Merge dbsp/live-source into dbsp/pushdown

# Conflicts:
#	context/code/source.md
#	context/test-evidence/core.md
#	context/test-evidence/improvise-incremental.md
#	crates/improvise-core/src/model/source.rs
This commit is contained in:
Claude (dbsp-sources)
2026-10-10 19:02:59 -07:00
committed by Edward Langley
35 changed files with 3134 additions and 136 deletions
+18 -1
View File
@@ -119,6 +119,14 @@ dead host's replies are refused. A duplicate answer also prunes every queued ent
and moves `seen` onto the live epoch, so a late notice from a dead host costs one extra round and cannot
livelock `sync`.
**The host that answers a request is live, whatever the client believed.** `left` is a guess, because
epochs of different hosts are unordered: a late reply from a host the client never joined moves it onto
that host, and writes off the live one. Refusing the live host's answers as "left" then strands the client
(`sync` returned `Exhausted`, found by the property test). So `apply_answer`, which `sync` and `poll` use
for the reply to a request they just sent, takes an epoch off `left` before it applies. `apply` is for a
reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round;
the next request it makes is answered by the live host and brings it back.
`poll` hands a pulled notice only to the shape it asked about. A notice carries only that shape's keys,
and applying it to another tracked shape would advance that shape past keys it never saw.
@@ -182,7 +190,9 @@ counts comparisons on a counting item type rather than reading a clock.
**The host instance** is a random id, never a time (`Host::new` takes one from `RandomState`).
`RandomState` has no entropy on `wasm32-unknown-unknown`, so a worker must pass a real one through
`Host::with_instance`. A restarted host with the same id would accept a stale watermark as current.
`Worker` still calls `Host::new` (improvise-bw4p.7).
`Worker` still calls `Host::new` (improvise-bw4p.7). Such a restart, with the same instance id and
generation 0, is not recognised as a restart: the client's `older` guard refuses its replies as coming from
an older generation of the host it is on (improvise-bw4p.7).
**Eviction** (`evict_idle`) takes a local count and an idle length in the same unit, never a wall clock
compared across machines. An evicted key is announced in the next notice and forgotten; naming it again
@@ -207,6 +217,13 @@ The token in a request is carried and echoed by the client and never read by the
## 8. Edge adapters
**Transport contract.** A reply must reach the call that made the request. `Request` and `Correction`
carry no id, so the client cannot tell a late reply from an answer on its own; `sync` and `poll` treat
whatever `Edge::call` returns as the answer (`apply_answer`), and only a reply handed to `apply` is
treated as late. A transport that reorders or duplicates replies must therefore correlate them itself
or drop the strays (improvise-bw4p.10 adds the id to the frames).
`Edge` is one end of a stream: `call(Request) -> Result<Reply, EdgeError>`. `InProcess` is the
function call; `WireEdge` passes request and reply through JSON text, the adapter at a wire boundary,
so the lossless properties are exercised by every test that uses it. Neither adds a `ClientFrame` or
+7 -3
View File
@@ -407,13 +407,17 @@ from the source's folds, never pushed down; `Derived` stays in this crate.
what a database does inside a read transaction and what `DataStore` does already (its log is
drained in the step that made the edit). `&self` because a live source is shared with the model that
reads it and changes behind it by interior mutability of its own.
2. **`Incremental::over(model, feed, none_cats)`**: the model is put over the feed (`Model::over`) and
keeps no cells of its own. Its categories, formulas and aggregations are the definition; the
2. **`Incremental::over(model, feed, none_cats)`** takes a `SharedFeed`, the alias core exports (a generic
`Arc<S>` could not take it: `S` is `Sized`). The model is put over the feed (`Model::over`), which
drops the cells it came with, so none is left for `get_cell`, persistence or the harness to read. Its categories, formulas and aggregations are the definition; the
source holds the data. Shapes and stems are built from the feed as it stands.
3. **`sync()` drains and maintains; `apply_changes(&[CellChange])` maintains changes the source already
reflects**, for a source whose changes reach the backend by another route (a trigger table, a
channel). `Maintained::apply` on a fed backend takes definition edits (a formula, an aggregation)
and drains: a cell edit is the source's to make, and `Model::set_cell` refuses it in a debug build.
and drains: a cell edit is the source's to make, and `apply` refuses it with an `assert!` in every
build (a release build lost it silently when only `Model::set_cell`'s `debug_assert` stood in the way).
**A feed has one drainer**: a drain is destructive, so two backends over one feed split its changes
(`context/code/source.md`, `ChangeSource`).
4. **Freshness is the source's.** ADR 003 left "polled, triggered, or stale until refreshed" open: `sync`
is the poll, and when it is called is the host's decision. Nothing here schedules it.
5. **What a change costs** is what it costs for the owned store: two group updates and two exact-sum
+18 -4
View File
@@ -67,10 +67,13 @@ the owned `DataStore`. That is the whole evaluator hook (improvise-rnak, bw4p.14
there is still one evaluator, `Model`'s, and it computes every formula value, over a source whose
cells it never owns.
1. **A model over a source is read-only.** `set_cell` and `clear_cell` `debug_assert` that no source
is set, because they write the owned store, which is then not read. A release build would drop the
write silently; the assert is the guard, not the design. The design is that a caller who holds a
source writes to the source (ADR 003: the trait has no writes), and the model is built over it.
1. **A model over a source is read-only, in every build.** `set_cell`, `clear_cell`, `remove_category` and
`remove_item` `assert!` that no source is set (a `debug_assert` first shipped and was lost in release:
the write went to the owned store, which nothing reads). A caller who holds a source writes to the
source (ADR 003: the trait has no writes). `Model::over` drops the model's own cells, so no stale copy
is left for `get_cell`, persistence or the harness to read. `data` is `pub`, so a caller can still write
it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed)
are refused too, because the source's cells would still carry them.
2. **A shared source changes behind its readers by interior mutability of its own.** The model holds
an `Arc`, so a database connection or a test double behind a lock is the same object the
backend feeds from.
@@ -112,3 +115,14 @@ fix one, or the passes of a document whose stems read each other may differ betw
`Select::admits` is the predicate of a select over one key, for a source that must test keys itself
(`DataStore::count`). `TableSource` does not use it: its own `holds` is the independent second reading.
backend and a model share. **A feed has one drainer**: a drain empties the pending list, so two
backends over one feed split the changes between them (measured in review: the same write gave one
backend a Revenue of 128 and the other 38, and the second then disagreed with its own exact reads).
`a_feed_has_one_drainer` pins the behaviour. Kb-style invariant: `invariant(change_source, one_drainer)`.
## Building a `Summary` from an engine's answer
`Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest), Agreement)`. `Agreement`
is what the non-number values hold: `None`, `One(v)` or `Mixed`. `from_parts` drops extremes that
contradict the count or are NaN, so a source cannot hand the evaluator a state `add` could not reach.
+22 -6
View File
@@ -33,11 +33,13 @@ link, so they call the `extern "C"` functions as plain functions.
4. **Failure leaves the Z-set unchanged.** `add`, `sub`, `neg`, `scale` and `decode_add`
compute a new value and then assign it, so an overflow half way through cannot leave a
half-applied update (`zs_zset_insert` is one addition, so it has no half).
5. **A failed step poisons the circuit.** A panic in `step` leaves operator state half
updated, and the circuit has no rollback (ADR 009 clause 11 is the host's; `zset.md`). So the first
failure in a step, or in a rebuild step, marks the circuit poisoned, and every later call
that uses it returns `ZS_ERR_POISONED`. A caller drops it and builds another. Ids that do
not exist, and malformed arguments, are refused before any state changes and do not poison.
5. **A failed step poisons the circuit, and a rollback revives it.** A panic in `step` leaves
operator state half updated. So the first failure in a step, or in a rebuild step, marks
the circuit poisoned, and every later call that uses it returns `ZS_ERR_POISONED`. The one
way out is `zs_circuit_rollback` of a transaction that the failed step ran in
(decision 13); a poison from outside a transaction is permanent, and the caller drops the
circuit and builds another. Ids that do not exist, and malformed arguments, are refused
before any state changes and do not poison.
6. **Buffers.** The caller owns every buffer it passes and may reuse it once the call
returns; the library copies. The library hands out no memory except handles. Output of
variable size uses a two-call protocol: `zs_zset_encode(z, buf, cap, &needed)` always sets
@@ -83,6 +85,20 @@ link, so they call the `extern "C"` functions as plain functions.
12. **Versioning.** `zs_abi_version()` and `ZS_ABI_VERSION` are 1. A change that breaks a
signature or a code bumps it.
13. **Transactions** (improvise-bw4p.5; `zset.md`, "Transactions"). `zs_circuit_begin`,
`zs_circuit_commit` and `zs_circuit_rollback` wrap `Circuit::begin/commit/rollback`; a
handle-returning `snapshot` was not chosen, because the circuit logs what changes and there
is no snapshot object to free. The new code is `ZS_ERR_TRANSACTION` (10) for a `begin` on an
open transaction, a `commit` or `rollback` with none open, and a `zs_circuit_take` while
one is open (take after commit: a rollback subtracts the step's output from the sink).
`begin` on a poisoned circuit is `ZS_ERR_POISONED`, and so is `commit` after a failed step:
the host rolls back. `rollback` is the only call that clears the poison, and only a poison
that came from a step inside the transaction. This is additive, so `ZS_ABI_VERSION` stays 1.
What a rollback cannot undo is a callback's own side effects and a `ZsRebuild` stepped in the
transaction, finished or not (it answers `ZS_ERR_INTERRUPTED` after, and the host starts another job; `rollback_abi.rs`). A
circuit left mid-step by a non-local exit answers `ZS_ERR_POISONED` to a rollback like to
everything else: the frames are skipped and the engine is not touched.
A rebuild is one logical tick however many `zs_rebuild_step` calls it takes (`zset.md`,
decision 8): a `delay` fed by it shows the previous tick once, and `done` is 1 as soon as the
last chunk ran, whatever the deadline callback says.
@@ -108,7 +124,7 @@ resolve to whatever is current, and only the nightly date is fixed. It passed at
this commit (about 12s):
```
nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi
nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi --test rollback_abi
```
It is not a CI job. A proposed one: a `miri` job in `.github/workflows/ci.yml` that takes the
+84 -4
View File
@@ -124,6 +124,7 @@ Clause 1's deferred scope, built without ADR 008. The ADR is not edited; this is
| `aggregate`, `aggregate_in` | group-by, re-folds only touched keys (`AggregateState`) |
| `Trace`, `MemTrace` | operator state behind lookup, range, apply |
| `Rebuild`, `Progress` | a snapshot loaded as a resumable job |
| `begin`, `commit`, `rollback` | an undo-log transaction over the circuit's state (below) |
Decisions, where the ADR left a choice:
@@ -185,7 +186,86 @@ Decisions, where the ADR left a choice:
The tests are `crates/zset/tests/circuit.rs`; the rivals and where they failed are in
`context/test-evidence/zset.md`.
A panic inside `step` (a weight overflow in an operator, say) leaves the circuit unusable:
nodes ahead of the panic have updated their state and the rest have not, and there is no
rollback. Drop the circuit and build another. ADR 009 clause 11's per-command snapshot and
restore is the host's job, not this crate's; `zset-ffi` poisons the circuit for this reason.
## Transactions (improvise-bw4p.5)
A panic inside `step` (a weight overflow in an operator, say) leaves nodes ahead of the panic
updated and the rest not. ADR 009 clause 11 says each command snapshots the circuit for
rollback; `Circuit::begin`, `commit` and `rollback` are that, and the ADR is not amended (the
clause's wording stands; this is how the crate reads it).
1. **An undo log, not a snapshot.** The ADR budgets memory, and a clone of the circuit's state
is the size of the circuit. `begin` instead makes every stateful node start logging, and
the log holds what the steps touch: the cost of a transaction is the keys a step touches,
not the circuit's size (`a_transaction_costs_the_keys_a_step_touches_and_not_the_size_of_the_circuit`
measures it in key clones: 16 extra for `begin; step; commit`, 22 for `rollback`, the same at
10 and at 300 keys of state, with an `integrate` in the pipeline). Outside a transaction
nothing is logged and nothing extra is cloned. Two costs are the circuit's size, and are
limits, not features:
- **A split tick.** A `step_continue` that is the first publish of an edge in a transaction
clones that edge's whole value (2009 key clones against 1006 outside one, at N=1000;
`a_split_tick_in_a_transaction_costs_the_size_of_the_edge`).
- **Memory held until `commit`.** Every edge's previous value is moved into the slot's undo
record and kept, so an `integrate -> delay` pipeline holds 4022 live keys during a
transaction against 2014 without one (N=1000). An integrate edge is back to three copies
of its integral for the length of the transaction.
2. **What is logged, per node.** Each integrated Z-set (`integrate`, `distinct`, a sink's sum)
keeps `(key, weight)` for every addition it applied; each trace (the join's two, the
aggregate's groups) keeps `(key, value, weight)` for every `apply`, through `Undoable`, a
wrapper over the caller's `Trace`; a `Delay` keeps one clone of what it holds on its first
eval, which is the previous tick's input; a `Source` keeps a clone of what was pending at
`begin`; an edge (`Slot`) keeps the `(piece, whole)` its first publish replaced, moved and
not cloned, except for a continued piece, which clones the edge once. `ticks` is restored.
3. **Rollback replays newest first.** Each undo is the exact inverse of an addition that
succeeded, so it never leaves `i64`, but only in this order: a key at `MAX` lowered and
raised again would reach `MAX + 1` undone oldest first
(`rollback_undoes_newest_first_so_no_undo_leaves_i64`). `i64::MIN` has no negation; a
trace undoes it as `MAX` then `1` (the weight before was non-negative, so neither leaves
`i64`), and a Z-set undoes it by `adjust(.., subtract)`.
4. **An entry is built before its change and logged after it.** An overflow panics before a
key changes (`insert` is one checked addition), and the entry's clones are made first, so
a panicking `Clone` changes nothing either (`c1_a_clone_panic_anywhere_in_a_step_is_rolled_back`
sweeps a panic over every key clone in a step). A step that dies half way has therefore
logged exactly the keys it applied. A custom `Trace` must do the same: a panicking `apply`
leaves the trace as it was. `MemTrace` does, since it makes every clone before it touches
its map (`a_panicking_value_clone_leaves_a_memtrace_unchanged`).
5. **The edge's two copies.** An integrate stream's piece is also its output, so a second copy
(the whole-tick value) only exists while a tick is split. `Slot::whole` is `None` otherwise
and `current()` reads the piece. `piece()` lends a borrow rather than a clone
(improvise-bw4p.11). An integrate stream now holds its integral twice (the node's `acc`
and the stream's piece), not three times; sharing those two needs an `Rc` in the slot and
was not done.
6. **What a transaction does not undo.**
- A closure's captured state (a `map` function that counts, an FFI callback's side
effects). Operators that keep state keep it in the crate's own types for this reason.
- A `Rebuild`: it owns an iterator over its snapshot, so a rolled-back chunk leaves the job
ahead of the circuit. The clock cannot tell: `rollback` rewinds `ticks`, which can make
a dead job's recorded tick current again. So the circuit also counts the rollbacks that
undid a step or piece (`undone`, never rewound), and a job records the count when it first
runs, before its first chunk can panic. The chunk's `push` is marked as a step too, since it
can overflow the pending delta before any step runs (`p1_a_chunk_whose_push_panics_in_a_txn_is_counted`). A later count makes `interrupted` true for good,
finished or not (`rollback_rebuild.rs`: begun before the transaction and continued inside,
finished inside, a first chunk that panicked, and a dead job revived by a later step).
- An `Output::take` made inside the transaction: a rollback subtracts the step's output
from the sink, which a `take` has emptied. Take after `commit`; `zset-ffi` refuses to take
while a transaction is open.
7. **Pushes.** `Input::push` is not logged (a handle has no circuit). `rollback` puts the
pending delta back to what it was at `begin`, so a push made after `begin` is dropped, and
one made before it is pending again. Operators built after `begin` start logging at once
and go back to empty; they are not removed.
8. **One transaction at a time, and misuse panics** like an overflow does: `begin` while open,
`commit` or `rollback` with none open. `zset-ffi` answers with a code instead.
The tests are `tests/rollback.rs` (the property: a step that panics half way, then a rollback,
leaves every stream and sink as before and the later ticks equal to a circuit that never ran
the step) and `tests/cost.rs`. A step that panics outside a transaction still leaves the
circuit unusable: drop it and build another.
## Rebuilds and circuits (improvise-bw4p.12)
A `Rebuild` takes its circuit's identity from its `Input`: the circuit holds an `Rc<()>` and every
`Input` it makes holds a clone, so `Rebuild::belongs_to(&circuit)` is a pointer comparison and no
global counter is needed. `Rebuild::step` on another circuit panics, where it used to report
`Done` and land its chunk in that circuit's next tick. An interrupted job (decision 8 above) can
never finish: `interrupted` stays true however many steps follow, because a split tick cannot
be mended and a rollback that took a chunk is counted forever (decision 6 above). Drop it and start another job from the same snapshot; the host keeps the
snapshot.
+42
View File
@@ -2401,3 +2401,45 @@ that does not exist, equal and in the same order, each stem once. Rivals, each f
the stems not deduplicated, sorted instead of first seen (this is the only test that sees it: the
evaluator's results do not depend on stem order for the fixtures), and the dropped categories kept (also
fails several evaluator tests).
### a_summary_built_from_an_engines_parts_reduces_as_one_built_from_the_values
Review of #356 (Codex): an engine outside core could not build a `Summary` without replaying rows. The
parts (count, exact sum, extremes, agreement of the non-numbers) are computed in the test without
`Summary::of`, for the 585 sequences of the reference test. Rivals: the greatest taken from the least;
a NaN allowed as an extreme. Each fails it.
### parts_that_contradict_themselves_are_made_consistent
Extremes with no numbers, and a NaN as an extreme, are dropped by `Numbers::from_parts`.
### keys_of_a_store_are_its_cells_keys_without_the_values
Review of #356 (Codex): `DataStore::keys` went through `cells` and cloned every value. It reads keys
only now; the test pins the keys and their order against `cells`. No clone-count test: a value clone is
not observable from outside, so the equality is all that can be asserted.
### a_model_over_a_source_refuses_a_clear
Review of #357: the `clear_cell` guard had no test. The refusal is an `assert!`, so the test holds in a
release build too (`cargo test --release -p improvise-core --lib over_tests` was run: the earlier
`debug_assert!` failed all four refusal tests there, `a_model_over_a_source_refuses_a_write` among
them, and was the bug). Rival: the guard removed from `clear_cell` fails this test only.
### a_model_over_a_source_refuses_to_remove_a_category
`remove_category` swept the owned store with no guard. Rival: the guard removed.
### a_model_over_a_source_refuses_to_remove_an_item
The same for `remove_item`.
### putting_a_populated_model_over_a_source_drops_its_own_cells
`Model::over` empties the owned store, so `get_cell` on an over-model is `None` for a cell the
source holds, and the evaluator still reads the source's value.
### records_mode_reads_the_records_from_a_source
Review of #357: a records-mode `GridLayout` over a source. `group_records` reading the owned store
left all 563 tests green; with this test it fails (the grid has no rows).
+20
View File
@@ -172,3 +172,23 @@ host, and the old reply then arrives. Rival: the `left` check removed.
every restart and swap. Rival: the `left` check removed fails it. The older-generation guard on its own is
not separable from `left` in the histories the property reaches, so it is held by
`a_late_window_reply_from_an_older_generation_does_not_regress` alone.
## The left-epoch livelock
### a_late_reply_from_a_host_the_client_never_joined_cannot_strand_it
The property test failed with `Exhausted` on a shrunk history of two restarts and four writes. Cause: the
client was on host 3, never having joined host 2; a late host-2 reply was applied (an unknown epoch is
accepted), which wrote host 3 into `left`, and every later answer from the live host 3 was then refused as
a duplicate. The test replays that history by hand: three hosts, the host-2 reply lost in flight, the client
settling on host 3, the late reply applied, the connection lost, and a sync that must settle on host 3's
grid. Rival: `apply_answer` treated like `apply` (the `left.retain` removed) fails this test. **The random
property does not catch it**: it passed 11 of 11 runs under that fault, and again after the late reply was
made an explicit event (a stashed reply from a host the client never joined, replayed before a lost
connection), because the property rarely produces a reply from a host the client never joined. This
test is the pin. Rival: the left-epoch guard removed fails
`a_reply_held_across_a_host_restart_cannot_pull_the_client_back` and the property, as before.
The property's invariant changed with it. "The watermark never returns to an epoch it left" was false by
design once an unknown-epoch reply can move the client, so the property now checks the narrower, true
rule: a replayed reply from an epoch the client was in and has left must be a duplicate.
@@ -899,3 +899,28 @@ occupancy; a sound source over the same fixture must pass.
`Pushdown` over `Remote` against `Eager` over random formula and cell edits on the formula fixture
(cycles, text, NaN and `-0.0` among the values), in its own binary. Rival: a refresh that does
nothing.
### a_backend_can_be_built_over_the_exported_feed_alias
Review of #358: `over<S>(Arc<S>)` could not take the `SharedFeed` the same PR exports. The test builds
a backend from a `SharedFeed`; it did not compile against the generic signature.
### a_populated_model_put_over_a_feed_keeps_no_cells_of_its_own
Review of #358: `Fed::new` emptied the store before the call, so no test passed a populated model. This
one does, asserts the owned store is empty afterwards and that every harness probe agrees with
`Eager`. Rival: `Model::over` keeping the cells fails it, and only it.
### a_cell_edit_on_a_fed_backend_is_refused
`Maintained::apply` with a `SetCell` panics with the backend's own message in every build. Rival: the
assert removed (the test then sees the model's message, not the backend's, and fails).
### a_cell_clear_on_a_fed_backend_is_refused
The same for `ClearCell`.
### a_feed_has_one_drainer
Review of #358: two backends over one feed; one write; `a.sync(); b.sync()`. The first sees it, the
second does not. This pins the behaviour as documented rather than as a guarantee: a drain is destructive.
+55
View File
@@ -166,3 +166,58 @@ A job outlives its circuit; sixty-four new circuits (some at the old address) al
Retyping `ZsKeepFn`'s return in the header fails. The `unary_op!` signatures are now read
from the macro body rather than written into the test.
## Transactions (improvise-bw4p.5)
Rivals were injected one at a time in `engine.rs` and watched fail by name; Miri passes on `abi`.
### a_rollback_after_a_failed_step_revives_the_circuit_in_its_old_state
An integrate held at `i64::MAX`; inside a transaction a step overflows (`ZS_ERR_OVERFLOW`, then
`ZS_ERR_POISONED`), `commit` of it is refused as poisoned, and `rollback` revives the circuit with
the integral where it was: `-1` then gives `MAX - 1`, with neither the failed `+1` nor the key
added beside it. Rivals: a rollback that leaves the poison on, and one that does not roll the
circuit back, fail this test and the callback one; a `commit` that does not check the poison
fails this test.
### a_committed_transaction_keeps_its_steps_and_a_new_one_may_follow
`commit` keeps the step and its sink, and a later transaction can be opened and rolled back.
### a_failed_callback_step_is_rolled_back_too
A failure the callback only reports (the step returns, the fault is noted) is rolled back like a
panic: the sink holds nothing after the rollback.
### transaction_misuse_is_a_code_and_changes_nothing
`ZS_ERR_TRANSACTION` for `commit` or `rollback` with none open, a second `begin`, and a `take`
while one is open (nothing written to the out-pointer). A rollback after a poison from outside any
transaction is also refused and the circuit stays poisoned. Rivals: `take` allowed in a
transaction, and a second `begin` allowed, each fail this test.
## Review of #355 at 51438911
### f1_rebuild_begun_before_a_txn_continued_inside_and_rolled_back
Through the ABI, a job continued inside a rolled-back transaction returned 0 with `done` 1 and
a partial load. It is now `ZS_ERR_INTERRUPTED`.
### f4_a_failed_rebuild_step_in_a_txn_then_rollback_then_resume
A job whose first chunk failed inside the transaction resumed without that chunk; now
`ZS_ERR_INTERRUPTED`.
### f2_a_rollback_from_inside_a_callback_is_refused_and_the_txn_stays_open
`ZS_ERR_POISONED` from the callback's rollback, and the transaction still commits.
### f3_begin_on_a_circuit_poisoned_outside_a_txn_is_poisoned
`begin` is `ZS_ERR_POISONED`; `commit` and `rollback` are `ZS_ERR_TRANSACTION`.
### f5_a_chunk_whose_push_overflows_in_a_txn_then_rollback_is_interrupted
The same path through the ABI: `zs_rebuild_step` returns `ZS_ERR_OVERFLOW` from the chunk's
push, `zs_circuit_rollback` succeeds, and the next `zs_rebuild_step` was a silent success; it is
now `ZS_ERR_INTERRUPTED`. Rival: the mark before the push removed fails `[f5]`.
+167
View File
@@ -429,3 +429,170 @@ A second job's first chunk is a tick, so the first job's next call panics.
### a_finished_rebuild_and_an_interrupted_check_do_not_trip_on_later_ticks
A finished job is never interrupted, and steps afterwards do not make it panic.
## `crates/zset/tests/{cost,rollback,rebuild}.rs`: rollback, the borrowed piece, the bound rebuild (improvise-bw4p.5, .11, .12)
Every rival below was injected one at a time and watched fail by test name (and tag where one
exists) before being reverted; the whole suite was re-run clean afterwards. Costs are counted
in key clones and live keys of a `Counted` key type, never in time, so they hold on any machine.
### an_eval_borrows_its_input_instead_of_cloning_it
bw4p.11. `input -> map(|k| k.0) -> output` over 100 keys makes **0** key clones in a step; before
the change it made 200 (the source's whole-tick copy and `piece()`'s clone). Rival: `Unary::eval`
clones the piece again fails `[piece]`.
### an_integrate_stream_holds_its_integral_twice_when_the_tick_is_whole
bw4p.11. After one step an `input -> integrate` pipeline holds 3N live keys (the input's
piece, the integral, the integrate stream's piece), where it held 5N before: the whole-tick copy
that duplicated the piece is kept only while a tick is split (`Slot::whole` is an `Option`).
Rival: a publish that always stores `whole = Some(v.clone())` fails `[whole]` (and the clone test).
### a_transaction_costs_the_keys_a_step_touches_and_not_the_size_of_the_circuit
bw4p.5. A pipeline with `distinct`, `delay`, `join_indexed`, `aggregate` and three sinks is
loaded with 10, then 300 keys, then given a two-key delta in each of two inputs. The extra key
clones of `begin; step; commit` over the same step outside a transaction are **16** at both
sizes (22 for `rollback`; the plain step is 20). Rival: `DistinctState::begin` clones what it
holds (a snapshot of the world) fails `[cost]` with 28 against 318. A `delay` clones what it
holds, the previous tick's input, so the test steps a small delta after the load. Review of
#355: the pipeline now includes an `integrate`; a rival `Acc::begin` that clones the integral
fails `[cost]`. (An `integrate` whose output is sunk is left out: its output is the integral,
so the sink logs all of it, which is the delta and not an overhead.)
### a_rolled_back_step_leaves_the_circuit_as_it_was_and_the_ticks_after_unchanged
bw4p.5, the property. A rig with every stateful operator (integrate, delay, differentiate,
distinct, join, aggregate, plus and eight sinks) runs a random prefix, then a key preloaded at
`i64::MAX - 1`. The failing step adds `(0,0): 1` and then `BIG: 5`, so operators have applied
the first key when the second overflows (the test asserts the step panicked iff it was meant
to). After `rollback`, every stream's `current()` and every sink equal what they were (`[rollback]`),
and the ticks after equal a reference circuit that never saw the step (`[post]`). The same test
rolls back a step that succeeds. `carry` is pending at `begin` and must stay pending.
Rivals, each failing this test: rollback that skips every node (`[rollback]`, and
`an_operator_built_inside_a_transaction_goes_back_to_empty`); an integrate log not undone
(`[post]`); a stream slot not restored (`[rollback]`); a `delay`'s held value not restored
(`[post]`); a source's pending delta not restored (`[post]`); a sink not undone (`[rollback]`);
`DistinctState` not undone (`[post]`); a trace not undone (`[post]`); an undo that drops the sign
(`[post]`); a log that records nothing (`[rollback]`, `[built]`).
### a_committed_transaction_equals_the_same_steps_outside_one
bw4p.5. A circuit that commits some of its steps equals one that never opened a transaction,
state and output, tick by tick. Rival: `commit` rolls back fails `[commit]`.
### rollback_undoes_the_clock
bw4p.5. A rolled-back step does not count as a tick. Rival: `ticks` not restored fails `[ticks]`.
### an_operator_built_inside_a_transaction_goes_back_to_empty
bw4p.5. An `integrate` built after `begin`, stepped, then rolled back, forgets the step
(`[built]`). Without the "a node built in a transaction starts recording" rule it kept it.
### rollback_undoes_newest_first_so_no_undo_leaves_i64
bw4p.5. A key at `i64::MAX` lowered by one and raised by one in one transaction. Undone oldest
first the first undo would reach `MAX + 1` and panic; undone newest first nothing leaves
`i64`. Rival: the log replayed oldest first fails this test (and no other: the property above
cannot see it, since its keys never sit at the boundary).
### a_minimum_weight_applied_to_every_kind_of_state_is_taken_back
bw4p.5. `i64::MIN` has no negation, so a trace undoes it as `i64::MAX` then `1`. Rival: the
second application dropped fails `[min]`.
### a_second_begin_is_refused_loudly
bw4p.5, with `a_rollback_with_nothing_open_is_refused_loudly` and
`a_commit_with_nothing_open_is_refused_loudly`: misuse panics, as an overflow does.
### a_rebuild_stepped_on_a_foreign_circuit_is_refused_loudly
bw4p.12. A job made from circuit A's input, stepped on circuit B, used to report Done and land
its chunk in B's next tick. It now panics. Rival: the check removed fails this test.
### a_rebuild_knows_the_circuit_its_input_came_from
bw4p.12. `Rebuild::belongs_to`, the question a host asks before the panic. Rival: always true
fails `[owner]`.
### an_interrupted_rebuild_stays_interrupted_and_a_new_job_loads_the_snapshot
bw4p.12. Documents, and pins, that an interrupted job can never finish: later steps never
clear it, and a new job over the same snapshot loads it. It did not fail before: the behaviour
was right and unstated.
## Review of #355 at 51438911 (improvise-bw4p.5, .12)
An adversarial review found a rebuild across a rollback losing data, a panicking `Clone`
mutating unlogged state, and two cost claims that did not hold. Its reproductions are adopted
as tests; each rival below was injected and watched fail by name.
### r1_rebuild_begun_before_txn_continued_inside_then_rolled_back_reports_interrupted
A job begun outside a transaction and continued inside, then rolled back, was not interrupted;
finishing it gave `{1,3,4}` for the snapshot `{1,2,3,4}`. Rival: `rollback` does not count the
undone step fails this and r2 to r4.
### r2_rebuild_finished_inside_a_rolled_back_txn_is_reported_done_with_nothing_loaded
A job that finished inside a rolled-back transaction was Done with an empty integral and not
interrupted.
### r3_an_interrupted_rebuild_resurrects_after_rollback_and_a_later_step
`rollback` rewinds `ticks`, so a later `step` made a dead job's recorded tick current again and
it continued without its first chunk.
### r4_rebuild_whose_first_chunk_panics_in_a_txn_then_rolled_back
A first chunk that panicked left `began_at` unset, and the job resumed without it. The count is
now recorded before the chunk runs. Rival: recording it never (the `get_or_insert` removed)
fails all four; a rival recording it after the chunk was not run separately.
### b1_a_closure_that_reads_a_stream_during_eval_does_not_panic
The borrowed `piece()` does not conflict with a closure that reads streams during an eval,
including its own source, inside a transaction. It passed before and after.
### c1_a_clone_panic_anywhere_in_a_step_is_rolled_back
Sweeps a panic over every key clone of one transactional step through `integrate`, `distinct`,
`join`, `delay`, `aggregate` and sinks; 138 panic points left state that differed after the
rollback. The post-rollback tick now pushes the keys the failed step touched, because a leaked
key is only visible when it is touched again (a rival that mutated `DistinctState` before
cloning its entry survived the first version). Rivals, each failing it: `add_logged`,
`DistinctState::step` and `Undoable::apply` each mutating before cloning the entry.
### a_panicking_value_clone_leaves_a_memtrace_unchanged
`MemTrace::apply` inserted an empty group before cloning the value. Rival: that order restored
fails `[c2]`.
### reviewer_rollback_matches_a_circuit_that_never_ran_the_step
The reviewer's model-based property (random circuits over every public operator, a custom
panicking trace, random push/step/continue/begin/commit/rollback histories, operators built in
a transaction), at the default 256 cases here (30,000 passed in review). Rival: the continued
path saving the slot without its `whole` fails it.
### a_rollback_across_a_split_tick_restores_the_whole_value
The tick is split before the transaction opens, so every edge holds a `whole` when the first
continued piece saves it. Rival: that save dropping `whole` fails `[split]` (an earlier version
split the tick inside the transaction and never reached the save).
### a_split_tick_in_a_transaction_costs_the_size_of_the_edge
Pins the limit `zset.md` states: a first continued piece in a transaction clones the edge's
whole value, growing with the state.
### p1_a_chunk_whose_push_panics_in_a_txn_is_counted
Delta review of #355 at b2445fed. `Rebuild::step` takes the chunk from its iterator and pushes
it before stepping; a push that overflows the pending delta panicked with no step run, so
`undone` was not bumped and the resumed job silently dropped the chunk (key 1's `+1`). The step
is now marked before the push. Rival: that mark removed fails `[p1]`.