test(zset): split-tick rollback, integrate and step_continue costs; docs and evidence for the review of #355 (improvise-bw4p.5)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

context/code/zset-ffi.md

@@ -95,7 +95,7 @@ link, so they call the `extern "C"` functions as plain functions.
     the host rolls back. `rollback` is the only call that clears the poison, and only a poison
     that came from a step inside the transaction. This is additive, so `ZS_ABI_VERSION` stays 1.
     What a rollback cannot undo is a callback's own side effects and a `ZsRebuild` stepped in the
-    transaction (it answers `ZS_ERR_INTERRUPTED` after, and the host starts another job). A
+    transaction, finished or not (it answers `ZS_ERR_INTERRUPTED` after, and the host starts another job; `rollback_abi.rs`). A
     circuit left mid-step by a non-local exit answers `ZS_ERR_POISONED` to a rollback like to
     everything else: the frames are skipped and the engine is not touched.
 
@@ -124,7 +124,7 @@ resolve to whatever is current, and only the nightly date is fixed. It passed at
 this commit (about 12s):
 
 ```
-nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi
+nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi --test rollback_abi
 ```
 
 It is not a CI job. A proposed one: a `miri` job in `.github/workflows/ci.yml` that takes the

context/code/zset.md

@@ -198,7 +198,16 @@ clause's wording stands; this is how the crate reads it).
    the log holds what the steps touch: the cost of a transaction is the keys a step touches,
    not the circuit's size (`a_transaction_costs_the_keys_a_step_touches_and_not_the_size_of_the_circuit`
    measures it in key clones: 16 extra for `begin; step; commit`, 22 for `rollback`, the same at
-   10 and at 300 keys of state). Outside a transaction nothing is logged and nothing extra is cloned.
+   10 and at 300 keys of state, with an `integrate` in the pipeline). Outside a transaction
+   nothing is logged and nothing extra is cloned. Two costs are the circuit's size, and are
+   limits, not features:
+   - **A split tick.** A `step_continue` that is the first publish of an edge in a transaction
+     clones that edge's whole value (2009 key clones against 1006 outside one, at N=1000;
+     `a_split_tick_in_a_transaction_costs_the_size_of_the_edge`).
+   - **Memory held until `commit`.** Every edge's previous value is moved into the slot's undo
+     record and kept, so an `integrate -> delay` pipeline holds 4022 live keys during a
+     transaction against 2014 without one (N=1000). An integrate edge is back to three copies
+     of its integral for the length of the transaction.
 2. **What is logged, per node.** Each integrated Z-set (`integrate`, `distinct`, a sink's sum)
    keeps `(key, weight)` for every addition it applied; each trace (the join's two, the
    aggregate's groups) keeps `(key, value, weight)` for every `apply`, through `Undoable`, a
@@ -212,10 +221,13 @@ clause's wording stands; this is how the crate reads it).
    (`rollback_undoes_newest_first_so_no_undo_leaves_i64`). `i64::MIN` has no negation; a
    trace undoes it as `MAX` then `1` (the weight before was non-negative, so neither leaves
    `i64`), and a Z-set undoes it by `adjust(.., subtract)`.
-4. **An entry is logged after its addition returned.** An overflow panics before a key changes
-   (`insert` is one checked addition), so a step that dies half way has logged exactly the
-   keys it applied. A custom `Trace` must do the same: a panicking `apply` leaves the trace as
-   it was. `MemTrace` does.
+4. **An entry is built before its change and logged after it.** An overflow panics before a
+   key changes (`insert` is one checked addition), and the entry's clones are made first, so
+   a panicking `Clone` changes nothing either (`c1_a_clone_panic_anywhere_in_a_step_is_rolled_back`
+   sweeps a panic over every key clone in a step). A step that dies half way has therefore
+   logged exactly the keys it applied. A custom `Trace` must do the same: a panicking `apply`
+   leaves the trace as it was. `MemTrace` does, since it makes every clone before it touches
+   its map (`c2_memtrace_value_clone_panic_leaves_an_empty_group`).
 5. **The edge's two copies.** An integrate stream's piece is also its output, so a second copy
    (the whole-tick value) only exists while a tick is split. `Slot::whole` is `None` otherwise
    and `current()` reads the piece. `piece()` lends a borrow rather than a clone
@@ -225,9 +237,13 @@ clause's wording stands; this is how the crate reads it).
 6. **What a transaction does not undo.**
    - A closure's captured state (a `map` function that counts, an FFI callback's side
      effects). Operators that keep state keep it in the crate's own types for this reason.
-   - A `Rebuild`: it owns an iterator over its snapshot. A rolled-back chunk leaves the job
-     ahead of the circuit, and the clock rolling back makes `interrupted` true, so the job
-     reports itself dead and the host starts another.
+   - A `Rebuild`: it owns an iterator over its snapshot, so a rolled-back chunk leaves the job
+     ahead of the circuit. The clock cannot tell: `rollback` rewinds `ticks`, which can make
+     a dead job's recorded tick current again. So the circuit also counts the rollbacks that
+     undid a step or piece (`undone`, never rewound), and a job records the count when it first
+     runs, before its first chunk can panic. A later count makes `interrupted` true for good,
+     finished or not (`rollback_rebuild.rs`: begun before the transaction and continued inside,
+     finished inside, a first chunk that panicked, and a dead job revived by a later step).
    - An `Output::take` made inside the transaction: a rollback subtracts the step's output
      from the sink, which a `take` has emptied. Take after `commit`; `zset-ffi` refuses to take
      while a transaction is open.
@@ -249,6 +265,6 @@ A `Rebuild` takes its circuit's identity from its `Input`: the circuit holds an
 `Input` it makes holds a clone, so `Rebuild::belongs_to(&circuit)` is a pointer comparison and no
 global counter is needed. `Rebuild::step` on another circuit panics, where it used to report
 `Done` and land its chunk in that circuit's next tick. An interrupted job (decision 8 above) can
-never finish: `interrupted` stays true however many steps follow, because the tick count it
-recorded is gone. Drop it and start another job from the same snapshot; the host keeps the
+never finish: `interrupted` stays true however many steps follow, because a split tick cannot
+be mended and a rollback that took a chunk is counted forever (decision 6 above). Drop it and start another job from the same snapshot; the host keeps the
 snapshot.

context/test-evidence/zset-ffi.md

@@ -195,3 +195,23 @@ panic: the sink holds nothing after the rollback.
 while one is open (nothing written to the out-pointer). A rollback after a poison from outside any
 transaction is also refused and the circuit stays poisoned. Rivals: `take` allowed in a
 transaction, and a second `begin` allowed, each fail this test.
+
+## Review of #355 at 51438911
+
+### f1_rebuild_begun_before_a_txn_continued_inside_and_rolled_back
+
+Through the ABI, a job continued inside a rolled-back transaction returned 0 with `done` 1 and
+a partial load. It is now `ZS_ERR_INTERRUPTED`.
+
+### f4_a_failed_rebuild_step_in_a_txn_then_rollback_then_resume
+
+A job whose first chunk failed inside the transaction resumed without that chunk; now
+`ZS_ERR_INTERRUPTED`.
+
+### f2_a_rollback_from_inside_a_callback_is_refused_and_the_txn_stays_open
+
+`ZS_ERR_POISONED` from the callback's rollback, and the transaction still commits.
+
+### f3_begin_on_a_circuit_poisoned_outside_a_txn_is_poisoned
+
+`begin` is `ZS_ERR_POISONED`; `commit` and `rollback` are `ZS_ERR_TRANSACTION`.

context/test-evidence/zset.md

@@ -456,7 +456,10 @@ loaded with 10, then 300 keys, then given a two-key delta in each of two inputs.
 clones of `begin; step; commit` over the same step outside a transaction are **16** at both
 sizes (22 for `rollback`; the plain step is 20). Rival: `DistinctState::begin` clones what it
 holds (a snapshot of the world) fails `[cost]` with 28 against 318. A `delay` clones what it
-holds, the previous tick's input, so the test steps a small delta after the load.
+holds, the previous tick's input, so the test steps a small delta after the load. Review of
+#355: the pipeline now includes an `integrate`; a rival `Acc::begin` that clones the integral
+fails `[cost]`. (An `integrate` whose output is sunk is left out: its output is the integral,
+so the sink logs all of it, which is the delta and not an overhead.)
 
 ### a_rolled_back_step_leaves_the_circuit_as_it_was_and_the_ticks_after_unchanged
 
@@ -520,3 +523,69 @@ fails `[owner]`.
 bw4p.12. Documents, and pins, that an interrupted job can never finish: later steps never
 clear it, and a new job over the same snapshot loads it. It did not fail before: the behaviour
 was right and unstated.
+
+## Review of #355 at 51438911 (improvise-bw4p.5, .12)
+
+An adversarial review found a rebuild across a rollback losing data, a panicking `Clone`
+mutating unlogged state, and two cost claims that did not hold. Its reproductions are adopted
+as tests; each rival below was injected and watched fail by name.
+
+### r1_rebuild_begun_before_txn_continued_inside_then_rolled_back_reports_interrupted
+
+A job begun outside a transaction and continued inside, then rolled back, was not interrupted;
+finishing it gave `{1,3,4}` for the snapshot `{1,2,3,4}`. Rival: `rollback` does not count the
+undone step fails this and r2 to r4.
+
+### r2_rebuild_finished_inside_a_rolled_back_txn_is_reported_done_with_nothing_loaded
+
+A job that finished inside a rolled-back transaction was Done with an empty integral and not
+interrupted.
+
+### r3_an_interrupted_rebuild_resurrects_after_rollback_and_a_later_step
+
+`rollback` rewinds `ticks`, so a later `step` made a dead job's recorded tick current again and
+it continued without its first chunk.
+
+### r4_rebuild_whose_first_chunk_panics_in_a_txn_then_rolled_back
+
+A first chunk that panicked left `began_at` unset, and the job resumed without it. The count is
+now recorded before the chunk runs. Rival: recording it never (the `get_or_insert` removed)
+fails all four; a rival recording it after the chunk was not run separately.
+
+### b1_a_closure_that_reads_a_stream_during_eval_does_not_panic
+
+The borrowed `piece()` does not conflict with a closure that reads streams during an eval,
+including its own source, inside a transaction. It passed before and after.
+
+### c1_a_clone_panic_anywhere_in_a_step_is_rolled_back
+
+Sweeps a panic over every key clone of one transactional step through `integrate`, `distinct`,
+`join`, `delay`, `aggregate` and sinks; 138 panic points left state that differed after the
+rollback. The post-rollback tick now pushes the keys the failed step touched, because a leaked
+key is only visible when it is touched again (a rival that mutated `DistinctState` before
+cloning its entry survived the first version). Rivals, each failing it: `add_logged`,
+`DistinctState::step` and `Undoable::apply` each mutating before cloning the entry.
+
+### c2_memtrace_value_clone_panic_leaves_an_empty_group
+
+`MemTrace::apply` inserted an empty group before cloning the value. Rival: that order restored
+fails `[c2]`.
+
+### reviewer_rollback_matches_a_circuit_that_never_ran_the_step
+
+The reviewer's model-based property (random circuits over every public operator, a custom
+panicking trace, random push/step/continue/begin/commit/rollback histories, operators built in
+a transaction), at the default 256 cases here (30,000 passed in review). Rival: the continued
+path saving the slot without its `whole` fails it.
+
+### a_rollback_across_a_split_tick_restores_the_whole_value
+
+The tick is split before the transaction opens, so every edge holds a `whole` when the first
+continued piece saves it. Rival: that save dropping `whole` fails `[split]` (an earlier version
+split the tick inside the transaction and never reached the save).
+
+### a_split_tick_in_a_transaction_costs_the_size_of_the_edge
+
+Pins the limit `zset.md` states: a first continued piece in a transaction clones the edge's
+whole value, growing with the state.
+