improvise/context/code/zset.md

zset: the reasoning

crates/zset is ADR 009 step 1 (improvise-tk51, epic improvise-bw4p): the Z-set group, first and alone. Nothing depends on it yet, so it changes no behaviour. The decisions are the ADR's (context/adr/009-incremental-evaluation.md, clause 1). This file records how the crate reads them, where the ADR left a choice open, and the evidence discipline. The evidence for each test is in context/test-evidence/zset.md.

The shape

item	what it is
ZSet<K: Ord>	a map from K to a non-zero i64 weight; an abelian group within i64 (interpretation 5)
+, -, unary -, * i64, scale	the group operations and scalar multiplication
new, singleton, insert, FromIterator, Extend	construction; weights of equal keys sum
weight, is_zero, len, iter, IntoIterator	reads; iteration is ascending by key
map, filter, distinct	the three per-element operators
integrate, differentiate	the stream operators, over a slice

There are no dependencies: not serde (clause 9 keeps the wire form in improvise-protocol), not even the workspace's own crates. The crate is wasm32-clean by construction and is in nix/ci.nix's wasmCrates.

Canonical form

A key with weight zero is never stored. insert removes a key whose sum reaches zero, and singleton(k, 0) is the zero Z-set. Two Z-sets are therefore == exactly when every key has the same weight, so the derived PartialEq, Hash and Ord are the group's equality. The clause 9 correction target - held relies on it: a correct state gives == the zero Z-set, and is_zero is len() == 0, not a scan.

The representation is a BTreeMap. Iteration order is the key order, the same in every process and for every insertion order, which clause 3 needs (eager folds in CellKey order) and which a HashMap would give up (context/code/core.md, gotcha 18's argument).

Interpretations

The ADR is one paragraph on the crate, and says more about what it is for than what it is. These are the readings this step takes.

1. Clause 1 lists a Circuit and more operators; step 1 built the pure part, and "The circuit" below built the rest (improvise-bw4p.1). The ADR's list is delay, integrate, differentiate, map, filter, flat_map, index_by, join_indexed, distinct and aggregate, run by a synchronous Circuit, plus resumable rebuilds and a trace interface. Step 1 as ticketed is the group and the operators that are functions of a Z-set or of a stream of them: map, filter, distinct, integrate, differentiate. The rest needs the circuit (a clock, state held between steps), or a second key type (index_by, join_indexed, aggregate), and is left to the steps that use it rather than designed without a caller. flat_map is map into several images and has no user yet either.

2. integrate and differentiate are slice functions, not circuit nodes. integrate(&[d_0, d_1, ..]) is the running sums d_0 + .. + d_i; differentiate is its inverse, s_0 then s_i - s_(i-1). The state before the first element is zero, as in DBSP. delay is not provided: over a slice it is a shift, and a circuit gives it meaning.

3. distinct is not linear, and is not a group homomorphism. It keeps weight 1 on the keys whose weight is positive and drops every other key, negative included. That is the set-semantics operator of DBSP. The ADR says nothing about negative weights; dropping them is the standard reading, and the one that makes distinct idempotent.

4. Operators are methods on &self, group operations are std::ops traits. K: Clone is required only where a result copies keys (filter, distinct, scale, the borrowed operators); map and the by-value operators need only Ord.

5. Overflow. Neither clause 1 nor clause 8 states a policy for weights (clause 8 is about f64 drift, which wrapping integers do not have). The crate currently uses checked_* on every weight operation, for the reason given below, and panics with zset: weight overflow in <op>, in debug and in release (checked_* does not depend on overflow-checks). The operations are add (including insert, Extend, FromIterator), sub, neg (so i64::MIN panics when it has to be negated) and scale. Scaling by zero cannot overflow and returns the zero Z-set. A panic, not a Result: Result does not belong in the majority of the code (AGENTS.md), and a weight near i64::MAX is a defect in the caller, not an input to recover from. The weights clause 3 produces are small (one per cell).

   The group is partial, and that is the stated reading. The ADR calls a Z-set an abelian group. With i64 weights and no wrapping, the laws hold wherever no weight, intermediate ones included, leaves i64; outside that, an operation panics rather than return a different value. So associativity is order-dependent at the boundary: (MAX + 1) + (-1) panics where MAX + (1 + (-1)) is MAX. The result is never wrong, only sometimes refused. Wrapping arithmetic would make the type a true group (the integers mod 2^64), and (w + x) - x == w would still hold exactly, so a retraction does cancel its insertion under wrapping. (An earlier version of this paragraph said otherwise; that was wrong.) What wrapping breaks is the intermediate reads. MAX + 1 stores i64::MIN, so distinct drops a key whose true weight is positive and weight() has the wrong sign. A true weight of 2^64 stores as 0 (by the add path; a scale product of 2^64 would store a zero too, which breaks canonical form), so the key vanishes and is_zero, len and == report a set that does not exist. Those are wrong answers returned silently, which is why the crate panics instead. A wider weight (i128) moves the boundary and does not remove it. The ADR's "abelian group" wording needs amending whichever policy is chosen: checked-and-panic gives a group only within range, wrapping gives a group whose reads can be wrong, and i128 is partial too. Which one is a ruling for the user, not for this step; the crate currently panics, and overflow_depends_on_the_order_of_evaluation_and_never_gives_a_wrong_value states it. sub is checked directly against the held weight, so a - a is zero for every a, i64::MIN included, and only an absent key's 0 - w can overflow (w is i64::MIN).

6. Stable error text. The panic message names the operation and both operands, so a failure in a long-running host is diagnosable from the log.

What the tests are

crates/zset/tests/properties.rs holds one checker per property, parameterised by an Alg trait. The property's test runs it on ZSet. A companion runs it on Model(Fault::None), a second implementation over a sorted vector, so the checker is shown not to reject a correct implementation that is not ZSet. Each rival is Model(Fault::X), run through the same checker with a deterministic seed, and the failure message must carry the property's tag ([inverse], [map], ..). A rival that fails for another reason does not count. The group laws, canonical form, scalar distributivity, the map homomorphism, filter linearity, distinct's definition, iteration order, the clause 9 correction identity and the integrate/differentiate inverses are all properties; each runs at the default 256 cases over keys from a range of six, so collisions and cancelling pairs are the common case. Weights come from -3..=3, an explicit zero, and ±2^40, so sums of three cannot wrap.

The overflow policy is not a property over the generators (they cannot reach it by design) and is tested by example in src/lib.rs, with should_panic(expected = ..).

The circuit (improvise-bw4p.1)

Clause 1's deferred scope, built without ADR 008. The ADR is not edited; this is how the crate reads it.

item	what it is
Circuit	creation-order list of boxed nodes; step() runs each once
Input<K>, Stream<ZSet<K>>, Output<K>	push a delta in; the value on an edge; a sink that sums every step's delta until take()
delay, integrate, differentiate, plus	the stream operators as nodes
map, filter, flat_map, index_by	linear, per-element
distinct	incremental, state is the integrated input (DistinctState)
join_indexed, join_indexed_in	bilinear delta join of two ZSet<(K, _)> (JoinState)
aggregate, aggregate_in	group-by, re-folds only touched keys (AggregateState)
Trace, MemTrace	operator state behind lookup, range, apply
Rebuild, Progress	a snapshot loaded as a resumable job
begin, commit, rollback	an undo-log transaction over the circuit's state (below)

Decisions, where the ADR left a choice:

1. Wiring is apart from logic. circuit.rs is the only file with the arena and Box<dyn Node>. ops.rs holds each stateful operator as a plain struct with a pure step(delta) -> delta, testable and usable without a circuit.

2. Scheduling is creation order. A node is built from streams that already exist, so creation order is a topological order, and the circuit has no cycles: the ADR has no recursion operator, and integrate is its own stateful node, not a delayed feedback loop. delay is a node with one held Z-set. Send, not Sync: one thread steps a circuit at a time (clause 1; "Threads" below).

3. Incremental means the integral of the output is the operator of the integral of the input. That is the property every operator is tested against, over random delta streams. distinct and aggregate are not linear, so they keep the integrated input and emit the difference of the operator at the touched keys only.

4. index_by is map to (key, element). A keyed stream is a ZSet<(K, V)>; there is no separate indexed collection type, so the join and the aggregate read the same shape.

5. The join is da x b_prev + a_prev x db + da x db, computed as da against the right trace, da applied to the left, then db against the updated left (which holds the cross term), then db applied. Output weights are products and panic on i64 overflow like every weight (interpretation 5). f(&K, &A, &B) -> O builds the output key.

6. aggregate's fold gets the key and the group's (value, weight) pairs (owned, because a paged trace cannot lend references) and returns Option<R>; None means the group has no row (an empty group, or a Min of nothing). Sum, count, min and max are closures; text unanimity and the ADR's ExactSum belong to the caller (clause 4, 8), not this crate.

7. The trace interface is apply(k, v, w), lookup(k) and range(lo, hi), returning owned vectors. MemTrace is a BTreeMap<K, ZSet<V>>; the _in constructors take any Trace, which is the seam for paging or compression. An empty or inverted range reads nothing (BTreeMap::range would panic).

8. The resumable step takes a predicate, not an Instant, and a load is one logical tick. Rebuild::step(circuit, expired) loads chunk keys at a time and stops when expired() is true, always running at least one chunk first, so every call progresses. It reports Done as soon as the snapshot is exhausted, whatever expired() would say. Instant::now() panics on wasm32 (web-time is a dependency this crate refuses), so the host closes over its own clock. The first chunk is Circuit::step; the rest are step_continue, which does not advance the clock. A circuit tick that arrives in pieces is exact for the incremental operators because their outputs are deltas. For the temporal ones the pieces are shaped so the outputs still sum to the one-step output: a continued delay emits nothing and adds to what it holds, a continued integrate emits only its piece. An empty snapshot still takes its one tick; a finished job takes none. Deltas pushed to the same input between calls are absorbed into the next piece. Buffering live deltas during a rebuild (clause 7) is the host's.

   current() is the whole tick. A stream keeps the latest piece, which downstream operators read, and the whole value of the tick, which Stream::current reads: pieces within one tick are absorbed into it, so integrate.current() after a chunked load is the full integral and a continued delay still reads the previous tick's input. Output sums pieces as before.

   A pending rebuild owns the clock. Circuit::step counts ticks, and a Rebuild records the count after its first chunk. If a Circuit::step runs before the job is Done (the host stepped between calls, or two jobs interleaved), the next Rebuild::step panics with "the rebuild was interrupted"; Rebuild::interrupted asks without panicking, so a host can refuse first. step_continue calls made by another job are not counted and are not detected. The state a load leaves for the next tick is tested by pushing another delta after Done and stepping twice.

9. Not built: Circuit does not own a job runner (ADR 008's), and nothing here is serialised (clause 9, improvise-protocol).

The tests are crates/zset/tests/circuit.rs; the rivals and where they failed are in context/test-evidence/zset.md.

Transactions (improvise-bw4p.5)

A panic inside step (a weight overflow in an operator, say) leaves nodes ahead of the panic updated and the rest not. ADR 009 clause 11 says each command snapshots the circuit for rollback; Circuit::begin, commit and rollback are that, and the ADR is not amended (the clause's wording stands; this is how the crate reads it).

1. An undo log, not a snapshot. The ADR budgets memory, and a clone of the circuit's state is the size of the circuit. begin instead makes every stateful node start logging, and the log holds what the steps touch: the cost of a transaction is the keys a step touches, not the circuit's size (a_transaction_costs_the_keys_a_step_touches_and_not_the_size_of_the_circuit measures it in key clones: 16 extra for begin; step; commit, 22 for rollback, the same at 10 and at 300 keys of state, with an integrate in the pipeline). Outside a transaction nothing is logged and nothing extra is cloned. Two costs are the circuit's size, and are limits, not features:
   • A split tick. A step_continue that is the first publish of an edge in a transaction clones that edge's whole value (2009 key clones against 1006 outside one, at N=1000; a_split_tick_in_a_transaction_costs_the_size_of_the_edge).
   • Memory held until commit. Every edge's previous value is moved into the slot's undo record and kept, so an integrate -> delay pipeline holds 4022 live keys during a transaction against 2014 without one (N=1000). An integrate edge is back to three copies of its integral for the length of the transaction.
2. What is logged, per node. Each integrated Z-set (integrate, distinct, a sink's sum) keeps (key, weight) for every addition it applied; each trace (the join's two, the aggregate's groups) keeps (key, value, weight) for every apply, through Undoable, a wrapper over the caller's Trace; a Delay keeps one clone of what it holds on its first eval, which is the previous tick's input; a Source keeps a clone of what was pending at begin; an edge (Slot) keeps the (piece, whole) its first publish replaced, moved and not cloned, except for a continued piece, which clones the edge once. ticks is restored.
3. Rollback replays newest first. Each undo is the exact inverse of an addition that succeeded, so it never leaves i64, but only in this order: a key at MAX lowered and raised again would reach MAX + 1 undone oldest first (rollback_undoes_newest_first_so_no_undo_leaves_i64). i64::MIN has no negation; a trace undoes it as MAX then 1 (the weight before was non-negative, so neither leaves i64), and a Z-set undoes it by adjust(.., subtract).
4. An entry is built before its change and logged after it. An overflow panics before a key changes (insert is one checked addition), and the entry's clones are made first, so a panicking Clone changes nothing either (c1_a_clone_panic_anywhere_in_a_step_is_rolled_back sweeps a panic over every key clone in a step). A step that dies half way has therefore logged exactly the keys it applied. A custom Trace must do the same: a panicking apply leaves the trace as it was. MemTrace does, since it makes every clone before it touches its map (a_panicking_value_clone_leaves_a_memtrace_unchanged).
5. The edge's two copies. An integrate stream's piece is also its output, so a second copy (the whole-tick value) only exists while a tick is split. Slot::whole is None otherwise and current() reads the piece. piece() lends a borrow rather than a clone (improvise-bw4p.11). An integrate stream now holds its integral twice (the node's acc and the stream's piece), not three times; sharing those two needs shared ownership in the slot and was not done.
6. What a transaction does not undo.
   • A closure's captured state (a map function that counts, an FFI callback's side effects). Operators that keep state keep it in the crate's own types for this reason.
   • A Rebuild: it owns an iterator over its snapshot, so a rolled-back chunk leaves the job ahead of the circuit. The clock cannot tell: rollback rewinds ticks, which can make a dead job's recorded tick current again. So the circuit also counts the rollbacks that undid a step or piece (undone, never rewound), and a job records the count when it first runs, before its first chunk can panic. The chunk's push is marked as a step too, since it can overflow the pending delta before any step runs (p1_a_chunk_whose_push_panics_in_a_txn_is_counted). A later count makes interrupted true for good, finished or not (rollback_rebuild.rs: begun before the transaction and continued inside, finished inside, a first chunk that panicked, and a dead job revived by a later step).
   • An Output::take made inside the transaction: a rollback subtracts the step's output from the sink, which a take has emptied. Take after commit; zset-ffi refuses to take while a transaction is open.
7. Pushes. Circuit::push is not logged. rollback puts the pending delta back to what it was at begin, so a push made after begin is dropped, and one made before it is pending again. Operators built after begin start logging at once and go back to empty; they are not removed.
8. One transaction at a time, and misuse panics like an overflow does: begin while open, commit or rollback with none open. zset-ffi answers with a code instead.

The tests are tests/rollback.rs (the property: a step that panics half way, then a rollback, leaves every stream and sink as before and the later ticks equal to a circuit that never ran the step) and tests/cost.rs. A step that panics outside a transaction still leaves the circuit unusable: drop it and build another.

Rebuilds and circuits (improvise-bw4p.12)

A Rebuild takes its circuit's identity from its Input: every handle a circuit makes holds the circuit's token (a process-wide counter since the arena, Rc::ptr_eq before), so Rebuild::belongs_to(&circuit) is a comparison. Rebuild::step on another circuit panics, where it used to report Done and land its chunk in that circuit's next tick. An interrupted job (decision 8 above) can never finish: interrupted stays true however many steps follow, because a split tick cannot be mended and a rollback that took a chunk is counted forever (decision 6 above). Drop it and start another job from the same snapshot; the host keeps the snapshot.

Threads (user ruling 2026-10-11; ADR 009 clause 1 and section 9)

The graph is network-agnostic and multi-threaded from the start, and multi-threading is scheduling: a circuit stays synchronous and single-threaded, and a host moves whole circuits between threads. Shared-nothing: one circuit per runnable unit, joined only by edges (the edge adapter, a later PR, is the one thing that crosses threads). Splitting one circuit's keys across threads (data parallelism) is not done and not precluded: its exchange step would be the same edge. This is a ruling recorded here, not an ADR amendment. Native threads are assumed; wasm threads are out of scope, and nothing in zset spawns one. Send and mpsc compile on wasm32.

1. Circuit is Send and not Sync, with no lock anywhere in it. It moves between threads, one at a time (tests/threads.rs: a circuit built on one thread, stepped on a new thread each tick, equals a circuit that never moved; a transaction begun on one thread is rolled back on another). Two compile_fail doctests on Circuit pin that it is not Sync and refuses an Rc closure.
2. The Rc<RefCell> graph is an arena. What made the circuit !Send was the sharing between the circuit and the handles it hands out. A circuit now owns every edge's slot, every input's pending delta and every sink's sum in one Vec<Box<dyn Any + Send>>; a node holds typed ids, and eval, begin, commit and rollback take the arena. A downcast on every access is the price (an Any type check; tests/cost.rs counts clones, which it leaves as they were, and time was not measured).
3. Handles are plain ids, used through the circuit. Stream, Input and Output are Copy, Send and Sync, and carry their circuit's token. The calls moved onto the circuit: Circuit::current(&stream), Circuit::push(&input, delta), Circuit::take(&output). A handle from another circuit panics ("handle from another circuit"), where before it was quietly wrong. A consequence: a closure can no longer read a stream during a step, since it has no circuit; the old b1_... case does not compile, so it is gone.
4. User closures are Send + 'static; keys and values are Send. A custom Trace handed to join_indexed_in or aggregate_in must be Send. Piecewise requires Send + 'static.
5. The sink does not clone. Its node reads the input edge and writes the sum through Arena::read_write, two disjoint entries, so an_eval_borrows_its_input_instead_of_cloning_it still holds.
6. Handing a circuit to a thread is the host's business (std::thread lives in host crates). Whoever owns the circuit pushes, steps and takes; a handle is a number and needs no thread.