improvise/context/code/improvise-incremental.md

improvise-incremental: the reasoning

crates/improvise-incremental is ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the plug. It holds the Derived trait, Eager (today's evaluator behind it) and the differential harness every later backend is checked with. There is no incremental backend here, the evaluator is untouched and no existing crate changed. The decisions are the ADR's (context/adr/009-incremental-evaluation.md, clause 2). The evidence for each test is in context/test-evidence/improvise-incremental.md.

The shape

item	what it is
Derived	the two things the evaluator reads: value(key, agg) and raw_fold(partial)
Eager	an owned Model plus its None categories; every answer is a call into Model
Maintained	a Derived that is told each Edit; Eager is one
Edit	one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation)
probes, Probe	every question worth asking of a model, and a way to put it to any backend
differential	runs one edit sequence on a candidate and on Eager, comparing every probe at every step
Divergence, Report	what a failure names (step, probe, both answers) and what a pass compared

Interpretations

Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw fold of the cells that match a partial key") and nothing else. These are the readings taken.

1. Derived lives in this crate, not in improvise-core. The ADR says improvise-core "gains a trait". Putting it there changes an existing crate's source, which the brief for this step ruled out, and the brief was ambiguous about it. This departs from the accepted ADR's text, so it is flagged to the user and not decided here: improvise-bw4p.2 moves it (one file and a pub use, since the trait names only core types). While it sits downstream, core cannot read through it, which is the clause's point.
2. Two methods, the minimum. value(key, agg) is the formula result at a key, or the cell as stored or folded; raw_fold(partial) is the fold of matching stored cells. agg is Aggregation, the evaluator's own type. Hidden-item covers, totals and occupancy joined the trait in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate replaced by its item's group is a probe.
3. raw_fold is never a formula, and Eager enforces it. For a partial naming a formula target Eager::raw_fold is None; otherwise it is evaluate_aggregated(partial, OverHidden), because the evaluator's own raw fold (aggregate_raw, aggregate_cell_values) is private to core. Calling it through the public path is exact for any partial that names no formula target. improvise-bw4p.2 exposes the raw fold from core so Eager stops going round.
4. Eager recomputes formulas after every edit. Production does it in rebuild_layout (Model::recompute_formulas(none_cats)), and an over-hidden read of a formula cell reads the cache it fills. So Eager carries the None categories of the view it stands for and recomputes under them after each edit, as a layout rebuild does.
5. Edit is the vocabulary of the harness, not of the wire. Five variants, each one public Model call. A later step adds variants for what it maintains (an item added, a category removed); the harness does not care which. An Edit fault reaches both sides of a differential alike, so it is pinned only where a direct Eager test reads the result: ClearCell, AddFormula and RemoveFormula, and SetMeasureAgg through the fold tests. SetCell is read by nearly every test.
6. Values agree by float rules with every NaN one value (same_value), the rule clause 3 states for cell deltas. Signed zero compares equal, so a later ExactSum cannot be told from eager's sum by the sign of a zero here; clause 8 has its own property test for that.
7. The zset dependency arrived with step 3, as the cell delta (below).

Step 3: the cell delta (improvise-5r8c)

delta.rs holds DeltaValue, CellDelta = ZSet<(CellKey, DeltaValue)>, cell_delta and moved_records. improvise-command's apply_effects reads them for the record tier and the dirty flag (context/code/command.md). Evidence: context/test-evidence/improvise-incremental.md. These are the readings taken, where ADR 009 clause 3 leaves a choice.

1. DataStore logs, improvise-incremental builds the Z-set. The ADR says the store "emits a ZSet", but core never depends on zset (clause 2). DataStore::record_changes turns on a Vec<CellChange> log (None while off, so an unwatched store keeps nothing), drain_changes empties it and cell_delta folds it. The log lives in the store, not in Model::set_cell, so every route that writes a cell is seen, including RemoveCategory's sweep.
2. DeltaValue canonicalises on construction: every NaN becomes f64::NAN, -0.0 becomes +0.0, and the order is total_cmp on the result, with numbers before text before errors. The inner value is private, so a non-canonical one cannot be built. Equality is the order's Equal, so == and Ord agree.
3. CellKey is Ord by derivation: the sorted coordinate pairs, compared as strings. Clause 3 calls this "the natural order of ids", which ADR 007's id form will change; the derive then moves with the key type.
4. The record tier is the set of records a consolidated batch delta names. A cell written and written back inside one batch cancels, as the old XOR did.
5. The dirty flag needs two inputs the ADR does not name. Besides a non-empty delta and the formula set, document_config stays (an AddItem writes no cell and the existing test requires it dirty) and the saved order joins, which no delta carries: DataStore::reorders counts SortData, and any logged insertion or removal dirties too, because a clear-then-set of one value nets to zero yet moves the cell to the end of the store (a same-item rename).
6. document_generation and the wire are untouched, and Eager and the Derived trait are not used: no backend consumes the delta yet. The differential for this step is the delta against the store it describes (the_cell_delta_of_an_edit_is_...), since there is nothing to compare with Eager.

What the harness checks

differential(model, none_cats, candidate, edits) builds Eager over a clone of the model, applies each edit to both, and after step 0 and after every edit asks both every probe. The probe set accumulates: probes from the model at every earlier step stay asked, so a backend that goes on answering for a removed formula diverges. A pass returns a Report of how many states and answers were compared.

probes(model) takes the base keys (each stored key, and each with one coordinate replaced by its item's group), every sub-partial of a base key dropping at most MAX_DROPPED = 3 coordinates plus the empty key, and each formula's target over each of those. Each key is asked as an exact value, an over-hidden value and a raw fold. A fold dropping more than three coordinates from a stored key is not asked about. edit_probes adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence reads divergence at step N: <probe>: eager <a> != candidate <b>.

Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4)

The subset probes above cannot see a drawn cell on a view with four or more None categories (it drops more than MAX_DROPPED coordinates of a stored key), and the trait could not be asked about hidden items, totals or occupancy. Both are closed. These are the readings taken.

1. Reading is how a view reads. The aggregation (Aggregation::of its None categories) and the hidden (category, item) pairs of View::hidden_items. covers(cat, item) is the predicate GridLayout::evaluator passes as its covers. It is a value, not a closure, so a probe can name it and a divergence can print it. A fold over a group or a spanned category leaves hidden members out; a read of a key naming neither does not (a hidden East still counts in {Product=A} when Region is None), while an occupancy count always leaves them out. The count follows the drawn items, not the value; Eager mirrors production in both.
2. Three methods join the trait: covered(key, reading) is Model::evaluate_covering; total(key, over, reading) is evaluate_spanning, a column total as ONE fold over the visible items of the omitted categories; occupancy(partial, reading) is ADR 009 clause 7's count by axis item: the stored cells matching partial, none of whose coordinates is hidden, where a coordinate naming a group matches every stored member of that group (a collapsed group is a remap, so the Coastal line of two visible members counts 2). A key naming a formula target has no stored cells and is not asked. "Has a value" (what prune_empty asks) is covered(..).is_some(), already probed.
3. drawn_probes(model, views, none_cats) asks what each view draws, built from the view's own GridLayout (pruned as production builds it): for every drawn cell the three subset questions plus Covered and Occupancy; for every column its column_key occupancy and, where GridLayout::draws_totals (the condition grid_body also reads), a Total over the row categories; for every row the occupancy of its cell key less the column coordinates. Lines carry the page coordinates, as a cell's key does, so a page is its own slice. A records view is skipped: its keys are synthetic _Index/_Dim coordinates naming no stored cell and it has no totals, while its stored keys are the subset probes' business. A view whose None categories differ from the oracle's is skipped, since Eager recomputes its formulas under one set (a backend keeping a cache per aggregation class is therefore not exercised on the others). differential_in(model, views, ..) adds them at every step; differential is that with no views. A view is a View, not a document, because the model holds none.
4. sub_partials is polynomial: it enumerates the sets of at most MAX_DROPPED dropped indices, not the 2^n masks. The probe set is unchanged for every key length (a test compares it with the old walk up to 12 coordinates, and counts it at 64). MAX_DROPPED stays: it bounds the subset probes, and the drawn probes carry the cells that drop more.
5. The hidden fixture is a workbook the corpus now includes (hidden): no shipped demo hides an item, so a backend wrong about hidden items would have passed the direct Eager-against-evaluator comparison as well.
6. Probe is Hash + Eq, so Asked deduplicates on the probe itself (Aggregation gained Hash in core for it).