improvise/context/code/improvise-incremental.md

improvise-incremental: the reasoning

crates/improvise-incremental began as ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the plug. It holds the Derived trait, Eager (today's evaluator behind it) and the differential harness every backend is checked with, and, from step 4 (improvise-u2le), the first backend that keeps state, Incremental (section "Step 4" below). The step 2 sections describe the trait and the oracle; they predate the backend. The decisions are the ADR's (context/adr/009-incremental-evaluation.md). The evidence for each test is in context/test-evidence/improvise-incremental.md.

The shape

item	what it is
Derived	the two things the evaluator reads: value(key, agg) and raw_fold(partial)
Eager	an owned Model plus its None categories; every answer is a call into Model
Maintained	a Derived that is told each Edit; Eager is one
Edit	one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation)
probes, Probe	every question worth asking of a model, and a way to put it to any backend
differential	runs one edit sequence on a candidate and on Eager, comparing every probe at every step
Divergence, Report	what a failure names (step, probe, both answers) and what a pass compared
Incremental	the backend of step 4: maintained groups and occupancy; formulas and Sum/Avg still go to the model (later steps change that)

Interpretations

Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw fold of the cells that match a partial key") and nothing else. These are the readings taken.

1. Derived lives in this crate, not in improvise-core. The ADR says improvise-core "gains a trait". Putting it there changes an existing crate's source, which the brief for this step ruled out, and the brief was ambiguous about it. This departs from the accepted ADR's text, so it is flagged to the user and not decided here: improvise-bw4p.2 moves it (one file and a pub use, since the trait names only core types). While it sits downstream, core cannot read through it, which is the clause's point.
2. Two methods, the minimum. value(key, agg) is the formula result at a key, or the cell as stored or folded; raw_fold(partial) is the fold of matching stored cells. agg is Aggregation, the evaluator's own type. Hidden-item covers, totals and occupancy joined the trait in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate replaced by its item's group is a probe.
3. raw_fold is never a formula, and Eager enforces it. For a partial naming a formula target Eager::raw_fold is None; otherwise it is evaluate_aggregated(partial, OverHidden), because the evaluator's own raw fold (aggregate_raw, aggregate_cell_values) is private to core. Calling it through the public path is exact for any partial that names no formula target. improvise-bw4p.2 exposes the raw fold from core so Eager stops going round.
4. Eager recomputes formulas after every edit. Production does it in rebuild_layout (Model::recompute_formulas(none_cats)), and an over-hidden read of a formula cell reads the cache it fills. So Eager carries the None categories of the view it stands for and recomputes under them after each edit, as a layout rebuild does.
5. Edit is the vocabulary of the harness, not of the wire. Five variants, each one public Model call. A later step adds variants for what it maintains (an item added, a category removed); the harness does not care which. An Edit fault reaches both sides of a differential alike, so it is pinned only where a direct Eager test reads the result: ClearCell, AddFormula and RemoveFormula, and SetMeasureAgg through the fold tests. SetCell is read by nearly every test.
6. Values agree by float rules with every NaN one value (same_value), the rule clause 3 states for cell deltas, except the two zeros: they differ, since a drawn -0.00 is not 0.00 and eager's Min and Max return the sign they were given. Clause 8 has its own tests for the sign of a zero sum.
7. The zset dependency arrived with step 3, as the cell delta (below).

Step 3: the cell delta (improvise-5r8c)

delta.rs holds DeltaValue, CellDelta = ZSet<(CellKey, DeltaValue)>, cell_delta and moved_records. improvise-command's apply_effects reads them for the record tier and the dirty flag (context/code/command.md). Evidence: context/test-evidence/improvise-incremental.md. These are the readings taken, where ADR 009 clause 3 leaves a choice.

1. DataStore logs, improvise-incremental builds the Z-set. The ADR says the store "emits a ZSet", but core never depends on zset (clause 2). DataStore::record_changes turns on a Vec<CellChange> log (None while off, so an unwatched store keeps nothing), drain_changes empties it and cell_delta folds it. The log lives in the store, not in Model::set_cell, so every route that writes a cell is seen, including RemoveCategory's sweep.
2. DeltaValue canonicalises on construction: every NaN becomes f64::NAN, -0.0 becomes +0.0, and the order is total_cmp on the result, with numbers before text before errors. The inner value is private, so a non-canonical one cannot be built. Equality is the order's Equal, so == and Ord agree.
3. CellKey is Ord by derivation: the sorted coordinate pairs, compared as strings. Clause 3 calls this "the natural order of ids", which ADR 007's id form will change; the derive then moves with the key type.
4. The record tier is the set of records a consolidated batch delta names. A cell written and written back inside one batch cancels, as the old XOR did.
5. The dirty flag needs two inputs the ADR does not name. Besides a non-empty delta and the formula set, document_config stays (an AddItem writes no cell and the existing test requires it dirty) and the saved order joins, which no delta carries: DataStore::reorders counts SortData, and any logged insertion or removal dirties too, because a clear-then-set of one value nets to zero yet moves the cell to the end of the store (a same-item rename).
6. document_generation and the wire are untouched, and Eager and the Derived trait are not used: no backend consumes the delta yet. The differential for this step is the delta against the store it describes (the_cell_delta_of_an_edit_is_...), since there is nothing to compare with Eager.

What the harness checks

differential(model, none_cats, candidate, edits) builds Eager over a clone of the model, applies each edit to both, and after step 0 and after every edit asks both every probe. The probe set accumulates: probes from the model at every earlier step stay asked, so a backend that goes on answering for a removed formula diverges. A pass returns a Report of how many states and answers were compared.

probes(model) takes the base keys (each stored key, and each with one coordinate replaced by its item's group), every sub-partial of a base key dropping at most MAX_DROPPED = 3 coordinates plus the empty key, and each formula's target over each of those. Each key is asked as an exact value, an over-hidden value and a raw fold. A fold dropping more than three coordinates from a stored key is not asked about. edit_probes adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence reads divergence at step N: <probe>: eager <a> != candidate <b>.

Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4)

The subset probes above cannot see a drawn cell on a view with four or more None categories (it drops more than MAX_DROPPED coordinates of a stored key), and the trait could not be asked about hidden items, totals or occupancy. Both are closed. These are the readings taken.

1. Reading is how a view reads. The aggregation (Aggregation::of its None categories) and the hidden (category, item) pairs of View::hidden_items. covers(cat, item) is the predicate GridLayout::evaluator passes as its covers. It is a value, not a closure, so a probe can name it and a divergence can print it. A fold over a group or a spanned category leaves hidden members out; a read of a key naming neither does not (a hidden East still counts in {Product=A} when Region is None), while an occupancy count always leaves them out. The count follows the drawn items, not the value; Eager mirrors production in both.
2. Three methods join the trait: covered(key, reading) is Model::evaluate_covering; total(key, over, reading) is evaluate_spanning, a column total as ONE fold over the visible items of the omitted categories; occupancy(partial, reading) is ADR 009 clause 7's count by axis item: the stored cells matching partial, none of whose coordinates is hidden, where a coordinate naming a group matches every stored member of that group (a collapsed group is a remap, so the Coastal line of two visible members counts 2). A key naming a formula target has no stored cells and is not asked. "Has a value" (what prune_empty asks) is covered(..).is_some(), already probed.
3. drawn_probes(model, views, none_cats) asks what each view draws, built from the view's own GridLayout (pruned as production builds it): for every drawn cell the three subset questions plus Covered and Occupancy; for every column its column_key occupancy and, where GridLayout::draws_totals (the condition grid_body also reads), a Total over the row categories; for every row the occupancy of its cell key less the column coordinates. Lines carry the page coordinates, as a cell's key does, so a page is its own slice. A records view is skipped: its keys are synthetic _Index/_Dim coordinates naming no stored cell and it has no totals, while its stored keys are the subset probes' business. A view whose None categories differ from the oracle's is skipped, since Eager recomputes its formulas under one set (a backend keeping a cache per aggregation class is therefore not exercised on the others). differential_in(model, views, ..) adds them at every step; differential is that with no views. A view is a View, not a document, because the model holds none.
4. sub_partials is polynomial: it enumerates the sets of at most MAX_DROPPED dropped indices, not the 2^n masks. The probe set is unchanged for every key length (a test compares it with the old walk up to 12 coordinates, and counts it at 64). MAX_DROPPED stays: it bounds the subset probes, and the drawn probes carry the cells that drop more.
5. The hidden fixture is a workbook the corpus now includes (hidden): no shipped demo hides an item, so a backend wrong about hidden items would have passed the direct Eager-against-evaluator comparison as well.
6. Probe is Hash + Eq, so Asked deduplicates on the probe itself (Aggregation gained Hash in core for it).

Step 4: the incremental backend (improvise-u2le)

Incremental (incremental.rs) is the first Derived that keeps state. It owns a Model (the store) and Shapes (shapes.rs, the groups). Maintained::apply applies the edit to the model, drains the store's change log into a CellDelta and feeds the delta to every shape. A read asks a group, never the store, for what step 4 owns. The rest of this section is the readings taken where clause 4 and clause 7 leave a choice; evidence is in context/test-evidence/improvise-incremental.md.

1. A shape is a set of category names, not a None list. Clause 4 says "a group is the cell key minus the None categories". The trait is asked about keys of any category set (the harness drops up to three coordinates of a stored key), so the unit is the set a key names: a Shape maps the projection of a stored cell onto those categories to a Group, and a cell lands in it only if it has all of them. A view's None list is one of these sets. Clause 7's "one circuit serves every shape key with the same None list" holds trivially, since the sets are shared by every reader.

2. A shape is built on its first read, by one pass over the store, and fed every delta after. Clause 7 makes a new key's first build a sliced job under ADR 008's job runner. That runner does not exist, so the pass is synchronous and counted (Meter::builds, built_cells). Nothing evicts a shape. Shapes holds its map in a RefCell because Derived reads take &self; the cell is private to shapes.rs and no reference to it leaves a method.

3. A group is a count, a count of numbers and a multiset (group.rs). One update touches one map entry (O(log d)); the count and the multiset serve every function, so SetMeasureAgg rebuilds nothing. A read of several groups (a collapsed group, a total) folds their union without merging them: fold_groups sums the counts and takes the smallest of the groups' minima, the largest of their maxima, or the one value they share.

4. Eager's rules are reproduced, not improved. An all-numbers group reduces; any other group is unanimous or Error("mixed"), so Count of a group holding text is that text. Min and Max skip NaN and are +inf/-inf over only NaN, as fold(INFINITY, f64::min) is. The multiset orders by the delta's total order, with NaN last, so the skip is a step past one key. A group holding a NaN and a non-number is mixed, because NaN is not equal to itself in CellValue's == and is a distinct key here.

5. Sum and Avg stay with the store until step 6. Their eager result depends on the order the cells are added in (insertion order in the store, not CellKey order, whatever clause 3 says), so a group cannot reproduce it without keeping that order. The backend routes a read whose measure aggregation is Sum or Avg to Model::evaluate_covering and evaluate_spanning, which is eager's own path. Step 6 replaces both sides with ExactSum. Because Sum is the default, the corpus reaches the groups only once a SetMeasureAgg edit has changed a measure; the differential tests apply those on purpose.

6. Formulas stay with the model until step 5. A key a formula owns is read through the model, and apply calls Model::recompute_formulas, as Eager does. Step 5 replaces the call.

7. A fold is the union of its members' groups. fold.rs is Model::group_expansion read off the public model: the cross product of the covered members of every group the key names and every covered item of every category in over that it omits. Under OverHidden the members are groups of one shape (the base's categories plus the spans'). Under Exact the cells admitted are exactly the member keys, so the read is a lookup per member and a throwaway Group. A total therefore reads one group per covered row, not one per stored cell, but still one per row: clause 7's maintained totals with a subtracted hidden overlay are not built (see "Not built").

8. Occupancy counts by inclusion and exclusion. occupancy(partial) is the cells under partial, where a coordinate naming a group matches each of its members' items, with no hidden coordinate. For each way of choosing a member per group coordinate the count is the group's cell count, less the cells holding a hidden item of a category the key leaves free, by inclusion and exclusion over those categories: N(P) - sum N(P + h) + .... The cost is the product of the hidden items per free category, a group read each. A key naming a hidden item itself is zero.

9. The meter counts operations. Meter is updates (group updates a delta caused), reads (groups a read looked at), builds and built_cells. tests/measure.rs fixes one table at two sizes and asserts the counts: an overwrite costs two updates per shape whatever the table holds, and a read of a group of 36 or 72 cells costs one group. Eager's cost for the same read is matching_values().len(), which the test prints beside it. This is the local fixture the brief asked for; ADR 009 step 4 says "measured at ADR 008's fixture", which needs ADR 008 (unapproved) and is deferred.

10. Min and Max keep the sign of a zero. The multiset calls -0.0 and +0.0 one value (clause 3), so a group also counts its negative zeros from the raw change log, as step 6's sums are fed. Eager's f64::min and f64::max leave the sign of a tie between the two zeros to the platform and the order (fmin on ARM returns -0.0; x86 may not). The backend reproduces the part that does not depend on either: all zeros negative gives -0.0, all positive gives +0.0. With both, Min is -0.0 and Max is +0.0, which is what ARM's fmin/fmax return; on other platforms eager may differ there, and tests/random.rs keeps +0.0 out of its pool for that reason.

Not built in step 4

• Maintained totals (clause 7): a total reads one group per covered row.
• Eviction of shapes, and the build as a sliced job.
• Sum, Avg (step 6) and formulas (step 5) are eager's, behind the same trait.
• The ADR 008 fixture measurement.

Step 5: formulas (improvise-shbc)

Incremental calls recompute_formulas only for the documents item 2 calls wide. Otherwise it keeps the formula cache itself (Model::formula_cell, set_formula_cell, clear_formula_cells and the now-public evaluate_formula_cell, the small additions to improvise-core) and decides which cells of it to evaluate again; the evaluator that computes each value is core's own. The ADR (clauses 5 and 6) is followed in outline, and the places it is not are the reason for this section.

1. A stem is the unit, because eager's cache is per stem. Stems (formulas.rs) holds, per target category, how many stored cells give each stem (a cell minus that category and the None categories, as recompute_formulas derives them), and which stems at least one category gives (live). Counts come from the raw store changes, one net count per stem per category per batch, so an overwrite never makes a stem vanish and reappear. A stem that appears is evaluated; one that goes has its cells removed.
2. A stem's cells are evaluated from nothing. recompute_formulas clears the cache and runs fixed-point passes over every stem together. When no formula reads another stem's cache the stems are independent, so running the same passes on one stem from an empty cache gives exactly what the whole run gives for it, including circular for a cell still moving after sixteen passes. That is why a dirty stem is cleared first (passes-start-warm shows what a warm start changes). Where they are not independent (Stems::is_wide: two target categories, or a name in another category that is not None) the backend falls back to recompute_formulas for the whole cache, so those documents are exact and not incremental.
3. Which stems a change reaches is a superset, read off the formulas. For formula f, Reads is the set of names in its expression and in the formulas those names target, and the categories that puts in play (its target's, every name's, every WHERE's and IS's). A stored cell can change what f evaluates to at stem s only if it agrees with s everywhere outside those categories. Readers indexes each live stem by what is left of it outside them, per category set, so a changed cell finds its stems by looking up its own projection on each set: no scan of the stems. This is clause 5's input rule made coarser, once for every shape of formula: it re-evaluates a stem for a cell of an item the formula never names (a change to a Note cell evaluates the stems Profit has). It cannot miss one, which is what the aggregate over an expression needs, since the cells it reads are chosen at run time by the records.
4. The aggregate-over-expression pipeline (clause 5, items 1 to 3) is not built. A dirty stem evaluates its formulas whole, so SUM(Revenue * Price) is re-folded over its records when any cell under the stem changes, not adjusted for the one record. That needs the evaluator to accept a maintained aggregate in place of folding, which is a change to core's evaluator and to the order its records are visited in (insertion order, not CellKey order). It is a bead.
5. Dependency order is an optimisation and is checked against the plain passes. Order is the formulas' dependency graph by name. When it has no cycle and there are fewer formulas than passes, one evaluation per cell in dependency order reaches the fixed point eager's passes reach (a chain of fifteen links needs fifteen passes and a sixteenth to see no change). It differs from the passes in two places. A cell holding NaN is never equal to itself, so eager's passes never see it settle and call it circular, and the cells that read it keep what they computed from the NaN; mark_nan_circular does the same afterwards. And eager stops on ==, which calls -0.0 equal to the 0.0 an early pass read from a stored cell, so a stem with a zero cell can settle on a sign dependency order never sees (a stored 0.0 at F1, F3 = F1, F4 = F3 ^ -1, then F1 = -0: eager's F4 is inf). A stem whose ordered evaluation holds any zero is therefore evaluated again by the passes (holds_a_zero). Evaluation::Plain always runs the passes; tests/ordering.rs compares the two after every edit, zero signs included, and tests/formulas.rs compares both with eager.
6. A formula edit rebuilds the stems. Adding, removing or replacing a formula (the fingerprint of the formula list moves) rebuilds the stems from the store by one pass, clears the cache and evaluates every stem, because a formula has a cell at each. Clause 5 wants a formula delta; the index is not maintained across a formula edit. Meter::rebuilds counts it.
7. A new aggregation dirties the stems it can reach. measure_agg is not a cell, so no change reaches the readers. A fold under a None category takes its aggregation from the _Measure coordinate of the key, which for a formula targeting another category comes from the stem and from no name the formula reads, so every live stem carrying that measure is dirtied (Stems::carrying). Where a formula names the measure (it targets _Measure) reads_name dirties every stem.
8. A write of the same bits changes nothing; a write of -0.0 over 0.0 does. The cell delta treats the two as one value (clause 3), but a formula may not (Units ^ -1 is inf and -inf), so the formula side reads the raw change log, not the canonical delta.

Not built in step 5

• The aggregate-over-expression pipeline and the per-record order key (item 4).
• Stems that survive a formula edit (item 6), and name-level selectivity within a stem (item 3).
• Incremental cache for documents with formulas in more than one target category, or a name in a non-None category other than the formula's own (item 2): they use the whole recompute.
• Strongly connected components as a maintained structure: Order is rebuilt with the stems, which is O(formulas) on a formula edit, and cycles are found by depth-first search, not Tarjan's.

Step 6: Sum, Avg and totals through an exact sum (improvise-hrjz)

ADR 009 clause 8, in both backends. Sum and Avg stop being the one reading that went to the store: a group now holds a numtower::ExactSum beside its counts and multiset, and Reduce (was OrderFree) has all five functions. Step 4's delegation, and its sentence in section 5 of that step, are gone; formula-owned keys still go to the model, whose cache step 5 maintains.

1. ExactSum already signs a zero total (numtower/src/sum.rs, PR #348, written for step 8's last test): -0.0 only if every finite addend is -0.0, +0.0 for any other zero, an exact cancellation included. An earlier version of this step wrote that rule as a wrapper, FloatSum; it was dropped when gh/main brought the same rule into ExactSum, and the tests of the rule are that PR's tests/zero_sign.rs. One difference from the wrapper: an empty sum reads +0.0, where Sum for f64 starts at -0.0. Core's SUM over no rows therefore changes from -0.0 to +0.0.
2. Core sums through it. apply_agg (Sum, Avg) and the formula-level SUM in eval_agg call exact_sum, so Eager is exact, and so is every reading of the model, with no flag: the user visible change clause 8 names (displayed sums change in their last bits) is this commit. improvise-core gains the numtower dependency; both of the new edges are in Cargo.nix and the harness lock.
3. A group's sum is fed from the raw changes, not the cell delta. The delta calls -0.0 and +0.0 one value (clause 3), so an overwrite of 0.0 by -0.0 is no delta at all, yet it moves the sign of a zero sum. Shape::apply takes the delta for counts and the multiset and the change log for sums, skipping only a change that left the same bits. The ADR's two clauses 3 and 8 are consistent only because the sum reads the store's values and not the delta's. The cost is a second walk of the batch per shape.
4. Totals and collapsed groups absorb. A read of several groups (total, a collapsed group, a hidden member left out) adds their accumulators with absorb and reads once, so a total is the same bits whatever the grouping and whatever the order of the rows. The clause 7 hidden overlay (subtracting the hidden cells from a kept total) is still not built; a total is a union of member groups, not a kept total less an overlay.
5. Avg is the exact sum over the cell count. Both backends round once in the sum and once in the division.
6. What a group costs. An ExactSum is 70 limbs of i64 and a few counters, about 650 bytes (size_of), in every group of every shape, whether or not the measure is summed. Clause 8 accepts 568 bytes "per summed group"; here the shape does not know which measure its groups serve, so a shape at cell granularity (a read of a full key) pays it for each cell. Sparing groups that are never summed, or holding the sum only above one cell, is the improvement; it is a bead.
7. The drift test is 20,000 overwrites, not a million. ADR 009 measured 1.7e-9 at 18,000 cells after a million cycles. tests/exact.rs runs 20,000 overwrites of a 1,000-cell group with values chosen so an f64 sum drifts, and asserts the two backends equal each other and a sum built afresh to the bit. The count is what a test binary can afford; the property it pins is independent of it.

Not built in step 6

• The hidden overlay and kept totals (clause 7).
• A group that does not pay for a sum it never reads (item 6).