# `improvise-incremental`: the reasoning

`crates/improvise-incremental` began as ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the
plug. It holds the `Derived` trait, `Eager` (today's evaluator behind it) and the
differential harness every backend is checked with, and, from step 4 (improvise-u2le), the
first backend that keeps state, `Incremental` (section "Step 4" below). The step 2 sections
describe the trait and the oracle; they predate the backend. The decisions are the ADR's
(`context/adr/009-incremental-evaluation.md`). The evidence for each test is in
`context/test-evidence/improvise-incremental.md`.

## The shape

| item | what it is |
|---|---|
| `Derived` | the two things the evaluator reads: `value(key, agg)` and `raw_fold(partial)` |
| `Eager` | an owned `Model` plus its None categories; every answer is a call into `Model` |
| `Maintained` | a `Derived` that is told each `Edit`; `Eager` is one |
| `Edit` | one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation) |
| `probes`, `Probe` | every question worth asking of a model, and a way to put it to any backend |
| `differential` | runs one edit sequence on a candidate and on `Eager`, comparing every probe at every step |
| `Divergence`, `Report` | what a failure names (step, probe, both answers) and what a pass compared |
| `Incremental` | the backend of step 4: maintained groups and occupancy; formulas and `Sum`/`Avg` still go to the model (later steps change that) |

## Interpretations

Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw
fold of the cells that match a partial key") and nothing else. These are the readings taken.

1. **`Derived` lives in this crate, not in improvise-core.** The ADR says improvise-core
   "gains a trait". Putting it there changes an existing crate's source, which the brief for
   this step ruled out, and the brief was ambiguous about it. This departs from the accepted
   ADR's text, so it is flagged to the user and not decided here: improvise-bw4p.2 moves it
   (one file and a `pub use`, since the trait names only core types). While it sits
   downstream, core cannot read through it, which is the clause's point.
2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell
   as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is
   `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy joined the trait
   in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate
   replaced by its item's group is a probe.
3. **`raw_fold` is never a formula, and `Eager` enforces it.** For a partial naming a formula
   target `Eager::raw_fold` is `None`; otherwise it is `evaluate_aggregated(partial, OverHidden)`,
   because the evaluator's own raw fold (`aggregate_raw`, `aggregate_cell_values`) is private to
   core. Calling it through the public path is exact for any partial that names no formula
   target. improvise-bw4p.2 exposes the raw fold from core so `Eager` stops going round.
4. **`Eager` recomputes formulas after every edit.** Production does it in `rebuild_layout`
   (`Model::recompute_formulas(none_cats)`), and an over-hidden read of a formula cell reads
   the cache it fills. So `Eager` carries the None categories of the view it stands for and
   recomputes under them after each edit, as a layout rebuild does.
5. **`Edit` is the vocabulary of the harness, not of the wire.** Five variants, each one
   public `Model` call. A later step adds variants for what it maintains (an item added, a
   category removed); the harness does not care which. An `Edit` fault reaches both sides of a
   differential alike, so it is pinned only where a direct `Eager` test reads the result:
   `ClearCell`, `AddFormula` and `RemoveFormula`, and `SetMeasureAgg` through the fold tests.
   `SetCell` is read by nearly every test.
6. **Values agree by float rules with every NaN one value** (`same_value`), the rule clause 3
   states for cell deltas, except the two zeros: they differ, since a drawn `-0.00` is not `0.00`
   and eager's `Min` and `Max` return the sign they were given. Clause 8 has its own tests
   for the sign of a zero sum.
7. **The `zset` dependency arrived with step 3**, as the cell delta (below).

## Step 3: the cell delta (improvise-5r8c)

`delta.rs` holds `DeltaValue`, `CellDelta = ZSet<(CellKey, DeltaValue)>`, `cell_delta` and
`moved_records`. `improvise-command`'s `apply_effects` reads them for the record tier and the
dirty flag (`context/code/command.md`). Evidence: `context/test-evidence/improvise-incremental.md`.
These are the readings taken, where ADR 009 clause 3 leaves a choice.

1. **`DataStore` logs, `improvise-incremental` builds the Z-set.** The ADR says the store
   "emits a `ZSet`", but core never depends on `zset` (clause 2). `DataStore::record_changes`
   turns on a `Vec<CellChange>` log (`None` while off, so an unwatched store keeps nothing),
   `drain_changes` empties it and `cell_delta` folds it. The log lives in the store, not in
   `Model::set_cell`, so every route that writes a cell is seen, including `RemoveCategory`'s
   sweep.
2. **`DeltaValue` canonicalises on construction**: every NaN becomes `f64::NAN`, `-0.0` becomes
   `+0.0`, and the order is `total_cmp` on the result, with numbers before text before errors.
   The inner value is private, so a non-canonical one cannot be built. Equality is the order's
   `Equal`, so `==` and `Ord` agree.
3. **`CellKey` is `Ord` by derivation**: the sorted coordinate pairs, compared as strings.
   Clause 3 calls this "the natural order of ids", which ADR 007's id form will change; the
   derive then moves with the key type.
4. **The record tier is the set of records a consolidated batch delta names.** A cell written
   and written back inside one batch cancels, as the old XOR did.
5. **The dirty flag needs two inputs the ADR does not name.** Besides a non-empty delta and
   the formula set, `document_config` stays (an `AddItem` writes no cell and the existing test
   requires it dirty) and the saved order joins, which no delta carries: `DataStore::reorders`
   counts `SortData`, and any logged insertion or removal dirties too, because a clear-then-set
   of one value nets to zero yet moves the cell to the end of the store (a same-item rename).
6. **`document_generation` and the wire are untouched**, and `Eager` and the `Derived` trait
   are not used: no backend consumes the delta yet. The differential for this step is the
   delta against the store it describes (`the_cell_delta_of_an_edit_is_...`), since there is
   nothing to compare with `Eager`.

## What the harness checks

`differential(model, none_cats, candidate, edits)` builds `Eager` over a clone of the model,
applies each edit to both, and after step 0 and after every edit asks both every probe. The
probe set **accumulates**: probes from the model at every earlier step stay asked, so a backend
that goes on answering for a removed formula diverges. A pass returns a `Report` of how many
states and answers were compared.

`probes(model)` takes the base keys (each stored key, and each with one coordinate replaced by
its item's group), every sub-partial of a base key dropping at most `MAX_DROPPED = 3`
coordinates plus the empty key, and each formula's target over each of those. Each key is asked
as an exact value, an over-hidden value and a raw fold. A fold dropping more than three
coordinates from a stored key is not asked about. `edit_probes`
adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence
reads `divergence at step N: <probe>: eager <a> != candidate <b>`.

## Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4)

The subset probes above cannot see a drawn cell on a view with four or more None categories
(it drops more than `MAX_DROPPED` coordinates of a stored key), and the trait could not be asked
about hidden items, totals or occupancy. Both are closed. These are the readings taken.

1. **`Reading` is how a view reads.** The aggregation (`Aggregation::of` its None categories)
   and the hidden `(category, item)` pairs of `View::hidden_items`. `covers(cat, item)` is the
   predicate `GridLayout::evaluator` passes as its `covers`. It is a value, not a closure, so a
   probe can name it and a divergence can print it. A fold over a group or a spanned category
   leaves hidden members out; a read of a key naming neither does not (a hidden East still counts
   in `{Product=A}` when Region is None), while an occupancy count always leaves them out. The
   count follows the drawn items, not the value; `Eager` mirrors production in both.
2. **Three methods join the trait**: `covered(key, reading)` is `Model::evaluate_covering`;
   `total(key, over, reading)` is `evaluate_spanning`, a column total as ONE fold over the
   visible items of the omitted categories; `occupancy(partial, reading)` is ADR 009 clause 7's
   count by axis item: the stored cells matching `partial`, none of whose coordinates is hidden,
   where a coordinate naming a group matches every stored member of that group (a collapsed group
   is a remap, so the Coastal line of two visible members counts 2). A key naming a formula
   target has no stored cells and is not asked. "Has a value" (what `prune_empty` asks) is
   `covered(..).is_some()`, already probed.
3. **`drawn_probes(model, views, none_cats)` asks what each view draws**, built from the view's
   own `GridLayout` (pruned as production builds it): for every drawn cell the three subset
   questions plus `Covered` and `Occupancy`; for every column its `column_key` occupancy and, where
   `GridLayout::draws_totals` (the condition `grid_body` also reads), a `Total` over the row
   categories; for every row the occupancy of its cell key less the column coordinates. Lines
   carry the page coordinates, as a cell's key does, so a page is its own slice. A records view is
   skipped: its keys are synthetic `_Index`/`_Dim` coordinates naming no stored cell and it has
   no totals, while its stored keys are the subset probes' business. A view whose None categories
   differ from the oracle's is skipped, since `Eager` recomputes its formulas under one set (a
   backend keeping a cache per aggregation class is therefore not exercised on the others).
   `differential_in(model, views, ..)` adds them at every step; `differential` is that with no
   views. A view is a `View`, not a document, because the model holds none.
4. **`sub_partials` is polynomial**: it enumerates the sets of at most `MAX_DROPPED` dropped
   indices, not the 2^n masks. The probe set is unchanged for every key length (a test compares
   it with the old walk up to 12 coordinates, and counts it at 64). `MAX_DROPPED` stays: it
   bounds the subset probes, and the drawn probes carry the cells that drop more.
5. **The hidden fixture is a workbook the corpus now includes** (`hidden`): no shipped demo
   hides an item, so a backend wrong about hidden items would have passed the direct
   `Eager`-against-evaluator comparison as well.
6. **`Probe` is `Hash + Eq`**, so `Asked` deduplicates on the probe itself (`Aggregation` gained
   `Hash` in core for it).

## Step 4: the incremental backend (improvise-u2le)

`Incremental` (`incremental.rs`) is the first `Derived` that keeps state. It owns a `Model`
(the store) and `Shapes` (`shapes.rs`, the groups). `Maintained::apply` applies the edit to the
model, drains the store's change log into a `CellDelta` and feeds the delta to every shape. A
read asks a group, never the store, for what step 4 owns. The rest of this section is the readings
taken where clause 4 and clause 7 leave a choice; evidence is in
`context/test-evidence/improvise-incremental.md`.

1. **A shape is a set of category names, not a None list.** Clause 4 says "a group is the cell key
   minus the None categories". The trait is asked about keys of any category set (the harness
   drops up to three coordinates of a stored key), so the unit is the set a key names: a `Shape`
   maps the projection of a stored cell onto those categories to a `Group`, and a cell lands in
   it only if it has all of them. A view's None list is one of these sets. Clause 7's "one
   circuit serves every shape key with the same None list" holds trivially, since the sets are
   shared by every reader.
2. **A shape is built on its first read, by one pass over the store, and fed every delta after.**
   Clause 7 makes a new key's first build a sliced job under ADR 008's job runner. That runner
   does not exist, so the pass is synchronous and counted (`Meter::builds`, `built_cells`).
   Nothing evicts a shape. `Shapes` holds its map in a `RefCell` because `Derived` reads take
   `&self`; the cell is private to `shapes.rs` and no reference to it leaves a method.
3. **A group is a count, a count of numbers and a multiset** (`group.rs`). One update touches one
   map entry (O(log d)); the count and the multiset serve every function, so
   `SetMeasureAgg` rebuilds nothing. A read of several groups (a collapsed group, a total) folds
   their union without merging them: `fold_groups` sums the counts and takes the smallest of the
   groups' minima, the largest of their maxima, or the one value they share.
4. **Eager's rules are reproduced, not improved.** An all-numbers group reduces; any other group
   is unanimous or `Error("mixed")`, so `Count` of a group holding text is that text. `Min` and
   `Max` skip NaN and are `+inf`/`-inf` over only NaN, as `fold(INFINITY, f64::min)` is. The
   multiset orders by the delta's total order, with NaN last, so the skip is a step past one
   key. A group holding a NaN and a non-number is mixed, because NaN is not equal to itself in
   `CellValue`'s `==` and is a distinct key here.
5. **`Sum` and `Avg` stay with the store until step 6.** Their eager result depends on the
   order the cells are added in (insertion order in the store, not `CellKey` order, whatever
   clause 3 says), so a group cannot reproduce it without keeping that order. The backend routes
   a read whose measure aggregation is `Sum` or `Avg` to `Model::evaluate_covering` and
   `evaluate_spanning`, which is eager's own path. Step 6 replaces both sides with `ExactSum`.
   Because `Sum` is the default, the corpus reaches the groups only once a `SetMeasureAgg`
   edit has changed a measure; the differential tests apply those on purpose.
6. **Formulas stay with the model until step 5.** A key a formula owns is read through the model,
   and `apply` calls `Model::recompute_formulas`, as `Eager` does. Step 5 replaces the call.
7. **A fold is the union of its members' groups.** `fold.rs` is `Model::group_expansion` read off
   the public model: the cross product of the covered members of every group the key names and
   every covered item of every category in `over` that it omits. Under `OverHidden` the members
   are groups of one shape (the base's categories plus the spans'). Under `Exact` the cells
   admitted are exactly the member keys, so the read is a lookup per member and a throwaway
   `Group`. A total therefore reads one group per covered row, not one per stored cell, but still
   one per row: clause 7's maintained totals with a subtracted hidden overlay are not built (see
   "Not built").
8. **Occupancy counts by inclusion and exclusion.** `occupancy(partial)` is the cells under
   `partial`, where a coordinate naming a group matches each of its members' items, with no
   hidden coordinate. For each way of choosing a member per group coordinate the count is the
   group's cell count, less the cells holding a hidden item of a category the key leaves free,
   by inclusion and exclusion over those categories: `N(P) - sum N(P + h) + ...`. The cost is
   the product of the hidden items per free category, a group read each. A key naming a hidden
   item itself is zero.
9. **The meter counts operations.** `Meter` is `updates` (group updates a delta caused), `reads`
   (groups a read looked at), `builds` and `built_cells`. `tests/measure.rs` fixes one table at
   two sizes and asserts the counts: an overwrite costs two updates per shape whatever the table
   holds, and a read of a group of 36 or 72 cells costs one group. Eager's cost for the same read
   is `matching_values().len()`, which the test prints beside it. This is the local fixture the
   brief asked for; ADR 009 step 4 says "measured at ADR 008's fixture", which needs ADR 008
   (unapproved) and is deferred.

10. **`Min` and `Max` keep the sign of a zero.** The multiset calls `-0.0` and `+0.0` one
    value (clause 3), so a group also counts its negative zeros from the raw change log, as step 6's
    sums are fed. Eager's `f64::min` and `f64::max` leave the sign of a tie between the two zeros to
    the platform and the order (`fmin` on ARM returns `-0.0`; x86 may not). The backend reproduces the
    part that does not depend on either: all zeros negative gives `-0.0`, all positive gives `+0.0`.
    With both, `Min` is `-0.0` and `Max` is `+0.0`, which is what ARM's `fmin`/`fmax` return; on other
    platforms eager may differ there, and `tests/random.rs` keeps `+0.0` out of its pool for that reason.

### Not built in step 4

- Maintained totals (clause 7): a total reads one group per covered row.
- Eviction of shapes, and the build as a sliced job.
- `Sum`, `Avg` (step 6) and formulas (step 5) are eager's, behind the same trait.
- The ADR 008 fixture measurement.

## Step 5: formulas (improvise-shbc)

`Incremental` calls `recompute_formulas` only for the documents item 2 calls wide. Otherwise it keeps
the formula cache itself (`Model::formula_cell`, `set_formula_cell`, `clear_formula_cells` and the
now-public `evaluate_formula_cell`, the small additions to `improvise-core`) and decides which cells of
it to evaluate again; the evaluator that computes each value is core's own. The ADR (clauses 5 and 6)
is followed in outline, and the places it is not are the reason for this section.

1. **A stem is the unit, because eager's cache is per stem.** `Stems` (`formulas.rs`) holds, per
   target category, how many stored cells give each stem (a cell minus that category and the
   None categories, as `recompute_formulas` derives them), and which stems at least one category
   gives (`live`). Counts come from the raw store changes, one net count per stem per category per
   batch, so an overwrite never makes a stem vanish and reappear. A stem that appears is
   evaluated; one that goes has its cells removed.
2. **A stem's cells are evaluated from nothing.** `recompute_formulas` clears the cache and runs
   fixed-point passes over every stem together. When no formula reads another stem's cache the stems
   are independent, so running the same passes on one stem from an empty cache gives exactly what
   the whole run gives for it, including `circular` for a cell still moving after sixteen
   passes. That is why a dirty stem is cleared first (`passes-start-warm` shows what a warm start
   changes). Where they are not independent (`Stems::is_wide`: two target categories, or a name in
   another category that is not None) the backend falls back to `recompute_formulas` for the whole
   cache, so those documents are exact and not incremental.
3. **Which stems a change reaches is a superset, read off the formulas.** For formula `f`,
   `Reads` is the set of names in its expression and in the formulas those names target, and the
   categories that puts in play (its target's, every name's, every `WHERE`'s and `IS`'s). A stored
   cell can change what `f` evaluates to at stem `s` only if it agrees with `s` everywhere outside
   those categories. `Readers` indexes each live stem by what is left of it outside them, per
   category set, so a changed cell finds its stems by looking up its own projection on each set:
   no scan of the stems. This is clause 5's input rule made coarser, once for every shape of
   formula: it re-evaluates a stem for a cell of an item the formula never names (a change to a
   `Note` cell evaluates the stems `Profit` has). It cannot miss one, which is what the aggregate
   over an expression needs, since the cells it reads are chosen at run time by the records.
4. **The aggregate-over-expression pipeline (clause 5, items 1 to 3) is not built.** A dirty stem
   evaluates its formulas whole, so `SUM(Revenue * Price)` is re-folded over its records when any
   cell under the stem changes, not adjusted for the one record. That needs the evaluator to
   accept a maintained aggregate in place of folding, which is a change to core's evaluator and
   to the order its records are visited in (insertion order, not `CellKey` order). It is a bead.
5. **Dependency order is an optimisation and is checked against the plain passes.** `Order` is the
   formulas' dependency graph by name. When it has no cycle and there are fewer formulas than
   passes, one evaluation per cell in dependency order reaches the fixed point eager's passes
   reach (a chain of fifteen links needs fifteen passes and a sixteenth to see no change). It
   differs from the passes in two places. A cell holding NaN is never equal to itself, so eager's
   passes never see it settle and call it `circular`, and the cells that read it keep what they
   computed from the NaN; `mark_nan_circular` does the same afterwards. And eager stops on `==`,
   which calls `-0.0` equal to the `0.0` an early pass read from a stored cell, so a stem with a
   zero cell can settle on a sign dependency order never sees (a stored `0.0` at `F1`, `F3 = F1`,
   `F4 = F3 ^ -1`, then `F1 = -0`: eager's `F4` is `inf`). A stem whose ordered evaluation holds any
   zero is therefore evaluated again by the passes (`holds_a_zero`). `Evaluation::Plain` always
   runs the passes; `tests/ordering.rs` compares the two after every edit, zero signs included, and
   `tests/formulas.rs` compares both with eager.
6. **A formula edit rebuilds the stems.** Adding, removing or replacing a formula (the fingerprint
   of the formula list moves) rebuilds the stems from the store by one pass, clears the cache and
   evaluates every stem, because a formula has a cell at each. Clause 5 wants a formula delta;
   the index is not maintained across a formula edit. `Meter::rebuilds` counts it.
7. **A new aggregation dirties the stems it can reach.** `measure_agg` is not a cell, so no change
   reaches the readers. A fold under a None category takes its aggregation from the `_Measure`
   coordinate of the key, which for a formula targeting another category comes from the stem and
   from no name the formula reads, so every live stem carrying that measure is dirtied
   (`Stems::carrying`). Where a formula names the measure (it targets `_Measure`) `reads_name` dirties every stem.
8. **A write of the same bits changes nothing; a write of `-0.0` over `0.0` does.** The cell delta
   treats the two as one value (clause 3), but a formula may not (`Units ^ -1` is `inf` and
   `-inf`), so the formula side reads the raw change log, not the canonical delta.

### Not built in step 5

- The aggregate-over-expression pipeline and the per-record order key (item 4).
- Stems that survive a formula edit (item 6), and name-level selectivity within a stem (item 3).
- Incremental cache for documents with formulas in more than one target category, or a name in
  a non-None category other than the formula's own (item 2): they use the whole recompute.
- Strongly connected components as a maintained structure: `Order` is rebuilt with the stems, which
  is O(formulas) on a formula edit, and cycles are found by depth-first search, not Tarjan's.

## Step 6: Sum, Avg and totals through an exact sum (improvise-hrjz)

ADR 009 clause 8, in both backends. `Sum` and `Avg` stop being the one reading that went to the
store: a group now holds a `numtower::ExactSum` beside its counts and multiset, and `Reduce`
(was `OrderFree`) has all five functions. Step 4's delegation, and its sentence in section 5
of that step, are gone; formula-owned keys still go to the model, whose cache step 5 maintains.

1. **`ExactSum` already signs a zero total** (`numtower/src/sum.rs`, PR #348, written for step 8's
   last test): `-0.0` only if every finite addend is `-0.0`, `+0.0` for any other zero, an exact
   cancellation included. An earlier version of this step wrote that rule as a wrapper, `FloatSum`;
   it was dropped when `gh/main` brought the same rule into `ExactSum`, and the tests of the rule are
   that PR's `tests/zero_sign.rs`. One difference from the wrapper: an empty sum reads `+0.0`, where
   `Sum for f64` starts at `-0.0`. Core's `SUM` over no rows therefore changes from `-0.0` to `+0.0`.
2. **Core sums through it.** `apply_agg` (`Sum`, `Avg`) and the formula-level `SUM` in `eval_agg` call
   `exact_sum`, so `Eager` is exact, and so is every reading of the model, with no flag: the user
   visible change clause 8 names (displayed sums change in their last bits) is this commit. `improvise-core`
   gains the `numtower` dependency; both of the new edges are in `Cargo.nix` and the harness lock.
3. **A group's sum is fed from the raw changes, not the cell delta.** The delta calls `-0.0` and
   `+0.0` one value (clause 3), so an overwrite of `0.0` by `-0.0` is no delta at all, yet it moves
   the sign of a zero sum. `Shape::apply` takes the delta for counts and the multiset and the change
   log for sums, skipping only a change that left the same bits. The ADR's two clauses 3 and 8 are
   consistent only because the sum reads the store's values and not the delta's. The cost is a second
   walk of the batch per shape.
4. **Totals and collapsed groups absorb.** A read of several groups (`total`, a collapsed group,
   a hidden member left out) adds their accumulators with `absorb` and reads once, so a total is the
   same bits whatever the grouping and whatever the order of the rows. The clause 7 hidden overlay
   (subtracting the hidden cells from a kept total) is still not built; a total is a union of member
   groups, not a kept total less an overlay.
5. **`Avg` is the exact sum over the cell count.** Both backends round once in the sum and once in
   the division.
6. **What a group costs.** An `ExactSum` is 70 limbs of `i64` and a few counters, about 650 bytes
   (`size_of`), in every group of every shape, whether or not the measure is summed. Clause 8
   accepts 568 bytes "per summed group"; here the shape does not know which measure its groups
   serve, so a shape at cell granularity (a read of a full key) pays it for each cell. Sparing
   groups that are never summed, or holding the sum only above one cell, is the improvement; it
   is a bead.
7. **The drift test is 20,000 overwrites, not a million.** ADR 009 measured 1.7e-9 at 18,000 cells
   after a million cycles. `tests/exact.rs` runs 20,000 overwrites of a 1,000-cell group with
   values chosen so an `f64` sum drifts, and asserts the two backends equal each other and a sum
   built afresh to the bit. The count is what a test binary can afford; the property it pins is
   independent of it.

### Not built in step 6

- The hidden overlay and kept totals (clause 7).
- A group that does not pay for a sum it never reads (item 6).