improvise/context/code/improvise-incremental.md

# `improvise-incremental`: the reasoning

`crates/improvise-incremental` is ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the
plug. It holds the `Derived` trait, `Eager` (today's evaluator behind it) and the
differential harness every later backend is checked with. There is no incremental backend
here, the evaluator is untouched and no existing crate changed. The decisions are the ADR's
(`context/adr/009-incremental-evaluation.md`, clause 2). The evidence for each test is in
`context/test-evidence/improvise-incremental.md`.

## The shape

| item | what it is |
|---|---|
| `Derived` | the two things the evaluator reads: `value(key, agg)` and `raw_fold(partial)` |
| `Eager` | an owned `Model` plus its None categories; every answer is a call into `Model` |
| `Maintained` | a `Derived` that is told each `Edit`; `Eager` is one |
| `Edit` | one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation) |
| `probes`, `Probe` | every question worth asking of a model, and a way to put it to any backend |
| `differential` | runs one edit sequence on a candidate and on `Eager`, comparing every probe at every step |
| `Divergence`, `Report` | what a failure names (step, probe, both answers) and what a pass compared |

## Interpretations

Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw
fold of the cells that match a partial key") and nothing else. These are the readings taken.

1. **`Derived` lives in this crate, not in improvise-core.** The ADR says improvise-core
   "gains a trait". Putting it there changes an existing crate's source, which the brief for
   this step ruled out, and the brief was ambiguous about it. This departs from the accepted
   ADR's text, so it is flagged to the user and not decided here: improvise-bw4p.2 moves it
   (one file and a `pub use`, since the trait names only core types). While it sits
   downstream, core cannot read through it, which is the clause's point.
2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell
   as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is
   `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy are not
   in the trait, so the differential cannot probe them, and a hidden item (view state) cannot
   be exercised at all. The ADR's differential compares "every drawn cell, total and occupancy
   count"; improvise-bw4p.3 adds them. A group is exercised: a stored key with one coordinate
   replaced by its item's group is a probe.
3. **`raw_fold` is never a formula, and `Eager` enforces it.** For a partial naming a formula
   target `Eager::raw_fold` is `None`; otherwise it is `evaluate_aggregated(partial, OverHidden)`,
   because the evaluator's own raw fold (`aggregate_raw`, `aggregate_cell_values`) is private to
   core. Calling it through the public path is exact for any partial that names no formula
   target. improvise-bw4p.2 exposes the raw fold from core so `Eager` stops going round.
4. **`Eager` recomputes formulas after every edit.** Production does it in `rebuild_layout`
   (`Model::recompute_formulas(none_cats)`), and an over-hidden read of a formula cell reads
   the cache it fills. So `Eager` carries the None categories of the view it stands for and
   recomputes under them after each edit, as a layout rebuild does.
5. **`Edit` is the vocabulary of the harness, not of the wire.** Five variants, each one
   public `Model` call. A later step adds variants for what it maintains (an item added, a
   category removed); the harness does not care which. An `Edit` fault reaches both sides of a
   differential alike, so it is pinned only where a direct `Eager` test reads the result:
   `ClearCell`, `AddFormula` and `RemoveFormula`, and `SetMeasureAgg` through the fold tests.
   `SetCell` is read by nearly every test.
6. **Values agree by float rules with every NaN one value** (`same_value`), the rule clause 3
   states for cell deltas. Signed zero compares equal, so a later `ExactSum` cannot be told
   from eager's sum by the sign of a zero here; clause 8 has its own property test for that.
7. **No `zset` dependency yet.** Nothing in this step uses a Z-set. It arrives with the first
   backend, as the cell delta of step 3.

## What the harness checks

`differential(model, none_cats, candidate, edits)` builds `Eager` over a clone of the model,
applies each edit to both, and after step 0 and after every edit asks both every probe. The
probe set **accumulates**: probes from the model at every earlier step stay asked, so a backend
that goes on answering for a removed formula diverges. A pass returns a `Report` of how many
states and answers were compared.

`probes(model)` takes the base keys (each stored key, and each with one coordinate replaced by
its item's group), every sub-partial of a base key dropping at most `MAX_DROPPED = 3`
coordinates plus the empty key, and each formula's target over each of those. Each key is asked
as an exact value, an over-hidden value and a raw fold. A fold dropping more than three
coordinates from a stored key is not asked about. `edit_probes`
adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence
reads `divergence at step N: <probe>: eager <a> != candidate <b>`.

**A limit, stated plainly (improvise-bw4p.4).** The probe set is every stored key's
sub-partials, not the keys a view draws. A view with four or more categories on None draws a cell
that drops four or more coordinates of a stored key, and the harness checks none of those cells:
a rival wrong only at `{C0=a, _Measure=Revenue}` over a five-category model passes. Neither
shipped demo reaches the limit, so the corpus does not show it. Raising `MAX_DROPPED` is not the
answer, because `sub_partials` walks all 2^n subsets of a key whatever the limit is (93ms at 20
categories and 225ms at 22 in a debug build, per call). The fix is to probe the keys each view
draws, with steps 3 and 4; until then a candidate on such a view is under-checked.

The corpus the tests use is `examples/demo.improv` and `web/demo.improv`, loaded through
`improvise_io::persistence::load`, and hand-built models (formulas, text, mixed values, a group).
`eager_matches_the_evaluator_on_the_shipped_documents` takes every view's None categories; the
differential runs take the first per document, and the largest document only its first four
edits, to keep the binary under two seconds in a debug build. A proptest (256 cases) runs random
edit sequences against a candidate that rebuilds an `Eager` from scratch on each edit.