test(incremental): formula cache reads, wide sets, NaN and the evidence for step 5 (improvise-shbc)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
2026-10-10 15:08:53 -07:00
parent 7e3c6b0c30
commit 933c9254a0
7 changed files with 380 additions and 46 deletions
@@ -216,3 +216,67 @@ taken where clause 4 and clause 7 leave a choice; evidence is in
 - Eviction of shapes, and the build as a sliced job.
 - `Sum`, `Avg` (step 6) and formulas (step 5) are eager's, behind the same trait.
 - The ADR 008 fixture measurement.
+
+## Step 5: formulas (improvise-shbc)
+
+`Incremental` no longer calls `recompute_formulas`. It keeps the formula cache (`Model::formula_cell`,
+`set_formula_cell`, `evaluate_formula_cell`, the three small additions to `improvise-core`) and decides
+which cells of it to evaluate again; the evaluator that computes each value is core's own. The ADR
+(clauses 5 and 6) is followed in outline, and the places it is not are the reason for this section.
+
+1. **A stem is the unit, because eager's cache is per stem.** `Stems` (`formulas.rs`) holds, per
+   target category, how many stored cells give each stem (a cell minus that category and the
+   None categories, as `recompute_formulas` derives them), and which stems at least one category
+   gives (`live`). Counts come from the raw store changes, one net count per stem per category per
+   batch, so an overwrite never makes a stem vanish and reappear. A stem that appears is
+   evaluated; one that goes has its cells removed.
+2. **A stem's cells are evaluated from nothing.** `recompute_formulas` clears the cache and runs
+   fixed-point passes over every stem together. When no formula reads another stem's cache the stems
+   are independent, so running the same passes on one stem from an empty cache gives exactly what
+   the whole run gives for it, including `circular` for a cell still moving after sixteen
+   passes. That is why a dirty stem is cleared first (`passes-start-warm` shows what a warm start
+   changes). Where they are not independent (`Stems::is_wide`: two target categories, or a name in
+   another category that is not None) the backend falls back to `recompute_formulas` for the whole
+   cache, so those documents are exact and not incremental.
+3. **Which stems a change reaches is a superset, read off the formulas.** For formula `f`,
+   `Reads` is the set of names in its expression and in the formulas those names target, and the
+   categories that puts in play (its target's, every name's, every `WHERE`'s and `IS`'s). A stored
+   cell can change what `f` evaluates to at stem `s` only if it agrees with `s` everywhere outside
+   those categories. `Readers` indexes each live stem by what is left of it outside them, per
+   category set, so a changed cell finds its stems by looking up its own projection on each set:
+   no scan of the stems. This is clause 5's input rule made coarser, once for every shape of
+   formula: it re-evaluates a stem for a cell of an item the formula never names (a change to a
+   `Note` cell evaluates the stems `Profit` has). It cannot miss one, which is what the aggregate
+   over an expression needs, since the cells it reads are chosen at run time by the records.
+4. **The aggregate-over-expression pipeline (clause 5, items 1 to 3) is not built.** A dirty stem
+   evaluates its formulas whole, so `SUM(Revenue * Price)` is re-folded over its records when any
+   cell under the stem changes, not adjusted for the one record. That needs the evaluator to
+   accept a maintained aggregate in place of folding, which is a change to core's evaluator and
+   to the order its records are visited in (insertion order, not `CellKey` order). It is a bead.
+5. **Dependency order is an optimisation and is checked against the plain passes.** `Order` is the
+   formulas' dependency graph by name. When it has no cycle and there are fewer formulas than
+   passes, one evaluation per cell in dependency order reaches the fixed point eager's passes
+   reach (a chain of fifteen links needs fifteen passes and a sixteenth to see no change). It
+   differs from the passes in one place: a cell holding NaN is never equal to itself, so eager's passes
+   never see it settle and call it `circular`, and the cells that read it keep what they computed
+   from the NaN. `mark_nan_circular` does the same afterwards. `Evaluation::Plain` always runs
+   the passes; `tests/ordering.rs` compares the two and `tests/formulas.rs` compares both with
+   eager.
+6. **A formula edit rebuilds the stems.** Adding, removing or replacing a formula (the fingerprint
+   of the formula list moves) rebuilds the stems from the store by one pass, clears the cache and
+   evaluates every stem, because a formula has a cell at each. Clause 5 wants a formula delta;
+   the index is not maintained across a formula edit. `Meter::rebuilds` counts it.
+7. **A new aggregation dirties every stem if a formula reads that measure.** `measure_agg` is not
+   a cell, so no change reaches the readers; `reads_name` asks the formulas.
+8. **A write of the same bits changes nothing; a write of `-0.0` over `0.0` does.** The cell delta
+   treats the two as one value (clause 3), but a formula may not (`Units ^ -1` is `inf` and
+   `-inf`), so the formula side reads the raw change log, not the canonical delta.
+
+### Not built in step 5
+
+- The aggregate-over-expression pipeline and the per-record order key (item 4).
+- Stems that survive a formula edit (item 6), and name-level selectivity within a stem (item 3).
+- Incremental cache for documents with formulas in more than one target category, or a name in
+  a non-None category other than the formula's own (item 2): they use the whole recompute.
+- Strongly connected components as a maintained structure: `Order` is rebuilt with the stems, which
+  is O(formulas) on a formula edit, and cycles are found by depth-first search, not Tarjan's.