feat(incremental): the Derived trait, Eager and the differential harness (improvise-t8ow)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

Cargo.lock

@@ -952,6 +952,14 @@ dependencies = [
  "improvise-formula",
 ]
 
+[[package]]
+name = "improvise-incremental"
+version = "0.1.0"
+dependencies = [
+ "improvise-core",
+ "improvise-io",
+]
+
 [[package]]
 name = "improvise-io"
 version = "0.1.0-rc2"

Cargo.toml

@@ -1,5 +1,5 @@
 [workspace]
-members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp", "crates/numtower", "crates/zset"]
+members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp", "crates/numtower", "crates/zset", "crates/improvise-incremental"]
 
 [package]
 name = "improvise"

context/code/improvise-incremental.md

@@ -0,0 +1,74 @@
+# `improvise-incremental`: the reasoning
+
+`crates/improvise-incremental` is ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the
+plug. It holds the `Derived` trait, `Eager` (today's evaluator behind it) and the
+differential harness every later backend is checked with. There is no incremental backend
+here, the evaluator is untouched and no existing crate changed. The decisions are the ADR's
+(`context/adr/009-incremental-evaluation.md`, clause 2). The evidence for each test is in
+`context/test-evidence/improvise-incremental.md`.
+
+## The shape
+
+| item | what it is |
+|---|---|
+| `Derived` | the two things the evaluator reads: `value(key, agg)` and `raw_fold(partial)` |
+| `Eager` | an owned `Model` plus its None categories; every answer is a call into `Model` |
+| `Maintained` | a `Derived` that is told each `Edit`; `Eager` is one |
+| `Edit` | one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation) |
+| `probes`, `Probe` | every question worth asking of a model, and a way to put it to any backend |
+| `differential` | runs one edit sequence on a candidate and on `Eager`, comparing every probe at every step |
+| `Divergence`, `Report` | what a failure names (step, probe, both answers) and what a pass compared |
+
+## Interpretations
+
+Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw
+fold of the cells that match a partial key") and nothing else. These are the readings taken.
+
+1. **`Derived` lives in this crate, not in improvise-core.** The ADR says improvise-core
+   "gains a trait". Putting it there changes an existing crate's source, which this step does
+   not do. Nothing is lost: the seam crate depends on core and core never on `zset`, which is
+   the clause's actual constraint. Moving the trait into core, when the evaluator is made to
+   read through it, is a move of one file.
+2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell
+   as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is
+   `Aggregation`, the evaluator's own type, because exact and over-hidden reads differ and
+   the harness must be able to ask both. Collapsed-group `covers`, totals and occupancy are
+   not in the trait: nothing reads them through it yet, and a method with no caller is
+   designed blind. The ADR's differential test compares "every drawn cell, total and occupancy
+   count"; those enter with the steps that maintain them.
+3. **`Eager::raw_fold` is `evaluate_aggregated(partial, OverHidden)`.** The evaluator's raw
+   fold (`aggregate_raw`, `aggregate_cell_values`) is private to core, and wrapping without
+   editing core leaves the public path. The two agree for a partial that names no formula
+   target and no collapsed group, which is what the harness probes. For a partial that does
+   name a formula target the cache answers instead, so `raw_fold` is not "never a formula"
+   there. Step 3 should expose the raw fold from core when it moves the evaluator behind the
+   trait, and `Eager` should call that.
+4. **`Eager` recomputes formulas after every edit.** Production does it in `rebuild_layout`
+   (`Model::recompute_formulas(none_cats)`), and an over-hidden read of a formula cell reads
+   the cache it fills. So `Eager` carries the None categories of the view it stands for and
+   recomputes under them after each edit, as a layout rebuild does.
+5. **`Edit` is the vocabulary of the harness, not of the wire.** Five variants, each one
+   public `Model` call. A later step adds variants for what it maintains (an item added, a
+   category removed); the harness does not care which.
+6. **Values agree by float rules with every NaN one value** (`same_value`), the rule clause 3
+   states for cell deltas. A NaN answer is not a divergence from itself. Signed zero compares
+   equal, so a later `ExactSum` cannot be told from eager's sum by the sign of a zero here;
+   clause 8 has its own property test for that.
+7. **No `zset` dependency yet.** The brief and the ADR name it, but nothing in this step uses
+   a Z-set, and an unused dependency is noise. It arrives with the first backend, as the
+   cell delta of step 3.
+
+## What the harness checks
+
+`differential(model, none_cats, candidate, edits)` builds `Eager` over a clone of the model,
+applies each edit to both, and after step 0 and after every edit asks both every probe. A
+probe set is derived from the post-edit model: each stored key, each partial of one and each
+formula cell over them under both aggregations, plus the raw fold of every partial. The key an
+edit names is probed too, so a cleared cell is asked about after it is gone. The first
+disagreement is returned as a `Divergence` whose message reads
+`divergence at step N: <probe>: eager <a> != candidate <b>`. A pass returns a `Report` of how
+many states and answers were compared, so a run that compared nothing shows as zero.
+
+The corpus the tests use is `examples/demo.improv` and `web/demo.improv`, loaded through
+`improvise_io::persistence::load`, every view's None categories in turn, plus a hand-built
+model with formulas, text and mixed values.

context/test-evidence/improvise-incremental.md

@@ -0,0 +1,102 @@
+# Test evidence: `improvise-incremental`
+
+What each test guards, which rival it was run against, and where that rival failed. The
+reasoning is in `context/code/improvise-incremental.md`; the decisions are ADR 009 clause 2's.
+
+**How the rivals are kept.** The `the_harness_catches_*` tests run `differential` on a
+`Rival`, an `Eager` told every edit and wrong in one way, and assert the step at which the
+harness reports the divergence. They run on every build, so each rival is watched failing on
+every run and not once by hand.
+
+**Injected in the source.** Six faults were also put into `src/eager.rs` and `src/harness.rs`
+one at a time, each run against the whole suite and each reverted and followed by a full
+rerun. The failing tests, by name:
+- `Eager::value` ignores its `Aggregation` (always `Exact`):
+  `eager_value_reads_a_formula_exactly_and_folded`, `eager_follows_an_edit_to_the_cell_a_fold_reads`,
+  `eager_matches_the_evaluator_on_the_shipped_documents`,
+  `the_harness_catches_a_backend_that_ignores_the_aggregation`;
+- `Eager::raw_fold` reads `Exact`: `eager_raw_fold_sums_by_default`,
+  `eager_raw_fold_follows_the_measures_aggregation`, `eager_raw_fold_of_text_is_unanimous_or_mixed`,
+  `eager_numbers_among_text_are_mixed`, `eager_follows_an_edit_to_the_cell_a_fold_reads`,
+  `eager_matches_the_evaluator_on_the_shipped_documents`;
+- `Eager::apply` does not recompute formulas: `eager_follows_an_edit_to_the_cell_a_fold_reads`;
+- the harness compares only the first edit: `the_harness_catches_a_backend_that_ignores_a_clear`,
+  `the_harness_catches_a_backend_that_ignores_the_measure_aggregation`,
+  `the_harness_catches_a_backend_that_goes_stale_after_two_edits`,
+  `the_harness_names_the_probe_the_two_backends_disagree_on`;
+- the harness never reports a divergence: the same four plus
+  `the_harness_catches_a_backend_that_ignores_the_aggregation`;
+- `same_value` says everything agrees: `the_harness_catches_a_backend_that_ignores_the_aggregation`.
+
+## eager_raw_fold_sums_by_default
+
+Hand-computed: Revenue under Product A is 10 + 5 = 15. Fails on the `raw_fold`-reads-`Exact`
+rival (a partial key names no stored cell, so the answer is `None`).
+
+## eager_raw_fold_follows_the_measures_aggregation
+
+After `measure_agg[Revenue] = Max` the same fold is 10, and Cost, unchanged, still sums to 6.
+A fold that always sums fails on the first; one that always takes the maximum fails on the second.
+
+## eager_raw_fold_of_text_is_unanimous_or_mixed
+
+Note under A is two equal texts and reads as that text; under B it is two different texts and
+reads `Error("mixed")`. A fold that drops text, which the evaluator used to do (gotcha 7),
+returns `None` for both.
+
+## eager_numbers_among_text_are_mixed
+
+A text among numbers is `Error("mixed")`, not the sum of the numbers.
+
+## eager_value_reads_a_formula_exactly_and_folded
+
+`Profit = Revenue - Cost` reads 6 at East/A exactly, and 9 at Product A with Region over
+hidden: (10 + 5) - (4 + 2), the fold taken before the subtraction. Fails on the
+`value`-ignores-`Aggregation` rival.
+
+## eager_follows_an_edit_to_the_cell_a_fold_reads
+
+After East/A Revenue becomes 20 the fold is 25 and the folded Profit 19. The Profit half is
+what fails when `apply` forgets to recompute formulas: the folded cache is then stale.
+
+## eager_matches_the_evaluator_on_the_shipped_documents
+
+Both shipped documents, every view's None categories, and the hand model: every probe of
+`Eager` equals a direct `evaluate_aggregated` on a model recomputed under the same None
+categories, and the probe set is non-empty. Fails on both aggregation rivals.
+
+## eager_agrees_with_itself_over_every_corpus_edit_sequence
+
+`differential` over a generated edit sequence on each corpus model returns a `Report` with
+`edits + 1` steps and a non-zero probe count. The count assertion is what a harness that
+compared nothing would fail; the step count is what one that stopped early would fail.
+
+## the_harness_passes_a_backend_that_is_right
+
+The control for the four `catches` tests: `Eager` as the candidate over the hand-built edits
+is a pass, so a harness that rejects everything is seen.
+
+## the_harness_catches_a_backend_that_ignores_the_aggregation
+
+A candidate that reads every `value` as `Exact` diverges at step 0. A harness that only
+compared after edits would not see it.
+
+## the_harness_catches_a_backend_that_ignores_the_measure_aggregation
+
+A candidate that drops `SetMeasureAgg` diverges at step 2, the edit that sets it. Fails on the
+compare-first-edit-only rival.
+
+## the_harness_catches_a_backend_that_goes_stale_after_two_edits
+
+A candidate that stops applying edits after the second diverges at step 3. Fails on the
+compare-first-edit-only rival: a prefix of steps cannot see a backend that goes wrong late.
+
+## the_harness_catches_a_backend_that_ignores_a_clear
+
+A candidate that keeps a cleared cell diverges at step 3, and the message names the eager
+answer. Fails on the compare-first-edit-only and never-fails rivals.
+
+## the_harness_names_the_probe_the_two_backends_disagree_on
+
+The `Divergence` carries a probe and two different answers, not only a message. Fails on the
+never-fails rival (no divergence is returned at all).

crates/improvise-incremental/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "improvise-incremental"
+version = "0.1.0"
+edition = "2024"
+license = "Apache-2.0"
+description = "The seam between improvise-core and the incremental circuit: the Derived trait, the Eager oracle and the differential harness (ADR 009 clause 2)."
+
+# ADR 009 clause 2: depends on improvise-core, never the other way round. The
+# `zset` edge arrives with the first backend that uses it (step 3), not before.
+[dependencies]
+improvise-core = { path = "../improvise-core" }
+
+[dev-dependencies]
+improvise-io = { path = "../improvise-io" }

crates/improvise-incremental/src/eager.rs

@@ -0,0 +1,42 @@
+//! Today's evaluator behind the trait. Temporary (ADR 009 clause 2).
+
+use crate::{Derived, Edit, Maintained};
+use improvise_core::model::cell::{CellKey, CellValue};
+use improvise_core::model::{Aggregation, Model};
+
+/// A model plus the None categories its formulas were last recomputed under.
+/// It adds no logic: every answer is a call into `Model`.
+#[derive(Debug, Clone)]
+pub struct Eager {
+    model: Model,
+    none_cats: Vec<String>,
+}
+
+impl Eager {
+    pub fn new(mut model: Model, none_cats: Vec<String>) -> Self {
+        model.recompute_formulas(&none_cats);
+        Eager { model, none_cats }
+    }
+
+    pub fn model(&self) -> &Model {
+        &self.model
+    }
+}
+
+impl Derived for Eager {
+    fn value(&self, key: &CellKey, agg: Aggregation) -> Option<CellValue> {
+        self.model.evaluate_aggregated(key, agg)
+    }
+
+    fn raw_fold(&self, partial: &CellKey) -> Option<CellValue> {
+        self.model
+            .evaluate_aggregated(partial, Aggregation::OverHidden)
+    }
+}
+
+impl Maintained for Eager {
+    fn apply(&mut self, edit: &Edit) {
+        edit.apply(&mut self.model);
+        self.model.recompute_formulas(&self.none_cats);
+    }
+}

crates/improvise-incremental/src/edit.rs

@@ -0,0 +1,46 @@
+//! One mutation of a model, as data, so a sequence can be replayed on two backends.
+
+use improvise_core::formula::ast::{AggFunc, Formula};
+use improvise_core::model::Model;
+use improvise_core::model::cell::{CellKey, CellValue};
+
+#[derive(Debug, Clone)]
+pub enum Edit {
+    SetCell(CellKey, CellValue),
+    ClearCell(CellKey),
+    AddFormula(Formula),
+    RemoveFormula {
+        target: String,
+        target_category: String,
+    },
+    SetMeasureAgg {
+        measure: String,
+        func: AggFunc,
+    },
+}
+
+impl Edit {
+    /// The one place an edit becomes a call on the model.
+    pub fn apply(&self, model: &mut Model) {
+        match self {
+            Edit::SetCell(key, value) => model.set_cell(key.clone(), value.clone()),
+            Edit::ClearCell(key) => model.clear_cell(key),
+            Edit::AddFormula(formula) => model.add_formula(formula.clone()),
+            Edit::RemoveFormula {
+                target,
+                target_category,
+            } => model.remove_formula(target, target_category),
+            Edit::SetMeasureAgg { measure, func } => {
+                model.measure_agg.insert(measure.clone(), func.clone());
+            }
+        }
+    }
+
+    /// The key an edit names, if it names one: always probed after it.
+    pub fn key(&self) -> Option<&CellKey> {
+        match self {
+            Edit::SetCell(key, _) | Edit::ClearCell(key) => Some(key),
+            _ => None,
+        }
+    }
+}

crates/improvise-incremental/src/harness.rs

@@ -0,0 +1,156 @@
+//! The differential harness: one edit sequence, two backends, every probe after every step.
+
+use crate::{Derived, Eager, Edit};
+use improvise_core::model::cell::{CellKey, CellValue};
+use improvise_core::model::{Aggregation, Model};
+use std::collections::HashSet;
+use std::fmt;
+
+/// A backend that holds state of its own and is told each edit.
+pub trait Maintained: Derived {
+    fn apply(&mut self, edit: &Edit);
+}
+
+/// One question put to both backends.
+#[derive(Debug, Clone, PartialEq)]
+pub enum Probe {
+    Value(CellKey, Aggregation),
+    RawFold(CellKey),
+}
+
+impl Probe {
+    pub fn ask(&self, backend: &dyn Derived) -> Option<CellValue> {
+        match self {
+            Probe::Value(key, agg) => backend.value(key, *agg),
+            Probe::RawFold(partial) => backend.raw_fold(partial),
+        }
+    }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct Divergence {
+    /// 0 is the model as built, `n` the state after edit `n`.
+    pub step: usize,
+    pub probe: Probe,
+    pub eager: Option<CellValue>,
+    pub candidate: Option<CellValue>,
+}
+
+impl fmt::Display for Divergence {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(
+            f,
+            "divergence at step {}: {:?}: eager {:?} != candidate {:?}",
+            self.step, self.probe, self.eager, self.candidate
+        )
+    }
+}
+
+/// How many states and how many answers were compared: a run that compared
+/// nothing must be visible as such.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct Report {
+    pub steps: usize,
+    pub probes: usize,
+}
+
+/// Values agree when equal by float rules with every NaN one value, so a NaN
+/// answer is not a divergence from itself.
+pub fn same_value(a: &Option<CellValue>, b: &Option<CellValue>) -> bool {
+    match (a, b) {
+        (Some(CellValue::Number(x)), Some(CellValue::Number(y))) => {
+            x == y || (x.is_nan() && y.is_nan())
+        }
+        _ => a == b,
+    }
+}
+
+/// Every question worth asking of `model`: each stored key, each partial of
+/// one and each formula cell over them, under both aggregations; and the raw
+/// fold of every partial.
+pub fn probes(model: &Model) -> Vec<Probe> {
+    let stored: Vec<CellKey> = model.data.iter_cells().map(|(k, _)| k).collect();
+    let partials: Vec<CellKey> = stored
+        .iter()
+        .flat_map(|k| k.0.iter().map(|(c, _)| k.without(c)))
+        .chain(std::iter::once(CellKey::new(vec![])))
+        .collect();
+    let formula_cells = stored.iter().flat_map(|k| {
+        model
+            .formulas()
+            .iter()
+            .map(move |f| k.clone().with(&f.target_category, &f.target))
+    });
+    let keys = stored
+        .iter()
+        .cloned()
+        .chain(partials.iter().cloned())
+        .chain(formula_cells);
+    let values = keys.flat_map(|k| {
+        [Aggregation::Exact, Aggregation::OverHidden]
+            .into_iter()
+            .map(move |a| Probe::Value(k.clone(), a))
+    });
+    let folds = partials.iter().cloned().map(Probe::RawFold);
+    let mut seen: HashSet<(CellKey, u8)> = HashSet::new();
+    values
+        .chain(folds)
+        .filter(|p| {
+            let tag = match p {
+                Probe::Value(k, Aggregation::Exact) => (k.clone(), 0),
+                Probe::Value(k, Aggregation::OverHidden) => (k.clone(), 1),
+                Probe::RawFold(k) => (k.clone(), 2),
+            };
+            seen.insert(tag)
+        })
+        .collect()
+}
+
+fn compare(
+    step: usize,
+    eager: &Eager,
+    candidate: &dyn Derived,
+    edit: Option<&Edit>,
+) -> Result<usize, Divergence> {
+    let mut asked = probes(eager.model());
+    let named = edit.and_then(Edit::key).into_iter().flat_map(|k| {
+        [Aggregation::Exact, Aggregation::OverHidden]
+            .into_iter()
+            .map(|a| Probe::Value(k.clone(), a))
+    });
+    asked.extend(named);
+    for probe in &asked {
+        let (want, got) = (probe.ask(eager), probe.ask(candidate));
+        if !same_value(&want, &got) {
+            return Err(Divergence {
+                step,
+                probe: probe.clone(),
+                eager: want,
+                candidate: got,
+            });
+        }
+    }
+    Ok(asked.len())
+}
+
+/// Applies `edits` to `candidate` and to an `Eager` over `model`, and
+/// compares every probe at step 0 and after each edit. `candidate` must have
+/// been built from `model`, with the same `none_cats`.
+pub fn differential<C: Maintained>(
+    model: &Model,
+    none_cats: &[String],
+    mut candidate: C,
+    edits: &[Edit],
+) -> Result<Report, Divergence> {
+    let mut eager = Eager::new(model.clone(), none_cats.to_vec());
+    let mut probes_run = compare(0, &eager, &candidate, None)?;
+    for (n, edit) in edits.iter().enumerate() {
+        eager.apply(edit);
+        candidate.apply(edit);
+        probes_run += compare(n + 1, &eager, &candidate, Some(edit))?;
+    }
+    Ok(Report {
+        steps: edits.len() + 1,
+        probes: probes_run,
+    })
+}

crates/improvise-incremental/src/lib.rs

@@ -0,0 +1,24 @@
+//! The plug: what the evaluator reads, and the oracle that answers it today.
+//! Reasoning: `context/code/improvise-incremental.md`.
+
+mod eager;
+mod edit;
+mod harness;
+
+pub use eager::Eager;
+pub use edit::Edit;
+pub use harness::{Divergence, Maintained, Probe, Report, differential, probes, same_value};
+
+use improvise_core::model::Aggregation;
+use improvise_core::model::cell::{CellKey, CellValue};
+
+/// The two things the evaluator reads (ADR 009 clause 2).
+pub trait Derived {
+    /// The value at `key`: the formula result when a formula owns it, else the
+    /// cell as stored or folded over what `agg` hides. `Model::evaluate_aggregated`.
+    fn value(&self, key: &CellKey, agg: Aggregation) -> Option<CellValue>;
+
+    /// The raw fold of the stored cells that match `partial`, by the measure's
+    /// aggregation. Never a formula.
+    fn raw_fold(&self, partial: &CellKey) -> Option<CellValue>;
+}

crates/improvise-incremental/tests/oracle.rs

@@ -0,0 +1,338 @@
+//! `Eager` against the evaluator and against hand-computed answers, and the
+//! differential harness against backends that are wrong on purpose.
+//! Evidence: `context/test-evidence/improvise-incremental.md`.
+
+use improvise_core::formula::ast::AggFunc;
+use improvise_core::formula::parse_formula;
+use improvise_core::model::cell::{CellKey, CellValue};
+use improvise_core::model::{Aggregation, Model};
+use improvise_core::view::Axis;
+use improvise_incremental::{Derived, Eager, Edit, Maintained, Probe, differential, probes};
+use std::path::Path;
+
+fn key(region: &str, product: &str, measure: &str) -> CellKey {
+    CellKey::new(vec![
+        ("Region".into(), region.into()),
+        ("Product".into(), product.into()),
+        ("_Measure".into(), measure.into()),
+    ])
+}
+
+fn partial(product: &str, measure: &str) -> CellKey {
+    CellKey::new(vec![
+        ("Product".into(), product.into()),
+        ("_Measure".into(), measure.into()),
+    ])
+}
+
+fn num(n: f64) -> Option<CellValue> {
+    Some(CellValue::Number(n))
+}
+
+fn text(s: &str) -> Option<CellValue> {
+    Some(CellValue::Text(s.into()))
+}
+
+fn mixed() -> Option<CellValue> {
+    Some(CellValue::Error("mixed".into()))
+}
+
+/// Region is the None category. Revenue and Cost are numbers; Note is text,
+/// unanimous under A and conflicting under B.
+fn hand_model() -> Model {
+    let mut m = Model::new("hand");
+    for (cat, items) in [("Region", ["East", "West"]), ("Product", ["A", "B"])] {
+        m.add_category(cat).unwrap();
+        for item in items {
+            m.category_mut(cat).unwrap().add_item(item);
+        }
+    }
+    for measure in ["Revenue", "Cost", "Note"] {
+        m.category_mut("_Measure").unwrap().add_item(measure);
+    }
+    let rows = [
+        ("East", "A", 10.0, 4.0, "n", "n"),
+        ("West", "A", 5.0, 2.0, "n", "n"),
+        ("East", "B", 7.0, 3.0, "x", "x"),
+        ("West", "B", 1.0, 1.0, "y", "y"),
+    ];
+    for (r, p, rev, cost, note, _) in rows {
+        m.set_cell(key(r, p, "Revenue"), CellValue::Number(rev));
+        m.set_cell(key(r, p, "Cost"), CellValue::Number(cost));
+        m.set_cell(key(r, p, "Note"), CellValue::Text(note.into()));
+    }
+    m.add_formula(parse_formula("Profit = Revenue - Cost", "_Measure").unwrap());
+    m
+}
+
+fn hand_eager() -> Eager {
+    Eager::new(hand_model(), vec!["Region".into()])
+}
+
+fn hand_edits() -> Vec<Edit> {
+    vec![
+        Edit::SetCell(key("East", "A", "Revenue"), CellValue::Number(20.0)),
+        Edit::SetMeasureAgg {
+            measure: "Revenue".into(),
+            func: AggFunc::Max,
+        },
+        Edit::ClearCell(key("West", "A", "Revenue")),
+        Edit::SetCell(key("West", "B", "Revenue"), CellValue::Text("oops".into())),
+        Edit::AddFormula(parse_formula("Double = Revenue * 2", "_Measure").unwrap()),
+        Edit::RemoveFormula {
+            target: "Profit".into(),
+            target_category: "_Measure".into(),
+        },
+    ]
+}
+
+#[test]
+fn eager_raw_fold_sums_by_default() {
+    assert_eq!(hand_eager().raw_fold(&partial("A", "Revenue")), num(15.0));
+}
+
+#[test]
+fn eager_raw_fold_follows_the_measures_aggregation() {
+    let mut eager = hand_eager();
+    eager.apply(&Edit::SetMeasureAgg {
+        measure: "Revenue".into(),
+        func: AggFunc::Max,
+    });
+    assert_eq!(eager.raw_fold(&partial("A", "Revenue")), num(10.0));
+    assert_eq!(eager.raw_fold(&partial("A", "Cost")), num(6.0));
+}
+
+#[test]
+fn eager_raw_fold_of_text_is_unanimous_or_mixed() {
+    let eager = hand_eager();
+    assert_eq!(eager.raw_fold(&partial("A", "Note")), text("n"));
+    assert_eq!(eager.raw_fold(&partial("B", "Note")), mixed());
+}
+
+#[test]
+fn eager_value_reads_a_formula_exactly_and_folded() {
+    let eager = hand_eager();
+    assert_eq!(
+        eager.value(&key("East", "A", "Profit"), Aggregation::Exact),
+        num(6.0)
+    );
+    assert_eq!(
+        eager.value(&partial("A", "Profit"), Aggregation::OverHidden),
+        num(9.0)
+    );
+}
+
+#[test]
+fn eager_follows_an_edit_to_the_cell_a_fold_reads() {
+    let mut eager = hand_eager();
+    eager.apply(&Edit::SetCell(
+        key("East", "A", "Revenue"),
+        CellValue::Number(20.0),
+    ));
+    assert_eq!(eager.raw_fold(&partial("A", "Revenue")), num(25.0));
+    assert_eq!(
+        eager.value(&partial("A", "Profit"), Aggregation::OverHidden),
+        num(19.0)
+    );
+}
+
+#[test]
+fn eager_numbers_among_text_are_mixed() {
+    let mut eager = hand_eager();
+    eager.apply(&Edit::SetCell(
+        key("West", "B", "Revenue"),
+        CellValue::Text("oops".into()),
+    ));
+    assert_eq!(eager.raw_fold(&partial("B", "Revenue")), mixed());
+}
+
+fn corpus() -> Vec<(String, Model, Vec<String>)> {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
+    let mut out = Vec::new();
+    for file in ["examples/demo.improv", "web/demo.improv"] {
+        let wb = improvise_io::persistence::load(&root.join(file)).unwrap();
+        for (name, view) in &wb.views {
+            let none = view
+                .categories_on(Axis::None)
+                .into_iter()
+                .map(String::from)
+                .collect();
+            out.push((format!("{file}:{name}"), wb.model.clone(), none));
+        }
+    }
+    out.push(("hand".into(), hand_model(), vec!["Region".into()]));
+    out
+}
+
+/// Edits built from the model's own first cell: overwrite, text, clear, a new
+/// record, a new aggregation, and the first formula removed then restored.
+fn edits_for(model: &Model) -> Vec<Edit> {
+    let Some((first, _)) = model.data.iter_cells().next() else {
+        return vec![];
+    };
+    let (last_cat, _) = first.0.last().unwrap().clone();
+    let measure = first.get("_Measure").map(String::from);
+    let fresh = first.clone().with(last_cat, "Zz-new");
+    let mut edits = vec![
+        Edit::SetCell(first.clone(), CellValue::Number(7.5)),
+        Edit::SetCell(first.clone(), CellValue::Text("t".into())),
+        Edit::SetCell(fresh.clone(), CellValue::Number(3.0)),
+        Edit::ClearCell(first),
+    ];
+    edits.extend(measure.map(|m| Edit::SetMeasureAgg {
+        measure: m,
+        func: AggFunc::Max,
+    }));
+    if let Some(f) = model.formulas().first() {
+        edits.push(Edit::RemoveFormula {
+            target: f.target.clone(),
+            target_category: f.target_category.clone(),
+        });
+        edits.push(Edit::AddFormula(f.clone()));
+    }
+    edits.push(Edit::ClearCell(fresh));
+    edits
+}
+
+#[test]
+fn eager_matches_the_evaluator_on_the_shipped_documents() {
+    for (name, model, none) in corpus() {
+        let eager = Eager::new(model.clone(), none.clone());
+        let mut direct = model.clone();
+        direct.recompute_formulas(&none);
+        let asked = probes(eager.model());
+        assert!(!asked.is_empty(), "{name}: no probes");
+        for probe in asked {
+            let want = match &probe {
+                Probe::Value(k, a) => direct.evaluate_aggregated(k, *a),
+                Probe::RawFold(k) => direct.evaluate_aggregated(k, Aggregation::OverHidden),
+            };
+            assert_eq!(probe.ask(&eager), want, "{name}: {probe:?}");
+        }
+    }
+}
+
+#[test]
+fn eager_agrees_with_itself_over_every_corpus_edit_sequence() {
+    for (name, model, none) in corpus() {
+        let edits = edits_for(&model);
+        let report = differential(
+            &model,
+            &none,
+            Eager::new(model.clone(), none.clone()),
+            &edits,
+        )
+        .unwrap_or_else(|d| panic!("{name}: {d}"));
+        assert_eq!(report.steps, edits.len() + 1, "{name}");
+        assert!(report.probes > 0, "{name}");
+    }
+}
+
+/// A wrong backend: an `Eager` told every edit, answering one thing wrongly.
+#[derive(Clone, Copy)]
+enum Fault {
+    IgnoresAggregation,
+    IgnoresMeasureAggregation,
+    GoesStaleAfterTwoEdits,
+    IgnoresClear,
+}
+
+struct Rival {
+    inner: Eager,
+    fault: Fault,
+    seen: usize,
+}
+
+impl Rival {
+    fn of(fault: Fault) -> Self {
+        Rival {
+            inner: hand_eager(),
+            fault,
+            seen: 0,
+        }
+    }
+}
+
+impl Derived for Rival {
+    fn value(&self, key: &CellKey, agg: Aggregation) -> Option<CellValue> {
+        match self.fault {
+            Fault::IgnoresAggregation => self.inner.value(key, Aggregation::Exact),
+            _ => self.inner.value(key, agg),
+        }
+    }
+
+    fn raw_fold(&self, partial: &CellKey) -> Option<CellValue> {
+        self.inner.raw_fold(partial)
+    }
+}
+
+impl Maintained for Rival {
+    fn apply(&mut self, edit: &Edit) {
+        self.seen += 1;
+        let skip = match (self.fault, edit) {
+            (Fault::IgnoresMeasureAggregation, Edit::SetMeasureAgg { .. }) => true,
+            (Fault::GoesStaleAfterTwoEdits, _) => self.seen > 2,
+            (Fault::IgnoresClear, Edit::ClearCell(_)) => true,
+            _ => false,
+        };
+        if !skip {
+            self.inner.apply(edit);
+        }
+    }
+}
+
+fn run_rival(
+    fault: Fault,
+) -> Result<improvise_incremental::Report, improvise_incremental::Divergence> {
+    differential(
+        &hand_model(),
+        &["Region".to_string()],
+        Rival::of(fault),
+        &hand_edits(),
+    )
+}
+
+#[test]
+fn the_harness_passes_a_backend_that_is_right() {
+    let report = differential(
+        &hand_model(),
+        &["Region".to_string()],
+        hand_eager(),
+        &hand_edits(),
+    )
+    .unwrap();
+    assert_eq!(report.steps, hand_edits().len() + 1);
+}
+
+#[test]
+fn the_harness_catches_a_backend_that_ignores_the_aggregation() {
+    let d = run_rival(Fault::IgnoresAggregation).unwrap_err();
+    assert_eq!(d.step, 0);
+    assert!(d.to_string().starts_with("divergence at step 0:"), "{d}");
+}
+
+#[test]
+fn the_harness_catches_a_backend_that_ignores_the_measure_aggregation() {
+    let d = run_rival(Fault::IgnoresMeasureAggregation).unwrap_err();
+    assert_eq!(d.step, 2, "{d}");
+}
+
+#[test]
+fn the_harness_catches_a_backend_that_goes_stale_after_two_edits() {
+    let d = run_rival(Fault::GoesStaleAfterTwoEdits).unwrap_err();
+    assert_eq!(d.step, 3, "{d}");
+}
+
+#[test]
+fn the_harness_catches_a_backend_that_ignores_a_clear() {
+    let d = run_rival(Fault::IgnoresClear).unwrap_err();
+    assert_eq!(d.step, 3, "{d}");
+    assert!(d.to_string().contains("eager"), "{d}");
+}
+
+#[test]
+fn the_harness_names_the_probe_the_two_backends_disagree_on() {
+    let d = run_rival(Fault::IgnoresClear).unwrap_err();
+    assert_ne!(d.eager, d.candidate);
+    assert!(matches!(d.probe, Probe::Value(..) | Probe::RawFold(_)));
+}

nix/ci.nix

@@ -50,6 +50,8 @@
     "numtower"
     # ADR 009 clause 1: std-only; nothing depends on it yet.
     "zset"
+    # ADR 009 clause 2: the Derived trait, Eager and the differential harness.
+    "improvise-incremental"
   ];
   pFlags = crates: lib.concatMapStringsSep " " (c: "-p ${c}") crates;
 

scripts/build-doc-site.mjs

@@ -69,6 +69,7 @@ const SECTIONS = [
       ["context/code/gp.md", "code/gp", "gp-tree and improvise-gp"],
       ["context/code/numtower.md", "code/numtower", "numtower, the numeric tower"],
       ["context/code/zset.md", "code/zset", "zset, the Z-set group"],
+      ["context/code/improvise-incremental.md", "code/improvise-incremental", "improvise-incremental, the plug"],
       ["context/code/kb-tool.md", "code/kb-tool", "The kb tool"],
       ["context/code/tooling.md", "code/tooling", "The repository's own checkers"],
     ],
@@ -89,6 +90,7 @@ const SECTIONS = [
       ["context/test-evidence/gp.md", "test-evidence/gp", "gp-tree and improvise-gp"],
       ["context/test-evidence/numtower.md", "test-evidence/numtower", "numtower"],
       ["context/test-evidence/zset.md", "test-evidence/zset", "zset"],
+      ["context/test-evidence/improvise-incremental.md", "test-evidence/improvise-incremental", "improvise-incremental"],
       ["context/test-evidence/tooling.md", "test-evidence/tooling", "The repository's own gates"],
       ["context/test-evidence/layout-cost.md", "test-evidence/layout-cost", "Layout and keypress cost"],
       ["context/test-evidence/scale-bisect.md", "test-evidence/scale-bisect", "Scale bisect"],