docs(incremental): the step 4 groundwork and its evidence; box the divergence's probe (improvise-bw4p.3, improvise-bw4p.4)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

context/code/improvise-incremental.md

@@ -32,10 +32,8 @@ fold of the cells that match a partial key") and nothing else. These are the rea
    downstream, core cannot read through it, which is the clause's point.
 2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell
    as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is
-   `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy are not
-   in the trait, so the differential cannot probe them, and a hidden item (view state) cannot
-   be exercised at all. The ADR's differential compares "every drawn cell, total and occupancy
-   count"; improvise-bw4p.3 adds them. A group is exercised: a stored key with one coordinate
+   `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy joined the trait
+   in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate
    replaced by its item's group is a probe.
 3. **`raw_fold` is never a formula, and `Eager` enforces it.** For a partial naming a formula
    target `Eager::raw_fold` is `None`; otherwise it is `evaluate_aggregated(partial, OverHidden)`,
@@ -104,14 +102,38 @@ coordinates from a stored key is not asked about. `edit_probes`
 adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence
 reads `divergence at step N: <probe>: eager <a> != candidate <b>`.
 
-**A limit, stated plainly (improvise-bw4p.4).** The probe set is every stored key's
-sub-partials, not the keys a view draws. A view with four or more categories on None draws a cell
-that drops four or more coordinates of a stored key, and the harness checks none of those cells:
-a rival wrong only at `{C0=a, _Measure=Revenue}` over a five-category model passes. Neither
-shipped demo reaches the limit, so the corpus does not show it. Raising `MAX_DROPPED` is not the
-answer, because `sub_partials` walks all 2^n subsets of a key whatever the limit is (93ms at 20
-categories and 225ms at 22 in a debug build, per call). The fix is to probe the keys each view
-draws, with steps 3 and 4; until then a candidate on such a view is under-checked.
+## Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4)
+
+The subset probes above cannot see a drawn cell on a view with four or more None categories
+(it drops more than `MAX_DROPPED` coordinates of a stored key), and the trait could not be asked
+about hidden items, totals or occupancy. Both are closed. These are the readings taken.
+
+1. **`Reading` is how a view reads.** The aggregation (`Aggregation::of` its None categories)
+   and the hidden `(category, item)` pairs of `View::hidden_items`. `covers(cat, item)` is the
+   predicate `GridLayout::evaluator` passes as its `covers`. It is a value, not a closure, so a
+   probe can name it and a divergence can print it.
+2. **Three methods join the trait**: `covered(key, reading)` is `Model::evaluate_covering`;
+   `total(key, over, reading)` is `evaluate_spanning`, a column total as ONE fold over the
+   visible items of the omitted categories; `occupancy(partial, reading)` counts the stored
+   cells matching `partial` none of whose coordinates is hidden. Occupancy is a stored-cell
+   count, the quantity ADR 009 clause 7 keeps by axis item; "has a value" (what `prune_empty`
+   asks) is `covered(..).is_some()`, already probed.
+3. **`drawn_probes(model, views)` asks what each view draws**, built from the view's own
+   `GridLayout` (pruned as production builds it, records mode skipped): for every drawn cell
+   the three subset questions plus `Covered` and `Occupancy`; for every column `Total` over
+   the row categories and the column line's occupancy; for every row the row line's occupancy.
+   `differential_in(model, views, ..)` adds them at every step with the rest; `differential`
+   is that with no views. A view is a `View`, not a document, because the model holds none.
+4. **`sub_partials` is polynomial**: it enumerates the sets of at most `MAX_DROPPED` dropped
+   indices, not the 2^n masks. The probe set is the same; the cost at 26 coordinates falls from
+   about 800ms to under 70ms (debug build). `MAX_DROPPED` stays: it bounds the subset probes,
+   and the drawn probes carry the cells that drop more.
+5. **The hidden fixture is a workbook the corpus now includes** (`hidden`): no shipped demo
+   hides an item, so a backend wrong about hidden items would have passed the direct
+   `Eager`-against-evaluator comparison as well.
+6. **Not here**: a view's None categories fix the formulas' cache, and `Eager` holds one set.
+   The drawn probes of a view under another None set are read from that one cache, by both
+   sides alike, so they test the backend and not the evaluator's per-view recompute.
 
 The corpus the tests use is `examples/demo.improv` and `web/demo.improv`, loaded through
 `improvise_io::persistence::load`, and hand-built models (formulas, text, mixed values, a group).

context/test-evidence/improvise-incremental.md

@@ -215,3 +215,39 @@ The differential for step 3. A proptest (256 cases, up to 12 edits drawn from se
 both zeros and three NaN spellings) checks that the delta `DataStore` logs for an edit equals
 the store as a Z-set after it minus the store before. No `Derived` backend exists yet, so
 this is the harness's step-3 form: the delta against the store it describes.
+
+## the_harness_checks_a_cell_a_view_draws_that_drops_four_coordinates
+
+improvise-bw4p.4. Six categories, four on None, a rival wrong only at `{C0=a, _Measure=Revenue}`.
+Before the drawn probes the differential passed it (`is_err` failed); the cell drops four
+coordinates of a stored key and `MAX_DROPPED` is three. Now the first divergence is the
+drawn cell's exact value. Fault: `drawn_probes` skipping every view fails this and five more.
+
+## probing_a_key_of_twenty_six_coordinates_is_not_exponential
+
+The old `sub_partials` walked all 2^n masks (93ms at 20 categories, 225ms at 22). With the mask
+walk put back, 26 coordinates take 807ms against a 250ms bound; the combination walk takes
+under 70ms. The probe set is unchanged for keys of up to 3 coordinates over any bound.
+
+## the_harness_catches_a_fold_that_covers_a_hidden_item
+
+improvise-bw4p.3. East hidden, Coastal collapsed: the drawn Coastal/A is 5, not 15. A rival
+whose `covered` reads with no hidden items diverges there. Fault in `Eager::covered` (covers
+everything) fails this and `eager_matches_the_evaluator_on_the_shipped_documents`, which now
+includes the `hidden` document, because no shipped demo hides an item.
+
+## the_harness_catches_a_total_that_counts_a_hidden_item
+
+A rival total reading with no hidden items diverges at a column total. The same fault in
+`Eager::total` fails this, the next test and the shipped-documents comparison.
+
+## the_harness_catches_a_total_that_does_not_span_its_rows
+
+A rival answering `covered` for a column key (so the aggregate over its rows, not one fold over
+the pooled rows) is told apart from `total`: with no None category the first has no cell.
+
+## the_harness_catches_an_occupancy_that_counts_a_hidden_item
+
+A rival counting stored cells with no hidden filter diverges at a drawn cell's or line's
+occupancy. `Eager::occupancy` ignoring the hidden items fails this and the shipped-documents
+comparison, whose count is taken independently from `iter_cells` and `matches_partial`.

crates/improvise-incremental/src/harness.rs

@@ -40,7 +40,7 @@ impl Probe {
 pub struct Divergence {
     /// 0 is the model as built, `n` the state after edit `n`.
     pub step: usize,
-    pub probe: Probe,
+    pub probe: Box<Probe>,
     pub eager: Option<CellValue>,
     pub candidate: Option<CellValue>,
 }
@@ -123,10 +123,12 @@ fn drop_sets(n: usize, max: usize) -> Vec<Vec<usize>> {
 /// Every key `key` keeps after dropping at most `MAX_DROPPED` coordinates:
 /// polynomial in the key's length, not a walk of all its subsets.
 fn sub_partials(key: &CellKey) -> impl Iterator<Item = CellKey> + '_ {
-    drop_sets(key.0.len(), MAX_DROPPED).into_iter().map(move |dropped| {
-        let kept = (0..key.0.len()).filter(|i| !dropped.contains(i));
-        CellKey::new(kept.map(|i| key.0[i].clone()).collect())
-    })
+    drop_sets(key.0.len(), MAX_DROPPED)
+        .into_iter()
+        .map(move |dropped| {
+            let kept = (0..key.0.len()).filter(|i| !dropped.contains(i));
+            CellKey::new(kept.map(|i| key.0[i].clone()).collect())
+        })
 }
 
 /// Every question worth asking of `model`; the bound and the key set are
@@ -186,11 +188,13 @@ fn view_probes(view: &View, layout: &GridLayout) -> Vec<Probe> {
             all_three(key).into_iter().chain([covered, occupied])
         });
     let columns = (0..layout.col_count()).flat_map(|c| {
-        let total = layout.column_key(c).map(|key| {
-            Probe::Total(key, layout.row_cats.clone(), reading.clone())
-        });
+        let total = layout
+            .column_key(c)
+            .map(|key| Probe::Total(key, layout.row_cats.clone(), reading.clone()));
         let line = axis_partial(&layout.col_cats, layout.col_tuple(c));
-        total.into_iter().chain([Probe::Occupancy(line, reading.clone())])
+        total
+            .into_iter()
+            .chain([Probe::Occupancy(line, reading.clone())])
     });
     let rows = (0..layout.row_count()).map(|r| {
         let line = axis_partial(&layout.row_cats, layout.row_tuple(r));
@@ -219,7 +223,7 @@ fn compare(
         if !same_value(&want, &got) {
             return Err(Divergence {
                 step,
-                probe: probe.clone(),
+                probe: Box::new(probe.clone()),
                 eager: want,
                 candidate: got,
             });

crates/improvise-incremental/tests/oracle.rs

@@ -171,6 +171,14 @@ fn corpus() -> Vec<Doc> {
         }
     }
     out.push(("hand".into(), hand_model(), vec!["Region".into()], vec![]));
+    let hidden = hidden_workbook();
+    let views = hidden.views.values().cloned().collect();
+    out.push((
+        "hidden".into(),
+        hidden.model,
+        vec!["_Measure".into()],
+        views,
+    ));
     out
 }
 
@@ -412,7 +420,7 @@ fn the_harness_catches_a_backend_that_ignores_the_aggregation() {
     let d = run_rival(Fault::IgnoresAggregation).unwrap_err();
     assert_eq!(d.step, 0);
     assert_eq!(
-        d.probe,
+        *d.probe,
         Probe::Value(key("East", "A", "Profit"), Aggregation::OverHidden)
     );
     assert_eq!((d.eager, d.candidate), (None, num(6.0)));
@@ -445,7 +453,7 @@ fn the_harness_catches_a_wrong_folded_formula_value() {
     let d = run_rival(Fault::WrongFoldedFormula).unwrap_err();
     assert_eq!(d.step, 0);
     assert_eq!(
-        d.probe,
+        *d.probe,
         Probe::Value(partial("A", "Profit"), Aggregation::OverHidden)
     );
     assert_eq!((d.eager, d.candidate), (num(9.0), num(-999.0)));
@@ -463,7 +471,7 @@ fn the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula() {
 fn the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped() {
     let d = run_rival(Fault::WrongWhenTwoAreDropped).unwrap_err();
     assert_eq!(d.step, 0);
-    assert_eq!(d.probe, Probe::Value(only_revenue(), Aggregation::Exact));
+    assert_eq!(*d.probe, Probe::Value(only_revenue(), Aggregation::Exact));
     assert_eq!(d.candidate, num(-999.0));
 }
 
@@ -471,7 +479,7 @@ fn the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped
 fn the_harness_catches_a_raw_fold_that_returns_a_formula() {
     let d = run_rival(Fault::RawFoldReturnsFormula).unwrap_err();
     assert_eq!(d.step, 0);
-    assert!(matches!(d.probe, Probe::RawFold(_)), "{d}");
+    assert!(matches!(*d.probe, Probe::RawFold(_)), "{d}");
     assert_eq!(measure_of(&d.probe), Some("Profit"), "{d}");
     assert!(d.eager.is_none() && d.candidate.is_some(), "{d}");
 }
@@ -486,7 +494,7 @@ fn the_harness_catches_a_wrong_group_fold() {
         &[],
     )
     .unwrap_err();
-    assert_eq!(d.probe, Probe::Value(coastal(), Aggregation::Exact));
+    assert_eq!(*d.probe, Probe::Value(coastal(), Aggregation::Exact));
     assert_eq!((d.eager, d.candidate), (num(15.0), num(-999.0)));
 }
 
@@ -798,7 +806,7 @@ fn the_harness_checks_a_cell_a_view_draws_that_drops_four_coordinates() {
     let views: Vec<_> = wb.views.values().cloned().collect();
     let d = differential_in(&wb.model, &views, &none, rival, &[]).unwrap_err();
     assert_eq!(d.step, 0);
-    assert_eq!(d.probe, Probe::Value(wide_cell(), Aggregation::Exact));
+    assert_eq!(*d.probe, Probe::Value(wide_cell(), Aggregation::Exact));
 }
 
 /// East is hidden and Coastal is collapsed (a group is, by default), so the Coastal row folds West only;
@@ -822,7 +830,8 @@ fn hidden_workbook() -> improvise_core::workbook::Workbook {
         ("Inland", "A", 2.0),
     ];
     for (r, p, v) in cells {
-        wb.model.set_cell(key(r, p, "Revenue"), CellValue::Number(v));
+        wb.model
+            .set_cell(key(r, p, "Revenue"), CellValue::Number(v));
     }
     let view = wb.active_view_mut();
     view.set_axis("Region", Axis::Row);
@@ -897,26 +906,26 @@ fn the_harness_passes_a_view_reader_that_is_right() {
 #[test]
 fn the_harness_catches_a_fold_that_covers_a_hidden_item() {
     let d = run_blind(Some(Blindness::CoversIgnoreHidden)).unwrap_err();
-    assert!(matches!(d.probe, Probe::Covered(..)), "{d}");
+    assert!(matches!(*d.probe, Probe::Covered(..)), "{d}");
     assert_eq!((d.eager, d.candidate), (num(5.0), num(15.0)), "Coastal/A");
 }
 
 #[test]
 fn the_harness_catches_a_total_that_counts_a_hidden_item() {
     let d = run_blind(Some(Blindness::TotalsIgnoreHidden)).unwrap_err();
-    assert!(matches!(d.probe, Probe::Total(..)), "{d}");
+    assert!(matches!(*d.probe, Probe::Total(..)), "{d}");
 }
 
 #[test]
 fn the_harness_catches_a_total_that_does_not_span_its_rows() {
     let d = run_blind(Some(Blindness::TotalsDoNotSpan)).unwrap_err();
-    assert!(matches!(d.probe, Probe::Total(..)), "{d}");
+    assert!(matches!(*d.probe, Probe::Total(..)), "{d}");
 }
 
 #[test]
 fn the_harness_catches_an_occupancy_that_counts_a_hidden_item() {
     let d = run_blind(Some(Blindness::OccupancyCountsHidden)).unwrap_err();
-    assert!(matches!(d.probe, Probe::Occupancy(..)), "{d}");
+    assert!(matches!(*d.probe, Probe::Occupancy(..)), "{d}");
 }
 
 #[test]
@@ -933,12 +942,18 @@ fn a_view_is_drawn_as_cells_totals_and_occupancy() {
 }
 
 #[test]
-fn probing_a_key_of_twenty_two_coordinates_is_not_exponential() {
-    let coords = (0..22).map(|i| (format!("C{i:02}"), "a".to_string())).collect();
+fn probing_a_key_of_twenty_six_coordinates_is_not_exponential() {
+    let coords = (0..26)
+        .map(|i| (format!("C{i:02}"), "a".to_string()))
+        .collect();
     let mut m = Model::new("deep");
     m.set_cell(CellKey::new(coords), CellValue::Number(1.0));
     let started = std::time::Instant::now();
     let n = probes(&m).len();
     assert!(n > 0);
-    assert!(started.elapsed().as_millis() < 100, "{:?}", started.elapsed());
+    assert!(
+        started.elapsed().as_millis() < 250,
+        "{:?}",
+        started.elapsed()
+    );
 }