test(incremental): hidden members under every aggregation, evidence and the step 4 reasoning (improvise-u2le)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

context/code/improvise-incremental.md

@@ -145,3 +145,74 @@ about hidden items, totals or occupancy. Both are closed. These are the readings
    `Eager`-against-evaluator comparison as well.
 6. **`Probe` is `Hash + Eq`**, so `Asked` deduplicates on the probe itself (`Aggregation` gained
    `Hash` in core for it).
+
+## Step 4: the incremental backend (improvise-u2le)
+
+`Incremental` (`incremental.rs`) is the first `Derived` that keeps state. It owns a `Model`
+(the store) and `Shapes` (`shapes.rs`, the groups). `Maintained::apply` applies the edit to the
+model, drains the store's change log into a `CellDelta` and feeds the delta to every shape. A
+read asks a group, never the store, for what step 4 owns. The rest of this section is the readings
+taken where clause 4 and clause 7 leave a choice; evidence is in
+`context/test-evidence/improvise-incremental.md`.
+
+1. **A shape is a set of category names, not a None list.** Clause 4 says "a group is the cell key
+   minus the None categories". The trait is asked about keys of any category set (the harness
+   drops up to three coordinates of a stored key), so the unit is the set a key names: a `Shape`
+   maps the projection of a stored cell onto those categories to a `Group`, and a cell lands in
+   it only if it has all of them. A view's None list is one of these sets. Clause 7's "one
+   circuit serves every shape key with the same None list" holds trivially, since the sets are
+   shared by every reader.
+2. **A shape is built on its first read, by one pass over the store, and fed every delta after.**
+   Clause 7 makes a new key's first build a sliced job under ADR 008's job runner. That runner
+   does not exist, so the pass is synchronous and counted (`Meter::builds`, `built_cells`).
+   Nothing evicts a shape. `Shapes` holds its map in a `RefCell` because `Derived` reads take
+   `&self`; the cell is private to `shapes.rs` and no reference to it leaves a method.
+3. **A group is a count, a count of numbers and a multiset** (`group.rs`). One update touches one
+   map entry (O(log d)); the count and the multiset serve every function, so
+   `SetMeasureAgg` rebuilds nothing. A read of several groups (a collapsed group, a total) folds
+   their union without merging them: `fold_groups` sums the counts and takes the smallest of the
+   groups' minima, the largest of their maxima, or the one value they share.
+4. **Eager's rules are reproduced, not improved.** An all-numbers group reduces; any other group
+   is unanimous or `Error("mixed")`, so `Count` of a group holding text is that text. `Min` and
+   `Max` skip NaN and are `+inf`/`-inf` over only NaN, as `fold(INFINITY, f64::min)` is. The
+   multiset orders by the delta's total order, with NaN last, so the skip is a step past one
+   key. A group holding a NaN and a non-number is mixed, because NaN is not equal to itself in
+   `CellValue`'s `==` and is a distinct key here.
+5. **`Sum` and `Avg` stay with the store until step 6.** Their eager result depends on the
+   order the cells are added in (insertion order in the store, not `CellKey` order, whatever
+   clause 3 says), so a group cannot reproduce it without keeping that order. The backend routes
+   a read whose measure aggregation is `Sum` or `Avg` to `Model::evaluate_covering` and
+   `evaluate_spanning`, which is eager's own path. Step 6 replaces both sides with `ExactSum`.
+   Because `Sum` is the default, the corpus reaches the groups only once a `SetMeasureAgg`
+   edit has changed a measure; the differential tests apply those on purpose.
+6. **Formulas stay with the model until step 5.** A key a formula owns is read through the model,
+   and `apply` calls `Model::recompute_formulas`, as `Eager` does. Step 5 replaces the call.
+7. **A fold is the union of its members' groups.** `fold.rs` is `Model::group_expansion` read off
+   the public model: the cross product of the covered members of every group the key names and
+   every covered item of every category in `over` that it omits. Under `OverHidden` the members
+   are groups of one shape (the base's categories plus the spans'). Under `Exact` the cells
+   admitted are exactly the member keys, so the read is a lookup per member and a throwaway
+   `Group`. A total therefore reads one group per covered row, not one per stored cell, but still
+   one per row: clause 7's maintained totals with a subtracted hidden overlay are not built (see
+   "Not built").
+8. **Occupancy counts by inclusion and exclusion.** `occupancy(partial)` is the cells under
+   `partial`, where a coordinate naming a group matches each of its members' items, with no
+   hidden coordinate. For each way of choosing a member per group coordinate the count is the
+   group's cell count, less the cells holding a hidden item of a category the key leaves free,
+   by inclusion and exclusion over those categories: `N(P) - sum N(P + h) + ...`. The cost is
+   the product of the hidden items per free category, a group read each. A key naming a hidden
+   item itself is zero.
+9. **The meter counts operations.** `Meter` is `updates` (group updates a delta caused), `reads`
+   (groups a read looked at), `builds` and `built_cells`. `tests/measure.rs` fixes one table at
+   two sizes and asserts the counts: an overwrite costs two updates per shape whatever the table
+   holds, and a read of a group of 36 or 72 cells costs one group. Eager's cost for the same read
+   is `matching_values().len()`, which the test prints beside it. This is the local fixture the
+   brief asked for; ADR 009 step 4 says "measured at ADR 008's fixture", which needs ADR 008
+   (unapproved) and is deferred.
+
+### Not built in step 4
+
+- Maintained totals (clause 7): a total reads one group per covered row.
+- Eviction of shapes, and the build as a sliced job.
+- `Sum`, `Avg` (step 6) and formulas (step 5) are eager's, behind the same trait.
+- The ADR 008 fixture measurement.

context/test-evidence/improvise-incremental.md

@@ -303,3 +303,148 @@ its stored keys are asked by the subset probes. Fault: not skipping it fails thi
 
 The oracle recomputes its formulas under one None set; a view under another is not asked, since
 the oracle would answer it from the wrong cache. Fault: dropping the filter fails this test.
+
+## Step 4: the incremental backend (improvise-u2le)
+
+**How the faults were injected.** Twenty-two faults were put into `src/group.rs`, `src/shapes.rs`,
+`src/fold.rs` and `src/incremental.rs` one at a time, each run against the whole crate
+(`cargo test -p improvise-incremental --no-fail-fast`) and reverted before the next. Two survive
+because they change nothing observable (below); every other is named by at least one test:
+- a group ignores a retraction: `group::tests::a_retraction_removes_the_value_it_added`,
+  `group::tests::an_emptied_group_is_absent_and_keeps_no_entries`,
+  `incremental_matches_eager_on_the_shipped_documents`, `incremental_equals_eager_over_random_edit_sequences`;
+- a group never counts its numbers: `a_group_read_comes_from_a_maintained_group`,
+  `group::tests::min_and_max_skip_nan_and_are_infinite_over_nothing_else` and eight differentials;
+- `Max` takes the first key: `group::tests::min_and_max_skip_nan_and_are_infinite_over_nothing_else`,
+  `an_overwrite_costs_two_updates_per_shape_whatever_the_table_size` and five differentials;
+- `Count` counts one: `a_collapsed_group_reads_one_group_per_member_it_covers`, `group::tests::*`;
+- unanimity ignores disagreement: `group::tests::text_is_unanimous_or_mixed_across_groups`,
+  `incremental_matches_eager_on_the_hand_model_under_every_aggregation`;
+- an `Exact` fold reads groups: `incremental_matches_eager_on_a_group_read_without_a_view`,
+  `incremental_matches_eager_on_hidden_items_and_collapsed_groups`;
+- a formula-owned key is served from groups: `incremental_matches_eager_on_a_formula_read_under_every_aggregation`;
+- `Sum` read as `Count`: seven differentials, so the delegation is observed;
+- occupancy counts a hidden coordinate / ignores a free hidden category / adds the hidden cells
+  / ignores a named group: the hidden-item differentials and
+  `occupancy_with_a_hidden_item_reads_the_groups_it_subtracts` (the last two of those);
+- shapes are not fed after they are built: `a_group_read_comes_from_a_maintained_group`,
+  `an_overwrite_costs_two_updates_per_shape_whatever_the_table_size`;
+- a shape keeps an emptied group: `a_group_whose_cells_are_all_cleared_is_no_longer_read`;
+- the expansion ignores `covers`:
+  `incremental_matches_eager_on_a_collapsed_group_with_a_hidden_member_under_every_aggregation`;
+- the members ignore the spans: `a_collapsed_group_reads_one_group_per_member_it_covers`;
+- `apply` skips the model or leaves the formulas stale: the differentials, with and without formulas;
+- the measure's aggregation is ignored: `a_collapsed_group_reads_one_group_per_member_it_covers`,
+  `a_group_read_comes_from_a_maintained_group`;
+- `raw_fold` answers for a formula target: `incremental_matches_eager_on_a_pooled_total_of_a_ratio`,
+  `incremental_matches_eager_on_the_hand_model_under_every_aggregation`.
+
+**Faults that cannot be seen, and why.** `Group::smallest` reading the first key even when it is NaN
+returns NaN, and `f64::min(INFINITY, NaN)` is `INFINITY`, which is the answer for a group of only
+NaN. A projection that keeps the cells lacking a shape's category files them under a key that
+names fewer categories, and a read of that shape never asks for such a key.
+
+**The fixture, and a first version that was blind.** `hidden_workbook` draws the measure on
+no axis, so its cells carry no `_Measure` coordinate, the measure's aggregation never applies and the
+expansion fault above survived. `hidden_counted_workbook` draws the measure on the columns.
+
+### incremental_matches_eager_on_the_hand_model_under_every_aggregation
+
+Applies `Count`, `Min`, `Max`, `Avg`, `Sum` to a numeric measure and to a text measure, then a text
+over a number and a clear. Fails on a group that ignores retractions, one that never counts numbers,
+`Max` reading the first key and unanimity that ignores disagreement.
+
+### incremental_matches_eager_on_the_shipped_documents
+
+Both shipped documents and the hand and hidden fixtures, over `edits_for`. It reaches the group path
+only after the edit that sets a measure's aggregation, so it fails on the faults that touch the
+fold, not on one in the `Sum` path.
+
+### incremental_matches_eager_on_hidden_items_and_collapsed_groups
+
+East is hidden and Coastal collapsed. Fails on occupancy that counts a hidden coordinate and on an
+`Exact` fold that reads groups. The measure is not drawn here, so it does not see `covers`.
+
+### incremental_matches_eager_on_a_collapsed_group_with_a_hidden_member_under_every_aggregation
+
+The same workbook with the measure on the columns, so the Coastal row folds West only and the
+measure's aggregation applies to it. Fails on an expansion that ignores `covers`.
+
+### incremental_matches_eager_on_a_formula_read_under_every_aggregation
+
+`Profit` is a formula target, so its key must be read by the model. Fails when the backend serves a
+formula-owned key from groups.
+
+### incremental_matches_eager_on_a_group_read_without_a_view
+
+`grouped_model` with a NaN, a clear and every aggregation, once with no None category and once with
+Product. Fails on an `Exact` fold that reads groups.
+
+### incremental_matches_eager_on_a_cell_a_view_draws_that_drops_four_coordinates
+
+The wide fixture under the new backend: a drawn cell that drops four coordinates of every stored
+key. Fails on a group that never counts numbers and on shapes not fed after the build.
+
+### incremental_matches_eager_on_a_pooled_total_of_a_ratio
+
+The ratio fixture's formula `Margin` pooled over the rows. Fails when `apply` leaves the formulas
+stale and when a formula key is read as a raw fold.
+
+### incremental_equals_eager_over_random_edit_sequences
+
+Up to six edits drawn from each fixture's own cells (an overwrite, a record in a made-up item, a
+clear, a new aggregation) with values from a pool that has both zeros, two NaNs, an infinity, text
+and an error, 256 sequences over five fixtures, probing everything each view draws. Fails on every
+fault listed above that a deterministic test did not already see.
+
+### a_group_read_comes_from_a_maintained_group
+
+A `Min` read builds one shape and reads one group; after an overwrite the new minimum comes from the
+delta and the store is not scanned again. Fails on a group that never counts numbers, on shapes that
+are not fed after the build and on a backend that ignores the measure's aggregation.
+
+### an_overwrite_costs_two_updates_per_shape_whatever_the_table_size
+
+At 216 and 864 cells, with one and with two shapes built, one overwrite costs a retraction and an
+insertion per shape and builds nothing; the two sizes cost the same. Fails on shapes that are not
+fed, which cost nothing, and on `Max` reading the first key, since the read after the write is wrong.
+
+### a_read_visits_one_group_where_eager_visits_every_cell_it_covers
+
+A group read at two table sizes costs one group each; eager's `matching_values` visits the 36 and 72
+cells the region covers. The counts are deterministic operations; no wall clock is read.
+
+### a_collapsed_group_reads_one_group_per_member_it_covers
+
+Coast holds three regions, so a `Count` of it reads three groups and answers 108. Fails on members
+that ignore the spans and on a `Count` that counts once.
+
+### occupancy_with_a_hidden_item_reads_the_groups_it_subtracts
+
+A line with one hidden product is its cells less the hidden product's, and reads two groups (the
+line and the line under that product). Fails on occupancy that ignores or adds the hidden cells.
+
+### a_group_whose_cells_are_all_cleared_is_no_longer_read
+
+Clearing the only cell of a group removes the group, and a later read looks at none. Fails when a
+shape keeps the emptied group.
+
+### group::tests::a_retraction_removes_the_value_it_added
+
+A multiset that ignores a negative weight keeps the minimum it should have lost.
+
+### group::tests::an_emptied_group_is_absent_and_keeps_no_entries
+
+After the last retraction the group reads as absent and its map holds no zero-weight entry.
+
+### group::tests::min_and_max_skip_nan_and_are_infinite_over_nothing_else
+
+NaN is skipped by `Min` and `Max`, a group of only NaN gives `+inf` and `-inf` and still counts.
+
+### group::tests::a_fold_over_several_groups_is_the_fold_of_their_union
+
+The smallest of the minima, the largest of the maxima and the sum of the counts.
+
+### group::tests::text_is_unanimous_or_mixed_across_groups
+
+One shared text is that text, two are `mixed`, a number among text is `mixed`, and so is a NaN.

crates/improvise-incremental/tests/incremental.rs

@@ -9,8 +9,8 @@ use improvise_core::model::cell::CellValue;
 use improvise_core::view::View;
 use improvise_incremental::{Edit, Incremental, differential_in};
 use support::{
-    corpus, edits_for, grouped_model, hand_model, hidden_workbook, none_of, ratio_workbook,
-    wide_workbook,
+    corpus, edits_for, grouped_model, hand_model, hidden_counted_workbook, hidden_workbook,
+    none_of, ratio_workbook, wide_workbook,
 };
 
 fn run(model: &Model, views: &[View], none: &[String], edits: &[Edit]) -> Result<(), String> {
@@ -77,6 +77,27 @@ fn incremental_matches_eager_on_hidden_items_and_collapsed_groups() {
     run(&wb.model, &views, &none, &edits).unwrap();
 }
 
+#[test]
+fn incremental_matches_eager_on_a_collapsed_group_with_a_hidden_member_under_every_aggregation() {
+    let wb = hidden_counted_workbook();
+    let views: Vec<View> = wb.views.values().cloned().collect();
+    let none = none_of(&wb);
+    let mut edits = agg_edits("Revenue");
+    edits.push(Edit::SetCell(
+        support::key("Inland", "A", "Revenue"),
+        CellValue::Number(f64::NAN),
+    ));
+    edits.extend(agg_edits("Revenue"));
+    run(&wb.model, &views, &none, &edits).unwrap();
+}
+
+#[test]
+fn incremental_matches_eager_on_a_formula_read_under_every_aggregation() {
+    let model = hand_model();
+    let none = vec!["Region".to_string()];
+    run(&model, &[], &none, &agg_edits("Profit")).unwrap();
+}
+
 #[test]
 fn incremental_matches_eager_on_a_group_read_without_a_view() {
     let model = grouped_model();

crates/improvise-incremental/tests/measure.rs

@@ -181,3 +181,18 @@ fn occupancy_with_a_hidden_item_reads_the_groups_it_subtracts() {
         "the line, less p3 under it"
     );
 }
+
+#[test]
+fn a_group_whose_cells_are_all_cleared_is_no_longer_read() {
+    let mut backend = Incremental::new(table(2, AggFunc::Count), vec![]);
+    let one = cell(1, 1, 1);
+    assert_eq!(backend.raw_fold(&one), Some(CellValue::Number(1.0)));
+    backend.apply(&Edit::ClearCell(cell(1, 1, 1)));
+    let before = backend.meter();
+    assert_eq!(backend.raw_fold(&one), None, "nothing is left under it");
+    assert_eq!(
+        since(before, backend.meter()).reads,
+        0,
+        "and no empty group stands in for it"
+    );
+}

crates/improvise-incremental/tests/support/mod.rs

@@ -223,3 +223,13 @@ pub fn ratio_workbook() -> improvise_core::workbook::Workbook {
     view.set_axis("Product", Axis::None);
     wb
 }
+
+/// [`hidden_workbook`] drawn with the measure on the columns and Product folded: its cells
+/// carry a measure, so the measure's aggregation reaches every row, total and collapsed group.
+pub fn hidden_counted_workbook() -> improvise_core::workbook::Workbook {
+    let mut wb = hidden_workbook();
+    let view = wb.active_view_mut();
+    view.set_axis("Product", Axis::None);
+    view.set_axis("_Measure", Axis::Column);
+    wb
+}