diff --git a/context/code/improvise-incremental.md b/context/code/improvise-incremental.md index 24469197..f9265fc6 100644 --- a/context/code/improvise-incremental.md +++ b/context/code/improvise-incremental.md @@ -32,10 +32,8 @@ fold of the cells that match a partial key") and nothing else. These are the rea downstream, core cannot read through it, which is the clause's point. 2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is - `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy are not - in the trait, so the differential cannot probe them, and a hidden item (view state) cannot - be exercised at all. The ADR's differential compares "every drawn cell, total and occupancy - count"; improvise-bw4p.3 adds them. A group is exercised: a stored key with one coordinate + `Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy joined the trait + in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate replaced by its item's group is a probe. 3. **`raw_fold` is never a formula, and `Eager` enforces it.** For a partial naming a formula target `Eager::raw_fold` is `None`; otherwise it is `evaluate_aggregated(partial, OverHidden)`, @@ -105,18 +103,45 @@ coordinates from a stored key is not asked about. `edit_probes` adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence reads `divergence at step N: : eager != candidate `. -**A limit, stated plainly (improvise-bw4p.4).** The probe set is every stored key's -sub-partials, not the keys a view draws. A view with four or more categories on None draws a cell -that drops four or more coordinates of a stored key, and the harness checks none of those cells: -a rival wrong only at `{C0=a, _Measure=Revenue}` over a five-category model passes. Neither -shipped demo reaches the limit, so the corpus does not show it. Raising `MAX_DROPPED` is not the -answer, because `sub_partials` walks all 2^n subsets of a key whatever the limit is (93ms at 20 -categories and 225ms at 22 in a debug build, per call). The fix is to probe the keys each view -draws, with steps 3 and 4; until then a candidate on such a view is under-checked. +## Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4) -The corpus the tests use is `examples/demo.improv` and `web/demo.improv`, loaded through -`improvise_io::persistence::load`, and hand-built models (formulas, text, mixed values, a group). -`eager_matches_the_evaluator_on_the_shipped_documents` takes every view's None categories; the -differential runs take the first per document, and the largest document only its first four -edits, to keep the binary under two seconds in a debug build. A proptest (256 cases) runs random -edit sequences against a candidate that rebuilds an `Eager` from scratch on each edit. +The subset probes above cannot see a drawn cell on a view with four or more None categories +(it drops more than `MAX_DROPPED` coordinates of a stored key), and the trait could not be asked +about hidden items, totals or occupancy. Both are closed. These are the readings taken. + +1. **`Reading` is how a view reads.** The aggregation (`Aggregation::of` its None categories) + and the hidden `(category, item)` pairs of `View::hidden_items`. `covers(cat, item)` is the + predicate `GridLayout::evaluator` passes as its `covers`. It is a value, not a closure, so a + probe can name it and a divergence can print it. A fold over a group or a spanned category + leaves hidden members out; a read of a key naming neither does not (a hidden East still counts + in `{Product=A}` when Region is None), while an occupancy count always leaves them out. The + count follows the drawn items, not the value; `Eager` mirrors production in both. +2. **Three methods join the trait**: `covered(key, reading)` is `Model::evaluate_covering`; + `total(key, over, reading)` is `evaluate_spanning`, a column total as ONE fold over the + visible items of the omitted categories; `occupancy(partial, reading)` is ADR 009 clause 7's + count by axis item: the stored cells matching `partial`, none of whose coordinates is hidden, + where a coordinate naming a group matches every stored member of that group (a collapsed group + is a remap, so the Coastal line of two visible members counts 2). A key naming a formula + target has no stored cells and is not asked. "Has a value" (what `prune_empty` asks) is + `covered(..).is_some()`, already probed. +3. **`drawn_probes(model, views, none_cats)` asks what each view draws**, built from the view's + own `GridLayout` (pruned as production builds it): for every drawn cell the three subset + questions plus `Covered` and `Occupancy`; for every column its `column_key` occupancy and, where + `GridLayout::draws_totals` (the condition `grid_body` also reads), a `Total` over the row + categories; for every row the occupancy of its cell key less the column coordinates. Lines + carry the page coordinates, as a cell's key does, so a page is its own slice. A records view is + skipped: its keys are synthetic `_Index`/`_Dim` coordinates naming no stored cell and it has + no totals, while its stored keys are the subset probes' business. A view whose None categories + differ from the oracle's is skipped, since `Eager` recomputes its formulas under one set (a + backend keeping a cache per aggregation class is therefore not exercised on the others). + `differential_in(model, views, ..)` adds them at every step; `differential` is that with no + views. A view is a `View`, not a document, because the model holds none. +4. **`sub_partials` is polynomial**: it enumerates the sets of at most `MAX_DROPPED` dropped + indices, not the 2^n masks. The probe set is unchanged for every key length (a test compares + it with the old walk up to 12 coordinates, and counts it at 64). `MAX_DROPPED` stays: it + bounds the subset probes, and the drawn probes carry the cells that drop more. +5. **The hidden fixture is a workbook the corpus now includes** (`hidden`): no shipped demo + hides an item, so a backend wrong about hidden items would have passed the direct + `Eager`-against-evaluator comparison as well. +6. **`Probe` is `Hash + Eq`**, so `Asked` deduplicates on the probe itself (`Aggregation` gained + `Hash` in core for it). diff --git a/context/test-evidence/improvise-incremental.md b/context/test-evidence/improvise-incremental.md index a96c7cb7..5ec3baf4 100644 --- a/context/test-evidence/improvise-incremental.md +++ b/context/test-evidence/improvise-incremental.md @@ -51,142 +51,142 @@ none did): - `Edit::ClearCell` applied as `SetCell(0.0)`, which both sides of a differential share and so only a direct `Eager` test can see: `eager_clear_cell_leaves_the_key_absent`. -## eager_raw_fold_sums_by_default +### eager_raw_fold_sums_by_default Hand-computed: Revenue under Product A is 10 + 5 = 15. Fails on the `raw_fold`-reads-`Exact` rival (a partial key names no stored cell, so the answer is `None`). -## eager_raw_fold_follows_the_measures_aggregation +### eager_raw_fold_follows_the_measures_aggregation After `measure_agg[Revenue] = Max` the same fold is 10, and Cost, unchanged, still sums to 6. A fold that always sums fails on the first; one that always takes the maximum fails on the second. -## eager_raw_fold_of_text_is_unanimous_or_mixed +### eager_raw_fold_of_text_is_unanimous_or_mixed Note under A is two equal texts and reads as that text; under B it is two different texts and reads `Error("mixed")`. A fold that drops text, which the evaluator used to do (gotcha 7), returns `None` for both. -## eager_numbers_among_text_are_mixed +### eager_numbers_among_text_are_mixed A text among numbers is `Error("mixed")`, not the sum of the numbers. -## eager_value_reads_a_formula_exactly_and_folded +### eager_value_reads_a_formula_exactly_and_folded `Profit = Revenue - Cost` reads 6 at East/A exactly, and 9 at Product A with Region over hidden: (10 + 5) - (4 + 2), the fold taken before the subtraction. Fails on the `value`-ignores-`Aggregation` rival. -## eager_follows_an_edit_to_the_cell_a_fold_reads +### eager_follows_an_edit_to_the_cell_a_fold_reads After East/A Revenue becomes 20 the fold is 25 and the folded Profit 19. The Profit half is what fails when `apply` forgets to recompute formulas: the folded cache is then stale. -## eager_matches_the_evaluator_on_the_shipped_documents +### eager_matches_the_evaluator_on_the_shipped_documents Both shipped documents, every view's None categories, and the hand model: every probe of `Eager` equals a direct `evaluate_aggregated` on a model recomputed under the same None categories, and the probe set is non-empty. Fails on both aggregation rivals. -## eager_agrees_with_itself_over_every_corpus_edit_sequence +### eager_agrees_with_itself_over_every_corpus_edit_sequence `differential` over a generated edit sequence on each corpus model returns a `Report` with `edits + 1` steps and a non-zero probe count. The count assertion is what a harness that compared nothing would fail; the step count is what one that stopped early would fail. -## the_harness_passes_a_backend_that_is_right +### the_harness_passes_a_backend_that_is_right The control for the `catches` tests: `Eager` as the candidate over the hand-built edits is a pass, so a harness that rejects everything is seen. -## the_harness_catches_a_backend_that_ignores_the_aggregation +### the_harness_catches_a_backend_that_ignores_the_aggregation A candidate that reads every `value` as `Exact` diverges at step 0, first at the full-key formula cell `{East, A, Profit}` over hidden (eager `None`, candidate `6`). -## the_harness_catches_a_backend_that_ignores_the_measure_aggregation +### the_harness_catches_a_backend_that_ignores_the_measure_aggregation A candidate that drops `SetMeasureAgg` diverges at step 2, the edit that sets it, on a Revenue key. -## the_harness_catches_a_backend_that_goes_stale_after_two_edits +### the_harness_catches_a_backend_that_goes_stale_after_two_edits A candidate that stops applying edits after the second diverges at step 3, on a Revenue key. A prefix of steps cannot see a backend that goes wrong late. -## the_harness_catches_a_backend_that_ignores_a_clear +### the_harness_catches_a_backend_that_ignores_a_clear A candidate that keeps a cleared cell diverges at step 3 and the message names it. -## the_harness_catches_a_wrong_folded_formula_value +### the_harness_catches_a_wrong_folded_formula_value Review of PR 343 (Codex P1): a rival wrong only at `value({Product=A, _Measure=Profit}, OverHidden)`, the Profit cell a view with Region on None draws, passed 128 probes, because formula cells were built over full keys only. Now the divergence is at step 0 on exactly that probe, eager `9`, candidate `-999`. -## the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula +### the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula Review of PR 343: probes were rebuilt from the post-edit model, so a candidate that ignores `RemoveFormula` was never asked about Profit again. Probes now accumulate over every step. The divergence is at the last step, on a Profit key, eager `None`, candidate `Some`. -## the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped +### the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped Review of PR 343 (Copilot): only one-coordinate partials and the empty key were probed. A rival wrong only at `{_Measure=Revenue}` now diverges at step 0 on `Value({_Measure=Revenue}, Exact)`. The bound is `MAX_DROPPED = 3` coordinates; a fold dropping more is not asked about. -## the_harness_catches_a_raw_fold_that_returns_a_formula +### the_harness_catches_a_raw_fold_that_returns_a_formula Review of PR 343 (Copilot): `raw_fold` is documented never to be a formula. A rival whose `raw_fold` is `value(partial, OverHidden)` diverges at step 0 on a `RawFold` of a Profit partial (eager `None`, candidate `Some`). It could not be seen while `Eager` itself returned the formula. -## the_harness_catches_a_wrong_group_fold +### the_harness_catches_a_wrong_group_fold A group name stands in for an item in the probe keys. A rival wrong only at `{Region=Coastal, Product=A, _Measure=Revenue}` diverges at step 0, eager `15`, candidate `-999`. -## eager_folds_a_group_it_is_asked_about +### eager_folds_a_group_it_is_asked_about The group fold in the grouped model is East + West = 15 exactly, so the rival test above has a true answer to differ from. -## eager_raw_fold_is_never_a_formula +### eager_raw_fold_is_never_a_formula At `{Product=A, _Measure=Profit}` the folded value is `9` and the raw fold is `None`. Fails on the fault that lets `raw_fold` read the formula cache. -## eager_add_and_remove_formula_change_what_a_key_reads +### eager_add_and_remove_formula_change_what_a_key_reads Review of PR 343: `Edit::AddFormula` as a no-op left all 25 tests green, because an `Edit` fault reaches both sides of a differential alike. `Double = Revenue * 2` reads `None` before the edit and `20` at East/A after it, and `None` again after `RemoveFormula`. Fails on `AddFormula` as a no-op. -## eager_clear_cell_leaves_the_key_absent +### eager_clear_cell_leaves_the_key_absent After `ClearCell` the key reads `None` and the model holds no cell. A `ClearCell` applied as `SetCell(0.0)` hits both sides of a differential identically, so only this direct test fails. -## same_value_follows_the_float_rules_and_is_exact_elsewhere +### same_value_follows_the_float_rules_and_is_exact_elsewhere NaN equals NaN, `-0.0` equals `0.0`, `1.0` differs from `1.0000000000000002`, texts and errors compare exactly, and `None` equals only `None`. Each of three one-line faults in `same_value` (NaN unequal, signed zero by bits, any two texts equal) survived the whole suite until this test existed. -## probes_cover_folds_formulas_measures_and_groups +### probes_cover_folds_formulas_measures_and_groups `probes` includes a raw fold, a full key, a folded formula cell under both aggregations, the two-dropped key `{_Measure=Revenue}`, the empty key and a group key. Review of PR 343 showed `probes` omitting formula cells, `_Measure` keys or raw folds left every test green. -## an_edit_names_its_key_as_two_value_probes_and_a_fold +### an_edit_names_its_key_as_two_value_probes_and_a_fold `edit_probes` of a `ClearCell` names the cleared key three ways, and of an edit naming no key names nothing. -## a_maintained_eager_equals_a_fresh_one_over_random_edit_sequences +### a_maintained_eager_equals_a_fresh_one_over_random_edit_sequences Clause 2 asks for a property test over a random edit sequence. A proptest (256 cases, up to four edits of five kinds over the hand-built model) runs `differential` with a candidate that rebuilds @@ -215,3 +215,91 @@ The differential for step 3. A proptest (256 cases, up to 12 edits drawn from se both zeros and three NaN spellings) checks that the delta `DataStore` logs for an edit equals the store as a Z-set after it minus the store before. No `Derived` backend exists yet, so this is the harness's step-3 form: the delta against the store it describes. + +### the_harness_checks_a_cell_a_view_draws_that_drops_four_coordinates + +improvise-bw4p.4. Six categories, four on None, a rival wrong only at `{C0=a, _Measure=Revenue}`. +Before the drawn probes the differential passed it (`is_err` failed); the cell drops four +coordinates of a stored key and `MAX_DROPPED` is three. Now the first divergence is the +drawn cell's exact value. Fault: `drawn_probes` skipping every view fails this and five more. + +### harness::tests::sub_partials_are_the_keys_the_old_mask_walk_gave + +The subset walk no longer enumerates 2^n masks. For every key of up to 12 coordinates its keys +equal the old mask walk's as sorted sets, so only the order changed. Fault: dropping at most +two coordinates instead of three fails this and the next test. No wall clock is read. + +### harness::tests::sub_partials_of_sixty_four_coordinates_number_the_choices_of_at_most_three + +At 64 coordinates the old `1u64 << n` cannot run, so only the combination walk answers; its +count must be the sum of C(64, k) for k up to 3, 43745. This replaces a 250ms timing bound +(removed: a debug-build wall clock fails on a slow runner and the old walk passes in release). + +### the_harness_catches_a_fold_that_covers_a_hidden_item + +improvise-bw4p.3. East hidden, Coastal collapsed: the drawn Coastal/A is 5, not 15. A rival +whose `covered` reads with no hidden items diverges there. Fault in `Eager::covered` (covers +everything) fails this and `eager_matches_the_evaluator_on_the_shipped_documents`, which now +includes the `hidden` document, because no shipped demo hides an item. + +### the_harness_catches_a_total_that_counts_a_hidden_item + +A rival total reading with no hidden items diverges at a column total. The same fault in +`Eager::total` fails this, the next test and the shipped-documents comparison. + +### the_harness_catches_a_total_that_does_not_span_its_rows + +Nothing is hidden and Margin is `Revenue / Cost`, so only the span can separate the rival: +the pooled total of `{_Measure=Margin}` over the rows is 15 / 6 for eager and `None` for a +backend answering `covered` at a column key. The first version used the hidden fixture and +failed through the hidden East instead; it passed once nothing was hidden. Fault: `Eager::total` +ignoring hidden items does not fail this test, as it must not. + +### the_harness_catches_an_occupancy_that_counts_a_hidden_item + +A rival counting stored cells with no hidden filter diverges at a drawn cell's or line's +occupancy. `Eager::occupancy` ignoring the hidden items fails this and the shipped-documents +comparison, whose count is taken independently from `iter_cells` and `matches_partial`. + +### a_collapsed_groups_occupancy_counts_its_visible_members + +Occupancy is keyed by axis item and a collapsed group is a remap to its visible members. The +Coastal line draws two cells (West's, East being hidden) and `Eager` once counted 0 because it +matched the group name literally. Fault: matching literally fails this test. + +### eager_reads_what_the_grid_evaluator_reads + +Compares `Eager::covered` and `total` with `GridLayout::evaluator`'s own `cell` and +`column_total` for every drawn cell and total, so the `Reading` is checked against the layout's +hidden items and not against the same `Model` call. Fault: `Reading::of_view` dropping every +hidden item fails this and four others; before this test the shipped-documents comparison passed it. + +### a_page_selection_narrows_the_line_occupancy_it_is_asked_under + +Lines carry the page coordinates, as a drawn cell's key does. Fault: building the row line from +the row categories alone asks `{Product=A}` across every page and fails this test. + +### every_row_and_column_line_is_asked_its_occupancy + +Pins the row-line probes (Inland, and the collapsed Coastal) and the column line. Fault: dropping +the row lines fails this test and the pruning test. + +### a_pruned_view_is_asked_only_about_the_lines_it_draws + +A row with no cells is asked about unpruned and not once `prune_empty` is set. Fault: building the +layout with `GridLayout::new`, which does not prune, fails it. + +### a_records_view_is_not_asked_about + +A records view draws synthetic `_Index`/`_Dim` keys that name no stored cell and has no totals; +its stored keys are asked by the subset probes. Fault: not skipping it fails this test. + +### no_total_is_asked_where_the_grid_draws_none + +`GridLayout::draws_totals` is the one condition `grid_body` and `drawn_probes` share. With +`_Measure` on the rows a column mixes measures and has no total. Fault: asking totals always fails it. + +### a_view_under_another_none_set_is_not_asked + +The oracle recomputes its formulas under one None set; a view under another is not asked, since +the oracle would answer it from the wrong cache. Fault: dropping the filter fails this test. diff --git a/crates/improvise-command/src/ui/viewport.rs b/crates/improvise-command/src/ui/viewport.rs index d0a2eaca..ad2dc12c 100644 --- a/crates/improvise-command/src/ui/viewport.rs +++ b/crates/improvise-command/src/ui/viewport.rs @@ -7,7 +7,6 @@ use improvise_protocol::header_level_widths; use crate::format::{FormatPlan, NumberFormat}; use crate::model::Model; -use crate::model::category::MEASURE_CATEGORY; use crate::model::cell::{CellKey, CellValue}; use crate::view::{AxisEntry, GridEval, GridLayout}; @@ -50,9 +49,7 @@ pub fn grid_body( // evaluated once (improvise-h6z1). The model does not change during it. let eval = layout.evaluator(model); // A column mixing measures has no total to show: `context/code/command.md`. - let measures_on_rows = layout.row_cats.iter().any(|c| c == MEASURE_CATEGORY); - let totalled = !records && !measures_on_rows && layout.row_count() > 0 && n > 0; - let totals: Vec> = match totalled { + let totals: Vec> = match layout.draws_totals() { true => (0..n).map(|ci| eval.column_total(ci)).collect(), false => Vec::new(), }; diff --git a/crates/improvise-core/src/model/types.rs b/crates/improvise-core/src/model/types.rs index eeb6f71a..cea45e01 100644 --- a/crates/improvise-core/src/model/types.rs +++ b/crates/improvise-core/src/model/types.rs @@ -339,7 +339,7 @@ fn eval_is(coordinate: Option<&str>, filter: &Filter) -> f64 { /// Whether a grid hides dimensions a cell key does not name. The emptiness of /// the view's `Axis::None` list is the whole question — `context/code/core.md`. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum Aggregation { /// The key names every dimension, so the cell is read as stored. Exact, diff --git a/crates/improvise-core/src/view/layout.rs b/crates/improvise-core/src/view/layout.rs index 37a0c133..d7187d22 100644 --- a/crates/improvise-core/src/view/layout.rs +++ b/crates/improvise-core/src/view/layout.rs @@ -991,6 +991,13 @@ impl GridLayout { Some(CellKey::new(coords)) } + /// Whether the grid draws a total under each column: not in records mode, + /// not when a column mixes measures (`_Measure` on the rows), not when empty. + pub fn draws_totals(&self) -> bool { + let measures_on_rows = self.row_cats.iter().any(|c| c == MEASURE_CATEGORY); + !self.is_records_mode() && !measures_on_rows && self.row_count() > 0 && self.col_count() > 0 + } + /// The coordinates a COLUMN answers for once its rows are aggregated away: /// the page/filter constraints and the column's own tuple (core.md §6). pub fn column_key(&self, col: usize) -> Option { diff --git a/crates/improvise-incremental/src/eager.rs b/crates/improvise-incremental/src/eager.rs index 06362b3a..846c950d 100644 --- a/crates/improvise-incremental/src/eager.rs +++ b/crates/improvise-incremental/src/eager.rs @@ -1,6 +1,6 @@ //! Today's evaluator behind the trait. Temporary (ADR 009 clause 2). -use crate::{Derived, Edit, Maintained}; +use crate::{Derived, Edit, Maintained, Reading}; use improvise_core::model::cell::{CellKey, CellValue}; use improvise_core::model::{Aggregation, Model}; @@ -23,6 +23,17 @@ impl Eager { } } +impl Eager { + /// Whether `item` names a group of `cat` that covers `held`, a stored item. + fn group_holds(&self, cat: &str, item: &str, held: Option<&str>) -> bool { + let Some(category) = self.model.category(cat) else { + return false; + }; + let members = category.items_in_group(item); + !category.items.contains_key(item) && held.is_some_and(|h| members.contains(&h)) + } +} + impl Derived for Eager { fn value(&self, key: &CellKey, agg: Aggregation) -> Option { self.model.evaluate_aggregated(key, agg) @@ -36,6 +47,31 @@ impl Derived for Eager { .evaluate_aggregated(partial, Aggregation::OverHidden), } } + + fn covered(&self, key: &CellKey, reading: &Reading) -> Option { + let covers = |cat: &str, item: &str| reading.covers(cat, item); + self.model.evaluate_covering(key, reading.agg, &covers) + } + + fn total(&self, key: &CellKey, over: &[String], reading: &Reading) -> Option { + let covers = |cat: &str, item: &str| reading.covers(cat, item); + self.model + .evaluate_spanning(key, over, reading.agg, &covers) + } + + fn occupancy(&self, partial: &CellKey, reading: &Reading) -> usize { + let drawn = |(cat, item): &(String, String)| reading.covers(cat, item); + let named_by = |stored: &CellKey| { + partial.0.iter().all(|(cat, item)| { + let held = stored.get(cat); + held == Some(item.as_str()) || self.group_holds(cat, item, held) + }) + }; + let cells = self.model.data.iter_cells(); + cells + .filter(|(k, _)| named_by(k) && k.0.iter().all(drawn)) + .count() + } } impl Maintained for Eager { diff --git a/crates/improvise-incremental/src/harness.rs b/crates/improvise-incremental/src/harness.rs index 37281d85..2fea4b32 100644 --- a/crates/improvise-incremental/src/harness.rs +++ b/crates/improvise-incremental/src/harness.rs @@ -1,8 +1,9 @@ //! The differential harness: one edit sequence, two backends, every probe after every step. -use crate::{Derived, Eager, Edit}; +use crate::{Derived, Eager, Edit, Reading}; use improvise_core::model::cell::{CellKey, CellValue}; use improvise_core::model::{Aggregation, Model}; +use improvise_core::view::{GridLayout, View}; use std::collections::HashSet; use std::fmt; @@ -12,10 +13,13 @@ pub trait Maintained: Derived { } /// One question put to both backends. -#[derive(Debug, Clone, PartialEq)] +#[derive(Debug, Clone, PartialEq, Eq, Hash)] pub enum Probe { Value(CellKey, Aggregation), RawFold(CellKey), + Covered(CellKey, Reading), + Total(CellKey, Vec, Reading), + Occupancy(CellKey, Reading), } impl Probe { @@ -23,6 +27,11 @@ impl Probe { match self { Probe::Value(key, agg) => backend.value(key, *agg), Probe::RawFold(partial) => backend.raw_fold(partial), + Probe::Covered(key, reading) => backend.covered(key, reading), + Probe::Total(key, over, reading) => backend.total(key, over, reading), + Probe::Occupancy(key, reading) => { + Some(CellValue::Number(backend.occupancy(key, reading) as f64)) + } } } } @@ -31,7 +40,7 @@ impl Probe { pub struct Divergence { /// 0 is the model as built, `n` the state after edit `n`. pub step: usize, - pub probe: Probe, + pub probe: Box, pub eager: Option, pub candidate: Option, } @@ -72,19 +81,14 @@ pub const MAX_DROPPED: usize = 3; /// The questions asked of a backend, once each, in the order first added. #[derive(Default)] struct Asked { - seen: HashSet<(CellKey, u8)>, + seen: HashSet, order: Vec, } impl Asked { fn extend(&mut self, probes: impl IntoIterator) { for p in probes { - let tag = match &p { - Probe::Value(k, Aggregation::Exact) => (k.clone(), 0), - Probe::Value(k, Aggregation::OverHidden) => (k.clone(), 1), - Probe::RawFold(k) => (k.clone(), 2), - }; - if self.seen.insert(tag) { + if self.seen.insert(p.clone()) { self.order.push(p); } } @@ -99,16 +103,31 @@ fn all_three(key: CellKey) -> [Probe; 3] { ] } -/// Every key `key` keeps after dropping at most `MAX_DROPPED` coordinates. +/// Every set of at most `max` indices below `n`, ascending within each set. +fn drop_sets(n: usize, max: usize) -> Vec> { + let mut all = vec![vec![]]; + let mut level: Vec> = vec![vec![]]; + for _ in 0..max { + level = level + .iter() + .flat_map(|set| { + let from = set.last().map_or(0, |last| last + 1); + (from..n).map(move |i| [set.as_slice(), &[i]].concat()) + }) + .collect(); + all.extend(level.iter().cloned()); + } + all +} + +/// Every key `key` keeps after dropping at most `MAX_DROPPED` coordinates: +/// polynomial in the key's length, not a walk of all its subsets. fn sub_partials(key: &CellKey) -> impl Iterator + '_ { - let n = key.0.len(); - (0..1u64 << n) - .filter(|mask| mask.count_ones() as usize <= MAX_DROPPED) - .map(move |mask| { - let kept = (0..n) - .filter(|i| mask >> i & 1 == 0) - .map(|i| key.0[i].clone()); - CellKey::new(kept.collect()) + drop_sets(key.0.len(), MAX_DROPPED) + .into_iter() + .map(move |dropped| { + let kept = (0..key.0.len()).filter(|i| !dropped.contains(i)); + CellKey::new(kept.map(|i| key.0[i].clone()).collect()) }) } @@ -140,6 +159,63 @@ pub fn probes(model: &Model) -> Vec { asked.order } +/// What each view under `none_cats` draws of `model`: cells, totals, occupancy. +/// Why, and what is skipped: `context/code/improvise-incremental.md`, step 4. +pub fn drawn_probes(model: &Model, views: &[View], none_cats: &[String]) -> Vec { + let sorted = |cats: &[String]| { + let mut cats = cats.to_vec(); + cats.sort(); + cats + }; + let oracle_set = sorted(none_cats); + views + .iter() + .map(|view| (view, GridLayout::with_frozen_records(model, view, None))) + .filter(|(_, layout)| !layout.is_records_mode()) + .filter(|(_, layout)| sorted(&layout.none_cats) == oracle_set) + .flat_map(|(view, layout)| view_probes(model, view, &layout)) + .collect() +} + +/// A stored cell count is asked of a key only if something is stored under it. +fn occupancy_probe(model: &Model, key: CellKey, reading: &Reading) -> Option { + model + .formula_for(&key) + .is_none() + .then(|| Probe::Occupancy(key, reading.clone())) +} + +fn view_probes(model: &Model, view: &View, layout: &GridLayout) -> Vec { + let reading = Reading::of_view(view, &layout.none_cats); + let cells = (0..layout.row_count()) + .flat_map(|r| (0..layout.col_count()).map(move |c| (r, c))) + .filter_map(|(r, c)| layout.cell_key(r, c)) + .flat_map(|key| { + let covered = Probe::Covered(key.clone(), reading.clone()); + let occupied = occupancy_probe(model, key.clone(), &reading); + all_three(key).into_iter().chain([covered]).chain(occupied) + }); + let columns = (0..layout.col_count()).flat_map(|c| { + let key = layout.column_key(c); + let total = key.clone().filter(|_| layout.draws_totals()); + let total = total.map(|k| Probe::Total(k, layout.row_cats.clone(), reading.clone())); + let line = key.and_then(|k| occupancy_probe(model, k, &reading)); + total.into_iter().chain(line) + }); + let rows = (0..layout.row_count()).flat_map(|r| { + let line = layout + .cell_key(r, 0) + .map(|k| without_all(k, &layout.col_cats)); + line.and_then(|k| occupancy_probe(model, k, &reading)) + }); + cells.chain(columns).chain(rows).collect() +} + +/// `key` less each of `cats`: a row's line is a cell without its column coordinates. +fn without_all(key: CellKey, cats: &[String]) -> CellKey { + cats.iter().fold(key, |k, cat| k.without(cat)) +} + /// What an edit names: its key, asked all three ways, after the edit. pub fn edit_probes(edit: &Edit) -> Vec { edit.key() @@ -160,7 +236,7 @@ fn compare( if !same_value(&want, &got) { return Err(Divergence { step, - probe: probe.clone(), + probe: Box::new(probe.clone()), eager: want, candidate: got, }); @@ -174,17 +250,30 @@ fn compare( pub fn differential( model: &Model, none_cats: &[String], + candidate: C, + edits: &[Edit], +) -> Result { + differential_in(model, &[], none_cats, candidate, edits) +} + +/// [`differential`], also asking everything `views` draw (`drawn_probes`). +pub fn differential_in( + model: &Model, + views: &[View], + none_cats: &[String], mut candidate: C, edits: &[Edit], ) -> Result { let mut eager = Eager::new(model.clone(), none_cats.to_vec()); let mut asked = Asked::default(); asked.extend(probes(eager.model())); + asked.extend(drawn_probes(eager.model(), views, none_cats)); let mut probes_run = compare(0, &eager, &candidate, &asked)?; for (n, edit) in edits.iter().enumerate() { eager.apply(edit); candidate.apply(edit); asked.extend(probes(eager.model())); + asked.extend(drawn_probes(eager.model(), views, none_cats)); asked.extend(edit_probes(edit)); probes_run += compare(n + 1, &eager, &candidate, &asked)?; } @@ -193,3 +282,41 @@ pub fn differential( probes: probes_run, }) } + +#[cfg(test)] +mod tests { + use super::*; + + fn old_mask_walk(key: &CellKey) -> Vec { + let n = key.0.len(); + (0..1u64 << n) + .filter(|mask| mask.count_ones() as usize <= MAX_DROPPED) + .map(|mask| { + let kept = (0..n).filter(|i| mask >> i & 1 == 0); + CellKey::new(kept.map(|i| key.0[i].clone()).collect()) + }) + .collect() + } + + fn key_of(n: usize) -> CellKey { + CellKey::new((0..n).map(|i| (format!("C{i:02}"), "a".into())).collect()) + } + + #[test] + fn sub_partials_are_the_keys_the_old_mask_walk_gave() { + for n in 0..=12 { + let mut now: Vec<_> = sub_partials(&key_of(n)).collect(); + let mut was = old_mask_walk(&key_of(n)); + now.sort(); + was.sort(); + assert_eq!(now, was, "{n} coordinates"); + } + } + + #[test] + fn sub_partials_of_sixty_four_coordinates_number_the_choices_of_at_most_three() { + let n = 64; + let choices = 1 + n + n * (n - 1) / 2 + n * (n - 1) * (n - 2) / 6; + assert_eq!(sub_partials(&key_of(n)).count(), choices); + } +} diff --git a/crates/improvise-incremental/src/lib.rs b/crates/improvise-incremental/src/lib.rs index 69f4a18a..bec59249 100644 --- a/crates/improvise-incremental/src/lib.rs +++ b/crates/improvise-incremental/src/lib.rs @@ -5,19 +5,21 @@ mod delta; mod eager; mod edit; mod harness; +mod reading; pub use delta::{CellDelta, DeltaValue, cell_delta, moved_records}; pub use eager::Eager; pub use edit::Edit; pub use harness::{ - Divergence, MAX_DROPPED, Maintained, Probe, Report, differential, edit_probes, probes, - same_value, + Divergence, MAX_DROPPED, Maintained, Probe, Report, differential, differential_in, + drawn_probes, edit_probes, probes, same_value, }; +pub use reading::Reading; use improvise_core::model::Aggregation; use improvise_core::model::cell::{CellKey, CellValue}; -/// The two things the evaluator reads (ADR 009 clause 2). +/// What the evaluator reads (ADR 009 clause 2) and what a view draws from it. pub trait Derived { /// The value at `key`: the formula result when a formula owns it, else the /// cell as stored or folded over what `agg` hides. `Model::evaluate_aggregated`. @@ -26,4 +28,15 @@ pub trait Derived { /// The raw fold of the stored cells that match `partial`, by the measure's /// aggregation. Never a formula: `None` for a partial naming a formula target. fn raw_fold(&self, partial: &CellKey) -> Option; + + /// `key` as a view reads it: a collapsed group folds only the members + /// `reading` leaves visible. `Model::evaluate_covering`. + fn covered(&self, key: &CellKey, reading: &Reading) -> Option; + + /// A column's total: `key` as ONE fold over every visible item of the + /// `over` categories it omits. `Model::evaluate_spanning`. + fn total(&self, key: &CellKey, over: &[String], reading: &Reading) -> Option; + + /// How many stored cells match `partial` with no hidden coordinate. + fn occupancy(&self, partial: &CellKey, reading: &Reading) -> usize; } diff --git a/crates/improvise-incremental/src/reading.rs b/crates/improvise-incremental/src/reading.rs new file mode 100644 index 00000000..90ab8895 --- /dev/null +++ b/crates/improvise-incremental/src/reading.rs @@ -0,0 +1,36 @@ +//! What a view reads a backend under: the aggregation and the items it hides. + +use improvise_core::model::Aggregation; +use improvise_core::view::View; +use std::collections::BTreeSet; + +/// One view's aggregation and hidden `(category, item)` pairs; which reads +/// leave them out is in `context/code/improvise-incremental.md`, step 4. +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct Reading { + pub agg: Aggregation, + hidden: BTreeSet<(String, String)>, +} + +impl Reading { + pub fn new(agg: Aggregation, hidden: impl IntoIterator) -> Self { + Reading { + agg, + hidden: hidden.into_iter().collect(), + } + } + + /// The reading of `view`, whose None categories are `none_cats`. + pub fn of_view(view: &View, none_cats: &[String]) -> Self { + let hidden = view + .hidden_items + .iter() + .flat_map(|(cat, items)| items.iter().map(|i| (cat.clone(), i.clone()))); + Self::new(Aggregation::of(none_cats), hidden) + } + + /// Whether the view draws `item` of `cat`. + pub fn covers(&self, cat: &str, item: &str) -> bool { + !self.hidden.contains(&(cat.to_string(), item.to_string())) + } +} diff --git a/crates/improvise-incremental/tests/oracle.rs b/crates/improvise-incremental/tests/oracle.rs index 9fa50334..ea84022e 100644 --- a/crates/improvise-incremental/tests/oracle.rs +++ b/crates/improvise-incremental/tests/oracle.rs @@ -5,10 +5,10 @@ use improvise_core::formula::ast::AggFunc; use improvise_core::formula::parse_formula; use improvise_core::model::cell::{CellKey, CellValue}; use improvise_core::model::{Aggregation, Model}; -use improvise_core::view::Axis; +use improvise_core::view::{Axis, GridLayout, View}; use improvise_incremental::{ - CellDelta, DeltaValue, Derived, Divergence, Eager, Edit, Maintained, Probe, cell_delta, - differential, probes, same_value, + CellDelta, DeltaValue, Derived, Divergence, Eager, Edit, Maintained, Probe, Reading, + cell_delta, differential, differential_in, drawn_probes, probes, same_value, }; use proptest::prelude::*; use std::path::Path; @@ -149,7 +149,9 @@ fn eager_numbers_among_text_are_mixed() { assert_eq!(eager.raw_fold(&partial("B", "Revenue")), mixed()); } -fn corpus() -> Vec<(String, Model, Vec)> { +type Doc = (String, Model, Vec, Vec); + +fn corpus() -> Vec { let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); let mut out = Vec::new(); for file in ["examples/demo.improv", "web/demo.improv"] { @@ -163,11 +165,16 @@ fn corpus() -> Vec<(String, Model, Vec)> { .collect(); if !seen.contains(&none) { seen.push(none.clone()); - out.push((format!("{file}:{name}"), wb.model.clone(), none)); + let views = wb.views.values().cloned().collect(); + out.push((format!("{file}:{name}"), wb.model.clone(), none, views)); } } } - out.push(("hand".into(), hand_model(), vec!["Region".into()])); + out.push(("hand".into(), hand_model(), vec!["Region".into()], vec![])); + let hidden = hidden_workbook(); + let views = hidden.views.values().cloned().collect(); + let none = none_of(&hidden); + out.push(("hidden".into(), hidden.model, none, views)); out } @@ -203,17 +210,36 @@ fn edits_for(model: &Model) -> Vec { #[test] fn eager_matches_the_evaluator_on_the_shipped_documents() { - for (name, model, none) in corpus() { + for (name, model, none, views) in corpus() { let eager = Eager::new(model.clone(), none.clone()); let mut direct = model.clone(); direct.recompute_formulas(&none); - let asked = probes(eager.model()); + let mut asked = probes(eager.model()); assert!(!asked.is_empty(), "{name}: no probes"); + asked.extend(drawn_probes(eager.model(), &views, &none)); for probe in asked { let want = match &probe { Probe::Value(k, a) => direct.evaluate_aggregated(k, *a), Probe::RawFold(k) if direct.formula_for(k).is_some() => None, Probe::RawFold(k) => direct.evaluate_aggregated(k, Aggregation::OverHidden), + Probe::Covered(k, r) => direct.evaluate_covering(k, r.agg, &|c, i| r.covers(c, i)), + Probe::Total(k, over, r) => { + direct.evaluate_spanning(k, over, r.agg, &|c, i| r.covers(c, i)) + } + Probe::Occupancy(p, r) => { + let drawn = |(c, i): &(String, String)| r.covers(c, i); + let names = |k: &CellKey| { + p.0.iter().all(|(c, i)| { + let held = k.get(c); + let members = direct.category(c).map(|x| x.items_in_group(i)); + held == Some(i.as_str()) + || held.is_some_and(|h| members.is_some_and(|m| m.contains(&h))) + }) + }; + let stored = direct.data.iter_cells().map(|(k, _)| k); + let n = stored.filter(|k| names(k) && k.0.iter().all(drawn)); + Some(CellValue::Number(n.count() as f64)) + } }; assert_eq!(probe.ask(&eager), want, "{name}: {probe:?}"); } @@ -223,7 +249,7 @@ fn eager_matches_the_evaluator_on_the_shipped_documents() { #[test] fn eager_agrees_with_itself_over_every_corpus_edit_sequence() { let mut files_done: Vec = Vec::new(); - for (name, model, none) in corpus() { + for (name, model, none, views) in corpus() { let file = name.split(':').next().unwrap().to_string(); if files_done.contains(&file) { continue; @@ -234,8 +260,9 @@ fn eager_agrees_with_itself_over_every_corpus_edit_sequence() { if model.data.iter_cells().count() > 40 { edits.truncate(4); } - let report = differential( + let report = differential_in( &model, + &views, &none, Eager::new(model.clone(), none.clone()), &edits, @@ -327,6 +354,18 @@ impl Derived for Rival { _ => self.inner.raw_fold(p), } } + + fn covered(&self, k: &CellKey, r: &Reading) -> Option { + self.inner.covered(k, r) + } + + fn total(&self, k: &CellKey, over: &[String], r: &Reading) -> Option { + self.inner.total(k, over, r) + } + + fn occupancy(&self, p: &CellKey, r: &Reading) -> usize { + self.inner.occupancy(p, r) + } } impl Maintained for Rival { @@ -351,7 +390,11 @@ fn only_revenue() -> CellKey { fn measure_of(probe: &Probe) -> Option<&str> { match probe { - Probe::Value(k, _) | Probe::RawFold(k) => k.get("_Measure"), + Probe::Value(k, _) + | Probe::RawFold(k) + | Probe::Covered(k, _) + | Probe::Total(k, _, _) + | Probe::Occupancy(k, _) => k.get("_Measure"), } } @@ -381,7 +424,7 @@ fn the_harness_catches_a_backend_that_ignores_the_aggregation() { let d = run_rival(Fault::IgnoresAggregation).unwrap_err(); assert_eq!(d.step, 0); assert_eq!( - d.probe, + *d.probe, Probe::Value(key("East", "A", "Profit"), Aggregation::OverHidden) ); assert_eq!((d.eager, d.candidate), (None, num(6.0))); @@ -414,7 +457,7 @@ fn the_harness_catches_a_wrong_folded_formula_value() { let d = run_rival(Fault::WrongFoldedFormula).unwrap_err(); assert_eq!(d.step, 0); assert_eq!( - d.probe, + *d.probe, Probe::Value(partial("A", "Profit"), Aggregation::OverHidden) ); assert_eq!((d.eager, d.candidate), (num(9.0), num(-999.0))); @@ -432,7 +475,7 @@ fn the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula() { fn the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped() { let d = run_rival(Fault::WrongWhenTwoAreDropped).unwrap_err(); assert_eq!(d.step, 0); - assert_eq!(d.probe, Probe::Value(only_revenue(), Aggregation::Exact)); + assert_eq!(*d.probe, Probe::Value(only_revenue(), Aggregation::Exact)); assert_eq!(d.candidate, num(-999.0)); } @@ -440,7 +483,7 @@ fn the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped fn the_harness_catches_a_raw_fold_that_returns_a_formula() { let d = run_rival(Fault::RawFoldReturnsFormula).unwrap_err(); assert_eq!(d.step, 0); - assert!(matches!(d.probe, Probe::RawFold(_)), "{d}"); + assert!(matches!(*d.probe, Probe::RawFold(_)), "{d}"); assert_eq!(measure_of(&d.probe), Some("Profit"), "{d}"); assert!(d.eager.is_none() && d.candidate.is_some(), "{d}"); } @@ -455,7 +498,7 @@ fn the_harness_catches_a_wrong_group_fold() { &[], ) .unwrap_err(); - assert_eq!(d.probe, Probe::Value(coastal(), Aggregation::Exact)); + assert_eq!(*d.probe, Probe::Value(coastal(), Aggregation::Exact)); assert_eq!((d.eager, d.candidate), (num(15.0), num(-999.0))); } @@ -569,6 +612,18 @@ impl Derived for Fresh { fn raw_fold(&self, p: &CellKey) -> Option { self.eager.raw_fold(p) } + + fn covered(&self, k: &CellKey, r: &Reading) -> Option { + self.eager.covered(k, r) + } + + fn total(&self, k: &CellKey, over: &[String], r: &Reading) -> Option { + self.eager.total(k, over, r) + } + + fn occupancy(&self, p: &CellKey, r: &Reading) -> usize { + self.eager.occupancy(p, r) + } } impl Maintained for Fresh { @@ -676,3 +731,422 @@ proptest! { prop_assert!(run.is_ok(), "{}", run.unwrap_err()); } } + +/// Six categories, four of them None in the one view: the cell it draws +/// drops four coordinates of every stored key. +fn wide_workbook() -> improvise_core::workbook::Workbook { + let mut wb = improvise_core::workbook::Workbook::new("wide"); + for c in ["C0", "C1", "C2", "C3", "C4"] { + wb.add_category(c).unwrap(); + wb.add_item(c, "a").unwrap(); + } + wb.add_item("C0", "b").unwrap(); + wb.add_item("_Measure", "Revenue").unwrap(); + for (c0, v) in [("a", 1.0), ("b", 2.0)] { + let k = CellKey::new(vec![ + ("C0".into(), c0.into()), + ("C1".into(), "a".into()), + ("C2".into(), "a".into()), + ("C3".into(), "a".into()), + ("C4".into(), "a".into()), + ("_Measure".into(), "Revenue".into()), + ]); + wb.model.set_cell(k, CellValue::Number(v)); + } + let view = wb.active_view_mut(); + view.set_axis("C0", Axis::Row); + view.set_axis("_Measure", Axis::Column); + for c in ["C1", "C2", "C3", "C4"] { + view.set_axis(c, Axis::None); + } + wb +} + +/// The None categories of the active view, virtual ones included: the set an +/// `Eager` standing for that view is built with. +fn none_of(wb: &improvise_core::workbook::Workbook) -> Vec { + let none = wb.active_view().categories_on(Axis::None); + none.into_iter().map(String::from).collect() +} + +fn wide_cell() -> CellKey { + CellKey::new(vec![ + ("C0".into(), "a".into()), + ("_Measure".into(), "Revenue".into()), + ]) +} + +struct WrongAt(Eager, CellKey); + +impl Derived for WrongAt { + fn value(&self, k: &CellKey, agg: Aggregation) -> Option { + if *k == self.1 { + num(-999.0) + } else { + self.0.value(k, agg) + } + } + fn raw_fold(&self, p: &CellKey) -> Option { + self.0.raw_fold(p) + } + + fn covered(&self, k: &CellKey, r: &Reading) -> Option { + self.0.covered(k, r) + } + + fn total(&self, k: &CellKey, over: &[String], r: &Reading) -> Option { + self.0.total(k, over, r) + } + + fn occupancy(&self, p: &CellKey, r: &Reading) -> usize { + self.0.occupancy(p, r) + } +} + +impl Maintained for WrongAt { + fn apply(&mut self, edit: &Edit) { + self.0.apply(edit); + } +} + +#[test] +fn the_harness_checks_a_cell_a_view_draws_that_drops_four_coordinates() { + let wb = wide_workbook(); + let none = none_of(&wb); + let rival = WrongAt(Eager::new(wb.model.clone(), none.clone()), wide_cell()); + let views: Vec<_> = wb.views.values().cloned().collect(); + let d = differential_in(&wb.model, &views, &none, rival, &[]).unwrap_err(); + assert_eq!(d.step, 0); + assert_eq!(*d.probe, Probe::Value(wide_cell(), Aggregation::Exact)); +} + +/// East is hidden and Coastal is collapsed (a group is, by default), so the Coastal row folds West only; +/// Inland has a cell under A only, so B's column holds just Coastal's. +fn hidden_workbook() -> improvise_core::workbook::Workbook { + let mut wb = improvise_core::workbook::Workbook::new("hidden"); + for c in ["Region", "Product"] { + wb.add_category(c).unwrap(); + } + wb.add_item_in_group("Region", "East", "Coastal").unwrap(); + wb.add_item_in_group("Region", "West", "Coastal").unwrap(); + wb.add_item("Region", "Inland").unwrap(); + wb.add_item("Product", "A").unwrap(); + wb.add_item("Product", "B").unwrap(); + wb.add_item("_Measure", "Revenue").unwrap(); + let cells = [ + ("East", "A", 10.0), + ("West", "A", 5.0), + ("East", "B", 20.0), + ("West", "B", 1.0), + ("Inland", "A", 2.0), + ]; + for (r, p, v) in cells { + wb.model + .set_cell(key(r, p, "Revenue"), CellValue::Number(v)); + } + let view = wb.active_view_mut(); + view.set_axis("Region", Axis::Row); + view.set_axis("Product", Axis::Column); + view.set_axis("_Measure", Axis::None); + view.hide_item("Region", "East"); + wb +} + +/// A backend that reads the view wrongly in exactly one way. +#[derive(Clone, Copy)] +enum Blindness { + CoversIgnoreHidden, + TotalsIgnoreHidden, + TotalsDoNotSpan, + OccupancyCountsHidden, +} + +struct Blind(Eager, Blindness); + +impl Derived for Blind { + fn value(&self, k: &CellKey, agg: Aggregation) -> Option { + self.0.value(k, agg) + } + fn raw_fold(&self, p: &CellKey) -> Option { + self.0.raw_fold(p) + } + fn covered(&self, k: &CellKey, r: &Reading) -> Option { + match self.1 { + Blindness::CoversIgnoreHidden => self.0.covered(k, &Reading::new(r.agg, [])), + _ => self.0.covered(k, r), + } + } + fn total(&self, k: &CellKey, over: &[String], r: &Reading) -> Option { + match self.1 { + Blindness::TotalsIgnoreHidden => self.0.total(k, over, &Reading::new(r.agg, [])), + Blindness::TotalsDoNotSpan => self.0.covered(k, r), + _ => self.0.total(k, over, r), + } + } + fn occupancy(&self, p: &CellKey, r: &Reading) -> usize { + match self.1 { + Blindness::OccupancyCountsHidden => self.0.occupancy(p, &Reading::new(r.agg, [])), + _ => self.0.occupancy(p, r), + } + } +} + +impl Maintained for Blind { + fn apply(&mut self, edit: &Edit) { + self.0.apply(edit); + } +} + +fn run_blind(fault: Option) -> Result { + let wb = hidden_workbook(); + let edit = Edit::SetCell(key("West", "A", "Revenue"), CellValue::Number(6.0)); + run_blind_on(&wb, fault, edit) +} + +fn run_blind_on( + wb: &improvise_core::workbook::Workbook, + fault: Option, + edit: Edit, +) -> Result { + let none = none_of(wb); + let eager = Eager::new(wb.model.clone(), none.clone()); + let views: Vec<_> = wb.views.values().cloned().collect(); + match fault { + Some(f) => differential_in(&wb.model, &views, &none, Blind(eager, f), &[edit]), + None => differential_in(&wb.model, &views, &none, eager, &[edit]), + } +} + +/// Nothing is hidden and Margin is a ratio: the pooled total of Margin is +/// Revenue over Cost summed across the rows, which no single row's fold gives. +fn ratio_workbook() -> improvise_core::workbook::Workbook { + let mut wb = improvise_core::workbook::Workbook::new("ratio"); + for c in ["Region", "Product"] { + wb.add_category(c).unwrap(); + } + wb.add_item("Region", "East").unwrap(); + wb.add_item("Region", "West").unwrap(); + wb.add_item("Product", "A").unwrap(); + for m in ["Revenue", "Cost"] { + wb.add_item("_Measure", m).unwrap(); + } + for (r, rev, cost) in [("East", 10.0, 4.0), ("West", 5.0, 2.0)] { + wb.model + .set_cell(key(r, "A", "Revenue"), CellValue::Number(rev)); + wb.model + .set_cell(key(r, "A", "Cost"), CellValue::Number(cost)); + } + wb.model + .add_formula(parse_formula("Margin = Revenue / Cost", "_Measure").unwrap()); + let view = wb.active_view_mut(); + view.set_axis("Region", Axis::Row); + view.set_axis("_Measure", Axis::Column); + view.set_axis("Product", Axis::None); + wb +} + +#[test] +fn the_harness_passes_a_view_reader_that_is_right() { + run_blind(None).unwrap(); +} + +#[test] +fn the_harness_catches_a_fold_that_covers_a_hidden_item() { + let d = run_blind(Some(Blindness::CoversIgnoreHidden)).unwrap_err(); + assert!(matches!(*d.probe, Probe::Covered(..)), "{d}"); + assert_eq!((d.eager, d.candidate), (num(5.0), num(15.0)), "Coastal/A"); +} + +#[test] +fn the_harness_catches_a_total_that_counts_a_hidden_item() { + let d = run_blind(Some(Blindness::TotalsIgnoreHidden)).unwrap_err(); + assert!(matches!(*d.probe, Probe::Total(..)), "{d}"); +} + +#[test] +fn the_harness_catches_a_total_that_does_not_span_its_rows() { + let edit = Edit::SetCell(key("East", "A", "Cost"), CellValue::Number(5.0)); + let run = |fault| run_blind_on(&ratio_workbook(), fault, edit.clone()); + run(None).unwrap(); + let d = run(Some(Blindness::TotalsDoNotSpan)).unwrap_err(); + assert!(matches!(*d.probe, Probe::Total(..)), "{d}"); + assert_eq!(measure_of(&d.probe), Some("Margin"), "{d}"); + let pooled = 15.0 / 6.0; + assert_eq!( + (d.step, d.eager.clone(), d.candidate.clone()), + (0, num(pooled), None), + "{d}" + ); +} + +#[test] +fn the_harness_catches_an_occupancy_that_counts_a_hidden_item() { + let d = run_blind(Some(Blindness::OccupancyCountsHidden)).unwrap_err(); + assert!(matches!(*d.probe, Probe::Occupancy(..)), "{d}"); +} + +#[test] +fn a_view_is_drawn_as_cells_totals_and_occupancy() { + let wb = hidden_workbook(); + let views: Vec<_> = wb.views.values().cloned().collect(); + let drawn = drawn_probes(&wb.model, &views, &none_of(&wb)); + let kinds = [ + drawn.iter().any(|p| matches!(p, Probe::Covered(..))), + drawn.iter().any(|p| matches!(p, Probe::Total(..))), + drawn.iter().any(|p| matches!(p, Probe::Occupancy(..))), + ]; + assert_eq!(kinds, [true; 3]); +} + +fn coastal_line() -> CellKey { + CellKey::new(vec![("Region".into(), "Coastal".into())]) +} + +#[test] +fn a_collapsed_groups_occupancy_counts_its_visible_members() { + let wb = hidden_workbook(); + let eager = Eager::new(wb.model.clone(), none_of(&wb)); + let reading = Reading::of_view(wb.active_view(), &none_of(&wb)); + assert_eq!(eager.occupancy(&coastal_line(), &reading), 2); + let east = CellKey::new(vec![("Region".into(), "East".into())]); + assert_eq!(eager.occupancy(&east, &reading), 0, "East is hidden"); +} + +fn drawn_of(wb: &improvise_core::workbook::Workbook) -> Vec { + let views: Vec<_> = wb.views.values().cloned().collect(); + drawn_probes(&wb.model, &views, &none_of(wb)) +} + +fn occupancy_keys(probes: &[Probe]) -> Vec { + let keyed = |p: &Probe| match p { + Probe::Occupancy(k, _) => Some(k.clone()), + _ => None, + }; + probes.iter().filter_map(keyed).collect() +} + +fn line(coords: &[(&str, &str)]) -> CellKey { + CellKey::new( + coords + .iter() + .map(|(c, i)| (c.to_string(), i.to_string())) + .collect(), + ) +} + +/// What the grid's own evaluator reads for every cell and total each view draws. +#[test] +fn eager_reads_what_the_grid_evaluator_reads() { + let mut compared = 0; + for (name, model, none, views) in corpus() { + let eager = Eager::new(model, none.clone()); + for view in &views { + let layout = GridLayout::with_frozen_records(eager.model(), view, None); + let mut sorted = layout.none_cats.clone(); + sorted.sort(); + let mut mine = none.clone(); + mine.sort(); + if layout.is_records_mode() || sorted != mine { + continue; + } + let reading = Reading::of_view(view, &layout.none_cats); + let grid = layout.evaluator(eager.model()); + for (r, c) in + (0..layout.row_count()).flat_map(|r| (0..layout.col_count()).map(move |c| (r, c))) + { + let k = layout.cell_key(r, c).unwrap(); + assert_eq!(eager.covered(&k, &reading), grid.cell(&k), "{name}: {k:?}"); + compared += 1; + } + for c in (0..layout.col_count()).filter(|_| layout.draws_totals()) { + let k = layout.column_key(c).unwrap(); + let want = grid.column_total(c); + assert_eq!( + eager.total(&k, &layout.row_cats, &reading), + want, + "{name}: {k:?}" + ); + compared += 1; + } + } + } + assert!(compared > 100, "{compared}"); +} + +#[test] +fn a_page_selection_narrows_the_line_occupancy_it_is_asked_under() { + let mut wb = hidden_workbook(); + let view = wb.active_view_mut(); + view.set_axis("Region", Axis::Page); + view.set_axis("Product", Axis::Row); + view.set_page_selection("Region", "West"); + let keys = occupancy_keys(&drawn_of(&wb)); + assert!( + keys.contains(&line(&[("Product", "A"), ("Region", "West")])), + "{keys:?}" + ); + assert!(!keys.contains(&line(&[("Product", "A")])), "{keys:?}"); +} + +#[test] +fn every_row_and_column_line_is_asked_its_occupancy() { + let keys = occupancy_keys(&drawn_of(&hidden_workbook())); + let rows = [ + line(&[("Region", "Inland")]), + line(&[("Region", "Coastal")]), + ]; + assert!(rows.iter().all(|k| keys.contains(k)), "{keys:?}"); + assert!(keys.contains(&line(&[("Product", "B")])), "{keys:?}"); +} + +#[test] +fn a_pruned_view_is_asked_only_about_the_lines_it_draws() { + let mut wb = hidden_workbook(); + wb.add_item("Region", "Empty").unwrap(); + let empty = line(&[("Region", "Empty")]); + assert!( + occupancy_keys(&drawn_of(&wb)).contains(&empty), + "drawn unpruned" + ); + wb.active_view_mut().prune_empty = true; + assert!( + !occupancy_keys(&drawn_of(&wb)).contains(&empty), + "pruned away" + ); +} + +#[test] +fn a_records_view_is_not_asked_about() { + let mut wb = hidden_workbook(); + let view = wb.active_view_mut(); + view.set_axis("_Index", Axis::Row); + view.set_axis("_Dim", Axis::Column); + for cat in ["Region", "Product", "_Measure"] { + view.set_axis(cat, Axis::None); + } + assert!(view.is_records_layout()); + assert!(drawn_of(&wb).is_empty()); +} + +#[test] +fn no_total_is_asked_where_the_grid_draws_none() { + let mut wb = ratio_workbook(); + let totals = |wb: &improvise_core::workbook::Workbook| { + drawn_of(wb) + .iter() + .filter(|p| matches!(p, Probe::Total(..))) + .count() + }; + assert!(totals(&wb) > 0, "totals are drawn here"); + let view = wb.active_view_mut(); + view.set_axis("_Measure", Axis::Row); + view.set_axis("Region", Axis::Column); + assert_eq!(totals(&wb), 0, "a column mixing measures has no total"); +} + +#[test] +fn a_view_under_another_none_set_is_not_asked() { + let wb = hidden_workbook(); + let views: Vec<_> = wb.views.values().cloned().collect(); + assert!(drawn_probes(&wb.model, &views, &["Region".to_string()]).is_empty()); +}