diff --git a/Cargo.nix b/Cargo.nix index cacfd3f9..6db877fd 100644 --- a/Cargo.nix +++ b/Cargo.nix @@ -3095,6 +3095,11 @@ rec { } ]; devDependencies = [ + { + name = "improvise-core"; + packageId = "improvise-core"; + features = [ "test-support" ]; + } { name = "improvise-io"; packageId = "improvise-io"; diff --git a/context/code/improvise-incremental.md b/context/code/improvise-incremental.md index 0791e1de..c8021050 100644 --- a/context/code/improvise-incremental.md +++ b/context/code/improvise-incremental.md @@ -427,3 +427,50 @@ from the source's folds, never pushed down; `Derived` stays in this crate. `tests/support/live.rs` holds `LiveTable`, the scripted source (an indexed store behind a lock, with a pending list and a counter of snapshot rows) and `Fed`, an `Incremental` over one that a differential can drive: a cell edit becomes a write to the table followed by `sync`. + +### Query pushdown (`Pushdown`) + +For a source that answers aggregate queries and reports nothing. `Pushdown::new(definition, source, +none_cats)` puts the model over the source (`Model::over`) and keeps nothing else: no shapes, no +change log, no stems. These are the readings taken. + +1. **`AggregateQuery` is `CellSource`; there are two traits, not three.** The brief's `AggregateQuery` + (`fold`, `count`) is the trait core already has: `summarize` is the fold, `count` the count. `Derived` + stays here (user ruling) and stays a different thing: the values a view draws, which a backend + builds from a source's aggregates and its own state. A source implements `CellSource` and a backend + implements `Derived`. +2. **Folds are queries, formula cells are computed in RAM.** `value`, `raw_fold`, `covered` and `total` are + the model's evaluator over `Model::source()`, so a fold is one `summarize` and the rest is arithmetic + on the answers. The formula cache is the one thing `Pushdown` holds. That is the user's ruling + (2026-10-10), and it makes the evaluator hook the only change to core's evaluation. +3. **Staleness is `refresh()`.** A stored cell or a fold is current at the moment of the read. A formula + cell is the cache's value until `refresh` recomputes it from the source as it now stands, which + costs one `distinct_without` per formula and the folds each formula reads. When to refresh is the + host's (ADR 003's "polled, triggered, or explicitly stale"); nothing here schedules it. A + delivery layer that wants notices from a pushdown-backed shape has no moved keys to name, so it + marks them "correct" (ADR 009 clause 9), which is a bead (below). +4. **Occupancy is one `count`.** A group in the partial becomes a span over its own name and its + members; the view's hidden pairs go as `excluding`. `Eager::occupancy` lists every key to say the + same thing. +5. **`define(edit)` takes a formula or an aggregation edit and refreshes.** A cell edit is the source's: + `Model::set_cell` refuses it in a debug build. +6. **What is and is not proven.** The tests run against `Remote` (`tests/support/remote.rs`), a scanning + in-memory table with counters, that shares no evaluator with `Eager` and answers queries over rows. + Without a real SQL engine only the **contract** is proven: that a source answering these six + queries as specified gives `Eager`'s answers, and what each read costs in queries. Nothing here + shows that SQLite can answer `summarize` exactly (an exact sum is not what its `SUM` returns: + `context/code/source.md`), or what a query costs. The first real source is improvise-rucf, still + deferred. + +### What step 7 did not predict + +- **The hook came before the change feed**, not after (a fed backend's formulas are evaluated over the + source; see `context/code/source.md`). +- **`TableSource` and `LiveTable` found nothing in `Incremental`**; the corpus and the random edits pass + over a source on the first run. What the tests found was in the fixtures: a rival that read the owned + store for the stored fallback of a filtered formula passed until a cell was stored where the `WHERE` + fails, and a rival that forgot a group's own name as an item passed until a cell was stored under + one. +- **The two-second limit decided the shape of the tests** more than the design did: the corpus + differential over a scanning source is 2.5s untrimmed, so it follows `corpus_eager`'s trimming, and + each random test is its own binary. diff --git a/context/code/source.md b/context/code/source.md index 7ed4d1d1..a7e81176 100644 --- a/context/code/source.md +++ b/context/code/source.md @@ -14,6 +14,8 @@ they leave a choice. | `summarize(select)` | the summary of the cells a `Select` names: the aggregate, not the rows | | `keys(partial)` | the keys of the cells containing `partial`: stems, and the records behind an aggregate | | `cells(partial)` | the same with values: records mode and a drill | +| `count(select, excluding)` | how many cells a select names, leaving out those holding a hidden pair: occupancy | +| `distinct_without(cats)` | the cell keys with `cats` removed, each once, first seen first: the stems formulas are evaluated at | 1. **Aggregation is the interface.** ADR 003 clause 1: `GridLayout` asks for a value, not for the rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk, @@ -96,3 +98,17 @@ now plus the changes drained later counts no change twice or not at all. Why, an uses it: `context/code/improvise-incremental.md`, step 7. `DataStore` is not a `ChangeSource`: its log is `&mut` and is drained by the owner in the step that made the edit. A `SharedFeed` is the `Arc` a backend and a model share. + +## Pushdown needs two more queries + +`count` and `distinct_without` joined the trait for `Pushdown` (improvise-rnak). Both are what a query +engine answers without sending rows (`COUNT(*) ... WHERE`, `SELECT DISTINCT`), and each replaces a +read that would have transferred every key: `Eager::occupancy` lists all keys, and +`recompute_formulas` walked every cell to find its stems. `recompute_formulas` now asks +`distinct_without(target and None categories)` once per formula, and the order it returns is +the order the cells were first seen in, so the stems are inserted in the order they always were +(the passes of a fixed point visit stems in that order). A source with no order of its own must +fix one, or the passes of a document whose stems read each other may differ between refreshes. + +`Select::admits` is the predicate of a select over one key, for a source that must test keys itself +(`DataStore::count`). `TableSource` does not use it: its own `holds` is the independent second reading. diff --git a/context/test-evidence/core.md b/context/test-evidence/core.md index 69226c41..bd52dbcc 100644 --- a/context/test-evidence/core.md +++ b/context/test-evidence/core.md @@ -2380,3 +2380,24 @@ branch reads the owned store; it fails here. `set_cell` on a model over a source panics in a debug build, naming the reason. Without the assert the write goes to the owned store and nothing reads it. + +### model::source::tests::the_store_and_a_scanned_table_count_every_select_alike + +improvise-rnak. `count(select, excluding)` on the store and on a scanning table, over every select of +the fixture, at both widths, hiding none, each of four items and each pair of them: 31,680 counts, 3,370 +of them not zero (a count that was zero on both sides would pass). Rivals, each failing it: the count +ignores the excluded pairs (also fails `hiding_an_item_removes_exactly_the_cells_that_hold_it`), ignores the +spans, and ignores `exact`. + +### model::source::tests::hiding_an_item_removes_exactly_the_cells_that_hold_it + +The hand-checked half: all rows, all but those holding `R=E`, and a select naming a hidden item is +zero. A count that agreed with a wrong scan on both sides would pass the test above and fail this. + +### model::source::tests::the_store_and_a_scanned_table_list_the_same_stems + +`distinct_without` on the store and on a scanning table, for every subset of three categories and one +that does not exist, equal and in the same order, each stem once. Rivals, each failing it by name: +the stems not deduplicated, sorted instead of first seen (this is the only test that sees it: the +evaluator's results do not depend on stem order for the fixtures), and the dropped categories kept (also +fails several evaluator tests). diff --git a/context/test-evidence/improvise-incremental.md b/context/test-evidence/improvise-incremental.md index 61011e72..8578822e 100644 --- a/context/test-evidence/improvise-incremental.md +++ b/context/test-evidence/improvise-incremental.md @@ -842,3 +842,60 @@ the model not over the feed fails it with the others. `Fed` against `Eager` over random formula and cell edits on the formula fixture (no views, so the binary stays under two seconds). Rivals: `sync` drains nothing, the model not over the feed, a shape built from an empty store and the stems from the owned store all fail it. + +### a_pushdown_matches_eager_over_the_shipped_documents + +improvise-rnak. `Pushdown` over `Remote` (a scanning table that counts its queries; the model keeps no +cells) against `Eager`, over the shipped documents, their views and `edits_for` (trimmed as +`corpus_eager` is). Rivals, each failing it by name: the occupancy forgetting the hidden items; the +model not over the source; a raw fold answering for a formula target; a covered read covering every +member; a total covering every member; the engine ignoring `exact`; a refresh that does nothing; a +definition edit that does not refresh. + +### a_pushdown_matches_eager_on_hidden_items_and_collapsed_groups_under_every_aggregation + +The hidden and the counted fixtures, drawn, under each of five aggregations, with a NaN, a `-0.0` and +a text written to a hidden cell and a cell stored under the group's own name. The last is why the +rival "occupancy forgets the group's own name" is caught: before that edit, it passed everywhere. +Rivals: the occupancy treating a group as an item; the same forgetting the group's name; the occupancy +forgetting the hidden items; covered and total covering everything. + +### a_pushdown_matches_eager_on_formulas_groups_and_wide_keys + +The ratio fixture (a formula over two measures), the hand model under every aggregation of two +measures, a grouped model and the six-category wide model. Rivals: a refresh that does nothing, a +definition edit that does not refresh, a raw fold of a formula target, the engine ignoring `exact`. + +### a_fold_is_one_query_and_sends_no_row + +A raw fold, a collapsed group's covered read and a column total are three `summarize` calls between +them and no row sent back by the table (`rows_sent` unchanged); an exact read is one lookup. Rival: +the model not over the source (the reads then never reach the table). + +### occupancy_is_one_count_whatever_the_groups_and_hidden_items + +The Coastal-by-A occupancy of the hidden fixture (East hidden, West holding A) is 1, in one `count`, +with no row sent. Rivals: the hidden items forgotten, a group treated as an item. + +### formula_cells_are_computed_from_the_distinct_stems_not_from_every_cell + +Refreshing the hand model's one formula asks for stems once and gets two rows back (Product A and B) +from twelve cells, lists no keys and no cells, and still queries the folds the formula reads. +Rivals: a refresh that does nothing, the model not over the source. + +### a_stored_cell_is_current_and_a_formula_cell_waits_for_a_refresh + +The staleness contract of `Pushdown`. A write by another client is seen by the next exact read (50) and +not by the cached formula cell (9) until `refresh` (49). Rival: a refresh that does nothing. + +### a_source_that_forgets_the_hidden_items_is_caught_as_a_divergence + +The check of the checker, the user's ruling about faults: `IgnoresHidden` is a source whose `count` +drops the hidden pairs. The differential must report a `Divergence` and the probe must be an +occupancy; a sound source over the same fixture must pass. + +### a_pushdown_matches_eager_over_random_formula_and_cell_edits + +`Pushdown` over `Remote` against `Eager` over random formula and cell edits on the formula fixture +(cycles, text, NaN and `-0.0` among the values), in its own binary. Rival: a refresh that does +nothing. diff --git a/crates/improvise-core/src/model/source.rs b/crates/improvise-core/src/model/source.rs index 5c3fd8a0..661ecafc 100644 --- a/crates/improvise-core/src/model/source.rs +++ b/crates/improvise-core/src/model/source.rs @@ -24,6 +24,16 @@ impl<'a> Select<'a> { } } + /// Whether the cell at `key` is one this select names. + pub fn admits(&self, key: &CellKey) -> bool { + let spanned = |(cat, items): &(&str, Vec<&str>)| { + key.get(cat).is_some_and(|held| items.contains(&held)) + }; + key.matches_partial(self.base) + && self.spans.iter().all(spanned) + && (!self.exact || key.0.len() == self.base.len() + self.spans.len()) + } + /// Every cell containing `base` and one listed item of each span. pub fn spanning( base: &'a [(String, String)], @@ -200,6 +210,20 @@ pub trait CellSource { &'a self, partial: &'a [(String, String)], ) -> Box + 'a>; + + /// How many cells `select` names, leaving out those holding any of the `excluding` pairs. + fn count(&self, select: &Select, excluding: &[(String, String)]) -> usize; + + /// The key of every cell with the categories in `without` removed, each once, in the order + /// first seen: the stems a formula is evaluated at, as `SELECT DISTINCT` gives them. + fn distinct_without(&self, without: &[String]) -> Vec; +} + +/// Whether the cell at `key` holds any of the `pairs`. +fn holds_any(key: &CellKey, pairs: &[(String, String)]) -> bool { + pairs + .iter() + .any(|(cat, item)| key.get(cat) == Some(item.as_str())) } impl CellSource for DataStore { @@ -236,6 +260,19 @@ impl CellSource for DataStore { ), } } + + fn count(&self, select: &Select, excluding: &[(String, String)]) -> usize { + let named = self.keys(select.base).filter(|key| select.admits(key)); + named.filter(|key| !holds_any(key, excluding)).count() + } + + fn distinct_without(&self, without: &[String]) -> Vec { + let mut seen = std::collections::HashSet::new(); + let stems = self + .iter_cells() + .map(|(key, _)| without.iter().fold(key, |k, cat| k.without(cat))); + stems.filter(|stem| seen.insert(stem.clone())).collect() + } } /// A table of rows read by scanning, the way a query engine with no index would. A second @@ -291,6 +328,28 @@ impl CellSource for TableSource { .cloned(), ) } + + fn count(&self, select: &Select, excluding: &[(String, String)]) -> usize { + let has = |key: &CellKey, (cat, item): &(String, String)| { + key.0.iter().any(|(c, i)| c == cat && i == item) + }; + let named = self.rows.iter().filter(|(key, _)| Self::holds(key, select)); + named + .filter(|(key, _)| !excluding.iter().any(|pair| has(key, pair))) + .count() + } + + fn distinct_without(&self, without: &[String]) -> Vec { + let mut stems: Vec = Vec::new(); + for (key, _) in &self.rows { + let kept = key.0.iter().filter(|(c, _)| !without.contains(c)); + let stem = CellKey::new(kept.cloned().collect()); + if !stems.contains(&stem) { + stems.push(stem); + } + } + stems + } } #[cfg(test)] @@ -569,4 +628,84 @@ mod tests { "no cell is keyed by R=E alone" ); } + + /// Every way to hide items of the fixture: none, each item of each category, and two at once. + fn hidden_sets() -> Vec> { + let items = [("R", "E"), ("R", "W"), ("P", "A"), ("_Measure", "Note")]; + let mut sets = vec![vec![]]; + for (i, a) in items.iter().enumerate() { + sets.push(vec![pair(a.0, a.1)]); + for b in &items[i + 1..] { + sets.push(vec![pair(a.0, a.1), pair(b.0, b.1)]); + } + } + sets + } + + #[test] + fn the_store_and_a_scanned_table_count_every_select_alike() { + let rows = rows(); + let (store, table) = (store_of(&rows), TableSource { rows: rows.clone() }); + let (mut asked, mut nonzero) = (0, 0); + for (base, spans) in selects(&rows) { + let spans: Vec<(&str, Vec<&str>)> = spans + .iter() + .map(|(c, i)| (c.as_str(), i.iter().map(String::as_str).collect())) + .collect(); + for exact in [false, true] { + let select = Select::spanning(&base, &spans, exact); + for hidden in hidden_sets() { + let (a, b) = (store.count(&select, &hidden), table.count(&select, &hidden)); + assert_eq!(a, b, "{base:?} {spans:?} exact={exact} hiding {hidden:?}"); + asked += 1; + nonzero += usize::from(a > 0); + } + } + } + assert!(asked > 20_000, "the enumeration shrank to {asked}"); + assert!(nonzero > 2_000, "{nonzero} of {asked} counts were not zero"); + } + + #[test] + fn hiding_an_item_removes_exactly_the_cells_that_hold_it() { + let rows = rows(); + let store = store_of(&rows); + let none = Select::matching(&[]); + assert_eq!(store.count(&none, &[]), rows.len()); + let east = [pair("R", "E")]; + let held = rows.iter().filter(|(k, _)| k.get("R") == Some("E")).count(); + assert_eq!(store.count(&none, &east), rows.len() - held); + assert_eq!( + store.count(&Select::matching(&east), &east), + 0, + "a named hidden item matches nothing" + ); + } + + #[test] + fn the_store_and_a_scanned_table_list_the_same_stems() { + let rows = rows(); + let (store, table) = (store_of(&rows), TableSource { rows: rows.clone() }); + let cats = ["R", "P", "_Measure", "Nowhere"].map(String::from); + let mut asked = 0; + for mask in 0..(1u32 << cats.len()) { + let without: Vec = cats + .iter() + .enumerate() + .filter(|(i, _)| mask >> i & 1 == 1) + .map(|(_, c)| c.clone()) + .collect(); + let (a, b) = ( + store.distinct_without(&without), + table.distinct_without(&without), + ); + assert_eq!(a, b, "without {without:?}, in first-seen order"); + let mut sorted = a.clone(); + sorted.sort(); + sorted.dedup(); + assert_eq!(sorted.len(), a.len(), "each stem once"); + asked += a.len(); + } + assert!(asked > 40, "only {asked} stems were compared"); + } } diff --git a/crates/improvise-core/src/model/types.rs b/crates/improvise-core/src/model/types.rs index d9c161e8..4b128590 100644 --- a/crates/improvise-core/src/model/types.rs +++ b/crates/improvise-core/src/model/types.rs @@ -1027,15 +1027,9 @@ impl Model { // (improvise-6os replaced an O(n^2) Vec::contains). let mut stems: IndexSet = IndexSet::new(); for (target_cat, _) in &formula_cats { - for key in self.source().keys(&[]) { - let stem = key.without(target_cat); - // Strip none_cats too: the grid queries without them. - let mut stripped = stem; - for nc in none_cats { - stripped = stripped.without(nc); - } - stems.insert(stripped); - } + // The target and the none_cats go: the grid queries without them. + let without = [std::slice::from_ref(target_cat), none_cats].concat(); + stems.extend(self.source().distinct_without(&without)); } // Fixed-point iteration. diff --git a/crates/improvise-incremental/Cargo.toml b/crates/improvise-incremental/Cargo.toml index 20d49e06..f022965c 100644 --- a/crates/improvise-incremental/Cargo.toml +++ b/crates/improvise-incremental/Cargo.toml @@ -13,5 +13,7 @@ numtower = { path = "../numtower" } zset = { path = "../zset" } [dev-dependencies] +# `TableSource`, the scanning source the pushdown tests are built on. +improvise-core = { path = "../improvise-core", features = ["test-support"] } improvise-io = { path = "../improvise-io" } proptest = "1" diff --git a/crates/improvise-incremental/src/lib.rs b/crates/improvise-incremental/src/lib.rs index 8fddf659..ae6b9d70 100644 --- a/crates/improvise-incremental/src/lib.rs +++ b/crates/improvise-incremental/src/lib.rs @@ -9,6 +9,7 @@ mod formulas; mod group; mod harness; mod incremental; +mod pushdown; mod reading; mod shapes; @@ -20,6 +21,7 @@ pub use harness::{ drawn_probes, edit_probes, probes, same_value, }; pub use incremental::{Evaluation, Incremental}; +pub use pushdown::Pushdown; pub use reading::Reading; pub use shapes::Meter; diff --git a/crates/improvise-incremental/src/pushdown.rs b/crates/improvise-incremental/src/pushdown.rs new file mode 100644 index 00000000..90ef6f67 --- /dev/null +++ b/crates/improvise-incremental/src/pushdown.rs @@ -0,0 +1,104 @@ +//! Derived values over a source that answers aggregate queries and reports no changes. +//! Reasoning: `context/code/improvise-incremental.md`, step 7. + +use crate::{Derived, Edit, Reading}; +use improvise_core::model::cell::{CellKey, CellValue}; +use improvise_core::model::source::{CellSource, Select}; +use improvise_core::model::{Aggregation, Model}; +use std::sync::Arc; + +/// A model whose cells are in `source`: a fold is a query, the formula cells are computed in RAM +/// by the model's own evaluator from those folds, and nothing is kept per shape or per change. +pub struct Pushdown { + model: Model, + none_cats: Vec, +} + +impl Pushdown { + /// `definition` supplies the categories, formulas and aggregations; its own store is not read. + pub fn new( + definition: Model, + source: Arc, + none_cats: Vec, + ) -> Self { + let mut backend = Pushdown { + model: definition.over(source), + none_cats, + }; + backend.refresh(); + backend + } + + pub fn model(&self) -> &Model { + &self.model + } + + /// Computes the formula cells again from the source as it now stands. A read of a stored + /// cell or a fold is a query and always current; the formula cache is the one thing held. + pub fn refresh(&mut self) { + self.model.recompute_formulas(&self.none_cats); + } + + /// A change to the definition: a formula or an aggregation. Cells belong to the source. + pub fn define(&mut self, edit: &Edit) { + edit.apply(&mut self.model); + self.refresh(); + } + + /// The items a stored cell may hold in `cat` to be named by `item`, when `item` names a group. + fn members(&self, cat: &str, item: &str) -> Vec { + let group = self + .model + .category(cat) + .filter(|c| !c.items.contains_key(item)); + let members = group.map(|c| c.items_in_group(item)).unwrap_or_default(); + match members.is_empty() { + true => Vec::new(), + false => std::iter::once(item) + .chain(members) + .map(String::from) + .collect(), + } + } +} + +impl Derived for Pushdown { + fn value(&self, key: &CellKey, agg: Aggregation) -> Option { + self.model.evaluate_aggregated(key, agg) + } + + fn raw_fold(&self, partial: &CellKey) -> Option { + match self.model.formula_for(partial) { + Some(_) => None, + None => self.value(partial, Aggregation::OverHidden), + } + } + + fn covered(&self, key: &CellKey, reading: &Reading) -> Option { + let covers = |cat: &str, item: &str| reading.covers(cat, item); + self.model.evaluate_covering(key, reading.agg, &covers) + } + + fn total(&self, key: &CellKey, over: &[String], reading: &Reading) -> Option { + let covers = |cat: &str, item: &str| reading.covers(cat, item); + self.model + .evaluate_spanning(key, over, reading.agg, &covers) + } + + fn occupancy(&self, partial: &CellKey, reading: &Reading) -> usize { + let (mut base, mut held) = (Vec::new(), Vec::new()); + for (cat, item) in &partial.0 { + match self.members(cat, item) { + members if members.is_empty() => base.push((cat.clone(), item.clone())), + members => held.push((cat.clone(), members)), + } + } + let spans: Vec<(&str, Vec<&str>)> = held + .iter() + .map(|(cat, items)| (cat.as_str(), items.iter().map(String::as_str).collect())) + .collect(); + let hidden: Vec<(String, String)> = reading.hidden().cloned().collect(); + let select = Select::spanning(&base, &spans, false); + self.model.source().count(&select, &hidden) + } +} diff --git a/crates/improvise-incremental/tests/pushdown.rs b/crates/improvise-incremental/tests/pushdown.rs new file mode 100644 index 00000000..2b4eaea0 --- /dev/null +++ b/crates/improvise-incremental/tests/pushdown.rs @@ -0,0 +1,197 @@ +//! `Pushdown` over a remote table against `Eager`: values, hidden items, groups, formulas. +//! Without a real SQL engine only the contract is proven: `context/code/improvise-incremental.md`. + +mod support; + +use improvise_core::formula::ast::AggFunc; +use improvise_core::model::cell::{CellKey, CellValue}; +use improvise_core::model::{Aggregation, Model}; +use improvise_core::view::View; +use improvise_incremental::{Derived, Edit, Probe, Reading, differential_in}; +use support::remote::{IgnoresHidden, Pushed}; +use support::{ + corpus, edits_for, grouped_model, hand_model, hidden_counted_workbook, hidden_workbook, key, + none_of, ratio_workbook, wide_workbook, +}; + +fn run(model: &Model, views: &[View], none: &[String], edits: &[Edit]) -> usize { + let candidate = Pushed::new(model, none.to_vec()); + match differential_in(model, views, none, candidate, edits) { + Ok(report) => report.probes, + Err(d) => panic!("{d}"), + } +} + +fn agg_edits(measure: &str) -> Vec { + let all = [ + AggFunc::Count, + AggFunc::Min, + AggFunc::Max, + AggFunc::Avg, + AggFunc::Sum, + ]; + all.into_iter() + .map(|func| Edit::SetMeasureAgg { + measure: measure.into(), + func, + }) + .collect() +} + +#[test] +fn a_pushdown_matches_eager_over_the_shipped_documents() { + let (mut compared, mut files_done) = (0, Vec::::new()); + for (name, model, none, views) in corpus() { + let file = name.split(':').next().unwrap().to_string(); + if files_done.contains(&file) { + continue; + } + files_done.push(file); + let mut edits = edits_for(&model); + if model.data.iter_cells().count() > 40 { + let picked = [0, 2, 4].iter().filter_map(|&i| edits.get(i).cloned()); + edits = picked.collect(); + } + compared += run(&model, &views, &none, &edits); + } + assert!(compared > 10_000, "only {compared} answers were compared"); +} + +#[test] +fn a_pushdown_matches_eager_on_hidden_items_and_collapsed_groups_under_every_aggregation() { + for wb in [hidden_workbook(), hidden_counted_workbook()] { + let views: Vec = wb.views.values().cloned().collect(); + let none = none_of(&wb); + let mut edits = edits_for(&wb.model); + edits.extend(agg_edits("Revenue")); + let hidden = key("East", "A", "Revenue"); + edits.push(Edit::SetCell(hidden.clone(), CellValue::Number(f64::NAN))); + edits.push(Edit::SetCell(hidden.clone(), CellValue::Number(-0.0))); + edits.push(Edit::SetCell(hidden, CellValue::Text("t".into()))); + // A cell stored under the group's own name is a cell its fold and its count both see. + let named = key("Coastal", "B", "Revenue"); + edits.push(Edit::SetCell(named, CellValue::Number(4.0))); + assert!(run(&wb.model, &views, &none, &edits) > 1_000); + } +} + +#[test] +fn a_pushdown_matches_eager_on_formulas_groups_and_wide_keys() { + let ratio = ratio_workbook(); + let views: Vec = ratio.views.values().cloned().collect(); + let mut edits = edits_for(&ratio.model); + edits.extend(agg_edits("Revenue")); + run(&ratio.model, &views, &none_of(&ratio), &edits); + let hand = hand_model(); + let mut edits = agg_edits("Revenue"); + edits.extend(agg_edits("Note")); + run(&hand, &[], &["Region".to_string()], &edits); + run(&grouped_model(), &[], &[], &edits_for(&grouped_model())); + let wide = wide_workbook(); + let views: Vec = wide.views.values().cloned().collect(); + run( + &wide.model, + &views, + &none_of(&wide), + &edits_for(&wide.model), + ); +} + +#[test] +fn a_fold_is_one_query_and_sends_no_row() { + let wb = hidden_counted_workbook(); + let none = none_of(&wb); + let pushed = Pushed::new(&wb.model, none); + let at = |name: &str| key("Inland", name, "Revenue"); + let before = pushed.remote.asked(); + pushed.pd.raw_fold(&CellKey::new(vec![ + ("Product".into(), "A".into()), + ("_Measure".into(), "Revenue".into()), + ])); + let reading = Reading::of_view(wb.active_view(), &none_of(&wb)); + let coastal = key("Coastal", "A", "Revenue"); + pushed.pd.covered(&coastal, &reading); + let total = key("Inland", "A", "Revenue").without("Region"); + pushed.pd.total(&total, &["Region".to_string()], &reading); + let spent = pushed.remote.asked(); + assert_eq!(spent.summaries - before.summaries, 3, "one query per fold"); + assert_eq!(spent.rows_sent, before.rows_sent, "and no row sent back"); + let one = pushed.pd.value(&at("A"), Aggregation::Exact); + assert_eq!(one, Some(CellValue::Number(2.0))); + assert_eq!(pushed.remote.asked().lookups - spent.lookups, 1); +} + +#[test] +fn occupancy_is_one_count_whatever_the_groups_and_hidden_items() { + let wb = hidden_workbook(); + let pushed = Pushed::new(&wb.model, none_of(&wb)); + let reading = Reading::of_view(wb.active_view(), &none_of(&wb)); + let before = pushed.remote.asked(); + let coastal = CellKey::new(vec![ + ("Region".into(), "Coastal".into()), + ("Product".into(), "A".into()), + ]); + let seen = pushed.pd.occupancy(&coastal, &reading); + let spent = pushed.remote.asked(); + assert_eq!(seen, 1, "East is hidden, West holds A"); + assert_eq!(spent.counts - before.counts, 1); + assert_eq!(spent.rows_sent, before.rows_sent); +} + +#[test] +fn formula_cells_are_computed_from_the_distinct_stems_not_from_every_cell() { + let model = hand_model(); + let cells = model.data.iter_cells().count() as u64; + let pushed = Pushed::new(&model, vec!["Region".to_string()]); + let asked = pushed.remote.asked(); + assert_eq!(asked.stem_lists, 1, "one list for the one formula"); + assert_eq!( + asked.rows_sent, 2, + "Product A and Product B, of {cells} cells" + ); + assert_eq!((asked.key_lists, asked.cell_lists), (0, 0)); + assert!( + asked.summaries > 0, + "and the folds the formula reads are queries" + ); +} + +#[test] +fn a_stored_cell_is_current_and_a_formula_cell_waits_for_a_refresh() { + let mut pushed = Pushed::new(&hand_model(), vec!["Region".to_string()]); + let revenue = key("East", "A", "Revenue"); + let profit = CellKey::new(vec![ + ("Product".into(), "A".into()), + ("_Measure".into(), "Profit".into()), + ]); + let fold = |p: &Pushed| p.pd.value(&profit, Aggregation::OverHidden); + assert_eq!(fold(&pushed), Some(CellValue::Number(9.0))); + pushed + .remote + .write(revenue.clone(), Some(CellValue::Number(50.0))); + let now = pushed.pd.value(&revenue, Aggregation::Exact); + assert_eq!(now, Some(CellValue::Number(50.0)), "a read is a query"); + assert_eq!( + fold(&pushed), + Some(CellValue::Number(9.0)), + "the cache is stale" + ); + pushed.pd.refresh(); + assert_eq!(fold(&pushed), Some(CellValue::Number(49.0))); +} + +#[test] +fn a_source_that_forgets_the_hidden_items_is_caught_as_a_divergence() { + let wb = hidden_workbook(); + let views: Vec = wb.views.values().cloned().collect(); + let none = none_of(&wb); + let faulty = Pushed::through(&wb.model, none.clone(), IgnoresHidden); + let ran = differential_in(&wb.model, &views, &none, faulty, &[]); + let divergence = ran.expect_err("a wrong hidden filter must not pass"); + assert!( + matches!(*divergence.probe, Probe::Occupancy(..)), + "{divergence}" + ); + let sound = Pushed::new(&wb.model, none.clone()); + assert!(differential_in(&wb.model, &views, &none, sound, &[]).is_ok()); +} diff --git a/crates/improvise-incremental/tests/random_pushdown.rs b/crates/improvise-incremental/tests/random_pushdown.rs new file mode 100644 index 00000000..a9245f9d --- /dev/null +++ b/crates/improvise-incremental/tests/random_pushdown.rs @@ -0,0 +1,22 @@ +//! `Pushdown` over a remote table against `Eager` over random formula and cell edits (NaN, +//! `-0.0`, cycles and text among them). Evidence: `context/test-evidence/improvise-incremental.md`. + +mod support; + +use improvise_incremental::differential_in; +use proptest::prelude::*; +use support::remote::Pushed; +use support::{any_formula_edit, formula_workbook}; + +proptest! { + #[test] + fn a_pushdown_matches_eager_over_random_formula_and_cell_edits( + edits in prop::collection::vec(any_formula_edit(), 0..3), + ) { + let wb = formula_workbook(); + let none = vec!["Region".to_string()]; + let candidate = Pushed::new(&wb.model, none.clone()); + let ran = differential_in(&wb.model, &[], &none, candidate, &edits); + prop_assert!(ran.is_ok(), "{}", ran.unwrap_err()); + } +} diff --git a/crates/improvise-incremental/tests/support/live.rs b/crates/improvise-incremental/tests/support/live.rs index 8d6d2259..d88ebf60 100644 --- a/crates/improvise-incremental/tests/support/live.rs +++ b/crates/improvise-incremental/tests/support/live.rs @@ -88,6 +88,14 @@ impl CellSource for LiveTable { } Box::new(cells.into_iter()) } + + fn count(&self, select: &Select, excluding: &[(String, String)]) -> usize { + self.state.lock().unwrap().visible.count(select, excluding) + } + + fn distinct_without(&self, without: &[String]) -> Vec { + self.state.lock().unwrap().visible.distinct_without(without) + } } impl ChangeSource for LiveTable { diff --git a/crates/improvise-incremental/tests/support/mod.rs b/crates/improvise-incremental/tests/support/mod.rs index db732221..7a396c42 100644 --- a/crates/improvise-incremental/tests/support/mod.rs +++ b/crates/improvise-incremental/tests/support/mod.rs @@ -2,6 +2,7 @@ #![allow(dead_code)] pub mod live; +pub mod remote; use improvise_core::formula::ast::AggFunc; use improvise_core::formula::parse_formula; diff --git a/crates/improvise-incremental/tests/support/remote.rs b/crates/improvise-incremental/tests/support/remote.rs new file mode 100644 index 00000000..adce47c5 --- /dev/null +++ b/crates/improvise-incremental/tests/support/remote.rs @@ -0,0 +1,226 @@ +//! A remote table: an in-memory stand-in for a query engine, read by scanning and counted. +//! It is not `Eager` and shares no evaluator with it; it answers queries over rows. +#![allow(dead_code)] + +use improvise_core::model::cell::{CellKey, CellValue}; +use improvise_core::model::source::{CellSource, Select, Summary, TableSource}; +use improvise_core::model::{Aggregation, Model}; +use improvise_incremental::{Derived, Edit, Maintained, Pushdown, Reading}; +use std::sync::{Arc, Mutex}; + +/// What the engine was asked, and how many rows it sent back. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct Asked { + pub lookups: u64, + pub summaries: u64, + pub counts: u64, + pub stem_lists: u64, + pub key_lists: u64, + pub cell_lists: u64, + /// Keys and cells sent back by `keys` and `cells`: rows, not aggregates. + pub rows_sent: u64, +} + +#[derive(Default)] +pub struct Remote { + table: Mutex, + asked: Mutex, +} + +impl Remote { + pub fn of(rows: impl IntoIterator) -> Self { + let table = TableSource { + rows: rows.into_iter().collect(), + }; + Remote { + table: Mutex::new(table), + asked: Mutex::default(), + } + } + + /// Writes as another client of the database would: in place, or at the end, or removed. + pub fn write(&self, key: CellKey, value: Option) { + let mut table = self.table.lock().unwrap(); + let at = table.rows.iter().position(|(k, _)| *k == key); + match (at, value) { + (Some(i), Some(v)) => table.rows[i].1 = v, + (None, Some(v)) => table.rows.push((key, v)), + (Some(i), None) => { + table.rows.remove(i); + } + (None, None) => {} + } + } + + pub fn asked(&self) -> Asked { + *self.asked.lock().unwrap() + } + + fn ask(&self, f: impl FnOnce(&mut Asked)) { + f(&mut self.asked.lock().unwrap()); + } + + fn table(&self) -> std::sync::MutexGuard<'_, TableSource> { + self.table.lock().unwrap() + } +} + +impl CellSource for Remote { + fn value_at(&self, key: &CellKey) -> Option { + self.ask(|a| a.lookups += 1); + self.table().value_at(key) + } + + fn summarize(&self, select: &Select) -> Summary { + self.ask(|a| a.summaries += 1); + self.table().summarize(select) + } + + fn keys<'a>( + &'a self, + partial: &'a [(String, String)], + ) -> Box + 'a> { + let keys: Vec = self.table().keys(partial).collect(); + self.ask(|a| { + a.key_lists += 1; + a.rows_sent += keys.len() as u64; + }); + Box::new(keys.into_iter()) + } + + fn cells<'a>( + &'a self, + partial: &'a [(String, String)], + ) -> Box + 'a> { + let cells: Vec<_> = self.table().cells(partial).collect(); + self.ask(|a| { + a.cell_lists += 1; + a.rows_sent += cells.len() as u64; + }); + Box::new(cells.into_iter()) + } + + fn count(&self, select: &Select, excluding: &[(String, String)]) -> usize { + self.ask(|a| a.counts += 1); + self.table().count(select, excluding) + } + + fn distinct_without(&self, without: &[String]) -> Vec { + let stems = self.table().distinct_without(without); + self.ask(|a| { + a.stem_lists += 1; + a.rows_sent += stems.len() as u64; + }); + stems + } +} + +/// A source with a fault, for checking that a differential can see it: it forgets the hidden +/// items a count leaves out. +pub struct IgnoresHidden(pub Arc); + +impl CellSource for IgnoresHidden { + fn value_at(&self, key: &CellKey) -> Option { + self.0.value_at(key) + } + + fn summarize(&self, select: &Select) -> Summary { + self.0.summarize(select) + } + + fn keys<'a>( + &'a self, + partial: &'a [(String, String)], + ) -> Box + 'a> { + self.0.keys(partial) + } + + fn cells<'a>( + &'a self, + partial: &'a [(String, String)], + ) -> Box + 'a> { + self.0.cells(partial) + } + + fn count(&self, select: &Select, _excluding: &[(String, String)]) -> usize { + self.0.count(select, &[]) + } + + fn distinct_without(&self, without: &[String]) -> Vec { + self.0.distinct_without(without) + } +} + +/// A `Pushdown` over a remote table that holds the model's cells; the model keeps no store. +/// A cell edit is a write to the table, as another client's would be, then a refresh. +pub struct Pushed { + pub pd: Pushdown, + pub remote: Arc, +} + +impl Pushed { + pub fn new(model: &Model, none_cats: Vec) -> Pushed { + let remote = Arc::new(Remote::of(rows_of(model))); + let pd = Pushdown::new(bare(model), remote.clone(), none_cats); + Pushed { pd, remote } + } + + /// The same, reading the table through `wrap`. + pub fn through( + model: &Model, + none_cats: Vec, + wrap: impl FnOnce(Arc) -> Q, + ) -> Pushed { + let remote = Arc::new(Remote::of(rows_of(model))); + let pd = Pushdown::new(bare(model), Arc::new(wrap(remote.clone())), none_cats); + Pushed { pd, remote } + } +} + +fn rows_of(model: &Model) -> Vec<(CellKey, CellValue)> { + model + .data + .iter_cells() + .map(|(k, v)| (k, v.clone())) + .collect() +} + +/// `model` with no cells of its own. +fn bare(model: &Model) -> Model { + let mut bare = model.clone(); + bare.data = improvise_core::model::cell::DataStore::new(); + bare +} + +impl Derived for Pushed { + fn value(&self, key: &CellKey, agg: Aggregation) -> Option { + self.pd.value(key, agg) + } + + fn raw_fold(&self, partial: &CellKey) -> Option { + self.pd.raw_fold(partial) + } + + fn covered(&self, key: &CellKey, reading: &Reading) -> Option { + self.pd.covered(key, reading) + } + + fn total(&self, key: &CellKey, over: &[String], reading: &Reading) -> Option { + self.pd.total(key, over, reading) + } + + fn occupancy(&self, partial: &CellKey, reading: &Reading) -> usize { + self.pd.occupancy(partial, reading) + } +} + +impl Maintained for Pushed { + fn apply(&mut self, edit: &Edit) { + match edit { + Edit::SetCell(key, value) => self.remote.write(key.clone(), Some(value.clone())), + Edit::ClearCell(key) => self.remote.write(key.clone(), None), + _ => return self.pd.define(edit), + } + self.pd.refresh(); + } +}