refactor(core): a read-only CellSource trait, DataStore its first implementation (improvise-qddo)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

context/code/source.md

@@ -0,0 +1,72 @@
+# `CellSource`: where cell values are read from
+
+`crates/improvise-core/src/model/source.rs`. ADR 003 asked for a source trait in core with the
+in-RAM `DataStore` as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for
+the two live sources that need it. This file is the reasoning; the evidence for each test is in
+`context/test-evidence/core.md`. The ADRs are not amended: what follows are the readings taken where
+they leave a choice.
+
+## What is in the trait, and why
+
+| method | what the evaluator asks |
+|---|---|
+| `value_at(key)` | the value stored at exactly `key` (an exact read, the filter-mismatch fallback) |
+| `summarize(select)` | the summary of the cells a `Select` names: the aggregate, not the rows |
+| `keys(partial)` | the keys of the cells containing `partial`: stems, and the records behind an aggregate |
+| `cells(partial)` | the same with values: records mode and a drill |
+
+1. **Aggregation is the interface.** ADR 003 clause 1: `GridLayout` asks for a value, not for the
+   rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk,
+   `summarize`, returns a `Summary` and never a list of values. `keys` and `cells` exist because
+   some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and
+   a SQL source answers them with a `SELECT DISTINCT` or a plain select.
+2. **Writes are not in the trait.** `DataStore::set` and `remove` are inherent methods. A source that
+   cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than
+   enforced". The model's own writes (`set_cell`, `clear_cell`) go to the owned store.
+3. **The trait has no `Result`.** A source that can fail (a connection) decides at its own boundary
+   what a failed read is, and `improvise-io` is where that decision belongs. AGENTS.md keeps
+   `Option` and `Result` at the edges; `Option` is the one the trait uses, for "no such cell".
+4. **`Select` is `spanned_values`' arguments.** A fold names a partial key, the covered members of
+   each group or omitted category it spans, and whether only an exact-width cell counts. The
+   covered members are decided in RAM from the categories and the view's hidden items, so a source
+   is handed explicit item lists and never a predicate. A hidden item therefore reaches a source
+   only as the absence of an item from a list.
+
+## `Summary` and `Numbers`
+
+A `Summary` is what a source reports about the values of a read: a `Numbers` (count, an exact sum,
+the least and greatest non-NaN, in `total_cmp` order so `-0.0` is below `+0.0`) and whether the rest
+(text, errors) agree on one value. It reduces exactly as `aggregate_cell_values` did: numbers by the
+function, anything else by unanimity, else `Error("mixed")`, none at all absent. Three properties
+carry the design:
+
+- **It merges.** A read of a collapsed group, a total, or several contexts of a formula aggregate is
+  the merge of the summaries of its parts, equal to the summary of the union. That is what lets an
+  engine answer a pooled read without sending rows, and what lets `eval_agg` pool `Numbers` instead
+  of concatenating `Vec<f64>`. The sum is `ExactSum`, so a merge is the same bits in any order.
+- **It is order-free.** Unanimity needs the set of non-number values, not their order, so a source
+  that has no insertion order loses nothing by it.
+- **The reference is kept.** `tests::reference` is the old reduction, copied into the test module;
+  `a_summary_reduces_as_the_values_did` compares every sequence of an eight-value pool, up to three
+  long, with `Summary` over all five functions. `characterization_survives_the_move`: the existing
+  evaluator tests (423 in the crate) are unchanged and pass.
+
+What a real engine must supply is stated here because the trait cannot enforce it: a `Numbers` whose
+sum is exact. SQLite's `SUM` is a float sum, so an engine that cannot return an exact accumulator
+either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which
+`context/code/improvise-incremental.md` ("Step 6") calls a user-visible change.
+
+## `Model::source()`
+
+`Model::source()` returns the `&dyn CellSource` that the evaluator, the stems of `recompute_formulas`
+and the records of `GridLayout` read through. Today it is always the owned `DataStore`; the point of
+routing the reads is that PR 3 (improvise-rnak) changes only this accessor. Reads that bypass it
+(`model.data.*` in other crates, `Eager::occupancy`, `Shapes::of`) are the ones later steps move.
+
+## `TableSource`
+
+A scanned table of rows, behind `cfg(any(test, feature = "test-support"))`. It is the second
+implementation, written as a query engine with no index would answer, sharing only `Summary` with
+`DataStore`. `the_store_and_a_scanned_table_answer_every_select_alike` asks both every select over a
+fixture (every subset of every stored key as a base, with no span, one span of each of six item
+lists, and two), at both widths and for all five functions: over 5,000 triples.

context/test-evidence/core.md

@@ -2300,3 +2300,47 @@ key with the whole suite green. It is reached by `reopened_at` (an advance),
 `refused_here` (a commit) and `settle_declined_editor`. Rival: the arm passes
 `category`; the test fails on all three assertions, and the display is 36 characters
 against the key "Region".
+
+### model::source::tests::a_summary_reduces_as_the_values_did
+
+improvise-qddo. The reduction moved from `aggregate_cell_values` to `Summary::reduce`; the test keeps
+the old function as `reference` and compares all 585 value sequences of up to three values from an
+eight-value pool (NaN, both zeros, infinity, two texts, an error) on all five functions. Rivals, each
+failing it by name: unanimity ignored (`Others::One` kept for any second value), a number beside a
+unanimous text read as unanimous, and `<` for the order of the zeros (the sign of a zero extreme is
+lost). The first two also fail `aggregating_differing_text_values_is_an_explicit_error` and
+`aggregating_text_among_numbers_is_an_explicit_error`; the zero rival also fails
+`eager_math_tests::min_and_max_of_both_zeros_do_not_depend_on_their_order`.
+
+### model::source::tests::a_merge_of_summaries_is_the_summary_of_the_union
+
+A read of several parts is a merge. The test merges every ordered pair of sequences of up to two
+values and compares with the reference of the concatenation. Rivals: the least not merged, the
+greatest merged with the other side's least, the zeros unordered. Each fails it.
+
+### model::source::tests::numbers_pool_what_the_formula_aggregate_pools
+
+`eval_agg` pools `Numbers` now, not a concatenated `Vec<f64>`. Every split of a five-number list
+(a NaN and both zeros among them) pooled by merge must equal the whole, by bits for all five
+functions and for the total. Rivals: the least not merged, the greatest seeded from the other
+side's least. Both fail it.
+
+### model::source::tests::the_store_and_a_scanned_table_answer_every_select_alike
+
+`DataStore` and `TableSource` (a scan, sharing only `Summary`) are asked every select over the
+fixture, at both widths. Rivals, each failing it: the store ignores `exact` for a select with no
+span, and the store ignores `exact` for a spanned one. The first also fails
+`an_exact_select_with_no_span_names_one_cell`.
+
+### model::source::tests::the_store_and_a_scanned_table_list_the_same_cells
+
+`keys` and `cells` under every base of the fixture, and `value_at` for every row and an absent key.
+Rival: `DataStore::keys` ignores its partial. It fails here and in five evaluator tests that read
+the member records of an aggregate over an expression
+(`route_then_aggregate_tests::a_routing_formula_at_a_record_is_the_rule_itself` among them).
+
+### model::source::tests::an_exact_select_with_no_span_names_one_cell
+
+A select with `exact` and no span is the cell, not the cells containing the pairs. The store's
+`spanned_values` is never called for it (the evaluator reads an exact key through `value_at`), so
+this is the contract a second source implements. Rival: `exact` ignored when there is no span.

crates/improvise-core/src/model/mod.rs

@@ -2,6 +2,7 @@ pub mod category;
 pub mod cell;
 pub mod id;
 pub mod record;
+pub mod source;
 pub mod symbol;
 pub mod types;
 

crates/improvise-core/src/model/source.rs

@@ -0,0 +1,558 @@
+//! Where cell values are read from: a read-only trait, and the aggregate it answers in.
+//! Reasoning: `context/code/source.md` (ADR 003, ADR 009 clause 2; improvise-qddo).
+
+use super::cell::{CellKey, CellValue, DataStore};
+use crate::formula::AggFunc;
+use numtower::ExactSum;
+
+/// The stored cells one read asks about: those that contain every pair of `base`, and, for each
+/// span, one of its listed items. `exact` admits only a cell with no other coordinate.
+#[derive(Debug, Clone, Copy)]
+pub struct Select<'a> {
+    pub base: &'a [(String, String)],
+    pub spans: &'a [(&'a str, Vec<&'a str>)],
+    pub exact: bool,
+}
+
+impl<'a> Select<'a> {
+    /// Every cell containing `base`.
+    pub fn matching(base: &'a [(String, String)]) -> Self {
+        Select {
+            base,
+            spans: &[],
+            exact: false,
+        }
+    }
+
+    /// Every cell containing `base` and one listed item of each span.
+    pub fn spanning(
+        base: &'a [(String, String)],
+        spans: &'a [(&'a str, Vec<&'a str>)],
+        exact: bool,
+    ) -> Self {
+        Select { base, spans, exact }
+    }
+}
+
+/// What a source reports of the numbers in a read: count, exact sum, extremes. It merges.
+/// Why it is enough: `context/code/source.md`.
+#[derive(Debug, Clone, Default)]
+pub struct Numbers {
+    count: u64,
+    sum: ExactSum,
+    least: Option<f64>,
+    greatest: Option<f64>,
+}
+
+impl Numbers {
+    pub fn of(values: impl IntoIterator<Item = f64>) -> Self {
+        let mut numbers = Numbers::default();
+        values.into_iter().for_each(|x| numbers.add(x));
+        numbers
+    }
+
+    pub fn add(&mut self, x: f64) {
+        self.count += 1;
+        self.sum.add(x, 1);
+        if !x.is_nan() {
+            self.least = Some(self.least.map_or(x, |l| order_min(l, x)));
+            self.greatest = Some(self.greatest.map_or(x, |g| order_max(g, x)));
+        }
+    }
+
+    pub fn merge(&mut self, other: &Numbers) {
+        self.count += other.count;
+        self.sum.absorb(&other.sum);
+        self.least = merged(self.least, other.least, order_min);
+        self.greatest = merged(self.greatest, other.greatest, order_max);
+    }
+
+    pub fn count(&self) -> u64 {
+        self.count
+    }
+
+    /// The sum correctly rounded once, signed as IEEE signs a zero total; `+0.0` over none.
+    pub fn total(&self) -> f64 {
+        self.sum.value()
+    }
+
+    /// `None` over no numbers: an empty read is absent, not zero. `Min` and `Max` skip NaN, and
+    /// are NaN when nothing else is left; zeros order by sign.
+    pub fn reduce(&self, func: &AggFunc) -> Option<f64> {
+        (self.count > 0).then(|| match func {
+            AggFunc::Sum => self.sum.value(),
+            AggFunc::Avg => self.sum.mean(self.count),
+            AggFunc::Min => self.least.unwrap_or(f64::NAN),
+            AggFunc::Max => self.greatest.unwrap_or(f64::NAN),
+            AggFunc::Count => self.count as f64,
+        })
+    }
+}
+
+fn order_min(a: f64, b: f64) -> f64 {
+    if b.total_cmp(&a).is_lt() { b } else { a }
+}
+
+fn order_max(a: f64, b: f64) -> f64 {
+    if b.total_cmp(&a).is_gt() { b } else { a }
+}
+
+fn merged(a: Option<f64>, b: Option<f64>, pick: fn(f64, f64) -> f64) -> Option<f64> {
+    match (a, b) {
+        (Some(x), Some(y)) => Some(pick(x, y)),
+        (x, y) => x.or(y),
+    }
+}
+
+/// The values that are not numbers, as far as unanimity needs them.
+#[derive(Debug, Clone, Default, PartialEq)]
+enum Others {
+    #[default]
+    None,
+    One(CellValue),
+    Mixed,
+}
+
+/// A read of cells as a source reports it: the numbers, and whether the rest agree on one
+/// value. It reduces as raw values do (core.md §2); none at all is absent.
+#[derive(Debug, Clone, Default)]
+pub struct Summary {
+    numbers: Numbers,
+    others: Others,
+}
+
+impl Summary {
+    pub fn of<'a>(values: impl IntoIterator<Item = &'a CellValue>) -> Self {
+        let mut summary = Summary::default();
+        values.into_iter().for_each(|v| summary.add(v));
+        summary
+    }
+
+    pub fn add(&mut self, value: &CellValue) {
+        match value.as_f64() {
+            Some(x) => self.numbers.add(x),
+            None => self.add_other(value),
+        }
+    }
+
+    fn add_other(&mut self, value: &CellValue) {
+        self.others = match std::mem::take(&mut self.others) {
+            Others::None => Others::One(value.clone()),
+            Others::One(held) if held == *value => Others::One(held),
+            _ => Others::Mixed,
+        };
+    }
+
+    pub fn merge(&mut self, other: &Summary) {
+        self.numbers.merge(&other.numbers);
+        match &other.others {
+            Others::None => {}
+            Others::One(v) => self.add_other(v),
+            Others::Mixed => self.others = Others::Mixed,
+        }
+    }
+
+    /// Only the numbers: what a formula-level aggregate pools.
+    pub fn numbers(&self) -> &Numbers {
+        &self.numbers
+    }
+
+    pub fn reduce(&self, func: &AggFunc) -> Option<CellValue> {
+        match (&self.others, self.numbers.count()) {
+            (Others::None, _) => self.numbers.reduce(func).map(CellValue::Number),
+            (Others::One(only), 0) => Some(only.clone()),
+            _ => Some(CellValue::Error("mixed".to_string())),
+        }
+    }
+}
+
+/// The read side of cell storage (ADR 003). Nothing here writes, so a source that cannot be
+/// written to is not one that fails at runtime: it has no method to fail in.
+pub trait CellSource {
+    /// The value stored at exactly `key`.
+    fn value_at(&self, key: &CellKey) -> Option<CellValue>;
+
+    /// The summary of the values `select` names.
+    fn summarize(&self, select: &Select) -> Summary;
+
+    /// The keys of the cells containing every pair of `partial`.
+    fn keys<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = CellKey> + 'a>;
+
+    /// The cells containing every pair of `partial`, with their values.
+    fn cells<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = (CellKey, CellValue)> + 'a>;
+}
+
+impl CellSource for DataStore {
+    fn value_at(&self, key: &CellKey) -> Option<CellValue> {
+        self.get(key).cloned()
+    }
+
+    fn summarize(&self, select: &Select) -> Summary {
+        let values = match (select.spans.is_empty(), select.exact) {
+            (true, true) => Vec::from_iter(self.get(&CellKey::new(select.base.to_vec()))),
+            (true, false) => self.matching_values(select.base),
+            (false, _) => self.spanned_values(select.base, select.spans, select.exact),
+        };
+        Summary::of(values)
+    }
+
+    fn keys<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = CellKey> + 'a> {
+        Box::new(self.cells(partial).map(|(key, _)| key))
+    }
+
+    fn cells<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = (CellKey, CellValue)> + 'a> {
+        match partial.is_empty() {
+            true => Box::new(self.iter_cells().map(|(k, v)| (k, v.clone()))),
+            false => Box::new(
+                self.matching_cells(partial)
+                    .into_iter()
+                    .map(|(k, v)| (k, v.clone())),
+            ),
+        }
+    }
+}
+
+/// A table of rows read by scanning, the way a query engine with no index would. A second
+/// implementation of [`CellSource`] that shares no code with [`DataStore`] but the summary.
+#[cfg(any(test, feature = "test-support"))]
+#[derive(Debug, Clone, Default)]
+pub struct TableSource {
+    pub rows: Vec<(CellKey, CellValue)>,
+}
+
+#[cfg(any(test, feature = "test-support"))]
+impl TableSource {
+    fn holds(key: &CellKey, select: &Select) -> bool {
+        let has = |(cat, item): &(String, String)| key.get(cat) == Some(item.as_str());
+        let spanned = |(cat, items): &(&str, Vec<&str>)| {
+            key.get(cat).is_some_and(|held| items.contains(&held))
+        };
+        select.base.iter().all(has)
+            && select.spans.iter().all(spanned)
+            && (!select.exact || key.0.len() == select.base.len() + select.spans.len())
+    }
+}
+
+#[cfg(any(test, feature = "test-support"))]
+impl CellSource for TableSource {
+    fn value_at(&self, key: &CellKey) -> Option<CellValue> {
+        self.rows
+            .iter()
+            .find(|(k, _)| k == key)
+            .map(|(_, v)| v.clone())
+    }
+
+    fn summarize(&self, select: &Select) -> Summary {
+        let held = self.rows.iter().filter(|(key, _)| Self::holds(key, select));
+        Summary::of(held.map(|(_, v)| v))
+    }
+
+    fn keys<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = CellKey> + 'a> {
+        Box::new(self.cells(partial).map(|(key, _)| key))
+    }
+
+    fn cells<'a>(
+        &'a self,
+        partial: &'a [(String, String)],
+    ) -> Box<dyn Iterator<Item = (CellKey, CellValue)> + 'a> {
+        let select = Select::matching(partial);
+        let held = self.rows.iter();
+        Box::new(
+            held.filter(move |(key, _)| Self::holds(key, &select))
+                .cloned(),
+        )
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const FUNCS: [AggFunc; 5] = [
+        AggFunc::Sum,
+        AggFunc::Avg,
+        AggFunc::Min,
+        AggFunc::Max,
+        AggFunc::Count,
+    ];
+
+    fn pool() -> Vec<CellValue> {
+        let text = |s: &str| CellValue::Text(s.into());
+        vec![
+            CellValue::Number(1.5),
+            CellValue::Number(-0.0),
+            CellValue::Number(0.0),
+            CellValue::Number(f64::NAN),
+            CellValue::Number(f64::INFINITY),
+            text("a"),
+            text("b"),
+            CellValue::Error("e".into()),
+        ]
+    }
+
+    /// Every sequence of the pool up to `len` values.
+    fn sequences(len: usize) -> Vec<Vec<CellValue>> {
+        let mut all: Vec<Vec<CellValue>> = vec![vec![]];
+        let mut last = all.clone();
+        for _ in 0..len {
+            let extend = |s: &Vec<CellValue>| {
+                pool()
+                    .into_iter()
+                    .map(|v| [s.clone(), vec![v]].concat())
+                    .collect::<Vec<_>>()
+            };
+            last = last.iter().flat_map(extend).collect();
+            all.extend(last.clone());
+        }
+        all
+    }
+
+    /// The reduction as it was before the source moved it, kept to compare against.
+    fn reference(values: &[&CellValue], func: &AggFunc) -> Option<CellValue> {
+        let numbers: Vec<f64> = values.iter().filter_map(|v| v.as_f64()).collect();
+        values.split_first().and_then(|(first, rest)| {
+            if numbers.len() == values.len() {
+                let mut sum = ExactSum::new();
+                numbers.iter().for_each(|v| sum.add(*v, 1));
+                let extreme = |order: fn(&f64, &f64) -> std::cmp::Ordering| {
+                    let finite = numbers.iter().copied().filter(|v| !v.is_nan());
+                    finite.min_by(order).unwrap_or(f64::NAN)
+                };
+                Some(CellValue::Number(match func {
+                    AggFunc::Sum => sum.value(),
+                    AggFunc::Avg => sum.mean(numbers.len() as u64),
+                    AggFunc::Min => extreme(f64::total_cmp),
+                    AggFunc::Max => extreme(|a, b| b.total_cmp(a)),
+                    AggFunc::Count => numbers.len() as f64,
+                }))
+            } else if rest.iter().all(|v| v == first) {
+                Some((*first).clone())
+            } else {
+                Some(CellValue::Error("mixed".to_string()))
+            }
+        })
+    }
+
+    fn same(a: &Option<CellValue>, b: &Option<CellValue>) -> bool {
+        match (a, b) {
+            (Some(CellValue::Number(x)), Some(CellValue::Number(y))) => {
+                x.to_bits() == y.to_bits() || (x.is_nan() && y.is_nan())
+            }
+            _ => a == b,
+        }
+    }
+
+    #[test]
+    fn a_summary_reduces_as_the_values_did() {
+        for values in sequences(3) {
+            let refs: Vec<&CellValue> = values.iter().collect();
+            for func in &FUNCS {
+                let want = reference(&refs, func);
+                let got = Summary::of(&values).reduce(func);
+                assert!(
+                    same(&want, &got),
+                    "{func:?} of {values:?}: {want:?} != {got:?}"
+                );
+            }
+        }
+    }
+
+    #[test]
+    fn a_merge_of_summaries_is_the_summary_of_the_union() {
+        let all = sequences(2);
+        for (a, b) in all.iter().flat_map(|a| all.iter().map(move |b| (a, b))) {
+            let mut merged = Summary::of(a);
+            merged.merge(&Summary::of(b));
+            let union: Vec<&CellValue> = a.iter().chain(b).collect();
+            for func in &FUNCS {
+                let (want, got) = (reference(&union, func), merged.reduce(func));
+                assert!(
+                    same(&want, &got),
+                    "{func:?} of {a:?} + {b:?}: {want:?} != {got:?}"
+                );
+            }
+        }
+    }
+
+    #[test]
+    fn numbers_pool_what_the_formula_aggregate_pools() {
+        let xs = [1.0, -0.0, f64::NAN, 3.0, 0.0];
+        for split in 0..=xs.len() {
+            let mut pooled = Numbers::of(xs[..split].iter().copied());
+            pooled.merge(&Numbers::of(xs[split..].iter().copied()));
+            let whole = Numbers::of(xs);
+            for func in &FUNCS {
+                let (a, b) = (pooled.reduce(func), whole.reduce(func));
+                let equal = a.map(f64::to_bits) == b.map(f64::to_bits)
+                    || a.zip(b).is_some_and(|(x, y)| x.is_nan() && y.is_nan());
+                assert!(equal, "{func:?} split at {split}: {a:?} != {b:?}");
+            }
+            assert_eq!(pooled.total().to_bits(), whole.total().to_bits());
+        }
+        assert_eq!(Numbers::default().total().to_bits(), 0.0f64.to_bits());
+        assert_eq!(Numbers::default().reduce(&AggFunc::Sum), None);
+    }
+
+    fn pair(c: &str, i: &str) -> (String, String) {
+        (c.to_string(), i.to_string())
+    }
+
+    /// Rows of several key shapes over three categories, with text among the numbers.
+    fn rows() -> Vec<(CellKey, CellValue)> {
+        let mut rows = Vec::new();
+        let records = [("E", "A"), ("E", "B"), ("W", "A"), ("W", "B")];
+        for (n, (r, p)) in records.iter().enumerate() {
+            for (m, v) in [("Rev", n as f64 + 1.0), ("Cost", 0.5 * n as f64)] {
+                let key = CellKey::new(vec![pair("R", r), pair("P", p), pair("_Measure", m)]);
+                rows.push((key, CellValue::Number(v)));
+            }
+        }
+        let note = CellKey::new(vec![pair("R", "E"), pair("_Measure", "Note")]);
+        rows.push((note, CellValue::Text("x".into())));
+        let lone = CellKey::new(vec![pair("R", "W"), pair("_Measure", "Rev")]);
+        rows.push((lone, CellValue::Number(100.0)));
+        rows
+    }
+
+    fn store_of(rows: &[(CellKey, CellValue)]) -> DataStore {
+        let mut store = DataStore::new();
+        rows.iter()
+            .for_each(|(k, v)| store.set(k.clone(), v.clone()));
+        store
+    }
+
+    type Spans = Vec<(String, Vec<String>)>;
+
+    /// Every subset of every stored key's pairs as a base, each with no span, with one
+    /// category spanned over each of several item lists, and with two spanned.
+    fn selects(rows: &[(CellKey, CellValue)]) -> Vec<(Vec<(String, String)>, Spans)> {
+        let lists: Vec<Vec<String>> = [
+            vec!["E"],
+            vec!["E", "W"],
+            vec!["A", "B"],
+            vec!["Rev", "Note"],
+            vec!["Cost", "Rev", "Note"],
+            vec!["none"],
+        ]
+        .iter()
+        .map(|l| l.iter().map(|s| s.to_string()).collect())
+        .collect();
+        let mut out = Vec::new();
+        for (key, _) in rows {
+            for mask in 0..(1u32 << key.0.len()) {
+                let keep = |(i, _): &(usize, &(String, String))| mask >> i & 1 == 1;
+                let base: Vec<_> = key.0.iter().enumerate().filter(keep).collect();
+                let base: Vec<_> = base.into_iter().map(|(_, p)| p.clone()).collect();
+                out.push((base.clone(), vec![]));
+                let all = ["R", "P", "_Measure"].map(String::from);
+                let free: Vec<_> = all
+                    .iter()
+                    .filter(|c| base.iter().all(|(b, _)| b != *c))
+                    .collect();
+                for cat in free.iter().copied() {
+                    for list in &lists {
+                        out.push((base.clone(), vec![(cat.clone(), list.clone())]));
+                        for other in free.iter().copied().filter(|o| *o != cat) {
+                            let both = vec![
+                                (cat.clone(), list.clone()),
+                                (other.clone(), lists[1].clone()),
+                            ];
+                            out.push((base.clone(), both));
+                        }
+                    }
+                }
+            }
+        }
+        out
+    }
+
+    #[test]
+    fn the_store_and_a_scanned_table_answer_every_select_alike() {
+        let rows = rows();
+        let (store, table) = (store_of(&rows), TableSource { rows: rows.clone() });
+        let mut asked = 0;
+        for (base, spans) in selects(&rows) {
+            let spans: Vec<(&str, Vec<&str>)> = spans
+                .iter()
+                .map(|(c, i)| (c.as_str(), i.iter().map(String::as_str).collect()))
+                .collect();
+            for exact in [false, true] {
+                let select = Select::spanning(&base, &spans, exact);
+                for func in &FUNCS {
+                    let a = store.summarize(&select).reduce(func);
+                    let b = table.summarize(&select).reduce(func);
+                    assert!(
+                        same(&a, &b),
+                        "{func:?} {base:?} {spans:?} exact={exact}: {a:?} != {b:?}"
+                    );
+                    asked += 1;
+                }
+            }
+        }
+        assert!(asked > 5000, "the enumeration shrank to {asked}");
+    }
+
+    #[test]
+    fn the_store_and_a_scanned_table_list_the_same_cells() {
+        let rows = rows();
+        let (store, table) = (store_of(&rows), TableSource { rows: rows.clone() });
+        for (base, _) in selects(&rows) {
+            let sorted = |source: &dyn CellSource| {
+                let mut cells: Vec<_> = source.cells(&base).collect();
+                cells.sort_by(|a, b| a.0.cmp(&b.0));
+                cells
+            };
+            assert_eq!(sorted(&store), sorted(&table), "cells under {base:?}");
+            let mut keys: Vec<_> = store.keys(&base).collect();
+            keys.sort();
+            let listed: Vec<_> = sorted(&table).into_iter().map(|c| c.0).collect();
+            assert_eq!(keys, listed, "keys under {base:?}");
+        }
+        for (key, value) in &rows {
+            assert_eq!(store.value_at(key).as_ref(), Some(value));
+            assert_eq!(table.value_at(key).as_ref(), Some(value));
+        }
+        let absent = CellKey::new(vec![pair("R", "nowhere")]);
+        assert_eq!(
+            (store.value_at(&absent), table.value_at(&absent)),
+            (None, None)
+        );
+    }
+
+    #[test]
+    fn an_exact_select_with_no_span_names_one_cell() {
+        let store = store_of(&rows());
+        let note = vec![pair("R", "E"), pair("_Measure", "Note")];
+        let one = store.summarize(&Select::spanning(&note, &[], true));
+        assert_eq!(one.reduce(&AggFunc::Sum), Some(CellValue::Text("x".into())));
+        let wide = vec![pair("R", "E")];
+        let loose = store.summarize(&Select::matching(&wide));
+        let mixed = Some(CellValue::Error("mixed".into()));
+        assert_eq!(
+            loose.reduce(&AggFunc::Sum),
+            mixed,
+            "E holds text and numbers"
+        );
+        let exact = store.summarize(&Select::spanning(&wide, &[], true));
+        assert_eq!(
+            exact.reduce(&AggFunc::Sum),
+            None,
+            "no cell is keyed by R=E alone"
+        );
+    }
+}

crates/improvise-core/src/model/types.rs

@@ -5,6 +5,7 @@ use indexmap::{IndexMap, IndexSet};
 
 use super::category::{Category, CategoryId, CategoryKind, MEASURE_CATEGORY};
 use super::cell::{CellKey, CellValue, DataStore};
+use super::source::{CellSource, Numbers, Select};
 use crate::formula::{AggFunc, Expr, Filter, Formula, FormulaId};
 
 /// Regular categories a model may hold; labels and virtuals do not count.
@@ -15,30 +16,6 @@ pub const MAX_CATEGORIES: usize = 24;
 /// They must agree; why, and why 1e-10: context/code/core.md §4.
 const FLOAT_EQ_EPSILON: f64 = 1e-10;
 
-/// The sum correctly rounded once, whatever the order (ADR 009 clause 8), signed as IEEE
-/// signs a zero total.
-fn exact_sum(values: &[f64]) -> f64 {
-    exact(values).value()
-}
-
-/// The mean correctly rounded once: the exact total over the count, not a rounded sum divided.
-fn exact_mean(values: &[f64]) -> f64 {
-    exact(values).mean(values.len() as u64)
-}
-
-fn exact(values: &[f64]) -> numtower::ExactSum {
-    let mut sum = numtower::ExactSum::new();
-    values.iter().for_each(|v| sum.add(*v, 1));
-    sum
-}
-
-/// The least of `values` under `order`, skipping NaN; NaN if nothing else is left. Zeros order
-/// by sign (`-0.0` below `+0.0`), so the answer does not depend on the order or the platform.
-fn extreme(values: &[f64], order: impl Fn(&f64, &f64) -> std::cmp::Ordering) -> f64 {
-    let numbers = values.iter().copied().filter(|v| !v.is_nan());
-    numbers.min_by(order).unwrap_or(f64::NAN)
-}
-
 /// Whether two cached cells are the same value for a fixed point: numbers by bits, so a zero
 /// that changes sign is a change, except that every NaN is one value, so a NaN can settle.
 pub fn settled(a: Option<&CellValue>, b: Option<&CellValue>) -> bool {
@@ -52,36 +29,16 @@ pub fn settled(a: Option<&CellValue>, b: Option<&CellValue>) -> bool {
 
 /// Reduce raw values. `None` on empty: an empty cell reads as **absent**, not
 /// as zero. Both aggregation paths share it (improvise-kdp, core.md §2).
+#[cfg(test)]
 fn apply_agg(values: &[f64], func: &AggFunc) -> Option<f64> {
-    (!values.is_empty()).then(|| match func {
-        AggFunc::Sum => exact_sum(values),
-        AggFunc::Avg => exact_mean(values),
-        AggFunc::Min => extreme(values, f64::total_cmp),
-        AggFunc::Max => extreme(values, |a, b| b.total_cmp(a)),
-        AggFunc::Count => values.len() as f64,
-    })
-}
-
-/// Reduce a cell's raw values. Numbers through [`apply_agg`]; anything else
-/// by **unanimity**, else `Error("mixed")`. Branch order matters — §2.
-fn aggregate_cell_values(values: &[&CellValue], func: &AggFunc) -> Option<CellValue> {
-    let numbers: Vec<f64> = values.iter().filter_map(|v| v.as_f64()).collect();
-    values.split_first().and_then(|(first, rest)| {
-        if numbers.len() == values.len() {
-            apply_agg(&numbers, func).map(CellValue::Number)
-        } else if rest.iter().all(|v| v == first) {
-            Some((*first).clone())
-        } else {
-            Some(CellValue::Error("mixed".to_string()))
-        }
-    })
+    Numbers::of(values.iter().copied()).reduce(func)
 }
 
 /// The member records behind `partial`, in `DataStore` insertion order and
 /// without repeats — the coordinate tuples minus `_Measure` (§13).
 fn member_records(model: &Model, partial: &CellKey) -> Vec<CellKey> {
     let mut seen: IndexSet<CellKey> = IndexSet::new();
-    for (key, _) in model.data.matching_cells(&partial.0) {
+    for key in model.source().keys(&partial.0) {
         seen.insert(key.without(MEASURE_CATEGORY));
     }
     seen.into_iter().collect()
@@ -107,16 +64,16 @@ fn eval_agg(
             None => c.clone(),
         })
         .collect();
-    let values = distinct
-        .iter()
-        .map(|c| agg_values(model, c, target_category, inner, agg_filter, eval_at))
-        .collect::<Result<Vec<_>, String>>()?
-        .concat();
+    let mut pooled = Numbers::default();
+    for context in &distinct {
+        let found = agg_values(model, context, target_category, inner, agg_filter, eval_at)?;
+        pooled.merge(&found);
+    }
     match func {
-        AggFunc::Sum => Ok(exact_sum(&values)),
-        AggFunc::Count => Ok(values.len() as f64),
+        AggFunc::Sum => Ok(pooled.total()),
+        AggFunc::Count => Ok(pooled.count() as f64),
         AggFunc::Avg | AggFunc::Min | AggFunc::Max => {
-            apply_agg(&values, func).ok_or_else(|| "empty".to_string())
+            pooled.reduce(func).ok_or_else(|| "empty".to_string())
         }
     }
 }
@@ -130,7 +87,7 @@ fn agg_values(
     inner: &Expr,
     agg_filter: Option<&Filter>,
     eval_at: &dyn Fn(&Expr, &CellKey) -> Result<f64, String>,
-) -> Result<Vec<f64>, String> {
+) -> Result<Numbers, String> {
     let mut partial = context.without(target_category);
     // The fast path is the fold specialised to a ref that names a STORED
     // cell; where it does not, the premise fails — §13, s08t, s2w3.
@@ -156,11 +113,10 @@ fn agg_values(
         // A bare ref asks for the values that exist; anything else asks the
         // expression of each member record and folds the answers — §13.
         true => Ok(model
-            .data
-            .matching_values(&partial.0)
-            .into_iter()
-            .filter_map(|v| v.as_f64())
-            .collect()),
+            .source()
+            .summarize(&Select::matching(&partial.0))
+            .numbers()
+            .clone()),
         false => member_records(model, &partial)
             .into_iter()
             .map(|record| {
@@ -170,7 +126,8 @@ fn agg_values(
                     .fold(partial.clone(), |k, (cat, item)| k.with(cat, item));
                 eval_at(inner, &at)
             })
-            .collect(),
+            .collect::<Result<Vec<_>, String>>()
+            .map(Numbers::of),
     }
 }
 
@@ -765,10 +722,21 @@ impl Model {
         }
         match self.formula_for(key) {
             Some(formula) => self.eval_formula_depth(formula, key, depth - 1),
-            None => self.data.get(key).cloned(),
+            None => self.source().value_at(key),
         }
     }
 
+    /// Where raw cell values are read from: the owned store.
+    pub fn source(&self) -> &dyn CellSource {
+        &self.data
+    }
+
+    /// `select`'s cells reduced by `key`'s measure aggregation: numbers by the function, anything
+    /// else by unanimity, else `Error("mixed")` (core.md §2).
+    fn read(&self, select: &Select, key: &CellKey) -> Option<CellValue> {
+        self.source().summarize(select).reduce(self.agg_for(key))
+    }
+
     /// The aggregation to apply to a cell's raw values, from the measure
     /// coordinate's entry in `measure_agg`. Defaults to `Sum`.
     fn agg_for(&self, key: &CellKey) -> &AggFunc {
@@ -898,7 +866,7 @@ impl Model {
         if let Some(cached) = self.formula_cache.get(key) {
             return Some(cached.clone());
         }
-        aggregate_cell_values(&self.data.matching_values(&key.0), self.agg_for(key))
+        self.read(&Select::matching(&key.0), key)
     }
 
     /// Each category `key` folds, with its covered members: a collapsed GROUP it
@@ -979,10 +947,8 @@ impl Model {
         agg: Aggregation,
     ) -> Option<CellValue> {
         let exact = agg == Aggregation::Exact;
-        let values = self
-            .data
-            .spanned_values(&expansion.base.0, &expansion.spans, exact);
-        aggregate_cell_values(&values, self.agg_for(key))
+        let select = Select::spanning(&expansion.base.0, &expansion.spans, exact);
+        self.read(&select, key)
     }
 
     /// A formula at a group key, its refs read through the fold — the value the
@@ -1042,7 +1008,7 @@ impl Model {
         // (improvise-6os replaced an O(n^2) Vec::contains).
         let mut stems: IndexSet<CellKey> = IndexSet::new();
         for (target_cat, _) in &formula_cats {
-            for (key, _) in self.data.iter_cells() {
+            for key in self.source().keys(&[]) {
                 let stem = key.without(target_cat);
                 // Strip none_cats too: the grid queries without them.
                 let mut stripped = stem;
@@ -1262,9 +1228,9 @@ impl Model {
     /// Aggregate raw data (never formulas) over hidden dimensions.
     fn aggregate_raw(&self, key: &CellKey, none_cats: &[String]) -> Option<CellValue> {
         if none_cats.is_empty() {
-            return self.data.get(key).cloned();
+            return self.source().value_at(key);
         }
-        aggregate_cell_values(&self.data.matching_values(&key.0), self.agg_for(key))
+        self.read(&Select::matching(&key.0), key)
     }
 
     fn eval_formula_depth(
@@ -1280,7 +1246,7 @@ impl Model {
                 .map(|v| v == filter.item.as_str())
                 .unwrap_or(false);
             if !matches {
-                return self.data.get(context).cloned();
+                return self.source().value_at(context);
             }
         }
 

crates/improvise-core/src/view/layout.rs

@@ -91,20 +91,7 @@ pub fn drill_constraint(model: &Model, key: &CellKey) -> CellKey {
 /// The frozen record list a drill into `constraint` shows. **One home**: a
 /// second derivation would freeze a different order — core.md §8, gotcha 18.
 pub fn drill_snapshot(model: &Model, constraint: &CellKey) -> Vec<RecordRow> {
-    let cells: Vec<(CellKey, CellValue)> = match constraint.0.is_empty() {
-        true => model
-            .data
-            .iter_cells()
-            .map(|(k, v)| (k, v.clone()))
-            .collect(),
-        false => model
-            .data
-            .matching_cells(&constraint.0)
-            .into_iter()
-            .map(|(k, v)| (k, v.clone()))
-            .collect(),
-    };
-    RecordRow::group(cells)
+    RecordRow::group(model.source().cells(&constraint.0))
 }
 
 /// The records row `record` is, as the records grid groups it; empty when no
@@ -729,13 +716,7 @@ impl GridLayout {
     }
 
     fn group_records(model: &Model, page_coords: &[(String, String)]) -> Vec<RecordRow> {
-        RecordRow::group(
-            model
-                .data
-                .matching_cells(page_coords)
-                .into_iter()
-                .map(|(k, v)| (k, v.clone())),
-        )
+        RecordRow::group(model.source().cells(page_coords))
     }
 
     /// `[coordinates…, +cat, measures…, +]`. Formula targets are not columns:

scripts/build-doc-site.mjs

@@ -72,6 +72,7 @@ const SECTIONS = [
       ["context/code/zset-ffi.md", "code/zset-ffi", "zset-ffi, the C ABI"],
       ["context/code/delivery.md", "code/delivery", "delivery, notices and corrections"],
       ["context/code/improvise-incremental.md", "code/improvise-incremental", "improvise-incremental, the plug"],
+      ["context/code/source.md", "code/source", "CellSource, where cells are read from"],
       ["context/code/kb-tool.md", "code/kb-tool", "The kb tool"],
       ["context/code/tooling.md", "code/tooling", "The repository's own checkers"],
     ],