improvise/context/code/source.md

CellSource: where cell values are read from

crates/improvise-core/src/model/source.rs. ADR 003 asked for a source trait in core with the in-RAM DataStore as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for the two live sources that need it. This file is the reasoning; the evidence for each test is in context/test-evidence/core.md. The ADRs are not amended: what follows are the readings taken where they leave a choice.

What is in the trait, and why

method	what the evaluator asks
value_at(key)	the value stored at exactly key (an exact read, the filter-mismatch fallback)
summarize(select)	the summary of the cells a Select names: the aggregate, not the rows
keys(partial)	the keys of the cells containing partial: stems, and the records behind an aggregate
cells(partial)	the same with values: records mode and a drill

1. Aggregation is the interface. ADR 003 clause 1: GridLayout asks for a value, not for the rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk, summarize, returns a Summary and never a list of values. keys and cells exist because some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and a SQL source answers them with a SELECT DISTINCT or a plain select.
2. Writes are not in the trait. DataStore::set and remove are inherent methods. A source that cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than enforced". The model's own writes (set_cell, clear_cell) go to the owned store.
3. The trait has no Result. A source that can fail (a connection) decides at its own boundary what a failed read is, and improvise-io is where that decision belongs. AGENTS.md keeps Option and Result at the edges; Option is the one the trait uses, for "no such cell".
4. Select is spanned_values' arguments. A fold names a partial key, the covered members of each group or omitted category it spans, and whether only an exact-width cell counts. The covered members are decided in RAM from the categories and the view's hidden items, so a source is handed explicit item lists and never a predicate. A hidden item therefore reaches a source only as the absence of an item from a list.

Summary and Numbers

A Summary is what a source reports about the values of a read: a Numbers (count, an exact sum, the least and greatest non-NaN, in total_cmp order so -0.0 is below +0.0) and whether the rest (text, errors) agree on one value. It reduces exactly as aggregate_cell_values did: numbers by the function, anything else by unanimity, else Error("mixed"), none at all absent. Three properties carry the design:

• It merges. A read of a collapsed group, a total, or several contexts of a formula aggregate is the merge of the summaries of its parts, equal to the summary of the union. That is what lets an engine answer a pooled read without sending rows, and what lets eval_agg pool Numbers instead of concatenating Vec<f64>. The sum is ExactSum, so a merge is the same bits in any order.
• It is order-free. Unanimity needs the set of non-number values, not their order, so a source that has no insertion order loses nothing by it.
• The reference is kept. tests::reference is the old reduction, copied into the test module; a_summary_reduces_as_the_values_did compares every sequence of an eight-value pool, up to three long, with Summary over all five functions. characterization_survives_the_move: the existing evaluator tests (423 in the crate) are unchanged and pass.

What a real engine must supply is stated here because the trait cannot enforce it: a Numbers whose sum is exact. SQLite's SUM is a float sum, so an engine that cannot return an exact accumulator either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which context/code/improvise-incremental.md ("Step 6") calls a user-visible change.

Model::source() and Model::over(source)

Model::source() returns the &dyn CellSource that the evaluator, the stems of recompute_formulas, the member records of an aggregate over an expression and the records of GridLayout read through. Model::over(source) points it at a SharedSource (an Arc<dyn CellSource + Send + Sync>) instead of the owned DataStore. That is the whole evaluator hook (improvise-rnak, bw4p.14's prerequisite): there is still one evaluator, Model's, and it computes every formula value, over a source whose cells it never owns.

1. A model over a source is read-only, in every build. set_cell, clear_cell, remove_category and remove_item assert! that no source is set (a debug_assert first shipped and was lost in release: the write went to the owned store, which nothing reads). A caller who holds a source writes to the source (ADR 003: the trait has no writes). Model::over drops the model's own cells and the formula cells computed from them, so no stale copy is left for get_cell, persistence or the harness to read. data is pub, so a caller can still write it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed) are refused too, because the source's cells would still carry them.
2. A shared source changes behind its readers by interior mutability of its own. The model holds an Arc, so a database connection or a test double behind a lock is the same object the backend feeds from.
3. What still reads the owned store: get_cell (a production caller: improvise-command/src/ui/effect.rs, the old value of a rename), Model::data (pub), persistence, the import wizard and the effects in improvise-command. They are the facade of a document that owns its cells; a model over a source has none of those to offer, and routing them would change get_cell's &CellValue return for 129 call sites. Eager::occupancy, Shapes::of and Stems::build in improvise-incremental are routed by the PRs that need them.
4. Debug shows only whether a source is set. A source is not necessarily printable, and two models over different sources are not told apart by it.

TableSource

A scanned table of rows, behind cfg(any(test, feature = "test-support")). It is the second implementation, written as a query engine with no index would answer, sharing only Summary with DataStore. the_store_and_a_scanned_table_answer_every_select_alike asks both every select over a fixture (every subset of every stored key as a base, with no span, one span of each of six item lists, and two), at both widths and for all five functions: over 5,000 triples.

Building a Summary from an engine's answer

Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest)?, Agreement)?. Agreement is what the non-number values hold: None, One(v) or Mixed. Both constructors return a Result and refuse a state no sequence of values could produce (extremes with no numbers, a NaN extreme, least above greatest, one extreme without the other, a sum with no numbers, no extremes beside a sum that is not NaN, a number among the "others"), because repairing it would hide an engine's bug.