improvise/context/code/source.md

CellSource: where cell values are read from

crates/improvise-core/src/model/source.rs. ADR 003 asked for a source trait in core with the in-RAM DataStore as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for the two live sources that need it. This file is the reasoning; the evidence for each test is in context/test-evidence/core.md. The ADRs are not amended: what follows are the readings taken where they leave a choice.

What is in the trait, and why

method	what the evaluator asks
value_at(key)	the value stored at exactly key (an exact read, the filter-mismatch fallback)
summarize(select)	the summary of the cells a Select names: the aggregate, not the rows
keys(partial)	the keys of the cells containing partial: stems, and the records behind an aggregate
cells(partial)	the same with values: records mode and a drill
count(select, excluding)	how many cells a select names, leaving out those holding a hidden pair: occupancy
distinct_without(cats)	the cell keys with cats removed, each once, first seen first: the stems formulas are evaluated at

1. Aggregation is the interface. ADR 003 clause 1: GridLayout asks for a value, not for the rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk, summarize, returns a Summary and never a list of values. keys and cells exist because some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and a SQL source answers them with a SELECT DISTINCT or a plain select.
2. Writes are not in the trait. DataStore::set and remove are inherent methods. A source that cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than enforced". The model's own writes (set_cell, clear_cell) go to the owned store.
3. The trait has no Result. A source that can fail (a connection) decides at its own boundary what a failed read is, and improvise-io is where that decision belongs. AGENTS.md keeps Option and Result at the edges; Option is the one the trait uses, for "no such cell".
4. Select is spanned_values' arguments. A fold names a partial key, the covered members of each group or omitted category it spans, and whether only an exact-width cell counts. The covered members are decided in RAM from the categories and the view's hidden items, so a source is handed explicit item lists and never a predicate. A hidden item therefore reaches a source only as the absence of an item from a list.

Summary and Numbers

A Summary is what a source reports about the values of a read: a Numbers (count, an exact sum, the least and greatest non-NaN, in total_cmp order so -0.0 is below +0.0) and whether the rest (text, errors) agree on one value. It reduces exactly as aggregate_cell_values did: numbers by the function, anything else by unanimity, else Error("mixed"), none at all absent. Three properties carry the design:

• It merges. A read of a collapsed group, a total, or several contexts of a formula aggregate is the merge of the summaries of its parts, equal to the summary of the union. That is what lets an engine answer a pooled read without sending rows, and what lets eval_agg pool Numbers instead of concatenating Vec<f64>. The sum is ExactSum, so a merge is the same bits in any order.
• It is order-free. Unanimity needs the set of non-number values, not their order, so a source that has no insertion order loses nothing by it.
• The reference is kept. tests::reference is the old reduction, copied into the test module; a_summary_reduces_as_the_values_did compares every sequence of an eight-value pool, up to three long, with Summary over all five functions. characterization_survives_the_move: the existing evaluator tests (423 in the crate) are unchanged and pass.

What a real engine must supply is stated here because the trait cannot enforce it: a Numbers whose sum is exact. SQLite's SUM is a float sum, so an engine that cannot return an exact accumulator either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which context/code/improvise-incremental.md ("Step 6") calls a user-visible change.

Model::source() and Model::over(source)

Model::source() returns the &dyn CellSource that the evaluator, the stems of recompute_formulas, the member records of an aggregate over an expression and the records of GridLayout read through. Model::over(source) points it at a SharedSource (an Arc<dyn CellSource + Send + Sync>) instead of the owned DataStore. That is the whole evaluator hook (improvise-rnak, bw4p.14's prerequisite): there is still one evaluator, Model's, and it computes every formula value, over a source whose cells it never owns.

1. A model over a source is read-only, in every build. set_cell, clear_cell, remove_category and remove_item assert! that no source is set (a debug_assert first shipped and was lost in release: the write went to the owned store, which nothing reads). A caller who holds a source writes to the source (ADR 003: the trait has no writes). Model::over drops the model's own cells and the formula cells computed from them, so no stale copy is left for get_cell, persistence or the harness to read. data is pub, so a caller can still write it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed) are refused too, because the source's cells would still carry them.
2. A shared source changes behind its readers by interior mutability of its own. The model holds an Arc, so a database connection or a test double behind a lock is the same object the backend feeds from.
3. What still reads the owned store: get_cell (a production caller: improvise-command/src/ui/effect.rs, the old value of a rename), Model::data (pub), persistence, the import wizard and the effects in improvise-command. They are the facade of a document that owns its cells; a model over a source has none of those to offer, and routing them would change get_cell's &CellValue return for 129 call sites. Eager::occupancy, Shapes::of and Stems::build in improvise-incremental are routed by the PRs that need them.
4. Debug shows only whether a source is set. A source is not necessarily printable, and two models over different sources are not told apart by it.

TableSource

A scanned table of rows, behind cfg(any(test, feature = "test-support")). It is the second implementation, written as a query engine with no index would answer, sharing only Summary with DataStore. the_store_and_a_scanned_table_answer_every_select_alike asks both every select over a fixture (every subset of every stored key as a base, with no span, one span of each of six item lists, and two), at both widths and for all five functions: over 5,000 triples.

ChangeSource

CellSource plus drain_changes(&self) -> Vec<CellChange>: a source that reports what changed in it. The contract that matters is that a source's cells are those as of the last drain, so a snapshot read now plus the changes drained later counts no change twice or not at all. Why, and how Incremental uses it: context/code/improvise-incremental.md, step 7. DataStore is not a ChangeSource: its log is &mut and is drained by the owner in the step that made the edit. A SharedFeed is the Arc a backend and a model share.

Pushdown needs two more queries

count and distinct_without joined the trait for Pushdown (improvise-rnak). Both are what a query engine answers without sending rows (COUNT(*) ... WHERE, SELECT DISTINCT), and each replaces a read that would have transferred every key: Eager::occupancy lists all keys, and recompute_formulas walked every cell to find its stems. recompute_formulas now asks distinct_without(target and None categories) once per formula, and the order it returns is the order the cells were first seen in, so the stems are inserted in the order they always were. That order is not a contract a source can be held to: the passes of the fixed point visit stems in it and a chain of stems can settle or not by it. That is a known evaluator defect, improvise-bw4p.30 (context/code/improvise-incremental.md, step 7 item 8), and until it is fixed a source with no order of its own should list stems in a stable one.

Select::admits is the predicate of a select over one key, for a source that must test keys itself (DataStore::count). TableSource does not use it: its own holds is the independent second reading.

backend and a model share. A feed has one drainer: a drain empties the pending list, so two backends over one feed split the changes between them (measured in review: the same write gave one backend a Revenue of 128 and the other 38, and the second then disagreed with its own exact reads). a_feed_has_one_drainer pins the behaviour. Kb-style invariant: invariant(change_source, one_drainer).

Building a Summary from an engine's answer

Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest)?, Agreement)?. Agreement is what the non-number values hold: None, One(v) or Mixed. Both constructors return a Result and refuse a state no sequence of values could produce (extremes with no numbers, a NaN extreme, least above greatest, one extreme without the other, a sum with no numbers, no extremes beside a sum that is not NaN, a number among the "others"), because repairing it would hide an engine's bug.