improvise/context/code/source.md

# `CellSource`: where cell values are read from

`crates/improvise-core/src/model/source.rs`. ADR 003 asked for a source trait in core with the
in-RAM `DataStore` as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for
the two live sources that need it. This file is the reasoning; the evidence for each test is in
`context/test-evidence/core.md`. The ADRs are not amended: what follows are the readings taken where
they leave a choice.

## What is in the trait, and why

| method | what the evaluator asks |
|---|---|
| `value_at(key)` | the value stored at exactly `key` (an exact read, the filter-mismatch fallback) |
| `summarize(select)` | the summary of the cells a `Select` names: the aggregate, not the rows |
| `keys(partial)` | the keys of the cells containing `partial`: stems, and the records behind an aggregate |
| `cells(partial)` | the same with values: records mode and a drill |
| `count(select, excluding)` | how many cells a select names, leaving out those holding a hidden pair: occupancy |
| `distinct_without(cats)` | the cell keys with `cats` removed, each once, first seen first: the stems formulas are evaluated at |

1. **Aggregation is the interface.** ADR 003 clause 1: `GridLayout` asks for a value, not for the
   rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk,
   `summarize`, returns a `Summary` and never a list of values. `keys` and `cells` exist because
   some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and
   a SQL source answers them with a `SELECT DISTINCT` or a plain select.
2. **Writes are not in the trait.** `DataStore::set` and `remove` are inherent methods. A source that
   cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than
   enforced". The model's own writes (`set_cell`, `clear_cell`) go to the owned store.
3. **The trait has no `Result`.** A source that can fail (a connection) decides at its own boundary
   what a failed read is, and `improvise-io` is where that decision belongs. AGENTS.md keeps
   `Option` and `Result` at the edges; `Option` is the one the trait uses, for "no such cell".
4. **`Select` is `spanned_values`' arguments.** A fold names a partial key, the covered members of
   each group or omitted category it spans, and whether only an exact-width cell counts. The
   covered members are decided in RAM from the categories and the view's hidden items, so a source
   is handed explicit item lists and never a predicate. A hidden item therefore reaches a source
   only as the absence of an item from a list.

## `Summary` and `Numbers`

A `Summary` is what a source reports about the values of a read: a `Numbers` (count, an exact sum,
the least and greatest non-NaN, in `total_cmp` order so `-0.0` is below `+0.0`) and whether the rest
(text, errors) agree on one value. It reduces exactly as `aggregate_cell_values` did: numbers by the
function, anything else by unanimity, else `Error("mixed")`, none at all absent. Three properties
carry the design:

- **It merges.** A read of a collapsed group, a total, or several contexts of a formula aggregate is
  the merge of the summaries of its parts, equal to the summary of the union. That is what lets an
  engine answer a pooled read without sending rows, and what lets `eval_agg` pool `Numbers` instead
  of concatenating `Vec<f64>`. The sum is `ExactSum`, so a merge is the same bits in any order.
- **It is order-free.** Unanimity needs the set of non-number values, not their order, so a source
  that has no insertion order loses nothing by it.
- **The reference is kept.** `tests::reference` is the old reduction, copied into the test module;
  `a_summary_reduces_as_the_values_did` compares every sequence of an eight-value pool, up to three
  long, with `Summary` over all five functions. `characterization_survives_the_move`: the existing
  evaluator tests (423 in the crate) are unchanged and pass.

What a real engine must supply is stated here because the trait cannot enforce it: a `Numbers` whose
sum is exact. SQLite's `SUM` is a float sum, so an engine that cannot return an exact accumulator
either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which
`context/code/improvise-incremental.md` ("Step 6") calls a user-visible change.

## `Model::source()` and `Model::over(source)`

`Model::source()` returns the `&dyn CellSource` that the evaluator, the stems of `recompute_formulas`,
the member records of an aggregate over an expression and the records of `GridLayout` read through.
`Model::over(source)` points it at a `SharedSource` (an `Arc<dyn CellSource + Send + Sync>`) instead of
the owned `DataStore`. That is the whole evaluator hook (improvise-rnak, bw4p.14's prerequisite):
there is still one evaluator, `Model`'s, and it computes every formula value, over a source whose
cells it never owns.

1. **A model over a source is read-only, in every build.** `set_cell`, `clear_cell`, `remove_category` and
   `remove_item` `assert!` that no source is set (a `debug_assert` first shipped and was lost in release:
   the write went to the owned store, which nothing reads). A caller who holds a source writes to the
   source (ADR 003: the trait has no writes). `Model::over` drops the model's own cells and the formula cells computed from them, so no stale copy
   is left for `get_cell`, persistence or the harness to read. `data` is `pub`, so a caller can still write
   it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed)
   are refused too, because the source's cells would still carry them.
2. **A shared source changes behind its readers by interior mutability of its own.** The model holds
   an `Arc`, so a database connection or a test double behind a lock is the same object the
   backend feeds from.
3. **What still reads the owned store**: `get_cell` (a production caller: `improvise-command/src/ui/effect.rs`,
   the `old` value of a rename), `Model::data` (`pub`), persistence, the import wizard and the effects
   in `improvise-command`. They are the facade of a document that owns its cells; a model over a source
   has none of those to offer, and routing them would change `get_cell`'s `&CellValue` return for 129
   call sites. `Eager::occupancy`, `Shapes::of` and `Stems::build` in `improvise-incremental` are
   routed by the PRs that need them.
4. **`Debug` shows only whether a source is set.** A source is not necessarily printable, and two
   models over different sources are not told apart by it.

## `TableSource`

A scanned table of rows, behind `cfg(any(test, feature = "test-support"))`. It is the second
implementation, written as a query engine with no index would answer, sharing only `Summary` with
`DataStore`. `the_store_and_a_scanned_table_answer_every_select_alike` asks both every select over a
fixture (every subset of every stored key as a base, with no span, one span of each of six item
lists, and two), at both widths and for all five functions: over 5,000 triples.

## `ChangeSource`

`CellSource` plus `drain_changes(&self) -> Vec<CellChange>`: a source that reports what changed in it.
The contract that matters is that a source's cells are those as of the last drain, so a snapshot read
now plus the changes drained later counts no change twice or not at all. Why, and how `Incremental`
uses it: `context/code/improvise-incremental.md`, step 7. `DataStore` is not a `ChangeSource`: its log
is `&mut` and is drained by the owner in the step that made the edit. A `SharedFeed` is the `Arc` a
backend and a model share.

## Pushdown needs two more queries

`count` and `distinct_without` joined the trait for `Pushdown` (improvise-rnak). Both are what a query
engine answers without sending rows (`COUNT(*) ... WHERE`, `SELECT DISTINCT`), and each replaces a
read that would have transferred every key: `Eager::occupancy` lists all keys, and
`recompute_formulas` walked every cell to find its stems. `recompute_formulas` now asks
`distinct_without(target and None categories)` once per formula, and the order it returns is
the order the cells were first seen in, so the stems are inserted in the order they always were.
That order is not a contract a source can be held to: the passes of the fixed point visit stems in
it and a chain of stems can settle or not by it. That is a known evaluator defect, improvise-bw4p.30
(`context/code/improvise-incremental.md`, step 7 item 8), and until it is fixed a source with no order
of its own should list stems in a stable one.

`Select::admits` is the predicate of a select over one key, for a source that must test keys itself
(`DataStore::count`). `TableSource` does not use it: its own `holds` is the independent second reading.

backend and a model share. **A feed has one drainer**: a drain empties the pending list, so two
backends over one feed split the changes between them (measured in review: the same write gave one
backend a Revenue of 128 and the other 38, and the second then disagreed with its own exact reads).
`a_feed_has_one_drainer` pins the behaviour. Kb-style invariant: `invariant(change_source, one_drainer)`.

## Building a `Summary` from an engine's answer

`Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest)?, Agreement)?`. `Agreement`
is what the non-number values hold: `None`, `One(v)` or `Mixed`. Both constructors return a `Result`
and **refuse** a state no sequence of values could produce (extremes with no numbers, a NaN extreme,
least above greatest, one extreme without the other, a number among the "others"), because repairing
it would hide an engine's bug.