# `CellSource`: where cell values are read from

`crates/improvise-core/src/model/source.rs`. ADR 003 asked for a source trait in core with the
in-RAM `DataStore` as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for
the two live sources that need it. This file is the reasoning; the evidence for each test is in
`context/test-evidence/core.md`. The ADRs are not amended: what follows are the readings taken where
they leave a choice.

## What is in the trait, and why

| method | what the evaluator asks |
|---|---|
| `value_at(key)` | the value stored at exactly `key` (an exact read, the filter-mismatch fallback) |
| `summarize(select)` | the summary of the cells a `Select` names: the aggregate, not the rows |
| `keys(partial)` | the keys of the cells containing `partial`: stems, and the records behind an aggregate |
| `cells(partial)` | the same with values: records mode and a drill |

1. **Aggregation is the interface.** ADR 003 clause 1: `GridLayout` asks for a value, not for the
   rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk,
   `summarize`, returns a `Summary` and never a list of values. `keys` and `cells` exist because
   some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and
   a SQL source answers them with a `SELECT DISTINCT` or a plain select.
2. **Writes are not in the trait.** `DataStore::set` and `remove` are inherent methods. A source that
   cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than
   enforced". The model's own writes (`set_cell`, `clear_cell`) go to the owned store.
3. **The trait has no `Result`.** A source that can fail (a connection) decides at its own boundary
   what a failed read is, and `improvise-io` is where that decision belongs. AGENTS.md keeps
   `Option` and `Result` at the edges; `Option` is the one the trait uses, for "no such cell".
4. **`Select` is `spanned_values`' arguments.** A fold names a partial key, the covered members of
   each group or omitted category it spans, and whether only an exact-width cell counts. The
   covered members are decided in RAM from the categories and the view's hidden items, so a source
   is handed explicit item lists and never a predicate. A hidden item therefore reaches a source
   only as the absence of an item from a list.

## `Summary` and `Numbers`

A `Summary` is what a source reports about the values of a read: a `Numbers` (count, an exact sum,
the least and greatest non-NaN, in `total_cmp` order so `-0.0` is below `+0.0`) and whether the rest
(text, errors) agree on one value. It reduces exactly as `aggregate_cell_values` did: numbers by the
function, anything else by unanimity, else `Error("mixed")`, none at all absent. Three properties
carry the design:

- **It merges.** A read of a collapsed group, a total, or several contexts of a formula aggregate is
  the merge of the summaries of its parts, equal to the summary of the union. That is what lets an
  engine answer a pooled read without sending rows, and what lets `eval_agg` pool `Numbers` instead
  of concatenating `Vec<f64>`. The sum is `ExactSum`, so a merge is the same bits in any order.
- **It is order-free.** Unanimity needs the set of non-number values, not their order, so a source
  that has no insertion order loses nothing by it.
- **The reference is kept.** `tests::reference` is the old reduction, copied into the test module;
  `a_summary_reduces_as_the_values_did` compares every sequence of an eight-value pool, up to three
  long, with `Summary` over all five functions. `characterization_survives_the_move`: the existing
  evaluator tests (423 in the crate) are unchanged and pass.

What a real engine must supply is stated here because the trait cannot enforce it: a `Numbers` whose
sum is exact. SQLite's `SUM` is a float sum, so an engine that cannot return an exact accumulator
either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which
`context/code/improvise-incremental.md` ("Step 6") calls a user-visible change.

## `Model::source()` and `Model::over(source)`

`Model::source()` returns the `&dyn CellSource` that the evaluator, the stems of `recompute_formulas`,
the member records of an aggregate over an expression and the records of `GridLayout` read through.
`Model::over(source)` points it at a `SharedSource` (an `Arc<dyn CellSource + Send + Sync>`) instead of
the owned `DataStore`. That is the whole evaluator hook (improvise-rnak, bw4p.14's prerequisite):
there is still one evaluator, `Model`'s, and it computes every formula value, over a source whose
cells it never owns.

1. **A model over a source is read-only, in every build.** `set_cell`, `clear_cell`, `remove_category` and
   `remove_item` `assert!` that no source is set (a `debug_assert` first shipped and was lost in release:
   the write went to the owned store, which nothing reads). A caller who holds a source writes to the
   source (ADR 003: the trait has no writes). `Model::over` drops the model's own cells, so no stale copy
   is left for `get_cell`, persistence or the harness to read. `data` is `pub`, so a caller can still write
   it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed)
   are refused too, because the source's cells would still carry them.
2. **A shared source changes behind its readers by interior mutability of its own.** The model holds
   an `Arc`, so a database connection or a test double behind a lock is the same object the
   backend feeds from.
3. **What still reads the owned store**: `get_cell` (a production caller: `improvise-command/src/ui/effect.rs`,
   the `old` value of a rename), `Model::data` (`pub`), persistence, the import wizard and the effects
   in `improvise-command`. They are the facade of a document that owns its cells; a model over a source
   has none of those to offer, and routing them would change `get_cell`'s `&CellValue` return for 129
   call sites. `Eager::occupancy`, `Shapes::of` and `Stems::build` in `improvise-incremental` are
   routed by the PRs that need them.
4. **`Debug` shows only whether a source is set.** A source is not necessarily printable, and two
   models over different sources are not told apart by it.

## `TableSource`

A scanned table of rows, behind `cfg(any(test, feature = "test-support"))`. It is the second
implementation, written as a query engine with no index would answer, sharing only `Summary` with
`DataStore`. `the_store_and_a_scanned_table_answer_every_select_alike` asks both every select over a
fixture (every subset of every stored key as a base, with no span, one span of each of six item
lists, and two), at both widths and for all five functions: over 5,000 triples.

## Building a `Summary` from an engine's answer

`Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest)?, Agreement)?`. `Agreement`
is what the non-number values hold: `None`, `One(v)` or `Mixed`. Both constructors return a `Result`
and **refuse** a state no sequence of values could produce (extremes with no numbers, a NaN extreme,
least above greatest, one extreme without the other, a number among the "others"), because repairing
it would hide an engine's bug.