improvise/context/adr/003-live-backed-models.md

ADR 003 — A model may be backed by a live query source

Status: accepted, not a priority (2026-09-19) · Decided by: the user · Supersedes: the read-only framing in improvise-rucf

The goal is to have the option of essentially using improvise as a DB client. Direction, not schedule: nothing here outranks the browser MVP (improvise-1ey), and the tickets are filed low. This is written down now so the option stays open — several decisions being made this week could foreclose it by accident.

Context

improvise must handle arbitrary SQLite databases as models — a user points it at a database they already have and pivots over it.

Two readings were offered and the larger one was chosen. Import-once — read a query into a Model in RAM, exactly as CSV import does — is days of work behind an existing seam (csv_parser produces Vec<Value>, analyze_records classifies it). It was rejected because it makes the model a snapshot: source changes are invisible until re-import, and the data must fit in memory.

Live-backed was chosen: the model is a view over the source, and queries go to it as the user pivots.

This is not improvise-9ix. That ticket proposed SQLite as improvise's own persistence format and was declined on measurements — a second implementation of the pest grammar, 22% larger than .improv and 4.7× gzipped. Reading someone else's database has no round-trip obligation and no grammar duplication. What does carry over is the dependency constraint: improvise-io is in the wasm bundle, and libsqlite3-sys does not build for wasm32-unknown-unknown under this repo's nix cc-wrapper.

What the model is today

Model holds categories, a DataStore, formulas, and views. DataStore interns every category and item name into a SymbolTable, stores IndexMap<InternedKey, CellValue>, and maintains a secondary index (Symbol, Symbol) → {InternedKey} for hot-path lookups. A Workbook loads entire — there is no partial-load path anywhere in the tree, a fact verified while declining improvise-9ix.

So this decision changes what Model is, not merely where its bytes come from.

Decision

Model becomes a facade over a data source, of which the current in-RAM DataStore is one implementation and a SQL query engine is another.

Three things make this more tractable than it first looks, and they should shape the design rather than be discovered late.

1. Aggregation is already the interface, and it is already SQL. apply_agg computes Sum | Avg | Min | Max | Count over a matching cell set — those are exactly SQL's aggregates. The pushdown is not a translation layer to be invented; it is the operation the model already performs, expressed in the source's own language. GridLayout asks for a cell's value, not for the rows behind it.

2. Formula results are already separate from stored data. Evaluation is fixed-point (recompute_formulas iterates to stability, gotcha 9), and formula-derived values live in Model::formula_cache, not in DataStore. So the split is clean: raw data is queried, derived data stays in RAM. Formulas do not need to become SQL.

3. Clients are already insulated. The projection architecture means a client sees CellDisplays and never learns where a value came from — the server holds the App and therefore the connection. Live-backing and the browser MVP compose rather than compete, and a browser never needs SQLite at all.

The consequence nobody asked for, and it is the important one

A live-backed document splits into our configuration and their data, and .improv does not currently distinguish them. Today a .improv file holds categories, views, formulas and a ## Data section. A live-backed model's categories and cell values are the source's; the views, formulas and axis assignments are improvise's.

So the format grows a source specification where a live model would otherwise have data — a connection, a query, and the column-to-field mapping. Categories become derived (SELECT DISTINCT) rather than declared. That is a grammar change, and improv.pest is the single source of truth, so it comes first (design-principles §3).

Read through the DB-client framing, that split stops being a complication and becomes the natural shape of the artifact: a live-backed .improv is a saved query plus a pivot configuration — which is precisely what a saved report is in any BI tool. The document stops being "my data" and becomes "my question", which is portable between databases in a way a data file never is, and shareable with someone who has the same source and not the same rows.

Open questions, in the order they will bite

Writes — DECIDED: a live-backed model is read-only. Not a limitation accepted reluctantly: the user's framing is that for many use cases editing capabilities would be a downside. Pointing a pivot tool at a production database and being unable to write to it is a feature. So there is no write-through, no overlay, and no reconciliation.

Two things fall out, and both are simplifications rather than costs.

It dissolves the ADR 001 tension instead of requiring an amendment. ADR 001 governs concurrent edits; a read-only model has none, so the basis check simply does not apply to it. The guarantee is not voided, it is scoped to documents improvise owns — which is what it always meant. What remains is a smaller and different question: an external writer changes what clients see, so live-backed projections need a freshness story (polled, triggered, or explicitly stale-until-refreshed). That is projection freshness, not conflict resolution.

The machinery to express it already exists, built for something else. ADR 002 makes SetCell take a CellId::Record and nothing else, so a live-backed model that projects no Record ids makes a write unconstructible by the client — no runtime check, no error path, no "this model is read-only" dialog. The settability distinction built for aggregates carries read-only sources for free.

One design consequence for the source trait: writes should not be in the trait that a query source implements. If they were, a read-only source would have to fail at runtime for a method it structurally cannot serve. Keep reads (and aggregates) in the trait every source implements, and let writing be a capability only the owned in-RAM implementation has — so "read-only" is unrepresentable rather than enforced, the same discipline as CellId.

Interning and identity. DataStore interns names and ADR 002 hashes the coordinate tuple. Both still work — coordinates come from the query — but the RecordHash → record index that resolves a CellId becomes a cache with an invalidation story rather than a structure maintained alongside writes.

Which rows. A table name or a user-supplied SELECT. A query is strictly more useful (joins, filters, computed columns) and no harder to consume; a bare table name is sugar over it.

Type mapping. SQLite is dynamically typed per value, but columns carry declared types, which is better input than CSV's string sampling: detect_date_format currently guesses a date by pattern-matching text. Dates may be TEXT, INTEGER (epoch) or REAL (julian) — detect, do not assume. NULL is absence, not the string "null" (improvise-ych).

Where the boundary goes. improvise-core must not gain an IO dependency (forbidden(improvise_core, io)), so a query-backed source cannot simply be a DataStore variant inside core. The trait belongs in core and the SQLite implementation in improvise-io, feature-gated and native-only so the wasm gate keeps passing.

Consequences

Good. Data larger than memory. A source that stays current. Aggregation pushed to an engine built for it. And the client/server split already built means none of this reaches a browser client.

Bad. It changes Model, which is the centre of the codebase — GridLayout, evaluation, persistence and the import pipeline all touch it. Latency enters a path that is currently a HashMap lookup, so the viewport-bounded projection stops being an optimisation and becomes load-bearing. And the .improv grammar grows a concept.

Worth doing first regardless of the eventual shape: the source trait and its in-RAM implementation, so that today's DataStore becomes an implementation rather than the implementation. That is a refactor with no behaviour change, characterization_survives_the_move applies, and it is the seam everything else needs — including, if it is ever wanted, the import-once path this ADR rejected as the only answer.