improvise/site/docs/adr/003-live-backed-models.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>ADR 003 — Live-backed models — improvise docs</title>
<link rel="stylesheet" href="../style.css" />
</head>
<body>
<div class="shell">
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html" class="here">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
<main>
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/adr/003-live-backed-models.md">context/adr/003-live-backed-models.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#adr-003-a-model-may-be-backed-by-a-live-query-source">ADR 003 — A model may be backed by a live query source</a></li><li class="l2"><a href="#context">Context</a></li><li class="l2"><a href="#what-the-model-is-today">What the model is today</a></li><li class="l2"><a href="#decision">Decision</a></li><li class="l2"><a href="#the-consequence-nobody-asked-for-and-it-is-the-important-one">The consequence nobody asked for, and it is the important one</a></li><li class="l2"><a href="#open-questions-in-the-order-they-will-bite">Open questions, in the order they will bite</a></li><li class="l2"><a href="#consequences">Consequences</a></li></ul></details><h1 id="adr-003-a-model-may-be-backed-by-a-live-query-source">ADR 003 — A model may be backed by a live query source</h1>
<p><strong>Status:</strong> accepted, <strong>not a priority</strong> (2026-09-19) · <strong>Decided by:</strong> the user · <strong>Supersedes:</strong> the read-only framing in <a class="tid" href="../tickets.html#improvise-rucf" title="Read arbitrary SQLite databases as a data source">improvise-rucf</a></p>
<blockquote><p>The goal is to have <strong>the option</strong> of essentially using improvise as a <strong>DB client</strong>. Direction, not schedule: nothing here outranks the browser MVP (<a class="tid" href="../tickets.html#improvise-1ey" title="Browser frontend MVP: end-to-end working demo">improvise-1ey</a>), and the tickets are filed low. This is written down now so the option stays open — several decisions being made this week could foreclose it by accident.</p></blockquote>
<h2 id="context"><a class="anchor" href="#context">#</a>Context</h2>
<p>improvise must handle <strong>arbitrary SQLite databases as models</strong> — a user points it at a database they already have and pivots over it.</p>
<p>Two readings were offered and the larger one was chosen. <strong>Import-once</strong> — read a query into a <code>Model</code> in RAM, exactly as CSV import does — is days of work behind an existing seam (<code>csv_parser</code> produces <code>Vec&lt;Value&gt;</code>, <code>analyze_records</code> classifies it). It was rejected because it makes the model a <em>snapshot</em>: source changes are invisible until re-import, and the data must fit in memory.</p>
<p><strong>Live-backed</strong> was chosen: the model is a view over the source, and queries go to it as the user pivots.</p>
<p>This is not <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>. That ticket proposed SQLite as improvise&#39;s <em>own persistence format</em> and was declined on measurements — a second implementation of the pest grammar, 22% larger than <code>.improv</code> and 4.7× gzipped. Reading someone else&#39;s database has no round-trip obligation and no grammar duplication. What does carry over is the dependency constraint: <code>improvise-io</code> is in the wasm bundle, and <code>libsqlite3-sys</code> does not build for <code>wasm32-unknown-unknown</code> under this repo&#39;s nix cc-wrapper.</p>
<h2 id="what-the-model-is-today"><a class="anchor" href="#what-the-model-is-today">#</a>What the model is today</h2>
<p><code>Model</code> holds <code>categories</code>, a <code>DataStore</code>, <code>formulas</code>, and <code>views</code>. <code>DataStore</code> interns every category and item name into a <code>SymbolTable</code>, stores <code>IndexMap&lt;InternedKey, CellValue&gt;</code>, and maintains a secondary index <code>(Symbol, Symbol) → {InternedKey}</code> for hot-path lookups. A <code>Workbook</code> loads <strong>entire</strong> — there is no partial-load path anywhere in the tree, a fact verified while declining <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>.</p>
<p>So this decision changes what <code>Model</code> <em>is</em>, not merely where its bytes come from.</p>
<h2 id="decision"><a class="anchor" href="#decision">#</a>Decision</h2>
<p><strong><code>Model</code> becomes a facade over a data source, of which the current in-RAM <code>DataStore</code> is one implementation and a SQL query engine is another.</strong></p>
<p>Three things make this more tractable than it first looks, and they should shape the design rather than be discovered late.</p>
<p><strong>1. Aggregation is already the interface, and it is already SQL.</strong> <code>apply_agg</code> computes <code>Sum | Avg | Min | Max | Count</code> over a matching cell set — those are exactly SQL&#39;s aggregates. The pushdown is not a translation layer to be invented; it is the operation the model already performs, expressed in the source&#39;s own language. <code>GridLayout</code> asks for a cell&#39;s value, not for the rows behind it.</p>
<p><strong>2. Formula results are already separate from stored data.</strong> Evaluation is fixed-point (<code>recompute_formulas</code> iterates to stability, gotcha 9), and formula-derived values live in <code>Model::formula_cache</code>, not in <code>DataStore</code>. So the split is clean: <strong>raw data is queried, derived data stays in RAM.</strong> Formulas do not need to become SQL.</p>
<p><strong>3. Clients are already insulated.</strong> The projection architecture means a client sees <code>CellDisplay</code>s and never learns where a value came from — the server holds the <code>App</code> and therefore the connection. Live-backing and the browser MVP compose rather than compete, and a browser never needs SQLite at all.</p>
<h2 id="the-consequence-nobody-asked-for-and-it-is-the-important-one"><a class="anchor" href="#the-consequence-nobody-asked-for-and-it-is-the-important-one">#</a>The consequence nobody asked for, and it is the important one</h2>
<p><strong>A live-backed document splits into <em>our</em> configuration and <em>their</em> data</strong>, and <code>.improv</code> does not currently distinguish them. Today a <code>.improv</code> file holds categories, views, formulas <strong>and</strong> a <code>## Data</code> section. A live-backed model&#39;s categories and cell values are the source&#39;s; the views, formulas and axis assignments are improvise&#39;s.</p>
<p>So the format grows a <strong>source specification</strong> where a live model would otherwise have data — a connection, a query, and the column-to-field mapping. Categories become <em>derived</em> (<code>SELECT DISTINCT</code>) rather than declared. That is a grammar change, and <code>improv.pest</code> is the single source of truth, so it comes first (design-principles §3).</p>
<p>Read through the DB-client framing, that split stops being a complication and becomes the <strong>natural shape of the artifact</strong>: a live-backed <code>.improv</code> is <em>a saved query plus a pivot configuration</em> — which is precisely what a saved report is in any BI tool. The document stops being &quot;my data&quot; and becomes &quot;my question&quot;, which is portable between databases in a way a data file never is, and shareable with someone who has the same source and not the same rows.</p>
<h2 id="open-questions-in-the-order-they-will-bite"><a class="anchor" href="#open-questions-in-the-order-they-will-bite">#</a>Open questions, in the order they will bite</h2>
<p><strong>Writes — DECIDED: a live-backed model is read-only.</strong> Not a limitation accepted reluctantly: the user&#39;s framing is that for many use cases <strong>editing capabilities would be a downside</strong>. Pointing a pivot tool at a production database and being unable to write to it is a <em>feature</em>. So there is no write-through, no overlay, and no reconciliation.</p>
<p>Two things fall out, and both are simplifications rather than costs.</p>
<p><strong>It dissolves the ADR 001 tension instead of requiring an amendment.</strong> ADR 001 governs concurrent <em>edits</em>; a read-only model has none, so the basis check simply does not apply to it. The guarantee is not voided, it is <strong>scoped to documents improvise owns</strong> — which is what it always meant. What remains is a smaller and different question: an external writer changes what clients <em>see</em>, so live-backed projections need a freshness story (polled, triggered, or explicitly stale-until-refreshed). That is projection freshness, not conflict resolution.</p>
<p><strong>The machinery to express it already exists, built for something else.</strong> ADR 002 makes <code>SetCell</code> take a <code>CellId::Record</code> and nothing else, so a live-backed model that projects no <code>Record</code> ids makes a write <strong>unconstructible by the client</strong> — no runtime check, no error path, no &quot;this model is read-only&quot; dialog. The settability distinction built for aggregates carries read-only sources for free.</p>
<p><strong>One design consequence for the source trait:</strong> writes should not be <em>in</em> the trait that a query source implements. If they were, a read-only source would have to fail at runtime for a method it structurally cannot serve. Keep reads (and aggregates) in the trait every source implements, and let writing be a capability only the owned in-RAM implementation has — so &quot;read-only&quot; is unrepresentable rather than enforced, the same discipline as <code>CellId</code>.</p>
<p><strong>Interning and identity.</strong> <code>DataStore</code> interns names and ADR 002 hashes the coordinate tuple. Both still work — coordinates come from the query — but the <code>RecordHash → record</code> index that resolves a <code>CellId</code> becomes a cache with an invalidation story rather than a structure maintained alongside writes.</p>
<p><strong>Which rows.</strong> A table name or a user-supplied <code>SELECT</code>. A query is strictly more useful (joins, filters, computed columns) and no harder to consume; a bare table name is sugar over it.</p>
<p><strong>Type mapping.</strong> SQLite is dynamically typed per value, but columns carry declared types, which is <em>better</em> input than CSV&#39;s string sampling: <code>detect_date_format</code> currently guesses a date by pattern-matching text. Dates may be TEXT, INTEGER (epoch) or REAL (julian) — detect, do not assume. NULL is <strong>absence</strong>, not the string <code>&quot;null&quot;</code> (<a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item &quot;null&quot; on import">improvise-ych</a>).</p>
<p><strong>Where the boundary goes.</strong> <code>improvise-core</code> must not gain an IO dependency (<code>forbidden(improvise_core, io)</code>), so a query-backed source cannot simply be a <code>DataStore</code> variant inside core. The trait belongs in core and the SQLite implementation in <code>improvise-io</code>, feature-gated and native-only so the wasm gate keeps passing.</p>
<h2 id="consequences"><a class="anchor" href="#consequences">#</a>Consequences</h2>
<p><strong>Good.</strong> Data larger than memory. A source that stays current. Aggregation pushed to an engine built for it. And the client/server split already built means none of this reaches a browser client.</p>
<p><strong>Bad.</strong> It changes <code>Model</code>, which is the centre of the codebase — <code>GridLayout</code>, evaluation, persistence and the import pipeline all touch it. Latency enters a path that is currently a <code>HashMap</code> lookup, so the viewport-bounded projection stops being an optimisation and becomes load-bearing. And the <code>.improv</code> grammar grows a concept.</p>
<p><strong>Worth doing first regardless of the eventual shape:</strong> the source trait and its in-RAM implementation, so that today&#39;s <code>DataStore</code> becomes <em>an</em> implementation rather than <em>the</em> implementation. That is a refactor with no behaviour change, <code>characterization_survives_the_move</code> applies, and it is the seam everything else needs — including, if it is ever wanted, the import-once path this ADR rejected as the <em>only</em> answer.</p>
</main>
</div>
</body>
</html>