diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 15f28db2..29ed4914 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -129,6 +129,7 @@ {"_type":"issue","id":"improvise-pby","title":"2.2 Write the README","description":"Replace existing README.md with new one under 250 lines. Sections in order: Title, one-sentence pitch, inline demo GIF, why this exists (Lotus Improv reference), quick start, key bindings, installation (Nix/crates.io/prebuilt - no Homebrew), codebase overview, expectations disclaimer, license. No badges, TOC, or contributing guide.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:32Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:02:52Z","close_reason":"README written with all 10 required sections, committed. License updated to Apache-2.0 per user request.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ihv","title":"2.1 Add Nix tooling for asciinema, VHS, and cargo-dist","description":"Modify flake.nix to add pkgs.asciinema and pkgs.vhs to dev shell nativeBuildInputs. Add cargo-dist if packaged in nixpkgs. Add runtime deps for VHS (ttyd, ffmpeg) if needed. Verify tools work via nix develop --command. Add nix run app or shell script for demo recording workflow.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:30Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:30:19Z","close_reason":"asciinema, vhs, cargo-dist added to flake.nix dev shell and verified. cargo-dist wrapper enables 'cargo dist' subcommand. scripts/record-demo.sh added.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-yk4","title":"Phase 2: README and demo artifacts","description":"The main launch work. README is 80% of the launch, demo artifacts are 20%. Covers Nix tooling for asciinema/VHS, writing the README, creating demo GIF and asciinema casts, and verifying all artifacts.","status":"closed","priority":2,"issue_type":"feature","assignee":"spot","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:49Z","created_by":"Edward Langley","updated_at":"2026-04-14T07:47:43Z","closed_at":"2026-04-14T07:47:43Z","close_reason":"All 5 subtasks closed; artifacts verified present and tracked in git (README.md, docs/demo.gif, docs/demo.tape, docs/casts/*.cast).","dependencies":[{"issue_id":"improvise-yk4","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T23:37:37Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-ihv","type":"blocks","created_at":"2026-04-08T23:37:34Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-odx","type":"blocks","created_at":"2026-04-08T23:37:35Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-pby","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rucf","title":"Read arbitrary SQLite databases as a data source","description":"USER REQUIREMENT (2026-09-19): improvise must be able to handle ARBITRARY SQLite databases as models -- someone points it at a database they already have and pivots over it.\n\nTHIS IS NOT improvise-9ix AND ITS DECLINE DOES NOT BEAR ON IT. 9ix proposed SQLite as improvise OWN PERSISTENCE FORMAT -- writing our model semantics into a second container -- and was declined because that is a second implementation of the pest grammar with a silent-drift failure mode, and because it measured 22% larger than .improv and 4.7x larger gzipped. Reading SOMEONE ELSE database has none of those properties: no round-trip obligation, no grammar duplication, and the value is real (pivot over data you already have). The one thing that DOES carry over is the dependency constraint, below.\n\nTHE SEAM IS CLEAN AND ALREADY EXISTS. import/csv_parser.rs produces Vec\u003cserde_json::Value\u003e (rows as objects) and import/analyzer.rs::analyze_records(\u0026[Value]) -\u003e Vec\u003cFieldProposal\u003e does the Category/Measure/Time/Skip classification from there. A SQLite source slots in beside csv_parser.rs and needs only to produce Vec\u003cValue\u003e. ImportPipeline and the wizard then work unchanged.\n\nA GENUINE IMPROVEMENT OVER CSV, worth exploiting rather than ignoring: the analyzer currently infers field kinds from STRING SAMPLES -- detect_date_format guesses a date by pattern-matching text. A SQLite column has a DECLARED TYPE. So the classifier should take typed input where it has it and fall back to sampling only where it does not (SQLite being dynamically typed per-value, both cases are real). That is a better analyzer, not just a new source.\n\nDESIGN QUESTIONS TO SETTLE, none of them hard but all of them real:\n * WHICH ROWS? A table name, or a user-supplied SELECT? A query is strictly more useful (joins, filters, computed columns) and no harder to consume -- a result set is a result set. Prefer it, and let a bare table name be sugar.\n * MULTIPLE TABLES: answered by the above if the input is a query.\n * TYPE MAPPING: INTEGER/REAL -\u003e measure candidates, TEXT -\u003e category candidates, NULL -\u003e the missing path (see improvise-ych: null is ABSENCE, not the string \"null\"), BLOB -\u003e skip. Dates in SQLite may be TEXT, INTEGER (epoch) or REAL (julian) -- detect, do not assume.\n * READ-ONLY for now. Writing edits back to the source database is a separate and much larger question; say so in the code rather than leaving it implied.\n\nTHE CONSTRAINT THAT CARRIES OVER FROM 9ix: improvise-io IS IN THE WASM BUNDLE (.github/workflows/ci.yml gates it), and libsqlite3-sys C build fails for wasm32-unknown-unknown under this repo nix cc-wrapper. So this must be FEATURE-GATED AND NATIVE-ONLY, with the wasm job still passing untouched -- or the browser path uses a different SQLite (sqlite-wasm-rs / OPFS), which is a second decision and should be its own ticket. Verify the gate rather than assuming it.\n\nOPEN, and it changes the size of this ticket by an order of magnitude -- see the note being added: whether this is IMPORT-ONCE (read the database, build a Model in RAM) or a LIVE-BACKED model (queries go to SQLite as the user pivots).","notes":"\n--- SUPERSEDED IN FRAMING by ADR 003 (user, 2026-09-19) ---\nThe user chose LIVE-BACKED over import-once, and the goal is stated as: have\nTHE OPTION of essentially using improvise as a DB CLIENT. NOT A PRIORITY --\nnothing here outranks improvise-1ey. Re-prioritised P2 -\u003e P3.\n\nSo this ticket read-only import framing is not the target shape. See\ncontext/adr/003-live-backed-models.md. The import-once seam it described\n(csv_parser -\u003e Vec\u003cValue\u003e -\u003e analyze_records) is still accurate and still\nuseful, but as ONE implementation of a source rather than as the answer.\n\nWHAT SURVIVES UNCHANGED and is worth keeping from this ticket: the type-mapping\nnotes (SQLite columns carry declared types, which is BETTER input than CSV\nstring sampling -- detect_date_format currently guesses a date by pattern-\nmatching text; dates may be TEXT, INTEGER epoch or REAL julian, so detect\nrather than assume; NULL is ABSENCE not the string \"null\", per improvise-ych),\nand the wasm constraint (improvise-io is in the bundle, libsqlite3-sys does not\nbuild for wasm32 under this repo nix cc-wrapper, so any SQLite dependency is\nfeature-gated and native-only).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:14:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:16:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i9lt","title":"View::axis_of panics for a category the view has not seen","description":"Noticed during improvise-35e (2026-09-19), reported not fixed per the mention-do-not-fix rule for pre-existing issues.\n\nView::axis_of(cat) panics for a category the view has never seen, and View::tileable_cats admits exactly such categories -- it filters on try_axis_of(c).is_none_or(...). So the pairing is a live panic waiting for the right view state.\n\nThe old TileBar called axis_of over that list. The migrated widget cannot panic there any more, because tile_entries uses try_axis_of(...).unwrap_or(Axis::None) -- so the reachable path is gone and no failing test was written, since the old path no longer exists to fail.\n\nWorth a guard anyway: either make axis_of total (returning Axis::None like tile_entries does) or make it private and force callers through try_axis_of. A panicking accessor over a list built to include the panicking case is a trap for the next caller, and the next caller will not have the migration to protect them.","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T23:40:38Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:40:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-j31x","title":"Server hosts several documents: a registry above Sessions","description":"DEFERRED by the user (2026-09-19) -- recorded so nothing forecloses it, not to be built now.\n\nimprovise is a multi-document app and the server should be able to host several separate models. The shape:\n server -\u003e many documents\n document -\u003e one App + one Sessions + many ClientSessions\nThat is NOT one App per session (what improvise-q08 ticket text says, written before the session model landed) and NOT one App total.\n\nNOTHING THAT HAS LANDED NEEDS TO CHANGE. Sessions does not own the App -- connect(\u0026mut self, app: \u0026App, ..), apply(\u0026mut self, app: \u0026mut App, ..) and project_all(\u0026mut self, app: \u0026App, ..) all take it as a parameter. So Sessions is already document-agnostic and a Sessions instance is implicitly bound to one document generation sequence, which is right: document_generation lives on the App. Multi-document is a REGISTRY ABOVE Sessions mapping a document id to its (App, Sessions) pair.\n\nWHY IT MATTERS BEYOND TIDINESS: sessions editing independent models are trivially non-conflicting, so ADR 001 concurrency design scopes per document FOR FREE -- the basis version tiers live on the App and each document has its own. A server that shared one App across documents would throw that away.\n\nWHAT TO AVOID IN THE MEANTIME: improvise-q08 must not hard-code a single App into a static or a lazily-initialised singleton. The document should be a PARAMETER even while there is only one, and a connection should be attachable to a document even if the MVP answers that from a CLI argument naming one .improv file.\n\nScope when picked up: the registry, document lifecycle (open/close/last-session-leaves), and how a client names the document it wants. Auth and access control are explicitly NOT in scope here -- file them separately if wanted.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:32:31Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:32:31Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-2wi8","title":"ADR-001 e: a forced overwrite is visible to the client it overwrote","description":"ADR 001 layer 1, step e. Depends on step c.\n\nWithout this, force is silent data loss with a confirmation step in front of it. The overwritten client edit was already committed, so this is a NOTIFICATION rather than a conflict -- but it must exist, or \"someone else won\" is indistinguishable from \"my edit never happened\".\n\nDesign note: the notification is about the loser OWN prior edit, so it is per-client and therefore UI -- the model is shared, the UI is not. Do not broadcast it as document state.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:27:34Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:27:34Z","dependencies":[{"issue_id":"improvise-2wi8","depends_on_id":"improvise-3al1","type":"blocks","created_at":"2026-09-19T23:27:47Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} @@ -206,6 +207,7 @@ {"_type":"issue","id":"improvise-l36","title":"3.2 Publish to crates.io","description":"After cargo publish --dry-run is clean and user confirms, run cargo publish. Verify crate appears at crates.io/crates/improvise and cargo install improvise works.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:58Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-l36","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T21:09:29Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-l36","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-11a","title":"3.1 Configure cargo dist","description":"Run cargo dist init targeting x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin. Skip Windows and musl. Commit generated .github/workflows/release.yml and Cargo.toml additions. Test with v0.1.0-rc1 tag, verify builds, delete rc tag.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:55Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-11T07:49:57Z","close_reason":"Closed","dependencies":[{"issue_id":"improvise-11a","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-0s6","title":"Phase 3: Distribution","description":"Configure cargo dist, publish to crates.io, tag v0.1.0 release. Produces prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:51Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-0s6","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T23:37:39Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-3tj","type":"blocks","created_at":"2026-04-08T23:37:41Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-l36","type":"blocks","created_at":"2026-04-08T23:37:40Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-mjy7","title":"Export a model as SQLite for interop (one-way, native-only)","description":"The salvageable part of improvise-9ix, which was declined as a PERSISTENCE format (see its close reason for the measurements).\n\nThe one claim with real standing is INTEROP: a user opening their model in sqlite3 or DB Browser to run their own queries. That is EXPORT, not persistence -- ONE-WAY, no round-trip obligation, and therefore none of the grammar-duplication risk that killed 9ix. A second FORMAT must track every future change to improv.pest forever; a second VIEW need not, because nothing loads it back.\n\nIt sits naturally beside export_csv/export_csv_to in crates/improvise-io/src/persistence/export.rs.\n\nTHE ONE REAL COST: the rusqlite dependency. cargo check --target wasm32-unknown-unknown fails in libsqlite3-sys C build under this repo nix cc-wrapper, so this must be NATIVE-ONLY and FEATURE-GATED, and the wasm CI job must keep passing untouched. Verify that rather than assume it -- the 9ix agent was careful to note the wasm failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating is about protecting the browser bundle, not about an impossibility.\n\nScope: choose a schema that is USEFUL TO QUERY rather than one that mirrors internal storage -- a reader wants cells joined to their coordinates, not an interned key blob. Do not add a reader.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:08:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:08:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-zavn","title":"Drill previews on hover: a bounded record keyset per aggregate","description":"DEFERRED by the user (2026-09-19) -- a good idea, recorded so the shape is not lost and so nobody builds the UNBOUNDED version by accident.\n\nTHE IDEA: show the underlying rows when the user hovers an aggregate cell, without a round trip.\n\nTHE TRAP THIS AVOIDS. Projecting the full keyset for every aggregate is untenable and the cost runs OPPOSITE to the value. A stored cell has fixed coordinates so it lands in exactly one displayed cell -- the keysets PARTITION rather than multiply -- but that still makes the payload scale with N (the data) while everything else in a projection scales with the VIEWPORT. Aggregation is precisely what makes a large document fit a screen, so shipping the keysets undoes that compression. Worked numbers for a 1000-cell viewport at u128: ~16KB at one record per cell (where the cell would be settable anyway and no keyset is needed), ~160KB at ten, and ~16MB over a million-record document -- to render a thousand cells. Per client, re-diffed as data changes. And the cheap cases are the ones where drilling barely matters.\n\nTHE SHAPE THAT WORKS: ship at most k record hashes per aggregate PLUS A COUNT, so the payload is viewport-proportional again, and fetch the full set on demand when the user actually drills. k is a product decision -- enough rows to make a hover useful, e.g. 3 to 5.\n\nPREREQUISITE: ADR 002 (context/adr/002-naming-a-cell.md). An aggregate already carries CellId::Aggregate(AggregateHash), so drill-on-demand works without this ticket; this is purely the latency optimisation for hover. Do not conflate the two -- drilling must stay correct without any preview.\n\nNote the preview hashes are RECORD hashes, so they resolve through the same DataStore index and obey the same staleness rules: a preview row whose record has since gone simply fails to resolve.","notes":"\n--- BLOOM FILTER IDEA (user, 2026-09-19): right tolerance, different question ---\nProposal was a Bloom filter instead of an explicit keyset, on the grounds that\nfalse positives are fine given a deterministic aggregate id to ask\nauthoritatively. The tolerance argument is right; the direction needs\ncorrecting.\n\nA BLOOM FILTER CANNOT ENUMERATE. A hover preview needs ROWS -- identities and\nvalues -- while a filter answers \"is X probably in this set?\" for an X you\nalready hold. The client holds aggregates, not the underlying records, so it has\nno candidates to test. So this does not serve previews directly.\n\nWHERE IT FITS INSTEAD, and it fits well: the REVERSE question, \"which visible\naggregates does record R feed?\" A client staging an edit to R tests R against\neach visible cell filter and marks the hits pending -- greyed, spinner -- with\nno round trip. A false positive marks one extra cell pending; it never misses\none. That asymmetry is exactly gotcha 35 rule: over-reporting costs work,\nunder-reporting is the bug, which is why Dirty::of is a deliberate\nover-approximation. Same shape.\n\nSIZE CORRECTION THAT MAKES IT CHEAP: a conventionally-sized filter still scales\nwith N -- roughly 9.6 bits per element at 1% FP, so ~1.2KB for a cell\naggregating 1000 records, and the payload again scales with the DATA. FIX THE\nWIDTH instead: a fixed 64- or 128-bit filter per cell is viewport-proportional\nby construction. Small aggregates are precise; large ones SATURATE and answer\n\"maybe\" to everything, degrading to mark-everything -- the safe direction, and\nthe same failure the design already tolerates.\n\nThe deterministic aggregate id is what makes the sloppiness safe: the\nauthoritative answer is one Drill(AggregateHash) away, so the filter is never\nload-bearing for correctness. Same structure as ADR 001 advisory reservations,\nwhere correctness never rests on the advisory layer.\n\nSO THIS TICKET NOW HOLDS TWO SEPARABLE IDEAS -- split them when picked up:\n (a) hover previews: needs a BOUNDED EXPLICIT keyset (k hashes + a count),\n because a preview needs rows;\n (b) optimistic edit feedback: a FIXED-WIDTH Bloom filter per aggregate,\n answering which visible cells a pending record edit affects.\n(b) belongs with ADR 001 layer 2 rather than here.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:59:53Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:00:46Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ut7p","title":"improvise-gp fit_from_gen0 is a third copy of the GA loop","description":"Noted during improvise-0ai (2026-09-19), deliberately not changed. The generation loop now lives in gp_tree::fit::fit. improvise-gp test helper fit_from_gen0 is still its own loop, kept because it also returns the GENERATION-0 best, which the library fit does not expose -- and that is exactly what breeding_improves_on_generation_zero needs in order to prove the search moved rather than merely differed. So the duplication is load-bearing today. Two honest options: expose the generation-0 champion from gp_tree::fit (a fit result carrying the starting cost, or fit_with_progress) and rewrite the helper onto it, or leave it and document why. Prefer the first -- did the search improve on where it started is a question any caller of a GA wants answerable, not only a test.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T21:23:05Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:40:24Z","closed_at":"2026-09-19T23:40:24Z","close_reason":"Took the preferred option. gp_tree::fit gains Fit { tree, cost, start, start_cost } and fit_detailed; fit is now fit_detailed(..).tree. start is the best member of GENERATION 0 of the restart that produced tree -- the stronger claim, and the one improvise-gp needed. improvise-gp fit_from_gen0 is gone, replaced by seeded_fit over fit_detailed, so the GA loop has one home.\n\nLoop shape preserved deliberately: generation i is still what is scored at the top of iteration i, the sort is still stable so member 0 wins its own ties and generation 0 champion equals the seed ancestor whenever nothing beat it -- that is what makes the restructure behaviour-preserving rather than merely plausible. breed runs the same number of times with the same arguments; the extra scoring pass draws nothing from the RNG.\n\nREPRODUCIBILITY EVIDENCE RE-ESTABLISHED, not assumed: all six known-answer vectors pass unchanged and the nine-scenario fit_csv harness is byte-identical including model.json, so every recorded seed still names the same tree. Two faults injected, both fired.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-0vh1","title":"number_text emits grammar-rejected 'inf' and panics on NaN","description":"From the wave-1 gp-tree review (2026-09-19). Both are guarded by documented preconditions, hence P4.\n\nimprovise-gp/src/lib.rs:124. number_text(f32::INFINITY): at p=0 it yields \"inf\", and \"inf\".parse::\u003cf64\u003e() succeeds, so the round-trip loop ACCEPTS it and emits 'inf' -- which the formula grammar rejects (number has no such form). fitted_formula then returns Err, which is fine, but a bare formula_text caller gets silently unparseable text. number_text(f32::NAN) never matches (NaN != NaN) and falls through to unreachable!(), i.e. panics.\n\nAlso a doc imprecision worth correcting while there (lib.rs:22-25 and 114-123): the comment says it emits 'the exact decimal value of the f32'. It emits the shortest decimal that ROUND-TRIPS to the same f64 -- which is exactly what the routing argument needs, and is a weaker and more accurate claim. Verified: over 20,000 random f32 bit patterns plus the extremes, every value found a width inside 0..=149, worst observed width 60, none emitted e/E/inf/NaN, all parsed and round-tripped.","notes":"\n--- FIXED, AWAITING MERGE (2026-09-19) ---\nThe fix is on branch wave-1-review-fixes and is in PR #9\n(https://github.com/fiddlerwoaroof/improvise/pull/9), which is open pending\nCopilot review. Moved to in_progress rather than left open so nobody picks\nthis up again -- see the agent-issue-drift-pattern memory. Close it when #9\nmerges, not before.","status":"closed","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T20:43:13Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:50:38Z","started_at":"2026-09-19T21:15:05Z","closed_at":"2026-09-19T21:50:38Z","close_reason":"Merged to main in PR #9 as e1acac691d70992b5dfcce9f11ce2c2b50fa1c29 (2026-09-19). Verified a real merge commit with both parents, not a squash, so all 18 branch commits are reachable on main per AGENTS.md's preference for merges. Copilot's second review on that PR: approval recommended, findings none. All eight non-skipped CI checks green, including claude-review. Synced to origin and gitea.","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/context/adr/003-live-backed-models.md b/context/adr/003-live-backed-models.md new file mode 100644 index 00000000..4e15792b --- /dev/null +++ b/context/adr/003-live-backed-models.md @@ -0,0 +1,133 @@ +# ADR 003 — A model may be backed by a live query source + +**Status:** accepted, **not a priority** (2026-09-19) · **Decided by:** the user · +**Supersedes:** the read-only framing in improvise-rucf + +> The goal is to have **the option** of essentially using improvise as a **DB client**. +> Direction, not schedule: nothing here outranks the browser MVP (improvise-1ey), and the +> tickets are filed low. This is written down now so the option stays open — several +> decisions being made this week could foreclose it by accident. + +## Context + +improvise must handle **arbitrary SQLite databases as models** — a user points it at a +database they already have and pivots over it. + +Two readings were offered and the larger one was chosen. **Import-once** — read a query +into a `Model` in RAM, exactly as CSV import does — is days of work behind an existing +seam (`csv_parser` produces `Vec`, `analyze_records` classifies it). It was +rejected because it makes the model a *snapshot*: source changes are invisible until +re-import, and the data must fit in memory. + +**Live-backed** was chosen: the model is a view over the source, and queries go to it as +the user pivots. + +This is not improvise-9ix. That ticket proposed SQLite as improvise's *own persistence +format* and was declined on measurements — a second implementation of the pest grammar, +22% larger than `.improv` and 4.7× gzipped. Reading someone else's database has no +round-trip obligation and no grammar duplication. What does carry over is the dependency +constraint: `improvise-io` is in the wasm bundle, and `libsqlite3-sys` does not build for +`wasm32-unknown-unknown` under this repo's nix cc-wrapper. + +## What the model is today + +`Model` holds `categories`, a `DataStore`, `formulas`, and `views`. `DataStore` interns +every category and item name into a `SymbolTable`, stores `IndexMap`, +and maintains a secondary index `(Symbol, Symbol) → {InternedKey}` for hot-path lookups. +A `Workbook` loads **entire** — there is no partial-load path anywhere in the tree, a fact +verified while declining improvise-9ix. + +So this decision changes what `Model` *is*, not merely where its bytes come from. + +## Decision + +**`Model` becomes a facade over a data source, of which the current in-RAM `DataStore` is +one implementation and a SQL query engine is another.** + +Three things make this more tractable than it first looks, and they should shape the +design rather than be discovered late. + +**1. Aggregation is already the interface, and it is already SQL.** `apply_agg` computes +`Sum | Avg | Min | Max | Count` over a matching cell set — those are exactly SQL's +aggregates. The pushdown is not a translation layer to be invented; it is the operation +the model already performs, expressed in the source's own language. `GridLayout` asks for +a cell's value, not for the rows behind it. + +**2. Formula results are already separate from stored data.** Evaluation is fixed-point +(`recompute_formulas` iterates to stability, gotcha 9), and formula-derived values live in +`Model::formula_cache`, not in `DataStore`. So the split is clean: **raw data is queried, +derived data stays in RAM.** Formulas do not need to become SQL. + +**3. Clients are already insulated.** The projection architecture means a client sees +`CellDisplay`s and never learns where a value came from — the server holds the `App` and +therefore the connection. Live-backing and the browser MVP compose rather than compete, +and a browser never needs SQLite at all. + +## The consequence nobody asked for, and it is the important one + +**A live-backed document splits into *our* configuration and *their* data**, and `.improv` +does not currently distinguish them. Today a `.improv` file holds categories, views, +formulas **and** a `## Data` section. A live-backed model's categories and cell values are +the source's; the views, formulas and axis assignments are improvise's. + +So the format grows a **source specification** where a live model would otherwise have +data — a connection, a query, and the column-to-field mapping. Categories become *derived* +(`SELECT DISTINCT`) rather than declared. That is a grammar change, and `improv.pest` is +the single source of truth, so it comes first (design-principles §3). + +Read through the DB-client framing, that split stops being a complication and becomes the +**natural shape of the artifact**: a live-backed `.improv` is *a saved query plus a pivot +configuration* — which is precisely what a saved report is in any BI tool. The document +stops being "my data" and becomes "my question", which is portable between databases in a +way a data file never is, and shareable with someone who has the same source and not the +same rows. + +## Open questions, in the order they will bite + +**Writes.** `SetCell` against a live-backed model either writes through to the source, or +the model is read-only, or edits accumulate as an overlay. All three are defensible and +they are very different products. Not decided here. + +**External writers break ADR 001's assumption.** The basis check assumes improvise is the +only writer: `record_version` is bumped by *our* effects. Another process writing to the +database invalidates a client's basis with nothing to detect it. Options include a source +generation polled or triggered, accepting staleness for read-only models, or scoping the +guarantee explicitly to "no other writer". **ADR 001 should be amended once this is +settled** — it currently states a guarantee this ADR can silently void. + +**Interning and identity.** `DataStore` interns names and ADR 002 hashes the coordinate +tuple. Both still work — coordinates come from the query — but the `RecordHash → record` +index that resolves a `CellId` becomes a cache with an invalidation story rather than a +structure maintained alongside writes. + +**Which rows.** A table name or a user-supplied `SELECT`. A query is strictly more useful +(joins, filters, computed columns) and no harder to consume; a bare table name is sugar +over it. + +**Type mapping.** SQLite is dynamically typed per value, but columns carry declared types, +which is *better* input than CSV's string sampling: `detect_date_format` currently guesses +a date by pattern-matching text. Dates may be TEXT, INTEGER (epoch) or REAL (julian) — +detect, do not assume. NULL is **absence**, not the string `"null"` (improvise-ych). + +**Where the boundary goes.** `improvise-core` must not gain an IO dependency +(`forbidden(improvise_core, io)`), so a query-backed source cannot simply be a `DataStore` +variant inside core. The trait belongs in core and the SQLite implementation in +`improvise-io`, feature-gated and native-only so the wasm gate keeps passing. + +## Consequences + +**Good.** Data larger than memory. A source that stays current. Aggregation pushed to an +engine built for it. And the client/server split already built means none of this reaches +a browser client. + +**Bad.** It changes `Model`, which is the centre of the codebase — `GridLayout`, +evaluation, persistence and the import pipeline all touch it. Latency enters a path that +is currently a `HashMap` lookup, so the viewport-bounded projection stops being an +optimisation and becomes load-bearing. The `.improv` grammar grows a concept. And ADR 001 +holds a guarantee that an external writer voids. + +**Worth doing first regardless of the eventual shape:** the source trait and its in-RAM +implementation, so that today's `DataStore` becomes *an* implementation rather than *the* +implementation. That is a refactor with no behaviour change, `characterization_survives_the_move` +applies, and it is the seam everything else needs — including, if it is ever wanted, the +import-once path this ADR rejected as the *only* answer. diff --git a/context/repo-map.md b/context/repo-map.md index 0a418d0d..8945a48b 100644 --- a/context/repo-map.md +++ b/context/repo-map.md @@ -345,6 +345,8 @@ examples/pretty-print.rs Parse stdin, print formatted .improv .claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl scripts/nix-seed-flake-inputs.sh Seeds locked flake inputs over git so `nix develop` works in cloud sessions prolog-sessions/kb.pl Executable knowledge base: invariants, doctrine, decisions (CI-checked) +context/adr/003-live-backed-models.md A model may be backed by a live query source — + improvise as a DB client. Accepted, NOT a priority. context/adr/002-naming-a-cell.md How a client names a cell: an opaque surrogate on CellDisplay, whose absence means the cell is an aggregate. Records why _Index cannot serve.