doc-site: name context/code/pr-review.md as an omission
The `build` job fails on main and on every branch merged from it: doc-site: context/code/pr-review.md is neither published nor a named omission NOT a merge loss -- scripts/build-doc-site.mjs in this branch is byte-identical to main's, and neither lists the file. PR #96 added context/code/pr-review.md without a manifest line, which is exactly the gate improvise-res9 added: a doc added without a line publishes nothing and says nothing, so the generator refuses. Omitted rather than published, and the line is cloud-agents.md's rather than tooling.md's. tooling.md explains checkers a reader of the code will meet; pr-review.md is about THIS REPOSITORY'S PROCESS -- where a Codex review appears, how to tell it from our own agents' comments, why absence is not approval. Nobody visiting a website is about to read a PR thread. Both homes updated, because an omission is restated beside the manifest AND in doc-site.md section 2, and the prose there counted them ("Neither is referenced...") so it went stale at three. This strays into scripts/, which is Wave N's ground. Flagged in the hand-back; it is a one-line data entry either way if the orchestrator would rather publish it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
@@ -35,8 +35,14 @@ Not published:
|
|||||||
cloud session — tracker hydration, `bd` pitfalls, what the sandbox blocks.
|
cloud session — tracker hydration, `bd` pitfalls, what the sandbox blocks.
|
||||||
It is true and it is useful and it is addressed to nobody who visits a
|
It is true and it is useful and it is addressed to nobody who visits a
|
||||||
website.
|
website.
|
||||||
|
- **`context/code/pr-review.md`** is how an agent reads a PR's reviews — where
|
||||||
|
a ChatGPT Codex review appears, how to tell it from this repo's own agents'
|
||||||
|
comments, and why absence is not approval. It is about *this repository's
|
||||||
|
process*, not about the program, which is the `cloud-agents.md` line and not
|
||||||
|
the `tooling.md` one: `tooling.md` explains checkers a reader of the code
|
||||||
|
will meet, and nobody visiting a website is about to read a PR thread.
|
||||||
|
|
||||||
Neither is referenced by any published document except as a row in
|
None of the three is referenced by any published document except as a row in
|
||||||
`repo-map.md`'s file inventory, which is inside a code fence, so leaving them
|
`repo-map.md`'s file inventory, which is inside a code fence, so leaving them
|
||||||
out creates no dangling link. That was checked rather than assumed.
|
out creates no dangling link. That was checked rather than assumed.
|
||||||
|
|
||||||
|
|||||||
@@ -98,6 +98,7 @@ for (const s of SECTIONS) for (const [src, slug] of s.docs) PAGE_BY_SOURCE.set(s
|
|||||||
const OMITTED = new Map([
|
const OMITTED = new Map([
|
||||||
["context/plan.md", "a prompt, not a plan; cites a context/SPEC.md that no longer exists"],
|
["context/plan.md", "a prompt, not a plan; cites a context/SPEC.md that no longer exists"],
|
||||||
["context/cloud-agents.md", "operational notes for agents, not about the program"],
|
["context/cloud-agents.md", "operational notes for agents, not about the program"],
|
||||||
|
["context/code/pr-review.md", "how an agent reads a PR's reviews; about this repo's process, not about the program"],
|
||||||
]);
|
]);
|
||||||
|
|
||||||
// Derived, because the manifest only ever grows and a doc added without a line
|
// Derived, because the manifest only ever grows and a doc added without a line
|
||||||
|
|||||||
@@ -0,0 +1,99 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>ADR 001 — Concurrent edits — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html" class="here">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/adr/001-concurrent-edits.md">context/adr/001-concurrent-edits.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#adr-001-concurrent-edits-between-clients">ADR 001 — Concurrent edits between clients</a></li><li class="l2"><a href="#context">Context</a></li><li class="l3"><a href="#there-is-already-a-total-order">There is already a total order</a></li><li class="l3"><a href="#scope-conflicts-range-over-one-document">Scope: conflicts range over one document</a></li><li class="l3"><a href="#what-actually-conflicts">What actually conflicts</a></li><li class="l3"><a href="#machinery-that-already-exists">Machinery that already exists</a></li><li class="l3"><a href="#machinery-that-did-not-exist-and-the-hole-it-left">Machinery that did not exist, and the hole it left</a></li><li class="l2"><a href="#decision">Decision</a></li><li class="l3"><a href="#rejection-must-be-forceable-and-force-means-re-compose">Rejection must be forceable, and force means re-compose</a></li><li class="l3"><a href="#optional-second-layer-advisory-reservations">Optional second layer: advisory reservations</a></li><li class="l2"><a href="#why-not-a-causal-clock">Why not a causal clock</a></li><li class="l2"><a href="#consequences">Consequences</a></li><li class="l2"><a href="#implementation-order">Implementation order</a></li><li class="l3"><a href="#what-the-check-is-not">What the check is not</a></li><li class="l3"><a href="#two-places-the-check-deliberately-does-not-reach">Two places the check deliberately does not reach</a></li></ul></details><h1 id="adr-001-concurrent-edits-between-clients">ADR 001 — Concurrent edits between clients</h1>
|
||||||
|
<p><strong>Status:</strong> accepted (2026-09-19); <strong>layer 1 implemented</strong> (<a class="tid" href="../tickets.html#improvise-xwi6" title="Concurrent edits: basis check (ADR 001, layer 1 — the correctness floor)">improvise-xwi6</a>/fkfg, 2026-09-19) · <strong>Supersedes:</strong> nothing · <strong>Decided by:</strong> the user</p>
|
||||||
|
<h2 id="context"><a class="anchor" href="#context">#</a>Context</h2>
|
||||||
|
<p><code>improvise-cqq</code> made the client/server split real: one authoritative <code>App</code>, a <code>ClientSession</code> mirror per client, and projections diffed against those mirrors. <code>improvise-kmgs</code> then put the 28 model mutations on the wire. Two clients can now edit the same document, and nothing decides what happens when they edit the same part of it.</p>
|
||||||
|
<h3 id="there-is-already-a-total-order"><a class="anchor" href="#there-is-already-a-total-order">#</a>There is already a total order</h3>
|
||||||
|
<p>Every <code>ModelCommand</code> reaches the document through one path: <code>Sessions::apply</code> → <code>apply_command</code> → <code>App::apply_effects</code>, against a single <code>App</code>. So the server already serialises writes. <strong>We do not need to build linear history — it exists.</strong></p>
|
||||||
|
<p>What we lack is a way to notice that a client composed an edit against a picture of the document that has since moved. This is a <em>stale basis</em> problem, not an ordering problem — the same shape as the stale cursor (gotcha 24) and the stale session mirror after a document swap (gotcha 39), one level up.</p>
|
||||||
|
<h3 id="scope-conflicts-range-over-one-document"><a class="anchor" href="#scope-conflicts-range-over-one-document">#</a>Scope: conflicts range over one document</h3>
|
||||||
|
<p>improvise is a <strong>multi-document</strong> app, and the server is expected to host several separate models. Sessions editing <em>independent</em> models are trivially non-conflicting, so everything below scopes to a single document.</p>
|
||||||
|
<p>That falls out of the existing shape rather than needing to be built: <code>Sessions</code> does not own the <code>App</code> — <code>connect</code>, <code>apply</code> and <code>project_all</code> all take it as a parameter — so a <code>Sessions</code> is already bound to one document's generation sequence, and multi-document hosting is a registry <em>above</em> it mapping a document to its <code>(App, Sessions)</code> pair. The version tiers this ADR introduces live on the <code>App</code>, so they scope per document for free.</p>
|
||||||
|
<p>Multi-document hosting <strong>landed</strong> as <code>improvise-ws-server::registry</code> (<a class="tid" href="../tickets.html#improvise-j31x" title="Server hosts several documents: a registry above Sessions">improvise-j31x</a>): a <code>Registry</code> of <code>Host</code>s keyed by <code>DocumentId</code>, addressed by <code>ClientId { document, session }</code> because a <code>SessionId</code> is minted per <code>Sessions</code> and so is unique only within its document. The version tiers scope per document as predicted. Note one lifecycle rule that exists to protect <em>this</em> ADR: a document is never evicted while the server runs, because <code>document_generation</code> and <code>record_versions</code> start at zero in a fresh <code>App</code>, so re-opening would accept a reconnecting client's stale basis as current.</p>
|
||||||
|
<h3 id="what-actually-conflicts"><a class="anchor" href="#what-actually-conflicts">#</a>What actually conflicts</h3>
|
||||||
|
<ol>
|
||||||
|
<li><strong>Two writers, same record.</strong> Survivable under last-write-wins: someone's number is lost, the document stays coherent.</li>
|
||||||
|
<li><strong>Shape change vs. cell edit.</strong> A removes category <code>Region</code> while B commits an edit keyed by <code>Region=East</code>. B's <code>CellKey</code> names a coordinate that no longer exists. This is <em>corruption</em>, not a lost keystroke, and it is the class a cell-level lock does not cover — holding <code>Region=East × Revenue</code> does not stop anyone deleting <code>Region</code>.</li>
|
||||||
|
<li><strong>Formula vs. data.</strong> A adds a formula targeting the measure B is typing into; B's raw value is shadowed by a derived one.</li>
|
||||||
|
</ol>
|
||||||
|
<h3 id="machinery-that-already-exists"><a class="anchor" href="#machinery-that-already-exists">#</a>Machinery that already exists</h3>
|
||||||
|
<ul>
|
||||||
|
<li><code>App::document_generation</code> — bumped by <code>install_workbook</code>, consumed by <code>ClientSession::project</code> to reseed a stale mirror (gotcha 39).</li>
|
||||||
|
<li><code>ModelChange::{CellData, LayoutGeometry, WholeWorkbook}</code> — the effect taxonomy, already the single classification for what a mutation reaches (gotcha 35).</li>
|
||||||
|
<li><code>record_row_identity(wide, coord_group_excluding_measure)</code> — a <em>record</em> is the coordinate tuple excluding <code>_Measure</code> (gotcha 1).</li>
|
||||||
|
<li>Drill staging — <code>pending_edits</code> applied atomically by <code>ApplyAndClearDrill</code>, and ruled <strong>private to whoever staged them</strong> (<a class="tid" href="../tickets.html#improvise-tom1" title="Drill pending edits are invisible to remote clients, and there are now three answers to 'what text is in this cell'">improvise-tom1</a>).</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="machinery-that-did-not-exist-and-the-hole-it-left"><a class="anchor" href="#machinery-that-did-not-exist-and-the-hole-it-left">#</a>Machinery that did <em>not</em> exist, and the hole it left</h3>
|
||||||
|
<p><strong>Nothing in the projection stream carried any of the three tiers.</strong> This ADR says a batch carries the basis it was composed against and never says how a client comes to <em>know</em> one — so the check as originally specified was satisfiable only by a client <strong>forging</strong> a basis. That is the invariant deleted with extra steps, arriving through the front door rather than as a <code>force</code> flag.</p>
|
||||||
|
<p>Closed during implementation (<a class="tid" href="../tickets.html#improvise-fkfg" title="ADR-001 b: ModelCommand carries its basis; Applied gains a rejected case">improvise-fkfg</a>) by <code>Projection::Basis</code> + <code>BasisCache</code>: the server publishes the two document tiers plus the write counters for the records <strong>this client's grid names</strong> — exactly the set it can address a write to, per ADR 002 — diffed against what that client was last told and seeded in the bootstrap. It is emitted <strong>after</strong> the grid updates in a batch, because the counters describe the cells those updates carry: a basis arriving first would let a client claim a version for a value it had not been shown. That is the mirror image of gotcha 33.</p>
|
||||||
|
<p>Read this as step (a′) of the implementation order below: <strong>publication comes before the check, because the check is unsatisfiable without it.</strong></p>
|
||||||
|
<h2 id="decision"><a class="anchor" href="#decision">#</a>Decision</h2>
|
||||||
|
<p><strong>Record-level atomicity with optimistic concurrency per batch.</strong></p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>The unit of atomicity is the record</strong>, i.e. the coordinate tuple excluding <code>_Measure</code> — the thing <code>record_row_identity</code> already names. Not the cell: gotcha 1 says editing a category column in records mode moves <em>every</em> measure cell in the row, so a cell-level unit would be violated by a single legal edit. The unit must be at least as coarse as the coarsest legal write.</li>
|
||||||
|
<li><strong>The unit of commit is the batch</strong> — one <code>Vec<Box<dyn Effect>></code> applied by one <code>apply_effects</code>. This is already atomic in practice: a <code>Binding::Sequence</code> flattens into one batch, and a failed <code>Save</code> aborts the rest (gotcha 21).</li>
|
||||||
|
<li><p><strong>A batch carries the basis it was composed against</strong>, and the server rejects it if that basis moved. The basis is <strong>tiered by <code>ModelChange</code></strong>, which is what makes class 2 fall out for free:</p>
|
||||||
|
<table><thead><tr><th>tier</th><th>bumped by</th><th>a batch is stale if</th></tr></thead><tbody><tr><td><code>document_generation</code></td><td><code>WholeWorkbook</code></td><td>it moved at all</td></tr><tr><td><code>layout_version</code></td><td><code>LayoutGeometry</code></td><td>it moved at all</td></tr><tr><td><code>record_version</code></td><td><code>CellData</code> on that record</td><td>any record the batch touches moved</td></tr></tbody></table>
|
||||||
|
<p>A <code>CellData</code> batch is stale if <em>layout</em> moved, because the shape its keys hang off changed. That is class 2, detected without a lock.</p></li>
|
||||||
|
<li><strong>Rejection is a normal outcome, not an error.</strong> The client already receives the new state through projections; it re-composes against the new basis. <code>Applied</code> grows a third case alongside <code>Diff</code> and <code>Rebased</code> — matching is what forces a host to handle it (gotcha 39's reasoning).</li>
|
||||||
|
</ol>
|
||||||
|
<h3 id="rejection-must-be-forceable-and-force-means-re-compose"><a class="anchor" href="#rejection-must-be-forceable-and-force-means-re-compose">#</a>Rejection must be forceable, and force means re-compose</h3>
|
||||||
|
<p>A rejected edit has to give the user a way through. But <strong>force is a re-composition, not a bypass</strong> — it never applies a batch whose basis is stale. A mechanism whose only purpose is to violate the invariant is not a feature, it is the invariant deleted with extra steps. Force re-composes the same <em>intent</em> against the current state and submits a fresh batch with a current basis.</p>
|
||||||
|
<p>That distinction is not pedantic, because what force can mean differs by class:</p>
|
||||||
|
<table><thead><tr><th>class</th><th>can force?</th><th>what force does</th></tr></thead><tbody><tr><td>1 — two writers, same record</td><td><strong>yes, trivially</strong></td><td>re-compose and apply; the user has seen their value and chooses to overwrite it</td></tr><tr><td>2 — shape change vs. cell edit</td><td><strong>no</strong></td><td>the intent no longer type-checks: the key names a coordinate that no longer exists</td></tr><tr><td>3 — formula shadows a value</td><td>yes, with a warning</td><td>the write lands; the displayed value is still derived</td></tr><tr><td>4 — the batch names a record its basis says nothing about</td><td><strong>no — and it is not a race</strong></td><td>nothing to re-compose <em>against</em>; the client was never in the running</td></tr></tbody></table>
|
||||||
|
<p>Class 4 was not in the original draft and was added by the implementation (<code>Rejection::RecordUnstated</code>). It exists because <strong>"any record the batch touches" had to name whose list</strong>, and the answer is the <em>server's</em>: reading it off the client's claim would be a <strong>force by omission</strong> — a client that simply omits a record skips the record tier, with no flag to remove and nothing in the message saying so. <code>reduce.rs::records_named</code> is therefore an exhaustive read of the <code>ModelCommand</code>, with one or-pattern for the variants that name no record, so a new record-writing variant fails the build. A well-formed client cannot produce class 4; it is a case of its own rather than folded into class 1 because "you lost a race" and "you were never in one" are different things to tell a user.</p>
|
||||||
|
<p>For class 1, force is one keystroke and should be. For class 2 there is nothing coherent to force: applying the batch anyway writes a <code>CellKey</code> naming a deleted category, which is the ghost-coordinate corruption this ADR exists to prevent. So:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Force never creates structure.</strong> Re-instating a category someone deleted so an edit can land is a <em>different and larger</em> intent than the user expressed, and it must be an explicit action, not a side effect of forcing a cell edit.</li>
|
||||||
|
<li>When an intent cannot be re-composed, the UI owes the user <strong>what changed and the concrete alternatives</strong> — re-target the edit, or re-create the structure first — rather than a refusal.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Two consequences that pull work forward from layer 2:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>The rejected batch's values stay staged in the session</strong>, so rejection never costs the user their typing. This is the drill staging mechanism doing the job it already does, and it is what makes force cheap rather than a retype. It belongs in layer 1 even though <em>publishing</em> the staged keys is layer 2.</li>
|
||||||
|
<li><strong>A forced overwrite must be visible to the client it overwrote.</strong> Otherwise force is silent data loss with a confirmation step in front of it. The loser's own edit was already committed, so this is a notification, not a conflict — but it must exist, or "someone else won" becomes indistinguishable from "my edit never happened."</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="optional-second-layer-advisory-reservations"><a class="anchor" href="#optional-second-layer-advisory-reservations">#</a>Optional second layer: advisory reservations</h3>
|
||||||
|
<p>The staged-edit mechanism from drill, generalised: a client publishes the <em>keys</em> it has staged so others can grey them out and warn early. The <em>values</em> stay private, per <a class="tid" href="../tickets.html#improvise-tom1" title="Drill pending edits are invisible to remote clients, and there are now three answers to 'what text is in this cell'">improvise-tom1</a>.</p>
|
||||||
|
<p>This layer is <strong>advisory only</strong>. If a reservation is lost, never arrives, or leaks from a crashed client, the system degrades to "you find out at commit" — never to corruption. That is what makes it safe to be sloppy about timeouts and disconnects, and it is why correctness must not be built on it.</p>
|
||||||
|
<h2 id="why-not-a-causal-clock"><a class="anchor" href="#why-not-a-causal-clock">#</a>Why not a causal clock</h2>
|
||||||
|
<p>Considered and rejected, for four reasons.</p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>The server already serialises.</strong> Causal clocks earn their keep where no single node orders writes — peer-to-peer, or offline-capable clients merging on reconnect. With an authoritative <code>App</code> we need <em>staleness detection</em> (one counter per unit), not <em>concurrency detection</em> (a vector per client).</li>
|
||||||
|
<li><strong>A Lamport clock is insufficient.</strong> It gives a total order but cannot tell you two edits were concurrent, which is exactly the question.</li>
|
||||||
|
<li><strong>A vector clock is sufficient but does not answer the question that matters.</strong> It costs O(clients), needs entries reaped for departed clients, and when it reports "these were concurrent" you still need a policy for class 2 — no clock says whether B's edit into <code>Region=East</code> should survive A's deletion of <code>Region</code>. The clock tells you there is a decision to make; it does not make it.</li>
|
||||||
|
<li><strong>CRDTs converge, but not necessarily to something meaningful.</strong> A LWW-register per cell is easy; the hard part is the <em>shape</em> — categories, items, formulas, axis assignments — where convergence can mean an orphaned cell or a resurrected category.</li>
|
||||||
|
</ol>
|
||||||
|
<p>If offline editing ever becomes a goal, this decision should be revisited: that is the requirement that makes a vector clock worth its cost, and it is a much larger commitment than multi-client-while-connected.</p>
|
||||||
|
<h2 id="consequences"><a class="anchor" href="#consequences">#</a>Consequences</h2>
|
||||||
|
<p><strong>Good.</strong> No lock to acquire, so no timeout, no lease renewal, no disconnect handling, and no liveness risk — a crashed client cannot wedge anyone. Correctness does not depend on the advisory layer. The tiered basis reuses <code>ModelChange</code>, so there is no second classification to drift from gotcha 35's table. Class 2 is detected rather than prevented, which is cheaper and does not require predicting which shape changes matter.</p>
|
||||||
|
<p><strong>Bad.</strong> A client can compose an edit and be told no, and the UI owes real work there: keep the typing, show what moved, and offer force where force is coherent. Class 2 is the expensive case — the honest answer is a small dialogue about what changed, not a button. Record-level granularity means two clients editing <em>different measures of the same record</em> serialise against each other even though their writes do not overlap — acceptable at the expected scale, and refinable to <code>(record, measure)</code> if contention shows up. A busy <code>LayoutGeometry</code> editor invalidates every in-flight <code>CellData</code> batch, which is correct but could feel aggressive.</p>
|
||||||
|
<p><strong>Not solved here.</strong> Class 3 (formula shadows a raw value) is detected — the formula bumps <code>layout_version</code> — but the <em>policy</em> is unstated: the edit is rejected, and whether that is the right answer for the user is a separate question. Nor does this address a client that wants to keep editing while disconnected.</p>
|
||||||
|
<h2 id="implementation-order"><a class="anchor" href="#implementation-order">#</a>Implementation order</h2>
|
||||||
|
<p>Deliberately two tickets, so the correctness floor lands without waiting on the UX.</p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>Basis check.</strong> Version tiers, <code>ModelCommand</code> carries its basis, <code>Applied</code> grows the rejected case. Includes the decision of where the version counters live — they are shared document state, so they belong beside <code>document_generation</code> on the authoritative <code>App</code>, not on <code>ViewState</code>.</li>
|
||||||
|
<li><strong>Advisory reservations.</strong> Generalise drill staging; publish key sets, keep values private.</li>
|
||||||
|
</ol>
|
||||||
|
<p>The nullary wire variants <code>ApplyAndClearDrill</code> and <code>AcceptWizardModel</code> were to be resolved first or alongside. <strong>Already done</strong>: <a class="tid" href="../tickets.html#improvise-3x3u" title="Resolve the two unaddressed nullary wire variants before concurrency lands">improvise-3x3u</a> removed both from <code>ModelCommand</code> entirely, so they have no wire form to carry a basis or a <code>SessionId</code>, and layer 1 landed honestly without them (<a class="tid" href="../tickets.html#improvise-k2i4" title="The two nullary model commands are unaddressed and act on one client's staged state">improvise-k2i4</a> closed as stale). What survives is the <em>addressed</em> form multi-client drill will want, which is layer 2 work.</p>
|
||||||
|
<h3 id="what-the-check-is-not"><a class="anchor" href="#what-the-check-is-not">#</a>What the check is not</h3>
|
||||||
|
<p>It is an <strong>anti-staleness coordination check, not a security boundary.</strong> <code>Basis</code> derives <code>Deserialize</code> because it must cross the wire, and every field is public, so any value at all can arrive as a claim — versions start at 0 and increment by 1, so guessing one is trivial. <code>Rejection::RecordUnstated</code> closes <strong>omission</strong> — a client that leaves a record out of its claim — which is the bypass that would otherwise sit in the message format with nothing saying so. It does not close <strong>invention</strong>, and nothing here tries to: a hostile client that forges a current basis is indistinguishable from one that is genuinely caught up, and defending against that is authentication's job.</p>
|
||||||
|
<p><code>BasisCache::claim</code> being the only constructor is therefore a statement about <strong>clients written against this crate</strong>, not about the wire. It is still worth having: it makes the honest path the easy one, and it is what makes "a client that has folded nothing composes nothing" true by construction.</p>
|
||||||
|
<h3 id="two-places-the-check-deliberately-does-not-reach"><a class="anchor" href="#two-places-the-check-deliberately-does-not-reach">#</a>Two places the check deliberately does not reach</h3>
|
||||||
|
<p><strong><code>Command::SwitchView</code> carries no basis.</strong> It is <code>LayoutGeometry</code> but sits outside <code>Command::Model</code>, so <code>apply_switch</code> is unchecked. It cannot corrupt — the workbook refuses a view it does not have, and every mirror takes what was <em>accepted</em> (gotcha 36) — but it is a hole in "a batch carries its basis" and should be decided rather than left as an accident.</p>
|
||||||
|
<p><strong><code>apply_command</code> does not check at all</strong> (nor did <code>reduce_full</code>, which <a class="tid" href="../tickets.html#improvise-s17o" title="reduce_full and apply_command's view-slice arm have no production caller">improvise-s17o</a> has since deleted for having no caller), and the <em>principle</em> is correct: the check is a property of the <strong>session door</strong>, not of the mutation path, because a host holding the <code>App</code> is the truth and not a claimant. That is also why <code>ui::reduce::caught_up(&App, cmd)</code> is not a bypass — it takes <code>&App</code>, so nothing reachable from a wire can call it.</p>
|
||||||
|
<p>As implemented it was a <strong>second door, not an exemption</strong> — both functions took a wire <code>Command</code> whose <code>Command::Model { cmd, .. }</code> arm discarded the basis with <code>..</code>, while <code>ui/reduce.rs</code>'s module doc pointed every future host at them by name. <strong>Closed by <a class="tid" href="../tickets.html#improvise-ondj" title="[bug] reduce_full/apply_command apply a Command::Model without the basis check">improvise-ondj</a></strong> as layer-1 cleanup: both now take a <code>Checked<'_></code>, whose only constructor runs the check against the authoritative <code>App</code> and returns a <code>Rejection</code>. <code>reduce_full</code> was deleted outright shortly after (<a class="tid" href="../tickets.html#improvise-s17o" title="reduce_full and apply_command's view-slice arm have no production caller">improvise-s17o</a>): having no caller was the condition that let the defect hide, and a <code>Checked</code> signature on a function nobody calls fixes the type and not that. The exemption stands — a host's own <code>ModelCommand</code> still carries no basis — but it is no longer reachable as a door, and the unchecked path is unconstructible rather than merely discouraged.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>ADR 002 — Naming a cell — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html" class="here">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/adr/002-naming-a-cell.md">context/adr/002-naming-a-cell.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#adr-002-how-a-client-names-a-cell">ADR 002 — How a client names a cell</a></li><li class="l2"><a href="#context">Context</a></li><li class="l2"><a href="#decision">Decision</a></li><li class="l2"><a href="#decisions-made-along-the-way-and-the-traps-in-them">Decisions made along the way, and the traps in them</a></li><li class="l2"><a href="#consequences">Consequences</a></li><li class="l2"><a href="#extension-2026-09-20-two-more-kinds-by-the-same-reasoning">Extension (2026-09-20) — two more kinds, by the same reasoning</a></li></ul></details><h1 id="adr-002-how-a-client-names-a-cell">ADR 002 — How a client names a cell</h1>
|
||||||
|
<p><strong>Status:</strong> accepted (2026-09-19), extended (2026-09-20) · <strong>Decided by:</strong> the user · <strong>Blocks:</strong> <a class="tid" href="../tickets.html#improvise-1ey" title="Browser frontend MVP: end-to-end working demo">improvise-1ey</a> · <strong>Extended by:</strong> <a class="tid" href="../tickets.html#improvise-ir7l" title="[bug] A formula cell is offered as editable and silently reverts">improvise-ir7l</a>, <a class="tid" href="../tickets.html#improvise-poia" title="The records-mode + column has no projected identity">improvise-poia</a></p>
|
||||||
|
<h2 id="context"><a class="anchor" href="#context">#</a>Context</h2>
|
||||||
|
<p><code>improvise-q08</code> established that a thin client <strong>cannot compose <code>SetCell</code></strong>, and it is the gap that blocks the browser MVP, whose stated scope is "type a number into a cell and see the value update".</p>
|
||||||
|
<p>A client sees a grid of <em>label strings</em> indexed by viewport position. A <code>CellKey</code> is the full sorted coordinate tuple — every Row and Column category, <strong>plus every Page and Filter selection</strong> (gotcha 27). Nothing in <code>GridCache</code> says which category a row label belongs to; nested row axes flatten several categories into one label; page and filter constraints are not projected at all; and records mode encodes a <em>column index</em> in <code>_Dim</code> (gotcha 1). There is no client-side derivation, and projecting the coordinate space would make every client learn conventions the server exists to own.</p>
|
||||||
|
<h2 id="decision"><a class="anchor" href="#decision">#</a>Decision</h2>
|
||||||
|
<p><strong>Every projected cell carries an opaque surrogate identity, and the <em>kind</em> of identity says what may be done to it.</strong></p>
|
||||||
|
<pre><code>pub struct CellDisplay {
|
||||||
|
pub text: String,
|
||||||
|
pub numeric: bool,
|
||||||
|
pub error: bool,
|
||||||
|
pub id: CellId,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub enum CellId {
|
||||||
|
/// A stored cell. `SetCell` takes this and nothing else.
|
||||||
|
Record(RecordHash),
|
||||||
|
/// An aggregate over stored cells. `Drill` takes this and nothing else.
|
||||||
|
Aggregate(AggregateHash),
|
||||||
|
}</code></pre>
|
||||||
|
<p>An aggregate's surrogate is a hash of its <strong>coordinate constraint</strong> — the row and column coordinates plus the page and filter selections that define which stored cells it sums. One hash per cell, the same width and the same index machinery as a record's, and the server resolves it back and runs <code>drill_records</code> when the client asks.</p>
|
||||||
|
<p>Three things follow, and each is the reason to prefer this over alternatives.</p>
|
||||||
|
<p><strong>1. Both operations are unrepresentable-if-wrong, not checked.</strong> A client cannot compose <code>SetCell</code> for an aggregate, <em>and</em> cannot compose <code>Drill</code> for a stored cell, because neither carries an id of the other's type. That is design-principles §3 — make invalid states unrepresentable — and it is strictly stronger than a server-side rejection, which would be a runtime answer to a question the type can settle.</p>
|
||||||
|
<p>An earlier draft used <code>Option<CellId></code>, absence meaning "aggregate". The enum is better for a reason worth keeping: <strong>an aggregate is not the absence of an identity, it is a different kind of identity.</strong> Modelling it as <code>None</code> made <code>SetCell</code> safe and left <code>Drill</code> with nothing to address — which is precisely how drill addressing came to be an open question. Two variants close it by construction.</p>
|
||||||
|
<p><strong>2. The client never learns the coordinate space.</strong> No category names, no page/filter constraints, no <code>_Dim</code> column-index convention. The surrogate is a token to echo, not a structure to understand — so gotchas 1 and 27 stay knowledge the server owns, and a second client implementation cannot get them subtly wrong.</p>
|
||||||
|
<p><strong>3. It mirrors a fork that already exists.</strong> <code>EditOrDrill</code> (<code>cmd/mode.rs</code>) already branches on exactly this in the TUI: a stored cell is edited, an aggregate is drilled. The wire is not inventing a distinction, it is <strong>projecting one the model already makes</strong> — the same discipline as <code>decision(improvise-cqi, command_set_derived_not_invented)</code>.</p>
|
||||||
|
<p>It also follows the precedent <code>CellDisplay::numeric</code> set, whose doc already says the server decides "because it is the side that knows which it was". Settability is another such fact.</p>
|
||||||
|
<h2 id="decisions-made-along-the-way-and-the-traps-in-them"><a class="anchor" href="#decisions-made-along-the-way-and-the-traps-in-them">#</a>Decisions made along the way, and the traps in them</h2>
|
||||||
|
<p>Most of what started as open questions here got answered. They are kept with their reasoning because each rejected alternative is a thing someone will propose again.</p>
|
||||||
|
<p><strong>What the surrogate is — DECIDED: a hash of the normalized record.</strong> Not of the cell: the record is the coordinate tuple minus <code>_Measure</code>, which is already ADR 001's unit of atomicity and already what <code>record_versions</code> is keyed by. A cell is then <code>(record hash, measure)</code>.</p>
|
||||||
|
<p>The reason is that it makes <strong>stale edits cheap to identify</strong>, and it does so three ways at once. The surrogate a client already holds <em>is</em> the lookup key for its own version, so <code>record_versions</code> becomes <code>HashMap<RecordHash, u64></code> — fixed-size, no allocation, no <code>CellKey</code> reconstruction on the hot path. A basis is then a list of <code>(record hash, version)</code> pairs, which is compact on the wire and O(1) to check. And a record that no longer exists simply <strong>fails to resolve</strong>, so that whole class of staleness needs no version comparison at all. One identity serves addressing and staleness instead of each ADR carrying its own.</p>
|
||||||
|
<p>It is well-defined because <code>CellKey</code> already has a canonical form: <code>CellKey::new</code> sorts coordinates by category name and equality and hashing rely on it (gotcha 5). "Normalized" means that canonical sorted form with <code>_Measure</code> removed — the normalization is a property the type already guarantees, not a new convention to maintain.</p>
|
||||||
|
<p>An interned id was rejected: it would need a table with a lifetime nobody has specified, and it buys only collision-freedom, which width buys more cheaply.</p>
|
||||||
|
<p><strong>Three ways this silently breaks, all avoidable.</strong> They are listed because each produces a <em>wrong write</em> rather than an error, which is the failure mode this ADR exists to prevent:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Do not use <code>DefaultHasher</code> or anything built on <code>RandomState</code>.</strong> <code>std</code>'s default <code>HashMap</code> hasher is randomly seeded <strong>per process</strong>, so hashes would differ across a server restart and between any two processes. Specify a fixed hash (a keyed SipHash with a constant key, or FxHash/xxHash/blake3) and write the choice down.</li>
|
||||||
|
<li><strong>Do not use <code>#[derive(Hash)]</code> as the definition.</strong> Rust gives no stability guarantee for derived <code>Hash</code> across compiler versions or platforms. Hash an <strong>explicit byte serialization</strong> of the sorted <code>(category, item)</code> pairs, so the surrogate is defined by the format rather than by whatever the compiler did.</li>
|
||||||
|
<li><strong>Pick the width by consequence, not by convenience.</strong> A collision is a silent write to the wrong record — precisely why <code>_Index</code> was rejected. 64 bits reaches birthday risk around 2³² records, which a pivot tool will not see; 128 bits costs 8 more bytes per <em>visible</em> cell, so on a 50×20 viewport that is 8 KB a frame. The cost is small and the failure is silent, so argue the width explicitly rather than defaulting to <code>u64</code> because it is convenient.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong><code>_Index</code> was considered and rejected — right ontology, unsafe representation.</strong> It is the coordinate that identifies a <em>record</em>, and a record is exactly the thing that is settable rather than aggregated, so the instinct is sound. But <code>CategoryKind::VirtualIndex</code> is documented as <em>"Items are <code>"0"</code>, <code>"1"</code>, … N where N = number of matching cells"</em> — it is <strong>positional over the currently matching set</strong>, not an identity. So <code>_Index = "3"</code> names a different record after a filter change, an insert, or a delete, and names a different one again in another view. A client echoing a stale <code>_Index</code> back would write to the <strong>wrong cell, silently</strong> — strictly worse than a surrogate that fails to resolve, and the same hazard class as the viewport-relative-vs-absolute trap and the stale cursor under a renamed key.</p>
|
||||||
|
<p>What <code>_Index</code> is <em>pointing at</em> is worth keeping, though: <strong>the surrogate should name the record, not the cell.</strong> A record is the coordinate tuple minus <code>_Measure</code> (<code>record_row_identity</code>), which is already ADR 001's unit of atomicity and already what <code>record_versions</code> is keyed by. If a cell is addressed as <code>(record surrogate, measure)</code>, then one identity serves both addressing <em>and</em> staleness, and the two ADRs share a mechanism instead of each inventing one.</p>
|
||||||
|
<p><strong>How the server resolves a surrogate back — DECIDED: a maintained index, in <code>DataStore</code>.</strong> A hash is one-way, so resolution needs a reverse map, and regenerating it from the layout on demand would be O(visible cells) per resolve <em>and</em> would only ever cover the visible window — a client may legitimately hold a surrogate for a record that has scrolled out of view.</p>
|
||||||
|
<p><code>DataStore</code> is the right home because it already does exactly this. It interns every category and item name into a <code>SymbolTable</code>, and it already maintains a secondary index <code>HashMap<(Symbol, Symbol), HashSet<InternedKey>></code> for hot-path lookups. A <code>RecordHash → record</code> index is the same shape, updated at the same points, and the existing <code>(Symbol, Symbol)</code> index is what makes invalidation tractable: when a category or item disappears, the records that mention it are findable without a scan.</p>
|
||||||
|
<p>Two consequences worth stating, because they fall out and are easy to miss.</p>
|
||||||
|
<p><strong>The index and <code>record_versions</code> want to be one structure.</strong> Both are keyed by the record hash and both are invalidated at the same moments, so <code>RecordHash → (record, version)</code> gives ADR 001's basis check and this ADR's resolution in a single lookup.</p>
|
||||||
|
<p><strong>Structural and data staleness separate cleanly</strong>, which is better than the regenerate-on-demand shape would have given:</p>
|
||||||
|
<table><thead><tr><th>incoming surrogate</th><th>means</th><th>answer</th></tr></thead><tbody><tr><td>does not resolve</td><td>the record is <strong>gone</strong> — a category or item it named was removed</td><td>reject; there is nothing to write to</td></tr><tr><td>resolves, version matches</td><td>the client's basis is current</td><td>apply</td></tr><tr><td>resolves, version differs</td><td>the record <strong>changed</strong> since the client read it</td><td>reject as stale (ADR 001), offer force</td></tr></tbody></table>
|
||||||
|
<p>Regenerating from the layout would have conflated the first two rows, because a record scrolled out of the viewport would have looked identical to a record that no longer exists.</p>
|
||||||
|
<p><strong>Whether a forged or stale surrogate is an error or a rejection.</strong> Under the regenerate-from-layout shape it is simply unresolvable, which suggests the same <code>Applied</code> rejection path ADR 001 defines rather than a distinct error.</p>
|
||||||
|
<h2 id="consequences"><a class="anchor" href="#consequences">#</a>Consequences</h2>
|
||||||
|
<p><strong>Good.</strong> Unblocks <code>improvise-1ey</code>. Keeps the client thin. Makes an invalid <code>SetCell</code> unconstructible rather than rejected. Composes with ADR 001 instead of duplicating it. Projects a distinction the model already makes, so there is one source of truth for what is settable.</p>
|
||||||
|
<p><strong>Bad.</strong> <code>CellDisplay</code> grows a field, and it is on the hot path — every visible cell carries it, so the surrogate's width is a wire-size decision, not only a collision one. A client holding a surrogate across a layout change holds something that will fail; that is correct, but the UI must handle it as a normal outcome rather than an error (the same obligation ADR 001 already places on rejection).</p>
|
||||||
|
<p><strong>Not addressed.</strong> The other composability blockers <code>improvise-q08</code> listed — <code>ToggleGroup</code>, <code>AddItemInGroup</code>, <code>RemoveFormula</code>, <code>LoadModel</code> — which are separate projection gaps and are not solved by a cell surrogate.</p>
|
||||||
|
<h2 id="extension-2026-09-20-two-more-kinds-by-the-same-reasoning"><a class="anchor" href="#extension-2026-09-20-two-more-kinds-by-the-same-reasoning">#</a>Extension (2026-09-20) — two more kinds, by the same reasoning</h2>
|
||||||
|
<p>The decision above is unchanged. <code>CellId</code> has since grown two variants, both arrived at by applying clause 1 rather than departing from it: a cell whose permission differs gets a <strong>different kind of identity</strong>, so the wrong operation stays unconstructible instead of being refused at runtime.</p>
|
||||||
|
<pre><code>pub enum CellId {
|
||||||
|
Record(RecordCell),
|
||||||
|
Aggregate(AggregateHash),
|
||||||
|
/// A cell a formula owns (improvise-ir7l).
|
||||||
|
Derived { target: String },
|
||||||
|
/// A records-mode creating column (improvise-poia).
|
||||||
|
Create(CreateKind),
|
||||||
|
}</code></pre>
|
||||||
|
<p><strong><code>Derived</code></strong> — a formula target is a real stored coordinate, so the original two-variant enum called it a <code>Record</code> and every client offered an editor over it. The write <em>landed</em>, and was then shadowed forever by <code>formula_cache</code>. It carries the formula's target and <strong>no surrogate</strong>: there is nothing a client could spend on a <code>SetCell</code>, for the same reason <code>Aggregate</code> carries an <code>AggregateHash</code>.</p>
|
||||||
|
<p><strong><code>Create</code></strong> — the records-mode <code>+</code> and <code>+cat</code> columns projected <code>None</code>, which this ADR defines as "no projected cell". That conflation is the one clause 1's rejected <code>Option<CellId></code> draft was about, arriving from the other side: not an aggregate mistaken for an absence, but a column the user <em>can</em> type into mistaken for one. It carries only the kind, because what the user types into a creating column is a <strong>name</strong>, not a value — the operation is <code>ModelCommand::AddItem</code> / <code>AddCategory</code>, which already existed, so no new command was needed and the record under the cursor is not part of it.</p>
|
||||||
|
<p>What the original two clauses cost, and still cost, is unchanged: a client learns no coordinate space, and an invalid operation is unconstructible rather than rejected. What the extension adds is that the enum is now <strong>four</strong> answers to "what may be done here", so <code>None</code> finally means only what this ADR always said it meant — no projected cell at all.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>ADR 003 — Live-backed models — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html" class="here">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/adr/003-live-backed-models.md">context/adr/003-live-backed-models.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#adr-003-a-model-may-be-backed-by-a-live-query-source">ADR 003 — A model may be backed by a live query source</a></li><li class="l2"><a href="#context">Context</a></li><li class="l2"><a href="#what-the-model-is-today">What the model is today</a></li><li class="l2"><a href="#decision">Decision</a></li><li class="l2"><a href="#the-consequence-nobody-asked-for-and-it-is-the-important-one">The consequence nobody asked for, and it is the important one</a></li><li class="l2"><a href="#open-questions-in-the-order-they-will-bite">Open questions, in the order they will bite</a></li><li class="l2"><a href="#consequences">Consequences</a></li></ul></details><h1 id="adr-003-a-model-may-be-backed-by-a-live-query-source">ADR 003 — A model may be backed by a live query source</h1>
|
||||||
|
<p><strong>Status:</strong> accepted, <strong>not a priority</strong> (2026-09-19) · <strong>Decided by:</strong> the user · <strong>Supersedes:</strong> the read-only framing in <a class="tid" href="../tickets.html#improvise-rucf" title="Read arbitrary SQLite databases as a data source">improvise-rucf</a></p>
|
||||||
|
<blockquote><p>The goal is to have <strong>the option</strong> of essentially using improvise as a <strong>DB client</strong>. Direction, not schedule: nothing here outranks the browser MVP (<a class="tid" href="../tickets.html#improvise-1ey" title="Browser frontend MVP: end-to-end working demo">improvise-1ey</a>), and the tickets are filed low. This is written down now so the option stays open — several decisions being made this week could foreclose it by accident.</p></blockquote>
|
||||||
|
<h2 id="context"><a class="anchor" href="#context">#</a>Context</h2>
|
||||||
|
<p>improvise must handle <strong>arbitrary SQLite databases as models</strong> — a user points it at a database they already have and pivots over it.</p>
|
||||||
|
<p>Two readings were offered and the larger one was chosen. <strong>Import-once</strong> — read a query into a <code>Model</code> in RAM, exactly as CSV import does — is days of work behind an existing seam (<code>csv_parser</code> produces <code>Vec<Value></code>, <code>analyze_records</code> classifies it). It was rejected because it makes the model a <em>snapshot</em>: source changes are invisible until re-import, and the data must fit in memory.</p>
|
||||||
|
<p><strong>Live-backed</strong> was chosen: the model is a view over the source, and queries go to it as the user pivots.</p>
|
||||||
|
<p>This is not <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>. That ticket proposed SQLite as improvise's <em>own persistence format</em> and was declined on measurements — a second implementation of the pest grammar, 22% larger than <code>.improv</code> and 4.7× gzipped. Reading someone else's database has no round-trip obligation and no grammar duplication. What does carry over is the dependency constraint: <code>improvise-io</code> is in the wasm bundle, and <code>libsqlite3-sys</code> does not build for <code>wasm32-unknown-unknown</code> under this repo's nix cc-wrapper.</p>
|
||||||
|
<h2 id="what-the-model-is-today"><a class="anchor" href="#what-the-model-is-today">#</a>What the model is today</h2>
|
||||||
|
<p><code>Model</code> holds <code>categories</code>, a <code>DataStore</code>, <code>formulas</code>, and <code>views</code>. <code>DataStore</code> interns every category and item name into a <code>SymbolTable</code>, stores <code>IndexMap<InternedKey, CellValue></code>, and maintains a secondary index <code>(Symbol, Symbol) → {InternedKey}</code> for hot-path lookups. A <code>Workbook</code> loads <strong>entire</strong> — there is no partial-load path anywhere in the tree, a fact verified while declining <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>.</p>
|
||||||
|
<p>So this decision changes what <code>Model</code> <em>is</em>, not merely where its bytes come from.</p>
|
||||||
|
<h2 id="decision"><a class="anchor" href="#decision">#</a>Decision</h2>
|
||||||
|
<p><strong><code>Model</code> becomes a facade over a data source, of which the current in-RAM <code>DataStore</code> is one implementation and a SQL query engine is another.</strong></p>
|
||||||
|
<p>Three things make this more tractable than it first looks, and they should shape the design rather than be discovered late.</p>
|
||||||
|
<p><strong>1. Aggregation is already the interface, and it is already SQL.</strong> <code>apply_agg</code> computes <code>Sum | Avg | Min | Max | Count</code> over a matching cell set — those are exactly SQL's aggregates. The pushdown is not a translation layer to be invented; it is the operation the model already performs, expressed in the source's own language. <code>GridLayout</code> asks for a cell's value, not for the rows behind it.</p>
|
||||||
|
<p><strong>2. Formula results are already separate from stored data.</strong> Evaluation is fixed-point (<code>recompute_formulas</code> iterates to stability, gotcha 9), and formula-derived values live in <code>Model::formula_cache</code>, not in <code>DataStore</code>. So the split is clean: <strong>raw data is queried, derived data stays in RAM.</strong> Formulas do not need to become SQL.</p>
|
||||||
|
<p><strong>3. Clients are already insulated.</strong> The projection architecture means a client sees <code>CellDisplay</code>s and never learns where a value came from — the server holds the <code>App</code> and therefore the connection. Live-backing and the browser MVP compose rather than compete, and a browser never needs SQLite at all.</p>
|
||||||
|
<h2 id="the-consequence-nobody-asked-for-and-it-is-the-important-one"><a class="anchor" href="#the-consequence-nobody-asked-for-and-it-is-the-important-one">#</a>The consequence nobody asked for, and it is the important one</h2>
|
||||||
|
<p><strong>A live-backed document splits into <em>our</em> configuration and <em>their</em> data</strong>, and <code>.improv</code> does not currently distinguish them. Today a <code>.improv</code> file holds categories, views, formulas <strong>and</strong> a <code>## Data</code> section. A live-backed model's categories and cell values are the source's; the views, formulas and axis assignments are improvise's.</p>
|
||||||
|
<p>So the format grows a <strong>source specification</strong> where a live model would otherwise have data — a connection, a query, and the column-to-field mapping. Categories become <em>derived</em> (<code>SELECT DISTINCT</code>) rather than declared. That is a grammar change, and <code>improv.pest</code> is the single source of truth, so it comes first (design-principles §3).</p>
|
||||||
|
<p>Read through the DB-client framing, that split stops being a complication and becomes the <strong>natural shape of the artifact</strong>: a live-backed <code>.improv</code> is <em>a saved query plus a pivot configuration</em> — which is precisely what a saved report is in any BI tool. The document stops being "my data" and becomes "my question", which is portable between databases in a way a data file never is, and shareable with someone who has the same source and not the same rows.</p>
|
||||||
|
<h2 id="open-questions-in-the-order-they-will-bite"><a class="anchor" href="#open-questions-in-the-order-they-will-bite">#</a>Open questions, in the order they will bite</h2>
|
||||||
|
<p><strong>Writes — DECIDED: a live-backed model is read-only.</strong> Not a limitation accepted reluctantly: the user's framing is that for many use cases <strong>editing capabilities would be a downside</strong>. Pointing a pivot tool at a production database and being unable to write to it is a <em>feature</em>. So there is no write-through, no overlay, and no reconciliation.</p>
|
||||||
|
<p>Two things fall out, and both are simplifications rather than costs.</p>
|
||||||
|
<p><strong>It dissolves the ADR 001 tension instead of requiring an amendment.</strong> ADR 001 governs concurrent <em>edits</em>; a read-only model has none, so the basis check simply does not apply to it. The guarantee is not voided, it is <strong>scoped to documents improvise owns</strong> — which is what it always meant. What remains is a smaller and different question: an external writer changes what clients <em>see</em>, so live-backed projections need a freshness story (polled, triggered, or explicitly stale-until-refreshed). That is projection freshness, not conflict resolution.</p>
|
||||||
|
<p><strong>The machinery to express it already exists, built for something else.</strong> ADR 002 makes <code>SetCell</code> take a <code>CellId::Record</code> and nothing else, so a live-backed model that projects no <code>Record</code> ids makes a write <strong>unconstructible by the client</strong> — no runtime check, no error path, no "this model is read-only" dialog. The settability distinction built for aggregates carries read-only sources for free.</p>
|
||||||
|
<p><strong>One design consequence for the source trait:</strong> writes should not be <em>in</em> the trait that a query source implements. If they were, a read-only source would have to fail at runtime for a method it structurally cannot serve. Keep reads (and aggregates) in the trait every source implements, and let writing be a capability only the owned in-RAM implementation has — so "read-only" is unrepresentable rather than enforced, the same discipline as <code>CellId</code>.</p>
|
||||||
|
<p><strong>Interning and identity.</strong> <code>DataStore</code> interns names and ADR 002 hashes the coordinate tuple. Both still work — coordinates come from the query — but the <code>RecordHash → record</code> index that resolves a <code>CellId</code> becomes a cache with an invalidation story rather than a structure maintained alongside writes.</p>
|
||||||
|
<p><strong>Which rows.</strong> A table name or a user-supplied <code>SELECT</code>. A query is strictly more useful (joins, filters, computed columns) and no harder to consume; a bare table name is sugar over it.</p>
|
||||||
|
<p><strong>Type mapping.</strong> SQLite is dynamically typed per value, but columns carry declared types, which is <em>better</em> input than CSV's string sampling: <code>detect_date_format</code> currently guesses a date by pattern-matching text. Dates may be TEXT, INTEGER (epoch) or REAL (julian) — detect, do not assume. NULL is <strong>absence</strong>, not the string <code>"null"</code> (<a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item "null" on import">improvise-ych</a>).</p>
|
||||||
|
<p><strong>Where the boundary goes.</strong> <code>improvise-core</code> must not gain an IO dependency (<code>forbidden(improvise_core, io)</code>), so a query-backed source cannot simply be a <code>DataStore</code> variant inside core. The trait belongs in core and the SQLite implementation in <code>improvise-io</code>, feature-gated and native-only so the wasm gate keeps passing.</p>
|
||||||
|
<h2 id="consequences"><a class="anchor" href="#consequences">#</a>Consequences</h2>
|
||||||
|
<p><strong>Good.</strong> Data larger than memory. A source that stays current. Aggregation pushed to an engine built for it. And the client/server split already built means none of this reaches a browser client.</p>
|
||||||
|
<p><strong>Bad.</strong> It changes <code>Model</code>, which is the centre of the codebase — <code>GridLayout</code>, evaluation, persistence and the import pipeline all touch it. Latency enters a path that is currently a <code>HashMap</code> lookup, so the viewport-bounded projection stops being an optimisation and becomes load-bearing. And the <code>.improv</code> grammar grows a concept.</p>
|
||||||
|
<p><strong>Worth doing first regardless of the eventual shape:</strong> the source trait and its in-RAM implementation, so that today's <code>DataStore</code> becomes <em>an</em> implementation rather than <em>the</em> implementation. That is a refactor with no behaviour change, <code>characterization_survives_the_move</code> applies, and it is the seam everything else needs — including, if it is ever wanted, the import-once path this ADR rejected as the <em>only</em> answer.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,159 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>improvise-core — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html" class="here">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/core.md">context/code/core.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#improvise-core-the-arguments">improvise-core — the arguments</a></li><li class="l2"><a href="#1-cellkey-is-canonical-by-construction">1. CellKey is canonical by construction</a></li><li class="l2"><a href="#2-aggregating-a-cell-s-raw-values">2. Aggregating a cell's raw values</a></li><li class="l3"><a href="#evaluate-aggregated-takes-aggregation-not-a-list-it-cannot-read">evaluate_aggregated takes Aggregation, not a list it cannot read</a></li><li class="l3"><a href="#non-numeric-values-reduce-by-unanimity-and-the-branch-order-is-the-rule">Non-numeric values reduce by unanimity, and the branch order is the rule</a></li><li class="l2"><a href="#3-formula-evaluation">3. Formula evaluation</a></li><li class="l2"><a href="#4-float-equality-has-one-epsilon-and-the-div-0-guard-shares-it">4. Float equality has one epsilon, and the div/0 guard shares it</a></li><li class="l2"><a href="#5-surrogate-identities-model-record-rs-adr-002">5. Surrogate identities — model/record.rs (ADR 002)</a></li><li class="l2"><a href="#6-records-mode-is-wide-format-view-layout-rs">6. Records mode is wide format (view/layout.rs)</a></li><li class="l3"><a href="#prune-axis-items-takes-a-predicate-not-a-flag-vector">prune_axis_items takes a predicate, not a flag vector</a></li><li class="l2"><a href="#7-axes-page-and-filter-are-two-different-questions">7. Axes: Page and Filter are two different questions</a></li><li class="l3"><a href="#on-category-added-a-virtual-on-an-axis-is-a-configuration-not-a-vacancy">on_category_added: a virtual on an axis is a configuration, not a vacancy</a></li><li class="l3"><a href="#known-defect-two-answers-to-is-this-name-a-virtual">Known defect: two answers to "is this name a virtual"</a></li><li class="l2"><a href="#8-ordering-is-correctness-not-tidiness-model-cell-rs">8. Ordering is correctness, not tidiness (model/cell.rs)</a></li><li class="l3"><a href="#the-record-index-recordentry">The record index: RecordEntry</a></li><li class="l2"><a href="#9-categories">9. Categories</a></li><li class="l2"><a href="#10-workbook-exists-to-break-the-model-view-cycle">10. Workbook exists to break the Model ↔ View cycle</a></li><li class="l2"><a href="#11-virtual-views">11. Virtual views</a></li><li class="l2"><a href="#12-axis-of-panics-try-axis-of-does-not">12. axis_of panics, try_axis_of does not</a></li><li class="l3"><a href="#record-cell-key-is-a-match-and-the-match-is-the-guard-improvise-ir7l">record_cell_key is a match, and the match is the guard (improvise-ir7l)</a></li><li class="l2"><a href="#13-route-then-aggregate-sum-folds-over-the-member-records">13. Route then aggregate: SUM(<expr>) folds over the member records</a></li><li class="l2"><a href="#pageaxis-what-constrains-and-what-the-user-may-turn-improvise-p1ym">PageAxis — what constrains, and what the user may turn (improvise-p1ym)</a></li><li class="l2"><a href="#the-number-format-format-rs-improvise-711n">The number format — format.rs (improvise-711n)</a></li></ul></details><h1 id="improvise-core-the-arguments"><code>improvise-core</code> — the arguments</h1>
|
||||||
|
<p>Written for a reader with the code open. This is where the reasoning that used to live in <code>crates/improvise-core/**</code>'s comments now lives; the comments themselves are capped at two lines (design-principles §6). Bug narratives keyed by test name are in <a href="../test-evidence/core.html"><code>context/test-evidence/core.md</code></a>.</p>
|
||||||
|
<p>The crate is pure data: <code>Model</code>, <code>View</code>, <code>GridLayout</code>, <code>Workbook</code>, number formatting. It depends on <code>improvise-formula</code> for AST types and knows nothing about UI, IO or commands.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="1-cellkey-is-canonical-by-construction"><a class="anchor" href="#1-cellkey-is-canonical-by-construction">#</a>1. <code>CellKey</code> is canonical by construction</h2>
|
||||||
|
<p><code>CellKey(Vec<(category, item)>)</code> is <strong>always sorted by category name</strong>, and the constructors are the only way to get one: <code>CellKey::new(coords)</code> sorts, <code>with(cat, item)</code> replaces-or-inserts-and-re-sorts, <code>without(cat)</code> filters and so preserves the order. Equality, hashing and storage all rely on the canonical form, so two keys naming the same intersection in different orders <em>are</em> the same key. Never build the inner <code>Vec</code> directly (gotcha 5).</p>
|
||||||
|
<p>That <code>without</code> preserves the sort is what lets <code>record_of</code> be a one-liner rather than a normalisation step — see §5.</p>
|
||||||
|
<h2 id="2-aggregating-a-cell-s-raw-values"><a class="anchor" href="#2-aggregating-a-cell-s-raw-values">#</a>2. Aggregating a cell's raw values</h2>
|
||||||
|
<p>Three functions, deliberately with three different emptiness rules, and the differences are the whole content of the design (<a class="tid" href="../tickets.html#improvise-kdp" title="Duplicated aggregation dispatch block">improvise-kdp</a>):</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong><code>apply_agg(values: &[f64], func)</code></strong> — reduces numbers. <code>None</code> on empty: an empty cell reads as <em>absent</em>, not as zero, or every blank intersection in a pivot would render <code>0</code>. Both aggregation paths (<code>evaluate_aggregated</code> and <code>aggregate_raw</code>) go through it so their semantics cannot drift.</li>
|
||||||
|
<li><strong><code>aggregate_cell_values(values: &[&CellValue], func)</code></strong> — reduces a mixed set. See below.</li>
|
||||||
|
<li><strong><code>eval_agg(...)</code></strong> — evaluates a formula-level <code>SUM(x)</code> / <code>AVG(x WHERE …)</code>. Its emptiness rule is <em>not</em> <code>apply_agg</code>'s: an aggregate over no rows is <code>0</code> for <code>SUM</code>/<code>COUNT</code>, which have an identity, and an error for <code>AVG</code>/<code>MIN</code>/<code>MAX</code>, which do not.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="evaluate-aggregated-takes-aggregation-not-a-list-it-cannot-read"><a class="anchor" href="#evaluate-aggregated-takes-aggregation-not-a-list-it-cannot-read">#</a><code>evaluate_aggregated</code> takes <code>Aggregation</code>, not a list it cannot read</h3>
|
||||||
|
<p>The parameter was <code>none_cats: &[String]</code>, and the body consulted it exactly once, for emptiness — a boolean wearing a list's type, since the aggregation itself runs over <code>key.0</code> through <code>matching_values</code>. It never filtered by that list, in any revision: <code>git log -S</code> back to the commit that introduced <code>Axis::None</code> shows the same <code>is_empty()</code> and nothing else. So the name promised a filter the body has never performed, and the next caller to build a careful list would have threaded it into a parameter that cannot use it.</p>
|
||||||
|
<p><code>Aggregation::{Exact, OverHidden}</code> admits only the two states the body distinguishes, and <code>Aggregation::of(none_cats)</code> performs the reduction once, at the boundary that holds the list (<a class="tid" href="../tickets.html#improvise-5jm5" title="Model::evaluate_aggregated takes a list it only tests for emptiness">improvise-5jm5</a>, <code>doctrine(an_invariant_is_made_unrepresentable_not_validated)</code> — the same answer <code>prune_axis_items</code> and <code>gp-tree</code>'s <code>Row<T></code> got). It is an enum rather than a <code>bool</code> because the two states have names worth reading at the call site, and the regression guard is the <strong>signature</strong>: the thing the ticket describes — two different lists, same emptiness, same answer — is no longer constructible.</p>
|
||||||
|
<p>Note what does <em>not</em> change: <code>recompute_formulas</code> and <code>aggregate_raw</code> keep <code>&[String]</code>, because they walk the elements (stripping each hidden coordinate off a stem). Two parameters that looked alike were doing two different jobs.</p>
|
||||||
|
<h3 id="non-numeric-values-reduce-by-unanimity-and-the-branch-order-is-the-rule"><a class="anchor" href="#non-numeric-values-reduce-by-unanimity-and-the-branch-order-is-the-rule">#</a>Non-numeric values reduce by unanimity, and the branch order is the rule</h3>
|
||||||
|
<p><code>aggregate_cell_values</code> checks the <strong>all-numeric branch first</strong>. If every value is a number it reduces through <code>apply_agg</code>; otherwise, if every value equals the first, the result is that value; otherwise it is <code>CellValue::Error("mixed")</code>.</p>
|
||||||
|
<p>That ordering is the entire difference between this rule and "a single value passes through". <code>Count</code> is the only function that can tell them apart — every other function is the identity on a one-element set — so <code>Count</code> of a lone number must still be <code>1</code> and not the number itself. <code>a_lone_number_still_counts_ as_one_rather_than_passing_through</code> is the pin.</p>
|
||||||
|
<p>What the rule forbids is <strong>dropping</strong>. The old code did <code>filter_map(as_f64)</code>, and since <code>_Index</code>/<code>_Dim</code> sit on <code>Axis::None</code> in every default view, <code>none_cats</code> is non-empty almost always and the aggregating path was almost always taken — so every <code>CellValue::Text</code> in every document rendered blank, including the pipe-quoted text <code>.improv</code> carries as a first-class value (<code>Region=West, Measure=Revenue = |pending|</code>). A user could type a text value, save it, load it, and never see it again (<a class="tid" href="../tickets.html#improvise-l4qx" title="[bug] A text cell is invisible whenever any category is on the None axis">improvise-l4qx</a>).</p>
|
||||||
|
<p>Both <code>evaluate_aggregated</code> and <code>aggregate_raw</code> share the helper, so a formula ref over mixed values now errors <code>"mixed"</code> rather than silently summing the numeric ones and reading as authoritative while under-reporting.</p>
|
||||||
|
<h2 id="3-formula-evaluation"><a class="anchor" href="#3-formula-evaluation">#</a>3. Formula evaluation</h2>
|
||||||
|
<p><strong>Fixed point.</strong> <code>recompute_formulas(none_cats)</code> clears <code>formula_cache</code>, collects the distinct coordinate <em>stems</em> (each data key with the formula's target category and every <code>none_cat</code> stripped), and iterates <code>stem × formula target</code> until nothing changes, bounded by <code>MAX_EVAL_DEPTH = 16</code>. Chained formulas (<code>Margin = Profit / Revenue</code> over <code>Profit = Revenue - Cost</code>) converge naturally; circular references converge to <code>CellValue::Error("circular")</code> rather than overflowing the stack. The stems are an <code>IndexSet</code>: dedupe is O(1) per cell and insertion order is preserved (<a class="tid" href="../tickets.html#improvise-6os" title="O(n^2) stem deduplication in recompute_formulas">improvise-6os</a> replaced an O(n²) <code>Vec::contains</code>).</p>
|
||||||
|
<p>There are <strong>two</strong> evaluators, and they must be changed together:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>eval_formula_depth</code> / <code>eval_expr</code> — the direct, uncached path, recursion bounded by <code>depth</code>.</li>
|
||||||
|
<li><code>eval_formula_with_cache</code> / <code>eval_expr_cached</code> — the fixed-point path, which resolves a <code>Expr::Ref</code> through <code>formula_cache</code> first and falls back to <code>aggregate_raw</code>.</li>
|
||||||
|
</ul>
|
||||||
|
<p>A rule stated in one and not the other is a live bug; <a class="tid" href="../tickets.html#improvise-0bf" title="Inconsistent float-equality tolerance in formula eval">improvise-0bf</a> was exactly that (see §4).</p>
|
||||||
|
<p><strong><code>find_item_category</code></strong> searches real categories first and then formula targets, because <code>add_formula</code> never inserts its target into <code>_Measure</code> (<a class="tid" href="../tickets.html#improvise-nrs" title="Duplicated find_item_category helper in eval paths">improvise-nrs</a>). Without the fallback, a reference to a computed measure fails to resolve.</p>
|
||||||
|
<p><strong>Display rounding never reaches eval.</strong> <code>format_f64</code> is called only in rendering paths. Feeding a rounded value back into a formula is the bug <code>formula_chain_preserves_full_precision</code> exists to catch.</p>
|
||||||
|
<h2 id="4-float-equality-has-one-epsilon-and-the-div-0-guard-shares-it"><a class="anchor" href="#4-float-equality-has-one-epsilon-and-the-div-0-guard-shares-it">#</a>4. Float equality has one epsilon, and the div/0 guard shares it</h2>
|
||||||
|
<p><code>FLOAT_EQ_EPSILON = 1e-10</code> is used by <code>=</code>/<code>!=</code> <strong>and</strong> by the division-by-zero guard, in both evaluators. The two must agree or <code>IF(x = 0, a, y / x)</code> is incoherent: the condition calls <code>x</code> zero and the else branch divides by it anyway. With <code>x = 1e-11</code> the old exact <code>rv == 0.0</code> guard returned <code>1e13</code> (<a class="tid" href="../tickets.html#improvise-0bf" title="Inconsistent float-equality tolerance in formula eval">improvise-0bf</a>). <code>1e-10</code> is far below typical data magnitudes but absorbs accumulated f64 rounding noise from chained formulas.</p>
|
||||||
|
<p>Note the repo map's gotcha 17 (which called this "currently mixed") describes the <em>pre-fix</em> state; the guard and the comparison share the constant now.</p>
|
||||||
|
<h2 id="5-surrogate-identities-model-record-rs-adr-002"><a class="anchor" href="#5-surrogate-identities-model-record-rs-adr-002">#</a>5. Surrogate identities — <code>model/record.rs</code> (ADR 002)</h2>
|
||||||
|
<p>A client sees a grid of label strings indexed by viewport position. It holds no <code>Model</code>, no category names and no page/filter constraints, so it cannot build a <code>CellKey</code> — which is why <a class="tid" href="../tickets.html#improvise-q08" title="improvise-ws-server binary (tokio + tungstenite session wrapper)">improvise-q08</a> found a thin client cannot compose <code>SetCell</code>. The answer is an opaque surrogate the server projects per visible cell and the client echoes back.</p>
|
||||||
|
<p><strong>Two surrogates, because two kinds of thing are nameable and only one is settable.</strong> <code>RecordHash</code> names a <strong>record</strong> — the coordinate tuple minus <code>_Measure</code> (<code>record_of</code>). That unit is not chosen here: it is already ADR 001's unit of atomicity and already what the write counters are keyed by, because renaming a category coordinate in wide records mode moves every measure cell in the row (§6). One identity therefore serves addressing <em>and</em> staleness. <code>AggregateHash</code> names an <strong>aggregate's coordinate constraint</strong> — row and column coordinates plus the page <em>and filter</em> selections that decide which stored cells it sums (<code>View::filtering_cats()</code>, §7). An aggregate is not the <em>absence</em> of an identity; it is a different kind of identity, which is what makes <code>Drill</code> addressable at all. It is deliberately <strong>not</strong> normalised: a measure on Row or Column is part of what the aggregate sums, so dropping <code>_Measure</code> would give two visibly different totals one name.</p>
|
||||||
|
<p><strong>The hash is specified here, not inherited.</strong> Three ways it breaks silently, each producing a <em>wrong write</em> rather than an error:</p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>Not <code>DefaultHasher</code>, not <code>RandomState</code>.</strong> <code>std</code>'s default <code>HashMap</code> hasher is randomly seeded per process, so a surrogate would not survive a restart or agree between two processes. This is SipHash-2-4 under a constant key.</li>
|
||||||
|
<li><strong>Not <code>#[derive(Hash)]</code>.</strong> Rust gives no stability guarantee for a derived <code>Hash</code> across compiler versions or platforms. What is hashed is the explicit byte serialization in <code>hash_coords</code>, so the surrogate is defined by that format rather than by whatever the compiler did this week.</li>
|
||||||
|
<li><strong>128 bits, argued rather than defaulted.</strong> A collision is a silent write to the wrong record — precisely what got <code>_Index</code> rejected as a surrogate — and no later check catches it: a colliding surrogate resolves <em>successfully</em> to a record the client never saw. 64 bits reaches birthday risk around 2^32 records, which a pivot tool will not see by live record count; but the risk is over every record a long-lived document <em>ever names</em>, and the consequence is unbounded and undetectable. The other choice costs 8 more bytes per <em>visible</em> cell — 8 KB on a full 50×20 projection and far less on the diffs that carry steady-state traffic. Bounded cost against silent corruption is not close.</li>
|
||||||
|
</ol>
|
||||||
|
<p><strong>Serialization details that are load-bearing.</strong> The byte format is <code>tag ‖ count ‖ (len(category) ‖ category ‖ len(item) ‖ item)*</code>, every length a little-endian <code>u64</code>. The lengths are what make it injective: without them <code>("ab","c")</code> and <code>("a","bc")</code> serialize alike. Only <code>Hasher::write</code> is used, never <code>write_u64</code>/<code>write_u8</code>, which are free to feed the hasher in native byte order. The SipHash key constants are written as big-endian readings of <code>b"improvis"</code>/<code>b"e-record"</code> so the constant is the same on every platform. A domain tag byte (<code>0x01</code> record, <code>0x02</code> aggregate) is hashed first, because a record and an aggregate may be built from the same coordinates — an aggregate over a single record is the degenerate case — and they must not hash alike.</p>
|
||||||
|
<p><strong>Hex on the wire, not a JSON number.</strong> <code>serde_json</code> writes a <code>u128</code> as a bare number and a JavaScript host parses every JSON number as <code>f64</code>, so anything above 2^53 comes back changed — which for a surrogate means resolving to a different record or to none. The wire form is a fixed 32-character lowercase hex string.</p>
|
||||||
|
<h2 id="6-records-mode-is-wide-format-view-layout-rs"><a class="anchor" href="#6-records-mode-is-wide-format-view-layout-rs">#</a>6. Records mode is wide format (<code>view/layout.rs</code>)</h2>
|
||||||
|
<p>A row is one <strong>coordinate tuple</strong> — the cells sharing everything but <code>_Measure</code> — and each raw measure gets its own editable column. The columns are <code>[coordinates…, +cat, measures…, +]</code>: one per regular category, then the column that creates a <strong>new category</strong> (<a class="tid" href="../tickets.html#improvise-2zq" title="Records mode has no inline way to add a category, unlike the + column for measures">improvise-2zq</a>), then one per raw measure, then the column that creates a new measure (<a class="tid" href="../tickets.html#improvise-c5v" title="Improve records view data entry by making each measure a normal value column (wide format instead of synthetic Value column)">improvise-c5v</a>). Each <code>+</code> sits adjacent to what it adds to; two trailing columns would have made the user read a label to tell them apart, and position says it instead. Formula targets are not columns — they are computed, so there is nothing to type into them.</p>
|
||||||
|
<p>Consequences: two measures on one coordinate are <strong>one row</strong>; editing a category column moves <strong>every</strong> measure cell in the row, not just the one under the cursor; and <code>_Dim</code> in the synthetic key carries the column <strong>index</strong>, not a label, because a category and a measure item may share a name and a label could not say which column an edit landed on. Resolve the index through <code>GridLayout::record_col</code> into a <code>RecordCol</code>. Drill pending edits are keyed by <code>(row, RecordCol)</code>.</p>
|
||||||
|
<p><code>RecordRow::group</code> collapses loose cells into wide rows in <strong>first-appearance</strong> order, so the result follows the store's order rather than a hash walk (§8). A cell with no <code>_Measure</code> coordinate still anchors a row — that is how <code>add-record-row</code> seeds one — but contributes no column value.</p>
|
||||||
|
<p><strong>Records mode is a property of the view's axes, and <code>View</code> owns the one definition.</strong> <code>View::is_records_layout()</code> (<code>_Index</code> on Row and <code>_Dim</code> on Column) is what <code>GridLayout::new</code> asks and what <code>on_category_added</code> asks; two derivations would drift (<a class="tid" href="../tickets.html#improvise-edp" title="Establish ViewModel layer: derived data between RenderCache and widgets">improvise-edp</a>, the disease this codebase keeps curing).</p>
|
||||||
|
<h3 id="prune-axis-items-takes-a-predicate-not-a-flag-vector"><a class="anchor" href="#prune-axis-items-takes-a-predicate-not-a-flag-vector">#</a><code>prune_axis_items</code> takes a predicate, not a flag vector</h3>
|
||||||
|
<p><code>prune_empty</code> hands it <code>impl Fn(usize) -> bool</code> over the <strong>data</strong> index — the position a <code>row</code>/<code>col</code> argument means, which group headers do not consume. The predicate's domain is derived from <code>items</code> itself, so there is no parallel length a caller can get wrong and debug and release builds cannot disagree about an out-of-range flag. The earlier <code>&[bool]</code> + <code>debug_assert</code> pair could have, had a third caller ever sized <code>keep</code> from something other than <code>row_count()</code>/<code>col_count()</code> (<a class="tid" href="../tickets.html#improvise-n3wt" title="prune_axis_items: keep-slice length invariant is only a debug_assert">improvise-n3wt</a>, <code>doctrine(an_invariant_is_made_unrepresentable_not_validated)</code>).</p>
|
||||||
|
<p>A group header survives only when a data item that follows it survives. In <code>prune_empty</code> the row pass does not touch <code>col_items</code> or <code>has_value</code>, so the column pass still reads the original grid. Records mode is exempt entirely — the user drilled in to see all the raw data.</p>
|
||||||
|
<h2 id="7-axes-page-and-filter-are-two-different-questions"><a class="anchor" href="#7-axes-page-and-filter-are-two-different-questions">#</a>7. Axes: <code>Page</code> and <code>Filter</code> are two different questions</h2>
|
||||||
|
<p><code>Axis</code> is <code>Row | Column | Page | Filter | None</code>. <code>Filter</code> names a coordinate the <strong>drill</strong> fixed (<a class="tid" href="../tickets.html#improvise-dz8" title="Add Axis::Filter for drill-fixed coordinates">improvise-dz8</a>): like <code>Page</code> it constrains which records are visible, but unlike <code>Page</code> the user cannot navigate it. <code>Axis::next()</code> maps <code>Filter</code> to itself and no navigable axis rotates into it, so tile select cannot hand one out, and <code>is_user_assignable()</code> is false for it.</p>
|
||||||
|
<p>Picking the wrong question is the trap:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>"what data is visible"</strong> → <code>View::filtering_cats()</code> (Page ∪ Filter). Used by <code>GridLayout</code>, by <code>AggregateHash</code>, and by <code>add-record-row</code> when it builds a key from the current constraints.</li>
|
||||||
|
<li><strong>"what can the user act on"</strong> → <code>categories_on(Axis::Page)</code> for <code>[</code>/<code>]</code> page cycling, or <code>View::tileable_cats()</code> for all three tile-select sites.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>Axis</code> also owns four presentation vocabularies — <code>plain_label()</code>, <code>glyph()</code> (which <code>Display</code> appends, giving "Row ↕"), <code>short_label()</code> (three columns for the tile bar; <code>None</code> is "·", <strong>not</strong> a truncation of "None") and <code>initial()</code> (one letter for the view panel; <code>None</code> is empty) — plus <code>color() -> AxisColor</code>. <code>AxisColor</code> is a <em>semantic</em> palette because <code>improvise-core</code> must not name a ratatui type; <code>draw.rs::axis_fg</code> does the one mapping, exactly as <code>minibuf_fg</code> does for <code>MinibufColor</code> (<a class="tid" href="../tickets.html#improvise-rml" title="Consolidate Axis display into a method on Axis">improvise-rml</a>). The tile bar and the category panel each carried a copy of the colour table before, with nothing keeping them in agreement.</p>
|
||||||
|
<h3 id="on-category-added-a-virtual-on-an-axis-is-a-configuration-not-a-vacancy"><a class="anchor" href="#on-category-added-a-virtual-on-an-axis-is-a-configuration-not-a-vacancy">#</a><code>on_category_added</code>: a virtual on an axis is a configuration, not a vacancy</h3>
|
||||||
|
<p>Virtual (underscore) categories seat on <code>Axis::None</code>. A regular category takes the first slot no <em>regular</em> category holds — Row, then Column, then Page.</p>
|
||||||
|
<p>This used to <strong>bump</strong> a virtual off Row or Column to make room, assuming a virtual sitting there was a placeholder. It never is, and the assumption cost two live bugs (<a class="tid" href="../tickets.html#improvise-2zq" title="Records mode has no inline way to add a category, unlike the + column for measures">improvise-2zq</a>):</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>_Index</code> on Row together with <code>_Dim</code> on Column <em>is</em> records mode, and the only production writers of those axes are the records and drill setups. Bumping <code>_Index</code> <strong>ejected the user from the grid they were standing in</strong>: naming a category with <code>N</code> in records mode silently returned them to a pivot.</li>
|
||||||
|
<li><code>_Measure</code> on Column is an ordinary user configuration — it is what <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/demo.improv"><code>web/demo.improv</code></a> ships. Bumping it made every measure vanish from the grid the moment a second regular category was added.</li>
|
||||||
|
</ul>
|
||||||
|
<p>So a virtual is never moved. A records view seats the new category on <code>Axis::None</code>, which is where records mode keeps <strong>every</strong> regular category and where <code>records_columns</code> — which filters on <code>kind.is_regular()</code> and never looks at an axis — picks it up as a column immediately. Elsewhere the new category simply joins whatever virtual is already there, nesting a header rather than destroying one.</p>
|
||||||
|
<p>The negative arm is load-bearing and must stay tested: without <code>a_pivot_view_still_seats_categories_row_then_column_then_page</code>, "always seat on <code>None</code>" would pass every assertion about records mode. Probing confirmed the asymmetry — seating everything on <code>None</code> fails 12 tests, while removing the virtual bump failed none at all, which is how the bump came to be wrong in two directions with nothing noticing.</p>
|
||||||
|
<h3 id="known-defect-two-answers-to-is-this-name-a-virtual"><a class="anchor" href="#known-defect-two-answers-to-is-this-name-a-virtual">#</a>Known defect: two answers to "is this name a virtual"</h3>
|
||||||
|
<p><code>on_category_added</code> tests <code>cat_name.starts_with('_')</code> (twice) while <code>is_records_layout()</code> four lines away asks <code>CategoryKind::of_name</code>. That is <strong><a class="tid" href="../tickets.html#improvise-bidq" title="Two answers to 'is this category virtual', four lines apart in view/types.rs">improvise-bidq</a></strong> and it is open; do not assume either spelling is the intended one when editing near it.</p>
|
||||||
|
<h2 id="8-ordering-is-correctness-not-tidiness-model-cell-rs"><a class="anchor" href="#8-ordering-is-correctness-not-tidiness-model-cell-rs">#</a>8. Ordering is correctness, not tidiness (<code>model/cell.rs</code>)</h2>
|
||||||
|
<p><code>DataStore</code> holds cells in an <code>IndexMap</code> keyed by <code>InternedKey</code>, plus a secondary index <code>HashMap<(Symbol, Symbol), HashSet<InternedKey>></code>.</p>
|
||||||
|
<p><strong>An index whose values are <code>HashSet</code>s is for O(1) lookup, never for iteration.</strong> <code>matching_cells</code>/<code>matching_values</code> used to walk the index's <code>HashSet</code> directly, so their order was hash order — different in every process, and different between two stores in one process, because <code>RandomState</code> keys each <code>HashMap</code> afresh. That defeated the whole reason <code>cells</code> is an <code>IndexMap</code>, and it was not tidiness (<a class="tid" href="../tickets.html#improvise-shyt" title="[bug] Records-mode row order is not stable across processes">improvise-shyt</a>). It caused, in ascending order of seriousness:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>records-mode rows reordering between sessions</strong>;</li>
|
||||||
|
<li><strong><code>SUM</code>/<code>AVG</code> drifting in the last <code>f64</code> bits</strong>, because f64 addition is not associative (<code>Min</code>/<code>Max</code>/<code>Count</code> and the text-unanimity rule of §2 are order-free, so they were never affected);</li>
|
||||||
|
<li>worst, the <strong>drill snapshot</strong> freezing its record list in hash order while <code>drill_state.pending_edits</code> are keyed by <em>row index</em> (§6) — so the same drill opened in a second process addressed <strong>different records</strong>. Silent misdirected writes, and nothing in the tracker knew.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>matching_positions</code> is the fix: the index still <strong>narrows</strong> the candidates, and the primary map <strong>orders</strong> them — each surviving candidate resolves to its position with one hash lookup and the positions are sorted. Shape stays O(smallest set × |partial|) plus O(m log m) over the <em>matches</em>, not a scan. Measured at 18,000 cells: +20% worst case on <code>matching_values</code>, and <code>matching_cells</code> 7% <em>faster</em>, sorted positions resolving <code>CellKey</code>s with better locality. The alternatives both move the cost — an ordered index value type pays on every <code>set</code>; scanning the primary is O(store) per query.</p>
|
||||||
|
<p><strong>An empty <code>partial</code> matches every cell, vacuously, and <code>matching_positions</code> is where that is answered</strong> (<a class="tid" href="../tickets.html#improvise-hkt6" title="Three small totality gaps found in review">improvise-hkt6</a>). It used to index <code>sets[0]</code> and panic; both public callers happened to guard, so it was unreachable — but the guard lived one level away from the indexing, which is how this class of bug returns. The guards could not simply be pushed down as <code>vec![]</code>: "no coordinates constrain anything" means <em>all</em> positions, and answering "none" here would have made removing a caller's fast path change behaviour silently. The callers keep theirs as a fast path, not as a guard.</p>
|
||||||
|
<p><strong><code>live_records</code> has a decided order, not an inherited one</strong> (<a class="tid" href="../tickets.html#improvise-2yc2" title="DataStore::live_records walks a HashMap -- decide its order before writing the caller">improvise-2yc2</a>). It walks <code>cells</code> and projects insertion order onto records: a record sits where its <strong>first</strong> cell sits. It deliberately pays O(cells) rather than O(records), because <code>cells</code>' order is already the answer <code>matching_positions</code> gives and a second answer to "what order is this store in" is the disease. It is derived per call rather than memoised on <code>RecordEntry</code>, since a stored position is a copy and a record deleted and re-created would hold its old one while its cells sit at the end of the store. Note what is <strong>not</strong> on offer: an unordered return. Every iterator has an order, and a type that merely disclaims one still hands the caller hash order — the guard living one level away from the hazard, which is the shape this whole class of bug returns through. The function had no callers when this was decided, so the order was still free to <em>decide</em> rather than to discover; with no caller to benchmark against, correctness is the defensible default, and a hot-path caller can argue for a memoised position with numbers.</p>
|
||||||
|
<h3 id="the-record-index-recordentry"><a class="anchor" href="#the-record-index-recordentry">#</a>The record index: <code>RecordEntry</code></h3>
|
||||||
|
<p><code>DataStore::records: HashMap<RecordHash, RecordEntry></code> answers both of a client surrogate's questions — "which record do you mean?" (ADR 002) and "is your basis stale?" (ADR 001) — from <strong>one</strong> entry in <strong>one</strong> lookup, because both are keyed by the record and both are invalidated at the same moment.</p>
|
||||||
|
<p>It lives in the store because this is where the same job is already done: the store already interns every name and already maintains a secondary index updated at exactly these two points (<code>set</code>, <code>remove</code>), and it is that existing <code>(Symbol, Symbol)</code> index which makes <strong>invalidation</strong> tractable — when a category or item disappears, the cells naming it are found without a scan, so the records built from them are too.</p>
|
||||||
|
<p>It is deliberately <strong>not</strong> regenerated from the layout on demand. That would be O(visible cells) per resolve <em>and</em> would only ever cover the visible window, so a record scrolled out of view would look identical to a record that no longer exists — conflating "reject, nothing to write to" with "apply".</p>
|
||||||
|
<p>It is <strong>not serialized</strong>: <code>Deserialize</code> rebuilds it through <code>set</code>, and every version starts at zero, which is correct because a freshly loaded document is a new <code>document_generation</code> and no client holds a basis against it.</p>
|
||||||
|
<p><strong><code>cells == 0</code> is a tombstone, not a reason to forget the entry.</strong> A record that is deleted and then re-created must not hand a client holding the old basis its own version number back; dropping the entry would reset the counter to zero and make a stale basis look <em>current</em>, which is the silent accept the basis check exists to prevent. The counter tracks <strong>liveness, not writes</strong> — <code>set</code> bumps <code>cells</code> only for a genuinely new cell, and <code>version</code> is moved by the observed diff in <code>App::apply_effects</code>, never from <code>set</code>.</p>
|
||||||
|
<p><code>resolve_record</code> returning <code>None</code> means <strong>the record is gone</strong> — never stored, or every cell removed, which is what happens when a category or item it named is deleted. That is one arm of staleness and needs no version comparison: there is nothing to write to. <code>record_version</code> reporting <code>0</code> for an unknown surrogate is deliberate: a basis composed against an absent record and one composed against an untouched record are the same claim.</p>
|
||||||
|
<h2 id="9-categories"><a class="anchor" href="#9-categories">#</a>9. Categories</h2>
|
||||||
|
<p><code>CategoryKind</code> is <code>Regular | VirtualIndex | VirtualDim | VirtualMeasure | Label</code>. Business rules match on the enum rather than sniffing name prefixes.</p>
|
||||||
|
<p><code>CategoryKind::category_name()</code> is the <strong>one</strong> place the name↔kind mapping is written down, so adding a variant forces a decision there (<a class="tid" href="../tickets.html#improvise-2lh" title="Migrate name-string comparisons to CategoryKind enum checks">improvise-2lh</a>). <code>of_name(name)</code> is safe to use in place of looking the <code>Category</code> up, because <code>Model::add_category</code> hands back the existing virtual category rather than shadowing a reserved name — <code>stored_virtual_categories_match_their_names</code> is what makes that sound. <code>Label</code> categories are user-named and so read as <code>Regular</code> from <code>of_name</code>; consult the stored <code>Category::kind</code> when the distinction matters.</p>
|
||||||
|
<p><code>VirtualMeasure</code> items come from two sources: numeric data fields listed in the file, and formula targets added implicitly. They are virtual because formula-derived items are <em>implied</em> by the formula definitions — listing them explicitly would be redundant in the file format and confusing in the UI. <code>add_formula</code> does <strong>not</strong> add items to <code>_Measure</code>; use <code>measure_item_names()</code> / <code>effective_item_names("_Measure")</code>.</p>
|
||||||
|
<p><code>Label</code> is a high-cardinality per-row field (description, id, note): stored alongside the data so it shows up in record/drill views, but defaulting to <code>Axis::None</code> and excluded from the <code>MAX_CATEGORIES = 12</code> pivot limit and from auto Row/Column assignment.</p>
|
||||||
|
<p><code>Category::add_item</code> deduplicates by name and assigns IDs from a private counter, so external code cannot fabricate an <code>ItemId</code>.</p>
|
||||||
|
<h2 id="10-workbook-exists-to-break-the-model-view-cycle"><a class="anchor" href="#10-workbook-exists-to-break-the-model-view-cycle">#</a>10. <code>Workbook</code> exists to break the <code>Model ↔ View</code> cycle</h2>
|
||||||
|
<p><code>Model</code> knows nothing about views; <code>View</code> depends on <code>Model</code>, one direction. Cross-slice operations — adding or removing a category must update both the model's categories and <strong>every</strong> view's axis map — live on <code>Workbook</code>, so <code>Model</code> stays pure data and <code>improvise-core</code> can be extracted without pulling view code along. That is also why <code>AddCategory</code>/<code>RemoveCategory</code> are <code>LayoutGeometry</code> effects rather than <code>CellData</code> ones (gotcha 3).</p>
|
||||||
|
<p><code>Workbook::new</code> seeds a <code>Default</code> view, registers the three virtual categories through <code>on_category_added</code> (so all three land on <code>Axis::None</code>), then bumps <code>_Measure</code> to <code>Axis::Page</code> so aggregated pivot views show a single measure at a time (<a class="tid" href="../tickets.html#improvise-kos" title="_Dim and _Index should work as pivot axes outside Records/Drill view; must default to None axis on non-records views including empty models">improvise-kos</a>, commit 709f2df). Leaving <code>_Index</code>/<code>_Dim</code> on <code>None</code> keeps pivot mode the default — records mode activates only when the user moves both onto axes.</p>
|
||||||
|
<p><code>Workbook::add_item</code> errors on an unknown category so callers do not reach through <code>model.category_mut(..)</code> and invent their own handling (<a class="tid" href="../tickets.html#improvise-tvt" title="Route mutations through Workbook facade instead of app.workbook.model chains">improvise-tvt</a>). There is no <code>Model::add_item</code>.</p>
|
||||||
|
<h2 id="11-virtual-views"><a class="anchor" href="#11-virtual-views">#</a>11. Virtual views</h2>
|
||||||
|
<p><code>_Records</code> and <code>_Drill</code> (<code>view::VIRTUAL_VIEW_NAMES</code>) are scratch views the app builds on demand. Both put the virtual <code>_Index</code>/<code>_Dim</code> categories on axes, and those categories are never persisted — so these views are rebuilt from scratch each time rather than saved (<a class="tid" href="../tickets.html#improvise-60z" title="Virtual views _Records and _Drill should not be persisted to .improv files">improvise-60z</a>).</p>
|
||||||
|
<h2 id="12-axis-of-panics-try-axis-of-does-not"><a class="anchor" href="#12-axis-of-panics-try-axis-of-does-not">#</a>12. <code>axis_of</code> panics, <code>try_axis_of</code> does not</h2>
|
||||||
|
<p>Registration is guaranteed for every model category: <code>Workbook::new</code>, <code>add_category</code>, <code>add_label_category</code> and <code>create_view</code> all call <code>View::on_category_added</code> for every view. So any name obtained from <code>Model::categories</code> or from the view's own <code>category_axes</code> is safe for <code>axis_of</code>. Use <code>try_axis_of</code> when the name comes from anywhere else — <code>cycle_axis</code> does, and no-ops on an unknown category to match <code>set_axis</code> (<a class="tid" href="../tickets.html#improvise-t8s" title="View::axis_of panics on missing category">improvise-t8s</a>).</p>
|
||||||
|
<h3 id="record-cell-key-is-a-match-and-the-match-is-the-guard-improvise-ir7l"><a class="anchor" href="#record-cell-key-is-a-match-and-the-match-is-the-guard-improvise-ir7l">#</a><code>record_cell_key</code> is a match, and the match is the guard (<a class="tid" href="../tickets.html#improvise-ir7l" title="[bug] A formula cell is offered as editable and silently reverts">improvise-ir7l</a>)</h3>
|
||||||
|
<p><code>GridLayout::record_cell_key(row, col)</code> is the one mapping from a records position to a model cell, and it answers <code>Some</code> for <code>RecordCol::Measure</code> alone. The other three arms are spelled out rather than swept into a <code>_</code>, and that exhaustive match is the whole guard: a new <code>RecordCol</code> variant fails the build until someone decides whether it names a stored cell.</p>
|
||||||
|
<p><strong>There is no test for the <code>Category</code> arm, and the reason is not that the wrong answer would be harmless.</strong> It would be a real fault, and a subtle one worth naming so nobody re-derives it wrongly. <code>RecordCol::Category(name)</code> carries a <strong>category</strong> name, while <code>RecordRow::cell_key(arg)</code> puts its argument under <code>_Measure</code>. So answering <code>Some</code> there yields <code>coords + _Measure=<category name></code>, and <code>Model::formula_for</code> would match it whenever a <code>_Measure</code> item happens to share a name with a regular category — projecting <code>CellId::Derived</code> for a <strong>coordinate column</strong>, which is never a formula target and which the pivot path does not have at all. The two front ends would then disagree, which is the design-principles §5 defect this ticket exists to close. <code>RecordCol</code> exists for exactly that collision: its doc says the enum resolves a column from its index "so a category and a measure sharing a name stay distinguishable".</p>
|
||||||
|
<p>What earns the decision to leave it untested is narrower: the <strong>reachable</strong> version of that fault is a new variant answered carelessly, and the exhaustive match makes that unrepresentable rather than merely unlikely (<code>doctrine(an_invariant_is_made_unrepresentable_not_validated)</code>). A test would pin behaviour the type system already pins, and would not survive the arm being renamed. Found in review, and the first recorded reason for declining it was <strong>wrong</strong> — it claimed the injected behaviour was defensible by analogy with a formula targeting a regular category, which is a different key entirely.</p>
|
||||||
|
<h2 id="13-route-then-aggregate-sum-folds-over-the-member-records"><a class="anchor" href="#13-route-then-aggregate-sum-folds-over-the-member-records">#</a>13. Route then aggregate: <code>SUM(<expr>)</code> folds over the member records</h2>
|
||||||
|
<p><a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>, and the one place the evaluator reaches past the cell it was asked about.</p>
|
||||||
|
<p><strong>The problem.</strong> A formula is evaluated at a <code>CellKey</code>. In an aggregating cell <code>evaluate_aggregated</code> aggregates each <code>Expr::Ref</code> <em>first</em> and evaluates the expression once on the totals. That is exactly right for a linear formula — <code>SUM(Revenue) - SUM(Cost)</code> and <code>SUM(Revenue - Cost)</code> are the same number — and exactly wrong for anything that <strong>routes</strong>. <code>IF(Cost < 50, 1, 10)</code> over records costing 10 and 100 answers <code>1 + 10 = 11</code> per record and <code>10</code> on the total, because <em>aggregate-then-route is not route-then-aggregate</em>. Every fitted tree (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>, <a class="tid" href="../tickets.html#improvise-vx3" title="fit-measure command: fit a tree and add it as a formula">improvise-vx3</a>) is such a formula, so it meant one thing on a record row and something else one row up. The user's ruling was to <strong>aggregate the rule over the members</strong>.</p>
|
||||||
|
<p><strong>The mechanism, and why it needed no new vocabulary.</strong> <code>SUM(e)</code> already means "fold <code>e</code> over the members". It simply did not: <code>eval_agg</code> specialised <code>Expr::Ref</code> and, for every other inner expression, summed the raw cells matching the partial key and <em>ignored the expression entirely</em>. It now folds — one entry per member record, the expression evaluated at that record's own coordinates:</p>
|
||||||
|
<pre><code>partial = context.without(target_category) (plus any inline WHERE)
|
||||||
|
records = distinct(matching_cells(partial).map(key.without("_Measure")))
|
||||||
|
values = records.map(|r| eval(inner, context.with(every coord of r)))</code></pre>
|
||||||
|
<p>Three consequences worth stating.</p>
|
||||||
|
<p>The rule is <strong>written once, and so is the evaluator it folds with.</strong> <code>eval_agg</code> is shared by <code>eval_expr</code> and <code>eval_expr_cached</code>, and <strong>both</strong> hand it the same <code>eval_at</code>: the record-level <code>eval_expr</code>. That is not a convenience, it is the correctness. A member key is a <em>record</em>, and <code>formula_cache</code> is keyed by <strong>stems with <code>none_cats</code> stripped</strong> — so the cached evaluator's <code>Expr::Ref</code> can never hit the cache at a member key, falls through to <code>aggregate_raw</code>, finds no raw cell behind a formula target, and answers <code>Err("ref:…")</code>. A first version did exactly that, and the two evaluators then disagreed on any routing formula reading a <em>derived</em> measure: <code>Number(4.0)</code> uncached, <code>Error("ref:Profit")</code> cached, which is the very divergence this section exists to remove (§3: "a rule stated in one and not the other is a live bug").</p>
|
||||||
|
<p><strong>And making the lookup "work" would have been worse than the error.</strong> The only way to hit the cache from a member key is to strip <code>none_cats</code> off it — which returns the <em>group's</em> value for that formula and hands it to every member. That is aggregate-then-route again, laundered through a cache lookup, and it would have been silent. The cache cannot answer a per-record question, so the fold does not ask it. Only <code>Expr::Is</code> needed an arm in each evaluator, and since the fold routes through <code>eval_expr</code>, the cached arm is now reached <strong>only by a bare <code>Is</code> outside a <code>SUM</code></strong> — which is what <code>the_cached_evaluator_has_its_own_working_coordinate_predicate</code> pins.</p>
|
||||||
|
<p>The <code>Expr::Ref</code> fast path <strong>stays</strong>, and it is the same fold specialised: a record's <code>Ref</code> is that record's own cell, so summing them is summing the matching values. It differs in exactly one place — a record with no such cell is <em>skipped</em> there and an <strong>error</strong> in the fold. That is deliberate: a bare <code>SUM(Revenue)</code> asks for the values that exist, while <code>SUM(<expr>)</code> asks the expression of each member, and an expression that cannot be evaluated is an error rather than a zero. Silently dropping members is how a routed sum would quietly answer for a subset.</p>
|
||||||
|
<p>The asymmetry's <strong>blast radius</strong> is wider than it sounds, and worth stating: gotcha 7 makes a pipe-quoted text cell a first-class <code>.improv</code> value, so <strong>one</strong> text cell anywhere in a member set turns the whole aggregated fitted cell into <code>Error("ref:Cost")</code>, where the bare-ref path would have dropped it and returned a number. That is the loud failure this repo prefers to a quietly wrong sum, but it is a real behaviour difference and not a detail.</p>
|
||||||
|
<p>Records come out in <code>DataStore</code> insertion order, deduplicated by an <code>IndexSet</code>, for §8's reason — hash order here would make a fitted cell drift in its last bits between processes.</p>
|
||||||
|
<p><strong>Interaction with the fixed point (§3) and <code>formula_cache</code>.</strong> None of this is new state. The fold reads <em>raw</em> data through the same <code>evaluate_depth</code> / <code>aggregate_raw</code> the non-folding path uses, so a formula cell still depends only on raw cells and on other formula cells' cached values; <code>recompute_formulas</code> iterates the same stems to the same fixed point, and <code>formula_cache</code> still holds exactly the aggregated formula cells it held before. The fold does not cache and does not memoise: it is O(members) per evaluation per pass.</p>
|
||||||
|
<p><strong>Measured</strong>, because the cost was a blocker and not a footnote (<a class="tid" href="../tickets.html#improvise-9iyv" title="The route-then-aggregate fold's cost, measured">improvise-9iyv</a>; <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/crates/improvise-core/examples/fold_bench.rs"><code>crates/improvise-core/examples/fold_bench.rs</code></a>, five runs per point, best–worst). At 18,050 cells with one routing formula and <code>Tag</code> on <code>Axis::None</code>, <code>recompute_formulas</code> takes <strong>18.1–19.2 ms</strong> release and 132.9–135.1 ms debug, against 7.1–9.0 ms / 71.7–74.6 ms for a linear <code>Rev - Cost</code>. That is <strong>2.3×</strong> a linear formula and <strong>linear in cells</strong>: 400 → 0.4 ms, 2,000 → 2.0 ms, 18,050 → 18.1 ms, so 45× the cells costs 45× the time.</p>
|
||||||
|
<p>It was not always linear. Routing the fold through <code>eval_expr_cached</code> — the first version, above — made every member's <code>Ref</code> miss the cache and fall into <code>aggregate_raw</code>, one poorly-narrowed scan <strong>per member</strong>: 51.1–53.3 ms at the same 18,050 cells and superlinear with it (45× the cells cost 85× the time). So the divergence and the stall were <strong>one defect</strong>, and fixing the first fixed the second.</p>
|
||||||
|
<p><strong>And "18,050 cells" was not a sufficient description of that workload.</strong> At the same cell count the old code cost 51 ms at a 25×361 shape and 244 ms at 95×95 — a 4.8× spread from group <em>count</em> alone, which is what a per-member scan predicts and a constant factor does not. After the fix the two shapes are 16.9–17.2 ms and 16.4–17.5 ms: indistinguishable. <strong>The shape-dependence being gone is the confirmation of the cause</strong>, more than the absolute number is. Re-measuring this means stating the shape, not just the cells.</p>
|
||||||
|
<p>The fixed point runs exactly <strong>2</strong> passes (measured, not assumed), so hoisting the member set out of the loop has a 2× ceiling on the fold's share and was declined: a memo is state, and a memo keyed by anything less than the whole <code>partial</code> is a second answer to "what does this cell depend on".</p>
|
||||||
|
<p>One asymmetry not to tidy: the cached path hands <code>eval_expr</code> a fresh <code>MAX_EVAL_DEPTH</code> at each <code>Agg</code>, while the uncached <code>Agg</code> threads its own <code>depth</code> through. The second is what bounds a chain of folded formulas; making them agree by resetting both would unbound it.</p>
|
||||||
|
<p><strong><code>Expr::Is(Filter)</code> — a coordinate predicate as a number.</strong> <code>Cat IS Item</code> is <code>1.0</code> when the cell's own <code>Cat</code> coordinate is <code>Item</code> and <code>0.0</code> otherwise. It keeps the evaluator <code>f64</code>-only — no string comparison enters <code>eval_expr</code> — and it reuses the primitive <code>Formula::filter</code> already had, <code>context.get(&cat)</code>, including its <code>unwrap_or(false)</code>: a cell with <strong>no</strong> <code>Cat</code> coordinate is not that item. Under route-then-aggregate that arm is reached only where a record genuinely lacks the coordinate, because the fold hands the expression a record key. Evaluated bare, outside a <code>SUM</code>, it degrades exactly as <code>Formula::filter</code> does — which is why a fitted tree is always emitted wrapped.</p>
|
||||||
|
<h2 id="pageaxis-what-constrains-and-what-the-user-may-turn-improvise-p1ym"><a class="anchor" href="#pageaxis-what-constrains-and-what-the-user-may-turn-improvise-p1ym">#</a><code>PageAxis</code> — what constrains, and what the user may turn (<a class="tid" href="../tickets.html#improvise-p1ym" title="Project the page-axis item lists so [ and ] work in a browser">improvise-p1ym</a>)</h2>
|
||||||
|
<p><code>GridLayout::page_coords</code> answers two questions that gotcha 27 says must not be confused. The bar at the top of a view names <strong>every</strong> category constraining the visible records, which is <code>View::filtering_cats()</code> — Page ∪ Filter — because a drill fixes its coordinates on <code>Filter</code> and the user should still see them. <code>[</code>/<code>]</code> cycle only what the user may act on, which is <code>categories_on(Axis::Page)</code>.</p>
|
||||||
|
<p>Those were two walks before: <code>GridLayout</code> did the first, <code>navigation.rs:: page_cat_data</code> did the second, and each spelled the Page-vs-Filter rule itself. Two spellings of one rule is the shape design-principles §5 calls the disease, and here it had already leaked into the protocol — <code>GridCache::page_coords</code> was a verbatim copy of the layout's list, so a browser handed it would have tried to page a Filter axis.</p>
|
||||||
|
<p>So <code>page_coords</code> is now <code>Vec<PageAxis></code>, and <code>items</code> is <code>Some</code> exactly when the axis is navigable. The rule lives in the <em>type</em>: a <code>Filter</code> axis carries no item list, so no caller can compose a cycle for one, and <code>PageAxis::cycle()</code> is the one place that pairs a list with the index its selection sits at. <code>page_cat_data</code> and the browser's <code>page_cycle</code> both read it rather than deriving it.</p>
|
||||||
|
<p>Two consequences worth stating. The list is <code>Model::effective_item_names</code>, the same call that already chose the default selection, so the selection and the list it indexes into come from one place by construction — <code>page_cat_data</code> used <code>ordered_item_names</code> before, which for a <code>_Measure</code> page axis meant the default selection could be a formula target <code>[</code>/<code>]</code> could not reach. That incoherence is gone; the cost is that <code>[</code>/<code>]</code> over a <code>_Measure</code> page axis now visits formula targets, which is what the column list shows anyway.</p>
|
||||||
|
<p>And the list is <strong>not</strong> filtered to visible items. Hidden items are still cycled, because that is what the TUI has always done, and making the browser differ would be the divergence rather than the fix. Changing it is a change to both, together.</p>
|
||||||
|
<h2 id="the-number-format-format-rs-improvise-711n"><a class="anchor" href="#the-number-format-format-rs-improvise-711n">#</a>The number format — <code>format.rs</code> (<a class="tid" href="../tickets.html#improvise-711n" title="parse_number_format silently yields 0 decimals for the ,.2f spelling our own docs show">improvise-711n</a>)</h2>
|
||||||
|
<p>A view's <code>number_format</code> is a <strong>printf-style picture, and only that</strong>. A <code>,</code> anywhere turns on thousands grouping; a <code>.</code> introduces the precision, which is the <strong>leading digit run after the last <code>.</code></strong>; anything following those digits — a conversion letter — is ignored. So <code>,.0</code>, <code>,.2</code>, <code>.2</code>, <code>,.2f</code> and <code>%.2f</code> all name what they look like they name.</p>
|
||||||
|
<p><strong>Why the trailing <code>f</code> is accepted rather than refused.</strong> It round-trips into documents today and always has. <code>improv.pest</code> spells the line <code>format_line = { "format: " ~ rest_of_line ~ NEWLINE ~ blank_lines }</code> — the grammar takes the format string as an <strong>opaque line</strong>, so <code>format: ,.2f</code> is a document the grammar accepts; <code>:set-number-format</code> stores whatever the user typed verbatim (<code>cmd/effect_cmds.rs</code>), and <code>format_md</code> writes <code>view.number_format</code> back out unchanged. Refusing the spelling at load would therefore reject documents this repo's own grammar admits and its own <code>.improv</code> example spells, to no benefit: by gotcha 8 the format is display-only, so misreading it costs a wrong pixel and can never cost a wrong number. A refusal here buys nothing and costs a document.</p>
|
||||||
|
<p><strong>Why printf and not Excel.</strong> The two conventions disagree on exactly the spelling that distinguishes them: <code>.00</code> is <em>precision zero</em> to printf and <em>two places</em> to Excel, so a parser that tried to honour both would have to guess. One is picked, and it is the one every spelling in this repository already uses. Consequences worth stating rather than discovering: <code>0.00</code> reads as <code>(false, 0)</code>, and an Excel picture such as <code>#,##0.00</code> reads as <code>(true, 0)</code> — grouped, no decimals.</p>
|
||||||
|
<p><strong>Nothing is validated, deliberately.</strong> A spelling the rule cannot read names no precision and formats with none; there is no error channel and no <code>Result</code>, because the reader is display-only and four call sites would have to thread one.</p>
|
||||||
|
<p><strong>What the bug was, and why six tests sat on top of it.</strong> The precision was read as <code>fmt[i+1..].parse::<u8>()</code> over the <em>whole</em> tail, so <code>"2f"</code> failed to parse and fell through to <code>unwrap_or(0)</code> — <code>,.2f</code> displayed <strong>zero</strong> decimals, in the TUI, in the browser projection and in CSV export. <code>",.2f"</code> and its <code>,.3f</code>/<code>,.4f</code> siblings are what this workspace's own fixtures reach for, in about ten tests, and none could see it: they assert the stored <em>string</em> round-trips or use it as an opaque value to dirty a widget, or — in <code>export_csv_quotes_commas_per_rfc4180</code> — they assert the rendered value <code>starts_with("1,200")</code>, which <code>1,200</code> and <code>1,200.00</code> both satisfy. A prefix check over a value whose <em>suffix</em> is the thing in question answers a neighbouring question, which is why that one is now an exact equality.</p>
|
||||||
|
<p><strong>Only two of the three consumers are guarded.</strong> Reverting the fix fails <code>format.rs</code>'s own unit tests and <code>export_csv_quotes_commas_per_rfc4180</code>, and nothing else in the workspace: <code>ui/project.rs::grid_of</code> (the browser projection) and <code>ui/app.rs</code> both call <code>parse_number_format</code> and neither has a test that renders a cell and asserts its digits. That gap is named rather than closed — those files are <code>improvise-command</code>, which was another wave's ground when this landed — and it is the more useful half of this entry, because it says where the next regression would go unseen.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>improvise-formula and improvise-io — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html" class="here">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/formula-and-io.md">context/code/formula-and-io.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#improvise-formula-and-improvise-io">improvise-formula and improvise-io</a></li><li class="l2"><a href="#pipe-quoting-has-exactly-one-home">Pipe quoting has exactly one home</a></li><li class="l3"><a href="#the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it">The keyword guard is canonical, user-confirmed — do not revert it</a></li><li class="l3"><a href="#the-escape-table-and-the-one-input-outside-both-grammars">The escape table, and the one input outside both grammars</a></li><li class="l2"><a href="#cat-is-item-a-coordinate-predicate-in-the-expression-grammar">Cat IS Item: a coordinate predicate in the expression grammar</a></li><li class="l3"><a href="#where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable">WHERE carries the same boundary, and the mechanism has to be generatable</a></li><li class="l2"><a href="#a-formula-target-is-unquoted-exactly-as-a-reference-is">A formula target is unquoted, exactly as a reference is</a></li><li class="l2"><a href="#the-improv-grammar-is-the-single-source-of-truth">The .improv grammar is the single source of truth</a></li><li class="l3"><a href="#the-tree-walkers-are-infallible-by-construction">The tree walkers are infallible by construction</a></li><li class="l3"><a href="#two-passes-because-sections-may-appear-in-any-order">Two passes, because sections may appear in any order</a></li><li class="l3"><a href="#a-section-header-takes-its-name-as-the-whole-line-totally">A section header takes its name as the whole line, totally</a></li><li class="l2"><a href="#storage-is-whole-document-read-write-and-that-closed-the-sqlite-question">Storage is whole-document read/write, and that closed the SQLite question</a></li><li class="l3"><a href="#storage-contract-is-a-public-function-not-a-test-helper">storage_contract is a public function, not a test helper</a></li><li class="l3"><a href="#impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule">impl Storage for Rc<T> lives in improvise-io because of the orphan rule</a></li><li class="l2"><a href="#opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds">OpfsFs: why a browser backend can be synchronous, and what the index holds</a></li><li class="l3"><a href="#the-index-holds-documents-not-entries">The index holds documents, not entries</a></li><li class="l3"><a href="#list-joins-dir-like-the-other-backends">list joins dir like the other backends</a></li><li class="l2"><a href="#loading-decompression-and-decoding-are-two-failures-with-two-answers">Loading: decompression and decoding are two failures with two answers</a></li><li class="l2"><a href="#persistence-write-order-and-the-virtual-views">Persistence write order and the virtual views</a></li><li class="l2"><a href="#import-order-is-a-correctness-property-not-tidiness">Import: order is a correctness property, not tidiness</a></li><li class="l3"><a href="#the-ambiguous-date-heuristic-reads-american-first-deliberately">The ambiguous-date heuristic reads American-first, deliberately</a></li><li class="l3"><a href="#csv-parsing">CSV parsing</a></li><li class="l3"><a href="#the-wizard-s-back-path-is-a-history-stack">The wizard's back path is a history stack</a></li><li class="l2"><a href="#csv-export">CSV export</a></li></ul></details><h1 id="improvise-formula-and-improvise-io"><code>improvise-formula</code> and <code>improvise-io</code></h1>
|
||||||
|
<p>The arguments that used to live in the comments of these two crates. Written for a reader with the code open; the code says <em>what</em>, this says <em>why</em>, and why the obvious alternative is wrong. Bug narratives keyed by test name are in <a href="../test-evidence/formula-and-io.html"><code>context/test-evidence/formula-and-io.md</code></a>.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="pipe-quoting-has-exactly-one-home"><a class="anchor" href="#pipe-quoting-has-exactly-one-home">#</a>Pipe quoting has exactly one home</h2>
|
||||||
|
<p><code>improvise_formula::quote</code> owns <strong>both halves</strong> of the CL-style <code>|…|</code> convention — writing and reading — for both grammars. <code>formula.pest</code>'s <code>bare_ident</code>/<code>pipe_quoted</code> and <code>improv.pest</code>'s <code>bare_name</code>/<code>pipe_quoted</code> mirror each other character for character, and <code>quote.rs</code> is the one Rust statement of that rule.</p>
|
||||||
|
<p>It lives in <code>improvise-formula</code> because that crate owns the grammar that says what a bare name <em>is</em>, and because every emitter needs it: a name that came from a user's data has to be spelled so the parser hands it back unchanged. <code>improvise-gp</code> re-exports it, <code>improvise-io</code> calls it from <code>persistence/{format,parse}.rs</code>, and <code>parser.rs::unquote_pipe</code> is a thin shim over <code>unescape_pipe_body</code>.</p>
|
||||||
|
<p>Before <a class="tid" href="../tickets.html#improvise-i2m5" title="improvise-io still carries its own pipe-quoting copy, without the keyword guard">improvise-i2m5</a> there were <strong>three</strong> copies and they had drifted, over the keyword rule below. <strong><code>improvise-io</code> no longer has one</strong> — its <code>persistence/quoting.rs</code> is deleted. <a href="../repo-map.html"><code>context/repo-map.md</code></a>'s file inventory kept claiming io "still has its own unguarded copy" long after the deletion — it was corrected in wave 1, but any <em>other</em> doc or comment still saying so is wrong, and that is the shape this particular fact goes stale in.</p>
|
||||||
|
<p>Three entry points, because the callers do not want the same thing:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>quote_name</code> — a name as the grammar wants it: bare when <code>is_bare_name</code> allows, else pipe-quoted.</li>
|
||||||
|
<li><code>pipe_quote</code> — unconditional. A <code>.improv</code> text cell goes through this, not through <code>quote_name</code>, because a bare <code>42</code> would read back as a number.</li>
|
||||||
|
<li><code>unescape_pipe_body</code> — the inverse, over the body between the delimiters (what both grammars' <code>pipe_inner</code> matches). The escape table is written once, in the last two.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it"><a class="anchor" href="#the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it">#</a>The keyword guard is canonical, user-confirmed — do not revert it</h3>
|
||||||
|
<p>An item literally named <code>SUM</code> is written <code>|SUM|</code>. <code>is_bare_name</code> is deliberately <strong>stricter than the grammar strictly requires</strong>: bare keywords (<code>WHERE SUM AVG MIN MAX COUNT IF IS</code>) <em>do</em> parse today, but only because <code>agg_call</code> needs a <code>(</code> and PEG backtracking then falls through to <code>ref_expr</code> — an emergent property of <strong>rule ordering</strong>, not a stated rule. A pipe-quoted name is an identifier in every position, with nothing to fall through, so a name colliding with a keyword is quoted and stops depending on that ordering.</p>
|
||||||
|
<p><code>IS</code> joined the list with <code>is_expr</code> (<a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>), and it is the one keyword whose omission would have been <em>silently</em> wrong rather than merely fragile: a category literally named <code>IS</code> written bare turns <code>IS IS East</code> into a predicate on a category called <code>East</code>. Adding a production to <code>formula.pest</code> means adding its keyword here — the two are one change.</p>
|
||||||
|
<p>The cost is worth paying because a <code>## Formulas</code> line carries formula text <strong>verbatim</strong>: a name <code>format_md</code> writes is later read by <code>parse_formula</code>. One namespace, one spelling rule.</p>
|
||||||
|
<h3 id="the-escape-table-and-the-one-input-outside-both-grammars"><a class="anchor" href="#the-escape-table-and-the-one-input-outside-both-grammars">#</a>The escape table, and the one input outside both grammars</h3>
|
||||||
|
<p><code>\|</code> → <code>|</code>, <code>\\</code> → <code>\</code>, <code>\n</code> → newline; any other <code>\X</code> is preserved verbatim. The trailing arms of <code>unescape_pipe_body</code> are unreachable from either grammar (<code>pipe_inner</code> is <code>("\\" ~ ANY | !"|" ~ ANY)*</code>, so a backslash is always followed by a character). They are written as a <strong>total function</strong> rather than a panic so a caller arriving by some other route gets its bytes back. The two pre-i2m5 unquoting copies differed <em>only</em> on a trailing lone backslash, where io returned it and <code>parser.rs</code> panicked; the survivor returns it.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="cat-is-item-a-coordinate-predicate-in-the-expression-grammar"><a class="anchor" href="#cat-is-item-a-coordinate-predicate-in-the-expression-grammar">#</a><code>Cat IS Item</code>: a coordinate predicate in the expression grammar</h2>
|
||||||
|
<p><code>is_expr = { identifier ~ ^"IS" ~ filter_value }</code> sits in <code>primary</code> <strong>before</strong> <code>ref_expr</code>, which would otherwise consume the category name and leave <code>IS Item</code> unparseable. A bare name still parses as a reference: <code>is_expr</code> needs the keyword, fails without it, and PEG backtracks into <code>ref_expr</code>.</p>
|
||||||
|
<p>It reuses <code>where_clause</code>'s two operands — an <code>identifier</code> for the category and a <code>filter_value</code> for the item — and therefore <code>build_filter</code>, which already reads exactly that pair. The <code>^"IS"</code> literal is a string in the grammar rather than a rule, so it contributes no pair and <code>into_inner()</code> yields the two operands and nothing else. One builder, one quoting rule, both halves through <code>improvise_formula::quote</code>.</p>
|
||||||
|
<p>The grammar-walking generator needs no new case: it reads <code>formula.pest</code> through <code>pest_meta</code> at test time, so a production added to <code>primary</code> is generated and round-tripped from the day it exists.</p>
|
||||||
|
<p>What the <em>evaluator</em> does with it, and why a fitted tree is always wrapped in a <code>SUM</code>, is <a href="../code/core.html"><code>context/code/core.md</code></a> §13.</p>
|
||||||
|
<h3 id="where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable"><a class="anchor" href="#where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable">#</a><code>WHERE</code> carries the same boundary, and the mechanism has to be generatable</h3>
|
||||||
|
<p><code>where_clause</code> and <code>inline_where</code> are compound-atomic with <code>sp+</code> after the keyword, exactly as <code>is_expr</code> is (<a class="tid" href="../tickets.html#improvise-r08u" title="IS and WHERE differ on trailing word boundary in formula.pest">improvise-r08u</a>). Non-atomic, pest auto-skipped <code>WHITESPACE</code> after <code>^"WHERE"</code>, so <code>SUM(Rev WHEREAS x = y)</code> read as a <code>WHERE</code> on a category <code>AS</code> — a silently wrong filter where it should have been a syntax error, the same class of defect <code>IS</code> had.</p>
|
||||||
|
<p>Two things decided the shape, and the second is the one worth carrying. A negative lookahead <code>!bare_char</code> with <code>sp*</code> would have been <em>strictly</em> more permissive — it keeps <code>WHERE|Region|=East</code>, which the old rule accepted — but <strong>the grammar-walking generator cannot honour a predicate</strong>: it walks <code>formula.pest</code> through <code>pest_meta</code> and emits <code>WHEREa=""</code>, text the grammar then refuses, so <code>generated_formula_parses</code> goes red. A rule whose only guard is a lookahead is a rule the generator can violate. <code>sp+</code> is generatable, matches <code>IS</code>, and costs exactly one accepted spelling: a category may no longer abut the keyword, so <code>WHERE |Region|</code> needs its space.</p>
|
||||||
|
<p>Note compound-atomicity <em>alone</em> closes the ticket's own reproducer, because <code>AS x</code> cannot be one identifier once implicit whitespace is gone. <code>sp+</code> is what closes the residue, <code>WHEREAS=y</code>, and the negative test names both.</p>
|
||||||
|
<h2 id="a-formula-target-is-unquoted-exactly-as-a-reference-is"><a class="anchor" href="#a-formula-target-is-unquoted-exactly-as-a-reference-is">#</a>A formula target is unquoted, exactly as a reference is</h2>
|
||||||
|
<p><code>parse_formula</code> strips the pipes off the target the same way it does off an <code>Expr::Ref</code>, because the target <strong>names a <code>_Measure</code> item</strong>. A target that kept its delimiters made <code>Model::add_formula</code> create an item literally called <code>|Total Revenue|</code> — a ghost column the user's real item never matched, and that <code>Expr::Ref("Total Revenue")</code> never resolved to.</p>
|
||||||
|
<p>Both the TUI formula minibuffer (<code>AddFormula</code>) and <code>.improv</code> load go through that one call, so both were affected. The writer emits <code>Formula::raw</code>, never <code>target</code>, so the file text is unaffected either way — <strong>which is exactly why the round-trip tests never saw it</strong>, and why the fix repairs existing files on next load with no migration.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="the-improv-grammar-is-the-single-source-of-truth"><a class="anchor" href="#the-improv-grammar-is-the-single-source-of-truth">#</a>The <code>.improv</code> grammar is the single source of truth</h2>
|
||||||
|
<p><code>persistence/improv.pest</code> defines the format; <code>parse.rs</code> is a tree-walker over pest's parse tree, not a line scanner. Adding a format feature means changing the grammar first.</p>
|
||||||
|
<p>The test generator (<code>parse.rs::generator</code>, and its twin in <code>parser.rs::generator</code> for <code>formula.pest</code>) reads the grammar's AST at test time via <code>pest_meta</code> and emits random valid input. <strong>A new production therefore needs generator coverage</strong>, or the proptests quietly stop describing the format: an <code>Expr</code> variant the emitter does not handle produces nothing, and <code>generated_file_parses</code> then passes over input that never exercised the new rule. <code>formula.pest</code>'s generator is explicit about this — lookaheads emit nothing on purpose, and any unused <code>Expr</code> variant falls through the <code>_ => {}</code> arm; if the grammar starts using one, generated formulas fail to parse and the gap surfaces.</p>
|
||||||
|
<p>Two generator details worth not re-deriving. The <code>atomic</code> flag in the formula generator controls whether a <code>Seq</code> inserts a space between children, so non-atomic rules produce the whitespace-separated tokens pest's implicit <code>WHITESPACE</code> accepts. And <code>Choice</code> is 50/50 between <em>two</em> branches, so rules with three or more alternatives (nested <code>Choice</code>s) pick deeper branches less often — good enough for random generation, not a uniform distribution.</p>
|
||||||
|
<h3 id="the-tree-walkers-are-infallible-by-construction"><a class="anchor" href="#the-tree-walkers-are-infallible-by-construction">#</a>The tree walkers are infallible by construction</h3>
|
||||||
|
<p>Every <code>build_*</code> in <code>parser.rs</code> operates on a <code>Pair</code> pest has already validated. "A <code>formula</code> has exactly one <code>target</code>", "a <code>comparison</code> has lhs, op and rhs" — pest guarantees these before the walker sees the pair. So every <code>.expect(GRAMMAR_INVARIANT)</code> is a <strong>bug marker</strong>, not a runtime condition: if one fires, the grammar and the walker have diverged. The same reasoning makes the final <code>else</code> arm of each dispatch unlabelled-but-safe (<code>ref_expr</code> is the only remaining alternative of <code>primary</code>, <code>COUNT</code> the only remaining <code>agg_func</code>, <code>>=</code> the only remaining <code>cmp_op</code>).</p>
|
||||||
|
<p><code>parse.rs</code>'s walkers are the opposite and deliberately so: they return <code>Result</code> and name the section in the error, because the <code>.improv</code> file is hand-editable and a failure there is a user's problem, not a programmer's. <code>parse_tree</code> renders pest's error as a located message (line number plus the offending line), because "Parse error: {pest dump}" is unusable to somebody fixing a file by hand.</p>
|
||||||
|
<h3 id="two-passes-because-sections-may-appear-in-any-order"><a class="anchor" href="#two-passes-because-sections-may-appear-in-any-order">#</a>Two passes, because sections may appear in any order</h3>
|
||||||
|
<p>Pass 1 turns each <code>## </code> section into a plain collector (<code>PCategory</code>, <code>PView</code>) or a flat list; pass 2 builds the <code>Workbook</code>. That split is what makes the format order-independent — a <code>## View:</code> may name a category defined further down. <code>PCategory::items</code> is a <code>Vec</code> in file order because <code>IndexMap</code> insertion order is load-bearing (gotcha 18), so pass 2 must replay the list as written.</p>
|
||||||
|
<p>Two <code>## Category:</code> sections naming one category are <strong>one</strong> category, and the merge is a concatenation performed in pass 1 (<code>ParsedFile::push_category</code>, <a class="tid" href="../tickets.html#improvise-349m" title="format_md/parse_md is not idempotent when a .improv file has two Category sections with the same name">improvise-349m</a>). Merging in pass 2 — building each section in turn — was not idempotent. A group is created either by the item that names it or by a <code>></code> line, and <code>build_category</code> creates its own section's item-groups <em>before</em> its own <code>></code> lines; so the first section's <code>></code> groups landed ahead of the second section's item-groups, while <code>format_md</code> writes one section with all the items before all the <code>></code> lines. The reparse then built the other order, and the two <code>format_md</code> outputs differed. Concatenating first makes the built order the written order by construction. Refusing a duplicate header was the alternative and would have made a legal hand-written file an error for no gain.</p>
|
||||||
|
<h3 id="a-section-header-takes-its-name-as-the-whole-line-totally"><a class="anchor" href="#a-section-header-takes-its-name-as-the-whole-line-totally">#</a>A section header takes its name as the whole line, totally</h3>
|
||||||
|
<p><code>header_name = { pipe_quoted ~ &NEWLINE | line_name }</code> (<a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a>). The <code>&NEWLINE</code> lookahead is load-bearing: without it, <code>## Category: |A|B</code> parses as <code>A</code> and <strong>silently drops <code>B</code></strong>. <code>line_name</code> catches everything else, so a header name can no longer fail to parse. Four rules share it — <code>model_name</code>, <code>initial_view</code>, <code>category_section</code>, <code>view_section</code> — which is why it is one rule and not four copies.</p>
|
||||||
|
<p>The writing half is <code>format.rs::quote_header</code>, and it is <strong>deliberately not <code>quote_name</code></strong>: that rule is for a name embedded among other tokens, and applying it to a header would pipe-quote every ordinary spaced title the line carries perfectly well, rewriting the header of every existing document for no correctness gain. <code>quote_header</code> quotes exactly four things, each one thing the <em>reader</em> does:</p>
|
||||||
|
<ul>
|
||||||
|
<li>the empty name, and any name with leading or trailing whitespace — the reader trims, so the category comes back renamed while every data coordinate still names the untrimmed original: silent orphaning;</li>
|
||||||
|
<li>a name containing <code>\n</code> or <code>\r</code> — it would start a second line, and <code>Region\n## Data</code> a second <em>section</em>;</li>
|
||||||
|
<li>a name already spelled <code>|…|</code> — the reader would take the pipes off.</li>
|
||||||
|
</ul>
|
||||||
|
<p>So a header is <em>written</em> raw where a reference is quoted. The grammar accepts both arms, so that asymmetry is a <strong>writer convention, not a format rule</strong>.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="storage-is-whole-document-read-write-and-that-closed-the-sqlite-question"><a class="anchor" href="#storage-is-whole-document-read-write-and-that-closed-the-sqlite-question">#</a><code>Storage</code> is whole-document read/write, and that closed the SQLite question</h2>
|
||||||
|
<p>Saving or loading a workbook only ever touches one whole document, so a backend needs no seeking, metadata or partial reads (<a class="tid" href="../tickets.html#improvise-6mq" title="Introduce Storage/VFS abstraction in persistence layer">improvise-6mq</a>). It does need to enumerate and remove documents, because a browser build has no OS file manager behind it — hence <code>list</code> and <code>delete</code> beside <code>read</code> and <code>write</code> (<a class="tid" href="../tickets.html#improvise-i34" title="OPFS-backed Storage implementation for wasm target">improvise-i34</a>).</p>
|
||||||
|
<p>Paths are <strong>opaque document names, not a directory tree</strong>: a backend may treat them as filesystem paths, OPFS keys or map entries. The trait doc says "backends that are not trees treat <code>dir</code> as a name prefix" — keep that sentence reachable. It is the sentence that turned out to be <em>right while the <code>OpfsFs</code> impl was wrong</em>, and the bug was found by reading it.</p>
|
||||||
|
<p>Because the trait is two ops over whole documents, an <code>sqlite3_vfs</code> — which needs <code>xRead(offset,len)</code>, <code>xWrite</code>, <code>xTruncate</code>, <code>xLock</code> — has nothing to bind to. See gotcha 41 for the measurements that declined <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>.</p>
|
||||||
|
<h3 id="storage-contract-is-a-public-function-not-a-test-helper"><a class="anchor" href="#storage-contract-is-a-public-function-not-a-test-helper">#</a><code>storage_contract</code> is a public function, not a test helper</h3>
|
||||||
|
<p>It reads like a test helper and it <em>was</em> one: <code>#[cfg(test)]</code> in <code>storage.rs</code>, run against <code>MemoryFs</code> and <code>PhysicalFs</code> so they could not drift. The one backend that most needed it could never be handed to it — <code>OpfsFs</code> is <code>cfg(target_arch = "wasm32")</code> and its sync access handles exist in no host harness — so for that backend the contract lived only in a doc comment.</p>
|
||||||
|
<p>Moving it out of <code>cfg(test)</code> is what lets a browser run <strong>this</strong> sequence rather than a second transcription of it (<a class="tid" href="../tickets.html#improvise-vvl3" title="A headless browser harness: run storage_contract and the shell assertions under Playwright">improvise-vvl3</a>/xf1). A second copy of a contract is the disease this codebase keeps curing. It returns <code>Result</code> rather than asserting because a failure has to travel out of a Web Worker as a string; the host tests <code>.unwrap()</code> it and get the same message.</p>
|
||||||
|
<p><strong>Two bugs fell out of its first actual execution</strong> — see the <code>dir</code> parameter and the index rules below.</p>
|
||||||
|
<p><code>dir</code> is a parameter, not a constant, because the three backends must agree at <strong>every</strong> <code>dir</code>, not only at the flat root the browser happens to use (<a class="tid" href="../tickets.html#improvise-vhcz" title="[bug] OpfsFs::list disagrees with the other backends on path shape">improvise-vhcz</a>). A backend whose <code>list</code> answered in a different path shape passes at <code>Path::new("")</code> — where joining is the identity — and fails at anything else.</p>
|
||||||
|
<h3 id="impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule"><a class="anchor" href="#impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule">#</a><code>impl Storage for Rc<T></code> lives in <code>improvise-io</code> because of the orphan rule</h3>
|
||||||
|
<p><code>OpfsFs</code> keeps four genuinely async operations — <code>open</code>, <code>attach</code>, <code>refresh</code>, <code>commit_deletes</code> — off the trait, so <code>Storage</code> can stay synchronous (gotcha 28). But <code>App::storage</code> is a <code>Box<dyn Storage></code>, and a trait object can name none of the four: <strong>handing the backend to the <code>App</code> puts the async seam permanently out of reach</strong>. The host must hold the concrete backend <em>and</em> give the <code>App</code> a <code>Storage</code>, which means sharing one.</p>
|
||||||
|
<p><code>impl Storage for Rc<OpfsFs></code> is two foreign types in any other crate, so <a class="tid" href="../tickets.html#improvise-9x6" title="Worker-hosted server bundle (wasm, MessageChannel transport)">improvise-9x6</a> wrote a forwarding newtype in the worker and every future worker would have written it again. Here the trait is local, so the impl is allowed, and it is <strong>blanket</strong> rather than <code>OpfsFs</code>-specific because nothing about it is OPFS-shaped: it is the general statement that a shared handle to a backend is usable as that backend (<a class="tid" href="../tickets.html#improvise-hmws" title="Storage cannot expose an async seam through Box<dyn Storage>">improvise-hmws</a>). <code>Rc</code> and not <code>Arc</code> because the one caller is a Web Worker — a single thread, and <code>wasm32-unknown-unknown</code> has no others. An <code>Arc</code> impl would be speculative.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds"><a class="anchor" href="#opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds">#</a><code>OpfsFs</code>: why a browser backend can be synchronous, and what the index holds</h2>
|
||||||
|
<p>OPFS has two APIs. The promise-based one (<code>getFileHandle</code>, <code>createWritable</code>) is async, but <code>FileSystemFileHandle::createSyncAccessHandle()</code> yields a handle whose <code>read</code>/<code>write</code>/<code>truncate</code>/<code>flush</code>/<code>close</code> are ordinary <strong>blocking</strong> calls. It exists so wasm can do blocking file I/O — it is what SQLite's wasm build uses — and is available <strong>only inside a dedicated Web Worker</strong>, which is where <a class="tid" href="../tickets.html#improvise-9x6" title="Worker-hosted server bundle (wasm, MessageChannel transport)">improvise-9x6</a> puts the whole app. So <code>Storage</code> never had to become async.</p>
|
||||||
|
<p><em>Acquiring</em> a handle is still async, as are directory listing and removal, so the backend splits in two:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>async, explicit</strong>: <code>open</code>, <code>attach</code>, <code>refresh</code>, <code>commit_deletes</code>, called by the worker from its message loop where awaiting is free;</li>
|
||||||
|
<li><strong>sync, via the trait</strong>: <code>read</code>/<code>write</code> use an already-attached handle; <code>list</code> answers from an index; <code>delete</code> drops the name from that index and queues the removal.</li>
|
||||||
|
</ul>
|
||||||
|
<p>So a <code>delete</code> is visible to <code>read</code> and <code>list</code> immediately and lands in OPFS at the end of the batch. That ordering is the price of a synchronous trait and is invisible to a single worker. <code>write</code> therefore <strong>un-queues</strong> any pending delete of the same name: otherwise delete-then-write would be listed and readable and then removed by <code>commit_deletes</code> at the end of the batch — the one ordering the deferral can get wrong.</p>
|
||||||
|
<h3 id="the-index-holds-documents-not-entries"><a class="anchor" href="#the-index-holds-documents-not-entries">#</a>The index holds documents, not entries</h3>
|
||||||
|
<p><code>Storage::list</code> promises <em>document</em> names, and documents and entries stopped being the same thing the moment <code>attach</code> had to create a file to get a handle for a document that does not exist yet (<a class="tid" href="../tickets.html#improvise-vvl3" title="A headless browser harness: run storage_contract and the shell assertions under Playwright">improvise-vvl3</a>). Three writers, and the split between them is the whole rule:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong><code>attach</code> indexes nothing.</strong> What it creates is zero bytes. Indexing there made a fresh origin list <code>demo.improv</code> before anything had been written, so the seed was skipped as "already there" and <code>load_from</code> got an empty buffer.</li>
|
||||||
|
<li><strong><code>Storage::write</code> indexes the name</strong> (and un-queues a pending delete). A written document is the first thing there is to list.</li>
|
||||||
|
<li><strong><code>refresh</code> rebuilds from OPFS, skipping zero-byte entries</strong> — and counting an entry it cannot <em>measure</em> as a document, since the one cause of that is another tab holding the exclusive handle.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>holds_bytes</code> is where the last clause lives, and its two failure modes get <strong>opposite answers</strong>, which is the whole function:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>getFileHandle</code> fails → the name is not a file. <code>keys()</code> yields directories too, and a directory rejects with <code>TypeMismatchError</code>; an entry removed between <code>keys()</code> and here rejects as well. Neither is a document and neither can ever be attached or read, so listing one is the same picker bug this filter exists to remove (Copilot, PR #24).</li>
|
||||||
|
<li><code>getFile</code> fails on a real file handle → another context of the same origin holds the exclusive sync access handle (two tabs). An entry somebody else has open is certainly a document, so this <strong>fails open</strong>. Dropping it would hide a real document; failing the whole <code>refresh</code> would let a second tab break the first one's picker. Over-reporting costs one row — the direction <code>Dirty::of</code> is deliberately sloppy in.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Handles are exclusive per file, so <code>attach</code> closes any handle it replaces and <code>detach</code> releases the lock.</p>
|
||||||
|
<h3 id="list-joins-dir-like-the-other-backends"><a class="anchor" href="#list-joins-dir-like-the-other-backends">#</a><code>list</code> joins <code>dir</code> like the other backends</h3>
|
||||||
|
<p>The join is not cosmetic (<a class="tid" href="../tickets.html#improvise-vhcz" title="[bug] OpfsFs::list disagrees with the other backends on path shape">improvise-vhcz</a>). <code>MemoryFs</code> and <code>PhysicalFs</code> answer with <code>dir</code>-joined paths — paths a caller can hand straight back to <code>read</code> — and <code>OpfsFs</code> used to answer with bare names. At <code>Path::new("")</code> the two shapes coincide, which is the only <code>dir</code> the worker ever passes, so the disagreement was invisible: a worker that organised documents under <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/docs"><code>docs/</code></a> would have got a <strong>full picker in every host test and an empty one in a browser</strong>, with nothing saying why.</p>
|
||||||
|
<p><code>dir</code> is decoration rather than a filter, because this backend is flat by construction: <code>entry_name</code> reduces every path to its file name, so <code>docs/a.improv</code> and <code>a.improv</code> are the same OPFS entry and an entry name may not contain a separator at all. That is the trait's own "treat <code>dir</code> as a name prefix" — the doc was right and the impl was wrong.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="loading-decompression-and-decoding-are-two-failures-with-two-answers"><a class="anchor" href="#loading-decompression-and-decoding-are-two-failures-with-two-answers">#</a>Loading: decompression and decoding are two failures with two answers</h2>
|
||||||
|
<p><code>load_from</code> decompresses, <em>then</em> decodes UTF-8, with a separate <code>Context</code> on each. <code>read_to_string</code> would fuse them, and a valid gzip stream holding invalid UTF-8 would then be reported as unreadable gzip — a wrong diagnosis of a document that decompressed perfectly well. Decoding once, afterwards, also leaves <strong>one</strong> answer to "is this UTF-8" rather than one per branch.</p>
|
||||||
|
<p>The <code>.gz</code> suffix selects gzip regardless of backend: that is a property of the document name, not of the filesystem (<a class="tid" href="../tickets.html#improvise-6mq" title="Introduce Storage/VFS abstraction in persistence layer">improvise-6mq</a>). Legacy JSON is detected by a leading <code>{</code> and is never written.</p>
|
||||||
|
<p><code>parse_number_cell</code> returns <code>CellValue::Error</code> naming the offending text rather than a fabricated <code>0.0</code>. No token the current grammar admits actually fails <code>f64::from_str</code> (overflow saturates to ±inf), but if the two ever drift the mismatch must <strong>surface</strong> — never silently store a wrong number.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="persistence-write-order-and-the-virtual-views"><a class="anchor" href="#persistence-write-order-and-the-virtual-views">#</a>Persistence write order and the virtual views</h2>
|
||||||
|
<p><code>format_md</code> writes Views → Formulas → Categories → Data: views first because they are typically small and orient the reader, data last because it is the largest section. The parser tolerates any order (see the two-pass note above).</p>
|
||||||
|
<p><code>_Records</code> and <code>_Drill</code> are scratch views the app rebuilds on demand. They put <code>_Index</code>/<code>_Dim</code> on axes, and those categories are never written — so persisting the views would produce axis lines naming categories the loader will not find. They are skipped on write, and <strong>dropped on load</strong> if an older file carries them, rather than reinstating a view the app would rebuild anyway; <code>Initial View</code> then falls through to the default, since the name it pointed at is gone. <code>persisted_initial_view</code> is the writing half: saving from records or drill mode falls back to the first real view rather than naming one the file will not contain (<a class="tid" href="../tickets.html#improvise-60z" title="Virtual views _Records and _Drill should not be persisted to .improv files">improvise-60z</a>).</p>
|
||||||
|
<p><code>_Measure</code> persists only its non-formula items — formula targets are rebuilt from the <code>## Formulas</code> section, so <code>format_md</code> collects the targets and excludes them.</p>
|
||||||
|
<p><code>Page</code> and <code>Filter</code> both carry the selected item after the axis keyword; the other axes have nothing to select. <code>Axis::Filter</code> round-trips even though it occurs only inside the never-written <code>_Drill</code> view, because the alternative was silently writing a Filter axis as some other axis (<a class="tid" href="../tickets.html#improvise-dz8" title="Add Axis::Filter for drill-fixed coordinates">improvise-dz8</a>, gotcha 27).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="import-order-is-a-correctness-property-not-tidiness"><a class="anchor" href="#import-order-is-a-correctness-property-not-tidiness">#</a>Import: order is a correctness property, not tidiness</h2>
|
||||||
|
<p><code>analyzer::analyze_records</code> emits distinct values in <strong>first-seen order</strong>, never <code>HashSet</code> order (<a class="tid" href="../tickets.html#improvise-7yx" title="Close test gaps flagged by deep review">improvise-7yx</a>, gotcha 18). <code>build_model</code> adds items in exactly that order and <code>IndexMap</code> preserves it, so a hash order made the <em>same CSV</em> import to a different document on every run — different <code>.improv</code> bytes, different row order on screen, and a different detected date format, since <code>detect_date_format</code> only inspects the first ten samples and <em>which</em> ten it saw varied. The <code>HashSet</code> in <code>analyze_records</code> is there to deduplicate and <strong>never to iterate</strong>.</p>
|
||||||
|
<h3 id="the-ambiguous-date-heuristic-reads-american-first-deliberately"><a class="anchor" href="#the-ambiguous-date-heuristic-reads-american-first-deliberately">#</a>The ambiguous-date heuristic reads American-first, deliberately</h3>
|
||||||
|
<p><code>01/02/2025</code> parses under both <code>%m/%d/%Y</code> and <code>%d/%m/%Y</code> and nothing in the data can tell them apart. <code>detect_date_format</code> is first-match over <code>DATE_FORMATS</code>, so the American reading wins. That is <em>a</em> defensible answer and not obviously the right one: a European CSV imports with days and months swapped, <strong>silently</strong>, and the derived <code>_Month</code>/<code>_Quarter</code> categories inherit the error. What makes it tolerable is that the wizard shows the detected format and lets the user change it — the same argument two-digit years already rested on. It is pinned by a test so that changing the answer is a decision somebody makes rather than a side effect of reordering the table.</p>
|
||||||
|
<p><strong>Ruled a known limit</strong> (<a class="tid" href="../tickets.html#improvise-l8rh" title="The ambiguous-date heuristic reads American-first, so a European CSV imports with days and months swapped">improvise-l8rh</a>, the user, 2026-09-20). Two things the ruling rests on. Changing the default to day-first only moves which half of the world imports silently wrong; and the honest fix is larger than the heuristic — an ambiguous column should <em>surface</em> its ambiguity rather than resolve it, which changes <code>FieldProposal</code>'s shape and the wizard step that displays it, the way <code>Dataset::warnings</code> returns a decision the user should hear about as data. Worth doing deliberately, not as a patch.</p>
|
||||||
|
<p>What makes the limit reasonable to live with is that it is now <em>deterministic</em>: <code>analyze_records</code> iterated a <code>HashSet</code> until <a class="tid" href="../tickets.html#improvise-7yx" title="Close test gaps flagged by deep review">improvise-7yx</a>, so the ten samples <code>detect_date_format</code> inspects were a different ten every run and a column whose eleventh value ruled out <code>%m/%d/%Y</code> detected a different format run to run. The ten-sample window itself is unchanged and is part of the limit.</p>
|
||||||
|
<h3 id="csv-parsing"><a class="anchor" href="#csv-parsing">#</a>CSV parsing</h3>
|
||||||
|
<p>Short rows keep every header column, missing trailing fields filled with <code>Value::Null</code>, and each affected column is reported as a warning with a row count. The reader is <code>.flexible(true)</code>, so without this a short row parsed cleanly and the trailing columns were simply <em>absent</em> from the record map — no error, no warning, and skewed value counts in field-kind analysis (<a class="tid" href="../tickets.html#improvise-k8i" title="CSV import silently drops columns when row widths mismatch">improvise-k8i</a>).</p>
|
||||||
|
<p>A header name becomes a <strong>category name</strong> and survives into the <code>.improv</code> file, so anything riding along invisibly in one — a byte-order mark, a stray carriage return — is silent corruption rather than cosmetics. That is why the adversarial inputs are tested with <em>controls</em>: a check that cannot see a BOM where there is one proves nothing.</p>
|
||||||
|
<p><code>parse_csv_bytes_with_warnings</code> exists so CSV import can come from any <code>Storage</code> backend rather than only a file (<a class="tid" href="../tickets.html#improvise-fuj" title="Two effects read files directly instead of going through Storage">improvise-fuj</a>); the path-taking wrappers remain for the native CLI.</p>
|
||||||
|
<h3 id="the-wizard-s-back-path-is-a-history-stack"><a class="anchor" href="#the-wizard-s-back-path-is-a-history-stack">#</a>The wizard's back path is a history stack</h3>
|
||||||
|
<p><code>advance</code> pushes the step it leaves; <code>previous</code> pops it. Keeping the path actually taken, rather than re-deriving the reverse of each conditional transition, means back can never disagree with how the user really got here (<a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a>).</p>
|
||||||
|
<p>Preconditions live in <code>validation_error()</code> and are checked <strong>on the step where the user can still fix them</strong> — never deferred to <code>build_model</code> after the final confirm. Two gates earn that: too many categories (<a class="tid" href="../tickets.html#improvise-mzv" title="Import wizard does not pre-check 12-category limit">improvise-mzv</a>) and no category at all (<a class="tid" href="../tickets.html#improvise-gfq" title="Import wizard defers the 'no category accepted' failure to the very last keystroke">improvise-gfq</a>); both are fixable with <code>c</code> at <code>ReviewProposals</code>, and both used to refuse only at the last keystroke. <code>MAX_CATEGORIES</code> here mirrors the model's private limit and counts regular categories only.</p>
|
||||||
|
<p><code>NameModel</code> is terminal for <code>advance</code>: leaving it is a <strong>build</strong>, not a step change. <code>build()</code> parks the workbook in <code>built</code> on <code>Done</code>; a build failure leaves the user on <code>NameModel</code> with the error. Going back from <code>Done</code> discards <code>built</code>, so accepting an import always reflects the latest field decisions.</p>
|
||||||
|
<p><code>scalar_name</code> decides what a JSON scalar contributes as an item name. <code>Value::Null</code> — which is also what the CSV parser produces for a blank field — and a blank string are <strong>absences, not names</strong>; stringifying them used to invent an item literally called "null" (<a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item "null" on import">improvise-ych</a>). Every other value keeps its printed form.</p>
|
||||||
|
<p>A record missing a value for an accepted Category field has no place in the grid and is dropped; <code>build_model_with_warnings</code> counts it against the <em>first</em> category it is missing and reports the total, because without a warning that loss is invisible in a thousand-row import (<a class="tid" href="../tickets.html#improvise-5j9" title="ImportPipeline silently skips records missing category values">improvise-5j9</a>).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="csv-export"><a class="anchor" href="#csv-export">#</a>CSV export</h2>
|
||||||
|
<p><code>render_csv</code> is separated from writing so the bytes can go to any <code>Storage</code> backend, not just a file (<a class="tid" href="../tickets.html#improvise-fuj" title="Two effects read files directly instead of going through Storage">improvise-fuj</a>). It goes through the <code>csv</code> crate rather than joining with commas, and it honours the view's number format — a value of 1200 under a comma-grouping format renders <code>1,200</code>, and a label like <code>Smith, Jr.</code> splits into two fields, corrupting every row (<a class="tid" href="../tickets.html#improvise-1cz" title="export_csv emits unquoted CSV — commas in values corrupt output">improvise-1cz</a>).</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>gp-tree and improvise-gp — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html" class="here">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/gp.md">context/code/gp.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#gp-tree-and-improvise-gp">gp-tree and improvise-gp</a></li><li class="l2"><a href="#1-two-crates-one-seam">1. Two crates, one seam</a></li><li class="l2"><a href="#2-gp-tree-returns-data-and-never-prints">2. gp-tree returns data and never prints</a></li><li class="l2"><a href="#3-the-vendored-rng-and-why-its-constants-are-a-promise">3. The vendored RNG, and why its constants are a promise</a></li><li class="l2"><a href="#4-spec-ranges-is-a-soundness-precondition">4. Spec::ranges is a soundness precondition</a></li><li class="l2"><a href="#5-row-an-invariant-made-unrepresentable">5. Row<T>: an invariant made unrepresentable</a></li><li class="l2"><a href="#6-folds-and-cv-folds-answer-different-questions">6. folds and cv_folds answer different questions</a></li><li class="l2"><a href="#7-standardize-denormalize-the-affine-map-must-be-folded-back">7. standardize / denormalize: the affine map must be folded back</a></li><li class="l2"><a href="#8-regressionquality-carries-its-own-baseline">8. RegressionQuality carries its own baseline</a></li><li class="l2"><a href="#9-a-fit-takes-two-things-to-reproduce-and-here-four">9. A fit takes two things to reproduce — and here, four</a></li><li class="l2"><a href="#10-the-emitter-quoting-and-the-threshold-s-exact-decimal">10. The emitter: quoting, and the threshold's exact decimal</a></li><li class="l2"><a href="#11-a-fit-is-storable-only-when-the-document-can-name-its-features">11. A fit is storable only when the document can name its features</a></li><li class="l3"><a href="#features-one-entry-per-feature-index-read-off-the-plan">Features: one entry per feature index, read off the plan</a></li><li class="l3"><a href="#what-fitted-formula-still-asks-the-document">What fitted_formula still asks the document</a></li><li class="l3"><a href="#the-two-families-improvise-wd6n-declines-separately">The two families improvise-wd6n declines, separately</a></li><li class="l2"><a href="#12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na">12. A fit is persisted as its formula, and nothing else (improvise-7na)</a></li><li class="l2"><a href="#13-dataset-rs-what-a-view-shows-as-a-feature-matrix">13. dataset.rs: what a view shows, as a feature matrix</a></li><li class="l2"><a href="#14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess">14. improvise-gp::fit: budgets, and the enum that refuses to guess</a></li><li class="l3"><a href="#two-budgets-and-why-the-preflight-is-advisory">Two budgets, and why the preflight is advisory</a></li><li class="l3"><a href="#forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one">Forcing a numeric category is a representability choice, not a budget one</a></li><li class="l3"><a href="#quality-is-an-enum-not-an-option">Quality is an enum, not an Option</a></li><li class="l3"><a href="#fitted-measure-name-a-fit-is-a-rule-about-the-data">fitted_measure_name — a fit is a rule about the data</a></li><li class="l3"><a href="#one-line-is-public-because-the-refusals-need-it-too">one_line is public because the refusals need it too</a></li><li class="l2"><a href="#15-known-gaps-filed-rather-than-fixed">15. Known gaps, filed rather than fixed</a></li><li class="l2"><a href="#16-preprocess-what-each-column-kind-expands-into">16. preprocess: what each column kind expands into</a></li><li class="l2"><a href="#17-fit-csv-the-cli-s-own-documentation">17. fit_csv: the CLI's own documentation</a></li></ul></details><h1 id="gp-tree-and-improvise-gp"><code>gp-tree</code> and <code>improvise-gp</code></h1>
|
||||||
|
<p>The reasoning that used to live in the doc comments of <code>crates/gp-tree/**</code> and <code>crates/improvise-gp/**</code>. Written for a reader with the code open: the code says what happens, this says why, and why the obvious alternative is wrong.</p>
|
||||||
|
<p>Bug narratives keyed by test name are in <a href="../test-evidence/gp.html"><code>context/test-evidence/gp.md</code></a>. The condensed version is repo-map gotcha 37.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="1-two-crates-one-seam"><a class="anchor" href="#1-two-crates-one-seam">#</a>1. Two crates, one seam</h2>
|
||||||
|
<p><code>gp-tree</code> is problem-agnostic: a <code>Tree</code> is a regression tree over a fixed-width <code>f32</code> feature vector, and the crate owns the representation, the GP operators, <code>simplify</code>, <code>render</code>, one generation (<code>breed</code>), the search (<code>fit</code>), and the optional raw-table bridge (<code>data</code>). It knows nothing about <em>what</em> the features mean or <em>how</em> a candidate is scored — the caller supplies a <code>Spec</code> (per-feature ranges and names) and, outside <code>fit</code>, its own fitness. Determinism is the caller's too: the operators take an <code>Rng</code> trait implemented over a seeded generator.</p>
|
||||||
|
<p><code>improvise-gp</code> is the one seam between that crate and improvise. It cannot live in <code>gp-tree</code> — that inverts the dependency and drags improvise into a generic GP crate — and it cannot live in <code>improvise-formula</code>, which must not know GP exists. It holds three things:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>lib.rs</code> — the emitter: a fitted <code>Tree</code> as improvise formula text.</li>
|
||||||
|
<li><code>dataset.rs</code> — a <code>View</code>'s records as the feature matrix a fit consumes, the improvise-side counterpart of <code>preprocess</code>.</li>
|
||||||
|
<li><code>fit.rs</code> — one fit end to end: records in, an improvise <code>Formula</code> plus the evidence for believing it out.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>fit.rs</code> is here rather than in a <code>Cmd</code> because none of it is terminal-specific. A browser fitting the same measure over the same document must run exactly these steps in exactly this order, or it gets a different tree from the same seed. <code>improvise-command/src/command/cmd/fit.rs</code> is the thin terminal wrapper over it.</p>
|
||||||
|
<h2 id="2-gp-tree-returns-data-and-never-prints"><a class="anchor" href="#2-gp-tree-returns-data-and-never-prints">#</a>2. <code>gp-tree</code> returns data and never prints</h2>
|
||||||
|
<p>A decision the user should hear about comes back as <strong>data</strong>: <code>Dataset::warnings</code> carries the columns <code>preprocess</code> dropped for being entirely missing, <code>fit.rs</code> hands back pooled <code>(prediction, actual)</code> pairs and a score per feature rather than tables. <code>examples/fit_csv.rs</code> owns every <code>println!</code> and every <code>eprintln!</code>, and that is the only reason its output stayed byte-identical across these moves. <strong>Adding an <code>eprintln!</code> to <code>fit.rs</code> or <code>data.rs</code> is the regression</strong>, not a convenience.</p>
|
||||||
|
<p>The reason is not tidiness: a library does not own its caller's stderr. A TUI writing to an alternate screen, a wasm build, or a caller that wants the notice in its own status line can none of them suppress, redirect or translate an <code>eprintln!</code>. <code>preprocess</code> used to do exactly that, which was correct while this code <em>was</em> <code>examples/fit_csv.rs</code>.</p>
|
||||||
|
<p><code>improvise-gp</code> inherits the rule. <code>FitOutcome::status_line</code> is a <em>rendering</em> of the outcome data, offered so a second front end does not compose its own — the same argument that put <code>flatten_cat_tree</code> in the shared layer — and it is a returned <code>String</code>, never a <code>println!</code>.</p>
|
||||||
|
<h2 id="3-the-vendored-rng-and-why-its-constants-are-a-promise"><a class="anchor" href="#3-the-vendored-rng-and-why-its-constants-are-a-promise">#</a>3. The vendored RNG, and why its constants are a promise</h2>
|
||||||
|
<p><code>SplitMix64</code> is ~20 lines and no dependency. The <code>Rng</code> trait exists so a caller can bring its own generator, and every caller so far wrote out this same one.</p>
|
||||||
|
<p>A fit is not a throwaway: it writes a rule into a user's document, so re-running it on the same data must produce the same rule, and a bug report must be reconstructible from its seed alone. That is worth vendoring rather than depending — the point is reproducibility, not statistical quality, and SplitMix64's is ample for picking nodes and thresholds.</p>
|
||||||
|
<p><code>SplitMix64::seed()</code> keeps the seed the run started from, unchanged as the generator advances, so a caller can show it and record it. A generator that has advanced past its seed cannot answer that question.</p>
|
||||||
|
<p><code>next_u64</code> is public because <code>splitmix64_matches_the_reference_vectors</code> pins it. <strong>The constants are the algorithm, not an implementation detail</strong>: every seed already printed by a fit and every <code>data::Model</code> seed already saved is a promise that <em>these</em> numbers come back.</p>
|
||||||
|
<h2 id="4-spec-ranges-is-a-soundness-precondition"><a class="anchor" href="#4-spec-ranges-is-a-soundness-precondition">#</a>4. <code>Spec::ranges</code> is a soundness precondition</h2>
|
||||||
|
<p><code>ranges[i]</code> must be a <strong>superset</strong> of the values feature <code>i</code> can actually take. A too-narrow range makes <code>simplify</code> prune a branch that is reachable, silently changing the tree's outputs. <code>preprocess</code> derives the ranges from the columns themselves, which is what keeps this true; <code>every_training_value_lies_inside_its_feature_range</code> pins that it stays derived.</p>
|
||||||
|
<p><code>simplify</code> seeds each feature's feasible interval with its real range rather than ±∞, so an out-of-range threshold prunes. The interval is half-open <code>[lo, hi)</code> but a feature can hit its max, so <code>hi</code> is nudged by <code>EPS = 1e-3</code> and a split <em>at</em> the max (<code>x < max</code>) stays live. <code>EPS</code> is safely below typical threshold granularity.</p>
|
||||||
|
<h2 id="5-row-an-invariant-made-unrepresentable"><a class="anchor" href="#5-row-an-invariant-made-unrepresentable">#</a>5. <code>Row<T></code>: an invariant made unrepresentable</h2>
|
||||||
|
<p><code>Row<T></code> pairs a feature vector with what it is supposed to predict. The target is generic because the two supervised shapes differ only in it — <code>Row<f32></code> for regression, <code>Row<usize></code> for a class code — and every scoring function then reads <strong>one</strong> slice instead of two.</p>
|
||||||
|
<p>That is the point of the type. Parallel <code>x</code>/<code>y</code> slices could be handed in at different lengths, and <code>zip</code> answered by silently scoring the shorter prefix while a denominator taken from the longer one made the mean too small. There is no length to get wrong here, so there is nothing to validate and no <code>Result</code> to thread through five signatures (<a class="tid" href="../tickets.html#improvise-dkqn" title="gp-tree::fit should take one slice of rows, not parallel x/y slices">improvise-dkqn</a>, <code>doctrine(an_invariant_is_made_unrepresentable_not_validated)</code> — the same answer <code>prune_axis_items</code> got).</p>
|
||||||
|
<p><strong>The pairing is made at the boundary that builds the data.</strong> <code>preprocess</code> walks one row index and the rows live inside <code>Target</code>; a <code>Dataset</code> carrying <code>rows</code> and <code>target</code> as two collections would move the mismatched length one level up, into every caller.</p>
|
||||||
|
<p><code>mean_over(total, n)</code> is the one place the empty row set is answered. Every mean in <code>fit.rs</code> used to reach its own <code>0.0 / 0.0</code> on an empty slice: a NaN that propagates through <code>FitConfig::cost</code>, sorts unpredictably against real costs, and reaches a caller as a number rather than as an error. Scoring no rows finds no error, so the answer is zero, stated once instead of in <code>mse</code>, <code>misclass_rate</code> and <code>standardize</code> separately.</p>
|
||||||
|
<p><strong>Changing any of this needs bit-level evidence.</strong> <code>tests/scoring_known_answers.rs</code> pins every scoring path as <code>f32</code> bit patterns plus a <code>next_u64</code> witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. See <a href="../test-evidence/gp.html"><code>context/test-evidence/gp.md</code></a>.</p>
|
||||||
|
<h2 id="6-folds-and-cv-folds-answer-different-questions"><a class="anchor" href="#6-folds-and-cv-folds-answer-different-questions">#</a>6. <code>folds</code> and <code>cv_folds</code> answer different questions</h2>
|
||||||
|
<p><code>folds(n, k)</code> splits <code>0..n</code> into <code>k</code> pieces after a deterministic shuffle. <code>k</code> arrives from a host — a CLI flag, in this crate's only caller — so zero is a number to answer, not an impossible state: <code>out[pos % 0]</code> is a public library function taking its host down with it. One fold is the answer the shape already gives, and <code>fit</code> spells the same <code>max(1)</code> over <code>restarts</code>.</p>
|
||||||
|
<p><code>cv_folds(n, k)</code> yields nothing below two. A partition into one piece is still a partition, but a <strong>cross-validation</strong> over one fold is not a cross-validation: the single fold holds every index, the training complement is empty, and both cross-validators would fit a tree on nothing and then let it predict every row. Yielding no folds makes the pooled pairs empty, which is the true count of such predictions — and no fold is drawn, so the shuffle never touches the caller's generator either (<a class="tid" href="../tickets.html#improvise-ry89" title="cv_regress/cv_classify with k=1 now return plausible numbers instead of NaN">improvise-ry89</a>).</p>
|
||||||
|
<p><code>complement(n, test)</code> builds a bool mask rather than asking <code>test.contains(i)</code> per row, which was a scan of the fold per row and so quadratic in the row count (<a class="tid" href="../tickets.html#improvise-t3ux" title="cv_regress and cv_classify are O(n-squared)">improvise-t3ux</a>). It yields <strong>the same indices in the same order</strong>, which is what every downstream draw from the generator depends on.</p>
|
||||||
|
<h2 id="7-standardize-denormalize-the-affine-map-must-be-folded-back"><a class="anchor" href="#7-standardize-denormalize-the-affine-map-must-be-folded-back">#</a>7. <code>standardize</code> / <code>denormalize</code>: the affine map must be folded back</h2>
|
||||||
|
<p>The crate's leaf scores live in a ±5 range, so a target in the thousands cannot be reached by leaf mutation and <code>standardize</code> is not optional. Its consequence is that the champion predicts <strong>standard deviations</strong>.</p>
|
||||||
|
<p><code>fit_csv</code> answers that by <em>printing</em> <code>≈ score × std + mean</code> beside the tree. A document has nowhere to put that, so <code>improvise-gp::fit_measure</code> folds the affine map back into the leaves with <code>denormalize</code> before a single character of formula text is spelled. Omitting it emits a perfectly well-formed formula whose every cell is out by an affine map — <strong>wrong in the quiet way</strong>, since small numbers in the right shape look like an answer.</p>
|
||||||
|
<p><code>denormalize</code> touches leaves only, and that is the whole reason it is sound: <code>standardize</code> rescales the <em>target</em>, so a threshold is already in its feature's own units and re-scaling it would be the bug.</p>
|
||||||
|
<h2 id="8-regressionquality-carries-its-own-baseline"><a class="anchor" href="#8-regressionquality-carries-its-own-baseline">#</a>8. <code>RegressionQuality</code> carries its own baseline</h2>
|
||||||
|
<p>A tree with a plausible-looking RMSE that is no better than always predicting the mean is not a weak rule, it is <em>no rule</em> — and it is worse than no rule at all, because it is one a user will believe. So the comparison is part of the answer rather than something a caller may forget to compute. <code>r2</code> is <strong>not clamped</strong>: negative means the fit does worse than the mean, which is a real outcome, and clamping would make "learned nothing" and "actively misleading" read alike.</p>
|
||||||
|
<p><code>importance_ranking</code> reports each feature's share of the total importance. The all-zero case returns early rather than flooring the divisor: <code>fit_csv</code> used <code>.max(1e-9)</code>, which is right for the only input it ever had and wrong as a contract — a lone importance of <code>1e-10</code> then divides by the floor rather than by itself and reports <strong>10%</strong>, where the one feature the model uses carries all of it. Importances this small are ordinary, being error differences over a <em>standardized</em> target, so the shares have to sum to 100% at every scale.</p>
|
||||||
|
<h2 id="9-a-fit-takes-two-things-to-reproduce-and-here-four"><a class="anchor" href="#9-a-fit-takes-two-things-to-reproduce-and-here-four">#</a>9. A fit takes two things to reproduce — and here, four</h2>
|
||||||
|
<p><code>data::Model</code> records <code>seed: Option<u64></code> <strong>and</strong> <code>config: Option<FitConfig></code>, because the same seed under a different population/generations/elite/penalty lands on a different tree (<a class="tid" href="../tickets.html#improvise-ksst" title="A gp-tree fit records its seed but not its GA config">improvise-ksst</a>). Both are <code>Option</code> for one reason: every <code>FitConfig</code> value and the seed <code>0</code> are legal, so nothing can double as "unknown", and anything reporting a sentinel would state a seed that never ran.</p>
|
||||||
|
<p>Note <code>#[serde(default)]</code> on an <code>Option</code> field is <strong>documentation, not the mechanism</strong> — serde defaults it anyway, so deleting the attribute fails no test. A backward-compatibility test that only deletes the attribute is not discriminating; aim it at an <em>absent field</em> loading as the default instead.</p>
|
||||||
|
<p><code>improvise-gp::FitOutcome</code> carries the <strong>whole</strong> <code>FitRequest</code>, not the pair. <code>FitRequest</code> is <code>{ seed, config, cv_folds, importance_repeats, max_records }</code>, and every stochastic step draws from <strong>one</strong> generator in a fixed order — cross-validate, then fit, then explain. So <code>cv_folds</code> decides how far that generator has advanced before <code>fit</code> draws from it, and therefore changes the <em>champion</em> and not merely the evidence for it. Recording two of four fields documented the reproducibility contract without meeting it.</p>
|
||||||
|
<h2 id="10-the-emitter-quoting-and-the-threshold-s-exact-decimal"><a class="anchor" href="#10-the-emitter-quoting-and-the-threshold-s-exact-decimal">#</a>10. The emitter: quoting, and the threshold's exact decimal</h2>
|
||||||
|
<p>A regression tree <em>is</em> a nested conditional, and improvise formulas already have <code>IF(cond, then, else)</code>, so a fit can be written straight into a user's model as an ordinary formula with nothing GP-specific left in the file:</p>
|
||||||
|
<pre><code>Score = SUM(IF(humidity < 77.5, 1.25, IF(outlook IS rainy < 0.5, -0.5, 2)))</code></pre>
|
||||||
|
<p>(An earlier form of that sample sat in <code>improvise-gp/src/lib.rs</code>'s module doc inside a ``<code>text fence. rustdoc does not execute a </code>text` fence, so it was prose wearing a fence rather than a doctest, and the two-line cap applies to it like anything else.)</p>
|
||||||
|
<p>Two things in it are <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>'s, and both are load-bearing. The outer <code>SUM</code> is the <strong>route-then-aggregate rule made syntax</strong>: a tree is fitted on records, and <code>SUM(e)</code> is improvise's existing word for "fold <code>e</code> over the member records" (<a href="../code/core.html"><code>context/code/core.md</code></a> §13). Without it the formula routes on the totals one row up and means something else there; with it, a record row is a one-member fold and answers the rule itself. <code>outlook IS rainy</code> is the one-hot <code>outlook=rainy</code> said as a <strong>coordinate predicate</strong> rather than as a reference to an item no document has — see §11 for which features get which spelling.</p>
|
||||||
|
<p>Two properties the emitter owes its caller:</p>
|
||||||
|
<p><strong>It parses.</strong> Feature names come from the caller's data, so they carry spaces, operators, pipes and keyword collisions. Every name goes through <code>quote_name</code> — <code>improvise_formula::quote</code>, the one home for pipe quoting (gotcha 34); <code>improvise-gp</code> re-exports it because it was part of this crate's public surface. A formula <em>target</em> is unquoted on the way back in: it names a <code>_Measure</code> item, so a target keeping its delimiters writes the fit into an item literally named <code>|Total Revenue|</code>.</p>
|
||||||
|
<p><strong>It routes identically.</strong> <code>number_text</code> emits the <strong>shortest decimal that reads back as the same <code>f64</code></strong> the <code>f32</code> widens to — not the f32's full decimal expansion, which runs to 149 fractional digits, and not a shortest-round-trip-to-<code>f32</code> rendering. Widening an <code>f32</code> to <code>f64</code> is exact, so text that reads back as that <code>f64</code> makes <code>feature < thresh</code> give the same answer in <code>f64</code> formula evaluation as in <code>f32</code> <code>gp_tree::eval</code>. A short rendering (<code>0.1</code> for <code>0.1f32</code>) sits up to half an ULP away, and a row whose feature value equals the threshold exactly then falls on the <em>other</em> side of the split.</p>
|
||||||
|
<p><code>number_text</code> <strong>panics</strong> on a non-finite value. The grammar spells neither, and both fail quietly rather than loudly: <code>"inf"</code> parses as an <code>f64</code>, so the round-trip loop would return text no formula parser accepts, and NaN matches no width at all. Trees built by this crate's own operators over a finite <code>Spec</code> are finite, so this is a precondition rather than a runtime condition — but see §12, because a <code>.improv</code> cell holding <code>inf</code> makes it reachable from a document.</p>
|
||||||
|
<p><code>expr_text</code> and <code>mark_features</code> share one precondition and spell it the same way (<a class="tid" href="../tickets.html#improvise-hkt6" title="Three small totality gaps found in review">improvise-hkt6</a>): a bare <code>used[*feat]</code> fails with <code>index out of bounds</code>, which names neither the tree nor the spec. Two sites, one failure, one sentence. <code>used_features</code> is the single walk both the document check and the plan check read, because walking the tree twice two ways is how those answers drift apart.</p>
|
||||||
|
<h2 id="11-a-fit-is-storable-only-when-the-document-can-name-its-features"><a class="anchor" href="#11-a-fit-is-storable-only-when-the-document-can-name-its-features">#</a>11. A fit is storable only when the document can <em>name</em> its features</h2>
|
||||||
|
<p><code>preprocess</code> invents <code>Region=North</code> for a one-hot, <code>Cost.missing</code> for a missing indicator, <code>when.year</code> for a date part. Those are <strong>encoding artefacts, not items of the document</strong>, and <code>Expr::Ref</code> resolves through <code>Model::find_item_category</code> — so a tree splitting on one emitted a perfectly well-formed formula whose every cell then evaluated to <code>Error("ref:Region=North")</code>, with nothing between the fit and the file to say so (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>). <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a> closed the categorical third of that by giving the emitter a vocabulary for a coordinate, and by giving the <em>evaluator</em> a rule that makes such a formula mean the same thing at every level of aggregation (<a href="../code/core.html"><code>context/code/core.md</code></a> §13).</p>
|
||||||
|
<h3 id="features-one-entry-per-feature-index-read-off-the-plan"><a class="anchor" href="#features-one-entry-per-feature-index-read-off-the-plan">#</a><code>Features</code>: one entry per feature index, read off the plan</h3>
|
||||||
|
<p><code>Features::of(&Dataset, categories)</code> walks <code>gp_tree::data::feature_roles</code> — the <strong>one</strong> statement of <code>preprocess</code>'s expansion order, which <code>encode_row</code> drives off too — and pairs each role with the name <code>preprocess</code> gave it. A tree's <code>feat</code> indexes <code>Features</code> directly, so nothing anywhere reconstructs index → role from the feature <em>names</em>. <code>Spec::names</code> is <code>&[&str]</code> and could not have carried this; the earlier check matched names against the raw plan instead and said in this section that it was avoiding a second implementation of the expansion. Now there is no second implementation to avoid.</p>
|
||||||
|
<p>Each feature gets one of three <code>Spelling</code>s, decided <strong>by role and by column</strong>, never by the shape of the name:</p>
|
||||||
|
<table><thead><tr><th>role</th><th>column</th><th>spelling</th></tr></thead><tbody><tr><td><code>Numeric</code></td><td>a measure</td><td><code>Expr::Ref(measure)</code></td></tr><tr><td><code>OneHot { value }</code></td><td>a category, <code>value != "(missing)"</code></td><td><code>Cat IS Item</code></td></tr><tr><td><code>Numeric</code></td><td>a category</td><td><code>Artefact(NumericCategory)</code></td></tr><tr><td><code>DatePart</code></td><td></td><td><code>Artefact(DatePart)</code></td></tr><tr><td><code>Missing</code></td><td></td><td><code>Artefact(MissingFlag)</code></td></tr><tr><td><code>OneHot</code></td><td>a measure</td><td><code>Artefact(ValueOfAMeasure)</code></td></tr><tr><td><code>OneHot { "(missing)" }</code></td><td>a category</td><td><code>Artefact(AbsentCoordinate)</code></td></tr></tbody></table>
|
||||||
|
<p>The last five arms were <strong>one</strong> <code>_ => Artefact</code> until <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>, and the wildcard was not merely untidy. The refusal it fed read "a <code>col.missing</code> flag or a date part is an encoding artefact", so three of the five families were refused in words naming something they are not — and a future <code>FeatureRole</code> variant would have joined them silently, declined under whichever family the message happened to mention. The match is now exhaustive, so a sixth role is a compile error rather than a sixth thing that message misdescribes.</p>
|
||||||
|
<p>The two exclusions are not fussiness. A one-hot over a <strong>measure</strong> column names no coordinate at all — the record's value for that measure is text, and <code>Cat IS Item</code> asks about coordinates. <code>(missing)</code> is the <em>absence</em> of a coordinate, which <code>Expr::Is</code> reads as <code>0</code> by §13's <code>unwrap_or(false)</code>, so the one spelling that would be wrong is exactly the one omitted. A <code>Numeric</code> role over a <strong>category</strong> column is the trap the previous version of this section called "the hole a name cannot close": a category whose items all parse as numbers produces a feature named after the category, and <code>Expr::Ref</code> would resolve it to a <code>_Measure</code> item of the same name — a different column. Both halves now fall out of <code>is_category</code>, which is a lookup in the header's own first <code>category_columns</code> entries.</p>
|
||||||
|
<p>Duplicate header names are still refused outright, by <code>no_ambiguous_column_names</code> and before anything else: <code>is_category</code> decides by name, so a name that means two columns is a spelling with two meanings.</p>
|
||||||
|
<p><code>Dataset::columns</code> is <strong>not</strong> index-aligned with the header — <code>preprocess</code> <code>continue</code>s past an entirely-missing column without pushing a <code>ColumnSpec</code>, so one empty category shifts every later spec left. <code>FeatureRole::column()</code> indexes <code>columns</code>, not the header, so that shift is carried rather than tripped over.</p>
|
||||||
|
<h3 id="what-fitted-formula-still-asks-the-document"><a class="anchor" href="#what-fitted-formula-still-asks-the-document">#</a>What <code>fitted_formula</code> still asks the document</h3>
|
||||||
|
<p>Two refusals, in order. An <code>Artefact</code> is refused with no reference to the document at all — nothing could make <code>Cost.missing</code> spellable. Everything else is asked <strong>of the document</strong>: a <code>Measure</code> must be <code>find_item_category</code>-resolvable and a <code>Coordinate</code> must name a category the model has. A measure literally named <code>Region=North</code> is a legal document and is accepted, because the check never looks at the name's shape — and that is also why the date parts and the <code>.missing</code> flags stay covered without a second copy of <code>preprocess</code>'s naming rules.</p>
|
||||||
|
<p><code>formula_text</code> still spells whatever it is handed, being the tree's own text; an <code>Artefact</code> reaching it is a precondition failure and panics, the same split <code>number_text</code> makes for a non-finite value.</p>
|
||||||
|
<h3 id="the-two-families-improvise-wd6n-declines-separately"><a class="anchor" href="#the-two-families-improvise-wd6n-declines-separately">#</a>The two families <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a> declines, separately</h3>
|
||||||
|
<p>Both stay <code>Artefact</code> and both stay refused. The decline is written here rather than left open, because each needs a <strong>formula-language</strong> change and neither needs an emitter change — and they are answered separately because the vocabulary each wants is different. What changed with the decline is that the refusal now says which, in that family's own words.</p>
|
||||||
|
<p><strong><code>col.missing</code> is declined because the language has no word for absence.</strong> The flag means "this record has no value for measure X". <code>Expr::Ref</code> over an absent cell is an <code>Error</code>, not a zero, so there is no existing spelling to reuse and no way to phrase the question as an expression. Closing it means deciding what an absence <em>is</em> in a language whose aggregates already treat it as "no rows" — and that decision reaches <code>apply_agg</code>, <code>eval_agg</code> and the text-unanimity rule, not just this emitter. It is not worth that for a split a fit happens to like.</p>
|
||||||
|
<p><strong>The date sub-features are declined because improvise has no dates.</strong> <code>when.epoch_day</code>, <code>.year</code>, <code>.month</code>, <code>.day</code>, <code>.dow</code>, and with a time component <code>.hour</code>, <code>.minute</code> — <code>gp_tree::data::build_date</code> emits them. A <code>.improv</code> cell holding a date is <code>CellValue::Text</code>; nothing in <code>eval_expr</code> reads one, and there is no date type, no date literal and no date arithmetic anywhere in the crate. Closing this is a type addition to the value language, and the sub-features are the <em>smallest</em> part of it: a user who could split on <code>when.year</code> would immediately want to filter and aggregate by it.</p>
|
||||||
|
<p>Both are visible to the user only as a refusal naming the feature and the reason, which is the right interim: a fit that cannot be stored says so, rather than storing a formula whose every cell is a reference error (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>). <code>improvise-4zl7</code>'s route-then-aggregate rule (<a href="../code/core.html"><code>context/code/core.md</code></a> §13) is untouched — nothing new is emitted, so there is nothing new for it to hold over.</p>
|
||||||
|
<p>Two of the other three are declined permanently: a one-hot over a measure and the <code>(missing)</code> pseudo-item name no coordinate <em>by construction</em>, so no vocabulary closes them. They are refusals rather than gaps.</p>
|
||||||
|
<p><strong><code>NumericCategory</code> was not, and <a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a> has closed it.</strong> That it was <em>called</em> one is the more useful half of <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>: prose declaring something a permanent refusal is what stops anyone filing the ticket, and nothing but a reader could have caught it — the suite was green either way.</p>
|
||||||
|
<p>The accident was one argument. <code>fit_measure</code> handed <code>preprocess</code> an <code>Overrides::default()</code>, so it never said which header columns are document categories, <strong>although the very next line told <code>Features::of</code></strong>. <code>preprocess</code> then inferred <code>Numeric</code> → <code>Date</code> → <code>Categorical</code> from the cell values, so a <code>Year</code> category with items <code>2024</code>/<code>2025</code> became a <em>numeric column</em> and a category of ISO dates became a <em>date column</em> — and both were then refused as encoding artefacts with <code>Year IS 2024</code> spellable all along. <code>fit_measure</code> now builds its <code>Overrides</code> from the same <code>header[..category_columns]</code> slice it already hands <code>Features::of</code>, forcing each to <code>Forced::Categorical</code>, so the one place the document's own answer was known is now the only place either caller reads it from.</p>
|
||||||
|
<p>Three things follow, and the last is a cost rather than a win.</p>
|
||||||
|
<p><strong>The date decline above now means what it says.</strong> A <code>DatePart</code> could arrive over a <em>category</em> column by exactly this accident, so §11's date family was narrower than it read. It now covers a genuine date <strong>measure</strong> and nothing else, which is the scope the decline was always argued for.</p>
|
||||||
|
<p><strong>The <code>Artefact</code> arms stay, and stay correct.</strong> <code>crate::dataset</code> is <code>pub</code> and takes the caller’s <code>Overrides</code>, and <code>Features::of</code> reads roles off whatever <code>Dataset</code> it is handed — so a caller that builds one without forcing still produces a <code>Numeric</code> role over a category column, and <code>fitted_formula</code> still has to refuse it. The arm is unreachable from <code>fit_measure</code>, which is the only entry point a <em>document</em> goes through; it is not unreachable code, and deleting it would be deriving the classification from one caller’s habits rather than from the roles.</p>
|
||||||
|
<p><strong>Forcing widens the encoding, and that costs a document.</strong> A <code>Year</code> category with two items was one numeric feature and is now two one-hots; with a hundred items it is a hundred, and <code>MAX_FEATURES</code> refuses the fit outright.</p>
|
||||||
|
<p>The first draft of this paragraph claimed "no document that fitted before stops fitting", on the reasoning that a tree splitting on such a column was refused as an artefact anyway. <strong>That is false, and the review that caught it was of this prose rather than of the diff.</strong> The refusal is conditioned on <code>used_features</code>, so a fit that simply <em>never split</em> on the numeric category succeeded — and it is precisely the wide categories that a fit has least reason to split on. Measured, on a hundred records with a hundred-item numeric <code>Year</code> beside a <code>Cost</code> that predicts <code>Score</code> on its own: before, the fit lands on <code>IF(Cost < 48.0…)</code> and <code>Year</code> is never touched; after, the preflight refuses at 101 features. <code>a_wide_numeric_category_now_refuses_a_fit_that_used_to_ignore_it</code> pins both halves, including that the repair the message names really works.</p>
|
||||||
|
<p>It is still the right trade, for a reason that is <em>not</em> "both refuse". The old success was accidental: <code>Year</code> was encoded as a single continuous number over 1900–1999, an encoding that means nothing and that the fit could only use by producing a formula it would then be refused for. The document now gets a refusal that names the repair — mark the category a Label — and marking it restores the pre-zv8t feature set exactly. What is lost is the case where the user wanted neither: a wide numeric category left alone while the fit uses the measures. That is filed as <strong><a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a></strong>; the honest fix is a budget that can decline <em>a column</em> rather than the whole table, which is a change to what <code>encoded_width</code> returns and not to this decision.</p>
|
||||||
|
<p>The preflight <code>encoded_width</code> had to learn the same slice, or it counted such a column as one feature and handed the refusal to the exact check <em>after</em> materialising the matrix — which is the side of the cost the preflight exists to stay on (§14).</p>
|
||||||
|
<p><strong>And the refusal's own words must not name a column kind the arm does not know.</strong> <code>FeatureRole::Missing</code> is <em>not</em> conditioned on <code>is_category</code> — unlike its <code>Numeric</code> and <code>OneHot</code> neighbours — and <code>dataset.rs</code> writes <code>""</code> for a record with no coordinate for a category, so a <code>.missing</code> flag over a <strong>category</strong> is reachable. Its clause therefore says "that column" and not "that measure": naming the wrong kind there would be this section's own charge — declined in words naming something they are not — surviving one level down.</p>
|
||||||
|
<h2 id="12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na"><a class="anchor" href="#12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na">#</a>12. A fit is persisted as its formula, and nothing else (<a class="tid" href="../tickets.html#improvise-7na" title="Decide whether fitted gp-tree models are persisted alongside the formula">improvise-7na</a>)</h2>
|
||||||
|
<p><code>.improv</code> stores the emitted formula and <strong>no</strong> serialized <code>gp_tree::data::Model</code>. The grammar is unchanged by this decision, deliberately and not by omission.</p>
|
||||||
|
<p>The ticket offered three options: (a) formula only, refit to score new data; (b) the <code>data::Model</code> as a JSON blob; (c) seed + config so the fit reproduces. The answer is (a), with (c) already available in <code>gp-tree</code> for whoever wants provenance later.</p>
|
||||||
|
<p><strong>Against (b)</strong>, two arguments. The general one is the second half of gotcha 41: a blob is a <em>second copy</em> of information the document already carries — the tree is the formula — and nothing keeps the two in agreement. A user who edits the formula by hand, or a cell it was fitted on, leaves a blob that disagrees with the file it sits in, and a round-trip test over the blob cannot see that, because it never compares the two. (The <em>first</em> half of gotcha 41 does not transfer: an opaque blob inside one new grammar production is not a second implementation of the grammar.) The specific one is layering: <code>Tree</code> and <code>FitConfig</code> are <code>gp-tree</code> types, so <code>improvise-io</code> would gain a dependency on <code>gp-tree</code> and <code>.improv</code> would be obliged to track a third crate's serde layout forever — inverting exactly the dependency <code>improvise-gp</code> exists to keep out of the core.</p>
|
||||||
|
<p><strong>What (b) was for, and why it is not needed.</strong> The encoding plan exists so <code>fit_csv --predict</code> can score a row that arrives as untyped CSV text. improvise has no such row: a new row is a <em>record in the same document</em>, and <code>recompute_formulas</code> scores it through the stored formula from its own measures.</p>
|
||||||
|
<p><strong>Why (c) is not written to the file either.</strong> <code>data::Model</code> already records seed and config, so reproducing a fit is solved wherever that value is in hand; and unlike the CSV case the training data <em>is</em> the document. What is missing is a <em>reader</em> — writing provenance now would add a grammar production and a round-trip obligation for metadata nothing consumes. Provenance is the fit command's decision, alongside how it words the fit for the user.</p>
|
||||||
|
<p><strong>A premise the ticket got wrong, and it is the bigger finding.</strong> "The emitted formula is enough to recompute existing cells" holds only for a fit over numeric measures — §11 is that defect, and it is a defect in what a fit may emit, not an argument for persisting the model.</p>
|
||||||
|
<h2 id="13-dataset-rs-what-a-view-shows-as-a-feature-matrix"><a class="anchor" href="#13-dataset-rs-what-a-view-shows-as-a-feature-matrix">#</a>13. <code>dataset.rs</code>: what a view shows, as a feature matrix</h2>
|
||||||
|
<p><strong>The shape is not invented here.</strong> Records mode is already a feature matrix: a row is one coordinate tuple — every cell sharing all coordinates but <code>_Measure</code> — and each measure is its own column (gotcha 1). That is exactly "a sample, and its features". So the module enumerates the same rows records mode does, lays them out as the string table <code>preprocess</code> takes, and lets <code>preprocess</code> own type inference, one-hot expansion, imputation and the ranges.</p>
|
||||||
|
<p>What it does <em>not</em> reuse is <code>GridLayout</code>: reaching the records path means putting <code>_Index</code> on Row and <code>_Dim</code> on Column, which is a different view from the one the user is looking at. <code>RecordRow::group</code> is public, so the traversal here is the same two steps the layout takes, against the user's own view.</p>
|
||||||
|
<p>Four selection rules, each of which could reasonably have gone the other way:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Rows are scoped by <code>View::filtering_cats()</code></strong> — Page ∪ Filter — not by <code>categories_on(Axis::Page)</code>. Both constrain which records are visible; only Page is navigable (gotcha 27), and scoping by the navigable half would silently fit against rows a drill had already excluded.</li>
|
||||||
|
<li><strong>Hidden items scope the rows and the feature columns.</strong> The mental model is "fit what I can see", and hiding an item is an explicit statement that it is not of interest. An explicitly named target is exempt: the caller asked for it by name.</li>
|
||||||
|
<li><strong>Every regular category is a feature, whatever axis it sits on</strong> — including <code>Axis::None</code>. A category's coordinate is part of what makes a record distinct, so dropping a None category would leave two different rows with identical feature vectors and different targets: noise the fit can neither learn nor report. <code>Label</code> categories are excluded, as they are from records mode, by being not <code>is_regular</code> — they are high-cardinality per-row text and one-hot into a perfect memoriser.</li>
|
||||||
|
<li><strong>No formula-derived value ever enters the matrix.</strong> Formula targets are not features (a formula is a function of the other measures, so it leaks the target whenever the target is one of its inputs), and a formula target is refused <em>as</em> the target — a formula already is the rule, so fitting a tree to reproduce it from its own inputs is trivially perfect and says nothing. One rule covers both halves, with no dependency analysis to get wrong.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong><code>_Measure</code> is a filtering category and must not filter a fit.</strong> That exception in <code>filter_coords</code> is not a special case, it is the difference between a grid and a feature matrix: <code>_Measure</code> is the coordinate the <em>columns</em> are spread across, so pinning it to one item is not "which records am I looking at" but "throw away every column except this one". <code>Workbook::new</code>'s default view puts it on <code>Axis::Page</code>, so that was the behaviour on a freshly opened document (<a class="tid" href="../tickets.html#improvise-vx3" title="fit-measure command: fit a tree and add it as a formula">improvise-vx3</a>).</p>
|
||||||
|
<p>Two smaller decisions. Records are sorted by their coordinate tuple. That began as a workaround for <code>matching_cells</code> walking a <code>HashSet</code>; since <a class="tid" href="../tickets.html#improvise-shyt" title="[bug] Records-mode row order is not stable across processes">improvise-shyt</a> it yields insertion order, so the sort is no longer what buys stability. It is kept because a fit's row order should be a property of the <em>data</em> rather than of the typing history that entered it. And a <code>Number</code> cell is rendered with <code>f64::to_string</code>, never <code>format_f64</code>: display rounding is view-only and must not reach a fit (gotcha 8). An <code>Error</code> cell has no value, so it is spelled as missing and picks up the same imputation and <code>.missing</code> indicator an absent cell would.</p>
|
||||||
|
<h2 id="14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess"><a class="anchor" href="#14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess">#</a>14. <code>improvise-gp::fit</code>: budgets, and the enum that refuses to guess</h2>
|
||||||
|
<h3 id="two-budgets-and-why-the-preflight-is-advisory"><a class="anchor" href="#two-budgets-and-why-the-preflight-is-advisory">#</a>Two budgets, and why the preflight is advisory</h3>
|
||||||
|
<p><code>FitConfig</code> bounds the <em>search</em>; nothing bounded the <em>width</em>, which <code>preprocess</code> can multiply without limit (one column per distinct value of a categorical column). Permutation importance alone is <code>repeats × rows × features</code> tree evaluations, on the thread that draws the next frame.</p>
|
||||||
|
<p><code>MAX_FEATURES = 64</code> is comfortably above any hand-built pivot table — this repo's own 18,000-cell benchmark encodes far fewer — and comfortably below the one-column-per-row shape that is really a mis-typed category. The message names the fix (<code>Label</code>), because that is what the user actually wants.</p>
|
||||||
|
<p><code>encoded_width</code> is the <strong>advisory</strong> preflight, run on the raw table <em>before</em> <code>preprocess</code> materialises anything. Checking width afterwards guards the wrong side of the cost: a category with a distinct value per record allocates a rows × rows one-hot <em>during</em> preprocessing, so the refusal arrives after the memory and the O(rows²) work have been spent. It approximates <code>preprocess</code>'s type inference rather than sharing it. Copying the predicate is the lesser evil; the alternative is exporting the inference and calling this a second implementation of it.</p>
|
||||||
|
<p><strong>The debt that copy carries is that the preflight owes EVERY branch <code>preprocess</code> can take, and it paid two of three.</strong> <code>preprocess</code> infers <code>Numeric → Date → Categorical</code>, in that order, and the preflight had no date arm at all: a column whose cells parse as neither numbers nor a forced category fell through to the categorical count. So a <strong>measure column holding dates</strong> — one distinct timestamp in every row — promised <strong>1</strong> and materialised <strong>5</strong>, because <code>build_date</code> emits <code>epoch_day/year/month/day/dow</code>, and <strong>7</strong> once any cell carries a time (<a class="tid" href="../tickets.html#improvise-x1ns" title="encoded_width undercounts a forced category that has a missing coordinate">improvise-x1ns</a>, second instance, found by ChatGPT Codex on #97). That is an under-count of four or six on a single column, against a budget of 64.</p>
|
||||||
|
<p>The two instances are the same defect one inference-branch apart, which is why the shape is now structural rather than remembered. <code>inferred_kind</code> reproduces <code>preprocess</code>'s match arm for arm and returns a <code>Forced</code>, and <code>encoded_width</code> matches that <strong>with no wildcard</strong> — so a fourth <code>Forced</code> variant is a compile error here instead of silently taking the categorical arm, which is exactly how <code>Date</code> was missed. The <code>.missing</code> flag is added once outside the match, because every branch owes it on the same terms.</p>
|
||||||
|
<p>The count each arm owes:</p>
|
||||||
|
<table><thead><tr><th><code>preprocess</code> builds</th><th>features</th></tr></thead><tbody><tr><td><code>build_numeric</code></td><td>1</td></tr><tr><td><code>build_date</code></td><td>5, or 7 if any present cell has a time</td></tr><tr><td><code>build_categorical</code></td><td>distinct present values, <strong>+1</strong> for the reserved <code>(missing)</code></td></tr></tbody></table>
|
||||||
|
<p>plus <code>col.missing</code> wherever any cell is missing, and clamped to at least 1.</p>
|
||||||
|
<p><strong>The two directions of a drift do not cost the same, and this paragraph said they did</strong> (<a class="tid" href="../tickets.html#improvise-x1ns" title="encoded_width undercounts a forced category that has a missing coordinate">improvise-x1ns</a>). It read: <em>the cost of the copy drifting is bounded in both directions by the exact check</em> — over-counting costs a needless refusal, under-counting "only lets the exact check do its job". That is wrong, and it is wrong against the argument <strong>two paragraphs above it</strong>: the exact check runs on <code>Dataset::feature_names.len()</code>, which exists only <em>after</em> <code>preprocess</code> has materialised the matrix. So it bounds the <strong>error message</strong>, never the <strong>cost</strong> — which is precisely the thing the preflight was introduced to bound. Over-counting is cheap; under-counting spends the memory and the O(rows²) work the preflight exists to prevent, and then reports it politely.</p>
|
||||||
|
<p>The live instance was the <code>(missing)</code> column: the categorical arm counted <code>present</code> values only, so a forced category with N distinct present values and one record lacking that coordinate promised N and materialised N+1. <code>has_missing</code> was already computed and used in the numeric arm alone — <strong>two arms of one match disagreeing about what a column costs</strong>, which is <a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a>'s forcing-by-name / budgeting-by-index defect one level in. That one was repaired by making the ordering a data dependency and so a compile error; this one has no such shape available, so the guard is a test that asks <code>preprocess</code> what it actually built rather than asserting a number (<code>the_preflight_counts_the_missing_category_a_forced_column_materialises</code>).</p>
|
||||||
|
<p>The previous wave's review saw a version of this and reasoned it away as <em>"read, not injected"</em> and <em>"bounded by the exact check"</em>. Both halves of that are the failure mode this file exists to catch: the first declined to run the experiment, and the second repeated the sentence above instead of checking it. <strong>An argument that a cost is bounded is checkable, and this one was checkable the day it was written</strong>: <code>encoded_width</code> landed 2026-09-19 and the sentence 2026-09-20, so the code it described was one commit old and one file away.</p>
|
||||||
|
<p><code>encoded_width</code> remains <strong>advisory</strong> in the direction that is safe: it may over-count, and a column whose cells are <em>entirely</em> missing is counted as one where <code>preprocess</code> skips it altogether. It must never under-count.</p>
|
||||||
|
<p><strong>That sentence was written as a proposed <code>kb.pl</code> invariant while the tree held a counter-example</strong>, in the same change that fixed the categorical one — the date branch was missing the whole time and nothing in the hand-back noticed. An invariant proposed from the defect you just fixed generalises the <em>fix</em>, not the code; the thing that would have caught it is the question "what else does <code>preprocess</code> do that I have not mirrored", which is a question about the other function and not about the bug.</p>
|
||||||
|
<p><code>MAX_RECORDS = 20_000</code> bounds the <em>work</em>, which the width bound does not: a one-feature table with a million records still runs cross-validation, the final search and permutation importance over all of them. Measured at 105 ms for 2,000 records and roughly linear, so 20,000 is about a second — where a pause stops being a pause. Refusing is honest, and the limit is the caller's <code>FitRequest::max_records</code> rather than a constant, so a host with somewhere else to run a fit can raise it.</p>
|
||||||
|
<p><code>FitRequest::default()</code> is the interactive budget: <code>FitConfig::default()</code> is the CLI's 80 members over 600 generations, which is right for a batch tool and wrong for a synchronous command. <strong>Measured</strong>, release build, on the shape this module's tests use: 6 ms at 12 records, 12 ms at 200, 105 ms at 2,000 — the whole sequence, five-fold CV included. There is deliberately <strong>no timing test</strong>: a wall-clock assertion is flaky on shared CI and would fail for a reason that is not this code's. Re-measure after changing the numbers.</p>
|
||||||
|
<h3 id="forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one"><a class="anchor" href="#forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one">#</a>Forcing a numeric category is a <em>representability</em> choice, not a budget one</h3>
|
||||||
|
<p><a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a> made <code>fit_measure</code> force every document category to <code>Forced::Categorical</code>, and <a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a> records the cost: a hundred-item numeric category one-hots to a hundred columns, past <code>MAX_FEATURES</code>, and is refused where inference used to read it as one numeric column and the fit succeeded. <strong><a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a> is declined, and the refusal is correct.</strong></p>
|
||||||
|
<p>The ticket rules out making the forcing conditional on cardinality, and rightly: that is inference by another name, and it makes the <em>encoding</em> depend on a budget rather than on what the document says. But the stronger reason to decline is that the alternatives are not merely distasteful — they are <strong>unstorable</strong>, and the code already says so.</p>
|
||||||
|
<p>A coordinate becomes an <code>Expr</code> <strong>only by passing through <code>IS</code></strong>: <code>is_expr</code> is <code>identifier ~ sp+ ~ ^"IS" ~ sp+ ~ filter_value</code>, a single-value equality, and there is no <code>IN</code> and no ordering over a category's items. (<code>where_clause</code> and <code>inline_where</code> are coordinate predicates too, and also single-value equalities — but they <em>gate</em> a formula or an aggregate rather than being an <code>Expr</code>, which is gotcha 37(d)'s distinction and is why <code>IS</code> is the one that matters here.) So whatever a tree says about a category, the emitted formula names its items <strong>one at a time</strong>.</p>
|
||||||
|
<p><strong>A first draft of this paragraph said that made one-hot the only spellable encoding. That is false</strong>, and an adversarial review disproved it by running <code>parse_formula</code> rather than reading the grammar: <code>is_expr</code> is a <code>primary</code>, so it is an <code>Expr</code> yielding 1/0, and <code>cmp_op</code>'s operands are full <code>expr</code>s — so an ordinal split <em>is</em> spellable, as a weighted indicator sum (<code>IF(2020*(Year IS |2020|) + 2021*(Year IS |2021|) < 2020.5, …)</code>). The claim is recorded here rather than quietly deleted because it is the shape this file keeps catching: a grammar argument made by reading the rule that looked relevant instead of running the parser.</p>
|
||||||
|
<p>So <strong>ordinal is not ruled out by spellability, and is ruled out anyway</strong>, for two reasons that survive. It invents a <strong>metric the document does not assert</strong> — a category's items carry insertion order, not magnitude, so <code>Year < 3.5</code> means "the first four items as they happen to be ordered" and stops meaning that when someone reorders them. And the width it saves does not stay saved: the <em>encoding</em> would be one feature, but the <em>formula</em> still spells one <code>IS</code> per item, so it trades a <code>MAX_FEATURES</code> problem for a formula-size problem while adding a false order. What the current emitter does with it is narrower and is not the argument: <code>spell()</code> maps <code>FeatureRole::Numeric</code> over a <em>category</em> column to <code>Artefact::NumericCategory</code>, because a bare reference to the category's name would resolve to a <code>_Measure</code> item of the same name (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>) — a fact about the bare-ref spelling, not about ordinals in general. <strong>Hashing</strong> is ruled out outright: a bucket has no name in the document, so it cannot even be refused informatively.</p>
|
||||||
|
<p>And this is what makes the decline clean rather than resigned. The "fit that previously succeeded" was one of two things, never a third. Either the tree <strong>never split</strong> on the numeric category — in which case the column contributed nothing to the rule, and marking it a <code>Label</code> gives a fit over the same remaining features, inside the budget, which is exactly what the refusal already advises. Or the tree <strong>did</strong> split on it, and <code>fitted_formula</code> refused the result as <code>Artefact::NumericCategory</code> at storage time. So <strong>no fit is lost that was both storable and used the column</strong>.</p>
|
||||||
|
<p>Two things that claim less than a first draft of this paragraph did, both worth keeping. Marking the category a <code>Label</code> does <strong>not</strong> yield <em>the same fit</em>: the feature count changes, so the RNG draw sequence in <code>random_tree</code>/<code>sample_thresh</code> changes, so the champion for a given seed changes — gotcha 37(b)'s own rule, applied to this paragraph. "The same fit" is a checkable cost claim and it is false; what is true is a fit over the same remaining features. And the dichotomy above is <strong>argued</strong>, from <code>spell()</code>'s exhaustive match over <code>FeatureRole</code>, not asserted: what <code>a_split_on_a_numeric_category_read_as_a_number_cannot_be_stored</code> asserts is one instance of its second branch, over one three-row fixture. That is the most a test can do here, and saying so is the difference between evidence and a flourish.</p>
|
||||||
|
<p>The test still had no predecessor. <code>NumericCategory</code> had no coverage before it — <strong>and it was one of three, not the only one</strong>: grepping each <code>Artefact</code> variant and each variant's <code>reason()</code> text, <code>MissingFlag</code> and <code>DatePart</code> are covered by <code>the_missing_flag_and_the_date_sub_features_are_refused_too</code>, and <strong><code>ValueOfAMeasure</code> and <code>AbsentCoordinate</code> have no test at all</strong>, before this change or after it. Two families remain uncovered; that is filed, not fixed here.</p>
|
||||||
|
<p>What the decline does cost, stated rather than waved at: a user whose <code>Year</code> category really is a quantity they want fitted <strong>as a number</strong> has no way to say so through <code>fit_measure</code>. The document says <code>Year</code> is a category of items, and that is what is honoured. The remedy is a document change — <code>Year</code> belongs in <code>_Measure</code> if it is a measurement — and <code>crate::dataset</code> is <code>pub</code> and takes the caller's <code>Overrides</code>, so a host that wants to offer the choice can (gotcha 37d). Offering it <em>inside</em> <code>fit_measure</code> would be a new user-facing control, and nobody has asked for one.</p>
|
||||||
|
<p>The refusal's wording was narrower than the rule and is now aligned in both places that carry it: it said <em>a category with a distinct value per record becomes one column per record</em>, which diagnoses the runaway shape and not <a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a>'s — a category of modest cardinality that is simply wider than the budget. Both now say a category becomes one column per distinct item, <strong>whether its items read as numbers or not</strong>, since that clause is the whole of what changed under zv8t.</p>
|
||||||
|
<h3 id="quality-is-an-enum-not-an-option"><a class="anchor" href="#quality-is-an-enum-not-an-option">#</a><code>Quality</code> is an enum, not an <code>Option</code></h3>
|
||||||
|
<p><code>cv_regress</code> pools no pairs below two folds, and scoring an empty pool gives an R² of exactly zero — which reads as "this rule is worth nothing" when what happened is that nobody checked. <code>Quality::Measured</code> carrying an all-zero <code>RegressionQuality</code> means "this tree is no better than the mean", the loudest thing a fit can say, and it must not be reachable by simply not having asked. So the enum, and a host that wants the numbers has to name the case where there are none.</p>
|
||||||
|
<p>The fold cap is <code>rows / 2</code>, not <code>rows</code>. <code>fit_csv</code> clamps to the row count, which lets two rows become two folds of one: each tree is then fit on a <em>single</em> row — a constant — and the numbers come back meaning nothing while wearing the <code>Measured</code> label. That is the confusion <code>Quality</code> exists to prevent, moved from one row to two rather than removed.</p>
|
||||||
|
<h3 id="fitted-measure-name-a-fit-is-a-rule-about-the-data"><a class="anchor" href="#fitted-measure-name-a-fit-is-a-rule-about-the-data">#</a><code>fitted_measure_name</code> — a fit is a rule <em>about</em> the data</h3>
|
||||||
|
<p><code>"{target} (fit)"</code>, unless the caller names another. Storing it under <code>target</code> itself would make the formula shadow the very cells it was fitted to: gotcha 7's <code>_Measure</code> items are explicit items ∪ formula targets, so the two would be one item with a computed value, and the user's typed data would stop being visible.</p>
|
||||||
|
<p>The destination is refused when it names <strong>any</strong> existing item, not only an existing measure. <code>Expr::Ref</code> resolves through <code>Model::find_item_category</code>, which finds a <em>regular</em> category's item before it falls back into <code>_Measure</code> (gotcha 7) — so naming the fit after an item of some category installs cleanly and then makes every later formula that references it read the category coordinate instead. Asking the document the same question <code>Expr::Ref</code> will ask is both stronger than a measure-only check and simpler.</p>
|
||||||
|
<h3 id="one-line-is-public-because-the-refusals-need-it-too"><a class="anchor" href="#one-line-is-public-because-the-refusals-need-it-too">#</a><code>one_line</code> is public because the refusals need it too</h3>
|
||||||
|
<p><code>.improv</code> pipe quoting makes <code>\n</code> legal inside a category or item name, so a document can name a measure that splits the TUI status bar in two. The whole composed status line is flattened once at the end rather than each part being sanitised and one being forgotten.</p>
|
||||||
|
<p>The <strong>refusals</strong> need it as well, and they are composed by the host: several interpolate a document name, and a failure is when a user is most reading the bar. Flattening only the success path left the failure path able to split it.</p>
|
||||||
|
<p>A related trap with no mechanism guarding it: a <code>rustfmt</code> pass over a multi-line string literal leaves runs of ten spaces mid-sentence in a message a user actually sees — invisible in the source and in every test that only asks whether a substring is present. <code>a_name_containing_a_newline_cannot_split_the_status_line</code> asserts the one-line rule for that reason.</p>
|
||||||
|
<h2 id="15-known-gaps-filed-rather-than-fixed"><a class="anchor" href="#15-known-gaps-filed-rather-than-fixed">#</a>15. Known gaps, filed rather than fixed</h2>
|
||||||
|
<p><strong>Closed: aggregate-then-route is not route-then-aggregate (<a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>).</strong> Kept here because the shape recurs. A tree is fitted on individual <strong>records</strong>, and an improvise formula evaluated at an aggregating <code>CellKey</code> used to aggregate each <code>Expr::Ref</code> first and evaluate the <code>IF</code> once on the totals — the same answer for a linear formula, a different one for anything that routes, so the stored rule meant one thing on a record cell and another one row up. The emitter now wraps every tree in <code>SUM(…)</code> and <code>eval_agg</code> folds a non-<code>Ref</code> inner expression over the member records: <a href="../code/core.html"><code>context/code/core.md</code></a> §13. What remains of it is a <strong>cost</strong>, not a defect — the fold is O(members) per evaluation per fixed-point pass and is unbenchmarked (<a class="tid" href="../tickets.html#improvise-9iyv" title="The route-then-aggregate fold's cost, measured">improvise-9iyv</a>).</p>
|
||||||
|
<p><strong>Two artefact families are still unspellable (<a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>).</strong> <code>col.missing</code> and the date sub-features stay <code>Spelling::Artefact</code> and are still refused. §11 says what each would need; they are independent of each other.</p>
|
||||||
|
<p><strong>A mean-imputed feature is read raw by the formula.</strong> <code>preprocess</code> fills a missing numeric cell with the column mean, so a champion may split on a value the training rows had and the <em>document</em> does not. The stored formula reads the document, so that record evaluates to something other than what the fit predicted for it — quietly, since the measure name resolves and only the cell is absent. <code>fitted_formula</code>'s check asks whether the document can <em>name</em> the feature, a different question from whether every record has one. Closing it means choosing between refusing any feature with a missing cell — which would refuse most real documents, imputation being why <code>preprocess</code> imputes — and teaching the formula language an <code>IFMISSING</code>-shaped primitive. A product decision of the same kind as <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>.</p>
|
||||||
|
<p><strong>A boundary value can route differently in the formula than in the fit.</strong> Training narrows each <code>f64</code> cell to <code>f32</code>; the stored formula compares the original <code>f64</code> against the threshold. <code>number_text</code> guarantees the threshold reads back as the same <code>f64</code>, so the <em>threshold</em> is exact — but a cell value just below a threshold that <code>f32</code> rounds <em>to</em> it takes <code>hi</code> in <code>gp_tree::eval</code> and <code>lo</code> in formula evaluation. Pre-existing in the emitter (<a class="tid" href="../tickets.html#improvise-izk" title="gp-tree step 1: emit a fitted Tree as an improvise formula string">improvise-izk</a>); the fix, one numeric representation end to end, would rewrite the bit-level characterization tests that are the only evidence the routing argument holds.</p>
|
||||||
|
<p><strong><code>Quality::Measured</code> is transductive rather than fully held out.</strong> Preprocessing runs over <strong>all</strong> rows before <code>cv_regress</code> picks its folds, so an imputation mean and a categorical vocabulary are learned partly from the rows they are then validated against. And it is wider than imputation: <code>Spec::ranges</code> is computed over all rows and the <em>same</em> spec goes into every training fold, so the thresholds the search samples and the branches <code>simplify</code> prunes are bounded by a held-out minimum or maximum. <strong>That reaches a fit over complete numeric measures too</strong> — the only shape <code>fit_measure</code> currently stores — so "the case that stores leaks nothing" is wrong. Closing it means building the ranges and the preprocessing plan per training fold and encoding each validation fold with <em>that</em> plan, which <code>cv_regress</code> cannot express: it takes one spec and rows that are already encoded. That is an API change in <code>gp-tree</code>, and it is independent of <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>, which turned out to need no change to how <code>cv_regress</code> is fed.</p>
|
||||||
|
<p><strong>Which features a champion picks is a property of the search</strong>, which used to mean the same command refused or succeeded depending on the seed. For the categorical family that is gone: a one-hot over a real category is spellable whichever way the search goes. It survives for <code>col.missing</code> and the date parts (<a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>), and it is the reason those are worth closing rather than living with — a refusal that depends on a seed is not a contract.</p>
|
||||||
|
<h2 id="16-preprocess-what-each-column-kind-expands-into"><a class="anchor" href="#16-preprocess-what-each-column-kind-expands-into">#</a>16. <code>preprocess</code>: what each column kind expands into</h2>
|
||||||
|
<p>Real data is not a matrix of floats. Columns are numeric, categorical or dated; cells go missing; a category is unordered and a <code>< thresh</code> split over its arbitrary codes would be meaningless. So each raw column is inferred (or forced through <code>Overrides</code>) to a <code>ColumnKind</code> and expanded:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>numeric</strong> → one feature, missing cells mean-imputed;</li>
|
||||||
|
<li><strong>date/datetime</strong> → ordered parts (<code>col.epoch_day</code>, <code>col.year</code>, <code>.month</code>, <code>.day</code>, <code>.dow</code>, and <code>.hour</code>/<code>.minute</code> when the column has a time), so trend and seasonality are learnable rather than one opaque token per timestamp. <code>epoch_day</code> is Howard Hinnant's <code>days_from_civil</code>; <code>parse_datetime</code> requires a 4-digit year so plain hyphenated numbers ("1-2-3") are not mistaken for dates;</li>
|
||||||
|
<li><strong>categorical</strong> → one 0/1 column per distinct value (<code>col=value</code>), which is the correct treatment of an unordered category — the tree then splits on "is it this value?". A missing cell becomes the reserved <code>(missing)</code> level.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Any column that had a missing cell also emits a <code>col.missing</code> 0/1 indicator, so <strong>absence itself</strong> is something the tree can split on rather than a silent substitution. A clean column emits none, or every fit would carry a constant feature.</p>
|
||||||
|
<p><strong>The expansion is not reproducible from a fresh table</strong>: imputation means and the one-hot value list are properties of the <em>training</em> rows, and a later table may not even contain some of them. <code>preprocess</code> therefore records a <code>ColumnSpec</code> per raw column and <code>encode_row</code> replays it; that pairing of fitted trees with their encoding plan is <code>data::Model</code>. A value never seen in training is simply all-zeros, not a new column.</p>
|
||||||
|
<p>Note for §14's preflight: <code>build_categorical</code> pushes the range <code>(0.0, 1.0)</code> for every one-hot <strong>whatever the column contains</strong>, so a constant one-hot is <em>not</em> pruned by <code>simplify</code> — it cannot know the column is constant. A test fixture that relies on a degenerate range being pruned is relying on seed luck.</p>
|
||||||
|
<h2 id="17-fit-csv-the-cli-s-own-documentation"><a class="anchor" href="#17-fit-csv-the-cli-s-own-documentation">#</a>17. <code>fit_csv</code>: the CLI's own documentation</h2>
|
||||||
|
<p><code>examples/fit_csv.rs</code> is the crate's <em>second</em> use case (the first was a game bot), a wholly different domain — the point being that <code>gp-tree</code> knows nothing about features or fitness. Almost none of what it does is <strong>in</strong> that file: preprocessing is <code>gp_tree::data</code>, the search is <code>gp_tree::fit</code>, and what is left is flag parsing, the demo dataset, and the printing.</p>
|
||||||
|
<pre><code>cargo run -p gp-tree --example fit_csv -- path/to/data.csv
|
||||||
|
cargo run -p gp-tree --example fit_csv -- --save model.json path/to/data.csv
|
||||||
|
cargo run -p gp-tree --example fit_csv -- --predict model.json path/to/new.csv</code></pre>
|
||||||
|
<p>With no path a built-in categorical dataset (the classic "play tennis" decision-tree example) is fitted, so it runs standalone.</p>
|
||||||
|
<p><strong>Shape.</strong> The header names the columns; the LAST column is the target, the rest are features. A numeric target is a <strong>regression</strong> (one tree, minimising MSE); a categorical target is a <strong>classification</strong> — one tree per class fit to a 0/1 indicator (one-vs-rest), predicting by argmax.</p>
|
||||||
|
<p><strong>What a fit reports</strong>, in order: a <em>generalization</em> metric first — <code>k</code>-fold cross-validation against a trivial baseline (predict-the-mean / majority-class), with R²/RMSE for regression and a confusion matrix plus per-class precision/recall for classification — then the final tree(s) fit on all rows, whose train error is optimistic, then a ranked <strong>permutation feature-importance</strong> table.</p>
|
||||||
|
<p><strong>Tuning flags.</strong> <code>--cv <k></code> (folds, <code>< 2</code> disables; default 5), <code>--seeds <n></code> (keep the best of <code>n</code> GA restarts; default 1), <code>--penalty <λ></code> (per-node parsimony cost keeping trees small and general; default 0.01), <code>--seed <n></code> (the RNG seed, decimal or <code>0x</code>-hex; printed by every fit and saved with <code>--save</code>).</p>
|
||||||
|
<p><strong>Column directives.</strong> <code>--ignore a,b</code> drops columns (ids, free text); <code>--numeric</code> / <code>--categorical</code> / <code>--date a,b</code> force a column's type — a numeric-looking zip code that is really a category, say.</p>
|
||||||
|
<p><strong>Messy CSVs.</strong> Missing cells (blank or <code>NA</code>/<code>null</code>/<code>?</code>/…) are mean-imputed for numeric and date columns, with a <code>col.missing</code> indicator so absence is itself learnable, and become a <code>(missing)</code> level for categorical columns. Rows missing the <em>target</em> are dropped up front, with a count on stderr.</p>
|
||||||
|
<p><strong><code>--save <path></code></strong> additionally writes a JSON <code>data::Model</code>: the fitted tree(s) plus enough of the preprocessing plan to re-encode a fresh CSV's raw cells into the exact same feature layout, <strong>and</strong> the seed and the <code>FitConfig</code> those flags produced — because the seed alone does not reproduce a fit (§9).</p>
|
||||||
|
<p><strong><code>--predict <model.json> <data.csv></code></strong> loads that model and scores each row of a CSV of raw feature columns (same names as training, any order, target column optional and ignored), printing one prediction per row — plus a softmax <code>confidence</code> column for classification.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>The kb tool — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html" class="here">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/kb-tool.md">context/code/kb-tool.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#maintaining-the-knowledge-base">Maintaining the knowledge base</a></li><li class="l2"><a href="#why-the-mcp-was-retired">Why the MCP was retired</a></li><li class="l2"><a href="#what-replaced-it">What replaced it</a></li><li class="l2"><a href="#the-guards-and-why-each-exists">The guards, and why each exists</a></li><li class="l2"><a href="#what-no-test-here-reaches">What no test here reaches</a></li></ul></details><h1 id="maintaining-the-knowledge-base">Maintaining the knowledge base</h1>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/prolog-sessions/kb.pl"><code>prolog-sessions/kb.pl</code></a> is executable knowledge, checked by CI. This is the argument for how it is written, and why the Prolog MCP stopped being the way.</p>
|
||||||
|
<h2 id="why-the-mcp-was-retired"><a class="anchor" href="#why-the-mcp-was-retired">#</a>Why the MCP was retired</h2>
|
||||||
|
<p><code>loadProgram</code> <strong>replaces a whole predicate</strong>, so adding one clause means re-sending every existing clause of it through an agent's context window — and a short payload silently drops the rest. Three further blockers: <code>saveSession</code> re-adds the MCP's own <code>host_rpc</code> shims and reserializes the whole file, and <code>assertz</code> raises <code>permission_error(modify, static_procedure)</code> against a consulted predicate, so there is no incremental route through it at all.</p>
|
||||||
|
<h2 id="what-replaced-it"><a class="anchor" href="#what-replaced-it">#</a>What replaced it</h2>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb"><code>scripts/kb</code></a>, a shell entry point over <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb-merge.pl"><code>scripts/kb-merge.pl</code></a>.</p>
|
||||||
|
<p><strong>Clauses are read as DATA, never consulted.</strong> <code>read_term/3</code> in a loop yields a list of terms; merging is list manipulation; <code>portray_clause/1</code> writes them back. Nothing is ever asserted, so the static-procedure wall is not reached and the cost of adding a clause is the clause.</p>
|
||||||
|
<p><strong>Canonical form is <code>read_term</code> in, <code>portray_clause</code> out.</strong> This was not designed, it was measured: all 495 facts in the existing file — including every escaped quote — round-trip byte-identically, and the form is a fixed point in one pass. The only change on first use is that rules move from the MCP's two-line layout to one line, which is why the reformat lands with this tool rather than before it: the MCP would re-expand them on its next save and the two would thrash.</p>
|
||||||
|
<p><strong>trealla comes from the dev shell.</strong> Reaching it through <code>nix shell</code> per invocation cost more than the work: the tool's first real use timed out past two minutes on a write that takes 0.42 seconds with <code>tpl</code> on <code>PATH</code>.</p>
|
||||||
|
<h2 id="the-guards-and-why-each-exists"><a class="anchor" href="#the-guards-and-why-each-exists">#</a>The guards, and why each exists</h2>
|
||||||
|
<p>Every one of these was added because injecting its fault showed the tool accepting something it should refuse. None was found by reading.</p>
|
||||||
|
<p><strong>The round-trip check.</strong> After writing, the file is read back and required to be the clauses intended. A serializer bug is the wipe-the-knowledge-base failure with a new owner, and this is the only check that sees it. Note it compares <em>variant</em> equality via <code>numbervars</code> over a copy — reading a term back yields fresh variables, so <code>==</code> compares variable identity and reports every rule as a difference.</p>
|
||||||
|
<p><strong><code>kb_consistent</code> runs on the CANDIDATE.</strong> The file is replaced only after the check passes, so a delta that breaks the knowledge base leaves it untouched rather than requiring a revert.</p>
|
||||||
|
<p><strong>Parse errors are caught explicitly.</strong> An uncaught syntax error drops trealla into its REPL, which <em>hangs</em> with a terminal and <em>exits 0</em> without one — a refusal that reads as acceptance. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb"><code>scripts/kb</code></a> also closes stdin on every call as a backstop.</p>
|
||||||
|
<p><strong>Every run reaches a <code>halt/1</code>.</strong> <code>tpl</code> exits 0 unless a halt is <em>reached</em>: measured on 3.10.3, <code>-g "fail"</code> and <code>-g "throw(boom)"</code> both exit 0, and so do <code>fail,halt</code> and <code>throw(boom),halt</code>, because a goal that fails or throws never reaches the halt beside it. <code>halt(2)</code> alone exits 2, so halt works — the default is the hazard. The goal used is therefore</p>
|
||||||
|
<pre><code>(catch(main,E,(write(user_error,E),nl(user_error),halt(2))) -> halt(0) ; halt(1))</code></pre>
|
||||||
|
<p>which gives 0, 2 and 1 for success, throw and failure. Without it, <code>if ! merge ...</code> reads a crashed merge as a clean one and replaces the knowledge base.</p>
|
||||||
|
<p><strong>Terms must be callable.</strong> <code>42.</code> parses, is not a clause, and reached the knowledge base as a fact whose head is a number. <code>kb_consistent</code> has no opinion about stray terms, so nothing objected.</p>
|
||||||
|
<p><strong>A clause can be retired.</strong> <code>kb rm</code> takes keys and refuses if any of them is absent, because a typo'd key would otherwise report success and leave the clause in place. Without it no correction could ever drop a fact — the first thing it was used for was retiring <code>kb_is_maintained_through_the_mcp</code>, whose <em>name</em> was the claim that went false.</p>
|
||||||
|
<p><strong>Duplicate keys are refused by default.</strong> A key is the functor plus every argument but the last, so <code>doctrine(Name,_)</code> keys on the name and <code>decision(Id,Key,_)</code> on the pair. <code>replace</code> is the door a correction goes through, and it overwrites in place — a count that grew would be two clauses under one name.</p>
|
||||||
|
<h2 id="what-no-test-here-reaches"><a class="anchor" href="#what-no-test-here-reaches">#</a>What no test here reaches</h2>
|
||||||
|
<p>The key rule is looser than <code>functional/1</code> for <code>crate/3</code>, which keys on the first argument alone; <code>kb_consistent</code> is the backstop for that case rather than the merge.</p>
|
||||||
|
<p>A clause name cited <em>inside another clause's string</em> is unchecked — dropping or renaming a clause silently dangles every clause referring to it (<code>improvise-2f97</code>).</p>
|
||||||
|
<p>A shebang would let <code>kb-merge.pl</code> run directly and collapse the two files into one. It does not work on trealla 3.10.3: <code>#!</code> is a syntax error that aborts the consult, so <code>main/0</code> is never defined, and <code>tpl</code> wants <code>-- args</code> <em>after</em> the file, which a shebang cannot inject. Worth re-testing if trealla gains script support.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,79 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>The repository's own checkers — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html" class="here">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/tooling.md">context/code/tooling.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#the-repository-s-own-checkers">The repository's own checkers</a></li><li class="l2"><a href="#1-the-comment-cap">1. The comment cap</a></li><li class="l2"><a href="#2-the-evidence-check">2. The evidence check</a></li><li class="l3"><a href="#a-leaf-is-not-an-identity">A leaf is not an identity</a></li><li class="l3"><a href="#the-names-come-from-cargo-not-from-the-source">The names come from cargo, not from the source</a></li><li class="l3"><a href="#a-demotion-is-a-reviewer-s-job-not-a-gate-s">A demotion is a reviewer's job, not a gate's</a></li><li class="l3"><a href="#a-flag-adds-a-question-it-never-selects-one-away">A flag adds a question; it never selects one away</a></li><li class="l3"><a href="#two-headings-one-test">Two headings, one test</a></li><li class="l2"><a href="#3-the-others">3. The others</a></li><li class="l3"><a href="#why-the-pages-deploy-fails-on-found-0-artifact-s-improvise-864a">Why the Pages deploy fails on "Found 0 artifact(s)" (improvise-864a)</a></li></ul></details><h1 id="the-repository-s-own-checkers">The repository's own checkers</h1>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/tree/main/scripts"><code>scripts/</code></a> holds the gates this repo runs on itself. They are code, and they are held to the rule they exist to enforce: a check nobody has seen fail is indistinguishable from one that always passes, so each of them carries a self-test whose cases are injected faults (<code>doctrine(check_the_checker)</code>).</p>
|
||||||
|
<p>Four docstrings in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-comment-runs.py"><code>scripts/check-comment-runs.py</code></a> point here. This is the file they mean.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="1-the-comment-cap"><a class="anchor" href="#1-the-comment-cap">#</a>1. The comment cap</h2>
|
||||||
|
<p><code>check-comment-runs.py</code> fails a <code>.rs</code> file carrying more than two consecutive comment lines. The rule is <a href="../design-principles.html"><code>context/design-principles.md</code></a> §6: reasoning lives in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a> prose, where it can be read, corrected and audited, not in a comment that nothing checks against the code.</p>
|
||||||
|
<p>The whole difficulty is deciding what <em>is</em> a comment line. Ten silent successes have been found in this one function, every one of them failing in the direction that looks like passing — a file that should have been flagged and was not:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>"test" in line</code> matched <code>#[cfg(test)]</code>, so every cfg-gated fixture cleared a gate written to reject exactly those. <strong>Anchor the match</strong>; an unanchored substring is a filter that over-matches, and a filter that over-matches fails in the direction that looks like success. (The same class as the bd ID audit's exclusion list, and as <code>_JS_TEST</code>'s exclusion of <code>.skip</code>/<code>.only</code>.)</li>
|
||||||
|
<li>A <code>proptest!</code> flag was set and never cleared, so the first <code>proptest!</code> in a file made every later <code>fn</code> — helpers included — count as a test.</li>
|
||||||
|
<li>An attribute holding a bracket inside a string (<code>#[cfg(feature = "[")]</code>) left the bracket depth unbalanced and exempted the rest of the file. Every question is therefore asked of the <strong>scanner</strong>, never of the raw line: the scanner knows Rust strings, raw strings, char literals and nested block comments, and the attribute depth is counted over code text only.</li>
|
||||||
|
<li>A fence that opens and never closes exempted everything after it.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Two rules that are decisions rather than accidents. A <code> fence **suspends** the run rather than ending it, so two lines above a fence and two below is a run of four — a reader meets it as one uninterrupted block. And only a *runnable* fence earns the exemption: rustdoc executes a bare </code> or <code>rust and does not execute </code>text, <code>ignore or </code>no_run, so the rest is prose wearing a fence.</p>
|
||||||
|
<h2 id="2-the-evidence-check"><a class="anchor" href="#2-the-evidence-check">#</a>2. The evidence check</h2>
|
||||||
|
<p><code>--check-evidence</code> holds <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> and the tests it describes to each other, in two directions:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Orphans.</strong> Every heading shaped like a test name must name a test that runs.</li>
|
||||||
|
<li><strong>Unstated.</strong> Every test whose doc comment cites <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> must have an entry there.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Neither direction is "every test has evidence". Most tests have none and should not be forced to have one; what is checkable is a <em>claim</em>, and both directions are claims. The failure they close is the same one a dead ticket ID is: a pointer that reads as a promise the thing exists.</p>
|
||||||
|
<h3 id="a-leaf-is-not-an-identity"><a class="anchor" href="#a-leaf-is-not-an-identity">#</a>A leaf is not an identity</h3>
|
||||||
|
<p>The names cargo lists are <strong>module paths</strong>, and the checker keeps them whole. Six leaf names collide in this workspace — two of them cited in evidence — so reducing a path to its last segment made a heading satisfiable by a test in another module answering another question. Renaming <code>ui::project::tests::a_load_that_fails_is_not_a_rebase</code>, the one its entry actually describes, left the checker at <em>579 entries, all name a test that runs</em>, exit 0: the surviving <code>ui::host::tests</code> namesake answered for it.</p>
|
||||||
|
<p>So a bare leaf resolves only when it is <strong>unique</strong>, and an ambiguous one is a <strong>refusal</strong> naming the candidates — not a warning, because a warning re-creates the problem for anyone who does not read it. The heading is then written with its module path, which is also what makes the entry say which test it is about.</p>
|
||||||
|
<p>Note the two halves are both needed and neither is sufficient. The refusal forces qualification but cannot survive a rename on its own: rename one of a colliding pair and the leaf becomes <em>unique</em>, silently resolving to the survivor. Qualification is what closes that, and the refusal is what makes somebody write it.</p>
|
||||||
|
<p>What stays open is collision <em>across binaries</em>: <code>--list</code> prints module paths with no crate and no binary, so two crates sharing one could not be told apart (<a class="tid" href="../tickets.html#improvise-sxsf" title="cargo test -- --list cannot name the binary, so two crates could share a full test path">improvise-sxsf</a>). Measured 2026-09-21 and still latent — 1444 names, <strong>zero</strong> duplicate full paths, and zero duplicate <code>(crate, path)</code> pairs.</p>
|
||||||
|
<p>The fix the ticket proposes — <code>--list</code> per package with the crate prefixed — was measured and is <strong>declined twice over</strong>. It costs 45.2s against 2.4s for the one workspace call (11 members, warm), 18× for an unreachable case, and <code>--check-evidence</code> needs the <code>--ignored</code> twin too, so the real figure is 22 invocations. More to the point <strong>it does not close the class it names</strong>: a package is not a binary. <code>improvise</code> alone carries five integration test binaries beside its lib, and an integration test's top-level <code>fn</code> lists with no module path at all — 60 of the 1444 names are bare — so <code>-p improvise</code> prefixes all six identically and <code>tests/a.rs::foo</code> still cannot be told from <code>tests/b.rs::foo</code>. A fix that ever becomes worth having is per-<em>target</em> (<code>--lib</code>, <code>--test <name></code>), not per-package, and the ticket's proposal would have bought 18× for a partial cure.</p>
|
||||||
|
<h3 id="the-names-come-from-cargo-not-from-the-source"><a class="anchor" href="#the-names-come-from-cargo-not-from-the-source">#</a>The names come from cargo, not from the source</h3>
|
||||||
|
<p>A heading is diffed against <code>cargo test --workspace -- --list</code>, minus the names that <code>--list --ignored</code> reports. It used to be scraped out of <code>.rs</code> text, and four shapes defeated that, all of them "evidence for a check that never runs" (<a class="tid" href="../tickets.html#improvise-rm4s" title="The evidence check accepts headings naming tests that never run">improvise-rm4s</a>):</p>
|
||||||
|
<table><thead><tr><th>shape</th><th>why the scrape accepted it</th></tr></thead><tbody><tr><td><code>#[test] fn x</code> inside a raw string literal</td><td>the scanner reads inside strings</td></tr><tr><td><code>#[cfg(feature = "…")] #[test]</code></td><td>the feature is never enabled</td></tr><tr><td><code>#[cfg(target_arch = "wasm32")] #[test]</code></td><td><code>cargo test --workspace</code> is native</td></tr><tr><td><code>#[test] #[ignore]</code></td><td>it compiles, and never runs</td></tr></tbody></table>
|
||||||
|
<p>The third is the realistic one here: this repo has real wasm32-only code and a <code>wasm</code> CI job that only runs <code>cargo check</code>. Adopting the stricter source cost nothing — all 539 headings resolved against a real listing on the commit that made the change — which is the moment to tighten a gate rather than the moment to argue about it.</p>
|
||||||
|
<p>Demonstrated before the fix, by injecting all four shapes into <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/draw.rs"><code>src/draw.rs</code></a> and <a href="../test-evidence/tui.html"><code>context/test-evidence/tui.md</code></a> at once: the previous checker printed <em>543 entries, all name a test that exists</em>, exit 0. The current one names all four.</p>
|
||||||
|
<p>What this costs: <code>--check-evidence</code> now needs a toolchain and a compiled workspace, about three seconds warm. A missing <code>cargo</code> says so rather than raising. The comment-cap check is unaffected and needs neither.</p>
|
||||||
|
<p><code>--rust-tests FILE</code> takes a saved listing for a caller that has one, and it <strong>requires <code>--rust-tests-ignored FILE</code></strong>: a first version took one file, called <code>cargo_test_names</code> with one argument, and so skipped the ignored rule entirely while printing the same success line as the honest path. A flag that is strictly more permissive and indistinguishable in the log is a bypass, not a convenience — so the pair is mandatory and the success line <strong>names its source</strong>. CI passes neither and runs cargo itself.</p>
|
||||||
|
<p>A <strong>missing</strong> <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> is not an early return either. It used to be, and it exited 0 while 170 tests cited a directory that was not there; the walk now yields nothing and every citation is reported unstated, which is loud.</p>
|
||||||
|
<p>Note what the self-test can and cannot carry here. The listing <em>reader</em> is testable from fixtures — a leaf name taken off a module path, an <code>--ignored</code> name subtracted, cargo's own prose not mistaken for a test — and those cases are in <code>SELF_TEST_LISTING</code>. The four <em>source shapes</em> are not: once <code>defined</code> comes from cargo, a fixture asserting "a heading with an empty listing is an orphan" is the same case however the Rust beside it is spelled, and writing four of them would be four copies of one assertion dressed as coverage. The evidence for those is the injection above, against the real tree.</p>
|
||||||
|
<p>What it does <strong>not</strong> close: <code>citing</code> is still scraped, so a citation inside a string literal would be reported as a test owing an entry. That is a false positive in the safe direction — loud, not silent — and it has no instance.</p>
|
||||||
|
<h3 id="a-demotion-is-a-reviewer-s-job-not-a-gate-s"><a class="anchor" href="#a-demotion-is-a-reviewer-s-job-not-a-gate-s">#</a>A demotion is a reviewer's job, not a gate's</h3>
|
||||||
|
<p>A heading is classified by <strong>shape</strong>: a bare identifier is a Rust test, a quoted string a JS test, anything else is prose about the section (<code>Helper — x</code>, <code>Note — y</code>, a file's own scope). So any orphaned heading can be retired by rewriting it with a space and a dash, and nothing can check whether the prose is honest — no mechanism could, because the question is whether a sentence is true (<a class="tid" href="../tickets.html#improvise-9za9" title="Demoting an evidence heading to prose is an unguarded escape hatch">improvise-9za9</a>).</p>
|
||||||
|
<p>Two things make that tolerable. The unsafe direction is the one that fails loudly: prose that <em>looks</em> like an identifier is classified as a Rust claim and reported as an orphan, so the error you can make by accident is the noisy one. And <strong>the mechanism is <code>git diff context/test-evidence/</code></strong> — a demotion shows up there as a <code>### identifier</code> line becoming prose, in the diff every reviewer already reads. The live instance, PR #69 demoting a heading while PR #72 restored its coverage under another name, is what says a reviewer rather than a gate has to catch it.</p>
|
||||||
|
<p>The printed count of prose headings is a <strong>convenience, not a guard</strong>: nothing stores the previous count and nobody diffs two CI logs, so a retirement moves a number no one is watching. Saying otherwise would claim a detectability it cannot support, which is the class of defect §7's adjacent-check section is about.</p>
|
||||||
|
<h3 id="a-flag-adds-a-question-it-never-selects-one-away"><a class="anchor" href="#a-flag-adds-a-question-it-never-selects-one-away">#</a>A flag adds a question; it never selects one away</h3>
|
||||||
|
<p><code>--check-evidence</code> used to <strong>early-return</strong>, so the flagged run answered the evidence question and said nothing at all about the comment-run cap. Three agents hit that independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment (<a class="tid" href="../tickets.html#improvise-ijql" title="check-comment-runs.py --check-evidence silently skips the comment-run cap it is named for">improvise-ijql</a>). It is now additive: <code>main</code> runs <code>check_cap</code> first, then <code>check_evidence</code>, and returns the worse of the two.</p>
|
||||||
|
<p>The ticket offered a cheaper shape — leave the early return and print one line saying the cap did not run — on the precedent of PR #83 making <code>--rust-tests</code> name its source. That precedent does not reach this case, and the distinction is worth keeping. <strong>A provenance line and a disclaimer are not the same object.</strong> Naming which listing the names came from qualifies an answer the run actually gave; saying "the cap did not run" qualifies an answer it did not give, while still <strong>exiting 0</strong>. Exit 0 is what CI branches on and what a human reads as clean, so a disclaimer leaves the gate exactly as skippable and merely documents the skip. The rule the fix is written to is therefore stated over the exit code: <strong>a tree with a long comment run must not pass any invocation that can exit 0.</strong></p>
|
||||||
|
<p>Which invocations those are is a decision, not a sweep. <code>--self-test</code> and <code>--list-doctests</code> are excluded and neither is an exception to the rule: the first scans no tree, and the second is a <em>listing</em> that renders no verdict. Only the two that answer "is this tree clean" are in <code>SELF_TEST_DISPATCH</code>.</p>
|
||||||
|
<p>That table is checked by running the script as a <strong>subprocess</strong> against a temporary tree, which is the only level at which this bug is visible — every unit in this file was correct, and the defect lived in <code>main</code>'s dispatch. Both arms are needed and both were demonstrated: restoring the early return fails <code>DISPATCH: --check-evidence: exited 0 over a four-line comment run</code>, and forcing the cap to always report a violation fails the clean arm for both invocations, which is what stops an always-fail from satisfying the first. The temporary tree also has no <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> and no JS roots, so the evidence half is vacuously clean there and the cap is the only thing that can decide the exit code — the isolation is what makes the assertion name one question.</p>
|
||||||
|
<h3 id="two-headings-one-test"><a class="anchor" href="#two-headings-one-test">#</a>Two headings, one test</h3>
|
||||||
|
<p>An orphan is a heading whose test nobody can find; the mirror image is a test that <strong>two</strong> headings answer for, and it is invisible to every rule above — each copy names a test that runs, so each copy passes. PR #92 shipped two entries under one heading where the older described a fixture the test no longer had and repeated a claim an adversarial review had measured false. A reader meeting the older one first has no way to know a successor exists two thousand lines down, so the stale copy reads as current. That is worse than an orphan, which at least announces itself as broken.</p>
|
||||||
|
<p>It is a <strong>refusal and not a warning</strong>, on the precedent <code>improvise-rm4s</code> set for the ambiguous leaf: the two are the same defect seen from opposite ends — one name answered by two tests, and one test answered by two names — so they are answered the same way. The refusal names the file and every line, in file order.</p>
|
||||||
|
<p>The key is the <strong>resolved test, not the spelling</strong>. <code>### f</code> beside <code>### m::tests::f</code> is two headings for one test, and grouping by the string would miss it precisely because qualification is the repair <code>rm4s</code> asks for; a heading qualified in one file and bare in another would have been the first way this came back. Resolution reuses the same leaf table the ambiguity rule uses, so a leaf that is genuinely ambiguous canonicalises to itself and is reported as ambiguous rather than as a duplicate — two <em>different</em> tests sharing a leaf are not two entries for one test, and conflating them would file the wrong repair. The check is over the whole directory rather than per file, because a heading keys a test name and a test name is unique workspace-wide once qualified.</p>
|
||||||
|
<p>What it still cannot see, and neither can any of the others: an entry that exists, names a live test, is the only one that does — and describes it backwards.</p>
|
||||||
|
<h2 id="3-the-others"><a class="anchor" href="#3-the-others">#</a>3. The others</h2>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb"><code>scripts/kb</code></a> is the only writer of <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/prolog-sessions/kb.pl"><code>prolog-sessions/kb.pl</code></a>; its own reasoning is in <a href="../code/kb-tool.html"><code>context/code/kb-tool.md</code></a>. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/node-test.sh"><code>scripts/node-test.sh</code></a>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb-test.sh"><code>scripts/kb-test.sh</code></a>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/pr-watch-test.sh"><code>scripts/pr-watch-test.sh</code></a> and <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> are the same shape as this file's subject: a guard, plus a suite that makes the guard refuse what it claims to refuse. Each is wired into <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/.github/workflows/ci.yml"><code>.github/workflows/ci.yml</code></a> in the same job as the thing it guards, because a check that only runs locally is a convention.</p>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-ci-graph.py"><code>scripts/check-ci-graph.py</code></a> is the workflow's own gate: <code>pages-deploy</code> must <code>needs</code> every other job in <code>ci.yml</code>. The rule "every check job" was written down (<a class="tid" href="../tickets.html#improvise-x7cq" title="A kb.pl inconsistency can still reach the published site">improvise-x7cq</a>) and a job added during review of PR #83 was still ungated with nothing noticing, which is what moved it from a sentence to a check. It reads the job keys and the <code>needs:</code> list off indentation, tracking which job a <code>needs:</code> belongs to — taking the first one seen would read another job's list as the deploy's and pass. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/README.md"><code>README.md</code></a> therefore states the rule and deliberately <strong>does not list the jobs</strong>: a second copy with nothing holding it to the first is what went stale, twice.</p>
|
||||||
|
<h3 id="why-the-pages-deploy-fails-on-found-0-artifact-s-improvise-864a"><a class="anchor" href="#why-the-pages-deploy-fails-on-found-0-artifact-s-improvise-864a">#</a>Why the Pages deploy fails on "Found 0 artifact(s)" (<a class="tid" href="../tickets.html#improvise-864a" title="Concurrent merges to main race on one Pages environment and the superseded deploys fail">improvise-864a</a>)</h3>
|
||||||
|
<p><code>actions/deploy-pages@v5</code> intermittently fails a push to <code>main</code> with <em>No artifacts named "github-pages" were found for this workflow run</em>, while the upload step was green in that same run. The ticket framed this as concurrent merges racing on one Pages environment, and PR #46 proposed skipping the deploy unless the commit is the tip of <code>main</code>. <strong>Both the framing and the guard are wrong, and measurement says so.</strong></p>
|
||||||
|
<p><strong>What was measured.</strong> All 100 push-to-<code>main</code> CI runs from 2026-09-19T22:02Z to 2026-09-21T05:02Z, from the run, job, step and log APIs (job metadata and logs outlive artifacts, which expire in a day and are all 404 now — so this is reproducible where the ticket's own artifact-count table is not):</p>
|
||||||
|
<table><thead><tr><th>deploys</th><th>with another run's SUCCESSFUL deploy completing inside this run's own upload→deploy window</th></tr></thead><tbody><tr><td>12 artifact-not-found failures</td><td><strong>12 (100%)</strong></td></tr><tr><td>56 successes</td><td>2 (3.6%)</td></tr><tr><td>4 other failures</td><td>0 — all four are the unrelated <em>Create Pages site failed: Resource not accessible by integration</em>, from before <code>enablement: false</code> (<a class="tid" href="../tickets.html#improvise-3gy" title="4.2 Enable GitHub Pages">improvise-3gy</a>)</td></tr></tbody></table>
|
||||||
|
<p><strong>The direction is settled, and it is neither of the two the ticket offered.</strong> The oldest run of the 17:47Z batch failed at 17:47:23Z — <em>before</em> the next deploy in the batch had started (17:47:31Z) and before any Pages deployment existed at all, since <code>deploy-pages</code> fetches the artifact metadata <em>before</em> it creates a deployment. So the failure cannot have been caused by this run's own deploy, by supersession, or by a concurrent deploy. The artifact was already gone. <strong>The reaper is another run's <em>successful</em> deployment.</strong></p>
|
||||||
|
<p><strong>It is scoped by artifact NAME, not by run.</strong> In run 35560726456 the build job finalized three artifacts seconds apart — <code>harness-serve</code>, <code>wasm-bundles</code> and <code>github-pages</code>, each with an id — and 2m21s later that run's own lookup reported <em>Found 2 artifact(s)</em> and no <code>github-pages</code>. Same run, same token, same call, same retention: the two siblings survived and only <code>github-pages</code> was missing. That one observation rules out the whole family of general explanations — service flakiness, eventual consistency, retention, a token scope, a v3/v4 backend mismatch — because any of them would have taken the siblings too.</p>
|
||||||
|
<p><strong>Four things this kills.</strong> Queue depth is not the mechanism: the three deploys of the 17:47Z batch never queued, running 9s and 17s apart with each failure taking one second, so the <code>pages</code> group serialized nothing. Retention is not it either — all three shared <code>retention-days: 1</code>, and since the <em>newest</em>-finalized expires last, expiry can produce 0/1/1 or 0/0/1 but never the ticket's 0/1/0. Nothing in <code>deploy-pages</code> deletes an artifact (an upstream feature request asks for exactly that capability, which is only open because nothing does). And the action-version theory dies on the interleaving: successes and failures alternate minute by minute on identically-resolved action SHAs.</p>
|
||||||
|
<p><strong>One clause of the ticket's own correction is falsified, and the conclusion still holds.</strong> The correction argued the tip-of-<code>main</code> guard would have skipped the one deploy that published "and let 52507a6 deploy and fail anyway" — which assumed 52507a6's artifact loss was independent of its neighbour. It was not: 5f871f7's deploy completed at 17:47:40Z, four seconds before 52507a6's began. Fewer successful deploys really is the right lever, so PR #46 aimed at the right quantity for the wrong reason. <strong>It should still not be shipped</strong>, because the association is not a mechanism and the mechanism is not deterministic: 5f871f7 had a foreign deploy land inside its own window too and survived. A guard measured at nothing, against a probabilistic effect, is <code>doctrine(measure_the_obvious_fix_before_reporting_it_as_the_fix)</code> exactly.</p>
|
||||||
|
<p><strong>What the evidence does license.</strong> The risk lives entirely in the upload→deploy window — every victim had a foreign deploy land inside its own — so shortening that window is the one lever with evidence behind it, and consolidating the build already did some of that. Closing it properly means no two runs holding a <code>github-pages</code> artifact across a deployment, which is a real change to the job graph with real costs and is not attempted here. A retry is <strong>not</strong> a fix: the artifact is gone rather than transiently unlistable, so a retry cannot find it.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>The TUI — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html" class="here">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/tui.md">context/code/tui.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#the-terminal-front-end-src">The terminal front end (src/)</a></li><li class="l2"><a href="#src-ui-grid-rs-gridwidget">src/ui/grid.rs — GridWidget</a></li><li class="l3"><a href="#inside-render-grid">Inside render_grid</a></li><li class="l2"><a href="#src-draw-rs">src/draw.rs</a></li><li class="l3"><a href="#the-key-event-boundary">The key-event boundary</a></li><li class="l3"><a href="#semantic-to-terminal-colour-and-the-one-that-is-wrong">Semantic-to-terminal colour, and the one that is wrong</a></li><li class="l3"><a href="#the-minibuffer-shares-the-bottom-row-with-the-status-line">The minibuffer shares the bottom row with the status line</a></li><li class="l3"><a href="#the-caches-are-built-here-per-frame">The caches are built here, per frame</a></li><li class="l2"><a href="#src-ui-import-wizard-ui-rs-importwizardwidget">src/ui/import_wizard_ui.rs — ImportWizardWidget</a></li><li class="l3"><a href="#set-bottom-line-one-writer-for-the-bottom-row">set_bottom_line — one writer for the bottom row</a></li><li class="l3"><a href="#truncate-prefix-bytes-deliberately">truncate / prefix — bytes, deliberately</a></li><li class="l2"><a href="#src-ui-help-rs">src/ui/help.rs</a></li><li class="l2"><a href="#the-panel-widgets">The panel widgets</a></li></ul></details><h1 id="the-terminal-front-end-src">The terminal front end (<a href="https://github.com/fiddlerwoaroof/improvise/tree/main/src"><code>src/</code></a>)</h1>
|
||||||
|
<p>The root crate is the ratatui binary: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/draw.rs"><code>src/draw.rs</code></a> (event loop and frame composition), <code>src/ui/*.rs</code> (the widgets), <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/main.rs"><code>src/main.rs</code></a> (clap). It is one of two front ends, and design-principles §5 is the rule it is judged by — it paints and wires, and every decision with a silently wrong answer is made in <code>improvise-protocol</code> so the browser cannot answer it differently.</p>
|
||||||
|
<p>Test narratives live in <a href="../test-evidence/tui.html"><code>context/test-evidence/tui.md</code></a>. Facts about <code>GridCache</code>, <code>AxisFrame</code>, <code>Projection</code> and the <code>Command</code>/<code>Effect</code> layer belong to <a href="../code/protocol.html"><code>context/code/protocol.md</code></a> and <a href="../code/command.html"><code>context/code/command.md</code></a>; this file cites them and does not restate them.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="src-ui-grid-rs-gridwidget"><a class="anchor" href="#src-ui-grid-rs-gridwidget">#</a><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/grid.rs"><code>src/ui/grid.rs</code></a> — <code>GridWidget</code></h2>
|
||||||
|
<p>The grid is drawn from a <code>GridCache</code> (<a class="tid" href="../tickets.html#improvise-n10" title="Migrate GridWidget to consume RenderCache + ViewState (not &App)">improvise-n10</a>).</p>
|
||||||
|
<p><strong>No <code>&Model</code>.</strong> Every value the grid paints — each cell's text, whether it is an aggregate, the column widths, the page bar and the totals strip — arrives already decided, on the cache. A widget that cannot reach the model cannot quietly answer a question differently from the way the browser client answers it, and that divergence is the whole thing projecting exists to prevent.</p>
|
||||||
|
<p><strong>The frame comes off the cache too</strong> (<a class="tid" href="../tickets.html#improvise-anr9" title="Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once">improvise-anr9</a>). <code>GridCache</code> carries an <code>AxisFrame</code> per axis — the header levels, the label <em>tuples</em> and the groups with their collapse state — so the multi-level headers, the repeat suppression, the group spans and the pieces below are all read rather than derived, and a browser painting the same frame paints the same thing. It used to be derived here from <code>GridLayout</code> + <code>View::is_group_collapsed</code> while the protocol carried <code>GridLayout::row_label</code>'s slash-joined form — the rendering <code>two_row_categories_produce_cross_product_labels</code> asserts must not appear — so the browser drew <code>Food/Alice</code>.</p>
|
||||||
|
<p>Two things are still not on the cache and are passed separately, each a different kind of absence:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong><code>layout</code> resolves a records-mode column</strong> (<code>record_col</code>), and only that. It keys <code>drill_state</code>'s pending edits, so it is session-local by design and correct to keep off the wire (gotcha 1).</li>
|
||||||
|
<li><strong><code>drill_state</code> is session-local residue</strong> (gotcha 30). Pending drill edits belong to whoever staged them and are deliberately not projected, so they shadow the cache rather than being on it.</li>
|
||||||
|
</ul>
|
||||||
|
<p>The cache is expected to cover the <strong>whole</strong> grid — offsets zero, every row and every column — so a position in it is an absolute position and the widget's scrolling arithmetic is unchanged.</p>
|
||||||
|
<h3 id="inside-render-grid"><a class="anchor" href="#inside-render-grid">#</a>Inside <code>render_grid</code></h3>
|
||||||
|
<p><code>GridWidget::cursor</code> is this client's cursor in the active view (<a class="tid" href="../tickets.html#improvise-8uz" title="Cursor and scroll are per-client state stored in the shared View">improvise-8uz</a>), passed in rather than read off <code>View</code>, which no longer carries per-client state.</p>
|
||||||
|
<p>Everything about the frame is the cache's answer (<a class="tid" href="../tickets.html#improvise-anr9" title="Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once">improvise-anr9</a>): how many header levels, what each level's label is, which group a position sits in and whether that group is collapsed. Nothing in <code>render_grid</code> reads <code>GridLayout</code> or <code>View</code>, so a browser folding the same <code>AxisFrame</code> draws the same header.</p>
|
||||||
|
<p><strong>The bulk forms are built once per frame, and that is not a micro-optimisation.</strong> <code>header_labels(offset)</code>, <code>groups()</code> and <code>shares_ancestry(sel_col)</code> each make one pass. The per-position accessors (<code>item</code>, <code>label</code>, <code>shows_level</code>, <code>group_of</code>, <code>same_prefix</code>) scan the frame from the start, so asking them inside the draw loops is quadratic in the grid and, at a deep scroll offset, grows with the offset. <code>col_lines</code> transposes <code>header_labels</code> to level-major because the column header is drawn one line per level while <code>header_labels</code> answers per column.</p>
|
||||||
|
<p><strong>The underline on the column header</strong> marks columns sharing <code>sel_col</code>'s ancestor group through each level: at the bottom level that is only <code>sel_col</code>, at higher levels it spans all the sub-columns.</p>
|
||||||
|
<p><strong>Pending drill edits shadow the projected value.</strong> They are this session's uncommitted typing and by construction are not on the cache (gotcha 30).</p>
|
||||||
|
<p><strong>Aggregate cells are drawn in italic</strong> to signal "drill to edit", and the fact is <em>not</em> re-derived here: the projection already named what each cell is, and <code>CellId::Aggregate</code> is exactly that fact (ADR 002), decided once by the side that holds the model. The widget used to walk <code>none_cats</code> × <code>kind.is_regular()</code> — a second hand-rolled copy of <code>project::cell_id</code>.</p>
|
||||||
|
<p><strong>Whether there is a total row at all is the cache's answer.</strong> An empty totals strip means "no totals" (records mode, or an empty grid), which is why the widget does not read <code>is_records_mode</code> a second time.</p>
|
||||||
|
<p><strong>The collapse glyph</strong> (<code>indicator</code>) is the one thing that is legitimately local: the <em>fact</em> — whether the group is collapsed — is on the frame; only its spelling is the terminal's (design-principles §5).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="src-draw-rs"><a class="anchor" href="#src-draw-rs">#</a><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/draw.rs"><code>src/draw.rs</code></a></h2>
|
||||||
|
<h3 id="the-key-event-boundary"><a class="anchor" href="#the-key-event-boundary">#</a>The key-event boundary</h3>
|
||||||
|
<p><code>convert_key_event</code> / <code>convert_key_code</code> / <code>convert_modifiers</code> are the <strong>only</strong> place crossterm's key types and the crate's own <code>command::key</code> types may both appear. Three rules:</p>
|
||||||
|
<ul>
|
||||||
|
<li>The event loop converts every incoming <code>Event::Key</code> regardless of <code>KeyEventKind</code> (Press/Repeat/Release), matching the pre-conversion behaviour, which never inspected the kind.</li>
|
||||||
|
<li>Codes the command layer does not model (media keys, <code>Insert</code>, …) become <code>KeyCode::Null</code>, which no keymap binds.</li>
|
||||||
|
<li><code>SHIFT</code>/<code>CONTROL</code>/<code>ALT</code> survive; every other modifier bit (<code>SUPER</code>, <code>HYPER</code>, <code>META</code>) is dropped.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="semantic-to-terminal-colour-and-the-one-that-is-wrong"><a class="anchor" href="#semantic-to-terminal-colour-and-the-one-that-is-wrong">#</a>Semantic-to-terminal colour, and the one that is wrong</h3>
|
||||||
|
<p>This file holds the mapping design-principles §5 names as the shape everything else should imitate: <strong>the decision lives in the shared layer as a semantic enum, and only the last translation is local.</strong></p>
|
||||||
|
<ul>
|
||||||
|
<li><code>axis_fg(AxisColor) -> Color</code> — <code>Axis::color</code> is the single place each axis's accent is decided; this is the single place axis colouring is <em>rendered</em> (<a class="tid" href="../tickets.html#improvise-rml" title="Consolidate Axis display into a method on Axis">improvise-rml</a>, gotcha 26). Before it, the tile bar and the category panel each carried their own copy of the colour table with nothing keeping them in agreement.</li>
|
||||||
|
<li><code>minibuf_fg(MinibufColor) -> Color</code> — the same shape for the minibuffer.</li>
|
||||||
|
<li><code>src/ui/import_wizard_ui.rs::field_kind_fg(WizardFieldKind) -> Color</code> — the same shape again. <code>improvise-protocol</code> must not name a ratatui <code>Color</code>.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>mode_style(&AppMode) -> Style</code> stays here for the same reason (<code>Style</code> and <code>Color</code> are ratatui types and <code>improvise-command</code>, where <code>AppMode</code> lives, carries no rendering-framework dependency), and since <code>improvise-zx8t</code> it is <strong>exhaustive</strong>: the fourteen variants that take the default chrome are spelled out rather than swept up by a <code>_ =></code> wildcard, so adding an <code>AppMode</code> fails the build here until someone chooses its colour.</p>
|
||||||
|
<p>It is still not quite the same shape as <code>axis_fg</code> and its two siblings, and the difference is deliberate rather than pending. Those three map a <em>semantic enum</em> the shared layer owns; <code>mode_style</code> matches <code>AppMode</code> itself, because there is no <code>ModeColor</code>. <code>improvise-zx8t</code> offered making one, and it was declined here for two reasons: the colour is the only thing at stake, and the ticket says to decide it together with <code>improvise-k099</code>, which asks the same question of <code>hint_text</code>/<code>label</code> and is not in this unit. Exhaustiveness is what the repo map's "the compiler names every one you must fill in" promise needed; a semantic enum is a separate, larger answer and is still open.</p>
|
||||||
|
<p><strong>The guard is the compiler, not a test.</strong> Nothing asserts the colours, and a test enumerating the variants would not fail when a fifteenth is added — only the missing match arm does. Verified by deleting the <code>AppMode::Help</code> arm: <code>error[E0004]: non-exhaustive patterns: </code>&AppMode::Help<code> not covered</code>.</p>
|
||||||
|
<h3 id="the-minibuffer-shares-the-bottom-row-with-the-status-line"><a class="anchor" href="#the-minibuffer-shares-the-bottom-row-with-the-status-line">#</a>The minibuffer shares the bottom row with the status line</h3>
|
||||||
|
<p><code>draw_bottom_bar</code> used to render the minibuffer <em>instead of</em> <code>draw_status</code>, so a status message written while any text-entry mode was open was never drawn (<code>improvise-mai9</code>, and the stranding gotcha 21 describes). <code>minibuffer_line</code> composes both: the prompt and buffer on the left, any pending <code>status_msg</code> right-aligned. The alternatives and why they lost are in <a href="../test-evidence/tui.html"><code>context/test-evidence/tui.md</code></a>.</p>
|
||||||
|
<h3 id="the-caches-are-built-here-per-frame"><a class="anchor" href="#the-caches-are-built-here-per-frame">#</a>The caches are built here, per frame</h3>
|
||||||
|
<p><code>project::formulas_of</code>, <code>category_tree_of</code>, <code>view_list_of</code>, <code>tile_bar_of</code>, <code>grid_of</code> and <code>wizard_of</code> are all called from <code>draw</code>/<code>draw_content</code> against the authoritative <code>App</code>. This is the <strong>temporary half of <a class="tid" href="../tickets.html#improvise-35e" title="Temporary adapter: RenderCache::from_app bridge for incremental widget migration">improvise-35e</a></strong>: when the TUI becomes a client it will <em>hold</em> each cache and fold the matching <code>Projection</code> into it, and none of the widgets change either way.</p>
|
||||||
|
<p>Two consequences today:</p>
|
||||||
|
<ul>
|
||||||
|
<li>The grid's viewport covers the <strong>whole</strong> grid, so a cache position is an absolute position (gotcha 44). Only the viewport narrows when the TUI becomes a client.</li>
|
||||||
|
<li>The wizard is <strong>built here rather than sent</strong>: it is session-local and has no <code>Projection</code> (<a class="tid" href="../tickets.html#improvise-764" title="Migrate import wizard widget and state to consume RenderCache">improvise-764</a>). It still goes through <code>project</code>, so the widget sees the same shape a browser wizard would.</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>draw</code> renders overlays last so they appear on top. The which-key popup's armed prefix is stored as the <em>key that opened it</em> and resolved back to a sub-keymap by the keymap set (<a class="tid" href="../tickets.html#improvise-jc8" title="Evict host-only state from ViewState: flip the transient-keymap Arc">improvise-jc8</a>, gotcha 13).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="src-ui-import-wizard-ui-rs-importwizardwidget"><a class="anchor" href="#src-ui-import-wizard-ui-rs-importwizardwidget">#</a><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/import_wizard_ui.rs"><code>src/ui/import_wizard_ui.rs</code></a> — <code>ImportWizardWidget</code></h2>
|
||||||
|
<p>Drawn from a <code>WizardCache</code> (<a class="tid" href="../tickets.html#improvise-764" title="Migrate import wizard widget and state to consume RenderCache">improvise-764</a>).</p>
|
||||||
|
<p><strong>No <code>ImportWizard</code>, no <code>ImportPipeline</code>, no <code>Workbook</code>.</strong> The widget used to hold the live wizard and read through it into <code>improvise-io</code> types and into the model — it called <code>regular_category_names()</code> and <code>measure_item_names()</code> on the built <code>Workbook</code> to write its one-line confirmation summary. Everything it draws now arrives already decided, from <code>project::wizard_of</code>.</p>
|
||||||
|
<p>The cache is <strong>not</strong> a <code>Projection</code>, and that is the ticket's answer to where wizard state lives: a half-open import dialogue belongs to the session that opened it, not to every connected client. <code>WizardCache</code>'s own documentation carries the argument.</p>
|
||||||
|
<p>Smaller facts the drawing depends on: the root array path is already spelled <code>(root)</code> by the projection, so both ends say it the same way; the date components arrive in the order the wizard offers them, already paired with whether they are on, so the cursor index counts exactly those rows; and on <code>Done</code> the workbook is built but not yet installed, so the step shows what the user is about to accept, warnings included (<a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a>).</p>
|
||||||
|
<h3 id="set-bottom-line-one-writer-for-the-bottom-row"><a class="anchor" href="#set-bottom-line-one-writer-for-the-bottom-row">#</a><code>set_bottom_line</code> — one writer for the bottom row</h3>
|
||||||
|
<p>That row has two writers — each step's key hint, and the wizard message that overrides it while present — and they write in that order to the same <code>y</code>. <code>Buffer::set_string</code> writes exactly as many cells as the string has and leaves the rest alone, so a message shorter than the hint it replaced left the hint's tail sitting beside it: <code>something went wrongcel</code>, the <code>cel</code> from <code>Esc: cancel</code> (<a class="tid" href="../tickets.html#improvise-q0vb" title="Pad the import wizard's hint line to the panel width">improvise-q0vb</a>).</p>
|
||||||
|
<p><strong>Padding here rather than at each site is the point.</strong> The bug is not that one of the five writers forgot; it is that the row had five writers and no one of them owned clearing it.</p>
|
||||||
|
<p><strong>The padding is measured in terminal cells, not characters.</strong> <code>Buffer</code> lays a string out by display width, so a char count over-pads a message carrying a wide character — enough to run past the panel and overwrite the border — and under-pads a combining mark, which is how a sliver of the hint would be left behind. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/grid.rs"><code>src/ui/grid.rs</code></a> measures the same way.</p>
|
||||||
|
<h3 id="truncate-prefix-bytes-deliberately"><a class="anchor" href="#truncate-prefix-bytes-deliberately">#</a><code>truncate</code> / <code>prefix</code> — bytes, deliberately</h3>
|
||||||
|
<p><code>truncate</code> cuts by <strong>byte</strong> length. <code>prefix</code> returns the longest prefix at most <code>max</code> bytes that ends on a char boundary, because every string <code>truncate</code> cuts can carry a field name, a formula or a message the user typed — and a multi-byte character straddling the cut used to panic the whole TUI on a slice at a non-boundary index. Walking down to the nearest boundary is the smallest change that cannot: for ASCII, which is what every fixture in this file holds, <code>max</code> is already a boundary and the result is byte-identical, so the characterization tests keep saying what they said.</p>
|
||||||
|
<p>It is bytes and not columns <em>deliberately</em>. Byte length is the measure <code>truncate</code> has always used, and changing it to display width would move where every line wraps — a different question from not crashing, and one that belongs with the hint-line padding it travels with. Bytes are never fewer than cells, so <code>truncate</code> cutting by bytes and <code>set_bottom_line</code> padding by cells cannot together overrun.</p>
|
||||||
|
<p><strong>The blind spot, stated because the fixtures cannot state it</strong>: the eight exact-frame characterization fixtures in this file are entirely ASCII. They are therefore blind to every question about display width — a change to either measure is invisible to them, and the evidence for those has to be built directly (see <code>set_bottom_line_clears_every_cell_it_covers_and_no_more</code>).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="src-ui-help-rs"><a class="anchor" href="#src-ui-help-rs">#</a><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/help.rs"><code>src/ui/help.rs</code></a></h2>
|
||||||
|
<p>~600 lines of mostly-static content: five pages of styled <code>(key, description)</code> lines, a tab bar and a footer. <code>improvise-ktv1</code> proposes moving that content into the shared layer so a browser help overlay is not a second copy of it. Until then it is TUI-only and <code>help_page</code> is already in <code>ViewState</code>, which is why gotcha 32 says the overlay needs no <code>Projection</code>.</p>
|
||||||
|
<h2 id="the-panel-widgets"><a class="anchor" href="#the-panel-widgets">#</a>The panel widgets</h2>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/panel.rs"><code>src/ui/panel.rs</code></a> holds the <code>PanelContent</code> trait and the generic <code>Panel</code> widget. The one non-obvious member is <code>panel()</code>: each impl declares its own identity, and the widget compares it against <code>AppMode::panel()</code> to decide whether to draw itself as active, instead of restating the mode match per panel.</p>
|
||||||
|
<p><code>category_panel.rs</code>, <code>formula_panel.rs</code>, <code>view_panel.rs</code> and <code>tile_bar.rs</code> each draw one cache and nothing else. Two facts worth keeping:</p>
|
||||||
|
<ul>
|
||||||
|
<li>The category panel draws one flattened tree — the axis travels on the entry (<a class="tid" href="../tickets.html#improvise-edp" title="Establish ViewModel layer: derived data between RenderCache and widgets">improvise-edp</a>), so there is no <code>View</code> in it.</li>
|
||||||
|
<li>The view panel's "which view is this client on" is read off <code>ViewState</code>, not the workbook: the active view is per-client and is deliberately absent from <code>ViewListCache</code> (<a class="tid" href="../tickets.html#improvise-8uz" title="Cursor and scroll are per-client state stored in the shared View">improvise-8uz</a>, gotcha 29).</li>
|
||||||
|
</ul>
|
||||||
|
<p><code>tile_bar.rs</code> scrolls by whole tiles: it finds the smallest start index at which the selected tile still fits in the available width, and draws <code>‹</code>/<code>›</code> overflow indicators. That is the file's only piece of real arithmetic and it is local geometry, not a shared derivation.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>Design notes — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../index.html">← Project home</a></li><li><a href="../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="design-notes.html" class="here">Design notes</a></li><li><a href="design-principles.html">Design principles</a></li><li><a href="repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="code/core.html">improvise-core</a></li><li><a href="code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="code/protocol.html">improvise-protocol</a></li><li><a href="code/command.html">improvise-command</a></li><li><a href="code/clients.html">The clients</a></li><li><a href="code/tui.html">The TUI</a></li><li><a href="code/gp.html">gp-tree and improvise-gp</a></li><li><a href="code/kb-tool.html">The kb tool</a></li><li><a href="code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="test-evidence/core.html">improvise-core</a></li><li><a href="test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="test-evidence/protocol.html">improvise-protocol</a></li><li><a href="test-evidence/command.html">improvise-command</a></li><li><a href="test-evidence/clients.html">The clients</a></li><li><a href="test-evidence/tui.html">The TUI</a></li><li><a href="test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/docs/design-notes.md">docs/design-notes.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#design-notes">Design Notes</a></li><li class="l2"><a href="#product-vision">Product Vision</a></li><li class="l2"><a href="#non-goals-v1">Non-Goals (v1)</a></li></ul></details><h1 id="design-notes">Design Notes</h1>
|
||||||
|
<blockquote><p><strong>Staleness warning:</strong> This document captures conceptual framing from the original specification. It is not kept in sync with the code. For current architecture and types, see <a href="repo-map.html"><code>context/repo-map.md</code></a> and <a href="design-principles.html"><code>context/design-principles.md</code></a>.</p></blockquote>
|
||||||
|
<h2 id="product-vision"><a class="anchor" href="#product-vision">#</a>Product Vision</h2>
|
||||||
|
<p>Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without rewriting formulas.</p>
|
||||||
|
<p>Improvise treats data as a multi-dimensional, semantically labeled structure -- separating data, computation, and views into independent layers. Formulas reference meaningful names, views can be rearranged instantly, and the same dataset can be explored from multiple perspectives simultaneously.</p>
|
||||||
|
<p>The application compiles to a single static binary and provides a rich TUI experience.</p>
|
||||||
|
<h2 id="non-goals-v1"><a class="anchor" href="#non-goals-v1">#</a>Non-Goals (v1)</h2>
|
||||||
|
<ul>
|
||||||
|
<li>Scripting/macro language beyond the formula system.</li>
|
||||||
|
<li>Collaborative/multi-user editing.</li>
|
||||||
|
<li>Live external data sources (databases, APIs).</li>
|
||||||
|
<li>Charts or graphical visualization.</li>
|
||||||
|
<li>Multi-level undo history.</li>
|
||||||
|
</ul>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,193 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>Design principles — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../index.html">← Project home</a></li><li><a href="../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="design-notes.html">Design notes</a></li><li><a href="design-principles.html" class="here">Design principles</a></li><li><a href="repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="code/core.html">improvise-core</a></li><li><a href="code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="code/protocol.html">improvise-protocol</a></li><li><a href="code/command.html">improvise-command</a></li><li><a href="code/clients.html">The clients</a></li><li><a href="code/tui.html">The TUI</a></li><li><a href="code/gp.html">gp-tree and improvise-gp</a></li><li><a href="code/kb-tool.html">The kb tool</a></li><li><a href="code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="test-evidence/core.html">improvise-core</a></li><li><a href="test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="test-evidence/protocol.html">improvise-protocol</a></li><li><a href="test-evidence/command.html">improvise-command</a></li><li><a href="test-evidence/clients.html">The clients</a></li><li><a href="test-evidence/tui.html">The TUI</a></li><li><a href="test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/design-principles.md">context/design-principles.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#improvise-design-principles">Improvise Design Principles</a></li><li class="l2"><a href="#1-functional-first-architecture">1. Functional-First Architecture</a></li><li class="l3"><a href="#commands-are-pure-effects-are-side-effectful">Commands Are Pure, Effects Are Side-Effectful</a></li><li class="l3"><a href="#prefer-transformations-to-mutation">Prefer Transformations to Mutation</a></li><li class="l3"><a href="#compose-small-pieces">Compose Small Pieces</a></li><li class="l3"><a href="#decompose-rather-than-early-return">Decompose rather than early return</a></li><li class="l2"><a href="#2-polymorphism-over-conditionals">2. Polymorphism Over Conditionals</a></li><li class="l3"><a href="#dispatch-through-traits-and-registries-not-match-blocks">Dispatch Through Traits and Registries, Not Match Blocks</a></li><li class="l3"><a href="#use-enums-to-make-invalid-states-unrepresentable">Use Enums to Make Invalid States Unrepresentable</a></li><li class="l3"><a href="#when-you-add-a-variant-the-compiler-finds-every-call-site">When You Add a Variant, the Compiler Finds Every Call Site</a></li><li class="l2"><a href="#3-correctness-by-construction">3. Correctness by Construction</a></li><li class="l3"><a href="#canonical-forms-prevent-equivalence-bugs">Canonical Forms Prevent Equivalence Bugs</a></li><li class="l3"><a href="#smart-constructors-enforce-invariants">Smart Constructors Enforce Invariants</a></li><li class="l3"><a href="#type-safe-identifiers">Type-Safe Identifiers</a></li><li class="l3"><a href="#symbol-interning-for-data-integrity">Symbol Interning for Data Integrity</a></li><li class="l3"><a href="#parse-time-validation">Parse-Time Validation</a></li><li class="l3"><a href="#grammar-defined-file-format">Grammar-Defined File Format</a></li><li class="l3"><a href="#cl-style-pipe-quoting-for-names">CL-Style Pipe Quoting for Names</a></li><li class="l3"><a href="#formula-identifiers-and-quoting">Formula Identifiers and Quoting</a></li><li class="l2"><a href="#4-separation-of-concerns">4. Separation of Concerns</a></li><li class="l3"><a href="#four-layers">Four Layers</a></li><li class="l3"><a href="#formulas-are-data-not-code">Formulas Are Data, Not Code</a></li><li class="l3"><a href="#formula-evaluation-is-fixed-point">Formula Evaluation Is Fixed-Point</a></li><li class="l3"><a href="#display-rounding-is-view-only">Display Rounding Is View-Only</a></li><li class="l3"><a href="#drill-state-isolates-edits">Drill State Isolates Edits</a></li><li class="l2"><a href="#5-a-front-end-displays-state-it-does-not-reimplement-behaviour">5. A Front End Displays State; It Does Not Reimplement Behaviour</a></li><li class="l2"><a href="#6-guidelines-for-new-code">6. Guidelines for New Code</a></li><li class="l2"><a href="#7-testing">7. Testing</a></li><li class="l3"><a href="#coverage-and-ambition">Coverage and ambition</a></li><li class="l3"><a href="#demonstrate-bugs-before-fixing-them">Demonstrate bugs before fixing them</a></li><li class="l3"><a href="#a-check-that-looks-like-the-answer-answering-a-neighbouring-question">A check that looks like the answer, answering a neighbouring question</a></li><li class="l3"><a href="#e2e-pins-the-boundaries-unit-tests-work-between-them">E2E pins the boundaries; unit tests work between them</a></li><li class="l3"><a href="#use-property-tests-judiciously">Use property tests judiciously</a></li><li class="l3"><a href="#what-to-test">What to test</a></li><li class="l3"><a href="#what-not-to-test">What not to test</a></li><li class="l3"><a href="#test-the-property-not-the-implementation">Test the property, not the implementation</a></li></ul></details><h1 id="improvise-design-principles">Improvise Design Principles</h1>
|
||||||
|
<h2 id="1-functional-first-architecture"><a class="anchor" href="#1-functional-first-architecture">#</a>1. Functional-First Architecture</h2>
|
||||||
|
<h3 id="commands-are-pure-effects-are-side-effectful"><a class="anchor" href="#commands-are-pure-effects-are-side-effectful">#</a>Commands Are Pure, Effects Are Side-Effectful</h3>
|
||||||
|
<p>Every user action flows through a two-phase pipeline:</p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>Command</strong> (<code>Cmd</code> trait) — reads immutable context, returns a list of effects. The <code>CmdContext</code> is a read-only snapshot: model, layout, mode, cursor position. Commands never touch <code>&mut App</code>. All decision logic is pure.</li>
|
||||||
|
<li><strong>Effect</strong> (<code>Effect</code> trait) — a small struct with an <code>apply(&self, app: &mut App)</code> method. Each effect is one discrete, debuggable state change. The app applies them in order.</li>
|
||||||
|
</ol>
|
||||||
|
<p>This separation means:</p>
|
||||||
|
<ul>
|
||||||
|
<li>Commands are testable without a terminal or an <code>App</code> instance.</li>
|
||||||
|
<li>Effects can be logged, replayed, or composed.</li>
|
||||||
|
<li>The only place <code>App</code> is mutated is inside <code>Effect::apply</code>.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="prefer-transformations-to-mutation"><a class="anchor" href="#prefer-transformations-to-mutation">#</a>Prefer Transformations to Mutation</h3>
|
||||||
|
<p>Where possible, build new values rather than mutating in place:</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>CellKey::with(cat, item)</code> returns a new key with an added/replaced coordinate.</li>
|
||||||
|
<li><code>CellKey::without(cat)</code> returns a new key with a coordinate removed.</li>
|
||||||
|
<li>Viewport positioning is computed as a pure function (<code>viewport_effects</code>) that returns a <code>Vec<Effect></code>, not a method that pokes at scroll offsets directly.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="compose-small-pieces"><a class="anchor" href="#compose-small-pieces">#</a>Compose Small Pieces</h3>
|
||||||
|
<p>Commands compose via <code>Binding::Sequence</code> — a keymap entry can chain multiple commands, each contributing effects independently. The <code>o</code> key (add row + begin editing) is two commands composed at the binding level, not a monolithic handler.</p>
|
||||||
|
<h3 id="decompose-rather-than-early-return"><a class="anchor" href="#decompose-rather-than-early-return">#</a>Decompose rather than early return</h3>
|
||||||
|
<p>Early Returns usually are a signal of mixed responsibilities: if an early return would clarify a function, consider how the function could be decomposed for the same effect without the early return.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="2-polymorphism-over-conditionals"><a class="anchor" href="#2-polymorphism-over-conditionals">#</a>2. Polymorphism Over Conditionals</h2>
|
||||||
|
<h3 id="dispatch-through-traits-and-registries-not-match-blocks"><a class="anchor" href="#dispatch-through-traits-and-registries-not-match-blocks">#</a>Dispatch Through Traits and Registries, Not Match Blocks</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Commands</strong>: 40+ types each implement <code>Cmd</code>, organized by concern across submodules in <code>command/cmd/</code> (navigation, cell, commit, grid, mode, panel, search, text_buffer, tile, effect_cmds). A <code>CmdRegistry</code> maps names to constructor closures. Dispatching a key press looks up the binding, resolves the command name through the registry, and calls <code>execute</code>. No central <code>match command_name { ... }</code> block.</li>
|
||||||
|
<li><strong>Effects</strong>: 50+ types each implement <code>Effect</code>. Collected into a <code>Vec<Box<dyn Effect>></code> and applied in order. No <code>match effect_kind { ... }</code>.</li>
|
||||||
|
<li><strong>Keymaps</strong>: Each mode has its own <code>Keymap</code> (a <code>HashMap<KeyPattern, Binding></code>). Mode dispatch is one table lookup, not a nested <code>match (mode, key)</code>.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="use-enums-to-make-invalid-states-unrepresentable"><a class="anchor" href="#use-enums-to-make-invalid-states-unrepresentable">#</a>Use Enums to Make Invalid States Unrepresentable</h3>
|
||||||
|
<ul>
|
||||||
|
<li><code>BinOp</code> is an enum (<code>Add | Sub | Mul | ...</code>), not a string. Invalid operators are caught at parse time, not silently ignored at eval time.</li>
|
||||||
|
<li><code>Axis</code> is <code>Row | Column | Page | None</code>. A category is on exactly one axis. Cycling is a four-state rotation — no boolean flags, no "row_or_column" ambiguity.</li>
|
||||||
|
<li><code>Binding</code> is <code>Cmd | Prefix | Sequence</code>. The keymap lookup returns one of these three shapes; dispatch pattern-matches exhaustively.</li>
|
||||||
|
<li><code>CategoryKind</code> is <code>Regular | VirtualIndex | VirtualDim | VirtualMeasure | Label</code>. Business rules (e.g., the 12-category limit counts only <code>Regular</code>) are enforced by matching on the enum, not by checking name prefixes. Virtual categories (<code>_Index</code>, <code>_Dim</code>, <code>_Measure</code>) always exist: <code>_Index</code> and <code>_Dim</code> support drill-down/records mode; <code>_Measure</code> holds numeric data fields and formula targets (added automatically by <code>add_formula</code>). Use <code>Model::regular_category_names()</code> when selecting a default category for prompts or other user-visible choices.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="when-you-add-a-variant-the-compiler-finds-every-call-site"><a class="anchor" href="#when-you-add-a-variant-the-compiler-finds-every-call-site">#</a>When You Add a Variant, the Compiler Finds Every Call Site</h3>
|
||||||
|
<p>Prefer exhaustive <code>match</code> over <code>if let</code> or <code>_ =></code> wildcards. When a new <code>Axis</code> variant or <code>AppMode</code> is added, non-exhaustive matches produce compile errors that guide you to every place that needs updating.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="3-correctness-by-construction"><a class="anchor" href="#3-correctness-by-construction">#</a>3. Correctness by Construction</h2>
|
||||||
|
<h3 id="canonical-forms-prevent-equivalence-bugs"><a class="anchor" href="#canonical-forms-prevent-equivalence-bugs">#</a>Canonical Forms Prevent Equivalence Bugs</h3>
|
||||||
|
<p><code>CellKey::new()</code> sorts coordinates by category name. Two keys that name the same intersection but in different order are identical after construction. Equality, hashing, and storage all work correctly without callers needing to remember to sort. Property tests verify this invariant.</p>
|
||||||
|
<h3 id="smart-constructors-enforce-invariants"><a class="anchor" href="#smart-constructors-enforce-invariants">#</a>Smart Constructors Enforce Invariants</h3>
|
||||||
|
<ul>
|
||||||
|
<li><code>CellKey::new()</code> is the only way to build a key — it always sorts.</li>
|
||||||
|
<li><code>Category::add_item()</code> deduplicates by name and auto-assigns IDs via a private counter. External code cannot fabricate an <code>ItemId</code>.</li>
|
||||||
|
<li><code>Model::add_category()</code> checks the 12-category limit before insertion.</li>
|
||||||
|
<li><code>Formula::new()</code> takes all required fields; there is no default/empty formula to accidentally leave half-initialized.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="type-safe-identifiers"><a class="anchor" href="#type-safe-identifiers">#</a>Type-Safe Identifiers</h3>
|
||||||
|
<p><code>CategoryId</code> and <code>ItemId</code> are typed aliases. While they are <code>usize</code> underneath, using named types signals intent and prevents accidentally passing an item count where an item ID is expected.</p>
|
||||||
|
<h3 id="symbol-interning-for-data-integrity"><a class="anchor" href="#symbol-interning-for-data-integrity">#</a>Symbol Interning for Data Integrity</h3>
|
||||||
|
<p><code>DataStore</code> interns category and item names into <code>Symbol</code> values (small copyable handles). This means:</p>
|
||||||
|
<ul>
|
||||||
|
<li>String comparison is integer comparison — fast and allocation-free.</li>
|
||||||
|
<li>A secondary index maps <code>(Symbol, Symbol)</code> pairs to cell sets, enabling O(1) lookups for aggregation queries.</li>
|
||||||
|
<li>Symbols can only be created through the <code>SymbolTable</code>, so misspelled names produce a distinct symbol rather than silently matching a wrong cell.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="parse-time-validation"><a class="anchor" href="#parse-time-validation">#</a>Parse-Time Validation</h3>
|
||||||
|
<p>Formulas are parsed into a typed AST (<code>Expr</code> enum) at entry time. If the syntax is invalid, the user gets an error immediately. The evaluator only sees well-formed trees — it does not need to handle malformed input.</p>
|
||||||
|
<h3 id="grammar-defined-file-format"><a class="anchor" href="#grammar-defined-file-format">#</a>Grammar-Defined File Format</h3>
|
||||||
|
<p>The <code>.improv</code> file format is defined by a PEG grammar (<code>persistence/improv.pest</code>) and parsed by pest. The grammar is the single source of truth — the parser is a tree-walker over the grammar's parse tree, not an ad-hoc line scanner. This means:</p>
|
||||||
|
<ul>
|
||||||
|
<li>Adding a new format feature means updating the grammar first, then the walker.</li>
|
||||||
|
<li>The grammar can be read as a specification independent of the Rust code.</li>
|
||||||
|
<li>A grammar-walking test generator reads the grammar AST at test time (via <code>pest_meta</code>) and produces random valid files, ensuring the parser accepts everything the grammar describes.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="cl-style-pipe-quoting-for-names"><a class="anchor" href="#cl-style-pipe-quoting-for-names">#</a>CL-Style Pipe Quoting for Names</h3>
|
||||||
|
<p>Names in the <code>.improv</code> format use CL-style <code>|...|</code> pipe quoting. A name is bare if it matches <code>[A-Za-z_][A-Za-z0-9_-]*</code>; everything else must be pipe-quoted. Escapes inside pipes: <code>\|</code> (literal pipe), <code>\\</code> (backslash), <code>\n</code> (newline). This convention is shared between the <code>.improv</code> persistence format and the formula parser's identifier syntax.</p>
|
||||||
|
<h3 id="formula-identifiers-and-quoting"><a class="anchor" href="#formula-identifiers-and-quoting">#</a>Formula Identifiers and Quoting</h3>
|
||||||
|
<p><strong>Bare identifiers</strong> are <code>(ASCII_ALPHA | "_") ~ (ASCII_ALPHANUMERIC | "_" | "-")*</code> — no spaces, the same shape <code>improv.pest</code>'s <code>bare_name</code> has. A multi-word name such as <code>Total Revenue</code> must be pipe-quoted.</p>
|
||||||
|
<p><strong>Keywords need a word boundary after them, not before.</strong> <code>IS</code>, <code>WHERE</code> and the aggregate names are spelled <code>^"…"</code> in <code>formula.pest</code>, and <code>is_expr</code>/<code>where_clause</code>/ <code>inline_where</code> are compound-atomic with <code>sp+</code> so that <code>Region ISLAND</code> and <code>SUM(Rev WHEREAS x = y)</code> are parse errors rather than silently-wrong filters. A <code>SUM</code>/<code>IF</code> without parens falls through to <code>ref_expr</code>, which is why a bare keyword still names an item; the writer quotes it anyway (<code>quote::quote_name</code>).</p>
|
||||||
|
<p><strong>Pipe-quoted identifiers</strong> (<code>|...|</code>) allow any characters — including spaces, keywords, and operators — inside the delimiters. Use pipes when a category or item name collides with a keyword or contains special characters:</p>
|
||||||
|
<pre><code>|WHERE| — category named "WHERE"
|
||||||
|
|Revenue (USD)| — name with parens
|
||||||
|
|Cost + Tax| — name with operator chars
|
||||||
|
SUM(|Net Revenue| WHERE |Region Name| = |East Coast|)</code></pre>
|
||||||
|
<p><code>identifier</code> is <code>pipe_quoted | bare_ident</code>, so a pipe-quoted name works everywhere a bare one does: expressions, aggregate arguments, and a <code>WHERE</code> clause's category name. A double-quoted string is a <code>filter_value</code> only — the right-hand side of a <code>WHERE</code> or an <code>IS</code>, never an expression.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="4-separation-of-concerns"><a class="anchor" href="#4-separation-of-concerns">#</a>4. Separation of Concerns</h2>
|
||||||
|
<h3 id="four-layers"><a class="anchor" href="#four-layers">#</a>Four Layers</h3>
|
||||||
|
<table><thead><tr><th>Layer</th><th>Directory</th><th>Responsibility</th></tr></thead><tbody><tr><td><strong>Model</strong></td><td><code>src/model/</code></td><td>Categories, items, groups, cell data, formulas. Pure data, no rendering.</td></tr><tr><td><strong>View</strong></td><td><code>src/view/</code></td><td>Axis assignments, page selection, hidden items, layout computation. Derived from model.</td></tr><tr><td><strong>Command / Effect</strong></td><td><code>src/command/</code>, <code>src/ui/effect.rs</code></td><td>Intent (commands) and state mutation (effects). Bridges user input to model changes.</td></tr><tr><td><strong>Rendering</strong></td><td><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/draw.rs"><code>src/draw.rs</code></a>, <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/src/ui"><code>src/ui/</code></a></td><td>Terminal drawing. Reads model + view, writes pixels. No mutation.</td></tr></tbody></table>
|
||||||
|
<h3 id="formulas-are-data-not-code"><a class="anchor" href="#formulas-are-data-not-code">#</a>Formulas Are Data, Not Code</h3>
|
||||||
|
<p>A formula is a serializable struct: raw text, target name, category, AST, optional filter. It is stored in the model alongside regular data. The evaluator walks the AST at read time. Formulas never become closures or runtime-generated code.</p>
|
||||||
|
<h3 id="formula-evaluation-is-fixed-point"><a class="anchor" href="#formula-evaluation-is-fixed-point">#</a>Formula Evaluation Is Fixed-Point</h3>
|
||||||
|
<p><code>recompute_formulas(none_cats)</code> iterates formula evaluation until values stabilize. Each pass evaluates all formula cells using the current cache (for formula-derived values) and raw data aggregation (for data values). This avoids recursive evaluation through <code>evaluate_aggregated</code> and naturally handles chained formulas (<code>Margin = Profit / Revenue</code> where <code>Profit = Revenue - Cost</code>). Circular references converge to <code>CellValue::Error("circular")</code> after <code>MAX_EVAL_DEPTH</code> iterations.</p>
|
||||||
|
<h3 id="display-rounding-is-view-only"><a class="anchor" href="#display-rounding-is-view-only">#</a>Display Rounding Is View-Only</h3>
|
||||||
|
<p>Number formatting (<code>format_f64</code>) rounds for display. Formula evaluation always operates on full <code>f64</code> precision. The rounding function is only called in rendering paths — never in <code>eval_formula</code> or aggregation.</p>
|
||||||
|
<h3 id="drill-state-isolates-edits"><a class="anchor" href="#drill-state-isolates-edits">#</a>Drill State Isolates Edits</h3>
|
||||||
|
<p>When editing aggregated (drill-down) cells, a <code>DrillState</code> snapshot freezes the current cell set. Pending edits accumulate in a staging map. On commit, <code>ApplyAndClearDrill</code> writes them all atomically. On cancel, the snapshot is discarded. No partial writes reach the model.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="5-a-front-end-displays-state-it-does-not-reimplement-behaviour"><a class="anchor" href="#5-a-front-end-displays-state-it-does-not-reimplement-behaviour">#</a>5. A Front End Displays State; It Does Not Reimplement Behaviour</h2>
|
||||||
|
<p><strong>The goal is to minimise the amount of new code a new UI framework needs.</strong> Logic is centralised in a redux-like store, so a second or third front end is mostly a function from state to pixels plus a wire for events — not a second implementation of how the application behaves.</p>
|
||||||
|
<p>The pieces already exist and are worth naming as what they are:</p>
|
||||||
|
<table><thead><tr><th>redux</th><th>here</th></tr></thead><tbody><tr><td>action</td><td><code>Command</code> / <code>ModelCommand</code> (<code>improvise-protocol</code>)</td></tr><tr><td>reducer</td><td><code>reduce_view</code>, and <code>WidgetCache::apply</code> per widget</td></tr><tr><td>store</td><td><code>ViewState</code> + the per-widget caches</td></tr><tr><td>dispatch</td><td><code>Sessions::apply_checked</code> — the one door (gotcha 36)</td></tr></tbody></table>
|
||||||
|
<p>So the question to ask of any code in a front end is: <strong>would a third front end have to write this again?</strong> If yes, it is behaviour and it belongs in the shared layer. Three things make this concrete rather than aspirational:</p>
|
||||||
|
<p><strong>The measure is lines of framework-specific code, and it should trend to "paint and wire".</strong> <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/app.js"><code>web/app.js</code></a> is the current benchmark: a few hundred lines that know no protocol, reaching the application through a handful of string calls. A front end that needs to understand <code>CellKey</code> coordinates, or which category a row label belongs to, or what a keypress means, is one that has been handed a job the store should have done.</p>
|
||||||
|
<p><strong>A derivation done twice is the defect, even when both copies are correct today.</strong> Two front ends deriving the same fact will diverge, and the divergence is invisible until someone compares them side by side. <code>improvise-edp</code> is the worked example — the category tree was flattened twice, two different ways, and the fix put <code>viewmodel::flatten_cat_tree</code> where both ends call it. Gotcha 44 was the same defect and is now closed the same way (<a class="tid" href="tickets.html#improvise-anr9" title="Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once">improvise-anr9</a>): the grid's <em>frame</em> — header levels, label tuples, group runs and their collapse state — is <code>improvise_protocol::AxisFrame</code>, and both <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/src/ui/grid.rs"><code>src/ui/grid.rs</code></a> and <code>improvise-wasm-client/src/render.rs</code> ask it rather than deriving one. Note what the second cure cost that the first did not: a <code>Vec<String></code> of labels had to change <strong>type</strong>, and a shared derivation could not be added beside the old one — <code>row_levels</code> next to <code>row_labels</code> would have been two answers to "what are the labels", which is the disease and not the cure.</p>
|
||||||
|
<p><strong>Sharing the table beats testing two copies of it.</strong> Where a front end must turn input into actions, share the mapping rather than mirroring it: two hand-kept copies held in step by a test are one definition waiting to be made (<code>doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished)</code>). That is why the keymap is shared rather than reimplemented per front end, and why <code>doctrine(mirror_the_cmd_line_for_line)</code> is a rule about <em>resolvers</em> — the narrow, unavoidable residue — and never a licence to restate a table.</p>
|
||||||
|
<p><strong>What legitimately stays per-framework.</strong> Painting, event wiring, and the mapping from a semantic value to a framework type: <code>AxisColor → ratatui::Color</code> in <code>draw.rs::axis_fg</code>, <code>MinibufColor → minibuf_fg</code>. That mapping is the shape to imitate — the <em>decision</em> is in the shared layer as a semantic enum, and only the last translation is local. Anything larger than a lookup table is a candidate for the store.</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="6-guidelines-for-new-code"><a class="anchor" href="#6-guidelines-for-new-code">#</a>6. Guidelines for New Code</h2>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Add a command, not a special case.</strong> If you need new behavior on a keypress, implement <code>Cmd</code>, register it, and bind it in the keymap. Do not add an <code>if key == 'x'</code> branch inside <code>handle_key</code>.</li>
|
||||||
|
<li><strong>Return effects, do not mutate.</strong> Your command's <code>execute</code> receives <code>&CmdContext</code> (immutable). Produce a <code>Vec<Box<dyn Effect>></code>. If you need a new kind of state change, create a new <code>Effect</code> struct.</li>
|
||||||
|
<li><strong>Use the type system.</strong> If a value can only be one of N things, make it an enum. If an invariant must hold, enforce it in the constructor. If a field is optional, use <code>Option</code> — do not use sentinel values.</li>
|
||||||
|
<li><strong>Test the logic, not the wiring.</strong> Commands are pure functions of context; test them by building a <code>CmdContext</code> and asserting on the returned effects. You do not need a terminal.</li>
|
||||||
|
<li><strong>Keep <code>Option</code>/<code>Result</code>/<code>Box</code> at the boundaries.</strong> Core logic should work with concrete values. Wrap in <code>Option</code> at the edges (parsing, lookup, I/O) and unwrap early. Do not thread <code>Option</code> through deep call chains.</li>
|
||||||
|
<li><strong>Reasoning lives in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a>, not in comments.</strong> A comment run is <strong>at most two consecutive lines</strong>, <code>///</code> and <code>//!</code> included, enforced by <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-comment-runs.py"><code>scripts/check-comment-runs.py</code></a>. Where a reader would otherwise re-derive a rule wrongly, name the <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a> file that carries the argument — and only there, because a pointer on every function is noise. The prose lives in <code>context/code/<area>.md</code>; the rule exists because the arguments in this codebase are its most expensive artefact and a comment is the one place nothing checks them against the code (user, 2026-09-20; <a class="tid" href="tickets.html#improvise-9c8c" title="Move the reasoning out of code comments into context/ prose, cap comments at two lines">improvise-9c8c</a>). A ``` fence in a doc comment is a doctest — a running test, not prose — and is exempt.</li>
|
||||||
|
<li><strong>And an argument must not live only in a commit message.</strong> A message is write-once and unauditable: it cannot be corrected without rewriting merged history, and no check in this repo can read one — the ID audit greps <code>context</code>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/README.md"><code>README.md</code></a>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/AGENTS.md"><code>AGENTS.md</code></a>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/CLAUDE.md"><code>CLAUDE.md</code></a>, <code>crates</code>, <code>src</code>, <code>tests</code>, <code>examples</code>, <code>web</code>, <code>harness</code> and <code>docs</code>, and a commit message is in none of them. So a reason recorded only there is a reason nothing can ever find wrong. When one turns out to be wrong, correct <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a> and say in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a> that the message is superseded; that is the fix rather than a compromise. PR #51's <code>7d25762</code> is the worked example, corrected by #56.</li>
|
||||||
|
</ul>
|
||||||
|
<hr />
|
||||||
|
<h2 id="7-testing"><a class="anchor" href="#7-testing">#</a>7. Testing</h2>
|
||||||
|
<h3 id="coverage-and-ambition"><a class="anchor" href="#coverage-and-ambition">#</a>Coverage and ambition</h3>
|
||||||
|
<p>Aim for <strong>~80% line and branch coverage</strong> on logic code. This is a quality floor — go higher where the code is tricky or load-bearing, but don't pad coverage by testing trivial getters or chasing 100% on rendering widgets. The test suite should remain fast, and the rule is <strong>two numbers</strong> because the ambiguity between them is what let the old single "under 2 seconds" drift: <strong>no single test binary may exceed 2 seconds</strong>, and the workspace wall clock is the sum of them. Excludes compilation, and excludes <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/harness"><code>harness/</code></a>, which is Playwright and takes minutes in its own CI job. Slow tests erode the habit of running them.</p>
|
||||||
|
<p><strong>The threshold is a tripwire, and the seconds are not portable.</strong> The same commit measured <code>improvise-io</code>'s lib binary at 1.4s on one machine and 3.7–5.2s on another (<a class="tid" href="tickets.html#improvise-8ec3" title="improvise-io's test binary breaches the 2s per-binary rule on some machines, and the pinned figure is not reproducible">improvise-8ec3</a>), so a PR body citing "longest single binary 1.47s" is checkable and <em>false</em> for the reader who checks it, which is worse than no claim at all. Three consequences:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Re-measure; never re-assert.</strong> <code>cargo test --workspace</code> prints a <code>finished in</code> per binary. Read your own run.</li>
|
||||||
|
<li><strong>If you cite a figure, name the machine you took it on.</strong> A number with no machine is a property of the suite, which it is not.</li>
|
||||||
|
<li><strong>The portable signal is the ratio, not the seconds.</strong> One binary an order of magnitude above the run's <strong>median</strong> is the thing to look at — not above second place, which it need not be: on one machine <code>improvise-io</code> is 2.2× the runner-up and ~15× the median. It is the same binary on all three machines it has been measured on: <code>improvise-io</code>'s lib, which was proptest-dominated by the grammar-walking <code>.improv</code> generator and the round-trip properties over it.</li>
|
||||||
|
</ul>
|
||||||
|
<p><strong><a class="tid" href="tickets.html#improvise-8shu" title="Constrain the .improv grammar generator's strategy; improvise-io's lib binary is the run's outlier">improvise-8shu</a> is closed, and its diagnosis was wrong</strong> — which is the more useful half. The ticket blamed an unbounded grammar walk and asked for a constrained strategy. Instrumenting the two halves separately: <code>load_grammar()</code> ×1000 took <strong>2.50s</strong> against a <strong>15.7ms</strong> walk, so the walk was <strong>0.6%</strong> of the cost and <code>improv_file()</code> was re-parsing <code>improv.pest</code> once per generated case. Caching it in a <code>OnceLock</code> took <code>improvise-io</code>'s lib from <strong>1.42s to 0.21s</strong> with the strategy's input space <em>unchanged</em>; constraining it would have moved 15.7ms and shipped as "the fix". Measured at <code>f152d10</code> on the orchestrator's machine: <strong>1445 tests over 34 binaries</strong>, longest single binary <strong>0.6–1.0s across two runs</strong> — the spread on one machine in one sitting being its own argument for reading your own run. <code>improvise-io</code> is no longer the outlier. Lowering the case count below 256 is still not on the table (see below).</p>
|
||||||
|
<h3 id="demonstrate-bugs-before-fixing-them"><a class="anchor" href="#demonstrate-bugs-before-fixing-them">#</a>Demonstrate bugs before fixing them</h3>
|
||||||
|
<p>Write a test that <strong>fails on the current code</strong> before writing the fix. Prefer a small unit test targeting the broken function over an end-to-end test. After the fix, the test stays as a regression guard.</p>
|
||||||
|
<p><strong>Record the bug in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a>, keyed by test name</strong> — what broke, what fault was injected, and why the test can fail. It does not go in the test's doc-comment: comment runs are capped at two lines (see below), and the evidence is longer than that in every case worth having. The cost of the split is that a renamed test orphans its entry. <code>scripts/check-comment-runs.py --check-evidence</code> catches that, in two directions. Every <code>###</code>/<code>####</code> heading shaped like a test name must name a test that <strong>runs</strong> — the names are diffed against <code>cargo test --workspace -- --list</code> minus its <code>--ignored</code> twin, never scraped out of the source, because text that looks like a test and a test are indistinguishable to a scanner (<a class="tid" href="tickets.html#improvise-rm4s" title="The evidence check accepts headings naming tests that never run">improvise-rm4s</a>). And a test whose doc comment <em>cites</em> <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> must have an entry there, or the pointer reads as a promise the explanation exists.</p>
|
||||||
|
<p>Neither direction is "every test has evidence", deliberately: most tests have none and should not be forced to have one. The reasoning and the residual holes are in <a href="code/tooling.html"><code>context/code/tooling.md</code></a>.</p>
|
||||||
|
<h3 id="a-check-that-looks-like-the-answer-answering-a-neighbouring-question"><a class="anchor" href="#a-check-that-looks-like-the-answer-answering-a-neighbouring-question">#</a>A check that looks like the answer, answering a neighbouring question</h3>
|
||||||
|
<p>The characteristic defect in this repo is not a wrong line; it is a check that is <em>adjacent</em> to the thing it names. Three instances landed in one campaign (PRs #59/#60), in three different layers, which is why it is worth stating as a shape rather than three gotchas:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>A vacuous assertion.</strong> <code>set_panel_open_and_cursor</code> asserted <code>category_panel_open</code> was true after an effect set it true — once the launch default supplied true, the effect could be replaced by a <strong>no-op</strong> and 1391 tests passed. It answered <em>is the flag true</em> where it meant <em>did the effect set it</em>.</li>
|
||||||
|
<li><strong>A vacuous repair of it.</strong> The first fix folded <code>[open:false, open:true]</code> and asserted the final state — which a reducer that folds nothing also satisfies. It answered <em>does the text mention the other direction</em> where it meant <em>does the writer move the flag both ways</em>. <strong>Reading the diff is what failed</strong>, and it failed because the vacuous version contained <code>open: false</code>.</li>
|
||||||
|
<li><strong>A clean auto-merge.</strong> <code>git merge-tree</code> reporting no conflict answers <em>can git combine these texts</em>, where the question is <em>does the combination do what both sides meant</em>. Two PRs edited the same four files with no conflict and neither side CI-tested together.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Each has the same remedy shape: <strong>name the question you actually mean, then find an input that separates it from its neighbour.</strong></p>
|
||||||
|
<ul>
|
||||||
|
<li>For an assertion: <strong>assert the direction the default cannot supply.</strong> Closing the panel discriminates; opening it proves nothing.</li>
|
||||||
|
<li>For a repair: inject a <strong>one-directional</strong> fault (<code>if open { set true }</code> — can raise, never lower). A vacuous repair cannot see it, because a test that only asserts the true direction is satisfied by a writer that only ever writes true.</li>
|
||||||
|
<li>For a merge: <strong>materialise it and run the gates</strong>, and check that a count which should MOVE actually moved. Playwright going 16 → 17 is what showed the other side survived; "no conflict" and an unchanged total would both have been consistent with losing it.</li>
|
||||||
|
</ul>
|
||||||
|
<p>This generalises <code>doctrine(fold_tests_must_test_order_not_the_axiom)</code> — asserting <code>fold(prefix)+fold(rest) == fold(whole)</code> is the definition of a left fold, so it answers a neighbouring question too. What the campaign adds is that the pattern recurs in <strong>test repairs</strong> and in <strong>merge hygiene</strong>, not only in production code and not only in fold tests.</p>
|
||||||
|
<h3 id="e2e-pins-the-boundaries-unit-tests-work-between-them"><a class="anchor" href="#e2e-pins-the-boundaries-unit-tests-work-between-them">#</a>E2E pins the boundaries; unit tests work between them</h3>
|
||||||
|
<p><strong>E2E and integration tests are rare and pin <em>boundary contracts</em>. Unit tests <em>assume</em> those contracts and test between the boundaries</strong> (user, 2026-09-20).</p>
|
||||||
|
<p>So the question to ask of a slow test is never "what feature does this cover" — it is <strong>"which boundary is this, and what is the contract at it?"</strong> The answer changes what you write. Four browser widgets had never worked — the tile bar, the view panel, the formula panel, and the category sidebar ignoring <code>category_panel_open</code> — and all four were <strong>one</strong> boundary failure: a <code>RenderModel</code> field with no element painting it. Four per-widget Playwright specs cost four slow tests and would still have missed the fifth; a single contract assertion that <em>every field the render model exposes is painted</em> catches all four and the next one. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/harness/tests/opfs.spec.js"><code>harness/tests/opfs.spec.js</code></a> is the cleanest instance of the right shape already in the tree: it runs the <strong>storage contract</strong>, not a feature.</p>
|
||||||
|
<p>Three corollaries, in the order they bite:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>A unit test may assume the contract rather than re-establishing it.</strong> That assumption is what makes it cheap, and it is only sound because something slow pins the contract — so deleting the contract test silently invalidates every unit test that leaned on it.</li>
|
||||||
|
<li><strong>Never weaken a contract to make it cheap.</strong> If the honest contract needs a real browser, a real <code>Worker</code> or real OPFS, it stays slow. Moving it to a faster harness that cannot see the thing it names is how a suite goes green over a broken product.</li>
|
||||||
|
<li><strong>Each slow test must say which contract it pins.</strong> The difference between a contract test and a feature test is invisible from the code, so it has to be written down or the next person adds the fifth per-widget spec.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="use-property-tests-judiciously"><a class="anchor" href="#use-property-tests-judiciously">#</a>Use property tests judiciously</h3>
|
||||||
|
<p>Property tests (<code>proptest</code>) are for <strong>invariants that must hold across all inputs</strong> — not a replacement for example-based tests. Good candidates:</p>
|
||||||
|
<ul>
|
||||||
|
<li>Structural invariants: CellKey is always sorted, each category lives on exactly one axis, toggle-collapse is involutive, hide/show roundtrips.</li>
|
||||||
|
<li>Serialization roundtrips: save/load identity.</li>
|
||||||
|
<li>Determinism: <code>evaluate</code> returns the same result for the same inputs.</li>
|
||||||
|
</ul>
|
||||||
|
<p>Keep case counts at the default (256). Don't crank them to thousands — if a property needs more cases to feel safe, constrain the input space with better strategies rather than brute-forcing. Property tests that take hundreds of milliseconds each are a sign something is wrong.</p>
|
||||||
|
<h3 id="what-to-test"><a class="anchor" href="#what-to-test">#</a>What to test</h3>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Model, formula, view</strong>: the core logic. Unit tests for each operation and edge case. Property tests for invariants. These are the highest-value tests.</li>
|
||||||
|
<li><strong>Commands</strong>: build a <code>CmdContext</code>, call <code>execute</code>, assert on the returned effects. Pure functions — no terminal needed. Tests are colocated in each command submodule (<code>command/cmd/<module>.rs</code> → <code>mod tests</code>), with shared test helpers in <code>command/cmd/mod.rs::test_helpers</code>.</li>
|
||||||
|
<li><strong>Persistence</strong>: round-trip tests (<code>save → load → save</code> produces identical output) plus grammar-driven property tests. The generator walks the pest grammar AST to produce random valid files; proptests verify <code>parse(generate())</code> succeeds and <code>parse(format(parse(generate())))</code> is stable. Cover groups, formulas, views, hidden items, pipe quoting edges.</li>
|
||||||
|
<li><strong>Format</strong>: boundary cases for comma placement, rounding, negative numbers.</li>
|
||||||
|
<li><strong>Import</strong>: field classification heuristics, CSV quoting, multi-file merge.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="what-not-to-test"><a class="anchor" href="#what-not-to-test">#</a>What not to test</h3>
|
||||||
|
<ul>
|
||||||
|
<li>Ratatui <code>Widget::render</code> implementations — pure drawing code that changes often. Test the data they consume (layout, cat_tree, format) instead.</li>
|
||||||
|
<li>Trivial data definitions (<code>ast.rs</code>, <code>axis.rs</code>).</li>
|
||||||
|
<li>Module re-export files.</li>
|
||||||
|
</ul>
|
||||||
|
<h3 id="test-the-property-not-the-implementation"><a class="anchor" href="#test-the-property-not-the-implementation">#</a>Test the property, not the implementation</h3>
|
||||||
|
<p>A test like "calling <code>set_axis(cat, Row)</code> sets the internal map entry to <code>Row</code>" is brittle — it mirrors the implementation and breaks if the storage changes. Instead test the observable contract: "after <code>set_axis(cat, Row)</code>, <code>axis_of(cat)</code> returns <code>Row</code> and <code>categories_on(Row)</code> includes <code>cat</code>." This style survives refactoring and catches real bugs.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>About this site — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../index.html">← Project home</a></li><li><a href="../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="design-notes.html">Design notes</a></li><li><a href="design-principles.html">Design principles</a></li><li><a href="repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="code/core.html">improvise-core</a></li><li><a href="code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="code/protocol.html">improvise-protocol</a></li><li><a href="code/command.html">improvise-command</a></li><li><a href="code/clients.html">The clients</a></li><li><a href="code/tui.html">The TUI</a></li><li><a href="code/gp.html">gp-tree and improvise-gp</a></li><li><a href="code/kb-tool.html">The kb tool</a></li><li><a href="code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="test-evidence/core.html">improvise-core</a></li><li><a href="test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="test-evidence/protocol.html">improvise-protocol</a></li><li><a href="test-evidence/command.html">improvise-command</a></li><li><a href="test-evidence/clients.html">The clients</a></li><li><a href="test-evidence/tui.html">The TUI</a></li><li><a href="test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="doc-site.html" class="here">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/doc-site.md">context/code/doc-site.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#the-documentation-site-the-arguments">The documentation site — the arguments</a></li><li class="l2"><a href="#1-what-is-published-and-what-is-not">1. What is published, and what is not</a></li><li class="l2"><a href="#2-ticket-ids-resolve-to-a-page-never-to-a-url-that-does-not-exist">2. Ticket ids resolve to a page, never to a URL that does not exist</a></li><li class="l2"><a href="#3-source-paths-link-only-where-the-file-exists">3. Source paths link only where the file exists</a></li><li class="l2"><a href="#4-the-link-check-is-the-test-and-it-is-part-of-the-build">4. The link check is the test, and it is part of the build</a></li><li class="l2"><a href="#5-why-a-node-script-and-not-a-generator">5. Why a node script and not a generator</a></li><li class="l2"><a href="#6-navigation">6. Navigation</a></li><li class="l2"><a href="#7-the-phone-nav-is-a-checkbox-and-that-is-not-arbitrary">7. The phone nav is a checkbox, and that is not arbitrary</a></li></ul></details><h1 id="the-documentation-site-the-arguments">The documentation site — the arguments</h1>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/build-doc-site.mjs"><code>scripts/build-doc-site.mjs</code></a> turns the prose in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context"><code>context/</code></a> into the static site served at <code>/docs/</code> on GitHub Pages. This is the reasoning behind what it publishes, what it deliberately does not, and why it is a hundred-odd lines of node rather than a static site generator (<a class="tid" href="tickets.html#improvise-djoo" title="Publish the context/ prose as a documentation site folded into GitHub Pages">improvise-djoo</a>).</p>
|
||||||
|
<hr />
|
||||||
|
<h2 id="1-what-is-published-and-what-is-not"><a class="anchor" href="#1-what-is-published-and-what-is-not">#</a>1. What is published, and what is not</h2>
|
||||||
|
<p>The manifest is the <code>SECTIONS</code> array at the top of the script, and <strong>a document is published only by being named there</strong>. That is the point: a glob over <code>context/**</code> would publish the next working file somebody drops in, and the decision about what a stranger should read is not one to make by accident.</p>
|
||||||
|
<p>Published, in four sections:</p>
|
||||||
|
<table><thead><tr><th>Section</th><th>Contents</th><th>Why</th></tr></thead><tbody><tr><td>Start here</td><td><a href="design-notes.html"><code>docs/design-notes.md</code></a>, <a href="design-principles.html"><code>context/design-principles.md</code></a>, <a href="repo-map.html"><code>context/repo-map.md</code></a></td><td>What the project is for, the doctrine, and the single best explanation of how it is built.</td></tr><tr><td>Decision records</td><td><code>context/adr/*.md</code></td><td>An ADR is written for an outside reader by construction — it states the options that were rejected, which is the part that is worth reading and the part that is nowhere else.</td></tr><tr><td>The arguments, by area</td><td><code>context/code/*.md</code></td><td>This is the reasoning that used to live in code comments and was moved out under the two-line cap (design-principles §6). It is the most valuable prose in the repository and it is addressed to exactly the reader a documentation site has.</td></tr><tr><td>Test evidence</td><td><code>context/test-evidence/*.md</code></td><td>501 entries, one per test whose narrative is longer than its code.</td></tr></tbody></table>
|
||||||
|
<p>Not published:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/plan.md"><code>context/plan.md</code></a></strong> is a <em>prompt</em>: it opens "You are working in the <code>improvise</code> repository… Your job is to prepare this repository for a public Show HN launch", and it instructs a reader to audit a <code>context/SPEC.md</code> that no longer exists. It is an instruction to an agent about a launch, not a description of the program, and publishing a launch plan on the site the launch points at inverts the two.</li>
|
||||||
|
<li><strong><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/cloud-agents.md"><code>context/cloud-agents.md</code></a></strong> is operational notes for agents running in a cloud session — tracker hydration, <code>bd</code> pitfalls, what the sandbox blocks. It is true and it is useful and it is addressed to nobody who visits a website.</li>
|
||||||
|
<li><strong><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/pr-review.md"><code>context/code/pr-review.md</code></a></strong> is how an agent reads a PR's reviews — where a ChatGPT Codex review appears, how to tell it from this repo's own agents' comments, and why absence is not approval. It is about <em>this repository's process</em>, not about the program, which is the <code>cloud-agents.md</code> line and not the <code>tooling.md</code> one: <code>tooling.md</code> explains checkers a reader of the code will meet, and nobody visiting a website is about to read a PR thread.</li>
|
||||||
|
</ul>
|
||||||
|
<p>None of the three is referenced by any published document except as a row in <code>repo-map.md</code>'s file inventory, which is inside a code fence, so leaving them out creates no dangling link. That was checked rather than assumed.</p>
|
||||||
|
<p><strong>And "not published" is now derived rather than assumed</strong> (<a class="tid" href="tickets.html#improvise-res9" title="Doc site: derive the published set, so an unpublished context/ doc fails the build">improvise-res9</a>). The generator walks <code>context/**/*.md</code> plus <a href="design-notes.html"><code>docs/design-notes.md</code></a>, subtracts the manifest, and fails unless the remainder is exactly the <code>OMITTED</code> map in the script — which carries the reason beside the name, so this section and the code cannot drift into disagreeing. An allowlist and not a count, because an omission is a decision that should have to be argued in code; and the map is checked in <em>both</em> directions, because an entry naming a file nobody can find asserts a reason for nothing, which is the failure a dead ticket id is.</p>
|
||||||
|
<p>The check was latent when it was filed and <strong>live by the time it was written</strong>: <a href="code/kb-tool.html"><code>context/code/kb-tool.md</code></a> had been added after this table was drafted and was publishing nothing, with no argument for its absence anywhere. The table above says <code>context/code/*.md</code> is published; the manifest disagreed and nothing could see it. That is the whole failure mode in one file.</p>
|
||||||
|
<p><strong>Test evidence is the one judgement call worth stating.</strong> The argument for leaving it out is that 501 entries about individual tests are noise to a reader who wanted to know what a pivot table is. The argument for keeping it is stronger: every one of the seven <code>context/code/*.md</code> documents points at its evidence file by name, so omitting it would break the published prose in exactly the way this site is otherwise careful not to; and the evidence file is the most distinctive artefact the repository has — a written record of the injected fault behind each load-bearing test is not a thing most projects can show. It is therefore published as its <strong>own section</strong>, last, described as a contributor reference rather than a tour, so nobody wanders into it expecting a narrative.</p>
|
||||||
|
<h2 id="2-ticket-ids-resolve-to-a-page-never-to-a-url-that-does-not-exist"><a class="anchor" href="#2-ticket-ids-resolve-to-a-page-never-to-a-url-that-does-not-exist">#</a>2. Ticket ids resolve to a page, never to a URL that does not exist</h2>
|
||||||
|
<p>The prose cites bead ids roughly 190 times — this paragraph deliberately spells none, because a document that gives one as an example makes itself a source of audit noise, and a placeholder is worse than a real id: it is ID-shaped and resolves to nothing. On a public site those citations are meaningless strings, and the obvious repair — link them to an issue tracker — is unavailable: beads keeps its issues in a Dolt database on a git ref, and there is no public issue URL to point at. Inventing one would produce 190 links that 404, which is worse than plain text because a link is a promise.</p>
|
||||||
|
<p>So the ids resolve <strong>inward</strong>. <code>tickets.html</code> is generated from <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/.beads/issues.jsonl"><code>.beads/issues.jsonl</code></a> — already committed, already public — and carries id, status and title, nothing else; descriptions are internal working notes and are not the site's business. Each citation in the prose becomes a link to that page's anchor, with the title in a <code>title=</code> attribute so a hover answers the question without a navigation.</p>
|
||||||
|
<p><strong><code>issues.jsonl</code> is a passive export and lags the database</strong>, which is not a defect to work around but a fact to render honestly: an id the export does not carry is emitted as plain text with a dotted underline and no link, and the ticket page lists it as "not exported". The site therefore never links to an anchor it did not generate, whatever state the export is in. A generator that instead linked every id it saw would produce a page that is right today and broken the first time somebody cites a fresh ticket.</p>
|
||||||
|
<p><strong>The page publishes every cited id, open bugs and their titles included, and that is a decision rather than a default</strong> (user ruling). The data is already public in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/.beads/issues.jsonl"><code>.beads/issues.jsonl</code></a> in a repository anyone can clone; the page only makes it legible. An honest open-bug list is a normal thing for an open-source project to have, and filtering one while the page's own lede links the raw JSONL would be theatre. Note the second-order reason to keep it unfiltered: a citation whose row was suppressed would be a link into a page that does not answer it, which is the failure this whole section exists to avoid.</p>
|
||||||
|
<p><strong>Titles are tracker data and are escaped as attribute values.</strong> Five shipped bead titles contain a double quote; interpolated into <code>title="…"</code> by an <code>esc</code> that handled only <code>&</code>, <code><</code> and <code>></code>, they broke out of the attribute and the browser parsed the remains as three junk attributes. <code>esc</code> now escapes <code>"</code> and <code>'</code> as well. Today's corpus being friendly is not a reason to interpolate untrusted-shaped text unescaped — the tracker is written by whoever files a bug.</p>
|
||||||
|
<h2 id="3-source-paths-link-only-where-the-file-exists"><a class="anchor" href="#3-source-paths-link-only-where-the-file-exists">#</a>3. Source paths link only where the file exists</h2>
|
||||||
|
<p>A code span whose entire content is a repo path becomes a link — to the generated page when the path is one of the published documents, to GitHub otherwise. Two constraints make that safe. The whole span must be the path, so a sentence that merely mentions a directory is untouched; and <strong>the path must exist on disk at generation time</strong>, so a citation that has gone stale degrades to plain text rather than to a 404. The GitHub ref is the cache-bust token <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a> already receives, used only when it is a 40-hex sha and falling back to <code>main</code> otherwise — a local <code>dev</code> build must not emit <code>blob/dev/…</code>.</p>
|
||||||
|
<h2 id="4-the-link-check-is-the-test-and-it-is-part-of-the-build"><a class="anchor" href="#4-the-link-check-is-the-test-and-it-is-part-of-the-build">#</a>4. The link check is the test, and it is part of the build</h2>
|
||||||
|
<p>The generator collects every anchor it emits and every internal href it writes, resolves them against each other, and <strong>exits non-zero</strong> if any href names a page it did not generate or a fragment that page does not carry. <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a> runs under <code>set -eu</code>, so a broken cross-reference fails the assembly rather than shipping.</p>
|
||||||
|
<p>This is not ceremony. It caught two real defects on its first two runs: a fence-close regex that compared a three-character marker against a one-character string, which swallowed every document from its first code block onward; and a list parser that ended the gotcha list at the first blank line, so 47 of the 48 gotcha anchors were never emitted while the "jump to gotcha N" strip linked to all of them. Both were invisible in a rendered page that otherwise looked fine.</p>
|
||||||
|
<p>It is proved by injection rather than asserted: renaming the emitted anchor ids makes it report 48 missing anchors, and pointing the ticket links at a page name that is never generated makes it report 763 missing pages — two different diagnoses, because "the page is absent" and "the page is there but the anchor is not" call for different repairs.</p>
|
||||||
|
<p>Links that <strong>leave</strong> the <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/docs"><code>docs/</code></a> subtree — <code>../index.html</code>, <code>../app/</code> — name files this script does not generate, so they are resolved against the filesystem instead. That is the whole reason <code>assemble.sh</code> runs the generator <strong>last</strong>: its siblings have to exist before they can be checked. An earlier version skipped those 48 links while still counting them as checked, behind a comment promising they were "checked below" when there was no below.</p>
|
||||||
|
<p>That ordering is itself checked now (<a class="tid" href="tickets.html#improvise-res9" title="Doc site: derive the published set, so an unpublished context/ doc fails the build">improvise-res9</a>). The generator reads <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a> and refuses to run unless its own invocation is that script's last command. PR #63 appended a loop and it happened to land above the generator; one landing below would have made the outward-link check pass against a half-built tree, silently and in the one direction this section says must not go unnoticed.</p>
|
||||||
|
<p><strong>And the link check is not sufficient, which is the more important lesson.</strong> It answers one question — does every href resolve — and a defect outside that question is invisible to it however carefully it is run. The worked example is a nested <code><a></code>: a TOC entry wrapping a heading whose rendered text already contained a link produced <code><a href="#anchor"><a href="github…">…</a></a></code>, and the HTML parser closes the outer anchor early, so the entry became an empty unclickable stub and clicking it navigated to GitHub. Both hrefs resolved perfectly. It shipped past the link check, past the whole-site link check, and past a screenshot pass, and was caught by a reviewer <em>clicking a link</em>. The repair is therefore two things, not one: the TOC keeps a tag-free variant of each heading, <strong>and</strong> a separate check asserts no <code><a></code> appears inside another <code><a></code> — proved by reverting the first half and watching the second report 66.</p>
|
||||||
|
<p>The general form: when a check misses a defect, add the check that could have seen it rather than only fixing the defect, and ask what <em>question</em> the existing check was answering.</p>
|
||||||
|
<h2 id="5-why-a-node-script-and-not-a-generator"><a class="anchor" href="#5-why-a-node-script-and-not-a-generator">#</a>5. Why a node script and not a generator</h2>
|
||||||
|
<p>The landing page is hand-written HTML with no build step and one pinned CDN script; the same posture applies here. A static site generator would add a dependency tree, a lockfile and a toolchain to a repository whose site is currently four files, in exchange for features this corpus does not use — the docs contain no front matter, no shortcodes and, as it happens, <strong>not a single markdown link</strong>. Every cross-reference in them is prose naming a path or an id, which is precisely why §2 and §3 are the interesting parts and Markdown rendering is the boring part.</p>
|
||||||
|
<p>So the renderer is a deliberately small subset covering exactly what the corpus contains: ATX headings, fenced code, pipe tables, blockquotes, ordered and unordered lists with indented continuations, thematic breaks, and inline code, strong, emphasis and links. Anything it does not recognise becomes paragraph text, so <strong>no input line is ever dropped</strong> — the failure mode is an ugly paragraph, never a missing one.</p>
|
||||||
|
<p>Two details in it are load-bearing and easy to undo. <strong>Code spans are replaced by placeholders before emphasis is matched</strong>, because this corpus routinely opens <code>**</code> on one side of a code span and closes it on the other; matching emphasis per segment silently drops that bold, which is a fidelity loss with no visible symptom. And <strong>strong is matched before emphasis, with a body that may contain a single <code>*</code> pair</strong>, because <code>[^*]+</code> cannot match <code>**a *b* c**</code> and the leftover <code>*</code> then pairs across the bold — which is how a single sentence in gotcha 33 came out as three italic paragraphs.</p>
|
||||||
|
<p>A third, added later and for the same reason: <strong>a table cell boundary is an unescaped pipe outside a code span</strong>, so the row is walked rather than <code>split("|")</code>. This corpus documents a file format whose own quoting syntax is <code>|Time Period|</code>, and a sentence containing one inside a table cell would have been shredded into extra columns — a row that still parses, so no column count could detect it. It was latent when fixed, which is the point: the trigger is one ordinary sentence away in a document whose subject is pipe quoting.</p>
|
||||||
|
<p>One deliberate divergence from GFM inside that rule: a <code>\|</code> <strong>inside a code span</strong> in a table cell keeps its backslash, where GFM would unescape it to a bare pipe. Splitting is suppressed inside code spans, so the escape is never needed there and never consumed. The corpus contains no instance; it is recorded as a decision rather than left to be rediscovered as a bug.</p>
|
||||||
|
<h2 id="6-navigation"><a class="anchor" href="#6-navigation">#</a>6. Navigation</h2>
|
||||||
|
<p><code>repo-map.md</code> is one long document whose gotcha list is numbered 1 to 48, and splitting it per gotcha was rejected: the gotchas cross-reference each other constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a reader following those wants a scroll, not 48 page loads. It stays one page, with an id on every list item and a compact strip of numbered links under the heading. The ids come from the list item's <strong>own number</strong>, not from a counter, so an inserted gotcha renumbers the anchors exactly as it renumbers the prose that cites them.</p>
|
||||||
|
<p>The strip's contents are <strong>derived from the anchors actually emitted</strong>, never from a count. A hardcoded 48 failed in only one direction: deleting a gotcha left a link to a missing anchor, which the link check catches, while <em>adding</em> one produced an unlinked gotcha and a clean build. The list only ever grows, so the hardcoded form was silent in precisely the direction it would be used — the same shape as an exclusion list that over-matches, failing in the direction that looks like success.</p>
|
||||||
|
<p><strong>Long unbreakable headings need <code>overflow-wrap</code>.</strong> The test-evidence pages are hundreds of <code>snake_case</code> test names as headings; at phone width seven of them overflowed the viewport by up to 92%. The diagnosis is worth keeping because the first guess was wrong: <code>getBoundingClientRect</code> reported every element inside the viewport, because the text overflows its box without widening it. Only <code>scrollWidth</code> per element names the culprit. Measure the property that actually overflowed rather than the one that usually does.</p>
|
||||||
|
<h2 id="7-the-phone-nav-is-a-checkbox-and-that-is-not-arbitrary"><a class="anchor" href="#7-the-phone-nav-is-a-checkbox-and-that-is-not-arbitrary">#</a>7. The phone nav is a checkbox, and that is not arbitrary</h2>
|
||||||
|
<p>At phone width the sidebar's 25 links stacked above the content, so every page opened on roughly 1500px of nav. The wall was not inherited: <strong>all of this CSS is new</strong>, so it shipped for the first time with the site, which is the test for whether a defect is pre-existing and the one most easily got wrong in the direction of declining work you introduced.</p>
|
||||||
|
<p><code><details></code> is the obvious control and it is <strong>wrong here</strong>, for a reason no spec-level reading predicts. The sidebar layout has to force the disclosure <em>open</em> at wide widths, and a closed <code><details></code> hides its content through <code>::details-content</code> — which <code>display</code> cannot override. Measured in Chromium, the desktop sidebar came out collapsed to a single link with its toggle also hidden: unusable, and in the half nobody would think to re-check after a phone-only change. A hidden checkbox plus a <code><label></code> puts the hiding in <em>our</em> media query, so forcing it open is ordinary cascade. Anyone reading the markup will see a checkbox where <code><details></code> obviously belongs; this paragraph is why it must not be tidied into one.</p>
|
||||||
|
<p><strong>And the first probe of the fix could not see the difference.</strong> Counting nav links whose <code>getBoundingClientRect().height > 0</code> reported <strong>26 visible in both the collapsed and the expanded state</strong>, because Chrome reports a height for <code>content-visibility</code>-hidden children — so it would have passed the broken <code><details></code> build. <code>checkVisibility({contentVisibilityAuto: true, visibilityProperty: true})</code> discriminates.</p>
|
||||||
|
<p>That is the same instrument failing twice: §6's overflow fix also measured clean by <code>getBoundingClientRect</code> while the page still scrolled. <strong>A measurement that cannot distinguish the two states is not a measurement</strong> — the repo's signature defect, a test that passes while the thing it names is broken, arriving through the instrument rather than through the assertion. Before trusting a browser measurement, injure the thing being measured and check the number moves.</p>
|
||||||
|
<p><strong>And the fix's own probe had the same shape one level up.</strong> Counting <em>visible</em> nav links answers "does operating the toggle reveal the links" — correct and complete for a mouse — and is silent on whether any non-mouse route exists. The first version of the checkbox carried <code>hidden</code>, which is <code>display:none</code>, which removes it from the tab order <em>and</em> from the accessibility tree: the label is not focusable on its own, so at ≤900px the navigation could be clicked and reached no other way, and the accessibility snapshot contained neither the checkbox nor any of the 25 links. A visibility count cannot see that, however sharp the visibility instrument. The toggle is therefore visually hidden but focusable at narrow widths and <code>display:none</code> at wide ones, and the check is tab order, <code>.focus()</code> and the accessibility snapshot — measured at 390, 900, 901 and 1280, because the regression lives on one side of a media query.</p>
|
||||||
|
<p>So the rule from §6 has a second half: <strong>a measurement of the wrong property is not a measurement either, however precise it is.</strong> <code>checkVisibility</code> fixed the visibility instrument; it does not turn a visibility count into an operability check. Ask what the defect would <em>do</em> to a user, then pick the instrument that observes that.</p>
|
||||||
|
<p><strong>A correction, and the risk the original argument missed.</strong> The commit that added <code>content: "\2630 " / ""</code> said Firefox ignores the alt-text form and falls back to reading the content, so the change was "an improvement in two engines and a no-op in the third". <strong>That mechanism is wrong</strong> — Firefox shipped the syntax in 133, and all three engines parse it (measured: Chrome 141, Firefox 142, WebKit 26 each compute the declaration intact). The conclusion held; the reason given for it did not, and the reason was the load-bearing part, because it was what made the change look like a one-way door. It is one because every engine shipped to supports the syntax — not because the unsupported path is harmless. That commit message is superseded by this paragraph (design-principles §6: an argument must not live only in a commit message).</p>
|
||||||
|
<p><strong>The unsupported path is not harmless, and that is the risk neither the change nor its first review named.</strong> <code>/ alt</code> in an engine that does not know it is a <em>parse error for the whole <code>content</code> declaration</em>, so the glyph would not lose its alt text — it would <strong>vanish</strong>. A silent visual regression, the opposite of the assumed failure mode. The glyph is therefore verified painted in all three engines by pixel diff against <code>::before { content: none }</code>, with the text starting 20px further right in each.</p>
|
||||||
|
<p>That check also produced this file's fourth instance of the wrong-property trap, and the sharpest, because it produced a confident <strong>false positive</strong>: the first probe measured the label's <em>width</em> with and without <code>::before</code> and reported the glyph missing in all three engines. The label is <code>display: block</code>, so its width comes from the container and cannot see inline content at all. Right question, wrong property, and this time the wrong instrument accused working code rather than clearing broken code.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>Documentation — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../index.html">← Project home</a></li><li><a href="../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="design-notes.html">Design notes</a></li><li><a href="design-principles.html">Design principles</a></li><li><a href="repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="code/core.html">improvise-core</a></li><li><a href="code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="code/protocol.html">improvise-protocol</a></li><li><a href="code/command.html">improvise-command</a></li><li><a href="code/clients.html">The clients</a></li><li><a href="code/tui.html">The TUI</a></li><li><a href="code/gp.html">gp-tree and improvise-gp</a></li><li><a href="code/kb-tool.html">The kb tool</a></li><li><a href="code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="test-evidence/core.html">improvise-core</a></li><li><a href="test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="test-evidence/protocol.html">improvise-protocol</a></li><li><a href="test-evidence/command.html">improvise-command</a></li><li><a href="test-evidence/clients.html">The clients</a></li><li><a href="test-evidence/tui.html">The TUI</a></li><li><a href="test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<h1>improvise — documentation</h1>
|
||||||
|
<p class="lede">These are the working documents of the improvise repository, published as written. They are addressed to someone reading or changing the code: the arguments behind the architecture, the decisions that were made and the ones that were rejected, and the record of what each load-bearing test is guarding against.</p>
|
||||||
|
<p class="lede">Nothing here is a tutorial. For what the program does, see the <a href="../index.html">project home</a> or run the <a href="../app/">browser demo</a>.</p>
|
||||||
|
<h2 id="start"><a class="anchor" href="#start">#</a>Start here</h2><p class="lede">What the project is for, and the principles the code is held to.</p><div class="cards"><div class="card"><a href="design-notes.html">Design notes</a></div><div class="card"><a href="design-principles.html">Design principles</a></div><div class="card"><a href="repo-map.html">Repository map</a></div></div><h2 id="adr"><a class="anchor" href="#adr">#</a>Decision records</h2><p class="lede">Decisions with consequences, including the options that were rejected.</p><div class="cards"><div class="card"><a href="adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></div><div class="card"><a href="adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></div><div class="card"><a href="adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></div></div><h2 id="code"><a class="anchor" href="#code">#</a>The arguments, by area</h2><p class="lede">The reasoning that used to live in code comments. Written for a reader with the source open.</p><div class="cards"><div class="card"><a href="code/core.html">improvise-core</a></div><div class="card"><a href="code/formula-and-io.html">improvise-formula and improvise-io</a></div><div class="card"><a href="code/protocol.html">improvise-protocol</a></div><div class="card"><a href="code/command.html">improvise-command</a></div><div class="card"><a href="code/clients.html">The clients</a></div><div class="card"><a href="code/tui.html">The TUI</a></div><div class="card"><a href="code/gp.html">gp-tree and improvise-gp</a></div><div class="card"><a href="code/kb-tool.html">The kb tool</a></div><div class="card"><a href="code/tooling.html">The repository's own checkers</a></div></div><h2 id="evidence"><a class="anchor" href="#evidence">#</a>Test evidence</h2><p class="lede">For each test whose narrative is longer than its code: what broke, what fault was injected, and why the test can fail. Contributor reference, not a tour.</p><div class="cards"><div class="card"><a href="test-evidence/core.html">improvise-core</a></div><div class="card"><a href="test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></div><div class="card"><a href="test-evidence/protocol.html">improvise-protocol</a></div><div class="card"><a href="test-evidence/command.html">improvise-command</a></div><div class="card"><a href="test-evidence/clients.html">The clients</a></div><div class="card"><a href="test-evidence/tui.html">The TUI</a></div><div class="card"><a href="test-evidence/gp.html">gp-tree and improvise-gp</a></div><div class="card"><a href="test-evidence/tooling.html">The repository's own gates</a></div></div><h2 id="meta"><a class="anchor" href="#meta">#</a>About this site</h2><p class="lede">What is published here, what is not, and why.</p><div class="cards"><div class="card"><a href="doc-site.html">About this site</a></div></div>
|
||||||
|
<h2 id="conventions"><a class="anchor" href="#conventions">#</a>Conventions</h2>
|
||||||
|
<p>Ids like <code>improvise-9c8c</code> are tracker issues and link to the ticket index below. Paths like <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/crates/improvise-core/src/model/types.rs"><code>crates/improvise-core/src/model/types.rs</code></a> link to the source on GitHub at <code>main</code>.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,60 @@
|
|||||||
|
:root{--bg:#14161a;--panel:#1b1e24;--fg:#d7dae0;--dim:#8b93a1;--accent:#8ec07c;--rule:#2b2f37;--mono:ui-monospace,"SFMono-Regular","Menlo","Consolas",monospace}
|
||||||
|
*{box-sizing:border-box}
|
||||||
|
body{margin:0;background:var(--bg);color:var(--fg);font:16px/1.65 system-ui,-apple-system,"Segoe UI",sans-serif}
|
||||||
|
a{color:var(--accent)}
|
||||||
|
.shell{display:flex;align-items:flex-start;max-width:1400px;margin:0 auto}
|
||||||
|
nav.side{position:sticky;top:0;flex:0 0 17rem;max-height:100vh;overflow-y:auto;padding:1.5rem 1rem 3rem;border-right:1px solid var(--rule);font-size:.9rem}
|
||||||
|
nav.side .brand{font-family:var(--mono);font-size:1.15rem;color:var(--accent);text-decoration:none;font-weight:600}
|
||||||
|
nav.side .navtoggle{display:none}
|
||||||
|
nav.side .navsummary{display:none;cursor:pointer;margin:.9rem 0 .2rem;padding:.4rem .55rem;border:1px solid var(--rule);border-radius:4px;font-family:var(--mono);font-size:.85rem;color:var(--dim)}
|
||||||
|
nav.side .navsummary::before{content:"\2630 " / ""}
|
||||||
|
nav.side .navtoggle:checked~.navsummary{border-color:var(--accent);color:var(--fg)}
|
||||||
|
nav.side h3{font-size:.72rem;letter-spacing:.09em;text-transform:uppercase;color:var(--dim);margin:1.6rem 0 .4rem}
|
||||||
|
nav.side ul{list-style:none;margin:0;padding:0}
|
||||||
|
nav.side li{margin:.15rem 0}
|
||||||
|
nav.side a{display:block;padding:.2rem .45rem;border-radius:4px;color:var(--fg);text-decoration:none}
|
||||||
|
nav.side a:hover{background:var(--panel)}
|
||||||
|
nav.side a.here{background:var(--panel);color:var(--accent);font-weight:600}
|
||||||
|
main{flex:1 1 auto;min-width:0;max-width:60rem;padding:2.5rem 2rem 6rem}
|
||||||
|
h1,h2,h3,h4,h5{font-family:var(--mono);font-weight:600;letter-spacing:-.01em;line-height:1.3;overflow-wrap:anywhere}
|
||||||
|
h1{font-size:2rem;margin:0 0 1.2rem;color:var(--accent)}
|
||||||
|
h2{font-size:1.25rem;margin:2.6rem 0 .6rem;padding-top:1.2rem;border-top:1px solid var(--rule)}
|
||||||
|
h3{font-size:1.02rem;margin:1.8rem 0 .4rem;color:#c8cdd6}
|
||||||
|
h4,h5{font-size:.95rem;margin:1.3rem 0 .3rem;color:var(--dim)}
|
||||||
|
.anchor{float:left;margin-left:-1.1em;padding-right:.35em;color:var(--rule);text-decoration:none;opacity:0}
|
||||||
|
h2:hover .anchor,h3:hover .anchor,h4:hover .anchor,h5:hover .anchor{opacity:1}
|
||||||
|
p{max-width:74ch}
|
||||||
|
li{max-width:74ch}
|
||||||
|
code{font-family:var(--mono);font-size:.88em;background:var(--panel);padding:.1em .3em;border-radius:3px;overflow-wrap:anywhere}
|
||||||
|
a code{background:#22262e}
|
||||||
|
pre{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:1rem;overflow-x:auto}
|
||||||
|
pre code{background:none;padding:0;font-size:.84rem;line-height:1.5;overflow-wrap:normal}
|
||||||
|
blockquote{margin:1rem 0;padding:.5rem .9rem;border-left:3px solid var(--accent);background:var(--panel)}
|
||||||
|
blockquote p{margin:.4rem 0}
|
||||||
|
table{border-collapse:collapse;margin:1rem 0;font-size:.92rem;display:block;overflow-x:auto}
|
||||||
|
th,td{border:1px solid var(--rule);padding:.4rem .6rem;text-align:left;vertical-align:top}
|
||||||
|
th{background:var(--panel)}
|
||||||
|
hr{border:0;border-top:1px solid var(--rule);margin:2rem 0}
|
||||||
|
.tid{font-family:var(--mono);font-size:.86em;white-space:nowrap}
|
||||||
|
.tid-unknown{color:var(--dim);border-bottom:1px dotted var(--rule)}
|
||||||
|
.toc{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:.8rem 1.1rem;margin:0 0 2rem;font-size:.92rem}
|
||||||
|
.toc summary{cursor:pointer;color:var(--dim);font-family:var(--mono)}
|
||||||
|
.toc ul{list-style:none;margin:.6rem 0 0;padding:0;columns:2;column-gap:2rem}
|
||||||
|
.toc li{break-inside:avoid;margin:.12rem 0;overflow-wrap:anywhere}
|
||||||
|
.toc li.l3{padding-left:1rem;font-size:.92em}
|
||||||
|
.toc a{text-decoration:none;overflow-wrap:anywhere}
|
||||||
|
.toc a:hover{text-decoration:underline}
|
||||||
|
.jump{margin:0 0 2rem;font-size:.85rem;font-family:var(--mono)}
|
||||||
|
.jump a{display:inline-block;min-width:2.1rem;text-align:center;padding:.12rem .3rem;margin:.1rem;border:1px solid var(--rule);border-radius:4px;text-decoration:none}
|
||||||
|
.jump a:hover{border-color:var(--accent)}
|
||||||
|
.lede{color:var(--dim);max-width:70ch}
|
||||||
|
.src{font-size:.85rem;color:var(--dim);font-family:var(--mono);margin:-.6rem 0 1.8rem}
|
||||||
|
.cards{display:grid;gap:1rem;grid-template-columns:repeat(auto-fill,minmax(19rem,1fr));margin:1rem 0 2rem}
|
||||||
|
.card{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:.9rem 1.1rem}
|
||||||
|
.card a{font-family:var(--mono);text-decoration:none}
|
||||||
|
.card p{color:var(--dim);font-size:.88rem;margin:.3rem 0 0}
|
||||||
|
.tickets{width:100%}
|
||||||
|
.tickets td:first-child{white-space:nowrap;font-family:var(--mono);font-size:.86rem}
|
||||||
|
.tickets td:last-child{overflow-wrap:anywhere}
|
||||||
|
.st{font-size:.78rem;color:var(--dim);text-transform:uppercase;letter-spacing:.05em}
|
||||||
|
@media(max-width:900px){.toc ul{max-height:45vh;overflow-y:auto}nav.side .navtoggle{display:block;position:absolute;width:1px;height:1px;margin:-1px;clip-path:inset(50%);overflow:hidden}nav.side .navtoggle:focus-visible~.navsummary{outline:2px solid var(--accent);outline-offset:2px}nav.side .navsummary{display:block}nav.side .navlinks{display:none}nav.side .navtoggle:checked~.navlinks{display:block}.shell{display:block}nav.side{position:static;flex:none;max-height:none;border-right:0;border-bottom:1px solid var(--rule)}main{padding:1.5rem 1rem 4rem}.toc ul{columns:1}}
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,66 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||||
|
<title>The repository's own gates — improvise docs</title>
|
||||||
|
<link rel="stylesheet" href="../style.css" />
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="shell">
|
||||||
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html" class="here">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||||||
|
<main>
|
||||||
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/test-evidence/tooling.md">context/test-evidence/tooling.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#test-evidence-the-repository-s-own-gates">Test evidence — the repository's own gates</a></li><li class="l3"><a href="#no-test-in-web-tests-reaches-for-a-built-bundle">"no test in web/tests/ reaches for a built bundle"</a></li><li class="l3"><a href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">"the subdirectory that MAY use a bundle is not the one being checked"</a></li><li class="l3"><a href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">"no test in web/tests/ imports a module that reaches the bundle"</a></li><li class="l3"><a href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">Note — the checker's own blockers, found by review of PR #83</a></li><li class="l3"><a href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">Note — the flag that selected the cap away (improvise-ijql)</a></li><li class="l3"><a href="#note-the-stamp-gate-s-discrimination-measured">Note — the stamp gate's discrimination, measured</a></li><li class="l3"><a href="#note-the-stamp-gate-matches-literally">Note — the stamp gate matches literally</a></li><li class="l3"><a href="#note-the-doc-site-s-derived-checks">Note — the doc site's derived checks</a></li><li class="l3"><a href="#duplicate-evidence-headings-self-test-duplicates">Duplicate evidence headings — SELF_TEST_DUPLICATES</a></li></ul></details><h1 id="test-evidence-the-repository-s-own-gates">Test evidence — the repository's own gates</h1>
|
||||||
|
<p>Evidence for the checks in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/scripts"><code>scripts/</code></a> and for the tests that hold CI's shape. The arguments behind the checkers themselves are in <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>; this file records what fault each test was watched to catch.</p>
|
||||||
|
<p>Note what is deliberately absent. <code>scripts/check-comment-runs.py --self-test</code>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb-test.sh"><code>scripts/kb-test.sh</code></a> and <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> are check-the-checker suites whose cases are already injected faults, and none of their case names is a Rust <code>fn</code> or a JS <code>test(</code> the evidence checker could resolve — so their evidence is the case list itself plus <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>, and inventing headings for them here would be headings naming nothing.</p>
|
||||||
|
<h3 id="no-test-in-web-tests-reaches-for-a-built-bundle"><a class="anchor" href="#no-test-in-web-tests-reaches-for-a-built-bundle">#</a>"no test in web/tests/ reaches for a built bundle"</h3>
|
||||||
|
<p>The <code>js</code> CI job runs <code>node --test web/tests/*.test.js</code> and installs nothing; the <code>browser</code> job runs <code>web/tests/bundle/*.test.js</code> after three wasm-pack builds. The moment a flat <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests"><code>web/tests/</code></a> file reads <code>web/pkg/</code>, the fast job inherits the slow one's dependency — and the likely repair is to add a build step to it, silently spending the ~4m30s the split exists to avoid. Until <a class="tid" href="../tickets.html#improvise-4ktt" title="The js job's bundle-freedom is a convention, not a gate">improvise-4ktt</a> that constraint was a directory convention and a glob, checked by nothing, and it failed in the direction that looks like ordinary breakage: a confusing ENOENT in CI on a machine where the bundle happens not to be built.</p>
|
||||||
|
<p>Fault injected, 2026-09-21: appended <code>const injected = readFileSync("../pkg/improvise_wasm_client.js");</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The test failed naming that file and pointing at <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a>. Reverted; green again.</p>
|
||||||
|
<p>Two things the test does that a simpler one would not. It asserts its own filename is among the files it scans, so it cannot become the one place a violation may hide. And it strips comments before matching, because this file's own header and the test's header both name the directory in prose — a lexical grep over raw text would have had to exempt them, and an exemption is where the next violation goes.</p>
|
||||||
|
<p><strong>What it cannot see</strong>, stated because the check itself uses the evasion: a path assembled at runtime (<code>"../" + "p" + "kg"</code>) is invisible to a lexical match, and the test's own <code>FORBIDDEN</code> constant is built exactly that way to stay out of its own scan. That is a deliberate, disclosed hole — closing it needs a parser, and the failure it would catch is one somebody has to write on purpose, where this gate exists for the one written by accident.</p>
|
||||||
|
<h3 id="the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked"><a class="anchor" href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">#</a>"the subdirectory that MAY use a bundle is not the one being checked"</h3>
|
||||||
|
<p>The companion, and the one that stops the first from passing vacuously. If <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a> were emptied or its files stopped touching the bundle, the first test would still be green while having nothing to be scoped <em>against</em> — "no flat test uses the bundle" is uninteresting if no test anywhere does. This asserts the other half of the split exists and really does what the flat half is forbidden to do.</p>
|
||||||
|
<p>Fault injected, 2026-09-21: moved <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/bundle/wasm_surface.test.js"><code>web/tests/bundle/wasm_surface.test.js</code></a> out of the tree. The test failed with "web/tests/bundle/ is empty — the split lost its other half" rather than passing. Restored; green again.</p>
|
||||||
|
<h3 id="no-test-in-web-tests-imports-a-module-that-reaches-the-bundle"><a class="anchor" href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">#</a>"no test in web/tests/ imports a module that reaches the bundle"</h3>
|
||||||
|
<p>The companion to the lexical check above, and the reason that one is not enough: <code>import "../app.js"</code> needs the bundle <em>transitively</em> and names nothing forbidden, so the <code>js</code> job breaks loudly but unattributed. <code>app.js</code> and <code>worker.js</code> are the two entry points that load the bundle; <code>shell.js</code> and <code>serial.js</code> do not, and <code>serial.test.js</code> imports one legitimately.</p>
|
||||||
|
<p>Fault injected, 2026-09-21: appended <code>import "../app.js";</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The lexical test stayed green; this one failed naming the file. Reverted.</p>
|
||||||
|
<h3 id="note-the-checker-s-own-blockers-found-by-review-of-pr-83"><a class="anchor" href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">#</a>Note — the checker's own blockers, found by review of PR #83</h3>
|
||||||
|
<p>Two gates in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-comment-runs.py"><code>scripts/check-comment-runs.py</code></a> produced a pass while not doing what their prose said. Both were in the mode the same PR rewrote, which is the part worth keeping: writing the doctrine down does not exempt the writer.</p>
|
||||||
|
<p><strong>A leaf name is not an identity.</strong> Six leaf names collide in this workspace and two were cited in evidence. Renaming <code>ui::project::tests::a_load_that_fails_is_not_a_rebase</code> — the one its entry describes — left the checker printing <em>all name a test that runs</em>, exit 0, with the heading answered by the <code>ui::host::tests</code> namesake. Now an ambiguous leaf is refused with both candidates named, and the two entries are qualified. Re-verified both ways, 2026-09-21: the refusal fires on the unqualified headings and names exactly those two; with the heading qualified, the same rename is reported as an orphan rather than absorbed.</p>
|
||||||
|
<p><strong><code>--rust-tests</code> was strictly more permissive and said the same words.</strong> It called <code>cargo_test_names</code> with one argument, so the <code>--ignored</code> subtraction never ran, and the success line was byte-identical to the honest path's. Injecting <code>### eyeball_the_rendered_hints_and_pages</code> orphaned on the default path and passed on the flag's. Now the flag requires <code>--rust-tests-ignored</code> (one alone exits 2) and the success line names its source.</p>
|
||||||
|
<p><strong>A missing evidence directory exited 0.</strong> The early return preceded the citation scan, so deleting <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> reported nothing while 170 citations pointed at a directory that was not there. Verified 2026-09-21 by moving the directory aside: exit 1, <em>170 tests cite context/test-evidence and have no entry there</em>.</p>
|
||||||
|
<h3 id="note-the-flag-that-selected-the-cap-away-improvise-ijql"><a class="anchor" href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">#</a>Note — the flag that selected the cap away (<a class="tid" href="../tickets.html#improvise-ijql" title="check-comment-runs.py --check-evidence silently skips the comment-run cap it is named for">improvise-ijql</a>)</h3>
|
||||||
|
<p><code>scripts/check-comment-runs.py --check-evidence</code> early-returned in <code>main</code>, so the flagged run never answered the comment-run cap and printed nothing about having skipped it. The reviewer's position when it was first raised was that CI runs all three spellings (<code>ci.yml</code>, the three <code>check-comment-runs.py</code> steps in the <code>test</code> job) so this was documentation only. The counter that carried is that the flagged form is the <strong>expensive</strong> one — it needs a compiled workspace — which makes it the one a human runs alone; three agents hit it independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment.</p>
|
||||||
|
<p>Reproduced before the fix, 2026-09-21, in a temporary tree holding one <code>.rs</code> file with a four-line comment run:</p>
|
||||||
|
<pre><code>$ check-comment-runs.py src --check-evidence --rust-tests … --rust-tests-ignored …
|
||||||
|
test-evidence: 0 entries, all name a test that runs …
|
||||||
|
EXIT=0
|
||||||
|
$ check-comment-runs.py src
|
||||||
|
1 comment runs longer than 2 lines:
|
||||||
|
src/bad.rs:1 4 lines
|
||||||
|
EXIT=1</code></pre>
|
||||||
|
<p>Two invocations over one tree, opposite verdicts, and the one that exits 0 is the one a human is more likely to run by itself.</p>
|
||||||
|
<p>The guard is <code>_dispatch_self_test</code>, which runs the script as a <strong>subprocess</strong> — the only level at which this is visible, since every unit in the file was correct and the defect was in <code>main</code>'s dispatch. Both arms were demonstrated:</p>
|
||||||
|
<table><thead><tr><th>injected fault</th><th>what failed</th></tr></thead><tbody><tr><td>restore the <code>--check-evidence</code> early return</td><td><code>DISPATCH: --check-evidence: exited 0 over a four-line comment run</code></td></tr><tr><td>make the cap always report a violation</td><td><code>DISPATCH: bare: refused a clean tree</code> <em>and</em> the same for <code>--check-evidence</code></td></tr></tbody></table>
|
||||||
|
<p>The second arm is what stops an always-failing cap from satisfying the first, and it is why <code>SELF_TEST_DISPATCH</code> asserts a clean tree passes as well as a dirty one failing. Why the fix is additive rather than a printed disclaimer, and why <code>--self-test</code>/<code>--list-doctests</code> are excluded from the table rather than exempted from the rule: <a href="../code/tooling.html"><code>context/code/tooling.md</code></a> §2, "A flag adds a question".</p>
|
||||||
|
<h3 id="note-the-stamp-gate-s-discrimination-measured"><a class="anchor" href="#note-the-stamp-gate-s-discrimination-measured">#</a>Note — the stamp gate's discrimination, measured</h3>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> separates three outcomes rather than two, which is what makes it evidence rather than a smoke test. Measured 2026-09-21 over its nine cases: the real script passes 9; deleting the final <code>test -z "$missing"</code> (the PR #69 injection, which the inline version survived at exit 0) passes 4 and fails 5; a script that does nothing passes 2; one that always fails passes 0.</p>
|
||||||
|
<p>The PR body first cited "4 of 9" for the middle row, from a run taken when the suite had eight cases. A figure is what the next reader checks, so it is recorded here where it can be re-measured rather than only in a message nothing can audit (design-principles §6).</p>
|
||||||
|
<h3 id="note-the-stamp-gate-matches-literally"><a class="anchor" href="#note-the-stamp-gate-matches-literally">#</a>Note — the stamp gate matches literally</h3>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp.sh"><code>scripts/assemble-stamp.sh</code></a> greps with <code>-F</code>. As a regex, the <code>.</code> in <code>improvise_wasm_client.js?v=abc</code> matches any character, so a stamp corrupted in exactly that position reads as present — a pass in the one direction a guard must not fail in. Inherited from the inline version this was extracted from, and found by reading the extraction rather than by anything failing.</p>
|
||||||
|
<p>Fault injected, 2026-09-21: dropping the <code>-F</code> makes <code>improvise_wasm_clientXjs?v=abc</code> satisfy the pattern, and <code>a pattern is matched literally, not as a regex</code> goes from pass to fail.</p>
|
||||||
|
<h3 id="note-the-doc-site-s-derived-checks"><a class="anchor" href="#note-the-doc-site-s-derived-checks">#</a>Note — the doc site's derived checks</h3>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/build-doc-site.mjs"><code>scripts/build-doc-site.mjs</code></a> carries two gates that no test names, because the generator <em>is</em> the test: it runs on every assembly and exits non-zero.</p>
|
||||||
|
<p>Faults injected, 2026-09-21, each observed:</p>
|
||||||
|
<table><thead><tr><th>injected</th><th>reported</th></tr></thead><tbody><tr><td>a new <code>context/code/newarea.md</code></td><td><code>context/code/newarea.md is neither published nor a named omission</code></td></tr><tr><td><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/plan.md"><code>context/plan.md</code></a> moved out of the tree</td><td><code>context/plan.md is listed as omitted but does not exist</code></td></tr><tr><td>a line appended below the generator in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a></td><td><code>web/assemble.sh must invoke this generator LAST</code>, naming the offending command</td></tr><tr><td>a trailing <code>echo "see scripts/build-doc-site.mjs"</code></td><td>the same, once the check asks for an INVOCATION rather than a mention — <code>.includes</code> had accepted it (review, PR #83)</td></tr></tbody></table>
|
||||||
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-ci-graph.py"><code>scripts/check-ci-graph.py</code></a> is the third, and it is a script rather than a stanza because its own refusals need checking: adding a <code>zz-new-check:</code> job to <code>ci.yml</code> reports <em>pages-deploy does not need <code>zz-new-check</code></em> and exits 1, where before the rule was prose and nothing noticed. Its five self-test graphs include the one a naive parse gets wrong — another job's <code>needs:</code> read as the deploy's.</p>
|
||||||
|
<p>The second arm is the one that would not get written first, and it is the point: without it, deleting or renaming an omitted file leaves an entry asserting a reason for a file nobody can find — the same failure as a dead ticket id, which <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/AGENTS.md"><code>AGENTS.md</code></a> already calls worse than no id.</p>
|
||||||
|
<p>The first arm was filed as latent and was live when it was written: <a href="../code/kb-tool.html"><code>context/code/kb-tool.md</code></a> had been unpublished, unargued and unnoticed since it was added.</p>
|
||||||
|
<h3 id="duplicate-evidence-headings-self-test-duplicates"><a class="anchor" href="#duplicate-evidence-headings-self-test-duplicates">#</a>Duplicate evidence headings — <code>SELF_TEST_DUPLICATES</code></h3>
|
||||||
|
<p><a class="tid" href="../tickets.html#improvise-hsoe" title="check-evidence cannot see a duplicate heading, and three exist">improvise-hsoe</a>. <code>--check-evidence</code> verified that a heading names a test that <strong>runs</strong>, and a duplicate heading names one just as well, so both copies pass. PR #92 shipped two entries under <code>a_layout_change_moves_the_layout_version_and_no_record</code>, and two more pairs sat in <code>tui.md</code> — one of them asserting that <code>THE_MEMBERSHIP_CORRECT_SPANS</code> was "a live tripwire, do not remove it" some four hundred lines above the entry recording that <a class="tid" href="../tickets.html#improvise-gd83" title="AxisEntry carries no group-end boundary, so group_of gives an ungrouped item the preceding group">improvise-gd83</a> had removed it. A reader meeting the older copy first takes it as current, which is worse than an orphan: an orphan announces itself as broken.</p>
|
||||||
|
<p>The fault is injected by the six cases in <code>SELF_TEST_DUPLICATES</code>, and three of them are the ones that would not get written first. Two <em>different</em> tests sharing a leaf must come back <strong>ambiguous and not duplicate</strong>, because the wrong diagnosis files the wrong repair. A prose heading must be free to repeat, because it claims no test. And the bare and module-qualified spellings of one test must collide — grouping by the string rather than by the resolved test would have missed exactly the case <a class="tid" href="../tickets.html#improvise-rm4s" title="The evidence check accepts headings naming tests that never run">improvise-rm4s</a>'s qualification repair creates, which is the first way this would have come back.</p>
|
||||||
|
<p>The live tree was itself the first injection: before the three merges the check exited 1 naming all three pairs with both line numbers; after them it exits 0. The clean arms (two headings naming two tests, and prose repeated) are what stop an always-refuse from satisfying the rest.</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user