787e354b9b
The `build` job fails on main and on every branch merged from it: doc-site: context/code/pr-review.md is neither published nor a named omission NOT a merge loss -- scripts/build-doc-site.mjs in this branch is byte-identical to main's, and neither lists the file. PR #96 added context/code/pr-review.md without a manifest line, which is exactly the gate improvise-res9 added: a doc added without a line publishes nothing and says nothing, so the generator refuses. Omitted rather than published, and the line is cloud-agents.md's rather than tooling.md's. tooling.md explains checkers a reader of the code will meet; pr-review.md is about THIS REPOSITORY'S PROCESS -- where a Codex review appears, how to tell it from our own agents' comments, why absence is not approval. Nobody visiting a website is about to read a PR thread. Both homes updated, because an omission is restated beside the manifest AND in doc-site.md section 2, and the prose there counted them ("Neither is referenced...") so it went stale at three. This strays into scripts/, which is Wave N's ground. Flagged in the hand-back; it is a one-line data entry either way if the orchestrator would rather publish it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
138 lines
45 KiB
HTML
138 lines
45 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<title>improvise-formula and improvise-io — improvise docs</title>
|
|
<link rel="stylesheet" href="../style.css" />
|
|
</head>
|
|
<body>
|
|
<div class="shell">
|
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html" class="here">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
|
<main>
|
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/formula-and-io.md">context/code/formula-and-io.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#improvise-formula-and-improvise-io">improvise-formula and improvise-io</a></li><li class="l2"><a href="#pipe-quoting-has-exactly-one-home">Pipe quoting has exactly one home</a></li><li class="l3"><a href="#the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it">The keyword guard is canonical, user-confirmed — do not revert it</a></li><li class="l3"><a href="#the-escape-table-and-the-one-input-outside-both-grammars">The escape table, and the one input outside both grammars</a></li><li class="l2"><a href="#cat-is-item-a-coordinate-predicate-in-the-expression-grammar">Cat IS Item: a coordinate predicate in the expression grammar</a></li><li class="l3"><a href="#where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable">WHERE carries the same boundary, and the mechanism has to be generatable</a></li><li class="l2"><a href="#a-formula-target-is-unquoted-exactly-as-a-reference-is">A formula target is unquoted, exactly as a reference is</a></li><li class="l2"><a href="#the-improv-grammar-is-the-single-source-of-truth">The .improv grammar is the single source of truth</a></li><li class="l3"><a href="#the-tree-walkers-are-infallible-by-construction">The tree walkers are infallible by construction</a></li><li class="l3"><a href="#two-passes-because-sections-may-appear-in-any-order">Two passes, because sections may appear in any order</a></li><li class="l3"><a href="#a-section-header-takes-its-name-as-the-whole-line-totally">A section header takes its name as the whole line, totally</a></li><li class="l2"><a href="#storage-is-whole-document-read-write-and-that-closed-the-sqlite-question">Storage is whole-document read/write, and that closed the SQLite question</a></li><li class="l3"><a href="#storage-contract-is-a-public-function-not-a-test-helper">storage_contract is a public function, not a test helper</a></li><li class="l3"><a href="#impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule">impl Storage for Rc<T> lives in improvise-io because of the orphan rule</a></li><li class="l2"><a href="#opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds">OpfsFs: why a browser backend can be synchronous, and what the index holds</a></li><li class="l3"><a href="#the-index-holds-documents-not-entries">The index holds documents, not entries</a></li><li class="l3"><a href="#list-joins-dir-like-the-other-backends">list joins dir like the other backends</a></li><li class="l2"><a href="#loading-decompression-and-decoding-are-two-failures-with-two-answers">Loading: decompression and decoding are two failures with two answers</a></li><li class="l2"><a href="#persistence-write-order-and-the-virtual-views">Persistence write order and the virtual views</a></li><li class="l2"><a href="#import-order-is-a-correctness-property-not-tidiness">Import: order is a correctness property, not tidiness</a></li><li class="l3"><a href="#the-ambiguous-date-heuristic-reads-american-first-deliberately">The ambiguous-date heuristic reads American-first, deliberately</a></li><li class="l3"><a href="#csv-parsing">CSV parsing</a></li><li class="l3"><a href="#the-wizard-s-back-path-is-a-history-stack">The wizard's back path is a history stack</a></li><li class="l2"><a href="#csv-export">CSV export</a></li></ul></details><h1 id="improvise-formula-and-improvise-io"><code>improvise-formula</code> and <code>improvise-io</code></h1>
|
|
<p>The arguments that used to live in the comments of these two crates. Written for a reader with the code open; the code says <em>what</em>, this says <em>why</em>, and why the obvious alternative is wrong. Bug narratives keyed by test name are in <a href="../test-evidence/formula-and-io.html"><code>context/test-evidence/formula-and-io.md</code></a>.</p>
|
|
<hr />
|
|
<h2 id="pipe-quoting-has-exactly-one-home"><a class="anchor" href="#pipe-quoting-has-exactly-one-home">#</a>Pipe quoting has exactly one home</h2>
|
|
<p><code>improvise_formula::quote</code> owns <strong>both halves</strong> of the CL-style <code>|…|</code> convention — writing and reading — for both grammars. <code>formula.pest</code>'s <code>bare_ident</code>/<code>pipe_quoted</code> and <code>improv.pest</code>'s <code>bare_name</code>/<code>pipe_quoted</code> mirror each other character for character, and <code>quote.rs</code> is the one Rust statement of that rule.</p>
|
|
<p>It lives in <code>improvise-formula</code> because that crate owns the grammar that says what a bare name <em>is</em>, and because every emitter needs it: a name that came from a user's data has to be spelled so the parser hands it back unchanged. <code>improvise-gp</code> re-exports it, <code>improvise-io</code> calls it from <code>persistence/{format,parse}.rs</code>, and <code>parser.rs::unquote_pipe</code> is a thin shim over <code>unescape_pipe_body</code>.</p>
|
|
<p>Before <a class="tid" href="../tickets.html#improvise-i2m5" title="improvise-io still carries its own pipe-quoting copy, without the keyword guard">improvise-i2m5</a> there were <strong>three</strong> copies and they had drifted, over the keyword rule below. <strong><code>improvise-io</code> no longer has one</strong> — its <code>persistence/quoting.rs</code> is deleted. <a href="../repo-map.html"><code>context/repo-map.md</code></a>'s file inventory kept claiming io "still has its own unguarded copy" long after the deletion — it was corrected in wave 1, but any <em>other</em> doc or comment still saying so is wrong, and that is the shape this particular fact goes stale in.</p>
|
|
<p>Three entry points, because the callers do not want the same thing:</p>
|
|
<ul>
|
|
<li><code>quote_name</code> — a name as the grammar wants it: bare when <code>is_bare_name</code> allows, else pipe-quoted.</li>
|
|
<li><code>pipe_quote</code> — unconditional. A <code>.improv</code> text cell goes through this, not through <code>quote_name</code>, because a bare <code>42</code> would read back as a number.</li>
|
|
<li><code>unescape_pipe_body</code> — the inverse, over the body between the delimiters (what both grammars' <code>pipe_inner</code> matches). The escape table is written once, in the last two.</li>
|
|
</ul>
|
|
<h3 id="the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it"><a class="anchor" href="#the-keyword-guard-is-canonical-user-confirmed-do-not-revert-it">#</a>The keyword guard is canonical, user-confirmed — do not revert it</h3>
|
|
<p>An item literally named <code>SUM</code> is written <code>|SUM|</code>. <code>is_bare_name</code> is deliberately <strong>stricter than the grammar strictly requires</strong>: bare keywords (<code>WHERE SUM AVG MIN MAX COUNT IF IS</code>) <em>do</em> parse today, but only because <code>agg_call</code> needs a <code>(</code> and PEG backtracking then falls through to <code>ref_expr</code> — an emergent property of <strong>rule ordering</strong>, not a stated rule. A pipe-quoted name is an identifier in every position, with nothing to fall through, so a name colliding with a keyword is quoted and stops depending on that ordering.</p>
|
|
<p><code>IS</code> joined the list with <code>is_expr</code> (<a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>), and it is the one keyword whose omission would have been <em>silently</em> wrong rather than merely fragile: a category literally named <code>IS</code> written bare turns <code>IS IS East</code> into a predicate on a category called <code>East</code>. Adding a production to <code>formula.pest</code> means adding its keyword here — the two are one change.</p>
|
|
<p>The cost is worth paying because a <code>## Formulas</code> line carries formula text <strong>verbatim</strong>: a name <code>format_md</code> writes is later read by <code>parse_formula</code>. One namespace, one spelling rule.</p>
|
|
<h3 id="the-escape-table-and-the-one-input-outside-both-grammars"><a class="anchor" href="#the-escape-table-and-the-one-input-outside-both-grammars">#</a>The escape table, and the one input outside both grammars</h3>
|
|
<p><code>\|</code> → <code>|</code>, <code>\\</code> → <code>\</code>, <code>\n</code> → newline; any other <code>\X</code> is preserved verbatim. The trailing arms of <code>unescape_pipe_body</code> are unreachable from either grammar (<code>pipe_inner</code> is <code>("\\" ~ ANY | !"|" ~ ANY)*</code>, so a backslash is always followed by a character). They are written as a <strong>total function</strong> rather than a panic so a caller arriving by some other route gets its bytes back. The two pre-i2m5 unquoting copies differed <em>only</em> on a trailing lone backslash, where io returned it and <code>parser.rs</code> panicked; the survivor returns it.</p>
|
|
<hr />
|
|
<h2 id="cat-is-item-a-coordinate-predicate-in-the-expression-grammar"><a class="anchor" href="#cat-is-item-a-coordinate-predicate-in-the-expression-grammar">#</a><code>Cat IS Item</code>: a coordinate predicate in the expression grammar</h2>
|
|
<p><code>is_expr = { identifier ~ ^"IS" ~ filter_value }</code> sits in <code>primary</code> <strong>before</strong> <code>ref_expr</code>, which would otherwise consume the category name and leave <code>IS Item</code> unparseable. A bare name still parses as a reference: <code>is_expr</code> needs the keyword, fails without it, and PEG backtracks into <code>ref_expr</code>.</p>
|
|
<p>It reuses <code>where_clause</code>'s two operands — an <code>identifier</code> for the category and a <code>filter_value</code> for the item — and therefore <code>build_filter</code>, which already reads exactly that pair. The <code>^"IS"</code> literal is a string in the grammar rather than a rule, so it contributes no pair and <code>into_inner()</code> yields the two operands and nothing else. One builder, one quoting rule, both halves through <code>improvise_formula::quote</code>.</p>
|
|
<p>The grammar-walking generator needs no new case: it reads <code>formula.pest</code> through <code>pest_meta</code> at test time, so a production added to <code>primary</code> is generated and round-tripped from the day it exists.</p>
|
|
<p>What the <em>evaluator</em> does with it, and why a fitted tree is always wrapped in a <code>SUM</code>, is <a href="../code/core.html"><code>context/code/core.md</code></a> §13.</p>
|
|
<h3 id="where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable"><a class="anchor" href="#where-carries-the-same-boundary-and-the-mechanism-has-to-be-generatable">#</a><code>WHERE</code> carries the same boundary, and the mechanism has to be generatable</h3>
|
|
<p><code>where_clause</code> and <code>inline_where</code> are compound-atomic with <code>sp+</code> after the keyword, exactly as <code>is_expr</code> is (<a class="tid" href="../tickets.html#improvise-r08u" title="IS and WHERE differ on trailing word boundary in formula.pest">improvise-r08u</a>). Non-atomic, pest auto-skipped <code>WHITESPACE</code> after <code>^"WHERE"</code>, so <code>SUM(Rev WHEREAS x = y)</code> read as a <code>WHERE</code> on a category <code>AS</code> — a silently wrong filter where it should have been a syntax error, the same class of defect <code>IS</code> had.</p>
|
|
<p>Two things decided the shape, and the second is the one worth carrying. A negative lookahead <code>!bare_char</code> with <code>sp*</code> would have been <em>strictly</em> more permissive — it keeps <code>WHERE|Region|=East</code>, which the old rule accepted — but <strong>the grammar-walking generator cannot honour a predicate</strong>: it walks <code>formula.pest</code> through <code>pest_meta</code> and emits <code>WHEREa=""</code>, text the grammar then refuses, so <code>generated_formula_parses</code> goes red. A rule whose only guard is a lookahead is a rule the generator can violate. <code>sp+</code> is generatable, matches <code>IS</code>, and costs exactly one accepted spelling: a category may no longer abut the keyword, so <code>WHERE |Region|</code> needs its space.</p>
|
|
<p>Note compound-atomicity <em>alone</em> closes the ticket's own reproducer, because <code>AS x</code> cannot be one identifier once implicit whitespace is gone. <code>sp+</code> is what closes the residue, <code>WHEREAS=y</code>, and the negative test names both.</p>
|
|
<h2 id="a-formula-target-is-unquoted-exactly-as-a-reference-is"><a class="anchor" href="#a-formula-target-is-unquoted-exactly-as-a-reference-is">#</a>A formula target is unquoted, exactly as a reference is</h2>
|
|
<p><code>parse_formula</code> strips the pipes off the target the same way it does off an <code>Expr::Ref</code>, because the target <strong>names a <code>_Measure</code> item</strong>. A target that kept its delimiters made <code>Model::add_formula</code> create an item literally called <code>|Total Revenue|</code> — a ghost column the user's real item never matched, and that <code>Expr::Ref("Total Revenue")</code> never resolved to.</p>
|
|
<p>Both the TUI formula minibuffer (<code>AddFormula</code>) and <code>.improv</code> load go through that one call, so both were affected. The writer emits <code>Formula::raw</code>, never <code>target</code>, so the file text is unaffected either way — <strong>which is exactly why the round-trip tests never saw it</strong>, and why the fix repairs existing files on next load with no migration.</p>
|
|
<hr />
|
|
<h2 id="the-improv-grammar-is-the-single-source-of-truth"><a class="anchor" href="#the-improv-grammar-is-the-single-source-of-truth">#</a>The <code>.improv</code> grammar is the single source of truth</h2>
|
|
<p><code>persistence/improv.pest</code> defines the format; <code>parse.rs</code> is a tree-walker over pest's parse tree, not a line scanner. Adding a format feature means changing the grammar first.</p>
|
|
<p>The test generator (<code>parse.rs::generator</code>, and its twin in <code>parser.rs::generator</code> for <code>formula.pest</code>) reads the grammar's AST at test time via <code>pest_meta</code> and emits random valid input. <strong>A new production therefore needs generator coverage</strong>, or the proptests quietly stop describing the format: an <code>Expr</code> variant the emitter does not handle produces nothing, and <code>generated_file_parses</code> then passes over input that never exercised the new rule. <code>formula.pest</code>'s generator is explicit about this — lookaheads emit nothing on purpose, and any unused <code>Expr</code> variant falls through the <code>_ => {}</code> arm; if the grammar starts using one, generated formulas fail to parse and the gap surfaces.</p>
|
|
<p>Two generator details worth not re-deriving. The <code>atomic</code> flag in the formula generator controls whether a <code>Seq</code> inserts a space between children, so non-atomic rules produce the whitespace-separated tokens pest's implicit <code>WHITESPACE</code> accepts. And <code>Choice</code> is 50/50 between <em>two</em> branches, so rules with three or more alternatives (nested <code>Choice</code>s) pick deeper branches less often — good enough for random generation, not a uniform distribution.</p>
|
|
<h3 id="the-tree-walkers-are-infallible-by-construction"><a class="anchor" href="#the-tree-walkers-are-infallible-by-construction">#</a>The tree walkers are infallible by construction</h3>
|
|
<p>Every <code>build_*</code> in <code>parser.rs</code> operates on a <code>Pair</code> pest has already validated. "A <code>formula</code> has exactly one <code>target</code>", "a <code>comparison</code> has lhs, op and rhs" — pest guarantees these before the walker sees the pair. So every <code>.expect(GRAMMAR_INVARIANT)</code> is a <strong>bug marker</strong>, not a runtime condition: if one fires, the grammar and the walker have diverged. The same reasoning makes the final <code>else</code> arm of each dispatch unlabelled-but-safe (<code>ref_expr</code> is the only remaining alternative of <code>primary</code>, <code>COUNT</code> the only remaining <code>agg_func</code>, <code>>=</code> the only remaining <code>cmp_op</code>).</p>
|
|
<p><code>parse.rs</code>'s walkers are the opposite and deliberately so: they return <code>Result</code> and name the section in the error, because the <code>.improv</code> file is hand-editable and a failure there is a user's problem, not a programmer's. <code>parse_tree</code> renders pest's error as a located message (line number plus the offending line), because "Parse error: {pest dump}" is unusable to somebody fixing a file by hand.</p>
|
|
<h3 id="two-passes-because-sections-may-appear-in-any-order"><a class="anchor" href="#two-passes-because-sections-may-appear-in-any-order">#</a>Two passes, because sections may appear in any order</h3>
|
|
<p>Pass 1 turns each <code>## </code> section into a plain collector (<code>PCategory</code>, <code>PView</code>) or a flat list; pass 2 builds the <code>Workbook</code>. That split is what makes the format order-independent — a <code>## View:</code> may name a category defined further down. <code>PCategory::items</code> is a <code>Vec</code> in file order because <code>IndexMap</code> insertion order is load-bearing (gotcha 18), so pass 2 must replay the list as written.</p>
|
|
<p>Two <code>## Category:</code> sections naming one category are <strong>one</strong> category, and the merge is a concatenation performed in pass 1 (<code>ParsedFile::push_category</code>, <a class="tid" href="../tickets.html#improvise-349m" title="format_md/parse_md is not idempotent when a .improv file has two Category sections with the same name">improvise-349m</a>). Merging in pass 2 — building each section in turn — was not idempotent. A group is created either by the item that names it or by a <code>></code> line, and <code>build_category</code> creates its own section's item-groups <em>before</em> its own <code>></code> lines; so the first section's <code>></code> groups landed ahead of the second section's item-groups, while <code>format_md</code> writes one section with all the items before all the <code>></code> lines. The reparse then built the other order, and the two <code>format_md</code> outputs differed. Concatenating first makes the built order the written order by construction. Refusing a duplicate header was the alternative and would have made a legal hand-written file an error for no gain.</p>
|
|
<h3 id="a-section-header-takes-its-name-as-the-whole-line-totally"><a class="anchor" href="#a-section-header-takes-its-name-as-the-whole-line-totally">#</a>A section header takes its name as the whole line, totally</h3>
|
|
<p><code>header_name = { pipe_quoted ~ &NEWLINE | line_name }</code> (<a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a>). The <code>&NEWLINE</code> lookahead is load-bearing: without it, <code>## Category: |A|B</code> parses as <code>A</code> and <strong>silently drops <code>B</code></strong>. <code>line_name</code> catches everything else, so a header name can no longer fail to parse. Four rules share it — <code>model_name</code>, <code>initial_view</code>, <code>category_section</code>, <code>view_section</code> — which is why it is one rule and not four copies.</p>
|
|
<p>The writing half is <code>format.rs::quote_header</code>, and it is <strong>deliberately not <code>quote_name</code></strong>: that rule is for a name embedded among other tokens, and applying it to a header would pipe-quote every ordinary spaced title the line carries perfectly well, rewriting the header of every existing document for no correctness gain. <code>quote_header</code> quotes exactly four things, each one thing the <em>reader</em> does:</p>
|
|
<ul>
|
|
<li>the empty name, and any name with leading or trailing whitespace — the reader trims, so the category comes back renamed while every data coordinate still names the untrimmed original: silent orphaning;</li>
|
|
<li>a name containing <code>\n</code> or <code>\r</code> — it would start a second line, and <code>Region\n## Data</code> a second <em>section</em>;</li>
|
|
<li>a name already spelled <code>|…|</code> — the reader would take the pipes off.</li>
|
|
</ul>
|
|
<p>So a header is <em>written</em> raw where a reference is quoted. The grammar accepts both arms, so that asymmetry is a <strong>writer convention, not a format rule</strong>.</p>
|
|
<hr />
|
|
<h2 id="storage-is-whole-document-read-write-and-that-closed-the-sqlite-question"><a class="anchor" href="#storage-is-whole-document-read-write-and-that-closed-the-sqlite-question">#</a><code>Storage</code> is whole-document read/write, and that closed the SQLite question</h2>
|
|
<p>Saving or loading a workbook only ever touches one whole document, so a backend needs no seeking, metadata or partial reads (<a class="tid" href="../tickets.html#improvise-6mq" title="Introduce Storage/VFS abstraction in persistence layer">improvise-6mq</a>). It does need to enumerate and remove documents, because a browser build has no OS file manager behind it — hence <code>list</code> and <code>delete</code> beside <code>read</code> and <code>write</code> (<a class="tid" href="../tickets.html#improvise-i34" title="OPFS-backed Storage implementation for wasm target">improvise-i34</a>).</p>
|
|
<p>Paths are <strong>opaque document names, not a directory tree</strong>: a backend may treat them as filesystem paths, OPFS keys or map entries. The trait doc says "backends that are not trees treat <code>dir</code> as a name prefix" — keep that sentence reachable. It is the sentence that turned out to be <em>right while the <code>OpfsFs</code> impl was wrong</em>, and the bug was found by reading it.</p>
|
|
<p>Because the trait is two ops over whole documents, an <code>sqlite3_vfs</code> — which needs <code>xRead(offset,len)</code>, <code>xWrite</code>, <code>xTruncate</code>, <code>xLock</code> — has nothing to bind to. See gotcha 41 for the measurements that declined <a class="tid" href="../tickets.html#improvise-9ix" title="SQLite persistence format (alternative save/restore)">improvise-9ix</a>.</p>
|
|
<h3 id="storage-contract-is-a-public-function-not-a-test-helper"><a class="anchor" href="#storage-contract-is-a-public-function-not-a-test-helper">#</a><code>storage_contract</code> is a public function, not a test helper</h3>
|
|
<p>It reads like a test helper and it <em>was</em> one: <code>#[cfg(test)]</code> in <code>storage.rs</code>, run against <code>MemoryFs</code> and <code>PhysicalFs</code> so they could not drift. The one backend that most needed it could never be handed to it — <code>OpfsFs</code> is <code>cfg(target_arch = "wasm32")</code> and its sync access handles exist in no host harness — so for that backend the contract lived only in a doc comment.</p>
|
|
<p>Moving it out of <code>cfg(test)</code> is what lets a browser run <strong>this</strong> sequence rather than a second transcription of it (<a class="tid" href="../tickets.html#improvise-vvl3" title="A headless browser harness: run storage_contract and the shell assertions under Playwright">improvise-vvl3</a>/xf1). A second copy of a contract is the disease this codebase keeps curing. It returns <code>Result</code> rather than asserting because a failure has to travel out of a Web Worker as a string; the host tests <code>.unwrap()</code> it and get the same message.</p>
|
|
<p><strong>Two bugs fell out of its first actual execution</strong> — see the <code>dir</code> parameter and the index rules below.</p>
|
|
<p><code>dir</code> is a parameter, not a constant, because the three backends must agree at <strong>every</strong> <code>dir</code>, not only at the flat root the browser happens to use (<a class="tid" href="../tickets.html#improvise-vhcz" title="[bug] OpfsFs::list disagrees with the other backends on path shape">improvise-vhcz</a>). A backend whose <code>list</code> answered in a different path shape passes at <code>Path::new("")</code> — where joining is the identity — and fails at anything else.</p>
|
|
<h3 id="impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule"><a class="anchor" href="#impl-storage-for-rc-lives-in-improvise-io-because-of-the-orphan-rule">#</a><code>impl Storage for Rc<T></code> lives in <code>improvise-io</code> because of the orphan rule</h3>
|
|
<p><code>OpfsFs</code> keeps four genuinely async operations — <code>open</code>, <code>attach</code>, <code>refresh</code>, <code>commit_deletes</code> — off the trait, so <code>Storage</code> can stay synchronous (gotcha 28). But <code>App::storage</code> is a <code>Box<dyn Storage></code>, and a trait object can name none of the four: <strong>handing the backend to the <code>App</code> puts the async seam permanently out of reach</strong>. The host must hold the concrete backend <em>and</em> give the <code>App</code> a <code>Storage</code>, which means sharing one.</p>
|
|
<p><code>impl Storage for Rc<OpfsFs></code> is two foreign types in any other crate, so <a class="tid" href="../tickets.html#improvise-9x6" title="Worker-hosted server bundle (wasm, MessageChannel transport)">improvise-9x6</a> wrote a forwarding newtype in the worker and every future worker would have written it again. Here the trait is local, so the impl is allowed, and it is <strong>blanket</strong> rather than <code>OpfsFs</code>-specific because nothing about it is OPFS-shaped: it is the general statement that a shared handle to a backend is usable as that backend (<a class="tid" href="../tickets.html#improvise-hmws" title="Storage cannot expose an async seam through Box<dyn Storage>">improvise-hmws</a>). <code>Rc</code> and not <code>Arc</code> because the one caller is a Web Worker — a single thread, and <code>wasm32-unknown-unknown</code> has no others. An <code>Arc</code> impl would be speculative.</p>
|
|
<hr />
|
|
<h2 id="opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds"><a class="anchor" href="#opfsfs-why-a-browser-backend-can-be-synchronous-and-what-the-index-holds">#</a><code>OpfsFs</code>: why a browser backend can be synchronous, and what the index holds</h2>
|
|
<p>OPFS has two APIs. The promise-based one (<code>getFileHandle</code>, <code>createWritable</code>) is async, but <code>FileSystemFileHandle::createSyncAccessHandle()</code> yields a handle whose <code>read</code>/<code>write</code>/<code>truncate</code>/<code>flush</code>/<code>close</code> are ordinary <strong>blocking</strong> calls. It exists so wasm can do blocking file I/O — it is what SQLite's wasm build uses — and is available <strong>only inside a dedicated Web Worker</strong>, which is where <a class="tid" href="../tickets.html#improvise-9x6" title="Worker-hosted server bundle (wasm, MessageChannel transport)">improvise-9x6</a> puts the whole app. So <code>Storage</code> never had to become async.</p>
|
|
<p><em>Acquiring</em> a handle is still async, as are directory listing and removal, so the backend splits in two:</p>
|
|
<ul>
|
|
<li><strong>async, explicit</strong>: <code>open</code>, <code>attach</code>, <code>refresh</code>, <code>commit_deletes</code>, called by the worker from its message loop where awaiting is free;</li>
|
|
<li><strong>sync, via the trait</strong>: <code>read</code>/<code>write</code> use an already-attached handle; <code>list</code> answers from an index; <code>delete</code> drops the name from that index and queues the removal.</li>
|
|
</ul>
|
|
<p>So a <code>delete</code> is visible to <code>read</code> and <code>list</code> immediately and lands in OPFS at the end of the batch. That ordering is the price of a synchronous trait and is invisible to a single worker. <code>write</code> therefore <strong>un-queues</strong> any pending delete of the same name: otherwise delete-then-write would be listed and readable and then removed by <code>commit_deletes</code> at the end of the batch — the one ordering the deferral can get wrong.</p>
|
|
<h3 id="the-index-holds-documents-not-entries"><a class="anchor" href="#the-index-holds-documents-not-entries">#</a>The index holds documents, not entries</h3>
|
|
<p><code>Storage::list</code> promises <em>document</em> names, and documents and entries stopped being the same thing the moment <code>attach</code> had to create a file to get a handle for a document that does not exist yet (<a class="tid" href="../tickets.html#improvise-vvl3" title="A headless browser harness: run storage_contract and the shell assertions under Playwright">improvise-vvl3</a>). Three writers, and the split between them is the whole rule:</p>
|
|
<ul>
|
|
<li><strong><code>attach</code> indexes nothing.</strong> What it creates is zero bytes. Indexing there made a fresh origin list <code>demo.improv</code> before anything had been written, so the seed was skipped as "already there" and <code>load_from</code> got an empty buffer.</li>
|
|
<li><strong><code>Storage::write</code> indexes the name</strong> (and un-queues a pending delete). A written document is the first thing there is to list.</li>
|
|
<li><strong><code>refresh</code> rebuilds from OPFS, skipping zero-byte entries</strong> — and counting an entry it cannot <em>measure</em> as a document, since the one cause of that is another tab holding the exclusive handle.</li>
|
|
</ul>
|
|
<p><code>holds_bytes</code> is where the last clause lives, and its two failure modes get <strong>opposite answers</strong>, which is the whole function:</p>
|
|
<ul>
|
|
<li><code>getFileHandle</code> fails → the name is not a file. <code>keys()</code> yields directories too, and a directory rejects with <code>TypeMismatchError</code>; an entry removed between <code>keys()</code> and here rejects as well. Neither is a document and neither can ever be attached or read, so listing one is the same picker bug this filter exists to remove (Copilot, PR #24).</li>
|
|
<li><code>getFile</code> fails on a real file handle → another context of the same origin holds the exclusive sync access handle (two tabs). An entry somebody else has open is certainly a document, so this <strong>fails open</strong>. Dropping it would hide a real document; failing the whole <code>refresh</code> would let a second tab break the first one's picker. Over-reporting costs one row — the direction <code>Dirty::of</code> is deliberately sloppy in.</li>
|
|
</ul>
|
|
<p>Handles are exclusive per file, so <code>attach</code> closes any handle it replaces and <code>detach</code> releases the lock.</p>
|
|
<h3 id="list-joins-dir-like-the-other-backends"><a class="anchor" href="#list-joins-dir-like-the-other-backends">#</a><code>list</code> joins <code>dir</code> like the other backends</h3>
|
|
<p>The join is not cosmetic (<a class="tid" href="../tickets.html#improvise-vhcz" title="[bug] OpfsFs::list disagrees with the other backends on path shape">improvise-vhcz</a>). <code>MemoryFs</code> and <code>PhysicalFs</code> answer with <code>dir</code>-joined paths — paths a caller can hand straight back to <code>read</code> — and <code>OpfsFs</code> used to answer with bare names. At <code>Path::new("")</code> the two shapes coincide, which is the only <code>dir</code> the worker ever passes, so the disagreement was invisible: a worker that organised documents under <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/docs"><code>docs/</code></a> would have got a <strong>full picker in every host test and an empty one in a browser</strong>, with nothing saying why.</p>
|
|
<p><code>dir</code> is decoration rather than a filter, because this backend is flat by construction: <code>entry_name</code> reduces every path to its file name, so <code>docs/a.improv</code> and <code>a.improv</code> are the same OPFS entry and an entry name may not contain a separator at all. That is the trait's own "treat <code>dir</code> as a name prefix" — the doc was right and the impl was wrong.</p>
|
|
<hr />
|
|
<h2 id="loading-decompression-and-decoding-are-two-failures-with-two-answers"><a class="anchor" href="#loading-decompression-and-decoding-are-two-failures-with-two-answers">#</a>Loading: decompression and decoding are two failures with two answers</h2>
|
|
<p><code>load_from</code> decompresses, <em>then</em> decodes UTF-8, with a separate <code>Context</code> on each. <code>read_to_string</code> would fuse them, and a valid gzip stream holding invalid UTF-8 would then be reported as unreadable gzip — a wrong diagnosis of a document that decompressed perfectly well. Decoding once, afterwards, also leaves <strong>one</strong> answer to "is this UTF-8" rather than one per branch.</p>
|
|
<p>The <code>.gz</code> suffix selects gzip regardless of backend: that is a property of the document name, not of the filesystem (<a class="tid" href="../tickets.html#improvise-6mq" title="Introduce Storage/VFS abstraction in persistence layer">improvise-6mq</a>). Legacy JSON is detected by a leading <code>{</code> and is never written.</p>
|
|
<p><code>parse_number_cell</code> returns <code>CellValue::Error</code> naming the offending text rather than a fabricated <code>0.0</code>. No token the current grammar admits actually fails <code>f64::from_str</code> (overflow saturates to ±inf), but if the two ever drift the mismatch must <strong>surface</strong> — never silently store a wrong number.</p>
|
|
<hr />
|
|
<h2 id="persistence-write-order-and-the-virtual-views"><a class="anchor" href="#persistence-write-order-and-the-virtual-views">#</a>Persistence write order and the virtual views</h2>
|
|
<p><code>format_md</code> writes Views → Formulas → Categories → Data: views first because they are typically small and orient the reader, data last because it is the largest section. The parser tolerates any order (see the two-pass note above).</p>
|
|
<p><code>_Records</code> and <code>_Drill</code> are scratch views the app rebuilds on demand. They put <code>_Index</code>/<code>_Dim</code> on axes, and those categories are never written — so persisting the views would produce axis lines naming categories the loader will not find. They are skipped on write, and <strong>dropped on load</strong> if an older file carries them, rather than reinstating a view the app would rebuild anyway; <code>Initial View</code> then falls through to the default, since the name it pointed at is gone. <code>persisted_initial_view</code> is the writing half: saving from records or drill mode falls back to the first real view rather than naming one the file will not contain (<a class="tid" href="../tickets.html#improvise-60z" title="Virtual views _Records and _Drill should not be persisted to .improv files">improvise-60z</a>).</p>
|
|
<p><code>_Measure</code> persists only its non-formula items — formula targets are rebuilt from the <code>## Formulas</code> section, so <code>format_md</code> collects the targets and excludes them.</p>
|
|
<p><code>Page</code> and <code>Filter</code> both carry the selected item after the axis keyword; the other axes have nothing to select. <code>Axis::Filter</code> round-trips even though it occurs only inside the never-written <code>_Drill</code> view, because the alternative was silently writing a Filter axis as some other axis (<a class="tid" href="../tickets.html#improvise-dz8" title="Add Axis::Filter for drill-fixed coordinates">improvise-dz8</a>, gotcha 27).</p>
|
|
<hr />
|
|
<h2 id="import-order-is-a-correctness-property-not-tidiness"><a class="anchor" href="#import-order-is-a-correctness-property-not-tidiness">#</a>Import: order is a correctness property, not tidiness</h2>
|
|
<p><code>analyzer::analyze_records</code> emits distinct values in <strong>first-seen order</strong>, never <code>HashSet</code> order (<a class="tid" href="../tickets.html#improvise-7yx" title="Close test gaps flagged by deep review">improvise-7yx</a>, gotcha 18). <code>build_model</code> adds items in exactly that order and <code>IndexMap</code> preserves it, so a hash order made the <em>same CSV</em> import to a different document on every run — different <code>.improv</code> bytes, different row order on screen, and a different detected date format, since <code>detect_date_format</code> only inspects the first ten samples and <em>which</em> ten it saw varied. The <code>HashSet</code> in <code>analyze_records</code> is there to deduplicate and <strong>never to iterate</strong>.</p>
|
|
<h3 id="the-ambiguous-date-heuristic-reads-american-first-deliberately"><a class="anchor" href="#the-ambiguous-date-heuristic-reads-american-first-deliberately">#</a>The ambiguous-date heuristic reads American-first, deliberately</h3>
|
|
<p><code>01/02/2025</code> parses under both <code>%m/%d/%Y</code> and <code>%d/%m/%Y</code> and nothing in the data can tell them apart. <code>detect_date_format</code> is first-match over <code>DATE_FORMATS</code>, so the American reading wins. That is <em>a</em> defensible answer and not obviously the right one: a European CSV imports with days and months swapped, <strong>silently</strong>, and the derived <code>_Month</code>/<code>_Quarter</code> categories inherit the error. What makes it tolerable is that the wizard shows the detected format and lets the user change it — the same argument two-digit years already rested on. It is pinned by a test so that changing the answer is a decision somebody makes rather than a side effect of reordering the table.</p>
|
|
<p><strong>Ruled a known limit</strong> (<a class="tid" href="../tickets.html#improvise-l8rh" title="The ambiguous-date heuristic reads American-first, so a European CSV imports with days and months swapped">improvise-l8rh</a>, the user, 2026-09-20). Two things the ruling rests on. Changing the default to day-first only moves which half of the world imports silently wrong; and the honest fix is larger than the heuristic — an ambiguous column should <em>surface</em> its ambiguity rather than resolve it, which changes <code>FieldProposal</code>'s shape and the wizard step that displays it, the way <code>Dataset::warnings</code> returns a decision the user should hear about as data. Worth doing deliberately, not as a patch.</p>
|
|
<p>What makes the limit reasonable to live with is that it is now <em>deterministic</em>: <code>analyze_records</code> iterated a <code>HashSet</code> until <a class="tid" href="../tickets.html#improvise-7yx" title="Close test gaps flagged by deep review">improvise-7yx</a>, so the ten samples <code>detect_date_format</code> inspects were a different ten every run and a column whose eleventh value ruled out <code>%m/%d/%Y</code> detected a different format run to run. The ten-sample window itself is unchanged and is part of the limit.</p>
|
|
<h3 id="csv-parsing"><a class="anchor" href="#csv-parsing">#</a>CSV parsing</h3>
|
|
<p>Short rows keep every header column, missing trailing fields filled with <code>Value::Null</code>, and each affected column is reported as a warning with a row count. The reader is <code>.flexible(true)</code>, so without this a short row parsed cleanly and the trailing columns were simply <em>absent</em> from the record map — no error, no warning, and skewed value counts in field-kind analysis (<a class="tid" href="../tickets.html#improvise-k8i" title="CSV import silently drops columns when row widths mismatch">improvise-k8i</a>).</p>
|
|
<p>A header name becomes a <strong>category name</strong> and survives into the <code>.improv</code> file, so anything riding along invisibly in one — a byte-order mark, a stray carriage return — is silent corruption rather than cosmetics. That is why the adversarial inputs are tested with <em>controls</em>: a check that cannot see a BOM where there is one proves nothing.</p>
|
|
<p><code>parse_csv_bytes_with_warnings</code> exists so CSV import can come from any <code>Storage</code> backend rather than only a file (<a class="tid" href="../tickets.html#improvise-fuj" title="Two effects read files directly instead of going through Storage">improvise-fuj</a>); the path-taking wrappers remain for the native CLI.</p>
|
|
<h3 id="the-wizard-s-back-path-is-a-history-stack"><a class="anchor" href="#the-wizard-s-back-path-is-a-history-stack">#</a>The wizard's back path is a history stack</h3>
|
|
<p><code>advance</code> pushes the step it leaves; <code>previous</code> pops it. Keeping the path actually taken, rather than re-deriving the reverse of each conditional transition, means back can never disagree with how the user really got here (<a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a>).</p>
|
|
<p>Preconditions live in <code>validation_error()</code> and are checked <strong>on the step where the user can still fix them</strong> — never deferred to <code>build_model</code> after the final confirm. Two gates earn that: too many categories (<a class="tid" href="../tickets.html#improvise-mzv" title="Import wizard does not pre-check 12-category limit">improvise-mzv</a>) and no category at all (<a class="tid" href="../tickets.html#improvise-gfq" title="Import wizard defers the 'no category accepted' failure to the very last keystroke">improvise-gfq</a>); both are fixable with <code>c</code> at <code>ReviewProposals</code>, and both used to refuse only at the last keystroke. <code>MAX_CATEGORIES</code> here mirrors the model's private limit and counts regular categories only.</p>
|
|
<p><code>NameModel</code> is terminal for <code>advance</code>: leaving it is a <strong>build</strong>, not a step change. <code>build()</code> parks the workbook in <code>built</code> on <code>Done</code>; a build failure leaves the user on <code>NameModel</code> with the error. Going back from <code>Done</code> discards <code>built</code>, so accepting an import always reflects the latest field decisions.</p>
|
|
<p><code>scalar_name</code> decides what a JSON scalar contributes as an item name. <code>Value::Null</code> — which is also what the CSV parser produces for a blank field — and a blank string are <strong>absences, not names</strong>; stringifying them used to invent an item literally called "null" (<a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item "null" on import">improvise-ych</a>). Every other value keeps its printed form.</p>
|
|
<p>A record missing a value for an accepted Category field has no place in the grid and is dropped; <code>build_model_with_warnings</code> counts it against the <em>first</em> category it is missing and reports the total, because without a warning that loss is invisible in a thousand-row import (<a class="tid" href="../tickets.html#improvise-5j9" title="ImportPipeline silently skips records missing category values">improvise-5j9</a>).</p>
|
|
<hr />
|
|
<h2 id="csv-export"><a class="anchor" href="#csv-export">#</a>CSV export</h2>
|
|
<p><code>render_csv</code> is separated from writing so the bytes can go to any <code>Storage</code> backend, not just a file (<a class="tid" href="../tickets.html#improvise-fuj" title="Two effects read files directly instead of going through Storage">improvise-fuj</a>). It goes through the <code>csv</code> crate rather than joining with commas, and it honours the view's number format — a value of 1200 under a comma-grouping format renders <code>1,200</code>, and a label like <code>Smith, Jr.</code> splits into two fields, corrupting every row (<a class="tid" href="../tickets.html#improvise-1cz" title="export_csv emits unquoted CSV — commas in values corrupt output">improvise-1cz</a>).</p>
|
|
</main>
|
|
</div>
|
|
</body>
|
|
</html>
|