improvise/context/code/formula-and-io.md

improvise-formula and improvise-io

The arguments that used to live in the comments of these two crates. Written for a reader with the code open; the code says what, this says why, and why the obvious alternative is wrong. Bug narratives keyed by test name are in context/test-evidence/formula-and-io.md.

---

Pipe quoting has exactly one home

improvise_formula::quote owns both halves of the CL-style |…| convention — writing and reading — for both grammars. formula.pest's bare_ident/pipe_quoted and improv.pest's bare_name/pipe_quoted mirror each other character for character, and quote.rs is the one Rust statement of that rule.

It lives in improvise-formula because that crate owns the grammar that says what a bare name is, and because every emitter needs it: a name that came from a user's data has to be spelled so the parser hands it back unchanged. improvise-gp re-exports it, improvise-io calls it from persistence/{format,parse}.rs, and parser.rs::unquote_pipe is a thin shim over unescape_pipe_body.

Before improvise-i2m5 there were three copies and they had drifted, over the keyword rule below. improvise-io no longer has one — its persistence/quoting.rs is deleted. A stale line in context/repo-map.md claimed io "still has its own unguarded copy" for months after the deletion; if you find a comment or doc saying that, it is wrong.

Three entry points, because the callers do not want the same thing:

• quote_name — a name as the grammar wants it: bare when is_bare_name allows, else pipe-quoted.
• pipe_quote — unconditional. A .improv text cell goes through this, not through quote_name, because a bare 42 would read back as a number.
• unescape_pipe_body — the inverse, over the body between the delimiters (what both grammars' pipe_inner matches). The escape table is written once, in the last two.

The keyword guard is canonical, user-confirmed — do not revert it

An item literally named SUM is written |SUM|. is_bare_name is deliberately stricter than the grammar strictly requires: bare keywords (WHERE SUM AVG MIN MAX COUNT IF) do parse today, but only because agg_call needs a ( and PEG backtracking then falls through to ref_expr — an emergent property of rule ordering, not a stated rule. A pipe-quoted name is an identifier in every position, with nothing to fall through, so a name colliding with a keyword is quoted and stops depending on that ordering.

The cost is worth paying because a ## Formulas line carries formula text verbatim: a name format_md writes is later read by parse_formula. One namespace, one spelling rule.

The escape table, and the one input outside both grammars

\| → |, \\ → \, \n → newline; any other \X is preserved verbatim. The trailing arms of unescape_pipe_body are unreachable from either grammar (pipe_inner is ("\\" ~ ANY | !"|" ~ ANY)*, so a backslash is always followed by a character). They are written as a total function rather than a panic so a caller arriving by some other route gets its bytes back. The two pre-i2m5 unquoting copies differed only on a trailing lone backslash, where io returned it and parser.rs panicked; the survivor returns it.

---

A formula target is unquoted, exactly as a reference is

parse_formula strips the pipes off the target the same way it does off an Expr::Ref, because the target names a _Measure item. A target that kept its delimiters made Model::add_formula create an item literally called |Total Revenue| — a ghost column the user's real item never matched, and that Expr::Ref("Total Revenue") never resolved to.

Both the TUI formula minibuffer (AddFormula) and .improv load go through that one call, so both were affected. The writer emits Formula::raw, never target, so the file text is unaffected either way — which is exactly why the round-trip tests never saw it, and why the fix repairs existing files on next load with no migration.

---

The .improv grammar is the single source of truth

persistence/improv.pest defines the format; parse.rs is a tree-walker over pest's parse tree, not a line scanner. Adding a format feature means changing the grammar first.

The test generator (parse.rs::generator, and its twin in parser.rs::generator for formula.pest) reads the grammar's AST at test time via pest_meta and emits random valid input. A new production therefore needs generator coverage, or the proptests quietly stop describing the format: an Expr variant the emitter does not handle produces nothing, and generated_file_parses then passes over input that never exercised the new rule. formula.pest's generator is explicit about this — lookaheads emit nothing on purpose, and any unused Expr variant falls through the _ => {} arm; if the grammar starts using one, generated formulas fail to parse and the gap surfaces.

Two generator details worth not re-deriving. The atomic flag in the formula generator controls whether a Seq inserts a space between children, so non-atomic rules produce the whitespace-separated tokens pest's implicit WHITESPACE accepts. And Choice is 50/50 between two branches, so rules with three or more alternatives (nested Choices) pick deeper branches less often — good enough for random generation, not a uniform distribution.

The tree walkers are infallible by construction

Every build_* in parser.rs operates on a Pair pest has already validated. "A formula has exactly one target", "a comparison has lhs, op and rhs" — pest guarantees these before the walker sees the pair. So every .expect(GRAMMAR_INVARIANT) is a bug marker, not a runtime condition: if one fires, the grammar and the walker have diverged. The same reasoning makes the final else arm of each dispatch unlabelled-but-safe (ref_expr is the only remaining alternative of primary, COUNT the only remaining agg_func, >= the only remaining cmp_op).

parse.rs's walkers are the opposite and deliberately so: they return Result and name the section in the error, because the .improv file is hand-editable and a failure there is a user's problem, not a programmer's. parse_tree renders pest's error as a located message (line number plus the offending line), because "Parse error: {pest dump}" is unusable to somebody fixing a file by hand.

Two passes, because sections may appear in any order

Pass 1 turns each ## section into a plain collector (PCategory, PView) or a flat list; pass 2 builds the Workbook. That split is what makes the format order-independent — a ## View: may name a category defined further down. PCategory::items is a Vec in file order because IndexMap insertion order is load-bearing (gotcha 18), so pass 2 must replay the list as written.

A section header takes its name as the whole line, totally

header_name = { pipe_quoted ~ &NEWLINE | line_name } (improvise-pczv). The &NEWLINE lookahead is load-bearing: without it, ## Category: |A|B parses as A and silently drops B. line_name catches everything else, so a header name can no longer fail to parse. Four rules share it — model_name, initial_view, category_section, view_section — which is why it is one rule and not four copies.

The writing half is format.rs::quote_header, and it is deliberately not quote_name: that rule is for a name embedded among other tokens, and applying it to a header would pipe-quote every ordinary spaced title the line carries perfectly well, rewriting the header of every existing document for no correctness gain. quote_header quotes exactly four things, each one thing the reader does:

• the empty name, and any name with leading or trailing whitespace — the reader trims, so the category comes back renamed while every data coordinate still names the untrimmed original: silent orphaning;
• a name containing \n or \r — it would start a second line, and Region\n## Data a second section;
• a name already spelled |…| — the reader would take the pipes off.

So a header is written raw where a reference is quoted. The grammar accepts both arms, so that asymmetry is a writer convention, not a format rule.

---

Storage is whole-document read/write, and that closed the SQLite question

Saving or loading a workbook only ever touches one whole document, so a backend needs no seeking, metadata or partial reads (improvise-6mq). It does need to enumerate and remove documents, because a browser build has no OS file manager behind it — hence list and delete beside read and write (improvise-i34).

Paths are opaque document names, not a directory tree: a backend may treat them as filesystem paths, OPFS keys or map entries. The trait doc says "backends that are not trees treat dir as a name prefix" — keep that sentence reachable. It is the sentence that turned out to be right while the OpfsFs impl was wrong, and the bug was found by reading it.

Because the trait is two ops over whole documents, an sqlite3_vfs — which needs xRead(offset,len), xWrite, xTruncate, xLock — has nothing to bind to. See gotcha 41 for the measurements that declined improvise-9ix.

storage_contract is a public function, not a test helper

It reads like a test helper and it was one: #[cfg(test)] in storage.rs, run against MemoryFs and PhysicalFs so they could not drift. The one backend that most needed it could never be handed to it — OpfsFs is cfg(target_arch = "wasm32") and its sync access handles exist in no host harness — so for that backend the contract lived only in a doc comment.

Moving it out of cfg(test) is what lets a browser run this sequence rather than a second transcription of it (improvise-vvl3/xf1). A second copy of a contract is the disease this codebase keeps curing. It returns Result rather than asserting because a failure has to travel out of a Web Worker as a string; the host tests .unwrap() it and get the same message.

Two bugs fell out of its first actual execution — see the dir parameter and the index rules below.

dir is a parameter, not a constant, because the three backends must agree at every dir, not only at the flat root the browser happens to use (improvise-vhcz). A backend whose list answered in a different path shape passes at Path::new("") — where joining is the identity — and fails at anything else.

impl Storage for Rc<T> lives in improvise-io because of the orphan rule

OpfsFs keeps four genuinely async operations — open, attach, refresh, commit_deletes — off the trait, so Storage can stay synchronous (gotcha 28). But App::storage is a Box<dyn Storage>, and a trait object can name none of the four: handing the backend to the App puts the async seam permanently out of reach. The host must hold the concrete backend and give the App a Storage, which means sharing one.

impl Storage for Rc<OpfsFs> is two foreign types in any other crate, so improvise-9x6 wrote a forwarding newtype in the worker and every future worker would have written it again. Here the trait is local, so the impl is allowed, and it is blanket rather than OpfsFs-specific because nothing about it is OPFS-shaped: it is the general statement that a shared handle to a backend is usable as that backend (improvise-hmws). Rc and not Arc because the one caller is a Web Worker — a single thread, and wasm32-unknown-unknown has no others. An Arc impl would be speculative.

---

OpfsFs: why a browser backend can be synchronous, and what the index holds

OPFS has two APIs. The promise-based one (getFileHandle, createWritable) is async, but FileSystemFileHandle::createSyncAccessHandle() yields a handle whose read/write/truncate/flush/close are ordinary blocking calls. It exists so wasm can do blocking file I/O — it is what SQLite's wasm build uses — and is available only inside a dedicated Web Worker, which is where improvise-9x6 puts the whole app. So Storage never had to become async.

Acquiring a handle is still async, as are directory listing and removal, so the backend splits in two:

• async, explicit: open, attach, refresh, commit_deletes, called by the worker from its message loop where awaiting is free;
• sync, via the trait: read/write use an already-attached handle; list answers from an index; delete drops the name from that index and queues the removal.

So a delete is visible to read and list immediately and lands in OPFS at the end of the batch. That ordering is the price of a synchronous trait and is invisible to a single worker. write therefore un-queues any pending delete of the same name: otherwise delete-then-write would be listed and readable and then removed by commit_deletes at the end of the batch — the one ordering the deferral can get wrong.

The index holds documents, not entries

Storage::list promises document names, and documents and entries stopped being the same thing the moment attach had to create a file to get a handle for a document that does not exist yet (improvise-vvl3). Three writers, and the split between them is the whole rule:

• attach indexes nothing. What it creates is zero bytes. Indexing there made a fresh origin list demo.improv before anything had been written, so the seed was skipped as "already there" and load_from got an empty buffer.
• Storage::write indexes the name (and un-queues a pending delete). A written document is the first thing there is to list.
• refresh rebuilds from OPFS, skipping zero-byte entries — and counting an entry it cannot measure as a document, since the one cause of that is another tab holding the exclusive handle.

holds_bytes is where the last clause lives, and its two failure modes get opposite answers, which is the whole function:

• getFileHandle fails → the name is not a file. keys() yields directories too, and a directory rejects with TypeMismatchError; an entry removed between keys() and here rejects as well. Neither is a document and neither can ever be attached or read, so listing one is the same picker bug this filter exists to remove (Copilot, PR #24).
• getFile fails on a real file handle → another context of the same origin holds the exclusive sync access handle (two tabs). An entry somebody else has open is certainly a document, so this fails open. Dropping it would hide a real document; failing the whole refresh would let a second tab break the first one's picker. Over-reporting costs one row — the direction Dirty::of is deliberately sloppy in.

Handles are exclusive per file, so attach closes any handle it replaces and detach releases the lock.

list joins dir like the other backends

The join is not cosmetic (improvise-vhcz). MemoryFs and PhysicalFs answer with dir-joined paths — paths a caller can hand straight back to read — and OpfsFs used to answer with bare names. At Path::new("") the two shapes coincide, which is the only dir the worker ever passes, so the disagreement was invisible: a worker that organised documents under docs/ would have got a full picker in every host test and an empty one in a browser, with nothing saying why.

dir is decoration rather than a filter, because this backend is flat by construction: entry_name reduces every path to its file name, so docs/a.improv and a.improv are the same OPFS entry and an entry name may not contain a separator at all. That is the trait's own "treat dir as a name prefix" — the doc was right and the impl was wrong.

---

Loading: decompression and decoding are two failures with two answers

load_from decompresses, then decodes UTF-8, with a separate Context on each. read_to_string would fuse them, and a valid gzip stream holding invalid UTF-8 would then be reported as unreadable gzip — a wrong diagnosis of a document that decompressed perfectly well. Decoding once, afterwards, also leaves one answer to "is this UTF-8" rather than one per branch.

The .gz suffix selects gzip regardless of backend: that is a property of the document name, not of the filesystem (improvise-6mq). Legacy JSON is detected by a leading { and is never written.

parse_number_cell returns CellValue::Error naming the offending text rather than a fabricated 0.0. No token the current grammar admits actually fails f64::from_str (overflow saturates to ±inf), but if the two ever drift the mismatch must surface — never silently store a wrong number.

---

Persistence write order and the virtual views

format_md writes Views → Formulas → Categories → Data: views first because they are typically small and orient the reader, data last because it is the largest section. The parser tolerates any order (see the two-pass note above).

_Records and _Drill are scratch views the app rebuilds on demand. They put _Index/_Dim on axes, and those categories are never written — so persisting the views would produce axis lines naming categories the loader will not find. They are skipped on write, and dropped on load if an older file carries them, rather than reinstating a view the app would rebuild anyway; Initial View then falls through to the default, since the name it pointed at is gone. persisted_initial_view is the writing half: saving from records or drill mode falls back to the first real view rather than naming one the file will not contain (improvise-60z).

_Measure persists only its non-formula items — formula targets are rebuilt from the ## Formulas section, so format_md collects the targets and excludes them.

Page and Filter both carry the selected item after the axis keyword; the other axes have nothing to select. Axis::Filter round-trips even though it occurs only inside the never-written _Drill view, because the alternative was silently writing a Filter axis as some other axis (improvise-dz8, gotcha 27).

---

Import: order is a correctness property, not tidiness

analyzer::analyze_records emits distinct values in first-seen order, never HashSet order (improvise-7yx, gotcha 18). build_model adds items in exactly that order and IndexMap preserves it, so a hash order made the same CSV import to a different document on every run — different .improv bytes, different row order on screen, and a different detected date format, since detect_date_format only inspects the first ten samples and which ten it saw varied. The HashSet in analyze_records is there to deduplicate and never to iterate.

The ambiguous-date heuristic reads American-first, deliberately

01/02/2025 parses under both %m/%d/%Y and %d/%m/%Y and nothing in the data can tell them apart. detect_date_format is first-match over DATE_FORMATS, so the American reading wins. That is a defensible answer and not obviously the right one: a European CSV imports with days and months swapped, silently, and the derived _Month/_Quarter categories inherit the error. What makes it tolerable is that the wizard shows the detected format and lets the user change it — the same argument two-digit years already rested on. It is pinned by a test so that changing the answer is a decision somebody makes rather than a side effect of reordering the table. Filed as improvise-l8rh.

CSV parsing

Short rows keep every header column, missing trailing fields filled with Value::Null, and each affected column is reported as a warning with a row count. The reader is .flexible(true), so without this a short row parsed cleanly and the trailing columns were simply absent from the record map — no error, no warning, and skewed value counts in field-kind analysis (improvise-k8i).

A header name becomes a category name and survives into the .improv file, so anything riding along invisibly in one — a byte-order mark, a stray carriage return — is silent corruption rather than cosmetics. That is why the adversarial inputs are tested with controls: a check that cannot see a BOM where there is one proves nothing.

parse_csv_bytes_with_warnings exists so CSV import can come from any Storage backend rather than only a file (improvise-fuj); the path-taking wrappers remain for the native CLI.

The wizard's back path is a history stack

advance pushes the step it leaves; previous pops it. Keeping the path actually taken, rather than re-deriving the reverse of each conditional transition, means back can never disagree with how the user really got here (improvise-907).

Preconditions live in validation_error() and are checked on the step where the user can still fix them — never deferred to build_model after the final confirm. Two gates earn that: too many categories (improvise-mzv) and no category at all (improvise-gfq); both are fixable with c at ReviewProposals, and both used to refuse only at the last keystroke. MAX_CATEGORIES here mirrors the model's private limit and counts regular categories only.

NameModel is terminal for advance: leaving it is a build, not a step change. build() parks the workbook in built on Done; a build failure leaves the user on NameModel with the error. Going back from Done discards built, so accepting an import always reflects the latest field decisions.

scalar_name decides what a JSON scalar contributes as an item name. Value::Null — which is also what the CSV parser produces for a blank field — and a blank string are absences, not names; stringifying them used to invent an item literally called "null" (improvise-ych). Every other value keeps its printed form.

A record missing a value for an accepted Category field has no place in the grid and is dropped; build_model_with_warnings counts it against the first category it is missing and reports the total, because without a warning that loss is invisible in a thousand-row import (improvise-5j9).

---

CSV export

render_csv is separated from writing so the bytes can go to any Storage backend, not just a file (improvise-fuj). It goes through the csv crate rather than joining with commas, and it honours the view's number format — a value of 1200 under a comma-grouping format renders 1,200, and a label like Smith, Jr. splits into two fields, corrupting every row (improvise-1cz).