improvise/context/code/formula-and-io.md

improvise-formula and improvise-io

The arguments that used to live in the comments of these two crates. Written for a reader with the code open; the code says what, this says why, and why the obvious alternative is wrong. Bug narratives keyed by test name are in context/test-evidence/formula-and-io.md.

---

Pipe quoting has exactly one home

improvise_formula::quote owns both halves of the CL-style |…| convention — writing and reading — for both grammars. formula.pest's bare_ident/pipe_quoted and improv.pest's bare_name/pipe_quoted mirror each other character for character, and quote.rs is the one Rust statement of that rule.

It lives in improvise-formula because that crate owns the grammar that says what a bare name is, and because every emitter needs it: a name that came from a user's data has to be spelled so the parser hands it back unchanged. improvise-gp re-exports it, improvise-io calls it from persistence/{format,parse}.rs, and parser.rs::unquote_pipe is a thin shim over unescape_pipe_body.

Before improvise-i2m5 there were three copies and they had drifted, over the keyword rule below. improvise-io no longer has one — its persistence/quoting.rs is deleted. context/file-inventory.md kept claiming io "still has its own unguarded copy" long after the deletion — it was corrected in wave 1, but any other doc or comment still saying so is wrong, and that is the shape this particular fact goes stale in.

Three entry points, because the callers do not want the same thing:

• quote_name — a name as the grammar wants it: bare when is_bare_name allows, else pipe-quoted.
• pipe_quote — unconditional. A .improv text cell goes through this, not through quote_name, because a bare 42 would read back as a number.
• unescape_pipe_body — the inverse, over the body between the delimiters (what both grammars' pipe_inner matches). The escape table is written once, in the last two.

The keyword guard is canonical, user-confirmed — do not revert it

An item literally named SUM is written |SUM|. is_bare_name is deliberately stricter than the grammar strictly requires: bare keywords (WHERE SUM AVG MIN MAX COUNT IF IS) do parse today, but only because agg_call needs a ( and PEG backtracking then falls through to ref_expr — an emergent property of rule ordering, not a stated rule. A pipe-quoted name is an identifier in every position, with nothing to fall through, so a name colliding with a keyword is quoted and stops depending on that ordering.

IS joined the list with is_expr (improvise-4zl7), and it is the one keyword whose omission would have been silently wrong rather than merely fragile: a category literally named IS written bare turns IS IS East into a predicate on a category called East. Adding a production to formula.pest means adding its keyword here — the two are one change.

The cost is worth paying because a ## Formulas line carries formula text verbatim: a name format_md writes is later read by parse_formula. One namespace, one spelling rule.

The escape table, and the one input outside both grammars

\| → |, \\ → \, \n → newline; any other \X is preserved verbatim. The trailing arms of unescape_pipe_body are unreachable from either grammar (pipe_inner is ("\\" ~ ANY | !"|" ~ ANY)*, so a backslash is always followed by a character). They are written as a total function rather than a panic so a caller arriving by some other route gets its bytes back. The two pre-i2m5 unquoting copies differed only on a trailing lone backslash, where io returned it and parser.rs panicked; the survivor returns it.

---

Cat IS Item: a coordinate predicate in the expression grammar

is_expr = { identifier ~ ^"IS" ~ filter_value } sits in primary before ref_expr, which would otherwise consume the category name and leave IS Item unparseable. A bare name still parses as a reference: is_expr needs the keyword, fails without it, and PEG backtracks into ref_expr.

It reuses where_clause's two operands — an identifier for the category and a filter_value for the item — and therefore build_filter, which already reads exactly that pair. The ^"IS" literal is a string in the grammar rather than a rule, so it contributes no pair and into_inner() yields the two operands and nothing else. One builder, one quoting rule, both halves through improvise_formula::quote.

The grammar-walking generator needs no new case: it reads formula.pest through pest_meta at test time, so a production added to primary is generated and round-tripped from the day it exists.

What the evaluator does with it, and why a fitted tree is always wrapped in a SUM, is context/code/core.md §13.

WHERE carries the same boundary, and the mechanism has to be generatable

where_clause and inline_where are compound-atomic with sp+ after the keyword, exactly as is_expr is (improvise-r08u). Non-atomic, pest auto-skipped WHITESPACE after ^"WHERE", so SUM(Rev WHEREAS x = y) read as a WHERE on a category AS — a silently wrong filter where it should have been a syntax error, the same class of defect IS had.

Two things decided the shape, and the second is the one worth carrying. A negative lookahead !bare_char with sp* would have been strictly more permissive — it keeps WHERE|Region|=East, which the old rule accepted — but the grammar-walking generator cannot honour a predicate: it walks formula.pest through pest_meta and emits WHEREa="", text the grammar then refuses, so generated_formula_parses goes red. A rule whose only guard is a lookahead is a rule the generator can violate. sp+ is generatable, matches IS, and costs exactly one accepted spelling: a category may no longer abut the keyword, so WHERE |Region| needs its space.

Note compound-atomicity alone closes the ticket's own reproducer, because AS x cannot be one identifier once implicit whitespace is gone. sp+ is what closes the residue, WHEREAS=y, and the negative test names both.

A formula target is unquoted, exactly as a reference is

parse_formula strips the pipes off the target the same way it does off an Expr::Ref, because the target names a _Measure item. A target that kept its delimiters made Model::add_formula create an item literally called |Total Revenue| — a ghost column the user's real item never matched, and that Expr::Ref("Total Revenue") never resolved to.

Both the TUI formula minibuffer (AddFormula) and .improv load go through that one call, so both were affected. The writer emits Formula::raw, never target, so the file text is unaffected either way — which is exactly why the round-trip tests never saw it, and why the fix repairs existing files on next load with no migration.

---

The .improv grammar is the single source of truth

persistence/improv.pest defines the format; parse.rs is a tree-walker over pest's parse tree, not a line scanner. Adding a format feature means changing the grammar first.

The test generator (parse.rs::generator, and its twin in parser.rs::generator for formula.pest) reads the grammar's AST at test time via pest_meta and emits random valid input. A new production therefore needs generator coverage, or the proptests quietly stop describing the format: an Expr variant the emitter does not handle produces nothing, and generated_file_parses then passes over input that never exercised the new rule. formula.pest's generator is explicit about this — lookaheads emit nothing on purpose, and any unused Expr variant falls through the _ => {} arm; if the grammar starts using one, generated formulas fail to parse and the gap surfaces.

Two generator details worth not re-deriving. The atomic flag in the formula generator controls whether a Seq inserts a space between children, so non-atomic rules produce the whitespace-separated tokens pest's implicit WHITESPACE accepts. And Choice is 50/50 between two branches, so rules with three or more alternatives (nested Choices) pick deeper branches less often — good enough for random generation, not a uniform distribution.

The category kind is a line, not a header suffix (improvise-dy3j)

CategoryKind::Label had no spelling in the format at all. format_md wrote every category as a bare ## Category: Name and build_category read every section back through add_category, so the kind was dropped on the way out and on the way in. A round-trip equality test cannot see a loss both sides share, which is why the existing suite was silent — the same shape as the Formula::raw traps above, and the reason the new tests assert the parsed Category::kind rather than the emitted text.

The consequence was a lost document, not a lost flag. Label sits outside MAX_CATEGORIES on purpose (12 then, 24 since improvise-felo.1), and the import wizard's own precondition scores a label as zero, so it will happily build twelve categories plus a label. On load every label came back Regular, the thirteenth was refused, and parse_md failed: any document where regular + label exceeded twelve was permanently unloadable. Below the limit the failure was quieter — a label silently became a pivot category, entering regular_category_names() pickers and gaining axis auto-assignment.

The ticket proposed ## Category: Note [label], and the grammar forbids it. A section header takes its name as the whole line, totally (improvise-pczv) — header_name is pipe_quoted ~ &NEWLINE | line_name — so a suffix is not a suffix, it is part of a category literally named Note [label]. The two readings are indistinguishable, and adopting the suffix would have made a legal name unspellable while quietly re-opening the ambiguity header_name exists to close.

So the kind is an entry inside the section, beside the item lines:

## Category: Note
kind: label
- see the record view

Three properties earned it. It uses a real token ("kind: "), not a lookahead, because the grammar-walking generator reads the grammar through pest_meta and skips predicates — a rule guarded only by a lookahead is a rule the generator can violate, which is why WHERE carries sp+. It cannot collide with item_list, grouped_item or group_hierarchy, each of which needs a - or > lead. And absence is the compatible reading: kind_kw spells only label, the writer emits the line only for a label, so every file written before this commit parses unchanged and every pivot category's section is byte-identical to what it was. That is also why parse_kind_kw is an exhaustive match returning Result rather than a bool read off the rule's mere presence — a second keyword must fail loudly at the walker, not be read as label because it arrived on a kind_line.

One case the format already allowed and this had to answer: two ## Category: sections naming one category are one category (improvise-349m), merged in pass 1. Absence is not a claim, so the merge keeps a stated kind rather than letting a later kindless section erase it.

The generator coverage this section demands was measured, not assumed. kind_line is an alternative of category_entry, which the grammar walk reaches generically, so nothing in the generator needed changing — but that is the kind of claim this section exists to say must be checked. Probed with a throwaway test over 500 files from generator::improv_file(): 14 carried a kind: label line and all 14 parsed to a CategoryKind::Label, so at the default 256 cases each proptest run exercises the new production about seven times. The probe was removed rather than kept: its rate depends on proptest's internal RNG, so as a permanent assertion it would fail on a dependency bump for a reason unrelated to the format.

Folds are view lines: collapsed: and expanded(N): (improvise-felo.6)

Groups start collapsed, so a view persists what it opens. expanded(N): Cat opens a category's top N levels, expanded(N): Cat/Group opens that group and N-1 levels beneath it, a bare expanded: is N = 1, and collapsed: Cat/Group keeps its old meaning. The semantics, the precedence and the refusals are context/code/core.md §19; what is particular to the format:

• No version bump. The line is additive, and the user ruled that old files open closed like new ones, so v2025-04-09 still reads every file it did.
• N >= 1 is grammar, not validation: depth_number = @{ '1'..'9' ~ ASCII_DIGIT* }. Spelled with a Range and ASCII_DIGIT rather than ASCII_NONZERO_DIGIT, because the generator knows the first two and would emit nothing for the third — expanded(): — a file its own grammar refuses.
• axis_line comes before expanded_line in view_entry, so a category named expanded keeps its axis line, and the writer never spells a category depth bare (expanded: row would be that axis line).
• Two refusals the grammar cannot express — a cycle among > lines and one group given two folds — are a typed Refusal, which generated_file_parses accepts and nothing else does. The generator produces both at ordinary rates.
• The generator reaches the new rule, asserted by the_generator_writes_expanded_lines_with_and_without_a_depth over 512 deterministic files. Unlike the kind-line probe above, it asserts that each spelling occurs at all, not a rate, so a proptest bump fails it only if the walk stopped reaching the rule.

The tree walkers are infallible by construction

Every build_* in parser.rs operates on a Pair pest has already validated. "A formula has exactly one target", "a comparison has lhs, op and rhs" — pest guarantees these before the walker sees the pair. So every .expect(GRAMMAR_INVARIANT) is a bug marker, not a runtime condition: if one fires, the grammar and the walker have diverged.

That reasoning used to be extended to the final else arm of each dispatch, and the extension was wrong. The sentence here said the unlabelled arm was safe because ref_expr is the only remaining alternative of primary, COUNT the only remaining agg_func and >= the only remaining cmp_op. Each clause was true of the grammar as it stood, which is exactly the property a grammar edit changes -- so what the arm really encoded was a snapshot, silently re-interpreted as an invariant. "pest validated the pair" says the pair matches some alternative; it never said which, and an unlabelled arm reads the answer off a count the next commit is free to change.

Two different repairs, and which one is available is a property of the input:

• A &str, where no compile error exists. parse_agg_func and parse_cmp_op take the matched text, because agg_func and cmp_op are each one pest rule over string alternatives and pest_derive generates no per-alternative Rule. Nothing for a match to be exhaustive over, so they spell every alternative and unreachable!(GRAMMAR_INVARIANT) on anything else -- loud at the first parse rather than silent forever (improvise-492x).
• A Rule, where one does. The four dispatches on Pair::as_rule -- build_formula, build_unary, build_primary, filter_value_to_string -- match over an enum, so they name every rule, including a final or-pattern over the ones that cannot appear. Adding any rule to formula.pest is now a build failure at all four until someone decides what it means.

The greppable form: == Rule:: appears nowhere in parser.rs. An if rule == Rule::x chain cannot be exhaustive, and it is how the snapshot got written down as an invariant in the first place.

The cost is four or-patterns of ~28 enum names, which is genuinely noisy, and that is the price of the guarantee -- an invariant made unrepresentable rather than validated, applied to a grammar instead of to a type. The noise is also the tell that a walker has grown a dispatch, which is worth noticing.

build_primary is where this was demonstrated rather than argued, and the injection has to be chosen carefully: a new alternative that is an atomic rule is caught by accident, because first_inner then finds no child and panics on its own GRAMMAR_INVARIANT. Only a non-atomic alternative reaches the silent path. With pct_literal = { "#" ~ identifier } placed first in primary, the pre-fix tree built clean, the whole crate suite passed, and parse_formula("X = #Revenue") returned Ref("Revenue") -- the # discarded and the formula quietly wrong. An injection into an atomic rule would have reported the code safe when it was not.

parse.rs's walkers are the opposite and deliberately so: they return Result and name the section in the error, because the .improv file is hand-editable and a failure there is a user's problem, not a programmer's. parse_tree renders pest's error as a located message (line number plus the offending line), because "Parse error: {pest dump}" is unusable to somebody fixing a file by hand.

Two passes, because sections may appear in any order

Pass 1 turns each ## section into a plain collector (PCategory, PView) or a flat list; pass 2 builds the Workbook. That split is what makes the format order-independent — a ## View: may name a category defined further down. PCategory::items is a Vec in file order because IndexMap insertion order is load-bearing (gotcha 18), so pass 2 must replay the list as written.

Two ## Category: sections naming one category are one category, and the merge is a concatenation performed in pass 1 (ParsedFile::push_category, improvise-349m). Merging in pass 2 — building each section in turn — was not idempotent. A group is created either by the item that names it or by a > line, and build_category creates its own section's item-groups before its own > lines; so the first section's > groups landed ahead of the second section's item-groups, while format_md writes one section with all the items before all the > lines. The reparse then built the other order, and the two format_md outputs differed. Concatenating first makes the built order the written order by construction. Refusing a duplicate header was the alternative and would have made a legal hand-written file an error for no gain.

A section header takes its name as the whole line, totally

header_name = { pipe_quoted ~ &NEWLINE | line_name } (improvise-pczv). The &NEWLINE lookahead is load-bearing: without it, ## Category: |A|B parses as A and silently drops B. line_name catches everything else, so a header name can no longer fail to parse. Four rules share it — model_name, initial_view, category_section, view_section — which is why it is one rule and not four copies.

The writing half is format.rs::quote_header, and it is deliberately not quote_name: that rule is for a name embedded among other tokens, and applying it to a header would pipe-quote every ordinary spaced title the line carries perfectly well, rewriting the header of every existing document for no correctness gain. quote_header quotes exactly four things, each one thing the reader does:

• the empty name, and any name with leading or trailing whitespace — the reader trims, so the category comes back renamed while every data coordinate still names the untrimmed original: silent orphaning;
• a name containing \n or \r — it would start a second line, and Region\n## Data a second section;
• a name already spelled |…| — the reader would take the pipes off.

So a header is written raw where a reference is quoted. The grammar accepts both arms, so that asymmetry is a writer convention, not a format rule.

---

Storage is whole-document read/write, and that closed the SQLite question

Saving or loading a workbook only ever touches one whole document, so a backend needs no seeking, metadata or partial reads (improvise-6mq). It does need to enumerate and remove documents, because a browser build has no OS file manager behind it — hence list and delete beside read and write (improvise-i34).

Paths are opaque document names, not a directory tree: a backend may treat them as filesystem paths, OPFS keys or map entries. The trait doc says "backends that are not trees treat dir as a name prefix" — keep that sentence reachable. It is the sentence that turned out to be right while the OpfsFs impl was wrong, and the bug was found by reading it.

Because the trait is two ops over whole documents, an sqlite3_vfs — which needs xRead(offset,len), xWrite, xTruncate, xLock — has nothing to bind to. See gotcha 41 for the measurements that declined improvise-9ix.

storage_contract is a public function, not a test helper

It reads like a test helper and it was one: #[cfg(test)] in storage.rs, run against MemoryFs and PhysicalFs so they could not drift. The one backend that most needed it could never be handed to it — OpfsFs is cfg(target_arch = "wasm32") and its sync access handles exist in no host harness — so for that backend the contract lived only in a doc comment.

Moving it out of cfg(test) is what lets a browser run this sequence rather than a second transcription of it (improvise-vvl3/xf1). A second copy of a contract is the disease this codebase keeps curing. It returns Result rather than asserting because a failure has to travel out of a Web Worker as a string; the host tests .unwrap() it and get the same message.

Two bugs fell out of its first actual execution — see the dir parameter and the index rules below.

dir is a parameter, not a constant, because the three backends must agree at every dir, not only at the flat root the browser happens to use (improvise-vhcz). A backend whose list answered in a different path shape passes at Path::new("") — where joining is the identity — and fails at anything else.

impl Storage for Rc<T> lives in improvise-io because of the orphan rule

OpfsFs keeps four genuinely async operations — open, attach, refresh, commit_deletes — off the trait, so Storage can stay synchronous (gotcha 28). But App::storage is a Box<dyn Storage>, and a trait object can name none of the four: handing the backend to the App puts the async seam permanently out of reach. The host must hold the concrete backend and give the App a Storage, which means sharing one.

impl Storage for Rc<OpfsFs> is two foreign types in any other crate, so improvise-9x6 wrote a forwarding newtype in the worker and every future worker would have written it again. Here the trait is local, so the impl is allowed, and it is blanket rather than OpfsFs-specific because nothing about it is OPFS-shaped: it is the general statement that a shared handle to a backend is usable as that backend (improvise-hmws). Rc and not Arc because the one caller is a Web Worker — a single thread, and wasm32-unknown-unknown has no others. An Arc impl would be speculative.

---

OpfsFs: why a browser backend can be synchronous, and what the index holds

OPFS has two APIs. The promise-based one (getFileHandle, createWritable) is async, but FileSystemFileHandle::createSyncAccessHandle() yields a handle whose read/write/truncate/flush/close are ordinary blocking calls. It exists so wasm can do blocking file I/O — it is what SQLite's wasm build uses — and is available only inside a dedicated Web Worker, which is where improvise-9x6 puts the whole app. So Storage never had to become async.

Acquiring a handle is still async, as are directory listing and removal, so the backend splits in two:

• async, explicit: open, attach, refresh, commit_deletes, called by the worker from its message loop where awaiting is free;
• sync, via the trait: read/write use an already-attached handle; list answers from an index; delete drops the name from that index and queues the removal.

So a delete is visible to read and list immediately and lands in OPFS at the end of the batch. That ordering is the price of a synchronous trait and is invisible to a single worker. write therefore un-queues any pending delete of the same name: otherwise delete-then-write would be listed and readable and then removed by commit_deletes at the end of the batch — the one ordering the deferral can get wrong.

The index holds documents, not entries

Storage::list promises document names, and documents and entries stopped being the same thing the moment attach had to create a file to get a handle for a document that does not exist yet (improvise-vvl3). Three writers, and the split between them is the whole rule:

• attach indexes nothing. What it creates is zero bytes. Indexing there made a fresh origin list demo.improv before anything had been written, so the seed was skipped as "already there" and load_from got an empty buffer.
• Storage::write indexes the name (and un-queues a pending delete). A written document is the first thing there is to list.
• refresh rebuilds from OPFS, skipping zero-byte entries — and counting an entry it cannot measure as a document, since the one cause of that is another tab holding the exclusive handle.

holds_bytes is where the last clause lives, and its two failure modes get opposite answers, which is the whole function:

• getFileHandle fails → the name is not a file. keys() yields directories too, and a directory rejects with TypeMismatchError; an entry removed between keys() and here rejects as well. Neither is a document and neither can ever be attached or read, so listing one is the same picker bug this filter exists to remove (Copilot, PR #24).
• getFile fails on a real file handle → another context of the same origin holds the exclusive sync access handle (two tabs). An entry somebody else has open is certainly a document, so this fails open. Dropping it would hide a real document; failing the whole refresh would let a second tab break the first one's picker. Over-reporting costs one row — the direction Dirty::of is deliberately sloppy in.

Handles are exclusive per file, so attach closes any handle it replaces and detach releases the lock.

list joins dir like the other backends

The join is not cosmetic (improvise-vhcz). MemoryFs and PhysicalFs answer with dir-joined paths — paths a caller can hand straight back to read — and OpfsFs used to answer with bare names. At Path::new("") the two shapes coincide, which is the only dir the worker ever passes, so the disagreement was invisible: a worker that organised documents under docs/ would have got a full picker in every host test and an empty one in a browser, with nothing saying why.

dir is decoration rather than a filter, because this backend is flat by construction: entry_name reduces every path to its file name, so docs/a.improv and a.improv are the same OPFS entry and an entry name may not contain a separator at all. That is the trait's own "treat dir as a name prefix" — the doc was right and the impl was wrong.

---

Loading: decompression and decoding are two failures with two answers

load_from decompresses, then decodes UTF-8, with a separate Context on each. read_to_string would fuse them, and a valid gzip stream holding invalid UTF-8 would then be reported as unreadable gzip — a wrong diagnosis of a document that decompressed perfectly well. Decoding once, afterwards, also leaves one answer to "is this UTF-8" rather than one per branch.

The .gz suffix selects gzip regardless of backend: that is a property of the document name, not of the filesystem (improvise-6mq).

.improv text is the only native format (improvise-tt22). A legacy JSON workbook used to be detected by a leading { and loaded read-only through Workbook's serde derives. That path is gone, on the user's direction on PR #210. A document that still starts with { is refused with an error that names the format and says it is no longer supported. The test is a_legacy_json_workbook_is_refused_by_name. The error is decided rather than defaulted, because otherwise parse_md would report a grammar error at line 1, which diagnoses the wrong thing: the file is not malformed .improv. The check is on the decompressed text, so .improv.gz is covered too.

The serde derives that existed only for that path went with it: Workbook, Model, View, Category, Item, Group, CategoryKind, CellKey, DataStore's manual impls and the formula AST. So did indexmap's serde feature in improvise-core, improvise-io and the root crate, and improvise-formula's serde dependency. Each was removed and the whole workspace rebuilt, so the compiler named every other consumer. Two remained, and both are the wire: Axis (whose rename_all = "lowercase" is its wire spelling) and PageAxis. A third arrived with improvise-km71 by the same rule: FormulaId, which crosses the wire in ModelCommand::ReplaceFormula, FormulaSlot and FormulaEntry. So improvise-formula depends on serde again, for that one type. No derive went back on the formula AST. A fourth, CategoryKind, went back on when TileEntry began carrying it (improvise-bqm0; context/code/command.md). It returned as a wire type, not as a document one: nothing loads a Category through serde. CellValue was never a candidate, since it is carried on the wire too. JSON import (StartImportWizard, ImportJsonHeadless) is a separate feature and is untouched.

One test consumer had to move: the effect-classification tests fingerprinted the model and the views through serde_json. They now use Debug. format_md was rejected as the replacement, because it skips the virtual views and sorts the data, so it would have been a weaker fingerprint.

The first version of this paragraph said Debug was "no weaker" because it renders every field, including the derived formula_cache the JSON skipped. That was wrong (improvise-pc3b). A derived Debug prints a HashMap in hash order, and recompute_formulas clears and refills the cache. So a correct ViewOnly effect such as RebuildLayout changed the fingerprint in 90 of 200 runs. The suite passed only because the fixture had no formulas, so the cache was empty. Model now has a hand-written Debug that renders formula_cache and measure_agg in key order. The fingerprint keeps every field, including the derived one, and a refill in a different hash order is no longer read as a change. recomputing_formulas_leaves_the_model_fingerprint_alone pins this. view_only_effects_leave_shared_state_alone now runs over a formula-bearing fixture, for the same reason.

That fix was canonical within one instance, not across two (improvise-djcn). It fixed Model's own maps but not the maps inside its fields. DataStore has index, records and the symbol table, and View has page_selections, hidden_items and collapsed_groups. All still printed through a derived Debug, in hash order. Each HashMap gets its own RandomState, so two equal documents built separately printed differently in 32 of 32 runs.

• The effect-classification tests compare one instance with itself before and after, so they were unaffected.
• a_model_command_lands_where_its_keypress_lands compares two Apps. It passed only because its fixture had no cells and no multi-entry view maps.

Now:

• DataStore, SymbolTable and View have hand-written Debug impls. Each sorts wherever its storage is a hash map.
• DataStore resolves symbols to their strings. A Symbol's number is interning order, not content, so printing it would make two equal stores differ again.
• SymbolTable prints its interned strings, sorted.
• cells keeps its IndexMap order, because that order is part of the document (gotcha 18).
• Every hand-written Debug, Model's included, destructures self. A field added later is then a compile error there, where a debug_struct that names fields one by one would skip it silently. The pc3b version of Model's impl had exactly that hole.

parse_number_cell returns CellValue::Error naming the offending text rather than a fabricated 0.0. No token the current grammar admits actually fails f64::from_str (overflow saturates to ±inf), but if the two ever drift the mismatch must surface — never silently store a wrong number.

---

Persistence write order and the virtual views

format_md writes Views → Formulas → Categories → Data: views first because they are typically small and orient the reader, data last because it is the largest section. The parser tolerates any order (see the two-pass note above).

_Records and _Drill are scratch views the app rebuilds on demand. They put _Index/_Dim on axes, and those categories are never written — so persisting the views would produce axis lines naming categories the loader will not find. They are skipped on write, and dropped on load if an older file carries them, rather than reinstating a view the app would rebuild anyway; Initial View then falls through to the default, since the name it pointed at is gone. persisted_initial_view is the writing half: saving from records or drill mode falls back to the first real view rather than naming one the file will not contain (improvise-60z).

_Measure persists only its non-formula items — formula targets are rebuilt from the ## Formulas section, so format_md collects the targets and excludes them.

Page and Filter both carry the selected item after the axis keyword; the other axes have nothing to select. Axis::Filter round-trips even though it occurs only inside the never-written _Drill view, because the alternative was silently writing a Filter axis as some other axis (improvise-dz8, gotcha 27).

---

Import: order is a correctness property, not tidiness

analyzer::analyze_records emits distinct values in first-seen order, never HashSet order (improvise-7yx, gotcha 18). build_model adds items in exactly that order and IndexMap preserves it, so a hash order made the same CSV import to a different document on every run — different .improv bytes, different row order on screen, and a different detected date format, since detect_date_format only inspects the first ten samples and which ten it saw varied. The HashSet in analyze_records is there to deduplicate and never to iterate.

A date column imports as ONE hierarchical category (improvise-8jk6.3)

The user: "I think dates should usually be one category and we can use formulas to crosstab." So the default DateShape::Hierarchy builds one category named after the column, where the old import built up to three flat ones beside it (Date_Year, Date_Quarter, Date_Month):

• Its items are the finest chosen component, spelled so they stay unique across years: 2024-01, 2024-Q1, 2024, which are extract_date_component's existing spellings. A month is never Jan.
• The coarser chosen components are its groups, coarsest first (DateComponent::COARSE_TO_FINE): a month sits in its quarter, and the quarter's > line names its year. A level left unchosen is skipped rather than drawn empty, so Year + Month groups months straight into years.
• It opens folded, one row per year, with nothing to set, because every group starts closed (context/code/core.md §19).
• It costs one of the category slots, not up to four, and proposed_category_count counts it that way.
• The raw dates are gone from that category. The day is not a component, so two records of one month land on one cell and the collision step asks, as it does for any other collision. Split keeps the day.
• A value the detected format cannot read keeps its raw text as an ungrouped item, with a warning naming the column and the row count, rather than being dropped. The warning is separate from the split shape's, because that one reports an absent coordinate and this one a coordinate that is there and ungrouped. Unless its spelling is one the hierarchy generates (improvise-d0n6): a raw 2024-01 or 2024 is the same name as the generated month or year, so it would join the grouped item and the warning would be false. The build refuses, naming the column and the spellings and pointing at the split shape. That follows the collision rule below: where two records would silently be one thing, the user is asked and nothing is defaulted. A fallback to the split shape was rejected as a silent change of the user's choice, and a renamed raw item as altering the data. The check is Plan::spelling_clash.
• Item order is first-seen, not chronological. Order is a correctness property here (gotcha 18), and a group need not be contiguous, so a CSV that is not in date order still nests correctly. It is just drawn in the order it was read.

DateShape::Split is the old behaviour, unchanged, and stays a choice. s on the wizard's date step toggles the field under the cursor, and the header line for each field says which it will get ("one category: Year > Quarter > Month" or "split: Date, Date_Year, …"). It is worded once, in project.rs::date_shape_sentence. Headless, --split-date <field> does the same, and --extract keeps its meaning: which components to build.

Crosstabs by formula are not available yet. A formula can name a month (SUM(Amount WHERE Date = "2024-01"), Date IS "2024-01"), but a WHERE or IS naming a GROUP matches no record: both compare the coordinate to the name exactly. SUM(Amount WHERE Date = "2024") is an empty sum, and Date IS "2024" is 0 on every month of 2024. That was measured on this branch and reported for filing, not built here.

The ambiguous-date heuristic reads American-first, deliberately

01/02/2025 parses under both %m/%d/%Y and %d/%m/%Y and nothing in the data can tell them apart. detect_date_format is first-match over DATE_FORMATS, so the American reading wins. That is a defensible answer and not obviously the right one: a European CSV imports with days and months swapped, silently, and the derived _Month/_Quarter categories inherit the error. What makes it tolerable is that the wizard shows the detected format and lets the user change it — the same argument two-digit years already rested on. It is pinned by a test so that changing the answer is a decision somebody makes rather than a side effect of reordering the table.

Ruled a known limit (improvise-l8rh, the user, 2026-09-20). Two things the ruling rests on. Changing the default to day-first only moves which half of the world imports silently wrong; and the honest fix is larger than the heuristic — an ambiguous column should surface its ambiguity rather than resolve it, which changes FieldProposal's shape and the wizard step that displays it, the way Dataset::warnings returns a decision the user should hear about as data. Worth doing deliberately, not as a patch.

What makes the limit reasonable to live with is that it is now deterministic: analyze_records iterated a HashSet until improvise-7yx, so the ten samples detect_date_format inspects were a different ten every run and a column whose eleventh value ruled out %m/%d/%Y detected a different format run to run. The ten-sample window itself is unchanged and is part of the limit.

CSV parsing

Short rows keep every header column, missing trailing fields filled with Value::Null, and each affected column is reported as a warning with a row count. The reader is .flexible(true), so without this a short row parsed cleanly and the trailing columns were simply absent from the record map — no error, no warning, and skewed value counts in field-kind analysis (improvise-k8i).

A header name becomes a category name and survives into the .improv file, so anything riding along invisibly in one — a byte-order mark, a stray carriage return — is silent corruption rather than cosmetics. That is why the adversarial inputs are tested with controls: a check that cannot see a BOM where there is one proves nothing.

parse_csv_bytes_with_warnings exists so CSV import can come from any Storage backend rather than only a file (improvise-fuj); the path-taking wrappers remain for the native CLI.

A CSV field is text, and the field's role reads it (improvise-8jk6.4)

The parser used to type every field as it read it: text that parsed as an i64 or a finite f64 became a Value::Number. That is a lossy step, and it ran before anyone knew what the column was for. A column the user then forced to Category (--category, or the wizard's c) took its item names from the number's display, not from the file. So commit 00565796 became item 565796, and 220944e2 was read as scientific notation and became 22094400.0. The repo-stats pipeline (improvise-1xfb.5.1) found it.

CSV has no types. Whether 00565796 is a number depends on whether its column is a measure, and the parser cannot know that. So the parser now yields every non-empty field as a Value::String of its text (blank is still Null), and two readers apply the old numeric rule, csv_parser::number_in, only where a number is wanted:

• Analysis proposes a kind from a typed copy of the records (Scalars::typed_record), so a numeric CSV column is still proposed as a Measure, exactly as before.
• A measure takes its value through Scalars::number.

A category or a label is named from the text itself, so it is verbatim.

ImportPipeline carries the reading as a private scalars: Scalars: Typed for JSON and Text for CSV. JSON cannot share the text rule, because a JSON string is text by the file's own say. "02139" in JSON is a zip code, and reading it as a number would propose it as a Measure. A JSON number forced to Category has no source text to keep, because serde_json does not keep one. It is named by Value's own display, which is what writing it back as JSON would say: 42, 1.5, and 1000.0 for 1e3.

The parser returns CsvRecords, not Vec<Value>. It derefs to [Value], so a caller can read it. It converts to a Vec only inside the crate, and the only public way into a pipeline is ImportPipeline::from_csv. This is not caution for its own sake. When the parser first changed, eleven tests in four files failed. Every one handed parser output to ImportPipeline::new, the JSON constructor, which would read every column as text and propose Revenue as a Category. Production had the same shape at all three CSV entry points: main.rs's single file, ImportPipeline::from_merge (the merge path, from PR #314), and ImportJsonHeadless. from_merge merged in with Self::new and now calls Self::from_csv. Now that mistake does not compile. Two holes remain, and both take a deliberate step. ImportPipeline::records is still a public Vec<Value>, and a test helper once copied it out of a CSV pipeline into a JSON one. And CsvRecords derefs to a slice that .to_vec() will copy.

An import refuses what it cannot keep (improvise-8jk6.5, improvise-8jk6.6)

Two failures used to end in an import that saved and reported success.

A formula that does not parse. build_model_with_warnings ran if let Ok(formula) = parse_formula(..), so a bad --formula vanished. The headless run then saved and exited 0, and the wizard kept the bad text in its list and dropped it the same way at build. Both now go through one function, import_formula, whose error names the formula and carries pest's parse error: Formula error in 'Y = x +': …. The prefix matches the TUI formula minibuffer's Formula error:.

• ImportPipeline::parsed_formulas parses them all. If any fails, it refuses and names every formula that failed, one per line (a message the user needs is never truncated). The build calls it before creating anything, so a refusal writes nothing.
• The wizard checks at confirm_formula, the one place the user can still fix the text (gotcha 22). A formula that does not parse stays in the buffer with the error as the step's message, and never enters the list.

Nothing kept. The one skip left after improvise-8jk6.2 is a record that is not a JSON object. When every record is skipped, there are no fields, so there are no proposals and no categories. The build did refuse that case, but by accident, with "At least one category must be accepted". That tells the user to change a field's kind when there are no fields. The ticket's own repro ([1, 2, "three"]) already exited 1 on the lpzu base, so the headline did not reproduce there. What did reproduce was the reason. nothing_to_import now states the rule and runs before the category check: Nothing to import: all 3 records were skipped as not JSON objects, or the source has no records. The wizard asks it first on ReviewProposals, where another array path can still be chosen.

One case meets PR #314's "a header-only file is still a file". A folder whose CSVs are all header-only has no records, only sources. Built from from_merge's seeded proposals alone, it would save File items, empty columns and one empty records view per file. It is refused here, because it kept no record. A header-only file beside files with rows is unaffected, since some record was kept. The user confirmed this on PR #321: "Refuse (as #321 does)".

A partial skip stays a warning. Some records kept and some skipped still builds, exits 0, and warns with a count: Skipped 1 of 3 records that were not JSON objects. The argument: a non-object in an array of records is not a record that failed to import. It is structurally not a record, such as a trailing null or a stray scalar in a hand-edited file. Every record that is a record was kept, and the count is on stderr for a script to read. Making it fatal would fail every import that carries one stray element. Making it a distinct non-zero exit that still writes gives a contract where import … && next stops even though the file was written.

The user ruled on PR #321: "Add a --strict flag". The default stays exit 0 with the counted warning, pinned by an_import_that_skips_some_records_warns_with_a_count_and_still_writes. import --no-wizard --strict turns any skipped record into a refusal: --strict: Skipped 1 of 3 records that were not JSON objects, exit 1.

A strict refusal writes nothing. A refusal that still wrote the file would be the half-met contract argued against above: && stops while the output sits there, and a script that checks only the file gets a model that --strict existed to prevent. So the check runs in run_headless_import before the build. It reads ImportPipeline::skip_warning, the same function the build's warning comes from, so the two cannot disagree about what was skipped. "Skipped" here means a record that is not an object, the only skip left after 8jk6.2. An unparsed date or a short CSV row is a warning about a record that was kept, and --strict does not refuse it.

A folder of CSVs is one sparse model, with a view per file (improvise-lpzu)

improvise import <folder> exists for improvise-1xfb.5.1. code-maat writes one CSV per analysis into a directory, and the schemas differ: coupling has entity,coupled,degree,average-revs, age has entity,age-months, and abs-churn has date,added,deleted,commits. The user ruled (2026-09-25): "Records are already sparse: the model can just leave null/empty values for columns not in a given CSV file." So files with different headers merge into one model. It is not refused, and it is not split into one model per schema.

A folder argument means exactly its *.csv files. expand_import_paths takes the arguments in order and replaces each folder with its *.csv entries. The extension check is case-insensitive, the listing is not recursive, and the order is Storage::list's, which is sorted by contract. Then the existing path runs, with one difference: a folder always merges. Two or more files merge through merge_csvs_from with a File category, and so does a folder holding a single CSV. That gives the one file its File item and its per-file view.

The single-CSV rule is the user's ruling (2026-09-25), which replaced this PR's first answer, "import dir/ is exactly import dir/*.csv". Their words: "yes, we might want to have some kind of update import eventually (don't plan this yet, but we'd want file identity for this)". A folder names a collection that may grow, so a document built from it should record which file each record came from from the start.

A single CSV named directly, not through a folder, still imports alone, without File. That is deliberate. Naming one file asks for that file, not for a collection, and the shipped recipes (examples/demo.improv from examples/demo.csv) and every existing single-file script would otherwise gain a File category and a second view. File identity for update-import can be asked for by naming a folder. The test is a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not.

Apart from that, a folder still behaves as the glob would, and four cases follow from that rather than being chosen one by one:

• Non-CSV entries are skipped, not refused, because a glob skips them. This includes the output .improv, which a second run finds in the folder.
• A folder with no CSV at all is refused, naming the folder. A glob that matches nothing is an error too, and importing nothing would write an empty document.
• Files and folders mix freely, in argument order.
• A file named twice is read twice. For example, dir/a.csv dir/ reads a.csv twice, and its records collide with themselves. The headless path refuses that like any other collision. Nothing deduplicates, because the shell glob would not either.

The listing goes through Storage::list so the browser host can reuse it (gotcha 2). Storage knows documents, not folders, so whether an argument is a folder is the caller's question. The CLI answers it at the edge with Path::is_dir and passes an ImportPath::{File, Folder}. An OPFS host would answer it from its own UI.

Sparse merge. A record from a file without some column has no key for it, so Plan::place gives it no coordinate there (improvise-8jk6.2 above), and no measure cell. Nothing is fabricated: no 0 and no "". The analyzer already classified each column over the values present (improvise-l1f5), so a column only one file has is judged by that file's values alone. The category cap counts the union of every file's categories, header-only files included (see the next paragraph). Rows from different files cannot collide, because File differs. One caveat on "judged by its values": a JSON "" is not yet dropped by the analyzer the way null is, so a JSON numeric column holding "" is proposed as a Label (improvise-8jk6.11). CSV is unaffected, because its parser turns a blank field into null.

A header-only file is still a file (the Codex P2 on PR #314). code-maat writes a header and no rows whenever nothing passes its threshold, so coupling.csv is often exactly that. The pipeline is analyzed from records, and such a file has none, so it had no File item, its columns never reached the model, and its view filtered to an item that did not exist. ImportPipeline::from_merge therefore seeds from Merged::sources:

• the File proposal's items are the sources' items, in file order, so every file has its item whether it has rows or not;
• a source column that no record has a value in is proposed as a Label. That is the analyzer's own verdict for an all-null column (improvise-l1f5), and a column with no values is exactly that: nothing can be classified, so it takes no pivot slot and does not count toward the category cap until the user reclassifies it in the wizard or with --category/--measure.

This is also the file identity the user wants kept for a future update-import.

A file with no measure column gets a records view (improvise-8jk6.13). A record with no measure makes no cell, so such a file contributes items and no data. Its view had nothing to page _Measure to, and paged onto another file's measure, which the same view hides; the cell there was a Record, editable, under this file's File. Now source_view gives such a file the records layout (view::records_steps, the one home of what a records view is) with File still on Filter at the file's item. It draws zero rows, because there are no cells. A header-only file whose columns are all Labels is the same case. In records mode the columns are the whole model's categories and measures, not just this file's: records mode has no per-view column filter. With no rows that is cosmetic.

Each file gets a view (the user, 2026-09-25: "dir import should do this: one view filtered by the file column and projected to the columns in that file"). merge_csvs_from returns a Merged whose sources list each File item with the columns its header named. Two files with one stem share an item, so their columns are unioned. The pipeline carries sources, and the build adds one view per source after the merged Default view, which stays the initial view. Each per-file view:

• Is named after the file's stem. A name already taken, including the virtual _Records and _Drill, gets 2, 3 and so on. A view named _Drill would never be written and would be dropped on load.
• Puts File on Axis::Filter, fixed at that file's item. Filter and not Page, because the view is that file's projection. Paging to another file would show that file's records through this file's columns and hidden measures, which is wrong on every page but one. Filter constrains without being navigable (gotcha 27). Until now it appeared only in the never-saved _Drill view, but the .improv grammar has always round-tripped it. per_file_views_survive_a_round_trip_and_repeat_byte_for_byte proves that on a saved per-file view, and a_saved_per_file_view_projects_only_its_file proves the shared grid_in projection, which both the TUI and the browser draw from, carries it as a fixed coordinate with nothing to cycle.
• Seats the file's own categories (its columns, plus date parts derived from them) through View::on_category_added, in model order: the rule a single-file import uses. Every other category goes on None. The file's records lack those, so nothing splits.
• Puts _Measure on Page, as Workbook::new does, with other files' measures hidden and the page selection on the file's first measure. A formula target comes from no file, so it stays visible. A file with no measure gets a records view instead (above).
• Is never named like another view from the same import. Default.csv and Default 2.csv become Default 2 and Default 2 2; without the same-batch check both would be named Default 2 and one view would silently replace the other.

A per-file view is made only while File is an accepted category. A user who skipped or reclassified it has no filter to build views from.

Two things this does not do. Page cycling still steps onto a hidden item and draws an empty grid: [/] in a per-file view can reach another file's measure (improvise-yd2t; the Default view already behaved this way). And a CSV with its own File column has it overwritten by the merge's, which is the existing behaviour of merge_csvs.

The wizard's back path is a history stack

advance pushes the step it leaves; previous pops it. Keeping the path actually taken, rather than re-deriving the reverse of each conditional transition, means back can never disagree with how the user really got here (improvise-907).

Preconditions live in validation_error() and are checked on the step where the user can still fix them — never deferred to build_model after the final confirm. Two gates earn that: too many categories (improvise-mzv) and no category at all (improvise-gfq); both are fixable with c at ReviewProposals, and both used to refuse only at the last keystroke. MAX_CATEGORIES here is the model's own constant, which counts regular categories only. Until improvise-felo.1 it was a private copy (core.md §9).

NameModel is terminal for advance: leaving it is a build, not a step change. build() parks the workbook in built on Done; a build failure leaves the user on NameModel with the error. Going back from Done discards built, so accepting an import always reflects the latest field decisions.

scalar_name decides what a JSON scalar contributes as an item name. Value::Null — which is also what the CSV parser produces for a blank field — and a blank string are absences, not names; stringifying them used to invent an item literally called "null" (improvise-ych). Every other value keeps its printed form.

A record with no value for a category keeps its cell, without that coordinate (improvise-8jk6.2). Records are sparse: a CellKey names only the coordinates it has. So a missing field, a null and a blank string all mean "no coordinate here". None of them makes an item: not "null", not "", and no synthetic placeholder. The same rule covers labels, which used to become an item named "". The user's words, 2026-09-25: "this limit in import should be fixed, it dates to when this wasn't sparse." That limit was improvise-5j9's rule: such a record "has no place in the grid" and was dropped, counted and reported as Skipped N of M records missing category values. 5j9's warning made the loss visible, and the loss is now gone, so the warning went with it. The only record still skipped is one that is not a JSON object.

Where such a record shows up is decided by the axes, not by the import. The test is a_record_with_no_coordinate_is_drawn_only_where_that_category_is_off_the_grid:

• The category on Row or Column: no position holds the record. Its other coordinates may still draw a row, but the cell under the category's items is empty, and the column total leaves it out.
• The category on Page: no selection shows it.
• The category on None: it is aggregated like any other record.
• Records mode: its row shows a blank cell in that category's column.
• .improv: its Data line omits the category, and it round-trips.
• Drill: a drill that does not constrain the category reaches it.

No "(none)" position, by the user's decision (2026-09-25). They were asked whether Row, Column or Page should give such records a place of their own. Their answer: "I think leaving it out for now is fine, maybe a UI element somewhere that shows the number of records underneath the current view?" That indicator is improvise-gtn7.10, and it is not an import concern.

Colliding records are a question, not a default (improvise-cgqt)

Two source records that land on the same coordinate tuple are one cell, so something has to happen to the second one. improvise-2cw9 made that "sum and warn". The user ruled on 2026-09-25, after an Etrade import summed 31 cells and reported it in one truncated line, that the wizard asks. It offers three choices: (a) sum them, (b) keep them separate with a generated category, or (c) go back. So ImportPipeline::on_duplicate is a DuplicatePolicy (Sum | Separate(name) | Refuse), and its default is Refuse. Any caller that never asked therefore gets an error and never a silent sum. ImportJsonHeadless builds its pipeline as a struct literal, so the compiler made it choose too. It chose Refuse and points at :import.

Detection comes before the build and shares the build's placement. Where a record lands is Plan::place: its categories, then its date parts, then its labels, then its measure values. Only a record that is not an object is dropped, since a missing value is no coordinate rather than a reason to drop (improvise-8jk6.2). collisions() and build_model_with_warnings both call it, so the question the wizard asks and the thing the build does cannot disagree. Neither has a second walk of the record. A collision is a cell that more than one record writes, meaning a tuple plus a measure (improvise-9bit). A Collision names the tuple and only the records that share one of its cells. A record with no measure makes no cell, so it collides with nothing. The tuple includes label values, as CellKey always has.

This PR's first version keyed on the tuple alone. It was superseded in review (improvise-9bit), and the argument it gave is withdrawn. Under that rule, two records with disjoint measures on one tuple (one has Revenue, the other Cost) counted as colliding although no cell sums. The wizard then offered "Sum them into one cell" where nothing would sum, and with the Refuse default the CLI refused a sparse-measure CSV that had always imported cleanly. The question the user is asked is "these values would be added together", so the rule has to be the one that decides whether anything is added.

The data is structured. A Collision holds coords plus 1-based records positions. WizardCache::collisions carries it, and WizardCollision::describe is the one wording, used by the TUI and by the CLI. A browser wizard must draw the same rows (gotcha 32), not a second sentence.

Where the step sits. It comes after ReviewProposals and after ConfigureDates when that step is visited, and before DefineFormulas. It appears only when collisions() is non-empty, and the list is recomputed each time the step is entered, because it depends on the field choices. The two field steps are exactly the steps that can change it. Date parts are functions of a value that is already a coordinate, so they can never split or merge a collision, but they count toward the category cap, and that is why the question waits until they are known. The history stack (gotcha 22) handles the new paths without change: ResolveDuplicates → NameDuplicateCategory → DefineFormulas retraces backwards. The chooser acts on the cursor. Enter on this step is ChooseDuplicate, never a bare Advance: the policy starts at Refuse, which validation_error blocks, so a bare advance could never leave the step.

Separate is an occurrence index, not a row number. This is the user's preferred shape. The records in a collision get 1, 2, 3… in source order, and every other record gets 1. That includes a record on the same tuple that shares no cell with the others: it lands on occurrence 1 beside the first colliding record, and since the two write different measures, nothing sums. The category holds only as many items as the deepest collision, rather than one item per record, and every existing coordinate keeps its meaning. Items are added 1..=n in numeric order (gotcha 18). The category goes on Axis::None. Left to on_category_added it would land on Page, and the default grid would then show only occurrence 1, which silently hides the other records. On None the grid aggregates across occurrences, so the first screen shows the same totals Sum would. The records stay distinct, so Avg and records mode see each one. A policy is moot when nothing collides: no category is added.

The name is suggested and editable: Occurrence, or the first Occurrence_N that no header (accepted or not) and no category the build will create already uses. NameDuplicateCategory refuses an empty name or one that is already a category. suggested_duplicate_category lives on the pipeline, so the CLI's separate uses the same rule.

At the category cap, Separate is shown and disabled. The reported import had 12 regular categories, which was MAX_CATEGORIES when this was written. Option (b) adds a regular category, and the user declined a hidden virtual one. So at the cap the option is drawn "(unavailable)" with its reason and remedy: go back and Skip a column or make it a Measure, or choose Sum. The reason is decided in ImportWizard::duplicate_options, so every front end words it the same way. This was the conservative reading, and it left an open question for the user. The user answered it on 2026-09-25 by raising the cap to 24 (improvise-felo.1), so "keep separate" is available for the reported import now. The option is still disabled for any import that fills the new cap.

Headless paths refuse by default. improvise import --no-wizard has nobody to ask, so --on-duplicate refuse|sum|separate defaults to refuse. It prints every colliding tuple and the flag, exits non-zero, and writes nothing. Summing by default would repeat exactly the silent outcome the user called critical. Separating by default would reshape a scripted import's document and fail outright at the cap. A refusal is the one default that cannot be wrong without saying so. This changes behaviour: a script that relied on 2cw9's implicit sum now needs --on-duplicate sum. :import-json has no flag and refuses, naming :import.

The 2cw9 argument that summing is the right default is superseded here, as its own evidence entry anticipated. Its warning survives as the Sum confirmation.

---

CSV export

render_csv is separated from writing so the bytes can go to any Storage backend, not just a file (improvise-fuj). It goes through the csv crate rather than joining with commas, and it honours the view's number format — a value of 1200 under a comma-grouping format renders 1,200, and a label like Smith, Jr. splits into two fields, corrupting every row (improvise-1cz).