Files
improvise/site/docs/test-evidence/formula-and-io.html
T
Edward Langley 787e354b9b doc-site: name context/code/pr-review.md as an omission
The `build` job fails on main and on every branch merged from it:

  doc-site: context/code/pr-review.md is neither published nor a named omission

NOT a merge loss -- scripts/build-doc-site.mjs in this branch is
byte-identical to main's, and neither lists the file. PR #96 added
context/code/pr-review.md without a manifest line, which is exactly the
gate improvise-res9 added: a doc added without a line publishes nothing
and says nothing, so the generator refuses.

Omitted rather than published, and the line is cloud-agents.md's rather
than tooling.md's. tooling.md explains checkers a reader of the code will
meet; pr-review.md is about THIS REPOSITORY'S PROCESS -- where a Codex
review appears, how to tell it from our own agents' comments, why absence
is not approval. Nobody visiting a website is about to read a PR thread.

Both homes updated, because an omission is restated beside the manifest
AND in doc-site.md section 2, and the prose there counted them ("Neither
is referenced...") so it went stale at three.

This strays into scripts/, which is Wave N's ground. Flagged in the
hand-back; it is a one-line data entry either way if the orchestrator
would rather publish it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
2026-09-21 11:12:00 -07:00

228 lines
72 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>improvise-formula and improvise-io — improvise docs</title>
<link rel="stylesheet" href="../style.css" />
</head>
<body>
<div class="shell">
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html" class="here">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
<main>
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/test-evidence/formula-and-io.md">context/test-evidence/formula-and-io.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#test-evidence-improvise-formula-and-improvise-io">Test evidence — improvise-formula and improvise-io</a></li><li class="l2"><a href="#improvise-formula-src-parser-rs">improvise-formula/src/parser.rs</a></li><li class="l3"><a href="#parse-sum-with-inline-where-filter">parse_sum_with_inline_where_filter</a></li><li class="l3"><a href="#hyphen-in-bare-identifier-reference">hyphen_in_bare_identifier_reference</a></li><li class="l3"><a href="#hyphen-in-bare-identifier-inside-aggregate">hyphen_in_bare_identifier_inside_aggregate</a></li><li class="l3"><a href="#pipe-quoted-target-is-unquoted-like-a-reference">pipe_quoted_target_is_unquoted_like_a_reference</a></li><li class="l3"><a href="#pipe-quoted-target-applies-escapes">pipe_quoted_target_applies_escapes</a></li><li class="l3"><a href="#bare-target-is-unchanged">bare_target_is_unchanged</a></li><li class="l2"><a href="#improvise-formula-src-quote-rs">improvise-formula/src/quote.rs</a></li><li class="l3"><a href="#unescaping-inverts-quoting">unescaping_inverts_quoting</a></li><li class="l3"><a href="#a-trailing-lone-backslash-is-kept">a_trailing_lone_backslash_is_kept</a></li><li class="l3"><a href="#a-quoted-name-round-trips-through-the-parser">a_quoted_name_round_trips_through_the_parser</a></li><li class="l3"><a href="#a-keyword-is-quoted-even-though-it-looks-bare">a_keyword_is_quoted_even_though_it_looks_bare</a></li><li class="l2"><a href="#improvise-io-src-storage-rs">improvise-io/src/storage.rs</a></li><li class="l3"><a href="#a-shared-backend-is-a-backend-and-a-clone-is-the-same-one">a_shared_backend_is_a_backend_and_a_clone_is_the_same_one</a></li><li class="l3"><a href="#memory-fs-satisfies-the-storage-contract">memory_fs_satisfies_the_storage_contract</a></li><li class="l3"><a href="#physical-fs-satisfies-the-storage-contract">physical_fs_satisfies_the_storage_contract</a></li><li class="l2"><a href="#improvise-io-src-persistence-mod-rs">improvise-io/src/persistence/mod.rs</a></li><li class="l3"><a href="#a-truncated-gzip-document-fails-as-an-unexpected-eof">a_truncated_gzip_document_fails_as_an_unexpected_eof</a></li><li class="l3"><a href="#a-gzip-document-with-a-flipped-byte-fails-its-checksum">a_gzip_document_with_a_flipped_byte_fails_its_checksum</a></li><li class="l3"><a href="#a-valid-gzip-of-invalid-utf8-is-reported-as-not-utf8">a_valid_gzip_of_invalid_utf8_is_reported_as_not_utf8</a></li><li class="l3"><a href="#round-trips-through-memory-storage">round_trips_through_memory_storage</a></li><li class="l3"><a href="#gzip-round-trips-through-memory-storage">gzip_round_trips_through_memory_storage</a></li><li class="l3"><a href="#physical-and-memory-backends-write-identical-bytes">physical_and_memory_backends_write_identical_bytes</a></li><li class="l2"><a href="#improvise-io-src-persistence-parse-rs">improvise-io/src/persistence/parse.rs</a></li><li class="l3"><a href="#two-sections-naming-one-category-round-trip-to-a-stable-group-order">two_sections_naming_one_category_round_trip_to_a_stable_group_order</a></li><li class="l3"><a href="#parse-md-formula-category-suffix-without-space">parse_md_formula_category_suffix_without_space</a></li><li class="l3"><a href="#parse-md-formula-ending-in-bracketed-pipe-name-keeps-raw">parse_md_formula_ending_in_bracketed_pipe_name_keeps_raw</a></li><li class="l3"><a href="#unparseable-number-token-yields-error-not-zero">unparseable_number_token_yields_error_not_zero</a></li><li class="l3"><a href="#malformed-numbers-in-data-never-silently-zero">malformed_numbers_in_data_never_silently_zero</a></li><li class="l3"><a href="#parse-error-names-line-number-and-contents">parse_error_names_line_number_and_contents</a></li><li class="l3"><a href="#roundtrip-preserves-an-awkward-category-name">roundtrip_preserves_an_awkward_category_name</a></li><li class="l2"><a href="#improvise-io-src-persistence-format-rs">improvise-io/src/persistence/format.rs</a></li><li class="l3"><a href="#virtual-views-are-not-written">virtual_views_are_not_written</a></li><li class="l2"><a href="#improvise-io-src-persistence-roundtrip-tests-rs">improvise-io/src/persistence/roundtrip_tests.rs</a></li><li class="l3"><a href="#initial-view-skips-a-virtual-active-view">initial_view_skips_a_virtual_active_view</a></li><li class="l3"><a href="#virtual-views-in-an-existing-file-are-dropped-on-load">virtual_views_in_an_existing_file_are_dropped_on_load</a></li><li class="l3"><a href="#filter-axis-round-trips-with-its-selection">filter_axis_round_trips_with_its_selection</a></li><li class="l3"><a href="#parse-md-round-trips-formula-category-with-special-chars">parse_md_round_trips_formula_category_with_special_chars</a></li><li class="l3"><a href="#a-keyword-named-category-is-pipe-quoted">a_keyword_named_category_is_pipe_quoted</a></li><li class="l3"><a href="#a-keyword-named-category-round-trips-everywhere-it-appears">a_keyword_named_category_round_trips_everywhere_it_appears</a></li><li class="l3"><a href="#a-formula-targeting-a-keyword-named-measure-round-trips">a_formula_targeting_a_keyword_named_measure_round_trips</a></li><li class="l3"><a href="#a-category-name-the-header-line-cannot-carry-round-trips">a_category_name_the_header_line_cannot_carry_round_trips</a></li><li class="l3"><a href="#a-model-and-view-name-the-header-line-cannot-carry-round-trip">a_model_and_view_name_the_header_line_cannot_carry_round_trip</a></li><li class="l3"><a href="#an-ordinary-spaced-header-stays-bare">an_ordinary_spaced_header_stays_bare</a></li><li class="l3"><a href="#pipes-that-do-not-close-the-header-line-are-part-of-the-name">pipes_that_do_not_close_the_header_line_are_part_of_the_name</a></li><li class="l3"><a href="#a-header-written-by-an-older-build-still-loads">a_header_written_by_an_older_build_still_loads</a></li><li class="l3"><a href="#a-csv-header-with-an-embedded-newline-survives-a-save">a_csv_header_with_an_embedded_newline_survives_a_save</a></li><li class="l3"><a href="#an-item-name-the-item-line-cannot-carry-round-trips">an_item_name_the_item_line_cannot_carry_round_trips</a></li><li class="l3"><a href="#a-bare-item-name-with-a-space-is-rejected">a_bare_item_name_with_a_space_is_rejected</a></li><li class="l2"><a href="#improvise-io-src-persistence-export-rs">improvise-io/src/persistence/export.rs</a></li><li class="l3"><a href="#export-csv-quotes-commas-per-rfc4180">export_csv_quotes_commas_per_rfc4180</a></li><li class="l2"><a href="#improvise-io-src-import-analyzer-rs">improvise-io/src/import/analyzer.rs</a></li><li class="l3"><a href="#detect-short-year-format">detect_short_year_format</a></li><li class="l3"><a href="#an-ambiguous-date-is-read-american-first">an_ambiguous_date_is_read_american_first</a></li><li class="l3"><a href="#a-day-first-only-value-settles-the-ambiguity">a_day_first_only_value_settles_the_ambiguity</a></li><li class="l3"><a href="#the-distinct-values-are-in-first-seen-order">the_distinct_values_are_in_first_seen_order</a></li><li class="l3"><a href="#the-detected-date-format-does-not-depend-on-the-hash-seed">the_detected_date_format_does_not_depend_on_the_hash_seed</a></li><li class="l2"><a href="#improvise-io-src-import-csv-parser-rs">improvise-io/src/import/csv_parser.rs</a></li><li class="l3"><a href="#short-rows-pad-missing-trailing-columns-with-null">short_rows_pad_missing_trailing_columns_with_null</a></li><li class="l3"><a href="#short-rows-produce-warning-naming-column-and-count">short_rows_produce_warning_naming_column_and_count</a></li><li class="l3"><a href="#full-rows-produce-no-warnings">full_rows_produce_no_warnings</a></li><li class="l3"><a href="#a-leading-utf8-bom-is-not-part-of-the-first-header">a_leading_utf8_bom_is_not_part_of_the_first_header</a></li><li class="l3"><a href="#crlf-line-endings-leave-no-carriage-return-in-the-data">crlf_line_endings_leave_no_carriage_return_in_the_data</a></li><li class="l3"><a href="#an-empty-file-is-no-records-and-no-warnings">an_empty_file_is_no_records_and_no_warnings</a></li><li class="l3"><a href="#a-header-with-no-rows-is-no-records">a_header_with_no_rows_is_no_records</a></li><li class="l3"><a href="#a-single-column-file-yields-one-field-per-record">a_single_column_file_yields_one_field_per_record</a></li><li class="l2"><a href="#improvise-io-src-import-wizard-rs">improvise-io/src/import/wizard.rs</a></li><li class="l3"><a href="#wizard-previous-reverses-each-advance">wizard_previous_reverses_each_advance</a></li><li class="l3"><a href="#wizard-build-parks-the-workbook-on-done">wizard_build_parks_the_workbook_on_done</a></li><li class="l3"><a href="#wizard-build-failure-stays-on-name-model">wizard_build_failure_stays_on_name_model</a></li><li class="l3"><a href="#wizard-blocks-advance-when-over-category-limit">wizard_blocks_advance_when_over_category_limit</a></li><li class="l3"><a href="#wizard-blocks-advance-when-no-category-accepted">wizard_blocks_advance_when_no_category_accepted</a></li><li class="l3"><a href="#build-model-null-category-value-is-missing-not-the-item-null">build_model_null_category_value_is_missing_not_the_item_null</a></li><li class="l3"><a href="#build-model-literal-string-null-is-a-real-category-item">build_model_literal_string_null_is_a_real_category_item</a></li><li class="l3"><a href="#build-model-empty-string-category-value-is-missing">build_model_empty_string_category_value_is_missing</a></li><li class="l3"><a href="#build-model-warns-about-records-skipped-for-missing-category">build_model_warns_about_records_skipped_for_missing_category</a></li><li class="l3"><a href="#is-does-not-swallow-the-start-of-a-longer-word">is_does_not_swallow_the_start_of_a_longer_word</a></li><li class="l3"><a href="#where-does-not-swallow-the-start-of-a-longer-word">where_does_not_swallow_the_start_of_a_longer_word</a></li><li class="l3"><a href="#where-requires-a-space-after-the-keyword">where_requires_a_space_after_the_keyword</a></li><li class="l3"><a href="#the-grammar-is-parsed-once-however-many-files-the-strategy-yields">the_grammar_is_parsed_once_however_many_files_the_strategy_yields</a></li><li class="l3"><a href="#export-csv-omits-the-rows-the-view-prunes">export_csv_omits_the_rows_the_view_prunes</a></li></ul></details><h1 id="test-evidence-improvise-formula-and-improvise-io">Test evidence — <code>improvise-formula</code> and <code>improvise-io</code></h1>
<p>What each of these tests is evidence <em>of</em>: the bug it reproduced, the fault that was injected, and why it can fail. Keyed by exact test name — a rename orphans the entry, and <code>scripts/check-comment-runs.py --check-evidence</code> catches that. The mechanism arguments are in <a href="../code/formula-and-io.html"><code>context/code/formula-and-io.md</code></a>.</p>
<p><strong><code>###</code> is reserved for a name that really is an annotated test.</strong> A fixture, strategy or shared harness whose narrative is worth keeping gets a <code>####</code> subordinate heading that says so. The checker as written accepts any <code>fn</code>, so a helper under <code>###</code> would pass today and fail the moment that is tightened.</p>
<hr />
<h2 id="improvise-formula-src-parser-rs"><a class="anchor" href="#improvise-formula-src-parser-rs">#</a><code>improvise-formula/src/parser.rs</code></h2>
<h3 id="parse-sum-with-inline-where-filter"><a class="anchor" href="#parse-sum-with-inline-where-filter">#</a>parse_sum_with_inline_where_filter</h3>
<p>Regression: a <code>WHERE</code> inside the aggregate&#39;s parens must parse as the aggregate&#39;s <em>inline</em> filter, not as a top-level <code>WHERE</code> clause on the formula. The two land in different places on the AST — <code>Expr::Agg(_, _, Some(filter))</code> against <code>Formula::filter</code> — and the difference is silent at parse time.</p>
<h3 id="hyphen-in-bare-identifier-reference"><a class="anchor" href="#hyphen-in-bare-identifier-reference">#</a>hyphen_in_bare_identifier_reference</h3>
<p>Regression. The hand-rolled tokenizer that preceded the pest grammar did not allow <code>-</code> inside bare identifiers, so a reference to a persistence-legal name like <code>east-coast</code> was silently parsed as <code>east - coast</code>: a <strong>subtraction of two unrelated <code>Expr::Ref</code>s</strong>, which evaluates rather than erroring. The pest grammar mirrors <code>improv.pest</code>&#39;s <code>bare_name</code>, which does allow <code>-</code>.</p>
<h3 id="hyphen-in-bare-identifier-inside-aggregate"><a class="anchor" href="#hyphen-in-bare-identifier-inside-aggregate">#</a>hyphen_in_bare_identifier_inside_aggregate</h3>
<p>The same fix through the aggregate path, because <code>SUM(east-coast)</code> reaches the identifier rule by a different route.</p>
<h3 id="pipe-quoted-target-is-unquoted-like-a-reference"><a class="anchor" href="#pipe-quoted-target-is-unquoted-like-a-reference">#</a>pipe_quoted_target_is_unquoted_like_a_reference</h3>
<p>Regression, and a data bug rather than a cosmetic one. The tree walker took the target&#39;s <strong>raw span</strong>, so a pipe-quoted target came back with its delimiters still attached while an <code>Expr::Ref</code> for the very same name came back unquoted. Downstream, <code>Model::add_formula</code> then created a <code>_Measure</code> item literally named <code>|My Target|</code>, which never matched the user&#39;s item <code>My Target</code>: the formula wrote into a ghost column and nothing referring to it by name resolved.</p>
<p>The test asserts both halves — the stripped target <em>and</em> that the reference spelled identically names the same item — because asserting only the target would pass under a walker that unquoted both wrongly in the same way.</p>
<p>Why no existing test caught it: the writer emits <code>Formula::raw</code>, not <code>target</code>, so the file text is identical either way and every round-trip test was blind to it. See <a href="../code/formula-and-io.html"><code>context/code/formula-and-io.md</code></a> § &quot;A formula target is unquoted&quot;.</p>
<h3 id="pipe-quoted-target-applies-escapes"><a class="anchor" href="#pipe-quoted-target-applies-escapes">#</a>pipe_quoted_target_applies_escapes</h3>
<p>The escape table applies to the target too, not only to <code>Expr::Ref</code> — the companion that stops the fix above from being &quot;strip the outer pipes&quot;.</p>
<h3 id="bare-target-is-unchanged"><a class="anchor" href="#bare-target-is-unchanged">#</a>bare_target_is_unchanged</h3>
<p>The negative arm: a bare target has nothing to strip. Without it, &quot;always strip the first and last character&quot; would pass the two tests above.</p>
<hr />
<h2 id="improvise-formula-src-quote-rs"><a class="anchor" href="#improvise-formula-src-quote-rs">#</a><code>improvise-formula/src/quote.rs</code></h2>
<h3 id="unescaping-inverts-quoting"><a class="anchor" href="#unescaping-inverts-quoting">#</a>unescaping_inverts_quoting</h3>
<p>The reading half inverts the writing half over every escape both grammars define, plus the <code>\X</code> passthrough neither of them assigns a meaning to. The <code>&quot;42&quot;</code> case is there because <code>pipe_quote</code> is unconditional: a <code>.improv</code> text cell holding <code>42</code> must not read back as a number.</p>
<h3 id="a-trailing-lone-backslash-is-kept"><a class="anchor" href="#a-trailing-lone-backslash-is-kept">#</a>a_trailing_lone_backslash_is_kept</h3>
<p>Before <a class="tid" href="../tickets.html#improvise-i2m5" title="improvise-io still carries its own pipe-quoting copy, without the keyword guard">improvise-i2m5</a> the unescape table existed <strong>twice</strong> — here (as <code>parser.rs::unquote_pipe</code>) and as <code>improvise-io/src/persistence/quoting.rs::unescape_pipe</code>. The two agreed on every input either grammar&#39;s <code>pipe_inner</code> can produce, and differed <em>only</em> on a trailing lone backslash, which <code>pipe_inner</code> cannot emit: io returned it, <code>parser.rs</code> panicked. This pins the surviving answer for that unreachable input, so the surviving function is total.</p>
<h3 id="a-quoted-name-round-trips-through-the-parser"><a class="anchor" href="#a-quoted-name-round-trips-through-the-parser">#</a>a_quoted_name_round_trips_through_the_parser</h3>
<p>The property the whole module exists for: a name <code>quote_name</code> spells parses, and the parser hands back the name that went in — as a <strong>target</strong> and as a <strong>reference</strong>, which after the target-unquoting fix are the same rule. The fixture list is the interesting part: keywords (<code>WHERE</code>, <code>SUM</code>), a leading digit (<code>9lives</code>), the escapes, and the empty name.</p>
<h3 id="a-keyword-is-quoted-even-though-it-looks-bare"><a class="anchor" href="#a-keyword-is-quoted-even-though-it-looks-bare">#</a>a_keyword_is_quoted_even_though_it_looks_bare</h3>
<p>Pins the keyword guard, which is deliberately stricter than the grammar requires and is <strong>user-confirmed</strong> — do not revert it as over-caution. The argument (PEG rule ordering, and <code>## Formulas</code> carrying formula text verbatim) is in <a href="../code/formula-and-io.html"><code>context/code/formula-and-io.md</code></a>.</p>
<hr />
<h2 id="improvise-io-src-storage-rs"><a class="anchor" href="#improvise-io-src-storage-rs">#</a><code>improvise-io/src/storage.rs</code></h2>
<h3 id="a-shared-backend-is-a-backend-and-a-clone-is-the-same-one"><a class="anchor" href="#a-shared-backend-is-a-backend-and-a-clone-is-the-same-one">#</a>a_shared_backend_is_a_backend_and_a_clone_is_the_same_one</h3>
<p><a class="tid" href="../tickets.html#improvise-hmws" title="Storage cannot expose an async seam through Box&lt;dyn Storage&gt;">improvise-hmws</a>. A worker must hold the concrete backend — for the four async methods the trait does not carry — <em>and</em> hand the <code>App</code> a <code>Storage</code>. Before the blanket <code>impl Storage for Rc&lt;T&gt;</code>, <code>Rc&lt;OpfsFs&gt;</code> was not a <code>Storage</code> and could not be made one outside this crate (orphan rule), so <code>improvise-worker-server</code> wrote a forwarding newtype, and so would every worker after it.</p>
<p><strong>The regression guard is the signature</strong>: this does not compile without the impl. What it asserts <em>beyond</em> compiling is the part a newtype could still get wrong — that the clone the <code>App</code> holds and the handle the message loop keeps are one backend and not two.</p>
<h3 id="memory-fs-satisfies-the-storage-contract"><a class="anchor" href="#memory-fs-satisfies-the-storage-contract">#</a>memory_fs_satisfies_the_storage_contract</h3>
<h3 id="physical-fs-satisfies-the-storage-contract"><a class="anchor" href="#physical-fs-satisfies-the-storage-contract">#</a>physical_fs_satisfies_the_storage_contract</h3>
<p>Both run <code>contract::storage_contract</code> at a <strong>non-empty <code>dir</code></strong>, which is the whole point of <code>dir</code> being a parameter: at <code>Path::new(&quot;&quot;)</code> a backend that answers <code>list</code> in the wrong path shape passes, because joining is the identity there. <code>OpfsFs</code> did exactly that and the disagreement was invisible until <a class="tid" href="../tickets.html#improvise-vhcz" title="[bug] OpfsFs::list disagrees with the other backends on path shape">improvise-vhcz</a> (see <a href="../code/formula-and-io.html"><code>context/code/formula-and-io.md</code></a>).</p>
<hr />
<h2 id="improvise-io-src-persistence-mod-rs"><a class="anchor" href="#improvise-io-src-persistence-mod-rs">#</a><code>improvise-io/src/persistence/mod.rs</code></h2>
<h4 id="the-shared-harness-load-corrupt-gzip"><a class="anchor" href="#the-shared-harness-load-corrupt-gzip">#</a>The shared harness: <code>load_corrupt_gzip</code></h4>
<p>Not a test, and so deliberately not a <code>###</code> heading — the harness the two gzip-corruption tests below share. It asserts two <strong>preconditions</strong> before handing the damaged bytes to the loader: that the corruption actually changed the bytes, and that the result still looks like a gzip document by its magic bytes. Without the second, a corruption that also broke the routing would make the loader fail for a different reason and the test would pass while proving nothing.</p>
<h3 id="a-truncated-gzip-document-fails-as-an-unexpected-eof"><a class="anchor" href="#a-truncated-gzip-document-fails-as-an-unexpected-eof">#</a>a_truncated_gzip_document_fails_as_an_unexpected_eof</h3>
<p>A <code>.gz</code> document cut short — the disk filled, or the process died mid-save. The <strong>kind</strong> is asserted (<code>io::ErrorKind::UnexpectedEof</code>), not merely that an error occurred, because &quot;an error occurred&quot; would also pass if the loader had stopped decompressing altogether and choked on the compressed bytes as text.</p>
<h3 id="a-gzip-document-with-a-flipped-byte-fails-its-checksum"><a class="anchor" href="#a-gzip-document-with-a-flipped-byte-fails-its-checksum">#</a>a_gzip_document_with_a_flipped_byte_fails_its_checksum</h3>
<p>The other way a <code>.gz</code> arrives damaged, and it is a different kind (<code>InvalidInput</code>), which is the reason to test both: one test against one corruption cannot tell &quot;the decoder refuses damaged input&quot; from &quot;the decoder refuses short input&quot;. The first is <em>incomplete</em>, the second is <em>wrong</em> — &quot;you lost the tail&quot; versus &quot;you cannot trust any of it&quot;.</p>
<p>The byte flipped is in the <strong>CRC32 field</strong>, the first four of the eight-byte gzip trailer, and not in the DEFLATE payload (Copilot, PR #22). A payload byte also yields <code>InvalidInput</code>, but it can do so for malformed compressed data rather than for the mismatch the test is named after, and the name would then be a guess about which. Corrupting the trailer leaves the stream decodable, so the only thing left to fail is the checksum — hence the assertion on the <em>source</em> error&#39;s message, since anyhow&#39;s <code>to_string</code> shows only the outermost context.</p>
<h3 id="a-valid-gzip-of-invalid-utf8-is-reported-as-not-utf8"><a class="anchor" href="#a-valid-gzip-of-invalid-utf8-is-reported-as-not-utf8">#</a>a_valid_gzip_of_invalid_utf8_is_reported_as_not_utf8</h3>
<p>The third failure, and the one that made the other two worth keeping apart in the code as well as in the tests (Copilot, PR #22). A gzip stream can be perfectly well-formed and still hold bytes that are not UTF-8; <code>read_to_string</code> decodes as it decompresses, so a single <code>Context</code> around it called that document <em>unreadable gzip</em> — a wrong diagnosis of a document that decompressed without complaint.</p>
<p>The precondition asserts the payload really is invalid UTF-8; without it the test would also pass on a document that merely failed to decompress, which is the case the other two already own. It also asserts the message does <strong>not</strong> mention gzip: the gzip stream was fine, and blaming it is the bug.</p>
<h4 id="the-fixture-storage-fixture"><a class="anchor" href="#the-fixture-storage-fixture">#</a>The fixture: <code>storage_fixture</code></h4>
<p>Not a test. A workbook with a category, items, a formula and data — enough that a round-trip exercises every section of the format. Every gzip test above and every storage round-trip below reuses it, so that &quot;the document survived&quot; is a claim about a real document rather than about an empty one.</p>
<h3 id="round-trips-through-memory-storage"><a class="anchor" href="#round-trips-through-memory-storage">#</a>round_trips_through_memory_storage</h3>
<p><a class="tid" href="../tickets.html#improvise-6mq" title="Introduce Storage/VFS abstraction in persistence layer">improvise-6mq</a>: persistence goes through <code>Storage</code>, so a workbook can be saved and loaded with <strong>no filesystem at all</strong> — no <code>tempfile</code>, no disk. This is the shape the browser backend plugs into.</p>
<h3 id="gzip-round-trips-through-memory-storage"><a class="anchor" href="#gzip-round-trips-through-memory-storage">#</a>gzip_round_trips_through_memory_storage</h3>
<p>The <code>.gz</code> suffix selects gzip regardless of backend: a property of the document <em>name</em>, not of the filesystem.</p>
<h3 id="physical-and-memory-backends-write-identical-bytes"><a class="anchor" href="#physical-and-memory-backends-write-identical-bytes">#</a>physical_and_memory_backends_write_identical_bytes</h3>
<p>Swapping the backend must not change the bytes written. This is what stops a backend from quietly normalising or re-encoding what it is handed.</p>
<hr />
<h2 id="improvise-io-src-persistence-parse-rs"><a class="anchor" href="#improvise-io-src-persistence-parse-rs">#</a><code>improvise-io/src/persistence/parse.rs</code></h2>
<h3 id="two-sections-naming-one-category-round-trip-to-a-stable-group-order"><a class="anchor" href="#two-sections-naming-one-category-round-trip-to-a-stable-group-order">#</a>two_sections_naming_one_category_round_trip_to_a_stable_group_order</h3>
<p><a class="tid" href="../tickets.html#improvise-349m" title="format_md/parse_md is not idempotent when a .improv file has two Category sections with the same name">improvise-349m</a>: <code>format_md(parse_md(text))</code> was not idempotent when a file carried two <code>## Category:</code> sections with the same name — the two outputs differed only in the order of the <code>&gt;</code> group lines. It looked like a flake because only <code>grammar_prop_tests::generated_file_roundtrips</code> saw it, and only when the generator happened to emit a duplicate header; taking proptest&#39;s minimal input and running it deterministically is what turned it into a bug.</p>
<p>The fixture is the smallest shape that shows it: a first section defining a group by a <code>&gt;</code> line only, and a second whose <em>item</em> names a different group that also gets a <code>&gt;</code> line. Both groups must have parents, or <code>format_md</code> writes only one <code>&gt;</code> line and the differing internal order is invisible in the text — which is the trap when shrinking this fixture further. Fault injected: reverting <code>ParsedFile::push_category</code> to a plain <code>Vec::push</code> fails it, and the ticket&#39;s own minimal input was checked against the fix as well.</p>
<p><strong>Stability is only half of what the merge owes, and the first version of this test could see only that half.</strong> Adversarial review injected the other fault — <code>Some(_existing) =&gt; {}</code>, which <em>drops</em> the duplicate section outright — and the whole <code>improvise-io</code> suite stayed green at 2000 proptest cases while the fixture&#39;s <code>- X[G2]</code> and <code>&gt; G2[P2]</code> vanished from the document without a word. Main&#39;s behaviour was unstable-but-lossless; a test that compares two printed forms cannot tell that from stable-and-lossy, because both sides of the equality lose the same thing. The three <code>contains</code> assertions run <em>before</em> the equality for exactly that reason, and they fail under the drop.</p>
<h3 id="parse-md-formula-category-suffix-without-space"><a class="anchor" href="#parse-md-formula-category-suffix-without-space">#</a>parse_md_formula_category_suffix_without_space</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-zgb" title=".improv formula loader mis-splits category suffix via rfind(&quot; [&quot;)">improvise-zgb</a> (b). The formulas walker located the <code>[Category]</code> suffix with <code>raw.rfind(&quot; [&quot;)</code>, so a hand-written line with no space before the bracket — <code>- Total = Food + Gas[Type]</code> — was not recognised and the formula was <strong>silently retargeted to <code>_Measure</code></strong>. The suffix is now part of the grammar (<code>formula_line</code>) and walked structurally.</p>
<h3 id="parse-md-formula-ending-in-bracketed-pipe-name-keeps-raw"><a class="anchor" href="#parse-md-formula-ending-in-bracketed-pipe-name-keeps-raw">#</a>parse_md_formula_ending_in_bracketed_pipe_name_keeps_raw</h3>
<p>Companion to <a class="tid" href="../tickets.html#improvise-zgb" title=".improv formula loader mis-splits category suffix via rfind(&quot; [&quot;)">improvise-zgb</a>: a formula <em>ending</em> in a pipe-quoted identifier that contains <code>&quot; [&quot;</code> must not have the quoted name mistaken for a category suffix. It stays a <code>_Measure</code> formula with its raw text intact. Without this arm, &quot;look for the last <code>[</code>&quot; would pass the test above.</p>
<h3 id="unparseable-number-token-yields-error-not-zero"><a class="anchor" href="#unparseable-number-token-yields-error-not-zero">#</a>unparseable_number_token_yields_error_not_zero</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-4yc" title="Malformed numbers in .improv file silently become 0.0">improvise-4yc</a>. The <code>Rule::number</code> arm used <code>as_str().parse().unwrap_or(0.0)</code>, so any numeric token the grammar admitted but <code>f64::from_str</code> rejected silently became <code>Number(0.0)</code> — a fabricated value in a spreadsheet.</p>
<p>No token the <em>current</em> grammar admits actually fails <code>f64</code> parsing (overflow saturates to inf), so the test exercises <code>parse_number_cell</code> <strong>directly</strong>, with the forms the grammar could plausibly drift toward admitting. That is the point: the guard exists for the drift, not for today.</p>
<h3 id="malformed-numbers-in-data-never-silently-zero"><a class="anchor" href="#malformed-numbers-in-data-never-silently-zero">#</a>malformed_numbers_in_data_never_silently_zero</h3>
<p>The same bug through the grammar, covering all three routings: <code>--5</code> fails the <code>number</code> rule entirely and becomes <code>bare_value</code> → <code>Text</code>; <code>3.14.15</code> matches the <code>3.14</code> prefix, commits, and the rest fails → a whole-file parse error naming the text; <code>1e999</code> is admitted and saturates to <code>+inf</code>. None of the three is <code>0.0</code>.</p>
<h3 id="parse-error-names-line-number-and-contents"><a class="anchor" href="#parse-error-names-line-number-and-contents">#</a>parse_error_names_line_number_and_contents</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-6kj" title="Pest parse errors lack file:line context for users">improvise-6kj</a>: parse failures were wrapped as <code>Parse error: {e}</code>, leaning on pest&#39;s raw rendering instead of stating the location in plain words. A user fixing a hand-edited file needs the line number and the offending line.</p>
<h3 id="roundtrip-preserves-an-awkward-category-name"><a class="anchor" href="#roundtrip-preserves-an-awkward-category-name">#</a>roundtrip_preserves_an_awkward_category_name</h3>
<p><a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a> as a property rather than a list. <code>quote_header</code> decides which names the header line cannot carry, and that decision is a small set of clauses — so the property is what earns its place: for <em>any</em> name, save-then-load must give the name back. It runs against <code>populated_model</code> rather than a fresh one because a workbook with no data cannot show a coordinate losing its category, which is the silent half of the bug.</p>
<h4 id="why-populated-model-and-not-a-fresh-workbook"><a class="anchor" href="#why-populated-model-and-not-a-fresh-workbook">#</a>Why <code>populated_model</code> and not a fresh workbook</h4>
<p>Not a test — the fixture the property above runs against. It already holds categories, items and cells, and the <em>reason</em> is the evidence: a default-constructed workbook makes the round-trip test vacuous for the data-coordinate half, because there is no coordinate to lose its category.</p>
<h4 id="the-strategy-awkward-name"><a class="anchor" href="#the-strategy-awkward-name">#</a>The strategy: <code>awkward_name</code></h4>
<p>Not a test — the input strategy the property above draws from. The generated shapes are the whole-line contract&#39;s stress points: edge whitespace, embedded line breaks, pipes, and text that looks like another section marker (<code>\n## Data</code>).</p>
<hr />
<h2 id="improvise-io-src-persistence-format-rs"><a class="anchor" href="#improvise-io-src-persistence-format-rs">#</a><code>improvise-io/src/persistence/format.rs</code></h2>
<h3 id="virtual-views-are-not-written"><a class="anchor" href="#virtual-views-are-not-written">#</a>virtual_views_are_not_written</h3>
<p><a class="tid" href="../tickets.html#improvise-60z" title="Virtual views _Records and _Drill should not be persisted to .improv files">improvise-60z</a>. <code>_Records</code> and <code>_Drill</code> are scratch views the app builds on demand. They put <code>_Index</code>/<code>_Dim</code> on axes, and those categories are never written to the file — so persisting the views produces axis lines referring to categories the loader will not find. The test also asserts a real view <em>is</em> written, so &quot;write no views at all&quot; cannot pass it.</p>
<hr />
<h2 id="improvise-io-src-persistence-roundtrip-tests-rs"><a class="anchor" href="#improvise-io-src-persistence-roundtrip-tests-rs">#</a><code>improvise-io/src/persistence/roundtrip_tests.rs</code></h2>
<h3 id="initial-view-skips-a-virtual-active-view"><a class="anchor" href="#initial-view-skips-a-virtual-active-view">#</a>initial_view_skips_a_virtual_active_view</h3>
<p><a class="tid" href="../tickets.html#improvise-60z" title="Virtual views _Records and _Drill should not be persisted to .improv files">improvise-60z</a>&#39;s writing half: saving while in records mode must not leave <code>Initial View: _Records</code> pointing at a view the file no longer contains. It asserts the loaded workbook&#39;s active view <strong>exists in the loaded workbook</strong>, not merely that the string changed.</p>
<h3 id="virtual-views-in-an-existing-file-are-dropped-on-load"><a class="anchor" href="#virtual-views-in-an-existing-file-are-dropped-on-load">#</a>virtual_views_in_an_existing_file_are_dropped_on_load</h3>
<p>The reading half: a file written by an older build may already carry the scratch views, and loading one must not reinstate them.</p>
<h3 id="filter-axis-round-trips-with-its-selection"><a class="anchor" href="#filter-axis-round-trips-with-its-selection">#</a>filter_axis_round_trips_with_its_selection</h3>
<p><a class="tid" href="../tickets.html#improvise-dz8" title="Add Axis::Filter for drill-fixed coordinates">improvise-dz8</a>. <code>Axis::Filter</code> arises only inside the <code>_Drill</code> view, which is never written, so this is defensive — but the alternative to supporting it in the format is <strong>silently writing a Filter axis as something else</strong>.</p>
<h3 id="parse-md-round-trips-formula-category-with-special-chars"><a class="anchor" href="#parse-md-round-trips-formula-category-with-special-chars">#</a>parse_md_round_trips_formula_category_with_special_chars</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-zgb" title=".improv formula loader mis-splits category suffix via rfind(&quot; [&quot;)">improvise-zgb</a> (a). <code>rfind(&quot; [&quot;)</code> mis-split a suffix whose category name itself contains <code>&quot; [&quot;</code>: the writer emitted <code>[Margin [%]]</code> unquoted and the parser then split inside the name, producing garbage raw text and a category called <code>%]</code>. Category names in the suffix are now pipe-quoted on write and parsed via the grammar&#39;s <code>name</code> rule.</p>
<h3 id="a-keyword-named-category-is-pipe-quoted"><a class="anchor" href="#a-keyword-named-category-is-pipe-quoted">#</a>a_keyword_named_category_is_pipe_quoted</h3>
<p><a class="tid" href="../tickets.html#improvise-i2m5" title="improvise-io still carries its own pipe-quoting copy, without the keyword guard">improvise-i2m5</a>. <code>.improv</code> now spells names through <code>improvise_formula::quote_name</code>, which pipe-quotes a formula keyword. Before the switch, io&#39;s own copy of the rule had no keyword guard and wrote <code>## Category: SUM</code>, <code>- WHERE</code> and <code>SUM=WHERE = 1</code> bare — that was the <strong>only</strong> output difference the whole persistence suite showed.</p>
<p>The names round-trip either way (<code>improv.pest</code> accepts <code>pipe_quoted</code> wherever a <code>name</code> goes), so this pins the <em>spelling</em>, not the survival. It also pins the header exception: <code>category_section</code> takes the whole line and never goes through <code>quote_name</code> on either side, so only <em>references</em> are quoted.</p>
<h3 id="a-keyword-named-category-round-trips-everywhere-it-appears"><a class="anchor" href="#a-keyword-named-category-round-trips-everywhere-it-appears">#</a>a_keyword_named_category_round_trips_everywhere_it_appears</h3>
<p>The round trip the quoting exists for, at every position a keyword-named category or item reaches the file: the axis line, the page selection, <code>hidden:</code>, the category section and the data coordinates. A guard that covered only some of them would pass a single-position test.</p>
<h3 id="a-formula-targeting-a-keyword-named-measure-round-trips"><a class="anchor" href="#a-formula-targeting-a-keyword-named-measure-round-trips">#</a>a_formula_targeting_a_keyword_named_measure_round_trips</h3>
<p>Why the keyword guard is worth its cosmetic cost even though bare keywords parse: a <code>## Formulas</code> line carries formula text <strong>verbatim</strong>, so a name written by <code>format_md</code> is later read by <code>parse_formula</code>. The two grammars share one namespace, and <code>quote_name</code> is the one rule spelling names for both.</p>
<h3 id="a-category-name-the-header-line-cannot-carry-round-trips"><a class="anchor" href="#a-category-name-the-header-line-cannot-carry-round-trips">#</a>a_category_name_the_header_line_cannot_carry_round_trips</h3>
<p><a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a>. A section header was the one name position that never went through a quoting rule, so a name the header <em>line</em> could not carry was written raw into it. The table <code>NAMES_THE_LINE_CANNOT_CARRY</code> is the whole rule, and each entry is a different failure:</p>
<ul>
<li><code>Region\nName</code> — the newline ended the header and the rest became a stray line: the file would not parse at all.</li>
<li><code>Region\n## Data</code> — worse: the remainder looked like another section marker.</li>
<li><code>&quot;Region &quot;</code> / <code>&quot; Region&quot;</code> — the reader trims, so the category came back renamed while every data coordinate still named the untrimmed original: <strong>silent orphaning</strong>.</li>
<li><code>&quot;&quot;</code> and <code>&quot;|Region|&quot;</code> — would be taken for a quoted name and unquoted.</li>
</ul>
<p>This is reachable and not theoretical — see <code>a_csv_header_with_an_embedded_newline_survives_a_save</code>.</p>
<h3 id="a-model-and-view-name-the-header-line-cannot-carry-round-trip"><a class="anchor" href="#a-model-and-view-name-the-header-line-cannot-carry-round-trip">#</a>a_model_and_view_name_the_header_line_cannot_carry_round_trip</h3>
<p>The same rule at the other three header positions: the model title, the view title, and the <code>Initial View:</code> reference that must still resolve to the view.</p>
<h3 id="an-ordinary-spaced-header-stays-bare"><a class="anchor" href="#an-ordinary-spaced-header-stays-bare">#</a>an_ordinary_spaced_header_stays_bare</h3>
<p>The negative arm, and the one that stops <code>quote_header</code> from quietly becoming <code>quote_name</code>. The header carries its name as the whole line — that is the contract — and quoting a name the line carries fine would rewrite the header of every existing document for nothing. It also asserts that a <em>reference</em> to the same name is still quoted, so the two rules stay distinguishable.</p>
<h3 id="pipes-that-do-not-close-the-header-line-are-part-of-the-name"><a class="anchor" href="#pipes-that-do-not-close-the-header-line-are-part-of-the-name">#</a>pipes_that_do_not_close_the_header_line_are_part_of_the_name</h3>
<p>The one genuinely ambiguous line, decided by the grammar&#39;s <code>&amp;NEWLINE</code>: <code>## Category: |a| and |b|</code> is a category called <code>|a| and |b|</code>. Without the lookahead the pipe-quoted arm would match <code>a</code> and <strong>silently drop the rest of the line</strong>.</p>
<h3 id="a-header-written-by-an-older-build-still-loads"><a class="anchor" href="#a-header-written-by-an-older-build-still-loads">#</a>a_header_written_by_an_older_build_still_loads</h3>
<p>A header written before <a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a> is a bare line and must still load as one — spaces, pipes in the middle and all. The compatibility direction of the same grammar change.</p>
<h3 id="a-csv-header-with-an-embedded-newline-survives-a-save"><a class="anchor" href="#a-csv-header-with-an-embedded-newline-survives-a-save">#</a>a_csv_header_with_an_embedded_newline_survives_a_save</h3>
<p>The bug as a user meets it, and the reason <a class="tid" href="../tickets.html#improvise-pczv" title=".improv section headers are a second, unquoted naming rule">improvise-pczv</a> is a corruption bug rather than tidiness. An RFC 4180 CSV may quote a header field containing a newline; nothing on the import path forbids it, so the category carried one — and the <code>.improv</code> file the user then saved <strong>could not be reopened</strong>. The test asserts the precondition (the import still admits the header verbatim) so it cannot pass by the import having quietly started rejecting it.</p>
<h3 id="an-item-name-the-item-line-cannot-carry-round-trips"><a class="anchor" href="#an-item-name-the-item-line-cannot-carry-round-trips">#</a>an_item_name_the_item_line_cannot_carry_round_trips</h3>
<p><a class="tid" href="../tickets.html#improvise-odbc" title="[bug] bank-info.improv in the repo root does not parse">improvise-odbc</a>. <code>bank-info.improv</code> in the repo root failed to load at <code>- JORDAN MARTINEZ</code> (line 293) with <code>expected pipe_quoted</code>. The ticket named two readings — a stale file, or a writer/reader gap in which improvise emits an item line the grammar rejects — and said to decide by asking whether improvise can <em>write</em> that line today.</p>
<p>It cannot. <code>item_list</code>&#39;s <code>name</code> is <code>pipe_quoted | bare_name</code>, and <code>bare_name</code> is <code>[A-Za-z_][A-Za-z0-9_-]*</code> with no space; <code>quote_name</code> mirrors it character for character, so every one of those item names is emitted pipe-quoted and read straight back. The file was hand-edited in <code>c48a5cd</code> — the <code>v2025-04-09</code> migration quoted the <code>Date</code> items and left the <code>Entity</code> ones bare — and never round-tripped. So the reading is &quot;stale file&quot;, and the standing finding was that nothing exercised it. This test is what that finding turned into: the item names from the file that motivated the ticket, driven through the writer and back. It asserts both directions — the spaced name is quoted, the bare-legal name stays bare.</p>
<h3 id="a-bare-item-name-with-a-space-is-rejected"><a class="anchor" href="#a-bare-item-name-with-a-space-is-rejected">#</a>a_bare_item_name_with_a_space_is_rejected</h3>
<p>The other half of the same finding: the exact line the file carries is <em>not</em> something the reader accepts, so the file really was broken rather than the error message being misleading. Asserting the failure keeps the test above honest — without it, &quot;the writer quotes it&quot; would be consistent with the reader also accepting the unquoted form. The trailing precondition (the same line quoted <em>does</em> parse) stops it passing because of some unrelated syntax error.</p>
<hr />
<h2 id="improvise-io-src-persistence-export-rs"><a class="anchor" href="#improvise-io-src-persistence-export-rs">#</a><code>improvise-io/src/persistence/export.rs</code></h2>
<h3 id="export-csv-quotes-commas-per-rfc4180"><a class="anchor" href="#export-csv-quotes-commas-per-rfc4180">#</a>export_csv_quotes_commas_per_rfc4180</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-1cz" title="export_csv emits unquoted CSV — commas in values corrupt output">improvise-1cz</a>. <code>export_csv</code> joined labels and values with bare commas — no RFC 4180 quoting — and ignored the view&#39;s number format. Under a comma-grouping format a value of 1200 renders as <code>1,200</code> and a label like <code>Smith, Jr.</code> splits into two fields, <strong>corrupting every row</strong>. The test reads the file back with a real CSV reader and asserts the field <em>count</em>, which is the thing a string <code>contains</code> check cannot see.</p>
<p><strong>Strengthened by <a class="tid" href="../tickets.html#improvise-711n" title="parse_number_format silently yields 0 decimals for the ,.2f spelling our own docs show">improvise-711n</a>.</strong> The assertion on the value was <code>records[1][1].starts_with(&quot;1,200&quot;)</code>, and the fixture&#39;s format is <code>,.2f</code> — so the check could not tell <code>1,200</code> from <code>1,200.00</code> and sat green while <code>parse_number_format</code> threw the <code>.2</code> away. It is now an exact equality. Reverting the <code>format.rs</code> fix fails it with <code>left: &quot;1,200&quot;</code>, <code>right: &quot;1,200.00&quot;</code>, which is the cross-crate demonstration that the format bug reached a user&#39;s exported file and not only the screen.</p>
<hr />
<h2 id="improvise-io-src-import-analyzer-rs"><a class="anchor" href="#improvise-io-src-import-analyzer-rs">#</a><code>improvise-io/src/import/analyzer.rs</code></h2>
<h3 id="detect-short-year-format"><a class="anchor" href="#detect-short-year-format">#</a>detect_short_year_format</h3>
<p>Two-digit years are ambiguous with the four-digit format, so <code>%m/%d/%Y</code> matches first. Expected, not a defect: the wizard shows the detected format and the user can override it. The same argument as the ambiguity below.</p>
<h3 id="an-ambiguous-date-is-read-american-first"><a class="anchor" href="#an-ambiguous-date-is-read-american-first">#</a>an_ambiguous_date_is_read_american_first</h3>
<p><code>01/02/2025</code> and <code>02/03/2025</code> parse under <code>%m/%d/%Y</code> <em>and</em> <code>%d/%m/%Y</code>, and nothing in the data can tell them apart. The heuristic is first-match over <code>DATE_FORMATS</code>, so the American reading wins — a European CSV imports with its days and months swapped, silently, and the derived <code>_Month</code>/<code>_Quarter</code> categories inherit the error. <strong><a class="tid" href="../tickets.html#improvise-l8rh" title="The ambiguous-date heuristic reads American-first, so a European CSV imports with days and months swapped">improvise-l8rh</a> ruled this a known limit</strong> rather than a defect, so this test is what makes changing the answer a decision somebody makes rather than a side effect of reordering the table. The argument is in <a href="../code/formula-and-io.html"><code>context/code/formula-and-io.md</code></a>.</p>
<p>The loop at the end is the precondition that makes this a statement about <em>ambiguity</em> rather than about one format merely fitting: the other reading has to fit too, or there is nothing to resolve.</p>
<h3 id="a-day-first-only-value-settles-the-ambiguity"><a class="anchor" href="#a-day-first-only-value-settles-the-ambiguity">#</a>a_day_first_only_value_settles_the_ambiguity</h3>
<p>Where the two readings genuinely can be told apart, the unambiguous value decides — first match still, but <code>%m/%d/%Y</code> now fails. Without this, &quot;always American&quot; would pass the test above.</p>
<h3 id="the-distinct-values-are-in-first-seen-order"><a class="anchor" href="#the-distinct-values-are-in-first-seen-order">#</a>the_distinct_values_are_in_first_seen_order</h3>
<p><a class="tid" href="../tickets.html#improvise-7yx" title="Close test gaps flagged by deep review">improvise-7yx</a>, found while pinning the date heuristic. <code>analyze_records</code> built its distinct values by collecting into a <code>HashSet</code> and iterating it, so the order was different in every process. This is gotcha 18&#39;s bug in a second place, and it reaches further than a tidy-up: <code>build_model</code> adds items in exactly this order, <code>IndexMap</code> preserves insertion order, and persistence relies on it — so the <strong>same CSV imported twice produced two different <code>.improv</code> files</strong> and two different row orders on screen.</p>
<p>The answer is first-seen order: deterministic, and the order the records themselves are in, which is the one a user can predict. Five runs is enough to see it, because each <code>HashSet</code> gets its own <code>RandomState</code>; the pre-fix code produced five different orders here.</p>
<h3 id="the-detected-date-format-does-not-depend-on-the-hash-seed"><a class="anchor" href="#the-detected-date-format-does-not-depend-on-the-hash-seed">#</a>the_detected_date_format_does_not_depend_on_the_hash_seed</h3>
<p>The consequence of the same bug that a user would have found hardest to explain. <code>detect_date_format</code> tests only the first ten samples, so when the sample order was a hash order, <em>which</em> ten it saw varied — and with it the detected format, whenever the eleventh-and-later values are the ones that rule a format out.</p>
<p>Twelve values are ambiguous and one (<code>25/12/2025</code>) is day-first only. In hash order it landed inside the first ten about half the time, so the column imported as <code>%d/%m/%Y</code> on some runs and <code>%m/%d/%Y</code> on others — every date in it off by the difference. In first-seen order it is last, deliberately: the answer is <code>%m/%d/%Y</code> <strong>stably</strong>, which is the property asserted. The ten-sample truncation itself is unchanged and is still a real limit.</p>
<hr />
<h2 id="improvise-io-src-import-csv-parser-rs"><a class="anchor" href="#improvise-io-src-import-csv-parser-rs">#</a><code>improvise-io/src/import/csv_parser.rs</code></h2>
<h3 id="short-rows-pad-missing-trailing-columns-with-null"><a class="anchor" href="#short-rows-pad-missing-trailing-columns-with-null">#</a>short_rows_pad_missing_trailing_columns_with_null</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-k8i" title="CSV import silently drops columns when row widths mismatch">improvise-k8i</a>. The reader is configured <code>.flexible(true)</code>, so rows shorter than the header parsed without error — but the missing trailing columns were simply <strong>absent</strong> from the record map. No error, no warning, and field-kind analysis saw skewed value counts. Short rows must keep every header column, missing ones as <code>Null</code>.</p>
<h3 id="short-rows-produce-warning-naming-column-and-count"><a class="anchor" href="#short-rows-produce-warning-naming-column-and-count">#</a>short_rows_produce_warning_naming_column_and_count</h3>
<p>Companion to <a class="tid" href="../tickets.html#improvise-k8i" title="CSV import silently drops columns when row widths mismatch">improvise-k8i</a>: the warning must name the affected column and say how many rows were short. A warning that said only &quot;some rows were short&quot; would pass a weaker test and be useless to the user.</p>
<h3 id="full-rows-produce-no-warnings"><a class="anchor" href="#full-rows-produce-no-warnings">#</a>full_rows_produce_no_warnings</h3>
<p>The negative arm: padding must not fire on a complete file.</p>
<h3 id="a-leading-utf8-bom-is-not-part-of-the-first-header"><a class="anchor" href="#a-leading-utf8-bom-is-not-part-of-the-first-header">#</a>a_leading_utf8_bom_is_not_part_of_the_first_header</h3>
<p>A header name becomes a <strong>category name</strong> and survives into the <code>.improv</code> file, so a byte-order mark riding along invisibly is silent corruption rather than cosmetics. Excel writes one; the <code>csv</code> crate strips a <em>leading</em> one, so the first header is <code>Region</code> and not <code>\u{feff}Region</code>.</p>
<p>The control at the end is what makes this an assertion rather than a tautology: a BOM in the <em>second</em> header is not leading, is not stripped, and shows up in the key — so the check demonstrably can see a BOM where there is one.</p>
<h3 id="crlf-line-endings-leave-no-carriage-return-in-the-data"><a class="anchor" href="#crlf-line-endings-leave-no-carriage-return-in-the-data">#</a>crlf_line_endings_leave_no_carriage_return_in_the_data</h3>
<p>The <code>\r</code> belongs to the terminator, not to the last field of the row: a <code>Revenue</code> of <code>1\r</code> would be a string rather than a number, and every row would import as text.</p>
<p>The control: a <strong>lone</strong> <code>\r</code> is itself a record terminator to this reader, so <code>Ea\rst</code> splits one row into two. That is what a stray carriage return does when it is not part of a CRLF pair, and it is why &quot;two records, clean values&quot; is a real observation rather than a coincidence.</p>
<h3 id="an-empty-file-is-no-records-and-no-warnings"><a class="anchor" href="#an-empty-file-is-no-records-and-no-warnings">#</a>an_empty_file_is_no_records_and_no_warnings</h3>
<p>An empty file is no records and no complaint. The wizard&#39;s own preconditions (<code>wizard.rs::validation_error</code>) are what refuse to build a model from nothing; the parser&#39;s job is to say truthfully that there was nothing there.</p>
<h3 id="a-header-with-no-rows-is-no-records"><a class="anchor" href="#a-header-with-no-rows-is-no-records">#</a>a_header_with_no_rows_is_no_records</h3>
<p>The same answer, and the case a user actually hits — an export filtered down to nothing.</p>
<h3 id="a-single-column-file-yields-one-field-per-record"><a class="anchor" href="#a-single-column-file-yields-one-field-per-record">#</a>a_single_column_file_yields_one_field_per_record</h3>
<p>The degenerate shape for everything downstream that splits a row into coordinates and a measure: one field, present, and <strong>not</strong> mistaken for a short row.</p>
<hr />
<h2 id="improvise-io-src-import-wizard-rs"><a class="anchor" href="#improvise-io-src-import-wizard-rs">#</a><code>improvise-io/src/import/wizard.rs</code></h2>
<h3 id="wizard-previous-reverses-each-advance"><a class="anchor" href="#wizard-previous-reverses-each-advance">#</a>wizard_previous_reverses_each_advance</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a> (from <a class="tid" href="../tickets.html#improvise-o1c" title="Import wizard has no way back; Esc discards all field decisions silently">improvise-o1c</a>): no step bound a back key, so reaching a later step and wanting to change a field kind meant Esc — discarding every proposal toggle, formula and name. <code>previous</code> walks back through the path <strong>actually taken</strong>, including the steps a conditional transition skipped on the way forward, which is why it is a history stack and not a reverse table.</p>
<h3 id="wizard-build-parks-the-workbook-on-done"><a class="anchor" href="#wizard-build-parks-the-workbook-on-done">#</a>wizard_build_parks_the_workbook_on_done</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a> (from <a class="tid" href="../tickets.html#improvise-c6j" title="WizardStep::Done is unreachable from the UI">improvise-c6j</a>): <code>advance</code> mapped <code>NameModel</code> → <code>Done</code> but nothing ever called it there, so <code>Done</code> was reachable <strong>only from tests</strong>. <code>build</code> now parks the workbook on <code>Done</code> for confirmation.</p>
<h3 id="wizard-build-failure-stays-on-name-model"><a class="anchor" href="#wizard-build-failure-stays-on-name-model">#</a>wizard_build_failure_stays_on_name_model</h3>
<p>A build that fails despite the earlier gates leaves the user on <code>NameModel</code> with the error, rather than moving to a <code>Done</code> step with nothing to confirm.</p>
<h3 id="wizard-blocks-advance-when-over-category-limit"><a class="anchor" href="#wizard-blocks-advance-when-over-category-limit">#</a>wizard_blocks_advance_when_over_category_limit</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-mzv" title="Import wizard does not pre-check 12-category limit">improvise-mzv</a>: with more than <code>MAX_CATEGORIES</code> (12) accepted Category proposals, the wizard let the user advance through every remaining step and <code>build_model()</code> only failed at the final confirm. Advancing past <code>ReviewProposals</code> must be blocked, with a message telling the user to mark fields as Measure or toggle them off — the actions still available on that step.</p>
<h3 id="wizard-blocks-advance-when-no-category-accepted"><a class="anchor" href="#wizard-blocks-advance-when-no-category-accepted">#</a>wizard_blocks_advance_when_no_category_accepted</h3>
<p>Bug <a class="tid" href="../tickets.html#improvise-907" title="Import wizard step machine: validate early, allow going back, make the terminal step real">improvise-907</a> (from <a class="tid" href="../tickets.html#improvise-gfq" title="Import wizard defers the &#39;no category accepted&#39; failure to the very last keystroke">improvise-gfq</a>): <code>build_model</code> raised &quot;At least one category must be accepted&quot; only on Enter at <code>NameModel</code>, the final step. An import whose fields all classify as Measure walked the user through <code>DefineFormulas</code> and <code>NameModel</code> before refusing. The gate belongs at <code>ReviewProposals</code>, where <code>c</code> can still change a field&#39;s kind — the same treatment <a class="tid" href="../tickets.html#improvise-mzv" title="Import wizard does not pre-check 12-category limit">improvise-mzv</a> gave the too-many-categories case.</p>
<h3 id="build-model-null-category-value-is-missing-not-the-item-null"><a class="anchor" href="#build-model-null-category-value-is-missing-not-the-item-null">#</a>build_model_null_category_value_is_missing_not_the_item_null</h3>
<p><a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item &quot;null&quot; on import">improvise-ych</a>. A category value of JSON <code>null</code> used to be stringified by the <code>v.to_string()</code> fallback, <strong>inventing an item literally named &quot;null&quot;</strong> and placing the record under it. A CSV empty cell parses to <code>Value::Null</code> too, so every blank column value in a CSV import grew a phantom &quot;null&quot; item. Null means <em>absent</em>: the record must be skipped and counted, exactly as if the key were missing.</p>
<h3 id="build-model-literal-string-null-is-a-real-category-item"><a class="anchor" href="#build-model-literal-string-null-is-a-real-category-item">#</a>build_model_literal_string_null_is_a_real_category_item</h3>
<p>The other side of <a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item &quot;null&quot; on import">improvise-ych</a>: the <em>string</em> <code>&quot;null&quot;</code> is an ordinary value — a region really called &quot;null&quot; — and must survive as its own item. Only <code>Value::Null</code> means absent. Without this arm, &quot;drop anything that stringifies to null&quot; would pass the test above.</p>
<h3 id="build-model-empty-string-category-value-is-missing"><a class="anchor" href="#build-model-empty-string-category-value-is-missing">#</a>build_model_empty_string_category_value_is_missing</h3>
<p><a class="tid" href="../tickets.html#improvise-ych" title="Null category values become the literal item &quot;null&quot; on import">improvise-ych</a>, third arm. An empty string is the JSON spelling of a blank cell — the CSV parser already turns a blank field into <code>Value::Null</code>, so the two inputs must agree. An item with an empty name is unnameable in the grid, so it is treated as missing rather than created.</p>
<h3 id="build-model-warns-about-records-skipped-for-missing-category"><a class="anchor" href="#build-model-warns-about-records-skipped-for-missing-category">#</a>build_model_warns_about_records_skipped_for_missing_category</h3>
<p><a class="tid" href="../tickets.html#improvise-5j9" title="ImportPipeline silently skips records missing category values">improvise-5j9</a>: skipping a record for a missing category value used to be silent, so a 1000-row import could drop rows with no user-visible sign. The warning must name the count, the total, and the offending field(s) — a bare &quot;some records were skipped&quot; is not actionable.</p>
<h3 id="is-does-not-swallow-the-start-of-a-longer-word"><a class="anchor" href="#is-does-not-swallow-the-start-of-a-longer-word">#</a>is_does_not_swallow_the_start_of_a_longer_word</h3>
<p>Adversarial review of PR #54, low severity and fixed anyway. <code>is_expr</code> was <code>identifier ~ ^&quot;IS&quot; ~ filter_value</code>, so <code>Region ISLAND</code> parsed as <code>Is(Region, &quot;LAND&quot;)</code> and <code>Region isotope</code> as <code>Is(Region, &quot;otope&quot;)</code> — inputs that were <strong>parse errors</strong> before the production existed, now silently-wrong formulas. Gotcha 34&#39;s whole theme is that a silently wrong parse costs more than a refusal.</p>
<p>The repair went through two wrong shapes before the right one, and both are worth keeping. A negative lookahead <code>^&quot;IS&quot; ~ !(ASCII_ALPHANUMERIC | …)</code> in an ordinary rule <strong>cannot work</strong>: pest auto-inserts <code>WHITESPACE</code> between sequence elements, so it tested the character <em>after</em> the space — rejecting <code>Region IS East</code> while still accepting <code>Region ISLAND</code>, i.e. exactly backwards. Making <code>is_expr</code> compound-atomic (<code>$</code>) stops the auto-insertion and forces the separator to be written out as <code>sp+</code>.</p>
<p>At which point <strong>the lookahead was redundant and was removed</strong>, which only re-injecting the fault revealed: deleting <code>!bare_char</code> left the suite green, because <code>sp+</code> after <code>IS</code> already refuses <code>ISLAND</code> — <code>IS</code> matches, then <code>sp+</code> demands whitespace and finds <code>L</code>. A guard nobody has seen fail is indistinguishable from one that always passes (<code>doctrine(check_the_checker)</code>), so it went. <strong><code>sp+</code> is now the mechanism</strong>, and relaxing it to <code>sp*</code> — the obvious &quot;let people write <code>Region IS|East|</code>&quot; change — reopens the hole; that substitution is what this test was finally proved against. The negative arm is paired with a positive one, or &quot;reject everything&quot; would pass.</p>
<p><code>^&quot;WHERE&quot;</code> had the identical hole. It is closed now — see below.</p>
<h3 id="where-does-not-swallow-the-start-of-a-longer-word"><a class="anchor" href="#where-does-not-swallow-the-start-of-a-longer-word">#</a>where_does_not_swallow_the_start_of_a_longer_word</h3>
<p><a class="tid" href="../tickets.html#improvise-r08u" title="IS and WHERE differ on trailing word boundary in formula.pest">improvise-r08u</a>, the <code>WHERE</code> half of the same defect. <code>where_clause</code> and <code>inline_where</code> were ordinary rules, so pest auto-inserted <code>WHITESPACE</code> after <code>^&quot;WHERE&quot;</code> and <code>SUM(Rev WHEREAS x = y)</code> parsed as a <code>WHERE</code> on a category <code>AS</code>: a silently wrong filter where it should have been a syntax error.</p>
<p>The test carries <strong>two</strong> negative shapes because the repair has two independent halves, and this was found by injecting each separately. Making the rules compound-atomic alone already refuses <code>WHEREAS x = y</code> — once implicit whitespace is gone, <code>AS x</code> cannot be one identifier — so a test built only on the ticket&#39;s reproducer passes over a rule still spelled <code>sp*</code>, and <code>WHEREAS=y</code> is what that rule still reads as a filter. Faults injected: the original non-atomic rule (fails on the first assertion) and <code>sp+</code> relaxed to <code>sp*</code> (fails on the second). The positive arms are there so &quot;refuse every WHERE&quot; cannot pass.</p>
<h3 id="where-requires-a-space-after-the-keyword"><a class="anchor" href="#where-requires-a-space-after-the-keyword">#</a>where_requires_a_space_after_the_keyword</h3>
<p>The one spelling <code>sp+</code> costs, stated rather than discovered later: <code>WHERE|Region Name|=|East Coast|</code> used to parse and now does not. <code>IS</code> has required its space since <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>, so this is the two keywords agreeing. The paired positive arm — the same text with the space — is what stops it degrading into &quot;any WHERE with a quoted category fails&quot;.</p>
<h3 id="the-grammar-is-parsed-once-however-many-files-the-strategy-yields"><a class="anchor" href="#the-grammar-is-parsed-once-however-many-files-the-strategy-yields">#</a>the_grammar_is_parsed_once_however_many_files_the_strategy_yields</h3>
<p><a class="tid" href="../tickets.html#improvise-8shu" title="Constrain the .improv grammar generator&#39;s strategy; improvise-io&#39;s lib binary is the run&#39;s outlier">improvise-8shu</a>. <code>improvise-io</code>&#39;s lib binary was the run&#39;s outlier — 1.42s here against a 0.01s median, measured at 1.40–1.45s, 1.50s and 3.70–5.24s on three machines. The ticket&#39;s diagnosis was that the grammar walk had no size bound, so a few deeply nested documents dominated. <strong>That was wrong</strong>, and measuring before fixing is the only reason it did not ship as the fix: instrumenting the two halves separately gave <code>load_grammar()</code> ×1000 = <strong>2.50s</strong> and the walk itself ×1000 = <strong>15.7ms</strong>. The walk was 0.6% of the cost. <code>improv_file()</code> called <code>load_grammar()</code> inside its <code>prop_map</code>, so <code>pest_meta</code> re-parsed <code>improv.pest</code> once for every generated case — 1000 times across the two properties, plus shrink attempts.</p>
<p>The strategy&#39;s input space was never the problem and is unchanged: the same entropy range, the same 0..N repetitions, the same properties over groups, formulas, views, hidden items and pipe quoting. The grammar is now read once through a <code>OnceLock</code>.</p>
<p>The guard counts <em>parses</em> rather than calls, which is what makes it order-independent: other tests in the binary reach <code>grammar()</code> too, and the count stays 1 however many of them run first. It drives <code>improv_file()</code> itself through a <code>TestRunner</code> rather than re-implementing what the strategy does, so a future edit that reverts the call site is what it catches. Fault injected: <code>Gen::new(grammar(), …)</code> restored to <code>let rules = load_grammar();</code> — the test fails with <code>left: 64, right: 1</code>.</p>
<p>Case counts went 500 → 256 in the same file, which is the default <a href="../design-principles.html"><code>context/design-principles.md</code></a> §7 prescribes and never below it; that is worth a further 0.39s → 0.21s but it is not the fix, and reporting it as one would be the mistake this entry exists to record. Net: 1.42s → 0.21s, and <code>improvise-io</code> is no longer the run&#39;s longest binary.</p>
<h3 id="export-csv-omits-the-rows-the-view-prunes"><a class="anchor" href="#export-csv-omits-the-rows-the-view-prunes">#</a>export_csv_omits_the_rows_the_view_prunes</h3>
<p><a class="tid" href="../tickets.html#improvise-iot6" title="render_csv builds GridLayout::new, so CSV export carries rows the view prunes">improvise-iot6</a>. <code>render_csv</code> built its layout with <code>GridLayout::new</code>, which does <strong>not</strong> apply <code>prune_empty</code> — only <code>GridLayout::with_frozen_records</code> does. So <code>:export</code> in the TUI and the CLI export path both wrote rows the grid does not draw. User-visible, in production, and shipping for as long as the two constructors have differed.</p>
<p>This is the <strong>fourth</strong> instance of the same trap; the third was <code>project::grid_in</code>, fixed in PR #78, and this fix is that one line for line. No fault needed injecting: on the unfixed tree the test fails with</p>
<blockquote><p>CSV wrote a row the grid does not draw: left <code>[&quot;Food&quot;, &quot;Gas&quot;, &quot;Rent&quot;]</code>, right <code>[&quot;Food&quot;, &quot;Gas&quot;]</code></p></blockquote>
<p><strong>Why the fixture is discriminating, which is the whole difficulty here.</strong> The sibling property written for <a class="tid" href="../tickets.html#improvise-t6db" title="grid_in ignored prune_empty for a non-active view, hiding a staleness bug">improvise-t6db</a> passed on the buggy tree in its first version, because it made its two views differ by turning pruning <em>off</em> — plenty of differing config, and vacuous, since with nothing to prune both code paths agree. So this fixture does two things that one cannot fake:</p>
<ul>
<li>it contains a row that is <strong>actually empty</strong> (<code>Rent</code>, an item of <code>Type</code> that no cell names) and leaves <code>prune_empty</code> <strong>ON</strong>;</li>
<li>it asserts the precondition in both directions before exporting anything — <code>with_frozen_records</code> draws 2 rows and <code>new</code> holds 3. If a future change makes pruning a no-op, or makes the two constructors agree, the precondition fails and says so rather than the assertion passing vacuously.</li>
</ul>
<p>None of <code>export.rs</code>&#39;s three pre-existing tests mentions pruning at all, which is exactly why this shipped.</p>
<p><strong>Both axes, because <code>prune_empty</code> prunes both.</strong> A first version of this fixture had a single <code>Month</code>, so it demonstrated the row half and left the column half asserted by nothing -- the same defect one axis over. <code>Feb</code> is now the empty column, the precondition is a pair on each layout ((2, 1) drawn against (3, 2) unpruned), and the CSV header is asserted to carry <code>Jan</code> alone.</p>
</main>
</div>
</body>
</html>