- 8jk6.4: the CSV parser yields source text (CsvRecords), and only the
role reads it as a number. A forced category keeps 00565796 and
220944e2 verbatim; measures and analysis read the text through
csv_parser::number_in. CsvRecords reaches a pipeline only through
ImportPipeline::from_csv, so handing CSV text to the JSON constructor
no longer compiles.
- 8jk6.5: an import formula that does not parse is refused by name with
its parse error: the build refuses (headless exits non-zero and writes
nothing), and the wizard keeps it out of the list with the text still
in the buffer.
- 8jk6.6: an import that kept no record refuses and says why, from the
build and from the wizard's review step. A partial skip stays a
warning with a count and exit 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
When source records collide on every coordinate, the import wizard now asks:
sum them, keep them separate with a generated occurrence category, or go
back. Collisions are found before the build by the same placement pass the
build uses, and are carried to the TUI as structured data (WizardCache) and
listed in full, scrollable. At the 12-category cap, "keep separate" is shown
disabled with its reason and remedy.
Every wizard warning and message now wraps to the panel instead of being
truncated, and long lists scroll.
Headless paths no longer sum silently: `import --no-wizard` takes
`--on-duplicate refuse|sum|separate` (default refuse), and `:import-json`
refuses and points at `:import`.
Reasoning: context/code/formula-and-io.md, "Colliding records are a
question, not a default"; context/code/tui.md; context/code/cli.md §11.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
improvise-e0ex. `Margin = Profit / Revenue` drew as `1`: `View::new` seeded
`,.0` — zero decimals — so a 0.66 ratio rounded half-away-from-zero. The cause
is the format, not the arithmetic; exact decimal math would not change the
rendering by one character, so nothing here touches the evaluator and gotcha 8
is unchanged.
Two halves, per the user's ruling:
* `View::new` seeds `,.2`. Revenue draws `18,400.00` and Margin `0.66`. The
noise on whole numbers is the cheaper error — a currency column with two
redundant zeroes is legible, a ratio with none is wrong — and a format cannot
be guessed from a value.
* A formula may carry its own format, which wins for its target; a raw measure
keeps the view's. `formula.pest` gains
`format_clause = ${ ^"FORMAT" ~ sp+ ~ format_spec }`, with `sp+` for `WHERE`'s
reason (a predicate is invisible to the grammar-walking generator) and
`FORMAT` added to `quote::KEYWORDS`.
`improv.pest` is deliberately unchanged: `formula_raw` already carries the
`## Formulas` line verbatim, so the format rides in the formula text with
nothing to add — and the generator, which reads `improv.pest`, sees no new
spelling to violate.
Resolution has one home. `FormatPlan::of(model, view)` replaces the
`(comma, decimals)` pair every render path threaded — `GridLayout::display_text`,
`viewport::grid_body`/`compute_col_widths`, `project::grid_in` and
`persistence/export.rs::render_csv` — rather than being added beside it, which
would be two answers to "what format is this cell". It is asked of the cell's
`CellKey`, so a cell aggregating over `_Measure` keeps the view's format by
construction. `GridBody::col_formats` carries the answer to the totals strip,
which is formatted on its own path.
Persistence is untouched except for metadata: `format_f64` is display,
`persistence/format.rs::format_number` writes round-trip precision, and the two
have no call between them. Newly created views write `format: ,.2`; existing
documents are unaffected, because `format_md` has always written the line
explicitly. `web/demo.improv` moves to `,.2` on both views.
The trap this sets is gotcha 34's: `format_md` writes `Formula::raw`, so a
`FORMAT` suffix round-trips whether or not the walker parsed it. The round-trip
test therefore asserts the parsed field — injecting a walker that drops the
clause fails that one test and nothing else in `improvise-io`, 242 tests
including the whole characterization suite and the grammar properties.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Self-review with the code-review skill, against kb.pl's doctrine. All three are
tests that passed while the thing they named was not pinned.
F1. `doctrine(inject_the_rival_fix_not_only_the_bug)`. The jcfs test used one
twelve-value fixture, and `take(100)` — raising the cap rather than removing it
— passed the ENTIRE suite. That pins a behaviour, not the rule: the rule is that
detection does not depend on how many values there are, and any fixed fixture
pins one side of some cap. It now runs at 11 and 500 leading values, so every
`take(N)` fails one arm. Renamed to say what it asserts.
F2. `doctrine(an_injection_that_does_not_fail_names_an_untested_path)`.
`an_all_null_column_is_a_label` was never proved able to fail. Under the l1f5
fault it PASSES — an all-null column was Label before the fix too — so its
neighbours' injection says nothing about it. Injecting its own fault (dropping
the `!present.is_empty()` conjunct) fails it alone. The rival fix for l1f5 was
also injected: `is_number() || is_null()` over the full slice repairs Revenue
and leaves Region broken, which is why both `all_` bindings had to move and
`a_blank_value_does_not_demote_a_string_column` is the separating input.
F3. `doctrine(an_order_test_needs_two_histories)`. The cell-order test used one
insertion history five times, which cannot tell "listed its own order" from
"emitted one canonical order that matches this fixture". Same six regions now
imported forwards and backwards, each asserting its own. Reaching the second
direction needed the loop re-ordered under the fault, since an injection that
fails at iteration 1 leaves iteration 2 untested.
Also: `duplicate_csv_headers_not_silent` asserted its two values by map
ITERATION ORDER; now by key, which is also what it means.
Four controls were verified able to fail rather than assumed:
an_ordinary_float_is_still_a_number (float branch deleted),
distinct_headers_are_not_renamed_and_warn_nothing (rename every header),
an_all_null_column_is_a_label (F2), and the long-row control.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-pnf5 (P1). A cell keyed by the empty CellKey is a legal model state
-- what AddRecordRow leaves while a records row is unfinished -- and format_md
wrote it as " = 0" while coord_list required at least one coordinate. Saving
in records mode therefore overwrote a good document with one the application
could not reopen. Given a spelling rather than discarded, because the ticket
is explicit that legitimate data must not be dropped: data_coords makes the
coordinate list optional and the scalar cell is written "= 42". Unambiguous
under PEG, since bare_name cannot start with "=".
improvise-m4rv. Workbook::new seeds a Default view and on_category_added
fills it, so a file naming only Plan parsed into two views -- the phantom at
index 0, and appended to the user's document on any save. parse_md now drops
Default when the file named at least one view and none was Default. A file
naming no view, or naming Default itself, is untouched.
improvise-3yv3. format_md collected every ungrouped item and emitted the
whole list before any grouped one, so an interleaved category came back
reordered -- and grid axis ordering consumes item insertion order (gotcha 18).
Items are now written in insertion order, a run of bare items sharing a line.
improvise-qu4a. View::prune_empty never reached disk. The line is spelled
with a value rather than by its presence, so the grammar generator can emit
either; absent means false, the compatible reading.
improvise-ppru. Model::measure_agg was lost by the roundtrip. An Aggregation
section carries it, sorted -- measure_agg is a HashMap and hash order is not
an order (gotcha 18), so two saves of one document must be made to agree.
improvise-4lgj. The three wildcard arms in the walker silently skipped any
grammar rule they did not handle, so extending improv.pest caused silent data
loss rather than a loud mismatch. They now bail, and a new exhaustive match
over Rule names the walker that reads each one -- a new rule stops that test
compiling, which is design-principles section 2's "the compiler finds every
call site" applied to a pest-derived enum. It earned its keep immediately by
naming Rule::section, which the first draft had missed.
That arm is also what makes the change safe: the grammar-walking generator
reaches both new rules -- proved by injecting a bail into each and watching
generated_file_parses fail -- so an unwalked rule is now a test failure.
Un-ignores the five reproductions in tests/bug_hunt_reproductions.rs.
improvise-nvq4 did NOT reproduce and is kept as a passing control.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
The five reproductions for improvise-jc95, 5hfh, l1f5, 2cw9 and the long-row
half of k8i now run. Removing `#[ignore]` is part of the fix: an ignored
reproduction is an open bug, and the module doc now says so rather than claiming
the file is report-only.
Three of the five asserted warning PROSE against a module that emitted no
warnings at all, so they specified wording with no convention to match — and
`"overwritten"` does not contain `"overwrite"`, so the most natural wording
would have failed its own assertion. Each now asserts the OBSERVABLE outcome
first and the prose only as a companion:
- duplicate_csv_headers_not_silent: three fields with values 1, 2, 3 in order.
- repeated_import_coordinates_not_silently_overwritten: exactly one cell at 30.0.
- long_csv_row_is_not_silently_dropped: no row discarded, plus a warning that
counts the affected rows and names the widest, because the values here really
are still dropped and there is no observable outcome to assert instead.
Every fix was verified by injecting its own fault and watching the test fail:
the `unwrap_or(0)` restored, `disambiguate_headers` short-circuited, the
long-row counter disabled, both `all_` bindings restored to the full slice,
`take(10)` restored, the accumulate arm changed back to last-wins, and the cell
write order sorted. Each named test failed; each control did not.
Evidence in context/test-evidence/formula-and-io.md, keyed by test name,
including the correction to `the_detected_date_format_does_not_depend_on_the_hash_seed`
— its fixture had made the ten-sample truncation a specification.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-s08t: eval_agg narrowed its partial key only where
find_item_category resolved, so SUM(Typo) left the key unconstrained and
aggregated every numeric measure matching the context. An unknown name now
errors ref:{name} before the raw fast path.
improvise-s2w3: the same fast path assumed a bare Ref names a STORED cell. A
formula target has no raw cell behind it (improvise-nrs), so SUM(Double)
summed nothing and answered 0 where SUM(Double + 0) folded and answered 20.
core.md §13's defence of the specialisation holds — what failed is its
premise — so eval_agg asks formula_for, the one home of "is this cell
derived", and a derived ref falls into the fold.
improvise-7iww: recompute_formulas stopped at MAX_EVAL_DEPTH and left the last
iteration in formula_cache, so a circular formula read Error("circular")
directly and Number(26) through the cached path — which is the DISPLAYED path,
since default views put the virtual categories on Axis::None. Cells still
moving when the bound runs out are now marked circular, which is the same
bound evaluate_depth already enforces.
improvise-vyrx: eval_bool_cached wildcarded BinOp where eval_bool lists its
arms. Verified by adding a BinOp::Mod: four sites now fail the build where
three did before.
improvise-pikk: round_half_away scaled by 10^decimals before truncating, so
format_f64(1e307, false, 2) rendered "inf" for a finite number.
improvise-f86q: add_category checked MAX_CATEGORIES before contains_key, so
re-adding an existing name at the limit was refused with a false message — and
per gotcha 21 aborted the rest of the batch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-yae4. grid_body narrowed every measured width to u16 and only then
added the one-character gap and clamped to MAX_COL_WIDTH, so the arithmetic ran
at full width before the cap that makes it safe. A stored 65535-column text
value panicked at `w + 1` in debug; in release the sum wrapped to 0 and the
column was drawn at MIN_COL_WIDTH — the narrowest column, for the widest value.
A 65536-column value never reached the addition: `as u16` truncated it to 0
first, so it wrapped in debug builds too, silently.
Widths are now capped while still usize, in the one place every measurement
enters (`widen`), which is why header_level_seed folds through it rather than
keeping a second `as u16`. The answer is unchanged for every input that did not
overflow: (min(w,32) + 1).clamp(5,32) and (w + 1).clamp(5,32) agree everywhere,
both saturating at 32 for any w >= 31.
improvise_protocol::header_level_widths carried the identical defect on the
row-header axis — a 65535-column item label — and is fixed the same way. It is
the one home for that rule (gotcha 44), so the row header and the column widths
cannot end up with two answers.
wide_cell_clamps_without_overflow is un-ignored and extended to the 65536 arm
and to an ordinary 20-column value. It asserts `widths == vec![expect]` rather
than `all(<= 32)`: the length pins that the fixture really produces ONE column,
and the ordinary arm pins that the single column is the one carrying the value —
a fixture in which no column was wide would read MIN_COL_WIDTH there and pass a
`<= 32` check. Both injected faults are recorded in
context/test-evidence/core.md.
Not fixed, and out of this ticket's scope: compute_visible_cols accumulates
`acc + w` in u16, reachable only from a terminal ~65535 columns wide.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Expand the warning pattern matching to include various forms of the
overwrite warning, such as 'overwritten', 'overwriting', and 'overwrote',
while still ensuring the affected coordinates are present.
Co-Authored-By: fiddlerwoaroof/git-smart-commit (unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL)
Refine assertions in reproduction tests to be more specific and ensure they
correctly validate expected error messages and warnings.
This includes improvements to:
- duplicate_csv_headers_not_silent
- repeated_import_coordinates_not_silently_overwritten
- long_csv_row_is_not_silently_dropped
- wide_cell_clamps_without_overflow
Co-Authored-By: fiddlerwoaroof/git-smart-commit (unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL)
These tests reproduce known bugs and are marked with #[ignore] to prevent
them from failing standard test runs.
They can be run explicitly using 'cargo test --test bug_hunt_reproductions
-- --ignored --nocapture'.
Co-Authored-By: fiddlerwoaroof/git-smart-commit (unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL)
Preserves a collection of test cases designed to reproduce specific bugs in
the `improvise` and `improvise_io` libraries. This includes issues related
to:
- CSV parsing
- Non-finite values
- Circular formulas
- UI overflow from large strings
Twenty tests in this file are intentionally designed to fail to reproduce
known unfixed bugs, while two control tests are expected to pass.
Co-Authored-By: fiddlerwoaroof/git-smart-commit (unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL)
Claude (imp321)