A collapsed group is written to CSV as its own total row, because an export
writes what the grid draws (improvise-iot6). core.md §14 carries the argument;
the evidence entries record each rival and where it failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
`Repr::Nonfinite(&'a str)` could only be produced by a deserializer that
borrows from its input, so the new wire spelling round-tripped through
`from_str` and `from_slice` and through nothing else. Measured on the
pre-fix head:
from_str {"Number":"inf"} -> Ok
from_str {"Number":"inf"} -> Err (the SAME JSON document)
to_value then from_value -> Err (does not round-trip its own output)
from_reader -> Err
The escaped form is the serious half: escaping is not a distinction JSON
makes, so the wire accepted a subset of the encodings of the spelling it
claims to share with the file. `Cow<'a, str>` fixes all four; the serialized
form is unchanged.
The PR's own tests could not see this — both use `from_str` exclusively,
which is precisely the one path that borrows. A check adjacent to the
question it names, inside the change that defines the question.
Also tightened `a_string_that_is_not_one_of_the_three_tokens_is_refused`: its
second disjunct accepted `data did not match any variant`, which is exactly
the message the above bug produces, so it passed in both worlds.
Conflict was `context/test-evidence/core.md` only, against #143. Both entries
kept and verified line-by-line — neither test cites the file, so a dropped
entry would pass `--check-evidence`.
Gates on the merge: 1728 passed / 0 failed / 7 ignored over 38 binaries,
clippy -D warnings clean, fmt clean, and `cargo check --target
wasm32-unknown-unknown -p improvise-core` clean — the one gate the review
asked for and could not run.
Found by adversarial review (confirmed by injection, not reasoned).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
serde_json writes a non-finite f64 as null, and null does not read back as one,
so CellValue::Number(INFINITY) serialized to {"Number":null} and the frame
carrying it was unreadable. The websocket reader skips an unparseable frame and
the worker logs a console error, so one poisoned cell silently dropped a whole
batch — and to_string returned Ok, so the send site could not notice either.
The ticket offered two answers and the format had already chosen. Measured
before changing anything: inf, -inf and nan all round-trip losslessly through
.improv — format_number writes them, the grammar's bare_value accepts them, the
walker maps them back. So refusing them before sending would make a legal saved
document unprojectable to a browser, which is a new failure rather than a fix.
It also settles the ticket's second location: resolve.rs's typed() accepting inf
is correct, not half a defect.
Number now serializes through an untagged adaptor: a JSON number when finite,
one of the three tokens when not. A finite cell's bytes are unchanged, and a
JSON integer still reads as a number rather than as a token.
The tokens are one home — nonfinite_token/parse_nonfinite beside CellValue,
called by the adaptor and by persistence/{format,parse}.rs. Three copies stood
before and a fourth was about to be written. format_f64 is deliberately not
folded in: it is the display layer and these three are a format.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
`assert!(val >= 1.0)` on a fixture whose answer is exactly 1.0 cleared the bar
for every inflated count. The ticket's fault (`len + 999.0`) is not the worst
case: it moves the empty count from 0 to 999 and so fails
eval_agg_over_no_rows_is_zero_only_where_an_identity_exists. Preserve the empty
case and all 1699 workspace tests pass while COUNT over any non-empty set is
wrong by 999.
Two assertions, because one does not discriminate. The exact 1.0 at Region=East
fails the +999 rival; a Count arm returning a constant 1.0 satisfies it, so the
unscoped COUNT over all three regions being 3.0 is what fails that one. Both
rivals injected; both fail, at different assertions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-vw47. Create View 2 and View 3, configure View 3, delete View 2,
then ask for a new view: both resolvers spelled `format!("View {}", len + 1)`,
which is the count and not a free name. `Workbook::create_view` replaces
whatever holds the name, so View 3's format reset to defaults and no view was
added -- a symptom that looks like nothing happened.
The name is derived at two sites, `cmd/panel.rs::CreateAndSwitchView` and
`resolve.rs`'s "create-and-switch-view" arm, so fixing one would have left the
TUI and the browser naming different views. `view::unused_view_name` in
improvise-core is the one derivation both now call. It takes an iterator of
names because the browser holds no `Workbook`, only the `ViewListCache` that
`view_summaries` maps one-for-one off `workbook.views`.
`Workbook::create_view` is deliberately untouched. Its replacement semantics is
what the `_Records`/`_Drill` rebuilders need -- both reach it through
`effect::CreateView` on every entry -- so refusing or renaming there would turn
"a view was reset" into "the key does nothing". The argument is in
`context/code/core.md` section 11.
Three tests, written failing first. The helper's own cases pin that a run of
taken names is searched rather than bumped once. The `Cmd` test uses the
ticket's fixture and asserts the count is below the highest name before asking
anything, because the pre-existing sibling test's fixtures all leave the
derived name free -- which is how the bug lived under it. `lockstep.rs`
witnesses the agreement: note that `both_ends` PASSES against the unfixed
source, both ends being wrong identically, so that test carries the count and
format assertions too.
Rivals injected: fixing panel.rs alone fails lockstep on the document text;
starting the search at 1 fails the pre-existing incremental-naming test.
Baseline 1667 passed / 0 failed / 7 ignored over 37 result lines; now 1670.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-cj4u (P2). View::on_category_added took a name and asked
CategoryKind::of_name, which reads Label as Regular by design.
Workbook::add_label_category papered over that with a set_axis(None)
straight afterwards, so views existing at that moment were right;
Workbook::create_view had only names to hand and had no such correction,
so every view created after a label pivoted it onto Row — and on an
already-pivoted document onto Page, filtering the whole document down to
one description.
on_category_added now takes the CategoryKind, and the two set_axis
corrections are deleted: one answer to "where does this category go"
rather than two, of which only one caller knew to write the second. The
signature change is deliberate — it is what made the compiler name every
call site. Every site outside Workbook is test code and passes
of_name(name), which preserves what those fixtures meant.
Shares a theme with improvise-dy3j — a Label kind that cannot cross a
boundary carrying only a name — but not a mechanism: dy3j is the file
format, this is a function signature. Fixed separately for that reason.
The control tests carry as much as the fix: a blanket `unseated = true`
satisfies every label assertion in the repository, so the direction the
bug cannot supply is asserted too, and a label is shown not to consume
the Row slot the next regular category wants. Reasoning in
context/code/core.md §9; evidence in context/test-evidence/core.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Both findings from ChatGPT Codex's review of PR #125. Both are real, and both
are the same mistake in two places: answering "whose format is this" from
something other than the rule that answers "whose value is this".
**1. `FormatPlan` indexed formats in a `HashMap` keyed by target.** A target
category need not be `_Measure` — `[CustomCat]` is in the format's own
documentation — so a key can carry coordinates two formulas claim, and
iterating the map answered from whichever category hashed first: the displayed
format was nondeterministic per process while the value was not. That is
gotcha 18's shape. Worse, only *formatted* formulas were indexed, so a later
formula's `FORMAT` reached a cell whose value an earlier unformatted formula
produces.
`FormatPlan` now holds every formula's `(target_category, target,
Option<format>)` in the model's own order and takes the first coordinate match
— `Model::formula_for`'s rule and order exactly. A match naming no format falls
through to the view; it is a match, not a miss. The list is dropped entirely
when no formula names a format, so the common document still pays one
`is_empty` per cell.
**2. A column total took its format from row 0.** `context/code/core.md` argued
that was safe because "where `_Measure` is on the row axis a column is not a
measure at all". Wrong: a formula's target category is any category, so a
formula can own a row-axis item — the total then inherited a row-local override
for a row it had summed away, and re-ordering the rows changed its spelling.
New `GridLayout::column_key(col)` gives the coordinates that survive the row
aggregation (page/filter constraints plus the column's tuple), and `col_formats`
is resolved from that, once per column, outside the row loop.
Three regression tests, each with the rejected version injected as the fault:
- `the_formula_that_owns_a_cell_names_its_format_even_when_it_names_none` puts
the same two formulas in both orders, so preferring the formatted one fails
one half and ignoring formats fails the other.
- `the_owner_is_the_one_model_order_names_and_not_the_one_a_hash_does` rebuilds
the plan five times, since each `HashMap` gets its own `RandomState`.
- `a_column_total_does_not_inherit_a_row_local_format` builds one grid with the
owned row first and one with it last and requires the strips to agree. Every
raw cell holds the same value per measure, so the totals are numerically
identical and only the format can differ.
`context/code/core.md`'s wrong paragraph is replaced rather than amended, and
says what it got wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-e0ex. `Margin = Profit / Revenue` drew as `1`: `View::new` seeded
`,.0` — zero decimals — so a 0.66 ratio rounded half-away-from-zero. The cause
is the format, not the arithmetic; exact decimal math would not change the
rendering by one character, so nothing here touches the evaluator and gotcha 8
is unchanged.
Two halves, per the user's ruling:
* `View::new` seeds `,.2`. Revenue draws `18,400.00` and Margin `0.66`. The
noise on whole numbers is the cheaper error — a currency column with two
redundant zeroes is legible, a ratio with none is wrong — and a format cannot
be guessed from a value.
* A formula may carry its own format, which wins for its target; a raw measure
keeps the view's. `formula.pest` gains
`format_clause = ${ ^"FORMAT" ~ sp+ ~ format_spec }`, with `sp+` for `WHERE`'s
reason (a predicate is invisible to the grammar-walking generator) and
`FORMAT` added to `quote::KEYWORDS`.
`improv.pest` is deliberately unchanged: `formula_raw` already carries the
`## Formulas` line verbatim, so the format rides in the formula text with
nothing to add — and the generator, which reads `improv.pest`, sees no new
spelling to violate.
Resolution has one home. `FormatPlan::of(model, view)` replaces the
`(comma, decimals)` pair every render path threaded — `GridLayout::display_text`,
`viewport::grid_body`/`compute_col_widths`, `project::grid_in` and
`persistence/export.rs::render_csv` — rather than being added beside it, which
would be two answers to "what format is this cell". It is asked of the cell's
`CellKey`, so a cell aggregating over `_Measure` keeps the view's format by
construction. `GridBody::col_formats` carries the answer to the totals strip,
which is formatted on its own path.
Persistence is untouched except for metadata: `format_f64` is display,
`persistence/format.rs::format_number` writes round-trip precision, and the two
have no call between them. Newly created views write `format: ,.2`; existing
documents are unaffected, because `format_md` has always written the line
explicitly. `web/demo.improv` moves to `,.2` on both views.
The trap this sets is gotcha 34's: `format_md` writes `Formula::raw`, so a
`FORMAT` suffix round-trips whether or not the walker parsed it. The round-trip
test therefore asserts the parsed field — injecting a walker that drops the
clause fails that one test and nothing else in `improvise-io`, 242 tests
including the whole characterization suite and the grammar properties.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
context/code/core.md gets the two model rules (why removing a name removes
the formulas targeting it, and why a required virtual category refuses) and
the axis rule (why leaving the constraining axes forgets the page selection,
and why the predicate is filters_data rather than 'is it Page').
context/code/clients.md gets improvise-koh6's decision in full -- including
why the state-dependence rule is about Upstream and not about outcomes --
plus the cursor rule and the shared axis sentence.
Evidence entries are module-qualified where they must be: three leaf names
are answered by a test in both resolve.rs and panel.rs, and one in both
model/types.rs and workbook.rs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Integration review of #106. Three findings; two applied, one does not reproduce.
1. AxisFrame::group_of had NO coverage of its GroupEnd arm. Replacing it with
`FrameEntry::GroupEnd => None` left all 133 improvise-protocol lib tests
passing — only tests/grid_frame_agreement.rs in the root crate caught it,
and that file does no pruning. So improvise-gd83's property reached the
protocol by construction through project.rs and never by assertion: a
derivation living in two crates with one of them unchecked, which is what
gotcha 44 and design-principles §5 exist to prevent.
Two tests in render.rs over the exact frames prune_axis_items now produces
— the re-opened run closed by a GroupEnd, and the collapsed header with no
item under it. The second also asserts collapsed: true travels, because a
header arriving without the flag is there and inert, which is lhi4 again
one layer out. The injection now fails both where it failed nothing.
This is the landed half of u6gc's AC3. The other half — that
prune_axis_items -> axis_frame -> group_of really composes to these frames
— still needs project.rs and stays filed rather than landed.
2. a_collapsed_groups_header_survives_pruning passed with prune_empty disabled
entirely: its fixture funded every item, so nothing was ever pruned and the
name claimed more than the test could see. It now carries an unfunded Mar,
so turning pruning off leaves DataItem(["Mar"]) in the asserted sequence and
the test fails. Same vacuity class as the any() -> whole-sequence fix in
4075d66.
3. NOT APPLIED — "no fault separates the collapsed test from the GroupEnd
test" does not reproduce. Re-measured all three faults over all four tests:
inverting close_pending_run's condition to `had_item` fails the collapsed
test while the GroupEnd test PASSES, because a re-opened run is closed by
the GroupEnd arm whatever the header condition says. The evidence now
carries the full 3x4 matrix rather than prose, so the claim is checkable
instead of assertable, and every column is distinct.
Also moves wide_cell_clamps_without_overflow's evidence from core.md to
protocol.md. improvise-yae4 touches viewport.rs and render.rs and does not
touch improvise-core at all, so core.md was the wrong home; it was put there
by hand and the checker cannot see a misfiled entry — it asks whether a
heading names a real test, not whether it sits in the right file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Self-review under the code-review skill, attacking my own three prune tests by
trying to build a wrong prune_axis_items that passes all of them.
The gap it found: all three fixtures put a group in an ALL-or-NOTHING state —
fully collapsed, fully pruned, or intact — so between them they said nothing
about the mixed case, which is the one a user meets first. A group where some
items prune and some survive now has its own test.
a_collapsed_groups_header_survives_pruning now asserts the whole row_items
sequence instead of any(|e| is the Q1 header). An any() cannot see a header
emitted in the wrong position or an extra entry beside it.
The attack itself found no defect, and what it established is worth more than
if it had. Rewriting run_had_item to mean "some item was REJECTED" rather than
"there was an item" is a no-op with the whole workspace green: the header is
pushed eagerly by the FIRST SURVIVING item, so pending_header is already None
before any rejection can be recorded, and the flag is only ever read in runs
where nothing survived — where the two readings coincide. The eager push is
what makes the collapsed/pruned distinction robust rather than delicate.
Also recorded, because it bounds what the new assertion may claim: two
consecutive GroupEnds are not reachable, since expand_category never emits
them, so "no stray GroupEnd" is a property of the writer and not something any
fixture can falsify. The commit that introduced the run_open tracking claimed
it as a virtue; it is one, but it is argued and not tested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Self-review under the code-review skill, two findings, both narrow.
format.rs: the pikk test asserted a finite number never renders as inf and
that half-away rounding still happens, but said nothing about a genuine
f64::INFINITY. Proved the two assertions ask different questions by clamping
round_half_away to a bare passthrough: the rounding assertion fails (2.5 reads
"2"), the infinity one does not.
test-evidence: named the loop arm that looks reachable and is not -- a pass
computing None where the cache holds Some would spin to the bound and now be
marked circular. The only None evaluate_formula_cell can produce is a WHERE
filter that does not match falling through to aggregate_raw over no data, and
aggregate_raw reads self.data rather than the cache while the formulas do not
change during a recompute, so a Some-to-None transition cannot occur. Also
recorded the free cross-check between two of the fixes: the s08t test asserts
ref:Typo AFTER recompute_formulas, so a formula erroring identically every pass
must be seen to converge rather than be overwritten with circular.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Keyed by test name per design-principles section 7, recording for each one the
bug it reproduced, the fault injected to watch it fail, and -- where a test has
a neighbour that looks like it -- the input that separates the two.
Three entries are about that separation rather than about a bug:
- a_view_that_does_not_prune_writes_no_prune_line is the direction the default
supplies, so a writer emitting "prune: true" unconditionally passes the real
test and fails this one.
- an_interleaved_category_keeps_its_item_order explains why the ticket's own
two-item reproduction cannot discriminate: with one item of each kind, both
the right and the wrong order are a single bare run followed by a single
grouped one.
- slash_in_hidden_name_survives_roundtrip records that improvise-nvq4 does NOT
reproduce, and is kept as a passing control so the claim cannot be re-filed
without someone first watching it pass.
core.md is touched for the improvise-qx1z entries, which are View-level and
belong there rather than in formula-and-io.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
A sweep for the SHAPE of improvise-vyrx rather than the site it named. Both
eval_bool and eval_bool_cached also wildcarded the outer Expr match as
`_ => Err("type")`, so a new Expr variant that ought to be usable as a
condition would have become a silent type error in both. Not vyrx's asymmetry
-- the two agreed -- but vyrx's defect: a new variant answered wrongly with
nothing to notice. Adding an Expr::And now fails the build at four sites where
it previously failed at two; verified by adding the variant.
The sweep's other half is which wildcards to LEAVE, recorded in the evidence:
eval_agg's `match inner { Expr::Ref(..) => .., _ => false }` falls through to
the GENERAL path (the per-record fold), which is right for any expression that
is not a bare ref, including one that does not exist yet. The dangerous
wildcard is the one falling through to a refusal or to a specialisation.
CellValue::as_f64 and CategoryKind::of_name clear on the same reading.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-s08t: eval_agg narrowed its partial key only where
find_item_category resolved, so SUM(Typo) left the key unconstrained and
aggregated every numeric measure matching the context. An unknown name now
errors ref:{name} before the raw fast path.
improvise-s2w3: the same fast path assumed a bare Ref names a STORED cell. A
formula target has no raw cell behind it (improvise-nrs), so SUM(Double)
summed nothing and answered 0 where SUM(Double + 0) folded and answered 20.
core.md §13's defence of the specialisation holds — what failed is its
premise — so eval_agg asks formula_for, the one home of "is this cell
derived", and a derived ref falls into the fold.
improvise-7iww: recompute_formulas stopped at MAX_EVAL_DEPTH and left the last
iteration in formula_cache, so a circular formula read Error("circular")
directly and Number(26) through the cached path — which is the DISPLAYED path,
since default views put the virtual categories on Axis::None. Cells still
moving when the bound runs out are now marked circular, which is the same
bound evaluate_depth already enforces.
improvise-vyrx: eval_bool_cached wildcarded BinOp where eval_bool lists its
arms. Verified by adding a BinOp::Mod: four sites now fail the build where
three did before.
improvise-pikk: round_half_away scaled by 10^decimals before truncating, so
format_f64(1e307, false, 2) rendered "inf" for a finite number.
improvise-f86q: add_category checked MAX_CATEGORIES before contains_key, so
re-adding an existing name at the limit was refused with a false message — and
per gotcha 21 aborted the rest of the batch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-yae4. grid_body narrowed every measured width to u16 and only then
added the one-character gap and clamped to MAX_COL_WIDTH, so the arithmetic ran
at full width before the cap that makes it safe. A stored 65535-column text
value panicked at `w + 1` in debug; in release the sum wrapped to 0 and the
column was drawn at MIN_COL_WIDTH — the narrowest column, for the widest value.
A 65536-column value never reached the addition: `as u16` truncated it to 0
first, so it wrapped in debug builds too, silently.
Widths are now capped while still usize, in the one place every measurement
enters (`widen`), which is why header_level_seed folds through it rather than
keeping a second `as u16`. The answer is unchanged for every input that did not
overflow: (min(w,32) + 1).clamp(5,32) and (w + 1).clamp(5,32) agree everywhere,
both saturating at 32 for any w >= 31.
improvise_protocol::header_level_widths carried the identical defect on the
row-header axis — a 65535-column item label — and is fixed the same way. It is
the one home for that rule (gotcha 44), so the row header and the column widths
cannot end up with two answers.
wide_cell_clamps_without_overflow is un-ignored and extended to the 65536 arm
and to an ordinary 20-column value. It asserts `widths == vec![expect]` rather
than `all(<= 32)`: the length pins that the fixture really produces ONE column,
and the ordinary arm pins that the single column is the one carrying the value —
a fixture in which no column was wide would read MIN_COL_WIDTH there and pass a
`<= 32` check. Both injected faults are recorded in
context/test-evidence/core.md.
Not fixed, and out of this ticket's scope: compute_visible_cols accumulates
`acc + w` in u16, reachable only from a terminal ~65535 columns wide.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
`parse_number_format` read the precision as `fmt[i+1..].parse::<u8>()` over
everything after the last `.`, so `,.2f` — the printf spelling the repo-map's
own `.improv` example carries — failed to parse and fell through to zero
decimals, silently, in the TUI, the browser projection and CSV export.
The precision is now the leading digit run after the last `.`, with any
conversion letter ignored. The spellings that are legal are stated once, in
`context/code/core.md`, along with why the trailing `f` is accepted rather than
refused (the grammar takes the format line as an opaque `rest_of_line`, so
refusing it rejects documents this repo already writes) and why the reading is
printf's rather than Excel's (`.00` is the spelling the two disagree on).
`export_csv_quotes_commas_per_rfc4180` asserted the value `starts_with("1,200")`
against a `,.2f` fixture, so it could not tell two decimals from none. It is now
an exact equality, and reverting the fix fails it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
The parameter was none_cats: &[String] and the body consulted it exactly once,
for emptiness -- a boolean wearing a list's type, since the aggregation runs
over key.0 through matching_values. git log -S back to the commit that
introduced Axis::None shows it never examined an element, so the name promised a
filter the function has never performed.
Aggregation::{Exact, OverHidden} admits only the two states the body
distinguishes, and Aggregation::of reduces the list once at the boundary that
holds it. The regression guard is the signature: two lists with the same
emptiness and different contents is no longer a constructible call.
recompute_formulas and aggregate_raw keep &[String] -- they walk the elements.
improvise-5jm5
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
`View::on_category_added`'s early return read
cat_name.starts_with('_') || self.is_records_layout()
— the same expression mixing both derivations one operator apart, the left a
name prefix and the right `CategoryKind`. A reader who checks this line sees
the enum being used and has no reason to look at the operand beside it. The
second site, `holds_a_regular`, had the same split.
`Model::add_category` does not validate names, so a user category literally
named `_Foo` was `CategoryKind::Regular` to `records_columns`, `cell_id` and
`is_empty_model`, and "virtual" to these two lines — seated on `Axis::None`,
never appearing in a pivot, while every other consumer treated it as ordinary.
Decided explicitly, as the ticket asks: `add_category` is NOT made to refuse a
leading underscore. The three reserved names are the ones
`CategoryKind::of_name` knows, `_Foo` is a legal document, and a user-visible
restriction is a bigger change than making one answer out of two.
Both sites now ask `CategoryKind::of_name(..).is_virtual()`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Review of #54 found the fold reintroducing exactly the divergence the ruling
exists to remove, and the same line turned out to be the 244 ms stall. One
defect, two symptoms.
`eval_expr_cached`'s `Expr::Ref` looks the value up in `formula_cache` at
`context.with(cat, name)`. Inside the fold `context` is a MEMBER-RECORD key
carrying the none_cat coordinates, while `formula_cache` is keyed by stems with
none_cats STRIPPED — so the lookup could never hit, fell through to
`aggregate_raw`, found no raw cell behind a formula target, and answered
Err("ref:…"). A routing formula over a derived measure gave Number(11.0)
uncached and Error("ref:Margin") cached, and the cached path is the one a pivot
grid runs whenever any category sits on Axis::None.
`eval_expr`/`eval_bool` are hoisted to module level and BOTH evaluators now hand
the fold the same record-level `eval_expr`. The two agree by construction rather
than by a test. Note the repair not taken: teaching the fold to hit the cache by
stripping none_cats off the member key returns the GROUP's value to every
member — aggregate-then-route through a cache lookup, and silent.
That same per-member `aggregate_raw` was one poorly-narrowed scan per member, an
O(n^2) shape. Measured with the new `examples/fold_bench.rs`, five runs a point,
at 18,050 cells: 51.1-53.3 ms before, 18.1-19.2 ms after, against 7.1-9.0 ms for
a linear formula. Scaling went from superlinear (45x cells, 85x time) to linear
(45x cells, 45x time). Debug 2.75 s -> 132.9-135.1 ms. The fixed point runs
exactly 2 passes, measured by counting, so hoisting the member set has a 2x
ceiling and is declined on improvise-9iyv — a memo keyed by less than the whole
`partial` is a second answer to what a cell depends on.
`Expr::Is` has an arm in each evaluator and the cached one was untested: the
earlier fault was injected into the shared `eval_is`, which both arms call.
Replacing the cached arm alone left the suite green. The first replacement test
ALSO stayed green, which is the interesting part — once the fold routes through
`eval_expr`, an `Is` inside a SUM never reaches the cached arm, so that arm is
reachable only by a bare `Is` outside an aggregate. That is what the new test
uses, and injecting into that arm alone now fails it.
`IS` gained a word boundary: `Region ISLAND` parsed as Is(Region, "LAND"),
silently wrong where it was a parse error. `is_expr` is compound-atomic with
`sp+` written out, because a negative lookahead in an ordinary rule tests the
character AFTER pest's auto-inserted whitespace and fails backwards. The
lookahead was then redundant and removed — deleting it left the suite green,
while relaxing `sp+` to `sp*` fails the test, so `sp+` is the mechanism. `WHERE`
has the same hole, is far less exposed, and is filed as improvise-r08u.
Also: `ambiguous.sort_unstable()` before `dedup`, which only removes consecutive
duplicates, so the refusal could name a column twice.
Every fault re-injected after the refactor: no-fold 10 failures, shared eval_is
5, record-level Is arm 4, cached Is arm 1, feature_roles order 2, context-vs-
partial 1, the new cached-fold divergence 1, `sp+` relaxed 1. Baseline 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Self-review of the commit before it. The record key the fold evaluates at was
built by folding the record's coordinates onto `context`, which still carries
the formula's own target coordinate. Harmless for a `_Measure` target, because
`Expr::Ref` overwrites `_Measure` on the way in — and not harmless for a formula
targeting any other category: the record key keeps `Levy=Tax`, `evaluate_depth`
matches the same formula again at that key, and the cell converges to
`Error("circular")`. A formula silently self-referencing through its own
aggregate.
`partial` is `context.without(target_category)` plus any inline WHERE, so
folding onto it drops the target coordinate by construction. Every member
satisfies `partial` by definition — that is what selected them — so nothing
constraining is lost.
`a_formula_targeting_another_category_still_folds_over_its_records` is the
guard; restoring the `context` fold fails it with `Error("circular")` against
`11.0` and leaves the other eight passing. Evidence in
`context/test-evidence/core.md`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
improvise-4zl7. A tree is fitted on individual records; an improvise formula is
evaluated at a CellKey, and an aggregating cell used to aggregate each Expr::Ref
first and evaluate the IF once on the totals. That is the same answer for a
linear formula and a different one for anything that routes, so every fitted
tree meant one thing on a record row and something else one row up. The user
ruled: aggregate the rule over the members.
The rule needed no new aggregate vocabulary. SUM(e) already means "fold e over
the members" and simply did not: eval_agg specialised Expr::Ref and, for every
other inner expression, summed the raw matching cells and ignored the expression
entirely. It now folds — one entry per member record, the expression evaluated
at that record's own coordinates. eval_agg is shared by eval_expr and
eval_expr_cached, so the fold has exactly one home; each evaluator passes its
own recursion in.
Expr::Is(Filter) is the new arm in both evaluators: `Cat IS Item` is 1.0 when
the cell's own Cat coordinate is Item, keeping the evaluator f64-only and
reusing Formula::filter's own context.get/unwrap_or(false). A grammar production
in primary, before ref_expr, with both operands through
improvise_formula::quote's one home; IS joins the keyword guard.
The emitter reads feature index -> role off preprocess's plan rather than
reconstructing it: gp_tree::data::feature_roles is the one statement of the
expansion order and encode_row now drives off it too. improvise_gp::Features
carries one Spelling per feature — Measure, Coordinate or Artefact — decided by
role and column, never by the name's shape. A one-hot over a real category is
emitted as `Region IS North` and accepted; col.missing and the date parts stay
Artefact and stay refused (improvise-wd6n).
fit-measure on a document a category explains now lands instead of refusing, and
the refusal no longer depends on which features the search happened to pick.
Evidence: eight hand-computed tests in improvise-core over a four-record fixture
whose Cost straddles the threshold inside each region, so route-then-aggregate
and aggregate-then-route differ at every level. Two faults injected and watched
to fail — reverting the fold fails 7 of 8, and a constant-false eval_is fails
exactly the 3 coordinate tests. Reordering feature_roles fails both it and
encode_row_replays_the_training_expansion.
The fold is O(members) per evaluation per fixed-point pass and unbenchmarked:
improvise-9iyv.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
improvise-core's comments are the oldest in the repository and carry the most
expensive artefact in it. This lands the destination before the cap, so the
next commits are a move and not a deletion.
context/code/core.md carries the chains that cannot survive two lines: the
ordering-is-correctness argument for DataStore (improvise-shyt/2yc2), the
unanimity rule and its branch order (improvise-l4qx), the surrogate width and
byte-format argument (ADR 002), the wide-records column vocabulary, the
virtual-on-an-axis rule (improvise-2zq), and the two-evaluator hazard behind
improvise-0bf.
context/test-evidence/core.md carries the bug narratives, keyed by exact test
name, per design-principles §7. 52 entries, all verified against
`--check-evidence`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edward Langley