context/code/formula-and-io.md records why the category merge happens in pass 1
rather than pass 2, and why the WHERE boundary is sp+ rather than a lookahead --
the grammar-walking generator cannot honour a predicate, which is the part that
generalises.
context/test-evidence records the fault injected for each test, including the
one that made the WHERE test carry two negative shapes: compound-atomicity alone
already refuses the ticket's own reproducer, so a test built only on it passes
over a rule still spelled sp*.
improvise-349m improvise-r08u
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
improvise-4zl7. A tree is fitted on individual records; an improvise formula is
evaluated at a CellKey, and an aggregating cell used to aggregate each Expr::Ref
first and evaluate the IF once on the totals. That is the same answer for a
linear formula and a different one for anything that routes, so every fitted
tree meant one thing on a record row and something else one row up. The user
ruled: aggregate the rule over the members.
The rule needed no new aggregate vocabulary. SUM(e) already means "fold e over
the members" and simply did not: eval_agg specialised Expr::Ref and, for every
other inner expression, summed the raw matching cells and ignored the expression
entirely. It now folds — one entry per member record, the expression evaluated
at that record's own coordinates. eval_agg is shared by eval_expr and
eval_expr_cached, so the fold has exactly one home; each evaluator passes its
own recursion in.
Expr::Is(Filter) is the new arm in both evaluators: `Cat IS Item` is 1.0 when
the cell's own Cat coordinate is Item, keeping the evaluator f64-only and
reusing Formula::filter's own context.get/unwrap_or(false). A grammar production
in primary, before ref_expr, with both operands through
improvise_formula::quote's one home; IS joins the keyword guard.
The emitter reads feature index -> role off preprocess's plan rather than
reconstructing it: gp_tree::data::feature_roles is the one statement of the
expansion order and encode_row now drives off it too. improvise_gp::Features
carries one Spelling per feature — Measure, Coordinate or Artefact — decided by
role and column, never by the name's shape. A one-hot over a real category is
emitted as `Region IS North` and accepted; col.missing and the date parts stay
Artefact and stay refused (improvise-wd6n).
fit-measure on a document a category explains now lands instead of refusing, and
the refusal no longer depends on which features the search happened to pick.
Evidence: eight hand-computed tests in improvise-core over a four-record fixture
whose Cost straddles the threshold inside each region, so route-then-aggregate
and aggregate-then-route differ at every level. Two faults injected and watched
to fail — reverting the fold fails 7 of 8, and a constant-false eval_is fails
exactly the 3 coordinate tests. Reordering feature_roles fails both it and
encode_row_replays_the_training_expansion.
The fold is O(members) per evaluation per fixed-point pass and unbenchmarked:
improvise-9iyv.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
improvise-formula's comments carried three arguments worth keeping and
several restatements of the code. The arguments move to
context/code/formula-and-io.md (pipe quoting's one home, the keyword guard
and why PEG rule ordering is not a rule, the unquoted formula target, and
why a new grammar production needs generator coverage) and the bug
narratives to context/test-evidence/formula-and-io.md, keyed by test name.
No behaviour change: comments only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edward Langley