fit: route per record and aggregate, never route on the aggregate

improvise-4zl7. A tree is fitted on individual records; an improvise formula is
evaluated at a CellKey, and an aggregating cell used to aggregate each Expr::Ref
first and evaluate the IF once on the totals. That is the same answer for a
linear formula and a different one for anything that routes, so every fitted
tree meant one thing on a record row and something else one row up. The user
ruled: aggregate the rule over the members.

The rule needed no new aggregate vocabulary. SUM(e) already means "fold e over
the members" and simply did not: eval_agg specialised Expr::Ref and, for every
other inner expression, summed the raw matching cells and ignored the expression
entirely. It now folds — one entry per member record, the expression evaluated
at that record's own coordinates. eval_agg is shared by eval_expr and
eval_expr_cached, so the fold has exactly one home; each evaluator passes its
own recursion in.

Expr::Is(Filter) is the new arm in both evaluators: `Cat IS Item` is 1.0 when
the cell's own Cat coordinate is Item, keeping the evaluator f64-only and
reusing Formula::filter's own context.get/unwrap_or(false). A grammar production
in primary, before ref_expr, with both operands through
improvise_formula::quote's one home; IS joins the keyword guard.

The emitter reads feature index -> role off preprocess's plan rather than
reconstructing it: gp_tree::data::feature_roles is the one statement of the
expansion order and encode_row now drives off it too. improvise_gp::Features
carries one Spelling per feature — Measure, Coordinate or Artefact — decided by
role and column, never by the name's shape. A one-hot over a real category is
emitted as `Region IS North` and accepted; col.missing and the date parts stay
Artefact and stay refused (improvise-wd6n).

fit-measure on a document a category explains now lands instead of refusing, and
the refusal no longer depends on which features the search happened to pick.

Evidence: eight hand-computed tests in improvise-core over a four-record fixture
whose Cost straddles the threshold inside each region, so route-then-aggregate
and aggregate-then-route differ at every level. Two faults injected and watched
to fail — reverting the fold fails 7 of 8, and a constant-false eval_is fails
exactly the 3 coordinate tests. Reordering feature_roles fails both it and
encode_row_replays_the_training_expansion.

The fold is O(members) per evaluation per fixed-point pass and unbenchmarked:
improvise-9iyv.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Langley
2026-09-20 11:52:27 -07:00
parent 52507a6678
commit 022e12f9b4
14 changed files with 1090 additions and 313 deletions
+28 -1
View File
@@ -43,12 +43,18 @@ Three entry points, because the callers do not want the same thing:
An item literally named `SUM` is written `|SUM|`. `is_bare_name` is
deliberately **stricter than the grammar strictly requires**: bare keywords
(`WHERE SUM AVG MIN MAX COUNT IF`) *do* parse today, but only because
(`WHERE SUM AVG MIN MAX COUNT IF IS`) *do* parse today, but only because
`agg_call` needs a `(` and PEG backtracking then falls through to `ref_expr`
— an emergent property of **rule ordering**, not a stated rule. A pipe-quoted
name is an identifier in every position, with nothing to fall through, so a
name colliding with a keyword is quoted and stops depending on that ordering.
`IS` joined the list with `is_expr` (improvise-4zl7), and it is the one keyword
whose omission would have been *silently* wrong rather than merely fragile: a
category literally named `IS` written bare turns `IS IS East` into a predicate
on a category called `East`. Adding a production to `formula.pest` means adding
its keyword here — the two are one change.
The cost is worth paying because a `## Formulas` line carries formula text
**verbatim**: a name `format_md` writes is later read by `parse_formula`. One
namespace, one spelling rule.
@@ -65,6 +71,27 @@ io returned it and `parser.rs` panicked; the survivor returns it.
---
## `Cat IS Item`: a coordinate predicate in the expression grammar
`is_expr = { identifier ~ ^"IS" ~ filter_value }` sits in `primary` **before**
`ref_expr`, which would otherwise consume the category name and leave `IS Item`
unparseable. A bare name still parses as a reference: `is_expr` needs the
keyword, fails without it, and PEG backtracks into `ref_expr`.
It reuses `where_clause`'s two operands — an `identifier` for the category and a
`filter_value` for the item — and therefore `build_filter`, which already reads
exactly that pair. The `^"IS"` literal is a string in the grammar rather than a
rule, so it contributes no pair and `into_inner()` yields the two operands and
nothing else. One builder, one quoting rule, both halves through
`improvise_formula::quote`.
The grammar-walking generator needs no new case: it reads `formula.pest` through
`pest_meta` at test time, so a production added to `primary` is generated and
round-tripped from the day it exists.
What the *evaluator* does with it, and why a fitted tree is always wrapped in a
`SUM`, is `context/code/core.md` §13.
## A formula target is unquoted, exactly as a reference is
`parse_formula` strips the pipes off the target the same way it does off an