fit: route per record and aggregate, never route on the aggregate
improvise-4zl7. A tree is fitted on individual records; an improvise formula is evaluated at a CellKey, and an aggregating cell used to aggregate each Expr::Ref first and evaluate the IF once on the totals. That is the same answer for a linear formula and a different one for anything that routes, so every fitted tree meant one thing on a record row and something else one row up. The user ruled: aggregate the rule over the members. The rule needed no new aggregate vocabulary. SUM(e) already means "fold e over the members" and simply did not: eval_agg specialised Expr::Ref and, for every other inner expression, summed the raw matching cells and ignored the expression entirely. It now folds — one entry per member record, the expression evaluated at that record's own coordinates. eval_agg is shared by eval_expr and eval_expr_cached, so the fold has exactly one home; each evaluator passes its own recursion in. Expr::Is(Filter) is the new arm in both evaluators: `Cat IS Item` is 1.0 when the cell's own Cat coordinate is Item, keeping the evaluator f64-only and reusing Formula::filter's own context.get/unwrap_or(false). A grammar production in primary, before ref_expr, with both operands through improvise_formula::quote's one home; IS joins the keyword guard. The emitter reads feature index -> role off preprocess's plan rather than reconstructing it: gp_tree::data::feature_roles is the one statement of the expansion order and encode_row now drives off it too. improvise_gp::Features carries one Spelling per feature — Measure, Coordinate or Artefact — decided by role and column, never by the name's shape. A one-hot over a real category is emitted as `Region IS North` and accepted; col.missing and the date parts stay Artefact and stay refused (improvise-wd6n). fit-measure on a document a category explains now lands instead of refusing, and the refusal no longer depends on which features the search happened to pick. Evidence: eight hand-computed tests in improvise-core over a four-record fixture whose Cost straddles the threshold inside each region, so route-then-aggregate and aggregate-then-route differ at every level. Two faults injected and watched to fail — reverting the fold fails 7 of 8, and a constant-false eval_is fails exactly the 3 coordinate tests. Reordering feature_roles fails both it and encode_row_replays_the_training_expansion. The fold is O(members) per evaluation per fixed-point pass and unbenchmarked: improvise-9iyv. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -43,12 +43,18 @@ Three entry points, because the callers do not want the same thing:
|
||||
|
||||
An item literally named `SUM` is written `|SUM|`. `is_bare_name` is
|
||||
deliberately **stricter than the grammar strictly requires**: bare keywords
|
||||
(`WHERE SUM AVG MIN MAX COUNT IF`) *do* parse today, but only because
|
||||
(`WHERE SUM AVG MIN MAX COUNT IF IS`) *do* parse today, but only because
|
||||
`agg_call` needs a `(` and PEG backtracking then falls through to `ref_expr`
|
||||
— an emergent property of **rule ordering**, not a stated rule. A pipe-quoted
|
||||
name is an identifier in every position, with nothing to fall through, so a
|
||||
name colliding with a keyword is quoted and stops depending on that ordering.
|
||||
|
||||
`IS` joined the list with `is_expr` (improvise-4zl7), and it is the one keyword
|
||||
whose omission would have been *silently* wrong rather than merely fragile: a
|
||||
category literally named `IS` written bare turns `IS IS East` into a predicate
|
||||
on a category called `East`. Adding a production to `formula.pest` means adding
|
||||
its keyword here — the two are one change.
|
||||
|
||||
The cost is worth paying because a `## Formulas` line carries formula text
|
||||
**verbatim**: a name `format_md` writes is later read by `parse_formula`. One
|
||||
namespace, one spelling rule.
|
||||
@@ -65,6 +71,27 @@ io returned it and `parser.rs` panicked; the survivor returns it.
|
||||
|
||||
---
|
||||
|
||||
## `Cat IS Item`: a coordinate predicate in the expression grammar
|
||||
|
||||
`is_expr = { identifier ~ ^"IS" ~ filter_value }` sits in `primary` **before**
|
||||
`ref_expr`, which would otherwise consume the category name and leave `IS Item`
|
||||
unparseable. A bare name still parses as a reference: `is_expr` needs the
|
||||
keyword, fails without it, and PEG backtracks into `ref_expr`.
|
||||
|
||||
It reuses `where_clause`'s two operands — an `identifier` for the category and a
|
||||
`filter_value` for the item — and therefore `build_filter`, which already reads
|
||||
exactly that pair. The `^"IS"` literal is a string in the grammar rather than a
|
||||
rule, so it contributes no pair and `into_inner()` yields the two operands and
|
||||
nothing else. One builder, one quoting rule, both halves through
|
||||
`improvise_formula::quote`.
|
||||
|
||||
The grammar-walking generator needs no new case: it reads `formula.pest` through
|
||||
`pest_meta` at test time, so a production added to `primary` is generated and
|
||||
round-tripped from the day it exists.
|
||||
|
||||
What the *evaluator* does with it, and why a fitted tree is always wrapped in a
|
||||
`SUM`, is `context/code/core.md` §13.
|
||||
|
||||
## A formula target is unquoted, exactly as a reference is
|
||||
|
||||
`parse_formula` strips the pipes off the target the same way it does off an
|
||||
|
||||
Reference in New Issue
Block a user