fit: route per record and aggregate, never route on the aggregate
improvise-4zl7. A tree is fitted on individual records; an improvise formula is evaluated at a CellKey, and an aggregating cell used to aggregate each Expr::Ref first and evaluate the IF once on the totals. That is the same answer for a linear formula and a different one for anything that routes, so every fitted tree meant one thing on a record row and something else one row up. The user ruled: aggregate the rule over the members. The rule needed no new aggregate vocabulary. SUM(e) already means "fold e over the members" and simply did not: eval_agg specialised Expr::Ref and, for every other inner expression, summed the raw matching cells and ignored the expression entirely. It now folds — one entry per member record, the expression evaluated at that record's own coordinates. eval_agg is shared by eval_expr and eval_expr_cached, so the fold has exactly one home; each evaluator passes its own recursion in. Expr::Is(Filter) is the new arm in both evaluators: `Cat IS Item` is 1.0 when the cell's own Cat coordinate is Item, keeping the evaluator f64-only and reusing Formula::filter's own context.get/unwrap_or(false). A grammar production in primary, before ref_expr, with both operands through improvise_formula::quote's one home; IS joins the keyword guard. The emitter reads feature index -> role off preprocess's plan rather than reconstructing it: gp_tree::data::feature_roles is the one statement of the expansion order and encode_row now drives off it too. improvise_gp::Features carries one Spelling per feature — Measure, Coordinate or Artefact — decided by role and column, never by the name's shape. A one-hot over a real category is emitted as `Region IS North` and accepted; col.missing and the date parts stay Artefact and stay refused (improvise-wd6n). fit-measure on a document a category explains now lands instead of refusing, and the refusal no longer depends on which features the search happened to pick. Evidence: eight hand-computed tests in improvise-core over a four-record fixture whose Cost straddles the threshold inside each region, so route-then-aggregate and aggregate-then-route differ at every level. Two faults injected and watched to fail — reverting the fold fails 7 of 8, and a constant-false eval_is fails exactly the 3 coordinate tests. Reordering feature_roles fails both it and encode_row_replays_the_training_expansion. The fold is O(members) per evaluation per fixed-point pass and unbenchmarked: improvise-9iyv. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -411,3 +411,69 @@ with the entry above this pins the rotation without enumerating it twice.
|
||||
|
||||
Distinct axes must be distinguishable at a glance; `color()` is the only reason
|
||||
`AxisColor` exists, so a collision defeats it.
|
||||
|
||||
### a_routing_formula_over_a_group_sums_the_members_answers
|
||||
|
||||
improvise-4zl7, and the hand-computed half of its evidence. Every number in
|
||||
`route_then_aggregate_tests` is written out rather than obtained from a second
|
||||
call into the evaluator, because the bug being guarded is *the evaluator
|
||||
answering consistently and wrongly*.
|
||||
|
||||
The fixture is four records — `Cost` of 10 and 100 in North, 20 and 200 in
|
||||
South — and the formula is `Pred = SUM(IF(Cost < 50, 1, 10))`. `Cost` straddles
|
||||
the threshold **inside each region**, which is the whole design of the fixture:
|
||||
|
||||
| cell | route-then-aggregate | aggregate-then-route |
|
||||
|---|---|---|
|
||||
| one record (`Cost` 10) | 1 | 1 |
|
||||
| North (`Tag` away) | 1 + 10 = **11** | `110 < 50` → **10** |
|
||||
| everything | 1+10+1+10 = **22** | `330 < 50` → **10** |
|
||||
|
||||
A fixture whose records fall on the same side of the split would give the same
|
||||
answer either way at the record level and differ only by a factor of the record
|
||||
count, which is the kind of agreement that reads as correctness.
|
||||
|
||||
Two faults were injected and both were watched to fail. Restoring the old
|
||||
`eval_agg` — sum the matching raw values and ignore the inner expression —
|
||||
fails **7 of the 8** tests with 330 for 22, 110 for 11 and 10 for 1; the one
|
||||
survivor is `is_is_one_when_the_cells_own_coordinate_matches_and_zero_otherwise`,
|
||||
which has no `SUM` and so never reaches the fold. Making `eval_is` return `0.0`
|
||||
unconditionally fails exactly the **3** coordinate tests, with 400 for 214 and
|
||||
200 for 14. The two faults failing disjoint-but-overlapping sets is itself the
|
||||
evidence that the tests separate the fold from the predicate.
|
||||
|
||||
### a_coordinate_split_survives_aggregating_over_that_very_category
|
||||
|
||||
The case the ruling exists for, and the one the ticket's own text called the
|
||||
product question. With `Region` aggregated away the cell has no `Region`
|
||||
coordinate of its own, so a formula that read the *cell's* coordinate would see
|
||||
nothing and take `Expr::Is`'s `unwrap_or(false)` branch — answering 100 once.
|
||||
The fold hands the expression a **record** key, so each of the four records
|
||||
answers for its own coordinate: 7 + 7 + 100 + 100 = 214.
|
||||
|
||||
### a_member_whose_input_is_missing_errors_rather_than_being_dropped
|
||||
|
||||
The asymmetry between the `Expr::Ref` fast path and the fold, asserted rather
|
||||
than left to be discovered. `SUM(Revenue)` skips a record with no `Revenue`
|
||||
cell; `SUM(<expr>)` cannot, because the expression is what is being asked and an
|
||||
expression that will not evaluate is an error. The fixture clears one record's
|
||||
`Cost` and writes a different measure on the same record, so the record still
|
||||
exists and still has a member cell to be found by — otherwise the record would
|
||||
simply vanish from `matching_cells` and the test would prove nothing.
|
||||
|
||||
### feature_roles_line_up_with_the_names_preprocess_emitted
|
||||
|
||||
Lives in `gp-tree`, recorded here because the rule it guards is improvise-4zl7's.
|
||||
`feature_roles` is the one statement of `preprocess`'s expansion order and
|
||||
`encode_row` drives off it, so a drift is silent in both directions: the emitter
|
||||
would spell the wrong feature and a re-encoded row would land in the wrong
|
||||
column. The assertion rebuilds each feature's *name* from its role and compares
|
||||
it with the name `preprocess` emitted, which is the only thing that ties the two
|
||||
walks together.
|
||||
|
||||
The last four assertions exist because the loop is vacuous over a fixture that
|
||||
produces only one kind of feature. The fixture has a numeric column with a
|
||||
missing cell, a categorical column and a date column, so all four roles appear.
|
||||
Fault injected: swapping the `Numeric` and `Missing` pushes in `feature_roles`
|
||||
fails this test **and** `encode_row_replays_the_training_expansion`, which is
|
||||
the pair that would have disagreed.
|
||||
|
||||
Reference in New Issue
Block a user