# `gp-tree` and `improvise-gp` The reasoning that used to live in the doc comments of `crates/gp-tree/**` and `crates/improvise-gp/**`. Written for a reader with the code open: the code says what happens, this says why, and why the obvious alternative is wrong. Bug narratives keyed by test name are in `context/test-evidence/gp.md`. The condensed version is repo-map gotcha 37. --- ## 1. Two crates, one seam `gp-tree` is problem-agnostic: a `Tree` is a regression tree over a fixed-width `f32` feature vector, and the crate owns the representation, the GP operators, `simplify`, `render`, one generation (`breed`), the search (`fit`), and the optional raw-table bridge (`data`). It knows nothing about *what* the features mean or *how* a candidate is scored — the caller supplies a `Spec` (per-feature ranges and names) and, outside `fit`, its own fitness. Determinism is the caller's too: the operators take an `Rng` trait implemented over a seeded generator. `improvise-gp` is the one seam between that crate and improvise. It cannot live in `gp-tree` — that inverts the dependency and drags improvise into a generic GP crate — and it cannot live in `improvise-formula`, which must not know GP exists. It holds three things: - `lib.rs` — the emitter: a fitted `Tree` as improvise formula text. - `dataset.rs` — a `View`'s records as the feature matrix a fit consumes, the improvise-side counterpart of `preprocess`. - `fit.rs` — one fit end to end: records in, an improvise `Formula` plus the evidence for believing it out. `fit.rs` is here rather than in a `Cmd` because none of it is terminal-specific. A browser fitting the same measure over the same document must run exactly these steps in exactly this order, or it gets a different tree from the same seed. `improvise-command/src/command/cmd/fit.rs` is the thin terminal wrapper over it. ## 2. `gp-tree` returns data and never prints A decision the user should hear about comes back as **data**: `Dataset::warnings` carries the columns `preprocess` dropped for being entirely missing, `fit.rs` hands back pooled `(prediction, actual)` pairs and a score per feature rather than tables. `examples/fit_csv.rs` owns every `println!` and every `eprintln!`, and that is the only reason its output stayed byte-identical across these moves. **Adding an `eprintln!` to `fit.rs` or `data.rs` is the regression**, not a convenience. The reason is not tidiness: a library does not own its caller's stderr. A TUI writing to an alternate screen, a wasm build, or a caller that wants the notice in its own status line can none of them suppress, redirect or translate an `eprintln!`. `preprocess` used to do exactly that, which was correct while this code *was* `examples/fit_csv.rs`. `improvise-gp` inherits the rule. `FitOutcome::status_line` is a *rendering* of the outcome data, offered so a second front end does not compose its own — the same argument that put `flatten_cat_tree` in the shared layer — and it is a returned `String`, never a `println!`. ## 3. The vendored RNG, and why its constants are a promise `SplitMix64` is ~20 lines and no dependency. The `Rng` trait exists so a caller can bring its own generator, and every caller so far wrote out this same one. A fit is not a throwaway: it writes a rule into a user's document, so re-running it on the same data must produce the same rule, and a bug report must be reconstructible from its seed alone. That is worth vendoring rather than depending — the point is reproducibility, not statistical quality, and SplitMix64's is ample for picking nodes and thresholds. `SplitMix64::seed()` keeps the seed the run started from, unchanged as the generator advances, so a caller can show it and record it. A generator that has advanced past its seed cannot answer that question. `next_u64` is public because `splitmix64_matches_the_reference_vectors` pins it. **The constants are the algorithm, not an implementation detail**: every seed already printed by a fit and every `data::Model` seed already saved is a promise that *these* numbers come back. ## 4. `Spec::ranges` is a soundness precondition `ranges[i]` must be a **superset** of the values feature `i` can actually take. A too-narrow range makes `simplify` prune a branch that is reachable, silently changing the tree's outputs. `preprocess` derives the ranges from the columns themselves, which is what keeps this true; `every_training_value_lies_inside_its_feature_range` pins that it stays derived. `simplify` seeds each feature's feasible interval with its real range rather than ±∞, so an out-of-range threshold prunes. The interval is half-open `[lo, hi)` but a feature can hit its max, so `hi` is nudged by `EPS = 1e-3` and a split *at* the max (`x < max`) stays live. `EPS` is safely below typical threshold granularity. ## 5. `Row`: an invariant made unrepresentable `Row` pairs a feature vector with what it is supposed to predict. The target is generic because the two supervised shapes differ only in it — `Row` for regression, `Row` for a class code — and every scoring function then reads **one** slice instead of two. That is the point of the type. Parallel `x`/`y` slices could be handed in at different lengths, and `zip` answered by silently scoring the shorter prefix while a denominator taken from the longer one made the mean too small. There is no length to get wrong here, so there is nothing to validate and no `Result` to thread through five signatures (improvise-dkqn, `doctrine(an_invariant_is_made_unrepresentable_not_validated)` — the same answer `prune_axis_items` got). **The pairing is made at the boundary that builds the data.** `preprocess` walks one row index and the rows live inside `Target`; a `Dataset` carrying `rows` and `target` as two collections would move the mismatched length one level up, into every caller. `mean_over(total, n)` is the one place the empty row set is answered. Every mean in `fit.rs` used to reach its own `0.0 / 0.0` on an empty slice: a NaN that propagates through `FitConfig::cost`, sorts unpredictably against real costs, and reaches a caller as a number rather than as an error. Scoring no rows finds no error, so the answer is zero, stated once instead of in `mse`, `misclass_rate` and `standardize` separately. **Changing any of this needs bit-level evidence.** `tests/scoring_known_answers.rs` pins every scoring path as `f32` bit patterns plus a `next_u64` witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. See `context/test-evidence/gp.md`. ## 6. `folds` and `cv_folds` answer different questions `folds(n, k)` splits `0..n` into `k` pieces after a deterministic shuffle. `k` arrives from a host — a CLI flag, in this crate's only caller — so zero is a number to answer, not an impossible state: `out[pos % 0]` is a public library function taking its host down with it. One fold is the answer the shape already gives, and `fit` spells the same `max(1)` over `restarts`. `cv_folds(n, k)` yields nothing below two. A partition into one piece is still a partition, but a **cross-validation** over one fold is not a cross-validation: the single fold holds every index, the training complement is empty, and both cross-validators would fit a tree on nothing and then let it predict every row. Yielding no folds makes the pooled pairs empty, which is the true count of such predictions — and no fold is drawn, so the shuffle never touches the caller's generator either (improvise-ry89). `complement(n, test)` builds a bool mask rather than asking `test.contains(i)` per row, which was a scan of the fold per row and so quadratic in the row count (improvise-t3ux). It yields **the same indices in the same order**, which is what every downstream draw from the generator depends on. ## 7. `standardize` / `denormalize`: the affine map must be folded back The crate's leaf scores live in a ±5 range, so a target in the thousands cannot be reached by leaf mutation and `standardize` is not optional. Its consequence is that the champion predicts **standard deviations**. `fit_csv` answers that by *printing* `≈ score × std + mean` beside the tree, and **turns the refusal below into a diagnostic rather than a panic** — `run_fit` returns `Result<(), String>` and `main` exits 1, which is gotcha 37(a) read the right way round: the library returns data, the example owns every message, and that includes the refusals (improvise-pv3g, found on PR #111 by Codex). A document has nowhere to put the affine map, so `improvise-gp::fit_measure` folds it back into the leaves with `denormalize` before a single character of formula text is spelled. Omitting it emits a perfectly well-formed formula whose every cell is out by an affine map — **wrong in the quiet way**, since small numbers in the right shape look like an answer. `denormalize` touches leaves only, and that is the whole reason it is sound: `standardize` rescales the *target*, so a threshold is already in its feature's own units and re-scaling it would be the bug. **The map is a type, because a poisoned one is silent (improvise-pv3g).** A target of `1e20` is a perfectly finite `f32`, and `fit_measure` checked exactly that — every `r.target`, every `r.feat`, every `data.ranges` entry. The overflow is in the **derived** quantity: `standardize` sums `(t - mean)²`, and eight rows at `1.5e20` make `1.8e40`, past `f32::MAX`. `std` came back `inf`, every normalized target became `±0`, and `denormalize` computed `0.0 * inf + mean = NaN` — which surfaced as a **panic** in `number_text`, in a different crate, two layers away from the arithmetic that caused it. The condition is `sum((t-mean)²) > f32::MAX`, so the crashing spread shrinks as records are added: ~6e18 at eight rows, ~1.3e17 at `MAX_RECORDS`. So `standardize` returns `Option<(Vec>, Scaling)>`, and `Scaling` has private fields and one constructor that refuses a non-finite part. That is `doctrine(an_invariant_is_made_unrepresentable_not_validated)` again — the answer `Row` and `prune_axis_items` got — and it buys three things a finiteness check at the call site does not. `denormalize` **cannot** be handed a poisoned map; `cv_regress` cannot pool a prediction scaled by one (it pools nothing for such a fold, which is what `cross_validating_without_a_held_out_fold_pools_nothing` already means by "no evidence"); and the rival implementation cannot be written, so there is nothing left to inject a fault into. The arithmetic is byte-identical — `standardize_is_bit_for_bit_what_it_was` passes with every pattern unchanged. The `1e-6` floor lives **inside** the constructor, after the finiteness answer, because `NaN.max(1e-6)` is `1e-6` in Rust: flooring first launders the poison into a plausible number. Two things measured rather than reasoned, and one of them corrects the ticket. `mean` non-finite with `std` finite **does not occur from finite inputs** — `(t - inf)²` is `+inf`, so the variance is `+inf` and never `NaN`, and `inf.max(1e-6)` is `inf`. Guarding only `std` therefore fails no `standardize` test; the `mean` half of the constructor's check is a contract of the *type* (anyone may call `Scaling::new`), not a reachable path through `standardize`, and `a_scaling_cannot_carry_a_non_finite_part` is the only test that can see it. Saying so is the difference between a guard and a guard that is believed to do something it does not. **Deserialization is the second caller, and it makes that contract load-bearing** (improvise-nzeu). pv3g made a poisoned map unconstructible where it is *derived* and left it constructible where it is *read back*: `TargetModel::Regress` carried a raw `{ mean: f32, std: f32 }` pair, so a hand-edited or corrupted model reintroduced on `--predict` exactly the state the type exists to forbid. `Scaling` is now `#[serde(try_from = "RawScaling", into = "RawScaling")]`, so a saved model reaches the one constructor like every other caller. **Not a check at the use site**: a use-site check is the validated form of the invariant this whole section spent its argument making unrepresentable, and it would leave two answers to "is this map usable" with nothing holding them in agreement. Two things the fix had to preserve, and one it gains. `Regress` carries `#[serde(flatten)] scaling: Scaling`, so the **saved shape is unchanged** — the two numbers still sit beside `tree` — and an existing model file still loads. Measured both ways rather than reasoned: over a regression CSV at one seed the saved JSON is byte-identical, the fit report and the `--predict` output are byte-identical, the new code reads the old code's model identically, and the old code reads the new one's. The arithmetic is unchanged: the CLI's `eval(tree, &f) * std + mean` is now `scaling.restore(..)`, which is the same expression, and the `1e-6` floor is idempotent because the saved `std` came through the constructor already. What it gains is that the `mean` arm finally has a **reachable path and a test**: the poison arrives as a plain JSON number, `1e39`, well inside `f64` and over `f32::MAX`, which serde's `as f32` turns into infinity. **One refusal does not make an entrypoint safe, and the CLI had a second abort one function further along.** At `1e18` the map is finite, the fit lands, and `report_importance` dies with a *capacity overflow*, because `importance_ranking` computes `100.0 * imp[j] / total` — the multiplication first, so an importance over `f32::MAX / 100` is `inf` before the division that would have given 100.0 — and `"#".repeat((inf / 5.0).round() as usize)` is a saturating cast. The bar is clamped and the arithmetic was filed as improvise-k844, deliberately unfixed here. **The stated reason for deferring it was wrong, and is worth keeping as the correction rather than deleting.** This paragraph said reordering the arithmetic "moves the bit patterns `misclass_rate_and_importance_are_bit_for_bit_what_they_were` pins". It does not: that test pins `permutation_importance`'s **raw scores**, and `importance_ranking` consumes them afterwards, so no path it witnesses runs through the share. Measured when k844 was actually fixed — the whole file is byte-identical and not one literal moved. A cost predicted from a test's *name* rather than from what it calls is how a one-line fix stays filed; §8 carries the fix and what it really cost. The lesson to carry is the one the *test* taught — a "just under the overflow" fixture at `1e18` aborted for an unrelated reason, and the fixture that means what it says sits at `1e9`, with `1e18` kept as its own guard against the second abort. ## 8. `RegressionQuality` carries its own baseline A tree with a plausible-looking RMSE that is no better than always predicting the mean is not a weak rule, it is *no rule* — and it is worse than no rule at all, because it is one a user will believe. So the comparison is part of the answer rather than something a caller may forget to compute. `r2` is **not clamped**: negative means the fit does worse than the mean, which is a real outcome, and clamping would make "learned nothing" and "actively misleading" read alike. `importance_ranking` reports each feature's share of the total importance. The all-zero case returns early rather than flooring the divisor: `fit_csv` used `.max(1e-9)`, which is right for the only input it ever had and wrong as a contract — a lone importance of `1e-10` then divides by the floor rather than by itself and reports **10%**, where the one feature the model uses carries all of it. Importances this small are ordinary, being error differences over a *standardized* target, so the shares have to sum to 100% at every scale. **"Every scale" includes the top of `f32`, and it did not** (improvise-k844). The sum and the ratio are computed in `f64` and only the resulting share — which is bounded by 100 and therefore always finite — is narrowed back to `f32`. Two distinct overflows made that necessary, and only the first is the one the ticket named. `100.0 * imp[j] / total` associates left, so a lone importance above `3.4e36` overflowed on the **multiply**, before the divide that would have brought it back into range, and the feature carrying everything was reported as `inf%`. Reassociating to `100.0 * (imp[j] / total)` fixes exactly that and nothing else — which is the trap, because the second overflow is in `total` itself: several large importances sum past `f32::MAX`, and a client of the reassociated version then divides by `inf` and reports a serene **0%** for every feature. `f64` closes both at once, since no sum of finite `f32`s can overflow it, and the guard that remains (`!total.is_finite()`) answers only the case where an importance arrives non-finite — nothing can be apportioned out of an infinite total, so the ranking is empty rather than `NaN`. Note where this is *reachable from*, because it is not where importances are small: both callers score permutation importance in the target's **own** units rather than the standardized ones, so an importance is an MSE difference and grows as the square of the target. That puts a live window between improvise-pv3g's `standardize` refusal and this overflow — a document whose target `standardize` accepts can still have a derived importance that `f32` cannot scale. As there, **every input was finite and the overflow was in the derived quantity**, which is why guarding inputs sees neither bug. ## 9. A fit takes two things to reproduce — and here, four `data::Model` records `seed: Option` **and** `config: Option`, because the same seed under a different population/generations/elite/penalty lands on a different tree (improvise-ksst). Both are `Option` for one reason: every `FitConfig` value and the seed `0` are legal, so nothing can double as "unknown", and anything reporting a sentinel would state a seed that never ran. Note `#[serde(default)]` on an `Option` field is **documentation, not the mechanism** — serde defaults it anyway, so deleting the attribute fails no test. A backward-compatibility test that only deletes the attribute is not discriminating; aim it at an *absent field* loading as the default instead. `improvise-gp::FitOutcome` carries the **whole** `FitRequest`, not the pair. `FitRequest` is `{ seed, config, cv_folds, importance_repeats, max_records }`, and every stochastic step draws from **one** generator in a fixed order — cross-validate, then fit, then explain. So `cv_folds` decides how far that generator has advanced before `fit` draws from it, and therefore changes the *champion* and not merely the evidence for it. Recording two of four fields documented the reproducibility contract without meeting it. ## 10. The emitter: quoting, and the threshold's exact decimal A regression tree *is* a nested conditional, and improvise formulas already have `IF(cond, then, else)`, so a fit can be written straight into a user's model as an ordinary formula with nothing GP-specific left in the file: ```text Score = SUM(IF(humidity < 77.5, 1.25, IF(outlook IS rainy < 0.5, -0.5, 2))) ``` (An earlier form of that sample sat in `improvise-gp/src/lib.rs`'s module doc inside a ```text fence. rustdoc does not execute a `text` fence, so it was prose wearing a fence rather than a doctest, and the two-line cap applies to it like anything else.) Two things in it are improvise-4zl7's, and both are load-bearing. The outer `SUM` is the **route-then-aggregate rule made syntax**: a tree is fitted on records, and `SUM(e)` is improvise's existing word for "fold `e` over the member records" (`context/code/core.md` §13). Without it the formula routes on the totals one row up and means something else there; with it, a record row is a one-member fold and answers the rule itself. `outlook IS rainy` is the one-hot `outlook=rainy` said as a **coordinate predicate** rather than as a reference to an item no document has — see §11 for which features get which spelling. Two properties the emitter owes its caller: **It parses.** Feature names come from the caller's data, so they carry spaces, operators, pipes and keyword collisions. Every name goes through `quote_name` — `improvise_formula::quote`, the one home for pipe quoting (gotcha 34); `improvise-gp` re-exports it because it was part of this crate's public surface. A formula *target* is unquoted on the way back in: it names a `_Measure` item, so a target keeping its delimiters writes the fit into an item literally named `|Total Revenue|`. **It routes identically.** `number_text` emits the **shortest decimal that reads back as the same `f64`** the `f32` widens to — not the f32's full decimal expansion, which runs to 149 fractional digits, and not a shortest-round-trip-to-`f32` rendering. Widening an `f32` to `f64` is exact, so text that reads back as that `f64` makes `feature < thresh` give the same answer in `f64` formula evaluation as in `f32` `gp_tree::eval`. A short rendering (`0.1` for `0.1f32`) sits up to half an ULP away, and a row whose feature value equals the threshold exactly then falls on the *other* side of the split. `number_text` **panics** on a non-finite value. The grammar spells neither, and both fail quietly rather than loudly: `"inf"` parses as an `f64`, so the round-trip loop would return text no formula parser accepts, and NaN matches no width at all. Trees built by this crate's own operators over a finite `Spec` are finite, so this is a precondition rather than a runtime condition — but see §12, because a `.improv` cell holding `inf` makes it reachable from a document. `expr_text` and `mark_features` share one precondition and spell it the same way (improvise-hkt6): a bare `used[*feat]` fails with `index out of bounds`, which names neither the tree nor the spec. Two sites, one failure, one sentence. `used_features` is the single walk both the document check and the plan check read, because walking the tree twice two ways is how those answers drift apart. ## 11. A fit is storable only when the document can *name* its features `preprocess` invents `Region=North` for a one-hot, `Cost.missing` for a missing indicator, `when.year` for a date part. Those are **encoding artefacts, not items of the document**, and `Expr::Ref` resolves through `Model::find_item_category` — so a tree splitting on one emitted a perfectly well-formed formula whose every cell then evaluated to `Error("ref:Region=North")`, with nothing between the fit and the file to say so (improvise-8xhy). improvise-4zl7 closed the categorical third of that by giving the emitter a vocabulary for a coordinate, and by giving the *evaluator* a rule that makes such a formula mean the same thing at every level of aggregation (`context/code/core.md` §13). ### `Features`: one entry per feature index, read off the plan `Features::of(&Dataset, categories)` walks `gp_tree::data::feature_roles` — the **one** statement of `preprocess`'s expansion order, which `encode_row` drives off too — and pairs each role with the name `preprocess` gave it. A tree's `feat` indexes `Features` directly, so nothing anywhere reconstructs index → role from the feature *names*. `Spec::names` is `&[&str]` and could not have carried this; the earlier check matched names against the raw plan instead and said in this section that it was avoiding a second implementation of the expansion. Now there is no second implementation to avoid. Each feature gets one of three `Spelling`s, decided **by role and by column**, never by the shape of the name: | role | column | spelling | |---|---|---| | `Numeric` | a measure | `Expr::Ref(measure)` | | `OneHot { value }` | a category, `value != "(missing)"` | `Cat IS Item` | | `Numeric` | a category | `Artefact(NumericCategory)` | | `DatePart` | | `Artefact(DatePart)` | | `Missing` | | `Artefact(MissingFlag)` | | `OneHot` | a measure | `Artefact(ValueOfAMeasure)` | | `OneHot { "(missing)" }` | a category | `Artefact(AbsentCoordinate)` | The last five arms were **one** `_ => Artefact` until improvise-wd6n, and the wildcard was not merely untidy. The refusal it fed read "a `col.missing` flag or a date part is an encoding artefact", so three of the five families were refused in words naming something they are not — and a future `FeatureRole` variant would have joined them silently, declined under whichever family the message happened to mention. The match is now exhaustive, so a sixth role is a compile error rather than a sixth thing that message misdescribes. The two exclusions are not fussiness. A one-hot over a **measure** column names no coordinate at all — the record's value for that measure is text, and `Cat IS Item` asks about coordinates. `(missing)` is the *absence* of a coordinate, which `Expr::Is` reads as `0` by §13's `unwrap_or(false)`, so the one spelling that would be wrong is exactly the one omitted. A `Numeric` role over a **category** column is the trap the previous version of this section called "the hole a name cannot close": a category whose items all parse as numbers produces a feature named after the category, and `Expr::Ref` would resolve it to a `_Measure` item of the same name — a different column. Both halves now fall out of `is_category`, which is a lookup in the header's own first `category_columns` entries. Duplicate header names are still refused outright, by `no_ambiguous_column_names` and before anything else: `is_category` decides by name, so a name that means two columns is a spelling with two meanings. `Dataset::columns` is **not** index-aligned with the header — `preprocess` `continue`s past an entirely-missing column without pushing a `ColumnSpec`, so one empty category shifts every later spec left. `FeatureRole::column()` indexes `columns`, not the header, so that shift is carried rather than tripped over. ### What `fitted_formula` still asks the document Two refusals, in order. An `Artefact` is refused with no reference to the document at all — nothing could make `Cost.missing` spellable. Everything else is asked **of the document**: a `Measure` must be `find_item_category`-resolvable and a `Coordinate` must name a category the model has. A measure literally named `Region=North` is a legal document and is accepted, because the check never looks at the name's shape — and that is also why the date parts and the `.missing` flags stay covered without a second copy of `preprocess`'s naming rules. `formula_text` still spells whatever it is handed, being the tree's own text; an `Artefact` reaching it is a precondition failure and panics, the same split `number_text` makes for a non-finite value. ### The two families improvise-wd6n declines, separately Both stay `Artefact` and both stay refused. The decline is written here rather than left open, because each needs a **formula-language** change and neither needs an emitter change — and they are answered separately because the vocabulary each wants is different. What changed with the decline is that the refusal now says which, in that family's own words. **`col.missing` is declined because the language has no word for absence.** The flag means "this record has no value for measure X". `Expr::Ref` over an absent cell is an `Error`, not a zero, so there is no existing spelling to reuse and no way to phrase the question as an expression. Closing it means deciding what an absence *is* in a language whose aggregates already treat it as "no rows" — and that decision reaches `apply_agg`, `eval_agg` and the text-unanimity rule, not just this emitter. It is not worth that for a split a fit happens to like. **The date sub-features are declined because improvise has no dates.** `when.epoch_day`, `.year`, `.month`, `.day`, `.dow`, and with a time component `.hour`, `.minute` — `gp_tree::data::build_date` emits them. A `.improv` cell holding a date is `CellValue::Text`; nothing in `eval_expr` reads one, and there is no date type, no date literal and no date arithmetic anywhere in the crate. Closing this is a type addition to the value language, and the sub-features are the *smallest* part of it: a user who could split on `when.year` would immediately want to filter and aggregate by it. Both are visible to the user only as a refusal naming the feature and the reason, which is the right interim: a fit that cannot be stored says so, rather than storing a formula whose every cell is a reference error (improvise-8xhy). `improvise-4zl7`'s route-then-aggregate rule (`context/code/core.md` §13) is untouched — nothing new is emitted, so there is nothing new for it to hold over. Two of the other three are declined permanently: a one-hot over a measure and the `(missing)` pseudo-item name no coordinate *by construction*, so no vocabulary closes them. They are refusals rather than gaps. **`NumericCategory` was not, and improvise-zv8t has closed it.** That it was *called* one is the more useful half of improvise-wd6n: prose declaring something a permanent refusal is what stops anyone filing the ticket, and nothing but a reader could have caught it — the suite was green either way. The accident was one argument. `fit_measure` handed `preprocess` an `Overrides::default()`, so it never said which header columns are document categories, **although the very next line told `Features::of`**. `preprocess` then inferred `Numeric` → `Date` → `Categorical` from the cell values, so a `Year` category with items `2024`/`2025` became a *numeric column* and a category of ISO dates became a *date column* — and both were then refused as encoding artefacts with `Year IS 2024` spellable all along. `fit_measure` now builds its `Overrides` from the same `header[..category_columns]` slice it already hands `Features::of`, forcing each to `Forced::Categorical`, so the one place the document's own answer was known is now the only place either caller reads it from. Three things follow, and the last is a cost rather than a win. **The date decline above now means what it says.** A `DatePart` could arrive over a *category* column by exactly this accident, so §11's date family was narrower than it read. It now covers a genuine date **measure** and nothing else, which is the scope the decline was always argued for. **The `Artefact` arms stay, and stay correct.** `crate::dataset` is `pub` and takes the caller’s `Overrides`, and `Features::of` reads roles off whatever `Dataset` it is handed — so a caller that builds one without forcing still produces a `Numeric` role over a category column, and `fitted_formula` still has to refuse it. The arm is unreachable from `fit_measure`, which is the only entry point a *document* goes through; it is not unreachable code, and deleting it would be deriving the classification from one caller’s habits rather than from the roles. **Forcing widens the encoding, and that costs a document.** A `Year` category with two items was one numeric feature and is now two one-hots; with a hundred items it is a hundred, and `MAX_FEATURES` refuses the fit outright. The first draft of this paragraph claimed "no document that fitted before stops fitting", on the reasoning that a tree splitting on such a column was refused as an artefact anyway. **That is false, and the review that caught it was of this prose rather than of the diff.** The refusal is conditioned on `used_features`, so a fit that simply *never split* on the numeric category succeeded — and it is precisely the wide categories that a fit has least reason to split on. Measured, on a hundred records with a hundred-item numeric `Year` beside a `Cost` that predicts `Score` on its own: before, the fit lands on `IF(Cost < 48.0…)` and `Year` is never touched; after, the preflight refuses at 101 features. `a_wide_numeric_category_now_refuses_a_fit_that_used_to_ignore_it` pins both halves, including that the repair the message names really works. It is still the right trade, for a reason that is *not* "both refuse". The old success was accidental: `Year` was encoded as a single continuous number over 1900–1999, an encoding that means nothing and that the fit could only use by producing a formula it would then be refused for. The document now gets a refusal that names the repair — mark the category a Label — and marking it restores the pre-zv8t feature set exactly. What is lost is the case where the user wanted neither: a wide numeric category left alone while the fit uses the measures. That was filed as **improvise-e8no** and is **done**: the budget can decline *a column* rather than the whole table, which was a change to what `encoded_widths` returns and not to this decision. See §14. The preflight `encoded_widths` had to learn the same slice, or it counted such a column as one feature and handed the refusal to the exact check *after* materialising the matrix — which is the side of the cost the preflight exists to stay on (§14). **And the refusal's own words must not name a column kind the arm does not know.** `FeatureRole::Missing` is *not* conditioned on `is_category` — unlike its `Numeric` and `OneHot` neighbours — and `dataset.rs` writes `""` for a record with no coordinate for a category, so a `.missing` flag over a **category** is reachable. Its clause therefore says "that column" and not "that measure": naming the wrong kind there would be this section's own charge — declined in words naming something they are not — surviving one level down. ## 12. A fit is persisted as its formula, and nothing else (improvise-7na) `.improv` stores the emitted formula and **no** serialized `gp_tree::data::Model`. The grammar is unchanged by this decision, deliberately and not by omission. The ticket offered three options: (a) formula only, refit to score new data; (b) the `data::Model` as a JSON blob; (c) seed + config so the fit reproduces. The answer is (a), with (c) already available in `gp-tree` for whoever wants provenance later. **Against (b)**, two arguments. The general one is the second half of gotcha 41: a blob is a *second copy* of information the document already carries — the tree is the formula — and nothing keeps the two in agreement. A user who edits the formula by hand, or a cell it was fitted on, leaves a blob that disagrees with the file it sits in, and a round-trip test over the blob cannot see that, because it never compares the two. (The *first* half of gotcha 41 does not transfer: an opaque blob inside one new grammar production is not a second implementation of the grammar.) The specific one is layering: `Tree` and `FitConfig` are `gp-tree` types, so `improvise-io` would gain a dependency on `gp-tree` and `.improv` would be obliged to track a third crate's serde layout forever — inverting exactly the dependency `improvise-gp` exists to keep out of the core. **What (b) was for, and why it is not needed.** The encoding plan exists so `fit_csv --predict` can score a row that arrives as untyped CSV text. improvise has no such row: a new row is a *record in the same document*, and `recompute_formulas` scores it through the stored formula from its own measures. **Why (c) is not written to the file either.** `data::Model` already records seed and config, so reproducing a fit is solved wherever that value is in hand; and unlike the CSV case the training data *is* the document. What is missing is a *reader* — writing provenance now would add a grammar production and a round-trip obligation for metadata nothing consumes. Provenance is the fit command's decision, alongside how it words the fit for the user. **A premise the ticket got wrong, and it is the bigger finding.** "The emitted formula is enough to recompute existing cells" holds only for a fit over numeric measures — §11 is that defect, and it is a defect in what a fit may emit, not an argument for persisting the model. ## 13. `dataset.rs`: what a view shows, as a feature matrix **The shape is not invented here.** Records mode is already a feature matrix: a row is one coordinate tuple — every cell sharing all coordinates but `_Measure` — and each measure is its own column (gotcha 1). That is exactly "a sample, and its features". So the module enumerates the same rows records mode does, lays them out as the string table `preprocess` takes, and lets `preprocess` own type inference, one-hot expansion, imputation and the ranges. What it does *not* reuse is `GridLayout`: reaching the records path means putting `_Index` on Row and `_Dim` on Column, which is a different view from the one the user is looking at. `RecordRow::group` is public, so the traversal here is the same two steps the layout takes, against the user's own view. Four selection rules, each of which could reasonably have gone the other way: - **Rows are scoped by `View::filtering_cats()`** — Page ∪ Filter — not by `categories_on(Axis::Page)`. Both constrain which records are visible; only Page is navigable (gotcha 27), and scoping by the navigable half would silently fit against rows a drill had already excluded. - **Hidden items scope the rows and the feature columns.** The mental model is "fit what I can see", and hiding an item is an explicit statement that it is not of interest. An explicitly named target is exempt: the caller asked for it by name. - **Every regular category is a feature, whatever axis it sits on** — including `Axis::None`. A category's coordinate is part of what makes a record distinct, so dropping a None category would leave two different rows with identical feature vectors and different targets: noise the fit can neither learn nor report. `Label` categories are excluded, as they are from records mode, by being not `is_regular` — they are high-cardinality per-row text and one-hot into a perfect memoriser. - **No formula-derived value ever enters the matrix.** Formula targets are not features (a formula is a function of the other measures, so it leaks the target whenever the target is one of its inputs), and a formula target is refused *as* the target — a formula already is the rule, so fitting a tree to reproduce it from its own inputs is trivially perfect and says nothing. One rule covers both halves, with no dependency analysis to get wrong. **`_Measure` is a filtering category and must not filter a fit.** That exception in `filter_coords` is not a special case, it is the difference between a grid and a feature matrix: `_Measure` is the coordinate the *columns* are spread across, so pinning it to one item is not "which records am I looking at" but "throw away every column except this one". `Workbook::new`'s default view puts it on `Axis::Page`, so that was the behaviour on a freshly opened document (improvise-vx3). Two smaller decisions. Records are sorted by their coordinate tuple. That began as a workaround for `matching_cells` walking a `HashSet`; since improvise-shyt it yields insertion order, so the sort is no longer what buys stability. It is kept because a fit's row order should be a property of the *data* rather than of the typing history that entered it. And a `Number` cell is rendered with `f64::to_string`, never `format_f64`: display rounding is view-only and must not reach a fit (gotcha 8). An `Error` cell has no value, so it is spelled as missing and picks up the same imputation and `.missing` indicator an absent cell would. ## 14. `improvise-gp::fit`: budgets, and the enum that refuses to guess ### Two budgets, and why the preflight is advisory `FitConfig` bounds the *search*; nothing bounded the *width*, which `preprocess` can multiply without limit (one column per distinct value of a categorical column). Permutation importance alone is `repeats × rows × features` tree evaluations, on the thread that draws the next frame. `MAX_FEATURES = 64` is comfortably above any hand-built pivot table — this repo's own 18,000-cell benchmark encodes far fewer — and comfortably below the one-column-per-row shape that is really a mis-typed category. The message names the fix (`Label`), because that is what the user actually wants. `encoded_widths` is the **advisory** preflight, run on the raw table *before* `preprocess` materialises anything. Checking width afterwards guards the wrong side of the cost: a category with a distinct value per record allocates a rows × rows one-hot *during* preprocessing, so the refusal arrives after the memory and the O(rows²) work have been spent. It approximates `preprocess`'s type inference rather than sharing it. Copying the predicate is the lesser evil; the alternative is exporting the inference and calling this a second implementation of it. **The debt that copy carries is that the preflight owes EVERY branch `preprocess` can take, and it paid two of three.** `preprocess` infers `Numeric → Date → Categorical`, in that order, and the preflight had no date arm at all: a column whose cells parse as neither numbers nor a forced category fell through to the categorical count. So a **measure column holding dates** — one distinct timestamp in every row — promised **1** and materialised **5**, because `build_date` emits `epoch_day/year/month/day/dow`, and **7** once any cell carries a time (improvise-x1ns, second instance, found by ChatGPT Codex on #97). That is an under-count of four or six on a single column, against a budget of 64. The two instances are the same defect one inference-branch apart, which is why the shape is now structural rather than remembered. `inferred_kind` reproduces `preprocess`'s match arm for arm and returns a `Forced`, and `encoded_widths` matches that **with no wildcard** — so a fourth `Forced` variant is a compile error here instead of silently taking the categorical arm, which is exactly how `Date` was missed. The `.missing` flag is added once outside the match, because every branch owes it on the same terms. The count each arm owes: | `preprocess` builds | features | |---|---| | `build_numeric` | 1 | | `build_date` | 5, or 7 if any present cell has a time | | `build_categorical` | distinct present values, **+1** for the reserved `(missing)` | plus `col.missing` wherever any cell is missing, and clamped to at least 1. **The two directions of a drift do not cost the same, and this paragraph said they did** (improvise-x1ns). It read: *the cost of the copy drifting is bounded in both directions by the exact check* — over-counting costs a needless refusal, under-counting "only lets the exact check do its job". That is wrong, and it is wrong against the argument **two paragraphs above it**: the exact check runs on `Dataset::feature_names.len()`, which exists only *after* `preprocess` has materialised the matrix. So it bounds the **error message**, never the **cost** — which is precisely the thing the preflight was introduced to bound. Over-counting is cheap; under-counting spends the memory and the O(rows²) work the preflight exists to prevent, and then reports it politely. The live instance was the `(missing)` column: the categorical arm counted `present` values only, so a forced category with N distinct present values and one record lacking that coordinate promised N and materialised N+1. `has_missing` was already computed and used in the numeric arm alone — **two arms of one match disagreeing about what a column costs**, which is improvise-zv8t's forcing-by-name / budgeting-by-index defect one level in. That one was repaired by making the ordering a data dependency and so a compile error; this one has no such shape available, so the guard is a test that asks `preprocess` what it actually built rather than asserting a number (`the_preflight_counts_the_missing_category_a_forced_column_materialises`). The previous wave's review saw a version of this and reasoned it away as *"read, not injected"* and *"bounded by the exact check"*. Both halves of that are the failure mode this file exists to catch: the first declined to run the experiment, and the second repeated the sentence above instead of checking it. **An argument that a cost is bounded is checkable, and this one was checkable the day it was written**: `encoded_width` landed 2026-09-19 and the sentence 2026-09-20, so the code it described was one commit old and one file away. `encoded_widths` remains **advisory** in the direction that is safe: it may over-count, and a column whose cells are *entirely* missing is counted as one where `preprocess` skips it altogether. It must never under-count. **That sentence was written as a proposed `kb.pl` invariant while the tree held a counter-example**, in the same change that fixed the categorical one — the date branch was missing the whole time and nothing in the hand-back noticed. An invariant proposed from the defect you just fixed generalises the *fix*, not the code; the thing that would have caught it is the question "what else does `preprocess` do that I have not mirrored", which is a question about the other function and not about the bug. `MAX_RECORDS = 20_000` bounds the *work*, which the width bound does not: a one-feature table with a million records still runs cross-validation, the final search and permutation importance over all of them. Measured at 105 ms for 2,000 records and roughly linear, so 20,000 is about a second — where a pause stops being a pause. Refusing is honest, and the limit is the caller's `FitRequest::max_records` rather than a constant, so a host with somewhere else to run a fit can raise it. `FitRequest::default()` is the interactive budget: `FitConfig::default()` is the CLI's 80 members over 600 generations, which is right for a batch tool and wrong for a synchronous command. **Measured**, release build, on the shape this module's tests use: 6 ms at 12 records, 12 ms at 200, 105 ms at 2,000 — the whole sequence, five-fold CV included. There is deliberately **no timing test**: a wall-clock assertion is flaky on shared CI and would fail for a reason that is not this code's. Re-measure after changing the numbers. ### A budget declines a column, not the table (improvise-e8no) The width preflight used to refuse the whole fit as soon as the encoded matrix was over `MAX_FEATURES`, which meant one wide category cost the user every other column as well. `unaffordable_categories` now spends the budget on the cheapest columns: it drops **category** columns, widest first, until the remaining total fits, adds each to `Overrides::ignore`, and reports each as a `Dataset::warnings` notice on `FitOutcome` — gotcha 37(a), it comes back as data and nothing prints. The refusal survives for the case drops cannot reach: a table whose *measure* columns alone exceed the budget. Five things to keep. **This is a budget decision, not an encoding one**, which is the whole of why it does not reopen improvise-zv8t. A category is still never inferred numeric, and the forcing is still unconditional; what varies with the budget is only *whether the column is present*, never *what it would encode to*. Making the forcing conditional on cardinality would have made the encoding itself depend on a budget, and the ticket rules that out. **Categories only.** A measure is data the user put in the view and silently dropping one is worse than refusing; a category that does not fit is a coordinate the fit was not going to be able to spell usefully anyway (§11's dichotomy). So a table over budget on its measures alone still refuses, with the existing message — which then advises `Label` at a document with no categories left, the one wording gap this leaves. **Dropping is by NAME, and the ordering that makes that safe is a `?`.** `Overrides::ignore` is a name set while `encoded_widths` counts by index, which is exactly improvise-zv8t's forcing-by-name / budgeting-by-index defect. It is safe only because `as_categories` — which refuses an ambiguous header — is answered *before* any drop is decided, and the guard is the **data dependency** rather than the line order: the names go into `overrides.ignore`, and `overrides` does not exist until `as_categories` returned `Ok`, so the extend cannot precede the refusal without `error[E0425]`. That is `doctrine(forcing_by_name_while_budgeting_by_index_defeats_the_budget)`'s own repair shape, reused rather than re-derived. What it does not cover, stated rather than implied: deleting `no_ambiguous_column_names` from `as_categories`. **`encoded_width` became `encoded_widths`** and the sum moved to the call site, because a budget that declines a column needs the per-column figure. Everything §14 says about the preflight being advisory, over-counting in the safe direction, and owing every `preprocess` branch is unchanged — it is the same function with its `.sum()` removed. **The drop and the `Label` repair land on the same champion**, and that is asserted rather than assumed: both remove the column before `preprocess` counts features, so the feature set is identical and the RNG draw sequence is too. The weaker claim — that both merely *fit* — would have passed against a different tree, and gotcha 37(b) is the reason to notice the difference. It is **not** the pre-zv8t champion, and that was checked rather than assumed: the threshold and the high leaf coincide with the probe recorded in the evidence file, the low leaf does not, because pre-zv8t the column was *present* as one inferred numeric feature and the draws differed. One thing the change does to the approximation, worth saying because §14 above licenses it in the other direction. `encoded_widths` over-counts in the safe direction, and an over-count used to cost a needless *refusal*; it now costs a needlessly *dropped column* — the fit still lands, over one feature fewer than it could have afforded. That is a smaller cost and a quieter one, so the case for keeping the preflight honest is unchanged rather than weakened. ### Forcing a numeric category is a *representability* choice, not a budget one improvise-zv8t made `fit_measure` force every document category to `Forced::Categorical`, and improvise-e8no records the cost: a hundred-item numeric category one-hots to a hundred columns, past `MAX_FEATURES`, and was refused where inference used to read it as one numeric column and the fit succeeded. **This section said "improvise-e8no is declined, and the refusal is correct". That verdict is superseded; the reasoning under it is not.** What is unchanged is everything about the *encoding*: a category is never inferred numeric, the forcing is never made conditional on cardinality, and ordinal and hashing stay ruled out for the reasons below. What changed is what the **budget** does when it cannot afford a column — it now declines the column and says so, rather than declining the table (see §14, "A budget declines a column"). That is a different decision in a different place, which is why the paragraphs below survive intact. The ticket rules out making the forcing conditional on cardinality, and rightly: that is inference by another name, and it makes the *encoding* depend on a budget rather than on what the document says. But the stronger reason not to reach for it is that the alternatives are not merely distasteful — they are **unstorable**, and the code already says so. A coordinate becomes an `Expr` **only by passing through `IS`**: `is_expr` is `identifier ~ sp+ ~ ^"IS" ~ sp+ ~ filter_value`, a single-value equality, and there is no `IN` and no ordering over a category's items. (`where_clause` and `inline_where` are coordinate predicates too, and also single-value equalities — but they *gate* a formula or an aggregate rather than being an `Expr`, which is gotcha 37(d)'s distinction and is why `IS` is the one that matters here.) So whatever a tree says about a category, the emitted formula names its items **one at a time**. **A first draft of this paragraph said that made one-hot the only spellable encoding. That is false**, and an adversarial review disproved it by running `parse_formula` rather than reading the grammar: `is_expr` is a `primary`, so it is an `Expr` yielding 1/0, and `cmp_op`'s operands are full `expr`s — so an ordinal split *is* spellable, as a weighted indicator sum (`IF(2020*(Year IS |2020|) + 2021*(Year IS |2021|) < 2020.5, …)`). The claim is recorded here rather than quietly deleted because it is the shape this file keeps catching: a grammar argument made by reading the rule that looked relevant instead of running the parser. So **ordinal is not ruled out by spellability, and is ruled out anyway**, for two reasons that survive. It invents a **metric the document does not assert** — a category's items carry insertion order, not magnitude, so `Year < 3.5` means "the first four items as they happen to be ordered" and stops meaning that when someone reorders them. And the width it saves does not stay saved: the *encoding* would be one feature, but the *formula* still spells one `IS` per item, so it trades a `MAX_FEATURES` problem for a formula-size problem while adding a false order. What the current emitter does with it is narrower and is not the argument: `spell()` maps `FeatureRole::Numeric` over a *category* column to `Artefact::NumericCategory`, because a bare reference to the category's name would resolve to a `_Measure` item of the same name (improvise-8xhy) — a fact about the bare-ref spelling, not about ordinals in general. **Hashing** is ruled out outright: a bucket has no name in the document, so it cannot even be refused informatively. And this dichotomy is what makes dropping the column the right budget answer rather than a loss. The "fit that previously succeeded" was one of two things, never a third. Either the tree **never split** on the numeric category — in which case the column contributed nothing to the rule, and removing it gives a fit over the same remaining features, inside the budget. Or the tree **did** split on it, and `fitted_formula` refused the result as `Artefact::NumericCategory` at storage time. So **no fit is lost that was both storable and used the column**, and the budget may drop the column knowing that. Two things that claim less than a first draft of this paragraph did, both worth keeping. Marking the category a `Label` does **not** yield *the same fit*: the feature count changes, so the RNG draw sequence in `random_tree`/`sample_thresh` changes, so the champion for a given seed changes — gotcha 37(b)'s own rule, applied to this paragraph. "The same fit" is a checkable cost claim and it is false; what is true is a fit over the same remaining features. And the dichotomy above is **argued**, from `spell()`'s exhaustive match over `FeatureRole`, not asserted: what `a_split_on_a_numeric_category_read_as_a_number_cannot_be_stored` asserts is one instance of its second branch, over one three-row fixture. That is the most a test can do here, and saying so is the difference between evidence and a flourish. The test still had no predecessor. `NumericCategory` had no coverage before it — **and it was one of three, not the only one**: grepping each `Artefact` variant and each variant's `reason()` text, `MissingFlag` and `DatePart` are covered by `the_missing_flag_and_the_date_sub_features_are_refused_too`, and **`ValueOfAMeasure` and `AbsentCoordinate` have no test at all**, before this change or after it. Two families remain uncovered; that is filed, not fixed here. What this does cost, stated rather than waved at: a user whose `Year` category really is a quantity they want fitted **as a number** has no way to say so through `fit_measure`. The document says `Year` is a category of items, and that is what is honoured. The remedy is a document change — `Year` belongs in `_Measure` if it is a measurement — and `crate::dataset` is `pub` and takes the caller's `Overrides`, so a host that wants to offer the choice can (gotcha 37d). Offering it *inside* `fit_measure` would be a new user-facing control, and nobody has asked for one. The refusal's wording was narrower than the rule and is now aligned in both places that carry it: it said *a category with a distinct value per record becomes one column per record*, which diagnoses the runaway shape and not improvise-e8no's — a category of modest cardinality that is simply wider than the budget. Both now say a category becomes one column per distinct item, **whether its items read as numbers or not**, since that clause is the whole of what changed under zv8t. The refusal is still reached, but only when dropping every category leaves the *measures* alone over budget. ### `Quality` is an enum, not an `Option` `cv_regress` pools no pairs below two folds, and scoring an empty pool gives an R² of exactly zero — which reads as "this rule is worth nothing" when what happened is that nobody checked. `Quality::Measured` carrying an all-zero `RegressionQuality` means "this tree is no better than the mean", the loudest thing a fit can say, and it must not be reachable by simply not having asked. So the enum, and a host that wants the numbers has to name the case where there are none. The fold cap is `rows / 2`, not `rows`. `fit_csv` clamps to the row count, which lets two rows become two folds of one: each tree is then fit on a *single* row — a constant — and the numbers come back meaning nothing while wearing the `Measured` label. That is the confusion `Quality` exists to prevent, moved from one row to two rather than removed. ### `fitted_measure_name` — a fit is a rule *about* the data `"{target} (fit)"`, unless the caller names another. Storing it under `target` itself would make the formula shadow the very cells it was fitted to: gotcha 7's `_Measure` items are explicit items ∪ formula targets, so the two would be one item with a computed value, and the user's typed data would stop being visible. The destination is refused when it names **any** existing item, not only an existing measure. `Expr::Ref` resolves through `Model::find_item_category`, which finds a *regular* category's item before it falls back into `_Measure` (gotcha 7) — so naming the fit after an item of some category installs cleanly and then makes every later formula that references it read the category coordinate instead. Asking the document the same question `Expr::Ref` will ask is both stronger than a measure-only check and simpler. ### `one_line` is public because the refusals need it too `.improv` pipe quoting makes `\n` legal inside a category or item name, so a document can name a measure that splits the TUI status bar in two. The whole composed status line is flattened once at the end rather than each part being sanitised and one being forgotten. The **refusals** need it as well, and they are composed by the host: several interpolate a document name, and a failure is when a user is most reading the bar. Flattening only the success path left the failure path able to split it. A related trap with no mechanism guarding it: a `rustfmt` pass over a multi-line string literal leaves runs of ten spaces mid-sentence in a message a user actually sees — invisible in the source and in every test that only asks whether a substring is present. `a_name_containing_a_newline_cannot_split_the_status_line` asserts the one-line rule for that reason. ## 15. Known gaps, filed rather than fixed **Closed: aggregate-then-route is not route-then-aggregate (improvise-4zl7).** Kept here because the shape recurs. A tree is fitted on individual **records**, and an improvise formula evaluated at an aggregating `CellKey` used to aggregate each `Expr::Ref` first and evaluate the `IF` once on the totals — the same answer for a linear formula, a different one for anything that routes, so the stored rule meant one thing on a record cell and another one row up. The emitter now wraps every tree in `SUM(…)` and `eval_agg` folds a non-`Ref` inner expression over the member records: `context/code/core.md` §13. What remains of it is a **cost**, not a defect — the fold is O(members) per evaluation per fixed-point pass and is unbenchmarked (improvise-9iyv). **Two artefact families are still unspellable (improvise-wd6n).** `col.missing` and the date sub-features stay `Spelling::Artefact` and are still refused. §11 says what each would need; they are independent of each other. **A mean-imputed feature is read raw by the formula.** `preprocess` fills a missing numeric cell with the column mean, so a champion may split on a value the training rows had and the *document* does not. The stored formula reads the document, so that record evaluates to something other than what the fit predicted for it — quietly, since the measure name resolves and only the cell is absent. `fitted_formula`'s check asks whether the document can *name* the feature, a different question from whether every record has one. Closing it means choosing between refusing any feature with a missing cell — which would refuse most real documents, imputation being why `preprocess` imputes — and teaching the formula language an `IFMISSING`-shaped primitive. A product decision of the same kind as improvise-4zl7. **A boundary value can route differently in the formula than in the fit.** Training narrows each `f64` cell to `f32`; the stored formula compares the original `f64` against the threshold. `number_text` guarantees the threshold reads back as the same `f64`, so the *threshold* is exact — but a cell value just below a threshold that `f32` rounds *to* it takes `hi` in `gp_tree::eval` and `lo` in formula evaluation. Pre-existing in the emitter (improvise-izk); the fix, one numeric representation end to end, would rewrite the bit-level characterization tests that are the only evidence the routing argument holds. **`Quality::Measured` is transductive rather than fully held out.** Preprocessing runs over **all** rows before `cv_regress` picks its folds, so an imputation mean and a categorical vocabulary are learned partly from the rows they are then validated against. And it is wider than imputation: `Spec::ranges` is computed over all rows and the *same* spec goes into every training fold, so the thresholds the search samples and the branches `simplify` prunes are bounded by a held-out minimum or maximum. **That reaches a fit over complete numeric measures too** — the only shape `fit_measure` currently stores — so "the case that stores leaks nothing" is wrong. Closing it means building the ranges and the preprocessing plan per training fold and encoding each validation fold with *that* plan, which `cv_regress` cannot express: it takes one spec and rows that are already encoded. That is an API change in `gp-tree`, and it is independent of improvise-4zl7, which turned out to need no change to how `cv_regress` is fed. **Which features a champion picks is a property of the search**, which used to mean the same command refused or succeeded depending on the seed. For the categorical family that is gone: a one-hot over a real category is spellable whichever way the search goes. It survives for `col.missing` and the date parts (improvise-wd6n), and it is the reason those are worth closing rather than living with — a refusal that depends on a seed is not a contract. ## 16. `preprocess`: what each column kind expands into Real data is not a matrix of floats. Columns are numeric, categorical or dated; cells go missing; a category is unordered and a `< thresh` split over its arbitrary codes would be meaningless. So each raw column is inferred (or forced through `Overrides`) to a `ColumnKind` and expanded: - **numeric** → one feature, missing cells mean-imputed; - **date/datetime** → ordered parts (`col.epoch_day`, `col.year`, `.month`, `.day`, `.dow`, and `.hour`/`.minute` when the column has a time), so trend and seasonality are learnable rather than one opaque token per timestamp. `epoch_day` is Howard Hinnant's `days_from_civil`; `parse_datetime` requires a 4-digit year so plain hyphenated numbers ("1-2-3") are not mistaken for dates; - **categorical** → one 0/1 column per distinct value (`col=value`), which is the correct treatment of an unordered category — the tree then splits on "is it this value?". A missing cell becomes the reserved `(missing)` level. Any column that had a missing cell also emits a `col.missing` 0/1 indicator, so **absence itself** is something the tree can split on rather than a silent substitution. A clean column emits none, or every fit would carry a constant feature. **The expansion is not reproducible from a fresh table**: imputation means and the one-hot value list are properties of the *training* rows, and a later table may not even contain some of them. `preprocess` therefore records a `ColumnSpec` per raw column and `encode_row` replays it; that pairing of fitted trees with their encoding plan is `data::Model`. A value never seen in training is simply all-zeros, not a new column. Note for §14's preflight: `build_categorical` pushes the range `(0.0, 1.0)` for every one-hot **whatever the column contains**, so a constant one-hot is *not* pruned by `simplify` — it cannot know the column is constant. A test fixture that relies on a degenerate range being pruned is relying on seed luck. ## 17. `fit_csv`: the CLI's own documentation `examples/fit_csv.rs` is the crate's *second* use case (the first was a game bot), a wholly different domain — the point being that `gp-tree` knows nothing about features or fitness. Almost none of what it does is **in** that file: preprocessing is `gp_tree::data`, the search is `gp_tree::fit`, and what is left is flag parsing, the demo dataset, and the printing. ```text cargo run -p gp-tree --example fit_csv -- path/to/data.csv cargo run -p gp-tree --example fit_csv -- --save model.json path/to/data.csv cargo run -p gp-tree --example fit_csv -- --predict model.json path/to/new.csv ``` With no path a built-in categorical dataset (the classic "play tennis" decision-tree example) is fitted, so it runs standalone. **Shape.** The header names the columns; the LAST column is the target, the rest are features. A numeric target is a **regression** (one tree, minimising MSE); a categorical target is a **classification** — one tree per class fit to a 0/1 indicator (one-vs-rest), predicting by argmax. **What a fit reports**, in order: a *generalization* metric first — `k`-fold cross-validation against a trivial baseline (predict-the-mean / majority-class), with R²/RMSE for regression and a confusion matrix plus per-class precision/recall for classification — then the final tree(s) fit on all rows, whose train error is optimistic, then a ranked **permutation feature-importance** table. **Tuning flags.** `--cv ` (folds, `< 2` disables; default 5), `--seeds ` (keep the best of `n` GA restarts; default 1), `--penalty <λ>` (per-node parsimony cost keeping trees small and general; default 0.01), `--seed ` (the RNG seed, decimal or `0x`-hex; printed by every fit and saved with `--save`). **Column directives.** `--ignore a,b` drops columns (ids, free text); `--numeric` / `--categorical` / `--date a,b` force a column's type — a numeric-looking zip code that is really a category, say. **Messy CSVs.** Missing cells (blank or `NA`/`null`/`?`/…) are mean-imputed for numeric and date columns, with a `col.missing` indicator so absence is itself learnable, and become a `(missing)` level for categorical columns. Rows missing the *target* are dropped up front, with a count on stderr. **`--save `** additionally writes a JSON `data::Model`: the fitted tree(s) plus enough of the preprocessing plan to re-encode a fresh CSV's raw cells into the exact same feature layout, **and** the seed and the `FitConfig` those flags produced — because the seed alone does not reproduce a fit (§9). **`--predict `** loads that model and scores each row of a CSV of raw feature columns (same names as training, any order, target column optional and ignored), printing one prediction per row — plus a softmax `confidence` column for classification.