improvise/context/code/gp.md

gp-tree and improvise-gp

The reasoning that used to live in the doc comments of crates/gp-tree/** and crates/improvise-gp/**. Written for a reader with the code open: the code says what happens, this says why, and why the obvious alternative is wrong.

Bug narratives keyed by test name are in context/test-evidence/gp.md. The condensed version is repo-map gotcha 37.

---

1. Two crates, one seam

gp-tree is problem-agnostic: a Tree is a regression tree over a fixed-width f32 feature vector, and the crate owns the representation, the GP operators, simplify, render, one generation (breed), the search (fit), and the optional raw-table bridge (data). It knows nothing about what the features mean or how a candidate is scored — the caller supplies a Spec (per-feature ranges and names) and, outside fit, its own fitness. Determinism is the caller's too: the operators take an Rng trait implemented over a seeded generator.

improvise-gp is the one seam between that crate and improvise. It cannot live in gp-tree — that inverts the dependency and drags improvise into a generic GP crate — and it cannot live in improvise-formula, which must not know GP exists. It holds three things:

• lib.rs — the emitter: a fitted Tree as improvise formula text.
• dataset.rs — a View's records as the feature matrix a fit consumes, the improvise-side counterpart of preprocess.
• fit.rs — one fit end to end: records in, an improvise Formula plus the evidence for believing it out.

fit.rs is here rather than in a Cmd because none of it is terminal-specific. A browser fitting the same measure over the same document must run exactly these steps in exactly this order, or it gets a different tree from the same seed. improvise-command/src/command/cmd/fit.rs is the thin terminal wrapper over it.

2. gp-tree returns data and never prints

A decision the user should hear about comes back as data: Dataset::warnings carries the columns preprocess dropped for being entirely missing, fit.rs hands back pooled (prediction, actual) pairs and a score per feature rather than tables. examples/fit_csv.rs owns every println! and every eprintln!, and that is the only reason its output stayed byte-identical across these moves. Adding an eprintln! to fit.rs or data.rs is the regression, not a convenience.

The reason is not tidiness: a library does not own its caller's stderr. A TUI writing to an alternate screen, a wasm build, or a caller that wants the notice in its own status line can none of them suppress, redirect or translate an eprintln!. preprocess used to do exactly that, which was correct while this code was examples/fit_csv.rs.

improvise-gp inherits the rule. FitOutcome::status_line is a rendering of the outcome data, offered so a second front end does not compose its own — the same argument that put flatten_cat_tree in the shared layer — and it is a returned String, never a println!.

3. The vendored RNG, and why its constants are a promise

SplitMix64 is ~20 lines and no dependency. The Rng trait exists so a caller can bring its own generator, and every caller so far wrote out this same one.

A fit is not a throwaway: it writes a rule into a user's document, so re-running it on the same data must produce the same rule, and a bug report must be reconstructible from its seed alone. That is worth vendoring rather than depending — the point is reproducibility, not statistical quality, and SplitMix64's is ample for picking nodes and thresholds.

SplitMix64::seed() keeps the seed the run started from, unchanged as the generator advances, so a caller can show it and record it. A generator that has advanced past its seed cannot answer that question.

next_u64 is public because splitmix64_matches_the_reference_vectors pins it. The constants are the algorithm, not an implementation detail: every seed already printed by a fit and every data::Model seed already saved is a promise that these numbers come back.

4. Spec::ranges is a soundness precondition

ranges[i] must be a superset of the values feature i can actually take. A too-narrow range makes simplify prune a branch that is reachable, silently changing the tree's outputs. preprocess derives the ranges from the columns themselves, which is what keeps this true; every_training_value_lies_inside_its_feature_range pins that it stays derived.

simplify seeds each feature's feasible interval with its real range rather than ±∞, so an out-of-range threshold prunes. The interval is half-open [lo, hi) but a feature can hit its max, so hi is nudged by EPS = 1e-3 and a split at the max (x < max) stays live. EPS is safely below typical threshold granularity.

5. Row<T>: an invariant made unrepresentable

Row<T> pairs a feature vector with what it is supposed to predict. The target is generic because the two supervised shapes differ only in it — Row<f32> for regression, Row<usize> for a class code — and every scoring function then reads one slice instead of two.

That is the point of the type. Parallel x/y slices could be handed in at different lengths, and zip answered by silently scoring the shorter prefix while a denominator taken from the longer one made the mean too small. There is no length to get wrong here, so there is nothing to validate and no Result to thread through five signatures (improvise-dkqn, doctrine(an_invariant_is_made_unrepresentable_not_validated) — the same answer prune_axis_items got).

The pairing is made at the boundary that builds the data. preprocess walks one row index and the rows live inside Target; a Dataset carrying rows and target as two collections would move the mismatched length one level up, into every caller.

mean_over(total, n) is the one place the empty row set is answered. Every mean in fit.rs used to reach its own 0.0 / 0.0 on an empty slice: a NaN that propagates through FitConfig::cost, sorts unpredictably against real costs, and reaches a caller as a number rather than as an error. Scoring no rows finds no error, so the answer is zero, stated once instead of in mse, misclass_rate and standardize separately.

Changing any of this needs bit-level evidence. tests/scoring_known_answers.rs pins every scoring path as f32 bit patterns plus a next_u64 witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. See context/test-evidence/gp.md.

6. folds and cv_folds answer different questions

folds(n, k) splits 0..n into k pieces after a deterministic shuffle. k arrives from a host — a CLI flag, in this crate's only caller — so zero is a number to answer, not an impossible state: out[pos % 0] is a public library function taking its host down with it. One fold is the answer the shape already gives, and fit spells the same max(1) over restarts.

cv_folds(n, k) yields nothing below two. A partition into one piece is still a partition, but a cross-validation over one fold is not a cross-validation: the single fold holds every index, the training complement is empty, and both cross-validators would fit a tree on nothing and then let it predict every row. Yielding no folds makes the pooled pairs empty, which is the true count of such predictions — and no fold is drawn, so the shuffle never touches the caller's generator either (improvise-ry89).

complement(n, test) builds a bool mask rather than asking test.contains(i) per row, which was a scan of the fold per row and so quadratic in the row count (improvise-t3ux). It yields the same indices in the same order, which is what every downstream draw from the generator depends on.

7. standardize / denormalize: the affine map must be folded back

The crate's leaf scores live in a ±5 range, so a target in the thousands cannot be reached by leaf mutation and standardize is not optional. Its consequence is that the champion predicts standard deviations.

fit_csv answers that by printing ≈ score × std + mean beside the tree. A document has nowhere to put that, so improvise-gp::fit_measure folds the affine map back into the leaves with denormalize before a single character of formula text is spelled. Omitting it emits a perfectly well-formed formula whose every cell is out by an affine map — wrong in the quiet way, since small numbers in the right shape look like an answer.

denormalize touches leaves only, and that is the whole reason it is sound: standardize rescales the target, so a threshold is already in its feature's own units and re-scaling it would be the bug.

8. RegressionQuality carries its own baseline

A tree with a plausible-looking RMSE that is no better than always predicting the mean is not a weak rule, it is no rule — and it is worse than no rule at all, because it is one a user will believe. So the comparison is part of the answer rather than something a caller may forget to compute. r2 is not clamped: negative means the fit does worse than the mean, which is a real outcome, and clamping would make "learned nothing" and "actively misleading" read alike.

importance_ranking reports each feature's share of the total importance. The all-zero case returns early rather than flooring the divisor: fit_csv used .max(1e-9), which is right for the only input it ever had and wrong as a contract — a lone importance of 1e-10 then divides by the floor rather than by itself and reports 10%, where the one feature the model uses carries all of it. Importances this small are ordinary, being error differences over a standardized target, so the shares have to sum to 100% at every scale.

9. A fit takes two things to reproduce — and here, four

data::Model records seed: Option<u64> and config: Option<FitConfig>, because the same seed under a different population/generations/elite/penalty lands on a different tree (improvise-ksst). Both are Option for one reason: every FitConfig value and the seed 0 are legal, so nothing can double as "unknown", and anything reporting a sentinel would state a seed that never ran.

Note #[serde(default)] on an Option field is documentation, not the mechanism — serde defaults it anyway, so deleting the attribute fails no test. A backward-compatibility test that only deletes the attribute is not discriminating; aim it at an absent field loading as the default instead.

improvise-gp::FitOutcome carries the whole FitRequest, not the pair. FitRequest is { seed, config, cv_folds, importance_repeats, max_records }, and every stochastic step draws from one generator in a fixed order — cross-validate, then fit, then explain. So cv_folds decides how far that generator has advanced before fit draws from it, and therefore changes the champion and not merely the evidence for it. Recording two of four fields documented the reproducibility contract without meeting it.

10. The emitter: quoting, and the threshold's exact decimal

A regression tree is a nested conditional, and improvise formulas already have IF(cond, then, else), so a fit can be written straight into a user's model as an ordinary formula with nothing GP-specific left in the file:

Score = SUM(IF(humidity < 77.5, 1.25, IF(outlook IS rainy < 0.5, -0.5, 2)))

(An earlier form of that sample sat in improvise-gp/src/lib.rs's module doc inside a ```text fence. rustdoc does not execute a text fence, so it was prose wearing a fence rather than a doctest, and the two-line cap applies to it like anything else.)

Two things in it are improvise-4zl7's, and both are load-bearing. The outer SUM is the route-then-aggregate rule made syntax: a tree is fitted on records, and SUM(e) is improvise's existing word for "fold e over the member records" (context/code/core.md §13). Without it the formula routes on the totals one row up and means something else there; with it, a record row is a one-member fold and answers the rule itself. outlook IS rainy is the one-hot outlook=rainy said as a coordinate predicate rather than as a reference to an item no document has — see §11 for which features get which spelling.

Two properties the emitter owes its caller:

It parses. Feature names come from the caller's data, so they carry spaces, operators, pipes and keyword collisions. Every name goes through quote_name — improvise_formula::quote, the one home for pipe quoting (gotcha 34); improvise-gp re-exports it because it was part of this crate's public surface. A formula target is unquoted on the way back in: it names a _Measure item, so a target keeping its delimiters writes the fit into an item literally named |Total Revenue|.

It routes identically. number_text emits the shortest decimal that reads back as the same f64 the f32 widens to — not the f32's full decimal expansion, which runs to 149 fractional digits, and not a shortest-round-trip-to-f32 rendering. Widening an f32 to f64 is exact, so text that reads back as that f64 makes feature < thresh give the same answer in f64 formula evaluation as in f32 gp_tree::eval. A short rendering (0.1 for 0.1f32) sits up to half an ULP away, and a row whose feature value equals the threshold exactly then falls on the other side of the split.

number_text panics on a non-finite value. The grammar spells neither, and both fail quietly rather than loudly: "inf" parses as an f64, so the round-trip loop would return text no formula parser accepts, and NaN matches no width at all. Trees built by this crate's own operators over a finite Spec are finite, so this is a precondition rather than a runtime condition — but see §12, because a .improv cell holding inf makes it reachable from a document.

expr_text and mark_features share one precondition and spell it the same way (improvise-hkt6): a bare used[*feat] fails with index out of bounds, which names neither the tree nor the spec. Two sites, one failure, one sentence. used_features is the single walk both the document check and the plan check read, because walking the tree twice two ways is how those answers drift apart.

11. A fit is storable only when the document can name its features

preprocess invents Region=North for a one-hot, Cost.missing for a missing indicator, when.year for a date part. Those are encoding artefacts, not items of the document, and Expr::Ref resolves through Model::find_item_category — so a tree splitting on one emitted a perfectly well-formed formula whose every cell then evaluated to Error("ref:Region=North"), with nothing between the fit and the file to say so (improvise-8xhy). improvise-4zl7 closed the categorical third of that by giving the emitter a vocabulary for a coordinate, and by giving the evaluator a rule that makes such a formula mean the same thing at every level of aggregation (context/code/core.md §13).

Features: one entry per feature index, read off the plan

Features::of(&Dataset, categories) walks gp_tree::data::feature_roles — the one statement of preprocess's expansion order, which encode_row drives off too — and pairs each role with the name preprocess gave it. A tree's feat indexes Features directly, so nothing anywhere reconstructs index → role from the feature names. Spec::names is &[&str] and could not have carried this; the earlier check matched names against the raw plan instead and said in this section that it was avoiding a second implementation of the expansion. Now there is no second implementation to avoid.

Each feature gets one of three Spellings, decided by role and by column, never by the shape of the name:

role	column	spelling
Numeric	a measure	Expr::Ref(measure)
OneHot { value }	a category, value != "(missing)"	Cat IS Item
anything else		Artefact

The two exclusions are not fussiness. A one-hot over a measure column names no coordinate at all — the record's value for that measure is text, and Cat IS Item asks about coordinates. (missing) is the absence of a coordinate, which Expr::Is reads as 0 by §13's unwrap_or(false), so the one spelling that would be wrong is exactly the one omitted. A Numeric role over a category column is the trap the previous version of this section called "the hole a name cannot close": a category whose items all parse as numbers produces a feature named after the category, and Expr::Ref would resolve it to a _Measure item of the same name — a different column. Both halves now fall out of is_category, which is a lookup in the header's own first category_columns entries.

Duplicate header names are still refused outright, by no_ambiguous_column_names and before anything else: is_category decides by name, so a name that means two columns is a spelling with two meanings.

Dataset::columns is not index-aligned with the header — preprocess continues past an entirely-missing column without pushing a ColumnSpec, so one empty category shifts every later spec left. FeatureRole::column() indexes columns, not the header, so that shift is carried rather than tripped over.

What fitted_formula still asks the document

Two refusals, in order. An Artefact is refused with no reference to the document at all — nothing could make Cost.missing spellable. Everything else is asked of the document: a Measure must be find_item_category-resolvable and a Coordinate must name a category the model has. A measure literally named Region=North is a legal document and is accepted, because the check never looks at the name's shape — and that is also why the date parts and the .missing flags stay covered without a second copy of preprocess's naming rules.

formula_text still spells whatever it is handed, being the tree's own text; an Artefact reaching it is a precondition failure and panics, the same split number_text makes for a non-finite value.

The two thirds this still does not close

col.missing means "this record has no value for measure X" — a different vocabulary, and one the formula language has no word for at all. The date sub-features (when.year, when.month) need date arithmetic improvise does not have. Both stay Artefact, both stay refused, and both are filed as improvise-wd6n rather than smuggled in — separately from each other, because they need different vocabulary.

12. A fit is persisted as its formula, and nothing else (improvise-7na)

.improv stores the emitted formula and no serialized gp_tree::data::Model. The grammar is unchanged by this decision, deliberately and not by omission.

The ticket offered three options: (a) formula only, refit to score new data; (b) the data::Model as a JSON blob; (c) seed + config so the fit reproduces. The answer is (a), with (c) already available in gp-tree for whoever wants provenance later.

Against (b), two arguments. The general one is the second half of gotcha 41: a blob is a second copy of information the document already carries — the tree is the formula — and nothing keeps the two in agreement. A user who edits the formula by hand, or a cell it was fitted on, leaves a blob that disagrees with the file it sits in, and a round-trip test over the blob cannot see that, because it never compares the two. (The first half of gotcha 41 does not transfer: an opaque blob inside one new grammar production is not a second implementation of the grammar.) The specific one is layering: Tree and FitConfig are gp-tree types, so improvise-io would gain a dependency on gp-tree and .improv would be obliged to track a third crate's serde layout forever — inverting exactly the dependency improvise-gp exists to keep out of the core.

What (b) was for, and why it is not needed. The encoding plan exists so fit_csv --predict can score a row that arrives as untyped CSV text. improvise has no such row: a new row is a record in the same document, and recompute_formulas scores it through the stored formula from its own measures.

Why (c) is not written to the file either. data::Model already records seed and config, so reproducing a fit is solved wherever that value is in hand; and unlike the CSV case the training data is the document. What is missing is a reader — writing provenance now would add a grammar production and a round-trip obligation for metadata nothing consumes. Provenance is the fit command's decision, alongside how it words the fit for the user.

A premise the ticket got wrong, and it is the bigger finding. "The emitted formula is enough to recompute existing cells" holds only for a fit over numeric measures — §11 is that defect, and it is a defect in what a fit may emit, not an argument for persisting the model.

13. dataset.rs: what a view shows, as a feature matrix

The shape is not invented here. Records mode is already a feature matrix: a row is one coordinate tuple — every cell sharing all coordinates but _Measure — and each measure is its own column (gotcha 1). That is exactly "a sample, and its features". So the module enumerates the same rows records mode does, lays them out as the string table preprocess takes, and lets preprocess own type inference, one-hot expansion, imputation and the ranges.

What it does not reuse is GridLayout: reaching the records path means putting _Index on Row and _Dim on Column, which is a different view from the one the user is looking at. RecordRow::group is public, so the traversal here is the same two steps the layout takes, against the user's own view.

Four selection rules, each of which could reasonably have gone the other way:

• Rows are scoped by View::filtering_cats() — Page ∪ Filter — not by categories_on(Axis::Page). Both constrain which records are visible; only Page is navigable (gotcha 27), and scoping by the navigable half would silently fit against rows a drill had already excluded.
• Hidden items scope the rows and the feature columns. The mental model is "fit what I can see", and hiding an item is an explicit statement that it is not of interest. An explicitly named target is exempt: the caller asked for it by name.
• Every regular category is a feature, whatever axis it sits on — including Axis::None. A category's coordinate is part of what makes a record distinct, so dropping a None category would leave two different rows with identical feature vectors and different targets: noise the fit can neither learn nor report. Label categories are excluded, as they are from records mode, by being not is_regular — they are high-cardinality per-row text and one-hot into a perfect memoriser.
• No formula-derived value ever enters the matrix. Formula targets are not features (a formula is a function of the other measures, so it leaks the target whenever the target is one of its inputs), and a formula target is refused as the target — a formula already is the rule, so fitting a tree to reproduce it from its own inputs is trivially perfect and says nothing. One rule covers both halves, with no dependency analysis to get wrong.

_Measure is a filtering category and must not filter a fit. That exception in filter_coords is not a special case, it is the difference between a grid and a feature matrix: _Measure is the coordinate the columns are spread across, so pinning it to one item is not "which records am I looking at" but "throw away every column except this one". Workbook::new's default view puts it on Axis::Page, so that was the behaviour on a freshly opened document (improvise-vx3).

Two smaller decisions. Records are sorted by their coordinate tuple. That began as a workaround for matching_cells walking a HashSet; since improvise-shyt it yields insertion order, so the sort is no longer what buys stability. It is kept because a fit's row order should be a property of the data rather than of the typing history that entered it. And a Number cell is rendered with f64::to_string, never format_f64: display rounding is view-only and must not reach a fit (gotcha 8). An Error cell has no value, so it is spelled as missing and picks up the same imputation and .missing indicator an absent cell would.

14. improvise-gp::fit: budgets, and the enum that refuses to guess

Two budgets, and why the preflight is advisory

FitConfig bounds the search; nothing bounded the width, which preprocess can multiply without limit (one column per distinct value of a categorical column). Permutation importance alone is repeats × rows × features tree evaluations, on the thread that draws the next frame.

MAX_FEATURES = 64 is comfortably above any hand-built pivot table — this repo's own 18,000-cell benchmark encodes far fewer — and comfortably below the one-column-per-row shape that is really a mis-typed category. The message names the fix (Label), because that is what the user actually wants.

encoded_width is the advisory preflight, run on the raw table before preprocess materialises anything. Checking width afterwards guards the wrong side of the cost: a category with a distinct value per record allocates a rows × rows one-hot during preprocessing, so the refusal arrives after the memory and the O(rows²) work have been spent. It approximates preprocess's type inference rather than sharing it — a column whose present cells all parse as a number is one feature (two with a missing indicator), anything else is counted at its distinct-value count. Copying the predicate is the lesser evil; the alternative is exporting the inference and calling this a second implementation of it. The cost of the copy drifting is bounded in both directions by the exact check on Dataset::feature_names.len(): over-counting costs a refusal the exact check would not have made, under-counting only lets the exact check do its job.

MAX_RECORDS = 20_000 bounds the work, which the width bound does not: a one-feature table with a million records still runs cross-validation, the final search and permutation importance over all of them. Measured at 105 ms for 2,000 records and roughly linear, so 20,000 is about a second — where a pause stops being a pause. Refusing is honest, and the limit is the caller's FitRequest::max_records rather than a constant, so a host with somewhere else to run a fit can raise it.

FitRequest::default() is the interactive budget: FitConfig::default() is the CLI's 80 members over 600 generations, which is right for a batch tool and wrong for a synchronous command. Measured, release build, on the shape this module's tests use: 6 ms at 12 records, 12 ms at 200, 105 ms at 2,000 — the whole sequence, five-fold CV included. There is deliberately no timing test: a wall-clock assertion is flaky on shared CI and would fail for a reason that is not this code's. Re-measure after changing the numbers.

Quality is an enum, not an Option

cv_regress pools no pairs below two folds, and scoring an empty pool gives an R² of exactly zero — which reads as "this rule is worth nothing" when what happened is that nobody checked. Quality::Measured carrying an all-zero RegressionQuality means "this tree is no better than the mean", the loudest thing a fit can say, and it must not be reachable by simply not having asked. So the enum, and a host that wants the numbers has to name the case where there are none.

The fold cap is rows / 2, not rows. fit_csv clamps to the row count, which lets two rows become two folds of one: each tree is then fit on a single row — a constant — and the numbers come back meaning nothing while wearing the Measured label. That is the confusion Quality exists to prevent, moved from one row to two rather than removed.

fitted_measure_name — a fit is a rule about the data

"{target} (fit)", unless the caller names another. Storing it under target itself would make the formula shadow the very cells it was fitted to: gotcha 7's _Measure items are explicit items ∪ formula targets, so the two would be one item with a computed value, and the user's typed data would stop being visible.

The destination is refused when it names any existing item, not only an existing measure. Expr::Ref resolves through Model::find_item_category, which finds a regular category's item before it falls back into _Measure (gotcha 7) — so naming the fit after an item of some category installs cleanly and then makes every later formula that references it read the category coordinate instead. Asking the document the same question Expr::Ref will ask is both stronger than a measure-only check and simpler.

one_line is public because the refusals need it too

.improv pipe quoting makes \n legal inside a category or item name, so a document can name a measure that splits the TUI status bar in two. The whole composed status line is flattened once at the end rather than each part being sanitised and one being forgotten.

The refusals need it as well, and they are composed by the host: several interpolate a document name, and a failure is when a user is most reading the bar. Flattening only the success path left the failure path able to split it.

A related trap with no mechanism guarding it: a rustfmt pass over a multi-line string literal leaves runs of ten spaces mid-sentence in a message a user actually sees — invisible in the source and in every test that only asks whether a substring is present. a_name_containing_a_newline_cannot_split_the_status_line asserts the one-line rule for that reason.

15. Known gaps, filed rather than fixed

Closed: aggregate-then-route is not route-then-aggregate (improvise-4zl7). Kept here because the shape recurs. A tree is fitted on individual records, and an improvise formula evaluated at an aggregating CellKey used to aggregate each Expr::Ref first and evaluate the IF once on the totals — the same answer for a linear formula, a different one for anything that routes, so the stored rule meant one thing on a record cell and another one row up. The emitter now wraps every tree in SUM(…) and eval_agg folds a non-Ref inner expression over the member records: context/code/core.md §13. What remains of it is a cost, not a defect — the fold is O(members) per evaluation per fixed-point pass and is unbenchmarked (improvise-9iyv).

Two artefact families are still unspellable (improvise-wd6n). col.missing and the date sub-features stay Spelling::Artefact and are still refused. §11 says what each would need; they are independent of each other.

A mean-imputed feature is read raw by the formula. preprocess fills a missing numeric cell with the column mean, so a champion may split on a value the training rows had and the document does not. The stored formula reads the document, so that record evaluates to something other than what the fit predicted for it — quietly, since the measure name resolves and only the cell is absent. fitted_formula's check asks whether the document can name the feature, a different question from whether every record has one. Closing it means choosing between refusing any feature with a missing cell — which would refuse most real documents, imputation being why preprocess imputes — and teaching the formula language an IFMISSING-shaped primitive. A product decision of the same kind as improvise-4zl7.

A boundary value can route differently in the formula than in the fit. Training narrows each f64 cell to f32; the stored formula compares the original f64 against the threshold. number_text guarantees the threshold reads back as the same f64, so the threshold is exact — but a cell value just below a threshold that f32 rounds to it takes hi in gp_tree::eval and lo in formula evaluation. Pre-existing in the emitter (improvise-izk); the fix, one numeric representation end to end, would rewrite the bit-level characterization tests that are the only evidence the routing argument holds.

Quality::Measured is transductive rather than fully held out. Preprocessing runs over all rows before cv_regress picks its folds, so an imputation mean and a categorical vocabulary are learned partly from the rows they are then validated against. And it is wider than imputation: Spec::ranges is computed over all rows and the same spec goes into every training fold, so the thresholds the search samples and the branches simplify prunes are bounded by a held-out minimum or maximum. That reaches a fit over complete numeric measures too — the only shape fit_measure currently stores — so "the case that stores leaks nothing" is wrong. Closing it means building the ranges and the preprocessing plan per training fold and encoding each validation fold with that plan, which cv_regress cannot express: it takes one spec and rows that are already encoded. That is an API change in gp-tree, and it is independent of improvise-4zl7, which turned out to need no change to how cv_regress is fed.

Which features a champion picks is a property of the search, which used to mean the same command refused or succeeded depending on the seed. For the categorical family that is gone: a one-hot over a real category is spellable whichever way the search goes. It survives for col.missing and the date parts (improvise-wd6n), and it is the reason those are worth closing rather than living with — a refusal that depends on a seed is not a contract.

16. preprocess: what each column kind expands into

Real data is not a matrix of floats. Columns are numeric, categorical or dated; cells go missing; a category is unordered and a < thresh split over its arbitrary codes would be meaningless. So each raw column is inferred (or forced through Overrides) to a ColumnKind and expanded:

• numeric → one feature, missing cells mean-imputed;
• date/datetime → ordered parts (col.epoch_day, col.year, .month, .day, .dow, and .hour/.minute when the column has a time), so trend and seasonality are learnable rather than one opaque token per timestamp. epoch_day is Howard Hinnant's days_from_civil; parse_datetime requires a 4-digit year so plain hyphenated numbers ("1-2-3") are not mistaken for dates;
• categorical → one 0/1 column per distinct value (col=value), which is the correct treatment of an unordered category — the tree then splits on "is it this value?". A missing cell becomes the reserved (missing) level.

Any column that had a missing cell also emits a col.missing 0/1 indicator, so absence itself is something the tree can split on rather than a silent substitution. A clean column emits none, or every fit would carry a constant feature.

The expansion is not reproducible from a fresh table: imputation means and the one-hot value list are properties of the training rows, and a later table may not even contain some of them. preprocess therefore records a ColumnSpec per raw column and encode_row replays it; that pairing of fitted trees with their encoding plan is data::Model. A value never seen in training is simply all-zeros, not a new column.

Note for §14's preflight: build_categorical pushes the range (0.0, 1.0) for every one-hot whatever the column contains, so a constant one-hot is not pruned by simplify — it cannot know the column is constant. A test fixture that relies on a degenerate range being pruned is relying on seed luck.

17. fit_csv: the CLI's own documentation

examples/fit_csv.rs is the crate's second use case (the first was a game bot), a wholly different domain — the point being that gp-tree knows nothing about features or fitness. Almost none of what it does is in that file: preprocessing is gp_tree::data, the search is gp_tree::fit, and what is left is flag parsing, the demo dataset, and the printing.

cargo run -p gp-tree --example fit_csv -- path/to/data.csv
cargo run -p gp-tree --example fit_csv -- --save model.json path/to/data.csv
cargo run -p gp-tree --example fit_csv -- --predict model.json path/to/new.csv

With no path a built-in categorical dataset (the classic "play tennis" decision-tree example) is fitted, so it runs standalone.

Shape. The header names the columns; the LAST column is the target, the rest are features. A numeric target is a regression (one tree, minimising MSE); a categorical target is a classification — one tree per class fit to a 0/1 indicator (one-vs-rest), predicting by argmax.

What a fit reports, in order: a generalization metric first — k-fold cross-validation against a trivial baseline (predict-the-mean / majority-class), with R²/RMSE for regression and a confusion matrix plus per-class precision/recall for classification — then the final tree(s) fit on all rows, whose train error is optimistic, then a ranked permutation feature-importance table.

Tuning flags. --cv <k> (folds, < 2 disables; default 5), --seeds <n> (keep the best of n GA restarts; default 1), --penalty <λ> (per-node parsimony cost keeping trees small and general; default 0.01), --seed <n> (the RNG seed, decimal or 0x-hex; printed by every fit and saved with --save).

Column directives. --ignore a,b drops columns (ids, free text); --numeric / --categorical / --date a,b force a column's type — a numeric-looking zip code that is really a category, say.

Messy CSVs. Missing cells (blank or NA/null/?/…) are mean-imputed for numeric and date columns, with a col.missing indicator so absence is itself learnable, and become a (missing) level for categorical columns. Rows missing the target are dropped up front, with a count on stderr.

--save <path> additionally writes a JSON data::Model: the fitted tree(s) plus enough of the preprocessing plan to re-encode a fresh CSV's raw cells into the exact same feature layout, and the seed and the FitConfig those flags produced — because the seed alone does not reproduce a fit (§9).

--predict <model.json> <data.csv> loads that model and scores each row of a CSV of raw feature columns (same names as training, any order, target column optional and ignored), printing one prediction per row — plus a softmax confidence column for classification.