787e354b9b
The `build` job fails on main and on every branch merged from it: doc-site: context/code/pr-review.md is neither published nor a named omission NOT a merge loss -- scripts/build-doc-site.mjs in this branch is byte-identical to main's, and neither lists the file. PR #96 added context/code/pr-review.md without a manifest line, which is exactly the gate improvise-res9 added: a doc added without a line publishes nothing and says nothing, so the generator refuses. Omitted rather than published, and the line is cloud-agents.md's rather than tooling.md's. tooling.md explains checkers a reader of the code will meet; pr-review.md is about THIS REPOSITORY'S PROCESS -- where a Codex review appears, how to tell it from our own agents' comments, why absence is not approval. Nobody visiting a website is about to read a PR thread. Both homes updated, because an omission is restated beside the manifest AND in doc-site.md section 2, and the prose there counted them ("Neither is referenced...") so it went stale at three. This strays into scripts/, which is Wave N's ground. Flagged in the hand-back; it is a one-line data entry either way if the orchestrator would rather publish it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
189 lines
79 KiB
HTML
189 lines
79 KiB
HTML
<!doctype html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||
<title>gp-tree and improvise-gp — improvise docs</title>
|
||
<link rel="stylesheet" href="../style.css" />
|
||
</head>
|
||
<body>
|
||
<div class="shell">
|
||
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html" class="here">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
||
<main>
|
||
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/code/gp.md">context/code/gp.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#gp-tree-and-improvise-gp">gp-tree and improvise-gp</a></li><li class="l2"><a href="#1-two-crates-one-seam">1. Two crates, one seam</a></li><li class="l2"><a href="#2-gp-tree-returns-data-and-never-prints">2. gp-tree returns data and never prints</a></li><li class="l2"><a href="#3-the-vendored-rng-and-why-its-constants-are-a-promise">3. The vendored RNG, and why its constants are a promise</a></li><li class="l2"><a href="#4-spec-ranges-is-a-soundness-precondition">4. Spec::ranges is a soundness precondition</a></li><li class="l2"><a href="#5-row-an-invariant-made-unrepresentable">5. Row<T>: an invariant made unrepresentable</a></li><li class="l2"><a href="#6-folds-and-cv-folds-answer-different-questions">6. folds and cv_folds answer different questions</a></li><li class="l2"><a href="#7-standardize-denormalize-the-affine-map-must-be-folded-back">7. standardize / denormalize: the affine map must be folded back</a></li><li class="l2"><a href="#8-regressionquality-carries-its-own-baseline">8. RegressionQuality carries its own baseline</a></li><li class="l2"><a href="#9-a-fit-takes-two-things-to-reproduce-and-here-four">9. A fit takes two things to reproduce — and here, four</a></li><li class="l2"><a href="#10-the-emitter-quoting-and-the-threshold-s-exact-decimal">10. The emitter: quoting, and the threshold's exact decimal</a></li><li class="l2"><a href="#11-a-fit-is-storable-only-when-the-document-can-name-its-features">11. A fit is storable only when the document can name its features</a></li><li class="l3"><a href="#features-one-entry-per-feature-index-read-off-the-plan">Features: one entry per feature index, read off the plan</a></li><li class="l3"><a href="#what-fitted-formula-still-asks-the-document">What fitted_formula still asks the document</a></li><li class="l3"><a href="#the-two-families-improvise-wd6n-declines-separately">The two families improvise-wd6n declines, separately</a></li><li class="l2"><a href="#12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na">12. A fit is persisted as its formula, and nothing else (improvise-7na)</a></li><li class="l2"><a href="#13-dataset-rs-what-a-view-shows-as-a-feature-matrix">13. dataset.rs: what a view shows, as a feature matrix</a></li><li class="l2"><a href="#14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess">14. improvise-gp::fit: budgets, and the enum that refuses to guess</a></li><li class="l3"><a href="#two-budgets-and-why-the-preflight-is-advisory">Two budgets, and why the preflight is advisory</a></li><li class="l3"><a href="#forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one">Forcing a numeric category is a representability choice, not a budget one</a></li><li class="l3"><a href="#quality-is-an-enum-not-an-option">Quality is an enum, not an Option</a></li><li class="l3"><a href="#fitted-measure-name-a-fit-is-a-rule-about-the-data">fitted_measure_name — a fit is a rule about the data</a></li><li class="l3"><a href="#one-line-is-public-because-the-refusals-need-it-too">one_line is public because the refusals need it too</a></li><li class="l2"><a href="#15-known-gaps-filed-rather-than-fixed">15. Known gaps, filed rather than fixed</a></li><li class="l2"><a href="#16-preprocess-what-each-column-kind-expands-into">16. preprocess: what each column kind expands into</a></li><li class="l2"><a href="#17-fit-csv-the-cli-s-own-documentation">17. fit_csv: the CLI's own documentation</a></li></ul></details><h1 id="gp-tree-and-improvise-gp"><code>gp-tree</code> and <code>improvise-gp</code></h1>
|
||
<p>The reasoning that used to live in the doc comments of <code>crates/gp-tree/**</code> and <code>crates/improvise-gp/**</code>. Written for a reader with the code open: the code says what happens, this says why, and why the obvious alternative is wrong.</p>
|
||
<p>Bug narratives keyed by test name are in <a href="../test-evidence/gp.html"><code>context/test-evidence/gp.md</code></a>. The condensed version is repo-map gotcha 37.</p>
|
||
<hr />
|
||
<h2 id="1-two-crates-one-seam"><a class="anchor" href="#1-two-crates-one-seam">#</a>1. Two crates, one seam</h2>
|
||
<p><code>gp-tree</code> is problem-agnostic: a <code>Tree</code> is a regression tree over a fixed-width <code>f32</code> feature vector, and the crate owns the representation, the GP operators, <code>simplify</code>, <code>render</code>, one generation (<code>breed</code>), the search (<code>fit</code>), and the optional raw-table bridge (<code>data</code>). It knows nothing about <em>what</em> the features mean or <em>how</em> a candidate is scored — the caller supplies a <code>Spec</code> (per-feature ranges and names) and, outside <code>fit</code>, its own fitness. Determinism is the caller's too: the operators take an <code>Rng</code> trait implemented over a seeded generator.</p>
|
||
<p><code>improvise-gp</code> is the one seam between that crate and improvise. It cannot live in <code>gp-tree</code> — that inverts the dependency and drags improvise into a generic GP crate — and it cannot live in <code>improvise-formula</code>, which must not know GP exists. It holds three things:</p>
|
||
<ul>
|
||
<li><code>lib.rs</code> — the emitter: a fitted <code>Tree</code> as improvise formula text.</li>
|
||
<li><code>dataset.rs</code> — a <code>View</code>'s records as the feature matrix a fit consumes, the improvise-side counterpart of <code>preprocess</code>.</li>
|
||
<li><code>fit.rs</code> — one fit end to end: records in, an improvise <code>Formula</code> plus the evidence for believing it out.</li>
|
||
</ul>
|
||
<p><code>fit.rs</code> is here rather than in a <code>Cmd</code> because none of it is terminal-specific. A browser fitting the same measure over the same document must run exactly these steps in exactly this order, or it gets a different tree from the same seed. <code>improvise-command/src/command/cmd/fit.rs</code> is the thin terminal wrapper over it.</p>
|
||
<h2 id="2-gp-tree-returns-data-and-never-prints"><a class="anchor" href="#2-gp-tree-returns-data-and-never-prints">#</a>2. <code>gp-tree</code> returns data and never prints</h2>
|
||
<p>A decision the user should hear about comes back as <strong>data</strong>: <code>Dataset::warnings</code> carries the columns <code>preprocess</code> dropped for being entirely missing, <code>fit.rs</code> hands back pooled <code>(prediction, actual)</code> pairs and a score per feature rather than tables. <code>examples/fit_csv.rs</code> owns every <code>println!</code> and every <code>eprintln!</code>, and that is the only reason its output stayed byte-identical across these moves. <strong>Adding an <code>eprintln!</code> to <code>fit.rs</code> or <code>data.rs</code> is the regression</strong>, not a convenience.</p>
|
||
<p>The reason is not tidiness: a library does not own its caller's stderr. A TUI writing to an alternate screen, a wasm build, or a caller that wants the notice in its own status line can none of them suppress, redirect or translate an <code>eprintln!</code>. <code>preprocess</code> used to do exactly that, which was correct while this code <em>was</em> <code>examples/fit_csv.rs</code>.</p>
|
||
<p><code>improvise-gp</code> inherits the rule. <code>FitOutcome::status_line</code> is a <em>rendering</em> of the outcome data, offered so a second front end does not compose its own — the same argument that put <code>flatten_cat_tree</code> in the shared layer — and it is a returned <code>String</code>, never a <code>println!</code>.</p>
|
||
<h2 id="3-the-vendored-rng-and-why-its-constants-are-a-promise"><a class="anchor" href="#3-the-vendored-rng-and-why-its-constants-are-a-promise">#</a>3. The vendored RNG, and why its constants are a promise</h2>
|
||
<p><code>SplitMix64</code> is ~20 lines and no dependency. The <code>Rng</code> trait exists so a caller can bring its own generator, and every caller so far wrote out this same one.</p>
|
||
<p>A fit is not a throwaway: it writes a rule into a user's document, so re-running it on the same data must produce the same rule, and a bug report must be reconstructible from its seed alone. That is worth vendoring rather than depending — the point is reproducibility, not statistical quality, and SplitMix64's is ample for picking nodes and thresholds.</p>
|
||
<p><code>SplitMix64::seed()</code> keeps the seed the run started from, unchanged as the generator advances, so a caller can show it and record it. A generator that has advanced past its seed cannot answer that question.</p>
|
||
<p><code>next_u64</code> is public because <code>splitmix64_matches_the_reference_vectors</code> pins it. <strong>The constants are the algorithm, not an implementation detail</strong>: every seed already printed by a fit and every <code>data::Model</code> seed already saved is a promise that <em>these</em> numbers come back.</p>
|
||
<h2 id="4-spec-ranges-is-a-soundness-precondition"><a class="anchor" href="#4-spec-ranges-is-a-soundness-precondition">#</a>4. <code>Spec::ranges</code> is a soundness precondition</h2>
|
||
<p><code>ranges[i]</code> must be a <strong>superset</strong> of the values feature <code>i</code> can actually take. A too-narrow range makes <code>simplify</code> prune a branch that is reachable, silently changing the tree's outputs. <code>preprocess</code> derives the ranges from the columns themselves, which is what keeps this true; <code>every_training_value_lies_inside_its_feature_range</code> pins that it stays derived.</p>
|
||
<p><code>simplify</code> seeds each feature's feasible interval with its real range rather than ±∞, so an out-of-range threshold prunes. The interval is half-open <code>[lo, hi)</code> but a feature can hit its max, so <code>hi</code> is nudged by <code>EPS = 1e-3</code> and a split <em>at</em> the max (<code>x < max</code>) stays live. <code>EPS</code> is safely below typical threshold granularity.</p>
|
||
<h2 id="5-row-an-invariant-made-unrepresentable"><a class="anchor" href="#5-row-an-invariant-made-unrepresentable">#</a>5. <code>Row<T></code>: an invariant made unrepresentable</h2>
|
||
<p><code>Row<T></code> pairs a feature vector with what it is supposed to predict. The target is generic because the two supervised shapes differ only in it — <code>Row<f32></code> for regression, <code>Row<usize></code> for a class code — and every scoring function then reads <strong>one</strong> slice instead of two.</p>
|
||
<p>That is the point of the type. Parallel <code>x</code>/<code>y</code> slices could be handed in at different lengths, and <code>zip</code> answered by silently scoring the shorter prefix while a denominator taken from the longer one made the mean too small. There is no length to get wrong here, so there is nothing to validate and no <code>Result</code> to thread through five signatures (<a class="tid" href="../tickets.html#improvise-dkqn" title="gp-tree::fit should take one slice of rows, not parallel x/y slices">improvise-dkqn</a>, <code>doctrine(an_invariant_is_made_unrepresentable_not_validated)</code> — the same answer <code>prune_axis_items</code> got).</p>
|
||
<p><strong>The pairing is made at the boundary that builds the data.</strong> <code>preprocess</code> walks one row index and the rows live inside <code>Target</code>; a <code>Dataset</code> carrying <code>rows</code> and <code>target</code> as two collections would move the mismatched length one level up, into every caller.</p>
|
||
<p><code>mean_over(total, n)</code> is the one place the empty row set is answered. Every mean in <code>fit.rs</code> used to reach its own <code>0.0 / 0.0</code> on an empty slice: a NaN that propagates through <code>FitConfig::cost</code>, sorts unpredictably against real costs, and reaches a caller as a number rather than as an error. Scoring no rows finds no error, so the answer is zero, stated once instead of in <code>mse</code>, <code>misclass_rate</code> and <code>standardize</code> separately.</p>
|
||
<p><strong>Changing any of this needs bit-level evidence.</strong> <code>tests/scoring_known_answers.rs</code> pins every scoring path as <code>f32</code> bit patterns plus a <code>next_u64</code> witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. See <a href="../test-evidence/gp.html"><code>context/test-evidence/gp.md</code></a>.</p>
|
||
<h2 id="6-folds-and-cv-folds-answer-different-questions"><a class="anchor" href="#6-folds-and-cv-folds-answer-different-questions">#</a>6. <code>folds</code> and <code>cv_folds</code> answer different questions</h2>
|
||
<p><code>folds(n, k)</code> splits <code>0..n</code> into <code>k</code> pieces after a deterministic shuffle. <code>k</code> arrives from a host — a CLI flag, in this crate's only caller — so zero is a number to answer, not an impossible state: <code>out[pos % 0]</code> is a public library function taking its host down with it. One fold is the answer the shape already gives, and <code>fit</code> spells the same <code>max(1)</code> over <code>restarts</code>.</p>
|
||
<p><code>cv_folds(n, k)</code> yields nothing below two. A partition into one piece is still a partition, but a <strong>cross-validation</strong> over one fold is not a cross-validation: the single fold holds every index, the training complement is empty, and both cross-validators would fit a tree on nothing and then let it predict every row. Yielding no folds makes the pooled pairs empty, which is the true count of such predictions — and no fold is drawn, so the shuffle never touches the caller's generator either (<a class="tid" href="../tickets.html#improvise-ry89" title="cv_regress/cv_classify with k=1 now return plausible numbers instead of NaN">improvise-ry89</a>).</p>
|
||
<p><code>complement(n, test)</code> builds a bool mask rather than asking <code>test.contains(i)</code> per row, which was a scan of the fold per row and so quadratic in the row count (<a class="tid" href="../tickets.html#improvise-t3ux" title="cv_regress and cv_classify are O(n-squared)">improvise-t3ux</a>). It yields <strong>the same indices in the same order</strong>, which is what every downstream draw from the generator depends on.</p>
|
||
<h2 id="7-standardize-denormalize-the-affine-map-must-be-folded-back"><a class="anchor" href="#7-standardize-denormalize-the-affine-map-must-be-folded-back">#</a>7. <code>standardize</code> / <code>denormalize</code>: the affine map must be folded back</h2>
|
||
<p>The crate's leaf scores live in a ±5 range, so a target in the thousands cannot be reached by leaf mutation and <code>standardize</code> is not optional. Its consequence is that the champion predicts <strong>standard deviations</strong>.</p>
|
||
<p><code>fit_csv</code> answers that by <em>printing</em> <code>≈ score × std + mean</code> beside the tree. A document has nowhere to put that, so <code>improvise-gp::fit_measure</code> folds the affine map back into the leaves with <code>denormalize</code> before a single character of formula text is spelled. Omitting it emits a perfectly well-formed formula whose every cell is out by an affine map — <strong>wrong in the quiet way</strong>, since small numbers in the right shape look like an answer.</p>
|
||
<p><code>denormalize</code> touches leaves only, and that is the whole reason it is sound: <code>standardize</code> rescales the <em>target</em>, so a threshold is already in its feature's own units and re-scaling it would be the bug.</p>
|
||
<h2 id="8-regressionquality-carries-its-own-baseline"><a class="anchor" href="#8-regressionquality-carries-its-own-baseline">#</a>8. <code>RegressionQuality</code> carries its own baseline</h2>
|
||
<p>A tree with a plausible-looking RMSE that is no better than always predicting the mean is not a weak rule, it is <em>no rule</em> — and it is worse than no rule at all, because it is one a user will believe. So the comparison is part of the answer rather than something a caller may forget to compute. <code>r2</code> is <strong>not clamped</strong>: negative means the fit does worse than the mean, which is a real outcome, and clamping would make "learned nothing" and "actively misleading" read alike.</p>
|
||
<p><code>importance_ranking</code> reports each feature's share of the total importance. The all-zero case returns early rather than flooring the divisor: <code>fit_csv</code> used <code>.max(1e-9)</code>, which is right for the only input it ever had and wrong as a contract — a lone importance of <code>1e-10</code> then divides by the floor rather than by itself and reports <strong>10%</strong>, where the one feature the model uses carries all of it. Importances this small are ordinary, being error differences over a <em>standardized</em> target, so the shares have to sum to 100% at every scale.</p>
|
||
<h2 id="9-a-fit-takes-two-things-to-reproduce-and-here-four"><a class="anchor" href="#9-a-fit-takes-two-things-to-reproduce-and-here-four">#</a>9. A fit takes two things to reproduce — and here, four</h2>
|
||
<p><code>data::Model</code> records <code>seed: Option<u64></code> <strong>and</strong> <code>config: Option<FitConfig></code>, because the same seed under a different population/generations/elite/penalty lands on a different tree (<a class="tid" href="../tickets.html#improvise-ksst" title="A gp-tree fit records its seed but not its GA config">improvise-ksst</a>). Both are <code>Option</code> for one reason: every <code>FitConfig</code> value and the seed <code>0</code> are legal, so nothing can double as "unknown", and anything reporting a sentinel would state a seed that never ran.</p>
|
||
<p>Note <code>#[serde(default)]</code> on an <code>Option</code> field is <strong>documentation, not the mechanism</strong> — serde defaults it anyway, so deleting the attribute fails no test. A backward-compatibility test that only deletes the attribute is not discriminating; aim it at an <em>absent field</em> loading as the default instead.</p>
|
||
<p><code>improvise-gp::FitOutcome</code> carries the <strong>whole</strong> <code>FitRequest</code>, not the pair. <code>FitRequest</code> is <code>{ seed, config, cv_folds, importance_repeats, max_records }</code>, and every stochastic step draws from <strong>one</strong> generator in a fixed order — cross-validate, then fit, then explain. So <code>cv_folds</code> decides how far that generator has advanced before <code>fit</code> draws from it, and therefore changes the <em>champion</em> and not merely the evidence for it. Recording two of four fields documented the reproducibility contract without meeting it.</p>
|
||
<h2 id="10-the-emitter-quoting-and-the-threshold-s-exact-decimal"><a class="anchor" href="#10-the-emitter-quoting-and-the-threshold-s-exact-decimal">#</a>10. The emitter: quoting, and the threshold's exact decimal</h2>
|
||
<p>A regression tree <em>is</em> a nested conditional, and improvise formulas already have <code>IF(cond, then, else)</code>, so a fit can be written straight into a user's model as an ordinary formula with nothing GP-specific left in the file:</p>
|
||
<pre><code>Score = SUM(IF(humidity < 77.5, 1.25, IF(outlook IS rainy < 0.5, -0.5, 2)))</code></pre>
|
||
<p>(An earlier form of that sample sat in <code>improvise-gp/src/lib.rs</code>'s module doc inside a ``<code>text fence. rustdoc does not execute a </code>text` fence, so it was prose wearing a fence rather than a doctest, and the two-line cap applies to it like anything else.)</p>
|
||
<p>Two things in it are <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>'s, and both are load-bearing. The outer <code>SUM</code> is the <strong>route-then-aggregate rule made syntax</strong>: a tree is fitted on records, and <code>SUM(e)</code> is improvise's existing word for "fold <code>e</code> over the member records" (<a href="../code/core.html"><code>context/code/core.md</code></a> §13). Without it the formula routes on the totals one row up and means something else there; with it, a record row is a one-member fold and answers the rule itself. <code>outlook IS rainy</code> is the one-hot <code>outlook=rainy</code> said as a <strong>coordinate predicate</strong> rather than as a reference to an item no document has — see §11 for which features get which spelling.</p>
|
||
<p>Two properties the emitter owes its caller:</p>
|
||
<p><strong>It parses.</strong> Feature names come from the caller's data, so they carry spaces, operators, pipes and keyword collisions. Every name goes through <code>quote_name</code> — <code>improvise_formula::quote</code>, the one home for pipe quoting (gotcha 34); <code>improvise-gp</code> re-exports it because it was part of this crate's public surface. A formula <em>target</em> is unquoted on the way back in: it names a <code>_Measure</code> item, so a target keeping its delimiters writes the fit into an item literally named <code>|Total Revenue|</code>.</p>
|
||
<p><strong>It routes identically.</strong> <code>number_text</code> emits the <strong>shortest decimal that reads back as the same <code>f64</code></strong> the <code>f32</code> widens to — not the f32's full decimal expansion, which runs to 149 fractional digits, and not a shortest-round-trip-to-<code>f32</code> rendering. Widening an <code>f32</code> to <code>f64</code> is exact, so text that reads back as that <code>f64</code> makes <code>feature < thresh</code> give the same answer in <code>f64</code> formula evaluation as in <code>f32</code> <code>gp_tree::eval</code>. A short rendering (<code>0.1</code> for <code>0.1f32</code>) sits up to half an ULP away, and a row whose feature value equals the threshold exactly then falls on the <em>other</em> side of the split.</p>
|
||
<p><code>number_text</code> <strong>panics</strong> on a non-finite value. The grammar spells neither, and both fail quietly rather than loudly: <code>"inf"</code> parses as an <code>f64</code>, so the round-trip loop would return text no formula parser accepts, and NaN matches no width at all. Trees built by this crate's own operators over a finite <code>Spec</code> are finite, so this is a precondition rather than a runtime condition — but see §12, because a <code>.improv</code> cell holding <code>inf</code> makes it reachable from a document.</p>
|
||
<p><code>expr_text</code> and <code>mark_features</code> share one precondition and spell it the same way (<a class="tid" href="../tickets.html#improvise-hkt6" title="Three small totality gaps found in review">improvise-hkt6</a>): a bare <code>used[*feat]</code> fails with <code>index out of bounds</code>, which names neither the tree nor the spec. Two sites, one failure, one sentence. <code>used_features</code> is the single walk both the document check and the plan check read, because walking the tree twice two ways is how those answers drift apart.</p>
|
||
<h2 id="11-a-fit-is-storable-only-when-the-document-can-name-its-features"><a class="anchor" href="#11-a-fit-is-storable-only-when-the-document-can-name-its-features">#</a>11. A fit is storable only when the document can <em>name</em> its features</h2>
|
||
<p><code>preprocess</code> invents <code>Region=North</code> for a one-hot, <code>Cost.missing</code> for a missing indicator, <code>when.year</code> for a date part. Those are <strong>encoding artefacts, not items of the document</strong>, and <code>Expr::Ref</code> resolves through <code>Model::find_item_category</code> — so a tree splitting on one emitted a perfectly well-formed formula whose every cell then evaluated to <code>Error("ref:Region=North")</code>, with nothing between the fit and the file to say so (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>). <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a> closed the categorical third of that by giving the emitter a vocabulary for a coordinate, and by giving the <em>evaluator</em> a rule that makes such a formula mean the same thing at every level of aggregation (<a href="../code/core.html"><code>context/code/core.md</code></a> §13).</p>
|
||
<h3 id="features-one-entry-per-feature-index-read-off-the-plan"><a class="anchor" href="#features-one-entry-per-feature-index-read-off-the-plan">#</a><code>Features</code>: one entry per feature index, read off the plan</h3>
|
||
<p><code>Features::of(&Dataset, categories)</code> walks <code>gp_tree::data::feature_roles</code> — the <strong>one</strong> statement of <code>preprocess</code>'s expansion order, which <code>encode_row</code> drives off too — and pairs each role with the name <code>preprocess</code> gave it. A tree's <code>feat</code> indexes <code>Features</code> directly, so nothing anywhere reconstructs index → role from the feature <em>names</em>. <code>Spec::names</code> is <code>&[&str]</code> and could not have carried this; the earlier check matched names against the raw plan instead and said in this section that it was avoiding a second implementation of the expansion. Now there is no second implementation to avoid.</p>
|
||
<p>Each feature gets one of three <code>Spelling</code>s, decided <strong>by role and by column</strong>, never by the shape of the name:</p>
|
||
<table><thead><tr><th>role</th><th>column</th><th>spelling</th></tr></thead><tbody><tr><td><code>Numeric</code></td><td>a measure</td><td><code>Expr::Ref(measure)</code></td></tr><tr><td><code>OneHot { value }</code></td><td>a category, <code>value != "(missing)"</code></td><td><code>Cat IS Item</code></td></tr><tr><td><code>Numeric</code></td><td>a category</td><td><code>Artefact(NumericCategory)</code></td></tr><tr><td><code>DatePart</code></td><td></td><td><code>Artefact(DatePart)</code></td></tr><tr><td><code>Missing</code></td><td></td><td><code>Artefact(MissingFlag)</code></td></tr><tr><td><code>OneHot</code></td><td>a measure</td><td><code>Artefact(ValueOfAMeasure)</code></td></tr><tr><td><code>OneHot { "(missing)" }</code></td><td>a category</td><td><code>Artefact(AbsentCoordinate)</code></td></tr></tbody></table>
|
||
<p>The last five arms were <strong>one</strong> <code>_ => Artefact</code> until <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>, and the wildcard was not merely untidy. The refusal it fed read "a <code>col.missing</code> flag or a date part is an encoding artefact", so three of the five families were refused in words naming something they are not — and a future <code>FeatureRole</code> variant would have joined them silently, declined under whichever family the message happened to mention. The match is now exhaustive, so a sixth role is a compile error rather than a sixth thing that message misdescribes.</p>
|
||
<p>The two exclusions are not fussiness. A one-hot over a <strong>measure</strong> column names no coordinate at all — the record's value for that measure is text, and <code>Cat IS Item</code> asks about coordinates. <code>(missing)</code> is the <em>absence</em> of a coordinate, which <code>Expr::Is</code> reads as <code>0</code> by §13's <code>unwrap_or(false)</code>, so the one spelling that would be wrong is exactly the one omitted. A <code>Numeric</code> role over a <strong>category</strong> column is the trap the previous version of this section called "the hole a name cannot close": a category whose items all parse as numbers produces a feature named after the category, and <code>Expr::Ref</code> would resolve it to a <code>_Measure</code> item of the same name — a different column. Both halves now fall out of <code>is_category</code>, which is a lookup in the header's own first <code>category_columns</code> entries.</p>
|
||
<p>Duplicate header names are still refused outright, by <code>no_ambiguous_column_names</code> and before anything else: <code>is_category</code> decides by name, so a name that means two columns is a spelling with two meanings.</p>
|
||
<p><code>Dataset::columns</code> is <strong>not</strong> index-aligned with the header — <code>preprocess</code> <code>continue</code>s past an entirely-missing column without pushing a <code>ColumnSpec</code>, so one empty category shifts every later spec left. <code>FeatureRole::column()</code> indexes <code>columns</code>, not the header, so that shift is carried rather than tripped over.</p>
|
||
<h3 id="what-fitted-formula-still-asks-the-document"><a class="anchor" href="#what-fitted-formula-still-asks-the-document">#</a>What <code>fitted_formula</code> still asks the document</h3>
|
||
<p>Two refusals, in order. An <code>Artefact</code> is refused with no reference to the document at all — nothing could make <code>Cost.missing</code> spellable. Everything else is asked <strong>of the document</strong>: a <code>Measure</code> must be <code>find_item_category</code>-resolvable and a <code>Coordinate</code> must name a category the model has. A measure literally named <code>Region=North</code> is a legal document and is accepted, because the check never looks at the name's shape — and that is also why the date parts and the <code>.missing</code> flags stay covered without a second copy of <code>preprocess</code>'s naming rules.</p>
|
||
<p><code>formula_text</code> still spells whatever it is handed, being the tree's own text; an <code>Artefact</code> reaching it is a precondition failure and panics, the same split <code>number_text</code> makes for a non-finite value.</p>
|
||
<h3 id="the-two-families-improvise-wd6n-declines-separately"><a class="anchor" href="#the-two-families-improvise-wd6n-declines-separately">#</a>The two families <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a> declines, separately</h3>
|
||
<p>Both stay <code>Artefact</code> and both stay refused. The decline is written here rather than left open, because each needs a <strong>formula-language</strong> change and neither needs an emitter change — and they are answered separately because the vocabulary each wants is different. What changed with the decline is that the refusal now says which, in that family's own words.</p>
|
||
<p><strong><code>col.missing</code> is declined because the language has no word for absence.</strong> The flag means "this record has no value for measure X". <code>Expr::Ref</code> over an absent cell is an <code>Error</code>, not a zero, so there is no existing spelling to reuse and no way to phrase the question as an expression. Closing it means deciding what an absence <em>is</em> in a language whose aggregates already treat it as "no rows" — and that decision reaches <code>apply_agg</code>, <code>eval_agg</code> and the text-unanimity rule, not just this emitter. It is not worth that for a split a fit happens to like.</p>
|
||
<p><strong>The date sub-features are declined because improvise has no dates.</strong> <code>when.epoch_day</code>, <code>.year</code>, <code>.month</code>, <code>.day</code>, <code>.dow</code>, and with a time component <code>.hour</code>, <code>.minute</code> — <code>gp_tree::data::build_date</code> emits them. A <code>.improv</code> cell holding a date is <code>CellValue::Text</code>; nothing in <code>eval_expr</code> reads one, and there is no date type, no date literal and no date arithmetic anywhere in the crate. Closing this is a type addition to the value language, and the sub-features are the <em>smallest</em> part of it: a user who could split on <code>when.year</code> would immediately want to filter and aggregate by it.</p>
|
||
<p>Both are visible to the user only as a refusal naming the feature and the reason, which is the right interim: a fit that cannot be stored says so, rather than storing a formula whose every cell is a reference error (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>). <code>improvise-4zl7</code>'s route-then-aggregate rule (<a href="../code/core.html"><code>context/code/core.md</code></a> §13) is untouched — nothing new is emitted, so there is nothing new for it to hold over.</p>
|
||
<p>Two of the other three are declined permanently: a one-hot over a measure and the <code>(missing)</code> pseudo-item name no coordinate <em>by construction</em>, so no vocabulary closes them. They are refusals rather than gaps.</p>
|
||
<p><strong><code>NumericCategory</code> was not, and <a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a> has closed it.</strong> That it was <em>called</em> one is the more useful half of <a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>: prose declaring something a permanent refusal is what stops anyone filing the ticket, and nothing but a reader could have caught it — the suite was green either way.</p>
|
||
<p>The accident was one argument. <code>fit_measure</code> handed <code>preprocess</code> an <code>Overrides::default()</code>, so it never said which header columns are document categories, <strong>although the very next line told <code>Features::of</code></strong>. <code>preprocess</code> then inferred <code>Numeric</code> → <code>Date</code> → <code>Categorical</code> from the cell values, so a <code>Year</code> category with items <code>2024</code>/<code>2025</code> became a <em>numeric column</em> and a category of ISO dates became a <em>date column</em> — and both were then refused as encoding artefacts with <code>Year IS 2024</code> spellable all along. <code>fit_measure</code> now builds its <code>Overrides</code> from the same <code>header[..category_columns]</code> slice it already hands <code>Features::of</code>, forcing each to <code>Forced::Categorical</code>, so the one place the document's own answer was known is now the only place either caller reads it from.</p>
|
||
<p>Three things follow, and the last is a cost rather than a win.</p>
|
||
<p><strong>The date decline above now means what it says.</strong> A <code>DatePart</code> could arrive over a <em>category</em> column by exactly this accident, so §11's date family was narrower than it read. It now covers a genuine date <strong>measure</strong> and nothing else, which is the scope the decline was always argued for.</p>
|
||
<p><strong>The <code>Artefact</code> arms stay, and stay correct.</strong> <code>crate::dataset</code> is <code>pub</code> and takes the caller’s <code>Overrides</code>, and <code>Features::of</code> reads roles off whatever <code>Dataset</code> it is handed — so a caller that builds one without forcing still produces a <code>Numeric</code> role over a category column, and <code>fitted_formula</code> still has to refuse it. The arm is unreachable from <code>fit_measure</code>, which is the only entry point a <em>document</em> goes through; it is not unreachable code, and deleting it would be deriving the classification from one caller’s habits rather than from the roles.</p>
|
||
<p><strong>Forcing widens the encoding, and that costs a document.</strong> A <code>Year</code> category with two items was one numeric feature and is now two one-hots; with a hundred items it is a hundred, and <code>MAX_FEATURES</code> refuses the fit outright.</p>
|
||
<p>The first draft of this paragraph claimed "no document that fitted before stops fitting", on the reasoning that a tree splitting on such a column was refused as an artefact anyway. <strong>That is false, and the review that caught it was of this prose rather than of the diff.</strong> The refusal is conditioned on <code>used_features</code>, so a fit that simply <em>never split</em> on the numeric category succeeded — and it is precisely the wide categories that a fit has least reason to split on. Measured, on a hundred records with a hundred-item numeric <code>Year</code> beside a <code>Cost</code> that predicts <code>Score</code> on its own: before, the fit lands on <code>IF(Cost < 48.0…)</code> and <code>Year</code> is never touched; after, the preflight refuses at 101 features. <code>a_wide_numeric_category_now_refuses_a_fit_that_used_to_ignore_it</code> pins both halves, including that the repair the message names really works.</p>
|
||
<p>It is still the right trade, for a reason that is <em>not</em> "both refuse". The old success was accidental: <code>Year</code> was encoded as a single continuous number over 1900–1999, an encoding that means nothing and that the fit could only use by producing a formula it would then be refused for. The document now gets a refusal that names the repair — mark the category a Label — and marking it restores the pre-zv8t feature set exactly. What is lost is the case where the user wanted neither: a wide numeric category left alone while the fit uses the measures. That is filed as <strong><a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a></strong>; the honest fix is a budget that can decline <em>a column</em> rather than the whole table, which is a change to what <code>encoded_width</code> returns and not to this decision.</p>
|
||
<p>The preflight <code>encoded_width</code> had to learn the same slice, or it counted such a column as one feature and handed the refusal to the exact check <em>after</em> materialising the matrix — which is the side of the cost the preflight exists to stay on (§14).</p>
|
||
<p><strong>And the refusal's own words must not name a column kind the arm does not know.</strong> <code>FeatureRole::Missing</code> is <em>not</em> conditioned on <code>is_category</code> — unlike its <code>Numeric</code> and <code>OneHot</code> neighbours — and <code>dataset.rs</code> writes <code>""</code> for a record with no coordinate for a category, so a <code>.missing</code> flag over a <strong>category</strong> is reachable. Its clause therefore says "that column" and not "that measure": naming the wrong kind there would be this section's own charge — declined in words naming something they are not — surviving one level down.</p>
|
||
<h2 id="12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na"><a class="anchor" href="#12-a-fit-is-persisted-as-its-formula-and-nothing-else-improvise-7na">#</a>12. A fit is persisted as its formula, and nothing else (<a class="tid" href="../tickets.html#improvise-7na" title="Decide whether fitted gp-tree models are persisted alongside the formula">improvise-7na</a>)</h2>
|
||
<p><code>.improv</code> stores the emitted formula and <strong>no</strong> serialized <code>gp_tree::data::Model</code>. The grammar is unchanged by this decision, deliberately and not by omission.</p>
|
||
<p>The ticket offered three options: (a) formula only, refit to score new data; (b) the <code>data::Model</code> as a JSON blob; (c) seed + config so the fit reproduces. The answer is (a), with (c) already available in <code>gp-tree</code> for whoever wants provenance later.</p>
|
||
<p><strong>Against (b)</strong>, two arguments. The general one is the second half of gotcha 41: a blob is a <em>second copy</em> of information the document already carries — the tree is the formula — and nothing keeps the two in agreement. A user who edits the formula by hand, or a cell it was fitted on, leaves a blob that disagrees with the file it sits in, and a round-trip test over the blob cannot see that, because it never compares the two. (The <em>first</em> half of gotcha 41 does not transfer: an opaque blob inside one new grammar production is not a second implementation of the grammar.) The specific one is layering: <code>Tree</code> and <code>FitConfig</code> are <code>gp-tree</code> types, so <code>improvise-io</code> would gain a dependency on <code>gp-tree</code> and <code>.improv</code> would be obliged to track a third crate's serde layout forever — inverting exactly the dependency <code>improvise-gp</code> exists to keep out of the core.</p>
|
||
<p><strong>What (b) was for, and why it is not needed.</strong> The encoding plan exists so <code>fit_csv --predict</code> can score a row that arrives as untyped CSV text. improvise has no such row: a new row is a <em>record in the same document</em>, and <code>recompute_formulas</code> scores it through the stored formula from its own measures.</p>
|
||
<p><strong>Why (c) is not written to the file either.</strong> <code>data::Model</code> already records seed and config, so reproducing a fit is solved wherever that value is in hand; and unlike the CSV case the training data <em>is</em> the document. What is missing is a <em>reader</em> — writing provenance now would add a grammar production and a round-trip obligation for metadata nothing consumes. Provenance is the fit command's decision, alongside how it words the fit for the user.</p>
|
||
<p><strong>A premise the ticket got wrong, and it is the bigger finding.</strong> "The emitted formula is enough to recompute existing cells" holds only for a fit over numeric measures — §11 is that defect, and it is a defect in what a fit may emit, not an argument for persisting the model.</p>
|
||
<h2 id="13-dataset-rs-what-a-view-shows-as-a-feature-matrix"><a class="anchor" href="#13-dataset-rs-what-a-view-shows-as-a-feature-matrix">#</a>13. <code>dataset.rs</code>: what a view shows, as a feature matrix</h2>
|
||
<p><strong>The shape is not invented here.</strong> Records mode is already a feature matrix: a row is one coordinate tuple — every cell sharing all coordinates but <code>_Measure</code> — and each measure is its own column (gotcha 1). That is exactly "a sample, and its features". So the module enumerates the same rows records mode does, lays them out as the string table <code>preprocess</code> takes, and lets <code>preprocess</code> own type inference, one-hot expansion, imputation and the ranges.</p>
|
||
<p>What it does <em>not</em> reuse is <code>GridLayout</code>: reaching the records path means putting <code>_Index</code> on Row and <code>_Dim</code> on Column, which is a different view from the one the user is looking at. <code>RecordRow::group</code> is public, so the traversal here is the same two steps the layout takes, against the user's own view.</p>
|
||
<p>Four selection rules, each of which could reasonably have gone the other way:</p>
|
||
<ul>
|
||
<li><strong>Rows are scoped by <code>View::filtering_cats()</code></strong> — Page ∪ Filter — not by <code>categories_on(Axis::Page)</code>. Both constrain which records are visible; only Page is navigable (gotcha 27), and scoping by the navigable half would silently fit against rows a drill had already excluded.</li>
|
||
<li><strong>Hidden items scope the rows and the feature columns.</strong> The mental model is "fit what I can see", and hiding an item is an explicit statement that it is not of interest. An explicitly named target is exempt: the caller asked for it by name.</li>
|
||
<li><strong>Every regular category is a feature, whatever axis it sits on</strong> — including <code>Axis::None</code>. A category's coordinate is part of what makes a record distinct, so dropping a None category would leave two different rows with identical feature vectors and different targets: noise the fit can neither learn nor report. <code>Label</code> categories are excluded, as they are from records mode, by being not <code>is_regular</code> — they are high-cardinality per-row text and one-hot into a perfect memoriser.</li>
|
||
<li><strong>No formula-derived value ever enters the matrix.</strong> Formula targets are not features (a formula is a function of the other measures, so it leaks the target whenever the target is one of its inputs), and a formula target is refused <em>as</em> the target — a formula already is the rule, so fitting a tree to reproduce it from its own inputs is trivially perfect and says nothing. One rule covers both halves, with no dependency analysis to get wrong.</li>
|
||
</ul>
|
||
<p><strong><code>_Measure</code> is a filtering category and must not filter a fit.</strong> That exception in <code>filter_coords</code> is not a special case, it is the difference between a grid and a feature matrix: <code>_Measure</code> is the coordinate the <em>columns</em> are spread across, so pinning it to one item is not "which records am I looking at" but "throw away every column except this one". <code>Workbook::new</code>'s default view puts it on <code>Axis::Page</code>, so that was the behaviour on a freshly opened document (<a class="tid" href="../tickets.html#improvise-vx3" title="fit-measure command: fit a tree and add it as a formula">improvise-vx3</a>).</p>
|
||
<p>Two smaller decisions. Records are sorted by their coordinate tuple. That began as a workaround for <code>matching_cells</code> walking a <code>HashSet</code>; since <a class="tid" href="../tickets.html#improvise-shyt" title="[bug] Records-mode row order is not stable across processes">improvise-shyt</a> it yields insertion order, so the sort is no longer what buys stability. It is kept because a fit's row order should be a property of the <em>data</em> rather than of the typing history that entered it. And a <code>Number</code> cell is rendered with <code>f64::to_string</code>, never <code>format_f64</code>: display rounding is view-only and must not reach a fit (gotcha 8). An <code>Error</code> cell has no value, so it is spelled as missing and picks up the same imputation and <code>.missing</code> indicator an absent cell would.</p>
|
||
<h2 id="14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess"><a class="anchor" href="#14-improvise-gp-fit-budgets-and-the-enum-that-refuses-to-guess">#</a>14. <code>improvise-gp::fit</code>: budgets, and the enum that refuses to guess</h2>
|
||
<h3 id="two-budgets-and-why-the-preflight-is-advisory"><a class="anchor" href="#two-budgets-and-why-the-preflight-is-advisory">#</a>Two budgets, and why the preflight is advisory</h3>
|
||
<p><code>FitConfig</code> bounds the <em>search</em>; nothing bounded the <em>width</em>, which <code>preprocess</code> can multiply without limit (one column per distinct value of a categorical column). Permutation importance alone is <code>repeats × rows × features</code> tree evaluations, on the thread that draws the next frame.</p>
|
||
<p><code>MAX_FEATURES = 64</code> is comfortably above any hand-built pivot table — this repo's own 18,000-cell benchmark encodes far fewer — and comfortably below the one-column-per-row shape that is really a mis-typed category. The message names the fix (<code>Label</code>), because that is what the user actually wants.</p>
|
||
<p><code>encoded_width</code> is the <strong>advisory</strong> preflight, run on the raw table <em>before</em> <code>preprocess</code> materialises anything. Checking width afterwards guards the wrong side of the cost: a category with a distinct value per record allocates a rows × rows one-hot <em>during</em> preprocessing, so the refusal arrives after the memory and the O(rows²) work have been spent. It approximates <code>preprocess</code>'s type inference rather than sharing it. Copying the predicate is the lesser evil; the alternative is exporting the inference and calling this a second implementation of it.</p>
|
||
<p><strong>The debt that copy carries is that the preflight owes EVERY branch <code>preprocess</code> can take, and it paid two of three.</strong> <code>preprocess</code> infers <code>Numeric → Date → Categorical</code>, in that order, and the preflight had no date arm at all: a column whose cells parse as neither numbers nor a forced category fell through to the categorical count. So a <strong>measure column holding dates</strong> — one distinct timestamp in every row — promised <strong>1</strong> and materialised <strong>5</strong>, because <code>build_date</code> emits <code>epoch_day/year/month/day/dow</code>, and <strong>7</strong> once any cell carries a time (<a class="tid" href="../tickets.html#improvise-x1ns" title="encoded_width undercounts a forced category that has a missing coordinate">improvise-x1ns</a>, second instance, found by ChatGPT Codex on #97). That is an under-count of four or six on a single column, against a budget of 64.</p>
|
||
<p>The two instances are the same defect one inference-branch apart, which is why the shape is now structural rather than remembered. <code>inferred_kind</code> reproduces <code>preprocess</code>'s match arm for arm and returns a <code>Forced</code>, and <code>encoded_width</code> matches that <strong>with no wildcard</strong> — so a fourth <code>Forced</code> variant is a compile error here instead of silently taking the categorical arm, which is exactly how <code>Date</code> was missed. The <code>.missing</code> flag is added once outside the match, because every branch owes it on the same terms.</p>
|
||
<p>The count each arm owes:</p>
|
||
<table><thead><tr><th><code>preprocess</code> builds</th><th>features</th></tr></thead><tbody><tr><td><code>build_numeric</code></td><td>1</td></tr><tr><td><code>build_date</code></td><td>5, or 7 if any present cell has a time</td></tr><tr><td><code>build_categorical</code></td><td>distinct present values, <strong>+1</strong> for the reserved <code>(missing)</code></td></tr></tbody></table>
|
||
<p>plus <code>col.missing</code> wherever any cell is missing, and clamped to at least 1.</p>
|
||
<p><strong>The two directions of a drift do not cost the same, and this paragraph said they did</strong> (<a class="tid" href="../tickets.html#improvise-x1ns" title="encoded_width undercounts a forced category that has a missing coordinate">improvise-x1ns</a>). It read: <em>the cost of the copy drifting is bounded in both directions by the exact check</em> — over-counting costs a needless refusal, under-counting "only lets the exact check do its job". That is wrong, and it is wrong against the argument <strong>two paragraphs above it</strong>: the exact check runs on <code>Dataset::feature_names.len()</code>, which exists only <em>after</em> <code>preprocess</code> has materialised the matrix. So it bounds the <strong>error message</strong>, never the <strong>cost</strong> — which is precisely the thing the preflight was introduced to bound. Over-counting is cheap; under-counting spends the memory and the O(rows²) work the preflight exists to prevent, and then reports it politely.</p>
|
||
<p>The live instance was the <code>(missing)</code> column: the categorical arm counted <code>present</code> values only, so a forced category with N distinct present values and one record lacking that coordinate promised N and materialised N+1. <code>has_missing</code> was already computed and used in the numeric arm alone — <strong>two arms of one match disagreeing about what a column costs</strong>, which is <a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a>'s forcing-by-name / budgeting-by-index defect one level in. That one was repaired by making the ordering a data dependency and so a compile error; this one has no such shape available, so the guard is a test that asks <code>preprocess</code> what it actually built rather than asserting a number (<code>the_preflight_counts_the_missing_category_a_forced_column_materialises</code>).</p>
|
||
<p>The previous wave's review saw a version of this and reasoned it away as <em>"read, not injected"</em> and <em>"bounded by the exact check"</em>. Both halves of that are the failure mode this file exists to catch: the first declined to run the experiment, and the second repeated the sentence above instead of checking it. <strong>An argument that a cost is bounded is checkable, and this one was checkable the day it was written</strong>: <code>encoded_width</code> landed 2026-09-19 and the sentence 2026-09-20, so the code it described was one commit old and one file away.</p>
|
||
<p><code>encoded_width</code> remains <strong>advisory</strong> in the direction that is safe: it may over-count, and a column whose cells are <em>entirely</em> missing is counted as one where <code>preprocess</code> skips it altogether. It must never under-count.</p>
|
||
<p><strong>That sentence was written as a proposed <code>kb.pl</code> invariant while the tree held a counter-example</strong>, in the same change that fixed the categorical one — the date branch was missing the whole time and nothing in the hand-back noticed. An invariant proposed from the defect you just fixed generalises the <em>fix</em>, not the code; the thing that would have caught it is the question "what else does <code>preprocess</code> do that I have not mirrored", which is a question about the other function and not about the bug.</p>
|
||
<p><code>MAX_RECORDS = 20_000</code> bounds the <em>work</em>, which the width bound does not: a one-feature table with a million records still runs cross-validation, the final search and permutation importance over all of them. Measured at 105 ms for 2,000 records and roughly linear, so 20,000 is about a second — where a pause stops being a pause. Refusing is honest, and the limit is the caller's <code>FitRequest::max_records</code> rather than a constant, so a host with somewhere else to run a fit can raise it.</p>
|
||
<p><code>FitRequest::default()</code> is the interactive budget: <code>FitConfig::default()</code> is the CLI's 80 members over 600 generations, which is right for a batch tool and wrong for a synchronous command. <strong>Measured</strong>, release build, on the shape this module's tests use: 6 ms at 12 records, 12 ms at 200, 105 ms at 2,000 — the whole sequence, five-fold CV included. There is deliberately <strong>no timing test</strong>: a wall-clock assertion is flaky on shared CI and would fail for a reason that is not this code's. Re-measure after changing the numbers.</p>
|
||
<h3 id="forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one"><a class="anchor" href="#forcing-a-numeric-category-is-a-representability-choice-not-a-budget-one">#</a>Forcing a numeric category is a <em>representability</em> choice, not a budget one</h3>
|
||
<p><a class="tid" href="../tickets.html#improvise-zv8t" title="fit_measure never tells preprocess which columns are categories, so a numeric- or date-valued category is unspellable">improvise-zv8t</a> made <code>fit_measure</code> force every document category to <code>Forced::Categorical</code>, and <a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a> records the cost: a hundred-item numeric category one-hots to a hundred columns, past <code>MAX_FEATURES</code>, and is refused where inference used to read it as one numeric column and the fit succeeded. <strong><a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a> is declined, and the refusal is correct.</strong></p>
|
||
<p>The ticket rules out making the forcing conditional on cardinality, and rightly: that is inference by another name, and it makes the <em>encoding</em> depend on a budget rather than on what the document says. But the stronger reason to decline is that the alternatives are not merely distasteful — they are <strong>unstorable</strong>, and the code already says so.</p>
|
||
<p>A coordinate becomes an <code>Expr</code> <strong>only by passing through <code>IS</code></strong>: <code>is_expr</code> is <code>identifier ~ sp+ ~ ^"IS" ~ sp+ ~ filter_value</code>, a single-value equality, and there is no <code>IN</code> and no ordering over a category's items. (<code>where_clause</code> and <code>inline_where</code> are coordinate predicates too, and also single-value equalities — but they <em>gate</em> a formula or an aggregate rather than being an <code>Expr</code>, which is gotcha 37(d)'s distinction and is why <code>IS</code> is the one that matters here.) So whatever a tree says about a category, the emitted formula names its items <strong>one at a time</strong>.</p>
|
||
<p><strong>A first draft of this paragraph said that made one-hot the only spellable encoding. That is false</strong>, and an adversarial review disproved it by running <code>parse_formula</code> rather than reading the grammar: <code>is_expr</code> is a <code>primary</code>, so it is an <code>Expr</code> yielding 1/0, and <code>cmp_op</code>'s operands are full <code>expr</code>s — so an ordinal split <em>is</em> spellable, as a weighted indicator sum (<code>IF(2020*(Year IS |2020|) + 2021*(Year IS |2021|) < 2020.5, …)</code>). The claim is recorded here rather than quietly deleted because it is the shape this file keeps catching: a grammar argument made by reading the rule that looked relevant instead of running the parser.</p>
|
||
<p>So <strong>ordinal is not ruled out by spellability, and is ruled out anyway</strong>, for two reasons that survive. It invents a <strong>metric the document does not assert</strong> — a category's items carry insertion order, not magnitude, so <code>Year < 3.5</code> means "the first four items as they happen to be ordered" and stops meaning that when someone reorders them. And the width it saves does not stay saved: the <em>encoding</em> would be one feature, but the <em>formula</em> still spells one <code>IS</code> per item, so it trades a <code>MAX_FEATURES</code> problem for a formula-size problem while adding a false order. What the current emitter does with it is narrower and is not the argument: <code>spell()</code> maps <code>FeatureRole::Numeric</code> over a <em>category</em> column to <code>Artefact::NumericCategory</code>, because a bare reference to the category's name would resolve to a <code>_Measure</code> item of the same name (<a class="tid" href="../tickets.html#improvise-8xhy" title="[bug] A fit over a categorical feature emits an unevaluable formula">improvise-8xhy</a>) — a fact about the bare-ref spelling, not about ordinals in general. <strong>Hashing</strong> is ruled out outright: a bucket has no name in the document, so it cannot even be refused informatively.</p>
|
||
<p>And this is what makes the decline clean rather than resigned. The "fit that previously succeeded" was one of two things, never a third. Either the tree <strong>never split</strong> on the numeric category — in which case the column contributed nothing to the rule, and marking it a <code>Label</code> gives a fit over the same remaining features, inside the budget, which is exactly what the refusal already advises. Or the tree <strong>did</strong> split on it, and <code>fitted_formula</code> refused the result as <code>Artefact::NumericCategory</code> at storage time. So <strong>no fit is lost that was both storable and used the column</strong>.</p>
|
||
<p>Two things that claim less than a first draft of this paragraph did, both worth keeping. Marking the category a <code>Label</code> does <strong>not</strong> yield <em>the same fit</em>: the feature count changes, so the RNG draw sequence in <code>random_tree</code>/<code>sample_thresh</code> changes, so the champion for a given seed changes — gotcha 37(b)'s own rule, applied to this paragraph. "The same fit" is a checkable cost claim and it is false; what is true is a fit over the same remaining features. And the dichotomy above is <strong>argued</strong>, from <code>spell()</code>'s exhaustive match over <code>FeatureRole</code>, not asserted: what <code>a_split_on_a_numeric_category_read_as_a_number_cannot_be_stored</code> asserts is one instance of its second branch, over one three-row fixture. That is the most a test can do here, and saying so is the difference between evidence and a flourish.</p>
|
||
<p>The test still had no predecessor. <code>NumericCategory</code> had no coverage before it — <strong>and it was one of three, not the only one</strong>: grepping each <code>Artefact</code> variant and each variant's <code>reason()</code> text, <code>MissingFlag</code> and <code>DatePart</code> are covered by <code>the_missing_flag_and_the_date_sub_features_are_refused_too</code>, and <strong><code>ValueOfAMeasure</code> and <code>AbsentCoordinate</code> have no test at all</strong>, before this change or after it. Two families remain uncovered; that is filed, not fixed here.</p>
|
||
<p>What the decline does cost, stated rather than waved at: a user whose <code>Year</code> category really is a quantity they want fitted <strong>as a number</strong> has no way to say so through <code>fit_measure</code>. The document says <code>Year</code> is a category of items, and that is what is honoured. The remedy is a document change — <code>Year</code> belongs in <code>_Measure</code> if it is a measurement — and <code>crate::dataset</code> is <code>pub</code> and takes the caller's <code>Overrides</code>, so a host that wants to offer the choice can (gotcha 37d). Offering it <em>inside</em> <code>fit_measure</code> would be a new user-facing control, and nobody has asked for one.</p>
|
||
<p>The refusal's wording was narrower than the rule and is now aligned in both places that carry it: it said <em>a category with a distinct value per record becomes one column per record</em>, which diagnoses the runaway shape and not <a class="tid" href="../tickets.html#improvise-e8no" title="A wide numeric category now refuses a fit that used to ignore it">improvise-e8no</a>'s — a category of modest cardinality that is simply wider than the budget. Both now say a category becomes one column per distinct item, <strong>whether its items read as numbers or not</strong>, since that clause is the whole of what changed under zv8t.</p>
|
||
<h3 id="quality-is-an-enum-not-an-option"><a class="anchor" href="#quality-is-an-enum-not-an-option">#</a><code>Quality</code> is an enum, not an <code>Option</code></h3>
|
||
<p><code>cv_regress</code> pools no pairs below two folds, and scoring an empty pool gives an R² of exactly zero — which reads as "this rule is worth nothing" when what happened is that nobody checked. <code>Quality::Measured</code> carrying an all-zero <code>RegressionQuality</code> means "this tree is no better than the mean", the loudest thing a fit can say, and it must not be reachable by simply not having asked. So the enum, and a host that wants the numbers has to name the case where there are none.</p>
|
||
<p>The fold cap is <code>rows / 2</code>, not <code>rows</code>. <code>fit_csv</code> clamps to the row count, which lets two rows become two folds of one: each tree is then fit on a <em>single</em> row — a constant — and the numbers come back meaning nothing while wearing the <code>Measured</code> label. That is the confusion <code>Quality</code> exists to prevent, moved from one row to two rather than removed.</p>
|
||
<h3 id="fitted-measure-name-a-fit-is-a-rule-about-the-data"><a class="anchor" href="#fitted-measure-name-a-fit-is-a-rule-about-the-data">#</a><code>fitted_measure_name</code> — a fit is a rule <em>about</em> the data</h3>
|
||
<p><code>"{target} (fit)"</code>, unless the caller names another. Storing it under <code>target</code> itself would make the formula shadow the very cells it was fitted to: gotcha 7's <code>_Measure</code> items are explicit items ∪ formula targets, so the two would be one item with a computed value, and the user's typed data would stop being visible.</p>
|
||
<p>The destination is refused when it names <strong>any</strong> existing item, not only an existing measure. <code>Expr::Ref</code> resolves through <code>Model::find_item_category</code>, which finds a <em>regular</em> category's item before it falls back into <code>_Measure</code> (gotcha 7) — so naming the fit after an item of some category installs cleanly and then makes every later formula that references it read the category coordinate instead. Asking the document the same question <code>Expr::Ref</code> will ask is both stronger than a measure-only check and simpler.</p>
|
||
<h3 id="one-line-is-public-because-the-refusals-need-it-too"><a class="anchor" href="#one-line-is-public-because-the-refusals-need-it-too">#</a><code>one_line</code> is public because the refusals need it too</h3>
|
||
<p><code>.improv</code> pipe quoting makes <code>\n</code> legal inside a category or item name, so a document can name a measure that splits the TUI status bar in two. The whole composed status line is flattened once at the end rather than each part being sanitised and one being forgotten.</p>
|
||
<p>The <strong>refusals</strong> need it as well, and they are composed by the host: several interpolate a document name, and a failure is when a user is most reading the bar. Flattening only the success path left the failure path able to split it.</p>
|
||
<p>A related trap with no mechanism guarding it: a <code>rustfmt</code> pass over a multi-line string literal leaves runs of ten spaces mid-sentence in a message a user actually sees — invisible in the source and in every test that only asks whether a substring is present. <code>a_name_containing_a_newline_cannot_split_the_status_line</code> asserts the one-line rule for that reason.</p>
|
||
<h2 id="15-known-gaps-filed-rather-than-fixed"><a class="anchor" href="#15-known-gaps-filed-rather-than-fixed">#</a>15. Known gaps, filed rather than fixed</h2>
|
||
<p><strong>Closed: aggregate-then-route is not route-then-aggregate (<a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>).</strong> Kept here because the shape recurs. A tree is fitted on individual <strong>records</strong>, and an improvise formula evaluated at an aggregating <code>CellKey</code> used to aggregate each <code>Expr::Ref</code> first and evaluate the <code>IF</code> once on the totals — the same answer for a linear formula, a different one for anything that routes, so the stored rule meant one thing on a record cell and another one row up. The emitter now wraps every tree in <code>SUM(…)</code> and <code>eval_agg</code> folds a non-<code>Ref</code> inner expression over the member records: <a href="../code/core.html"><code>context/code/core.md</code></a> §13. What remains of it is a <strong>cost</strong>, not a defect — the fold is O(members) per evaluation per fixed-point pass and is unbenchmarked (<a class="tid" href="../tickets.html#improvise-9iyv" title="The route-then-aggregate fold's cost, measured">improvise-9iyv</a>).</p>
|
||
<p><strong>Two artefact families are still unspellable (<a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>).</strong> <code>col.missing</code> and the date sub-features stay <code>Spelling::Artefact</code> and are still refused. §11 says what each would need; they are independent of each other.</p>
|
||
<p><strong>A mean-imputed feature is read raw by the formula.</strong> <code>preprocess</code> fills a missing numeric cell with the column mean, so a champion may split on a value the training rows had and the <em>document</em> does not. The stored formula reads the document, so that record evaluates to something other than what the fit predicted for it — quietly, since the measure name resolves and only the cell is absent. <code>fitted_formula</code>'s check asks whether the document can <em>name</em> the feature, a different question from whether every record has one. Closing it means choosing between refusing any feature with a missing cell — which would refuse most real documents, imputation being why <code>preprocess</code> imputes — and teaching the formula language an <code>IFMISSING</code>-shaped primitive. A product decision of the same kind as <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>.</p>
|
||
<p><strong>A boundary value can route differently in the formula than in the fit.</strong> Training narrows each <code>f64</code> cell to <code>f32</code>; the stored formula compares the original <code>f64</code> against the threshold. <code>number_text</code> guarantees the threshold reads back as the same <code>f64</code>, so the <em>threshold</em> is exact — but a cell value just below a threshold that <code>f32</code> rounds <em>to</em> it takes <code>hi</code> in <code>gp_tree::eval</code> and <code>lo</code> in formula evaluation. Pre-existing in the emitter (<a class="tid" href="../tickets.html#improvise-izk" title="gp-tree step 1: emit a fitted Tree as an improvise formula string">improvise-izk</a>); the fix, one numeric representation end to end, would rewrite the bit-level characterization tests that are the only evidence the routing argument holds.</p>
|
||
<p><strong><code>Quality::Measured</code> is transductive rather than fully held out.</strong> Preprocessing runs over <strong>all</strong> rows before <code>cv_regress</code> picks its folds, so an imputation mean and a categorical vocabulary are learned partly from the rows they are then validated against. And it is wider than imputation: <code>Spec::ranges</code> is computed over all rows and the <em>same</em> spec goes into every training fold, so the thresholds the search samples and the branches <code>simplify</code> prunes are bounded by a held-out minimum or maximum. <strong>That reaches a fit over complete numeric measures too</strong> — the only shape <code>fit_measure</code> currently stores — so "the case that stores leaks nothing" is wrong. Closing it means building the ranges and the preprocessing plan per training fold and encoding each validation fold with <em>that</em> plan, which <code>cv_regress</code> cannot express: it takes one spec and rows that are already encoded. That is an API change in <code>gp-tree</code>, and it is independent of <a class="tid" href="../tickets.html#improvise-4zl7" title="A fitted tree must route per record and aggregate the results, not route on the aggregate">improvise-4zl7</a>, which turned out to need no change to how <code>cv_regress</code> is fed.</p>
|
||
<p><strong>Which features a champion picks is a property of the search</strong>, which used to mean the same command refused or succeeded depending on the seed. For the categorical family that is gone: a one-hot over a real category is spellable whichever way the search goes. It survives for <code>col.missing</code> and the date parts (<a class="tid" href="../tickets.html#improvise-wd6n" title="A fit may still not split on col.missing or a date sub-feature">improvise-wd6n</a>), and it is the reason those are worth closing rather than living with — a refusal that depends on a seed is not a contract.</p>
|
||
<h2 id="16-preprocess-what-each-column-kind-expands-into"><a class="anchor" href="#16-preprocess-what-each-column-kind-expands-into">#</a>16. <code>preprocess</code>: what each column kind expands into</h2>
|
||
<p>Real data is not a matrix of floats. Columns are numeric, categorical or dated; cells go missing; a category is unordered and a <code>< thresh</code> split over its arbitrary codes would be meaningless. So each raw column is inferred (or forced through <code>Overrides</code>) to a <code>ColumnKind</code> and expanded:</p>
|
||
<ul>
|
||
<li><strong>numeric</strong> → one feature, missing cells mean-imputed;</li>
|
||
<li><strong>date/datetime</strong> → ordered parts (<code>col.epoch_day</code>, <code>col.year</code>, <code>.month</code>, <code>.day</code>, <code>.dow</code>, and <code>.hour</code>/<code>.minute</code> when the column has a time), so trend and seasonality are learnable rather than one opaque token per timestamp. <code>epoch_day</code> is Howard Hinnant's <code>days_from_civil</code>; <code>parse_datetime</code> requires a 4-digit year so plain hyphenated numbers ("1-2-3") are not mistaken for dates;</li>
|
||
<li><strong>categorical</strong> → one 0/1 column per distinct value (<code>col=value</code>), which is the correct treatment of an unordered category — the tree then splits on "is it this value?". A missing cell becomes the reserved <code>(missing)</code> level.</li>
|
||
</ul>
|
||
<p>Any column that had a missing cell also emits a <code>col.missing</code> 0/1 indicator, so <strong>absence itself</strong> is something the tree can split on rather than a silent substitution. A clean column emits none, or every fit would carry a constant feature.</p>
|
||
<p><strong>The expansion is not reproducible from a fresh table</strong>: imputation means and the one-hot value list are properties of the <em>training</em> rows, and a later table may not even contain some of them. <code>preprocess</code> therefore records a <code>ColumnSpec</code> per raw column and <code>encode_row</code> replays it; that pairing of fitted trees with their encoding plan is <code>data::Model</code>. A value never seen in training is simply all-zeros, not a new column.</p>
|
||
<p>Note for §14's preflight: <code>build_categorical</code> pushes the range <code>(0.0, 1.0)</code> for every one-hot <strong>whatever the column contains</strong>, so a constant one-hot is <em>not</em> pruned by <code>simplify</code> — it cannot know the column is constant. A test fixture that relies on a degenerate range being pruned is relying on seed luck.</p>
|
||
<h2 id="17-fit-csv-the-cli-s-own-documentation"><a class="anchor" href="#17-fit-csv-the-cli-s-own-documentation">#</a>17. <code>fit_csv</code>: the CLI's own documentation</h2>
|
||
<p><code>examples/fit_csv.rs</code> is the crate's <em>second</em> use case (the first was a game bot), a wholly different domain — the point being that <code>gp-tree</code> knows nothing about features or fitness. Almost none of what it does is <strong>in</strong> that file: preprocessing is <code>gp_tree::data</code>, the search is <code>gp_tree::fit</code>, and what is left is flag parsing, the demo dataset, and the printing.</p>
|
||
<pre><code>cargo run -p gp-tree --example fit_csv -- path/to/data.csv
|
||
cargo run -p gp-tree --example fit_csv -- --save model.json path/to/data.csv
|
||
cargo run -p gp-tree --example fit_csv -- --predict model.json path/to/new.csv</code></pre>
|
||
<p>With no path a built-in categorical dataset (the classic "play tennis" decision-tree example) is fitted, so it runs standalone.</p>
|
||
<p><strong>Shape.</strong> The header names the columns; the LAST column is the target, the rest are features. A numeric target is a <strong>regression</strong> (one tree, minimising MSE); a categorical target is a <strong>classification</strong> — one tree per class fit to a 0/1 indicator (one-vs-rest), predicting by argmax.</p>
|
||
<p><strong>What a fit reports</strong>, in order: a <em>generalization</em> metric first — <code>k</code>-fold cross-validation against a trivial baseline (predict-the-mean / majority-class), with R²/RMSE for regression and a confusion matrix plus per-class precision/recall for classification — then the final tree(s) fit on all rows, whose train error is optimistic, then a ranked <strong>permutation feature-importance</strong> table.</p>
|
||
<p><strong>Tuning flags.</strong> <code>--cv <k></code> (folds, <code>< 2</code> disables; default 5), <code>--seeds <n></code> (keep the best of <code>n</code> GA restarts; default 1), <code>--penalty <λ></code> (per-node parsimony cost keeping trees small and general; default 0.01), <code>--seed <n></code> (the RNG seed, decimal or <code>0x</code>-hex; printed by every fit and saved with <code>--save</code>).</p>
|
||
<p><strong>Column directives.</strong> <code>--ignore a,b</code> drops columns (ids, free text); <code>--numeric</code> / <code>--categorical</code> / <code>--date a,b</code> force a column's type — a numeric-looking zip code that is really a category, say.</p>
|
||
<p><strong>Messy CSVs.</strong> Missing cells (blank or <code>NA</code>/<code>null</code>/<code>?</code>/…) are mean-imputed for numeric and date columns, with a <code>col.missing</code> indicator so absence is itself learnable, and become a <code>(missing)</code> level for categorical columns. Rows missing the <em>target</em> are dropped up front, with a count on stderr.</p>
|
||
<p><strong><code>--save <path></code></strong> additionally writes a JSON <code>data::Model</code>: the fitted tree(s) plus enough of the preprocessing plan to re-encode a fresh CSV's raw cells into the exact same feature layout, <strong>and</strong> the seed and the <code>FitConfig</code> those flags produced — because the seed alone does not reproduce a fit (§9).</p>
|
||
<p><strong><code>--predict <model.json> <data.csv></code></strong> loads that model and scores each row of a CSV of raw feature columns (same names as training, any order, target column optional and ignored), printing one prediction per row — plus a softmax <code>confidence</code> column for classification.</p>
|
||
</main>
|
||
</div>
|
||
</body>
|
||
</html>
|