improvise/context/test-evidence/gp.md

Test evidence: gp-tree and improvise-gp

What each test's bug was, what fault was injected to prove the test can fail, and why the obvious weaker version of it proved nothing. Keyed by exact test name — a rename orphans the entry, which scripts/check-comment-runs.py --check-evidence catches.

The architectural arguments these tests defend are in context/code/gp.md.

---

gp-tree/src/lib.rs

splitmix64_matches_the_reference_vectors

The vendored RNG's constants had nothing pinning them. Changing the gamma from 0x9E3779B97F4A7C15 to …7C17 left every other test in the workspace green, because they only ever assert that two runs at one seed agree.

The crate's promise is stronger: a fit's seed is printed to the user and saved in data::Model, so a later run at an old seed must reproduce the old fit. These are therefore the published SplitMix64 vectors for seed 0, not a golden capture of this code — which is what makes them evidence rather than a tautology.

splitmix64_matches_the_reference_vectors_from_a_nonzero_seed

Seed 0 alone cannot catch a change to how the seed enters the state (state = seed versus state = 0), because both give the same stream at 0. Also asserts seed() still answers the seed after the generator has advanced.

---

gp-tree/src/fit.rs

mse_averages_over_exactly_the_rows_it_was_given

mse used to take parallel x/y slices, sum over zip(x, y) — which stops at the shorter — and divide by x.len(). A caller that got the two out of step was handed a mean that was quietly too small rather than an error: the improvise-n3wt hazard, a count supplied alongside the data rather than taken from it.

The remedy was not a length check but a type: one slice of Row, so the shorter slice cannot be built (improvise-dkqn). This test is what is left to say about it — numerator and denominator count the same rows, for any prefix.

scoring_no_rows_is_zero_error_rather_than_nan

Every mean in the module reached 0.0 / 0.0 on an empty slice and handed back a NaN a caller could not tell from a score: it propagates through FitConfig::cost and sorts unpredictably against real costs. One answer, in mean_over, covers mse, misclass_rate and standardize, and the test asserts all three plus std == 1e-6 (a degenerate spread still never divides by zero).

asking_for_no_folds_yields_one_rather_than_dividing_by_zero

k reaches folds from a CLI flag, so zero is a value a caller can supply rather than an impossible state. out[pos % k] divided by it and panicked — a public library function taking its host down with it.

cross_validating_without_a_held_out_fold_pools_nothing

The sharper half of the same parameter (improvise-ry89). A cross-validated prediction is one made by a tree that never saw the row it scores, so k < 2 asks for something that cannot exist: the single fold holds every index, the training complement is empty, and the tree is fit and standardized on nothing before predicting every row.

This used to announce itself — standardize(&[]) gave a NaN mean and every pooled prediction came back NaN. Once mean_over answered the empty row set with 0.0, the same call handed back near-zero numbers indistinguishable from a model's output, and an R² computed off them is a number rather than a NaN. So fixing one bug hid another, and the count is the contract: no held-out row means no pair. The test also asserts the generator is untouched, because declining before the shuffle is what keeps a caller's next draw the one it would have made.

the_training_split_is_what_the_contains_scan_gave

The O(n²) fix (improvise-t3ux) replaced test.contains(i) per row with a bool mask. It is only safe because the mask yields the same indices in the same order — the training rows are what every downstream draw from the generator depends on. Pinned directly, at every fold count, rather than inferred from a fit that happened to agree.

a_fit_improves_on_the_tree_it_started_from

Without it, a fit that returned its seed tree unchanged would satisfy both the_same_seed_and_config_fit_the_same_tree and a_different_seed_fits_a_different_tree, because the initial random_tree already differs across seeds.

fit_detailed_starts_from_the_best_of_generation_zero

What start has to mean for the question it exists to answer: the best of generation 0, not the seed ancestor it grew from (improvise-ut7p). The two differ whenever a mutated member of the initial population beats member 0, and only the former makes "the search improved on where it started" a claim about breeding. Before this, improvise-gp kept a second copy of the whole GA loop just to reach that starting point.

a_tiny_lone_importance_is_still_the_whole_of_it

Copilot on PR #25. fit_csv divided by total.max(1e-9), which is right for the only input it ever had and wrong as a contract: one feature carrying an importance of 1e-10 carries all of what the model knows and was reported at 10%. Importances this small are ordinary — they are error differences over a standardized target. the_shares_sum_to_a_hundred_at_any_scale is the general form across four orders of magnitude.

denormalize_rescales_the_leaves_and_leaves_the_splits_alone

The negative arm is the load-bearing one: standardize rescales the target, so a threshold is already in its feature's own units and re-scaling it with the leaves would be the bug. a_tree_fitted_on_standardized_rows_predicts_original_units_after_denormalize pins the arithmetic against a hand-built tree rather than a search, so it does not depend on the GA.

---

gp-tree/src/data.rs

an_entirely_missing_column_is_skipped_and_reported_as_data

preprocess used to say so with an eprintln!, which was correct while this code was examples/fit_csv.rs. A library does not own its caller's stderr: a TUI on an alternate screen, a wasm build, or a caller wanting the notice in its own status line could none of them suppress, redirect or translate it. The notice now comes back as data.

seed_zero_is_distinguishable_from_no_seed

0 is a legal seed, so "unknown" and "fitted at 0" have to be two different values — anything reporting the seed of a pre-seed model would otherwise state a seed that never ran. That is why Model::seed is an Option and not a sentinel.

a_saved_model_round_trips_with_its_whole_ga_config

The seed alone is necessary and not sufficient: the same seed under a different FitConfig searches differently and lands somewhere else, so a model carrying only a seed looks reproducible while being reproducible only by whoever still remembers the flags.

The fixture a_fit_config() differs from FitConfig::default() in every field, and the test asserts that explicitly. A round trip that silently substituted the default would otherwise pass against a default-constructed value — the failure mode this test exists for.

a_model_saved_before_configs_still_loads

A backward-compatibility test aimed at the absent field, not at the attribute. #[serde(default)] on an Option is documentation rather than mechanism — serde defaults an Option anyway, so a test that merely deletes the attribute fails nothing and is not discriminating. This one removes config from the serialized JSON and asserts it loads as None (unknown), not as Some(FitConfig::default()). a_model_saved_before_seeds_still_loads is the same shape for seed.

---

gp-tree/tests/scoring_known_answers.rs

Whole-file note: these are f32 bit patterns, not approximate values, because a change that merely moves an accumulation around silently re-points every seed already saved at a different tree. assert!((a - b).abs() < 1e-6) cannot see that, and neither can the CLI's output, rounded to three or four decimals. The numbers were captured from the parallel-x/y code before improvise-dkqn reshaped the signatures and improvise-t3ux replaced the contains scan, and are unchanged by both. If you are changing the search on purpose these numbers change with it — but then every seed recorded by an older --save names a different tree, and that is the decision this file forces you to make out loud.

standardize_is_bit_for_bit_what_it_was

A witness that could not see the bit. The first and last rows plus mean/std were the whole witness, and four numbers cannot separate two ways of computing the same quantity: injecting r.target / std - mean / std for (r.target - mean) / std — algebraically identical, a different rounding — left this test and both cross-validation vectors green (improvise-ry89). All thirty-seven values do separate them: that reassociation moves 21 of the rows, and none of them were rows 0, 36 or 19.

Note what is not evidence here. Reversing the mean's accumulation order also left every vector green, but on this fixture the two orders sum to the same f32 — so that injection changed no behaviour to catch. A perturbation has to move a bit before a bit-level witness can be blamed for missing it.

cv_regress_pools_the_same_predictions_and_leaves_the_rng_where_it_was

The rng_after witness is the part that pins the order of the generator's calls: the next u64 after a cross-validation depends on exactly how many draws that run made, so a reshaped loop that drew differently would move it even if every pooled prediction happened to agree. 37 rows is a prime, so no fold count divides it evenly and the last fold is short — where an off-by-one in the train/test split shows.

cv_classify_pools_the_same_codes_and_leaves_the_rng_where_it_was

The classification half of the same guard. The mask replaced contains in both loops, and one arm passing says nothing about the other (doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)).

---

improvise-gp/src/lib.rs

improvise_evaluation_matches_gp_tree_eval

The acceptance property: over random trees and random feature vectors, improvise's evaluation of the emitted formula matches gp_tree::eval exactly — widening f32 to f64 is lossless and the thresholds are exact, so there is no tolerance to allow.

The improvise_eval helper, and the shape to remember

This test and a_value_sitting_on_the_threshold_falls_the_same_side both run through the improvise_eval helper, and its scope is what was once read as covering more than it does. It fabricates a _Measure item per feature name, so it asserts the emitter and gp_tree::eval agree against a model that could not exist. It says Ok to outlook=rainy because it just made an item called that. A helper that builds the world the code wants is not evidence that the world exists.

So the awkward names above are a quoting fixture and nothing more: outlook=rainy is there because it is awkward to spell, not because a document could hold it — it could not, and fitted_formula refuses it. The helper is still the right tool for the arithmetic-and-quoting claims; the property it could not state is asserted separately, against a real document, in evaluability_in_a_real_document.

an_awkward_target_name_is_quoted_and_round_trips

This test used to assert parsed.target == "|Total Revenue|" and call the pipes correct. They are not: the target names a _Measure item, so a target carrying its delimiters writes the fit into an item literally named |Total Revenue|, which is not the item the user has. What the emitter owes its caller is a semantic round trip — the name goes in, the same name comes back out. an_awkward_target_names_the_users_measure_item is the consequence that makes it a data bug rather than a cosmetic one.

an_infinite_threshold_is_refused_rather_than_emitted_as_inf

number_text(f32::INFINITY) emitted "inf", which "inf".parse::<f64>() accepts — so the round-trip loop returned it and a formula_text caller got text the grammar rejects, with nothing on the way saying so. fitted_formula would surface it as a parse error; a bare formula_text caller would write it into a file.

a_nan_threshold_is_refused_by_name

NaN fell through the loop to unreachable!, so the panic blamed the decimal expansion instead of the caller's value.

a_threshold_is_the_shortest_decimal_that_reads_back_the_same

The doc used to overclaim the emitted text as the f32's full decimal expansion. It is the shortest decimal that reads back as the same f64 — the weaker claim, and the one the routing argument actually needs: equal text means equal f64 means the same branch.

a_value_sitting_on_the_threshold_falls_the_same_side

The boundary case the whole number_text design exists for. 0.7f32 is really 0.699999988…; the shortest text that reads back as that f32 is "0.7", which parses as an f64 strictly greater than it. A row whose feature value equals the threshold routes hi under eval and would route lo through a formula carrying the rounded threshold. The test asserts the trap is live — "0.7".parse::<f64>() > thresh as f64 — before relying on it.

a_split_on_a_one_hot_feature_is_spelled_as_a_coordinate

improvise-8xhy's defect, and improvise-4zl7's answer to it. preprocess one-hots a categorical column into features named Region=North, an encoding artefact no document has an item for, so find_item_category returned None and every cell of the fitted measure came back as Error("ref:Region=North") — silently, with the emitter having said nothing. 8xhy shipped a refusal; 4zl7 replaced the refusal with a spelling, Region IS North.

The assertion is the emitted text, exactly, and not merely that the call is Ok: the whole failure mode here was a well-formed formula meaning the wrong thing, so "it parsed" is the assertion that missed it the first time. The fault to inject is Spelling::Coordinate rendering as quote_name(name) — byte-identical for every measure, and wrong for exactly this family.

a_coordinate_naming_no_category_of_this_document_is_refused

The negative arm of the same rule, and it is not redundant with the artefact refusal: a Coordinate is spellable and may still be unresolvable. Without it, resolves_in's Coordinate arm could return true unconditionally and every test above would still pass.

a_measure_that_merely_looks_like_a_one_hot_is_accepted

The negative arm, and why the check asks the document rather than refusing names that look like encoding artefacts: a measure literally named Region=North is a legal document. The two are indistinguishable as strings. the_missing_flag_and_the_date_sub_features_are_refused_too covers the other two artefact families with nothing enumerating them — the check asks the exact precondition Expr::Ref has, so a future naming scheme is covered without this crate learning about it.

a_formula_the_emitter_accepted_evaluates_everywhere

The property the lying test could not state: if fitted_formula says Ok, no cell of the target is a ref: error. It is also what stops the refusal tests above from passing by refusing everything.

a_tree_indexing_past_the_spec_says_so

improvise-hkt6. A tree whose feat runs past the spec violates the same precondition in two places, and the two answered differently: expr_text said what was wrong, mark_features said index out of bounds. Whichever ran first decided what the caller was told, and fitted_formula runs mark_features first — so the legible message was the one nobody ever saw. Driven through fitted_formula because both private functions are reached that way, and the message is asserted rather than merely the panic, since an out-of-bounds index panics too.

a_record_added_after_the_fit_is_scored_by_the_formula_alone

The load-bearing half of improvise-7na's answer: scoring a new row needs no saved encoding plan. fit_csv --predict calls encode_row because its new row arrives as untyped CSV text; improvise's new row is a record in the same document, scored through the stored formula from its own measures. So nothing about the --predict path argues for putting a gp_tree::data::Model in the file.

breeding_improves_on_generation_zero

What rules out a GA whose breed returned its input unchanged. a_different_seed_fits_a_different_formula is weak evidence on its own, since the initial random_tree already differs across seeds.

---

improvise-gp/src/dataset.rs

a_measure_on_the_page_axis_does_not_filter_away_the_training_rows

improvise-vx3. Workbook::new's default view puts _Measure on Axis::Page, so filtering_cats() named it, filter_coords pinned it to the first measure, and every "record" then carried exactly one measure cell — never the target. The whole table came back as "no visible record has a Revenue value to learn from" on a document plainly full of them, which reads as a data problem rather than as an axis one.

Every test in the module used View::new + on_category_added, under which a leading-underscore category lands on Axis::None — so the fixture was quietly building the one world the code worked in, and no test could see the default a real Workbook hands out.

The assertion compares the contents, not the shape. Pinning only the header and the row count passes on this fixture even with the bug present: _Measure's first item is Revenue, so every row still has a target and only the feature columns come back blank — the shape survives while the data does not.

a_paged_measure_that_is_not_the_target_does_not_empty_the_table

The sharper half: filter_coords pinned to the first item, so on a document whose target is not first the table came back empty.

the_feature_table_is_one_row_per_record_with_the_target_last

The literal row order is half of what this pins, and f32 accumulation over the rows is why: a reordered table makes a seeded fit irreproducible in a way nothing prints. Since improvise-shyt matching_cells yields insertion order, so the assertion now holds by construction; before it, a test in one process could not tell "sorted" from "this process happened to hash them in that order".

a_formula_measure_is_not_a_feature

The fixture lists Profit as an explicit _Measure item as well as making it a formula target, which is legal and is the only case this test can see: a target that is only implied by its formula never reaches the explicit item list, so a fixture without the explicit item passes whether the rule is implemented or not.

---

improvise-gp/src/fit.rs

Whole-module fixture note — document(). Every category is a Label, and that is the only thing that makes the success path deterministic. An earlier version left Region regular with a single item and claimed the constant one-hot had a degenerate (1, 1) range that simplify would prune. That claim is false: data.rs::build_categorical pushes (0.0, 1.0) for every one-hot whatever the column contains, so simplify cannot know it is constant and the champion was free to split on Region=North. These tests were passing on seed luck, and the comment asserting otherwise was the worst part (Copilot on PR #25). The targets are in the thousands so a fit that forgets to de-normalize is visibly wrong rather than subtly so. The view puts _Measure on Axis::Page because that is what Workbook::new does — fitting under the real default is not an extra case, it is the only honest one.

the_fitted_formula_predicts_in_the_documents_own_units

The de-normalization regression. fit runs against a standardized target, so a champion written down untouched predicts standard deviations: a well-formed formula, parsing cleanly, evaluating to ±5 where the user's data is 1000–4000. Nothing downstream can see that — it is a number in a cell.

Fault injected: dropping the denormalize call in fit_measure fails this and only this test. The loop panic!s rather than continues on a non-numeric cell, because a loop that skips every record asserts nothing and "no cell evaluated" is itself the improvise-8xhy failure. The claim is then read a second way off the formula text, which does not depend on evaluation at all.

a_category_and_a_measure_of_the_same_name_are_refused

Copilot on PR #25, and the sharpest finding of the round: asking the document whether it can resolve a feature name is not the same question as asking whether that name means the column the tree was fitted on. A numeric category called Cost encodes to a feature called Cost; with a _Measure item of the same name the emitted Expr::Ref("Cost") resolves cleanly — to the measure — and the stored rule silently reads a different column from the one it learned.

The first version of this test never built the collision. Adding a category Cost with items puts no Cost coordinate on the existing records, so feature_table emits an empty column, preprocess drops it, and there was no duplicate name to find; it passed on the positional role bug instead. The records here carry the coordinate, and the test asserts the fixture really produces two columns named Cost before asserting the refusal.

an_empty_category_before_a_measure_does_not_break_the_role_check

Copilot on PR #25. preprocess continues past an entirely-missing column without pushing a ColumnSpec, so Dataset::columns is not index-aligned with the header once anything is dropped. A positional i >= category_columns role test then read the surviving numeric measure as a category and refused a perfectly good fit. The test asserts the fixture really drops a column first.

a_coordinate_is_spelled_as_one_even_when_an_item_shares_the_one_hots_name

The collision a name cannot resolve, now closed by construction rather than by a refusal. The fixture adds a _Measure item literally named Region=North and asserts find_item_category resolves it — so a check that matched the one-hot's feature name against the document would have found the wrong column.

Under improvise-4zl7 the one-hot is never spelled by its feature name at all: Features::of reads the role off preprocess's plan and emits Region IS North, which cannot collide with any measure however the document is named. So the assertion is the absence of |Region=North| in the emitted text as much as the presence of Region IS. Injecting the fault means having spell fall through to Spelling::Measure for a OneHot role: the fit then succeeds, the formula reads |Region=North|, and it silently evaluates against the wrong column — which is the original bug wearing the new API.

a_high_cardinality_numeric_measure_is_one_feature_not_many

The preflight approximates preprocess's type inference, so it can disagree in both directions. A numeric measure with more distinct values than MAX_FEATURES encodes to one feature and must not be refused for having many values.

a_category_with_a_distinct_value_per_record_is_refused_not_searched

FitConfig bounds the search; nothing bounded the width. A regular category with one distinct value per record one-hots to one column per record, and permutation importance is repeats × rows × features evaluations on the draw thread.

The fixture had to be pushed past MAX_FEATURES, and the assertion names the width refusal specifically. At twelve records this document is refused too — but by fitted_formula noticing the champion split on Tag=r0, a different rule, which would have made the test pass with the guard removed entirely.

more_records_than_the_budget_is_refused_before_any_search

The width bound does not bound the work: every stage scales with rows. The assertion also requires the message to say what to do (Page or Filter).

a_document_with_no_usable_feature_is_refused_rather_than_panicking

preprocess drops an entirely-missing column, so a document whose only candidate feature is empty encodes to zero features — and random_tree then draws rng.below(0), which SplitMix64 answers with 0, and sample_thresh indexes spec.ranges[0]. A panic, on a legal document, for any seed that does not happen to open on a leaf.

This needs the all-Label fixture rather than merely an empty measure: with a regular category present its one-hot survives — range (0.0, 1.0) whatever it holds — and the fit returns a constant leaf instead of crashing.

a_non_finite_cell_is_refused_rather_than_panicking

A .improv cell holding inf parses as an f64, so this is reachable from a document rather than only from a bug. standardize turns an infinite target into NaN leaves, an inf..inf range makes sample_thresh produce a NaN threshold, and number_text then asserts — a panic out of a function whose signature promised a Result.

Verified by disabling the guard, and the two directions differ: an infinite target really does panic in number_text ("wants a finite value, got inf"), while an infinite feature poisons its range and quietly yields a constant leaf. A panic and a meaningless rule; both are refused.

NaN is deliberately absent from the cases: f64::NAN.to_string() is "NaN", which is_missing already spells as missing, so it is imputed away and never reaches the search.

a_name_containing_a_newline_cannot_split_the_status_line

.improv pipe quoting makes a newline legal inside a name, so a document can hold a measure whose name would split the TUI status bar in two.

a_fit_that_needs_a_category_now_spells_it_as_a_coordinate

improvise-8xhy reaching a real user, and what improvise-4zl7 bought: the same fixture, the same champion, and a formula instead of a refusal. The fixture clears both measures on every North record: clearing only Cost would leave North carrying both target values, so the one-hot would explain nothing and the champion would have no reason to split on it.

The second assertion is the one that earns its place. Checking only that the text says Region IS would pass for a rule that routed both groups to the same leaf — which is what a tree that could not see the coordinate would produce. The fixture's two groups are 1000 and 4000, so requiring one leaf under 2000 and one over 3000 separates "spelled it" from "spelled it and meant it".

The \n-and-double-space assertions moved with the refusal they guarded; the one-line rule itself is unchanged and is still asserted by a_name_containing_a_newline_cannot_split_the_status_line.

the_fold_count_moves_the_champion_and_so_is_recorded

Copilot on PR #25, and it is right: the outcome used to carry seed plus config, which is the pair gotcha 37b names — and that pair is not sufficient here. Cross-validation draws from the same generator the fit then draws from, so changing cv_folds alone moves the champion. Recording two of four fields documented the reproducibility contract without meeting it. The test asserts the seed and config are equal across the two requests first, so the difference can only be the fold count.

the_same_seed_under_a_different_config_is_a_different_search

The knob varied is pop, not generations. More generations over an exactly-learnable fixture can land on the same optimum and simplify normalises it, so that version of the test would fail one day for a reason that is not a regression. pop changes how many RNG draws each generation makes, so the two searches diverge from the first one.

a_fit_named_after_a_category_item_is_refused

Copilot on PR #25. The check used to ask only about measures, and Expr::Ref does not: find_item_category finds a regular category's item before falling back into _Measure (gotcha 7). So naming the fit after some category's item installed cleanly and then made every later formula referring to it read the category coordinate. a_later_formula_referring_to_the_fit_reads_the_fit is the property that refusal protects, asserted against a real document.

a_fitted_rule_at_an_aggregated_cell_routes_per_record_and_sums

The characterization PR #25 left here, now replaced by the rule it was waiting for. A tree is fitted on individual records; an improvise formula is evaluated at a CellKey, and evaluate_aggregated used to aggregate each Expr::Ref first and evaluate the IF once on the totals. Linear formulas are unaffected — SUM(Revenue) - SUM(Cost) is SUM(Revenue - Cost) — and a tree is not, so the stored rule meant one thing on a record cell and a different thing one row up. improvise-4zl7's ruling was to aggregate the rule over the members.

The old test asserted the negative and carried a message telling whoever broke it to replace it with the rule. It fired on the first run of the fold.

What this one asserts is the property the ticket names: the aggregated cell equals the sum of the per-record predictions. That compares two different code paths — a fully-specified key evaluates one member, an aggregating key folds twelve — rather than one path against itself, but it is still an end-to-end agreement rather than a hand-computed number. The hand-computed evidence is in improvise-core, under route_then_aggregate_tests, on a four-record fixture whose every expected value is written out in the test name or the assertion.

The third assertion is what stops this one being satisfiable by accident: routing once on the total lands on a single leaf, so the test refuses a fixture in which a leaf happens to equal the right answer.