improvise/context/test-evidence/improvise-incremental.md

Test evidence: improvise-incremental

What each test guards, which rival it was run against, and where that rival failed. The reasoning is in context/code/improvise-incremental.md; the decisions are ADR 009 clause 2's.

How the rivals are kept. The the_harness_catches_* tests run differential on a Rival, an Eager told every edit and wrong in one way, and assert the step at which the harness reports the divergence. They run on every build, so each rival is watched failing on every run and not once by hand.

Injected in the source. Eighteen faults were put into src/eager.rs, src/harness.rs and src/edit.rs one at a time, each run against the whole suite and each reverted and followed by a full rerun. The failing tests, by name (a fault no test names survived, and none did):

• Eager::value ignores its Aggregation: eager_follows_an_edit_to_the_cell_a_fold_reads, eager_matches_the_evaluator_on_the_shipped_documents, eager_raw_fold_is_never_a_formula, eager_value_reads_a_formula_exactly_and_folded, the_harness_catches_a_backend_that_ignores_the_aggregation, the_harness_catches_a_wrong_folded_formula_value;
• Eager::raw_fold reads Exact: eager_raw_fold_sums_by_default, eager_raw_fold_follows_the_measures_aggregation, eager_raw_fold_of_text_is_unanimous_or_mixed, eager_numbers_among_text_are_mixed, eager_follows_an_edit_to_the_cell_a_fold_reads, eager_matches_the_evaluator_on_the_shipped_documents, the_harness_catches_a_raw_fold_that_returns_a_formula;
• Eager::raw_fold returns a formula's value: eager_raw_fold_is_never_a_formula, eager_matches_the_evaluator_on_the_shipped_documents, the_harness_catches_a_raw_fold_that_returns_a_formula;
• Eager::apply does not recompute formulas: eager_follows_an_edit_to_the_cell_a_fold_reads, a_maintained_eager_equals_a_fresh_one_over_random_edit_sequences;
• the harness compares only the first edit: the_harness_catches_a_backend_that_ignores_a_clear, …_ignores_the_measure_aggregation, …_goes_stale_after_two_edits, the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula;
• the harness never reports a divergence: all nine the_harness_catches_* tests;
• same_value always true: same_value_follows_the_float_rules_and_is_exact_elsewhere and four the_harness_catches_* tests;
• same_value NaN unequal, signed zero by bits, any two text equal: each fails same_value_follows_the_float_rules_and_is_exact_elsewhere and nothing else;
• probes omit formula cells: probes_cover_folds_formulas_measures_and_groups and four the_harness_catches_* tests;
• probes drop at most one coordinate: probes_cover_folds_formulas_measures_and_groups, the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped;
• probes omit raw folds: probes_cover_folds_formulas_measures_and_groups, an_edit_names_its_key_as_two_value_probes_and_a_fold, the_harness_catches_a_raw_fold_that_returns_a_formula;
• probes omit group keys: probes_cover_folds_formulas_measures_and_groups, the_harness_catches_a_wrong_group_fold;
• the probe history is forgotten between steps: …_goes_stale_after_two_edits, …_ignores_a_clear, the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula;
• edit_probes names nothing: an_edit_names_its_key_as_two_value_probes_and_a_fold;
• Edit::AddFormula as a no-op: eager_add_and_remove_formula_change_what_a_key_reads;
• Edit::ClearCell applied as SetCell(0.0), which both sides of a differential share and so only a direct Eager test can see: eager_clear_cell_leaves_the_key_absent.

eager_raw_fold_sums_by_default

Hand-computed: Revenue under Product A is 10 + 5 = 15. Fails on the raw_fold-reads-Exact rival (a partial key names no stored cell, so the answer is None).

eager_raw_fold_follows_the_measures_aggregation

After measure_agg[Revenue] = Max the same fold is 10, and Cost, unchanged, still sums to 6. A fold that always sums fails on the first; one that always takes the maximum fails on the second.

eager_raw_fold_of_text_is_unanimous_or_mixed

Note under A is two equal texts and reads as that text; under B it is two different texts and reads Error("mixed"). A fold that drops text, which the evaluator used to do (gotcha 7), returns None for both.

eager_numbers_among_text_are_mixed

A text among numbers is Error("mixed"), not the sum of the numbers.

eager_value_reads_a_formula_exactly_and_folded

Profit = Revenue - Cost reads 6 at East/A exactly, and 9 at Product A with Region over hidden: (10 + 5) - (4 + 2), the fold taken before the subtraction. Fails on the value-ignores-Aggregation rival.

eager_follows_an_edit_to_the_cell_a_fold_reads

After East/A Revenue becomes 20 the fold is 25 and the folded Profit 19. The Profit half is what fails when apply forgets to recompute formulas: the folded cache is then stale.

eager_matches_the_evaluator_on_the_shipped_documents

Both shipped documents, every view's None categories, and the hand model: every probe of Eager equals a direct evaluate_aggregated on a model recomputed under the same None categories, and the probe set is non-empty. Fails on both aggregation rivals.

eager_agrees_with_itself_over_every_corpus_edit_sequence

differential over a generated edit sequence on each corpus model returns a Report with edits + 1 steps and a non-zero probe count. The count assertion is what a harness that compared nothing would fail; the step count is what one that stopped early would fail.

the_harness_passes_a_backend_that_is_right

The control for the catches tests: Eager as the candidate over the hand-built edits is a pass, so a harness that rejects everything is seen.

the_harness_catches_a_backend_that_ignores_the_aggregation

A candidate that reads every value as Exact diverges at step 0, first at the full-key formula cell {East, A, Profit} over hidden (eager None, candidate 6).

the_harness_catches_a_backend_that_ignores_the_measure_aggregation

A candidate that drops SetMeasureAgg diverges at step 2, the edit that sets it, on a Revenue key.

the_harness_catches_a_backend_that_goes_stale_after_two_edits

A candidate that stops applying edits after the second diverges at step 3, on a Revenue key. A prefix of steps cannot see a backend that goes wrong late.

the_harness_catches_a_backend_that_ignores_a_clear

A candidate that keeps a cleared cell diverges at step 3 and the message names it.

the_harness_catches_a_wrong_folded_formula_value

Review of PR 343 (Codex P1): a rival wrong only at value({Product=A, _Measure=Profit}, OverHidden), the Profit cell a view with Region on None draws, passed 128 probes, because formula cells were built over full keys only. Now the divergence is at step 0 on exactly that probe, eager 9, candidate -999.

the_harness_catches_a_backend_that_keeps_answering_for_a_removed_formula

Review of PR 343: probes were rebuilt from the post-edit model, so a candidate that ignores RemoveFormula was never asked about Profit again. Probes now accumulate over every step. The divergence is at the last step, on a Profit key, eager None, candidate Some.

the_harness_catches_a_fault_that_only_shows_after_two_coordinates_are_dropped

Review of PR 343 (Copilot): only one-coordinate partials and the empty key were probed. A rival wrong only at {_Measure=Revenue} now diverges at step 0 on Value({_Measure=Revenue}, Exact). The bound is MAX_DROPPED = 3 coordinates; a fold dropping more is not asked about.

the_harness_catches_a_raw_fold_that_returns_a_formula

Review of PR 343 (Copilot): raw_fold is documented never to be a formula. A rival whose raw_fold is value(partial, OverHidden) diverges at step 0 on a RawFold of a Profit partial (eager None, candidate Some). It could not be seen while Eager itself returned the formula.

the_harness_catches_a_wrong_group_fold

A group name stands in for an item in the probe keys. A rival wrong only at {Region=Coastal, Product=A, _Measure=Revenue} diverges at step 0, eager 15, candidate -999.

eager_folds_a_group_it_is_asked_about

The group fold in the grouped model is East + West = 15 exactly, so the rival test above has a true answer to differ from.

eager_raw_fold_is_never_a_formula

At {Product=A, _Measure=Profit} the folded value is 9 and the raw fold is None. Fails on the fault that lets raw_fold read the formula cache.

eager_add_and_remove_formula_change_what_a_key_reads

Review of PR 343: Edit::AddFormula as a no-op left all 25 tests green, because an Edit fault reaches both sides of a differential alike. Double = Revenue * 2 reads None before the edit and 20 at East/A after it, and None again after RemoveFormula. Fails on AddFormula as a no-op.

eager_clear_cell_leaves_the_key_absent

After ClearCell the key reads None and the model holds no cell. A ClearCell applied as SetCell(0.0) hits both sides of a differential identically, so only this direct test fails.

same_value_follows_the_float_rules_and_is_exact_elsewhere

NaN equals NaN, -0.0 equals 0.0, 1.0 differs from 1.0000000000000002, texts and errors compare exactly, and None equals only None. Each of three one-line faults in same_value (NaN unequal, signed zero by bits, any two texts equal) survived the whole suite until this test existed.

probes_cover_folds_formulas_measures_and_groups

probes includes a raw fold, a full key, a folded formula cell under both aggregations, the two-dropped key {_Measure=Revenue}, the empty key and a group key. Review of PR 343 showed probes omitting formula cells, _Measure keys or raw folds left every test green.

an_edit_names_its_key_as_two_value_probes_and_a_fold

edit_probes of a ClearCell names the cleared key three ways, and of an edit naming no key names nothing.

a_maintained_eager_equals_a_fresh_one_over_random_edit_sequences

Clause 2 asks for a property test over a random edit sequence. A proptest (256 cases, up to four edits of five kinds over the hand-built model) runs differential with a candidate that rebuilds an Eager from scratch on every edit. It fails on the Eager::apply-skips-recompute fault, which is the property: maintaining in place equals building afresh.