improvise/context/test-evidence/core.md

Test evidence — improvise-core

What broke, what fault was injected, and why each test can fail. Keyed by exact test name (design-principles §7). A renamed test orphans its entry; scripts/check-comment-runs.py --check-evidence catches a heading naming a test that does not exist.

Only tests whose narrative is longer than the code appear here. Most tests need none.

---

model/types.rs

apply_agg_over_nothing_is_absent_for_every_function

An empty cell must read as absent, not as zero — otherwise every blank intersection in a pivot renders 0 and a sparse model looks fully populated. Fails if apply_agg ever gains a Sum => 0.0 identity arm for the empty case. eval_agg deliberately has the opposite rule; see the next entry.

eval_agg_over_no_rows_is_zero_only_where_an_identity_exists

A formula aggregate over no rows is 0 for SUM/COUNT, which have an identity, and an error for AVG/MIN/MAX, which do not. This is deliberately not apply_agg's rule, and the two are separate functions precisely so the difference is visible. improvise-kdp named both after they had been duplicated inline in two evaluators; the test pins the contract directly rather than only through formula eval.

find_item_category_falls_back_to_formula_targets

add_formula never inserts its target into _Measure (the item is implied by the formula definition). So find_item_category must fall back to the formula list, or Expr::Ref("Profit") fails to resolve and every cell of a chained formula errors ref:Profit (improvise-nrs). The test asserts the setup — that the target really is not a stored item — before asserting the lookup, so it cannot pass by the item having been added after all.

stored_virtual_categories_match_their_names

CategoryKind::of_name is used all over the codebase in place of looking the Category up. That substitution is only sound because (a) a fresh Model really does store each reserved name under its virtual kind and (b) add_category hands back the existing virtual category rather than shadowing a reserved name with a Regular one. The second half is the arm that would break silently: a shadowed _Measure would still answer of_name correctly while the stored category had the wrong kind (improvise-2lh).

aggregating_one_text_value_yields_that_text

The head of the improvise-l4qx family. aggregate_cell_values used to be filter_map(as_f64), which dropped text. _Index/_Dim sit on Axis::None in every default view, so none_cats is non-empty almost always and the aggregating path was almost always taken — making every text cell in every document render blank. .improv carries pipe-quoted text as a first-class value, so a user could type a text value, save, load, and never see it again.

aggregating_differing_text_values_is_an_explicit_error

Two different words have no reduction. The user is told mixed rather than shown one of them arbitrarily, or a blank.

aggregating_text_among_numbers_is_an_explicit_error

The case that produced a plausible number before: the text was dropped and the rest summed, so the cell read as authoritative while under-reporting. A blank would have been noticed; a wrong total is not.

a_lone_number_still_counts_as_one_rather_than_passing_through

The regression guard for the hot path, and the test that pins the branch order in aggregate_cell_values. The all-numeric branch is checked first, which is the entire difference between this rule and "a single value passes through". Count is the only function that can tell them apart — every other function is the identity on a one-element set — so this test sets measure_agg["Status"] = Count and requires 1, not 7. Reordering the two branches passes every other test in the family.

aggregating_nothing_is_still_absent

The text rule must not turn apply_agg's emptiness rule into an error or a zero. Fails if aggregate_cell_values answers on an empty slice at all.

where_filter_absent_category_does_not_apply_formula

Profit = 42 WHERE Region = "East" evaluated against a key with no Region coordinate must not apply the formula: the condition cannot be satisfied, so the raw cell value (absent) is the answer. The bug was an if let Some(item_val) that failed to bind when the category was absent and fell through, applying the formula unconditionally — so a grand-total cell showed the filtered formula's value.

sum_inner_expression_constrains_which_cells_are_summed

SUM(Revenue) ignored its inner expression and summed every numeric cell matching the partial key. With Revenue=100 and Cost=50 both stored for Region=East, Total = SUM(Revenue) returned 150. The fixture needs two measures on one coordinate or the bug is invisible.

add_formula_same_target_name_different_category_both_coexist

add_formula deduplicated by target name alone, ignoring target_category, so Profit in _Measure and Profit in KPI could not coexist — the second silently replaced the first.

add_formula_same_target_name_different_category_evaluates_independently

The consequence of the same bug seen through evaluation rather than through formulas.len(): the dropped formula's cell evaluated to absent. Worth keeping beside the count assertion because a fix that deduplicates correctly but dispatches on name alone would pass the first and fail this one.

remove_formula_only_removes_specified_target_category

The mirror image: remove_formula matched by target name alone, so removing Profit from _Measure also destroyed the Profit formula in KPI.

near_zero_divisor_consistent_with_equality_epsilon

improvise-0bf. =/!= compare with FLOAT_EQ_EPSILON = 1e-10, but the division-by-zero guard checked rv == 0.0 exactly. With X = 1e-11 the comparison X = 0 was true and 100 / X happily returned 1e13 — incoherent semantics for IF(X = 0, a, y / X), whose else branch divides by a value its own condition called zero. Any value the equality operator treats as zero must also trigger div/0. This exercises the direct eval path (eval_expr inside eval_formula_depth).

near_zero_divisor_consistent_in_cached_eval_path

The same bug in the fixed-point/cached path (eval_expr_cached inside eval_formula_with_cache), whose division guard also used exact == 0.0. Two tests because there are two evaluators and a rule stated in one and not the other is a live bug — which is exactly how this one shipped.

recompute_formulas_multi_stem_values_stable

improvise-6os replaced an O(n²) Vec::contains dedupe of coordinate stems with an IndexSet — a pure performance change. The test pins the observable contract instead of the mechanism: a multi-stem model (chained formulas, several measures per stem) produces the stated values, and a second recompute_formulas call reproduces the exact same cache, so a dedupe that lost or duplicated stems would show up as a changed value or a non-idempotent cache.

formula_chain_preserves_full_precision

Display rounds; eval must not. Tax = Price * 0.075 on Price = 10 is exactly 0.75, which format_f64(_, true, 0) displays as "1". The test asserts both: that the display really does round (so the fixture is discriminating) and that Total = Price + Tax is 10.75 and not 11. Without the first assertion the test would pass on a fixture where rounding changes nothing.

model/cell.rs

matching_cells_yields_the_primary_insertion_order

improvise-shyt. matching_cells walked the secondary index's HashSet, so its order was hash order rather than entry order. DataStore::cells is an IndexMap precisely so records mode can list rows in entry order. The check is against iter_cells — the primary map's own order — and not against a literal, which would only pin whatever the hasher happened to do on this machine today.

the_same_cells_entered_in_opposite_orders_list_in_opposite_orders

The half a single-process test cannot reach by calling twice: two stores holding the same logical cells, built by opposite insertion histories. Each must list its own history, so one result is the reverse of the other. Under the index walk the two orders were unrelated to both histories and to each other, because RandomState keys each fresh HashMap differently. doctrine(an_order_test_needs_two_histories).

matching_values_yields_the_primary_insertion_order

The same guarantee for the value-only hot path, which aggregation folds f64 over. f64 addition is not associative, so an unstable order made a SUM reproducible only within one process — drift in the last bits, which is the kind of wrongness nobody reports and nobody can reproduce. Min/Max/Count and the text-unanimity rule are order-free and were never affected.

an_empty_partial_matches_every_cell_from_either_door

improvise-hkt6. matching_positions indexed sets[0] and panicked on an empty partial. Both public callers guard, so it was unreachable — but a guard one level away from the indexing is how this class of bug returns, so the input the guards prevent is fed in directly here.

It asserts the answer, not merely the absence of a panic: an empty partial constrains nothing, so every cell matches, and that must be the same answer the callers' fast paths give. Returning vec![] would also have been total and would have made deleting a fast path silently flip "all" to "none".

live_records_come_back_in_the_stores_own_insertion_order

improvise-2yc2. live_records walked the records: HashMap, so it yielded a different order in every process and in every instance — the improvise-shyt defect, caught here before a caller could inherit it. The order decided on is the store's own: a record sits where its first cell sits in the primary IndexMap. The fixture interleaves the measures deliberately — East's second measure arrives after West, and must not move East behind it.

the_same_records_entered_in_opposite_orders_list_in_opposite_orders

doctrine(an_order_test_needs_two_histories) applied to records: one history cannot tell "ordered" from "hashed that way today, on this machine".

a_recreated_record_takes_its_new_place_not_its_old_one

The claim live_records makes about why it derives the order from cells rather than memoising a position on RecordEntry — and which nothing else could see (Copilot, PR 20). A memoised position passes all three order tests above and still puts a re-created record back at its original place while its cells sit at the end of the store. The entry survives the deletion on purpose (it is a tombstone, below), which is exactly what makes the memoised position available to be wrong.

deleting_and_recreating_a_record_does_not_reset_its_version

The tombstone earns its place here. If RecordEntry were dropped when the last cell went, a delete-and-recreate would reset the counter to zero — and a client holding version 0 against the old record would be told its basis is current and its write applied. That is precisely the silent accept ADR 001's basis check exists to prevent.

a_surrogate_for_a_removed_record_fails_to_resolve

The first arm of staleness: a surrogate for a record whose cells are all gone must fail to resolve rather than resolve to some other record. The fixture holds two records, so a bug returning "any record" is visible; with one, the assertion would pass by naming the only record there is.

a_second_measure_on_a_record_resolves_to_the_same_record

The records-mode unit seen from the index side: two measures on one coordinate tuple are one record, so they share one entry. A cell-level unit would be violated by one legal edit (renaming a category coordinate moves every measure cell in the row).

a_live_records_surrogate_is_the_one_that_resolves_it

The pair live_records yields is not two facts that could drift apart: the surrogate is the record's own, so it is the one that resolves it back.

a_deserialized_store_can_resolve_its_records

The record index is derived state and is #[serde(skip)]-equivalent — it must come back with the data, rebuilt through set, rather than being written alongside it.

model/record.rs

a_records_surrogate_is_pinned_to_a_literal

The surrogate must mean the same thing in the process that minted it and in the one that restarts under it, so it is pinned to a literal rather than to "whatever this build computes". A failure here is not a bug in the test: it means the hash, the key or the byte serialization moved, and every surrogate any client holds is now a name for something else. That is a wire-format break.

the_two_kinds_do_not_share_a_hash

A record and an aggregate may be built from the same coordinates — an aggregate over a single record is the degenerate case. A client holding an aggregate id must not be able to spend it as a record id by accident, which is what the domain tag byte is for.

coordinates_that_concatenate_alike_do_not_hash_alike

The length prefixes in hash_coords earn their place: without them ("Ab","c") and ("A","bc") serialize to the same bytes, and a collision here is a silent write to the wrong record.

renaming_either_half_of_a_coordinate_changes_the_hash

Two categories, not one: a one-coordinate fixture cannot tell "the item moved" from "the category moved" — the same blind spot as the one-category fixture that could not distinguish TransposeAxes from CycleAxis.

a_surrogate_round_trips_through_json_as_a_string

serde_json writes a u128 as a bare number and a JavaScript host parses every JSON number as f64, so anything above 2^53 comes back changed — resolving to a different record or to none. The test asserts the wire shape (a quoted 32-hex string, 34 bytes with quotes), not merely that a round trip succeeds, because a numeric encoding round-trips fine inside Rust.

construction_order_does_not_reach_the_hash

CellKey is canonically sorted and without preserves that order, so "the normalized record" is a property the type guarantees rather than a convention to maintain.

view/types.rs

a_pivot_view_still_seats_categories_row_then_column_then_page

improvise-2zq, the negative arm, and the load-bearing one. The fix makes a records view seat a new category on Axis::None; without this test, "always seat on None" would pass every assertion about records mode while quietly deleting the auto-assignment on_category_added exists for.

Probing confirmed the asymmetry: seating every category on None fails 12 tests, while removing the virtual bump failed none at all — which is how the bump came to be wrong in two directions with nothing noticing.

a_records_view_seats_a_new_category_on_none_and_stays_a_records_view

on_category_added used to bump a virtual off Row or Column to make room. With _Index on Row and _Dim on Column that is records mode, so naming a category with N while in records mode silently ejected the user from the grid they were standing in and returned them to a pivot.

a_measure_on_column_is_not_bumped_off_it

The second thing the bump broke, and the reason the fix is about virtuals in general rather than about records mode: _Measure on Column is an ordinary configuration — it is what web/demo.improv ships — and adding a second regular category kicked it off the axis, making every measure vanish from the grid. The fixture is deliberately not a records layout, so it exercises the pivot path rather than the early return.

cycle_axis_unknown_category_is_noop

improvise-t8s: cycle_axis panicked through axis_of's expect when given a category not registered with the view, while set_axis silently no-ops on the same input. It now goes through try_axis_of.

tileable_cats_omits_filter_but_keeps_every_other_axis

improvise-dz8: a coordinate the drill fixed must not appear in tile select — the user cannot move it, so offering it offers an action that cannot be taken. The test flips the category back to Page afterwards, so it distinguishes "omits Filter" from "omits this particular category".

filtering_cats_covers_page_and_filter_but_not_the_rest

Filter constrains the data exactly as Page does; only Page is navigable. The fixture puts a category on each of Page, Filter, Row and None so the result separates the two questions rather than merely counting.

view/layout.rs

prune_axis_items_indexes_data_positions_not_visual_ones

prune_axis_items asks keep about data indices — the positions a row/col argument means — so a group header between two data items must not shift the numbering the predicate sees. The fixture places the header at visual index 1 and prunes data index 1: if the predicate saw visual indices, "A" would be the one dropped. improvise-n3wt also removed the &[bool] + debug_assert pair, whose length could disagree with the axis in release builds only.

a_text_cell_renders_in_a_default_view

The user-visible face of improvise-l4qx, asserted at the layout rather than at the aggregation helper. The test asserts its own premise first — !layout.none_cats.is_empty() — because if a default view ever stopped putting _Index/_Dim on Axis::None the test would pass without exercising the bug at all.

records_mode_gives_each_measure_its_own_value_column

Pins the whole column vocabulary in order: [Region, +cat, Revenue, Cost, +] — the regular categories, the new-category column (improvise-2zq, sitting where the column it creates will appear), one column per raw measure, then the new-measure column (improvise-c5v). It checks both the labels and the RecordCol variants, so a column drawn with the right label but resolved to the wrong kind fails.

records_mode_initial_layout_is_sorted

On entry into records mode the store is sorted by CellKey, so rows appear in a deterministic order regardless of insertion order. The fixture inserts East after North, the opposite of alphabetical, so "did nothing" fails.

records_mode_new_record_appends_at_bottom

The complement: once the layout exists, a new record appends rather than re-sorting into the middle. The new record's coordinate (AAA) sorts before every existing one, so a re-sort would move it to the top and move the row the user was looking at.

records_display_out_of_bounds_col_returns_none

improvise-byz: records_display for an out-of-bounds column called col_label(col), which returns ""; "" was then looked up as a category name in the record, never found, and the function returned Some("") — masking view/data mismatches as blank cells. It must return None, symmetric with cell_key.

view/axis.rs

the_label_vocabularies_are_unchanged

improvise-rml, characterization: these strings were scattered across four render/command sites before being pulled onto Axis. Pinned verbatim so the move could not quietly restyle the UI.

filter_is_outside_the_rotation

improvise-dz8: Filter names a coordinate the drill fixed, so it is outside the Row→Column→Page→None rotation entirely. Both directions are asserted — cycling a Filter must not hand the user a navigable axis, and no navigable axis may rotate into Filter, or tile select could hand one out.

next_is_a_four_cycle_over_the_navigable_axes

Applying next four times is the identity from every starting point and no navigable axis is a fixed point, so no variant is stranded or skipped. Together with the entry above this pins the rotation without enumerating it twice.

every_axis_gets_its_own_color

Distinct axes must be distinguishable at a glance; color() is the only reason AxisColor exists, so a collision defeats it.

a_routing_formula_over_a_group_sums_the_members_answers

improvise-4zl7, and the hand-computed half of its evidence. Every number in route_then_aggregate_tests is written out rather than obtained from a second call into the evaluator, because the bug being guarded is the evaluator answering consistently and wrongly.

The fixture is four records — Cost of 10 and 100 in North, 20 and 200 in South — and the formula is Pred = SUM(IF(Cost < 50, 1, 10)). Cost straddles the threshold inside each region, which is the whole design of the fixture:

cell	route-then-aggregate	aggregate-then-route
one record (Cost 10)	1	1
North (Tag away)	1 + 10 = 11	110 < 50 → 10
everything	1+10+1+10 = 22	330 < 50 → 10

A fixture whose records fall on the same side of the split would give the same answer either way at the record level and differ only by a factor of the record count, which is the kind of agreement that reads as correctness.

Two faults were injected and both were watched to fail. Restoring the old eval_agg — sum the matching raw values and ignore the inner expression — fails 7 of the 8 tests with 330 for 22, 110 for 11 and 10 for 1; the one survivor is is_is_one_when_the_cells_own_coordinate_matches_and_zero_otherwise, which has no SUM and so never reaches the fold. Making eval_is return 0.0 unconditionally fails exactly the 3 coordinate tests, with 400 for 214 and 200 for 14. The two faults failing disjoint-but-overlapping sets is itself the evidence that the tests separate the fold from the predicate.

a_coordinate_split_survives_aggregating_over_that_very_category

The case the ruling exists for, and the one the ticket's own text called the product question. With Region aggregated away the cell has no Region coordinate of its own, so a formula that read the cell's coordinate would see nothing and take Expr::Is's unwrap_or(false) branch — answering 100 once. The fold hands the expression a record key, so each of the four records answers for its own coordinate: 7 + 7 + 100 + 100 = 214.

a_member_whose_input_is_missing_errors_rather_than_being_dropped

The asymmetry between the Expr::Ref fast path and the fold, asserted rather than left to be discovered. SUM(Revenue) skips a record with no Revenue cell; SUM(<expr>) cannot, because the expression is what is being asked and an expression that will not evaluate is an error. The fixture clears one record's Cost and writes a different measure on the same record, so the record still exists and still has a member cell to be found by — otherwise the record would simply vanish from matching_cells and the test would prove nothing.

feature_roles_line_up_with_the_names_preprocess_emitted

Lives in gp-tree, recorded here because the rule it guards is improvise-4zl7's. feature_roles is the one statement of preprocess's expansion order and encode_row drives off it, so a drift is silent in both directions: the emitter would spell the wrong feature and a re-encoded row would land in the wrong column. The assertion rebuilds each feature's name from its role and compares it with the name preprocess emitted, which is the only thing that ties the two walks together.

The last four assertions exist because the loop is vacuous over a fixture that produces only one kind of feature. The fixture has a numeric column with a missing cell, a categorical column and a date column, so all four roles appear. Fault injected: swapping the Numeric and Missing pushes in feature_roles fails this test and encode_row_replays_the_training_expansion, which is the pair that would have disagreed.

a_formula_targeting_another_category_still_folds_over_its_records

Found by self-review, not by the ticket. The fold evaluates the inner expression at each member record's key, and the first version built that key by folding the record's coordinates onto context — which still carries the formula's own target coordinate. For the ordinary _Measure target that is harmless, because Expr::Ref overwrites _Measure on the way in. For a formula targeting any other category it is not: the record key keeps Levy=Tax, so evaluate_depth matches the formula again at that key and the whole cell converges to Error("circular") — a formula quietly self-referencing through its own aggregate.

The key is now folded onto partial (context.without(target_category), plus any inline WHERE), which drops the target coordinate by construction rather than by every caller remembering to. Fault injected by restoring the context fold: this test fails with Error("circular") against 11.0 and the other eight pass, which is why it is a separate test and not an extra assertion on one of them.

the_two_evaluators_agree_when_the_fold_reads_a_derived_measure

Adversarial review of PR #54, and the sharpest finding of the round: the first version of the fold reintroduced exactly the divergence the ruling exists to remove, through the cached path.

eval_expr_cached's Expr::Ref looks the value up in formula_cache at context.with(cat, name). Inside the fold, context is a member-record key carrying the none_cats coordinates, while formula_cache is keyed by stems with none_cats stripped. The lookup could never hit, fell through to aggregate_raw, found no raw cell behind a formula target, and answered Err("ref:…"). So a routing formula reading a derived measure gave Number(11.0) uncached and Error("ref:Margin") cached — and the cached path is the one a pivot grid runs whenever any category sits on Axis::None.

The fixture is the four-record one plus Margin = Cost * 2, so the fold's input is a formula rather than raw data. The test asserts the uncached value and that the two agree; asserting only agreement would pass if both broke.

The fix makes them agree by construction rather than by this test: eval_expr is hoisted to module level and both evaluators hand it to eval_agg as eval_at. Note the repair that was not taken — teaching the fold to hit the cache by stripping none_cats off the member key returns the group's value to every member, which is aggregate-then-route through a cache lookup, silent. Fault injected by restoring eval_expr_cached as the fold's evaluator: this test fails with Error("ref:Margin") and the other eleven pass.

the_cached_evaluator_has_its_own_working_coordinate_predicate

The same review's second finding, and a lesson about where to inject a fault. Expr::Is has an arm in each evaluator. The original evidence injected the fault into the shared eval_is, which both arms call — proving the tests see the predicate, not that they see the cached copy. Replacing only the cached arm with Ok(0.0) left the suite at 1330 passing.

The first attempt to close it — a coordinate split run through recompute_formulas + evaluate_aggregated — also stayed green, and that is the interesting part: once the fold routes through eval_expr, an Is inside a SUM never reaches the cached arm at all. The cached arm is now reachable only by a bare Is outside an aggregate, which is what this test uses. Fault injected into that arm alone: it fails, Number(0.0) against Number(1.0).

The general shape: an injected fault proves coverage only of the code path it is injected into. Two arms of the same rule need two injections, and if one of them turns out to be unreachable, that is a fact about the design worth writing down.

a_user_category_whose_name_leads_with_an_underscore_is_seated_as_regular

improvise-bidq. View::on_category_added's early return read cat_name.starts_with('_') || self.is_records_layout() — the same expression mixing two derivations one operator apart, the left a name prefix and the right CategoryKind. Four lines up, is_records_layout asks the enum.

Model::add_category does not validate names, so a user category literally named _Foo was CategoryKind::Regular to records_columns, cell_id and is_empty_model, and "virtual" to these two lines — seated on Axis::None, so it never appeared in a pivot while every other consumer treated it as ordinary. The second site (holds_a_regular) had the same split, which is why the test asserts a second category's seat too: fixing only the first leaves _Foo on Row while Region also takes Row.

Decided explicitly, per the ticket: add_category is not made to refuse a leading underscore. The three reserved names are the ones CategoryKind::of_name knows, _Foo is a legal document, and a user-visible restriction is a larger change than making one answer out of two. The fix is that both sites now ask CategoryKind::of_name(..).is_virtual().

Fault injected by restoring either starts_with('_'): the test fails, None against Row on the first site and Row against Column on the second.

an_ungrouped_item_after_a_grouped_run_belongs_to_no_group

a_grouped_run_does_not_leak_into_the_next_prefix

improvise-gd83, the two arms of one defect. expand_category emitted a GroupHeader on entering a group and no boundary on leaving one, so row_group_for/col_group_for's nearest-preceding walk answered with a group that had already ended.

The arms are separate because the two escapes are different and a fix can close one and miss the other. The first is within one expansion — Jan[Q1], Feb[Q1], Sep — closed by emitting AxisEntry::GroupEnd on a Some -> None transition. The second is across expansions: cross_product expands the inner category once per prefix, each call starting with last_group = None, so Food/Bob[Family] is immediately followed by Clothing/Carol, which is ungrouped and inherited Family with no transition anywhere to hook onto. That one needs a GroupEnd at the end of an expansion whose last item was grouped.

GroupEnd is deliberately NOT emitted on Some(A) -> Some(B): B's own header already ends A's run, and emitting one there moved data_row_to_visual_skips_headers and data_col_to_visual_skips_headers for no gain. Those two are untouched, which is the evidence the boundary is as narrow as it claims.

Fault injected by deleting either emission: the corresponding test fails with Some(("Month", "Q1")) / Some(("Recipient", "Family")) against None, and the_two_front_ends_draw_the_same_column_header fails with it.