improvise/context/test-evidence/delivery.md

Test evidence: delivery

What each delivery test guards and which injected fault it was seen to fail on. Keyed by test name. The reasoning is in context/code/delivery.md. Each fault was put into the source one at a time, the suite run, the fault reverted with git checkout -- <file>, and the suite run again.

The host-side tests are in crates/improvise-command/src/ui/delivery_tests.rs, the client's in crates/improvise-protocol/src/delivery/client_tests.rs, the wire's in delivery/wire.rs.

The injected faults, by what failed

• The host reports no moved keys (let keys = BTreeSet::new() in Delivery::commit): seven host-side tests fail: a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes, a_pull_only_client_follows_the_log_without_a_correction, a_notice_names_the_moved_cells_including_a_formula_cell_it_did_not_write, values_keep_every_bit_from_the_cell_to_the_client, an_oversized_notice_says_correct_and_costs_every_reader_a_correction, a_dropped_notice_is_detected_and_a_correction_repairs_it and the random property. a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction still passes, on purpose: its AddItem makes the client refetch the window, which also fetches the write.
• The client treats every notice as contiguous (false => true in on_notice): a_dropped_notice_is_detected, a_dropped_notice_is_detected_and_a_correction_repairs_it and the random property.
• A failed frame does not put the Delivery back (undo_frame drops published): a_refused_command_consumes_no_publication_number, commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001 and a_refused_frame_is_never_read_back.
• A moved correction replaces instead of subtracting held (target for target - held_there): a_moved_correction_applies_only_at_the_watermark_it_starts_from and a_moved_key_the_target_no_longer_holds_is_removed. The same for a window correction (target for target - held): a_correction_equals_the_target_and_a_second_one_is_the_zero_zset, a_correction_replaces_what_is_held_not_adds_to_it and a_correction_already_covered_is_idempotent_and_never_regresses.
• A wire number loses the sign of zero (Num::inexact(x + 0.0) in WireNum::of_f64): a_float_crosses_in_canonical_text_with_the_sign_of_zero and a_float_keeps_every_bit_but_the_nan_payload.
• covered ignores the log's start and the shape's liveness: a_watermark_the_log_covers_gets_the_moved_keys_and_one_it_does_not_gets_the_window and an_evicted_shape_is_announced_and_its_reader_corrects.
• The client does not remember an axis change (restructured never set): an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window. The host's own restructured test alone fails only the random property, since the client's window flag also forces a window.
• Stamp::same_epoch ignores the host instance: a_notice_from_another_host_or_generation_asks_for_a_correction, a_stamp_from_another_epoch_is_behind_every_current_one, a_reconnect_after_a_host_restart_asks_for_a_correction_from_the_old_watermark and a_notice_from_a_restarted_host_asks_for_a_correction_too.

Entries

a_refused_frame_is_never_read_back

The first fixture filled the document to the category limit and then refused the frame at its basis check, so the write never ran and the test passed with the Delivery not restored. A fixture that fails in a wider shape than the fault: the frame never reached mutating_frame. It now writes North, then loads a missing document, and asserts Rejection::MutationRefused so the write is known to have run. The rival above fails it with 77.0 held.

a_corrected_state_is_the_target_and_correcting_it_again_is_the_zero_zset

The random property: histories of up to twelve single-command frames (writes of 0.0, -0.0, both infinities, NaN and 0.1 + 0.2, item adds and removes, a formula), a random window, notices delivered or dropped, connections lost, requests dropped, replies dropped and garbled. It ends with a poll, then asserts the client holds exactly what the grid draws in its window, and that target - held is the zero Z-set. The oracle (drawn) reads GridLayout itself and shares nothing with Output.

Without the closing poll a lost last notice is not detectable by a listener, and the property reported it as a stale client (minimal case: one Remove, notice dropped, no later notice).

The first version of the property also failed on a window of one cell after a remove, and that was a real gap in the correction, not in the test: the remove moved South into the window without moving its value. It is the reason for the restructured rule (delivery.md §3).

a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction

Step 8's first required test. The frame's first command carries one basis stating both records (Host's frame rule: the first mutation's basis stands for the frame), which the test helper builds.

commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001

Step 8 requires that the commands before a refused one stay published. Today's host undoes the whole frame (ADR 001 clause 4), so this test asserts the opposite and names the reason. Flip it when ADR 006 makes each command commit on its own.

a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes

The three transport faults (request dropped, reply dropped, reply unreadable) each leave the client asking for Ask::Correction from the watermark it held, and the next request carries the same session token and that watermark.

an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window

A one-cell window, a remove that moves South into it. The notice lists only North, so a client that fetches moved keys alone ends with an empty window.

Review of #348, round one

Each test below was written against the behaviour the review reported and failed before its fix, or, for the cost tests, was run against a quadratic rival.

diffing_a_category_compares_each_item_a_bounded_number_of_times

Items are a type whose eq is counted. Rival: contains on the slices in diff_items, which makes the count quadratic and fails every size. A wall-clock test would have passed on a fast machine.

a_commit_with_nothing_live_and_no_transport_reads_nothing

Counts Structure::of calls. Rival: the idle skip removed.

the_outbox_and_the_clone_a_frame_takes_are_bounded

Rival: the outbox truncation removed, so twenty commits leave twenty queued.

a_late_window_reply_from_an_older_generation_does_not_regress

Rival: the older-generation guard disabled (fails this test in the protocol crate).

a_late_notice_from_a_dead_host_settles_against_the_live_one

Rival: the duplicate arm no longer prunes other epochs' entries; the random property also fails.

a_poll_for_one_shape_leaves_the_other_shapes_alone

Rival: pulled notices applied to every shape.

a_transpose_empties_a_partial_window_of_the_cell_that_left_it

A transpose moves no key. Rival: flags.views not counted as a structural change in the client and the host. a_dropped_views_notice_is_answered_with_the_window_not_an_empty_key_list fails with the host alone changed.

wide_integers_are_canonical_on_input

Both spellings are refused outside their range: a JSON number at or above 2^53, a string below it, a sign, leading zeros and trailing space.

an_item_insert_position_crosses_as_a_wide_integer

ItemInserted.at is a u128 through wide.

a_fetch_covers_only_the_notices_it_named_when_a_later_commit_is_already_in_the_state

Rival: a fetch answered with through: at. The first version of the random property did not catch it (it delivered each notice before the next commit); the property now splits, delays, duplicates and replays notices and old replies, and fails the rival in most runs (17 of 20 at 256 cases, measured on the author's machine); this test fails it on every run, by name.

equal_values_compare_equal_whatever_the_zero_counts_say

ExactSum's equality compared the sign-of-zero counters even when the total was nonzero.

Review of #348, round two

a_late_window_reply_from_a_dead_host_does_not_regress

A client that has applied a correction from host 2 receives a late Window reply from host 1. Epochs of different hosts are unordered, so the first fix (older generation of the same host) did not cover it. Rival: the left check removed; this test and the two below fail.

a_reply_from_a_generation_the_client_left_does_not_return

Generations 1, 2, 3 of one host; a late generation-2 reply. Rival: both guards removed.

a_reply_held_across_a_host_restart_cannot_pull_the_client_back

The host-side form: the reply is computed and lost, the host restarts, the client settles against the new host, and the old reply then arrives. Rival: the left check removed.

The random property, round two. The property now checks after every step that the watermark's epoch never goes back to one it had left (so a mid-history regression cannot be repaired by the closing poll), and replays an earlier reply after every restart and swap. Rival: the left check removed fails it. The older-generation guard on its own is not separable from left in the histories the property reaches, so it is held by a_late_window_reply_from_an_older_generation_does_not_regress alone.

The left-epoch livelock

a_late_reply_from_a_host_the_client_never_joined_cannot_strand_it

The property test failed with Exhausted on a shrunk history of two restarts and four writes. Cause: the client was on host 3, never having joined host 2; a late host-2 reply was applied (an unknown epoch is accepted), which wrote host 3 into left, and every later answer from the live host 3 was then refused as a duplicate. The test replays that history by hand: three hosts, the host-2 reply lost in flight, the client settling on host 3, the late reply applied, the connection lost, and a sync that must settle on host 3's grid. Rival: apply_answer treated like apply (the left.retain removed) fails this test. The random property does not catch it: it passed 11 of 11 runs under that fault, and again after the late reply was made an explicit event (a stashed reply from a host the client never joined, replayed before a lost connection), because the property rarely produces a reply from a host the client never joined. This test is the pin. Rival: the left-epoch guard removed fails a_reply_held_across_a_host_restart_cannot_pull_the_client_back and the property, as before.

The property's invariant changed with it. "The watermark never returns to an epoch it left" was false by design once an unknown-epoch reply can move the client, so the property now checks the narrower, true rule: a replayed reply from an epoch the client was in and has left must be a duplicate.

The old-notice livelock

a_late_notice_from_an_older_generation_cannot_rewind_the_client

The property failed with Exhausted on a shrunk history (two swaps, a transpose, a limit change, the first swap's notice delivered three steps late, an old reply replayed). The client had settled on generation 2 when the generation-1 notice arrived: seen was rewound into generation 1, a notice from the same history set restructured, and the window correction that answered was covered by the watermark, so it never cleared the mark and poll retried until Exhausted. The test replays the history step for step, with the property's own weather and a Flaky edge. Rival at the time: the notice-ignore test in accept disabled; this test and the random property failed. Since the round-three change (a foreign notice only marks the shape stale) nothing fails with that shortcut disabled: it is an optimisation that saves one round, not a correctness rule. The test still replays the seed. The seed file is not committed (proptest-regressions/ is ignored).

Run after the fix, the property passed 4 x 4000 cases and 2 x 20000 cases, with no third failure.

Round three: the same livelock by other routes

route_a_a_dead_hosts_notice_cannot_make_the_live_hosts_old_notices_look_new

A client goes host 1 to host 3 and never joins host 2. A late host-2 notice rewinds seen, then host 3's own old notices look new and set restructured at the watermark, and the answers are duplicates that never clear it. Failed on the head before the change (wants stayed Fetch { window: true }).

route_b_a_covered_reply_cannot_set_seen_below_a_queued_entry

The same rewind, then an unsolicited covered correction: the duplicate branch set seen to its through, below a queued entry, and each fetch for that through came back a duplicate. Failed on the head before the change.

a_foreign_notice_marks_the_shape_stale_and_the_home_hosts_answer_settles_it

A foreign notice sets stale, and the home host's answer then settles the shape. It does not see seen stay put (an answer heals it either way); route A and route B are what pin that.

notices_cut_for_different_shapes_or_not_cut_do_not_merge

A merge of notices cut for different shapes, or a cut one with an uncut one, would keep one shape's mark and make the other shape see a gap. merge refuses it. Rival: the cut_for comparison removed.

a_pulled_notice_pushed_through_on_notice_still_reaches_only_its_shape

A notice cut for shape a by for_shape, delivered through on_notice, must leave shape b alone. Failed before Notice::cut_for existed.

two_shapes_survive_late_repeated_and_dead_messages

The reviewer's generator, kept: two shapes, notices late, repeated and reordered (the due set is reversed on odd steps), dead hosts' notices kept across restarts, stored notices and stored replies replayed on their own, hosts that restart often. At the default 256 cases it did not find the old head's livelock, and at 20,000 it did within ten seconds (an Exhausted), so the default run is a smoke test and the deep run is the check. After the change it passed three runs of 20,000 cases, and the older property passed 2 x 20,000 with it.