From eb15df6c368523251fa86729cfd21b8e95b186cb Mon Sep 17 00:00:00 2001 From: "Claude (dbsp-delivery)" Date: Sat, 10 Oct 2026 12:11:48 -0700 Subject: [PATCH] docs(context): the reasoning and the evidence for delivery (improvise-z2vr) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/delivery.md | 198 ++++++++++++++++++++++++++++++ context/test-evidence/delivery.md | 95 ++++++++++++++ scripts/build-doc-site.mjs | 2 + 3 files changed, 295 insertions(+) create mode 100644 context/code/delivery.md create mode 100644 context/test-evidence/delivery.md diff --git a/context/code/delivery.md b/context/code/delivery.md new file mode 100644 index 00000000..2ee9781c --- /dev/null +++ b/context/code/delivery.md @@ -0,0 +1,198 @@ +# Delivery: the reasoning + +ADR 009 step 8 (improvise-z2vr): notices per committed command, watermarks, corrections, the shared +bounded log, the lossless wire and the edge adapters. The decisions are the ADR's clause 9 +(`context/adr/009-incremental-evaluation.md`). This file records how the code reads them, the places +the ADR left a choice open, and the places it could not be satisfied yet. The evidence for each test is +in `context/test-evidence/delivery.md`. + +**What is built, and against what.** Neither the incremental backend nor ADR 008's render model +exists. So the source of every target is the existing Eager path: the cells a `GridLayout` draws for a +view. Everything below is correct against that source and does not depend on what replaces it. The +circuit will supply "moved output keys" where `Delivery::commit` diffs two `Output`s today, and +nothing else changes. + +| piece | where | +|---|---| +| wire values, keys, Z-set elements | `improvise-protocol/src/delivery/wire.rs` | +| stamp, range, notice, merge | `.../notice.rs` | +| request, correction, `Edge` | `.../exchange.rs` | +| the client end: watermark, held cells, when to ask | `.../client.rs` | +| the publisher, the log, live shapes, answering | `improvise-command/src/ui/delivery.rs` | +| `Host` integration, in-process and JSON edges | `ui/host.rs`, `ui/delivery.rs` | + +## 1. The wire form + +A number crosses as ADR 005's canonical text through `numtower::Num`: `#i0.1`, `#i-0.0`, `inf`, +`-inf`, `nan`. `WireNum` accepts only text the tower itself writes (`parse` re-prints and compares), +because a second spelling of one value would make textual equality wrong. Equality of wire values is +textual, so `-0.0` and `+0.0` are different wire values. That is the opposite of the cell delta in +clause 3, which makes them equal; the two do not meet, because the delta is a store-side signal and a +correction is a value the client displays. + +An integer is a JSON number below 2^53 and a decimal string at or above it (`wide`, `wide_weight`). +That covers stamps, positions, window sizes and weights. Text is a JSON string, byte-exact. + +`CellValue` is still an `f64`, so `WireValue::of` writes it through `Num::inexact` and +`to_cell_value` narrows an exact number to `f64`. The round-trip properties therefore run over +28-digit decimals and `i128`-wide fractions at the `WireNum` level, where they are lossless, and over +`f64` at the `CellValue` level, where only the NaN payload is dropped. + +`CellKey` is not `Serialize`, so a cell is named by `WireKey`, its `(category, item)` tuple. Under +ADR 007 those become ids; nothing here reads the strings. + +## 2. Stamps and notices + +`Stamp` is `(host instance, document_generation, publication)`. A stamp from another host or +generation is behind every current one (`Stamp::is_behind`). The publication number increments once +per committed command, and it keeps counting across a document swap rather than restarting, so a +number is never reused within one host instance. + +**What "a committed command" is.** The host publishes one notice for each `Command::Model` that +lands. A view command (cursor, mode, a view switch) publishes nothing: it is view-store state and +the model store has no business announcing it. A write that changes nothing still publishes, with +empty `moved`, because the contract is one notice per commit and a client counts them. + +**A notice names moved keys per live shape**, plus axis changes, relabels, flags, evicted shapes. Axis +changes are computed from the categories' item lists before and after (`axis_changes`): insert and +remove by item, and a reorder when the surviving items changed order. `Relabel` exists and is never +emitted: with names as identity a rename is a remove and an insert, and display writes arrive with +ADR 007. Flags come from comparing the formula texts, `measure_agg`, the views and the default view. + +`Notice::merge` joins two notices only when the second starts where the first ends (same epoch). +Anything else returns both unchanged, so a merge cannot hide a loss. `Moved::Correct` absorbs key +lists in a merge. + +## 3. Requests and corrections + +A request is a shape key, a window and one of three asks. `Ask::Correction` is "bring me up from my +watermark"; `Ask::Fetch` is "the current values of these keys, covering the publications up to +`through`"; `Ask::Notices` is "the notices after this stamp". + +**A correction's elements are `target`, not `target - held`.** The host does not know what the client +holds, so it sends the target restricted as the body says, and the client computes `target - held` +with ordinary Z-set subtraction. A `Moved` body restricts both sides to the named keys, so a key the +target no longer holds is removed. A `Window` body replaces everything held. + +**`through` is not `at`.** `at` is the state the values were read at, always the latest committed one. +`through` is how far the correction advances the client's watermark. They differ for a `Fetch`: +the client named the keys of the notices it has seen, and a later commit may have moved other keys +whose notice it has not seen yet. Advancing to `at` would make that notice look already applied, so +the watermark advances only to `through` and the later notice refetches its keys. The values are +at least as new as the watermark claims, never older. + +**An axis change makes the correction a window refetch.** A moved key list says which values changed, +not which keys sit in the client's window, and an insert or remove shifts cells into and out of it +without moving them. The ADR has the client re-derive positions from the axis changes and fetch what +it lacks. That needs ADR 008's render model, which this layer does not have. So the client records a +notice that moved an axis (`restructured`), asks `Fetch { window: true }`, and the host answers with a +`Window` body whose `carried` notice holds the axis changes, in order. This is a fetch, not a +correction, so the required test "a frame `[AddItem, SetCell]` ... and no client asks for a correction" +holds: it asserts no `Ask::Correction` was sent. When the render model exists the client can narrow +the fetch to the cells that entered its window. + +## 4. The client + +`EdgeClient` holds, per shape: the window, the held cells (a `ZSet`), the **watermark** (applied +through), `seen` (the last notice received), the notices received and not yet fetched, `restructured` +and `stale`. + +`on_notice` ignores a duplicate (`to <= seen`), treats a gap, another host or generation, a shape in the +evicted list, or `Moved::Correct` as "ask for a correction", and otherwise queues the moved keys. The +watermark advances over a notice only when nothing is owed, so it never claims a key it has not fetched. + +`apply` takes a correction. It is a duplicate if the watermark already covers `through`; a `Moved` +body is a gap unless it starts at or below the watermark. Otherwise `held += target - held`, the +watermark becomes `through`, and the queued notices it covers are dropped. A late reply older than +the watermark therefore cannot regress the state, which is what makes a retry safe. + +`connection_lost`, `request_failed` and `unreadable` set `stale`, which `wants` turns into +`Ask::Correction` from the watermark the client held. A duplicate whose `through` equals the watermark +also clears `stale`: the host has said there is nothing newer. + +**A lost final notice is invisible to a listener.** The next notice shows the gap, but if none comes the +client never learns. `poll` is the answer: it asks `Ask::Notices { after: seen }`, which the host +answers from the log. The property test ends every history with a `poll` for this reason. + +`Event`s (`Axis`, `Relabel`, `Flags`, `Resync`) are what the shape's consumer applies to its +structure. They are deduplicated by `events_through`, because an axis change is not idempotent. +`Resync` means positions are unknown: re-derive them from what is held. + +## 5. The Eager stand-in for a shape key + +The real key is the circuit's: axes, None categories, filters, pages, `measure_agg`, in canonical id +form. Here a shape key is `eager:view/` (`shape_of_view`) and the output is that view's layout. +That is finer than the real key, which is always safe: two views that would share state do not, and +nothing else is lost. A view that no longer exists is announced as evicted. + +`Output` is the shape's whole output, keyed by `WireKey` with the position attached. `target` restricts +it to a window. `moved_since` compares two outputs by value and presence; a changed position is an axis +change, not a key. + +**This costs O(grid) per live shape per commit.** It stands in for the circuit's delta, and it is the +reason the module exists beside, not instead of, the incremental work: nothing else in the host reads +the whole grid per command. + +## 6. The host + +`Delivery` is one value per document, held by `Host`. `Host::handle` calls `Delivery::commit` after +each `Command::Model` that lands, and `Host::deliver` answers a `Request`. + +**Reads see only the last committed state.** `answer` reads `Live::output`, which `commit` refreshed, +and never the workbook. A value a failed command wrote is not in it. + +**Rollback.** `Host::mutating_frame` clones the `Delivery` beside the app snapshot and the session +mirrors, and `undo_frame` puts it back. The clone is cheap (`Arc` for outputs, notices and structure, +bounded by the log's capacity). So a refused frame consumes no publication number and publishes nothing. + +**The ADR and ADR 001 disagree here, and ADR 001 wins until ADR 006 exists.** Clause 9 says a frame +whose later command fails keeps its earlier commands and their notices, "each command commits on its own +(ADR 006)". Today's host undoes the whole frame (ADR 001 clause 4, `Host::mutating_frame`), and ADR 006 is +not in the tree. The required test "a refused command consumes no publication number, and the commands +before it in its frame stay published" is split in two. The first half is tested. The second is +tested as its opposite, `commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001`, +which is the test to flip when per-command commit lands: the publisher already supports it, since it +publishes whatever the host tells it committed. + +**The host instance** is a random id, never a time (`Host::new` takes one from `RandomState`). +`RandomState` has no entropy on `wasm32-unknown-unknown`, so a worker must pass a real one through +`Host::with_instance`. A restarted host with the same id would accept a stale watermark as current. + +**Eviction** (`evict_idle`) takes a local count and an idle length in the same unit, never a wall clock +compared across machines. An evicted key is announced in the next notice and forgotten; naming it again +revives it with `since` set to the current publication, so a watermark older than that is answered with +a window. + +## 7. Answering + +`covered(shape, watermark)` is true when the watermark is in the current epoch, is not ahead of the +host, and either equals the current publication or is no older than both the log's start and the time +the shape went live. Only then are the moved keys of `(watermark, now]` complete. + +- covered, only moved keys: a `Moved` body with those keys' current values in the window; +- covered, with axis changes or relabels in range: a `Window` body with `carried`; +- a `Moved::Correct` in range, or not covered: a `Window` body from `None`, which is `target - empty`. + +A notice lists no keys for a shape whose list passed `MAX_KEYS`; it says `Moved::Correct` instead, and +costs every reader a correction. `LOG_CAPACITY` bounds the single shared log. Both are set per host by +`limit_delivery` so a test can reach them. + +The token in a request is carried and echoed by the client and never read by the host. + +## 8. Edge adapters + +`Edge` is one end of a stream: `call(Request) -> Result`. `InProcess` is the +function call; `WireEdge` passes request and reply through JSON text, the adapter at a wire boundary, +so the lossless properties are exercised by every test that uses it. Neither adds a `ClientFrame` or +`ServerFrame` variant: the page does not read notices yet, and adding the frames means changing the +browser and the worker together (ADR 008 replaces `Projection`). A transport pushes +`Host::take_notices()`. + +## Not done, and why + +- **The notices are not sent by the socket or worker hosts.** They need frame variants and the page + (above). +- **A twin at the serving end** (a server-side watermark or filter) is not built; the ADR allows it and + does not require it. +- **A pushed filter** is the client's `window`, applied on every request. +- **Hidden-item overlays, pruning queries and column widths** are ADR 008 and clause 10. diff --git a/context/test-evidence/delivery.md b/context/test-evidence/delivery.md new file mode 100644 index 00000000..c617c311 --- /dev/null +++ b/context/test-evidence/delivery.md @@ -0,0 +1,95 @@ +# Test evidence: delivery + +What each delivery test guards and which injected fault it was seen to fail on. Keyed by test name. +The reasoning is in `context/code/delivery.md`. Each fault was put into the source one at a time, +the suite run, the fault reverted with `git checkout -- `, and the suite run again. + +The host-side tests are in `crates/improvise-command/src/ui/delivery_tests.rs`, the client's in +`crates/improvise-protocol/src/delivery/client_tests.rs`, the wire's in `delivery/wire.rs`. + +## The injected faults, by what failed + +- **The host reports no moved keys** (`let keys = BTreeSet::new()` in `Delivery::commit`): seven + host-side tests fail: `a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes`, + `a_pull_only_client_follows_the_log_without_a_correction`, + `a_notice_names_the_moved_cells_including_a_formula_cell_it_did_not_write`, + `values_keep_every_bit_from_the_cell_to_the_client`, + `an_oversized_notice_says_correct_and_costs_every_reader_a_correction`, + `a_dropped_notice_is_detected_and_a_correction_repairs_it` and the random property. + `a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction` still + passes, on purpose: its `AddItem` makes the client refetch the window, which also fetches the write. +- **The client treats every notice as contiguous** (`false => true` in `on_notice`): + `a_dropped_notice_is_detected`, `a_dropped_notice_is_detected_and_a_correction_repairs_it` and the + random property. +- **A failed frame does not put the `Delivery` back** (`undo_frame` drops `published`): + `a_refused_command_consumes_no_publication_number`, + `commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001` and + `a_refused_frame_is_never_read_back`. +- **A moved correction replaces instead of subtracting held** (`target` for `target - held_there`): + `a_moved_correction_applies_only_at_the_watermark_it_starts_from` and + `a_moved_key_the_target_no_longer_holds_is_removed`. The same for a window correction (`target` for + `target - held`): `a_correction_equals_the_target_and_a_second_one_is_the_zero_zset`, + `a_correction_replaces_what_is_held_not_adds_to_it` and + `a_correction_already_covered_is_idempotent_and_never_regresses`. +- **A wire number loses the sign of zero** (`Num::inexact(x + 0.0)` in `WireNum::of_f64`): + `a_float_crosses_in_canonical_text_with_the_sign_of_zero` and + `a_float_keeps_every_bit_but_the_nan_payload`. +- **`covered` ignores the log's start and the shape's liveness**: + `a_watermark_the_log_covers_gets_the_moved_keys_and_one_it_does_not_gets_the_window` and + `an_evicted_shape_is_announced_and_its_reader_corrects`. +- **The client does not remember an axis change** (`restructured` never set): + `an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window`. The host's own + `restructured` test alone fails only the random property, since the client's `window` flag also forces + a window. +- **`Stamp::same_epoch` ignores the host instance**: + `a_notice_from_another_host_or_generation_asks_for_a_correction`, + `a_stamp_from_another_epoch_is_behind_every_current_one`, + `a_reconnect_after_a_host_restart_asks_for_a_correction_from_the_old_watermark` and + `a_notice_from_a_restarted_host_asks_for_a_correction_too`. + +## Entries + +### a_refused_frame_is_never_read_back + +The first fixture filled the document to the category limit and then refused the frame at its basis +check, so the write never ran and the test passed with the `Delivery` not restored. A fixture that fails +in a wider shape than the fault: the frame never reached `mutating_frame`. It now writes North, then +loads a missing document, and asserts `Rejection::MutationRefused` so the write is known to have run. The +rival above fails it with `77.0` held. + +### a_corrected_state_is_the_target_and_correcting_it_again_is_the_zero_zset + +The random property: histories of up to twelve single-command frames (writes of `0.0`, `-0.0`, both +infinities, NaN and `0.1 + 0.2`, item adds and removes, a formula), a random window, notices delivered +or dropped, connections lost, requests dropped, replies dropped and garbled. It ends with a `poll`, then +asserts the client holds exactly what the grid draws in its window, and that `target - held` is the zero +Z-set. The oracle (`drawn`) reads `GridLayout` itself and shares nothing with `Output`. + +Without the closing `poll` a lost last notice is not detectable by a listener, and the property +reported it as a stale client (minimal case: one `Remove`, notice dropped, no later notice). + +The first version of the property also failed on a window of one cell after a remove, and that was a +real gap in the correction, not in the test: the remove moved South into the window without moving its +value. It is the reason for the `restructured` rule (`delivery.md` ยง3). + +### a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction + +Step 8's first required test. The frame's first command carries one basis stating both records +(`Host`'s frame rule: the first mutation's basis stands for the frame), which the test helper builds. + +### commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001 + +Step 8 requires that the commands before a refused one stay published. Today's host undoes the whole +frame (ADR 001 clause 4), so this test asserts the opposite and names the reason. Flip it when ADR 006 +makes each command commit on its own. + +### a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes + +The three transport faults (request dropped, reply dropped, reply unreadable) each leave the client +asking for `Ask::Correction` from the watermark it held, and the next request carries the same session +token and that watermark. + +### an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window + +A one-cell window, a remove that moves South into it. The notice lists only North, so a client that +fetches moved keys alone ends with an empty window. diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index 0ccf5cc6..9414b1c3 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -69,6 +69,7 @@ const SECTIONS = [ ["context/code/gp.md", "code/gp", "gp-tree and improvise-gp"], ["context/code/numtower.md", "code/numtower", "numtower, the numeric tower"], ["context/code/zset.md", "code/zset", "zset, the Z-set group"], + ["context/code/delivery.md", "code/delivery", "delivery, notices and corrections"], ["context/code/kb-tool.md", "code/kb-tool", "The kb tool"], ["context/code/tooling.md", "code/tooling", "The repository's own checkers"], ], @@ -89,6 +90,7 @@ const SECTIONS = [ ["context/test-evidence/gp.md", "test-evidence/gp", "gp-tree and improvise-gp"], ["context/test-evidence/numtower.md", "test-evidence/numtower", "numtower"], ["context/test-evidence/zset.md", "test-evidence/zset", "zset"], + ["context/test-evidence/delivery.md", "test-evidence/delivery", "delivery"], ["context/test-evidence/tooling.md", "test-evidence/tooling", "The repository's own gates"], ["context/test-evidence/layout-cost.md", "test-evidence/layout-cost", "Layout and keypress cost"], ["context/test-evidence/scale-bisect.md", "test-evidence/scale-bisect", "Scale bisect"],