improvise/context/code/delivery.md

Delivery: the reasoning

ADR 009 step 8 (improvise-z2vr): notices per committed command, watermarks, corrections, the shared bounded log, the lossless wire and the edge adapters. The decisions are the ADR's clause 9 (context/adr/009-incremental-evaluation.md). This file records how the code reads them, the places the ADR left a choice open, and the places it could not be satisfied yet. The evidence for each test is in context/test-evidence/delivery.md.

What is built, and against what. Neither the incremental backend nor ADR 008's render model exists. So the source of every target is the existing Eager path: the cells a GridLayout draws for a view. Everything below is correct against that source and does not depend on what replaces it. The circuit will supply "moved output keys" where Delivery::commit diffs two Outputs today, and nothing else changes.

piece	where
wire values, keys, Z-set elements	improvise-protocol/src/delivery/wire.rs
stamp, range, notice, merge	.../notice.rs
request, correction, Edge	.../exchange.rs
the client end: watermark, held cells, when to ask	.../client.rs
the publisher, the log, live shapes, answering	improvise-command/src/ui/delivery.rs
Host integration, in-process and JSON edges	ui/host.rs, ui/delivery.rs

1. The wire form

A number crosses as ADR 005's canonical text through numtower::Num: #i0.1, #i-0.0, inf, -inf, nan. WireNum accepts only text the tower itself writes (parse re-prints and compares), because a second spelling of one value would make textual equality wrong. Equality of wire values is textual, so -0.0 and +0.0 are different wire values. That is the opposite of the cell delta in clause 3, which makes them equal; the two do not meet, because the delta is a store-side signal and a correction is a value the client displays.

An integer is a JSON number below 2^53 and a decimal string at or above it (wide, wide_weight). That covers stamps, positions, window sizes and weights. Text is a JSON string, byte-exact.

CellValue is still an f64, so WireValue::of writes it through Num::inexact and to_cell_value narrows an exact number to f64. The round-trip properties therefore run over 28-digit decimals and i128-wide fractions at the WireNum level, where they are lossless, and over f64 at the CellValue level, where only the NaN payload is dropped.

CellKey is not Serialize, so a cell is named by WireKey, its (category, item) tuple. Under ADR 007 those become ids; nothing here reads the strings.

2. Stamps and notices

Stamp is (host instance, document_generation, publication). A stamp from another host or generation is behind every current one (Stamp::is_behind). The publication number increments once per committed command, and it keeps counting across a document swap rather than restarting, so a number is never reused within one host instance.

What "a committed command" is. The host publishes one notice for each Command::Model that lands. A view command (cursor, mode, a view switch) publishes nothing: it is view-store state and the model store has no business announcing it. A write that changes nothing still publishes, with empty moved, because the contract is one notice per commit and a client counts them.

A notice names moved keys per live shape, plus axis changes, relabels, flags, evicted shapes. Axis changes are computed from the categories' item lists before and after (axis_changes): insert and remove by item, and a reorder when the surviving items changed order. Relabel exists and is never emitted: with names as identity a rename is a remove and an insert, and display writes arrive with ADR 007. Flags come from comparing the formula texts, measure_agg, the views and the default view.

Notice::merge joins two notices only when the second starts where the first ends (same epoch). Anything else returns both unchanged, so a merge cannot hide a loss. Moved::Correct absorbs key lists in a merge.

3. Requests and corrections

A request is a shape key, a window and one of three asks. Ask::Correction is "bring me up from my watermark"; Ask::Fetch is "the current values of these keys, covering the publications up to through"; Ask::Notices is "the notices after this stamp".

A correction's elements are target, not target - held. The host does not know what the client holds, so it sends the target restricted as the body says, and the client computes target - held with ordinary Z-set subtraction. A Moved body restricts both sides to the named keys, so a key the target no longer holds is removed. A Window body replaces everything held.

through is not at. at is the state the values were read at, always the latest committed one. through is how far the correction advances the client's watermark. They differ for a Fetch: the client named the keys of the notices it has seen, and a later commit may have moved other keys whose notice it has not seen yet. Advancing to at would make that notice look already applied, so the watermark advances only to through and the later notice refetches its keys. The values are at least as new as the watermark claims, never older.

An axis change makes the correction a window refetch. A moved key list says which values changed, not which keys sit in the client's window, and an insert or remove shifts cells into and out of it without moving them. The ADR has the client re-derive positions from the axis changes and fetch what it lacks. That needs ADR 008's render model, which this layer does not have. So the client records a notice that moved an axis or set flags.views (restructured), asks Fetch { window: true }, and the host answers with a Window body whose carried notice holds the axis changes, in order. This is a fetch, not a correction, so the required test "a frame [AddItem, SetCell] ... and no client asks for a correction" holds: it asserts no Ask::Correction was sent. When the render model exists the client can narrow the fetch to the cells that entered its window (improvise-bw4p.9).

A view command (SetAxis, TransposeAxes, a page selection, a hide) moves positions and no key, and reaches the client only as flags.views. So that flag counts as an axis change on both sides: the client fetches its window, and the host answers a correction over a range holding the flag with a window body.

4. The client

EdgeClient holds, per shape: the window, the held cells (a ZSet), the watermark (applied through), seen (the last notice received), the notices received and not yet fetched, restructured and stale.

on_notice ignores a duplicate (to <= seen), treats a gap, another host or generation, a shape in the evicted list, or Moved::Correct as "ask for a correction", and otherwise queues the moved keys. The watermark advances over a notice only when nothing is owed, so it never claims a key it has not fetched.

apply takes a correction. It is a duplicate if the watermark already covers through; a Moved body is a gap unless it starts at or below the watermark. Otherwise held += target - held, the watermark becomes through, and the queued notices it covers are dropped. A late reply older than the watermark therefore cannot regress the state, which is what makes a retry safe.

A correction from a generation or host the client has left is a duplicate, whatever the watermark. A shape records the epochs its watermark has moved out of (left, the newest 64), and a correction from one of them, or from an older generation of the host it is on, changes nothing. Epochs of different hosts are not ordered, so a client that has never applied anything from the newer host cannot tell a late reply from a dead one from a legitimate first answer, and applies it; once it has moved on, the dead host's replies are refused. A duplicate answer also prunes every queued entry from another epoch and moves seen onto the live epoch, so a late notice from a dead host costs one extra round and cannot livelock sync.

The host that answers a request is live, whatever the client believed. left is a guess, because epochs of different hosts are unordered: a late reply from a host the client never joined moves it onto that host, and writes off the live one. Refusing the live host's answers as "left" then strands the client (sync returned Exhausted, found by the property test). So apply_answer, which sync and poll use for the reply to a request they just sent, takes an epoch off left before it applies. apply is for a reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round; the next request it makes is answered by the live host and brings it back.

A notice from another epoch only marks the shape stale; seen moves only through an answer. The first two livelocks (improvise-z2vr) were the same defect reached by different routes, which is why ignoring one kind of foreign notice after another was not enough. Epochs of different hosts are unordered, so a notice from an epoch other than seen's cannot be placed. Reading it used to rewind seen into the foreign epoch, and then the live host's own old notices looked new, queued work and set restructured; the answers that followed were covered by the watermark, were duplicates, and never cleared it. Now such a notice sets stale and nothing else, the next request is answered by the live host, and that answer (apply_answer) is what moves seen. Two cheaper guards stay: a notice from an older generation of the host the shape is on, or from an epoch it has left, is ignored without the extra round; and a duplicate answer clears a restructured mark it covers. The last is defensive: no test reaches it now that seen cannot be rewound, and it is kept because a covered answer that cannot clear a mark it covers is the shape of both livelocks. Merging notices cut for different shapes, or a cut one with an uncut one, is refused. A notice the client asked for (poll) skips the left-epoch test, since the host that answers is live.

poll hands a pulled notice only to the shape it asked about, and the notice says so: for_shape sets Notice::cut_for, and accept limits any notice that carries it to that shape, however it reaches the client (on_notice included). Without the mark, a pulled notice delivered as if pushed made the other shape skip keys it never saw.

connection_lost, request_failed and unreadable set stale, which wants turns into Ask::Correction from the watermark the client held. A duplicate whose through equals the watermark also clears stale: the host has said there is nothing newer.

A lost final notice is invisible to a listener. The next notice shows the gap, but if none comes the client never learns. poll is the answer: it asks Ask::Notices { after: seen }, which the host answers from the log. The property test ends every history with a poll for this reason.

Events (Axis, Relabel, Flags, Resync) are what the shape's consumer applies to its structure. They are deduplicated by events_through, because an axis change is not idempotent. Resync means positions are unknown: re-derive them from what is held.

5. The Eager stand-in for a shape key

The real key is the circuit's: axes, None categories, filters, pages, measure_agg, in canonical id form. Here a shape key is eager:view/<name> (shape_of_view) and the output is that view's layout. That is finer than the real key, which is always safe: two views that would share state do not, and nothing else is lost. A view that no longer exists is announced as evicted.

Output is the shape's whole output, keyed by WireKey with the position attached. target restricts it to a window. moved_since compares two outputs by value and presence; a changed position is an axis change, not a key.

This costs O(grid) per live shape per commit. It stands in for the circuit's delta, and it is the reason the module exists beside, not instead of, the incremental work: nothing else in the host reads the whole grid per command.

6. The host

Delivery is one value per document, held by Host. Host::handle calls Delivery::commit after each Command::Model that lands, and Host::deliver answers a Request.

Reads see only the last committed state. answer reads Live::output, which commit refreshed, and never the workbook. A value a failed command wrote is not in it.

Rollback. Host::mutating_frame clones the Delivery beside the app snapshot and the session mirrors, and undo_frame puts it back. The clone is cheap: Arc for outputs, notices and structure, and both the log and the outbox hold at most capacity notices. So a refused frame consumes no publication number and publishes nothing.

The ADR and ADR 001 disagree here, and ADR 001 wins until ADR 006 exists. Clause 9 says a frame whose later command fails keeps its earlier commands and their notices, "each command commits on its own (ADR 006)". Today's host undoes the whole frame (ADR 001 clause 4, Host::mutating_frame), and ADR 006 is not in the tree. The required test "a refused command consumes no publication number, and the commands before it in its frame stay published" is split in two. The first half is tested. The second is tested as its opposite, commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001, which is the test to flip when per-command commit lands (improvise-bw4p.8): the publisher already supports it, since it publishes whatever the host tells it committed.

Nothing is published while nobody can hear it. With no live shape and no attached transport (Host::attach_notices), a commit only advances the publication number: no structure is read, nothing is diffed, the log is cleared and the queued structure is forgotten. A shape that goes live, or a transport that attaches, re-reads the structure, so its first diff is against a known state. Without this every host pays for delivery on every command while nothing consumes it. The outbox exists only while a transport is attached, and keeps the newest capacity notices; a transport that falls behind sees a gap and asks for a correction. The axis diff is linear in the items (diff_items, set lookups), and its test counts comparisons on a counting item type rather than reading a clock.

The host instance is a random id, never a time (Host::new takes one from RandomState). RandomState has no entropy on wasm32-unknown-unknown, so a worker must pass a real one through Host::with_instance. A restarted host with the same id would accept a stale watermark as current. Worker still calls Host::new (improvise-bw4p.7). Such a restart, with the same instance id and generation 0, is not recognised as a restart: the client's older guard refuses its replies as coming from an older generation of the host it is on (improvise-bw4p.7).

Eviction (evict_idle) takes a local count and an idle length in the same unit, never a wall clock compared across machines. An evicted key is announced in the next notice and forgotten; naming it again revives it with since set to the current publication, so a watermark older than that is answered with a window.

7. Answering

covered(shape, watermark) is true when the watermark is in the current epoch, is not ahead of the host, and either equals the current publication or is no older than both the log's start and the time the shape went live. Only then are the moved keys of (watermark, now] complete.

• covered, only moved keys: a Moved body with those keys' current values in the window;
• covered, with axis changes or relabels in range: a Window body with carried;
• a Moved::Correct in range, or not covered: a Window body from None, which is target - empty.

A notice lists no keys for a shape whose list passed MAX_KEYS; it says Moved::Correct instead, and costs every reader a correction. LOG_CAPACITY bounds the single shared log. Both are set per host by limit_delivery so a test can reach them.

The token in a request is carried and echoed by the client and never read by the host.

8. Edge adapters

Transport contract. A reply must reach the call that made the request. Request and Correction carry no id, so the client cannot tell a late reply from an answer on its own; sync and poll treat whatever Edge::call returns as the answer (apply_answer), and only a reply handed to apply is treated as late. A transport that reorders or duplicates replies must therefore correlate them itself or drop the strays (improvise-bw4p.10 adds the id to the frames).

Edge is one end of a stream: call(Request) -> Result<Reply, EdgeError>. InProcess is the function call; WireEdge passes request and reply through JSON text, the adapter at a wire boundary, so the lossless properties are exercised by every test that uses it. Neither adds a ClientFrame or ServerFrame variant: the page does not read notices yet, and adding the frames means changing the browser and the worker together (ADR 008 replaces Projection; improvise-bw4p.10). A transport pushes Host::take_notices().

Not done, and why

• The notices are not sent by the socket or worker hosts. They need frame variants and the page (above).
• Positions and column totals are not on the wire. A correction carries (key, value) and no row or column, and Output holds body cells only. The ADR attaches the position when output is sent and lists totals among the moved keys. Both need the render model and the totals' own keys (ADR 008, clause 10).
• A twin at the serving end (a server-side watermark or filter) is not built; the ADR allows it and does not require it.
• A pushed filter is the client's window, applied on every request.
• Hidden-item overlays, pruning queries and column widths are ADR 008 and clause 10.

The same edge, generalised in zset. The zset crate's edge module, crates/zset/src/edge.rs (context/code/zset.md, "Edges") carries Z-set deltas between two circuits with the rules of this file (stamps, coalescing by summing, target - held, token-matched answers) and no log; this layer is unchanged and does not use it.