improvise/context/code/delivery.md

# Delivery: the reasoning

ADR 009 step 8 (improvise-z2vr): notices per committed command, watermarks, corrections, the shared
bounded log, the lossless wire and the edge adapters. The decisions are the ADR's clause 9
(`context/adr/009-incremental-evaluation.md`). This file records how the code reads them, the places
the ADR left a choice open, and the places it could not be satisfied yet. The evidence for each test is
in `context/test-evidence/delivery.md`.

**What is built, and against what.** Neither the incremental backend nor ADR 008's render model
exists. So the source of every target is the existing Eager path: the cells a `GridLayout` draws for a
view. Everything below is correct against that source and does not depend on what replaces it. The
circuit will supply "moved output keys" where `Delivery::commit` diffs two `Output`s today, and
nothing else changes.

| piece | where |
|---|---|
| wire values, keys, Z-set elements | `improvise-protocol/src/delivery/wire.rs` |
| stamp, range, notice, merge | `.../notice.rs` |
| request, correction, `Edge` | `.../exchange.rs` |
| the client end: watermark, held cells, when to ask | `.../client.rs` |
| the publisher, the log, live shapes, answering | `improvise-command/src/ui/delivery.rs` |
| `Host` integration, in-process and JSON edges | `ui/host.rs`, `ui/delivery.rs` |

## 1. The wire form

A number crosses as ADR 005's canonical text through `numtower::Num`: `#i0.1`, `#i-0.0`, `inf`,
`-inf`, `nan`. `WireNum` accepts only text the tower itself writes (`parse` re-prints and compares),
because a second spelling of one value would make textual equality wrong. Equality of wire values is
textual, so `-0.0` and `+0.0` are different wire values. That is the opposite of the cell delta in
clause 3, which makes them equal; the two do not meet, because the delta is a store-side signal and a
correction is a value the client displays.

An integer is a JSON number below 2^53 and a decimal string at or above it (`wide`, `wide_weight`).
That covers stamps, positions, window sizes and weights. Text is a JSON string, byte-exact.

`CellValue` is still an `f64`, so `WireValue::of` writes it through `Num::inexact` and
`to_cell_value` narrows an exact number to `f64`. The round-trip properties therefore run over
28-digit decimals and `i128`-wide fractions at the `WireNum` level, where they are lossless, and over
`f64` at the `CellValue` level, where only the NaN payload is dropped.

`CellKey` is not `Serialize`, so a cell is named by `WireKey`, its `(category, item)` tuple. Under
ADR 007 those become ids; nothing here reads the strings.

## 2. Stamps and notices

`Stamp` is `(host instance, document_generation, publication)`. A stamp from another host or
generation is behind every current one (`Stamp::is_behind`). The publication number increments once
per committed command, and it keeps counting across a document swap rather than restarting, so a
number is never reused within one host instance.

**What "a committed command" is.** The host publishes one notice for each `Command::Model` that
lands. A view command (cursor, mode, a view switch) publishes nothing: it is view-store state and
the model store has no business announcing it. A write that changes nothing still publishes, with
empty `moved`, because the contract is one notice per commit and a client counts them.

**A notice names moved keys per live shape**, plus axis changes, relabels, flags, evicted shapes. Axis
changes are computed from the categories' item lists before and after (`axis_changes`): insert and
remove by item, and a reorder when the surviving items changed order. `Relabel` exists and is never
emitted: with names as identity a rename is a remove and an insert, and display writes arrive with
ADR 007. Flags come from comparing the formula texts, `measure_agg`, the views and the default view.

`Notice::merge` joins two notices only when the second starts where the first ends (same epoch).
Anything else returns both unchanged, so a merge cannot hide a loss. `Moved::Correct` absorbs key
lists in a merge.

## 3. Requests and corrections

A request is a shape key, a window and one of three asks. `Ask::Correction` is "bring me up from my
watermark"; `Ask::Fetch` is "the current values of these keys, covering the publications up to
`through`"; `Ask::Notices` is "the notices after this stamp".

**A correction's elements are `target`, not `target - held`.** The host does not know what the client
holds, so it sends the target restricted as the body says, and the client computes `target - held`
with ordinary Z-set subtraction. A `Moved` body restricts both sides to the named keys, so a key the
target no longer holds is removed. A `Window` body replaces everything held.

**`through` is not `at`.** `at` is the state the values were read at, always the latest committed one.
`through` is how far the correction advances the client's watermark. They differ for a `Fetch`:
the client named the keys of the notices it has seen, and a later commit may have moved other keys
whose notice it has not seen yet. Advancing to `at` would make that notice look already applied, so
the watermark advances only to `through` and the later notice refetches its keys. The values are
at least as new as the watermark claims, never older.

**An axis change makes the correction a window refetch.** A moved key list says which values changed,
not which keys sit in the client's window, and an insert or remove shifts cells into and out of it
without moving them. The ADR has the client re-derive positions from the axis changes and fetch what
it lacks. That needs ADR 008's render model, which this layer does not have. So the client records a
notice that moved an axis or set `flags.views` (`restructured`), asks `Fetch { window: true }`, and the host answers with a
`Window` body whose `carried` notice holds the axis changes, in order. This is a fetch, not a
correction, so the required test "a frame `[AddItem, SetCell]` ... and no client asks for a correction"
holds: it asserts no `Ask::Correction` was sent. When the render model exists the client can narrow
the fetch to the cells that entered its window (improvise-bw4p.9).

A view command (`SetAxis`, `TransposeAxes`, a page selection, a hide) moves positions and no key, and
reaches the client only as `flags.views`. So that flag counts as an axis change on both sides: the client
fetches its window, and the host answers a correction over a range holding the flag with a window body.

## 4. The client

`EdgeClient` holds, per shape: the window, the held cells (a `ZSet`), the **watermark** (applied
through), `seen` (the last notice received), the notices received and not yet fetched, `restructured`
and `stale`.

`on_notice` ignores a duplicate (`to <= seen`), treats a gap, another host or generation, a shape in the
evicted list, or `Moved::Correct` as "ask for a correction", and otherwise queues the moved keys. The
watermark advances over a notice only when nothing is owed, so it never claims a key it has not fetched.

`apply` takes a correction. It is a duplicate if the watermark already covers `through`; a `Moved`
body is a gap unless it starts at or below the watermark. Otherwise `held += target - held`, the
watermark becomes `through`, and the queued notices it covers are dropped. A late reply older than
the watermark therefore cannot regress the state, which is what makes a retry safe.

**A correction from a generation or host the client has left is a duplicate, whatever the
watermark.** A shape records the epochs its watermark has moved out of (`left`, the newest 64), and a
correction from one of them, or from an older generation of the host it is on, changes nothing. Epochs of
different hosts are not ordered, so a client that has never applied anything from the newer host cannot
tell a late reply from a dead one from a legitimate first answer, and applies it; once it has moved on, the
dead host's replies are refused. A duplicate answer also prunes every queued entry from another epoch
and moves `seen` onto the live epoch, so a late notice from a dead host costs one extra round and cannot
livelock `sync`.

**The host that answers a request is live, whatever the client believed.** `left` is a guess, because
epochs of different hosts are unordered: a late reply from a host the client never joined moves it onto
that host, and writes off the live one. Refusing the live host's answers as "left" then strands the client
(`sync` returned `Exhausted`, found by the property test). So `apply_answer`, which `sync` and `poll` use
for the reply to a request they just sent, takes an epoch off `left` before it applies. `apply` is for a
reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round;
the next request it makes is answered by the live host and brings it back.

**A notice from another epoch only marks the shape stale; `seen` moves only through an answer.** The
first two livelocks (improvise-z2vr) were the same defect reached by different routes, which is why
ignoring one kind of foreign notice after another was not enough. Epochs of different hosts are
unordered, so a notice from an epoch other than `seen`'s cannot be placed. Reading it used to rewind
`seen` into the foreign epoch, and then the live host's own old notices looked new, queued work and set
`restructured`; the answers that followed were covered by the watermark, were duplicates, and never
cleared it. Now such a notice sets `stale` and nothing else, the next request is answered by the live
host, and that answer (`apply_answer`) is what moves `seen`. Two cheaper guards stay: a notice from an
older generation of the host the shape is on, or from an epoch it has left, is ignored without the extra
round; and a duplicate answer clears a `restructured` mark it covers. The last is defensive: no test
reaches it now that `seen` cannot be rewound, and it is kept because a covered answer that cannot clear a mark
it covers is the shape of both livelocks. Merging notices cut for different shapes, or a cut one with an
uncut one, is refused. A notice the client asked for
(`poll`) skips the left-epoch test, since the host that answers is live.

`poll` hands a pulled notice only to the shape it asked about, and the notice says so: `for_shape` sets
`Notice::cut_for`, and `accept` limits any notice that carries it to that shape, however it reaches the
client (`on_notice` included). Without the mark, a pulled notice delivered as if pushed made the other
shape skip keys it never saw.

`connection_lost`, `request_failed` and `unreadable` set `stale`, which `wants` turns into
`Ask::Correction` from the watermark the client held. A duplicate whose `through` equals the watermark
also clears `stale`: the host has said there is nothing newer.

**A lost final notice is invisible to a listener.** The next notice shows the gap, but if none comes the
client never learns. `poll` is the answer: it asks `Ask::Notices { after: seen }`, which the host
answers from the log. The property test ends every history with a `poll` for this reason.

`Event`s (`Axis`, `Relabel`, `Flags`, `Resync`) are what the shape's consumer applies to its
structure. They are deduplicated by `events_through`, because an axis change is not idempotent.
`Resync` means positions are unknown: re-derive them from what is held.

## 5. The Eager stand-in for a shape key

The real key is the circuit's: axes, None categories, filters, pages, `measure_agg`, in canonical id
form. Here a shape key is `eager:view/<name>` (`shape_of_view`) and the output is that view's layout.
That is finer than the real key, which is always safe: two views that would share state do not, and
nothing else is lost. A view that no longer exists is announced as evicted.

`Output` is the shape's whole output, keyed by `WireKey` with the position attached. `target` restricts
it to a window. `moved_since` compares two outputs by value and presence; a changed position is an axis
change, not a key.

**This costs O(grid) per live shape per commit.** It stands in for the circuit's delta, and it is the
reason the module exists beside, not instead of, the incremental work: nothing else in the host reads
the whole grid per command.

## 6. The host

`Delivery` is one value per document, held by `Host`. `Host::handle` calls `Delivery::commit` after
each `Command::Model` that lands, and `Host::deliver` answers a `Request`.

**Reads see only the last committed state.** `answer` reads `Live::output`, which `commit` refreshed,
and never the workbook. A value a failed command wrote is not in it.

**Rollback.** `Host::mutating_frame` clones the `Delivery` beside the app snapshot and the session
mirrors, and `undo_frame` puts it back. The clone is cheap: `Arc` for outputs, notices and structure,
and both the log and the outbox hold at most `capacity` notices. So a refused frame consumes no publication number and publishes nothing.

**The ADR and ADR 001 disagree here, and ADR 001 wins until ADR 006 exists.** Clause 9 says a frame
whose later command fails keeps its earlier commands and their notices, "each command commits on its own
(ADR 006)". Today's host undoes the whole frame (ADR 001 clause 4, `Host::mutating_frame`), and ADR 006 is
not in the tree. The required test "a refused command consumes no publication number, and the commands
before it in its frame stay published" is split in two. The first half is tested. The second is
tested as its opposite, `commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001`,
which is the test to flip when per-command commit lands (improvise-bw4p.8): the publisher already supports it, since it
publishes whatever the host tells it committed.

**Nothing is published while nobody can hear it.** With no live shape and no attached transport
(`Host::attach_notices`), a commit only advances the publication number: no structure is read, nothing is
diffed, the log is cleared and the queued structure is forgotten. A shape that goes live, or a transport
that attaches, re-reads the structure, so its first diff is against a known state. Without this every
host pays for delivery on every command while nothing consumes it. The outbox exists only while a
transport is attached, and keeps the newest `capacity` notices; a transport that falls behind sees a gap
and asks for a correction. The axis diff is linear in the items (`diff_items`, set lookups), and its test
counts comparisons on a counting item type rather than reading a clock.

**The host instance** is a random id, never a time (`Host::new` takes one from `RandomState`).
`RandomState` has no entropy on `wasm32-unknown-unknown`, so a worker must pass a real one through
`Host::with_instance`. A restarted host with the same id would accept a stale watermark as current.
`Worker` still calls `Host::new` (improvise-bw4p.7). Such a restart, with the same instance id and
generation 0, is not recognised as a restart: the client's `older` guard refuses its replies as coming from
an older generation of the host it is on (improvise-bw4p.7).

**Eviction** (`evict_idle`) takes a local count and an idle length in the same unit, never a wall clock
compared across machines. An evicted key is announced in the next notice and forgotten; naming it again
revives it with `since` set to the current publication, so a watermark older than that is answered with
a window.

## 7. Answering

`covered(shape, watermark)` is true when the watermark is in the current epoch, is not ahead of the
host, and either equals the current publication or is no older than both the log's start and the time
the shape went live. Only then are the moved keys of `(watermark, now]` complete.

- covered, only moved keys: a `Moved` body with those keys' current values in the window;
- covered, with axis changes or relabels in range: a `Window` body with `carried`;
- a `Moved::Correct` in range, or not covered: a `Window` body from `None`, which is `target - empty`.

A notice lists no keys for a shape whose list passed `MAX_KEYS`; it says `Moved::Correct` instead, and
costs every reader a correction. `LOG_CAPACITY` bounds the single shared log. Both are set per host by
`limit_delivery` so a test can reach them.

The token in a request is carried and echoed by the client and never read by the host.

## 8. Edge adapters

**Transport contract.** A reply must reach the call that made the request. `Request` and `Correction`
carry no id, so the client cannot tell a late reply from an answer on its own; `sync` and `poll` treat
whatever `Edge::call` returns as the answer (`apply_answer`), and only a reply handed to `apply` is
treated as late. A transport that reorders or duplicates replies must therefore correlate them itself
or drop the strays (improvise-bw4p.10 adds the id to the frames).


`Edge` is one end of a stream: `call(Request) -> Result<Reply, EdgeError>`. `InProcess` is the
function call; `WireEdge` passes request and reply through JSON text, the adapter at a wire boundary,
so the lossless properties are exercised by every test that uses it. Neither adds a `ClientFrame` or
`ServerFrame` variant: the page does not read notices yet, and adding the frames means changing the
browser and the worker together (ADR 008 replaces `Projection`; improvise-bw4p.10). A transport
pushes `Host::take_notices()`.

## Not done, and why

- **The notices are not sent by the socket or worker hosts.** They need frame variants and the page
  (above).
- **Positions and column totals are not on the wire.** A correction carries `(key, value)` and no row or
  column, and `Output` holds body cells only. The ADR attaches the position when output is sent and lists
  totals among the moved keys. Both need the render model and the totals' own keys (ADR 008, clause 10).
- **A twin at the serving end** (a server-side watermark or filter) is not built; the ADR allows it and
  does not require it.
- **A pushed filter** is the client's `window`, applied on every request.
- **Hidden-item overlays, pruning queries and column widths** are ADR 008 and clause 10.

**The same edge, generalised in `zset`.** The `zset` crate's edge module, `crates/zset/src/edge.rs` (`context/code/zset.md`, "Edges") carries Z-set deltas between two circuits with the rules of this file (stamps, coalescing by summing, `target - held`, token-matched answers) and no log; this layer is unchanged and does not use it.