fix(protocol): a notice from another epoch only marks the shape stale, and a pulled notice stays with its shape (improvise-z2vr)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (dbsp-delivery)
2026-10-10 20:08:34 -07:00
committed by Edward Langley
parent f47efef5ae
commit 69fb5e625e
6 changed files with 281 additions and 18 deletions
+15 -8
View File
@@ -127,15 +127,22 @@ for the reply to a request they just sent, takes an epoch off `left` before it a
reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round;
the next request it makes is answered by the live host and brings it back.
**A notice from an older generation of the host the client is on, or from an epoch it has left, is
ignored.** Reading one rewinds `seen` into the old epoch and queues a correction entry, and the
correction that answers it is then covered by the watermark and a duplicate, so a `restructured` mark
set from the same notice is never cleared and `sync` returns `Exhausted`. A notice the client asked for
(`poll`) skips the left-epoch test, since the host that answers is live (see the answer rule above); it
still skips an older generation. When every shape ignores a notice, its events are dropped too.
**A notice from another epoch only marks the shape stale; `seen` moves only through an answer.** The
first two livelocks (improvise-z2vr) were the same defect reached by different routes, which is why
ignoring one kind of foreign notice after another was not enough. Epochs of different hosts are
unordered, so a notice from an epoch other than `seen`'s cannot be placed. Reading it used to rewind
`seen` into the foreign epoch, and then the live host's own old notices looked new, queued work and set
`restructured`; the answers that followed were covered by the watermark, were duplicates, and never
cleared it. Now such a notice sets `stale` and nothing else, the next request is answered by the live
host, and that answer (`apply_answer`) is what moves `seen`. Two cheaper guards stay: a notice from an
older generation of the host the shape is on, or from an epoch it has left, is ignored without the extra
round; and a duplicate answer clears a `restructured` mark it covers. A notice the client asked for
(`poll`) skips the left-epoch test, since the host that answers is live.
`poll` hands a pulled notice only to the shape it asked about. A notice carries only that shape's keys,
and applying it to another tracked shape would advance that shape past keys it never saw.
`poll` hands a pulled notice only to the shape it asked about, and the notice says so: `for_shape` sets
`Notice::cut_for`, and `accept` limits any notice that carries it to that shape, however it reaches the
client (`on_notice` included). Without the mark, a pulled notice delivered as if pushed made the other
shape skip keys it never saw.
`connection_lost`, `request_failed` and `unreadable` set `stale`, which `wants` turns into
`Ask::Correction` from the watermark the client held. A duplicate whose `through` equals the watermark
+32
View File
@@ -206,3 +206,35 @@ property's own weather and a Flaky edge. Rival: the notice-ignore test in `accep
and the random property fail. The seed file is not committed (`proptest-regressions/` is ignored).
Run after the fix, the property passed 4 x 4000 cases and 2 x 20000 cases, with no third failure.
## Round three: the same livelock by other routes
### route_a_a_dead_hosts_notice_cannot_make_the_live_hosts_old_notices_look_new
A client goes host 1 to host 3 and never joins host 2. A late host-2 notice rewinds `seen`, then host 3's
own old notices look new and set `restructured` at the watermark, and the answers are duplicates that
never clear it. Failed on the head before the change (`wants` stayed `Fetch { window: true }`).
### route_b_a_covered_reply_cannot_set_seen_below_a_queued_entry
The same rewind, then an unsolicited covered correction: the duplicate branch set `seen` to its `through`,
below a queued entry, and each fetch for that `through` came back a duplicate. Failed on the head before
the change.
### a_notice_from_another_epoch_marks_the_shape_stale_and_moves_nothing_else
The new rule itself: a foreign notice sets `stale`; the home host's next notice is still contiguous.
### a_pulled_notice_pushed_through_on_notice_still_reaches_only_its_shape
A notice cut for shape `a` by `for_shape`, delivered through `on_notice`, must leave shape `b` alone.
Failed before `Notice::cut_for` existed.
### two_shapes_survive_late_repeated_and_dead_messages
The reviewer's generator, kept: two shapes, notices late, repeated and reordered (the due set is reversed
on odd steps), dead hosts' notices kept across restarts, stored notices and stored replies replayed on
their own, hosts that restart often. At the default 256 cases it did not find the old head's livelock, and
at 20,000 it did within ten seconds (an `Exhausted`), so the default run is a smoke test and the deep run
is the check. After the change it passed three runs of 20,000 cases, and the older property passed
2 x 20,000 with it.