fix(protocol): a notice from another epoch only marks the shape stale, and a pulled notice stays with its shape (improvise-z2vr)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
committed by
Edward Langley
parent
f47efef5ae
commit
69fb5e625e
@@ -127,15 +127,22 @@ for the reply to a request they just sent, takes an epoch off `left` before it a
|
||||
reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round;
|
||||
the next request it makes is answered by the live host and brings it back.
|
||||
|
||||
**A notice from an older generation of the host the client is on, or from an epoch it has left, is
|
||||
ignored.** Reading one rewinds `seen` into the old epoch and queues a correction entry, and the
|
||||
correction that answers it is then covered by the watermark and a duplicate, so a `restructured` mark
|
||||
set from the same notice is never cleared and `sync` returns `Exhausted`. A notice the client asked for
|
||||
(`poll`) skips the left-epoch test, since the host that answers is live (see the answer rule above); it
|
||||
still skips an older generation. When every shape ignores a notice, its events are dropped too.
|
||||
**A notice from another epoch only marks the shape stale; `seen` moves only through an answer.** The
|
||||
first two livelocks (improvise-z2vr) were the same defect reached by different routes, which is why
|
||||
ignoring one kind of foreign notice after another was not enough. Epochs of different hosts are
|
||||
unordered, so a notice from an epoch other than `seen`'s cannot be placed. Reading it used to rewind
|
||||
`seen` into the foreign epoch, and then the live host's own old notices looked new, queued work and set
|
||||
`restructured`; the answers that followed were covered by the watermark, were duplicates, and never
|
||||
cleared it. Now such a notice sets `stale` and nothing else, the next request is answered by the live
|
||||
host, and that answer (`apply_answer`) is what moves `seen`. Two cheaper guards stay: a notice from an
|
||||
older generation of the host the shape is on, or from an epoch it has left, is ignored without the extra
|
||||
round; and a duplicate answer clears a `restructured` mark it covers. A notice the client asked for
|
||||
(`poll`) skips the left-epoch test, since the host that answers is live.
|
||||
|
||||
`poll` hands a pulled notice only to the shape it asked about. A notice carries only that shape's keys,
|
||||
and applying it to another tracked shape would advance that shape past keys it never saw.
|
||||
`poll` hands a pulled notice only to the shape it asked about, and the notice says so: `for_shape` sets
|
||||
`Notice::cut_for`, and `accept` limits any notice that carries it to that shape, however it reaches the
|
||||
client (`on_notice` included). Without the mark, a pulled notice delivered as if pushed made the other
|
||||
shape skip keys it never saw.
|
||||
|
||||
`connection_lost`, `request_failed` and `unreadable` set `stale`, which `wants` turns into
|
||||
`Ask::Correction` from the watermark the client held. A duplicate whose `through` equals the watermark
|
||||
|
||||
@@ -206,3 +206,35 @@ property's own weather and a Flaky edge. Rival: the notice-ignore test in `accep
|
||||
and the random property fail. The seed file is not committed (`proptest-regressions/` is ignored).
|
||||
|
||||
Run after the fix, the property passed 4 x 4000 cases and 2 x 20000 cases, with no third failure.
|
||||
|
||||
## Round three: the same livelock by other routes
|
||||
|
||||
### route_a_a_dead_hosts_notice_cannot_make_the_live_hosts_old_notices_look_new
|
||||
|
||||
A client goes host 1 to host 3 and never joins host 2. A late host-2 notice rewinds `seen`, then host 3's
|
||||
own old notices look new and set `restructured` at the watermark, and the answers are duplicates that
|
||||
never clear it. Failed on the head before the change (`wants` stayed `Fetch { window: true }`).
|
||||
|
||||
### route_b_a_covered_reply_cannot_set_seen_below_a_queued_entry
|
||||
|
||||
The same rewind, then an unsolicited covered correction: the duplicate branch set `seen` to its `through`,
|
||||
below a queued entry, and each fetch for that `through` came back a duplicate. Failed on the head before
|
||||
the change.
|
||||
|
||||
### a_notice_from_another_epoch_marks_the_shape_stale_and_moves_nothing_else
|
||||
|
||||
The new rule itself: a foreign notice sets `stale`; the home host's next notice is still contiguous.
|
||||
|
||||
### a_pulled_notice_pushed_through_on_notice_still_reaches_only_its_shape
|
||||
|
||||
A notice cut for shape `a` by `for_shape`, delivered through `on_notice`, must leave shape `b` alone.
|
||||
Failed before `Notice::cut_for` existed.
|
||||
|
||||
### two_shapes_survive_late_repeated_and_dead_messages
|
||||
|
||||
The reviewer's generator, kept: two shapes, notices late, repeated and reordered (the due set is reversed
|
||||
on odd steps), dead hosts' notices kept across restarts, stored notices and stored replies replayed on
|
||||
their own, hosts that restart often. At the default 256 cases it did not find the old head's livelock, and
|
||||
at 20,000 it did within ten seconds (an `Exhausted`), so the default run is a smoke test and the deep run
|
||||
is the check. After the change it passed three runs of 20,000 cases, and the older property passed
|
||||
2 x 20,000 with it.
|
||||
|
||||
Reference in New Issue
Block a user