No code fix. The ticket asked that any fix first establish why two of three
green uploads had no artifact at deploy time, and the answer is neither of the
two directions it offered.
Measured over all 100 push-to-main CI runs from 2026-09-19T22:02Z to
2026-09-21T05:02Z, via the run/job/step/log APIs -- job metadata and logs
outlive artifacts, which expire in a day and are all 404 now, so this is
reproducible where the ticket's own artifact-count table no longer is:
12 artifact-not-found failures -- 12 (100%) had another run's SUCCESSFUL
deploy complete inside their own upload->deploy window
56 successes -- 2 (3.6%) did
4 other failures -- 0; all four are the unrelated "Create Pages
site failed: Resource not accessible by integration", pre-improvise-3gy
Direction: the oldest run of the 17:47Z batch failed at 17:47:23Z, before the
next deploy in the batch started (17:47:31Z) and before any Pages deployment
existed at all, since deploy-pages fetches artifact metadata BEFORE creating a
deployment. So it cannot have been caused by its own deploy, by supersession or
by a concurrent one. The artifact was already gone.
Scoped by artifact NAME, not by run: run 35560726456 finalized harness-serve,
wasm-bundles and github-pages seconds apart, and 2m21s later its own lookup
reported "Found 2 artifact(s)" with no github-pages. Same run, same token, same
call, same retention -- which rules out service flakiness, eventual consistency,
retention, token scope and a v3/v4 backend mismatch in one observation, since
any of them would have taken the siblings too.
One clause of the ticket's own correction is falsified: 52507a6 did not fail
"anyway" -- 5f871f7's deploy completed four seconds before its own began. Fewer
successful deploys is the right lever, so PR #46 aimed at the right quantity for
the wrong reason. It still should not ship: the association is not a mechanism
and the mechanism is not deterministic (5f871f7 had a foreign deploy inside its
own window and survived), so shipping it would be exactly
doctrine(measure_the_obvious_fix_before_reporting_it_as_the_fix).
Also corrects the pages-deploy comment in ci.yml, which stated the superseded-run
mechanism this evidence refutes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
scripts/check-comment-runs.py early-returned on args.check_evidence, so the
flagged run answered the evidence question and said nothing at all about the
comment-run cap. The flagged form is the expensive one -- it needs a compiled
workspace -- which makes it the one a human runs alone; three agents hit this
independently in one campaign, and in one case a local gate went green while
CI went red on a three-line doc comment (improvise-ijql).
The cap moves into check_cap(), and --check-evidence now runs it first and
returns the worse of the two verdicts. The ticket's cheaper alternative -- keep
the early return and print a line saying the cap did not run -- is declined and
the reason is recorded: a provenance line qualifies an answer the run gave,
where a disclaimer qualifies one it did not, while still exiting 0. Exit 0 is
what CI branches on and what a human reads as clean, so the rule is stated over
the exit code: a tree with a long comment run must not pass any invocation that
can exit 0.
The guard is _dispatch_self_test, which runs the script as a subprocess against
a temporary tree -- the only level at which this is visible, since every unit in
the file was correct and the defect was in main()'s dispatch. Both arms were
demonstrated by injection: restoring the early return fails
"DISPATCH: --check-evidence: exited 0 over a four-line comment run", and forcing
the cap to always report a violation fails the clean arm for both invocations,
which is what stops an always-fail from satisfying the first.
Measured: the cap adds 0.29s to a 6.5s --check-evidence run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Both blockers from the adversarial review of #83, both in the mode this PR
rewrote, both producing a pass while the gate did not do what its prose said.
1. cargo_test_names reduced every listed path to its last segment. Six leaf
names collide in this workspace and two are cited in evidence, so a heading was
satisfiable by a test in another module answering another question -- renaming
ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry
describes, left the checker at exit 0 with the ui::host namesake answering for
it. Paths are kept whole; a bare leaf resolves only when unique, and an
ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so
a qualified heading is a claim rather than prose, and a citation is matched on
the leaf so qualifying a heading cannot orphan its own test. The two colliding
entries are qualified. Neither half suffices alone: rename one of a colliding
pair and the leaf becomes unique again, which is why qualification is the fix
and the refusal is what forces it.
2. --rust-tests called cargo_test_names with one argument, so the --ignored
subtraction never ran, and it printed a byte-identical success line. It now
requires --rust-tests-ignored (one alone exits 2) and the line names its source.
Also, from the same review: a missing context/test-evidence/ exited 0 while 170
citations pointed at nothing (early return, now gone); the doc-site "runs last"
check accepted a trailing `echo` that merely NAMED the generator, so it asks for
an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`,
which reaches the bundle while naming nothing forbidden; and two prose overclaims
are corrected -- the prose-heading count is a convenience rather than a guard
(git diff is the mechanism), and the ratio rule says "above the run MEDIAN",
which is defensible where "above the others" is false against second place.
scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs`
every other job in ci.yml. The rule was stated in the last commit and a
zz-new-check: job added during review was still ungated with nothing noticing.
README now states the rule and lists no jobs, the second copy being what went
stale twice.
Every fault re-injected and observed; figures in context/test-evidence/tooling.md
rather than only here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
1. A self-test case that was a duplicate wearing different clothes. "a heading
cargo's listing does not carry is an orphan however the source reads" passed on
an empty listing -- which is the first case in the table, already. Once defined
comes from cargo, the Rust beside a fixture is not read at all, so four cases
for the four source shapes would be four copies of one assertion dressed as
coverage. Deleted; the evidence for those shapes is the real-tree injection, and
context/code/tooling.md now says which half a fixture can carry and which it
cannot. This is exactly the shape improvise-84wp's prose, applied two commits
ago, tells a reviewer to look for.
2. assemble-stamp.sh grepped its patterns as REGEXES, inherited from the inline
version it was extracted from. The `.` in improvise_wasm_client.js?v=abc matches
any character, so a stamp corrupted in that position read as present -- a pass in
the one direction a guard must not fail in. grep -F, plus a case proved by
injection: dropping the -F flips it.
3. A missing cargo raised a traceback out of --check-evidence. It now names the
flag that takes a saved listing instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Four shapes let a heading name something that never runs: a test inside a raw
string literal, one behind an unenabled cfg feature, one behind
cfg(target_arch = "wasm32"), and one behind #[ignore]. All four cleared the
scrape; injecting them together made the previous checker print "543 entries,
all name a test that exists", exit 0.
--check-evidence now takes the Rust names from `cargo test --workspace -- --list`
minus its `--ignored` twin. All 539 live headings resolve against a real listing,
so the stricter source costs nothing today. rust_tests is rust_citations: the
scrape survives only for the reverse direction, where a false positive is loud.
Prose headings become a third kind, counted and printed, so retiring a claim by
rewriting it as prose moves a number instead of passing in silence
(improvise-9za9). No gate can read prose for truth; the reviewer is the
mechanism, and the diff is what to read.
context/code/tooling.md is new and is the file four docstrings in
check-comment-runs.py already pointed at. design-principles §7 stopped saying
the check runs in one direction, which improvise-lyyc had already made false.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Edward Langley