Both found by ChatGPT Codex on PR #133, both invisible on macOS, both
arriving as exit 2 with no output -- jq's answer to an invalid --argjson,
so a shell fault at the top of the script wears jq's exit code.
`local n="$1" name="$2" f="$DIR/$n.$name.json"` creates every name before
running the assignments on bash 5, so $n is unbound in the third clause
and set -u kills the shell. The dead command substitution then hands
--argjson an empty string. Works on macOS bash 3.2: the suite passed
19/19 locally and 2/19 in CI on the same commit. Each local now gets its
own statement.
`gh api --paginate` emits one JSON array PER PAGE, so a comment thread
past the page size is several concatenated documents that --argjson
rejects the same way. Both fetch paths now flatten through one
pages_to_array (`jq -s 'add // []'`), which is a no-op on a single page.
The fixture path shares that helper, which is the only reason a fixture
can test it: a fixture holding a single array would exercise a shape the
live fetch never produces. Three cases added, and removing the helper
from the fixture path fails exactly those three, the wait case with the
live exit-2 signature. Suite is 22 cases.
No PR in this repo has yet had enough inline comments to reach a second
page, so nothing but the test will see that one before a user does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
It printed review BODIES only and was blind to inline review comments --
the surface a human reviewer is most likely to use. On PR #122 the user
reviewed with the body "Generally good, added comments inline." and five
substantive rulings underneath it; pr-reviews printed the one-liner, and
an agent reported the PR as awaiting review while the answers sat on it.
pulls/{n}/comments and issues/{n}/comments join pulls/{n}/reviews, all
three interleaved in time order, each inline comment carrying its path
and line (falling back to original_line, which is what an outdated
comment has left).
The exit contract is unchanged: 0 means something exists against the
current head, 1 means the wait elapsed, 4 is usage. An inline comment at
the head now satisfies --wait -- it carries a commit_id and is exactly
the case that failed. An issue-level comment does not: it names no
commit and so makes no claim about which tree was read.
The usage's exit-code warning is widened from "do not pipe into
head/tail" to the real rule: nothing may follow the script in the same
`;` list either, because a trailing `echo "exit=$?"` makes the status
the echo's.
scripts/pr-reviews-test.sh is the check_the_checker, fixture-driven and
needing no network, wired into CI's kb job beside pr-watch-test.sh and
release-preflight-test.sh. Five faults were injected and watched to
fail; context/test-evidence/tooling.md records which case caught each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
All three found by the adversarial review of 019cbc1, each by injection.
1. every_model_command() is NOT compiler-maintained, and both this PR's prose
and reduce.rs's own doc comment said it was. variant_name's exhaustive match
forces an arm in variant_name, not an entry in the value list -- deleting
ModelCommand::TogglePruneEmpty from it failed the new completeness check and
NOTHING ELSE in the workspace. So the diff is two independent hand-written
lists: strictly better than one list against its own length, and not the
derived enumeration improvise-5ubs got for the two enums. Doc comment
corrected; the prose in context/code/command.md and the evidence entry now
say what it is. Closing it properly wants improvise_protocol::variants made
pub so declared::<ModelCommand>() can bound the list -- another wave's crate,
reported rather than taken.
The assertion message was also ONE-DIRECTIONAL ("a name only on the right
..."), which is the shape that injection produces and which invites the wrong
repair: delete the case on the left, degrading both sides together. It now
names both directions and says which is which.
2. SELF_TEST_DUPLICATES case 3 was named for the ambiguity guard and never
reached it: both its headings were module-qualified, so canonical() returned
on the exact-match arm before touching by_leaf. Dropping the guard
(`if matches` for `if len(matches) == 1`) left --self-test at exit 0. The
fixture is now one BARE heading over an ambiguous leaf plus BOTH qualified
spellings, so the injection collides with one of them whichever way
next(iter(set)) falls -- deterministic across five runs, verified.
3. The evidence entry said five pre-existing records-mode tests fail under the
settle-before-rebuild injection; measured, it is six. And "three hundred
lines above" is four hundred.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
--check-evidence verified that a heading names a test that RUNS, and a
duplicate names one just as well, so both copies pass. PR #92 shipped two
entries under one heading; the older described a fixture the test no longer
had.
Refuse rather than warn, on the improvise-rm4s precedent, naming the file and
every line in file order. Keyed by the RESOLVED test, so `### f` beside
`### m::tests::f` collides too -- the case rm4s's qualification repair creates.
Six SELF_TEST_DUPLICATES cases, including the three clean arms: two different
tests sharing a leaf are ambiguous and not duplicate, prose repeats freely, and
two headings naming two tests are fine.
Resolves the two tui.md duplicates: the older column-header copy called
THE_MEMBERSHIP_CORRECT_SPANS a live tripwire three hundred lines above the
entry recording its removal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
scripts/check-comment-runs.py early-returned on args.check_evidence, so the
flagged run answered the evidence question and said nothing at all about the
comment-run cap. The flagged form is the expensive one -- it needs a compiled
workspace -- which makes it the one a human runs alone; three agents hit this
independently in one campaign, and in one case a local gate went green while
CI went red on a three-line doc comment (improvise-ijql).
The cap moves into check_cap(), and --check-evidence now runs it first and
returns the worse of the two verdicts. The ticket's cheaper alternative -- keep
the early return and print a line saying the cap did not run -- is declined and
the reason is recorded: a provenance line qualifies an answer the run gave,
where a disclaimer qualifies one it did not, while still exiting 0. Exit 0 is
what CI branches on and what a human reads as clean, so the rule is stated over
the exit code: a tree with a long comment run must not pass any invocation that
can exit 0.
The guard is _dispatch_self_test, which runs the script as a subprocess against
a temporary tree -- the only level at which this is visible, since every unit in
the file was correct and the defect was in main()'s dispatch. Both arms were
demonstrated by injection: restoring the early return fails
"DISPATCH: --check-evidence: exited 0 over a four-line comment run", and forcing
the cap to always report a violation fails the clean arm for both invocations,
which is what stops an always-fail from satisfying the first.
Measured: the cap adds 0.29s to a 6.5s --check-evidence run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Both blockers from the adversarial review of #83, both in the mode this PR
rewrote, both producing a pass while the gate did not do what its prose said.
1. cargo_test_names reduced every listed path to its last segment. Six leaf
names collide in this workspace and two are cited in evidence, so a heading was
satisfiable by a test in another module answering another question -- renaming
ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry
describes, left the checker at exit 0 with the ui::host namesake answering for
it. Paths are kept whole; a bare leaf resolves only when unique, and an
ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so
a qualified heading is a claim rather than prose, and a citation is matched on
the leaf so qualifying a heading cannot orphan its own test. The two colliding
entries are qualified. Neither half suffices alone: rename one of a colliding
pair and the leaf becomes unique again, which is why qualification is the fix
and the refusal is what forces it.
2. --rust-tests called cargo_test_names with one argument, so the --ignored
subtraction never ran, and it printed a byte-identical success line. It now
requires --rust-tests-ignored (one alone exits 2) and the line names its source.
Also, from the same review: a missing context/test-evidence/ exited 0 while 170
citations pointed at nothing (early return, now gone); the doc-site "runs last"
check accepted a trailing `echo` that merely NAMED the generator, so it asks for
an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`,
which reaches the bundle while naming nothing forbidden; and two prose overclaims
are corrected -- the prose-heading count is a convenience rather than a guard
(git diff is the mechanism), and the ratio rule says "above the run MEDIAN",
which is defensible where "above the others" is false against second place.
scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs`
every other job in ci.yml. The rule was stated in the last commit and a
zz-new-check: job added during review was still ungated with nothing noticing.
README now states the rule and lists no jobs, the second copy being what went
stale twice.
Every fault re-injected and observed; figures in context/test-evidence/tooling.md
rather than only here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
1. A self-test case that was a duplicate wearing different clothes. "a heading
cargo's listing does not carry is an orphan however the source reads" passed on
an empty listing -- which is the first case in the table, already. Once defined
comes from cargo, the Rust beside a fixture is not read at all, so four cases
for the four source shapes would be four copies of one assertion dressed as
coverage. Deleted; the evidence for those shapes is the real-tree injection, and
context/code/tooling.md now says which half a fixture can carry and which it
cannot. This is exactly the shape improvise-84wp's prose, applied two commits
ago, tells a reviewer to look for.
2. assemble-stamp.sh grepped its patterns as REGEXES, inherited from the inline
version it was extracted from. The `.` in improvise_wasm_client.js?v=abc matches
any character, so a stamp corrupted in that position read as present -- a pass in
the one direction a guard must not fail in. grep -F, plus a case proved by
injection: dropping the -F flips it.
3. A missing cargo raised a traceback out of --check-evidence. It now names the
flag that takes a saved listing instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
The manifest only ever grows, so a doc added without a line in it published
nothing and said nothing: build green, link check green, doc simply absent.
The generator now walks context/**/*.md plus docs/design-notes.md, subtracts the
manifest, and fails unless the remainder is exactly the OMITTED map -- checked
in BOTH directions, because an entry naming a file nobody can find asserts a
reason for nothing.
Filed as latent; it was live. context/code/kb-tool.md had been unpublished,
unargued and unnoticed since it was added, while doc-site.md's own table says
context/code/*.md is published. It is published now, with context/code/tooling.md
and context/test-evidence/tooling.md: 23 pages to 26.
Second gate, same file: the generator reads web/assemble.sh and refuses unless
its own invocation is that script's last command. Its outward-link check resolves
../index.html and ../app/ against the filesystem, so a line appended below it
would make that check pass against a half-built tree.
Both run at module scope, before any page is generated, so a manifest error is
answered before the 48 outward-link failures a bare run reports -- a missing doc
is actionable wherever you ran the generator, and the two are unrelated
diagnoses.
Three faults injected and observed; evidence in context/test-evidence/tooling.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
The stamp gate is scripts/assemble-stamp.sh, taking FILE PATTERN pairs, and
scripts/assemble-stamp-test.sh is its check_the_checker. Deleting the final
`test -z "$missing"` makes four of its eight cases fail; before the extraction
that same deletion was invisible, both misses printing while assemble.sh exited
0. It is a separate script because exercising it inside assemble.sh would need a
fabricated web/pkg, which is a second answer to "what does the site assemble
from" (improvise-vvl3). Wired into the `build` job, first, because it needs
nothing and guards a step in that job.
web/tests/bundle_freedom.test.js refuses a flat web/tests/ file that names the
bundle directory -- the constraint keeping the `js` job at seconds rather than
~4m30s, stated in prose and checked by nothing. It scans its own filename too,
so it cannot become the place a violation hides, and its companion asserts
web/tests/bundle/ still holds a test that DOES use the bundle: a scoping rule is
uninteresting if nothing falls outside it.
Both faults were injected and watched to fail; evidence in
context/test-evidence/tooling.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Edward Langley