fix(tooling): a leaf is not an identity, and a convenience flag was a bypass

Both blockers from the adversarial review of #83, both in the mode this PR
rewrote, both producing a pass while the gate did not do what its prose said.

1. cargo_test_names reduced every listed path to its last segment. Six leaf
names collide in this workspace and two are cited in evidence, so a heading was
satisfiable by a test in another module answering another question -- renaming
ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry
describes, left the checker at exit 0 with the ui::host namesake answering for
it. Paths are kept whole; a bare leaf resolves only when unique, and an
ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so
a qualified heading is a claim rather than prose, and a citation is matched on
the leaf so qualifying a heading cannot orphan its own test. The two colliding
entries are qualified. Neither half suffices alone: rename one of a colliding
pair and the leaf becomes unique again, which is why qualification is the fix
and the refusal is what forces it.

2. --rust-tests called cargo_test_names with one argument, so the --ignored
subtraction never ran, and it printed a byte-identical success line. It now
requires --rust-tests-ignored (one alone exits 2) and the line names its source.

Also, from the same review: a missing context/test-evidence/ exited 0 while 170
citations pointed at nothing (early return, now gone); the doc-site "runs last"
check accepted a trailing `echo` that merely NAMED the generator, so it asks for
an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`,
which reaches the bundle while naming nothing forbidden; and two prose overclaims
are corrected -- the prose-heading count is a convenience rather than a guard
(git diff is the mechanism), and the ratio rule says "above the run MEDIAN",
which is defensible where "above the others" is false against second place.

scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs`
every other job in ci.yml. The rule was stated in the last commit and a
zz-new-check: job added during review was still ungated with nothing noticing.
README now states the rule and lists no jobs, the second copy being what went
stale twice.

Every fault re-injected and observed; figures in context/test-evidence/tooling.md
rather than only here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 21:42:07 -07:00
parent 29cdab1a4a
commit 3542d6f717
11 changed files with 418 additions and 56 deletions
+56 -9
View File
@@ -56,6 +56,27 @@ not be forced to have one; what is checkable is a *claim*, and both directions
are claims. The failure they close is the same one a dead ticket ID is: a
pointer that reads as a promise the thing exists.
### A leaf is not an identity
The names cargo lists are **module paths**, and the checker keeps them whole.
Six leaf names collide in this workspace — two of them cited in evidence — so
reducing a path to its last segment made a heading satisfiable by a test in
another module answering another question. Renaming
`ui::project::tests::a_load_that_fails_is_not_a_rebase`, the one its entry
actually describes, left the checker at *579 entries, all name a test that
runs*, exit 0: the surviving `ui::host::tests` namesake answered for it.
So a bare leaf resolves only when it is **unique**, and an ambiguous one is a
**refusal** naming the candidates — not a warning, because a warning re-creates
the problem for anyone who does not read it. The heading is then written with
its module path, which is also what makes the entry say which test it is about.
Note the two halves are both needed and neither is sufficient. The refusal
forces qualification but cannot survive a rename on its own: rename one of a
colliding pair and the leaf becomes *unique*, silently resolving to the
survivor. Qualification is what closes that, and the refusal is what makes
somebody write it.
### The names come from cargo, not from the source
A heading is diffed against `cargo test --workspace -- --list`, minus the names
@@ -81,9 +102,20 @@ Demonstrated before the fix, by injecting all four shapes into `src/draw.rs` and
entries, all name a test that exists*, exit 0. The current one names all four.
What this costs: `--check-evidence` now needs a toolchain and a compiled
workspace, about three seconds warm. `--rust-tests FILE` takes a saved listing
for a caller that has one, and a missing `cargo` says so rather than raising.
The comment-cap check is unaffected and needs neither.
workspace, about three seconds warm. A missing `cargo` says so rather than
raising. The comment-cap check is unaffected and needs neither.
`--rust-tests FILE` takes a saved listing for a caller that has one, and it
**requires `--rust-tests-ignored FILE`**: a first version took one file, called
`cargo_test_names` with one argument, and so skipped the ignored rule entirely
while printing the same success line as the honest path. A flag that is
strictly more permissive and indistinguishable in the log is a bypass, not a
convenience — so the pair is mandatory and the success line **names its
source**. CI passes neither and runs cargo itself.
A **missing** `context/test-evidence/` is not an early return either. It used to
be, and it exited 0 while 170 tests cited a directory that was not there; the
walk now yields nothing and every citation is reported unstated, which is loud.
Note what the self-test can and cannot carry here. The listing *reader* is
testable from fixtures — a leaf name taken off a module path, an `--ignored`
@@ -110,12 +142,17 @@ honest — no mechanism could, because the question is whether a sentence is tru
Two things make that tolerable. The unsafe direction is the one that fails
loudly: prose that *looks* like an identifier is classified as a Rust claim and
reported as an orphan, so the error you can make by accident is the noisy one.
And prose headings are **counted and printed** beside the claims, so a
retirement moves a number in the gate's own output instead of passing in
silence. The live instance — PR #69 demoting a heading while PR #72 restored its
coverage under a different name — is what says the reviewer is the mechanism:
`git diff context/test-evidence/` shows a `### identifier` line becoming prose,
and that diff is the thing to read.
And **the mechanism is `git diff context/test-evidence/`** — a demotion shows
up there as a `### identifier` line becoming prose, in the diff every reviewer
already reads. The live instance, PR #69 demoting a heading while PR #72
restored its coverage under another name, is what says a reviewer rather than a
gate has to catch it.
The printed count of prose headings is a **convenience, not a guard**: nothing
stores the previous count and nobody diffs two CI logs, so a retirement moves a
number no one is watching. Saying otherwise would claim a detectability it
cannot support, which is the class of defect §7's adjacent-check section is
about.
## 3. The others
@@ -126,3 +163,13 @@ shape as this file's subject: a guard, plus a suite that makes the guard refuse
what it claims to refuse. Each is wired into `.github/workflows/ci.yml` in the
same job as the thing it guards, because a check that only runs locally is a
convention.
`scripts/check-ci-graph.py` is the workflow's own gate: `pages-deploy` must
`needs` every other job in `ci.yml`. The rule "every check job" was written down
(improvise-x7cq) and a job added during review of PR #83 was still ungated with
nothing noticing, which is what moved it from a sentence to a check. It reads
the job keys and the `needs:` list off indentation, tracking which job a
`needs:` belongs to — taking the first one seen would read another job's list as
the deploy's and pass. `README.md` therefore states the rule and deliberately
**does not list the jobs**: a second copy with nothing holding it to the first
is what went stale, twice.