fix(tooling): a leaf is not an identity, and a convenience flag was a bypass
Both blockers from the adversarial review of #83, both in the mode this PR rewrote, both producing a pass while the gate did not do what its prose said. 1. cargo_test_names reduced every listed path to its last segment. Six leaf names collide in this workspace and two are cited in evidence, so a heading was satisfiable by a test in another module answering another question -- renaming ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry describes, left the checker at exit 0 with the ui::host namesake answering for it. Paths are kept whole; a bare leaf resolves only when unique, and an ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so a qualified heading is a claim rather than prose, and a citation is matched on the leaf so qualifying a heading cannot orphan its own test. The two colliding entries are qualified. Neither half suffices alone: rename one of a colliding pair and the leaf becomes unique again, which is why qualification is the fix and the refusal is what forces it. 2. --rust-tests called cargo_test_names with one argument, so the --ignored subtraction never ran, and it printed a byte-identical success line. It now requires --rust-tests-ignored (one alone exits 2) and the line names its source. Also, from the same review: a missing context/test-evidence/ exited 0 while 170 citations pointed at nothing (early return, now gone); the doc-site "runs last" check accepted a trailing `echo` that merely NAMED the generator, so it asks for an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`, which reaches the bundle while naming nothing forbidden; and two prose overclaims are corrected -- the prose-heading count is a convenience rather than a guard (git diff is the mechanism), and the ratio rule says "above the run MEDIAN", which is defensible where "above the others" is false against second place. scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs` every other job in ci.yml. The rule was stated in the last commit and a zz-new-check: job added during review was still ungated with nothing noticing. README now states the rule and lists no jobs, the second copy being what went stale twice. Every fault re-injected and observed; figures in context/test-evidence/tooling.md rather than only here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
+56
-9
@@ -56,6 +56,27 @@ not be forced to have one; what is checkable is a *claim*, and both directions
|
||||
are claims. The failure they close is the same one a dead ticket ID is: a
|
||||
pointer that reads as a promise the thing exists.
|
||||
|
||||
### A leaf is not an identity
|
||||
|
||||
The names cargo lists are **module paths**, and the checker keeps them whole.
|
||||
Six leaf names collide in this workspace — two of them cited in evidence — so
|
||||
reducing a path to its last segment made a heading satisfiable by a test in
|
||||
another module answering another question. Renaming
|
||||
`ui::project::tests::a_load_that_fails_is_not_a_rebase`, the one its entry
|
||||
actually describes, left the checker at *579 entries, all name a test that
|
||||
runs*, exit 0: the surviving `ui::host::tests` namesake answered for it.
|
||||
|
||||
So a bare leaf resolves only when it is **unique**, and an ambiguous one is a
|
||||
**refusal** naming the candidates — not a warning, because a warning re-creates
|
||||
the problem for anyone who does not read it. The heading is then written with
|
||||
its module path, which is also what makes the entry say which test it is about.
|
||||
|
||||
Note the two halves are both needed and neither is sufficient. The refusal
|
||||
forces qualification but cannot survive a rename on its own: rename one of a
|
||||
colliding pair and the leaf becomes *unique*, silently resolving to the
|
||||
survivor. Qualification is what closes that, and the refusal is what makes
|
||||
somebody write it.
|
||||
|
||||
### The names come from cargo, not from the source
|
||||
|
||||
A heading is diffed against `cargo test --workspace -- --list`, minus the names
|
||||
@@ -81,9 +102,20 @@ Demonstrated before the fix, by injecting all four shapes into `src/draw.rs` and
|
||||
entries, all name a test that exists*, exit 0. The current one names all four.
|
||||
|
||||
What this costs: `--check-evidence` now needs a toolchain and a compiled
|
||||
workspace, about three seconds warm. `--rust-tests FILE` takes a saved listing
|
||||
for a caller that has one, and a missing `cargo` says so rather than raising.
|
||||
The comment-cap check is unaffected and needs neither.
|
||||
workspace, about three seconds warm. A missing `cargo` says so rather than
|
||||
raising. The comment-cap check is unaffected and needs neither.
|
||||
|
||||
`--rust-tests FILE` takes a saved listing for a caller that has one, and it
|
||||
**requires `--rust-tests-ignored FILE`**: a first version took one file, called
|
||||
`cargo_test_names` with one argument, and so skipped the ignored rule entirely
|
||||
while printing the same success line as the honest path. A flag that is
|
||||
strictly more permissive and indistinguishable in the log is a bypass, not a
|
||||
convenience — so the pair is mandatory and the success line **names its
|
||||
source**. CI passes neither and runs cargo itself.
|
||||
|
||||
A **missing** `context/test-evidence/` is not an early return either. It used to
|
||||
be, and it exited 0 while 170 tests cited a directory that was not there; the
|
||||
walk now yields nothing and every citation is reported unstated, which is loud.
|
||||
|
||||
Note what the self-test can and cannot carry here. The listing *reader* is
|
||||
testable from fixtures — a leaf name taken off a module path, an `--ignored`
|
||||
@@ -110,12 +142,17 @@ honest — no mechanism could, because the question is whether a sentence is tru
|
||||
Two things make that tolerable. The unsafe direction is the one that fails
|
||||
loudly: prose that *looks* like an identifier is classified as a Rust claim and
|
||||
reported as an orphan, so the error you can make by accident is the noisy one.
|
||||
And prose headings are **counted and printed** beside the claims, so a
|
||||
retirement moves a number in the gate's own output instead of passing in
|
||||
silence. The live instance — PR #69 demoting a heading while PR #72 restored its
|
||||
coverage under a different name — is what says the reviewer is the mechanism:
|
||||
`git diff context/test-evidence/` shows a `### identifier` line becoming prose,
|
||||
and that diff is the thing to read.
|
||||
And **the mechanism is `git diff context/test-evidence/`** — a demotion shows
|
||||
up there as a `### identifier` line becoming prose, in the diff every reviewer
|
||||
already reads. The live instance, PR #69 demoting a heading while PR #72
|
||||
restored its coverage under another name, is what says a reviewer rather than a
|
||||
gate has to catch it.
|
||||
|
||||
The printed count of prose headings is a **convenience, not a guard**: nothing
|
||||
stores the previous count and nobody diffs two CI logs, so a retirement moves a
|
||||
number no one is watching. Saying otherwise would claim a detectability it
|
||||
cannot support, which is the class of defect §7's adjacent-check section is
|
||||
about.
|
||||
|
||||
## 3. The others
|
||||
|
||||
@@ -126,3 +163,13 @@ shape as this file's subject: a guard, plus a suite that makes the guard refuse
|
||||
what it claims to refuse. Each is wired into `.github/workflows/ci.yml` in the
|
||||
same job as the thing it guards, because a check that only runs locally is a
|
||||
convention.
|
||||
|
||||
`scripts/check-ci-graph.py` is the workflow's own gate: `pages-deploy` must
|
||||
`needs` every other job in `ci.yml`. The rule "every check job" was written down
|
||||
(improvise-x7cq) and a job added during review of PR #83 was still ungated with
|
||||
nothing noticing, which is what moved it from a sentence to a check. It reads
|
||||
the job keys and the `needs:` list off indentation, tracking which job a
|
||||
`needs:` belongs to — taking the first one seen would read another job's list as
|
||||
the deploy's and pass. `README.md` therefore states the rule and deliberately
|
||||
**does not list the jobs**: a second copy with nothing holding it to the first
|
||||
is what went stale, twice.
|
||||
|
||||
Reference in New Issue
Block a user