fix(tooling): a leaf is not an identity, and a convenience flag was a bypass

Both blockers from the adversarial review of #83, both in the mode this PR
rewrote, both producing a pass while the gate did not do what its prose said.

1. cargo_test_names reduced every listed path to its last segment. Six leaf
names collide in this workspace and two are cited in evidence, so a heading was
satisfiable by a test in another module answering another question -- renaming
ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry
describes, left the checker at exit 0 with the ui::host namesake answering for
it. Paths are kept whole; a bare leaf resolves only when unique, and an
ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so
a qualified heading is a claim rather than prose, and a citation is matched on
the leaf so qualifying a heading cannot orphan its own test. The two colliding
entries are qualified. Neither half suffices alone: rename one of a colliding
pair and the leaf becomes unique again, which is why qualification is the fix
and the refusal is what forces it.

2. --rust-tests called cargo_test_names with one argument, so the --ignored
subtraction never ran, and it printed a byte-identical success line. It now
requires --rust-tests-ignored (one alone exits 2) and the line names its source.

Also, from the same review: a missing context/test-evidence/ exited 0 while 170
citations pointed at nothing (early return, now gone); the doc-site "runs last"
check accepted a trailing `echo` that merely NAMED the generator, so it asks for
an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`,
which reaches the bundle while naming nothing forbidden; and two prose overclaims
are corrected -- the prose-heading count is a convenience rather than a guard
(git diff is the mechanism), and the ratio rule says "above the run MEDIAN",
which is defensible where "above the others" is false against second place.

scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs`
every other job in ci.yml. The rule was stated in the last commit and a
zz-new-check: job added during review was still ungated with nothing noticing.
README now states the rule and lists no jobs, the second copy being what went
stale twice.

Every fault re-injected and observed; figures in context/test-evidence/tooling.md
rather than only here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 21:42:07 -07:00
parent 29cdab1a4a
commit 3542d6f717
11 changed files with 418 additions and 56 deletions
+4 -2
View File
@@ -345,8 +345,10 @@ worse than no claim at all. Three consequences:
- **If you cite a figure, name the machine you took it on.** A number with no
machine is a property of the suite, which it is not.
- **The portable signal is the ratio, not the seconds.** One binary an order of
magnitude above the others is the thing to look at, and it is the same
binary on every machine it has been measured on: `improvise-io`'s lib, which
magnitude above the run's **median** is the thing to look at — not above
second place, which it need not be: on one machine `improvise-io` is 2.2×
the runner-up and ~15× the median. It is the same binary on all three
machines it has been measured on: `improvise-io`'s lib, which
is proptest-dominated by the grammar-walking `.improv` generator and the
round-trip properties over it. Constraining that generator's strategy is the
open work (improvise-8shu); lowering the case count below 256 is not (see