fix(tooling): evidence headings are diffed against cargo, not scraped (improvise-rm4s, improvise-9za9)

Four shapes let a heading name something that never runs: a test inside a raw
string literal, one behind an unenabled cfg feature, one behind
cfg(target_arch = "wasm32"), and one behind #[ignore]. All four cleared the
scrape; injecting them together made the previous checker print "543 entries,
all name a test that exists", exit 0.

--check-evidence now takes the Rust names from `cargo test --workspace -- --list`
minus its `--ignored` twin. All 539 live headings resolve against a real listing,
so the stricter source costs nothing today. rust_tests is rust_citations: the
scrape survives only for the reverse direction, where a false positive is loud.

Prose headings become a third kind, counted and printed, so retiring a claim by
rewriting it as prose moves a number instead of passing in silence
(improvise-9za9). No gate can read prose for truth; the reviewer is the
mechanism, and the diff is what to read.

context/code/tooling.md is new and is the file four docstrings in
check-comment-runs.py already pointed at. design-principles §7 stopped saying
the check runs in one direction, which improvise-lyyc had already made false.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 21:08:25 -07:00
parent 8a559b2686
commit 61473ef4d3
3 changed files with 265 additions and 45 deletions
+11 -3
View File
@@ -342,9 +342,17 @@ what fault was injected, and why the test can fail. It does not go in the test's
doc-comment: comment runs are capped at two lines (see below), and the evidence
is longer than that in every case worth having. The cost of the split is that a
renamed test orphans its entry. `scripts/check-comment-runs.py --check-evidence`
catches that: every `### name` heading must name a test that exists. It checks
one direction only, deliberately — a test may have no evidence (most do not),
but evidence naming a test nobody can find is a dangling explanation.
catches that, in two directions. Every `###`/`####` heading shaped like a test
name must name a test that **runs** — the names are diffed against
`cargo test --workspace -- --list` minus its `--ignored` twin, never scraped out
of the source, because text that looks like a test and a test are
indistinguishable to a scanner (improvise-rm4s). And a test whose doc comment
*cites* `context/test-evidence/` must have an entry there, or the pointer reads
as a promise the explanation exists.
Neither direction is "every test has evidence", deliberately: most tests have
none and should not be forced to have one. The reasoning and the residual holes
are in `context/code/tooling.md`.
### E2E pins the boundaries; unit tests work between them