fix(tooling): three findings from reviewing this branch against itself

1. A self-test case that was a duplicate wearing different clothes. "a heading
cargo's listing does not carry is an orphan however the source reads" passed on
an empty listing -- which is the first case in the table, already. Once defined
comes from cargo, the Rust beside a fixture is not read at all, so four cases
for the four source shapes would be four copies of one assertion dressed as
coverage. Deleted; the evidence for those shapes is the real-tree injection, and
context/code/tooling.md now says which half a fixture can carry and which it
cannot. This is exactly the shape improvise-84wp's prose, applied two commits
ago, tells a reviewer to look for.

2. assemble-stamp.sh grepped its patterns as REGEXES, inherited from the inline
version it was extracted from. The `.` in improvise_wasm_client.js?v=abc matches
any character, so a stamp corrupted in that position read as present -- a pass in
the one direction a guard must not fail in. grep -F, plus a case proved by
injection: dropping the -F flips it.

3. A missing cargo raised a traceback out of --check-evidence. It now names the
flag that takes a saved listing instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 21:17:41 -07:00
parent f4d9d14fcb
commit 1d4d9107cf
5 changed files with 47 additions and 10 deletions
+11 -2
View File
@@ -82,8 +82,17 @@ entries, all name a test that exists*, exit 0. The current one names all four.
What this costs: `--check-evidence` now needs a toolchain and a compiled
workspace, about three seconds warm. `--rust-tests FILE` takes a saved listing
for a caller that has one. The comment-cap check is unaffected and needs
neither.
for a caller that has one, and a missing `cargo` says so rather than raising.
The comment-cap check is unaffected and needs neither.
Note what the self-test can and cannot carry here. The listing *reader* is
testable from fixtures — a leaf name taken off a module path, an `--ignored`
name subtracted, cargo's own prose not mistaken for a test — and those cases
are in `SELF_TEST_LISTING`. The four *source shapes* are not: once `defined`
comes from cargo, a fixture asserting "a heading with an empty listing is an
orphan" is the same case however the Rust beside it is spelled, and writing
four of them would be four copies of one assertion dressed as coverage. The
evidence for those is the injection above, against the real tree.
What it does **not** close: `citing` is still scraped, so a citation inside a
string literal would be reported as a test owing an entry. That is a false
+19
View File
@@ -34,6 +34,13 @@ own header and the test's header both name the directory in prose — a lexical
grep over raw text would have had to exempt them, and an exemption is where the
next violation goes.
**What it cannot see**, stated because the check itself uses the evasion: a path
assembled at runtime (`"../" + "p" + "kg"`) is invisible to a lexical match, and
the test's own `FORBIDDEN` constant is built exactly that way to stay out of its
own scan. That is a deliberate, disclosed hole — closing it needs a parser, and
the failure it would catch is one somebody has to write on purpose, where this
gate exists for the one written by accident.
### "the subdirectory that MAY use a bundle is not the one being checked"
The companion, and the one that stops the first from passing vacuously. If
@@ -47,6 +54,18 @@ Fault injected, 2026-09-21: moved `web/tests/bundle/wasm_surface.test.js` out of
the tree. The test failed with "web/tests/bundle/ is empty — the split lost its
other half" rather than passing. Restored; green again.
### Note — the stamp gate matches literally
`scripts/assemble-stamp.sh` greps with `-F`. As a regex, the `.` in
`improvise_wasm_client.js?v=abc` matches any character, so a stamp corrupted in
exactly that position reads as present — a pass in the one direction a guard
must not fail in. Inherited from the inline version this was extracted from,
and found by reading the extraction rather than by anything failing.
Fault injected, 2026-09-21: dropping the `-F` makes
`improvise_wasm_clientXjs?v=abc` satisfy the pattern, and
`a pattern is matched literally, not as a regex` goes from pass to fail.
### Note — the doc site's derived checks
`scripts/build-doc-site.mjs` carries two gates that no test names, because the
+7
View File
@@ -64,6 +64,13 @@ run_case "a file that does not exist is a miss, not an abort" 1 "vanished.js" \
# A caller that drops one argument would otherwise have the last FILE read as
# a PATTERN over the file before it -- a check that passes by accident.
# The pattern is a literal. As a regex, `.` matches any character, so a stamp
# corrupted in exactly that position would read as present -- the one direction
# a guard must not fail in.
printf 'import "./pkg/improvise_wasm_clientXjs?v=abc";\n' > "$work/corrupt.js"
run_case "a pattern is matched literally, not as a regex" 1 "corrupt.js" \
"$work/corrupt.js" "pkg/improvise_wasm_client.js?v=abc"
run_case "an odd number of arguments is a usage error" 2 "pairs" \
"$work/stamped.js" "pkg/improvise_wasm_client.js?v=abc" "$work/bare.js"
+4 -2
View File
@@ -24,8 +24,10 @@ while [ $# -ge 2 ]; do
file=$1
pattern=$2
shift 2
if [ -f "$file" ] && grep -q "$pattern" "$file"; then
grep -n "$pattern" "$file"
# -F, because every pattern is a literal and `.` as "any character" would
# let a corrupted stamp match -- a pass in the direction that looks like one.
if [ -f "$file" ] && grep -qF "$pattern" "$file"; then
grep -nF "$pattern" "$file"
else
echo "assemble: $file is missing the cache-bust stamp: $pattern" >&2
missing="${missing} $file"
+6 -6
View File
@@ -313,7 +313,12 @@ def cargo_test_names(listing, ignored_listing=""):
def _cargo_listing(extra):
cmd = ["cargo", "test", "--workspace", "--", "--list", *extra]
run = subprocess.run(cmd, capture_output=True, text=True)
try:
run = subprocess.run(cmd, capture_output=True, text=True)
except OSError as e:
raise SystemExit(f"error: cannot run cargo ({e}).\n"
"--check-evidence needs a toolchain, or --rust-tests FILE "
"holding a saved `cargo test --workspace -- --list`.")
if run.returncode != 0:
raise SystemExit(f"error: {' '.join(cmd)} failed:\n{run.stderr}")
return run.stdout
@@ -754,11 +759,6 @@ SELF_TEST_EVIDENCE = [
("a ### naming a helper rather than a test is still an orphan",
"### a_helper\n", "fn a_helper() {}\n", "", "", ["a_helper"], []),
("a heading cargo's listing does not carry is an orphan however the source reads",
"### a_test_that_is_only_text\n",
'pub const S: &str = r#"\n#[test]\nfn a_test_that_is_only_text() {}\n"#;\n',
"", "", ["a_test_that_is_only_text"], []),
("a test that cites the evidence file and has no entry is caught",
"", "/// See `context/test-evidence/clients.md`.\n#[test]\nfn a_cited_test() {}\n",
"", "m::a_cited_test: test\n", [], ["a_cited_test"]),