improvise/context/test-evidence/tooling.md

Test evidence — the repository's own gates

Evidence for the checks in scripts/ and for the tests that hold CI's shape. The arguments behind the checkers themselves are in context/code/tooling.md; this file records what fault each test was watched to catch.

Note what is deliberately absent. scripts/check-comment-runs.py --self-test, scripts/kb-test.sh and scripts/assemble-stamp-test.sh are check-the-checker suites whose cases are already injected faults, and none of their case names is a Rust fn or a JS test( the evidence checker could resolve — so their evidence is the case list itself plus context/code/tooling.md, and inventing headings for them here would be headings naming nothing.

"no test in web/tests/ reaches for a built bundle"

The js CI job runs node --test web/tests/*.test.js and installs nothing; the browser job runs web/tests/bundle/*.test.js after three wasm-pack builds. The moment a flat web/tests/ file reads web/pkg/, the fast job inherits the slow one's dependency — and the likely repair is to add a build step to it, silently spending the ~4m30s the split exists to avoid. Until improvise-4ktt that constraint was a directory convention and a glob, checked by nothing, and it failed in the direction that looks like ordinary breakage: a confusing ENOENT in CI on a machine where the bundle happens not to be built.

Fault injected, 2026-09-21: appended const injected = readFileSync("../pkg/improvise_wasm_client.js"); to web/tests/grid_plan.test.js. The test failed naming that file and pointing at web/tests/bundle/. Reverted; green again.

Two things the test does that a simpler one would not. It asserts its own filename is among the files it scans, so it cannot become the one place a violation may hide. And it strips comments before matching, because this file's own header and the test's header both name the directory in prose — a lexical grep over raw text would have had to exempt them, and an exemption is where the next violation goes.

What it cannot see, stated because the check itself uses the evasion: a path assembled at runtime ("../" + "p" + "kg") is invisible to a lexical match, and the test's own FORBIDDEN constant is built exactly that way to stay out of its own scan. That is a deliberate, disclosed hole — closing it needs a parser, and the failure it would catch is one somebody has to write on purpose, where this gate exists for the one written by accident.

"the subdirectory that MAY use a bundle is not the one being checked"

The companion, and the one that stops the first from passing vacuously. If web/tests/bundle/ were emptied or its files stopped touching the bundle, the first test would still be green while having nothing to be scoped against — "no flat test uses the bundle" is uninteresting if no test anywhere does. This asserts the other half of the split exists and really does what the flat half is forbidden to do.

Fault injected, 2026-09-21: moved web/tests/bundle/wasm_surface.test.js out of the tree. The test failed with "web/tests/bundle/ is empty — the split lost its other half" rather than passing. Restored; green again.

"no test in web/tests/ imports a module that reaches the bundle"

The companion to the lexical check above, and the reason that one is not enough: import "../app.js" needs the bundle transitively and names nothing forbidden, so the js job breaks loudly but unattributed. app.js and worker.js are the two entry points that load the bundle; shell.js and serial.js do not, and serial.test.js imports one legitimately.

Fault injected, 2026-09-21: appended import "../app.js"; to web/tests/grid_plan.test.js. The lexical test stayed green; this one failed naming the file. Reverted.

Note — the checker's own blockers, found by review of PR #83

Two gates in scripts/check-comment-runs.py produced a pass while not doing what their prose said. Both were in the mode the same PR rewrote, which is the part worth keeping: writing the doctrine down does not exempt the writer.

A leaf name is not an identity. Six leaf names collide in this workspace and two were cited in evidence. Renaming ui::project::tests::a_load_that_fails_is_not_a_rebase — the one its entry describes — left the checker printing all name a test that runs, exit 0, with the heading answered by the ui::host::tests namesake. Now an ambiguous leaf is refused with both candidates named, and the two entries are qualified. Re-verified both ways, 2026-09-21: the refusal fires on the unqualified headings and names exactly those two; with the heading qualified, the same rename is reported as an orphan rather than absorbed.

--rust-tests was strictly more permissive and said the same words. It called cargo_test_names with one argument, so the --ignored subtraction never ran, and the success line was byte-identical to the honest path's. Injecting ### eyeball_the_rendered_hints_and_pages orphaned on the default path and passed on the flag's. Now the flag requires --rust-tests-ignored (one alone exits 2) and the success line names its source.

A missing evidence directory exited 0. The early return preceded the citation scan, so deleting context/test-evidence/ reported nothing while 170 citations pointed at a directory that was not there. Verified 2026-09-21 by moving the directory aside: exit 1, 170 tests cite context/test-evidence and have no entry there.

Note — the flag that selected the cap away (improvise-ijql)

scripts/check-comment-runs.py --check-evidence early-returned in main, so the flagged run never answered the comment-run cap and printed nothing about having skipped it. The reviewer's position when it was first raised was that CI runs all three spellings (ci.yml, the three check-comment-runs.py steps in the test job) so this was documentation only. The counter that carried is that the flagged form is the expensive one — it needs a compiled workspace — which makes it the one a human runs alone; three agents hit it independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment.

Reproduced before the fix, 2026-09-21, in a temporary tree holding one .rs file with a four-line comment run:

$ check-comment-runs.py src --check-evidence --rust-tests … --rust-tests-ignored …
test-evidence: 0 entries, all name a test that runs …
EXIT=0
$ check-comment-runs.py src
1 comment runs longer than 2 lines:
  src/bad.rs:1  4 lines
EXIT=1

Two invocations over one tree, opposite verdicts, and the one that exits 0 is the one a human is more likely to run by itself.

The guard is _dispatch_self_test, which runs the script as a subprocess — the only level at which this is visible, since every unit in the file was correct and the defect was in main's dispatch. Both arms were demonstrated:

injected fault	what failed
restore the --check-evidence early return	DISPATCH: --check-evidence: exited 0 over a four-line comment run
make the cap always report a violation	DISPATCH: bare: refused a clean tree and the same for --check-evidence

The second arm is what stops an always-failing cap from satisfying the first, and it is why SELF_TEST_DISPATCH asserts a clean tree passes as well as a dirty one failing. Why the fix is additive rather than a printed disclaimer, and why --self-test/--list-doctests are excluded from the table rather than exempted from the rule: context/code/tooling.md §2, "A flag adds a question".

Note — the stamp gate's discrimination, measured

scripts/assemble-stamp-test.sh separates three outcomes rather than two, which is what makes it evidence rather than a smoke test. Measured 2026-09-21 over its nine cases: the real script passes 9; deleting the final test -z "$missing" (the PR #69 injection, which the inline version survived at exit 0) passes 4 and fails 5; a script that does nothing passes 2; one that always fails passes 0.

The PR body first cited "4 of 9" for the middle row, from a run taken when the suite had eight cases. A figure is what the next reader checks, so it is recorded here where it can be re-measured rather than only in a message nothing can audit (design-principles §6).

Note — the stamp gate matches literally

scripts/assemble-stamp.sh greps with -F. As a regex, the . in improvise_wasm_client.js?v=abc matches any character, so a stamp corrupted in exactly that position reads as present — a pass in the one direction a guard must not fail in. Inherited from the inline version this was extracted from, and found by reading the extraction rather than by anything failing.

Fault injected, 2026-09-21: dropping the -F makes improvise_wasm_clientXjs?v=abc satisfy the pattern, and a pattern is matched literally, not as a regex goes from pass to fail.

Note — the doc site's derived checks

scripts/build-doc-site.mjs carries two gates that no test names, because the generator is the test: it runs on every assembly and exits non-zero.

Faults injected, 2026-09-21, each observed:

injected	reported
a new context/code/newarea.md	context/code/newarea.md is neither published nor a named omission
context/plan.md moved out of the tree	context/plan.md is listed as omitted but does not exist
a line appended below the generator in web/assemble.sh	web/assemble.sh must invoke this generator LAST, naming the offending command
a trailing echo "see scripts/build-doc-site.mjs"	the same, once the check asks for an INVOCATION rather than a mention — .includes had accepted it (review, PR #83)

scripts/check-ci-graph.py is the third, and it is a script rather than a stanza because its own refusals need checking: adding a zz-new-check: job to ci.yml reports pages-deploy does not need zz-new-check and exits 1, where before the rule was prose and nothing noticed. Its five self-test graphs include the one a naive parse gets wrong — another job's needs: read as the deploy's.

The second arm is the one that would not get written first, and it is the point: without it, deleting or renaming an omitted file leaves an entry asserting a reason for a file nobody can find — the same failure as a dead ticket id, which AGENTS.md already calls worse than no id.

The first arm was filed as latent and was live when it was written: context/code/kb-tool.md had been unpublished, unargued and unnoticed since it was added.

Duplicate evidence headings — SELF_TEST_DUPLICATES

improvise-hsoe. --check-evidence verified that a heading names a test that runs, and a duplicate heading names one just as well, so both copies pass. PR #92 shipped two entries under a_layout_change_moves_the_layout_version_and_no_record, and two more pairs sat in tui.md — one of them asserting that THE_MEMBERSHIP_CORRECT_SPANS was "a live tripwire, do not remove it" some four hundred lines above the entry recording that improvise-gd83 had removed it. A reader meeting the older copy first takes it as current, which is worse than an orphan: an orphan announces itself as broken.

The fault is injected by the six cases in SELF_TEST_DUPLICATES, and three of them are the ones that would not get written first. Two different tests sharing a leaf must come back ambiguous and not duplicate, because the wrong diagnosis files the wrong repair. A prose heading must be free to repeat, because it claims no test. And the bare and module-qualified spellings of one test must collide — grouping by the string rather than by the resolved test would have missed exactly the case improvise-rm4s's qualification repair creates, which is the first way this would have come back.

The live tree was itself the first injection: before the three merges the check exited 1 naming all three pairs with both line numbers; after them it exits 0. The clean arms (two headings naming two tests, and prose repeated) are what stop an always-refuse from satisfying the rest.

The PR review reader's own suite — scripts/pr-reviews-test.sh

improvise-xgra. Like kb-test.sh and pr-watch-test.sh, its cases are already injected faults and none of their names is a Rust fn the evidence checker could resolve, so the case list is the evidence. What is recorded here is the five faults it was watched to fail against, and which case caught each — a suite that only asked "did it print something" passes all five.

Fault 1, the inline surface dropped entirely, which is the live bug as it shipped: inline: [] in both fetch paths. Eight cases fail, including both --wait arms. This is the one that matters, because the old script's behaviour was this fault and it exited 0 on PR #122's review body while printing none of the five rulings underneath it.

Fault 2, an inline comment counted rather than compared: ((.inline|length) > 0) in place of the commit_id == head test. One case fails, an inline comment at an EARLIER head does not satisfy --wait, and that is the pair the two inline fixtures exist to discriminate — the at-head case passes under this fault, so on its own it proves nothing.

Fault 3, the time sort dropped (| . for | sort_by(.at)). Only the three surfaces interleave in time order fails: it reads the three bodies back as SECOND THIRD FIRST, the per-endpoint concatenation order. Every content assertion still passes, because each body is still printed.

Fault 4, no original_line fallback. Only the outdated-comment case fails. line is null exactly when the line has moved under the comment, so a renderer reading .line alone prints draw.rs:null on precisely the comments whose location is hardest to recover by hand.

Fault 5, --wait widened to any issue-level comment. Only an issue-level comment does not satisfy --wait fails. That is the fault a well-meaning repair would introduce — "wait for anyone to say anything" — and the fixture that separates it is a PR thread comment with no commit_id at all.

Two cases are there for the contract rather than the bug: a review body at the current head still satisfies --wait and --wait 0 over an empty PR is success, because other briefs depend on exit 0 and exit 1 meaning what they did before. no PR number is exit 4 pins the third code, which is distinguishable from 1 only if nothing collapses them.

the usage warns about a trailing $? reader asserts a sentence rather than a behaviour, which is unusual and deliberate: the ; echo "exit=$?" trap costs a wrong diagnosis every time it is hit, the warning is the only guard against it, and a header is exactly the thing a later edit trims without noticing.

Three more cases were added after ChatGPT Codex reviewed PR #133, and the pair of bugs it found is the reason the suite has to run in CI rather than only on the author's machine. Both produce exit 2 with no output whatsoever, which is jq refusing an invalid --argjson — so a shell fault at the top of the script arrives wearing jq's exit code, and 17 of 19 cases failed identically with nothing to read.

The first, a multi-assignment local whose third clause reads its first, is fine on macOS's bash 3.2 and fatal on bash 5 under set -u. The suite passed 19/19 locally and 2/19 in CI on the same commit, which is the case for wiring it into the kb job in the same change rather than afterwards.

The second is gh api --paginate emitting one array per page. Fault 6 removes | pages_to_array from the fixture path and exactly the three new cases fail — a comment on the first page survives pagination, …the second page…, and a paginated thread still satisfies --wait (exit 2, the live signature). The fixture carries two concatenated arrays and both paths flatten through the same helper; a fixture holding one array could not have caught this, because it would exercise a shape the live fetch never produces. No PR in this repo has yet had enough inline comments to reach a second page, so the test is the only thing that will ever see it before a user does.