# Test evidence — the repository's own gates

Evidence for the checks in `scripts/` and for the tests that hold CI's shape.
The arguments behind the checkers themselves are in `context/code/tooling.md`;
this file records what fault each test was watched to catch.

Note what is deliberately absent. `scripts/check-comment-runs.py --self-test`,
`scripts/kb-test.sh` and `scripts/assemble-stamp-test.sh` are check-the-checker
suites whose cases are already injected faults, and none of their case names is
a Rust `fn` or a JS `test(` the evidence checker could resolve — so their
evidence is the case list itself plus `context/code/tooling.md`, and inventing
headings for them here would be headings naming nothing.

### "no test in web/tests/ reaches for a built bundle"

The `js` CI job runs `node --test web/tests/*.test.js` and installs nothing; the
`browser` job runs `web/tests/bundle/*.test.js` after three wasm-pack builds.
The moment a flat `web/tests/` file reads `web/pkg/`, the fast job inherits the
slow one's dependency — and the likely repair is to add a build step to it,
silently spending the ~4m30s the split exists to avoid. Until improvise-4ktt
that constraint was a directory convention and a glob, checked by nothing, and
it failed in the direction that looks like ordinary breakage: a confusing ENOENT
in CI on a machine where the bundle happens not to be built.

Fault injected, 2026-09-21: appended
`const injected = readFileSync("../pkg/improvise_wasm_client.js");` to
`web/tests/grid_plan.test.js`. The test failed naming that file and pointing at
`web/tests/bundle/`. Reverted; green again.

Two things the test does that a simpler one would not. It asserts its own
filename is among the files it scans, so it cannot become the one place a
violation may hide. And it strips comments before matching, because this file's
own header and the test's header both name the directory in prose — a lexical
grep over raw text would have had to exempt them, and an exemption is where the
next violation goes.

**What it cannot see**, stated because the check itself uses the evasion: a path
assembled at runtime (`"../" + "p" + "kg"`) is invisible to a lexical match, and
the test's own `FORBIDDEN` constant is built exactly that way to stay out of its
own scan. That is a deliberate, disclosed hole — closing it needs a parser, and
the failure it would catch is one somebody has to write on purpose, where this
gate exists for the one written by accident.

### "the subdirectory that MAY use a bundle is not the one being checked"

The companion, and the one that stops the first from passing vacuously. If
`web/tests/bundle/` were emptied or its files stopped touching the bundle, the
first test would still be green while having nothing to be scoped *against* —
"no flat test uses the bundle" is uninteresting if no test anywhere does. This
asserts the other half of the split exists and really does what the flat half
is forbidden to do.

Fault injected, 2026-09-21: moved `web/tests/bundle/wasm_surface.test.js` out of
the tree. The test failed with "web/tests/bundle/ is empty — the split lost its
other half" rather than passing. Restored; green again.

### "no test in web/tests/ imports a module that reaches the bundle"

The companion to the lexical check above, and the reason that one is not
enough: `import "../app.js"` needs the bundle *transitively* and names nothing
forbidden, so the `js` job breaks loudly but unattributed. `app.js` and
`worker.js` are the two entry points that load the bundle; `shell.js` and
`serial.js` do not, and `serial.test.js` imports one legitimately.

Fault injected, 2026-09-21: appended `import "../app.js";` to
`web/tests/grid_plan.test.js`. The lexical test stayed green; this one failed
naming the file. Reverted.

### Note — the checker's own blockers, found by review of PR #83

Two gates in `scripts/check-comment-runs.py` produced a pass while not doing
what their prose said. Both were in the mode the same PR rewrote, which is the
part worth keeping: writing the doctrine down does not exempt the writer.

**A leaf name is not an identity.** Six leaf names collide in this workspace and
two were cited in evidence. Renaming
`ui::project::tests::a_load_that_fails_is_not_a_rebase` — the one its entry
describes — left the checker printing *all name a test that runs*, exit 0, with
the heading answered by the `ui::host::tests` namesake. Now an ambiguous leaf is
refused with both candidates named, and the two entries are qualified.
Re-verified both ways, 2026-09-21: the refusal fires on the unqualified
headings and names exactly those two; with the heading qualified, the same
rename is reported as an orphan rather than absorbed.

**`--rust-tests` was strictly more permissive and said the same words.** It
called `cargo_test_names` with one argument, so the `--ignored` subtraction
never ran, and the success line was byte-identical to the honest path's.
Injecting `### eyeball_the_rendered_hints_and_pages` orphaned on the default
path and passed on the flag's. Now the flag requires
`--rust-tests-ignored` (one alone exits 2) and the success line names its
source.

**A missing evidence directory exited 0.** The early return preceded the
citation scan, so deleting `context/test-evidence/` reported nothing while 170
citations pointed at a directory that was not there. Verified 2026-09-21 by
moving the directory aside: exit 1, *170 tests cite context/test-evidence and
have no entry there*.

### Note — the flag that selected the cap away (improvise-ijql)

`scripts/check-comment-runs.py --check-evidence` early-returned in `main`, so
the flagged run never answered the comment-run cap and printed nothing about
having skipped it. The reviewer's position when it was first raised was that CI
runs all three spellings (`ci.yml`, the three `check-comment-runs.py` steps in
the `test` job) so this was documentation only. The counter that carried is that
the flagged form is the **expensive** one — it needs a compiled workspace — which
makes it the one a human runs alone; three agents hit it independently in one
campaign, and in one case a local gate went green while CI went red on a
three-line doc comment.

Reproduced before the fix, 2026-09-21, in a temporary tree holding one `.rs`
file with a four-line comment run:

```
$ check-comment-runs.py src --check-evidence --rust-tests … --rust-tests-ignored …
test-evidence: 0 entries, all name a test that runs …
EXIT=0
$ check-comment-runs.py src
1 comment runs longer than 2 lines:
  src/bad.rs:1  4 lines
EXIT=1
```

Two invocations over one tree, opposite verdicts, and the one that exits 0 is
the one a human is more likely to run by itself.

The guard is `_dispatch_self_test`, which runs the script as a **subprocess** —
the only level at which this is visible, since every unit in the file was
correct and the defect was in `main`'s dispatch. Both arms were demonstrated:

| injected fault | what failed |
|---|---|
| restore the `--check-evidence` early return | `DISPATCH: --check-evidence: exited 0 over a four-line comment run` |
| make the cap always report a violation | `DISPATCH: bare: refused a clean tree` *and* the same for `--check-evidence` |

The second arm is what stops an always-failing cap from satisfying the first,
and it is why `SELF_TEST_DISPATCH` asserts a clean tree passes as well as a
dirty one failing. Why the fix is additive rather than a printed disclaimer, and
why `--self-test`/`--list-doctests` are excluded from the table rather than
exempted from the rule: `context/code/tooling.md` §2, "A flag adds a question".

### Note — the stamp gate's discrimination, measured

`scripts/assemble-stamp-test.sh` separates three outcomes rather than two, which
is what makes it evidence rather than a smoke test. Measured 2026-09-21 over its
nine cases: the real script passes 9; deleting the final `test -z "$missing"`
(the PR #69 injection, which the inline version survived at exit 0) passes 4 and
fails 5; a script that does nothing passes 2; one that always fails passes 0.

The PR body first cited "4 of 9" for the middle row, from a run taken when the
suite had eight cases. A figure is what the next reader checks, so it is
recorded here where it can be re-measured rather than only in a message nothing
can audit (design-principles §6).

### Note — the stamp gate matches literally

`scripts/assemble-stamp.sh` greps with `-F`. As a regex, the `.` in
`improvise_wasm_client.js?v=abc` matches any character, so a stamp corrupted in
exactly that position reads as present — a pass in the one direction a guard
must not fail in. Inherited from the inline version this was extracted from,
and found by reading the extraction rather than by anything failing.

Fault injected, 2026-09-21: dropping the `-F` makes
`improvise_wasm_clientXjs?v=abc` satisfy the pattern, and
`a pattern is matched literally, not as a regex` goes from pass to fail.

### Note — the doc site's derived checks

`scripts/build-doc-site.mjs` carries two gates that no test names, because the
generator *is* the test: it runs on every assembly and exits non-zero.

Faults injected, 2026-09-21, each observed:

| injected | reported |
|---|---|
| a new `context/code/newarea.md` | `context/code/newarea.md is neither published nor a named omission` |
| `context/plan.md` moved out of the tree | `context/plan.md is listed as omitted but does not exist` |
| a line appended below the generator in `web/assemble.sh` | `web/assemble.sh must invoke this generator LAST`, naming the offending command |
| a trailing `echo "see scripts/build-doc-site.mjs"` | the same, once the check asks for an INVOCATION rather than a mention — `.includes` had accepted it (review, PR #83) |

`scripts/check-ci-graph.py` is the third, and it is a script rather than a
stanza because its own refusals need checking: adding a `zz-new-check:` job to
`ci.yml` reports *pages-deploy does not need `zz-new-check`* and exits 1, where
before the rule was prose and nothing noticed. Its five self-test graphs include
the one a naive parse gets wrong — another job's `needs:` read as the deploy's.

The second arm is the one that would not get written first, and it is the point:
without it, deleting or renaming an omitted file leaves an entry asserting a
reason for a file nobody can find — the same failure as a dead ticket id, which
`AGENTS.md` already calls worse than no id.

The first arm was filed as latent and was live when it was written:
`context/code/kb-tool.md` had been unpublished, unargued and unnoticed since it
was added.


### Duplicate evidence headings — `SELF_TEST_DUPLICATES`

improvise-hsoe. `--check-evidence` verified that a heading names a test that
**runs**, and a duplicate heading names one just as well, so both copies pass.
PR #92 shipped two entries under
`a_layout_change_moves_the_layout_version_and_no_record`, and two more pairs
sat in `tui.md` — one of them asserting that `THE_MEMBERSHIP_CORRECT_SPANS` was
"a live tripwire, do not remove it" some four hundred lines above the entry
recording that improvise-gd83 had removed it. A reader meeting the older copy
first takes it as current, which is worse than an orphan: an orphan announces
itself as broken.

The fault is injected by the six cases in `SELF_TEST_DUPLICATES`, and three of
them are the ones that would not get written first. Two *different* tests
sharing a leaf must come back **ambiguous and not duplicate**, because the
wrong diagnosis files the wrong repair. A prose heading must be free to repeat,
because it claims no test. And the bare and module-qualified spellings of one
test must collide — grouping by the string rather than by the resolved test
would have missed exactly the case improvise-rm4s's qualification repair
creates, which is the first way this would have come back.

The live tree was itself the first injection: before the three merges the check
exited 1 naming all three pairs with both line numbers; after them it exits 0.
The clean arms (two headings naming two tests, and prose repeated) are what
stop an always-refuse from satisfying the rest.

### The PR review reader's own suite — `scripts/pr-reviews-test.sh`

improvise-xgra. Like `kb-test.sh` and `pr-watch-test.sh`, its cases are already
injected faults and none of their names is a Rust `fn` the evidence checker
could resolve, so the case list is the evidence. What is recorded here is the
five faults it was watched to fail against, and which case caught each — a
suite that only asked "did it print something" passes all five.

Fault 1, **the inline surface dropped entirely**, which is the live bug as it
shipped: `inline: []` in both fetch paths. Eight cases fail, including both
`--wait` arms. This is the one that matters, because the old script's behaviour
*was* this fault and it exited 0 on PR #122's review body while printing none
of the five rulings underneath it.

Fault 2, **an inline comment counted rather than compared**:
`((.inline|length) > 0)` in place of the `commit_id == head` test. One case
fails, `an inline comment at an EARLIER head does not satisfy --wait`, and that
is the pair the two inline fixtures exist to discriminate — the at-head case
passes under this fault, so on its own it proves nothing.

Fault 3, **the time sort dropped** (`| .` for `| sort_by(.at)`). Only
`the three surfaces interleave in time order` fails: it reads the three bodies
back as `SECOND THIRD FIRST`, the per-endpoint concatenation order. Every
content assertion still passes, because each body is still printed.

Fault 4, **no `original_line` fallback**. Only the outdated-comment case fails.
`line` is null exactly when the line has moved under the comment, so a renderer
reading `.line` alone prints `draw.rs:null` on precisely the comments whose
location is hardest to recover by hand.

Fault 5, **`--wait` widened to any issue-level comment**. Only
`an issue-level comment does not satisfy --wait` fails. That is the fault a
well-meaning repair would introduce — "wait for anyone to say anything" — and
the fixture that separates it is a PR thread comment with no `commit_id` at
all.

Two cases are there for the contract rather than the bug:
`a review body at the current head still satisfies --wait` and
`--wait 0 over an empty PR is success`, because other briefs depend on exit 0
and exit 1 meaning what they did before. `no PR number is exit 4` pins the
third code, which is distinguishable from 1 only if nothing collapses them.

`the usage warns about a trailing $? reader` asserts a sentence rather than a
behaviour, which is unusual and deliberate: the `; echo "exit=$?"` trap costs a
wrong diagnosis every time it is hit, the warning is the only guard against it,
and a header is exactly the thing a later edit trims without noticing.

Three more cases were added after ChatGPT Codex reviewed PR #133, and the pair
of bugs it found is the reason the suite has to run in CI rather than only on
the author's machine. Both produce **exit 2 with no output whatsoever**, which
is jq refusing an invalid `--argjson` — so a shell fault at the top of the
script arrives wearing jq's exit code, and 17 of 19 cases failed identically
with nothing to read.

The first, a multi-assignment `local` whose third clause reads its first, is
fine on macOS's bash 3.2 and fatal on bash 5 under `set -u`. **The suite passed
19/19 locally and 2/19 in CI on the same commit**, which is the case for wiring
it into the `kb` job in the same change rather than afterwards.

The second is `gh api --paginate` emitting one array per page. Fault 6 removes
`| pages_to_array` from the fixture path and exactly the three new cases fail —
`a comment on the first page survives pagination`, `…the second page…`, and
`a paginated thread still satisfies --wait` (exit 2, the live signature). The
fixture carries two concatenated arrays and both paths flatten through the same
helper; a fixture holding one array could not have caught this, because it
would exercise a shape the live fetch never produces. No PR in this repo has
yet had enough inline comments to reach a second page, so the test is the only
thing that will ever see it before a user does.