Edward Langley
8af37d44a1
The stamp gate is scripts/assemble-stamp.sh, taking FILE PATTERN pairs, and scripts/assemble-stamp-test.sh is its check_the_checker. Deleting the final `test -z "$missing"` makes four of its eight cases fail; before the extraction that same deletion was invisible, both misses printing while assemble.sh exited 0. It is a separate script because exercising it inside assemble.sh would need a fabricated web/pkg, which is a second answer to "what does the site assemble from" (improvise-vvl3). Wired into the `build` job, first, because it needs nothing and guards a step in that job. web/tests/bundle_freedom.test.js refuses a flat web/tests/ file that names the bundle directory -- the constraint keeping the `js` job at seconds rather than ~4m30s, stated in prose and checked by nothing. It scans its own filename too, so it cannot become the place a violation hides, and its companion asserts web/tests/bundle/ still holds a test that DOES use the bundle: a scoping rule is uninteresting if nothing falls outside it. Both faults were injected and watched to fail; evidence in context/test-evidence/tooling.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
2.7 KiB
Test evidence — the repository's own gates
Evidence for the checks in scripts/ and for the tests that hold CI's shape.
The arguments behind the checkers themselves are in context/code/tooling.md;
this file records what fault each test was watched to catch.
Note what is deliberately absent. scripts/check-comment-runs.py --self-test,
scripts/kb-test.sh and scripts/assemble-stamp-test.sh are check-the-checker
suites whose cases are already injected faults, and none of their case names is
a Rust fn or a JS test( the evidence checker could resolve — so their
evidence is the case list itself plus context/code/tooling.md, and inventing
headings for them here would be headings naming nothing.
"no test in web/tests/ reaches for a built bundle"
The js CI job runs node --test web/tests/*.test.js and installs nothing; the
browser job runs web/tests/bundle/*.test.js after three wasm-pack builds.
The moment a flat web/tests/ file reads web/pkg/, the fast job inherits the
slow one's dependency — and the likely repair is to add a build step to it,
silently spending the ~4m30s the split exists to avoid. Until improvise-4ktt
that constraint was a directory convention and a glob, checked by nothing, and
it failed in the direction that looks like ordinary breakage: a confusing ENOENT
in CI on a machine where the bundle happens not to be built.
Fault injected, 2026-09-21: appended
const injected = readFileSync("../pkg/improvise_wasm_client.js"); to
web/tests/grid_plan.test.js. The test failed naming that file and pointing at
web/tests/bundle/. Reverted; green again.
Two things the test does that a simpler one would not. It asserts its own filename is among the files it scans, so it cannot become the one place a violation may hide. And it strips comments before matching, because this file's own header and the test's header both name the directory in prose — a lexical grep over raw text would have had to exempt them, and an exemption is where the next violation goes.
"the subdirectory that MAY use a bundle is not the one being checked"
The companion, and the one that stops the first from passing vacuously. If
web/tests/bundle/ were emptied or its files stopped touching the bundle, the
first test would still be green while having nothing to be scoped against —
"no flat test uses the bundle" is uninteresting if no test anywhere does. This
asserts the other half of the split exists and really does what the flat half
is forbidden to do.
Fault injected, 2026-09-21: moved web/tests/bundle/wasm_surface.test.js out of
the tree. The test failed with "web/tests/bundle/ is empty — the split lost its
other half" rather than passing. Restored; green again.