improvise/context/code/tooling.md

The repository's own checkers

scripts/ holds the gates this repo runs on itself. They are code, and they are held to the rule they exist to enforce: a check nobody has seen fail is indistinguishable from one that always passes, so each of them carries a self-test whose cases are injected faults (doctrine(check_the_checker)).

Four docstrings in scripts/check-comment-runs.py point here. This is the file they mean.

---

1. The comment cap

check-comment-runs.py fails a .rs file carrying more than two consecutive comment lines. The rule is context/design-principles.md §6: reasoning lives in context/ prose, where it can be read, corrected and audited, not in a comment that nothing checks against the code.

The whole difficulty is deciding what is a comment line. Ten silent successes have been found in this one function, every one of them failing in the direction that looks like passing — a file that should have been flagged and was not:

• "test" in line matched #[cfg(test)], so every cfg-gated fixture cleared a gate written to reject exactly those. Anchor the match; an unanchored substring is a filter that over-matches, and a filter that over-matches fails in the direction that looks like success. (The same class as the bd ID audit's exclusion list, and as _JS_TEST's exclusion of .skip/.only.)
• A proptest! flag was set and never cleared, so the first proptest! in a file made every later fn — helpers included — count as a test.
• An attribute holding a bracket inside a string (#[cfg(feature = "[")]) left the bracket depth unbalanced and exempted the rest of the file. Every question is therefore asked of the scanner, never of the raw line: the scanner knows Rust strings, raw strings, char literals and nested block comments, and the attribute depth is counted over code text only.
• A fence that opens and never closes exempted everything after it.

Two rules that are decisions rather than accidents. A fence **suspends** the run rather than ending it, so two lines above a fence and two below is a run of four — a reader meets it as one uninterrupted block. And only a *runnable* fence earns the exemption: rustdoc executes a bare or rust and does not execute text, ignore or no_run, so the rest is prose wearing a fence.

2. The evidence check

--check-evidence holds context/test-evidence/ and the tests it describes to each other, in two directions:

• Orphans. Every heading shaped like a test name must name a test that runs.
• Unstated. Every test whose doc comment cites context/test-evidence/ must have an entry there.

Neither direction is "every test has evidence". Most tests have none and should not be forced to have one; what is checkable is a claim, and both directions are claims. The failure they close is the same one a dead ticket ID is: a pointer that reads as a promise the thing exists.

A leaf is not an identity

The names cargo lists are module paths, and the checker keeps them whole. Six leaf names collide in this workspace — two of them cited in evidence — so reducing a path to its last segment made a heading satisfiable by a test in another module answering another question. Renaming ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry actually describes, left the checker at 579 entries, all name a test that runs, exit 0: the surviving ui::host::tests namesake answered for it.

So a bare leaf resolves only when it is unique, and an ambiguous one is a refusal naming the candidates — not a warning, because a warning re-creates the problem for anyone who does not read it. The heading is then written with its module path, which is also what makes the entry say which test it is about.

Note the two halves are both needed and neither is sufficient. The refusal forces qualification but cannot survive a rename on its own: rename one of a colliding pair and the leaf becomes unique, silently resolving to the survivor. Qualification is what closes that, and the refusal is what makes somebody write it.

The names come from cargo, not from the source

A heading is diffed against cargo test --workspace -- --list, minus the names that --list --ignored reports. It used to be scraped out of .rs text, and four shapes defeated that, all of them "evidence for a check that never runs" (improvise-rm4s):

shape	why the scrape accepted it
#[test] fn x inside a raw string literal	the scanner reads inside strings
#[cfg(feature = "…")] #[test]	the feature is never enabled
#[cfg(target_arch = "wasm32")] #[test]	cargo test --workspace is native
#[test] #[ignore]	it compiles, and never runs

The third is the realistic one here: this repo has real wasm32-only code and a wasm CI job that only runs cargo check. Adopting the stricter source cost nothing — all 539 headings resolved against a real listing on the commit that made the change — which is the moment to tighten a gate rather than the moment to argue about it.

Demonstrated before the fix, by injecting all four shapes into src/draw.rs and context/test-evidence/tui.md at once: the previous checker printed 543 entries, all name a test that exists, exit 0. The current one names all four.

What this costs: --check-evidence now needs a toolchain and a compiled workspace, about three seconds warm. A missing cargo says so rather than raising. The comment-cap check is unaffected and needs neither.

--rust-tests FILE takes a saved listing for a caller that has one, and it requires --rust-tests-ignored FILE: a first version took one file, called cargo_test_names with one argument, and so skipped the ignored rule entirely while printing the same success line as the honest path. A flag that is strictly more permissive and indistinguishable in the log is a bypass, not a convenience — so the pair is mandatory and the success line names its source. CI passes neither and runs cargo itself.

A missing context/test-evidence/ is not an early return either. It used to be, and it exited 0 while 170 tests cited a directory that was not there; the walk now yields nothing and every citation is reported unstated, which is loud.

Note what the self-test can and cannot carry here. The listing reader is testable from fixtures — a leaf name taken off a module path, an --ignored name subtracted, cargo's own prose not mistaken for a test — and those cases are in SELF_TEST_LISTING. The four source shapes are not: once defined comes from cargo, a fixture asserting "a heading with an empty listing is an orphan" is the same case however the Rust beside it is spelled, and writing four of them would be four copies of one assertion dressed as coverage. The evidence for those is the injection above, against the real tree.

What it does not close: citing is still scraped, so a citation inside a string literal would be reported as a test owing an entry. That is a false positive in the safe direction — loud, not silent — and it has no instance.

A demotion is a reviewer's job, not a gate's

A heading is classified by shape: a bare identifier is a Rust test, a quoted string a JS test, anything else is prose about the section (Helper — x, Note — y, a file's own scope). So any orphaned heading can be retired by rewriting it with a space and a dash, and nothing can check whether the prose is honest — no mechanism could, because the question is whether a sentence is true (improvise-9za9).

Two things make that tolerable. The unsafe direction is the one that fails loudly: prose that looks like an identifier is classified as a Rust claim and reported as an orphan, so the error you can make by accident is the noisy one. And the mechanism is git diff context/test-evidence/ — a demotion shows up there as a ### identifier line becoming prose, in the diff every reviewer already reads. The live instance, PR #69 demoting a heading while PR #72 restored its coverage under another name, is what says a reviewer rather than a gate has to catch it.

The printed count of prose headings is a convenience, not a guard: nothing stores the previous count and nobody diffs two CI logs, so a retirement moves a number no one is watching. Saying otherwise would claim a detectability it cannot support, which is the class of defect §7's adjacent-check section is about.

3. The others

scripts/kb is the only writer of prolog-sessions/kb.pl; its own reasoning is in context/code/kb-tool.md. scripts/node-test.sh, scripts/kb-test.sh, scripts/pr-watch-test.sh and scripts/assemble-stamp-test.sh are the same shape as this file's subject: a guard, plus a suite that makes the guard refuse what it claims to refuse. Each is wired into .github/workflows/ci.yml in the same job as the thing it guards, because a check that only runs locally is a convention.

scripts/check-ci-graph.py is the workflow's own gate: pages-deploy must needs every other job in ci.yml. The rule "every check job" was written down (improvise-x7cq) and a job added during review of PR #83 was still ungated with nothing noticing, which is what moved it from a sentence to a check. It reads the job keys and the needs: list off indentation, tracking which job a needs: belongs to — taking the first one seen would read another job's list as the deploy's and pass. README.md therefore states the rule and deliberately does not list the jobs: a second copy with nothing holding it to the first is what went stale, twice.