Make --check-evidence additive, so no clean exit skips the comment cap

scripts/check-comment-runs.py early-returned on args.check_evidence, so the
flagged run answered the evidence question and said nothing at all about the
comment-run cap. The flagged form is the expensive one -- it needs a compiled
workspace -- which makes it the one a human runs alone; three agents hit this
independently in one campaign, and in one case a local gate went green while
CI went red on a three-line doc comment (improvise-ijql).

The cap moves into check_cap(), and --check-evidence now runs it first and
returns the worse of the two verdicts. The ticket's cheaper alternative -- keep
the early return and print a line saying the cap did not run -- is declined and
the reason is recorded: a provenance line qualifies an answer the run gave,
where a disclaimer qualifies one it did not, while still exiting 0. Exit 0 is
what CI branches on and what a human reads as clean, so the rule is stated over
the exit code: a tree with a long comment run must not pass any invocation that
can exit 0.

The guard is _dispatch_self_test, which runs the script as a subprocess against
a temporary tree -- the only level at which this is visible, since every unit in
the file was correct and the defect was in main()'s dispatch. Both arms were
demonstrated by injection: restoring the early return fails
"DISPATCH: --check-evidence: exited 0 over a four-line comment run", and forcing
the cap to always report a violation fails the clean arm for both invocations,
which is what stops an always-fail from satisfying the first.

Measured: the cap adds 0.29s to a 6.5s --check-evidence run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 22:14:39 -07:00
parent bc35ac51c1
commit da564dc4c1
3 changed files with 155 additions and 18 deletions
+36
View File
@@ -154,6 +154,42 @@ number no one is watching. Saying otherwise would claim a detectability it
cannot support, which is the class of defect §7's adjacent-check section is
about.
### A flag adds a question; it never selects one away
`--check-evidence` used to **early-return**, so the flagged run answered the
evidence question and said nothing at all about the comment-run cap. Three
agents hit that independently in one campaign, and in one case a local gate went
green while CI went red on a three-line doc comment (improvise-ijql). It is now
additive: `main` runs `check_cap` first, then `check_evidence`, and returns the
worse of the two.
The ticket offered a cheaper shape — leave the early return and print one line
saying the cap did not run — on the precedent of PR #83 making `--rust-tests`
name its source. That precedent does not reach this case, and the distinction is
worth keeping. **A provenance line and a disclaimer are not the same object.**
Naming which listing the names came from qualifies an answer the run actually
gave; saying "the cap did not run" qualifies an answer it did not give, while
still **exiting 0**. Exit 0 is what CI branches on and what a human reads as
clean, so a disclaimer leaves the gate exactly as skippable and merely documents
the skip. The rule the fix is written to is therefore stated over the exit code:
**a tree with a long comment run must not pass any invocation that can exit 0.**
Which invocations those are is a decision, not a sweep. `--self-test` and
`--list-doctests` are excluded and neither is an exception to the rule: the
first scans no tree, and the second is a *listing* that renders no verdict. Only
the two that answer "is this tree clean" are in `SELF_TEST_DISPATCH`.
That table is checked by running the script as a **subprocess** against a
temporary tree, which is the only level at which this bug is visible — every
unit in this file was correct, and the defect lived in `main`'s dispatch. Both
arms are needed and both were demonstrated: restoring the early return fails
`DISPATCH: --check-evidence: exited 0 over a four-line comment run`, and forcing
the cap to always report a violation fails the clean arm for both invocations,
which is what stops an always-fail from satisfying the first. The temporary tree
also has no `context/test-evidence/` and no JS roots, so the evidence half is
vacuously clean there and the cap is the only thing that can decide the exit
code — the isolation is what makes the assertion name one question.
## 3. The others
`scripts/kb` is the only writer of `prolog-sessions/kb.pl`; its own reasoning is