Five findings from the second review round, and the first is this PR's own
standard applied to its own P1.
**The two fixes improvise-dj1l names had no discriminating test.** Reverting
`has_key` to unification, or dropping `canon_key` from the duplicate gate, left
kb-test at 27 passed. Case 11 was carried by `check_total` refusing with a
DIFFERENT message, and case 10 by the other half of the fix. Both now assert
which refusal fired (`said()`), plus two cases aimed at each half: a rm key with
a variable in a non-final position must be refused as ABSENT rather than
matching everything, and a keyed FACT whose key holds a variable must ADD
without excluding the siblings it would unify with. Each proved by reverting
exactly one fix and watching exactly that case fail.
**Keying a rule on its whole clause made `replace` on an edited rule silently
become `add`** -- both arms live, the stale one still firing, and `check_total`
cannot see it because `NO + NF` is correct arithmetic for an add. Reachable on
the 15 `kb_error/1` arms and the `kb_text/2` rules, so editing the check
improvise-2f97 adds is the first thing to hit it. `check_rule_replace` refuses
and names the route (rm, then add); the negative arm -- replace of a rule
present verbatim -- is tested, or the refusal would just be "replace never
works on rules".
**The `decision(` arm was dead code.** `cited_name` took the first field, and
the repo spells `decision(<ticket>, <name>)`, whose ticket id fails
`name_shaped/1` on the hyphen -- so that marker could never fire for a real
citation. A second `cited_name` clause takes the second field. All three
markers are now exercised.
Minor, from the same round: `_JS_TEST` no longer accepts `test.only`, which
disables every other test in its file, with a case asserting a heading cannot
be satisfied by one; and `node-test-test.sh`'s last case no longer duplicates
an earlier one.
Filed rather than fixed: improvise-w8ih, extracting `web/assemble.sh`'s gate so
its own absence is visible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Two findings from the PR #69 review, one class: a lexical `test(` scan is
satisfied by things that execute no assertion -- a commented-out call, the
words in a string, `test.skip`/`test.todo`, and an empty `test.describe`.
scripts/node-test.sh drops the scan and reads node's report instead. `# pass`
is 0 for a skip-only file, a todo-only file and an empty suite; and a file that
registered no test is reported as one passing test WHOSE NAME IS THE FILE PATH,
which is both how a barren file is detected and why `# tests` alone reads as
the file count. That naming convention is an assumption about node, so the
self-test pins it: a node that stops doing it fails the case rather than
letting the guard go quiet. Six cases added, including a barren file hiding
among passing ones.
--check-evidence had the parallel hole: _JS_TEST accepted `.skip`, `.todo` and
`.describe`, so a quoted heading could be satisfied by disabled coverage or by
an empty suite. Now runnable leaves only (`.only` included), with three cases
asserting that a skipped, todo or suite-only name leaves the heading orphaned
while a test nested in a suite still satisfies one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Two blind spots with one consequence: an evidence file could promise a guard
that was not there, and a test could point at prose that did not exist.
5ray: headings were collected with `startswith("### ")` and resolved against
Rust fn names only. context/test-evidence/clients.md uses `#### "name"` for
every browser test precisely BECAUSE a `###` naming a Playwright test would
have been reported as an orphan -- so the convention that kept the checker
quiet was the same one that made it blind. Now `###` and `####` alike, decided
by SHAPE: a bare identifier is a Rust test, a quoted string is a JS test
(harness/tests and web/tests), anything else is prose naming nothing.
lyyc: the check was one-directional. A test whose doc comment names
context/test-evidence must now appear there. Deliberately only that narrower
obligation -- most tests have no evidence and should not be forced to. Neither
direction can see a STALE entry; that stays discipline.
Each half found a real instance on its first run. A `#### "? opens help..."`
heading outlived the test 36ade35 deleted, and is demoted to prose with the
coverage filed as improvise-v5pc. `an_entry_that_holds_no_document_is_seeded_over`
cited the file and had no entry; one is written from the test.
The checker's --self-test covered the comment scanner and nothing else, which
is how both survived. It grows ten evidence cases.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
Adversarial review of the previous commit: scan() survived 16 literal-shape
attacks, and both HIGH findings were in the code that did NOT get rewritten.
The attribute handling still asked the RAW LINE the questions this commit's
whole argument says a raw line cannot answer.
- Bracket depth was counted over the raw line, so `#[cfg(feature = "[")]`
set attr_depth=1 with no reset path and exempted EVERY REMAINING LINE IN THE
FILE. The raw-string spelling of it is the exact defect class this rewrite
was for: scan() knew those bytes were string content and line 150 asked the
line instead. The tree has 14 `#[arg(...)]` and 3 `#[command(...)]` clap
attributes whose help strings are free text.
- `s.startswith("#[doc")` is an UNANCHORED PREFIX, so `#[doc(hidden)]` counted
as a comment and flagged correct code. Fourth instance of that class in this
campaign, eleven lines above the comment naming the previous three.
Also from the same review: `#![doc = "..."]` is the inner spelling of a doc
comment and was never counted (unlimited free crate doc); an inner attribute
between two `//!` blocks split one crate doc in half, where the rule exists to
separate a crate doc from an ITEM doc — so it now ends a run only where the doc
KIND changes; and a block comment closing on a line with code got one free line.
Attributes are now asked of the scanner: `scan()` returns the code-only text,
and depth is counted over that. `_is_attribute` is deleted — dead since the
logic moved inline, and an orphaned explanation is what --check-evidence exists
to catch one directory over.
The coverage finding was the important one. Two of scan()'s four literal
mechanisms had NO self-test case: reverting char-literal or plain-string
handling left the suite green. Both are load-bearing in opposite directions —
without char literals, `'"'` opens a string that eats the file (false negative);
without plain strings, a multi-line string holding `//` lines is a false
positive, the CI-breaking direction. Both now have cases, and so does every
shape above. The attribute-depth mechanism needed a fixture with code AFTER the
attribute, because comments inside one are counted either way: the first
fixture I wrote passed under its own injection.
Self-test: 17 violations caught, 9 legitimate shapes allowed. Every mechanism
proved discriminating by reverting it alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adversarial review demonstrated that the previous fix closed one class and
left three, then reintroduced the bug this file's own history is about.
- A ``` fence in a PLAIN `//` comment earned the doctest exemption. rustdoc
runs no test there ("running 0 tests"), so two lines exempted anything after
them. The cheapest bypass yet, and `--list-doctests` in the same file already
had it right: one of the two was wrong and it was this one.
- A block comment opened after code on the same line was invisible, since only
a line STARTING with /* was seen. That is the most natural way to write one.
- Nested block comments closed at the first */ and the rest read as code.
- And the false positive: `in_block` had no reset path, so a /* inside a raw
string swallowed the rest of the file. Latent — no .rs file starts a line
with /* today, so the block fix had zero effect on this repo — but 8 files
hold raw strings and this repo's habit is Rust fixtures inside them. The
first such fixture breaks CI with a diagnosis pointing at a string.
A line-based scanner cannot decide any of these, so `scan()` now walks
characters, tracking nested block comments, strings, raw strings (with hashes)
and char literals. `#![...]` ends a run where `#[...]` stays transparent,
because a crate doc and an item doc separated by an inner attribute are two
blocks a reader does not meet as one. `#[doc = "..."]` counts: it is the one
attribute that is a comment.
The self-test carries all of it: 12 bypasses, 6 shapes that must stay legal.
Each new fix was proved discriminating by injecting it alone.
One of those injections caught a VACUOUS test of my own: with raw-string
handling disabled, a plain-string fallback reached the same verdict, so the
fixture proved nothing. It now contains r#"x"y, which a non-raw parse closes
early — so the fault is visible. That is the defect this repo is named for,
found in the test written to prevent it.
Closes improvise-mbmf, which PR #45 claimed to close and did not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adversarial review of PR #44 constructed all three, each compiling clean with
no warnings, each passing the checker.
Block comments were invisible: the scanner only ever tested startswith("//"),
so a six-line /* */ passed — and /** */ is a DOC comment that rustdoc renders
exactly as /// does, making `/**` the cheapest way to write the long argument
the rule exists to forbid. There are zero block comments in the tree today,
which is the only reason this was unexploited; #44 is the change that gives
anyone a reason to look.
Fence state was per-FILE, so one unclosed ``` exempted every later comment in
the file. A dangling doctest would fail the build in a lib target, but doctests
do not run for bins, tests/, examples/ or harness/ — all default scan roots.
Fence state now belongs to the comment block: any non-comment line clears it.
Attributes terminated a run, so `#[inline]` between two pairs of doc lines
split one argument of four into two legal runs of two. The module docstring
had always promised attributes are "ignored"; the code disagreed with it, and
disagreed in the direction that passes.
Closing the last of those found two real violations the old checker could not
see. crates/improvise-command/src/command/cmd/text_buffer.rs carried TWO doc
comments straddling #[derive(Debug)] — both attach to the struct, so rustdoc
rendered both; a leftover from the comment wave. transport.rs had a three-line
run split by #[allow].
A checker with ten silent successes and no test of its own is the thing it
warns about, so --self-test runs every bypass this file has been fixed for
plus three shapes that must stay legal. Proved discriminating by reverting the
attribute fix: it fails and names that case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventh silent success, found by CC-command on PR #41, and the THIRD instance
in this campaign of one class: matching a substring where an exact token was
meant.
'test' in s also matched '#[cfg(test)]', so every cfg-gated fixture was accepted
as a test by --check-evidence — a gate written to reject exactly those. Two
plain fixtures cleared it: Host::model and Sessions::apply_caught_up.
The first instance was the bd dead-ID audit's unanchored exclusion list, where
a real ticket improvise-gp12 would have been silently excluded because it
starts with a crate name. The second was --check-evidence accepting any fn.
This is the third.
Now matched by exact attribute path (test, tokio::test, async_std::test,
wasm_bindgen_test). Fault-injected: a #[cfg(test)] fn under a ### heading is
reported as an orphan, a #[test] fn is not. Re-ran over the five merged units:
215 entries still pass, so none of them relied on the hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sixth silent success, and I introduced it fixing the fourth. Found by review.
Deciding 'are we inside a fence' and 'is this fence exempt' from one flag meant
a ```text block's BARE CLOSING ``` looked runnable and opened an exemption --
so prose after a non-runnable fence evaded the cap entirely. Reproduced: four
prose lines after a closed ```text block reported clean.
They are now tracked separately, and the delimiters belong to the block they
bound, so a runnable fence's closing ``` is exempt too -- otherwise it lands in
the prose run and a genuine two-line doc reads as three.
Fault-injected both directions: prose after a closed ```text block is a
violation; a runnable block with one prose line either side is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fifth silent success in this file, found by review on PR #40.
in_proptest was set on the first 'proptest!' line and NEVER CLEARED, so every
later fn in that file -- helpers included -- was accepted as a test by
--check-evidence. A file with one property test made the whole file's functions
eligible, which is exactly the hole the any-fn fix was meant to close.
Now the block's braces are tracked and the flag leaves when they balance.
Fault-injected: a helper defined after a proptest! block is reported as an
orphan, while the #[test] inside the block still passes.
Five silent successes in one small checker in one day, every one failing in the
direction that looks like passing, and none found by the person who wrote it.
The gate and the thing it gates were written by the same hand and nobody
injected a fault -- which is the wave's own finding turned on its tooling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourth silent success in the same file, and the largest.
rustdoc does not execute ```text, ```ignore or ```no_run. The exemption
treated every fence alike, so prose wearing a fence was exempt from the cap --
and 74 lines of CLI documentation in gp-tree were sitting in exactly that hole.
Measured before narrowing it: `cargo test --doc` across the whole workspace was
0 passed / 2 ignored. NOT ONE of the six fences ran. So the exemption written to
protect doctests was protecting nothing, while shielding the longest comment
blocks in the tree from the rule it was written to enforce.
Both ignore blocks had also drifted out of agreement with the code, which is
precisely what ignore cannot catch: one called fit::fit with the parallel x/y
signature improvise-dkqn removed, against a Dataset with no rows field; the
other called a rank() that has never existed. CC-gp promoted both to running
doctests, so the workspace now has 2 that actually run.
Fault-injected: a ```text block with two prose lines either side is now
reported as a run of four, while a bare ``` block is still exempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two surprises reported by agents in the same hour, neither a behaviour change.
A fence SUSPENDS the count rather than ending the run, so two prose lines above
a fence and two below is a run of four. That is deliberate -- a reader meets it
as one uninterrupted block -- but an agent capped a file to what looked like
2 and 2 and the checker correctly called it 5.
And rustdoc does not execute ```text, ```ignore or ```no_run. Two of the six
fences in this tree were ```text, so they were claiming the doctest exemption
for prose. Only a runnable fence earns it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by two wave agents inside an hour, and all three are the same class as
the bd-audit exclusion list this repo already has a doctrine about: a filter
that over-matches reports success.
1. A FILE path walked nothing and printed 'none longer than 2 lines'. roots was
filtered by os.path.isdir, so 'check-comment-runs.py src/foo.rs' silently
scanned an empty set. One agent nearly shipped a file believing it done, and
every 'my unit reports none' claim taken with a file argument was worthless.
2. A NONEXISTENT path did the same thing, so a typo'd directory read as a pass.
Both now exit 2 and name the path. Defaults are unaffected.
3. --check-evidence accepted ANY fn, so a '### heading' naming a fixture or a
helper was reported as satisfied. One unit's '68 entries, all name a test'
was passing on that accident; re-checked strictly it was 64. A heading must
now name a fn carrying #[test] (or one inside a proptest! block), which is
what the convention always claimed.
Each fix fault-injected before keeping: a file path and a missing path now
error, and a heading naming a non-test fn is reported as an orphan while a real
test passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both findings from the review on #33, and the second is the sharper one.
The checker skipped harness/opfs-contract/src/lib.rs — a shipped Rust source
tree with 14 runs over the limit, including a 22-line module doc — so it could
have reported success while the convention was violated. harness/ is now in the
default roots.
And design-principles claimed a renamed test was made detectable because 'the
checker looks for both halves'. It did not: it counted comment lines and listed
doctest fences, and nothing read context/test-evidence/ at all. So the cost the
convention says is controlled was uncontrolled, in the same commit that
introduced the convention — a reviewed paragraph is not evidence, and this one
was mine.
--check-evidence now requires every '### name' heading to name a test that
exists. One direction only: a test may have no evidence (most do not), but
evidence naming a test nobody can find is a dangling explanation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user's instruction (2026-09-20): reasoning moves out of code comments into
context/ prose, and a comment run is at most two consecutive lines — `///` and
`//!` included, and test doc-comments too.
scripts/check-comment-runs.py makes the rule checkable rather than remembered,
which is the difference between a convention and a guard. It reports 1321 runs
over the limit today; the move itself is improvise-9c8c.
A ``` fence in a doc comment is exempt: a doctest is a running test rather than
prose, and capping around one would delete it. `--list-doctests` finds the six
blocks that need individual decisions.
design-principles §7 said to record a bug in the test's doc-comment, which the
cap contradicts — that evidence now goes to context/test-evidence/ keyed by test
name, with the orphaning cost stated rather than discovered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Edward Langley