Adversarial review demonstrated that the previous fix closed one class and
left three, then reintroduced the bug this file's own history is about.
- A ``` fence in a PLAIN `//` comment earned the doctest exemption. rustdoc
runs no test there ("running 0 tests"), so two lines exempted anything after
them. The cheapest bypass yet, and `--list-doctests` in the same file already
had it right: one of the two was wrong and it was this one.
- A block comment opened after code on the same line was invisible, since only
a line STARTING with /* was seen. That is the most natural way to write one.
- Nested block comments closed at the first */ and the rest read as code.
- And the false positive: `in_block` had no reset path, so a /* inside a raw
string swallowed the rest of the file. Latent — no .rs file starts a line
with /* today, so the block fix had zero effect on this repo — but 8 files
hold raw strings and this repo's habit is Rust fixtures inside them. The
first such fixture breaks CI with a diagnosis pointing at a string.
A line-based scanner cannot decide any of these, so `scan()` now walks
characters, tracking nested block comments, strings, raw strings (with hashes)
and char literals. `#![...]` ends a run where `#[...]` stays transparent,
because a crate doc and an item doc separated by an inner attribute are two
blocks a reader does not meet as one. `#[doc = "..."]` counts: it is the one
attribute that is a comment.
The self-test carries all of it: 12 bypasses, 6 shapes that must stay legal.
Each new fix was proved discriminating by injecting it alone.
One of those injections caught a VACUOUS test of my own: with raw-string
handling disabled, a plain-string fallback reached the same verdict, so the
fixture proved nothing. It now contains r#"x"y, which a non-raw parse closes
early — so the fault is visible. That is the defect this repo is named for,
found in the test written to prevent it.
Closes improvise-mbmf, which PR #45 claimed to close and did not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adversarial review of PR #44 constructed all three, each compiling clean with
no warnings, each passing the checker.
Block comments were invisible: the scanner only ever tested startswith("//"),
so a six-line /* */ passed — and /** */ is a DOC comment that rustdoc renders
exactly as /// does, making `/**` the cheapest way to write the long argument
the rule exists to forbid. There are zero block comments in the tree today,
which is the only reason this was unexploited; #44 is the change that gives
anyone a reason to look.
Fence state was per-FILE, so one unclosed ``` exempted every later comment in
the file. A dangling doctest would fail the build in a lib target, but doctests
do not run for bins, tests/, examples/ or harness/ — all default scan roots.
Fence state now belongs to the comment block: any non-comment line clears it.
Attributes terminated a run, so `#[inline]` between two pairs of doc lines
split one argument of four into two legal runs of two. The module docstring
had always promised attributes are "ignored"; the code disagreed with it, and
disagreed in the direction that passes.
Closing the last of those found two real violations the old checker could not
see. crates/improvise-command/src/command/cmd/text_buffer.rs carried TWO doc
comments straddling #[derive(Debug)] — both attach to the struct, so rustdoc
rendered both; a leftover from the comment wave. transport.rs had a three-line
run split by #[allow].
A checker with ten silent successes and no test of its own is the thing it
warns about, so --self-test runs every bypass this file has been fixed for
plus three shapes that must stay legal. Proved discriminating by reverting the
attribute fix: it fails and names that case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventh silent success, found by CC-command on PR #41, and the THIRD instance
in this campaign of one class: matching a substring where an exact token was
meant.
'test' in s also matched '#[cfg(test)]', so every cfg-gated fixture was accepted
as a test by --check-evidence — a gate written to reject exactly those. Two
plain fixtures cleared it: Host::model and Sessions::apply_caught_up.
The first instance was the bd dead-ID audit's unanchored exclusion list, where
a real ticket improvise-gp12 would have been silently excluded because it
starts with a crate name. The second was --check-evidence accepting any fn.
This is the third.
Now matched by exact attribute path (test, tokio::test, async_std::test,
wasm_bindgen_test). Fault-injected: a #[cfg(test)] fn under a ### heading is
reported as an orphan, a #[test] fn is not. Re-ran over the five merged units:
215 entries still pass, so none of them relied on the hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sixth silent success, and I introduced it fixing the fourth. Found by review.
Deciding 'are we inside a fence' and 'is this fence exempt' from one flag meant
a ```text block's BARE CLOSING ``` looked runnable and opened an exemption --
so prose after a non-runnable fence evaded the cap entirely. Reproduced: four
prose lines after a closed ```text block reported clean.
They are now tracked separately, and the delimiters belong to the block they
bound, so a runnable fence's closing ``` is exempt too -- otherwise it lands in
the prose run and a genuine two-line doc reads as three.
Fault-injected both directions: prose after a closed ```text block is a
violation; a runnable block with one prose line either side is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fifth silent success in this file, found by review on PR #40.
in_proptest was set on the first 'proptest!' line and NEVER CLEARED, so every
later fn in that file -- helpers included -- was accepted as a test by
--check-evidence. A file with one property test made the whole file's functions
eligible, which is exactly the hole the any-fn fix was meant to close.
Now the block's braces are tracked and the flag leaves when they balance.
Fault-injected: a helper defined after a proptest! block is reported as an
orphan, while the #[test] inside the block still passes.
Five silent successes in one small checker in one day, every one failing in the
direction that looks like passing, and none found by the person who wrote it.
The gate and the thing it gates were written by the same hand and nobody
injected a fault -- which is the wave's own finding turned on its tooling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourth silent success in the same file, and the largest.
rustdoc does not execute ```text, ```ignore or ```no_run. The exemption
treated every fence alike, so prose wearing a fence was exempt from the cap --
and 74 lines of CLI documentation in gp-tree were sitting in exactly that hole.
Measured before narrowing it: `cargo test --doc` across the whole workspace was
0 passed / 2 ignored. NOT ONE of the six fences ran. So the exemption written to
protect doctests was protecting nothing, while shielding the longest comment
blocks in the tree from the rule it was written to enforce.
Both ignore blocks had also drifted out of agreement with the code, which is
precisely what ignore cannot catch: one called fit::fit with the parallel x/y
signature improvise-dkqn removed, against a Dataset with no rows field; the
other called a rank() that has never existed. CC-gp promoted both to running
doctests, so the workspace now has 2 that actually run.
Fault-injected: a ```text block with two prose lines either side is now
reported as a run of four, while a bare ``` block is still exempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two surprises reported by agents in the same hour, neither a behaviour change.
A fence SUSPENDS the count rather than ending the run, so two prose lines above
a fence and two below is a run of four. That is deliberate -- a reader meets it
as one uninterrupted block -- but an agent capped a file to what looked like
2 and 2 and the checker correctly called it 5.
And rustdoc does not execute ```text, ```ignore or ```no_run. Two of the six
fences in this tree were ```text, so they were claiming the doctest exemption
for prose. Only a runnable fence earns it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by two wave agents inside an hour, and all three are the same class as
the bd-audit exclusion list this repo already has a doctrine about: a filter
that over-matches reports success.
1. A FILE path walked nothing and printed 'none longer than 2 lines'. roots was
filtered by os.path.isdir, so 'check-comment-runs.py src/foo.rs' silently
scanned an empty set. One agent nearly shipped a file believing it done, and
every 'my unit reports none' claim taken with a file argument was worthless.
2. A NONEXISTENT path did the same thing, so a typo'd directory read as a pass.
Both now exit 2 and name the path. Defaults are unaffected.
3. --check-evidence accepted ANY fn, so a '### heading' naming a fixture or a
helper was reported as satisfied. One unit's '68 entries, all name a test'
was passing on that accident; re-checked strictly it was 64. A heading must
now name a fn carrying #[test] (or one inside a proptest! block), which is
what the convention always claimed.
Each fix fault-injected before keeping: a file path and a missing path now
error, and a heading naming a non-test fn is reported as an orphan while a real
test passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both findings from the review on #33, and the second is the sharper one.
The checker skipped harness/opfs-contract/src/lib.rs — a shipped Rust source
tree with 14 runs over the limit, including a 22-line module doc — so it could
have reported success while the convention was violated. harness/ is now in the
default roots.
And design-principles claimed a renamed test was made detectable because 'the
checker looks for both halves'. It did not: it counted comment lines and listed
doctest fences, and nothing read context/test-evidence/ at all. So the cost the
convention says is controlled was uncontrolled, in the same commit that
introduced the convention — a reviewed paragraph is not evidence, and this one
was mine.
--check-evidence now requires every '### name' heading to name a test that
exists. One direction only: a test may have no evidence (most do not), but
evidence naming a test nobody can find is a dangling explanation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The user's instruction (2026-09-20): reasoning moves out of code comments into
context/ prose, and a comment run is at most two consecutive lines — `///` and
`//!` included, and test doc-comments too.
scripts/check-comment-runs.py makes the rule checkable rather than remembered,
which is the difference between a convention and a guard. It reports 1321 runs
over the limit today; the move itself is improvise-9c8c.
A ``` fence in a doc comment is exempt: a doctest is a running test rather than
prose, and capping around one would delete it. `--list-doctests` finds the six
blocks that need individual decisions.
design-principles §7 said to record a bug in the test's doc-comment, which the
cap contradicts — that evidence now goes to context/test-evidence/ keyed by test
name, with the orphaning cost stated rather than discovered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two findings from the review of #7.
The hook piped an unpinned installer from beads' `main` straight into a
shell. That runs code nobody has seen, from a URL whose contents can
change under us, and leaves nothing to inspect when something goes wrong.
It now downloads first and runs the file, so /tmp/bd-install.sh is
readable afterwards, and pins the revision: the peeled commit of tag
v1.3.0, which is the version this work was done against. The comment
carries the ls-remote incantation for bumping it. Byte-identical to
`main` today, so the pin costs nothing now and is the whole point later.
The seeding script read its input with `mapfile < <(jq …)`, and a process
substitution discards the exit status of what ran inside it. A malformed
or unreadable flake.lock therefore produced an empty list, which the next
branch reported as "no github-sourced inputs; nothing to seed" and exited
0 — the script would claim success having seeded nothing. jq's output is
now captured with the status checked, and emptiness is a separate case
from failure.
Verified by exercising the paths rather than reading them:
malformed flake.lock -> error names the file, rc=1
valid lock, no github inputs -> "nothing to seed", rc=0
missing flake.lock -> error, rc=1
installer download -> /tmp/bd-install.sh matches f45b249 byte for byte
shellcheck -S style clean on both scripts; the real seeding run still
completes with flake.lock unmodified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DyHK7L8S1XDX3EihTnSKaX
Nix's `github:` fetcher wants a tarball from
github.com/<owner>/<repo>/archive/<rev>.tar.gz, and that endpoint is not
served in a cloud session. Three things that look like fixes are not:
- Raising the environment's network access to Full, or allowlisting
codeload.github.com, changes nothing: GitHub traffic bypasses the
domain allowlist entirely and goes through a proxy that scopes API
and release-asset requests to repositories attached to the session.
An input from any other owner gets a 403.
- Attaching the dependency repos is refused -- cross-tier adds are not
supported once the session has sources from another owner.
- Nix has no setting that makes `github:` inputs use the git transport.
The lock's own integrity machinery is what works. flake.lock records a
narHash per input, and Nix looks a source up in the store and its
substituters before fetching anything. The git and tarball fetchers
produce byte-identical trees for a revision, so prefetching the locked
rev over git+https lands the input at exactly the store path the lock
names:
lock narHash: sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=
git prefetch: sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=
scripts/nix-seed-flake-inputs.sh does that for every locked input. It
reads flake.lock and never writes it. Nix still verifies each tree
against the recorded hash, so this satisfies the lock rather than working
around it -- a mismatch fails loudly. Rewriting the lock's nodes to `git`
type and carrying the old narHash across would have been the opposite:
that hash is the claim the lock exists to make.
Verified from a fresh clone with a cleared git cache: 15/15 inputs
seeded, then `nix develop` gives rustc 1.98.1, cargo and crate2nix, with
flake.nix and flake.lock unmodified.
Not wired into the session-start hook: it costs minutes and several GB
once the devShell builds, and most work only needs the preinstalled cargo
toolchain.
Corrects three places that claimed nix could not work here --
context/cloud-agents.md, the design-review skill's kb.pl check, and the
hook's environment notes -- all written before the git transport was
tested. The doc also records the two alternatives, a binary-cache
substituter and a self-hosted environment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DyHK7L8S1XDX3EihTnSKaX
Add demo recording tools (asciinema, vhs) and release tooling
(cargo-dist) to nativeBuildInputs. Include a cargo-dist wrapper so
`cargo dist` works as a subcommand. Add scripts/record-demo.sh for
consistent asciinema cast recording at 100x30.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Edward Langley
Claude