cb3afb559ed74171f94b69563e26d4d9f93749de
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Edward Langley
|
cb3afb559e |
Split a fmt usage error from a fmt verdict across both workspace roots
improvise-b4ks. From the repo root `cargo fmt -p opfs-contract -- --check`
exits 1 with "package `opfs-contract` is not a member of the workspace" and
never runs --check, so a caller reading $? cannot tell "misformatted" from
"nothing was checked". Reproduced at
|
||
|
Edward Langley
|
cb6866b2c6 |
fix(scripts): one shell and one environment for every checker (improvise-f9qf)
`scripts/pr-reviews-test.sh` passed 19/19 locally and 2/19 in CI on the same commit, because a multi-assignment `local` is not left-to-right on bash 5 while macOS's bash 3.2 tolerates it. The obvious half of the fix is pinning the shell. The half that decides the design is not. .envrc DOES already put a nix bash 5.3.15 on PATH, which is CI's semantics -- and it is not enough. `pkgs.mkShell` exports `name`, `out`, `outputs`, `system`, `shell`, `builder`, `stdenv`, `phases` and about twenty more generically-named variables that CI does not have, so a script reading one by accident under `set -u` is SILENT in the dev shell and fatal in CI. That is exactly what happened: the broken clause read `$name`, and in the dev shell `$name` was `nix-shell-env`. Measured at 051f541^ on one machine: macOS /bin/bash 3.2 19 passed, 0 failed direnv exec . ./scripts/pr-reviews-test.sh 19 passed, 0 failed the same, with `name` unset 2 passed, 17 failed Two reasons for green, one of them the correct interpreter. A dev shell is a third environment, not a reproduction of CI. A nix shebang was measured and declined, on availability rather than speed: warm, `nix shell nixpkgs#bash --command true` is 61.8ms against an ambient 5.1ms, and `nix-shell -p bash --run true` -- the manpage form -- is 675.2ms, 132x, inside git's merge machinery for beads-merge-driver.sh. But only the `kb` job installs nix; build, test, wasm, js, harness-lint and browser have none, so a nix shebang needs an installer step on three more jobs to reach a bash those runners already ship. zsh is not a shebang change and the runner does not have it either. Both declined in context/code/tooling.md section 5. What lands instead pins interpreter and environment together, in one place: scripts/checkers.sh is the one entry point for every check_the_checker, used verbatim by CI and by the local gate. It pins a bash >= 5 (ambient first, nix only when the ambient one is older, exit 3 when neither), scrubs the environment to an allowlist, and refuses any shell shebang in scripts/ that is not `#!/usr/bin/env bash`. The pinned bash goes on the front of PATH so inner invocations get it too -- which is where the fault actually lived. The second undetected split is closed with it: the four `#!/bin/sh` and one `#!/usr/bin/env sh` scripts are now bash, and the six places that ran a script through an explicit `sh` -- overriding the callee's own shebang, which is the same disease -- now invoke it directly. Verification, injected against the real script: restoring pr-reviews' multi-assignment `local` leaves `sh scripts/pr-reviews-test.sh` at 22/22 and `direnv exec . ./scripts/pr-reviews-test.sh` at 22/22, while `./scripts/checkers.sh pr-reviews` reports 2 passed, 20 failed, exit 1 -- matching the 2/19 CI actually reported. scripts/checkers-test.sh is twelve cases, each injecting rather than asserting; the pair that carries it runs one fixture both ways, so the `set -u` case is a test of the runner and not of `set -u`. The old local gate was `for t in scripts/*-test.sh; do sh "$t"; done`, which overrides every shebang with /bin/sh -- so it ran every suite under bash 3.2 in sh-mode regardless of what the file said. It is replaced by `./scripts/checkers.sh`. Workspace suite unchanged at 1689 passing over 38 binaries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
051f54111b |
fix(scripts): a local is not left-to-right on bash 5, and --paginate emits one array per page
Both found by ChatGPT Codex on PR #133, both invisible on macOS, both arriving as exit 2 with no output -- jq's answer to an invalid --argjson, so a shell fault at the top of the script wears jq's exit code. `local n="$1" name="$2" f="$DIR/$n.$name.json"` creates every name before running the assignments on bash 5, so $n is unbound in the third clause and set -u kills the shell. The dead command substitution then hands --argjson an empty string. Works on macOS bash 3.2: the suite passed 19/19 locally and 2/19 in CI on the same commit. Each local now gets its own statement. `gh api --paginate` emits one JSON array PER PAGE, so a comment thread past the page size is several concatenated documents that --argjson rejects the same way. Both fetch paths now flatten through one pages_to_array (`jq -s 'add // []'`), which is a no-op on a single page. The fixture path shares that helper, which is the only reason a fixture can test it: a fixture holding a single array would exercise a shape the live fetch never produces. Three cases added, and removing the helper from the fixture path fails exactly those three, the wait case with the live exit-2 signature. Suite is 22 cases. No PR in this repo has yet had enough inline comments to reach a second page, so nothing but the test will see that one before a user does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
a47310d50f |
fix(scripts): pr-reviews reads all three review surfaces (improvise-xgra)
It printed review BODIES only and was blind to inline review comments -- the surface a human reviewer is most likely to use. On PR #122 the user reviewed with the body "Generally good, added comments inline." and five substantive rulings underneath it; pr-reviews printed the one-liner, and an agent reported the PR as awaiting review while the answers sat on it. pulls/{n}/comments and issues/{n}/comments join pulls/{n}/reviews, all three interleaved in time order, each inline comment carrying its path and line (falling back to original_line, which is what an outdated comment has left). The exit contract is unchanged: 0 means something exists against the current head, 1 means the wait elapsed, 4 is usage. An inline comment at the head now satisfies --wait -- it carries a commit_id and is exactly the case that failed. An issue-level comment does not: it names no commit and so makes no claim about which tree was read. The usage's exit-code warning is widened from "do not pipe into head/tail" to the real rule: nothing may follow the script in the same `;` list either, because a trailing `echo "exit=$?"` makes the status the echo's. scripts/pr-reviews-test.sh is the check_the_checker, fixture-driven and needing no network, wired into CI's kb job beside pr-watch-test.sh and release-preflight-test.sh. Five faults were injected and watched to fail; context/test-evidence/tooling.md records which case caught each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
7bc26f80d5 |
Review round 1: three checks that did not check what they named
All three found by the adversarial review of
|
||
|
Edward Langley
|
383e33a1a0 |
Refuse a duplicate test-evidence heading (improvise-hsoe)
--check-evidence verified that a heading names a test that RUNS, and a duplicate names one just as well, so both copies pass. PR #92 shipped two entries under one heading; the older described a fixture the test no longer had. Refuse rather than warn, on the improvise-rm4s precedent, naming the file and every line in file order. Keyed by the RESOLVED test, so `### f` beside `### m::tests::f` collides too -- the case rm4s's qualification repair creates. Six SELF_TEST_DUPLICATES cases, including the three clean arms: two different tests sharing a leaf are ambiguous and not duplicate, prose repeats freely, and two headings naming two tests are fine. Resolves the two tui.md duplicates: the older column-header copy called THE_MEMBERSHIP_CORRECT_SPANS a live tripwire three hundred lines above the entry recording its removal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
da564dc4c1 |
Make --check-evidence additive, so no clean exit skips the comment cap
scripts/check-comment-runs.py early-returned on args.check_evidence, so the flagged run answered the evidence question and said nothing at all about the comment-run cap. The flagged form is the expensive one -- it needs a compiled workspace -- which makes it the one a human runs alone; three agents hit this independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment (improvise-ijql). The cap moves into check_cap(), and --check-evidence now runs it first and returns the worse of the two verdicts. The ticket's cheaper alternative -- keep the early return and print a line saying the cap did not run -- is declined and the reason is recorded: a provenance line qualifies an answer the run gave, where a disclaimer qualifies one it did not, while still exiting 0. Exit 0 is what CI branches on and what a human reads as clean, so the rule is stated over the exit code: a tree with a long comment run must not pass any invocation that can exit 0. The guard is _dispatch_self_test, which runs the script as a subprocess against a temporary tree -- the only level at which this is visible, since every unit in the file was correct and the defect was in main()'s dispatch. Both arms were demonstrated by injection: restoring the early return fails "DISPATCH: --check-evidence: exited 0 over a four-line comment run", and forcing the cap to always report a violation fails the clean arm for both invocations, which is what stops an always-fail from satisfying the first. Measured: the cap adds 0.29s to a 6.5s --check-evidence run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
3542d6f717 |
fix(tooling): a leaf is not an identity, and a convenience flag was a bypass
Both blockers from the adversarial review of #83, both in the mode this PR rewrote, both producing a pass while the gate did not do what its prose said. 1. cargo_test_names reduced every listed path to its last segment. Six leaf names collide in this workspace and two are cited in evidence, so a heading was satisfiable by a test in another module answering another question -- renaming ui::project::tests::a_load_that_fails_is_not_a_rebase, the one its entry describes, left the checker at exit 0 with the ui::host namesake answering for it. Paths are kept whole; a bare leaf resolves only when unique, and an ambiguous one is REFUSED with both candidates named. _RUST_NAME accepts `::`, so a qualified heading is a claim rather than prose, and a citation is matched on the leaf so qualifying a heading cannot orphan its own test. The two colliding entries are qualified. Neither half suffices alone: rename one of a colliding pair and the leaf becomes unique again, which is why qualification is the fix and the refusal is what forces it. 2. --rust-tests called cargo_test_names with one argument, so the --ignored subtraction never ran, and it printed a byte-identical success line. It now requires --rust-tests-ignored (one alone exits 2) and the line names its source. Also, from the same review: a missing context/test-evidence/ exited 0 while 170 citations pointed at nothing (early return, now gone); the doc-site "runs last" check accepted a trailing `echo` that merely NAMED the generator, so it asks for an invocation; bundle_freedom gains an arm for a transitive `import "../app.js"`, which reaches the bundle while naming nothing forbidden; and two prose overclaims are corrected -- the prose-heading count is a convenience rather than a guard (git diff is the mechanism), and the ratio rule says "above the run MEDIAN", which is defensible where "above the others" is false against second place. scripts/check-ci-graph.py makes claim 6's rule a gate: pages-deploy must `needs` every other job in ci.yml. The rule was stated in the last commit and a zz-new-check: job added during review was still ungated with nothing noticing. README now states the rule and lists no jobs, the second copy being what went stale twice. Every fault re-injected and observed; figures in context/test-evidence/tooling.md rather than only here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
1d4d9107cf |
fix(tooling): three findings from reviewing this branch against itself
1. A self-test case that was a duplicate wearing different clothes. "a heading cargo's listing does not carry is an orphan however the source reads" passed on an empty listing -- which is the first case in the table, already. Once defined comes from cargo, the Rust beside a fixture is not read at all, so four cases for the four source shapes would be four copies of one assertion dressed as coverage. Deleted; the evidence for those shapes is the real-tree injection, and context/code/tooling.md now says which half a fixture can carry and which it cannot. This is exactly the shape improvise-84wp's prose, applied two commits ago, tells a reviewer to look for. 2. assemble-stamp.sh grepped its patterns as REGEXES, inherited from the inline version it was extracted from. The `.` in improvise_wasm_client.js?v=abc matches any character, so a stamp corrupted in that position read as present -- a pass in the one direction a guard must not fail in. grep -F, plus a case proved by injection: dropping the -F flips it. 3. A missing cargo raised a traceback out of --check-evidence. It now names the flag that takes a saved listing instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
1c3e1bd968 |
feat(doc-site): the published set is derived, and the generator must run last (improvise-res9)
The manifest only ever grows, so a doc added without a line in it published nothing and said nothing: build green, link check green, doc simply absent. The generator now walks context/**/*.md plus docs/design-notes.md, subtracts the manifest, and fails unless the remainder is exactly the OMITTED map -- checked in BOTH directions, because an entry naming a file nobody can find asserts a reason for nothing. Filed as latent; it was live. context/code/kb-tool.md had been unpublished, unargued and unnoticed since it was added, while doc-site.md's own table says context/code/*.md is published. It is published now, with context/code/tooling.md and context/test-evidence/tooling.md: 23 pages to 26. Second gate, same file: the generator reads web/assemble.sh and refuses unless its own invocation is that script's last command. Its outward-link check resolves ../index.html and ../app/ against the filesystem, so a line appended below it would make that check pass against a half-built tree. Both run at module scope, before any page is generated, so a manifest error is answered before the 48 outward-link failures a bare run reports -- a missing doc is actionable wherever you ran the generator, and the two are unrelated diagnoses. Three faults injected and observed; evidence in context/test-evidence/tooling.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |
||
|
Edward Langley
|
8af37d44a1 |
feat(ci): two conventions become gates (improvise-w8ih, improvise-4ktt)
The stamp gate is scripts/assemble-stamp.sh, taking FILE PATTERN pairs, and scripts/assemble-stamp-test.sh is its check_the_checker. Deleting the final `test -z "$missing"` makes four of its eight cases fail; before the extraction that same deletion was invisible, both misses printing while assemble.sh exited 0. It is a separate script because exercising it inside assemble.sh would need a fabricated web/pkg, which is a second answer to "what does the site assemble from" (improvise-vvl3). Wired into the `build` job, first, because it needs nothing and guards a step in that job. web/tests/bundle_freedom.test.js refuses a flat web/tests/ file that names the bundle directory -- the constraint keeping the `js` job at seconds rather than ~4m30s, stated in prose and checked by nothing. It scans its own filename too, so it cannot become the place a violation hides, and its companion asserts web/tests/bundle/ still holds a test that DOES use the bundle: a scoping rule is uninteresting if nothing falls outside it. Both faults were injected and watched to fail; evidence in context/test-evidence/tooling.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH |