tooling: check-ci-graph asks checkers.sh what a step runs (improvise-maax)

The #228 review and Codex's P1 found the suite check failing open. A bare
scripts/checkers.sh after an inline comment, in a step name or as echo's
argument read as "every suite runs". --list alone read as every suite, and
--self-test's discarded names, and --list's, counted as run.

checkers.sh gains --would-run, honoured only as the first argument. It
reports what the rest of the arguments would run, from the real argument
loop and name resolution. check-ci-graph.py only finds invocations: in run:
text, comments stripped by bash's rule, command position, every command on
the line, quotes removed. It then asks --would-run about each. An
unparseable line contributes nothing and fails closed.

16 suite self-test cases, 8 of which the old parser gets wrong. Six
rivals each fail a case only they can. checkers-test.sh gains four
--would-run cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Edward Langley
2026-09-23 08:36:13 -07:00
parent af61e1bf8d
commit c141735977
5 changed files with 287 additions and 68 deletions
+17 -5
View File
@@ -264,11 +264,23 @@ It holds a second rule of the same kind: every suite `scripts/checkers.sh
suites by hand, because they need different jobs and environments (`kb` runs
under a nix trealla, `node-test` in the `js` job), so a single bare
`checkers.sh` was not an option. The check does not re-derive what a suite is.
It asks `checkers.sh --list`, so the naming rule keeps its one home. It then
reads every `checkers.sh` invocation in the workflow. A bare one covers every
suite, and `--self-test` covers `checkers`. A YAML comment is skipped, because a
bare mention there would otherwise read as "everything runs", which is the
failure that looks like success.
It asks `checkers.sh --list`, so the naming rule keeps its one home.
**Nor does it re-derive what an invocation runs** (improvise-maax). The first
version did, and got `checkers.sh`'s argument rules wrong in the direction that
passes. It read `--list` as "run everything", when `--list` runs nothing. It
counted the names after `--self-test`, which `checkers.sh` throws away. And it
read any mention of `scripts/checkers.sh` as a command: in a trailing comment,
a step `name:`, or `echo`'s argument. Now `checkers.sh --would-run <args>`
answers what `<args>` would run, from the same argument loop and name
resolution the real run uses. `--would-run` is honoured only as the first
argument, so a CI step that spells it anywhere is a usage error rather than a
green step that ran nothing. The check itself only finds invocations. It reads
the shell text of `run:` values alone, strips a comment by bash's rule (a `#`
at the start of a word, outside quotes, so `nixpkgs#trealla` survives), splits
at `;`, `&&` and `|`, and counts a command only when `checkers.sh` is its
command word or what `nix shell … --command` runs. A line `shlex` cannot parse
contributes nothing, which fails closed.
### pr-watch and mergeability (improvise-edpl, improvise-zlj2)
+59
View File
@@ -841,3 +841,62 @@ the next line with a backslash; and a step naming a suite that does not exist.
The script was restored after each (`cmp` against a saved copy), and the
self-test is back to "5 graphs and 7 suite lists judged".
### The suite check asks checkers.sh what a step runs — `scripts/check-ci-graph.py` (improvise-maax)
**What broke.** The adversarial review of #228 at `cee61b94`, and ChatGPT
Codex's P1 on it, found the first version failing open in four shapes. Each let
the check pass while a suite never ran:
- a bare `scripts/checkers.sh` outside a command read as "every suite runs":
after an inline `#` (Codex), in a step `name:`, or as `echo`'s argument;
- `checkers.sh --list` alone read as every suite, though `--list` runs nothing;
- `--self-test zz-new kb` counted `zz-new` and `kb`, though `--self-test`
discards them;
- `--list kb zz-new` counted both.
Two more were loud but wrong: only the first invocation on a line counted, and
quoted names kept their quotes.
**The fix moves the argument rules back into `checkers.sh`.** It gains
`--would-run`, which is honoured only as the first argument and reports rather
than runs. `check-ci-graph.py` finds invocations: `run:` text only, comments
stripped by bash's rule, command position only. It then asks `--would-run`
about each one.
**Every new case run against the OLD parser** (`/tmp/ccg-old.py`, the file at
`cee61b94`): eight of the sixteen suite cases give the wrong answer there. The
five false passes are `--list` alone, the inline comment (Codex), the non-run
block scalar, the step `name:` and `echo`'s argument. Wrong in a noisier way:
`--self-test zz-new kb` (the old parser counts `kb`, so it reports only a
bogus "zz-new is not a suite"), `--list kb zz-new` (the same), and `a && b`
(the second suite is missed).
**A vacuity fixed before the rivals ran.** As first written, the bare-mention
cases required `zz-new`, which no real `checkers.sh` ever runs. A false bare
run then still left exactly `zz-new` unrun, the expected answer, so the
any-word-position rival passed all sixteen. The cases now require `kb` and
`pr-watch`, and the real step runs only `kb`. A false bare run is then the only
way `pr-watch` counts as run.
**Rivals, one function swapped in at a time and each run against the real
`--self-test`:**
- **no comment stripping:** fails only "a suite named inside a trailing shell
comment is not run";
- **every line scanned** (run: prefixes still recognised): fails "a
command-shaped line in a non-run block scalar" and the block-run case;
- **`checkers.sh` accepted in any word position:** fails only "a bare mention
as echo's argument";
- **argv modelled naively** (names, else everything): fails the `--self-test`
case and both `--list` cases;
- **first invocation per line only:** fails "every invocation on a line
counts" and the block-run case;
- **`--would-run` honoured in any position**, in `checkers.sh` itself: fails
`checkers-test.sh`'s "--would-run anywhere but first is a usage error" (exit
0, wanted 4) and the parser's "a step spelling --would-run runs nothing".
One more rival crashed instead: an early every-line rival hit `shlex`'s
"No escaped character" on a lone trailing backslash. So a line that cannot be
parsed now contributes no invocation, which fails closed. Everything was
restored (`cmp` for `checkers.sh`) and re-run: `--self-test` reports 16 suite
lists and `checkers-test.sh` 17/17. A planted `zz-planted` suite still fails
the real run, naming it.
+160 -55
View File
@@ -8,12 +8,14 @@ nothing noticed, so the rule is a gate. See context/code/tooling.md.
Second rule: every suite `scripts/checkers.sh --list` knows must run in some
CI step. ci.yml names its suites by hand, so a new `*-test.sh` ran locally and
never in CI until someone remembered to add it (improvise-dak5).
never in CI until someone remembered to add it (improvise-dak5). What a step
runs is asked of `checkers.sh --would-run`, never re-derived (improvise-maax).
`--self-test` checks the checker: each case is a graph this must refuse.
"""
import re
import shlex
import subprocess
import sys
@@ -65,49 +67,116 @@ def verdict(jobs, needs):
return out
_CHECKERS = re.compile(r"(?:^|[\s/])scripts/checkers\.sh\b(.*)$")
ALL = object()
_RUN = re.compile(r"^(\s*)(?:-\s+)?run:\s?(.*)$")
_ASSIGN = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*=")
_OPERATOR = set(";&|()")
def run_scripts(text):
"""The shell text of every `run:` value. Nothing else in ci.yml is a
command, so a step `name:` or a YAML comment never reads as one."""
lines, out, i = text.split("\n"), [], 0
while i < len(lines):
m = _RUN.match(lines[i])
i += 1
if not m:
continue
indent, value = len(m.group(1)), m.group(2).strip()
if value[:1] not in ("|", ">"):
out.append(value)
continue
body = []
while i < len(lines) and (not lines[i].strip() or
len(lines[i]) - len(lines[i].lstrip()) > indent):
body.append(lines[i])
i += 1
out.append("\n".join(body))
return out
def strip_comment(line):
"""Bash's rule: `#` opens a comment only at the start of a word and outside
quotes, so `nixpkgs#trealla` survives and `echo x # checkers.sh` does not."""
quote, prev = None, " "
for j, c in enumerate(line):
if quote:
if c == quote:
quote = None
elif c in "'\"":
quote = c
elif c == "#" and (prev.isspace() or prev in _OPERATOR):
return line[:j]
prev = c
return line
def simple_commands(script):
"""Each simple command in a script, as its words with quotes removed."""
for line in script.replace("\\\n", " ").split("\n"):
lexer = shlex.shlex(strip_comment(line), posix=True, punctuation_chars=True)
lexer.whitespace_split = True
lexer.commenters = ""
try:
tokens = list(lexer)
except ValueError:
continue # unparseable: no invocation, so it fails closed
words = []
for tok in tokens:
if set(tok) <= _OPERATOR:
if words:
yield words
words = []
else:
words.append(tok)
if words:
yield words
def checkers_invocations(text):
"""The arguments of every command that IS checkers.sh: the command word
after any VAR=value, or the program `nix shell ... --command` runs."""
for script in run_scripts(text):
for words in simple_commands(script):
k = 0
while k < len(words) and _ASSIGN.match(words[k]):
k += 1
if "--command" in words[k:]:
k = words.index("--command", k) + 1
if k < len(words) and words[k].endswith("scripts/checkers.sh"):
yield words[k + 1:]
def would_run(args):
"""What checkers.sh itself says these arguments run. Its argument rules are
not restated here: --list and --self-test mean what the script makes them."""
out = subprocess.run(["scripts/checkers.sh", "--would-run", *args],
capture_output=True, text=True).stdout
ran, missing = set(), set()
for line in out.split("\n"):
if line.startswith("would-run: "):
ran.add(line[len("would-run: "):])
elif line.startswith("missing: "):
missing.add(line[len("missing: "):])
return ran, missing
def suites_run(text):
"""The suite names CI runs, or ALL when some step runs checkers.sh bare.
A YAML comment is skipped: a bare mention there must not read as "every
suite runs". A trailing backslash continues onto the next line.
"""
lines = text.split("\n")
names, i = set(), 0
while i < len(lines):
raw = lines[i]
i += 1
if raw.lstrip().startswith("#"):
continue
m = _CHECKERS.search(raw)
if not m:
continue
rest = m.group(1)
while rest.rstrip().endswith("\\") and i < len(lines):
rest = rest.rstrip()[:-1] + " " + lines[i]
i += 1
rest = re.split(r"&&|\|\||[;|#]", rest)[0]
args = rest.split()
if "--self-test" in args:
names.add("checkers")
picked = [a for a in args if not a.startswith("-")]
if not picked and "--self-test" not in args:
return ALL
names.update(picked)
return names
"""(suites some step runs, names a step asks for that are not suites)."""
ran, missing = set(), set()
for args in checkers_invocations(text):
r, m = would_run(args)
ran |= r
missing |= m
return ran, missing
def suite_verdict(suites, ran):
"""The complaints about suites CI never runs, or runs by a wrong name."""
if ran is ALL:
return []
def suite_verdict(suites, run):
"""The complaints about suites CI never runs, or asks for by a wrong name."""
ran, missing = run
out = [f"`{s}` is a checkers.sh suite that no CI step runs; add it to ci.yml"
for s in sorted(set(suites) - ran)]
out += [f"ci.yml runs checkers.sh `{s}`, which is not a suite"
for s in sorted(ran - set(suites))]
for s in sorted(missing)]
return out
@@ -118,35 +187,71 @@ def list_suites():
return [s for s in out.split("\n") if s.strip()]
# (name, suites checkers.sh knows, workflow text, the complaints it must draw).
# (name, suites to require, workflow text, the complaints it must draw). Every
# case asks the REAL checkers.sh. A false "bare run" must be the ONLY way pr-watch runs.
_UNRUN = "`zz-new` is a checkers.sh suite that no CI step runs; add it to ci.yml"
_NO_PW = "`pr-watch` is a checkers.sh suite that no CI step runs; add it to ci.yml"
_KB_PW = ["kb", "pr-watch", "zz-new"]
_REQ = ["kb", "pr-watch"]
_ONLY_KB = " run: ./scripts/checkers.sh kb\n"
SUITE_SELF_TEST = [
("a suite no step runs is named",
["kb", "zz-new"], " run: ./scripts/checkers.sh kb\n",
["`zz-new` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
_KB_PW, " run: ./scripts/checkers.sh kb pr-watch\n", [_UNRUN]),
("every suite named across steps is accepted",
("suites named across steps, one inside nix shell --command, are accepted",
["kb", "pr-watch", "checkers"],
" run: nix shell x --command ./scripts/checkers.sh kb\n"
" run: nix shell nixpkgs#trealla --command ./scripts/checkers.sh kb\n"
" run: ./scripts/checkers.sh --self-test\n"
" run: ./scripts/checkers.sh pr-watch\n", []),
" - run: ./scripts/checkers.sh pr-watch\n", []),
("a bare run covers every suite",
["kb", "zz-new"], " run: ./scripts/checkers.sh\n", []),
("a bare run covers every real suite",
["kb", "pr-watch", "checkers"], " run: ./scripts/checkers.sh\n", []),
("--self-test covers the checkers suite and nothing else",
["checkers", "kb"], " run: ./scripts/checkers.sh --self-test\n",
("--self-test runs checkers and DISCARDS the names beside it",
["checkers", "kb"], " run: ./scripts/checkers.sh --self-test zz-new kb\n",
["`kb` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
("a bare mention in a comment does not count as running everything",
["kb", "zz-new"],
" # Every suite goes through scripts/checkers.sh\n"
" run: ./scripts/checkers.sh kb\n",
["`zz-new` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
("--list alone runs nothing",
["kb"], " run: ./scripts/checkers.sh --list\n",
["`kb` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
("names continued onto the next line count, and are not a bare run",
["kb", "pr-watch", "zz-new"],
" run: ./scripts/checkers.sh \\\n kb pr-watch\n",
["`zz-new` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
("--list with names runs nothing either",
["kb"], " run: ./scripts/checkers.sh --list kb zz-new\n",
["`kb` is a checkers.sh suite that no CI step runs; add it to ci.yml"]),
("a bare mention after an inline shell comment is not a run (Codex, #228)",
_REQ, " run: echo skipped # ./scripts/checkers.sh\n" + _ONLY_KB, [_NO_PW]),
("a suite named inside a trailing shell comment is not run",
_REQ, " run: ./scripts/checkers.sh kb # pr-watch\n", [_NO_PW]),
("a command-shaped line in a non-run block scalar is not a run",
_REQ,
" - name: notes\n with:\n notes: |\n"
" ./scripts/checkers.sh\n " + _ONLY_KB, [_NO_PW]),
("a bare mention in a step name is not a run",
_REQ, " - name: Runs scripts/checkers.sh\n " + _ONLY_KB, [_NO_PW]),
("a bare mention as echo's argument is not a run",
_REQ, " run: echo scripts/checkers.sh\n" + _ONLY_KB, [_NO_PW]),
("a bare mention in a YAML comment is not a run",
_REQ, " # Every suite goes through scripts/checkers.sh\n" + _ONLY_KB,
[_NO_PW]),
("every invocation on a line counts, and quotes are removed",
_REQ,
" run: ./scripts/checkers.sh kb && ./scripts/checkers.sh 'pr-watch'\n", []),
("a block run is read, continuation lines included",
_REQ,
" run: |\n set -e\n ./scripts/checkers.sh \\\n"
" kb pr-watch\n - name: next\n", []),
("a step spelling --would-run runs nothing, wherever it puts it",
_REQ, _ONLY_KB + " run: ./scripts/checkers.sh pr-watch --would-run\n"
" run: ./scripts/checkers.sh --would-run pr-watch\n", [_NO_PW]),
("a step naming a suite that does not exist is caught",
["kb"], " run: ./scripts/checkers.sh kb ghost\n",
+29
View File
@@ -176,5 +176,34 @@ listed="$(tr '\n' ' ' < "$work/out")"
&& ok "an X-test.sh with no X beside it is not a suite" \
|| bad "an X-test.sh with no X beside it is not a suite" "got [$listed]"
# ---------------------------------------------------------------------------
# 8. --would-run reports what the rest of the arguments run, and runs nothing.
# check-ci-graph.py asks it, so each argument rule below is answered HERE.
d="$(sandbox wouldrun)"
for n in alpha beta; do
suite "$d" "$n" <<EOF
#!/usr/bin/env bash
touch "$work/ran-$n"
EOF
done
rm -f "$work"/ran-*
"$d/checkers.sh" --would-run > "$work/out" 2>&1
got="$(tr '\n' ' ' < "$work/out")"
[ "$got" = "would-run: alpha would-run: beta " ] && [ ! -e "$work/ran-alpha" ] \
&& ok "--would-run with no names reports every suite and runs none" \
|| bad "--would-run with no names reports every suite and runs none" "got [$got]"
"$d/checkers.sh" --would-run --self-test alpha > "$work/out" 2>&1
got="$(tr '\n' ' ' < "$work/out")"
[ "$got" = "missing: checkers " ] \
&& ok "--would-run --self-test reports checkers and discards the names" \
|| bad "--would-run --self-test reports checkers and discards the names" "got [$got]"
"$d/checkers.sh" --would-run --list alpha > "$work/out" 2>&1
grep -q '^would-run:' "$work/out" \
&& bad "--would-run --list reports nothing as run" "$(tr '\n' ' ' < "$work/out")" \
|| ok "--would-run --list reports nothing as run"
expect "--would-run anywhere but first is a usage error" 4 "$d/checkers.sh" alpha --would-run
[ ! -e "$work/ran-alpha" ] && ok "and it runs nothing" \
|| bad "and it runs nothing" "alpha ran"
printf '\ncheckers-test: %d passed, %d failed\n' "$pass" "$fail"
[ "$fail" -eq 0 ]
+22 -8
View File
@@ -27,6 +27,10 @@ usage: checkers.sh [--list] [--self-test] [name ...]
--list print the suite names and exit
--self-test run scripts/checkers-test.sh, which injects the faults this
runner exists to catch
--would-run as the FIRST argument only: run nothing, and print
`would-run: <suite>` (or `missing: <name>`) for each suite
the rest of the arguments would run. check-ci-graph.py
asks this, so the argument rules have one home
exit 0 every suite passed
exit 1 a suite failed, or a shebang is not `#!/usr/bin/env bash`
@@ -107,8 +111,10 @@ run_suite() {
}
main() {
local self_test=0
local self_test=0 would_run=0
local -a names=()
# Only as the FIRST argument, so a CI step spelling it anywhere is an error.
if [ "${1:-}" = --would-run ]; then would_run=1; shift; fi
while [ $# -gt 0 ]; do
case "$1" in
--list) suites; return 0 ;;
@@ -120,6 +126,21 @@ main() {
shift
done
if [ "$self_test" = 1 ]; then
names=(checkers)
elif [ ${#names[@]} -eq 0 ]; then
while IFS= read -r line; do names+=("$line"); done < <(suites)
fi
local n file
if [ "$would_run" = 1 ]; then
for n in ${names[@]+"${names[@]}"}; do
if [ -f "$HERE/$n-test.sh" ]; then printf 'would-run: %s\n' "$n"
else printf 'missing: %s\n' "$n"; fi
done
return 0
fi
local pinned; pinned="$(find_modern_bash)"
if [ -z "$pinned" ]; then
echo 'checkers: no bash >= 5 on PATH and nix cannot supply one.' >&2
@@ -132,13 +153,6 @@ main() {
local rc=0
check_shebangs || rc=1
if [ "$self_test" = 1 ]; then
names=(checkers)
elif [ ${#names[@]} -eq 0 ]; then
while IFS= read -r line; do names+=("$line"); done < <(suites)
fi
local n file
for n in ${names[@]+"${names[@]}"}; do
file="$HERE/$n-test.sh"
if [ ! -f "$file" ]; then