Edward Langley
ea347ccab4
Under bash an unmatched glob survives as a literal; node 22+ globs the literal itself, matches nothing, reports 0 tests and exits 0. Both CI node steps -- the `js` job and the browser job's bundle suite -- were exposed: rename or move either directory and the step goes green having asserted nothing. Reproduced under `bash -e`, which is the shell CI names; zsh ABORTS on an unmatched glob, so the failure is invisible from a developer's shell. The glob becomes the script's rather than the shell's. scripts/node-test.sh expands it, refuses an empty match, and also refuses a matched file that declares no test -- checked LEXICALLY, because node reports such a file as one passing test named after the file, so the `# tests N` a minimum-count assertion would read is the FILE count and is satisfied by the offender. Measured on node 24.20. scripts/node-test-test.sh proves each refusal by injecting its fault, including that the wrapper does not buy its refusals by swallowing a real failure, and runs in the `js` job: a check wired into nothing is a convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
87 lines
3.1 KiB
Bash
Executable File
87 lines
3.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# check_the_checker for scripts/node-test.sh. Every case is a way a node test
|
|
# step can exit 0 having asserted nothing -- which is what the step it replaces
|
|
# did, on both CI jobs, for the whole life of web/tests/ (improvise-vz54).
|
|
set -uo pipefail
|
|
|
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
RUN="$here/node-test.sh"
|
|
work="$(mktemp -d)"
|
|
trap 'rm -rf "$work"' EXIT
|
|
|
|
pass=0; fail=0
|
|
ok() { printf ' ok %s\n' "$1"; pass=$((pass+1)); }
|
|
bad() { printf ' FAIL %s (exit %s, wanted %s)\n' "$1" "$2" "$3"; fail=$((fail+1)); }
|
|
|
|
# $1 name, $2 expected exit, rest: patterns
|
|
run_case() {
|
|
local name="$1" want="$2"; shift 2
|
|
"$RUN" "$@" >/dev/null 2>&1
|
|
local got=$?
|
|
if [ "$want" = "nonzero" ]; then
|
|
[ "$got" -ne 0 ] && ok "$name" || bad "$name" "$got" "$want"
|
|
else
|
|
[ "$got" = "$want" ] && ok "$name" || bad "$name" "$got" "$want"
|
|
fi
|
|
}
|
|
|
|
mkdir -p "$work/tests" "$work/empty"
|
|
|
|
cat > "$work/tests/green.test.js" <<'EOF'
|
|
const { test } = require("node:test");
|
|
test("a test that passes", () => {});
|
|
EOF
|
|
|
|
cat > "$work/tests/red.test.js" <<'EOF'
|
|
const { test } = require("node:test");
|
|
const assert = require("node:assert");
|
|
test("a test that fails", () => { assert.equal(1, 2); });
|
|
EOF
|
|
|
|
# A file that matches the glob and declares no test -- a helper left in the
|
|
# directory, or a suite emptied by a refactor. node reports it as ONE
|
|
# PASSING TEST named after the file, so its own count cannot see it and a
|
|
# minimum-count assertion over `# tests` is satisfied by the offender.
|
|
cat > "$work/empty/nothing.test.js" <<'EOF'
|
|
module.exports = { helper: () => 1 };
|
|
EOF
|
|
|
|
# The lexical check must not call a real suite barren. Both spellings the
|
|
# repo uses, and the indented form, live here.
|
|
mkdir -p "$work/shapes"
|
|
cat > "$work/shapes/shapes.test.js" <<'EOF'
|
|
const { test } = require("node:test");
|
|
test.describe("a group", () => {
|
|
test("an indented test", () => {});
|
|
});
|
|
EOF
|
|
|
|
# The bug itself. Under bash the literal survives, node globs it, matches
|
|
# nothing, and the step goes green. Demonstrated at improvise-vz54.
|
|
run_case "a pattern that matches nothing is refused" 1 "$work/gone/*.test.js"
|
|
|
|
run_case "a matching file with a passing test is accepted" 0 "$work/tests/green.test.js"
|
|
|
|
# The refusal must not be bought by swallowing real failures.
|
|
run_case "a failing test still fails" nonzero "$work/tests/red.test.js"
|
|
|
|
run_case "a file that declares no test is refused" 1 "$work/empty/*.test.js"
|
|
|
|
run_case "a suite using describe and an indented test is not called barren" 0 \
|
|
"$work/shapes/*.test.js"
|
|
|
|
# Two globs where the SECOND dies is the shape a directory move makes: the
|
|
# first still matches, so a check over the union would pass.
|
|
run_case "one dead pattern among several is refused" 1 \
|
|
"$work/tests/green.test.js" "$work/gone/*.test.js"
|
|
|
|
run_case "no argument at all is a usage error, not a pass" 2
|
|
|
|
# The literal the caller would pass if they forgot to quote AND the glob
|
|
# matched: ordinary filenames, which expand to themselves.
|
|
run_case "already-expanded filenames still run" 0 \
|
|
"$work/tests/green.test.js"
|
|
|
|
printf '\nnode-test-test: %d passed, %d failed\n' "$pass" "$fail"
|
|
[ "$fail" -eq 0 ]
|