check-ci-graph: a line holding a YAML-only line break or a control character is refused (xpzh)

The gate split on \n alone while YAML also breaks at U+0085, U+2028 and
U+2029, so an if: after one was invisible and the gate exited 0. Every line,
comments and block-scalar bodies included, is now checked for C0 (but tab),
DEL, C1 and U+2028/2029. CR is folded by the one read path, read_workflow, and
file-based controls pin that. Argument: context/code/tooling.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze

context/code/tooling.md

@@ -435,6 +435,37 @@ because its value is. So a quoted key deep in a job, such as `"FOO": bar` under
 `env:`, is now refused, because the gate cannot tell it from an unclosed quote.
 That is a valid style refused loudly, beside the others in this section.
 
+**A line holding a line break YAML knows and the gate does not, or a control
+character, is refused** (improvise-xpzh). The gate splits the text on `\n`
+only. YAML also breaks lines at U+0085 (NEL), U+2028 and U+2029, so
+`runs-on: x<U+2028>    if: false` was one line to the gate and two to go-yaml.
+The job's `if:` was invisible, and the gate exited 0. The same break inside a
+full-line comment, or inside a `run: |` body, ends that line for YAML too. A
+body line `echo a<U+2028>    if: false` makes the `if:` a real job key.
+
+So the check runs on **every** line, before the comment and owner skips, and
+refuses:
+- C0 controls other than tab (CR included);
+- DEL;
+- the whole C1 block, which contains U+0085;
+- U+2028 and U+2029.
+
+CR is the one break the read folds. `read_workflow` opens with Python's
+universal newlines, which turn CR and CRLF into `\n`, and a lone CR is a line
+break to YAML too, so the two agree. A CR that survives to the split can only
+come from a read that did not fold it, and it is refused. Two self-test
+controls write real files, one CRLF and one lone CR, and read them back
+through `read_workflow`. The code path is the point: a string handed straight
+to `suite_verdict` never meets the fold.
+
+**Why not refuse all non-ASCII,** which would be the simpler allowlist. The
+real `ci.yml` has 27 non-ASCII characters (em dash, `§`, arrows) in its
+comments, so it would have to be rewritten to pass. The set of characters that
+changes line structure is not open-ended, though. It is YAML's own `b-char`:
+LF and CR, plus NEL, LS and PS in YAML 1.1. So refusing that set, and the
+controls around it, is complete by the spec's definition, not a list that
+grows with each review.
+
 `read_graph` refuses nothing, so it cannot rely on refusal. It opens a job on
 any `  name:` line followed by whitespace or the end of the line, which is
 wider than the exact header `ci_steps` accepts. Reading too wide only adds jobs

context/test-evidence/tooling.md

@@ -1304,3 +1304,51 @@ which fails 12 cases), and both `_opens_quote` rivals. The four new ones:
 as `"FOO": bar` under `env:`, is refused, because the gate cannot tell it from
 an unclosed quote. Checked directly: refused, and the plain `FOO: bar` control
 is accepted.
+
+### A line break YAML knows and the gate does not is refused — `scripts/check-ci-graph.py` (improvise-xpzh)
+
+**What broke (P2, fail-open).** Found in the delta review of #228 at
+`7ae871cb`. The gate splits on `\n` only, but YAML also breaks lines at U+0085,
+U+2028 and U+2029. `runs-on: ubuntu-latest<U+2028>    if: false` on the real
+`kb` job is one line to the gate and two to go-yaml. yq-go reads
+`kb.if == false` with each of the three characters, and the gate exited 0.
+
+**Fix.** `_UNSAFE_CHAR` is checked on every line, before the comment and owner
+skips. It covers C0 except tab, DEL, all of C1 (which holds U+0085), U+2028 and
+U+2029. The workflow is read through `read_workflow`, whose universal newlines
+fold CR and CRLF. The argument, including why all non-ASCII is not refused, is
+in `context/code/tooling.md`. The real `ci.yml` has 0 CR bytes, no BOM and 27
+printable non-ASCII characters, none of them in the refused set.
+
+**Red first** against `7ae871cb`. All five new suite cases gave `saw []`:
+- one per break character, each on a `runs-on:` line;
+- a U+2028 inside a full-line comment;
+- a U+2028 inside a `run: |` body, on the job's last step. yq-go ends the
+  block there and reads the `if:` as a job key.
+
+**Controls.** Two cases write a real file and read it back through
+`read_workflow`:
+- CRLF line ends are accepted with no complaint;
+- a lone CR is a line break, so the following `if:` makes kb conditional, as
+  YAML reads it.
+
+**Replayed on the real `ci.yml`.** The three plants, one per character, on
+kb's `runs-on:`, exit 0 at `7ae871cb`. Now all three exit 1, naming line 255.
+All 13 plants from the five rounds exit 1. The file was written with
+`newline=""` so the characters landed literally, and it was restored byte for
+byte (sha1 `a00288a8…`).
+
+**Rivals, each a source edit run against the real `--self-test` (65 suite
+lists, 2 files).** All 27 fail. The 22 from earlier rounds fail as before.
+Five are new:
+
+| Rival | Fails |
+|---|---|
+| The character check removed | the five new cases |
+| C1 allowed | the U+0085 case alone |
+| Comments not checked | the comment case alone |
+| Owned content not checked | the block-scalar case, and the comment case |
+| The read does not fold CR (`newline=""`) | both file controls |
+
+The owned-content rival also fails the comment case because `owner` still
+holds the preceding `runs-on:` key's column when that line is checked.

scripts/check-ci-graph.py

@@ -15,9 +15,11 @@ here, and what a row runs is asked of `--would-run` (improvise-maax, st8q).
 `--self-test` checks the checker: each case is a graph this must refuse.
 """
 
+import os
 import re
 import subprocess
 import sys
+import tempfile
 
 WORKFLOW = ".github/workflows/ci.yml"
 DEPLOY = "pages-deploy"
@@ -84,6 +86,7 @@ _TOP_KEY = re.compile(rf"^{_NAME}:(\s|$)")
 _NEEDS_VALUE = re.compile(rf"^(|{_NAME}|\[\s*{_NAME}(\s*,\s*{_NAME})*\s*\])$")
 _NEEDS_ITEM = re.compile(rf"^      - {_NAME}$")
 _COMMENT = re.compile(r"\s#")
+_UNSAFE_CHAR = re.compile("[\x00-\x08\x0b-\x1f\x7f-\x9f\u2028\u2029]")
 
 
 def _readable(line, indent, in_steps, in_needs, job_keyed):
@@ -160,6 +163,9 @@ def ci_steps(text):
             steps.append(step)
 
     for n, raw in enumerate(text.split("\n"), 1):
+        if _UNSAFE_CHAR.search(raw):
+            unread.append((n, "char"))
+            continue
         indent = len(raw) - len(raw.lstrip())
         if owner is not None and (not raw.strip() or indent > owner):
             if "scripts/checkers.sh" in raw:
@@ -268,6 +274,9 @@ def _unrun(s):
 
 
 def _unread(n, kind="plain"):
+    if kind == "char":
+        return (f"ci.yml line {n} holds a control character or a line break YAML sees and "
+                "this gate does not (U+0085, U+2028, U+2029); remove it")
     if kind == "value":
         return (f"ci.yml line {n} has a value that is not one plain line (a multi-line "
                 "quote or flow, a nested or quoted flow, an anchor, alias or tag); spell "
@@ -309,6 +318,12 @@ def suite_verdict(suites, text):
     return out
 
 
+def read_workflow(path):
+    """The workflow's text. Universal newlines fold CR and CRLF into the one break the
+    gate splits on; every other break YAML knows is refused (improvise-xpzh)."""
+    return open(path, encoding="utf8").read()
+
+
 def list_suites():
     """The suites, from checkers.sh itself, so its naming rule has one home."""
     out = subprocess.run(["scripts/checkers.sh", "--list"], capture_output=True,
@@ -336,6 +351,7 @@ def _stray(n):
 
 _GATE = "  gate:\n    if: false\n    steps:\n      - run: echo\n"
 _KB_RUN = "      - run: ./scripts/checkers.sh --ci kb\n"
+_YAML_ONLY_BREAKS = ("\x85", "\u2028", "\u2029")
 _INNER_RUN = "        run: ./scripts/checkers.sh --ci kb\n"
 _NEEDS_GATE = ["job `kb`'s `checkers.sh --ci` step is conditional (needs gate, which has if), "
                "so it does not count"] + _NO_PW
@@ -640,12 +656,38 @@ SUITE_SELF_TEST = [
                                  + "        c: x\"]\n")),
                 'name: !!seq', kind="value"),
 
+    *[_unreadable(f"a U+{ord(sep):04X} inside a line is refused (xpzh)",
+                  _wf(_BUILD, "  kb:\n    runs-on: x" + sep + "    if: false\n    steps:\n" + _KB_RUN),
+                  sep, kind="char")
+      for sep in _YAML_ONLY_BREAKS],
+
+    _unreadable("a line break inside a block scalar's body is refused too (xpzh)",
+                _wf(_BUILD, _job("kb", _KB_RUN + "      - run: |\n          echo a\u2028    if: false\n")),
+                "echo a", kind="char"),
+
+    _unreadable("a line break inside a full-line comment is refused too (xpzh)",
+                _wf(_BUILD, "  kb:\n    runs-on: x\n    # note\u2028    if: false\n    steps:\n"
+                    + _KB_RUN), "# note", kind="char"),
+
     ("a YAML comment mentioning checkers.sh is fine", _REQ,
      _wf(_BUILD, _job("kb", "      # every suite goes through scripts/checkers.sh\n"
                             "      - run: ./scripts/checkers.sh --ci kb\n")), []),
 ]
 
 
+# (name, file bytes as text, the complaints it must draw), read back through
+# read_workflow, because what a CR means depends on that read (improvise-xpzh).
+FILE_SELF_TEST = [
+    ("control: CRLF line ends are folded by the read and accepted (xpzh)",
+     _wf(_BUILD, _job("kb", _KB_RUN)).replace("\n", "\r\n"), []),
+
+    ("control: a lone CR is a line break, as YAML reads it (xpzh)",
+     _wf(_BUILD, "  kb:\n    runs-on: x\r    if: false\n    steps:\n" + _KB_RUN),
+     ["job `kb`'s `checkers.sh --ci` step is conditional (job if), so it does not count"]
+     + _NO_PW),
+]
+
+
 # (name, workflow text, the complaints it must draw).
 SELF_TEST = [
     ("a job nobody gates is named",
@@ -690,19 +732,27 @@ def self_test():
         got = suite_verdict(suites, text)
         if got != want:
             failures.append(f"  {name}: expected {want}, saw {got}")
+    for name, content, want in FILE_SELF_TEST:
+        with tempfile.TemporaryDirectory() as d:
+            path = os.path.join(d, "ci.yml")
+            with open(path, "w", encoding="utf8", newline="") as f:
+                f.write(content)
+            got = suite_verdict(_REQ, read_workflow(path))
+        if got != want:
+            failures.append(f"  {name}: expected {want}, saw {got}")
     if failures:
         print("ci-graph self-test FAILED:", file=sys.stderr)
         print("\n".join(failures), file=sys.stderr)
         return 1
-    print(f"ci-graph self-test: {len(SELF_TEST)} graphs and "
-          f"{len(SUITE_SELF_TEST)} suite lists judged")
+    print(f"ci-graph self-test: {len(SELF_TEST)} graphs, "
+          f"{len(SUITE_SELF_TEST)} suite lists and {len(FILE_SELF_TEST)} files judged")
     return 0
 
 
 def main():
     if "--self-test" in sys.argv[1:]:
         return self_test()
-    text = open(WORKFLOW, encoding="utf8").read()
+    text = read_workflow(WORKFLOW)
     jobs, needs = read_graph(text)
     if len(jobs) < 2:
         print(f"error: {WORKFLOW}: found {len(jobs)} jobs; the parse is wrong",