787e354b9b
The `build` job fails on main and on every branch merged from it: doc-site: context/code/pr-review.md is neither published nor a named omission NOT a merge loss -- scripts/build-doc-site.mjs in this branch is byte-identical to main's, and neither lists the file. PR #96 added context/code/pr-review.md without a manifest line, which is exactly the gate improvise-res9 added: a doc added without a line publishes nothing and says nothing, so the generator refuses. Omitted rather than published, and the line is cloud-agents.md's rather than tooling.md's. tooling.md explains checkers a reader of the code will meet; pr-review.md is about THIS REPOSITORY'S PROCESS -- where a Codex review appears, how to tell it from our own agents' comments, why absence is not approval. Nobody visiting a website is about to read a PR thread. Both homes updated, because an omission is restated beside the manifest AND in doc-site.md section 2, and the prose there counted them ("Neither is referenced...") so it went stale at three. This strays into scripts/, which is Wave N's ground. Flagged in the hand-back; it is a one-line data entry either way if the orchestrator would rather publish it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
67 lines
21 KiB
HTML
67 lines
21 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
|
<title>The repository's own gates — improvise docs</title>
|
|
<link rel="stylesheet" href="../style.css" />
|
|
</head>
|
|
<body>
|
|
<div class="shell">
|
|
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html" class="here">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
|
|
<main>
|
|
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/test-evidence/tooling.md">context/test-evidence/tooling.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#test-evidence-the-repository-s-own-gates">Test evidence — the repository's own gates</a></li><li class="l3"><a href="#no-test-in-web-tests-reaches-for-a-built-bundle">"no test in web/tests/ reaches for a built bundle"</a></li><li class="l3"><a href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">"the subdirectory that MAY use a bundle is not the one being checked"</a></li><li class="l3"><a href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">"no test in web/tests/ imports a module that reaches the bundle"</a></li><li class="l3"><a href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">Note — the checker's own blockers, found by review of PR #83</a></li><li class="l3"><a href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">Note — the flag that selected the cap away (improvise-ijql)</a></li><li class="l3"><a href="#note-the-stamp-gate-s-discrimination-measured">Note — the stamp gate's discrimination, measured</a></li><li class="l3"><a href="#note-the-stamp-gate-matches-literally">Note — the stamp gate matches literally</a></li><li class="l3"><a href="#note-the-doc-site-s-derived-checks">Note — the doc site's derived checks</a></li><li class="l3"><a href="#duplicate-evidence-headings-self-test-duplicates">Duplicate evidence headings — SELF_TEST_DUPLICATES</a></li></ul></details><h1 id="test-evidence-the-repository-s-own-gates">Test evidence — the repository's own gates</h1>
|
|
<p>Evidence for the checks in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/scripts"><code>scripts/</code></a> and for the tests that hold CI's shape. The arguments behind the checkers themselves are in <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>; this file records what fault each test was watched to catch.</p>
|
|
<p>Note what is deliberately absent. <code>scripts/check-comment-runs.py --self-test</code>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb-test.sh"><code>scripts/kb-test.sh</code></a> and <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> are check-the-checker suites whose cases are already injected faults, and none of their case names is a Rust <code>fn</code> or a JS <code>test(</code> the evidence checker could resolve — so their evidence is the case list itself plus <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>, and inventing headings for them here would be headings naming nothing.</p>
|
|
<h3 id="no-test-in-web-tests-reaches-for-a-built-bundle"><a class="anchor" href="#no-test-in-web-tests-reaches-for-a-built-bundle">#</a>"no test in web/tests/ reaches for a built bundle"</h3>
|
|
<p>The <code>js</code> CI job runs <code>node --test web/tests/*.test.js</code> and installs nothing; the <code>browser</code> job runs <code>web/tests/bundle/*.test.js</code> after three wasm-pack builds. The moment a flat <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests"><code>web/tests/</code></a> file reads <code>web/pkg/</code>, the fast job inherits the slow one's dependency — and the likely repair is to add a build step to it, silently spending the ~4m30s the split exists to avoid. Until <a class="tid" href="../tickets.html#improvise-4ktt" title="The js job's bundle-freedom is a convention, not a gate">improvise-4ktt</a> that constraint was a directory convention and a glob, checked by nothing, and it failed in the direction that looks like ordinary breakage: a confusing ENOENT in CI on a machine where the bundle happens not to be built.</p>
|
|
<p>Fault injected, 2026-09-21: appended <code>const injected = readFileSync("../pkg/improvise_wasm_client.js");</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The test failed naming that file and pointing at <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a>. Reverted; green again.</p>
|
|
<p>Two things the test does that a simpler one would not. It asserts its own filename is among the files it scans, so it cannot become the one place a violation may hide. And it strips comments before matching, because this file's own header and the test's header both name the directory in prose — a lexical grep over raw text would have had to exempt them, and an exemption is where the next violation goes.</p>
|
|
<p><strong>What it cannot see</strong>, stated because the check itself uses the evasion: a path assembled at runtime (<code>"../" + "p" + "kg"</code>) is invisible to a lexical match, and the test's own <code>FORBIDDEN</code> constant is built exactly that way to stay out of its own scan. That is a deliberate, disclosed hole — closing it needs a parser, and the failure it would catch is one somebody has to write on purpose, where this gate exists for the one written by accident.</p>
|
|
<h3 id="the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked"><a class="anchor" href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">#</a>"the subdirectory that MAY use a bundle is not the one being checked"</h3>
|
|
<p>The companion, and the one that stops the first from passing vacuously. If <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a> were emptied or its files stopped touching the bundle, the first test would still be green while having nothing to be scoped <em>against</em> — "no flat test uses the bundle" is uninteresting if no test anywhere does. This asserts the other half of the split exists and really does what the flat half is forbidden to do.</p>
|
|
<p>Fault injected, 2026-09-21: moved <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/bundle/wasm_surface.test.js"><code>web/tests/bundle/wasm_surface.test.js</code></a> out of the tree. The test failed with "web/tests/bundle/ is empty — the split lost its other half" rather than passing. Restored; green again.</p>
|
|
<h3 id="no-test-in-web-tests-imports-a-module-that-reaches-the-bundle"><a class="anchor" href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">#</a>"no test in web/tests/ imports a module that reaches the bundle"</h3>
|
|
<p>The companion to the lexical check above, and the reason that one is not enough: <code>import "../app.js"</code> needs the bundle <em>transitively</em> and names nothing forbidden, so the <code>js</code> job breaks loudly but unattributed. <code>app.js</code> and <code>worker.js</code> are the two entry points that load the bundle; <code>shell.js</code> and <code>serial.js</code> do not, and <code>serial.test.js</code> imports one legitimately.</p>
|
|
<p>Fault injected, 2026-09-21: appended <code>import "../app.js";</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The lexical test stayed green; this one failed naming the file. Reverted.</p>
|
|
<h3 id="note-the-checker-s-own-blockers-found-by-review-of-pr-83"><a class="anchor" href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">#</a>Note — the checker's own blockers, found by review of PR #83</h3>
|
|
<p>Two gates in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-comment-runs.py"><code>scripts/check-comment-runs.py</code></a> produced a pass while not doing what their prose said. Both were in the mode the same PR rewrote, which is the part worth keeping: writing the doctrine down does not exempt the writer.</p>
|
|
<p><strong>A leaf name is not an identity.</strong> Six leaf names collide in this workspace and two were cited in evidence. Renaming <code>ui::project::tests::a_load_that_fails_is_not_a_rebase</code> — the one its entry describes — left the checker printing <em>all name a test that runs</em>, exit 0, with the heading answered by the <code>ui::host::tests</code> namesake. Now an ambiguous leaf is refused with both candidates named, and the two entries are qualified. Re-verified both ways, 2026-09-21: the refusal fires on the unqualified headings and names exactly those two; with the heading qualified, the same rename is reported as an orphan rather than absorbed.</p>
|
|
<p><strong><code>--rust-tests</code> was strictly more permissive and said the same words.</strong> It called <code>cargo_test_names</code> with one argument, so the <code>--ignored</code> subtraction never ran, and the success line was byte-identical to the honest path's. Injecting <code>### eyeball_the_rendered_hints_and_pages</code> orphaned on the default path and passed on the flag's. Now the flag requires <code>--rust-tests-ignored</code> (one alone exits 2) and the success line names its source.</p>
|
|
<p><strong>A missing evidence directory exited 0.</strong> The early return preceded the citation scan, so deleting <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> reported nothing while 170 citations pointed at a directory that was not there. Verified 2026-09-21 by moving the directory aside: exit 1, <em>170 tests cite context/test-evidence and have no entry there</em>.</p>
|
|
<h3 id="note-the-flag-that-selected-the-cap-away-improvise-ijql"><a class="anchor" href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">#</a>Note — the flag that selected the cap away (<a class="tid" href="../tickets.html#improvise-ijql" title="check-comment-runs.py --check-evidence silently skips the comment-run cap it is named for">improvise-ijql</a>)</h3>
|
|
<p><code>scripts/check-comment-runs.py --check-evidence</code> early-returned in <code>main</code>, so the flagged run never answered the comment-run cap and printed nothing about having skipped it. The reviewer's position when it was first raised was that CI runs all three spellings (<code>ci.yml</code>, the three <code>check-comment-runs.py</code> steps in the <code>test</code> job) so this was documentation only. The counter that carried is that the flagged form is the <strong>expensive</strong> one — it needs a compiled workspace — which makes it the one a human runs alone; three agents hit it independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment.</p>
|
|
<p>Reproduced before the fix, 2026-09-21, in a temporary tree holding one <code>.rs</code> file with a four-line comment run:</p>
|
|
<pre><code>$ check-comment-runs.py src --check-evidence --rust-tests … --rust-tests-ignored …
|
|
test-evidence: 0 entries, all name a test that runs …
|
|
EXIT=0
|
|
$ check-comment-runs.py src
|
|
1 comment runs longer than 2 lines:
|
|
src/bad.rs:1 4 lines
|
|
EXIT=1</code></pre>
|
|
<p>Two invocations over one tree, opposite verdicts, and the one that exits 0 is the one a human is more likely to run by itself.</p>
|
|
<p>The guard is <code>_dispatch_self_test</code>, which runs the script as a <strong>subprocess</strong> — the only level at which this is visible, since every unit in the file was correct and the defect was in <code>main</code>'s dispatch. Both arms were demonstrated:</p>
|
|
<table><thead><tr><th>injected fault</th><th>what failed</th></tr></thead><tbody><tr><td>restore the <code>--check-evidence</code> early return</td><td><code>DISPATCH: --check-evidence: exited 0 over a four-line comment run</code></td></tr><tr><td>make the cap always report a violation</td><td><code>DISPATCH: bare: refused a clean tree</code> <em>and</em> the same for <code>--check-evidence</code></td></tr></tbody></table>
|
|
<p>The second arm is what stops an always-failing cap from satisfying the first, and it is why <code>SELF_TEST_DISPATCH</code> asserts a clean tree passes as well as a dirty one failing. Why the fix is additive rather than a printed disclaimer, and why <code>--self-test</code>/<code>--list-doctests</code> are excluded from the table rather than exempted from the rule: <a href="../code/tooling.html"><code>context/code/tooling.md</code></a> §2, "A flag adds a question".</p>
|
|
<h3 id="note-the-stamp-gate-s-discrimination-measured"><a class="anchor" href="#note-the-stamp-gate-s-discrimination-measured">#</a>Note — the stamp gate's discrimination, measured</h3>
|
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> separates three outcomes rather than two, which is what makes it evidence rather than a smoke test. Measured 2026-09-21 over its nine cases: the real script passes 9; deleting the final <code>test -z "$missing"</code> (the PR #69 injection, which the inline version survived at exit 0) passes 4 and fails 5; a script that does nothing passes 2; one that always fails passes 0.</p>
|
|
<p>The PR body first cited "4 of 9" for the middle row, from a run taken when the suite had eight cases. A figure is what the next reader checks, so it is recorded here where it can be re-measured rather than only in a message nothing can audit (design-principles §6).</p>
|
|
<h3 id="note-the-stamp-gate-matches-literally"><a class="anchor" href="#note-the-stamp-gate-matches-literally">#</a>Note — the stamp gate matches literally</h3>
|
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp.sh"><code>scripts/assemble-stamp.sh</code></a> greps with <code>-F</code>. As a regex, the <code>.</code> in <code>improvise_wasm_client.js?v=abc</code> matches any character, so a stamp corrupted in exactly that position reads as present — a pass in the one direction a guard must not fail in. Inherited from the inline version this was extracted from, and found by reading the extraction rather than by anything failing.</p>
|
|
<p>Fault injected, 2026-09-21: dropping the <code>-F</code> makes <code>improvise_wasm_clientXjs?v=abc</code> satisfy the pattern, and <code>a pattern is matched literally, not as a regex</code> goes from pass to fail.</p>
|
|
<h3 id="note-the-doc-site-s-derived-checks"><a class="anchor" href="#note-the-doc-site-s-derived-checks">#</a>Note — the doc site's derived checks</h3>
|
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/build-doc-site.mjs"><code>scripts/build-doc-site.mjs</code></a> carries two gates that no test names, because the generator <em>is</em> the test: it runs on every assembly and exits non-zero.</p>
|
|
<p>Faults injected, 2026-09-21, each observed:</p>
|
|
<table><thead><tr><th>injected</th><th>reported</th></tr></thead><tbody><tr><td>a new <code>context/code/newarea.md</code></td><td><code>context/code/newarea.md is neither published nor a named omission</code></td></tr><tr><td><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/plan.md"><code>context/plan.md</code></a> moved out of the tree</td><td><code>context/plan.md is listed as omitted but does not exist</code></td></tr><tr><td>a line appended below the generator in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a></td><td><code>web/assemble.sh must invoke this generator LAST</code>, naming the offending command</td></tr><tr><td>a trailing <code>echo "see scripts/build-doc-site.mjs"</code></td><td>the same, once the check asks for an INVOCATION rather than a mention — <code>.includes</code> had accepted it (review, PR #83)</td></tr></tbody></table>
|
|
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-ci-graph.py"><code>scripts/check-ci-graph.py</code></a> is the third, and it is a script rather than a stanza because its own refusals need checking: adding a <code>zz-new-check:</code> job to <code>ci.yml</code> reports <em>pages-deploy does not need <code>zz-new-check</code></em> and exits 1, where before the rule was prose and nothing noticed. Its five self-test graphs include the one a naive parse gets wrong — another job's <code>needs:</code> read as the deploy's.</p>
|
|
<p>The second arm is the one that would not get written first, and it is the point: without it, deleting or renaming an omitted file leaves an entry asserting a reason for a file nobody can find — the same failure as a dead ticket id, which <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/AGENTS.md"><code>AGENTS.md</code></a> already calls worse than no id.</p>
|
|
<p>The first arm was filed as latent and was live when it was written: <a href="../code/kb-tool.html"><code>context/code/kb-tool.md</code></a> had been unpublished, unargued and unnoticed since it was added.</p>
|
|
<h3 id="duplicate-evidence-headings-self-test-duplicates"><a class="anchor" href="#duplicate-evidence-headings-self-test-duplicates">#</a>Duplicate evidence headings — <code>SELF_TEST_DUPLICATES</code></h3>
|
|
<p><a class="tid" href="../tickets.html#improvise-hsoe" title="check-evidence cannot see a duplicate heading, and three exist">improvise-hsoe</a>. <code>--check-evidence</code> verified that a heading names a test that <strong>runs</strong>, and a duplicate heading names one just as well, so both copies pass. PR #92 shipped two entries under <code>a_layout_change_moves_the_layout_version_and_no_record</code>, and two more pairs sat in <code>tui.md</code> — one of them asserting that <code>THE_MEMBERSHIP_CORRECT_SPANS</code> was "a live tripwire, do not remove it" some four hundred lines above the entry recording that <a class="tid" href="../tickets.html#improvise-gd83" title="AxisEntry carries no group-end boundary, so group_of gives an ungrouped item the preceding group">improvise-gd83</a> had removed it. A reader meeting the older copy first takes it as current, which is worse than an orphan: an orphan announces itself as broken.</p>
|
|
<p>The fault is injected by the six cases in <code>SELF_TEST_DUPLICATES</code>, and three of them are the ones that would not get written first. Two <em>different</em> tests sharing a leaf must come back <strong>ambiguous and not duplicate</strong>, because the wrong diagnosis files the wrong repair. A prose heading must be free to repeat, because it claims no test. And the bare and module-qualified spellings of one test must collide — grouping by the string rather than by the resolved test would have missed exactly the case <a class="tid" href="../tickets.html#improvise-rm4s" title="The evidence check accepts headings naming tests that never run">improvise-rm4s</a>'s qualification repair creates, which is the first way this would have come back.</p>
|
|
<p>The live tree was itself the first injection: before the three merges the check exited 1 naming all three pairs with both line numbers; after them it exits 0. The clean arms (two headings naming two tests, and prose repeated) are what stop an always-refuse from satisfying the rest.</p>
|
|
</main>
|
|
</div>
|
|
</body>
|
|
</html>
|