Files
improvise/site/docs/test-evidence/tooling.html
T
2026-09-21 13:46:31 -07:00

67 lines
22 KiB
HTML

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>The repository&#39;s own gates — improvise docs</title>
<link rel="stylesheet" href="../style.css" />
</head>
<body>
<div class="shell">
<nav class="side"><a class="brand" href="../index.html">improvise docs</a><input type="checkbox" id="navtoggle" class="navtoggle" /><label class="navsummary" for="navtoggle">Pages</label><div class="navlinks"><ul><li><a href="../../index.html">← Project home</a></li><li><a href="../../app/">Browser demo</a></li><li><a href="https://github.com/fiddlerwoaroof/improvise">Source on GitHub</a></li></ul><h3>Start here</h3><ul><li><a href="../design-notes.html">Design notes</a></li><li><a href="../design-principles.html">Design principles</a></li><li><a href="../repo-map.html">Repository map</a></li></ul><h3>Decision records</h3><ul><li><a href="../adr/001-concurrent-edits.html">ADR 001 — Concurrent edits</a></li><li><a href="../adr/002-naming-a-cell.html">ADR 002 — Naming a cell</a></li><li><a href="../adr/003-live-backed-models.html">ADR 003 — Live-backed models</a></li></ul><h3>The arguments, by area</h3><ul><li><a href="../code/core.html">improvise-core</a></li><li><a href="../code/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../code/protocol.html">improvise-protocol</a></li><li><a href="../code/command.html">improvise-command</a></li><li><a href="../code/clients.html">The clients</a></li><li><a href="../code/tui.html">The TUI</a></li><li><a href="../code/gp.html">gp-tree and improvise-gp</a></li><li><a href="../code/kb-tool.html">The kb tool</a></li><li><a href="../code/tooling.html">The repository's own checkers</a></li></ul><h3>Test evidence</h3><ul><li><a href="../test-evidence/core.html">improvise-core</a></li><li><a href="../test-evidence/formula-and-io.html">improvise-formula and improvise-io</a></li><li><a href="../test-evidence/protocol.html">improvise-protocol</a></li><li><a href="../test-evidence/command.html">improvise-command</a></li><li><a href="../test-evidence/clients.html">The clients</a></li><li><a href="../test-evidence/tui.html">The TUI</a></li><li><a href="../test-evidence/gp.html">gp-tree and improvise-gp</a></li><li><a href="../test-evidence/tooling.html" class="here">The repository's own gates</a></li></ul><h3>About this site</h3><ul><li><a href="../doc-site.html">About this site</a></li></ul><h3>Reference</h3><ul><li><a href="../tickets.html">Ticket index</a></li></ul></div></nav>
<main>
<p class="src">Source: <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/test-evidence/tooling.md">context/test-evidence/tooling.md</a></p><details class="toc" open><summary>On this page</summary><ul><li class="l1"><a href="#test-evidence-the-repository-s-own-gates">Test evidence — the repository&#39;s own gates</a></li><li class="l3"><a href="#no-test-in-web-tests-reaches-for-a-built-bundle">&quot;no test in web/tests/ reaches for a built bundle&quot;</a></li><li class="l3"><a href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">&quot;the subdirectory that MAY use a bundle is not the one being checked&quot;</a></li><li class="l3"><a href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">&quot;no test in web/tests/ imports a module that reaches the bundle&quot;</a></li><li class="l3"><a href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">Note — the checker&#39;s own blockers, found by review of PR #83</a></li><li class="l3"><a href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">Note — the flag that selected the cap away (improvise-ijql)</a></li><li class="l3"><a href="#note-the-stamp-gate-s-discrimination-measured">Note — the stamp gate&#39;s discrimination, measured</a></li><li class="l3"><a href="#note-the-stamp-gate-matches-literally">Note — the stamp gate matches literally</a></li><li class="l3"><a href="#note-the-doc-site-s-derived-checks">Note — the doc site&#39;s derived checks</a></li><li class="l3"><a href="#duplicate-evidence-headings-self-test-duplicates">Duplicate evidence headings — SELF_TEST_DUPLICATES</a></li></ul></details><h1 id="test-evidence-the-repository-s-own-gates">Test evidence — the repository&#39;s own gates</h1>
<p>Evidence for the checks in <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/scripts"><code>scripts/</code></a> and for the tests that hold CI&#39;s shape. The arguments behind the checkers themselves are in <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>; this file records what fault each test was watched to catch.</p>
<p>Note what is deliberately absent. <code>scripts/check-comment-runs.py --self-test</code>, <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/kb-test.sh"><code>scripts/kb-test.sh</code></a> and <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> are check-the-checker suites whose cases are already injected faults, and none of their case names is a Rust <code>fn</code> or a JS <code>test(</code> the evidence checker could resolve — so their evidence is the case list itself plus <a href="../code/tooling.html"><code>context/code/tooling.md</code></a>, and inventing headings for them here would be headings naming nothing.</p>
<h3 id="no-test-in-web-tests-reaches-for-a-built-bundle"><a class="anchor" href="#no-test-in-web-tests-reaches-for-a-built-bundle">#</a>&quot;no test in web/tests/ reaches for a built bundle&quot;</h3>
<p>The <code>js</code> CI job runs <code>node --test web/tests/*.test.js</code> and installs nothing; the <code>browser</code> job runs <code>web/tests/bundle/*.test.js</code> after three wasm-pack builds. The moment a flat <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests"><code>web/tests/</code></a> file reads <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/pkg"><code>web/pkg/</code></a>, the fast job inherits the slow one&#39;s dependency — and the likely repair is to add a build step to it, silently spending the ~4m30s the split exists to avoid. Until <a class="tid" href="../tickets.html#improvise-4ktt" title="The js job&#39;s bundle-freedom is a convention, not a gate">improvise-4ktt</a> that constraint was a directory convention and a glob, checked by nothing, and it failed in the direction that looks like ordinary breakage: a confusing ENOENT in CI on a machine where the bundle happens not to be built.</p>
<p>Fault injected, 2026-09-21: appended <code>const injected = readFileSync(&quot;../pkg/improvise_wasm_client.js&quot;);</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The test failed naming that file and pointing at <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a>. Reverted; green again.</p>
<p>Two things the test does that a simpler one would not. It asserts its own filename is among the files it scans, so it cannot become the one place a violation may hide. And it strips comments before matching, because this file&#39;s own header and the test&#39;s header both name the directory in prose — a lexical grep over raw text would have had to exempt them, and an exemption is where the next violation goes.</p>
<p><strong>What it cannot see</strong>, stated because the check itself uses the evasion: a path assembled at runtime (<code>&quot;../&quot; + &quot;p&quot; + &quot;kg&quot;</code>) is invisible to a lexical match, and the test&#39;s own <code>FORBIDDEN</code> constant is built exactly that way to stay out of its own scan. That is a deliberate, disclosed hole — closing it needs a parser, and the failure it would catch is one somebody has to write on purpose, where this gate exists for the one written by accident.</p>
<h3 id="the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked"><a class="anchor" href="#the-subdirectory-that-may-use-a-bundle-is-not-the-one-being-checked">#</a>&quot;the subdirectory that MAY use a bundle is not the one being checked&quot;</h3>
<p>The companion, and the one that stops the first from passing vacuously. If <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/web/tests/bundle"><code>web/tests/bundle/</code></a> were emptied or its files stopped touching the bundle, the first test would still be green while having nothing to be scoped <em>against</em> — &quot;no flat test uses the bundle&quot; is uninteresting if no test anywhere does. This asserts the other half of the split exists and really does what the flat half is forbidden to do.</p>
<p>Fault injected, 2026-09-21: moved <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/bundle/wasm_surface.test.js"><code>web/tests/bundle/wasm_surface.test.js</code></a> out of the tree. The test failed with &quot;web/tests/bundle/ is empty — the split lost its other half&quot; rather than passing. Restored; green again.</p>
<h3 id="no-test-in-web-tests-imports-a-module-that-reaches-the-bundle"><a class="anchor" href="#no-test-in-web-tests-imports-a-module-that-reaches-the-bundle">#</a>&quot;no test in web/tests/ imports a module that reaches the bundle&quot;</h3>
<p>The companion to the lexical check above, and the reason that one is not enough: <code>import &quot;../app.js&quot;</code> needs the bundle <em>transitively</em> and names nothing forbidden, so the <code>js</code> job breaks loudly but unattributed. <code>app.js</code> and <code>worker.js</code> are the two entry points that load the bundle; <code>shell.js</code> and <code>serial.js</code> do not, and <code>serial.test.js</code> imports one legitimately.</p>
<p>Fault injected, 2026-09-21: appended <code>import &quot;../app.js&quot;;</code> to <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/tests/grid_plan.test.js"><code>web/tests/grid_plan.test.js</code></a>. The lexical test stayed green; this one failed naming the file. Reverted.</p>
<h3 id="note-the-checker-s-own-blockers-found-by-review-of-pr-83"><a class="anchor" href="#note-the-checker-s-own-blockers-found-by-review-of-pr-83">#</a>Note — the checker&#39;s own blockers, found by review of PR #83</h3>
<p>Two gates in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-comment-runs.py"><code>scripts/check-comment-runs.py</code></a> produced a pass while not doing what their prose said. Both were in the mode the same PR rewrote, which is the part worth keeping: writing the doctrine down does not exempt the writer.</p>
<p><strong>A leaf name is not an identity.</strong> Six leaf names collide in this workspace and two were cited in evidence. Renaming <code>ui::project::tests::a_load_that_fails_is_not_a_rebase</code> — the one its entry describes — left the checker printing <em>all name a test that runs</em>, exit 0, with the heading answered by the <code>ui::host::tests</code> namesake. Now an ambiguous leaf is refused with both candidates named, and the two entries are qualified. Re-verified both ways, 2026-09-21: the refusal fires on the unqualified headings and names exactly those two; with the heading qualified, the same rename is reported as an orphan rather than absorbed.</p>
<p><strong><code>--rust-tests</code> was strictly more permissive and said the same words.</strong> It called <code>cargo_test_names</code> with one argument, so the <code>--ignored</code> subtraction never ran, and the success line was byte-identical to the honest path&#39;s. Injecting <code>### eyeball_the_rendered_hints_and_pages</code> orphaned on the default path and passed on the flag&#39;s. Now the flag requires <code>--rust-tests-ignored</code> (one alone exits 2) and the success line names its source.</p>
<p><strong>A missing evidence directory exited 0.</strong> The early return preceded the citation scan, so deleting <a href="https://github.com/fiddlerwoaroof/improvise/tree/main/context/test-evidence"><code>context/test-evidence/</code></a> reported nothing while 170 citations pointed at a directory that was not there. Verified 2026-09-21 by moving the directory aside: exit 1, <em>170 tests cite context/test-evidence and have no entry there</em>.</p>
<h3 id="note-the-flag-that-selected-the-cap-away-improvise-ijql"><a class="anchor" href="#note-the-flag-that-selected-the-cap-away-improvise-ijql">#</a>Note — the flag that selected the cap away (<a class="tid" href="../tickets.html#improvise-ijql" title="check-comment-runs.py --check-evidence silently skips the comment-run cap it is named for">improvise-ijql</a>)</h3>
<p><code>scripts/check-comment-runs.py --check-evidence</code> early-returned in <code>main</code>, so the flagged run never answered the comment-run cap and printed nothing about having skipped it. The reviewer&#39;s position when it was first raised was that CI runs all three spellings (<code>ci.yml</code>, the three <code>check-comment-runs.py</code> steps in the <code>test</code> job) so this was documentation only. The counter that carried is that the flagged form is the <strong>expensive</strong> one — it needs a compiled workspace — which makes it the one a human runs alone; three agents hit it independently in one campaign, and in one case a local gate went green while CI went red on a three-line doc comment.</p>
<p>Reproduced before the fix, 2026-09-21, in a temporary tree holding one <code>.rs</code> file with a four-line comment run:</p>
<pre><code>$ check-comment-runs.py src --check-evidence --rust-tests … --rust-tests-ignored …
test-evidence: 0 entries, all name a test that runs …
EXIT=0
$ check-comment-runs.py src
1 comment runs longer than 2 lines:
src/bad.rs:1 4 lines
EXIT=1</code></pre>
<p>Two invocations over one tree, opposite verdicts, and the one that exits 0 is the one a human is more likely to run by itself.</p>
<p>The guard is <code>_dispatch_self_test</code>, which runs the script as a <strong>subprocess</strong> — the only level at which this is visible, since every unit in the file was correct and the defect was in <code>main</code>&#39;s dispatch. Both arms were demonstrated:</p>
<table><thead><tr><th>injected fault</th><th>what failed</th></tr></thead><tbody><tr><td>restore the <code>--check-evidence</code> early return</td><td><code>DISPATCH: --check-evidence: exited 0 over a four-line comment run</code></td></tr><tr><td>make the cap always report a violation</td><td><code>DISPATCH: bare: refused a clean tree</code> <em>and</em> the same for <code>--check-evidence</code></td></tr></tbody></table>
<p>The second arm is what stops an always-failing cap from satisfying the first, and it is why <code>SELF_TEST_DISPATCH</code> asserts a clean tree passes as well as a dirty one failing. Why the fix is additive rather than a printed disclaimer, and why <code>--self-test</code>/<code>--list-doctests</code> are excluded from the table rather than exempted from the rule: <a href="../code/tooling.html"><code>context/code/tooling.md</code></a> §2, &quot;A flag adds a question&quot;.</p>
<h3 id="note-the-stamp-gate-s-discrimination-measured"><a class="anchor" href="#note-the-stamp-gate-s-discrimination-measured">#</a>Note — the stamp gate&#39;s discrimination, measured</h3>
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp-test.sh"><code>scripts/assemble-stamp-test.sh</code></a> separates three outcomes rather than two, which is what makes it evidence rather than a smoke test. Measured 2026-09-21 over its nine cases: the real script passes 9; deleting the final <code>test -z &quot;$missing&quot;</code> (the PR #69 injection, which the inline version survived at exit 0) passes 4 and fails 5; a script that does nothing passes 2; one that always fails passes 0.</p>
<p>The PR body first cited &quot;4 of 9&quot; for the middle row, from a run taken when the suite had eight cases. A figure is what the next reader checks, so it is recorded here where it can be re-measured rather than only in a message nothing can audit (design-principles §6).</p>
<h3 id="note-the-stamp-gate-matches-literally"><a class="anchor" href="#note-the-stamp-gate-matches-literally">#</a>Note — the stamp gate matches literally</h3>
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/assemble-stamp.sh"><code>scripts/assemble-stamp.sh</code></a> greps with <code>-F</code>. As a regex, the <code>.</code> in <code>improvise_wasm_client.js?v=abc</code> matches any character, so a stamp corrupted in exactly that position reads as present — a pass in the one direction a guard must not fail in. Inherited from the inline version this was extracted from, and found by reading the extraction rather than by anything failing.</p>
<p>Fault injected, 2026-09-21: dropping the <code>-F</code> makes <code>improvise_wasm_clientXjs?v=abc</code> satisfy the pattern, and <code>a pattern is matched literally, not as a regex</code> goes from pass to fail.</p>
<h3 id="note-the-doc-site-s-derived-checks"><a class="anchor" href="#note-the-doc-site-s-derived-checks">#</a>Note — the doc site&#39;s derived checks</h3>
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/build-doc-site.mjs"><code>scripts/build-doc-site.mjs</code></a> carries two gates that no test names, because the generator <em>is</em> the test: it runs on every assembly and exits non-zero.</p>
<p>Faults injected, 2026-09-21, each observed:</p>
<table><thead><tr><th>injected</th><th>reported</th></tr></thead><tbody><tr><td>a new <code>context/code/newarea.md</code></td><td><code>context/code/newarea.md is neither published nor a named omission</code></td></tr><tr><td><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/context/plan.md"><code>context/plan.md</code></a> moved out of the tree</td><td><code>context/plan.md is listed as omitted but does not exist</code></td></tr><tr><td>a line appended below the generator in <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/web/assemble.sh"><code>web/assemble.sh</code></a></td><td><code>web/assemble.sh must invoke this generator LAST</code>, naming the offending command</td></tr><tr><td>a trailing <code>echo &quot;see scripts/build-doc-site.mjs&quot;</code></td><td>the same, once the check asks for an INVOCATION rather than a mention — <code>.includes</code> had accepted it (review, PR #83)</td></tr></tbody></table>
<p><a href="https://github.com/fiddlerwoaroof/improvise/blob/main/scripts/check-ci-graph.py"><code>scripts/check-ci-graph.py</code></a> is the third, and it is a script rather than a stanza because its own refusals need checking: adding a <code>zz-new-check:</code> job to <code>ci.yml</code> reports <em>pages-deploy does not need <code>zz-new-check</code></em> and exits 1, where before the rule was prose and nothing noticed. Its five self-test graphs include the one a naive parse gets wrong — another job&#39;s <code>needs:</code> read as the deploy&#39;s.</p>
<p>The second arm is the one that would not get written first, and it is the point: without it, deleting or renaming an omitted file leaves an entry asserting a reason for a file nobody can find — the same failure as a dead ticket id, which <a href="https://github.com/fiddlerwoaroof/improvise/blob/main/AGENTS.md"><code>AGENTS.md</code></a> already calls worse than no id.</p>
<p>The first arm was filed as latent and was live when it was written: <a href="../code/kb-tool.html"><code>context/code/kb-tool.md</code></a> had been unpublished, unargued and unnoticed since it was added.</p>
<h3 id="duplicate-evidence-headings-self-test-duplicates"><a class="anchor" href="#duplicate-evidence-headings-self-test-duplicates">#</a>Duplicate evidence headings — <code>SELF_TEST_DUPLICATES</code></h3>
<p><a class="tid" href="../tickets.html#improvise-hsoe" title="check-evidence cannot see a duplicate heading, and three exist">improvise-hsoe</a>. <code>--check-evidence</code> verified that a heading names a test that <strong>runs</strong>, and a duplicate heading names one just as well, so both copies pass. PR #92 shipped two entries under <code>a_layout_change_moves_the_layout_version_and_no_record</code>, and two more pairs sat in <code>tui.md</code> — one of them asserting that <code>THE_MEMBERSHIP_CORRECT_SPANS</code> was &quot;a live tripwire, do not remove it&quot; some four hundred lines above the entry recording that <a class="tid" href="../tickets.html#improvise-gd83" title="AxisEntry carries no group-end boundary, so group_of gives an ungrouped item the preceding group">improvise-gd83</a> had removed it. A reader meeting the older copy first takes it as current, which is worse than an orphan: an orphan announces itself as broken.</p>
<p>The fault is injected by the six cases in <code>SELF_TEST_DUPLICATES</code>, and three of them are the ones that would not get written first. Two <em>different</em> tests sharing a leaf must come back <strong>ambiguous and not duplicate</strong>, because the wrong diagnosis files the wrong repair. A prose heading must be free to repeat, because it claims no test. And the bare and module-qualified spellings of one test must collide — grouping by the string rather than by the resolved test would have missed exactly the case <a class="tid" href="../tickets.html#improvise-rm4s" title="The evidence check accepts headings naming tests that never run">improvise-rm4s</a>&#39;s qualification repair creates, which is the first way this would have come back.</p>
<p>The live tree was itself the first injection: before the three merges the check exited 1 naming all three pairs with both line numbers; after them it exits 0. The clean arms (two headings naming two tests, and prose repeated) are what stop an always-refuse from satisfying the rest.</p>
</main>
</div>
</body>
</html>