docs: what the harness now reaches, and the evidence for each assertion

The engine-coverage and measurement passages in context/code/clients.md, the
boundary table (four boundaries become six), and README's caveat -- which
named the grouped and multi-level DOM as never executed, and it now is. A fact
stated in more than one place is a grep, not an edit (gotcha 48).

context/test-evidence/clients.md records, per test, what fault was injected to
prove it can fail: the blanked help keys, the five grid-frame injections, the
404'd casts and the dead front-door link, and the mis-wired item-row click --
plus the two-command test-name diff that found that last one.

Playwright: 26 passed + 3 skipped, 34.4s / 33.7s (was 17 + 3, 15.1s); the
landing page's cast playback is most of the difference. node --test web/tests
26, cargo test --workspace 1394, both unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FC2jVn8Au7E9JPkw6bwXXH
This commit is contained in:
Edward Langley
2026-09-20 16:34:31 -07:00
parent ab96f3651f
commit f9580a33af
3 changed files with 159 additions and 12 deletions
+7 -4
View File
@@ -429,10 +429,13 @@ binary whose job is a pivot table". If sharing a TUI-hosted document to a
remote browser ever becomes a common case rather than an occasional one, this
is worth revisiting on that evidence.
Caveat, narrowed: `harness/` runs `web/app.js` under Chromium and `OpfsFs` under
Chromium and Firefox, in CI. What is still never executed is `bindings.rs`, the
grouped and multi-level DOM, the CSS and the `ResizeObserver` arithmetic — and
Safari, whose Playwright build provides no OPFS at all.
Caveat, narrowed twice: `harness/` runs `web/app.js` under Chromium — the
standalone deployment under Firefox too — and `OpfsFs` under Chromium and
Firefox, in CI. The grouped and multi-level DOM is executed since
improvise-plcx, against a fixture document the harness serves beside the demo.
What is still never executed is `bindings.rs`, the CSS and the
`ResizeObserver` arithmetic — and Safari, whose Playwright build provides no
OPFS at all.
## What's interesting about the architecture
+30 -8
View File
@@ -1253,9 +1253,21 @@ OPFS it fails rather than skipping. An unexpected skip on any other engine is a
failure — a silently skipped check is exactly what this harness exists to
prevent.
The shell spec is **Chromium-only on purpose**: it is about `web/app.js`'s own
logic, which is engine-independent, and `routeWebSocket` plus the worker
handshake are slower and flakier to no benefit across three.
The shell spec's **mocked-peer** block is Chromium-only on purpose: it is about
`web/app.js`'s own logic, which is engine-independent, and `routeWebSocket` is
slower and flakier to no benefit across three. That argument is about those four
tests and was never made about the two beside them: the **standalone** block
mocks nothing, it is the deployment improvise-bck's criterion (1) names in so
many words ("Chrome and Firefox"), and the engine's OPFS is already exercised by
`opfs.spec.js`, so the path is known to be there. It runs on Firefox too since
improvise-zkkw — `testMatch` widened, `grepInvert: /@mocked-peer/` — measured at
**2.0s** for the pair and 10/10 green under `--repeat-each=5`. The split is a
tag rather than a third copy of the file, and widening the whole file and then
quieting what flaked would be adopting a reason nobody gave.
Every other spec is Chromium-only, including the three added by improvise-plcx,
78v6 and bgtf: a fixture document, a help overlay and a landing page decide
nothing per engine.
### `harness/opfs-contract`
@@ -1296,14 +1308,17 @@ to check that `refresh` skips one, has to happen outside it.
**E2E and integration tests are rare and pin boundary contracts; unit tests
assume those contracts and test between them.** (User ruling, 2026-09-20.) So
the question is not "can this assertion be made cheaper" but "which boundary is
it about", and the browser half has four:
it about", and the browser half has six:
| boundary | the contract | where it is pinned |
|---|---|---|
| `RenderModel` → DOM | every field the render model projects reaches an element | `web/tests/render_surface.test.js` (statically) **and** `harness/tests/contract.spec.js` (on the screen) |
| JS → wasm | the calls the page makes exist, with the arity the bundle declares, and their answers are painted | `web/tests/bundle/wasm_surface.test.js` (the surface) **and** `contract.spec.js`'s pointer test (the wiring) |
| page → peer | the handshake goes out first, a frame is answered whole, a failure is named, **and the page routes each failure through the call that disarms the watchdog** | `harness/tests/shell.spec.js`, `harness/tests/drill.spec.js` |
| `help::pages()` → DOM | the overlay's rows are the shared content — the keys off the keymap and the words beside them — and not the right number of blank rows | `harness/tests/help.spec.js` |
| worker → OPFS | `storage_contract`, run against a real `OpfsFs` | `harness/tests/opfs.spec.js` |
| `AxisFrame` → DOM | the grid's *frame* — header levels, group spans, the collapse indicator and the group row that carries no cells — is drawn as the frame says, not slash-joined | `harness/tests/grouped.spec.js`, over `harness/fixtures/grouped.improv`; `web/tests/grid_plan.test.js` pins the plan it draws from |
| the site → the app | the landing page's casts are really fetched and really play, and its front door reaches `./app/` | `harness/tests/landing.spec.js` |
**A unit test of a rule does not test the wiring that reaches it**, and the
sharpest case in this file is the watchdog. `shell.test.js` proves that
@@ -1399,12 +1414,19 @@ trap below). Two runs each; the pair is the variance.
|---|---|---|---|---|
| Playwright, before (`1af51c9`) | 23 | 26 passed + 3 skipped | 36.2s / 36.3s | ~4m30s in CI |
| Playwright, after | 14 | 17 passed + 3 skipped | 15.1s / 14.9s | ~4m30s in CI |
| Playwright, + wave 4 | 21 | 26 passed + 3 skipped | 34.4s / 33.7s | ~4m30s in CI |
| `node --test web/tests` | — | 26 | 0.12s | seconds |
**Two columns because one number was ambiguous three times.** Every spec but
`opfs.spec.js` is Chromium-only, so a count taken off a run is three larger than
the count of tests in the files (Firefox's OPFS battery) with three more skipped
(WebKit's). The columns must also **agree**: chromium + 3 is the passed total,
The last row is improvise-78v6/igxp/plcx/bgtf/zkkw: seven more Chromium tests
and two Firefox ones. **Most of the 19 seconds is one test** — the landing
page's four asciinema casts are played until each terminal draws, which is the
only assertion that can tell a cast from a 404 (evidence file). The Firefox pair
is 2.0s of it.
**Two columns because one number was ambiguous three times.** Firefox runs
`opfs.spec.js` and the standalone half of `shell.spec.js` and every other spec
is Chromium-only, so a count taken off a run is five larger than the chromium
count, with three more skipped (WebKit's OPFS battery). The columns must also **agree**: chromium + 3 is the passed total,
and a row where they do not is a row built from two different trees.
`npx playwright test --list` prints the per-project truth; quote which one, and
this is the same "two numbers, not one" the Rust suite learned in
+122
View File
@@ -1703,3 +1703,125 @@ not the text, so a cell drawn blank passed. The assertion was strengthened until
the injection failed, which is the only reason it is worth having — and it is
the same lesson as the eleven tests above: a class is a flag, and a flag is not
a pixel.
---
## `harness/tests/help.spec.js` — the overlay, asserted as words
PR #61 deleted `"? opens help, and its rows are the shared ones"` and nothing
replaced it, so `paintHelp`, `help`, `help_page` and `help_titles` reached no
assertion at all (improvise-78v6). What `contract.spec.js` carried in its place
said only that a `dt` was VISIBLE — and an empty `dt` is a visible `dt`
(improvise-igxp), which is the same shape as the caret bug above: a row is a
frame, and a frame is not a word.
**The injection both tickets name**: blank `keys.textContent` in `paintHelp`,
patched in `harness/serve/app/app.js`. Before this work it left all three
contract tests passing. It now fails both tests in this file (the Welcome
banner reads empty; the advertised key reads empty) **and** `contract.spec.js`,
whose two help lines were strengthened from `toBeVisible` to the banner's text
and the first `dt.help-keys`'s text.
**What the pairs helper is for.** The overlay is a `<dl>`, and the association
between a key row and its description is document order — `dt` then `dd`. Read
as two lists they zip wrong, because a prose row emits **no** `dd` at all
(deliberately: an empty one would leave a blank grid cell). Welcome is prose end
to end, so `dd` count zero there is an assertion about that rule and not a
coincidence.
#### "a key the overlay advertises does what the overlay says it does"
The assertions above are still about a string on a screen; this one asks whether
the string is **true**. The key for `Jump to last row` is read OUT OF THE DOM,
the overlay closed, the key pressed, and the selection asserted onto the last
row. A page drawing a plausible key it does not have — what a hand-spelled table
decays into — fails here, and the spec holds no second copy of the keymap.
It presses the FIRST key the row names because a row is a compaction of every
binding that reaches the command (`h j k l ← ↑ → ↓` is eight).
## `harness/tests/grouped.spec.js` — the first browser run of improvise-anr9
`web/demo.improv` has no groups and one category per visible axis, deliberately
(gotcha 46), so the multi-level header, the group span, the collapse indicator
and the inline group row — exactly the code improvise-anr9 added — had never
been rendered anywhere a test could see (improvise-plcx). The fixture is
`harness/fixtures/grouped.improv`, served beside the shipped demo by
`harness/build.sh`, and the shipped document is unchanged.
**A missing seed is not an error**: `web/worker.js` answers a fetch that 404s
with an empty model. So a typo in the filename yields a page that draws a grid,
and "a grid is drawn" would pass on it. Every assertion here is about labels the
fixture really carries.
**Five injections, five distinct failures**, patched in `harness/serve/` so no
rebuild stands between the fault and the run:
- `gridPlan` slash-joins the level tuples into one line (the exact defect anr9
fixed) — fails the header-levels test and the span test;
- `th.colSpan = 1` on every column group header — fails the span test alone;
- a stray `tr.insertCell()` on the inline group row — fails the group-row test
alone, which is the DOM half of `RenderRow::Group` carrying no cells;
- `indicator(false)` hard-coded — fails on `▶ Upland` while `▼ Inland` still
passes, which is why the fixture has one collapsed group and one open one;
- `collapsed: Region/Upland` removed from the served document — fails on the
rows that reappear, so the assertions see the collapse and not just the glyph.
**A bug this fixture surfaced rather than predicted.** `Coastal` is the row
frame's FIRST entry and its header is never drawn: the spec asserts **two**
group rows where the document has three, with improvise-0o6x named on the line.
That count is a record of an open bug, not an endorsement — when 0o6x lands it
becomes 3 and this line is where the change must be argued.
## `harness/tests/landing.spec.js` — the front door
`docs/index.html` is copied to the site root by `web/assemble.sh`, so the
harness has been serving it all along with nothing asserted about it
(improvise-bgtf).
**Counting the four `.ap-player` elements proves nothing**, and that is the
whole reason this is a spec rather than a one-liner: `autoPlay` is false, so
asciinema-player fetches no cast until play is pressed. A page whose every cast
path 404s constructs four players, throws no page error, and looks identical to
a working one. Injected while the page was being written (`./casts/` ->
`./cats/`) the construction-only check passed on the broken page.
Re-injected into `harness/serve/index.html` now, the spec fails at the first
player: `.ap-line` elements exist only once there is a cast to lay out. The spec
also asserts, BEFORE pressing anything, that no `.cast` has been requested —
without which the network evidence below could be satisfied by a page that
fetched them eagerly, which is not the page we ship.
Two assertions where one looks sufficient: the rendering says a cast played, and
the recorded responses say **four distinct** casts answered 200. Three players
showing the same recording satisfies the first and fails the second.
`"the front door reaches the app it advertises"` follows the footer link rather
than asserting its text, because whether `./app/` resolves is a fact about the
assembled tree. Injecting `href="./aplication/"` fails it.
## `harness/tests/contract.spec.js` — the ninth injection, and a lost guard
**The lost guard.** improvise-78v6 asks for the test-name diff over PR #61's
range, and it finds a fourth loss beyond the help test: `"a click on an item row
filters, as Enter does"`. `contract.spec.js` asserts the CATEGORY arm of
`click_category_row` (the row expands) and nothing asserts the ITEM arm, whose
rule is "do what Enter does here" — filter. Only `#status` distinguishes them,
so a click wired to the wrong arm passed the whole file. Restored inside the
click test, and proved by injecting `click_category(row.is_category ? index :
0)`, which fails that line alone.
The diff that finds this is two commands, and is worth running on any PR whose
subject is a lost guard:
```sh
M=$(gh pr view 61 --json mergeCommit -q .mergeCommit.oid)
for r in "$M^1" "$M"; do
git ls-tree -r --name-only "$r" -- harness/tests web/tests |
while read -r f; do git show "$r:$f" | grep -oE 'test\("[^"]*"'; done
done
```
The other two losses are deliberate: `pointer.spec.js`'s sidebar test was
superseded by the open-at-launch ruling, and the widgets file was merged into
this one.