name: CI on: # `push` unrestricted plus `pull_request` ran every job TWICE on any branch # with an open PR — visible as the (push) / (pull_request) pairs in the checks # list. Restricting push to main means a PR branch is covered once by # pull_request, and main once by push, which is also what pages-deploy needs. push: branches: [main] pull_request: env: CARGO_TERM_COLOR: always # Every job runs on the self-hosted NixOS microVM runners, and every `run:` # step inside the flake's `ci` dev shell. Generic-Linux downloads (rustup, # setup-node's node, Playwright's browsers) cannot run on NixOS, so the # toolchain comes from flake.nix, the same one `nix develop` gives locally. # Each step re-enters the shell; after the first, the eval cache makes it cheap. defaults: run: shell: nix develop .#ci --command bash --noprofile --norc -eo pipefail {0} jobs: # --------------------------------------------------------------------------- # The one place wasm-pack runs (improvise CI dedup). # # Before this job there were two: `pages-build` ran `web/build.sh`, and the # browser harness ran `harness/build.sh`, which CALLS `web/build.sh`. The two # wasm-pack builds were therefore done twice per run, in parallel jobs, with # separate rust-cache keys (`add-job-id-key` is on by default, so `browser` # and `pages-build` never shared a target dir). Measured on run 35535225637 # (push to main, caches warm): 68s inside `pages-build`, and the same 68s # inside the browser job's 110s build step. # # Everything downstream now DOWNLOADS. The invariant that buys is artifact # identity: the harness and Pages are served bytes from one wasm-pack run, # so "the harness runs against the tree Pages publishes" (improvise-vvl3) is # true by construction rather than because two scripts agree. # # The native binary is deliberately NOT built here. Nothing downstream # consumes one — `cargo test --workspace` in the `test` job builds its own, # for its own target dir and its own cache key, and a `needs:` on this job # would serialise a 109s job behind a 126s one for no artifact at all. build: name: Build all bundles runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # First, because it needs nothing and takes a second: the gate that stops # an unstamped bundle shipping was deleted by injection on PR #69 and the # assembly still exited 0 (improvise-w8ih). It guards the assemble step # below, so it fails under this job's name rather than another's. - name: The stamp gate refuses what it claims to run: ./scripts/checkers.sh assemble-stamp # Built by Nix (packages.web and packages.site in nix/ci.nix), which runs # the same harness/build.sh and web/assemble.sh a developer runs, so there # is still one answer to "how is the bundle built". `web` is the three # wasm bundles plus `harness/serve`; `site` re-assembles from web's pkg # dirs with the commit SHA as the cache-bust token (improvise-9x6), where # the harness tree is stamped `harness`. Dependencies are their own # derivations, substituted from titan's cache when Cargo.lock is unchanged. - name: Build the bundles, the harness tree and the site run: | nix build --print-build-logs --out-link result-web .#web nix build --print-build-logs --out-link result-site .#site # The upload steps below read these paths. Copied rather than linked: # an artifact of a symlink into /nix/store is a dangling link elsewhere. - name: Lay the outputs out where the uploads read them run: | rm -rf harness/serve web/pkg web/worker-pkg site cp -rL result-web/harness-serve harness/serve cp -rL result-web/pkg web/pkg cp -rL result-web/worker-pkg web/worker-pkg cp -rL result-site site chmod -R u+w harness/serve web/pkg web/worker-pkg site - run: find site -type f | sort - run: du -sh site # The tree the browser harness serves. Uploaded whole rather than # reassembled downstream: a second assembly path is a second answer to # "what does the harness serve", which is the thing improvise-vvl3 exists # to prevent. - name: Upload the harness tree uses: actions/upload-artifact@v7 with: name: harness-serve path: harness/serve retention-days: 1 include-hidden-files: true # For a job that needs the raw wasm-pack output rather than the # assembled site — the Node bundle suite under `web/tests/bundle/`. - name: Upload the wasm bundles uses: actions/upload-artifact@v7 with: name: wasm-bundles path: | web/pkg web/worker-pkg retention-days: 1 include-hidden-files: true - uses: actions/upload-pages-artifact@v5 with: path: site test: name: Test and lint (host) runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # NOT `cargo fmt --all`, which reaches only ONE of the two workspace # roots. Exit 2 here is a usage error, never a verdict (improvise-b4ks). - name: Every workspace root is formatted run: ./scripts/fmt-workspaces.sh # checks.clippy and checks.test in nix/ci.nix: the same cargo commands, # on the dev/test profiles, with dependencies as a cached derivation. - name: cargo clippy --workspace --all-targets -- -D warnings run: nix build --print-build-logs .#checks.x86_64-linux.clippy # Also runs the test-evidence gate (check-comment-runs.py # --check-evidence), which lists the tests this derivation compiled. - name: cargo test --workspace, and test-evidence headings name real tests run: nix build --print-build-logs .#checks.x86_64-linux.test # The comment cap and the test-evidence keys (improvise-9c8c); without # this step the rule is advisory. See context/design-principles.md §6. - name: The comment checker catches what it has been fixed for run: python3 scripts/check-comment-runs.py --self-test - name: Comment runs capped at two lines run: python3 scripts/check-comment-runs.py # `pages-deploy` needs every other job, and saying so was not enough: a # job added during review of PR #83 was ungated and nothing noticed. - name: The deploy gate refuses what it claims to run: python3 scripts/check-ci-graph.py --self-test - name: Every check job gates the deploy run: python3 scripts/check-ci-graph.py # --------------------------------------------------------------------------- # The crate2nix build (Cargo.nix), which `cargo` never exercises. It was off # because GitHub-hosted runners start with an empty store and rebuilt every # dependency crate each run. The NixOS runners substitute from titan's # harmonia first, and titan builds gh/main's `#improvise` hourly, so a PR that # leaves Cargo.lock alone rebuilds only the workspace crates it changed. nix-build: name: Nix build (crate2nix) runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 - name: nix build .#improvise run: nix build --print-build-logs .#improvise wasm: name: wasm32 check (browser bundle crates) runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # checks.wasm-check in nix/ci.nix: `cargo check --target wasm32` for each # crate in the browser bundle, one at a time, so features are not unified # across them. The root `improvise` crate is the terminal binary and # depends on crossterm, which does not build for wasm — that is correct, # not a gap (improvise-ywd). The log names each crate as it is checked. - name: cargo check --target wasm32-unknown-unknown, per bundle crate run: nix build --print-build-logs .#checks.x86_64-linux.wasm-check # --------------------------------------------------------------------------- # The browser JavaScript's unit tests. Its own job, deliberately: a failure # here is a decision `web/shell.js` or `web/serial.js` got wrong, and # reporting that as "Browser harness" would put it behind an 80 MB browser # download and two wasm-pack builds. It needs neither — `node --test` is in # the runtime, which the dev shell supplies; this job installs nothing. js: name: Browser JS unit tests (node --test) runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # The glob is the SCRIPT's, not the shell's and not node's. Under bash an # unmatched glob survives as a literal, node globs it, matches nothing and # exits 0 having run no test -- so this step gated nothing the moment # `web/tests/` moved (improvise-vz54). `web/tests/bundle/` needs a built # wasm bundle and runs in the browser job instead. - name: What the page decides between its boundaries run: ./scripts/node-test.sh 'web/tests/*.test.js' # The guard above is only worth its line if its refusals are themselves # checked -- the step it replaces was green while asserting nothing. - name: The node test runner refuses what it claims to run: ./scripts/checkers.sh node-test kb: name: Knowledge base consistency runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # kb.pl is executable knowledge, so it gets checked like code: rules # must reject what they should, domains must close, and functional keys # must not carry conflicting values. See kb_report/0 in the file. # trealla comes from the dev shell, as it does locally. - name: kb_consistent run: | tpl prolog-sessions/kb.pl \ -g "(kb_consistent -> halt(0) ; (kb_report, halt(1)))" # scripts/kb is the only thing that writes kb.pl, so its refusals are as # load-bearing as kb_consistent itself. Without this the replacement # writer could regress green -- the review that found the directive # bypass also found that nothing ran this suite. - name: The kb tool refuses what it claims to run: ./scripts/checkers.sh kb # Every suite goes through scripts/checkers.sh, which scrubs the # environment, so the variables a suite passes under here are the ones it # passes under locally (improvise-f9qf). Its bash is the dev shell's. - name: The checker runner refuses what it claims to run: ./scripts/checkers.sh --self-test # pr-watch is driven entirely from fixtures. The release preflight is the # one guard whose absence is invisible until a tag is public and # immutable. pr-reviews stands between a merge and an unread review, and # was blind to inline comments for as long as it existed (improvise-xgra). # fmt-workspaces is here because its own failure mode is the bug it # exists for: a usage error wearing a formatting failure's exit code. # bd-audit-ids is the dead-ID audit, whose recipe once read a bd that # never ran as a clean audit (improvise-52eq). - name: The remaining checkers refuse what they claim to run: ./scripts/checkers.sh pr-watch release-preflight pr-reviews fmt-workspaces bd-audit-ids # --------------------------------------------------------------------------- # Lints the one crate the `test` job cannot reach. # # `harness/opfs-contract` carries an empty `[workspace]` table, so it is its # own root and `cargo clippy --workspace` at the repo root never sees it. # # Its own job rather than a step inside `browser`, for two reasons. It took # 37s of the browser job's 263s on run 35535225637 while gating nothing the # harness needs — a lint failure and a harness failure are different reports, # and here the lint failure wore the harness's name. And `cargo fmt --all` # from inside that directory formats the WHOLE outer workspace (114 files, # measured) rather than the one crate, because cargo-fmt's `--all` walks past # the nested workspace boundary — so a rustfmt violation anywhere in the repo # failed the *Browser harness* job. `-p opfs-contract` scopes it to 1 file. harness-lint: name: Lint the harness crate runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 # checks.harness-lint in nix/ci.nix, run from inside harness/opfs-contract: # `cargo fmt -p opfs-contract -- --check` and clippy for wasm32. - name: cargo fmt -p opfs-contract and clippy --target wasm32 run: nix build --print-build-logs .#checks.x86_64-linux.harness-lint # --------------------------------------------------------------------------- # The browser harness (improvise-vvl3, absorbing improvise-xf1). # # This is the only job that RUNS any of the browser half. `OpfsFs`, # `web/app.js` and `web/worker.js` were previously compile-checked, # syntax-checked and read — never executed — so every bug in them was found # by reading or by a person pressing keys, neither of which is repeatable. # # It can fail the build, deliberately: a harness nobody runs is worse than # none, because it reads as coverage. # # It now needs NO Rust toolchain: the tree it serves is downloaded from # `build`, which is what makes "the harness runs against the tree Pages # publishes" (improvise-vvl3) a fact about bytes rather than about two # scripts agreeing. The download is `harness/serve` whole, so there is still # exactly one place that arrangement is decided — `harness/build.sh`. browser: name: Browser harness (OPFS contract + shell) needs: build runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 - name: Install the harness working-directory: harness run: npm ci # No browser install step and no version probe: the dev shell sets # PLAYWRIGHT_BROWSERS_PATH to nixpkgs' browsers, which pin them. `npm ci` # stays because @playwright/test itself is not nix-provided. # The tree `build` assembled, not one this job rebuilds. Same bytes as # the Pages artifact, modulo the cache-bust token `web/assemble.sh` # stamps — the harness tree is stamped `harness` and the site the commit # SHA, which was already true when both jobs built their own. - name: Download the harness tree uses: actions/download-artifact@v7 with: name: harness-serve path: harness/serve # `web/pkg` and `web/worker-pkg` as wasm-pack left them, for the .d.ts # the step below reads. The artifact's root is `web/`, so this restores # both directories at the paths the test resolves relative to itself. - name: Download the wasm bundles uses: actions/download-artifact@v7 with: name: wasm-bundles path: web # The JS↔wasm boundary: every call web/app.js makes is declared by the # bundle, with that arity. It needs the bundle and nothing else, so it # runs here rather than in the `js` job, and before Playwright — a # renamed export is cheaper to read here than as a blank page. - name: The page's calls match the bundle it calls run: ./scripts/node-test.sh 'web/tests/bundle/*.test.js' # Chromium and Firefox only, for now. nixpkgs' WebKit (WPEWebProcess) # aborts on its first page inside the runner VMs, though not on the # host; its four specs skip on the OPFS probe anyway, so nothing gated # is lost. Put `--project=webkit` back once that abort is understood. - name: Run the harness working-directory: harness run: npx playwright test --project=chromium --project=firefox - if: failure() uses: actions/upload-artifact@v7 with: name: playwright-traces path: harness/test-results retention-days: 7 # --------------------------------------------------------------------------- # GitHub Pages delivery (improvise-d31). # # The artifact is uploaded by `build`, which runs on every push and pull # request, so a change that breaks `web/build.sh` or the bundle fails on the # PR that made it — the build is a gate. `pages-deploy` runs only on a push # to main and `needs` the check jobs, so nothing that failed `cargo test`, # the wasm32 gate or the browser harness can reach the published site. The # deploy depends on those checks rather than repeating them (improvise-d31 # AC 2). # # Not path-filtered, deliberately. The bundles are `improvise-wasm-client` and # `improvise-worker-server` plus, between them, most of the workspace — the # worker bundle is the whole App: core, formula, io, protocol and command. A # filter # that meant "skip when the bundle cannot have changed" would have to name # most of the workspace, and the failure mode of getting it wrong is a stale # site with no signal that it is stale. A redundant build is the cheaper error. # # On improvise-864a: the mechanism is NOT supersession and NOT queue depth — # another run's SUCCESSFUL deployment invalidates this run's `github-pages` # artifact, measured over 100 runs. Building once shortens the upload→deploy # window (182s → ~115s on the run 35535225637 shape), and the window is where # the whole risk lives. It narrows the race; it does not close it. The # evidence and what it rules out: context/code/tooling.md §3. # `needs` is EVERY check job, which is the only rule that does not go stale. # `kb` was the one absent, on the unstated ground that kb.pl is not shipped by # the site — but neither is `harness-lint`'s crate, and it was in the list, so # the ground proved too much (improvise-x7cq). A failing consistency check # means main is broken, and publishing from a broken main is what this list # exists to stop. pages-deploy: name: Deploy to GitHub Pages needs: [test, wasm, js, browser, harness-lint, kb, build, nix-build] if: github.event_name == 'push' && github.ref == 'refs/heads/main' runs-on: [self-hosted, nixos, microvm] permissions: contents: read pages: write id-token: write # A superseded deploy is allowed to finish rather than being cancelled: # `deploy-pages` cancelled midway can leave the Pages deployment in a # half-applied state, and the next push republishes anyway. concurrency: group: pages cancel-in-progress: false environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} steps: # `enablement: false` (the default) configures the environment URL # without attempting to create or modify the Pages site via the API. # GitHub Pages must be enabled once in the repository settings # (Settings → Pages → Source: GitHub Actions) before this job will # produce a live URL. Using `enablement: true` fails when the workflow # token lacks admin permission to create the site (improvise-3gy). - uses: actions/configure-pages@v6 with: enablement: false - id: deployment uses: actions/deploy-pages@v5