Merge pull request #119 from fiddlerwoaroof/fix/release-preflight

ci: a release procedure, the preflight that enforces it, and the blind spot its own repair opened
This commit is contained in:
Ed Langley
2026-09-21 22:55:22 -07:00
committed by GitHub
5 changed files with 475 additions and 0 deletions
+4
View File
@@ -264,6 +264,10 @@ jobs:
# No trealla needed: pr-watch is driven entirely from fixtures.
- name: The PR watcher refuses what it claims to
run: ./scripts/pr-watch-test.sh
# The release preflight is the one guard whose absence is invisible until
# a tag is already public and immutable, so its refusals are checked here.
- name: The release preflight refuses what it claims to
run: ./scripts/release-preflight-test.sh
# ---------------------------------------------------------------------------
# Lints the one crate the `test` job cannot reach.
+71
View File
@@ -0,0 +1,71 @@
# Releasing
A checklist, not an explanation. The reasoning is `context/code/tooling.md` §4;
read it if a step surprises you, not before.
Its own file rather than a `README.md` section because `README.md` §Installation
addresses someone *installing* improvise and this addresses the one person
*publishing* it — folding them together lengthens the user-facing document for
no user benefit.
## The one rule
**Bump the version and commit it BEFORE you tag.** `dist` matches the tag
against the root package's version and refuses when they disagree — and by the
time CI tells you, the tag is already published and a published tag is not
moved. Every step below exists to put that check before the irreversible one.
## Cutting a release
1. **Bump the version.** `Cargo.toml`'s `[package]` version is the one `dist`
announces (`dist-workspace.toml` says `members = ["cargo:."]`). Keep the
crates under `crates/` in step with it; `gp-tree` sits at its own version
deliberately, and the preflight lists such crates as a note rather than an
error.
2. **Commit the bump**, and let `Cargo.lock` be part of that commit.
3. **Run the preflight**, with the tag you intend to push:
```sh
scripts/release-preflight.sh vX.Y.Z
```
It refuses a dirty tree, refuses a tag that does not match the version, runs
`dist plan` as a second opinion when `dist` is on `PATH`, and prints the tag
commands only once it has passed. It cuts nothing itself.
4. **Tag and push**, using the commands it printed:
```sh
git tag -a vX.Y.Z -m "vX.Y.Z"
git push gh vX.Y.Z
```
Push the tag to **`gh`** — `.github/workflows/release.yml` triggers on a tag
push there, matching `**[0-9]+.[0-9]+.[0-9]+*`. `origin` and `gitea` are
mirrors and build nothing.
5. **Watch the run.** A green release takes about three minutes and publishes
tarballs for all four targets with their `.sha256` files, `sha256.sum`,
`dist-manifest.json`, `source.tar.gz` and `improvise-installer.sh`.
The failure signature is unmistakable once you know it: **`plan` fails in
about twenty seconds and the four jobs behind it skip.** That is the tag not
matching the version — see below. A run that gets past `plan` is building.
## When a tag has already been pushed against the wrong version
**Leave it dead and cut the next one.** Do not move or re-push the tag: it is
already public, and a published tag that changes meaning is worse than one that
never built. Bump to the next version, commit, run the preflight, and tag that.
The worked example is `v0.1.0-rc3`: it was pushed while every crate still said
`0.1.0-rc2`, `dist` refused, and the four jobs behind `plan` skipped. No release
was ever built from it.
**Ask which tags exist; do not trust a document that tells you.** `git tag -l`,
and `git ls-remote --tags <remote>` for each remote you push to. Which tags are
where is a fact with a shelf life — it changed twice while this file was being
written — so this file deliberately records no inventory of it. Cleaning up a
dead tag is a separate decision and is not a step in cutting a release.
+169
View File
@@ -325,3 +325,172 @@ build already did some of that. Closing it properly means no two runs holding a
`github-pages` artifact across a deployment, which is a real change to the job
graph with real costs and is not attempted here. A retry is **not** a fix: the
artifact is gone rather than transiently unlistable, so a retry cannot find it.
### 4. The release preflight, and why a loud failure was not enough
`scripts/release-preflight.sh` answers, before a tag exists, the question the
Release workflow answers after. **The procedure is `RELEASING.md`**; this
section is the argument behind it, and the split is deliberate — a checklist
followed at 2am is not read for its reasoning, and one that carries it does not
get followed.
**The root cause was a missing document, not a missing check.** Nothing in this
repo said *bump the version before you tag* — `release.yml` is cargo-dist's
autogenerated mechanism, `README.md` §Installation describes the result, and
`context/plan.md` Phase 3 carries one-time *setup* instructions from the launch
plan. So the preflight is the second half of the fix; `RELEASING.md` is the
first.
**What happened.** `v0.1.0-rc3` was pushed while every
crate still said `0.1.0-rc2`. dist matches a tag against the root package's
version, refused with *"This workspace doesn't have anything for dist to
Release!"*, and the four jobs behind `plan` skipped. **Nothing was wrong with
the workflow**, which is worth stating plainly because the instinct on a failed
release run is to edit the YAML. The trigger is correct, the pattern matches,
the workflow is enabled; `dist plan --tag=v0.1.0-rc2` succeeds today and
`--tag=v0.1.0-rc3` reproduces the CI error exactly.
**The decisive evidence is a shipped release, not that local run**, and an
earlier draft of this passage rested on the weaker half. `v0.1.0-rc1` and
`v0.1.0-rc2` both carry **complete published GitHub Releases** from
2026-04-11 — tarballs for all four targets, their `.sha256` files,
`sha256.sum`, `dist-manifest.json`, `source.tar.gz` and
`improvise-installer.sh` (`gh release view v0.1.0-rc2 --json assets`). So the
workflow is not merely *expected* to work: it **demonstrably ran end to end**
the last time a tag and the version agreed. Note how nearly the opposite was
concluded — `gh run list` came back empty and was read as "only one tag push
ever ran this", when Actions logs simply age out at 90 days and those tags are
five months old. **An empty log is a statement about retention, not about
history**; the artefact a run *published* outlives the run, so ask the releases
before asking the runs. "The four jobs behind `plan` skipped" is true of the
rc3 run alone and must not be read as "no release has ever been built".
**A tag push has failed this way before — and how much that proves is worth
being exact about.** `gh run list --workflow=release.yml --event=push` keeps the
whole tag history, and it reads: rc1 **failed twice** (14s each) before
succeeding on the third push (3m25s), rc2 succeeded first time (3m9s), rc3
failed (21s). All three failures carry the **identical signature** — `plan:
failure` with `build-local-artifacts`, `build-global-artifacts`, `host` and
`announce` all `skipped` — checked per job, not inferred from the durations.
**A shared signature is not a shared cause, and rc1's is not knowable now.**
Only rc3's was reproduced (`dist plan --tag=v0.1.0-rc3`); rc1's logs aged out at
90 days, and its two failures sit on commits titled *"feat: cargo dist info"*
and *"chore: update version"*, i.e. during initial setup, where `plan` could as
easily have failed on an incomplete `dist` config. So the honest claim is: this
failure **shape** has now cost four pushes across two releases, and the one time
anyone has diagnosed it, it was the version/tag disagreement. That is ample
reason for `RELEASING.md` and a check before the irreversible step, and it is
**not** a claim that the same mistake was made three times — a first draft of
this paragraph said exactly that, from the durations alone, which is the same
asserted-not-asked error the two paragraphs around it correct.
The signature is worth memorising because it is so cheap to read: **`plan` fails
in ~20s with four jobs skipped**, where a real build takes about three minutes.
Note
the workflow also triggers on `pull_request`, where it plans without a tag and
passes in ~20s — so a *green* 20s Release run on a PR and a *failed* 20s one on
a tag look alike at a glance and mean opposite things.
**Why a loud, precise failure still cost something.** dist named the right
answer in its own error text — *"`--tag=v0.1.0-rc2` will Announce: improvise"*.
The problem is not the message, it is **when** it arrives: a tag is public and
immutable by the time CI reads it, so the repair is a new version rather than an
edit, and the tag that failed stays published forever. A check that runs
after the irreversible step is a check in the wrong place, however good its
message. So the preflight runs *before* tagging and prints the tag command only
once it has passed — the honest path made the easy one, the same shape as
`Basis` having no public constructor (gotcha 43).
**"All three remotes" was wrong when it was finally checked.** `aa4e901`'s
commit message said `v0.1.0-rc3` had been pushed to all three, and that was
copied from there into this section and into `RELEASING.md` before anyone ran
`git ls-remote`. Run, it showed the tag on `gh` alone. **That commit message is
superseded on this point and cannot be corrected**, which is why
`context/design-principles.md` §6 says an argument must not live only in one.
The irony was local: it sat two paragraphs from *an empty log is a statement
about retention, not about history* — the same failure, a remote-shaped fact
asserted rather than asked.
**Then the fact changed again, within hours, and that is the better lesson.**
Three measurements of the same thing in one session: present on all three
remotes and locally; then on `gh` alone; then **nowhere at all** — `rc3` was
deleted while this branch was in review. Only the first reading was careless.
The second and third were each correct when taken, by two different people, and
they disagree.
So the rule this section reaches for is **not** "ask rather than assert". It is
that **a remote-shaped fact has a shelf life even after you ask**: a verified
answer decays, and prose cannot distinguish a stale line from a live one, so a
reader at 2am has no way to tell which they are holding. A document that records
an inventory is therefore wrong on a schedule it does not control. This section
and `RELEASING.md` consequently record **how to ask** — `git tag -l`,
`git ls-remote --tags <remote>` — and deliberately record no inventory of which
tags exist where. What never decayed is the principle: a wrong tag is left dead
and the next version is cut. Note which half of the original correction survives
and which did not: *the commit message was wrong* is durable, because it is a
fact about a fixed text; *the tag is on `gh` alone* was true and is now false.
The keeper is the one whose subject cannot change.
**The primary check is textual and the `dist plan` run is a second opinion.**
An absent `dist` must not read as a pass, so the script says which of the two
ran. It also tolerates a locally-installed dist whose version differs from the
one pinned in `dist-workspace.toml`: that stops dist before it plans, and
reporting it as a refusal would train the reader to ignore refusals.
**Only the root package's version can disagree with a tag.**
`dist-workspace.toml` says `members = ["cargo:."]`, so `gp-tree` sitting at
`0.1.0` while everything else is on `0.1.0-rc2` is not a finding. The script
lists such crates and says so rather than failing, because a deliberate split
should stay deliberate and a silent one should be visible.
**The test's first version was vacuous, and the fault injection is what showed
it.** `scripts/release-preflight-test.sh` originally asserted only *did it
refuse*. Every negative case passed with the tag comparison replaced by
`if false;` — because the script refuses a **dirty working tree** first, and
this repo's tree is almost always dirty, bd auto-exporting `.beads/issues.jsonl`
on nearly every command. Four of five checks were answering *is the tree dirty*
while claiming to answer *does it catch a bad tag*. The test now runs the
negative cases against a clean throwaway checkout **and asserts the refusal
names the version mismatch**, so a refusal for the wrong reason is a failure
rather than a pass. Two rival faults were injected against the repaired
version — the comparison disabled, and `expected` derived from the tag itself —
and each produces four failures and exit 1.
That is `context/design-principles.md` §7's "a check that looks like the answer,
answering a neighbouring question", found in a checker written to prevent
exactly that class, within an hour of writing it.
**And the repair opened a second blind spot in the same place — which is the
more useful half.** Moving the negative cases to a clean checkout is what made
the tag comparison reachable at all, and it thereby removed the only
circumstance in which the test could observe the **dirty-tree guard**.
Disabling that guard outright left all six checks green and the script exiting
`0`: the guard whose *precedence* caused the original vacuity had become the one
thing the suite could not see, and the repair's own shape is what hid it. A
vacuity and its repair are not two events here; the repair relocated the blind
spot rather than closing it.
The check that closes it is therefore aimed the other way — a tag that
**matches** the version, run against a dirtied checkout, so the comparison
passes and a refusal can only be the tree. That is §7's *assert the direction
the default cannot supply* applied to a **fixture** rather than to a default:
the clean checkout is the fixture, and the discriminating input is the one that
makes it dirty. The two faults are independent in both directions — with the
comparison disabled the dirty-tree check still passes, and with the guard
disabled the three tag checks still pass — which is what makes the seventh
check worth having rather than a restatement of the other six.
**What is still unchecked, and why that is a decision.** The `dist plan` second
opinion can be disabled — its refusal arm rewritten never to match — with the
suite green, because every input the test feeds it is one the textual check has
already refused. That follows from the arm's own design rather than being an
oversight: it runs *after* the primary check, it is allowed to be absent
entirely, and an absent `dist` already prints that only the textual check ran.
Pinning it would mean installing a `dist` matching `cargo-dist-version`, which
is the very mismatch the script tolerates on purpose. Stated here rather than
guarded, so the next reader does not mistake the silence for coverage.
**Where it runs.** The `kb` job, beside `scripts/pr-watch-test.sh` — this
repo keeps its check_the_checkers there, not in `test`. The step needs no
trealla and no nix of its own; it is `sh`, `git archive` and `awk`.
+91
View File
@@ -0,0 +1,91 @@
#!/bin/sh
# check_the_checker for scripts/release-preflight.sh.
#
# It exists for the reason scripts/assemble-stamp-test.sh does: the guard is the
# whole value, and a guard that silently stopped guarding is invisible from the
# outside -- a preflight that accepted everything would print the same reassuring
# lines it prints today.
#
# IT ASSERTS THE REASON, NOT THE EXIT CODE, and that is the point. A first version
# checked only "did it refuse", and every negative case passed while the tag
# comparison was disabled -- because the script refuses a DIRTY WORKING TREE first,
# and this repo's tree is usually dirty (bd auto-exports .beads/issues.jsonl). Four
# of five checks were answering "is the tree dirty" while claiming to answer "does
# it catch a bad tag". Injecting `if false;` into the comparison is what exposed it.
# See context/design-principles.md section 7 -- a check adjacent to the thing it names.
set -eu
root=$(cd "$(dirname "$0")/.." && pwd)
script="$root/scripts/release-preflight.sh"
fails=0
# Refuses, AND the refusal names the tag/version mismatch rather than the tree.
refuses_for_the_tag() {
what=$1; tag=$2
out=$(sh "$script" "$tag" 2>&1) && { echo "FAIL: $what -- accepted" >&2; fails=$((fails+1)); return; }
case "$out" in
*"Cargo.toml has"*) echo " ok: $what (refused, naming the version)" ;;
*"uncommitted changes"*) echo "FAIL: $what -- refused for the DIRTY TREE, so this check proved nothing" >&2; fails=$((fails+1)) ;;
*) echo "FAIL: $what -- refused for an unrecognised reason:" >&2; echo "$out" | head -3 >&2; fails=$((fails+1)) ;;
esac
}
version=$(awk '/^\[package\]/{p=1;next} /^\[/{p=0} p&&/^version *=/{gsub(/[" ]/,"");sub(/^version=/,"");print;exit}' \
"$root/Cargo.toml")
test -n "$version" || { echo "FAIL: could not read the root version" >&2; exit 1; }
echo "release-preflight-test: root version is $version"
# The dirty-tree guard runs first, so the negative cases are exercised against a
# CLEAN checkout of this tree rather than the working copy. Without this the test
# cannot reach the comparison it exists to check.
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
git -C "$root" archive HEAD | (cd "$tmp" && tar x)
cp -f "$script" "$tmp/scripts/release-preflight.sh"
cp -f "$root/Cargo.toml" "$tmp/Cargo.toml"
(cd "$tmp" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm t)
script="$tmp/scripts/release-preflight.sh"
if sh "$script" --self-test >/dev/null 2>&1; then
echo " ok: --self-test (accepted)"
else
echo "FAIL: --self-test refused" >&2; fails=$((fails+1))
fi
if sh "$script" >/dev/null 2>&1; then
echo "FAIL: no argument should be a usage error" >&2; fails=$((fails+1))
else
echo " ok: no argument is a usage error (refused)"
fi
refuses_for_the_tag "a tag one character off" "v${version}x"
refuses_for_the_tag "the shape that actually broke CI" "v0.1.0-rc99"
refuses_for_the_tag "a bare version with no v" "$version"
# The positive direction, which only a clean tree can exercise.
if sh "$script" "v$version" >/dev/null 2>&1; then
echo " ok: the matching tag v$version (accepted)"
else
echo "FAIL: the matching tag v$version was refused" >&2; fails=$((fails+1))
fi
# The dirty-tree guard, which every check above is deliberately blind to: they
# run on a CLEAN checkout precisely so the tag comparison is reachable at all.
# The tag here MATCHES the version, so the comparison passes and a refusal can
# only be the tree -- which is what separates this check from the ones above
# instead of duplicating them. Disabling the guard leaves all six of those green.
echo "dirtied by release-preflight-test" >> "$tmp/README.md"
if out=$(sh "$script" "v$version" 2>&1); then
echo "FAIL: a dirty tree -- accepted" >&2; fails=$((fails+1))
else
case "$out" in
*"uncommitted changes"*) echo " ok: a dirty tree (refused, naming the tree)" ;;
*) echo "FAIL: a dirty tree -- refused, but not for the tree:" >&2; echo "$out" | head -3 >&2; fails=$((fails+1)) ;;
esac
fi
if [ "$fails" -ne 0 ]; then
echo "release-preflight-test: $fails failure(s)" >&2
exit 1
fi
echo "release-preflight-test: all checks passed"
+140
View File
@@ -0,0 +1,140 @@
#!/bin/sh
# Answer, before a tag exists, the question the Release workflow answers after.
#
# Usage: scripts/release-preflight.sh vX.Y.Z[-suffix]
# scripts/release-preflight.sh --self-test
#
# WHY THIS EXISTS. `v0.1.0-rc3` was pushed while every crate
# still said `0.1.0-rc2`. dist matches the tag against the root package version,
# refused with "This workspace doesn't have anything for dist to Release!", and
# the four jobs behind `plan` skipped. Nothing was wrong with the workflow.
#
# The cost is not the failure, it is WHEN the failure arrives: a tag is already
# public and immutable by the time CI reads it, so the repair is a new version
# rather than an edit. That is why this runs BEFORE tagging and prints the tag
# command only once it has passed -- the honest path made the easy one, the same
# shape as `Basis` having no public constructor (gotcha 43).
#
# The primary check is textual and always runs. `dist plan` is a SECOND opinion
# and is allowed to be unavailable; what it could not confirm is stated rather
# than assumed. Reasoning: context/code/tooling.md. scripts/release-preflight-test.sh
# is the check_the_checker.
set -eu
root=$(cd "$(dirname "$0")/.." && pwd)
# The root package's version, read from the FIRST [package] block of the root
# Cargo.toml. dist-workspace.toml says `members = ["cargo:."]`, so the root
# package is the only one dist announces and the only one whose version can
# disagree with a tag. gp-tree sitting at 0.1.0 is therefore not a finding.
root_version() {
awk '/^\[package\]/{p=1;next} /^\[/{p=0} p&&/^version *=/{gsub(/[" ]/,"");sub(/^version=/,"");print;exit}' \
"$root/Cargo.toml"
}
fail() { echo "release-preflight: $*" >&2; exit 1; }
preflight() {
tag=$1
version=$(root_version)
test -n "$version" || fail "could not read a version from Cargo.toml's [package] block"
# A tag names a version; anything else is a spelling this repo does not use.
expected="v$version"
if [ "$tag" != "$expected" ]; then
echo "release-preflight: REFUSED" >&2
echo " tag asked for : $tag" >&2
echo " Cargo.toml has: $version (so the tag must be $expected)" >&2
echo "" >&2
echo " Bump every crate to the version you mean to release and commit that" >&2
echo " FIRST. A tag pushed ahead of its version fails in CI after the tag is" >&2
echo " already public, and a published tag should not be moved." >&2
return 1
fi
echo "release-preflight: tag $tag matches Cargo.toml version $version"
# Every crate, because a workspace that disagrees with itself is the other
# half of the same error and dist says so in the same breath.
mismatched=$(grep -l '^version = ' "$root"/crates/*/Cargo.toml 2>/dev/null | while read -r f; do
v=$(awk '/^\[package\]/{p=1;next} /^\[/{p=0} p&&/^version *=/{gsub(/[" ]/,"");sub(/^version=/,"");print;exit}' "$f")
case "$v" in "$version"|"") ;; *) echo " $f: $v" ;; esac
done)
if [ -n "$mismatched" ]; then
echo "release-preflight: note -- these crates are on a different version:"
echo "$mismatched"
echo " Not fatal: dist announces only the root package (dist-workspace.toml"
echo " says members = [\"cargo:.\"]). Listed so a deliberate split stays deliberate."
fi
return 0
}
self_test() {
# check_the_checker. Both directions, because a checker that only ever passes
# and a checker that only ever fails are indistinguishable from one that works.
version=$(root_version)
test -n "$version" || { echo "self-test: could not read the root version" >&2; exit 1; }
if ! preflight "v$version" >/dev/null 2>&1; then
echo "self-test FAILED: the matching tag v$version was refused" >&2
exit 1
fi
if preflight "v$version-nope" >/dev/null 2>&1; then
echo "self-test FAILED: a tag that does not match the version was accepted" >&2
exit 1
fi
# The exact shape that was actually pushed, spelled out, because this is the
# case the script was written for and a regression here is the whole failure.
if preflight "v0.1.0-rc3" >/dev/null 2>&1 && [ "$version" != "0.1.0-rc3" ]; then
echo "self-test FAILED: v0.1.0-rc3 accepted against version $version" >&2
exit 1
fi
echo "release-preflight: self-test passed (accepts v$version, refuses a mismatch)"
}
case "${1:-}" in
--self-test) self_test; exit 0 ;;
"" | -h | --help)
echo "usage: release-preflight.sh vX.Y.Z[-suffix] | --self-test" >&2
exit 2 ;;
esac
tag=$1
# A dirty tree means the tag would not name what was tested.
if [ -n "$(git -C "$root" status --porcelain --untracked-files=no)" ]; then
fail "the working tree has uncommitted changes; tag a commit, not a guess"
fi
preflight "$tag" || exit 1
# The second opinion. dist is the thing that actually refused in CI, so running
# it is worth more than any restatement of its rules -- but it is optional, and
# an absent dist must not read as a pass.
if command -v dist >/dev/null 2>&1; then
out=$(cd "$root" && dist plan "--tag=$tag" 2>&1) || true
case "$out" in
*"doesn't have anything for dist to Release"*)
echo "release-preflight: REFUSED -- dist rejected the tag:" >&2
echo "$out" | sed -n '/×/,$p' | head -8 >&2
exit 1 ;;
*"but 'cargo-dist-version"*)
echo "release-preflight: dist ran but is a different version from the one"
echo " pinned in dist-workspace.toml, so it stopped before planning. The"
echo " tag/version check above still passed; CI installs the pinned version."
;;
*) echo "release-preflight: dist plan accepted $tag" ;;
esac
else
echo "release-preflight: dist not on PATH, so only the textual check ran."
echo " That is the check that catches the failure this script exists for."
fi
cat <<EOF
Preflight passed. To cut the release:
git tag -a $tag -m "$tag"
git push gh $tag
Push the tag to gh; the Release workflow triggers on the tag push there.
EOF