diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 22801196..e605a8f6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,6 +68,12 @@ jobs: "wasm-pack-${WASM_PACK_VERSION}-x86_64-unknown-linux-musl/wasm-pack" wasm-pack --version + # Both assemblies below generate the prose site with node. The runner + # image ships one, but pinning it is what makes a local run evidence. + - uses: actions/setup-node@v4 + with: + node-version: "24" + # The same script a developer runs locally, so there is one answer to # "how is the bundle built" rather than two that can drift. It builds all # three wasm bundles (client, worker, opfs-contract) and assembles diff --git a/context/code/doc-site.md b/context/code/doc-site.md new file mode 100644 index 00000000..61667fcb --- /dev/null +++ b/context/code/doc-site.md @@ -0,0 +1,303 @@ +# The documentation site — the arguments + +`scripts/build-doc-site.mjs` turns the prose in `context/` into the static site +served at `/docs/` on GitHub Pages. This is the reasoning behind what it +publishes, what it deliberately does not, and why it is a hundred-odd lines of +node rather than a static site generator (improvise-djoo). + +--- + +## 1. What is published, and what is not + +The manifest is the `SECTIONS` array at the top of the script, and **a document +is published only by being named there**. That is the point: a glob over +`context/**` would publish the next working file somebody drops in, and the +decision about what a stranger should read is not one to make by accident. + +Published, in four sections: + +| Section | Contents | Why | +|---|---|---| +| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md` | What the project is for, the doctrine, and the single best explanation of how it is built. | +| Decision records | `context/adr/*.md` | An ADR is written for an outside reader by construction — it states the options that were rejected, which is the part that is worth reading and the part that is nowhere else. | +| The arguments, by area | `context/code/*.md` | This is the reasoning that used to live in code comments and was moved out under the two-line cap (design-principles §6). It is the most valuable prose in the repository and it is addressed to exactly the reader a documentation site has. | +| Test evidence | `context/test-evidence/*.md` | 501 entries, one per test whose narrative is longer than its code. | + +Not published: + +- **`context/plan.md`** is a *prompt*: it opens "You are working in the + `improvise` repository… Your job is to prepare this repository for a public + Show HN launch", and it instructs a reader to audit a `context/SPEC.md` that + no longer exists. It is an instruction to an agent about a launch, not a + description of the program, and publishing a launch plan on the site the + launch points at inverts the two. +- **`context/cloud-agents.md`** is operational notes for agents running in a + cloud session — tracker hydration, `bd` pitfalls, what the sandbox blocks. + It is true and it is useful and it is addressed to nobody who visits a + website. + +Neither is referenced by any published document except as a row in +`repo-map.md`'s file inventory, which is inside a code fence, so leaving them +out creates no dangling link. That was checked rather than assumed. + +**Test evidence is the one judgement call worth stating.** The argument for +leaving it out is that 501 entries about individual tests are noise to a reader +who wanted to know what a pivot table is. The argument for keeping it is +stronger: every one of the seven `context/code/*.md` documents points at its +evidence file by name, so omitting it would break the published prose in +exactly the way this site is otherwise careful not to; and the evidence file is +the most distinctive artefact the repository has — a written record of the +injected fault behind each load-bearing test is not a thing most projects can +show. It is therefore published as its **own section**, last, described as a +contributor reference rather than a tour, so nobody wanders into it expecting +a narrative. + +## 2. Ticket ids resolve to a page, never to a URL that does not exist + +The prose cites bead ids roughly 190 times — this paragraph deliberately spells +none, because a document that gives one as an example makes itself a source of +audit noise, and a placeholder is worse than a real id: it is ID-shaped and +resolves to nothing. On a public site those citations are +meaningless strings, and the obvious repair — link them to an issue tracker — +is unavailable: beads keeps its issues in a Dolt database on a git ref, and +there is no public issue URL to point at. Inventing one would produce 190 +links that 404, which is worse than plain text because a link is a promise. + +So the ids resolve **inward**. `tickets.html` is generated from +`.beads/issues.jsonl` — already committed, already public — and carries id, +status and title, nothing else; descriptions are internal working notes and are +not the site's business. Each citation in the prose becomes a link to that +page's anchor, with the title in a `title=` attribute so a hover answers the +question without a navigation. + +**`issues.jsonl` is a passive export and lags the database**, which is not a +defect to work around but a fact to render honestly: an id the export does not +carry is emitted as plain text with a dotted underline and no link, and the +ticket page lists it as "not exported". The site therefore never links to an +anchor it did not generate, whatever state the export is in. A generator that +instead linked every id it saw would produce a page that is right today and +broken the first time somebody cites a fresh ticket. + +**The page publishes every cited id, open bugs and their titles included, and +that is a decision rather than a default** (user ruling). The data is already +public in `.beads/issues.jsonl` in a repository anyone can clone; the page only +makes it legible. An honest open-bug list is a normal thing for an open-source +project to have, and filtering one while the page's own lede links the raw +JSONL would be theatre. Note the second-order reason to keep it unfiltered: +a citation whose row was suppressed would be a link into a page that does not +answer it, which is the failure this whole section exists to avoid. + +**Titles are tracker data and are escaped as attribute values.** Five shipped +bead titles contain a double quote; interpolated into `title="…"` by an `esc` +that handled only `&`, `<` and `>`, they broke out of the attribute and the +browser parsed the remains as three junk attributes. `esc` now escapes `"` and +`'` as well. Today's corpus being friendly is not a reason to interpolate +untrusted-shaped text unescaped — the tracker is written by whoever files a +bug. + +## 3. Source paths link only where the file exists + +A code span whose entire content is a repo path becomes a link — to the +generated page when the path is one of the published documents, to GitHub +otherwise. Two constraints make that safe. The whole span must be the path, so +a sentence that merely mentions a directory is untouched; and **the path must +exist on disk at generation time**, so a citation that has gone stale degrades +to plain text rather than to a 404. The GitHub ref is the cache-bust token +`web/assemble.sh` already receives, used only when it is a 40-hex sha and +falling back to `main` otherwise — a local `dev` build must not emit +`blob/dev/…`. + +## 4. The link check is the test, and it is part of the build + +The generator collects every anchor it emits and every internal href it writes, +resolves them against each other, and **exits non-zero** if any href names a +page it did not generate or a fragment that page does not carry. +`web/assemble.sh` runs under `set -eu`, so a broken cross-reference fails the +assembly rather than shipping. + +This is not ceremony. It caught two real defects on its first two runs: a +fence-close regex that compared a three-character marker against a +one-character string, which swallowed every document from its first code block +onward; and a list parser that ended the gotcha list at the first blank line, +so 47 of the 48 gotcha anchors were never emitted while the "jump to gotcha N" +strip linked to all of them. Both were invisible in a rendered page that +otherwise looked fine. + +It is proved by injection rather than asserted: renaming the emitted anchor ids +makes it report 48 missing anchors, and pointing the ticket links at a page +name that is never generated makes it report 763 missing pages — two different +diagnoses, because "the page is absent" and "the page is there but the anchor +is not" call for different repairs. + +Links that **leave** the `docs/` subtree — `../index.html`, `../app/` — name +files this script does not generate, so they are resolved against the +filesystem instead. That is the whole reason `assemble.sh` runs the generator +**last**: its siblings have to exist before they can be checked. An earlier +version skipped those 48 links while still counting them as checked, behind a +comment promising they were "checked below" when there was no below. + +**And the link check is not sufficient, which is the more important lesson.** +It answers one question — does every href resolve — and a defect outside that +question is invisible to it however carefully it is run. The worked example is +a nested ``: a TOC entry wrapping a heading whose rendered text already +contained a link produced `…`, and +the HTML parser closes the outer anchor early, so the entry became an empty +unclickable stub and clicking it navigated to GitHub. Both hrefs resolved +perfectly. It shipped past the link check, past the whole-site link check, and +past a screenshot pass, and was caught by a reviewer *clicking a link*. The +repair is therefore two things, not one: the TOC keeps a tag-free variant of +each heading, **and** a separate check asserts no `` appears inside another +`` — proved by reverting the first half and watching the second report 66. + +The general form: when a check misses a defect, add the check that could have +seen it rather than only fixing the defect, and ask what *question* the +existing check was answering. + +## 5. Why a node script and not a generator + +The landing page is hand-written HTML with no build step and one pinned CDN +script; the same posture applies here. A static site generator would add a +dependency tree, a lockfile and a toolchain to a repository whose site is +currently four files, in exchange for features this corpus does not use — the +docs contain no front matter, no shortcodes and, as it happens, **not a single +markdown link**. Every cross-reference in them is prose naming a path or an +id, which is precisely why §2 and §3 are the interesting parts and Markdown +rendering is the boring part. + +So the renderer is a deliberately small subset covering exactly what the corpus +contains: ATX headings, fenced code, pipe tables, blockquotes, ordered and +unordered lists with indented continuations, thematic breaks, and inline code, +strong, emphasis and links. Anything it does not recognise becomes paragraph +text, so **no input line is ever dropped** — the failure mode is an ugly +paragraph, never a missing one. + +Two details in it are load-bearing and easy to undo. **Code spans are replaced +by placeholders before emphasis is matched**, because this corpus routinely +opens `**` on one side of a code span and closes it on the other; matching +emphasis per segment silently drops that bold, which is a fidelity loss with +no visible symptom. And **strong is matched before emphasis, with a body that +may contain a single `*` pair**, because `[^*]+` cannot match +`**a *b* c**` and the leftover `*` then pairs across the bold — which is how a +single sentence in gotcha 33 came out as three italic paragraphs. + +A third, added later and for the same reason: **a table cell boundary is an +unescaped pipe outside a code span**, so the row is walked rather than +`split("|")`. This corpus documents a file format whose own quoting syntax is +`|Time Period|`, and a sentence containing one inside a table cell would have +been shredded into extra columns — a row that still parses, so no column count +could detect it. It was latent when fixed, which is the point: the trigger is +one ordinary sentence away in a document whose subject is pipe quoting. + +One deliberate divergence from GFM inside that rule: a `\|` **inside a code +span** in a table cell keeps its backslash, where GFM would unescape it to a +bare pipe. Splitting is suppressed inside code spans, so the escape is never +needed there and never consumed. The corpus contains no instance; it is +recorded as a decision rather than left to be rediscovered as a bug. + +## 6. Navigation + +`repo-map.md` is one long document whose gotcha list is numbered 1 to 48, and +splitting it per gotcha was rejected: the gotchas cross-reference each other +constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a reader +following those wants a scroll, not 48 page loads. It stays one page, with an +id on every list item and a compact strip of numbered links under the heading. +The ids come from the list item's **own number**, not from a counter, so an +inserted gotcha renumbers the anchors exactly as it renumbers the prose that +cites them. + +The strip's contents are **derived from the anchors actually emitted**, never +from a count. A hardcoded 48 failed in only one direction: deleting a gotcha +left a link to a missing anchor, which the link check catches, while *adding* +one produced an unlinked gotcha and a clean build. The list only ever grows, so +the hardcoded form was silent in precisely the direction it would be used — +the same shape as an exclusion list that over-matches, failing in the direction +that looks like success. + +**Long unbreakable headings need `overflow-wrap`.** The test-evidence pages are +hundreds of `snake_case` test names as headings; at phone width seven of them +overflowed the viewport by up to 92%. The diagnosis is worth keeping because +the first guess was wrong: `getBoundingClientRect` reported every element +inside the viewport, because the text overflows its box without widening it. +Only `scrollWidth` per element names the culprit. Measure the property that +actually overflowed rather than the one that usually does. + +## 7. The phone nav is a checkbox, and that is not arbitrary + +At phone width the sidebar's 25 links stacked above the content, so every page +opened on roughly 1500px of nav. The wall was not inherited: **all of this CSS +is new**, so it shipped for the first time with the site, which is the test for +whether a defect is pre-existing and the one most easily got wrong in the +direction of declining work you introduced. + +`
` is the obvious control and it is **wrong here**, for a reason no +spec-level reading predicts. The sidebar layout has to force the disclosure +*open* at wide widths, and a closed `
` hides its content through +`::details-content` — which `display` cannot override. Measured in Chromium, +the desktop sidebar came out collapsed to a single link with its toggle also +hidden: unusable, and in the half nobody would think to re-check after a +phone-only change. A hidden checkbox plus a `
` obviously belongs; this paragraph is why +it must not be tidied into one. + +**And the first probe of the fix could not see the difference.** Counting nav +links whose `getBoundingClientRect().height > 0` reported **26 visible in both +the collapsed and the expanded state**, because Chrome reports a height for +`content-visibility`-hidden children — so it would have passed the broken +`
` build. `checkVisibility({contentVisibilityAuto: true, +visibilityProperty: true})` discriminates. + +That is the same instrument failing twice: §6's overflow fix also measured +clean by `getBoundingClientRect` while the page still scrolled. **A measurement +that cannot distinguish the two states is not a measurement** — the repo's +signature defect, a test that passes while the thing it names is broken, +arriving through the instrument rather than through the assertion. Before +trusting a browser measurement, injure the thing being measured and check the +number moves. + +**And the fix's own probe had the same shape one level up.** Counting *visible* +nav links answers "does operating the toggle reveal the links" — correct and +complete for a mouse — and is silent on whether any non-mouse route exists. The +first version of the checkbox carried `hidden`, which is `display:none`, which +removes it from the tab order *and* from the accessibility tree: the label is +not focusable on its own, so at ≤900px the navigation could be clicked and +reached no other way, and the accessibility snapshot contained neither the +checkbox nor any of the 25 links. A visibility count cannot see that, however +sharp the visibility instrument. The toggle is therefore visually hidden but +focusable at narrow widths and `display:none` at wide ones, and the check is +tab order, `.focus()` and the accessibility snapshot — measured at 390, 900, +901 and 1280, because the regression lives on one side of a media query. + +So the rule from §6 has a second half: **a measurement of the wrong property is +not a measurement either, however precise it is.** `checkVisibility` fixed the +visibility instrument; it does not turn a visibility count into an operability +check. Ask what the defect would *do* to a user, then pick the instrument that +observes that. + +**A correction, and the risk the original argument missed.** The commit that +added `content: "\2630 " / ""` said Firefox ignores the alt-text form and falls +back to reading the content, so the change was "an improvement in two engines +and a no-op in the third". **That mechanism is wrong** — Firefox shipped the +syntax in 133, and all three engines parse it (measured: Chrome 141, Firefox +142, WebKit 26 each compute the declaration intact). The conclusion held; the +reason given for it did not, and the reason was the load-bearing part, because +it was what made the change look like a one-way door. It is one because every +engine shipped to supports the syntax — not because the unsupported path is +harmless. That commit message is superseded by this paragraph (design-principles +§6: an argument must not live only in a commit message). + +**The unsupported path is not harmless, and that is the risk neither the change +nor its first review named.** `/ alt` in an engine that does not know it is a +*parse error for the whole `content` declaration*, so the glyph would not lose +its alt text — it would **vanish**. A silent visual regression, the opposite of +the assumed failure mode. The glyph is therefore verified painted in all three +engines by pixel diff against `::before { content: none }`, with the text +starting 20px further right in each. + +That check also produced this file's fourth instance of the wrong-property +trap, and the sharpest, because it produced a confident **false positive**: the +first probe measured the label's *width* with and without `::before` and +reported the glyph missing in all three engines. The label is `display: block`, +so its width comes from the container and cannot see inline content at all. +Right question, wrong property, and this time the wrong instrument accused +working code rather than clearing broken code. diff --git a/docs/index.html b/docs/index.html index 090bea45..ebcfc879 100644 --- a/docs/index.html +++ b/docs/index.html @@ -134,6 +134,7 @@ ▶ Run improvise in your browser + Documentation Source on GitHub @@ -191,7 +192,8 @@ nix profile install github:fiddlerwoaroof/improvise diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs new file mode 100644 index 00000000..e82dada9 --- /dev/null +++ b/scripts/build-doc-site.mjs @@ -0,0 +1,688 @@ +#!/usr/bin/env node +// Generate the documentation site from context/ into OUT_DIR (default site/docs). +// The argument and the rationale live in context/code/doc-site.md. +// +// NOT standalone: the link check resolves ../index.html and ../app/ on disk, so +// OUT_DIR's parent must already be an assembled site. Run web/assemble.sh, which +// invokes this last for that reason. Bare `node scripts/build-doc-site.mjs out` +// exits 1 on 48 outward links, and that strictness is the point — the hole it +// replaced was counting those links as checked while skipping them. +// +// The reorder also moved WHERE a failure shows: a dead link now leaves a site/ +// that looks complete with a partial docs/, where before it left one with no +// app/. Contained only because assemble.sh is `set -eu` and CI gates on it. +import { readFileSync, writeFileSync, mkdirSync, rmSync, existsSync, statSync } from "node:fs"; +import { dirname, join, relative } from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = join(dirname(fileURLToPath(import.meta.url)), ".."); +const out = process.argv[2] || join(root, "site", "docs"); +const token = process.argv[3] || ""; +const ref = /^[0-9a-f]{40}$/.test(token) ? token : "main"; +const REPO = "https://github.com/fiddlerwoaroof/improvise"; + +// --------------------------------------------------------------------------- +// The manifest. A doc is published only by being named here — see +// context/code/doc-site.md for what was left out and why. +const SECTIONS = [ + { + id: "start", + title: "Start here", + blurb: + "What the project is for, and the principles the code is held to.", + docs: [ + ["docs/design-notes.md", "design-notes", "Design notes"], + ["context/design-principles.md", "design-principles", "Design principles"], + ["context/repo-map.md", "repo-map", "Repository map"], + ], + }, + { + id: "adr", + title: "Decision records", + blurb: "Decisions with consequences, including the options that were rejected.", + docs: [ + ["context/adr/001-concurrent-edits.md", "adr/001-concurrent-edits", "ADR 001 — Concurrent edits"], + ["context/adr/002-naming-a-cell.md", "adr/002-naming-a-cell", "ADR 002 — Naming a cell"], + ["context/adr/003-live-backed-models.md", "adr/003-live-backed-models", "ADR 003 — Live-backed models"], + ], + }, + { + id: "code", + title: "The arguments, by area", + blurb: + "The reasoning that used to live in code comments. Written for a reader with the source open.", + docs: [ + ["context/code/core.md", "code/core", "improvise-core"], + ["context/code/formula-and-io.md", "code/formula-and-io", "improvise-formula and improvise-io"], + ["context/code/protocol.md", "code/protocol", "improvise-protocol"], + ["context/code/command.md", "code/command", "improvise-command"], + ["context/code/clients.md", "code/clients", "The clients"], + ["context/code/tui.md", "code/tui", "The TUI"], + ["context/code/gp.md", "code/gp", "gp-tree and improvise-gp"], + ], + }, + { + id: "evidence", + title: "Test evidence", + blurb: + "For each test whose narrative is longer than its code: what broke, what fault was injected, and why the test can fail. Contributor reference, not a tour.", + docs: [ + ["context/test-evidence/core.md", "test-evidence/core", "improvise-core"], + ["context/test-evidence/formula-and-io.md", "test-evidence/formula-and-io", "improvise-formula and improvise-io"], + ["context/test-evidence/protocol.md", "test-evidence/protocol", "improvise-protocol"], + ["context/test-evidence/command.md", "test-evidence/command", "improvise-command"], + ["context/test-evidence/clients.md", "test-evidence/clients", "The clients"], + ["context/test-evidence/tui.md", "test-evidence/tui", "The TUI"], + ["context/test-evidence/gp.md", "test-evidence/gp", "gp-tree and improvise-gp"], + ], + }, + { + id: "meta", + title: "About this site", + blurb: "What is published here, what is not, and why.", + docs: [["context/code/doc-site.md", "doc-site", "About this site"]], + }, +]; + +const PAGE_BY_SOURCE = new Map(); +for (const s of SECTIONS) for (const [src, slug] of s.docs) PAGE_BY_SOURCE.set(src, `${slug}.html`); + +// --------------------------------------------------------------------------- +// The ticket index. `.beads/issues.jsonl` is a passive export and lags the Dolt +// database, so an id it does not carry is rendered as plain text rather than a +// link that 404s. +function readTickets() { + const path = join(root, ".beads", "issues.jsonl"); + const map = new Map(); + if (!existsSync(path)) return map; + for (const line of readFileSync(path, "utf8").split("\n")) { + if (!line.trim()) continue; + let o; + try { + o = JSON.parse(line); + } catch { + continue; + } + if (o && o._type === "issue" && typeof o.id === "string" && typeof o.title === "string") { + map.set(o.id, { title: o.title, status: o.status || "unknown", kind: o.issue_type || o.type || "" }); + } + } + return map; +} +const TICKETS = readTickets(); + +// Tokens that look like an id but name a crate or a package. Anchored, because +// an unanchored alternative is a prefix match that silently excludes a real id. +const NON_TICKET = new Set( + "core formula io protocol command gp ws worker wasm harness registry side".split(" "), +); + +// --------------------------------------------------------------------------- +// Markdown. A deliberately small subset, covering exactly what the corpus uses; +// anything unrecognised becomes paragraph text, so no input line is ever lost. +const esc = (s) => + s + .replace(/&/g, "&") + .replace(//g, ">") + .replace(/"/g, """) + .replace(/'/g, "'"); + +// A TOC entry must not nest an anchor inside the one wrapping it: a heading +// whose text is a path or a ticket id already carries an , and the parser +// closes the outer one early, leaving an empty unclickable stub. +const stripTags = (html) => html.replace(/<[^>]+>/g, ""); + +function slugify(s) { + return ( + s + .toLowerCase() + .replace(/<[^>]+>/g, "") + .replace(/&[a-z]+;/g, "") + .replace(/[^a-z0-9]+/g, "-") + .replace(/^-+|-+$/g, "") || "section" + ); +} + +// A code span whose whole content is a repo path becomes a link: to the +// generated page when the path is a published doc, to GitHub otherwise. The +// path must exist on disk, so a stale citation stays plain text. +function pathHref(p, depth) { + const clean = p.replace(/^\.\//, ""); + if (!/^[A-Za-z0-9_.\/-]+$/.test(clean) || !/[\/.]/.test(clean)) return null; + // A span of punctuation alone (`.`, `-`, `/`) is prose, not a path. + if (!/[A-Za-z0-9]/.test(clean)) return null; + if (clean.includes("..") || clean.startsWith("/")) return null; + if (PAGE_BY_SOURCE.has(clean)) return "../".repeat(depth) + PAGE_BY_SOURCE.get(clean); + const abs = join(root, clean); + if (!existsSync(abs)) return null; + // GitHub serves a directory under /tree/, not /blob/; relying on its + // redirect is a 404 waiting on a change of behaviour we do not control. + const kind = statSync(abs).isDirectory() ? "tree" : "blob"; + return `${REPO}/${kind}/${ref}/${clean.replace(/\/$/, "")}`; +} + +function inline(text, ctx) { + // Code spans first: a backtick run opens a span only if a run of the same + // length closes it, so an unmatched ``` in prose stays literal. + const parts = []; + let i = 0; + while (i < text.length) { + if (text[i] === "`") { + let n = 0; + while (text[i + n] === "`") n++; + const fence = "`".repeat(n); + const end = text.indexOf(fence, i + n); + const after = end >= 0 ? end + n : -1; + if (end >= 0 && (after >= text.length || text[after] !== "`")) { + parts.push({ code: text.slice(i + n, end) }); + i = after; + continue; + } + } + const next = text.indexOf("`", i + 1); + parts.push({ text: text.slice(i, next < 0 ? text.length : next) }); + i = next < 0 ? text.length : next; + } + + // Code spans stand aside as placeholders while emphasis is matched, because + // the corpus routinely opens `**` on one side of a code span and closes it on + // the other; matching per segment silently drops that bold. + const held = []; + let s = parts + .map((p) => { + if (p.code === undefined) return esc(p.text); + const body = `${esc(p.code)}`; + const href = pathHref(p.code.trim(), ctx.depth); + held.push(href ? `${body}` : body); + return `\u0000${held.length - 1}\u0000`; + }) + .join(""); + + s = s.replace(/\[([^\]]+)\]\(([^)\s]+)\)/g, (_m, label, href) => `${label}`); + // Strong first, and its body may contain a single `*` pair: `[^*]+` cannot + // match `**a *b* c**`, and the leftover `*` then pairs across the bold. + s = s.replace(/\*\*((?:[^*]|\*(?!\*))+)\*\*/g, "$1"); + s = s.replace(/\*([^*]+)\*/g, "$1"); + s = s.replace(/\bimprovise-([a-z0-9]{3,8})\b/g, (m, tail) => { + if (NON_TICKET.has(tail)) return m; + ctx.cited.add(m); + if (!TICKETS.has(m)) { + ctx.unresolved.add(m); + return `${m}`; + } + const t = TICKETS.get(m); + const href = `${"../".repeat(ctx.depth)}tickets.html#${m}`; + return `${m}`; + }); + return s.replace(/\u0000(\d+)\u0000/g, (_m, n) => held[Number(n)]); +} + +// Block parse. Returns { html, toc }. +function renderMarkdown(src, ctx) { + const lines = src.split("\n"); + const toc = []; + const seen = new Map(); + const html = blocks(lines); + return { html: html.join("\n"), toc }; + + function heading(level, text) { + const rendered = inline(text, ctx); + let id = slugify(text); + const n = (seen.get(id) || 0) + 1; + seen.set(id, n); + if (n > 1) id = `${id}-${n}`; + ctx.anchors.add(id); + if (level <= 3) toc.push({ level, id, text: stripTags(rendered) }); + const mark = level > 1 ? `#` : ""; + return `${mark}${rendered}`; + } + + function blocks(ls) { + const outHtml = []; + let i = 0; + let para = []; + const flush = () => { + if (para.length) outHtml.push(`

${inline(para.join(" "), ctx)}

`); + para = []; + }; + while (i < ls.length) { + const line = ls[i]; + const fence = line.match(/^(\s*)(`{3,}|~{3,})\s*([A-Za-z0-9_-]*)\s*$/); + if (fence) { + flush(); + const close = new RegExp(`^\\s*${fence[2][0] === "~" ? "~" : "\`"}{3,}\\s*$`); + const body = []; + i++; + while (i < ls.length && !close.test(ls[i])) { + body.push(ls[i]); + i++; + } + i++; + outHtml.push(`
${esc(body.join("\n"))}
`); + continue; + } + const h = line.match(/^(#{1,6})\s+(.*)$/); + if (h) { + flush(); + outHtml.push(heading(h[1].length, h[2].trim())); + i++; + continue; + } + if (/^\s*(-{3,}|\*{3,}|_{3,})\s*$/.test(line) && !para.length) { + flush(); + outHtml.push("
"); + i++; + continue; + } + if (/^\s*\|/.test(line) && i + 1 < ls.length && /^\s*\|[\s:|-]+\|?\s*$/.test(ls[i + 1])) { + flush(); + const rows = []; + while (i < ls.length && /^\s*\|/.test(ls[i])) { + rows.push(ls[i]); + i++; + } + outHtml.push(table(rows)); + continue; + } + if (/^\s*>/.test(line)) { + flush(); + const body = []; + while (i < ls.length && /^\s*>/.test(ls[i])) { + body.push(ls[i].replace(/^\s*>\s?/, "")); + i++; + } + outHtml.push(`
${blocks(body).join("\n")}
`); + continue; + } + const li = line.match(/^(\s*)([-*+]|\d{1,3}\.)\s+(.*)$/); + if (li) { + flush(); + const [listHtml, next] = list(ls, i, li[1].length); + outHtml.push(listHtml); + i = next; + continue; + } + if (!line.trim()) { + flush(); + i++; + continue; + } + para.push(line.trim()); + i++; + } + flush(); + return outHtml; + } + + // An item's continuation lines are dedented by that item's own marker width, + // so a nested list keeps its relative indentation and reparses as a list. + function list(ls, start, indent) { + const ordered = /^\s*\d{1,3}\./.test(ls[start]); + const items = []; + let i = start; + let first = null; + while (i < ls.length) { + const m = ls[i].match(/^(\s*)([-*+]|(\d{1,3})\.)\s+(.*)$/); + if (m && m[1].length === indent) { + if (ordered && first === null) first = Number(m[3]); + items.push({ num: m[3] ? Number(m[3]) : null, cw: m[0].length - m[4].length, body: [m[4]] }); + i++; + continue; + } + if (!items.length) break; + const cur = items[items.length - 1]; + if (!ls[i].trim()) { + let j = i; + while (j < ls.length && !ls[j].trim()) j++; + if (j >= ls.length) break; + // A blank line ends the list only if what follows is neither the next + // item nor a continuation of the open one. + const isSibling = /^(\s*)([-*+]|\d{1,3}\.)\s+/.test(ls[j]) && ls[j].search(/\S/) === indent; + if (isSibling) { + i = j; + continue; + } + if (ls[j].search(/\S/) > indent) { + cur.body.push(""); + i = j; + continue; + } + break; + } + const col = ls[i].search(/\S/); + if (col <= indent) break; + cur.body.push(ls[i].slice(Math.min(col, cur.cw))); + i++; + } + const tag = ordered ? "ol" : "ul"; + const attr = ordered && first !== null && first !== 1 ? ` start="${first}"` : ""; + const rendered = items + .map((it) => { + const inner = blocks(it.body); + const body = inner.length === 1 && inner[0].startsWith("

") + ? inner[0].slice(3, -4) + : inner.join("\n"); + const id = ctx.listItemId && it.num !== null ? ctx.listItemId(it.num) : null; + if (id) ctx.anchors.add(id); + return `${body}`; + }) + .join("\n"); + return [`<${tag}${attr}>\n${rendered}\n`, i]; + } + + // A cell boundary is an UNESCAPED pipe outside a code span. Splitting the raw + // row on every `|` shreds `\|` and this repo's own `|Time Period|` syntax into + // extra columns, which no column count can detect because the row still parses. + function splitRow(r) { + const out = []; + let cur = ""; + let tick = 0; + for (let i = 0; i < r.length; i++) { + const c = r[i]; + if (c === "\\" && r[i + 1] === "|" && !tick) { + cur += "|"; + i++; + continue; + } + if (c === "`") { + let n = 0; + while (r[i + n] === "`") n++; + if (!tick) tick = n; + else if (tick === n) tick = 0; + cur += "`".repeat(n); + i += n - 1; + continue; + } + if (c === "|" && !tick) { + out.push(cur); + cur = ""; + continue; + } + cur += c; + } + out.push(cur); + if (out.length && !out[0].trim()) out.shift(); + if (out.length && !out[out.length - 1].trim()) out.pop(); + return out; + } + + function table(rows) { + const cells = (r) => splitRow(r.trim()).map((c) => inline(c.trim(), ctx)); + const head = cells(rows[0]); + const body = rows.slice(2).map(cells); + return ( + `${head.map((c) => ``).join("")}` + + body.map((r) => `${r.map((c) => ``).join("")}`).join("") + + "
${c}
${c}
" + ); + } +} + +// --------------------------------------------------------------------------- +const CSS = `:root{--bg:#14161a;--panel:#1b1e24;--fg:#d7dae0;--dim:#8b93a1;--accent:#8ec07c;--rule:#2b2f37;--mono:ui-monospace,"SFMono-Regular","Menlo","Consolas",monospace} +*{box-sizing:border-box} +body{margin:0;background:var(--bg);color:var(--fg);font:16px/1.65 system-ui,-apple-system,"Segoe UI",sans-serif} +a{color:var(--accent)} +.shell{display:flex;align-items:flex-start;max-width:1400px;margin:0 auto} +nav.side{position:sticky;top:0;flex:0 0 17rem;max-height:100vh;overflow-y:auto;padding:1.5rem 1rem 3rem;border-right:1px solid var(--rule);font-size:.9rem} +nav.side .brand{font-family:var(--mono);font-size:1.15rem;color:var(--accent);text-decoration:none;font-weight:600} +nav.side .navtoggle{display:none} +nav.side .navsummary{display:none;cursor:pointer;margin:.9rem 0 .2rem;padding:.4rem .55rem;border:1px solid var(--rule);border-radius:4px;font-family:var(--mono);font-size:.85rem;color:var(--dim)} +nav.side .navsummary::before{content:"\\2630 " / ""} +nav.side .navtoggle:checked~.navsummary{border-color:var(--accent);color:var(--fg)} +nav.side h3{font-size:.72rem;letter-spacing:.09em;text-transform:uppercase;color:var(--dim);margin:1.6rem 0 .4rem} +nav.side ul{list-style:none;margin:0;padding:0} +nav.side li{margin:.15rem 0} +nav.side a{display:block;padding:.2rem .45rem;border-radius:4px;color:var(--fg);text-decoration:none} +nav.side a:hover{background:var(--panel)} +nav.side a.here{background:var(--panel);color:var(--accent);font-weight:600} +main{flex:1 1 auto;min-width:0;max-width:60rem;padding:2.5rem 2rem 6rem} +h1,h2,h3,h4,h5{font-family:var(--mono);font-weight:600;letter-spacing:-.01em;line-height:1.3;overflow-wrap:anywhere} +h1{font-size:2rem;margin:0 0 1.2rem;color:var(--accent)} +h2{font-size:1.25rem;margin:2.6rem 0 .6rem;padding-top:1.2rem;border-top:1px solid var(--rule)} +h3{font-size:1.02rem;margin:1.8rem 0 .4rem;color:#c8cdd6} +h4,h5{font-size:.95rem;margin:1.3rem 0 .3rem;color:var(--dim)} +.anchor{float:left;margin-left:-1.1em;padding-right:.35em;color:var(--rule);text-decoration:none;opacity:0} +h2:hover .anchor,h3:hover .anchor,h4:hover .anchor,h5:hover .anchor{opacity:1} +p{max-width:74ch} +li{max-width:74ch} +code{font-family:var(--mono);font-size:.88em;background:var(--panel);padding:.1em .3em;border-radius:3px;overflow-wrap:anywhere} +a code{background:#22262e} +pre{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:1rem;overflow-x:auto} +pre code{background:none;padding:0;font-size:.84rem;line-height:1.5;overflow-wrap:normal} +blockquote{margin:1rem 0;padding:.5rem .9rem;border-left:3px solid var(--accent);background:var(--panel)} +blockquote p{margin:.4rem 0} +table{border-collapse:collapse;margin:1rem 0;font-size:.92rem;display:block;overflow-x:auto} +th,td{border:1px solid var(--rule);padding:.4rem .6rem;text-align:left;vertical-align:top} +th{background:var(--panel)} +hr{border:0;border-top:1px solid var(--rule);margin:2rem 0} +.tid{font-family:var(--mono);font-size:.86em;white-space:nowrap} +.tid-unknown{color:var(--dim);border-bottom:1px dotted var(--rule)} +.toc{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:.8rem 1.1rem;margin:0 0 2rem;font-size:.92rem} +.toc summary{cursor:pointer;color:var(--dim);font-family:var(--mono)} +.toc ul{list-style:none;margin:.6rem 0 0;padding:0;columns:2;column-gap:2rem} +.toc li{break-inside:avoid;margin:.12rem 0;overflow-wrap:anywhere} +.toc li.l3{padding-left:1rem;font-size:.92em} +.toc a{text-decoration:none;overflow-wrap:anywhere} +.toc a:hover{text-decoration:underline} +.jump{margin:0 0 2rem;font-size:.85rem;font-family:var(--mono)} +.jump a{display:inline-block;min-width:2.1rem;text-align:center;padding:.12rem .3rem;margin:.1rem;border:1px solid var(--rule);border-radius:4px;text-decoration:none} +.jump a:hover{border-color:var(--accent)} +.lede{color:var(--dim);max-width:70ch} +.src{font-size:.85rem;color:var(--dim);font-family:var(--mono);margin:-.6rem 0 1.8rem} +.cards{display:grid;gap:1rem;grid-template-columns:repeat(auto-fill,minmax(19rem,1fr));margin:1rem 0 2rem} +.card{background:var(--panel);border:1px solid var(--rule);border-radius:6px;padding:.9rem 1.1rem} +.card a{font-family:var(--mono);text-decoration:none} +.card p{color:var(--dim);font-size:.88rem;margin:.3rem 0 0} +.tickets{width:100%} +.tickets td:first-child{white-space:nowrap;font-family:var(--mono);font-size:.86rem} +.tickets td:last-child{overflow-wrap:anywhere} +.st{font-size:.78rem;color:var(--dim);text-transform:uppercase;letter-spacing:.05em} +@media(max-width:900px){.toc ul{max-height:45vh;overflow-y:auto}nav.side .navtoggle{display:block;position:absolute;width:1px;height:1px;margin:-1px;clip-path:inset(50%);overflow:hidden}nav.side .navtoggle:focus-visible~.navsummary{outline:2px solid var(--accent);outline-offset:2px}nav.side .navsummary{display:block}nav.side .navlinks{display:none}nav.side .navtoggle:checked~.navlinks{display:block}.shell{display:block}nav.side{position:static;flex:none;max-height:none;border-right:0;border-bottom:1px solid var(--rule)}main{padding:1.5rem 1rem 4rem}.toc ul{columns:1}}`; + +function nav(currentSlug, depth) { + const up = "../".repeat(depth); + const link = (href, label, here) => + `

  • ${label}
  • `; + // Do NOT "simplify" this to
    : the sidebar must force it OPEN, which + // `display` cannot do, and the desktop nav collapses. doc-site.md §7. + const parts = [ + `"); + return parts.join(""); +} + +function page({ slug, title, body, depth }) { + const up = "../".repeat(depth); + return ` + + + + +${esc(title)} — improvise docs + + + +
    +${nav(slug, depth)} +
    +${body} +
    +
    + + +`; +} + +function write(rel, content) { + const p = join(out, rel); + mkdirSync(dirname(p), { recursive: true }); + writeFileSync(p, content); + return p; +} + +// --------------------------------------------------------------------------- +rmSync(out, { recursive: true, force: true }); +mkdirSync(out, { recursive: true }); +write("style.css", CSS); + +const cited = new Set(); +const unresolved = new Set(); +const written = []; + +for (const section of SECTIONS) { + for (const [src, slug, title] of section.docs) { + const depth = slug.split("/").length - 1; + const ctx = { depth, cited, unresolved, anchors: new Set() }; + if (slug === "repo-map") ctx.listItemId = (n) => (n >= 1 && n <= 200 ? `gotcha-${n}` : null); + const md = readFileSync(join(root, src), "utf8"); + const { html, toc } = renderMarkdown(md, ctx); + + const tocHtml = toc.length + ? `
    On this page
      ${toc + .filter((t) => t.level <= 3) + .map((t) => `
    • ${t.text}
    • `) + .join("")}
    ` + : ""; + + // Derived from the anchors actually emitted. A hardcoded count fails closed + // on a deleted gotcha (the link check catches it) and SILENTLY on an added + // one, which is the direction the list moves in. + const gotchas = [...ctx.anchors] + .map((a) => /^gotcha-(\d+)$/.exec(a)) + .filter(Boolean) + .map((m) => Number(m[1])) + .sort((a, b) => a - b); + const jump = gotchas.length + ? `

    Gotchas: ${gotchas.map((n) => `${n}`).join("")}

    ` + : ""; + + const srcLine = `

    Source: ${src}

    `; + written.push({ + rel: `${slug}.html`, + html: page({ + slug, + title, + depth, + body: `${srcLine}${tocHtml}${jump}${html}`, + }), + anchors: ctx.anchors, + }); + } +} + +// Ticket index. +{ + const ctx = { depth: 0, cited, unresolved, anchors: new Set() }; + const ids = [...cited].sort(); + const rows = ids + .map((id) => { + const t = TICKETS.get(id); + ctx.anchors.add(id); + if (!t) + return `${id}not exportedCited in the prose; the committed tracker export does not carry it.`; + return `${id}${esc(t.status)}${esc(t.title)}`; + }) + .join(""); + const body = `

    Ticket index

    +

    The prose on this site cites issue ids like improvise-9c8c. They are this project's own tracker ids, kept in-repo with beads; there is no public issue URL to link to, so this page is what they resolve to. Titles come from .beads/issues.jsonl, a passive export that lags the database — an id it does not carry is listed here without a title and is not linked from the prose.

    +

    ${ids.length} ids are cited, of which ${ids.length - unresolved.size} resolve.

    +${rows}
    IdStatusTitle
    `; + written.push({ rel: "tickets.html", html: page({ slug: "tickets", title: "Ticket index", depth: 0, body }), anchors: ctx.anchors }); +} + +// Landing page for the docs section. +{ + // The two conventions go through `inline` rather than being hand-written, so + // the sentence describing the linking demonstrates it instead of asserting it. + const conventions = inline( + "Ids like `improvise-9c8c` are tracker issues and link to the ticket index below. " + + "Paths like `crates/improvise-core/src/model/types.rs` link to the source on GitHub at `" + + ref + + "`.", + { depth: 0, cited, unresolved, anchors: new Set() }, + ); + const cards = SECTIONS.map( + (s) => + `

    #${s.title}

    ${s.blurb}

    ${s.docs + .map(([, slug, title]) => ``) + .join("")}
    `, + ).join(""); + const body = `

    improvise — documentation

    +

    These are the working documents of the improvise repository, published as written. They are addressed to someone reading or changing the code: the arguments behind the architecture, the decisions that were made and the ones that were rejected, and the record of what each load-bearing test is guarding against.

    +

    Nothing here is a tutorial. For what the program does, see the project home or run the browser demo.

    +${cards} +

    #Conventions

    +

    ${conventions}

    `; + written.push({ rel: "index.html", html: page({ slug: "", title: "Documentation", depth: 0, body }), anchors: new Set(["conventions", ...SECTIONS.map((s) => s.id)]) }); +} + +for (const w of written) write(w.rel, w.html); + +// --------------------------------------------------------------------------- +// An nested in an is closed early by the parser, so the outer link +// becomes an empty unclickable stub and the inner one wins. The link check +// cannot see it: both hrefs resolve. It asks a different question. +const nested = []; +for (const w of written) { + for (const m of w.html.matchAll(/]*>((?:(?!<\/a>)[\s\S])*?) inside near ${JSON.stringify(m[1].slice(0, 60))}`); +} + +// Link check over the generated output. Every internal href must resolve to a +// generated file, and every fragment to an id that page really carries. +const anchorsByPage = new Map(written.map((w) => [w.rel, w.anchors])); +const files = new Set(written.map((w) => w.rel).concat(["style.css"])); +const broken = []; +let checked = 0; +for (const w of written) { + const base = dirname(w.rel); + for (const m of w.html.matchAll(/(?:href|src)="([^"]+)"/g)) { + const raw = m[1]; + if (/^(https?:|mailto:|data:)/.test(raw)) continue; + checked++; + const [pathPart, frag] = raw.split("#"); + if (!pathPart) { + if (frag && !w.anchors.has(frag)) broken.push(`${w.rel} -> ${raw} (no such anchor on this page)`); + continue; + } + const resolved = relative(".", join(base === "." ? "" : base, pathPart)); + // A link out of the docs tree (../index.html, ../app/) names a file this + // script does not generate, so it is checked on disk instead — which is why + // assemble.sh builds the docs LAST, once its siblings exist. + if (resolved.startsWith("..")) { + const onDisk = join(out, base === "." ? "" : base, pathPart); + const withIndex = raw.endsWith("/") ? join(onDisk, "index.html") : onDisk; + if (!existsSync(withIndex) && !existsSync(onDisk)) + broken.push(`${w.rel} -> ${raw} (no such file outside the docs tree: ${withIndex})`); + continue; + } + const target = resolved.endsWith("/") ? `${resolved}index.html` : resolved; + if (!files.has(target)) broken.push(`${w.rel} -> ${raw} (no such page: ${target})`); + else if (frag && !anchorsByPage.get(target)?.has(frag)) + broken.push(`${w.rel} -> ${raw} (no such anchor: ${frag})`); + } +} + +console.log(`doc-site: ${written.length} pages -> ${out}`); +console.log(`doc-site: ${cited.size} ticket ids cited, ${cited.size - unresolved.size} resolve, ${unresolved.size} not in the export`); +console.log(`doc-site: ${checked} internal links checked, ${broken.length} broken`); +console.log(`doc-site: ${nested.length} nested anchors`); +for (const n of nested) console.error(`doc-site: nested anchor: ${n}`); +if (broken.length || nested.length) { + for (const b of broken) console.error(`doc-site: broken link: ${b}`); + process.exit(1); +} diff --git a/web/assemble.sh b/web/assemble.sh index f6ba1e73..1fe1cd20 100755 --- a/web/assemble.sh +++ b/web/assemble.sh @@ -98,3 +98,10 @@ for entry in app.js worker.js; do } done done + +# The prose site at `docs/`, generated from context/ by the same script the +# harness serves. It runs LAST because its link check verifies the links that +# leave its own tree (`../index.html`, `../app/`) against the filesystem, which +# means their targets must already be here. It exits non-zero on a break, so a +# dead cross-reference fails the assembly rather than shipping. +node "$root/scripts/build-doc-site.mjs" "$out/docs" "$token"