Compare commits
19 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Edward Langley
|
00499fc2bf | ||
|
Edward Langley
|
178983bcbf | ||
|
Edward Langley
|
e09ddf71a7 | ||
|
Edward Langley
|
f8f8f537c3 | ||
|
Edward Langley
|
334597d825 | ||
|
Edward Langley
|
9329f04082 | ||
|
Edward Langley
|
631067b011 | ||
|
Edward Langley
|
bd5dcfe1f7 | ||
|
Edward Langley
|
0249afe33d | ||
|
Edward Langley
|
132f017c79 | ||
|
Edward Langley
|
f1a777670f | ||
|
Edward Langley
|
92f351bce3 | ||
|
Edward Langley
|
6f4bc5e798 | ||
|
Edward Langley
|
8e0c06d888 | ||
|
Edward Langley
|
32677141de | ||
|
Edward Langley
|
ecc2987963 | ||
|
Edward Langley
|
6d5138d904 | ||
|
Edward Langley
|
def3902eb9 | ||
|
Edward Langley
|
9d88ad3205 |
@@ -1,95 +0,0 @@
|
||||
---
|
||||
name: beads-merge
|
||||
description: Resolve or prevent git conflicts in .beads/issues.jsonl, and reconcile genuinely divergent issues. Use when a merge, rebase, cherry-pick or stash pop conflicts on that file, when auto-export reports "invalid character", or when setting up a fresh clone.
|
||||
---
|
||||
|
||||
# Merging `.beads/issues.jsonl`
|
||||
|
||||
**The database is the truth; the file is derived.** `.beads/issues.jsonl` is a
|
||||
passive export of the local Dolt DB, which syncs over `refs/dolt/data`. Nothing
|
||||
here ever merges the *file* — it regenerates it.
|
||||
|
||||
## The one rule
|
||||
|
||||
**Never hand-edit it, never `git checkout` it** (that fires the post-checkout
|
||||
hook and desynchronises `export-state.json`), and never resolve its conflict
|
||||
markers by editing. Use `bd` and regenerate.
|
||||
|
||||
## Setup — required once per clone
|
||||
|
||||
`.gitattributes` names the driver; the driver itself lives in git config, which
|
||||
is per-clone and **not** carried by a checkout. Without this, git silently falls
|
||||
back to line-merging and writes conflict markers into the JSONL.
|
||||
|
||||
```sh
|
||||
git config merge.beads.name "beads issues.jsonl: import both sides, regenerate from the database"
|
||||
git config merge.beads.driver "$PWD/scripts/beads-merge-driver.sh %O %A %B %P"
|
||||
git check-attr merge -- .beads/issues.jsonl # must print "merge: beads"
|
||||
```
|
||||
|
||||
A worktree shares `.git/config` with its parent, so a worktree needs no separate
|
||||
setup — but a fresh `git clone` does.
|
||||
|
||||
## What the driver does, and why it needs no merge logic
|
||||
|
||||
`scripts/beads-merge-driver.sh` imports **both** sides and re-exports. It writes
|
||||
no merge logic of its own because `bd import` already has the semantics:
|
||||
|
||||
- an id the DB has not seen is **created**
|
||||
- a row whose `updated_at` is **strictly newer** than the local one **wins**
|
||||
- an older row is **skipped** (reported as `stale_skipped_ids`)
|
||||
- a tie keeps **every local column** — `updated_at` has second granularity, so a
|
||||
same-second tie can be two distinct updates, and local wins
|
||||
|
||||
So neither import can move the database backwards, which is why importing *ours*
|
||||
as well as *theirs* is safe: it restores a database that has lost rows without
|
||||
overwriting one that is ahead.
|
||||
|
||||
Set `BEADS_MERGE_PULL=1` to `bd dolt pull` first (remote via `BEADS_MERGE_REMOTE`,
|
||||
default `gh`). Off by default: a merge driver that reaches the network fails in
|
||||
ways a merge cannot report.
|
||||
|
||||
## Reconciling a genuinely divergent issue
|
||||
|
||||
The driver resolves by recency, which is right for the common case (two sides
|
||||
filing different issues) and **wrong when both sides meaningfully edited the same
|
||||
issue** — the older edit is skipped silently.
|
||||
|
||||
Do **not** reconstruct that from the JSONL. `bd` already does it field by field
|
||||
against the Dolt working set, which is the real sync path:
|
||||
|
||||
```sh
|
||||
bd conflicts list # which issues are conflicted
|
||||
bd conflicts show improvise-XXXX # base / ours / theirs, per field
|
||||
bd conflicts resolve improvise-XXXX --ours # or --theirs
|
||||
bd conflicts resolve --all --theirs
|
||||
```
|
||||
|
||||
If the driver already took the newer side and you need the other, the losing
|
||||
content is in git history: `git show <their-commit>:.beads/issues.jsonl`, find
|
||||
the row, and apply the fields with `bd update`. Never import a whole stale file
|
||||
to recover one issue — that is what `--allow-stale` does, and it overwrites every
|
||||
newer local row.
|
||||
|
||||
## Recovering a corrupted file
|
||||
|
||||
If auto-export reports `failed to compare existing JSONL against local store:
|
||||
invalid character '<'`, the file has conflict markers in it and **auto-export has
|
||||
been silently skipping** — possibly for many commits.
|
||||
|
||||
```sh
|
||||
grep -nE '^(<<<<<<<|=======|>>>>>>>)' .beads/issues.jsonl # confirm
|
||||
bd export -o .beads/issues.jsonl # regenerate from the DB
|
||||
```
|
||||
|
||||
Check whether the markers reached a commit (`git log -S '<<<<<<<' -- .beads/issues.jsonl`).
|
||||
They are usually working-tree only.
|
||||
|
||||
## The trap that caused this
|
||||
|
||||
A **`git stash pop`** conflict writes `<<<<<<< Updated upstream` /
|
||||
`>>>>>>> Stashed changes` into the file. Git's autostash during a dirty-tree
|
||||
merge or rebase does this **without anyone running `stash` explicitly**, and the
|
||||
orphaned entry stays in `git stash list`. The merge driver covers stash pops too,
|
||||
but the habit that avoids it is: **commit before merging**, and treat a stray
|
||||
`autostash` in `git stash list` as a live hazard.
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
name: beads
|
||||
description: Use when working in a repository that uses bd or Beads for durable project task tracking, issue dependencies, blocker management, multi-session handoff, or shared work memory. Trigger when the user asks to find ready work, claim or close tasks, create follow-up work, inspect blockers, recover project context, or choose between local planning and persistent project tracking.
|
||||
---
|
||||
|
||||
# Beads
|
||||
|
||||
Use Beads as the shared project task system. Local plans, scratch files, and personal memories are useful, but they are not the durable source of truth for project work.
|
||||
|
||||
## First Step
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
bd prime
|
||||
```
|
||||
|
||||
If that prints nothing, check whether the repository has an active Beads workspace:
|
||||
|
||||
```bash
|
||||
bd where
|
||||
```
|
||||
|
||||
## Preferred Route
|
||||
|
||||
Use the `bd` CLI when shell access is available. It is the most compact and direct Beads interface.
|
||||
|
||||
## Core CLI Workflow
|
||||
|
||||
1. Find work:
|
||||
|
||||
```bash
|
||||
bd ready
|
||||
bd list --status=open
|
||||
bd list --status=in_progress
|
||||
```
|
||||
|
||||
2. Inspect before editing:
|
||||
|
||||
```bash
|
||||
bd show <id>
|
||||
```
|
||||
|
||||
3. Claim work atomically:
|
||||
|
||||
```bash
|
||||
bd update <id> --claim
|
||||
```
|
||||
|
||||
4. Create durable follow-up work when implementation reveals new tasks:
|
||||
|
||||
```bash
|
||||
bd create "Short title" --description="Why this exists and what needs to be done" --type=task --priority=2
|
||||
```
|
||||
|
||||
5. Close completed work:
|
||||
|
||||
```bash
|
||||
bd close <id> --reason="Completed"
|
||||
```
|
||||
|
||||
## What Belongs In Beads
|
||||
|
||||
Use Beads for:
|
||||
|
||||
- shared project tasks
|
||||
- blockers and dependencies
|
||||
- discovered follow-up work
|
||||
- work that must survive thread reset, compaction, or handoff
|
||||
- status that another person or agent should be able to resume
|
||||
|
||||
Use agent-local planning tools only for the current turn's execution checklist. Do not treat them as shared project state.
|
||||
|
||||
## Rules
|
||||
|
||||
- Do not create markdown TODO files as the source of truth when Beads is available.
|
||||
- Do not use `bd edit`; it opens an interactive editor. Use `bd update` flags instead.
|
||||
- Prefer `--json` when parsing `bd` output programmatically.
|
||||
- If hooks are installed, `bd prime` may already be injected. Run it manually when context is missing.
|
||||
- Do not auto-close or mutate tasks unless the work is actually complete.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Beads"
|
||||
short_description: "Project task tracking with bd"
|
||||
default_prompt: "Use $beads to inspect ready work and manage durable project tasks."
|
||||
@@ -1,138 +0,0 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Review a pull request or diff against this repository's own written architecture and testing doctrine — the Cmd/Effect split, exhaustive enums over AppMode/Axis/CategoryKind, crate layering, the CellKey/layout/persistence invariants, and when a green suite is not evidence. Use when reviewing any change to this repo, and before pushing your own work.
|
||||
---
|
||||
|
||||
# Code review
|
||||
|
||||
This repo has already written down what "correct shape" means, in more detail
|
||||
than most. A review here is mostly checking a change against that, rather than
|
||||
against generic standards.
|
||||
|
||||
Two passes, in order:
|
||||
|
||||
1. **Correctness** — does the change do what it claims? Wrong logic, missed
|
||||
edge cases, error paths that swallow failures. Ordinary review judgement.
|
||||
2. **Conformance** — does it fit the design? That is what the rest of this
|
||||
file is for, and it is the part a reviewer who has not read `context/`
|
||||
will miss.
|
||||
|
||||
## Read the sources, don't trust this file
|
||||
|
||||
This skill is an index. The rules live in files that change, and restating
|
||||
them here would let the copy drift:
|
||||
|
||||
| Source | What it holds |
|
||||
|---|---|
|
||||
| `context/design-principles.md` | The "why" — functional-first, polymorphism over conditionals, correctness by construction, layering, §7 testing doctrine |
|
||||
| `context/repo-map.md` | The "where", plus a **numbered gotcha digest** (the full text is `context/gotchas.md`) — read the long form of the ones your diff touches |
|
||||
| `prolog-sessions/kb.pl` | Executable knowledge: `invariant/2`, `doctrine/2`, `forbidden/2`, `decision/3`, and prior `work_done/3` |
|
||||
| `AGENTS.md` | Working rules: bug-first tests, `Option`/`Result`/`Box` at the edges, history reconciled only by merge, a new conventional commit by default (amend only an unpushed HEAD, for a small fix), and nothing that needs a force push |
|
||||
| `.agents/skills/wave/agent-rules.md`, "GitHub" | The reviewer's rule: what to read before your first comment, how to answer each existing review, and what every review you post must state. Read it before reviewing a PR |
|
||||
|
||||
`kb.pl` is the highest-value and least-obvious of these. Grep it before
|
||||
reviewing: `grep -n "invariant(\|doctrine(\|decision(" prolog-sessions/kb.pl`.
|
||||
A `decision/3` fact often explains why the thing you are about to flag looks
|
||||
the way it does.
|
||||
|
||||
## What to check
|
||||
|
||||
**Command/effect split.** `Cmd::execute` takes `&CmdContext` and returns
|
||||
`Vec<Box<dyn Effect>>`. Mutation happens only in `Effect::apply`. A command
|
||||
reaching for mutable state is the single most common shape error.
|
||||
|
||||
**New effects.** `kind()` has no default, so the compiler asks — but check
|
||||
the classification is right, not just present (`doctrine(derive_then_verify)`
|
||||
records naive extraction mis-classifying three effects). Note there is no
|
||||
`changes_mode()` any more: mode settling is observed after the batch, not
|
||||
predicted per type (gotcha 31, improvise-toc). Don't reintroduce a static
|
||||
flag for a conditional behaviour.
|
||||
|
||||
**Exhaustive matches.** `AppMode`, `Axis`, `BinOp`, `CategoryKind` and friends
|
||||
should match exhaustively — a `_ =>` arm is how adding a variant silently
|
||||
skips a site. When a new variant shares one property with an existing variant
|
||||
but not another, name both as predicates and route call sites through the one
|
||||
they mean, rather than testing for the variant
|
||||
(`doctrine(two_questions_one_enum)`; `Axis::filters_data` /
|
||||
`is_user_assignable` are the worked example).
|
||||
|
||||
**Layering.** `forbidden/2` in `kb.pl` is the authority:
|
||||
`improvise-core` no UI/IO, `improvise-io` no UI/commands, `improvise-command`
|
||||
no ratatui/crossterm. A ratatui type in core is the usual violation — the fix
|
||||
is a semantic value the render layer maps; for appearance that is a `Face` in
|
||||
`improvise-protocol`, resolved by `draw.rs::face_style` (ADR 004).
|
||||
|
||||
**`Option`/`Result`/`Box` at the boundaries.** Wrap at parse/lookup/IO, unwrap
|
||||
early, keep core logic on concrete values. Heap-container plumbing should be
|
||||
split from logic (`AGENTS.md`).
|
||||
|
||||
**Invariants the type system does not carry.** Check the relevant gotchas
|
||||
rather than reasoning from first principles — `CellKey` built through its
|
||||
constructors, `IndexMap` order being load-bearing for persistence, display
|
||||
rounding never fed back into eval, `app.layout` going stale mid-batch and
|
||||
needing `RebuildLayout` in front.
|
||||
|
||||
**Persistence.** The pest grammar is the source of truth; update it first.
|
||||
Wire tokens are not UI labels and must not follow them.
|
||||
|
||||
## Testing doctrine
|
||||
|
||||
From `design-principles.md` §7 and `doctrine/2`:
|
||||
|
||||
- **A bug fix needs a test that failed before it.** If the diff claims a fix
|
||||
with no such test, that is the finding. Ask whether it was run against the
|
||||
old code.
|
||||
- Prefer unit tests over integration/e2e; test the observable contract, not
|
||||
internal storage.
|
||||
- Suite stays **under 2 seconds**. Proptest stays at 256 cases — constrain the
|
||||
strategy instead of raising them.
|
||||
- `doctrine(check_the_checker)`: a new check should be proven by injecting the
|
||||
fault it targets. A check nobody has seen fail is indistinguishable from one
|
||||
that always passes.
|
||||
- `doctrine(populate_before_roundtrip)`: a roundtrip over a default-constructed
|
||||
value proves almost nothing.
|
||||
- `doctrine(binding_dump_diff)`: most keybindings have no direct test, so a
|
||||
green suite is **not** evidence for a keymap refactor — diff the dumped
|
||||
bindings.
|
||||
|
||||
## Verifying
|
||||
|
||||
Run the gate **CI runs**, flags and all. The list is
|
||||
`.agents/skills/wave/agent-rules.md`, "Gates", and it is kept only there. Run
|
||||
it from that file rather than from memory or from an older copy. A second
|
||||
copy of that list lived here and drifted: it said `cargo fmt --all --check`,
|
||||
which reaches only one of this repo's two workspace roots, and it left out
|
||||
`direnv exec .`, `scripts/checkers.sh` and `scripts/node-test.sh`
|
||||
(improvise-vjus). Flags matter. Clippy exits 0 on warnings, so without
|
||||
`-D warnings` a clippy run was green on exactly the input CI is red on.
|
||||
|
||||
**`--check-evidence` is additive**, so it answers the comment cap too and both
|
||||
verdicts print. It did not always: the flag used to *select*, so the flagged
|
||||
run skipped the cap and said nothing about having skipped it, and three
|
||||
separate agents hit that in one campaign — a local gate using only the flagged
|
||||
form went green while CI went red on a three-line doc comment (improvise-ijql).
|
||||
Run the bare form anyway when you only want the cap: it is 0.3s and needs no
|
||||
toolchain, where the flagged one needs a compiled workspace.
|
||||
|
||||
The knowledge base is checked in CI as code:
|
||||
|
||||
```bash
|
||||
nix shell nixpkgs#trealla --command tpl prolog-sessions/kb.pl \
|
||||
-g "(kb_consistent -> halt(0) ; (kb_report, halt(1)))"
|
||||
```
|
||||
|
||||
In a cloud container that needs one step first: `nix shell` resolves nixpkgs
|
||||
as a GitHub tarball, which is not served there. Run
|
||||
`scripts/nix-seed-flake-inputs.sh` to seed the locked inputs over git (it does
|
||||
not modify `flake.lock`), after which the command works. Details and the two
|
||||
alternatives are in `context/cloud-agents.md`. If you would rather not spend
|
||||
the time, review `kb.pl` edits by reading and let CI run the check.
|
||||
|
||||
## Closing the loop
|
||||
|
||||
If the review surfaces a durable fact — a decision, a new invariant, a method
|
||||
that worked — add it to `prolog-sessions/kb.pl` as `decision/3`,
|
||||
`invariant/2` or `doctrine/2`. If it changes where something lives or how it
|
||||
behaves, update `context/repo-map.md`. Those two files and
|
||||
`context/design-principles.md` are maintained by hand and are the reason this
|
||||
review is cheap next time.
|
||||
@@ -1,118 +0,0 @@
|
||||
---
|
||||
name: kb
|
||||
description: Read and write the Prolog knowledge base at prolog-sessions/kb.pl. Use when adding a doctrine, invariant, decision or work_done fact, correcting or retiring one, or querying the formal model. Never edit that file by hand and never write it through the Prolog MCP.
|
||||
---
|
||||
|
||||
# Writing the knowledge base
|
||||
|
||||
`prolog-sessions/kb.pl` is executable knowledge, gated by CI. **`scripts/kb` is
|
||||
the only thing that writes it.** `tpl` is assumed on `PATH` from the dev shell.
|
||||
|
||||
```
|
||||
scripts/kb check # kb_consistent, exactly what CI runs
|
||||
scripts/kb check --with <d.pl># does the model accept a design you have not built?
|
||||
scripts/kb count # clauses per keyed predicate
|
||||
scripts/kb query '<goal>' # run a goal; exit 1 if it fails
|
||||
scripts/kb add <delta.pl> # add clauses; refuses if a key already exists
|
||||
scripts/kb replace <delta.pl> # add, overwriting existing keys
|
||||
scripts/kb rm <keys.pl> # remove by key; refuses if a key is absent
|
||||
scripts/kb fmt # canonical form (idempotent)
|
||||
```
|
||||
|
||||
A delta is an ordinary Prolog file of clauses. For `rm`, a key is a clause head
|
||||
with the last argument left as `_`:
|
||||
|
||||
```prolog
|
||||
doctrine(a_name_that_went_false,_).
|
||||
```
|
||||
|
||||
## Designing against the model
|
||||
|
||||
Before implementing a ticket, state the design as facts and ask whether the
|
||||
model accepts it:
|
||||
|
||||
```prolog
|
||||
crate(improvise_report,'crates/improvise-report','render a View as a report').
|
||||
crate_depends_on(improvise_report,improvise_core).
|
||||
layer(report,improvise_report).
|
||||
```
|
||||
|
||||
`scripts/kb check --with design.pl` merges those into a throwaway candidate and
|
||||
runs `kb_consistent` over the union. Nothing is written either way.
|
||||
|
||||
It derives consequences you would not enumerate: one wrong dependency edge
|
||||
returns six `layering_inverted` violations through the transitive closure. It
|
||||
also catches schema mistakes — `layer/2` is `(layer, crate)`, and the wrong
|
||||
order comes back as `layer_names_unknown_crate` rather than being accepted.
|
||||
|
||||
What the model can refuse today: layering and dependency direction
|
||||
(`illegal_dependency`, `layering_inverted`, `may_use`), `ViewState` membership
|
||||
(`portable`, `snapshot_safe`, `view_state_admits`, `field_misplaced_in_view_state`),
|
||||
record obligations, and functional-key conflicts. A design touching those areas
|
||||
is worth stating; one that touches none of them will be accepted vacuously, so
|
||||
extend the model rather than trusting the green.
|
||||
|
||||
## Rules
|
||||
|
||||
**Write a fact when you learn it.** There is no reason to batch.
|
||||
|
||||
**Verify the claim against the tree before writing it.** A clause stating the
|
||||
negation of what the code does is worse than no clause
|
||||
(`doctrine(a_stale_gotcha_asserts_the_negation)`).
|
||||
|
||||
**If the NAME went false, retire it** — `rm` the old key, `add` the new one, and
|
||||
say in the new clause that it was renamed. A reader who trusts a name never
|
||||
reads the body.
|
||||
|
||||
**Put the argument in the clause.** Doctrine and decision text is where worked
|
||||
examples and history belong; prose docs should stay operational.
|
||||
|
||||
## Traps
|
||||
|
||||
- **`assertz` does not work here.** A consulted predicate is static;
|
||||
`assertz`/`retract` raise `permission_error(modify, static_procedure)`. This
|
||||
is why the tool reads clauses as data.
|
||||
- **`kb_consistent` does not check everything.** A clause name cited *inside
|
||||
another clause's string* is unchecked, so retiring a clause can dangle every
|
||||
clause referring to it (`improvise-2f97`).
|
||||
- **Reading through the MCP is fine.** `loadSession` at session start is still
|
||||
how a session gets the model in hand; only writing moved.
|
||||
|
||||
Every write checks three things before it replaces the file: the result reads
|
||||
back as intended, it still holds every clause **HEAD** has, and `kb_consistent`
|
||||
passes. So a bad delta leaves the file untouched. `count` before and after is
|
||||
still the cheapest evidence a write did what you meant.
|
||||
|
||||
## When a write is refused
|
||||
|
||||
Exit codes: `0` done, `1` refused, `2` usage, `3` HEAD's `kb.pl` could not be
|
||||
read, so nothing was compared, `127` no `tpl`. Only `1` is a verdict.
|
||||
|
||||
**"clauses HEAD has are gone, with no kb rm recorded"** means the working-tree
|
||||
`kb.pl` is short of the committed one. Your delta did not cause it. The file
|
||||
was already damaged, by a killed write, a bad merge resolution or an edit. The
|
||||
refusal lists the clauses that are missing. The check counts clauses **per
|
||||
key**, so losing one of two clauses that share a key is caught too.
|
||||
|
||||
- **Recover:** `git show HEAD:prolog-sessions/kb.pl > prolog-sessions/kb.pl`,
|
||||
then re-run your `add`/`replace`/`rm` deltas.
|
||||
- **Do not commit the short file to get past the refusal.** HEAD is what the
|
||||
check compares against, so a committed loss becomes the new floor and is
|
||||
never reported again.
|
||||
- **To remove a clause on purpose, use `kb rm`.** It records the removal in a
|
||||
per-worktree ledger (`git rev-parse --git-path kb-removed.pl`, never
|
||||
committed), and that record is what lets later writes accept the missing
|
||||
clause. A hand deletion has no record, so it is refused.
|
||||
|
||||
The ledger vouches only for the exact `kb.pl` file the tool last wrote. Any
|
||||
rewrite outside the tool voids it, including a restore, `git checkout`, `cp`
|
||||
or an editor. After an `rm`, a restore therefore also undoes the `rm`, and you
|
||||
must redo it. That is deliberate. Otherwise the abandoned `rm` would go on
|
||||
excusing a later, accidental loss of the same clause.
|
||||
|
||||
A killed write can leave `kb.pl.cand.*` files and a `kb.pl.lock` directory in
|
||||
`prolog-sessions/`. Git ignores them. The lock makes the next write wait 30
|
||||
seconds and then fail, so delete it if nothing else is writing.
|
||||
|
||||
`context/code/kb-tool.md` has the argument. `scripts/kb-test.sh` proves each
|
||||
refusal by injecting the fault it names.
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
name: pr-watch
|
||||
description: Wait for a GitHub PR's checks to finish before merging or reporting on it. Use whenever you need to know whether a PR is green. Do not hand-roll a polling loop over gh pr view.
|
||||
---
|
||||
|
||||
# Waiting on a PR
|
||||
|
||||
"Watching a PR" means tracking **both** CI checks **and** GitHub comments/reviews.
|
||||
A PR is not ready to merge just because checks are green — review feedback must be
|
||||
read and answered first.
|
||||
|
||||
## CI Checks
|
||||
|
||||
```
|
||||
scripts/pr-watch <pr-number> [--expect <sha>] [--timeout <sec>] [--interval <sec>] [--reviews]
|
||||
```
|
||||
|
||||
```
|
||||
exit 0 every check COMPLETED, none failed, MERGEABLE with a state of
|
||||
CLEAN, UNSTABLE or HAS_HOOKS, and reviews fetched (if --reviews)
|
||||
exit 1 a check failed (under any mergeability, BLOCKED included), or the
|
||||
PR is not mergeable (CONFLICTING, DIRTY, BEHIND, DRAFT, or a state
|
||||
it does not know)
|
||||
exit 2 the head moved during the wait, or is not --expect
|
||||
exit 3 timed out with checks still running, mergeability still UNKNOWN,
|
||||
or still BLOCKED with every registered check green
|
||||
exit 4 usage or environment error
|
||||
exit 5 the PR is MERGED or CLOSED, at the head the watch pinned; it ends on
|
||||
the first poll that sees it closed, which may be mid-wait
|
||||
exit 6 checks green/mergeable, but pr-reviews failed (fetch error, not "no reviews")
|
||||
```
|
||||
|
||||
**`--reviews`** (default): after checks complete, print all reviews and comments
|
||||
via `scripts/pr-reviews`. Use `--no-reviews` to skip this. The reviews are shown
|
||||
but do not change the exit code — answering them is your responsibility.
|
||||
|
||||
**Exit 0 now means mergeable, not only green.** GitHub computes mergeability
|
||||
lazily and reports `UNKNOWN` for a while after the base moves. It also reports
|
||||
`BLOCKED` for the moment before a required check that `needs:` another job has
|
||||
registered. The watch waits out both, so you do not need to re-check
|
||||
`mergeable` by hand after it. **Exit 3 is not a pass.** It may be a required
|
||||
check that never ran. Why each state counts the way it does:
|
||||
`context/code/tooling.md`, "pr-watch and mergeability" (improvise-edpl).
|
||||
|
||||
**Pass `--expect <sha>` whenever you know which commit you are asking about.**
|
||||
It is the difference between "this PR is green" and "the thing I pushed is
|
||||
green", and a PR here was once merged at a head that predated its own fix.
|
||||
|
||||
## Reviews and Comments
|
||||
|
||||
Reviews are a **separate API surface** from the comment thread. Codex/ChatGPT
|
||||
reviews arrive as *reviews*, not comments — checking only the comment thread
|
||||
misses them entirely. `pr-watch --reviews` (the default) shows all three:
|
||||
|
||||
| Surface | What it is |
|
||||
|---|---|
|
||||
| `[REVIEW]` | The review's own body (pulls/{n}/reviews) |
|
||||
| `[INLINE]` | A comment on a line of the diff (pulls/{n}/comments) |
|
||||
| `[COMMENT]` | A comment on the PR thread (issues/{n}/comments) |
|
||||
|
||||
To fetch reviews directly: `./scripts/pr-reviews <n>` — **do not pipe it**. The
|
||||
script's `$?` must be read directly; piping or chaining loses the exit code.
|
||||
Redirect to a file if you need to process it.
|
||||
|
||||
### Review Obligations
|
||||
|
||||
Before treating a PR as done or merging:
|
||||
|
||||
1. **Read all review feedback** — reviews against an earlier head may still apply
|
||||
2. **Answer substantive feedback with attribution** — every reply must include
|
||||
the attribution line (see below)
|
||||
3. **Re-check after each push** that new reviews target the current head SHA
|
||||
4. **Don't wait indefinitely on Copilot/Codex** — reviews arrive as quota allows;
|
||||
an absent review and an unrequested one look identical. The adversarial review
|
||||
that gates a merge is dispatched by the orchestrator; external reviews are
|
||||
read *in addition*, not instead.
|
||||
|
||||
### GitHub Attribution (Hard Rule)
|
||||
|
||||
**Every GitHub write must include an attribution line.** Posts go out under the
|
||||
repository owner's account — an unattributed post reads as the human's words.
|
||||
|
||||
Open the body with: `_Posted by <Agent Stack> (<model>)_`
|
||||
|
||||
Examples:
|
||||
- `_Posted by Claude (Opus 4.5)_`
|
||||
- `_Posted by Cursor Agent (claude-sonnet-5)_`
|
||||
|
||||
A thread is not "answered" until the reply carries this attribution.
|
||||
|
||||
See AGENTS.md "GitHub Attribution" for the full rule and "Shipping a wave" for
|
||||
Copilot request mechanics and how to answer declined findings.
|
||||
|
||||
## Do not write the loop yourself
|
||||
|
||||
Four signals look like "finished" and are not:
|
||||
|
||||
- **"nothing pending"** — also true when no check has registered yet. Require a
|
||||
**non-zero total** as well.
|
||||
- **`conclusion == null`** — `conclusion` is a *string*, empty while running.
|
||||
Test `status == "COMPLETED"`.
|
||||
- **a verdict from before the head moved** — re-check the head, or `--expect`.
|
||||
- **`mergeable: UNKNOWN`** — GitHub has not computed it yet. It is not a
|
||||
pass, and PRs #172 and #179 both went green on it.
|
||||
|
||||
`gh` is shell-aliased to a different tool in these shells; `pr-watch` locates a
|
||||
real one itself. Do not reintroduce a bare `gh` call in a script here.
|
||||
|
||||
## Before deciding a PR is stuck
|
||||
|
||||
Check suites can take several minutes to appear at all, and a Copilot review
|
||||
takes about six. Compare `date -u` against the GitHub timeline rather than a
|
||||
felt interval, and do not close and reopen a PR to unstick it.
|
||||
|
||||
`scripts/pr-watch-test.sh` drives every case from fixtures.
|
||||
@@ -1,121 +0,0 @@
|
||||
---
|
||||
name: wave
|
||||
description: Dispatch parallel agents to work through open beads and shepherd each to a merged PR. Use when an orchestrator is running a wave, partitioning the backlog, or writing a brief for a work-unit agent.
|
||||
---
|
||||
|
||||
# Running a wave
|
||||
|
||||
An orchestrator chunks the open backlog, fans agents out, reviews what comes
|
||||
back, and merges. `AGENTS.md` has the history; this is the operating procedure.
|
||||
This file is the brief checklist's one home, and `agent-rules.md` beside it
|
||||
holds the rules each agent follows.
|
||||
|
||||
## Every agent gets its own worktree. No exceptions.
|
||||
|
||||
Dispatch with `isolation: "worktree"`. Without it every agent edits the **same**
|
||||
working tree: one `git status` shows four agents' half-finished files, any commit
|
||||
sweeps up the others' work, and any `git checkout` destroys it. This has happened
|
||||
here. If you discover it mid-wave, stop every agent, have each create a worktree
|
||||
and copy across **only files it can prove it touched**, then `git checkout --`
|
||||
those files in the primary tree.
|
||||
|
||||
In the worktree `.envrc` must be allowed, or `bd` fails confusingly:
|
||||
`agent-rules.md`, "Discipline per ticket".
|
||||
|
||||
## Partitioning
|
||||
|
||||
By **crate or directory**, never by ticket count. **Tickets that share a file go
|
||||
to one agent**, to be done sequentially — four widget migrations all touching
|
||||
`src/draw.rs` are a guaranteed conflict split four ways. Group first, then count.
|
||||
|
||||
Tell each agent what ground is *not* theirs, and to report rather than guess when
|
||||
it must cross. Two agents adding a workspace member will still collide in
|
||||
`Cargo.toml`/`Cargo.lock`; keep both sides.
|
||||
|
||||
## What every brief must carry
|
||||
|
||||
**Point the agent at [`agent-rules.md`](agent-rules.md) and tell it to read it in
|
||||
full first.** That file is the agent's standing rules: base and push mechanics,
|
||||
the gate list, the claim, the vacuity shapes, evidence, the files it must not
|
||||
write, GitHub conventions (including how authors and reviewers answer reviews
|
||||
on the PR thread), the hand-back and disk cleanup. Do not restate them
|
||||
in the brief. A restated rule drifts from its source, and the brief is the copy
|
||||
nobody corrects. The brief adds only what varies per wave:
|
||||
|
||||
1. **The expected base commit, by name.** Worktrees do not reliably branch from
|
||||
HEAD. One was created three merges behind and was only noticed because it
|
||||
was told to check. The agent checks against it: `agent-rules.md` rule 1.
|
||||
It then merges `gh/main` as it moves, not only at the start: rule 9.
|
||||
2. **A reference baseline**, so "N passing" is checkable and a drop is
|
||||
visible: Rust test count, harness counts, node counts, **and the machine**.
|
||||
The same commit once measured `improvise-io`'s binary at 1.4s on one machine
|
||||
and 3.7–5.2s on another, so a figure with no machine is a claim about the
|
||||
suite, which it is not. The agent still measures its own.
|
||||
3. **Its ground, and the ground that is not.** Name any orchestrator-only file
|
||||
it is exceptionally allowed to edit.
|
||||
4. **Its `BEADS_ACTOR` and its tickets.** The actor also names the agent's
|
||||
commits: it commits through `scripts/agent-git`, which authors each one as
|
||||
`Claude (<BEADS_ACTOR>)` (`agent-rules.md`, "Discipline per ticket"). So
|
||||
give each agent its own actor, and do not tell it to set `user.name`.
|
||||
5. **The attribution line, commit trailers and PR footer**, spelled out.
|
||||
They name the agent (Claude's line opens `_Posted by Claude (<model>)…_`)
|
||||
and this session's URL, which is why the shared `agent-rules.md` cannot
|
||||
hold them.
|
||||
6. **An explicit ask for what the ticket did not predict.** `agent-rules.md`
|
||||
lists it in the hand-back. Say it in the brief anyway: it is consistently
|
||||
the most valuable part of a hand-back, and agents omit it unless asked.
|
||||
|
||||
When a hand-back teaches a new rule, add it to `agent-rules.md` in the same
|
||||
wave, not to the next brief.
|
||||
|
||||
## Method to require
|
||||
|
||||
`agent-rules.md`, "Discipline per ticket" and "Gates": failing test first, a
|
||||
rival injected and watched failing by name, `--no-fail-fast`, and test *names*
|
||||
diffed after any merge. Hold the hand-back to it. This repo's characteristic
|
||||
defect is a test that passes.
|
||||
|
||||
## Shepherding
|
||||
|
||||
- Wait on checks with `scripts/pr-watch <pr> --expect <sha>`. Never hand-roll a
|
||||
poll; see the `pr-watch` skill for the four ways they report green while wrong.
|
||||
- Review before merging. Dispatch an adversarial reviewer and give it the head
|
||||
SHA, the base, and a test-merge, so it reviews the tree that will exist. Point
|
||||
it at `agent-rules.md`, "GitHub", for what a reviewer reads and answers first.
|
||||
Copilot reviews here again (2026-09-24), mostly in inline comments, and Codex
|
||||
as quota allows. Read both with `scripts/pr-reviews <n>` and see each finding
|
||||
answered before merging. Do not wait on either: an absent review and an
|
||||
unrequested one both read as "pending".
|
||||
- Only the user's explicit approval accepts an ADR, and lets its PR merge. The
|
||||
review gates still apply, and none of them is approval:
|
||||
`AGENTS.md`, "ADR pull requests". **While an ADR PR is open, keep a
|
||||
background poll** for new non-agent comments and reviews, and resume its
|
||||
steward on each one. The user answers inline, and an answer left unread for
|
||||
hours is a failure.
|
||||
- **Before merging, check `baseRefName` is `main`**
|
||||
(`gh pr view <n> --json baseRefName`). A stacked PR whose base has already
|
||||
merged still targets the stale base branch, and merging it lands nothing in
|
||||
`main`. Retarget it with `gh pr edit <n> --base main` first. `AGENTS.md`,
|
||||
"Shipping a wave", has the 2026-09-24 incident.
|
||||
- **Land a batch of approved PRs as one integration PR.** Evidence files are
|
||||
append-at-EOF, so merging one at a time conflicts every other open PR. Then
|
||||
diff every conflict resolution against both parents and the merge base, and
|
||||
run the gates on the merged tree: `AGENTS.md`, "Merging many PRs at once".
|
||||
- A re-review is of the **delta**, at a named SHA. A review of a superseded
|
||||
commit reads as a pass.
|
||||
- Merge on GitHub with a merge commit, never a squash or rebase merge
|
||||
(agent-rules, rule 8). Then `git fetch gh && git merge --ff-only gh/main`. Close
|
||||
the bead only once the merge has landed. That is when the work is done:
|
||||
`AGENTS.md`, "Repository policy over the Beads blocks".
|
||||
- Anything posted to GitHub goes out under the owner's account: open every
|
||||
comment with a line naming the model and saying an agent wrote it. That
|
||||
wording is agent-neutral because this one file serves Claude and Codex
|
||||
(`.claude/skills/wave` is a symlink to `.agents/skills/wave`).
|
||||
|
||||
## Between waves
|
||||
|
||||
Run the code-review skill on the merged range. The characteristic find here is
|
||||
not a wrong line, it is **a test that passes while the thing it names is
|
||||
broken** — a vacuous fold test, a one-category fixture, a helper that fabricates
|
||||
what the code under test needs. Ask the reviewer to inject the fault and watch
|
||||
the check fail.
|
||||
@@ -1,415 +0,0 @@
|
||||
# Standing rules for a wave agent
|
||||
|
||||
Every work-unit agent reads this **in full before its first command**. A brief
|
||||
points here and adds only what is specific to its wave: the expected base
|
||||
commit, the agent's ground, its `BEADS_ACTOR`, and its tickets. Every rule below
|
||||
was bought by a failure.
|
||||
|
||||
This file holds each agent-side rule **once**. The brief checklist is
|
||||
`SKILL.md` beside it. Where `AGENTS.md` ("The Wave Workflow") carries a longer
|
||||
history, the rule here points there. Change a rule here, not in a brief, and
|
||||
not in a second copy.
|
||||
|
||||
## Base, branch, push
|
||||
|
||||
1. **Verify your base against the commit your brief names, first.** Run
|
||||
`git log --oneline -1` and compare. Worktrees do not reliably branch from
|
||||
the orchestrator's HEAD: one was created three merges behind, and it was
|
||||
only noticed because the agent was told to look.
|
||||
2. **Branch from `gh/main`, never `main` or `origin/main`.** `git fetch gh &&
|
||||
git checkout -b <branch> gh/main`. A worktree handed to you may already sit
|
||||
on unpushed commits. Why: `AGENTS.md`, "Shipping a wave" (the two remotes
|
||||
both spelled `main`, and the worktree-base vector).
|
||||
3. **Before opening a PR, check `git diff --numstat gh/main...<branch>`.** Do
|
||||
not use `gh pr view --json files`: that is the union of your commits' files,
|
||||
not the effective diff.
|
||||
4. **Push with an explicit refspec:** `git push gh HEAD:refs/heads/<branch>`. A
|
||||
branch cut from `gh/main` tracks `gh/main`, so a bare push targets `main` and
|
||||
is refused.
|
||||
5. **Before pushing a fix to an existing PR, check `gh pr view <n> --json
|
||||
state`.** GitHub deletes a merged PR's branch. A push to it then prints
|
||||
`* [new branch]` and lands nowhere. Why: `AGENTS.md`, "Shipping a wave".
|
||||
6. **The Beads "Session Completion" block near the end of `AGENTS.md` does not
|
||||
set your git policy.** This repository does not opt in to the block's
|
||||
"team-maintainer" profile. On a wave branch that profile's pull and bare
|
||||
`git push` would be refused (rule 4), or worse. Rules 1–5, 7, 8 and 9 are your
|
||||
git policy. The block is bd's and bd regenerates it whole, so it is not
|
||||
edited by hand: `AGENTS.md`, "Repository policy over the Beads blocks" (improvise-x1uk).
|
||||
7. **Pin `gh/main` to a literal SHA when the task starts.** Run
|
||||
`git rev-parse gh/main` once. Use that SHA, not the ref name, for every
|
||||
merge, merge-base and diff. Remote refs are shared by every worktree of the
|
||||
repository, so `gh/main` moves whenever another session fetches. Re-resolve
|
||||
only on purpose, for example when you mean to merge a newer main. Rule 3's
|
||||
check before a PR is such a time.
|
||||
8. **Reconcile diverging history only by merge.** To take a newer main,
|
||||
re-pin on purpose first (rule 7): `git fetch gh`, then `git rev-parse
|
||||
gh/main`. Then run `git merge <new SHA>`. Merge commits and new
|
||||
commits are what the rule wants.
|
||||
|
||||
**The default is a new commit.** The user: "As a general rule, you shuold
|
||||
just add a new conventional commit. Small adjustments can ammend HEAD, but
|
||||
when in doubt don't." [sic] (2026-09-25; improvise-1xfb.2). That narrows an
|
||||
earlier ruling: "I don't mind amending on a branch that hasn't been pushed,
|
||||
but don't do anything that requires a force push" (2026-09-25;
|
||||
improvise-79h5).
|
||||
- **Amend only HEAD, only for a small adjustment, and only while HEAD is
|
||||
unpushed.** A typo, a file left out of the commit you just made, a
|
||||
formatter's fix to it. When in doubt, add a new commit.
|
||||
- **Never reorder, fold or squash commits, pushed or not**, and never move
|
||||
them onto another base: no `git rebase` onto main and no `git reset
|
||||
--soft gh/main`. Both incidents below were exactly that, on commits that
|
||||
had not been pushed.
|
||||
- **Once a commit is pushed**, nothing may rewrite it: no rebase, squash
|
||||
(`git reset --soft`, `git merge --squash`, a `rebase -i` fixup), amend or
|
||||
`git reset` that drops it. Anything that would need a force push is
|
||||
forbidden, and so is the force push itself.
|
||||
|
||||
**Messages are Conventional Commits**: `type(scope): subject`, with the
|
||||
scope optional. The types are Conventional Commits' own: `feat`, `fix`,
|
||||
`refactor`, `perf`, `style`, `test`, `docs`, `ci` and `chore`. Do not
|
||||
invent new ones. History also has ad-hoc prefixes (`tooling:`, `context:`,
|
||||
`clippy:`, `checkers:`); write those as a standard type with a scope
|
||||
instead, e.g. `chore(tooling):`. A ticket id goes at the end of the subject
|
||||
in parentheses, as most recent conventional commits carry it: `fix(tui): the grid and the tile bar scroll
|
||||
by the width they are drawn in (improvise-gtn7.1)`. A merge commit keeps
|
||||
the message git writes for it. The message says what changed. The argument
|
||||
for the change lives in `context/`, never only in the message
|
||||
(`context/design-principles.md` §6).
|
||||
|
||||
The user's reason: "the merge structure of git history is a causal record"
|
||||
(2026-09-25; improvise-1xfb.1). A merge keeps that record, and a rewrite
|
||||
replaces it with a story. The rewrite is also where the damage comes from.
|
||||
A squash or rebase computes a new commit against a ref another session can
|
||||
move, and when that ref moves, the commit silently reverts whatever landed
|
||||
in between. It has happened twice, and making the operation safer did not
|
||||
stop the second:
|
||||
- **#196:** an agent ran `git reset --soft gh/main` mid-task and staged a
|
||||
reverse diff of #196. Only `git status` caught it (improvise-diho).
|
||||
- **#303:** after #303 merged mid-session, the improvise-cgqt agent squashed
|
||||
with `git reset --soft gh/main`. The staged commit reverted #303, and only
|
||||
rule 3's numstat caught it before the push.
|
||||
|
||||
A cherry-pick copies a change and drops its causal link, so use a merge
|
||||
wherever a merge does the job. That includes a branch that carries another
|
||||
PR's unmerged commits, when that PR is open and will land as a merge
|
||||
commit. Either stack on that PR, or wait for it to merge, re-pin, and merge
|
||||
the new SHA. The foreign commits then leave your diff.
|
||||
|
||||
The one place a merge cannot help is commits that will never reach main as
|
||||
they are. That means an abandoned or rebuilt PR, or commits that exist only
|
||||
on a mirror or a local main (#148). Only then, cut a fresh branch from your
|
||||
pinned SHA and `git cherry-pick -x` your own commits onto it (`-x` records
|
||||
where each came from). Close the old branch rather than force-pushing it:
|
||||
`AGENTS.md`, "Shipping a wave".
|
||||
9. **Merge main as soon as you can, and keep merging it.** The user:
|
||||
"Agents should merge main asap" and, when asked why, "Reduces conflicts"
|
||||
(2026-10-09 UTC; bd memory `agents-merge-main-asap`, written by the
|
||||
orchestrator; PR #335). Those two quotations are the user's words. The
|
||||
timing below is the orchestrator's operationalisation of them.
|
||||
- **When.** `git fetch gh`, then merge the new `gh/main` SHA (rule 7's
|
||||
re-pin, done on purpose) at the start of a task, before the first review
|
||||
request, before every push to a PR under review, and every time you
|
||||
resume. Main only gets further from your base, and a conflict met early
|
||||
is small. Never `origin/main`, never rebase, never squash (rule 8).
|
||||
Commit the merge through `scripts/agent-git`.
|
||||
- **After every merge of main, clean or not**, re-run the gates (see
|
||||
"Gates") on the merged tree, not on each side, and diff the test names
|
||||
against your pre-merge HEAD to see that none was lost. A clean merge is
|
||||
the dangerous case: it can delete a test module or break a shared test
|
||||
file with no conflict marker.
|
||||
- **Re-measure your baseline** after a mid-task merge, and report it
|
||||
against the merged SHA, so main's changes are not read as yours.
|
||||
- **Resolve a conflict by hand.** For each conflicted file, diff the
|
||||
result against both parents and the merge base, so a side's
|
||||
non-conflicting changes are not lost to a whole-file `--ours` or
|
||||
`--theirs`.
|
||||
- **A stacked PR merges its base branch, not `gh/main`.** GitHub diffs it
|
||||
against that base, so merging a `gh/main` ahead of the base puts main's
|
||||
commits in its diff (the #145/#148 symptom, which rule 3's numstat
|
||||
cannot see). The orchestrator retargets it to main once the base lands.
|
||||
- **The exception is narrow.** A durability push of WIP (see "Commit
|
||||
early") need not wait for a merge of main that conflicts. If you cannot
|
||||
resolve the conflict yet, push the WIP to a separate wip branch, never
|
||||
to the PR.
|
||||
- **Each merge-and-push moves the head SHA** an in-flight review is aimed
|
||||
at. Reviewers re-fetch the tip before posting, but do not merge main
|
||||
only to push a reply.
|
||||
- The mechanics are `AGENTS.md`, "Shipping a wave" (merge `gh/main`, never
|
||||
`origin/main`) and "Merging many PRs at once" (diff each parent; two
|
||||
clean changes to one test file can still fail together). Not restated here.
|
||||
|
||||
## Commit early: a worktree is not durable storage
|
||||
|
||||
Commit a WIP as soon as your first test run is green, and again after every
|
||||
meaningful step. Push your branch (rule 4) once it is worth keeping. An agent
|
||||
finished a whole ticket with every gate green and lost all of it, uncommitted,
|
||||
when its worktree was deleted in a disk cleanup. A local commit survives a
|
||||
deleted worktree. Make each step its own conventional commit, and leave it
|
||||
that way: steps are not reordered or folded together afterwards. A small fix
|
||||
to the step you just committed may amend it while it is unpushed. Anything
|
||||
more, or any doubt, is a new commit (rule 8).
|
||||
|
||||
## Gates
|
||||
|
||||
```
|
||||
direnv exec . ./scripts/fmt-workspaces.sh # NOT cargo fmt --all; exit 2 = the gate did not run
|
||||
direnv exec . ./scripts/check-harness-lock.sh # offline; a Cargo.toml change can stale harness/opfs-contract's lock
|
||||
direnv exec . cargo clippy --workspace --all-targets -- -D warnings
|
||||
direnv exec . cargo test --workspace --no-fail-fast
|
||||
direnv exec . python3 scripts/check-comment-runs.py --self-test
|
||||
direnv exec . python3 scripts/check-comment-runs.py --check-evidence # also answers the comment cap
|
||||
direnv exec . python3 scripts/check-ci-graph.py --self-test
|
||||
direnv exec . python3 scripts/check-ci-graph.py
|
||||
direnv exec . ./scripts/checkers.sh # every suite in scripts/, its own self-test included
|
||||
direnv exec . tpl prolog-sessions/kb.pl -g "(kb_consistent -> halt(0) ; (kb_report, halt(1)))"
|
||||
direnv exec . ./scripts/node-test.sh 'web/tests/*.test.js'
|
||||
```
|
||||
|
||||
This list is the `test`, `js` and `kb` jobs of `.github/workflows/ci.yml`,
|
||||
step for step, and this file is the one place it is written down. Change it in
|
||||
the same PR that changes CI. The other CI jobs are conditional, below, or left
|
||||
out on purpose:
|
||||
|
||||
- **`wasm`**: when you touch a crate it checks (next bullet).
|
||||
- **`harness-lint`**: when you touch `harness/opfs-contract`, run
|
||||
`cargo clippy --target wasm32-unknown-unknown --all-targets -- -D warnings`
|
||||
from that directory. Its format half is already `fmt-workspaces.sh`.
|
||||
- **`build` and `browser`**: when you touch `web/`, `harness/`, or the surface
|
||||
the page calls in `improvise-wasm-client` or `improvise-worker-server`
|
||||
(third bullet).
|
||||
- **`pages-deploy`** is left out. It deploys, and it runs only on a push to
|
||||
`main`. What guards it locally is `check-ci-graph.py`, which is in the list.
|
||||
|
||||
- **Never `cargo fmt --all`.** It reaches only **one** of this repo's two
|
||||
workspace roots, because `harness/opfs-contract` declares its own
|
||||
`[workspace]`. The obvious repair is worse: `cargo fmt -p opfs-contract` from
|
||||
the repo root exits **1**, *"not a member of the workspace"*, which looks
|
||||
exactly like a real formatting failure (improvise-b4ks). Both spellings fail
|
||||
in the direction that looks like success.
|
||||
- **Run every gate under `direnv exec .`.** `./scripts/checkers.sh` scrubs the
|
||||
environment to an allowlist, so a bare run drops `tpl` off `PATH` and reports
|
||||
spurious failures.
|
||||
- **Never pipe a gate.** `$?` must be the gate's own. And
|
||||
`check-comment-runs.py --check-evidence` prints two lines with the comment
|
||||
cap **first**, so `| tail -1` silently drops half of it.
|
||||
`fmt-workspaces.sh` is the one most often lost this way: an unformatted tree
|
||||
exits **1** and prints a diff, and `| tail` shows the diff's last lines and
|
||||
exits 0. Read its exit status, or run it with `--fix` and commit what it
|
||||
rewrites.
|
||||
- **CI takes every tool from `nix develop .#ci`, and so should your scripts**:
|
||||
no `/bin/…` or `/usr/bin/…` paths, and no bash-version or tool-version
|
||||
probes (`AGENTS.md`, "CI takes its tools from the dev shell"). A test that
|
||||
needs a restricted `PATH` builds one on purpose. It does not assume the
|
||||
runner's `/bin` holds anything, because on the NixOS runners it holds only
|
||||
`env` and `sh`.
|
||||
- **`--no-fail-fast`**, because plain `cargo test --workspace` stops at the
|
||||
first failing binary. Use it for any claim about blast radius.
|
||||
- If you touch a browser-bundle crate (core, formula, io, protocol, command,
|
||||
wasm-client, worker-server) or `improvise-gp`, also run
|
||||
`cargo check --target wasm32-unknown-unknown -p <crate>` **for each crate**.
|
||||
CI checks each one separately, so feature unification cannot hide a leak.
|
||||
- If you touch `web/`, `harness/` or the page's wasm surface, run
|
||||
`./harness/build.sh`, then
|
||||
`./scripts/node-test.sh 'web/tests/bundle/*.test.js'`, then `npm ci` and
|
||||
`npx playwright test` in `harness/`. The bundle tests sit outside the
|
||||
`web/tests/*.test.js` glob and read the built `web/pkg`, so the standard list
|
||||
never runs them. They failed only in CI once. A fresh worktree has no
|
||||
`harness/node_modules`, and Playwright then reports a missing
|
||||
`@playwright/test`, which reads like a broken checkout.
|
||||
- **Playwright serves `harness/serve/`, not `web/`.** `harness/build.sh`
|
||||
assembles that tree and it is gitignored, so after any change to `web/*.js`
|
||||
— yours, or one a merge brought in — re-run `./harness/build.sh` before the
|
||||
specs. Otherwise they test the old page. That failure looks exactly like a
|
||||
real regression (`doctrine(rebuild_the_assembled_site_after_merging_web)`).
|
||||
CI runs Chromium and Firefox only (`--project=chromium --project=firefox`),
|
||||
so run those two locally.
|
||||
- **If you change the grid's row or column count, also grep
|
||||
`harness/fixtures/*.improv` and `harness/tests/*.spec.js`.** On PR #176 a
|
||||
Playwright spec pinned the old collapsed-group row count. `cargo test` cannot
|
||||
see that.
|
||||
- **A PR that conflicts with its base gets no CI run.** So anything "left to
|
||||
CI" is unchecked until the PR is mergeable (PR #177).
|
||||
- **Measure your own baseline** before changing anything, and report it with
|
||||
the machine it was measured on. A figure from the brief is a reference, not
|
||||
your baseline.
|
||||
- **Merge `gh/main` and re-run every gate before calling a PR green.** Then
|
||||
check that no test *disappeared*. Diff test **names**, not counts: a
|
||||
textually clean merge can delete a test module with no conflict marker, and
|
||||
the other side's new tests mask the loss in a count.
|
||||
- **Build each side's test list in a fresh `CARGO_TARGET_DIR`, every time.** If
|
||||
a parent's tree comes from `git archive`, a reused target dir gives a wrong
|
||||
answer, not just a slow one. Archive stamps every file with its commit time,
|
||||
so cargo judges the sources older than the last build and lists the previous
|
||||
tree's tests. On #188 that made three of the branch's own tests look
|
||||
merge-only.
|
||||
- **The shell guard refusing a compound command means NEITHER part ran.** Split
|
||||
the command. After a refusal, re-check the state it was meant to produce.
|
||||
- **Re-run the full suite after reverting an injected fault.** A revert applied
|
||||
to the wrong occurrence once shipped two faults past every other gate.
|
||||
|
||||
## Discipline per ticket
|
||||
|
||||
- **Claim first:** `export BEADS_ACTOR=<your actor>` in every shell, then
|
||||
`bd update <id> --claim`. Exit 13 means another actor holds it. Pick another;
|
||||
do not retry.
|
||||
- **Author every commit as `Claude (<BEADS_ACTOR>)`.** The user: "agents
|
||||
should override git author name, I think to include an agent identifier
|
||||
(matching BEADS_ACTOR)" (2026-09-25; improvise-1xfb.3). Make every commit
|
||||
through the wrapper, merge commits included:
|
||||
`BEADS_ACTOR=<your actor> scripts/agent-git commit …`, and the same for
|
||||
`merge`. The wrapper sets the author name for that one command, and it refuses (exit 2) when `BEADS_ACTOR` is unset or empty. It
|
||||
reads `BEADS_ACTOR` from its own environment, so the prefix works in a shell
|
||||
that keeps nothing between commands.
|
||||
- **The author email, the committer and the `Co-Authored-By` trailer stay as
|
||||
they are.** The email is the owner's configured one, so GitHub still links
|
||||
the commit.
|
||||
- **Never set `user.name` with `git config`.** Worktrees share
|
||||
`.git/config`, so it renames every agent at once. Do not use
|
||||
`git -c user.name=…` either: `user.name` names the committer as well as
|
||||
the author. Why the wrapper uses `GIT_AUTHOR_NAME`:
|
||||
`context/test-evidence/tooling.md`.
|
||||
- A plain `git commit --amend` keeps the author it amends, so amend with
|
||||
plain git or with the wrapper.
|
||||
- **A cherry-pick keeps its source's author**, through the wrapper or not:
|
||||
git ignores `GIT_AUTHOR_NAME` there. That is right, because rule 8 allows
|
||||
only `cherry-pick -x` of commits that keep whoever wrote them.
|
||||
- Before you push, check `git log --format='%an <%ae> | %cn' <pinned
|
||||
SHA>..HEAD`. Every commit **you made** must name your actor. A
|
||||
cherry-picked commit, or one from a PR you stack on, keeps its own author.
|
||||
- **The format is a proposal.** `Claude (<BEADS_ACTOR>)` is the default
|
||||
proposed on improvise-1xfb.3's PR. The user may change it there. The format
|
||||
lives in `scripts/agent-git` and nowhere else.
|
||||
- **`bd` needs `.envrc` allowed in your worktree.** Without it you get a
|
||||
**version mismatch**, not a clean failure. Run `direnv allow` in the worktree,
|
||||
or rely on the `[whitelist] prefix` in `~/.config/direnv/direnv.toml`.
|
||||
`direnv status` reports `Found RC allowed 0`, and `0` means **allowed** (the
|
||||
enum is 0=Allowed, 1=NotAllowed). It reads like the opposite.
|
||||
- **Reproduce at HEAD before fixing.** A dozen tickets were stale-open in one
|
||||
campaign, one fixed two days before it was filed. If it does not reproduce,
|
||||
find the fixing commit, confirm with `git merge-base --is-ancestor <sha> HEAD`,
|
||||
report the SHA, and close nothing.
|
||||
- **Write a failing test first, then inject a rival and watch it fail.** Run
|
||||
it; do not reason about it. Report *where* it failed, by case name. An
|
||||
injection that does not compile reads as a clean run, so get a positive
|
||||
signal that the build ran.
|
||||
- **Bound any test loop that walks a state machine until a step.** An
|
||||
unbounded `while step != target { advance() }` hangs the suite when the
|
||||
transition it counts on is broken. Loop a fixed number of times and assert
|
||||
the step was reached, so the failure is a named assertion, not a timeout.
|
||||
- **Check your test against every vacuity shape seen here:**
|
||||
- an assertion the default state already satisfies;
|
||||
- `f(x) == f(x)`: calling a pure `Cmd` twice against one immutable context;
|
||||
- a disjunctive assertion whose other branch is the bug's own symptom;
|
||||
- a rival that fails the *setup*, which proves the neighbouring question
|
||||
(e.g. setup and subject share an `EffectKind`);
|
||||
- a fixture that is right by accident (expected value 1, group at row 0,
|
||||
bystander already on the relayed value);
|
||||
- a fixture or stub that lacks the structure the hazard needs, or fails in a
|
||||
wider shape than the real fault;
|
||||
- hand-picked fixture cases where the set could be **enumerated**: four
|
||||
vacuities in `fmt-workspaces-test.sh` were each a case nobody thought of
|
||||
(improvise-90ea);
|
||||
- a rendered-output test at a width where the thing asserted is never drawn;
|
||||
- a fold test that batches differently from the real producer (a
|
||||
`Binding::Sequence` is ONE effect vec);
|
||||
- **the cursor at index 0**: "the item under the cursor" and "the first item"
|
||||
are indistinguishable there, and that was four of five findings in one
|
||||
review. Seat the cursor at a nonzero index of a list with at least two
|
||||
elements;
|
||||
- a determinism check against ONE context instance: every `HashMap` iterates
|
||||
the same way twice. Build two equal contexts independently
|
||||
(improvise-hzjo);
|
||||
- two derivations of one fact where the fixture makes them agree (drawn vs
|
||||
committed, edit text vs value, `1200` vs `1.23456789`). Use a value where
|
||||
they disagree.
|
||||
- **Record each bug in `context/test-evidence/<area>.md`**, keyed by test name
|
||||
(module-qualify the name when the leaf is not unique). It is append-at-EOF, so
|
||||
**expect a conflict**. Keep both sides and verify line by line against both
|
||||
parents: `--check-evidence` cannot see a dropped entry.
|
||||
- **Never resolve a conflict with `git checkout --ours` or `--theirs`.** Either
|
||||
one takes the whole file from one side and silently drops the other side's
|
||||
*non*-conflicting changes to it. That is how a `ci.yml` step vanished while
|
||||
the merge message said it was unchanged (improvise-dndu). After resolving,
|
||||
diff the merge against each parent and against the merge base
|
||||
(`AGENTS.md`, "Merging many PRs at once").
|
||||
- **Comment runs are at most two lines.** Reasoning goes in `context/`, not in
|
||||
comments (`context/design-principles.md` §6).
|
||||
- **If the acceptance criteria look wrong, argue it** rather than following
|
||||
them silently. Three agents inverted theirs in one campaign, and all three
|
||||
were right.
|
||||
|
||||
## Files you must not write
|
||||
|
||||
`.beads/**` (never commit `issues.jsonl`, never `git checkout` it),
|
||||
`prolog-sessions/kb.pl`, `context/repo-map.md`, `context/gotchas.md`,
|
||||
`context/file-inventory.md`: `AGENTS.md`, "Who owns what".
|
||||
Also `AGENTS.md`, unless your brief grants you a named section. Report proposed
|
||||
text for any of these in your hand-back.
|
||||
|
||||
## GitHub
|
||||
|
||||
- `gh` is shell-aliased to a different tool. Use
|
||||
`nix shell nixpkgs#gh --command gh …`, and pass bodies with `-F body=@file`.
|
||||
- **Everything posted goes out under the repository owner's account.** Open
|
||||
every PR body and comment with the attribution line your brief gives you,
|
||||
verbatim. It names your agent and model, which is how our posts are told
|
||||
apart from an automated Codex review (`AGENTS.md`, "Shipping a wave").
|
||||
- The attribution line, commit trailers and PR footer all come from your brief.
|
||||
They name the agent and the session, so this shared file cannot hold them.
|
||||
- Read reviews with `./scripts/pr-reviews <n>`, and do not pipe it. It also
|
||||
shows inline comments, which is where Copilot puts most of its findings.
|
||||
Copilot reviews here again (2026-09-24), so answer its findings as you answer
|
||||
Codex's. How to tell a Codex review from ours: `AGENTS.md`, "Shipping a
|
||||
wave".
|
||||
- **As a PR's author, reply on the PR to every review finding you address.**
|
||||
Name the commit that fixes it, or say why you declined it. Putting it only in
|
||||
your hand-back to the orchestrator is not enough: the PR thread is where the
|
||||
next reader looks.
|
||||
- **As a reviewer, read the existing reviews first**, with
|
||||
`./scripts/pr-reviews <n>`, unpiped. Reply on its thread to every Codex
|
||||
review: fixed (name the commit), declined (say why), or already one of our
|
||||
tickets (name it). Every review you post states the head SHA it reviewed.
|
||||
- **Answer a review even if the PR has already merged.** Reviews arrive late,
|
||||
and the thread is still where the next reader looks.
|
||||
- **Do not merge, and do not close beads.** Your part ends at an open PR plus
|
||||
your hand-back. What "done" means is in `AGENTS.md`, "Repository policy over
|
||||
the Beads blocks". Avoid stacked PRs. If you must stack, say in the PR body which PR it is
|
||||
stacked on. Once that base PR merges, check the stacked PR's `baseRefName`
|
||||
(`gh pr view <n> --json baseRefName`) and, unless it is `main`, retarget it
|
||||
with `gh pr edit <n> --base main` before anyone merges it: merging it
|
||||
otherwise lands it in the stale base branch, not in `main`. Fifteen PRs did
|
||||
that on 2026-09-24 (`AGENTS.md`, "Shipping a wave").
|
||||
|
||||
## Hand-back, per ticket
|
||||
|
||||
Report:
|
||||
|
||||
- your branch, head SHA and PR number. Review is dispatched against the SHA;
|
||||
- what changed, per ticket;
|
||||
- the claim result;
|
||||
- your measured baseline;
|
||||
- whether the ticket reproduced (the SHA if it did not);
|
||||
- the rival and where it failed;
|
||||
- what you could not verify, and why;
|
||||
- proposed repo-map and `kb.pl` text;
|
||||
- **what the ticket did not predict**, consistently the most valuable part.
|
||||
|
||||
## Disk
|
||||
|
||||
Each worktree's `target/` grows to 3–6G, and a full disk (ENOSPC) once stopped
|
||||
every agent. **When you hand back, delete your own build output, and only
|
||||
yours.**
|
||||
|
||||
Run `pwd` and `git rev-parse --show-toplevel` first, and read the answer. Then
|
||||
delete by **literal absolute path**, typed out:
|
||||
`rm -rf /Users/.../.claude/worktrees/agent-<you>/target`.
|
||||
|
||||
**Never interpolate a computed path into `rm -rf`.** One agent's worktree had
|
||||
been removed, so `$(git rev-parse --show-toplevel)` printed nothing and the
|
||||
command became `rm -rf "/target"`. A guarded one-liner was tried, but the
|
||||
worktree shell guard refuses it as too complex, which pushed agents back to an
|
||||
unguarded form. A literal path is the version that is both allowed and safe.
|
||||
|
||||
Keep your branch and any uncommitted work. Delete only `target/` (and
|
||||
`harness/opfs-contract/target` if you built it). If you see ENOSPC, stop and
|
||||
report rather than deleting anything outside your worktree.
|
||||
@@ -1,74 +0,0 @@
|
||||
# Dolt database (managed by Dolt, not git)
|
||||
dolt/
|
||||
embeddeddolt/
|
||||
|
||||
# Runtime files
|
||||
bd.sock
|
||||
bd.sock.startlock
|
||||
sync-state.json
|
||||
last-touched
|
||||
.exclusive-lock
|
||||
|
||||
# Daemon runtime (lock, log, pid)
|
||||
daemon.*
|
||||
|
||||
# Interactions log (runtime, not versioned)
|
||||
interactions.jsonl
|
||||
|
||||
# Push state (runtime, per-machine)
|
||||
push-state.json
|
||||
|
||||
# Lock files (various runtime locks)
|
||||
*.lock
|
||||
|
||||
# Credential key (encryption key for federation peer auth — never commit)
|
||||
.beads-credential-key
|
||||
|
||||
# Local version tracking (prevents upgrade notification spam after git ops)
|
||||
.local_version
|
||||
|
||||
# Worktree redirect file (contains relative path to main repo's .beads/)
|
||||
# Must not be committed as paths would be wrong in other clones
|
||||
redirect
|
||||
|
||||
# Sync state (local-only, per-machine)
|
||||
# These files are machine-specific and should not be shared across clones
|
||||
.sync.lock
|
||||
export-state/
|
||||
export-state.json
|
||||
|
||||
# Ephemeral store (SQLite - wisps/molecules, intentionally not versioned)
|
||||
ephemeral.sqlite3
|
||||
ephemeral.sqlite3-journal
|
||||
ephemeral.sqlite3-wal
|
||||
ephemeral.sqlite3-shm
|
||||
|
||||
# Dolt server management (auto-started by bd)
|
||||
dolt-server.pid
|
||||
dolt-server.log
|
||||
dolt-server.lock
|
||||
dolt-server.port
|
||||
dolt-server.activity
|
||||
dolt-server-config.yaml
|
||||
|
||||
# Corrupt backup directories (created by bd doctor --fix recovery)
|
||||
*.corrupt.backup/
|
||||
|
||||
# Backup data (auto-exported JSONL, local-only)
|
||||
backup/
|
||||
|
||||
# Per-project environment file (Dolt connection config, GH#2520)
|
||||
.env
|
||||
|
||||
# Legacy files (from pre-Dolt versions)
|
||||
*.db
|
||||
*.db?*
|
||||
*.db-journal
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
db.sqlite
|
||||
bd.db
|
||||
# NOTE: Do NOT add negation patterns here.
|
||||
# They would override fork protection in .git/info/exclude.
|
||||
# Config files (metadata.json, config.yaml) are tracked by git by default
|
||||
# since no pattern above ignores them.
|
||||
@@ -1,81 +0,0 @@
|
||||
# Beads - AI-Native Issue Tracking
|
||||
|
||||
Welcome to Beads! This repository uses **Beads** for issue tracking - a modern, AI-native tool designed to live directly in your codebase alongside your code.
|
||||
|
||||
## What is Beads?
|
||||
|
||||
Beads is issue tracking that lives in your repo, making it perfect for AI coding agents and developers who want their issues close to their code. No web UI required - everything works through the CLI and integrates seamlessly with git.
|
||||
|
||||
**Learn more:** [github.com/steveyegge/beads](https://github.com/steveyegge/beads)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Essential Commands
|
||||
|
||||
```bash
|
||||
# Create new issues
|
||||
bd create "Add user authentication"
|
||||
|
||||
# View all issues
|
||||
bd list
|
||||
|
||||
# View issue details
|
||||
bd show <issue-id>
|
||||
|
||||
# Update issue status
|
||||
bd update <issue-id> --claim
|
||||
bd update <issue-id> --status done
|
||||
|
||||
# Sync with Dolt remote
|
||||
bd dolt push
|
||||
```
|
||||
|
||||
### Working with Issues
|
||||
|
||||
Issues in Beads are:
|
||||
- **Git-native**: Stored in Dolt database with version control and branching
|
||||
- **AI-friendly**: CLI-first design works perfectly with AI coding agents
|
||||
- **Branch-aware**: Issues can follow your branch workflow
|
||||
- **Always in sync**: Auto-syncs with your commits
|
||||
|
||||
## Why Beads?
|
||||
|
||||
✨ **AI-Native Design**
|
||||
- Built specifically for AI-assisted development workflows
|
||||
- CLI-first interface works seamlessly with AI coding agents
|
||||
- No context switching to web UIs
|
||||
|
||||
🚀 **Developer Focused**
|
||||
- Issues live in your repo, right next to your code
|
||||
- Works offline, syncs when you push
|
||||
- Fast, lightweight, and stays out of your way
|
||||
|
||||
🔧 **Git Integration**
|
||||
- Automatic sync with git commits
|
||||
- Branch-aware issue tracking
|
||||
- Dolt-native three-way merge resolution
|
||||
|
||||
## Get Started with Beads
|
||||
|
||||
Try Beads in your own projects:
|
||||
|
||||
```bash
|
||||
# Install Beads
|
||||
curl -sSL https://raw.githubusercontent.com/steveyegge/beads/main/scripts/install.sh | bash
|
||||
|
||||
# Initialize in your repo
|
||||
bd init
|
||||
|
||||
# Create your first issue
|
||||
bd create "Try out Beads"
|
||||
```
|
||||
|
||||
## Learn More
|
||||
|
||||
- **Documentation**: [github.com/steveyegge/beads/docs](https://github.com/steveyegge/beads/tree/main/docs)
|
||||
- **Quick Start Guide**: Run `bd quickstart`
|
||||
- **Examples**: [github.com/steveyegge/beads/examples](https://github.com/steveyegge/beads/tree/main/examples)
|
||||
|
||||
---
|
||||
|
||||
*Beads: Issue tracking that moves at the speed of thought* ⚡
|
||||
@@ -1,60 +0,0 @@
|
||||
# Beads Configuration File
|
||||
# This file configures default behavior for all bd commands in this repository
|
||||
# All settings can also be set via environment variables (BD_* prefix)
|
||||
# or overridden with command-line flags
|
||||
|
||||
# Issue prefix for this repository (used by bd init)
|
||||
# If not set, bd init will auto-detect from directory name
|
||||
# Example: issue-prefix: "myproject" creates issues like "myproject-1", "myproject-2", etc.
|
||||
# issue-prefix: ""
|
||||
|
||||
# Use no-db mode: JSONL-only, no Dolt database
|
||||
# When true, bd will use .beads/issues.jsonl as the source of truth
|
||||
# no-db: false
|
||||
|
||||
# Enable JSON output by default
|
||||
# json: false
|
||||
|
||||
# Feedback title formatting for mutating commands (create/update/close/dep/edit)
|
||||
# 0 = hide titles, N > 0 = truncate to N characters
|
||||
# output:
|
||||
# title-length: 255
|
||||
|
||||
# Default actor for audit trails (overridden by BEADS_ACTOR or --actor)
|
||||
# actor: ""
|
||||
|
||||
# Export events (audit trail) to .beads/events.jsonl on each flush/sync
|
||||
# When enabled, new events are appended incrementally using a high-water mark.
|
||||
# Use 'bd export --events' to trigger manually regardless of this setting.
|
||||
events-export: true
|
||||
# Multi-repo configuration (experimental - bd-307)
|
||||
# Allows hydrating from multiple repositories and routing writes to the correct database
|
||||
# repos:
|
||||
# primary: "." # Primary repo (where this database lives)
|
||||
# additional: # Additional repos to hydrate from (read-only)
|
||||
# - ~/beads-planning # Personal planning repo
|
||||
# - ~/work-planning # Work planning repo
|
||||
|
||||
# JSONL backup (periodic export for off-machine recovery)
|
||||
# Auto-enabled when a git remote exists. Override explicitly:
|
||||
# backup:
|
||||
# enabled: false # Disable auto-backup entirely
|
||||
# interval: 15m # Minimum time between auto-exports
|
||||
# git-push: false # Disable git push (export locally only)
|
||||
# git-repo: "" # Separate git repo for backups (default: project repo)
|
||||
|
||||
# Integration settings (access with 'bd config get/set')
|
||||
# These are stored in the database, not in this file:
|
||||
# - jira.url
|
||||
# - jira.project
|
||||
# - linear.url
|
||||
# - linear.api-key
|
||||
# - github.org
|
||||
# - github.repo
|
||||
sync.remote: "git+https://github.com/fiddlerwoaroof/improvise.git"
|
||||
export:
|
||||
auto: true
|
||||
git-add: false
|
||||
include-memories: false
|
||||
validation:
|
||||
on-create: warn
|
||||
@@ -1,24 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
# --- BEGIN BEADS INTEGRATION v1.0.2 ---
|
||||
# This section is managed by beads. Do not remove these markers.
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
export BD_GIT_HOOK=1
|
||||
_bd_timeout=${BEADS_HOOK_TIMEOUT:-300}
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$_bd_timeout" bd hooks run post-checkout "$@"
|
||||
_bd_exit=$?
|
||||
if [ $_bd_exit -eq 124 ]; then
|
||||
echo >&2 "beads: hook 'post-checkout' timed out after ${_bd_timeout}s — continuing without beads"
|
||||
_bd_exit=0
|
||||
fi
|
||||
else
|
||||
bd hooks run post-checkout "$@"
|
||||
_bd_exit=$?
|
||||
fi
|
||||
if [ $_bd_exit -eq 3 ]; then
|
||||
echo >&2 "beads: database not initialized — skipping hook 'post-checkout'"
|
||||
_bd_exit=0
|
||||
fi
|
||||
if [ $_bd_exit -ne 0 ]; then exit $_bd_exit; fi
|
||||
fi
|
||||
# --- END BEADS INTEGRATION v1.0.2 ---
|
||||
@@ -1 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
# --- BEGIN BEADS INTEGRATION v1.0.2 ---
|
||||
# This section is managed by beads. Do not remove these markers.
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
export BD_GIT_HOOK=1
|
||||
_bd_timeout=${BEADS_HOOK_TIMEOUT:-300}
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$_bd_timeout" bd hooks run post-merge "$@"
|
||||
_bd_exit=$?
|
||||
if [ $_bd_exit -eq 124 ]; then
|
||||
echo >&2 "beads: hook 'post-merge' timed out after ${_bd_timeout}s — continuing without beads"
|
||||
_bd_exit=0
|
||||
fi
|
||||
else
|
||||
bd hooks run post-merge "$@"
|
||||
_bd_exit=$?
|
||||
fi
|
||||
if [ $_bd_exit -eq 3 ]; then
|
||||
echo >&2 "beads: database not initialized — skipping hook 'post-merge'"
|
||||
_bd_exit=0
|
||||
fi
|
||||
if [ $_bd_exit -ne 0 ]; then exit $_bd_exit; fi
|
||||
fi
|
||||
# --- END BEADS INTEGRATION v1.0.2 ---
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
# --- BEGIN BEADS INTEGRATION v1.0.2 ---
|
||||
# This section is managed by beads. Do not remove these markers.
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
export BD_GIT_HOOK=1
|
||||
_bd_timeout=${BEADS_HOOK_TIMEOUT:-300}
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$_bd_timeout" bd hooks run pre-commit "$@"
|
||||
_bd_exit=$?
|
||||
if [ $_bd_exit -eq 124 ]; then
|
||||
echo >&2 "beads: hook 'pre-commit' timed out after ${_bd_timeout}s — continuing without beads"
|
||||
_bd_exit=0
|
||||
fi
|
||||
else
|
||||
bd hooks run pre-commit "$@"
|
||||
_bd_exit=$?
|
||||
fi
|
||||
if [ $_bd_exit -eq 3 ]; then
|
||||
echo >&2 "beads: database not initialized — skipping hook 'pre-commit'"
|
||||
_bd_exit=0
|
||||
fi
|
||||
if [ $_bd_exit -ne 0 ]; then exit $_bd_exit; fi
|
||||
fi
|
||||
# --- END BEADS INTEGRATION v1.0.2 ---
|
||||
@@ -1,25 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
# --- BEGIN BEADS INTEGRATION v1.0.2 ---
|
||||
# This section is managed by beads. Do not remove these markers.
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
export BD_GIT_HOOK=1
|
||||
_bd_timeout=${BEADS_HOOK_TIMEOUT:-300}
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$_bd_timeout" bd hooks run pre-push "$@"
|
||||
_bd_exit=$?
|
||||
if [ $_bd_exit -eq 124 ]; then
|
||||
echo >&2 "beads: hook 'pre-push' timed out after ${_bd_timeout}s — continuing without beads"
|
||||
_bd_exit=0
|
||||
fi
|
||||
else
|
||||
bd hooks run pre-push "$@"
|
||||
_bd_exit=$?
|
||||
fi
|
||||
if [ $_bd_exit -eq 3 ]; then
|
||||
echo >&2 "beads: database not initialized — skipping hook 'pre-push'"
|
||||
_bd_exit=0
|
||||
fi
|
||||
if [ $_bd_exit -ne 0 ]; then exit $_bd_exit; fi
|
||||
fi
|
||||
# --- END BEADS INTEGRATION v1.0.2 ---
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
#!/usr/bin/env sh
|
||||
# --- BEGIN BEADS INTEGRATION v1.0.2 ---
|
||||
# This section is managed by beads. Do not remove these markers.
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
export BD_GIT_HOOK=1
|
||||
_bd_timeout=${BEADS_HOOK_TIMEOUT:-300}
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
timeout "$_bd_timeout" bd hooks run prepare-commit-msg "$@"
|
||||
_bd_exit=$?
|
||||
if [ $_bd_exit -eq 124 ]; then
|
||||
echo >&2 "beads: hook 'prepare-commit-msg' timed out after ${_bd_timeout}s — continuing without beads"
|
||||
_bd_exit=0
|
||||
fi
|
||||
else
|
||||
bd hooks run prepare-commit-msg "$@"
|
||||
_bd_exit=$?
|
||||
fi
|
||||
if [ $_bd_exit -eq 3 ]; then
|
||||
echo >&2 "beads: database not initialized — skipping hook 'prepare-commit-msg'"
|
||||
_bd_exit=0
|
||||
fi
|
||||
if [ $_bd_exit -ne 0 ]; then exit $_bd_exit; fi
|
||||
fi
|
||||
# --- END BEADS INTEGRATION v1.0.2 ---
|
||||
File diff suppressed because one or more lines are too long
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"database": "dolt",
|
||||
"backend": "dolt",
|
||||
"dolt_mode": "embedded",
|
||||
"dolt_database": "improvise",
|
||||
"project_id": "1ccea08a-5afb-4b57-acad-78282e9e3af6"
|
||||
}
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Prepare a Claude Code on the web container for work on this repository.
|
||||
#
|
||||
# This hook delegates to the shared `scripts/agent-bootstrap.sh`, which works
|
||||
# for any coding agent stack (Claude Code, Cursor, etc.). This file exists
|
||||
# because Claude Code expects hooks at `.claude/hooks/session-start.sh`.
|
||||
#
|
||||
# For local checkouts, only the bd pin assertion runs.
|
||||
# For cloud sessions (CLAUDE_CODE_REMOTE=true), full bootstrap runs.
|
||||
#
|
||||
# See also: scripts/agent-bootstrap.sh (the stack-agnostic implementation)
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="${CLAUDE_PROJECT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
|
||||
BOOTSTRAP="$ROOT/scripts/agent-bootstrap.sh"
|
||||
|
||||
# Forward the Claude env file variable to the generic name the bootstrap uses
|
||||
if [ -n "${CLAUDE_ENV_FILE:-}" ]; then
|
||||
export AGENT_ENV_FILE="$CLAUDE_ENV_FILE"
|
||||
fi
|
||||
|
||||
# For local checkouts, only verify the bd pin (no install, no sync)
|
||||
if [ "${CLAUDE_CODE_REMOTE:-}" != "true" ]; then
|
||||
# Inline the pin-check only, since the full bootstrap is cloud-only
|
||||
cd "$ROOT" || exit 0
|
||||
beads_pin() {
|
||||
sed -n 's|.*beads\.url[[:space:]]*=[[:space:]]*"github:[^/]*/[^/]*/v\{0,1\}\([^"]*\)".*|\1|p' \
|
||||
flake.nix 2>/dev/null | head -1
|
||||
}
|
||||
bd_running_version() {
|
||||
bd version 2>/dev/null | sed -n 's/^bd version \([0-9][0-9.]*\).*/\1/p' | head -1
|
||||
}
|
||||
if command -v bd >/dev/null 2>&1; then
|
||||
pinned=$(beads_pin)
|
||||
running=$(bd_running_version)
|
||||
# Fail closed: if we can't parse either version, warn rather than silently pass
|
||||
if [ -z "$pinned" ]; then
|
||||
printf '[session-start] WARNING: cannot parse beads pin from flake.nix\n' >&2
|
||||
printf '[session-start] The pin check is skipped, but mutating bd commands may be unsafe.\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ -z "$running" ]; then
|
||||
printf '[session-start] WARNING: cannot parse bd version from "bd version"\n' >&2
|
||||
printf '[session-start] bd: %s\n' "$(command -v bd)" >&2
|
||||
printf '[session-start] The pin check is skipped, but mutating bd commands may be unsafe.\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$running" != "$pinned" ]; then
|
||||
printf '\n[session-start] ===== BEADS VERSION SKEW =====\n' >&2
|
||||
printf '[session-start] pinned (flake.nix): %s\n' "$pinned" >&2
|
||||
printf '[session-start] running: %s (%s)\n' "$running" "$(command -v bd)" >&2
|
||||
printf '[session-start] Reading is safe. Do NOT run mutating bd commands.\n' >&2
|
||||
printf '[session-start] Fix: nix develop, or direnv allow.\n' >&2
|
||||
printf '[session-start] =============================\n\n' >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Cloud session: run the full bootstrap
|
||||
exec "$BOOTSTRAP"
|
||||
@@ -1,52 +0,0 @@
|
||||
{
|
||||
"hooks": {
|
||||
"SessionStart": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh",
|
||||
"type": "command"
|
||||
}
|
||||
],
|
||||
"matcher": ""
|
||||
},
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "bd prime --hook-json",
|
||||
"type": "command"
|
||||
}
|
||||
],
|
||||
"matcher": ""
|
||||
}
|
||||
]
|
||||
},
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(cargo build:*)",
|
||||
"Bash(cargo test:*)",
|
||||
"Bash(cargo clippy:*)",
|
||||
"Bash(cargo fmt:*)",
|
||||
"Bash(cargo run -- cmd:*)",
|
||||
"Bash(cargo run -- script:*)",
|
||||
"Bash(bd ready:*)",
|
||||
"Bash(bd list:*)",
|
||||
"Bash(bd show:*)",
|
||||
"Bash(bd prime:*)",
|
||||
"Bash(bd stats:*)",
|
||||
"Bash(bd update:*)",
|
||||
"Bash(bd close:*)",
|
||||
"Bash(bd create:*)",
|
||||
"Bash(git status:*)",
|
||||
"Bash(git diff:*)",
|
||||
"Bash(git log:*)",
|
||||
"Bash(git show:*)",
|
||||
"Bash(git branch:*)",
|
||||
"Bash(git fetch:*)",
|
||||
"Bash(git ls-remote:*)"
|
||||
],
|
||||
"deny": [
|
||||
"Edit(./.beads/issues.jsonl)"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/beads
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/beads-merge
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/code-review
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/kb
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/pr-watch
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/wave
|
||||
@@ -1,6 +0,0 @@
|
||||
[features]
|
||||
hooks = true
|
||||
|
||||
[mcp_servers.prolog-mcp]
|
||||
command = "node"
|
||||
args = ["/Users/edwlan/git_repos/github.com/adamrybinski/prolog-mcp/dist/index.js"]
|
||||
@@ -1,51 +0,0 @@
|
||||
{
|
||||
"hooks": {
|
||||
"PostCompact": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "bd codex-hook PostCompact",
|
||||
"statusMessage": "Scheduling Beads context refresh",
|
||||
"type": "command"
|
||||
}
|
||||
],
|
||||
"matcher": "manual|auto"
|
||||
}
|
||||
],
|
||||
"PreCompact": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "bd codex-hook PreCompact",
|
||||
"statusMessage": "Checking Beads context",
|
||||
"type": "command"
|
||||
}
|
||||
],
|
||||
"matcher": "manual|auto"
|
||||
}
|
||||
],
|
||||
"SessionStart": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "bd codex-hook SessionStart",
|
||||
"statusMessage": "Loading Beads context",
|
||||
"type": "command"
|
||||
}
|
||||
],
|
||||
"matcher": "startup|resume|clear"
|
||||
}
|
||||
],
|
||||
"UserPromptSubmit": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"command": "bd codex-hook UserPromptSubmit",
|
||||
"statusMessage": "Refreshing Beads context",
|
||||
"type": "command"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"name": "Improvise",
|
||||
"install": "nix-daemon >/tmp/nix-daemon.log 2>&1 & ./scripts/agent-bootstrap.sh",
|
||||
"start": "nix-daemon &"
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/beads
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/beads-merge
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/code-review
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/kb
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/pr-watch
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/wave
|
||||
@@ -1 +0,0 @@
|
||||
.beads/issues.jsonl merge=beads
|
||||
@@ -1,6 +0,0 @@
|
||||
# Labels the self-hosted NixOS microVM runners register with.
|
||||
self-hosted-runner:
|
||||
labels:
|
||||
- nixos
|
||||
- microvm
|
||||
- audited
|
||||
@@ -1,388 +0,0 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
# `push` unrestricted plus `pull_request` ran every job TWICE on any branch
|
||||
# with an open PR — visible as the (push) / (pull_request) pairs in the checks
|
||||
# list. Restricting push to main means a PR branch is covered once by
|
||||
# pull_request, and main once by push, which is also what pages-deploy needs.
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
|
||||
# Every job runs on the self-hosted NixOS microVM runners, and every `run:`
|
||||
# step inside the flake's `ci` dev shell. Generic-Linux downloads (rustup,
|
||||
# setup-node's node, Playwright's browsers) cannot run on NixOS, so the
|
||||
# toolchain comes from flake.nix, the same one `nix develop` gives locally.
|
||||
# Each step re-enters the shell; after the first, the eval cache makes it cheap.
|
||||
defaults:
|
||||
run:
|
||||
shell: nix develop .#ci --command bash --noprofile --norc -eo pipefail {0}
|
||||
|
||||
jobs:
|
||||
# ---------------------------------------------------------------------------
|
||||
# The one place wasm-pack runs (improvise CI dedup).
|
||||
#
|
||||
# Before this job there were two: `pages-build` ran `web/build.sh`, and the
|
||||
# browser harness ran `harness/build.sh`, which CALLS `web/build.sh`. The two
|
||||
# wasm-pack builds were therefore done twice per run, in parallel jobs, with
|
||||
# separate rust-cache keys (`add-job-id-key` is on by default, so `browser`
|
||||
# and `pages-build` never shared a target dir). Measured on run 35535225637
|
||||
# (push to main, caches warm): 68s inside `pages-build`, and the same 68s
|
||||
# inside the browser job's 110s build step.
|
||||
#
|
||||
# Everything downstream now DOWNLOADS. The invariant that buys is artifact
|
||||
# identity: the harness and Pages are served bytes from one wasm-pack run,
|
||||
# so "the harness runs against the tree Pages publishes" (improvise-vvl3) is
|
||||
# true by construction rather than because two scripts agree.
|
||||
#
|
||||
# The native binary is deliberately NOT built here. Nothing downstream
|
||||
# consumes one — `cargo test --workspace` in the `test` job builds its own,
|
||||
# for its own target dir and its own cache key, and a `needs:` on this job
|
||||
# would serialise a 109s job behind a 126s one for no artifact at all.
|
||||
build:
|
||||
name: Build all bundles
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# First, because it needs nothing and takes a second: the gate that stops
|
||||
# an unstamped bundle shipping was deleted by injection on PR #69 and the
|
||||
# assembly still exited 0 (improvise-w8ih). It guards the assemble step
|
||||
# below, so it fails under this job's name rather than another's.
|
||||
- name: The stamp gate refuses what it claims to
|
||||
run: ./scripts/checkers.sh --ci build
|
||||
# Built by Nix (packages.web and packages.site in nix/ci.nix), which runs
|
||||
# the same harness/build.sh and web/assemble.sh a developer runs, so there
|
||||
# is still one answer to "how is the bundle built". `web` is the three
|
||||
# wasm bundles plus `harness/serve`; `site` re-assembles from web's pkg
|
||||
# dirs with the commit SHA as the cache-bust token (improvise-9x6), where
|
||||
# the harness tree is stamped `harness`. Dependencies are their own
|
||||
# derivations, substituted from titan's cache when Cargo.lock is unchanged.
|
||||
- name: Build the bundles, the harness tree and the site
|
||||
run: |
|
||||
nix build --print-build-logs --out-link result-web .#web
|
||||
nix build --print-build-logs --out-link result-site .#site
|
||||
# The upload steps below read these paths. Copied rather than linked:
|
||||
# an artifact of a symlink into /nix/store is a dangling link elsewhere.
|
||||
- name: Lay the outputs out where the uploads read them
|
||||
run: |
|
||||
rm -rf harness/serve web/pkg web/worker-pkg site
|
||||
cp -rL result-web/harness-serve harness/serve
|
||||
cp -rL result-web/pkg web/pkg
|
||||
cp -rL result-web/worker-pkg web/worker-pkg
|
||||
cp -rL result-site site
|
||||
chmod -R u+w harness/serve web/pkg web/worker-pkg site
|
||||
|
||||
- run: find site -type f | sort
|
||||
- run: du -sh site
|
||||
|
||||
# The tree the browser harness serves. Uploaded whole rather than
|
||||
# reassembled downstream: a second assembly path is a second answer to
|
||||
# "what does the harness serve", which is the thing improvise-vvl3 exists
|
||||
# to prevent.
|
||||
- name: Upload the harness tree
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: harness-serve
|
||||
path: harness/serve
|
||||
retention-days: 1
|
||||
include-hidden-files: true
|
||||
|
||||
# For a job that needs the raw wasm-pack output rather than the
|
||||
# assembled site — the Node bundle suite under `web/tests/bundle/`.
|
||||
- name: Upload the wasm bundles
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: wasm-bundles
|
||||
path: |
|
||||
web/pkg
|
||||
web/worker-pkg
|
||||
retention-days: 1
|
||||
include-hidden-files: true
|
||||
|
||||
- uses: actions/upload-pages-artifact@v5
|
||||
with:
|
||||
path: site
|
||||
|
||||
test:
|
||||
name: Test and lint (host)
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# NOT `cargo fmt --all`, which reaches only ONE of the two workspace
|
||||
# roots. Exit 2 here is a usage error, never a verdict (improvise-b4ks).
|
||||
- name: Every workspace root is formatted
|
||||
run: ./scripts/fmt-workspaces.sh
|
||||
# A manifest-only change to a crate the harness reaches by path stales
|
||||
# its own lock, which only the slow `build` job saw (improvise-zs2r).
|
||||
- name: The harness lock matches the manifests it resolves against
|
||||
run: ./scripts/check-harness-lock.sh --allow-network
|
||||
# checks.clippy and checks.test in nix/ci.nix: the same cargo commands,
|
||||
# on the dev/test profiles, with dependencies as a cached derivation.
|
||||
- name: cargo clippy --workspace --all-targets -- -D warnings
|
||||
run: nix build --print-build-logs .#checks.x86_64-linux.clippy
|
||||
# Also runs the test-evidence gate (check-comment-runs.py
|
||||
# --check-evidence), which lists the tests this derivation compiled.
|
||||
- name: cargo test --workspace, and test-evidence headings name real tests
|
||||
run: nix build --print-build-logs .#checks.x86_64-linux.test
|
||||
# The comment cap and the test-evidence keys (improvise-9c8c); without
|
||||
# this step the rule is advisory. See context/design-principles.md §6.
|
||||
- name: The comment checker catches what it has been fixed for
|
||||
run: python3 scripts/check-comment-runs.py --self-test
|
||||
- name: Comment runs capped at two lines
|
||||
run: python3 scripts/check-comment-runs.py
|
||||
# `pages-deploy` needs every other job, and saying so was not enough: a
|
||||
# job added during review of PR #83 was ungated and nothing noticed.
|
||||
- name: The deploy gate refuses what it claims to
|
||||
run: python3 scripts/check-ci-graph.py --self-test
|
||||
- name: Every check job gates the deploy
|
||||
run: python3 scripts/check-ci-graph.py
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The crate2nix build (Cargo.nix), which `cargo` never exercises. It was off
|
||||
# because GitHub-hosted runners start with an empty store and rebuilt every
|
||||
# dependency crate each run. The NixOS runners substitute from titan's
|
||||
# harmonia first, and titan builds gh/main's `#improvise` hourly, so a PR that
|
||||
# leaves Cargo.lock alone rebuilds only the workspace crates it changed.
|
||||
#
|
||||
# The repo-stats pipeline runs here because it needs this job's binary:
|
||||
# `.#repo-stats-tools` is that build plus code-maat and its JRE, which stay
|
||||
# out of the ci shell every other job enters.
|
||||
# Its suite gates; the report on a push to main never fails the build, and
|
||||
# its step fails only when improvise refuses the pipeline's input
|
||||
# (context/code/tooling.md section 7). Full history is for the report.
|
||||
nix-build:
|
||||
name: Nix build (crate2nix) and the repo-stats pipeline
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: nix build .#improvise
|
||||
run: nix build --print-build-logs .#improvise
|
||||
- name: The repo-stats pipeline computes what it claims to
|
||||
run: nix shell .#repo-stats-tools --command ./scripts/checkers.sh --ci nix-build
|
||||
- name: Churn, coupling and doc-growth report (report only)
|
||||
if: github.event_name == 'push'
|
||||
continue-on-error: true
|
||||
run: |
|
||||
rc=0
|
||||
nix shell .#repo-stats-tools --command ./scripts/repo-stats --out repo-stats > repo-stats.txt || rc=$?
|
||||
{ echo '```'; cat repo-stats.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
|
||||
exit "$rc"
|
||||
- name: Upload the repo-stats model and its CSVs
|
||||
if: github.event_name == 'push'
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: repo-stats
|
||||
path: repo-stats
|
||||
retention-days: 30
|
||||
|
||||
wasm:
|
||||
name: wasm32 check (browser bundle crates)
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# checks.wasm-check in nix/ci.nix: `cargo check --target wasm32` for each
|
||||
# crate in the browser bundle, one at a time, so features are not unified
|
||||
# across them. The root `improvise` crate is the terminal binary and
|
||||
# depends on crossterm, which does not build for wasm — that is correct,
|
||||
# not a gap (improvise-ywd). The log names each crate as it is checked.
|
||||
- name: cargo check --target wasm32-unknown-unknown, per bundle crate
|
||||
run: nix build --print-build-logs .#checks.x86_64-linux.wasm-check
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The browser JavaScript's unit tests. Its own job, deliberately: a failure
|
||||
# here is a decision `web/shell.js` or `web/serial.js` got wrong, and
|
||||
# reporting that as "Browser harness" would put it behind an 80 MB browser
|
||||
# download and two wasm-pack builds. It needs neither — `node --test` is in
|
||||
# the runtime, which the dev shell supplies; this job installs nothing.
|
||||
js:
|
||||
name: Browser JS unit tests (node --test)
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# The glob is the SCRIPT's, not the shell's and not node's. Under bash an
|
||||
# unmatched glob survives as a literal, node globs it, matches nothing and
|
||||
# exits 0 having run no test -- so this step gated nothing the moment
|
||||
# `web/tests/` moved (improvise-vz54). `web/tests/bundle/` needs a built
|
||||
# wasm bundle and runs in the browser job instead.
|
||||
- name: What the page decides between its boundaries
|
||||
run: ./scripts/node-test.sh 'web/tests/*.test.js'
|
||||
# The guard above is only worth its line if its refusals are themselves
|
||||
# checked -- the step it replaces was green while asserting nothing.
|
||||
- name: The node test runner refuses what it claims to
|
||||
run: ./scripts/checkers.sh --ci js
|
||||
|
||||
kb:
|
||||
name: Knowledge base consistency
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# kb.pl is executable knowledge, so it gets checked like code: rules
|
||||
# must reject what they should, domains must close, and functional keys
|
||||
# must not carry conflicting values. See kb_report/0 in the file.
|
||||
# trealla comes from the dev shell, as it does locally.
|
||||
- name: kb_consistent
|
||||
run: |
|
||||
tpl prolog-sessions/kb.pl \
|
||||
-g "(kb_consistent -> halt(0) ; (kb_report, halt(1)))"
|
||||
# Every suite goes through scripts/checkers.sh, which scrubs the
|
||||
# environment (improvise-f9qf); its bash is the dev shell's. Which suites
|
||||
# this job runs is the `kb` row of the CI table in checkers.sh, deliberately
|
||||
# not listed here. check-ci-graph.py holds that table to `checkers.sh --list`.
|
||||
- name: Every checker in scripts/ refuses what it claims to
|
||||
run: ./scripts/checkers.sh --ci kb
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lints the one crate the `test` job cannot reach.
|
||||
#
|
||||
# `harness/opfs-contract` carries an empty `[workspace]` table, so it is its
|
||||
# own root and `cargo clippy --workspace` at the repo root never sees it.
|
||||
#
|
||||
# Its own job rather than a step inside `browser`, for two reasons. It took
|
||||
# 37s of the browser job's 263s on run 35535225637 while gating nothing the
|
||||
# harness needs — a lint failure and a harness failure are different reports,
|
||||
# and here the lint failure wore the harness's name. And `cargo fmt --all`
|
||||
# from inside that directory formats the WHOLE outer workspace (114 files,
|
||||
# measured) rather than the one crate, because cargo-fmt's `--all` walks past
|
||||
# the nested workspace boundary — so a rustfmt violation anywhere in the repo
|
||||
# failed the *Browser harness* job. `-p opfs-contract` scopes it to 1 file.
|
||||
harness-lint:
|
||||
name: Lint the harness crate
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# checks.harness-lint in nix/ci.nix, run from inside harness/opfs-contract:
|
||||
# `cargo fmt -p opfs-contract -- --check` and clippy for wasm32.
|
||||
- name: cargo fmt -p opfs-contract and clippy --target wasm32
|
||||
run: nix build --print-build-logs .#checks.x86_64-linux.harness-lint
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The browser harness (improvise-vvl3, absorbing improvise-xf1).
|
||||
#
|
||||
# This is the only job that RUNS any of the browser half. `OpfsFs`,
|
||||
# `web/app.js` and `web/worker.js` were previously compile-checked,
|
||||
# syntax-checked and read — never executed — so every bug in them was found
|
||||
# by reading or by a person pressing keys, neither of which is repeatable.
|
||||
#
|
||||
# It can fail the build, deliberately: a harness nobody runs is worse than
|
||||
# none, because it reads as coverage.
|
||||
#
|
||||
# It now needs NO Rust toolchain: the tree it serves is downloaded from
|
||||
# `build`, which is what makes "the harness runs against the tree Pages
|
||||
# publishes" (improvise-vvl3) a fact about bytes rather than about two
|
||||
# scripts agreeing. The download is `harness/serve` whole, so there is still
|
||||
# exactly one place that arrangement is decided — `harness/build.sh`.
|
||||
browser:
|
||||
name: Browser harness (OPFS contract + shell)
|
||||
needs: build
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Install the harness
|
||||
working-directory: harness
|
||||
run: npm ci
|
||||
# No browser install step and no version probe: the dev shell sets
|
||||
# PLAYWRIGHT_BROWSERS_PATH to nixpkgs' browsers, which pin them. `npm ci`
|
||||
# stays because @playwright/test itself is not nix-provided.
|
||||
# The tree `build` assembled, not one this job rebuilds. Same bytes as
|
||||
# the Pages artifact, modulo the cache-bust token `web/assemble.sh`
|
||||
# stamps — the harness tree is stamped `harness` and the site the commit
|
||||
# SHA, which was already true when both jobs built their own.
|
||||
- name: Download the harness tree
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: harness-serve
|
||||
path: harness/serve
|
||||
# `web/pkg` and `web/worker-pkg` as wasm-pack left them, for the .d.ts
|
||||
# the step below reads. The artifact's root is `web/`, so this restores
|
||||
# both directories at the paths the test resolves relative to itself.
|
||||
- name: Download the wasm bundles
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: wasm-bundles
|
||||
path: web
|
||||
# The JS↔wasm boundary: every call web/app.js makes is declared by the
|
||||
# bundle, with that arity. It needs the bundle and nothing else, so it
|
||||
# runs here rather than in the `js` job, and before Playwright — a
|
||||
# renamed export is cheaper to read here than as a blank page.
|
||||
- name: The page's calls match the bundle it calls
|
||||
run: ./scripts/node-test.sh 'web/tests/bundle/*.test.js'
|
||||
# Chromium and Firefox only, for now. nixpkgs' WebKit (WPEWebProcess)
|
||||
# aborts on its first page inside the runner VMs, though not on the
|
||||
# host; its four specs skip on the OPFS probe anyway, so nothing gated
|
||||
# is lost. Put `--project=webkit` back once that abort is understood.
|
||||
- name: Run the harness
|
||||
working-directory: harness
|
||||
run: npx playwright test --project=chromium --project=firefox
|
||||
- if: failure()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: playwright-traces
|
||||
path: harness/test-results
|
||||
retention-days: 7
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GitHub Pages delivery (improvise-d31).
|
||||
#
|
||||
# The artifact is uploaded by `build`, which runs on every push and pull
|
||||
# request, so a change that breaks `web/build.sh` or the bundle fails on the
|
||||
# PR that made it — the build is a gate. `pages-deploy` runs only on a push
|
||||
# to main and `needs` the check jobs, so nothing that failed `cargo test`,
|
||||
# the wasm32 gate or the browser harness can reach the published site. The
|
||||
# deploy depends on those checks rather than repeating them (improvise-d31
|
||||
# AC 2).
|
||||
#
|
||||
# Not path-filtered, deliberately. The bundles are `improvise-wasm-client` and
|
||||
# `improvise-worker-server` plus, between them, most of the workspace — the
|
||||
# worker bundle is the whole App: core, formula, io, protocol and command. A
|
||||
# filter
|
||||
# that meant "skip when the bundle cannot have changed" would have to name
|
||||
# most of the workspace, and the failure mode of getting it wrong is a stale
|
||||
# site with no signal that it is stale. A redundant build is the cheaper error.
|
||||
#
|
||||
# On improvise-864a: the mechanism is NOT supersession and NOT queue depth —
|
||||
# another run's SUCCESSFUL deployment invalidates this run's `github-pages`
|
||||
# artifact, measured over 100 runs. Building once shortens the upload→deploy
|
||||
# window (182s → ~115s on the run 35535225637 shape), and the window is where
|
||||
# the whole risk lives. It narrows the race; it does not close it. The
|
||||
# evidence and what it rules out: context/code/tooling.md §3.
|
||||
# `needs` is EVERY check job, which is the only rule that does not go stale.
|
||||
# `kb` was the one absent, on the unstated ground that kb.pl is not shipped by
|
||||
# the site — but neither is `harness-lint`'s crate, and it was in the list, so
|
||||
# the ground proved too much (improvise-x7cq). A failing consistency check
|
||||
# means main is broken, and publishing from a broken main is what this list
|
||||
# exists to stop.
|
||||
pages-deploy:
|
||||
name: Deploy to GitHub Pages
|
||||
needs: [test, wasm, js, browser, harness-lint, kb, build, nix-build]
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
# A superseded deploy is allowed to finish rather than being cancelled:
|
||||
# `deploy-pages` cancelled midway can leave the Pages deployment in a
|
||||
# half-applied state, and the next push republishes anyway.
|
||||
concurrency:
|
||||
group: pages
|
||||
cancel-in-progress: false
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
# `enablement: false` (the default) configures the environment URL
|
||||
# without attempting to create or modify the Pages site via the API.
|
||||
# GitHub Pages must be enabled once in the repository settings
|
||||
# (Settings → Pages → Source: GitHub Actions) before this job will
|
||||
# produce a live URL. Using `enablement: true` fails when the workflow
|
||||
# token lacks admin permission to create the site (improvise-3gy).
|
||||
- uses: actions/configure-pages@v6
|
||||
with:
|
||||
enablement: false
|
||||
- id: deployment
|
||||
uses: actions/deploy-pages@v5
|
||||
@@ -1,50 +0,0 @@
|
||||
name: Claude Code
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
issues:
|
||||
types: [opened, assigned]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
if: |
|
||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
|
||||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
issues: read
|
||||
id-token: write
|
||||
actions: read # Required for Claude to read CI results on PRs
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Run Claude Code
|
||||
id: claude
|
||||
uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
|
||||
# This is an optional setting that allows Claude to read CI results on PRs
|
||||
additional_permissions: |
|
||||
actions: read
|
||||
|
||||
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
|
||||
# prompt: 'Update the pull request description to include a summary of changes.'
|
||||
|
||||
# Optional: Add claude_args to customize behavior and configuration
|
||||
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
|
||||
# or https://code.claude.com/docs/en/cli-reference for available options
|
||||
# claude_args: '--allowed-tools Bash(gh pr:*)'
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
name: Cloud agent smoke
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- ".claude/**"
|
||||
- ".cursor/**"
|
||||
- "scripts/agent-bootstrap.sh"
|
||||
- "scripts/nix-seed-flake-inputs.sh"
|
||||
- ".github/workflows/cloud-agent-smoke.yml"
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
name: Claude entry point + bead
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bootstrap through Claude session-start hook
|
||||
shell: bash
|
||||
env:
|
||||
CLAUDE_CODE_REMOTE: "true"
|
||||
run: |
|
||||
if ! ./.claude/hooks/session-start.sh; then
|
||||
echo "=== /tmp/nix-daemon.log ==="
|
||||
cat /tmp/nix-daemon.log 2>/dev/null || true
|
||||
echo "=== /tmp/nix-seed.log ==="
|
||||
cat /tmp/nix-seed.log 2>/dev/null || true
|
||||
echo "=== /tmp/bd-install.log ==="
|
||||
cat /tmp/bd-install.log 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
- name: Create, read back, and close a smoke-test bead
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
out="$(bd create "Cloud agent smoke: Claude entry point" --description="Ephemeral local-only bead used to verify the Claude cloud bootstrap entry point." --type=task --priority=4 --json)"
|
||||
id="$(printf '%s' "$out" | jq -r '.id // .issue.id // empty')"
|
||||
test -n "$id"
|
||||
bd show "$id"
|
||||
bd close "$id" --reason="Cloud-agent smoke test completed"
|
||||
bd show "$id" | grep -qi closed
|
||||
|
||||
cursor:
|
||||
name: Cursor install entry point + bead
|
||||
runs-on: [self-hosted, nixos, microvm]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- name: Bootstrap through Cursor install command
|
||||
shell: bash
|
||||
run: |
|
||||
nix-daemon >/tmp/nix-daemon.log 2>&1 &
|
||||
if ! ./scripts/agent-bootstrap.sh; then
|
||||
echo "=== /tmp/nix-daemon.log ==="
|
||||
cat /tmp/nix-daemon.log 2>/dev/null || true
|
||||
echo "=== /tmp/nix-seed.log ==="
|
||||
cat /tmp/nix-seed.log 2>/dev/null || true
|
||||
echo "=== /tmp/bd-install.log ==="
|
||||
cat /tmp/bd-install.log 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
- name: Create, read back, and close a smoke-test bead
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
out="$(bd create "Cloud agent smoke: Cursor entry point" --description="Ephemeral local-only bead used to verify the Cursor cloud bootstrap entry point." --type=task --priority=4 --json)"
|
||||
id="$(printf '%s' "$out" | jq -r '.id // .issue.id // empty')"
|
||||
test -n "$id"
|
||||
bd show "$id"
|
||||
bd close "$id" --reason="Cloud-agent smoke test completed"
|
||||
bd show "$id" | grep -qi closed
|
||||
@@ -1,296 +0,0 @@
|
||||
# This file was autogenerated by dist: https://axodotdev.github.io/cargo-dist
|
||||
#
|
||||
# Copyright 2022-2024, axodotdev
|
||||
# SPDX-License-Identifier: MIT or Apache-2.0
|
||||
#
|
||||
# CI that:
|
||||
#
|
||||
# * checks for a Git Tag that looks like a release
|
||||
# * builds artifacts with dist (archives, installers, hashes)
|
||||
# * uploads those artifacts to temporary workflow zip
|
||||
# * on success, uploads the artifacts to a GitHub Release
|
||||
#
|
||||
# Note that the GitHub Release will be created with a generated
|
||||
# title/body based on your changelogs.
|
||||
|
||||
name: Release
|
||||
permissions:
|
||||
"contents": "write"
|
||||
|
||||
# This task will run whenever you push a git tag that looks like a version
|
||||
# like "1.0.0", "v0.1.0-prerelease.1", "my-app/0.1.0", "releases/v1.0.0", etc.
|
||||
# Various formats will be parsed into a VERSION and an optional PACKAGE_NAME, where
|
||||
# PACKAGE_NAME must be the name of a Cargo package in your workspace, and VERSION
|
||||
# must be a Cargo-style SemVer Version (must have at least major.minor.patch).
|
||||
#
|
||||
# If PACKAGE_NAME is specified, then the announcement will be for that
|
||||
# package (erroring out if it doesn't have the given version or isn't dist-able).
|
||||
#
|
||||
# If PACKAGE_NAME isn't specified, then the announcement will be for all
|
||||
# (dist-able) packages in the workspace with that version (this mode is
|
||||
# intended for workspaces with only one dist-able package, or with all dist-able
|
||||
# packages versioned/released in lockstep).
|
||||
#
|
||||
# If you push multiple tags at once, separate instances of this workflow will
|
||||
# spin up, creating an independent announcement for each one. However, GitHub
|
||||
# will hard limit this to 3 tags per commit, as it will assume more tags is a
|
||||
# mistake.
|
||||
#
|
||||
# If there's a prerelease-style suffix to the version, then the release(s)
|
||||
# will be marked as a prerelease.
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
tags:
|
||||
- '**[0-9]+.[0-9]+.[0-9]+*'
|
||||
|
||||
jobs:
|
||||
# Run 'dist plan' (or host) to determine what tasks we need to do
|
||||
plan:
|
||||
runs-on: "ubuntu-22.04"
|
||||
outputs:
|
||||
val: ${{ steps.plan.outputs.manifest }}
|
||||
tag: ${{ !github.event.pull_request && github.ref_name || '' }}
|
||||
tag-flag: ${{ !github.event.pull_request && format('--tag={0}', github.ref_name) || '' }}
|
||||
publishing: ${{ !github.event.pull_request }}
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
submodules: recursive
|
||||
- name: Install dist
|
||||
# we specify bash to get pipefail; it guards against the `curl` command
|
||||
# failing. otherwise `sh` won't catch that `curl` returned non-0
|
||||
shell: bash
|
||||
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.4/cargo-dist-installer.sh | sh"
|
||||
- name: Cache dist
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: cargo-dist-cache
|
||||
path: ~/.cargo/bin/dist
|
||||
# sure would be cool if github gave us proper conditionals...
|
||||
# so here's a doubly-nested ternary-via-truthiness to try to provide the best possible
|
||||
# functionality based on whether this is a pull_request, and whether it's from a fork.
|
||||
# (PRs run on the *source* but secrets are usually on the *target* -- that's *good*
|
||||
# but also really annoying to build CI around when it needs secrets to work right.)
|
||||
- id: plan
|
||||
run: |
|
||||
dist ${{ (!github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name)) || 'plan' }} --output-format=json > plan-dist-manifest.json
|
||||
echo "dist ran successfully"
|
||||
cat plan-dist-manifest.json
|
||||
echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
|
||||
- name: "Upload dist-manifest.json"
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: artifacts-plan-dist-manifest
|
||||
path: plan-dist-manifest.json
|
||||
|
||||
# Build and packages all the platform-specific things
|
||||
build-local-artifacts:
|
||||
name: build-local-artifacts (${{ join(matrix.targets, ', ') }})
|
||||
# Let the initial task tell us to not run (currently very blunt)
|
||||
needs:
|
||||
- plan
|
||||
if: ${{ fromJson(needs.plan.outputs.val).ci.github.artifacts_matrix.include != null && (needs.plan.outputs.publishing == 'true' || fromJson(needs.plan.outputs.val).ci.github.pr_run_mode == 'upload') }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
# Target platforms/runners are computed by dist in create-release.
|
||||
# Each member of the matrix has the following arguments:
|
||||
#
|
||||
# - runner: the github runner
|
||||
# - dist-args: cli flags to pass to dist
|
||||
# - install-dist: expression to run to install dist on the runner
|
||||
#
|
||||
# Typically there will be:
|
||||
# - 1 "global" task that builds universal installers
|
||||
# - N "local" tasks that build each platform's binaries and platform-specific installers
|
||||
matrix: ${{ fromJson(needs.plan.outputs.val).ci.github.artifacts_matrix }}
|
||||
runs-on: ${{ matrix.runner }}
|
||||
container: ${{ matrix.container && matrix.container.image || null }}
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
BUILD_MANIFEST_NAME: target/distrib/${{ join(matrix.targets, '-') }}-dist-manifest.json
|
||||
steps:
|
||||
- name: enable windows longpaths
|
||||
run: |
|
||||
git config --global core.longpaths true
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
submodules: recursive
|
||||
- name: Install Rust non-interactively if not already installed
|
||||
if: ${{ matrix.container }}
|
||||
run: |
|
||||
if ! command -v cargo > /dev/null 2>&1; then
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
|
||||
fi
|
||||
- name: Install dist
|
||||
run: ${{ matrix.install_dist.run }}
|
||||
# Get the dist-manifest
|
||||
- name: Fetch local artifacts
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: artifacts-*
|
||||
path: target/distrib/
|
||||
merge-multiple: true
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
${{ matrix.packages_install }}
|
||||
- name: Build artifacts
|
||||
run: |
|
||||
# Actually do builds and make zips and whatnot
|
||||
dist build ${{ needs.plan.outputs.tag-flag }} --print=linkage --output-format=json ${{ matrix.dist_args }} > dist-manifest.json
|
||||
echo "dist ran successfully"
|
||||
- id: cargo-dist
|
||||
name: Post-build
|
||||
# We force bash here just because github makes it really hard to get values up
|
||||
# to "real" actions without writing to env-vars, and writing to env-vars has
|
||||
# inconsistent syntax between shell and powershell.
|
||||
shell: bash
|
||||
run: |
|
||||
# Parse out what we just built and upload it to scratch storage
|
||||
echo "paths<<EOF" >> "$GITHUB_OUTPUT"
|
||||
dist print-upload-files-from-manifest --manifest dist-manifest.json >> "$GITHUB_OUTPUT"
|
||||
echo "EOF" >> "$GITHUB_OUTPUT"
|
||||
|
||||
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
|
||||
- name: "Upload artifacts"
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: artifacts-build-local-${{ join(matrix.targets, '_') }}
|
||||
path: |
|
||||
${{ steps.cargo-dist.outputs.paths }}
|
||||
${{ env.BUILD_MANIFEST_NAME }}
|
||||
|
||||
# Build and package all the platform-agnostic(ish) things
|
||||
build-global-artifacts:
|
||||
needs:
|
||||
- plan
|
||||
- build-local-artifacts
|
||||
runs-on: "ubuntu-22.04"
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
submodules: recursive
|
||||
- name: Install cached dist
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: cargo-dist-cache
|
||||
path: ~/.cargo/bin/
|
||||
- run: chmod +x ~/.cargo/bin/dist
|
||||
# Get all the local artifacts for the global tasks to use (for e.g. checksums)
|
||||
- name: Fetch local artifacts
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: artifacts-*
|
||||
path: target/distrib/
|
||||
merge-multiple: true
|
||||
- id: cargo-dist
|
||||
shell: bash
|
||||
run: |
|
||||
dist build ${{ needs.plan.outputs.tag-flag }} --output-format=json "--artifacts=global" > dist-manifest.json
|
||||
echo "dist ran successfully"
|
||||
|
||||
# Parse out what we just built and upload it to scratch storage
|
||||
echo "paths<<EOF" >> "$GITHUB_OUTPUT"
|
||||
jq --raw-output ".upload_files[]" dist-manifest.json >> "$GITHUB_OUTPUT"
|
||||
echo "EOF" >> "$GITHUB_OUTPUT"
|
||||
|
||||
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
|
||||
- name: "Upload artifacts"
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: artifacts-build-global
|
||||
path: |
|
||||
${{ steps.cargo-dist.outputs.paths }}
|
||||
${{ env.BUILD_MANIFEST_NAME }}
|
||||
# Determines if we should publish/announce
|
||||
host:
|
||||
needs:
|
||||
- plan
|
||||
- build-local-artifacts
|
||||
- build-global-artifacts
|
||||
# Only run if we're "publishing", and only if plan, local and global didn't fail (skipped is fine)
|
||||
if: ${{ always() && needs.plan.result == 'success' && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
runs-on: "ubuntu-22.04"
|
||||
outputs:
|
||||
val: ${{ steps.host.outputs.manifest }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
submodules: recursive
|
||||
- name: Install cached dist
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
name: cargo-dist-cache
|
||||
path: ~/.cargo/bin/
|
||||
- run: chmod +x ~/.cargo/bin/dist
|
||||
# Fetch artifacts from scratch-storage
|
||||
- name: Fetch artifacts
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: artifacts-*
|
||||
path: target/distrib/
|
||||
merge-multiple: true
|
||||
- id: host
|
||||
shell: bash
|
||||
run: |
|
||||
dist host ${{ needs.plan.outputs.tag-flag }} --steps=upload --steps=release --output-format=json > dist-manifest.json
|
||||
echo "artifacts uploaded and released successfully"
|
||||
cat dist-manifest.json
|
||||
echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
|
||||
- name: "Upload dist-manifest.json"
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
# Overwrite the previous copy
|
||||
name: artifacts-dist-manifest
|
||||
path: dist-manifest.json
|
||||
# Create a GitHub Release while uploading all files to it
|
||||
- name: "Download GitHub Artifacts"
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: artifacts-*
|
||||
path: artifacts
|
||||
merge-multiple: true
|
||||
- name: Cleanup
|
||||
run: |
|
||||
# Remove the granular manifests
|
||||
rm -f artifacts/*-dist-manifest.json
|
||||
- name: Create GitHub Release
|
||||
env:
|
||||
PRERELEASE_FLAG: "${{ fromJson(steps.host.outputs.manifest).announcement_is_prerelease && '--prerelease' || '' }}"
|
||||
ANNOUNCEMENT_TITLE: "${{ fromJson(steps.host.outputs.manifest).announcement_title }}"
|
||||
ANNOUNCEMENT_BODY: "${{ fromJson(steps.host.outputs.manifest).announcement_github_body }}"
|
||||
RELEASE_COMMIT: "${{ github.sha }}"
|
||||
run: |
|
||||
# Write and read notes from a file to avoid quoting breaking things
|
||||
echo "$ANNOUNCEMENT_BODY" > $RUNNER_TEMP/notes.txt
|
||||
|
||||
gh release create "${{ needs.plan.outputs.tag }}" --target "$RELEASE_COMMIT" $PRERELEASE_FLAG --title "$ANNOUNCEMENT_TITLE" --notes-file "$RUNNER_TEMP/notes.txt" artifacts/*
|
||||
|
||||
announce:
|
||||
needs:
|
||||
- plan
|
||||
- host
|
||||
# use "always() && ..." to allow us to wait for all publish jobs while
|
||||
# still allowing individual publish jobs to skip themselves (for prereleases).
|
||||
# "host" however must run to completion, no skipping allowed!
|
||||
if: ${{ always() && needs.host.result == 'success' }}
|
||||
runs-on: "ubuntu-22.04"
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
submodules: recursive
|
||||
-61
@@ -8,64 +8,3 @@ symbols.json
|
||||
profile.json
|
||||
profile.json.gz
|
||||
bench/*.txt
|
||||
|
||||
# Added by git-smart-commit
|
||||
*.patch
|
||||
*.improv
|
||||
!examples/*.improv
|
||||
# The document the static site ships and seeds a fresh browser from
|
||||
# (improvise-bck). A shipped asset, not somebody's working file.
|
||||
!web/*.improv
|
||||
# The harness's own documents. A test fixture, not somebody's working file:
|
||||
# the grouped, multi-level model no shipped document is (improvise-plcx).
|
||||
!harness/fixtures/*.improv
|
||||
|
||||
# Beads / Dolt files (added by bd init)
|
||||
.dolt/
|
||||
*.db
|
||||
.beads-credential-key
|
||||
.sqlhistory
|
||||
backup.jsonl
|
||||
|
||||
# Added by git-smart-commit
|
||||
*.swp
|
||||
proptest-regressions/
|
||||
.cursor
|
||||
.claude/worktrees/
|
||||
*~
|
||||
/roadmap.html
|
||||
|
||||
# prolog-mcp session files (server writes ./prolog-sessions relative to cwd)
|
||||
# ...except the checked-in seed knowledge base, which agents load at session start.
|
||||
prolog-sessions/*
|
||||
!prolog-sessions/kb.pl
|
||||
|
||||
# Beads runtime locks (bd creates these next to .beads/)
|
||||
*.gate.lock*
|
||||
.beads/embeddeddolt.incoming/
|
||||
.beads/embeddeddolt.previous/
|
||||
|
||||
# GNU Global tag files (generated by the gtags devShell tooling)
|
||||
GTAGS
|
||||
GRTAGS
|
||||
GPATH
|
||||
GSYMS
|
||||
|
||||
# graphviz/dependency-graph artifacts at the repo root
|
||||
/graph.dot
|
||||
/graph.html
|
||||
/graph.svg
|
||||
.human
|
||||
|
||||
# Python bytecode from scripts/ (interpreter-version-specific, regenerated)
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# The assembled site, written by web/assemble.sh (and the doc generator's
|
||||
# default OUT_DIR). Build output: CI assembles it from source (improvise-1xfb.4).
|
||||
/site/
|
||||
|
||||
# The release web capture, written to the repo root by
|
||||
# scripts/build-web-release.sh (improvise-oujb).
|
||||
/improvise-web.tar.gz
|
||||
.opencode
|
||||
|
||||
@@ -1,703 +0,0 @@
|
||||
- Always use tests to demonstrate the existence of a bug before fixing the bug.
|
||||
- If you suspect that a bug exists, use a test to demonstrate it first:
|
||||
- prefer unit tests testing a small amount of code to integration or e2e tests
|
||||
- Option<...> or Result<...> are fine but should not be present in the majority of the code.
|
||||
- Similarly, code managing Box<...> or RC<...>, etc. for containers pointing to heap data should be split
|
||||
from logic
|
||||
- @context/repo-map.md is your "road map" for the repository. use it to reduce exploration and keep it updated.
|
||||
- @context/design-principles.md is also important for keeping the repository consistent.
|
||||
- reconcile diverging history only by merge: the merge structure of git history is a causal record (the
|
||||
user, 2026-09-25). "As a general rule, you shuold just add a new conventional commit. Small adjustments
|
||||
can ammend HEAD, but when in doubt don't." (the user). So the default is a new Conventional Commit. Amend
|
||||
only an unpushed HEAD, and only for a small fix. Never reorder, fold, squash or rebase commits, and never
|
||||
force-push. The rule, the message convention and the incidents are
|
||||
`.agents/skills/wave/agent-rules.md`, rule 8.
|
||||
- always start responses with bananaS!
|
||||
- **Knowledge base: use `scripts/kb`, not Prolog MCP.** The Prolog MCP is optional and not
|
||||
available in all agent stacks (Cursor cloud agents do not have it). All agents must use
|
||||
`scripts/kb add|replace|rm|query|check` for KB operations — see the `kb` skill at
|
||||
`.agents/skills/kb/SKILL.md`. Reading `prolog-sessions/kb.pl` directly is fine for queries.
|
||||
- save structured facts as you go — work done, interesting locations, important decisions — with
|
||||
`scripts/kb add` (see the `kb` skill). Write a fact when you learn it; there is no reason to batch.
|
||||
- query the knowledge base as needed and use Prolog reasoning when it makes sense.
|
||||
- Prolog should be used as a design tool to reason about the code and the implementation of tickets and to
|
||||
persist a formal model of it.
|
||||
|
||||
# Agent Instructions
|
||||
|
||||
This project uses **bd** (beads) for issue tracking. Run `bd prime` for full workflow context.
|
||||
|
||||
## Multi-Stack Agent Support
|
||||
|
||||
This repo supports multiple coding-agent stacks: Claude Code, Cursor (IDE and cloud), and
|
||||
potentially others. All stacks share the same operational setup:
|
||||
|
||||
| Component | Location | Notes |
|
||||
|---|---|---|
|
||||
| Toolchain | `nix develop` / `direnv` | **Required.** Rust from flake, wasm target, beads pin, trealla |
|
||||
| Skills | `.agents/skills/` | Canonical source; `.claude/skills/` and `.cursor/skills/` symlink here |
|
||||
| Session bootstrap | `scripts/agent-bootstrap.sh` | Seeds nix inputs, installs bd, syncs tracker, warms cargo |
|
||||
| Knowledge base | `scripts/kb` | **All stacks use this**, not Prolog MCP |
|
||||
| Issue tracker | `bd` CLI | Synced from Dolt remote at session start |
|
||||
|
||||
### Toolchain Requirement
|
||||
|
||||
**Agents must work inside `nix develop` (or direnv) for this repo.** Bare system cargo that
|
||||
fails `edition2024` or missing wasm target is a misconfigured session. The bootstrap seeds
|
||||
nix flake inputs automatically (GitHub tarballs are blocked in cloud), so `nix develop`
|
||||
should work after install completes.
|
||||
|
||||
Run commands inside the dev shell:
|
||||
```bash
|
||||
nix develop --command cargo test
|
||||
nix develop --command cargo build --workspace
|
||||
```
|
||||
|
||||
Or enter an interactive shell: `nix develop`
|
||||
|
||||
**Claude Code**: `.claude/hooks/session-start.sh` calls the bootstrap automatically.
|
||||
|
||||
**Cursor cloud**: `.cursor/environment.json` runs the bootstrap via `install`. Skills are
|
||||
discovered from `.cursor/skills/` (symlinks to `.agents/skills/`).
|
||||
|
||||
**Local agents**: Run `scripts/agent-bootstrap.sh` manually, or use `direnv allow` for
|
||||
automatic dev shell activation.
|
||||
|
||||
See `context/cloud-agents.md` for environment-specific notes (nix, network, etc.).
|
||||
|
||||
### GitHub Attribution (Hard Rule)
|
||||
|
||||
**Every GitHub write (PR, comment, review reply) must include an attribution line identifying
|
||||
the AI agent.** Posts go out under the repository owner's account, and an unattributed post
|
||||
reads as the human owner's words to everyone who sees it.
|
||||
|
||||
**Format:** Open the body with an italicized attribution line:
|
||||
|
||||
```
|
||||
_Posted by <Agent Stack> (<model>)_
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `_Posted by Claude (Opus 4.5)_`
|
||||
- `_Posted by Cursor Agent (claude-sonnet-5)_`
|
||||
- `_Posted by Claude Code_`
|
||||
|
||||
For PR bodies, the `🤖 Generated with …` footer that some tools add covers this. For
|
||||
comments and review replies, add the line yourself — there is no automatic footer.
|
||||
|
||||
**This applies to:** PR creation, PR updates, issue comments, review comments, and review
|
||||
replies. A thread is not "answered" until the reply carries this attribution.
|
||||
|
||||
### PR Tracking: Checks AND Reviews
|
||||
|
||||
For any PR an agent owns or is working on, **track both CI checks and GitHub comments/reviews**.
|
||||
The `pr-watch` skill (`.agents/skills/pr-watch/SKILL.md`) is the agent-facing recipe.
|
||||
|
||||
**`scripts/pr-watch`** now prints reviews after checks are green (use `--no-reviews` to skip).
|
||||
Reviews are a separate API surface from comments — Codex/ChatGPT reviews arrive as *reviews*,
|
||||
not comments, and are easy to miss if you only read the comment thread.
|
||||
|
||||
1. **Run `scripts/pr-watch <n>`** — waits for checks, then prints all reviews/comments
|
||||
2. **Read all review feedback** — reviews against an earlier head may still apply
|
||||
3. **Answer substantive feedback with attribution** before merging (see format above)
|
||||
4. **Re-check after each push** that new reviews target the current head SHA
|
||||
5. **Don't wait indefinitely on Copilot/Codex** — reviews arrive as quota allows
|
||||
|
||||
To fetch reviews directly: `./scripts/pr-reviews <n>` — **do not pipe it** (exit code trap).
|
||||
|
||||
See the `pr-watch` skill for the full recipe, and "Shipping a wave" below for Copilot
|
||||
request mechanics and how to answer declined findings.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
bd ready # Find available work
|
||||
bd show <id> # View issue details
|
||||
bd update <id> --claim # Claim work atomically
|
||||
bd close <id> # Complete work
|
||||
bd dolt push # Push beads data to remote
|
||||
```
|
||||
|
||||
## Non-Interactive Shell Commands
|
||||
|
||||
**ALWAYS use non-interactive flags** with file operations to avoid hanging on confirmation prompts.
|
||||
|
||||
Shell commands like `cp`, `mv`, and `rm` may be aliased to include `-i` (interactive) mode on some systems, causing the agent to hang indefinitely waiting for y/n input.
|
||||
|
||||
**Use these forms instead:**
|
||||
```bash
|
||||
# Force overwrite without prompting
|
||||
cp -f source dest # NOT: cp source dest
|
||||
mv -f source dest # NOT: mv source dest
|
||||
rm -f file # NOT: rm file
|
||||
|
||||
# For recursive operations
|
||||
rm -rf directory # NOT: rm -r directory
|
||||
cp -rf source dest # NOT: cp -r source dest
|
||||
```
|
||||
|
||||
**Other commands that may prompt:**
|
||||
- `scp` - use `-o BatchMode=yes` for non-interactive
|
||||
- `ssh` - use `-o BatchMode=yes` to fail instead of prompting
|
||||
- `apt-get` - use `-y` flag
|
||||
- `brew` - use `HOMEBREW_NO_AUTO_UPDATE=1` env var
|
||||
|
||||
## The Wave Workflow
|
||||
|
||||
Most work here is done by **waves**: an orchestrator chunks the open backlog,
|
||||
fans several agents out into their own git worktrees, and shepherds each
|
||||
agent's pull request through review to a merge on GitHub. This section is what
|
||||
that costs to get right. Everything in it was learned by getting it wrong first.
|
||||
|
||||
### Who owns what
|
||||
|
||||
These files are **orchestrator-only**, and the rule exists because they are
|
||||
guaranteed conflicts — every agent has a reason to touch them and they merge
|
||||
badly:
|
||||
|
||||
| File | Why |
|
||||
|---|---|
|
||||
| `.beads/` (incl. `issues.jsonl`) | bd owns it. See "Never hand-edit" below. |
|
||||
| `prolog-sessions/kb.pl` | Written by `scripts/kb` only. See `context/code/kb-tool.md`. |
|
||||
| `context/repo-map.md` | Every agent wants to add a gotcha; every wave would conflict. |
|
||||
| `context/gotchas.md`, `context/file-inventory.md` | The long forms the map digests; same conflict, same owner. |
|
||||
|
||||
Wave agents **read** all of them freely (`bd show`, `bd ready`, grep `kb.pl`) and
|
||||
**write** none of them. They report proposed repo-map lines and `kb.pl` facts as
|
||||
*text* in their hand-back, and the orchestrator applies them. Repo-map lines
|
||||
are batched at the end of the wave, after the review, because they conflict.
|
||||
`kb.pl` facts are not batched: the orchestrator writes each one when it arrives.
|
||||
**`kb.pl` is written by `scripts/kb add|replace|rm` and by nothing else** — not
|
||||
the Prolog MCP, not an editor. Every write is gated on `kb_consistent` against
|
||||
the candidate, so a bad delta leaves the file untouched, which is why a fact
|
||||
can be written the moment it is learned. There is a `kb` skill; read it before
|
||||
your first write. Reading is unaffected — `loadSession` at session start is
|
||||
still how a session gets the model in hand.
|
||||
|
||||
### Partitioning a wave
|
||||
|
||||
Partition by **crate or directory**, not by ticket count, and tell each agent
|
||||
explicitly which ground is *not* theirs. Two rules that fall out of the ones we
|
||||
broke:
|
||||
|
||||
- **Tickets that share a file belong to one agent.** The four TUI widget
|
||||
migrations all touch `src/draw.rs`; split across agents they are a guaranteed
|
||||
conflict, so they go to one agent to do sequentially. Group first, then count.
|
||||
- **An agent that must stray says so rather than guessing.** Give the boundary
|
||||
and ask for a report when it has to be crossed.
|
||||
|
||||
Two agents each adding a workspace member will still collide in
|
||||
`Cargo.toml`/`Cargo.lock`. That one is expected; keep both sides.
|
||||
|
||||
### Every wave agent's brief must include
|
||||
|
||||
The wave skill holds both halves, each in one place (improvise-1f8e):
|
||||
|
||||
- **The brief checklist** — what an orchestrator must put in a brief — is
|
||||
`.agents/skills/wave/SKILL.md`, "What every brief must carry".
|
||||
- **The agent's standing rules** — base and push, gates, claim, vacuity shapes,
|
||||
evidence, forbidden files, GitHub, hand-back, disk, commit early — are
|
||||
`.agents/skills/wave/agent-rules.md`. A brief points there rather than
|
||||
restating them.
|
||||
|
||||
`.claude/skills/wave` is a symlink to `.agents/skills/wave`, so Claude and Codex
|
||||
read the same files. A new rule goes into one of those two files, not here and
|
||||
not into the next brief: a restated rule is the copy nobody corrects, and this
|
||||
list was once a third copy that drifted from the skill's.
|
||||
|
||||
### Between waves
|
||||
|
||||
Run the **code-review skill** on the merged range and iterate until its backlog
|
||||
is empty. This is not ceremony: in this repo the review's characteristic find is
|
||||
not a wrong line, it is **a test that passes while the thing it names is
|
||||
broken** — a vacuous fold test, a bit-witness that cannot see the bit, a
|
||||
one-category fixture, a helper that fabricates the `_Measure` items the code
|
||||
under test needs, a check-the-checker that injected malformed JSON and so proved
|
||||
nothing. A green suite is not evidence. Ask the reviewer to *inject the fault*
|
||||
and watch the check fail.
|
||||
|
||||
### Shipping a wave
|
||||
|
||||
**Do not push `main` to `gh/main`.** Each agent's work reaches GitHub as its own
|
||||
branch and its own pull request, and lands on `main` by being merged there, once
|
||||
CI is green and an adversarial review agent has passed the PR's head SHA. An ADR
|
||||
PR also needs the user's explicit approval, and nothing else stands in for it:
|
||||
"ADR pull requests" below. The orchestrator does not integrate agent branches
|
||||
locally: its `main` only fast-forwards to `gh/main` after a merge. The procedure is
|
||||
`.agents/skills/wave/SKILL.md`, "Shepherding". `origin` and `gitea` are mirrors
|
||||
and take `main` directly.
|
||||
|
||||
**Because the mirrors take `main` directly, `origin/main` can run ahead of
|
||||
`gh/main`, and a branch destined for a GitHub PR must merge `gh/main` — never
|
||||
`origin/main`.** This is a trap in this repo's own topology rather than a slip:
|
||||
the two remotes are both spelled `main`, and nothing about
|
||||
`git merge origin/main` looks wrong. bd's generated Session Completion block
|
||||
once prescribed exactly that command (see "Repository policy over the Beads blocks" below).
|
||||
It cost a review round on PR #145,
|
||||
where four unmerged commits — including another PR's whole diff — were pulled
|
||||
into a branch based on `gh/main` and had to be rebuilt by cherry-picking.
|
||||
Today the repair **starts with a merge**. If the foreign commits belong to
|
||||
another open PR that will land as a merge commit, either stack on that PR (see
|
||||
the stacked-PR rule below) or wait for it to merge. Then re-pin `gh/main` and
|
||||
merge the new SHA, and the foreign commits drop out of your diff. Never revert
|
||||
them in your branch: merging the branch would then land an undo of that PR.
|
||||
|
||||
`git cherry-pick -x` is sanctioned **only** when those commits will never reach
|
||||
`main` as they are: an abandoned or rebuilt PR, or commits that exist only on a
|
||||
mirror or a local `main`, as in #148 below. Only then, cut a fresh branch from
|
||||
the pinned `gh/main` SHA and cherry-pick only your own commits. `-x` keeps the
|
||||
link to each original. Open a new PR, and close the old one without
|
||||
force-pushing it.
|
||||
|
||||
ChatGPT Codex caught the #145 base problem; nothing in CI could, because the
|
||||
result compiles and passes. The tell is a PR diff naming files the ticket has no reason to touch,
|
||||
so **read your own PR's file list before asking for review** — and diff against `gh/main...<branch>`, never the local equivalent. **The second vector is the worktree base**: a branch cut from the orchestrator's *local* `main` inherits whatever that checkout has merged but not yet pushed, and nothing in the branch says so. That hit PR #148 with another PR's whole diff — including `.beads/`, `kb.pl` and `context/repo-map.md`, the three files agents are forbidden to touch, so the review read as a rule violation rather than a base problem. Note the window
|
||||
is transient — the two remotes are equal immediately after the orchestrator
|
||||
syncs — which makes it worse, not better: the hazard is absent exactly when
|
||||
you would think to check for it.
|
||||
|
||||
This is easy to drift out of, because merging locally and pushing feels like
|
||||
finishing. It happened in this repo: 27 commits across six agent branches —
|
||||
including a new crate, a new protocol type, and a type moving between crates —
|
||||
went straight to `main` unreviewed, while `AGENTS.md` already said not to. If
|
||||
you notice a batch has bypassed the loop, the cheap repair is a **review-only
|
||||
PR**: base a branch at the last PR merge, open a PR with `main` as the head,
|
||||
have an adversarial review agent review the diff, and land the findings as an
|
||||
ordinary PR afterwards. It
|
||||
cannot be merged — the commits are already in — so close it once its findings
|
||||
are filed or fixed. Rewinding the default branch to fake a mergeable PR is not
|
||||
worth it on a repo with other collaborators.
|
||||
|
||||
**Read the PR's reviews before merging it.** This is separate from CI and from
|
||||
the adversarial agent, and it has been missed: two ChatGPT Codex reviews landed
|
||||
on merged PRs and were not read until afterwards, one of them a live P2
|
||||
(improvise-x1ns). Both were correct.
|
||||
|
||||
```bash
|
||||
./scripts/pr-reviews <n> # bodies, inline comments and issue comments, in time order; do not pipe it
|
||||
```
|
||||
|
||||
`gh pr view <n> --json reviews` is not a substitute: it shows review bodies
|
||||
only, and most of Copilot's findings are inline comments.
|
||||
|
||||
Two things make a Codex review easy to miss. It arrives as a **review**, not a
|
||||
comment, so a glance at the comment thread does not show it. And it posts under
|
||||
the repository owner's account — the same account our own agents post under —
|
||||
so **authorship does not distinguish it**. The tell is the trailing line
|
||||
`_Automated review by ChatGPT Codex (…)_`; ours open with
|
||||
`_Posted by Claude (…)_`. Check the `commit.oid` too: a review may target an
|
||||
earlier head, and the finding may already be fixed — verify rather than assume
|
||||
in either direction.
|
||||
|
||||
Reviews arrive **as the owner's quota allows**, so absence is not evidence.
|
||||
Wait where a review is plausible. How a review is answered once found, by the
|
||||
author and by a reviewer, is `.agents/skills/wave/agent-rules.md`, "GitHub".
|
||||
|
||||
**Copilot is reviewing this repository again, so read its findings and answer
|
||||
them exactly as you answer Codex's.** On 2026-09-24 it posted inline findings on
|
||||
#221, #251, #274, #295 and #298, after a lapse that began between #27 and #61
|
||||
(measured 2026-09-20: #12, #21 and #27 carry 2, 8 and 4 Copilot reviews, and
|
||||
#61, #66, #68 and #70 carry none, while both request paths reported success and
|
||||
registered nothing). Two things carry over from the lapse:
|
||||
|
||||
- **Its findings arrive mostly as inline review comments**, not as a review
|
||||
body. `scripts/pr-reviews <n>` reads bodies, inline comments and issue
|
||||
comments together; `gh pr view --json reviews` shows only the bodies, so it
|
||||
misses them. Answer each on its own thread, as agent-rules, "GitHub" says.
|
||||
- **Do not wait on it.** It reviews as its quota allows, and an absent review
|
||||
and an unrequested one still look identical from outside, which is
|
||||
`doctrine(review_is_requested_not_inherited)` seen from the requester's side.
|
||||
The review that gates a merge is still an adversarial agent the orchestrator
|
||||
dispatches, given the head SHA. Copilot's findings are read *in addition*,
|
||||
before the merge, like Codex's.
|
||||
|
||||
The request mechanics below are what worked before the lapse. Whether they work
|
||||
now is not re-measured.
|
||||
|
||||
Copilot review must be **re-requested after every fix push**; it does not
|
||||
re-review on its own. The mutation takes a **`botIds`** field — `userIds` fails
|
||||
with *"Could not resolve to User node"*:
|
||||
|
||||
```bash
|
||||
gh api graphql -f query='mutation($pr:ID!,$ids:[ID!]!){requestReviews(input:{pullRequestId:$pr,botIds:$ids,union:true}){clientMutationId}}' \
|
||||
-f pr="$(gh pr view <n> --json id -q .id)" -f ids=BOT_kgDOCnlnWA
|
||||
```
|
||||
|
||||
REST also works, with the login spelled `copilot-pull-request-reviewer[bot]` —
|
||||
without the `[bot]` suffix it 422s as "not a collaborator".
|
||||
|
||||
In a worktree-isolated agent the shell guard refuses an inline
|
||||
`gh api graphql -f query='mutation(...)'` as "too complex to verify". Put the
|
||||
query in a file and pass `-F query=@file`; PR comment bodies likewise go
|
||||
through `-F body=@file` rather than command substitution.
|
||||
|
||||
**Anything you post to GitHub goes out under the repository owner's account, so
|
||||
say who is speaking.** Open every PR comment, review and issue body with a line
|
||||
naming the model and that it is an agent posting in his session. A PR body's
|
||||
`🤖 Generated with Claude Code` footer covers that case; comments have nothing
|
||||
unless you add it, and an unattributed one reads as his own words to everyone
|
||||
who sees it.
|
||||
|
||||
Two environment notes. **`gh` is not on `PATH` here** — the name is
|
||||
shell-aliased to `git-hub`, a different tool — so use
|
||||
`nix shell nixpkgs#gh --command gh …`. And **a Copilot review takes about six
|
||||
minutes**; tool calls return fast, which makes elapsed time easy to misjudge, so
|
||||
check `date -u` against the GitHub timeline before concluding a review is stuck.
|
||||
(One agent closed and reopened a PR to "unstick" a review that was proceeding
|
||||
normally.)
|
||||
|
||||
Across four PRs Copilot has found genuine bugs and no false positives, so its
|
||||
findings are worth the loop. **But judge each one against the repo's doctrine
|
||||
rather than applying it reflexively**. How a declined finding is answered is
|
||||
agent-rules, "GitHub". The distinction that has mattered: a finding can be *correct*
|
||||
and still be wrong to fix in that PR. On #12, a non-ASCII `truncate` panic was
|
||||
applied (byte-identical on ASCII, so it cost no evidence) while a hint-line
|
||||
padding bug in the same file was declined — fixing it would have rewritten the
|
||||
exact-frame characterization tests that were the PR's only evidence the
|
||||
migration changed nothing. Two right findings, different costs. File the
|
||||
declined one.
|
||||
|
||||
**Under merge-as-soon-as-green, a review round can land after its own merge.**
|
||||
GitHub deletes the branch when a PR merges, so a fix push to the old branch name
|
||||
**silently succeeds and accomplishes nothing** — it re-creates a branch `main`
|
||||
will never see, CI runs on it, and the commit is nowhere. The tell is the push
|
||||
printing `* [new branch]` where you expected an `old..new` range. It happened on
|
||||
#51: the third review round completed four minutes after the merge, and the
|
||||
correction it prompted went to a re-created `wave/b-projection-gaps`. The repair
|
||||
is a merge, not a cherry-pick. The PR merged with a merge commit, so the stray
|
||||
branch's base is already an ancestor of `main`. Branch from the pinned
|
||||
`gh/main` SHA, `git merge` the stray branch's SHA, open a follow-up PR, and
|
||||
delete the stray re-creation. The merge brings exactly the fix commits, under
|
||||
their own SHAs, with the link to the PR they answered. A cherry-pick would copy
|
||||
them and drop that link. This recipe works only because PRs here never
|
||||
squash-merge. Check `gh pr view <n> --json state` *before* pushing a fix to
|
||||
a PR you have not looked at in a few minutes, rather than after.
|
||||
|
||||
**A stacked PR whose base PR has already merged merges into that stale base
|
||||
branch, not into `main`.** GitHub merges a PR into whatever `baseRefName` says,
|
||||
and it did not retarget these. On 2026-09-24 fifteen approved stacked PRs
|
||||
(#255–#286) were merged into a base whose own PR had **already** merged, and
|
||||
the merges read as done. Recovery took seven landing PRs (#289–#295), each
|
||||
opened from a stack's tip against `main`, and their titles name exactly those
|
||||
fifteen. Four more in that range (#259, #263, #272, #273) also merged into a
|
||||
non-`main` base, but one whose PR was still open; they reached `main` when their
|
||||
base did, which is stacking working as intended. The repository reports `delete_branch_on_merge: true` as measured on
|
||||
2026-09-24, and the stale bases (`collapse-fu/h6z1`, `fix/enz6-advance-onto-fold`
|
||||
and others) were still on GitHub afterwards, so that setting is not the guard.
|
||||
The guard is a check before every merge:
|
||||
|
||||
```bash
|
||||
gh pr view <n> --json baseRefName -q .baseRefName # must print main
|
||||
gh pr edit <n> --base main # if not, once its base PR has landed
|
||||
```
|
||||
|
||||
Then read the PR's diff again: a retargeted PR's diff now includes its base's
|
||||
commits if the base did not land.
|
||||
|
||||
Merge only once a review of the PR's current head has passed — an adversarial
|
||||
agent's — and every Codex and Copilot finding on the PR has an answer. What a
|
||||
review must state is agent-rules, "GitHub". Merge with a **merge commit**. Never
|
||||
use GitHub's squash or rebase merge: history is reconciled only by merge (the
|
||||
rule at the top of this file). A PR's commits are individually meaningful, and
|
||||
the fix-after-merge recipe above needs the PR's head to be an ancestor of
|
||||
`main`.
|
||||
|
||||
### Merging many PRs at once
|
||||
|
||||
Three things went wrong when a day's approved PRs landed together, and each is
|
||||
invisible to CI on either side alone.
|
||||
|
||||
**Land a batch as one integration PR, not one PR at a time.** Every PR appends
|
||||
its bug evidence at the end of a `context/test-evidence/*.md` file. So each
|
||||
merge makes every other open PR conflict at EOF, and merging one at a time
|
||||
means re-resolving the same files once per PR — and a PR that conflicts gets no
|
||||
CI run at all (agent-rules, "Gates"). On 2026-09-24 the approved PRs landed as
|
||||
two integration PRs instead: #298 (nineteen PRs) and #299 (the browser-editor
|
||||
stack). This is what "one PR per wave" above already intends.
|
||||
|
||||
**After resolving a conflict, diff the result against each parent and against
|
||||
the merge base.** `git checkout --ours <file>` or `--theirs` takes the whole
|
||||
file from one side, so it silently drops every *non*-conflicting change the
|
||||
other side made to that file. #245 resolved a `ci.yml` conflict with #228 by
|
||||
`git checkout --theirs`, took #228's whole file, and lost its own
|
||||
`check-harness-lock --allow-network` step, while the merge message said the step
|
||||
was unchanged (improvise-dndu). Nothing refused it: `check-ci-graph.py` counts
|
||||
suites, not that step. Resolve by hand, then check the merge commit:
|
||||
|
||||
```bash
|
||||
git diff <merge-base> HEAD -- <file> # both sides' changes together, and nothing else
|
||||
git diff HEAD^1 HEAD -- <file> # what the second side brought
|
||||
git diff HEAD^2 HEAD -- <file> # what the first side brought
|
||||
```
|
||||
|
||||
**Two changes to one test file can merge cleanly and still fail together.** #287
|
||||
(the NixOS runners) found that `/usr/bin:/bin` there holds only `env` and `sh`,
|
||||
and gave section 5 of `checkers-test.sh` a hermetic `utilbin` PATH. #237 added
|
||||
section 5b on a branch cut before that, still using `PATH=…:/usr/bin:/bin`. Each
|
||||
passed CI alone, the merge was textually clean, and 5b failed on every PR after
|
||||
it (#296). So a fix to an idiom — a hermetic PATH, a scrubbed environment, a
|
||||
helper every case must call — has to be applied to **every case in the file**,
|
||||
including cases that exist only on another open branch. When you merge, grep
|
||||
the merged file for the old idiom. Then run the gates on the merged tree, not
|
||||
on each side.
|
||||
|
||||
### CI takes its tools from the dev shell
|
||||
|
||||
These are the user's rulings of 2026-09-24:
|
||||
|
||||
- **CI runs on self-hosted NixOS microVM runners** (`runs-on: [self-hosted,
|
||||
nixos, microvm]`). Every `run:` step runs in `nix develop .#ci`, through the
|
||||
workflow's `defaults.run.shell`. A `uses:` step (an action such as
|
||||
`actions/checkout`) is not a shell step and does not go through it. A missing tool goes into the `.#ci` shell in
|
||||
`flake.nix`, not into a per-step `nix shell` wrapper.
|
||||
- **No hard-coded `/bin` or `/usr/bin` paths.** On these runners those
|
||||
directories hold only `env` and `sh`. Take tools from `PATH`.
|
||||
- **No version floors and no probes.** Scripts and tests must not care which
|
||||
bash or standard utilities are present. `checkers.sh` no longer pins a bash
|
||||
(it did, at 5.1, until #297). A script that needs a newer feature fails where
|
||||
it uses it. The argument is `context/code/tooling.md` §5.
|
||||
- **Playwright's browsers come from nix.** The dev shell sets
|
||||
`PLAYWRIGHT_BROWSERS_PATH` to nixpkgs' browsers, so there is no install step
|
||||
and no version check. `npm ci` stays, because `@playwright/test` is not
|
||||
nix-provided.
|
||||
|
||||
A test that fakes an environment must build it on purpose, for example a
|
||||
hermetic `utilbin` of linked tools. It must not assume the runner's `/bin`
|
||||
holds anything.
|
||||
|
||||
### ADR pull requests
|
||||
|
||||
**Only the user's explicit approval accepts an ADR (`context/adr/`), and only
|
||||
it lets an ADR PR merge.** "An ADR isn't approved until I say it is" (the
|
||||
user). Green CI, every review answered and a passing adversarial review
|
||||
("Shipping a wave") are still required. They are gates, never approval: an ADR
|
||||
PR that passes all of them is still unapproved and does not merge. The steps,
|
||||
modelled on PR #122 (ADR 004):
|
||||
|
||||
1. The PR proposes the ADR with its **open questions numbered**.
|
||||
2. The user answers them on the PR.
|
||||
3. The answers are folded into the ADR, in a section headed **"What the user
|
||||
decided"**.
|
||||
4. The PR merges only after the user comments **"Approved"**, and only while
|
||||
the gates above still hold. Nobody else's comment or review stands in for
|
||||
it. "Read the PR's reviews before merging it" still
|
||||
applies after the approval. On #122 the user's own review landed as the PR
|
||||
merged.
|
||||
|
||||
**An open ADR PR is watched, not visited.** The user answers the numbered
|
||||
questions inline, and reviews arrive as they arrive. An answer left unread for
|
||||
hours is a failure (the user, 2026-09-24). So while an ADR PR is open the
|
||||
orchestrator keeps a background poll for new non-agent comments and reviews
|
||||
since its last pass, and resumes the steward agent on each new item. The
|
||||
steward folds each answer in and replies on its thread.
|
||||
|
||||
### Never hand-edit `.beads/issues.jsonl`
|
||||
|
||||
bd manipulates that file and auto-exports to it. Editing it — or even
|
||||
`git checkout`-ing it, which fires the post-checkout hook — breaks auto-export
|
||||
and desynchronises `export-state.json`. Use `bd` commands, let auto-export run,
|
||||
and commit whatever it leaves.
|
||||
|
||||
### Cite a ticket ID you have read back
|
||||
|
||||
`bd create` assigns the ID; it is not derivable from the title. An orchestrator
|
||||
that files with output suppressed and then writes the ticket into a gotcha or an
|
||||
ADR **will invent one** — six invented IDs survived into `context/` in a single
|
||||
session before an agent tried one and found nothing. Read the ID back after
|
||||
filing, and before pushing docs run the audit:
|
||||
|
||||
```bash
|
||||
./scripts/bd-audit-ids
|
||||
```
|
||||
|
||||
It exits 0, printing `all N cited ids resolve`, only when every cited id is in
|
||||
what `bd list --all` returns. A dead id is exit 1, and each one is named. A bd
|
||||
that is missing, fails, or lists no issues is exit 3. The comparison is
|
||||
positive because the recipe the script replaced was not: it searched `bd show`
|
||||
for "not found" lines and printed "all resolve" when there were none, so a bd
|
||||
that never ran read as a clean audit (improvise-52eq). Its suite,
|
||||
`scripts/bd-audit-ids-test.sh`, runs in `checkers.sh` and checks both halves:
|
||||
a dead id, and a bd that does not answer.
|
||||
|
||||
In the script, `git ls-files` picks the files, and grep only reads them. The file list is the
|
||||
tracked files plus the untracked ones, minus anything git ignores. That leaves
|
||||
out two things a `grep -r` over the same roots reads:
|
||||
|
||||
- **Build output.** After `./harness/build.sh`, `harness/serve/` holds
|
||||
rendered copies of `context/`, and `grep -r` reported three dead ids from them
|
||||
that no tracked file contains (improvise-8haj).
|
||||
- **Other agents' worktrees.** In the orchestrator's checkout
|
||||
`.claude/worktrees/` holds every agent's worktree, dozens of copies of the
|
||||
repository with their unpushed work.
|
||||
|
||||
`-d skip` is there for the skills' symlinks, and it changes no answer. The
|
||||
`.claude/skills/*` entries are symlinks to directories under `.agents/skills/`,
|
||||
and `git ls-files` lists each one, so each reaches grep as a command-line
|
||||
operand. The grep has no `-r`, so it never reads a directory operand. It prints
|
||||
`grep: <path>: Is a directory` on stderr instead, and `-d skip` only silences
|
||||
those lines. The output goes through `sort -u`, so a duplicate could not change
|
||||
the answer anyway. Measured with BSD grep 2.6.0 and GNU grep 3.12: 3092 matches
|
||||
with or without the flag, 462 distinct ids. Do not add `-r`: GNU grep then
|
||||
follows those symlinks and reads the skills a second time (3107 matches).
|
||||
|
||||
Plain `git grep` is not a substitute: on macOS its `-E` silently ignores `\b`
|
||||
and finds a fraction of the ids.
|
||||
|
||||
Three things the earlier version of this recipe got wrong, each of which let a
|
||||
real defect through:
|
||||
|
||||
- **It scanned two paths** (`context/repo-map.md context/adr/*.md`), so it
|
||||
found one dead id in the repo map and missed both copies of it in
|
||||
`harness/tests/shell.spec.js`. Ticket IDs are cited in code comments,
|
||||
in `README.md` and in `web/`, not only in `context/`.
|
||||
- **`[a-z0-9]{3,4}` cannot see a commit SHA wearing the prefix.** A seven-hex
|
||||
commit sha cited as `improvise-<sha>` survived in
|
||||
`crates/improvise-core/src/workbook.rs` precisely because it was too long
|
||||
to match. An ID-shaped non-ID is worse than a dead ID, because the
|
||||
diagnosis is identical and the repair is not.
|
||||
- **The exclusion list named only `core` and `wasm`**, so every other crate
|
||||
name would have been reported missing. The script's exclusions also cover the
|
||||
non-ticket tokens this repo really contains: a package name, a temp file
|
||||
name, the prose word "improvise-side", and the example crate name in the
|
||||
`kb` skill's `crate(...)` fact. **They are anchored `^…$` for the
|
||||
same reason the recipe is read as code**: unanchored, the `gp` alternative is
|
||||
a *prefix* match, so a future real ticket whose id happens to begin `gp` is
|
||||
silently excluded and the audit reports nothing while skipping it. An
|
||||
exclusion list is a filter, and a filter that over-matches fails in the
|
||||
direction that looks like success. (Anchoring it is also what caught the
|
||||
example id this very paragraph originally spelled out — see the note below.)
|
||||
- **A release artifact's name is ID-shaped, and `improvise-installer.sh`
|
||||
escaped only by being ten characters long.** `improvise-web.tar.gz`
|
||||
(improvise-oujb) is three, so it matches `{3,8}` and resolves to nothing;
|
||||
`web` is therefore excluded in the script. The general point is that the length
|
||||
bound, not any rule, is what has been keeping `dist`'s artifact names out of
|
||||
this audit — so **a new artifact whose name is a short lowercase word must be
|
||||
added to the exclusion list in the same commit**, or it arrives as a dead id
|
||||
that no amount of `bd` archaeology explains.
|
||||
|
||||
The recipe's own prose spells no id as an *example*, because a document that
|
||||
lists example ids makes itself a source of audit noise. The ids it does cite,
|
||||
improvise-8haj and improvise-oujb, are the tickets behind two of the rules
|
||||
above. They resolve, and the audit checks them like any other.
|
||||
|
||||
One environment note: run the audit by its name. Which `bd` command forms a
|
||||
worktree-isolated agent's shell guard allows has varied between sessions. In
|
||||
one session it refused the old recipe's `bd show $(cat …)`, and also
|
||||
`xargs bd show < file` and `sh file`. In another, both `xargs` forms ran.
|
||||
`./scripts/bd-audit-ids` passes `bd` no argument computed at runtime, and it
|
||||
ran in the session that refused the other three.
|
||||
|
||||
A dead ID is worse than no ID: it reads as a promise that the follow-up exists.
|
||||
|
||||
### Before filing a ticket, search
|
||||
|
||||
Agent-filed issues duplicate work that is already open, sometimes already
|
||||
in progress. Two duplicates were filed in this repo within minutes of the
|
||||
hand-backs that prompted them. `bd search` the core concept *and* scan
|
||||
in-progress issues first.
|
||||
|
||||
## Repository policy over the Beads blocks
|
||||
|
||||
Everything between a `BEGIN BEADS …` marker and its `END` is generated by bd,
|
||||
and bd replaces it whole. `bd init` and `bd setup factory|opencode|mux` rewrite
|
||||
the integration block, and the last three call it "stale" and invite the
|
||||
rewrite whenever bd's template has moved on. `bd setup codex` rewrites the
|
||||
Codex block. So a hand edit inside the markers is silently reverted by the next
|
||||
regeneration. Repository policy lives here, outside them:
|
||||
|
||||
- **Work is not done until it is in a pull request and that pull request has
|
||||
merged.** A pushed branch is not done, and neither is a green one. In a wave,
|
||||
the merge is the orchestrator's, after an adversarial review passes the head
|
||||
SHA ("Shipping a wave"). An ADR PR merges only on the user's explicit
|
||||
approval as well ("ADR pull requests"). So a wave agent's part ends at an
|
||||
open PR plus its hand-back, and the ticket is done, and its bead closed, only
|
||||
when the merge lands.
|
||||
- **This repository does not opt in to the Beads "team-maintainer" profile.**
|
||||
So the pull (`git pull`, or `git pull --rebase` in bd's "full" profile) and
|
||||
the bare `git push` that the block lists for that profile are not
|
||||
instructions here. Work reaches `main` as "Shipping a wave" describes,
|
||||
and a wave agent pushes with the explicit refspec in
|
||||
`.agents/skills/wave/agent-rules.md`, "Base, branch, push".
|
||||
- **The no-markdown-TODO and no-`MEMORY.md` rules have one exception:**
|
||||
`context/design-principles.md` and `context/repo-map.md` are kept by hand
|
||||
(the repo map by the orchestrator only; see "Who owns what").
|
||||
|
||||
<!-- BEGIN BEADS INTEGRATION v:1 profile:minimal hash:46cd31e7 -->
|
||||
## Beads Issue Tracker
|
||||
|
||||
This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands.
|
||||
|
||||
### Quick Reference
|
||||
|
||||
```bash
|
||||
bd ready # Find available work
|
||||
bd show <id> # View issue details
|
||||
bd update <id> --claim # Claim work
|
||||
bd close <id> # Complete work
|
||||
```
|
||||
|
||||
### Rules
|
||||
|
||||
- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
|
||||
- Run `bd prime` for detailed command reference and session close protocol
|
||||
- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files
|
||||
|
||||
**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns.
|
||||
|
||||
## Agent Context Profiles
|
||||
|
||||
The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions.
|
||||
|
||||
- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands.
|
||||
- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise.
|
||||
- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins.
|
||||
|
||||
## Session Completion
|
||||
|
||||
This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions.
|
||||
|
||||
1. **File issues for remaining work** - Create beads for anything that needs follow-up
|
||||
2. **Run quality gates** (if code changed) - Tests, linters, builds
|
||||
3. **Update issue status** - Close finished work, update in-progress items
|
||||
4. **Handle git/sync by active profile**:
|
||||
```bash
|
||||
# Conservative/minimal/default: report status and proposed commands; wait for approval.
|
||||
git status
|
||||
|
||||
# Team-maintainer opt-in only, unless current instructions forbid it:
|
||||
git pull
|
||||
bd dolt push
|
||||
git push
|
||||
git status
|
||||
```
|
||||
5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step
|
||||
|
||||
**Critical rules:**
|
||||
- Explicit user or orchestrator instructions override this Beads block.
|
||||
- Do not commit or push without clear authority from the active profile or the current user request.
|
||||
- If a required sync or push is blocked, stop and report the exact command and error.
|
||||
<!-- END BEADS INTEGRATION -->
|
||||
|
||||
<!-- BEGIN BEADS CODEX SETUP: generated by bd setup codex -->
|
||||
## Beads Issue Tracker
|
||||
|
||||
Use Beads (`bd`) for durable task tracking in repositories that include it. Use the `beads` skill at `.agents/skills/beads/SKILL.md` (project install) or `~/.agents/skills/beads/SKILL.md` (global install) for Beads workflow guidance, then use the `bd` CLI for issue operations.
|
||||
|
||||
### Quick Reference
|
||||
|
||||
```bash
|
||||
bd ready # Find available work
|
||||
bd show <id> # View issue details
|
||||
bd update <id> --claim # Claim work
|
||||
bd close <id> # Complete work
|
||||
bd prime # Refresh Beads context
|
||||
```
|
||||
|
||||
### Rules
|
||||
|
||||
- Use `bd` for all task tracking; do not create markdown TODO lists.
|
||||
- Run `bd prime` when Beads context is missing or stale. Codex 0.129.0+ can load Beads context automatically through native hooks; use `/hooks` to inspect or toggle them.
|
||||
- Keep persistent project memory in Beads via `bd remember`; do not create ad hoc memory files.
|
||||
|
||||
**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns.
|
||||
<!-- END BEADS CODEX SETUP -->
|
||||
@@ -1,121 +1,6 @@
|
||||
- see @AGENTS.md
|
||||
|
||||
Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.
|
||||
|
||||
**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
|
||||
|
||||
## 1. Think Before Coding
|
||||
|
||||
**Don't assume. Don't hide confusion. Surface tradeoffs.**
|
||||
|
||||
Before implementing:
|
||||
- State your assumptions explicitly. If uncertain, ask.
|
||||
- If multiple interpretations exist, present them - don't pick silently.
|
||||
- If a simpler approach exists, say so. Push back when warranted.
|
||||
- If something is unclear, stop. Name what's confusing. Ask.
|
||||
|
||||
## 2. Simplicity First
|
||||
|
||||
**Minimum code that solves the problem. Nothing speculative.**
|
||||
|
||||
- No features beyond what was asked.
|
||||
- No abstractions for single-use code.
|
||||
- No "flexibility" or "configurability" that wasn't requested.
|
||||
- No error handling for impossible scenarios.
|
||||
- If you write 200 lines and it could be 50, rewrite it.
|
||||
|
||||
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
|
||||
|
||||
## 3. Surgical Changes
|
||||
|
||||
**Touch only what you must. Clean up only your own mess.**
|
||||
|
||||
When editing existing code:
|
||||
- Don't "improve" adjacent code, comments, or formatting.
|
||||
- Don't refactor things that aren't broken.
|
||||
- Match existing style, even if you'd do it differently.
|
||||
- If you notice unrelated dead code, mention it - don't delete it.
|
||||
|
||||
When your changes create orphans:
|
||||
- Remove imports/variables/functions that YOUR changes made unused.
|
||||
- Don't remove pre-existing dead code unless asked.
|
||||
|
||||
The test: Every changed line should trace directly to the user's request.
|
||||
|
||||
## 4. Goal-Driven Execution
|
||||
|
||||
**Define success criteria. Loop until verified.**
|
||||
|
||||
Transform tasks into verifiable goals:
|
||||
- "Add validation" → "Write tests for invalid inputs, then make them pass"
|
||||
- "Fix the bug" → "Write a test that reproduces it, then make it pass"
|
||||
- "Refactor X" → "Ensure tests pass before and after"
|
||||
|
||||
For multi-step tasks, state a brief plan:
|
||||
```
|
||||
1. [Step] → verify: [check]
|
||||
2. [Step] → verify: [check]
|
||||
3. [Step] → verify: [check]
|
||||
```
|
||||
|
||||
Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
|
||||
|
||||
---
|
||||
|
||||
**These guidelines are working if:** fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.
|
||||
|
||||
<!-- BEGIN BEADS INTEGRATION v:1 profile:minimal hash:46cd31e7 -->
|
||||
## Beads Issue Tracker
|
||||
|
||||
This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands.
|
||||
|
||||
### Quick Reference
|
||||
|
||||
```bash
|
||||
bd ready # Find available work
|
||||
bd show <id> # View issue details
|
||||
bd update <id> --claim # Claim work
|
||||
bd close <id> # Complete work
|
||||
```
|
||||
|
||||
### Rules
|
||||
|
||||
- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists
|
||||
- Run `bd prime` for detailed command reference and session close protocol
|
||||
- Use `bd remember` for persistent knowledge — do NOT use MEMORY.md files
|
||||
|
||||
**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns.
|
||||
|
||||
## Agent Context Profiles
|
||||
|
||||
The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions.
|
||||
|
||||
- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands.
|
||||
- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise.
|
||||
- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins.
|
||||
|
||||
## Session Completion
|
||||
|
||||
This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions.
|
||||
|
||||
1. **File issues for remaining work** - Create beads for anything that needs follow-up
|
||||
2. **Run quality gates** (if code changed) - Tests, linters, builds
|
||||
3. **Update issue status** - Close finished work, update in-progress items
|
||||
4. **Handle git/sync by active profile**:
|
||||
```bash
|
||||
# Conservative/minimal/default: report status and proposed commands; wait for approval.
|
||||
git status
|
||||
|
||||
# Team-maintainer opt-in only, unless current instructions forbid it:
|
||||
git pull
|
||||
bd dolt push
|
||||
git push
|
||||
git status
|
||||
```
|
||||
5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step
|
||||
|
||||
**Critical rules:**
|
||||
- Explicit user or orchestrator instructions override this Beads block.
|
||||
- Do not commit or push without clear authority from the active profile or the current user request.
|
||||
- If a required sync or push is blocked, stop and report the exact command and error.
|
||||
<!-- END BEADS INTEGRATION -->
|
||||
- Always use tests to demonstrate the existence of a bug before fixing the bug.
|
||||
- If you suspect that a bug exists, use a test to demonstrate it first:
|
||||
- prefer unit tests testing a small amount of code to integration or e2e tests
|
||||
- Option<...> or Result<...> are fine but should not be present in the majority of the code.
|
||||
- Similarly, code managing Box<...> or RC<...>, etc. for containers pointing to heap data should be split
|
||||
from logic
|
||||
|
||||
Generated
+507
-1702
File diff suppressed because it is too large
Load Diff
+12
-64
@@ -1,78 +1,30 @@
|
||||
[workspace]
|
||||
members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp", "crates/numtower", "crates/zset", "crates/improvise-incremental", "crates/zset-ffi"]
|
||||
|
||||
[package]
|
||||
name = "improvise"
|
||||
version = "0.1.0-rc2"
|
||||
edition = "2024"
|
||||
description = "Terminal pivot-table modeling in the spirit of Lotus Improv"
|
||||
license = "Apache-2.0"
|
||||
repository = "https://github.com/fiddlerwoaroof/improvise"
|
||||
homepage = "https://github.com/fiddlerwoaroof/improvise"
|
||||
readme = "README.md"
|
||||
keywords = ["tui", "pivot", "spreadsheet", "data", "improv"]
|
||||
categories = ["command-line-utilities", "visualization"]
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "Multi-dimensional data modeling terminal application"
|
||||
license = "MIT"
|
||||
|
||||
[[bin]]
|
||||
name = "improvise"
|
||||
path = "src/main.rs"
|
||||
|
||||
# improvise-i5tk's acceptance criterion: whether the TUI's whole-grid GridCache
|
||||
# is worth narrowing to a viewport is a measurement, not an argument.
|
||||
# `test = false`: it takes seconds by design, and the suite's rule is that no
|
||||
# single test binary exceeds two of them.
|
||||
[[bench]]
|
||||
name = "grid_projection"
|
||||
harness = false
|
||||
test = false
|
||||
|
||||
[[bench]]
|
||||
name = "reshape_observation"
|
||||
harness = false
|
||||
test = false
|
||||
|
||||
[[bench]]
|
||||
name = "scale_bisect"
|
||||
harness = false
|
||||
test = false
|
||||
|
||||
[dependencies]
|
||||
gp-tree = { path = "crates/gp-tree" }
|
||||
improvise-core = { path = "crates/improvise-core" }
|
||||
improvise-formula = { path = "crates/improvise-formula" }
|
||||
improvise-io = { path = "crates/improvise-io" }
|
||||
improvise-command = { path = "crates/improvise-command" }
|
||||
ratatui = "0.30"
|
||||
crossterm = "0.29"
|
||||
ratatui = "0.29"
|
||||
crossterm = "0.28"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
# Parses JSON a user's numbers cross, so it must read back the f64 written: per crate,
|
||||
# because each build stands alone (improvise-6yvg, context/code/protocol.md).
|
||||
serde_json = { version = "1", features = ["float_roundtrip"] }
|
||||
serde_json = "1"
|
||||
anyhow = "1"
|
||||
thiserror = "2"
|
||||
indexmap = "2"
|
||||
thiserror = "1"
|
||||
indexmap = { version = "2", features = ["serde"] }
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
flate2 = "1"
|
||||
unicode-width = "^0.2"
|
||||
unicode-width = "0.2"
|
||||
dirs = "5"
|
||||
csv = "1"
|
||||
clap = { version = "4.6.7", features = ["derive"] }
|
||||
enum_dispatch = "0.3.13"
|
||||
clap = { version = "4.6.0", features = ["derive"] }
|
||||
|
||||
[dev-dependencies]
|
||||
# Only `tests/grid_frame_agreement.rs` needs it: that test drives the TUI
|
||||
# widget and the browser client over ONE `GridCache` and asserts they draw
|
||||
# the same grid frame (improvise-anr9). A dev-dependency, so the terminal
|
||||
# binary does not grow a wasm client; `improvise-wasm-client` builds
|
||||
# natively, which is how its own tests already run.
|
||||
improvise-wasm-client = { path = "crates/improvise-wasm-client" }
|
||||
# `test-support` gives `FormulaEntry: From<&str>`, which the tests use; asked for
|
||||
# here, or `-p improvise` alone does not compile (improvise-a61f).
|
||||
improvise-protocol = { path = "crates/improvise-protocol", features = ["test-support"] }
|
||||
# `tests/headless_cli.rs` detaches the binary from any terminal with `setsid`.
|
||||
libc = "0.2"
|
||||
pest = "2.9.3"
|
||||
pest_derive = "2.9.3"
|
||||
pest_meta = "2.9.3"
|
||||
proptest = "1"
|
||||
tempfile = "3"
|
||||
|
||||
@@ -86,7 +38,3 @@ strip = true
|
||||
inherits = "release"
|
||||
strip = false
|
||||
debug = 2
|
||||
|
||||
# The profile that 'dist' will build with
|
||||
[profile.dist]
|
||||
inherits = "release"
|
||||
|
||||
@@ -1,202 +0,0 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -1,513 +0,0 @@
|
||||
# improvise
|
||||
|
||||
*Terminal pivot-table modeling in the spirit of Lotus Improv — multidimensional cells, formulas over dimensions instead of cell addresses, and vim-style keybindings for reassigning axes on the fly.*
|
||||
|
||||
|
||||
|
||||
## Why this exists
|
||||
|
||||
Lotus Improv (NeXT, 1991) separated data from its presentation: cells were
|
||||
addressed by named dimensions, not grid coordinates, and rearranging a view
|
||||
didn't break formulas. That idea never made it into mainstream tools. Excel
|
||||
pivot tables borrowed the visual rearrangement but kept cell-address formulas.
|
||||
Terminal data tools like sc-im and VisiData do different things well — sc-im is
|
||||
a traditional spreadsheet, VisiData is a data explorer — but neither offers the
|
||||
dimension-keyed data model that made Improv interesting. improvise is a small
|
||||
attempt to bring that model to a modern terminal, with a formula language that
|
||||
references category and item names, views that can be rearranged with a single
|
||||
keystroke, and a plain-text file format you can diff in git.
|
||||
|
||||
See [docs/design-notes.md](docs/design-notes.md) for the original product
|
||||
vision and non-goals.
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
nix build .
|
||||
./result/bin/improvise examples/demo.improv
|
||||
```
|
||||
|
||||
Or import your own CSV:
|
||||
|
||||
```sh
|
||||
./result/bin/improvise import path/to/data.csv
|
||||
```
|
||||
|
||||
The included `examples/demo.improv` was generated from `examples/demo.csv`:
|
||||
|
||||
```sh
|
||||
improvise import examples/demo.csv \
|
||||
--no-wizard \
|
||||
--category Region --category Product --category Customer \
|
||||
--measure Revenue --measure Cost \
|
||||
--time Date --extract Date:Month --split-date Date \
|
||||
--axis Region:row --axis Product:row \
|
||||
--axis Date_Month:column --axis _Measure:column \
|
||||
--axis Customer:page --axis Date:none \
|
||||
--formula "Profit = Revenue - Cost" \
|
||||
--name "Acme Sales Demo" \
|
||||
-o examples/demo.improv
|
||||
```
|
||||
|
||||
## Key bindings to try first
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| `T` | Enter tile mode — reassign category axes |
|
||||
| `[` / `]` | Cycle through page-axis items |
|
||||
| `>` | Drill into an aggregated cell |
|
||||
| `<` | Return from drill-down |
|
||||
| `F` | Open the formula panel |
|
||||
| `t` | Transpose rows and columns |
|
||||
| `?` / `F1` | Full key reference |
|
||||
| `:w` | Save |
|
||||
| `:q` | Quit |
|
||||
|
||||
## Installation
|
||||
|
||||
### With Nix (preferred)
|
||||
|
||||
```sh
|
||||
nix build .
|
||||
# or install into your profile:
|
||||
nix profile install .
|
||||
```
|
||||
|
||||
### From crates.io
|
||||
|
||||
Not yet published — `improvise` is not on crates.io, so `cargo install
|
||||
improvise` does not work today. It is tracked as improvise-l36.
|
||||
|
||||
### Prebuilt binaries
|
||||
|
||||
The [GitHub releases page](https://github.com/fiddlerwoaroof/improvise/releases)
|
||||
carries prebuilt binaries for Linux (x86_64 and aarch64) and macOS (Intel and
|
||||
Apple Silicon), built by `cargo dist`. The current release is the `v0.1.0-rc2`
|
||||
prerelease; there is no `v0.1.0` tag yet (improvise-3tj).
|
||||
|
||||
Each release also carries `improvise-web.tar.gz`: the assembled browser app and
|
||||
landing page, ready to serve from any static host.
|
||||
|
||||
### The published site and the released binaries are not the same code
|
||||
|
||||
Deliberately, and it is worth knowing before you file a bug. The site deploys
|
||||
from `main` on every merge, so it is the latest code; a release tarball is
|
||||
whatever was tagged, which may be months older. **They are different lineages.**
|
||||
The web page names the build it was assembled from in its header, so quote that
|
||||
rather than a release number when reporting something you saw in a browser.
|
||||
`improvise-web.tar.gz` is a self-hostable copy of the site as of the tag and a
|
||||
rollback target — it is not what the published site is running.
|
||||
|
||||
## The data model
|
||||
|
||||
Every cell lives at the intersection of named categories and items —
|
||||
`(Region=East, Measure=Revenue)` — not at a grid address like `B3`. A model
|
||||
can have up to 24 categories, each with an ordered list of items that can be
|
||||
organized into collapsible groups (e.g. months grouped into quarters).
|
||||
|
||||
Formulas reference dimension names, not cell addresses:
|
||||
|
||||
```
|
||||
Profit = Revenue - Cost
|
||||
Tax = Revenue * 0.08
|
||||
Margin = IF(Revenue > 0, Profit / Revenue, 0)
|
||||
BigDeal = Sum(Revenue WHERE Region = "West")
|
||||
```
|
||||
|
||||
The formula language supports `+` `-` `*` `/` `^`, comparisons, `IF`, and
|
||||
aggregation functions (`Sum`, `Avg`, `Min`, `Max`, `Count`) with optional
|
||||
`WHERE` clauses. Formulas apply uniformly across all intersections — no
|
||||
copying, no dragging, no `$A$1` anchoring.
|
||||
|
||||
## Views and axes
|
||||
|
||||
A view assigns each category to one of four axes: **row**, **column**,
|
||||
**page** (slicer), or **hidden**. The grid layout is a pure function of
|
||||
`(Model, View)` → `GridLayout` — transposing is just swapping row and column
|
||||
assignments, and it happens in one keystroke (`t`). Page-axis categories act
|
||||
as filters: `[` and `]` cycle through items.
|
||||
|
||||
Press `T` to enter tile mode, where each category appears as a tile in the
|
||||
tile bar. Move between tiles with `h`/`l`, then press `Space` to cycle axes
|
||||
or `r`/`c`/`p` to set one directly.
|
||||
|
||||
Records mode (`R`) flips to a long-format view by assigning the virtual
|
||||
`_Index` and `_Dim` categories to row and column axes. Drill-down (`>`) on an
|
||||
aggregated cell captures a snapshot; edits accumulate in a staging area and
|
||||
commit atomically on exit.
|
||||
|
||||
## File format
|
||||
|
||||
Models persist to a plain-text `.improv` format that reads like markdown and
|
||||
diffs cleanly in git:
|
||||
|
||||
```
|
||||
# Sales 2025
|
||||
|
||||
## Category: Region
|
||||
- North
|
||||
- South
|
||||
- East [Coastal]
|
||||
- West [Coastal]
|
||||
> Coastal
|
||||
|
||||
## Category: Measure
|
||||
- Revenue
|
||||
- Cost
|
||||
- Profit
|
||||
|
||||
## Formulas
|
||||
- Profit = Revenue - Cost [Measure]
|
||||
|
||||
## Data
|
||||
Region=East, Measure=Revenue = 1200
|
||||
Region=East, Measure=Cost = 800
|
||||
|
||||
## View: Default (active)
|
||||
Region: row
|
||||
Measure: column
|
||||
format: ,.2f
|
||||
```
|
||||
|
||||
Gzip-compressed `.improv.gz` is also supported. `.improv` text is the only
|
||||
native document format: the old JSON document format is no longer read, and
|
||||
opening one fails with an error that says so. JSON *import* is unaffected.
|
||||
|
||||
## Import and scripting
|
||||
|
||||
The import wizard analyzes CSV columns and proposes each as a category
|
||||
(string-valued), measure (numeric), time dimension (date-like, with optional
|
||||
year/month/quarter extraction), or skip. Multiple CSVs merge automatically
|
||||
with a synthetic "File" category, even when their headers differ. A record
|
||||
simply has no value for a column its file lacks. A folder argument imports
|
||||
every `*.csv` in it, in name order, even when it holds a single CSV. Each file
|
||||
also gets its own view, filtered to that file and showing only its columns:
|
||||
|
||||
```sh
|
||||
improvise import reports/ --no-wizard -o reports.improv
|
||||
```
|
||||
|
||||
All model mutations go through a typed command registry, so the same
|
||||
operations that back the TUI work headless:
|
||||
|
||||
```sh
|
||||
# single commands
|
||||
improvise cmd 'add-cat Region' 'add-item Region East' -f model.improv
|
||||
|
||||
# script file (one command per line, # comments)
|
||||
improvise script setup.txt -f model.improv
|
||||
```
|
||||
|
||||
The language is the `:` command line, exactly as typed in the TUI — not JSON.
|
||||
Arguments split on whitespace, so anything containing a space is quoted:
|
||||
|
||||
```sh
|
||||
improvise cmd 'formula Product "Total = Widget + Gadget"' -f model.improv
|
||||
```
|
||||
|
||||
### What the headless commands promise
|
||||
|
||||
- **Everything a command reports goes to stderr**, as it would appear on the
|
||||
TUI's status line, and stdout is left empty. A run that had nothing to report
|
||||
prints nothing, so `2>/dev/null` is the whole suppression story and there is
|
||||
no quiet flag.
|
||||
- **Exit 1** means a line did not parse, or a command refused what it was
|
||||
asked — a failed `:w`, `add-item` naming a category that does not exist.
|
||||
A command that did its work and then had nowhere further to go — a
|
||||
`commit-cell-edit` on the last cell, whose advance stops — is not a refusal
|
||||
and exits 0. A few refusals are not yet classified and still exit 0
|
||||
(improvise-i3o5), so check stderr as well as `$?` when it matters.
|
||||
- **A run behaves as if its terminal were 80x24, whatever it runs in.** A page
|
||||
of rows is a real distance — `page-scroll` moves the cursor by one — so
|
||||
reading the actual window would make the same script write a different record
|
||||
in a different terminal, and a different one again in CI.
|
||||
- **A run that changed nothing does not rewrite the document.** Writing it back
|
||||
regardless silently normalises it — `1e20` becomes its twenty digits — which
|
||||
turns a hand-edited fixture into a no-op on the next run.
|
||||
|
||||
## In the browser
|
||||
|
||||
The browser page is at **<https://fiddlerwoaroof.github.io/improvise/app/>**,
|
||||
built and published from `main` by the `build` / `pages-deploy` jobs in
|
||||
`.github/workflows/ci.yml`. A deploy only happens after **every** check job in
|
||||
that workflow has passed. The list is deliberately not repeated here — naming a
|
||||
subset is how this sentence went stale before, and `scripts/check-ci-graph.py`
|
||||
now holds `pages-deploy`'s `needs` to the workflow's own job keys.
|
||||
|
||||
The site **root** is the landing page — `docs/index.html`, with the recorded
|
||||
terminal casts — and it links to the app. `web/assemble.sh` puts the two
|
||||
together, which is why the app sits under `app/` rather than at the root: every
|
||||
path in `app.js` and `worker.js` is relative, so the bundles move with it and
|
||||
the generated wasm-pack glue needs no editing.
|
||||
|
||||
**It needs no server.** The page is still a thin client — it holds no model,
|
||||
and every question about a cell, its text and whether it may be typed into, is
|
||||
answered by a server and arrives already decided — but the server it talks to
|
||||
is an in-page Web Worker hosting the whole `App` over the browser's private
|
||||
storage (OPFS). It is seeded on first load from `web/demo.improv`, so a visitor
|
||||
lands on a document, and edits persist across a reload without leaving the
|
||||
machine. `?doc=` and `?seed=` name another document and the file it is first
|
||||
seeded from.
|
||||
|
||||
To run the same page locally:
|
||||
|
||||
```sh
|
||||
./web/build.sh # wasm-pack -> web/{pkg,worker-pkg}
|
||||
python3 -m http.server 8080 --directory web
|
||||
```
|
||||
|
||||
The page and the OPFS backend are exercised by a Playwright harness in
|
||||
`harness/` — `npm ci` and `./harness/build.sh`, then `npx playwright test` from
|
||||
that directory. `tests/shell.spec.js` runs `web/app.js` under Chromium against a
|
||||
mocked peer; `tests/opfs.spec.js` runs the `Storage` contract against a real
|
||||
`OpfsFs` in a real Web Worker under Chromium and Firefox (WebKit provides no
|
||||
OPFS and is skipped on a runtime probe). CI's `browser` job runs both and can
|
||||
fail the build.
|
||||
|
||||
To open a document hosted by a separate process instead — which is how a
|
||||
document open in the TUI is shared — start the websocket server and pass
|
||||
`?ws=`:
|
||||
|
||||
```sh
|
||||
cargo run -p improvise-ws-server -- examples/demo.improv --port 8787
|
||||
```
|
||||
|
||||
then reload with `?ws=127.0.0.1:8787`.
|
||||
|
||||
**The scheme is not yours to supply and defaults to TLS.** `?ws=` takes a
|
||||
`host:port`; the page uses `wss://` when it is itself served over https and
|
||||
`ws://` when it is served over http, because an https origin refuses a `ws://`
|
||||
address as mixed content. An explicit `ws://` from an https page is upgraded
|
||||
rather than passed through — passing it through can only produce a failure
|
||||
that reads like "the server is not running".
|
||||
|
||||
So the local pairing above works as written, and it is the case worth
|
||||
optimising for: `http://localhost` is not a secure-context problem, so nothing
|
||||
below is needed to develop, to demo locally, or to share a TUI-hosted document
|
||||
with another tab on the same machine.
|
||||
|
||||
The server checks `Origin` on the handshake, which is what makes that safe. A
|
||||
WebSocket is exempt from the same-origin policy — there is no CORS preflight —
|
||||
so without the check any page you visited could connect to `ws://127.0.0.1:8787`
|
||||
and read or edit the document while the server runs. **Pages served from
|
||||
loopback are allowed by default**, which is exactly the recipe above; every
|
||||
other page is refused unless you name it with `--allow-origin`. A client that
|
||||
sends no `Origin` at all — curl, a script, a native client — is allowed, because
|
||||
no browser omits it and refusing them would answer a question `--bind` already
|
||||
answers.
|
||||
|
||||
This is not authentication (see the next section) and it does not depend on the
|
||||
browser: whatever Chromium's **Local Network Access** work ends up covering, the
|
||||
handshake is refused here rather than at the browser's discretion.
|
||||
|
||||
### Sharing a TUI-hosted document with a remote browser
|
||||
|
||||
Reaching `improvise-ws-server` from the **deployed** https page means
|
||||
terminating TLS in front of it. `improvise-ws-server` speaks plain `ws://`
|
||||
only — there is no rustls, no native-tls, nothing in the crate or its
|
||||
dependencies — so `wss://` is not obtainable from the binary itself, and this
|
||||
is the supported way to get it.
|
||||
|
||||
**Say the cost first, because it is real.** This means the documented path to
|
||||
sharing involves a second piece of software, which is a poor answer for a tool
|
||||
whose selling point is that it is one binary. It is the answer anyway, for a
|
||||
reason given under "Why not native `wss://`" below. If you only want *a*
|
||||
document in *a* browser, you do not need any of this — that is the standalone
|
||||
worker deployment, it is what the Pages site is for, and it needs no server
|
||||
process at all.
|
||||
|
||||
**Read this before following any of it.** `improvise-ws-server` has **no
|
||||
authentication of any kind**, and that is a deliberate MVP limit rather than an
|
||||
oversight — its own module doc says so. Anyone who can open a socket to it can
|
||||
**rewrite every document it serves**. Loopback binding is the only thing
|
||||
currently standing between that and the network, and *terminating TLS in front
|
||||
of it removes exactly that protection*: the proxy is public, so the endpoint is
|
||||
public, and `--bind 127.0.0.1` protects the port while doing nothing about the
|
||||
proxy. TLS gets you confidentiality, not access control.
|
||||
|
||||
**And the control has to be one a browser can actually satisfy**, which rules
|
||||
out most of the obvious ones. The page opens the socket as
|
||||
`new WebSocket(url)` — one argument, a URL — and the distinction that decides
|
||||
everything here is **ambient credentials versus challenge–response**:
|
||||
|
||||
- The handshake **is** an HTTP request, so the browser attaches what it
|
||||
attaches *automatically* — cookies for that host above all. That is why an
|
||||
identity-aware proxy can work at all.
|
||||
- What the page cannot do is **set a header** (there is no `Authorization`
|
||||
argument to `new WebSocket`) or **answer a prompt** (a handshake surfaces no
|
||||
login dialog, and its failure reaches script as a bare connection error).
|
||||
|
||||
So **HTTP basic auth is not a locked door here, it is a 401 the page can
|
||||
neither answer nor explain** — the same shape as the self-signed certificate
|
||||
that rules out native `wss://` below. Anything challenge-shaped fails for that
|
||||
reason; anything ambient may work. Client-certificate mTLS sits awkwardly
|
||||
between the two — the certificate is offered at the TLS layer rather than by
|
||||
the page, but the selection UI browsers show is a top-level-navigation
|
||||
affordance, and whether a cross-origin socket handshake gets it is **not
|
||||
something verified here**.
|
||||
|
||||
**And "who may connect" is two questions, not one.** A WebSocket is **exempt
|
||||
from the same-origin policy** — there is no CORS preflight on a handshake — so
|
||||
any hostile page an allowed user visits can open a socket to your endpoint
|
||||
*from that user's own allowed address*, and read and write the document. An
|
||||
address check cannot see that, because by then the request genuinely does come
|
||||
from an allowed client. Only the **`Origin` header** distinguishes them, and
|
||||
`improvise-ws-server` checks it: pages served from loopback are allowed, and
|
||||
anything else must be named with `--allow-origin`. **Deploying behind a proxy
|
||||
therefore needs the flag** — the deployed page's origin is
|
||||
`https://fiddlerwoaroof.github.io`, which is not loopback. Keep the Origin
|
||||
allow-list in the proxy as well: it is defence in depth now rather than the only
|
||||
defence, and it is what covers a proxy configuration that strips or rewrites the
|
||||
header before the server can see it.
|
||||
|
||||
So the honest controls are the ones that need **no credential from the page**:
|
||||
|
||||
- **An IP allow-list in the proxy** — the cheapest, and the worked example
|
||||
below. It is about where the client is, so the page sends nothing. On its
|
||||
own it answers only the first of the two questions above.
|
||||
- **A private network** (Tailscale, WireGuard). The peer is only reachable by
|
||||
machines already on it; again the page sends nothing.
|
||||
- **An identity-aware tunnel** (`cloudflared` with Cloudflare Access) — the
|
||||
*ambient* case above, and the reason that distinction is drawn at all. It
|
||||
works by a session **cookie** set after an interactive login on the proxy's
|
||||
own hostname, which the handshake then carries by itself; so it needs the
|
||||
user to visit that hostname first, and it is
|
||||
subject to the browser's third-party-cookie rules. Plausible and *not
|
||||
verified here* — the Playwright harness mocks its peer and reaches no
|
||||
identity-aware proxy — so it is named, not written out as a recipe.
|
||||
|
||||
What would make this genuinely easy is a **token in the URL**, because the URL
|
||||
is the one place the page can put a credential *deliberately* — everything else
|
||||
it sends is ambient. The server has
|
||||
no such capability today, and whether it should grow one — or whether "auth is
|
||||
the proxy's job" becomes a stated design position — is **improvise-kpnr**.
|
||||
|
||||
Until then: share only documents you would be willing to let anyone on the
|
||||
other side of that control edit, and take the tunnel down when you are done.
|
||||
|
||||
A worked example with [Caddy](https://caddyserver.com/), which obtains and
|
||||
renews the certificate itself. Point a DNS name at the host, then:
|
||||
|
||||
```caddyfile
|
||||
# Caddyfile
|
||||
improvise.example.com {
|
||||
# NOT optional — see above. Without it the document is world-writable.
|
||||
# An allow-list rather than a password, because the page cannot answer a
|
||||
# challenge on a socket handshake: this is checked against the address.
|
||||
@denied not remote_ip 203.0.113.4 198.51.100.0/24
|
||||
abort @denied
|
||||
|
||||
# Defence in depth, and NOT redundant with the address check. A WebSocket
|
||||
# is exempt from the same-origin policy, so any hostile page an allowed
|
||||
# user visits can open this socket FROM THAT ALLOWED ADDRESS. The address
|
||||
# check cannot see that; only the Origin can. The server checks it too —
|
||||
# this is what covers a proxy that strips or rewrites the header.
|
||||
@badorigin not header Origin https://fiddlerwoaroof.github.io
|
||||
abort @badorigin
|
||||
|
||||
reverse_proxy 127.0.0.1:8787
|
||||
}
|
||||
```
|
||||
|
||||
```sh
|
||||
# --allow-origin is required here: the deployed page is not a loopback origin,
|
||||
# so without it the server refuses the handshake the proxy forwards.
|
||||
improvise-ws-server bank-info.improv --port 8787 \
|
||||
--allow-origin https://fiddlerwoaroof.github.io # still plain ws, still loopback
|
||||
caddy run # terminates TLS, proxies to it
|
||||
```
|
||||
|
||||
On the same terms as the browser caveat below: **this configuration has not
|
||||
been executed in this repository** — there is no Caddy here, and the browser
|
||||
harness that does exist never puts a proxy in front of the socket. It
|
||||
is the form Caddy documents for `remote_ip`, and the reasoning about what a
|
||||
browser can satisfy is checked against `web/app.js`, which opens
|
||||
`new WebSocket(url)` and can therefore send no header. Verify the deny actually
|
||||
denies before trusting it with a document.
|
||||
|
||||
Then, from <https://fiddlerwoaroof.github.io/improvise/app/>, reload with
|
||||
`?ws=improvise.example.com`. Caddy preserves the request path, so the
|
||||
multi-document form works too: serve several files and name one with
|
||||
`?ws=improvise.example.com/bank-info`.
|
||||
|
||||
Caddy needs no websocket-specific configuration — `reverse_proxy` passes the
|
||||
`Upgrade` handshake through. nginx does: it needs `proxy_http_version 1.1` and
|
||||
the `Upgrade`/`Connection` headers set explicitly, or the handshake is
|
||||
answered as an ordinary request and the socket never opens.
|
||||
|
||||
For a one-off share with no DNS name and no certificate of your own,
|
||||
`cloudflared tunnel --url http://127.0.0.1:8787` or `ngrok http 8787` each
|
||||
print an https hostname that works the same way — pass the hostname to `?ws=`
|
||||
without a scheme, and pass the page's own origin to `--allow-origin`, since it
|
||||
is not loopback. Both print a **public** URL and neither authenticates anything
|
||||
by default, and an unguessable URL is not a control.
|
||||
|
||||
Two things carry over from above. `ngrok http --basic-auth` does **not** help
|
||||
here — the page cannot answer a basic-auth challenge on a socket handshake —
|
||||
and what you want instead is the same **address check** the Caddy example
|
||||
uses. But no command line for it is given here on purpose: ngrok has moved IP
|
||||
restriction from the (now deprecated) `--cidr-allow` flag to a Traffic Policy
|
||||
`restrict-ips` action passed with `--traffic-policy-file`, and cloudflared's
|
||||
equivalent is a product feature of its own. Those spellings have already
|
||||
changed once, none of them has been run in this repository, and a recipe that
|
||||
is correct today and silently wrong at the next major version is worse than a
|
||||
pointer — so: **consult their current documentation, and confirm the
|
||||
restriction denies before you trust it with a document.** If you cannot put an
|
||||
address check in front of the tunnel, do not expose it.
|
||||
|
||||
Note the server binds `127.0.0.1` by default and `--bind` is the only way to
|
||||
widen that. Behind a proxy on the same host you should *not* widen it: the
|
||||
proxy reaches loopback, and leaving the default keeps the **unencrypted** port
|
||||
off the network. That is worth doing and is not access control — the proxy is
|
||||
still the open door, which is why the **access controls** above belong there.
|
||||
Those are access controls and not authentication, which the server still has
|
||||
none of: they answer *which clients and which origins may connect*, never
|
||||
*who* is connecting. The distinction is the whole of improvise-kpnr.
|
||||
|
||||
### Why not native `wss://`
|
||||
|
||||
Declined deliberately rather than merely unimplemented (improvise-o8fa). Adding
|
||||
rustls is the easy half; the certificate is the hard one, and it is what makes
|
||||
the feature worse than the proxy rather than better. A self-signed certificate
|
||||
makes the browser refuse the socket outright — a websocket handshake has no
|
||||
interstitial to click through, so the failure arrives as a bare connection
|
||||
error with no usable indication that a certificate was the reason. That leaves
|
||||
a real certificate or an explicitly trusted local CA (`mkcert`) as the only
|
||||
pleasant options, and both of those are a second piece of software too. So the
|
||||
choice is not "one binary versus two", it is "two either way, and one of the
|
||||
options additionally ships a TLS stack and a certificate-loading surface in a
|
||||
binary whose job is a pivot table". If sharing a TUI-hosted document to a
|
||||
remote browser ever becomes a common case rather than an occasional one, this
|
||||
is worth revisiting on that evidence.
|
||||
|
||||
Caveat, narrowed twice: `harness/` runs `web/app.js` under Chromium — the
|
||||
standalone deployment under Firefox too — and `OpfsFs` under Chromium and
|
||||
Firefox, in CI. The grouped and multi-level DOM is executed since
|
||||
improvise-plcx, against a fixture document the harness serves beside the demo.
|
||||
What is still never executed is the CSS and the `ResizeObserver` arithmetic —
|
||||
and Safari, whose Playwright build provides no OPFS at all. `bindings.rs` **is**
|
||||
run: the no-`?ws` test loads the app, which spawns the worker, which calls
|
||||
`start()` and `OpfsFs::open()`.
|
||||
|
||||
## What's interesting about the architecture
|
||||
|
||||
All user actions flow through a two-phase command/effect pipeline. Commands
|
||||
are pure functions: they receive an immutable `CmdContext` (model, layout,
|
||||
cursor position, mode) and return a list of effects. Effects are the only
|
||||
things that mutate app state, and each one is a small, debuggable struct.
|
||||
This means commands are testable without a terminal, effects can be logged or
|
||||
replayed, and the 40+ commands and 50+ effect types are all polymorphic —
|
||||
dispatched through trait objects and a registry, not a central match block.
|
||||
The keybinding system gives each of the 17 modes (`ModeKey`) its own keymap, with
|
||||
Emacs-style prefix keys for multi-stroke sequences.
|
||||
|
||||
## Expectations
|
||||
|
||||
improvise is a personal project I built for my own use. I'm sharing it because
|
||||
other people might find it useful, but I can't promise active maintenance or
|
||||
feature development. Issues and PRs are welcome but may not get a fast
|
||||
response. If you want to build on it, fork away.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -1,83 +0,0 @@
|
||||
# Releasing
|
||||
|
||||
A checklist, not an explanation. The reasoning is `context/code/tooling.md` §4;
|
||||
read it if a step surprises you, not before.
|
||||
|
||||
Its own file rather than a `README.md` section because `README.md` §Installation
|
||||
addresses someone *installing* improvise and this addresses the one person
|
||||
*publishing* it — folding them together lengthens the user-facing document for
|
||||
no user benefit.
|
||||
|
||||
## The one rule
|
||||
|
||||
**Bump the version and commit it BEFORE you tag.** `dist` matches the tag
|
||||
against the root package's version and refuses when they disagree — and by the
|
||||
time CI tells you, the tag is already published and a published tag is not
|
||||
moved. Every step below exists to put that check before the irreversible one.
|
||||
|
||||
## Cutting a release
|
||||
|
||||
1. **Bump the version.** `Cargo.toml`'s `[package]` version is the one `dist`
|
||||
announces (`dist-workspace.toml` says `members = ["cargo:."]`). Keep the
|
||||
crates under `crates/` in step with it; `gp-tree` sits at its own version
|
||||
deliberately, and the preflight lists such crates as a note rather than an
|
||||
error.
|
||||
|
||||
2. **Commit the bump**, and let `Cargo.lock` be part of that commit.
|
||||
|
||||
3. **Run the preflight**, with the tag you intend to push:
|
||||
|
||||
```sh
|
||||
scripts/release-preflight.sh vX.Y.Z
|
||||
```
|
||||
|
||||
It refuses a dirty tree, refuses a tag that does not match the version, runs
|
||||
`dist plan` as a second opinion when `dist` is on `PATH`, and prints the tag
|
||||
commands only once it has passed. It cuts nothing itself.
|
||||
|
||||
4. **Tag and push**, using the commands it printed:
|
||||
|
||||
```sh
|
||||
git tag -a vX.Y.Z -m "vX.Y.Z"
|
||||
git push gh vX.Y.Z
|
||||
```
|
||||
|
||||
Push the tag to **`gh`** — `.github/workflows/release.yml` triggers on a tag
|
||||
push there, matching `**[0-9]+.[0-9]+.[0-9]+*`. `origin` and `gitea` are
|
||||
mirrors and build nothing.
|
||||
|
||||
5. **Watch the run.** A green release takes about three minutes and publishes
|
||||
tarballs for all four targets with their `.sha256` files, `sha256.sum`,
|
||||
`dist-manifest.json`, `source.tar.gz`, `improvise-installer.sh` and
|
||||
`improvise-web.tar.gz` — the assembled landing page and browser app, stamped
|
||||
with the tag, which the page then shows in its header (improvise-oujb).
|
||||
|
||||
**`improvise-web.tar.gz` is a self-hostable copy and a rollback target, not
|
||||
what the published site is serving.** GitHub Pages tracks `main` and a tag
|
||||
does not deploy it, so the site and the release are deliberately different
|
||||
code. Two things follow when the web half goes wrong. `release.yml` is
|
||||
autogenerated, so never hand-edit it: change `dist-workspace.toml` and re-run
|
||||
`dist generate`, which should leave the workflow byte-identical. And a
|
||||
**pull-request** run of the Release workflow skips `build-global-artifacts`,
|
||||
so a green PR says nothing about the web capture — to exercise it without
|
||||
cutting a tag, run `dist build --artifacts=global` locally.
|
||||
|
||||
The failure signature is unmistakable once you know it: **`plan` fails in
|
||||
about twenty seconds and the four jobs behind it skip.** That is the tag not
|
||||
matching the version — see below. A run that gets past `plan` is building.
|
||||
|
||||
## When a tag has already been pushed against the wrong version
|
||||
|
||||
**Leave it dead and cut the next one.** Do not move or re-push the tag: it is
|
||||
already public, and a published tag that changes meaning is worse than one that
|
||||
never built. Bump to the next version, commit, run the preflight, and tag that.
|
||||
|
||||
The worked example is `v0.1.0-rc3`: it was pushed while every crate still said
|
||||
`0.1.0-rc2`, `dist` refused, and the four jobs behind `plan` skipped. No release
|
||||
was ever built from it.
|
||||
|
||||
**Ask which tags exist; do not trust a document that tells you.** `git tag -l`,
|
||||
and `git ls-remote --tags <remote>` for each remote you push to. Which tags are
|
||||
where is a fact with a shelf life — it changed twice while this file was being
|
||||
written — so this file deliberately records no inventory of it. Cleaning up a
|
||||
dead tag is a separate decision and is not a step in cutting a release.
|
||||
+1189
File diff suppressed because it is too large
Load Diff
@@ -1,8 +0,0 @@
|
||||
import sys; sys.path.insert(0,'bench')
|
||||
from scale_bisect import run
|
||||
cfgs=[dict(k=k,prune=0) for k in (16,17,18,26,27)]+[dict(k=k,prune=1) for k in (7,11,12,13,14,21,22)]+[dict(k=1000,n=n,mode='records') for n in (1000,1200,1500,1800,2000,4600,14500)]
|
||||
for c in cfgs:
|
||||
c.setdefault('n',100000); c['ops']='open,key,window'
|
||||
v=[run('/tmp/sb',c,timeout=200) for _ in range(3)]
|
||||
f=lambda k:'/'.join(f"{r[k][1]:.1f}" if k in r else '-' for r in v)
|
||||
print({k:c[k] for k in c if k!='ops'},'rows',v[0].get('rows'),'open',f('open'),'key',f('keypress_down'),'win',f('browser_grid_of_top'),'tuiframe',f('tui_frame_grid_of_whole'),flush=True)
|
||||
@@ -1,10 +0,0 @@
|
||||
import sys, json; sys.path.insert(0,'bench')
|
||||
from scale_bisect import run
|
||||
cfgs=[dict(k=9,prune=0),dict(k=10,prune=0),dict(k=17,prune=0,groups=1),dict(k=26,prune=0,groups=1),
|
||||
dict(k=8,prune=1),dict(k=13,prune=1,groups=1),
|
||||
dict(k=1000,n=1300,mode='records'),dict(k=1000,n=4400,mode='records'),dict(k=1000,n=14000,mode='records')]
|
||||
for c in cfgs:
|
||||
c.setdefault('n',100000)
|
||||
for rep in range(3):
|
||||
r=run('/tmp/sb',c,timeout=300)
|
||||
print(json.dumps({'cfg':c,'rep':rep,**r}),flush=True)
|
||||
@@ -1,38 +0,0 @@
|
||||
"""Drive benches/scale_bisect one configuration per process; parse its lines.
|
||||
usage: scale_bisect.py BIN k=.. n=.. [cats=..] [prune=..] [mode=..] [ops=..] [--timeout S]
|
||||
"""
|
||||
import subprocess, sys, re, json
|
||||
|
||||
def run(binary, cfg, timeout=90):
|
||||
args = [binary] + [f"{k}={v}" for k, v in cfg.items()]
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
out, status = p.stdout, "ok"
|
||||
except subprocess.TimeoutExpired as e:
|
||||
out, status = (e.stdout or b"").decode() if isinstance(e.stdout, bytes) else (e.stdout or ""), "timeout"
|
||||
res = {"status": status}
|
||||
for line in out.splitlines():
|
||||
if line.startswith("REFUSED"):
|
||||
res["status"] = "refused"
|
||||
m = re.match(r"(\w+) min=([\d.]+) med=([\d.]+) max=([\d.]+)", line)
|
||||
if m:
|
||||
res[m[1]] = (float(m[2]), float(m[3]), float(m[4]))
|
||||
for kv in re.findall(r"(\w+)=([\d.]+)", line):
|
||||
if kv[0] in ("rows", "cols", "maxrss_mb", "improv_mb", "snapshot_kb", "rss_after_open_mb", "cells"):
|
||||
res[kv[0]] = float(kv[1])
|
||||
m = re.match(r"json_encode_setcell_updates ms=(.*)", line)
|
||||
if m:
|
||||
res["json_setcell"] = m[1]
|
||||
return res
|
||||
|
||||
if __name__ == "__main__":
|
||||
binary = sys.argv[1]
|
||||
cfg, timeout = {}, 90
|
||||
rest = sys.argv[2:]
|
||||
for a in rest:
|
||||
if a.startswith("--timeout"):
|
||||
timeout = int(a.split("=")[1])
|
||||
else:
|
||||
k, v = a.split("=", 1)
|
||||
cfg[k] = v
|
||||
print(json.dumps(run(binary, cfg, timeout)))
|
||||
@@ -1,10 +0,0 @@
|
||||
import sys; sys.path.insert(0,'bench')
|
||||
from scale_bisect import run
|
||||
import itertools
|
||||
OPS=sys.argv[1];
|
||||
cfgs=eval(sys.argv[2])
|
||||
cols=sys.argv[3].split(',')
|
||||
for c in cfgs:
|
||||
c=dict(c); c['ops']=OPS
|
||||
r=run('/tmp/sb',c,timeout=int(sys.argv[4]) if len(sys.argv)>4 else 90)
|
||||
print(c, r['status'], 'rows',r.get('rows'), *[f"{k}={r[k][1]:.1f}" if k in r else f"{k}=-" for k in cols], f"rss={r.get('maxrss_mb')}", flush=True)
|
||||
@@ -1,261 +0,0 @@
|
||||
//! Does the TUI's whole-grid `GridCache` dominate a frame? See improvise-i5tk,
|
||||
//! improvise-jmjz and `context/code/command.md`. `cargo bench --bench grid_projection`.
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use improvise::format::FormatPlan;
|
||||
use improvise::model::cell::{CellKey, CellValue};
|
||||
use improvise::view::{Axis, GridLayout};
|
||||
use improvise::workbook::Workbook;
|
||||
use improvise_command::ui::app::App;
|
||||
use improvise_command::ui::project::{self, Viewport};
|
||||
use improvise_command::ui::viewport::compute_col_widths;
|
||||
|
||||
/// A dense pivot grid: `regions * products` rows by `measures` columns, every
|
||||
/// cell filled, which is the worst case for a per-cell evaluation.
|
||||
fn dense_app(regions: usize, products: usize, measures: usize) -> App {
|
||||
let mut wb = Workbook::new("Bench");
|
||||
wb.add_category("Region").unwrap();
|
||||
wb.add_category("Product").unwrap();
|
||||
let region_names: Vec<String> = (0..regions).map(|i| format!("Region{i}")).collect();
|
||||
let product_names: Vec<String> = (0..products).map(|i| format!("Product{i}")).collect();
|
||||
let measure_names: Vec<String> = (0..measures).map(|i| format!("Measure{i}")).collect();
|
||||
for r in ®ion_names {
|
||||
wb.add_item("Region", r.clone()).unwrap();
|
||||
}
|
||||
for p in &product_names {
|
||||
wb.add_item("Product", p.clone()).unwrap();
|
||||
}
|
||||
for m in &measure_names {
|
||||
wb.add_item("_Measure", m.clone()).unwrap();
|
||||
}
|
||||
{
|
||||
let v = wb.active_view_mut();
|
||||
v.set_axis("Region", Axis::Row);
|
||||
v.set_axis("Product", Axis::Row);
|
||||
v.set_axis("_Measure", Axis::Column);
|
||||
}
|
||||
let mut n = 0.0f64;
|
||||
for r in ®ion_names {
|
||||
for p in &product_names {
|
||||
for m in &measure_names {
|
||||
n += 1.0;
|
||||
let key = CellKey::new(vec![
|
||||
("Region".to_string(), r.clone()),
|
||||
("Product".to_string(), p.clone()),
|
||||
("_Measure".to_string(), m.clone()),
|
||||
]);
|
||||
wb.model.set_cell(key, CellValue::Number(n));
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut app = App::new(wb, None);
|
||||
app.rebuild_layout();
|
||||
app
|
||||
}
|
||||
|
||||
/// Collapsed groups on the row axis and a formula chain on the column axis, so
|
||||
/// every fold cell and every total re-derives the chain (improvise-h6z1).
|
||||
fn chain_app(regions: usize, products: usize) -> App {
|
||||
let mut wb = Workbook::new("Chain");
|
||||
wb.add_category("Region").unwrap();
|
||||
wb.add_category("Product").unwrap();
|
||||
for r in 0..regions {
|
||||
wb.add_item_in_group("Region", format!("Region{r}"), format!("G{}", r / 10))
|
||||
.unwrap();
|
||||
}
|
||||
for p in 0..products {
|
||||
wb.add_item("Product", format!("Product{p}")).unwrap();
|
||||
}
|
||||
wb.add_item("_Measure", "Revenue").unwrap();
|
||||
let chain = [
|
||||
"F1 = Revenue * 2",
|
||||
"F2 = F1 + Revenue",
|
||||
"F3 = F2 + F1",
|
||||
"F4 = F3 + F2",
|
||||
"F5 = F4 / F1",
|
||||
];
|
||||
for text in chain {
|
||||
let formula = improvise::formula::parse_formula(text, "_Measure").unwrap();
|
||||
wb.model.add_formula(formula);
|
||||
}
|
||||
{
|
||||
let v = wb.active_view_mut();
|
||||
v.set_axis("Region", Axis::Row);
|
||||
v.set_axis("Product", Axis::Row);
|
||||
v.set_axis("_Measure", Axis::Column);
|
||||
}
|
||||
let mut n = 0.0f64;
|
||||
for r in 0..regions {
|
||||
for p in 0..products {
|
||||
n += 1.0;
|
||||
let key = CellKey::new(vec![
|
||||
("Region".to_string(), format!("Region{r}")),
|
||||
("Product".to_string(), format!("Product{p}")),
|
||||
("_Measure".to_string(), "Revenue".to_string()),
|
||||
]);
|
||||
wb.model.set_cell(key, CellValue::Number(n));
|
||||
}
|
||||
}
|
||||
let mut app = App::new(wb, None);
|
||||
app.rebuild_layout();
|
||||
app
|
||||
}
|
||||
|
||||
/// Days in collapsed groups of `per_group` on Row, with Product: every row is a
|
||||
/// fold over a few days, never the whole category (improvise-jq0m).
|
||||
fn grouped_app(days: usize, per_group: usize, products: usize, measures: usize) -> App {
|
||||
let mut wb = Workbook::new("Grouped");
|
||||
wb.add_category("Day").unwrap();
|
||||
wb.add_category("Product").unwrap();
|
||||
for d in 0..days {
|
||||
let group = format!("G{}", d / per_group);
|
||||
wb.add_item_in_group("Day", format!("Day{d}"), group)
|
||||
.unwrap();
|
||||
}
|
||||
for p in 0..products {
|
||||
wb.add_item("Product", format!("Product{p}")).unwrap();
|
||||
}
|
||||
for m in 0..measures {
|
||||
wb.add_item("_Measure", format!("Measure{m}")).unwrap();
|
||||
}
|
||||
{
|
||||
let v = wb.active_view_mut();
|
||||
v.set_axis("Day", Axis::Row);
|
||||
v.set_axis("Product", Axis::Row);
|
||||
v.set_axis("_Measure", Axis::Column);
|
||||
}
|
||||
let mut n = 0.0f64;
|
||||
for d in 0..days {
|
||||
for p in 0..products {
|
||||
for m in 0..measures {
|
||||
n += 1.0;
|
||||
let key = CellKey::new(vec![
|
||||
("Day".to_string(), format!("Day{d}")),
|
||||
("Product".to_string(), format!("Product{p}")),
|
||||
("_Measure".to_string(), format!("Measure{m}")),
|
||||
]);
|
||||
wb.model.set_cell(key, CellValue::Number(n));
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut app = App::new(wb, None);
|
||||
app.rebuild_layout();
|
||||
app
|
||||
}
|
||||
|
||||
fn whole_grid(app: &App) -> Viewport {
|
||||
Viewport {
|
||||
row_offset: 0,
|
||||
col_offset: 0,
|
||||
rows: app.layout.row_count(),
|
||||
cols: app.layout.col_count(),
|
||||
}
|
||||
}
|
||||
|
||||
/// A terminal window: what the TUI would ask for if a cache position were
|
||||
/// relative, which is the browser client's arrangement.
|
||||
fn terminal_window() -> Viewport {
|
||||
Viewport {
|
||||
row_offset: 0,
|
||||
col_offset: 0,
|
||||
rows: 45,
|
||||
cols: 12,
|
||||
}
|
||||
}
|
||||
|
||||
/// Min-of-N, not the mean: a mean over a handful of iterations is dominated by
|
||||
/// whatever else the machine was doing, which is how a share above 100% happens.
|
||||
fn time(iters: u32, app: &App, viewport: Viewport) -> Duration {
|
||||
let mut best = Duration::MAX;
|
||||
for _ in 0..iters.max(3) {
|
||||
let start = Instant::now();
|
||||
std::hint::black_box(project::grid_of(app, viewport));
|
||||
best = best.min(start.elapsed());
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
/// `compute_col_widths` is whole-grid whatever the viewport says, so it is the part
|
||||
/// narrowing the cells loop cannot reach. It must be timed with the arguments
|
||||
/// `grid_of` passes it, or the share is of a call that never happens.
|
||||
// Since improvise-jmjz it IS the fused walk, with an empty window, so its share
|
||||
// rose while the frame fell: it is now the work, not the redundancy.
|
||||
fn time_widths(iters: u32, app: &App) -> Duration {
|
||||
let layout = &app.layout;
|
||||
let model = &app.model_state.workbook.model;
|
||||
let plan = FormatPlan::of(model, app.model_state.workbook.active_view());
|
||||
let mut best = Duration::MAX;
|
||||
for _ in 0..iters.max(3) {
|
||||
let start = Instant::now();
|
||||
std::hint::black_box(compute_col_widths(model, layout, &plan));
|
||||
best = best.min(start.elapsed());
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
fn case(label: &str, regions: usize, products: usize, measures: usize, iters: u32) {
|
||||
report(label, dense_app(regions, products, measures), iters);
|
||||
}
|
||||
|
||||
fn report(label: &str, app: App, iters: u32) {
|
||||
let rows = app.layout.row_count();
|
||||
let cols = app.layout.col_count();
|
||||
let cells = rows * cols;
|
||||
let full = time(iters, &app, whole_grid(&app));
|
||||
let windowed = time(iters, &app, terminal_window());
|
||||
let widths = time_widths(iters, &app);
|
||||
let ratio = full.as_secs_f64() / windowed.as_secs_f64().max(f64::EPSILON);
|
||||
let share = 100.0 * widths.as_secs_f64() / full.as_secs_f64().max(f64::EPSILON);
|
||||
// A part cannot take longer than the whole that contains it. Say so rather
|
||||
// than printing a number that reads like a measurement.
|
||||
let share = match share > 100.0 {
|
||||
true => "UNSTABLE".to_string(),
|
||||
false => format!("{share:.0}%"),
|
||||
};
|
||||
// The TUI pays both on every keypress: widths in `cmd_context`, the grid in draw.
|
||||
let keypress = full + widths;
|
||||
println!(
|
||||
"{label:<22} {rows:>5} x {cols:>3} = {cells:>7} cells whole-grid {full:>11.3?} \
|
||||
windowed {windowed:>11.3?} ratio {ratio:>4.1}x col_widths {widths:>11.3?} ({share} of the frame) \
|
||||
keypress {keypress:>11.3?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The layout build with `prune_empty` on: every cell evaluated to find the empty
|
||||
/// rows and columns, on the same keypress path as `grid_of` (improvise-metf).
|
||||
fn report_prune(label: &str, mut app: App, iters: u32) {
|
||||
app.model_state.workbook.active_view_mut().prune_empty = true;
|
||||
let model = &app.model_state.workbook.model;
|
||||
let view = app.model_state.workbook.active_view();
|
||||
let mut best = Duration::MAX;
|
||||
for _ in 0..iters.max(3) {
|
||||
let start = Instant::now();
|
||||
std::hint::black_box(GridLayout::with_frozen_records(model, view, None));
|
||||
best = best.min(start.elapsed());
|
||||
}
|
||||
println!("{label:<22} layout build with prune_empty {best:>11.3?}");
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("project::grid_of, one call = one TUI frame's grid cache");
|
||||
println!("a 60fps frame budget is 16.7ms; the TUI redraws on each keypress\n");
|
||||
case("typical", 20, 5, 4, 200);
|
||||
case("1.2k cells", 60, 5, 4, 100);
|
||||
case("2.4k cells", 60, 10, 4, 100);
|
||||
case("4.8k cells", 60, 20, 4, 50);
|
||||
case("large", 60, 20, 8, 50);
|
||||
case("18k cells (gotcha 41)", 150, 20, 6, 20);
|
||||
case("pathological", 300, 30, 10, 5);
|
||||
println!("\nfolded rows and a five-formula chain (improvise-h6z1)\n");
|
||||
report("folded chain ~18k", chain_app(1000, 30), 5);
|
||||
report("folded chain ~90k", chain_app(5000, 30), 3);
|
||||
println!("\nthe same, pruned (improvise-metf)\n");
|
||||
report_prune("dense 18k", dense_app(150, 20, 6), 5);
|
||||
report_prune("folded chain ~18k", chain_app(1000, 30), 5);
|
||||
report_prune("folded chain ~90k", chain_app(5000, 30), 3);
|
||||
println!("\ncollapsed small groups on the row axis (improvise-jq0m)\n");
|
||||
report("364 days in weeks", grouped_app(364, 7, 20, 6), 5);
|
||||
report("364 days in pairs", grouped_app(364, 2, 20, 6), 5);
|
||||
report("1000 days in pairs", grouped_app(1000, 2, 10, 4), 3);
|
||||
}
|
||||
@@ -1,138 +0,0 @@
|
||||
//! What a model batch costs on the keypress path, which is where the reshape
|
||||
//! observation sits (improvise-ni6k). Run with
|
||||
//! `cargo bench --bench reshape_observation`.
|
||||
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use improvise::model::cell::{CellKey, CellValue};
|
||||
use improvise::view::Axis;
|
||||
use improvise::workbook::Workbook;
|
||||
use improvise_command::ui::app::App;
|
||||
use improvise_command::ui::effect::{CycleAxis, Effect, SetCell, ViewTarget};
|
||||
|
||||
/// The fixture `grid_projection` uses, so the two numbers are comparable.
|
||||
fn dense_app(regions: usize, products: usize, measures: usize) -> App {
|
||||
let mut wb = Workbook::new("Bench");
|
||||
wb.add_category("Region").unwrap();
|
||||
wb.add_category("Product").unwrap();
|
||||
let regions: Vec<String> = (0..regions).map(|i| format!("Region{i}")).collect();
|
||||
let products: Vec<String> = (0..products).map(|i| format!("Product{i}")).collect();
|
||||
let measures: Vec<String> = (0..measures).map(|i| format!("Measure{i}")).collect();
|
||||
for r in ®ions {
|
||||
wb.add_item("Region", r.clone()).unwrap();
|
||||
}
|
||||
for p in &products {
|
||||
wb.add_item("Product", p.clone()).unwrap();
|
||||
}
|
||||
for m in &measures {
|
||||
wb.add_item("_Measure", m.clone()).unwrap();
|
||||
}
|
||||
{
|
||||
let v = wb.active_view_mut();
|
||||
v.set_axis("Region", Axis::Row);
|
||||
v.set_axis("Product", Axis::Row);
|
||||
v.set_axis("_Measure", Axis::Column);
|
||||
}
|
||||
let mut n = 0.0f64;
|
||||
for r in ®ions {
|
||||
for p in &products {
|
||||
for m in &measures {
|
||||
n += 1.0;
|
||||
let key = CellKey::new(vec![
|
||||
("Region".to_string(), r.clone()),
|
||||
("Product".to_string(), p.clone()),
|
||||
("_Measure".to_string(), m.clone()),
|
||||
]);
|
||||
wb.model.set_cell(key, CellValue::Number(n));
|
||||
}
|
||||
}
|
||||
}
|
||||
// More than one view, because the observation runs over all of them.
|
||||
wb.create_view("Second");
|
||||
wb.switch_view("Default").unwrap();
|
||||
let mut app = App::new(wb, None);
|
||||
app.rebuild_layout();
|
||||
app
|
||||
}
|
||||
|
||||
/// Min-of-N for the reason `grid_projection` says: a mean over a few runs
|
||||
/// measures the machine, not the code.
|
||||
fn time_batch(iters: u32, app: &mut App, key: &CellKey) -> Duration {
|
||||
let mut best = Duration::MAX;
|
||||
for i in 0..iters.max(3) {
|
||||
let effects: Vec<Box<dyn Effect>> = vec![Box::new(SetCell(
|
||||
key.clone(),
|
||||
CellValue::Number(i as f64 + 1.0),
|
||||
))];
|
||||
let start = Instant::now();
|
||||
std::hint::black_box(app.apply_effects(effects));
|
||||
best = best.min(start.elapsed());
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
fn case(label: &str, regions: usize, products: usize, measures: usize, iters: u32) {
|
||||
let mut app = dense_app(regions, products, measures);
|
||||
let (rows, cols) = (app.layout.row_count(), app.layout.col_count());
|
||||
let key = CellKey::new(vec![
|
||||
("Region".to_string(), "Region0".to_string()),
|
||||
("Product".to_string(), "Product0".to_string()),
|
||||
("_Measure".to_string(), "Measure0".to_string()),
|
||||
]);
|
||||
let batch = time_batch(iters, &mut app, &key);
|
||||
println!(
|
||||
"{label:<22} {rows:>5} x {cols:>3} = {:>7} cells one SetCell batch {batch:>11.3?}",
|
||||
rows * cols
|
||||
);
|
||||
}
|
||||
|
||||
/// The **held-key** path improvise-748f exists for: a geometry batch writes no
|
||||
/// cell, so `document_content` never runs and the frame fingerprint is alone.
|
||||
fn geometry_case(label: &str, regions: usize, products: usize, measures: usize, iters: u32) {
|
||||
// DO NOT set this false to "simplify". `prune_empty` is the ONLY thing that
|
||||
// makes the fingerprint walk cells — see `context/code/command.md`.
|
||||
geometry_at(label, regions, products, measures, iters, true);
|
||||
}
|
||||
|
||||
fn geometry_at(
|
||||
label: &str,
|
||||
regions: usize,
|
||||
products: usize,
|
||||
measures: usize,
|
||||
iters: u32,
|
||||
prune: bool,
|
||||
) {
|
||||
let mut app = dense_app(regions, products, measures);
|
||||
app.model_state.workbook.active_view_mut().prune_empty = prune;
|
||||
app.rebuild_layout();
|
||||
let (rows, cols) = (app.layout.row_count(), app.layout.col_count());
|
||||
let mut best = Duration::MAX;
|
||||
for i in 0..iters.max(3) {
|
||||
let effects: Vec<Box<dyn Effect>> = vec![Box::new(CycleAxis(
|
||||
ViewTarget::Active,
|
||||
"Product".to_string(),
|
||||
))];
|
||||
let start = Instant::now();
|
||||
std::hint::black_box(app.apply_effects(effects));
|
||||
best = best.min(start.elapsed());
|
||||
let _ = i;
|
||||
}
|
||||
println!(
|
||||
"{label:<22} {rows:>5} x {cols:>3} = {:>7} cells one CycleAxis batch {best:>11.3?}",
|
||||
rows * cols
|
||||
);
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("App::apply_effects, one model batch = one keypress that writes a cell");
|
||||
println!("a 60fps frame budget is 16.7ms\n");
|
||||
case("typical", 20, 5, 4, 200);
|
||||
case("large", 60, 20, 8, 50);
|
||||
case("18k cells (gotcha 41)", 150, 20, 6, 20);
|
||||
case("pathological", 300, 30, 10, 5);
|
||||
println!("\nthe held-key path (improvise-748f): a batch that writes no cell");
|
||||
geometry_case("typical", 20, 5, 4, 200);
|
||||
geometry_case("large", 60, 20, 8, 50);
|
||||
geometry_case("18k cells (gotcha 41)", 150, 20, 6, 20);
|
||||
geometry_case("pathological", 300, 30, 10, 5);
|
||||
}
|
||||
@@ -1,347 +0,0 @@
|
||||
//! One configuration per process: build a sparse document, time the real paths.
|
||||
//! `scale_bisect k=1000 n=100000 cats=3 prune=1 mode=grid`; see context/test-evidence/scale-bisect.md.
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::io::Write;
|
||||
use std::time::Instant;
|
||||
|
||||
use improvise::model::cell::{CellKey, CellValue};
|
||||
use improvise::view::Axis;
|
||||
use improvise::workbook::Workbook;
|
||||
use improvise_command::ui::app::App;
|
||||
use improvise_command::ui::effect::{self, ViewTarget};
|
||||
use improvise_command::ui::host::Host;
|
||||
use improvise_command::ui::project::{self, Viewport};
|
||||
use improvise_command::ui::reduce::caught_up;
|
||||
use improvise_io::persistence::{format_md, parse_md, save_to};
|
||||
use improvise_io::storage::MemoryFs;
|
||||
use improvise_protocol::{
|
||||
ClientFrame, KeyCode, KeyEvent, KeyModifiers, ModelCommand, RecordHash, render::RecordCell,
|
||||
};
|
||||
|
||||
fn lcg(s: &mut u64, n: usize) -> usize {
|
||||
*s = s
|
||||
.wrapping_mul(6_364_136_223_846_793_005)
|
||||
.wrapping_add(1_442_695_040_888_963_407);
|
||||
(*s >> 33) as usize % n
|
||||
}
|
||||
|
||||
struct Cfg {
|
||||
k: usize,
|
||||
n: usize,
|
||||
cats: usize,
|
||||
prune: bool,
|
||||
mode: String,
|
||||
groups: bool,
|
||||
ops: String,
|
||||
}
|
||||
|
||||
fn arg(args: &HashMap<String, String>, k: &str, d: &str) -> String {
|
||||
args.get(k).cloned().unwrap_or_else(|| d.to_string())
|
||||
}
|
||||
|
||||
fn document(c: &Cfg) -> (Workbook, Vec<CellKey>) {
|
||||
let mut wb = Workbook::new("bench");
|
||||
for ci in 0..c.cats {
|
||||
let cat = format!("C{ci}");
|
||||
wb.add_category(cat.clone()).expect("under the limit");
|
||||
for i in 0..c.k {
|
||||
if c.groups && ci == 0 {
|
||||
wb.add_item_in_group(&cat, format!("i{i}"), format!("G{}", i / 10))
|
||||
.expect("added");
|
||||
} else {
|
||||
wb.add_item(&cat, format!("i{i}")).expect("added");
|
||||
}
|
||||
}
|
||||
}
|
||||
for m in ["Cost", "Rev"] {
|
||||
wb.add_item("_Measure", m).expect("virtual");
|
||||
}
|
||||
let space = (c.k as f64).powi(c.cats as i32);
|
||||
let wanted = c.n.min((space / 2.0).max(1.0) as usize);
|
||||
let (mut s, mut seen, mut recs) = (0x9E37_79B9_7F4A_7C15u64, HashSet::new(), Vec::new());
|
||||
while seen.len() < wanted {
|
||||
let d: Vec<usize> = (0..c.cats).map(|_| lcg(&mut s, c.k)).collect();
|
||||
if !seen.insert(d.clone()) {
|
||||
continue;
|
||||
}
|
||||
let coords: Vec<(String, String)> = d
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(ci, x)| (format!("C{ci}"), format!("i{x}")))
|
||||
.collect();
|
||||
recs.push(CellKey::new(coords.clone()));
|
||||
for (m, v) in [("Cost", 3.0), ("Rev", 5.0)] {
|
||||
let mut key = coords.clone();
|
||||
key.push(("_Measure".to_string(), m.to_string()));
|
||||
wb.model.set_cell(CellKey::new(key), CellValue::Number(v));
|
||||
}
|
||||
}
|
||||
let view = wb.active_view_mut();
|
||||
for ci in 0..c.cats {
|
||||
let axis = if c.mode == "records" || c.mode == "none" {
|
||||
Axis::None
|
||||
} else {
|
||||
Axis::Row
|
||||
};
|
||||
view.set_axis(&format!("C{ci}"), axis);
|
||||
}
|
||||
if c.mode == "records" {
|
||||
view.set_axis("_Index", Axis::Row);
|
||||
view.set_axis("_Dim", Axis::Column);
|
||||
} else {
|
||||
view.set_axis("_Measure", Axis::Column);
|
||||
}
|
||||
view.prune_empty = c.prune;
|
||||
(wb, recs)
|
||||
}
|
||||
|
||||
fn maxrss_mb() -> f64 {
|
||||
let mut u: libc::rusage = unsafe { std::mem::zeroed() };
|
||||
unsafe { libc::getrusage(libc::RUSAGE_SELF, &mut u) };
|
||||
u.ru_maxrss as f64 / 1048576.0
|
||||
}
|
||||
|
||||
fn rss_mb() -> f64 {
|
||||
let out = std::process::Command::new("ps")
|
||||
.args(["-o", "rss=", "-p", &std::process::id().to_string()])
|
||||
.output()
|
||||
.expect("ps");
|
||||
String::from_utf8_lossy(&out.stdout)
|
||||
.trim()
|
||||
.parse::<f64>()
|
||||
.unwrap_or(0.0)
|
||||
/ 1024.0
|
||||
}
|
||||
|
||||
/// Sample `f` up to `reps` times, stopping once `budget_ms` is spent (one sample minimum).
|
||||
fn sample(name: &str, reps: usize, budget_ms: f64, mut f: impl FnMut(usize)) {
|
||||
let mut v = Vec::new();
|
||||
let start = Instant::now();
|
||||
for r in 0..reps {
|
||||
let t = Instant::now();
|
||||
f(r);
|
||||
v.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
if start.elapsed().as_secs_f64() * 1000.0 > budget_ms {
|
||||
break;
|
||||
}
|
||||
}
|
||||
v.sort_by(|a, b| a.partial_cmp(b).expect("finite"));
|
||||
println!(
|
||||
"{name} min={:.3} med={:.3} max={:.3} reps={}",
|
||||
v[0],
|
||||
v[v.len() / 2],
|
||||
v[v.len() - 1],
|
||||
v.len()
|
||||
);
|
||||
std::io::stdout().flush().ok();
|
||||
}
|
||||
|
||||
fn once<T>(name: &str, f: impl FnOnce() -> T) -> T {
|
||||
let t = Instant::now();
|
||||
let v = f();
|
||||
println!(
|
||||
"{name} min={0:.3} med={0:.3} max={0:.3} reps=1",
|
||||
t.elapsed().as_secs_f64() * 1000.0
|
||||
);
|
||||
std::io::stdout().flush().ok();
|
||||
v
|
||||
}
|
||||
|
||||
fn key(code: KeyCode) -> KeyEvent {
|
||||
KeyEvent::new(code, KeyModifiers::NONE)
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let args: HashMap<String, String> = std::env::args()
|
||||
.skip(1)
|
||||
.filter_map(|a| {
|
||||
a.split_once('=')
|
||||
.map(|(k, v)| (k.to_string(), v.to_string()))
|
||||
})
|
||||
.collect();
|
||||
let c = Cfg {
|
||||
k: arg(&args, "k", "1000").parse().expect("k"),
|
||||
n: arg(&args, "n", "100000").parse().expect("n"),
|
||||
cats: arg(&args, "cats", "3").parse().expect("cats"),
|
||||
prune: arg(&args, "prune", "1") == "1",
|
||||
mode: arg(&args, "mode", "grid"),
|
||||
groups: arg(&args, "groups", "0") == "1",
|
||||
ops: arg(
|
||||
&args,
|
||||
"ops",
|
||||
"open,key,frame,window,setcell,host,reshape,io",
|
||||
),
|
||||
};
|
||||
let rows_cap: f64 = arg(&args, "rows_cap", "30000000").parse().expect("cap");
|
||||
let product = (c.k as f64).powi(c.cats as i32);
|
||||
if c.mode == "grid" && product > rows_cap {
|
||||
println!(
|
||||
"REFUSED product_rows={product:.3e} (> {rows_cap:.0}: layout would materialize it)"
|
||||
);
|
||||
return;
|
||||
}
|
||||
let want = |o: &str| c.ops.split(',').any(|x| x == o);
|
||||
let (wb, recs) = document(&c);
|
||||
println!(
|
||||
"cells={} records={} base_rss_mb={:.0}",
|
||||
wb.model.data.iter_cells().count(),
|
||||
recs.len(),
|
||||
rss_mb()
|
||||
);
|
||||
let mut app = once("open", || App::new(wb, None));
|
||||
println!(
|
||||
"rows={} cols={} rss_after_open_mb={:.0}",
|
||||
app.layout.row_count(),
|
||||
app.layout.col_count(),
|
||||
rss_mb()
|
||||
);
|
||||
app.term_width = 200;
|
||||
app.term_height = 50;
|
||||
let name = app.view_name().to_string();
|
||||
|
||||
if want("key") {
|
||||
sample("keypress_down", 8, 6000.0, |_| {
|
||||
app.dispatch_key(key(KeyCode::Down));
|
||||
});
|
||||
}
|
||||
if want("frame") {
|
||||
let whole = |app: &App| Viewport {
|
||||
row_offset: 0,
|
||||
col_offset: 0,
|
||||
rows: app.layout.row_count(),
|
||||
cols: app.layout.col_count(),
|
||||
};
|
||||
sample("tui_frame_grid_of_whole", 3, 8000.0, |_| {
|
||||
std::hint::black_box(project::grid_of(&app, whole(&app)));
|
||||
});
|
||||
let cache = project::grid_of(&app, whole(&app));
|
||||
let area = ratatui::layout::Rect::new(0, 0, 200, 50);
|
||||
sample("tui_render_gridwidget_buffer", 5, 4000.0, |_| {
|
||||
let mut buf = ratatui::buffer::Buffer::empty(area);
|
||||
let w = improvise::ui::grid::GridWidget::new(
|
||||
&cache,
|
||||
app.cursor(),
|
||||
&app.view_state.active_view,
|
||||
&app.layout,
|
||||
&app.view_state.mode,
|
||||
&app.view_state.search_query,
|
||||
&app.view_state.buffers,
|
||||
app.drill_state.as_ref(),
|
||||
);
|
||||
ratatui::widgets::Widget::render(w, area, &mut buf);
|
||||
std::hint::black_box(buf);
|
||||
});
|
||||
}
|
||||
if want("window") {
|
||||
let mid = app.layout.row_count() / 2;
|
||||
for (label, off) in [("top", 0), ("mid", mid)] {
|
||||
let vp = Viewport {
|
||||
row_offset: off,
|
||||
col_offset: 0,
|
||||
rows: 45,
|
||||
cols: 12,
|
||||
};
|
||||
sample(&format!("browser_grid_of_{label}"), 5, 6000.0, |_| {
|
||||
std::hint::black_box(project::grid_of(&app, vp));
|
||||
});
|
||||
}
|
||||
}
|
||||
if want("setcell") && !recs.is_empty() {
|
||||
let existing = recs[recs.len() / 2].clone().with("_Measure", "Rev");
|
||||
sample("setcell_existing", 5, 6000.0, |r| {
|
||||
app.apply_effects(vec![Box::new(effect::SetCell(
|
||||
existing.clone(),
|
||||
CellValue::Number(10.0 + r as f64),
|
||||
))]);
|
||||
});
|
||||
let mut s = 99u64;
|
||||
sample("setcell_new_record", 5, 6000.0, |_| {
|
||||
let mut coords: Vec<(String, String)> = (0..c.cats)
|
||||
.map(|ci| (format!("C{ci}"), format!("i{}", lcg(&mut s, c.k))))
|
||||
.collect();
|
||||
coords.push(("_Measure".into(), "Rev".into()));
|
||||
app.apply_effects(vec![Box::new(effect::SetCell(
|
||||
CellKey::new(coords),
|
||||
CellValue::Number(1.0),
|
||||
))]);
|
||||
});
|
||||
}
|
||||
if want("reshape") && c.mode == "grid" {
|
||||
let v = || ViewTarget::Active;
|
||||
sample("reshape_hide_item", 3, 6000.0, |r| {
|
||||
app.apply_effects(vec![Box::new(effect::HideItem {
|
||||
view: v(),
|
||||
category: "C1".into(),
|
||||
item: format!("i{r}"),
|
||||
})]);
|
||||
});
|
||||
if c.groups {
|
||||
sample("reshape_toggle_group", 3, 6000.0, |_| {
|
||||
app.apply_effects(vec![Box::new(effect::ToggleGroup {
|
||||
view: v(),
|
||||
category: "C0".into(),
|
||||
group: "G0".into(),
|
||||
})]);
|
||||
});
|
||||
}
|
||||
let last = format!("C{}", c.cats - 1);
|
||||
sample("reshape_set_axis", 4, 6000.0, |r| {
|
||||
let axis = if r % 2 == 0 { Axis::Page } else { Axis::Row };
|
||||
app.apply_effects(vec![Box::new(effect::SetAxis {
|
||||
view: v(),
|
||||
category: last.clone(),
|
||||
axis,
|
||||
})]);
|
||||
});
|
||||
sample("reshape_toggle_prune", 2, 6000.0, |_| {
|
||||
app.apply_effects(vec![Box::new(effect::TogglePruneEmpty(v()))]);
|
||||
});
|
||||
}
|
||||
if want("io") {
|
||||
let wbr = &app.model_state.workbook;
|
||||
let text = once("format_md", || format_md(wbr));
|
||||
println!("improv_mb={:.2}", text.len() as f64 / 1048576.0);
|
||||
once("parse_md", || parse_md(&text).expect("parses"));
|
||||
let fs = MemoryFs::new();
|
||||
sample("persist_per_command_save_to", 3, 6000.0, |_| {
|
||||
save_to(&fs, wbr, std::path::Path::new("d.improv")).expect("write");
|
||||
});
|
||||
}
|
||||
if want("host") && !recs.is_empty() {
|
||||
let mut host = Host::new(app);
|
||||
let (id, snap) = once("host_connect_snapshot", || host.connect(45, 12));
|
||||
sample("json_encode_snapshot", 3, 4000.0, |_| {
|
||||
std::hint::black_box(serde_json::to_vec(&snap.msg).expect("json"));
|
||||
});
|
||||
println!(
|
||||
"snapshot_kb={:.1}",
|
||||
serde_json::to_vec(&snap.msg).expect("json").len() as f64 / 1024.0
|
||||
);
|
||||
let rec = recs[recs.len() / 2].clone();
|
||||
let cell = RecordCell {
|
||||
record: RecordHash::of(&rec),
|
||||
measure: Some("Rev".into()),
|
||||
};
|
||||
let mut enc: Vec<(f64, usize)> = Vec::new();
|
||||
sample("host_handle_setcell_and_project", 4, 8000.0, |r| {
|
||||
let wire = caught_up(
|
||||
host.app(),
|
||||
ModelCommand::SetCell {
|
||||
view: name.clone(),
|
||||
cell: cell.clone(),
|
||||
value: CellValue::Number(50.0 + r as f64),
|
||||
},
|
||||
);
|
||||
let out = host.handle(id, ClientFrame::command(wire));
|
||||
let t = Instant::now();
|
||||
let bytes: usize = out
|
||||
.iter()
|
||||
.map(|a| serde_json::to_vec(&a.msg).expect("json").len())
|
||||
.sum();
|
||||
enc.push((t.elapsed().as_secs_f64() * 1000.0, bytes));
|
||||
});
|
||||
println!("json_encode_setcell_updates ms={:?}", enc);
|
||||
}
|
||||
println!("maxrss_mb={:.0}", maxrss_mb());
|
||||
}
|
||||
+283
@@ -0,0 +1,283 @@
|
||||
# Improvise — Multi-Dimensional Data Modeling Terminal Application
|
||||
|
||||
## Context
|
||||
|
||||
Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque
|
||||
cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without
|
||||
rewriting formulas. We are building a terminal application that treats data as a multi-dimensional,
|
||||
semantically labeled structure — separating data, computation, and views into independent layers. The result
|
||||
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
|
||||
can be explored from multiple perspectives simultaneously.
|
||||
|
||||
The application compiles to a single static binary (`x86_64-unknown-linux-musl`) and provides a rich TUI
|
||||
experience.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Data Model
|
||||
|
||||
### 1.1 Categories and Items
|
||||
- Data is organized into **categories** (dimensions) and **items** (members of a dimension).
|
||||
- Example: Category "Region" contains items "North", "South", "East", "West".
|
||||
- Example: Category "Time" contains items "Q1", "Q2", "Q3", "Q4".
|
||||
- Items within a category can be organized into **groups** forming a hierarchy.
|
||||
- Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
|
||||
- Groups are collapsible/expandable for drill-down.
|
||||
- A model supports up to **12 categories**.
|
||||
|
||||
### 1.2 Data Cells
|
||||
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
|
||||
- Cells hold numeric values, text, or empty/null.
|
||||
- The underlying storage is a sparse multi-dimensional array (`HashMap<CellKey, CellValue>`).
|
||||
|
||||
### 1.3 Models
|
||||
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
|
||||
- Models are saved to and loaded from a single `.improv` file (JSON format).
|
||||
|
||||
---
|
||||
|
||||
## 2. Formula System
|
||||
|
||||
### 2.1 Named Formulas
|
||||
- Formulas reference categories and items by name, not by cell address.
|
||||
- Example: `Profit = Revenue - Cost`
|
||||
- Example: `Tax = Revenue * 0.08`
|
||||
- Example: `Margin = Profit / Revenue`
|
||||
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.
|
||||
|
||||
### 2.2 Formula Panel
|
||||
- Formulas are defined in a **dedicated formula panel**, separate from the data grid.
|
||||
- All formulas are visible in one place for easy auditing.
|
||||
- Formulas cannot be accidentally overwritten by data entry.
|
||||
|
||||
### 2.3 Scoped Formulas (WHERE clause)
|
||||
- A formula can be scoped to a subset of items:
|
||||
- Example: `Discount = 0.10 * Price WHERE Region = "West"`
|
||||
|
||||
### 2.4 Aggregation
|
||||
- Built-in aggregation functions: `SUM`, `AVG`, `MIN`, `MAX`, `COUNT`.
|
||||
|
||||
### 2.5 Formula Language
|
||||
- Expression-based (not Turing-complete).
|
||||
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
|
||||
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
|
||||
- Conditionals: `IF(condition, then, else)`.
|
||||
- `WHERE` clause for filtering: `SUM(Sales WHERE Region = "East")`.
|
||||
- Parentheses for grouping.
|
||||
- Literal numbers and quoted strings.
|
||||
|
||||
---
|
||||
|
||||
## 3. View System
|
||||
|
||||
### 3.1 Views as First-Class Objects
|
||||
- A **view** is a named configuration specifying:
|
||||
- Which categories are assigned to **rows**, **columns**, and **pages** (filters/slicers).
|
||||
- Which items/groups are visible vs. hidden.
|
||||
- Sort order (future).
|
||||
- Number formatting.
|
||||
- Multiple views can exist per model. Each is independent.
|
||||
- Editing data in any view updates the underlying model; all other views reflect the change.
|
||||
|
||||
### 3.2 Category Tiles
|
||||
- Each category is represented as a **tile** displayed in the tile bar.
|
||||
- The user can move tiles between row, column, and page axes to instantly pivot/rearrange the view.
|
||||
- Moving a tile triggers an instant recalculation and re-render of the grid.
|
||||
|
||||
### 3.3 Page Axis (Slicing)
|
||||
- Categories assigned to the page axis act as filters.
|
||||
- The user selects a single item from a paged category using `[` and `]`.
|
||||
|
||||
### 3.4 Collapsing and Expanding
|
||||
- Groups can be collapsed/expanded per-view (future: keyboard shortcut in grid).
|
||||
|
||||
---
|
||||
|
||||
## 4. JSON Import Wizard
|
||||
|
||||
### 4.1 Purpose
|
||||
- Users can import arbitrary JSON files to bootstrap a model.
|
||||
|
||||
### 4.2 Wizard Flow (interactive TUI)
|
||||
|
||||
**Step 1: Preview** — Structural summary of the JSON.
|
||||
|
||||
**Step 2: Select Array Path** — If the JSON is not a flat array, the user selects which key path contains the primary record array.
|
||||
|
||||
**Step 3: Review Proposals** — Fields are analyzed and proposed as:
|
||||
- Category (small number of distinct string values)
|
||||
- Measure (numeric)
|
||||
- Time Category (date-like strings)
|
||||
- Label/Identifier (skip)
|
||||
|
||||
**Step 4: Name the Model** — User names the model and confirms.
|
||||
|
||||
### 4.3 Headless Import
|
||||
```
|
||||
improvise --cmd '{"op":"ImportJson","path":"data.json"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Terminal UI
|
||||
|
||||
### 5.1 Layout
|
||||
```
|
||||
+---------------------------------------------------------------+
|
||||
| Improvise | Model: Sales 2025 [*] [F1 Help] [Ctrl+Q] |
|
||||
+---------------------------------------------------------------+
|
||||
| [Page: Region = East] |
|
||||
| | Q1 | Q2 | Q3 | Q4 | |
|
||||
|--------------+---------+---------+---------+---------+--------|
|
||||
| Shirts | 1,200 | 1,450 | 1,100 | 1,800 | |
|
||||
| Pants | 800 | 920 | 750 | 1,200 | |
|
||||
| ... |
|
||||
|--------------+---------+---------+---------+---------+--------|
|
||||
| Total | 4,100 | 4,670 | 3,750 | 5,800 | |
|
||||
+---------------------------------------------------------------+
|
||||
| Tiles: [Time ↔] [Product ↕] [Region ☰] Ctrl+↑↓←→ tiles |
|
||||
+---------------------------------------------------------------+
|
||||
| NORMAL | Default | Ctrl+F:formulas Ctrl+C:categories ... |
|
||||
+---------------------------------------------------------------+
|
||||
```
|
||||
|
||||
### 5.2 Panels
|
||||
- **Grid panel** (main): Scrollable table of the current view.
|
||||
- **Tile bar**: Category tiles with axis symbols. `Ctrl+Arrow` enters tile-select mode.
|
||||
- **Formula panel**: `Ctrl+F` — list and edit formulas.
|
||||
- **Category panel**: `Ctrl+C` — manage categories and axis assignments.
|
||||
- **View panel**: `Ctrl+V` — switch, create, delete views.
|
||||
- **Status bar**: Mode, active view name, keyboard hints.
|
||||
|
||||
### 5.3 Navigation and Editing
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| ↑↓←→ / hjkl | Move cursor |
|
||||
| Enter | Edit cell |
|
||||
| Esc | Cancel edit |
|
||||
| Tab | Focus next open panel |
|
||||
| / | Search |
|
||||
| [ / ] | Page axis prev/next |
|
||||
| Ctrl+Arrow | Tile select mode |
|
||||
| Enter/Space (tile) | Cycle axis (Row→Col→Page) |
|
||||
| r / c / p (tile) | Set axis directly |
|
||||
| Ctrl+F | Toggle formula panel |
|
||||
| Ctrl+C | Toggle category panel |
|
||||
| Ctrl+V | Toggle view panel |
|
||||
| Ctrl+S | Save |
|
||||
| Ctrl+E | Export CSV |
|
||||
| F1 | Help |
|
||||
| Ctrl+Q | Quit |
|
||||
|
||||
---
|
||||
|
||||
## 6. Command Layer (Headless Mode)
|
||||
|
||||
All model mutations go through a typed command layer. This enables:
|
||||
- Scripting without the TUI
|
||||
- Replay / audit log
|
||||
- Testing without rendering
|
||||
|
||||
### 6.1 Command Format
|
||||
JSON object with an `op` field:
|
||||
```json
|
||||
{"op": "CommandName", ...args}
|
||||
```
|
||||
|
||||
### 6.2 Available Commands
|
||||
|
||||
| op | Required fields | Description |
|
||||
|----|-----------------|-------------|
|
||||
| `AddCategory` | `name` | Add a category/dimension |
|
||||
| `AddItem` | `category`, `item` | Add an item to a category |
|
||||
| `AddItemInGroup` | `category`, `item`, `group` | Add an item in a named group |
|
||||
| `SetCell` | `coords: [[cat,item],...]`, `number` or `text` | Set a cell value |
|
||||
| `ClearCell` | `coords` | Clear a cell |
|
||||
| `AddFormula` | `raw`, `target_category` | Add/replace a formula |
|
||||
| `RemoveFormula` | `target` | Remove a formula by target name |
|
||||
| `CreateView` | `name` | Create a new view |
|
||||
| `DeleteView` | `name` | Delete a view |
|
||||
| `SwitchView` | `name` | Switch the active view |
|
||||
| `SetAxis` | `category`, `axis` (`"row"/"column"/"page"`) | Set category axis |
|
||||
| `SetPageSelection` | `category`, `item` | Set page-axis filter |
|
||||
| `ToggleGroup` | `category`, `group` | Toggle group collapse |
|
||||
| `Save` | `path` | Save model to file |
|
||||
| `Load` | `path` | Load model from file |
|
||||
| `ExportCsv` | `path` | Export active view to CSV |
|
||||
| `ImportJson` | `path`, `model_name?`, `array_path?` | Import JSON file |
|
||||
|
||||
### 6.3 Response Format
|
||||
```json
|
||||
{"ok": true, "message": "optional message"}
|
||||
{"ok": false, "message": "error description"}
|
||||
```
|
||||
|
||||
### 6.4 Invocation
|
||||
```bash
|
||||
# Single command
|
||||
improvise model.improv --cmd '{"op":"SetCell","coords":[["Region","East"],["Measure","Revenue"]],"number":1200}'
|
||||
|
||||
# Script file (one JSON object per line, # comments allowed)
|
||||
improvise model.improv --script setup.jsonl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Persistence
|
||||
|
||||
### 7.1 File Format
|
||||
Native format: JSON-based `.improv` file containing all categories, items, groups, data cells, formulas, and view definitions.
|
||||
|
||||
Compressed variant: `.improv.gz` (gzip, same JSON payload).
|
||||
|
||||
### 7.2 Export
|
||||
- `Ctrl+E` in TUI or `ExportCsv` command: exports active view to CSV.
|
||||
|
||||
### 7.3 Autosave
|
||||
- Periodic autosave (every 30 seconds when dirty) to `.model.improv.autosave`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Technology
|
||||
|
||||
| Concern | Choice |
|
||||
|---------|--------|
|
||||
| Language | Rust (stable) |
|
||||
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
|
||||
| Serialization | `serde` + `serde_json` |
|
||||
| Static binary | `x86_64-unknown-linux-musl` via `musl-gcc` |
|
||||
| Dev environment | Nix flake with `rust-overlay` |
|
||||
| No runtime deps | Single binary, no database, no network |
|
||||
|
||||
---
|
||||
|
||||
## 9. Non-Goals (v1)
|
||||
|
||||
- Scripting/macro language beyond the formula system.
|
||||
- Collaborative/multi-user editing.
|
||||
- Live external data sources (databases, APIs).
|
||||
- Charts or graphical visualization.
|
||||
- Multi-level undo history.
|
||||
|
||||
---
|
||||
|
||||
## 10. Verification
|
||||
|
||||
```bash
|
||||
# Build
|
||||
nix develop --command cargo build --release
|
||||
file target/x86_64-unknown-linux-musl/release/improvise # → statically linked
|
||||
|
||||
# Import test
|
||||
./improvise --cmd '{"op":"ImportJson","path":"sample.json"}' --cmd '{"op":"Save","path":"test.improv"}'
|
||||
|
||||
# Formula test
|
||||
./improvise test.improv \
|
||||
--cmd '{"op":"AddFormula","raw":"Profit = Revenue - Cost","target_category":"Measure"}'
|
||||
|
||||
# Headless script
|
||||
./improvise new.improv --script tests/setup.jsonl
|
||||
|
||||
# TUI
|
||||
./improvise model.improv
|
||||
```
|
||||
@@ -1,291 +0,0 @@
|
||||
# ADR 001 — Concurrent edits between clients
|
||||
|
||||
**Status:** accepted (2026-09-19); **layer 1 implemented** (improvise-xwi6/fkfg, 2026-09-19) · **Supersedes:** nothing · **Decided by:** the user
|
||||
|
||||
## Context
|
||||
|
||||
`improvise-cqq` made the client/server split real: one authoritative `App`, a
|
||||
`ClientSession` mirror per client, and projections diffed against those mirrors.
|
||||
`improvise-kmgs` then put the 28 model mutations on the wire. Two clients can now
|
||||
edit the same document, and nothing decides what happens when they edit the same
|
||||
part of it.
|
||||
|
||||
### There is already a total order
|
||||
|
||||
Every `ModelCommand` reaches the document through one path:
|
||||
`Sessions::apply` → `apply_command` → `App::apply_effects`, against a single `App`.
|
||||
So the server already serialises writes. **We do not need to build linear history —
|
||||
it exists.**
|
||||
|
||||
What we lack is a way to notice that a client composed an edit against a picture of
|
||||
the document that has since moved. This is a *stale basis* problem, not an ordering
|
||||
problem — the same shape as the stale cursor (gotcha 24) and the stale session
|
||||
mirror after a document swap (gotcha 39), one level up.
|
||||
|
||||
### Scope: conflicts range over one document
|
||||
|
||||
improvise is a **multi-document** app, and the server is expected to host several
|
||||
separate models. Sessions editing *independent* models are trivially
|
||||
non-conflicting, so everything below scopes to a single document.
|
||||
|
||||
That falls out of the existing shape rather than needing to be built: `Sessions`
|
||||
does not own the `App` — `connect`, `apply` and `project_all` all take it as a
|
||||
parameter — so a `Sessions` is already bound to one document's generation sequence,
|
||||
and multi-document hosting is a registry *above* it mapping a document to its
|
||||
`(App, Sessions)` pair. The version tiers this ADR introduces live on the `App`, so
|
||||
they scope per document for free.
|
||||
|
||||
Multi-document hosting **landed** as `improvise-ws-server::registry` (improvise-j31x): a
|
||||
`Registry` of `Host`s keyed by `DocumentId`, addressed by `ClientId { document, session }`
|
||||
because a `SessionId` is minted per `Sessions` and so is unique only within its document.
|
||||
The version tiers scope per document as predicted. Note one lifecycle rule that exists to
|
||||
protect *this* ADR: a document is never evicted while the server runs, because
|
||||
`document_generation` and `record_versions` start at zero in a fresh `App`, so re-opening
|
||||
would accept a reconnecting client's stale basis as current.
|
||||
|
||||
### What actually conflicts
|
||||
|
||||
1. **Two writers, same record.** Survivable under last-write-wins: someone's number
|
||||
is lost, the document stays coherent.
|
||||
2. **Shape change vs. cell edit.** A removes category `Region` while B commits an
|
||||
edit keyed by `Region=East`. B's `CellKey` names a coordinate that no longer
|
||||
exists. This is *corruption*, not a lost keystroke, and it is the class a
|
||||
cell-level lock does not cover — holding `Region=East × Revenue` does not stop
|
||||
anyone deleting `Region`.
|
||||
3. **Formula vs. data.** A adds a formula targeting the measure B is typing into;
|
||||
B's raw value is shadowed by a derived one.
|
||||
|
||||
### Machinery that already exists
|
||||
|
||||
- `App::document_generation` — bumped by `install_workbook`, consumed by
|
||||
`ClientSession::project` to reseed a stale mirror (gotcha 39).
|
||||
- `ModelChange::{CellData, LayoutGeometry, WholeWorkbook}` — the effect taxonomy,
|
||||
already the single classification for what a mutation reaches (gotcha 35).
|
||||
- `record_row_identity(wide, coord_group_excluding_measure)` — a *record* is the
|
||||
coordinate tuple excluding `_Measure` (gotcha 1).
|
||||
- Drill staging — `pending_edits` applied atomically by `ApplyAndClearDrill`, and
|
||||
ruled **private to whoever staged them** (improvise-tom1).
|
||||
|
||||
### Machinery that did *not* exist, and the hole it left
|
||||
|
||||
**Nothing in the projection stream carried any of the three tiers.** This ADR
|
||||
says a batch carries the basis it was composed against and never says how a
|
||||
client comes to *know* one — so the check as originally specified was
|
||||
satisfiable only by a client **forging** a basis. That is the invariant deleted
|
||||
with extra steps, arriving through the front door rather than as a `force` flag.
|
||||
|
||||
Closed during implementation (improvise-fkfg) by `Projection::Basis` +
|
||||
`BasisCache`: the server publishes the two document tiers plus the write
|
||||
counters for the records **this client's grid names** — exactly the set it can
|
||||
address a write to, per ADR 002 — diffed against what that client was last told
|
||||
and seeded in the bootstrap. It is emitted **after** the grid updates in a
|
||||
batch, because the counters describe the cells those updates carry: a basis
|
||||
arriving first would let a client claim a version for a value it had not been
|
||||
shown. That is the mirror image of gotcha 33.
|
||||
|
||||
Read this as step (a′) of the implementation order below: **publication comes
|
||||
before the check, because the check is unsatisfiable without it.**
|
||||
|
||||
## Decision
|
||||
|
||||
**Record-level atomicity with optimistic concurrency per batch.**
|
||||
|
||||
1. **The unit of atomicity is the record**, i.e. the coordinate tuple excluding
|
||||
`_Measure` — the thing `record_row_identity` already names. Not the cell:
|
||||
gotcha 1 says editing a category column in records mode moves *every* measure
|
||||
cell in the row, so a cell-level unit would be violated by a single legal edit.
|
||||
The unit must be at least as coarse as the coarsest legal write.
|
||||
|
||||
2. **The unit of commit is the batch** — one `Vec<Box<dyn Effect>>` applied by one
|
||||
`apply_effects`. This is already atomic in practice: a `Binding::Sequence`
|
||||
flattens into one batch, and a failed `Save` aborts the rest (gotcha 21).
|
||||
|
||||
3. **A batch carries the basis it was composed against**, and the server rejects it
|
||||
if that basis moved. The basis is **tiered by `ModelChange`**, which is what
|
||||
makes class 2 fall out for free:
|
||||
|
||||
| tier | bumped by | a batch is stale if |
|
||||
|---|---|---|
|
||||
| `document_generation` | `WholeWorkbook` | it moved at all |
|
||||
| `layout_version` | `LayoutGeometry` | it moved at all |
|
||||
| `record_version` | `CellData` on that record | any record the batch touches moved |
|
||||
|
||||
A `CellData` batch is stale if *layout* moved, because the shape its keys hang
|
||||
off changed. That is class 2, detected without a lock.
|
||||
|
||||
4. **Rejection is a normal outcome, not an error.** The client already receives the
|
||||
new state through projections; it re-composes against the new basis. `Applied`
|
||||
grows a third case alongside `Diff` and `Rebased` — matching is what forces a
|
||||
host to handle it (gotcha 39's reasoning).
|
||||
|
||||
### Rejection must be forceable, and force means re-compose
|
||||
|
||||
A rejected edit has to give the user a way through. But **force is a re-composition,
|
||||
not a bypass** — it never applies a batch whose basis is stale. A mechanism whose
|
||||
only purpose is to violate the invariant is not a feature, it is the invariant
|
||||
deleted with extra steps. Force re-composes the same *intent* against the current
|
||||
state and submits a fresh batch with a current basis.
|
||||
|
||||
That distinction is not pedantic, because what force can mean differs by class:
|
||||
|
||||
| class | can force? | what force does |
|
||||
|---|---|---|
|
||||
| 1 — two writers, same record | **yes, trivially** | re-compose and apply; the user has seen their value and chooses to overwrite it |
|
||||
| 2 — shape change vs. cell edit | **no** | the intent no longer type-checks: the key names a coordinate that no longer exists |
|
||||
| 3 — formula shadows a value | yes, with a warning | the write lands; the displayed value is still derived |
|
||||
| 4 — the batch names a record its basis says nothing about | **no — and it is not a race** | nothing to re-compose *against*; the client was never in the running |
|
||||
|
||||
Class 4 was not in the original draft and was added by the implementation
|
||||
(`Rejection::RecordUnstated`). It exists because **"any record the batch
|
||||
touches" had to name whose list**, and the answer is the *server's*: reading it
|
||||
off the client's claim would be a **force by omission** — a client that simply
|
||||
omits a record skips the record tier, with no flag to remove and nothing in the
|
||||
message saying so. `reduce.rs::records_named` is therefore an exhaustive read of
|
||||
the `ModelCommand`, with one or-pattern for the variants that name no record, so
|
||||
a new record-writing variant fails the build. A well-formed client cannot
|
||||
produce class 4; it is a case of its own rather than folded into class 1 because
|
||||
"you lost a race" and "you were never in one" are different things to tell a
|
||||
user.
|
||||
|
||||
For class 1, force is one keystroke and should be. For class 2 there is nothing
|
||||
coherent to force: applying the batch anyway writes a `CellKey` naming a deleted
|
||||
category, which is the ghost-coordinate corruption this ADR exists to prevent. So:
|
||||
|
||||
- **Force never creates structure.** Re-instating a category someone deleted so an
|
||||
edit can land is a *different and larger* intent than the user expressed, and it
|
||||
must be an explicit action, not a side effect of forcing a cell edit.
|
||||
- When an intent cannot be re-composed, the UI owes the user **what changed and the
|
||||
concrete alternatives** — re-target the edit, or re-create the structure first —
|
||||
rather than a refusal.
|
||||
|
||||
Two consequences that pull work forward from layer 2:
|
||||
|
||||
- **The rejected batch's values stay staged in the session**, so rejection never
|
||||
costs the user their typing. This is the drill staging mechanism doing the job it
|
||||
already does, and it is what makes force cheap rather than a retype. It belongs in
|
||||
layer 1 even though *publishing* the staged keys is layer 2.
|
||||
- **A forced overwrite must be visible to the client it overwrote.** Otherwise force
|
||||
is silent data loss with a confirmation step in front of it. The loser's own edit
|
||||
was already committed, so this is a notification, not a conflict — but it must
|
||||
exist, or "someone else won" becomes indistinguishable from "my edit never
|
||||
happened."
|
||||
|
||||
### Optional second layer: advisory reservations
|
||||
|
||||
The staged-edit mechanism from drill, generalised: a client publishes the *keys* it
|
||||
has staged so others can grey them out and warn early. The *values* stay private,
|
||||
per improvise-tom1.
|
||||
|
||||
This layer is **advisory only**. If a reservation is lost, never arrives, or leaks
|
||||
from a crashed client, the system degrades to "you find out at commit" — never to
|
||||
corruption. That is what makes it safe to be sloppy about timeouts and disconnects,
|
||||
and it is why correctness must not be built on it.
|
||||
|
||||
## Why not a causal clock
|
||||
|
||||
Considered and rejected, for four reasons.
|
||||
|
||||
1. **The server already serialises.** Causal clocks earn their keep where no single
|
||||
node orders writes — peer-to-peer, or offline-capable clients merging on
|
||||
reconnect. With an authoritative `App` we need *staleness detection* (one
|
||||
counter per unit), not *concurrency detection* (a vector per client).
|
||||
2. **A Lamport clock is insufficient.** It gives a total order but cannot tell you
|
||||
two edits were concurrent, which is exactly the question.
|
||||
3. **A vector clock is sufficient but does not answer the question that matters.**
|
||||
It costs O(clients), needs entries reaped for departed clients, and when it
|
||||
reports "these were concurrent" you still need a policy for class 2 — no clock
|
||||
says whether B's edit into `Region=East` should survive A's deletion of `Region`.
|
||||
The clock tells you there is a decision to make; it does not make it.
|
||||
4. **CRDTs converge, but not necessarily to something meaningful.** A LWW-register
|
||||
per cell is easy; the hard part is the *shape* — categories, items, formulas,
|
||||
axis assignments — where convergence can mean an orphaned cell or a resurrected
|
||||
category.
|
||||
|
||||
If offline editing ever becomes a goal, this decision should be revisited: that is
|
||||
the requirement that makes a vector clock worth its cost, and it is a much larger
|
||||
commitment than multi-client-while-connected.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good.** No lock to acquire, so no timeout, no lease renewal, no disconnect
|
||||
handling, and no liveness risk — a crashed client cannot wedge anyone. Correctness
|
||||
does not depend on the advisory layer. The tiered basis reuses `ModelChange`, so
|
||||
there is no second classification to drift from gotcha 35's table. Class 2 is
|
||||
detected rather than prevented, which is cheaper and does not require predicting
|
||||
which shape changes matter.
|
||||
|
||||
**Bad.** A client can compose an edit and be told no, and the UI owes real work
|
||||
there: keep the typing, show what moved, and offer force where force is coherent.
|
||||
Class 2 is the expensive case — the honest answer is a small dialogue about what
|
||||
changed, not a button. Record-level granularity means two clients
|
||||
editing *different measures of the same record* serialise against each other even
|
||||
though their writes do not overlap — acceptable at the expected scale, and
|
||||
refinable to `(record, measure)` if contention shows up. A busy `LayoutGeometry`
|
||||
editor invalidates every in-flight `CellData` batch, which is correct but could feel
|
||||
aggressive.
|
||||
|
||||
**Not solved here.** Class 3 (formula shadows a raw value) is detected — the formula
|
||||
bumps `layout_version` — but the *policy* is unstated: the edit is rejected, and
|
||||
whether that is the right answer for the user is a separate question. Nor does this
|
||||
address a client that wants to keep editing while disconnected.
|
||||
|
||||
## Implementation order
|
||||
|
||||
Deliberately two tickets, so the correctness floor lands without waiting on the UX.
|
||||
|
||||
1. **Basis check.** Version tiers, `ModelCommand` carries its basis, `Applied` grows
|
||||
the rejected case. Includes the decision of where the version counters live —
|
||||
they are shared document state, so they belong beside `document_generation` on
|
||||
the authoritative `App`, not on `ViewState`.
|
||||
2. **Advisory reservations.** Generalise drill staging; publish key sets, keep values
|
||||
private.
|
||||
|
||||
The nullary wire variants `ApplyAndClearDrill` and `AcceptWizardModel` were to
|
||||
be resolved first or alongside. **Already done**: improvise-3x3u removed both
|
||||
from `ModelCommand` entirely, so they have no wire form to carry a basis or a
|
||||
`SessionId`, and layer 1 landed honestly without them (improvise-k2i4 closed as
|
||||
stale). What survives is the *addressed* form multi-client drill will want,
|
||||
which is layer 2 work.
|
||||
|
||||
### What the check is not
|
||||
|
||||
It is an **anti-staleness coordination check, not a security boundary.** `Basis`
|
||||
derives `Deserialize` because it must cross the wire, and every field is public,
|
||||
so any value at all can arrive as a claim — versions start at 0 and increment by
|
||||
1, so guessing one is trivial. `Rejection::RecordUnstated` closes **omission** —
|
||||
a client that leaves a record out of its claim — which is the bypass that would
|
||||
otherwise sit in the message format with nothing saying so. It does not close
|
||||
**invention**, and nothing here tries to: a hostile client that forges a current
|
||||
basis is indistinguishable from one that is genuinely caught up, and defending
|
||||
against that is authentication's job.
|
||||
|
||||
`BasisCache::claim` being the only constructor is therefore a statement about
|
||||
**clients written against this crate**, not about the wire. It is still worth
|
||||
having: it makes the honest path the easy one, and it is what makes "a client
|
||||
that has folded nothing composes nothing" true by construction.
|
||||
|
||||
### Two places the check deliberately does not reach
|
||||
|
||||
**`Command::SwitchView` carries no basis.** It is `LayoutGeometry` but sits
|
||||
outside `Command::Model`, so `apply_switch` is unchecked. It cannot corrupt —
|
||||
the workbook refuses a view it does not have, and every mirror takes what was
|
||||
*accepted* (gotcha 36) — but it is a hole in "a batch carries its basis" and
|
||||
should be decided rather than left as an accident.
|
||||
|
||||
**`apply_command` does not check at all** (nor did `reduce_full`, which
|
||||
improvise-s17o has since deleted for having no caller), and the *principle* is
|
||||
correct: the check is a property of the **session door**, not of the mutation
|
||||
path, because a host holding the `App` is the truth and not a claimant. That is
|
||||
also why `ui::reduce::caught_up(&App, cmd)` is not a bypass — it takes `&App`,
|
||||
so nothing reachable from a wire can call it.
|
||||
|
||||
As implemented it was a **second door, not an exemption** — both functions took
|
||||
a wire `Command` whose `Command::Model { cmd, .. }` arm discarded the basis with
|
||||
`..`, while `ui/reduce.rs`'s module doc pointed every future host at them by
|
||||
name. **Closed by improvise-ondj** as layer-1 cleanup: both now take a
|
||||
`Checked<'_>`, whose only constructor runs the check against the authoritative
|
||||
`App` and returns a `Rejection`. `reduce_full` was deleted outright shortly after
|
||||
(improvise-s17o): having no caller was the condition that let the defect hide,
|
||||
and a `Checked` signature on a function nobody calls fixes the type and not that. The exemption stands — a host's own
|
||||
`ModelCommand` still carries no basis — but it is no longer reachable as a door,
|
||||
and the unchecked path is unconstructible rather than merely discouraged.
|
||||
@@ -1,234 +0,0 @@
|
||||
# ADR 002 — How a client names a cell
|
||||
|
||||
**Status:** accepted (2026-09-19), extended (2026-09-20) · **Decided by:** the user
|
||||
· **Blocks:** improvise-1ey · **Extended by:** improvise-ir7l, improvise-poia
|
||||
|
||||
## Context
|
||||
|
||||
`improvise-q08` established that a thin client **cannot compose `SetCell`**, and it is
|
||||
the gap that blocks the browser MVP, whose stated scope is "type a number into a cell
|
||||
and see the value update".
|
||||
|
||||
A client sees a grid of *label strings* indexed by viewport position. A `CellKey` is the
|
||||
full sorted coordinate tuple — every Row and Column category, **plus every Page and
|
||||
Filter selection** (gotcha 27). Nothing in `GridCache` says which category a row label
|
||||
belongs to; nested row axes flatten several categories into one label; page and filter
|
||||
constraints are not projected at all; and records mode encodes a *column index* in
|
||||
`_Dim` (gotcha 1). There is no client-side derivation, and projecting the coordinate
|
||||
space would make every client learn conventions the server exists to own.
|
||||
|
||||
## Decision
|
||||
|
||||
**Every projected cell carries an opaque surrogate identity, and the *kind* of identity
|
||||
says what may be done to it.**
|
||||
|
||||
```rust
|
||||
pub struct CellDisplay {
|
||||
pub text: String,
|
||||
pub numeric: bool,
|
||||
pub error: bool,
|
||||
pub id: CellId,
|
||||
}
|
||||
|
||||
pub enum CellId {
|
||||
/// A stored cell. `SetCell` takes this and nothing else.
|
||||
Record(RecordHash),
|
||||
/// An aggregate over stored cells. `Drill` takes this and nothing else.
|
||||
Aggregate(AggregateHash),
|
||||
}
|
||||
```
|
||||
|
||||
An aggregate's surrogate is a hash of its **coordinate constraint** — the row and column
|
||||
coordinates plus the page and filter selections that define which stored cells it sums.
|
||||
One hash per cell, the same width and the same index machinery as a record's, and the
|
||||
server resolves it back and runs `drill_records` when the client asks.
|
||||
|
||||
Three things follow, and each is the reason to prefer this over alternatives.
|
||||
|
||||
**1. Both operations are unrepresentable-if-wrong, not checked.** A client cannot compose
|
||||
`SetCell` for an aggregate, *and* cannot compose `Drill` for a stored cell, because
|
||||
neither carries an id of the other's type. That is design-principles §3 — make invalid
|
||||
states unrepresentable — and it is strictly stronger than a server-side rejection, which
|
||||
would be a runtime answer to a question the type can settle.
|
||||
|
||||
An earlier draft used `Option<CellId>`, absence meaning "aggregate". The enum is better
|
||||
for a reason worth keeping: **an aggregate is not the absence of an identity, it is a
|
||||
different kind of identity.** Modelling it as `None` made `SetCell` safe and left `Drill`
|
||||
with nothing to address — which is precisely how drill addressing came to be an open
|
||||
question. Two variants close it by construction.
|
||||
|
||||
**2. The client never learns the coordinate space.** No category names, no page/filter
|
||||
constraints, no `_Dim` column-index convention. The surrogate is a token to echo, not a
|
||||
structure to understand — so gotchas 1 and 27 stay knowledge the server owns, and a
|
||||
second client implementation cannot get them subtly wrong.
|
||||
|
||||
**3. It mirrors a fork that already exists.** `EditOrDrill` (`cmd/mode.rs`) already
|
||||
branches on exactly this in the TUI: a stored cell is edited, an aggregate is drilled.
|
||||
The wire is not inventing a distinction, it is **projecting one the model already
|
||||
makes** — the same discipline as `decision(improvise-cqi, command_set_derived_not_invented)`.
|
||||
|
||||
It also follows the precedent `CellDisplay::numeric` set, whose doc already says the
|
||||
server decides "because it is the side that knows which it was". Settability is another
|
||||
such fact.
|
||||
|
||||
## Decisions made along the way, and the traps in them
|
||||
|
||||
Most of what started as open questions here got answered. They are kept with their
|
||||
reasoning because each rejected alternative is a thing someone will propose again.
|
||||
|
||||
**What the surrogate is — DECIDED: a hash of the normalized record.** Not of the cell:
|
||||
the record is the coordinate tuple minus `_Measure`, which is already ADR 001's unit of
|
||||
atomicity and already what `record_versions` is keyed by. A cell is then
|
||||
`(record hash, measure)`.
|
||||
|
||||
The reason is that it makes **stale edits cheap to identify**, and it does so three ways
|
||||
at once. The surrogate a client already holds *is* the lookup key for its own version, so
|
||||
`record_versions` becomes `HashMap<RecordHash, u64>` — fixed-size, no allocation, no
|
||||
`CellKey` reconstruction on the hot path. A basis is then a list of
|
||||
`(record hash, version)` pairs, which is compact on the wire and O(1) to check. And a
|
||||
record that no longer exists simply **fails to resolve**, so that whole class of
|
||||
staleness needs no version comparison at all. One identity serves addressing and
|
||||
staleness instead of each ADR carrying its own.
|
||||
|
||||
It is well-defined because `CellKey` already has a canonical form: `CellKey::new` sorts
|
||||
coordinates by category name and equality and hashing rely on it (gotcha 5). "Normalized"
|
||||
means that canonical sorted form with `_Measure` removed — the normalization is a
|
||||
property the type already guarantees, not a new convention to maintain.
|
||||
|
||||
An interned id was rejected: it would need a table with a lifetime nobody has specified,
|
||||
and it buys only collision-freedom, which width buys more cheaply.
|
||||
|
||||
**Three ways this silently breaks, all avoidable.** They are listed because each produces
|
||||
a *wrong write* rather than an error, which is the failure mode this ADR exists to
|
||||
prevent:
|
||||
|
||||
- **Do not use `DefaultHasher` or anything built on `RandomState`.** `std`'s default
|
||||
`HashMap` hasher is randomly seeded **per process**, so hashes would differ across a
|
||||
server restart and between any two processes. Specify a fixed hash (a keyed SipHash
|
||||
with a constant key, or FxHash/xxHash/blake3) and write the choice down.
|
||||
- **Do not use `#[derive(Hash)]` as the definition.** Rust gives no stability guarantee
|
||||
for derived `Hash` across compiler versions or platforms. Hash an **explicit byte
|
||||
serialization** of the sorted `(category, item)` pairs, so the surrogate is defined by
|
||||
the format rather than by whatever the compiler did.
|
||||
- **Pick the width by consequence, not by convenience.** A collision is a silent write to
|
||||
the wrong record — precisely why `_Index` was rejected. 64 bits reaches birthday risk
|
||||
around 2³² records, which a pivot tool will not see; 128 bits costs 8 more bytes per
|
||||
*visible* cell, so on a 50×20 viewport that is 8 KB a frame. The cost is small and the
|
||||
failure is silent, so argue the width explicitly rather than defaulting to `u64` because
|
||||
it is convenient.
|
||||
|
||||
**`_Index` was considered and rejected — right ontology, unsafe representation.** It is
|
||||
the coordinate that identifies a *record*, and a record is exactly the thing that is
|
||||
settable rather than aggregated, so the instinct is sound. But `CategoryKind::VirtualIndex`
|
||||
is documented as *"Items are `"0"`, `"1"`, … N where N = number of matching cells"* — it is
|
||||
**positional over the currently matching set**, not an identity. So `_Index = "3"` names a
|
||||
different record after a filter change, an insert, or a delete, and names a different one
|
||||
again in another view. A client echoing a stale `_Index` back would write to the **wrong
|
||||
cell, silently** — strictly worse than a surrogate that fails to resolve, and the same
|
||||
hazard class as the viewport-relative-vs-absolute trap and the stale cursor under a
|
||||
renamed key.
|
||||
|
||||
What `_Index` is *pointing at* is worth keeping, though: **the surrogate should name the
|
||||
record, not the cell.** A record is the coordinate tuple minus `_Measure`
|
||||
(`record_row_identity`), which is already ADR 001's unit of atomicity and already what
|
||||
`record_versions` is keyed by. If a cell is addressed as `(record surrogate, measure)`,
|
||||
then one identity serves both addressing *and* staleness, and the two ADRs share a
|
||||
mechanism instead of each inventing one.
|
||||
|
||||
**How the server resolves a surrogate back — DECIDED: a maintained index, in
|
||||
`DataStore`.** A hash is one-way, so resolution needs a reverse map, and regenerating it
|
||||
from the layout on demand would be O(visible cells) per resolve *and* would only ever
|
||||
cover the visible window — a client may legitimately hold a surrogate for a record that
|
||||
has scrolled out of view.
|
||||
|
||||
`DataStore` is the right home because it already does exactly this. It interns every
|
||||
category and item name into a `SymbolTable`, and it already maintains a secondary index
|
||||
`HashMap<(Symbol, Symbol), HashSet<InternedKey>>` for hot-path lookups. A
|
||||
`RecordHash → record` index is the same shape, updated at the same points, and the
|
||||
existing `(Symbol, Symbol)` index is what makes invalidation tractable: when a category or
|
||||
item disappears, the records that mention it are findable without a scan.
|
||||
|
||||
Two consequences worth stating, because they fall out and are easy to miss.
|
||||
|
||||
**The index and `record_versions` want to be one structure.** Both are keyed by the record
|
||||
hash and both are invalidated at the same moments, so `RecordHash → (record, version)`
|
||||
gives ADR 001's basis check and this ADR's resolution in a single lookup.
|
||||
|
||||
**Structural and data staleness separate cleanly**, which is better than the
|
||||
regenerate-on-demand shape would have given:
|
||||
|
||||
| incoming surrogate | means | answer |
|
||||
|---|---|---|
|
||||
| does not resolve | the record is **gone** — a category or item it named was removed | reject; there is nothing to write to |
|
||||
| resolves, version matches | the client's basis is current | apply |
|
||||
| resolves, version differs | the record **changed** since the client read it | reject as stale (ADR 001), offer force |
|
||||
|
||||
Regenerating from the layout would have conflated the first two rows, because a record
|
||||
scrolled out of the viewport would have looked identical to a record that no longer
|
||||
exists.
|
||||
|
||||
**Whether a forged or stale surrogate is an error or a rejection.** Under the
|
||||
regenerate-from-layout shape it is simply unresolvable, which suggests the same
|
||||
`Applied` rejection path ADR 001 defines rather than a distinct error.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good.** Unblocks `improvise-1ey`. Keeps the client thin. Makes an invalid `SetCell`
|
||||
unconstructible rather than rejected. Composes with ADR 001 instead of duplicating it.
|
||||
Projects a distinction the model already makes, so there is one source of truth for what
|
||||
is settable.
|
||||
|
||||
**Bad.** `CellDisplay` grows a field, and it is on the hot path — every visible cell
|
||||
carries it, so the surrogate's width is a wire-size decision, not only a collision one.
|
||||
A client holding a surrogate across a layout change holds something that will fail; that
|
||||
is correct, but the UI must handle it as a normal outcome rather than an error (the same
|
||||
obligation ADR 001 already places on rejection).
|
||||
|
||||
**Not addressed.** The other composability blockers `improvise-q08`
|
||||
listed — `ToggleGroup`, `AddItemInGroup`, `RemoveFormula`, `LoadModel` — which are
|
||||
separate projection gaps and are not solved by a cell surrogate.
|
||||
|
||||
## Extension (2026-09-20) — two more kinds, by the same reasoning
|
||||
|
||||
The decision above is unchanged. `CellId` has since grown two variants, both
|
||||
arrived at by applying clause 1 rather than departing from it: a cell whose
|
||||
permission differs gets a **different kind of identity**, so the wrong operation
|
||||
stays unconstructible instead of being refused at runtime.
|
||||
|
||||
```rust
|
||||
pub enum CellId {
|
||||
Record(RecordCell),
|
||||
Aggregate(AggregateHash),
|
||||
/// A cell a formula owns (improvise-ir7l).
|
||||
Derived { target: String },
|
||||
/// A records-mode creating column (improvise-poia).
|
||||
Create(CreateKind),
|
||||
}
|
||||
```
|
||||
|
||||
**`Derived`** — a formula target is a real stored coordinate, so the original
|
||||
two-variant enum called it a `Record` and every client offered an editor over it.
|
||||
The write *landed*, and was then shadowed forever by `formula_cache`. It carries
|
||||
the formula's target and **no surrogate**: there is nothing a client could spend
|
||||
on a `SetCell`, for the same reason `Aggregate` carries an `AggregateHash`.
|
||||
|
||||
**`Create`** — the records-mode `+` and `+cat` columns projected `None`, which
|
||||
this ADR defines as "no projected cell". That conflation is the one clause 1's
|
||||
rejected `Option<CellId>` draft was about, arriving from the other side: not an
|
||||
aggregate mistaken for an absence, but a column the user *can* type into
|
||||
mistaken for one. It carries only the kind, because what the user types into a
|
||||
creating column is a **name**, not a value — the operation is
|
||||
`ModelCommand::AddItem` / `AddCategory`, which already existed, so no new
|
||||
command was needed and the record under the cursor is not part of it.
|
||||
|
||||
What the original two clauses cost, and still cost, is unchanged: a client
|
||||
learns no coordinate space, and an invalid operation is unconstructible rather
|
||||
than rejected. What the extension adds is that the enum is now **four** answers
|
||||
to "what may be done here", so `None` finally means only what this ADR always
|
||||
said it meant — no projected cell at all.
|
||||
|
||||
**A fifth kind (2026-09-22, improvise-exdg).** The records-mode coordinate cell
|
||||
was the last `None` that named a visible cell. It is now `CellId::Coordinate`.
|
||||
The variant carries no operand, because no client composes a coordinate rename.
|
||||
The previous paragraph's "four answers" is therefore five. `None` now means
|
||||
only a padded or blanked slot. Reasoning: `context/code/protocol.md` §3.
|
||||
@@ -1,149 +0,0 @@
|
||||
# ADR 003 — A model may be backed by a live query source
|
||||
|
||||
**Status:** accepted, **not a priority** (2026-09-19) · **Decided by:** the user ·
|
||||
**Supersedes:** the read-only framing in improvise-rucf
|
||||
|
||||
> The goal is to have **the option** of essentially using improvise as a **DB client**.
|
||||
> Direction, not schedule: nothing here outranks the browser MVP (improvise-1ey), and the
|
||||
> tickets are filed low. This is written down now so the option stays open — several
|
||||
> decisions being made this week could foreclose it by accident.
|
||||
|
||||
## Context
|
||||
|
||||
improvise must handle **arbitrary SQLite databases as models** — a user points it at a
|
||||
database they already have and pivots over it.
|
||||
|
||||
Two readings were offered and the larger one was chosen. **Import-once** — read a query
|
||||
into a `Model` in RAM, exactly as CSV import does — is days of work behind an existing
|
||||
seam (`csv_parser` produces `Vec<Value>`, `analyze_records` classifies it). It was
|
||||
rejected because it makes the model a *snapshot*: source changes are invisible until
|
||||
re-import, and the data must fit in memory.
|
||||
|
||||
**Live-backed** was chosen: the model is a view over the source, and queries go to it as
|
||||
the user pivots.
|
||||
|
||||
This is not improvise-9ix. That ticket proposed SQLite as improvise's *own persistence
|
||||
format* and was declined on measurements — a second implementation of the pest grammar,
|
||||
22% larger than `.improv` and 4.7× gzipped. Reading someone else's database has no
|
||||
round-trip obligation and no grammar duplication. What does carry over is the dependency
|
||||
constraint: `improvise-io` is in the wasm bundle, and `libsqlite3-sys` does not build for
|
||||
`wasm32-unknown-unknown` under this repo's nix cc-wrapper.
|
||||
|
||||
## What the model is today
|
||||
|
||||
`Model` holds `categories`, a `DataStore`, `formulas`, and `views`. `DataStore` interns
|
||||
every category and item name into a `SymbolTable`, stores `IndexMap<InternedKey, CellValue>`,
|
||||
and maintains a secondary index `(Symbol, Symbol) → {InternedKey}` for hot-path lookups.
|
||||
A `Workbook` loads **entire** — there is no partial-load path anywhere in the tree, a fact
|
||||
verified while declining improvise-9ix.
|
||||
|
||||
So this decision changes what `Model` *is*, not merely where its bytes come from.
|
||||
|
||||
## Decision
|
||||
|
||||
**`Model` becomes a facade over a data source, of which the current in-RAM `DataStore` is
|
||||
one implementation and a SQL query engine is another.**
|
||||
|
||||
Three things make this more tractable than it first looks, and they should shape the
|
||||
design rather than be discovered late.
|
||||
|
||||
**1. Aggregation is already the interface, and it is already SQL.** `apply_agg` computes
|
||||
`Sum | Avg | Min | Max | Count` over a matching cell set — those are exactly SQL's
|
||||
aggregates. The pushdown is not a translation layer to be invented; it is the operation
|
||||
the model already performs, expressed in the source's own language. `GridLayout` asks for
|
||||
a cell's value, not for the rows behind it.
|
||||
|
||||
**2. Formula results are already separate from stored data.** Evaluation is fixed-point
|
||||
(`recompute_formulas` iterates to stability, gotcha 9), and formula-derived values live in
|
||||
`Model::formula_cache`, not in `DataStore`. So the split is clean: **raw data is queried,
|
||||
derived data stays in RAM.** Formulas do not need to become SQL.
|
||||
|
||||
**3. Clients are already insulated.** The projection architecture means a client sees
|
||||
`CellDisplay`s and never learns where a value came from — the server holds the `App` and
|
||||
therefore the connection. Live-backing and the browser MVP compose rather than compete,
|
||||
and a browser never needs SQLite at all.
|
||||
|
||||
## The consequence nobody asked for, and it is the important one
|
||||
|
||||
**A live-backed document splits into *our* configuration and *their* data**, and `.improv`
|
||||
does not currently distinguish them. Today a `.improv` file holds categories, views,
|
||||
formulas **and** a `## Data` section. A live-backed model's categories and cell values are
|
||||
the source's; the views, formulas and axis assignments are improvise's.
|
||||
|
||||
So the format grows a **source specification** where a live model would otherwise have
|
||||
data — a connection, a query, and the column-to-field mapping. Categories become *derived*
|
||||
(`SELECT DISTINCT`) rather than declared. That is a grammar change, and `improv.pest` is
|
||||
the single source of truth, so it comes first (design-principles §3).
|
||||
|
||||
Read through the DB-client framing, that split stops being a complication and becomes the
|
||||
**natural shape of the artifact**: a live-backed `.improv` is *a saved query plus a pivot
|
||||
configuration* — which is precisely what a saved report is in any BI tool. The document
|
||||
stops being "my data" and becomes "my question", which is portable between databases in a
|
||||
way a data file never is, and shareable with someone who has the same source and not the
|
||||
same rows.
|
||||
|
||||
## Open questions, in the order they will bite
|
||||
|
||||
**Writes — DECIDED: a live-backed model is read-only.** Not a limitation accepted
|
||||
reluctantly: the user's framing is that for many use cases **editing capabilities would be
|
||||
a downside**. Pointing a pivot tool at a production database and being unable to write to
|
||||
it is a *feature*. So there is no write-through, no overlay, and no reconciliation.
|
||||
|
||||
Two things fall out, and both are simplifications rather than costs.
|
||||
|
||||
**It dissolves the ADR 001 tension instead of requiring an amendment.** ADR 001 governs
|
||||
concurrent *edits*; a read-only model has none, so the basis check simply does not apply
|
||||
to it. The guarantee is not voided, it is **scoped to documents improvise owns** — which
|
||||
is what it always meant. What remains is a smaller and different question: an external
|
||||
writer changes what clients *see*, so live-backed projections need a freshness story
|
||||
(polled, triggered, or explicitly stale-until-refreshed). That is projection freshness,
|
||||
not conflict resolution.
|
||||
|
||||
**The machinery to express it already exists, built for something else.** ADR 002 makes
|
||||
`SetCell` take a `CellId::Record` and nothing else, so a live-backed model that projects
|
||||
no `Record` ids makes a write **unconstructible by the client** — no runtime check, no
|
||||
error path, no "this model is read-only" dialog. The settability distinction built for
|
||||
aggregates carries read-only sources for free.
|
||||
|
||||
**One design consequence for the source trait:** writes should not be *in* the trait that
|
||||
a query source implements. If they were, a read-only source would have to fail at runtime
|
||||
for a method it structurally cannot serve. Keep reads (and aggregates) in the trait every
|
||||
source implements, and let writing be a capability only the owned in-RAM implementation
|
||||
has — so "read-only" is unrepresentable rather than enforced, the same discipline as
|
||||
`CellId`.
|
||||
|
||||
**Interning and identity.** `DataStore` interns names and ADR 002 hashes the coordinate
|
||||
tuple. Both still work — coordinates come from the query — but the `RecordHash → record`
|
||||
index that resolves a `CellId` becomes a cache with an invalidation story rather than a
|
||||
structure maintained alongside writes.
|
||||
|
||||
**Which rows.** A table name or a user-supplied `SELECT`. A query is strictly more useful
|
||||
(joins, filters, computed columns) and no harder to consume; a bare table name is sugar
|
||||
over it.
|
||||
|
||||
**Type mapping.** SQLite is dynamically typed per value, but columns carry declared types,
|
||||
which is *better* input than CSV's string sampling: `detect_date_format` currently guesses
|
||||
a date by pattern-matching text. Dates may be TEXT, INTEGER (epoch) or REAL (julian) —
|
||||
detect, do not assume. NULL is **absence**, not the string `"null"` (improvise-ych).
|
||||
|
||||
**Where the boundary goes.** `improvise-core` must not gain an IO dependency
|
||||
(`forbidden(improvise_core, io)`), so a query-backed source cannot simply be a `DataStore`
|
||||
variant inside core. The trait belongs in core and the SQLite implementation in
|
||||
`improvise-io`, feature-gated and native-only so the wasm gate keeps passing.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good.** Data larger than memory. A source that stays current. Aggregation pushed to an
|
||||
engine built for it. And the client/server split already built means none of this reaches
|
||||
a browser client.
|
||||
|
||||
**Bad.** It changes `Model`, which is the centre of the codebase — `GridLayout`,
|
||||
evaluation, persistence and the import pipeline all touch it. Latency enters a path that
|
||||
is currently a `HashMap` lookup, so the viewport-bounded projection stops being an
|
||||
optimisation and becomes load-bearing. And the `.improv` grammar grows a concept.
|
||||
|
||||
**Worth doing first regardless of the eventual shape:** the source trait and its in-RAM
|
||||
implementation, so that today's `DataStore` becomes *an* implementation rather than *the*
|
||||
implementation. That is a refactor with no behaviour change, `characterization_survives_the_move`
|
||||
applies, and it is the seam everything else needs — including, if it is ever wanted, the
|
||||
import-once path this ADR rejected as the *only* answer.
|
||||
@@ -1,830 +0,0 @@
|
||||
# ADR 004 — Faces: appearance as UI-agnostic data
|
||||
|
||||
**Status:** accepted (2026-09-22) · **Decided by:** the user, who answered all five open
|
||||
questions on PR #122 and then ruled on the one choice they left ·
|
||||
**Direction from:** the user (improvise-nn4s) ·
|
||||
**Governed by:** `context/design-principles.md` §5
|
||||
|
||||
> "I think we want to iterate towards having a way to theme the app like emacs does:
|
||||
> faces, etc. described in a UI-agnostic format that the different frontends can apply
|
||||
> (there's probably a swift version down the road and maybe Qt, etc.)"
|
||||
|
||||
**Iterate towards.** This ADR fixes a vocabulary and a set of constraints so that
|
||||
incremental work cannot paint the repo into a corner. It deliberately does *not* specify a
|
||||
finished theming system, and what is still deferred is named at the end.
|
||||
|
||||
It was written with five questions left open on the grounds that they were the user's
|
||||
rather than the ADR's, and **the user answered all five on PR #122** — then ruled on the
|
||||
one choice those answers left standing. Their answers are folded into the clauses below
|
||||
and recorded, with the reasoning, in *[What the user decided](#what-the-user-decided)*.
|
||||
Two of them changed the decision rather than merely confirming it: `reverse` is in the
|
||||
portable set, and the file format is no longer deferred.
|
||||
|
||||
## Context
|
||||
|
||||
### What the tree actually contains
|
||||
|
||||
Measured at `f0687b1`, and stated as counts because the ticket's figures were re-derived
|
||||
rather than repeated:
|
||||
|
||||
**124 lines carrying 125 ratatui `Color::` literals, across exactly 10 files, all under
|
||||
`src/`.** Not one lives in a shared crate — `improvise-core`, `improvise-protocol`,
|
||||
`improvise-command` and `improvise-wasm-client` name no ratatui type. So
|
||||
`forbidden(improvise_core, ui)` is holding, and the problem this ADR addresses is not a
|
||||
layering breach. It is that **the 125 literals are the only place a large part of the
|
||||
app's appearance is decided, and that place is one front end.**
|
||||
|
||||
The distribution is itself informative. `DarkGray` is 33 of the 125 — a quarter of the
|
||||
app's colouring is one role, "recede". After it: `Yellow` 19, `Cyan` 18, `Green` 15,
|
||||
`Black` 12, `Blue` 11, `Magenta` 7, `White` 3, `Gray` 3, `Red` 2, `Indexed` 2. **Nine
|
||||
named ANSI colours, two indexed greys, and nothing else.** The whole TUI is painted from
|
||||
a palette of eleven, and only the two `Indexed` values (235 for the minibuffer strip, 237
|
||||
for the cursor row) are not a name the user's terminal gets to define.
|
||||
|
||||
Attributes are smaller still: `BOLD` 25, `UNDERLINED` 3, `ITALIC` 1. Ratatui offers nine
|
||||
modifiers and the app uses three.
|
||||
|
||||
### The pattern already works, three times, and is wrong once
|
||||
|
||||
`context/code/tui.md` already carries a section called "Semantic-to-terminal colour, and
|
||||
the one that is wrong", and `invariant(semantic_palette_at_the_core_boundary)` already
|
||||
states the rule: a shared type that needs a colour names a semantic enum, and the render
|
||||
layer owns the one mapping.
|
||||
|
||||
| where | shape | verdict |
|
||||
|---|---|---|
|
||||
| `AxisColor` → `draw.rs::axis_fg` | 5 arms named for **roles** (Row, Column, Page, Filter, Inactive) | the pattern working (improvise-rml) |
|
||||
| `WizardFieldKind` → `import_wizard_ui.rs::field_kind_fg` | 4 arms, the enum is a domain type | the pattern working |
|
||||
| `PanelContent::active_color()` | 3 arms (Formula=Yellow, Category=Cyan, View=Blue) | the pattern working, but **private to the TUI** — never lifted |
|
||||
| `AppMode` → `draw.rs::mode_style` | matches the mode enum directly; there is no `ModeColor` | the gap improvise-zx8t left open on purpose |
|
||||
| `MinibufColor` → `draw.rs::minibuf_fg` | 3 arms named **Green, Cyan, Yellow**; the map is the identity | **the one that is wrong** |
|
||||
|
||||
`MinibufColor`'s doc comment says "Semantic palette only". The type makes that false:
|
||||
there is nothing to theme, because the name *is* the colour.
|
||||
|
||||
### The cost of that is already paid, in the browser, in hex
|
||||
|
||||
`web/app.js` sets `el.minibuffer.className = mb.color.toLowerCase()`, and `web/style.css`
|
||||
answers with **six hand-written hex values** — `#6aa84f`/`#4f7f3c`, `#3f9ba3`/`#2f7980`,
|
||||
`#b8952f`/`#8a6f21`, plus three more under `prefers-color-scheme: dark`. Its comment says
|
||||
they are "resolved to a terminal-descended colour". Every other colour in that file —
|
||||
**77 uses** — goes through a `var(--…)` custom property. The minibuffer's are outside the
|
||||
token system for exactly one reason: **the wire handed the page a colour, and a page
|
||||
cannot ask its own palette for "cyan".**
|
||||
|
||||
That is not a hypothetical. It is the regression this ADR exists to stop, already
|
||||
measurable in one file.
|
||||
|
||||
### The fork
|
||||
|
||||
PR #120 gave the browser a real palette: `--bg`, `--surface`, `--raised`, `--fg`, `--dim`,
|
||||
`--line`, `--line-strong`, `--accent`, `--accent-soft`, `--accent-ink`, `--selected`,
|
||||
`--error`, in light and dark. That palette is deliberately the landing page's, so the site
|
||||
and the app read as one product.
|
||||
|
||||
Note what those names are: **a surface/ink/accent token set, not a list of app roles.** It
|
||||
decomposes along a different axis from the TUI's 125 literals, and nothing relates them.
|
||||
So the two front ends are at a fork: either appearance becomes shared vocabulary that each
|
||||
end maps to its own palette, or PR #120's tokens and the TUI's literals drift permanently
|
||||
and one document looks like two unrelated products. improvise-pcje is already open on the
|
||||
browser dropping every `AxisColor`.
|
||||
|
||||
## Decision
|
||||
|
||||
**Appearance is named by *faces* — roles, not colours. A face carries a small portable
|
||||
attribute set. A theme binds faces to values, resolves once in shared code, and is
|
||||
client-local. A front end translates resolved attributes to its own framework and ignores
|
||||
what it cannot render.**
|
||||
|
||||
Nine clauses. Each is a place this gets decided by accident otherwise.
|
||||
|
||||
### 1. A face names a role. A colour is never a role.
|
||||
|
||||
`AxisColor::Row` is a face name. `MinibufColor::Green` is not. The test is whether the
|
||||
name survives a theme that re-colours everything: "the row axis" does, "green" does not.
|
||||
|
||||
This is the same move improvise-k099 already made for *text* — `AppMode::label` and
|
||||
`hint_text` left the wire enum and became `Hint { keys, command, caption }`, a semantic
|
||||
value a localized client resolves. `invariant(a_ui_string_is_keyed_by_a_command_never_by_a_key)`
|
||||
is that rule for strings; this ADR is the same rule for pixels.
|
||||
|
||||
### 2. Faces are named by *derivation* where a domain enum exists.
|
||||
|
||||
Four of the five families above are indexed by an enum the app already has. A face per arm
|
||||
means **adding an `Axis` variant adds a face by construction** rather than by remembering
|
||||
to edit a list — the same discipline `mode_style`'s exhaustive match already buys
|
||||
(improvise-zx8t), and the reason the vocabulary does not become a 124-entry flat file.
|
||||
|
||||
The face *set* is therefore mostly generated, and only the cross-cutting base faces are
|
||||
written out by hand.
|
||||
|
||||
### 3. Faces inherit, and inheritance resolves exactly once, in shared code.
|
||||
|
||||
Emacs makes a face system tractable by deriving most faces from a few; without it this is
|
||||
124 independent decisions. A face names a parent and overrides some attributes;
|
||||
`Theme::resolve(Face) -> ResolvedFace` walks the chain and returns a struct in which every
|
||||
attribute is concrete.
|
||||
|
||||
**A front end never walks the chain.** Three front ends resolving inheritance
|
||||
independently is three derivations of one answer — improvise-edp is the worked example
|
||||
(the category tree, flattened twice, two different ways) and improvise-anr9 is the second
|
||||
(the grid frame). design-principles §5 states the general rule and this is a direct
|
||||
application of it: *the decision is shared, and only the last translation is local.*
|
||||
|
||||
The cheapest thing this buys is visible in the call sites today. Twelve sites spell
|
||||
`fg(Color::Black).bg(<accent>)` by hand — a hand-rolled reverse video whose ink is
|
||||
hardcoded `Black`. Under inheritance that is *one* rule — a `selection` base face carrying
|
||||
`reverse` (clause 5) over whatever background it inherits — expressed as a **derived face
|
||||
per surface**; as twelve literals it is twelve chances to get a light-background terminal
|
||||
wrong.
|
||||
|
||||
**`resolve` takes a `Face` and nothing else, so "the surface's accent" is a parent and
|
||||
never a parameter.** This is the clause's sharpest edge and it was found by review. The
|
||||
grid's selected cell, the formula panel's and the view panel's are three *different*
|
||||
backgrounds, so they are three faces — `grid.cursor-cell`, `panel.formula.selected`,
|
||||
`panel.view.selected` — each naming `selection` as its parent and overriding `background`
|
||||
from its own surface's accent. They are **not** one `Face::Selection` resolved with
|
||||
context. A resolution that took a context argument would be a second input the front ends
|
||||
would each have to supply correctly, which is the three-derivations hazard reappearing one
|
||||
level down; and a context-free `resolve` is what lets a front end cache a resolved theme
|
||||
as a flat table. Clause 2 already generates faces from domain enums, so "one face per
|
||||
(surface, state) pair" is the mechanism this repo already has, not a new one.
|
||||
|
||||
**`reverse` is not a counter-example to any of that**, though it looks like one: it is
|
||||
applied by the renderer *after* `resolve` returns, against colours the front end already
|
||||
holds, so it adds no input to resolution. Clause 5 works it through.
|
||||
|
||||
### 4. A colour is palette-relative or absolute, and the shipped theme is never absolute.
|
||||
|
||||
**This is the hard question and the clause most expensive to get wrong.**
|
||||
|
||||
Today `Color::Green` means whatever the *user's terminal* calls green. Their terminal
|
||||
theme applies to improvise for free, and 123 of the 125 literals are named colours, so
|
||||
this is nearly the whole TUI. A theme format carrying only RGB silently takes that
|
||||
away — a **regression, not a gap**, and one that would be found late.
|
||||
|
||||
So a colour attribute is:
|
||||
|
||||
```
|
||||
Colour ::= Default -- inherit the surface; state nothing
|
||||
| Named(AnsiSlot) -- one of the 16 slots, meaning "whatever
|
||||
-- THIS DISPLAY calls that slot"
|
||||
| Indexed(u8) -- one of the 256, same meaning, weaker claim
|
||||
| Rgb(r, g, b) -- an absolute colour, deliberately
|
||||
```
|
||||
|
||||
**The line that matters is not where the arms are, it is between the first three and the
|
||||
last: `Default`, `Named` and `Indexed` are *palette-relative*, and `Rgb` is *absolute*.**
|
||||
A palette-relative value says which of the viewer's colours; an absolute one says which
|
||||
colour. So the shipped-theme rule is stated over that line — **the shipped default theme
|
||||
contains no `Rgb`** — and it is the only rule that actually protects the terminal theme.
|
||||
An earlier draft wrote the rule as "only `Default` and `Named`" while also promising the
|
||||
two existing `Indexed` greys would keep rendering exactly, and left the fourth arm as "a
|
||||
detail". Those three cannot all hold: folding an index into `Rgb` changes what it *means*
|
||||
and breaks the rule in the same stroke. Review caught it; the fourth arm is the fix.
|
||||
|
||||
**Which spelling does what to a terminal user, stated as a table because the difference is
|
||||
invisible in a theme file and the wrong one is a thing to stumble into rather than choose:**
|
||||
|
||||
| spelling | follows the user's terminal theme? | what it is for |
|
||||
|---|---|---|
|
||||
| `Default` | yes — states nothing at all | ordinary text, the terminal's own background |
|
||||
| `Named(slot)` | **yes**, fully: slots 0–15 are what a terminal theme redefines | almost everything; the shipped default theme is written in these |
|
||||
| `Indexed(n)` | in principle, in practice no for 16–255 — nothing re-themes the xterm cube | a shade the sixteen cannot express, such as "just off the background" (the two greys today) |
|
||||
| `Rgb(r,g,b)` | **no** | an exact colour: a brand shade, or a hue outside any palette |
|
||||
|
||||
So a theme author reaching for `Rgb` gets exactly what the user ruled they should be able
|
||||
to have — an exact colour, which is the honest answer for a brand shade or anything the
|
||||
sixteen slots cannot say — **and pays for it by opting that theme's users out of terminal
|
||||
palette tracking.** Fine when chosen, bad when stumbled into, which is why the trade is
|
||||
written here rather than left to be discovered.
|
||||
|
||||
Four things follow, and the first is the point of the whole clause.
|
||||
|
||||
**A named slot is not a colour, it is an index into the viewer's palette.** A theme that
|
||||
says `axis.row: Named(Green)` has not chosen green; it has chosen *the user's* green. That
|
||||
is how the terminal theme survives, and it survives by the theme *declining to state a
|
||||
colour* rather than by a special case.
|
||||
|
||||
**`Default` is a third value and not a synonym for anything.** The TUI already never sets
|
||||
a background for ordinary grid content, so the terminal's own background shows through; a
|
||||
theme that could only say `Rgb` or `Named` would have to invent one. It is the same
|
||||
distinction ADR 002 drew between an aggregate and an absence — "unstated" is a real
|
||||
answer.
|
||||
|
||||
**A theme MAY carry two slot tables, and a single-table theme stays legal** (the user's
|
||||
answer to Q4: themes "should usually have the option to split out light/dark palettes").
|
||||
Optional, not mandatory — so `Theme { slots: SlotTable, dark: Option<SlotTable>, … }`
|
||||
rather than a pair, because a theme with one palette should not have to say it twice.
|
||||
|
||||
**What a terminal does with a two-table theme is: almost nothing, and that falls out of
|
||||
clause 4 rather than needing a rule.** A terminal *has* the user's palette, so `Named` and
|
||||
`Indexed` never consult a slot table at all — the tables are the browser's *substitute*
|
||||
for a terminal palette, not a second opinion about one. They therefore bite in exactly two
|
||||
places: a browser resolving a slot, and an `Rgb` value a theme chose to state twice. For
|
||||
that second case a two-table theme names which table a single-scheme front end takes
|
||||
(`default_scheme`), because a terminal cannot be asked which it is in: `prefers-color-scheme`
|
||||
has no terminal equivalent, and while some terminals answer an OSC 11 background query,
|
||||
crossterm exposes nothing portable, so **detection is not on the table and is not needed.**
|
||||
One declared field beats a guess. Since the shipped theme is palette-relative throughout,
|
||||
the practical answer is that the tables are a browser concern and the TUI is unaffected.
|
||||
|
||||
**What a browser does with "terminal green":** it ships a 16-slot table per scheme and
|
||||
resolves the slot through it. A browser has no terminal to ask, so it *is* the
|
||||
terminal — this is precisely what a terminal emulator does, and it costs 16 custom
|
||||
properties. Those sixteen are where PR #120's palette meets the face system: slot 2 should
|
||||
resolve to the product's green (`#2f6f4f` light / `#8ec07c` dark), not to a generic ANSI
|
||||
green, so the app keeps its identity while the TUI keeps the user's. **One table, one
|
||||
place, both facts served.** It also retires the nine stray hex values in `style.css` — six
|
||||
under the light scheme and three under dark — which are the only colours in that file not
|
||||
reached through a `var(--…)`.
|
||||
|
||||
**What a browser does with `Indexed(237)`:** slots 0–15 go through the same 16-slot table,
|
||||
and 16–255 through the standard xterm formula — the 6×6×6 cube at `16 + 36r + 6g + b` and
|
||||
the 24-step greyscale ramp at 232–255. That is one well-defined function, written once in
|
||||
the shared resolver rather than in each front end, and it is what makes an index portable
|
||||
at all.
|
||||
|
||||
**What a terminal does with `#3f7a4d`:** ratatui has `Color::Rgb`, so a truecolor terminal
|
||||
renders it exactly; a 256-colour terminal must approximate and a 16-colour one must
|
||||
approximate badly. So RGB is *permitted and degrades*, where a palette-relative value is
|
||||
*exact everywhere*. Hence the rule: **the shipped default theme contains no `Rgb`. Using
|
||||
`Rgb` is a user's deliberate opt-out of their own terminal theme**, which is a thing a
|
||||
user is allowed to want and is not a thing improvise should do on their behalf.
|
||||
|
||||
**`Indexed` is palette-relative but makes the weaker claim, and that is worth saying
|
||||
rather than glossing.** Slots 0–15 are what a user's terminal theme actually re-defines;
|
||||
16–255 are the xterm cube and almost no theme touches them. So an `Indexed` value is
|
||||
portable and exact, and it is *not* meaningfully themeable by the user. That is precisely
|
||||
why `Indexed(235)`/`(237)` exist today: no ANSI slot names "very slightly lighter than
|
||||
whatever the background is", and those two greys are the app reaching past the sixteen for
|
||||
something the sixteen cannot express. They stay, spelled as the 256-colour slots they are,
|
||||
and a theme author choosing `Indexed` over `Named` should know they have traded the user's
|
||||
palette for a guaranteed shade.
|
||||
|
||||
### 5. Six portable attributes, `reverse` among them
|
||||
|
||||
An attribute a front end cannot render is ignored.
|
||||
|
||||
`foreground`, `background`, `weight` (normal/bold), `slant` (normal/italic), `underline`,
|
||||
`reverse`. The first five are what the app uses today — `BOLD` 25 times, `UNDERLINED` 3,
|
||||
`ITALIC` once — so that part of the set is not a guess.
|
||||
|
||||
All six map onto ratatui `Modifier` (`REVERSED` is `ratatui-core`'s `style.rs:111`), onto
|
||||
CSS, onto `NSAttributedString`/SwiftUI and onto `QTextCharFormat`. Box, shadow,
|
||||
border-radius and font family do not reach a terminal and are **out of the face system**,
|
||||
not optional within it: an attribute that half the front ends structurally cannot serve
|
||||
belongs to the front end that can, the same way ADR 003 keeps writes out of the trait a
|
||||
read-only source implements. *Strike-through remains out* — nothing in the app strikes
|
||||
anything through, so including it would be the speculation this clause is shaped to avoid.
|
||||
|
||||
**A front end ignores an attribute it cannot render**, as Emacs does on a tty. Said
|
||||
explicitly because the alternative — refusing, or falling back — makes every future
|
||||
attribute a compatibility negotiation.
|
||||
|
||||
#### `reverse` is in, and the objection to it was wrong
|
||||
|
||||
An earlier draft left `reverse` out on the ground that "CSS, Swift and Qt have no such
|
||||
thing and would have to compute it, which needs a resolved background and therefore makes
|
||||
resolution stateful". The user asked for it anyway, and asked whether CSS could do it with
|
||||
a class that overrides custom properties. **It can, and working that through shows the
|
||||
objection was simply mistaken.**
|
||||
|
||||
The swap happens in the **renderer, after resolution**, against two values the element
|
||||
already carries — so `resolve` stays context-free and clause 3 is untouched:
|
||||
|
||||
```css
|
||||
[data-face] { color: var(--face-fg, var(--fg));
|
||||
background: var(--face-bg, var(--bg)); }
|
||||
[data-face].reversed { color: var(--face-bg, var(--bg));
|
||||
background: var(--face-fg, var(--fg)); }
|
||||
```
|
||||
|
||||
Three things make this the right answer rather than a trick. **It is a re-reference, not a
|
||||
computation** — both colours are already custom properties on the element, so the reversed
|
||||
rule just reads them crossed; nothing is derived and no JavaScript is involved. **The
|
||||
`Default` case, which is the one that would have broken it, is covered by the `var()`
|
||||
fallback**: a face that states no foreground falls through to the page's own `--fg`/`--bg`
|
||||
tokens, which is exactly what a terminal's own reverse does against the terminal's current
|
||||
colours. And **it needs no reversed *palette***, which was the shape the question offered:
|
||||
a second palette would be a second set of values to keep in step, which is the disease this
|
||||
whole ADR is about. One class, two rules.
|
||||
|
||||
**Swift and Qt are not helped by a CSS trick, and do not need to be.** Both compose runs
|
||||
with an explicit foreground and background — `NSAttributedString`'s `.foregroundColor` /
|
||||
`.backgroundColor`, `QTextCharFormat`'s `setForeground` / `setBackground` — so the swap is
|
||||
two assignments against values they already hold, with the same `Default` substitution
|
||||
from their own surface. So the rule generalises and is worth stating as the rule rather
|
||||
than as three answers: **`reverse` is applied by the front end after resolution, against
|
||||
the surface colours that front end knows.** The resolver never learns what a surface is.
|
||||
|
||||
The TUI gets the better end of this. `Modifier::REVERSED` is handled by the *terminal*,
|
||||
which swaps against whatever the cell's actual colours are — so the twelve hand-rolled
|
||||
`fg(Color::Black).bg(accent)` sites become `reverse` over the accent and stop asserting
|
||||
that the ink is black, which is the light-terminal bug clause 3 describes.
|
||||
|
||||
### 6. The vocabulary lives in `improvise-protocol`, and `improvise-core` names no face.
|
||||
|
||||
Today `AxisColor` is in `improvise-core` and `MinibufColor` is in `improvise-protocol`.
|
||||
The ticket asks the ADR to resolve that rather than inherit it.
|
||||
|
||||
**A face is UI.** `improvise-core` is the crate that is supposed to have none, and
|
||||
`AxisColor` squeaked past `forbidden(improvise_core, ui)` only because it is *semantic*
|
||||
UI. Its presence there is an accident of `Axis::color()` being a method on a core type, not
|
||||
a decision. So the face vocabulary goes where `Panel`, `AppMode` and `Hint` already are —
|
||||
`improvise-protocol`, which depends on core and is depended on by both front ends — and
|
||||
`Axis::color()` becomes `face_of(Axis)` there.
|
||||
|
||||
The cost is real and should be weighed: gotcha 26 says "axis presentation lives on `Axis`",
|
||||
and this moves one of its five vocabularies off the type. improvise-rml's consolidation is
|
||||
not undone — its point was *one* home, not that the home be core — but a reader who knows
|
||||
gotcha 26 will look in the wrong place for a release or two. The alternative (put faces in
|
||||
core, so `Axis::color()` stays) keeps that pointer true and costs the core-has-no-UI rule,
|
||||
which is the more expensive of the two to bend.
|
||||
|
||||
**A role may cross the wire; a theme never does.** `MinibufferConfig.color` is already
|
||||
serialized into the Snapshot, and that stays correct: the *server* says "this prompt is an
|
||||
edit prompt", and the *client* decides what an edit prompt looks like. Faces in
|
||||
`improvise-protocol` is exactly that division.
|
||||
|
||||
### 7. A theme is client-local, and is not document data.
|
||||
|
||||
Three negatives, all cheap now and a protocol change later:
|
||||
|
||||
- **A theme is not a `Command`, is not in `ViewState`, and is not projected.** Gotcha 29
|
||||
and the user's own ruling on improvise-g0f: the model is shared and the UI is not. A
|
||||
theme reaching another client's screen is that bug exactly. `ViewState` membership is
|
||||
gated on being *snapshot-safe* — "a fresh client inheriting this value is correct" —
|
||||
and a theme fails it by definition (gotcha 30).
|
||||
- **A theme is not in `.improv`.** A document must not dictate its reader's colours. This
|
||||
is not merely tidiness: ADR 003 is already splitting `.improv` into *our configuration*
|
||||
and *their data*, and a theme is neither.
|
||||
- **A theme is not in the `Projection` stream.** Nothing a client themes is dirty for
|
||||
anybody else, so there is nothing to diff.
|
||||
|
||||
### 8. Migration: the default theme reproduces today's rendering, with one exception
|
||||
|
||||
**The mechanism that makes a flag day unnecessary is not a plan, it is a property.** If
|
||||
the built-in default theme resolves every face to precisely the colour and modifier the
|
||||
literal at that call site produces today, then:
|
||||
|
||||
- migrating one call site is a **no-op**, verifiable by a `TestBackend` buffer comparison;
|
||||
- a **partially** migrated file is not visibly wrong, so there is no pressure to finish;
|
||||
- the 125 literals move opportunistically, when a file is touched for another reason.
|
||||
|
||||
**The one exception is `minibuffer.category-add`, and it must stay an exception rather
|
||||
than becoming a habit.** The user ruled (Q1) that the green/yellow split between
|
||||
`item-add` and `category-add` is not meaningful, so the default theme makes them agree and
|
||||
`category-add` changes from yellow to green. That is a deliberate one-line behaviour
|
||||
change riding along with a migration whose whole safety argument is that it changes
|
||||
nothing.
|
||||
|
||||
Two consequences follow and both are the point of writing it down. The
|
||||
`TestBackend` comparison for that one call site **must be updated rather than expected to
|
||||
pass**, so it is the single place a reviewer has to actually look. And **any future
|
||||
exception needs the same treatment** — named in this clause, with who decided it. A
|
||||
migration that quietly accumulates "while we were in there" changes loses the property
|
||||
that makes it safe to do piecemeal, and then the flag day comes back.
|
||||
|
||||
The order that foreclosed nothing, cheapest first:
|
||||
|
||||
1. `Face`, `Theme`, `resolve` and the default theme land in `improvise-protocol`. Nothing
|
||||
else changes; no call site moves.
|
||||
2. `draw.rs` gains **one** `face_style(Face) -> Style`. `axis_fg`, `minibuf_fg`,
|
||||
`field_kind_fg` and `mode_style` become callers of it and then disappear.
|
||||
3. `PanelContent::active_color()` and `field_kind_fg` are lifted into faces — they are
|
||||
already the right shape and only lack a home.
|
||||
4. `MinibufColor` is retired in favour of a face per text-entry mode (see below).
|
||||
5. The browser's 16-slot table replaces `style.css`'s nine stray hex values, and the
|
||||
`--` tokens become that theme's surface faces.
|
||||
6. Everything else, as files are touched.
|
||||
|
||||
**A note on the ticket's own step (a), which does not survive contact with the call
|
||||
sites.** improvise-nn4s proposes renaming `MinibufColor`'s three variants to roles as the
|
||||
cheapest first move. It cannot be done honestly: there is no three-way role split that
|
||||
fits the eight construction sites (below). The right move is a face per text-entry mode —
|
||||
eight faces, three of which happen to resolve to the same value in the default theme.
|
||||
That is barely more work and it is the difference between a true type and a second false
|
||||
one.
|
||||
|
||||
### 9. A theme is JSON, and a published face name is a compatibility surface.
|
||||
|
||||
The format was drafted as deferred, on the stated condition that improvise has no
|
||||
configuration story of any kind. **The user's answer to Q5 dissolves that condition rather
|
||||
than meeting it:** "They should be external and I should be able to download a JSON dump
|
||||
of the current theme, edit it and upload."
|
||||
|
||||
**Download-edit-upload is not a configuration file**, and that distinction is what lets
|
||||
this land without first deciding how improvise is configured. A config file needs a search
|
||||
path, a discovery order, a precedence rule between system and user and project, and a
|
||||
policy for what happens when it is absent or malformed. An export/import pair needs none
|
||||
of them: it is a value the user round-trips, which is the shape `export_csv` already has.
|
||||
So the larger decision stays unmade and this one no longer waits on it.
|
||||
|
||||
**JSON, and the choice costs nothing.** `serde_json` is already a dependency of every
|
||||
crate in the workspace except `improvise-protocol`, where it is optional and dev-only — so
|
||||
JSON adds no dependency anywhere, while TOML would have added one, and to a *bundle* crate
|
||||
if the parsing lived beside the types. The `Face` and `Theme` types therefore derive
|
||||
`Serialize`/`Deserialize` in `improvise-protocol` (serde alone, already a dependency) and
|
||||
the `serde_json` call happens at the **front ends**, which already have it. That keeps
|
||||
protocol's dependency footprint unchanged, which the `wasm` CI job checks per crate, and
|
||||
it is the same division `ViewState` already has: derive in the protocol, encode at the
|
||||
edge. A pest grammar was the other candidate and loses here — its advantage is parse
|
||||
errors on hand-written text, and a file the user obtained by *downloading it from the app*
|
||||
starts well-formed.
|
||||
|
||||
**"Editable by hand" is a constraint on the shape, not a nicety.** Face names are the
|
||||
keys, spelled as the vocabulary spells them; ordering is stable, so a re-download after an
|
||||
edit produces a reviewable diff rather than a reshuffle (`serde_json`'s `preserve_order`,
|
||||
or sorted keys — `indexmap` is already in the tree); and there are **no surrogate keys**,
|
||||
because a numeric id in a file a human edits is a lookup the human has to perform.
|
||||
|
||||
**A published face name is permanent.** This is the real cost of Q5 and it is worth
|
||||
stating as plainly as the benefit: once a downloadable theme spells `grid.column-header`,
|
||||
a user's saved theme contains that string, and renaming the face breaks their file.
|
||||
Face names are therefore a **compatibility surface** on the same footing as the `.improv`
|
||||
grammar and the wire frames — which is an argument for deriving names from domain enums
|
||||
(clause 2), since those change rarely and visibly, and an argument against inventing
|
||||
decorative sub-names that a later tidy-up would want back. What a theme may safely do is
|
||||
*omit* a face it does not override; inheritance (clause 3) then supplies it, so a theme
|
||||
written today keeps working when a face is added tomorrow.
|
||||
|
||||
The download/upload path itself — a browser half and a TUI half — is **follow-up work and
|
||||
not part of this ADR**; see the deferred section.
|
||||
|
||||
## The role vocabulary, derived from the 125 call sites
|
||||
|
||||
This is the part an ADR gets wrong by inventing, so it is derived rather than designed —
|
||||
and where the call sites disagree with themselves, that is recorded rather than smoothed
|
||||
over.
|
||||
|
||||
**Base faces** (everything inherits from these):
|
||||
|
||||
| face | today | sites |
|
||||
|---|---|---|
|
||||
| `default` | unset fg, unset bg | the ordinary grid cell, `HelpStyles::normal` |
|
||||
| `dim` | `DarkGray` | **33** — hints, empty cells, inactive borders, rules, secondary text |
|
||||
| `emphasis` | `BOLD`, no colour | 25 |
|
||||
| `error` | `Red` | 2, both in the wizard |
|
||||
|
||||
`dim` at 33 sites is the single largest role in the app and has no name today.
|
||||
|
||||
**Chrome:** `title-bar` (Black on Blue, bold), `rule` (the horizontal separators),
|
||||
`popup-border`, `hint`, `banner`. `rule` and `hint` both resolve to `dim` today and are
|
||||
separate faces anyway — a face system's job is to make them *separable*, not to assert
|
||||
they differ.
|
||||
|
||||
**Grid:** `grid.column-header` (Yellow bold), `grid.group-header` (Green bold),
|
||||
`grid.total` (Yellow bold), `grid.cell-empty` (→ `dim`), `grid.cursor-row`
|
||||
(Cyan on `Indexed(237)`, bold), `grid.cursor-cell` (→ `selection`), `grid.search-match`
|
||||
(Black on Yellow), `grid.edit` (Green, underlined), `grid.page-bar` (→ `axis.page`).
|
||||
|
||||
**Derived:** `selection` is a **base** face, not a face anything draws with. The twelve
|
||||
`fg(Black).bg(accent)` sites are one *rule* and several faces — `grid.cursor-cell`,
|
||||
`panel.{formula,category,view}.selected`, `tile-bar.selected`, `wizard.selected` — each
|
||||
inheriting `selection` and overriding `background` from its own surface's accent, per
|
||||
clause 3. Written out because "one parameterised selection face" reads as the tidier
|
||||
answer and cannot be resolved by `resolve(Face)`.
|
||||
|
||||
**Parameterised families**, one face per arm of an enum that already exists:
|
||||
|
||||
- `axis.{row, column, page, filter, none}` — is `AxisColor`, unchanged.
|
||||
- `panel.{formula, category, view}` — is `active_color()`, lifted.
|
||||
- `wizard.field.{category, measure, time, label}` — is `field_kind_fg`, lifted.
|
||||
- `mode.*` — does **not** exist. `mode_style` matches `AppMode` directly, and
|
||||
improvise-zx8t explicitly left `ModeColor` open, deferring it to be decided with
|
||||
improvise-k099. k099 has since closed by doing exactly the analogous thing for text, so
|
||||
the precedent this ADR needs is already set. Its shape is the next section.
|
||||
- `minibuffer.{edit, formula, command, category-add, item-add, export, save}` — is
|
||||
`MinibufColor`, re-derived.
|
||||
|
||||
### What the thirteen DarkGray modes are hiding, and what they are not
|
||||
|
||||
The user's answer to Q2 was that "the dark gray group is probably hiding semantic
|
||||
differences that should be resolved by inheritance". **Probably** is the right hedge, so
|
||||
the sub-roles below are derived from distinctions `AppMode` *already draws* — its
|
||||
`minibuffer()`, `panel()`, `is_editing()` and `is_records()` methods — rather than
|
||||
invented one per variant, and the splits that could not be justified are named as such.
|
||||
|
||||
`mode_style` colours four variants (Editing/RecordsEditing green, CommandMode yellow,
|
||||
TileSelect magenta) and sweeps thirteen into DarkGray. Those thirteen partition as:
|
||||
|
||||
| face | variants | the fact that groups them |
|
||||
|---|---|---|
|
||||
| `mode.resting` | Normal, RecordsNormal | 2 |
|
||||
| `mode.text-entry` | FormulaEdit, CategoryAdd, ItemAdd, ExportPrompt, SavePrompt | 5 — `minibuffer()` is `Some` |
|
||||
| `mode.panel.{formula,category,view}` | FormulaPanel, CategoryPanel, ViewPanel | 3 — is `Panel::ALL`, so clause 2 derives it |
|
||||
| `mode.overlay` | ImportWizard, Help | 2 |
|
||||
| `mode.transient` | Quit | 1 |
|
||||
|
||||
**The text-entry group is the one that was genuinely hiding something, and it is visible
|
||||
without any judgement call.** All five have a `MinibufferConfig`, so their *minibuffer*
|
||||
is painted green, cyan or yellow while their *mode indicator* says nothing at all — and
|
||||
`Editing` and `CommandMode`, which are the same kind of mode, are the two that do get a
|
||||
colour. So the existing split is not "these three matter and those thirteen do not"; it is
|
||||
one family, partly coloured. The chain is therefore `mode` → `mode.text-entry` →
|
||||
`mode.text-entry.<variant>`, with `editing` and `command` overriding today and the other
|
||||
five inheriting — which is exactly the user's "resolved by inheritance", and it changes no
|
||||
pixel until someone decides a value.
|
||||
|
||||
**Three splits I could not justify, recorded rather than guessed:**
|
||||
|
||||
- **Normal vs RecordsNormal.** The difference is the grid's *shape*, which the grid itself
|
||||
shows plainly; a mode indicator saying it too is redundant. One face.
|
||||
- **Help vs ImportWizard.** Both are full-screen overlays. They differ in whether keys
|
||||
mutate anything, which is real — but the mode indicator is not where a user learns it,
|
||||
and inventing two faces to encode it would be the ADR deciding a UI question it has no
|
||||
evidence for. One face; split it when someone wants them to look different.
|
||||
- **Quit.** Drawn for at most one frame, so whether it is its own face is unobservable. It
|
||||
has one only because clause 2 derives a face per arm, and that is a better reason than
|
||||
any aesthetic one.
|
||||
|
||||
Note the caution this section is under. Q2's answer contains "probably", and
|
||||
`doctrine(an_invariant_generalised_from_a_fix_generalises_the_fix)` is about exactly the
|
||||
move of writing down more structure than the evidence carries because the evidence is in
|
||||
front of you. Five faces where there were none is a decomposition; thirteen would have
|
||||
been a transcription.
|
||||
|
||||
### What `MinibufColor`'s three colours actually distinguish
|
||||
|
||||
Read off the eight construction sites in `improvise-protocol/src/lib.rs`:
|
||||
|
||||
| face | prompt | today |
|
||||
|---|---|---|
|
||||
| `minibuffer.edit` (cell), `minibuffer.edit` (records) | `edit: ` | Green |
|
||||
| `minibuffer.item-add` | `add item to X: ` | **Green** |
|
||||
| `minibuffer.formula` | `formula: ` | **Cyan** |
|
||||
| `minibuffer.command` | `:` | Yellow |
|
||||
| `minibuffer.category-add` | `new category: ` | **Yellow** |
|
||||
| `minibuffer.export` | `export path: ` | Yellow |
|
||||
| `minibuffer.save` | `save as: ` | Yellow |
|
||||
|
||||
Two things this contradicts, both worth stating because a plausible reading of these
|
||||
colours is wrong in both directions:
|
||||
|
||||
**Cyan is the formula prompt, not search.** There is no search minibuffer at all —
|
||||
`SearchMode` is `Normal` plus a `search_mode` flag (gotcha 14), drawn in `draw_status`
|
||||
under `mode_style`, and the search *match* in the grid is Black-on-Yellow. Nothing in the
|
||||
minibuffer is search.
|
||||
|
||||
**Green and yellow do not split on "editing vs prompting".** `item-add` is green and
|
||||
`category-add` is yellow, and those are the same kind of act — typing a name that becomes
|
||||
part of the document. Either green means "this text becomes content" (which `category-add`
|
||||
then violates) or the split is an accident. The code cannot say which, so this was put to
|
||||
the user as Q1, and **the user's answer was "No" — it is an accident.**
|
||||
|
||||
Three things follow, and the first is the one worth guarding.
|
||||
|
||||
**No face name encodes the distinction.** `minibuffer.item-add` and
|
||||
`minibuffer.category-add` both exist — clause 2 derives a face per text-entry mode, so
|
||||
they exist by construction — but neither is named for a difference, and both inherit
|
||||
`mode.text-entry` directly rather than being grouped under some invented parent that would
|
||||
re-assert it. A vocabulary that spelled the accident would have outlived the colours,
|
||||
which is why this was the question flagged as expensive.
|
||||
|
||||
**The default theme makes them agree**, which is the declared exception to clause 8's
|
||||
no-op property: `category-add` becomes green.
|
||||
|
||||
**The ability to theme them apart is retained and costs nothing.** Two faces that resolve
|
||||
alike are not a redundancy — they are the seam at which a user who *does* want the
|
||||
distinction can have it, without anyone having to add a face first. That is what naming
|
||||
roles buys over naming colours, and it is worth noticing that the answer "no, this is not
|
||||
meaningful" leaves the face vocabulary in exactly the shape clause 2 already required.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
**Per-front-end palettes, kept in step by a test.** The cheapest thing that looks like it
|
||||
works: the browser keeps `style.css`, the TUI keeps its literals, and something asserts
|
||||
they agree. Rejected by
|
||||
`doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished)` — two
|
||||
definitions held in step by a test are one definition waiting to be made — and by the fact
|
||||
that no such test can be written here: the two ends do not agree *today* (the browser
|
||||
drops every `AxisColor`, improvise-pcje) and a test would have to encode the disagreement.
|
||||
|
||||
**Ship RGB only, the way a web palette naturally is.** Simplest to specify, one value per
|
||||
attribute, no slot table anywhere. Rejected because it is the regression in clause 4: it
|
||||
silently ends the user's terminal theme applying to improvise, for 123 of 125 literals,
|
||||
and the loss would be discovered by a user rather than by a test. Emacs solved this
|
||||
decades ago with named colours plus tty approximation and there is no reason to relearn it.
|
||||
|
||||
**Palette-relative only, no `Rgb` arm at all — proposed, and REJECTED BY THE USER.**
|
||||
Genuinely tempting, because it makes the regression *unrepresentable* rather than merely
|
||||
defaulted-against, which is the strongest shape this repo has and the one `CellId` and
|
||||
`Row<T>` both took. The draft kept the arm on its own reasoning — a theme author on a
|
||||
truecolor terminal has a legitimate reason to want an exact colour, a browser theme is
|
||||
naturally written in RGB, and the arm would be reintroduced by the first person who needed
|
||||
it — but flagged the weakness and put the choice to the user rather than settling it.
|
||||
|
||||
> **"Keep full color support in themes"** — the user, on `6a4c7dc`
|
||||
|
||||
So `Rgb` stays, by decision rather than by default. **The protection against losing the
|
||||
user's terminal theme is therefore a property of the *shipped theme* and not of the type,
|
||||
and that is now a chosen cost rather than an unexamined one.** It is the honest trade: a
|
||||
theme *may* say anything a display can show, and what keeps the terminal palette alive is
|
||||
that the theme improvise ships declines to.
|
||||
|
||||
**One distinction this ruling does not collapse, stated because it is the easy misreading.**
|
||||
"Full colour support in themes" is about what the *format may express*; the shipped-theme
|
||||
rule is about what *improvise's own default says*. They are independent, and clause 4
|
||||
keeps both: the format admits `Rgb`, and the default theme contains none. If the intent
|
||||
was also that the shipped default use absolute colour, that is a different change and
|
||||
would reopen the regression clause 4 exists to prevent — so it is worth saying out loud
|
||||
rather than inferring.
|
||||
|
||||
**A `Style` struct on the wire instead of a face name.** i.e. the server resolves the
|
||||
theme and projects concrete attributes. Rejected by clause 7: the theme would then have to
|
||||
be server-side, which puts one client's colours on another client's screen — improvise-g0f
|
||||
decided the general form of this question and the answer was the model is shared and the UI
|
||||
is not.
|
||||
|
||||
**A face system in `improvise-core`.** Keeps `Axis::color()` where gotcha 26 says it is.
|
||||
Rejected in clause 6, but narrowly, and a reader who weighs the two costs differently is
|
||||
not being unreasonable.
|
||||
|
||||
**A theme section in `.improv`.** Rejected: a document dictating its reader's colours. Also
|
||||
a grammar change (`improv.pest` is the single source of truth) for something that is not
|
||||
document data — precisely the cost that sank improvise-9ix's second serialization format.
|
||||
|
||||
## What the user decided
|
||||
|
||||
The first five were left open on the ground that they were the user's rather than the
|
||||
ADR's, and were answered as inline comments on PR #122 at `246e8f4a`; the sixth is the
|
||||
choice those answers left standing, ruled on at `6a4c7dc`. Each answer is quoted verbatim,
|
||||
with what it settled and where it now lives — the questions are kept because a decision
|
||||
whose alternatives have been deleted reads as though it was never in doubt.
|
||||
|
||||
**1. Is the `item-add` / `category-add` colour split meaningful?** Green for adding an item
|
||||
and yellow for adding a category. If it encodes something ("green means this becomes
|
||||
content"), the face names must carry it and every later migration is spelled in them; if it
|
||||
is an accident, the default theme should quietly make them agree. Flagged as the one
|
||||
mistake here that is expensive, because face names outlive colours.
|
||||
|
||||
> **"No"**
|
||||
|
||||
Settled: an accident. **No face name encodes it, the default theme makes them agree, and
|
||||
that is clause 8's one declared exception** to the reproduces-today-exactly property. The
|
||||
two faces still exist, because clause 2 derives one per mode, so a user who wants the
|
||||
distinction can still have it. Worth recording that this went the *cheap* way: had the
|
||||
answer been yes, the distinction would have had to be carried in the vocabulary
|
||||
permanently.
|
||||
|
||||
**2. What is `mode_style`'s role split?** Editing=Green, CommandMode=Yellow,
|
||||
TileSelect=Magenta, and the remaining *thirteen* variants all DarkGray — counted, because
|
||||
`context/code/tui.md` says fourteen and is off by one. Is that three roles and a
|
||||
default, or is the DarkGray group hiding modes that want their own? improvise-zx8t declined
|
||||
to invent `ModeColor` on the grounds that the colour was the only thing at stake; under a
|
||||
face system that is no longer true.
|
||||
|
||||
> **"The dark gray group is probably hiding semantic differences that should be resolved
|
||||
> by inheritance"**
|
||||
|
||||
Settled: decompose, **and inheritance is the mechanism** — which makes clause 3
|
||||
load-bearing rather than convenient. The thirteen become five faces, derived from
|
||||
distinctions `AppMode` already draws, and the three splits that could not be justified are
|
||||
named. *[What the thirteen DarkGray modes are hiding](#what-the-thirteen-darkgray-modes-are-hiding-and-what-they-are-not)*
|
||||
has the working. The "probably" was respected: five faces is a decomposition, thirteen
|
||||
would have been a transcription.
|
||||
|
||||
**3. Should `reverse` be a portable attribute?** Terminals have it natively and it is the
|
||||
honest spelling of the twelve `fg(Black).bg(accent)` sites — it inverts against *whatever*
|
||||
the surface is, so it cannot get a light terminal wrong. The draft left it out because
|
||||
CSS, Swift and Qt would have to compute it, which needs a resolved background and
|
||||
therefore looked as though it made resolution stateful.
|
||||
|
||||
> **"Yes, reverse is important can we compute it in CSS using a reversed palette or
|
||||
> something that overrides CSS variables and a class to activate it?"**
|
||||
|
||||
Settled: **yes, and the objection was wrong.** A class that reads the face's two custom
|
||||
properties crossed does it in pure CSS — a re-reference, not a computation — and the
|
||||
`var()` fallback covers the `Default` case that would otherwise have broken it. No
|
||||
reversed *palette* is needed, which is the better outcome, since a second palette is a
|
||||
second set of values to keep in step — and PR #120's `--accent-ink`, which the question
|
||||
offered as the alternative, is likewise not needed: a hardcoded ink token is the same
|
||||
assertion about the surface that `fg(Color::Black)` makes, one layer up. Swift and Qt are
|
||||
not helped by the CSS mechanism but
|
||||
do not need to be: the general rule is that **`reverse` is applied by the front end after
|
||||
resolution**, so `resolve` stays context-free and clause 3 is untouched. Clause 5 has the
|
||||
rules and the reasoning. The portable set is now six.
|
||||
|
||||
**4. Does the browser honour `prefers-color-scheme` under a theme?** A terminal has one
|
||||
scheme and cannot be asked which; a browser has two and PR #120 already serves both. Either
|
||||
a theme carries two slot tables, or a theme *is* per-scheme and the browser picks one.
|
||||
|
||||
> **"Themes should usually have the option to split out light/dark palettes and wire each
|
||||
> into the CSS appropriately."**
|
||||
|
||||
Settled: **optional, not mandatory** — "usually have the option". A single-table theme
|
||||
stays legal, so the second table is an `Option` rather than half of a pair. What a
|
||||
terminal does with a two-table theme is answered in clause 4 and is mostly "nothing": a
|
||||
terminal already has the user's palette, so the tables are the browser's *substitute* for
|
||||
one and only bite for an `Rgb` value stated twice, which a `default_scheme` field names.
|
||||
Detection was considered and declined — crossterm exposes nothing portable.
|
||||
|
||||
**5. Is a face's *name* user-visible?** Emacs faces are user-facing identifiers people put
|
||||
in their init file, which makes renaming one a breaking change. If improvise's faces are
|
||||
only ever internal, the vocabulary can be refactored freely; if they are addressable by a
|
||||
user theme, the first published name is permanent.
|
||||
|
||||
> **"They should be external and I should be able to download a JSON dump of the current
|
||||
> theme, edit it and upload."**
|
||||
|
||||
Settled, and it settled more than it was asked. Face names are **external and permanent
|
||||
once published** — a compatibility surface on the footing of the `.improv` grammar. And it
|
||||
**ends the format deferral**, which had been conditional on improvise growing a
|
||||
configuration story: download-edit-upload is an export/import pair, not a config file, so
|
||||
the condition is dissolved rather than met and the larger question stays unmade. The format
|
||||
is **JSON**, which costs no new dependency anywhere. Clause 9 has all of it, including
|
||||
what "editable by hand" constrains.
|
||||
|
||||
**6. Should the `Rgb` arm be dropped?** Not one of the original five: it is the choice the
|
||||
other answers left standing, raised because clause 4's protection against losing a user's
|
||||
terminal theme is a property of the *shipped theme* rather than of the *type*, which is
|
||||
weaker than the unconstructible-bad-state shape this repo usually reaches for.
|
||||
|
||||
> **"Keep full color support in themes"**
|
||||
|
||||
Settled: **the arm stays.** A theme may express any colour a display can show, and what
|
||||
keeps the terminal palette alive is that improvise's own default theme declines to. The
|
||||
alternatives section carries it as a rejected alternative with the reasoning, rather than
|
||||
as an open question — and the weakness is now a **chosen** cost, which is the difference
|
||||
this answer makes. Read narrowly, as clause 4's own note says: this is about what the
|
||||
format may express, not about what the shipped default says.
|
||||
|
||||
## Deferred, with the condition that would settle each
|
||||
|
||||
**The file format is no longer here.** It was deferred on the condition that improvise
|
||||
grows a configuration story; the user's Q5 answer dissolved that condition rather than
|
||||
meeting it, and the decision is clause 9. The paragraph is not merely deleted: a reader
|
||||
who remembers the deferral should find out what happened to it, because a deferral that
|
||||
silently disappears is indistinguishable from one everybody forgot.
|
||||
|
||||
**Download and upload — follow-up work, with a browser half and a TUI half.** This is a
|
||||
*feature*, not a vocabulary question, and it is the one thing Q5 asked for that this ADR
|
||||
does not deliver. What it will need, so that the shape is not rediscovered: the browser
|
||||
half is a `Blob` download and a file input, neither of which touches the protocol; the TUI
|
||||
half is a path, and **that** is where the configuration question the format deferral was
|
||||
about actually arrives — a TUI writing a theme to a path is choosing a directory. So the
|
||||
browser half can land first and alone, and it should, because it needs no answer to
|
||||
anything. The condition for the TUI half is the one clause 9 names: it waits on improvise
|
||||
deciding how it is configured.
|
||||
|
||||
**A theme picker and live reload — deferred.** Both follow the vocabulary and neither is
|
||||
the hard part: once `Theme::resolve` exists, a picker is a field and a reload is a
|
||||
re-resolve. Building them before the vocabulary lands would fix the interface around an
|
||||
unfinished face set, which is the wrong order.
|
||||
|
||||
**Nothing about `Rgb` — the user ruled, and it is not deferred.** The draft left dropping
|
||||
the `Rgb` arm as a live choice; the answer is "keep full colour support in themes", so the
|
||||
arm stays and the alternatives section records it as a rejected alternative with its
|
||||
reasoning rather than an open one. What that fixes permanently is that **clause 4's
|
||||
protection is a property of the shipped theme rather than of the type** — a chosen cost.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good.** The browser and the TUI stop diverging, which is a fork that would otherwise widen
|
||||
every release. improvise-pcje becomes a matter of resolving faces the browser already has
|
||||
names for. `MinibufColor`'s doc comment becomes true. The nine stray hex values in
|
||||
`style.css` rejoin the token system. `context/code/tui.md`'s "the one that is wrong" section
|
||||
gets to be deleted rather than maintained. And a Swift or Qt front end starts from a
|
||||
vocabulary rather than from reading 125 literals.
|
||||
|
||||
Two more since the user's answers. The twelve hand-rolled `fg(Black).bg(accent)` sites
|
||||
become `reverse`, which **fixes a latent light-terminal bug** rather than merely tidying
|
||||
them — the ink stops being asserted. And a user can have their own theme without anyone
|
||||
building a theme *system* first: clause 9's JSON plus the browser's download half is a
|
||||
small feature, and it works because the vocabulary is the hard part and it is done.
|
||||
|
||||
**Bad, and this one first because it is the clause most carefully argued.** With `Rgb` in
|
||||
the vocabulary by decision, **a user's terminal theme is protected by what improvise
|
||||
ships, not by what the type permits.** A theme may state an absolute colour, and one that
|
||||
does will not track the terminal's palette — so the guarantee is "the default theme does
|
||||
not do this" and not "this cannot happen". That is weaker than this repo's usual bar,
|
||||
where `CellId`, `Row<T>` and `Checked` all make the bad state unconstructible, and it was
|
||||
said plainly before the ruling and is worth saying just as plainly after: **the ruling
|
||||
settles what we ship, not what the type prevents, and those are different guarantees.**
|
||||
What it buys is real — an exact brand colour, or a shade no sixteen-slot palette can
|
||||
express — and clause 4's table says which spelling costs what.
|
||||
|
||||
It is indirection where there is currently none: `Color::Yellow` is legible and
|
||||
`face_style(Face::GridColumnHeader)` requires a lookup to understand. That cost is real and
|
||||
is the honest argument for doing nothing. The answer is that the app already pays it three
|
||||
times (`axis_fg`, `field_kind_fg`, `active_color`) and nobody has objected — the indirection
|
||||
is only unfamiliar where it is absent.
|
||||
|
||||
It also moves `Axis::color()` out of `improvise-core`, making gotcha 26's "axis presentation
|
||||
lives on `Axis`" partially false until the repo map catches up, and it adds a resolution
|
||||
step to a rendering path that is currently a constant. The path is per-frame and the
|
||||
resolution is a short chain walk, so this is a nothing; it is listed because "it is
|
||||
obviously fast" is how a per-frame cost gets in.
|
||||
|
||||
And a face name becomes a **compatibility surface** the moment a theme is downloadable
|
||||
(clause 9). That is a genuine cost of accepting Q5's answer: the vocabulary stops being
|
||||
freely refactorable, and it stops being so *before* every face has been migrated, since
|
||||
the 125 literals move opportunistically. Names derived from domain enums (clause 2) are
|
||||
the mitigation, because they change rarely and visibly.
|
||||
|
||||
**Not addressed.** Fonts, spacing, and layout — a face system is colour and emphasis, and
|
||||
the browser's type scale and the TUI's column widths are not the same kind of fact.
|
||||
Accessibility contrast checking, which a theme makes *possible* to verify and does not
|
||||
itself provide. The download/upload feature itself. And nothing here touches what is
|
||||
drawn — only how.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,318 +0,0 @@
|
||||
# ADR 007, amendment 1 — Stable identities
|
||||
|
||||
**Ticket:** improvise-sboz (this amendment improvise-cq1q.1; implementation epic
|
||||
improvise-cq1q) · **Builds on:** ADR 001, ADR 002
|
||||
|
||||
## Status
|
||||
|
||||
**Accepted** (2026-10-09, PR #326). It amends ADR 007 (accepted 2026-09-25, PR #210,
|
||||
`007-stable-identities.md`). This file states only what the amendment changes or
|
||||
adds, numbered by the accepted ADR's clauses; everything else in ADR 007 is still
|
||||
true. Where the amendment replaces an ADR 007 rule or an existing behaviour:
|
||||
|
||||
| clause | accepted rule or existing behaviour | the amendment |
|
||||
|---|---|---|
|
||||
| 1 | groups have no id | a group has an id and a display, and its id behaves like every other id |
|
||||
| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill` |
|
||||
| 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays and referred to by id (`_measure`); the `## Category: _Measure` header lists the stored measures but declares nothing |
|
||||
| 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-<n>` beside a stored `profit` |
|
||||
| 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order |
|
||||
| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` |
|
||||
| 4 | ids sort by plain string order (`CellKey::new` sorts with `String::cmp`, and the writer orders data lines by their text) | ids sort in natural order. `CellKey`'s canonical coordinate order changes with it, so `RecordHash` values change (they are on the wire, not in files), and a saved file's data lines and their coordinates are written in the new order; the loader canonicalises keys, so the data read back is the same |
|
||||
| 4 | records mode string-sorts the store on entry (`DataStore::sort_by_key`, run by `SortData`) | records-mode rows come out in natural order, and so does the order `matching_positions` folds in (gotcha 18), so a `SUM`/`AVG` can change in the last bits |
|
||||
| 4 | the writer string-sorts `hidden:`, `expanded(…)`, `collapsed:` and `## Aggregation` lines | those lines are written in natural order too |
|
||||
| 4 | a GP fit string-sorts its records (`improvise-gp` `dataset.rs`) | the feature matrix's row order, and so a seeded fit's folds, follow natural order: a seeded fit can produce a different tree |
|
||||
| 4 | `app.rs` view-shape fingerprints and `Model`'s `Debug` string-sort ids | they sort naturally too; nothing visible changes |
|
||||
| 2, 5 | a collapsed entry and a group reference name a group by name | by its id |
|
||||
| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit |
|
||||
| 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id |
|
||||
| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders read display names in files and command-language text, which names things by id; interactive input matches a display or an id |
|
||||
| 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text |
|
||||
| 12 | the regression test commits the unedited text | the test edits a non-name token and commits |
|
||||
|
||||
## Context
|
||||
|
||||
Implementing the accepted ADR (epic improvise-cq1q) and using it raised points the
|
||||
accepted text does not settle: whether groups and measures carry ids, what a virtual
|
||||
category is in a file, exactly what slugify produces, which inputs may read display
|
||||
names, and which side resolves names in a formula edit. This amendment settles them.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. What carries an identity
|
||||
|
||||
**Groups have ids.** A group has an id and a display, like a category or an item, and
|
||||
its id is an ordinary id: unique in the document, generated by slugify (`group` is an
|
||||
empty slug's `<kind>`), declared beside its display, referred to by id, created by
|
||||
display, and renamed by a display rename. Its wire address is `GroupId`. In code,
|
||||
PR #336 (improvise-cq1q.12) already adds a display to `Group` beside its key; that key
|
||||
becomes the group's id.
|
||||
|
||||
**Virtual ids.**
|
||||
- **A leading `_` is reserved for virtual ids in general.** The virtual views have ids
|
||||
`_records` and `_drill`, displayed `_Records` and `_Drill`: they are addressed on the
|
||||
wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could
|
||||
collide with a user's view. Slugify never produces a leading `_`, so the grammar does
|
||||
not check the reservation.
|
||||
- **Virtual ids are lower case.** The code, the wire, `ViewState` and the browser
|
||||
client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, and move
|
||||
to `_index`, `_dim` and `_measure` in the re-key step.
|
||||
- **Virtual categories and measures are defined by criteria, not declared.** Each
|
||||
contains whatever in the document meets its criterion: `_measure` holds the measures,
|
||||
and `_index` and `_dim` hold the record positions. Their ids and displays are
|
||||
hardcoded in the application, so a file never declares one, and a line that refers to
|
||||
one spells its id: `_measure: column`, `region=east, _measure=revenue = 1200`.
|
||||
- **The stored measures are still listed in a `## Category: _Measure` section**, each
|
||||
with its display and id (`- Revenue(revenue), Cost(cost)`). Its header names the
|
||||
hardcoded virtual category and is not a declaration of it. The section is complete:
|
||||
it lists every stored measure, so a file never relies on its data lines to discover
|
||||
one. A formula's measure is not listed there; it is declared by its `## Formulas`
|
||||
line, which carries the id it shares with its formula. The loader recognises
|
||||
`_Measure`, `_Index` and `_Dim` by their hardcoded displays, and references are
|
||||
written `_measure`.
|
||||
- **Records mode's synthetic positions are values, not ids.** The row coordinate under
|
||||
`_index`, and by the same reasoning the column coordinate under `_dim`, are values of
|
||||
those virtual categories. They take no part in id uniqueness and cannot collide with
|
||||
a user's item.
|
||||
|
||||
**Formulas and measures.** A measure is the normal formula target (`profit = revenue -
|
||||
cost`); a dimension target is rare, for a derived dimension.
|
||||
- **A formula is a kind of measure, so a `_Measure` formula's id is its measure's id.**
|
||||
`Profit = Revenue - Cost` is one thing, `profit`: other formulas, each Profit cell's
|
||||
`CellKey`, and `ReplaceFormula`/`RemoveFormula` all name `profit`.
|
||||
- **Each measure's cells are keyed by one handle:** `ItemHandle::Item(id)` for a stored
|
||||
measure, `ItemHandle::FormulaTarget(id)` for a formula's measure, never both.
|
||||
- **A formula never merges into a stored measure.** With a stored `profit` (displayed
|
||||
`Profit`), adding `Profit = Revenue - Cost` creates a second measure with a fresh id,
|
||||
`profit-<n>`, by the ordinary collision rule. The stored measure keeps `profit` and
|
||||
its data; removing the formula removes only `profit-<n>`.
|
||||
- Both are named `Profit` in `_measure`'s scope, so they are shown as `Profit` and
|
||||
`Profit #2` (clause 2's suffix rule).
|
||||
- **A target outside `_Measure` is a real item** with its own `ItemId`, and its formula
|
||||
has a separate `FormulaId`: a formula `Tax` in `CustomCat` gets item id `tax`, then
|
||||
formula id `tax-<n>`.
|
||||
- **Formulas may refer to anything by id.** A function in a formula may accept any
|
||||
id-bearing thing (category, item, measure, formula, view, group) by its id. Whether a
|
||||
function takes the thing or its value (an item's display, say) is part of that
|
||||
function's signature.
|
||||
|
||||
### 2. Storage keys
|
||||
|
||||
- **`CellKey` pairs are `(CategoryId, ItemHandle)`**, so a `_Measure` formula cell's
|
||||
coordinate is `FormulaTarget` without claiming an `ItemId` the document does not
|
||||
hold.
|
||||
- **A collapsed entry names its group by the group's id**, and `Group` gains a
|
||||
`display`.
|
||||
- **Displays stay unique within a scope by suffixing.** The accepted text refuses a
|
||||
display already in use in the scope; a formula's measure beside a stored measure of
|
||||
the same name (clause 1) cannot be refused. So when several things in one scope have
|
||||
the same display, they are shown with a suffix: the first keeps the bare display, and
|
||||
the rest get ` #2`, ` #3`, … (a space, then `#` and the number) as a gapless
|
||||
sequence in the order their ids sort.
|
||||
- The order is the ids' natural order (clause 4), so `profit`, `profit-1`,
|
||||
`profit-2` and `profit-10` show as `Profit`, `Profit #2`, `Profit #3` and
|
||||
`Profit #4`.
|
||||
- Gapless: if the thing shown as `Profit #2` is removed, `Profit #3` becomes
|
||||
`Profit #2`.
|
||||
- The suffix is computed when a display is shown, from the things that share it, and
|
||||
is never stored or written to a file.
|
||||
- A suffixed display is a display like any other: interactive input that types
|
||||
`Profit #2` picks that thing (clause 6), and the ids stay the unambiguous name.
|
||||
- Old files cannot hold two things with one name in a scope, so loading one adds no
|
||||
suffix.
|
||||
- A suffix never repeats a display already in the scope. If something is literally
|
||||
displayed `Profit #2`, the shared `Profit`s take the next free numbers instead:
|
||||
`Profit`, `Profit #3`, `Profit #4`, still in id order. "Gapless" counts only the
|
||||
suffixes actually assigned, so a number taken by a literal display is skipped,
|
||||
like an id collision candidate that is already held.
|
||||
|
||||
### 4. How an id is made
|
||||
|
||||
**Slugify's steps:**
|
||||
1. **Casefold**, with Unicode full case folding (`Straße` → `strasse`).
|
||||
2. **Replace every character outside `a`–`z`, `0`–`9` and `-` with `_`.** Ids'
|
||||
alphanumerics are ASCII and include `-`; whitespace, punctuation and every non-ASCII
|
||||
letter become `_`.
|
||||
3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 or
|
||||
were in the display.
|
||||
4. **Remove every leading and trailing `_` and `-`**, repeating until neither end holds
|
||||
either.
|
||||
5. **If the slug starts with a digit, prepend `n`**, so no id is purely numeric or
|
||||
starts with a digit: `2024` → `n2024`.
|
||||
6. **If nothing is left, use `<kind>-<n>`** from the shared counter. A display of only
|
||||
`-`, only `_` or only non-ASCII letters ends here.
|
||||
|
||||
**The syntax of an id:** a lower-case ASCII letter, then `a`–`z`, `0`–`9`, `-` and
|
||||
single underscores, never ending in `_` or `-`. Slugify's output has this shape, so
|
||||
slugifying an id returns it unchanged, and a collision suffix `-<n>` keeps the shape.
|
||||
|
||||
| display | id |
|
||||
|---|---|
|
||||
| `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` |
|
||||
| `a_ b`, `a__b` | `a_b` |
|
||||
| `Q1 - 2024` | `q1_-_2024` |
|
||||
| `East `, `East_`, `-East-` | `east` |
|
||||
| `Region(code)` | `region_code` |
|
||||
| `Crème Brûlée` | `cr_me_br_l_e` |
|
||||
| `2024`, `_2024` | `n2024` |
|
||||
| `2024-Q1` | `n2024-q1` |
|
||||
| `Ü`, `-`, `%` | `<kind>-<n>` |
|
||||
|
||||
**A suffix and a hyphenated slug can look alike.** `East-1` slugs to `east-1`, which is
|
||||
also what the counter may make of `east`. They never collide, because every id a
|
||||
document holds is taken however it was made: a display slugging to a held `east-1`
|
||||
becomes `east-1-<n>`, and a held counter candidate is skipped.
|
||||
|
||||
**Ids are always sorted in natural order**, wherever ids are ordered. Split each id into
|
||||
alternating runs of digits and non-digits; compare the runs pairwise, a non-digit run by
|
||||
byte order and a digit run by value: strip its leading zeros, then the shorter run is
|
||||
smaller, then compare bytes (so runs of any length work, with no fixed-width parse); when
|
||||
two digit runs have equal value, the one with fewer leading zeros sorts first; when one
|
||||
id runs out of runs first, it sorts first. So `profit`, `profit-1`, `profit-2`, `profit-10`, and `a-7` before `a-007`.
|
||||
|
||||
**Hand-written ids obey every syntax rule of a generated id.** `east-1`, `item-4` and
|
||||
`q1_-_2024` load; `-east`, `east-`, `2024`, `a__b` and `_draft` are refused at load.
|
||||
|
||||
**Two branches of one file can merge into a file that will not load**, if each adds a
|
||||
different thing that generates the same slug: the merged file declares that id twice,
|
||||
and the load is refused naming both. Stable identity across copies of a file is not
|
||||
promised (improvise-8tk5).
|
||||
|
||||
**Built code.** `crates/improvise-core/src/model/id.rs` implements this clause
|
||||
(`context/code/core.md` §20): `slugify` follows the six steps, `is_id_syntax` is the
|
||||
syntax, `claim` refuses what a generated id could not be, `VIRTUAL_IDS` holds the five
|
||||
virtual ids, `IdKind::Group` exists, and `natural_cmp` and `display_suffixes` implement
|
||||
natural order and clause 2's suffixes. No other code calls them yet.
|
||||
|
||||
### 5. Persistence
|
||||
|
||||
- **The loader also refuses a hand-written id that breaks the syntax above**, naming
|
||||
the line; the grammar's `id_token` stays permissive. A declared id never begins with
|
||||
`_`, since a file never declares a virtual category.
|
||||
- **Formula text quotes only keyword ids** (`|sum|`). No id starts with a digit, and
|
||||
the formula grammar's bare identifier already admits `-`.
|
||||
- **Groups:** a group reference (`[coastal]`) and the group in a `collapsed:` line are
|
||||
ids, and a group definition declares a display and an id (`> Coastal(coastal)`).
|
||||
- **A header such as `## Category: Budget (2024)`** in a new-version file gives the id
|
||||
`2024`, which is refused. Import pipe-quotes `Region(code)` as
|
||||
`## Category: |Region(code)|(region_code)`.
|
||||
|
||||
**An axis line is tried before `format:`, and a format that reads as an axis line is
|
||||
pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice and `format_line`
|
||||
takes the rest of the line, so a category with id `format` would lose its axis line to
|
||||
the number format; the new-version arm puts `axis_line` first. Then `:set-format none`
|
||||
would write `format: none`, read as an axis line, so the writer pipe-quotes a format
|
||||
that would also match `axis_line` (an axis keyword alone or followed by `, <name>`, or a
|
||||
value starting with `|`), and the new arm's `format_line` is
|
||||
`pipe_quoted ~ &NEWLINE | rest_of_line`. An old file's `format: none` is read as before
|
||||
and written quoted on first save. Required tests: a category displayed `Format` on a
|
||||
row axis round-trips; a view whose format is each axis keyword round-trips beside a
|
||||
category `format` on an axis.
|
||||
|
||||
**A formula line.** Today:
|
||||
|
||||
```
|
||||
## Formulas
|
||||
- Profit = Revenue - Cost
|
||||
- Tax = Revenue * 0.1 [CustomCat]
|
||||
```
|
||||
|
||||
Under the amendment:
|
||||
|
||||
```
|
||||
## Formulas
|
||||
- Profit(profit) = revenue - cost
|
||||
- tax(tax-1) = revenue * 0.1 [customcat]
|
||||
```
|
||||
|
||||
- **A measure formula** has no tag: `_measure` is implicit and never written as a tag.
|
||||
The line is the measure's only declaration: `Profit` is its display and `(profit)`
|
||||
the id it shares with its formula.
|
||||
- **A dimension formula** has a tag, which is always the target category's id. The
|
||||
target is already declared with its display in its category, so the line names it by
|
||||
item id (`tax`), and `(tax-1)` is the formula's own id.
|
||||
- So the loader tells them apart by the tag. The expression is over ids; a reference to
|
||||
a formula target, and a `WHERE`/`IS` value, is the id text of its `ItemHandle`. A
|
||||
target display that looks like `Name(x)` is pipe-quoted: `- |Margin(%)|(margin) = …`.
|
||||
|
||||
### 6. Display names and the wire
|
||||
|
||||
- **Only CSV import and the `.improv` loaders turn display names in input text into
|
||||
ids**, where input text means files and command-language text. They read text people
|
||||
wrote. An old file's names are resolved there as a migration.
|
||||
- **Command-language text names things by id.** Anything that uses the command language
|
||||
as a scripting language is a script, wherever the text comes from (a file,
|
||||
`improvise cmd`, another program). A script naming something by display fails; it
|
||||
gets a display only through a dedicated function such as `display_for(id)`.
|
||||
- **Interactive input** (typing in the TUI or the browser, which is not input text)
|
||||
matches either a display or an id, as long as it
|
||||
is unambiguous which thing the user is picking. Ids win when it is not, and the
|
||||
display is shown as a preview (clause 12).
|
||||
- **Groups on the wire:** `AddItemInGroup`'s `group` is a group id, a new group is
|
||||
created by display, a group is renamed by id and new display, and `FrameGroup`
|
||||
carries the group's id and display.
|
||||
- **The formula panel** treats a display equal to another thing's id as ambiguous, so it
|
||||
never shows a bare name that ids-first would bind elsewhere.
|
||||
- **`ViewState::active_view` and `Workbook::active_view`** are keyed by id
|
||||
(improvise-p003).
|
||||
- **A `_Measure` formula-target row** is removed by `RemoveFormula` and renamed by a
|
||||
display rename of the measure (improvise-i0n7).
|
||||
- **CheckedBatch's created-record exemption is removed** (improvise-p003). It exists for
|
||||
frames like `[AddItem(East), SetCell(East)]`, and a frame cannot address what it
|
||||
creates, so nothing produces one.
|
||||
|
||||
### 7. Rejections
|
||||
|
||||
`Rejection::Gone(Named)` gains `Named::Group`.
|
||||
|
||||
### 8. `RecordHash`
|
||||
|
||||
`RecordHash` hashes `(CategoryId, ItemHandle)` pairs spelled as id text. Ids are unique
|
||||
across kinds, so it stays canonical; a record excludes `_measure`, so a `FormulaTarget`
|
||||
is never part of one.
|
||||
|
||||
### 11. Migration
|
||||
|
||||
- Step 2 adds `GroupId`, a display on `Group`, and `ItemHandle`.
|
||||
- Step 3 moves the virtual categories' keys to `_index`, `_dim` and `_measure`.
|
||||
- A new step 7 builds clause 12's client-side capture and resolver, ids first from the
|
||||
start.
|
||||
|
||||
### 12. Who resolves names in a formula edit
|
||||
|
||||
**The client resolves; the host never sees a display.**
|
||||
- The client holds the display→id mapping captured when the editor opened, as it holds
|
||||
`OpenedCell`. The TUI and the browser share one resolver in `improvise-protocol`.
|
||||
- **A name new to the edit** is resolved by the client against its id-bearing
|
||||
projection: a typed name equal to an id names that thing, even if it is also another
|
||||
thing's display; otherwise a display exactly one thing has names it; a display
|
||||
matching several things is refused with their ids; a name matching nothing is
|
||||
refused. Ids are matched exactly. Ids win wherever there is ambiguity, now and once an
|
||||
autocomplete exists; the autocomplete prefers id matches and previews the display.
|
||||
- **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are
|
||||
in id form; an id that no longer exists is `Rejection::Gone`.
|
||||
|
||||
**What ids-first costs.** Item A is displayed `East` (id `east`) and item B is displayed
|
||||
`east`, so it got `east-1`. Typing `east` names A; B is reached as `east-1`. The panel
|
||||
therefore shows B as `east-1`, never as a bare `east`.
|
||||
|
||||
**Tests.**
|
||||
- The accepted regression test's step 3 becomes: edit something that is not a name
|
||||
(`Revenue - Cost * 1`) and commit. Committing unedited text cannot tell a correct
|
||||
capture from none, because a client may pass unchanged text as a no-op
|
||||
(improvise-hexb).
|
||||
- Ids-first: with A (`East`, `east`) and B (`east`, `east-1`), typing `east` in text not
|
||||
captured at open must bind A (and fail under displays-first), `east-1` must bind B,
|
||||
and a display that is nobody's id must still resolve by display. A fixture where no
|
||||
display equals an id proves nothing.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Scripts that name things by display break and must be rewritten over ids.
|
||||
- Two measures can share a display, and are shown as `Profit` and `Profit #2`.
|
||||
- Old files' `_Measure` declarations and hyphen- or digit-led names change on first save.
|
||||
- `id.rs` implements clause 4; the code that sorts, shows and creates ids does not call it yet.
|
||||
|
||||
## Open questions
|
||||
|
||||
None.
|
||||
@@ -1,919 +0,0 @@
|
||||
# ADR 007 — Stable identities for categories, items, formulas and views
|
||||
|
||||
**Amended:** by `007-stable-identities-amendment-1.md` (accepted 2026-10-09, PR #326),
|
||||
which states every rule it replaces.
|
||||
|
||||
**Status:** accepted (2026-09-25). The user approved revision 7. Revision 8 then
|
||||
followed the adversarial review of that head: it adds clause 12, the ADR's own reading
|
||||
of when a formula edit's names become ids, and corrects stale facts about main. The
|
||||
user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17:10Z:
|
||||
"looks good, approved"). ·
|
||||
**Decided by:** the user; see "What the user decided" ·
|
||||
**Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 ·
|
||||
**Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and
|
||||
improvise-fdk1's `RenameRecordCoordinate` (landed in PR #221) ·
|
||||
**Builds on:** ADR 001, ADR 002 · **Depended on:** improvise-tt22 (remove legacy JSON),
|
||||
which landed in PR #224
|
||||
|
||||
**Numbering.** The ADR was first filed as 005, but improvise-l2b8 already claims ADR 005
|
||||
(exact decimal arithmetic) and improvise-ilvl claims 006. The review tickets filed
|
||||
against the first draft still say "ADR 005" in their titles; they mean this document.
|
||||
|
||||
**Direction from** the user, 2026-09-23, in three messages:
|
||||
|
||||
> "there would be a lot of benefit to categories/formulas having stable identities, even if
|
||||
> they were only transient: it would help pin editing operations and distinguish them from
|
||||
> delete/remove"
|
||||
|
||||
> "It's probably worth a file format change (in a opt-in way, that gets rectified on save if
|
||||
> the input file was missing them)"
|
||||
|
||||
> "Maybe something like `View: Default(default)` and the standard
|
||||
> `View: |Default (default)|(default)` for quoting."
|
||||
|
||||
## What the user decided
|
||||
|
||||
The user answered the **first draft** (`context/adr/005-stable-identities.md` at
|
||||
`9d97664c`) with inline comments on PR #210, and then posted one comment on the thread.
|
||||
Each answer is quoted verbatim below, with where it was attached and what it overturned.
|
||||
**Five of the first draft's positions were reversed.** The rest of this document is
|
||||
rewritten to match.
|
||||
|
||||
**1. Storage keys by id; the human name becomes a display field.** On first-draft
|
||||
line 92, the clause "`CellKey`, `DataStore`, views and formula text **stay keyed by
|
||||
name**":
|
||||
|
||||
> "Shouldn't it be easy to replace this to key by ID? There's no type change so, at worst,
|
||||
> it would be renaming field access. We could also add the name as something like
|
||||
> `display` and reserve `name` or whatever it is currently for the id."
|
||||
|
||||
**2. The point of ids is that keys use them.** On first-draft line 97, "ids do not make a
|
||||
rename cheap. They make it *expressible*":
|
||||
|
||||
> "This is a lot lower value if CellKeys, etc. don't refer to the new id"
|
||||
|
||||
**3. Every reference uses the id.** On first-draft line 275, the heading of clause 8,
|
||||
"ADR 002: `RecordHash` stays defined over names":
|
||||
|
||||
> "No, anything referring to something with an id should refer to the id as the stable
|
||||
> identity to avoid glitches."
|
||||
|
||||
This answers Q7 by reversing it, and reverses clause 5's "references carry names" too.
|
||||
|
||||
**4. Q1, the id's internal type.** On first-draft line 469, the recommendation to wrap "a
|
||||
small string":
|
||||
|
||||
> "We probably can reuse symbols as the internal type with interning?"
|
||||
|
||||
Ids are strings, interned through the existing `SymbolTable`.
|
||||
|
||||
**5. Q2, formula id placement.** On first-draft line 473, recommendation F1
|
||||
(`- Profit(3) = Revenue - Cost`):
|
||||
|
||||
> "Yes, before the equal to the right of the name"
|
||||
|
||||
**6. Q3, header syntax.** On first-draft line 480, the claim that "a header with a space
|
||||
before its parenthesis (`Sales (copy)`) is unaffected":
|
||||
|
||||
> "Space before the open paren is optional: document this edge case that might break
|
||||
> existing docs."
|
||||
|
||||
**7. Q4, data keys.** On first-draft line 485, the recommendation "no — names only":
|
||||
|
||||
> "In Data, we should prefer ids that are guaranteed unique to names that might not be."
|
||||
|
||||
**8. Import generates ids; legacy JSON goes.** Thread comment, 2026-09-23 06:51:
|
||||
|
||||
> "For CSV or JSON import, the dimensions are to be treated as the human friendly name and
|
||||
> never parsed for IDs: during import, ids should be generated by a slugify algorithm that
|
||||
> strips non-alphanumeric characters and replaces spaces with hyphens(collapsing runs).
|
||||
> The only supported native file format is .improv, cleanup any legacy JSON format."
|
||||
|
||||
Legacy JSON removal is its own implementation ticket, **improvise-tt22**. This ADR
|
||||
depended on it and did not implement it; it landed in PR #224. It also dissolves
|
||||
improvise-65kf: with no legacy JSON to load, there are no numeric JSON ids to accept.
|
||||
|
||||
**The answers change the ADR's centre of gravity.** The first draft put an id *beside*
|
||||
the name as an index. The user's answers make the **id the key** and the name a
|
||||
**display** string. So a rename becomes cheap: it changes one display field and moves no
|
||||
key, no surrogate and no basis tier. The first draft had called that impossible.
|
||||
|
||||
### Second round: answers on revision 2
|
||||
|
||||
The user answered revision 2 (`context/adr/007-stable-identities.md` at `5e46ce03`)
|
||||
with thirteen inline comments on PR #210, posted 2026-09-23 15:03–15:11Z. Each answer
|
||||
is mapped to the question **on the line it is attached to**, and quoted verbatim.
|
||||
|
||||
**9. Q5, group ids.** On line 468, inside Q5 ("Recommendation: not now … the one
|
||||
exception to 'every reference is an id'"):
|
||||
|
||||
> "This seems right"
|
||||
|
||||
Groups get no id in this ADR.
|
||||
|
||||
**10. Q6, per-entity versions.** On line 469, Q6 ("Recommendation: not in this ADR"):
|
||||
|
||||
> "Agreed"
|
||||
|
||||
**11. Q8, a duplicate or unknown id.** On line 473, the recommendation to refuse the
|
||||
load:
|
||||
|
||||
> "Yeah, don't load the file because you can't safely resave it"
|
||||
|
||||
**12. Q9, the reading of "opt-in".** On line 476, the recommendation that an id-less
|
||||
file is read forever and every save writes ids:
|
||||
|
||||
> "Yes"
|
||||
|
||||
**13. Q10, id reuse after a reload.** On line 485, inside option **(b)**, "Accept
|
||||
reuse across a reload (recommended)":
|
||||
|
||||
> "Yes, b"
|
||||
|
||||
This is option (b), and (b) was the recommendation. Within a session, retirement still
|
||||
prevents reuse. Across a save and reload, a removed name that comes back gets its old
|
||||
slug back.
|
||||
|
||||
**14. Q11, who assigns ids.** On line 492, inside option (a):
|
||||
|
||||
> "The host assigns"
|
||||
|
||||
**15. Q12, old-version files and the new header rule.** On line 502, inside option (a),
|
||||
"No (recommended)":
|
||||
|
||||
> "Yes, a as long as version is unambiguously marked"
|
||||
|
||||
Old files are read as they always were, on the condition that the version is marked
|
||||
unambiguously. Clause 9 says how the loader meets that condition.
|
||||
|
||||
**16. Q13, slugify.** Five answers, one per item:
|
||||
- **(a) where it applies.** On line 510:
|
||||
> "All id generation"
|
||||
- **(b) case.** On line 512, against the recommendation to keep case:
|
||||
> "lower case only"
|
||||
- **(c) `_` and `-`.** On line 515:
|
||||
> "Yeah, keep both"
|
||||
- **(d) non-ASCII letters.** On line 518:
|
||||
> "Strip non lower alphanumeric or -_"
|
||||
- **(e) the empty slug.** On line 520, the recommendation `<kind>-<n>`:
|
||||
> "Yes"
|
||||
|
||||
So an id is lower case, and its alphabet is `a`–`z`, `0`–`9`, `-` and `_`. Clause 4
|
||||
states the rule. Q15 asked the user to confirm the order of its steps (answer 18).
|
||||
|
||||
**17. Q14, formula text.** On line 529, the recommendation "ids in the file":
|
||||
|
||||
> "Display in the formula panel but not as column/row/page headers."
|
||||
|
||||
This does not contradict ids in the file, which answer 3 already requires. It answers
|
||||
**where formula text is displayed**, and it could be read two ways. That was Q16,
|
||||
which answer 19 settles.
|
||||
|
||||
### Third round: answers on revision 3
|
||||
|
||||
The user answered revision 3 (`context/adr/007-stable-identities.md` at `18c82e1c`)
|
||||
with three inline comments on PR #210, posted 2026-09-24 00:34–00:36Z. Each is mapped
|
||||
to the question on the line it is attached to, and quoted verbatim.
|
||||
|
||||
**18. Q15, the order of slugify's steps.** On line 571, Q15's recommendation "fold, then
|
||||
strip":
|
||||
|
||||
> "casefold, replace space with `_`, strip"
|
||||
|
||||
The fold comes first, as recommended. The answer also **changes answer 8's middle
|
||||
step**: a space becomes `_`, not `-`. Two details of that sentence are Q18.
|
||||
|
||||
**19. Q16, what the formula panel shows.** On line 584, the paragraph that declined to
|
||||
recommend a reading:
|
||||
|
||||
> "It should be at least `Profit (profit) = Revenue - Cost` but if `Revenue` is
|
||||
> ambiguous, you'll have to display the slugs. Working with slugs internally and making
|
||||
> the friendly names view-layer is almost always the right move here."
|
||||
|
||||
So the panel shows formula text over **display names**, with the target's id beside it
|
||||
in parentheses. Where a display name in the text is ambiguous, the panel shows the id
|
||||
instead. Ids are what the file and the model hold, and display names belong to the view
|
||||
layer. This also settles answer 17: the thing displayed "in the formula panel but not as
|
||||
column/row/page headers" is the target's **id**. Whether an id is enough to resolve an
|
||||
ambiguous name is Q19.
|
||||
|
||||
**20. Q17, capitals in a hand-written id.** On line 589, the question "May a
|
||||
hand-written id contain capitals?":
|
||||
|
||||
> "no."
|
||||
|
||||
This is option (a). `id_token` is lower case only, and a header ending in a
|
||||
parenthesised run that contains a capital loads as a whole-line display with a load
|
||||
warning. The virtual categories are spelled with capitals today. That is Q20.
|
||||
(Answer 21 later made a new-version file all-or-nothing, so that header is now refused
|
||||
rather than warned about. See clause 5.)
|
||||
|
||||
### Fourth round: answers on revision 4
|
||||
|
||||
The user answered revision 4 (`context/adr/007-stable-identities.md` at `024aa2d3`)
|
||||
with five inline comments on PR #210, posted 2026-09-24 06:05–06:08Z. Each is mapped
|
||||
to the question on the line it is attached to, and quoted verbatim.
|
||||
|
||||
**21. Q18(a), runs of spaces, and the collision suffix.** On line 658, Q18(a) ("Does a
|
||||
run of spaces collapse to one `_`?"):
|
||||
|
||||
> "runs of spaces are collapsed to one `_` and then `-###` numeric suffix is added if
|
||||
> duplicate (only when importing a file without IDs: a file must either have no IDs at
|
||||
> all or always have IDs)"
|
||||
|
||||
This decides three things:
|
||||
- a run of spaces collapses to one `_`, as recommended;
|
||||
- a duplicate slug gets a numeric suffix after a `-`;
|
||||
- **a file has ids on everything or on nothing.** A new-version file with a declaration
|
||||
missing its id is refused, like any file that cannot be resaved safely (answer 11).
|
||||
So the earlier rule that a hand-written id wins over a generated one in the same file
|
||||
is gone.
|
||||
|
||||
The parenthesis also limits the suffix to generating ids for a file that has none. Two
|
||||
details of that are Q22. Q18(b), what "casefold" means, was not answered in this round
|
||||
(answer 26 settles it).
|
||||
|
||||
**22. Q19, whether an id disambiguates formula text.** On line 678, option (a) ("Make
|
||||
item ids unique in the document"):
|
||||
|
||||
> "Ids have to be globally unique."
|
||||
|
||||
This is stronger than option (a). **No two ids in a document are equal, whatever their
|
||||
kind**, so a category, an item, a formula and a view never share one. An id in formula
|
||||
text or on the wire therefore always names exactly one thing.
|
||||
|
||||
**23. Q20, the virtual categories.** On line 691, option (a) ("Lower-case ids,
|
||||
capitalised displays"):
|
||||
|
||||
> "Virtual categories have IDs: we need to add a "slug" version such as `_index`, but
|
||||
> no non-virtual id can start with `_`"
|
||||
|
||||
The ids are `_index`, `_dim` and `_measure`, and the displays stay `_Index`, `_Dim` and
|
||||
`_Measure`. A leading `_` is reserved for them. Clause 4 says how slugify keeps a
|
||||
generated id out of that space, and clause 5 says what happens to a hand-written one.
|
||||
|
||||
**24. Q21, an old file naming an undeclared item.** On line 708, option (a) ("Declare
|
||||
the item at load"):
|
||||
|
||||
> "yes, this is right."
|
||||
|
||||
**25. The file version.** On line 731, the Consequences line "A new file version.":
|
||||
|
||||
> "File version should be added explciitly if it's not already part of the format."
|
||||
|
||||
It is already part of the format. The first line of every `.improv` file is a
|
||||
mandatory, exact version literal (`version_line = { "v2025-04-09" ~ NEWLINE ~ … }` in
|
||||
`improv.pest`), and the writer emits it. So the new format is marked by a **new
|
||||
literal**, and nothing is inferred. Clause 10, new in this revision, states the rule in
|
||||
one place.
|
||||
|
||||
### Fifth round: answers on revision 5
|
||||
|
||||
The user answered revision 5 (`context/adr/007-stable-identities.md` at `e044c968`)
|
||||
with three inline comments on PR #210, posted 2026-09-24 19:50–19:51Z. Each is mapped
|
||||
to the question on the line it is attached to, and quoted verbatim.
|
||||
|
||||
**26. Q18(b), what "casefold" means.** On line 766, the recommendation to take
|
||||
"casefold" literally, as full case folding:
|
||||
|
||||
> "full unicode"
|
||||
|
||||
Slugify's first step is **Unicode full case folding**, so `Straße` → `strasse`.
|
||||
|
||||
**27. Q22(a), how the suffix is spelled.** On line 771, the recommendation "a plain
|
||||
number":
|
||||
|
||||
> "Just a plane number."
|
||||
|
||||
The suffix is `-` followed by a plain decimal number, with no padding.
|
||||
|
||||
**28. Q22(b), collisions outside import.** On line 782, option (i), "Suffix it there
|
||||
too":
|
||||
|
||||
> "Any ambiguous id gets a suffix, if if the id is between different types of things
|
||||
> identified. Don't bother implementing a strict order here: `east` `east-1`
|
||||
> `east-2`, etc. just do what gensym does in CL: global increment-only counter appended
|
||||
> when dedupe needed."
|
||||
|
||||
This decides three things:
|
||||
- **every** colliding id gets a suffix: on import, in a session, and across kinds (a
|
||||
category and an item that slug alike);
|
||||
- the number comes from **one counter for the document that only ever increments**, as
|
||||
Common Lisp's `gensym` does, so it is not "the lowest free number";
|
||||
- the suffix is appended **only when a dedupe is needed**, so an id that does not
|
||||
collide stays bare.
|
||||
|
||||
Clause 4 is rewritten to match. The rules this replaces (lowest free number, and
|
||||
suffixing only on import) were revision 5's.
|
||||
|
||||
### Sixth round: answer on revision 6
|
||||
|
||||
The user replied at 2026-09-24 22:09Z to their own answer 28, on line 782 of
|
||||
`e044c968`:
|
||||
|
||||
**29. Hand-written ids are canonical.**
|
||||
|
||||
> "You have to treat the file's hand-written ids as canonical, doing otherwise will
|
||||
> corrupt the data section."
|
||||
|
||||
This constrains answer 28. The `gensym` suffix applies only to an id the program is
|
||||
**generating**, on import, when an id-less file loads, or when the host creates
|
||||
something in a session. **An id written in a file is never renamed**, because every
|
||||
data key and reference in that file spells it. Clause 4 says what follows:
|
||||
- a generated id that would collide with a file's id takes the suffix, and the file's
|
||||
id keeps its spelling;
|
||||
- two identical hand-written ids in one file are refused at load, not repaired.
|
||||
|
||||
## Context
|
||||
|
||||
**Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name.
|
||||
`CellKey` and `DataStore`'s interned symbols are names. Views, axis maps, page
|
||||
selections, hidden and collapsed sets, and formula text all refer by name. The user's
|
||||
first message names what that costs:
|
||||
|
||||
- **An edit cannot be told apart from a delete followed by an add.** No rename operation
|
||||
exists for a category or an item. A rename is remove plus add, which loses every cell
|
||||
(improvise-fz77). A formula is effectively identified by `(target, target_category)`,
|
||||
and `Model::add_formula` *upserts* on that pair. improvise-km71 (PR #214) made formula
|
||||
editing its own operation, with an interim id built from that pair (below).
|
||||
- **An edit whose target was concurrently replaced cannot be told apart from an edit of
|
||||
the replacement.**
|
||||
- **A records-mode coordinate rename names its category and item by name.**
|
||||
improvise-fdk1 (PR #221) landed `RenameRecordCoordinate { record, category, item }`
|
||||
with `String` operands (below).
|
||||
|
||||
**What already exists.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize`
|
||||
aliases, assigned from monotone counters. **No production caller uses either value.**
|
||||
`Model::add_category` and `Category::add_item` return them on the idempotent path, and
|
||||
nothing reads the returned value; the only reader is one test. Neither has ever been written to `.improv`. `View` has no
|
||||
id.
|
||||
|
||||
**`FormulaId` exists, as an interim.** km71 landed an opaque `FormulaId` whose private
|
||||
fields are `(target, target_category)`, built only by `Formula::id()`
|
||||
(`context/code/core.md` §18). It is on the wire in
|
||||
`ModelCommand::ReplaceFormula { id, was, raw }`, in `FormulaSlot::Existing` and in
|
||||
`FormulaEntry`. It was made opaque so that this ADR changes one struct rather than every
|
||||
signature. It is still a name pair, so a formula whose target is renamed gets a
|
||||
different `FormulaId`. `Model::replace_formula` therefore refuses to rename a target
|
||||
outside `_Measure` until this ADR lands (improvise-kb1d).
|
||||
|
||||
**Legacy JSON is gone.** improvise-tt22 landed in PR #224, so `.improv` is the only
|
||||
native format.
|
||||
|
||||
## Decision
|
||||
|
||||
**Categories, items, formulas and views each have an id, and the id is their key.** A
|
||||
category or item's current `name` field *becomes* the id (the user's suggestion), and the
|
||||
human name moves to a new `display` field.
|
||||
- **Every reference is an id.** That covers `CellKey`s, axis maps, page selections,
|
||||
hidden and collapsed sets, formula references, data keys, and wire operands that name
|
||||
an existing thing.
|
||||
- **Display names are for people.** They appear on screen, are typed into minibuffers,
|
||||
and are resolved to ids at the input edge.
|
||||
|
||||
Twelve clauses.
|
||||
|
||||
### 1. What carries an identity
|
||||
|
||||
| kind | id | unique within | wire address |
|
||||
|---|---|---|---|
|
||||
| category | yes | the document, across every kind (answer 22) | `CategoryId` |
|
||||
| item | yes | the document, across every kind (answer 22) | `ItemId` |
|
||||
| formula | yes | the document, across every kind (answer 22) | `FormulaId` |
|
||||
| view | yes (the user's example) | the document, across every kind (answer 22) | `ViewId` |
|
||||
| record | `RecordHash`, now over ids (clause 8) | the document | unchanged shape |
|
||||
| group | **no**, not in this ADR (answer 9) | | |
|
||||
|
||||
**Ids are unique in the whole document** (answer 22). One namespace covers categories,
|
||||
items, formulas and views, so an item id names one item without its category. A
|
||||
`CellKey` still pairs each item with its category, because the key needs the
|
||||
categories. An operation that creates an item still names the parent category.
|
||||
|
||||
**Virtual categories have reserved ids** (answer 23): `_index`, `_dim` and `_measure`,
|
||||
with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.**
|
||||
A `_Measure` formula target has no item of its own. Its handle is its
|
||||
formula's `FormulaId`, which is why the category panel's `_Measure` rows need
|
||||
`ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6).
|
||||
|
||||
### 2. Storage keys by id; `display` is the only place a human name lives
|
||||
|
||||
Following answers 1 to 3:
|
||||
- **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id,
|
||||
and each `Category`, `Item` and `View` gains a `display: String`.
|
||||
- **Cells and derived surrogates:** a `CellKey` is a sorted list of
|
||||
`(CategoryId, ItemId)` pairs. It stays sorted, now **by category id**, so gotcha 5's
|
||||
canonical form is kept with a different sort key. `DataStore` interns ids exactly as it
|
||||
interns names today.
|
||||
- **View configuration:** the axis maps, page selections, and hidden and collapsed sets
|
||||
are keyed by ids.
|
||||
- **Formulas:** a formula's references are ids (clause 5). The formula panel renders
|
||||
them as display names (answer 19; clause 6).
|
||||
|
||||
**What a rename now costs:** one `display` write. It moves no `CellKey`, no `RecordHash`
|
||||
and no `layout_version`, so an edit a client composed before the rename still lands after
|
||||
it. That is the "avoid glitches" the user's rule is for. The grid's labels are display
|
||||
strings, so the rename's effect must still dirty the grid and the panels
|
||||
(`Dirty::of` gotcha 35), but no basis tier moves.
|
||||
|
||||
**What the migration costs, stated honestly.** Every API that takes a name today
|
||||
(`Model::category(name)`, `Workbook::add_item(cat, item)`, `CellKey::new` callers, the
|
||||
formula evaluator's `Expr::Ref` lookup, `View` methods) takes an id instead. The
|
||||
display→id resolution moves to the places a human types, such as the minibuffer commits,
|
||||
the formula parser's reference resolution, and import. The change is mechanical but wide,
|
||||
and it is the bulk of the implementation. The user judged it cheap ("at worst, it would
|
||||
be renaming field access"), and for the field that is right. The call sites are the
|
||||
cost.
|
||||
|
||||
**Display names stay unique within their scope**, as names are today. So a human-typed
|
||||
name resolves to exactly one id, and a rename to a display name already in use is
|
||||
refused. An item's scope is its category, so two categories may each have an item
|
||||
displayed `Revenue`. Formula text crosses categories, which is where answer 19's
|
||||
"ambiguous" arises. Their ids differ (answer 22), so the panel resolves the ambiguity
|
||||
by showing the ids.
|
||||
|
||||
### 3. Ids are interned symbols
|
||||
|
||||
Following answer 4, an id is a **string**, and in memory it is a `Symbol` from the
|
||||
existing `SymbolTable`: copyable, with integer comparison. Two consequences.
|
||||
|
||||
- **A `Symbol` never crosses the wire or reaches the file.** It is an index into one
|
||||
process's table. The wire and `.improv` carry the id's **text**, and each end interns
|
||||
it on arrival.
|
||||
- **`Rejection` stops being `Copy`**, because `Rejection::Gone` (clause 7) carries an id's
|
||||
text. `basis.rs`'s `SaveFailed` carries no message *because* `Rejection` was `Copy`, so
|
||||
that reason lapses. Whether `SaveFailed` now gains its message is the implementer's
|
||||
call.
|
||||
|
||||
### 4. How an id is made: slugify, one rule everywhere
|
||||
|
||||
Following answers 8, 16 and 18, **every** id the app generates comes from the display
|
||||
name by **slugify**. That covers import, interactive creation, and assigning ids to an
|
||||
id-less `.improv` file at load (answer 16(a), "All id generation"). So a thing's id
|
||||
never depends on how it was made. The steps, in order (answer 18, "casefold, replace
|
||||
space with `_`, strip"):
|
||||
|
||||
1. **Casefold, with Unicode full case folding** (answers 16(b), 18 and 26). Rust's
|
||||
`str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped), so this
|
||||
needs a case-folding table or a small crate in `improvise-core`. It must pass the
|
||||
`wasm` gate, because that crate is in the browser bundle.
|
||||
2. **Replace each run of spaces with one `_`** (answers 18 and 21; answer 18 changed
|
||||
answer 8's `-`).
|
||||
3. **Strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`** (answers 16(c)
|
||||
and 16(d)). This strips non-ASCII letters, so an id never needs quoting in an
|
||||
`.improv` reference (clause 5 notes where formula text differs).
|
||||
4. **Remove any leading `_`** (answer 23: no non-virtual id may begin with `_`). The
|
||||
answer states the constraint. Removing the underscores is this ADR's way of meeting
|
||||
it, so `_Draft` → `draft`.
|
||||
5. **If nothing is left, use `<kind>-<n>`** (answer 16(e)), such as `category-1` or
|
||||
`item-4`. `n` comes from the same counter as every other suffix (answer 28), not
|
||||
from the lowest free number.
|
||||
|
||||
Examples: `East` → `east`, `Time Period` → `time_period`, `Coastal_East` →
|
||||
`coastal_east`, `Q1-2024` → `q1-2024`, `Region(code)` → `regioncode`, `Café` → `caf`,
|
||||
`Ü` → `category-<n>`, `Straße` → `strasse` (answer 26). `Time Period` and
|
||||
`Time_Period` slug alike, so whichever comes second gets `time_period-<n>`. `East` in
|
||||
`Region` and `East` in `Direction` slug alike too, and since ids are global (answer
|
||||
22), the second is `east-<n>`. Here `<n>` is the counter's next value, for example
|
||||
`east-1`.
|
||||
|
||||
The remaining rules:
|
||||
|
||||
- **A file has ids on everything or on nothing** (answer 21). An id-less file (old
|
||||
version, or a CSV or JSON import) gets every id from slugify. A new-version file must
|
||||
carry an id on every declaration, and one that does not is refused (clause 5). So no
|
||||
file mixes hand-written and generated ids.
|
||||
- **Any colliding id gets a suffix, `gensym`-style** (answers 21, 27 and 28).
|
||||
- **When:** whenever a generated slug is already taken, whatever kind of thing holds
|
||||
it. That applies on import, when an id-less file loads, and when the host creates
|
||||
something in a session. An id that does not collide stays bare.
|
||||
- **What:** `-` and the next value of **one increment-only counter per document**,
|
||||
as a plain number (`east-1`, not `east-001`). The counter never goes down and is
|
||||
never reused, so no order is enforced among suffixes: `east-7` may follow `east`
|
||||
directly.
|
||||
- **Retirement:** within a session a removed id is never handed out again. The
|
||||
counter has already moved past any suffix it issued, and a bare slug that was
|
||||
removed is still reserved in memory.
|
||||
- **Determinism:** the counter advances in file order or creation order, so the same
|
||||
input gives the same ids, never hash order (gotcha 18).
|
||||
- **Hand-written ids are canonical** (answer 29). The suffix only ever goes on the
|
||||
id being generated. An id read from a new-version file keeps its spelling, even
|
||||
when it looks like a slug or a suffixed one (`east-3`), because the file's data
|
||||
keys and references spell it. So when a generated candidate `slug-<n>` is already
|
||||
taken by a file's id, the counter advances again until the candidate is free.
|
||||
- **Two identical hand-written ids in one file are refused at load** (answers 11,
|
||||
22 and 29). By answer 22 they are a duplicate even when they declare different
|
||||
kinds of thing, such as a category and an item. Renaming either one is exactly
|
||||
what answer 29 forbids, and answer 11 already refuses a duplicate because such a
|
||||
file "can't safely resave". The error names both lines (clause 5). This follows
|
||||
from the answers, so it is not asked as a new question.
|
||||
- **The counter** lives in memory and starts at 1 when a document loads. It is not
|
||||
written to the file, since retired ids are not persisted (answer 13). That is
|
||||
this ADR's reading.
|
||||
- **Across a save and reload, reuse is accepted** (answer 13, Q10(b)). Retired ids are
|
||||
not persisted. So remove `East`, save, reload and add `East`, and the new item gets
|
||||
`east` again. An in-flight command from before the reload is refused by the
|
||||
generation tier. The one remaining risk is an outside reference that outlives the
|
||||
file, and none exists today.
|
||||
|
||||
### 5. Persistence: ids on declarations *and* references
|
||||
|
||||
Following answers 3 and 7, the file now carries ids everywhere:
|
||||
- **Declarations** (section headers, item lines, formula lines) carry the display name
|
||||
and the id.
|
||||
- **Every reference** carries the id: axis lines, `hidden:` and `collapsed:`, page
|
||||
selections, `## Aggregation`, `Initial View:`, `## Data` keys, and names inside formula
|
||||
text (answer 3). The formula panel shows display names instead (answer 19).
|
||||
|
||||
**Id alphabet.** Every id, generated or hand-written, is spelled only in `a`–`z`,
|
||||
`0`–`9`, `_` and `-` (clause 4; answer 20). The grammar spells them **with ranges**,
|
||||
because the generator drops `ASCII_ALPHANUMERIC` (improvise-rxz9). So an id never needs
|
||||
pipe quoting in an `.improv` declaration or reference. A header ending in a
|
||||
parenthesised run that contains a capital, such as `Sales(Copy)`, does not match
|
||||
`id_tag`, and neither does an empty or trailing-space tag. In a new-version file that
|
||||
declaration has no id, so the load is refused (answer 21), and the error names the
|
||||
line. Revision 4 generated an id and warned. A hand-written id beginning with `_`,
|
||||
other than the three virtual ids, is refused too (answer 23).
|
||||
|
||||
**Inside formula text an id may still need quoting.** The formula grammar's
|
||||
`bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So
|
||||
the writer spells an id in formula text through `quote::quote_name`, as it spells a
|
||||
name today. An id beginning with a digit (`2024`) or equal to a keyword (`sum`, `if`,
|
||||
`where`, `is`) is written `|2024|` or `|sum|`. No new rule is needed, but "never needs
|
||||
quoting" does not extend to formula text.
|
||||
|
||||
```pest
|
||||
id_token = @{ ('a'..'z' | '0'..'9' | "_" | "-")+ }
|
||||
// The space is optional (answer 6).
|
||||
id_tag = ${ " "* ~ "(" ~ id_token ~ ")" }
|
||||
|
||||
// Declarations: display name, then its id. The id is optional in the grammar so the
|
||||
// loader, not the parser, refuses a missing one and can name the line (answer 21).
|
||||
header_decl = { pipe_quoted ~ id_tag ~ &NEWLINE | header_display ~ id_tag ~ &NEWLINE | line_name }
|
||||
header_display = @{ (!(id_tag ~ NEWLINE) ~ !NEWLINE ~ ANY)+ }
|
||||
item_decl = { name ~ id_tag? }
|
||||
grouped_item = { "- " ~ name ~ id_tag? ~ "[" ~ name ~ "]" ~ NEWLINE ~ blank_lines }
|
||||
|
||||
// References: an id, never a display name.
|
||||
id_ref = { id_token }
|
||||
```
|
||||
|
||||
`model_name` has no id, so it keeps `header_name`. `Initial View:` becomes a reference
|
||||
and takes an `id_ref`. A group reference (`[Coastal]`) stays a name while groups have no
|
||||
id (answer 9).
|
||||
|
||||
**`header_display` is guarded only by a lookahead**, which the generator skips (gotcha
|
||||
34). So the generator may emit a display ending in something that later parses as an id.
|
||||
Round-trip property 3 still holds, because the first parse settles the split. But
|
||||
property 1 must be stated over the parsed result, not over the generated text.
|
||||
|
||||
**Which regime a file is read under is decided by its version line alone** (clause
|
||||
10). In an old-version file, a header is the whole line, a reference resolves against
|
||||
display names, and ids are assigned by slugify. In a new-version file, declarations
|
||||
carry ids and a reference is an id.
|
||||
|
||||
**An old file naming an item it never declares** declares it at load (answer 24). This
|
||||
happens today when a `## Data` line names an undeclared item (improvise-98pd, second
|
||||
source). The item gets a slug like any other, and the first save writes it, so a file
|
||||
that loads today keeps loading (the lesson of improvise-80rm). The NonItem work
|
||||
(improvise-gk51, PR #265) projects such a cell as `CellId::NonItem`. From a loaded file
|
||||
that case no longer arises. Ids do not settle 98pd's other two sources, `set-cell` and
|
||||
a stale page selection: a page selection holds an item id, and removing the item leaves
|
||||
it dangling exactly as a removed name does today.
|
||||
|
||||
**Slugify folds case (answer 16(b)), so old data keys change spelling on first save.**
|
||||
An old file's `Region=East` resolves at load through the display `East`. The first
|
||||
save writes the id: `Region=East` becomes `region=east`. The data is the same, but a
|
||||
`git diff` across that first save touches every data line.
|
||||
|
||||
**Round-trip properties:**
|
||||
1. an id-less (old-version) file loads, gets ids, and is written in the new version on
|
||||
its first save;
|
||||
2. a new-version file loads with every id kept verbatim;
|
||||
3. `format(parse(format(parse(x)))) == format(parse(x))`.
|
||||
|
||||
A **formula-line generator strategy** is required. The grammar walker emits no `=` and
|
||||
the walker drops such lines, so without its own strategy every formula property is
|
||||
vacuous (improvise-rxz9).
|
||||
|
||||
**A duplicate, unknown or missing id in a new-version file refuses the load** (answers
|
||||
11, 21, 22 and 29; a duplicate across two kinds of thing counts, and a hand-written id
|
||||
is never renamed to resolve one; answer 11:
|
||||
"don't load the file because you can't safely resave it"). The error names the id,
|
||||
and for a duplicate it names both declarations. Repairing silently is ruled out,
|
||||
because references carry ids (answer 3): reassigning a duplicate would silently change
|
||||
what its references point at.
|
||||
|
||||
**Legacy JSON is removed** (improvise-tt22), so it needs no migration here.
|
||||
|
||||
**Import** (CSV, JSON) never parses ids out of dimension names (answer 8). A header
|
||||
`Region(code)` is the display name `Region(code)`, and its id is its slug. The writer
|
||||
pipe-quotes any display name that would otherwise read as display plus id:
|
||||
`## Category: |Region(code)|(regioncode)`.
|
||||
|
||||
### 6. The wire: address by id, create by display name, learn ids from projections
|
||||
|
||||
- **The host assigns every id** (answer 14). A client never proposes one.
|
||||
- **Every operand naming an existing thing is an id.** The one operand naming the thing
|
||||
being created is a display name, and the host slugifies it:
|
||||
- `AddCategory(display)` and `CreateView(display)`;
|
||||
- `AddItem { category: CategoryId, display }`,
|
||||
`AddItemInGroup { category: CategoryId, display, group }` and
|
||||
`AddFormula { target_category: CategoryId, raw }`.
|
||||
- **A frame cannot address a thing it creates**, because it does not know the id the
|
||||
host will assign. So a create-then-act compound becomes **one command** carrying
|
||||
both halves. `create-and-switch-view` sends `[CreateView, SwitchView]` today and
|
||||
becomes `CreateView { display, switch: true }`. Any other compound of this shape
|
||||
gets the same treatment.
|
||||
- **New operations are born with ids:**
|
||||
- `RenameCategory { category, to: display }`;
|
||||
- `RenameItem { category, item, to: display }`;
|
||||
- `ReplaceFormula { id, was, raw }`, which km71 already landed with an interim id
|
||||
(below);
|
||||
- fdk1's `RenameRecordCoordinate`, which landed with name operands (below);
|
||||
- the removes (`RemoveCategory`, `RemoveItem`, `RemoveFormula`, `DeleteView`).
|
||||
- **Older variants** (`SetAxis`, `SetPageSelection`, `HideItem`, …) **must also move to
|
||||
ids**. Their operands are references, and a reference is now an id. They may move one
|
||||
whole variant at a time. That includes `SetCell { view, … }` and
|
||||
`ClearCell { view, … }`, which improvise-wpxr (PR #281, merged) gave a view named by
|
||||
its name, like every other view-scoped variant (improvise-lltt).
|
||||
- **Projections carry id and display together:**
|
||||
- `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }`;
|
||||
- `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows
|
||||
(answer 19): the target's display name with its id in parentheses, then the
|
||||
expression over display names, with an id wherever a display name is ambiguous
|
||||
(ids are unique, answer 22). `stored` is the id-form text the file carries, and it is what
|
||||
`ReplaceFormula`'s `was` is compared against. So a rename of `Revenue` elsewhere
|
||||
changes what the panel shows but does not refuse an edit of `Profit`, which is
|
||||
the "avoid glitches" of answer 3. What makes that hold for an edit already open
|
||||
is clause 12;
|
||||
- `ViewSummary { id, display, … }`.
|
||||
|
||||
`AxisFrame` labels stay display strings. They gain ids when a consumer needs them.
|
||||
- **`ViewState` keys by id**: the cursors, `expanded_cats`, and the view history
|
||||
(`view_back_stack` / `view_forward_stack`). Answer 1 makes this natural rather than a
|
||||
migration of its own. A view or category rename then orphans nothing.
|
||||
|
||||
### 7. ADR 001: an id that no longer resolves is gone
|
||||
|
||||
A command naming an id that does not resolve is rejected before anything is applied, as
|
||||
`Rejection::Gone(Named)`:
|
||||
- `Named` is one of `Category`, `Item`, `Formula` or `View`, and carries the id's text.
|
||||
- It is **structural**: force never creates structure.
|
||||
|
||||
The check has two halves, like the record tier:
|
||||
- **Before the frame:** an exhaustive `ids_named` feeds `Checked::against`.
|
||||
- **Within the frame:** an **id arm in `broken_chain`** refuses
|
||||
`[RemoveFormula(f), ReplaceFormula(f)]` (improvise-gkyo).
|
||||
|
||||
Per-entity versions are not added (answer 10).
|
||||
|
||||
### 8. ADR 002: `RecordHash` is now defined over ids
|
||||
|
||||
Following answer 3, `RecordHash` hashes the explicit serialization of the sorted
|
||||
`(CategoryId, ItemId)` text pairs, the same function over different strings.
|
||||
- **ADR 002's decision is unchanged in substance.** It still hashes an explicit
|
||||
serialization of a normalized record, with a fixed key and a width argued for.
|
||||
- **Its prose will be out of date once this lands.** ADR 002 says `CellKey::new`
|
||||
"sorts coordinates by category name", and a record is spelled in names. Both become
|
||||
ids. ADR 002 is **not edited here**. Updating that wording belongs to the
|
||||
implementation PR.
|
||||
|
||||
**What changes in behaviour:** a display rename no longer changes any `RecordHash` or
|
||||
bumps `layout_version`, because `view_shape` hashes ids too. So a client's surrogate
|
||||
survives a rename, and its in-flight edit still lands. A **coordinate** change, as in
|
||||
fdk1 or a records-mode move, still produces a different record and a different hash, as
|
||||
it should: it *is* a different record.
|
||||
|
||||
### 9. Headers that change meaning: the edge case answer 6 asks to document
|
||||
|
||||
Under answer 6, `header_decl` reads a trailing `(token)`, with or without a space before
|
||||
it, as the id. **It applies only to new-version files** (answer 15, Q12(a)). An
|
||||
old-version file is read as it always was, so its headers never split. In a new-version
|
||||
file the writer never produces an ambiguous header, because it pipe-quotes any display
|
||||
that would split. So the edge cases below arise **only when a new-version file is
|
||||
written by hand**. The table shows what the header rule would do to lines that exist
|
||||
in old files today. That is why old files do not get the rule.
|
||||
|
||||
| header line | whole-line reading (today) | under `header_decl` |
|
||||
|---|---|---|
|
||||
| `## View: Default(default)` | `Default(default)` | display `Default`, id `default` |
|
||||
| `## View: \|Default (default)\|(default)` | pipes literal | display `Default (default)`, id `default` |
|
||||
| `## View: Sales (copy)` | `Sales (copy)` | **display `Sales`, id `copy`** |
|
||||
| `## Category: Region(code)` | `Region(code)` | **display `Region`, id `code`** |
|
||||
| `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** |
|
||||
| `## Category: Time Period` | `Time Period` | unchanged; id `time_period` generated |
|
||||
|
||||
**The documents that change meaning** are those with a view or category header ending in
|
||||
a parenthesised run of letters, digits, `_` or `-`. **The app writes such headers
|
||||
itself**: `improvise import` on a CSV with headers like `Region(code)` or `Size(cm)`
|
||||
saves `## Category: Region(code)` today (improvise-80rm). Any view someone named
|
||||
`Sales (copy)` does the same.
|
||||
|
||||
**What it would cost if old files got the new rule** (the case answer 15 rules out):
|
||||
- The category is renamed on load: display `Region`, id `code`.
|
||||
- Every reference in that old file spells `Region(code)`, which resolves to neither the
|
||||
display nor the id.
|
||||
- So the axis lines and every data key for that category do not resolve.
|
||||
- Today's loader would silently drop the axis lines and orphan the data, so nothing is
|
||||
drawn.
|
||||
|
||||
**Decided (answer 15):** old-version files keep the whole-line reading, so none of
|
||||
this happens to a file that exists today. On the first save the writer pipe-quotes
|
||||
such a display, for example `## Category: |Region(code)|(regioncode)`, so the new file
|
||||
cannot split it either.
|
||||
|
||||
### 10. The file version is explicit, and it alone picks the regime
|
||||
|
||||
Following answers 15 and 25:
|
||||
- **The version is already explicit.** The first line of every `.improv` file is a
|
||||
mandatory version line, matched as an **exact literal**
|
||||
(`version_line = { "v2025-04-09" ~ NEWLINE ~ blank_lines }` in `improv.pest`), and
|
||||
`format_md` writes it. Nothing is added to make the version explicit. What is added is
|
||||
a second literal.
|
||||
- **The new format gets a new literal.** The grammar's `file` rule gets one arm per
|
||||
literal, and each arm uses its own header and reference rules. The new literal's
|
||||
exact spelling is left to the implementation.
|
||||
- **The regime is chosen only by the first line**, never inferred from what the body
|
||||
looks like. So the old literal means no ids anywhere, and the new literal means ids
|
||||
on every declaration (answer 21).
|
||||
- **A file whose first line is neither literal is refused**, as it is today.
|
||||
- **Every save writes the new literal** (answer 12). So an old file changes version on
|
||||
its first save.
|
||||
|
||||
### 11. Migration
|
||||
|
||||
1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224.
|
||||
2. **Types:** the id newtypes over `Symbol`, the `display` fields, and `FormulaId` and
|
||||
`ViewId`.
|
||||
3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with
|
||||
display→id resolution moved to every input edge.
|
||||
4. **Persistence:** the new version literal (clause 10), the grammar in clause 5, the
|
||||
loader and its refusals, slugify, the writer, and the round-trip properties with a
|
||||
formula-line strategy.
|
||||
5. **Projections and `ViewState`** (clause 6).
|
||||
6. **Vocabulary:** the id-addressed variants, `Rejection::Gone`, `ids_named` and
|
||||
`broken_chain`'s id arm.
|
||||
|
||||
### 12. A formula edit's names are fixed when the editor opens
|
||||
|
||||
**This clause is the ADR's own reading, not a user answer.** It closes a contradiction
|
||||
the adversarial review of `4830fa30` found between clause 6 and the km71 section, which
|
||||
is also ChatGPT Codex's P1 on `29ad14f4`.
|
||||
|
||||
**The problem.** The formula panel shows display names (answer 19), and the user edits
|
||||
that text. If the host maps those names to ids at **commit**, a concurrent rename breaks
|
||||
the edit one of two ways:
|
||||
- the name the user typed no longer resolves, so the edit is refused, which contradicts
|
||||
clause 6's "does not refuse an edit of `Profit`";
|
||||
- or the name now belongs to something else (`Revenue` renamed away, another item
|
||||
renamed to `Revenue`), so the edit silently binds to a different thing. That is the
|
||||
glitch answer 3 exists to prevent.
|
||||
|
||||
**The rule.** The display-name-to-id mapping is **captured when the formula editor
|
||||
opens**, and the commit resolves the edited text through that captured mapping. This
|
||||
is the capture-at-open rule the cell editor already follows: `OpenedCell` in
|
||||
`improvise-protocol/src/client.rs` records the record, its version, the document
|
||||
generation and the view at open, and the commit answers for those rather than for
|
||||
whatever the document holds by then.
|
||||
- The mapping covers every display name the panel showed in that formula's text,
|
||||
including the target's.
|
||||
- A name the user types that was **not** in the captured mapping is new to the edit. It
|
||||
is resolved against the document at commit, as any typed name is. If it does not
|
||||
resolve, the edit is refused and says which name.
|
||||
- A captured id that no longer exists at commit is `Rejection::Gone` (clause 7).
|
||||
- `was` is still compared against the stored id-form text (clause 6), so a concurrent
|
||||
*replace* of the formula is still seen.
|
||||
|
||||
**Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. From
|
||||
another client, rename the item `Revenue` to `Sales` and rename a different item to
|
||||
`Revenue`. Commit the unedited text. The formula must still reference the original
|
||||
item's id, and the commit must not be refused. The test must fail under
|
||||
commit-time resolution, which binds to the new `Revenue`, and it must fail under a
|
||||
refusal on rename. Both are the rivals this clause rules out.
|
||||
|
||||
## What this means for you
|
||||
|
||||
### improvise-km71 — editing a formula (landed as an interim)
|
||||
|
||||
km71 landed in PR #214, before this ADR, and was built to be revised by it
|
||||
(`context/code/core.md` §18):
|
||||
- **What landed.** `ModelCommand::ReplaceFormula { id: FormulaId, was, raw }`.
|
||||
`FormulaId` is opaque, with private fields `(target, target_category)`.
|
||||
`Model::replace_formula` writes in place, keeping the position. It refuses a new
|
||||
target that another formula owns, and it refuses any target change outside
|
||||
`_Measure` (improvise-kb1d). `was` is the text the editor opened on. It makes a
|
||||
concurrent *replace* visible, which even a stable id cannot do, so **`was` stays**.
|
||||
- **What this ADR changes.**
|
||||
- `FormulaId`'s fields become one id. The signatures that carry `FormulaId` do not
|
||||
change, which is what making it opaque was for.
|
||||
- A target change becomes a **display** rename of the target. The id does not move,
|
||||
and nothing that references the target is rewritten. So kb1d's refusal outside
|
||||
`_Measure` can be lifted, and so can the dangling-reference report core.md §18
|
||||
describes.
|
||||
- The panel shows `Profit (profit) = Revenue - Cost` (answer 19). The user edits that
|
||||
text. Its display names are resolved to ids through the mapping captured when the
|
||||
editor opened (clause 12), not through the names the document holds at commit, and
|
||||
`was` is compared against the stored id-form text (clause 6). The parenthesised id is not the user's to
|
||||
change, since the host assigns ids (answer 14). An edit that changes it is refused.
|
||||
This is the ADR's reading, not a user answer.
|
||||
- **Rejections.** Another client's deletion gives `Rejection::Gone(Formula(id))`. A
|
||||
deletion earlier in the same frame gives the broken-chain refusal.
|
||||
|
||||
### improvise-fz77 — renaming a category or item
|
||||
|
||||
- **The operations.** `RenameCategory { category, to }` and
|
||||
`RenameItem { category, item, to }`. The panel rows carry the ids.
|
||||
- **It is now cheap** (answers 1 and 2): one `display` write. No `CellKey`, axis map,
|
||||
page selection, hidden or collapsed set, or formula text is rewritten, because all of
|
||||
them hold ids. No `RecordHash` or `layout_version` moves, so in-flight edits survive.
|
||||
- **What it still owes:**
|
||||
- **Uniqueness.** Refuse a display name already used in the same scope.
|
||||
- **Dirtiness.** Mark the effect so the grid labels and panels are re-projected
|
||||
(gotcha 35). The labels are display strings.
|
||||
- **Tests.** Assert that the *labels* changed and the *surrogates did not*. That is
|
||||
the property this design exists for.
|
||||
- **It depends on step 3 of the migration**, the re-keying. Until then there is no cheap
|
||||
rename to build on.
|
||||
|
||||
### improvise-fdk1 — renaming a records coordinate
|
||||
|
||||
fdk1 landed in PR #221, before this ADR. What exists on main:
|
||||
- `ModelCommand::RenameRecordCoordinate { record: RecordHash, category: String, item:
|
||||
String }`, which moves a whole record to a new coordinate and creates the item if need
|
||||
be;
|
||||
- `CellId::Coordinate { record: RecordHash, category: String }`, the surrogate a
|
||||
records-mode coordinate cell carries.
|
||||
|
||||
**What this ADR changes**, as it does for km71's `FormulaId`:
|
||||
- `category` becomes a `CategoryId`, in the command and in `CellId::Coordinate`.
|
||||
- `item` becomes an `ItemRef`: `Existing(ItemId)` or `New(display)`. That is the
|
||||
create-by-display half of clause 6, since the host assigns the new item's id.
|
||||
- Unlike a display rename, this **changes a coordinate**, so the record's `CellKey`s and
|
||||
`RecordHash` change. It is a different record, and that does not change.
|
||||
|
||||
## Questions as asked, and where each now stands
|
||||
|
||||
The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14,
|
||||
revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All
|
||||
are answered.**
|
||||
|
||||
| # | question | my recommendation | status |
|
||||
|---|---|---|---|
|
||||
| 1 | Integer or word ids? | first draft: words; revision 1: integers only | **answered**: interned symbols (answer 4), generated by slugify (answers 8, 16). Words. |
|
||||
| 2 | Formula id placement | F1, `- Profit(3) = …` | **answered: F1** (answer 5) |
|
||||
| 3 | Headers in old files | first draft: accept the rare break; revision 1: gate on version | **answered**: the space is optional (answer 6), and old files keep the whole-line reading (answer 15, Q12) |
|
||||
| 4 | Do `## Data` keys carry ids? | no | **answered: yes, ids** (answer 7), reversing the recommendation |
|
||||
| 5 | Group ids? | not now | **answered: not now** (answer 9) |
|
||||
| 6 | Per-entity versions? | not in this ADR | **answered: not in this ADR** (answer 10) |
|
||||
| 7 | `RecordHash` over ids? | no, keep names | **answered: over ids** (answer 3), reversing the recommendation |
|
||||
| 8 | Duplicate id: repair or refuse? | revision 2: refuse | **answered: refuse the load** (answer 11) |
|
||||
| 9 | Reading of "opt-in" | read optional, write always | **answered: yes** (answer 12) |
|
||||
| 10 | Id reuse after a reload | revision 2: (b) accept reuse | **answered: (b)** (answer 13) |
|
||||
| 11 | Who assigns ids? | the host | **answered: the host** (answer 14) |
|
||||
| 12 | Old files and the new header rule | (a) no | **answered: (a), with the version marked unambiguously** (answer 15) |
|
||||
| 13 | Slugify details | all generation; keep case; keep `_` `-`; strip non-ASCII; `<kind>-<n>` | **answered** (answer 16): all generation, **lower case only** (reversing the recommendation), keep both, strip, `<kind>-<n>` |
|
||||
| 14 | Formula text: ids or display names? | ids in the file | **answered**: ids in the file (answer 3); the panel is Q16 |
|
||||
| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-` |
|
||||
| 16 | What the formula panel shows | none | **answered: display names, target shown as `Profit (profit)`, ids where a name is ambiguous** (answer 19) |
|
||||
| 17 | Capitals in a hand-written id | (a) no | **answered: no** (answer 20) |
|
||||
| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse**, plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) |
|
||||
| 18(b) | What "casefold" means | full case folding | **answered: full Unicode case folding** (answer 26) |
|
||||
| 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answer 22), stronger than (a) |
|
||||
| 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23) |
|
||||
| 21 | An old file naming an undeclared item | (a) declare it at load | **answered: (a)** (answer 24) |
|
||||
| — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal |
|
||||
| 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) |
|
||||
|
||||
## Still open
|
||||
|
||||
Nothing is open among Q1–Q22, and answer 29 settles the one case it raises: two
|
||||
identical hand-written ids are refused at load. **The user approved revision 7.
|
||||
Revision 8 adds clause 12, which is the ADR's own reading, and the user approved it
|
||||
("looks good, approved", 2026-09-25 00:17:10Z).** Three follow-ups from the same
|
||||
review stay as tickets, not ADR text: improvise-le4p, improvise-m56i and
|
||||
improvise-p003.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good.**
|
||||
- Edit, rename and replace are distinct, addressed operations.
|
||||
- A rename is one display write: it moves no key, surrogate or basis tier, and in-flight
|
||||
edits survive it.
|
||||
- `## Data` keys are unique by construction.
|
||||
- km71's `FormulaId` becomes a real id by changing one struct, and kb1d's refusal can be
|
||||
lifted.
|
||||
|
||||
**Bad.**
|
||||
- Re-keying storage is wide: every name-taking API moves to ids, and display→id
|
||||
resolution moves to the input edges.
|
||||
- A second version literal (clause 10), and every old file changes version on its first
|
||||
save.
|
||||
- `Rejection` loses `Copy`.
|
||||
- `Formula::raw` is no longer verbatim: the file carries formula text over ids.
|
||||
- The formula panel is a second rendering of formula text, over display names, so the
|
||||
host must resolve display names typed there back to ids (answer 19).
|
||||
- Ids are lower case, so an old file's data keys change spelling on its first save.
|
||||
- ADR 002's prose needs a wording update when this lands.
|
||||
|
||||
**Not addressed.** Per-entity versions (Q6), group ids (Q5), and ids in `AxisFrame` and
|
||||
`CellId::Derived`.
|
||||
@@ -1,447 +0,0 @@
|
||||
# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait
|
||||
|
||||
**Status:** accepted. ·
|
||||
**Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own
|
||||
epic) ·
|
||||
**Relates to:** ADR 001 (the basis a write claims), ADR 002 (surrogates on the wire),
|
||||
ADR 003, ADR 005, ADR 006, ADR 007, ADR 008
|
||||
|
||||
## Problem
|
||||
|
||||
Every change recomputes every formula cell (`Model::recompute_formulas`, called by
|
||||
`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That
|
||||
does not scale to a live source (ADR 003) streaming changes into a large table, nor to
|
||||
ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without
|
||||
building the grid. A change should cost in proportion to what it touches.
|
||||
|
||||
## Decision
|
||||
|
||||
Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A
|
||||
cell write becomes a delta, aggregates are maintained, and only the formula cells whose
|
||||
inputs moved are re-evaluated by the existing evaluator. The incremental results must
|
||||
equal today's eager results exactly.
|
||||
|
||||
**Principle: everything is as incremental as possible**, so the work done per model
|
||||
mutation is minimised. Where this ADR does work that grows with more than the
|
||||
mutation, it says so and names the incremental path that normally applies.
|
||||
|
||||
### 1. The `zset` crate
|
||||
|
||||
A problem-agnostic workspace crate with no dependencies beyond `std` (so it is
|
||||
wasm32-clean and serde-free). It provides:
|
||||
- `ZSet<K>`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group;
|
||||
- a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`,
|
||||
`differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join
|
||||
`join_indexed`, `distinct` and `aggregate`;
|
||||
- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it
|
||||
can run as a sliced job;
|
||||
- operator state behind one trace interface (look up, read a range, apply a delta), so
|
||||
later store-memory work can page or compress it.
|
||||
|
||||
**Operators are self-contained.** An operator's state and behaviour depend only on its
|
||||
inputs and its own state, never on a shared in-process global or on another operator's
|
||||
internals. That keeps the graph distributable: in principle the model could run across a
|
||||
cluster of machines. Nothing more is designed for that here.
|
||||
|
||||
There are no nested streams, no recursion operator and no timestamps beyond one clock.
|
||||
**The formula language is the query language.** What a request asks of the model beyond
|
||||
its axes and window (filters, aggregation, derived values) is written in formula text
|
||||
over ids (ADR 007). It goes through the same evaluator and circuit as a document
|
||||
formula, so there is no second query language.
|
||||
|
||||
### 2. The plug
|
||||
|
||||
improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the
|
||||
formula result at a key, and the raw fold of the cells that match a partial key. The
|
||||
incremental backend implements it in a seam crate, `improvise-incremental`, which
|
||||
depends on `zset` and improvise-core. improvise-core never depends on `zset`.
|
||||
|
||||
A live source (ADR 003) that reports its changes feeds cell deltas like `DataStore`. A
|
||||
source that answers aggregate queries but reports no changes implements the derived-value
|
||||
operator directly, by query pushdown, with no change feed.
|
||||
|
||||
The circuit decides *which* formula cells to re-evaluate. The one existing evaluator
|
||||
computes each value, so formulas are never compiled into operators.
|
||||
|
||||
**`Eager` is temporary.** Today's code moves behind the trait as `Eager`, unchanged. It
|
||||
exists only to show discrepancies between the new evaluator and the old one: a
|
||||
differential property test applies one random edit sequence to both backends and
|
||||
compares every drawn cell, total and occupancy count. Eventually it is deleted, and
|
||||
nothing depends on it surviving. Long-term tests compare against exact references, such
|
||||
as `ExactSum` against rational sums, not against `Eager`.
|
||||
|
||||
### 3. The cell delta
|
||||
|
||||
`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue)>`:
|
||||
- `+(k, v)` for a new cell;
|
||||
- `−(k, v₀) + (k, v₁)` for an overwrite;
|
||||
- `−(k, v₀)` for a removal.
|
||||
|
||||
Values compare by normal float rules, except that every NaN is one canonical value:
|
||||
`-0.0` equals `+0.0`, and an equal-value overwrite consolidates to nothing. NaN payloads
|
||||
are not preserved, so nothing can be smuggled through them. Today's record tier compares
|
||||
bits, so overwriting `0.0` with `-0.0` stops counting as a change. Multisets order values
|
||||
by `f64::total_cmp` after mapping every NaN to the canonical one and `-0.0` to `+0.0`. `CellKey` gains `Ord`, which is its canonical order: the
|
||||
natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The
|
||||
circuit steps once per model effect.
|
||||
|
||||
The record tier and the dirty flag read this delta instead of scanning the store. The
|
||||
dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the
|
||||
formula set moves, because formula edits move no cell.
|
||||
|
||||
Under ADR 007 and its amendment, `CellKey` pairs are `(CategoryId, ItemHandle)`. So a
|
||||
rename is a display write and emits no delta. The formula-set fingerprint must use the id
|
||||
form too, or a rename reads as a formula change.
|
||||
|
||||
### 4. Aggregates
|
||||
|
||||
A group is the cell key minus the None categories. One aggregation circuit serves every
|
||||
shape key with the same None list, which removes the per-session model clone.
|
||||
|
||||
| aggregate | state per group | per delta element |
|
||||
|---|---|---|
|
||||
| `Count` | count of values, count of numbers | O(1) |
|
||||
| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) |
|
||||
| `Min`, `Max`, text unanimity | a multiset of values | O(log d) |
|
||||
|
||||
A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and
|
||||
semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the
|
||||
ordering category's order. Those tickets choose the key. It must keep a reorder or an
|
||||
insert incremental: only the entries under the moved items change, and nothing is
|
||||
rebuilt.
|
||||
|
||||
### 5. Formulas
|
||||
|
||||
**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None
|
||||
categories". A formula counts at a stem where its top-level `WHERE` matches or raw data
|
||||
exists at the target.
|
||||
|
||||
**Inputs.** A formula cell is re-evaluated when one of its inputs moved:
|
||||
- a `Ref(R)`: the group at the stem with `R`'s category set to `R`;
|
||||
- an `Agg`: the group at the stem;
|
||||
- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set
|
||||
to `Item`. This is the one join: stems indexed by "stem minus `Cat`".
|
||||
|
||||
The affected pairs are evaluated in the formula graph's topological order.
|
||||
|
||||
**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline:
|
||||
1. A per-record stage retracts and reinserts each touched record. It reads the record's
|
||||
previous cells as the store minus the delta, so it keeps no copy of the values. The
|
||||
store is the `integrate` operator's trace over the cell delta, read as an input of
|
||||
the stage, so the stage stays self-contained.
|
||||
2. A map produces `(context, order key, expr)`. The order key is (context rank, record
|
||||
key in `CellKey` order), which is eager's concatenation order.
|
||||
3. The group aggregate, with an error channel that reads the first `Err` in order.
|
||||
|
||||
No cross product is built. A formula whose expression reads other records is evaluated
|
||||
over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is
|
||||
unchanged, because a context with no records contributes nothing. One exception: an
|
||||
inner bare ref that names no item errors at every context, so for it the ref is checked
|
||||
before enumerating. This replaces `Expansion::members`.
|
||||
|
||||
### 6. Circular references
|
||||
|
||||
DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the
|
||||
answer must be `Error("circular")`. The formula graph's strongly connected components
|
||||
are maintained incrementally as formulas change. Acyclic formulas are evaluated once
|
||||
each, in topological order. A cyclic component is maintained incrementally too: a change
|
||||
re-iterates only the cells it reaches. A component that does not converge within the
|
||||
existing bound of sixteen passes (`MAX_EVAL_DEPTH`) propagates `Error("circular")`, as
|
||||
today. An optimisation that speeds convergence is allowed only where it is exact: it
|
||||
gives the same result as the plain iteration, and a test compares the two.
|
||||
|
||||
### 7. Operator state follows the request's shape key
|
||||
|
||||
A projection request's **shape key** selects operator state, shared by every request
|
||||
with the same key, from any client. The key is:
|
||||
- the categories on each axis, by level;
|
||||
- the None categories;
|
||||
- the filter keys: each filter category's item, and the hidden items of categories on
|
||||
None, Page or Filter;
|
||||
- the page categories, so each page is a maintained slice and paging rebuilds nothing;
|
||||
- `measure_agg`.
|
||||
|
||||
The key uses the canonical parsed id form of the request's query text, so two spellings
|
||||
of one query share their operator state. Everything else in a request is applied when output is read: hidden and collapsed items
|
||||
on axis categories, item order, the page selection and prune. Occupancy and column
|
||||
aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item
|
||||
changes no operator. The visible axis is a remap.
|
||||
|
||||
- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas
|
||||
arriving during the build are buffered and added at install. The finished result is the
|
||||
response to the request that asked for it, not a notice.
|
||||
- **A formula edit** is a formula delta fed to the circuit.
|
||||
- **A document swap** rebuilds everything, and a new key's first build is a full pass.
|
||||
These are the only rebuilds; every other change is a delta.
|
||||
- **Saving a view definition** touches no operator.
|
||||
- **Live keys are a cache.** A key is evicted after a period in which no request names
|
||||
it. The period is a local duration on the host, set by measurement, and never compared
|
||||
with another machine's clock. Every eviction is announced (clause 9).
|
||||
|
||||
**Column totals and hidden rows.** Totals are kept over all rows, because the shape key
|
||||
is shared and a hide is one client's. Whether hidden items travel in a request is ADR
|
||||
008's open question. If they do not, a total is read as it is kept. If they do, a total
|
||||
must leave the hidden rows out, so a read subtracts the request's H hidden cells as a
|
||||
temporary overlay:
|
||||
- `Count` subtracts;
|
||||
- `Min` and `Max` skip keys whose whole count is hidden;
|
||||
- text counts the distinct keys left;
|
||||
- `Sum` subtracts exactly through the accumulator.
|
||||
|
||||
That is O(H log d) per read for every aggregate, with no view-store state in the model
|
||||
store.
|
||||
Aggregate-over-expression totals index their entries by row tuple for the same
|
||||
subtraction.
|
||||
|
||||
### 8. Exact sums
|
||||
|
||||
An f64 sum maintained by `+v` and `−v` drifts. Measured by
|
||||
`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million
|
||||
cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer
|
||||
differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both
|
||||
backends:
|
||||
- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which
|
||||
makes it an abelian group that rounds once, on read;
|
||||
- measured bit-equal to a fresh sum after a million cycles, and exact against rational
|
||||
sums on 20,000 cases;
|
||||
- 568 bytes per group, and about 1.3× the cost of an f64 add per update.
|
||||
|
||||
The sign of a zero sum follows IEEE rules for the same addends: it is `-0.0` only if
|
||||
every addend is `-0.0`, and an exact cancellation gives `+0.0`. It lives in ADR 005's
|
||||
`numtower` crate. **User-visible change:** displayed sums change in their last bits,
|
||||
because they become correctly rounded.
|
||||
|
||||
### 9. Delivery
|
||||
|
||||
The **model store** (the document model and the DBSP graph) and the **view store** (the
|
||||
cursor, the window and unsaved view definitions) are conceptual boxes, not a network
|
||||
boundary. Where a piece of state physically lives (a client, the host or a worker) is
|
||||
an implementation detail. The model store's decisions depend only on the model. A
|
||||
server-side twin of a client's state, such as its filter, watermark or window, may be
|
||||
held at the serving end of an edge where that makes the graph genuinely simpler. It is
|
||||
then part of the edge, not of the model store, and it is disposable: losing it never
|
||||
gives wrong data. A client that keeps its own watermark recovers with a correction from
|
||||
it. A client that relied on the twin for its watermark recovers with a full refetch.
|
||||
|
||||
**Every edge may cross a network.** An edge, meaning the stream from one operator to the
|
||||
next, may be in-process or remote. No edge assumes a function call or shared
|
||||
memory. An operator that sits at a wire boundary is just one implementation of an edge,
|
||||
an **edge adapter**. So every edge has the same semantics:
|
||||
- **A wire form.** Every element type that flows on an edge has a serializable form,
|
||||
defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays
|
||||
free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its
|
||||
tuple of ids, its position or an ADR 002 surrogate.
|
||||
- **The wire form is lossless.** Nothing is lost in transport, numeric precision above
|
||||
all:
|
||||
- a value never crosses as a JSON or JS double. It crosses in ADR 005's canonical text:
|
||||
an inexact finite value as `#i` with its shortest round-trip spelling, which keeps
|
||||
the sign of zero, infinities as `inf` and `-inf`, and every NaN as the one token
|
||||
`nan`;
|
||||
- positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as
|
||||
strings;
|
||||
- text crosses byte-exact, with no normalisation;
|
||||
- the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs
|
||||
a sum gets the host's exact read or rebuilds it from the exact deltas.
|
||||
|
||||
The stamp rule detects lost messages, but nothing would detect a lossy encoding. So
|
||||
every element type has a round-trip property test, where encoding then decoding is
|
||||
the identity for every value, including `-0.0`, infinities, 28-digit decimals and
|
||||
`i128`-wide fractions. A NaN round-trips as the canonical NaN.
|
||||
- **The clock is causal, never a wall clock.** Ordering, staleness and loss detection
|
||||
use only the stamp sequence (below), a logical clock the host advances once per
|
||||
committed command. Each process runs its own synchronous sub-circuit, and across an
|
||||
edge the stamped commands are the clock. No decision compares wall-clock times across
|
||||
machines, so clock skew cannot affect it. A delta crosses a wire only after its command
|
||||
commits, so a remote operator never sees a delta that is later rolled back.
|
||||
- **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces
|
||||
exactly: deltas are summed, and notices merged. A merged notice carries the range of
|
||||
publications it covers, `(from, to]`, so the client that merges notices sees no gap
|
||||
where nothing was lost.
|
||||
- **An edge is a session, not a connection.** An edge may be one long-lived connection,
|
||||
or a sequence of requests carrying the same session token or cookie, for example
|
||||
several HTTP requests. The host decides nothing from the token: it is transport, and
|
||||
any later authentication is out of scope. What continues across connections is the
|
||||
client's own watermark and shape keys, which it sends with each request. Every request
|
||||
can be retried, and a resync from a watermark is idempotent. A client asks for
|
||||
"notices after my watermark", and the host answers from **one shared, bounded log
|
||||
of published notices**: a single log, not one per edge or client, holding the notices
|
||||
themselves (keys and flags, not values). A request older than the log's start is
|
||||
answered with a correction (below).
|
||||
- **Failure.** Timeouts are local monotonic durations, never compared across machines.
|
||||
On a long-lived connection, both ends keep alive, and a host-side timeout is tied to
|
||||
that open connection. On a sequence of requests, only the
|
||||
client detects failure: a request that fails or times out. In either case the client
|
||||
then asks for a correction from its watermark.
|
||||
|
||||
- **Correction is Z-set arithmetic.** A correction is `target − held`, where `target`
|
||||
is the current state for the client's window and `held` is what the client holds.
|
||||
Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so
|
||||
the group structure of `ZSet` provides it, with no separate machinery. The client's
|
||||
position is a **watermark**: the whole stamp `(host instance, document_generation,
|
||||
publication)` up to which it has applied the stream, so everything at or before it is
|
||||
applied. A watermark from another host instance or generation is behind every current
|
||||
stamp. The client holds one per shape key it reads. A watermark is view-store state:
|
||||
the model store keeps none, so a client sends its own with each request unless a twin
|
||||
at the serving end holds it.
|
||||
- where the shared log covers the watermark and the covered range holds only moved
|
||||
keys, the host sends the current values of those keys, restricted to the client's
|
||||
window, and the client applies `target − held` over them. The log holds keys and
|
||||
flags, not values, so the model store keeps no view-store state;
|
||||
- where the covered range also holds axis changes, relabels or flags, which positions
|
||||
and labels depend on, the correction carries them too, in order, as a merged notice
|
||||
does, so the client re-derives positions incrementally;
|
||||
- only where the log no longer covers the watermark, or the covered range holds an
|
||||
oversized notice that lists no keys, does the host fall back to `target`
|
||||
for the whole window, and the client applies `target − held`;
|
||||
- a full refetch is `target − ∅`.
|
||||
|
||||
**Every read sees only the last committed state**, whether it is a correction, a
|
||||
fetch or `project`, and it is labelled with that state's stamp. A value a failed
|
||||
command wrote and then rolled back is never read. A correction is labelled with the
|
||||
publication range it covers, so it is applied only at the watermark it starts from,
|
||||
which makes it idempotent and resumable. Its elements are `(key, value)` pairs with
|
||||
weights, in the lossless wire form, so a client and a worker apply it with the same
|
||||
code.
|
||||
|
||||
Aggregation stays on the model side (ADR 008), so the operators that aggregate run on
|
||||
the model side. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger
|
||||
relies on, is a correction trigger below, followed from a displayed cell's key back to
|
||||
the model.
|
||||
|
||||
**A request** is a shape key plus a window, given as an absolute cross-product top-left
|
||||
and dimensions. The window is view-store state, and a request costs O(visible).
|
||||
- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings.
|
||||
- **Output** is keyed by tuple identity, with the position attached when sent.
|
||||
- **An item insert, removal or reorder** is sent as an axis change, and each client
|
||||
re-derives its positions.
|
||||
|
||||
**Pruning support.** Pruning is window-relative: a row or column empty within the
|
||||
displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model
|
||||
answers one restricted successor query: "the next k rows after position p with a value
|
||||
in a column of S", and the same for columns. A per-column sorted index of occupied rows
|
||||
answers it in O(|S| log n + k|S|), holding O(occupied cells). Whether pruning runs on
|
||||
the client and which parameters travel are ADR 008's questions, and this design supports
|
||||
either answer. If the client prunes, it prunes its window, fills forward with that
|
||||
query, and, when scrolling up or left, finds a new top-left by galloping backwards in
|
||||
doubling jumps (about 14 to 28 probes at density 10⁻⁴).
|
||||
|
||||
**The change notice.** Each command commits on its own (ADR 006), and after each
|
||||
committed command the host emits one notice, addressed to no client. A frame
|
||||
whose later command fails keeps its earlier commands, and their notices, published. A
|
||||
notice carries:
|
||||
- the stamp `(host instance, document_generation, publication)`. The publication number
|
||||
increments once per committed command. The host instance changes on restart; it is a
|
||||
random id, never a wall-clock time;
|
||||
- axis changes, display relabels as `(id, display)`, and a flag when the formula set,
|
||||
`measure_agg`, the view definitions or the default view moved. A shared display's
|
||||
` #2` suffix is never sent: the client computes it from the displays in scope;
|
||||
- the shape keys evicted since the last notice;
|
||||
- for each live shape key, the moved output keys (cells, formula cells and totals), as
|
||||
the circuit computes them. Moved records alone would miss formulas with cross-stem
|
||||
inputs.
|
||||
|
||||
**A client receives only what it needs.** Filtering is part of the DBSP graph, not a
|
||||
separate mechanism. A client either pulls just the state it needs, or pushes a filter to
|
||||
the host's side of its connection: a filter operator on its shape's query, written in the
|
||||
query language, through which its notices pass. Whether the filter also carries the
|
||||
client's window is an optimisation, since the model store's decisions depend only on the
|
||||
model. A pushed filter is held by the open connection, or by a twin. Otherwise the
|
||||
client sends it with each request. While a connection's filter is open, it counts as a request
|
||||
naming its shape, so its shape is not evicted.
|
||||
|
||||
A notice whose key lists exceed a fixed size carries "correct" instead. That is a
|
||||
fallback, and it costs every reader a correction.
|
||||
|
||||
**What a client does.** It intersects the moved keys with its window. A fold row has no
|
||||
maintained key, so it maps member keys through its own remap first. It then fetches what
|
||||
moved. It asks for a correction when it sees a publication gap, a new
|
||||
`document_generation` or host instance, its shape in the eviction list, a notice it
|
||||
cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of
|
||||
keys, the covered range of publications, and axis changes kept in order. Because the
|
||||
wire is lossless, outputs keep the circuit's value identity end to end.
|
||||
|
||||
**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`,
|
||||
`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no
|
||||
view-store state, only the shared notice log. `GridExtent` is derived by the client, and in records
|
||||
mode the record count is fetched. `Projection` also published the ADR 001 basis a write
|
||||
claims. Whether `project` returns it instead, and what replaces the layout tier, are
|
||||
ADR 008's questions.
|
||||
|
||||
**Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the
|
||||
circuit, the client's adapter is a sink for results and notices and a source of requests.
|
||||
Toward the client it is the mirror image. The client's end keeps the watermark. In
|
||||
process, an edge is a function call.
|
||||
|
||||
### 10. Column width hints
|
||||
|
||||
Every column incrementally maintains a width hint, which optimises grid sizing and
|
||||
reduces jitter. The hint is an integrated aggregate: a histogram from display width to
|
||||
count, so a retraction is O(log d). Where in the graph the hint is computed is a
|
||||
scheduling choice, made as an optimisation later.
|
||||
|
||||
### 11. Rollback and memory
|
||||
|
||||
- **Rollback.** The circuit's state joins ADR 006's per-command snapshot, which is taken
|
||||
before a command's effects and restored if the command fails. An inverse delta is not
|
||||
a rewind, because `delay` and `differentiate` keep history. The requirement is that
|
||||
this snapshot does not cost O(cells) per command against ADR 008's 8 ms budget, and
|
||||
that writes do not block reads. How is for the implementation epic.
|
||||
- **Memory.** Resident state beside the store is the multisets, `ExactSum` per summed
|
||||
group, occupancy per live shape, the successor-query index, the width histograms and
|
||||
the bounded notice log. The store itself is the integrated base and is never copied. It
|
||||
is measured at ADR 008's fixture before the store-memory work begins.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it
|
||||
fails to build for wasm32.
|
||||
- **Compiling formulas into operators:** a second evaluator.
|
||||
- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a
|
||||
second description of evaluation.
|
||||
- **A topic or subscription mechanism beside the graph:** filtering is an operator in
|
||||
the graph (clause 9).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:**
|
||||
- A write's evaluation costs O(delta) plus the formula cells it reaches.
|
||||
- The store scans and the per-view model clones go.
|
||||
- ADR 008 gets occupancy, totals and widths from one evaluator.
|
||||
- Live sources that report changes plug in.
|
||||
- The model store holds no view-store state.
|
||||
|
||||
**Bad:**
|
||||
- Two backends exist until `Eager` goes.
|
||||
- Integrated state is O(cells) beside the store.
|
||||
- `ExactSum` costs 568 bytes per summed group.
|
||||
- Each command snapshots the circuit's state for rollback.
|
||||
|
||||
**Order** (each step is a ticket in the implementation epic, behind the differential
|
||||
test):
|
||||
1. `zset`;
|
||||
2. the `Derived` trait with `Eager`;
|
||||
3. the cell delta, the record tier and the dirty flag;
|
||||
4. order-free aggregates and occupancy, measured at ADR 008's fixture;
|
||||
5. formulas and the aggregate-over-expression pipeline;
|
||||
6. `Sum`, `Avg` and totals through `ExactSum`;
|
||||
7. live sources;
|
||||
8. delivery.
|
||||
|
||||
Step 8's required tests:
|
||||
- a frame `[AddItem, SetCell]` publishes one notice per committed command, numbered
|
||||
consecutively, and no client asks for a correction;
|
||||
- a dropped notice is detected;
|
||||
- a reconnect after a host restart asks for a correction;
|
||||
- a corrected state equals the target, `held + (target − held) = target`, and
|
||||
correcting a state that is already correct is the zero Z-set (property tests over
|
||||
random edit sequences and losses);
|
||||
- a refused command consumes no publication number, and the commands before it in its
|
||||
frame stay published;
|
||||
- a dropped connection or a failed request triggers a correction, and a later request
|
||||
with the same session resumes from the watermark.
|
||||
- a zero sum has the sign IEEE gives the same addends: `-0.0` only if every addend is
|
||||
`-0.0`, and `+0.0` for an exact cancellation (a property test against IEEE).
|
||||
|
||||
## Open questions
|
||||
|
||||
None.
|
||||
@@ -1,214 +0,0 @@
|
||||
# Cloud Agents (LLM Reference)
|
||||
|
||||
Environment notes for agents running in cloud containers — Claude Code on the web,
|
||||
Cursor cloud agents, and similar setups. Local checkouts are unaffected.
|
||||
|
||||
## Session Bootstrap
|
||||
|
||||
All agent stacks use the same bootstrap script: `scripts/agent-bootstrap.sh`. It:
|
||||
|
||||
1. **Seeds nix flake inputs** (GitHub tarballs are blocked; seeds over git instead)
|
||||
2. Verifies the `bd` version matches the flake.nix pin
|
||||
3. Installs `bd` if missing
|
||||
4. Syncs the Dolt issue tracker from the git remote
|
||||
5. Warms the cargo build **inside `nix develop`**
|
||||
|
||||
**After bootstrap, `nix develop` works.** This is the required toolchain — bare system
|
||||
cargo that fails `edition2024` or missing wasm target is a misconfigured session.
|
||||
|
||||
**Claude Code**: `.claude/hooks/session-start.sh` calls the bootstrap automatically when
|
||||
`CLAUDE_CODE_REMOTE=true`.
|
||||
|
||||
**Cursor cloud**: `.cursor/environment.json` runs the bootstrap via `install`. Skills are
|
||||
in `.cursor/skills/` (symlinks to `.agents/skills/`).
|
||||
|
||||
**Other stacks**: Run `scripts/agent-bootstrap.sh` manually after checkout.
|
||||
|
||||
This doc is the "why", and the record of what bites when setup doesn't run.
|
||||
|
||||
---
|
||||
|
||||
## The issue tracker is not `issues.jsonl`
|
||||
|
||||
**`.beads/issues.jsonl` is a stale export.** It is a viewer/interchange
|
||||
artifact, not the source of truth — `.beads/README.md` says as much
|
||||
("Cross-machine sync and backups use Dolt remotes/backups, not JSONL
|
||||
import/export"). The real tracker is the Dolt database published to
|
||||
`refs/dolt/data` on the git remote.
|
||||
|
||||
The gap is not cosmetic. A session that hydrated from the JSONL saw 121
|
||||
issues against the remote's 168, with every P1 bug showing OPEN though all
|
||||
four were fixed and closed. Two beads were implemented from scratch that
|
||||
the remote had recorded as closed weeks earlier, and which turned out to
|
||||
be already implemented on an unmerged branch.
|
||||
|
||||
The hook syncs the database from the remote at session start. If it did
|
||||
not run, or reported a failure:
|
||||
|
||||
```bash
|
||||
bd list --all | wc -l # 121 means you are on the stale export
|
||||
```
|
||||
|
||||
**Verify any bead against the code before starting it.** `git log --oneline
|
||||
--all | grep -oE 'improvise-[a-z0-9]{3}'` names the ones already
|
||||
referenced by a commit, and `git log origin/main` is worth a look for work
|
||||
that landed without a bead reference.
|
||||
|
||||
---
|
||||
|
||||
## `bd init` is destructive here
|
||||
|
||||
A bare `bd init` in this repo rewrites `AGENTS.md`, `CLAUDE.md` and
|
||||
`.claude/settings.json`, installs Cursor/Codex integration files, and
|
||||
**commits all of it**. It also rewrites `sync.remote` in
|
||||
`.beads/config.yaml`.
|
||||
|
||||
Always pass the opt-outs:
|
||||
|
||||
```bash
|
||||
bd init --non-interactive --skip-agents --skip-hooks -q --remote git+<origin>
|
||||
```
|
||||
|
||||
Even then it commits a small `.gitignore` change, so the hook runs it in a
|
||||
throwaway directory and copies only `.beads/embeddeddolt` back. Do the
|
||||
same if you need to re-sync by hand; never run it in the working tree.
|
||||
|
||||
---
|
||||
|
||||
## What does not work in this environment
|
||||
|
||||
| Thing | Status | Use instead |
|
||||
|---|---|---|
|
||||
| `nix develop` **without bootstrap** | Fails: flake inputs are GitHub **tarballs**, which are blocked | Bootstrap seeds inputs automatically; if it failed, run `scripts/nix-seed-flake-inputs.sh` manually |
|
||||
| Bare system cargo | Fails: `edition2024`, missing wasm target | **Always use `nix develop`** — this is the required toolchain |
|
||||
| `bd dolt push` | Fails: 403 outbound | Nothing; bead status changes stay local. Say so rather than implying the tracker was updated |
|
||||
| `git+ssh://` Dolt remote in `config.yaml` | No ssh client in the container | `git+https://<origin>`, which the hook passes explicitly |
|
||||
| `git push`, GitHub API | **Work normally** | — |
|
||||
|
||||
Nix itself installs and runs fine (no init system, so the daemon needs
|
||||
starting by hand). Reads of public GitHub repos over the git protocol work —
|
||||
it is specifically the `/archive/*.tar.gz` and `codeload` endpoints that are
|
||||
refused.
|
||||
|
||||
### How nix develop works after bootstrap
|
||||
|
||||
The bootstrap runs `scripts/nix-seed-flake-inputs.sh` automatically, which seeds
|
||||
the flake inputs over git **without touching `flake.nix` or `flake.lock`**.
|
||||
After bootstrap, `nix develop` should work. If it fails, check `/tmp/nix-seed.log`.
|
||||
|
||||
Why the seeding is needed:
|
||||
|
||||
- Nix's `github:` fetcher wants a tarball. GitHub traffic **bypasses the
|
||||
environment's domain allowlist entirely** and goes through a proxy that
|
||||
scopes API and release-asset requests to repositories *attached to the
|
||||
session*. A flake input from any other owner gets a 403. Raising network
|
||||
access to **Full**, or allowlisting `codeload.github.com`, therefore does
|
||||
nothing.
|
||||
- Attaching the dependency repos is refused: `cross-tier adds are not
|
||||
supported in v1` once the session already has sources from another owner.
|
||||
- Nix has no setting that makes `github:` inputs use the git transport.
|
||||
|
||||
What does work is the lock's own integrity machinery. `flake.lock` records a
|
||||
`narHash` per input, and Nix looks a source up in the store and its
|
||||
substituters *before* fetching. The git and tarball fetchers produce
|
||||
byte-identical trees for a revision, so a `git+https` prefetch at the locked
|
||||
rev lands the input at exactly the store path the lock names:
|
||||
|
||||
```
|
||||
lock narHash: sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=
|
||||
git prefetch: sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=
|
||||
```
|
||||
|
||||
The script does that for every locked input. Nix still verifies each tree
|
||||
against the recorded hash, so this **satisfies** the lock rather than working
|
||||
around it — a mismatch fails loudly. Do not instead rewrite the lock's nodes
|
||||
to `git` type: carrying the old `narHash` onto a different fetcher is exactly
|
||||
the claim the lock exists to make.
|
||||
|
||||
The bootstrap runs this automatically. It takes a few minutes and several GB of
|
||||
store once the devShell builds. If bootstrap reported a seed failure, check
|
||||
`/tmp/nix-seed.log` and re-run `scripts/nix-seed-flake-inputs.sh` manually.
|
||||
|
||||
Two alternatives if per-container seeding is unattractive: push the inputs to
|
||||
a binary cache with `nix flake archive` and point sessions at it with
|
||||
`NIX_CONFIG`/`extra-substituters` plus a **Custom** allowlist entry for the
|
||||
cache host (not GitHub, so the allowlist does apply); or use a self-hosted
|
||||
environment, where egress leaves through your own network boundary.
|
||||
|
||||
`cache.nixos.org` is reachable, so the build itself substitutes normally.
|
||||
|
||||
---
|
||||
|
||||
## Sharp edges in `bd`
|
||||
|
||||
- **`bd dolt push <remote>` ignores its argument** and pushes to `origin`.
|
||||
Pointing it at a throwaway remote does not keep the real one out of it;
|
||||
only the 403 does. Treat the named form as unsafe.
|
||||
- `bd dolt pull` into a divergent local database fails with `local changes
|
||||
would be stomped by merge`. Re-sync from the remote instead of merging.
|
||||
- `bd` warns if `.beads` is not mode 0700.
|
||||
|
||||
---
|
||||
|
||||
## Working conventions
|
||||
|
||||
- One bead per pull request. Stacking a second bead onto an open PR has
|
||||
been rejected in review.
|
||||
- A merged PR cannot carry follow-up work: restart from the latest default
|
||||
branch and open a new PR.
|
||||
- Reconcile diverging history only by merge, including when bringing the
|
||||
default branch into a working branch. Never rebase onto main, and never do
|
||||
anything that needs a force push. Add a new Conventional Commit by default.
|
||||
Only an unpushed HEAD may be amended, and only for a small fix. Never
|
||||
reorder or fold commits (`.agents/skills/wave/agent-rules.md`, rule 8).
|
||||
- Keep `context/repo-map.md` and `context/design-principles.md` current —
|
||||
they are the two docs agents are expected to maintain by hand.
|
||||
|
||||
---
|
||||
|
||||
## PR Tracking
|
||||
|
||||
For any PR you own, **track both CI checks and GitHub comments/reviews**:
|
||||
|
||||
- **`scripts/pr-watch <n>`**: Waits for checks, then prints all reviews/comments (default).
|
||||
Reviews are a separate API surface — Codex/ChatGPT reviews arrive as *reviews* not comments.
|
||||
- **Answer feedback with attribution** before marking work done (see below).
|
||||
- **Re-check after push** that reviews target the current head SHA.
|
||||
|
||||
The `pr-watch` skill (`.agents/skills/pr-watch/SKILL.md`) is the agent-facing recipe.
|
||||
See AGENTS.md "PR Tracking: Checks AND Reviews" and "Shipping a wave" for the full workflow.
|
||||
|
||||
---
|
||||
|
||||
## GitHub Attribution (Hard Rule)
|
||||
|
||||
**Every GitHub write (PR, comment, review reply) must include an attribution line.**
|
||||
Posts go out under the repository owner's account — an unattributed post reads as the
|
||||
human's words.
|
||||
|
||||
Open the body with: `_Posted by <Agent Stack> (<model>)_`
|
||||
|
||||
Examples:
|
||||
- `_Posted by Claude (Opus 4.5)_`
|
||||
- `_Posted by Cursor Agent (claude-sonnet-5)_`
|
||||
|
||||
A thread is not "answered" until the reply carries this attribution. See AGENTS.md
|
||||
"GitHub Attribution" for the full rule.
|
||||
|
||||
---
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
**Use `scripts/kb`, not Prolog MCP.** The Prolog MCP is not available in all agent stacks
|
||||
(Cursor cloud agents do not have it). All agents must use `scripts/kb` for KB operations:
|
||||
|
||||
```bash
|
||||
scripts/kb check # kb_consistent, exactly what CI runs
|
||||
scripts/kb query '<goal>' # run a goal; exit 1 if it fails
|
||||
scripts/kb add <delta.pl> # add clauses; refuses if a key already exists
|
||||
scripts/kb replace <delta.pl> # add, overwriting existing keys
|
||||
scripts/kb rm <keys.pl> # remove by key; refuses if a key is absent
|
||||
```
|
||||
|
||||
Reading `prolog-sessions/kb.pl` directly for queries is fine. See the `kb` skill at
|
||||
`.agents/skills/kb/SKILL.md` for the full interface.
|
||||
@@ -1,540 +0,0 @@
|
||||
# The headless CLI
|
||||
|
||||
`improvise cmd` and `improvise script` are one function — `run_headless_commands`
|
||||
in `src/main.rs`; `run_headless_script` reads the file and delegates. Every claim
|
||||
below therefore holds for both, and a fix to one is a fix to both by
|
||||
construction rather than by remembering.
|
||||
|
||||
## 1 — the status line is the only channel, so it must be printed
|
||||
|
||||
`App` has no return channel for "what happened". A command's outcome — a
|
||||
refusal, a warning, a `fit-measure`'s reproducing seed — is written to
|
||||
`ViewState::status_msg` and rendered by the TUI's bottom bar. Headless, nothing
|
||||
read it, so **success and refusal were byte-identical: silence and exit 0**
|
||||
(improvise-q914, improvise-j6x8). Two things followed that are worth keeping
|
||||
in mind, because both were mistaken for correct behaviour by agents working
|
||||
above this layer:
|
||||
|
||||
- A P1's carefully-worded refusal can be written, tested at the effect, and
|
||||
still reach nobody, because the test asserts the *effect's* status and the
|
||||
surface that would show it drops it. That is this repo's characteristic
|
||||
defect one layer out, and the only cure is a test at the process boundary —
|
||||
`tests/headless_cli.rs`, which spawns the real binary.
|
||||
- `improvise cmd 'fit-measure …'` looked like it did nothing whether it worked
|
||||
or not, which made the whole GP feature read as broken from a script.
|
||||
|
||||
The loop takes the message with `std::mem::take` and clears it per command, so
|
||||
one command's status cannot be reported twice or attributed to its successor.
|
||||
|
||||
## 2 — stderr, all of it, and why no split was possible
|
||||
|
||||
The tempting design is stdout for a report and stderr for a refusal. **It
|
||||
cannot be built honestly**, because `status_msg` is a bare `String` with no
|
||||
severity, and both candidate classifiers are disproved by code in the tree:
|
||||
|
||||
- *"A refusal is a batch that aborted."* Only three effects set
|
||||
`abort_effects` (gotcha 21). `AddItem` writes `workbook.add_item`'s `Err`
|
||||
straight to the status and does not abort — and that is q914's own worked
|
||||
example of a refusal. So this classifier puts a refusal on stdout labelled a
|
||||
success.
|
||||
- *"A refusal is anything that spoke."* `cmd/fit.rs` sets a status on
|
||||
**success**, carrying the seed that reproduces the fit. So this one labels a
|
||||
deliverable a failure.
|
||||
|
||||
Guessing between them would be precisely the adjacent-check defect
|
||||
`context/design-principles.md` §7 names. So everything goes to stderr: it is
|
||||
the diagnostics stream, it leaves stdout free for a future data output, and it
|
||||
matches every other diagnostic `main.rs` already emits (`Saved to …`,
|
||||
`warning: …`, `Parse error: …`).
|
||||
|
||||
**No `--quiet` flag.** Silence is never the default and `2>/dev/null` is the
|
||||
standard suppression; a flag would be a second knob with nothing asking for it.
|
||||
|
||||
## 3 — the exit code (superseded by §10)
|
||||
|
||||
This section used to say: exit 1 on a parse error or on
|
||||
`App::last_batch_aborted()`, and a refusal that only writes a status exits 0,
|
||||
because closing that needed a severity beside `status_msg` — a `ViewState`
|
||||
change, and so a wire change (gotcha 25). Both halves were wrong in a way worth
|
||||
keeping. The abort half made a *success* exit 1 (improvise-c5ip), and the
|
||||
severity half named the wrong home: the question an exit code asks is about one
|
||||
batch, not about a message, so it belongs beside `batch_aborted` on `App` and
|
||||
never reaches the wire. §10 is what replaced it.
|
||||
|
||||
## 4 — a run that changed nothing does not write
|
||||
|
||||
`persistence::save` used to run unconditionally, so `improvise cmd` normalised
|
||||
the document on every invocation — `1e20` expanded to twenty digits. The cost
|
||||
was not cosmetic: an agent editing a fixture by string-replacing a value after
|
||||
one run got a silent no-op, and two misleading test results out of it.
|
||||
|
||||
The gate is `changed`, accumulated from `model_state.dirty` **after each
|
||||
command**, not read once at the end. Reading it at the end is wrong because
|
||||
`effect::Save` clears the flag, so `[add-item …, w other.improv]` would look
|
||||
unchanged and drop the edit from the `-f` document. `!path.exists()` is or-ed
|
||||
in so that naming a new file still creates it.
|
||||
|
||||
Over-approximation is the safe direction here, the same one `Dirty::of` takes
|
||||
(gotcha 35): a `Model(_)` effect that happens to write the value already there
|
||||
still costs one redundant save.
|
||||
|
||||
## 5 — the refusal names the word the user typed
|
||||
|
||||
`CmdRegistry::resolve` maps `formula` → `add-formula` *before* the arity check
|
||||
runs, so every validator that spelled its own name reported a command nobody
|
||||
typed (improvise-z05h). The fix moves the name to the one place that knows
|
||||
which of the two the user wrote: `require_args` no longer takes a `word`, and
|
||||
`CmdRegistry::parse` prefixes `"{typed}: "` onto any `Err`.
|
||||
|
||||
That removed 41 name literals from `cmd/`, which is the same argument gotcha 15
|
||||
makes about `registry.rs` having no name literals — a name spelled twice can be
|
||||
spelled wrong once. `interactive` deliberately does **not** prefix: `dispatch`
|
||||
in `command/keymap.rs` already formats `"{name}: {e}"` with the *bound* name
|
||||
(improvise-55l5, PR #126), and prefixing in both would double it.
|
||||
|
||||
The message states the **shape** rather than the arity, because
|
||||
"requires 1-2 argument(s), got 6" does not tell anyone to quote anything.
|
||||
|
||||
## 6 — the help's ex-commands were the unguarded surface
|
||||
|
||||
`help.rs` derives `HelpLine::Keys` rows from the keymap and drops any row whose
|
||||
command does not resolve, so a key row cannot go stale. `HelpLine::Literal` (an
|
||||
ex-command only, since improvise-wk18: built by `ex!`) and
|
||||
`HelpLine::Accent` rows are hand-written prose and had nothing checking them —
|
||||
which is how all three documented spellings of `:formula` came to be ones the
|
||||
parser refuses.
|
||||
|
||||
`every_ex_command_the_help_pages_show_is_one_the_parser_accepts` closes that
|
||||
surface rather than the one instance: every help line whose trimmed text opens
|
||||
with `:` is split on its colons and each piece handed to `parse_line`. The
|
||||
split exists because one row shows two examples side by side.
|
||||
|
||||
What it still cannot see, and should be closed if the welcome text moves again:
|
||||
`draw.rs::draw_welcome` builds its lines as a local array inside a ratatui
|
||||
render function, so nothing can walk them. Its `:formula` line was corrected by
|
||||
hand.
|
||||
|
||||
## 7 — a script's command is built against the cursor it will run against
|
||||
|
||||
`improvise cmd` and `improvise script` used to resolve every command through
|
||||
`CmdRegistry::parse` — the **pure** arm, which builds `CursorState::default()`:
|
||||
row 0, col 0, and a grid of zero rows and zero columns. The interactive arm,
|
||||
which builds `CursorState::from_ctx(ctx)`, was never reached from a script.
|
||||
|
||||
So every `Move` (`move-selection`, `scroll-rows`, `page-scroll`, the four
|
||||
`jump-*`) and `enter-advance` snapshotted a zeroed cursor at **parse** time,
|
||||
clamped to `(0, 0)`, and emitted `SetSelected(0, 0)`. That is worse than a
|
||||
no-op: it *resets* the cursor. Nothing could position it, so every
|
||||
cursor-reading command — `clear-cell`, `yank`, `paste`, `cycle-axis-at-cursor`,
|
||||
`hide-selected-row-item`, `edit-or-drill`, `drill-into-cell`,
|
||||
`toggle-group-under-cursor`, the three `delete-*-at-cursor`,
|
||||
`commit-and-advance` — was pinned to the origin (improvise-6uva). It was
|
||||
introduced with `CursorState` and the two-arm registry in April and predates the
|
||||
browser work by five months.
|
||||
|
||||
**A registry with two construction arms silently demotes every caller of the
|
||||
weaker arm, and nothing type-checks which arm a caller deserves.** That is the
|
||||
shape to remember, and it has a second instance in the same fix: the
|
||||
`register_pure` entries had an interactive arm that refused **empty** args while
|
||||
their pure arm accepted them, so `:w` (save to the current path) parsed one way
|
||||
and refused the other. Both arms are now `parse`, which is what "needs no
|
||||
interactive context" was supposed to mean;
|
||||
`a_pure_commands_two_arms_agree_on_no_arguments` is the guard.
|
||||
|
||||
The repair is `parse::segments` plus `parse::parse_segment_interactive`, and the
|
||||
loop re-parses **per segment**, because command *n* must see command *n-1*'s
|
||||
cursor — which means construction cannot be hoisted out of the loop. Line
|
||||
atomicity is preserved by keeping the existing whole-line `parse_line` as a
|
||||
**syntax gate** before anything in the line runs: a line that parsed before
|
||||
still parses, and a line that did not still does not, so the only behavioural
|
||||
change is which cursor the commands see.
|
||||
|
||||
**That atomicity claim was false as written** (improvise-2bac), and §8 is what
|
||||
makes it true. The gate runs the pure arm; the loop runs the interactive arm. A
|
||||
gate is only a gate if *every argument list the pure arm accepts, the
|
||||
interactive arm accepts too* — and `commit-cell-edit` broke it:
|
||||
`add-category Foo . commit-cell-edit editing 5 Foo/x` on an empty document
|
||||
passed the gate, applied the first segment, and then refused the second with
|
||||
`no cell at cursor`. Half the line landed.
|
||||
|
||||
The TUI's `:` line had the same defect and is fixed the same way, with one
|
||||
limitation that is not new: `ExecuteCommand` returns one effect batch, so the
|
||||
segments of `:a . b` all see the pre-batch cursor, exactly as a
|
||||
`Binding::Sequence` does (gotchas 21 and 23).
|
||||
|
||||
## 8 — the interactive arm is `parse`, then the cursor; it is never a second parser
|
||||
|
||||
#151 fixed three instances of *a registry with two construction arms demotes
|
||||
whichever caller reaches the weaker one*, one patch each. A post-merge review
|
||||
found the fourth (improvise-2bac): `commit-cell-edit <mode> <value> <coords>`,
|
||||
whose interactive arm read only `<mode>`, took the value from the `edit` buffer
|
||||
and the key from the cursor. So a script that wrote 5 to `Region/East`
|
||||
**deleted the cell under the cursor** — the empty buffer, committed — with exit
|
||||
0 and nothing on stderr. The fifth was already in the tree:
|
||||
`toggle-panel-and-focus <panel> false false` ignored both flags and toggled.
|
||||
|
||||
The shape rather than the instances is what changed. `CmdRegistry::register`
|
||||
used to take two free functions, `parse(args)` and `interactive(args, ctx)`,
|
||||
and nothing related them; each command re-derived its arguments twice and every
|
||||
disagreement was silent. The interactive arm is now **built from `parse`**, not
|
||||
beside it:
|
||||
|
||||
- `parse(args) -> C` — the only function that reads arguments a caller supplied.
|
||||
- `fallback: Option<(args, ctx) -> C>` — reached **only** when `parse` refused.
|
||||
This is where "no coordinates means the cursor" lives: `clear-cell` bare, and
|
||||
`commit-cell-edit <mode>` bare, which is what the keymap binds.
|
||||
- For `register_at_cursor` only, the registry itself then sets the command's
|
||||
**cursor** — its `CursorState`, reached through `AtCursor::cursor_mut` — to the
|
||||
live one. That is the whole of what the interactive arm adds to a parsed
|
||||
command, and no registration supplies it.
|
||||
|
||||
Three consequences hold by construction rather than by test. **Whatever `parse`
|
||||
accepts, the interactive arm accepts** — nothing after `parse` can fail — so the
|
||||
§7 gate cannot pass a segment the loop then refuses. **An accepted argument list
|
||||
cannot be overridden by the cursor**, because the only function that sees both
|
||||
args and context never runs on an accepted list. And **no field `parse` set can
|
||||
be overwritten from context**, because nothing a registration writes can name
|
||||
one: the cursor slot is typed `&mut CursorState`, and no command parses a
|
||||
`CursorState` out of its arguments. `register_pure` and `register_nullary`
|
||||
already had one arm; they are `register` with no fallback.
|
||||
|
||||
That last one is improvise-yf7p, and it is why the registry lost a parameter.
|
||||
#171 took a third function, `contextualize(C, &CmdContext) -> C`, handed the
|
||||
built command and not the args, and stated honestly that it could still
|
||||
**overwrite** a parsed field — naming the field was "a visible line in review".
|
||||
Review then injected five such overwrites on commands other than the one with a
|
||||
test (yank, paste and drill-into-cell taking their key from the cursor over the
|
||||
parsed one; toggle-panel-visibility ignoring its parsed panel; move-panel-cursor
|
||||
clamping its parsed delta to ±1) and the whole workspace stayed green. A hole
|
||||
guarded by review discipline plus one command's test is not guarded. Every
|
||||
contextualizer in the tree fell into one of two shapes, and each had an answer
|
||||
that removes the function rather than checking it:
|
||||
|
||||
- **The cursor** (`Move`, `EnterAdvance`, `CommitAndAdvance`). Always one
|
||||
`CursorState` field, always `CursorState::from_ctx`. So the registry does it,
|
||||
through a slot whose *type* is the guard: an `AtCursor` impl pointing at a
|
||||
`CellKey` or a delta is a compile error, which was tried.
|
||||
- **A read of context that belonged in `execute`** (`toggle-panel-visibility`'s
|
||||
`currently_open`, `cycle-panel-focus`'s three flags, `move-panel-cursor`'s
|
||||
cursor and bound). These copied context into the command only to read it back;
|
||||
`execute` now reads `ctx` directly, the same answer `toggle-panel-and-focus`
|
||||
got below. Those three are `register`/`register_nullary` now, so their
|
||||
interactive arm is exactly `parse`.
|
||||
|
||||
What `execute` does with a parsed field is still the command's own logic and is
|
||||
tested as such — `toggle_panel_visibility_reads_the_panel_it_names` and
|
||||
`move_panel_cursor_moves_from_the_named_panels_own_cursor` fail if the panel or
|
||||
the delta is ignored there. The difference is that the registry is no longer a
|
||||
second place such a bug can be written.
|
||||
|
||||
What is still representable, stated rather than hidden:
|
||||
|
||||
- A `fallback` sees the args `parse` refused and could ignore some of them.
|
||||
`commit-cell-edit`'s refuses anything but a lone mode, so the malformed
|
||||
`commit-cell-edit editing 5` is an error on the `:` line — which has no pure
|
||||
gate — rather than a commit of the edit buffer to the cursor cell.
|
||||
- The gate is **stricter** than the loop wherever a fallback exists: a bare
|
||||
`clear-cell` is refused in a script. That is the safe direction — the whole
|
||||
line is refused — and it is unchanged from before #151.
|
||||
- A new command could declare a *second* `CursorState` field and parse it from
|
||||
args; `cursor_mut` could then point at the wrong one. No command has one, and
|
||||
a parsed cursor would be a new idea deserving its own argument.
|
||||
|
||||
`toggle-panel-and-focus` did not fit #171's `contextualize` either, because its
|
||||
pure arm defaulted the omitted flags to `true` and a contextualizer cannot tell an
|
||||
omitted flag from an explicit one. So the omission is now in the type —
|
||||
`target: Option<(bool, bool)>`, `None` meaning *toggle* — and the toggle is
|
||||
resolved in `execute` against the same context. That is the honest home: the
|
||||
decision depends on context, and `execute` is where context is.
|
||||
|
||||
Guards: `every_argument_list_the_gate_accepts_the_run_accepts_too` enumerates
|
||||
**every registration** against every candidate argument list, in a context whose
|
||||
cursor sits on no cell — the context in which the old `commit-cell-edit` arm
|
||||
refused. It lists no commands, so a new registration is covered the day it
|
||||
lands. It is backed by the process-level tests in `tests/headless_cli.rs`.
|
||||
|
||||
## 9 — the extent is fixed, because it reaches the document
|
||||
|
||||
A headless run used to read `crossterm::terminal::size()`, and the page size
|
||||
derived from it decides where `page-scroll` lands and therefore which record the
|
||||
next cursor-reading command writes (improvise-up8g). So one script wrote a
|
||||
different document in a different window, and with no tty — CI, cron — the read
|
||||
failed and left `App`'s default, so CI and a shell disagreed too. It was
|
||||
invisible until §7 let a script see its cursor.
|
||||
|
||||
`HEADLESS_EXTENT` in `main.rs` pins it to 80x24 — the size CI already had, so
|
||||
nothing that passed there moves. There is **no flag** to change it, for §2's
|
||||
reason: a second knob with nothing asking for it. A script that wants a
|
||||
particular row says so with `move-selection` or `jump-*`, which do not depend on
|
||||
any window.
|
||||
|
||||
What this does not close: the **TUI** still takes its extent from the terminal,
|
||||
and that is correct there, because the window really is an input to an
|
||||
interactive session. The ticket's wider claim — that a replayable log needs the
|
||||
extent in it (improvise-ilvl) — is about that log, which does not exist yet; a
|
||||
headless run has nothing to replay *except* its script, so pinning is the whole
|
||||
fix for this surface.
|
||||
|
||||
## 10 — the exit code asks whether the change happened, never whether a chain stopped
|
||||
|
||||
`App::last_batch_aborted()` answered the wrong question for an exit code, in
|
||||
both directions at once.
|
||||
|
||||
- **A success that exited 1** (improvise-c5ip). `commit-cell-edit` ends in
|
||||
`EnterAdvance`, which emits `AbortChain` when the cursor is already
|
||||
bottom-right or the grid is empty: *the advance's premise no longer holds*.
|
||||
The write had already landed and was saved, and the run exited 1 with
|
||||
nothing on stderr. Before §7 that was every scripted commit.
|
||||
- **A refusal that exited 0** (improvise-udt2). `AddItem` into a missing
|
||||
category writes the error and does not abort, because nothing after it
|
||||
depends on the item. So the flag never saw it.
|
||||
|
||||
An abort is a statement about the **chain**; a refusal is a statement about the
|
||||
**request**. `AbortChain` stops a chain without refusing anything, and
|
||||
`AddItem` refuses without stopping anything. The two coincide only for
|
||||
`AddCategory` and the file writes, which is why reading one for the other
|
||||
looked right for so long.
|
||||
|
||||
So `App` carries a second per-batch outcome, `batch_refused`, reset where
|
||||
`batch_aborted` is and written by exactly one method, `App::refuse(msg)`, which
|
||||
sets the status and the flag together — a refusal cannot write its message and
|
||||
forget the flag. Commands reach it through `effect::refuse`, the `Refuse`
|
||||
effect, beside `effect::set_status` for reports. `main.rs` exits 1 on
|
||||
`last_batch_refused()` and no longer reads `last_batch_aborted()` at all.
|
||||
|
||||
**Why not a severity on `status_msg`**, which udt2 proposed. A severity on the
|
||||
message would live in `ViewState`, so it would be portable state sent to every
|
||||
client in a `Snapshot`, and it would go **stale**: every one of the ~60 writers
|
||||
of `status_msg` would have to reset it or leave the previous message's severity
|
||||
in place. A per-batch flag cannot go stale, because `apply_effects` clears it
|
||||
before every batch, and it is not in `ViewState`, so nothing on the wire
|
||||
carries it or has to agree with it. The TUI colouring the ticket hoped for is
|
||||
still possible later, and would want the severity *on the message*; that is a
|
||||
separate change with a separate staleness problem.
|
||||
|
||||
**The classification is per writer.** Every refusal in `ui/effect.rs`, in
|
||||
`cmd/{cell,commit,fit,grid,mode,panel,search,text_buffer}.rs` and in
|
||||
`command/keymap.rs::dispatch` goes through `refuse`. The last of them were
|
||||
converted in two steps:
|
||||
- **improvise-i3o5:** `EditOrDrill`'s collapsed-group arm, and the three
|
||||
spelled-paste declines in `cell.rs` (aggregate, coordinate column, inside a
|
||||
drill).
|
||||
- **improvise-cwfj:** the three `cmd/panel.rs` declines, and `dispatch`'s
|
||||
arg-error arm.
|
||||
|
||||
The sweeps that pin refusals revert `refuse(` calls, so they **cannot see a
|
||||
refusal written as `set_status`**. Finding one takes a read of every
|
||||
`set_status` on a decline path, and that is how both batches were found.
|
||||
|
||||
**A bound key whose command will not build is a refusal** (the orchestrator's
|
||||
ruling, improvise-cwfj). The key asked for something that did not happen,
|
||||
which is this section's definition. So `dispatch` answers a constructor's
|
||||
`Err` with `refuse(e)` rather than `set_status(e)`. That arm is where every
|
||||
key-path decline lands, including `p`, `x` and `i` onto an aggregate, a fold,
|
||||
a coordinate column or inside a drill.
|
||||
|
||||
Before the conversion, the key path was checked for any reader of the flag:
|
||||
- `last_batch_refused` is read in production only by `Host::mutating_frame`,
|
||||
which handles frames and not keys, and by `main.rs`'s headless loop, which
|
||||
handles `:` lines and not keys.
|
||||
- `Refuse` is `ViewOnly`, like `SetStatus`, so the `Dirty` a key returns and
|
||||
the projections built from it are unchanged.
|
||||
- The browser lockstep suite compares `ViewState`, including `status_msg`,
|
||||
never the flag, and `Refuse` writes the same message.
|
||||
|
||||
The whole workspace stayed green with the change. `q` on a dirty document is deliberately left a
|
||||
report: headless, its text is false — the CLI saves anyway (improvise-ep5o).
|
||||
The two rivals udt2 named both stay green: a successful `:w` and a successful
|
||||
`fit-measure` report through `set_status` and exit 0.
|
||||
|
||||
**`remove-formula` of a formula that is not there now exits 1** (improvise-tej1,
|
||||
improvise-9trl). It used to exit 0: the effect was keyed by names and removing
|
||||
nothing was a silent no-op. Since tej1, `RemoveFormula` names its formula by
|
||||
`FormulaId`, and `remove-formula <category> <target>` resolves the two names
|
||||
to an id in `execute`. When nothing matches it `refuse`s with `No formula for
|
||||
<target> in <category>`, so a script that removes a formula it never added now
|
||||
fails where it used to pass. That is the §10 rule applied: nothing changed, so
|
||||
the request was not met. `remove_formula_of_a_missing_formula_is_a_refusal`
|
||||
pins it, with a present formula as the control.
|
||||
|
||||
## 11 — a headless import refuses colliding records unless told otherwise (improvise-cgqt)
|
||||
|
||||
`import --no-wizard` takes `--on-duplicate refuse|sum|separate`, and the
|
||||
default is `refuse`. The argument is in `context/code/formula-and-io.md`,
|
||||
"Colliding records are a question, not a default". In short, a default of sum
|
||||
is the silent outcome the user called critical, and a default of separate
|
||||
reshapes a scripted document. The refusal prints every colliding tuple through
|
||||
`WizardCollision::describe`, the wizard's own wording, then the flag. It exits
|
||||
non-zero and writes nothing. It is checked before the build rather than read
|
||||
off the build's error, so the listing is structured data and not a parsed
|
||||
sentence. `separate` uses `ImportPipeline::suggested_duplicate_category`, and
|
||||
there is deliberately no flag to name the category yet.
|
||||
|
||||
## 12 — `import` takes folders, and the CLI only says which arguments are folders (improvise-lpzu)
|
||||
|
||||
`improvise import` takes any mix of files and folders. A folder stands for its
|
||||
`*.csv` files, in name order. A folder always merges, even when it holds a
|
||||
single CSV, so that file keeps its `File` item and view. A single file named
|
||||
directly still imports alone (the user's ruling on the PR). The rules
|
||||
and their reasons are in `context/code/formula-and-io.md`, "A folder of CSVs is
|
||||
one sparse model, with a view per file". The CLI's part is the one question
|
||||
`Storage` cannot answer: whether a path is a folder. `get_import_data` asks
|
||||
`Path::is_dir` for each argument, hands the resulting `ImportPath`s to
|
||||
`expand_import_paths` over `PhysicalFs`, and from there takes the path a list
|
||||
of files always took. A folder with no CSV in it is refused before anything is
|
||||
read. The message names the folder, and the run exits non-zero and writes
|
||||
nothing.
|
||||
|
||||
A merge now reaches the build as an `ImportPipeline` carrying the merged files'
|
||||
`sources`, rather than as a bare JSON value. Both `--no-wizard` and the TUI
|
||||
wizard (`run_tui` → `App::start_import`) therefore build the per-file views.
|
||||
The headless path matters most, because the code-maat pipeline
|
||||
(improvise-1xfb.5.1) runs there. `a_headless_import_of_a_folder_merges_its_csv_files_and_repeats_byte_for_byte`
|
||||
pins the views, the sparse Data lines and the byte-identical second run, with
|
||||
the first run's output sitting in the folder it reads.
|
||||
|
||||
## 13 — a headless import refuses rather than saving less than was asked (improvise-8jk6.4, 8jk6.5, 8jk6.6)
|
||||
|
||||
The user treats the headless command and model layer as a product of its own,
|
||||
and the scripted `import --no-wizard` is where it meets a pipeline. The
|
||||
repo-stats pipeline (improvise-1xfb.5.1) found three ways that import saved
|
||||
something other than what the flags asked for, and still exited 0. The
|
||||
reasoning for each is in `context/code/formula-and-io.md`, "A CSV field is
|
||||
text, and the field's role reads it" and "An import refuses what it cannot
|
||||
keep". The CLI's contract:
|
||||
|
||||
- **`--category` keeps the CSV text.** A forced category's items are the
|
||||
file's text, verbatim: `00565796` stays `00565796`, and `220944e2` stays
|
||||
`220944e2`. Measures still read as numbers. A JSON number has no source
|
||||
text, so it is named as JSON writes it (`1e3` is `1000.0`).
|
||||
- **An unparseable `--formula` is a refusal.** It exits non-zero and writes
|
||||
nothing. Every bad formula is named on stderr, each with its parse error.
|
||||
This is §10's rule: the request was not met, so the exit code says so. The
|
||||
wizard refuses the same text at the formula step, with the same wording.
|
||||
- **An import that kept no record is a refusal.** It exits non-zero, writes
|
||||
nothing, and says why: every record was skipped as not a JSON object, or
|
||||
the source had none.
|
||||
- **An import that skipped *some* records exits 0 by default.** It writes the
|
||||
model and warns with a count. With `--strict` (the user's ruling on PR #321)
|
||||
any skipped record is a refusal instead: exit 1, the same count named, and
|
||||
nothing written. The argument is in `formula-and-io.md`.
|
||||
|
||||
Two refusals bail in `run_headless_import` before the build: the `--strict`
|
||||
skip check (`src/main.rs:305`) and the `--on-duplicate refuse` collision
|
||||
check. The rest come from `build_model_with_warnings` as an `Err`. All of
|
||||
them reach `main` as exit 1 before `persistence::save`. So "writes nothing"
|
||||
holds by statement order, not by where the refusal is raised. The headless
|
||||
tests check it by asserting that the output path does not exist.
|
||||
|
||||
## 14 — the command language reads `|pipe quoting|`, and a surplus argument is refused (improvise-31oe.3)
|
||||
|
||||
`create-view |By area|` used to make a view named `|By` and drop `area|` without
|
||||
a word. Two separate defects, fixed separately.
|
||||
|
||||
**The quoting.** `parse.rs` now has one lexer, `lex`, and everything else is
|
||||
built on it: `parse_line_with`, `segments` and `parse_segment_interactive`. The
|
||||
rules are:
|
||||
|
||||
- A `|…|` piece is read by `improvise_formula::quote::unescape_pipe_body`, the
|
||||
one home of pipe quoting (gotcha 34). So `\|`, `\\` and `\n` mean exactly what
|
||||
they mean in `.improv` and in formulas. The lexer only finds the closing pipe:
|
||||
a backslash takes the next character with it, so `\|` does not close.
|
||||
- A piece may sit **anywhere in a word**, and the pieces and bare text are
|
||||
joined. So a coordinate quotes each half, `|Sales Region|/|East Coast|`, the
|
||||
way a `.improv` data line quotes each name. The alternative, a pipe only at
|
||||
the start of a word, would have left no way to spell a coordinate with a
|
||||
space in it.
|
||||
- `"…"` is **kept**, verbatim, with no escapes, and a `|` inside it stays a
|
||||
`|`. It is not a workaround to retire. Formula text carries its own pipes:
|
||||
`add-formula "Net = |Gross Rev| - Cost"` must reach `parse_formula` with the
|
||||
pipes on. If the line had pipes only, every pipe in a formula would need
|
||||
escaping. Its old leniency is kept too: a `"` that is never closed runs to
|
||||
the end of the line. A `|` that is never closed is refused, naming the text
|
||||
from the open pipe on, because a name is never meant to run to the end.
|
||||
- The separator is a **bare** `.` word. A quoted `|.|` or `"."` is an argument,
|
||||
and a `.` inside `|a . b|` does not split the line. The old `split_on_dot`
|
||||
was a second scanner that knew double quotes and not pipes. With one lexer
|
||||
there is no second scanner to disagree.
|
||||
- A comment line (`#`, `//`) is not lexed, so an unmatched pipe in a comment is
|
||||
not an error.
|
||||
|
||||
`segments` now returns `Result`, because an unterminated pipe is a fact about
|
||||
the whole line. `main.rs` refuses the line before running any of it, as §7 does
|
||||
for a parse error, and `ExecuteCommand` refuses it on the status line.
|
||||
|
||||
What is still not spellable: an item whose name contains `/`. The coordinate is
|
||||
split on the first `/` **after** unquoting, so `|a/b|/x` is still read as
|
||||
category `a` and item `b/x`. Closing that needs the lexer to hand structured
|
||||
coordinates to the commands, which is a larger change than this ticket.
|
||||
|
||||
**The arity.** Every registration now declares `MaxArgs::AtMost(n)` or
|
||||
`MaxArgs::Any` beside its prototype. `CmdRegistry::parse` and
|
||||
`CmdRegistry::interactive` apply it after construction succeeds. So the rule is
|
||||
written once and every command gets it, and a new registration cannot compile
|
||||
without a declaration. The refusal names the command, the limit and every
|
||||
surplus argument (`create-view: takes at most 1 argument(s), got 2; unexpected
|
||||
"area"`). A headless run exits non-zero through the §7 parse gate, and the `:`
|
||||
line refuses on the status line.
|
||||
|
||||
It runs **after** the command's own parse, deliberately. A command whose own
|
||||
refusal states its shape keeps it: `add-formula`'s "quote the formula"
|
||||
(improvise-z05h) is more use than an arity, and it is what `formula Product
|
||||
Total = Widget + Gadget` still says.
|
||||
|
||||
`MaxArgs::Any` is a claim, and the list of commands making it is short:
|
||||
`add-items`, `set-cell`, `set-format` (it joins its words), `clear-cell`,
|
||||
`yank`, `paste`, `drill-into-cell` and `commit-cell-edit`. Each takes a list of
|
||||
items or coordinates. `no_command_silently_drops_an_argument_it_was_given` names
|
||||
that list, so widening another command to `Any` has to add it there.
|
||||
|
||||
## 15 — a command that names something the document lacks is refused (improvise-31oe.2)
|
||||
|
||||
`set-axis Regoin row`, `set-page Region Nowhere` and `set-axis Region sideways`
|
||||
all exited 0 and said nothing. §10's rule applies: nothing changed, so the
|
||||
request was not met.
|
||||
|
||||
- **An unreadable axis is an argument error.** `set-axis` now checks its axis
|
||||
with `parse_axis` when it is parsed, so a script is refused before any of it
|
||||
runs. `execute` still refuses an axis it cannot read rather than returning
|
||||
nothing. That arm cannot be reached through the registry.
|
||||
- **An unknown category, item, group or view is a refusal**, from `RequireKnown`
|
||||
(below), because only the document knows what it holds. `effect::unknown_name`
|
||||
asks the active view (`try_axis_of`) for a category. That is exactly the
|
||||
condition under which `View::set_axis` was a silent no-op. It asks
|
||||
`effective_item_names` for an item, so a formula target is a known `_Measure`
|
||||
item. It asks both an item's own group and the category's `groups` for a
|
||||
group. `add_item_in_group` registers no `Group`, and a parent group has no
|
||||
items, so either one alone misses a real group. A view is asked of
|
||||
`workbook.views`.
|
||||
|
||||
The siblings fixed with it: `hide-item` and `show-item` (hiding an unknown item
|
||||
wrote it into `hidden_items`), `toggle-group`, and `switch-view` (the effect
|
||||
ignores a view it lacks, and says nothing). `set-page` with an unknown item was
|
||||
worse than a no-op: it wrote the name into `page_selections`.
|
||||
|
||||
The check runs at **apply** time, as an effect of its own. `execute` runs
|
||||
against one context built before the line ran, and a `:` line builds every
|
||||
segment up front, so a check in `execute` refused
|
||||
`create-view Second . switch-view Second` and
|
||||
`add-item Type Tools . set-page Type Tools` after their first halves had landed,
|
||||
while `improvise cmd` (which rebuilds the context per segment) accepted them.
|
||||
Each of these commands therefore returns `RequireKnown(needs)` followed by the
|
||||
real effect. `RequireKnown` refuses and aborts the batch (gotcha 21), so the
|
||||
real effect never applies. It is a `ViewOnly` effect that writes nothing and is
|
||||
off the wire on purpose: the wire's own effects (`model_effects`, gotcha 38)
|
||||
stay non-refusing, since a refusing effect there becomes a frame rejection
|
||||
(`MutationRefused`). That is arguably right, but it changes the protocol, and
|
||||
a browser composes these only from names it was sent. It is left for a ticket
|
||||
of its own.
|
||||
|
||||
A refused name stops the rest of a `:` line, because the abort ends the one batch
|
||||
the whole line runs as. Headless runs each segment as its own batch, so an abort
|
||||
never crosses a `.` there: `:set-page Type Nowhere . add-item Type X` adds
|
||||
nothing, while `improvise cmd` on the same line exits 1 and still adds `X`.
|
||||
`load`/`Save` already behaved this way. The two front ends' chain semantics are
|
||||
left for a ticket of their own.
|
||||
|
||||
The interactive cursor fallbacks of `clear-cell`, `yank`, `paste` and
|
||||
`drill-into-cell` answer only for coordinates that were OMITTED
|
||||
(`only_when_omitted`). Coordinates that were given and are malformed are
|
||||
refused, never replaced by the cursor's cell.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,263 +0,0 @@
|
||||
# Delivery: the reasoning
|
||||
|
||||
ADR 009 step 8 (improvise-z2vr): notices per committed command, watermarks, corrections, the shared
|
||||
bounded log, the lossless wire and the edge adapters. The decisions are the ADR's clause 9
|
||||
(`context/adr/009-incremental-evaluation.md`). This file records how the code reads them, the places
|
||||
the ADR left a choice open, and the places it could not be satisfied yet. The evidence for each test is
|
||||
in `context/test-evidence/delivery.md`.
|
||||
|
||||
**What is built, and against what.** Neither the incremental backend nor ADR 008's render model
|
||||
exists. So the source of every target is the existing Eager path: the cells a `GridLayout` draws for a
|
||||
view. Everything below is correct against that source and does not depend on what replaces it. The
|
||||
circuit will supply "moved output keys" where `Delivery::commit` diffs two `Output`s today, and
|
||||
nothing else changes.
|
||||
|
||||
| piece | where |
|
||||
|---|---|
|
||||
| wire values, keys, Z-set elements | `improvise-protocol/src/delivery/wire.rs` |
|
||||
| stamp, range, notice, merge | `.../notice.rs` |
|
||||
| request, correction, `Edge` | `.../exchange.rs` |
|
||||
| the client end: watermark, held cells, when to ask | `.../client.rs` |
|
||||
| the publisher, the log, live shapes, answering | `improvise-command/src/ui/delivery.rs` |
|
||||
| `Host` integration, in-process and JSON edges | `ui/host.rs`, `ui/delivery.rs` |
|
||||
|
||||
## 1. The wire form
|
||||
|
||||
A number crosses as ADR 005's canonical text through `numtower::Num`: `#i0.1`, `#i-0.0`, `inf`,
|
||||
`-inf`, `nan`. `WireNum` accepts only text the tower itself writes (`parse` re-prints and compares),
|
||||
because a second spelling of one value would make textual equality wrong. Equality of wire values is
|
||||
textual, so `-0.0` and `+0.0` are different wire values. That is the opposite of the cell delta in
|
||||
clause 3, which makes them equal; the two do not meet, because the delta is a store-side signal and a
|
||||
correction is a value the client displays.
|
||||
|
||||
An integer is a JSON number below 2^53 and a decimal string at or above it (`wide`, `wide_weight`).
|
||||
That covers stamps, positions, window sizes and weights. Text is a JSON string, byte-exact.
|
||||
|
||||
`CellValue` is still an `f64`, so `WireValue::of` writes it through `Num::inexact` and
|
||||
`to_cell_value` narrows an exact number to `f64`. The round-trip properties therefore run over
|
||||
28-digit decimals and `i128`-wide fractions at the `WireNum` level, where they are lossless, and over
|
||||
`f64` at the `CellValue` level, where only the NaN payload is dropped.
|
||||
|
||||
`CellKey` is not `Serialize`, so a cell is named by `WireKey`, its `(category, item)` tuple. Under
|
||||
ADR 007 those become ids; nothing here reads the strings.
|
||||
|
||||
## 2. Stamps and notices
|
||||
|
||||
`Stamp` is `(host instance, document_generation, publication)`. A stamp from another host or
|
||||
generation is behind every current one (`Stamp::is_behind`). The publication number increments once
|
||||
per committed command, and it keeps counting across a document swap rather than restarting, so a
|
||||
number is never reused within one host instance.
|
||||
|
||||
**What "a committed command" is.** The host publishes one notice for each `Command::Model` that
|
||||
lands. A view command (cursor, mode, a view switch) publishes nothing: it is view-store state and
|
||||
the model store has no business announcing it. A write that changes nothing still publishes, with
|
||||
empty `moved`, because the contract is one notice per commit and a client counts them.
|
||||
|
||||
**A notice names moved keys per live shape**, plus axis changes, relabels, flags, evicted shapes. Axis
|
||||
changes are computed from the categories' item lists before and after (`axis_changes`): insert and
|
||||
remove by item, and a reorder when the surviving items changed order. `Relabel` exists and is never
|
||||
emitted: with names as identity a rename is a remove and an insert, and display writes arrive with
|
||||
ADR 007. Flags come from comparing the formula texts, `measure_agg`, the views and the default view.
|
||||
|
||||
`Notice::merge` joins two notices only when the second starts where the first ends (same epoch).
|
||||
Anything else returns both unchanged, so a merge cannot hide a loss. `Moved::Correct` absorbs key
|
||||
lists in a merge.
|
||||
|
||||
## 3. Requests and corrections
|
||||
|
||||
A request is a shape key, a window and one of three asks. `Ask::Correction` is "bring me up from my
|
||||
watermark"; `Ask::Fetch` is "the current values of these keys, covering the publications up to
|
||||
`through`"; `Ask::Notices` is "the notices after this stamp".
|
||||
|
||||
**A correction's elements are `target`, not `target - held`.** The host does not know what the client
|
||||
holds, so it sends the target restricted as the body says, and the client computes `target - held`
|
||||
with ordinary Z-set subtraction. A `Moved` body restricts both sides to the named keys, so a key the
|
||||
target no longer holds is removed. A `Window` body replaces everything held.
|
||||
|
||||
**`through` is not `at`.** `at` is the state the values were read at, always the latest committed one.
|
||||
`through` is how far the correction advances the client's watermark. They differ for a `Fetch`:
|
||||
the client named the keys of the notices it has seen, and a later commit may have moved other keys
|
||||
whose notice it has not seen yet. Advancing to `at` would make that notice look already applied, so
|
||||
the watermark advances only to `through` and the later notice refetches its keys. The values are
|
||||
at least as new as the watermark claims, never older.
|
||||
|
||||
**An axis change makes the correction a window refetch.** A moved key list says which values changed,
|
||||
not which keys sit in the client's window, and an insert or remove shifts cells into and out of it
|
||||
without moving them. The ADR has the client re-derive positions from the axis changes and fetch what
|
||||
it lacks. That needs ADR 008's render model, which this layer does not have. So the client records a
|
||||
notice that moved an axis or set `flags.views` (`restructured`), asks `Fetch { window: true }`, and the host answers with a
|
||||
`Window` body whose `carried` notice holds the axis changes, in order. This is a fetch, not a
|
||||
correction, so the required test "a frame `[AddItem, SetCell]` ... and no client asks for a correction"
|
||||
holds: it asserts no `Ask::Correction` was sent. When the render model exists the client can narrow
|
||||
the fetch to the cells that entered its window (improvise-bw4p.9).
|
||||
|
||||
A view command (`SetAxis`, `TransposeAxes`, a page selection, a hide) moves positions and no key, and
|
||||
reaches the client only as `flags.views`. So that flag counts as an axis change on both sides: the client
|
||||
fetches its window, and the host answers a correction over a range holding the flag with a window body.
|
||||
|
||||
## 4. The client
|
||||
|
||||
`EdgeClient` holds, per shape: the window, the held cells (a `ZSet`), the **watermark** (applied
|
||||
through), `seen` (the last notice received), the notices received and not yet fetched, `restructured`
|
||||
and `stale`.
|
||||
|
||||
`on_notice` ignores a duplicate (`to <= seen`), treats a gap, another host or generation, a shape in the
|
||||
evicted list, or `Moved::Correct` as "ask for a correction", and otherwise queues the moved keys. The
|
||||
watermark advances over a notice only when nothing is owed, so it never claims a key it has not fetched.
|
||||
|
||||
`apply` takes a correction. It is a duplicate if the watermark already covers `through`; a `Moved`
|
||||
body is a gap unless it starts at or below the watermark. Otherwise `held += target - held`, the
|
||||
watermark becomes `through`, and the queued notices it covers are dropped. A late reply older than
|
||||
the watermark therefore cannot regress the state, which is what makes a retry safe.
|
||||
|
||||
**A correction from a generation or host the client has left is a duplicate, whatever the
|
||||
watermark.** A shape records the epochs its watermark has moved out of (`left`, the newest 64), and a
|
||||
correction from one of them, or from an older generation of the host it is on, changes nothing. Epochs of
|
||||
different hosts are not ordered, so a client that has never applied anything from the newer host cannot
|
||||
tell a late reply from a dead one from a legitimate first answer, and applies it; once it has moved on, the
|
||||
dead host's replies are refused. A duplicate answer also prunes every queued entry from another epoch
|
||||
and moves `seen` onto the live epoch, so a late notice from a dead host costs one extra round and cannot
|
||||
livelock `sync`.
|
||||
|
||||
**The host that answers a request is live, whatever the client believed.** `left` is a guess, because
|
||||
epochs of different hosts are unordered: a late reply from a host the client never joined moves it onto
|
||||
that host, and writes off the live one. Refusing the live host's answers as "left" then strands the client
|
||||
(`sync` returned `Exhausted`, found by the property test). So `apply_answer`, which `sync` and `poll` use
|
||||
for the reply to a request they just sent, takes an epoch off `left` before it applies. `apply` is for a
|
||||
reply that arrives on its own, and keeps the guard. A stale reply can still move the client for one round;
|
||||
the next request it makes is answered by the live host and brings it back.
|
||||
|
||||
**A notice from another epoch only marks the shape stale; `seen` moves only through an answer.** The
|
||||
first two livelocks (improvise-z2vr) were the same defect reached by different routes, which is why
|
||||
ignoring one kind of foreign notice after another was not enough. Epochs of different hosts are
|
||||
unordered, so a notice from an epoch other than `seen`'s cannot be placed. Reading it used to rewind
|
||||
`seen` into the foreign epoch, and then the live host's own old notices looked new, queued work and set
|
||||
`restructured`; the answers that followed were covered by the watermark, were duplicates, and never
|
||||
cleared it. Now such a notice sets `stale` and nothing else, the next request is answered by the live
|
||||
host, and that answer (`apply_answer`) is what moves `seen`. Two cheaper guards stay: a notice from an
|
||||
older generation of the host the shape is on, or from an epoch it has left, is ignored without the extra
|
||||
round; and a duplicate answer clears a `restructured` mark it covers. The last is defensive: no test
|
||||
reaches it now that `seen` cannot be rewound, and it is kept because a covered answer that cannot clear a mark
|
||||
it covers is the shape of both livelocks. Merging notices cut for different shapes, or a cut one with an
|
||||
uncut one, is refused. A notice the client asked for
|
||||
(`poll`) skips the left-epoch test, since the host that answers is live.
|
||||
|
||||
`poll` hands a pulled notice only to the shape it asked about, and the notice says so: `for_shape` sets
|
||||
`Notice::cut_for`, and `accept` limits any notice that carries it to that shape, however it reaches the
|
||||
client (`on_notice` included). Without the mark, a pulled notice delivered as if pushed made the other
|
||||
shape skip keys it never saw.
|
||||
|
||||
`connection_lost`, `request_failed` and `unreadable` set `stale`, which `wants` turns into
|
||||
`Ask::Correction` from the watermark the client held. A duplicate whose `through` equals the watermark
|
||||
also clears `stale`: the host has said there is nothing newer.
|
||||
|
||||
**A lost final notice is invisible to a listener.** The next notice shows the gap, but if none comes the
|
||||
client never learns. `poll` is the answer: it asks `Ask::Notices { after: seen }`, which the host
|
||||
answers from the log. The property test ends every history with a `poll` for this reason.
|
||||
|
||||
`Event`s (`Axis`, `Relabel`, `Flags`, `Resync`) are what the shape's consumer applies to its
|
||||
structure. They are deduplicated by `events_through`, because an axis change is not idempotent.
|
||||
`Resync` means positions are unknown: re-derive them from what is held.
|
||||
|
||||
## 5. The Eager stand-in for a shape key
|
||||
|
||||
The real key is the circuit's: axes, None categories, filters, pages, `measure_agg`, in canonical id
|
||||
form. Here a shape key is `eager:view/<name>` (`shape_of_view`) and the output is that view's layout.
|
||||
That is finer than the real key, which is always safe: two views that would share state do not, and
|
||||
nothing else is lost. A view that no longer exists is announced as evicted.
|
||||
|
||||
`Output` is the shape's whole output, keyed by `WireKey` with the position attached. `target` restricts
|
||||
it to a window. `moved_since` compares two outputs by value and presence; a changed position is an axis
|
||||
change, not a key.
|
||||
|
||||
**This costs O(grid) per live shape per commit.** It stands in for the circuit's delta, and it is the
|
||||
reason the module exists beside, not instead of, the incremental work: nothing else in the host reads
|
||||
the whole grid per command.
|
||||
|
||||
## 6. The host
|
||||
|
||||
`Delivery` is one value per document, held by `Host`. `Host::handle` calls `Delivery::commit` after
|
||||
each `Command::Model` that lands, and `Host::deliver` answers a `Request`.
|
||||
|
||||
**Reads see only the last committed state.** `answer` reads `Live::output`, which `commit` refreshed,
|
||||
and never the workbook. A value a failed command wrote is not in it.
|
||||
|
||||
**Rollback.** `Host::mutating_frame` clones the `Delivery` beside the app snapshot and the session
|
||||
mirrors, and `undo_frame` puts it back. The clone is cheap: `Arc` for outputs, notices and structure,
|
||||
and both the log and the outbox hold at most `capacity` notices. So a refused frame consumes no publication number and publishes nothing.
|
||||
|
||||
**The ADR and ADR 001 disagree here, and ADR 001 wins until ADR 006 exists.** Clause 9 says a frame
|
||||
whose later command fails keeps its earlier commands and their notices, "each command commits on its own
|
||||
(ADR 006)". Today's host undoes the whole frame (ADR 001 clause 4, `Host::mutating_frame`), and ADR 006 is
|
||||
not in the tree. The required test "a refused command consumes no publication number, and the commands
|
||||
before it in its frame stay published" is split in two. The first half is tested. The second is
|
||||
tested as its opposite, `commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001`,
|
||||
which is the test to flip when per-command commit lands (improvise-bw4p.8): the publisher already supports it, since it
|
||||
publishes whatever the host tells it committed.
|
||||
|
||||
**Nothing is published while nobody can hear it.** With no live shape and no attached transport
|
||||
(`Host::attach_notices`), a commit only advances the publication number: no structure is read, nothing is
|
||||
diffed, the log is cleared and the queued structure is forgotten. A shape that goes live, or a transport
|
||||
that attaches, re-reads the structure, so its first diff is against a known state. Without this every
|
||||
host pays for delivery on every command while nothing consumes it. The outbox exists only while a
|
||||
transport is attached, and keeps the newest `capacity` notices; a transport that falls behind sees a gap
|
||||
and asks for a correction. The axis diff is linear in the items (`diff_items`, set lookups), and its test
|
||||
counts comparisons on a counting item type rather than reading a clock.
|
||||
|
||||
**The host instance** is a random id, never a time (`Host::new` takes one from `RandomState`).
|
||||
`RandomState` has no entropy on `wasm32-unknown-unknown`, so a worker must pass a real one through
|
||||
`Host::with_instance`. A restarted host with the same id would accept a stale watermark as current.
|
||||
`Worker` still calls `Host::new` (improvise-bw4p.7). Such a restart, with the same instance id and
|
||||
generation 0, is not recognised as a restart: the client's `older` guard refuses its replies as coming from
|
||||
an older generation of the host it is on (improvise-bw4p.7).
|
||||
|
||||
**Eviction** (`evict_idle`) takes a local count and an idle length in the same unit, never a wall clock
|
||||
compared across machines. An evicted key is announced in the next notice and forgotten; naming it again
|
||||
revives it with `since` set to the current publication, so a watermark older than that is answered with
|
||||
a window.
|
||||
|
||||
## 7. Answering
|
||||
|
||||
`covered(shape, watermark)` is true when the watermark is in the current epoch, is not ahead of the
|
||||
host, and either equals the current publication or is no older than both the log's start and the time
|
||||
the shape went live. Only then are the moved keys of `(watermark, now]` complete.
|
||||
|
||||
- covered, only moved keys: a `Moved` body with those keys' current values in the window;
|
||||
- covered, with axis changes or relabels in range: a `Window` body with `carried`;
|
||||
- a `Moved::Correct` in range, or not covered: a `Window` body from `None`, which is `target - empty`.
|
||||
|
||||
A notice lists no keys for a shape whose list passed `MAX_KEYS`; it says `Moved::Correct` instead, and
|
||||
costs every reader a correction. `LOG_CAPACITY` bounds the single shared log. Both are set per host by
|
||||
`limit_delivery` so a test can reach them.
|
||||
|
||||
The token in a request is carried and echoed by the client and never read by the host.
|
||||
|
||||
## 8. Edge adapters
|
||||
|
||||
**Transport contract.** A reply must reach the call that made the request. `Request` and `Correction`
|
||||
carry no id, so the client cannot tell a late reply from an answer on its own; `sync` and `poll` treat
|
||||
whatever `Edge::call` returns as the answer (`apply_answer`), and only a reply handed to `apply` is
|
||||
treated as late. A transport that reorders or duplicates replies must therefore correlate them itself
|
||||
or drop the strays (improvise-bw4p.10 adds the id to the frames).
|
||||
|
||||
|
||||
`Edge` is one end of a stream: `call(Request) -> Result<Reply, EdgeError>`. `InProcess` is the
|
||||
function call; `WireEdge` passes request and reply through JSON text, the adapter at a wire boundary,
|
||||
so the lossless properties are exercised by every test that uses it. Neither adds a `ClientFrame` or
|
||||
`ServerFrame` variant: the page does not read notices yet, and adding the frames means changing the
|
||||
browser and the worker together (ADR 008 replaces `Projection`; improvise-bw4p.10). A transport
|
||||
pushes `Host::take_notices()`.
|
||||
|
||||
## Not done, and why
|
||||
|
||||
- **The notices are not sent by the socket or worker hosts.** They need frame variants and the page
|
||||
(above).
|
||||
- **Positions and column totals are not on the wire.** A correction carries `(key, value)` and no row or
|
||||
column, and `Output` holds body cells only. The ADR attaches the position when output is sent and lists
|
||||
totals among the moved keys. Both need the render model and the totals' own keys (ADR 008, clause 10).
|
||||
- **A twin at the serving end** (a server-side watermark or filter) is not built; the ADR allows it and
|
||||
does not require it.
|
||||
- **A pushed filter** is the client's `window`, applied on every request.
|
||||
- **Hidden-item overlays, pruning queries and column widths** are ADR 008 and clause 10.
|
||||
|
||||
**The same edge, generalised in `zset`.** The `zset` crate's edge module, `crates/zset/src/edge.rs` (`context/code/zset.md`, "Edges") carries Z-set deltas between two circuits with the rules of this file (stamps, coalescing by summing, `target - held`, token-matched answers) and no log; this layer is unchanged and does not use it.
|
||||
@@ -1,342 +0,0 @@
|
||||
# The documentation site — the arguments
|
||||
|
||||
`scripts/build-doc-site.mjs` turns the prose in `context/` into the static site
|
||||
served at `/docs/` on GitHub Pages. This is the reasoning behind what it
|
||||
publishes, what it deliberately does not, and why it is a hundred-odd lines of
|
||||
node rather than a static site generator (improvise-djoo).
|
||||
|
||||
---
|
||||
|
||||
## 1. What is published, and what is not
|
||||
|
||||
The manifest is the `SECTIONS` array at the top of the script, and **a document
|
||||
is published only by being named there**. That is the point: a glob over
|
||||
`context/**` would publish the next working file somebody drops in, and the
|
||||
decision about what a stranger should read is not one to make by accident.
|
||||
|
||||
Published, in four sections:
|
||||
|
||||
| Section | Contents | Why |
|
||||
|---|---|---|
|
||||
| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md`, `context/gotchas.md`, `context/file-inventory.md` | What the project is for, the doctrine, and the single best explanation of how it is built. The map is a digest; the last two are its long forms, published so a `gotcha N` citation and the per-file listing have somewhere to land. |
|
||||
| Decision records | `context/adr/*.md` | An ADR is written for an outside reader by construction — it states the options that were rejected, which is the part that is worth reading and the part that is nowhere else. |
|
||||
| The arguments, by area | `context/code/*.md` | This is the reasoning that used to live in code comments and was moved out under the two-line cap (design-principles §6). It is the most valuable prose in the repository and it is addressed to exactly the reader a documentation site has. |
|
||||
| Test evidence | `context/test-evidence/*.md` | One entry per test whose narrative is longer than its code. |
|
||||
|
||||
Not published:
|
||||
|
||||
- **`context/plan.md`** is a *prompt*: it opens "You are working in the
|
||||
`improvise` repository… Your job is to prepare this repository for a public
|
||||
Show HN launch", and it instructs a reader to audit a `context/SPEC.md` that
|
||||
no longer exists. It is an instruction to an agent about a launch, not a
|
||||
description of the program, and publishing a launch plan on the site the
|
||||
launch points at inverts the two.
|
||||
- **`context/cloud-agents.md`** is operational notes for agents running in a
|
||||
cloud session — tracker hydration, `bd` pitfalls, what the sandbox blocks.
|
||||
It is true and it is useful and it is addressed to nobody who visits a
|
||||
website.
|
||||
- **`context/code/pr-review.md`** is how an agent reads a PR's reviews — where
|
||||
a ChatGPT Codex review appears, how to tell it from this repo's own agents'
|
||||
comments, and why absence is not approval. It is about *this repository's
|
||||
process*, not about the program, which is the `cloud-agents.md` line and not
|
||||
the `tooling.md` one: `tooling.md` explains checkers a reader of the code
|
||||
will meet, and nobody visiting a website is about to read a PR thread.
|
||||
|
||||
None of the three is referenced by any published document except as a row in
|
||||
the file inventory (`file-inventory.md`), which is inside a code fence, so leaving them
|
||||
out creates no dangling link. That was checked rather than assumed.
|
||||
|
||||
**And "not published" is now derived rather than assumed** (improvise-res9).
|
||||
The generator walks `context/**/*.md` plus `docs/design-notes.md`, subtracts
|
||||
the manifest, and fails unless the remainder is exactly the `OMITTED` map in
|
||||
the script — which carries the reason beside the name, so this section and the
|
||||
code cannot drift into disagreeing. An allowlist and not a count, because an
|
||||
omission is a decision that should have to be argued in code; and the map is
|
||||
checked in *both* directions, because an entry naming a file nobody can find
|
||||
asserts a reason for nothing, which is the failure a dead ticket id is.
|
||||
|
||||
The check was latent when it was filed and **live by the time it was written**:
|
||||
`context/code/kb-tool.md` had been added after this table was drafted and was
|
||||
publishing nothing, with no argument for its absence anywhere. The table above
|
||||
says `context/code/*.md` is published; the manifest disagreed and nothing could
|
||||
see it. That is the whole failure mode in one file.
|
||||
|
||||
**Test evidence is the one judgement call worth stating.** The argument for
|
||||
leaving it out is that hundreds of entries about individual tests are noise
|
||||
to a reader who wanted to know what a pivot table is. The argument for keeping it is
|
||||
stronger: every one of the `context/code/*.md` documents points at its
|
||||
evidence file by name, so omitting it would break the published prose in
|
||||
exactly the way this site is otherwise careful not to; and the evidence file is
|
||||
the most distinctive artefact the repository has — a written record of the
|
||||
injected fault behind each load-bearing test is not a thing most projects can
|
||||
show. It is therefore published as its **own section**, last, described as a
|
||||
contributor reference rather than a tour, so nobody wanders into it expecting
|
||||
a narrative.
|
||||
|
||||
## 2. Ticket ids resolve to a page, never to a URL that does not exist
|
||||
|
||||
The prose cites bead ids roughly 190 times — this paragraph deliberately spells
|
||||
none, because a document that gives one as an example makes itself a source of
|
||||
audit noise, and a placeholder is worse than a real id: it is ID-shaped and
|
||||
resolves to nothing. On a public site those citations are
|
||||
meaningless strings, and the obvious repair — link them to an issue tracker —
|
||||
is unavailable: beads keeps its issues in a Dolt database on a git ref, and
|
||||
there is no public issue URL to point at. Inventing one would produce 190
|
||||
links that 404, which is worse than plain text because a link is a promise.
|
||||
|
||||
So the ids resolve **inward**. `tickets.html` is generated from
|
||||
`.beads/issues.jsonl` — already committed, already public — and carries id,
|
||||
status and title, nothing else; descriptions are internal working notes and are
|
||||
not the site's business. Each citation in the prose becomes a link to that
|
||||
page's anchor, with the title in a `title=` attribute so a hover answers the
|
||||
question without a navigation.
|
||||
|
||||
**`issues.jsonl` is a passive export and lags the database**, which is not a
|
||||
defect to work around but a fact to render honestly: an id the export does not
|
||||
carry is emitted as plain text with a dotted underline and no link, and the
|
||||
ticket page lists it as "not exported". The site therefore never links to an
|
||||
anchor it did not generate, whatever state the export is in. A generator that
|
||||
instead linked every id it saw would produce a page that is right today and
|
||||
broken the first time somebody cites a fresh ticket.
|
||||
|
||||
**The page publishes every cited id, open bugs and their titles included, and
|
||||
that is a decision rather than a default** (user ruling). The data is already
|
||||
public in `.beads/issues.jsonl` in a repository anyone can clone; the page only
|
||||
makes it legible. An honest open-bug list is a normal thing for an open-source
|
||||
project to have, and filtering one while the page's own lede links the raw
|
||||
JSONL would be theatre. Note the second-order reason to keep it unfiltered:
|
||||
a citation whose row was suppressed would be a link into a page that does not
|
||||
answer it, which is the failure this whole section exists to avoid.
|
||||
|
||||
**Titles are tracker data and are escaped as attribute values.** Five shipped
|
||||
bead titles contain a double quote; interpolated into `title="…"` by an `esc`
|
||||
that handled only `&`, `<` and `>`, they broke out of the attribute and the
|
||||
browser parsed the remains as three junk attributes. `esc` now escapes `"` and
|
||||
`'` as well. Today's corpus being friendly is not a reason to interpolate
|
||||
untrusted-shaped text unescaped — the tracker is written by whoever files a
|
||||
bug.
|
||||
|
||||
## 3. Source paths link only where the file exists
|
||||
|
||||
A code span whose entire content is a repo path becomes a link — to the
|
||||
generated page when the path is one of the published documents, to GitHub
|
||||
otherwise. Two constraints make that safe. The whole span must be the path, so
|
||||
a sentence that merely mentions a directory is untouched; and **the path must
|
||||
exist on disk at generation time**, so a citation that has gone stale degrades
|
||||
to plain text rather than to a 404. The GitHub ref is the cache-bust token
|
||||
`web/assemble.sh` already receives, used only when it is a 40-hex sha and
|
||||
falling back to `main` otherwise — a local `dev` build must not emit
|
||||
`blob/dev/…`.
|
||||
|
||||
## 4. The link check is the test, and it is part of the build
|
||||
|
||||
The generator collects every anchor it emits and every internal href it writes,
|
||||
resolves them against each other, and **exits non-zero** if any href names a
|
||||
page it did not generate or a fragment that page does not carry.
|
||||
`web/assemble.sh` runs under `set -eu`, so a broken cross-reference fails the
|
||||
assembly rather than shipping.
|
||||
|
||||
This is not ceremony. It caught two real defects on its first two runs: a
|
||||
fence-close regex that compared a three-character marker against a
|
||||
one-character string, which swallowed every document from its first code block
|
||||
onward; and a list parser that ended the gotcha list at the first blank line,
|
||||
so 47 of the 48 gotcha anchors were never emitted while the "jump to gotcha N"
|
||||
strip linked to all of them. Both were invisible in a rendered page that
|
||||
otherwise looked fine.
|
||||
|
||||
It is proved by injection rather than asserted: renaming the emitted anchor ids
|
||||
makes it report 48 missing anchors, and pointing the ticket links at a page
|
||||
name that is never generated makes it report 763 missing pages — two different
|
||||
diagnoses, because "the page is absent" and "the page is there but the anchor
|
||||
is not" call for different repairs.
|
||||
|
||||
Links that **leave** the `docs/` subtree — `../index.html`, `../app/` — name
|
||||
files this script does not generate, so they are resolved against the
|
||||
filesystem instead. That is the whole reason `assemble.sh` runs the generator
|
||||
**last**: its siblings have to exist before they can be checked. An earlier
|
||||
version skipped those 48 links while still counting them as checked, behind a
|
||||
comment promising they were "checked below" when there was no below.
|
||||
|
||||
That ordering is itself checked now (improvise-res9). The generator reads
|
||||
`web/assemble.sh` and refuses to run unless its own invocation is that script's
|
||||
last command. PR #63 appended a loop and it happened to land above the
|
||||
generator; one landing below would have made the outward-link check pass
|
||||
against a half-built tree, silently and in the one direction this section says
|
||||
must not go unnoticed.
|
||||
|
||||
**And the link check is not sufficient, which is the more important lesson.**
|
||||
It answers one question — does every href resolve — and a defect outside that
|
||||
question is invisible to it however carefully it is run. The worked example is
|
||||
a nested `<a>`: a TOC entry wrapping a heading whose rendered text already
|
||||
contained a link produced `<a href="#anchor"><a href="github…">…</a></a>`, and
|
||||
the HTML parser closes the outer anchor early, so the entry became an empty
|
||||
unclickable stub and clicking it navigated to GitHub. Both hrefs resolved
|
||||
perfectly. It shipped past the link check, past the whole-site link check, and
|
||||
past a screenshot pass, and was caught by a reviewer *clicking a link*. The
|
||||
repair is therefore two things, not one: the TOC keeps a tag-free variant of
|
||||
each heading, **and** a separate check asserts no `<a>` appears inside another
|
||||
`<a>` — proved by reverting the first half and watching the second report 66.
|
||||
|
||||
The general form: when a check misses a defect, add the check that could have
|
||||
seen it rather than only fixing the defect, and ask what *question* the
|
||||
existing check was answering.
|
||||
|
||||
**`site/` is build output and is not tracked** (improvise-1xfb.4). It is
|
||||
`web/assemble.sh`'s default `OUT_DIR`, and `site/docs` is this script's.
|
||||
Every reader builds its own copy from source: CI's `build` job runs
|
||||
`nix build .#site`, which assembles into the store; the harness assembles
|
||||
into `harness/serve/`; `scripts/build-web-release.sh` assembles into a
|
||||
temporary stage. For a while 27 generated pages under `site/docs` were
|
||||
committed by accident. They went stale, still saying 12 categories after the
|
||||
cap was raised, and nothing read them.
|
||||
|
||||
## 5. Why a node script and not a generator
|
||||
|
||||
The landing page is hand-written HTML with no build step and one pinned CDN
|
||||
script; the same posture applies here. A static site generator would add a
|
||||
dependency tree, a lockfile and a toolchain to a repository whose site is
|
||||
currently four files, in exchange for features this corpus does not use — the
|
||||
docs contain no front matter, no shortcodes and, as it happens, **not a single
|
||||
markdown link**. Every cross-reference in them is prose naming a path or an
|
||||
id, which is precisely why §2 and §3 are the interesting parts and Markdown
|
||||
rendering is the boring part.
|
||||
|
||||
So the renderer is a deliberately small subset covering exactly what the corpus
|
||||
contains: ATX headings, fenced code, pipe tables, blockquotes, ordered and
|
||||
unordered lists with indented continuations, thematic breaks, and inline code,
|
||||
strong, emphasis and links. Anything it does not recognise becomes paragraph
|
||||
text, so **no input line is ever dropped** — the failure mode is an ugly
|
||||
paragraph, never a missing one.
|
||||
|
||||
Two details in it are load-bearing and easy to undo. **Code spans are replaced
|
||||
by placeholders before emphasis is matched**, because this corpus routinely
|
||||
opens `**` on one side of a code span and closes it on the other; matching
|
||||
emphasis per segment silently drops that bold, which is a fidelity loss with
|
||||
no visible symptom. And **strong is matched before emphasis, with a body that
|
||||
may contain a single `*` pair**, because `[^*]+` cannot match
|
||||
`**a *b* c**` and the leftover `*` then pairs across the bold — which is how a
|
||||
single sentence in gotcha 33 came out as three italic paragraphs.
|
||||
|
||||
A third, added later and for the same reason: **a table cell boundary is an
|
||||
unescaped pipe outside a code span**, so the row is walked rather than
|
||||
`split("|")`. This corpus documents a file format whose own quoting syntax is
|
||||
`|Time Period|`, and a sentence containing one inside a table cell would have
|
||||
been shredded into extra columns — a row that still parses, so no column count
|
||||
could detect it. It was latent when fixed, which is the point: the trigger is
|
||||
one ordinary sentence away in a document whose subject is pipe quoting.
|
||||
|
||||
One deliberate divergence from GFM inside that rule: a `\|` **inside a code
|
||||
span** in a table cell keeps its backslash, where GFM would unescape it to a
|
||||
bare pipe. Splitting is suppressed inside code spans, so the escape is never
|
||||
needed there and never consumed. The corpus contains no instance; it is
|
||||
recorded as a decision rather than left to be rediscovered as a bug.
|
||||
|
||||
## 6. Navigation
|
||||
|
||||
`repo-map.md` carries a digest of the numbered gotchas and `gotchas.md` the long
|
||||
form; the map was condensed because it was too long to put in every new agent's
|
||||
context. Splitting per gotcha was rejected: the gotchas cross-reference each
|
||||
other constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a
|
||||
reader following those wants a scroll, not one page load per number. Each of the
|
||||
two stays one page, with an id on every list item and a compact strip of
|
||||
numbered links under the heading.
|
||||
The ids come from the list item's **own number**, not from a counter, so an
|
||||
inserted gotcha renumbers the anchors exactly as it renumbers the prose that
|
||||
cites them.
|
||||
|
||||
The strip's contents are **derived from the anchors actually emitted**, never
|
||||
from a count. A hardcoded 48 failed in only one direction: deleting a gotcha
|
||||
left a link to a missing anchor, which the link check catches, while *adding*
|
||||
one produced an unlinked gotcha and a clean build. The list only ever grows, so
|
||||
the hardcoded form was silent in precisely the direction it would be used —
|
||||
the same shape as an exclusion list that over-matches, failing in the direction
|
||||
that looks like success.
|
||||
|
||||
**Long unbreakable headings need `overflow-wrap`.** The test-evidence pages are
|
||||
hundreds of `snake_case` test names as headings; at phone width seven of them
|
||||
overflowed the viewport by up to 92%. The diagnosis is worth keeping because
|
||||
the first guess was wrong: `getBoundingClientRect` reported every element
|
||||
inside the viewport, because the text overflows its box without widening it.
|
||||
Only `scrollWidth` per element names the culprit. Measure the property that
|
||||
actually overflowed rather than the one that usually does.
|
||||
|
||||
## 7. The phone nav is a checkbox, and that is not arbitrary
|
||||
|
||||
At phone width the sidebar's 25 links stacked above the content, so every page
|
||||
opened on roughly 1500px of nav. The wall was not inherited: **all of this CSS
|
||||
is new**, so it shipped for the first time with the site, which is the test for
|
||||
whether a defect is pre-existing and the one most easily got wrong in the
|
||||
direction of declining work you introduced.
|
||||
|
||||
`<details>` is the obvious control and it is **wrong here**, for a reason no
|
||||
spec-level reading predicts. The sidebar layout has to force the disclosure
|
||||
*open* at wide widths, and a closed `<details>` hides its content through
|
||||
`::details-content` — which `display` cannot override. Measured in Chromium,
|
||||
the desktop sidebar came out collapsed to a single link with its toggle also
|
||||
hidden: unusable, and in the half nobody would think to re-check after a
|
||||
phone-only change. A hidden checkbox plus a `<label>` puts the hiding in *our*
|
||||
media query, so forcing it open is ordinary cascade. Anyone reading the markup
|
||||
will see a checkbox where `<details>` obviously belongs; this paragraph is why
|
||||
it must not be tidied into one.
|
||||
|
||||
**And the first probe of the fix could not see the difference.** Counting nav
|
||||
links whose `getBoundingClientRect().height > 0` reported **26 visible in both
|
||||
the collapsed and the expanded state**, because Chrome reports a height for
|
||||
`content-visibility`-hidden children — so it would have passed the broken
|
||||
`<details>` build. `checkVisibility({contentVisibilityAuto: true,
|
||||
visibilityProperty: true})` discriminates.
|
||||
|
||||
That is the same instrument failing twice: §6's overflow fix also measured
|
||||
clean by `getBoundingClientRect` while the page still scrolled. **A measurement
|
||||
that cannot distinguish the two states is not a measurement** — the repo's
|
||||
signature defect, a test that passes while the thing it names is broken,
|
||||
arriving through the instrument rather than through the assertion. Before
|
||||
trusting a browser measurement, injure the thing being measured and check the
|
||||
number moves.
|
||||
|
||||
**And the fix's own probe had the same shape one level up.** Counting *visible*
|
||||
nav links answers "does operating the toggle reveal the links" — correct and
|
||||
complete for a mouse — and is silent on whether any non-mouse route exists. The
|
||||
first version of the checkbox carried `hidden`, which is `display:none`, which
|
||||
removes it from the tab order *and* from the accessibility tree: the label is
|
||||
not focusable on its own, so at ≤900px the navigation could be clicked and
|
||||
reached no other way, and the accessibility snapshot contained neither the
|
||||
checkbox nor any of the 25 links. A visibility count cannot see that, however
|
||||
sharp the visibility instrument. The toggle is therefore visually hidden but
|
||||
focusable at narrow widths and `display:none` at wide ones, and the check is
|
||||
tab order, `.focus()` and the accessibility snapshot — measured at 390, 900,
|
||||
901 and 1280, because the regression lives on one side of a media query.
|
||||
|
||||
So the rule from §6 has a second half: **a measurement of the wrong property is
|
||||
not a measurement either, however precise it is.** `checkVisibility` fixed the
|
||||
visibility instrument; it does not turn a visibility count into an operability
|
||||
check. Ask what the defect would *do* to a user, then pick the instrument that
|
||||
observes that.
|
||||
|
||||
**A correction, and the risk the original argument missed.** The commit that
|
||||
added `content: "\2630 " / ""` said Firefox ignores the alt-text form and falls
|
||||
back to reading the content, so the change was "an improvement in two engines
|
||||
and a no-op in the third". **That mechanism is wrong** — Firefox shipped the
|
||||
syntax in 133, and all three engines parse it (measured: Chrome 141, Firefox
|
||||
142, WebKit 26 each compute the declaration intact). The conclusion held; the
|
||||
reason given for it did not, and the reason was the load-bearing part, because
|
||||
it was what made the change look like a one-way door. It is one because every
|
||||
engine shipped to supports the syntax — not because the unsupported path is
|
||||
harmless. That commit message is superseded by this paragraph (design-principles
|
||||
§6: an argument must not live only in a commit message).
|
||||
|
||||
**The unsupported path is not harmless, and that is the risk neither the change
|
||||
nor its first review named.** `/ alt` in an engine that does not know it is a
|
||||
*parse error for the whole `content` declaration*, so the glyph would not lose
|
||||
its alt text — it would **vanish**. A silent visual regression, the opposite of
|
||||
the assumed failure mode. The glyph is therefore verified painted in all three
|
||||
engines by pixel diff against `::before { content: none }`, with the text
|
||||
starting 20px further right in each.
|
||||
|
||||
That check also produced this file's fourth instance of the wrong-property
|
||||
trap, and the sharpest, because it produced a confident **false positive**: the
|
||||
first probe measured the label's *width* with and without `::before` and
|
||||
reported the glyph missing in all three engines. The label is `display: block`,
|
||||
so its width comes from the container and cannot see inline content at all.
|
||||
Right question, wrong property, and this time the wrong instrument accused
|
||||
working code rather than clearing broken code.
|
||||
File diff suppressed because it is too large
Load Diff
-1129
File diff suppressed because it is too large
Load Diff
@@ -1,580 +0,0 @@
|
||||
# `improvise-incremental`: the reasoning
|
||||
|
||||
`crates/improvise-incremental` began as ADR 009 step 2 (improvise-t8ow, epic improvise-bw4p): the
|
||||
plug. It holds the `Derived` trait, `Eager` (today's evaluator behind it) and the
|
||||
differential harness every backend is checked with, and, from step 4 (improvise-u2le), the
|
||||
first backend that keeps state, `Incremental` (section "Step 4" below). The step 2 sections
|
||||
describe the trait and the oracle; they predate the backend. The decisions are the ADR's
|
||||
(`context/adr/009-incremental-evaluation.md`). The evidence for each test is in
|
||||
`context/test-evidence/improvise-incremental.md`.
|
||||
|
||||
## The shape
|
||||
|
||||
| item | what it is |
|
||||
|---|---|
|
||||
| `Derived` | the two things the evaluator reads: `value(key, agg)` and `raw_fold(partial)` |
|
||||
| `Eager` | an owned `Model` plus its None categories; every answer is a call into `Model` |
|
||||
| `Maintained` | a `Derived` that is told each `Edit`; `Eager` is one |
|
||||
| `Edit` | one model mutation as data (set, clear, add or remove a formula, set a measure's aggregation) |
|
||||
| `probes`, `Probe` | every question worth asking of a model, and a way to put it to any backend |
|
||||
| `differential` | runs one edit sequence on a candidate and on `Eager`, comparing every probe at every step |
|
||||
| `Divergence`, `Report` | what a failure names (step, probe, both answers) and what a pass compared |
|
||||
| `Incremental` | the backend of step 4: maintained groups and occupancy; formulas and `Sum`/`Avg` still go to the model (later steps change that) |
|
||||
|
||||
## Interpretations
|
||||
|
||||
Clause 2 names the trait and says what it carries ("the formula result at a key, and the raw
|
||||
fold of the cells that match a partial key") and nothing else. These are the readings taken.
|
||||
|
||||
1. **`Derived` lives in this crate, not in improvise-core.** The ADR says improvise-core
|
||||
"gains a trait". Putting it there changes an existing crate's source, which the brief for
|
||||
this step ruled out, and the brief was ambiguous about it. This departs from the accepted
|
||||
ADR's text, so it is flagged to the user and not decided here: improvise-bw4p.2 moves it
|
||||
(one file and a `pub use`, since the trait names only core types). While it sits
|
||||
downstream, core cannot read through it, which is the clause's point.
|
||||
2. **Two methods, the minimum.** `value(key, agg)` is the formula result at a key, or the cell
|
||||
as stored or folded; `raw_fold(partial)` is the fold of matching stored cells. `agg` is
|
||||
`Aggregation`, the evaluator's own type. Hidden-item `covers`, totals and occupancy joined the trait
|
||||
in improvise-bw4p.3 (section "Step 4 groundwork"). A group is exercised: a stored key with one coordinate
|
||||
replaced by its item's group is a probe.
|
||||
3. **`raw_fold` is never a formula, and `Eager` enforces it.** For a partial naming a formula
|
||||
target `Eager::raw_fold` is `None`; otherwise it is `evaluate_aggregated(partial, OverHidden)`,
|
||||
because the evaluator's own raw fold (`aggregate_raw`, `aggregate_cell_values`) is private to
|
||||
core. Calling it through the public path is exact for any partial that names no formula
|
||||
target. improvise-bw4p.2 exposes the raw fold from core so `Eager` stops going round.
|
||||
4. **`Eager` recomputes formulas after every edit.** Production does it in `rebuild_layout`
|
||||
(`Model::recompute_formulas(none_cats)`), and an over-hidden read of a formula cell reads
|
||||
the cache it fills. So `Eager` carries the None categories of the view it stands for and
|
||||
recomputes under them after each edit, as a layout rebuild does.
|
||||
5. **`Edit` is the vocabulary of the harness, not of the wire.** Five variants, each one
|
||||
public `Model` call. A later step adds variants for what it maintains (an item added, a
|
||||
category removed); the harness does not care which. An `Edit` fault reaches both sides of a
|
||||
differential alike, so it is pinned only where a direct `Eager` test reads the result:
|
||||
`ClearCell`, `AddFormula` and `RemoveFormula`, and `SetMeasureAgg` through the fold tests.
|
||||
`SetCell` is read by nearly every test.
|
||||
6. **Values agree by float rules with every NaN one value** (`same_value`), the rule clause 3
|
||||
states for cell deltas, except the two zeros: they differ, since a drawn `-0.00` is not `0.00`
|
||||
and eager's `Min` and `Max` return the sign they were given. Clause 8 has its own tests
|
||||
for the sign of a zero sum.
|
||||
7. **The `zset` dependency arrived with step 3**, as the cell delta (below).
|
||||
|
||||
## Step 3: the cell delta (improvise-5r8c)
|
||||
|
||||
`delta.rs` holds `DeltaValue`, `CellDelta = ZSet<(CellKey, DeltaValue)>`, `cell_delta` and
|
||||
`moved_records`. `improvise-command`'s `apply_effects` reads them for the record tier and the
|
||||
dirty flag (`context/code/command.md`). Evidence: `context/test-evidence/improvise-incremental.md`.
|
||||
These are the readings taken, where ADR 009 clause 3 leaves a choice.
|
||||
|
||||
1. **`DataStore` logs, `improvise-incremental` builds the Z-set.** The ADR says the store
|
||||
"emits a `ZSet`", but core never depends on `zset` (clause 2). `DataStore::record_changes`
|
||||
turns on a `Vec<CellChange>` log (`None` while off, so an unwatched store keeps nothing),
|
||||
`drain_changes` empties it and `cell_delta` folds it. The log lives in the store, not in
|
||||
`Model::set_cell`, so every route that writes a cell is seen, including `RemoveCategory`'s
|
||||
sweep.
|
||||
2. **`DeltaValue` canonicalises on construction**: every NaN becomes `f64::NAN`, `-0.0` becomes
|
||||
`+0.0`, and the order is `total_cmp` on the result, with numbers before text before errors.
|
||||
The inner value is private, so a non-canonical one cannot be built. Equality is the order's
|
||||
`Equal`, so `==` and `Ord` agree.
|
||||
3. **`CellKey` is `Ord` by derivation**: the sorted coordinate pairs, compared as strings.
|
||||
Clause 3 calls this "the natural order of ids", which ADR 007's id form will change; the
|
||||
derive then moves with the key type.
|
||||
4. **The record tier is the set of records a consolidated batch delta names.** A cell written
|
||||
and written back inside one batch cancels, as the old XOR did.
|
||||
5. **The dirty flag needs two inputs the ADR does not name.** Besides a non-empty delta and
|
||||
the formula set, `document_config` stays (an `AddItem` writes no cell and the existing test
|
||||
requires it dirty) and the saved order joins, which no delta carries: `DataStore::reorders`
|
||||
counts `SortData`, and any logged insertion or removal dirties too, because a clear-then-set
|
||||
of one value nets to zero yet moves the cell to the end of the store (a same-item rename).
|
||||
6. **`document_generation` and the wire are untouched**, and `Eager` and the `Derived` trait
|
||||
are not used: no backend consumes the delta yet. The differential for this step is the
|
||||
delta against the store it describes (`the_cell_delta_of_an_edit_is_...`), since there is
|
||||
nothing to compare with `Eager`.
|
||||
|
||||
## What the harness checks
|
||||
|
||||
`differential(model, none_cats, candidate, edits)` builds `Eager` over a clone of the model,
|
||||
applies each edit to both, and after step 0 and after every edit asks both every probe. The
|
||||
probe set **accumulates**: probes from the model at every earlier step stay asked, so a backend
|
||||
that goes on answering for a removed formula diverges. A pass returns a `Report` of how many
|
||||
states and answers were compared.
|
||||
|
||||
`probes(model)` takes the base keys (each stored key, and each with one coordinate replaced by
|
||||
its item's group), every sub-partial of a base key dropping at most `MAX_DROPPED = 3`
|
||||
coordinates plus the empty key, and each formula's target over each of those. Each key is asked
|
||||
as an exact value, an over-hidden value and a raw fold. A fold dropping more than three
|
||||
coordinates from a stored key is not asked about. `edit_probes`
|
||||
adds the key an edit names, so a cleared cell is asked about after it is gone. A divergence
|
||||
reads `divergence at step N: <probe>: eager <a> != candidate <b>`.
|
||||
|
||||
## Step 4 groundwork: what a view draws (improvise-bw4p.3, improvise-bw4p.4)
|
||||
|
||||
The subset probes above cannot see a drawn cell on a view with four or more None categories
|
||||
(it drops more than `MAX_DROPPED` coordinates of a stored key), and the trait could not be asked
|
||||
about hidden items, totals or occupancy. Both are closed. These are the readings taken.
|
||||
|
||||
1. **`Reading` is how a view reads.** The aggregation (`Aggregation::of` its None categories)
|
||||
and the hidden `(category, item)` pairs of `View::hidden_items`. `covers(cat, item)` is the
|
||||
predicate `GridLayout::evaluator` passes as its `covers`. It is a value, not a closure, so a
|
||||
probe can name it and a divergence can print it. A fold over a group or a spanned category
|
||||
leaves hidden members out; a read of a key naming neither does not (a hidden East still counts
|
||||
in `{Product=A}` when Region is None), while an occupancy count always leaves them out. The
|
||||
count follows the drawn items, not the value; `Eager` mirrors production in both.
|
||||
2. **Three methods join the trait**: `covered(key, reading)` is `Model::evaluate_covering`;
|
||||
`total(key, over, reading)` is `evaluate_spanning`, a column total as ONE fold over the
|
||||
visible items of the omitted categories; `occupancy(partial, reading)` is ADR 009 clause 7's
|
||||
count by axis item: the stored cells matching `partial`, none of whose coordinates is hidden,
|
||||
where a coordinate naming a group matches every stored member of that group (a collapsed group
|
||||
is a remap, so the Coastal line of two visible members counts 2). A key naming a formula
|
||||
target has no stored cells and is not asked. "Has a value" (what `prune_empty` asks) is
|
||||
`covered(..).is_some()`, already probed.
|
||||
3. **`drawn_probes(model, views, none_cats)` asks what each view draws**, built from the view's
|
||||
own `GridLayout` (pruned as production builds it): for every drawn cell the three subset
|
||||
questions plus `Covered` and `Occupancy`; for every column its `column_key` occupancy and, where
|
||||
`GridLayout::draws_totals` (the condition `grid_body` also reads), a `Total` over the row
|
||||
categories; for every row the occupancy of its cell key less the column coordinates. Lines
|
||||
carry the page coordinates, as a cell's key does, so a page is its own slice. A records view is
|
||||
skipped: its keys are synthetic `_Index`/`_Dim` coordinates naming no stored cell and it has
|
||||
no totals, while its stored keys are the subset probes' business. A view whose None categories
|
||||
differ from the oracle's is skipped, since `Eager` recomputes its formulas under one set (a
|
||||
backend keeping a cache per aggregation class is therefore not exercised on the others).
|
||||
`differential_in(model, views, ..)` adds them at every step; `differential` is that with no
|
||||
views. A view is a `View`, not a document, because the model holds none.
|
||||
4. **`sub_partials` is polynomial**: it enumerates the sets of at most `MAX_DROPPED` dropped
|
||||
indices, not the 2^n masks. The probe set is unchanged for every key length (a test compares
|
||||
it with the old walk up to 12 coordinates, and counts it at 64). `MAX_DROPPED` stays: it
|
||||
bounds the subset probes, and the drawn probes carry the cells that drop more.
|
||||
5. **The hidden fixture is a workbook the corpus now includes** (`hidden`): no shipped demo
|
||||
hides an item, so a backend wrong about hidden items would have passed the direct
|
||||
`Eager`-against-evaluator comparison as well.
|
||||
6. **`Probe` is `Hash + Eq`**, so `Asked` deduplicates on the probe itself (`Aggregation` gained
|
||||
`Hash` in core for it).
|
||||
|
||||
## Step 4: the incremental backend (improvise-u2le)
|
||||
|
||||
`Incremental` (`incremental.rs`) is the first `Derived` that keeps state. It owns a `Model`
|
||||
(the store) and `Shapes` (`shapes.rs`, the groups). `Maintained::apply` applies the edit to the
|
||||
model, drains the store's change log into a `CellDelta` and feeds the delta to every shape. A
|
||||
read asks a group, never the store, for what step 4 owns. The rest of this section is the readings
|
||||
taken where clause 4 and clause 7 leave a choice; evidence is in
|
||||
`context/test-evidence/improvise-incremental.md`.
|
||||
|
||||
1. **A shape is a set of category names, not a None list.** Clause 4 says "a group is the cell key
|
||||
minus the None categories". The trait is asked about keys of any category set (the harness
|
||||
drops up to three coordinates of a stored key), so the unit is the set a key names: a `Shape`
|
||||
maps the projection of a stored cell onto those categories to a `Group`, and a cell lands in
|
||||
it only if it has all of them. A view's None list is one of these sets. Clause 7's "one
|
||||
circuit serves every shape key with the same None list" holds trivially, since the sets are
|
||||
shared by every reader.
|
||||
2. **A shape is built on its first read, by one pass over the store, and fed every delta after.**
|
||||
Clause 7 makes a new key's first build a sliced job under ADR 008's job runner. That runner
|
||||
does not exist, so the pass is synchronous and counted (`Meter::builds`, `built_cells`).
|
||||
Nothing evicts a shape. `Shapes` holds its map in a `RefCell` because `Derived` reads take
|
||||
`&self`; the cell is private to `shapes.rs` and no reference to it leaves a method.
|
||||
3. **A group is a count, a count of numbers and a multiset** (`group.rs`). One update touches one
|
||||
map entry (O(log d)); the count and the multiset serve every function, so
|
||||
`SetMeasureAgg` rebuilds nothing. A read of several groups (a collapsed group, a total) folds
|
||||
their union without merging them: `fold_groups` sums the counts and takes the smallest of the
|
||||
groups' minima, the largest of their maxima, or the one value they share.
|
||||
4. **Eager's rules are reproduced, not improved** (superseded for NaN and zeros by "Eager's math, fixed" below). An all-numbers group reduces; any other group
|
||||
is unanimous or `Error("mixed")`, so `Count` of a group holding text is that text. `Min` and
|
||||
`Max` skip NaN and are `+inf`/`-inf` over only NaN, as `fold(INFINITY, f64::min)` is. The
|
||||
multiset orders by the delta's total order, with NaN last, so the skip is a step past one
|
||||
key. A group holding a NaN and a non-number is mixed, because NaN is not equal to itself in
|
||||
`CellValue`'s `==` and is a distinct key here.
|
||||
5. **`Sum` and `Avg` stay with the store until step 6.** Their eager result depends on the
|
||||
order the cells are added in (insertion order in the store, not `CellKey` order, whatever
|
||||
clause 3 says), so a group cannot reproduce it without keeping that order. The backend routes
|
||||
a read whose measure aggregation is `Sum` or `Avg` to `Model::evaluate_covering` and
|
||||
`evaluate_spanning`, which is eager's own path. Step 6 replaces both sides with `ExactSum`.
|
||||
Because `Sum` is the default, the corpus reaches the groups only once a `SetMeasureAgg`
|
||||
edit has changed a measure; the differential tests apply those on purpose.
|
||||
6. **Formulas stay with the model until step 5.** A key a formula owns is read through the model,
|
||||
and `apply` calls `Model::recompute_formulas`, as `Eager` does. Step 5 replaces the call.
|
||||
7. **A fold is the union of its members' groups.** `fold.rs` is `Model::group_expansion` read off
|
||||
the public model: the cross product of the covered members of every group the key names and
|
||||
every covered item of every category in `over` that it omits. Under `OverHidden` the members
|
||||
are groups of one shape (the base's categories plus the spans'). Under `Exact` the cells
|
||||
admitted are exactly the member keys, so the read is a lookup per member and a throwaway
|
||||
`Group`. A total therefore reads one group per covered row, not one per stored cell, but still
|
||||
one per row: clause 7's maintained totals with a subtracted hidden overlay are not built (see
|
||||
"Not built").
|
||||
8. **Occupancy counts by inclusion and exclusion.** `occupancy(partial)` is the cells under
|
||||
`partial`, where a coordinate naming a group matches each of its members' items, with no
|
||||
hidden coordinate. For each way of choosing a member per group coordinate the count is the
|
||||
group's cell count, less the cells holding a hidden item of a category the key leaves free,
|
||||
by inclusion and exclusion over those categories: `N(P) - sum N(P + h) + ...`. The cost is
|
||||
the product of the hidden items per free category, a group read each. A key naming a hidden
|
||||
item itself is zero.
|
||||
9. **The meter counts operations.** `Meter` is `updates` (group updates a delta caused), `reads`
|
||||
(groups a read looked at), `builds` and `built_cells`. `tests/measure.rs` fixes one table at
|
||||
two sizes and asserts the counts: an overwrite costs two updates per shape whatever the table
|
||||
holds, and a read of a group of 36 or 72 cells costs one group. Eager's cost for the same read
|
||||
is `matching_values().len()`, which the test prints beside it. This is the local fixture the
|
||||
brief asked for; ADR 009 step 4 says "measured at ADR 008's fixture", which needs ADR 008
|
||||
(unapproved) and is deferred.
|
||||
|
||||
10. **`Min` and `Max` keep the sign of a zero.** The multiset calls `-0.0` and `+0.0` one
|
||||
value (clause 3), so a group also counts its negative zeros from the raw change log, as step 6's
|
||||
sums are fed. Eager's `f64::min` and `f64::max` leave the sign of a tie between the two zeros to
|
||||
the platform and the order (`fmin` on ARM returns `-0.0`; x86 may not). The backend reproduces the
|
||||
part that does not depend on either: all zeros negative gives `-0.0`, all positive gives `+0.0`.
|
||||
With both, `Min` is `-0.0` and `Max` is `+0.0`; eager now returns the same (see "Eager's math,
|
||||
fixed").
|
||||
|
||||
### Not built in step 4
|
||||
|
||||
- Maintained totals (clause 7): a total reads one group per covered row.
|
||||
- Eviction of shapes, and the build as a sliced job.
|
||||
- `Sum`, `Avg` (step 6) and formulas (step 5) are eager's, behind the same trait.
|
||||
- The ADR 008 fixture measurement.
|
||||
|
||||
## Step 5: formulas (improvise-shbc)
|
||||
|
||||
`Incremental` calls `recompute_formulas` only for the documents item 2 calls wide. Otherwise it keeps
|
||||
the formula cache itself (`Model::formula_cell`, `set_formula_cell`, `clear_formula_cells` and the
|
||||
now-public `evaluate_formula_cell`, the small additions to `improvise-core`) and decides which cells of
|
||||
it to evaluate again; the evaluator that computes each value is core's own. The ADR (clauses 5 and 6)
|
||||
is followed in outline, and the places it is not are the reason for this section.
|
||||
|
||||
1. **A stem is the unit, because eager's cache is per stem.** `Stems` (`formulas.rs`) holds, per
|
||||
target category, how many stored cells give each stem (a cell minus that category and the
|
||||
None categories, as `recompute_formulas` derives them), and which stems at least one category
|
||||
gives (`live`). Counts come from the raw store changes, one net count per stem per category per
|
||||
batch, so an overwrite never makes a stem vanish and reappear. A stem that appears is
|
||||
evaluated; one that goes has its cells removed.
|
||||
2. **A stem's cells are evaluated from nothing.** `recompute_formulas` clears the cache and runs
|
||||
fixed-point passes over every stem together. When no formula reads another stem's cache the stems
|
||||
are independent, so running the same passes on one stem from an empty cache gives exactly what
|
||||
the whole run gives for it, including `circular` for a cell still moving after sixteen
|
||||
passes. That is why a dirty stem is cleared first (`passes-start-warm` shows what a warm start
|
||||
changes). Where they are not independent (`Stems::is_wide`: two target categories, or a name in
|
||||
another category that is not None) the backend falls back to `recompute_formulas` for the whole
|
||||
cache, so those documents are exact and not incremental.
|
||||
3. **Which stems a change reaches is a superset, read off the formulas.** For formula `f`,
|
||||
`Reads` is the set of names in its expression and in the formulas those names target, and the
|
||||
categories that puts in play (its target's, every name's, every `WHERE`'s and `IS`'s). A stored
|
||||
cell can change what `f` evaluates to at stem `s` only if it agrees with `s` everywhere outside
|
||||
those categories. `Readers` indexes each live stem by what is left of it outside them, per
|
||||
category set, so a changed cell finds its stems by looking up its own projection on each set:
|
||||
no scan of the stems. This is clause 5's input rule made coarser, once for every shape of
|
||||
formula: it re-evaluates a stem for a cell of an item the formula never names (a change to a
|
||||
`Note` cell evaluates the stems `Profit` has). It cannot miss one, which is what the aggregate
|
||||
over an expression needs, since the cells it reads are chosen at run time by the records.
|
||||
4. **The aggregate-over-expression pipeline (clause 5, items 1 to 3) is not built.** A dirty stem
|
||||
evaluates its formulas whole, so `SUM(Revenue * Price)` is re-folded over its records when any
|
||||
cell under the stem changes, not adjusted for the one record. That needs the evaluator to
|
||||
accept a maintained aggregate in place of folding, which is a change to core's evaluator and
|
||||
to the order its records are visited in (insertion order, not `CellKey` order). It is a bead.
|
||||
5. **Dependency order is an optimisation and is checked against the plain passes** (the NaN and zero differences below are gone: "Eager's math, fixed"). `Order` is the
|
||||
formulas' dependency graph by name. When it has no cycle and there are fewer formulas than
|
||||
passes, one evaluation per cell in dependency order reaches the fixed point eager's passes
|
||||
reach (a chain of fifteen links needs fifteen passes and a sixteenth to see no change). It is the same fixed point, because Eager's convergence test compares by bits ("Eager's math, fixed").
|
||||
`Evaluation::Plain` always
|
||||
runs the passes; `tests/ordering.rs` compares the two after every edit, zero signs included, and
|
||||
`tests/formulas.rs` compares both with eager.
|
||||
6. **A formula edit rebuilds the stems.** Adding, removing or replacing a formula (the fingerprint
|
||||
of the formula list moves) rebuilds the stems from the store by one pass, clears the cache and
|
||||
evaluates every stem, because a formula has a cell at each. Clause 5 wants a formula delta;
|
||||
the index is not maintained across a formula edit. `Meter::rebuilds` counts it.
|
||||
7. **A new aggregation dirties the stems it can reach.** `measure_agg` is not a cell, so no change
|
||||
reaches the readers. A fold under a None category takes its aggregation from the `_Measure`
|
||||
coordinate of the key, which for a formula targeting another category comes from the stem and
|
||||
from no name the formula reads, so every live stem carrying that measure is dirtied
|
||||
(`Stems::carrying`). Where a formula names the measure (it targets `_Measure`) `reads_name` dirties every stem.
|
||||
8. **A write of the same bits changes nothing; a write of `-0.0` over `0.0` does.** The cell delta
|
||||
treats the two as one value (clause 3), but a formula may not (`Units ^ -1` is `inf` and
|
||||
`-inf`), so the formula side reads the raw change log, not the canonical delta.
|
||||
|
||||
### Not built in step 5
|
||||
|
||||
- The aggregate-over-expression pipeline and the per-record order key (item 4).
|
||||
- Stems that survive a formula edit (item 6), and name-level selectivity within a stem (item 3).
|
||||
- Incremental cache for documents with formulas in more than one target category, or a name in
|
||||
a non-None category other than the formula's own (item 2): they use the whole recompute.
|
||||
- Strongly connected components as a maintained structure: `Order` is rebuilt with the stems, which
|
||||
is O(formulas) on a formula edit, and cycles are found by depth-first search, not Tarjan's.
|
||||
|
||||
## Step 6: Sum, Avg and totals through an exact sum (improvise-hrjz)
|
||||
|
||||
ADR 009 clause 8, in both backends. `Sum` and `Avg` stop being the one reading that went to the
|
||||
store: a group now holds a `numtower::ExactSum` beside its counts and multiset, and `Reduce`
|
||||
(was `OrderFree`) has all five functions. Step 4's delegation, and its sentence in section 5
|
||||
of that step, are gone; formula-owned keys still go to the model, whose cache step 5 maintains.
|
||||
|
||||
1. **`ExactSum` already signs a zero total** (`numtower/src/sum.rs`, PR #348, written for step 8's
|
||||
last test): `-0.0` only if every finite addend is `-0.0`, `+0.0` for any other zero, an exact
|
||||
cancellation included. An earlier version of this step wrote that rule as a wrapper, `FloatSum`;
|
||||
it was dropped when `gh/main` brought the same rule into `ExactSum`, and the tests of the rule are
|
||||
that PR's `tests/zero_sign.rs`. One difference from the wrapper: an empty sum reads `+0.0`, where
|
||||
`Sum for f64` starts at `-0.0`. Core's `SUM` over no rows therefore changes from `-0.0` to `+0.0`
|
||||
(a formula `SUM` whose `WHERE` matches nothing); `a_formula_sum_over_nothing_is_positive_zero`, in core,
|
||||
pins it.
|
||||
2. **Core sums through it.** `apply_agg` (`Sum`, `Avg`) and the formula-level `SUM` in `eval_agg` call
|
||||
`exact_sum`, so `Eager` is exact, and so is every reading of the model, with no flag: the user
|
||||
visible change clause 8 names (displayed sums change in their last bits) is this commit. `improvise-core`
|
||||
gains the `numtower` dependency; both of the new edges are in `Cargo.nix` and the harness lock.
|
||||
3. **A group's sum is fed from the raw changes, not the cell delta.** The delta calls `-0.0` and
|
||||
`+0.0` one value (clause 3), so an overwrite of `0.0` by `-0.0` is no delta at all, yet it moves
|
||||
the sign of a zero sum. `Shape::apply` takes the delta for counts and the multiset and the change
|
||||
log for sums, skipping only a change that left the same bits. The ADR's two clauses 3 and 8 are
|
||||
consistent only because the sum reads the store's values and not the delta's. The cost is a second
|
||||
walk of the batch per shape.
|
||||
4. **Totals and collapsed groups absorb.** A read of several groups (`total`, a collapsed group,
|
||||
a hidden member left out) adds their accumulators with `absorb` and reads once, so a total is the
|
||||
same bits whatever the grouping and whatever the order of the rows. The clause 7 hidden overlay
|
||||
(subtracting the hidden cells from a kept total) is still not built; a total is a union of member
|
||||
groups, not a kept total less an overlay.
|
||||
5. **`Avg` is the exact total over the cell count, rounded once** (`ExactSum::mean`). The first
|
||||
version divided a rounded sum, which rounds twice and overflows with the sum: `Avg([MAX, MAX])`
|
||||
was `inf` and a correctly rounded mean differs from the twice-rounded one for about 2% of random
|
||||
multisets. Both backends now use `mean`, so this is a second user-visible numeric change
|
||||
beside the sums: a displayed average can differ in its last bit, and a mean whose sum would
|
||||
overflow is finite. `numtower/tests/mean.rs` checks it against exact rationals.
|
||||
6. **What a group costs.** An `ExactSum` is 70 limbs of `i64` and a few counters, about 650 bytes
|
||||
(`size_of`), in every group of every shape, whether or not the measure is summed. Clause 8
|
||||
accepts 568 bytes "per summed group"; here the shape does not know which measure its groups
|
||||
serve, so a shape at cell granularity (a read of a full key) pays it for each cell. Sparing
|
||||
groups that are never summed, or holding the sum only above one cell, is the improvement; it
|
||||
is a bead.
|
||||
7. **The drift test is 20,000 overwrites, not a million.** ADR 009 measured 1.7e-9 at 18,000 cells
|
||||
after a million cycles. `tests/exact.rs` runs 20,000 overwrites of a 1,000-cell group with
|
||||
values chosen so an `f64` sum drifts, and asserts the two backends equal each other and a sum
|
||||
built afresh to the bit. The count is what a test binary can afford; the property it pins is
|
||||
independent of it.
|
||||
|
||||
### Not built in step 6
|
||||
|
||||
- The hidden overlay and kept totals (clause 7).
|
||||
- A group that does not pay for a sum it never reads (item 6).
|
||||
|
||||
## Eager's math, fixed (user ruling 2026-10-10)
|
||||
|
||||
"Don't reproduce math bugs you might find in the Eager mode." Every place the earlier steps matched
|
||||
Eager's arithmetic "as eager does" was reviewed. The items below supersede the sentences in steps 4 and 5
|
||||
that say otherwise (step 4 items 4 and 10, step 5 item 5). Each is a user-visible numeric change.
|
||||
|
||||
1. **Min and Max of both zeros** is `-0.0` and `+0.0`, whatever the order or the platform. Core's fold used
|
||||
`f64::min` and `f64::max`, whose tie between the zeros is left to the platform; it now orders by
|
||||
`total_cmp` over the non-NaN values. Incremental already returned this. (improvise-bw4p.20)
|
||||
2. **Min and Max over only NaN** are NaN, where they were `+inf` and `-inf` (the seed of the old fold leaking
|
||||
out). NaN among numbers is still skipped.
|
||||
3. **A NaN formula result is a value, not a cycle.** The fixed point compared cached cells with `!=`, so a
|
||||
cell that settled on NaN never settled and became `circular`, dragging the cells that read it. It now
|
||||
compares with `model::types::settled`: numbers by bits, every NaN one value. A cell that really oscillates
|
||||
is still `circular` after sixteen passes. `mark_nan_circular` is gone.
|
||||
4. **A zero's sign is a change.** `settled` compares by bits, so `0.0` becoming `-0.0` between passes moves
|
||||
the cell, and eager's fixed point and dependency order's agree. Dependency order is the unique fixed point
|
||||
of an acyclic set under bit equality, up to NaN payload and sign (`settled` merges NaNs): each cell is a
|
||||
function of the final values it reads. So the
|
||||
`holds_a_zero` fallback is dropped and `Ordered` needs no extra precondition.
|
||||
5. **Empty `SUM`** is `+0.0` (step 6), listed here for completeness.
|
||||
6. **Cycles over zeros and NaN.** A cycle that only flips a zero's sign (`X = -X`, `X = X * -1`,
|
||||
`IF(X < 1, -X, X)`, `A = -B` with `B = A` over zeros) used to settle, because `-0.0 == 0.0`; it is now
|
||||
`circular`, as the same cycle over nonzero data always was. A cycle fed a NaN (`A = A + 1`, or `A = A`
|
||||
with `A` NaN) used to be `circular` and now settles on NaN.
|
||||
7. **`Min` and `Max` skip NaN while `Sum` and `Avg` propagate it.** Kept: it is IEEE's `minimumNumber` and
|
||||
`maximumNumber` for the extremes and ordinary addition for the sums, and consistent with ADR 005.
|
||||
Whether a spreadsheet user expects one rule for both is a product question (bead text in the hand-back).
|
||||
|
||||
### Left alone, with the reasoning
|
||||
|
||||
- **`Count` of a group that holds text** returns the text (the unanimity branch of
|
||||
`aggregate_cell_values`), not the number of cells. That is probably not what a user wants, but it is a
|
||||
decision about what a `Count` measure means for non-numbers, not an arithmetic error, so it is filed
|
||||
rather than changed.
|
||||
- **The division and equality epsilon** (`FLOAT_EQ_EPSILON`, 1e-10) is shared and deliberate (gotcha 17).
|
||||
- **A group holding NaN beside text** is `mixed`: NaN is a number and text is not, so no unanimity exists.
|
||||
- **`Avg` and `Sum`** are already exact and correctly rounded (step 6).
|
||||
- **Overflow of the exact sum to infinity** follows `ExactSum::value` (round to nearest, so an overflowing
|
||||
total is infinite): correct IEEE behaviour.
|
||||
|
||||
|
||||
## Step 7: live sources (improvise-rnak)
|
||||
|
||||
ADR 009 clause 2 names two kinds of live source: one that reports its changes, which feeds cell deltas
|
||||
like `DataStore`, and one that answers aggregate queries but reports none, which implements the derived
|
||||
value by pushdown. The core half of both is in `context/code/source.md` (`CellSource`, `Model::over`).
|
||||
The brief ordered this half after the evaluator hook; the hook came first because a source that reports
|
||||
changes still has its formulas evaluated over it. User ruling 2026-10-10: formulas are computed in RAM
|
||||
from the source's folds, never pushed down; `Derived` stays in this crate.
|
||||
|
||||
### Change-reporting (`Incremental::over`)
|
||||
|
||||
1. **`ChangeSource` is `CellSource` plus `drain_changes(&self)`, and a source's cells are those as of the
|
||||
last drain.** A shape is built lazily, by a pass over the source, at its first read. If a source
|
||||
showed a write before reporting it, a shape built in between would hold the write and the next
|
||||
drain would add it again. So the contract is that nothing is visible until it is drained, which is
|
||||
what a database does inside a read transaction and what `DataStore` does already (its log is
|
||||
drained in the step that made the edit). `&self` because a live source is shared with the model that
|
||||
reads it and changes behind it by interior mutability of its own.
|
||||
2. **`Incremental::over(model, feed, none_cats)`** takes a `SharedFeed`, the alias core exports (a generic
|
||||
`Arc<S>` could not take it: `S` is `Sized`). The model is put over the feed (`Model::over`), which
|
||||
drops the cells it came with, so none is left for `get_cell`, persistence or the harness to read. Its categories, formulas and aggregations are the definition; the
|
||||
source holds the data. Shapes and stems are built from the feed as it stands.
|
||||
3. **`sync()` drains and maintains; `apply_changes(&[CellChange])` maintains changes the source already
|
||||
reflects**, for a source whose changes reach the backend by another route (a trigger table, a
|
||||
channel). `Maintained::apply` on a fed backend takes definition edits (a formula, an aggregation)
|
||||
and drains: a cell edit is the source's to make, and `apply` refuses it with an `assert!` in every
|
||||
build (a release build lost it silently when only `Model::set_cell`'s `debug_assert` stood in the way).
|
||||
**A feed has one drainer**: a drain is destructive, so two backends over one feed split its changes
|
||||
(`context/code/source.md`, `ChangeSource`).
|
||||
4. **Freshness is the source's.** ADR 003 left "polled, triggered, or stale until refreshed" open: `sync`
|
||||
is the poll, and when it is called is the host's decision. Nothing here schedules it.
|
||||
5. **What a change costs** is what it costs for the owned store: two group updates and two exact-sum
|
||||
updates per maintained shape, no pass over the source (`the_cost_of_a_reported_change_...` counts the
|
||||
rows the source serves for snapshots: zero after the build). Formulas are evaluated at the stems the
|
||||
change reaches, as in step 5.
|
||||
6. **Three routes into the store were routed through the source**: `Shape::of` (the build) reads
|
||||
`cells(&[])`, `Stems::build` reads `keys(&[])`, and `Incremental::read` reads `value_at`. `Eager::occupancy`
|
||||
reads `keys(&[])` too, so an `Eager` over a source would answer it.
|
||||
|
||||
`tests/support/live.rs` holds `LiveTable`, the scripted source (an indexed store behind a lock, with
|
||||
a pending list and a counter of snapshot rows) and `Fed`, an `Incremental` over one that a differential
|
||||
can drive: a cell edit becomes a write to the table followed by `sync`.
|
||||
|
||||
### Query pushdown (`Pushdown`)
|
||||
|
||||
For a source that answers aggregate queries and reports nothing. `Pushdown::new(definition, source,
|
||||
none_cats)` puts the model over the source (`Model::over`) and keeps nothing else: no shapes, no
|
||||
change log, no stems. These are the readings taken.
|
||||
|
||||
1. **`AggregateQuery` is `CellSource`; there are two traits, not three.** The brief's `AggregateQuery`
|
||||
(`fold`, `count`) is the trait core already has: `summarize` is the fold, `count` the count. `Derived`
|
||||
stays here (user ruling) and stays a different thing: the values a view draws, which a backend
|
||||
builds from a source's aggregates and its own state. A source implements `CellSource` and a backend
|
||||
implements `Derived`.
|
||||
2. **Folds are queries, formula cells are computed in RAM.** `value`, `raw_fold`, `covered` and `total` are
|
||||
the model's evaluator over `Model::source()`, so a fold is one `summarize` and the rest is arithmetic
|
||||
on the answers. The formula cache is the one thing `Pushdown` holds. That is the user's ruling
|
||||
(2026-10-10), and it makes the evaluator hook the only change to core's evaluation.
|
||||
3. **Staleness is `refresh()`.** A stored cell or a fold is current at the moment of the read. A formula
|
||||
cell is the cache's value until `refresh` recomputes it from the source as it now stands, which
|
||||
costs one `distinct_without` per formula and the folds each formula reads. When to refresh is the
|
||||
host's (ADR 003's "polled, triggered, or explicitly stale"); nothing here schedules it. A
|
||||
delivery layer that wants notices from a pushdown-backed shape has no moved keys to name, so it
|
||||
marks them "correct" (ADR 009 clause 9), which is a bead (below).
|
||||
4. **Occupancy is one `count`.** A group in the partial becomes a span over its own name and its
|
||||
members; the view's hidden pairs go as `excluding`. `Eager::occupancy` lists every key to say the
|
||||
same thing.
|
||||
5. **`define(edit)` takes a formula or an aggregation edit and refreshes.** A cell edit is the source's:
|
||||
`define` refuses it with an `assert!` of its own in every build (`a_cell_edit_on_a_pushdown_is_refused`).
|
||||
6. **What the differential proves, and what it does not.** `value`, `raw_fold`, `covered` and `total` are
|
||||
the same four functions for `Eager` and `Pushdown` (`reads.rs`; they were copies, and are shared now),
|
||||
so a Pushdown-against-Eager differential cannot see a mistake in them. `Pushdown::members` repeats
|
||||
`Eager::group_holds` in a different form (a span list against a predicate), and that repetition
|
||||
is compared. What the differential proves is the **source's** semantics: that `summarize`, `count`
|
||||
and `distinct_without` give, for every select the evaluator and the views make, the answers a
|
||||
`DataStore` gives. The evaluator over a source is proven equal to the evaluator over the owned
|
||||
store in `improvise-core` (`over_tests`). Nothing else is claimed.
|
||||
7. **A refresh wants one snapshot.** `refresh` asks `distinct_without` and then folds, so a source that
|
||||
changes between the queries can serve them from different revisions. The passes heal what they
|
||||
can: a write between two folds is read again by the next pass, and the cache ends equal to the new
|
||||
state's values (`writes_between_two_queries_of_a_refresh_are_healed_by_the_fixed_point`). A stem
|
||||
that appears after the stem list was taken has no cell until the next refresh
|
||||
(`a_stem_that_appears_between_the_stem_list_and_the_folds_waits_for_the_next_refresh`). A real source
|
||||
should serve a refresh from one read transaction; nothing here can ask it to.
|
||||
8. **Known evaluator defect: the result can depend on the order of the stems** (improvise-bw4p.30). The
|
||||
passes of the fixed point update the cache in place, visiting stems in the order `distinct_without`
|
||||
lists them, and stop at sixteen. A document whose stems read each other settles in one pass listed
|
||||
along its dependency chain and takes one pass per link listed against it, so an eighteen-link chain
|
||||
is `circular` the second way. That is the evaluator's fault, not an obligation on a source: a SQL
|
||||
source has no order to keep, and bw4p.30 is to make the fixed point independent of it. Until then
|
||||
the first-seen order (`DataStore`'s insertion order) is what the oracle uses, and
|
||||
`the_order_stems_are_listed_in_decides_a_result_today` pins today's behaviour: along the chain 18,
|
||||
sorted `circular`, and a source listing its stems sorted is a divergence. No fixture before it had a
|
||||
result that depended on the order. When bw4p.30 lands the test's sorted half must change to expect
|
||||
agreement, and this item goes.
|
||||
9. **What is and is not proven.** The tests run against `Remote` (`tests/support/remote.rs`), a scanning
|
||||
in-memory table with counters, that shares no evaluator with `Eager` and answers queries over rows.
|
||||
Without a real SQL engine only the **contract** is proven: that a source answering these six
|
||||
queries as specified gives `Eager`'s answers, and what each read costs in queries. Nothing here
|
||||
shows that SQLite can answer `summarize` exactly (an exact sum is not what its `SUM` returns:
|
||||
`context/code/source.md`), or what a query costs. The first real source is improvise-rucf, still
|
||||
deferred.
|
||||
|
||||
### What step 7 did not predict
|
||||
|
||||
- **The hook came before the change feed**, not after (a fed backend's formulas are evaluated over the
|
||||
source; see `context/code/source.md`).
|
||||
- **`TableSource` and `LiveTable` found nothing in `Incremental`**; the corpus and the random edits pass
|
||||
over a source on the first run. What the tests found was in the fixtures: a rival that read the owned
|
||||
store for the stored fallback of a filtered formula passed until a cell was stored where the `WHERE`
|
||||
fails, and a rival that forgot a group's own name as an item passed until a cell was stored under
|
||||
one.
|
||||
- **The two-second limit decided the shape of the tests** more than the design did: the corpus
|
||||
differential over a scanning source is 2.5s untrimmed, so it follows `corpus_eager`'s trimming, and
|
||||
each random test is its own binary.
|
||||
|
||||
## Scheduling: sub-circuits on a worker pool (user ruling 2026-10-11)
|
||||
|
||||
Multi-threading here is **scheduling**. Each sub-circuit stays synchronous and single-threaded
|
||||
(`zset.md`, "Threads"), the only thing that crosses threads is an edge (`zset.md`, "Edges"), and
|
||||
a small executor decides which thread runs which sub-circuit next. This is a ruling recorded here,
|
||||
not an ADR amendment. Native threads are assumed. `sched.rs` and `staged.rs` compile for wasm32
|
||||
(a `Mutex` and a `Condvar`), but `improvise-command` depends on this crate, so the module is in
|
||||
the browser bundle: `Pool`, the only thing that spawns a thread, and the blocking calls
|
||||
(`claim_blocking`, `wait_idle`, which `Condvar::wait` would make panic there) are
|
||||
`cfg(not(target_arch = "wasm32"))`. Nothing here runs on wasm.
|
||||
|
||||
1. **A unit is a sub-circuit with the edge ends around it** (`Unit`: `turn(budget)` and
|
||||
`neighbours()`). Placement is manual: which nodes sit in which unit is how the graph was built,
|
||||
and `staged::PLACEMENT` names it for the example. Two node-building functions (`aggregate_nodes`,
|
||||
`grid_nodes`) build the same nodes into one circuit (`single_circuit`, the reference) or into
|
||||
two.
|
||||
2. **Runnable means queued.** `Executor::kick(id)` queues a unit once. A unit is kicked by the
|
||||
host (`Staged::submit`), and by a neighbour whose turn moved a frame (`Turn::progress`, which a
|
||||
unit derives from its edges' `traffic()`). A kick reaches both directions, because a request
|
||||
flows back along an edge. Progress is a count of frames, so it ends: two neighbours cannot kick
|
||||
each other for ever.
|
||||
3. **The ownership token is the unit itself.** A claimed unit is taken out of its slot, so a second
|
||||
worker cannot reach it. A kick that arrives while it runs sets `again`, and the owner queues it
|
||||
when it hands the unit back (`finish`). No lock is held during a turn; one mutex guards the
|
||||
queue and the slots, and a turn takes microseconds beside it.
|
||||
4. **Fairness is the budget and the back of the queue.** A turn handles at most `budget` frames
|
||||
(`EdgeSource::poll(max)`), then says `more`, and `finish` queues it behind the units already
|
||||
waiting. A full edge is **not** `more`: the sender would spin through the queue while its
|
||||
consumer is not running (`a_full_edge_does_not_make_its_sender_spin`). The consumer that frees
|
||||
room reports progress and kicks its producer back, so a backlogged sender runs when it can
|
||||
send. For the same reason a middle unit (the aggregate) needs no `more` at its budget: its
|
||||
consumer's progress kicks it. The tail (the projection) has no consumer to kick it back, so it
|
||||
says `more` when it read a full budget (`a_unit_whose_kicks_coalesced_still_drains_its_edge`,
|
||||
because kicks coalesce into one queue entry). Both clauses were first written untested; a
|
||||
review injected their removal and everything passed, which is how the first was found to be a
|
||||
spin and the second redundant. The resumable `step(deadline)` predicate of `Rebuild` is for a sliced snapshot load,
|
||||
and this example has none.
|
||||
5. **The executor spawns nothing.** `claim` and `finish` are what a worker calls. `Pool` is N
|
||||
threads looping on them; the tests' scheduler calls them in an order a seed chooses, with up to N
|
||||
units held at once and the turn of a held unit already run, so a kick can land on a running
|
||||
unit. Both drive the same code.
|
||||
6. **Results do not depend on N or the order.** The deltas sum, and every operator in the graph is
|
||||
a function of its input's integral. So any schedule settles where the single circuit does
|
||||
(`placed_circuits_equal_one_circuit_for_any_workers_and_order`, 1 to 4 workers, random
|
||||
commands, random order, commands arriving while it runs; and a real pool of 1 to 4 threads).
|
||||
Per-tick outputs (`delay`, `differentiate`) would differ with coalescing, and the example has
|
||||
none.
|
||||
7. **What the example is.** `Staged`: the host's end of the first edge, an aggregate stage (sum of
|
||||
a cell's value per row and column, `aggregate`) and a grid projection (the cell each group draws,
|
||||
a total per row). The incremental backend (`Incremental`) keeps its groups in its own shapes
|
||||
and does not run on a `Circuit`, so the stage is the same computation built as zset nodes rather than a
|
||||
split of `Incremental` itself. Running `Incremental` on circuits is a later step.
|
||||
8. **A panic in a turn kills that unit and nothing else.** A turn runs outside the lock, so
|
||||
nothing is poisoned and, in the first version, the pool waited for ever on a unit that never
|
||||
came back. Now the worker catches the unwind and calls `Executor::abandon`: ownership is
|
||||
released, the unit is dead (never run or kicked again), the panic is recorded
|
||||
(`Executor::panics`, `UnitPanic { unit, message }`), and neighbours go on. `wait_idle` and
|
||||
`Pool::finish` return `Err(UnitPanic)` for the first one once the rest is quiet. A unit that
|
||||
cannot name its neighbours is buried the same way. A kick of a unit that does not exist panics
|
||||
before it takes the lock (`no such unit`), so a bad index cannot poison the executor; the
|
||||
`lock()` message now names the only thing that can poison it, a panic in the executor itself.
|
||||
9. **Workers are counted and a dropped `Pool` stops them.** `Executor::workers()` counts live
|
||||
worker threads, so a worker that leaves on a spurious wake is visible
|
||||
(`every_worker_stays_through_wakes_with_nothing_queued`); `Drop for Pool` shuts down and joins
|
||||
(`a_pool_that_is_dropped_waits_for_its_workers`), discarding what is queued. A dead unit is
|
||||
dropped inside `catch_unwind` (`bury`), because a drop that panics would otherwise kill the
|
||||
worker after the bookkeeping and hang or shrink the pool; `Pool::join` does not panic over a
|
||||
dead worker while the host is already unwinding. `claim_blocking`
|
||||
returns `None` as soon as `shutdown` is set. The real-thread tests run behind a watchdog
|
||||
(`tests/support/watchdog.rs`), so a lost `notify_all` fails by name and not by hanging.
|
||||
10. **Not done.** A work-stealing or priority scheme (one FIFO queue); a unit with a `Rebuild`
|
||||
job sliced by the deadline predicate; a timeout that calls `connection_lost` on the edges (the
|
||||
edges here never lose a frame); restarting a dead unit.
|
||||
@@ -1,235 +0,0 @@
|
||||
# Maintaining the knowledge base
|
||||
|
||||
`prolog-sessions/kb.pl` is executable knowledge, checked by CI. This is the
|
||||
argument for how it is written, and why the Prolog MCP stopped being the way.
|
||||
|
||||
## Why the MCP was retired
|
||||
|
||||
`loadProgram` **replaces a whole predicate**, so adding one clause means
|
||||
re-sending every existing clause of it through an agent's context window — and a
|
||||
short payload silently drops the rest. Three further blockers: `saveSession`
|
||||
re-adds the MCP's own `host_rpc` shims and reserializes the whole file, and
|
||||
`assertz` raises `permission_error(modify, static_procedure)` against a
|
||||
consulted predicate, so there is no incremental route through it at all.
|
||||
|
||||
## What replaced it
|
||||
|
||||
`scripts/kb`, a shell entry point over `scripts/kb-merge.pl`.
|
||||
|
||||
**Clauses are read as DATA, never consulted.** `read_term/3` in a loop yields a
|
||||
list of terms; merging is list manipulation; `portray_clause/1` writes them
|
||||
back. Nothing is ever asserted, so the static-procedure wall is not reached and
|
||||
the cost of adding a clause is the clause.
|
||||
|
||||
**Canonical form is `read_term` in, `portray_clause` out.** This was not
|
||||
designed, it was measured: all 495 facts in the existing file — including every
|
||||
escaped quote — round-trip byte-identically, and the form is a fixed point in
|
||||
one pass. The only change on first use is that rules move from the MCP's
|
||||
two-line layout to one line, which is why the reformat lands with this tool
|
||||
rather than before it: the MCP would re-expand them on its next save and the two
|
||||
would thrash.
|
||||
|
||||
**trealla comes from the dev shell.** Reaching it through `nix shell` per
|
||||
invocation cost more than the work: the tool's first real use timed out past two
|
||||
minutes on a write that takes 0.42 seconds with `tpl` on `PATH`.
|
||||
|
||||
## The guards, and why each exists
|
||||
|
||||
Every one of these was added because injecting its fault showed the tool
|
||||
accepting something it should refuse. None was found by reading.
|
||||
|
||||
**The round-trip check.** After writing, the file is read back and required to
|
||||
be the clauses intended. A serializer bug is the wipe-the-knowledge-base failure
|
||||
with a new owner, and this is the only check that sees it. Note it compares
|
||||
*variant* equality via `numbervars` over a copy — reading a term back yields
|
||||
fresh variables, so `==` compares variable identity and reports every rule as a
|
||||
difference.
|
||||
|
||||
**`kb_consistent` runs on the CANDIDATE.** The file is replaced only after the
|
||||
check passes, so a delta that breaks the knowledge base leaves it untouched
|
||||
rather than requiring a revert.
|
||||
|
||||
**Parse errors are caught explicitly.** An uncaught syntax error drops trealla
|
||||
into its REPL, which *hangs* with a terminal and *exits 0* without one — a
|
||||
refusal that reads as acceptance. `scripts/kb` also closes stdin on every call
|
||||
as a backstop.
|
||||
|
||||
**Every run reaches a `halt/1`.** `tpl` exits 0 unless a halt is *reached*:
|
||||
measured on 3.10.3, `-g "fail"` and `-g "throw(boom)"` both exit 0, and so do
|
||||
`fail,halt` and `throw(boom),halt`, because a goal that fails or throws never
|
||||
reaches the halt beside it. `halt(2)` alone exits 2, so halt works — the default
|
||||
is the hazard. The goal used is therefore
|
||||
|
||||
```
|
||||
(catch(main,E,(write(user_error,E),nl(user_error),halt(2))) -> halt(0) ; halt(1))
|
||||
```
|
||||
|
||||
which gives 0, 2 and 1 for success, throw and failure. Without it, `if ! merge
|
||||
...` reads a crashed merge as a clean one and replaces the knowledge base.
|
||||
|
||||
**Terms must be callable.** `42.` parses, is not a clause, and reached the
|
||||
knowledge base as a fact whose head is a number. `kb_consistent` has no opinion
|
||||
about stray terms, so nothing objected.
|
||||
|
||||
**A clause can be retired.** `kb rm` takes keys and refuses if any of them is
|
||||
absent, because a typo'd key would otherwise report success and leave the clause
|
||||
in place. Without it no correction could ever drop a fact — the first thing it
|
||||
was used for was retiring `kb_is_maintained_through_the_mcp`, whose *name* was
|
||||
the claim that went false.
|
||||
|
||||
**Duplicate keys are refused by default.** A key is the functor plus every
|
||||
argument but the last, so `doctrine(Name,_)` keys on the name and
|
||||
`decision(Id,Key,_)` on the pair. `replace` is the door a correction goes
|
||||
through, and it overwrites in place — a count that grew would be two clauses
|
||||
under one name.
|
||||
|
||||
**A write may not end up short of HEAD** (improvise-usus). `kb add` wrote four
|
||||
facts onto a working-tree `kb.pl` that had already lost 276 of its 801, and
|
||||
reported success: `kb_consistent` asks whether the facts are *consistent*, never
|
||||
whether they are all *there*, and a knowledge base missing a prefix of a
|
||||
predicate is perfectly consistent. `kb-merge` printed `529 before` beside a last
|
||||
known 805 and nothing compared them. The cause was never settled between a
|
||||
`kb replace` killed mid-write and git being moved under the session, so the
|
||||
guard had to catch both, and only a comparison against something *outside* the
|
||||
working tree can: a guard that compares the file against its own pre-write
|
||||
count passes a file that was truncated before the write began, which is exactly
|
||||
what happened. So every write runs `kb-merge.pl complete` on the candidate,
|
||||
before `kb_consistent`, against `git show HEAD:prolog-sessions/kb.pl`, and
|
||||
refuses unless every clause HEAD has is still present **by key**. It runs first
|
||||
so that the refusal names the loss rather than whatever consistency error the
|
||||
loss happens to cause.
|
||||
|
||||
It compares keys rather than totals, although the ticket asked for a count,
|
||||
because a total can be masked: uncommitted adds in one predicate offset a
|
||||
truncation in another, and the count comes out even. Keys cost the same (0.26s
|
||||
for a write, most of it trealla start-up) and the message still names both
|
||||
totals and the number missing.
|
||||
|
||||
**Keys are counted, not collected** (improvise-ih3k). The guard first compared
|
||||
the *set* of keys, and a set cannot see one of two clauses sharing a key.
|
||||
`work_done/3`, `decision/3` and `work_followup/3` key on two arguments and are
|
||||
not `functional/1`, so `kb_consistent` accepts two clauses under one key. `kb
|
||||
add` refuses to *write* such a pair, but a git merge is not a `kb add`: two
|
||||
branches that each record `work_done(T, bug_fix, …)` conflict at the end of the
|
||||
block, and keep-both is the natural resolution. Once HEAD holds a pair, losing
|
||||
either clause left the key present and the write was accepted, with both totals
|
||||
printed equal. So the guard counts clauses per key, and a key is short when the
|
||||
result holds fewer than HEAD does. The refusal names the HEAD clauses the result
|
||||
holds no copy of, which is the one that went.
|
||||
|
||||
Two alternatives were rejected. Comparing whole clauses would refuse every
|
||||
`replace`, whose purpose is to change the text under a key. Making the three
|
||||
predicates functional would refuse the pair itself at `kb_consistent`. That
|
||||
would force a merge to resolve it, and may be worth doing, but it is a change to
|
||||
the gate in `kb.pl`, not to this tool. `functional/1` also keys an arity-3
|
||||
predicate on its first argument, which is wrong for `decision/3`, so it would
|
||||
need a new two-argument form. And the tool must not lose a clause whatever the
|
||||
gate allows, so counting is needed either way.
|
||||
|
||||
A pair also had no way out through the tool. `rm` checked its result against
|
||||
`before - clauses in the delta`, which assumes one clause per key, so removing a
|
||||
paired key was refused as "a clause was dropped". The expected total is now
|
||||
`before - clauses the delta's keys match`, which is still counted separately
|
||||
from the fold that removes them. `replace` of a paired key is still refused by
|
||||
its total. That is the right answer, because it would silently collapse two
|
||||
clauses into one. The route for a pair is `kb rm` the key, then `kb add` the one
|
||||
clause to keep.
|
||||
|
||||
**How the count legitimately falls.** `add` never lowers it and `replace`
|
||||
overwrites in place, so neither can trip the guard. `kb rm` is the only
|
||||
intended removal, and it records the keys it removed in a **ledger** that later
|
||||
writes read: a key the result is short of is excused only if the ledger names
|
||||
it. The ledger lives in the git dir (`git rev-parse --git-path
|
||||
kb-removed.pl`), so it is per-worktree and can never be committed; with
|
||||
`KB_FILE` set it sits beside that file. Every write rewrites it, keeping only
|
||||
the keys the result is still short of, so an entry drops once the removal is
|
||||
committed and once the key is back to HEAD's count. That second case matters:
|
||||
without it, a re-added clause that was lost again would still be excused. There
|
||||
is deliberately no override flag.
|
||||
|
||||
The way out of a refusal is to restore `kb.pl` from HEAD and re-apply your
|
||||
deltas. Do **not** commit the short `kb.pl` to get past it. The refusal
|
||||
message used to offer that, and taken on a loss nobody intended it makes the
|
||||
loss permanent: the committed file becomes the floor this guard compares
|
||||
against, and nothing ever reports it again. A removal you intend goes through
|
||||
`kb rm`, onto a restored file.
|
||||
|
||||
**A ledger speaks only for the file its write published** (improvise-yea8).
|
||||
Pruning happens only when the tool writes. A commit, a `git checkout --
|
||||
prolog-sessions/kb.pl`, a `cp`, and a branch switch are not writes. So an
|
||||
entry from an `rm` that was later abandoned would excuse a later loss of that
|
||||
same clause by any path. That is easy to reach, because the recommended
|
||||
recovery is exactly a restore. Pruning against the kb read at the next write
|
||||
cannot close it, because that kb has already lost the clause. So the ledger's
|
||||
first line is a `% kb-stamp` comment naming the kb file the write published,
|
||||
and a later write reads the entries only if the kb it finds is still that file.
|
||||
|
||||
The stamp is the content hash **and** the file's inode and nanosecond mtime.
|
||||
Content alone is not enough, and the test that showed it was the ticket's own
|
||||
reproduction. `kb.pl` is kept in canonical form, so `grep -v` of one clause
|
||||
from HEAD produces output **byte-identical** to what `kb rm` of that clause
|
||||
writes. After `rm X`, a restore, and then the same loss by hand, a content-only
|
||||
stamp still matched. `rename(2)` carries the candidate's inode and mtime onto
|
||||
`kb.pl`. Anything else that rewrites the file, whether a `cp` onto it, a git
|
||||
checkout or an editor, gives a new mtime and usually a new inode. `stat -c` is
|
||||
tried before `stat -f` because GNU `stat -f` means *filesystem* status and
|
||||
would silently stamp the wrong thing. It is not keyed by HEAD commit, because a
|
||||
commit that does not touch `kb.pl` does not make the ledger wrong. And an
|
||||
uncommitted `rm` carried across a branch switch, with `kb.pl` untouched, is
|
||||
still exactly the kb it describes.
|
||||
|
||||
The cost is a false refusal after an intended `rm` followed by anything that
|
||||
rewrites the file without the tool. That refusal is loud, and the recovery is
|
||||
the usual one: restore and redo the `rm`.
|
||||
|
||||
**The write is a rename.** The candidate used to come from `mktemp` in
|
||||
`$TMPDIR`, and `mv` from there onto the repository crosses a device wherever
|
||||
`/tmp` is its own filesystem. That makes `mv` a copy and an unlink, so a kill
|
||||
partway through leaves a truncated `kb.pl`. The candidate is now a sibling of
|
||||
`kb.pl` (`kb.pl.cand.XXXXXX`), so publishing it is `rename(2)`. The ledger is
|
||||
renamed first. A kill between the two renames leaves a ledger stamped for a
|
||||
candidate that was never published. It therefore matches nothing, and the next
|
||||
write ignores it. That can refuse a write an earlier `rm` should have excused.
|
||||
It cannot excuse a loss, which was the hole before the stamp.
|
||||
|
||||
**Exit codes.** 0 done, 1 refused, 2 usage, 3 HEAD's `kb.pl` could not be read,
|
||||
127 no `tpl`. Only 1 is a verdict. 3 is separate because "could not compare"
|
||||
is not "compared and refused" (improvise-b4ks). A pre-existing wart survives:
|
||||
failing to acquire the lock exits 1 too.
|
||||
|
||||
## What no test here reaches
|
||||
|
||||
The completeness guard trusts HEAD. A loss that has been **committed** is
|
||||
invisible to it by construction. That was the right trade for this incident,
|
||||
where HEAD held all 801, but a bad commit of `kb.pl` becomes the new floor.
|
||||
|
||||
A `SIGKILL` leaves six files behind, because no trap runs: the candidate
|
||||
`kb.pl.cand.XXXXXX`, its `.base`, `.ledger`, `.removed` and `.stamped`
|
||||
siblings, and the `kb.pl.lock` directory. None of them is ever read again,
|
||||
and **git does not see them**. `.gitignore` ignores `prolog-sessions/*`, so
|
||||
neither `git status --untracked-files=all` nor `git add -A` lists them. That
|
||||
was measured, and an earlier version of this paragraph warned about `git add
|
||||
-A` staging them, which does not happen. The one that matters is the lock: the
|
||||
next write waits 30 seconds for it and then exits 1 with "remove it if stale".
|
||||
Delete it and the candidates by hand.
|
||||
|
||||
The ledger's stamp names a file by inode and nanosecond mtime. A rewrite that
|
||||
preserves both, such as a `touch -r`, a filesystem with coarse timestamps
|
||||
rewriting within one tick, or an inode reused at the same mtime, with the bytes
|
||||
also identical to what the tool published, is indistinguishable from the file
|
||||
the tool wrote. That takes deliberate effort or a filesystem this repository
|
||||
does not run on.
|
||||
|
||||
The key rule is looser than `functional/1` for `crate/3`, which keys on the
|
||||
first argument alone; `kb_consistent` is the backstop for that case rather than
|
||||
the merge.
|
||||
|
||||
A clause name cited *inside another clause's string* is unchecked — dropping or
|
||||
renaming a clause silently dangles every clause referring to it
|
||||
(`improvise-2f97`).
|
||||
|
||||
A shebang would let `kb-merge.pl` run directly and collapse the two files into
|
||||
one. It does not work on trealla 3.10.3: `#!` is a syntax error that aborts the
|
||||
consult, so `main/0` is never defined, and `tpl` wants `-- args` *after* the
|
||||
file, which a shebang cannot inject. Worth re-testing if trealla gains script
|
||||
support.
|
||||
@@ -1,332 +0,0 @@
|
||||
# `numtower`: the reasoning
|
||||
|
||||
`crates/numtower` is ADR 005 migration step 1 (improvise-59nw.1): the numeric tower,
|
||||
first and alone. Nothing in improvise depends on it yet, so it changes no behaviour.
|
||||
The decisions are the ADR's (`context/adr/005-exact-decimal-arithmetic.md`). This file
|
||||
records how the crate reads them, the places the ADR left a choice open or said two
|
||||
things, and the measurements. The evidence for each test is in
|
||||
`context/test-evidence/numtower.md`.
|
||||
|
||||
## The shape
|
||||
|
||||
| type | what it is | ADR |
|
||||
|---|---|---|
|
||||
| `Num` | the stored value: an exact `Decimal`, an exact `Ratio<i128>`, or an `f64` | clauses 1, 2 |
|
||||
| `NumRef` | a borrowed view of a `Num`'s arm, for exhaustive matching | — |
|
||||
| `Interim` | a value inside one evaluation; exact intermediates widen to a bigint rational | question 13 (b) |
|
||||
| `NumSum` | a retractable, order-free `SUM` over any `Num` | clause 8, question 16 |
|
||||
| `ExactSum` | ADR 009's f64 superaccumulator | question 15 (a) |
|
||||
| `parse_number`, `is_number`, `Typing` | clause 3's number language, gate and parse | clause 3, question 12 |
|
||||
| `Display` / `Num::from_canonical` | the canonical text: `12.50`, `10/3`, `#i0.1`, `inf`, `-inf`, `nan` | clauses 10, 11, 14 |
|
||||
| `Num::round_to_places` | display rounding, the mode a parameter | clause 12 "Format" |
|
||||
|
||||
**`Num` is a struct with private arms, not the ADR's `pub enum`.** The ADR's sketch
|
||||
(clause 2) shows `pub enum Num { Exact(Decimal), Fraction(Box<Ratio<i128>>),
|
||||
Inexact(f64) }`. A public enum would let any caller build values the tower never
|
||||
produces: `Fraction(1/2)` (which must be the decimal `0.5`), an unreduced fraction, or an
|
||||
exact `-0`. Each would break exact equality or the fingerprint of clause 9. So the arms
|
||||
are private, every constructor normalizes, and callers match on `NumRef`. That is
|
||||
`context/design-principles.md` §3, smart constructors. It is also what clause 14's table
|
||||
asks for, since improvise calls "`Num`'s accessors for exactness, mantissa, scale and
|
||||
fraction parts". The `Box` lives in one constructor, as `AGENTS.md` asks. `Num` is 24
|
||||
bytes, as the ADR measured, and `a_num_is_24_bytes` pins it.
|
||||
|
||||
**The normal form.** An exact zero has no sign. A fraction is reduced and has a positive
|
||||
denominator. A value whose reduced denominator is `2^p·5^q`, with `k = max(p, q) <= 28`
|
||||
and a mantissa that fits 96 bits, is always the `Decimal`, never a `Fraction`. So
|
||||
`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. Every `i128`
|
||||
is a valid numerator, `i128::MIN` included, because the arm is `Ratio<i128>`. That one
|
||||
value cannot be negated in `i128`, so `Operand::parts` keeps it out of the `i128` fast
|
||||
path and its arithmetic takes the bigint path (Copilot on PR #337).
|
||||
|
||||
## Arithmetic
|
||||
|
||||
### Why there are two types: `Num` and `Interim`
|
||||
|
||||
Question 13 (b) bounds only the **stored** result: "every intermediate inside one formula
|
||||
evaluation is unbounded up to the ceiling". The bounded `Num` cannot be such an
|
||||
intermediate, so `Interim` is the widened type. It is either a stored `Num` or an exact
|
||||
`BigQ` (a reduced `BigInt` fraction) that does not fit `i128`. `Interim::store` applies
|
||||
the stored bound. The operators on `&Num` are one-operation evaluations: each builds the
|
||||
`Interim`, applies the operation and stores the result. So `(a * a) / a` returns
|
||||
`a` exactly when computed through `Interim`, and is inexact when computed through `Num`
|
||||
(`an_intermediate_beyond_i128_comes_back_exact`). That difference is (b)'s whole point.
|
||||
Step 3's evaluators should evaluate through `Interim` and store once per cell.
|
||||
|
||||
**So `Interim` compares, and knows zero, without storing.** `IF(a*a/a = a, …)` and the
|
||||
division guard both run before the stored bound. If `=` had to store its operands first,
|
||||
its answer would depend on how the formula is spelled, which is what question 13 (b)
|
||||
rules out. `Interim::cmp_exact` is clause 6's exact order over every pairing: two stored
|
||||
values go through `Num::cmp_exact`, and anything wide compares as a bigint rational,
|
||||
against an f64 by its exact binary value. `Interim::is_zero` is the one zero predicate. A
|
||||
wide value is never zero, because zero always fits a stored arm. The comparison through
|
||||
storing is the rival (`an_interim_compares_exactly_without_storing_catches_storing_first`):
|
||||
`a` and `a ± 1/(3·10^30)` store to one f64 and compare equal. The review of `302594bf`
|
||||
asked for this or for a note naming it as a step-3 item; the crate does the first.
|
||||
|
||||
### Contagion, and how many roundings a demotion has
|
||||
|
||||
An inexact operand makes the result inexact. Each exact operand is converted by the
|
||||
correctly rounded conversion, and then the f64 operation runs. That is two roundings,
|
||||
which is clause 4's "computed in f64".
|
||||
|
||||
**An exact ∘ exact result that does not fit is rounded once.** It is the correctly rounded
|
||||
f64 of the exact value. Clause 4 still says a demoted sum or product is "rounded twice,
|
||||
once per operand and once in the operation". That text predates option (d). Option (d)'s
|
||||
"Operations" and question 13 (b) both say that a result failing the stored bound is "the
|
||||
correctly rounded f64 of the exact fraction", and under (b) the exact value is always in
|
||||
hand. The crate follows (d), because (d) is decided and later. Both texts cannot hold for
|
||||
one operation, so this is a reading, and it is flagged.
|
||||
|
||||
### Clause 4: the hint, the recompute, the stored scale
|
||||
|
||||
`Decimal + Decimal` and `Decimal × Decimal` first try `rust_decimal`'s checked operation.
|
||||
The result is trusted only when its scale equals the nominal one (`max(sa, sb)` for `+`
|
||||
and `−`, `sa + sb` for `×`). A lower scale is the **hint**, and so is a `None`. On the
|
||||
hint the exact integer result is recomputed in a 256-bit integer, and its trailing zeros
|
||||
are stripped. If the stripped mantissa fits 96 bits at a scale of at most 28, the result
|
||||
is a decimal at the **stored scale**, `min(nominal, 28)`, padded back with zeros while the
|
||||
mantissa fits. Otherwise the result goes to the fraction arm, then to a wide `Interim`.
|
||||
The ADR's own example holds: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28
|
||||
(`a_recomputed_result_keeps_the_stored_scale_clause_4_names`). A `None` counts as the
|
||||
hint, so that an overflow `rust_decimal` reports is never mistaken for an exact result.
|
||||
|
||||
### Division: the direct test, one implementation
|
||||
|
||||
Clause 5 is normative: write the quotient as a reduced `n / (2^p·5^q)`, then it is a
|
||||
decimal iff `k <= 28` and the padded mantissa fits. The crate does this the same way for
|
||||
every exact pairing, not only for decimals. It reduces the quotient (in `i128` when that
|
||||
cannot overflow, and in a bigint otherwise), then normalizes. `small_ratio` and
|
||||
`decimal_of` are the one normalization, so a decimal reached through any path has the
|
||||
same representation. Multiplying back is never used.
|
||||
|
||||
**A quotient carries its minimal scale** (`10.00 / 2` is `5`, `1 / 4` is `0.25`). The ADR
|
||||
does not say what scale a quotient has. A quotient's scale is not a function of its
|
||||
operands' scales (clause 5), and a result that passed through the fraction arm carries
|
||||
its minimal scale (option (d)), so the minimal scale is the consistent choice. It is a
|
||||
reading, and it is flagged.
|
||||
|
||||
**The ADR's argument against multiplying back is weaker than it says.** Clause 5 rejects
|
||||
it as incomplete, on `1e-28 / 0.5`: `q × 0.5` "needs scale 29, so the check flags it".
|
||||
That holds under revision 6's rule, which demotes on the hint
|
||||
(`division_decides_fit_directly_catches_multiplying_back` keeps that rival). But under
|
||||
revision 7's recompute the product `1e-28` is found exact after its zeros are stripped.
|
||||
So multiplying back by `Num`'s own `*` accepts the ADR's counterexample. In 20 runs of the
|
||||
division property (5,120 random cases plus the three seeds), it agreed with the direct
|
||||
test every time. That was an experiment and is not committed. The direct test stays the
|
||||
rule, because the ADR decided it and because it needs no second division.
|
||||
|
||||
### `POW`
|
||||
|
||||
- **The exponent.** It is exact when it is an exact **integer value**, so `2.0` counts.
|
||||
- **The size, decided before any multiplication.** The result's numerator has at least
|
||||
`|n|·(bits(num) − 1)` bits, and likewise its denominator. If either lower bound passes
|
||||
the ceiling, the result is `powf`, inexact, with nothing allocated. So
|
||||
`POW(1/3, 10^9)` costs nothing. Inside the bound the result is computed exactly, and it
|
||||
can still exceed the ceiling (`3^6000` is 9,510 bits). It then demotes, which is the
|
||||
ceiling's stated exception.
|
||||
- **The scale.** A positive power of a decimal takes clause 4's stored scale with nominal
|
||||
`s·n`, the scale repeated multiplication would give, so `1.50^2` is `2.2500`. A
|
||||
negative power is a division and has its minimal scale, **unit bases included**:
|
||||
`1.0^-2` is `1` and `(-1.0)^-3` is `-1`, as `2.0^-2` is `0.25`. Until the review of
|
||||
`302594bf` the unit-base arm padded every power to `s·|n|`, so `1.0^-2` read `1.00`
|
||||
(Copilot on PR #337). The power property now checks the scale of every decimal result
|
||||
against this rule, with unit bases among its seeds.
|
||||
- **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`,
|
||||
including one too large to compute.
|
||||
- **A demoted power takes its sign from the exact exponent.** `powf` gets the base's
|
||||
magnitude, and the sign is set from `base < 0 && n odd`. An exponent above 2^53 rounds
|
||||
to an even f64, so handing `powf` the signed base made `(-2)^(2^53 + 1)` read `+inf`
|
||||
(Copilot on PR #337).
|
||||
|
||||
### The ceiling
|
||||
|
||||
`CEILING_BITS = 8,192`. It applies to the bit length of an intermediate's numerator *or*
|
||||
denominator. The ADR states the ceiling as a width, not as which of the two it measures.
|
||||
Past it, the intermediate becomes the correctly rounded f64 of its exact value.
|
||||
|
||||
## Comparison
|
||||
|
||||
- **Exact with exact.** Cross-multiplication in 256 bits. A decimal is `m / 10^s` with
|
||||
`10^28 < 2^94`, so every product fits.
|
||||
- **Exact with inexact.** The ADR's method: convert the exact operand, and the rounded
|
||||
value decides unless it lands on the f64 itself. A tie is resolved against the f64's
|
||||
exact binary value, in a bigint.
|
||||
- **That fast path is sound only because the conversion is correctly rounded.** Injecting
|
||||
`rust_decimal`'s `to_f64` failed `exact_mixed_comparison_is_transitive` as well as the
|
||||
conversion test. A future integer conversion (clause 4 allows one) must keep that.
|
||||
|
||||
`MIN` and `MAX` skip NaN. Equal values break their tie as follows. Only the first rule is
|
||||
the ADR's (question 11); the other two are this crate's, and they make the result
|
||||
independent of argument order:
|
||||
- the exact value is preferred to the inexact one (question 11);
|
||||
- of two equal decimals, the larger scale is kept, for the same reason: it keeps more of
|
||||
what was typed;
|
||||
- of `0.0` and `-0.0`, `MIN` takes `-0.0` and `MAX` takes `0.0`, as IEEE 754's
|
||||
`minimum` and `maximum` do.
|
||||
|
||||
## The number language
|
||||
|
||||
**The gate is literally `f64::from_str` reading a finite value.** The parse is the
|
||||
crate's own lexer, and it never goes through an f64 on an exact path. The property test
|
||||
checks that the two agree, and that the exact value read rounds to the f64
|
||||
`f64::from_str` reads.
|
||||
|
||||
- **A lexeme's scale.** It is the scale the lexeme implies (`1.50` → 2, `1.5e3` → 0). When
|
||||
the implied scale is above 28 but the value fits a decimal (`0.1` written with 29
|
||||
places), clause 4's stored-scale rule applies: strip the zeros, then pad toward 28.
|
||||
The ADR only says "with the scale the lexeme implies", which cannot exceed 28.
|
||||
- **Zero.** It is exact, unsigned, at the implied scale clamped to 0–28, so
|
||||
`0e99999999999999999999` is `0`.
|
||||
- **An exponent too long to read** saturates, because no exact arm can reach it. A
|
||||
significand of more than 180 digits, or a leading digit outside 10^-130 to 10^40, goes
|
||||
straight to f64. Those bounds are wider than the fraction arm can reach (a reduced
|
||||
denominator `10^k / 2^a` is at least `2^k`, so `k < 127`).
|
||||
- **The forced `n/d` arm.** Its value is a decimal, then a fraction, then **the nearest
|
||||
f64**. The ADR says `n/d` "is a fraction or, once normalized, a decimal", and does not
|
||||
say what a forced `n/d` too large for `i128` becomes. The crate applies clause 3's order
|
||||
to it, as it does to every other spelled number. This is a reading, and it is flagged.
|
||||
**Except that the nearest f64 must be finite.** A forced `n/d` whose value rounds to
|
||||
infinity (`n` of 400 digits, `/1`) is text, exactly as the gate makes `1e999` text.
|
||||
Clause 3: "a finite lexeme must never read as infinity". One that underflows is the
|
||||
inexact zero, as `1e-999` is. `is_number` and `parse_number` share the one helper,
|
||||
`forced_fraction`, so the gate and the parse cannot disagree on it.
|
||||
|
||||
**The canonical text is `Display`.** An inexact finite value is `#i` followed by Rust's
|
||||
`{:?}` spelling, which is the shortest that reads back (`#i1e300`, `#i0.1`).
|
||||
`from_canonical` reads all six spellings back, and reads an exact spelling only as an
|
||||
exact value. Clause 14's list puts "`#i…`" in the crate, while its table says "the `#i`
|
||||
prefix and the grammar are improvise's". Those two contradict each other. The crate
|
||||
holds the prefix, because the wire and the file both need the one spelling, and step 7
|
||||
may move the prefix out if the grammar wants it.
|
||||
|
||||
## Rounding to places
|
||||
|
||||
Every arm goes through one exact path: the value as a bigint fraction,
|
||||
`n·10^places / d`, then rounded by the caller's mode (`HalfAwayFromZero` or `HalfEven`).
|
||||
An inexact value is first read exactly from its shortest round-trip spelling, which is
|
||||
improvise-felo.9's fix, so a typed-then-computed `1.005` shows `1.01`. A result that
|
||||
rounds to zero has no sign (`-0.001` at two places is `0.00`). The ADR does not decide
|
||||
that, and today's f64 path would print `-0.00`.
|
||||
|
||||
## Sums
|
||||
|
||||
`NumSum` keeps five things, and every one is an abelian group, so the state is a function
|
||||
of the multiset of terms and not of their order.
|
||||
|
||||
| part | holds | width |
|
||||
|---|---|---|
|
||||
| the decimal partition | every decimal, and every fraction whose denominator is `2^p·5^q` with `k <= 28`, as an integer at scale 28 | 320 bits |
|
||||
| the groups | per other reduced denominator, the summed numerator | 256 bits each, in a `BTreeMap` |
|
||||
| `group_floats` | an `ExactSum` of each group's correctly rounded value | ADR 009's |
|
||||
| `inexact` | an `ExactSum` of the f64 terms | ADR 009's |
|
||||
| `inexact_terms` | the net weight per f64 bit pattern, entries at zero dropped | a `BTreeMap` |
|
||||
|
||||
**The read.** It is exact when no inexact term is present and at most K = 8 groups
|
||||
remain. The groups and the decimal partition are then combined **over the lcm of their
|
||||
denominators, and reduced once**, and the stored bound applies to the result. Past K the
|
||||
read is the f64 accumulators: `group_floats`, the decimal partition converted once, and
|
||||
the inexact terms. That reading rounds each group, then the total. With inexact terms and
|
||||
at most K groups, the exact total is converted once and added to the inexact terms. A
|
||||
group whose numerator returns to zero is dropped, so "how many groups" is a function of
|
||||
state.
|
||||
|
||||
**"Is an inexact term present" is asked per value, never of the net weight.** The first
|
||||
version kept one net count. Inserting `1.0` and retracting `2.0` leaves the count at zero
|
||||
while the accumulator holds −1.0, so the read took the exact branch and dropped it
|
||||
(ChatGPT Codex, P1 on PR #337). Presence is now a map from f64 bit pattern to net weight,
|
||||
dropped at zero, the same rule the groups follow. It costs O(distinct inexact values),
|
||||
the same order as the groups. The K cutoff overrides representability: nine denominators whose exact total fits
|
||||
still read inexact (`more_than_k_denominators_read_inexact_even_when_the_total_fits`).
|
||||
That is question 13's second exception, and the ADR's flagged reading.
|
||||
|
||||
**`ExactSum` is ADR 009's prototype**, from `crates/improvise-core/examples/retraction_drift.rs`
|
||||
on PR #309. It has three changes:
|
||||
- it splits a weight into base-2^30 digits, as ADR 009 says the `numtower` version must,
|
||||
instead of looping per chunk;
|
||||
- it holds 70 limbs, not 67, because a digit's shift adds up to 60 bits;
|
||||
- it reads through the crate's one rational-to-f64 rounding.
|
||||
|
||||
It counts NaN and the infinities, so they can be retracted. A count that is nonzero in
|
||||
either direction is treated as present.
|
||||
|
||||
**`ExactSum`'s `==` compares sums, not representations.** A derived equality compared the
|
||||
raw limbs and the `pending` carry counter, so `0.5 + 0.5 != 1.0`, an insert then retract
|
||||
was not equal to an empty sum, and the same weights added in another order compared
|
||||
unequal (Copilot on PR #337). For a type whose whole point is order-freedom, and which
|
||||
ADR 009's group newtype will wrap, that equality is a trap. It now compares the
|
||||
propagated limbs, which are canonical (every limb but the top in `[0, 2^32)`), plus the
|
||||
three special-value counts, and ignores `pending`. `NumSum` derives no equality: compare
|
||||
its `read()`.
|
||||
|
||||
**`improvise-core` and `improvise-incremental` sum through `ExactSum`** (improvise-hrjz): the
|
||||
sign rule is the one already in it, and the reasoning for the two callers is
|
||||
`context/code/improvise-incremental.md`, step 6.
|
||||
|
||||
**For ADR 009's seam crate.** `NumSum` and `ExactSum` both have the shape a Z-set group
|
||||
needs: `add(&Num, i64)` / `add(f64, i64)`, `insert`, `retract`, `read`/`value`, `Clone`.
|
||||
`ExactSum` also has `absorb`, to merge two accumulators. Neither implements a group trait
|
||||
yet, because `zset` does not exist yet. The newtype of question 15 is where that goes.
|
||||
|
||||
## The wide integers
|
||||
|
||||
`Fixed<N>` is a two's-complement integer of `N` 64-bit limbs that never allocates. It is
|
||||
used for the clause 4 recompute (256 bits), exact comparison (256 bits), the decimal
|
||||
partition (320 bits) and the group numerators (256 bits). Clause 14 also names "256-bit
|
||||
helpers for the division reduction". Division reduces in `i128` when the products cannot
|
||||
overflow, and otherwise in a bigint. It does not have a hand-rolled 256-bit gcd. That
|
||||
path is the slow path, and it is measured below.
|
||||
|
||||
## Dependencies
|
||||
|
||||
The normal edges are `rust_decimal` 1.43.0 (std only), `num-rational` 0.4.2 (no default
|
||||
features), `num-bigint` 0.4.8, `num-integer` 0.1.47 and `num-traits` 0.2.19. The last two
|
||||
are direct dependencies, used for `gcd`/`lcm` and the numeric traits. They were already
|
||||
in the tree through `num-rational` and `num-bigint`, so they add no crate. The ADR names
|
||||
only the first three. There is no serde, and `cargo tree --edges normal` shows no
|
||||
`borsh`. `BigRational` is a **dev**-dependency feature only, for the oracle, so the
|
||||
oracle and the implementation do not share a rational type.
|
||||
|
||||
## Measured
|
||||
|
||||
On an Apple M4 Max (64 GB), rustc 1.98.1, `cargo run --release -p numtower --example
|
||||
bench`. Each range is two runs.
|
||||
|
||||
| what | figure |
|
||||
|---|---|
|
||||
| SUM, 18k and 90k typed two-place values, ns per element | f64 0.53; `rust_decimal` fold 2.7–3.0; `Num` `+` fold 7.8–8.2; `NumSum` insert and read 7.9–8.1 |
|
||||
| per operation over 18,000 typed pairs, ns | `Num` add 3.1–3.3, mul 3.0–3.2, div 146–153; `rust_decimal` checked add 1.4–1.5, mul 1.2, div 24.7–25.3 |
|
||||
| `(Rev − Cost) / Rev`, ns per record | 151–153 |
|
||||
| `to_f64` of a decimal (the text path), ns | 36.7–37.1 |
|
||||
| `round_to_places(2)`, ns | 370–382 |
|
||||
| `NumSum` read, K distinct 126-bit denominators, numerators of about 190 bits | K = 2: 5.2–5.4 µs; K = 8: 41.4–42.0 µs |
|
||||
| `ExactSum`, 1M values | 1.78–1.81 ns per insert, 0.29 µs per read |
|
||||
|
||||
Here is how these sit against the ADR:
|
||||
- **The grouped read.** At K = 8 it is inside the ADR's 52–127 µs. The ADR measured
|
||||
250-bit numerators, and these are about 190 bits, because `NumSum` can only reach a
|
||||
group's numerator through its public `add`.
|
||||
- **`ExactSum`.** It agrees with ADR 009's 2.0 ns per update and 0.58 µs per read.
|
||||
- **Division.** At about 150 ns it is above the ADR's *estimate* of 60–110 ns. Most
|
||||
quotients of typed values do not terminate, so each one pays two `i128` gcds and the
|
||||
`Box` of a `Fraction`.
|
||||
- **The rest.** The ADR's whole-recompute measurement at 18k and 90k cells waits for
|
||||
step 3, as the ADR says.
|
||||
|
||||
**The wasm32 cost** comes from a probe outside the repository, built exactly as the
|
||||
ADR's appendix builds one: a `cdylib`, `opt-level = "s"`, LTO, `wasm-opt -Oz
|
||||
--enable-bulk-memory --enable-nontrapping-float-to-int`, `gzip -9`. The baseline already
|
||||
parses and prints an f64.
|
||||
|
||||
| probe | after `wasm-opt -Oz` | gzipped |
|
||||
|---|---|---|
|
||||
| f64 only | 48,331 B | 29,162 B |
|
||||
| the ADR's five operations (parse, `+`, `−`, `/`, print) through `numtower` | +89,375 B | **+36,714 B** |
|
||||
| plus `NumSum`, `cmp_exact` and `round_to_places` | +119,277 B | **+48,221 B** |
|
||||
|
||||
The narrow figure confirms the ADR's estimate of "up to about +36 KB together". The full
|
||||
surface is about +47 KB gzipped. That is what the bundle pays once step 4 sums through
|
||||
the crate and step 6 rounds through it. It is more than the +15.4 KB the user answered
|
||||
question 8 against, and the ADR says question 8 "reopens" if the figure changes the
|
||||
answer. Neither figure is the real bundle, which shares allocator and formatting code
|
||||
with everything else.
|
||||
@@ -1,130 +0,0 @@
|
||||
# Reading a PR's reviews
|
||||
|
||||
Twenty PRs merged in one campaign with their reviews unread. CI was green and
|
||||
an adversarial agent had been dispatched, which felt like the whole gate. Two
|
||||
ChatGPT Codex reviews were found afterwards on merged PRs; both were correct
|
||||
and one was a live P2. Three more were found on the next three PRs — one per
|
||||
PR, every one a P1.
|
||||
|
||||
`scripts/pr-reviews <n>` prints them. `--wait <sec>` blocks until one lands
|
||||
against the current head.
|
||||
|
||||
## Three surfaces, because a reviewer writes on whichever one is nearest
|
||||
|
||||
GitHub has three places to put a review, and they are three different API
|
||||
endpoints:
|
||||
|
||||
| printed as | endpoint | what it is |
|
||||
|---|---|---|
|
||||
| `[REVIEW]` | `pulls/{n}/reviews` | the review's own body |
|
||||
| `[INLINE]` | `pulls/{n}/comments` | a comment on a line of the diff |
|
||||
| `[COMMENT]` | `issues/{n}/comments` | a comment on the PR thread |
|
||||
|
||||
**`pr-reviews` read only the first for as long as it existed** (improvise-xgra),
|
||||
and the surface it could not see is the one a human is most likely to use. On
|
||||
PR #122 the user reviewed with the body *"Generally good, added comments
|
||||
inline."* and put five substantive rulings underneath it as inline comments.
|
||||
`pr-reviews` printed the one-line body faithfully and nothing else, and an
|
||||
agent reported the PR as awaiting review while the answers sat on it. The user
|
||||
had to say so.
|
||||
|
||||
`scripts/pr-watch` does not cover the gap: that one watches **checks**. Nothing
|
||||
in `scripts/` surfaced the surface a human writes on.
|
||||
|
||||
The three are **interleaved in time order**, not printed in three blocks,
|
||||
because a review body and its inline comments are one act of reading and a
|
||||
block-per-endpoint layout hides which body the rulings belong under.
|
||||
|
||||
## What satisfies `--wait`
|
||||
|
||||
An **inline comment at the current head does**, and that is the whole point of
|
||||
the fix rather than an incidental widening. `--wait`'s question is *has anyone
|
||||
read this tree yet*, and a `pulls/{n}/comments` entry carries a `commit_id`: it
|
||||
is a review comment, written by a reviewer, about a specific commit. The case
|
||||
that failed on #122 is exactly one where the substance was inline — treating it
|
||||
as not-a-review would leave `--wait` blind to the reviews it most needs to
|
||||
catch. Note that a single "Add single comment" on GitHub creates a review whose
|
||||
**body is empty**, so under the old rule that reviewer's exit code was right
|
||||
and their words were invisible; the surfaces and the wait have to move together.
|
||||
|
||||
An **issue-level comment does not**, because it carries no commit. It makes no
|
||||
claim about which tree was read, and "I am rebasing this shortly" satisfying a
|
||||
wait for a review would be the widening that looks like the fix.
|
||||
|
||||
A comment on an **earlier** head does not satisfy the wait either — same rule
|
||||
the review bodies already had, and the same reason: it may be fixed or still
|
||||
live, and only reading it tells you which. An outdated comment has `line: null`
|
||||
and only `original_line` left, which is what is printed.
|
||||
|
||||
**A `commit_id` is per-comment, not per-review, and they differ inside one
|
||||
submission.** Measured on PR #122: one review submitted between 19:48 and 19:53
|
||||
carries five inline comments, of which three name `246e8f4a` and two name
|
||||
`6a4c7dcd`, and the review body names `246e8f4a` — GitHub pins each comment to
|
||||
the file version it was written against, so a comment on an outdated diff
|
||||
position keeps the older commit while its siblings move on. Two consequences.
|
||||
The per-entry `<- an EARLIER head` flag is honest but can make a *current*
|
||||
review look partly stale, so read the timestamps together before discounting
|
||||
one. And a head comparison against the review **body** is not even a proxy for
|
||||
its own comments: on that PR the body read as an earlier head while two of its
|
||||
rulings sat on the current one.
|
||||
|
||||
## `$?` must be this script's own
|
||||
|
||||
The usage text used to warn against piping into `head`/`tail`. That is narrower
|
||||
than the real rule: **nothing may follow the script in the same `;` list
|
||||
either**, because a trailing `echo "exit=$?"` makes the status the echo's. An
|
||||
agent nearly reported a review that did not exist that way. Redirect to a file
|
||||
and read the file in a later command.
|
||||
|
||||
## Two things the shell and `gh` do that this script got wrong first
|
||||
|
||||
Both were found by ChatGPT Codex on PR #133, and both are invisible on macOS.
|
||||
|
||||
**A multi-assignment `local` is not left-to-right on bash 5.** Written
|
||||
`local n="$1" name="$2" f="$DIR/$n.$name.json"`, bash creates every name
|
||||
*before* running the assignments, so `$n` is unbound in the third one and
|
||||
`set -u` kills the shell. It works on macOS's bash 3.2, so the whole suite
|
||||
passed locally and 17 of 19 cases failed in CI. **The signature to recognise is
|
||||
`exit 2` with no output at all**: the dead command substitution handed
|
||||
`jq --argjson` an empty string, and jq answers an invalid `--argjson` with a
|
||||
usage error, which is 2. So a shell bug at the top of the script arrives
|
||||
wearing jq's exit code. Give each `local` its own statement.
|
||||
|
||||
**`gh api --paginate` emits one JSON array per page**, not one array. A comment
|
||||
thread longer than the page size therefore arrives as several concatenated
|
||||
documents, and `--argjson` rejects it with the same exit 2. `jq -s 'add // []'`
|
||||
flattens the pages and is a no-op on a single page. The fixture path runs
|
||||
through the **same** helper, which is the only reason a fixture can test it at
|
||||
all: a fixture that could hold just one array would be exercising a shape the
|
||||
live fetch never produces. This was unreachable on every PR in this repo so far
|
||||
— #122 has five inline comments — which is exactly why it needed a test rather
|
||||
than a look.
|
||||
|
||||
## Why a review is easy to miss here
|
||||
|
||||
It arrives as a **review**, not a comment, so reading the comment thread does
|
||||
not show it.
|
||||
|
||||
It posts under the repository **owner's** account — the same account this
|
||||
repo's own agents post under — so **authorship does not distinguish it**. The
|
||||
tell is placement: a Codex review *ends* with
|
||||
`_Automated review by ChatGPT Codex (…)_`, where an agent's comment *opens*
|
||||
with `_Posted by Claude (…)_`.
|
||||
|
||||
It may target an **earlier head**. `pr-reviews` says which, because that
|
||||
changes what you do: a finding against an old commit may already be fixed or
|
||||
may still be live, and only checking tells you which. Both cases have occurred
|
||||
— one fixed by a later round, one live at the head after three more commits.
|
||||
|
||||
## Absence is not approval
|
||||
|
||||
Reviews arrive as the owner's quota allows. Observed latency is ten to fifteen
|
||||
minutes after a push. `--wait` exists so that a merge four minutes after a
|
||||
push does not race the review that was about to arrive; when it times out,
|
||||
that is a decision to merge unreviewed rather than evidence of a clean bill.
|
||||
|
||||
## Answer them on the thread
|
||||
|
||||
Including after the PR has merged. That is where the next reader looks, and a
|
||||
finding that was declined or already-fixed is worth as much as one that was
|
||||
taken — the reasoning is the artefact, not the verdict.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,134 +0,0 @@
|
||||
# `CellSource`: where cell values are read from
|
||||
|
||||
`crates/improvise-core/src/model/source.rs`. ADR 003 asked for a source trait in core with the
|
||||
in-RAM `DataStore` as one implementation (improvise-qddo). ADR 009 step 7 (improvise-rnak) asks for
|
||||
the two live sources that need it. This file is the reasoning; the evidence for each test is in
|
||||
`context/test-evidence/core.md`. The ADRs are not amended: what follows are the readings taken where
|
||||
they leave a choice.
|
||||
|
||||
## What is in the trait, and why
|
||||
|
||||
| method | what the evaluator asks |
|
||||
|---|---|
|
||||
| `value_at(key)` | the value stored at exactly `key` (an exact read, the filter-mismatch fallback) |
|
||||
| `summarize(select)` | the summary of the cells a `Select` names: the aggregate, not the rows |
|
||||
| `keys(partial)` | the keys of the cells containing `partial`: stems, and the records behind an aggregate |
|
||||
| `cells(partial)` | the same with values: records mode and a drill |
|
||||
| `count(select, excluding)` | how many cells a select names, leaving out those holding a hidden pair: occupancy |
|
||||
| `distinct_without(cats)` | the cell keys with `cats` removed, each once, first seen first: the stems formulas are evaluated at |
|
||||
|
||||
1. **Aggregation is the interface.** ADR 003 clause 1: `GridLayout` asks for a value, not for the
|
||||
rows behind it, and the aggregates are SQL's. So the one read the evaluator makes in bulk,
|
||||
`summarize`, returns a `Summary` and never a list of values. `keys` and `cells` exist because
|
||||
some reads are of records, not of a fold (the stems a formula is evaluated at; records mode), and
|
||||
a SQL source answers them with a `SELECT DISTINCT` or a plain select.
|
||||
2. **Writes are not in the trait.** `DataStore::set` and `remove` are inherent methods. A source that
|
||||
cannot be written to has no method to fail in, which is ADR 003's "unrepresentable rather than
|
||||
enforced". The model's own writes (`set_cell`, `clear_cell`) go to the owned store.
|
||||
3. **The trait has no `Result`.** A source that can fail (a connection) decides at its own boundary
|
||||
what a failed read is, and `improvise-io` is where that decision belongs. AGENTS.md keeps
|
||||
`Option` and `Result` at the edges; `Option` is the one the trait uses, for "no such cell".
|
||||
4. **`Select` is `spanned_values`' arguments.** A fold names a partial key, the covered members of
|
||||
each group or omitted category it spans, and whether only an exact-width cell counts. The
|
||||
covered members are decided in RAM from the categories and the view's hidden items, so a source
|
||||
is handed explicit item lists and never a predicate. A hidden item therefore reaches a source
|
||||
only as the absence of an item from a list.
|
||||
|
||||
## `Summary` and `Numbers`
|
||||
|
||||
A `Summary` is what a source reports about the values of a read: a `Numbers` (count, an exact sum,
|
||||
the least and greatest non-NaN, in `total_cmp` order so `-0.0` is below `+0.0`) and whether the rest
|
||||
(text, errors) agree on one value. It reduces exactly as `aggregate_cell_values` did: numbers by the
|
||||
function, anything else by unanimity, else `Error("mixed")`, none at all absent. Three properties
|
||||
carry the design:
|
||||
|
||||
- **It merges.** A read of a collapsed group, a total, or several contexts of a formula aggregate is
|
||||
the merge of the summaries of its parts, equal to the summary of the union. That is what lets an
|
||||
engine answer a pooled read without sending rows, and what lets `eval_agg` pool `Numbers` instead
|
||||
of concatenating `Vec<f64>`. The sum is `ExactSum`, so a merge is the same bits in any order.
|
||||
- **It is order-free.** Unanimity needs the set of non-number values, not their order, so a source
|
||||
that has no insertion order loses nothing by it.
|
||||
- **The reference is kept.** `tests::reference` is the old reduction, copied into the test module;
|
||||
`a_summary_reduces_as_the_values_did` compares every sequence of an eight-value pool, up to three
|
||||
long, with `Summary` over all five functions. `characterization_survives_the_move`: the existing
|
||||
evaluator tests (423 in the crate) are unchanged and pass.
|
||||
|
||||
What a real engine must supply is stated here because the trait cannot enforce it: a `Numbers` whose
|
||||
sum is exact. SQLite's `SUM` is a float sum, so an engine that cannot return an exact accumulator
|
||||
either returns the numbers (and the adapter sums them) or accepts a last-bit difference, which
|
||||
`context/code/improvise-incremental.md` ("Step 6") calls a user-visible change.
|
||||
|
||||
## `Model::source()` and `Model::over(source)`
|
||||
|
||||
`Model::source()` returns the `&dyn CellSource` that the evaluator, the stems of `recompute_formulas`,
|
||||
the member records of an aggregate over an expression and the records of `GridLayout` read through.
|
||||
`Model::over(source)` points it at a `SharedSource` (an `Arc<dyn CellSource + Send + Sync>`) instead of
|
||||
the owned `DataStore`. That is the whole evaluator hook (improvise-rnak, bw4p.14's prerequisite):
|
||||
there is still one evaluator, `Model`'s, and it computes every formula value, over a source whose
|
||||
cells it never owns.
|
||||
|
||||
1. **A model over a source is read-only, in every build.** `set_cell`, `clear_cell`, `remove_category` and
|
||||
`remove_item` `assert!` that no source is set (a `debug_assert` first shipped and was lost in release:
|
||||
the write went to the owned store, which nothing reads). A caller who holds a source writes to the
|
||||
source (ADR 003: the trait has no writes). `Model::over` drops the model's own cells and the formula cells computed from them, so no stale copy
|
||||
is left for `get_cell`, persistence or the harness to read. `data` is `pub`, so a caller can still write
|
||||
it directly; it is not read, and nothing prevents that. Structural edits (a category or item removed)
|
||||
are refused too, because the source's cells would still carry them.
|
||||
2. **A shared source changes behind its readers by interior mutability of its own.** The model holds
|
||||
an `Arc`, so a database connection or a test double behind a lock is the same object the
|
||||
backend feeds from.
|
||||
3. **What still reads the owned store**: `get_cell` (a production caller: `improvise-command/src/ui/effect.rs`,
|
||||
the `old` value of a rename), `Model::data` (`pub`), persistence, the import wizard and the effects
|
||||
in `improvise-command`. They are the facade of a document that owns its cells; a model over a source
|
||||
has none of those to offer, and routing them would change `get_cell`'s `&CellValue` return for 129
|
||||
call sites. `Eager::occupancy`, `Shapes::of` and `Stems::build` in `improvise-incremental` are
|
||||
routed by the PRs that need them.
|
||||
4. **`Debug` shows only whether a source is set.** A source is not necessarily printable, and two
|
||||
models over different sources are not told apart by it.
|
||||
|
||||
## `TableSource`
|
||||
|
||||
A scanned table of rows, behind `cfg(any(test, feature = "test-support"))`. It is the second
|
||||
implementation, written as a query engine with no index would answer, sharing only `Summary` with
|
||||
`DataStore`. `the_store_and_a_scanned_table_answer_every_select_alike` asks both every select over a
|
||||
fixture (every subset of every stored key as a base, with no span, one span of each of six item
|
||||
lists, and two), at both widths and for all five functions: over 5,000 triples.
|
||||
|
||||
## `ChangeSource`
|
||||
|
||||
`CellSource` plus `drain_changes(&self) -> Vec<CellChange>`: a source that reports what changed in it.
|
||||
The contract that matters is that a source's cells are those as of the last drain, so a snapshot read
|
||||
now plus the changes drained later counts no change twice or not at all. Why, and how `Incremental`
|
||||
uses it: `context/code/improvise-incremental.md`, step 7. `DataStore` is not a `ChangeSource`: its log
|
||||
is `&mut` and is drained by the owner in the step that made the edit. A `SharedFeed` is the `Arc` a
|
||||
backend and a model share.
|
||||
|
||||
## Pushdown needs two more queries
|
||||
|
||||
`count` and `distinct_without` joined the trait for `Pushdown` (improvise-rnak). Both are what a query
|
||||
engine answers without sending rows (`COUNT(*) ... WHERE`, `SELECT DISTINCT`), and each replaces a
|
||||
read that would have transferred every key: `Eager::occupancy` lists all keys, and
|
||||
`recompute_formulas` walked every cell to find its stems. `recompute_formulas` now asks
|
||||
`distinct_without(target and None categories)` once per formula, and the order it returns is
|
||||
the order the cells were first seen in, so the stems are inserted in the order they always were.
|
||||
That order is not a contract a source can be held to: the passes of the fixed point visit stems in
|
||||
it and a chain of stems can settle or not by it. That is a known evaluator defect, improvise-bw4p.30
|
||||
(`context/code/improvise-incremental.md`, step 7 item 8), and until it is fixed a source with no order
|
||||
of its own should list stems in a stable one.
|
||||
|
||||
`Select::admits` is the predicate of a select over one key, for a source that must test keys itself
|
||||
(`DataStore::count`). `TableSource` does not use it: its own `holds` is the independent second reading.
|
||||
|
||||
backend and a model share. **A feed has one drainer**: a drain empties the pending list, so two
|
||||
backends over one feed split the changes between them (measured in review: the same write gave one
|
||||
backend a Revenue of 128 and the other 38, and the second then disagreed with its own exact reads).
|
||||
`a_feed_has_one_drainer` pins the behaviour. Kb-style invariant: `invariant(change_source, one_drainer)`.
|
||||
|
||||
## Building a `Summary` from an engine's answer
|
||||
|
||||
`Summary::from_parts(Numbers::from_parts(count, exact_sum, least, greatest)?, Agreement)?`. `Agreement`
|
||||
is what the non-number values hold: `None`, `One(v)` or `Mixed`. Both constructors return a `Result`
|
||||
and **refuse** a state no sequence of values could produce (extremes with no numbers, a NaN extreme,
|
||||
least above greatest, one extreme without the other, a sum with no numbers, no extremes beside a sum
|
||||
that is not NaN, a number among the "others"), because repairing
|
||||
it would hide an engine's bug.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,605 +0,0 @@
|
||||
# The terminal front end (`src/`)
|
||||
|
||||
The root crate is the ratatui binary: `src/draw.rs` (event loop and frame
|
||||
composition), `src/ui/*.rs` (the widgets), `src/main.rs` (clap). It is one of
|
||||
two front ends, and design-principles §5 is the rule it is judged by — it
|
||||
paints and wires, and every decision with a silently wrong answer is made in
|
||||
`improvise-protocol` so the browser cannot answer it differently.
|
||||
|
||||
Test narratives live in `context/test-evidence/tui.md`. Facts about
|
||||
`GridCache`, `AxisFrame`, `Projection` and the `Command`/`Effect` layer belong
|
||||
to `context/code/protocol.md` and `context/code/command.md`; this file cites
|
||||
them and does not restate them.
|
||||
|
||||
---
|
||||
|
||||
## `src/ui/grid.rs` — `GridWidget`
|
||||
|
||||
The grid is drawn from a `GridCache` (improvise-n10).
|
||||
|
||||
**No `&Model`.** Every value the grid paints — each cell's text, whether it is
|
||||
an aggregate, the column widths, the page bar and the totals strip — arrives
|
||||
already decided, on the cache. A widget that cannot reach the model cannot
|
||||
quietly answer a question differently from the way the browser client answers
|
||||
it, and that divergence is the whole thing projecting exists to prevent.
|
||||
|
||||
**The frame comes off the cache too** (improvise-anr9). `GridCache` carries an
|
||||
`AxisFrame` per axis — the header levels, the label *tuples* and the groups
|
||||
with their collapse state — so the multi-level headers, the repeat suppression,
|
||||
the group spans and the pieces below are all read rather than derived, and a
|
||||
browser painting the same frame paints the same thing. It used to be derived
|
||||
here from `GridLayout` + `View::is_group_collapsed` while the protocol carried
|
||||
`GridLayout::row_label`'s slash-joined form — the rendering
|
||||
`two_row_categories_produce_cross_product_labels` asserts must not appear — so
|
||||
the browser drew `Food/Alice`.
|
||||
|
||||
Two things are still not on the cache and are passed separately, each a
|
||||
different kind of absence:
|
||||
|
||||
- **`layout` resolves a records-mode column** (`record_col`), and only that. It
|
||||
keys `drill_state`'s pending edits, so it is session-local by design and
|
||||
correct to keep off the wire (gotcha 1).
|
||||
- **`drill_state` is session-local residue** (gotcha 30). Pending drill edits
|
||||
belong to whoever staged them and are deliberately not projected, so they
|
||||
shadow the cache rather than being on it.
|
||||
|
||||
The cache is expected to cover the **whole** grid — offsets zero, every row and
|
||||
every column — so a position in it is an absolute position and the widget's
|
||||
scrolling arithmetic is unchanged.
|
||||
|
||||
### Inside `render_grid`
|
||||
|
||||
`GridWidget::cursor` is this client's cursor in the active view (improvise-8uz),
|
||||
passed in rather than read off `View`, which no longer carries per-client state.
|
||||
|
||||
Everything about the frame is the cache's answer (improvise-anr9): how many
|
||||
header levels, what each level's label is, which group a position sits in and
|
||||
whether that group is collapsed. Nothing in `render_grid` reads `GridLayout` or
|
||||
`View`, so a browser folding the same `AxisFrame` draws the same header.
|
||||
|
||||
**The bulk forms are built once per frame, and that is not a micro-optimisation.**
|
||||
`header_labels(offset)`, `groups()` and `shares_ancestry(sel_col)` each make one
|
||||
pass. The per-position accessors (`item`, `label`, `shows_level`, `group_of`,
|
||||
`same_prefix`) scan the frame from the start, so asking them inside the draw
|
||||
loops is quadratic in the grid and, at a deep scroll offset, grows with the
|
||||
offset. `col_lines` transposes `header_labels` to level-major because the column
|
||||
header is drawn one line per level while `header_labels` answers per column.
|
||||
|
||||
**The underline on the column header** marks columns sharing `sel_col`'s
|
||||
ancestor group through each level: at the bottom level that is only `sel_col`,
|
||||
at higher levels it spans all the sub-columns.
|
||||
|
||||
**Pending drill edits shadow the projected value.** They are this session's
|
||||
uncommitted typing and by construction are not on the cache (gotcha 30).
|
||||
|
||||
**Aggregate cells are drawn in italic** to signal "drill to edit", and the fact
|
||||
is *not* re-derived here: the projection already named what each cell is, and
|
||||
`CellId::Aggregate` is exactly that fact (ADR 002), decided once by the side
|
||||
that holds the model. The widget used to walk `none_cats` × `kind.is_regular()`
|
||||
— a second hand-rolled copy of `project::cell_id`.
|
||||
|
||||
**Whether there is a total row at all is the cache's answer.** An empty totals
|
||||
strip means "no totals" (records mode, or an empty grid), which is why the
|
||||
widget does not read `is_records_mode` a second time.
|
||||
|
||||
**The collapse glyph** (`indicator`) is the one thing that is legitimately
|
||||
local: the *fact* — whether the group is collapsed — is on the frame; only its
|
||||
spelling is the terminal's (design-principles §5).
|
||||
|
||||
---
|
||||
|
||||
## `src/draw.rs`
|
||||
|
||||
### The key-event boundary
|
||||
|
||||
`convert_key_event` / `convert_key_code` / `convert_modifiers` are the **only**
|
||||
place crossterm's key types and the crate's own `command::key` types may both
|
||||
appear. Three rules:
|
||||
|
||||
- The event loop converts every incoming `Event::Key` regardless of
|
||||
`KeyEventKind` (Press/Repeat/Release), matching the pre-conversion behaviour,
|
||||
which never inspected the kind.
|
||||
- Codes the command layer does not model (media keys, `Insert`, …) become
|
||||
`KeyCode::Null`, which no keymap binds.
|
||||
- `SHIFT`/`CONTROL`/`ALT` survive; every other modifier bit (`SUPER`, `HYPER`,
|
||||
`META`) is dropped.
|
||||
|
||||
### `face_style` — the one place a face becomes a terminal style
|
||||
|
||||
ADR 004 replaced the four per-family maps this section used to describe
|
||||
(`axis_fg`, `minibuf_fg`, `field_kind_fg`, `mode_style`) with one function:
|
||||
`face_style(Face) -> Style` resolves the face against the shipped theme
|
||||
(`Theme::resolve`, in `improvise-protocol`) and translates the result. That is
|
||||
design-principles §5's shape with nothing left over: **the decision is shared,
|
||||
and only the last translation is local** — `Colour` → `ratatui::Color` and the
|
||||
six attributes → `Modifier`. The chain is never walked here (ADR 004 clause 3).
|
||||
What the vocabulary is and why is `context/code/protocol.md` §10.
|
||||
|
||||
Three things to know.
|
||||
|
||||
- **`Colour::Default` sets nothing.** It maps to "no `fg`/`bg` on the `Style`",
|
||||
so the terminal's own colour shows through. That is also why a face with no
|
||||
stated colour produces `Style::default()` and not a `Reset`.
|
||||
- **The slot names follow ANSI, not ratatui.** ANSI slot 7 is `White`, which
|
||||
ratatui calls `Gray`; slot 8 is `BrightBlack`, ratatui's `DarkGray`; slot 15
|
||||
is `BrightWhite`, ratatui's `White`. `terminal_colour` is the one place that
|
||||
translation is written.
|
||||
- **The shipped theme is resolved once**, into a `OnceLock`, because there is no
|
||||
theme picker yet (ADR 004 "Deferred"). When one arrives the theme becomes the
|
||||
TUI's own state rather than a static, and a theme is still never on `App`'s
|
||||
`ViewState` (clause 7).
|
||||
|
||||
**The mode indicator is still exhaustive over `AppMode`**, now in
|
||||
`Face::of_mode` rather than here: adding a variant fails the build there
|
||||
(improvise-zx8t's guard, moved with the table). The thirteen DarkGray variants
|
||||
are no longer one arm: they are the five faces ADR 004 derives
|
||||
(`mode.resting`, `mode.text-entry`, `mode.panel.*`, `mode.overlay`,
|
||||
`mode.transient`), which all still inherit the same colours from `mode`.
|
||||
|
||||
**No-op evidence.** `face_tests::the_shipped_theme_reproduces_every_migrated_literal`
|
||||
holds the old literals as a table and asserts the shipped theme draws each one,
|
||||
which is clause 8's property. The one deliberate difference, the category-add
|
||||
prompt turning green, is asserted by
|
||||
`the_category_add_prompt_draws_as_the_item_add_prompt_does` rather than being
|
||||
left out of the table silently.
|
||||
|
||||
**Still literal**: every other `Color::` in `src/` (the selection sites, the
|
||||
panels' accents, the grid's headers, help, the welcome popup). They move as
|
||||
their files are touched (improvise-nn4s.3, improvise-nn4s.6). Do not touch
|
||||
`src/ui/grid.rs` for this while improvise-felo.6 is in flight.
|
||||
|
||||
### The minibuffer shares the bottom row with the status line
|
||||
|
||||
`draw_bottom_bar` used to render the minibuffer *instead of* `draw_status`, so a
|
||||
status message written while any text-entry mode was open was never drawn
|
||||
(`improvise-mai9`, and the stranding gotcha 21 describes). `minibuffer_line`
|
||||
composes both: the prompt and buffer on the left, any pending `status_msg`
|
||||
right-aligned. The alternatives and why they lost are in
|
||||
`context/test-evidence/tui.md`.
|
||||
|
||||
### `fill_line` — which segment yields is a per-call-site decision
|
||||
|
||||
Three rows in the TUI are one left segment and one right segment sharing a line:
|
||||
the title bar, `minibuffer_line` and `draw_status`. `fill_line` composes them,
|
||||
and it owes two things that it did not originally do — it must measure in
|
||||
**display cells** (it padded with `left.len()`, bytes, and the title's own `·`
|
||||
is two of them, so every document's right segment sat one cell in from its edge)
|
||||
and it must **truncate**, because concatenating and letting ratatui clip clips
|
||||
the *right*.
|
||||
|
||||
What clipping the right costs depends on the row, which is why `Keeps` is a
|
||||
parameter and not a rule:
|
||||
|
||||
- **The title bar keeps the right.** Its left segment is decoration whose length
|
||||
is unbounded — a document name plus a filename — and its right segment is the
|
||||
fixed `" ?:help :q quit "`, the only thing on screen telling a stuck user how
|
||||
to get out. Unconditional left priority made the escape hatch the first thing
|
||||
sacrificed, at 100 columns with an ordinary document name (improvise-t14d).
|
||||
- **Both bottom-bar rows keep the left**, which is what they already did. There
|
||||
the left is the user's own typing plus the cursor (`minibuffer_line`) or the
|
||||
status message (`draw_status`), and the right is a view badge or the message's
|
||||
own margin. Right priority there would let a long message eat the text being
|
||||
edited — the same mistake in the other direction.
|
||||
|
||||
So the rule is not "the right wins" but "the affordance wins over the
|
||||
decoration", and which is which is local. Truncation goes through
|
||||
`ui::text::truncate` (improvise-8l0c), so an overflow is a `…` rather than a
|
||||
mid-token clip of `?:`.
|
||||
|
||||
### One key table on screen, and it is the one derived from the keymap
|
||||
|
||||
The TUI stacks the tile bar (row 3) directly above the status bar (row 4), and
|
||||
the import wizard's overlay sits above the status bar too, so a mode can put
|
||||
**two** key tables on screen at once. When it does, they disagree — different
|
||||
notation, different content, and in one case a flat contradiction.
|
||||
|
||||
The rule is not "the overlay wins" or "the status bar wins". It is **keep
|
||||
whichever table is derived from the thing it describes** (improvise-rr8n):
|
||||
|
||||
- **Tile select: the status bar wins.** `hint_line(&mode)` is read off the
|
||||
keymap, so it cannot go stale, and it is a strict superset of what the tile
|
||||
bar's literal carried — `Space`, `h`/`l` and `Tab` were only ever on the
|
||||
derived line. The tile bar's own hints are therefore **deleted**, both of
|
||||
them, and `TileBar::render` now advertises no keys at all. The one it drew in
|
||||
Normal was false: it named `Ctrl+↑↓←→` as "to move tiles" while those four
|
||||
keys are bound to `enter-tile-select`, which moves nothing.
|
||||
- **The import wizard: the overlay wins.** `hint_line(&AppMode::ImportWizard)`
|
||||
is derived too, but from the *wrong input* — the wizard's real keys are
|
||||
resolved per step inside `HandleWizardKey`, and one static line off the mode's
|
||||
keymap is wrong on four of its eight steps, including telling the user `Esc`
|
||||
cancels on `Done` where `Esc` goes back. So `draw_status` suppresses its hint
|
||||
fallback for that one mode and lets the overlay's per-step hints stand alone.
|
||||
|
||||
Two things worth keeping from how this survived. `every_hint_key_resolves_in_its_mode`
|
||||
checks `AppMode::hint_text()` against the keymaps and **cannot see a string
|
||||
literal in a widget**, which is the same blind spot improvise-wk18 names for
|
||||
`help.rs`. And the widget's own characterization test renders at 80 columns,
|
||||
where the hint does not fit and is never drawn — so the test that pins this
|
||||
widget's output was structurally incapable of showing either hint. A fixture
|
||||
width is part of what a test can see.
|
||||
|
||||
The longer-term shape the ticket names is still open: the wizard's per-step keys
|
||||
belong in `advertised()` keyed by step, which would make them derived like
|
||||
everything else and let the status bar carry them correctly instead of being
|
||||
switched off.
|
||||
|
||||
### The grid's width is one answer, and a scroll is measured where it is drawn
|
||||
|
||||
Two parts of the TUI scroll horizontally, and both lost their cursor for the
|
||||
same reason: **the code that chose the scroll offset measured a different width
|
||||
from the code that drew it** (improvise-gtn7.1).
|
||||
|
||||
- **The grid.** `App::cmd_context` counts `visible_cols` for the keypress path,
|
||||
and `GridWidget` counts again from the `Rect` it is handed. The keypress path
|
||||
counted from `term_width`, while `draw_content` gave the grid `term_width`
|
||||
minus the 32-column side panel. So with any of the three panels open the
|
||||
cursor was scrolled to a column the widget never drew: it went "behind the
|
||||
panel". Now `viewport::grid_width(term_width, side_panel_open)` is the one
|
||||
answer, reached through `App::grid_width`. `draw_content` splits the content
|
||||
row on it, and `cmd_context` counts on it. `App::side_panel_open` is the one
|
||||
answer to "is a panel open", so a panel that did not count toward the width
|
||||
would not be drawn either, and the two sides still agree.
|
||||
- **The tile bar.** `TileBar` chose its first tile against the width after
|
||||
the prefix, but drew tiles one column later, and two columns later once
|
||||
scrolled, to leave room for `◀`. When the selected tile fitted within those
|
||||
one or two columns, the scroll stopped too early and the draw loop dropped the
|
||||
tile. `tiles_left(start)` is now where both the scroll (`first_tile`) and the
|
||||
draw begin.
|
||||
|
||||
**Why it looked like a regression.** The grid defect dates from `b5418f2e`
|
||||
(2026-04-06), when `compute_visible_cols` replaced a rough estimate and took
|
||||
`term_width`. It was always wrong with a panel open. But until `50e4a372`
|
||||
(2026-09-20) no panel was open at launch. That commit opened the category panel
|
||||
by default, in the shared `ViewState::default()`, and from then on every TUI
|
||||
session started in the broken configuration. The tile bar's offset has been
|
||||
wrong since `4686f470` (2026-04-09), when it first scrolled.
|
||||
|
||||
**What is still wrong: improvise-gtn7.2.** `visible_cols` is a *count*, taken at
|
||||
the *old* offset, and `viewport_effects` applies it at the *new* one. Across
|
||||
columns of mixed width no single count is right, so the cursor can still leave
|
||||
the grid with no panel open. The fix is to measure the new offset against the
|
||||
widths, as `first_tile` does. It is not in this change because
|
||||
`resolve.rs::scrolled_to` mirrors `viewport_effects` over a window counted in
|
||||
columns, and `lockstep.rs` holds the two in agreement.
|
||||
|
||||
### The caches are built here, per frame
|
||||
|
||||
`project::formulas_of`, `category_tree_of`, `view_list_of`, `tile_bar_of`,
|
||||
`grid_of` and `wizard_of` are all called from `draw`/`draw_content` against the
|
||||
authoritative `App`. This is the **temporary half of improvise-35e**: when the
|
||||
TUI becomes a client it will *hold* each cache and fold the matching
|
||||
`Projection` into it, and none of the widgets change either way.
|
||||
|
||||
Two consequences today:
|
||||
|
||||
- The grid's viewport covers the **whole** grid, so a cache position is an
|
||||
absolute position (gotcha 44). Only the viewport narrows when the TUI becomes
|
||||
a client.
|
||||
- The wizard is **built here rather than sent**: it is session-local and has no
|
||||
`Projection` (improvise-764). It still goes through `project`, so the widget
|
||||
sees the same shape a browser wizard would.
|
||||
|
||||
`draw` renders overlays last so they appear on top. The which-key popup's armed
|
||||
prefix is stored as the *key that opened it* and resolved back to a sub-keymap
|
||||
by the keymap set (improvise-jc8, gotcha 13).
|
||||
|
||||
---
|
||||
|
||||
## `src/ui/import_wizard_ui.rs` — `ImportWizardWidget`
|
||||
|
||||
Drawn from a `WizardCache` (improvise-764).
|
||||
|
||||
**No `ImportWizard`, no `ImportPipeline`, no `Workbook`.** The widget used to
|
||||
hold the live wizard and read through it into `improvise-io` types and into the
|
||||
model — it called `regular_category_names()` and `measure_item_names()` on the
|
||||
built `Workbook` to write its one-line confirmation summary. Everything it draws
|
||||
now arrives already decided, from `project::wizard_of`.
|
||||
|
||||
The cache is **not** a `Projection`, and that is the ticket's answer to where
|
||||
wizard state lives: a half-open import dialogue belongs to the session that
|
||||
opened it, not to every connected client. `WizardCache`'s own documentation
|
||||
carries the argument.
|
||||
|
||||
Smaller facts the drawing depends on: the root array path is already spelled
|
||||
`(root)` by the projection, so both ends say it the same way; the date
|
||||
components arrive in the order the wizard offers them, already paired with
|
||||
whether they are on, so the cursor index counts exactly those rows; and on
|
||||
`Done` the workbook is built but not yet installed, so the step shows what the
|
||||
user is about to accept, warnings included (improvise-907).
|
||||
|
||||
### `set_bottom_line` — one writer for the bottom row
|
||||
|
||||
That row has two writers — each step's key hint, and the wizard message that
|
||||
overrides it while present — and they write in that order to the same `y`.
|
||||
`Buffer::set_string` writes exactly as many cells as the string has and leaves
|
||||
the rest alone, so a message shorter than the hint it replaced left the hint's
|
||||
tail sitting beside it: `something went wrongcel`, the `cel` from `Esc: cancel`
|
||||
(improvise-q0vb).
|
||||
|
||||
**Padding here rather than at each site is the point.** The bug is not that one
|
||||
of the five writers forgot; it is that the row had five writers and no one of
|
||||
them owned clearing it.
|
||||
|
||||
**The padding is measured in terminal cells, not characters.** `Buffer` lays a
|
||||
string out by display width, so a char count over-pads a message carrying a wide
|
||||
character — enough to run past the panel and overwrite the border — and
|
||||
under-pads a combining mark, which is how a sliver of the hint would be left
|
||||
behind. `src/ui/grid.rs` measures the same way.
|
||||
|
||||
### `truncate` / `prefix` — bytes, deliberately
|
||||
|
||||
`truncate` cuts by **byte** length. `prefix` returns the longest prefix at most
|
||||
`max` bytes that ends on a char boundary, because every string `truncate` cuts
|
||||
can carry a field name, a formula or a message the user typed — and a multi-byte
|
||||
character straddling the cut used to panic the whole TUI on a slice at a
|
||||
non-boundary index. Walking down to the nearest boundary is the smallest change
|
||||
that cannot: for ASCII, which is what every fixture in this file holds, `max` is
|
||||
already a boundary and the result is byte-identical, so the characterization
|
||||
tests keep saying what they said.
|
||||
|
||||
It is bytes and not columns *deliberately*. Byte length is the measure
|
||||
`truncate` has always used, and changing it to display width would move where
|
||||
every line wraps — a different question from not crashing, and one that belongs
|
||||
with the hint-line padding it travels with. Bytes are never fewer than cells, so
|
||||
`truncate` cutting by bytes and `set_bottom_line` padding by cells cannot
|
||||
together overrun.
|
||||
|
||||
**The blind spot, stated because the fixtures cannot state it**: the eight
|
||||
exact-frame characterization fixtures in this file are entirely ASCII. They are
|
||||
therefore blind to every question about display width — a change to either
|
||||
measure is invisible to them, and the evidence for those has to be built
|
||||
directly (see `set_bottom_line_clears_every_cell_it_covers_and_no_more`).
|
||||
|
||||
---
|
||||
|
||||
## `src/ui/help.rs`
|
||||
|
||||
~600 lines of mostly-static content: five pages of styled `(key, description)`
|
||||
lines, a tab bar and a footer. `improvise-ktv1` proposes moving that content
|
||||
into the shared layer so a browser help overlay is not a second copy of it.
|
||||
Until then it is TUI-only and `help_page` is already in `ViewState`, which is
|
||||
why gotcha 32 says the overlay needs no `Projection`.
|
||||
|
||||
## The panel widgets
|
||||
|
||||
`src/ui/panel.rs` holds the `PanelContent` trait and the generic `Panel` widget.
|
||||
The one non-obvious member is `panel()`: each impl declares its own identity, and
|
||||
the widget compares it against `AppMode::panel()` to decide whether to draw
|
||||
itself as active, instead of restating the mode match per panel.
|
||||
|
||||
`category_panel.rs`, `formula_panel.rs`, `view_panel.rs` and `tile_bar.rs` each
|
||||
draw one cache and nothing else. Two facts worth keeping:
|
||||
|
||||
- The category panel draws one flattened tree — the axis travels on the entry
|
||||
(improvise-edp), so there is no `View` in it.
|
||||
- The view panel's "which view is this client on" is read off `ViewState`, not
|
||||
the workbook: the active view is per-client and is deliberately absent from
|
||||
`ViewListCache` (improvise-8uz, gotcha 29).
|
||||
|
||||
`tile_bar.rs` scrolls by whole tiles: it finds the smallest start index at which
|
||||
the selected tile still fits in the available width, and draws `‹`/`›` overflow
|
||||
indicators. That is the file's only piece of real arithmetic and it is local
|
||||
geometry, not a shared derivation.
|
||||
|
||||
## `,` walks the view history forward (improvise-htl6)
|
||||
|
||||
`view-forward` was bound in **no** keymap, in either front end, from the day it
|
||||
was written. The TUI reached it as `:view-forward` and a browser could not reach
|
||||
it at all, because `declined()` refuses to enter `CommandMode` at the door. It
|
||||
survived because pressing `<` twice is the only way to observe the forward stack
|
||||
in the first place, and until PR #115 no client maintained one.
|
||||
|
||||
**The key is `,`, and `>` is why the obvious answer is wrong.** `<` is
|
||||
`view-back` and `>` is `drill-into-cell`, so the pair that *looks* symmetric is
|
||||
not one — `>` moves within the grid, `<` moves between views. Binding the
|
||||
forward half to anything that reads as `>`'s partner would deepen exactly the
|
||||
confusion that hid the gap. `,` is the unshifted twin of `<`, so both halves of
|
||||
the history sit on one physical key, and it carries no direction-in-the-grid
|
||||
meaning to collide with.
|
||||
|
||||
**It is deliberately not a `Ctrl-` key.** `Ctrl-R`, the universal "forward
|
||||
through a history" spelling, is browser reload; `Ctrl-S` and `Ctrl-W` are
|
||||
already bound here and `Ctrl-S` is already a filed bug for precisely that reason
|
||||
(improvise-clso, "Ctrl-S saves and also opens the browser's own save dialog").
|
||||
The keymap is **shared** between the two front ends, so a binding chosen for the
|
||||
terminal is a binding the browser must also live with, and adding a second
|
||||
instance of an open bug is not a thing to do by accident.
|
||||
|
||||
Nothing else moved. `help.rs` keys its lines off the command *name* and looks
|
||||
the key up in the keymap, so the overlay and the hint line follow the binding;
|
||||
`every_bound_command_name_is_registered` already covers the new entry. The
|
||||
absence assertion PR #117 put in
|
||||
`resolve::tests::the_forward_step_mirrors_its_cmd_and_no_key_reaches_it` was
|
||||
**inverted rather than deleted** — the test is renamed `…_and_one_key_reaches_it`
|
||||
and now asserts the binding exists, with `view-back` kept as the control so it
|
||||
cannot pass against a table that binds everything or nothing. Coming there and
|
||||
saying so was the point of that assertion, and an inverted guard keeps the
|
||||
property under watch where deleting it would have let the key quietly go away
|
||||
again.
|
||||
|
||||
`,` is not advertised in `AppMode::Normal`'s hint line, and neither is `<`: the
|
||||
hint is width-constrained and the history lives on help page 5 under "View
|
||||
history" with both halves. That is the #149 lesson — a hint that does not fit is
|
||||
never drawn, so the test pinning the widget cannot see it.
|
||||
|
||||
## A status message lasts until the next key (improvise-cq3d)
|
||||
|
||||
`draw_status` draws `status_msg` *instead of* the mode's derived hint, and
|
||||
nothing in production ever cleared `status_msg`: it was only replaced by the next
|
||||
message. So after the first "Saved", "Loaded" or error of a session, the derived
|
||||
key hint (the one key table on screen, improvise-rr8n) was never drawn again, in
|
||||
any mode.
|
||||
|
||||
**The rule: a message expires at the start of the next keypress.** `draw.rs::press`
|
||||
applies `SetStatus("")` and then dispatches the key, so:
|
||||
|
||||
- a message a key produces is drawn after that key, whatever it is, errors
|
||||
included. It stays until the user presses something, which is the moment they
|
||||
have acted on it. An error is never wiped before it has been drawn.
|
||||
- a message that arrives outside a key, such as one from startup or an import,
|
||||
also stands until the next key.
|
||||
- the next key clears it *before* its own effects run. So a key that writes a
|
||||
new message shows that message, and a key that writes none shows the hint.
|
||||
|
||||
**Rejected alternatives.**
|
||||
|
||||
- *Expire on mode change.* This fails the ticket's own case: a message written in
|
||||
Normal would stand in Normal indefinitely.
|
||||
- *Expire on the next successful command.* Effects carry no notion of success,
|
||||
and a failed command would leave the stale message in place. That is exactly
|
||||
the case where the user most needs the hint back.
|
||||
- *Clear after dispatch.* Every message would be wiped before it is drawn. The
|
||||
test's middle step exists to catch that.
|
||||
- *Compose message and hint on one line.* A long message still eats the hint, and
|
||||
the stale message still stands forever.
|
||||
|
||||
**Why here and not in `App::dispatch_key`.** `dispatch_key` is shared with the
|
||||
lockstep harness (`reduce_key`), which compares `status_msg` on the host and the
|
||||
browser after every key. Expiring there without changing the browser's resolver
|
||||
too would make the two ends disagree. The expiry is also a presentation decision
|
||||
about the TUI's one-line status row, so it belongs to the TUI front end.
|
||||
|
||||
**The browser already behaves this way.** Its `status_msg` is state that is never
|
||||
cleared either, but `web/app.js` shows it in a toast that hides after
|
||||
`TOAST_MS` (`web/shell.js::createToasts`), and `#hint` is a separate element that
|
||||
is always drawn. So a browser user sees a message briefly and never loses the
|
||||
hint. The TUI now does the same, on a keypress rather than a timer, because a
|
||||
terminal status row has one line to share and a keypress is the event the TUI
|
||||
already has.
|
||||
|
||||
### Except while a minibuffer stays open under it (improvise-su4u)
|
||||
|
||||
In a text-entry mode `minibuffer_line` draws the message to the right of the
|
||||
prompt. Since improvise-v4hg, a formula that does not parse keeps the user in
|
||||
`FormulaEdit` with the error. Under the plain rule, the error vanished on the
|
||||
first correcting key: one Backspace and the user was fixing text they could no
|
||||
longer see the complaint about. The same happened to any refusal that leaves
|
||||
the user in a minibuffer.
|
||||
|
||||
**The rule: a key that leaves the user in the same editor, and says nothing of
|
||||
its own, puts the stale message back.** `press` remembers the editor session and
|
||||
the message, clears the message as before, dispatches the key, and then restores
|
||||
the message only if all three of these hold:
|
||||
- a text-entry mode was open before the key;
|
||||
- the same editor session is open after it (below);
|
||||
- the key wrote no message of its own.
|
||||
|
||||
So:
|
||||
|
||||
- editing keys keep the refusal on screen until the user acts on it;
|
||||
- a key's own message always wins, so a second refused `Enter` shows the new
|
||||
error, not the old one;
|
||||
- leaving the minibuffer, by a successful commit or by `Esc`, drops it, exactly
|
||||
as the plain rule does.
|
||||
|
||||
**Rejected alternatives.**
|
||||
|
||||
- *Expire only on the commit key.* That needs to know which keys commit, which
|
||||
is the keymap's knowledge (`TextEntry::commit`), and a second copy of it in
|
||||
the TUI loop. "The mode did not change" is observed rather than predicted,
|
||||
like gotcha 31's `ReturnToNormalUnlessMoved`.
|
||||
- *Never clear in a text-entry mode.* The message would then outlive `Esc` and
|
||||
stand in the mode the user returns to, which is the cq3d bug again.
|
||||
- *Compare the message after the key with the stale one.* A key that wrote the
|
||||
same text would be indistinguishable from a key that wrote none. Clearing
|
||||
first and restoring only an empty slot has no such ambiguity.
|
||||
|
||||
### What "the same editor" means (improvise-sx7h)
|
||||
|
||||
The first version of this rule compared modes: an equal `AppMode` before and
|
||||
after the key meant the same editor. That is false for the two cell editors.
|
||||
`Editing` and `RecordsEditing` commit with `CommitAndAdvance`, which drops to
|
||||
Normal, writes the cell, moves the cursor and re-opens the editor through
|
||||
`EnterEditAtCursor`, which sets an **equal** `AppMode`. So after an `Enter` or
|
||||
`Tab` that committed and moved on, the mode looked unchanged, and the stale
|
||||
message was restored beside a different cell's editor. It came back again on
|
||||
every later commit until `Esc`. The reviewer's repro was `R`, `o`, `5`, `Tab`,
|
||||
`7`: "Added new record row" survived the `Tab`, and so did `o`'s refusal
|
||||
"That record row is already here…" after the `Tab` that filled the row in.
|
||||
`FormulaEdit` could not show it, because its successful commit leaves the mode
|
||||
and writes a message of its own.
|
||||
|
||||
An editor session is one *opening* of a minibuffer. The TUI loop cannot see
|
||||
openings, so `EditorSession` compares what an opening sets:
|
||||
- **the whole `AppMode`**, not only its variant: an `ItemAdd` for `Month` is
|
||||
not the `ItemAdd` for `Region` it replaced;
|
||||
- **the cursor cell**, which is a cell editor's target, so a commit that
|
||||
advances is a new editor. The prompt modes do not move the cursor while the
|
||||
user types, so for them this is inert;
|
||||
- **the editor's seed** (`opened_editor.seed`). `EnterEditAtCursor` re-seeds
|
||||
from the cell. So a commit that re-opens on the **same** cell, such as `Tab`
|
||||
in the last column, is a new editor too, because the seed is now the
|
||||
committed text.
|
||||
|
||||
What it cannot see: a commit that re-opens on the same cell with the same seed.
|
||||
That means the user committed the text the cell already held, which writes
|
||||
nothing. Restoring a message across a commit that changed nothing is harmless.
|
||||
|
||||
**Rejected: an editor generation counter.** It would be exact, but an editor is
|
||||
opened by more than one effect (`EnterEditAtCursor`, and `ChangeMode` from
|
||||
`enter-mode`). A counter bumped in each is a hand-kept list of the places a mode
|
||||
is written, which is the shape gotcha 24 retired. The three fields are set by
|
||||
the opening itself, so they need no list.
|
||||
|
||||
**Not included: the active view.** The cursor is per view, so in principle the
|
||||
view belongs with the cell. But no key a minibuffer binds can switch the view
|
||||
and stay in the minibuffer: a `:` line that switches view settles back to
|
||||
Normal. So nothing could test the field, and an untestable field is a claim no
|
||||
test can check.
|
||||
|
||||
**The browser asks the same question (improvise-1x9h).** `EditorSession` now
|
||||
lives in `improvise-protocol` and is built from a `ViewState`, so the TUI's
|
||||
`press` and the browser's `ClientState::status_stands` compare the same value.
|
||||
The browser's mechanism differs: its message is a toast on a timer, and there
|
||||
is no per-key clear. So it keeps the message's *owner* and suspends the timer
|
||||
while that owner is open. See `context/code/clients.md`, "A message beside its
|
||||
open editor stands until that editor closes".
|
||||
|
||||
## The import wizard wraps and scrolls; it never truncates a message (improvise-cgqt)
|
||||
|
||||
The user's ruling of 2026-09-25: "a message like this should wrap not truncate
|
||||
so I can read the whole thing." Every warning and message the wizard draws goes
|
||||
through `ui::text::wrap`. That covers the Confirm step's warnings, the bottom-line message,
|
||||
the duplicates step's intro, options, disabled reasons and colliding records,
|
||||
and the name step's explanation. `wrap` measures in cells, like `truncate`
|
||||
(gotcha 50). It breaks at spaces, cuts a word only when the word alone is wider
|
||||
than the line, and emits a lone double-width character rather than dropping it
|
||||
when the line is narrower than that character. A leading indent is kept on every
|
||||
line, so an indented row hangs under itself.
|
||||
|
||||
**The bottom-line message wraps upward** from the row the key hint uses. A
|
||||
one-line message therefore draws exactly as before, which is why the pinned
|
||||
frames (`every_step_draws_what_it_drew_before`,
|
||||
`the_formula_step_draws_its_transient_state`) did not move.
|
||||
|
||||
**Lists scroll by item, and the header says what is shown.** `Panel::scrolling`
|
||||
draws the items from `WizardCache::scroll` on, each one wrapped whole. The
|
||||
list's own header row reads `Colliding records 3–9 of 14 (PgUp/PgDn):` whenever
|
||||
some items are off screen. A first version put "more above" and "more below"
|
||||
markers inside the list's rows. When the first item filled the list exactly,
|
||||
the marker landed on the hint row and was drawn over (Codex, PR #304). The
|
||||
header row belongs to no item, so no item can cover it, and no item loses a line
|
||||
to make room for a marker. `ImportWizard::scroll` clamps to `len - 1`, so the last item can reach the top
|
||||
and the list can never scroll away. It is by item, not by line, because the
|
||||
wizard state cannot know the panel width that line-level clamping needs. The
|
||||
residual is honest. A **single** item taller than the whole list area, roughly
|
||||
1,500 characters at 80 columns, is still clipped at the panel's bottom. A
|
||||
collision tuple has at most `MAX_CATEGORIES` coordinates (24 since
|
||||
improvise-felo.1, up from 12) plus labels. With short names that is a few
|
||||
hundred characters. Neither the names nor a label value is bounded, though, so
|
||||
reaching the residual got easier when the cap doubled.
|
||||
|
||||
**Every write is clipped to the panel.** The widget draws through `Panel`,
|
||||
whose `put` writes only inside the panel's inner rectangle, and its height
|
||||
arithmetic saturates. Before this change, the duplicates step wrote at an
|
||||
unbounded row and ratatui panicked at 80×10, and four older steps panicked on
|
||||
`u16` underflow at heights of 3 to 5 (review B2 on PR #304). The bottom row's
|
||||
`set_bottom_line` now truncates in cells through `ui::text::truncate`. It had
|
||||
used the widget's byte-measured `truncate`, so a wrapped line holding a `←`
|
||||
was cut one character early, with `…`, although it fit (review B1). The local
|
||||
byte `truncate` still draws field names and labels. It is byte-identical on
|
||||
ASCII, which keeps every pinned frame, and those rows are labels rather than
|
||||
messages.
|
||||
|
||||
Other TUI surfaces still truncate text a user needs. They are out of this
|
||||
ticket's scope and reported in its hand-back rather than fixed here.
|
||||
@@ -1,159 +0,0 @@
|
||||
# `zset-ffi`: the reasoning
|
||||
|
||||
`crates/zset-ffi` is a C ABI over `zset` for CFFI callers (the first is Common Lisp). It is a
|
||||
separate crate so `zset` stays std-only, serde-free and wasm32-clean (ADR 009 clause 1). It is
|
||||
native only: it is **not** in `nix/ci.nix`'s `wasmCrates`, and the workspace-wide native
|
||||
build (`--workspace`) already compiles it, so nothing in the nix files changes. The contract
|
||||
is `crates/zset-ffi/include/zset_ffi.h`; this file records the decisions behind it. The
|
||||
evidence is in `context/test-evidence/zset-ffi.md`.
|
||||
|
||||
## Build
|
||||
|
||||
`cargo build -p zset-ffi --release` writes `target/release/libzset_ffi.{so,dylib}` (the
|
||||
`cdylib`), `libzset_ffi.a` (the `staticlib`) and the rlib. The `rlib` is what the Rust tests
|
||||
link, so they call the `extern "C"` functions as plain functions.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. **Monomorphic.** Keys are byte strings `(ptr, len)`, weights are `int64_t`. The generic
|
||||
`ZSet<K>` is instantiated once, at `Vec<u8>`. A caller encodes whatever it keys by.
|
||||
2. **Opaque handles, explicit create and free.** `ZsZSet`, `ZsCircuit` and `ZsRebuild` are
|
||||
`Box::into_raw` pointers; `zs_X_new(&out)` makes one and `zs_X_free` releases it. Freeing
|
||||
null is a no-op. A double free or a use after free is undefined, as in C. Circuit streams,
|
||||
inputs and sinks are `uint32_t` ids local to their circuit, not handles: they own nothing.
|
||||
3. **Error codes, never unwinding.** Every function but the frees and two accessors returns
|
||||
`int32_t`: 0, or a `ZS_ERR_*`. Every entry point body runs in `catch_unwind` (`guard`).
|
||||
A `zset` weight-overflow panic is recognised by its message prefix and becomes
|
||||
`ZS_ERR_OVERFLOW`; any other panic is `ZS_ERR_PANIC`. Results come back through
|
||||
out-pointers written only on success. `zs_last_error()` is a thread-local, NUL-terminated
|
||||
message, never null, valid until the next failure on that thread.
|
||||
The Rust panic hook still prints the overflow message to stderr before the code returns,
|
||||
because replacing a process-wide hook from a library is not this library's call. A host that
|
||||
wants silence installs its own hook, or avoids the overflow.
|
||||
4. **Failure leaves the Z-set unchanged.** `add`, `sub`, `neg`, `scale` and `decode_add`
|
||||
compute a new value and then assign it, so an overflow half way through cannot leave a
|
||||
half-applied update (`zs_zset_insert` is one addition, so it has no half).
|
||||
5. **A failed step poisons the circuit, and a rollback revives it.** A panic in `step` leaves
|
||||
operator state half updated. So the first failure in a step, or in a rebuild step, marks
|
||||
the circuit poisoned, and every later call that uses it returns `ZS_ERR_POISONED`. The one
|
||||
way out is `zs_circuit_rollback` of a transaction that the failed step ran in
|
||||
(decision 13); a poison from outside a transaction is permanent, and the caller drops the
|
||||
circuit and builds another. Ids that do not exist, and malformed arguments, are refused
|
||||
before any state changes and do not poison.
|
||||
6. **Buffers.** The caller owns every buffer it passes and may reuse it once the call
|
||||
returns; the library copies. The library hands out no memory except handles. Output of
|
||||
variable size uses a two-call protocol: `zs_zset_encode(z, buf, cap, &needed)` always sets
|
||||
`needed`, and writes only if `cap >= needed` (else `ZS_ERR_BUFFER_TOO_SMALL`).
|
||||
7. **Wire shape.** A Z-set encodes as entries ascending by key, each `[u32 LE key length][key]
|
||||
[i64 LE weight]`. A keyed element is `pair(k, v) = [u32 LE length of k][k][v]`. These are
|
||||
this crate's own, not clause 9's wire form (that lives in `improvise-protocol`).
|
||||
8. **What the circuit offers.** Input, output sinks, `delay`, `integrate`, `differentiate`,
|
||||
`distinct`, `plus`, `filter` and `flat_map` (C callbacks), `index_by` (a callback emits the
|
||||
one group key), `join` over two pair streams, and `aggregate` with count, sum, min and max
|
||||
over `int64` values. `map` is `flat_map` that emits once. `aggregate`'s fold is fixed to
|
||||
those four because a C callback cannot return an `Option` of a group result without a
|
||||
second encoding; richer folds belong to a caller that wants one. The `Trace` interface and
|
||||
the `_in` constructors are Rust-side seams and are not exposed.
|
||||
9. **Callbacks** are `extern "C"` function pointers with a `void *ctx`; the library never
|
||||
dereferences `ctx`, and keeps both for the circuit's life. A callback reports failure by
|
||||
its return value (`flat_map`/`index_by` non-zero, `keep` and `expired` negative), and the
|
||||
step then fails with `ZS_ERR_CALLBACK`. A callback must return normally. A Rust panic across
|
||||
`extern "C"` aborts, and a Lisp `handler-case` outside `zs_circuit_step` that unwinds
|
||||
through the Rust frames skips destructors and leaves the circuit half way through a step.
|
||||
The ABI cannot stop the unwinding, so it detects it: `ZsCircuit::in_step` is set for the
|
||||
duration of `step` and `rebuild_step`, and cleared only when they return. Every entry
|
||||
point first reads it, so a circuit whose frames were skipped answers `ZS_ERR_POISONED`
|
||||
forever, and a callback that re-enters its own circuit gets `ZS_ERR_POISONED` without the
|
||||
library touching the engine. `zs_rebuild_step` reads `circuit(c)` before it borrows the job,
|
||||
so re-entry from an `expired` callback is refused first. `zs_circuit_free` on a mid-step
|
||||
circuit and `zs_rebuild_free` on a job being stepped do nothing: the object is leaked, not
|
||||
freed under the frames that are running. The first fault recorded in a step is the one
|
||||
reported (`note`), so a failing `index_by` callback is `ZS_ERR_CALLBACK`, not the
|
||||
consequent `BAD_VALUE`. The CL rule is to put `handler-case` *inside* the callback and
|
||||
return the failure value (`tests/smoke.lisp` does). `index_by` needs exactly one emitted
|
||||
key and a keyed stream valid pairs, else `ZS_ERR_BAD_VALUE`; these faults are recorded in a
|
||||
shared cell rather than panicking.
|
||||
10. **Threads** (`zset.md`, "Threads"). A `ZsCircuit` may be moved between threads and is used
|
||||
by one at a time; it is `Send` and not `Sync`, with no lock. Callbacks run on whichever
|
||||
thread calls `step` or `rebuild_step`, with that thread's thread-local state (Lisp dynamic
|
||||
bindings, `errno`, `zs_last_error`), so the caller's `ctx` must be usable from every
|
||||
thread that may step (the wrapper `Ctx` is `unsafe impl Send` on that contract). The
|
||||
first-fault cell is the one lock, and it is the FFI's own. A `ZsRebuild` no longer shares
|
||||
a reference count with its circuit (the handles are ids), so it is a separate object, and
|
||||
a `zs_rebuild_step` call uses both. The caller still keeps each to one thread at a time.
|
||||
Z-sets are separate handles. The error message is per thread
|
||||
(`tests/threads_abi.rs`: callbacks ran on the thread that stepped).
|
||||
11. **The header is hand-written** (cbindgen is not a dependency of this repo). `tests/header.rs`
|
||||
keeps it honest. It maps each Rust parameter and return type of every exported function
|
||||
(including the `unary_op!` expansions) to its C spelling, through a table that panics on
|
||||
an unlisted type, and compares them with the header's declarations. Every `Code` is
|
||||
declared by the `codes!` macro, which generates the enum, `Code::ALL` and the header
|
||||
name, so a new code cannot exist without a `#define` line and a matching value.
|
||||
12. **Versioning.** `zs_abi_version()` and `ZS_ABI_VERSION` are 1. A change that breaks a
|
||||
signature or a code bumps it.
|
||||
|
||||
13. **Transactions** (improvise-bw4p.5; `zset.md`, "Transactions"). `zs_circuit_begin`,
|
||||
`zs_circuit_commit` and `zs_circuit_rollback` wrap `Circuit::begin/commit/rollback`; a
|
||||
handle-returning `snapshot` was not chosen, because the circuit logs what changes and there
|
||||
is no snapshot object to free. The new code is `ZS_ERR_TRANSACTION` (10) for a `begin` on an
|
||||
open transaction, a `commit` or `rollback` with none open, and a `zs_circuit_take` while
|
||||
one is open (take after commit: a rollback subtracts the step's output from the sink).
|
||||
`begin` on a poisoned circuit is `ZS_ERR_POISONED`, and so is `commit` after a failed step:
|
||||
the host rolls back. `rollback` is the only call that clears the poison, and only a poison
|
||||
that came from a step inside the transaction. This is additive, so `ZS_ABI_VERSION` stays 1.
|
||||
What a rollback cannot undo is a callback's own side effects and a `ZsRebuild` stepped in the
|
||||
transaction, finished or not (it answers `ZS_ERR_INTERRUPTED` after, and the host starts another job; `rollback_abi.rs`). A
|
||||
circuit left mid-step by a non-local exit answers `ZS_ERR_POISONED` to a rollback like to
|
||||
everything else: the frames are skipped and the engine is not touched.
|
||||
|
||||
A rebuild is one logical tick however many `zs_rebuild_step` calls it takes (`zset.md`,
|
||||
decision 8): a `delay` fed by it shows the previous tick once, and `done` is 1 as soon as the
|
||||
last chunk ran, whatever the deadline callback says.
|
||||
|
||||
## Ids, handles and unsafety
|
||||
|
||||
All exported functions are `pub unsafe extern "C" fn`: the rlib is public API, and a safe
|
||||
Rust caller could otherwise pass any pointer. Output pointers are checked before any work
|
||||
(`with_out`): a refused call changes nothing and leaks nothing. Stream ids and sink ids are
|
||||
different spaces (sink ids carry the top bit). A `ZsRebuild` records its circuit's id (a process-wide counter, not an address, which can be
|
||||
reused) and is refused on another. A `Circuit::step` between two `zs_rebuild_step` calls is
|
||||
`ZS_ERR_INTERRUPTED`, without poisoning: zset's `Rebuild::interrupted`, checked first. An id from another circuit is not detected. `zs_zset_add(a, a)` is
|
||||
defined: the sum is computed through shared references and then stored. Keys of 4 GiB or
|
||||
longer are `ZS_ERR_BAD_VALUE`, not truncated. A non-empty aggregate group always has a row,
|
||||
even when its count or sum is zero.
|
||||
|
||||
## Miri
|
||||
|
||||
Miri checks the aliasing, leak and null rules the C compiler cannot. It runs `tests/abi.rs`
|
||||
on a nightly built through the flake's `rust-overlay` input. The recipe below is **not
|
||||
pinned**: `builtins.getFlake "github:oxalica/rust-overlay"` and `<nixpkgs>` (hence `--impure`)
|
||||
resolve to whatever is current, and only the nightly date is fixed. It passed at the date of
|
||||
this commit (about 12s):
|
||||
|
||||
```
|
||||
nix shell --impure --expr 'let p = import <nixpkgs> { overlays = [ (builtins.getFlake "github:oxalica/rust-overlay").overlays.default ]; }; in p.rust-bin.nightly."2026-09-24".default.override { extensions = [ "miri" "rust-src" ]; }' --command cargo miri test -p zset-ffi --lib --test abi --test rollback_abi
|
||||
```
|
||||
|
||||
It is not a CI job. A proposed one: a `miri` job in `.github/workflows/ci.yml` that takes the
|
||||
nightly from the flake's own pinned `rust-overlay` input in `nix develop .#ci`.
|
||||
|
||||
## Calling it from Common Lisp
|
||||
|
||||
`crates/zset-ffi/tests/smoke.lisp` is a runnable CFFI program (`defcfun` for each call, a
|
||||
foreign string for keys, `with-foreign-object` for out-parameters, the overflow code checked
|
||||
and the message read back with `zs-last-error`). It loads the built library and passes. It is
|
||||
**not a CI gate**: `sbcl` and `cffi` come from the user's profile and Quicklisp here, and are
|
||||
not in the nix dev shell. The command to run it is in the file's header. Sketch of the
|
||||
definitions it uses:
|
||||
|
||||
```lisp
|
||||
(defcfun "zs_zset_new" :int32 (out :pointer))
|
||||
(defcfun "zs_zset_insert" :int32 (z :pointer) (key :pointer) (len :size) (weight :int64))
|
||||
(defcfun "zs_zset_weight" :int32 (z :pointer) (key :pointer) (len :size) (out :pointer))
|
||||
(defcfun "zs_last_error" :string)
|
||||
(defcfun "zs_zset_free" :void (z :pointer))
|
||||
```
|
||||
|
||||
A callback (`filter`, `flat_map`) is a `defcallback` passed with `(callback name)`, with a
|
||||
`handler-case` inside it that returns the failure value; its `emit` argument is a foreign
|
||||
function pointer to call with `foreign-funcall-pointer`.
|
||||
@@ -1,400 +0,0 @@
|
||||
# `zset`: the reasoning
|
||||
|
||||
`crates/zset` is ADR 009 step 1 (improvise-tk51, epic improvise-bw4p): the Z-set group, first
|
||||
and alone. Nothing depends on it yet, so it changes no behaviour. The decisions are the
|
||||
ADR's (`context/adr/009-incremental-evaluation.md`, clause 1). This file records how the
|
||||
crate reads them, where the ADR left a choice open, and the evidence discipline. The
|
||||
evidence for each test is in `context/test-evidence/zset.md`.
|
||||
|
||||
## The shape
|
||||
|
||||
| item | what it is |
|
||||
|---|---|
|
||||
| `ZSet<K: Ord>` | a map from `K` to a non-zero `i64` weight; an abelian group within `i64` (interpretation 5) |
|
||||
| `+`, `-`, unary `-`, `* i64`, `scale` | the group operations and scalar multiplication |
|
||||
| `new`, `singleton`, `insert`, `FromIterator`, `Extend` | construction; weights of equal keys sum |
|
||||
| `weight`, `is_zero`, `len`, `iter`, `IntoIterator` | reads; iteration is ascending by key |
|
||||
| `map`, `filter`, `distinct` | the three per-element operators |
|
||||
| `integrate`, `differentiate` | the stream operators, over a slice |
|
||||
|
||||
There are no dependencies: not serde (clause 9 keeps the wire form in `improvise-protocol`),
|
||||
not even the workspace's own crates. The crate is wasm32-clean by construction and is in
|
||||
`nix/ci.nix`'s `wasmCrates`.
|
||||
|
||||
## Canonical form
|
||||
|
||||
A key with weight zero is never stored. `insert` removes a key whose sum reaches zero, and
|
||||
`singleton(k, 0)` is the zero Z-set. Two Z-sets are therefore `==` exactly when every key has
|
||||
the same weight, so the derived `PartialEq`, `Hash` and `Ord` are the group's equality. The
|
||||
clause 9 correction `target - held` relies on it: a correct state gives `==` the zero Z-set,
|
||||
and `is_zero` is `len() == 0`, not a scan.
|
||||
|
||||
The representation is a `BTreeMap`. Iteration order is the key order, the same in every
|
||||
process and for every insertion order, which clause 3 needs (eager folds in `CellKey` order)
|
||||
and which a `HashMap` would give up (`context/code/core.md`, gotcha 18's argument).
|
||||
|
||||
## Interpretations
|
||||
|
||||
The ADR is one paragraph on the crate, and says more about what it is for than what it is.
|
||||
These are the readings this step takes.
|
||||
|
||||
1. **Clause 1 lists a `Circuit` and more operators; step 1 built the pure part, and "The circuit" below built the rest (improvise-bw4p.1).** The
|
||||
ADR's list is `delay`, `integrate`, `differentiate`, `map`, `filter`, `flat_map`,
|
||||
`index_by`, `join_indexed`, `distinct` and `aggregate`, run by a synchronous `Circuit`,
|
||||
plus resumable rebuilds and a trace interface. Step 1 as ticketed is the group and the
|
||||
operators that are functions of a Z-set or of a stream of them: `map`, `filter`,
|
||||
`distinct`, `integrate`, `differentiate`. The rest needs the circuit (a clock, state held
|
||||
between steps), or a second key type (`index_by`, `join_indexed`, `aggregate`), and is
|
||||
left to the steps that use it rather than designed without a caller. `flat_map` is
|
||||
`map` into several images and has no user yet either.
|
||||
2. **`integrate` and `differentiate` are slice functions, not circuit nodes.**
|
||||
`integrate(&[d_0, d_1, ..])` is the running sums `d_0 + .. + d_i`; `differentiate` is
|
||||
its inverse, `s_0` then `s_i - s_(i-1)`. The state before the first element is zero, as in
|
||||
DBSP. `delay` is not provided: over a slice it is a shift, and a circuit gives it meaning.
|
||||
3. **`distinct` is not linear, and is not a group homomorphism.** It keeps weight 1 on the
|
||||
keys whose weight is positive and drops every other key, negative included. That is the
|
||||
set-semantics operator of DBSP. The ADR says nothing about negative weights; dropping
|
||||
them is the standard reading, and the one that makes `distinct` idempotent.
|
||||
4. **Operators are methods on `&self`, group operations are `std::ops` traits.** `K: Clone`
|
||||
is required only where a result copies keys (`filter`, `distinct`, `scale`, the borrowed
|
||||
operators); `map` and the by-value operators need only `Ord`.
|
||||
5. **Overflow.** Neither clause 1 nor clause 8 states a policy for weights (clause 8 is about
|
||||
`f64` drift, which wrapping integers do not have). The crate currently uses `checked_*`
|
||||
on every weight operation, for the reason given below, and **panics** with `zset: weight overflow in <op>`, in debug
|
||||
and in release (`checked_*` does not depend on `overflow-checks`). The operations are
|
||||
`add` (including `insert`, `Extend`, `FromIterator`), `sub`, `neg` (so `i64::MIN` panics
|
||||
when it has to be negated) and `scale`. Scaling by zero cannot overflow and returns the
|
||||
zero Z-set. A panic, not a `Result`: `Result` does not belong in the majority of the code
|
||||
(`AGENTS.md`), and a weight near `i64::MAX` is a defect in the caller, not an input to
|
||||
recover from. The weights clause 3 produces are small (one per cell).
|
||||
|
||||
**The group is partial, and that is the stated reading.** The ADR calls a Z-set an abelian
|
||||
group. With `i64` weights and no wrapping, the laws hold wherever no weight, intermediate
|
||||
ones included, leaves `i64`; outside that, an operation panics rather than return a
|
||||
different value. So associativity is order-dependent at the boundary:
|
||||
`(MAX + 1) + (-1)` panics where `MAX + (1 + (-1))` is `MAX`. The result is never wrong,
|
||||
only sometimes refused. Wrapping arithmetic would make the type a true group (the integers mod 2^64), and
|
||||
`(w + x) - x == w` would still hold exactly, so a retraction does cancel its insertion
|
||||
under wrapping. (An earlier version of this paragraph said otherwise; that was wrong.) What
|
||||
wrapping breaks is the intermediate reads. `MAX + 1` stores `i64::MIN`, so `distinct`
|
||||
drops a key whose true weight is positive and `weight()` has the wrong sign. A true weight
|
||||
of 2^64 stores as 0 (by the add path; a `scale` product of 2^64 would store a zero too, which
|
||||
breaks canonical form), so the key vanishes and `is_zero`, `len` and `==` report a set that
|
||||
does not exist. Those are wrong answers returned silently, which is why the crate panics
|
||||
instead. A wider weight (`i128`) moves the boundary and does not remove it. The ADR's
|
||||
"abelian group" wording needs amending whichever policy is chosen: checked-and-panic gives
|
||||
a group only within range, wrapping gives a group whose reads can be wrong, and `i128` is
|
||||
partial too. Which one is a ruling for the user, not for this step; the crate currently
|
||||
panics, and
|
||||
`overflow_depends_on_the_order_of_evaluation_and_never_gives_a_wrong_value` states it.
|
||||
`sub` is checked directly against the held weight, so `a - a` is zero for every `a`, `i64::MIN`
|
||||
included, and only an absent key's `0 - w` can overflow (`w` is `i64::MIN`).
|
||||
6. **Stable error text.** The panic message names the operation and both operands, so a
|
||||
failure in a long-running host is diagnosable from the log.
|
||||
|
||||
## What the tests are
|
||||
|
||||
`crates/zset/tests/properties.rs` holds one checker per property, parameterised by an `Alg`
|
||||
trait. The property's test runs it on `ZSet`. A companion runs it on `Model(Fault::None)`, a
|
||||
second implementation over a sorted vector, so the checker is shown not to reject a correct
|
||||
implementation that is not `ZSet`. Each rival is `Model(Fault::X)`, run through the same
|
||||
checker with a deterministic seed, and the failure message must carry the property's tag
|
||||
(`[inverse]`, `[map]`, ..). A rival that fails for another reason does not count. The group
|
||||
laws, canonical form, scalar distributivity, the `map` homomorphism, `filter` linearity,
|
||||
`distinct`'s definition, iteration order, the clause 9 correction identity and the
|
||||
`integrate`/`differentiate` inverses are all properties; each runs at the default 256 cases
|
||||
over keys from a range of six, so collisions and cancelling pairs are the common case.
|
||||
Weights come from `-3..=3`, an explicit zero, and `±2^40`, so sums of three cannot wrap.
|
||||
|
||||
The overflow policy is not a property over the generators (they cannot reach it by design)
|
||||
and is tested by example in `src/lib.rs`, with `should_panic(expected = ..)`.
|
||||
|
||||
## The circuit (improvise-bw4p.1)
|
||||
|
||||
Clause 1's deferred scope, built without ADR 008. The ADR is not edited; this is how the crate reads it.
|
||||
|
||||
| item | what it is |
|
||||
|---|---|
|
||||
| `Circuit` | creation-order list of boxed nodes; `step()` runs each once |
|
||||
| `Input<K>`, `Stream<ZSet<K>>`, `Output<K>` | push a delta in; the value on an edge; a sink that sums every step's delta until `take()` |
|
||||
| `delay`, `integrate`, `differentiate`, `plus` | the stream operators as nodes |
|
||||
| `map`, `filter`, `flat_map`, `index_by` | linear, per-element |
|
||||
| `distinct` | incremental, state is the integrated input (`DistinctState`) |
|
||||
| `join_indexed`, `join_indexed_in` | bilinear delta join of two `ZSet<(K, _)>` (`JoinState`) |
|
||||
| `aggregate`, `aggregate_in` | group-by, re-folds only touched keys (`AggregateState`) |
|
||||
| `Trace`, `MemTrace` | operator state behind lookup, range, apply |
|
||||
| `Rebuild`, `Progress` | a snapshot loaded as a resumable job |
|
||||
| `begin`, `commit`, `rollback` | an undo-log transaction over the circuit's state (below) |
|
||||
|
||||
Decisions, where the ADR left a choice:
|
||||
|
||||
1. **Wiring is apart from logic.** `circuit.rs` is the only file with the arena and
|
||||
`Box<dyn Node>`. `ops.rs` holds each stateful operator as a plain struct with a pure
|
||||
`step(delta) -> delta`, testable and usable without a circuit.
|
||||
2. **Scheduling is creation order.** A node is built from streams that already exist, so
|
||||
creation order is a topological order, and the circuit has no cycles: the ADR has no
|
||||
recursion operator, and `integrate` is its own stateful node, not a delayed feedback loop.
|
||||
`delay` is a node with one held Z-set. `Send`, not `Sync`: one thread steps a circuit at a
|
||||
time (clause 1; "Threads" below).
|
||||
3. **Incremental means the integral of the output is the operator of the integral of the
|
||||
input.** That is the property every operator is tested against, over random delta streams.
|
||||
`distinct` and `aggregate` are not linear, so they keep the integrated input and emit the
|
||||
difference of the operator at the touched keys only.
|
||||
4. **`index_by` is `map` to `(key, element)`.** A keyed stream is a `ZSet<(K, V)>`; there is no
|
||||
separate indexed collection type, so the join and the aggregate read the same shape.
|
||||
5. **The join** is `da x b_prev + a_prev x db + da x db`, computed as `da` against the right
|
||||
trace, `da` applied to the left, then `db` against the updated left (which holds the cross
|
||||
term), then `db` applied. Output weights are products and panic on `i64` overflow like
|
||||
every weight (interpretation 5). `f(&K, &A, &B) -> O` builds the output key.
|
||||
6. **`aggregate`'s fold gets the key and the group's `(value, weight)` pairs** (owned, because
|
||||
a paged trace cannot lend references) and returns `Option<R>`; `None` means the group has no
|
||||
row (an empty group, or a `Min` of nothing). Sum, count, min and max are closures; text
|
||||
unanimity and the ADR's `ExactSum` belong to the caller (clause 4, 8), not this crate.
|
||||
7. **The trace interface** is `apply(k, v, w)`, `lookup(k)` and `range(lo, hi)`, returning
|
||||
owned vectors. `MemTrace` is a `BTreeMap<K, ZSet<V>>`; the `_in` constructors take any
|
||||
`Trace`, which is the seam for paging or compression. An empty or inverted range reads
|
||||
nothing (`BTreeMap::range` would panic).
|
||||
8. **The resumable step takes a predicate, not an `Instant`, and a load is one logical tick.**
|
||||
`Rebuild::step(circuit, expired)` loads `chunk` keys at a time and stops when `expired()` is
|
||||
true, always running at least one chunk first, so every call progresses. It reports `Done`
|
||||
as soon as the snapshot is exhausted, whatever `expired()` would say. `Instant::now()`
|
||||
panics on wasm32 (`web-time` is a dependency this crate refuses), so the host closes over
|
||||
its own clock. The first chunk is `Circuit::step`; the rest are `step_continue`, which does
|
||||
not advance the clock. A circuit tick that arrives in pieces is exact for the incremental
|
||||
operators because their outputs are deltas. For the temporal ones the pieces are shaped so
|
||||
the outputs still sum to the one-step output: a continued `delay` emits nothing and adds to
|
||||
what it holds, a continued `integrate` emits only its piece. An empty snapshot still takes
|
||||
its one tick; a finished job takes none. Deltas pushed to the same input between calls
|
||||
are absorbed into the next piece. Buffering live deltas during a rebuild (clause 7) is the
|
||||
host's.
|
||||
|
||||
**`current()` is the whole tick.** A stream keeps the latest piece, which downstream
|
||||
operators read, and the whole value of the tick, which `Circuit::current(&s)` reads: pieces
|
||||
within one tick are absorbed into it, so `current` of an integrate stream after a chunked load is the
|
||||
full integral and a continued `delay` still reads the previous tick's input. `Output`
|
||||
sums pieces as before.
|
||||
|
||||
**A pending rebuild owns the clock.** `Circuit::step` counts ticks, and a `Rebuild` records
|
||||
the count after its first chunk. If a `Circuit::step` runs before the job is Done (the host
|
||||
stepped between calls, or two jobs interleaved), the next `Rebuild::step` panics with
|
||||
"the rebuild was interrupted"; `Rebuild::interrupted` asks without panicking, so a host
|
||||
can refuse first. `step_continue` calls made by another job are not counted and are not
|
||||
detected. The state a load leaves for the next tick is tested by pushing another delta
|
||||
after Done and stepping twice.
|
||||
9. **Not built:** `Circuit` does not own a job runner (ADR 008's), and nothing here is
|
||||
serialised (clause 9, `improvise-protocol`).
|
||||
|
||||
The tests are `crates/zset/tests/circuit.rs`; the rivals and where they failed are in
|
||||
`context/test-evidence/zset.md`.
|
||||
|
||||
## Transactions (improvise-bw4p.5)
|
||||
|
||||
A panic inside `step` (a weight overflow in an operator, say) leaves nodes ahead of the panic
|
||||
updated and the rest not. ADR 009 clause 11 says each command snapshots the circuit for
|
||||
rollback; `Circuit::begin`, `commit` and `rollback` are that, and the ADR is not amended (the
|
||||
clause's wording stands; this is how the crate reads it).
|
||||
|
||||
1. **An undo log, not a snapshot.** The ADR budgets memory, and a clone of the circuit's state
|
||||
is the size of the circuit. `begin` instead makes every stateful node start logging, and
|
||||
the log holds what the steps touch: the cost of a transaction is the keys a step touches,
|
||||
not the circuit's size (`a_transaction_costs_the_keys_a_step_touches_and_not_the_size_of_the_circuit`
|
||||
measures it in key clones: 16 extra for `begin; step; commit`, 22 for `rollback`, the same at
|
||||
10 and at 300 keys of state, with an `integrate` in the pipeline). Outside a transaction
|
||||
nothing is logged and nothing extra is cloned. Two costs are the circuit's size, and are
|
||||
limits, not features:
|
||||
- **A split tick.** A `step_continue` that is the first publish of an edge in a transaction
|
||||
clones that edge's whole value (2009 key clones against 1006 outside one, at N=1000;
|
||||
`a_split_tick_in_a_transaction_costs_the_size_of_the_edge`).
|
||||
- **Memory held until `commit`.** Every edge's previous value is moved into the slot's undo
|
||||
record and kept, so an `integrate -> delay` pipeline holds 4022 live keys during a
|
||||
transaction against 2014 without one (N=1000). An integrate edge is back to three copies
|
||||
of its integral for the length of the transaction.
|
||||
2. **What is logged, per node.** Each integrated Z-set (`integrate`, `distinct`, a sink's sum)
|
||||
keeps `(key, weight)` for every addition it applied; each trace (the join's two, the
|
||||
aggregate's groups) keeps `(key, value, weight)` for every `apply`, through `Undoable`, a
|
||||
wrapper over the caller's `Trace`; a `Delay` keeps one clone of what it holds on its first
|
||||
eval, which is the previous tick's input; a `Source` keeps a clone of what was pending at
|
||||
`begin`; an edge (`Slot`) keeps the `(piece, whole)` its first publish replaced, moved and
|
||||
not cloned, except for a continued piece, which clones the edge once. `ticks` is restored.
|
||||
3. **Rollback replays newest first.** Each undo is the exact inverse of an addition that
|
||||
succeeded, so it never leaves `i64`, but only in this order: a key at `MAX` lowered and
|
||||
raised again would reach `MAX + 1` undone oldest first
|
||||
(`rollback_undoes_newest_first_so_no_undo_leaves_i64`). `i64::MIN` has no negation; a
|
||||
trace undoes it as `MAX` then `1` (the weight before was non-negative, so neither leaves
|
||||
`i64`), and a Z-set undoes it by `adjust(.., subtract)`.
|
||||
4. **An entry is built before its change and logged after it.** An overflow panics before a
|
||||
key changes (`insert` is one checked addition), and the entry's clones are made first, so
|
||||
a panicking `Clone` changes nothing either (`c1_a_clone_panic_anywhere_in_a_step_is_rolled_back`
|
||||
sweeps a panic over every key clone in a step). A step that dies half way has therefore
|
||||
logged exactly the keys it applied. A custom `Trace` must do the same: a panicking `apply`
|
||||
leaves the trace as it was. `MemTrace` does, since it makes every clone before it touches
|
||||
its map (`a_panicking_value_clone_leaves_a_memtrace_unchanged`).
|
||||
5. **The edge's two copies.** An integrate stream's piece is also its output, so a second copy
|
||||
(the whole-tick value) only exists while a tick is split. `Slot::whole` is `None` otherwise
|
||||
and `Circuit::current` reads the piece. An eval reads its input's piece in the arena by
|
||||
reference rather than cloning it (improvise-bw4p.11). An integrate stream now holds its integral twice (the node's `acc`
|
||||
and the stream's piece), not three times; sharing those two needs shared ownership in the
|
||||
slot and was not done.
|
||||
6. **What a transaction does not undo.**
|
||||
- A closure's captured state (a `map` function that counts, an FFI callback's side
|
||||
effects). Operators that keep state keep it in the crate's own types for this reason.
|
||||
- A `Rebuild`: it owns an iterator over its snapshot, so a rolled-back chunk leaves the job
|
||||
ahead of the circuit. The clock cannot tell: `rollback` rewinds `ticks`, which can make
|
||||
a dead job's recorded tick current again. So the circuit also counts the rollbacks that
|
||||
undid a step or piece (`undone`, never rewound), and a job records the count when it first
|
||||
runs, before its first chunk can panic. The chunk's `push` is marked as a step too, since it
|
||||
can overflow the pending delta before any step runs (`p1_a_chunk_whose_push_panics_in_a_txn_is_counted`). A later count makes `interrupted` true for good,
|
||||
finished or not (`rollback_rebuild.rs`: begun before the transaction and continued inside,
|
||||
finished inside, a first chunk that panicked, and a dead job revived by a later step).
|
||||
- A `Circuit::take` of an output made inside the transaction: a rollback subtracts the step's
|
||||
output from the sink, which a `take` has emptied. Take after `commit`; `zset-ffi` refuses to take
|
||||
while a transaction is open.
|
||||
7. **Pushes.** `Circuit::push` is not logged. `rollback` puts the
|
||||
pending delta back to what it was at `begin`, so a push made after `begin` is dropped, and
|
||||
one made before it is pending again. Operators built after `begin` start logging at once
|
||||
and go back to empty; they are not removed.
|
||||
8. **One transaction at a time, and misuse panics** like an overflow does: `begin` while open,
|
||||
`commit` or `rollback` with none open. `zset-ffi` answers with a code instead.
|
||||
|
||||
The tests are `tests/rollback.rs` (the property: a step that panics half way, then a rollback,
|
||||
leaves every stream and sink as before and the later ticks equal to a circuit that never ran
|
||||
the step) and `tests/cost.rs`. A step that panics outside a transaction still leaves the
|
||||
circuit unusable: drop it and build another.
|
||||
|
||||
## Rebuilds and circuits (improvise-bw4p.12)
|
||||
|
||||
A `Rebuild` takes its circuit's identity from its `Input`: every handle a circuit makes holds the
|
||||
circuit's token (a process-wide counter since the arena, `Rc::ptr_eq` before), so
|
||||
`Rebuild::belongs_to(&circuit)` is a comparison. `Rebuild::step` on another circuit panics, where it used to report
|
||||
`Done` and land its chunk in that circuit's next tick. An interrupted job (decision 8 above) can
|
||||
never finish: `interrupted` stays true however many steps follow, because a split tick cannot
|
||||
be mended and a rollback that took a chunk is counted forever (decision 6 above). Drop it and start another job from the same snapshot; the host keeps the
|
||||
snapshot.
|
||||
|
||||
## Threads (user ruling 2026-10-11; ADR 009 clause 1 and section 9)
|
||||
|
||||
The graph is network-agnostic and multi-threaded from the start, and **multi-threading is
|
||||
scheduling**: a circuit stays synchronous and single-threaded, and a host moves whole circuits
|
||||
between threads. Shared-nothing: one circuit per runnable unit, joined only by edges (the edge
|
||||
adapter, a later PR, is the one thing that crosses threads). Splitting one circuit's keys across
|
||||
threads (data parallelism) is not done and not precluded: its exchange step would be the same
|
||||
edge. This is a ruling recorded here, not an ADR amendment. Native threads are assumed; wasm
|
||||
threads are out of scope, and nothing in `zset` spawns one. `Send` and `mpsc` compile on wasm32.
|
||||
|
||||
1. **`Circuit` is `Send` and not `Sync`, with no lock anywhere in it.** It moves between
|
||||
threads, one at a time (`tests/threads.rs`: a circuit built on one thread, stepped on a new
|
||||
thread each tick, equals a circuit that never moved; a transaction begun on one thread is
|
||||
rolled back on another). Not `Sync` is pinned by `a_circuit_is_not_sync`, an ordinary test
|
||||
using the ambiguous-impl trick: it fails to build (E0283) if `Sync` is ever implemented.
|
||||
2. **The `Rc<RefCell>` graph is an arena.** What made the circuit `!Send` was the sharing
|
||||
between the circuit and the handles it hands out. A circuit now owns every edge's slot, every
|
||||
input's pending delta and every sink's sum in one `Vec<Box<dyn Any + Send>>`; a node holds
|
||||
typed ids, and `eval`, `begin`, `commit` and `rollback` take the arena. A downcast on every
|
||||
access is the price. Measured on an M-series Mac under load, a release build of a 1001-node
|
||||
chain (map, filter, distinct, delay) over 3000 ticks, best of 5, alternating with main: main
|
||||
127k-131k ticks/s, this 118k-122k, so 7-9% slower per tick (about 1 ns per node per tick).
|
||||
The reviewer saw 10-25% on a more loaded run. Not reduced.
|
||||
3. **Handles are plain ids, used through the circuit.** `Stream`, `Input` and `Output` are
|
||||
`Copy`, `Send` and `Sync`, and carry their circuit's token. The calls moved onto the circuit:
|
||||
`Circuit::current(&stream)`, `Circuit::push(&input, delta)`, `Circuit::take(&output)`. A
|
||||
handle from another circuit panics ("handle from another circuit"), where before it was
|
||||
quietly wrong. A consequence: a closure can no longer read a stream during a step, since it
|
||||
has no circuit; the old `b1_...` case does not compile, so it is gone. The handle types are
|
||||
invariant in their element (`PhantomData<fn(T) -> T>`), as main's `Rc<RefCell<T>>` was, so a
|
||||
stream of `for<'a> fn(&'a u8)` cannot be coerced into a stream of `fn(&'static u8)`, where
|
||||
`current` would panic on a failed downcast. Nine call sites check the token, and
|
||||
`every_call_that_takes_a_handle_refuses_a_foreign_one` has one case for each.
|
||||
4. **User closures are `Send + 'static`;** keys and values are `Send`. A custom `Trace` handed to
|
||||
`join_indexed_in` or `aggregate_in` must be `Send`. `Piecewise` requires `Send + 'static`.
|
||||
5. **The sink does not clone.** Its node reads the input edge and writes the sum through
|
||||
`Arena::read_write`, two disjoint entries (the read one is always allocated first).
|
||||
`a_sink_adds_its_input_with_one_clone_per_key` pins it over `Counted` keys.
|
||||
6. **Handing a circuit to a thread is the host's business** (`std::thread` lives in host crates).
|
||||
Whoever owns the circuit pushes, steps and takes; a handle is a number and needs no thread.
|
||||
|
||||
## Edges (user ruling 2026-10-11; ADR 009 section 9)
|
||||
|
||||
`crates/zset/src/edge.rs`, std only. An edge joins two circuits that share nothing: the
|
||||
producer's `Output` feeds an `EdgeSink`, the consumer's `Input` is fed by an `EdgeSource`, and a
|
||||
`Pipe` carries `Frame`s between them. A channel and a network are the same kind of edge: the
|
||||
channel edge is `Direct` over `ChannelPipe`, a network edge is a codec over a socket pipe.
|
||||
Nothing here spawns a thread; whoever owns each circuit calls `pump` and `poll`.
|
||||
|
||||
1. **The clock is a counter the sender increments per committed command.** `Stamp { epoch, seq }`:
|
||||
`epoch` names the sender's incarnation and is passed in, never read from a clock (the host
|
||||
uses a random id, as `Host::with_instance` does). `delivery`'s `(host instance,
|
||||
document_generation, publication)` maps to `(epoch, seq)` with the first two folded into
|
||||
`epoch`; a delta is a notice with its `(from, to]` range. Delivery is not changed.
|
||||
2. **Only committed deltas cross.** `EdgeSink::committed_from(circuit, &output)` takes the output
|
||||
after `Circuit::commit` and panics inside a transaction (`Circuit::in_transaction`). A rolled
|
||||
back command is never taken, so it consumes no stamp
|
||||
(`a_rolled_back_command_does_not_cross`). An empty command is still a stamp, as in delivery.
|
||||
3. **Backpressure coalesces exactly.** The sink keeps one waiting delta `(sent, seq]`. A command
|
||||
adds to it (Z-set sum) and moves its `to`; `pump` sends it and, if the pipe says `Full`, leaves
|
||||
it to grow. Nothing is dropped and the range has no gap
|
||||
(`a_full_pipe_coalesces_exactly_and_keeps_the_range`).
|
||||
4. **Order and gaps.** The receiver applies a delta only if it is in the epoch it joined and
|
||||
`from == watermark`. `to <= watermark` is a duplicate and changes nothing. Anything else (a
|
||||
gap, an overlap, a frame that does not decode) is a *mark*: `stale` is set and a counter of
|
||||
troubles moves. `stale` makes the next `poll` send a `Request`.
|
||||
5. **A receiver joins an incarnation only by the answer to its latest request.** A `Request`
|
||||
carries a token (the receiver's counter, which `EdgeSource::restart` keeps), and the sender
|
||||
echoes the token of the last request it read in its `Correction { epoch, through: seq, token,
|
||||
target: integral }`. The receiver pushes `target - held` (delivery's rule). This carries over
|
||||
delivery's third fix, "the host that answers a request is live" (`delivery.md` section 4): a
|
||||
frame from an incarnation the receiver has not joined only marks it stale and never moves the
|
||||
watermark, unless it is the answer to the request just sent. A reply to an older request, a
|
||||
duplicate, or a late reply from a dead sender is therefore never a join. A correction from the
|
||||
joined incarnation applies when it is ahead of the watermark, whoever asked. There is no shared
|
||||
log as in delivery, so every answer is a full correction. Two earlier designs were wrong, each
|
||||
found by a property: joining by a first delta with `from == 0` (a restarted sender with a
|
||||
non-empty integral at `seq 0` looks fresh), and a `left` list of incarnations (a late reply
|
||||
from an incarnation never joined sent the receiver back and shut the live sender out:
|
||||
`a_late_reply_from_a_dead_incarnation_changes_nothing`). There is no `left` list now.
|
||||
6. **What clears `stale`.** Only an answer, and only if it is not behind the watermark
|
||||
(`an_answer_older_than_the_watermark_does_not_clear_stale`) and no mark has been made since
|
||||
its request went out (`trouble_seen_after_a_request_keeps_its_answer_from_clearing_stale`);
|
||||
an answer at the watermark says nothing newer exists
|
||||
(`an_answer_at_the_watermark_says_nothing_newer_and_clears_stale`).
|
||||
7. **A restart is a new epoch, and an epoch is never reused.** `EdgeSink::restart(epoch, seq,
|
||||
integral)` keeps the pipe and codec. The first frame of an unjoined epoch makes the receiver
|
||||
re-ask once for that epoch, because the request may have died with the sender that read it
|
||||
(`a_new_epoch_frame_re_asks_once_after_a_lost_request`). A sender that reuses an epoch id at
|
||||
a lower seq cannot be told from a reordered old frame, so the receiver ignores it as a
|
||||
duplicate and is stranded until the sender restarts again: the caller must pass a fresh epoch.
|
||||
A receiver that restarts (`EdgeSource::restart`) is stale again and keeps its token counter.
|
||||
A brand-new receiver (`EdgeSource::new`, a process that lost its state) seeds the counter from
|
||||
a hash of a process-wide counter under `RandomState`: random across processes natively and
|
||||
unique per process on wasm32, which has no entropy. Seeding at 0 let a dead receiver's late
|
||||
reply match a new receiver's request and join an older incarnation
|
||||
(`review_token_collision_after_new_source`).
|
||||
8. **A lost final message is invisible to the receiver.** `connection_lost()` is the host's
|
||||
timeout: it marks the receiver stale and lets it ask again. The host calls it on its own local
|
||||
monotonic timeout, never by comparing times across machines, and it must be longer than the
|
||||
round trip, or every answer is one request behind; a host that times out repeatedly must back
|
||||
off, not retry at a fixed short interval. Requests carry the receiver's stamp and
|
||||
this sender does not read it.
|
||||
9. **The wire form is a hook.** `EdgeCodec<K>` turns a `Frame<K>` into `Wire` and back, with
|
||||
`CodecError` for what does not decode; `Direct` is the identity. `improvise-protocol` is to
|
||||
implement it for its own key types in the lossless text of ADR 005, and `zset` stays free of
|
||||
serde. The tests run every history through a hand-written text codec
|
||||
(`the_same_histories_through_a_text_codec`).
|
||||
10. **Threads.** `EdgeSink` and `EdgeSource` are `Send` when the pipe and codec are. The real
|
||||
two-thread test steps a producer on one thread and a consumer on another with a capacity-2 channel
|
||||
(`a_circuit_on_each_of_two_threads_equals_one_circuit`). Two properties simulate arbitrary
|
||||
interleavings: `tests/edge.rs` over FIFO pipes that lose frames, and `tests/edge_net.rs` over a
|
||||
network that drops, duplicates, garbles and reorders in-flight frames both ways, restarts either
|
||||
end at any seq and times out (capacity 1 to 4). It checks after every action that the
|
||||
receiver's applied state is the sender's integral at the stamp it claims, and that it never
|
||||
moves to an older incarnation.
|
||||
`poll` and `committed_from` panic inside a transaction: a rollback drops what a poll pushed,
|
||||
and the stamps would not know (`a_poll_inside_a_transaction_is_refused`).
|
||||
`committed` checks overflow before it changes anything, so a caught panic leaves the sink as
|
||||
it was. `pump` clones the waiting delta each attempt against a full pipe; that is left.
|
||||
Native threads are assumed; `mpsc` and `Send` compile on wasm32 and nothing runs there.
|
||||
11. **Not built.** A sink with a log to answer a watermark with only the moved keys (delivery's
|
||||
`Moved` correction). A pipe that blocks or notifies, because `try_send` and `try_recv` are
|
||||
polling. Key-sharded data parallelism inside a circuit: its exchange step would be this same
|
||||
edge, and nothing here forbids it.
|
||||
|
||||
11. **Progress is counted.** `EdgeSink::traffic` and `EdgeSource::traffic` count frames sent and read and requests sent and read, so a scheduler can tell that a turn moved something (`improvise-incremental.md`, "Scheduling").
|
||||
@@ -1,509 +0,0 @@
|
||||
# Improvise Design Principles
|
||||
|
||||
## 1. Functional-First Architecture
|
||||
|
||||
### Commands Are Pure, Effects Are Side-Effectful
|
||||
|
||||
Every user action flows through a two-phase pipeline:
|
||||
|
||||
1. **Command** (`Cmd` trait) — reads immutable context, returns a list of effects.
|
||||
The `CmdContext` is a read-only snapshot: model, layout, mode, cursor position.
|
||||
Commands never touch `&mut App`. All decision logic is pure.
|
||||
|
||||
2. **Effect** (`Effect` trait) — a small struct with an `apply(&self, app: &mut App)` method.
|
||||
Each effect is one discrete, debuggable state change. The app applies them in order.
|
||||
|
||||
This separation means:
|
||||
- Commands are testable without a terminal or an `App` instance.
|
||||
- Effects can be logged, replayed, or composed.
|
||||
- The only place `App` is mutated is inside `Effect::apply`.
|
||||
|
||||
### Prefer Transformations to Mutation
|
||||
|
||||
Where possible, build new values rather than mutating in place:
|
||||
- `CellKey::with(cat, item)` returns a new key with an added/replaced coordinate.
|
||||
- `CellKey::without(cat)` returns a new key with a coordinate removed.
|
||||
- Viewport positioning is computed as a pure function (`viewport_effects`) that
|
||||
returns a `Vec<Effect>`, not a method that pokes at scroll offsets directly.
|
||||
|
||||
### Compose Small Pieces
|
||||
|
||||
Commands compose via `Binding::Sequence` — a keymap entry can chain multiple
|
||||
commands, each contributing effects independently. The `o` key (add row + begin
|
||||
editing) is two commands composed at the binding level, not a monolithic handler.
|
||||
|
||||
### Decompose rather than early return
|
||||
|
||||
Early Returns usually are a signal of mixed responsibilities: if an early return
|
||||
would clarify a function, consider how the function could be decomposed for the
|
||||
same effect without the early return.
|
||||
|
||||
---
|
||||
|
||||
## 2. Polymorphism Over Conditionals
|
||||
|
||||
### Dispatch Through Traits and Registries, Not Match Blocks
|
||||
|
||||
- **Commands**: 40+ types each implement `Cmd`, organized by concern across
|
||||
submodules in `command/cmd/` (navigation, cell, commit, grid, mode, panel,
|
||||
search, text_buffer, tile, effect_cmds). A `CmdRegistry` maps names to
|
||||
constructor closures. Dispatching a key press looks up the binding, resolves
|
||||
the command name through the registry, and calls `execute`. No central
|
||||
`match command_name { ... }` block.
|
||||
|
||||
- **Effects**: 50+ types each implement `Effect`. Collected into a `Vec<Box<dyn Effect>>`
|
||||
and applied in order. No `match effect_kind { ... }`.
|
||||
|
||||
- **Keymaps**: Each mode has its own `Keymap` (a `HashMap<KeyPattern, Binding>`).
|
||||
Mode dispatch is one table lookup, not a nested `match (mode, key)`.
|
||||
|
||||
### Use Enums to Make Invalid States Unrepresentable
|
||||
|
||||
- `BinOp` is an enum (`Add | Sub | Mul | ...`), not a string. Invalid operators
|
||||
are caught at parse time, not silently ignored at eval time.
|
||||
|
||||
- `Axis` is `Row | Column | Page | None`. A category is on exactly one axis.
|
||||
Cycling is a four-state rotation — no boolean flags, no "row_or_column" ambiguity.
|
||||
|
||||
- `Binding` is `Cmd | Prefix | Sequence`. The keymap lookup returns one of these
|
||||
three shapes; dispatch pattern-matches exhaustively.
|
||||
|
||||
- `CategoryKind` is `Regular | VirtualIndex | VirtualDim | VirtualMeasure | Label`.
|
||||
Business rules (e.g., the 24-category limit counts only `Regular`) are
|
||||
enforced by matching on the enum, not by checking name prefixes. Virtual
|
||||
categories (`_Index`, `_Dim`, `_Measure`) always exist: `_Index` and `_Dim`
|
||||
support drill-down/records mode; `_Measure` holds numeric data fields and
|
||||
formula targets (added automatically by `add_formula`). Use
|
||||
`Model::regular_category_names()` when selecting a default category for
|
||||
prompts or other user-visible choices.
|
||||
|
||||
### When You Add a Variant, the Compiler Finds Every Call Site
|
||||
|
||||
Prefer exhaustive `match` over `if let` or `_ =>` wildcards. When a new `Axis`
|
||||
variant or `AppMode` is added, non-exhaustive matches produce compile errors
|
||||
that guide you to every place that needs updating.
|
||||
|
||||
---
|
||||
|
||||
## 3. Correctness by Construction
|
||||
|
||||
### Canonical Forms Prevent Equivalence Bugs
|
||||
|
||||
`CellKey::new()` sorts coordinates by category name. Two keys that name the same
|
||||
intersection but in different order are identical after construction. Equality,
|
||||
hashing, and storage all work correctly without callers needing to remember to
|
||||
sort. Property tests verify this invariant.
|
||||
|
||||
### Smart Constructors Enforce Invariants
|
||||
|
||||
- `CellKey::new()` is the only way to build a key — it always sorts.
|
||||
- `Category::add_item()` deduplicates by name and auto-assigns IDs via a private
|
||||
counter. External code cannot fabricate an `ItemId`.
|
||||
- `Model::add_category()` checks the category limit (`MAX_CATEGORIES`, 24)
|
||||
before insertion.
|
||||
- `Formula::new()` takes all required fields; there is no default/empty formula
|
||||
to accidentally leave half-initialized.
|
||||
|
||||
### Type-Safe Identifiers
|
||||
|
||||
`CategoryId` and `ItemId` are typed aliases. While they are `usize` underneath,
|
||||
using named types signals intent and prevents accidentally passing an item count
|
||||
where an item ID is expected.
|
||||
|
||||
### Symbol Interning for Data Integrity
|
||||
|
||||
`DataStore` interns category and item names into `Symbol` values (small copyable
|
||||
handles). This means:
|
||||
- String comparison is integer comparison — fast and allocation-free.
|
||||
- A secondary index maps `(Symbol, Symbol)` pairs to cell sets, enabling O(1)
|
||||
lookups for aggregation queries.
|
||||
- Symbols can only be created through the `SymbolTable`, so misspelled names
|
||||
produce a distinct symbol rather than silently matching a wrong cell.
|
||||
|
||||
### Parse-Time Validation
|
||||
|
||||
Formulas are parsed into a typed AST (`Expr` enum) at entry time. If the syntax
|
||||
is invalid, the user gets an error immediately. The evaluator only sees
|
||||
well-formed trees — it does not need to handle malformed input.
|
||||
|
||||
### Grammar-Defined File Format
|
||||
|
||||
The `.improv` file format is defined by a PEG grammar (`persistence/improv.pest`)
|
||||
and parsed by pest. The grammar is the single source of truth — the parser is a
|
||||
tree-walker over the grammar's parse tree, not an ad-hoc line scanner. This means:
|
||||
|
||||
- Adding a new format feature means updating the grammar first, then the walker.
|
||||
- The grammar can be read as a specification independent of the Rust code.
|
||||
- A grammar-walking test generator reads the grammar AST at test time (via
|
||||
`pest_meta`) and produces random valid files, ensuring the parser accepts
|
||||
everything the grammar describes.
|
||||
|
||||
### CL-Style Pipe Quoting for Names
|
||||
|
||||
Names in the `.improv` format use CL-style `|...|` pipe quoting. A name is bare
|
||||
if it matches `[A-Za-z_][A-Za-z0-9_-]*`; everything else must be pipe-quoted.
|
||||
Escapes inside pipes: `\|` (literal pipe), `\\` (backslash), `\n` (newline).
|
||||
This convention is shared between the `.improv` persistence format and the
|
||||
formula parser's identifier syntax.
|
||||
|
||||
### Formula Identifiers and Quoting
|
||||
|
||||
**Bare identifiers** are `(ASCII_ALPHA | "_") ~ (ASCII_ALPHANUMERIC | "_" | "-")*`
|
||||
— no spaces, the same shape `improv.pest`'s `bare_name` has. A multi-word name
|
||||
such as `Total Revenue` must be pipe-quoted.
|
||||
|
||||
**Keywords need a word boundary after them, not before.** `IS`, `WHERE` and the
|
||||
aggregate names are spelled `^"…"` in `formula.pest`, and `is_expr`/`where_clause`/
|
||||
`inline_where` are compound-atomic with `sp+` so that `Region ISLAND` and
|
||||
`SUM(Rev WHEREAS x = y)` are parse errors rather than silently-wrong filters. A
|
||||
`SUM`/`IF` without parens falls through to `ref_expr`, which is why a bare
|
||||
keyword still names an item; the writer quotes it anyway (`quote::quote_name`).
|
||||
|
||||
**Pipe-quoted identifiers** (`|...|`) allow any characters — including spaces,
|
||||
keywords, and operators — inside the delimiters. Use pipes when a category or
|
||||
item name collides with a keyword or contains special characters:
|
||||
|
||||
```
|
||||
|WHERE| — category named "WHERE"
|
||||
|Revenue (USD)| — name with parens
|
||||
|Cost + Tax| — name with operator chars
|
||||
SUM(|Net Revenue| WHERE |Region Name| = |East Coast|)
|
||||
```
|
||||
|
||||
`identifier` is `pipe_quoted | bare_ident`, so a pipe-quoted name works
|
||||
everywhere a bare one does: expressions, aggregate arguments, and a `WHERE`
|
||||
clause's category name. A double-quoted string is a `filter_value` only — the
|
||||
right-hand side of a `WHERE` or an `IS`, never an expression.
|
||||
|
||||
---
|
||||
|
||||
## 4. Separation of Concerns
|
||||
|
||||
### Four Layers
|
||||
|
||||
| Layer | Directory | Responsibility |
|
||||
|-------|-----------|----------------|
|
||||
| **Model** | `src/model/` | Categories, items, groups, cell data, formulas. Pure data, no rendering. |
|
||||
| **View** | `src/view/` | Axis assignments, page selection, hidden items, layout computation. Derived from model. |
|
||||
| **Command / Effect** | `src/command/`, `src/ui/effect.rs` | Intent (commands) and state mutation (effects). Bridges user input to model changes. |
|
||||
| **Rendering** | `src/draw.rs`, `src/ui/` | Terminal drawing. Reads model + view, writes pixels. No mutation. |
|
||||
|
||||
### Formulas Are Data, Not Code
|
||||
|
||||
A formula is a serializable struct: raw text, target name, category, AST, optional
|
||||
filter. It is stored in the model alongside regular data. The evaluator walks the
|
||||
AST at read time. Formulas never become closures or runtime-generated code.
|
||||
|
||||
### Formula Evaluation Is Fixed-Point
|
||||
|
||||
`recompute_formulas(none_cats)` iterates formula evaluation until values
|
||||
stabilize. Each pass evaluates all formula cells using the current cache
|
||||
(for formula-derived values) and raw data aggregation (for data values).
|
||||
This avoids recursive evaluation through `evaluate_aggregated` and
|
||||
naturally handles chained formulas (`Margin = Profit / Revenue` where
|
||||
`Profit = Revenue - Cost`). Circular references converge to
|
||||
`CellValue::Error("circular")` after `MAX_EVAL_DEPTH` iterations.
|
||||
|
||||
### Display Rounding Is View-Only
|
||||
|
||||
Number formatting (`format_f64`) rounds for display. Formula evaluation always
|
||||
operates on full `f64` precision. The rounding function is only called in
|
||||
rendering paths — never in `eval_formula` or aggregation.
|
||||
|
||||
### Drill State Isolates Edits
|
||||
|
||||
When editing aggregated (drill-down) cells, a `DrillState` snapshot freezes the
|
||||
current cell set. Pending edits accumulate in a staging map. On commit,
|
||||
`ApplyAndClearDrill` writes them all atomically. On cancel, the snapshot is
|
||||
discarded. No partial writes reach the model.
|
||||
|
||||
---
|
||||
|
||||
## 5. A Front End Displays State; It Does Not Reimplement Behaviour
|
||||
|
||||
**The goal is to minimise the amount of new code a new UI framework needs.**
|
||||
Logic is centralised in a redux-like store, so a second or third front end is
|
||||
mostly a function from state to pixels plus a wire for events — not a second
|
||||
implementation of how the application behaves.
|
||||
|
||||
The pieces already exist and are worth naming as what they are:
|
||||
|
||||
| redux | here |
|
||||
|---|---|
|
||||
| action | `Command` / `ModelCommand` (`improvise-protocol`) |
|
||||
| reducer | `reduce_view`, and `WidgetCache::apply` per widget |
|
||||
| store | `ViewState` + the per-widget caches |
|
||||
| dispatch | `Sessions::apply_checked` — the one door (gotcha 36) |
|
||||
|
||||
So the question to ask of any code in a front end is: **would a third front end
|
||||
have to write this again?** If yes, it is behaviour and it belongs in the shared
|
||||
layer. Three things make this concrete rather than aspirational:
|
||||
|
||||
**The measure is lines of framework-specific code, and it should trend to
|
||||
"paint and wire".** `web/app.js` is the current benchmark: a few hundred lines
|
||||
that know no protocol, reaching the application through a handful of string
|
||||
calls. A front end that needs to understand `CellKey` coordinates, or which
|
||||
category a row label belongs to, or what a keypress means, is one that has been
|
||||
handed a job the store should have done.
|
||||
|
||||
**A derivation done twice is the defect, even when both copies are correct
|
||||
today.** Two front ends deriving the same fact will diverge, and the divergence
|
||||
is invisible until someone compares them side by side. `improvise-edp` is the
|
||||
worked example — the category tree was flattened twice, two different ways, and
|
||||
the fix put `viewmodel::flatten_cat_tree` where both ends call it. Gotcha 44
|
||||
was the same defect and is now closed the same way (improvise-anr9): the grid's
|
||||
*frame* — header levels, label tuples, group runs and their collapse state —
|
||||
is `improvise_protocol::AxisFrame`, and both `src/ui/grid.rs` and
|
||||
`improvise-wasm-client/src/render.rs` ask it rather than deriving one. Note
|
||||
what the second cure cost that the first did not: a `Vec<String>` of labels
|
||||
had to change **type**, and a shared derivation could not be added beside the
|
||||
old one — `row_levels` next to `row_labels` would have been two answers to
|
||||
"what are the labels", which is the disease and not the cure.
|
||||
|
||||
**Sharing the table beats testing two copies of it.** Where a front end must
|
||||
turn input into actions, share the mapping rather than mirroring it: two
|
||||
hand-kept copies held in step by a test are one definition waiting to be made
|
||||
(`doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished)`).
|
||||
That is why the keymap is shared rather than reimplemented per front end, and
|
||||
why `doctrine(mirror_the_cmd_line_for_line)` is a rule about *resolvers* — the
|
||||
narrow, unavoidable residue — and never a licence to restate a table.
|
||||
|
||||
**What legitimately stays per-framework.** Painting, event wiring, and the
|
||||
mapping from a semantic value to a framework type: a `Face`, resolved by the
|
||||
shared `Theme`, becomes a `ratatui::Style` in `draw.rs::face_style` (ADR 004). That mapping is the shape
|
||||
to imitate — the *decision* is in the shared layer as a semantic enum, and only
|
||||
the last translation is local. Anything larger than a lookup table is a
|
||||
candidate for the store.
|
||||
|
||||
---
|
||||
|
||||
## 6. Guidelines for New Code
|
||||
|
||||
- **Add a command, not a special case.** If you need new behavior on a keypress,
|
||||
implement `Cmd`, register it, and bind it in the keymap. Do not add an
|
||||
`if key == 'x'` branch inside `handle_key`.
|
||||
|
||||
- **Return effects, do not mutate.** Your command's `execute` receives `&CmdContext`
|
||||
(immutable). Produce a `Vec<Box<dyn Effect>>`. If you need a new kind of state
|
||||
change, create a new `Effect` struct.
|
||||
|
||||
- **Use the type system.** If a value can only be one of N things, make it an enum.
|
||||
If an invariant must hold, enforce it in the constructor. If a field is
|
||||
optional, use `Option` — do not use sentinel values.
|
||||
|
||||
- **Test the logic, not the wiring.** Commands are pure functions of context;
|
||||
test them by building a `CmdContext` and asserting on the returned effects.
|
||||
You do not need a terminal.
|
||||
|
||||
- **Keep `Option`/`Result`/`Box` at the boundaries.** Core logic should work with
|
||||
concrete values. Wrap in `Option` at the edges (parsing, lookup, I/O) and
|
||||
unwrap early. Do not thread `Option` through deep call chains.
|
||||
|
||||
- **Reasoning lives in `context/`, not in comments.** A comment run is **at most
|
||||
two consecutive lines**, `///` and `//!` included, enforced by
|
||||
`scripts/check-comment-runs.py`. Where a reader would otherwise re-derive a
|
||||
rule wrongly, name the `context/` file that carries the argument — and only
|
||||
there, because a pointer on every function is noise. The prose lives in
|
||||
`context/code/<area>.md`; the rule exists because the arguments in this
|
||||
codebase are its most expensive artefact and a comment is the one place
|
||||
nothing checks them against the code (user, 2026-09-20; improvise-9c8c).
|
||||
A ``` fence in a doc comment is a doctest — a running test, not prose — and
|
||||
is exempt.
|
||||
|
||||
- **And an argument must not live only in a commit message.** A message is
|
||||
write-once and unauditable: it cannot be corrected without rewriting merged
|
||||
history, and no check in this repo can read one — the ID audit
|
||||
(`AGENTS.md`, "Cite a ticket ID you have read back") greps files in the
|
||||
tree, and a commit message is in none of them. So a reason recorded only there is a reason nothing can ever find wrong.
|
||||
When one turns out to be wrong, correct `context/` and say in `context/` that
|
||||
the message is superseded; that is the fix rather than a compromise. PR #51's
|
||||
`7d25762` is the worked example, corrected by #56.
|
||||
|
||||
---
|
||||
|
||||
## 7. Testing
|
||||
|
||||
### Coverage and ambition
|
||||
|
||||
Aim for **~80% line and branch coverage** on logic code. This is a quality floor —
|
||||
go higher where the code is tricky or load-bearing, but don't pad coverage by
|
||||
testing trivial getters or chasing 100% on rendering widgets. The test suite
|
||||
should remain fast, and the rule is **two numbers** because the ambiguity
|
||||
between them is what let the old single "under 2 seconds" drift: **no single
|
||||
test binary may exceed 2 seconds**, and the workspace wall clock is the sum of
|
||||
them. Excludes compilation, and excludes `harness/`, which is Playwright and
|
||||
takes minutes in its own CI job. Slow tests erode the habit of running them.
|
||||
|
||||
**The threshold is a tripwire, and the seconds are not portable.** The same
|
||||
commit measured `improvise-io`'s lib binary at 1.4s on one machine and
|
||||
3.7–5.2s on another (improvise-8ec3), so a PR body citing "longest single
|
||||
binary 1.47s" is checkable and *false* for the reader who checks it, which is
|
||||
worse than no claim at all. Three consequences:
|
||||
|
||||
- **Re-measure; never re-assert.** `cargo test --workspace` prints a
|
||||
`finished in` per binary. Read your own run.
|
||||
- **If you cite a figure, name the machine you took it on.** A number with no
|
||||
machine is a property of the suite, which it is not.
|
||||
- **The portable signal is a ratio, not the seconds — and not the ratio to
|
||||
the median.** Most binaries print `0.00s` at two decimals, so the median is
|
||||
zero and a ratio to it is undefined (improvise-i5z6, 39 binaries). Compare
|
||||
the slowest binary with the next few slowest, across several runs: an
|
||||
outlier is one that stays well clear of them *in every run*. Both outliers so
|
||||
far were found that way and had one cause, a generator re-parsing its pest
|
||||
grammar per case: `improvise-io`'s lib (improvise-8shu) and
|
||||
`improvise-formula`'s lib (longest of 39 in all three i5z6 runs).
|
||||
|
||||
**improvise-8shu is closed, and its diagnosis was wrong** — which is the more
|
||||
useful half. The ticket blamed an unbounded grammar walk and asked for a
|
||||
constrained strategy. Instrumenting the two halves separately: `load_grammar()`
|
||||
×1000 took **2.50s** against a **15.7ms** walk, so the walk was **0.6%** of the
|
||||
cost and `improv_file()` was re-parsing `improv.pest` once per generated case.
|
||||
Caching it in a `OnceLock` took `improvise-io`'s lib from **1.42s to 0.21s**
|
||||
with the strategy's input space *unchanged*; constraining it would have moved
|
||||
15.7ms and shipped as "the fix". Measured at `f152d10` on the orchestrator's
|
||||
machine: **1445 tests over 34 binaries**, longest single binary **0.6–1.0s
|
||||
across two runs** — the spread on one machine in one sitting being its own
|
||||
argument for reading your own run. `improvise-io` is no longer the outlier.
|
||||
Lowering the case count below 256 is still not on the table (see below).
|
||||
|
||||
### Demonstrate bugs before fixing them
|
||||
|
||||
Write a test that **fails on the current code** before writing the fix. Prefer a
|
||||
small unit test targeting the broken function over an end-to-end test. After the
|
||||
fix, the test stays as a regression guard.
|
||||
|
||||
**Record the bug in `context/test-evidence/`, keyed by test name** — what broke,
|
||||
what fault was injected, and why the test can fail. It does not go in the test's
|
||||
doc-comment: comment runs are capped at two lines (see below), and the evidence
|
||||
is longer than that in every case worth having. The cost of the split is that a
|
||||
renamed test orphans its entry. `scripts/check-comment-runs.py --check-evidence`
|
||||
catches that, in two directions. Every `###`/`####` heading shaped like a test
|
||||
name must name a test that **runs** — the names are diffed against
|
||||
`cargo test --workspace -- --list` minus its `--ignored` twin, never scraped out
|
||||
of the source, because text that looks like a test and a test are
|
||||
indistinguishable to a scanner (improvise-rm4s). And a test whose doc comment
|
||||
*cites* `context/test-evidence/` must have an entry there, or the pointer reads
|
||||
as a promise the explanation exists.
|
||||
|
||||
Neither direction is "every test has evidence", deliberately: most tests have
|
||||
none and should not be forced to have one. The reasoning and the residual holes
|
||||
are in `context/code/tooling.md`.
|
||||
|
||||
### A check that looks like the answer, answering a neighbouring question
|
||||
|
||||
The characteristic defect in this repo is not a wrong line; it is a check that
|
||||
is *adjacent* to the thing it names. Three instances landed in one campaign
|
||||
(PRs #59/#60), in three different layers, which is why it is worth stating as a
|
||||
shape rather than three gotchas:
|
||||
|
||||
- **A vacuous assertion.** `set_panel_open_and_cursor` asserted
|
||||
`category_panel_open` was true after an effect set it true — once the launch
|
||||
default supplied true, the effect could be replaced by a **no-op** and 1391
|
||||
tests passed. It answered *is the flag true* where it meant *did the effect
|
||||
set it*.
|
||||
- **A vacuous repair of it.** The first fix folded `[open:false, open:true]`
|
||||
and asserted the final state — which a reducer that folds nothing also
|
||||
satisfies. It answered *does the text mention the other direction* where it
|
||||
meant *does the writer move the flag both ways*. **Reading the diff is what
|
||||
failed**, and it failed because the vacuous version contained `open: false`.
|
||||
- **A clean auto-merge.** `git merge-tree` reporting no conflict answers *can
|
||||
git combine these texts*, where the question is *does the combination do what
|
||||
both sides meant*. Two PRs edited the same four files with no conflict and
|
||||
neither side CI-tested together.
|
||||
|
||||
Each has the same remedy shape: **name the question you actually mean, then
|
||||
find an input that separates it from its neighbour.**
|
||||
|
||||
- For an assertion: **assert the direction the default cannot supply.** Closing
|
||||
the panel discriminates; opening it proves nothing.
|
||||
- For a repair: inject a **one-directional** fault (`if open { set true }` —
|
||||
can raise, never lower). A vacuous repair cannot see it, because a test that
|
||||
only asserts the true direction is satisfied by a writer that only ever
|
||||
writes true.
|
||||
- For a merge: **materialise it and run the gates**, and check that a count
|
||||
which should MOVE actually moved. Playwright going 16 → 17 is what showed the
|
||||
other side survived; "no conflict" and an unchanged total would both have
|
||||
been consistent with losing it.
|
||||
|
||||
This generalises `doctrine(fold_tests_must_test_order_not_the_axiom)` —
|
||||
asserting `fold(prefix)+fold(rest) == fold(whole)` is the definition of a left
|
||||
fold, so it answers a neighbouring question too. What the campaign adds is that
|
||||
the pattern recurs in **test repairs** and in **merge hygiene**, not only in
|
||||
production code and not only in fold tests.
|
||||
|
||||
### E2E pins the boundaries; unit tests work between them
|
||||
|
||||
**E2E and integration tests are rare and pin *boundary contracts*. Unit tests
|
||||
*assume* those contracts and test between the boundaries** (user, 2026-09-20).
|
||||
|
||||
So the question to ask of a slow test is never "what feature does this cover" —
|
||||
it is **"which boundary is this, and what is the contract at it?"** The answer
|
||||
changes what you write. Four browser widgets had never worked — the tile bar,
|
||||
the view panel, the formula panel, and the category sidebar ignoring
|
||||
`category_panel_open` — and all four were **one** boundary failure: a
|
||||
`RenderModel` field with no element painting it. Four per-widget Playwright
|
||||
specs cost four slow tests and would still have missed the fifth; a single
|
||||
contract assertion that *every field the render model exposes is painted*
|
||||
catches all four and the next one. `harness/tests/opfs.spec.js` is the cleanest
|
||||
instance of the right shape already in the tree: it runs the **storage
|
||||
contract**, not a feature.
|
||||
|
||||
Three corollaries, in the order they bite:
|
||||
|
||||
- **A unit test may assume the contract rather than re-establishing it.** That
|
||||
assumption is what makes it cheap, and it is only sound because something
|
||||
slow pins the contract — so deleting the contract test silently invalidates
|
||||
every unit test that leaned on it.
|
||||
- **Never weaken a contract to make it cheap.** If the honest contract needs a
|
||||
real browser, a real `Worker` or real OPFS, it stays slow. Moving it to a
|
||||
faster harness that cannot see the thing it names is how a suite goes green
|
||||
over a broken product.
|
||||
- **Each slow test must say which contract it pins.** The difference between a
|
||||
contract test and a feature test is invisible from the code, so it has to be
|
||||
written down or the next person adds the fifth per-widget spec.
|
||||
|
||||
### Use property tests judiciously
|
||||
|
||||
Property tests (`proptest`) are for **invariants that must hold across all
|
||||
inputs** — not a replacement for example-based tests. Good candidates:
|
||||
|
||||
- Structural invariants: CellKey is always sorted, each category lives on exactly
|
||||
one axis, toggle-collapse is involutive, hide/show roundtrips.
|
||||
- Serialization roundtrips: save/load identity.
|
||||
- Determinism: `evaluate` returns the same result for the same inputs.
|
||||
|
||||
Keep case counts at the default (256). Don't crank them to thousands — if a
|
||||
property needs more cases to feel safe, constrain the input space with better
|
||||
strategies rather than brute-forcing. Property tests that take hundreds of
|
||||
milliseconds each are a sign something is wrong.
|
||||
|
||||
### What to test
|
||||
|
||||
- **Model, formula, view**: the core logic. Unit tests for each operation and
|
||||
edge case. Property tests for invariants. These are the highest-value tests.
|
||||
- **Commands**: build a `CmdContext`, call `execute`, assert on the returned
|
||||
effects. Pure functions — no terminal needed. Tests are colocated in each
|
||||
command submodule (`command/cmd/<module>.rs` → `mod tests`), with shared
|
||||
test helpers in `command/cmd/mod.rs::test_helpers`.
|
||||
- **Persistence**: round-trip tests (`save → load → save` produces identical
|
||||
output) plus grammar-driven property tests. The generator walks the pest
|
||||
grammar AST to produce random valid files; proptests verify
|
||||
`parse(generate())` succeeds and `parse(format(parse(generate())))` is
|
||||
stable. Cover groups, formulas, views, hidden items, pipe quoting edges.
|
||||
- **Format**: boundary cases for comma placement, rounding, negative numbers.
|
||||
- **Import**: field classification heuristics, CSV quoting, multi-file merge.
|
||||
|
||||
### What not to test
|
||||
|
||||
- Ratatui `Widget::render` implementations — pure drawing code that changes
|
||||
often. Test the data they consume (layout, cat_tree, format) instead.
|
||||
- Trivial data definitions (`ast.rs`, `axis.rs`).
|
||||
- Module re-export files.
|
||||
|
||||
### Test the property, not the implementation
|
||||
|
||||
A test like "calling `set_axis(cat, Row)` sets the internal map entry to `Row`"
|
||||
is brittle — it mirrors the implementation and breaks if the storage changes.
|
||||
Instead test the observable contract: "after `set_axis(cat, Row)`,
|
||||
`axis_of(cat)` returns `Row` and `categories_on(Row)` includes `cat`." This
|
||||
style survives refactoring and catches real bugs.
|
||||
@@ -1,449 +0,0 @@
|
||||
# File Inventory (the long form)
|
||||
|
||||
The per-file listing that `context/repo-map.md` summarises. Moved here unchanged.
|
||||
|
||||
Line counts are a **dated snapshot**, not a maintained fact: measured 2026-09-24 at `c1772ca9`, and **70 of 105 had drifted past 10%** in the three days since the 2026-09-21 refresh — eleven had doubled or more (`client.rs` 255 → 943, `lockstep.rs` 1126 → 4064). Treat them as "is this file big", never as a figure to cite; re-derive with `wc -l`. The `ws-server/` and `wasm-client/` entries use a shorthand prefix, so a refresh must map it to `crates/improvise-<name>/{src,}/` by hand or they drift silently. Test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted.
|
||||
|
||||
### `improvise-core` (`crates/improvise-core/src/`)
|
||||
```
|
||||
model/types.rs 3560 / 112t Model, formula eval (fixed-point), CRUD
|
||||
model/cell.rs 1358 / 47t CellKey (sorted), CellValue, DataStore (interned + index),
|
||||
live_records (insertion order — gotcha 18)
|
||||
model/record.rs 226 / 8t RecordHash / CellId surrogates: the opaque identities a
|
||||
thin client names a cell by (ADR 002)
|
||||
model/category.rs 310 / 9t Category, Item, Group, CategoryKind
|
||||
model/symbol.rs 89 / 3t Symbol interning
|
||||
view/layout.rs 2604 / 57t GridLayout, drill, records mode, records_columns; a private
|
||||
per-axis position index (`row_pos`/`col_pos`, rebuilt by `reindex`)
|
||||
so a data position resolves in O(1) (improvise-felo.13)
|
||||
view/types.rs 817 / 43t View config (axes, pages, hidden, collapsed, format)
|
||||
model/id.rs 290 slugify + IdAllocator: how a generated id is made (ADR 007
|
||||
clause 4, core.md §20). Nothing outside the file calls it yet
|
||||
view/axis.rs 190 / 5t Axis enum + label/next methods (no colour: that is a Face)
|
||||
workbook.rs 413 / 22t Workbook: Model + cross-view ops, item facade
|
||||
format.rs 519 / 40t format_f64, parse_number_format (display only)
|
||||
```
|
||||
|
||||
### `improvise-formula` (`crates/improvise-formula/src/`)
|
||||
```
|
||||
parser.rs 1394 / 79t pest grammar + tokenizer → Formula AST
|
||||
quote.rs 160 / 7t quote_name / is_bare_name / pipe_quote / unescape_pipe_body:
|
||||
the one home, both halves (improvise-i2m5). improvise-gp
|
||||
re-exports it; improvise-io calls it from
|
||||
persistence/{format,parse}.rs
|
||||
formula.pest 108 PEG grammar for formula text
|
||||
ast.rs 130 Expr, BinOp, AggFunc, Formula, Filter
|
||||
```
|
||||
|
||||
### `improvise-io` (`crates/improvise-io/src/`)
|
||||
```
|
||||
storage.rs 334 / 13t Storage trait (read/write/list/delete), PhysicalFs, MemoryFs,
|
||||
`impl Storage for Rc<T>` (improvise-hmws), and
|
||||
contract::storage_contract — public, not cfg(test), because
|
||||
a browser must run that sequence (harness/)
|
||||
opfs.rs 289 / 0t OpfsFs: Storage over the browser's Origin Private File
|
||||
System. wasm32 only. Run by harness/, not by cargo test
|
||||
persistence/improv.pest 179 PEG grammar — single source of truth
|
||||
persistence/mod.rs 441 / 11t document lifecycle: save/load/gzip/legacy-JSON over Storage;
|
||||
facade re-exporting format_md/parse_md/export_csv*
|
||||
persistence/format.rs 401 / 11t format_md: Workbook → .improv text
|
||||
persistence/parse.rs 1684 / 46t parse_md: pest tree-walk → Workbook (+ grammar generator)
|
||||
persistence/export.rs 243 / 5t export_csv / render_csv. Builds its layout with
|
||||
with_frozen_records, NOT new — `new` skips prune_empty
|
||||
(improvise-iot6)
|
||||
persistence/roundtrip_tests.rs
|
||||
1236 / 62t format_md ↔ parse_md characterization (tests only)
|
||||
import/wizard.rs 1967 / 62t ImportPipeline + ImportWizard. `Plan::place` is the one
|
||||
placement pass, shared by `collisions()` and the build so
|
||||
the two cannot disagree; `DuplicatePolicy` / `Collision`
|
||||
(improvise-cgqt)
|
||||
import/analyzer.rs 434 / 17t Field kind detection (Category/Measure/Time/Skip);
|
||||
analyze_records is first-seen order, not HashSet (gotcha 18)
|
||||
import/csv_parser.rs 590 / 26t CSV parsing, multi-file merge
|
||||
```
|
||||
|
||||
### `improvise-protocol` (`crates/improvise-protocol/src/`)
|
||||
```
|
||||
lib.rs 619 / 2t ViewState, ViewCursor, ViewFrame, AppMode (17), Panel,
|
||||
MinibufferConfig, MinibufColor, HELP_PAGE_COUNT
|
||||
reduce.rs 749 / 10t Command + ModelCommand (the wire format), HelpPage, reduce_view
|
||||
render.rs 2157 / 46t CellDisplay, WidgetCache trait + one cache per widget
|
||||
(GridCache/CategoryTreeCache/FormulaCache/ViewListCache/
|
||||
TileBarCache), AxisFrame/FrameEntry/FrameGroup and
|
||||
header_level_widths (the grid frame, gotcha 44),
|
||||
GridCache::seed (the bootstrap compaction), Projection,
|
||||
bootstrap-only Snapshot
|
||||
keymap.rs 975 / 0t KeyPattern, Binding, Keymap, ModeKey (17), TextEntry,
|
||||
PanelKeymap, Keymaps::default_keymaps/prefix_keymap — the
|
||||
shared table both front ends look up. Tests live with
|
||||
dispatch in improvise-command/src/command/keymap.rs
|
||||
viewmodel.rs 170 / 5t flatten_cat_tree — derivations between a cache and a
|
||||
renderer, called by both front ends (improvise-edp)
|
||||
face.rs 682 Face (a role: Axis, WizardField, Mode*, MinibufferOf…), Theme,
|
||||
AnsiSlot — appearance as data a front end applies (ADR 004,
|
||||
protocol.md §10); draw.rs::face_style is the TUI's one translation
|
||||
key.rs 96 / 3t KeyCode, KeyModifiers, KeyEvent (moved from command/)
|
||||
frame.rs 156 / 3t ClientFrame/ServerFrame — the ONE envelope pair, both
|
||||
deployments (improvise-xks6). Nothing else defines one.
|
||||
ClientFrame::Save is a request, not a Command (gotcha 45)
|
||||
basis.rs 257 / 6t Basis (the three tiers), Rejection, BasisCache — ADR 001
|
||||
layer 1. Basis has no Default and no public constructor
|
||||
help.rs 1241 / 14t HelpPage content and Hint { keys: Vec<(KeyCode, KeyModifiers)>,
|
||||
command, caption }. `compact` drops a chord by its MODIFIERS
|
||||
(anything beyond shift), never by parsing a label it just
|
||||
formatted (improvise-thpe) — the guard is the signature
|
||||
resolve.rs 4883 / 88t the browser's key resolvers: a BrowserKey to the Commands it
|
||||
composes, mirroring each Cmd line for line INCLUDING where
|
||||
siblings disagree (gotcha 47). The largest file in the crate
|
||||
client.rs 943 / 15t ClientState: ViewState + one cache per widget, and the fold;
|
||||
in_flight, `opened` (OpenedCell), notify, settle_status
|
||||
(gotcha 47). EditorSession is in lib.rs beside ViewState
|
||||
```
|
||||
|
||||
### `gp-tree` / `improvise-gp` (`crates/gp-tree/`, `crates/improvise-gp/`)
|
||||
```
|
||||
gp-tree/src/lib.rs 575 / 8t Tree, Spec, eval, simplify, render, breed, Population,
|
||||
SplitMix64 (vendored seeded Rng; seed() keeps the seed so
|
||||
a fit can be shown and recorded, pinned to the published
|
||||
reference vectors)
|
||||
gp-tree/src/data.rs 1000 / 21t raw string rows -> f32 feature vectors: per-column type
|
||||
inference, one-hot, mean-imputation + col.missing flag,
|
||||
ColumnSpec/encode_row, Dataset (incl. warnings),
|
||||
Model = trees + plan + seed + FitConfig
|
||||
gp-tree/src/fit.rs 961 / 32t the search: FitConfig (Default = the CLI's numbers), fit()
|
||||
-> simplified champion, cv_regress/cv_classify,
|
||||
permutation_importance, folds/standardize/mse.
|
||||
Returns data; never prints
|
||||
gp-tree/examples/fit_csv.rs 583 CLI over the above: flags, demo dataset, all printing
|
||||
gp-tree/tests/scoring_known_answers.rs
|
||||
250 / 6t f32 bit patterns + a next_u64 witness: the evidence any
|
||||
change to the scoring paths needs (gotcha 37c)
|
||||
improvise-gp/src/lib.rs 978 / 25t formula_text / fitted_formula: a Tree as IF(...) formula
|
||||
text, validated back through parse_formula. A threshold is
|
||||
the SHORTEST decimal that reads back as the same f64 (not
|
||||
the exact expansion), so a row on a split boundary routes
|
||||
the same in f64 as in f32 eval; non-finite values panic
|
||||
improvise-gp/src/dataset.rs 506 / 15t a View's records -> the feature matrix a fit consumes —
|
||||
the improvise-side counterpart of preprocess. Records mode
|
||||
is already a feature matrix (gotcha 1)
|
||||
improvise-gp/src/fit.rs 1778 / 35t one fit end to end: a View's records in, an improvise
|
||||
Formula and the evidence for believing it out. Not in a
|
||||
command, because none of it is terminal-specific
|
||||
```
|
||||
|
||||
### Command layer (`crates/improvise-command/src/command/`)
|
||||
```
|
||||
cmd/core.rs 475 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers
|
||||
cmd/registry.rs 903 / 13t default_registry() — all registrations
|
||||
cmd/navigation.rs 507 / 12t Move, EnterAdvance, Page*
|
||||
cmd/cell.rs 724 / 20t ClearCellCommand, YankCell, PasteCell, TransposeAxes, SaveCmd
|
||||
cmd/commit.rs 1304 / 27t CommitAndAdvance, CommitBuffer(CommitKind), CommitExport,
|
||||
CommitSave
|
||||
cmd/effect_cmds.rs 519 / 6t effect_cmd! macro, 25+ simple wrappers, `unless_unknown` (gotcha 51)
|
||||
cmd/fit.rs 293 / 8t fit-measure: evolve a tree over the active view and store it
|
||||
as an ordinary `## Formulas` line — nothing GP-specific is
|
||||
left in the document afterwards
|
||||
cmd/grid.rs 988 / 22t ToggleGroupAtCursor, ViewNavigate, DrillIntoCell, TogglePruneEmpty
|
||||
cmd/mode.rs 469 / 13t EnterMode, Quit, EditOrDrill, EnterTileSelect
|
||||
cmd/panel.rs 936 / 24t Panel toggle/cycle/cursor, formula/category/view panels
|
||||
cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
|
||||
cmd/text_buffer.rs 399 / 14t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
|
||||
cmd/tile.rs 170 / 6t MoveTileCursor, TileAxisOp
|
||||
cmd/wizard.rs 244 / 6t HandleWizardKey — per-step import-wizard key tables
|
||||
keymap.rs 1353 / 44t KeymapSet, Keymap::dispatch/dispatch_transient, effects_of,
|
||||
SetTransientPrefix — what a thin client cannot have. The
|
||||
table itself is improvise-protocol/src/keymap.rs
|
||||
parse.rs 350 / 23t Script/command-line parser (prefix syntax)
|
||||
```
|
||||
|
||||
### Semantic UI state (`crates/improvise-command/src/ui/`)
|
||||
```
|
||||
ui/effect.rs 4020 / 99t Effect trait. 63 effects: 53 `impl Effect for` blocks, 9
|
||||
simple_effect! expansions, and SetTransientPrefix in
|
||||
command/keymap.rs. The 10 outside the brace-match are all
|
||||
ViewOnly; 28 of the 53 are Model(_) (gotcha 38, h3y7)
|
||||
ui/app.rs 5558 / 134t App, ModelState, ViewState + dispatch methods, Panel, handle_key, dispatch_key, document_generation, autosave; apply_effects declares `dirty` for WholeWorkbook and observes it for CellData/LayoutGeometry (gotcha 45)
|
||||
ui/project.rs 6383 / 118t Dirty, Viewport, grid_of/category_rows/formula_texts/
|
||||
view_summaries/tile_entries, grid_updates (the diff),
|
||||
ClientSession + Sessions (connect/apply/rebase/project_all)
|
||||
ui/reduce.rs 1217 / 24t Checked / CheckedBatch / apply_command / model_effects /
|
||||
reduce_key — one door from a Command or a keypress to the
|
||||
authoritative App. `reduce_full` is GONE (improvise-h3y7)
|
||||
ui/host.rs 3410 / 67t Host: one App + one Sessions; handle, commands, project_all,
|
||||
rebase. Lives here, not in ws-server (gotcha 45)
|
||||
ui/cat_tree.rs 174 / 8t Category tree flattener for panel
|
||||
ui/viewport.rs 259 compute_col_widths / compute_row_header_width / visible_cols
|
||||
(pure geometry; the width rule itself is the protocol's);
|
||||
grid_width, SIDE_PANEL_WIDTH, MIN_GRID_WIDTH (improvise-gtn7.1);
|
||||
GridSummary / grid_summary / grid_window (improvise-felo.13)
|
||||
```
|
||||
|
||||
### TUI rendering, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`)
|
||||
```
|
||||
ui/grid.rs 905 / 14t GridWidget (ratatui): no &Model, no &View — the frame comes
|
||||
off GridCache (gotcha 44). Its module doc is the best-written
|
||||
statement of that rule in the repo
|
||||
ui/help.rs 277 5-page help overlay (render only)
|
||||
ui/import_wizard_ui.rs 829 / 9t Import wizard rendering. `Panel` is the one writer, and
|
||||
every write is clipped to the panel's rows and width
|
||||
ui/text.rs 169 truncate and wrap, both measured in terminal cells, never bytes
|
||||
draw.rs 1122 / 20t TUI event loop, frame composition; `content_tests` draws a
|
||||
whole frame to check the selection beside a side panel
|
||||
main.rs 428 CLI entry (clap): open, import, cmd, script
|
||||
# other ui/*.rs are small panel renderers — skip unless changing layout/style
|
||||
tests/grid_frame_agreement.rs
|
||||
479 / 6t the two front ends draw the same grid frame (improvise-anr9)
|
||||
tests/records_wide_smoke.rs 67 / 1t
|
||||
tests/shipped_documents_load.rs 333 / 8t every TRACKED .improv/.improv.gz loads, and every record
|
||||
it projects is writable. "Shipped" is read off `git ls-files`,
|
||||
not a .gitignore reading (improvise-c0aj), and the walk's git
|
||||
is scrubbed of every hostile GIT_* setting the caller may carry
|
||||
tests/web_demo_model.rs 83 / 2t every visible cell in web/demo.improv is a Record
|
||||
```
|
||||
|
||||
### Browser host and client (`crates/improvise-ws-server/src/`, `crates/improvise-wasm-client/src/`, `web/`)
|
||||
```
|
||||
ws-server/registry.rs 616 / 14t Registry of Hosts keyed by DocumentId; ClientId {document,
|
||||
session} — a SessionId is unique only within its document.
|
||||
A document is never evicted while the server runs (ADR 001)
|
||||
ws-server/host.rs 4 / 0t a re-export shim only. The session loop is
|
||||
improvise-command/src/ui/host.rs (gotcha 45)
|
||||
ws-server/origin.rs 132 / 6t OriginPolicy: which origins may open a socket. Loopback by
|
||||
default, --allow-origin widens, absent allowed, two refused
|
||||
(improvise-kc7n). The only decision transport::admitted makes
|
||||
ws-server/transport.rs 330 / 4t tokio + tungstenite plumbing, plus the Origin check at the
|
||||
handshake — a refused upgrade, not an ignored socket
|
||||
ws-server/main.rs 190 / 5t CLI entry: serve a file on a port; --bind, --allow-origin
|
||||
ws-server/tests/socket.rs 389 / 6t the one test that uses a real socket: handshake, JSON
|
||||
framing, SessionId routing, which document a URL asked for
|
||||
wasm-client/lib.rs 455 / 8t Client: the wasm-bindgen surface (init, on_message, on_frame,
|
||||
viewport/set_viewport, render_model, set_cell_at, typed_cell)
|
||||
wasm-client/keys.rs 187 / 5t BrowserKey ONLY — a KeyboardEvent flattened to an improvise
|
||||
key. It does NOT resolve: `resolve` and KeyOutcome live in
|
||||
improvise-protocol/src/resolve.rs, shared by both front ends
|
||||
wasm-client/render.rs 1249 / 21t RenderModel: editable / selected / cat tree, decided here
|
||||
wasm-client/socket.rs 547 / 13t the fold (fold_frame/handle_frame), the two compositions
|
||||
and typed() text→CellValue. Defines no enums
|
||||
# state.rs is GONE — ClientState moved to improvise-protocol/src/client.rs
|
||||
wasm-client/tests/browser_round_trip.rs 2136 / 34t improvise-1ey's scope against a real Host
|
||||
wasm-client/tests/lockstep.rs 4064 / 78t host and client agree on a keypress
|
||||
web/app.js 816 string calls only, no protocol knowledge, no framework.
|
||||
Run by harness/tests/shell.spec.js (Chromium; the standalone half on Firefox too) — see
|
||||
gotcha 48 for what that does and does not reach
|
||||
web/worker.js 86 three string calls, no protocol knowledge. RUN by
|
||||
shell.spec.js's no-?ws path, with bindings.rs (gotcha 45)
|
||||
web/shell.js 399 everything app.js DECIDES that is not painting: no DOM,
|
||||
no browser API, one `node --test` away. Owns socketUrl,
|
||||
gridPlan, frameCostOf, documentTitle — so a chunk of the
|
||||
browser's render decisions is NOT in the file gotcha 42
|
||||
and gotcha 46 name. Reasoning: context/code/clients.md
|
||||
web/serial.js 19 the one rule worker.js exists to keep: one frame at a
|
||||
time, in arrival order (gotcha 48's async-listener race),
|
||||
extracted so it is testable without worker globals
|
||||
web/index.html, style.css the page; legibility only
|
||||
web/build.sh 38 wasm-pack build → web/pkg (gitignored). Needs
|
||||
wasm-opt = ["-O", "--enable-bulk-memory"] or the bundled
|
||||
wasm-opt rejects LLVM's memory.copy
|
||||
web/assemble.sh 112 assembles the deployable site (copies web/*.improv)
|
||||
web/demo.improv 65 the document the site ships and seeds from. Every cell a
|
||||
Record — asserted (gotcha 46)
|
||||
```
|
||||
|
||||
### Browser worker host (`crates/improvise-worker-server/`)
|
||||
```
|
||||
improvise-worker-server/src/worker.rs 765 / 17t Worker: one port, one client; handshake, Addressed
|
||||
filter, persist trigger (model_state.dirty). Every decision, every test
|
||||
improvise-worker-server/src/vfs.rs 348 / 8t documents/open: seeding is a Storage::write
|
||||
improvise-worker-server/src/bindings.rs 185 / 0t wasm32 shim: OpfsFs awaits + postMessage. No Rust test
|
||||
reaches it, but harness/tests/shell.spec.js DOES: the
|
||||
no-?ws path runs start()/OpfsFs::open under Chromium
|
||||
and Firefox.
|
||||
Called NEVER RUN here for months: see gotcha 45
|
||||
improvise-worker-server/src/lib.rs 51 / 1t the module doc that states exactly what is and is not
|
||||
covered — the one correct copy of that fact
|
||||
improvise-worker-server/tests/post_message.rs 407 / 8t main-thread traffic, strings only
|
||||
```
|
||||
|
||||
### Browser harness (`harness/`) — not a workspace member
|
||||
|
||||
```
|
||||
harness/playwright.config.js 50 projects: chromium (every spec), firefox (opfs, plus
|
||||
shell.spec's standalone half: grepInvert @mocked-peer),
|
||||
webkit (opfs only, which skips on a runtime probe: no
|
||||
OPFS). CI runs `--project=chromium --project=firefox` only:
|
||||
nixpkgs' WebKit aborts inside the runner VMs (ci.yml)
|
||||
harness/tests/shell.spec.js 345 the first execution of web/app.js. A change to web/*.js
|
||||
belongs here (gotcha 48). Playwright serves harness/serve/,
|
||||
which harness/build.sh ASSEMBLES: edit web/*.js, rebuild,
|
||||
or the specs run the old page
|
||||
harness/tests/opfs.spec.js 103 drives the OPFS contract page and reads back its result
|
||||
harness/opfs-contract/src/lib.rs 470 runs improvise_io::storage::contract::storage_contract
|
||||
against a real OpfsFs inside a real Web Worker
|
||||
harness/www/opfs/{index.html,worker.js} the page and worker that host it
|
||||
harness/build.sh 50 assembles harness/serve/ (web/assemble.sh, the fixtures,
|
||||
and the contract crate's wasm-pack build) — gitignored
|
||||
# gated by the `browser` job in .github/workflows/ci.yml, which can fail the build
|
||||
```
|
||||
|
||||
### Examples
|
||||
```
|
||||
examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta)
|
||||
examples/pretty-print.rs Parse stdin, print formatted .improv
|
||||
crates/improvise-core/examples/category_width_bench.rs
|
||||
The same records at 12 and at 24 regular categories
|
||||
(improvise-felo.1; context/code/core.md)
|
||||
```
|
||||
|
||||
### Sources, incremental evaluation and circuits (`crates/improvise-core/src/model/`, `crates/{improvise-incremental,zset,zset-ffi,numtower}/`)
|
||||
```
|
||||
improvise-core/src/model/source.rs CellSource, ChangeSource, Select, Summary/Numbers/Agreement, BadParts, SharedSource; TableSource (test-support); `Model::over` is in model/types.rs
|
||||
improvise-core/src/model/over_tests.rs tests of a model over a source
|
||||
improvise-incremental/src/lib.rs Derived trait and re-exports
|
||||
reading.rs, delta.rs, eager.rs Reading, CellDelta/DeltaValue/cell_delta/moved_records, Eager (the oracle)
|
||||
group.rs, shapes.rs, formulas.rs maintained groups (ExactSum), shapes per category set, formulas by stem
|
||||
incremental.rs, edit.rs, fold.rs Incremental (over/sync/apply_changes), Edit, the fold
|
||||
pushdown.rs, reads.rs Pushdown over a query-answering source; reads shared with Eager
|
||||
harness.rs differential, probes, Maintained
|
||||
sched.rs, staged.rs Executor/Unit/Pool (Pool native-only); the aggregate/grid example on three units (head, aggregate, grid)
|
||||
zset/src/{circuit,ops,trace,edge}.rs Circuit (arena, Send not Sync, begin/commit/rollback), operators, Trace, edges (EdgeSink/EdgeSource/Pipe/EdgeCodec/Stamp)
|
||||
zset-ffi/src/{lib,engine}.rs the C ABI; include/zset_ffi.h (zs_circuit_begin/commit/rollback, ZS_ERR_TRANSACTION, ZS_ERR_INTERRUPTED)
|
||||
```
|
||||
|
||||
### Context docs
|
||||
```
|
||||
.claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl
|
||||
scripts/check-comment-runs.py The two-line comment cap AND --check-evidence. The flag is
|
||||
ADDITIVE since improvise-ijql — every invocation that can
|
||||
exit 0 answers the cap — guarded by a subprocess self-test,
|
||||
the only level a dispatch bug is visible at. Reasoning:
|
||||
context/code/tooling.md §2
|
||||
scripts/check-ci-graph.py pages-deploy must `needs` every job in ci.yml, and every
|
||||
suite `checkers.sh --list` knows must run in some CI job
|
||||
(improvise-dak5). It REFUSES what it cannot read rather than
|
||||
parsing YAML: the values it accepts are an allowlist
|
||||
(tooling.md). --self-test; run in the `test` job. README
|
||||
states the rule and lists no jobs (improvise-x7cq)
|
||||
scripts/checkers.sh The one entry point for every *-test.sh suite: scrubs the
|
||||
env to an allowlist, runs under the bash that runs it (no
|
||||
version floor since #297). `ci_suites` is the table of which
|
||||
suites each CI job runs (`--ci <job>`); ci.yml names none
|
||||
scripts/pr-watch Waits on a PR's checks (the pr-watch skill). Exit 5: the PR
|
||||
is already MERGED or CLOSED, so there is nothing to wait for
|
||||
scripts/assemble-stamp.sh The cache-bust stamp gate web/assemble.sh calls, extracted
|
||||
so its refusals can be exercised without a web/pkg
|
||||
scripts/assemble-stamp-test.sh Its check_the_checker, run first in the `build` CI job
|
||||
context/code/source.md CellSource, Model::over, Summary, ChangeSource (ADR 009 step 7)
|
||||
context/code/improvise-incremental.md The Derived plug: steps 3-7, Eager's math, Scheduling
|
||||
context/code/zset.md Circuit, transactions, threads, edges
|
||||
context/code/tooling.md The repository's own checkers: why an evidence heading's
|
||||
names come from cargo rather than from the source, and
|
||||
why a leaf name is not an identity
|
||||
context/test-evidence/tooling.md Evidence for the gates in scripts/ and for the tests that
|
||||
hold CI's shape
|
||||
scripts/nix-seed-flake-inputs.sh Seeds locked flake inputs over git so `nix develop` works in cloud sessions
|
||||
prolog-sessions/kb.pl Executable knowledge base: invariants, doctrine, decisions (CI-checked)
|
||||
context/adr/007-stable-identities.md Stable identities: ids unique across the document are the
|
||||
keys and names are display-only; gensym suffixes apply to
|
||||
generated ids only, and hand-written ids are canonical.
|
||||
Accepted 2026-09-25. (There is no ADR 005 or 006 file:
|
||||
improvise-l2b8 and improvise-ilvl claim those numbers.)
|
||||
context/adr/004-faces-and-themes.md Faces: appearance as UI-agnostic data a front end
|
||||
applies, emacs-style. Accepted 2026-09-22.
|
||||
context/adr/003-live-backed-models.md A model may be backed by a live query source —
|
||||
improvise as a DB client. Accepted, NOT a priority.
|
||||
context/adr/002-naming-a-cell.md How a client names a cell: an opaque surrogate on
|
||||
CellDisplay, whose absence means the cell is an
|
||||
aggregate. Records why _Index cannot serve.
|
||||
context/adr/001-concurrent-edits.md Concurrent edits between clients: record-level atomicity,
|
||||
optimistic basis check per batch, advisory reservations.
|
||||
Records why a causal clock was rejected.
|
||||
context/cloud-agents.md Cloud-session environment: tracker sync, bd pitfalls, what is blocked
|
||||
context/design-principles.md Architectural principles & testing doctrine
|
||||
context/plan.md Show HN launch plan
|
||||
context/repo-map.md The map: crates, how-to-find-things, the gotcha digest, a file map
|
||||
context/gotchas.md The long form of every numbered gotcha (the digest is in repo-map.md)
|
||||
context/file-inventory.md This file: the per-file listing the repo map summarises
|
||||
docs/design-notes.md Product vision & non-goals
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# Wasm, CLI, testing and dependencies (long form)
|
||||
|
||||
The three repo-map sections of these names, as they stood before the map was condensed.
|
||||
|
||||
## Wasm
|
||||
|
||||
The browser bundle is `improvise-core`, `improvise-formula`, `improvise-io`,
|
||||
`improvise-protocol`, `improvise-command`, `improvise-wasm-client`,
|
||||
`improvise-worker-server`, and (ADR 005, ADR 009) `numtower`, `zset` and
|
||||
`improvise-incremental`, the last three because `improvise-protocol` and
|
||||
`improvise-command` depend on them. Each
|
||||
`cargo check --target wasm32-unknown-unknown` clean with no feature flags
|
||||
(improvise-ywd); `wasmCrates` in `nix/ci.nix` lists the set the CI gate checks, and
|
||||
it also holds `improvise-gp` (improvise-z5eg), a candidate rather than a member,
|
||||
because staying wasm-clean is free now and expensive to retrofit. `zset-ffi` is
|
||||
native only and not in that list.
|
||||
**A feature that adds a dependency to a bundle crate must be checked with `cargo tree --edges normal`** — the default output includes **dev** edges, so a plain `cargo tree -p improvise-protocol --target wasm32-unknown-unknown` reports a dev-only `serde_json` as *present* and reads exactly like a leak. Check per-crate too, because the `wasm` CI job invokes each crate separately and feature unification cannot reach across that (improvise-26dm added a `test-support` feature on `improvise-protocol` on those terms; it is off in every shipped build).
|
||||
|
||||
`improvise-ws-server` is the host and is native only — tokio, tungstenite and
|
||||
`Box<dyn Storage>` — so `improvise-wasm-client` cannot depend on it in anger. It
|
||||
no longer needs to: both ends `pub use` `improvise_protocol::frame` and neither
|
||||
defines a pair (gotcha 42). The **dev**-dependency remains, for
|
||||
`tests/browser_round_trip.rs`, which hands the client's own serialized frame to
|
||||
a real `Host::handle`. The root `improvise` crate is
|
||||
the terminal binary and does **not** build for wasm — it depends on
|
||||
crossterm; that is correct, not a gap.
|
||||
|
||||
`cargo check` passing is not the same as running. Two known runtime hazards
|
||||
it cannot see: `Instant::now()` panics on wasm (hence `web-time` in
|
||||
`improvise-command`), and any `std::fs` call compiles but fails in a browser
|
||||
— improvise-fuj closed the effects that bypassed `Storage`; autosave and the headless final write still do (improvise-31oe.11).
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
improvise [model.improv] # open TUI (default)
|
||||
improvise import data.csv [--no-wizard] [-o out] # import CSV/JSON
|
||||
improvise cmd 'add-cat Region' -f model.improv # headless command(s)
|
||||
improvise script setup.txt -f model.improv # run script file
|
||||
```
|
||||
|
||||
Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`, `--split-date <field>` (a date column imports as ONE hierarchical category by default; this restores the flat split, and `--extract` still picks the components — improvise-8jk6.3), `--strict` (with `--no-wizard`, any skipped record is a refusal and nothing is written, where the default writes and warns with a count — improvise-8jk6.6), and `--on-duplicate refuse|sum|separate` (default `refuse`; applies with `--no-wizard`, where the wizard would otherwise ask — improvise-cgqt). A headless import also refuses a `--split-date` that names a non-date field and an `--axis` that names a category the import did not build (exit 1).
|
||||
|
||||
---
|
||||
|
||||
## Testing — the short version
|
||||
|
||||
Full guidance lives in `context/design-principles.md` §7. Quick reminders:
|
||||
|
||||
- **Two numbers, not one** — the ambiguity between them is what let the old
|
||||
"suite runs in <2s" drift: **no single test binary over 2 seconds**, and the
|
||||
workspace wall clock is the sum. Measured at `63a5c13`: **1464 tests over 35
|
||||
binaries** (1 ignored), longest single binary **0.6–1.0s across two runs on one machine** —
|
||||
the spread being its own argument for reading your own run. Excludes compilation,
|
||||
and excludes `harness/` (Playwright, minutes, its own CI job). *Re-measure* rather
|
||||
than re-assert — `cargo test --workspace` prints a `finished in` per binary, and
|
||||
**the portable signal is the ratio, not the seconds**. **Not the ratio to the
|
||||
median, though: it is degenerate.** Most binaries print `0.00s` at two decimals,
|
||||
so the median is zero and every ratio to it is undefined (improvise-i5z6, 39
|
||||
binaries). Compare the slowest binary to the next few slowest instead: an outlier
|
||||
is one that sits well clear of them *in every run*. That is how i5z6 was read —
|
||||
`improvise-formula`'s lib longest of 39 in all three runs, at 0.91–1.95s against a
|
||||
runner-up's 0.50–0.92s. Both outliers so far had one cause: a generator re-parsing
|
||||
its pest grammar per case. `improvise-io` (improvise-8shu, 1.42s → 0.21s) and
|
||||
`improvise-formula` (improvise-i5z6, 0.84–0.98s → 0.06–0.07s) each read it once through a
|
||||
`OnceLock` now. A new grammar-walking strategy should do the same from the start.
|
||||
The full argument is `context/design-principles.md` §7.
|
||||
- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed.
|
||||
- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability.
|
||||
- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Record the bug in `context/test-evidence/`, **keyed by test name** — *not* in the test's doc comment, which the two-line comment cap forbids and `scripts/check-comment-runs.py` enforces. A heading may be module-qualified and must be when its leaf is not unique.
|
||||
- Coverage target ~80% line/branch on logic code; skip ratatui render paths.
|
||||
|
||||
---
|
||||
|
||||
## Key dependencies
|
||||
|
||||
ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
|
||||
File diff suppressed because one or more lines are too long
-288
@@ -1,288 +0,0 @@
|
||||
# improvise — launch preparation plan
|
||||
|
||||
You are working in the `improvise` repository, a Rust terminal application that implements a Lotus Improv-style pivot-table modeling tool. Cells are keyed by sets of `(category, item)` coordinates rather than 2D grid addresses; views assign categories to row/column/page axes; formulas reference dimension names rather than cell addresses. The application is built with `ratatui` + `crossterm`, persists models in a markdown-based `.improv` format (with optional gzip), and supports both an interactive TUI and a headless command/script mode through a unified registry.
|
||||
|
||||
Your job is to prepare this repository for a public Show HN launch. The bar is "ready for strangers to install and try in 60 seconds," not "feature-complete." Do tasks in the order given. Do not start a later phase before earlier phases are complete. Do not add features, refactor warts, or restructure modules — those are explicitly out of scope and listed at the end.
|
||||
|
||||
The target audience is developers who use vim, who have opinions about data modeling, and who would understand a Lotus Improv reference. The pitch positions improvise as a pivot-table modeling tool, not "a spreadsheet in a terminal" — that framing competes with sc-im and VisiData and loses.
|
||||
|
||||
The development environment is Nix-based (the repo has a `flake.nix`). The user does not have Homebrew installed and will not install it. All tooling must be available through Nix, and any process that needs new tools must be codified as additions to `flake.nix` so the user can reach them via `nix develop` or `nix run`. Do not suggest `brew install` anywhere.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Repository hygiene
|
||||
|
||||
Goal: raise the quality floor of what a stranger sees when they land on the repo. Roughly 90 minutes of work.
|
||||
|
||||
### 1.1 Audit and remove or update `context/SPEC.md`
|
||||
|
||||
Read `context/SPEC.md` and compare it against the actual code. The spec is likely stale: it may describe a JSON import wizard when the real wizard handles CSV, and it may describe JSON persistence when the real format is markdown (look at `src/persistence/mod.rs` to confirm). It may also list commands that no longer exist in the registry.
|
||||
|
||||
If the spec contradicts the code in significant ways, delete it. If you want to preserve design intent, move salvageable conceptual content to `docs/design-notes.md` with a header that reads:
|
||||
|
||||
> **Note:** This document captures design intent and may not reflect the current implementation. The README and source code are authoritative.
|
||||
|
||||
Do not try to bring the spec back into sync with the code — that is not worth the effort for a personal project, and a stale spec is worse than no spec.
|
||||
|
||||
### 1.2 Fix `Cargo.toml` metadata
|
||||
|
||||
The current `[package]` section is missing fields that crates.io and `cargo dist` need. Update it to include:
|
||||
|
||||
- `description` — replace any generic placeholder with: `"Terminal pivot-table modeling in the spirit of Lotus Improv"`
|
||||
- `repository` — ask the user for the GitHub URL if you don't already know it; otherwise leave a `TODO` comment and flag it
|
||||
- `homepage` — same URL as `repository`
|
||||
- `documentation` — same URL as `repository` for now
|
||||
- `readme = "README.md"`
|
||||
- `keywords = ["tui", "pivot", "spreadsheet", "data", "improv"]` (max 5 keywords, each ≤20 chars)
|
||||
- `categories = ["command-line-utilities", "visualization"]`
|
||||
- `license` — confirm this is set; if not, ask the user
|
||||
|
||||
Preserve all existing fields and dependency entries.
|
||||
|
||||
### 1.3 Verify publish-readiness
|
||||
|
||||
Run `cargo publish --dry-run` from inside `nix develop`. Fix any errors or warnings. Common issues: missing license file, files too large to publish, dependencies with incompatible versions. Do not actually publish — that happens in Phase 3.
|
||||
|
||||
### 1.4 Audit CSV quote handling
|
||||
|
||||
Look at `src/import/csv_parser.rs` (or wherever CSV parsing lives). Verify that it correctly handles RFC 4180 quoted fields: fields enclosed in double quotes, embedded commas inside quoted fields, and escaped quotes (`""` inside a quoted field).
|
||||
|
||||
If the parser is using the `csv` crate from `Cargo.toml`, this should be handled correctly by default — verify that the code is actually using it and not doing manual `split(',')` anywhere. If there are any places that do manual splitting or fragile quote handling, fix them by routing through the `csv` crate. Add a unit test that round-trips a CSV row with `"O'Reilly, Inc."` (embedded comma) and `"She said ""hi"""` (embedded escaped quotes).
|
||||
|
||||
If the parser is fundamentally broken in ways that can't be fixed quickly, do not block on this. Add a one-line note to the README under a "Known limitations" section: "CSV files with unusual quoting may not parse correctly; PRs welcome."
|
||||
|
||||
### 1.5 Create `examples/demo.improv` and `examples/demo.csv`
|
||||
|
||||
Create two synthetic example files with obviously-fake data. These exist so a new user can run `improvise examples/demo.improv` and immediately see a working pivot, or `improvise --import examples/demo.csv` and walk through the wizard.
|
||||
|
||||
**`examples/demo.csv`** should contain ~30-50 rows with columns like:
|
||||
- `Date` (mix of dates across at least 2 quarters of one year)
|
||||
- `Region` (3-4 values: North, South, East, West)
|
||||
- `Product` (3-4 values: Widget, Gadget, Sprocket, Doohickey)
|
||||
- `Customer` (5-8 fake company names: Acme Corp, Globex, Initech, Umbrella, Soylent, etc.)
|
||||
- `Revenue` (round numbers, 100-10000)
|
||||
- `Cost` (round numbers, less than Revenue)
|
||||
|
||||
**`examples/demo.improv`** should be the result of importing `demo.csv` through the tool, then manually saving. To create it: build the tool, run the import on `demo.csv`, optionally pivot it into an interesting default view (e.g., Region on rows, Date_Quarter on columns, Product on page axis), add a sample formula like `Profit = Revenue - Cost` if the formula system supports it, save as `examples/demo.improv`, and commit both files.
|
||||
|
||||
If you cannot determine the exact format syntax, look at existing test fixtures or run the tool's save path on a small in-memory model to generate one.
|
||||
|
||||
The data must be obviously synthetic. Do not copy data from any other file in the repo. Do not use any real-looking names, real amounts, or anything that looks like it might be from a real bank export.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — README and demo artifacts
|
||||
|
||||
This is the main weekend's work. The README is 80% of the launch. The demo artifacts are the other 20%. Nothing in Phase 3 or 4 matters if Phase 2 is weak.
|
||||
|
||||
### 2.1 Add Nix tooling for asciinema and VHS
|
||||
|
||||
The user does not have Homebrew. Both asciinema (terminal session recording) and VHS (Charmbracelet's terminal-to-GIF tool, package name `vhs` in nixpkgs) need to be available through Nix.
|
||||
|
||||
Modify `flake.nix` to add `pkgs.asciinema` and `pkgs.vhs` to the dev shell's `nativeBuildInputs`. Both packages exist in nixpkgs unstable; verify before adding. If `vhs` requires additional runtime dependencies (it uses `ttyd` and `ffmpeg` internally), add those too — check the nixpkgs `vhs` derivation to see what's already bundled.
|
||||
|
||||
After modifying the flake, verify that `nix develop --command asciinema --version` and `nix develop --command vhs --version` both work.
|
||||
|
||||
While you're in the flake, also add:
|
||||
- `pkgs.cargo-dist` for the release tooling in Phase 3 (if it's packaged in nixpkgs; if not, fall back to running it via `cargo install` inside the dev shell and note this in a comment)
|
||||
- A `nix run` app or shell alias for the demo recording workflow (see 2.4)
|
||||
|
||||
### 2.2 Write the README
|
||||
|
||||
Replace any existing `README.md` with a new one structured as follows. Aim for under 250 lines total.
|
||||
|
||||
**Section order (do not deviate):**
|
||||
|
||||
1. **Title** — `# improvise`
|
||||
|
||||
2. **One-sentence pitch** — italicized, immediately under the title:
|
||||
> *Terminal pivot-table modeling in the spirit of Lotus Improv — multidimensional cells, formulas over dimensions instead of cell addresses, and vim-style keybindings for reassigning axes on the fly.*
|
||||
|
||||
3. **Inline animated demo** — `` (the GIF generated in step 2.5). This must be near the top so it shows in HN previews and the GitHub repo card.
|
||||
|
||||
4. **Why this exists** — exactly one paragraph. Explain the Improv data/view separation and why no terminal tool currently does it. Reference Lotus Improv (NeXT, 1991) by name. Briefly note that Excel pivot tables took the visual idea but not the formula model. Do not bash other tools — name sc-im and VisiData neutrally as adjacent-but-different.
|
||||
|
||||
5. **Quick start** — three code blocks:
|
||||
```
|
||||
nix develop
|
||||
cargo build --release
|
||||
./target/release/improvise examples/demo.improv
|
||||
```
|
||||
Followed by a one-line "or import your own CSV":
|
||||
```
|
||||
./target/release/improvise --import path/to/data.csv
|
||||
```
|
||||
|
||||
6. **Key bindings to try first** — short list, not a complete reference. `T` for tile mode (reassign axes), `[` `]` for page axis cycling, `>` to drill into a cell, `<` to return, `F` for formula panel, `:w` to save, `:q` to quit, `?` or `F1` for full help.
|
||||
|
||||
7. **Installation** — three subsections:
|
||||
- **From source with Nix** (preferred): `nix develop` then `cargo build --release`, then optionally `cargo install --path .`
|
||||
- **From crates.io**: `cargo install improvise` (will be valid after Phase 3)
|
||||
- **Prebuilt binaries**: link to GitHub releases page (will be populated after Phase 3)
|
||||
|
||||
Do not mention Homebrew. Do not mention `apt`, `dnf`, or other distro package managers.
|
||||
|
||||
8. **What's interesting about the codebase** — 10-15 lines, prose, no bullets. Cover: the multidimensional data model with categories instead of grid coordinates; the view layer as a pure function from `(Model, View)` to `GridLayout`; records mode as just another axis assignment; the command/effect architecture that lets the same registry serve both interactive dispatch and headless scripts; the markdown `.improv` persistence format that's human-readable and git-diffable.
|
||||
|
||||
9. **Expectations** — mandatory disclaimer paragraph, exact wording:
|
||||
> improvise is a personal project I built for my own use. I'm sharing it because other people might find it useful, but I can't promise active maintenance or feature development. Issues and PRs are welcome but may not get a fast response. If you want to build on it, fork away.
|
||||
|
||||
10. **License** — one line.
|
||||
|
||||
Do not add: a table of contents, badges (build status, version, license badges, etc.), a contributing guide, a code of conduct, an "inspired by" gratitude section. These are noise for a personal project launch.
|
||||
|
||||
### 2.3 Create `docs/demo.tape` and generate `docs/demo.gif`
|
||||
|
||||
VHS is scripted: you write a `.tape` file describing keystrokes and timing, and VHS produces a GIF. This GIF goes inline in the README and is the single highest-leverage artifact in the launch — it's what shows up in HN preview cards and Google search results.
|
||||
|
||||
Create `docs/demo.tape` that scripts a ~20-second flow showing the pivot reassignment killer demo. Rough script structure (consult VHS docs for exact syntax):
|
||||
|
||||
```
|
||||
Output docs/demo.gif
|
||||
Set FontSize 14
|
||||
Set Width 1000
|
||||
Set Height 600
|
||||
Set Theme "Dracula"
|
||||
|
||||
Type "improvise examples/demo.improv"
|
||||
Enter
|
||||
Sleep 1500ms
|
||||
|
||||
# Show initial pivot
|
||||
Sleep 1s
|
||||
|
||||
# Enter tile mode and reassign an axis
|
||||
Type "T"
|
||||
Sleep 800ms
|
||||
# (whatever keystrokes reassign Region from rows to columns)
|
||||
Sleep 1500ms
|
||||
|
||||
# Show the new pivot
|
||||
Sleep 1s
|
||||
|
||||
# Reassign again to demonstrate the speed
|
||||
# (more keystrokes)
|
||||
Sleep 1500ms
|
||||
|
||||
# Exit
|
||||
Type ":q"
|
||||
Enter
|
||||
```
|
||||
|
||||
Generate the GIF by running `nix develop --command vhs docs/demo.tape`. Iterate on the script until the GIF is under 5MB, the timing is readable (not too fast, not too slow), and the key beats are clear: start in pivot view → press T → axis reassigns → press T again → axis reassigns again. The viewer should be able to understand "this tool re-pivots data with one keystroke" without any narration.
|
||||
|
||||
### 2.4 Record asciinema casts
|
||||
|
||||
Asciinema produces `.cast` files that the asciinema-player JS widget can replay in a browser, with selectable text and pixel-perfect terminal rendering. These will be embedded in the GitHub Pages landing page in Phase 4.
|
||||
|
||||
Before recording, set the terminal to match what `docs/index.html` declares. That is **120 columns by 37 rows** today, not the 100x30 this plan originally said: the casts that were actually recorded are 120x37, and the landing page follows the recordings rather than the other way round, because a 30-row player clips the bottom seven rows of a 37-row cast — which is where the tile bar and the status line are. `stty cols 120 rows 37` (or resize the window manually if `stty` doesn't take). Whichever geometry is used, the two must agree: the player renders at the size it is told and a mismatch looks broken. If a re-recording changes it, change `cols`/`rows` in `docs/index.html` in the same commit (improvise-2oxt).
|
||||
|
||||
Record four casts under `docs/casts/`:
|
||||
|
||||
- **`docs/casts/import.cast`** — start with `improvise` (no args, empty model), trigger CSV import for `examples/demo.csv`, walk through the wizard accepting defaults, end in pivot view.
|
||||
- **`docs/casts/pivot.cast`** — start from `improvise examples/demo.improv`, demonstrate axis reassignment with `T`. This is the same flow as the README GIF but longer and more complete. Show 2-3 different pivots.
|
||||
- **`docs/casts/drill.cast`** — from a pivot view, press `>` to drill into an aggregated cell, show the records view, demonstrate that you can edit a record, press `<` to return.
|
||||
- **`docs/casts/formulas.cast`** — from a pivot view, open the formula panel with `F`, add `Profit = Revenue - Cost`, show it appearing across the pivot.
|
||||
|
||||
Record each with `nix develop --command asciinema rec -i 2 docs/casts/<name>.cast`. The `-i 2` flag caps idle gaps at 2 seconds, which prevents long pauses from making the playback feel dead. Each cast should be under 60 seconds.
|
||||
|
||||
If a take has flubs, delete it and re-record. Do not try to edit the JSON cast files manually.
|
||||
|
||||
Add a `nix run` app to the flake or a shell script in `scripts/record-demo.sh` that wraps the recording workflow with the right `stty` setup, so the user can re-record consistently in the future without remembering the exact incantation.
|
||||
|
||||
### 2.5 Verify all artifacts exist and are committed
|
||||
|
||||
Before moving to Phase 3, confirm:
|
||||
- `README.md` exists with all 10 sections
|
||||
- `docs/demo.gif` exists, is referenced from the README, and is under 5MB
|
||||
- `docs/demo.tape` exists and regenerates the GIF when run through VHS
|
||||
- `docs/casts/import.cast`, `pivot.cast`, `drill.cast`, `formulas.cast` all exist
|
||||
- `examples/demo.csv` and `examples/demo.improv` exist and contain only synthetic data
|
||||
- `flake.nix` includes `asciinema`, `vhs`, and `cargo-dist` (or a fallback for `cargo-dist`)
|
||||
- `nix develop` succeeds and all the tools above are on PATH
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Distribution
|
||||
|
||||
### 3.1 Configure `cargo dist`
|
||||
|
||||
`cargo dist` generates a GitHub Actions workflow that produces release tarballs and installer scripts when you push a version tag. Run `nix develop --command cargo dist init` and configure for these targets:
|
||||
|
||||
- `x86_64-unknown-linux-gnu`
|
||||
- `aarch64-apple-darwin`
|
||||
- `x86_64-apple-darwin`
|
||||
|
||||
Skip Windows. Skip musl unless `cargo dist init` strongly recommends it — the existing flake doesn't build for musl and adding that complication is out of scope.
|
||||
|
||||
Commit the generated `.github/workflows/release.yml` and the additions to `Cargo.toml`. Test the workflow by tagging `v0.1.0-rc1` and pushing the tag to a branch — verify the release builds successfully on GitHub Actions before doing a real release. Delete the rc tag afterward.
|
||||
|
||||
### 3.2 Publish to crates.io
|
||||
|
||||
After `cargo publish --dry-run` is clean (from step 1.3) and the user confirms they're ready, run `nix develop --command cargo publish`. Verify the crate appears at `https://crates.io/crates/improvise` and that `cargo install improvise` works on a clean machine (or in a fresh `nix shell --packages cargo`).
|
||||
|
||||
### 3.3 Tag the v0.1.0 release
|
||||
|
||||
Create a git tag `v0.1.0`, push it, and verify the `cargo dist` workflow produces release artifacts. Update the README's "Prebuilt binaries" link to point at the actual release.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Landing page (optional but recommended)
|
||||
|
||||
### 4.1 Create `docs/index.html`
|
||||
|
||||
Vanilla HTML, single file, under 200 lines. No framework, no build step. Dark background, monospace headings to match the terminal aesthetic. Embed the asciinema-player from jsdelivr CDN (`https://cdn.jsdelivr.net/npm/asciinema-player@3/dist/bundle/`) and reference the four `.cast` files from `docs/casts/`.
|
||||
|
||||
Section structure: title and one-line tagline → "Reassign axes on the fly" with the pivot cast → "Drill into aggregated cells" with the drill cast → "Formulas over dimensions" with the formulas cast → "Import a CSV" with the import cast → "Get it" with the install commands and a GitHub link.
|
||||
|
||||
Each cast should be embedded with `AsciinemaPlayer.create()` configured with `theme: 'monokai', autoPlay: false, loop: true`, and `cols`/`rows` set to the casts' own geometry — 120x37 as recorded, not the 100x30 this plan first specified. See 2.4.
|
||||
|
||||
### 4.2 Enable GitHub Pages
|
||||
|
||||
**Not branch-and-folder.** Pages is served from the GitHub Actions artifact that `build` / `pages-deploy` publish in `.github/workflows/ci.yml` (Settings → Pages → Source: GitHub Actions), and the tree is assembled by `web/assemble.sh`: `docs/index.html` and `docs/casts/` at the root, the browser app under `app/`.
|
||||
|
||||
Setting the source to branch `main` / folder `/docs` instead would serve `docs/` raw. That used to fail loudly — there was no `docs/index.html`, so it was a 404 — and since improvise-e61 it fails **quietly**: the landing page renders perfectly and its "Run improvise in your browser" link 404s, because `app/` exists only in the assembled tree. A page that looks right with a dead front door is worse than a page that is missing.
|
||||
|
||||
Verify the site is live at `https://<user>.github.io/improvise/` within a few minutes of pushing, and click through to the app rather than only loading the root.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Buffer
|
||||
|
||||
The second weekend (or the second half of the first weekend) is reserved for whatever was missed in Phases 1-4. Do not use it to add features. Do not use it to fix the issues listed in "Out of scope" below. If everything is done early, stop.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope — do not do these before launch
|
||||
|
||||
The following are real issues that have been identified but are explicitly not launch-blocking. Do not start any of them as part of this plan. They can be addressed after launch based on what real users actually trip over.
|
||||
|
||||
- ~~**Do not** raise or restructure the `MAX_CATEGORIES = 12` limit, even though it's tight for some real use cases.~~ **Superseded** by the user's decision of 2026-09-25 ("12 categories is feeling small, we should increase to 24"). The limit is 24 (improvise-felo.1).
|
||||
- **Do not** refactor the `ExecuteCommand` vim command match ladder, even though it duplicates registry logic.
|
||||
- **Do not** fix the hardcoded `"Measure"` category name in `evaluate_aggregated`, even though it's the one place in the code that assumes a specific user-facing name.
|
||||
- **Do not** change `Model::evaluate` from a linear formula walk to a HashMap lookup, even though it's O(N·M·F) in the render hot path.
|
||||
- **Do not** add render-pass memoization to `matching_values` calls.
|
||||
- **Do not** fix the `ApplyAndClearDrill` HashMap iteration ordering issue around coordinate rename + value edit interactions.
|
||||
- **Do not** add new features: no YoY references, no parameterized formulas, no Datalog backend, no undo/redo, no charts, no plugin system.
|
||||
- **Do not** write a documentation site beyond the single GitHub Pages landing page in Phase 4.
|
||||
- **Do not** add Windows support.
|
||||
- **Do not** rename types, restructure modules, or do any "while I'm in here" cleanups.
|
||||
|
||||
Each of these is worth doing eventually. None of them is worth doing before the launch post is up. The goal of this plan is to get the existing tool in front of strangers in a form they can install and try, not to make the tool perfect first.
|
||||
|
||||
---
|
||||
|
||||
## Definition of done
|
||||
|
||||
The plan is complete when, on a clean machine with only Nix installed:
|
||||
|
||||
1. `git clone <repo> && cd improvise && nix develop` succeeds.
|
||||
2. `cargo build --release` succeeds.
|
||||
3. `./target/release/improvise examples/demo.improv` opens an interesting pivot immediately.
|
||||
4. The README renders correctly on GitHub with the inline GIF playing.
|
||||
5. `cargo install improvise` from a fresh shell works and produces a runnable binary.
|
||||
6. The GitHub releases page has prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).
|
||||
7. The GitHub Pages site at `https://<user>.github.io/improvise/` loads and the asciinema casts play.
|
||||
8. Nothing in the "Out of scope" list has been touched.
|
||||
|
||||
When all eight conditions hold, stop. Report back to the user with a summary of what was done and any blockers encountered. Do not post to Hacker News yourself — that's the user's call and their timing.
|
||||
@@ -1,301 +0,0 @@
|
||||
# Repository Map (LLM Reference)
|
||||
|
||||
Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture.
|
||||
Apache-2.0. Root binary+lib `improvise`; sub-crates under `crates/`:
|
||||
|
||||
- `improvise-core` — `Model`, `View`, `Workbook`, number formatting, and `model/source.rs` (`CellSource`: where raw cells are read from; `Model::over` is in `model/types.rs`). Depends on `improvise-formula` + `numtower` (exact sums). No UI/IO. `context/code/source.md`.
|
||||
- `improvise-formula` — formula parser, AST, `parse_formula`.
|
||||
- `improvise-io` — `.improv` save/load, CSV/JSON import, `Storage` backends. No UI/commands.
|
||||
- `improvise-protocol` — the wire vocabulary and the portable session slice: `Command`, `Projection`, `ViewState`, the per-widget caches, `reduce_view`, key types, and `delivery/{wire,notice,exchange,client}.rs` (notices, watermarks, corrections). Depends on `improvise-core`, `zset`, `numtower` + serde. No IO, no command layer, no terminal.
|
||||
- `improvise-command` — command/effect layer + semantic `App` state (app.rs, effect.rs, cat_tree.rs, viewport.rs, delivery.rs). Depends on `improvise-protocol`, `improvise-gp`, `zset`, `improvise-incremental`. No ratatui/crossterm.
|
||||
- `numtower` — exact decimal arithmetic tower (`Num`, canonical text), ADR 005. Pure Rust, no workspace deps; `improvise-protocol` uses it for the wire form and `improvise-core` for exact sums. Long form `context/code/numtower.md`.
|
||||
- `zset` — std-only Z-sets (non-zero checked `i64` weights, panic on overflow) plus `Circuit` (`Send`, not `Sync`; transactions `begin`/`commit`/`rollback`), operators, `Trace`, `Rebuild`, and `edge.rs` (stamped, coalescing delta edges between circuits). No dependencies. `context/code/zset.md`.
|
||||
- `zset-ffi` — C ABI over `zset` (`cdylib`/`staticlib`/`rlib`) for CFFI; `include/zset_ffi.h`. Native only, **not** in `wasmCrates`. `context/code/zset-ffi.md`.
|
||||
- `improvise-incremental` — the `Derived` trait, `Reading`, `Eager`, `differential_in`/`drawn_probes`, and `cell_delta`/`DeltaValue`/`moved_records` (the store's `CellChange` log to a `ZSet`), `Incremental` (maintained groups/shapes, formulas by stem, `ExactSum`; `over`/`sync`/`apply_changes` on a `ChangeSource`), `Pushdown` (a query-answering source), and `sched.rs`/`staged.rs` (an `Executor` of `Unit`s; `Pool` is native-only). Depends on `improvise-core`, `zset` + `numtower`; `core` never depends on `zset`. `context/code/improvise-incremental.md`.
|
||||
- `gp-tree` — problem-agnostic genetic-programming regression trees (`Tree`, `Spec`, `breed`, `simplify`, `render`). Knows nothing about improvise.
|
||||
- `improvise-gp` — the one seam between the two: emits a fitted `gp_tree::Tree` as improvise formula text (`formula_text` / `fitted_formula`). Depends on `gp-tree`, `improvise-formula` **and `improvise-core`** — `dataset.rs` builds a feature matrix from a `View`'s records (improvise-4s8) and `fitted_formula` takes the `&Model` it must resolve against (improvise-8xhy). `improvise-command` depends on it.
|
||||
- `improvise-ws-server` — the host: tokio + tungstenite, a `Registry` of `Host`s keyed by `DocumentId` (multi-document, ADR 001), `Sessions` fan-out, and the socket framing (`wire.rs`). Native only.
|
||||
- `improvise-worker-server` — the *other* host: the same `App`, `Sessions` and `Host` as the socket server, behind `self.onmessage`/`self.postMessage` with OPFS underneath. wasm32 only in anger; `worker.rs`/`vfs.rs` are ordinary Rust and are where every decision and every test is. Nothing depends on it.
|
||||
- `improvise-wasm-client` — the browser client: the fold, the key classifier, the socket framing it mirrors (`socket.rs`), and the render model the DOM paints (`render.rs`). Non-dev deps are `improvise-protocol` + serde + wasm-bindgen and nothing else. The page itself is `web/`.
|
||||
|
||||
The main crate (and `improvise-command`) re-export the sub-crates as `crate::{model, view, workbook, format, formula, persistence, import}` so consumer paths stay stable when crates shuffle. The main crate also re-exports `improvise_command::command` as `crate::command` and `improvise_command::ui::{app, cat_tree, effect, viewport}` under `crate::ui`. `ui::app` re-exports the protocol's `AppMode`, `ViewState`, `ViewCursor`, `ViewFrame`, `Panel`, `MinibufferConfig`, `MinibufColor` and `HELP_PAGE_COUNT`, and `command::key` re-exports the protocol's key types, so every pre-existing path still resolves.
|
||||
|
||||
Architectural intent lives in `context/design-principles.md` — read that for the "why". This doc is the "where".
|
||||
|
||||
---
|
||||
|
||||
## How to Find Things
|
||||
|
||||
| I need to... | Look in |
|
||||
|-------------------------------------|-------------------------------------------------|
|
||||
| Add a keybinding | `improvise-protocol/src/keymap.rs` → `Keymaps::default_keymaps()` (the table; the browser looks up the same bindings). Resolving a bound *name* into effects is `improvise-command/src/command/keymap.rs`. |
|
||||
| Add a user command | `command/cmd/<submodule>.rs`, register in `registry.rs` |
|
||||
| Add a state mutation | `ui/effect.rs` → implement `Effect` |
|
||||
| Change formula eval | `model/types.rs` → `eval_formula` / `eval_expr` |
|
||||
| Change cell storage / lookup | `model/cell.rs` → `DataStore` |
|
||||
| Change category/item behavior | `model/category.rs` → `Category` |
|
||||
| Change view axis logic | `view/types.rs` → `View` |
|
||||
| Change an axis label or cycle | `view/axis.rs` → `impl Axis`. An axis's **colour** is `Face::Axis(axis)` in `improvise-protocol/src/face.rs`, resolved by the `Theme` and turned into a style by `draw.rs::face_style` (ADR 004) |
|
||||
| Ask "what data is visible" vs "what can the user page" | `View::filtering_cats()` vs `categories_on(Axis::Page)` |
|
||||
| Change grid layout | `view/layout.rs` → `GridLayout` |
|
||||
| Change `.improv` format | `persistence/improv.pest` + `persistence/{format,parse}.rs` |
|
||||
| Change where documents are stored | `storage.rs` → `Storage` impls (`opfs.rs` for wasm) |
|
||||
| Change number display | `format.rs` → `format_f64` |
|
||||
| Change CLI args | `main.rs` (clap) |
|
||||
| Change import logic | `import/wizard.rs` → `ImportPipeline` |
|
||||
| Change how a gp-tree fit searches | `gp-tree/src/fit.rs` → `FitConfig` / `fit` |
|
||||
| Change gp-tree CSV preprocessing | `gp-tree/src/data.rs` → `preprocess` |
|
||||
| Change how a name is quoted anywhere | `improvise-formula/src/quote.rs` — the one home, both halves |
|
||||
| Change import-wizard keys/steps | `command/cmd/wizard.rs` (keys) + `import/wizard.rs` (steps) |
|
||||
| Change frame layout | `draw.rs` → `draw()` |
|
||||
| Change an incremental operator or the circuit | `crates/zset/src/{ops,circuit}.rs` |
|
||||
| Change where raw cells are read from | `improvise-core/src/model/source.rs` (`CellSource`) + `model/types.rs` (`Model::over`); `context/code/source.md` |
|
||||
| Change an incremental aggregate/formula | `improvise-incremental/src/{group,shapes,formulas}.rs` (`Incremental`); reads shared with `Eager`/`Pushdown` in `reads.rs` |
|
||||
| Run sub-circuits on threads | `improvise-incremental/src/sched.rs` (`Executor`, `Unit`, `Pool`) + `staged.rs` |
|
||||
| Change the edge protocol between circuits | `crates/zset/src/edge.rs` + `context/code/zset.md`, "Edges" |
|
||||
| Change the C ABI | `crates/zset-ffi/src/lib.rs` + `crates/zset-ffi/include/zset_ffi.h` |
|
||||
| Change the wire form of a value or delta | `improvise-protocol/src/delivery/wire.rs` |
|
||||
| Change what the host publishes/answers | `improvise-command/src/ui/delivery.rs` |
|
||||
| Change what counts as a document or record write | `improvise-command/src/ui/app.rs::apply_effects` (reads the batch cell delta) |
|
||||
| Ask "does this grid draw column totals" | `GridLayout::draws_totals` (`view/layout.rs`) — the one condition |
|
||||
| Change how wide the TUI grid is beside a side panel | `improvise-command/src/ui/viewport.rs` → `grid_width` (reached through `App::grid_width`); `draw.rs::draw_content` splits on it and `App::cmd_context` counts `visible_cols` on it (improvise-gtn7.1) |
|
||||
| Change what the browser page draws | `improvise-wasm-client/src/render.rs` → `RenderModel` |
|
||||
| Change a browser↔server frame | `improvise-protocol/src/frame.rs` → `ClientFrame`/`ServerFrame`. The **one** pair; both ends `pub use` it and neither defines one (improvise-xks6) |
|
||||
| Change the worker's message loop | `improvise-worker-server/src/worker.rs` → `Worker::on_frame` |
|
||||
| Change the session loop (either host) | `improvise-command/src/ui/host.rs` → `Host::handle` |
|
||||
| Change how the browser opens or seeds a document | `improvise-worker-server/src/vfs.rs` → `open` / `documents` |
|
||||
| Change what a browser resolves without the host | `improvise-protocol/src/resolve.rs` → `resolve`. **Not** `wasm-client/src/keys.rs`, which holds only `BrowserKey` (`KeyboardEvent` → improvise key) — this row named it until 2026-09-21 and that pointer is how a false claim reached `context/` (improvise-e5pf) |
|
||||
| Change the browser page itself | `web/app.js` (thin), `web/index.html`, `web/style.css` |
|
||||
| Host another document | `improvise-ws-server/src/registry.rs` → `Registry`, `ClientId` |
|
||||
| Change app state / modes | `ui/app.rs` → `App`, `AppMode` |
|
||||
| Write a test for model logic | `model/types.rs` → `mod tests` / `formula_tests` |
|
||||
| Write a test for a command | `command/cmd/<module>.rs` → colocated `mod tests`; helpers in `cmd/mod.rs::test_helpers` |
|
||||
|
||||
---
|
||||
|
||||
## Central Pattern: Cmd → Effect
|
||||
|
||||
```
|
||||
keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
|
||||
(pure, read-only ctx) (list of mutations) (only mutation site)
|
||||
```
|
||||
|
||||
```rust
|
||||
pub trait Cmd: Debug + Send + Sync {
|
||||
fn name(&self) -> &'static str;
|
||||
fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
|
||||
}
|
||||
|
||||
pub trait Effect: Debug {
|
||||
fn apply(&self, app: &mut App);
|
||||
fn kind(&self) -> EffectKind; // NO default — the compiler makes you classify
|
||||
}
|
||||
|
||||
// EffectKind::{ViewOnly, Model(ModelChange)}
|
||||
// ModelChange::{CellData, LayoutGeometry, WholeWorkbook}
|
||||
```
|
||||
|
||||
`CmdContext` (see `command/cmd/core.rs`) holds immutable refs to `Model`, `Workbook`, `View`, `GridLayout`, the `CmdRegistry`, `AppMode`, cursor/offsets, buffers, yanked cell, key code, expanded cats, panel/tile cursors, the opened editor, the view back/forward stacks, `visible_rows`/`visible_cols` and the cursor cell's `display_value`.
|
||||
|
||||
**Add a command**: implement `Cmd` in a `command/cmd/` submodule, register in `registry.rs`, bind in `default_keymaps()`. Simple wrappers go through the `effect_cmd!` macro in `effect_cmds.rs`.
|
||||
|
||||
**Add an effect**: implement `Effect` in `ui/effect.rs`, add a constructor fn if it helps composition.
|
||||
|
||||
---
|
||||
|
||||
## Key Types (skim; read the source for fields/methods)
|
||||
|
||||
**Model** (`model/types.rs`): `categories: IndexMap<String, Category>`, `data: DataStore`, `formulas: Vec<Formula>`, `views: IndexMap<String, View>`, `active_view`, `measure_agg`. `MAX_CATEGORIES = 24` (regular only) — `pub`, re-exported as `improvise_core::model::MAX_CATEGORIES`, and the one home: the import wizard reads it rather than keeping a copy (improvise-felo.1). Virtual categories `_Index`, `_Dim`, `_Measure` always exist. Use `regular_category_names()` for user-facing pickers.
|
||||
|
||||
**CellKey** (`model/cell.rs`): `Vec<(cat, item)>` — **always sorted by category**. Build with `CellKey::new(coords)` / `with(cat, item)` / `without(cat)`. Never construct the inner `Vec` directly.
|
||||
|
||||
**CellValue**: `Number(f64) | Text(String) | Error(String)`. Errors surface from formula eval (circular, div/0, etc.).
|
||||
|
||||
**DataStore** (`model/cell.rs`): interned symbols + secondary index `(Symbol, Symbol) → {InternedKey}`. Hot-path lookups go through `matching_values` / `matching_cells`.
|
||||
|
||||
**Formula AST** (`improvise-formula/src/ast.rs`):
|
||||
- `Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }`
|
||||
- `BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }`
|
||||
- `AggFunc { Sum, Avg, Min, Max, Count }`
|
||||
- `Formula { raw, target, target_category, expr, filter }`
|
||||
|
||||
**View** (`view/types.rs`): `category_axes: IndexMap<cat, Axis>`, page selections, hidden items, collapsed groups, `number_format`, `prune_empty`. It carries **no cursor** — `selected`/`row_offset`/`col_offset` moved to `ViewState::cursors` (improvise-8uz). `Axis { Row, Column, Page, Filter, None }`. `cycle_axis` rotates Row → Column → Page → None; `Filter` is outside the rotation (see gotcha 27).
|
||||
|
||||
**GridLayout** (`view/layout.rs`): pure function of `Model + View`. `cell_key(r,c)`, `cell_value(r,c)`, `drill_records(r,c)`. **Records mode** auto-detects when `_Index` is on Row and `_Dim` is on Column.
|
||||
|
||||
**AppMode** (`ui/app.rs`): 17 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, SavePrompt, CommandMode, ImportWizard, Help, Quit, RecordsNormal, RecordsEditing). Eight text-entry variants carry a `MinibufferConfig`. `SearchMode` is Normal + `search_mode: bool`, not its own variant. Per-variant UI facts live on the enum as exhaustive-match methods — `label()`, `hint_text()`, `panel()`, `minibuffer()`, `is_editing()`, `is_records()` (improvise-2hi); add a variant and the compiler names every one you must fill in. A mode's colour is a `Face` (`Face::Mode*`, `Face::of_minibuffer`), turned into a ratatui `Style` only by `src/draw.rs::face_style`. `App` state is split into `ModelState` (workbook, file_path, dirty) and `ViewState` (mode, cursor/offsets, buffers, panels, search) — see improvise-x2c/ew0. `wizard` and `drill_state` live on `App`, not `ViewState` (improvise-h0b) — see gotcha 24.
|
||||
|
||||
**Keymap** (`improvise-protocol/src/keymap.rs`; `dispatch` and `KeymapSet` in `improvise-command/src/command/keymap.rs`): `Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }`. Lookup fallback: `exact(key,mods) → Char(NONE) → AnyChar → Any → parent`. 17 mode keymaps, one per `ModeKey`, built by `Keymaps::default_keymaps()`; mode resolved via `ModeKey::from_app_mode()`. `ModeKey` and `AppMode` are two different enumerations that happen to be the same size — `SearchMode` is a `ModeKey` and not an `AppMode`; `Quit` is an `AppMode` with no keymap — so do not "fix" one count to match the other.
|
||||
|
||||
---
|
||||
|
||||
## `.improv` File Format
|
||||
|
||||
Plain-text, markdown-like, defined by `persistence/improv.pest`. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via `pest_meta`).
|
||||
|
||||
```
|
||||
v2025-04-09
|
||||
# Model Name
|
||||
Initial View: Default
|
||||
|
||||
## View: Default
|
||||
Region: row
|
||||
Measure: column
|
||||
|Time Period|: page, Q1
|
||||
hidden: Region/Internal
|
||||
collapsed: |Time Period|/|2024|
|
||||
format: ,.2f
|
||||
|
||||
## Formulas
|
||||
- Profit = Revenue - Cost # defaults to [_Measure]
|
||||
- Tax = Revenue * 0.1 [CustomCat]
|
||||
|
||||
## Category: Region
|
||||
- North, South, East, West # bare items, comma-separated
|
||||
- Coastal_East[Coastal] # grouped item, one per line
|
||||
> Coastal # group definition
|
||||
|
||||
## Data
|
||||
Region=East, Measure=Revenue = 1200
|
||||
Region=West, Measure=Revenue = |pending| # pipe-quoted text
|
||||
```
|
||||
|
||||
- **Name quoting**: bare `[A-Za-z_][A-Za-z0-9_-]*`, else CL-style `|…|` with escapes `\|`, `\\`, `\n`. Same convention in the formula tokenizer.
|
||||
- **Write order**: Views → Formulas → Categories → Data. Parser tolerates any order.
|
||||
- **Gzip**: `.improv.gz` (same text, gzipped).
|
||||
- **Legacy JSON**: auto-detected by leading `{`; never written.
|
||||
- **Axis keywords**: `row`, `column`, `page`, `filter`, `none`; `page`/`filter` carry the selected item after a comma.
|
||||
- **Virtual categories**: `_Index`/`_Dim` never persist; `_Measure` persists only non-formula items (formula targets are rebuilt from the `## Formulas` section).
|
||||
- **Virtual views**: `_Records`/`_Drill` (`view::VIRTUAL_VIEW_NAMES`) are never written and are dropped on load if an older file carries them; saving from records/drill mode writes the first real view as `Initial View`. Their commands rebuild them on demand.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas (digest — read before editing)
|
||||
|
||||
One or two lines each: the rule, and the trap. **The numbers are stable and cited across the repo; a
|
||||
citation of `gotcha N` means entry N in `context/gotchas.md`, which holds the full argument,
|
||||
the incidents and the ticket ids.** Read the long form of every gotcha your diff touches. Counts and
|
||||
"closed/open" claims in the long form are dated snapshots.
|
||||
|
||||
1. **Records mode is wide format.** A row is a coordinate tuple; columns are `[coords…, +cat, raw measures…, +]`. Editing a category column moves every measure cell in the row. `_Dim` carries a column *index*, resolved by `GridLayout::record_col`. `GridLayout::new` does **not** prune; `with_frozen_records(m, v, None)` does — production code wants the latter. `prune_axis_items` takes a predicate and re-marks survivors.
|
||||
2. **Effects never touch `std::fs`.** Documents go through `App::storage: Box<dyn Storage>`. Autosave and the headless final write still bypass it (improvise-31oe.11).
|
||||
3. **Every effect declares an `EffectKind`; there is no default.** `ViewOnly` never reaches other clients; `Model(CellData | LayoutGeometry | WholeWorkbook)` does. The view lives inside the workbook, so axis/page/format are `LayoutGeometry` and cursor/scroll/`SwitchView` are `ViewOnly`; `Add/RemoveCategory` are `LayoutGeometry`. `EffectKind` is too coarse to check the `ModelCommand` mapping — `every_model_command_names_the_effect_of_the_same_name` does.
|
||||
4. **Commands never mutate.** `&CmdContext` is read-only; return effects.
|
||||
5. **`CellKey` is always sorted.** Use the constructors.
|
||||
6. **No `Model::add_item`.** Use the `Workbook` facade (`wb.add_item`) or go through the category.
|
||||
7. **Virtual categories.** `_Measure` items = explicit ∪ formula targets (`measure_item_names()`). Cell aggregation (`apply_agg`) and formula `SUM/AVG` (`eval_agg`) differ on empty input; non-numeric values reduce by unanimity, else `Error("mixed")` — never dropped. `evaluate_aggregated` takes `Aggregation::{Exact, OverHidden}`.
|
||||
8. **Display rounding is view-only.** `format_f64` only in rendering; never feed a rounded value back into eval.
|
||||
9. **Formula eval is fixed-point.** `recompute_formulas` iterates to stability; cycles become `Error("circular")`.
|
||||
10. **Drilling a formula cell** strips the `_Measure` coordinate so the raw feeding records come back.
|
||||
11. **Minibuffer keymaps come from `TextEntry`** (eight text-entry modes); the buffer a prompt names and the keys edit are one string by construction. Read/write a runtime-named buffer with `read_buffer`/`write_buffer`. `SearchMode` is hand-written.
|
||||
12. **Panel keymaps come from `PanelKeymap`**; `Panel::ALL`/`name()`/`key()` drive the toggle keys.
|
||||
13. **A prefix key is stored as an address** (`ViewState::transient_prefix`), resolved by `Keymaps::prefix_keymap`; prefixes must stay flat (`prefix_keymaps_are_flat`). `ViewState` stays free of `improvise-command` types.
|
||||
14. **Keybindings are per-mode** (`ModeKey::from_app_mode`); minibuffer Enter/Esc are `Binding::Sequence`. `every_hint_key_resolves_in_its_mode` checks which keys a hint names, not what they do.
|
||||
15. **Registry names come from prototypes, not strings** — except the `Move` literals and four aliases, guarded by tests; a wrong-variant prototype shows up as a *duplicate* registration.
|
||||
16. **`effect_cmd!` / `simple_effect!`** wrap one effect; `simple_effect!`'s kind argument is mandatory. Six wrappers return `unless_unknown(...)` (gotcha 51). Mode entry is `enter-mode <name>`; no per-mode shim commands.
|
||||
17. **One float epsilon** (`FLOAT_EQ_EPSILON = 1e-10`) for `=`/`!=` *and* the div/0 guard, in both evaluators.
|
||||
18. **Order is data.** `IndexMap` order is persisted; an index of `HashSet`s is for lookup, never iteration (`DataStore::matching_*` sort to store order; `analyze_records` is first-seen order). A group's fold reads through the same rule (`spanned_positions`).
|
||||
19. **Cell-edit commit paths split** (`commit_regular_cell_value` / `commit_plain_records_edit`); keep both in sync. `cmd::core::uneditable` is the one decline, asked at open *and* commit. `CellId::NonItem` is a stored value naming no item.
|
||||
20. **Minibuffer commits go through `commit-buffer <formula|category|item>`**; the add effect is emitted last so its failure message wins.
|
||||
21. **An abort stops the chain, never the bookkeeping.** `Save`/`SaveAs`/`AddCategory` set `abort_effects`; tidy-up (mode restore, buffer clear) goes at the *head* of a batch. An abort is not a rollback (gotcha 47).
|
||||
22. **The import wizard's back path is a history stack.** Preconditions live in `validation_error()`; `ResolveDuplicates` appears only when `collisions()` is non-empty; headless defaults to `DuplicatePolicy::Refuse`. A collision is a cell written by two records.
|
||||
23. **`app.layout` goes stale mid-batch** — put a `RebuildLayout` effect before any effect that reads the layout after a grid change. The rebuild is skipped when nothing it reads moved (felo.13); a writer bypassing effects must call `rebuild_layout()`.
|
||||
24. **The cursor is per-client, keyed by view** (`ViewState::cursors`; use `App::cursor()/cursor_mut()`). Replacing the document clears the map and bumps `document_generation`. Cursor invalidation is *observed* from the grid frame (`cursor_epoch`), not called by effects; two arms (`LayoutGeometry` hashes the active view, `CellData` every view).
|
||||
25. **`ViewState` and its field types derive `Serialize`/`Deserialize`** — no borrowed-`'static` fields anywhere it reaches.
|
||||
26. **Axis presentation lives on `Axis`; colour is a `Face`** resolved by the `Theme`; `draw.rs::face_style` is the one translation to ratatui.
|
||||
27. **`Axis::Filter` constrains without being navigable.** Visible data = `filtering_cats()`; user-actionable = `categories_on(Page)` / `tileable_cats()`. What a virtual view must be is model-side (`records_steps`, `drill_steps`).
|
||||
28. **The browser backend is sync on purpose, with an async seam** (`OpfsFs`; `open/attach/refresh/commit_deletes`). It needs a *dedicated* worker — `createSyncAccessHandle` is absent in a `SharedWorker`.
|
||||
29. **"Active view" is stored twice and may differ:** `Workbook::active_view` is the default, `ViewState::active_view` is authoritative per client. `App::resolve_view` is the one fallback; `Sessions::adopt_deleted_views` handles a deleted view.
|
||||
30. **Two tests gate `ViewState` membership:** portable (no `Rc`/`Arc`) and snapshot-safe. `wizard` and `drill_state` live on `App` as session-local residue.
|
||||
31. **`:` commands settle their mode by observation** (`ReturnToNormalUnlessMoved`), not by a per-effect constant.
|
||||
32. **There is no aggregate render cache.** One cache per widget (`WidgetCache`), a `Projection` is one update to one widget; bootstrap is a *compaction* (`Snapshot { view, seed }`). The wizard has a cache but deliberately no projection.
|
||||
33. **The grid's labels frame its cells; the fold is not commutative.** A `Cells` update that overtakes its frame is dropped; `apply` invalidates relabelled positions itself; every arm touching labels or widths re-fits. Don't coalesce a batch.
|
||||
34. **A formula target is unquoted; pipe quoting has one home** (`improvise_formula::quote`). The keyword guard is canonical. `header_name` is total; duplicate `## Category:` sections are one category; `WHERE` needs `sp+`.
|
||||
35. **`Dirty` over-reports on purpose** — a redundant projection is cheap, a missed one leaves a client wrong forever. `LayoutGeometry` includes `formulas`.
|
||||
36. **A client's command has one door:** `Sessions::apply_checked(app, id, Checked)`. Assert the `Broadcast` itself, and seat a bystander where a wrong relay can move it.
|
||||
37. **`gp-tree` returns data and never prints; a fit needs seed *and* `FitConfig`.** A scorer takes `Row<T>` slices. `fitted_formula` refuses features the document cannot name; categories are forced `Categorical` (zv8t).
|
||||
38. **The wire vocabulary is derived twice** (`Command` view half, `ModelCommand` model half). Completeness is checked (`vocabulary_is_complete.rs`). `ApplyAndClearDrill` and `AcceptWizardModel` are deliberately off the wire (session-local operands). Recount, don't cite.
|
||||
39. **`Applied` is an enum (`Diff | Rebased | Rejected`)** because a document swap is not a diff; the trigger is the generation moving, not the variant. A swap fixture must keep the view *name*.
|
||||
40. **A batch's basis is three tiers** (`document_generation`, `layout_version`, `record_versions`); the unit is the record; lower tiers are *observed*, not read off the effect. The record tier is the consolidated batch cell delta (`cell_delta` over the store's change log, `moved_records`); `record_fingerprints` is gone.
|
||||
41. **`Storage` is not a VFS;** SQLite was declined on measurement (a second serialization is a second grammar).
|
||||
42. **Browser render decisions are made in Rust** (`RenderModel`: `editable` from the `CellId` variant, selection reconciled with offsets, `flatten_cat_tree`). `improvise_protocol::frame` is the one envelope pair for both deployments.
|
||||
43. **A batch carries its basis, and the server publishes it** (`Projection::Basis`). `Basis` has no `Default` or public constructor; `Checked`/`CheckedBatch` make "ignoring the basis" unrepresentable; `Rejection` keeps `RecordGone` / `RecordMoved` / `RecordUnstated` apart. Anti-staleness, not security. The check is a property of the session door (`apply_command` does not check), and only two of the three tiers hoist per frame — the record tier is asked per command (`records_named`) against the frame's basis. Read gotcha 43 before touching it.
|
||||
44. **The TUI grid draws from `GridCache`**, which frames it too (`AxisFrame`; width rule `header_level_widths`). Use the one-pass accessors, not per-position scans. `RecordHash` is document-scoped, `AggregateHash` layout-scoped.
|
||||
45. **One session loop (`Host`), two transports.** Saving is a request frame (`ClientFrame::Save`); dirtiness is *declared* for `WholeWorkbook` and *observed* for `CellData`/`LayoutGeometry` (`MarkDirty` is gone); for `CellData` the flag is each effect's own delta (drained after every `CellData` effect, so it keeps its order with `Save`) plus reorders, formulas and config, not a document fingerprint; writes that cancel across effects still leave it set. `impl Storage for Rc<T>` lives in `improvise-io`. `harness/` runs `OpfsFs` for real; Safari is unseen.
|
||||
46. **The page picks a peer:** `?ws=` → socket, else a module `Worker`. `#status` composes two facts; the first-frame watchdog is per-peer-agnostic. The demo model is built so every cell is a `Record`.
|
||||
47. **Keys become commands in the browser;** the server must not know the keymap. `GridExtent`, `set_viewport`, `Commands(Vec)`, `resolve::NEEDS_THE_HOST` and the five client rules (`projected_at`, `opened`, `refused_here`, `Step::Declined`, `notify`).
|
||||
48. **The browser JS is partly executed** (`harness/tests/shell.spec.js`); changes to `web/*.js` belong there. Serialize frames on one tail promise; disarm, don't pre-empt, a watchdog. Evidence headings may be module-qualified (and must be when the leaf collides).
|
||||
49. **A websocket handshake is exempt from same-origin policy** — `src/origin.rs` refuses the upgrade (loopback and absent allowed, two refused). Not authentication.
|
||||
50. **A composed TUI row measures in cells and names which side yields** (`fill_line` + `Keeps`). Keep whichever key table is *derived from what it describes*. Scrolls are measured where drawn (`grid_width`).
|
||||
51. **The command language reads pipes, refuses surplus, and checks names when applied** (`only_when_omitted`; `RequireKnown`/`unless_unknown`).
|
||||
52. **A date column imports as one hierarchical category;** `--split-date` restores flat. `Plan::spelling_clash` refuses what would silently merge; `--strict` refuses partial skips.
|
||||
53. **Delivery: a watermark is a `Stamp`; a correction is `target - held`, applied only at the watermark it starts from.** `through` is not `at` (the watermark advances only to `through`), a refused frame restores `Delivery` with the app snapshot, and a lost last notice is found only by `poll`. Axis changes refetch the window until ADR 008's render model. A notice from a foreign epoch only marks the shape stale (a foreign-epoch notice never becomes `seen`; only `apply_answer` moves `seen` to another epoch), and notices cut for different shapes (`cut_for`) never merge. Long form `context/code/delivery.md`.
|
||||
54. **Sources and exactness:** one evaluator (`Model`'s) over any `CellSource`; a `Model::over` source is read-only in every build, and a `ChangeSource` has one drainer (its cells are as of its last drain). Eager's math is fixed (`settled` compares by bits with every NaN one value; Min/Max by `total_cmp`; NaN-only Min/Max is NaN; a stable NaN is not `circular`). An exactness test needs an input that is inexact in floats. Stem order is a known evaluator defect (improvise-bw4p.30). `context/code/source.md`, `improvise-incremental.md`.
|
||||
55. **zset circuits are ids, not shared pointers.** `Stream`/`Input`/`Output` are `Copy` ids used through the circuit (`current`/`push`/`take`); a closure cannot read a stream during a step. Rollback replays newest first, `take` an output only after `commit`, and a panicking step outside a transaction leaves the circuit unusable. `context/code/zset.md`, "Threads" and "Transactions".
|
||||
56. **An edge joins by correction only.** A receiver joins an incarnation only by the answer to its latest request (a token per receiver incarnation, seeded fresh by `EdgeSource::new`); the host's timeout must exceed the round trip. `context/code/zset.md`, "Edges".
|
||||
57. **A scheduler turn runs outside the lock.** `Pool` catches a panicking turn and buries its unit (dropped inside `catch_unwind`), so `wait_idle` does not hang on a dead unit; a caller driving `Executor::claim` itself must catch `Claimed::run` and call `Executor::abandon`; a panic payload whose own drop panics is still open (improvise-bw4p.40). `context/code/improvise-incremental.md`, "Scheduling".
|
||||
58. **Tooling notes:** `Cargo.nix` has drifted from a fresh crate2nix run (improvise-pqf1). `proptest-regressions/` is gitignored, but zset commits its seeds (`crates/zset/tests/*.proptest-regressions`, a file beside the test, which that pattern does not match). `scripts/build-doc-site.mjs` lists every `context/` doc by hand: register a new one there. CI's test derivation sets `CARGO_PROFILE_DEV_DEBUG = "0"` (`nix/ci.nix`).
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
Per-crate entry points. Per-file notes, test counts and a dated line-count snapshot are in
|
||||
`context/file-inventory.md`; re-derive sizes with `wc -l`.
|
||||
|
||||
| Area | Where | Entry points |
|
||||
|---|---|---|
|
||||
| Core model | `crates/improvise-core/src/` | `model/{types,cell,record,category,id}.rs`, `view/{layout,types,axis}.rs`, `workbook.rs`, `format.rs` |
|
||||
| Formula | `crates/improvise-formula/src/` | `parser.rs`, `formula.pest`, `ast.rs`, `quote.rs` |
|
||||
| I/O | `crates/improvise-io/src/` | `storage.rs`, `opfs.rs`, `persistence/{improv.pest,format,parse,export,mod}.rs`, `import/{wizard,analyzer,csv_parser}.rs` |
|
||||
| Protocol | `crates/improvise-protocol/src/` | `lib.rs` (ViewState), `reduce.rs`, `render.rs`, `keymap.rs`, `face.rs`, `frame.rs`, `basis.rs`, `resolve.rs`, `client.rs`, `viewmodel.rs`, `help.rs` |
|
||||
| Command layer | `crates/improvise-command/src/command/` | `cmd/{core,registry,navigation,cell,commit,effect_cmds,fit,grid,mode,panel,search,text_buffer,tile,wizard}.rs`, `keymap.rs`, `parse.rs` |
|
||||
| Semantic UI state | `crates/improvise-command/src/ui/` | `effect.rs`, `app.rs`, `project.rs`, `reduce.rs`, `host.rs`, `viewport.rs`, `cat_tree.rs` |
|
||||
| TUI + CLI | `src/` | `draw.rs`, `main.rs`, `ui/{grid,help,import_wizard_ui,text}.rs`; integration tests in `tests/` |
|
||||
| GP | `crates/{gp-tree,improvise-gp}/` | `gp-tree/src/{lib,data,fit}.rs`, `improvise-gp/src/{lib,dataset,fit}.rs` |
|
||||
| Socket host | `crates/improvise-ws-server/src/` | `registry.rs`, `origin.rs`, `transport.rs`, `main.rs`; `host.rs` is a re-export shim |
|
||||
| Worker host | `crates/improvise-worker-server/src/` | `worker.rs`, `vfs.rs`, `bindings.rs` (wasm32 shim) |
|
||||
| Browser client | `crates/improvise-wasm-client/src/`, `web/` | `lib.rs`, `keys.rs`, `render.rs`, `socket.rs`; `web/{app,shell,worker,serial}.js`, `index.html`, `demo.improv`, `build.sh`, `assemble.sh` |
|
||||
| Browser harness | `harness/` | Playwright (`tests/shell.spec.js`, `tests/opfs.spec.js`), `opfs-contract/`, `build.sh` — not a workspace member; its own CI job |
|
||||
| Examples | `examples/`, `crates/*/examples/` | `gen-grammar.rs`, `pretty-print.rs`, `category_width_bench.rs`, `fit_csv.rs` |
|
||||
| Scripts | `scripts/` | `checkers.sh` (every `*-test.sh` suite), `check-comment-runs.py`, `check-ci-graph.py`, `bd-audit-ids`, `kb`, `pr-watch`, `pr-reviews`, `build-doc-site.mjs` |
|
||||
| Context docs | `context/` | `design-principles.md`, `gotchas.md`, `file-inventory.md`, `code/<area>.md` (arguments), `test-evidence/` (bug evidence by test name), `adr/` (001–005, 007, 009; no 006/008 file yet), `cloud-agents.md`; `prolog-sessions/kb.pl` is the KB; `docs/design-notes.md` is product vision |
|
||||
|
||||
---
|
||||
|
||||
## Wasm
|
||||
|
||||
The browser bundle is `improvise-{core,formula,io,protocol,command,wasm-client,worker-server,incremental}`
|
||||
plus `zset` and `numtower`; each `cargo check --target wasm32-unknown-unknown` clean with no feature flags
|
||||
(`wasmCrates` in `nix/ci.nix`, which also gates `improvise-gp`, a candidate). `improvise-ws-server`,
|
||||
`zset-ffi` and the root crate are native-only.
|
||||
Check new bundle dependencies with `cargo tree --edges normal` (the default output includes dev edges),
|
||||
per crate. `cargo check` cannot see runtime hazards: `Instant::now()` panics on wasm (`web-time`), and
|
||||
`std::fs` compiles but fails in a browser. Long form: `context/file-inventory.md`, appendix.
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
improvise [model.improv] # open TUI (default)
|
||||
improvise import data.csv [--no-wizard] [-o out] # import CSV/JSON
|
||||
improvise cmd 'add-cat Region' -f model.improv # headless command(s)
|
||||
improvise script setup.txt -f model.improv # run script file
|
||||
```
|
||||
|
||||
Import flags: `--category --measure --time --skip --extract --axis --formula --name`, `--split-date <field>`
|
||||
(flat date split; default is one hierarchical category), `--strict` (refuse any skipped record), and
|
||||
`--on-duplicate refuse|sum|separate` (default `refuse`). Details: `context/code/cli.md`, gotchas 51–52.
|
||||
|
||||
## Testing — the short version
|
||||
|
||||
Full guidance: `context/design-principles.md` §7.
|
||||
|
||||
- **No single test binary over 2 seconds**; the workspace wall clock is the sum. Re-measure with `cargo test --workspace` (a `finished in` per binary) and compare the slowest binary to the next few — a ratio, not seconds, and not a ratio to the median (it is zero).
|
||||
- Commands: build a `CmdContext`, call `execute`, assert on the returned effects. No terminal.
|
||||
- Property tests (`proptest`, default 256 cases) for invariants and round-trips; a grammar-walking strategy must read its pest grammar once (`OnceLock`).
|
||||
- **Bug-fix workflow:** write the failing test first. Record the bug in `context/test-evidence/`, keyed by test name — not in a doc comment (two-line cap, `scripts/check-comment-runs.py`).
|
||||
- Coverage target ~80% on logic code; skip ratatui render paths.
|
||||
|
||||
## Key dependencies
|
||||
|
||||
ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest, flate2, csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
|
||||
@@ -1,660 +0,0 @@
|
||||
# Test evidence — the headless CLI (`src/main.rs`, `tests/headless_cli.rs`)
|
||||
|
||||
Keyed by exact test name, per design-principles §7. A renamed test orphans its
|
||||
entry; `python3 scripts/check-comment-runs.py --check-evidence` catches a
|
||||
heading that names no test.
|
||||
|
||||
---
|
||||
|
||||
## `tests/headless_cli.rs` — the contract at the process boundary
|
||||
|
||||
improvise-q914 / improvise-j6x8 / improvise-z05h. The reasoning is
|
||||
`context/code/cli.md`; this file records what broke and what each test can see.
|
||||
|
||||
**Why a spawned binary rather than a unit test.** The three things under test
|
||||
are an exit code, a choice of stream, and whether a file was written — none of
|
||||
which exists below `std::process::exit`. Per design-principles §7 this is a
|
||||
boundary contract, so it is pinned once, slowly and honestly, and the
|
||||
`command/cmd/` unit tests assume it. Measured on the orchestrator's machine the
|
||||
whole binary runs in ~0.5s, twelve process spawns included, which is well
|
||||
inside the per-binary budget.
|
||||
|
||||
**What was broken.** `run_headless_commands` applied effects and never read
|
||||
`app.view_state.status_msg`, and set `exit_code` only for parse errors. So a
|
||||
successful command and a refused one produced byte-identical output — nothing —
|
||||
and exit 0 for both. Measured at `bf83294` before the fix: six of the twelve
|
||||
tests below failed.
|
||||
|
||||
### a_refused_command_says_what_went_wrong
|
||||
|
||||
`add-item Nonexistent Thing` against a document that has no such category.
|
||||
`Workbook::add_item`'s `Err` reaches `status_msg` and the effect does **not**
|
||||
set `abort_effects`, which is what makes this the load-bearing case: it is a
|
||||
genuine refusal that no flag in the codebase classifies as one.
|
||||
|
||||
Asserts the category's name appears, not merely that something was printed —
|
||||
"something was printed" is satisfied by a fixed `ok` on every command.
|
||||
|
||||
### a_command_with_nothing_to_say_says_nothing
|
||||
|
||||
**The rival this exists to kill**, and it is the one the brief named: a fix
|
||||
that prints on the happy path and stays silent on the refusal. Injected as
|
||||
`if said.trim().is_empty() { eprintln!("ok") }` — the exact inversion — and it
|
||||
fails this test *and* `a_refused_command_says_what_went_wrong`, five tests in
|
||||
all. A test that only asked "was anything printed" passes that rival.
|
||||
|
||||
`add-category Region` is the input because it sets no status at all, so the
|
||||
assertion is about the writer rather than about this command's vocabulary.
|
||||
|
||||
### an_aborted_batch_exits_non_zero_and_says_why
|
||||
|
||||
`:w` into a directory that does not exist. `effect::Save` reports through
|
||||
`report_file_write`, which sets the status **and** `abort_effects` (gotcha 21),
|
||||
so this is the one arm the exit code can honestly read. Before the fix this
|
||||
exited 0 with no output, which is what made `improvise cmd … && next-step`
|
||||
proceed over a failed write.
|
||||
|
||||
### a_command_that_succeeds_while_reporting_still_exits_zero
|
||||
|
||||
**The rival this exists to kill**: making any emitted status a non-zero exit,
|
||||
which is the obvious way to widen the previous test and is wrong. `effect::Save`
|
||||
on **success** also writes a status, as does `cmd/fit.rs` — whose status carries
|
||||
the seed that reproduces the fit. So "it spoke" cannot mean "it failed".
|
||||
|
||||
The first assertion is a check-the-checker: if the save ever stops reporting,
|
||||
the test proves nothing and says so rather than passing quietly.
|
||||
|
||||
### a_parse_error_still_exits_non_zero
|
||||
|
||||
The pre-existing arm, kept so the widening cannot silently drop it. Also
|
||||
asserts the message names the word typed, which the `Unknown command` arm
|
||||
reports post-alias-resolution.
|
||||
|
||||
### what_a_command_says_goes_to_stderr_and_stdout_stays_empty
|
||||
|
||||
Pins the stream choice itself. Both halves are asserted: a rival that writes to
|
||||
both streams passes an stderr-only check.
|
||||
|
||||
### a_script_reports_exactly_as_cmd_does
|
||||
|
||||
`improvise script` is the same function, so this cannot fail while `cmd`
|
||||
passes — which is the point. It is here because the ticket asked whether the
|
||||
hole was shared, and a claim that two paths are one is worth one cheap
|
||||
assertion rather than a sentence. The script also carries a `#` comment, so the
|
||||
comment-skipping arm is exercised on the way through.
|
||||
|
||||
### a_run_that_changes_nothing_does_not_rewrite_the_document
|
||||
|
||||
`move-cursor down` is `ViewOnly`, so nothing is dirty. **This test passed
|
||||
before the fix** — a freshly written document re-serialises byte-identically —
|
||||
and it is kept as the control that says so. The discriminating one is below.
|
||||
|
||||
### a_no_op_run_leaves_an_unnormalised_literal_alone
|
||||
|
||||
The reported instance. A hand-written document carrying `1e20` is rewritten as
|
||||
`100000000000000000000` by a run asked to change nothing, because
|
||||
`persistence::save` ran unconditionally. It fails before the fix where its
|
||||
sibling does not, which is exactly why both are here: the sibling cannot see a
|
||||
normalisation, only a reordering.
|
||||
|
||||
The first assertion checks the fixture still contains `1e20` before the run, so
|
||||
a future grammar change that rejects the literal fails loudly instead of making
|
||||
the test vacuous.
|
||||
|
||||
### a_run_that_changed_the_document_still_writes_it
|
||||
|
||||
The regression the dirty-gate could have introduced. `effect::Save` sets
|
||||
`dirty = false`, so a gate reading the flag once at the end would see
|
||||
`[add-item East, w other.improv]` as unchanged and silently drop the edit from
|
||||
the `-f` document. Accumulating `changed` per command is what this asserts; a
|
||||
gate written the obvious way fails it.
|
||||
|
||||
### the_cmd_help_does_not_advertise_a_language_it_refuses
|
||||
|
||||
Three clap doc strings said JSON three times over. Checks both subcommands,
|
||||
because `script`'s `<PATH>` help carried the same claim and a fix to `cmd`
|
||||
alone reads as complete.
|
||||
|
||||
### the_language_the_help_names_is_the_one_it_accepts
|
||||
|
||||
The other direction, and the reason the test above is not merely a spell check:
|
||||
it demonstrates the `:` command line works end to end *and* that the JSON form
|
||||
the help advertised is genuinely refused. Without this half, deleting the word
|
||||
"JSON" from the help would satisfy the check while the CLI still accepted
|
||||
neither language.
|
||||
|
||||
#### Scratch (helper)
|
||||
|
||||
A per-test temp directory named from the test's own tag and the process id, so
|
||||
the twelve tests cannot collide, with a `Drop` that removes it. Not
|
||||
`tempfile::TempDir` only because the fixture also has to be readable by a
|
||||
spawned child at a stable path.
|
||||
|
||||
#### seeded (helper)
|
||||
|
||||
Runs `add-category Region` to create the document. It asserts the file exists
|
||||
afterwards rather than trusting the seed, because every test that follows would
|
||||
otherwise misreport as "the command did nothing" when the seed was what failed.
|
||||
|
||||
### a_script_can_move_the_cursor_and_the_next_command_sees_it
|
||||
|
||||
`improvise cmd` and `improvise script` resolved every command through
|
||||
`CmdRegistry::parse` — the **pure** arm, which builds `CursorState::default()`
|
||||
(row 0, col 0, zero rows, zero columns). The interactive arm, which builds
|
||||
`CursorState::from_ctx(ctx)`, was never reached from a script, so every `Move`
|
||||
snapshotted a zeroed cursor at *parse* time, clamped to `(0, 0)` and emitted
|
||||
`SetSelected(0, 0)` — worse than a no-op, since it **resets** the cursor
|
||||
(improvise-6uva). The reasoning is `context/code/cli.md` section 7.
|
||||
|
||||
Reproduced at `a9cad15` before the fix, on a seeded document with `Region` on
|
||||
row carrying `North, South, East`: `move-selection 2 0 . hide-selected-row-item`
|
||||
and a bare `hide-selected-row-item` both wrote `hidden: Region/North`. After the
|
||||
fix the first writes `hidden: Region/East`.
|
||||
|
||||
**Why the test has two halves.** Asserting only that the moved script hides
|
||||
`East` cannot separate "the move worked" from "the origin happens to be East".
|
||||
The second document asserts the origin is `North`, so the first assertion is
|
||||
discriminating, and its message says what breaks if that ever changes — three
|
||||
identical replays of a fixture whose moves are silent no-ops prove nothing,
|
||||
which is how this survived five months.
|
||||
|
||||
Two faults were injected, one per layer, and both fail this test:
|
||||
|
||||
- **`parse_segment_interactive` calls `ctx.registry.parse` instead of
|
||||
`.interactive`** — the bug itself, one level down.
|
||||
`left: "hidden: Region/North" right: "hidden: Region/East"`.
|
||||
- **`run_headless_commands` builds the segment up front with `parse_line`**
|
||||
rather than per segment against a live `CmdContext` — the composition half.
|
||||
Same failure. The unit test in `parse.rs` passes under this second one, which
|
||||
is why both tests exist.
|
||||
|
||||
### a_script_cursor_move_survives_the_line_that_made_it
|
||||
|
||||
The same property across two `-f` arguments rather than one dot-separated line,
|
||||
because `improvise cmd` takes several and `improvise script` is one per line:
|
||||
the cursor is `App` state and a line boundary is not a reset.
|
||||
|
||||
### a_segment_is_built_against_the_cursor_it_will_run_against
|
||||
|
||||
The unit test for the layer that owns the bug. It seats a `CmdContext` at
|
||||
`(1, 0)` in a two-by-two grid and asserts `move-selection 0 1` executes to
|
||||
`SetSelected(1, 1)`.
|
||||
|
||||
Its second half is what keeps it honest: it builds the *same* command through
|
||||
`registry.parse` and asserts that one gives `SetSelected(0, 0)`. If the pure arm
|
||||
ever learns the cursor, the first assertion stops discriminating and this half
|
||||
fails first, saying so.
|
||||
|
||||
### a_pure_commands_two_arms_agree_on_no_arguments
|
||||
|
||||
A second instance of the same shape, found by the first fix rather than
|
||||
predicted by the ticket. `CmdRegistry::register_pure` built an interactive arm
|
||||
that refused **empty** args while its own pure arm accepted them, so `w` with no
|
||||
path — save to the current document — parsed one way and refused the other. Once
|
||||
a script re-parsed through the interactive arm, `improvise cmd 'w'` and `:w`
|
||||
both broke; the suite caught it as
|
||||
`a_failed_colon_write_lands_in_normal_where_its_error_is_drawn` reporting
|
||||
`":w — w: this command requires arguments"`.
|
||||
|
||||
Both arms are now `parse`, which is what "needs no interactive context" was
|
||||
supposed to mean. Restoring the empty-args guard fails this test on the first
|
||||
`expect`: the interactive arm refuses a bare `w` the pure arm accepts.
|
||||
|
||||
### segments_drops_comments_and_blanks
|
||||
|
||||
`segments` is the split half of `parse_line_with`, extracted so the headless
|
||||
loop can re-parse one segment at a time. It answers the same three cases the
|
||||
whole-line parser did — a `#` comment, a `//` comment and a blank line are no
|
||||
commands — plus the empty segment in `a . . b`.
|
||||
|
||||
### an_unknown_segment_is_an_error_not_a_silent_skip
|
||||
|
||||
`parse_segment_interactive` returns `Ok(None)` for a segment with no word, which
|
||||
is a shape an unknown *word* must not be able to reach: a silently skipped
|
||||
command in a script is the partial-and-silent failure this repo keeps finding.
|
||||
The error also names the word, matching what `CmdRegistry::parse` does.
|
||||
|
||||
### an_explicit_coordinate_still_beats_the_cursor
|
||||
|
||||
Found by the ChatGPT Codex review of PR #151 at head `64bcb82`, and correct: the
|
||||
first version of improvise-6uva's fix introduced a regression it was not looking
|
||||
for. Four registry entries — `clear-cell`, `yank`, `paste`, `drill-into-cell` —
|
||||
were registered with `register`, and their **interactive** arms took `|_args,
|
||||
ctx|`: they discarded the arguments and used `ctx.cell_key()`. That was harmless
|
||||
while a script only ever reached the *pure* arm, which reads the coordinates. The
|
||||
moment the headless loop re-parsed interactively, `clear-cell Region/East` passed
|
||||
the syntax gate and then cleared whatever the cursor was sitting on — the origin,
|
||||
`Region/North`.
|
||||
|
||||
So improvise-6uva's own lesson had a **third** instance in its own fix: two
|
||||
construction arms that mean different things demote whichever caller reaches the
|
||||
weaker one, and here the "weaker" one was the arm that had the context. The four
|
||||
arms went through `key_from_args_or_cursor`: explicit coordinates win, the
|
||||
cursor is the fallback, which is also what the keymap path wants since a binding
|
||||
carries no arguments. That helper is gone since improvise-2bac: the cursor is
|
||||
now the registry's `fallback`, reached only when `parse` refused, so explicit
|
||||
coordinates win by construction (`context/code/cli.md` §8).
|
||||
|
||||
The fixture writes three cells and clears the one the cursor is **not** on, so
|
||||
the assertion separates "cleared the named cell" from "cleared the cursor cell"
|
||||
— clearing `North` by name would have passed against the broken code. Restoring
|
||||
the arg-discarding arm fails it: `the cursor cell was cleared instead of the
|
||||
named one: ["Region=East = 3", "Region=South = 2"]`.
|
||||
|
||||
### commit_cell_edit_writes_the_named_cell_and_spares_the_cursors
|
||||
|
||||
improvise-2bac, found by the post-merge adversarial review of PR #151: the
|
||||
fourth instance of that PR's own lesson. `commit-cell-edit` was registered with
|
||||
two free arms. The pure arm read `<mode> <value> <coords>`; the interactive arm
|
||||
read only `<mode>`, took the value from the `edit` buffer and the key from the
|
||||
cursor. Once #151 made scripts construct through the interactive arm,
|
||||
`commit-cell-edit editing 5 Region/East` passed the syntax gate and then
|
||||
committed an **empty** buffer to the cursor cell — deleting it — with exit 0
|
||||
and nothing on stderr.
|
||||
|
||||
Reproduced at `40b8913` before the fix, through the real binary: the document
|
||||
came back as `["Region=East = 3", "Region=North = 1"]` — `South`, the cell the
|
||||
cursor had been moved onto, was gone and `East` was unchanged.
|
||||
|
||||
The fixture writes three cells and **moves the cursor to `South`** in the same
|
||||
line before committing to `East`, so the cursor sits on neither the target nor
|
||||
the origin: a test with the cursor on the target is right by accident, and one
|
||||
that only asserts the target misses the destruction. It asserts all three.
|
||||
|
||||
Injected rivals, run: (a) `contextualize` for `CommitAndAdvance` overwriting
|
||||
`key` and `value` from the cursor and the buffer — fails this test and
|
||||
`a_line_whose_gate_passes_runs_whole`; (b) the registry dispatching to the
|
||||
fallback first whenever one exists (the pre-fix shape) — fails this test,
|
||||
`a_line_whose_gate_passes_runs_whole`, `an_explicit_coordinate_still_beats_the_cursor`
|
||||
and both registry unit tests below.
|
||||
|
||||
### a_line_whose_gate_passes_runs_whole
|
||||
|
||||
improvise-2bac's second symptom, reported on the ticket: `context/code/cli.md`
|
||||
§7 said the whole-line `parse_line` gate preserved line atomicity. It did not,
|
||||
wherever the interactive arm refused an argument list the pure arm accepted.
|
||||
`add-category Foo . commit-cell-edit editing 5 Foo/x` on an empty document
|
||||
printed `Parse error: commit-cell-edit: no cell at cursor` after `Foo` had
|
||||
already been added and saved: half the line landed.
|
||||
|
||||
The assertion is on `Parse error` and on the committed cell, **not** on the exit
|
||||
code. The line exits 1 both before #151 and after this fix, for a different and
|
||||
pre-existing reason: the grid has no rows, so `commit-cell-edit`'s advance
|
||||
emits `AbortChain`, and the CLI maps an aborted batch to exit 1
|
||||
(`context/code/cli.md` §3). Asserting success would have made this test fail on
|
||||
the correct behaviour.
|
||||
|
||||
### a_malformed_commit_line_is_refused_whole
|
||||
|
||||
The other half of atomicity, which must survive the fix: a segment the gate
|
||||
refuses (`commit-cell-edit editing 5` — a value and no coordinates) refuses the
|
||||
whole line, so the preceding `add-category Foo` never lands. It passed before
|
||||
the fix and must keep passing; its job is to catch a "fix" that loosens the
|
||||
pure arm to make the gate agree.
|
||||
|
||||
### commit_cell_edit_honours_an_explicit_value_and_cell_over_the_cursor
|
||||
|
||||
The unit-level form of the headless test, against `CmdRegistry::interactive`
|
||||
directly: cursor on `Type=Food, Month=Jan`, a decoy `edit` buffer, and an
|
||||
explicit `5` for `Type/Clothing Month/Feb`. It asserts the built command's key
|
||||
and value. Rival (a) above — a `contextualize` that overwrites the parsed
|
||||
fields — failed it. improvise-yf7p then removed `contextualize` altogether
|
||||
(`context/code/cli.md` §8), so rival (a) no longer compiles; the test stays as
|
||||
the behavioural statement of what the signature now guarantees.
|
||||
|
||||
### every_argument_list_the_gate_accepts_the_run_accepts_too
|
||||
|
||||
Enumerates **every** registration (`CmdRegistry::names`, no hand-kept list)
|
||||
against every candidate argument list, and requires the interactive arm to
|
||||
accept whatever the pure arm accepts, in a context whose cursor sits on **no
|
||||
cell** — a category with no items. That context is what made the old
|
||||
`commit-cell-edit` arm refuse, so a context with a cell under the cursor would
|
||||
have passed the bug. The assertion that the cursor has no cell is checked
|
||||
first; a plain `Workbook::new` does not satisfy it, which is how the fixture
|
||||
came to add the empty category. Rival (b) fails it.
|
||||
|
||||
### explicit_open_and_focus_arguments_beat_the_toggle
|
||||
|
||||
The fifth instance, named on improvise-2bac as plausible and not yet run:
|
||||
`toggle-panel-and-focus <panel> false false` ignored both flags and toggled, so
|
||||
on a **closed** panel it opened it. The fixture closes every panel first,
|
||||
because on an open-and-focused panel the toggle also closes it and the test
|
||||
would pass against the bug. Rival: `execute` ignoring `target` and always
|
||||
toggling — fails this test for every panel.
|
||||
|
||||
### a_script_writes_the_same_record_whatever_terminal_ran_it
|
||||
|
||||
improvise-up8g. `run_headless_commands` read `crossterm::terminal::size()` into
|
||||
`App::term_{width,height}`, and `CmdContext::visible_rows` derives from the
|
||||
height, so `page-scroll 1 . hide-selected-row-item` over a 60-item row axis hid
|
||||
`R13` at 24 rows and `R40` at 60 — one script, two documents. It became live
|
||||
when #151 let a script's commands see the cursor (improvise-6uva); before that
|
||||
the cursor was always the origin and every run hid `R01`.
|
||||
|
||||
The test runs the same script against two copies of one document at `24x80`
|
||||
and `60x200` and asserts the same record is hidden. One size cannot see the
|
||||
bug, which is the ticket's own requirement. The size is driven through `LINES`
|
||||
and `COLUMNS` with the child detached from any terminal by `setsid`: crossterm
|
||||
asks `/dev/tty` first and falls back to `tput`, which honours those variables,
|
||||
so without the detach a developer's real terminal would override both sizes and
|
||||
the test would pass against the bug on a laptop and fail it only in CI.
|
||||
|
||||
Two guards against vacuity. The hidden record must not be `R01`, because a
|
||||
cursor that never moved agrees at every size (the pre-#151 shape, and the shape
|
||||
of a fix that re-broke 6uva). And something must be hidden at all. Failed at
|
||||
`9e78051` with `R13` against `R40`. It cannot see a *width* leak, and does not
|
||||
need to: width moves only `col_offset`, which no command today turns into a
|
||||
written record.
|
||||
|
||||
### a_commit_whose_advance_has_nowhere_to_go_still_exits_zero
|
||||
|
||||
improvise-c5ip. `commit-cell-edit` ends in `EnterAdvance`, which emits
|
||||
`AbortChain` when the cursor is already bottom-right; the CLI mapped
|
||||
`last_batch_aborted()` to exit 1, so a write that landed **and was saved**
|
||||
exited 1 with nothing on stderr. Two cases: the cursor moved to the last of
|
||||
three rows and the write named explicitly, and a commit on a grid that is empty
|
||||
until the same line creates it. The value is asserted **first**, so the test
|
||||
cannot pass by the write simply not happening. Failed at `487f0b6` on the exit
|
||||
assertion with `South = 9` already written. Rival: keeping
|
||||
`|| last_batch_aborted()` in `main.rs` beside the new refusal check — fails
|
||||
here, at the first exit assertion.
|
||||
|
||||
### a_refusal_that_only_writes_a_status_exits_non_zero
|
||||
|
||||
improvise-udt2. `add-item Nonexistent Thing` writes the workbook's error to the
|
||||
status and does not abort, so it exited 0. It asserts the refusal was printed
|
||||
before asserting the exit, so it cannot pass on a run that failed for some other
|
||||
reason. Rival: `AddItem` writing `status_msg` directly instead of calling
|
||||
`App::refuse` — fails here. The two rivals the ticket named are the existing
|
||||
`a_command_that_succeeds_while_reporting_still_exits_zero` (a successful `:w`
|
||||
calling `refuse` fails it, measured) and `a_fit_that_reports_its_seed_is_not_a_refusal`.
|
||||
|
||||
### a_refusal_marks_only_its_own_batch
|
||||
|
||||
A refusing batch followed by a reporting one. The second assertion is the one
|
||||
the default cannot supply: a flag that is never reset passes the first half and
|
||||
fails the second. Rival: deleting the reset in `apply_effects` — fails at the
|
||||
second assertion. Rival: `AddItem` not calling `refuse` — fails at the first.
|
||||
|
||||
### a_chain_with_nowhere_to_go_aborts_without_refusing
|
||||
|
||||
`AbortChain` must abort — asserted, so the test cannot pass on a batch that did
|
||||
nothing — and must not refuse. Rival: `AbortChain` calling `refuse` — fails
|
||||
here, and fails the process-level c5ip test too.
|
||||
|
||||
### a_failed_save_is_a_refusal_as_well_as_an_abort
|
||||
|
||||
The one place the two flags coincide, and the reason the old exit code looked
|
||||
right: a failed file write both stops the chain (gotcha 21) and refuses. It
|
||||
keeps `an_aborted_batch_exits_non_zero_and_says_why` meaningful now that the CLI
|
||||
reads only the refusal.
|
||||
|
||||
### a_fit_that_reports_its_seed_is_not_a_refusal
|
||||
|
||||
udt2's second rival: `fit-measure` speaks on **success**, carrying the seed.
|
||||
It asserts the status mentions the seed first, so a fit that said nothing
|
||||
cannot pass. Rival: the success arm emitting `refuse` instead of `set_status` —
|
||||
fails here. The refusal arm is asserted in
|
||||
`a_refusal_names_what_went_wrong_and_stores_nothing`.
|
||||
|
||||
### a_headless_import_with_duplicates_refuses_by_default_and_names_them
|
||||
|
||||
improvise-cgqt, CLI parity: `import --no-wizard` summed colliding records
|
||||
silently apart from a warning, as 2cw9 left it. Red before the flag existed,
|
||||
because the import succeeded. It asserts a non-zero exit, the colliding records
|
||||
named in the wizard's own wording (`2 records (#1, #3): Region=East`), the flag
|
||||
named, and no output file. It has two siblings. `..._sums_duplicates_when_told_to`
|
||||
asserts `= 30` and no `Occurrence`. `..._keeps_duplicates_separate_when_told_to`
|
||||
asserts the category, `= 10` and `= 20`, and no `= 30`. So the two flags cannot
|
||||
both be summing.
|
||||
|
||||
### a_headless_import_of_a_folder_merges_its_csv_files_and_repeats_byte_for_byte
|
||||
|
||||
improvise-lpzu, at the process boundary. The folder holds code-maat's three
|
||||
schemas (coupling, age and abs-churn) and a `README.txt`. At `e65d87fd`'s
|
||||
parent the run failed with `Cannot read '…/maat': Is a directory (os error
|
||||
21)`, because a single path went straight to `read_to_string`. The test
|
||||
asserts:
|
||||
|
||||
- sparse Data lines for all three schemas, with no README;
|
||||
- the four views, `Default` first and initial, then one per file;
|
||||
- `File: filter, coupling`;
|
||||
- a second run, with the first run's `maat.improv` now inside the folder,
|
||||
writing the same bytes.
|
||||
|
||||
Rivals R2, R5, R6 and R7 each fail it.
|
||||
|
||||
### a_headless_import_takes_files_and_folders_together_and_refuses_an_empty_folder
|
||||
|
||||
improvise-lpzu. A file argument and a folder argument together are one merge,
|
||||
and the extra file's records carry its own `File` item. Before the fix this
|
||||
failed with `Multi-file import only supports CSV files`, because the folder was
|
||||
not a `.csv` path. An empty folder exits non-zero, writes nothing, and names
|
||||
itself: `no CSV files in folder …/empty`. Rivals R2, R4 and R5 each fail it.
|
||||
|
||||
### a_saved_per_file_view_projects_only_its_file
|
||||
|
||||
improvise-lpzu (`tests/folder_import_views.rs`). A folder import is saved,
|
||||
reloaded and projected through `project::grid_in`, the one derivation the TUI
|
||||
grid and the browser both draw from (gotcha 44). This is why no separate TUI
|
||||
or browser test was written. For each file's view:
|
||||
|
||||
- the page bar carries `File` fixed at that file with nothing to cycle
|
||||
(`items: None`, gotcha 27);
|
||||
- the drawn cells are that file's alone: `2.00, 7.00` for age and
|
||||
`5.00, 3.00` for coupling.
|
||||
|
||||
Rivals R2, R7 and R9 each fail it. Since the review of `4e68372c` the fixture
|
||||
also holds a header-only `soc.csv`. Its view must carry `File` at `soc`, be a
|
||||
records view, and draw zero rows. Rivals F7 and F8 fail that half.
|
||||
|
||||
### a_csv_column_forced_to_category_keeps_its_text_verbatim
|
||||
|
||||
improvise-8jk6.4. Before the fix, `commit,lines` with `00565796` and
|
||||
`220944e2` forced to Category imported as items `565796` and `22094400.0`.
|
||||
The CSV parser had typed both as numbers, and the category was named from the
|
||||
number. On the base this failed with the items line `- |565796|, |22094400.0|`.
|
||||
The test also asserts both Data lines, which proves the measure `lines` still
|
||||
reads 3 and 5 through the same records.
|
||||
|
||||
- **R1**, a category named from the typed value, fails it.
|
||||
- **R2**, a CSV measure read as a JSON value (`as_f64` on the text), fails it
|
||||
on the Data lines.
|
||||
|
||||
### a_json_number_forced_to_category_is_named_as_json_writes_it
|
||||
|
||||
improvise-8jk6.4, the JSON half. This is not a bug demonstration: it passed
|
||||
on the base, and it pins that the fix left JSON alone. JSON keeps no source
|
||||
text for a number, so `42`, `1.5` and `1e3` name items `42`, `1.5` and
|
||||
`1000.0`. `1e3` is the discriminating value, because its source text, its f64
|
||||
display (`1000`) and its JSON display (`1000.0`) all differ. **R10**, naming a
|
||||
number by its `f64` display, fails it on `1000.0`.
|
||||
|
||||
### an_import_formula_that_does_not_parse_refuses_names_it_and_writes_nothing
|
||||
|
||||
improvise-8jk6.5. Before the fix, `--formula 'Y = x +'` was dropped: the run
|
||||
printed `Saved to …` and exited 0. That is the base failure (`a dropped
|
||||
formula exited 0`). The test passes a good formula too (`Z = x`) and asserts
|
||||
it is not named, so a refusal that lists every formula cannot pass.
|
||||
`expected` comes from pest's error, which proves the parse error is carried
|
||||
and not only the formula text. **R4**, a build that drops what does not
|
||||
parse, fails it. Since improvise-8jk6.17 it passes two bad formulas, so
|
||||
**R5**, naming only the first, fails it here too. At `efaa5050` only the io
|
||||
unit test saw R5.
|
||||
|
||||
### an_import_that_keeps_no_record_refuses_and_says_why
|
||||
|
||||
improvise-8jk6.6. On the lpzu base `[1, 2, "three"]` already exited 1, so the
|
||||
exit-code half was not the bug. The reason was: `At least one category must
|
||||
be accepted`, which is true only because no fields exist, and which points
|
||||
the user at a fix that cannot help. The base failure was on the message.
|
||||
**R7**, removing `nothing_to_import` from the build, fails it the same way.
|
||||
|
||||
### an_import_that_skips_some_records_warns_with_a_count_and_still_writes
|
||||
|
||||
improvise-8jk6.6, the partial case. It passed on the base. It pins the
|
||||
default the user kept on PR #321 ("Add a --strict flag"), so that changing
|
||||
the default is a decision and not a drift. It asserts the warning's count (`1 of 3`), exit 0,
|
||||
and a Data line from the record *after* the skipped one. That last check
|
||||
shows the skip did not end the import early. **R9**, making any skip fatal,
|
||||
fails it.
|
||||
|
||||
### a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not
|
||||
|
||||
improvise-lpzu, after the user's ruling on PR #314 ("we'd want file identity").
|
||||
At `4e68372c`, a folder holding one CSV took the single-file path, because
|
||||
`import dir/` was exactly `dir/*.csv`. That document had no `File` category and
|
||||
no per-file view, and the test failed at the `## Category: File` assertion. The
|
||||
other half pins the direction the ruling did *not* change: the same CSV named
|
||||
directly imports without any `File`. The rival that merges every all-CSV
|
||||
import, lone files included, fails that half ("a lone file keeps today's
|
||||
shape"). So the test separates "a folder merges" from "everything merges".
|
||||
|
||||
### a_pipe_quoted_argument_is_one_argument_without_its_pipes
|
||||
|
||||
improvise-31oe.3. Red before the fix: the old tokenizer split on whitespace
|
||||
only, so `create-view |By area|` built `CreateViewCmd(["|By", "area|"])`. It
|
||||
asserts the whole built command, so a lexer that joined the words but kept the
|
||||
pipes fails as well as one that split them. The second case has two quoted
|
||||
arguments side by side, so a lexer that reads one pipe piece per line fails.
|
||||
Rival: `bare_word` pushing `|` as a plain character. Fails here and in
|
||||
`a_coordinate_may_pipe_quote_either_half`.
|
||||
|
||||
### a_pipe_quoted_argument_reads_the_escapes_improv_does
|
||||
|
||||
The three escapes `pipe_quote` writes, `\|`, `\\` and `\n`, each in its own
|
||||
case. Red before the fix. Rival: `pipe_body` returning the raw body without
|
||||
`unescape_pipe_body`. Fails here, at the first case, and nowhere else in the
|
||||
crate, which is why the case exists. A lexer that treated `\|` as a closing
|
||||
pipe would also fail the first case, because the pipe after `b` then opens a
|
||||
piece that never closes.
|
||||
|
||||
### a_command_given_more_arguments_than_it_takes_is_refused
|
||||
|
||||
improvise-31oe.3's second half. Red before the fix: `create-view By area`
|
||||
parsed and dropped `area`. It asserts the refusal opens with the command's name
|
||||
and quotes the surplus word, and that the pipe-quoted spelling of the same line
|
||||
is the control that parses. Rival: `MaxArgs::admit` answering `Ok` always.
|
||||
Fails here, in `no_command_silently_drops_an_argument_it_was_given`, and in
|
||||
`a_colon_line_naming_something_missing_refuses_on_the_status_line`.
|
||||
|
||||
### no_command_silently_drops_an_argument_it_was_given
|
||||
|
||||
The rule is one rule only if every registration obeys it, so this enumerates
|
||||
every registered name against every candidate argument list, on both arms. For
|
||||
each list a command accepts, it appends **four** surplus words. One would be
|
||||
absorbed by an optional argument (`add-formula ["1"]` plus one more is a valid
|
||||
two-argument call), and four is more than any optional span in the registry.
|
||||
The last word must be named in the refusal. The exceptions are stated, not
|
||||
inferred: the `Any` list, and three commands whose own shape refusal runs
|
||||
first. Rivals, each run: declaring `add-item-in-group` as `Any` fails here,
|
||||
naming the dropped list. Removing the check from the interactive arm alone
|
||||
fails here and in the `:` line test. A test of the pure arm alone would have
|
||||
missed that.
|
||||
|
||||
### a_command_naming_something_the_document_lacks_is_a_refusal
|
||||
|
||||
improvise-31oe.2 and its siblings: `set-axis`, `set-page`, `hide-item`,
|
||||
`show-item`, `toggle-group` and `switch-view`, each naming a missing category,
|
||||
item, group or view. Red before the fix at the first row, `set-axis Regoin
|
||||
column`, with an empty status. Each row asserts the refusal flag, the missing
|
||||
name in the message, and that the document's `.improv` text and the active view
|
||||
are unchanged. The last two catch a refusal written after the write. The
|
||||
control is `a_command_naming_what_the_document_has_is_not_refused`, whose group
|
||||
row is the one that caught a group check reading only `Category::groups`:
|
||||
`add_item_in_group` registers no `Group`. The checks are `effect::unknown_name`,
|
||||
run by `RequireKnown` when the batch applies. Rivals, each run: the
|
||||
`Needs::Item` arm answering `None` fails at `set-page ["Type", "Nowhere"]`. The
|
||||
category test (`try_axis_of(..).is_none()`) never firing fails at the first
|
||||
row. The `Needs::View` arm answering `None` fails at the `switch-view` row.
|
||||
`RequireKnown` placed after the write, or refusing without `refuse_and_stop`'s
|
||||
abort, fails the document-unchanged assertion.
|
||||
|
||||
### a_colon_line_naming_something_missing_refuses_on_the_status_line
|
||||
|
||||
The TUI path for both tickets, through `ExecuteCommand` and `App::apply_effects`,
|
||||
not through the command alone. It asserts the refusal flag, the name on the
|
||||
status line, and the mode back at Normal. Rivals: an unknown category accepted
|
||||
fails at `:set-axis Regoin row`, and the arity check removed from the
|
||||
interactive arm fails at `:create-view By area`.
|
||||
|
||||
### an_unknown_category_axis_or_item_exits_non_zero_and_names_it
|
||||
|
||||
The three repros from improvise-31oe.2, at the process boundary. Red against
|
||||
the `gh/main` source with this test file: `set-axis Regoin row` exited 0 with an
|
||||
empty stderr. The control, `set-page Region East`, must exit 0. Without it, a
|
||||
fix that refused every `set-page` would pass.
|
||||
|
||||
### a_pipe_quoted_name_reaches_the_document_whole
|
||||
|
||||
improvise-31oe.3's repro at the process boundary: the saved document must carry
|
||||
`## View: By area` and no `|By`. The second half, `create-view By area`, must
|
||||
exit non-zero naming `"area"`. Red against the `gh/main` source: the document
|
||||
carried the view `|By`.
|
||||
|
||||
### command::cmd::registry::tests::a_malformed_coordinate_is_refused_not_dropped
|
||||
|
||||
Codex's P2 on PR #316: `set-cell` and `parse_cell_key_from_args` (`clear-cell`,
|
||||
`yank`, `paste`, `drill-into-cell`, `commit-cell-edit`) built coordinates with
|
||||
`filter_map(split_once('/'))`, so `set-cell 1 Type/Food unexpected` dropped
|
||||
`unexpected` without a word. The test names every caller, a half-empty
|
||||
coordinate, and a control line that must still parse. Rival: the `filter_map`
|
||||
restored in `parse_cell_key_from_args` fails at the first line,
|
||||
`set-cell 1 Type/Food unexpected dropped "unexpected" without a word`.
|
||||
|
||||
### command::cmd::text_buffer::tests::colon_refuses_a_malformed_coordinate_and_leaves_the_cursor_cell_alone
|
||||
|
||||
The PR #316 review's blocker: on `:`, `parse_cell_key_from_args` failing sent
|
||||
`clear-cell`, `yank`, `paste` and `drill-into-cell` into the cursor fallback,
|
||||
which ignores its arguments. `:clear-cell Type/Clothing Month/Feb junk` cleared
|
||||
the cursor's Food/Jan without a word. The test runs each line through the full
|
||||
`:` path and asserts a refusal naming `"junk"`, both cells intact, nothing
|
||||
yanked and no drill opened. Rival: `only_when_omitted` made unconditionally
|
||||
`Ok` fails it at the first line.
|
||||
|
||||
### command::cmd::text_buffer::tests::colon_lets_a_later_segment_use_what_an_earlier_one_created
|
||||
|
||||
The review's second blocker: a `:` line is built before any segment runs, so the
|
||||
name checks in `execute` refused `create-view Second . switch-view Second` and
|
||||
`add-item Type Tools . set-page Type Tools` after the first half landed. The
|
||||
checks now run when the effect applies. Rival: the checks back in `execute`
|
||||
fails both lines with `No view named "Second"` and `Unknown item 'Tools'`.
|
||||
|
||||
### a_strict_import_that_skips_a_record_refuses_and_writes_nothing
|
||||
|
||||
improvise-8jk6.6, the user's ruling on PR #321. Written before `--strict`
|
||||
existed, it failed on clap's unknown-argument error. It asserts the refusal
|
||||
names the flag and the same count as the default warning, and that the output
|
||||
path does not exist.
|
||||
|
||||
- **S1**, the flag parsed but ignored, fails it on the exit status.
|
||||
- **S3**, refusing only after `persistence::save`, fails it on the output
|
||||
path. That is the "a strict refusal still writes" contract this PR decided
|
||||
against.
|
||||
|
||||
### a_strict_import_that_skips_nothing_still_writes
|
||||
|
||||
The control for the entry above. Without it, a `--strict` that refuses
|
||||
**every** import passes that test. **S2** is that rival, and it fails here.
|
||||
It failed before the flag existed too, but only on clap's error, which is not
|
||||
the discriminating failure. The discriminating one is S2.
|
||||
|
||||
### a_csv_decimal_measure_imports_as_its_number
|
||||
|
||||
improvise-8jk6.17 (F3), from the adversarial review of #321. The fixture has
|
||||
decimal CSV measures (`1000.50`, `-2.25`). With `number_in`'s `f64` branch
|
||||
disabled (integers only), no headless test failed at `efaa5050`, and a decimal
|
||||
measure was silently dropped. **F3** now fails this test and the next one.
|
||||
This is coverage, not a bug: it passed when written.
|
||||
|
||||
### import_json_on_a_csv_reads_its_numeric_column_as_a_measure
|
||||
|
||||
improvise-8jk6.17 (F1). `improvise cmd 'import-json f.csv'` reaches
|
||||
`ImportJsonHeadless`, a third CSV entry point besides `import`. **F1**
|
||||
restores its CSV branch to the JSON constructor via
|
||||
`ImportPipeline::new(Value::Array(recs.to_vec()))`. That still compiles,
|
||||
because `CsvRecords` derefs to a slice. `Revenue` then becomes a category of
|
||||
text items, and this test fails on its Data lines. At `efaa5050` only
|
||||
`import_headless_refuses_records_that_collide` failed under F1, and only by
|
||||
accident. The `## Category: Revenue` assertion names the symptom directly.
|
||||
This is coverage: it passed when written.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,249 +0,0 @@
|
||||
# Test evidence: delivery
|
||||
|
||||
What each delivery test guards and which injected fault it was seen to fail on. Keyed by test name.
|
||||
The reasoning is in `context/code/delivery.md`. Each fault was put into the source one at a time,
|
||||
the suite run, the fault reverted with `git checkout -- <file>`, and the suite run again.
|
||||
|
||||
The host-side tests are in `crates/improvise-command/src/ui/delivery_tests.rs`, the client's in
|
||||
`crates/improvise-protocol/src/delivery/client_tests.rs`, the wire's in `delivery/wire.rs`.
|
||||
|
||||
## The injected faults, by what failed
|
||||
|
||||
- **The host reports no moved keys** (`let keys = BTreeSet::new()` in `Delivery::commit`): seven
|
||||
host-side tests fail: `a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes`,
|
||||
`a_pull_only_client_follows_the_log_without_a_correction`,
|
||||
`a_notice_names_the_moved_cells_including_a_formula_cell_it_did_not_write`,
|
||||
`values_keep_every_bit_from_the_cell_to_the_client`,
|
||||
`an_oversized_notice_says_correct_and_costs_every_reader_a_correction`,
|
||||
`a_dropped_notice_is_detected_and_a_correction_repairs_it` and the random property.
|
||||
`a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction` still
|
||||
passes, on purpose: its `AddItem` makes the client refetch the window, which also fetches the write.
|
||||
- **The client treats every notice as contiguous** (`false => true` in `on_notice`):
|
||||
`a_dropped_notice_is_detected`, `a_dropped_notice_is_detected_and_a_correction_repairs_it` and the
|
||||
random property.
|
||||
- **A failed frame does not put the `Delivery` back** (`undo_frame` drops `published`):
|
||||
`a_refused_command_consumes_no_publication_number`,
|
||||
`commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001` and
|
||||
`a_refused_frame_is_never_read_back`.
|
||||
- **A moved correction replaces instead of subtracting held** (`target` for `target - held_there`):
|
||||
`a_moved_correction_applies_only_at_the_watermark_it_starts_from` and
|
||||
`a_moved_key_the_target_no_longer_holds_is_removed`. The same for a window correction (`target` for
|
||||
`target - held`): `a_correction_equals_the_target_and_a_second_one_is_the_zero_zset`,
|
||||
`a_correction_replaces_what_is_held_not_adds_to_it` and
|
||||
`a_correction_already_covered_is_idempotent_and_never_regresses`.
|
||||
- **A wire number loses the sign of zero** (`Num::inexact(x + 0.0)` in `WireNum::of_f64`):
|
||||
`a_float_crosses_in_canonical_text_with_the_sign_of_zero` and
|
||||
`a_float_keeps_every_bit_but_the_nan_payload`.
|
||||
- **`covered` ignores the log's start and the shape's liveness**:
|
||||
`a_watermark_the_log_covers_gets_the_moved_keys_and_one_it_does_not_gets_the_window` and
|
||||
`an_evicted_shape_is_announced_and_its_reader_corrects`.
|
||||
- **The client does not remember an axis change** (`restructured` never set):
|
||||
`an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window`. The host's own
|
||||
`restructured` test alone fails only the random property, since the client's `window` flag also forces
|
||||
a window.
|
||||
- **`Stamp::same_epoch` ignores the host instance**:
|
||||
`a_notice_from_another_host_or_generation_asks_for_a_correction`,
|
||||
`a_stamp_from_another_epoch_is_behind_every_current_one`,
|
||||
`a_reconnect_after_a_host_restart_asks_for_a_correction_from_the_old_watermark` and
|
||||
`a_notice_from_a_restarted_host_asks_for_a_correction_too`.
|
||||
|
||||
## Entries
|
||||
|
||||
### a_refused_frame_is_never_read_back
|
||||
|
||||
The first fixture filled the document to the category limit and then refused the frame at its basis
|
||||
check, so the write never ran and the test passed with the `Delivery` not restored. A fixture that fails
|
||||
in a wider shape than the fault: the frame never reached `mutating_frame`. It now writes North, then
|
||||
loads a missing document, and asserts `Rejection::MutationRefused` so the write is known to have run. The
|
||||
rival above fails it with `77.0` held.
|
||||
|
||||
### a_corrected_state_is_the_target_and_correcting_it_again_is_the_zero_zset
|
||||
|
||||
The random property: histories of up to twelve single-command frames (writes of `0.0`, `-0.0`, both
|
||||
infinities, NaN and `0.1 + 0.2`, item adds and removes, a formula), a random window, notices delivered
|
||||
or dropped, connections lost, requests dropped, replies dropped and garbled. It ends with a `poll`, then
|
||||
asserts the client holds exactly what the grid draws in its window, and that `target - held` is the zero
|
||||
Z-set. The oracle (`drawn`) reads `GridLayout` itself and shares nothing with `Output`.
|
||||
|
||||
Without the closing `poll` a lost last notice is not detectable by a listener, and the property
|
||||
reported it as a stale client (minimal case: one `Remove`, notice dropped, no later notice).
|
||||
|
||||
The first version of the property also failed on a window of one cell after a remove, and that was a
|
||||
real gap in the correction, not in the test: the remove moved South into the window without moving its
|
||||
value. It is the reason for the `restructured` rule (`delivery.md` §3).
|
||||
|
||||
### a_frame_of_two_commands_publishes_one_notice_each_and_no_client_asks_for_a_correction
|
||||
|
||||
Step 8's first required test. The frame's first command carries one basis stating both records
|
||||
(`Host`'s frame rule: the first mutation's basis stands for the frame), which the test helper builds.
|
||||
|
||||
### commands_before_a_refused_one_are_rolled_back_with_their_notices_under_adr_001
|
||||
|
||||
Step 8 requires that the commands before a refused one stay published. Today's host undoes the whole
|
||||
frame (ADR 001 clause 4), so this test asserts the opposite and names the reason. Flip it when ADR 006
|
||||
makes each command commit on its own.
|
||||
|
||||
### a_dropped_connection_or_a_failed_request_triggers_a_correction_and_the_session_resumes
|
||||
|
||||
The three transport faults (request dropped, reply dropped, reply unreadable) each leave the client
|
||||
asking for `Ask::Correction` from the watermark it held, and the next request carries the same session
|
||||
token and that watermark.
|
||||
|
||||
### an_axis_change_that_shifts_an_unmoved_cell_into_the_window_refetches_the_window
|
||||
|
||||
A one-cell window, a remove that moves South into it. The notice lists only North, so a client that
|
||||
fetches moved keys alone ends with an empty window.
|
||||
|
||||
## Review of #348, round one
|
||||
|
||||
Each test below was written against the behaviour the review reported and failed before its fix, or, for
|
||||
the cost tests, was run against a quadratic rival.
|
||||
|
||||
### diffing_a_category_compares_each_item_a_bounded_number_of_times
|
||||
|
||||
Items are a type whose `eq` is counted. Rival: `contains` on the slices in `diff_items`, which makes the
|
||||
count quadratic and fails every size. A wall-clock test would have passed on a fast machine.
|
||||
|
||||
### a_commit_with_nothing_live_and_no_transport_reads_nothing
|
||||
|
||||
Counts `Structure::of` calls. Rival: the idle skip removed.
|
||||
|
||||
### the_outbox_and_the_clone_a_frame_takes_are_bounded
|
||||
|
||||
Rival: the outbox truncation removed, so twenty commits leave twenty queued.
|
||||
|
||||
### a_late_window_reply_from_an_older_generation_does_not_regress
|
||||
|
||||
Rival: the older-generation guard disabled (fails this test in the protocol crate).
|
||||
|
||||
### a_late_notice_from_a_dead_host_settles_against_the_live_one
|
||||
|
||||
Rival: the duplicate arm no longer prunes other epochs' entries; the random property also fails.
|
||||
|
||||
### a_poll_for_one_shape_leaves_the_other_shapes_alone
|
||||
|
||||
Rival: pulled notices applied to every shape.
|
||||
|
||||
### a_transpose_empties_a_partial_window_of_the_cell_that_left_it
|
||||
|
||||
A transpose moves no key. Rival: `flags.views` not counted as a structural change in the client and the
|
||||
host. `a_dropped_views_notice_is_answered_with_the_window_not_an_empty_key_list` fails with the host
|
||||
alone changed.
|
||||
|
||||
### wide_integers_are_canonical_on_input
|
||||
|
||||
Both spellings are refused outside their range: a JSON number at or above 2^53, a string below it, a
|
||||
sign, leading zeros and trailing space.
|
||||
|
||||
### an_item_insert_position_crosses_as_a_wide_integer
|
||||
|
||||
`ItemInserted.at` is a `u128` through `wide`.
|
||||
|
||||
### a_fetch_covers_only_the_notices_it_named_when_a_later_commit_is_already_in_the_state
|
||||
|
||||
Rival: a fetch answered with `through: at`. The first version of the random property did not catch it (it
|
||||
delivered each notice before the next commit); the property now splits, delays, duplicates and replays
|
||||
notices and old replies, and fails the rival in most runs (17 of 20 at 256 cases, measured on the
|
||||
author's machine); this test fails it on every run, by name.
|
||||
|
||||
### equal_values_compare_equal_whatever_the_zero_counts_say
|
||||
|
||||
`ExactSum`'s equality compared the sign-of-zero counters even when the total was nonzero.
|
||||
|
||||
## Review of #348, round two
|
||||
|
||||
### a_late_window_reply_from_a_dead_host_does_not_regress
|
||||
|
||||
A client that has applied a correction from host 2 receives a late `Window` reply from host 1. Epochs of
|
||||
different hosts are unordered, so the first fix (older generation of the same host) did not cover it.
|
||||
Rival: the `left` check removed; this test and the two below fail.
|
||||
|
||||
### a_reply_from_a_generation_the_client_left_does_not_return
|
||||
|
||||
Generations 1, 2, 3 of one host; a late generation-2 reply. Rival: both guards removed.
|
||||
|
||||
### a_reply_held_across_a_host_restart_cannot_pull_the_client_back
|
||||
|
||||
The host-side form: the reply is computed and lost, the host restarts, the client settles against the new
|
||||
host, and the old reply then arrives. Rival: the `left` check removed.
|
||||
|
||||
**The random property, round two.** The property now checks after every step that the watermark's epoch never goes back to one it had left
|
||||
(so a mid-history regression cannot be repaired by the closing `poll`), and replays an earlier reply after
|
||||
every restart and swap. Rival: the `left` check removed fails it. The older-generation guard on its own is
|
||||
not separable from `left` in the histories the property reaches, so it is held by
|
||||
`a_late_window_reply_from_an_older_generation_does_not_regress` alone.
|
||||
|
||||
## The left-epoch livelock
|
||||
|
||||
### a_late_reply_from_a_host_the_client_never_joined_cannot_strand_it
|
||||
|
||||
The property test failed with `Exhausted` on a shrunk history of two restarts and four writes. Cause: the
|
||||
client was on host 3, never having joined host 2; a late host-2 reply was applied (an unknown epoch is
|
||||
accepted), which wrote host 3 into `left`, and every later answer from the live host 3 was then refused as
|
||||
a duplicate. The test replays that history by hand: three hosts, the host-2 reply lost in flight, the client
|
||||
settling on host 3, the late reply applied, the connection lost, and a sync that must settle on host 3's
|
||||
grid. Rival: `apply_answer` treated like `apply` (the `left.retain` removed) fails this test. **The random
|
||||
property does not catch it**: it passed 11 of 11 runs under that fault, and again after the late reply was
|
||||
made an explicit event (a stashed reply from a host the client never joined, replayed before a lost
|
||||
connection), because the property rarely produces a reply from a host the client never joined. This
|
||||
test is the pin. Rival: the left-epoch guard removed fails
|
||||
`a_reply_held_across_a_host_restart_cannot_pull_the_client_back` and the property, as before.
|
||||
|
||||
The property's invariant changed with it. "The watermark never returns to an epoch it left" was false by
|
||||
design once an unknown-epoch reply can move the client, so the property now checks the narrower, true
|
||||
rule: a replayed reply from an epoch the client was in and has left must be a duplicate.
|
||||
|
||||
## The old-notice livelock
|
||||
|
||||
### a_late_notice_from_an_older_generation_cannot_rewind_the_client
|
||||
|
||||
The property failed with `Exhausted` on a shrunk history (two swaps, a transpose, a limit change, the first
|
||||
swap's notice delivered three steps late, an old reply replayed). The client had settled on generation 2 when
|
||||
the generation-1 notice arrived: `seen` was rewound into generation 1, a notice from the same history set
|
||||
`restructured`, and the window correction that answered was covered by the watermark, so it never cleared
|
||||
the mark and `poll` retried until `Exhausted`. The test replays the history step for step, with the
|
||||
property's own weather and a Flaky edge. Rival at the time: the notice-ignore test in `accept` disabled;
|
||||
this test and the random property failed. **Since the round-three change (a foreign notice only marks the
|
||||
shape stale) nothing fails with that shortcut disabled**: it is an optimisation that saves one round, not a
|
||||
correctness rule. The test still replays the seed. The seed file is not committed
|
||||
(`proptest-regressions/` is ignored).
|
||||
|
||||
Run after the fix, the property passed 4 x 4000 cases and 2 x 20000 cases, with no third failure.
|
||||
|
||||
## Round three: the same livelock by other routes
|
||||
|
||||
### route_a_a_dead_hosts_notice_cannot_make_the_live_hosts_old_notices_look_new
|
||||
|
||||
A client goes host 1 to host 3 and never joins host 2. A late host-2 notice rewinds `seen`, then host 3's
|
||||
own old notices look new and set `restructured` at the watermark, and the answers are duplicates that
|
||||
never clear it. Failed on the head before the change (`wants` stayed `Fetch { window: true }`).
|
||||
|
||||
### route_b_a_covered_reply_cannot_set_seen_below_a_queued_entry
|
||||
|
||||
The same rewind, then an unsolicited covered correction: the duplicate branch set `seen` to its `through`,
|
||||
below a queued entry, and each fetch for that `through` came back a duplicate. Failed on the head before
|
||||
the change.
|
||||
|
||||
### a_foreign_notice_marks_the_shape_stale_and_the_home_hosts_answer_settles_it
|
||||
|
||||
A foreign notice sets `stale`, and the home host's answer then settles the shape. It does not see `seen`
|
||||
stay put (an answer heals it either way); route A and route B are what pin that.
|
||||
|
||||
### notices_cut_for_different_shapes_or_not_cut_do_not_merge
|
||||
|
||||
A merge of notices cut for different shapes, or a cut one with an uncut one, would keep one shape's
|
||||
mark and make the other shape see a gap. `merge` refuses it. Rival: the `cut_for` comparison removed.
|
||||
|
||||
### a_pulled_notice_pushed_through_on_notice_still_reaches_only_its_shape
|
||||
|
||||
A notice cut for shape `a` by `for_shape`, delivered through `on_notice`, must leave shape `b` alone.
|
||||
Failed before `Notice::cut_for` existed.
|
||||
|
||||
### two_shapes_survive_late_repeated_and_dead_messages
|
||||
|
||||
The reviewer's generator, kept: two shapes, notices late, repeated and reordered (the due set is reversed
|
||||
on odd steps), dead hosts' notices kept across restarts, stored notices and stored replies replayed on
|
||||
their own, hosts that restart often. At the default 256 cases it did not find the old head's livelock, and
|
||||
at 20,000 it did within ten seconds (an `Exhausted`), so the default run is a smoke test and the deep run
|
||||
is the check. After the change it passed three runs of 20,000 cases, and the older property passed
|
||||
2 x 20,000 with it.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user