From 322875b30affcf751b8a43148d6d3694a026e8d3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:14:37 -0700 Subject: [PATCH 001/106] docs(adr): ADR 009 incremental evaluation (proposed) (improvise-felo.4) An in-house DBSP core as a problem-agnostic crate, plugged in behind a trait in improvise-core, with the eager evaluator kept as the oracle. Adds examples/retraction_drift.rs, which measures f64 retraction drift against the eager fold and the exact decimal sum. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 612 ++++++++++++++++++ .../examples/retraction_drift.rs | 191 ++++++ scripts/build-doc-site.mjs | 1 + 3 files changed, 804 insertions(+) create mode 100644 context/adr/009-incremental-evaluation.md create mode 100644 crates/improvise-core/examples/retraction_drift.rs diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md new file mode 100644 index 00000000..b61fe0ad --- /dev/null +++ b/context/adr/009-incremental-evaluation.md @@ -0,0 +1,612 @@ +# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait + +**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end +are theirs to answer on the PR. · +**Ticket:** improvise-felo.4 · +**Decision from:** the user, 2026-09-25 (below) · +**Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · +**Relates to:** ADR 005 (exact decimals, improvise-l2b8, no file yet), ADR 008 (the +virtualized grid, PR #308, which is revised to assume this ADR) · +**Governed by:** `context/design-principles.md` §3 and §5 + +**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl and have +no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +no open PR claimed it when this was written. + +**Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are +dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means +revision 2 of `context/adr/008-virtualized-grid.md` on PR #308. + +## The decision this ADR writes down + +The user pointed at a teaching implementation of the DBSP paper, +[brurucy/dbsp-from-scratch](https://github.com/brurucy/dbsp-from-scratch/blob/master/dbsp_paper_walkthrough_implementation.ipynb), +as a way to evaluate formulas so that data can stream. The notebook builds, in order: + +- abelian groups; +- streams, with `Delay` (z⁻¹), `Differentiate` and `Integrate`; +- Z-sets, which are maps from element to an integer weight; +- lifted and doubly-lifted joins, and a delta join that keeps less state; +- `distinct` through the `H` function over an integrated state; +- recursion by nested streams, shown on graph reachability. + +The user then decided: + +> "I think doing our own implementation of the core of dbsp as a crate we can plugin is +> probably dieal [sic] here. Write the ADR and Update ADR 008 to assume the decisions in +> that one" +> +> — the user, 2026-09-25, recorded on improvise-felo.4 + +So the choice between Feldera's crate and our own is already made (see Alternatives for +what was checked). This ADR says what "the core of dbsp" is for improvise, where it +plugs in, and what it may not change. + +## Context + +### Evaluation today is a from-scratch fold, rerun after every batch + +- **Every batch recomputes every formula cell.** `App::apply_effects` + (`crates/improvise-command/src/ui/app.rs:989`) ends in `rebuild_layout` (`:880-891`). + That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). + It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), + and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` + (`:699`, `:1001-1022`). A cursor move and a cell write pay the same. +- **Raw aggregation scans the matching cells on every read.** `evaluate_unfolded` + (`types.rs:837-845`) folds `DataStore::matching_values` through + `aggregate_cell_values` (`:30-42`). `fold_stored` (`:918-930`) does the same for a + collapsed group, through `spanned_values` (`crates/improvise-core/src/model/cell.rs:445`). +- **`formula_cache` holds one aggregation shape at a time.** It is keyed by cell and + computed for one `none_cats` list. A session on another view with a different None + axis gets a *cloned model* recomputed for its own list (`rederived_for`, + `crates/improvise-command/src/ui/project.rs:195-210`). +- **A formula total with an aggregate inside walks a cross product.** `contexts_of` + (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every + combination of the spanned categories' items. ADR 008 clause 5 names this as the one + O(grid) path left in totals. +- **Two of the record tiers are observed by scanning the store.** `record_fingerprints` + (`app.rs:254-264`) hashes every cell before and after a cell-writing batch, and + `document_content` (`app.rs:165-178`) does the same for the dirty flag. Both are + O(cells) per batch, whatever the batch wrote. + +### What an incremental evaluator must not change + +These rules come from the repo map and `context/code/core.md`. Each one is a place an +incremental evaluator could quietly disagree with the eager one. + +- **Gotcha 7, text unanimity.** `aggregate_cell_values` checks the all-numeric branch + first. Otherwise one distinct value is that value, and anything else is + `Error("mixed")`. `Count` of a lone number is `1`, not the number. +- **Gotcha 9 and core.md §3, circularity.** A cell still moving after 16 passes is + written as `Error("circular")` (improvise-7iww). **A chain that needs all sixteen + passes is also reported circular**, because the fixed point can only confirm + convergence with a pass that changes nothing. The direct evaluator agrees because its + recursion depth is the same 16 (`types.rs:705-714`). +- **Gotcha 17, one epsilon.** `FLOAT_EQ_EPSILON = 1e-10` (`types.rs:14`) is shared by + `=`, `!=` and the div/0 guard, in both evaluators. +- **Gotcha 18, ordering is correctness.** A `HashSet` index is for lookup, never for + iteration. `SUM` and `AVG` drifted in the last bits when the store was walked in hash + order (improvise-shyt), because f64 addition is not associative. +- **Gotcha 40, the tiers are observed.** A write that lands the value already there + bumps nothing. The record set is collected into a *set*, so a rewritten record counts + once. + +## Decision + +**A new, problem-agnostic workspace crate implements the DBSP core. improvise-core +defines a trait for the derived state it reads. Today's evaluator is one implementation +of that trait, and a circuit built on the new crate is the other. Both stay in the tree, +and a differential test holds them equal.** Eleven clauses. + +### 1. The crate: `zset` (name is question 1) + +Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond +`std`, so it is wasm32-clean by construction. + +**What it contains:** + +- **`ZSet`**: a map from `K: Ord + Clone` to a non-zero `i64` weight, with `add`, + `neg` and `zero`. It forms an abelian group, and its laws are property-tested as the + notebook's first cell does: commutativity, associativity, identity and inverse. + Weights use checked arithmetic, so an overflow is a panic, not a wrap. A weight that + reaches zero is removed, so equal Z-sets are equal as maps. +- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one input + delta per source and returns one output delta per sink. There is one clock, one + thread, and no scheduler. +- **Stream operators:** `delay` (z⁻¹), `integrate` (I) and `differentiate` (D), with + I∘D = D∘I = id tested as a property. +- **Linear operators, which are their own incremental form:** `map`, `filter` and + `flat_map`. +- **`index_by(key)`.** It turns a `ZSet` into an indexed Z-set `K → ZSet`, which + is the input shape for grouping and joins. +- **`join_indexed`.** The bilinear delta join: Δ(a ⋈ b) = Δa ⋈ Δb + I(a)₋₁ ⋈ Δb + + Δa ⋈ I(b)₋₁, over integrated indexes. It is needed for one purpose, clause 5's + cross-stem references. +- **`distinct`.** It is `H` over the integrated input, as in the notebook. The output + delta is non-empty exactly when an element's total weight crosses zero. +- **`aggregate(group, Aggregator)`.** Its output delta retracts the group's old value + and inserts its new one. The trait has two kinds of implementation: + - **linear:** `Count`, and `Sum` for any `G: AbelianGroup`. The state is one value + per group, and each delta element costs O(1). + - **integrated:** the aggregator keeps the group's integrated multiset, + `BTreeMap`, and reads its answer off it. `Min` and `Max` are the first and + last keys. Unanimity (clause 4) is "exactly one distinct key". Each delta element + costs O(log d), where d is the number of distinct values in the group. + +**What it leaves out, deliberately:** + +- **Nested streams and least-fixed-point recursion** (the notebook's + `LiftedLiftedJoin`, `DeltaLiftedDeltaLiftedDistinct` and `IncrementalGraphReachability`). + improvise's circular semantics are not a least fixed point (clause 6). +- **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies + batches in one total order (ADR 001, "There is already a total order"). +- **Persistence, spilling to disk, and threads.** State lives in memory, and a browser + worker has one thread. +- **A query language or planner.** Circuits are wired by hand in the seam crate + (clause 2). + +**The weight type is `i64`.** Integer weights make `ZSet` an exact group, so +consolidation and retraction never drift. The drift problem is in the *values* being +summed, not in the weights (clause 8). + +### 2. The plug: a trait in improvise-core, and a seam crate that implements it + +**improvise-core gains a trait for the derived state its evaluator reads.** The working +name is `Derived`. It replaces the two places where eager evaluation folds the store: + +- the formula result at a key, which is `formula_cache` today (`types.rs:370`); +- the raw fold of the cells matching a partial key under an aggregation, which is + `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). + +It has one mutation, `apply(&mut self, delta: &CellDelta)`, plus a rebuild for the +changes that are not cell deltas (clause 7). + +**The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code +and read through the trait. The circuit decides *which* formula cells to re-evaluate +and supplies their inputs, and the existing per-cell evaluator computes the value. +Formulas are never compiled into operators. A compiled `IF`/`POW`/`WHERE` would be a +second implementation of evaluation, which design-principles §5 calls the defect +itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental totals. + +**Two implementations:** + +- **`Eager`** is today's code moved behind the trait, with no behaviour change: `apply` + records the delta, and the post-batch refresh is `recompute_formulas`. +- **`Incremental`** lives in a seam crate, working name `improvise-incremental`, which + depends on `zset` and `improvise-core`. This is the `gp-tree`/`improvise-gp` shape: + improvise-core never depends on `zset`, and it can be built and tested eager-only + (question 2). + +**Why keep both.** `Eager` is the oracle for the differential test. It is also the +backend a query-only source keeps (clause 10). The differential test is a property test +in the seam crate. It generates documents with groups, collapse, hidden items, formulas +(including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text +cells and non-finite numbers, applies the same random edit sequence to both backends, +and compares every drawn cell, every column total and every occupancy count. + +### 3. The input: a cell write is a retraction plus an insertion + +**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue)>` with weights +±1.** `DataStore::set` and `DataStore::remove` (`cell.rs:286`, `:325`) are the only +places a cell changes, so the delta is emitted there: + +- a new cell is `+(k, v)`; +- an overwrite of `v₀` with `v₁` is `−(k, v₀) + (k, v₁)`; +- a removal is `−(k, v₀)`. + +An overwrite with an equal value consolidates to the empty Z-set. The circuit steps once +per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy +inside the effect loop (`app.rs:1073-1084`). + +**This lines up with ADR 001's record tier with no extra rule.** The records a batch +moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys +in the batch's consolidated delta. So: + +- a write of the value already there moves nothing, which is gotcha 40's rule; +- a record gained, lost or rewritten is named once, because a Z-set is already a set of + keys. + +`bump_changed_records` can then read the delta instead of diffing two +`record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta +comes from the mutation sites, not from an `EffectKind`. The same holds for the content +half of the dirty flag, where a non-empty delta means dirty. `layout_version` is +unchanged, because it observes `view_shape` and not cells. + +**`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` +(`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only +`PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by +`f64::total_cmp`. Clause 4 says where that differs from `==`. + +### 4. Aggregates: linear where they can be, an integrated multiset where they cannot + +**The group** is the cell key with the view's `none_cats` removed. That is the partial +key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct +`none_cats` list, shared by every view that has that list. That replaces +`rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group +comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does +(`types.rs:717-722`). + +**What each group keeps:** + +| answer | state per group | cost per delta element | exact on f64? | +|---|---|---|---| +| `Count` | count of values, count of numbers | O(1) | yes (integers) | +| `Sum` | running sum | O(1) | **no** (clause 8) | +| `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | +| `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | +| unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | + +Reading a group reproduces `aggregate_cell_values`'s branch order: + +- if the count of numbers equals the count of values, the numeric answer; +- else if the value multiset has exactly one distinct key, that value; +- else `Error("mixed")`. + +The empty group is absent. The formula-level `eval_agg` (`types.rs:55-87`) reads the +same state with its own emptiness rule, `0` for `Sum` and `Count`, at read time. + +**Memory.** Summed over all groups, the multisets hold at most one entry per stored +cell. So the state for one `none_cats` list is O(cells), on the order of the store it +summarizes. **Estimate, not measured:** a `BTreeMap` entry of about 40–60 bytes plus the +key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the measures +whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A +`Sum`-only document keeps one number per group. + +**`total_cmp` is not `==`, and the differential test must say so.** `NaN != NaN`, so +today two NaNs in one group read as `mixed`, and `f64::min` skips a NaN. A multiset keyed +by `total_cmp` would call two NaNs one value and would order NaN at an end. The test +generates non-finite values (core.md §16), and the aggregator matches eager +behaviour explicitly rather than inheriting `total_cmp`'s. + +### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it + +**Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key +with the target category and each `none_cats` entry removed. That is +`distinct(map(project))` over the cell delta. A stem enters when its first cell arrives +and leaves when its last cell goes. + +**A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula +`T` at stem `s` are: + +- for each `Ref(R)`, the group `s.with(cat(R), R)`. `cat(R)` is resolved once per + formula set through `find_item_category`, including its `_Measure` fallback for + formula targets (`types.rs:725-740`). Whether `R` is derived (`formula_for`) is also + decided per formula set, so a ref to another formula becomes an edge between formula + nodes, and a ref to raw data becomes an edge from an aggregate group. +- for an `Agg` with no filter, the group at `s` itself, because `eval_agg` pools over + `context.without(target_category)`. +- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at + `s.with(Cat, Item)`. That is a different stem, so a delta at `Cat = Item` reaches + every stem that differs only in `Cat`. **This is the one join**: stems indexed by + "stem minus `Cat`", joined with the aggregate's output delta. + +The affected `(stem, formula)` pairs are consolidated, ordered topologically over the +**formula graph**, and handed to the existing evaluator. Its cached path already reads +refs through `formula_cache` and falls back to raw aggregation (`types.rs:1066-1086`). +With the trait, both reads go to the maintained state. + +**A formula's own `WHERE`** is decided per cell, as today. A cell whose coordinate does +not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own +group and nothing else. + +**An aggregate over an expression replaces `Expansion::members`.** `SUM(expr)` pools +`expr` evaluated at each member record (`agg_values`, `types.rs:89-139`). Incrementally, +that is `map(record ↦ expr at record)`, then `index_by(base key)`, then `Sum`. The +member records never have to be listed. **The boundary:** the map is sound only when +`expr` reads the record's own cells. An inner expression that reads another record, or +holds a nested aggregate, stays on the eager path for that formula, and the +differential test pins the boundary. Until `Sum` is exact, this path carries clause 8's +caveat. ADR 008 clause 5 adopts the same boundary. + +### 6. Circular references: DBSP's recursion is not our semantics + +DBSP handles recursion by computing a **least fixed point**: nested streams iterate a +monotone operator, and `distinct` keeps the lattice finite, so reachability converges. +improvise has no such guarantee: + +- `A = B + 1` with `B = A + 1` has no fixed point at all; +- `X = X + 1` over a stored `X = 10` climbs forever; +- the required answer is `Error("circular")`, not a limit or a non-terminating step. + +So the crate has no recursion operator, and circularity is decided *outside* it: + +- **The formula graph** (target → the targets its refs name) is rebuilt when the formula + set changes, and its strongly connected components are found (Tarjan). Formulas + outside every cycle are evaluated in topological order, each exactly once per step. +- **A formula in a cycle, a self-loop included, has its affected cells evaluated by + today's bounded fixed point, restricted to that component.** A formula-level cycle is + not always circular at every cell. `A = IF(x, B, 1)` with `B = A` converges wherever + `x` is false. The per-cell fixed point is the only thing that gets those cells right, + and reusing it keeps `Error("circular")` exactly as it is. +- **The depth bound is part of the semantics today.** An acyclic chain that needs all + sixteen passes is reported circular by both eager evaluators (core.md §3). A + topological evaluator would compute it correctly, and so would disagree with eager. + Question 4 asks whether to reproduce the bound. The recommendation is to reproduce + it, by carrying each formula's topological depth, so the differential test compares + like with like. Changing the rule would be its own decision, for both backends at + once. + +### 7. What is not a cell delta rebuilds, once + +Some changes alter the circuit's *shape*, not its input: + +- the formula set: add, remove or replace; +- `measure_agg`; +- category and item structure: add or remove a category, remove an item, and group + membership; +- a view's axes, hidden items and collapsed groups, which change `view_shape`; +- a document swap, `document_generation`. + +Each rebuilds only the affected operators, from the integrated base Z-set. That costs +O(cells), once per change, and is never repeated per keypress. **The trigger is the same +observation the tiers use**: a `view_shape` diff (`app.rs:40-72`) rebuilds that view's +operators, the same diff that bumps `layout_version`. So there is one signal and not two. + +**A page selection is not a rebuild.** The group key includes the page categories, so +`[` and `]` read another slice of state that already exists. The held-key path stays +O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the +state is still bounded by the stored cells, however many pages there are (question 7). + +### 8. Floating point: retraction drifts, and exactness is ADR 005's + +f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum +the eager fold computes. **Measured**, not asserted: +`cargo run --release -p improvise-core --example retraction_drift` +(`crates/improvise-core/examples/retraction_drift.rs`, added by this PR). It was run on +an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N +retract-and-insert cycles. It compares the sum against the eager fold, +`evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count +of 10⁻⁵ units rounded to f64 once. Values are signed two-place decimals spread over 3 or +9 decades. + +| store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | +|---|---|---|---|---| +| 1,000 cells | 3 decades | 6.4e-11 – 1.3e-10 (35–140 ulp) | in 1 of 2 seeds | ≤ 7.3e-12 | +| 18,000 cells | 3 decades | 1.7e-9 – 2.3e-9 (57–80 ulp) | yes | 1.5e-10 – 2.6e-10 | +| 18,000 cells | 9 decades | 2.3e-5 – 3.2e-4 (3–42 ulp) | yes | 2.4e-4 – 2.5e-4 | + +A fourth case holds 1e17, then adds 1,000 cells of 0.01, then retracts the 1e17. The +naive running sum reads **0.00**, and the eager fold and the exact sum both read +**10.00** (at `,.2`). The small values were absorbed and never came back. + +A Neumaier-compensated running sum recovers that case exactly. In the random cases it +stays about as close to the exact sum as eager does, **but it is never bit-equal to +eager**, because eager folds in store order and is itself inexact. + +Four conclusions follow: + +- **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by + any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, + unanimity and occupancy bit for bit today, because they are order-free. It can compare + `Sum` and `Avg` only with a tolerance. +- **The gap reaches the epsilon.** In most 18k-cell cases the incremental sum and the + eager sum differ by more than `FLOAT_EQ_EPSILON`. So `IF(SUM(..) = x, ..)` can answer + differently on the two paths. That is a behaviour difference, not a display one. +- **This is already a bug in eager, which the measurement found in passing.** The eager + 18k-cell sum is 1.5e-10 to 5.8e-10 from the exact decimal sum, so `=` against a + decimal a user typed can already be wrong today. That supports ADR 005 on its own + merits, and it is recorded here for improvise-l2b8. +- **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true + `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's + whole class), and the differential test is bit-exact everywhere. + +**So ADR 005 is a prerequisite for the maintained `Sum`/`Avg` on the displayed path, but +not for the rest of this ADR** (question 3). Occupancy, `Count`, `Min`, `Max`, +unanimity, stems and invalidation can all land before it. Until ADR 005 lands, a `Sum` +or `Avg` the grid displays, including a column total that holds one, is read from the +eager fold. The maintained sum runs only inside the differential test, under a stated +tolerance. If the user wants maintained sums before ADR 005, the fallback is a +compensated sum that re-folds a group from its values when cancellation is detected. +The measurement above is the evidence for and against that fallback. + +### 9. What ADR 008 gets from it + +ADR 008 (revision 2, PR #308) assumes these, under the shared names below. + +- **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and + per column tuple of cells that hold a value. Stored cells pass the view's filter, with + hidden items dropped, and a collapsed group's members map to the group's tuple. + Formula cells count at their stems, which is today's `formula_cache` stem set. A tuple + is on the axis iff its count is above zero. **This ADR does not settle ADR 008 + question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those + cells are values with no stored cell under them. The stem definition drops them, and + if the user rules otherwise, those views fall back to a non-occupancy path. Occupancy + is integer and order-free, so it does not wait for ADR 005. +- **The occupancy delta** (ADR 008 clause 8). `distinct` over occupancy emits a + non-empty delta exactly when a tuple enters or leaves an axis. A reframe is a + non-empty occupancy delta. The two-arm asymmetry of the frame observation (gotcha 24, + improvise-mcq4) is unchanged, but each arm costs O(delta) instead of a rebuilt grid + per view. +- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over + the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), + not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) + aggregate, and the formula layer re-evaluates only the columns whose inputs moved, + through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that + hold a `Sum` or `Avg` wait for ADR 005 (clause 8). +- **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, + or maintain) is struck there as decided by the user through this ADR. Its questions 1 + and 2 stay open there. +- **What it does not get.** Column widths depend on formatted text, which is + display-only (gotcha 8), so they are not maintained here. They stay ADR 008 question 2. + +### 10. ADR 003's live sources are delta streams, when they can be + +A live source that can report its changes, by polling and diffing, by triggers, or by a +change feed, produces a cell delta. That is the same input `DataStore` produces, and the +circuit cannot tell them apart. That is the "data can stream" half of the user's +prompt. + +**A source that can only answer aggregate queries stays on `Eager`,** with ADR 003's +pushdown. There is a tension: ADR 003 wants data larger than memory, and every +integrated operator here keeps state proportional to its input (clause 4). So the +backend is chosen per document, which is what "pluggable" buys (question 8). A +read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. + +### 11. Constraints + +- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset` and + `improvise-core`. If improvise-command selects the backend, both join the browser + bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds + them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). +- **Two seconds per test binary.** The crate's laws and its "incremental equals + from-scratch" properties run at the default 256 cases over small inputs. The + differential test lives in the seam crate's own binary, so its cost is visible on its + own. Its document sizes are bounded by the strategy, never by raising or lowering the + case count (design-principles §7). +- **Ordering determinism (gotcha 18).** A `ZSet` iterates in `K`'s `Ord` order, never + in hash order. A hash index is allowed for lookup only. Output deltas are + consolidated and sorted before they leave an operator. Nothing in the circuit decides + a display order: record order stays `DataStore`'s (`live_records`, `cell.rs:366`). + The determinism test builds **two** circuits independently from the same history and + compares them. One circuit compared with itself proves nothing (improvise-hzjo's + vacuity). +- **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` + (`app.rs:647`). The backend's state is part of the snapshot. Undoing by applying the + inverse delta is exact for integer weights and order-free aggregators, but not for an + f64 `Sum`, so the snapshot is kept until clause 8's prerequisite lands (question 9). +- **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the + delta sequence, so replaying commands reproduces its state. + +## Alternatives considered + +**Feldera's `dbsp` crate.** What was verified on 2026-09-24: + +- The latest release is 0.354.0, a 4.8 MB compressed tarball, MIT OR Apache-2.0, + requiring Rust 1.96.1. +- It has 67 direct normal dependencies, none optional, and two features + (`backend-mode`, `default`). tokio, mimalloc, nix, zstd, libc and core_affinity are + all required. +- `cargo tree -e normal` resolves about 455 crates natively and 421 for + `wasm32-unknown-unknown`. For comparison, improvise-core's whole normal tree for wasm32 + is 25 crates, and improvise-command's is 68. +- `cargo check` succeeds natively. **`cargo check --target wasm32-unknown-unknown` + fails**: `mio` (tokio's) reports 48 errors. + +What was **not** verified: whether anything past `mio` would also fail. The build stops +at the first failure, so that is estimated, not measured. It is likely, because +mimalloc's sys crate and zstd build C code, and `libsqlite3-sys` already fails under this +repo's nix cc-wrapper (gotcha 41). Even if it built, it is a multi-threaded, +storage-backed query engine where improvise needs a single-threaded operator library. +The user's decision already excludes it. This records why that decision loses nothing. + +**Keep eager evaluation and memoize.** improvise-swre proposes keying `grid_of` on the +existing tiers, so an unchanged grid costs nothing. That is worth doing under either +backend. But every write still recomputes every formula and rescans the store, and a +second view with a different None axis still clones the model. It removes the cost of +*not* changing and leaves the cost of changing at O(cells). + +**An ad-hoc incremental cache.** Track dirty stems by hand: invalidate the groups a +write touches, and recompute them. It is cheaper to start. But each derived fact, such +as a group aggregate, occupancy, a column total, a stem or a cross-stem `WHERE`, needs +its own invalidation rule. Each rule is a second description of evaluation, free to +disagree with the first. The Z-set algebra gives **one** rule, linearity and +bilinearity, whose laws can be property-tested once in the crate. That is the same +argument as design-principles §5's "a derivation done twice is the defect". + +**Compile formulas into operators.** This is the fully DBSP-shaped answer. It is +rejected in clause 2, because it would be a second evaluator. + +## Consequences and migration + +**Order.** Each step is its own ticket and PR, with gates green, and none switches the +displayed path until the differential test holds. + +1. **The `zset` crate**, with its group laws, operator properties and determinism + test. Nothing in improvise uses it yet. +2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the + existing suite is the evidence. `CellKey` gains `Ord`. +3. **The cell delta from `DataStore`.** `bump_changed_records` and the content half of + the dirty flag move onto it. That removes two O(cells) scans per cell-writing batch. + The existing tier tests are the evidence. +4. **`Incremental`, for the order-free parts**: stems, `Count`, `Min`, `Max`, + unanimity, occupancy and the occupancy delta, under the differential test. **ADR 008 + clauses 3 and 8 (pruning and the frame observation) depend on this step.** +5. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's fallback). + **ADR 008 clause 5 depends on this step.** +6. **Formulas**: the formula graph, topological re-evaluation, the component fixed + point, and the aggregate-over-expression map that replaces `Expansion::members`. +7. **Live sources as delta streams**, when ADR 003's source trait exists. + +**Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is +question 10. improvise-swre stays independent and is useful under either backend. +improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause 8. + +**Good.** + +- A write costs O(delta) plus the formula cells it reaches, not O(cells). +- The store scans behind the record tier and the dirty flag disappear. +- Per-view model clones for a different None axis disappear. +- ADR 008 gets occupancy, the occupancy delta and column totals without a second + evaluator. +- Live sources that can report changes plug in without touching the evaluator. + +**Bad.** + +- Two backends, kept equal by a differential test that must itself be kept honest. +- Integrated state is O(cells) in memory, per distinct None axis, in addition to the + store. +- `Sum` and `Avg` cannot be maintained on the displayed path until ADR 005 lands, and + the measurement says so. +- Circularity is decided in two layers: the formula graph's components and the + per-cell bound. +- The browser bundle grows by two small crates. + +## Open questions + +Each question has a recommendation. The user's answers will be folded in under "What +the user decided", as ADRs 004 and 007 did. + +1. **The crate's name.** `zset`, `dbsp-core`, or `improvise-dbsp`? + *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not + carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. + +2. **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on + `zset` and `improvise-core`), or inside improvise-core? + *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and + `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. + +3. **Floating point.** Should ADR 005 (exact decimals) be a prerequisite for maintaining + `Sum` and `Avg` on the displayed path, with the eager fold kept for them until then? + Or should a compensated f64 sum, re-folded on detected cancellation, ship first with + a tolerance in the differential test? + *Recommendation:* ADR 005 as the prerequisite. The measured drift reaches + `FLOAT_EQ_EPSILON`, so `=` could answer differently on the two paths. No f64 technique + makes the differential test bit-exact. Everything else in this ADR lands without it. + +4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported + `circular` today (core.md §3). Should `Incremental` reproduce that? + *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to + change, change it in both backends at once, as its own decision. + +5. **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by + today's bounded fixed point, restricted to their component, rather than by any + operator in the crate? + *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the + cells a guarded cycle leaves convergent. The crate needs no recursion operator. + +6. **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? + *Recommendation:* yes, permanently. It is the differential test's oracle and the + backend for ADR 003's query-only sources. Deleting it would delete the test. + +7. **Page selection as a slice.** Should the group key include the page categories, so + switching page reads existing state, rather than rebuilding the view's operators per + page change? + *Recommendation:* the slice. The state stays bounded by the stored cells, and the + held-key path (`[`/`]`) stays O(visible). ADR 008 revision 2 assumes this. + +8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, + with query-only sources kept on `Eager` and ADR 003's pushdown? + *Recommendation:* yes, chosen per document. Integrated state is proportional to the + source, which contradicts ADR 003's larger-than-memory goal for exactly the sources + that cannot report their changes anyway. + +9. **Rollback.** For ADR 001 clause 4, should the backend's state be snapshotted with + the `App`, or undone by applying the inverse delta? + *Recommendation:* snapshot until ADR 005 lands, then the inverse delta. The inverse + is exact only when every aggregator is. + +10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps + 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by + steps 4 and 5? + *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are + how it is met. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs new file mode 100644 index 00000000..9e627378 --- /dev/null +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -0,0 +1,191 @@ +//! How far a SUM maintained by insert/retract deltas drifts from the eager fold, +//! in f64. ADR 009 (improvise-felo.4), "Floating point". + +use improvise_core::format::format_f64; +use improvise_core::model::cell::{CellKey, CellValue}; +use improvise_core::model::types::{Aggregation, Model}; + +const MEASURE: &str = "_Measure"; + +/// SplitMix64: seeded, so a run is reproducible and a second seed is a new sample. +struct Rng(u64); + +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + fn below(&mut self, n: u64) -> u64 { + self.next() % n + } +} + +/// A user-entered decimal as an exact count of 1e-5 units, its magnitude spread +/// over `decades` decades. The f64 the store holds is `typed(units)`. +fn decimal(rng: &mut Rng, decades: u32) -> i64 { + let scale = 10i64.pow(rng.below(u64::from(decades)) as u32); + let cents = rng.below(100_000) as i64; + let sign = if rng.below(4) == 0 { -1 } else { 1 }; + sign * cents * scale +} + +fn typed(units: i64) -> f64 { + units as f64 / 1e5 +} + +/// The exact decimal sum, rounded to f64 once: what ADR 005's arithmetic would show. +fn exact(units: &[i64]) -> f64 { + units.iter().map(|&u| i128::from(u)).sum::() as f64 / 1e5 +} + +fn key(row: usize) -> CellKey { + CellKey::new(vec![ + ("Row".to_string(), format!("r{row}")), + (MEASURE.to_string(), "Amount".to_string()), + ]) +} + +/// `cells` rows of one measure; the eager SUM is the one the grid draws. +fn document(rng: &mut Rng, cells: usize, decades: u32) -> (Model, Vec) { + let mut m = Model::new("drift"); + m.add_category("Row").expect("under the limit"); + let values: Vec = (0..cells).map(|_| decimal(rng, decades)).collect(); + for (row, v) in values.iter().enumerate() { + m.category_mut("Row") + .expect("just added") + .add_item(format!("r{row}")); + m.set_cell(key(row), CellValue::Number(typed(*v))); + } + (m, values) +} + +fn eager(m: &Model) -> f64 { + let total = CellKey::new(vec![(MEASURE.to_string(), "Amount".to_string())]); + m.evaluate_aggregated_f64(&total, Aggregation::OverHidden) +} + +/// Neumaier's compensated running sum: the cheap mitigation, measured beside the naive one. +#[derive(Default)] +struct Compensated { + sum: f64, + carry: f64, +} + +impl Compensated { + fn add(&mut self, x: f64) { + let t = self.sum + x; + self.carry += match self.sum.abs() >= x.abs() { + true => (self.sum - t) + x, + false => (x - t) + self.sum, + }; + self.sum = t; + } + + fn value(&self) -> f64 { + self.sum + self.carry + } +} + +fn ulps(a: f64, b: f64) -> u64 { + (a.to_bits() as i64).abs_diff(b.to_bits() as i64) +} + +/// Each running sum against the eager fold (the differential oracle) and against the +/// exact decimal sum. `flip`: the gap reaches `FLOAT_EQ_EPSILON`, so `=` can answer differently. +fn report(label: &str, cycles: usize, exact: f64, eager: f64, naive: f64, comp: f64) { + let shown = |v: f64| format_f64(v, true, 2); + println!( + "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e}", + (naive - eager).abs(), + ulps(naive, eager), + (comp - eager).abs(), + ulps(comp, eager), + (naive - eager).abs() >= 1e-10, + (eager - exact).abs(), + (naive - exact).abs(), + (comp - exact).abs(), + ); + if shown(naive) != shown(eager) || shown(eager) != shown(exact) { + println!( + " shown at ,.2: naive {} eager {} exact {}", + shown(naive), + shown(eager), + shown(exact) + ); + } +} + +/// Each cycle retracts one live value and inserts a fresh one at the same key. +fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) { + let mut rng = Rng(seed); + let (mut m, mut values) = document(&mut rng, cells, decades); + let mut naive = eager(&m); + let mut comp = Compensated::default(); + comp.add(naive); + let mut done = 0; + for &at in checkpoints { + while done < at { + let row = rng.below(cells as u64) as usize; + let new = decimal(&mut rng, decades); + naive = naive - typed(values[row]) + typed(new); + comp.add(-typed(values[row])); + comp.add(typed(new)); + values[row] = new; + m.set_cell(key(row), CellValue::Number(typed(new))); + done += 1; + } + let label = format!("{cells} cells, {decades} dec"); + report(&label, at, exact(&values), eager(&m), naive, comp.value()); + } +} + +/// A large value arrives, small ones accumulate under it, then it is retracted. +fn absorbed_by_a_large_value() { + let mut m = Model::new("absorb"); + m.add_category("Row").expect("under the limit"); + m.category_mut("Row").expect("just added").add_item("r0"); + let mut naive = 0.0; + let mut comp = Compensated::default(); + m.set_cell(key(0), CellValue::Number(1e17)); + naive += 1e17; + comp.add(1e17); + for row in 1..=1000 { + m.category_mut("Row") + .expect("just added") + .add_item(format!("r{row}")); + m.set_cell(key(row), CellValue::Number(0.01)); + naive += 0.01; + comp.add(0.01); + } + m.clear_cell(&key(0)); + naive -= 1e17; + comp.add(-1e17); + report("1e17, 1000 x 0.01, -1e17", 1001, 10.0, eager(&m), naive, comp.value()); +} + +fn main() { + println!( + "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10}", + "case", + "cycles", + "naive-eag", + "ulp", + "comp-eag", + "ulp", + "flip", + "eag-exact", + "nai-exact", + "cmp-exact" + ); + let checkpoints = [1_000, 10_000, 100_000, 1_000_000]; + for seed in [1, 2] { + random_updates(seed, 1_000, 3, &checkpoints); + random_updates(seed, 18_000, 3, &checkpoints); + random_updates(seed, 18_000, 9, &checkpoints); + } + absorbed_by_a_large_value(); +} diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index f9bab7b3..a2967458 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -46,6 +46,7 @@ const SECTIONS = [ ["context/adr/003-live-backed-models.md", "adr/003-live-backed-models", "ADR 003 — Live-backed models"], ["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"], ["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"], + ["context/adr/009-incremental-evaluation.md", "adr/009-incremental-evaluation", "ADR 009 — Incremental evaluation (proposed)"], ], }, { From 5e60e59acd5b9df028699b25f1acf8c4157c7c11 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:15:47 -0700 Subject: [PATCH 002/106] docs(adr): state ADR 009's drift counts per checkpoint; rustfmt the drift example (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 ++++++++------ crates/improvise-core/examples/retraction_drift.rs | 9 ++++++++- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b61fe0ad..802addc3 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -379,12 +379,14 @@ Four conclusions follow: any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit today, because they are order-free. It can compare `Sum` and `Avg` only with a tolerance. -- **The gap reaches the epsilon.** In most 18k-cell cases the incremental sum and the - eager sum differ by more than `FLOAT_EQ_EPSILON`. So `IF(SUM(..) = x, ..)` can answer - differently on the two paths. That is a behaviour difference, not a display one. -- **This is already a bug in eager, which the measurement found in passing.** The eager - 18k-cell sum is 1.5e-10 to 5.8e-10 from the exact decimal sum, so `=` against a - decimal a user typed can already be wrong today. That supports ADR 005 on its own +- **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum + differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So + `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour + difference, not a display one. +- **This is already a bug in eager, which the measurement found in passing.** With 3 + decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal + sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed + can already be wrong today. That supports ADR 005 on its own merits, and it is recorded here for improvise-l2b8. - **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 9e627378..92dae6e5 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -164,7 +164,14 @@ fn absorbed_by_a_large_value() { m.clear_cell(&key(0)); naive -= 1e17; comp.add(-1e17); - report("1e17, 1000 x 0.01, -1e17", 1001, 10.0, eager(&m), naive, comp.value()); + report( + "1e17, 1000 x 0.01, -1e17", + 1001, + 10.0, + eager(&m), + naive, + comp.value(), + ); } fn main() { From 9be37faf79471208c06df90b1e3e4d1c0d8d50dc Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:17:24 -0700 Subject: [PATCH 003/106] docs(adr): name ADR 009's crate options without an id-shaped token (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 802addc3..edcb5ac3 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -558,7 +558,7 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. -1. **The crate's name.** `zset`, `dbsp-core`, or `improvise-dbsp`? +1. **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. From c456be2d5400ea25c8fc1b2a273b68ac54099839 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 20:45:59 -0700 Subject: [PATCH 004/106] docs(adr): ADR 009 follows the document per model effect; answer the joint review (improvise-felo.4) - clause 7: the operator shape and the formula-set fingerprint are re-read after each model effect inside the loop. Formula edits are an explicit input, and a document swap rebuilds everything. Stated in the same words as ADR 008 revision 3. - clause 3: a delta key's identity is bit identity, matching hash_value. The dirty flag keeps a formula fingerprint, so formula edits still persist. - clause 5: the stem rule for a formula with a top-level WHERE. - clause 11 and question 9: rollback is a snapshot, with its second O(cells) clone stated. The inverse delta is not a rewind. - migration: formulas become step 5, and Sum/Avg and column totals step 6. - the drift example's values are five-place, not two-place. Clause 7 and question 7 are worded as recommendations. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 221 ++++++++++++++++------ 1 file changed, 166 insertions(+), 55 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index edcb5ac3..ffee8dd0 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -15,7 +15,7 @@ no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means -revision 2 of `context/adr/008-virtualized-grid.md` on PR #308. +revision 3 of `context/adr/008-virtualized-grid.md` on PR #308. ## The decision this ADR writes down @@ -64,10 +64,15 @@ plugs in, and what it may not change. (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every combination of the spanned categories' items. ADR 008 clause 5 names this as the one O(grid) path left in totals. -- **Two of the record tiers are observed by scanning the store.** `record_fingerprints` - (`app.rs:254-264`) hashes every cell before and after a cell-writing batch, and - `document_content` (`app.rs:165-178`) does the same for the dirty flag. Both are - O(cells) per batch, whatever the batch wrote. +- **Two observations scan the store.** `record_fingerprints` (`app.rs:254-264`) hashes + every cell before and after a cell-writing batch. `document_content` + (`app.rs:165-180`) hashes every cell and every formula for the dirty flag, and it is + re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) + per batch and the second O(cells) per effect, whatever was written. +- **Every mutating frame from a client already clones the document.** + `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:162`, + `app.rs:627-629`), for ADR 001 clause 4. That is O(cells) per frame on the network + path, whatever this ADR does. ### What an incremental evaluator must not change @@ -158,8 +163,8 @@ name is `Derived`. It replaces the two places where eager evaluation folds the s - the raw fold of the cells matching a partial key under an aggregation, which is `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). -It has one mutation, `apply(&mut self, delta: &CellDelta)`, plus a rebuild for the -changes that are not cell deltas (clause 7). +It takes two inputs, both applied after each model effect (clause 7): the cell delta, +and the formula delta. It also has a rebuild, for an operator shape that moved. **The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code and read through the trait. The circuit decides *which* formula cells to re-evaluate @@ -198,6 +203,25 @@ An overwrite with an equal value consolidates to the empty Z-set. The circuit st per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). +**A delta key's identity is bit identity, the same identity the record tier uses +today.** Two `(CellKey, CellValue)` pairs are one element exactly when their keys are +equal and their values are equal *by bits*: `Number` by `f64::to_bits`, `Text` and +`Error` by their strings. That is `hash_value` (`app.rs:233-250`), which +`record_fingerprints` already hashes. So the cell delta reproduces today's tier +exactly: + +- overwriting `0.0` with `-0.0` is a non-empty delta and bumps the record, as it does + today; +- overwriting a NaN with a NaN of the same bits is empty and bumps nothing, as today. + +It is **not** `CellValue`'s derived `PartialEq`, under which `-0.0 == 0.0` and +`NaN != NaN`. `recompute_formulas` uses that one for its "did this cell move" test +(`types.rs:1010`), and it stays there. The ordering behind a multiset (clause 4) is +`f64::total_cmp`, whose equality *is* bit equality for every finite and non-finite +value. So the delta identity and the multiset ordering agree, and neither is +`PartialEq`. Step 3's tests pin both cases: signed zero, and a repeated same-bits +NaN. + **This lines up with ADR 001's record tier with no extra rule.** The records a batch moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys in the batch's consolidated delta. So: @@ -208,9 +232,19 @@ in the batch's consolidated delta. So: `bump_changed_records` can then read the delta instead of diffing two `record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta -comes from the mutation sites, not from an `EffectKind`. The same holds for the content -half of the dirty flag, where a non-empty delta means dirty. `layout_version` is -unchanged, because it observes `view_shape` and not cells. +comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, +because it observes `view_shape` and not cells. + +**The dirty flag's content half does not move onto the cell delta alone.** +`document_content` (`app.rs:165-180`) hashes the formulas as well as the cells, and it +is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, +`RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, +`:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a +non-empty cell delta means dirty" would never save a formula edit in the browser. The +content half becomes: **a non-empty cell delta, or a moved formula-set fingerprint**. +The fingerprint is the one clause 7 already reads after each model effect, so nothing +new is observed. Its cost is O(formulas), where `document_content` is O(cells) per +`CellData` effect. **`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` (`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only @@ -252,11 +286,13 @@ key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the me whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A `Sum`-only document keeps one number per group. -**`total_cmp` is not `==`, and the differential test must say so.** `NaN != NaN`, so -today two NaNs in one group read as `mixed`, and `f64::min` skips a NaN. A multiset keyed -by `total_cmp` would call two NaNs one value and would order NaN at an end. The test -generates non-finite values (core.md §16), and the aggregator matches eager -behaviour explicitly rather than inheriting `total_cmp`'s. +**The multiset's identity is clause 3's bit identity, and eager's *reading* is not.** A +multiset keyed by `total_cmp` holds the values the store holds, by bits, and that is +correct for the state. But eager *reads* a group through `PartialEq` and `f64::min`: +`NaN != NaN`, so two NaNs in one group read as `mixed` today, `-0.0` and `0.0` read as +one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eager's +rules explicitly and does not inherit `total_cmp`'s. The differential test generates +signed zeros and non-finite values (core.md §16) to pin it. ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it @@ -265,6 +301,21 @@ with the target category and each `none_cats` entry removed. That is `distinct(map(project))` over the cell delta. A stem enters when its first cell arrives and leaves when its last cell goes. +**The stem rule: a formula cell exists at a stem only where its value would be cached +today.** `recompute_formulas` caches a formula cell only when the cached evaluator +returns a value (`types.rs:1010-1014`). For a formula with a top-level `WHERE`, a stem +whose coordinate fails the filter falls back to `aggregate_raw` (`types.rs:1047-1055`), +which is absent when no raw data lies at the target key. So: + +- **a formula counts at a stem only where its top-level `WHERE` (if any) matches the + stem's coordinate, or where raw data exists at stem + target.** + +This is a coordinate predicate plus one count lookup, never an evaluation. It is the +rule ADR 008 clause 3's occupancy uses for formula cells, and it keeps today's pruning. +The review of ADR 008 found the case that needs it: `Double = Revenue * 2 WHERE +Region = East` with data only at West is pruned today, and a rule that counted every +stem would have kept the column. + **A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula `T` at stem `s` are: @@ -326,26 +377,51 @@ So the crate has no recursion operator, and circularity is decided *outside* it: like with like. Changing the rule would be its own decision, for both backends at once. -### 7. What is not a cell delta rebuilds, once +### 7. The circuit follows the document per model effect -Some changes alter the circuit's *shape*, not its input: +Some changes alter what the circuit computes, not only its input. They are observed +**inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same +position as the frame and dirty observations (`app.rs:1036-1084`). They are not the +post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). +That diff comes too late, and it cannot see most of these changes. ADR 008 states this +clause in the same words. -- the formula set: add, remove or replace; -- `measure_agg`; -- category and item structure: add or remove a category, remove an item, and group - membership; -- a view's axes, hidden items and collapsed groups, which change `view_shape`; -- a document swap, `document_generation`. +- **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), + plus the group membership of every drawn item, collapsed or not, plus + `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is + unchanged. A view whose operator shape moved has its operators rebuilt from the + integrated base Z-set **before the next effect's cell delta is applied**. +- **The formula set is an input, not a rebuild.** After each model effect the host + compares a fingerprint of the formula set, over (`raw`, `target`, + `target_category`), with its previous read. If it moved, the formula delta (−the old + formulas, +the new ones) is fed to the circuit as an explicit input. The formula + layer then re-derives the formula graph (clause 6) and re-evaluates the affected + targets at their stems. This is observed, not declared by effect type, as the tiers + are. It is what catches a `ReplaceFormula` that keeps its target, which moves no + `view_shape`. +- **A document swap rebuilds every circuit** from the new store. `install_workbook` + bumps `document_generation`, and the same after-each-effect read sees it. -Each rebuilds only the affected operators, from the integrated base Z-set. That costs -O(cells), once per change, and is never repeated per keypress. **The trigger is the same -observation the tiers use**: a `view_shape` diff (`app.rs:40-72`) rebuilds that view's -operators, the same diff that bumps `layout_version`. So there is one signal and not two. +Re-reading the operator shape costs O(items) per view per model effect, and the formula +fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs +O(cells), once per actual reshape. The same `view_shape` function is therefore read at +two moments: by the circuit after each effect, and by `layout_version` after the loop. +They answer different questions, and neither replaces the other. -**A page selection is not a rebuild.** The group key includes the page categories, so -`[` and `]` read another slice of state that already exists. The held-key path stays -O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the -state is still bounded by the stored cells, however many pages there are (question 7). +**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** `AddItem` +moves the operator shape, so the view is rebuilt with East in its sequences. Only then +does `SetCell`'s delta arrive, at operators that already know East. So the in-loop +`CellData` frame observation reads occupancy for the new shape. The `LayoutGeometry` +arm (`app.rs:1062-1071`) reads the frame immediately after each geometry effect, and it +reads a rebuilt occupancy for the same reason. + +**Recommended: a page selection is not a rebuild (question 7).** Under the recommended +answer, the group key includes the page categories, so `[` and `]` read another slice +of state that already exists. The held-key path stays O(visible) (improvise-748f's +constraint). The slices cost nothing extra, because the state is still bounded by the +stored cells, however many pages there are. Filter coordinates still filter; only page +coordinates key the slice. If the user answers the other way, a page change is an +operator-shape change like any other. ### 8. Floating point: retraction drifts, and exactness is ADR 005's @@ -356,8 +432,10 @@ the eager fold computes. **Measured**, not asserted: an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N retract-and-insert cycles. It compares the sum against the eager fold, `evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count -of 10⁻⁵ units rounded to f64 once. Values are signed two-place decimals spread over 3 or -9 decades. +of 10⁻⁵ units rounded to f64 once. Each value is a signed whole number of 10⁻⁵ units: +up to 99,999 units, times a power of ten spread over 3 or 9 decades. So the smallest +values carry five decimal places, not the two a user usually types. Two-place input +would absorb differently, and it was **not** measured. | store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | |---|---|---|---|---| @@ -403,12 +481,15 @@ The measurement above is the evidence for and against that fallback. ### 9. What ADR 008 gets from it -ADR 008 (revision 2, PR #308) assumes these, under the shared names below. +ADR 008 (revision 3, PR #308) assumes these, under the shared names below. - **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and per column tuple of cells that hold a value. Stored cells pass the view's filter, with hidden items dropped, and a collapsed group's members map to the group's tuple. - Formula cells count at their stems, which is today's `formula_cache` stem set. A tuple + Formula cells count by clause 5's stem rule, which is when today's `formula_cache` + holds a value: a top-level `WHERE` must match, or raw data must exist at the target. + The operators are rebuilt per clause 7, so a batch that reshapes and then writes feeds + its write to the new shape. Page coordinates key a slice, subject to question 7. A tuple is on the axis iff its count is above zero. **This ADR does not settle ADR 008 question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those cells are values with no stored cell under them. The stem definition drops them, and @@ -463,9 +544,20 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:647`). The backend's state is part of the snapshot. Undoing by applying the - inverse delta is exact for integer weights and order-free aggregators, but not for an - f64 `Sum`, so the snapshot is kept until clause 8's prerequisite lands (question 9). + (`app.rs:647`). **Recommended (question 9): the backend's state joins the snapshot.** + Applying the inverse delta is **not** a rollback, even with exact aggregators. It + restores integrated values, but `step(d)` then `step(−d)` still advances the circuit + and changes the history that `delay` and `differentiate` hold. So the next step after + a "rewind" can differ from the next step after nothing happened. A real rewind would + have to restore every stateful operator, with a test of observational equivalence + across all of them. This ADR does not propose one. + + **The snapshot costs a second O(cells) clone per mutating frame.** + `Host::mutating_frame` already clones the workbook on every mutating frame + (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So + on the network path a write is not O(delta): it is O(cells) for the snapshot today, + and roughly twice that with the circuit in it. The keyboard path (`apply_effects` + without a `Host`) takes no snapshot and keeps the O(delta) cost. - **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the delta sequence, so replaying commands reproduces its state. @@ -517,16 +609,23 @@ displayed path until the differential test holds. test. Nothing in improvise uses it yet. 2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the existing suite is the evidence. `CellKey` gains `Ord`. -3. **The cell delta from `DataStore`.** `bump_changed_records` and the content half of - the dirty flag move onto it. That removes two O(cells) scans per cell-writing batch. - The existing tier tests are the evidence. -4. **`Incremental`, for the order-free parts**: stems, `Count`, `Min`, `Max`, - unanimity, occupancy and the occupancy delta, under the differential test. **ADR 008 - clauses 3 and 8 (pruning and the frame observation) depend on this step.** -5. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's fallback). - **ADR 008 clause 5 depends on this step.** -6. **Formulas**: the formula graph, topological re-evaluation, the component fixed - point, and the aggregate-over-expression map that replaces `Expansion::members`. +3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero + and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half + becomes "a non-empty cell delta, or a moved formula-set fingerprint" (clause 3), with + a test that a formula edit alone still marks the document dirty. That removes the + O(cells) scans per cell write. The existing tier tests are the evidence. +4. **`Incremental`, for the order-free parts**, under the differential test: stems with + clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy + delta, all following clause 7's per-effect operator shape. **ADR 008 clauses 3 and 8 + (pruning and the frame observation) depend on this step.** +5. **Formulas**: the formula graph and the formula delta (clause 7), topological + re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression + map that replaces `Expansion::members` (clause 5). Formula values that depend on a + `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of + `Expansion::members` depends on this step.** +6. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's + fallback). A column total is a formula over the pooled fold, so it needs step 5's + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is @@ -535,7 +634,9 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause **Good.** -- A write costs O(delta) plus the formula cells it reaches, not O(cells). +- On the keyboard path, a write's *evaluation* costs O(delta) plus the formula cells it + reaches, not O(cells). On the network path, the ADR 001 snapshot still costs O(cells) + per mutating frame (see "Bad"). - The store scans behind the record tier and the dirty flag disappear. - Per-view model clones for a different None axis disappear. - ADR 008 gets occupancy, the occupancy delta and column totals without a second @@ -551,6 +652,11 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause the measurement says so. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. +- A mutating frame from a client already clones the workbook for rollback + (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) + clone per frame. Incremental evaluation does not make the network path O(delta). +- The operator shape is re-read after every model effect: O(items) per view, a cost the + post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. ## Open questions @@ -594,7 +700,8 @@ the user decided", as ADRs 004 and 007 did. switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the - held-key path (`[`/`]`) stays O(visible). ADR 008 revision 2 assumes this. + held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this + recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". 8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? @@ -602,13 +709,17 @@ the user decided", as ADRs 004 and 007 did. source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. **Rollback.** For ADR 001 clause 4, should the backend's state be snapshotted with - the `App`, or undone by applying the inverse delta? - *Recommendation:* snapshot until ADR 005 lands, then the inverse delta. The inverse - is exact only when every aggregator is. +9. **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` + snapshot that `Host::mutating_frame` already takes? + *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying + `−d` restores integrated values but not the history `delay` and `differentiate` + hold, so it is not a rewind even with exact aggregators (clause 11). The cost is a + second O(cells) clone per mutating frame, beside the workbook clone that exists + today. A true rewind would need its own design and an observational-equivalence test + over every stateful operator. 10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by - steps 4 and 5? + steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. From 0ab3ac7dbf5186f848400459dfd3bea9674513a1 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 21:27:56 -0700 Subject: [PATCH 005/106] docs(adr): ADR 009 rebuilds before the frame observation, and before step 5 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ffee8dd0..5c26445a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -383,14 +383,16 @@ Some changes alter what the circuit computes, not only its input. They are obser **inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same position as the frame and dirty observations (`app.rs:1036-1084`). They are not the post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). -That diff comes too late, and it cannot see most of these changes. ADR 008 states this -clause in the same words. +That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 +states the same rule. - **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), plus the group membership of every drawn item, collapsed or not, plus `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is unchanged. A view whose operator shape moved has its operators rebuilt from the - integrated base Z-set **before the next effect's cell delta is applied**. + integrated base Z-set **before the frame observation that follows this effect**. + The `LayoutGeometry` frame arm (`app.rs:1062-1071`) reads occupancy right after the + effect that moved the shape, so the next effect's cell delta is too late. - **The formula set is an input, not a rebuild.** After each model effect the host compares a fingerprint of the formula set, over (`raw`, `target`, `target_category`), with its previous read. If it moved, the formula delta (−the old @@ -616,8 +618,10 @@ displayed path until the differential test holds. O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's per-effect operator shape. **ADR 008 clauses 3 and 8 - (pruning and the frame observation) depend on this step.** + delta, all following clause 7's per-effect operator shape. The formula delta arrives + only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in + step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the + frame observation) depend on this step.** 5. **Formulas**: the formula graph and the formula delta (clause 7), topological re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression map that replaces `Expansion::members` (clause 5). Formula values that depend on a From a2836f44f57fec90caf299bd9650d10d2fd6e81e Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 21:53:29 -0700 Subject: [PATCH 006/106] docs(adr): ADR 009 leads with incremental formulas; exact f64 sums; ordered aggregates (improvise-felo.4) - Reframed: incremental formula evaluation for streamed data (ADR 003) is the primary benefit, and ADR 008's grid is the second consumer. - New clause 5a: aggregate over an expression, as a lifted map feeding the group aggregate, still evaluated by eval_expr. - Clause 4: ordered state for LAST/FIRST (felo.7) and semi-additive measures (felo.8). New question 11. - Clause 8: DBSP alone gives no floating-point benefit. An exact superaccumulator makes f64 SUM a true group. It is measured in retraction_drift: bit-equal after 1M cycles, about 3.2 ns per update, 568 bytes per group. It is correct against math.fsum on 20,000 cases, and xsum 0.1.6 is rejected. The principle of deferring lossy operations, and a table of what each exact domain covers. Question 3 gains option (c) and recommends it. No clause or question number moved. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 330 +++++++++++++++--- .../examples/retraction_drift.rs | 284 +++++++++++++-- 2 files changed, 544 insertions(+), 70 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 5c26445a..5e9d4aa8 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -42,8 +42,38 @@ So the choice between Feldera's crate and our own is already made (see Alternati what was checked). This ADR says what "the core of dbsp" is for improvise, where it plugs in, and what it may not change. +## What it is for + +**The primary benefit is incremental formula evaluation, so that data can stream.** +That is the user's own framing, both in the prompt that led to this ADR and in their +later question: "Isn't the benefit of adopting DBSP primarily incremental formula +updates?" Yes. When a cell changes, whether from a keystroke, a client's frame or a +live source (ADR 003, clause 10), only the aggregates that cell feeds are adjusted, +and only the formula cells whose inputs moved are re-evaluated. Nothing is recomputed +from scratch. The machinery is: + +- the aggregates formulas read are **maintained**, not re-folded (clause 4); +- an aggregate over a per-record expression such as `SUM(Revenue * Price)` is a + maintained operator too (clause 5a); +- the circuit decides **which** formula cells moved, and the one existing evaluator + recomputes only those, in dependency order (clauses 5 and 6). + +**The grid is the second consumer.** ADR 008 needs the same maintained state for other +purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, +and column totals over rows never scrolled to (clause 9). It gets them from the same +circuit, not from a second mechanism. + ## Context +### The primary case: data that streams + +A live-backed model (ADR 003) is a source whose rows change underneath the document. +Today the only way to reflect a change is the path every edit takes: recompute every +formula cell of the document (next subsection). That is tolerable when a human types +one cell at a time. It is not tolerable when a source delivers a stream of changes, +each touching a few records of a large table. Incremental evaluation makes the cost of +a change proportional to what the change touches, not to the document. + ### Evaluation today is a from-scratch fold, rerun after every batch - **Every batch recomputes every formula cell.** `App::apply_effects` @@ -98,10 +128,14 @@ incremental evaluator could quietly disagree with the eager one. ## Decision -**A new, problem-agnostic workspace crate implements the DBSP core. improvise-core -defines a trait for the derived state it reads. Today's evaluator is one implementation -of that trait, and a circuit built on the new crate is the other. Both stay in the tree, -and a differential test holds them equal.** Eleven clauses. +**Formulas are evaluated incrementally. A change re-evaluates only the formula cells +whose inputs moved, over aggregates that are maintained rather than re-folded. The +machinery is a new, problem-agnostic workspace crate implementing the DBSP core. +improvise-core defines a trait for the derived state its evaluator reads. Today's +evaluator is one implementation of that trait, and a circuit built on the new crate is +the other. Both stay in the tree, and a differential test holds them equal.** Eleven +clauses, plus clause 5a. It is numbered 5a so that no clause number or question number +cited elsewhere moves. ### 1. The crate: `zset` (name is question 1) @@ -265,7 +299,7 @@ comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does | answer | state per group | cost per delta element | exact on f64? | |---|---|---|---| | `Count` | count of values, count of numbers | O(1) | yes (integers) | -| `Sum` | running sum | O(1) | **no** (clause 8) | +| `Sum` | running sum (an exact accumulator under question 3's recommendation) | O(1) | **no** as an f64; **yes** as the accumulator (clause 8) | | `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | | `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | | unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | @@ -294,6 +328,43 @@ one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eag rules explicitly and does not inherit `total_cmp`'s. The differential test generates signed zeros and non-finite values (core.md §16) to pin it. +#### Ordered aggregates: `LAST` and `FIRST` need state ordered by an item, not a value + +improvise-felo.7 asks for `LAST` and `FIRST` over a fold: the closing or opening value +of a stock price, say, across the months of a year. improvise-felo.8 asks for +semi-additive measures: `sum` across one category and `last` across another. Both are +open tickets with their own semantics to decide. What this ADR must say is whether its +per-group state can carry them. Today it cannot: the integrated state above is an +**unordered** multiset, keyed by the value itself. + +**The structure.** An ordered aggregate keeps, per group, a `BTreeMap` keyed by the +**ordering category's item order**, with the tie-break felo.7 decides. Each entry holds +the values at that position with their weights. `LAST` reads the entry at the greatest +key, and `FIRST` the least. + +- **It is the same shape as `Min`/`Max`.** Those also keep an integrated ordered map and + read an extreme; they order by the value, where `LAST`/`FIRST` order by the item's + position and read the value stored there. So one generic operator covers all four: + an integrated map keyed by `K`, read at an extreme. `Min`/`Max` instantiate it with + `K = value`, and `LAST`/`FIRST` with `K = (item position, tie-break)`. +- **Cost:** O(log d) per delta element, where d is the number of distinct positions in + the group, and one entry per contributing record. That is the same order as + `Min`/`Max`'s multisets. +- **What moves the order.** Adding an item in the middle of the ordering category, or + reordering its items, changes every stored position. So the ordering category's + item order must be part of the operator shape (clause 7), and a change to it rebuilds + the ordered aggregators that use it. The alternative is to key by a stable item + identity (ADR 007) and look the order up at read time. That makes a reorder free, but + a read then costs O(d). This is question 11. +- **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, + `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a + delta (retract the old value, insert the new), and the outer consumes it. So the + composition is incremental at O(log d) per stage. Which stage is inner is felo.8's + semantic decision, because sum-of-last and last-of-sum differ. The circuit takes + either order. +- **Exactness.** `LAST`/`FIRST` return a stored value unchanged, so they are exact on + f64 like `Min`/`Max`. A semi-additive `sum` stage still needs question 3's answer. + ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it **Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key @@ -340,14 +411,61 @@ With the trait, both reads go to the maintained state. not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own group and nothing else. -**An aggregate over an expression replaces `Expansion::members`.** `SUM(expr)` pools -`expr` evaluated at each member record (`agg_values`, `types.rs:89-139`). Incrementally, -that is `map(record ↦ expr at record)`, then `index_by(base key)`, then `Sum`. The -member records never have to be listed. **The boundary:** the map is sound only when -`expr` reads the record's own cells. An inner expression that reads another record, or -holds a nested aggregate, stays on the eager path for that formula, and the -differential test pins the boundary. Until `Sum` is exact, this path carries clause 8's -caveat. ADR 008 clause 5 adopts the same boundary. +**An aggregate over an expression** is the one place a formula needs an expression +*inside* an operator. It has its own clause, 5a. + +### 5a. Aggregate over an expression: a lifted map feeding the group aggregate + +`SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per +record**, `Revenue * Price` at that record, and pools those values across the records +of a group. Today `agg_values` (`types.rs:89-139`) lists the member records of each +context and calls the record-level evaluator on each (`member_records`, then +`eval_at`). At a fold or a column total, the contexts come from `Expansion::members` +(`types.rs:201-213`), which materializes the cross product of the spanned categories' +items. That is the one O(grid) path in totals (ADR 008 clause 5). + +**The design.** It is a three-stage pipeline over the record stream, and each stage is +an operator from clause 1: + +1. **A lifted map over the per-record expression.** For each record in the cell + delta's `record_of` set, the map produces `(base key, expr at record)`. A record + whose cells changed is retracted at its old value and inserted at its new one, so + the map's output is itself a delta. `map` is linear, so this is its own incremental + form. +2. **`index_by(base key)`.** The base key is the record's key with the target category + removed, which is what `eval_agg` pools over (`types.rs:55-87`). An inner `WHERE` + (`SUM(expr WHERE Cat = Item)`) is a `filter` before the index. It is the same + rewrite `eval_agg` applies to its contexts (`types.rs:66-72`). +3. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the + mapped values, with `Avg` derived. + +Member records never have to be listed, and no cross product is ever built. A context +in today's cross product that holds no records contributed nothing to the pool, so +pooling only the records that exist is the same answer. + +**This is not compiling formulas into operators.** The operator is the map; the +expression inside it is still evaluated by `eval_expr` at the record, exactly as +`agg_values` calls it today. The circuit decides *which* records to re-evaluate (those +the delta names), and the one evaluator computes each value. Clause 2's rule stands: +there is one evaluator. + +**The boundary.** The map is sound only when `expr` reads the record's own cells, so +that its value at a record changes only when that record changes. An inner expression +that reads another record, holds a nested aggregate, or refers to a formula whose value +depends on other records, is not a function of one record. That formula stays on the +eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the +same boundary. + +**What stays eager until step 5, and until step 6.** +- Before step 5 (formulas), every aggregate over an expression is evaluated as today, + through `Expansion::members`. +- From step 5, the map and the index are maintained, so the cross product is gone. + `Count`, `Min` and `Max` over the mapped values are read from the maintained state, + since they are order-free and bit-exact on f64 (clause 8). +- `Sum` and `Avg` over the mapped values are **re-folded** from the group's maintained + values, in the store's record order, until step 6. That keeps them bit-equal to + eager, at a cost of O(group) per change rather than O(grid). +- From step 6 (on question 3's answer), `Sum` is maintained like any other. ### 6. Circular references: DBSP's recursion is not our semantics @@ -425,7 +543,7 @@ stored cells, however many pages there are. Filter coordinates still filter; onl coordinates key the slice. If the user answers the other way, a page change is an operator-shape change like any other. -### 8. Floating point: retraction drifts, and exactness is ADR 005's +### 8. Floating point: retraction drifts, and exactness comes from the representation f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum the eager fold computes. **Measured**, not asserted: @@ -453,12 +571,12 @@ A Neumaier-compensated running sum recovers that case exactly. In the random cas stays about as close to the exact sum as eager does, **but it is never bit-equal to eager**, because eager folds in store order and is itself inexact. -Four conclusions follow: +Four conclusions follow about a **floating-point** running sum: - **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by - any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, - unanimity and occupancy bit for bit today, because they are order-free. It can compare - `Sum` and `Avg` only with a tolerance. + any f64 running-sum technique, naive or compensated. So the differential test can + compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit, because they are + order-free, but it can compare an f64 `Sum` and `Avg` only with a tolerance. - **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour @@ -466,20 +584,115 @@ Four conclusions follow: - **This is already a bug in eager, which the measurement found in passing.** With 3 decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed - can already be wrong today. That supports ADR 005 on its own - merits, and it is recorded here for improvise-l2b8. -- **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true - `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's - whole class), and the differential test is bit-exact everywhere. + can already be wrong today. That is recorded here for improvise-l2b8. +- **Only an exact representation dissolves the problem.** The next two subsections give + two: an exact accumulator for f64 sums, and ADR 005's decimals. -**So ADR 005 is a prerequisite for the maintained `Sum`/`Avg` on the displayed path, but -not for the rest of this ADR** (question 3). Occupancy, `Count`, `Min`, `Max`, -unanimity, stems and invalidation can all land before it. Until ADR 005 lands, a `Sum` -or `Avg` the grid displays, including a column total that holds one, is read from the -eager fold. The maintained sum runs only inside the differential test, under a stated -tolerance. If the user wants maintained sums before ADR 005, the fallback is a -compensated sum that re-folds a group from its values when cancellation is detected. -The measurement above is the evidence for and against that fallback. +#### Does DBSP itself help with floating point? No. + +The user asked whether rebuilding the evaluator on DBSP brings any correctness benefit +for floating point. On its own, **none**: + +- **Only the Z-set weights are exact.** They are integers (clause 1). The *values* being + summed are whatever type the aggregator holds. +- **An f64 `Sum` maintained by retraction is worse than eager, not better.** The table + above is the evidence: it drifts further from the exact sum the longer it runs, and + the absorption case loses the answer entirely. + +**The benefits come from two other places.** +- **The representation.** An exact accumulator (next subsection) or exact decimals + (ADR 005) make `Sum` a true abelian group. Then retraction is exact, incremental + equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent + `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes + with it. +- **The differential test** between `Eager` and `Incremental` (clause 2) is itself a + correctness gain. Two independent evaluations of every formula cell catch evaluator + bugs in either one, which nothing in the tree does today. + +What DBSP does contribute is **placement**. Every aggregate is a named operator with a +chosen algebra, so the circuit states exactly where values enter an exact domain and +where they leave it. That is the next subsection's principle. + +#### An exact accumulator for f64 sums, and deferring the lossy operations + +**A superaccumulator sums f64 values exactly.** It is a fixed-point integer wide enough +for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit integer shifted +by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's +xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. +Adding or retracting a value is integer addition, so the accumulator is a true abelian +group over the finite f64s. **Rounding happens once, when the value is read.** Then: + +- `Sum` is the exact sum, rounded once. It is independent of order. +- `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded + if the division is done on the exact value; dividing the rounded sum rounds twice, + and step 6 must say which it does. +- `Min` and `Max` are already exact. +- Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. + +**Measured**, in the same example, by `ExactSum` (147 lines, including its rounding): + +- **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` + maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over + the current values. That includes the absorption case, which reads exactly 10.00. + Against the exact *decimal* sum, it differs at 1 checkpoint of 25, by 9.1e-13. That + gap is the rounding of the typed decimals into f64, not the sum. +- **It is correct.** It was checked (not committed) against Python's `math.fsum` on + 20,000 random cases. The cases include subnormals, values near the overflow bound, + and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the + 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. +- **Cost, Apple M4 Max, best of 3 over 10M values:** + - about 3.2 ns per insert or retract, against about 2.3 ns for a plain f64 add in the + same loop; + - about 1.1 µs per read, which is the rounding, paid once per group read; + - **568 bytes per group**, against 8 for an f64. + + The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB + of accumulators. A sparse form that stores only the occupied span of limbs would be a + few limbs for typical data. **That is an estimate, not measured.** + +**No existing crate was usable.** `xsum` 0.1.6 (MIT, no dependencies, and it builds for +`wasm32-unknown-unknown`) was checked and rejected on three counts: + +- **It is wrong on a single negative value.** It rounds about a quarter of single + negative inputs to a neighbouring f64: 49,819 of 200,000 sampled values, all + negative, for example `-5890.88` read back as `-5890.880000000001`. +- It keeps one infinity and one NaN flag, not counts, so a non-finite value cannot be + retracted. +- Its accumulator is not `Clone`, and reading it takes `&mut self`. + +`accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not +evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and +a differential test against an exact rational sum. + +**Defer the lossy operations: the general principle.** Keep intermediate results in a +domain where the operation is exact, and apply the lossy operations at the edge: + +- sums go in the accumulator; +- decimal add, subtract and multiply are exact under ADR 005; +- division, `POW` with a non-integer exponent, and rounding for display happen only + where a value leaves the exact domain. + +This is honest only about where the gain comes from. **The representation earns it, not +DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named +operator with a chosen algebra, so "where does rounding happen" has one answer per +operator rather than one per call site. + +#### What each exact domain covers + +| | exact accumulator | ADR 005 decimals | +|---|---|---| +| `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | +| per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | +| typed values such as money | no | yes | +| a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | +| cost | 568 bytes per group, about 3.2 ns per update | bignum weight in the wasm bundle (improvise-l2b8's question 3) | + +**So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. +The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this +ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting +for ADR 005**, provided `Eager` uses the same accumulator. That is question 3's new +option, and its recommendation. ### 9. What ADR 008 gets from it @@ -507,7 +720,8 @@ ADR 008 (revision 3, PR #308) assumes these, under the shared names below. not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate, and the formula layer re-evaluates only the columns whose inputs moved, through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that - hold a `Sum` or `Avg` wait for ADR 005 (clause 8). + hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the + exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, or maintain) is struck there as decided by the user through this ADR. Its questions 1 and 2 stay open there. @@ -627,14 +841,16 @@ displayed path until the differential test holds. map that replaces `Expansion::members` (clause 5). Formula values that depend on a `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's - fallback). A column total is a formula over the pooled fold, so it needs step 5's +6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact + accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is question 10. improvise-swre stays independent and is useful under either backend. -improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause 8. +improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is +already inexact beyond `FLOAT_EQ_EPSILON`. Under question 3's recommendation it is no +longer a prerequisite of step 6. **Good.** @@ -652,8 +868,12 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause - Two backends, kept equal by a differential test that must itself be kept honest. - Integrated state is O(cells) in memory, per distinct None axis, in addition to the store. -- `Sum` and `Avg` cannot be maintained on the displayed path until ADR 005 lands, and - the measurement says so. +- An f64 `Sum` cannot be maintained on the displayed path at all. Maintaining it needs + an exact representation: the accumulator costs 568 bytes per group (clause 8), or + ADR 005's decimals. +- Under question 3's recommendation, `Eager`'s displayed sums change in the last bits, + because they become correctly rounded. That is a behaviour change, even though it is + a fix. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. - A mutating frame from a client already clones the workbook for rollback @@ -677,13 +897,22 @@ the user decided", as ADRs 004 and 007 did. *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. **Floating point.** Should ADR 005 (exact decimals) be a prerequisite for maintaining - `Sum` and `Avg` on the displayed path, with the eager fold kept for them until then? - Or should a compensated f64 sum, re-folded on detected cancellation, ship first with - a tolerance in the differential test? - *Recommendation:* ADR 005 as the prerequisite. The measured drift reaches - `FLOAT_EQ_EPSILON`, so `=` could answer differently on the two paths. No f64 technique - makes the differential test bit-exact. Everything else in this ADR lands without it. +3. **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? + - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums + until then. + - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in + the differential test. + - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by + **both** backends, so that incremental equals eager bit for bit. + + *Recommendation:* (c), in `zset`. It is measured bit-equal after a million + retract-and-insert cycles, and exact against `math.fsum` on 20,000 cases. It costs + about 3.2 ns per update and 568 bytes per group, and it does not wait for ADR 005. + (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for + per-cell arithmetic and typed values, and it composes with (c): under ADR 005 a + decimal sum is exact on its own. The cost to accept is that today's displayed sums + change in their last bits, because they become correctly rounded, and that + `Avg`'s one rounding must be stated in step 6. 4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? @@ -727,3 +956,16 @@ the user decided", as ADRs 004 and 007 did. steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. + +11. **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). + Should `LAST`/`FIRST` key their per-group state by the ordering category's item + position, with that category's item order added to the operator shape so that a + reorder rebuilds them? Or should they key by stable item identity (ADR 007), and + look the order up at read time? + *Recommendation:* item position, with the order in the operator shape. Reads stay + O(log d), like `Min`/`Max`, and the same generic operator serves all four. An item + reorder is a rare structural edit, and clause 7 already rebuilds on those. This + extends the operator shape's `measure_agg` part to cover "and the item order of + each ordering category". ADR 008 clause 3 lists the parts as `view_shape`, group + membership and `measure_agg`. This reading changes what the `measure_agg` part + covers, not the list, so ADR 008's text and cross-references still hold. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 92dae6e5..26487b9e 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -1,5 +1,8 @@ //! How far a SUM maintained by insert/retract deltas drifts from the eager fold, -//! in f64. ADR 009 (improvise-felo.4), "Floating point". +//! in f64, and what an exact accumulator costs. ADR 009 (improvise-felo.4), clause 8. + +use std::hint::black_box; +use std::time::Instant; use improvise_core::format::format_f64; use improvise_core::model::cell::{CellKey, CellValue}; @@ -90,31 +93,196 @@ impl Compensated { } } +const LIMB_BITS: u32 = 32; +/// Every finite f64 is `mant * 2^(shift - 1074)` with `shift <= 2045` and `mant < 2^53`. +const LIMBS: usize = 67; + +/// An exact sum of f64s: a signed fixed-point integer in units of 2^-1074, wide enough +/// for every finite f64, with non-finite values counted so they can be retracted too. +#[derive(Clone)] +struct ExactSum { + limbs: [i64; LIMBS], + pending: u32, + nan: i64, + pos_inf: i64, + neg_inf: i64, +} + +impl ExactSum { + fn new() -> Self { + ExactSum { + limbs: [0; LIMBS], + pending: 0, + nan: 0, + pos_inf: 0, + neg_inf: 0, + } + } + + /// Add `x` with `weight` +1 (insert) or -1 (retract). Exact: no rounding happens here. + fn add(&mut self, x: f64, weight: i64) { + match (x.is_nan(), x.is_infinite()) { + (true, _) => self.nan += weight, + (false, true) if x > 0.0 => self.pos_inf += weight, + (false, true) => self.neg_inf += weight, + (false, false) => self.add_finite(x, weight), + } + } + + fn add_finite(&mut self, x: f64, weight: i64) { + let bits = x.to_bits(); + let biased = ((bits >> 52) & 0x7ff) as u32; + let fraction = bits & ((1 << 52) - 1); + let (mant, shift) = match biased { + 0 => (fraction, 0), + e => (fraction | (1 << 52), e - 1), + }; + let sign = if x.is_sign_negative() { + -weight + } else { + weight + }; + let wide = u128::from(mant) << (shift % LIMB_BITS); + let k = (shift / LIMB_BITS) as usize; + for i in 0..3 { + let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64; + self.limbs[k + i] += sign * part; + } + self.pending += 1; + if self.pending == 1 << 30 { + propagate(&mut self.limbs); + self.pending = 0; + } + } + + /// The exact sum rounded to the nearest f64, ties to even: the one rounding. + fn value(&self) -> f64 { + if self.nan > 0 || (self.pos_inf > 0 && self.neg_inf > 0) { + return f64::NAN; + } + if self.pos_inf > 0 { + return f64::INFINITY; + } + if self.neg_inf > 0 { + return f64::NEG_INFINITY; + } + let mut limbs = self.limbs; + propagate(&mut limbs); + let negative = limbs[LIMBS - 1] < 0; + if negative { + limbs.iter_mut().for_each(|l| *l = -*l); + propagate(&mut limbs); + } + let magnitude = round_to_f64(&digits(&limbs)); + match negative { + true => -magnitude, + false => magnitude, + } + } +} + +/// Carry each limb into the next, leaving every limb but the top in [0, 2^32). +fn propagate(limbs: &mut [i64; LIMBS]) { + for k in 0..LIMBS - 1 { + let carry = limbs[k] >> LIMB_BITS; + limbs[k] -= carry << LIMB_BITS; + limbs[k + 1] += carry; + } +} + +/// A non-negative normalized accumulator as 32-bit digits, the top limb split out. +fn digits(limbs: &[i64; LIMBS]) -> Vec { + let mut out: Vec = limbs.iter().map(|&l| l as u64).collect(); + let mut top = out.pop().unwrap_or(0); + loop { + out.push(top & 0xffff_ffff); + top >>= LIMB_BITS; + if top == 0 { + break; + } + } + out +} + +fn bit(d: &[u64], pos: usize) -> u64 { + (d[pos / 32] >> (pos % 32)) & 1 +} + +/// Round an exact multiple of 2^-1074 to the nearest f64, ties to even. +fn round_to_f64(d: &[u64]) -> f64 { + let Some(k) = d.iter().rposition(|&x| x != 0) else { + return 0.0; + }; + let msb = 32 * k + 63 - d[k].leading_zeros() as usize; + if msb < 53 { + let int = d[0] | (d[1] << 32); + return int as f64 * f64::from_bits(1); + } + let mut mant = (0..53).fold(0u64, |m, i| m | (bit(d, msb - 52 + i) << i)); + let round = bit(d, msb - 53) == 1; + let sticky = (0..msb - 53).any(|p| bit(d, p) == 1); + let mut top = msb; + if round && (sticky || mant & 1 == 1) { + mant += 1; + if mant == 1 << 53 { + mant >>= 1; + top += 1; + } + } + let exp = top as i64 - 52 - 1074; + if exp > 971 { + return f64::INFINITY; + } + let scale = match exp >= -1022 { + true => f64::from_bits(((exp + 1023) as u64) << 52), + false => f64::from_bits(1u64 << (exp + 1074)), + }; + mant as f64 * scale +} + +fn exact_over(values: &[i64]) -> f64 { + let mut acc = ExactSum::new(); + values.iter().for_each(|&v| acc.add(typed(v), 1)); + acc.value() +} + fn ulps(a: f64, b: f64) -> u64 { (a.to_bits() as i64).abs_diff(b.to_bits() as i64) } +struct Sums { + exact: f64, + eager: f64, + naive: f64, + comp: f64, + xs_maintained: f64, + xs_fresh: f64, +} + /// Each running sum against the eager fold (the differential oracle) and against the /// exact decimal sum. `flip`: the gap reaches `FLOAT_EQ_EPSILON`, so `=` can answer differently. -fn report(label: &str, cycles: usize, exact: f64, eager: f64, naive: f64, comp: f64) { +fn report(label: &str, cycles: usize, s: &Sums) { let shown = |v: f64| format_f64(v, true, 2); println!( - "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e}", - (naive - eager).abs(), - ulps(naive, eager), - (comp - eager).abs(), - ulps(comp, eager), - (naive - eager).abs() >= 1e-10, - (eager - exact).abs(), - (naive - exact).abs(), - (comp - exact).abs(), + "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e} {:>7} {:>10.2e}", + (s.naive - s.eager).abs(), + ulps(s.naive, s.eager), + (s.comp - s.eager).abs(), + ulps(s.comp, s.eager), + (s.naive - s.eager).abs() >= 1e-10, + (s.eager - s.exact).abs(), + (s.naive - s.exact).abs(), + (s.comp - s.exact).abs(), + s.xs_maintained.to_bits() == s.xs_fresh.to_bits(), + (s.xs_maintained - s.exact).abs(), ); - if shown(naive) != shown(eager) || shown(eager) != shown(exact) { + if shown(s.naive) != shown(s.eager) || shown(s.eager) != shown(s.exact) { println!( - " shown at ,.2: naive {} eager {} exact {}", - shown(naive), - shown(eager), - shown(exact) + " shown at ,.2: naive {} eager {} exact {} maintained-exact-sum {}", + shown(s.naive), + shown(s.eager), + shown(s.exact), + shown(s.xs_maintained) ); } } @@ -126,6 +294,8 @@ fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) let mut naive = eager(&m); let mut comp = Compensated::default(); comp.add(naive); + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| xs.add(typed(v), 1)); let mut done = 0; for &at in checkpoints { while done < at { @@ -134,12 +304,21 @@ fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) naive = naive - typed(values[row]) + typed(new); comp.add(-typed(values[row])); comp.add(typed(new)); + xs.add(typed(values[row]), -1); + xs.add(typed(new), 1); values[row] = new; m.set_cell(key(row), CellValue::Number(typed(new))); done += 1; } - let label = format!("{cells} cells, {decades} dec"); - report(&label, at, exact(&values), eager(&m), naive, comp.value()); + let sums = Sums { + exact: exact(&values), + eager: eager(&m), + naive, + comp: comp.value(), + xs_maintained: xs.value(), + xs_fresh: exact_over(&values), + }; + report(&format!("{cells} cells, {decades} dec"), at, &sums); } } @@ -150,9 +329,11 @@ fn absorbed_by_a_large_value() { m.category_mut("Row").expect("just added").add_item("r0"); let mut naive = 0.0; let mut comp = Compensated::default(); + let mut xs = ExactSum::new(); m.set_cell(key(0), CellValue::Number(1e17)); naive += 1e17; comp.add(1e17); + xs.add(1e17, 1); for row in 1..=1000 { m.category_mut("Row") .expect("just added") @@ -160,23 +341,71 @@ fn absorbed_by_a_large_value() { m.set_cell(key(row), CellValue::Number(0.01)); naive += 0.01; comp.add(0.01); + xs.add(0.01, 1); } m.clear_cell(&key(0)); naive -= 1e17; comp.add(-1e17); - report( - "1e17, 1000 x 0.01, -1e17", - 1001, - 10.0, - eager(&m), + xs.add(1e17, -1); + let mut fresh = ExactSum::new(); + (0..1000).for_each(|_| fresh.add(0.01, 1)); + let sums = Sums { + exact: 10.0, + eager: eager(&m), naive, - comp.value(), + comp: comp.value(), + xs_maintained: xs.value(), + xs_fresh: fresh.value(), + }; + report("1e17, 1000 x 0.01, -1e17", 1001, &sums); +} + +/// Nanoseconds per operation over `n` values, best of three runs. +fn per_op(n: usize, mut op: impl FnMut() -> f64) -> f64 { + (0..3) + .map(|_| { + let t = Instant::now(); + black_box(op()); + t.elapsed().as_secs_f64() * 1e9 / n as f64 + }) + .fold(f64::INFINITY, f64::min) +} + +fn cost() { + let n = 10_000_000; + let mut rng = Rng(7); + let values: Vec = (0..n).map(|_| typed(decimal(&mut rng, 9))).collect(); + let plain = per_op(n, || { + values + .iter() + .fold(0.0, |s, &v| black_box(s + v) - black_box(v) + v) + }); + let exact_add = per_op(n, || { + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| { + xs.add(black_box(v), 1); + xs.add(black_box(v), -1); + xs.add(v, 1); + }); + xs.limbs[0] as f64 + }); + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| xs.add(v, 1)); + let read = per_op(1, || xs.value()); + println!(); + println!("cost, {n} values, best of 3 (per op = one insert + one retract + one insert):"); + println!(" plain f64 add/sub {plain:>8.2} ns per op"); + println!(" exact accumulator {exact_add:>8.2} ns per op"); + println!(" exact read (round) {read:>8.0} ns per read"); + println!( + " state per group {} bytes, against 8 for an f64", + std::mem::size_of::() ); } fn main() { println!( - "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10}", + "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10} {:>7} {:>10}", "case", "cycles", "naive-eag", @@ -186,7 +415,9 @@ fn main() { "flip", "eag-exact", "nai-exact", - "cmp-exact" + "cmp-exact", + "xs=fresh", + "xs-exact" ); let checkpoints = [1_000, 10_000, 100_000, 1_000_000]; for seed in [1, 2] { @@ -195,4 +426,5 @@ fn main() { random_updates(seed, 18_000, 9, &checkpoints); } absorbed_by_a_large_value(); + cost(); } From 25ebf641a92299ade54f25be2892de88020cb645 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 22:13:53 -0700 Subject: [PATCH 007/106] docs(adr): ADR 009 clause 5a computes today's aggregate; bound ExactSum's weights (improvise-felo.4) - 5a: group by context (the stem projection, plus the fold/total projection), not by record key. The inner WHERE is a filter plus clause 5's join. There is an integrated per-record stage before the map, and a rule for several writes to one record. The order key is (context rank, record store order), so the pre-step-6 re-fold is bit-equal to eager at folds and totals. The first-Err error channel is kept. - ExactSum splits weights into chunks of 2^30 and counts magnitude between carries. The example checks weight 2^33, and 20,000 weighted cases match exact fractions. The example prints the cost per update. - Q3(c) states the signed-zero difference and Eager's per-read cost. Q11 is narrowed: drawn-item order is already in view_shape. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 215 ++++++++++++------ .../examples/retraction_drift.rs | 54 +++-- 2 files changed, 190 insertions(+), 79 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 5e9d4aa8..d85c62ab 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -351,9 +351,11 @@ key, and `FIRST` the least. the group, and one entry per contributing record. That is the same order as `Min`/`Max`'s multisets. - **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. So the ordering category's - item order must be part of the operator shape (clause 7), and a change to it rebuilds - the ordered aggregators that use it. The alternative is to key by a stable item + reordering its items, changes every stored position. **Most of that is already in the + operator shape**: `view_shape` hashes each category's *drawn* items, in order + (`app.rs:47-52`), so a reorder of drawn items already rebuilds. What it does not see + is the position of a **hidden** item, which matters only if `LAST`/`FIRST` must order + values that come from hidden items. The alternative is to key by a stable item identity (ADR 007) and look the order up at read time. That makes a reorder free, but a read then costs O(d). This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, @@ -418,54 +420,108 @@ group and nothing else. `SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per record**, `Revenue * Price` at that record, and pools those values across the records -of a group. Today `agg_values` (`types.rs:89-139`) lists the member records of each -context and calls the record-level evaluator on each (`member_records`, then -`eval_at`). At a fold or a column total, the contexts come from `Expansion::members` -(`types.rs:201-213`), which materializes the cross product of the spanned categories' -items. That is the one O(grid) path in totals (ADR 008 clause 5). +of a **context**. It is the one place a formula needs an expression *inside* an +operator. This clause first states exactly what today's evaluator computes, because the +incremental form must compute the same thing. -**The design.** It is a three-stage pipeline over the record stream, and each stage is -an operator from clause 1: +**What eager computes** (`eval_agg`, `types.rs:55-87`, and `agg_values`, `:89-139`): -1. **A lifted map over the per-record expression.** For each record in the cell - delta's `record_of` set, the map produces `(base key, expr at record)`. A record - whose cells changed is retracted at its old value and inserted at its new one, so - the map's output is itself a delta. `map` is linear, so this is its own incremental - form. -2. **`index_by(base key)`.** The base key is the record's key with the target category - removed, which is what `eval_agg` pools over (`types.rs:55-87`). An inner `WHERE` - (`SUM(expr WHERE Cat = Item)`) is a `filter` before the index. It is the same - rewrite `eval_agg` applies to its contexts (`types.rs:66-72`). -3. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the - mapped values, with `Avg` derived. +- **Contexts.** The formula cell's key is the context at a stem. At a fold or a column + total, the contexts are the members `contexts_of` lists through `Expansion::members` + (`types.rs:201-213`, `:905-916`), the cross product of the spanned categories' covered + items. That is the one O(grid) path in totals (ADR 008 clause 5). +- **An inner `WHERE` rewrites the context.** A filter `Cat = Item` *replaces* the + context's `Cat` coordinate (`c.with(&f.category, &f.item)`, `types.rs:69`, and again at + `:116`), and the distinct rewritten contexts are pooled once each. So the stem + `Region=West`, evaluating `SUM(x WHERE Region=East)`, pools **East's** records. +- **Pooling.** For each context, `partial = context.without(target_category)` + (`types.rs:97`). The context's member records are those `member_records` finds under + `partial`, in the store's record order (a record sits where its first cell sits). + The expression is evaluated at each, and the per-context value lists are concatenated + **in context order** (`types.rs:73-77`). So the order is (context rank, record store + order): expansion-major at a fold or total, and plain store order at a stem. +- **Errors.** `agg_values` collects `Result`s (`types.rs:126-136`), and so does + `eval_agg` (`:76`). So the aggregate's value is the **first** `Err` in that order, + if any member errs. -Member records never have to be listed, and no cross product is ever built. A context -in today's cross product that holds no records contributed nothing to the pool, so -pooling only the records that exist is the same answer. +**The design.** Four stages, each an operator from clause 1: -**This is not compiling formulas into operators.** The operator is the map; the -expression inside it is still evaluated by `eval_expr` at the record, exactly as -`agg_values` calls it today. The circuit decides *which* records to re-evaluate (those -the delta names), and the one evaluator computes each value. Clause 2's rule stands: -there is one evaluator. +1. **An integrated per-record stage.** A cell delta does not carry a record: when only + `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged + `Revenue`, so the old mapped value cannot be computed from the delta alone. So the + first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each + live record's cells, as `ZSet<(measure, value)>` per record. On each step it emits, + for every record the consolidated delta touches: + - a retraction of the record's **previous** snapshot, read from the integrated state + before the step (`I(x)₋₁`); + - an insertion of its **new** snapshot. + + **Several writes to one record in one effect** consolidate before this stage (clause + 3 steps once per model effect), so the record is retracted once and inserted once, + at its net state. Writes in separate effects are separate steps. A record whose + snapshot is unchanged after consolidation emits nothing. +2. **A lifted map over record snapshots.** For each snapshot, the map produces one + entry per context the record belongs to: `(context key, order key, expr at record)`. + The value is a `Result`, so an error is a mapped value like any other. `map` is linear, + so a retracted snapshot retracts exactly the entries it once produced. + - **The context key** is the record projected onto the context's shape: the target + category removed and each `none_cats` category removed, per `none_cats` list, + exactly as clause 4 defines "the group". At a fold or total, a collapsed group's + member is also mapped to its group, and each spanned category is removed. That is + the same projection occupancy uses (ADR 008 clause 3). + - **The order key** is (context rank, record store order). The context rank is the + record's position in `Expansion::members` order, and it is 0 at a stem. The record + store order is the insertion sequence of the record's first live cell. `DataStore` + does not keep one today, so this stage assigns it; `IndexMap`'s relative order is + exactly first-insertion order among live cells. `DataStore::sort_by_key` + (`cell.rs:269`, on entry into records mode) reorders the store, so it is an + operator-shape change and rebuilds this stage. +3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records + with `Cat = Item` pass the filter, and are indexed by their context key **without** + `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of + clause 5 (stems indexed by "stem minus `Cat`"). A filter alone would put East's + records in East's group and leave West's read empty; the join is what makes West's + stem read East's pool, as the rewrite does today. +4. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the + mapped `Ok` values, with `Avg` derived. **The error channel** is kept beside it: per + group, the set of erring entries, ordered by order key. The aggregate reads the + first of them if the set is non-empty, reproducing today's first-`Err`. + +Member records are never listed and no cross product is built. A context in today's +cross product that holds no records contributed nothing to the pool, so pooling only +the records that exist gives the same answer. + +**This is not compiling formulas into operators.** The operators are the record stage, +the map, the index and the aggregate. The expression inside the map is still evaluated +by `eval_expr` at the record, exactly as `agg_values` calls it today through `eval_at` +(`types.rs:127-134`). The circuit decides *which* records to re-evaluate (those the +delta names), and the one evaluator computes each value. Clause 2's rule stands: there +is one evaluator. **The boundary.** The map is sound only when `expr` reads the record's own cells, so that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. That formula stays on the eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. +same boundary. The regression that pins the record stage changes **one operand** of a +two-operand expression (`Price` alone, under `SUM(Revenue * Price)`) and compares with +eager. -**What stays eager until step 5, and until step 6.** -- Before step 5 (formulas), every aggregate over an expression is evaluated as today, - through `Expansion::members`. -- From step 5, the map and the index are maintained, so the cross product is gone. - `Count`, `Min` and `Max` over the mapped values are read from the maintained state, - since they are order-free and bit-exact on f64 (clause 8). -- `Sum` and `Avg` over the mapped values are **re-folded** from the group's maintained - values, in the store's record order, until step 6. That keeps them bit-equal to - eager, at a cost of O(group) per change rather than O(grid). -- From step 6 (on question 3's answer), `Sum` is maintained like any other. +**What stays eager until step 5, and what is bit-equal before step 6.** +- **Before step 5** (formulas), every aggregate over an expression is evaluated as + today, through `Expansion::members`. +- **From step 5**, the record stage, the map and the index are maintained, so the cross + product is gone. `Count`, `Min` and `Max` over the mapped values, and the error + channel, are read from the maintained state. They are order-free, or ordered by the + order key, and bit-exact on f64 (clause 8). +- **`Sum` and `Avg` over the mapped values are re-folded** from the group's maintained + values in **order-key order** until step 6. That is (context rank, record store + order), which is exactly the order eager concatenates in, so the re-fold is bit-equal + to eager at a stem, a fold and a total. It costs O(group log group) per change, + against O(grid). +- **From step 6** (on question 3's answer), `Sum` is maintained like any other. Under + question 3's recommendation (c) the order key no longer matters for `Sum`, because + the exact accumulator is order-free. ### 6. Circular references: DBSP's recursion is not our semantics @@ -620,7 +676,15 @@ for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit i by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. Adding or retracting a value is integer addition, so the accumulator is a true abelian -group over the finite f64s. **Rounding happens once, when the value is read.** Then: +group over the finite f64s. **Rounding happens once, when the value is read.** + +**The group claim needs one stated bound.** A consolidated Z-set carries weights, not +just ±1: equal mapped values consolidate, so one element can arrive with a large weight. +The accumulator therefore splits any weight into chunks of at most 2³⁰, and counts +*magnitude* between carry propagations rather than calls. Each limb then stays below +2⁶² between carries, for any `i64` weight. The first version counted calls, and review +found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is +`1.82e-304`. The example now checks that case, and it reads the exact answer. Then: - `Sum` is the exact sum, rounded once. It is independent of order. - `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded @@ -629,7 +693,7 @@ group over the finite f64s. **Rounding happens once, when the value is read.** T - `Min` and `Max` are already exact. - Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. -**Measured**, in the same example, by `ExactSum` (147 lines, including its rounding): +**Measured**, in the same example, by `ExactSum` (158 lines, including its rounding and the weight bound): - **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over @@ -639,13 +703,21 @@ group over the finite f64s. **Rounding happens once, when the value is read.** T - **It is correct.** It was checked (not committed) against Python's `math.fsum` on 20,000 random cases. The cases include subnormals, values near the overflow bound, and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the - 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. -- **Cost, Apple M4 Max, best of 3 over 10M values:** - - about 3.2 ns per insert or retract, against about 2.3 ns for a plain f64 add in the - same loop; - - about 1.1 µs per read, which is the rounding, paid once per group read; + 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. After + the weight bound, it was checked again, on 20,000 cases with weights up to 2⁴⁰ + against exact `Fraction` sums: 0 mismatches. The review's own oracle run agreed. +- **Cost, Apple M4 Max, best of 3 over 10M values.** The example prints per single + update. + - This revision's three runs agreed: 2.0 ns per insert or retract, against 1.5 ns + for a plain f64 add in the same black-boxed loop, and 0.58 µs per read. + - The previous revision's run on the same machine gave 3.2 against 2.3 ns and 1.1 µs, + and the review's run gave 3.6 against 2.7 ns. + - So an update costs roughly 1.3× a plain add. A read, which is the rounding paid + once per group read, costs 0.6–1.1 µs. - **568 bytes per group**, against 8 for an f64. + The ranges below are between runs on one kind of machine; read your own run. + The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB of accumulators. A sparse form that stores only the occupied span of limbs would be a few limbs for typical data. **That is an estimate, not measured.** @@ -686,7 +758,7 @@ operator rather than one per call site. | `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, about 3.2 ns per update | bignum weight in the wasm bundle (improvise-l2b8's question 3) | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | bignum weight in the wasm bundle (improvise-l2b8's question 3) | **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -837,10 +909,12 @@ displayed path until the differential test holds. step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the frame observation) depend on this step.** 5. **Formulas**: the formula graph and the formula delta (clause 7), topological - re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression - map that replaces `Expansion::members` (clause 5). Formula values that depend on a - `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of - `Expansion::members` depends on this step.** + re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that + replaces `Expansion::members`: the integrated per-record stage, the map keyed by + context and order key, the inner-`WHERE` join, and the error channel. Its tests + include a change to one operand of a two-operand expression. Formula values that + depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR + 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** @@ -906,13 +980,20 @@ the user decided", as ADRs 004 and 007 did. **both** backends, so that incremental equals eager bit for bit. *Recommendation:* (c), in `zset`. It is measured bit-equal after a million - retract-and-insert cycles, and exact against `math.fsum` on 20,000 cases. It costs - about 3.2 ns per update and 568 bytes per group, and it does not wait for ADR 005. - (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for - per-cell arithmetic and typed values, and it composes with (c): under ADR 005 a - decimal sum is exact on its own. The cost to accept is that today's displayed sums - change in their last bits, because they become correctly rounded, and that - `Avg`'s one rounding must be stated in step 6. + retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and + its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and + typed values, and it composes with (c): under ADR 005 a decimal sum is exact on its + own. **The costs to accept:** + - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. + - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it + would use the same accumulator. A whole-grid eager projection pays it once per + summed cell. + - Today's displayed sums change in their last bits, because they become correctly + rounded. + - **Signed zero:** the accumulator reads `[−0.0]` as `+0.0`, where the f64 `sum()` + that `eval_agg` uses (`types.rs:79`) gives `−0.0`. + - `Avg`'s one rounding must be stated in step 6. 4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? @@ -962,10 +1043,12 @@ the user decided", as ADRs 004 and 007 did. position, with that category's item order added to the operator shape so that a reorder rebuilds them? Or should they key by stable item identity (ADR 007), and look the order up at read time? - *Recommendation:* item position, with the order in the operator shape. Reads stay - O(log d), like `Min`/`Max`, and the same generic operator serves all four. An item - reorder is a rare structural edit, and clause 7 already rebuilds on those. This - extends the operator shape's `measure_agg` part to cover "and the item order of - each ordering category". ADR 008 clause 3 lists the parts as `view_shape`, group - membership and `measure_agg`. This reading changes what the `measure_agg` part - covers, not the list, so ADR 008's text and cross-references still hold. + *Recommendation:* item position. Reads stay O(log d), like `Min`/`Max`, and the + same generic operator serves all four. The order of **drawn** items is already in + `view_shape` (`app.rs:47-52`), so a reorder of drawn items already rebuilds, and + nothing new is needed for them. The one open part is **hidden** items, whose + positions `view_shape` does not hash. Recommended: `LAST`/`FIRST` order only the + values the fold covers, which are the drawn members. Then the operator shape needs + no widening at all, and ADR 008's text and cross-references hold unchanged. If the + user wants hidden items ordered too, the `measure_agg` part must also cover their + positions. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 26487b9e..407a7ab5 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -96,13 +96,15 @@ impl Compensated { const LIMB_BITS: u32 = 32; /// Every finite f64 is `mant * 2^(shift - 1074)` with `shift <= 2045` and `mant < 2^53`. const LIMBS: usize = 67; +/// The most weight one limb absorbs between carries: each add moves a limb by < 2^32. +const CHUNK: i64 = 1 << 30; /// An exact sum of f64s: a signed fixed-point integer in units of 2^-1074, wide enough /// for every finite f64, with non-finite values counted so they can be retracted too. #[derive(Clone)] struct ExactSum { limbs: [i64; LIMBS], - pending: u32, + pending: u64, nan: i64, pos_inf: i64, neg_inf: i64, @@ -119,17 +121,30 @@ impl ExactSum { } } - /// Add `x` with `weight` +1 (insert) or -1 (retract). Exact: no rounding happens here. + /// Add `x` with any `weight` (a consolidated Z-set weight). Exact: no rounding here. fn add(&mut self, x: f64, weight: i64) { match (x.is_nan(), x.is_infinite()) { (true, _) => self.nan += weight, (false, true) if x > 0.0 => self.pos_inf += weight, (false, true) => self.neg_inf += weight, - (false, false) => self.add_finite(x, weight), + (false, false) => { + let mut left = weight; + while left != 0 { + let chunk = left.clamp(-CHUNK, CHUNK); + self.add_finite(x, chunk); + left -= chunk; + } + } } } + /// Bounded so each limb stays under 2^62: `|weight| <= CHUNK` and `pending` + /// counts magnitude, not calls. fn add_finite(&mut self, x: f64, weight: i64) { + if self.pending + weight.unsigned_abs() > CHUNK as u64 { + propagate(&mut self.limbs); + self.pending = 0; + } let bits = x.to_bits(); let biased = ((bits >> 52) & 0x7ff) as u32; let fraction = bits & ((1 << 52) - 1); @@ -148,11 +163,7 @@ impl ExactSum { let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64; self.limbs[k + i] += sign * part; } - self.pending += 1; - if self.pending == 1 << 30 { - propagate(&mut self.limbs); - self.pending = 0; - } + self.pending += weight.unsigned_abs(); } /// The exact sum rounded to the nearest f64, ties to even: the one rounding. @@ -375,12 +386,13 @@ fn cost() { let n = 10_000_000; let mut rng = Rng(7); let values: Vec = (0..n).map(|_| typed(decimal(&mut rng, 9))).collect(); - let plain = per_op(n, || { + let updates = 3 * n; + let plain = per_op(updates, || { values .iter() .fold(0.0, |s, &v| black_box(s + v) - black_box(v) + v) }); - let exact_add = per_op(n, || { + let exact_add = per_op(updates, || { let mut xs = ExactSum::new(); values.iter().for_each(|&v| { xs.add(black_box(v), 1); @@ -393,9 +405,9 @@ fn cost() { values.iter().for_each(|&v| xs.add(v, 1)); let read = per_op(1, || xs.value()); println!(); - println!("cost, {n} values, best of 3 (per op = one insert + one retract + one insert):"); - println!(" plain f64 add/sub {plain:>8.2} ns per op"); - println!(" exact accumulator {exact_add:>8.2} ns per op"); + println!("cost, {n} values x (insert, retract, insert), best of 3, per single update:"); + println!(" plain f64 add/sub {plain:>8.2} ns per update"); + println!(" exact accumulator {exact_add:>8.2} ns per update"); println!(" exact read (round) {read:>8.0} ns per read"); println!( " state per group {} bytes, against 8 for an f64", @@ -403,6 +415,21 @@ fn cost() { ); } +/// A consolidated Z-set weight can be large; the accumulator must stay exact at 2^33. +fn large_weight() { + let x = f64::from_bits((1u64 << 32) - 1); + let mut xs = ExactSum::new(); + xs.add(x, 1 << 33); + let want = ((1u64 << 32) - 1) as f64 * 2f64.powi(33) * f64::from_bits(1); + println!(); + println!( + "large weight: (2^32-1) x 2^-1074 at weight 2^33 reads {:e}, exact {:e}, equal {}", + xs.value(), + want, + xs.value().to_bits() == want.to_bits() + ); +} + fn main() { println!( "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10} {:>7} {:>10}", @@ -426,5 +453,6 @@ fn main() { random_updates(seed, 18_000, 9, &checkpoints); } absorbed_by_a_large_value(); + large_weight(); cost(); } From 8ebcf889bdcb3f6976565870f4c4db90ef15d849 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 23:02:00 -0700 Subject: [PATCH 008/106] docs(adr): ADR 009 matches ADR 005 revision 2 on exact decimal sums (improvise-felo.4) A decimal Sum is a group only through ADR 005's wide accumulator, used by both backends. rust_decimal's own + is not associative at its 28-digit bound. ExactSum does not make = against a typed decimal right: it can miss in the last bit. The decimal cost cell now uses ADR 005's measurement, and exact-to-f64 conversion goes through the decimal string. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 ++++++++++++++++++----- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index d85c62ab..b246be9b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -5,7 +5,7 @@ are theirs to answer on the PR. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8, no file yet), ADR 008 (the +**Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid, PR #308, which is revised to assume this ADR) · **Governed by:** `context/design-principles.md` §3 and §5 @@ -656,8 +656,12 @@ for floating point. On its own, **none**: the absorption case loses the answer entirely. **The benefits come from two other places.** -- **The representation.** An exact accumulator (next subsection) or exact decimals - (ADR 005) make `Sum` a true abelian group. Then retraction is exact, incremental +- **The representation.** An exact accumulator (next subsection), or ADR 005's + decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian + group, provided both backends use the same accumulator. Decimals alone do not: ADR + 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at + its 28-digit bound, so the order of summation changes both the value and the + exactness. Then retraction is exact, incremental equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes with it. @@ -740,9 +744,14 @@ a differential test against an exact rational sum. domain where the operation is exact, and apply the lossy operations at the edge: - sums go in the accumulator; -- decimal add, subtract and multiply are exact under ADR 005; +- decimal add, subtract and multiply are exact under ADR 005, within `rust_decimal`'s + 28-digit bound, which ADR 005 detects rather than silently rounds; - division, `POW` with a non-integer exponent, and rounding for display happen only - where a value leaves the exact domain. + where a value leaves the exact domain; +- **an exact value becomes an f64 through its decimal string and `str::parse::`**, + which is correctly rounded. `rust_decimal`'s `Decimal::to_f64` is not: ADR 005 + measured it disagreeing in 17,375 of 200,000 values. `ExactSum`'s own read is + correctly rounded by construction (clause 8's oracle checks). This is honest only about where the gain comes from. **The representation earns it, not DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named @@ -755,10 +764,25 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | **no**, in general: it sums the stored f64s exactly, and those are already rounded, so the result can miss the typed total in the last bit (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | bignum weight in the wasm bundle (improvise-l2b8's question 3) | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | + +**The accumulator does not make `=` against a typed decimal right.** It sums the +*stored* f64s exactly, and each stored f64 is already the rounded form of what the user +typed. So the exact sum of the stored values, rounded once, can differ in the last bit +from the typed total, rounded once. +- **ADR 005 revision 2 measured it** with this ADR's `ExactSum` as the stand-in, on + typed two-place values. The result missed the typed total in 21%, 16%, 7.5% and 2.5% + of 2-, 3-, 10- and 100-term sums, and in 0 of 8 seeds at 18k and 90k cells. +- **A spot check** (not committed) with two-place values up to ±1,000 missed in about + 38–40% of 2- to 100-term sums. +- **This ADR's own example** missed in 1 of 25 large-sum checkpoints. + +So the miss rate depends on the values' magnitudes, not only on the term count. The +misses are last-bit: they are visible to `=` and in the editor, not in a display +format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -983,8 +1007,10 @@ the user decided", as ADRs 004 and 007 did. retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and - typed values, and it composes with (c): under ADR 005 a decimal sum is exact on its - own. **The costs to accept:** + typed values, and it composes with (c): under ADR 005, a decimal sum is exact + through ADR 005's own wide accumulator (its question 10), provided both backends use + it, exactly as (c) requires for `ExactSum`. `rust_decimal`'s own `+` is not + associative at its bound, so decimals alone are not enough. **The costs to accept:** - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it would use the same accumulator. A whole-grid eager projection pays it once per From 09df35dffbe722a2ee775c24382fefbef45f3fb0 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:01:06 -0700 Subject: [PATCH 009/106] docs(adr): ADR 009 names ADR 005's stand-in correctly; the miss rate follows the total's size (improvise-felo.4) ADR 005 measured with the exact rational of the stored f64s, rounded once, not with ExactSum itself. The typed-total miss rate depends on how large the total is relative to its terms, not on the values' magnitudes. Both of ADR 005's distributions are cited. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b246be9b..3ed26294 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -773,14 +773,21 @@ operator rather than one per call site. *stored* f64s exactly, and each stored f64 is already the rounded form of what the user typed. So the exact sum of the stored values, rounded once, can differ in the last bit from the typed total, rounded once. -- **ADR 005 revision 2 measured it** with this ADR's `ExactSum` as the stand-in, on - typed two-place values. The result missed the typed total in 21%, 16%, 7.5% and 2.5% - of 2-, 3-, 10- and 100-term sums, and in 0 of 8 seeds at 18k and 90k cells. -- **A spot check** (not committed) with two-place values up to ±1,000 missed in about - 38–40% of 2- to 100-term sums. +- **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, + rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not + `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result + missed the typed total in 21%, 16%, 7.5% and 2.5% of 2-, 3-, 10- and 100-term sums, + and in 0 of 8 seeds at 18k and 90k cells. On two-place values uniform in ±1,000.00, it + missed in 38–40% at every one of those term counts. +- **A spot check** of this ADR (not committed), on the ±1,000 values, agreed: about + 37–40% of 2- to 100-term sums missed. - **This ADR's own example** missed in 1 of 25 large-sum checkpoints. -So the miss rate depends on the values' magnitudes, not only on the term count. The +So the miss rate depends on **how large the total is relative to its terms**, not on +the term count or on how large the values are. When the terms share a sign, the total +grows, and its last-bit step grows with it, so the terms' own rounding errors fall +below it and misses become rare. When the terms cancel, the total stays small, and +misses stay near 40%. The misses are last-bit: they are visible to `=` and in the editor, not in a display format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. From 284ac5b3e30281bb52b905493619d5a034d7bfac Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:06:54 -0700 Subject: [PATCH 010/106] docs(adr): ADR 009 hedges ExactSum's home on ADR 005 question 15 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 3ed26294..1012dd2c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -738,7 +738,9 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. +a differential test against an exact rational sum. Subject to ADR 005 question 15, +`ExactSum` may live in `numtower` instead and be wrapped by the seam crate, so that +`zset` stays free of numbers. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -1010,7 +1012,8 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c), in `zset`. It is measured bit-equal after a million + *Recommendation:* (c), in `zset`. Subject to ADR 005 question 15, `ExactSum` may + live in `numtower` instead and be wrapped by the seam crate. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and From b2c125918c024eae1ade3b2e234a3691418d895a Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:54:23 -0700 Subject: [PATCH 011/106] docs(adr): ADR 009 orders records by their current leading cell; qualify "visible to =" (improvise-felo.4) - 5a: a per-cell store sequence. A record's order key is the minimum over its live cells, carried in the snapshot, so deleting a leading cell re-emits the record as eager's shift_remove reorders it. A delete-leading-cell regression is added. Formulas with a non-_Measure target stay eager until 5a defines their projection. - A last-bit miss against a typed total is visible to = only above about 2^19, because = is an absolute 1e-10. The table row and paragraph now match ADR 005. - "not on the term count as such"; the numbering note says ADR 005 is on #320; ExactSum's home defers to ADR 005 question 15; the chunk loop's cost is noted. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 100 ++++++++++++++++------ 1 file changed, 74 insertions(+), 26 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1012dd2c..d60aec48 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -9,8 +9,8 @@ are theirs to answer on the PR. · virtualized grid, PR #308, which is revised to assume this ADR) · **Governed by:** `context/design-principles.md` §3 and §5 -**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl and have -no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 +is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are @@ -450,8 +450,9 @@ incremental form must compute the same thing. `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged `Revenue`, so the old mapped value cannot be computed from the delta alone. So the first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each - live record's cells, as `ZSet<(measure, value)>` per record. On each step it emits, - for every record the consolidated delta touches: + live record's cells, as `ZSet<(measure, value)>` per record, **each cell with its + store sequence** (below). On each step it emits, for every record the consolidated + delta touches, including a cell the expression never reads: - a retraction of the record's **previous** snapshot, read from the integrated state before the step (`I(x)₋₁`); - an insertion of its **new** snapshot. @@ -470,12 +471,30 @@ incremental form must compute the same thing. member is also mapped to its group, and each spanned category is removed. That is the same projection occupancy uses (ADR 008 clause 3). - **The order key** is (context rank, record store order). The context rank is the - record's position in `Expansion::members` order, and it is 0 at a stem. The record - store order is the insertion sequence of the record's first live cell. `DataStore` - does not keep one today, so this stage assigns it; `IndexMap`'s relative order is - exactly first-insertion order among live cells. `DataStore::sort_by_key` - (`cell.rs:269`, on entry into records mode) reorders the store, so it is an - operator-shape change and rebuilds this stage. + record's position in `Expansion::members` order, and it is 0 at a stem. + - **The record store order is the minimum store sequence over the record's *live* + cells**, recomputed every time the record is touched, and carried in the snapshot. + Eager orders records by their **current** earliest live cell in `DataStore::cells` + (`member_records`), and `shift_remove` changes that when a record's leading cell + goes. So the stage keeps a **per-cell** sequence, assigned when a key is first + inserted: + - a same-key retract and insert within one step is an in-place overwrite, so the + cell keeps its sequence, as `IndexMap::insert` does; + - a key inserted again after a removal in an earlier step gets a fresh sequence, at + the end, as `IndexMap` puts it; + - losing a record's leading cell changes its minimum, so the record is re-emitted + with its new order key. + + `IndexMap`'s relative order is exactly this sequence order among live cells. + `DataStore::sort_by_key` (`cell.rs:269`, on entry into records mode) reorders the + store, so it reassigns the sequences, and as an operator-shape change it rebuilds + this stage. + + The review demonstrated why a per-*record* sequence is not enough, against eager at + `57d385e1`. With `S = SUM(Rev * 1)` and store order `A.Other=5, B.Rev=1, + C.Rev=1e16, A.Rev=1, D.Rev=-1e16`, eager reads 2.0. Removing `A.Other`, a cell the + expression never reads, moves A behind C, and eager reads 0.0. A sequence fixed + when A first appeared would still read 2.0. 3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records with `Cat = Item` pass the filter, and are indexed by their context key **without** `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of @@ -503,9 +522,19 @@ that its value at a record changes only when that record changes. An inner expre that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. That formula stays on the eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. The regression that pins the record stage changes **one operand** of a -two-operand expression (`Price` alone, under `SUM(Revenue * Price)`) and compares with -eager. +same boundary. Two regressions pin the record stage against eager: +- one changes **one operand** of a two-operand expression (`Price` alone, under + `SUM(Revenue * Price)`); +- one **deletes a record's leading cell**, including a cell the expression never reads + (the `A.Other` case above), and checks both the value and which error is reported + first. + +**5a covers formulas whose target category is `_Measure`.** For a formula that targets +another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A record, +whose key `record_of` takes without `_Measure`, then belongs to the context only through +its cell under that measure, and eager orders it by *that* cell's position, not by the +record's first live cell. Until 5a defines that projection and its order key, such +formulas stay on the eager path, and the differential test pins that they do. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as @@ -688,7 +717,13 @@ The accumulator therefore splits any weight into chunks of at most 2³⁰, and c *magnitude* between carry propagations rather than calls. Each limb then stays below 2⁶² between carries, for any `i64` weight. The first version counted calls, and review found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is -`1.82e-304`. The example now checks that case, and it reads the exact answer. Then: +`1.82e-304`. The example now checks that case, and it reads the exact answer. + +The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight +near `i64::MAX`. The version in `zset` (or `numtower`) splits the weight into base-2³⁰ +digits instead and adds each as a shifted value. That is exact in the fixed-point +representation and O(1) per add. In practice a weight is bounded by a record count +anyway. Then: - `Sum` is the exact sum, rounded once. It is independent of order. - `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded @@ -738,9 +773,10 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. Subject to ADR 005 question 15, -`ExactSum` may live in `numtower` instead and be wrapped by the seam crate, so that -`zset` stays free of numbers. +a differential test against an exact rational sum. **Where it lives is ADR 005 +question 15's decision** (recommended there: a separate `numtower` crate, wrapped by +this ADR's seam crate so that `zset` stays free of numbers). This ADR defers to that +answer. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -766,7 +802,7 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | **no**, in general: it sums the stored f64s exactly, and those are already rounded, so the result can miss the typed total in the last bit (see below) | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | yes below about 5e5, no above: it sums the stored, already-rounded f64s exactly, so it can miss the typed total in the last bit, and `=`'s absolute 1e-10 absorbs that only while the total is below about 2¹⁹ (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | | cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | @@ -786,12 +822,23 @@ from the typed total, rounded once. - **This ADR's own example** missed in 1 of 25 large-sum checkpoints. So the miss rate depends on **how large the total is relative to its terms**, not on -the term count or on how large the values are. When the terms share a sign, the total +the term count as such, nor on how large the values are. When the terms share a sign, the total grows, and its last-bit step grows with it, so the terms' own rounding errors fall below it and misses become rare. When the terms cancel, the total stays small, and -misses stay near 40%. The -misses are last-bit: they are visible to `=` and in the editor, not in a display -format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. +misses stay near 40%. + +**Where a miss is visible.** The misses are last-bit, so they show in the editor and +to a raw comparison, not in a display format. **They are visible to `=` only once the +total is above about 2¹⁹ ≈ 5.2e5**, because `=` is an absolute `1e-10` (gotcha 17), +and one ulp reaches it only at that magnitude. The same holds if ADR 005's epsilon-free +`=` is adopted (its question 3), since that has no tolerance at all. So: +- on the ±1,000 values, where misses are most frequent, the review counted 0 of 7,506, + 0 of 7,901 and 0 of 7,963 misses visible to `=` at 2, 10 and 100 terms; +- on positive values up to 999,999.99, most misses are visible, because the totals are + large. + +ADR 005 states the same condition. Only exact decimal cells (ADR 005) make `=` against +a typed total exact at every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -945,7 +992,8 @@ displayed path until the differential test holds. re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that replaces `Expansion::members`: the integrated per-record stage, the map keyed by context and order key, the inner-`WHERE` join, and the error channel. Its tests - include a change to one operand of a two-operand expression. Formula values that + include a change to one operand of a two-operand expression, and the deletion of a + record's leading cell. Formula values that depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact @@ -1012,9 +1060,9 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c), in `zset`. Subject to ADR 005 question 15, `ExactSum` may - live in `numtower` instead and be wrapped by the seam crate. It is measured bit-equal after a million - retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + *Recommendation:* (c). Where `ExactSum` lives is ADR 005 question 15's decision + (recommended there: `numtower`, wrapped by the seam crate), and this ADR defers to + it. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and typed values, and it composes with (c): under ADR 005, a decimal sum is exact From 051c2389f6f4d9fca335a49f738909a388a7d1eb Mon Sep 17 00:00:00 2001 From: "Claude (agent-adr007)" Date: Fri, 25 Sep 2026 11:58:41 -0700 Subject: [PATCH 012/106] docs(adr): ADR 007 revision 9, answers 30-32 and the review text fixes (improvise-cq1q.1) - Folds in the user's answers on improvise-m56i (items 2 and 6) and improvise-hexb (item 2) as answers 30-32. The m56i readings (items 1, 3, 4 and 5) go in clause 5 as the formula line. - Applies the planned text fixes: le4p, p003, the i0n7 remainder, hexb items 1, 3 and 4, 8vs2 item 4, the 8tk5 merge sentence, and PR #318's two slugify readings. - Asks one new question, Q23: how a formula line spells its target. - Changes no earlier decision. As an amendment to an accepted ADR, it needs the user's approval. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 199 ++++++++++++++++++++++++--- 1 file changed, 178 insertions(+), 21 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 20c98836..694fa401 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -4,7 +4,12 @@ followed the adversarial review of that head: it adds clause 12, the ADR's own reading of when a formula edit's names become ids, and corrects stale facts about main. The user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17:10Z: -"looks good, approved"). · +"looks good, approved"). +**Revision 9 is a proposed amendment, not yet approved.** It folds in the user's +answers on the implementation tickets (answers 30–32) and clarifies text the +reviews found unclear or stale (improvise-cq1q.1). It changes no earlier decision. +It asks one new question, Q23 (a formula line's spelling), and flags three places +where it states the ADR's own reading. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -321,6 +326,35 @@ data key and reference in that file spells it. Clause 4 says what follows: id keeps its spelling; - two identical hand-written ids in one file are refused at load, not repaired. +### Seventh round: answers on the implementation tickets + +After the ADR was accepted, planning its implementation (epic improvise-cq1q) +raised three questions the ADR does not answer. The user answered them on +2026-09-25. The answers are recorded on the tickets and quoted here verbatim. + +**30. A formula target outside `_Measure`** (improvise-m56i item 2): + +> "Real item + separate formula id" + +`Model::add_formula` keeps creating a real item for a target outside `_Measure`. +That item has its own `ItemId`, the formula has its own `FormulaId`, and by +answer 22 the two differ. A formula line names such a target by the item's id. +This was chosen over the recommended alternative, which was to give every formula +target the `_Measure` treatment (no real item, handled by its `FormulaId`). + +**31. A typed name that is both one thing's display and another thing's id** +(improvise-m56i item 6): + +> "Display first; client holds it" + +A typed name is matched against displays first, then against ids. A display that +matches more than one thing is refused, and the refusal lists the candidates' ids. + +**32. Which side holds clause 12's capture** (improvise-hexb item 2): the +**client**, as it holds `OpenedCell`. It resolves typed names against its +id-bearing projection, the wire carries only id-form text, and the host never +resolves a display name. Clause 12 is rewritten to match. + ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. @@ -341,9 +375,11 @@ first message names what that costs: **What already exists.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize` aliases, assigned from monotone counters. **No production caller uses either value.** -`Model::add_category` and `Category::add_item` return them on the idempotent path, and -nothing reads the returned value; the only reader is one test. Neither has ever been written to `.improv`. `View` has no -id. +`Category::add_item` returns the id on every call, and `Model::add_category` returns +one too. Tests read them (`category.rs`'s `add_item_returns_sequential_ids` and +`add_item_duplicate_returns_same_id`, and one in `types.rs`), but no production +caller does: `parse.rs` discards the value. Neither has ever been written to +`.improv`. `View` has no id. **`FormulaId` exists, as an interim.** km71 landed an opaque `FormulaId` whose private fields are `(target, target_category)`, built only by `Formula::id()` @@ -390,7 +426,23 @@ categories. An operation that creates an item still names the parent category. with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** A `_Measure` formula target has no item of its own. Its handle is its formula's `FormulaId`, which is why the category panel's `_Measure` rows need -`ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). +`ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). **A formula +target outside `_Measure` is a real item** (answer 30). It has its own `ItemId`, +and its formula has a separate `FormulaId`. Generated ids follow clause 4, so a +formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-`. +The panel lists that row as `ItemHandle::Item`, like any other item. + +**The virtual views have reserved ids too:** `_records` and `_drill`, with the +displays `_Records` and `_Drill`. They are never persisted, but they are +addressed on the wire (`SwitchView`, improvise-obtu). Answer 23 reserves a +leading `_` for virtual things, and a slugified `_Drill` would otherwise be +`drill`, which could collide with a user's view (improvise-p003 item 1). + +**The ADR's reading, not a user answer: `_Index` and `_Dim` coordinates are +outside the id namespace.** They are synthetic record positions (`_Dim` carries a +column index, gotcha 1). They are never persisted and never typed, so they take no +part in answer 22's uniqueness, and `_index`'s `0` cannot collide with a user item +whose id is `0` (improvise-p003 item 2). ### 2. Storage keys by id; `display` is the only place a human name lives @@ -402,7 +454,10 @@ Following answers 1 to 3: canonical form is kept with a different sort key. `DataStore` interns ids exactly as it interns names today. - **View configuration:** the axis maps, page selections, and hidden and collapsed sets - are keyed by ids. + are keyed by ids. The one exception is a collapsed entry, which names a group: + groups have no id (answer 9), so it is `(CategoryId, group name)`, and + `collapsed:` writes the category id, then the group name (improvise-p003 + item 4). - **Formulas:** a formula's references are ids (clause 5). The formula panel renders them as display names (answer 19; clause 6). @@ -473,6 +528,16 @@ Examples: `East` → `east`, `Time Period` → `time_period`, `Coastal_East` → 22), the second is `east-`. Here `` is the counter's next value, for example `east-1`. +**Two readings of the steps, the ADR's own and not user answers**, chosen by the +implementation (PR #318; `context/code/core.md` §20): +- **"Space" means U+0020.** Only a run of U+0020 becomes `_`. A tab, a no-break + space (U+00A0) and an ideographic space (U+3000) are not spaces here, so step 3 + strips them: `Time\tPeriod` → `timeperiod`, which collides with `TimePeriod`, + not with `Time Period`. A tab-separated import header is the realistic case. +- **A slug of only `-` is a legal bare id.** `-` and `---` slug to themselves. + Step 5's "if nothing is left" means the empty string only. The id alphabet + admits them, so a line such as `-: row` is legal but hard to read. + The remaining rules: - **A file has ids on everything or on nothing** (answer 21). An id-less file (old @@ -511,6 +576,11 @@ The remaining rules: `east` again. An in-flight command from before the reload is refused by the generation tier. The one remaining risk is an outside reference that outlives the file, and none exists today. +- **Two branches of one file can merge into a file that will not load.** If each + branch adds a different thing whose id generates to the same slug, the merged + file declares that id twice. Its load is refused (answers 11 and 29), and the + error names both declarations, so the merge must be repaired by hand. Stable + identity across copies and branches of a file is not promised (improvise-8tk5). ### 5. Persistence: ids on declarations *and* references @@ -558,6 +628,32 @@ id_ref = { id_token } and takes an `id_ref`. A group reference (`[Coastal]`) stays a name while groups have no id (answer 9). +**An axis line is tried before `format:`** (improvise-le4p). `view_entry` is an +ordered choice, and `format_line` takes the rest of the line, so a category whose +id is `format` would lose its axis line to the view's number format. In the +new-version arm, `axis_line`, which requires an axis keyword after `: `, comes +before `format_line`. No number format is spelled as an axis keyword, so neither +rule loses a line. The id `format` needs no quoting, and `id_ref` keeps no quoting +arm. The required test is a category displayed `Format`, on a row axis, that +round-trips. + +**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). The shape is +`- () = []`: +- **The parenthesised id is the formula's own `FormulaId`** (answer 5's + placement). +- **`` is the target item's id when the target is an item** (outside + `_Measure`, answer 30). For a `_Measure` target, which has no item, it is the + target's display. + - `Formula` carries that display, and a display rename of a `_Measure` target + writes it. + - Whether one line should spell both cases this way is **Q23**, below. +- **The expression is over ids**, spelled through `quote::quote_name` as above. +- **A `WHERE`/`IS` filter value is an item id**, bare or quoted (answer 3: every + reference is an id). The formula panel shows it as a display. +- **A formula cell's `CellKey`** carries the target item's `ItemId` outside + `_Measure`. For a `_Measure` target, it carries the `FormulaId` text under + `_measure`, which answer 22 makes unique. + **`header_display` is guarded only by a lookahead**, which the generator skips (gotcha 34). So the generator may emit a display ending in something that later parses as an id. Round-trip property 3 still holds, because the first parse settles the split. But @@ -646,9 +742,19 @@ pipe-quotes any display name that would otherwise read as display plus id: - `ViewSummary { id, display, … }`. `AxisFrame` labels stay display strings. They gain ids when a consumer needs them. -- **`ViewState` keys by id**: the cursors, `expanded_cats`, and the view history - (`view_back_stack` / `view_forward_stack`). Answer 1 makes this natural rather than a - migration of its own. A view or category rename then orphans nothing. +- **`ViewState` keys by id**: the cursors, `expanded_cats`, the view history + (`view_back_stack` / `view_forward_stack`) and `active_view` itself, as does + `Workbook::active_view` (improvise-p003 item 3). Answer 1 makes this natural + rather than a migration of its own. A view or category rename then orphans + nothing. +- **A `_Measure` formula-target row** (`ItemHandle::FormulaTarget(FormulaId)`): + removing it means `RemoveFormula`, and renaming it means a display rename of + the formula's target (improvise-i0n7). +- **CheckedBatch's created-record exemption goes** (improvise-p003 item 5). It + exempts a record that neither resolves nor is claimed, for frames like + `[AddItem(East), SetCell(East)]`. Under this clause a frame cannot address what + it creates, so nothing produces such a frame any more. The wire step removes + the exemption rather than leaving dead reasoning in place. ### 7. ADR 001: an id that no longer resolves is gone @@ -750,6 +856,14 @@ Following answers 15 and 25: 5. **Projections and `ViewState`** (clause 6). 6. **Vocabulary:** the id-addressed variants, `Rejection::Gone`, `ids_named` and `broken_chain`'s id arm. +7. **Clause 12's capture:** the client-side display→id mapping taken when the + formula editor opens, and the resolver that turns typed names into id-form + text (answers 31 and 32). + +The implementation epic (improvise-cq1q) orders these steps to keep `main` +shippable after each one, which the list above does not: the display field +before the re-key, and projection ids before the re-key. That ordering is the +epic's, not a change to this list. ### 12. A formula edit's names are fixed when the editor opens @@ -772,22 +886,34 @@ is the capture-at-open rule the cell editor already follows: `OpenedCell` in `improvise-protocol/src/client.rs` records the record, its version, the document generation and the view at open, and the commit answers for those rather than for whatever the document holds by then. +- **The client holds the mapping** (answer 32), as it holds `OpenedCell`. The TUI + and the browser both use the same resolver in `improvise-protocol`. - The mapping covers every display name the panel showed in that formula's text, including the target's. -- A name the user types that was **not** in the captured mapping is new to the edit. It - is resolved against the document at commit, as any typed name is. If it does not - resolve, the edit is refused and says which name. -- A captured id that no longer exists at commit is `Rejection::Gone` (clause 7). +- A name the user types that was **not** in the captured mapping is new to the + edit. The client resolves it against its own id-bearing projection (clause 6) at + commit, in answer 31's order: displays first, then ids. If a display matches more + than one thing, the edit is refused and the refusal lists the candidates' ids. If + the name does not resolve at all, the edit is refused and names it. +- **The wire carries only id-form text.** `ReplaceFormula.raw` and + `AddFormula.raw` are in the file's id form, and the host never resolves a display + name (answer 32). The host parses ids, and one that no longer exists is + `Rejection::Gone` (clause 7). - `was` is still compared against the stored id-form text (clause 6), so a concurrent *replace* of the formula is still seen. -**Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. From -another client, rename the item `Revenue` to `Sales` and rename a different item to -`Revenue`. Commit the unedited text. The formula must still reference the original +**Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. +From another client, rename the item `Revenue` to `Sales` and rename a different +item to `Revenue`. Then edit something that is **not a name**: change the text to +`Revenue - Cost * 1` and commit it. The stored formula must reference the original item's id, and the commit must not be refused. The test must fail under commit-time resolution, which binds to the new `Revenue`, and it must fail under a refusal on rename. Both are the rivals this clause rules out. +Revision 8's version committed the *unedited* text. A client that treats an +unchanged text as a no-op passes it without resolving any name, so the test could +not tell a correct capture from no capture at all (improvise-hexb item 1). + ## What this means for you ### improvise-km71 — editing a formula (landed as an interim) @@ -852,7 +978,8 @@ fdk1 landed in PR #221, before this ADR. What exists on main: The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14, revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All -are answered.** +of those are answered.** After acceptance, three implementation questions were +answered on their tickets (answers 30–32). Revision 9 adds Q23, which is open. | # | question | my recommendation | status | |---|---|---|---| @@ -880,6 +1007,10 @@ are answered.** | 21 | An old file naming an undeclared item | (a) declare it at load | **answered: (a)** (answer 24) | | — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal | | 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) | +| — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation | +| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31) | +| — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32) | +| 23 | A formula line's spelling (revision 9) | see "Still open" | **open** | ## Still open @@ -887,8 +1018,33 @@ Nothing is open among Q1–Q22, and answer 29 settles the one case it raises: tw identical hand-written ids are refused at load. **The user approved revision 7. Revision 8 adds clause 12, which is the ADR's own reading, and the user approved it ("looks good, approved", 2026-09-25 00:17:10Z).** Three follow-ups from the same -review stay as tickets, not ADR text: improvise-le4p, improvise-m56i and -improvise-p003. +review were left as tickets rather than ADR text: improvise-le4p, improvise-m56i and +improvise-p003. Revision 9 folds all three in, and improvise-m56i's two +questions are answers 30 and 31. + +**Revision 9 needs the user's approval, and it asks one question.** + +**Q23. How does a formula line spell its target?** Answer 30 makes a target +outside `_Measure` a real item, and says a formula line names it "by the item's +id". A `_Measure` target has no item and so no item id. Clause 5 above therefore +reads the line as `- () = …`, where `` is an item id +outside `_Measure` and a display inside it. That puts two different kinds of +thing in one position, and the category after `[` is what tells them apart. +- **(a) Recommended: as clause 5 reads it.** No extra syntax. The one asymmetry + is that a `_Measure` target's display appears in the file bare, while every + other name is an id. +- **(b) Always a display, and the item id in a second tag:** + `- Tax(tax-1)[tax] = …`. The two positions never mix kinds, but a formula line + outside `_Measure` gains a third bracketed token. +- **(c) Always the target's id**, giving a `_Measure` target an id of its own + that is distinct from the formula's. This reverses clause 1's "its handle is its + formula's `FormulaId`". + +Three places in revision 9 state the ADR's own reading rather than a user +answer. Each is flagged where it appears: +- the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); +- "space" means U+0020; +- a slug of only `-` is a legal id (both in clause 4). ## Consequences @@ -907,8 +1063,9 @@ improvise-p003. save. - `Rejection` loses `Copy`. - `Formula::raw` is no longer verbatim: the file carries formula text over ids. -- The formula panel is a second rendering of formula text, over display names, so the - host must resolve display names typed there back to ids (answer 19). +- The formula panel is a second rendering of formula text, over display names, so + the **client** must resolve display names typed there back to ids (answers 19, + 31 and 32). The host sees only id-form text. - Ids are lower case, so an old file's data keys change spelling on its first save. - ADR 002's prose needs a wording update when this lands. From 4973845fda79f818da7a4ef6266b7760c89cc6f2 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 12:03:43 -0700 Subject: [PATCH 013/106] docs(adr): ADR 009 states =-visibility on the terms, not the total; sequences come from the store (improvise-felo.4) - A miss against a typed total is bounded by the terms' rounding, about half an ulp of each term, so it is last-bit relative to the largest term. It can reach = only when the terms reach about 2^19. The large-term, small-total case (1e15 + 0.10 - 1e15 reads 0.125) is added. The epsilon-free = shows every miss. - 5a: cell sequences are assigned at DataStore's mutation sites and carried in each delta element, so a remove-then-set of one key within an effect moves it to the end, as IndexMap does. The generator includes that case. The record tier projects to (key, value). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 62 ++++++++++++++++------- 1 file changed, 44 insertions(+), 18 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index d60aec48..b4e6ccdd 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -478,10 +478,19 @@ incremental form must compute the same thing. (`member_records`), and `shift_remove` changes that when a record's leading cell goes. So the stage keeps a **per-cell** sequence, assigned when a key is first inserted: - - a same-key retract and insert within one step is an in-place overwrite, so the - cell keeps its sequence, as `IndexMap::insert` does; - - a key inserted again after a removal in an earlier step gets a fresh sequence, at - the end, as `IndexMap` puts it; + - **the sequence is assigned where `IndexMap` assigns position, at `DataStore`'s + own mutation sites**, and each cell-delta element carries it. An overwrite + (`set` on a live key) keeps the key's sequence, as `IndexMap::insert` does. A + `remove` followed by a `set` of the same key moves the key to the end, even + within one effect, and gets a fresh sequence. Because the element carries the + sequence, the two cases stay distinct after consolidation: an overwrite is + `−(k, v₀, s) + (k, v₁, s)`, and a remove-then-set is + `−(k, v₀, s) + (k, v₁, s′)`. A stage that inferred the sequence from a + consolidated `(key, value)` delta could not tell them apart. The differential + test's generator includes a same-key remove-then-set within one effect; + - the record tier (clause 3) projects the delta to `(key, value)` before it + consolidates, so moving a cell without changing its value still bumps nothing, + as today's fingerprint does; - losing a record's leading cell changes its minimum, so the record is re-emitted with its new order key. @@ -802,15 +811,16 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | yes below about 5e5, no above: it sums the stored, already-rounded f64s exactly, so it can miss the typed total in the last bit, and `=`'s absolute 1e-10 absorbs that only while the total is below about 2¹⁹ (see below) | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | | cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | **The accumulator does not make `=` against a typed decimal right.** It sums the *stored* f64s exactly, and each stored f64 is already the rounded form of what the user -typed. So the exact sum of the stored values, rounded once, can differ in the last bit -from the typed total, rounded once. +typed. So the exact sum of the stored values, rounded once, can differ from the typed +total, rounded once, by up to about half an ulp of each term. That is the last bit of +the largest term, which is more than the last bit of the total when the terms cancel. - **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result @@ -827,18 +837,34 @@ grows, and its last-bit step grows with it, so the terms' own rounding errors fa below it and misses become rare. When the terms cancel, the total stays small, and misses stay near 40%. -**Where a miss is visible.** The misses are last-bit, so they show in the editor and -to a raw comparison, not in a display format. **They are visible to `=` only once the -total is above about 2¹⁹ ≈ 5.2e5**, because `=` is an absolute `1e-10` (gotcha 17), -and one ulp reaches it only at that magnitude. The same holds if ADR 005's epsilon-free -`=` is adopted (its question 3), since that has no tolerance at all. So: -- on the ±1,000 values, where misses are most frequent, the review counted 0 of 7,506, - 0 of 7,901 and 0 of 7,963 misses visible to `=` at 2, 10 and 100 terms; -- on positive values up to 999,999.99, most misses are visible, because the totals are - large. +**Where a miss is visible.** A miss is bounded by the **terms'** rounding, not by the +total's. Each stored term is off from its typed value by at most half an ulp of that +term, so the exact sum of the stored terms is off from the typed total by at most about +½·Σ ulp(term), plus the total's own half-ulp at the final rounding. So the misses are +last-bit **relative to the largest term**, not relative to the total. When terms cancel, +a miss can be far larger than the total's ulp. -ADR 005 states the same condition. Only exact decimal cells (ADR 005) make `=` against -a typed total exact at every magnitude. +- **The large-term, small-total case (from ChatGPT Codex on this PR).** The terms are + `1000000000000000.10 + (-1000000000000000.00)`, and the typed total is 0.10. The + first term is stored as `1e15 + 0.125`, because the ulp at 1e15 is 0.125. So the + exact sum of the stored terms is **0.125**. The miss is 0.025, on a total of 0.1, and + `=` sees it. +- **The condition, stated on the terms.** `=` is an absolute `1e-10` (gotcha 17), so a + miss can reach it only when ½·Σ ulp(term) reaches `1e-10`. That needs at least one + term of about 2¹⁹ ≈ 5.2e5 or larger, or many terms just below it. It does not depend + on how large the total is. +- **The measured cases fit the condition.** On the ±1,000 values every term is far + below 2¹⁹, and the review counted 0 of 7,506, 0 of 7,901 and 0 of 7,963 misses + visible to `=` at 2, 10 and 100 terms. On positive values up to 999,999.99 the terms + themselves pass 2¹⁹, and most misses are visible. +- **Under ADR 005's epsilon-free `=`** (its question 3), every miss is visible, since + there is no tolerance at all. + +The misses show in the editor and to a raw comparison, and not in a display format +unless the terms are large enough, as in the case above. ADR 005's "once the magnitude +is above 2¹⁹" needs the same correction, to the terms' magnitude, and its author has +been told. Only exact decimal cells (ADR 005) make `=` against a typed total exact at +every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this From 8d520d038815f731f8f579ade3d534348e66d0a0 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 12:12:00 -0700 Subject: [PATCH 014/106] docs(adr): ExactSum lives in numtower, as the user decided on ADR 005 question 15 (improvise-felo.4) The user answered (a) on PR #320. ADR 009's seam crate wraps ExactSum for zset's trait, so zset stays free of numbers. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b4e6ccdd..fe3efbd4 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -729,7 +729,7 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `1.82e-304`. The example now checks that case, and it reads the exact answer. The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight -near `i64::MAX`. The version in `zset` (or `numtower`) splits the weight into base-2³⁰ +near `i64::MAX`. The version in `numtower` splits the weight into base-2³⁰ digits instead and adds each as a shifted value. That is exact in the fixed-point representation and O(1) per add. In practice a weight is bounded by a record count anyway. Then: @@ -781,11 +781,11 @@ anyway. Then: - Its accumulator is not `Clone`, and reading it takes `&mut self`. `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not -evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. **Where it lives is ADR 005 -question 15's decision** (recommended there: a separate `numtower` crate, wrapped by -this ADR's seam crate so that `zset` stays free of numbers). This ADR defers to that -answer. +evaluated. So the accumulator is about 150 lines of our own, with its own tests and a +differential test against an exact rational sum. **It lives in `numtower`, ADR 005's +numeric crate.** The user decided that on ADR 005 question 15, answering "(a)" on PR +#320 on 2026-09-25. This ADR's seam crate wraps it in a newtype for `zset`'s group +trait, so `zset` stays free of numbers. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -921,8 +921,8 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 11. Constraints -- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset` and - `improvise-core`. If improvise-command selects the backend, both join the browser +- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, + `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, both join the browser bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). - **Two seconds per test binary.** The crate's laws and its "incremental equals @@ -1086,9 +1086,9 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c). Where `ExactSum` lives is ADR 005 question 15's decision - (recommended there: `numtower`, wrapped by the seam crate), and this ADR defers to - it. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + *Recommendation:* (c). `ExactSum` lives in `numtower`, wrapped by the seam crate, + as the user decided on ADR 005 question 15. It is measured bit-equal after a million + retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and typed values, and it composes with (c): under ADR 005, a decimal sum is exact From f6e5145aa5b0d495b454038505441adf9eaf27f4 Mon Sep 17 00:00:00 2001 From: "Claude (agent-adr007)" Date: Fri, 25 Sep 2026 12:14:14 -0700 Subject: [PATCH 015/106] docs(adr): a CellKey coordinate is (CategoryId, ItemHandle) (improvise-cq1q.1) Revision 9 put a FormulaId in a CellKey's item position while clause 2 typed that position ItemId, a contradiction Codex found on PR #326. Clause 2, the formula-cell key in clause 5 and clause 8's RecordHash now use clause 6's own ItemHandle. Answer 22 keeps the id text unambiguous. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 694fa401..a2e385e9 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -450,9 +450,16 @@ Following answers 1 to 3: - **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id, and each `Category`, `Item` and `View` gains a `display: String`. - **Cells and derived surrogates:** a `CellKey` is a sorted list of - `(CategoryId, ItemId)` pairs. It stays sorted, now **by category id**, so gotcha 5's - canonical form is kept with a different sort key. `DataStore` interns ids exactly as it - interns names today. + `(CategoryId, ItemHandle)` pairs, where `ItemHandle` is clause 6's + `{Item(ItemId), FormulaTarget(FormulaId)}`. Outside `_Measure` every coordinate is + an `Item`. A `_Measure` formula cell's coordinate is `FormulaTarget`, because + that target has no item. The key stays sorted, now **by category id**, so gotcha + 5's canonical form is kept with a different sort key. `DataStore` interns ids + exactly as it interns names today. Answer 22 makes every id unique across kinds, + so an id's text alone names one thing; the tag is there so that the types + never claim an `ItemId` the document does not hold. Revision 9's first draft + wrote this pair as `(CategoryId, ItemId)` while also putting a `FormulaId` in it + (Codex on PR #326). - **View configuration:** the axis maps, page selections, and hidden and collapsed sets are keyed by ids. The one exception is a collapsed entry, which names a group: groups have no id (answer 9), so it is `(CategoryId, group name)`, and @@ -650,9 +657,9 @@ round-trips. - **The expression is over ids**, spelled through `quote::quote_name` as above. - **A `WHERE`/`IS` filter value is an item id**, bare or quoted (answer 3: every reference is an id). The formula panel shows it as a display. -- **A formula cell's `CellKey`** carries the target item's `ItemId` outside - `_Measure`. For a `_Measure` target, it carries the `FormulaId` text under - `_measure`, which answer 22 makes unique. +- **A formula cell's `CellKey`** carries `ItemHandle::Item()` + outside `_Measure`. For a `_Measure` target it carries + `ItemHandle::FormulaTarget()` under `_measure` (clause 2). **`header_display` is guarded only by a lookahead**, which the generator skips (gotcha 34). So the generator may emit a display ending in something that later parses as an id. @@ -773,7 +780,11 @@ Per-entity versions are not added (answer 10). ### 8. ADR 002: `RecordHash` is now defined over ids Following answer 3, `RecordHash` hashes the explicit serialization of the sorted -`(CategoryId, ItemId)` text pairs, the same function over different strings. +`(CategoryId, ItemHandle)` pairs, spelled as their id text, the same function +over different strings. The text is enough: answer 22 makes ids unique across +kinds, so no `Item` and `FormulaTarget` can spell alike, and the hash stays +canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate +is never part of a record in the first place. - **ADR 002's decision is unchanged in substance.** It still hashes an explicit serialization of a normalized record, with a fixed key and a width argued for. - **Its prose will be out of date once this lands.** ADR 002 says `CellKey::new` From d766605b304e293a7177644b33c1208417895472 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-fix)" Date: Fri, 2 Oct 2026 15:51:27 -0700 Subject: [PATCH 016/106] docs(adr): the cell delta carries the store sequence; two stale statements fixed (improvise-felo.4) Clause 3 defines the canonical element with Seq, derives the (key, value) projection for the record tier, and states bit identity over that part only. Clause 11 names all three crates; the 2^19 note is past tense. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++-------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index fe3efbd4..711c4e36 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -225,21 +225,32 @@ and compares every drawn cell, every column total and every occupancy count. ### 3. The input: a cell write is a retraction plus an insertion -**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue)>` with weights -±1.** `DataStore::set` and `DataStore::remove` (`cell.rs:286`, `:325`) are the only -places a cell changes, so the delta is emitted there: +**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue, Seq)>` with +weights ±1, where `Seq` is the cell's store sequence (clause 5a).** `DataStore::set` and +`DataStore::remove` (`cell.rs:286`, `:325`) are the only places a cell changes, so the +delta is emitted there, each element carrying the sequence `s` of the cell it names: -- a new cell is `+(k, v)`; -- an overwrite of `v₀` with `v₁` is `−(k, v₀) + (k, v₁)`; -- a removal is `−(k, v₀)`. +- a new cell is `+(k, v, s)`, with a fresh `s`; +- an overwrite of `v₀` with `v₁` is `−(k, v₀, s) + (k, v₁, s)`, the key keeping its `s`; +- a removal is `−(k, v₀, s)`; +- a remove-then-set within one effect is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. -An overwrite with an equal value consolidates to the empty Z-set. The circuit steps once +**The `(key, value)` projection is derived, not a second stream.** The record tier +and any consumer that does not need order read the delta mapped through +`(k, v, s) ↦ (k, v)` and consolidated, a `ZSet<(CellKey, CellValue)>`. Only clause 5a's +stage reads `s`. Defining the canonical element without `s` would strip it, and a +same-value remove-then-set would cancel, defeating 5a. + +An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty +Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). **A delta key's identity is bit identity, the same identity the record tier uses -today.** Two `(CellKey, CellValue)` pairs are one element exactly when their keys are -equal and their values are equal *by bits*: `Number` by `f64::to_bits`, `Text` and +today.** Two elements have the same `(key, value)` part exactly when their keys are +equal and their values are equal *by bits*, and bit identity is stated over that part +only (the sequence is compared as the integer it is, which keeps an +equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and `Error` by their strings. That is `hash_value` (`app.rs:233-250`), which `record_fingerprints` already hashes. So the cell delta reproduces today's tier exactly: @@ -862,8 +873,8 @@ a miss can be far larger than the total's ulp. The misses show in the editor and to a raw comparison, and not in a display format unless the terms are large enough, as in the case above. ADR 005's "once the magnitude -is above 2¹⁹" needs the same correction, to the terms' magnitude, and its author has -been told. Only exact decimal cells (ADR 005) make `=` against a typed total exact at +is above 2¹⁹" needed the same correction, to the terms' magnitude, and ADR 005 has +made it. Only exact decimal cells (ADR 005) make `=` against a typed total exact at every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. @@ -922,7 +933,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 11. Constraints - **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, - `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, both join the browser + `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, all three join the browser bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). - **Two seconds per test binary.** The crate's laws and its "incremental equals From 24ee72aff979384fe9eb5d2f5304153cbef2d63b Mon Sep 17 00:00:00 2001 From: "Claude (agent-adr007)" Date: Fri, 2 Oct 2026 15:51:58 -0700 Subject: [PATCH 017/106] docs(adr): a format that reads as an axis line is pipe-quoted, and flag every reading (improvise-cq1q.1) The le4p fix said no number format is spelled as an axis keyword. That is false: set-format stores its argument verbatim and the writer emits `format: `, so `:set-format none` writes `format: none`, which the axis-first order would read as the axis line of a category `format`. Clause 5 now states one rule, as the ADR's own reading: the writer pipe-quotes a format that would also match axis_line, and the new arm's format_line unquotes it. Also: - answer 32 quotes the recorded comment whole, and the section says what its quotations are and why authorship in the tracker does not settle them; - the readings that are not user answers are flagged in place and listed under Q23; - clause 11 step 2 lists ItemHandle; - Q23 states how an expression refers to a formula target and that the target position's pipe-quoting covers a `_Measure` display that looks like `Name(x)`. Nothing is renumbered, and no earlier decision changes. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 120 +++++++++++++++++++-------- 1 file changed, 86 insertions(+), 34 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index a2e385e9..7b88e513 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -329,8 +329,14 @@ data key and reference in that file spells it. Clause 4 says what follows: ### Seventh round: answers on the implementation tickets After the ADR was accepted, planning its implementation (epic improvise-cq1q) -raised three questions the ADR does not answer. The user answered them on -2026-09-25. The answers are recorded on the tickets and quoted here verbatim. +raised three questions the ADR does not answer. They were answered on +2026-09-25, and the answers are recorded as comments on the tickets +(improvise-m56i and improvise-hexb). The tracker shows the repository owner's +account as the author of those comments, and agents write to it under the same +account, so authorship there does not show whose words they are. The user's +"Approved" on this PR is what confirms them. Answers 30 and 31 quote the option +chosen, as recorded, and the sentences after each are the ADR's own statement of +what it means. Answer 32 was recorded as one sentence and is quoted whole. **30. A formula target outside `_Measure`** (improvise-m56i item 2): @@ -350,10 +356,14 @@ target the `_Measure` treatment (no real item, handled by its `FormulaId`). A typed name is matched against displays first, then against ids. A display that matches more than one thing is refused, and the refusal lists the candidates' ids. -**32. Which side holds clause 12's capture** (improvise-hexb item 2): the -**client**, as it holds `OpenedCell`. It resolves typed names against its -id-bearing projection, the wire carries only id-form text, and the host never -resolves a display name. Clause 12 is rewritten to match. +**32. Which side holds clause 12's capture** (improvise-hexb item 2), as recorded: + +> "USER DECISION 2026-09-25 (item 2): the CLIENT holds clause 12's display->id +> capture (like OpenedCell) and resolves names against its id-bearing +> projection; the wire carries only id-form text; the host never resolves a +> display name." + +Clause 12 is rewritten to match. ## Context @@ -432,11 +442,12 @@ and its formula has a separate `FormulaId`. Generated ids follow clause 4, so a formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-`. The panel lists that row as `ItemHandle::Item`, like any other item. -**The virtual views have reserved ids too:** `_records` and `_drill`, with the -displays `_Records` and `_Drill`. They are never persisted, but they are -addressed on the wire (`SwitchView`, improvise-obtu). Answer 23 reserves a -leading `_` for virtual things, and a slugified `_Drill` would otherwise be -`drill`, which could collide with a user's view (improvise-p003 item 1). +**The virtual views have reserved ids too** (the ADR's own reading, not a user +answer): `_records` and `_drill`, with the displays `_Records` and `_Drill`. They +are never persisted, but they are addressed on the wire (`SwitchView`, +improvise-obtu). Answer 23 reserves a leading `_` for the virtual *categories*, +and this extends it to the virtual views, because a slugified `_Drill` would +otherwise be `drill`, which could collide with a user's view (improvise-p003 item 1). **The ADR's reading, not a user answer: `_Index` and `_Dim` coordinates are outside the id namespace.** They are synthetic record positions (`_Dim` carries a @@ -450,7 +461,8 @@ Following answers 1 to 3: - **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id, and each `Category`, `Item` and `View` gains a `display: String`. - **Cells and derived surrogates:** a `CellKey` is a sorted list of - `(CategoryId, ItemHandle)` pairs, where `ItemHandle` is clause 6's + `(CategoryId, ItemHandle)` pairs (the ADR's own reading of answer 30, not a + user answer), where `ItemHandle` is clause 6's `{Item(ItemId), FormulaTarget(FormulaId)}`. Outside `_Measure` every coordinate is an `Item`. A `_Measure` formula cell's coordinate is `FormulaTarget`, because that target has no item. The key stays sorted, now **by category id**, so gotcha @@ -463,8 +475,8 @@ Following answers 1 to 3: - **View configuration:** the axis maps, page selections, and hidden and collapsed sets are keyed by ids. The one exception is a collapsed entry, which names a group: groups have no id (answer 9), so it is `(CategoryId, group name)`, and - `collapsed:` writes the category id, then the group name (improvise-p003 - item 4). + `collapsed:` writes the category id, then the group name (the ADR's own + reading, not a user answer; improvise-p003 item 4). - **Formulas:** a formula's references are ids (clause 5). The formula panel renders them as display names (answer 19; clause 6). @@ -635,16 +647,36 @@ id_ref = { id_token } and takes an `id_ref`. A group reference (`[Coastal]`) stays a name while groups have no id (answer 9). -**An axis line is tried before `format:`** (improvise-le4p). `view_entry` is an -ordered choice, and `format_line` takes the rest of the line, so a category whose -id is `format` would lose its axis line to the view's number format. In the -new-version arm, `axis_line`, which requires an axis keyword after `: `, comes -before `format_line`. No number format is spelled as an axis keyword, so neither -rule loses a line. The id `format` needs no quoting, and `id_ref` keeps no quoting -arm. The required test is a category displayed `Format`, on a row axis, that -round-trips. +**An axis line is tried before `format:`, and a format that reads as an axis line is +pipe-quoted** (improvise-le4p; the ADR's own reading, not a user answer). +`view_entry` is an ordered choice, and `format_line` takes the rest of the line, +so a category whose id is `format` would lose its axis line to the view's number +format. In the new-version arm, `axis_line` therefore comes before `format_line`. +That moves the collision to the other side. `set-format` stores its argument +verbatim and the writer emits `format: {number_format}`, so `:set-format none` +writes `format: none`, which would now read as the axis line of a category +`format`: the format is lost, or the id is unknown and the load is refused +(answer 11), so a document that saves would not load. -**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). The shape is +The rule is that **the writer pipe-quotes a number format whenever the line would +also match `axis_line`**, that is, when the value is an axis keyword (`row`, +`column`, `page`, `filter` or `none`) alone or followed by `, `, or when it +begins with `|`. The new arm's `format_line` takes +`pipe_quoted ~ &NEWLINE | rest_of_line`, as `header_name` does (gotcha 34). So +`format: |none|` is the format `none`, and nothing is refused. An old-version +file's `format: none` is read as before, because the old arm keeps `format_line` +first, and it is written quoted on its first save. The alternatives were to refuse +such a format in `set-format`, which does nothing for a file that already holds +one, or in the writer, which leaves a document that cannot be saved. The id +`format` needs no quoting, and `id_ref` keeps no quoting arm. + +Required tests: a category displayed `Format`, on a row axis, round-trips; and a +view whose number format is each axis keyword round-trips, in a document where +the same view also holds a category `format` on an axis. + +**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). The shape is the ADR's +own reading, not a user answer, except that the formula id's place is answer 5's and +an item target's id is answer 30's. It is `- () = []`: - **The parenthesised id is the formula's own `FormulaId`** (answer 5's placement). @@ -652,8 +684,9 @@ round-trips. `_Measure`, answer 30). For a `_Measure` target, which has no item, it is the target's display. - `Formula` carries that display, and a display rename of a `_Measure` target - writes it. - - Whether one line should spell both cases this way is **Q23**, below. + writes it. Where that display lives is also the ADR's own reading. + - Whether one line should spell both cases this way is **Q23**, below, and so + are how an expression refers to a target and how a `_Measure` display is quoted. - **The expression is over ids**, spelled through `quote::quote_name` as above. - **A `WHERE`/`IS` filter value is an item id**, bare or quoted (answer 3: every reference is an id). The formula panel shows it as a display. @@ -757,7 +790,8 @@ pipe-quotes any display name that would otherwise read as display plus id: - **A `_Measure` formula-target row** (`ItemHandle::FormulaTarget(FormulaId)`): removing it means `RemoveFormula`, and renaming it means a display rename of the formula's target (improvise-i0n7). -- **CheckedBatch's created-record exemption goes** (improvise-p003 item 5). It +- **CheckedBatch's created-record exemption goes** (the ADR's own reading, not a + user answer; improvise-p003 item 5). It exempts a record that neither resolves nor is claimed, for frames like `[AddItem(East), SetCell(East)]`. Under this clause a frame cannot address what it creates, so nothing produces such a frame any more. The wire step removes @@ -857,8 +891,9 @@ Following answers 15 and 25: ### 11. Migration 1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224. -2. **Types:** the id newtypes over `Symbol`, the `display` fields, and `FormulaId` and - `ViewId`. +2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId`, + `ViewId`, and `ItemHandle`, which the re-key (step 3) needs for `CellKey` before + clause 6's step (5) uses it in projections. 3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with display→id resolution moved to every input edge. 4. **Persistence:** the new version literal (clause 10), the grammar in clause 5, the @@ -1045,17 +1080,34 @@ thing in one position, and the category after `[` is what tells them apart. is that a `_Measure` target's display appears in the file bare, while every other name is an id. - **(b) Always a display, and the item id in a second tag:** - `- Tax(tax-1)[tax] = …`. The two positions never mix kinds, but a formula line - outside `_Measure` gains a third bracketed token. + `- Tax(tax-1)(tax) [customcat] = …`. The two positions never mix kinds, but a + formula line outside `_Measure` gains a second parenthesised tag. - **(c) Always the target's id**, giving a `_Measure` target an id of its own that is distinct from the formula's. This reverses clause 1's "its handle is its formula's `FormulaId`". -Three places in revision 9 state the ADR's own reading rather than a user -answer. Each is flagged where it appears: +Two consequences of (a) that the answer should take in, both the ADR's own reading: +- **References to a formula target.** An expression, and a `WHERE`/`IS` value, + names a target outside `_Measure` by its **item id**, and a `_Measure` target by + its **formula id**. That follows from `ItemHandle` (clause 2). +- **Quoting the target position.** A `_Measure` display in the target position can + itself look like `Name(x)`, which would read back as a display and a formula id. + The writer's rule that pipe-quotes such a display therefore covers a + formula-line target too: `- |Margin(%)|(margin) = …`. An item id never contains + a parenthesis, so only a `_Measure` display can need it. + +Revision 9 states these readings of its own, which are not user answers. Each is +flagged where it appears, and approving the PR approves them: +- the virtual views' reserved ids `_records` and `_drill` (clause 1); - the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); -- "space" means U+0020; -- a slug of only `-` is a legal id (both in clause 4). +- a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2); +- `collapsed:` is `(CategoryId, group name)` (clause 2); +- "space" means U+0020, and a slug of only `-` is a legal id (clause 4); +- the axis-before-`format:` order and the quoting of a format that reads as an axis + line (clause 5); +- the formula-line shape, and a `_Measure` target's display living on the `Formula` + (clause 5); +- CheckedBatch's created-record exemption is removed (clause 6). ## Consequences From f786f835aca659d61d4439c55a1c6d767c8f1c65 Mon Sep 17 00:00:00 2001 From: "Claude (agent-adr007)" Date: Fri, 2 Oct 2026 16:24:05 -0700 Subject: [PATCH 018/106] docs(adr): one rule for a reference to a formula target, and the right count of readings (improvise-cq1q.1) Copilot found two inconsistencies in the previous push: - clause 5 said a WHERE/IS value is an item id while Q23 said it could be a formula id. Clause 5 now states one rule, the id text of an ItemHandle, for an expression and a filter value alike, and Q23 points to it instead of restating it; - the status header still said three readings were flagged, where Q23 lists eight. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 7b88e513..b6ca9e81 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -8,8 +8,8 @@ user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17: **Revision 9 is a proposed amendment, not yet approved.** It folds in the user's answers on the implementation tickets (answers 30–32) and clarifies text the reviews found unclear or stale (improvise-cq1q.1). It changes no earlier decision. -It asks one new question, Q23 (a formula line's spelling), and flags three places -where it states the ADR's own reading. · +It asks one new question, Q23 (a formula line's spelling), and flags eight readings +of its own that are not user answers, listed under Q23. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -687,9 +687,13 @@ an item target's id is answer 30's. It is writes it. Where that display lives is also the ADR's own reading. - Whether one line should spell both cases this way is **Q23**, below, and so are how an expression refers to a target and how a `_Measure` display is quoted. -- **The expression is over ids**, spelled through `quote::quote_name` as above. -- **A `WHERE`/`IS` filter value is an item id**, bare or quoted (answer 3: every - reference is an id). The formula panel shows it as a display. +- **The expression is over ids**, spelled through `quote::quote_name` as above. A + reference to a formula target is the id text of its `ItemHandle`: the item id + outside `_Measure`, and the formula id for a `_Measure` target. +- **A `WHERE`/`IS` filter value follows the same rule** (answer 3: every reference + is an id). It is the id text of an `ItemHandle` in the filtered category, bare or + quoted: an item id, or a formula id when the category is `_Measure` and the value + names a formula target. The formula panel shows it as a display. - **A formula cell's `CellKey`** carries `ItemHandle::Item()` outside `_Measure`. For a `_Measure` target it carries `ItemHandle::FormulaTarget()` under `_measure` (clause 2). @@ -1087,9 +1091,10 @@ thing in one position, and the category after `[` is what tells them apart. formula's `FormulaId`". Two consequences of (a) that the answer should take in, both the ADR's own reading: -- **References to a formula target.** An expression, and a `WHERE`/`IS` value, - names a target outside `_Measure` by its **item id**, and a `_Measure` target by - its **formula id**. That follows from `ItemHandle` (clause 2). +- **References to a formula target** are as clause 5 states them, for an expression + and a filter value alike: the id text of an `ItemHandle`, which is the item id + outside `_Measure` and the formula id for a `_Measure` target. That holds under + (a) and (b). Under (c) a `_Measure` target would be named by its own id instead. - **Quoting the target position.** A `_Measure` display in the target position can itself look like `Name(x)`, which would read back as a display and a formula id. The writer's rule that pipe-quotes such a display therefore covers a From 7878decbaa3ccdf00eb5af3869597a4ac13d0166 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-fix2)" Date: Fri, 2 Oct 2026 17:12:57 -0700 Subject: [PATCH 019/106] docs(adr): dirty flag reads the projected cell delta (improvise-felo.4) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 711c4e36..f76ff26c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -286,7 +286,8 @@ is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a non-empty cell delta means dirty" would never save a formula edit in the browser. The -content half becomes: **a non-empty cell delta, or a moved formula-set fingerprint**. +content half becomes: **a non-empty *projected* cell delta (the `(k, v)` projection, so a same-value +remove-then-set raises nothing), or a moved formula-set fingerprint**. The fingerprint is the one clause 7 already reads after each model effect, so nothing new is observed. Its cost is O(formulas), where `document_content` is O(cells) per `CellData` effect. @@ -1016,7 +1017,7 @@ displayed path until the differential test holds. existing suite is the evidence. `CellKey` gains `Ord`. 3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half - becomes "a non-empty cell delta, or a moved formula-set fingerprint" (clause 3), with + becomes "a non-empty projected cell delta, or a moved formula-set fingerprint" (clause 3), with a test that a formula edit alone still marks the document dirty. That removes the O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with From 4aed8c071f8aab2bf4e832206592b4ade71d57e0 Mon Sep 17 00:00:00 2001 From: "Claude (agent-adr007)" Date: Fri, 2 Oct 2026 17:26:42 -0700 Subject: [PATCH 020/106] docs(adr): put option (b)'s category tag after the expression, and count places (improvise-cq1q.1) Option (b)'s example had `[customcat]` before the `=`, while clause 5's formula-line shape puts the category tag after the expression. It was the only copy of the slip. The status header and the list under Q23 now say eight places, since three of the eight bullets each hold two readings. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index b6ca9e81..d1151945 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -8,8 +8,8 @@ user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17: **Revision 9 is a proposed amendment, not yet approved.** It folds in the user's answers on the implementation tickets (answers 30–32) and clarifies text the reviews found unclear or stale (improvise-cq1q.1). It changes no earlier decision. -It asks one new question, Q23 (a formula line's spelling), and flags eight readings -of its own that are not user answers, listed under Q23. · +It asks one new question, Q23 (a formula line's spelling), and flags eight places +where it states a reading of its own that is not a user answer, listed under Q23. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -1084,7 +1084,7 @@ thing in one position, and the category after `[` is what tells them apart. is that a `_Measure` target's display appears in the file bare, while every other name is an id. - **(b) Always a display, and the item id in a second tag:** - `- Tax(tax-1)(tax) [customcat] = …`. The two positions never mix kinds, but a + `- Tax(tax-1)(tax) = … [customcat]`. The two positions never mix kinds, but a formula line outside `_Measure` gains a second parenthesised tag. - **(c) Always the target's id**, giving a `_Measure` target an id of its own that is distinct from the formula's. This reverses clause 1's "its handle is its @@ -1101,8 +1101,9 @@ Two consequences of (a) that the answer should take in, both the ADR's own readi formula-line target too: `- |Margin(%)|(margin) = …`. An item id never contains a parenthesis, so only a `_Measure` display can need it. -Revision 9 states these readings of its own, which are not user answers. Each is -flagged where it appears, and approving the PR approves them: +Revision 9 states a reading of its own, which is not a user answer, in each of these +eight places (three of them hold two readings). Each is flagged where it appears, and +approving the PR approves them: - the virtual views' reserved ids `_records` and `_drill` (clause 1); - the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); - a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2); From e9659903d80d13bb2e85e873c9d011949d652ede Mon Sep 17 00:00:00 2001 From: "Claude (adr7-steward)" Date: Mon, 5 Oct 2026 13:28:07 -0700 Subject: [PATCH 021/106] docs(adr): fold answers 33 and 34 into ADR 007 (improvise-cq1q.1) Answer 33: typed names match ids first until a view-layer autocomplete exists, reversing answer 31's order. Answer 34: "space" means whitespace and a run of it becomes one underscore, overriding the ADR's U+0020 reading. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 162 ++++++++++++++++++++++----- 1 file changed, 132 insertions(+), 30 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index d1151945..f7c3522b 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -6,10 +6,13 @@ of when a formula edit's names become ids, and corrects stale facts about main. user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17:10Z: "looks good, approved"). **Revision 9 is a proposed amendment, not yet approved.** It folds in the user's -answers on the implementation tickets (answers 30–32) and clarifies text the -reviews found unclear or stale (improvise-cq1q.1). It changes no earlier decision. -It asks one new question, Q23 (a formula line's spelling), and flags eight places -where it states a reading of its own that is not a user answer, listed under Q23. · +answers on the implementation tickets (answers 30–32), the user's two inline +comments on its head `4aed8c07` and a clarification in chat (answers 33 and 34), +and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 +reverses the matching order that answer 31 recorded, and answer 34 overrides the +ADR's own reading of "space". It asks one new question, Q23 (a formula line's +spelling), and flags nine places where it states a reading of its own that is not +a user answer, listed under Q23. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -224,7 +227,8 @@ run of spaces collapse to one `_`?"): > all or always have IDs)" This decides three things: -- a run of spaces collapses to one `_`, as recommended; +- a run of spaces collapses to one `_`, as recommended (a run of whitespace, by + answer 34); - a duplicate slug gets a numeric suffix after a `-`; - **a file has ids on everything or on nothing.** A new-version file with a declaration missing its id is refused, like any file that cannot be resaved safely (answer 11). @@ -353,8 +357,12 @@ target the `_Measure` treatment (no real item, handled by its `FormulaId`). > "Display first; client holds it" -A typed name is matched against displays first, then against ids. A display that -matches more than one thing is refused, and the refusal lists the candidates' ids. +The option is quoted as recorded. **The order it names was reversed by the user's own +comment on this answer (answer 33): until autocomplete exists, a typed name is +matched against ids first, then against displays.** What stands from answer 31 is +that the client holds the resolution (answer 32), and that a display matching more +than one thing is refused, with the refusal listing the candidates' ids. Where +answers 31 and 33 differ, answer 33 governs. **32. Which side holds clause 12's capture** (improvise-hexb item 2), as recorded: @@ -365,6 +373,43 @@ matches more than one thing is refused, and the refusal lists the candidates' id Clause 12 is rewritten to match. +### Eighth round: answers on revision 9 + +The user answered revision 9 (`context/adr/007-stable-identities.md` at `4aed8c07`) +with two inline comments on PR #326, posted 2026-10-04 02:48Z and 02:49Z, and then +clarified the second one in chat (not on the PR, relayed to the steward). Each is +mapped to the line it is attached to, and quoted verbatim. + +**33. The order of answer 31.** On line 356, the sentence under answer 31, "A typed +name is matched against displays first, then against ids": + +> "This should be handled by separating the view layer autocomplete of typed display names from the model layer reference to IDs: but, before we have autocomplete, IDs are matched first." + +This decides two things: +- **The long-term design.** A view-layer autocomplete resolves typed display text to + ids. The model layer refers to ids only, and never matches a display. +- **The interim rule, until autocomplete exists.** A typed name is matched against + **ids first, then displays.** This reverses answer 31's "displays first". + +Clause 12 states the rule and what it means for a document where one thing's display +equals another's id. Nothing built yet implements either order (clause 11, step 7). + +**34. What "space" means in slugify.** On line 553, the bullet that read "Space" +as U+0020 only, so that a tab was stripped and `Time\tPeriod` slugged to +`timeperiod`: + +> "This shouldn’t matter because consecutive spaces should normalize to a single underscore" + +and, in chat: + +> "space should generally mean "whitespace"" + +This **overrides the ADR's own reading** (revision 9's, which was not a user answer). +"Space" in answers 18 and 21 means whitespace, and a run of whitespace becomes one +`_`. The user did not enumerate which characters count, so the definition in clause +4 is **the ADR's own reading of "whitespace"**, flagged there. Answers 18 and 21 +keep their wording, which this reads as "space, meaning whitespace". + ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. @@ -496,8 +541,9 @@ be renaming field access"), and for the field that is right. The call sites are cost. **Display names stay unique within their scope**, as names are today. So a human-typed -name resolves to exactly one id, and a rename to a display name already in use is -refused. An item's scope is its category, so two categories may each have an item +display resolves to at most one id within a scope, and a rename to a display name +already in use is refused. Typed text is matched against ids before displays +(answer 33; clause 12). An item's scope is its category, so two categories may each have an item displayed `Revenue`. Formula text crosses categories, which is where answer 19's "ambiguous" arises. Their ids differ (answer 22), so the panel resolves the ambiguity by showing the ids. @@ -527,8 +573,9 @@ space with `_`, strip"): `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped), so this needs a case-folding table or a small crate in `improvise-core`. It must pass the `wasm` gate, because that crate is in the browser bundle. -2. **Replace each run of spaces with one `_`** (answers 18 and 21; answer 18 changed - answer 8's `-`). +2. **Replace each run of whitespace with one `_`** (answers 18, 21 and 34; answer 18 + changed answer 8's `-`). A run is any unbroken sequence of whitespace + characters, whatever mix, so it becomes one `_`, never one per character. 3. **Strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`** (answers 16(c) and 16(d)). This strips non-ASCII letters, so an id never needs quoting in an `.improv` reference (clause 5 notes where formula text differs). @@ -547,16 +594,36 @@ Examples: `East` → `east`, `Time Period` → `time_period`, `Coastal_East` → 22), the second is `east-`. Here `` is the counter's next value, for example `east-1`. -**Two readings of the steps, the ADR's own and not user answers**, chosen by the -implementation (PR #318; `context/code/core.md` §20): -- **"Space" means U+0020.** Only a run of U+0020 becomes `_`. A tab, a no-break - space (U+00A0) and an ideographic space (U+3000) are not spaces here, so step 3 - strips them: `Time\tPeriod` → `timeperiod`, which collides with `TimePeriod`, - not with `Time Period`. A tab-separated import header is the realistic case. +**Two readings of the steps, the ADR's own and not user answers.** The first is +the ADR's reading of the word "whitespace", which answer 34 requires and does not +define. The second was chosen by the implementation (PR #318; `context/code/core.md` +§20): +- **"Whitespace" means the Unicode `White_Space` property**, the set Rust's + `char::is_whitespace` tests. That is U+0009–U+000D, U+0020, U+0085, U+00A0, + U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. It needs no + table in `improvise-core`. A tab, a no-break space (U+00A0) and an ideographic + space (U+3000) are whitespace, so each becomes `_` like a space does: + `Time\tPeriod`, `Time\u{A0}Period` and `Time Period` all slug to `time_period`, + and `Time \t Period`, one run of three characters, does too. A zero-width space + (U+200B) and a byte-order mark (U+FEFF) are **not** `White_Space`, so step 3 + strips them. The realistic case is a header pasted from a web page, where a + no-break space stands for a space. The ADR chose this set because it is the standard + definition and the user asked for "whitespace", not for a list. The user may + prefer a narrower one (ASCII whitespace only, say), and that is a one-line change. + An underscore the display already had is still never merged with the one a run + makes (`a_ b` → `a__b`), because the user said *consecutive spaces* collapse, not + consecutive underscores. - **A slug of only `-` is a legal bare id.** `-` and `---` slug to themselves. Step 5's "if nothing is left" means the empty string only. The id alphabet admits them, so a line such as `-: row` is legal but hard to read. +**What this changes in built code (a follow-up, not made here).** PR #318's +`slugify` (`crates/improvise-core/src/model/id.rs`, `underscore_space_runs`) tests +`c == ' '`, and `context/code/core.md` §20 states the U+0020-only reading as a +rule. So today `Time\tPeriod` slugs to `timeperiod`. Answer 34 makes that wrong, and +the implementation needs a ticket to test whitespace instead and to repair §20 and +its tests. Nothing calls `slugify` yet (§20), so no stored id is affected. + The remaining rules: - **A file has ids on everything or on nothing** (answer 21). An id-less file (old @@ -778,7 +845,10 @@ pipe-quotes any display name that would otherwise read as display plus id: - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows (answer 19): the target's display name with its id in parentheses, then the expression over display names, with an id wherever a display name is ambiguous - (ids are unique, answer 22). `stored` is the id-form text the file carries, and it is what + (ids are unique, answer 22). A display that equals another thing's id is + ambiguous in this sense (the ADR's own reading of answers 19 and 33, flagged + under Q23), so the panel never shows a bare name that the ids-first rule would + bind elsewhere. `stored` is the id-form text the file carries, and it is what `ReplaceFormula`'s `was` is compared against. So a rename of `Revenue` elsewhere changes what the panel shows but does not refuse an edit of `Profit`, which is the "avoid glitches" of answer 3. What makes that hold for an edit already open @@ -908,7 +978,9 @@ Following answers 15 and 25: `broken_chain`'s id arm. 7. **Clause 12's capture:** the client-side display→id mapping taken when the formula editor opens, and the resolver that turns typed names into id-form - text (answers 31 and 32). + text (answers 32 and 33). Neither the resolver nor any display-matching code + exists yet, so the interim order is built ids first from the start and nothing + has to be migrated. The autocomplete that retires it is not in this list. The implementation epic (improvise-cq1q) orders these steps to keep `main` shippable after each one, which the list above does not: the display field @@ -942,9 +1014,17 @@ whatever the document holds by then. including the target's. - A name the user types that was **not** in the captured mapping is new to the edit. The client resolves it against its own id-bearing projection (clause 6) at - commit, in answer 31's order: displays first, then ids. If a display matches more - than one thing, the edit is refused and the refusal lists the candidates' ids. If - the name does not resolve at all, the edit is refused and names it. + commit, in answer 33's order: **ids first, then displays**. A typed name equal to + an id names that id's thing, even where it is also another thing's display. + Otherwise, if exactly one thing in scope has that display, it names that thing. If + a display matches more than one thing, the edit is refused and the refusal lists + the candidates' ids. If the name does not resolve at all, the edit is refused and + names it. An id is matched exactly: no case folding. +- **The ids-first order is interim** (answer 33). The long-term design separates two + layers. A view-layer autocomplete of typed display names turns text into ids + before it is committed, and the model layer refers to ids only. When it exists, + this resolver's display stage and the ids-first order go with it. This ADR does not + specify the autocomplete. - **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are in the file's id form, and the host never resolves a display name (answer 32). The host parses ids, and one that no longer exists is @@ -952,6 +1032,16 @@ whatever the document holds by then. - `was` is still compared against the stored id-form text (clause 6), so a concurrent *replace* of the formula is still seen. +**What the interim order means for a document where an item's display equals another +item's id.** Item A is displayed `East` and has id `east`. Item B is displayed `east`, +so slugify finds `east` taken and gives it `east-1` (clause 4). Typing `east` names +**A**, because the id wins. B is reached by typing `east-1`, or by autocomplete once +it exists. A user who typed `east` meaning B is bound to A without being told, which +is the cost of the interim order and the reason the panel shows B as `east-1`, not as +a bare `east` (clause 6). The captured mapping cannot disagree with the rule, since +it holds only names the panel showed bare, and the panel shows none that ids-first +would bind elsewhere. + **Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. From another client, rename the item `Revenue` to `Sales` and rename a different item to `Revenue`. Then edit something that is **not a name**: change the text to @@ -960,6 +1050,13 @@ item's id, and the commit must not be refused. The test must fail under commit-time resolution, which binds to the new `Revenue`, and it must fail under a refusal on rename. Both are the rivals this clause rules out. +**Required test for the interim order** (answer 33). In a document with items A +(display `East`, id `east`) and B (display `east`, id `east-1`), type `east` into a +formula editor, in text not captured at open. The stored formula must reference A. +The test must fail under displays-first, which binds B. Typing `east-1` must bind B, +and typing a display that is nobody's id must still resolve by display. A fixture +where no display equals an id passes under either order and proves nothing. + Revision 8's version committed the *unedited* text. A client that treats an unchanged text as a no-op passes it without resolving any name, so the test could not tell a correct capture from no capture at all (improvise-hexb item 1). @@ -1029,7 +1126,8 @@ fdk1 landed in PR #221, before this ADR. What exists on main: The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14, revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All of those are answered.** After acceptance, three implementation questions were -answered on their tickets (answers 30–32). Revision 9 adds Q23, which is open. +answered on their tickets (answers 30–32), and the user's comments on revision 9 are +answers 33 and 34. Revision 9 adds Q23, which is open. | # | question | my recommendation | status | |---|---|---|---| @@ -1047,10 +1145,10 @@ answered on their tickets (answers 30–32). Revision 9 adds Q23, which is open. | 12 | Old files and the new header rule | (a) no | **answered: (a), with the version marked unambiguously** (answer 15) | | 13 | Slugify details | all generation; keep case; keep `_` `-`; strip non-ASCII; `-` | **answered** (answer 16): all generation, **lower case only** (reversing the recommendation), keep both, strip, `-` | | 14 | Formula text: ids or display names? | ids in the file | **answered**: ids in the file (answer 3); the panel is Q16 | -| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-` | +| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-`, and "space" means whitespace (answer 34) | | 16 | What the formula panel shows | none | **answered: display names, target shown as `Profit (profit)`, ids where a name is ambiguous** (answer 19) | | 17 | Capitals in a hand-written id | (a) no | **answered: no** (answer 20) | -| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse**, plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | +| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse** (a run of whitespace, answer 34), plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | | 18(b) | What "casefold" means | full case folding | **answered: full Unicode case folding** (answer 26) | | 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answer 22), stronger than (a) | | 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23) | @@ -1058,7 +1156,7 @@ answered on their tickets (answers 30–32). Revision 9 adds Q23, which is open. | — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal | | 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) | | — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation | -| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31) | +| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31). **Order reversed: ids first until there is autocomplete** (answer 33) | | — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32) | | 23 | A formula line's spelling (revision 9) | see "Still open" | **open** | @@ -1102,18 +1200,21 @@ Two consequences of (a) that the answer should take in, both the ADR's own readi a parenthesis, so only a `_Measure` display can need it. Revision 9 states a reading of its own, which is not a user answer, in each of these -eight places (three of them hold two readings). Each is flagged where it appears, and +nine places (three of them hold two readings). Each is flagged where it appears, and approving the PR approves them: - the virtual views' reserved ids `_records` and `_drill` (clause 1); - the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); - a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2); - `collapsed:` is `(CategoryId, group name)` (clause 2); -- "space" means U+0020, and a slug of only `-` is a legal id (clause 4); +- "whitespace" means the Unicode `White_Space` property (replacing the U+0020 reading, + by answer 34), and a slug of only `-` is a legal id (clause 4); - the axis-before-`format:` order and the quoting of a format that reads as an axis line (clause 5); - the formula-line shape, and a `_Measure` target's display living on the `Formula` (clause 5); -- CheckedBatch's created-record exemption is removed (clause 6). +- CheckedBatch's created-record exemption is removed (clause 6); +- a display equal to another thing's id counts as ambiguous in the formula panel + (clause 6). ## Consequences @@ -1134,7 +1235,8 @@ approving the PR approves them: - `Formula::raw` is no longer verbatim: the file carries formula text over ids. - The formula panel is a second rendering of formula text, over display names, so the **client** must resolve display names typed there back to ids (answers 19, - 31 and 32). The host sees only id-form text. + 32 and 33), ids first until an autocomplete exists. The host sees only id-form + text. - Ids are lower case, so an old file's data keys change spelling on its first save. - ADR 002's prose needs a wording update when this lands. From 5385bb36bc2d007080ca0425eb7e212be1e0d4fa Mon Sep 17 00:00:00 2001 From: "Claude (adr7-steward)" Date: Mon, 5 Oct 2026 13:31:14 -0700 Subject: [PATCH 022/106] docs(adr): fold answers 35 and 36, reopen Q23, add Q24 and Q25 (improvise-cq1q.1) Measures get the id/display split and are the normal formula target; dimension targets are rare. The mechanism is left to Q23 as options. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 162 ++++++++++++++++++++------- 1 file changed, 123 insertions(+), 39 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index f7c3522b..b6776588 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -8,11 +8,14 @@ user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17: **Revision 9 is a proposed amendment, not yet approved.** It folds in the user's answers on the implementation tickets (answers 30–32), the user's two inline comments on its head `4aed8c07` and a clarification in chat (answers 33 and 34), -and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 +its two inline comments on `e9659903` (answers 35 and 36), and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 reverses the matching order that answer 31 recorded, and answer 34 overrides the -ADR's own reading of "space". It asks one new question, Q23 (a formula line's -spelling), and flags nine places where it states a reading of its own that is not -a user answer, listed under Q23. · +ADR's own reading of "space". Answers 35 and 36 make measures the normal formula +target and give them an id/display split, which reopens Q23. It asks three new +questions, Q23 (how a formula-target measure carries its id, and how a formula line +spells its target), Q24 and Q25 (two details of slugify's whitespace rule), and +flags nine places where it states a reading of its own that is not a user answer, +listed under Q23. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -410,6 +413,42 @@ This **overrides the ADR's own reading** (revision 9's, which was not a user ans 4 is **the ADR's own reading of "whitespace"**, flagged there. Answers 18 and 21 keep their wording, which this reads as "space, meaning whitespace". +### Ninth round: answers on revision 9's head `e9659903` + +The user posted two inline comments on PR #326 at 2026-10-05 20:22:52Z and 20:23:40Z, +both on the line holding Q23's option (c). Each is quoted verbatim. + +**35. Measures get the id/display split too.** + +> "Measures should have ID/display split too. Measures are the normal case for formulas, anyways: something like `profit = revenue - cost`" + +**36. Dimensions are rarely formula targets.** + +> "Dimensions are mainly in formulas when you're calculating some kind of derived dimension, but usually dimensions should be relatively static and not interesting in formulas." + +These decide two things, and no more: +- **A measure has an id and a display**, as every other thing does. +- **The normal formula target is a measure** (`profit = revenue - cost`), and a + dimension (a non-`_Measure` category) target is the rare, derived case. + +They do **not** decide what a formula-target measure's id is, how its formula line +names it, or whether it is an item. That is **Q23**, reopened and reworded. Four +things in earlier text meet them, and none is resolved here: +- **Clause 1 and revision 9's flagged readings (clauses 1, 2, 5, 8)** say a `_Measure` + formula target "has no item of its own" and is handled by `FormulaTarget(FormulaId)`. + A stored measure (a non-formula `_Measure` item) is an ordinary item and already has + an id and a display. A measure that exists only because a formula targets it has + its display on the `Formula` and its id in the `FormulaId`. Whether that is the + split the user wants, or whether the target should be an item, is Q23. +- **Answer 3** ("every reference is an id") sits badly with revision 9's line for the + normal case, where the target position holds a bare display. +- **Answers 5 and 19** picture the measure case as a display plus one id + (`Profit(3)`, `Profit (profit)`), which fits an id/display split. +- **Answer 30** and the ADR's worked example (`Tax` in `CustomCat`) are about the + *rare* case. The decision stands. The emphasis of the surrounding text changed: + revision 9 wrote the dimension target as the main case and the `_Measure` target as + the exception. + ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. @@ -479,10 +518,13 @@ categories. An operation that creates an item still names the parent category. **Virtual categories have reserved ids** (answer 23): `_index`, `_dim` and `_measure`, with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** -A `_Measure` formula target has no item of its own. Its handle is its -formula's `FormulaId`, which is why the category panel's `_Measure` rows need +**Measures are the normal formula target** (answers 35 and 36), and **a measure +has an id and a display**. A stored measure is an ordinary item. Revision 9 says a +`_Measure` formula target has no item of its own and that its handle is its +formula's `FormulaId`; that is **under question again (Q23)**, because it is +the normal case. On revision 9's reading, this is why the category panel's `_Measure` rows need `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). **A formula -target outside `_Measure` is a real item** (answer 30). It has its own `ItemId`, +target outside `_Measure`, the rare case (answer 36), is a real item** (answer 30). It has its own `ItemId`, and its formula has a separate `FormulaId`. Generated ids follow clause 4, so a formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-`. The panel lists that row as `ItemHandle::Item`, like any other item. @@ -612,7 +654,8 @@ define. The second was chosen by the implementation (PR #318; `context/code/core prefer a narrower one (ASCII whitespace only, say), and that is a one-line change. An underscore the display already had is still never merged with the one a run makes (`a_ b` → `a__b`), because the user said *consecutive spaces* collapse, not - consecutive underscores. + consecutive underscores. That is **Q24**. A trailing run slugs to a trailing + `_` (`East ` → `east_`), which is **Q25**. - **A slug of only `-` is a legal bare id.** `-` and `---` slug to themselves. Step 5's "if nothing is left" means the empty string only. The id alphabet admits them, so a line such as `-: row` is legal but hard to read. @@ -741,7 +784,9 @@ Required tests: a category displayed `Format`, on a row axis, round-trips; and a view whose number format is each axis keyword round-trips, in a document where the same view also holds a category `format` on an axis. -**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). The shape is the ADR's +**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). **Under question +again by answers 35 and 36, because the `_Measure` target is the normal case (Q23).** +The shape below is revision 9's. It is the ADR's own reading, not a user answer, except that the formula id's place is answer 5's and an item target's id is answer 30's. It is `- () = []`: @@ -861,7 +906,8 @@ pipe-quotes any display name that would otherwise read as display plus id: `Workbook::active_view` (improvise-p003 item 3). Answer 1 makes this natural rather than a migration of its own. A view or category rename then orphans nothing. -- **A `_Measure` formula-target row** (`ItemHandle::FormulaTarget(FormulaId)`): +- **A `_Measure` formula-target row** (`ItemHandle::FormulaTarget(FormulaId)`; under + question again, Q23): removing it means `RemoveFormula`, and renaming it means a display rename of the formula's target (improvise-i0n7). - **CheckedBatch's created-record exemption goes** (the ADR's own reading, not a @@ -1127,7 +1173,7 @@ The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q1 revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All of those are answered.** After acceptance, three implementation questions were answered on their tickets (answers 30–32), and the user's comments on revision 9 are -answers 33 and 34. Revision 9 adds Q23, which is open. +answers 33 and 34, and its two later comments are answers 35 and 36. Revision 9 adds Q23 to Q25, which are open. | # | question | my recommendation | status | |---|---|---|---| @@ -1158,7 +1204,10 @@ answers 33 and 34. Revision 9 adds Q23, which is open. | — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation | | — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31). **Order reversed: ids first until there is autocomplete** (answer 33) | | — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32) | -| 23 | A formula line's spelling (revision 9) | see "Still open" | **open** | +| — | Whether a measure has an id/display split, and which formula targets are normal (user comments on revision 9) | — | **answered in principle**: measures have the split (answer 35), and the normal target is a measure (answers 35, 36). The mechanism is Q23 | +| 23 | How a formula-target measure carries its id, and how a formula line spells its target (revision 9; reworded after answers 35 and 36) | none until the user chooses | **open** | +| 24 | Does `a_ b` slug to `a__b`? (answer 34) | (a) keep | **open** | +| 25 | Does a trailing whitespace run slug to a trailing `_`? | (a) keep | **open** | ## Still open @@ -1170,34 +1219,69 @@ review were left as tickets rather than ADR text: improvise-le4p, improvise-m56i improvise-p003. Revision 9 folds all three in, and improvise-m56i's two questions are answers 30 and 31. -**Revision 9 needs the user's approval, and it asks one question.** +**Revision 9 needs the user's approval, and it asks three questions**, Q23 to Q25. -**Q23. How does a formula line spell its target?** Answer 30 makes a target -outside `_Measure` a real item, and says a formula line names it "by the item's -id". A `_Measure` target has no item and so no item id. Clause 5 above therefore -reads the line as `- () = …`, where `` is an item id -outside `_Measure` and a display inside it. That puts two different kinds of -thing in one position, and the category after `[` is what tells them apart. -- **(a) Recommended: as clause 5 reads it.** No extra syntax. The one asymmetry - is that a `_Measure` target's display appears in the file bare, while every - other name is an id. -- **(b) Always a display, and the item id in a second tag:** - `- Tax(tax-1)(tax) = … [customcat]`. The two positions never mix kinds, but a - formula line outside `_Measure` gains a second parenthesised tag. -- **(c) Always the target's id**, giving a `_Measure` target an id of its own - that is distinct from the formula's. This reverses clause 1's "its handle is its - formula's `FormulaId`". +**Q23. How does a formula-target measure carry its id, and how does a formula line +spell its target?** (Reworded after answers 35 and 36. Revision 9 asked only how +the line spelled its target, and it had the dimension case in view.) -Two consequences of (a) that the answer should take in, both the ADR's own reading: -- **References to a formula target** are as clause 5 states them, for an expression - and a filter value alike: the id text of an `ItemHandle`, which is the item id - outside `_Measure` and the formula id for a `_Measure` target. That holds under - (a) and (b). Under (c) a `_Measure` target would be named by its own id instead. -- **Quoting the target position.** A `_Measure` display in the target position can - itself look like `Name(x)`, which would read back as a display and a formula id. - The writer's rule that pipe-quotes such a display therefore covers a - formula-line target too: `- |Margin(%)|(margin) = …`. An item id never contains - a parenthesis, so only a `_Measure` display can need it. +The user's words: measures have an id and a display, and measures are the normal +formula target. What is open is the *mechanism*. A stored measure is already an item +with both. A measure created by a formula (`profit`) has, in revision 9, no item: +its display lives on the `Formula`, and its id is the formula's own id. Clause 5 +therefore spells the line `- Profit() = … [_measure]` (the display bare) +for the normal case, and `- () = … []` for the rare +dimension case, so the two cases put different kinds of thing in one position. + +- **(a) Keep the handle: the target's id is the formula's id.** The display lives on + the `Formula`. The id is generated from the target's display, and with no item to + collide with it comes out bare: `Profit` → `profit`, not `profit-`. + `- Profit(profit) = revenue - cost [_measure]`, and a dimension target is + `- tax(tax-1) = … [customcat]`. References to the measure are `profit`. This is + the smallest change, and matches the pictures in answers 5 and 19. Whether it + *is* the split the user asks for is the thing to decide. Its cost is that the + normal case writes a bare display in a position that holds an id for the rare one, + which sits against answer 3. +- **(b) Always a display, then ids in tags.** Measure: `- Profit(profit) = … [_measure]`. + Dimension: `- Tax(tax-1)(tax) = … [customcat]`. The target position never mixes + kinds. A dimension line gains a second tag. +- **(c) A formula-target measure is a real `_Measure` item**, like a dimension target + under answer 30: its own `ItemId` and display, and a separate `FormulaId` + (`profit`, then `profit-`). The target position is the item id in every case: + `- profit(profit-1) = revenue - cost [_measure]`. This is the reading closest to + "measures have an id/display split too". It removes `ItemHandle::FormulaTarget`, + returns `CellKey` to `(CategoryId, ItemId)`, and ends the `CellKey` typing problem + Codex raised on this PR. It reverses clause 1's "no item of its own" and revision + 9's readings in clauses 1, 2, 5 and 8. It also needs the item and its display + written somewhere: today only non-formula `_Measure` items persist, and formula + targets are rebuilt from `## Formulas`. + +**No recommendation is made until the user chooses.** Revision 9 recommended (a), +written when the dimension case was the one in view. (c) is the literal reading of +answer 35 and (a) is the minimal change. + +Two consequences, both the ADR's own reading, for whichever is chosen: +- **References to a formula target** are the id text of its handle: under (a) and + (b) the item id outside `_Measure` and the formula id for a `_Measure` target; + under (c) always the target's item id. +- **Quoting the target position.** Where a display sits in the target position + (a, b), a display that looks like `Name(x)` would read back as a display and an id, + so the writer pipe-quotes it: `- |Margin(%)|(margin) = …`. An item id never + contains a parenthesis, so only a display needs it. (c) has no such position. + +**Q24. Does `a_ b` slug to `a__b`?** Answer 34 collapses a run of whitespace to one +`_`, and the ADR applies the steps as written, so an underscore the display already +had is not merged with the one a run makes. The user did not say. +- **(a) Keep it.** `a_ b` → `a__b`, and a deliberate `a__b` stays. +- **(b) Collapse any run of whitespace and `_` to one `_`.** `a_ b` → `a_b`, and + `a__b` → `a_b`. That changes ids for displays that hold a double underscore. + +**Q25. Does a trailing run slug to a trailing `_`?** Step 4 removes leading `_` and +nothing removes trailing ones, so `East ` → `east_`. The user did not say. +- **(a) Keep it.** It is what PR #318 does. +- **(b) Remove trailing `_` as step 4 removes leading `_`.** `East ` → `east`, and + also `East_` → `east`. +- **(c) Trim whitespace before step 2.** `East ` → `east`, and `East_` stays `east_`. Revision 9 states a reading of its own, which is not a user answer, in each of these nine places (three of them hold two readings). Each is flagged where it appears, and @@ -1211,7 +1295,7 @@ approving the PR approves them: - the axis-before-`format:` order and the quoting of a format that reads as an axis line (clause 5); - the formula-line shape, and a `_Measure` target's display living on the `Formula` - (clause 5); + (clause 5; **reopened by answers 35 and 36, Q23**); - CheckedBatch's created-record exemption is removed (clause 6); - a display equal to another thing's id counts as ambiguous in the formula panel (clause 6). From 769f7b251f4fad26126c430f88d1635b5325309e Mon Sep 17 00:00:00 2001 From: "Claude (adr7-steward)" Date: Mon, 5 Oct 2026 13:49:46 -0700 Subject: [PATCH 023/106] docs(adr): put answers 35 and 36 on their true head and mark Q23's premises (improvise-cq1q.1) Merge the eighth and ninth rounds (all four comments are on 4aed8c07), quote the anchored line, drop the lean toward option (c), add pending-Q23 markers, and record the review's non-blocking points as flagged text. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 101 +++++++++++++++++++-------- 1 file changed, 70 insertions(+), 31 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index b6776588..e2d3533d 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -8,13 +8,13 @@ user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17: **Revision 9 is a proposed amendment, not yet approved.** It folds in the user's answers on the implementation tickets (answers 30–32), the user's two inline comments on its head `4aed8c07` and a clarification in chat (answers 33 and 34), -its two inline comments on `e9659903` (answers 35 and 36), and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 +its two later inline comments, also on `4aed8c07` (answers 35 and 36), and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 reverses the matching order that answer 31 recorded, and answer 34 overrides the ADR's own reading of "space". Answers 35 and 36 make measures the normal formula target and give them an id/display split, which reopens Q23. It asks three new questions, Q23 (how a formula-target measure carries its id, and how a formula line spells its target), Q24 and Q25 (two details of slugify's whitespace rule), and -flags nine places where it states a reading of its own that is not a user answer, +flags eleven places where it states a reading of its own that is not a user answer, listed under Q23. · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · @@ -376,12 +376,16 @@ answers 31 and 33 differ, answer 33 governs. Clause 12 is rewritten to match. -### Eighth round: answers on revision 9 +### Eighth round: answers on revision 9 (head `4aed8c07`) The user answered revision 9 (`context/adr/007-stable-identities.md` at `4aed8c07`) -with two inline comments on PR #326, posted 2026-10-04 02:48Z and 02:49Z, and then -clarified the second one in chat (not on the PR, relayed to the steward). Each is -mapped to the line it is attached to, and quoted verbatim. +with four inline comments on PR #326: two posted 2026-10-04 02:48Z and 02:49Z, and +two posted 2026-10-05 20:22:52Z and 20:23:40Z. The GitHub API gives all four the +original commit `4aed8c07`. **All four are one round**: the user wrote the last two +before `e9659903`, which folded in the first two, existed, so they had not seen +answers 33 and 34 written up. The second comment was also clarified in chat (not on +the PR, relayed to the steward). Each is mapped to the line it was attached to on +`4aed8c07`, and quoted verbatim. **33. The order of answer 31.** On line 356, the sentence under answer 31, "A typed name is matched against displays first, then against ids": @@ -413,16 +417,14 @@ This **overrides the ADR's own reading** (revision 9's, which was not a user ans 4 is **the ADR's own reading of "whitespace"**, flagged there. Answers 18 and 21 keep their wording, which this reads as "space, meaning whitespace". -### Ninth round: answers on revision 9's head `e9659903` - -The user posted two inline comments on PR #326 at 2026-10-05 20:22:52Z and 20:23:40Z, -both on the line holding Q23's option (c). Each is quoted verbatim. - -**35. Measures get the id/display split too.** +**35. Measures get the id/display split too.** On line 1091, the end of Q23's +option (c), "Always the target's id, giving a `_Measure` target an id of its own that is +distinct from the formula's. This reverses clause 1's 'its handle is its formula's +`FormulaId`'": > "Measures should have ID/display split too. Measures are the normal case for formulas, anyways: something like `profit = revenue - cost`" -**36. Dimensions are rarely formula targets.** +**36. Dimensions are rarely formula targets.** Also on line 1091, 48 seconds later: > "Dimensions are mainly in formulas when you're calculating some kind of derived dimension, but usually dimensions should be relatively static and not interesting in formulas." @@ -440,8 +442,10 @@ things in earlier text meet them, and none is resolved here: an id and a display. A measure that exists only because a formula targets it has its display on the `Formula` and its id in the `FormulaId`. Whether that is the split the user wants, or whether the target should be an item, is Q23. -- **Answer 3** ("every reference is an id") sits badly with revision 9's line for the - normal case, where the target position holds a bare display. +- **Answer 3** ("every reference is an id") is about references. Clause 5 calls a + formula line a declaration, so the bare display in its target position is a + declaration's display and not a reference. Whether the normal case should still + spell an id there is part of Q23. - **Answers 5 and 19** picture the measure case as a display plus one id (`Profit(3)`, `Profit (profit)`), which fits an id/display split. - **Answer 30** and the ADR's worked example (`Tax` in `CustomCat`) are about the @@ -552,7 +556,8 @@ Following answers 1 to 3: user answer), where `ItemHandle` is clause 6's `{Item(ItemId), FormulaTarget(FormulaId)}`. Outside `_Measure` every coordinate is an `Item`. A `_Measure` formula cell's coordinate is `FormulaTarget`, because - that target has no item. The key stays sorted, now **by category id**, so gotcha + that target has no item (**pending Q23**: option (c) gives it an item, and this + reading falls). The key stays sorted, now **by category id**, so gotcha 5's canonical form is kept with a different sort key. `DataStore` interns ids exactly as it interns names today. Answer 22 makes every id unique across kinds, so an id's text alone names one thing; the tag is there so that the types @@ -938,7 +943,8 @@ Following answer 3, `RecordHash` hashes the explicit serialization of the sorted over different strings. The text is enough: answer 22 makes ids unique across kinds, so no `Item` and `FormulaTarget` can spell alike, and the hash stays canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate -is never part of a record in the first place. +is never part of a record in the first place. **Pending Q23**: under option (c) no +`FormulaTarget` exists, and this paragraph reduces to ids over `ItemId`s. - **ADR 002's decision is unchanged in substance.** It still hashes an explicit serialization of a normalized record, with a fixed key and a width argued for. - **Its prose will be out of date once this lands.** ADR 002 says `CellKey::new` @@ -1013,7 +1019,8 @@ Following answers 15 and 25: 1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224. 2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId`, `ViewId`, and `ItemHandle`, which the re-key (step 3) needs for `CellKey` before - clause 6's step (5) uses it in projections. + clause 6's step (5) uses it in projections (**pending Q23**: under option (c) + there is no `ItemHandle`). 3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with display→id resolution moved to every input edge. 4. **Persistence:** the new version literal (clause 10), the grammar in clause 5, the @@ -1238,27 +1245,49 @@ dimension case, so the two cases put different kinds of thing in one position. collide with it comes out bare: `Profit` → `profit`, not `profit-`. `- Profit(profit) = revenue - cost [_measure]`, and a dimension target is `- tax(tax-1) = … [customcat]`. References to the measure are `profit`. This is - the smallest change, and matches the pictures in answers 5 and 19. Whether it - *is* the split the user asks for is the thing to decide. Its cost is that the - normal case writes a bare display in a position that holds an id for the rare one, - which sits against answer 3. + the smallest change. Its cost is that the normal case writes a display in a + position that holds an id for the rare one. Under (a) a measure does have an id + and a display; whether that is the split the user asks for is the thing to decide. - **(b) Always a display, then ids in tags.** Measure: `- Profit(profit) = … [_measure]`. Dimension: `- Tax(tax-1)(tax) = … [customcat]`. The target position never mixes kinds. A dimension line gains a second tag. - **(c) A formula-target measure is a real `_Measure` item**, like a dimension target under answer 30: its own `ItemId` and display, and a separate `FormulaId` (`profit`, then `profit-`). The target position is the item id in every case: - `- profit(profit-1) = revenue - cost [_measure]`. This is the reading closest to - "measures have an id/display split too". It removes `ItemHandle::FormulaTarget`, + `- profit(profit-1) = revenue - cost [_measure]`. It removes `ItemHandle::FormulaTarget`, returns `CellKey` to `(CategoryId, ItemId)`, and ends the `CellKey` typing problem Codex raised on this PR. It reverses clause 1's "no item of its own" and revision 9's readings in clauses 1, 2, 5 and 8. It also needs the item and its display written somewhere: today only non-formula `_Measure` items persist, and formula targets are rebuilt from `## Formulas`. +Answers 5 and 19 picture the measure case as `Profit(3)` and `Profit (profit)`. Both +pictures fit (a), (b) and (c): each gives the measure a display and an id, and under +(c) the `profit` in answer 19's picture is the item id. + **No recommendation is made until the user chooses.** Revision 9 recommended (a), -written when the dimension case was the one in view. (c) is the literal reading of -answer 35 and (a) is the minimal change. +written when the dimension case was the one in view. Any ranking of how well an option +fits answer 35 would be the ADR's own judgement and not the user's, since (a) and (b) +also give a measure an id and a display. (a) is the smallest change. + +Four further points the choice must settle, all the ADR's own reading and none a +user answer: +- **A formula targeting a stored measure of the same name.** `measure_item_names` + merges them today, so a stored `Profit` column and a formula `Profit = …` are one + measure. Under (a) and (b) that one column has two ids (the item's `profit` and the + formula's `profit-`), and the ADR must say which one keys and references use. + Under (c) it is the natural case, and the formula attaches to the existing item. +- **Removing a formula under (c).** `RemoveFormula` drops the formula and leaves its + `_Measure` item, as it leaves a dimension target's item today (answer 30). The + measure column then has no formula and possibly no data. +- **The `[_measure]` tag.** Today's writer omits the tag for a `_Measure` formula + (`- {raw}`), and the user's own example, `profit = revenue - cost`, has none. The + examples above write it only to show the category. Whether the tag is omitted for + `_measure` is not decided here. +- **Other ADRs.** ADR 008 (PR #308) anchors the cursor by item identity, which under + (a) or (b) a formula-measure column does not have, so it would need `ItemHandle`. + ADR 009 (PR #309) relies on `find_item_category`'s `_Measure` fallback for formula + targets, which (c) would remove. Neither ADR is edited here. Two consequences, both the ADR's own reading, for whichever is chosen: - **References to a formula target** are the id text of its handle: under (a) and @@ -1271,24 +1300,31 @@ Two consequences, both the ADR's own reading, for whichever is chosen: **Q24. Does `a_ b` slug to `a__b`?** Answer 34 collapses a run of whitespace to one `_`, and the ADR applies the steps as written, so an underscore the display already -had is not merged with the one a run makes. The user did not say. +had is not merged with the one a run makes. The user did not say. The author's +lean, not the user's, is (a): it is what the user's wording ("consecutive spaces") +and PR #318 give. - **(a) Keep it.** `a_ b` → `a__b`, and a deliberate `a__b` stays. - **(b) Collapse any run of whitespace and `_` to one `_`.** `a_ b` → `a_b`, and `a__b` → `a_b`. That changes ids for displays that hold a double underscore. +A neighbouring edge: step 2 runs before step 3 strips non-alphabet characters, so +`Time \u{200B} Period` (a zero-width space between two spaces) slugs to +`time__period`, two runs and not one. + **Q25. Does a trailing run slug to a trailing `_`?** Step 4 removes leading `_` and -nothing removes trailing ones, so `East ` → `east_`. The user did not say. +nothing removes trailing ones, so `East ` → `east_`. The user did not say. The +author's lean, not the user's, is (a). - **(a) Keep it.** It is what PR #318 does. - **(b) Remove trailing `_` as step 4 removes leading `_`.** `East ` → `east`, and also `East_` → `east`. - **(c) Trim whitespace before step 2.** `East ` → `east`, and `East_` stays `east_`. Revision 9 states a reading of its own, which is not a user answer, in each of these -nine places (three of them hold two readings). Each is flagged where it appears, and +eleven places (three of them hold two readings). Each is flagged where it appears, and approving the PR approves them: - the virtual views' reserved ids `_records` and `_drill` (clause 1); - the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); -- a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2); +- a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2; **pending Q23**); - `collapsed:` is `(CategoryId, group name)` (clause 2); - "whitespace" means the Unicode `White_Space` property (replacing the U+0020 reading, by answer 34), and a slug of only `-` is a legal id (clause 4); @@ -1298,7 +1334,10 @@ approving the PR approves them: (clause 5; **reopened by answers 35 and 36, Q23**); - CheckedBatch's created-record exemption is removed (clause 6); - a display equal to another thing's id counts as ambiguous in the formula panel - (clause 6). + (clause 6); +- an id is matched exactly, with no case folding (clause 12); +- the display stage of the typed-name resolver goes once autocomplete exists + (clause 12). ## Consequences From 068e4130714ccb08d45e34564af381de6f3a4a78 Mon Sep 17 00:00:00 2001 From: "Claude (adr7-steward)" Date: Mon, 5 Oct 2026 14:07:05 -0700 Subject: [PATCH 024/106] docs(adr): mark CategoryRow pending Q23 and qualify answer 5's fit (improvise-cq1q.1) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index e2d3533d..bc22f671 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -447,7 +447,8 @@ things in earlier text meet them, and none is resolved here: declaration's display and not a reference. Whether the normal case should still spell an id there is part of Q23. - **Answers 5 and 19** picture the measure case as a display plus one id - (`Profit(3)`, `Profit (profit)`), which fits an id/display split. + (`Profit(3)`, `Profit (profit)`), which fits an id/display split (answer 5's + file-line picture fits option (c) only loosely; see Q23). - **Answer 30** and the ADR's worked example (`Tax` in `CustomCat`) are about the *rare* case. The decision stands. The emphasis of the surrounding text changed: revision 9 wrote the dimension target as the main case and the `_Measure` target as @@ -891,7 +892,8 @@ pipe-quotes any display name that would otherwise read as display plus id: `ClearCell { view, … }`, which improvise-wpxr (PR #281, merged) gave a view named by its name, like every other view-scoped variant (improvise-lltt). - **Projections carry id and display together:** - - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }`; + - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }` + (**pending Q23**: under option (c) there is no `ItemHandle`); - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows (answer 19): the target's display name with its id in parentheses, then the expression over display names, with an id wherever a display name is ambiguous @@ -1261,9 +1263,11 @@ dimension case, so the two cases put different kinds of thing in one position. written somewhere: today only non-formula `_Measure` items persist, and formula targets are rebuilt from `## Formulas`. -Answers 5 and 19 picture the measure case as `Profit(3)` and `Profit (profit)`. Both -pictures fit (a), (b) and (c): each gives the measure a display and an id, and under -(c) the `profit` in answer 19's picture is the item id. +Answers 5 and 19 picture the measure case as `Profit(3)` and `Profit (profit)`. +Answer 19's panel picture fits (a), (b) and (c): each gives the measure a display and +an id, and under (c) the `profit` is the item id. Answer 5's file-line picture, +`- Profit(3) = …`, fits (a) and (b) exactly, and (c) only loosely, because (c) writes +the item id in the target position where answer 5 shows the display. **No recommendation is made until the user chooses.** Revision 9 recommended (a), written when the dimension case was the one in view. Any ranking of how well an option From a9bcb816e9d3d44c40982fa41b6ac723a1126fd0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:54:30 -0700 Subject: [PATCH 025/106] docs(adr): ADR 009 revised to fit ADR 008's view/host split (improvise-felo.4) Adds the user's ADR 008 statements that bear on this ADR, clauses 12-14 (network transparency, incremental max width, resident state), keys operator state by projection shape at item grain, retires the per-mirror delivery assumptions, and appends questions 12-21. Questions 1-11 stay open. Reads ADR 008 at 69f0a7c6. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 786 ++++++++++++++++++++-- 1 file changed, 714 insertions(+), 72 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index f76ff26c..eaa01619 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,12 +1,19 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. · +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 19** +(PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in +the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details +ADR 008 routes here. They are in the section "What ADR 008's review established for this +ADR", in clauses 12 to 14, and in questions 12 to 18. **The user has answered none of +this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline +comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the -virtualized grid, PR #308, which is revised to assume this ADR) · +virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision +reads it at `69f0a7c6`, its revision 19) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -14,8 +21,13 @@ is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #3 no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are -dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means -revision 3 of `context/adr/008-virtualized-grid.md` on PR #308. +dated, so re-derive them before citing this document elsewhere. **The 2026-10-05 +revision's new citations are at `gh/main` `50e1314d`**, and each says so. The older +citations were not re-derived, and some have moved. For example, `Host::mutating_frame` +takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and +`App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means +revision 19 of `context/adr/008-virtualized-grid.md` on PR #308, at `69f0a7c6`. Clauses 3, +5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -61,7 +73,123 @@ from scratch. The machinery is: **The grid is the second consumer.** ADR 008 needs the same maintained state for other purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, and column totals over rows never scrolled to (clause 9). It gets them from the same -circuit, not from a second mechanism. +circuit, not from a second mechanism. Since ADR 008's view/host split, the grid's consumer +may be in another process, and clause 12 says how the same circuit serves both. + +## What ADR 008's review established for this ADR + +At its revisions 18 and 19, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +user routed part of it here. This section quotes what the user said that bears on this ADR, +and says where each statement came from. +- **PR comments** are cited by their GitHub comment id and UTC time. The question numbers + are ADR 008's, as ADR 008 assigns them. +- **The two chat statements** were given to the orchestrator and relayed to this ADR. They + are on no PR, and ADR 008 records them the same way. The user's "Approved" on this ADR + would confirm them. + +None of this answers this ADR's own questions 1 to 11. + +**Routed here, in chat:** +- "The DBSP machinery needs to be implemented in a network transparent way, but the + details of that go to ADR 009" (chat, relayed on 2026-10-05). Clause 12. +- "Thinking about it a bit more, we might need to keep track of recent projections + + clients to stream aggregation results back to the UI correctly. But this should be as + stateless as possible" (chat, relayed on 2026-10-05). Clause 12. +- On ADR 008's question 53: "Q53 is cross product: I'm still ambivalent about which side + of the wire pruning should happen on" (chat, relayed on 2026-10-05). So a request's + top-left is a cross-product position, and which side prunes is open. ADR 008 records it + at revision 19 as its question 53 answered and its question 57 open. Clause 12, + question 19. +- "But the top left should be as absolute as possible" (chat, relayed on 2026-10-05). + Clause 12, question 20 (ADR 008's question 58). +- "After pruning we might have to stream rows and columns to attempt to fill out the full + grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). + +**Asked on ADR 008, and not yet answered:** +- "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, + 2026-10-03T03:15:38Z, on ADR 008's question 2). Clause 13 is this ADR's reading of it. + Question 12. + +**The read side and the write side** (#308, inline comments of 2026-10-05): +- Q43 (4189071865, 21:34:45Z): "yeah, the read side of the model is a projection of the + model's full space into a particular set of axes using a `project` verb, with + offset+dimensions. The write side is an apply function that can only take model + commands and applies them to the model using Redux-style Flux architecture." +- Q38 (4189007864, 21:27:55Z): "No, axes are a model concept because they define the + grid. Aggregation should always be done model-side for consistency." +- Q38 (4189027924, 21:30:08Z): "A grid request would look something like:" + ``` + Axes: Row(A,B,C), Col(D,E), Page(F), None(G,H) + Top-left (4,5) # relative to the cross-product of the axes in the category-native order (which might be arbitrary) + Dimensions (10,10) # so get ten rows and ten columns worth of data from the grid defined by the axes. + ``` +- Q33 (4188936570, 21:20:11Z): "The host shouldn't have to know anything about the + viewport: it's mostly a question of the parameters of the grid the user is actually + looking at, rather than a "viewport" strictly speaking." +- Q37 (4189000059, 21:27:04Z): "I don't think the host needs to no [sic] anything about + the UI: the initial load of the UI should just be the default view stored in the model + (or, in some cases, a particularly requested view). Sending UI state over the wire is + an architecture smell." +- Q42 (4189051110, 21:32:31Z): "The space scrolling happens in is defined by the axes, top + left corner offset and with/height. But how much of this space is actually materialized + and when fetches, etc. happens (and model values are cached) is client-policy driven." +- Q32 (4188917796, 21:18:17Z): "I can't think of any cursor command with model-side + effects." +- Q27 (4188816383, 21:07:43Z): "go with the recommendation: the ordered list of records is + a model-side concept." +- Q31 (4188911659, 21:17:39Z): "You have to distinguish view in the sense of UI and view + in the sense written into `.improv` files. P has to be a model-side effect because it + prunes data for a particular view. It is not a UI-only command." And (4188926877, + 21:19:10Z): "The only real issue here is that pruning shouldn't infect every client + here. It might need to be UI-only until a view is saved, and then the view definition + goes to the model." +- Q29 (4188887721, 21:15:14Z): "Prune should only be written if the user explicitly + unpruned the view." +- The cursor: "The cursor should never jump: it should stay where the user left it as much + as possible." (Q12, 4171501445, 2026-10-03T03:21:05Z). And "As far as possible, updates + should shift the grid being viewed to keep the cursor in the same visual place on the + screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). + +**Answers ADR 008 records that this ADR's text had not caught up with:** +- Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. +- Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases + where the column size has been stably smaller." And Q22 (4188309022, + 2026-10-05T20:15:12Z): "hysteresis as recommended." Clauses 9 and 13. +- Q6 (4171493762, 2026-10-03T03:18:12Z): "maintained incrementally". Clause 9. +- Q13, Q14 and Q15: "the recommendation is fine.", three comments (4171504439, + 4171505979 and 4171509012, 2026-10-03T03:22:16Z to 03:23:36Z), which ADR 008 records as + its answers to those three questions. Q13 sets the frame budget, and Q15 sends the + formulas the per-record map cannot express to the occupied contexts. Clauses 5a and 14. +- The store's memory (4171481179, 2026-10-03T03:13:14Z): "We have to be careful here to + minimize memory requirements of the store: offloading inactive regions to disk (maybe + mmap or similar), double-delta/gorilla/lz4 compression, etc." And Q17 (4188271710, + 2026-10-05T20:11:02Z): "Let's defer this as recommended." Clause 14. +- Jobs (4171458281, 2026-10-03T03:04:52Z): "We also have to have a way to + invalidate/cancel in-flight jobs efficiently and probably a way to throttle them (a + priority queue or similar) to avoid pinning the CPU." Then (4171694278, 04:30:31Z): + "priority queue for work stealing is also important here." Clause 7. + +**What this ADR reads from them.** These are the author's readings, and each one names +the clause it changes: +1. **Reads and writes go through `project` and `apply`.** A consumer reads the circuit's + outputs through `project`, and the circuit's inputs change only through `apply` of + model commands. So nothing here may depend on a host-held mirror of a client + (`Sessions`, `ClientSession`, `reduce_view` on the host), on the per-mirror diff + (`grid_updates`), or on resetting a client's cursor. Clauses 9 and 12. +2. **Aggregation stays on the model side.** No client runs an aggregate operator, so + network transparency concerns the circuit's outputs, and a live source's inputs. It + does not mean splitting operators across a wire. Clause 12. +3. **State is keyed by the request's shape, not by a view's name.** Axes are a model + concept, so operator state keyed by axes is model state, not UI state. But a `project` + request names its own axes, and an unsaved request may carry parameters no saved view + holds (ADR 008's flagged reading 1, its question 51). Clause 7, question 14. +4. **The record order belongs to the store.** The ordered list of records is the store's + (Q27). The circuit may consume it, and does not own it. Clause 9. +5. **The memory requirement covers this ADR too.** The store-memory requirement covers the + circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. + Clause 14, question 17. +6. **Five of ADR 008's questions are answered.** Those are questions 1, 2, 6, 13 and 15, + and clauses 5a, 9, 13 and 14 now follow the answers. ## Context @@ -133,9 +261,9 @@ whose inputs moved, over aggregates that are maintained rather than re-folded. T machinery is a new, problem-agnostic workspace crate implementing the DBSP core. improvise-core defines a trait for the derived state its evaluator reads. Today's evaluator is one implementation of that trait, and a circuit built on the new crate is -the other. Both stay in the tree, and a differential test holds them equal.** Eleven +the other. Both stay in the tree, and a differential test holds them equal.** Fourteen clauses, plus clause 5a. It is numbered 5a so that no clause number or question number -cited elsewhere moves. +cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. ### 1. The crate: `zset` (name is question 1) @@ -179,8 +307,12 @@ Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** bey improvise's circular semantics are not a least fixed point (clause 6). - **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies batches in one total order (ADR 001, "There is already a total order"). -- **Persistence, spilling to disk, and threads.** State lives in memory, and a browser - worker has one thread. +- **Spilling to disk, compression, and threads, in the first version.** State lives in + memory, and a browser worker has one thread. **This was a non-goal and is now only a + deferral.** The user's store-memory requirement (ADR 008 clause 12) covers this state + too, and clause 14 says what the crate must not rule out. A host with threads may run + independent circuits on a pool (ADR 008 question 23's answer). One circuit's `step` + stays single-threaded. - **A query language or planner.** Circuits are wired by hand in the seam crate (clause 2). @@ -244,7 +376,9 @@ same-value remove-then-set would cancel, defeating 5a. An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy -inside the effect loop (`app.rs:1073-1084`). +inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation +under the view/host split. The per-effect step stays, because each step is what clause +12 stamps. **A delta key's identity is bit identity, the same identity the record tier uses today.** Two elements have the same `(key, value)` part exactly when their keys are @@ -301,7 +435,7 @@ new is observed. Its cost is O(formulas), where `document_content` is O(cells) p **The group** is the cell key with the view's `none_cats` removed. That is the partial key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct -`none_cats` list, shared by every view that has that list. That replaces +`none_cats` list, shared by every projection shape that has that list (clause 7). That replaces `rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does (`types.rs:717-722`). @@ -541,9 +675,16 @@ is one evaluator. **The boundary.** The map is sound only when `expr` reads the record's own cells, so that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value -depends on other records, is not a function of one record. That formula stays on the -eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. Two regressions pin the record stage against eager: +depends on other records, is not a function of one record. **Revised on 2026-10-05:** +such a formula does **not** stay on today's eager path, because that path is +`Expansion::members`, the cross product (ADR 008 clause 5, a finding of Copilot's on +#308). It takes ADR 008's **store-driven fallback**, which evaluates the expression only +over the **occupied contexts**, the contexts some record projects to. The user accepted +that recommendation, as ADR 008's question 15 ("the recommendation is fine."). It differs +from today's eager answer wherever the expression is non-empty at a context that holds no +record, and ADR 008 carves those cases out of its differential test by name. Whether this +ADR's `Eager` adopts the same rule, so that its own differential test needs no carve-out, +is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -555,7 +696,10 @@ another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A r whose key `record_of` takes without `_Measure`, then belongs to the context only through its cell under that measure, and eager orders it by *that* cell's position, not by the record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. +formulas stay on the eager path, and the differential test pins that they do. **That +eager path is `Expansion::members`, so while it lasts it blocks ADR 008's retirement of +`Expansion::members` (its migration step 6) for these formulas.** Sending them to the +occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as @@ -610,13 +754,58 @@ post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1 That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 states the same rule. -- **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), - plus the group membership of every drawn item, collapsed or not, plus - `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is - unchanged. A view whose operator shape moved has its operators rebuilt from the - integrated base Z-set **before the frame observation that follows this effect**. - The `LayoutGeometry` frame arm (`app.rs:1062-1071`) reads occupancy right after the - effect that moved the shape, so the next effect's cell delta is too late. +- **Operator shape, per projection shape (revised on 2026-10-05).** Operator state is + keyed by the **operator shape of a `project` request**, not by the name of a view + (question 14). The shape is made of: + - the categories of each axis, by level; + - the None categories; + - the filter and page keys; + - the hidden items and collapsed groups the request carries; + - `measure_agg`. + + A saved `.improv` view supplies these to a request that names it. An unsaved request + carries its own, for example an unsaved prune or hide (ADR 008's flagged reading 1, its + question 51). Requests with the same shape share one set of operators, whichever + clients sent them. The live shapes are a **cache**. A shape enters when a request names + it, and it is evicted once no recent request has named it (clause 12). For a saved + view the shape still includes what `view_shape` (`app.rs:40-72` at `57d385e1`) hashes, + plus the group membership of every drawn item and `measure_agg`. `view_shape` itself is + not widened, so what bumps `layout_version` is unchanged. + + *As first written* (superseded): a view whose operator shape moved had its operators + rebuilt **before the frame observation that follows this effect**. That rule served the + per-view frame observation, which ADR 008 clause 8 revises under the split. The three + rules below replace it. +- **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column + aggregates are keyed by the **items** of the axis categories, together with the filter + and page keys. They are never keyed by a position in a level sequence. So a collapse, + expand, hide or show of an **axis** category's items changes no operator. The visible + axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms + for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item changes no + operator either, because a new item tuple is just a new key. Three things still + rebuild a shape: + - a change of which categories are its levels; + - a change of its filter or page keys, which includes hiding an item of a category that + is not on an axis; + - a change of `measure_agg`. + + A collapsed group's value is a pooled fold over its members. With item grain it is + combined from the members' state when it is read. For `Sum` and `Count` that combine is + linear. For the integrated aggregates it costs O(members) per read, and whether that + fits the frame budget at a 1,000-member group is **not measured**. +- **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it + cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a + 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). + So a rebuild is a resumable `step(deadline)` over an owned snapshot of its input. The + deltas that arrive meanwhile are buffered and added at install, which Z-set addition + makes exact. A circuit `step` on a cell delta is O(delta), and it is never sliced. + Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the + install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and + priority queue the user required. This ADR only makes its rebuilds steppable and + cancellable between slices. +- **A crossing count (ADR 008's second request).** Each step's occupancy output carries + the number of tuples whose count crossed zero. ADR 008's ordered index uses it to + choose between an insertion and a rebuild by sorting. - **The formula set is an input, not a rebuild.** After each model effect the host compares a fingerprint of the formula set, over (`raw`, `target`, `target_category`), with its previous read. If it moved, the formula delta (−the old @@ -628,18 +817,18 @@ states the same rule. - **A document swap rebuilds every circuit** from the new store. `install_workbook` bumps `document_generation`, and the same after-each-effect read sees it. -Re-reading the operator shape costs O(items) per view per model effect, and the formula +Re-reading the operator shape costs O(items) per live shape per model effect, and the formula fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs O(cells), once per actual reshape. The same `view_shape` function is therefore read at two moments: by the circuit after each effect, and by `layout_version` after the loop. They answer different questions, and neither replaces the other. -**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** `AddItem` -moves the operator shape, so the view is rebuilt with East in its sequences. Only then -does `SetCell`'s delta arrive, at operators that already know East. So the in-loop -`CellData` frame observation reads occupancy for the new shape. The `LayoutGeometry` -arm (`app.rs:1062-1071`) reads the frame immediately after each geometry effect, and it -reads a rebuilt occupancy for the same reason. +**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first +written* (superseded by item grain): `AddItem` moved the operator shape, so the view was +rebuilt with East in its sequences before `SetCell`'s delta arrived. **Under item grain, +`AddItem` moves no operator.** `SetCell`'s delta inserts the item tuple `East` as a new +key, and only the visible axis, which is derived, gains a row. The case needs no +ordering rule at all. **Recommended: a page selection is not a rebuild (question 7).** Under the recommended answer, the group key includes the page categories, so `[` and `]` read another slice @@ -886,37 +1075,61 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 3, PR #308) assumes these, under the shared names below. +ADR 008 (revision 19 at `69f0a7c6`, PR #308) assumes these, under the shared names below. +**Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. +A consumer reads it through `project` (clause 12), and none of it is addressed to a +host-held mirror of a client. -- **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and - per column tuple of cells that hold a value. Stored cells pass the view's filter, with - hidden items dropped, and a collapsed group's members map to the group's tuple. - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` - holds a value: a top-level `WHERE` must match, or raw data must exist at the target. - The operators are rebuilt per clause 7, so a batch that reshapes and then writes feeds - its write to the new shape. Page coordinates key a slice, subject to question 7. A tuple - is on the axis iff its count is above zero. **This ADR does not settle ADR 008 - question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those - cells are values with no stored cell under them. The stem definition drops them, and - if the user rules otherwise, those views fall back to a non-occupancy path. Occupancy - is integer and order-free, so it does not wait for ADR 005. -- **The occupancy delta** (ADR 008 clause 8). `distinct` over occupancy emits a - non-empty delta exactly when a tuple enters or leaves an axis. A reframe is a - non-empty occupancy delta. The two-arm asymmetry of the frame observation (gotcha 24, - improvise-mcq4) is unchanged, but each arm costs O(delta) instead of a rebuilt grid - per view. -- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over - the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), +- **Occupancy** (ADR 008 clause 3). For each projection shape (clause 7), there is a count + per row tuple and per column tuple of cells that hold a value. The counts are kept at + **item grain**, and the visible axis is a remap of them. + - Stored cells pass the shape's filter. + - A hidden item of an axis category is masked only when the visible axis is derived. + - A collapsed group's members map to the group's tuple at read time. + - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` + holds a value: a top-level `WHERE` must match, or raw data must exist at the target. + - Page coordinates key a slice, subject to question 7. + + A tuple is on the axis exactly when its count is above zero. **ADR 008's question 1 is + answered: "go with occupied"** (#308, comment 4171484686). So the two cases where + today's pruning keeps a row with no stored cell under it are dropped on purpose: a fold + row with a formula column, and `Aggregation::Exact`. Clause 5's stem rule already drops + them, there is no "non-occupancy path" for them, and the differential test carves them + out by name. *As first written*, this item said this ADR "does not settle" that + question; that is superseded. Occupancy is integer and order-free, so it does not wait + for ADR 005. +- **The ordered record list** (ADR 008 clause 1, `Records`). The user: "the ordered list + of records is a model-side concept" (Q27). It is the store's order (gotcha 18, + `DataStore::live_records`), and this ADR does **not** maintain it as a per-shape + operator. Clause 5a's record stage keys records by `record_of` and orders them by the + store sequence, so it *consumes* the store's order and never defines a second one. How + a page and filter select from that list is ADR 008's question 55. It is not this ADR's + to answer, because this ADR keeps no per-view copy of the list. Whether records mode is + `project` over the virtual axes (ADR 008 question 54) changes nothing here: occupancy + over `_Index` and `_Dim` is the same operator as over any other axis. +- **The occupancy delta** (ADR 008 clause 8, which the split revises). `distinct` over + occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** + axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first + written*, its consumer was the per-view frame observation and its reset of each client's + cursor (gotcha 24). Under the split the host holds no cursor. So the delta is a model + fact, published on clause 12's delivery stream to the consumers of that shape, and a + client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the + host pushes it is question 13, which is ADR 008's question 52 seen from this side. +- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the + pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate, and the formula layer re-evaluates only the columns whose inputs moved, - through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that - hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the - exact accumulator (recommended) or ADR 005 (clause 8). -- **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, - or maintain) is struck there as decided by the user through this ADR. Its questions 1 - and 2 stay open there. -- **What it does not get.** Column widths depend on formatted text, which is - display-only (gotcha 8), so they are not maintained here. They stay ADR 008 question 2. + aggregate at item grain, and the formula layer re-evaluates only the columns whose + inputs moved, through clause 2's trait. Clause 5 replaces `Expansion::members`. Totals + that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: + the exact accumulator (recommended) or ADR 005 (clause 8). +- **What ADR 008 no longer asks.** Its question 6 is answered ("maintained + incrementally"). Its questions 1 and 2 are answered there too: 1 is "go with occupied", + and 2 is (b), sticky per session, with hysteresis as the easing (its question 22). *As + first written*, this item said they "stay open there", which had gone stale. +- **Widths.** A width depends on formatted text, which is display-only (gotcha 8). Under + the user's answer to ADR 008's question 2, widths are taken over the rows realized and + held by the view layer. The user's question whether DBSP can maintain a maximum width is + answered in clause 13. ### 10. ADR 003's live sources are delta streams, when they can be @@ -963,10 +1176,336 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So on the network path a write is not O(delta): it is O(cells) for the snapshot today, and roughly twice that with the circuit in it. The keyboard path (`apply_effects` - without a `Host`) takes no snapshot and keeps the O(delta) cost. + without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the + store-memory requirement (clause 14), that second clone is also peak memory, which a + rewind would avoid. Question 9 stays open, and is now weighed against clause 14 as + well. - **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the delta sequence, so replaying commands reproduces its state. +### 12. Network transparency: one circuit, consumed in process or across a wire + +**The requirement is the user's:** "The DBSP machinery needs to be implemented in a +network transparent way, but the details of that go to ADR 009" (chat). **Everything +else in this clause is this ADR's design, and the author's choices are marked as such. +None of it is measured.** + +**What crosses a wire, and what does not** (the author's reading). Two boundaries can be +a wire. +- **The output boundary** carries the results of `project` and their changes: a window's + cells, its axis occupancy, its totals, and ADR 008 clause 4's width facts. In the TUI + the consumer is in the same process. In the browser it is the page, across `postMessage` + from the worker or a socket from `improvise-ws-server` (gotcha 45). +- **The input boundary of a live source** (clause 10). A source that reports its changes + delivers a cell delta, possibly from another machine. The circuit cannot tell that + delta from one `DataStore` emits. + +Three things do not cross: +- **The write side carries no deltas.** A client changes the model only by `apply` of + model commands (the user, Q43), so traffic from a client to the host stays + `ModelCommand`s with their basis (ADR 001), as today. +- **The circuit's internal streams stay on the host.** The cell delta, the per-record + stage and the group aggregates never cross: aggregation is model-side (the user, Q38), + so no operator runs in a client. +- **No UI state crosses.** The host is told the request and nothing about a cursor, a + scroll or a key (Q32, Q33, Q37). + +**What a projection request is to the circuit** (the author's answer to "is the result +keyed by request, by axes, or shared?"). A request has two parts. +- **Its operator shape** (clause 7) selects the circuit's state. The shape is shared by + every request, from any client, that has it. +- **Its window**, the top-left and the dimensions, selects a part of that state's output. + The window is never state. + +So maintained results are keyed by **shape**, and the only per-request work is +restricting a shape's output to a window. That is O(visible) per request and needs +nothing kept between requests. + +**Positions are cross-product positions.** The user answered ADR 008's question 53 in +chat: "Q53 is cross product: I'm still ambivalent about which side of the wire pruning +should happen on" (relayed on 2026-10-05; on no PR). So a request's top-left is a position +in the **cross product** of its axes, in each category's own item order. That position is +a bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed +contexts, we can represent bigints as strings in JSON"). A cross-product position names +the same tuple whatever is occupied, so **an occupancy change never moves a request's +window**. It changes only which positions inside the window hold data. **Which side of +the wire prunes is open** (the user is "still ambivalent"), and is question 19. The +circuit must serve both sides, and what it must expose in each case is the author's +analysis: +- **Pruning on the model side.** `project` returns occupied rows and columns. The circuit + must then answer two order-statistic queries over a shape's visible occupancy, from a + cross-product position: + - the next k occupied tuples at or after a position, so a pruned window can start where + the request points; + - the rank of a tuple among the occupied ones, and the occupied tuple of a given rank, + so a client can scroll a pruned axis by rows and show "row i of n". + + That is ADR 008's ordered index over occupied tuples (its clause 3), the sorted `Vec` + first and the order-statistic B-tree when measured (its question 16), fed by this ADR's + occupancy delta and crossing count. Under this case a client sees only occupied rows, + and its scroll space is the pruned one, built on top of the cross-product offset. +- **Pruning on the client side.** `project` returns the cross-product window, empty + positions included, together with a **skip structure** the client can query: for each + axis, the occupied runs that intersect the window and the next occupied position past + each edge. The circuit then has to expose the same order-statistic queries, but as a + model fact a client can request, not as a re-indexing the model performs on its behalf. + The cost is that a sparse grid at 3 × 1,000 (10^9 positions, about 10^5 occupied in ADR + 008's fixture) sends mostly empty windows unless the client asks for skips first. + +Both cases need the same maintained structure. They differ only in who calls it, and in +whether a `project` result is already pruned. Prune would be a request parameter in the +first case (ADR 008's flagged reading 1) and a client policy in the second. + +**"As absolute as possible", and what still moves a position.** The user, in chat: "But +the top left should be as absolute as possible" (relayed on 2026-10-05; on no PR). *The +rest of this paragraph is the author's analysis.* A cross-product position does not move +when occupancy changes, pruning included. **It does move when an item is inserted, +removed or reordered** in a category's own order. Positions are mixed-radix numbers over +the items of each level, so inserting item j at level L shifts every later item of L, and +changes the radix that every outer position is multiplied by. No position over a +changing item list is fully absolute. Three things follow: +- **Deltas are keyed by identity, and positions are attached when sent.** An output + element names its row and column by tuple identity: item ids under ADR 007, and names + until then. The host attaches the absolute position from the current item order when it + sends the element. A delta keyed by position alone would go stale with the next item + insert. +- **An item edit is sent as an axis change, not as moved rows.** "Item inserted at index + j of level L", or removed, or moved. A client re-derives every position it holds + arithmetically, at O(levels) per position, and its top-left can be re-anchored to the + same tuple. That is the cursor's identity rule from ADR 008 clause 11 applied to the + window. Resending every row whose position moved would cost O(rows after j). +- **The cost is rank and select over each category's items.** "The position of item x" + and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` + first, a B-tree when measured, its question 16). A tuple's position is O(levels × log + items). The ordered occupancy index must compare tuples by **current** item position, + not by a position stored at insert time. With item grain (clause 7) its keys are item + identities, so an item insert re-orders nothing in it, and comparison goes through the + order-statistic lookup, at O(levels × log items) per comparison. That is **not + measured**. Question 20. + +**A pruned answer may arrive as a stream.** The user, in chat: "After pruning we might +have to stream rows and columns to attempt to fill out the full grid" (relayed on +2026-10-05; on no PR). *The author's analysis:* under model-side pruning, `project` scans +forward from the absolute top-left over each axis's occupancy until it has the requested +number of occupied rows and columns, or reaches the end of the axis. It sends the rows +and columns it finds as it finds them. What the circuit must support: +- **Skipping empty space in O(log n + k).** "The next k occupied tuples at or after a + position" over the ordered occupancy index. It must never walk empty cross-product + space, which at 3 × 1,000 is mostly empty. +- **The absolute position of every row and column sent**, so that a client can place it, + and a later delta can name it, without counting. +- **Bounded work per slice.** The fill is a job under ADR 008 clause 11's runner when k + is large, and it can be resumed from the last absolute position it sent. So the only + state a fill keeps is `(request, fill front)`, which is "as stateless as possible" in + the user's words. A lost fill restarts from the client's last known position. +- **Changes during a fill are identity-keyed deltas, stamped by step** (T3). Suppose a + row becomes occupied between the top-left and the fill front after it was passed. The + subscriber receives it as an ordinary occupancy delta carrying its absolute position, + so its window now holds one row more than asked, and it drops or keeps the extra row by + client policy (Q42). A row that becomes occupied ahead of the front is simply found by + the fill. Each chunk of the fill is a delta to the window's Z-set, stamped like any + other. So a fill and a live update are one stream, which keeps the in-process and the + remote consumer the same (T6). + +- **Rows and columns fill independently** (the author's, on ADR 008's question 59(c)). + This ADR's occupancy is kept **per axis**: a count per row tuple and a count per + column tuple (clause 9). That matches today's `prune_empty`, which drops a row only + when it is empty across the whole grid, and prunes the columns against the same + unpruned grid (`layout.rs:752-757` at `50e1314d`). So a row occupied only in a column that falls + outside the window is still an occupied row, and the row fill never waits on the + column fill. A joint rule (a row occupied *within the requested columns*) would need a + two-dimensional count per (row tuple, column tuple). That is O(occupied cells) more + state, and it would change today's pruning, so it is not proposed. + +Whether a pruned response is streamed or returned whole, and what a client sees while a +fill is incomplete, is question 21. + +**What network transparency requires** (T1 to T6, the author's): +- **T1. Results are plain data.** Every element on an output stream is a value the + protocol crate can serialize, with no `Rc`, no borrowed store key and no closure. Facts + at `50e1314d`: `CellValue` is `Serialize`, with numbers going through `wire_f64` + (`crates/improvise-core/src/model/cell.rs:86-118`). `CellKey` is **not**: it derives only + `Debug, Clone, PartialEq, Eq, Hash` (`cell.rs:10`). So an output element names a cell by + the request's row and column tuples, or by cross-product positions. Where a write may + follow, it also carries ADR 002's surrogate (`CellId`). It never names a cell by + `CellKey`. +- **T2. One identity on both sides of the wire.** A consumer across a wire must + consolidate exactly as an in-process one does. **On NaN they would not.** Clause 3's + delta identity is bit identity, under which two NaNs with different payloads differ. But + the wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114` at `50e1314d`), and the + projection compares values with `same_on_the_wire` (`crates/improvise-protocol/src/render.rs:173-180` + at `50e1314d`; `context/code/protocol.md` gives the reason). So an output delta + `−(c, NaN₁) + (c, NaN₂)` is non-empty in process and consolidates to nothing after a + round trip. **Proposed:** output streams are consolidated under the wire's identity + (every NaN is one value, and every other number is its bits) before they leave the host. + The record tier keeps bit identity, as it does today. Question 15. +- **T3. Every output change is stamped.** A delta carries the step that computed it, as + `(document_generation, step)`. The circuit steps once per model effect (clause 3), and + ADR 001 already puts batches in one total order. A consumer that has applied step n + drops anything at or below n, so a duplicate does no harm, and it treats a gap as a + loss, which a fresh snapshot answers. What is published is the **batch's** consolidated + delta, and only once the batch is accepted. A refused frame is restored (clause 11) and + publishes nothing. +- **T4. A snapshot, then deltas.** A subscription starts with the integrated result at + step n: the window as a Z-set of current values, each with weight +1. Deltas from step + n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current + result. `GridCache::folded(&cache.seed()) == *cache` states the same property for + today's bootstrap compaction (gotcha 32). +- **T5. Order holds across streams, and coalescing happens within a stream.** A slow + consumer's pending deltas on one stream can be **summed** into one delta. Z-set + addition is commutative and associative, so the sum is exact, and the queue is bounded + by the size of the result, not by the number of steps. When the summed delta outgrows + the snapshot, a snapshot replaces it. That is the backpressure, and nothing is dropped + silently. **Across streams, order stays load-bearing**, as gotchas 33 and 43 require: + the occupancy change comes before the cells it places, and the cells come before the + basis that describes them. This does not break gotcha 42's "never coalesce". That rule + is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple + identity, or by a stable cross-product position, is not positional in that sense. +- **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives + the same result and delta types as a remote client. Its transport is a function call. + That is what makes the requirement testable: run one edit sequence through two + consumers, one in process and one behind a serialize and deserialize round trip, and + require their integrated results to be equal under T2's identity. A TUI shortcut that + read operator state directly would be a second read path, which is the derivation-twice + defect (design-principles §5). + +Weights on the output are ±1, because a result is a set of current values. A consolidated +weight that does reach past 2^53 follows the user's bigint-as-string rule (Q36). + +**Where the wire form lives** (the author's). `zset` stays dependent on `std` alone (clause 1). +The wire types live in `improvise-protocol`, with the one envelope pair (gotcha 42). A +consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that boundary. +The alternative is a `serde` feature on `zset`. The author leans to the conversion, +because what crosses is the projected result, not the crate's own element type. + +**How stateless the host can be.** The user: "as stateless as possible". These are the +options of question 13, and ADR 008's question 52 asks the push half of the same thing. +- **(a) Pure fetch.** The host keeps nothing per client. A client calls + `project(request)`. It learns its result may be stale from a change notice addressed to + no one: the accepted batch's step. It then refetches when its own policy says to (Q42). + Each refetch costs O(visible). +- **(b) Soft-state subscriptions** (the author's leaning). A request may ask to be kept + current. For each subscription the host keeps the request and the last step it sent, + and nothing else: no copy of the window, no cursor and no viewport. After each accepted + batch it reads the delta of each subscribed shape, restricts it to the request's window, + and sends it under T3 to T5. **The table is a cache.** If it is lost (a restart, an + eviction, a reconnect), a client resumes by resending its request and the last step it + applied, and the host answers with a snapshot. The host could instead keep a short ring + of past deltas per shape; the author's lean is no ring at first. So the user's "recent + projections + clients" are this table plus clause 7's shape cache, and **both can be + rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). +- **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and + `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, + `:822` and `:981` at `50e1314d`). It keeps each client's `ViewState` and grid on the host + and diffs against them. ADR 008 lists it as obsolete under the split, and this ADR does + not propose it. + +A shape is kept while some subscription or recent request names it. Eviction drops its +operators, and the next request rebuilds them as a job (clause 7). How long "recent" lasts +is a cost the user has not set (question 14). + +**What a client must hold to resume:** its request, the last step it applied, its +integrated window, and its basis (ADR 001). The host holds nothing that a client needs +back. + +**The basis.** ADR 001's basis is published with the result it describes, and after it +(gotcha 43). The host builds it from the request's window (ADR 008, lens row 18), not from +a mirror. `published_basis` (`project.rs:585` at `50e1314d`) already takes the `GridCache` +it describes, not a cursor. + +### 13. Incremental max width (the user's question on ADR 008) + +The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" +**This clause is the author's analysis, and none of it is measured.** The user's decision +on widths stands as ADR 008 records it: (b), sticky per session, eased by hysteresis, +taken over the rows realized and held by the view layer (ADR 008 clause 4). + +**DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. +When the widest value is retracted, the new maximum cannot be computed from the old +maximum and the retraction. So it belongs to clause 4's **integrated** class, beside `Min` +and `Max`: each group keeps its integrated multiset and reads off the last key. A *top-k* +(keeping only the k widest) holds less state, but it is wrong once the top k have all been +retracted, and then it needs a rescan. So it is not proposed. + +**For widths the multiset is small.** The group is one column of a projection shape, +together with its number format, since a width is measured on formatted text. The values +are display widths, which are small integers. So the state is a histogram from width to +count, per column. It is O(distinct widths) per column, whatever the number of rows, and +an update costs O(log d), where d is the number of distinct widths. + +**The cost is upstream, in what feeds the histogram.** Its input is the display width of +**every evaluated cell in the column**, under the request's number format. That needs +three things: +- **A delta of evaluated cells over the whole column**, not a window. Clause 4's group + aggregates and clause 5's formula re-evaluation already produce an output delta for + every unfolded cell that moves, with its old value as the retraction. So unfolded rows + need no new state. **Fold rows are not covered.** Under item grain (clause 7) a + collapsed group's value is combined when it is read, not maintained, so its width would + need the fold maintained per column at the visible grain. +- **Formatting every changed cell on the host**, on the write path: `format_f64` and a + display width for each changed cell. Each one is cheap, but it would be the first + display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value + back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But + today the host is asked for widths only over the rows requested (ADR 008 clause 4), and + this would widen that to whole columns. +- **A rebuild whenever the format changes.** `number_format` is a parameter of the view + definition, and a rebuild costs O(occupied rows) per column, as a job. + +**What it would buy, set against what the user decided.** A maintained whole-column width +is ADR 008's question 2 option (c), "size each column from its whole contents", which the +user did **not** choose. Under (b) a window's widths are an O(visible) computation over +cells the window already evaluates, so maintaining them incrementally adds nothing. So +the real question is whether the user wants (c) after all, now that its state is cheap. It +would remove the jitter at its source and make hysteresis unnecessary, at the upstream +cost above. That is question 12. + +### 14. Resident state, and the store-memory requirement + +The user, on ADR 008: "We have to be careful here to minimize memory requirements of the +store: offloading inactive regions to disk (maybe mmap or similar), +double-delta/gorilla/lz4 compression, etc." (#308, comment 4171481179). And then, on ADR +008's question 17: "Let's defer this as recommended." (4188271710). So the store's memory +gets its own ADR, once this ADR's operator state has been measured at ADR 008's fixture. +**This clause withdraws the earlier non-goal** ("Persistence, spilling to disk, and +threads", clause 1), and it says what this ADR must not rule out. + +**What this ADR keeps resident, beside the store.** This is the author's inventory. The +orders come from the clauses and are not measured. + +| state | clause | order | +|---|---|---| +| the integrated base that a rebuild reads | 7 | O(cells), if it is kept as a copy | +| clause 5a's per-record stage | 5a | O(cells of the records an aggregate over an expression reads), as written | +| multisets for `Min`, `Max` and text | 4 | O(cells) of the measures that need them | +| `ExactSum`, per summed group | 8 | 568 bytes per group | +| occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | +| the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | +| the subscription table | 12 | O(request) per subscription | + +**Proposed by the author:** +- **The store is the integral, so it is not copied.** The integrated base a rebuild reads + is `DataStore` itself, which is I(cell delta) by definition. Clause 5a's per-record + stage keeps each record's cells only to compute the retraction `I(x)₋₁`. That equals + `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. So + the stage can read the store and keep no copy of the values. It still needs each cell's + store sequence (clause 5a), one integer per cell, which `DataStore` would keep beside + the value rather than in a second map. This is **not measured**, and the retraction of + the order key has to be checked against it (question 17). +- **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a + range, and apply a delta. The first implementation keeps it in memory, and the + store-memory ADR can page or compress it without changing any operator. +- **Shapes are evicted** (clause 12). So per-shape state is bounded by the shapes in + recent use, not by every saved view. +- **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, + `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and + O(groups), and the store-memory ADR has to count them. + +**The frame budget.** ADR 008 clause 11 sets it, and the user accepted the recommendation +(ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms +slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is +a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration +step 4 must. + ## Alternatives considered **Feldera's `dbsp` crate.** What was verified on 2026-09-24: @@ -1038,6 +1577,12 @@ displayed path until the differential test holds. accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. +8. **The delivery stream** (clause 12), on the answers to questions 13, 15, 19, 20 and + 21. It brings the output types in `improvise-protocol`, the step stamp, + snapshot-then-delta, the order-statistic queries over occupancy, and the test with two + consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff + only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, + the cost of a step and of a rebuild (clause 14). **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is question 10. improvise-swre stays independent and is useful under either backend. @@ -1055,6 +1600,9 @@ longer a prerequisite of step 6. - ADR 008 gets occupancy, the occupancy delta and column totals without a second evaluator. - Live sources that can report changes plug in without touching the evaluator. +- One delivery path serves the TUI in process and every remote client (clause 12), and + the host keeps no per-client UI state: only a subscription table that can be rebuilt + from the requests. **Bad.** @@ -1072,25 +1620,33 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- The operator shape is re-read after every model effect: O(items) per view, a cost the +- The operator shape is re-read after every model effect: O(items) per live shape, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. +- Per-shape state lives only as long as requests name it. A shape that comes back after + eviction is rebuilt as a job, and the client waits through its pending state. +- The output identity (every NaN is one value) differs from the record tier's (bit + identity), as T2 explains. Two identities need stating everywhere a delta crosses. +- A position over a changing item list is never fully absolute. An item insert moves + every later position, and is sent as an axis change (clause 12). ## Open questions Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. +the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered +none of questions 1 to 11 on #309.** Questions 12 to 21 were added by the 2026-10-05 +revision, and none is renumbered. -1. **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? +1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. -2. **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on +2. *[Open.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on `zset` and `improvise-core`), or inside improvise-core? *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? +3. *[Open.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums until then. - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in @@ -1117,35 +1673,35 @@ the user decided", as ADRs 004 and 007 did. that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - `Avg`'s one rounding must be stated in step 6. -4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported +4. *[Open.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to change, change it in both backends at once, as its own decision. -5. **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by +5. *[Open.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by today's bounded fixed point, restricted to their component, rather than by any operator in the crate? *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the cells a guarded cycle leaves convergent. The crate needs no recursion operator. -6. **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? +6. *[Open.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. **Page selection as a slice.** Should the group key include the page categories, so +7. *[Open. ADR 008's flagged reading 1 makes the page selection a parameter of the request, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". -8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, +8. *[Open.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? *Recommendation:* yes, chosen per document. Integrated state is proportional to the source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` +9. *[Open. Clause 14 now weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` snapshot that `Host::mutating_frame` already takes? *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying `−d` restores integrated values but not the history `delay` and `differentiate` @@ -1154,13 +1710,13 @@ the user decided", as ADRs 004 and 007 did. today. A true rewind would need its own design and an observational-equivalence test over every stateful operator. -10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps +10. *[Open.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. -11. **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). +11. *[Open, and affected by question 16. Under item grain a reorder of an axis category's items rebuilds nothing, so the recommendation's premise (key by position and rely on `view_shape`'s rebuild) no longer holds. Keying by item identity and reading the order at read time, which clause 12 already needs (question 20), becomes the natural form. The recommendation is left as written until the user answers both.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering category's item position, with that category's item order added to the operator shape so that a reorder rebuilds them? Or should they key by stable item identity (ADR 007), and @@ -1174,3 +1730,89 @@ the user decided", as ADRs 004 and 007 did. no widening at all, and ADR 008's text and cross-references hold unchanged. If the user wants hidden items ordered too, the `measure_agg` part must also cover their positions. + +12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). + Should the host maintain each column's whole-column maximum display width? That is + ADR 008's question 2 option (c), and its state would now be a small histogram per + column. *Recommendation (the author's):* no, not now. The user decided on (b) with + hysteresis, and under (b) a window's widths cost O(visible) and need no maintenance. + The maintained form also needs fold values maintained per column, and formatting on + the write path. Revisit it if a measured workload shows jitter that hysteresis does + not cure. + +13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per + client** (clause 12; ADR 008's question 52 asks the push half). *Recommendation (the + author's):* (b), soft-state subscriptions that can be rebuilt from the request and the + last step, with no copy of a client's window on the host. When its table is lost, (b) + falls back to (a). (c) is obsolete under ADR 008's split. One of the two ADRs should + hold this question, and the other should point to it. + +14. *[New.]* **What keys operator state, and how long a shape lives** (clause 7). Is it + keyed per projection shape, cached and evicted (the author's)? Or per saved `.improv` + view only, so that a request whose shape is unsaved (an unsaved prune or hide, ADR + 008's question 51) is served without incremental state? *Recommendation:* per shape, + cached, and evicted once no request has named it for a while. The time is an + implementation number. An unsaved shape is then as incremental as a saved one, which + keeps "pruning shouldn't infect every client" from costing an eager path. + +15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate + under the wire's identity, where every NaN is one value, while the record tier keeps + bit identity? *Recommendation:* yes. Otherwise a remote consumer and an in-process + one disagree when only a NaN's payload changes. + +16. *[New.]* **ADR 008's three requests** (ADR 008, "What revision 4 asks of ADR 009"): + occupancy and the per-column aggregates kept at item grain, a crossing count per + step, and rebuilds as steppable jobs. *Recommendation:* adopt all three, as clause 7 + is now written. This changes clause 7's list of what rebuilds, removes the rebuild + from gotcha 43's worked case, and undercuts question 11's premise. + +17. *[New.]* **Resident state** (clause 14). Should this ADR withdraw "spilling to disk" + as a non-goal, keep no second copy of the store (the store is the integral), and put + operator state behind one interface that the store-memory ADR can page? + *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the + order the user's answer to ADR 008's question 17 already sets. + +18. *[New.]* **Formulas the per-record map cannot express** (clause 5a). ADR 008's + question 15 sends them to a fallback over the occupied contexts, never to + `Expansion::members`; the user's answer was "the recommendation is fine.". Two parts: + - Does this ADR's `Eager` adopt the same rule, so that `Eager` and `Incremental` + agree and this ADR's differential test needs no carve-out? + - Do the formulas that target a category other than `_Measure` take the same + fallback, so that they do not block ADR 008's retirement of `Expansion::members`? + + *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's + question 1 and question 15 cases, as a stated change of behaviour. + +19. *[New. The user is "still ambivalent".]* **Which side of the wire prunes** (clause 12; + the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, + and one ADR should hold it). On the model side, `project` + returns occupied rows and columns, and the circuit answers next-occupied and + rank/select queries for it. On the client side, `project` returns the cross-product + window together with a skip structure, and the client asks for the queries itself. + *Recommendation (the author's):* the model side, with prune as a request parameter. + A sparse grid at 3 × 1,000 then never ships empty space, and a client stays thin. + Either way, the circuit maintains the same structure (clause 12). + +20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left + should be as absolute as possible"; ADR 008's question 58 asks the same from the + request side). A cross-product position stays fixed when + occupancy changes, but it moves when an item is inserted, removed or reordered. Should + deltas be keyed by tuple identity, with absolute positions attached when they are + sent, and should an item edit be sent as an axis change, so that clients re-derive + their positions arithmetically? *Recommendation:* yes. It costs rank and select over + each category's items (ADR 008 clause 10), and comparisons in the occupancy index at + O(levels × log items) each, which is not measured. It is compatible with ADR 008's + question 58 recommendation, which makes a tuple of item ids the primary form of the + top-left. Whether hidden items belong to the absolute space is ADR 008's question 58. + Under item grain (clause 7) this ADR serves either answer, because hiding is a remap. + +21. *[New.]* **A pruned answer as a stream** (clause 12; the user: "After pruning we + might have to stream rows and columns to attempt to fill out the full grid"). Should + `project` under pruning fill forward from the absolute top-left, as a sliced job that + can be resumed from its fill front, and send each found row and column as a stamped + delta? Should a row that becomes occupied behind the front arrive as an ordinary + occupancy delta, and leave the window one row larger than asked, for the client's + policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one + stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 + asks the same, and asks in its part (c) whether rows or columns fill first. Clause 12's + answer is that the two fill independently (the author's). From 4851eec8a6174f50eacf1335c6ea8fc0d5f73dea Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:56:02 -0700 Subject: [PATCH 026/106] docs(adr): ADR 009 records window-relative pruning and the restricted skip query (improvise-felo.4) The user's chat statements make pruning a filter over displayed values. Clause 12 now names the query a client's fill loop needs and three ways to answer it; question 19's recommendation is revised and question 22 added. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 107 ++++++++++++++++++---- 1 file changed, 88 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index eaa01619..bd29138a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -5,7 +5,7 @@ are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision (PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 18. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -104,6 +104,11 @@ None of this answers this ADR's own questions 1 to 11. Clause 12, question 20 (ADR 008's question 58). - "After pruning we might have to stream rows and columns to attempt to fill out the full grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). +- "Pruning is just a filter applied to full columns a rows, right? Hide a column/row in + the grid if it's all null. That sounds like a UI side concern because that's where the + visible grid lives" and "The point of pruning is to make the displayed values more + dense, so a value being present outside the viewport doesn't matter." (chat, relayed on + 2026-10-05). Clause 12, question 22. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1307,15 +1312,55 @@ and columns it finds as it finds them. What the circuit must support: other. So a fill and a live update are one stream, which keeps the in-process and the remote consumer the same (T6). -- **Rows and columns fill independently** (the author's, on ADR 008's question 59(c)). - This ADR's occupancy is kept **per axis**: a count per row tuple and a count per - column tuple (clause 9). That matches today's `prune_empty`, which drops a row only - when it is empty across the whole grid, and prunes the columns against the same - unpruned grid (`layout.rs:752-757` at `50e1314d`). So a row occupied only in a column that falls - outside the window is still an occupied row, and the row fill never waits on the - column fill. A joint rule (a row occupied *within the requested columns*) would need a - two-dimensional count per (row tuple, column tuple). That is O(occupied cells) more - state, and it would change today's pruning, so it is not proposed. +**Pruning is a filter on the values displayed (the user, later in chat).** "Pruning is +just a filter applied to full columns a rows, right? Hide a column/row in the grid if it's +all null. That sounds like a UI side concern because that's where the visible grid lives" +and "The point of pruning is to make the displayed values more dense, so a value being +present outside the viewport doesn't matter." (both relayed on 2026-10-05; on no PR). +**This is a change from today's code**, and it is recorded here, not decided. Today, +`prune_empty` drops a row only when it is empty across the **whole** grid, and prunes the +columns against the same unpruned grid (`layout.rs:752-757` at `50e1314d`). In the user's +words, emptiness is judged over the **displayed** values, so a row that holds a value only +in a column outside the window is hidden. *The rest of this paragraph is the author's +analysis.* +- **What the model then owes a client** is not a pruned axis. It owes windows (an + absolute top-left and dimensions) and a **restricted skip query** for the client's fill + loop: "the next k rows, from absolute position p, that hold a value in at least one of + these columns", and the same with rows and columns swapped. The question is about a set + of columns, not about the whole grid, so a per-axis count cannot answer it. A row can be + occupied overall and still be empty in every column the client shows. +- **Three ways to answer the query**, all unmeasured: + - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then + test each candidate row against the column set by lookup. The state is the per-axis + counts clause 9 already keeps. The worst case scans many rows that are occupied + overall but empty in the column set, at O(|columns|) lookups each. + - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that + hold a value in it. The query is a k-way merge of |columns| sorted lists from p, at + O(|columns| × log n + k × |columns|). The state is O(occupied cells at grid grain), + the two-dimensional count clause 9 does not keep. It is maintained from the cell delta + and from formula stems (clause 5's stem rule decides when a formula cell holds a + value). A collapsed group's column merges its members' lists when it is read. + - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, + intersected with the column set as the fill walks forward. It has the same state as + (ii), and its scan behaves like (i), with an exact test per row. + + The author leans to (ii): a window has tens of columns, so the merge is small, and it + never walks rows that are empty in the window. Its cost is a second O(occupied cells) + structure, beside the store (clause 14). +- **The filter has a fixpoint.** Rows are judged against the displayed columns, and the + columns against the displayed rows. Hiding a row can empty a column, and hiding that + column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records + it, in its questions 60 and later); the skip query only has to be cheap enough to call + once per iteration. +- **What this changes in this ADR.** If pruning is a client filter over the window, + nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The + pruned axis, its ordered index and "row i of n" are not needed either, because a + window-relative filter has no global n. What remains model-side is **which cells hold a + value**, including formula stems, which only the model can evaluate. ADR 008's question + 1 ("go with occupied") is about that, and it still applies, to the cells of a window. Its + question 11 ("prune empty should be the default") becomes a default for a client's + display. Whether clause 9's occupancy should be restated as the index of (ii), and + whether ADR 008 clause 3's `Occupied` axis survives, is question 22. Whether a pruned response is streamed or returned whole, and what a client sees while a fill is incomplete, is question 21. @@ -1577,8 +1622,8 @@ displayed path until the differential test holds. accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. -8. **The delivery stream** (clause 12), on the answers to questions 13, 15, 19, 20 and - 21. It brings the output types in `improvise-protocol`, the step stamp, +8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to + 22. It brings the output types in `improvise-protocol`, the step stamp, snapshot-then-delta, the order-statistic queries over occupancy, and the test with two consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, @@ -1634,7 +1679,7 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 21 were added by the 2026-10-05 +none of questions 1 to 11 on #309.** Questions 12 to 22 were added by the 2026-10-05 revision, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -1783,15 +1828,16 @@ revision, and none is renumbered. *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's question 1 and question 15 cases, as a stated change of behaviour. -19. *[New. The user is "still ambivalent".]* **Which side of the wire prunes** (clause 12; +19. *[New. The user was "still ambivalent", then said in chat that pruning "sounds like a UI side concern"; see question 22. Not confirmed as a decision.]* **Which side of the wire prunes** (clause 12; the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, and one ADR should hold it). On the model side, `project` returns occupied rows and columns, and the circuit answers next-occupied and rank/select queries for it. On the client side, `project` returns the cross-product window together with a skip structure, and the client asks for the queries itself. - *Recommendation (the author's):* the model side, with prune as a request parameter. - A sparse grid at 3 × 1,000 then never ships empty space, and a client stays thin. - Either way, the circuit maintains the same structure (clause 12). + *Recommendation (the author's), revised after the user's later chat:* the client + side, as the user leans. The model offers windows and question 22's restricted skip + query, so a sparse grid still never ships empty space. *As first written*, this + recommended the model side; that is withdrawn. 20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left should be as absolute as possible"; ADR 008's question 58 asks the same from the @@ -1814,5 +1860,28 @@ revision, and none is renumbered. occupancy delta, and leave the window one row larger than asked, for the client's policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 - asks the same, and asks in its part (c) whether rows or columns fill first. Clause 12's - answer is that the two fill independently (the author's). + asks the same, and asks in its part (c) whether rows or columns fill first. Under the + user's later chat (question 22), the fill loop is the **client's**: it calls the + restricted skip query once per step of its row and column fixpoint. The host then + keeps no fill state at all, which is more stateless than the sliced host job this + question first described. + +22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause + 12; the user, in chat: "Pruning is just a filter applied to full columns a rows ... + That sounds like a UI side concern" and "a value being present outside the viewport + doesn't matter"). This changes today's semantics, which prune against the whole + grid. Three parts: + - (a) Should the circuit offer a **restricted skip query**, "the next k rows from + absolute position p with a value in at least one of these columns", and the same for + columns? How should it answer: (i) per-axis occupancy as a pre-filter, (ii) a column + index of occupied rows, merged, or (iii) a row index of occupied columns? + - (b) Should clause 9's whole-extent, per-axis occupancy be **restated** as that + index, and ADR 008 clause 3's `Occupied` axis and its "row i of n" be dropped, + since a window-relative filter has no global count? + - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty + should be the default") need restating? Question 1 would become the rule for which + cells of a window hold a value, and question 11 a client's display default. + + *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms + that pruning is the client's. (c) restate both in those terms, in ADR 008, since they + are ADR 008's answers. From 09938f0064f2d55e8b075c68c367537b0310b074 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:56:49 -0700 Subject: [PATCH 027/106] docs(adr): ADR 009 analyses the client's fetch, prune and fill loop (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index bd29138a..8e2a12ea 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1352,6 +1352,38 @@ analysis.* column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records it, in its questions 60 and later); the skip query only has to be cheap enough to call once per iteration. +- **The loop the user describes, and what makes it converge cheaply.** The user, in chat: + "The problem pruning solves is visualizing sparse datasets: screens that are mostly + empty are useless and so you drop out categories from rows and columns until every + visible row or column has at least one value on screen." (relayed on 2026-10-05; on no + PR). The criterion is a fixpoint on the displayed grid. *The author's analysis:* a client + alternates three steps. It fetches a window. It prunes, removing every row with no + value in the visible columns and every column with no value in the visible rows, until + nothing changes. It fills, asking for the next rows (or columns) past its last absolute + position that have a value in the columns (or rows) still visible. Three things the + circuit must answer for that loop: + - **"Any value in row r among columns S", and its batched form**, the next k such rows + from p. With index (ii) the batched form costs O(|S| × log n + k × |S|) per call, keeps + no state per request, and is recomputed for each request. That is the "as stateless + as possible" side of it. Without an index, each candidate row costs |S| store lookups, + and a sparse dataset puts most candidates in empty space. That is exactly the case + pruning exists for, so the index is not optional at 3 × 1,000. + - **Maintenance is incremental and cheap per write.** A cell that starts or stops + holding a value inserts or removes one entry in one column's list, at O(log n). Index + (ii) is a set of `(column, row)` pairs, so it is itself a Z-set fed from the cell delta + and from formula stems. Its delta is "row r gained or lost a value in column c". + - **The answers are friendly to deltas.** A client subscribed to a window can be sent + just those `(row, column)` changes that fall in its window's columns (clause 12's + subscription, restricted to S). It re-runs its own prune step locally, and it fills + again only when a row or column has been removed. So the host never re-runs a + client's fixpoint. + + Whether the loop always terminates is **not proven**. Within a fixed candidate set, + iterated removal reaches the unique greatest fixpoint, as a 1-core of the bipartite + row-column graph does. The fill only moves forward through absolute positions, so it + adds finitely many candidates before the axis ends. A client that bounds its fills per + frame stays within ADR 008's frame budget whether or not the loop has converged. That + bound is a client policy (Q42). - **What this changes in this ADR.** If pruning is a client filter over the window, nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The pruned axis, its ordered index and "row i of n" are not needed either, because a @@ -1882,6 +1914,9 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. + The user's later statement makes the criterion a fixpoint on the displayed grid, + reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is + what each fill step calls. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From 530ec2941bf4d0e8b84893c55aea4145fd5e1434 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:57:34 -0700 Subject: [PATCH 028/106] docs(adr): ADR 009 reads ADR 008 revision 20 and cross-references its questions 60-65 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 8e2a12ea..dc461009 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 19** -(PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 20** +(PR #308, read at `b0dc277c`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `69f0a7c6`, its revision 19) · +reads it at `b0dc277c`, its revision 20) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 19 of `context/adr/008-virtualized-grid.md` on PR #308, at `69f0a7c6`. Clauses 3, +revision 20 of `context/adr/008-virtualized-grid.md` on PR #308, at `b0dc277c`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 and 19, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 20, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1080,7 +1080,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 19 at `69f0a7c6`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 20 at `b0dc277c`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1379,7 +1379,9 @@ analysis.* client's fixpoint. Whether the loop always terminates is **not proven**. Within a fixed candidate set, - iterated removal reaches the unique greatest fixpoint, as a 1-core of the bipartite + iterated removal reaches the greatest fixpoint, which is unique (the union of any two + sets with the property has it too). ADR 008's question 61 says a fixpoint "may be + non-unique"; that holds for other readings, not for this one. Iterated removal reaches it, as computing the 1-core of the bipartite row-column graph does. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A client that bounds its fills per frame stays within ADR 008's frame budget whether or not the loop has converged. That @@ -1914,6 +1916,12 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. + ADR 008 revision 20 asks the same from its side: its question 62 (the skip query), + question 65 (whether "occupied" stays whole-extent) and question 61 (one pass or a + fixpoint). Its question 65 recommends keeping per-axis, whole-extent occupancy and + using it only to skip. Under (i) above, that skips only rows that are empty + everywhere, not rows that are empty in the displayed columns. So it does not answer + the restricted query by itself. The user's later statement makes the criterion a fixpoint on the displayed grid, reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is what each fill step calls. From 037f795473133258e2b326ae11a2e379f63b2082 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:59:09 -0700 Subject: [PATCH 029/106] docs(adr): ADR 009 corrects the single-window fixpoint claim and reads ADR 008 revision 21 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 ++++++++++++----------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index dc461009..48f2f643 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 20** -(PR #308, read at `b0dc277c`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 21** +(PR #308, read at `3b70a0f1`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `b0dc277c`, its revision 20) · +reads it at `3b70a0f1`, its revision 21) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 20 of `context/adr/008-virtualized-grid.md` on PR #308, at `b0dc277c`. Clauses 3, +revision 21 of `context/adr/008-virtualized-grid.md` on PR #308, at `3b70a0f1`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 20, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 21, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1080,7 +1080,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 20 at `b0dc277c`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 21 at `3b70a0f1`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1347,11 +1347,13 @@ analysis.* The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) structure, beside the store (clause 14). -- **The filter has a fixpoint.** Rows are judged against the displayed columns, and the - columns against the displayed rows. Hiding a row can empty a column, and hiding that - column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records - it, in its questions 60 and later); the skip query only has to be cheap enough to call - once per iteration. +- **The fixpoint comes from the fill, not from one window.** *As first written*, this + item said that hiding a row can empty a column. **That was wrong.** A row dropped for + having no value among the window's columns contributes no value to any of those + columns. So on a fixed window, one pass that drops the rows and columns empty within + the window already is the fixpoint (ADR 008 revision 21, question 61, makes the same + argument). The iteration comes from the fill step, which brings in rows and columns from + outside the window. ADR 008's question 66 records that loop. - **The loop the user describes, and what makes it converge cheaply.** The user, in chat: "The problem pruning solves is visualizing sparse datasets: screens that are mostly empty are useless and so you drop out categories from rows and columns until every @@ -1380,9 +1382,8 @@ analysis.* Whether the loop always terminates is **not proven**. Within a fixed candidate set, iterated removal reaches the greatest fixpoint, which is unique (the union of any two - sets with the property has it too). ADR 008's question 61 says a fixpoint "may be - non-unique"; that holds for other readings, not for this one. Iterated removal reaches it, as computing the 1-core of the bipartite - row-column graph does. The fill only moves forward through absolute positions, so it + sets with the property has it too, as ADR 008 revision 21's question 61 also says). On + a fixed window that takes one pass. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A client that bounds its fills per frame stays within ADR 008's frame budget whether or not the loop has converged. That bound is a client policy (Q42). @@ -1916,9 +1917,9 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. - ADR 008 revision 20 asks the same from its side: its question 62 (the skip query), - question 65 (whether "occupied" stays whole-extent) and question 61 (one pass or a - fixpoint). Its question 65 recommends keeping per-axis, whole-extent occupancy and + ADR 008 revision 21 asks the same from its side: its question 62 (the skip query), + question 65 (whether "occupied" stays whole-extent), question 61 (one pass or a + fixpoint) and question 66 (the fill loop). Its question 65 recommends keeping per-axis, whole-extent occupancy and using it only to skip. Under (i) above, that skips only rows that are empty everywhere, not rows that are empty in the displayed columns. So it does not answer the restricted query by itself. From 6305fa3bc7de606bfd2bb30763b1d718c15b3035 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:05:38 -0700 Subject: [PATCH 030/106] docs(adr): ADR 009 analyses forward-only, right-and-down pruning growth (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 48f2f643..4d749786 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1347,6 +1347,38 @@ analysis.* The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) structure, beside the store (clause 14). +- **Growth is only rightward and downward (the user, in chat).** "We should be able to + optimize pruning on the right and down assumption of growth (truncated at the bottom + right corner of the axis space)" (relayed on 2026-10-05; on no PR). *The rest of this + item is the author's analysis.* The fill loop only grows away from the absolute + top-left, and it stops at the end of each axis. So every skip query is a one-directional + **successor** query: "the next row (or column) after position p with a value among this + set", bounded by the end of the axis. Pruning never needs a predecessor or a reverse + scan. Scrolling up or left is not a reverse fill. It is a **new request with a new + absolute top-left**, filled forward from there, and nothing from the old request has to + be walked backwards. What that simplifies: + - **The index options.** (ii)'s per-column sorted lists need only a seek (a lower + bound at p) followed by forward iteration. A k-way merge of forward cursors keeps + O(|S|) state and never steps back. Option (i)'s pre-filter and option (iii)'s per-row + test also walk forward only. So any ordered structure with a lower-bound seek and a + forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 + question 16). No order statistics (rank, or "row i of n") are needed for pruning. + They remain only for a client that wants a scrollbar over an unpruned axis, where + rank is arithmetic over the cross product anyway. + - **Maintenance.** An insert or removal at position q affects only fills whose front is + already past q, and only within their window. A fill whose front is before q finds the + change when it gets there. So a subscription needs a delta only for positions between + its top-left and its front, which is a range test against two absolute positions. + - **Resumable streams.** A fill's whole state is its request and its last absolute + position on each axis. A resume is a seek past that position, so a lost stream, a + reconnect or a sliced job (ADR 008 clause 11) all resume the same way, with nothing + else kept. Positions are stable under occupancy changes because they are + cross-product positions. An item insert before the front shifts them, which is + question 20's axis-change message: the client re-derives its front arithmetically. + - **The corner truncates the work.** The fill is bounded by the requested dimensions + and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × + |S|) per round, never a scan of empty space past the last occupied position. + - **The fixpoint comes from the fill, not from one window.** *As first written*, this item said that hiding a row can empty a column. **That was wrong.** A row dropped for having no value among the window's columns contributes no value to any of those @@ -1926,6 +1958,9 @@ revision, and none is renumbered. The user's later statement makes the criterion a fixpoint on the displayed grid, reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is what each fill step calls. + Under the user's right-and-down growth (clause 12), the query is a forward successor + query only, and so are all three options; no rank or reverse scan is needed for + pruning. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From de3b2bc3234c4ecf2cc559086be6aa42b7b57089 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:08:52 -0700 Subject: [PATCH 031/106] docs(adr): ADR 009 analyses predecessor search and a top-left on a pruned row (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 4d749786..c311be43 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1379,6 +1379,58 @@ analysis.* and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × |S|) per round, never a scan of empty space past the last occupied position. +- **Scrolling up or left, and a top-left that lands on a pruned row (the user, in + chat).** "The app is free to optimize fetching for up or left by just fetching the new + rows, until it finds one that has values in the viewport but, conceptually, scrolling up + or left would request a new viewport with a new top left corner. The requested top left + corner falling on a pruned row or column is an annoying edge case, maybe just fail the + fetch and retry?" (relayed on 2026-10-05; on no PR). The user's "maybe" is tentative, + and nothing here is decided. *The rest of this item is the author's analysis.* + - **The optimisation is a predecessor search.** Fetching only the new rows above the + window, and scanning upward until one has a value among the displayed columns, asks + "the previous row before p with a value in S". The forward-only queries above do not + serve it directly. There are two ways to serve it: + - **The client does it with forward requests.** It guesses an earlier top-left, + requests forward from there, and widens the guess if nothing qualifies. Each guess + costs one forward query, and the number of guesses grows with the gap. At the + fixture's density of 10^-4 a blind step of tens of rows usually finds nothing, so + it needs widening, about log₂(gap) guesses if each guess doubles the step. + - **The model offers a predecessor query.** It is cheap under every option. A sorted + `Vec` and a B-tree both give an upper-bound seek and a reverse iterator at the same + cost as the forward ones, O(log n) and then O(1) per step. Option (ii)'s per-column + sorted lists merge backwards exactly as they merge forwards. Options (i) and (iii) + walk backwards with the same per-row test. So offering it adds no state. It adds + only a second direction to the query surface, which the conceptual model (always + fill forward from a new top-left) does not need. *Author's lean:* offer it, because + it costs no state and saves the client a guessing loop. The user's to confirm. + - **The edge case: a top-left on a pruned (empty) row or column.** Listed neutrally: + - (a) **Fail, then retry** (the user's tentative "maybe"). The model answers with an + error. If the error names the nearest occupied successor (and predecessor), the + retry is one round trip. If it names nothing, the client must guess, as above. + - (b) **Snap forward, and report the actual top-left.** The model starts the fill at + the next occupied position at or after the requested one, and returns that + position in the response. No extra round trip. The client sees its window start + somewhere other than it asked. + - (c) **Snap toward the scroll direction.** Like (b), but snap backward when the + request came from an upward scroll. That needs the request to say which direction + it came from, which the request sketch does not carry. + + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the + ordered index, O(log n), in either direction, so (a) with named neighbours and (b) + cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap + to the next occupied row is on the order of 10^4 cross-product rows, so a client that + guesses blind under (a) would need about log₂(10^4) ≈ 14 doubling retries. **Unpruned + space is not an edge case at all**: a top-left on an empty position is answered with + empty cells. + - **How it fits identity-keyed deltas and resumable streams.** Under (b) and under (a) + with named neighbours, the response's actual top-left is an absolute position, and + the tuple there is an identity. So the stream's resumable state stays `(request, + actual top-left, last position)`, and a later delta names rows by identity whichever + top-left was asked for. A top-left whose row is pruned after the stream started is the + same case in time. A delta reports the row's removal, the client's window keeps its + absolute top-left, and the next fill skips past the empty position. That is (b)'s + behaviour without a new request. + - **The fixpoint comes from the fill, not from one window.** *As first written*, this item said that hiding a row can empty a column. **That was wrong.** A row dropped for having no value among the window's columns contributes no value to any of those @@ -1961,6 +2013,11 @@ revision, and none is renumbered. Under the user's right-and-down growth (clause 12), the query is a forward successor query only, and so are all three options; no rank or reverse scan is needed for pruning. + The user's later chat adds a client-side optimisation for scrolling up or left (a + predecessor search, which every option serves as cheaply as a successor search). It + also raises the edge case of a top-left that lands on a pruned row: fail and retry, + snap forward, or snap toward the scroll direction (clause 12). That edge case is + part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From 885e8eb4693e9d99f9e79fd65f4846c503ce067c Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:10:01 -0700 Subject: [PATCH 032/106] docs(adr): ADR 009 records the objection to snap-forward and argues the alternatives cannot stick (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 32 ++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index c311be43..099361e9 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1415,6 +1415,35 @@ analysis.* request came from an upward scroll. That needs the request to say which direction it came from, which the request sketch does not carry. + **The user objects to (b)** (chat, relayed on 2026-10-05; on no PR): "Snap forward is + going to cause issues with the viewport getting stuck". *The author's reading of the + reason:* a client scrolling up asks for a top-left just above its window. If that + position is empty, (b) snaps forward to the first occupied position, which is the + window's old top-left. So the window never moves past an empty stretch above it. (b) + is therefore set aside, and (a) and (c) remain. **Both need the predecessor query**: + (a) to name the nearest occupied position before the request, and (c) to snap backward. + So the author's lean to offer a predecessor query is stronger now, and it is still the + author's. + + **Neither (a) nor (c) can get stuck, and neither oscillates** (the author's argument): + - **(a) moves the client.** The error names the nearest occupied predecessor and + successor. A client scrolling up retries at the predecessor, which is strictly + before its old top-left, so each scroll step makes progress. A client scrolling + down retries at the successor, which is strictly after. Neither retry can land on an + empty position, because both are occupied by definition. One retry suffices, unless + a write empties that position in between; then the next error names new neighbours, + still in the same direction. + - **(c) moves in the direction asked.** It snaps backward for an upward request and + forward for a downward one, so the actual top-left moves in the direction of the + scroll, or stays put only when there is nothing further that way. + - **At the bounds.** At the top of the axis there is no predecessor. Under (a) the + error names the successor only, and says there is no predecessor. Under (c) an + upward snap with nothing above it returns the first occupied position (the top of + the occupied axis) and says it is the top. Either way the client learns it is at + the edge and stops scrolling, so it does not bounce between "fail" and "retry". + The bottom-right corner is symmetric. An axis with **no** occupied position + answers "empty axis" under both, and the window is empty. + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the ordered index, O(log n), in either direction, so (a) with named neighbours and (b) cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap @@ -2016,7 +2045,8 @@ revision, and none is renumbered. The user's later chat adds a client-side optimisation for scrolling up or left (a predecessor search, which every option serves as cheaply as a successor search). It also raises the edge case of a top-left that lands on a pruned row: fail and retry, - snap forward, or snap toward the scroll direction (clause 12). That edge case is + snap forward (which the user objects to: "the viewport getting stuck"), or snap toward + the scroll direction (clause 12). That edge case is part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they From 3698c68025e42edc5593275bb22a11aa28f9d3a5 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:12:18 -0700 Subject: [PATCH 033/106] docs(adr): ADR 009 records the user's galloping top-left search and revises the predecessor lean (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 ++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 099361e9..79255763 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1444,6 +1444,44 @@ analysis.* The bottom-right corner is symmetric. An axis with **no** occupied position answers "empty axis" under both, and the window is empty. + **The user's simpler procedure** (chat, relayed on 2026-10-05; on no PR), quoted + exactly as sent: "The simplest might be some kind of binary search for a new top left + corner: increasing jumps of the top left until you get a hit and then move down/right + until + You only have one new row column." *The author's reading:* the procedure belongs to the + client and uses forward requests only. The client moves its requested top-left up (or + left) in growing jumps until a request returns an occupied row before its old + top-left, then narrows back down (or right) until exactly one new row (or column) is + gained. A pruned top-left is no longer an edge case at all: the response simply starts + at a later position than the one asked for. *The rest is the author's analysis.* + - **What it requires of the response:** the **actual** absolute position of every + row and column returned, the first one included. The client compares those + positions with its old top-left, so this is T1 again, and it already holds under + clause 12. Nothing else is needed: no error, no snap rule, no direction in the + request, and no predecessor query. + - **Cost.** With doubling jumps, the gallop takes about log₂(gap) requests. A plain + binary search over the last interval takes about log₂(gap) more. At density 10^-4 + the expected gap is about 10^4 rows, so that is 14 + 14 ≈ 28 forward requests, + against one predecessor request. The refinement is often much cheaper. A forward + response holds k occupied rows with their positions, so if the hit request's k rows + reach the old top-left, the client reads the last row before it directly, and the + search ends after the gallop, about 14 requests. Refinement is needed only when + more than k occupied rows lie between the hit and the old top-left. These figures + are arithmetic, not measured. + - **Statelessness and network transparency.** The host keeps nothing for the search. + Every request is an ordinary `project` from an absolute top-left, so the in-process + and remote consumers do exactly the same thing (T6). The cost is round trips: about + 14 for a remote client and about 14 function calls for the TUI. For the TUI that is + nothing. Over a socket it is about 14 latencies for one scroll step into a sparse + stretch. + - **The author's lean, revised.** Take the user's procedure as the **baseline**. It + makes (a), (b) and (c) unnecessary, because a pruned top-left just returns a later + first position. Keep a model-side predecessor query only as an **optional + optimisation** for a remote client. It costs no state (see above) and collapses + the gallop into one request. Offer it once a measured round-trip cost shows it is + worth a second query direction. *As first written*, this item leaned to offering it + outright; that lean is withdrawn. + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the ordered index, O(log n), in either direction, so (a) with named neighbours and (b) cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap @@ -2046,7 +2084,11 @@ revision, and none is renumbered. predecessor search, which every option serves as cheaply as a successor search). It also raises the edge case of a top-left that lands on a pruned row: fail and retry, snap forward (which the user objects to: "the viewport getting stuck"), or snap toward - the scroll direction (clause 12). That edge case is + the scroll direction (clause 12). The user has since suggested a simpler client procedure: + gallop the top-left with forward requests until one hits, then narrow. A pruned + top-left then needs no special answer, provided responses report actual positions. + The author now takes that as the baseline, and keeps a predecessor query only as an + optional optimisation. That edge case is part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they From 71a56234792f5963bbbb22d19ecae9e3c9b702aa Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:30:55 -0700 Subject: [PATCH 034/106] docs(adr): ADR 009 answers the review of 3698c680 (improvise-felo.4) B1: clause 7 splits a shape key from read-time parameters and says view_shape is not the key; clause 4, Q7 and Q11 follow. B2: one numbered list of the twelve relayed chat statements. B3: withdrawn leans marked superseded; clause 12's pruning text rewritten for the gallop baseline. Also: clause 5a's fallback is answer-preserving (agg_values pools only member_records), ADR 007 keys, T5 per-subscription coalescing, item edits reach every subscriber, fold Sum exactness, owners for questions shared with ADR 008, re-derived citations. Reads ADR 008 at 3180327b. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 964 ++++++++++------------ 1 file changed, 454 insertions(+), 510 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 79255763..3fc94606 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 21** -(PR #308, read at `3b70a0f1`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 24** +(PR #308, read at `3180327b`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `3b70a0f1`, its revision 21) · +reads it at `3180327b`, its revision 24) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 21 of `context/adr/008-virtualized-grid.md` on PR #308, at `3b70a0f1`. Clauses 3, +revision 24 of `context/adr/008-virtualized-grid.md` on PR #308, at `3180327b`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,37 +78,52 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 21, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 24, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **The two chat statements** were given to the orchestrator and relayed to this ADR. They - are on no PR, and ADR 008 records them the same way. The user's "Approved" on this ADR - would confirm them. +- **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed + to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR + would confirm them. ADR 008 records the same statements, also as relayed and not on the + PR, but spread over several groups of its own. This list is the one place this ADR + quotes them, and the clauses cite them by number. None of this answers this ADR's own questions 1 to 11. -**Routed here, in chat:** -- "The DBSP machinery needs to be implemented in a network transparent way, but the - details of that go to ADR 009" (chat, relayed on 2026-10-05). Clause 12. -- "Thinking about it a bit more, we might need to keep track of recent projections + - clients to stream aggregation results back to the UI correctly. But this should be as - stateless as possible" (chat, relayed on 2026-10-05). Clause 12. -- On ADR 008's question 53: "Q53 is cross product: I'm still ambivalent about which side - of the wire pruning should happen on" (chat, relayed on 2026-10-05). So a request's - top-left is a cross-product position, and which side prunes is open. ADR 008 records it - at revision 19 as its question 53 answered and its question 57 open. Clause 12, - question 19. -- "But the top left should be as absolute as possible" (chat, relayed on 2026-10-05). - Clause 12, question 20 (ADR 008's question 58). -- "After pruning we might have to stream rows and columns to attempt to fill out the full - grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). -- "Pruning is just a filter applied to full columns a rows, right? Hide a column/row in - the grid if it's all null. That sounds like a UI side concern because that's where the - visible grid lives" and "The point of pruning is to make the displayed values more - dense, so a value being present outside the viewport doesn't matter." (chat, relayed on - 2026-10-05). Clause 12, question 22. +**Relayed in chat, on no PR** (all on 2026-10-05; quoted exactly as relayed): +- **C1.** "The DBSP machinery needs to be implemented in a network transparent way, but + the details of that go to ADR 009". Clause 12. +- **C2.** "Thinking about it a bit more, we might need to keep track of recent projections + + clients to stream aggregation results back to the UI correctly. But this should be as + stateless as possible". Clause 12. +- **C3.** "Q53 is cross product: I'm still ambivalent about which side of the wire pruning + should happen on". ADR 008 records it as its question 53 answered and its question 57 + open. Clause 12, question 19. +- **C4.** "But the top left should be as absolute as possible". Clause 12, question 20. +- **C5.** "After pruning we might have to stream rows and columns to attempt to fill out + the full grid". Clause 12, question 21. +- **C6.** "Pruning is just a filter applied to full columns a rows, right? Hide a + column/row in the grid if it's all null. That sounds like a UI side concern because + that's where the visible grid lives". Clause 12, question 22. +- **C7.** "The point of pruning is to make the displayed values more dense, so a value + being present outside the viewport doesn't matter." Clause 12, question 22. +- **C8.** "The problem pruning solves is visualizing sparse datasets: screens that are + mostly empty are useless and so you drop out categories from rows and columns until + every visible row or column has at least one value on screen." Clause 12, question 22. +- **C9.** "We should be able to optimize pruning on the right and down assumption of + growth (truncated at the bottom right corner of the axis space)". Clause 12; ADR 008's + question 67. +- **C10.** "The app is free to optimize fetching for up or left by just fetching the new + rows, until it finds one that has values in the viewport but, conceptually, scrolling up + or left would request a new viewport with a new top left corner. The requested top left + corner falling on a pruned row or column is an annoying edge case, maybe just fail the + fetch and retry?" Clause 12, question 22(d). +- **C11.** "Snap forward is going to cause issues with the viewport getting stuck". + Clause 12, question 22(d). +- **C12.** "The simplest might be some kind of binary search for a new top left corner: + increasing jumps of the top left until you get a hit and then move down/right until + You only have one new row column." Clause 12, question 22(d). **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -187,14 +202,16 @@ the clause it changes: 3. **State is keyed by the request's shape, not by a view's name.** Axes are a model concept, so operator state keyed by axes is model state, not UI state. But a `project` request names its own axes, and an unsaved request may carry parameters no saved view - holds (ADR 008's flagged reading 1, its question 51). Clause 7, question 14. + holds (ADR 008's flagged readings 1 and 3, its question 51). Clause 7, question 14. 4. **The record order belongs to the store.** The ordered list of records is the store's (Q27). The circuit may consume it, and does not own it. Clause 9. 5. **The memory requirement covers this ADR too.** The store-memory requirement covers the circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. Clause 14, question 17. -6. **Five of ADR 008's questions are answered.** Those are questions 1, 2, 6, 13 and 15, - and clauses 5a, 9, 13 and 14 now follow the answers. +6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's + questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 42 and + 43, quoted above. + Clauses 5a, 7, 9, 12, 13 and 14 follow them. ## Context @@ -210,7 +227,8 @@ a change proportional to what the change touches, not to the document. ### Evaluation today is a from-scratch fold, rerun after every batch - **Every batch recomputes every formula cell.** `App::apply_effects` - (`crates/improvise-command/src/ui/app.rs:989`) ends in `rebuild_layout` (`:880-891`). + (`crates/improvise-command/src/ui/app.rs:1069` at `50e1314d`) ends in `rebuild_layout` + (`:926`). That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` @@ -227,14 +245,14 @@ a change proportional to what the change touches, not to the document. (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every combination of the spanned categories' items. ADR 008 clause 5 names this as the one O(grid) path left in totals. -- **Two observations scan the store.** `record_fingerprints` (`app.rs:254-264`) hashes +- **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes every cell before and after a cell-writing batch. `document_content` - (`app.rs:165-180`) hashes every cell and every formula for the dirty flag, and it is + (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) per batch and the second O(cells) per effect, whatever was written. - **Every mutating frame from a client already clones the document.** - `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:162`, - `app.rs:627-629`), for ADR 001 clause 4. That is O(cells) per frame on the network + `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, + `app.rs:673` at `50e1314d`), for ADR 001 clause 4. That is O(cells) per frame on the network path, whatever this ADR does. ### What an incremental evaluator must not change @@ -335,7 +353,7 @@ name is `Derived`. It replaces the two places where eager evaluation folds the s `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). It takes two inputs, both applied after each model effect (clause 7): the cell delta, -and the formula delta. It also has a rebuild, for an operator shape that moved. +and the formula delta. It also has a rebuild, for a new shape key (clause 7). **The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code and read through the trait. The circuit decides *which* formula cells to re-evaluate @@ -390,7 +408,7 @@ today.** Two elements have the same `(key, value)` part exactly when their keys equal and their values are equal *by bits*, and bit identity is stated over that part only (the sequence is compared as the integer it is, which keeps an equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and -`Error` by their strings. That is `hash_value` (`app.rs:233-250`), which +`Error` by their strings. That is `hash_value` (`app.rs:246-262` at `50e1314d`), which `record_fingerprints` already hashes. So the cell delta reproduces today's tier exactly: @@ -415,12 +433,12 @@ in the batch's consolidated delta. So: keys. `bump_changed_records` can then read the delta instead of diffing two -`record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta +`record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, because it observes `view_shape` and not cells. **The dirty flag's content half does not move onto the cell delta alone.** -`document_content` (`app.rs:165-180`) hashes the formulas as well as the cells, and it +`document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a @@ -436,6 +454,25 @@ new is observed. Its cost is O(formulas), where `document_content` is O(cells) p `PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by `f64::total_cmp`. Clause 4 says where that differs from `==`. +**Names and ADR 007 (added on 2026-10-05).** ADR 007 is accepted. It makes ids the keys +and names display-only, and in it a `CellKey` becomes sorted `(CategoryId, ItemId)` pairs. +The user said there: "This is a lot lower value if CellKeys, etc. don't refer to the new +id". Every key this ADR defines follows `CellKey`: the cell delta, `CellKey`'s `Ord`, the +stems, `record_of`, the group keys, the shape key's categories, and the output's tuple +identities (clause 12). So **under ADR 007 they are id-keyed with no change to this ADR, +and a rename emits no cell delta and rebuilds no operator.** It is one `display` write, +which the output carries as a relabelling, not as a retraction and an insertion. Two +places do not follow `CellKey`, and must move with ADR 007: +- **the formula fingerprint** (clause 7). It is taken over `(raw, target, + target_category)`, which are names today. Under ADR 007 it must be taken over the + formula's id-form, or a rename would read as a formula delta and re-evaluate every + affected target; +- **the cross-product order** (clause 12), which orders items by the category's own + order. That order is unchanged by a rename. + +Until ADR 007 lands, a rename is what it is today: a remove and an insert of every cell +under the old name, and this ADR's deltas say exactly that. + ### 4. Aggregates: linear where they can be, an integrated multiset where they cannot **The group** is the cell key with the view's `none_cats` removed. That is the partial @@ -502,11 +539,12 @@ key, and `FIRST` the least. the group, and one entry per contributing record. That is the same order as `Min`/`Max`'s multisets. - **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. **Most of that is already in the - operator shape**: `view_shape` hashes each category's *drawn* items, in order - (`app.rs:47-52`), so a reorder of drawn items already rebuilds. What it does not see - is the position of a **hidden** item, which matters only if `LAST`/`FIRST` must order - values that come from hidden items. The alternative is to key by a stable item + reordering its items, changes every stored position. *As first written*, this item + said a reorder of drawn items already rebuilds, through `view_shape`. **That is + superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a + reorder rebuilds nothing, and a state keyed by position would go stale. The position of + a **hidden** item matters only if `LAST`/`FIRST` must order values that come from + hidden items. The alternative is to key by a stable item identity (ADR 007) and look the order up at read time. That makes a reorder free, but a read then costs O(d). This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, @@ -681,15 +719,20 @@ is one evaluator. that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. **Revised on 2026-10-05:** -such a formula does **not** stay on today's eager path, because that path is -`Expansion::members`, the cross product (ADR 008 clause 5, a finding of Copilot's on -#308). It takes ADR 008's **store-driven fallback**, which evaluates the expression only -over the **occupied contexts**, the contexts some record projects to. The user accepted -that recommendation, as ADR 008's question 15 ("the recommendation is fine."). It differs -from today's eager answer wherever the expression is non-empty at a context that holds no -record, and ADR 008 carves those cases out of its differential test by name. Whether this -ADR's `Eager` adopts the same rule, so that its own differential test needs no carve-out, -is question 18. Two regressions pin the record stage against eager: +such a formula is not maintained, and at a fold or a total it does **not** enumerate +today's eager contexts. Those are `Expansion::members`, the cross product (ADR 008 clause +5, a finding of Copilot's on #308). At a stem, eager pools one context and there is no +product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's +**store-driven fallback**, which evaluates the expression only over the **occupied +contexts**, the contexts some record projects to. The user accepted that recommendation, +as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as +eager's**, as the paragraph above says of the pipeline. `agg_values` evaluates the +expression only at the `member_records` of each context (`types.rs:47-53`, `:91-139`), +so a context with no record contributes no value and no error, whatever the expression. +The fallback changes the cost and not the answer. *The first revision of 2026-10-05 said +it "differs from today's eager answer" wherever the expression is non-empty at an empty +context. That was wrong, and ADR 008's question 15 carries the same premise.* Whether +this ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -701,9 +744,9 @@ another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A r whose key `record_of` takes without `_Measure`, then belongs to the context only through its cell under that measure, and eager orders it by *that* cell's position, not by the record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. **That -eager path is `Expansion::members`, so while it lasts it blocks ADR 008's retirement of -`Expansion::members` (its migration step 6) for these formulas.** Sending them to the +formulas stay on the eager path, and the differential test pins that they do. **At a fold +or a total that eager path is `Expansion::members`, so while it lasts it blocks ADR 008's +retirement of `Expansion::members` (its migration step 6) for these formulas.** Sending them to the occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** @@ -759,44 +802,67 @@ post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1 That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 states the same rule. -- **Operator shape, per projection shape (revised on 2026-10-05).** Operator state is - keyed by the **operator shape of a `project` request**, not by the name of a view - (question 14). The shape is made of: +- **A shape key, and read-time parameters (revised on 2026-10-05).** Operator state is + selected by the **shape key** of a `project` request, not by the name of a view + (question 14). The key is exactly what changes which cells feed which operator: - the categories of each axis, by level; - the None categories; - - the filter and page keys; - - the hidden items and collapsed groups the request carries; + - the filter keys: each `Axis::Filter` category with its selected item, and the hidden + items of every category that is **not** on an axis (such an item drops cells from the + fold, so it filters); + - the page **categories**, which key the slices under question 7's recommendation; - `measure_agg`. - A saved `.improv` view supplies these to a request that names it. An unsaved request - carries its own, for example an unsaved prune or hide (ADR 008's flagged reading 1, its - question 51). Requests with the same shape share one set of operators, whichever - clients sent them. The live shapes are a **cache**. A shape enters when a request names - it, and it is evicted once no recent request has named it (clause 12). For a saved - view the shape still includes what `view_shape` (`app.rs:40-72` at `57d385e1`) hashes, - plus the group membership of every drawn item and `measure_agg`. `view_shape` itself is - not widened, so what bumps `layout_version` is unchanged. + Everything else a request carries is a **read-time parameter**, applied when the + output is read, without touching operator state: + - the hidden items of an axis category; + - the collapsed groups, and the group membership of axis items; + - the drawn order of items; + - the page **selection**, which picks a slice; + - prune, which under client-side pruning is not the host's at all (clause 12). - *As first written* (superseded): a view whose operator shape moved had its operators - rebuilt **before the frame observation that follows this effect**. That rule served the - per-view frame observation, which ADR 008 clause 8 revises under the split. The three - rules below replace it. + **`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72` at `50e1314d`) + hashes each category's drawn items in order (hidden items excluded), its collapsed + groups, its page selection and `prune_empty`. Those are all read-time parameters except + the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, + `AddItem`, reorder and page step, which is what item grain and question 7 exist to + avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40), and it is + unchanged. + + A saved `.improv` view supplies the key and the parameters to a request that names it. + An unsaved request carries its own (ADR 008's flagged readings 1 and 3, its question + 51). + Requests with the same key share one set of operators, whichever clients sent them. The + live keys are a **cache**: a key enters when a request names it, and it is evicted once + no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a + shape key. + + *As first written* (superseded twice): the operator shape was `view_shape` plus group + membership and `measure_agg`, and a view whose shape moved had its operators rebuilt + **before the frame observation that follows this effect**. The first revision of + 2026-10-05 still listed hidden items and collapsed groups in the shape, and still + deferred to `view_shape` for a saved view. That contradicted the item-grain rule below + and question 7, as the review of `3698c680` found. - **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column aggregates are keyed by the **items** of the axis categories, together with the filter and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no operator. The visible axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms - for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item changes no - operator either, because a new item tuple is just a new key. Three things still - rebuild a shape: - - a change of which categories are its levels; - - a change of its filter or page keys, which includes hiding an item of a category that - is not on an axis; + for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or + reordering items, changes no operator either: a new item tuple is just a new key, and + the order is a read-time parameter. Only a change of the **shape key** selects other + operators, which a new key builds as a job: + - a change of which categories are its levels or on None; + - a change of its filter keys, which includes hiding an item of a category that is not + on an axis; + - a change of its page categories; - a change of `measure_agg`. A collapsed group's value is a pooled fold over its members. With item grain it is - combined from the members' state when it is read. For `Sum` and `Count` that combine is - linear. For the integrated aggregates it costs O(members) per read, and whether that + combined from the members' state when it is read. For `Count` that combine is exact. + For `Sum` it is exact only under question 3's (c), or under ADR 005's wide accumulator. + A sum of the members' f64 sums is not bit-equal to eager's one store-order fold + (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a @@ -807,7 +873,9 @@ states the same rule. Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and priority queue the user required. This ADR only makes its rebuilds steppable and - cancellable between slices. + cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user + ruled. Any measurement this ADR later reports against them is stated as a ratio + against an unrelated operation on the same run, as ADR 008's question 13 requires. - **A crossing count (ADR 008's second request).** Each step's occupancy output carries the number of tuples whose count crossed zero. ADR 008's ordered index uses it to choose between an insertion and a rebuild by sorting. @@ -822,11 +890,10 @@ states the same rule. - **A document swap rebuilds every circuit** from the new store. `install_workbook` bumps `document_generation`, and the same after-each-effect read sees it. -Re-reading the operator shape costs O(items) per live shape per model effect, and the formula -fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs -O(cells), once per actual reshape. The same `view_shape` function is therefore read at -two moments: by the circuit after each effect, and by `layout_version` after the loop. -They answer different questions, and neither replaces the other. +Re-reading the shape keys costs O(categories) per live key per model effect, and the +formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a +new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by +`layout_version` after the loop, as today. **The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first written* (superseded by item grain): `AddItem` moved the operator shape, so the view was @@ -836,7 +903,7 @@ key, and only the visible axis, which is derived, gains a row. The case needs no ordering rule at all. **Recommended: a page selection is not a rebuild (question 7).** Under the recommended -answer, the group key includes the page categories, so `[` and `]` read another slice +answer, the shape key includes the page categories and not the page selection, so `[` and `]` read another slice of state that already exists. The held-key path stays O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the state is still bounded by the stored cells, however many pages there are. Filter coordinates still filter; only page @@ -1080,7 +1147,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 21 at `3b70a0f1`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 24 at `3180327b`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1168,7 +1235,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:647`). **Recommended (question 9): the backend's state joins the snapshot.** + (`app.rs:693` at `50e1314d`). **Recommended (question 9): the backend's state joins the snapshot.** Applying the inverse delta is **not** a rollback, even with exact aggregators. It restores integrated values, but `step(d)` then `step(−d)` still advances the circuit and changes the history that `delay` and `differentiate` hold. So the next step after @@ -1178,7 +1245,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n **The snapshot costs a second O(cells) clone per mutating frame.** `Host::mutating_frame` already clones the workbook on every mutating frame - (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So + (`host.rs:224-225`, `app.rs:673` at `50e1314d`), and the integrated state is of the same order. So on the network path a write is not O(delta): it is O(cells) for the snapshot today, and roughly twice that with the circuit in it. The keyboard path (`apply_effects` without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the @@ -1190,8 +1257,8 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 12. Network transparency: one circuit, consumed in process or across a wire -**The requirement is the user's:** "The DBSP machinery needs to be implemented in a -network transparent way, but the details of that go to ADR 009" (chat). **Everything +**The requirement is the user's** (C1): "The DBSP machinery needs to be implemented in a +network transparent way, but the details of that go to ADR 009". **Everything else in this clause is this ADR's design, and the author's choices are marked as such. None of it is measured.** @@ -1217,8 +1284,9 @@ Three things do not cross: **What a projection request is to the circuit** (the author's answer to "is the result keyed by request, by axes, or shared?"). A request has two parts. -- **Its operator shape** (clause 7) selects the circuit's state. The shape is shared by - every request, from any client, that has it. +- **Its shape key** (clause 7) selects the circuit's state. The key is shared by every + request, from any client, that has it. Its read-time parameters (hidden and collapsed + axis items, drawn order, page selection) are applied to that state's output. - **Its window**, the top-left and the dimensions, selects a part of that state's output. The window is never state. @@ -1226,330 +1294,205 @@ So maintained results are keyed by **shape**, and the only per-request work is restricting a shape's output to a window. That is O(visible) per request and needs nothing kept between requests. -**Positions are cross-product positions.** The user answered ADR 008's question 53 in -chat: "Q53 is cross product: I'm still ambivalent about which side of the wire pruning -should happen on" (relayed on 2026-10-05; on no PR). So a request's top-left is a position -in the **cross product** of its axes, in each category's own item order. That position is -a bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed +**Positions are cross-product positions** (C3). A request's top-left is a position in the +**cross product** of its axes, in each category's own item order. That position is a +bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed contexts, we can represent bigints as strings in JSON"). A cross-product position names the same tuple whatever is occupied, so **an occupancy change never moves a request's window**. It changes only which positions inside the window hold data. **Which side of -the wire prunes is open** (the user is "still ambivalent"), and is question 19. The -circuit must serve both sides, and what it must expose in each case is the author's -analysis: -- **Pruning on the model side.** `project` returns occupied rows and columns. The circuit - must then answer two order-statistic queries over a shape's visible occupancy, from a - cross-product position: - - the next k occupied tuples at or after a position, so a pruned window can start where - the request points; - - the rank of a tuple among the occupied ones, and the occupied tuple of a given rank, - so a client can scroll a pruned axis by rows and show "row i of n". +the wire prunes is open.** C3 says the user is "still ambivalent", and C6 says pruning +"sounds like a UI side concern". That is question 19. The rest of this clause is the +author's analysis, and it is written for **client-side pruning** first, because that is +the author's lean after C6 to C8. Where model-side pruning would differ, it says so. - That is ADR 008's ordered index over occupied tuples (its clause 3), the sorted `Vec` - first and the order-statistic B-tree when measured (its question 16), fed by this ADR's - occupancy delta and crossing count. Under this case a client sees only occupied rows, - and its scroll space is the pruned one, built on top of the cross-product offset. -- **Pruning on the client side.** `project` returns the cross-product window, empty - positions included, together with a **skip structure** the client can query: for each - axis, the occupied runs that intersect the window and the next occupied position past - each edge. The circuit then has to expose the same order-statistic queries, but as a - model fact a client can request, not as a re-indexing the model performs on its behalf. - The cost is that a sparse grid at 3 × 1,000 (10^9 positions, about 10^5 occupied in ADR - 008's fixture) sends mostly empty windows unless the client asks for skips first. - -Both cases need the same maintained structure. They differ only in who calls it, and in -whether a `project` result is already pruned. Prune would be a request parameter in the -first case (ADR 008's flagged reading 1) and a client policy in the second. - -**"As absolute as possible", and what still moves a position.** The user, in chat: "But -the top left should be as absolute as possible" (relayed on 2026-10-05; on no PR). *The -rest of this paragraph is the author's analysis.* A cross-product position does not move -when occupancy changes, pruning included. **It does move when an item is inserted, -removed or reordered** in a category's own order. Positions are mixed-radix numbers over -the items of each level, so inserting item j at level L shifts every later item of L, and -changes the radix that every outer position is multiplied by. No position over a -changing item list is fully absolute. Three things follow: +**"As absolute as possible", and what still moves a position** (C4). *The author's +analysis.* A cross-product position does not move when occupancy changes, pruning +included. **It does move when an item is inserted, removed or reordered** in a category's +own order. Positions are mixed-radix numbers over the items of each level, so inserting +item j at level L shifts every later item of L, and changes the radix that every outer +position is multiplied by. No position over a changing item list is fully absolute. +Three things follow: - **Deltas are keyed by identity, and positions are attached when sent.** An output element names its row and column by tuple identity: item ids under ADR 007, and names - until then. The host attaches the absolute position from the current item order when it - sends the element. A delta keyed by position alone would go stale with the next item - insert. -- **An item edit is sent as an axis change, not as moved rows.** "Item inserted at index - j of level L", or removed, or moved. A client re-derives every position it holds - arithmetically, at O(levels) per position, and its top-left can be re-anchored to the - same tuple. That is the cursor's identity rule from ADR 008 clause 11 applied to the - window. Resending every row whose position moved would cost O(rows after j). + until then (see "Names and ADR 007" in clause 3). The host attaches the absolute + position from the current item order when it sends the element. A delta keyed by + position alone would go stale with the next item insert. +- **An item edit is sent to every subscriber as an axis change.** "Item inserted at index + j of level L", or removed, or moved. It goes to **every** subscriber of a shape with + that category on an axis, wherever its window lies, because an insert before a + window's top-left shifts every position that subscriber holds. A client re-derives its + positions arithmetically, at O(levels) per position, and re-anchors its top-left to the + same tuple, as ADR 008 clause 11 anchors the cursor. Resending every row whose position + moved would cost O(rows after j). - **The cost is rank and select over each category's items.** "The position of item x" and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` first, a B-tree when measured, its question 16). A tuple's position is O(levels × log - items). The ordered occupancy index must compare tuples by **current** item position, + items). An index ordered by position must compare tuples by **current** item position, not by a position stored at insert time. With item grain (clause 7) its keys are item - identities, so an item insert re-orders nothing in it, and comparison goes through the - order-statistic lookup, at O(levels × log items) per comparison. That is **not - measured**. Question 20. + identities, so an item insert re-orders nothing in it, and a comparison goes through + the order-statistic lookup, at O(levels × log items). That is **not measured**. + Question 20. -**A pruned answer may arrive as a stream.** The user, in chat: "After pruning we might -have to stream rows and columns to attempt to fill out the full grid" (relayed on -2026-10-05; on no PR). *The author's analysis:* under model-side pruning, `project` scans -forward from the absolute top-left over each axis's occupancy until it has the requested -number of occupied rows and columns, or reaches the end of the axis. It sends the rows -and columns it finds as it finds them. What the circuit must support: -- **Skipping empty space in O(log n + k).** "The next k occupied tuples at or after a - position" over the ordered occupancy index. It must never walk empty cross-product - space, which at 3 × 1,000 is mostly empty. -- **The absolute position of every row and column sent**, so that a client can place it, - and a later delta can name it, without counting. -- **Bounded work per slice.** The fill is a job under ADR 008 clause 11's runner when k - is large, and it can be resumed from the last absolute position it sent. So the only - state a fill keeps is `(request, fill front)`, which is "as stateless as possible" in - the user's words. A lost fill restarts from the client's last known position. -- **Changes during a fill are identity-keyed deltas, stamped by step** (T3). Suppose a - row becomes occupied between the top-left and the fill front after it was passed. The - subscriber receives it as an ordinary occupancy delta carrying its absolute position, - so its window now holds one row more than asked, and it drops or keeps the extra row by - client policy (Q42). A row that becomes occupied ahead of the front is simply found by - the fill. Each chunk of the fill is a delta to the window's Z-set, stamped like any - other. So a fill and a live update are one stream, which keeps the in-process and the - remote consumer the same (T6). - -**Pruning is a filter on the values displayed (the user, later in chat).** "Pruning is -just a filter applied to full columns a rows, right? Hide a column/row in the grid if it's -all null. That sounds like a UI side concern because that's where the visible grid lives" -and "The point of pruning is to make the displayed values more dense, so a value being -present outside the viewport doesn't matter." (both relayed on 2026-10-05; on no PR). -**This is a change from today's code**, and it is recorded here, not decided. Today, -`prune_empty` drops a row only when it is empty across the **whole** grid, and prunes the -columns against the same unpruned grid (`layout.rs:752-757` at `50e1314d`). In the user's -words, emptiness is judged over the **displayed** values, so a row that holds a value only -in a column outside the window is hidden. *The rest of this paragraph is the author's -analysis.* -- **What the model then owes a client** is not a pruned axis. It owes windows (an - absolute top-left and dimensions) and a **restricted skip query** for the client's fill - loop: "the next k rows, from absolute position p, that hold a value in at least one of - these columns", and the same with rows and columns swapped. The question is about a set - of columns, not about the whole grid, so a per-axis count cannot answer it. A row can be - occupied overall and still be empty in every column the client shows. -- **Three ways to answer the query**, all unmeasured: +**Pruning as a filter on the displayed values** (C6, C7). **This is a change from today's +code**, and it is recorded here, not decided. Today, `prune_empty` drops a row only when +it is empty across the **whole** grid, and prunes the columns against the same unpruned +grid (`layout.rs:752-757` at `50e1314d`). The user's intent, as ADR 008 reads it (its +questions 60 to 65), is that emptiness is judged over the **displayed** values, so a row +that holds a value only in a column outside the window is hidden. *The rest of this +paragraph is the author's analysis.* +- **What the model owes a client** is then not a pruned axis. It owes cross-product + windows (an absolute top-left and dimensions) and a **restricted successor query**: + "the next k rows after absolute position p that hold a value in at least one column of + the set S", and the same with rows and columns swapped. **S travels in the call**, + because only the client knows which columns it displays. A per-axis count cannot + answer it: a row can be occupied overall and still be empty in every column the client + shows. +- **Three ways to answer it**, all unmeasured: - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then - test each candidate row against the column set by lookup. The state is the per-axis - counts clause 9 already keeps. The worst case scans many rows that are occupied - overall but empty in the column set, at O(|columns|) lookups each. + test each candidate against S by lookup. The state is the per-axis counts clause 9 + keeps. The worst case scans many rows that are occupied overall but empty in S, at + |S| lookups each. - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that - hold a value in it. The query is a k-way merge of |columns| sorted lists from p, at - O(|columns| × log n + k × |columns|). The state is O(occupied cells at grid grain), - the two-dimensional count clause 9 does not keep. It is maintained from the cell delta - and from formula stems (clause 5's stem rule decides when a formula cell holds a - value). A collapsed group's column merges its members' lists when it is read. + hold a value in it. The query seeks each of S's lists to p and merges them forward, + choosing the minimum by a linear scan over the |S| cursors: O(|S| × log n + k × |S|). + The state is O(occupied cells), the two-dimensional count clause 9 does not keep. It + is maintained from the cell delta and from formula stems (clause 5's stem rule decides + when a formula cell holds a value). A collapsed group's column merges its members' + lists when it is read. - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, - intersected with the column set as the fill walks forward. It has the same state as - (ii), and its scan behaves like (i), with an exact test per row. + tested against S as the scan walks forward. It has the same state as (ii), and it + scans like (i), with an exact test per row. The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) - structure, beside the store (clause 14). -- **Growth is only rightward and downward (the user, in chat).** "We should be able to - optimize pruning on the right and down assumption of growth (truncated at the bottom - right corner of the axis space)" (relayed on 2026-10-05; on no PR). *The rest of this - item is the author's analysis.* The fill loop only grows away from the absolute - top-left, and it stops at the end of each axis. So every skip query is a one-directional - **successor** query: "the next row (or column) after position p with a value among this - set", bounded by the end of the axis. Pruning never needs a predecessor or a reverse - scan. Scrolling up or left is not a reverse fill. It is a **new request with a new - absolute top-left**, filled forward from there, and nothing from the old request has to - be walked backwards. What that simplifies: - - **The index options.** (ii)'s per-column sorted lists need only a seek (a lower - bound at p) followed by forward iteration. A k-way merge of forward cursors keeps - O(|S|) state and never steps back. Option (i)'s pre-filter and option (iii)'s per-row - test also walk forward only. So any ordered structure with a lower-bound seek and a - forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 - question 16). No order statistics (rank, or "row i of n") are needed for pruning. - They remain only for a client that wants a scrollbar over an unpruned axis, where - rank is arithmetic over the cross product anyway. - - **Maintenance.** An insert or removal at position q affects only fills whose front is - already past q, and only within their window. A fill whose front is before q finds the - change when it gets there. So a subscription needs a delta only for positions between - its top-left and its front, which is a range test against two absolute positions. - - **Resumable streams.** A fill's whole state is its request and its last absolute - position on each axis. A resume is a seek past that position, so a lost stream, a - reconnect or a sliced job (ADR 008 clause 11) all resume the same way, with nothing - else kept. Positions are stable under occupancy changes because they are - cross-product positions. An item insert before the front shifts them, which is - question 20's axis-change message: the client re-derives its front arithmetically. - - **The corner truncates the work.** The fill is bounded by the requested dimensions - and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × - |S|) per round, never a scan of empty space past the last occupied position. + structure beside the store (clause 14). Index (ii) is a set of `(column, row)` pairs, + so it is itself a Z-set fed from the cell delta and from formula stems. A write costs + O(log n), and its delta, "row r gained or lost a value in column c", is what a + subscription restricted to S receives. +- **Under model-side pruning instead**, the model would keep a pruned axis per shape: ADR + 008 clause 3's ordered index over occupied tuples, with rank and select for "row i of + n". Pruning would be a request parameter (ADR 008's flagged reading 1), and the host + would judge "pruned". Everything below that mentions a host judging emptiness applies + only to this alternative. -- **Scrolling up or left, and a top-left that lands on a pruned row (the user, in - chat).** "The app is free to optimize fetching for up or left by just fetching the new - rows, until it finds one that has values in the viewport but, conceptually, scrolling up - or left would request a new viewport with a new top left corner. The requested top left - corner falling on a pruned row or column is an annoying edge case, maybe just fail the - fetch and retry?" (relayed on 2026-10-05; on no PR). The user's "maybe" is tentative, - and nothing here is decided. *The rest of this item is the author's analysis.* - - **The optimisation is a predecessor search.** Fetching only the new rows above the - window, and scanning upward until one has a value among the displayed columns, asks - "the previous row before p with a value in S". The forward-only queries above do not - serve it directly. There are two ways to serve it: - - **The client does it with forward requests.** It guesses an earlier top-left, - requests forward from there, and widens the guess if nothing qualifies. Each guess - costs one forward query, and the number of guesses grows with the gap. At the - fixture's density of 10^-4 a blind step of tens of rows usually finds nothing, so - it needs widening, about log₂(gap) guesses if each guess doubles the step. - - **The model offers a predecessor query.** It is cheap under every option. A sorted - `Vec` and a B-tree both give an upper-bound seek and a reverse iterator at the same - cost as the forward ones, O(log n) and then O(1) per step. Option (ii)'s per-column - sorted lists merge backwards exactly as they merge forwards. Options (i) and (iii) - walk backwards with the same per-row test. So offering it adds no state. It adds - only a second direction to the query surface, which the conceptual model (always - fill forward from a new top-left) does not need. *Author's lean:* offer it, because - it costs no state and saves the client a guessing loop. The user's to confirm. - - **The edge case: a top-left on a pruned (empty) row or column.** Listed neutrally: - - (a) **Fail, then retry** (the user's tentative "maybe"). The model answers with an - error. If the error names the nearest occupied successor (and predecessor), the - retry is one round trip. If it names nothing, the client must guess, as above. - - (b) **Snap forward, and report the actual top-left.** The model starts the fill at - the next occupied position at or after the requested one, and returns that - position in the response. No extra round trip. The client sees its window start - somewhere other than it asked. - - (c) **Snap toward the scroll direction.** Like (b), but snap backward when the - request came from an upward scroll. That needs the request to say which direction - it came from, which the request sketch does not carry. +**The fixpoint comes from the fill, not from one window** (C8). *The author reads C8* as +a fixpoint on the displayed grid: every visible row has a value among the visible +columns, and every visible column has one among the visible rows. C8 says "drop out +categories", which the author reads as dropping rows and columns, that is, items or +tuples. The user may mean dropping a whole category from an axis. ADR 008 flags the same +ambiguity (its question 66(a)), and it is the user's to settle. *As first written*, this +clause said that hiding a row can empty a column. **That was wrong.** A row dropped for +having no value among the window's columns contributes no value to any of those columns. +So on a fixed window, one pass that drops the rows and columns empty within the window +already is the fixpoint, which is unique (ADR 008's question 61 makes the same argument). +The iteration comes from the fill step, which brings in rows and columns from outside the +window (ADR 008 question 66). A client alternates three steps: +- it fetches a window; +- it prunes, in one pass; +- it fills, asking for the next rows past its last absolute row position with a value + in its displayed columns, and likewise for columns. - **The user objects to (b)** (chat, relayed on 2026-10-05; on no PR): "Snap forward is - going to cause issues with the viewport getting stuck". *The author's reading of the - reason:* a client scrolling up asks for a top-left just above its window. If that - position is empty, (b) snaps forward to the first occupied position, which is the - window's old top-left. So the window never moves past an empty stretch above it. (b) - is therefore set aside, and (a) and (c) remain. **Both need the predecessor query**: - (a) to name the nearest occupied position before the request, and (c) to snap backward. - So the author's lean to offer a predecessor query is stronger now, and it is still the - author's. +The model never re-runs a client's fixpoint. A subscribed client receives index (ii)'s +deltas for its S, prunes again locally, and fills again only when a row or column has +gone. Whether the loop always terminates is **not proven**. The fill only moves forward +through absolute positions, so it adds finitely many candidates before the axis ends. A +client that bounds its fills per frame stays within ADR 008's frame budget whether or not +the loop has converged, and that bound is client policy (Q42). - **Neither (a) nor (c) can get stuck, and neither oscillates** (the author's argument): - - **(a) moves the client.** The error names the nearest occupied predecessor and - successor. A client scrolling up retries at the predecessor, which is strictly - before its old top-left, so each scroll step makes progress. A client scrolling - down retries at the successor, which is strictly after. Neither retry can land on an - empty position, because both are occupied by definition. One retry suffices, unless - a write empties that position in between; then the next error names new neighbours, - still in the same direction. - - **(c) moves in the direction asked.** It snaps backward for an upward request and - forward for a downward one, so the actual top-left moves in the direction of the - scroll, or stays put only when there is nothing further that way. - - **At the bounds.** At the top of the axis there is no predecessor. Under (a) the - error names the successor only, and says there is no predecessor. Under (c) an - upward snap with nothing above it returns the first occupied position (the top of - the occupied axis) and says it is the top. Either way the client learns it is at - the edge and stops scrolling, so it does not bounce between "fail" and "retry". - The bottom-right corner is symmetric. An axis with **no** occupied position - answers "empty axis" under both, and the window is empty. +**Growth is only rightward and downward** (C9). *The author's analysis.* The fill only +grows away from the absolute top-left, and it stops at the end of each axis. So **the +fill** only ever asks successor queries: "the next row (or column) after p with a value +in S", bounded by the end of the axis. Scrolling up or left is not a reverse fill: it is +a **new request with a new absolute top-left** (C10), and it is the one place a backward +search appears (next paragraph). What forward-only growth simplifies: +- **The index options.** Each needs only a seek to p and forward iteration, so any + ordered structure with a lower-bound seek and a forward iterator suffices: the sorted + `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no + rank or "row i of n" is needed. A scrollbar over the unpruned cross product gets its + rank by arithmetic. +- **Maintenance.** The range test that follows applies to **occupancy and cell deltas + only**. An item edit is an axis change and goes to every subscriber (above). A change at + position q matters to a fill only if q lies between its top-left and its front. A fill + whose front is before q finds the change when it gets there. +- **Resumable fills.** A fill's whole state is its request and its last absolute position + on each axis. A resume is a seek past that position, so a lost stream, a reconnect or a + sliced job (ADR 008 clause 11) all resume the same way, with nothing else kept. If a + column's list changes during a merge, a re-seek from the last position is enough. An + item insert before the front shifts positions, and the axis-change message above lets + the client re-derive its front. +- **The corner truncates the work.** A fill is bounded by the requested dimensions and by + the bottom-right corner of the axis space. One round costs O(|S| × log n + k × |S|), + never a scan of empty space past the last occupied position. - **The user's simpler procedure** (chat, relayed on 2026-10-05; on no PR), quoted - exactly as sent: "The simplest might be some kind of binary search for a new top left - corner: increasing jumps of the top left until you get a hit and then move down/right - until - You only have one new row column." *The author's reading:* the procedure belongs to the - client and uses forward requests only. The client moves its requested top-left up (or - left) in growing jumps until a request returns an occupied row before its old - top-left, then narrows back down (or right) until exactly one new row (or column) is - gained. A pruned top-left is no longer an edge case at all: the response simply starts - at a later position than the one asked for. *The rest is the author's analysis.* - - **What it requires of the response:** the **actual** absolute position of every - row and column returned, the first one included. The client compares those - positions with its old top-left, so this is T1 again, and it already holds under - clause 12. Nothing else is needed: no error, no snap rule, no direction in the - request, and no predecessor query. - - **Cost.** With doubling jumps, the gallop takes about log₂(gap) requests. A plain - binary search over the last interval takes about log₂(gap) more. At density 10^-4 - the expected gap is about 10^4 rows, so that is 14 + 14 ≈ 28 forward requests, - against one predecessor request. The refinement is often much cheaper. A forward - response holds k occupied rows with their positions, so if the hit request's k rows - reach the old top-left, the client reads the last row before it directly, and the - search ends after the gallop, about 14 requests. Refinement is needed only when - more than k occupied rows lie between the hit and the old top-left. These figures - are arithmetic, not measured. - - **Statelessness and network transparency.** The host keeps nothing for the search. - Every request is an ordinary `project` from an absolute top-left, so the in-process - and remote consumers do exactly the same thing (T6). The cost is round trips: about - 14 for a remote client and about 14 function calls for the TUI. For the TUI that is - nothing. Over a socket it is about 14 latencies for one scroll step into a sparse - stretch. - - **The author's lean, revised.** Take the user's procedure as the **baseline**. It - makes (a), (b) and (c) unnecessary, because a pruned top-left just returns a later - first position. Keep a model-side predecessor query only as an **optional - optimisation** for a remote client. It costs no state (see above) and collapses - the gallop into one request. Offer it once a measured round-trip cost shows it is - worth a second query direction. *As first written*, this item leaned to offering it - outright; that lean is withdrawn. +**A pruned answer as a stream** (C5). Under client-side pruning the fill is the client's, +and each step is one restricted successor query, so the host keeps no fill state at all. +Under model-side pruning, `project` itself would scan forward from the top-left and send +rows and columns as it finds them, as a sliced job resumable from `(request, fill +front)`. Either way, changes during a fill are identity-keyed deltas stamped by step +(T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an +ordinary delta with its absolute position, and the client keeps or drops the extra row by +its own policy (Q42). So a fill and a live update are one stream, which keeps the +in-process and the remote consumer the same (T6). Question 21. - Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the - ordered index, O(log n), in either direction, so (a) with named neighbours and (b) - cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap - to the next occupied row is on the order of 10^4 cross-product rows, so a client that - guesses blind under (a) would need about log₂(10^4) ≈ 14 doubling retries. **Unpruned - space is not an edge case at all**: a top-left on an empty position is answered with - empty cells. - - **How it fits identity-keyed deltas and resumable streams.** Under (b) and under (a) - with named neighbours, the response's actual top-left is an absolute position, and - the tuple there is an identity. So the stream's resumable state stays `(request, - actual top-left, last position)`, and a later delta names rows by identity whichever - top-left was asked for. A top-left whose row is pruned after the stream started is the - same case in time. A delta reports the row's removal, the client's window keeps its - absolute top-left, and the next fill skips past the empty position. That is (b)'s - behaviour without a new request. +**Scrolling up or left, and a top-left in empty space** (C10 to C12). +- **The baseline is the user's galloping search** (C12; "the simplest might be"). *The + author's reading:* the client moves its top-left up (or left) in growing jumps until a + request hits, then narrows back until exactly one new row (or column) is gained. **Each + probe is a restricted successor query** from the candidate top-left t, with the client's + current S, bounded by its old top-left: "is there a row in [t, old top-left) with a + value in S, and which?" It is not a bare cross-product `project`, which would return + mostly empty rows. +- **What it requires:** the response reports the **actual** absolute position of every + row and column returned, which T1 already requires. Under client-side pruning a + top-left that lands on an empty row is no edge case for the host. The host returns the + cross-product window or the successor answer, and the client's own prune decides what + shows. +- **Cost** (arithmetic, not measured). Doubling jumps take about log₂(gap) probes, and a + binary narrowing about log₂(gap) more. At density 10^-4 the gap is about 10^4 rows, + so 14 + 14 ≈ 28 probes. It is often about 14, because a probe returns up to k rows with + their positions. If the hit's rows reach the old top-left, the client reads the answer + directly. Under concurrent writes neither phase can run forever: the gallop ends at the + top of the axis, and the narrowing is a bounded bisection. +- **Statelessness and transparency.** The host keeps nothing for the search, and the + in-process and remote consumers make the same calls (T6). The cost is round trips: about + 14 over a socket for one scroll step into a sparse stretch, and negligible in the TUI. +- **Mixed cases** (the author's). A diagonal scroll moves both axes. The client gallops + one axis at a time. It gallops rows against its current displayed columns, then + columns against the rows it now displays, then prunes once. When only the row axis's + new top-left is empty, only the row gallop does work. When both are empty, both do, in + turn. Whether rows-then-columns and columns-then-rows always give the same window is + **not proven**. ADR 008's question 61(b) notes that order matters for a one-sided + pass, and the order is client policy. +- **A predecessor query is an optional optimisation.** The model could answer "the + previous row before p with a value in S" in one call. It costs no new state under any + of the three options: a sorted `Vec` and a B-tree both seek an upper bound and iterate + in reverse at the same cost, option (ii)'s lists merge backwards as they merge + forwards, and options (i) and (iii) walk backwards with the same test. *The author's + lean:* offer it only once a measured round-trip cost shows that one call is worth a + second query direction. +- *Superseded:* earlier text in this clause leaned to offering the predecessor query + outright, and then called that lean "stronger" once options (a) and (c) below needed + it. The galloping baseline replaces both leans. Options (a) to (c) are kept below as + a record. -- **The fixpoint comes from the fill, not from one window.** *As first written*, this - item said that hiding a row can empty a column. **That was wrong.** A row dropped for - having no value among the window's columns contributes no value to any of those - columns. So on a fixed window, one pass that drops the rows and columns empty within - the window already is the fixpoint (ADR 008 revision 21, question 61, makes the same - argument). The iteration comes from the fill step, which brings in rows and columns from - outside the window. ADR 008's question 66 records that loop. -- **The loop the user describes, and what makes it converge cheaply.** The user, in chat: - "The problem pruning solves is visualizing sparse datasets: screens that are mostly - empty are useless and so you drop out categories from rows and columns until every - visible row or column has at least one value on screen." (relayed on 2026-10-05; on no - PR). The criterion is a fixpoint on the displayed grid. *The author's analysis:* a client - alternates three steps. It fetches a window. It prunes, removing every row with no - value in the visible columns and every column with no value in the visible rows, until - nothing changes. It fills, asking for the next rows (or columns) past its last absolute - position that have a value in the columns (or rows) still visible. Three things the - circuit must answer for that loop: - - **"Any value in row r among columns S", and its batched form**, the next k such rows - from p. With index (ii) the batched form costs O(|S| × log n + k × |S|) per call, keeps - no state per request, and is recomputed for each request. That is the "as stateless - as possible" side of it. Without an index, each candidate row costs |S| store lookups, - and a sparse dataset puts most candidates in empty space. That is exactly the case - pruning exists for, so the index is not optional at 3 × 1,000. - - **Maintenance is incremental and cheap per write.** A cell that starts or stops - holding a value inserts or removes one entry in one column's list, at O(log n). Index - (ii) is a set of `(column, row)` pairs, so it is itself a Z-set fed from the cell delta - and from formula stems. Its delta is "row r gained or lost a value in column c". - - **The answers are friendly to deltas.** A client subscribed to a window can be sent - just those `(row, column)` changes that fall in its window's columns (clause 12's - subscription, restricted to S). It re-runs its own prune step locally, and it fills - again only when a row or column has been removed. So the host never re-runs a - client's fixpoint. +**Superseded: the pruned top-left under model-side pruning.** These options assume the +host judges "pruned", which holds only under model-side pruning. They are kept so that +the user's objection is recorded with what it answered. +- (a) **Fail, then retry** (C10's tentative "maybe"). The error names the nearest + occupied successor and predecessor. +- (b) **Snap forward and report the actual top-left.** **The user objects** (C11). *The + author's reading of the reason:* a client scrolling up asks for a top-left just above + its window. If that row is empty, (b) snaps forward to the window's old top-left, so the + window never moves past an empty stretch. +- (c) **Snap in the scroll direction.** It needs the request to carry its direction. - Whether the loop always terminates is **not proven**. Within a fixed candidate set, - iterated removal reaches the greatest fixpoint, which is unique (the union of any two - sets with the property has it too, as ADR 008 revision 21's question 61 also says). On - a fixed window that takes one pass. The fill only moves forward through absolute positions, so it - adds finitely many candidates before the axis ends. A client that bounds its fills per - frame stays within ADR 008's frame budget whether or not the loop has converged. That - bound is a client policy (Q42). -- **What this changes in this ADR.** If pruning is a client filter over the window, - nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The - pruned axis, its ordered index and "row i of n" are not needed either, because a - window-relative filter has no global n. What remains model-side is **which cells hold a - value**, including formula stems, which only the model can evaluate. ADR 008's question - 1 ("go with occupied") is about that, and it still applies, to the cells of a window. Its - question 11 ("prune empty should be the default") becomes a default for a client's - display. Whether clause 9's occupancy should be restated as the index of (ii), and - whether ADR 008 clause 3's `Occupied` axis survives, is question 22. - -Whether a pruned response is streamed or returned whole, and what a client sees while a -fill is incomplete, is question 21. +Neither (a) nor (c) can get stuck. A retry under (a) goes to a named neighbour that is +strictly beyond the old top-left in the scroll direction, and occupied. A write that +empties it only delays the search, by at most the gap. (c) moves in the direction asked, +and stays put only at the end of the axis. At the top of the axis, (a) names only the +successor and reports that there is no predecessor, and (c) returns the first occupied +position and reports that it is the top. So neither bounces. An axis with nothing +occupied answers "empty axis". Both need the predecessor query. **What network transparency requires** (T1 to T6, the author's): - **T1. Results are plain data.** Every element on an output stream is a value the @@ -1592,6 +1535,11 @@ fill is incomplete, is question 21. basis that describes them. This does not break gotcha 42's "never coalesce". That rule is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple identity, or by a stable cross-product position, is not positional in that sense. + **Coalescing is therefore per subscription, never per stream.** All of a + subscription's pending deltas, the basis included, are summed together up to one step, + and sent as one batch in the fixed order. Coalescing one stream while another is sent as + it arrives would let a basis for step n+2 overtake the cells it describes, which is + gotcha 43's failure. - **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives the same result and delta types as a remote client. Its transport is a function call. That is what makes the requirement testable: run one edit sequence through two @@ -1609,7 +1557,7 @@ consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that bo The alternative is a `serde` feature on `zset`. The author leans to the conversion, because what crosses is the projected result, not the crate's own element type. -**How stateless the host can be.** The user: "as stateless as possible". These are the +**How stateless the host can be.** The user: "as stateless as possible" (C2). These are the options of question 13, and ADR 008's question 52 asks the push half of the same thing. - **(a) Pure fetch.** The host keeps nothing per client. A client calls `project(request)`. It learns its result may be stale from a change notice addressed to @@ -1623,7 +1571,7 @@ options of question 13, and ADR 008's question 52 asks the push half of the same eviction, a reconnect), a client resumes by resending its request and the last step it applied, and the host answers with a snapshot. The host could instead keep a short ring of past deltas per shape; the author's lean is no ring at first. So the user's "recent - projections + clients" are this table plus clause 7's shape cache, and **both can be + projections + clients" (C2) are this table plus clause 7's shape cache, and **both can be rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). - **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, @@ -1735,7 +1683,8 @@ orders come from the clauses and are not measured. (ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration -step 4 must. +step 4 must. It reports both as ratios against an unrelated operation on the same run, +as ADR 008's question 13 requires, not as bare milliseconds. ## Alternatives considered @@ -1792,7 +1741,7 @@ displayed path until the differential test holds. O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's per-effect operator shape. The formula delta arrives + delta, all following clause 7's shape key and read-time parameters. The formula delta arrives only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the frame observation) depend on this step.** @@ -1849,9 +1798,9 @@ longer a prerequisite of step 6. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. - A mutating frame from a client already clones the workbook for rollback - (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) + (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- The operator shape is re-read after every model effect: O(items) per live shape, a cost the +- The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. - Per-shape state lives only as long as requests name it. A shape that comes back after @@ -1919,7 +1868,7 @@ revision, and none is renumbered. *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. *[Open. ADR 008's flagged reading 1 makes the page selection a parameter of the request, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so +7. *[Open. ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the @@ -1947,20 +1896,22 @@ revision, and none is renumbered. *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. -11. *[Open, and affected by question 16. Under item grain a reorder of an axis category's items rebuilds nothing, so the recommendation's premise (key by position and rely on `view_shape`'s rebuild) no longer holds. Keying by item identity and reading the order at read time, which clause 12 already needs (question 20), becomes the natural form. The recommendation is left as written until the user answers both.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). - Should `LAST`/`FIRST` key their per-group state by the ordering category's item - position, with that category's item order added to the operator shape so that a - reorder rebuilds them? Or should they key by stable item identity (ADR 007), and - look the order up at read time? - *Recommendation:* item position. Reads stay O(log d), like `Min`/`Max`, and the - same generic operator serves all four. The order of **drawn** items is already in - `view_shape` (`app.rs:47-52`), so a reorder of drawn items already rebuilds, and - nothing new is needed for them. The one open part is **hidden** items, whose - positions `view_shape` does not hash. Recommended: `LAST`/`FIRST` order only the - values the fold covers, which are the drawn members. Then the operator shape needs - no widening at all, and ADR 008's text and cross-references hold unchanged. If the - user wants hidden items ordered too, the `measure_agg` part must also cover their - positions. +11. *[Open. The recommendation was revised on 2026-10-05 by the author, because clause 7 + now makes the drawn order a read-time parameter; the user has answered neither + version.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and + improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering + category's item position, and rebuild on a reorder? Or should they key by stable item + identity (ADR 007), and look the order up at read time? + *Recommendation (revised):* neither as first stated. Key each entry by an **order + label**: a key that compares items in their current order and is not renumbered when + an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's + question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator + serves all four. An insert or a reorder relabels only the items that moved, and + rebuilds nothing. This is not measured. `LAST`/`FIRST` order only the values the fold + covers, which are the drawn members. + *As first written* (superseded): key by item position, relying on `view_shape`'s + rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn + order, so that rebuild does not happen, and a position key would go stale. 12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). Should the host maintain each column's whole-column maximum display width? That is @@ -1972,19 +1923,21 @@ revision, and none is renumbered. not cure. 13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per - client** (clause 12; ADR 008's question 52 asks the push half). *Recommendation (the - author's):* (b), soft-state subscriptions that can be rebuilt from the request and the - last step, with no copy of a client's window on the host. When its table is lost, (b) - falls back to (a). (c) is obsolete under ADR 008's split. One of the two ADRs should - hold this question, and the other should point to it. + client** (clause 12). **Owner: this ADR.** ADR 008's question 52 asks the push half, + and should point here. *Recommendation (the author's):* (b), soft-state + subscriptions that can be rebuilt from the request and the last step, with no copy of + a client's window on the host. When its table is lost, (b) falls back to (a). (c) is + obsolete under ADR 008's split. -14. *[New.]* **What keys operator state, and how long a shape lives** (clause 7). Is it - keyed per projection shape, cached and evicted (the author's)? Or per saved `.improv` - view only, so that a request whose shape is unsaved (an unsaved prune or hide, ADR - 008's question 51) is served without incremental state? *Recommendation:* per shape, - cached, and evicted once no request has named it for a while. The time is an - implementation number. An unsaved shape is then as incremental as a saved one, which - keeps "pruning shouldn't infect every client" from costing an eager path. +14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state + selected by a request's **shape key** (axes by level, None categories, filter keys, + page categories, `measure_agg`), cached and evicted, with every other parameter + applied at read time (the author's)? Or is it kept per saved `.improv` view only, so + that a request with an unsaved key is served without incremental state? + *Recommendation:* per shape key, cached, and evicted once no request has named it for + a while. The time is an implementation number. An unsaved hide, collapse, reorder, + page step or prune then shares the saved view's key and costs nothing. Only an + unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2003,93 +1956,84 @@ revision, and none is renumbered. *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the order the user's answer to ADR 008's question 17 already sets. -18. *[New.]* **Formulas the per-record map cannot express** (clause 5a). ADR 008's - question 15 sends them to a fallback over the occupied contexts, never to - `Expansion::members`; the user's answer was "the recommendation is fine.". Two parts: - - Does this ADR's `Eager` adopt the same rule, so that `Eager` and `Incremental` - agree and this ADR's differential test needs no carve-out? - - Do the formulas that target a category other than `_Measure` take the same +18. *[New; premise corrected on 2026-10-05.]* **Formulas the per-record map cannot + express** (clause 5a). ADR 008's question 15 sends them to a fallback over the + occupied contexts, never to `Expansion::members`; the user's answer was "the + recommendation is fine.". *As first written*, this question said the fallback changes + `Eager`'s answers. **The code says it does not.** At `50e1314d`, `agg_values` evaluates + the expression only over `member_records` of each context (`types.rs:91-139`; `member_records` at `:47-53`). So a + context that holds no record contributes no value, and no error, whatever the + expression is. The fallback changes the cost, not the answer. And today's eager path + reaches `Expansion::members` only at a fold or a total (`contexts_of`, + `types.rs:916-925`). At a stem it pools one context. Two parts remain: + - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be + a cost change with bit-equal answers, since dropping empty contexts keeps the order + of the rest, and the differential test then needs no carve-out. + - Should the formulas that target a category other than `_Measure` take the same fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's - question 1 and question 15 cases, as a stated change of behaviour. + *Recommendation:* yes to both, with no change of behaviour. This was argued from the + code and not run. The differential test pins it. -19. *[New. The user was "still ambivalent", then said in chat that pruning "sounds like a UI side concern"; see question 22. Not confirmed as a decision.]* **Which side of the wire prunes** (clause 12; - the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, - and one ADR should hold it). On the model side, `project` - returns occupied rows and columns, and the circuit answers next-occupied and - rank/select queries for it. On the client side, `project` returns the cross-product - window together with a skip structure, and the client asks for the queries itself. - *Recommendation (the author's), revised after the user's later chat:* the client - side, as the user leans. The model offers windows and question 22's restricted skip - query, so a sparse grid still never ships empty space. *As first written*, this - recommended the model side; that is withdrawn. +19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side + concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** + (clause 12). **Owner: ADR 008** (its question 57), because it is a question about the + view/host split. This question records what each answer asks of the circuit. On the + model side, the circuit keeps a pruned axis per shape key, with rank and select. On + the client side, it answers question 22's restricted successor query, and the client + prunes. *Recommendation (the author's), revised after C6 to C8:* the client side. The + model offers cross-product windows and the restricted successor query, so a sparse + grid still never ships empty space. *As first written*, this recommended the model + side; that is withdrawn. -20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left - should be as absolute as possible"; ADR 008's question 58 asks the same from the - request side). A cross-product position stays fixed when - occupancy changes, but it moves when an item is inserted, removed or reordered. Should - deltas be keyed by tuple identity, with absolute positions attached when they are - sent, and should an item edit be sent as an axis change, so that clients re-derive - their positions arithmetically? *Recommendation:* yes. It costs rank and select over - each category's items (ADR 008 clause 10), and comparisons in the occupancy index at - O(levels × log items) each, which is not measured. It is compatible with ADR 008's - question 58 recommendation, which makes a tuple of item ids the primary form of the - top-left. Whether hidden items belong to the absolute space is ADR 008's question 58. - Under item grain (clause 7) this ADR serves either answer, because hiding is a remap. +20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR for the + delta keys; ADR 008 (its question 58) for the form of the request's top-left.** A + cross-product position stays fixed when occupancy changes, but it moves when an item + is inserted, removed or reordered. Should deltas be keyed by tuple identity, with + absolute positions attached when they are sent? And should an item edit be sent to + every subscriber of the key as an axis change, so that clients re-derive their + positions arithmetically? *Recommendation:* yes. It costs rank and select over each + category's items (ADR 008 clause 10), at O(levels × log items) per lookup, which is + not measured. It is compatible with ADR 008's question 58 recommendation, a tuple of + item ids as the primary form of the top-left. Item grain (clause 7) serves either + answer to whether hidden items belong to the absolute space, because hiding is a + remap. -21. *[New.]* **A pruned answer as a stream** (clause 12; the user: "After pruning we - might have to stream rows and columns to attempt to fill out the full grid"). Should - `project` under pruning fill forward from the absolute top-left, as a sliced job that - can be resumed from its fill front, and send each found row and column as a stamped - delta? Should a row that becomes occupied behind the front arrive as an ordinary - occupancy delta, and leave the window one row larger than asked, for the client's - policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one - stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 - asks the same, and asks in its part (c) whether rows or columns fill first. Under the - user's later chat (question 22), the fill loop is the **client's**: it calls the - restricted skip query once per step of its row and column fixpoint. The host then - keeps no fill state at all, which is more stateless than the sliced host job this - question first described. +21. *[New.]* **How a pruned answer streams** (clause 12; C5). **Owner: ADR 008** (its + question 59) for the semantics: when a fill stops, and in what order rows and columns + fill. **This ADR** owns the delivery. *Recommendation:* under client-side pruning the + fill is the client's, one restricted successor query per step, and the host keeps no + fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, + so a fill and live updates are one stream. Under model-side pruning the same holds, + with the fill a sliced host job resumable from `(request, fill front)`. 22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause - 12; the user, in chat: "Pruning is just a filter applied to full columns a rows ... - That sounds like a UI side concern" and "a value being present outside the viewport - doesn't matter"). This changes today's semantics, which prune against the whole - grid. Three parts: - - (a) Should the circuit offer a **restricted skip query**, "the next k rows from - absolute position p with a value in at least one of these columns", and the same for - columns? How should it answer: (i) per-axis occupancy as a pre-filter, (ii) a column - index of occupied rows, merged, or (iii) a row index of occupied columns? - - (b) Should clause 9's whole-extent, per-axis occupancy be **restated** as that - index, and ADR 008 clause 3's `Occupied` axis and its "row i of n" be dropped, - since a window-relative filter has no global count? - - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty - should be the default") need restating? Question 1 would become the rule for which - cells of a window hold a value, and question 11 a client's display default. + 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. + Four parts, with owners: + - (a) **Owner: this ADR** (ADR 008's question 62 should point here). Should the + circuit offer a **restricted successor query**, "the next k rows after absolute + position p with a value in at least one column of S", with S in the call, and the + same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, + (ii) a column index of occupied rows, merged, or (iii) a row index of occupied + columns? Under C9 it only ever looks forward. + - (b) **Owner: this ADR for clause 9; ADR 008 (its question 65) for its `Occupied` + axis.** Should clause 9's whole-extent, per-axis occupancy be restated as index + (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it + only to skip. That skips only rows empty everywhere, not rows empty in S, so on its + own it does not answer (a). + - (c) **Owner: ADR 008.** Do its question 1 ("go with occupied") and question 11 + ("prune empty should be the default") need restating? Question 1 would become the + rule for which cells of a window hold a value, and question 11 a client's display + default. + - (d) **Owner: ADR 008 for the semantics; this ADR records the costs.** What happens + when scrolling up or left, and when a requested top-left lands on an empty row + (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It + probes with the restricted successor query, and it needs responses to report actual + positions. It costs about 14 to 28 probes at density 10^-4, by arithmetic. The + earlier fail, snap-forward and direction-snap options are kept there as superseded, + with the user's objection to snapping forward (C11). - ADR 008 revision 21 asks the same from its side: its question 62 (the skip query), - question 65 (whether "occupied" stays whole-extent), question 61 (one pass or a - fixpoint) and question 66 (the fill loop). Its question 65 recommends keeping per-axis, whole-extent occupancy and - using it only to skip. Under (i) above, that skips only rows that are empty - everywhere, not rows that are empty in the displayed columns. So it does not answer - the restricted query by itself. - The user's later statement makes the criterion a fixpoint on the displayed grid, - reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is - what each fill step calls. - Under the user's right-and-down growth (clause 12), the query is a forward successor - query only, and so are all three options; no rank or reverse scan is needed for - pruning. - The user's later chat adds a client-side optimisation for scrolling up or left (a - predecessor search, which every option serves as cheaply as a successor search). It - also raises the edge case of a top-left that lands on a pruned row: fail and retry, - snap forward (which the user objects to: "the viewport getting stuck"), or snap toward - the scroll direction (clause 12). The user has since suggested a simpler client procedure: - gallop the top-left with forward requests until one hits, then narrow. A pruned - top-left then needs no special answer, provided responses report actual positions. - The author now takes that as the baseline, and keeps a predecessor query only as an - optional optimisation. That edge case is - part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms - that pruning is the client's. (c) restate both in those terms, in ADR 008, since they - are ADR 008's answers. + that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the + gallop as the baseline, and a predecessor query only as an optional optimisation, + offered once a measured round-trip cost justifies it. From 255e3d6d96ada6e95a451c23fb5933db00f7abcd Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:32:39 -0700 Subject: [PATCH 035/106] docs(adr): ADR 009 reads ADR 008 revision 25: the ref edge in clause 5a and the proposed question owners (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 80 +++++++++++++---------- 1 file changed, 47 insertions(+), 33 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 3fc94606..6720b9a2 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 24** -(PR #308, read at `3180327b`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 25** +(PR #308, read at `5c208f54`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `3180327b`, its revision 24) · +reads it at `5c208f54`, its revision 25) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 24 of `context/adr/008-virtualized-grid.md` on PR #308, at `3180327b`. Clauses 3, +revision 25 of `context/adr/008-virtualized-grid.md` on PR #308, at `5c208f54`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 24, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 25, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -726,13 +726,17 @@ product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's **store-driven fallback**, which evaluates the expression only over the **occupied contexts**, the contexts some record projects to. The user accepted that recommendation, as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as -eager's**, as the paragraph above says of the pipeline. `agg_values` evaluates the -expression only at the `member_records` of each context (`types.rs:47-53`, `:91-139`), -so a context with no record contributes no value and no error, whatever the expression. -The fallback changes the cost and not the answer. *The first revision of 2026-10-05 said -it "differs from today's eager answer" wherever the expression is non-empty at an empty -context. That was wrong, and ADR 008's question 15 carries the same premise.* Whether -this ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: +eager's, with one edge.** `agg_values` evaluates the expression only at the +`member_records` of each context (`types.rs:47-53`, `:91-139`), so a context with no +record contributes no value, as the paragraph above says of the pipeline. **The edge:** +an inner bare `Expr::Ref` that names no item errors `ref:` at the top of +`agg_values`, for every context it is called with, records or not. Evaluated over +**zero** occupied contexts the call never happens, and the aggregate yields its empty +value instead of the error. ADR 008 revision 25 found this edge, and it is question 15's +whole difference. With at least one occupied context the error is the same. *The first +revision of 2026-10-05 said the fallback differs "wherever the expression is non-empty at +an empty context", and a later one said it never differs. Both were wrong.* Whether this +ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -1147,7 +1151,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 24 at `3180327b`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 25 at `5c208f54`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1408,8 +1412,10 @@ search appears (next paragraph). What forward-only growth simplifies: - **The index options.** Each needs only a seek to p and forward iteration, so any ordered structure with a lower-bound seek and a forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no - rank or "row i of n" is needed. A scrollbar over the unpruned cross product gets its - rank by arithmetic. + rank over **occupied tuples**, and no "row i of n", is needed. Rank and select over + **each category's items** stay, because a cross-product position is built from them + (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its + rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas only**. An item edit is an axis change and goes to every subscriber (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill @@ -1960,20 +1966,25 @@ revision, and none is renumbered. express** (clause 5a). ADR 008's question 15 sends them to a fallback over the occupied contexts, never to `Expansion::members`; the user's answer was "the recommendation is fine.". *As first written*, this question said the fallback changes - `Eager`'s answers. **The code says it does not.** At `50e1314d`, `agg_values` evaluates - the expression only over `member_records` of each context (`types.rs:91-139`; `member_records` at `:47-53`). So a - context that holds no record contributes no value, and no error, whatever the - expression is. The fallback changes the cost, not the answer. And today's eager path + `Eager`'s answers wherever the expression is non-empty at an empty context. **The code + says it does not, except at one edge.** At `50e1314d`, `agg_values` evaluates the + expression only over the `member_records` of each context (`types.rs:91-139`; + `member_records` at `:47-53`), so a context that holds no record contributes no + value. The edge is an inner bare ref naming no item. It errors for every context it + is called with, so over zero occupied contexts it yields the empty value instead of + `ref:` (clause 5a; ADR 008 revision 25). And today's eager path reaches `Expansion::members` only at a fold or a total (`contexts_of`, `types.rs:916-925`). At a stem it pools one context. Two parts remain: - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be - a cost change with bit-equal answers, since dropping empty contexts keeps the order - of the rest, and the differential test then needs no carve-out. + a cost change with bit-equal answers, except at the ref edge, since dropping empty + contexts keeps the order of the rest. The differential test then carves out only + that edge, by name. - Should the formulas that target a category other than `_Measure` take the same fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - *Recommendation:* yes to both, with no change of behaviour. This was argued from the - code and not run. The differential test pins it. + *Recommendation:* yes to both. The only change of behaviour is the ref edge, which + could also be kept by checking the ref once before enumerating. This was argued from + the code and not run. The differential test pins it. 19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** @@ -1986,8 +1997,9 @@ revision, and none is renumbered. grid still never ships empty space. *As first written*, this recommended the model side; that is withdrawn. -20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR for the - delta keys; ADR 008 (its question 58) for the form of the request's top-left.** A +20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR**, for the + delta keys and the form of the top-left both, as ADR 008 revision 25 proposes. Its + question 58 points here. A cross-product position stays fixed when occupancy changes, but it moves when an item is inserted, removed or reordered. Should deltas be keyed by tuple identity, with absolute positions attached when they are sent? And should an item edit be sent to @@ -1999,9 +2011,9 @@ revision, and none is renumbered. answer to whether hidden items belong to the absolute space, because hiding is a remap. -21. *[New.]* **How a pruned answer streams** (clause 12; C5). **Owner: ADR 008** (its - question 59) for the semantics: when a fill stops, and in what order rows and columns - fill. **This ADR** owns the delivery. *Recommendation:* under client-side pruning the +21. *[New.]* **How a pruned answer streams, and the fill step** (clause 12; C5). **Owner: + this ADR**, as ADR 008 revision 25 proposes. Its questions 59 and 66 point here. That + includes when a fill stops, and in what order rows and columns fill. *Recommendation:* under client-side pruning the fill is the client's, one restricted successor query per step, and the host keeps no fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, so a fill and live updates are one stream. Under model-side pruning the same holds, @@ -2010,22 +2022,24 @@ revision, and none is renumbered. 22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. Four parts, with owners: - - (a) **Owner: this ADR** (ADR 008's question 62 should point here). Should the + **Owner: this ADR** for the whole question, as ADR 008 revision 25 proposes (its + questions 61, 62 and 65 point here). Part (c) is the exception: it restates answers + ADR 008 records, so its text can only change there. + - (a) Should the circuit offer a **restricted successor query**, "the next k rows after absolute position p with a value in at least one column of S", with S in the call, and the same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, (ii) a column index of occupied rows, merged, or (iii) a row index of occupied columns? Under C9 it only ever looks forward. - - (b) **Owner: this ADR for clause 9; ADR 008 (its question 65) for its `Occupied` - axis.** Should clause 9's whole-extent, per-axis occupancy be restated as index + - (b) Should clause 9's whole-extent, per-axis occupancy be restated as index (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it only to skip. That skips only rows empty everywhere, not rows empty in S, so on its own it does not answer (a). - - (c) **Owner: ADR 008.** Do its question 1 ("go with occupied") and question 11 + - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. - - (d) **Owner: ADR 008 for the semantics; this ADR records the costs.** What happens + - (d) What happens when scrolling up or left, and when a requested top-left lands on an empty row (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It probes with the restricted successor query, and it needs responses to report actual From 15cb987a938149260a980916442baa1d7217b32b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:39:32 -0700 Subject: [PATCH 036/106] docs(adr): ADR 009 rebuilds column totals on a row hide, and fixes the review's nits (improvise-felo.4) Today a column total pools only the drawn rows, so a hide changes it; clause 7 now rebuilds totals on a row-axis hide or show by default (B), with read-time subtraction (A) as question 23. Hidden page-category items are filter keys; step 8, the stem path, the order label's cost, the ADR 008 chat-list note and two citations are corrected. Reads ADR 008 at 99322d0e. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 119 ++++++++++++++++------ 1 file changed, 88 insertions(+), 31 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 6720b9a2..e034ecae 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,11 +1,11 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 25** -(PR #308, read at `5c208f54`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 26** +(PR #308, read at `99322d0e`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `5c208f54`, its revision 25) · +reads it at `99322d0e`, its revision 26) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 25 of `context/adr/008-virtualized-grid.md` on PR #308, at `5c208f54`. Clauses 3, +revision 26 of `context/adr/008-virtualized-grid.md` on PR #308, at `99322d0e`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 25, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 26, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -86,7 +86,9 @@ and says where each statement came from. - **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the - PR, but spread over several groups of its own. This list is the one place this ADR + PR, but spread over several groups of its own. ADR 008's list also has six more (its + items 1 to 6: the five standing design statements, and the "q19 … q24" corrections). + This ADR relies on none of those six. This list is the one place this ADR quotes them, and the clauses cite them by number. None of this answers this ADR's own questions 1 to 11. @@ -248,7 +250,7 @@ a change proportional to what the change touches, not to the document. - **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes every cell before and after a cell-writing batch. `document_content` (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is - re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) + re-read after **each** `CellData` effect (`app.rs:1119-1125` at `50e1314d`). So the first is O(cells) per batch and the second O(cells) per effect, whatever was written. - **Every mutating frame from a client already clones the document.** `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, @@ -439,7 +441,7 @@ because it observes `view_shape` and not cells. **The dirty flag's content half does not move onto the cell delta alone.** `document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it -is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, +is re-read after every `CellData` effect (`app.rs:1119-1125` at `50e1314d`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a non-empty cell delta means dirty" would never save a formula edit in the browser. The @@ -544,9 +546,15 @@ key, and `FIRST` the least. superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a reorder rebuilds nothing, and a state keyed by position would go stale. The position of a **hidden** item matters only if `LAST`/`FIRST` must order values that come from - hidden items. The alternative is to key by a stable item - identity (ADR 007) and look the order up at read time. That makes a reorder free, but - a read then costs O(d). This is question 11. + hidden items. There are three keys: + - **position**, which goes stale on a reorder now that nothing rebuilds; + - **stable item identity** (ADR 007), with the order looked up at read time, which + makes a reorder free, but then a read costs O(d); + - an **order label**, which compares in the current order and is not renumbered by an + insert elsewhere. Reads stay O(log d). A reorder relabels the moved items, and + re-keys every group entry stored under them, at O(entries under the moved items). + + This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a delta (retract the old value, insert the new), and the outer consumes it. So the @@ -755,7 +763,8 @@ occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as - today, through `Expansion::members`. + today: through `Expansion::members` at a fold or a total, and over the one context at a + stem. - **From step 5**, the record stage, the map and the index are maintained, so the cross product is gone. `Count`, `Min` and `Max` over the mapped values, and the error channel, are read from the maintained state. They are order-free, or ordered by the @@ -812,14 +821,16 @@ states the same rule. - the categories of each axis, by level; - the None categories; - the filter keys: each `Axis::Filter` category with its selected item, and the hidden - items of every category that is **not** on an axis (such an item drops cells from the - fold, so it filters); + items of every category on None, Page or Filter, as ADR 008 clause 3 also says. Such + an item drops cells from the fold, so it filters. A hidden item of a **page** + category is a filter key, not a read-time parameter; - the page **categories**, which key the slices under question 7's recommendation; - `measure_agg`. Everything else a request carries is a **read-time parameter**, applied when the - output is read, without touching operator state: - - the hidden items of an axis category; + output is read, without touching occupancy or cell operators: + - the hidden items of an axis category, which also rebuild the column totals when they + are on the row axis (see "Hiding a row moves the column totals" below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -850,15 +861,16 @@ states the same rule. - **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column aggregates are keyed by the **items** of the axis categories, together with the filter and page keys. They are never keyed by a position in a level sequence. So a collapse, - expand, hide or show of an **axis** category's items changes no operator. The visible - axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms + expand, hide or show of an **axis** category's items changes no occupancy or cell + operator, and the visible axis is a remap of the item-grain state. **The exception is + the column totals:** a hide or show of a row-axis item rebuilds them (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other operators, which a new key builds as a job: - a change of which categories are its levels or on None; - - a change of its filter keys, which includes hiding an item of a category that is not - on an axis; + - a change of its filter keys, which includes hiding an item of a category on None, + Page or Filter; - a change of its page categories; - a change of `measure_agg`. @@ -868,6 +880,30 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. +- **Hiding a row moves the column totals (the author's default; question 23).** Today a + column total pools only the **drawn** rows. The total's evaluator excludes hidden items + through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` + evaluates over that memo (`:1066-1070`). The test + `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take + `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide + Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, + and no read-time parameter can take Feb back out of it. So: + - **(B), the default:** a hide or show of a **row-axis** item rebuilds the column + totals of that shape, as a job. ADR 008 says the same ("totals still rebuild on hide + and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and + expand do not move a total, because it pools the members either way, which is what + the test above pins. A hide or show of a column-axis item only hides or shows that + column's total. + - **(A), the open alternative:** build each total from clause 4's (row tuple, column + tuple) group state, and drop the hidden rows when the total is read. For `Sum` and + `Count` that is O(hidden rows) per read: subtract the hidden rows' state. That is + exact only under question 3's (c), or under ADR 005's wide accumulator, because + f64 subtraction is not exact. For `Min`, `Max` and the one-distinct-value text rule + it is O(rows) per read, a recombination of every visible row's state. A hide then + costs nothing to install, and reads cost more. + + The author takes (B), because it matches the code and ADR 008, and it is the smaller + change. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1151,7 +1187,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 25 at `5c208f54`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 26 at `99322d0e`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1194,8 +1230,10 @@ host-held mirror of a client. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate at item grain, and the formula layer re-evaluates only the columns whose - inputs moved, through clause 2's trait. Clause 5 replaces `Expansion::members`. Totals + aggregate at item grain over the drawn rows, and the formula layer re-evaluates only + the columns whose inputs moved, through clause 2's trait. A hide or show of a row-axis + item rebuilds the totals (clause 7, "Hiding a row moves the column totals"; question + 23). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1761,11 +1799,12 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** The totals rebuild on a hide or + show of a row-axis item (clause 7), unless the user takes question 23's (A). 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, - snapshot-then-delta, the order-statistic queries over occupancy, and the test with two + snapshot-then-delta, question 22's restricted successor query, and the test with two consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -1806,6 +1845,8 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). +- A hide or show of a row-axis item rebuilds the column totals of that shape, as a job, + under question 23's default (clause 7). Occupancy and cells do not rebuild. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -1820,7 +1861,7 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 22 were added by the 2026-10-05 +none of questions 1 to 11 on #309.** Questions 12 to 23 were added by the 2026-10-05 revision, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -1913,7 +1954,9 @@ revision, and none is renumbered. an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator serves all four. An insert or a reorder relabels only the items that moved, and - rebuilds nothing. This is not measured. `LAST`/`FIRST` order only the values the fold + rebuilds nothing. **But each relabelled item re-keys every group entry stored under + it**, so a reorder costs O(entries under the moved items), not O(moved items). This + is not measured. `LAST`/`FIRST` order only the values the fold covers, which are the drawn members. *As first written* (superseded): key by item position, relying on `view_shape`'s rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn @@ -1941,9 +1984,11 @@ revision, and none is renumbered. applied at read time (the author's)? Or is it kept per saved `.improv` view only, so that a request with an unsaved key is served without incremental state? *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while. The time is an implementation number. An unsaved hide, collapse, reorder, - page step or prune then shares the saved view's key and costs nothing. Only an - unsaved change of axes, filters or `measure_agg` builds a new key, as a job. + a while. The time is an implementation number. An unsaved collapse, reorder, page + step or prune then shares the saved view's key and builds no operator. An unsaved + hide or show of a row-axis item shares the key too, but it rebuilds that request's + column totals (question 23's default). Only an unsaved change of axes, filters or + `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2051,3 +2096,15 @@ revision, and none is renumbered. that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the gallop as the baseline, and a predecessor query only as an optional optimisation, offered once a measured round-trip cost justifies it. + +23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column + totals"). Today a column total pools only the drawn rows, so hiding a row changes it + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) + aggregate cannot remove a hidden row at read time. Should a hide or show of a + row-axis item **(B)** rebuild that shape's column totals as a job? Or **(A)** should + totals be read from the (row tuple, column tuple) group state, dropping the hidden rows + when read? (A) costs O(hidden rows) per read for `Sum` and `Count`, which is exact only + under question 3's (c) or ADR 005's accumulator. For `Min`, `Max` and the text rule it + costs O(rows) per read. *Recommendation (the author's):* (B). It matches today's code + and ADR 008's finding 10, and it is the smaller change. (A) is the alternative if a + measured hide-heavy workload shows the rebuild is too slow. From 405d433732f34662cea0c6c5a9efd82a706e5a10 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:41:30 -0700 Subject: [PATCH 037/106] docs(adr): ADR 009 keys column totals by the hidden row set, because the shape key is shared (improvise-felo.4) Plain (B) is withdrawn: two clients on one shape key with different hidden rows would share one wrong total. The author now leans to (B'), totals per (shape key, hidden row-axis set), with (A) and a mix still open in question 23; question 14 states the cache cost. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 91 ++++++++++++++++------- 1 file changed, 64 insertions(+), 27 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e034ecae..9525ae13 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -829,8 +829,9 @@ states the same rule. Everything else a request carries is a **read-time parameter**, applied when the output is read, without touching occupancy or cell operators: - - the hidden items of an axis category, which also rebuild the column totals when they - are on the row axis (see "Hiding a row moves the column totals" below); + - the hidden items of an axis category. For the column totals only, the hidden + row-axis set selects a totals entry (see "Hiding a row moves the column totals" + below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -863,7 +864,8 @@ states the same rule. and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no occupancy or cell operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item rebuilds them (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms + the column totals:** a hide or show of a row-axis item selects or builds a totals entry + (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other @@ -880,7 +882,7 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default; question 23).** Today a +- **Hiding a row moves the column totals (the author's default, (B′); question 23).** Today a column total pools only the **drawn** rows. The total's evaluator excludes hidden items through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` evaluates over that memo (`:1066-1070`). The test @@ -888,8 +890,8 @@ states the same rule. `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, and no read-time parameter can take Feb back out of it. So: - - **(B), the default:** a hide or show of a **row-axis** item rebuilds the column - totals of that shape, as a job. ADR 008 says the same ("totals still rebuild on hide + - **(B):** a hide or show of a **row-axis** item rebuilds the column totals of that + shape, as a job. ADR 008 says the same ("totals still rebuild on hide and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what the test above pins. A hide or show of a column-axis item only hides or shows that @@ -902,8 +904,27 @@ states the same rule. it is O(rows) per read, a recombination of every visible row's state. A hide then costs nothing to install, and reads cost more. - The author takes (B), because it matches the code and ADR 008, and it is the smaller - change. The user decides it in question 23. + **(B) has a defect: the key is shared and the hide is not.** Operator state is shared + by every request with the same shape key (above), and an unsaved hide is a parameter of + one client's request. If client X hides Feb and client Y, on the same key, does not, + (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: + - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and + cells stay shared under the shape key. Totals are kept per (shape key, hidden + row-axis set). Each distinct hidden set in use costs one O(cells) rebuild, as a job, + and one set of per-column aggregates. The totals cache multiplies by the number of + distinct hidden sets that recent requests name, and it is evicted like any key + (question 14). + - **(A)**, as above, needs no per-set state at all. + + *The author's lean, re-evaluated:* **(B′)**. Reads stay O(1) per total. And the common + case shares one entry: clients reading a saved view carry its saved hidden set. An + unsaved hide costs that one client a job and a cache entry, and nobody else anything. + (A) is stateless and installs a hide for free. But its `Min`, `Max` and text totals cost + O(rows) per read, per column drawn, which at 100k rows and tens of columns is unlikely + to fit the frame budget (not measured). And its `Sum` waits for question 3's (c). So + (A) is better only for `Sum` and `Count` once question 3's (c) lands, and a mix (A for + those, B′ for the rest) is possible. Plain (B) without the hidden set in the key is + **withdrawn**. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1228,12 +1249,13 @@ host-held mirror of a client. client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the host pushes it is question 13, which is ADR 008's question 52 seen from this side. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the - pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), + pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at + `50e1314d`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. A hide or show of a row-axis - item rebuilds the totals (clause 7, "Hiding a row moves the column totals"; question - 23). Clause 5 replaces `Expansion::members`. Totals + the columns whose inputs moved, through clause 2's trait. Totals are kept per (shape + key, hidden row-axis set), because a hidden row leaves the total and the key is shared + across clients (clause 7, "Hiding a row moves the column totals"; question 23). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1799,8 +1821,8 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** The totals rebuild on a hide or - show of a row-axis item (clause 7), unless the user takes question 23's (A). + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Totals are kept per (shape key, + hidden row-axis set) under question 23's (B′) (clause 7), unless the user takes (A). 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, @@ -1845,8 +1867,9 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- A hide or show of a row-axis item rebuilds the column totals of that shape, as a job, - under question 23's default (clause 7). Occupancy and cells do not rebuild. +- A hide or show of a row-axis item builds a totals entry per (shape key, hidden + row-axis set), as a job, under question 23's (B′) (clause 7). Occupancy and cells do + not rebuild. The totals cache can hold one entry per distinct hidden set in use. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -1986,9 +2009,12 @@ revision, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key too, but it rebuilds that request's - column totals (question 23's default). Only an unsaved change of axes, filters or - `measure_agg` builds a new key, as a job. + hide or show of a row-axis item shares the key for occupancy and cells. Under + question 23's (B′), it adds a **totals entry** keyed by (shape key, hidden row-axis + set), which is built as a job and evicted like a key. So the totals cache holds up to + one entry per distinct hidden set that recent requests name, not one per key, and a + client that toggles hides through many sets costs a rebuild for each new set. Only an + unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2100,11 +2126,22 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) - aggregate cannot remove a hidden row at read time. Should a hide or show of a - row-axis item **(B)** rebuild that shape's column totals as a job? Or **(A)** should - totals be read from the (row tuple, column tuple) group state, dropping the hidden rows - when read? (A) costs O(hidden rows) per read for `Sum` and `Count`, which is exact only - under question 3's (c) or ADR 005's accumulator. For `Min`, `Max` and the text rule it - costs O(rows) per read. *Recommendation (the author's):* (B). It matches today's code - and ADR 008's finding 10, and it is the smaller change. (A) is the alternative if a - measured hide-heavy workload shows the rebuild is too slow. + aggregate cannot remove a hidden row at read time. The options: + - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, + because the shape key is shared across clients and an unsaved hide belongs to one + client's request. Two clients on one key with different hidden rows would get one + wrong total. + - **(B′)** make the hidden row-axis set part of the key **for totals only**. Each + distinct set in use costs one O(cells) rebuild, as a job, and one set of per-column + aggregates. The cache multiplies by the number of distinct sets (question 14). + - **(A)** read each total from the (row tuple, column tuple) group state, and drop the + hidden rows when it is read. It keeps no per-set state. It costs O(hidden rows) per + read for `Sum` and `Count`, exact only under question 3's (c) or ADR 005's + accumulator, and O(rows) per read for `Min`, `Max` and the text rule. + + *Recommendation (the author's, re-evaluated):* (B′). Reads stay O(1). Clients on a + saved view share its saved hidden set, so the common case is one entry, and an unsaved + hide costs only its own client a job and a cache entry. (A)'s O(rows) reads for `Min` + and `Max` are unlikely to fit the frame budget at ADR 008's fixture (not measured). A + mix is open: (A) for `Sum` and `Count` once question 3's (c) lands, and (B′) for the + rest. From ad60ad2b7ee31e81d6c40d6753ea633cf69568bb Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:45:28 -0700 Subject: [PATCH 038/106] docs(adr): ADR 009 marks (B) withdrawn where it first appears and states ADR 008 revision 28's totals rule (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 9525ae13..770e8d1c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -890,9 +890,11 @@ states the same rule. `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, and no read-time parameter can take Feb back out of it. So: - - **(B):** a hide or show of a **row-axis** item rebuilds the column totals of that - shape, as a job. ADR 008 says the same ("totals still rebuild on hide - and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and + - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** + item rebuilds the column totals of that shape, as a job. ADR 008 once said the same + (its finding 10). It no longer does: its clause 5, at revision 28, says totals are + **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's + question 23. Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what the test above pins. A hide or show of a column-axis item only hides or shows that column's total. From 0cf16b043a9dc2aa168d743f94ab63354e260618 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:46:25 -0700 Subject: [PATCH 039/106] docs(adr): ADR 009 reads ADR 008 at revision 29 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 770e8d1c..b082cfb2 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 26** -(PR #308, read at `99322d0e`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 29** +(PR #308, read at `0edd6f2a`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `99322d0e`, its revision 26) · +reads it at `0edd6f2a`, its revision 29) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 26 of `context/adr/008-virtualized-grid.md` on PR #308, at `99322d0e`. Clauses 3, +revision 29 of `context/adr/008-virtualized-grid.md` on PR #308, at `0edd6f2a`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 26, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 29, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1210,7 +1210,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 26 at `99322d0e`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 29 at `0edd6f2a`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. From 0b066d46cb9ca9aeb9f0bbc6ac89136518c2b7be Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:47:59 -0700 Subject: [PATCH 040/106] docs(adr): ADR 009 corrects option (A)'s total cost to O(H log d) and leans to (A) (improvise-felo.4) A total over all rows can subtract the hidden rows' groups at read time for every aggregate (Min/Max skip fully hidden keys; the text rule counts distinct keys left), so (A) is stateless and cheap; (B') becomes a memo on it. Also: a hidden column item under a collapsed column group, clause 5a totals under both options, and (B')'s cost stated as an upper bound. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 143 +++++++++++++--------- 1 file changed, 85 insertions(+), 58 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b082cfb2..1f904f47 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -829,9 +829,8 @@ states the same rule. Everything else a request carries is a **read-time parameter**, applied when the output is read, without touching occupancy or cell operators: - - the hidden items of an axis category. For the column totals only, the hidden - row-axis set selects a totals entry (see "Hiding a row moves the column totals" - below); + - the hidden items of an axis category. For the column totals, the hidden row-axis + set is subtracted at read time (see "Hiding a row moves the column totals" below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -864,8 +863,8 @@ states the same rule. and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no occupancy or cell operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item selects or builds a totals entry - (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms + the column totals:** a hide or show of a row-axis item changes them, and they subtract + the hidden rows when read (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other @@ -882,7 +881,7 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default, (B′); question 23).** Today a +- **Hiding a row moves the column totals (the author's default, (A); question 23).** Today a column total pools only the **drawn** rows. The total's evaluator excludes hidden items through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` evaluates over that memo (`:1066-1070`). The test @@ -896,15 +895,32 @@ states the same rule. **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's question 23. Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what - the test above pins. A hide or show of a column-axis item only hides or shows that - column's total. - - **(A), the open alternative:** build each total from clause 4's (row tuple, column - tuple) group state, and drop the hidden rows when the total is read. For `Sum` and - `Count` that is O(hidden rows) per read: subtract the hidden rows' state. That is - exact only under question 3's (c), or under ADR 005's wide accumulator, because - f64 subtraction is not exact. For `Min`, `Max` and the one-distinct-value text rule - it is O(rows) per read, a recombination of every visible row's state. A hide then - costs nothing to install, and reads cost more. + the test above pins. A hide or show of a column-axis item hides or shows that + column's total. Under a **collapsed** column group it also changes the group column's + cells and total, because the fold memo skips hidden items on every category. That is a + read-time recombination of the members' item-grain state, and nothing is rebuilt. + - **(A):** keep the per-(column tuple, measure) total state over **all** rows, and + subtract the hidden rows when the total is read. Clause 4 keeps, per group (a row + tuple and a column tuple at item grain), a count of values, a count of numbers, a + running sum and, where `measure_agg` or text needs it, a multiset of values with a + count per key. The column's state is the sum of its groups' states. So at read time + the H cells of the hidden rows in that column are taken out as a **temporary + overlay**, never written back: + - `Count`, and the two counts behind the numeric-first branch: subtract. O(H). + - `Min` and `Max`: walk from the extreme key, skipping each key whose whole count is + hidden. At most H keys are skipped, so O(H log d), with d the distinct keys. + - The one-distinct-value text rule, and `mixed`: count the keys whose whole count is + hidden, and subtract that from the column's number of distinct keys. O(H log d). + Eager's own reading rules (NaN under `PartialEq`, clause 4) apply after the overlay, + unchanged. + - `Sum`: subtract the hidden rows' sums. That is exact only under question 3's (c), + or ADR 005's wide accumulator, because f64 subtraction is not exact. Before step 6 + a `Sum` total stays on today's path anyway. `Avg` is `Sum / Count`. + + So (A) costs **O(H log d) per read for every aggregate**. It is exact, and it keeps no + per-set state. *As first written* (superseded), this item said `Min`, `Max` and the + text rule cost O(rows) per read. That overstated it, as the review of `ad60ad2b` + found. The argument above is the author's and is not measured. **(B) has a defect: the key is shared and the hide is not.** Operator state is shared by every request with the same shape key (above), and an unsaved hide is a parameter of @@ -912,21 +928,28 @@ states the same rule. (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and cells stay shared under the shape key. Totals are kept per (shape key, hidden - row-axis set). Each distinct hidden set in use costs one O(cells) rebuild, as a job, - and one set of per-column aggregates. The totals cache multiplies by the number of - distinct hidden sets that recent requests name, and it is evicted like any key - (question 14). + row-axis set). Each distinct hidden set in use costs at most one O(cells) rebuild, as + a job, and one set of per-column aggregates. The bound is an upper one: an entry can + be derived from the all-rows state by (A)'s subtraction, made once and kept, at O(H) + per column, with `Sum` exact only under question 3's (c). So (B′) is (A) memoised. + The totals cache multiplies by the number of distinct hidden sets that recent + requests name, and it is evicted like any key (question 14). - **(A)**, as above, needs no per-set state at all. - *The author's lean, re-evaluated:* **(B′)**. Reads stay O(1) per total. And the common - case shares one entry: clients reading a saved view carry its saved hidden set. An - unsaved hide costs that one client a job and a cache entry, and nobody else anything. - (A) is stateless and installs a hide for free. But its `Min`, `Max` and text totals cost - O(rows) per read, per column drawn, which at 100k rows and tens of columns is unlikely - to fit the frame budget (not measured). And its `Sum` waits for question 3's (c). So - (A) is better only for `Sum` and `Count` once question 3's (c) lands, and a mix (A for - those, B′ for the rest) is possible. Plain (B) without the hidden set in the key is - **withdrawn**. The user decides it in question 23. + **Aggregates over an expression** (clause 5a) follow the same split. A 5a total's + entries are keyed by context, with the row categories removed. To subtract hidden rows + under (A), the 5a stage's entries must also be indexed by row tuple, at O(entries) more + state, so a read costs O(hidden records). Under (B′), a 5a total per hidden set is its + own map and aggregate stage, at O(records) to build. + + *The author's lean, re-evaluated after the corrected cost:* **(A)**. A read costs + O(H log d), and a hidden set is usually a handful of rows, so a read costs about as + much as (B′)'s O(1) in practice (not measured). (A) keeps no per-set state, which is + what C2's "as stateless as possible" asks, and it never rebuilds or multiplies the + cache. (B′) is kept as a **memo** on top of (A), for a hidden set so large that H log + d per read would not fit the frame budget. The earlier lean to (B′) rested on the + overstated O(rows) cost, and it is **superseded**. Plain (B) without the hidden set in + the key stays **withdrawn**. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1255,9 +1278,10 @@ host-held mirror of a client. `50e1314d`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. Totals are kept per (shape - key, hidden row-axis set), because a hidden row leaves the total and the key is shared - across clients (clause 7, "Hiding a row moves the column totals"; question 23). Clause 5 replaces `Expansion::members`. Totals + the columns whose inputs moved, through clause 2's trait. A hidden row leaves the total + and the key is shared across clients, so under question 23's default (A) a total + subtracts the request's hidden rows when it is read, at O(H log d) (clause 7, "Hiding a + row moves the column totals"). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1823,8 +1847,9 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Totals are kept per (shape key, - hidden row-axis set) under question 23's (B′) (clause 7), unless the user takes (A). + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), + a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs + question 3's (c) for that subtraction to be exact. 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, @@ -1869,9 +1894,9 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- A hide or show of a row-axis item builds a totals entry per (shape key, hidden - row-axis set), as a job, under question 23's (B′) (clause 7). Occupancy and cells do - not rebuild. The totals cache can hold one entry per distinct hidden set in use. +- Under question 23's default (A), every column-total read pays O(H log d) to subtract + the request's hidden rows, and clause 5a's totals need their entries indexed by row + tuple as well. Occupancy and cells do not rebuild. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -2011,11 +2036,12 @@ revision, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key for occupancy and cells. Under - question 23's (B′), it adds a **totals entry** keyed by (shape key, hidden row-axis - set), which is built as a job and evicted like a key. So the totals cache holds up to - one entry per distinct hidden set that recent requests name, not one per key, and a - client that toggles hides through many sets costs a rebuild for each new set. Only an + hide or show of a row-axis item shares the key too. Under question 23's default (A) + its totals subtract the hidden rows at read time and add nothing to the cache. Under + (B′), it would add a **totals entry** keyed by (shape key, hidden row-axis set). Then + the cache would hold up to one entry per distinct hidden set that recent requests + name, and a client that toggles hides through many sets would pay to derive each new + entry. Only an unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate @@ -2127,23 +2153,24 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) - aggregate cannot remove a hidden row at read time. The options: + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). The options: - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, because the shape key is shared across clients and an unsaved hide belongs to one - client's request. Two clients on one key with different hidden rows would get one - wrong total. - - **(B′)** make the hidden row-axis set part of the key **for totals only**. Each - distinct set in use costs one O(cells) rebuild, as a job, and one set of per-column - aggregates. The cache multiplies by the number of distinct sets (question 14). - - **(A)** read each total from the (row tuple, column tuple) group state, and drop the - hidden rows when it is read. It keeps no per-set state. It costs O(hidden rows) per - read for `Sum` and `Count`, exact only under question 3's (c) or ADR 005's - accumulator, and O(rows) per read for `Min`, `Max` and the text rule. + client's request. + - **(A)** keep each total over all rows, and subtract the request's H hidden cells in + that column when it is read, as a temporary overlay. It costs O(H log d) per read + for every aggregate. `Count` subtracts, `Min` and `Max` skip fully hidden keys from + the extreme, and the text rule counts the distinct keys left. It keeps no per-set + state. `Sum` is exact only under question 3's (c) or ADR 005's accumulator. + Clause 5a's totals need their entries indexed by row tuple too. + - **(B′)** keep totals per (shape key, hidden row-axis set). Reads are O(1), but the + cache multiplies by the distinct hidden sets in use. Each entry costs at most one + O(cells) rebuild, or one O(H) derivation by (A)'s subtraction. That makes (B′) a + memo on (A). - *Recommendation (the author's, re-evaluated):* (B′). Reads stay O(1). Clients on a - saved view share its saved hidden set, so the common case is one entry, and an unsaved - hide costs only its own client a job and a cache entry. (A)'s O(rows) reads for `Min` - and `Max` are unlikely to fit the frame budget at ADR 008's fixture (not measured). A - mix is open: (A) for `Sum` and `Count` once question 3's (c) lands, and (B′) for the - rest. + *Recommendation (the author's, re-evaluated on 2026-10-05):* **(A)**, with (B′) kept + only as a memo for a hidden set so large that its reads would not fit the frame + budget. It is the stateless option, which C2 asks for, and a typical hidden set is + small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on + an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of + `ad60ad2b` found. None of these costs is measured. From dce3ab63c725290450d56223bfa2b886b1924195 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:05:11 -0700 Subject: [PATCH 041/106] docs(adr): ADR 009 folds ADR 008's 2026-10-06 answers: client-side view state, no layout tier, delivery by unaddressed notice (improvise-felo.4) Quotes Q33, Q39 and Q46-Q52 with ids. Request parameters come from a client's ephemeral UI state and saved views only supply defaults; SaveView touches no operator; the circuit relies on no view-fact tier. Q13 (now this ADR's, per "defer this to ADR 009") leans to an unaddressed change notice, with the fates of GridExtent, Dirty and the occupancy delta. Q23 concerns unsaved hides; Q24 asks which parameters must travel. Reads ADR 008 at 5b29335a. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 233 ++++++++++++++++------ 1 file changed, 170 insertions(+), 63 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1f904f47..980ff33c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,11 +1,11 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 29** -(PR #308, read at `0edd6f2a`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 32** +(PR #308, read at `5b29335a`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `0edd6f2a`, its revision 29) · +reads it at `5b29335a`, its revision 32) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 29 of `context/adr/008-virtualized-grid.md` on PR #308, at `0edd6f2a`. Clauses 3, +revision 32 of `context/adr/008-virtualized-grid.md` on PR #308, at `5b29335a`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 29, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 32, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -172,6 +172,29 @@ None of this answers this ADR's own questions 1 to 11. should shift the grid being viewed to keep the cursor in the same visual place on the screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). +**Answered on ADR 008 on 2026-10-06** (#308 inline comments; question numbers are ADR +008's): +- Q33 (4191973298, 05:50:48Z): "The backend should only hold model-relevant information. + If we need to persist cursor position/last view or other UI concerns, that would go in + localStorage/cookies/fragment routing/etc." +- Q39 (4191981468, 05:52:14Z): "I think layout version is mostly a view fact here, we have + to separate this from model mutations (value editing, etc.)" +- Q46 (4191986207, 05:53:05Z): "go with an explicit saye-view command" [sic]. +- Q47 (4191992428, 05:53:56Z): "The transient UI state never is reflected in a .improv + file unless an there is an explicit decision to persist it as a saved view (or by + replacing an existing view)" [sic]. +- Q48 (4191999772, 05:55:06Z): "A .improv file has to have at least a default view + definition. UI state is always ephemeral unless explicitly saved: when a new View is + created through the UI, it must not be lost if the model confirms that it was saved." +- Q49 (4192005787, 05:56:07Z): "The only one of these that should be "global" is set + number format as far as I can tell: the other ones are all ephemeral UI state." +- Q50 (4192010101, 05:56:53Z): "Disabling pruning should only be persisted if the + resulting view is explicitly saved." +- Q51 (4192014974, 05:57:40Z): "page is a third axis, do the others need to actually + travel across the wire?" +- Q52 (4192016491, 05:57:59Z): "defer this to ADR 009". This is ADR 008's question + about what replaces the pushed projection stream, and it is this ADR's question 13. + **Answers ADR 008 records that this ADR's text had not caught up with:** - Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. - Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases @@ -211,9 +234,16 @@ the clause it changes: circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. Clause 14, question 17. 6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's - questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 42 and - 43, quoted above. + questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 39, 42, + 43 and 46 to 52, quoted above. Clauses 5a, 7, 9, 12, 13 and 14 follow them. +7. **View configuration is the client's until saved** (Q46 to Q50). Hiding, collapsing, + page selection, axis assignment and pruning are ephemeral UI state per client until an + explicit `SaveView`. Only number format is "global" (Q49). So operator state is keyed + by the request, never by a client's unsaved settings held on the host. Clause 7, + questions 23 and 24. +8. **No basis tier this ADR relies on is a view fact** (Q39), and the host keeps no + per-client UI state (Q33). Clauses 7 and 12, question 13. ## Context @@ -437,7 +467,10 @@ in the batch's consolidated delta. So: `bump_changed_records` can then read the delta instead of diffing two `record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, -because it observes `view_shape` and not cells. +because it observes `view_shape` and not cells. Nothing in this ADR reads or bumps it. +The user calls it "mostly a view fact", to be separated from model mutations (ADR 008 +Q39). ADR 001's layout tier is to be split (ADR 008 question 68), and this ADR relies +only on `document_generation` and the record tier. **The dirty flag's content half does not move onto the cell delta alone.** `document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it @@ -841,12 +874,20 @@ states the same rule. groups, its page selection and `prune_empty`. Those are all read-time parameters except the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, `AddItem`, reorder and page step, which is what item grain and question 7 exist to - avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40), and it is - unchanged. + avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40). That tier + is a view fact by the user's Q39 answer, and the circuit does not depend on it. - A saved `.improv` view supplies the key and the parameters to a request that names it. - An unsaved request carries its own (ADR 008's flagged readings 1 and 3, its question - 51). + **Where the key and the parameters come from (revised for ADR 008's 2026-10-06 + answers).** Hiding, collapsing, the page selection, axis assignment and pruning are + ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number + format is "global" (Q49). So a request carries its own key and parameters, taken from + that client's UI state. A saved `.improv` view only supplies the **defaults** a client + starts from, and the circuit never stores a client's unsaved settings. Number format is + display-only (gotcha 8) and never reaches the circuit, except through clause 13's + widths if those were adopted. Whether every read-time parameter must travel in the + request (the user's Q51: "do the others need to actually travel across the wire?") is + question 24. A hide and a collapse change model-side aggregates (the totals, and a + collapsed group's fold), so the author reads them as needing to travel. Requests with the same key share one set of operators, whichever clients sent them. The live keys are a **cache**: a key enters when a request names it, and it is evicted once no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a @@ -888,7 +929,11 @@ states the same rule. `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, - and no read-time parameter can take Feb back out of it. So: + and no read-time parameter can take Feb back out of it. **Whose hide this is** (ADR 008, + Q46 to Q50): a hide is one client's ephemeral UI state until an explicit `SaveView`, + and never a model mutation. So the options below concern a request's **unsaved** hidden + set. A saved view's hidden set arrives only through `SaveView`, as part of the saved + definition a request starts from, and it is subtracted the same way. So: - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** item rebuilds the column totals of that shape, as a job. ADR 008 once said the same (its finding 10). It no longer does: its clause 5, at revision 28, says totals are @@ -962,6 +1007,13 @@ states the same rule. cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user ruled. Any measurement this ADR later reports against them is stated as a ratio against an unrelated operation on the same run, as ADR 008's question 13 requires. +- **`SaveView`, and the default view (Q46 to Q48).** `SaveView` is a model command that + writes a view definition, acknowledged by the model, and a document always has at + least a default view. **Neither touches the circuit.** A saved definition moves no + cell and no formula, so it emits no cell delta and no formula delta. Shape keys come + from requests, so saving builds no operator. The next request that names the saved + view simply starts from its defaults. The acknowledgement is ADR 008's + (`ServerFrame::Accepted`, its question 69), not this ADR's. - **A crossing count (ADR 008's second request).** Each step's occupancy output carries the number of tuples whose count crossed zero. ADR 008's ordered index uses it to choose between an insertion and a rebuild by sorting. @@ -979,7 +1031,7 @@ states the same rule. Re-reading the shape keys costs O(categories) per live key per model effect, and the formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by -`layout_version` after the loop, as today. +`layout_version` after the loop, as today, and the circuit reads neither. **The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first written* (superseded by item grain): `AddItem` moved the operator shape, so the view was @@ -1233,7 +1285,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 29 at `0edd6f2a`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 32 at `5b29335a`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1407,8 +1459,10 @@ Three things follow: until then (see "Names and ADR 007" in clause 3). The host attaches the absolute position from the current item order when it sends the element. A delta keyed by position alone would go stale with the next item insert. -- **An item edit is sent to every subscriber as an axis change.** "Item inserted at index - j of level L", or removed, or moved. It goes to **every** subscriber of a shape with +- **An item edit is sent to every consumer as an axis change.** "Item inserted at index + j of level L", or removed, or moved. Under question 13's (b) it rides on the change + notice, which every consumer receives. Under (c) it goes to **every** subscriber of a + shape with that category on an axis, wherever its window lies, because an insert before a window's top-left shifts every position that subscriber holds. A client re-derives its positions arithmetically, at O(levels) per position, and re-anchors its top-left to the @@ -1482,7 +1536,8 @@ window (ADR 008 question 66). A client alternates three steps: - it fills, asking for the next rows past its last absolute row position with a value in its displayed columns, and likewise for columns. -The model never re-runs a client's fixpoint. A subscribed client receives index (ii)'s +The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from +the change notice which records moved, and re-asks. Under (c), a subscribed client receives index (ii)'s deltas for its S, prunes again locally, and fills again only when a row or column has gone. Whether the loop always terminates is **not proven**. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A @@ -1503,7 +1558,7 @@ search appears (next paragraph). What forward-only growth simplifies: (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**. An item edit is an axis change and goes to every subscriber (above). A change at + only**. An item edit is an axis change and goes to every consumer (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill whose front is before q finds the change when it gets there. - **Resumable fills.** A fill's whole state is its request and its last absolute position @@ -1612,7 +1667,8 @@ occupied answers "empty axis". Both need the predecessor query. loss, which a fresh snapshot answers. What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. -- **T4. A snapshot, then deltas.** A subscription starts with the integrated result at +- **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the + snapshot and the notices are the deltas' stamps). A subscription starts with the integrated result at step n: the window as a Z-set of current values, each with weight +1. Deltas from step n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for @@ -1649,40 +1705,66 @@ consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that bo The alternative is a `serde` feature on `zset`. The author leans to the conversion, because what crosses is the projected result, not the crate's own element type. -**How stateless the host can be.** The user: "as stateless as possible" (C2). These are the -options of question 13, and ADR 008's question 52 asks the push half of the same thing. -- **(a) Pure fetch.** The host keeps nothing per client. A client calls - `project(request)`. It learns its result may be stale from a change notice addressed to - no one: the accepted batch's step. It then refetches when its own policy says to (Q42). - Each refetch costs O(visible). -- **(b) Soft-state subscriptions** (the author's leaning). A request may ask to be kept - current. For each subscription the host keeps the request and the last step it sent, - and nothing else: no copy of the window, no cursor and no viewport. After each accepted - batch it reads the delta of each subscribed shape, restricts it to the request's window, - and sends it under T3 to T5. **The table is a cache.** If it is lost (a restart, an - eviction, a reconnect), a client resumes by resending its request and the last step it - applied, and the host answers with a snapshot. The host could instead keep a short ring - of past deltas per shape; the author's lean is no ring at first. So the user's "recent - projections + clients" (C2) are this table plus clause 7's shape cache, and **both can be - rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). -- **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and +**How stateless the host can be, and what replaces the pushed stream.** The user: "as +stateless as possible" (C2), and "The backend should only hold model-relevant +information" (ADR 008 Q33). ADR 008's question 52, what replaces the pushed projection +stream, was answered "defer this to ADR 009", so it is this ADR's question 13. The +options, in ADR 008's lettering: +- **(a) Never learns: refetch by policy.** The host keeps nothing and says nothing. A + client refetches when its own policy says to (Q42). A change made by another client + stays invisible until then. +- **(b) An unaddressed change notice** (Flux style). After each accepted batch, the + model emits one notice to every consumer, addressed to none. It carries the step stamp + `(document_generation, step)` (T3), the identities of the records the batch moved (the + record tier, clause 3), each item edit as an axis change (above), and a flag when the + formula set moved (clause 7). Each client + decides for itself whether its window is affected. A moved record whose coordinates + fall under the client's axes and filters can change a cell or a total in its window, so + the client refetches by `project`, or by question 22's successor query. A formula change + means refetching. The host keeps no per-client state. A notice costs O(records moved), + and every client pays a test per notice. +- **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask + to be kept current. The host keeps the request and the last step it sent, then pushes + the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost + table costs a snapshot). But it is per-client state on the backend, and it holds a + window, which is a client's view parameter. +- **(d) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, - `:822` and `:981` at `50e1314d`). It keeps each client's `ViewState` and grid on the host - and diffs against them. ADR 008 lists it as obsolete under the split, and this ADR does - not propose it. + `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -A shape is kept while some subscription or recent request names it. Eviction drops its -operators, and the next request rebuilds them as a job (clause 7). How long "recent" lasts -is a cost the user has not set (question 14). +*The author's recommendation, revised for Q33:* **(b)**. It is the only option that keeps +a client informed while the host holds nothing per client. It is also the most network +transparent: one broadcast stream of model events, consumed the same way in process (the +TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would +otherwise refetch too much, if a measured round-trip cost justifies keeping per-client +state. *As first written* (superseded), this clause leaned to (c), which it then called +(b). That was before the user said the backend holds only model-relevant information. +Under (b) the user's "recent projections + clients" (C2) shrinks to clause 7's shape +cache, which is keyed by requests and holds no client. -**What a client must hold to resume:** its request, the last step it applied, its -integrated window, and its basis (ADR 001). The host holds nothing that a client needs -back. +**What becomes of today's pieces under (b)** (the author's): +- **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its + levels' item counts, which a client derives from the category lists it fetches. A + pruned axis has no global extent under client-side pruning (clause 12), and question + 22's successor query answers "is there more". +- **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which + widget a change can reach, moves into the client's test of the notice. +- **The occupancy delta** is not pushed. It stays a model-side output that keeps question + 22's index current. A client learns of a new or vanished row from the notice's records, + and re-asks. -**The basis.** ADR 001's basis is published with the result it describes, and after it -(gotcha 43). The host builds it from the request's window (ADR 008, lens row 18), not from -a mirror. `published_basis` (`project.rs:585` at `50e1314d`) already takes the `GridCache` -it describes, not a cursor. +A shape is kept while recent requests name it. Eviction drops its operators, and the next +request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has +not set (question 14). + +**What a client must hold to resume:** its request, the last step it applied, its window, +and its basis. The host holds nothing that a client needs back. + +**The basis.** ADR 001's basis is answered with the result it describes, and after it +(gotcha 43), built from the request's window, never from a mirror. `published_basis` +(`project.rs:585` at `50e1314d`) already takes the `GridCache` it describes, not a +cursor. Under the user's Q39 answer, only `document_generation` and the record tier are +model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). ### 13. Incremental max width (the user's question on ADR 008) @@ -1751,7 +1833,7 @@ orders come from the clauses and are not measured. | `ExactSum`, per summed group | 8 | 568 bytes per group | | occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | | the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | -| the subscription table | 12 | O(request) per subscription | +| the subscription table, only under question 13's (c) | 12 | O(request) per subscription | **Proposed by the author:** - **The store is the integral, so it is not copied.** The integrated base a rebuild reads @@ -1875,7 +1957,7 @@ longer a prerequisite of step 6. evaluator. - Live sources that can report changes plug in without touching the evaluator. - One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client UI state: only a subscription table that can be rebuilt + the host keeps no per-client state under question 13's (b). Option (c) would add only a subscription table that can be rebuilt from the requests. **Bad.** @@ -1911,8 +1993,8 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 23 were added by the 2026-10-05 -revision, and none is renumbered. +none of questions 1 to 11 on #309.** Questions 12 to 24 were added by the 2026-10-05 and 2026-10-06 +revisions, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not @@ -2021,12 +2103,20 @@ revision, and none is renumbered. the write path. Revisit it if a measured workload shows jitter that hysteresis does not cure. -13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per - client** (clause 12). **Owner: this ADR.** ADR 008's question 52 asks the push half, - and should point here. *Recommendation (the author's):* (b), soft-state - subscriptions that can be rebuilt from the request and the last step, with no copy of - a client's window on the host. When its table is lost, (b) falls back to (a). (c) is - obsolete under ADR 008's split. +13. *[New; now this ADR's to carry, by the user's "defer this to ADR 009" on ADR 008's + question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: + this ADR.** The options are: + - (a) never learn, and refetch by policy; + - (b) an unaddressed change notice per accepted batch, carrying the step stamp, the + moved records, axis changes and a formula flag, after which each client decides; + - (c) push the model's data changes through soft-state subscriptions; + - (d) today's per-mirror push, which is obsolete. + + *Recommendation (the author's, revised for Q33):* (b). The host keeps nothing per + client, and in-process and remote consumers read one stream. (c) is an optimisation + to adopt only if measured round trips justify per-client state. Under (b), + `GridExtent` is derived by the client, `Dirty` is retired with the mirrors, and the + occupancy delta stays model-side, feeding question 22's index. 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2153,7 +2243,10 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). The options: + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). Under ADR 008's 2026-10-06 + answers (Q46 to Q50), a hide is a client's UI parameter until an explicit `SaveView`. + So this question concerns the hidden set a request carries, and it assumes that set + travels (question 24). The options: - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, because the shape key is shared across clients and an unsaved hide belongs to one client's request. @@ -2174,3 +2267,17 @@ revision, and none is renumbered. small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of `ad60ad2b` found. None of these costs is measured. + +24. *[New.]* **Which read-time parameters must travel in a request** (clause 7; the user + on ADR 008's Q51: "page is a third axis, do the others need to actually travel across + the wire?"). **Owner: ADR 008** (its question 51). This question states only what the + circuit needs. The page selection travels as the third axis. A **hide** changes + model-side aggregates (the column totals, and a collapsed group's fold), and so does a + **collapse** (a fold row's value). The user: "Aggregation should always be done + model-side for consistency" (Q38). So if hidden items and collapsed groups did not + travel, a client could not get its totals and folds from the model. It would have to + compute them itself, against Q38, or show totals over all rows. The **drawn order**, + **number format** and **prune** (under client-side pruning) change no aggregate, and + need not travel. *Recommendation (the author's):* hidden items and collapsed groups + travel with the request; the drawn order, number format and prune stay with the + client. Not decided. From 5a8dc486ca14e1aaf112e81f27411c0d2b5ae907 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:11:03 -0700 Subject: [PATCH 042/106] docs(adr): ADR 009's change notice carries moved output keys, not records alone (improvise-felo.4) A record test misses formulas whose inputs sit at another stem; the notice now carries the moved keys per live shape (the author's lean), with a conservative client rule as the alternative, and a formula or measure_agg flag. Also: the occupancy delta is not pushed under (b), T4 and T5 are scoped, measure_agg is model state, Q24 names ADR 008's alternative forms, and records-mode extent is fetched. Reads ADR 008 at 9df2e478. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 105 +++++++++++++++------- 1 file changed, 75 insertions(+), 30 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 980ff33c..ac524b74 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 32** -(PR #308, read at `5b29335a`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 33** +(PR #308, read at `9df2e478`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `5b29335a`, its revision 32) · +reads it at `9df2e478`, its revision 33) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 32 of `context/adr/008-virtualized-grid.md` on PR #308, at `5b29335a`. Clauses 3, +revision 33 of `context/adr/008-virtualized-grid.md` on PR #308, at `9df2e478`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 32, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 33, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -881,7 +881,10 @@ states the same rule. answers).** Hiding, collapsing, the page selection, axis assignment and pruning are ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number format is "global" (Q49). So a request carries its own key and parameters, taken from - that client's UI state. A saved `.improv` view only supplies the **defaults** a client + that client's UI state. **`measure_agg` is the exception:** it is model state + (`Model::measure_agg`, `types.rs:370` at `50e1314d`), part of every shape key but read + from the model, never taken from a request (ADR 008 Q51 says the same). A saved + `.improv` view only supplies the **defaults** a client starts from, and the circuit never stores a client's unsaved settings. Number format is display-only (gotcha 8) and never reaches the circuit, except through clause 13's widths if those were adopted. Whether every read-time parameter must travel in the @@ -1285,7 +1288,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 32 at `5b29335a`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 33 at `9df2e478`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1321,10 +1324,12 @@ host-held mirror of a client. occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first written*, its consumer was the per-view frame observation and its reset of each client's - cursor (gotcha 24). Under the split the host holds no cursor. So the delta is a model - fact, published on clause 12's delivery stream to the consumers of that shape, and a - client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the - host pushes it is question 13, which is ADR 008's question 52 seen from this side. + cursor (gotcha 24). Under the split the host holds no cursor, and no client's read-time + parameters, so it cannot compute any client's **visible** axis. The delta is a + model-side output at item grain that keeps question 22's index current. Under question + 13's (b) it is **not pushed**. A client learns of a new or vanished row from the change + notice, recomputes its own visible axis, and shifts its grid so that the cursor keeps + its place (Q34). Only under (c) would a window's delta be pushed. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at `50e1314d`), @@ -1537,7 +1542,7 @@ window (ADR 008 question 66). A client alternates three steps: in its displayed columns, and likewise for columns. The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from -the change notice which records moved, and re-asks. Under (c), a subscribed client receives index (ii)'s +the change notice which output keys moved, and re-asks. Under (c), a subscribed client receives index (ii)'s deltas for its S, prunes again locally, and fills again only when a row or column has gone. Whether the loop always terminates is **not proven**. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A @@ -1558,7 +1563,9 @@ search appears (next paragraph). What forward-only growth simplifies: (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**. An item edit is an axis change and goes to every consumer (above). A change at + only**, and to their moved keys in a question-13 (b) notice. A formula cell inside the + range moves when any of its inputs move, wherever those sit, so it is tested by the + moved output keys, never by the position of the record that moved. An item edit is an axis change and goes to every consumer (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill whose front is before q finds the change when it gets there. - **Resumable fills.** A fill's whole state is its request and its last absolute position @@ -1668,12 +1675,15 @@ occupied answers "empty axis". Both need the predecessor query. delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the - snapshot and the notices are the deltas' stamps). A subscription starts with the integrated result at + snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at step n: the window as a Z-set of current values, each with weight +1. Deltas from step n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for today's bootstrap compaction (gotcha 32). -- **T5. Order holds across streams, and coalescing happens within a stream.** A slow +- **T5. Order holds across streams, and coalescing happens within a stream** (as stated, + for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client + merges pending notices into one, taking the union of their moved keys and the latest + stamp, and refetches once). A slow consumer's pending deltas on one stream can be **summed** into one delta. Z-set addition is commutative and associative, so the sum is exact, and the queue is bounded by the size of the result, not by the number of steps. When the summed delta outgrows @@ -1714,15 +1724,37 @@ options, in ADR 008's lettering: client refetches when its own policy says to (Q42). A change made by another client stays invisible until then. - **(b) An unaddressed change notice** (Flux style). After each accepted batch, the - model emits one notice to every consumer, addressed to none. It carries the step stamp - `(document_generation, step)` (T3), the identities of the records the batch moved (the - record tier, clause 3), each item edit as an axis change (above), and a flag when the - formula set moved (clause 7). Each client - decides for itself whether its window is affected. A moved record whose coordinates - fall under the client's axes and filters can change a cell or a total in its window, so - the client refetches by `project`, or by question 22's successor query. A formula change - means refetching. The host keeps no per-client state. A notice costs O(records moved), - and every client pays a test per notice. + model emits one notice to every consumer, addressed to none. It carries: + - the step stamp `(document_generation, step)` (T3); + - each item edit, as an axis change (above); + - a flag when the formula set or `measure_agg` moved. Either may move every aggregate + without moving a record. `measure_agg` has no wire command today, and changes only + with a load, which moves `document_generation`; + - **what moved, as the model computes it** (the author's lean, below). + + **Records alone are not enough** (found by the review of `dce3ab63`). A client could + test whether each moved record falls under its own axes and filters, but that misses + every formula whose inputs sit at another stem: an inner `WHERE` naming another item, + or a ref to an item of a page, filter or None category (clause 5's one join). For + example, a client on page `Region=West` shows `EastShare = Revenue / SUM(Revenue WHERE + Region=East)`. Another client writes an East record. The West client's test says "not + mine", and its cells stay stale until its own refetch policy fires. Two remedies: + - **The author's lean: the notice carries the moved keys the circuit already + computes.** Clause 5 finds the affected `(stem, formula)` pairs, and clause 4 the + moved group keys. That is the circuit's whole job. For each **live shape key**, which + is a model-side cache keyed by requests and never by a client, the notice lists the + moved output keys: cells, formula cells and column-tuple totals. A client intersects + them with its window by identity, so the model decides relevance and the client only + looks it up. A notice then costs O(moved output keys across live shapes), not + O(records moved). + - **The alternative: a conservative client rule.** The notice carries records only, + and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to + a page, filter or None item, or an aggregate) treats **every** moved record as + relevant. The notice is small, but such windows refetch on every batch. + + Under either remedy, the client refetches what it needs by `project`, or by question + 22's successor query. A formula or `measure_agg` change means refetching the window. + The host keeps no per-client state. - **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask to be kept current. The host keeps the request and the last step it sent, then pushes the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost @@ -1732,8 +1764,12 @@ options, in ADR 008's lettering: `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -*The author's recommendation, revised for Q33:* **(b)**. It is the only option that keeps -a client informed while the host holds nothing per client. It is also the most network +*The author's recommendation, revised for Q33:* **(b)**, with moved keys per live shape. +It is the only option that keeps a client informed while the host holds nothing per +client. **The bigger notice does not change the choice against (c).** (b)'s notice lists +keys per live shape, one broadcast for all clients. (c) pushes values per window, and so +needs per-client state. The key lists cost O(moved keys) per batch, where (c)'s deltas +cost the same per subscriber. It is also the most network transparent: one broadcast stream of model events, consumed the same way in process (the TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would otherwise refetch too much, if a measured round-trip cost justifies keeping per-client @@ -1746,7 +1782,9 @@ cache, which is keyed by requests and holds no client. - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A pruned axis has no global extent under client-side pruning (clause 12), and question - 22's successor query answers "is there more". + 22's successor query answers "is there more". **In records mode** the row extent is the + record count that the page and filter admit. That is a model fact, which the client must + **fetch**, because it cannot derive it. - **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which widget a change can reach, moves into the client's test of the notice. - **The occupancy delta** is not pushed. It stays a model-side output that keeps question @@ -2107,8 +2145,10 @@ revisions, and none is renumbered. question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: this ADR.** The options are: - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the step stamp, the - moved records, axis changes and a formula flag, after which each client decides; + - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis + changes, a formula or `measure_agg` flag, and the moved output keys per live shape + key (the author's lean). The alternative is moved records plus a conservative rule + for windows with cross-stem formulas; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2276,7 +2316,12 @@ revisions, and none is renumbered. **collapse** (a fold row's value). The user: "Aggregation should always be done model-side for consistency" (Q38). So if hidden items and collapsed groups did not travel, a client could not get its totals and folds from the model. It would have to - compute them itself, against Q38, or show totals over all rows. The **drawn order**, + compute them itself, against Q38, or show totals over all rows. **Alternative forms**, + which ADR 008's Q51 names and which are closer to the user's lean that only the axes + travel: a **collapse** as an axis at the group's level (the request's row axis names + the group level instead of the items), and a **hide** as a filter-key exclusion. Both + are already in the shape key's vocabulary (clause 7), so either form gives the model + what it needs without a separate parameter. The **drawn order**, **number format** and **prune** (under client-side pruning) change no aggregate, and need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the From e152b667fc3043bac964fb2e44946502c406a707 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:15:38 -0700 Subject: [PATCH 043/106] docs(adr): ADR 009 states delivery as host state against wire traffic and leans to one topic per shape (improvise-felo.4) Corrects the (b)/(c) comparison, adds (b'): notices by shape-key topic, O(1) delivery state per client. Also: fold rows match through the client's member-to-group remap, axis changes do not coalesce, display relabels ride the notice under ADR 007, and Q24's alternative forms change the shape key. Reads ADR 008 at 385784e7. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 112 +++++++++++++++------- 1 file changed, 78 insertions(+), 34 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ac524b74..00167007 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 33** -(PR #308, read at `9df2e478`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 34** +(PR #308, read at `385784e7`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `9df2e478`, its revision 33) · +reads it at `385784e7`, its revision 34) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 33 of `context/adr/008-virtualized-grid.md` on PR #308, at `9df2e478`. Clauses 3, +revision 34 of `context/adr/008-virtualized-grid.md` on PR #308, at `385784e7`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 33, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 34, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1288,7 +1288,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 33 at `9df2e478`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 34 at `385784e7`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1682,8 +1682,11 @@ occupied answers "empty axis". Both need the predecessor query. today's bootstrap compaction (gotcha 32). - **T5. Order holds across streams, and coalescing happens within a stream** (as stated, for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client - merges pending notices into one, taking the union of their moved keys and the latest - stamp, and refetches once). A slow + merges pending notices into one, taking the union of their moved keys and flags and the + latest stamp, and refetches once. **Axis changes do not merge.** They are positional, + so they are kept as a list and applied in their original order. A + `document_generation` change inside the merged run voids the merge, because the old + keys belong to a document that is gone, and the client refetches from scratch). A slow consumer's pending deltas on one stream can be **summed** into one delta. Z-set addition is commutative and associative, so the sum is exact, and the queue is bounded by the size of the result, not by the number of steps. When the summed delta outgrows @@ -1730,7 +1733,12 @@ options, in ADR 008's lettering: - a flag when the formula set or `measure_agg` moved. Either may move every aggregate without moving a record. `measure_agg` has no wire command today, and changes only with a load, which moves `document_generation`; - - **what moved, as the model computes it** (the author's lean, below). + - **what moved, as the model computes it** (the author's lean, below); + - **display relabels.** Under ADR 007 a rename is one `display` write, which moves no + key (clause 3), so no moved-key list can carry it. The notice carries it as `(id, + new display)`, and the client relabels what it holds without refetching. Before ADR + 007, a rename is a removal and an insertion of every cell under the old name, and it + arrives as moved keys. **Records alone are not enough** (found by the review of `dce3ab63`). A client could test whether each moved record falls under its own axes and filters, but that misses @@ -1745,7 +1753,10 @@ options, in ADR 008's lettering: is a model-side cache keyed by requests and never by a client, the notice lists the moved output keys: cells, formula cells and column-tuple totals. A client intersects them with its window by identity, so the model decides relevance and the client only - looks it up. A notice then costs O(moved output keys across live shapes), not + looks it up. **A collapsed fold row has no maintained output key under item grain** + (clause 7: it is combined at read time). So a moved member key goes through the + client's own member-to-group remap before it is matched against the window. The + client holds that remap, since collapse is its UI state. A notice then costs O(moved output keys across live shapes), not O(records moved). - **The alternative: a conservative client rule.** The notice carries records only, and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to @@ -1764,21 +1775,46 @@ options, in ADR 008's lettering: `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -*The author's recommendation, revised for Q33:* **(b)**, with moved keys per live shape. -It is the only option that keeps a client informed while the host holds nothing per -client. **The bigger notice does not change the choice against (c).** (b)'s notice lists -keys per live shape, one broadcast for all clients. (c) pushes values per window, and so -needs per-client state. The key lists cost O(moved keys) per batch, where (c)'s deltas -cost the same per subscriber. It is also the most network -transparent: one broadcast stream of model events, consumed the same way in process (the -TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would -otherwise refetch too much, if a measured round-trip cost justifies keeping per-client -state. *As first written* (superseded), this clause leaned to (c), which it then called -(b). That was before the user said the backend holds only model-relevant information. -Under (b) the user's "recent projections + clients" (C2) shrinks to clause 7's shape -cache, which is keyed by requests and holds no client. +**The trade between (b) and (c) is host state against wire traffic** (the author's; +corrected after the review of `5a8dc486`; nothing here is measured). +- **(b) keeps no per-client state, and pays in traffic.** Its one broadcast is still + delivered to all N clients, and each receives every live shape's moved keys. That is + O(N × all moved keys) on the wire per batch. In this ADR's primary case, a live source + streaming into a large table, every remote client would receive every moved key, most + of them for shapes it does not read. +- **(c) keeps per-client state, and saves traffic.** It holds the request and the last + step per subscriber, which is a client's window and so a view parameter on the host, + against Q33. In return it sends each client only its window's deltas. +- *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) + per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. -**What becomes of today's pieces under (b)** (the author's): +**A middle form, (b′): one topic per shape key.** A client subscribes to the topic of +its own shape key, and receives notices only for that shape. The host keeps O(1) per +client: a topic subscription, which names a shape key and holds no window, cursor or +view. It is the same kind of delivery state a socket connection already is. The costs: +- **Traffic.** O(Σ over clients of the moved keys of that client's shape) per batch. It + saves nothing when every client reads one shape, and much when shapes differ. It is + still not restricted to the window; only (c) does that. +- **"As stateless as possible" (C2) and Q33.** It is per-client state, but it holds no UI + state, and it can be rebuilt by the next request, which names its shape. The author + reads it as delivery state, not UI state, but the user has not said so. +- **Network transparency.** The topic is a filter applied when sending, so in-process + and remote consumers still read the same notices (T6). The TUI simply subscribes to its + own shape. + +*The author's recommendation, revised:* **(b′)**. It keeps the host free of UI state, +while cutting (b)'s traffic to the shapes each client reads. A client that names no +topic receives the full broadcast, which is (b). (c) stays an optimisation for a client +whose window is a small part of a busy shape, if measured traffic justifies holding a +window on the host. All of this is unmeasured. + +*Earlier still* (superseded), this clause leaned to (c), which it then called (b). That +was before the user said the backend holds only model-relevant information. Under (b′), +the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is +keyed by requests and holds no client, plus the topic subscriptions, which hold one shape +key per client. + +**What becomes of today's pieces under (b) and (b′)** (the author's): - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A pruned axis has no global extent under client-side pruning (clause 12), and question @@ -1995,7 +2031,7 @@ longer a prerequisite of step 6. evaluator. - Live sources that can report changes plug in without touching the evaluator. - One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client state under question 13's (b). Option (c) would add only a subscription table that can be rebuilt + the host keeps no per-client UI state under question 13's (b′), only one topic per client (none under plain (b)). Option (c) would add a subscription table that can be rebuilt from the requests. **Bad.** @@ -2146,17 +2182,21 @@ revisions, and none is renumbered. this ADR.** The options are: - (a) never learn, and refetch by policy; - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis - changes, a formula or `measure_agg` flag, and the moved output keys per live shape - key (the author's lean). The alternative is moved records plus a conservative rule - for windows with cross-stem formulas; + changes, display relabels, a formula or `measure_agg` flag, and the moved output + keys per live shape key. The alternative is moved records plus a conservative rule + for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); + - (b′) the same notices, delivered by topic: one topic per shape key, with O(1) + per-client state that holds no UI state; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. - *Recommendation (the author's, revised for Q33):* (b). The host keeps nothing per - client, and in-process and remote consumers read one stream. (c) is an optimisation - to adopt only if measured round trips justify per-client state. Under (b), - `GridExtent` is derived by the client, `Dirty` is retired with the mirrors, and the - occupancy delta stays model-side, feeding question 22's index. + *Recommendation (the author's, revised):* (b′). It holds no UI state on the host and + cuts (b)'s traffic to the shapes each client reads. (b) is what a client that names no + topic gets. (c) is an optimisation if measured traffic justifies holding a window. + Under all three, `GridExtent` is derived by the client (in records mode it is + fetched), `Dirty` is retired with the mirrors, and the occupancy delta stays + model-side, feeding question 22's index. *As first written*, this recommended plain + (b); the trade it skipped is wire traffic. 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2321,7 +2361,11 @@ revisions, and none is renumbered. travel: a **collapse** as an axis at the group's level (the request's row axis names the group level instead of the items), and a **hide** as a filter-key exclusion. Both are already in the shape key's vocabulary (clause 7), so either form gives the model - what it needs without a separate parameter. The **drawn order**, + what it needs without a separate parameter. **But both change the shape key.** A hide + as a filter-key exclusion makes each distinct hidden set its own key, at an O(cells) + rebuild per set, and it loses question 23's (A), where every hidden set shares one + key and is subtracted at read time. A collapse as an axis level does the same for + each collapse state. The **drawn order**, **number format** and **prune** (under client-side pruning) change no aggregate, and need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the From 8ee0d78c64d7c676c703ebf8f133d174d0a256f5 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:19:00 -0700 Subject: [PATCH 044/106] docs(adr): ADR 009 sends every batch header to every topic and pins subscribed shapes (improvise-felo.4) Answers the Opus review's B8 and ChatGPT Codex's cache-lifecycle P on e152b667: headers (stamp, generation, axis changes, relabels, flags, eviction list) reach every client so T3's gap rule holds; only moved-key lists are filtered by shape; a topic subscription is a lease that pins its shape and evictions are announced; the general refetch rule covers gaps, restarts and evictions; the shape-key switch rule is stated; Q13 puts the Q33 question about topic state to the user. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 81 ++++++++++++++++++----- 1 file changed, 64 insertions(+), 17 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 00167007..e02328c1 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -893,7 +893,8 @@ states the same rule. collapsed group's fold), so the author reads them as needing to travel. Requests with the same key share one set of operators, whichever clients sent them. The live keys are a **cache**: a key enters when a request names it, and it is evicted once - no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a + no recent request has named it **and no live topic subscription pins it**. Every + eviction is announced in the next notice header (clause 12). Elsewhere in this ADR, "a shape" means a shape key. *As first written* (superseded twice): the operator shape was `view_shape` plus group @@ -1671,7 +1672,13 @@ occupied answers "empty axis". Both need the predecessor query. `(document_generation, step)`. The circuit steps once per model effect (clause 3), and ADR 001 already puts batches in one total order. A consumer that has applied step n drops anything at or below n, so a duplicate does no harm, and it treats a gap as a - loss, which a fresh snapshot answers. What is published is the **batch's** consolidated + loss, which a fresh snapshot answers. Under question 13's (b′), every client receives + every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is + always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees + `document_generation` change, finds its shape in an eviction list, or receives a + notice it cannot interpret, **refetches**. That covers a host restart: the stamp + sequence starts again, which is a discontinuity, and a client holding a cached window + across it refetches. The same rule holds for a client under (a), (b) and (c). What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the @@ -1789,15 +1796,45 @@ corrected after the review of `5a8dc486`; nothing here is measured). per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. **A middle form, (b′): one topic per shape key.** A client subscribes to the topic of -its own shape key, and receives notices only for that shape. The host keeps O(1) per -client: a topic subscription, which names a shape key and holds no window, cursor or -view. It is the same kind of delivery state a socket connection already is. The costs: -- **Traffic.** O(Σ over clients of the moved keys of that client's shape) per batch. It - saves nothing when every client reads one shape, and much when shapes differ. It is - still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** It is per-client state, but it holds no UI - state, and it can be rebuilt by the next request, which names its shape. The author - reads it as delivery state, not UI state, but the user has not said so. +its own shape key. The host keeps O(1) per client: a topic subscription, which names a +shape key and holds no window, cursor or view. **A topic filters only the moved-key +lists; every batch's header goes to everyone** (found by the review of `e152b667`): +- **The header goes to every topic, and to every topic-less client, for every accepted + batch.** It holds the step stamp, `document_generation`, the axis changes, the display + relabels, the formula and `measure_agg` flags, and the eviction list (below). None of + these is per shape: an item insert shifts every shape with that category on an axis. + So a filter must not withhold them. Headers are also what keep T3's loss detection + working. Every client sees every stamp, so a gap is a real loss and never a quiet + batch. That costs O(N) small headers per batch. +- **The moved-key lists are filtered by shape.** A client receives only its own shape's + list. +- **Switching shape key.** A client that changes its request's shape fetches under the + new key, which returns the stamp n the fetch was computed at. It subscribes to the new + topic from n on and drops the old topic. Headers never stopped arriving, so the switch + leaves no gap. +- **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a + shape once no recent request names it. A client still subscribed to that topic would + then receive no moved keys, which looks exactly like "nothing changed", and its window + would go stale. *The author's lean* combines two invariants: + - **(i) a live topic subscription pins its shape.** Subscriptions are leases, renewed + by the client's requests or a heartbeat. Host state grows with the live + subscriptions, and a client that vanishes is released when its lease expires. + - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under + memory pressure (clause 14), puts the shape key in the next header's **eviction + list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + + The alternative, **(iii)**: every header states, for each shape it carries keys for, + that the shape was evaluated, and a subscribed shape missing from that statement means + a conservative refetch. It needs no leases, and it costs a list of evaluated shapes + per header. +- **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) + per batch. It saves nothing when every client reads one shape, and much when shapes + differ. It is still not restricted to the window; only (c) does that. +- **"As stateless as possible" (C2) and Q33.** A subscription is per-client state, with + a lease. It holds no UI state, and the next request, which names its shape, rebuilds it. + The author reads it as delivery state, not UI state. **But the user said "The backend + should only hold model-relevant information"** (ADR 008 Q33), and whether a topic + subscription counts as that is the user's question, put in question 13. - **Network transparency.** The topic is a filter applied when sending, so in-process and remote consumers still read the same notices (T6). The TUI simply subscribes to its own shape. @@ -1827,8 +1864,9 @@ key per client. 22's index current. A client learns of a new or vanished row from the notice's records, and re-asks. -A shape is kept while recent requests name it. Eviction drops its operators, and the next -request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has +A shape is kept while recent requests name it or a live topic subscription pins it. +Eviction drops its operators, is announced in the next header's eviction list, and the +next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has not set (question 14). **What a client must hold to resume:** its request, the last step it applied, its window, @@ -1921,7 +1959,7 @@ orders come from the clauses and are not measured. - **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a range, and apply a delta. The first implementation keeps it in memory, and the store-memory ADR can page or compress it without changing any operator. -- **Shapes are evicted** (clause 12). So per-shape state is bounded by the shapes in +- **Shapes are evicted** unless a live topic subscription pins them (clause 12). So per-shape state is bounded by the shapes in recent use, not by every saved view. - **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and @@ -2185,8 +2223,14 @@ revisions, and none is renumbered. changes, display relabels, a formula or `measure_agg` flag, and the moved output keys per live shape key. The alternative is moved records plus a conservative rule for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - - (b′) the same notices, delivered by topic: one topic per shape key, with O(1) - per-client state that holds no UI state; + - (b′) the same notices, delivered by topic. Every batch's header (stamp, + `document_generation`, axis changes, relabels, flags, eviction list) goes to every + client, and only the moved-key lists are filtered by shape key. A topic subscription + is a lease that pins its shape, and any eviction is announced in the header. It costs + O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question + for the user:** is a topic subscription "model-relevant information", in the sense + of your answer to ADR 008's Q33 ("The backend should only hold model-relevant + information")? (b′) depends on the answer; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2204,7 +2248,10 @@ revisions, and none is renumbered. applied at read time (the author's)? Or is it kept per saved `.improv` view only, so that a request with an unsaved key is served without incremental state? *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while. The time is an implementation number. An unsaved collapse, reorder, page + a while **and no live topic subscription pins it** (question 13's (b′)). An eviction + is announced in the next notice header, so a subscriber refetches and is never left + with a window that looks current and is not. The time and the lease length are + implementation numbers. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved hide or show of a row-axis item shares the key too. Under question 23's default (A) its totals subtract the hidden rows at read time and add nothing to the cache. Under From 70ababfe0c6c0f5616ba5103df586e70f9224cd4 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:19:46 -0700 Subject: [PATCH 045/106] docs(adr): ADR 009 adds a host instance to the stamp and covers plain (b) by the eviction header (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 ++++++++++++++++------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e02328c1..286ddabc 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1669,16 +1669,20 @@ occupied answers "empty axis". Both need the predecessor query. (every NaN is one value, and every other number is its bits) before they leave the host. The record tier keeps bit identity, as it does today. Question 15. - **T3. Every output change is stamped.** A delta carries the step that computed it, as - `(document_generation, step)`. The circuit steps once per model effect (clause 3), and - ADR 001 already puts batches in one total order. A consumer that has applied step n - drops anything at or below n, so a duplicate does no harm, and it treats a gap as a + `(host instance, document_generation, step)`. The **host instance** is an id chosen + when the host starts. The other two counters live in memory and start again at their + initial values after a restart, so without the instance id a reconnected client would + drop the new host's notices as duplicates. The circuit steps once per model effect + (clause 3), and ADR 001 already puts batches in one total order. A consumer that has + applied step n of the same instance and generation drops anything at or below n, so a + duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot answers. Under question 13's (b′), every client receives every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. That covers a host restart: the stamp - sequence starts again, which is a discontinuity, and a client holding a cached window - across it refetches. The same rule holds for a client under (a), (b) and (c). What is published is the **batch's** consolidated + notice it cannot interpret, **refetches**. A changed host instance is such a + discontinuity, so a reconnect after a restart always refetches. The same rule holds for + a client under (a), (b) and (c). What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the @@ -1800,8 +1804,9 @@ its own shape key. The host keeps O(1) per client: a topic subscription, which n shape key and holds no window, cursor or view. **A topic filters only the moved-key lists; every batch's header goes to everyone** (found by the review of `e152b667`): - **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the step stamp, `document_generation`, the axis changes, the display - relabels, the formula and `measure_agg` flags, and the eviction list (below). None of + batch.** It holds the step stamp (with the host instance, T3), `document_generation`, + the axis changes, the display relabels, the formula and `measure_agg` flags, and the + eviction list (below). None of these is per shape: an item insert shifts every shape with that category on an axis. So a filter must not withhold them. Headers are also what keep T3's loss detection working. Every client sees every stamp, so a gap is a real loss and never a quiet @@ -1823,6 +1828,15 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists + moved keys only for live shape keys, and a topic-less client pins nothing. So its shape + can be evicted silently. The header's eviction list reaches it too, and it refetches. + Only the "records plus a conservative rule" alternative of (b) escapes the hole, because + it lists no shape at all. Instead of an eviction list, a header could carry the set of + live shape keys, or a shape-cache generation that moves on every eviction. A client + whose key is missing from the live set would then re-request. Both say the same thing, + and the eviction list is smaller. + The alternative, **(iii)**: every header states, for each shape it carries keys for, that the shape was evaluated, and a subscribed shape missing from that statement means a conservative refetch. It needs no leases, and it costs a list of evaluated shapes @@ -2225,7 +2239,8 @@ revisions, and none is renumbered. for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - (b′) the same notices, delivered by topic. Every batch's header (stamp, `document_generation`, axis changes, relabels, flags, eviction list) goes to every - client, and only the moved-key lists are filtered by shape key. A topic subscription + client, with the host instance in its stamp. Only the moved-key lists are filtered + by shape key. A topic subscription is a lease that pins its shape, and any eviction is announced in the header. It costs O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense From 85dc0ad8ba8c789182995ec3bef0ef2e6f319e8c Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:23:30 -0700 Subject: [PATCH 046/106] docs(adr): ADR 009 registers subscriptions atomically and stamps batches by a publication number (improvise-felo.4) B9: a fetch registers its topic in the same step, so no batch can fall between them; ack-and-recheck is the alternative; refetch-storm options and the lease heartbeat's cost are stated. Codex on 70ababfe: the stamp counts published batches, not per-effect circuit steps, so a multi-effect batch is one header; required tests added to step 8. Reads ADR 008 at 1c974bf7. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 121 +++++++++++++++------- 1 file changed, 81 insertions(+), 40 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 286ddabc..29ffee43 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 34** -(PR #308, read at `385784e7`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 35** +(PR #308, read at `1c974bf7`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `385784e7`, its revision 34) · +reads it at `1c974bf7`, its revision 35) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 34 of `context/adr/008-virtualized-grid.md` on PR #308, at `385784e7`. Clauses 3, +revision 35 of `context/adr/008-virtualized-grid.md` on PR #308, at `1c974bf7`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 34, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 35, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -432,8 +432,9 @@ An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to th Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation -under the view/host split. The per-effect step stays, because each step is what clause -12 stamps. +under the view/host split. The per-effect step stays as the circuit's own clock. What a +client counts is clause 12's publication number, which moves once per published batch +(T3). **A delta key's identity is bit identity, the same identity the record tier uses today.** Two elements have the same `(key, value)` part exactly when their keys are @@ -1289,7 +1290,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 34 at `385784e7`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 35 at `1c974bf7`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1668,33 +1669,48 @@ occupied answers "empty axis". Both need the predecessor query. round trip. **Proposed:** output streams are consolidated under the wire's identity (every NaN is one value, and every other number is its bits) before they leave the host. The record tier keeps bit identity, as it does today. Question 15. -- **T3. Every output change is stamped.** A delta carries the step that computed it, as - `(host instance, document_generation, step)`. The **host instance** is an id chosen - when the host starts. The other two counters live in memory and start again at their - initial values after a restart, so without the instance id a reconnected client would - drop the new host's notices as duplicates. The circuit steps once per model effect - (clause 3), and ADR 001 already puts batches in one total order. A consumer that has - applied step n of the same instance and generation drops anything at or below n, so a - duplicate does no harm, and it treats a gap as a - loss, which a fresh snapshot answers. Under question 13's (b′), every client receives - every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is - always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees - `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. A changed host instance is such a - discontinuity, so a reconnect after a restart always refetches. The same rule holds for - a client under (a), (b) and (c). What is published is the **batch's** consolidated - delta, and only once the batch is accepted. A refused frame is restored (clause 11) and - publishes nothing. +- **T3. Every published batch is stamped, by a publication sequence.** The stamp is + `(host instance, document_generation, publication)`. + - **The publication number** increments once per **published batch**, and that is + what a client counts. The circuit's internal `step` advances once per model effect + (clause 3), and it is never a client-visible gap detector. *As first written* + (superseded), the stamp carried `step`. A batch of two model effects then advanced + from n to n+2 under one header, and a client read the missing n+1 as a loss (found by + ChatGPT Codex on `70ababfe`). *The author's lean* is this separate sequence (i). The + alternatives: (ii) a header per model-effect step, which multiplies headers and + exposes a batch's internal steps, which ADR 001 treats as one atomic unit; or (iii) a + header that carries the previous published stamp, so a client can tell a + consolidated range from a loss, at a little more per header. + - **The host instance** is an id chosen when the host starts. The other two counters + live in memory and start again at their initial values after a restart. Without the + instance id, a reconnected client would drop the new host's notices as duplicates. + - ADR 001 already puts batches in one total order. A consumer that has applied + publication n, of the same instance and generation, drops anything at or below n, so + a duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot + answers. + - Under question 13's (b′), every client receives every batch's **header** (clause 12), + so the publication numbers it sees are contiguous, and a gap is always a real loss. + Contiguous numbers prove only that no header was lost, not that a key list was routed + to the client. That holds only because a subscription is registered atomically with + its fetch (clause 12). + - **The general rule:** a client that misses a notice, sees a gap, sees + `document_generation` change, finds its shape in an eviction list, or receives a + notice it cannot interpret, **refetches**. A changed host instance is such a + discontinuity, so a reconnect after a restart always refetches. The same rule holds + for a client under (a), (b) and (c). + - What is published is the **batch's** consolidated delta, and only once the batch is + accepted. A refused frame is restored (clause 11), publishes nothing, and consumes no + publication number. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at - step n: the window as a Z-set of current values, each with weight +1. Deltas from step - n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current + publication n: the window as a Z-set of current values, each with weight +1. Deltas + from publication n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for today's bootstrap compaction (gotcha 32). - **T5. Order holds across streams, and coalescing happens within a stream** (as stated, for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client merges pending notices into one, taking the union of their moved keys and flags and the - latest stamp, and refetches once. **Axis changes do not merge.** They are positional, + latest publication number, and refetches once. **Axis changes do not merge.** They are positional, so they are kept as a list and applied in their original order. A `document_generation` change inside the merged run voids the merge, because the old keys belong to a document that is gone, and the client refetches from scratch). A slow @@ -1708,9 +1724,10 @@ occupied answers "empty axis". Both need the predecessor query. is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple identity, or by a stable cross-product position, is not positional in that sense. **Coalescing is therefore per subscription, never per stream.** All of a - subscription's pending deltas, the basis included, are summed together up to one step, + subscription's pending deltas, the basis included, are summed together up to one + publication number, and sent as one batch in the fixed order. Coalescing one stream while another is sent as - it arrives would let a basis for step n+2 overtake the cells it describes, which is + it arrives would let a basis for publication n+2 overtake the cells it describes, which is gotcha 43's failure. - **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives the same result and delta types as a remote client. Its transport is a function call. @@ -1739,7 +1756,7 @@ options, in ADR 008's lettering: stays invisible until then. - **(b) An unaddressed change notice** (Flux style). After each accepted batch, the model emits one notice to every consumer, addressed to none. It carries: - - the step stamp `(document_generation, step)` (T3); + - the stamp `(host instance, document_generation, publication)` (T3); - each item edit, as an axis change (above); - a flag when the formula set or `measure_agg` moved. Either may move every aggregate without moving a record. `measure_agg` has no wire command today, and changes only @@ -1804,7 +1821,7 @@ its own shape key. The host keeps O(1) per client: a topic subscription, which n shape key and holds no window, cursor or view. **A topic filters only the moved-key lists; every batch's header goes to everyone** (found by the review of `e152b667`): - **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the step stamp (with the host instance, T3), `document_generation`, + batch.** It holds the stamp (host instance, `document_generation`, publication; T3), the axis changes, the display relabels, the formula and `measure_agg` flags, and the eviction list (below). None of these is per shape: an item insert shifts every shape with that category on an axis. @@ -1813,10 +1830,19 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 batch. That costs O(N) small headers per batch. - **The moved-key lists are filtered by shape.** A client receives only its own shape's list. -- **Switching shape key.** A client that changes its request's shape fetches under the - new key, which returns the stamp n the fetch was computed at. It subscribes to the new - topic from n on and drops the old topic. Headers never stopped arriving, so the switch - leaves no gap. +- **Subscribing, first or after a switch of shape key.** *As first written* + (superseded), this said a client fetches at publication n and then subscribes from n, and + that the switch "leaves no gap". **That was wrong** (found by the review of `70ababfe`). + The host keeps no past moved keys. So if batch n+1 moves a key in the new shape before + the subscription is registered, the client gets header n+1, with contiguous stamps, + but not n+1's key list, and its window goes stale silently. A client's first + subscription has the same race. *The author's lean:* **(i) the fetch registers the + subscription atomically.** The request names its topic, and the host registers the + lease in the same step that computes the fetch at n. Every later batch then routes to + it, and this fits "leases renewed by requests". The alternative, **(ii)**: the + subscription is acknowledged with the current publication number m, and the client refetches if + m > n. That costs a second round trip whenever a batch lands in between. The client + drops the old topic once the new fetch is in. - **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a shape once no recent request names it. A client still subscribed to that topic would then receive no moved keys, which looks exactly like "nothing changed", and its window @@ -1827,6 +1853,12 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + Evicting a popular shape therefore causes a **refetch storm**. *The author's + options*, unmeasured: refuse to evict a shape with many live subscriptions + (hysteresis), stagger the refetches with a per-client jitter, or rate-limit + rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list + names only the shapes evicted since the previous header, so it does not grow without + bound. **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists moved keys only for live shape keys, and a topic-less client pins nothing. So its shape @@ -2060,9 +2092,15 @@ displayed path until the differential test holds. question 3's (c) for that subtraction to be exact. 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to - 22. It brings the output types in `improvise-protocol`, the step stamp, + 22. It brings the output types in `improvise-protocol`, the publication stamp (T3), snapshot-then-delta, question 22's restricted successor query, and the test with two - consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff + consumers (T6). **Required tests:** + - a multi-effect batch such as `[AddItem, SetCell]` publishes exactly one header, and + no client refetches; + - a dropped header **is** detected as a gap, and the client refetches; + - a batch that lands between a fetch and its subscription reaches the client's key + list (question 13's atomic registration); + - a reconnect after a host restart refetches, and drops nothing as a duplicate. It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -2233,7 +2271,7 @@ revisions, and none is renumbered. question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: this ADR.** The options are: - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis + - (b) an unaddressed change notice per accepted batch, carrying the publication stamp, axis changes, display relabels, a formula or `measure_agg` flag, and the moved output keys per live shape key. The alternative is moved records plus a conservative rule for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); @@ -2242,7 +2280,10 @@ revisions, and none is renumbered. client, with the host instance in its stamp. Only the moved-key lists are filtered by shape key. A topic subscription is a lease that pins its shape, and any eviction is announced in the header. It costs - O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question + O(N) headers per batch, O(1) per-client state (which holds no UI state), and a + lease heartbeat, which is a second small message stream per client, between + requests. A subscription is registered atomically with the fetch that names its + topic, so no batch can fall between them. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant information")? (b′) depends on the answer; From 6ad3643790f4da64fbae06c325c816fd24323950 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:27:19 -0700 Subject: [PATCH 047/106] docs(adr): ADR 009 ties topic subscriptions to connections and leans to plain (b) with a size cap (improvise-felo.4) B10: a lease could lapse silently while another client pinned the shape; a subscription now lives exactly as long as its connection, which drops the heartbeat. Q13 compares the forms in complexity as well as traffic, and the author's lean returns to plain (b) with a size cap, with (b') as a measured optimisation. Client-visible step uses now say publication; two more required tests. Reads ADR 008 at a25fea81. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 112 ++++++++++++++-------- 1 file changed, 71 insertions(+), 41 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 29ffee43..48421fcc 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 35** -(PR #308, read at `1c974bf7`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 36** +(PR #308, read at `a25fea81`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `1c974bf7`, its revision 35) · +reads it at `a25fea81`, its revision 36) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 35 of `context/adr/008-virtualized-grid.md` on PR #308, at `1c974bf7`. Clauses 3, +revision 36 of `context/adr/008-virtualized-grid.md` on PR #308, at `a25fea81`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 35, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 36, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1290,7 +1290,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 35 at `1c974bf7`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 36 at `a25fea81`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1584,7 +1584,7 @@ search appears (next paragraph). What forward-only growth simplifies: and each step is one restricted successor query, so the host keeps no fill state at all. Under model-side pruning, `project` itself would scan forward from the top-left and send rows and columns as it finds them, as a sliced job resumable from `(request, fill -front)`. Either way, changes during a fill are identity-keyed deltas stamped by step +front)`. Either way, changes during a fill are identity-keyed deltas stamped by publication number (T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an ordinary delta with its absolute position, and the client keeps or drops the extra row by its own policy (Q42). So a fill and a live update are one stream, which keeps the @@ -1795,7 +1795,7 @@ options, in ADR 008's lettering: 22's successor query. A formula or `measure_agg` change means refetching the window. The host keeps no per-client state. - **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask - to be kept current. The host keeps the request and the last step it sent, then pushes + to be kept current. The host keeps the request and the last publication number it sent, then pushes the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost table costs a snapshot). But it is per-client state on the backend, and it holds a window, which is a client's view parameter. @@ -1811,7 +1811,7 @@ corrected after the review of `5a8dc486`; nothing here is measured). streaming into a large table, every remote client would receive every moved key, most of them for shapes it does not read. - **(c) keeps per-client state, and saves traffic.** It holds the request and the last - step per subscriber, which is a client's window and so a view parameter on the host, + publication number per subscriber, which is a client's window and so a view parameter on the host, against Q33. In return it sends each client only its window's deltas. - *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. @@ -1838,8 +1838,8 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 but not n+1's key list, and its window goes stale silently. A client's first subscription has the same race. *The author's lean:* **(i) the fetch registers the subscription atomically.** The request names its topic, and the host registers the - lease in the same step that computes the fetch at n. Every later batch then routes to - it, and this fits "leases renewed by requests". The alternative, **(ii)**: the + subscription in the same step that computes the fetch at n. Every later batch then + routes to it. The alternative, **(ii)**: the subscription is acknowledged with the current publication number m, and the client refetches if m > n. That costs a second round trip whenever a batch lands in between. The client drops the old topic once the new fetch is in. @@ -1847,12 +1847,22 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 shape once no recent request names it. A client still subscribed to that topic would then receive no moved keys, which looks exactly like "nothing changed", and its window would go stale. *The author's lean* combines two invariants: - - **(i) a live topic subscription pins its shape.** Subscriptions are leases, renewed - by the client's requests or a heartbeat. Host state grows with the live - subscriptions, and a client that vanishes is released when its lease expires. - - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under + - **(i) a live topic subscription pins its shape, and a subscription lives exactly as + long as its connection** (revised after the review of `85dc0ad8`). *As first + written* (superseded), subscriptions were leases renewed by requests or a + heartbeat. That let a client's own lease lapse silently while another client still + pinned the shape. The shape was then not evicted and nothing was announced, so the + client kept receiving contiguous headers but no key lists. Tying the subscription to + the connection (the socket, the worker's port, or the TUI's in-process call) removes + the heartbeat and the lapse together. A live connection keeps its subscriptions. A + dropped connection drops them. A client always knows when its connection dropped, + and it refetches on reconnect (T3's general rule). *The author's lean*, because it is + the simplest of the three. The alternatives were a per-client subscription epoch in + that client's header, or acknowledged renewals with a self-unsubscribe on a missed + acknowledgement. Each adds a mechanism. + - **(ii) every eviction is announced.** Any eviction, by ageing out or under memory pressure (clause 14), puts the shape key in the next header's **eviction - list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. Evicting a popular shape therefore causes a **refetch storm**. *The author's options*, unmeasured: refuse to evict a shape with many live subscriptions (hysteresis), stagger the refetches with a per-client jitter, or rate-limit @@ -1871,13 +1881,13 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 The alternative, **(iii)**: every header states, for each shape it carries keys for, that the shape was evaluated, and a subscribed shape missing from that statement means - a conservative refetch. It needs no leases, and it costs a list of evaluated shapes + a conservative refetch. It needs no pins, and it costs a list of evaluated shapes per header. - **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) per batch. It saves nothing when every client reads one shape, and much when shapes differ. It is still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** A subscription is per-client state, with - a lease. It holds no UI state, and the next request, which names its shape, rebuilds it. +- **"As stateless as possible" (C2) and Q33.** A subscription is per-connection state. + It holds no UI state, and the next request, which names its shape, rebuilds it. The author reads it as delivery state, not UI state. **But the user said "The backend should only hold model-relevant information"** (ADR 008 Q33), and whether a topic subscription counts as that is the user's question, put in question 13. @@ -1885,11 +1895,31 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 and remote consumers still read the same notices (T6). The TUI simply subscribes to its own shape. -*The author's recommendation, revised:* **(b′)**. It keeps the host free of UI state, -while cutting (b)'s traffic to the shapes each client reads. A client that names no -topic receives the full broadcast, which is (b). (c) stays an optimisation for a client -whose window is a small part of a busy shape, if measured traffic justifies holding a -window on the host. All of this is unmeasured. +**What each form costs in complexity, not only in traffic** (the author's; nothing +measured). The mechanism (b′) needed has grown review by review: topics, pins, +atomic registration, an eviction list, and a lease (now a connection tie). That growth +is itself a signal. +- **Plain (b): moved keys per live shape, broadcast to every client.** It keeps no + per-client state and needs no topics, pins or registration. It still needs the eviction + list, because a client's shape can be evicted. Its traffic is O(N × all moved keys) per + batch. **With a size cap**, a header whose key lists exceed a fixed size carries "too + many changes, refetch" instead of the lists. That bounds every notice, at the price of a + refetch by every client in a busy batch. +- **The records-only variant of (b).** It sends moved records and a formula flag, and + windows with cross-stem formulas refetch on every batch. It needs no shape lists, so it + needs no eviction list either. It is the simplest form, and its notices are the + smallest. It refetches the most. +- **(b′): (b) filtered by connection-tied topics.** It cuts traffic to the shapes each + client reads. The price is per-connection state, which is up against Q33, plus atomic + registration at fetch and pins on the shape cache. + +*The author's recommendation, revised again:* **plain (b) with a size cap**, keeping no +per-client state, with **(b′)** as the optimisation to adopt only if traffic is measured to +matter with many clients on different shapes. (c) stays the further optimisation for a +small window on a busy shape. (b′) is defensible against (b) with a cap only where +clients read many different shapes of one busy document, and nothing here shows that +case is common. The user may well prefer the simpler design. *As first written* +(superseded), the lean was (b′). *Earlier still* (superseded), this clause leaned to (c), which it then called (b). That was before the user said the backend holds only model-relevant information. Under (b′), @@ -1915,7 +1945,7 @@ Eviction drops its operators, is announced in the next header's eviction list, a next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has not set (question 14). -**What a client must hold to resume:** its request, the last step it applied, its window, +**What a client must hold to resume:** its request, the last publication number it applied, its window, and its basis. The host holds nothing that a client needs back. **The basis.** ADR 001's basis is answered with the result it describes, and after it @@ -2100,7 +2130,10 @@ displayed path until the differential test holds. - a dropped header **is** detected as a gap, and the client refetches; - a batch that lands between a fetch and its subscription reaches the client's key list (question 13's atomic registration); - - a reconnect after a host restart refetches, and drops nothing as a duplicate. It depends on step 4 for occupancy. It replaces the per-mirror diff + - a reconnect after a host restart refetches, and drops nothing as a duplicate; + - a refused frame consumes no publication number; + - under (b′), a dropped connection drops its subscriptions, and the client refetches on + reconnect, even while another client still pins the shape. It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -2278,25 +2311,23 @@ revisions, and none is renumbered. - (b′) the same notices, delivered by topic. Every batch's header (stamp, `document_generation`, axis changes, relabels, flags, eviction list) goes to every client, with the host instance in its stamp. Only the moved-key lists are filtered - by shape key. A topic subscription - is a lease that pins its shape, and any eviction is announced in the header. It costs - O(N) headers per batch, O(1) per-client state (which holds no UI state), and a - lease heartbeat, which is a second small message stream per client, between - requests. A subscription is registered atomically with the fetch that names its - topic, so no batch can fall between them. **Question + by shape key. A topic subscription lives exactly as long as its connection, and pins + its shape. It is registered atomically with the fetch that names its topic. It costs + O(N) headers per batch and per-connection state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant information")? (b′) depends on the answer; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. - *Recommendation (the author's, revised):* (b′). It holds no UI state on the host and - cuts (b)'s traffic to the shapes each client reads. (b) is what a client that names no - topic gets. (c) is an optimisation if measured traffic justifies holding a window. - Under all three, `GridExtent` is derived by the client (in records mode it is - fetched), `Dirty` is retired with the mirrors, and the occupancy delta stays - model-side, feeding question 22's index. *As first written*, this recommended plain - (b); the trade it skipped is wire traffic. + *Recommendation (the author's, revised again):* **plain (b) with a size cap**, which + keeps no per-client state, with (b′) as a measured optimisation and (c) beyond it. The + simpler choice is the user's to prefer. (b′) buys traffic savings at the price of + connection-tied topics, pins, atomic registration and per-connection state against + Q33. The records-only variant of (b) is simpler still, and refetches more. Under every + option, `GridExtent` is derived by the client (in records mode it is fetched), `Dirty` + is retired with the mirrors, and the occupancy delta stays model-side, feeding + question 22's index. *Earlier* (superseded): (b), then (b′). 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2306,8 +2337,7 @@ revisions, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while **and no live topic subscription pins it** (question 13's (b′)). An eviction is announced in the next notice header, so a subscriber refetches and is never left - with a window that looks current and is not. The time and the lease length are - implementation numbers. An unsaved collapse, reorder, page + with a window that looks current and is not. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved hide or show of a row-axis item shares the key too. Under question 23's default (A) its totals subtract the hidden rows at read time and add nothing to the cache. Under From e6a52790554398307fe2aa4e33354579d342833f Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:36:41 -0700 Subject: [PATCH 048/106] docs(adr): ADR 009 folds the user's answers to questions 1-10 and 12 (improvise-felo.4) Adds "What the user decided" with the eleven comments of review 5465446805, verbatim. zset, the seam crate and the exact accumulator are decided; Eager is deprecated and kept for now; the cycle bound and component fixed point stay for testing with a DBSP-native plan; the rollback snapshot stands; felo.4 records the ADR only and a new epic takes the steps; clause 13 becomes the decided direction for string-value columns. New questions 25-27 hold the residue. Also: dead- connection detection and the idle-reader eviction storm. Reads ADR 008 at c8be7720; merges gh/main f0fb6880. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 249 +++++++++++++++++----- 1 file changed, 198 insertions(+), 51 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 48421fcc..05e14ef8 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,23 +1,23 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 36** -(PR #308, read at `a25fea81`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and +12** (see "What the user decided"). Question 11 and questions 13 to 27 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** +(PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of -this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline -comments are Copilot's two and our replies. · +ADR", in clauses 12 to 14, and in questions 12 to 27. *Superseded:* until 2026-10-09 the +user had answered none of questions 1 to 11. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the +**Relates to:** ADR 005 (exact decimals, improvise-l2b8; PR #320 merged on 2026-10-09, and its own status line still reads proposed), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `a25fea81`, its revision 36) · +reads it at `c8be7720`, its revision 37) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 -is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +is on main since PR #320 merged (2026-10-09), and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 36 of `context/adr/008-virtualized-grid.md` on PR #308, at `a25fea81`. Clauses 3, +revision 37 of `context/adr/008-virtualized-grid.md` on PR #308, at `c8be7720`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 36, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 37, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -320,7 +320,7 @@ the other. Both stay in the tree, and a differential test holds them equal.** Fo clauses, plus clause 5a. It is numbered 5a so that no clause number or question number cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. -### 1. The crate: `zset` (name is question 1) +### 1. The crate: `zset` (the user's answer to question 1) Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond `std`, so it is wasm32-clean by construction. @@ -403,8 +403,13 @@ itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental improvise-core never depends on `zset`, and it can be built and tested eager-only (question 2). -**Why keep both.** `Eager` is the oracle for the differential test. It is also the -backend a query-only source keeps (clause 10). The differential test is a property test +**Why keep both, for now.** `Eager` is the oracle for the differential test. It was +also to be the backend a query-only source keeps (clause 10). **The user: "Eager should +be deprecated", and "For now: the goal is to only have DBSP formulas"** (questions 8 and +6). So `Eager` stays only as long as the differential test needs an oracle, and what +replaces it for query-only sources and for clause 5a's non-record-local formulas is +question 26. *As first written* (superseded), this paragraph and question 6 kept `Eager` +permanently. The differential test is a property test in the seam crate. It generates documents with groups, collapse, hidden items, formulas (including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text cells and non-finite numbers, applies the same random edit sequence to both backends, @@ -814,6 +819,14 @@ occupied-contexts fallback above as well is question 18. ### 6. Circular references: DBSP's recursion is not our semantics +**The user, on questions 4 and 5:** the sixteen-pass bound is kept "for testing purposes +but eventually remove it because DBSP should fix the cycle issue", and the component +fixed point "For validation, but if there's a more natural DBSP-native way we should plan +on that." So what follows is the **current** plan, kept for testing and validation, and a +DBSP-native treatment of cycles is to be planned. The argument below, that DBSP's least +fixed point is not improvise's circular semantics, is the author's. It is the open part, +question 25. + DBSP handles recursion by computing a **least fixed point**: nested streams iterate a monotone operator, and `distinct` keeps the lattice finite, so reachability converges. improvise has no such guarantee: @@ -1285,12 +1298,12 @@ every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting -for ADR 005**, provided `Eager` uses the same accumulator. That is question 3's new -option, and its recommendation. +for ADR 005**, provided `Eager` uses the same accumulator. That was question 3's option +(c), and **the user chose it**: "use the exact accumulator." (4226373188). ### 9. What ADR 008 gets from it -ADR 008 (revision 36 at `a25fea81`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 37 at `c8be7720`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1359,8 +1372,9 @@ change feed, produces a cell delta. That is the same input `DataStore` produces, circuit cannot tell them apart. That is the "data can stream" half of the user's prompt. -**A source that can only answer aggregate queries stays on `Eager`,** with ADR 003's -pushdown. There is a tension: ADR 003 wants data larger than memory, and every +**A source that can only answer aggregate queries stays on `Eager` for now,** with ADR +003's pushdown. The user has said "Eager should be deprecated" (question 8), so this is +an interim, and its replacement is question 26. There is a tension: ADR 003 wants data larger than memory, and every integrated operator here keeps state proportional to its input (clause 4). So the backend is chosen per document, which is what "pluggable" buys (question 8). A read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. @@ -1384,7 +1398,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:693` at `50e1314d`). **Recommended (question 9): the backend's state joins the snapshot.** + (`app.rs:693` at `50e1314d`). **Decided (question 9, the user: "The recommendation is fine, I think"): the backend's state joins the snapshot.** Applying the inverse delta is **not** a rollback, even with exact aggregators. It restores integrated values, but `step(d)` then `step(−d)` still advances the circuit and changes the history that `delay` and `differentiate` hold. So the next step after @@ -1855,7 +1869,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 client kept receiving contiguous headers but no key lists. Tying the subscription to the connection (the socket, the worker's port, or the TUI's in-process call) removes the heartbeat and the lapse together. A live connection keeps its subscriptions. A - dropped connection drops them. A client always knows when its connection dropped, + dropped connection drops them. **The transport must detect a dead connection**: a + websocket ping with a timeout on the socket server, a closed port for the worker, and + nothing for the TUI, which is in process. A half-open socket that is never detected + would hold its pins forever. A client always knows when its connection dropped, and it refetches on reconnect (T3's general rule). *The author's lean*, because it is the simplest of the three. The alternatives were a per-client subscription epoch in that client's header, or acknowledged renewals with a self-unsubscribe on a missed @@ -1863,7 +1880,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 - **(ii) every eviction is announced.** Any eviction, by ageing out or under memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. - Evicting a popular shape therefore causes a **refetch storm**. *The author's + Evicting a popular shape therefore causes a **refetch storm**. **Idle readers make + it likely under plain (b):** a shape that clients only watch, without requesting, + ages out, and every one of them refetches at once. That refetch is also what brings + the shape back. *The author's options*, unmeasured: refuse to evict a shape with many live subscriptions (hysteresis), stagger the refetches with a per-client jitter, or rate-limit rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list @@ -1921,6 +1941,13 @@ clients read many different shapes of one busy document, and nothing here shows case is common. The user may well prefer the simpler design. *As first written* (superseded), the lean was (b′). +**The user's answer to question 7 bears on this, and does not decide it.** "As much state +as possible should be maintained in the backend, unless it's UI-specific state", and "we +probably need to have a mix of caching and a pull model to expand the cached data". Plain +(b) keeps the maintained model state in the backend and has clients pull on a notice, +which fits both sentences. Whether a per-connection topic is backend state of the kind +the user means, or UI-specific, is still question 13's question to the user. + *Earlier still* (superseded), this clause leaned to (c), which it then called (b). That was before the user said the backend holds only model-relevant information. Under (b′), the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is @@ -1954,12 +1981,26 @@ and its basis. The host holds nothing that a client needs back. cursor. Under the user's Q39 answer, only `document_generation` and the record tier are model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). -### 13. Incremental max width (the user's question on ADR 008) +### 13. Incremental max width: the host maintains it for string-value columns The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" -**This clause is the author's analysis, and none of it is measured.** The user's decision -on widths stands as ADR 008 records it: (b), sticky per session, eased by hysteresis, -taken over the rows realized and held by the view layer (ADR 008 clause 4). +**The user then decided** (question 12, 4226624890): "Yes, I think the host should +maintain the max width for string-value columns incrementally." So **the host maintains +each string-value column's maximum display width incrementally**. This clause is now +that direction. Its cost analysis is the author's, and none of it is measured. + +**Scope.** "String-value columns" is read, by the author, as columns whose cells hold +text values (`CellValue::Text`), as opposed to numbers formatted for display. Numbers' +widths depend on the number format, which is display-only, so they stay with ADR 008's +rule: (b), sticky per session, eased by hysteresis, over the rows realized, held by the +view layer (ADR 008 clause 4). Whether the user means this split, and what a column +holding both text and numbers does, is question 27. + +**For text, two costs below shrink.** A text value's display width does not depend on +`number_format`, so there is no rebuild when the format changes, and no formatting on +the write path, only a display-width measurement of the string. What remains is the +histogram per column, and the delta of text cells over the whole column, including fold +rows. **DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. When the widest value is retracted, the new maximum cannot be computed from the old @@ -1992,13 +2033,12 @@ three things: - **A rebuild whenever the format changes.** `number_format` is a parameter of the view definition, and a rebuild costs O(occupied rows) per column, as a job. -**What it would buy, set against what the user decided.** A maintained whole-column width -is ADR 008's question 2 option (c), "size each column from its whole contents", which the -user did **not** choose. Under (b) a window's widths are an O(visible) computation over -cells the window already evaluates, so maintaining them incrementally adds nothing. So -the real question is whether the user wants (c) after all, now that its state is cheap. It -would remove the jitter at its source and make hysteresis unnecessary, at the upstream -cost above. That is question 12. +**Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 +option (c), "size each column from its whole contents". The user had chosen (b) there, +and now chooses the maintained width for string-value columns. So the two rules split by +column kind, and ADR 008 should record that (question 27). *As first written* +(superseded), this paragraph recommended not maintaining widths "now", and asked the +user in question 12. The user answered yes. ### 14. Resident state, and the store-memory requirement @@ -2115,8 +2155,8 @@ displayed path until the differential test holds. record's leading cell. Formula values that depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact - accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's +6. **`Sum` and `Avg`, and column totals**, with the exact accumulator in both backends, + as the user decided on question 3. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs question 3's (c) for that subtraction to be exact. @@ -2137,10 +2177,11 @@ displayed path until the differential test holds. only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). -**Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is -question 10. improvise-swre stays independent and is useful under either backend. +**Tickets.** improvise-felo.4 is this ADR's record **only**. The user: "No, new epic and +felo.4 records ADR only" (question 10). So the steps above go to a **new implementation +epic**, which this revision does not file. improvise-swre stays independent and is useful under either backend. improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is -already inexact beyond `FLOAT_EQ_EPSILON`. Under question 3's recommendation it is no +already inexact beyond `FLOAT_EQ_EPSILON`. Under the user's answer to question 3 it is no longer a prerequisite of step 6. **Good.** @@ -2186,23 +2227,92 @@ longer a prerequisite of step 6. - A position over a changing item list is never fully absolute. An item insert moves every later position, and is sent as an axis change (clause 12). +## What the user decided + +**On 2026-10-09 the user answered questions 1 to 10 and 12** in inline comments on #309, +in review 5465446805 (03:34:52Z to 04:32:13Z). They are quoted exactly below, double +spaces and the "is is" kept. Question 3's comment is anchored on an older head +(`d766605b`, line 1100), where that line is question 3's option (c). Each other comment +is mapped from its `original_line` on `6ad36437` to the question it closes. **The user +has not said "Approved"**, so the ADR stays proposed. Question 11 is unanswered. + +- **1.** (4226370667) "zset is is fine." The crate is `zset` (clause 1). +- **2.** (4226371183) "seam crate". `Incremental` lives in the seam crate + `improvise-incremental` (clause 2). +- **3.** (4226373188) "use the exact accumulator." Option (c): `ExactSum` for `Sum` and + `Avg` over f64 (clause 8, migration step 6). +- **4.** (4226588257) "Yeah, for testing purposes but eventually remove it because DBSP + should fix the cycle issue." The sixteen-pass bound is reproduced so that the + differential test compares like with like, and is to be removed later (clause 6; + question 25). +- **5.** (4226590267) "For validation, but if there's a more natural DBSP-native way we + should plan on that." The component fixed point is kept for validation, and a + DBSP-native treatment of cycles is to be planned (clause 6; question 25). +- **6.** (4226592632) "For now: the goal is to only have DBSP formulas". `Eager` stays + for now, and the goal is that formulas are evaluated by DBSP only (clause 2; + question 26). +- **7.** (4226599186) "As much state as possible should be maintained in the backend, + unless it's UI-specific state. As far as possible, state should be incrementally + recomputed using DBSP operators to minimize per-request recalculation." And + (4226616602) "However, this is intrinsically limited because the grid is infinite, so + we probably need to have a mix of caching and a pull model to expand the cached data." + *The author's reading:* this endorses question 7's slice, which keeps every page's + state maintained in the backend, and it states a principle that bears on questions 13 + and 14 (clause 12). +- **8.** (4226618377) "Eager should be deprecated." This **overrides** question 8's + recommendation, which kept query-only sources on `Eager` (clause 10; question 26). +- **9.** (4226622610) "The recommendation is fine, I think". The backend's state joins + the rollback snapshot, with no later switch to the inverse delta (clause 11). +- **10.** (4226621783) "No, new epic and felo.4 records ADR only". This **overrides** the + recommendation: improvise-felo.4 stays the record of this ADR, and the implementation + steps go to a new epic. Nothing is filed by this revision. +- **12.** (4226624890) "Yes, I think the host should maintain the max width for + string-value columns incrementally." This **overrides** the author's recommendation + ("no, not now"). Clause 13 is rewritten as the decided direction; its scope is + question 27. + +**What the answers change, and where they pull against earlier text** (the author's +readings, flagged): +- **`Eager` is deprecated, not permanent** (questions 6 and 8). Clause 2's "why keep + both", clause 10's query-only sources on `Eager`, and clause 5a's eager path for + formulas the per-record map cannot express all assumed a permanent `Eager`. They now + hold "for now", as the differential test's oracle. What replaces `Eager` for those two + uses is question 26. +- **"Only DBSP formulas"** (question 6) pulls against clause 2's rule that formulas are + never compiled into operators, and that the one existing evaluator computes each + value. The user's words may mean "every formula goes through the circuit", which + clause 2 already allows, or "formulas become operators". Question 26. +- **"DBSP should fix the cycle issue"** (question 4) pulls against clause 6's argument + that DBSP's least fixed point is not improvise's circular semantics. `X = X + 1` has no + fixed point. Question 25. +- **"As much state as possible ... in the backend"** (question 7) sits beside the + user's ADR 008 Q33 answer, "The backend should only hold model-relevant information". + The two agree for derived model state, such as the shape cache and maintained + aggregates, and both exclude UI state. "A mix of caching and a pull model to expand the + cached data" fits question 13's plain (b), where clients pull on a notice, and question + 14's shape cache. It does not decide question 13 or 14, and does not say whether a + per-connection topic counts as backend state. +- **String-value column widths are maintained by the host** (question 12). That + narrows, for those columns, ADR 008's decided width rule: (b), sticky per session with + hysteresis, over the rows realized. ADR 008 should record it. Question 27. + ## Open questions Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 24 were added by the 2026-10-05 and 2026-10-06 -revisions, and none is renumbered. +the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered +questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10-05, +2026-10-06 and 2026-10-09 revisions, and none is renumbered. -1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? +1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. -2. *[Open.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on +2. *[Answered: a seam crate.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on `zset` and `improvise-core`), or inside improvise-core? *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. *[Open.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? +3. *[Answered: (c), the exact accumulator.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums until then. - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in @@ -2229,35 +2339,35 @@ revisions, and none is renumbered. that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - `Avg`'s one rounding must be stated in step 6. -4. *[Open.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported +4. *[Answered: reproduce it for testing, and remove it eventually; question 25.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to change, change it in both backends at once, as its own decision. -5. *[Open.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by +5. *[Answered: yes, for validation, and plan a DBSP-native way; question 25.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by today's bounded fixed point, restricted to their component, rather than by any operator in the crate? *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the cells a guarded cycle leaves convergent. The crate needs no recursion operator. -6. *[Open.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? +6. *[Answered: for now; the goal is DBSP formulas only; question 26.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. *[Open. ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so +7. *[Answered in principle (the author's reading: the slice); see "What the user decided". Earlier note: ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". -8. *[Open.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, +8. *[Answered: "Eager should be deprecated", which overrides the recommendation; question 26.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? *Recommendation:* yes, chosen per document. Integrated state is proportional to the source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. *[Open. Clause 14 now weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` +9. *[Answered: the recommendation, hedged "I think". Clause 14 also weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` snapshot that `Host::mutating_frame` already takes? *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying `−d` restores integrated values but not the history `delay` and `differentiate` @@ -2266,7 +2376,7 @@ revisions, and none is renumbered. today. A true rewind would need its own design and an observational-equivalence test over every stateful operator. -10. *[Open.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps +10. *[Answered: no; a new epic, and felo.4 records the ADR only.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are @@ -2291,7 +2401,7 @@ revisions, and none is renumbered. rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn order, so that rebuild does not happen, and a position key would go stale. -12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). +12. *[Answered: yes, for string-value columns; the recommendation below is overridden; question 27.]* **Incremental max width** (clause 13; the user's question on ADR 008). Should the host maintain each column's whole-column maximum display width? That is ADR 008's question 2 option (c), and its state would now be a small histogram per column. *Recommendation (the author's):* no, not now. The user decided on (b) with @@ -2503,3 +2613,40 @@ revisions, and none is renumbered. need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the client. Not decided. + +25. *[New.]* **What "DBSP should fix the cycle issue" means** (clause 6; the user's + answers to questions 4 and 5). Clause 6 argues that DBSP's recursion computes a least + fixed point, which improvise's circularity is not: `X = X + 1` over a stored 10 has + no fixed point, and the required answer is `Error("circular")`. A DBSP-native + treatment could be: + - (a) a recursion operator bounded at sixteen iterations, which writes `circular` + where it has not converged, so the bound stays part of the semantics; + - (b) structural detection: every cell in a cycle of the formula graph is `circular`, + which changes today's answer for guarded cycles such as `A = IF(x, B, 1)`; + - (c) a monotone reformulation, where one exists. + + *Recommendation (the author's):* (a) for the first DBSP-native step, because it + keeps today's answers, and the differential test can then retire the per-cell fixed + point. Not decided. + +26. *[New.]* **What replaces `Eager`, and what "only DBSP formulas" means** (clause 2; + the user's answers to questions 6 and 8). Two parts: + - (a) Does "only DBSP formulas" mean every formula is evaluated through the circuit, + with the one per-cell evaluator inside it as clause 2 has it? Or that formulas are + compiled into operators, which clause 2 rejects as a second evaluator? + - (b) Once `Eager` is gone, what serves a query-only live source (clause 10), and the + formulas the per-record map cannot express (clause 5a)? And what is the + differential test's oracle? + + *Recommendation (the author's):* (a) the first reading, which keeps one evaluator. + (b) the occupied-contexts fallback becomes a circuit stage. A query-only source gets + its own pushdown backend behind clause 2's trait, not `Eager`. The differential test's + oracle becomes a small, slow, from-scratch reference implementation kept in the test + crate only. Not decided. + +27. *[New.]* **Which columns are "string-value columns"** (clause 13; the user's answer + to question 12). Is it columns whose cells are text, so that numeric columns keep ADR + 008's (b) with hysteresis? And what does a column holding both text and numbers do? + *Recommendation (the author's):* a column whose drawn cells include any text keeps a + maintained width for its text cells. The view layer takes the larger of that and its + held numeric width. ADR 008 should record the split. Not decided. From 74e66b34bbe149f0c2696c2f16baac0cbc6a0337 Mon Sep 17 00:00:00 2001 From: "Claude (rules-merge-main)" Date: Thu, 8 Oct 2026 21:41:30 -0700 Subject: [PATCH 049/106] docs(wave): agents merge gh/main early and often (rule 9) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- .agents/skills/wave/SKILL.md | 1 + .agents/skills/wave/agent-rules.md | 17 ++++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/.agents/skills/wave/SKILL.md b/.agents/skills/wave/SKILL.md index 3772166c..10b62ce1 100644 --- a/.agents/skills/wave/SKILL.md +++ b/.agents/skills/wave/SKILL.md @@ -45,6 +45,7 @@ nobody corrects. The brief adds only what varies per wave: 1. **The expected base commit, by name.** Worktrees do not reliably branch from HEAD. One was created three merges behind and was only noticed because it was told to check. The agent checks against it: `agent-rules.md` rule 1. + It then merges `gh/main` as it moves, not only at the start: rule 9. 2. **A reference baseline**, so "N passing" is checkable and a drop is visible: Rust test count, harness counts, node counts, **and the machine**. The same commit once measured `improvise-io`'s binary at 1.4s on one machine diff --git a/.agents/skills/wave/agent-rules.md b/.agents/skills/wave/agent-rules.md index 9bcc7934..fc6c1756 100644 --- a/.agents/skills/wave/agent-rules.md +++ b/.agents/skills/wave/agent-rules.md @@ -32,7 +32,7 @@ not in a second copy. 6. **The Beads "Session Completion" block near the end of `AGENTS.md` does not set your git policy.** This repository does not opt in to the block's "team-maintainer" profile. On a wave branch that profile's pull and bare - `git push` would be refused (rule 4), or worse. Rules 1–5, 7 and 8 are your + `git push` would be refused (rule 4), or worse. Rules 1–5, 7, 8 and 9 are your git policy. The block is bd's and bd regenerates it whole, so it is not edited by hand: `AGENTS.md`, "Repository policy over the Beads blocks" (improvise-x1uk). 7. **Pin `gh/main` to a literal SHA when the task starts.** Run @@ -101,6 +101,21 @@ not in a second copy. pinned SHA and `git cherry-pick -x` your own commits onto it (`-x` records where each came from). Close the old branch rather than force-pushing it: `AGENTS.md`, "Shipping a wave". +9. **Merge main as soon as you can, and keep merging it.** The user: + "Agents should merge main asap" and, asked the point, "Reduces conflicts" + (2026-10-09). `git fetch gh`, then merge the new `gh/main` SHA (rule 7's + re-pin, done on purpose) at the start of a task, before every push, and + every time you resume. Main only gets further from your base, and a + conflict met early is small. Never `origin/main`, never rebase, never + squash (rule 8). Commit the merge through `scripts/agent-git`. + - **Resolve a conflict by hand.** For each conflicted file, diff the + result against both parents and the merge base, so a side's + non-conflicting changes are not lost to a whole-file `--ours` or + `--theirs`. Check that no test name was lost. Then re-run the gates + (see "Gates") on the merged tree, not on each side. + - The mechanics are `AGENTS.md`, "Shipping a wave" (merge `gh/main`, never + `origin/main`) and "Merging many PRs at once" (diff each parent; two + clean changes to one test file can still fail together). Not restated here. ## Commit early: a worktree is not durable storage From 3335ace76e3be98f3a0ed65d996f09f8ca3668e0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:42:22 -0700 Subject: [PATCH 050/106] docs(adr): ADR 009 scopes clause 13's costs to numbers, restates ADR 005 as merged, pings both ways (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 26 ++++++++++++++++------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 05e14ef8..f1d2c49d 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1125,7 +1125,10 @@ for floating point. On its own, **none**: **The benefits come from two other places.** - **The representation.** An exact accumulator (next subsection), or ADR 005's decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian - group, provided both backends use the same accumulator. Decimals alone do not: ADR + group, provided both backends use the same accumulator. Under ADR 005 as merged that + holds **per partition**: the 320-bit decimal partition and each fraction group. A read + that combines more than K = 8 distinct non-decimal denominators is inexact by ADR 005's + cutoff. Decimals alone do not: ADR 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at its 28-digit bound, so the order of summation changes both the value and the exactness. Then retraction is exact, incremental @@ -1243,7 +1246,7 @@ operator rather than one per call site. | `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 as merged (option (d)): a three-level tower, `Decimal` / `Fraction(Box>)` / `Inexact(f64)`, in `numtower`. `rust_decimal` alone was measured at +15.4 KB gzipped, and `num-rational` with `num-bigint` at about +20.8 KB on their own; the combination is unprobed. `numtower` is not yet in the workspace (implementation epic improvise-59nw). Its sum keeps a 320-bit decimal partition and a 256-bit numerator per fraction group, and a read combines at most K = 8 groups (the user's final answer; K = 32 was superseded). The 40 bytes per summed group cover the decimal partition only | **The accumulator does not make `=` against a typed decimal right.** It sums the *stored* f64s exactly, and each stored f64 is already the rounded form of what the user @@ -1871,7 +1874,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 the heartbeat and the lapse together. A live connection keeps its subscriptions. A dropped connection drops them. **The transport must detect a dead connection**: a websocket ping with a timeout on the socket server, a closed port for the worker, and - nothing for the TUI, which is in process. A half-open socket that is never detected + nothing for the TUI, which is in process. **The ping runs both ways.** A client must + also detect a dead host, or it reads silence as "no batches" and keeps a stale + window. So a client that hears neither a header nor a ping within the timeout treats + its connection as dropped, and refetches on reconnect. A half-open socket that is never detected would hold its pins forever. A client always knows when its connection dropped, and it refetches on reconnect (T3's general rule). *The author's lean*, because it is the simplest of the three. The alternatives were a per-client subscription epoch in @@ -2016,21 +2022,25 @@ count, per column. It is O(distinct widths) per column, whatever the number of r an update costs O(log d), where d is the number of distinct widths. **The cost is upstream, in what feeds the histogram.** Its input is the display width of -**every evaluated cell in the column**, under the request's number format. That needs -three things: +**every evaluated cell in the column**. Of the three costs below, the second and third +apply **only to numbers**, which a host-maintained width would cover only if question 27 +extends the decision to them. For text, the first one remains: - **A delta of evaluated cells over the whole column**, not a window. Clause 4's group aggregates and clause 5's formula re-evaluation already produce an output delta for every unfolded cell that moves, with its old value as the retraction. So unfolded rows need no new state. **Fold rows are not covered.** Under item grain (clause 7) a collapsed group's value is combined when it is read, not maintained, so its width would - need the fold maintained per column at the visible grain. -- **Formatting every changed cell on the host**, on the write path: `format_f64` and a + need the fold maintained per column at the visible grain. That holds for text as well: + a fold row's text, under the unanimity rule, is a combined value. +- **A client's hidden rows** come out of the whole-column histogram the way they come out + of a total: subtracted at read time by question 23's (A) overlay, at O(H log d). +- **Numbers only: formatting every changed cell on the host**, on the write path: `format_f64` and a display width for each changed cell. Each one is cheap, but it would be the first display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But today the host is asked for widths only over the rows requested (ADR 008 clause 4), and this would widen that to whole columns. -- **A rebuild whenever the format changes.** `number_format` is a parameter of the view +- **Numbers only: a rebuild whenever the format changes.** `number_format` is a parameter of the view definition, and a rebuild costs O(occupied rows) per column, as a job. **Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 From 4d04b61b69978935f51570791bf6a058cf77578a Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 21:53:37 -0700 Subject: [PATCH 051/106] feat(numtower): Num, contagion, exact comparison, number language and sums (improvise-59nw.1) WIP: the crate compiles and its unit tests pass; property and differential tests follow. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- Cargo.lock | 167 ++++++++++++- Cargo.toml | 2 +- crates/numtower/Cargo.toml | 25 ++ crates/numtower/examples/bench.rs | 1 + crates/numtower/src/arith.rs | 378 ++++++++++++++++++++++++++++++ crates/numtower/src/cmp.rs | 92 ++++++++ crates/numtower/src/fixed.rs | 232 ++++++++++++++++++ crates/numtower/src/lang.rs | 181 ++++++++++++++ crates/numtower/src/lib.rs | 20 ++ crates/numtower/src/num.rs | 202 ++++++++++++++++ crates/numtower/src/ratio.rs | 172 ++++++++++++++ crates/numtower/src/round.rs | 85 +++++++ crates/numtower/src/sum.rs | 258 ++++++++++++++++++++ 13 files changed, 1812 insertions(+), 3 deletions(-) create mode 100644 crates/numtower/Cargo.toml create mode 100644 crates/numtower/examples/bench.rs create mode 100644 crates/numtower/src/arith.rs create mode 100644 crates/numtower/src/cmp.rs create mode 100644 crates/numtower/src/fixed.rs create mode 100644 crates/numtower/src/lang.rs create mode 100644 crates/numtower/src/lib.rs create mode 100644 crates/numtower/src/num.rs create mode 100644 crates/numtower/src/ratio.rs create mode 100644 crates/numtower/src/round.rs create mode 100644 crates/numtower/src/sum.rs diff --git a/Cargo.lock b/Cargo.lock index 77a3c5b8..51a21dda 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -97,6 +97,12 @@ dependencies = [ "num-traits", ] +[[package]] +name = "arrayvec" +version = "0.7.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56" + [[package]] name = "atomic" version = "0.6.1" @@ -178,6 +184,30 @@ dependencies = [ "hybrid-array", ] +[[package]] +name = "borsh" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a" +dependencies = [ + "borsh-derive", + "bytes", + "cfg_aliases", +] + +[[package]] +name = "borsh-derive" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56" +dependencies = [ + "once_cell", + "proc-macro-crate", + "proc-macro2", + "quote", + "syn 3.0.6", +] + [[package]] name = "bumpalo" version = "3.20.3" @@ -705,6 +735,17 @@ dependencies = [ "version_check", ] +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + [[package]] name = "getrandom" version = "0.3.4" @@ -1216,6 +1257,16 @@ dependencies = [ "minimal-lexical", ] +[[package]] +name = "num-bigint" +version = "0.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367" +dependencies = [ + "num-integer", + "num-traits", +] + [[package]] name = "num-conv" version = "0.2.2" @@ -1233,6 +1284,26 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "num-integer" +version = "0.1.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + [[package]] name = "num-traits" version = "0.2.19" @@ -1251,6 +1322,18 @@ dependencies = [ "libc", ] +[[package]] +name = "numtower" +version = "0.1.0" +dependencies = [ + "num-bigint", + "num-integer", + "num-rational", + "num-traits", + "proptest", + "rust_decimal", +] + [[package]] name = "once_cell" version = "1.21.4" @@ -1449,6 +1532,15 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "proc-macro-crate" +version = "3.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" +dependencies = [ + "toml_edit", +] + [[package]] name = "proc-macro2" version = "1.0.107" @@ -1469,7 +1561,7 @@ dependencies = [ "bitflags 2.13.2", "num-traits", "rand 0.9.5", - "rand_chacha", + "rand_chacha 0.9.0", "rand_xorshift", "regex-syntax", "rusty-fork", @@ -1510,6 +1602,8 @@ version = "0.8.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c" dependencies = [ + "libc", + "rand_chacha 0.3.1", "rand_core 0.6.4", ] @@ -1519,7 +1613,7 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" dependencies = [ - "rand_chacha", + "rand_chacha 0.9.0", "rand_core 0.9.5", ] @@ -1534,6 +1628,16 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "rand_chacha" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" +dependencies = [ + "ppv-lite86", + "rand_core 0.6.4", +] + [[package]] name = "rand_chacha" version = "0.9.0" @@ -1549,6 +1653,9 @@ name = "rand_core" version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", +] [[package]] name = "rand_core" @@ -1713,6 +1820,23 @@ version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" +[[package]] +name = "rust_decimal" +version = "1.43.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a" +dependencies = [ + "arrayvec", + "borsh", + "bytes", + "num-traits", + "rand 0.8.8", + "rand 0.9.5", + "serde", + "serde_json", + "wasm-bindgen", +] + [[package]] name = "rustc_version" version = "0.4.1" @@ -2161,6 +2285,36 @@ dependencies = [ "tungstenite", ] +[[package]] +name = "toml_datetime" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_edit" +version = "0.25.17+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c" +dependencies = [ + "indexmap", + "toml_datetime", + "toml_parser", + "winnow", +] + +[[package]] +name = "toml_parser" +version = "1.1.5+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c" +dependencies = [ + "winnow", +] + [[package]] name = "tungstenite" version = "0.30.0" @@ -2525,6 +2679,15 @@ dependencies = [ "windows-link", ] +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + [[package]] name = "wit-bindgen" version = "0.57.1" diff --git a/Cargo.toml b/Cargo.toml index 1a5467a7..a07fe043 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,5 @@ [workspace] -members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp"] +members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp", "crates/numtower"] [package] name = "improvise" diff --git a/crates/numtower/Cargo.toml b/crates/numtower/Cargo.toml new file mode 100644 index 00000000..54ca4e27 --- /dev/null +++ b/crates/numtower/Cargo.toml @@ -0,0 +1,25 @@ +[package] +name = "numtower" +version = "0.1.0" +edition = "2024" +license = "Apache-2.0" +description = "A three-level numeric tower: exact decimals, bounded exact fractions and inexact f64, with Scheme's contagion rules and exact, order-free sums." +keywords = ["decimal", "rational", "exact", "numeric-tower"] +categories = ["mathematics"] + +# ADR 005 clause 14: rust_decimal (std only), num-rational without its default +# features (for Ratio), and num-bigint always. No serde. +[dependencies] +rust_decimal = { version = "1.43", default-features = false, features = ["std"] } +num-rational = { version = "0.4.2", default-features = false } +num-bigint = { version = "0.4", default-features = false, features = ["std"] } +num-integer = { version = "0.1", default-features = false } +num-traits = { version = "0.2", default-features = false, features = ["std"] } + +[dev-dependencies] +num-rational = { version = "0.4.2", features = ["num-bigint-std"] } +proptest = "1" + +[[example]] +name = "bench" +test = false diff --git a/crates/numtower/examples/bench.rs b/crates/numtower/examples/bench.rs new file mode 100644 index 00000000..f328e4d9 --- /dev/null +++ b/crates/numtower/examples/bench.rs @@ -0,0 +1 @@ +fn main() {} diff --git a/crates/numtower/src/arith.rs b/crates/numtower/src/arith.rs new file mode 100644 index 00000000..31664681 --- /dev/null +++ b/crates/numtower/src/arith.rs @@ -0,0 +1,378 @@ +//! Arithmetic and contagion (ADR 005 clauses 4 and 5, question 13 (b)). +//! The reasoning for each rule is in `context/code/numtower.md`, "Arithmetic". + +use std::ops::{Add, Mul, Neg, Sub}; + +use num_bigint::{BigInt, BigUint}; +use num_integer::Integer; +use num_traits::{One, Signed, ToPrimitive, Zero}; +use rust_decimal::Decimal; + +use crate::fixed::I256; +use crate::num::{MANTISSA_LIMIT, MAX_SCALE, Num, NumRef, decimal, pow10, small_ratio}; +use crate::ratio::BigQ; + +/// The widest an exact intermediate may grow, in bits of numerator or denominator. +pub const CEILING_BITS: u64 = 8192; + +/// A value inside one evaluation: exact intermediates are unbounded up to the ceiling. +#[derive(Clone, Debug)] +pub struct Interim(Inner); + +#[derive(Clone, Debug)] +enum Inner { + Stored(Num), + Wide(Box), +} + +/// The divisor was exactly zero, exact or inexact (clause 5). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DivisionByZero; + +impl From for Interim { + fn from(n: Num) -> Interim { + Interim(Inner::Stored(n)) + } +} + +impl Interim { + fn wide(q: BigQ) -> Interim { + Interim(Inner::Wide(Box::new(q))) + } + + /// The stored bound of question 13 (b): a wide value is the nearest f64. + pub fn store(self) -> Num { + match self.0 { + Inner::Stored(n) => n, + Inner::Wide(q) => Num::inexact(q.to_f64()), + } + } + + pub fn is_exact(&self) -> bool { + match &self.0 { + Inner::Stored(n) => n.is_exact(), + Inner::Wide(_) => true, + } + } + + fn operand(&self) -> Operand<'_> { + match &self.0 { + Inner::Stored(n) => Operand::of(n), + Inner::Wide(q) => Operand::Big(q), + } + } + + pub fn checked_div(&self, o: &Interim) -> Result { + divide(self.operand(), o.operand()) + } + + pub fn pow(&self, o: &Interim) -> Result { + power(self.operand(), o.operand()) + } +} + +impl Num { + pub fn checked_div(&self, o: &Num) -> Result { + divide(Operand::of(self), Operand::of(o)).map(Interim::store) + } + + pub fn pow(&self, o: &Num) -> Result { + power(Operand::of(self), Operand::of(o)).map(Interim::store) + } +} + +macro_rules! binary_ops { + ($($trait:ident $method:ident $f:ident;)*) => {$( + impl $trait<&Interim> for &Interim { + type Output = Interim; + fn $method(self, o: &Interim) -> Interim { + $f(self.operand(), o.operand()) + } + } + impl $trait<&Num> for &Num { + type Output = Num; + fn $method(self, o: &Num) -> Num { + $f(Operand::of(self), Operand::of(o)).store() + } + } + )*}; +} + +binary_ops! { + Add add plus; + Sub sub minus; + Mul mul times; +} + +impl Neg for &Interim { + type Output = Interim; + fn neg(self) -> Interim { + negate(self.operand()) + } +} + +impl Neg for &Num { + type Output = Num; + fn neg(self) -> Num { + negate(Operand::of(self)).store() + } +} + +#[derive(Clone, Copy)] +enum Operand<'a> { + Dec(&'a Decimal), + Small(i128, i128), + Big(&'a BigQ), + Float(f64), +} + +impl<'a> Operand<'a> { + fn of(n: &'a Num) -> Operand<'a> { + match n.kind() { + NumRef::Exact(d) => Operand::Dec(d), + NumRef::Fraction(r) => Operand::Small(*r.numer(), *r.denom()), + NumRef::Inexact(x) => Operand::Float(x), + } + } + + fn parts(self) -> Option<(i128, i128)> { + match self { + Operand::Dec(d) => Some((d.mantissa(), pow10(d.scale()))), + Operand::Small(n, d) => Some((n, d)), + Operand::Big(_) | Operand::Float(_) => None, + } + } + + fn big(self) -> Option { + match self { + Operand::Big(q) => Some(q.clone()), + Operand::Float(_) => None, + exact => exact.parts().map(|(n, d)| BigQ::from_i128(n, d)), + } + } + + fn float(self) -> f64 { + match self { + Operand::Dec(d) => crate::num::decimal_to_f64(d), + Operand::Small(n, d) => crate::ratio::ratio_to_f64(&BigInt::from(n), &BigInt::from(d)), + Operand::Big(q) => q.to_f64(), + Operand::Float(x) => x, + } + } + + fn is_float(self) -> bool { + matches!(self, Operand::Float(_)) + } + + fn is_zero(self) -> bool { + match self { + Operand::Dec(d) => d.is_zero(), + Operand::Small(..) | Operand::Big(_) => false, + Operand::Float(x) => x == 0.0, + } + } +} + +fn inexact(x: f64) -> Interim { + Interim::from(Num::inexact(x)) +} + +/// An exact result in the first arm it fits; wide while under the ceiling, else demoted. +fn settle_wide(q: BigQ) -> Interim { + match Num::fit(&q) { + Some(n) => Interim::from(n), + None if q.bits() > CEILING_BITS => inexact(q.to_f64()), + None => Interim::wide(q), + } +} + +fn exact_or_wide(a: Operand, b: Operand, small: impl Fn(i128, i128, i128, i128) -> Option<(i128, i128)>, big: impl Fn(&BigQ, &BigQ) -> BigQ) -> Interim { + let fast = a.parts().zip(b.parts()).and_then(|((n1, d1), (n2, d2))| small(n1, d1, n2, d2)); + fast.and_then(|(n, d)| small_ratio(n, d)) + .map(Interim::from) + .unwrap_or_else(|| match a.big().zip(b.big()) { + Some((x, y)) => settle_wide(big(&x, &y)), + None => unreachable!("both operands are exact here"), + }) +} + +fn plus(a: Operand, b: Operand) -> Interim { + match (a, b) { + (x, y) if x.is_float() || y.is_float() => inexact(x.float() + y.float()), + (Operand::Dec(x), Operand::Dec(y)) => decimal_sum(x, y), + _ => exact_or_wide( + a, + b, + |n1, d1, n2, d2| { + let n = n1.checked_mul(d2)?.checked_add(n2.checked_mul(d1)?)?; + Some((n, d1.checked_mul(d2)?)) + }, + BigQ::add, + ), + } +} + +fn minus(a: Operand, b: Operand) -> Interim { + plus(a, negate(b).operand()) +} + +fn times(a: Operand, b: Operand) -> Interim { + match (a, b) { + (x, y) if x.is_float() || y.is_float() => inexact(x.float() * y.float()), + (Operand::Dec(x), Operand::Dec(y)) => decimal_product(x, y), + _ => exact_or_wide( + a, + b, + |n1, d1, n2, d2| { + let (g1, g2) = (n1.gcd(&d2).max(1), n2.gcd(&d1).max(1)); + Some(((n1 / g1).checked_mul(n2 / g2)?, (d1 / g2).checked_mul(d2 / g1)?)) + }, + BigQ::mul, + ), + } +} + +fn negate(a: Operand) -> Interim { + match a { + Operand::Dec(d) => Interim::from(Num::exact(-*d)), + Operand::Float(x) => inexact(-x), + Operand::Small(n, d) => small_ratio(-n, d) + .map(Interim::from) + .unwrap_or_else(|| settle_wide(BigQ::from_i128(n, d).neg())), + Operand::Big(q) => settle_wide(q.neg()), + } +} + +fn divide(a: Operand, b: Operand) -> Result { + match (a, b) { + (_, y) if y.is_zero() => Err(DivisionByZero), + (x, y) if x.is_float() || y.is_float() => Ok(inexact(x.float() / y.float())), + _ => Ok(exact_or_wide( + a, + b, + |n1, d1, n2, d2| { + let (g1, g2) = (n1.gcd(&n2).max(1), d1.gcd(&d2).max(1)); + Some(((n1 / g1).checked_mul(d2 / g2)?, (d1 / g2).checked_mul(n2 / g1)?)) + }, + |x, y| x.div(y).unwrap_or_else(|| unreachable!("the divisor is not zero")), + )), + } +} + +/// Clause 4's hint: a returned scale below the nominal one may mean rounding. +fn decimal_sum(x: &Decimal, y: &Decimal) -> Interim { + let nominal = x.scale().max(y.scale()); + match x.checked_add(*y).filter(|r| r.scale() == nominal) { + Some(r) => Interim::from(Num::exact(r)), + None => { + let align = |d: &Decimal| { + I256::from_i128(d.mantissa()) + .checked_mul_u128(pow10(nominal - d.scale()) as u128) + .expect("an aligned mantissa is below 2^190") + }; + let sum = align(x).checked_add(&align(y)).expect("below 2^191"); + settle_decimal_integer(sum, nominal, nominal) + } + } +} + +fn decimal_product(x: &Decimal, y: &Decimal) -> Interim { + let nominal = x.scale() + y.scale(); + match x.checked_mul(*y).filter(|r| r.scale() == nominal) { + Some(r) => Interim::from(Num::exact(r)), + None => settle_decimal_integer(I256::mul_i128(x.mantissa(), y.mantissa()), nominal, nominal), + } +} + +/// The recompute of clause 4: strip trailing zeros, fit 96 bits at scale <= 28, else a fraction. +fn settle_decimal_integer(m: I256, scale: u32, nominal: u32) -> Interim { + let (mut m, mut s) = (m, scale); + while s > 0 { + let (q, r) = m.div_rem_u64(10); + if r != 0 { + break; + } + m = q; + s -= 1; + } + let fits = m.to_i128().filter(|v| v.unsigned_abs() < MANTISSA_LIMIT as u128); + match fits { + Some(v) if s <= MAX_SCALE => Interim::from(padded(v, s, nominal.min(MAX_SCALE))), + _ => settle_wide(BigQ::new(m.to_bigint(), BigInt::from(10).pow(s))), + } +} + +/// The stored scale of clause 4: pad toward `target` while the mantissa fits 96 bits. +pub(crate) fn padded(mantissa: i128, scale: u32, target: u32) -> Num { + let (mut m, mut s) = (mantissa, scale); + while s < target && (m * 10).unsigned_abs() < MANTISSA_LIMIT as u128 { + m *= 10; + s += 1; + } + decimal(m, s).expect("a padded mantissa still fits") +} + +/// The exponent of an exact integer-valued operand. +fn integer_value(b: Operand) -> Option { + match b { + Operand::Dec(d) if d.fract().is_zero() => Some(BigInt::from(d.mantissa() / pow10(d.scale()))), + Operand::Small(n, 1) => Some(BigInt::from(n)), + Operand::Big(q) if q.den.is_one() => Some(q.num.clone()), + _ => None, + } +} + +fn power(a: Operand, b: Operand) -> Result { + match (a.is_float(), integer_value(b)) { + (false, Some(n)) => exact_power(a, &n), + _ => Ok(inexact(a.float().powf(b.float()))), + } +} + +/// Clause 5's `POW`: the size is checked before anything is computed. +fn exact_power(a: Operand, n: &BigInt) -> Result { + let q = a.big().expect("an exact base"); + let base_scale = match a { + Operand::Dec(d) => d.scale(), + _ => 0, + }; + let k = n.magnitude(); + let target = k + .to_u32() + .and_then(|k| k.checked_mul(base_scale)) + .map_or(MAX_SCALE, |s| s.min(MAX_SCALE)); + let unit = q.den.is_one() && q.num.magnitude().is_one(); + match (q.num.is_zero(), n.is_negative(), unit) { + (_, _, _) if n.is_zero() => Ok(Interim::from(padded(1, 0, 0))), + (true, true, _) => Err(DivisionByZero), + (true, false, _) => Ok(Interim::from(padded(0, 0, target))), + (false, _, true) => { + let negative = q.num.is_negative() && k.is_odd(); + Ok(Interim::from(padded(if negative { -1 } else { 1 }, 0, target))) + } + (false, negative, false) => { + let lower = |x: &BigInt| BigUint::from(x.bits().saturating_sub(1)) * k; + let too_big = lower(&q.num).max(lower(&q.den)) > BigUint::from(CEILING_BITS); + match (too_big, k.to_u32()) { + (false, Some(k)) => Ok(exact_integer_power(&q, k, negative, target)), + _ => Ok(inexact(a.float().powf(n.to_f64().unwrap_or(f64::INFINITY)))), + } + } + } +} + +fn exact_integer_power(q: &BigQ, k: u32, negative: bool, target: u32) -> Interim { + let (num, den) = (q.num.pow(k), q.den.pow(k)); + let value = match negative { + true => BigQ::new(den, num), + false => BigQ { num, den }, + }; + let settled = settle_wide(value); + match (&settled.0, negative) { + (Inner::Stored(n), false) => match n.kind() { + NumRef::Exact(d) => Interim::from(padded(d.mantissa(), d.scale(), target)), + _ => settled, + }, + _ => settled, + } +} diff --git a/crates/numtower/src/cmp.rs b/crates/numtower/src/cmp.rs new file mode 100644 index 00000000..ffa2e40a --- /dev/null +++ b/crates/numtower/src/cmp.rs @@ -0,0 +1,92 @@ +//! Exact comparison at every pairing of arms (clause 6), and the `MIN`/`MAX` +//! tie-breaks (clause 8). Why: `context/code/numtower.md`, "Comparison". + +use std::cmp::Ordering; + +use crate::fixed::I256; +use crate::num::{Num, NumRef}; +use crate::ratio::BigQ; + +impl Num { + /// The order of the two values, exactly; `None` only when either is NaN. + pub fn cmp_exact(&self, o: &Num) -> Option { + match (self.exact_parts(), o.exact_parts()) { + (Some(a), Some(b)) => Some(cross(a, b)), + (Some(_), None) => mixed(self, o.to_f64()), + (None, Some(_)) => mixed(o, self.to_f64()).map(Ordering::reverse), + (None, None) => self.to_f64().partial_cmp(&o.to_f64()), + } + } + + /// The smaller, NaN skipped; ties keep the most information (clause 8, question 11). + pub fn min_of<'a>(&'a self, o: &'a Num) -> &'a Num { + pick(self, o, Ordering::Less) + } + + pub fn max_of<'a>(&'a self, o: &'a Num) -> &'a Num { + pick(self, o, Ordering::Greater) + } +} + +impl PartialEq for Num { + fn eq(&self, o: &Num) -> bool { + self.cmp_exact(o) == Some(Ordering::Equal) + } +} + +impl PartialOrd for Num { + fn partial_cmp(&self, o: &Num) -> Option { + self.cmp_exact(o) + } +} + +/// `a.0/a.1` against `b.0/b.1`, denominators positive, in 256 bits. +fn cross(a: (i128, i128), b: (i128, i128)) -> Ordering { + I256::mul_i128(a.0, b.1).cmp(&I256::mul_i128(b.0, a.1)) +} + +/// An exact value against an f64: the rounded value decides unless it ties. +fn mixed(exact: &Num, x: f64) -> Option { + match (x.is_nan(), x.is_infinite()) { + (true, _) => None, + (false, true) => Some(if x > 0.0 { Ordering::Less } else { Ordering::Greater }), + (false, false) => match exact.to_f64().partial_cmp(&x)? { + Ordering::Equal => { + let q = exact.exact_q().expect("an exact operand"); + Some(q.cmp_value(&BigQ::from_f64(x))) + } + decided => Some(decided), + }, + } +} + +fn pick<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num { + match (a.cmp_exact(b), is_nan(a), is_nan(b)) { + (_, true, _) => b, + (_, false, true) => a, + (Some(o), _, _) if o == want => a, + (Some(Ordering::Equal), _, _) => tie(a, b, want), + _ => b, + } +} + +fn is_nan(n: &Num) -> bool { + matches!(n.kind(), NumRef::Inexact(x) if x.is_nan()) +} + +/// Equal values: exact over inexact, the larger scale, and IEEE's signed zero. +fn tie<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num { + match (a.kind(), b.kind()) { + (NumRef::Inexact(_), NumRef::Exact(_) | NumRef::Fraction(_)) => b, + (NumRef::Exact(_) | NumRef::Fraction(_), NumRef::Inexact(_)) => a, + (NumRef::Exact(x), NumRef::Exact(y)) if y.scale() > x.scale() => b, + (NumRef::Inexact(x), NumRef::Inexact(y)) => { + let negative_wanted = want == Ordering::Less; + match y.is_sign_negative() == negative_wanted && x.is_sign_negative() != negative_wanted { + true => b, + false => a, + } + } + _ => a, + } +} diff --git a/crates/numtower/src/fixed.rs b/crates/numtower/src/fixed.rs new file mode 100644 index 00000000..89175010 --- /dev/null +++ b/crates/numtower/src/fixed.rs @@ -0,0 +1,232 @@ +//! Fixed-width two's-complement integers for the hot paths: no allocation. +//! Why they are hand-rolled: `context/code/numtower.md`, "The wide integers". + +use std::cmp::Ordering; + +use num_bigint::{BigInt, Sign}; + +/// A signed integer of `64 * N` bits, little-endian limbs, two's complement. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub(crate) struct Fixed { + limbs: [u64; N], +} + +/// The 256-bit integer clauses 4–6 use, and the 320-bit decimal partition of clause 8. +pub(crate) type I256 = Fixed<4>; +pub(crate) type I320 = Fixed<5>; + +impl Fixed { + pub(crate) const ZERO: Self = Fixed { limbs: [0; N] }; + + pub(crate) fn from_i128(x: i128) -> Self { + let fill = if x < 0 { u64::MAX } else { 0 }; + let mut limbs = [fill; N]; + limbs[0] = x as u64; + limbs[1] = (x >> 64) as u64; + Fixed { limbs } + } + + pub(crate) fn is_negative(&self) -> bool { + self.limbs[N - 1] >> 63 == 1 + } + + pub(crate) fn is_zero(&self) -> bool { + self.limbs.iter().all(|&l| l == 0) + } + + fn wrapping_neg(&self) -> Self { + let mut out = [0u64; N]; + let mut carry = 1u64; + for (o, &l) in out.iter_mut().zip(&self.limbs) { + let (v, c) = (!l).overflowing_add(carry); + *o = v; + carry = c as u64; + } + Fixed { limbs: out } + } + + fn wrapping_add(&self, other: &Self) -> Self { + let mut out = [0u64; N]; + let mut carry = false; + for i in 0..N { + let (a, c1) = self.limbs[i].overflowing_add(other.limbs[i]); + let (b, c2) = a.overflowing_add(carry as u64); + out[i] = b; + carry = c1 || c2; + } + Fixed { limbs: out } + } + + pub(crate) fn checked_add(&self, other: &Self) -> Option { + let sum = self.wrapping_add(other); + let same = self.is_negative() == other.is_negative(); + (!same || sum.is_negative() == self.is_negative()).then_some(sum) + } + + pub(crate) fn checked_neg(&self) -> Option { + let neg = self.wrapping_neg(); + (self.is_zero() || neg.is_negative() != self.is_negative()).then_some(neg) + } + + /// The magnitude as unsigned limbs, with the sign split out. + fn magnitude(&self) -> (bool, [u64; N]) { + match self.is_negative() { + true => (true, self.wrapping_neg().limbs), + false => (false, self.limbs), + } + } + + fn from_magnitude(negative: bool, limbs: [u64; N]) -> Option { + let unsigned = Fixed { limbs }; + match (negative, unsigned.is_negative()) { + (false, false) => Some(unsigned), + (true, _) => { + let neg = unsigned.wrapping_neg(); + (neg.is_negative() || neg.is_zero()).then_some(neg) + } + (false, true) => None, + } + } + + pub(crate) fn checked_mul_u128(&self, k: u128) -> Option { + let (negative, mag) = self.magnitude(); + let mut out = [0u64; N]; + for (j, kk) in [k as u64, (k >> 64) as u64].into_iter().enumerate() { + if kk == 0 { + continue; + } + let mut carry = 0u128; + for i in 0..N { + let idx = i + j; + let prod = mag[i] as u128 * kk as u128 + carry; + match idx < N { + true => { + let s = out[idx] as u128 + (prod as u64) as u128; + out[idx] = s as u64; + carry = (prod >> 64) + (s >> 64); + } + false if prod != 0 => return None, + false => carry = 0, + } + } + if carry != 0 { + return None; + } + } + Self::from_magnitude(negative, out) + } + + /// Divides the magnitude by `k`, returning the quotient (sign kept) and the remainder. + pub(crate) fn div_rem_u64(&self, k: u64) -> (Self, u64) { + let (negative, mag) = self.magnitude(); + let mut out = [0u64; N]; + let mut rem = 0u128; + for i in (0..N).rev() { + let cur = (rem << 64) | mag[i] as u128; + out[i] = (cur / k as u128) as u64; + rem = cur % k as u128; + } + let q = Self::from_magnitude(negative, out).unwrap_or(Self::ZERO); + (q, rem as u64) + } + + pub(crate) fn to_i128(self) -> Option { + let fill = if self.is_negative() { u64::MAX } else { 0 }; + let hi_ok = self.limbs[2..].iter().all(|&l| l == fill); + let v = (self.limbs[0] as u128 | ((self.limbs[1] as u128) << 64)) as i128; + (hi_ok && (v < 0) == self.is_negative()).then_some(v) + } + + pub(crate) fn to_bigint(self) -> BigInt { + let (negative, mag) = self.magnitude(); + let digits: Vec = mag + .iter() + .flat_map(|&l| [l as u32, (l >> 32) as u32]) + .collect(); + let sign = if negative { Sign::Minus } else { Sign::Plus }; + BigInt::from_slice(sign, &digits) + } + + /// The product of two `i128`s; needs `N >= 4`. + pub(crate) fn mul_i128(a: i128, b: i128) -> Self { + let negative = (a < 0) != (b < 0); + let (x, y) = (a.unsigned_abs(), b.unsigned_abs()); + let (x0, x1) = (x as u64 as u128, x >> 64); + let (y0, y1) = (y as u64 as u128, y >> 64); + let p00 = x0 * y0; + let p01 = x0 * y1; + let p10 = x1 * y0; + let p11 = x1 * y1; + let mid = (p00 >> 64) + (p01 as u64 as u128) + (p10 as u64 as u128); + let mut limbs = [0u64; N]; + limbs[0] = p00 as u64; + limbs[1] = mid as u64; + let hi = (mid >> 64) + (p01 >> 64) + (p10 >> 64) + p11; + limbs[2] = hi as u64; + limbs[3] = (hi >> 64) as u64; + let unsigned = Fixed { limbs }; + match negative { + true => unsigned.wrapping_neg(), + false => unsigned, + } + } +} + +impl Default for Fixed { + fn default() -> Self { + Self::ZERO + } +} + +impl Ord for Fixed { + fn cmp(&self, other: &Self) -> Ordering { + match (self.is_negative(), other.is_negative()) { + (true, false) => Ordering::Less, + (false, true) => Ordering::Greater, + _ => self.limbs.iter().rev().cmp(other.limbs.iter().rev()), + } + } +} + +impl PartialOrd for Fixed { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_256_bit_product_of_two_i128s_matches_bigint() { + let cases = [ + (i128::MAX, i128::MAX), + (i128::MAX, -i128::MAX), + (-1, 1), + (0, i128::MIN), + (123_456_789_012_345_678_901_234, -98_765_432_109_876_543_210), + ]; + for (a, b) in cases { + let want = BigInt::from(a) * BigInt::from(b); + assert_eq!(I256::mul_i128(a, b).to_bigint(), want, "{a} * {b}"); + } + } + + #[test] + fn checked_arithmetic_refuses_what_overflows_and_keeps_what_fits() { + let max = I256::from_i128(i128::MAX).checked_mul_u128(u128::MAX).unwrap(); + assert_eq!( + max.to_bigint(), + BigInt::from(i128::MAX) * BigInt::from(u128::MAX) + ); + assert!(max.checked_mul_u128(u128::MAX).is_none()); + let neg = max.checked_neg().unwrap(); + assert_eq!(neg.to_bigint(), -max.to_bigint()); + let (q, r) = neg.div_rem_u64(10); + assert_eq!(q.to_bigint() * 10 - BigInt::from(r), neg.to_bigint()); + assert_eq!(I256::from_i128(-5).to_i128(), Some(-5)); + assert_eq!(max.to_i128(), None); + assert!(I256::from_i128(-1) < I256::ZERO); + } +} diff --git a/crates/numtower/src/lang.rs b/crates/numtower/src/lang.rs new file mode 100644 index 00000000..7bb16c2b --- /dev/null +++ b/crates/numtower/src/lang.rs @@ -0,0 +1,181 @@ +//! Clause 3's one number language: the gate, the exact parse, the forced `n/d` arm, +//! and the canonical text. Why each rule: `context/code/numtower.md`, "The number language". + +use std::fmt; + +use num_bigint::BigInt; +use num_traits::{One, Zero}; + +use crate::arith::padded; +use crate::num::{MAX_SCALE, Num, NumRef}; +use crate::ratio::BigQ; + +/// Whether the type of the text is being guessed or was forced (question 12). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Typing { + Inferred, + Forced, +} + +/// The gate: the strings `f64::from_str` accepts and reads as a finite value. +pub fn is_number(text: &str, typing: Typing) -> bool { + gate(text) || (typing == Typing::Forced && fraction_lexeme(text).is_some()) +} + +/// A number if the text is one, read exactly; `None` means the text is text. +pub fn parse_number(text: &str, typing: Typing) -> Option { + match (gate(text), typing) { + (true, _) => Lexeme::read(text).map(|lx| lx.value(text)), + (false, Typing::Forced) => fraction_lexeme(text).map(|q| Num::settle(&q)), + (false, Typing::Inferred) => None, + } +} + +fn gate(text: &str) -> bool { + text.parse::().is_ok_and(f64::is_finite) +} + +/// `[+-]digits/digits` with a nonzero denominator, as an exact value. +fn fraction_lexeme(text: &str) -> Option { + let (n, d) = text.split_once('/')?; + let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit()); + let unsigned = n.strip_prefix(['+', '-']).unwrap_or(n); + let ok = digits(unsigned) && digits(d); + let num: BigInt = ok.then(|| n.trim_start_matches('+').parse().ok())??; + let den: BigInt = d.parse().ok()?; + (!den.is_zero()).then(|| BigQ::new(num, den)) +} + +/// A lexeme of the gate's language: sign, significant digits, and the power of ten. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct Lexeme { + pub(crate) negative: bool, + pub(crate) digits: String, + pub(crate) exp10: i64, +} + +/// Exponents beyond this cannot reach either exact arm, and saturate. +const EXP_SATURATE: i64 = 1 << 40; + +impl Lexeme { + /// `[+-]? (d+ (. d*)? | . d+) ([eE] [+-]? d+)?`, and nothing else. + pub(crate) fn read(text: &str) -> Option { + let (negative, rest) = match text.as_bytes().first()? { + b'-' => (true, &text[1..]), + b'+' => (false, &text[1..]), + _ => (false, text), + }; + let (mantissa, exponent) = match rest.find(['e', 'E']) { + Some(i) => (&rest[..i], Some(&rest[i + 1..])), + None => (rest, None), + }; + let (int, frac) = mantissa.split_once('.').unwrap_or((mantissa, "")); + let all_digits = |s: &str| s.bytes().all(|b| b.is_ascii_digit()); + let mantissa_ok = all_digits(int) && all_digits(frac) && !(int.is_empty() && frac.is_empty()); + let exp = match exponent { + Some(e) => read_exponent(e)?, + None => 0, + }; + let digits = format!("{int}{frac}"); + mantissa_ok.then(|| Lexeme { + negative, + digits, + exp10: exp.saturating_sub(frac.len() as i64), + }) + } + + /// Clause 3's order: a decimal at the lexeme's scale, then a fraction, then the nearest f64. + fn value(&self, text: &str) -> Num { + let significant = self.digits.trim_start_matches('0'); + let stripped = significant.trim_end_matches('0'); + let implied_scale = (-self.exp10).clamp(0, MAX_SCALE as i64) as u32; + match stripped.is_empty() { + true => padded(0, 0, implied_scale), + false => { + let shift = self.exp10.saturating_add((significant.len() - stripped.len()) as i64); + self.exact(stripped, shift, implied_scale) + .unwrap_or_else(|| Num::inexact(text.parse().unwrap_or(f64::NAN))) + } + } + } + + fn exact(&self, stripped: &str, shift: i64, implied_scale: u32) -> Option { + let leading = stripped.len() as i64 + shift; + let reachable = stripped.len() <= 180 && (-130..=40).contains(&leading) && shift >= -400; + let q = reachable.then(|| self.rational(stripped, shift))?; + Num::fit(&q).map(|n| match n.kind() { + NumRef::Exact(d) => padded(d.mantissa(), d.scale(), implied_scale), + _ => n, + }) + } + + pub(crate) fn rational(&self, stripped: &str, shift: i64) -> BigQ { + let m: BigInt = stripped.parse().unwrap_or_default(); + let m = if self.negative { -m } else { m }; + let ten = BigInt::from(10); + match shift >= 0 { + true => BigQ::new(m * ten.pow(shift as u32), BigInt::one()), + false => BigQ::new(m, ten.pow((-shift) as u32)), + } + } + + /// The exact value, whatever its size: for display rounding. + pub(crate) fn exact_value(&self) -> BigQ { + let stripped = self.digits.trim_start_matches('0'); + match stripped.is_empty() { + true => BigQ::new(BigInt::zero(), BigInt::one()), + false => self.rational(stripped, self.exp10), + } + } +} + +fn read_exponent(e: &str) -> Option { + let (negative, digits) = match e.as_bytes().first()? { + b'-' => (true, &e[1..]), + b'+' => (false, &e[1..]), + _ => (false, e), + }; + let ok = !digits.is_empty() && digits.bytes().all(|b| b.is_ascii_digit()); + let significant = digits.trim_start_matches('0'); + let magnitude = match significant.len() > 12 { + true => EXP_SATURATE, + false => significant.parse::().unwrap_or(0).min(EXP_SATURATE), + }; + ok.then_some(if negative { -magnitude } else { magnitude }) +} + +/// The shortest text that reads back as `x`; `None` for a non-finite value. +pub fn shortest_round_trip(x: f64) -> Option { + x.is_finite().then(|| format!("{x:?}")) +} + +const INEXACT_MARK: &str = "#i"; + +impl Num { + /// Reads every spelling `Display` writes: a decimal, `n/d`, `#i…`, `inf`, `-inf`, `nan`. + pub fn from_canonical(text: &str) -> Option { + match text { + "inf" => Some(Num::inexact(f64::INFINITY)), + "-inf" => Some(Num::inexact(f64::NEG_INFINITY)), + "nan" => Some(Num::inexact(f64::NAN)), + _ => match text.strip_prefix(INEXACT_MARK) { + Some(rest) if gate(rest) => rest.parse().ok().map(Num::inexact), + Some(_) => None, + None => parse_number(text, Typing::Forced).filter(Num::is_exact), + }, + } + } +} + +/// The canonical text: a decimal with its scale, `n/d`, `#i` and the shortest spelling, or a token. +impl fmt::Display for Num { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.kind() { + NumRef::Exact(d) => write!(f, "{d}"), + NumRef::Fraction(r) => write!(f, "{}/{}", r.numer(), r.denom()), + NumRef::Inexact(x) if x.is_nan() => f.write_str("nan"), + NumRef::Inexact(x) if x.is_infinite() => f.write_str(if x > 0.0 { "inf" } else { "-inf" }), + NumRef::Inexact(x) => write!(f, "{INEXACT_MARK}{x:?}"), + } + } +} diff --git a/crates/numtower/src/lib.rs b/crates/numtower/src/lib.rs new file mode 100644 index 00000000..458d3e9f --- /dev/null +++ b/crates/numtower/src/lib.rs @@ -0,0 +1,20 @@ +//! A three-level numeric tower: exact decimal, exact bounded fraction, inexact f64. +//! The decisions are ADR 005; the reasoning is `context/code/numtower.md`. + +mod arith; +mod cmp; +mod fixed; +mod lang; +mod num; +mod ratio; +mod round; +mod sum; + +pub use arith::{CEILING_BITS, DivisionByZero, Interim}; +pub use lang::{Typing, is_number, parse_number, shortest_round_trip}; +pub use num::{Num, NumRef}; +pub use round::Rounding; +pub use sum::{ExactSum, K, NumSum}; + +pub use num_rational::Ratio; +pub use rust_decimal::Decimal; diff --git a/crates/numtower/src/num.rs b/crates/numtower/src/num.rs new file mode 100644 index 00000000..50075814 --- /dev/null +++ b/crates/numtower/src/num.rs @@ -0,0 +1,202 @@ +//! The stored number: an exact decimal, an exact bounded fraction, or an f64. +//! The arms are private so a value is always normalized: `context/code/numtower.md`. + +use num_bigint::BigInt; +use num_integer::Integer; +use num_rational::Ratio; +use num_traits::{One, ToPrimitive}; +use rust_decimal::Decimal; + +use crate::ratio::{BigQ, ratio_to_f64}; + +/// The most significant digits a `Decimal` holds, and its mantissa bound. +pub(crate) const MAX_SCALE: u32 = 28; +pub(crate) const MANTISSA_LIMIT: i128 = 1 << 96; + +/// A number of the tower (ADR 005 clause 2), always in its one normal form. +#[derive(Clone, Debug)] +pub struct Num(Repr); + +#[derive(Clone, Debug)] +enum Repr { + Exact(Decimal), + Fraction(Box>), + Inexact(f64), +} + +/// A borrowed view of a `Num`'s arm, for exhaustive matching by callers. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum NumRef<'a> { + Exact(&'a Decimal), + Fraction(&'a Ratio), + Inexact(f64), +} + +impl Num { + /// An exact decimal. Zero loses its sign: the sign of zero is an f64 artefact. + pub fn exact(d: Decimal) -> Num { + let mut d = d; + if d.is_zero() { + d.set_sign_positive(true); + } + Num(Repr::Exact(d)) + } + + pub fn inexact(x: f64) -> Num { + Num(Repr::Inexact(x)) + } + + /// The exact value `num / den`, normalized; `None` when `den` is zero. + pub fn ratio(num: i128, den: i128) -> Option { + (den != 0).then(|| Num::settle(&BigQ::from_i128(num, den))) + } + + fn fraction(num: i128, den: i128) -> Num { + Num(Repr::Fraction(Box::new(Ratio::new_raw(num, den)))) + } + + pub fn kind(&self) -> NumRef<'_> { + match &self.0 { + Repr::Exact(d) => NumRef::Exact(d), + Repr::Fraction(r) => NumRef::Fraction(r), + Repr::Inexact(x) => NumRef::Inexact(*x), + } + } + + pub fn is_exact(&self) -> bool { + !matches!(self.0, Repr::Inexact(_)) + } + + /// Exactly zero, at either level; the one zero predicate of clause 6. + pub fn is_zero(&self) -> bool { + match &self.0 { + Repr::Exact(d) => d.is_zero(), + Repr::Fraction(_) => false, + Repr::Inexact(x) => *x == 0.0, + } + } + + /// The correctly rounded f64 (clause 4): a decimal through its text, a fraction exactly. + pub fn to_f64(&self) -> f64 { + match &self.0 { + Repr::Exact(d) => decimal_to_f64(d), + Repr::Fraction(r) => ratio_to_f64(&BigInt::from(*r.numer()), &BigInt::from(*r.denom())), + Repr::Inexact(x) => *x, + } + } + + /// The exact value as a rational; `None` for an inexact number. + pub(crate) fn exact_parts(&self) -> Option<(i128, i128)> { + match &self.0 { + Repr::Exact(d) => Some((d.mantissa(), pow10(d.scale()))), + Repr::Fraction(r) => Some((*r.numer(), *r.denom())), + Repr::Inexact(_) => None, + } + } + + pub(crate) fn exact_q(&self) -> Option { + self.exact_parts().map(|(n, d)| BigQ::from_i128(n, d)) + } + + /// The stored bound: a decimal if it fits, else an `i128` fraction, else the nearest f64. + pub(crate) fn settle(q: &BigQ) -> Num { + Num::fit(q).unwrap_or_else(|| Num::inexact(q.to_f64())) + } + + /// `q` in an exact arm, at its minimal scale, if it fits one. + pub(crate) fn fit(q: &BigQ) -> Option { + decimal_of(q).or_else(|| { + let num = to_i128(&q.num)?; + let den = to_i128(&q.den)?; + Some(Num::fraction(num, den)) + }) + } +} + +/// `num / den` normalized without allocating; `None` hands the case to the bigint path. +pub(crate) fn small_ratio(num: i128, den: i128) -> Option { + let representable = num != i128::MIN && den != i128::MIN && den != 0; + representable.then(|| { + let g = num.gcd(&den); + let (n, d) = match den < 0 { + true => (-num / g, -den / g), + false => (num / g, den / g), + }; + let p = d.trailing_zeros(); + let mut rest = d >> p; + let mut q5 = 0; + while rest % 5 == 0 { + rest /= 5; + q5 += 1; + } + let k = p.max(q5); + let as_decimal = (rest == 1 && k <= MAX_SCALE) + .then(|| 2i128.pow(k - p) * 5i128.pow(k - q5)) + .and_then(|pad| n.checked_mul(pad)) + .and_then(|m| decimal(m, k)); + as_decimal.unwrap_or_else(|| Num::fraction(n, d)) + }) +} + +fn to_i128(x: &BigInt) -> Option { + x.to_i128().filter(|v| *v != i128::MIN) +} + +pub(crate) fn pow10(k: u32) -> i128 { + 10i128.pow(k) +} + +pub(crate) fn decimal_to_f64(d: &Decimal) -> f64 { + d.to_string().parse().unwrap_or(f64::NAN) +} + +/// A decimal from a mantissa and scale, if the mantissa fits 96 bits. +pub(crate) fn decimal(mantissa: i128, scale: u32) -> Option { + let fits = mantissa.unsigned_abs() < MANTISSA_LIMIT as u128 && scale <= MAX_SCALE; + fits.then(|| Num::exact(Decimal::from_i128_with_scale(mantissa, scale))) +} + +/// The direct test of clause 5: a reduced `n / (2^p 5^q)` is a decimal iff it fits. +fn decimal_of(q: &BigQ) -> Option { + let p = q.den.trailing_zeros().unwrap_or(0); + let mut rest = &q.den >> p as usize; + let five = BigInt::from(5); + let mut q5 = 0u64; + while rest.is_multiple_of(&five) && q5 <= MAX_SCALE as u64 { + rest /= &five; + q5 += 1; + } + let k = p.max(q5); + if !rest.is_one() || k > MAX_SCALE as u64 { + return None; + } + let pad = (BigInt::one() << (k - p) as usize) * five.pow((k - q5) as u32); + let mantissa = (&q.num * pad).to_i128()?; + decimal(mantissa, k as u32) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_num_is_24_bytes() { + assert_eq!(std::mem::size_of::(), 24); + } + + #[test] + fn a_fraction_that_is_a_fitting_decimal_is_stored_as_the_decimal() { + let half = Num::ratio(1, 2).unwrap(); + assert_eq!(half.kind(), NumRef::Exact(&Decimal::new(5, 1))); + let third = Num::ratio(-10, 3).unwrap(); + assert_eq!(third.kind(), NumRef::Fraction(&Ratio::new_raw(-10, 3))); + let tiny = Num::ratio(1, 1 << 40).unwrap(); + assert!(matches!(tiny.kind(), NumRef::Fraction(_)), "{tiny:?}"); + let mut minus_zero = Decimal::new(0, 2); + minus_zero.set_sign_negative(true); + assert!(minus_zero.is_sign_negative()); + let zero = Num::exact(minus_zero); + assert!(!zero.to_f64().is_sign_negative()); + assert!(Num::ratio(1, 0).is_none()); + } +} diff --git a/crates/numtower/src/ratio.rs b/crates/numtower/src/ratio.rs new file mode 100644 index 00000000..0c1a2d5c --- /dev/null +++ b/crates/numtower/src/ratio.rs @@ -0,0 +1,172 @@ +//! Exact rationals in bigints: the widened intermediate of question 13 (b), and +//! the one correctly rounded conversion of a rational to f64 (clauses 4 and 5). + +use std::cmp::Ordering; + +use num_bigint::{BigInt, BigUint, Sign}; +use num_integer::Integer; +use num_traits::{One, Signed, ToPrimitive, Zero}; + +/// A reduced fraction with a positive denominator. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct BigQ { + pub(crate) num: BigInt, + pub(crate) den: BigInt, +} + +impl BigQ { + pub(crate) fn new(num: BigInt, den: BigInt) -> BigQ { + let (num, den) = match den.is_negative() { + true => (-num, -den), + false => (num, den), + }; + let g = num.gcd(&den); + match g.is_zero() || g.is_one() { + true => BigQ { num, den }, + false => BigQ { + num: num / &g, + den: den / &g, + }, + } + } + + pub(crate) fn from_i128(num: i128, den: i128) -> BigQ { + BigQ::new(BigInt::from(num), BigInt::from(den)) + } + + pub(crate) fn add(&self, o: &BigQ) -> BigQ { + BigQ::new(&self.num * &o.den + &o.num * &self.den, &self.den * &o.den) + } + + pub(crate) fn neg(&self) -> BigQ { + BigQ { + num: -&self.num, + den: self.den.clone(), + } + } + + pub(crate) fn mul(&self, o: &BigQ) -> BigQ { + BigQ::new(&self.num * &o.num, &self.den * &o.den) + } + + /// `None` when `o` is zero. + pub(crate) fn div(&self, o: &BigQ) -> Option { + (!o.num.is_zero()).then(|| BigQ::new(&self.num * &o.den, &self.den * &o.num)) + } + + pub(crate) fn cmp_value(&self, o: &BigQ) -> Ordering { + (&self.num * &o.den).cmp(&(&o.num * &self.den)) + } + + pub(crate) fn bits(&self) -> u64 { + self.num.bits().max(self.den.bits()) + } + + pub(crate) fn to_f64(&self) -> f64 { + ratio_to_f64(&self.num, &self.den) + } + + /// The exact value of a finite f64. + pub(crate) fn from_f64(x: f64) -> BigQ { + let bits = x.to_bits(); + let biased = ((bits >> 52) & 0x7ff) as i64; + let fraction = bits & ((1u64 << 52) - 1); + let (mant, exp) = match biased { + 0 => (fraction, -1074), + e => (fraction | (1 << 52), e - 1075), + }; + let signed = match x.is_sign_negative() { + true => -BigInt::from(mant), + false => BigInt::from(mant), + }; + match exp >= 0 { + true => BigQ::new(signed << exp as usize, BigInt::one()), + false => BigQ::new(signed, BigInt::one() << (-exp) as usize), + } + } +} + +/// `num / den` rounded to the nearest f64, ties to even, subnormals and overflow included. +pub(crate) fn ratio_to_f64(num: &BigInt, den: &BigInt) -> f64 { + let negative = (num.sign() == Sign::Minus) != (den.sign() == Sign::Minus); + let a = num.magnitude(); + let b = den.magnitude(); + let magnitude = match a.is_zero() { + true => 0.0, + false => magnitude_to_f64(a, b), + }; + match negative { + true => -magnitude, + false => magnitude, + } +} + +fn magnitude_to_f64(a: &BigUint, b: &BigUint) -> f64 { + let shift = 54 - (a.bits() as i64 - b.bits() as i64); + let (n, d) = match shift >= 0 { + true => (a << shift as usize, b.clone()), + false => (a.clone(), b << (-shift) as usize), + }; + let (q, r) = n.div_rem(&d); + let qbits = q.bits() as i64; + let drop = (qbits - 53).max(shift - 1074); + if drop > qbits + 1 { + return 0.0; + } + let kept = (&q >> drop as usize).to_u64().unwrap_or(0); + let round = q.bit(drop as u64 - 1); + let sticky = !r.is_zero() || q.trailing_zeros().is_some_and(|t| (t as i64) < drop - 1); + let up = round && (sticky || kept & 1 == 1); + ldexp(kept + up as u64, drop - shift) +} + +/// `m * 2^e` for `m <= 2^53`, exact unless it overflows to infinity. +fn ldexp(m: u64, e: i64) -> f64 { + let top = e + 63 - m.leading_zeros() as i64; + match (m == 0, top > 1023) { + (true, _) => 0.0, + (false, true) => f64::INFINITY, + (false, false) => { + let half = e / 2; + m as f64 * pow2(half) * pow2(e - half) + } + } +} + +fn pow2(e: i64) -> f64 { + match e { + e if e >= -1022 => f64::from_bits(((e + 1023) as u64) << 52), + e => f64::from_bits(1u64 << (e + 1074)), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rounding_to_f64_reaches_subnormals_ties_and_overflow() { + let one = BigInt::one(); + let min_sub = f64::from_bits(1); + assert_eq!(ratio_to_f64(&one, &(BigInt::one() << 1074)), min_sub); + assert_eq!(ratio_to_f64(&one, &(BigInt::one() << 1075)), 0.0); + let three = BigInt::from(3); + assert_eq!(ratio_to_f64(&three, &(BigInt::one() << 1076)), min_sub); + assert_eq!(ratio_to_f64(&one, &BigInt::from(3)), 1.0 / 3.0); + let tie = (BigInt::one() << 53) + 1; + assert_eq!(ratio_to_f64(&tie, &one), 9007199254740992.0); + let big = BigInt::one() << 1024; + assert_eq!(ratio_to_f64(&big, &one), f64::INFINITY); + assert_eq!(ratio_to_f64(&-big, &one), f64::NEG_INFINITY); + let max = BigQ::from_f64(f64::MAX); + assert_eq!(max.to_f64(), f64::MAX); + } + + #[test] + fn every_f64_converts_to_its_exact_value_and_back() { + for x in [0.1, -2.5e-310, 1e300, f64::MIN_POSITIVE, 123456.789, -7.0] { + assert_eq!(BigQ::from_f64(x).to_f64().to_bits(), x.to_bits(), "{x}"); + } + assert!(BigQ::from_f64(-0.0).num.is_zero()); + } +} diff --git a/crates/numtower/src/round.rs b/crates/numtower/src/round.rs new file mode 100644 index 00000000..ec6a060e --- /dev/null +++ b/crates/numtower/src/round.rs @@ -0,0 +1,85 @@ +//! Rounding any value to `places` for display, exactly (clause 12, "Format"). +//! An inexact value is rounded over its shortest round-trip spelling (improvise-felo.9). + +use num_bigint::BigInt; +use num_integer::Integer; +use num_traits::Signed; + +use crate::lang::{Lexeme, shortest_round_trip}; +use crate::num::Num; +use crate::ratio::BigQ; + +/// The rounding mode is the caller's policy, not the crate's. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Rounding { + HalfAwayFromZero, + HalfEven, +} + +impl Num { + /// Plain decimal text with exactly `places` fractional digits; `None` for a non-finite value. + pub fn round_to_places(&self, places: u8, mode: Rounding) -> Option { + let q = match self.exact_q() { + Some(q) => q, + None => Lexeme::read(&shortest_round_trip(self.to_f64())?)?.exact_value(), + }; + Some(render(round(&q, places, mode), places)) + } +} + +/// `q * 10^places`, rounded to an integer by `mode`. +fn round(q: &BigQ, places: u8, mode: Rounding) -> BigInt { + let scaled = q.num.abs() * BigInt::from(10).pow(places as u32); + let (whole, rest) = scaled.div_rem(&q.den); + let twice: BigInt = rest * 2; + let up = match twice.cmp(&q.den) { + std::cmp::Ordering::Greater => true, + std::cmp::Ordering::Equal => mode == Rounding::HalfAwayFromZero || whole.is_odd(), + std::cmp::Ordering::Less => false, + }; + let magnitude = whole + BigInt::from(up as u8); + match q.num.is_negative() { + true => -magnitude, + false => magnitude, + } +} + +fn render(units: BigInt, places: u8) -> String { + let digits = units.abs().to_string(); + let width = places as usize + 1; + let padded = format!("{digits:0>width$}"); + let (int, frac) = padded.split_at(padded.len() - places as usize); + let sign = if units.is_negative() { "-" } else { "" }; + match places { + 0 => format!("{sign}{int}"), + _ => format!("{sign}{int}.{frac}"), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::lang::{Typing, parse_number}; + + fn typed(s: &str) -> Num { + parse_number(s, Typing::Inferred).unwrap() + } + + #[test] + fn a_typed_value_rounds_as_it_was_typed() { + let at2 = |n: &Num| n.round_to_places(2, Rounding::HalfAwayFromZero).unwrap(); + for (text, want) in [("1.005", "1.01"), ("1.015", "1.02"), ("0.285", "0.29"), ("-2.675", "-2.68")] { + assert_eq!(at2(&typed(text)), want, "{text}"); + assert_eq!(at2(&Num::inexact(text.parse().unwrap())), want, "inexact {text}"); + } + assert_eq!(at2(&Num::ratio(10, 3).unwrap()), "3.33"); + assert_eq!(at2(&Num::ratio(2, 3).unwrap()), "0.67"); + assert_eq!(at2(&typed("-0.001")), "0.00"); + let even = typed("0.125").round_to_places(2, Rounding::HalfEven).unwrap(); + assert_eq!(even, "0.12"); + assert_eq!(typed("12.5").round_to_places(0, Rounding::HalfEven).unwrap(), "12"); + assert_eq!(Num::inexact(f64::NAN).round_to_places(2, Rounding::HalfEven), None); + let wide = Num::ratio(1, 3).unwrap().round_to_places(40, Rounding::HalfEven).unwrap(); + assert_eq!(wide, format!("0.{}", "3".repeat(40))); + } +} diff --git a/crates/numtower/src/sum.rs b/crates/numtower/src/sum.rs new file mode 100644 index 00000000..f9b27b11 --- /dev/null +++ b/crates/numtower/src/sum.rs @@ -0,0 +1,258 @@ +//! Order-free, retractable sums (clause 8): ADR 009's f64 `ExactSum`, and `NumSum`, +//! which adds the exact partitions. The design: `context/code/numtower.md`, "Sums". + +use std::collections::BTreeMap; + +use num_bigint::{BigInt, BigUint}; +use num_integer::Integer; +use num_traits::{One, Zero}; + +use crate::fixed::{I256, I320}; +use crate::num::{MAX_SCALE, Num, NumRef, pow10}; +use crate::ratio::{BigQ, ratio_to_f64}; + +/// The most non-decimal denominator groups a sum reads exactly (question 16: K = 8). +pub const K: usize = 8; + +const LIMB_BITS: u32 = 32; +/// A finite f64 is `mant * 2^(shift - 1074)`, `shift <= 2045`; weights add 60 bits. +const LIMBS: usize = 70; +const CHUNK: u64 = 1 << 30; + +/// An exact sum of weighted f64s, rounded once when read (ADR 009). +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ExactSum { + limbs: [i64; LIMBS], + pending: u64, + nan: i128, + pos_inf: i128, + neg_inf: i128, +} + +impl Default for ExactSum { + fn default() -> Self { + ExactSum { + limbs: [0; LIMBS], + pending: 0, + nan: 0, + pos_inf: 0, + neg_inf: 0, + } + } +} + +impl ExactSum { + pub fn new() -> ExactSum { + ExactSum::default() + } + + /// Adds `x` with a Z-set `weight`; the weight is split into base-2^30 digits. + pub fn add(&mut self, x: f64, weight: i64) { + match (x.is_nan(), x.is_infinite(), x > 0.0) { + (true, _, _) => self.nan += weight as i128, + (false, true, true) => self.pos_inf += weight as i128, + (false, true, false) => self.neg_inf += weight as i128, + (false, false, _) => { + let magnitude = weight.unsigned_abs(); + for j in 0..3 { + let digit = (magnitude >> (30 * j)) & (CHUNK - 1); + if digit != 0 { + let signed = if weight < 0 { -(digit as i64) } else { digit as i64 }; + self.add_digit(x, signed, 30 * j); + } + } + } + } + } + + fn add_digit(&mut self, x: f64, weight: i64, extra: u32) { + if self.pending + weight.unsigned_abs() > CHUNK { + propagate(&mut self.limbs); + self.pending = 0; + } + let bits = x.to_bits(); + let biased = ((bits >> 52) & 0x7ff) as u32; + let fraction = bits & ((1 << 52) - 1); + let (mant, shift) = match biased { + 0 => (fraction, 0), + e => (fraction | (1 << 52), e - 1), + }; + let sign = if x.is_sign_negative() { -weight } else { weight }; + let at = shift + extra; + let wide = u128::from(mant) << (at % LIMB_BITS); + let k = (at / LIMB_BITS) as usize; + for i in 0..3 { + let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64; + self.limbs[k + i] += sign * part; + } + self.pending += weight.unsigned_abs(); + } + + /// Adds another accumulator's contents. + pub fn absorb(&mut self, o: &ExactSum) { + let mut theirs = o.limbs; + propagate(&mut theirs); + propagate(&mut self.limbs); + for (mine, t) in self.limbs.iter_mut().zip(theirs) { + *mine += t; + } + propagate(&mut self.limbs); + self.pending = 0; + self.nan += o.nan; + self.pos_inf += o.pos_inf; + self.neg_inf += o.neg_inf; + } + + /// The exact sum rounded to the nearest f64, ties to even. + pub fn value(&self) -> f64 { + match (self.nan != 0, self.pos_inf != 0, self.neg_inf != 0) { + (true, _, _) | (false, true, true) => f64::NAN, + (false, true, false) => f64::INFINITY, + (false, false, true) => f64::NEG_INFINITY, + (false, false, false) => ratio_to_f64(&self.total(), &(BigInt::one() << 1074)), + } + } + + /// The finite part as an integer in units of 2^-1074. + fn total(&self) -> BigInt { + let mut limbs = self.limbs; + propagate(&mut limbs); + let digits: Vec = limbs[..LIMBS - 1].iter().map(|&l| l as u32).collect(); + let low = BigInt::from(BigUint::from_slice(&digits)); + (BigInt::from(limbs[LIMBS - 1]) << (LIMB_BITS as usize * (LIMBS - 1))) + low + } +} + +/// Carry each limb into the next, leaving every limb but the top in [0, 2^32). +fn propagate(limbs: &mut [i64; LIMBS]) { + for k in 0..LIMBS - 1 { + let carry = limbs[k] >> LIMB_BITS; + limbs[k] -= carry << LIMB_BITS; + limbs[k + 1] += carry; + } +} + +/// A `SUM` over any `Num`s: exact when every term is exact and at most K groups remain. +#[derive(Clone, Debug, Default)] +pub struct NumSum { + decimals: I320, + groups: BTreeMap, + group_floats: ExactSum, + inexact: ExactSum, + inexact_weight: i128, +} + +impl NumSum { + pub fn new() -> NumSum { + NumSum::default() + } + + pub fn insert(&mut self, x: &Num) { + self.add(x, 1); + } + + pub fn retract(&mut self, x: &Num) { + self.add(x, -1); + } + + /// Adds `x` with a Z-set `weight`. Exact and associative: the order cannot matter. + pub fn add(&mut self, x: &Num, weight: i64) { + match (x.kind(), x.exact_parts()) { + (NumRef::Inexact(f), _) => { + self.inexact.add(f, weight); + self.inexact_weight += weight as i128; + } + (_, Some((n, d))) => match at_scale_28(d) { + Some(unit) => self.add_scaled(n, unit, weight), + None => self.add_to_group(n, d, weight), + }, + (_, None) => unreachable!("an exact value has parts"), + } + } + + fn add_scaled(&mut self, n: i128, unit: u128, weight: i64) { + let term = I320::from_i128(n) + .checked_mul_u128(unit) + .and_then(|t| t.checked_mul_u128(weight.unsigned_abs() as u128)) + .and_then(|t| if weight < 0 { t.checked_neg() } else { Some(t) }) + .expect("a weighted term is below 2^284"); + self.decimals = self.decimals.checked_add(&term).expect("320 bits holds 2^36 terms"); + } + + fn add_to_group(&mut self, n: i128, d: i128, weight: i64) { + let old = self.groups.get(&d).copied().unwrap_or(I256::ZERO); + let new = old + .checked_add(&I256::mul_i128(n, weight as i128)) + .expect("256 bits holds 2^64 terms"); + let value = |num: I256| ratio_to_f64(&num.to_bigint(), &BigInt::from(d)); + if !old.is_zero() { + self.group_floats.add(value(old), -1); + } + match new.is_zero() { + true => { + self.groups.remove(&d); + } + false => { + self.group_floats.add(value(new), 1); + self.groups.insert(d, new); + } + } + } + + /// The current non-decimal denominator groups. + pub fn groups(&self) -> usize { + self.groups.len() + } + + /// One read: exact over at most K groups, else the f64 accumulators (clause 8). + pub fn read(&self) -> Num { + match (self.inexact_weight != 0, self.groups.len() > K) { + (false, false) => Num::settle(&self.exact_total()), + (_, true) => { + let mut acc = self.inexact.clone(); + acc.absorb(&self.group_floats); + acc.add(self.decimal_total().to_f64(), 1); + Num::inexact(acc.value()) + } + (true, false) => { + let mut acc = self.inexact.clone(); + acc.add(self.exact_total().to_f64(), 1); + Num::inexact(acc.value()) + } + } + } + + fn decimal_total(&self) -> BigQ { + BigQ::new(self.decimals.to_bigint(), BigInt::from(pow10(MAX_SCALE))) + } + + fn exact_total(&self) -> BigQ { + let decimal = (!self.decimals.is_zero()) + .then(|| (self.decimals.to_bigint(), BigInt::from(pow10(MAX_SCALE)))); + let terms: Vec<(BigInt, BigInt)> = decimal + .into_iter() + .chain(self.groups.iter().map(|(d, n)| (n.to_bigint(), BigInt::from(*d)))) + .collect(); + let (num, den) = combine_over_lcm(&terms); + BigQ::new(num, den) + } +} + +/// `Σ n/d` over the lcm of the denominators, not yet reduced: the read reduces once. +pub(crate) fn combine_over_lcm(terms: &[(BigInt, BigInt)]) -> (BigInt, BigInt) { + let lcm = terms.iter().fold(BigInt::one(), |l, (_, d)| l.lcm(d)); + let num = terms.iter().fold(BigInt::zero(), |s, (n, d)| s + n * (&lcm / d)); + (num, lcm) +} + +/// `10^28 / d` when `d = 2^p 5^q` with `max(p, q) <= 28`: the term joins the decimal partition. +fn at_scale_28(d: i128) -> Option { + let p = d.trailing_zeros(); + let mut rest = d >> p; + let mut q = 0; + while rest % 5 == 0 { + rest /= 5; + q += 1; + } + (rest == 1 && p.max(q) <= MAX_SCALE).then(|| (pow10(MAX_SCALE) / d) as u128) +} From 0dc881f934de451df714be6e13b8cea8fef6545b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:58:58 -0700 Subject: [PATCH 052/106] docs(adr): ADR 009 records the user's connection-operator consideration (C13) and its bearing on delivery (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 87 +++++++++++++++++++++-- 1 file changed, 82 insertions(+), 5 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index f1d2c49d..89f939da 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -2,11 +2,11 @@ **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and -12** (see "What the user decided"). Question 11 and questions 13 to 27 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** +12** (see "What the user decided"). Question 11 and questions 13 to 28 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** (PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 27. *Superseded:* until 2026-10-09 the +ADR", in clauses 12 to 14, and in questions 12 to 28. *Superseded:* until 2026-10-09 the user had answered none of questions 1 to 11. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · @@ -83,7 +83,7 @@ user routed part of it here. This section quotes what the user said that bears o and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed +- **Thirteen chat statements**, C1 to C13 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the PR, but spread over several groups of its own. ADR 008's list also has six more (its @@ -126,6 +126,11 @@ None of this answers this ADR's own questions 1 to 11. - **C12.** "The simplest might be some kind of binary search for a new top left corner: increasing jumps of the top left until you get a hit and then move down/right until You only have one new row column." Clause 12, question 22(d). +- **C13.** "I think we can minimize the amount of UI state on the backend by having an + operator of some kind responsible for both ends of the connection between the frontend + and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). + A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, + 14 and 28. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1960,6 +1965,55 @@ the user's "recent projections + clients" (C2) becomes clause 7's shape cache, w keyed by requests and holds no client, plus the topic subscriptions, which hold one shape key per client. +**A connection operator (C13, the user's consideration; everything below is the +author's reading).** The user: "I think we can minimize the amount of UI state on the +backend by having an operator of some kind responsible for both ends of the connection +between the frontend and backend and handling keep alive / session resumption / etc". +- **What "operator" means is ambiguous.** It is read here **not** as a DBSP operator, + but as a session or connection manager that sits between a frontend and the host and + owns both ends of the link: a gateway, or an actor. Question 28. +- **What it would hold, so that the host does not:** the connection; the keep-alive, with + the ping both ways; session resumption, meaning a reconnect with a token and then the + replay-or-refetch rules (T3's host instance and publication number); the topic + subscriptions and their pins ((b′)); and each client's last applied stamp. The host + would then answer `project` and `apply`, publish its unaddressed notices, and know + nothing about clients. The operator is the one place that does. +- **Where it could live, in today's code:** + - in process, for the TUI; + - beside the host in the browser worker: `improvise-worker-server`'s `Worker` already + owns one `Host` and one port (`worker.rs:14-15` at `f0fb6880`); + - as the socket server's front: `improvise-ws-server`'s `Registry` and transport are + the nearest thing today (`registry.rs:88-89`). + + Whether it is a new crate, a new process, or a role that existing code takes on is + question 28. +- **What it changes for question 13.** If the operator subscribes to the host's + unaddressed notices **on behalf of its clients** and fans them out, the host sends + O(operators) notices per batch, not O(clients). The per-client filtering of (b′), and + even the window restriction of (c), move into the operator. The wire between host and + operator then carries (b)'s broadcast, and the wire between operator and client carries + (b′)'s or (c)'s traffic. That changes the trade the review of `5a8dc486` caught, of host + state against wire traffic: with an operator, (b′) and (c) cost the **operator** state, + not the host. The costs are a new component, a second hop, and a place where a lost + notice can happen. When the operator is in-process or in the worker, the second hop is + a function call. None of this is measured. +- **What it changes for question 14.** Pins and subscriptions belong to the operator. It + re-requests the shapes its clients read, and that keeps them alive in the host's cache, + so the host's eviction rule (no recent request) holds unchanged. A host-side pin is no + longer needed. +- **The Q33 question.** If topic subscriptions live in the operator, the host holds none, + and a subscription is no longer "backend" state in the sense of the user's Q33 answer. + That holds only if the operator does not count as backend, which is part of question + 28. +- **Network transparency and "as stateless as possible".** T1 to T6 then hold on each + hop, and the operator must not change a result, only route it. The host becomes as + stateless toward clients as plain (b), and the per-client state is concentrated in one + component that can be restarted, because a client then refetches (T3's general rule). + In ADR 008's view/host split, the operator sits on the **host side of the wire, outside + the model**. It knows clients and holds no UI state beyond delivery state. +- **What it does not answer.** Question 13's choice, question 14's eviction policy, and + whether the operator is a new component at all. + **What becomes of today's pieces under (b) and (b′)** (the author's): - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A @@ -2310,7 +2364,7 @@ readings, flagged): Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered -questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10-05, +questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10-05, 2026-10-06 and 2026-10-09 revisions, and none is renumbered. 1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -2436,7 +2490,9 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 O(N) headers per batch and per-connection state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant - information")? (b′) depends on the answer; + information")? (b′) depends on the answer. The user's consideration C13, a + connection operator that owns both ends, would move topic subscriptions out of the + host, so they stop being host state (clause 12, question 28); - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2466,6 +2522,9 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 name, and a client that toggles hides through many sets would pay to derive each new entry. Only an unsaved change of axes, filters or `measure_agg` builds a new key, as a job. + *Under C13* (the user's consideration, not decided), pins move to a connection + operator, which keeps shapes alive by re-requesting them. The host's rule is then only + "evict once no recent request names it" (clause 12). 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2660,3 +2719,21 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 *Recommendation (the author's):* a column whose drawn cells include any text keeps a maintained width for its text cells. The view layer takes the larger of that and its held numeric width. ADR 008 should record the split. Not decided. + +28. *[New.]* **The connection "operator" of C13** (clause 12). The user: "an operator of + some kind responsible for both ends of the connection between the frontend and + backend and handling keep alive / session resumption / etc". Four parts: + - (a) Is "operator" a session or connection manager (a gateway or actor that owns + both ends of the link), as the author reads it, and not a DBSP operator? + - (b) Is it a new crate or process, or a role that existing code takes on: + `improvise-worker-server`'s `Worker` in the browser, `improvise-ws-server`'s + `Registry` and transport for sockets, and in-process code in the TUI? + - (c) Does it count as "backend" in the sense of the user's Q33 answer, so that + topic subscriptions held there are or are not backend state? + - (d) Does it subscribe to the host's notices on behalf of its clients and fan them + out, so that the host sends O(operators) per batch? + + *Recommendation (the author's):* (a) yes. (b) a role of the existing transport + crates, not a new process, so the TUI and the worker get it with no extra hop. (c) + not backend: it is delivery and holds no UI state. (d) yes. With that, question 13's + choice moves to the operator–client hop, and the host sends plain (b). Not decided. From 74eb5aaaadc4d062681362bbc9abf4aa5256d6e6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:59:29 -0700 Subject: [PATCH 053/106] docs(adr): ADR 009 reads C14: the operator is a sink/source adapter at the wire boundary (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 79 +++++++++++++++++------ 1 file changed, 61 insertions(+), 18 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 89f939da..8d2e9bd5 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -83,7 +83,7 @@ user routed part of it here. This section quotes what the user said that bears o and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **Thirteen chat statements**, C1 to C13 below, were given to the orchestrator and relayed +- **Fourteen chat statements**, C1 to C14 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the PR, but spread over several groups of its own. ADR 008's list also has six more (its @@ -131,6 +131,9 @@ None of this answers this ADR's own questions 1 to 11. and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, 14 and 28. +- **C14.** "And it asks as a sink/source for the DBSP graph part on the backend and a + source/sink for the client side" (relayed 2026-10-09, after C13). Quoted exactly as + sent. *The ADR reads "asks" as "acts"*, flagged. Clause 12, question 28. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1969,9 +1972,38 @@ key per client. author's reading).** The user: "I think we can minimize the amount of UI state on the backend by having an operator of some kind responsible for both ends of the connection between the frontend and backend and handling keep alive / session resumption / etc". -- **What "operator" means is ambiguous.** It is read here **not** as a DBSP operator, - but as a session or connection manager that sits between a frontend and the host and - owns both ends of the link: a gateway, or an actor. Question 28. +- **What "operator" means, after C14.** C14 resolves most of C13's ambiguity: "And it + asks as a sink/source for the DBSP graph part on the backend and a source/sink for the + client side", with "asks" read as "acts" (flagged). *The author's reading:* the + operator is **shaped like the DBSP graph**. Toward the host it is a **sink** for the + graph's output streams (projection results and change notices) and a **source** of the + requests and model commands it feeds in. Toward the client it is a **source** of those + results and notices, and a **sink** for the client's requests and events. So the + connection between frontend and backend is itself a pair of sink/source endpoints, and + the operator owns both. What stays ambiguous is whether "both ends" means both halves of + that pair inside one component, or a component in each of the two processes. Question + 28. +- **Why this fits C1** ("network transparent"). The circuit's streams are the same + whether consumed in process or across a wire, and the operator is the **adapter at the + wire boundary**. It owns: + - the serialization of Z-set deltas, under T1 (`CellKey` is not `Serialize`, so the wire + form names tuples and ADR 002 surrogates) and T2 (the wire's NaN identity); + - the stamps and the gap rule (T3); + - snapshot-then-delta (T4); + - coalescing and backpressure (T5); + - session resumption. + + In process, both of its halves collapse to function calls, which is T6. +- **Where it sits among the crates** (the author's reading). The host-facing half + consumes the circuit's output types, so it belongs with the seam crate + `improvise-incremental` (question 2's answer) or right above it. The wire types it + serializes belong in `improvise-protocol`, beside the one envelope pair (gotcha 42). + `zset` stays dependent on `std` alone (question 1's answer), with no serde, because the + conversion happens in the operator. The **client-facing half** would be written again + by every front end unless it is shared, so by design-principles §5 it belongs in the + shared layer, as `improvise-protocol`'s client fold (`ClientState`, `client.rs`) + already is. Who owns keep-alive and resumption state, and whether the TUI, the browser + and a CLI share one client half, are question 28. - **What it would hold, so that the host does not:** the connection; the keep-alive, with the ping both ways; session resumption, meaning a reconnect with a token and then the replay-or-refetch rules (T3's host instance and publication number); the topic @@ -2720,20 +2752,31 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10 maintained width for its text cells. The view layer takes the larger of that and its held numeric width. ADR 008 should record the split. Not decided. -28. *[New.]* **The connection "operator" of C13** (clause 12). The user: "an operator of - some kind responsible for both ends of the connection between the frontend and - backend and handling keep alive / session resumption / etc". Four parts: - - (a) Is "operator" a session or connection manager (a gateway or actor that owns - both ends of the link), as the author reads it, and not a DBSP operator? - - (b) Is it a new crate or process, or a role that existing code takes on: - `improvise-worker-server`'s `Worker` in the browser, `improvise-ws-server`'s - `Registry` and transport for sockets, and in-process code in the TUI? - - (c) Does it count as "backend" in the sense of the user's Q33 answer, so that - topic subscriptions held there are or are not backend state? +28. *[New.]* **The connection "operator" of C13 and C14** (clause 12). The user: "an + operator of some kind responsible for both ends of the connection between the + frontend and backend and handling keep alive / session resumption / etc", and "And it + asks as a sink/source for the DBSP graph part on the backend and a source/sink for the + client side" ("asks" read as "acts", flagged). Part (a) is read as answered by C14. + - (a) *Read as answered by C14:* it is a DBSP-graph-shaped adapter, a sink and source + toward the graph and a source and sink toward the client, not a DBSP operator inside + the circuit. Still open: does "both ends" mean both halves of the sink/source pair + inside one component, or a component on each side of the wire? + - (b) Is it a new crate or process, or a role existing code takes on? Its host-facing + half could live with `improvise-incremental`, its wire types in + `improvise-protocol`, and its client-facing half beside `improvise-protocol`'s + client fold, shared by the TUI, the browser and any CLI (design-principles §5). The + browser's `Worker` (`worker.rs:14-15` at `f0fb6880`) and the socket server's + `Registry` and transport (`registry.rs:88-89`) are the nearest code today. + - (c) Who owns keep-alive and resumption state: the operator's host-facing half, its + client-facing half, or both? And does it count as "backend" in the sense of the + user's Q33 answer? - (d) Does it subscribe to the host's notices on behalf of its clients and fan them out, so that the host sends O(operators) per batch? - *Recommendation (the author's):* (a) yes. (b) a role of the existing transport - crates, not a new process, so the TUI and the worker get it with no extra hop. (c) - not backend: it is delivery and holds no UI state. (d) yes. With that, question 13's - choice moves to the operator–client hop, and the host sends plain (b). Not decided. + *Recommendation (the author's):* (a) one component per connection that owns both + halves, with the client half as a shared library each front end links. (b) the + host-facing half in or above `improvise-incremental`, wire types in + `improvise-protocol`, and the client half shared beside `ClientState`. (c) resumption + state in the host-facing half, keyed by a token the client holds, and treated as + delivery state, not backend UI state. (d) yes. Then question 13's choice moves to the + operator–client hop, and the host sends plain (b). Not decided. From df6b80842fe2d89fcfd77de41e8e75a035127821 Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 22:02:18 -0700 Subject: [PATCH 054/106] test(numtower): ADR 005 step 1's property tests, each beside the rival it catches (improvise-59nw.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- crates/numtower/src/arith.rs | 45 +- crates/numtower/src/cmp.rs | 9 +- crates/numtower/src/fixed.rs | 16 +- crates/numtower/src/lang.rs | 11 +- crates/numtower/src/round.rs | 34 +- crates/numtower/src/sum.rs | 122 +++- crates/numtower/tests/oracle/mod.rs | 187 +++++ crates/numtower/tests/properties.rs | 1019 +++++++++++++++++++++++++++ 8 files changed, 1411 insertions(+), 32 deletions(-) create mode 100644 crates/numtower/tests/oracle/mod.rs create mode 100644 crates/numtower/tests/properties.rs diff --git a/crates/numtower/src/arith.rs b/crates/numtower/src/arith.rs index 31664681..18cf3174 100644 --- a/crates/numtower/src/arith.rs +++ b/crates/numtower/src/arith.rs @@ -186,8 +186,16 @@ fn settle_wide(q: BigQ) -> Interim { } } -fn exact_or_wide(a: Operand, b: Operand, small: impl Fn(i128, i128, i128, i128) -> Option<(i128, i128)>, big: impl Fn(&BigQ, &BigQ) -> BigQ) -> Interim { - let fast = a.parts().zip(b.parts()).and_then(|((n1, d1), (n2, d2))| small(n1, d1, n2, d2)); +fn exact_or_wide( + a: Operand, + b: Operand, + small: impl Fn(i128, i128, i128, i128) -> Option<(i128, i128)>, + big: impl Fn(&BigQ, &BigQ) -> BigQ, +) -> Interim { + let fast = a + .parts() + .zip(b.parts()) + .and_then(|((n1, d1), (n2, d2))| small(n1, d1, n2, d2)); fast.and_then(|(n, d)| small_ratio(n, d)) .map(Interim::from) .unwrap_or_else(|| match a.big().zip(b.big()) { @@ -225,7 +233,10 @@ fn times(a: Operand, b: Operand) -> Interim { b, |n1, d1, n2, d2| { let (g1, g2) = (n1.gcd(&d2).max(1), n2.gcd(&d1).max(1)); - Some(((n1 / g1).checked_mul(n2 / g2)?, (d1 / g2).checked_mul(d2 / g1)?)) + Some(( + (n1 / g1).checked_mul(n2 / g2)?, + (d1 / g2).checked_mul(d2 / g1)?, + )) }, BigQ::mul, ), @@ -252,9 +263,15 @@ fn divide(a: Operand, b: Operand) -> Result { b, |n1, d1, n2, d2| { let (g1, g2) = (n1.gcd(&n2).max(1), d1.gcd(&d2).max(1)); - Some(((n1 / g1).checked_mul(d2 / g2)?, (d1 / g2).checked_mul(n2 / g1)?)) + Some(( + (n1 / g1).checked_mul(d2 / g2)?, + (d1 / g2).checked_mul(n2 / g1)?, + )) + }, + |x, y| { + x.div(y) + .unwrap_or_else(|| unreachable!("the divisor is not zero")) }, - |x, y| x.div(y).unwrap_or_else(|| unreachable!("the divisor is not zero")), )), } } @@ -280,7 +297,9 @@ fn decimal_product(x: &Decimal, y: &Decimal) -> Interim { let nominal = x.scale() + y.scale(); match x.checked_mul(*y).filter(|r| r.scale() == nominal) { Some(r) => Interim::from(Num::exact(r)), - None => settle_decimal_integer(I256::mul_i128(x.mantissa(), y.mantissa()), nominal, nominal), + None => { + settle_decimal_integer(I256::mul_i128(x.mantissa(), y.mantissa()), nominal, nominal) + } } } @@ -295,7 +314,9 @@ fn settle_decimal_integer(m: I256, scale: u32, nominal: u32) -> Interim { m = q; s -= 1; } - let fits = m.to_i128().filter(|v| v.unsigned_abs() < MANTISSA_LIMIT as u128); + let fits = m + .to_i128() + .filter(|v| v.unsigned_abs() < MANTISSA_LIMIT as u128); match fits { Some(v) if s <= MAX_SCALE => Interim::from(padded(v, s, nominal.min(MAX_SCALE))), _ => settle_wide(BigQ::new(m.to_bigint(), BigInt::from(10).pow(s))), @@ -315,7 +336,9 @@ pub(crate) fn padded(mantissa: i128, scale: u32, target: u32) -> Num { /// The exponent of an exact integer-valued operand. fn integer_value(b: Operand) -> Option { match b { - Operand::Dec(d) if d.fract().is_zero() => Some(BigInt::from(d.mantissa() / pow10(d.scale()))), + Operand::Dec(d) if d.fract().is_zero() => { + Some(BigInt::from(d.mantissa() / pow10(d.scale()))) + } Operand::Small(n, 1) => Some(BigInt::from(n)), Operand::Big(q) if q.den.is_one() => Some(q.num.clone()), _ => None, @@ -348,7 +371,11 @@ fn exact_power(a: Operand, n: &BigInt) -> Result { (true, false, _) => Ok(Interim::from(padded(0, 0, target))), (false, _, true) => { let negative = q.num.is_negative() && k.is_odd(); - Ok(Interim::from(padded(if negative { -1 } else { 1 }, 0, target))) + Ok(Interim::from(padded( + if negative { -1 } else { 1 }, + 0, + target, + ))) } (false, negative, false) => { let lower = |x: &BigInt| BigUint::from(x.bits().saturating_sub(1)) * k; diff --git a/crates/numtower/src/cmp.rs b/crates/numtower/src/cmp.rs index ffa2e40a..2466b017 100644 --- a/crates/numtower/src/cmp.rs +++ b/crates/numtower/src/cmp.rs @@ -49,7 +49,11 @@ fn cross(a: (i128, i128), b: (i128, i128)) -> Ordering { fn mixed(exact: &Num, x: f64) -> Option { match (x.is_nan(), x.is_infinite()) { (true, _) => None, - (false, true) => Some(if x > 0.0 { Ordering::Less } else { Ordering::Greater }), + (false, true) => Some(if x > 0.0 { + Ordering::Less + } else { + Ordering::Greater + }), (false, false) => match exact.to_f64().partial_cmp(&x)? { Ordering::Equal => { let q = exact.exact_q().expect("an exact operand"); @@ -82,7 +86,8 @@ fn tie<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num { (NumRef::Exact(x), NumRef::Exact(y)) if y.scale() > x.scale() => b, (NumRef::Inexact(x), NumRef::Inexact(y)) => { let negative_wanted = want == Ordering::Less; - match y.is_sign_negative() == negative_wanted && x.is_sign_negative() != negative_wanted { + match y.is_sign_negative() == negative_wanted && x.is_sign_negative() != negative_wanted + { true => b, false => a, } diff --git a/crates/numtower/src/fixed.rs b/crates/numtower/src/fixed.rs index 89175010..ba551181 100644 --- a/crates/numtower/src/fixed.rs +++ b/crates/numtower/src/fixed.rs @@ -48,10 +48,10 @@ impl Fixed { fn wrapping_add(&self, other: &Self) -> Self { let mut out = [0u64; N]; let mut carry = false; - for i in 0..N { - let (a, c1) = self.limbs[i].overflowing_add(other.limbs[i]); - let (b, c2) = a.overflowing_add(carry as u64); - out[i] = b; + for (o, (a, b)) in out.iter_mut().zip(self.limbs.iter().zip(&other.limbs)) { + let (s, c1) = a.overflowing_add(*b); + let (s, c2) = s.overflowing_add(carry as u64); + *o = s; carry = c1 || c2; } Fixed { limbs: out } @@ -96,9 +96,9 @@ impl Fixed { continue; } let mut carry = 0u128; - for i in 0..N { + for (i, &limb) in mag.iter().enumerate() { let idx = i + j; - let prod = mag[i] as u128 * kk as u128 + carry; + let prod = limb as u128 * kk as u128 + carry; match idx < N { true => { let s = out[idx] as u128 + (prod as u64) as u128; @@ -215,7 +215,9 @@ mod tests { #[test] fn checked_arithmetic_refuses_what_overflows_and_keeps_what_fits() { - let max = I256::from_i128(i128::MAX).checked_mul_u128(u128::MAX).unwrap(); + let max = I256::from_i128(i128::MAX) + .checked_mul_u128(u128::MAX) + .unwrap(); assert_eq!( max.to_bigint(), BigInt::from(i128::MAX) * BigInt::from(u128::MAX) diff --git a/crates/numtower/src/lang.rs b/crates/numtower/src/lang.rs index 7bb16c2b..e2a7d666 100644 --- a/crates/numtower/src/lang.rs +++ b/crates/numtower/src/lang.rs @@ -71,7 +71,8 @@ impl Lexeme { }; let (int, frac) = mantissa.split_once('.').unwrap_or((mantissa, "")); let all_digits = |s: &str| s.bytes().all(|b| b.is_ascii_digit()); - let mantissa_ok = all_digits(int) && all_digits(frac) && !(int.is_empty() && frac.is_empty()); + let mantissa_ok = + all_digits(int) && all_digits(frac) && !(int.is_empty() && frac.is_empty()); let exp = match exponent { Some(e) => read_exponent(e)?, None => 0, @@ -92,7 +93,9 @@ impl Lexeme { match stripped.is_empty() { true => padded(0, 0, implied_scale), false => { - let shift = self.exp10.saturating_add((significant.len() - stripped.len()) as i64); + let shift = self + .exp10 + .saturating_add((significant.len() - stripped.len()) as i64); self.exact(stripped, shift, implied_scale) .unwrap_or_else(|| Num::inexact(text.parse().unwrap_or(f64::NAN))) } @@ -174,7 +177,9 @@ impl fmt::Display for Num { NumRef::Exact(d) => write!(f, "{d}"), NumRef::Fraction(r) => write!(f, "{}/{}", r.numer(), r.denom()), NumRef::Inexact(x) if x.is_nan() => f.write_str("nan"), - NumRef::Inexact(x) if x.is_infinite() => f.write_str(if x > 0.0 { "inf" } else { "-inf" }), + NumRef::Inexact(x) if x.is_infinite() => { + f.write_str(if x > 0.0 { "inf" } else { "-inf" }) + } NumRef::Inexact(x) => write!(f, "{INEXACT_MARK}{x:?}"), } } diff --git a/crates/numtower/src/round.rs b/crates/numtower/src/round.rs index ec6a060e..c1a17633 100644 --- a/crates/numtower/src/round.rs +++ b/crates/numtower/src/round.rs @@ -68,18 +68,40 @@ mod tests { #[test] fn a_typed_value_rounds_as_it_was_typed() { let at2 = |n: &Num| n.round_to_places(2, Rounding::HalfAwayFromZero).unwrap(); - for (text, want) in [("1.005", "1.01"), ("1.015", "1.02"), ("0.285", "0.29"), ("-2.675", "-2.68")] { + for (text, want) in [ + ("1.005", "1.01"), + ("1.015", "1.02"), + ("0.285", "0.29"), + ("-2.675", "-2.68"), + ] { assert_eq!(at2(&typed(text)), want, "{text}"); - assert_eq!(at2(&Num::inexact(text.parse().unwrap())), want, "inexact {text}"); + assert_eq!( + at2(&Num::inexact(text.parse().unwrap())), + want, + "inexact {text}" + ); } assert_eq!(at2(&Num::ratio(10, 3).unwrap()), "3.33"); assert_eq!(at2(&Num::ratio(2, 3).unwrap()), "0.67"); assert_eq!(at2(&typed("-0.001")), "0.00"); - let even = typed("0.125").round_to_places(2, Rounding::HalfEven).unwrap(); + let even = typed("0.125") + .round_to_places(2, Rounding::HalfEven) + .unwrap(); assert_eq!(even, "0.12"); - assert_eq!(typed("12.5").round_to_places(0, Rounding::HalfEven).unwrap(), "12"); - assert_eq!(Num::inexact(f64::NAN).round_to_places(2, Rounding::HalfEven), None); - let wide = Num::ratio(1, 3).unwrap().round_to_places(40, Rounding::HalfEven).unwrap(); + assert_eq!( + typed("12.5") + .round_to_places(0, Rounding::HalfEven) + .unwrap(), + "12" + ); + assert_eq!( + Num::inexact(f64::NAN).round_to_places(2, Rounding::HalfEven), + None + ); + let wide = Num::ratio(1, 3) + .unwrap() + .round_to_places(40, Rounding::HalfEven) + .unwrap(); assert_eq!(wide, format!("0.{}", "3".repeat(40))); } } diff --git a/crates/numtower/src/sum.rs b/crates/numtower/src/sum.rs index f9b27b11..5276a50f 100644 --- a/crates/numtower/src/sum.rs +++ b/crates/numtower/src/sum.rs @@ -57,7 +57,11 @@ impl ExactSum { for j in 0..3 { let digit = (magnitude >> (30 * j)) & (CHUNK - 1); if digit != 0 { - let signed = if weight < 0 { -(digit as i64) } else { digit as i64 }; + let signed = if weight < 0 { + -(digit as i64) + } else { + digit as i64 + }; self.add_digit(x, signed, 30 * j); } } @@ -77,7 +81,11 @@ impl ExactSum { 0 => (fraction, 0), e => (fraction | (1 << 52), e - 1), }; - let sign = if x.is_sign_negative() { -weight } else { weight }; + let sign = if x.is_sign_negative() { + -weight + } else { + weight + }; let at = shift + extra; let wide = u128::from(mant) << (at % LIMB_BITS); let k = (at / LIMB_BITS) as usize; @@ -176,7 +184,10 @@ impl NumSum { .and_then(|t| t.checked_mul_u128(weight.unsigned_abs() as u128)) .and_then(|t| if weight < 0 { t.checked_neg() } else { Some(t) }) .expect("a weighted term is below 2^284"); - self.decimals = self.decimals.checked_add(&term).expect("320 bits holds 2^36 terms"); + self.decimals = self + .decimals + .checked_add(&term) + .expect("320 bits holds 2^36 terms"); } fn add_to_group(&mut self, n: i128, d: i128, weight: i64) { @@ -231,7 +242,11 @@ impl NumSum { .then(|| (self.decimals.to_bigint(), BigInt::from(pow10(MAX_SCALE)))); let terms: Vec<(BigInt, BigInt)> = decimal .into_iter() - .chain(self.groups.iter().map(|(d, n)| (n.to_bigint(), BigInt::from(*d)))) + .chain( + self.groups + .iter() + .map(|(d, n)| (n.to_bigint(), BigInt::from(*d))), + ) .collect(); let (num, den) = combine_over_lcm(&terms); BigQ::new(num, den) @@ -241,7 +256,9 @@ impl NumSum { /// `Σ n/d` over the lcm of the denominators, not yet reduced: the read reduces once. pub(crate) fn combine_over_lcm(terms: &[(BigInt, BigInt)]) -> (BigInt, BigInt) { let lcm = terms.iter().fold(BigInt::one(), |l, (_, d)| l.lcm(d)); - let num = terms.iter().fold(BigInt::zero(), |s, (n, d)| s + n * (&lcm / d)); + let num = terms + .iter() + .fold(BigInt::zero(), |s, (n, d)| s + n * (&lcm / d)); (num, lcm) } @@ -256,3 +273,98 @@ fn at_scale_28(d: i128) -> Option { } (rest == 1 && p.max(q) <= MAX_SCALE).then(|| (pow10(MAX_SCALE) / d) as u128) } + +#[cfg(test)] +mod tests { + use super::*; + use num_rational::BigRational; + use proptest::prelude::*; + use proptest::test_runner::{Config, RngAlgorithm, TestRng, TestRunner}; + + type Combine = fn(&[(BigInt, BigInt)]) -> (BigInt, BigInt); + + /// Up to K groups of wide numerators whose denominators share factors. + fn groups() -> impl Strategy> { + let shared = prop::sample::select(vec![1i128, 3, 7 * 11, 1 << 20, 3i128.pow(40)]); + let group = + (any::(), any::(), shared, 1i128..(1 << 60)).prop_map(|(n, w, s, d)| { + ( + BigInt::from(n) * BigInt::from(w), + BigInt::from(s) * BigInt::from(d), + ) + }); + prop::collection::vec(group, 1..=K) + } + + fn check_grouped_read(combine: Combine, deterministic: bool) -> Result<(), String> { + let config = Config { + cases: 256, + failure_persistence: None, + ..Config::default() + }; + let mut runner = match deterministic { + true => { + TestRunner::new_with_rng(config, TestRng::deterministic_rng(RngAlgorithm::ChaCha)) + } + false => TestRunner::new(config), + }; + let result = runner.run(&groups(), |terms| { + let (num, den) = combine(&terms); + let lcm = terms.iter().fold(BigInt::one(), |l, (_, d)| l.lcm(d)); + prop_assert_eq!(&den, &lcm, "the read did not combine over the lcm"); + let oracle: BigRational = terms + .iter() + .map(|(n, d)| BigRational::new(n.clone(), d.clone())) + .sum(); + let read = BigQ::new(num, den); + let got = BigRational::new(read.num.clone(), read.den.clone()); + prop_assert_eq!(got, oracle, "the read is not the sum"); + prop_assert!(read.num.gcd(&read.den).is_one(), "the read is not reduced"); + Ok(()) + }); + result.map_err(|e| e.to_string()) + } + + #[test] + fn the_grouped_read_combines_over_the_lcm_and_reduces_once() { + check_grouped_read(combine_over_lcm, false).unwrap(); + } + + fn over_the_product(terms: &[(BigInt, BigInt)]) -> (BigInt, BigInt) { + let den = terms.iter().fold(BigInt::one(), |p, (_, d)| p * d); + let num = terms + .iter() + .fold(BigInt::zero(), |s, (n, d)| s + n * (&den / d)); + (num, den) + } + + #[test] + fn the_grouped_read_combines_over_the_lcm_and_reduces_once_catches_the_product() { + let err = check_grouped_read(over_the_product, true).expect_err("the rival passed"); + assert!( + err.contains("the read did not combine over the lcm"), + "{err}" + ); + } + + #[test] + fn a_fraction_with_a_decimal_denominator_joins_the_decimal_partition() { + let mut s = NumSum::new(); + s.insert(&Num::ratio(10i128.pow(35), 1).unwrap()); + s.insert(&Num::ratio(3, 1 << 20).unwrap()); + assert_eq!(s.groups(), 0); + s.insert(&Num::ratio(1, 1i128 << 40).unwrap()); + s.insert(&Num::ratio(1, 3).unwrap()); + assert_eq!( + s.groups(), + 2, + "2^40 has scale 40, past 28: a group of its own" + ); + s.retract(&Num::ratio(1, 3).unwrap()); + assert_eq!( + s.groups(), + 1, + "a group whose numerator returns to zero is dropped" + ); + } +} diff --git a/crates/numtower/tests/oracle/mod.rs b/crates/numtower/tests/oracle/mod.rs new file mode 100644 index 00000000..6acb8f5c --- /dev/null +++ b/crates/numtower/tests/oracle/mod.rs @@ -0,0 +1,187 @@ +//! The `BigRational` oracle and the generators the property tests share. +//! Each property is a checker taking the implementation, so a rival can be run through it. + +#![allow(dead_code)] + +use std::cmp::Ordering; + +use num_bigint::BigInt; +use num_rational::BigRational; +use num_traits::{One, Signed, ToPrimitive, Zero}; +use numtower::{Decimal, Num, NumRef}; +use proptest::prelude::*; +use proptest::test_runner::{Config, RngAlgorithm, TestCaseError, TestRng, TestRunner}; + +pub fn q(n: &Num) -> Option { + match n.kind() { + NumRef::Exact(d) => Some(BigRational::new( + d.mantissa().into(), + BigInt::from(10).pow(d.scale()), + )), + NumRef::Fraction(r) => Some(BigRational::new((*r.numer()).into(), (*r.denom()).into())), + NumRef::Inexact(x) => BigRational::from_float(x), + } +} + +/// Whether `v` fits a stored exact arm: a decimal at scale <= 28, or an `i128` fraction. +pub fn fits(v: &BigRational) -> bool { + let limit = BigInt::one() << 96; + let decimal = (0..=28u32).any(|s| { + let scaled = v * BigRational::from_integer(BigInt::from(10).pow(s)); + scaled.is_integer() && scaled.to_integer().abs() < limit + }); + let max = BigInt::from(i128::MAX); + decimal || (v.numer().abs() <= max && v.denom() <= &max) +} + +/// Whether `f` is the round-to-nearest-even f64 of `v`: the review's interval form. +pub fn is_nearest(f: f64, v: &BigRational) -> bool { + let exact = |x: f64| BigRational::from_float(x).expect("finite"); + let two = BigRational::from_integer(BigInt::from(2)); + let half_ulp_past = |x: f64, up: bool| { + let (next, inward) = match up { + true => (x.next_up(), x.next_down()), + false => (x.next_down(), x.next_up()), + }; + match next.is_finite() { + true => (exact(x) + exact(next)) / &two, + false => exact(x) + (exact(x) - exact(inward)) / &two, + } + }; + match f.is_finite() { + false => { + let edge = half_ulp_past(f64::MAX.copysign(f), f > 0.0); + (f > 0.0 && v >= &edge) || (f < 0.0 && v <= &edge) + } + true => { + let lo = half_ulp_past(f, false); + let hi = half_ulp_past(f, true); + let even = f.to_bits() & 1 == 0; + match even { + true => &lo <= v && v <= &hi, + false => &lo < v && v < &hi, + } + } + } +} + +/// A decimal's text built digit by digit, without `rust_decimal`'s `Display`. +pub fn decimal_text(mantissa: i128, scale: u32) -> String { + let digits = format!( + "{:0>width$}", + mantissa.unsigned_abs(), + width = scale as usize + 1 + ); + let (int, frac) = digits.split_at(digits.len() - scale as usize); + let sign = if mantissa < 0 { "-" } else { "" }; + format!("{sign}{int}.{frac}0") +} + +/// The correctly rounded f64 of an exact value, independent of `numtower`'s own. +pub fn independent_f64(n: &Num) -> f64 { + match n.kind() { + NumRef::Exact(d) => decimal_text(d.mantissa(), d.scale()).parse().unwrap(), + NumRef::Fraction(_) => q(n).unwrap().to_f64().unwrap(), + NumRef::Inexact(x) => x, + } +} + +/// Same arm, same value and same scale: what a fingerprint would see. +pub fn same(a: &Num, b: &Num) -> bool { + match (a.kind(), b.kind()) { + (NumRef::Exact(x), NumRef::Exact(y)) => { + x.mantissa() == y.mantissa() && x.scale() == y.scale() + } + (NumRef::Fraction(x), NumRef::Fraction(y)) => x == y, + (NumRef::Inexact(x), NumRef::Inexact(y)) => x.to_bits() == y.to_bits(), + _ => false, + } +} + +pub fn arm(n: &Num) -> &'static str { + match n.kind() { + NumRef::Exact(_) => "exact", + NumRef::Fraction(_) => "fraction", + NumRef::Inexact(_) => "inexact", + } +} + +pub fn mantissa96() -> impl Strategy { + let max = (1i128 << 96) - 1; + prop_oneof![ + -max..=max, + -1_000_000i128..1_000_000, + -(10i128.pow(15))..10i128.pow(15), + ] +} + +pub fn decimal() -> impl Strategy { + (mantissa96(), 0u32..=28).prop_map(|(m, s)| Num::exact(Decimal::from_i128_with_scale(m, s))) +} + +fn of_width(bits: u32) -> impl Strategy { + (any::(), any::()).prop_map(move |(r, neg)| { + let v = ((r >> (128 - bits)) | (1u128 << (bits - 1))) as i128; + if neg { -v } else { v } + }) +} + +pub fn fraction() -> impl Strategy { + (1u32..=126, 1u32..=126) + .prop_flat_map(|(a, b)| (of_width(a), of_width(b))) + .prop_map(|(n, d)| Num::ratio(n, d).unwrap()) +} + +pub fn finite_f64() -> impl Strategy { + use proptest::num::f64::{NEGATIVE, NORMAL, POSITIVE, SUBNORMAL, ZERO}; + prop_oneof![ + (POSITIVE | NEGATIVE | NORMAL | SUBNORMAL | ZERO), + -1e6f64..1e6, + (-1000i64..1000).prop_map(|k| k as f64 / 8.0), + ] +} + +pub fn exact() -> impl Strategy { + prop_oneof![decimal(), fraction()] +} + +pub fn any_num() -> impl Strategy { + prop_oneof![ + 3 => exact(), + 1 => finite_f64().prop_map(Num::inexact), + 1 => prop_oneof![Just(f64::NAN), Just(f64::INFINITY), Just(f64::NEG_INFINITY)].prop_map(Num::inexact), + ] +} + +/// Runs a property; `deterministic` for a rival, so its failure is not luck. +pub fn run( + strategy: S, + cases: u32, + deterministic: bool, + test: impl Fn(S::Value) -> Result<(), TestCaseError>, +) -> Result<(), String> { + let config = Config { + cases, + failure_persistence: None, + ..Config::default() + }; + let mut runner = match deterministic { + true => TestRunner::new_with_rng(config, TestRng::deterministic_rng(RngAlgorithm::ChaCha)), + false => TestRunner::new(config), + }; + runner.run(&strategy, test).map_err(|e| e.to_string()) +} + +/// A rival must fail on the property's own assertion, never on something else. +pub fn assert_caught(result: Result<(), String>, marker: &str) { + let err = result.expect_err("the rival passed: the property cannot see it"); + assert!(err.contains(marker), "failed for another reason: {err}"); +} + +pub fn oracle_cmp(a: &Num, b: &Num) -> Option { + Some(q(a)?.cmp(&q(b)?)) +} + +pub fn integer(v: &BigRational) -> bool { + v.is_integer() && !v.numer().is_zero() +} diff --git a/crates/numtower/tests/properties.rs b/crates/numtower/tests/properties.rs new file mode 100644 index 00000000..68cc76e5 --- /dev/null +++ b/crates/numtower/tests/properties.rs @@ -0,0 +1,1019 @@ +//! ADR 005 migration step 1's property tests, each beside the rival it must catch. +//! What each rival is and why it matters: `context/test-evidence/numtower.md`. + +mod oracle; + +use std::cmp::Ordering; +use std::str::FromStr; + +use num_bigint::BigInt; +use num_rational::BigRational; +use num_traits::{ToPrimitive, Zero}; +use numtower::{Decimal, Interim, Num, NumRef, NumSum, Typing, is_number, parse_number}; +use oracle::*; +use proptest::prelude::*; + +fn typed(s: &str) -> Num { + parse_number(s, Typing::Inferred).unwrap_or_else(|| panic!("{s} is a number")) +} + +fn dec(s: &str) -> Num { + Num::exact(Decimal::from_str(s).unwrap()) +} + +// --------------------------------------------------------------------------- +// An Exact result is never a rounded one. + +type BinOp<'a> = &'a dyn Fn(&Num, &Num) -> Option; + +fn exact_op(name: &str) -> fn(&BigRational, &BigRational) -> Option { + match name { + "+" => |a, b| Some(a + b), + "-" => |a, b| Some(a - b), + "*" => |a, b| Some(a * b), + _ => |a, b| (b.numer() != &BigInt::from(0)).then(|| a / b), + } +} + +fn check_never_rounded(name: &str, op: BinOp, deterministic: bool) -> Result<(), String> { + let oracle_op = exact_op(name); + run((exact(), exact()), 256, deterministic, |(a, b)| { + if let Some(r) = op(&a, &b).filter(Num::is_exact) { + let want = oracle_op(&q(&a).unwrap(), &q(&b).unwrap()); + prop_assert!( + q(&r) == want, + "an exact result was rounded: {a} {name} {b} = {r}" + ); + } + Ok(()) + }) +} + +fn real(name: &str) -> impl Fn(&Num, &Num) -> Option + '_ { + move |a: &Num, b: &Num| match name { + "+" => Some(a + b), + "-" => Some(a - b), + "*" => Some(a * b), + _ => a.checked_div(b).ok(), + } +} + +#[test] +fn an_exact_result_is_never_a_rounded_one() { + for name in ["+", "-", "*", "/"] { + check_never_rounded(name, &real(name), false).unwrap(); + } + let third = dec("0.3333333333333333333333333333"); + let sum = &dec("1000") + &third; + assert!(sum.is_exact(), "{sum}"); + assert_eq!(q(&sum), Some(q(&dec("1000")).unwrap() + q(&third).unwrap())); +} + +/// `rust_decimal`'s own `+`, trusted as exact: it rounds past 28 digits and says nothing. +fn silently_rounding_add(a: &Num, b: &Num) -> Option { + match (a.kind(), b.kind()) { + (NumRef::Exact(x), NumRef::Exact(y)) => { + Some(x.checked_add(*y).map_or_else(|| a + b, Num::exact)) + } + _ => Some(a + b), + } +} + +#[test] +fn an_exact_result_is_never_a_rounded_one_catches_a_silently_rounding_add() { + assert_caught( + check_never_rounded("+", &silently_rounding_add, true), + "an exact result was rounded", + ); +} + +// --------------------------------------------------------------------------- +// Contagion is total. + +fn check_contagion(add: BinOp, deterministic: bool) -> Result<(), String> { + let seed = Just((dec("2"), Num::inexact(3.0))); + let cases = prop_oneof![1 => seed, 8 => (any_num(), any_num())]; + run(cases, 512, deterministic, |(a, b)| { + let results = [ + ("+", add(&a, &b)), + ("-", Some(&a - &b)), + ("*", Some(&a * &b)), + ("/", a.checked_div(&b).ok()), + ]; + for (name, r) in results { + match (r, a.is_exact() && b.is_exact()) { + (None, _) => prop_assert!( + name == "/" && b.is_zero(), + "contagion: {a} {name} {b} refused" + ), + (Some(r), false) => { + prop_assert!(!r.is_exact(), "contagion: {a} {name} {b} = {r} is exact") + } + (Some(r), true) => { + let v = exact_op(name)(&q(&a).unwrap(), &q(&b).unwrap()).unwrap(); + prop_assert_eq!( + r.is_exact(), + fits(&v), + "contagion: {} {} {} = {}", + a, + name, + b, + r + ); + prop_assert!( + r.is_exact() || is_nearest(r.to_f64(), &v), + "contagion: {a} {name} {b} = {r}" + ); + } + } + } + prop_assert_eq!( + a.checked_div(&b).is_err(), + b.is_zero(), + "contagion: div/0 for {} / {}", + a, + b + ); + Ok(()) + }) +} + +#[test] +fn contagion_is_total() { + check_contagion(&|a, b| Some(a + b), false).unwrap(); +} + +/// Keeps an integral f64 exact, as a "harmless" promotion would. +fn integral_inexact_kept_exact(a: &Num, b: &Num) -> Option { + match (a.is_exact(), b.kind()) { + (true, NumRef::Inexact(x)) if x.is_finite() && x.fract() == 0.0 && x.abs() < 1e15 => { + Some(a + &Num::exact(Decimal::from(x as i64))) + } + _ => Some(a + b), + } +} + +#[test] +fn contagion_is_total_catches_an_integral_inexact_kept_exact() { + assert_caught( + check_contagion(&integral_inexact_kept_exact, true), + "contagion", + ); +} + +#[test] +fn pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits() { + run((any_num(), -6i32..=6), 256, false, |(a, k)| { + let r = a.pow(&Num::exact(Decimal::from(k))); + match (q(&a).filter(|_| a.is_exact()), r) { + (_, Err(_)) => prop_assert!(a.is_zero() && k < 0, "{a}^{k} refused"), + (None, Ok(r)) => prop_assert!(!r.is_exact(), "{a}^{k} = {r} is exact"), + (Some(v), Ok(r)) => { + let want = v.pow(k); + prop_assert_eq!(r.is_exact(), fits(&want), "{}^{} = {}", a, k, r); + prop_assert!( + !r.is_exact() || q(&r) == Some(want.clone()), + "{a}^{k} = {r}" + ); + prop_assert!( + r.is_exact() || is_nearest(r.to_f64(), &want), + "{a}^{k} = {r}" + ); + } + } + Ok(()) + }) + .unwrap(); +} + +#[test] +fn pow_decides_its_size_before_computing_and_honours_the_ceiling() { + let third = Num::ratio(1, 3).unwrap(); + let huge = third + .pow(&Num::exact(Decimal::from(1_000_000_000))) + .unwrap(); + assert!(!huge.is_exact() && huge.to_f64() == 0.0, "{huge}"); + let past = third.pow(&Num::exact(Decimal::from(6000))).unwrap(); + assert!( + !past.is_exact(), + "3^6000 has 9,510 bits, past the 8,192-bit ceiling" + ); + let growth = dec("1.05").pow(&Num::exact(Decimal::from(28))).unwrap(); + assert!(growth.is_exact(), "1.05^28 is exact under (d)"); + assert!( + !dec("1.05") + .pow(&Num::exact(Decimal::from(29))) + .unwrap() + .is_exact() + ); + let unit = dec("-1.0") + .pow(&Num::exact(Decimal::from_i128_with_scale( + 10i128.pow(20), + 0, + ))) + .unwrap(); + assert!( + same(&unit, &dec("1.0000000000000000000000000000")), + "{unit}" + ); + assert!(dec("0").pow(&dec("-1")).is_err()); + assert!(!dec("2").pow(&dec("0.5")).unwrap().is_exact()); + assert!(same(&dec("2").pow(&dec("-2")).unwrap(), &dec("0.25"))); +} + +// --------------------------------------------------------------------------- +// The conversion to f64 agrees with str::parse of the decimal text. + +fn check_to_f64( + conv: &dyn Fn(&Decimal) -> f64, + range: i128, + deterministic: bool, +) -> Result<(), String> { + run((-range..=range, 0u32..=28), 256, deterministic, |(m, s)| { + let d = Decimal::from_i128_with_scale(m, s); + let want: f64 = decimal_text(m, s).parse().unwrap(); + prop_assert_eq!( + conv(&d).to_bits(), + want.to_bits(), + "to_f64 disagrees with the text of {}", + d + ); + Ok(()) + }) +} + +const FULL: i128 = (1 << 96) - 1; + +#[test] +fn the_f64_conversion_agrees_with_parsing_the_decimal_text() { + check_to_f64(&|d| Num::exact(*d).to_f64(), FULL, false).unwrap(); + let review = dec("63294.607176721429540940386554"); + assert_eq!( + review.to_f64(), + "63294.607176721429540940386554".parse::().unwrap() + ); +} + +fn crate_to_f64(d: &Decimal) -> f64 { + rust_decimal::prelude::ToPrimitive::to_f64(d).unwrap() +} + +#[test] +fn the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_to_f64() { + assert_caught(check_to_f64(&crate_to_f64, FULL, true), "to_f64 disagrees"); + let review = Decimal::from_str("63294.607176721429540940386554").unwrap(); + assert_ne!( + crate_to_f64(&review), + "63294.607176721429540940386554".parse::().unwrap() + ); +} + +#[test] +fn rust_decimal_to_f64_agrees_on_fifteen_digit_mantissas_so_a_narrow_generator_is_vacuous() { + let narrow = 10i128.pow(15) - 1; + let result = run((-narrow..=narrow, 0u32..=15), 4096, true, |(m, s)| { + let d = Decimal::from_i128_with_scale(m, s); + prop_assert_eq!( + crate_to_f64(&d).to_bits(), + decimal_text(m, s).parse::().unwrap().to_bits() + ); + Ok(()) + }); + result.unwrap(); +} + +// --------------------------------------------------------------------------- +// A fraction's conversion lies inside the round-to-nearest-even interval. + +fn wide_fraction() -> impl Strategy { + ( + 20u32..=126, + 20u32..=126, + any::(), + any::(), + any::(), + ) + .prop_map(|(a, b, x, y, neg)| { + let n = ((x >> (128 - a)) | (1 << (a - 1))) as i128; + let d = ((y >> (128 - b)) | (1 << (b - 1))) as i128; + Num::ratio(if neg { -n } else { n }, d).unwrap() + }) +} + +fn check_fraction_conversion( + conv: &dyn Fn(&Num) -> f64, + deterministic: bool, +) -> Result<(), String> { + run( + (wide_fraction(), wide_fraction()), + 256, + deterministic, + |(x, y)| { + prop_assert!( + is_nearest(conv(&x), &q(&x).unwrap()), + "outside the interval: {x}" + ); + let wide = (&Interim::from(x.clone()) * &Interim::from(y.clone())).store(); + let exact = q(&x).unwrap() * q(&y).unwrap(); + prop_assert!( + wide.is_exact() || is_nearest(wide.to_f64(), &exact), + "outside the interval: {x} * {y}" + ); + Ok(()) + }, + ) +} + +#[test] +fn a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval() { + check_fraction_conversion(&Num::to_f64, false).unwrap(); +} + +fn divide_in_f64(x: &Num) -> f64 { + match x.kind() { + NumRef::Fraction(r) => *r.numer() as f64 / *r.denom() as f64, + _ => x.to_f64(), + } +} + +#[test] +fn a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval_catches_an_f64_division() { + assert_caught( + check_fraction_conversion(÷_in_f64, true), + "outside the interval", + ); +} + +// --------------------------------------------------------------------------- +// Exact mixed comparison is transitive. + +/// Values crowded around one f64: equal-looking decimals, the f64 and its neighbours. +fn crowd() -> impl Strategy> { + (-1e6f64..1e6, 1i64..9).prop_map(|(x, k)| { + let shortest = typed(&format!("{x:?}")); + let nudge = Num::exact(Decimal::new(k, 25)); + vec![ + &shortest + &nudge, + Num::inexact(x), + &shortest - &nudge, + shortest.clone(), + Num::inexact(x.next_up()), + Num::inexact(x.next_down()), + shortest.checked_div(&Num::exact(Decimal::from(3))).unwrap(), + ] + }) +} + +type Cmp<'a> = &'a dyn Fn(&Num, &Num) -> Option; + +fn check_transitive(cmp: Cmp, deterministic: bool) -> Result<(), String> { + run(crowd(), 256, deterministic, |xs| { + for a in &xs { + for b in &xs { + let reversed = cmp(b, a).map(Ordering::reverse); + prop_assert_eq!( + cmp(a, b), + reversed, + "comparison: not antisymmetric: {} vs {}", + a, + b + ); + for c in &xs { + let (ab, bc, ac) = (cmp(a, b), cmp(b, c), cmp(a, c)); + let chained = ab.is_some_and(|o| o != Ordering::Greater) + && bc.is_some_and(|o| o != Ordering::Greater); + let strict = ab == Some(Ordering::Less) || bc == Some(Ordering::Less); + let want = if strict { + Ordering::Less + } else { + Ordering::Equal + }; + prop_assert!( + !chained || ac == Some(want), + "comparison: not transitive: {a}, {b}, {c}" + ); + } + } + } + Ok(()) + }) +} + +#[test] +fn exact_mixed_comparison_is_transitive() { + check_transitive(&Num::cmp_exact, false).unwrap(); + assert_ne!(dec("0.1"), Num::inexact(0.1)); +} + +#[test] +fn comparison_agrees_with_the_oracle_at_every_pairing_of_arms() { + let finite = prop_oneof![exact(), finite_f64().prop_map(Num::inexact)]; + let pools = prop_oneof![crowd(), prop::collection::vec(finite, 2..6)]; + run(pools, 256, false, |xs| { + for a in &xs { + for b in &xs { + prop_assert_eq!(a.cmp_exact(b), oracle_cmp(a, b), "{} vs {}", a, b); + } + } + Ok(()) + }) + .unwrap(); + assert_eq!(Num::inexact(f64::NAN).cmp_exact(&dec("1")), None); + assert_eq!( + dec("1").cmp_exact(&Num::inexact(f64::INFINITY)), + Some(Ordering::Less) + ); +} + +/// Revision 1's rule: the exact operand converted to f64, then compared. +fn compare_through_f64(a: &Num, b: &Num) -> Option { + match a.is_exact() == b.is_exact() { + true => a.cmp_exact(b), + false => a.to_f64().partial_cmp(&b.to_f64()), + } +} + +#[test] +fn exact_mixed_comparison_is_transitive_catches_comparing_through_f64() { + assert_caught( + check_transitive(&compare_through_f64, true), + "comparison: not transitive", + ); +} + +#[test] +fn min_and_max_prefer_the_exact_value_and_do_not_depend_on_order() { + let half = dec("0.50"); + let pairs = [ + (half.clone(), Num::inexact(0.5)), + (dec("1.5"), dec("1.50")), + (Num::inexact(0.0), Num::inexact(-0.0)), + (Num::inexact(f64::NAN), dec("2")), + ]; + for (a, b) in &pairs { + assert!(same(a.min_of(b), b.min_of(a)), "min of {a}, {b}"); + assert!(same(a.max_of(b), b.max_of(a)), "max of {a}, {b}"); + } + assert!(same(half.min_of(&Num::inexact(0.5)), &half)); + assert!(same(dec("1.5").max_of(&dec("1.50")), &dec("1.50"))); + assert!( + Num::inexact(0.0) + .min_of(&Num::inexact(-0.0)) + .to_f64() + .is_sign_negative() + ); + assert!( + !Num::inexact(-0.0) + .max_of(&Num::inexact(0.0)) + .to_f64() + .is_sign_negative() + ); + assert!(same(Num::inexact(f64::NAN).min_of(&dec("2")), &dec("2"))); +} + +// --------------------------------------------------------------------------- +// The accumulator is independent of order. + +fn term() -> impl Strategy { + let pooled = ( + -1000i128..1000, + prop::sample::select(vec![3i128, 7, 12, 49, 1 << 40, 3 * (1 << 33)]), + ) + .prop_map(|(n, d)| Num::ratio(n, d).unwrap()); + prop_oneof![ + 4 => decimal(), + 2 => pooled, + 1 => fraction(), + 1 => finite_f64().prop_map(Num::inexact), + ] +} + +fn sum_all(xs: &[Num]) -> Num { + let mut s = NumSum::new(); + xs.iter().for_each(|x| s.insert(x)); + s.read() +} + +fn check_order_free(sum: &dyn Fn(&[Num]) -> Num, deterministic: bool) -> Result<(), String> { + let seed = vec![ + dec("0.30000000000000004"), + dec("1000000000000"), + dec("-1000000000000"), + ]; + let cases = prop_oneof![ + 1 => Just(seed), + 4 => prop::collection::vec(term(), 0..24), + ]; + let shuffled = cases.prop_flat_map(|xs| (Just(xs.clone()), Just(xs).prop_shuffle())); + run(shuffled, 256, deterministic, |(xs, ys)| { + let (a, b) = (sum(&xs), sum(&ys)); + prop_assert!(same(&a, &b), "depends on order: {a} against {b}"); + Ok(()) + }) +} + +#[test] +fn the_accumulator_is_independent_of_order() { + check_order_free(&sum_all, false).unwrap(); + let seed = sum_all(&[ + dec("0.30000000000000004"), + dec("1000000000000"), + dec("-1000000000000"), + ]); + assert!(same(&seed, &dec("0.30000000000000004")), "{seed}"); +} + +/// Clause 8's rival: `Decimal`'s own `+`, demoting to f64 where it overflows. +fn decimal_fold(xs: &[Num]) -> Num { + xs.iter().fold(Num::exact(Decimal::ZERO), |acc, x| { + match (acc.kind(), x.kind()) { + (NumRef::Exact(a), NumRef::Exact(b)) => { + a.checked_add(*b).map_or_else(|| &acc + x, Num::exact) + } + _ => &acc + x, + } + }) +} + +#[test] +fn the_accumulator_is_independent_of_order_catches_decimal_addition() { + assert_caught(check_order_free(&decimal_fold, true), "depends on order"); +} + +#[test] +fn the_accumulator_retracts_exactly() { + run( + ( + prop::collection::vec(term(), 0..16), + prop::collection::vec(term(), 0..16), + ), + 256, + false, + |(xs, ys)| { + let mut s = NumSum::new(); + xs.iter().chain(&ys).for_each(|x| s.insert(x)); + ys.iter().rev().for_each(|y| s.retract(y)); + let fresh = sum_all(&xs); + prop_assert!( + same(&s.read(), &fresh), + "{} after retraction, {} fresh", + s.read(), + fresh + ); + Ok(()) + }, + ) + .unwrap(); +} + +#[test] +fn the_accumulator_reads_the_exact_sum_or_its_stated_roundings() { + run(prop::collection::vec(term(), 0..24), 256, false, |xs| { + let got = sum_all(&xs); + let mut groups = std::collections::BTreeMap::::new(); + let mut exact = BigRational::from_integer(BigInt::from(0)); + let mut floats = Vec::new(); + for x in &xs { + match x.kind() { + NumRef::Inexact(f) => floats.push(f), + NumRef::Fraction(r) if !decimal_denominator(*r.denom()) => { + *groups.entry(*r.denom()).or_insert_with(BigRational::zero) += q(x).unwrap(); + exact += q(x).unwrap(); + } + _ => exact += q(x).unwrap(), + } + } + let live = groups.values().filter(|v| !v.is_zero()).count(); + match (floats.is_empty(), live <= numtower::K) { + (true, true) => { + prop_assert_eq!(got.is_exact(), fits(&exact), "{}", got); + prop_assert!(!got.is_exact() || q(&got) == Some(exact.clone()), "{got}"); + prop_assert!(got.is_exact() || is_nearest(got.to_f64(), &exact), "{got}"); + } + (false, true) => { + let mut total = BigRational::from_float(exact.to_f64().unwrap()).unwrap(); + floats + .iter() + .for_each(|f| total += BigRational::from_float(*f).unwrap()); + prop_assert!(!got.is_exact() && is_nearest(got.to_f64(), &total), "{got}"); + } + (_, false) => prop_assert!(!got.is_exact(), "past K groups: {got}"), + } + Ok(()) + }) + .unwrap(); +} + +fn decimal_denominator(d: i128) -> bool { + let p = d.trailing_zeros(); + let mut rest = d >> p; + let mut q5 = 0; + while rest % 5 == 0 { + rest /= 5; + q5 += 1; + } + rest == 1 && p.max(q5) <= 28 +} + +#[test] +fn more_than_k_denominators_read_inexact_even_when_the_total_fits() { + let primes = [3i128, 7, 11, 13, 17, 19, 23, 29, 31]; + let terms: Vec = primes.iter().map(|&p| Num::ratio(1, p).unwrap()).collect(); + let eight = sum_all(&terms[..8]); + assert!(matches!(eight.kind(), NumRef::Fraction(_)), "{eight}"); + let nine = sum_all(&terms); + assert!(!nine.is_exact(), "{nine}"); + let exact: BigRational = terms.iter().map(|t| q(t).unwrap()).sum(); + assert!( + fits(&exact), + "the exact total fits, so the cutoff is what demotes" + ); + let each_rounded: BigRational = terms + .iter() + .map(|t| BigRational::from_float(q(t).unwrap().to_f64().unwrap()).unwrap()) + .sum(); + assert!( + is_nearest(nine.to_f64(), &each_rounded), + "each group rounded, then the sum once" + ); +} + +#[test] +fn exact_sum_is_the_exact_weighted_total_rounded_once() { + let weights = prop_oneof![-3i64..=3, any::(), (-(1i64 << 40))..(1i64 << 40)]; + run( + prop::collection::vec((finite_f64(), weights), 0..20), + 256, + false, + |terms| { + let mut acc = numtower::ExactSum::new(); + let mut exact = BigRational::from_integer(BigInt::from(0)); + for (x, w) in &terms { + acc.add(*x, *w); + exact += BigRational::from_float(*x).unwrap() + * BigRational::from_integer(BigInt::from(*w)); + } + prop_assert!( + is_nearest(acc.value(), &exact), + "{} for {:?}", + acc.value(), + terms + ); + Ok(()) + }, + ) + .unwrap(); +} + +// --------------------------------------------------------------------------- +// Division's direct fit test. + +fn check_division(div: BinOp, deterministic: bool) -> Result<(), String> { + let seeds = prop_oneof![ + Just((dec("1"), dec("1099511627776"))), + Just((dec("1"), dec("931322574615478515625"))), + Just((dec("0.0000000000000000000000000001"), dec("0.5"))), + ]; + let cases = prop_oneof![1 => seeds, 3 => (decimal(), decimal())]; + run(cases, 256, deterministic, |(a, b)| { + if let (Some(r), Some(v)) = (div(&a, &b), exact_op("/")(&q(&a).unwrap(), &q(&b).unwrap())) { + prop_assert_eq!( + r.is_exact(), + fits(&v), + "division fit: {} / {} = {}", + a, + b, + r + ); + prop_assert!( + !r.is_exact() || q(&r) == Some(v.clone()), + "division fit: {a} / {b} = {r}" + ); + let decimal = (0..=28u32).any(|s| { + let scaled = &v * BigRational::from_integer(BigInt::from(10).pow(s)); + scaled.is_integer() && scaled.to_integer().magnitude().bits() <= 96 + }); + prop_assert_eq!( + matches!(r.kind(), NumRef::Exact(_)), + decimal, + "division fit: {} / {} = {}", + a, + b, + r + ); + } + Ok(()) + }) +} + +#[test] +fn division_decides_fit_directly() { + check_division(&|a, b| a.checked_div(b).ok(), false).unwrap(); + let frac = |n: &Num| matches!(n.kind(), NumRef::Fraction(_)); + assert!( + frac(&dec("1").checked_div(&dec("1099511627776")).unwrap()), + "1/2^40" + ); + assert!( + frac(&dec("1").checked_div(&dec("931322574615478515625")).unwrap()), + "1/5^30" + ); + let tiny = dec("0.0000000000000000000000000001") + .checked_div(&dec("0.5")) + .unwrap(); + assert!( + same(&tiny, &dec("0.0000000000000000000000000002")), + "{tiny}" + ); + let avg = &(&(&dec("10.00") + &dec("20.00")) + &dec("40.00")); + assert!(frac(&avg.checked_div(&dec("3")).unwrap())); + assert!(same( + &dec("1").checked_div(&dec("4")).unwrap(), + &dec("0.25") + )); + let third = Num::ratio(1, 3).unwrap(); + assert!( + same(&(&third * &dec("3")), &dec("1")), + "1/3 * 3 is exactly 1" + ); + assert_eq!( + Num::inexact(1.0).checked_div(&Num::inexact(-0.0)), + Err(numtower::DivisionByZero) + ); +} + +/// Multiplying back, with revision 6's scale test: sound, and incomplete. +fn multiply_back(a: &Num, b: &Num) -> Option { + match (a.kind(), b.kind()) { + (NumRef::Exact(x), NumRef::Exact(y)) if !y.is_zero() => { + let candidate = x.checked_div(*y)?; + let back = candidate.checked_mul(*y)?; + let kept = back.scale() == candidate.scale() + y.scale() && back == *x; + Some(match kept { + true => Num::exact(candidate), + false => Num::inexact(a.to_f64() / b.to_f64()), + }) + } + _ => a.checked_div(b).ok(), + } +} + +#[test] +fn division_decides_fit_directly_catches_multiplying_back() { + assert_caught(check_division(&multiply_back, true), "division fit"); +} + +// --------------------------------------------------------------------------- +// The number language: the gate and the parse agree. + +fn lexeme() -> impl Strategy { + let digits = "[0-9]{0,40}"; + let structured = ( + prop::option::of("[+-]"), + digits, + prop::option::of(("\\.", digits)), + prop::option::of(("[eE]", prop::option::of("[+-]"), "[0-9]{0,4}")), + ) + .prop_map(|(sign, int, frac, exp)| { + let frac = frac.map(|(p, f)| format!("{p}{f}")).unwrap_or_default(); + let exp = exp + .map(|(e, s, d)| format!("{e}{}{d}", s.unwrap_or_default())) + .unwrap_or_default(); + format!("{}{int}{frac}{exp}", sign.unwrap_or_default()) + }); + prop_oneof![ + 4 => structured, + 1 => "[0-9eE+\\-._/aifnIN ]{0,10}", + 1 => "[+-]?[0-9]{1,30}/[0-9]{1,30}", + 1 => prop::sample::select(vec!["inf", "-inf", "nan", "NaN", "Infinity", "-infinity", "1_000", "+.5", "1.", ".", "e5"]).prop_map(String::from), + ] +} + +fn check_language( + parse: &dyn Fn(&str, Typing) -> Option, + deterministic: bool, +) -> Result<(), String> { + run(lexeme(), 512, deterministic, |s| { + let gate = s.parse::().is_ok_and(f64::is_finite); + let parsed = parse(&s, Typing::Inferred); + prop_assert_eq!( + parsed.is_some(), + gate, + "the gate and the parse disagree on {:?}", + s + ); + prop_assert_eq!( + is_number(&s, Typing::Inferred), + gate, + "the gate and the parse disagree on {:?}", + s + ); + if let Some(n) = &parsed { + let nearest: f64 = s.parse().unwrap(); + prop_assert!( + n.to_f64() == nearest, + "the gate and the parse disagree on the value of {s:?}: {n}" + ); + let v = q(n).unwrap(); + prop_assert!( + !n.is_exact() || is_nearest(nearest, &v), + "the gate and the parse disagree on {s:?}" + ); + } + let forced = parse(&s, Typing::Forced); + prop_assert_eq!( + forced.is_some(), + is_number(&s, Typing::Forced), + "the gate and the parse disagree on forced {:?}", + s + ); + Ok(()) + }) +} + +#[test] +fn the_number_languages_gate_and_parse_agree() { + check_language(&parse_number, false).unwrap(); +} + +/// `rust_decimal`'s exact parse: it refuses `1.5e3` and accepts `1_000`. +fn from_str_exact(s: &str, typing: Typing) -> Option { + match Decimal::from_str_exact(s) { + Ok(d) => Some(Num::exact(d)), + Err(_) if typing == Typing::Forced => parse_number(s, typing), + Err(_) => None, + } +} + +#[test] +fn the_number_languages_gate_and_parse_agree_catches_from_str_exact() { + assert_caught( + check_language(&from_str_exact, true), + "the gate and the parse disagree", + ); +} + +#[test] +fn the_number_language_seed_cases() { + let inferred = |s: &str| parse_number(s, Typing::Inferred); + let forced = |s: &str| parse_number(s, Typing::Forced); + assert!(same(&typed("1.5e3"), &dec("1500"))); + assert!(same(&typed("1.50"), &dec("1.50"))); + assert_eq!(inferred("1_000").map(|n| n.to_string()), None); + let zero = typed("-0.0"); + assert!(same(&zero, &dec("0.0")) && !zero.to_f64().is_sign_negative()); + assert!(inferred("3/4").is_none()); + assert!(same(&forced("3/4").unwrap(), &dec("0.75"))); + assert!(matches!( + forced("-10/3").unwrap().kind(), + NumRef::Fraction(_) + )); + assert!(forced("3/0").is_none()); + for text in ["1e999", "-1e999", "nan", "inf", "Infinity", "NaN", "-inf"] { + assert!( + inferred(text).is_none() && forced(text).is_none(), + "{text} is text" + ); + } + let underflow = typed("1e-999"); + assert!(!underflow.is_exact() && underflow.to_f64() == 0.0); + assert_eq!(typed("1e-30"), Num::ratio(1, 10i128.pow(30)).unwrap()); + assert!(matches!(typed("1e-30").kind(), NumRef::Fraction(_))); + let long = typed("0.12345678901234567890123456789"); + let want = BigRational::new( + BigInt::from(12345678901234567890123456789i128), + BigInt::from(10).pow(29), + ); + assert!(matches!(long.kind(), NumRef::Fraction(_)) && q(&long) == Some(want)); + let big = (1i128 << 120).to_string(); + let n = typed(&big); + assert!(matches!(n.kind(), NumRef::Fraction(r) if *r.denom() == 1 && *r.numer() == 1 << 120)); + assert!(!typed("1e-40").is_exact()); + assert!(!typed("1234567890123456789012345678901234567890").is_exact()); + assert!(same(&typed("0e99999999999999999999"), &dec("0"))); +} + +// --------------------------------------------------------------------------- +// A typed value and the same value computed are equally exact. + +fn small_decimal() -> impl Strategy { + let mantissa = prop_oneof![-1000i128..1000, mantissa96()]; + (mantissa, 0u32..=28).prop_map(|(m, s)| Num::exact(Decimal::from_i128_with_scale(m, s))) +} + +type Mul<'a> = &'a dyn Fn(&Num, &Num) -> Num; + +fn check_typed_equals_computed(mul: Mul, deterministic: bool) -> Result<(), String> { + let seed = Just((dec("0.000000000000001"), dec("0.000000000000001"))); + run( + prop_oneof![1 => seed, 3 => (small_decimal(), small_decimal())], + 256, + deterministic, + |(a, b)| { + let v = q(&a).unwrap() * q(&b).unwrap(); + let ten = |k: u32| BigRational::from_integer(BigInt::from(10).pow(k)); + let k = (0..=60u32) + .find(|&k| (&v * ten(k)).is_integer()) + .expect("a decimal product"); + let text = format!("{}e-{k}", (&v * ten(k)).to_integer()); + let parsed = typed(&text); + let computed = mul(&a, &b); + prop_assert_eq!( + arm(&parsed), + arm(&computed), + "typed and computed differ: {} against {} * {}", + text, + a, + b + ); + prop_assert!( + parsed.cmp_exact(&computed) == Some(Ordering::Equal), + "typed and computed differ: {text} against {computed}" + ); + Ok(()) + }, + ) +} + +#[test] +fn a_typed_value_and_the_same_value_computed_are_equally_exact() { + check_typed_equals_computed(&|a, b| a * b, false).unwrap(); + let computed = &typed("1e-15") * &typed("1e-15"); + assert!(same(&typed("1e-30"), &computed), "{computed}"); +} + +/// Revision 6's rule: demote whenever the returned scale drops. +fn demote_on_the_hint(a: &Num, b: &Num) -> Num { + match (a.kind(), b.kind()) { + (NumRef::Exact(x), NumRef::Exact(y)) => match x.checked_mul(*y) { + Some(r) if r.scale() == x.scale() + y.scale() => Num::exact(r), + _ => Num::inexact(a.to_f64() * b.to_f64()), + }, + _ => a * b, + } +} + +#[test] +fn a_typed_value_and_the_same_value_computed_are_equally_exact_catches_demoting_on_the_hint() { + assert_caught( + check_typed_equals_computed(&demote_on_the_hint, true), + "typed and computed differ", + ); +} + +// --------------------------------------------------------------------------- +// Clause 4's stored scale, question 13 (b)'s widening, and the canonical text. + +#[test] +fn a_recomputed_result_keeps_the_stored_scale_clause_4_names() { + let product = &dec("1.2500000000000000") * &dec("4.0000000000000000"); + assert!( + same(&product, &dec("5.0000000000000000000000000000")), + "{product}" + ); + let common = &dec("12.50") + &dec("1.5"); + assert!(same(&common, &dec("14.00")), "{common}"); + let rounded = &dec("1000") + &dec("0.3333333333333333333333333333"); + assert!(matches!(rounded.kind(), NumRef::Fraction(_)), "{rounded}"); + assert!(same(&(&dec("-1.5") + &dec("1.5")), &dec("0.0"))); +} + +#[test] +fn an_intermediate_beyond_i128_comes_back_exact() { + let a = Num::ratio(10i128.pow(30), 7).unwrap(); + let wide = &Interim::from(a.clone()) * &Interim::from(a.clone()); + assert!( + wide.is_exact(), + "an intermediate is unbounded up to the ceiling" + ); + let back = wide.checked_div(&Interim::from(a.clone())).unwrap().store(); + assert!(same(&back, &a), "{back}"); + let stored = (&a * &a).to_f64(); + assert!(!(&a * &a).is_exact() && is_nearest(stored, &(q(&a).unwrap() * q(&a).unwrap()))); +} + +#[test] +fn the_canonical_text_reads_back_as_the_same_number() { + run(any_num(), 512, false, |n| { + let text = n.to_string(); + let back = Num::from_canonical(&text); + prop_assert!( + back.as_ref() + .is_some_and(|b| same(b, &n) || (n.to_f64().is_nan() && b.to_f64().is_nan())), + "{text}" + ); + Ok(()) + }) + .unwrap(); + assert_eq!(dec("12.50").to_string(), "12.50"); + assert_eq!(Num::ratio(-10, 3).unwrap().to_string(), "-10/3"); + assert_eq!(Num::inexact(1.0 / 3.0).to_string(), "#i0.3333333333333333"); + assert_eq!(Num::inexact(f64::NEG_INFINITY).to_string(), "-inf"); + assert!(Num::from_canonical("#inan").is_none()); + assert!( + Num::from_canonical("1e-999").is_none(), + "canonical exact text never reads inexact" + ); + let ten35 = Num::ratio(10i128.pow(35), 1).unwrap(); + assert_eq!(ten35.to_string(), format!("{}/1", 10i128.pow(35))); +} From edbb50c42603c669a5a920ddd4a7bf7ed1382f72 Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 22:07:44 -0700 Subject: [PATCH 055/106] feat(numtower): benchmark, wasm CI entry, and the reasoning and evidence docs (improvise-59nw.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/numtower.md | 284 ++++++++++++++++++++++++++ context/test-evidence/numtower.md | 302 ++++++++++++++++++++++++++++ crates/numtower/examples/bench.rs | 188 ++++++++++++++++- crates/numtower/src/sum.rs | 26 ++- crates/numtower/tests/properties.rs | 2 + nix/ci.nix | 2 + scripts/build-doc-site.mjs | 2 + 7 files changed, 796 insertions(+), 10 deletions(-) create mode 100644 context/code/numtower.md create mode 100644 context/test-evidence/numtower.md diff --git a/context/code/numtower.md b/context/code/numtower.md new file mode 100644 index 00000000..cb794e83 --- /dev/null +++ b/context/code/numtower.md @@ -0,0 +1,284 @@ +# `numtower`: the reasoning + +`crates/numtower` is ADR 005 migration step 1 (improvise-59nw.1): the numeric tower, +first and alone. Nothing in improvise depends on it yet, so it changes no behaviour. +The decisions are the ADR's (`context/adr/005-exact-decimal-arithmetic.md`). This file +records how the crate reads them, the places the ADR left a choice open or said two +things, and the measurements. The evidence for each test is in +`context/test-evidence/numtower.md`. + +## The shape + +| type | what it is | ADR | +|---|---|---| +| `Num` | the stored value: an exact `Decimal`, an exact `Ratio`, or an `f64` | clauses 1, 2 | +| `NumRef` | a borrowed view of a `Num`'s arm, for exhaustive matching | — | +| `Interim` | a value inside one evaluation; exact intermediates widen to a bigint rational | question 13 (b) | +| `NumSum` | a retractable, order-free `SUM` over any `Num` | clause 8, question 16 | +| `ExactSum` | ADR 009's f64 superaccumulator | question 15 (a) | +| `parse_number`, `is_number`, `Typing` | clause 3's number language, gate and parse | clause 3, question 12 | +| `Display` / `Num::from_canonical` | the canonical text: `12.50`, `10/3`, `#i0.1`, `inf`, `-inf`, `nan` | clauses 10, 11, 14 | +| `Num::round_to_places` | display rounding, the mode a parameter | clause 12 "Format" | + +**`Num` is a struct with private arms, not the ADR's `pub enum`.** The ADR's sketch +(clause 2) shows `pub enum Num { Exact(Decimal), Fraction(Box>), +Inexact(f64) }`. A public enum would let any caller build values the tower never +produces: `Fraction(1/2)` (which must be the decimal `0.5`), an unreduced fraction, or an +exact `-0`. Each would break exact equality or the fingerprint of clause 9. So the arms +are private, every constructor normalizes, and callers match on `NumRef`. That is +`context/design-principles.md` §3, smart constructors. It is also what clause 14's table +asks for, since improvise calls "`Num`'s accessors for exactness, mantissa, scale and +fraction parts". The `Box` lives in one constructor, as `AGENTS.md` asks. `Num` is 24 +bytes, as the ADR measured, and `a_num_is_24_bytes` pins it. + +**The normal form.** An exact zero has no sign. A fraction is reduced and has a positive +denominator. A value whose reduced denominator is `2^p·5^q`, with `k = max(p, q) <= 28` +and a mantissa that fits 96 bits, is always the `Decimal`, never a `Fraction`. So +`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. + +## Arithmetic + +### Why there are two types: `Num` and `Interim` + +Question 13 (b) bounds only the **stored** result: "every intermediate inside one formula +evaluation is unbounded up to the ceiling". The bounded `Num` cannot be such an +intermediate, so `Interim` is the widened type. It is either a stored `Num` or an exact +`BigQ` (a reduced `BigInt` fraction) that does not fit `i128`. `Interim::store` applies +the stored bound. The operators on `&Num` are one-operation evaluations: each builds the +`Interim`, applies the operation and stores the result. So `(a * a) / a` returns +`a` exactly when computed through `Interim`, and is inexact when computed through `Num` +(`an_intermediate_beyond_i128_comes_back_exact`). That difference is (b)'s whole point. +Step 3's evaluators should evaluate through `Interim` and store once per cell. + +### Contagion, and how many roundings a demotion has + +An inexact operand makes the result inexact. Each exact operand is converted by the +correctly rounded conversion, and then the f64 operation runs. That is two roundings, +which is clause 4's "computed in f64". + +**An exact ∘ exact result that does not fit is rounded once.** It is the correctly rounded +f64 of the exact value. Clause 4 still says a demoted sum or product is "rounded twice, +once per operand and once in the operation". That text predates option (d). Option (d)'s +"Operations" and question 13 (b) both say that a result failing the stored bound is "the +correctly rounded f64 of the exact fraction", and under (b) the exact value is always in +hand. The crate follows (d), because (d) is decided and later. Both texts cannot hold for +one operation, so this is a reading, and it is flagged. + +### Clause 4: the hint, the recompute, the stored scale + +`Decimal + Decimal` and `Decimal × Decimal` first try `rust_decimal`'s checked operation. +The result is trusted only when its scale equals the nominal one (`max(sa, sb)` for `+` +and `−`, `sa + sb` for `×`). A lower scale is the **hint**, and so is a `None`. On the +hint the exact integer result is recomputed in a 256-bit integer, and its trailing zeros +are stripped. If the stripped mantissa fits 96 bits at a scale of at most 28, the result +is a decimal at the **stored scale**, `min(nominal, 28)`, padded back with zeros while the +mantissa fits. Otherwise the result goes to the fraction arm, then to a wide `Interim`. +The ADR's own example holds: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28 +(`a_recomputed_result_keeps_the_stored_scale_clause_4_names`). A `None` counts as the +hint, so that an overflow `rust_decimal` reports is never mistaken for an exact result. + +### Division: the direct test, one implementation + +Clause 5 is normative: write the quotient as a reduced `n / (2^p·5^q)`, then it is a +decimal iff `k <= 28` and the padded mantissa fits. The crate does this the same way for +every exact pairing, not only for decimals. It reduces the quotient (in `i128` when that +cannot overflow, and in a bigint otherwise), then normalizes. `small_ratio` and +`decimal_of` are the one normalization, so a decimal reached through any path has the +same representation. Multiplying back is never used. + +**A quotient carries its minimal scale** (`10.00 / 2` is `5`, `1 / 4` is `0.25`). The ADR +does not say what scale a quotient has. A quotient's scale is not a function of its +operands' scales (clause 5), and a result that passed through the fraction arm carries +its minimal scale (option (d)), so the minimal scale is the consistent choice. It is a +reading, and it is flagged. + +**The ADR's argument against multiplying back is weaker than it says.** Clause 5 rejects +it as incomplete, on `1e-28 / 0.5`: `q × 0.5` "needs scale 29, so the check flags it". +That holds under revision 6's rule, which demotes on the hint +(`division_decides_fit_directly_catches_multiplying_back` keeps that rival). But under +revision 7's recompute the product `1e-28` is found exact after its zeros are stripped. +So multiplying back by `Num`'s own `*` accepts the ADR's counterexample. In 20 runs of the +division property (5,120 random cases plus the three seeds), it agreed with the direct +test every time. That was an experiment and is not committed. The direct test stays the +rule, because the ADR decided it and because it needs no second division. + +### `POW` + +- **The exponent.** It is exact when it is an exact **integer value**, so `2.0` counts. +- **The size, decided before any multiplication.** The result's numerator has at least + `|n|·(bits(num) − 1)` bits, and likewise its denominator. If either lower bound passes + the ceiling, the result is `powf`, inexact, with nothing allocated. So + `POW(1/3, 10^9)` costs nothing. Inside the bound the result is computed exactly, and it + can still exceed the ceiling (`3^6000` is 9,510 bits). It then demotes, which is the + ceiling's stated exception. +- **The scale.** A positive power of a decimal takes clause 4's stored scale with nominal + `s·n`, the scale repeated multiplication would give, so `1.50^2` is `2.2500`. A + negative power is a division and has its minimal scale. +- **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`, + including one too large to compute. + +### The ceiling + +`CEILING_BITS = 8,192`. It applies to the bit length of an intermediate's numerator *or* +denominator. The ADR states the ceiling as a width, not as which of the two it measures. +Past it, the intermediate becomes the correctly rounded f64 of its exact value. + +## Comparison + +- **Exact with exact.** Cross-multiplication in 256 bits. A decimal is `m / 10^s` with + `10^28 < 2^94`, so every product fits. +- **Exact with inexact.** The ADR's method: convert the exact operand, and the rounded + value decides unless it lands on the f64 itself. A tie is resolved against the f64's + exact binary value, in a bigint. +- **That fast path is sound only because the conversion is correctly rounded.** Injecting + `rust_decimal`'s `to_f64` failed `exact_mixed_comparison_is_transitive` as well as the + conversion test. A future integer conversion (clause 4 allows one) must keep that. + +`MIN` and `MAX` skip NaN. Equal values break their tie as follows. Only the first rule is +the ADR's (question 11); the other two are this crate's, and they make the result +independent of argument order: +- the exact value is preferred to the inexact one (question 11); +- of two equal decimals, the larger scale is kept, for the same reason: it keeps more of + what was typed; +- of `0.0` and `-0.0`, `MIN` takes `-0.0` and `MAX` takes `0.0`, as IEEE 754's + `minimum` and `maximum` do. + +## The number language + +**The gate is literally `f64::from_str` reading a finite value.** The parse is the +crate's own lexer, and it never goes through an f64 on an exact path. The property test +checks that the two agree, and that the exact value read rounds to the f64 +`f64::from_str` reads. + +- **A lexeme's scale.** It is the scale the lexeme implies (`1.50` → 2, `1.5e3` → 0). When + the implied scale is above 28 but the value fits a decimal (`0.1` written with 29 + places), clause 4's stored-scale rule applies: strip the zeros, then pad toward 28. + The ADR only says "with the scale the lexeme implies", which cannot exceed 28. +- **Zero.** It is exact, unsigned, at the implied scale clamped to 0–28, so + `0e99999999999999999999` is `0`. +- **An exponent too long to read** saturates, because no exact arm can reach it. A + significand of more than 180 digits, or a leading digit outside 10^-130 to 10^40, goes + straight to f64. Those bounds are wider than the fraction arm can reach (a reduced + denominator `10^k / 2^a` is at least `2^k`, so `k < 127`). +- **The forced `n/d` arm.** Its value is a decimal, then a fraction, then **the nearest + f64**. The ADR says `n/d` "is a fraction or, once normalized, a decimal", and does not + say what a forced `n/d` too large for `i128` becomes. The crate applies clause 3's order + to it, as it does to every other spelled number. This is a reading, and it is flagged. + +**The canonical text is `Display`.** An inexact finite value is `#i` followed by Rust's +`{:?}` spelling, which is the shortest that reads back (`#i1e300`, `#i0.1`). +`from_canonical` reads all six spellings back, and reads an exact spelling only as an +exact value. Clause 14's list puts "`#i…`" in the crate, while its table says "the `#i` +prefix and the grammar are improvise's". Those two contradict each other. The crate +holds the prefix, because the wire and the file both need the one spelling, and step 7 +may move the prefix out if the grammar wants it. + +## Rounding to places + +Every arm goes through one exact path: the value as a bigint fraction, +`n·10^places / d`, then rounded by the caller's mode (`HalfAwayFromZero` or `HalfEven`). +An inexact value is first read exactly from its shortest round-trip spelling, which is +improvise-felo.9's fix, so a typed-then-computed `1.005` shows `1.01`. A result that +rounds to zero has no sign (`-0.001` at two places is `0.00`). The ADR does not decide +that, and today's f64 path would print `-0.00`. + +## Sums + +`NumSum` keeps four things, and every one is an abelian group, so the state is a function +of the multiset of terms and not of their order. + +| part | holds | width | +|---|---|---| +| the decimal partition | every decimal, and every fraction whose denominator is `2^p·5^q` with `k <= 28`, as an integer at scale 28 | 320 bits | +| the groups | per other reduced denominator, the summed numerator | 256 bits each, in a `BTreeMap` | +| `group_floats` | an `ExactSum` of each group's correctly rounded value | ADR 009's | +| `inexact` | an `ExactSum` of the f64 terms, plus their net weight | ADR 009's | + +**The read.** It is exact when the net inexact weight is zero and at most K = 8 groups +remain. The groups and the decimal partition are then combined **over the lcm of their +denominators, and reduced once**, and the stored bound applies to the result. Past K the +read is the f64 accumulators: `group_floats`, the decimal partition converted once, and +the inexact terms. That reading rounds each group, then the total. With inexact terms and +at most K groups, the exact total is converted once and added to the inexact terms. A +group whose numerator returns to zero is dropped, so "how many groups" is a function of +state. The K cutoff overrides representability: nine denominators whose exact total fits +still read inexact (`more_than_k_denominators_read_inexact_even_when_the_total_fits`). +That is question 13's second exception, and the ADR's flagged reading. + +**`ExactSum` is ADR 009's prototype**, from `crates/improvise-core/examples/retraction_drift.rs` +on PR #309. It has three changes: +- it splits a weight into base-2^30 digits, as ADR 009 says the `numtower` version must, + instead of looping per chunk; +- it holds 70 limbs, not 67, because a digit's shift adds up to 60 bits; +- it reads through the crate's one rational-to-f64 rounding. + +It counts NaN and the infinities, so they can be retracted. A count that is nonzero in +either direction is treated as present. + +**For ADR 009's seam crate.** `NumSum` and `ExactSum` both have the shape a Z-set group +needs: `add(&Num, i64)` / `add(f64, i64)`, `insert`, `retract`, `read`/`value`, `Clone`. +`ExactSum` also has `absorb`, to merge two accumulators. Neither implements a group trait +yet, because `zset` does not exist yet. The newtype of question 15 is where that goes. + +## The wide integers + +`Fixed` is a two's-complement integer of `N` 64-bit limbs that never allocates. It is +used for the clause 4 recompute (256 bits), exact comparison (256 bits), the decimal +partition (320 bits) and the group numerators (256 bits). Clause 14 also names "256-bit +helpers for the division reduction". Division reduces in `i128` when the products cannot +overflow, and otherwise in a bigint. It does not have a hand-rolled 256-bit gcd. That +path is the slow path, and it is measured below. + +## Dependencies + +The normal edges are `rust_decimal` 1.43.0 (std only), `num-rational` 0.4.2 (no default +features), `num-bigint` 0.4.8, `num-integer` 0.1.47 and `num-traits` 0.2.19. The last two +are direct dependencies, used for `gcd`/`lcm` and the numeric traits. They were already +in the tree through `num-rational` and `num-bigint`, so they add no crate. The ADR names +only the first three. There is no serde, and `cargo tree --edges normal` shows no +`borsh`. `BigRational` is a **dev**-dependency feature only, for the oracle, so the +oracle and the implementation do not share a rational type. + +## Measured + +On an Apple M4 Max (64 GB), rustc 1.98.1, `cargo run --release -p numtower --example +bench`. Each range is two runs. + +| what | figure | +|---|---| +| SUM, 18k and 90k typed two-place values, ns per element | f64 0.53; `rust_decimal` fold 2.7–3.0; `Num` `+` fold 7.8–8.2; `NumSum` insert and read 7.9–8.1 | +| per operation over 18,000 typed pairs, ns | `Num` add 3.1–3.3, mul 3.0–3.2, div 146–153; `rust_decimal` checked add 1.4–1.5, mul 1.2, div 24.7–25.3 | +| `(Rev − Cost) / Rev`, ns per record | 151–153 | +| `to_f64` of a decimal (the text path), ns | 36.7–37.1 | +| `round_to_places(2)`, ns | 370–382 | +| `NumSum` read, K distinct 126-bit denominators, numerators of about 190 bits | K = 2: 5.2–5.4 µs; K = 8: 41.4–42.0 µs | +| `ExactSum`, 1M values | 1.78–1.81 ns per insert, 0.29 µs per read | + +Here is how these sit against the ADR: +- **The grouped read.** At K = 8 it is inside the ADR's 52–127 µs. The ADR measured + 250-bit numerators, and these are about 190 bits, because `NumSum` can only reach a + group's numerator through its public `add`. +- **`ExactSum`.** It agrees with ADR 009's 2.0 ns per update and 0.58 µs per read. +- **Division.** At about 150 ns it is above the ADR's *estimate* of 60–110 ns. Most + quotients of typed values do not terminate, so each one pays two `i128` gcds and the + `Box` of a `Fraction`. +- **The rest.** The ADR's whole-recompute measurement at 18k and 90k cells waits for + step 3, as the ADR says. + +**The wasm32 cost** comes from a probe outside the repository, built exactly as the +ADR's appendix builds one: a `cdylib`, `opt-level = "s"`, LTO, `wasm-opt -Oz +--enable-bulk-memory --enable-nontrapping-float-to-int`, `gzip -9`. The baseline already +parses and prints an f64. + +| probe | after `wasm-opt -Oz` | gzipped | +|---|---|---| +| f64 only | 48,331 B | 29,162 B | +| the ADR's five operations (parse, `+`, `−`, `/`, print) through `numtower` | +89,375 B | **+36,714 B** | +| plus `NumSum`, `cmp_exact` and `round_to_places` | +119,277 B | **+48,221 B** | + +The narrow figure confirms the ADR's estimate of "up to about +36 KB together". The full +surface is about +47 KB gzipped. That is what the bundle pays once step 4 sums through +the crate and step 6 rounds through it. It is more than the +15.4 KB the user answered +question 8 against, and the ADR says question 8 "reopens" if the figure changes the +answer. Neither figure is the real bundle, which shares allocator and formatting code +with everything else. diff --git a/context/test-evidence/numtower.md b/context/test-evidence/numtower.md new file mode 100644 index 00000000..7938728d --- /dev/null +++ b/context/test-evidence/numtower.md @@ -0,0 +1,302 @@ +# Test evidence: `numtower` + +What each test guards, which rival it was run against, and where that rival failed. The +entries are keyed by exact test name, and a rename orphans its entry, which +`scripts/check-comment-runs.py --check-evidence` catches. The reasoning these tests defend +is in `context/code/numtower.md`. The decisions are ADR 005's. + +**How the rivals are kept.** Every property in `crates/numtower/tests/properties.rs` is a +checker that takes the implementation as a parameter. The property's own test runs it on +the real code. A companion test, named `_catches_`, runs the same checker +on the rival, with a deterministic seed, and asserts two things: +- the run fails; +- the failure message carries **the property's own assertion text**, so a rival that + fails in setup, panics, or fails some other assertion does not count (`assert_caught` + in `tests/oracle/mod.rs`). + +So the rival is watched failing on every run of the suite, not once by hand. Two rivals +were also injected into the crate's source, and the run was recorded. Those runs are noted +under their entries. + +**The oracle** is `num-rational`'s `BigRational`, a dev-dependency feature. The crate +itself never uses `BigRational`, so the oracle shares no rational type with the code under +test. "Correctly rounded" is checked in the review's interval form (`is_nearest`). The +exact value must lie inside the round-to-nearest-even interval of the returned f64, with +the endpoints included only when the f64's mantissa is even. Comparing against +`BigRational::to_f64` would have been circular in the way ADR 005's option (d) section +describes. + +--- + +## `crates/numtower/tests/properties.rs` + +### an_exact_result_is_never_a_rounded_one + +Every exact pairing of arms and every operation (`+ − × ÷`) is checked against the oracle. +When the result claims to be exact, its value must be the exact value. The generator +spans 96-bit mantissas, scales 0–28 and `i128` fractions. A fixed case pins ADR 005's own +example, `1000 + 0.3333333333333333333333333333`. `rust_decimal` returns that at scale +25, rounded, and the crate must not. + +**Injected in the source.** In `arith.rs::decimal_sum`, the scale filter was deleted, so +`checked_add`'s result was trusted. Three tests failed: this one, +`contagion_is_total`, and `a_recomputed_result_keeps_the_stored_scale_clause_4_names`. +The revert was followed by a full rerun. + +### an_exact_result_is_never_a_rounded_one_catches_a_silently_rounding_add + +The rival is `rust_decimal`'s own `checked_add`, trusted as exact. That is the one +outcome ADR 005 clause 4 exists to prevent. + +### contagion_is_total + +Every pairing of the three arms, NaN and the infinities included, through `+ − × ÷`: +- an inexact operand makes the result inexact; +- an exact pair gives an exact result **iff** the oracle value fits a stored arm, and + otherwise the correctly rounded f64 of that value; +- division refuses iff the divisor is exactly zero, at either level. + +"Total" means no pairing is missing. A pairing the code never handled would panic, and the +runner reports a panic as a failure. + +### contagion_is_total_catches_an_integral_inexact_kept_exact + +The rival is a "harmless" promotion: an integral f64 is treated as exact, so `2 + 3.0` is +exact. It is caught on the seed pair `(2, Inexact(3.0))`. That pair is in the generator on +purpose. Integral f64s are rare among random floats, so without the seed the rival was +caught only by luck. + +### pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits + +Bases from every arm, exponents −6 to 6. An exact base gives an exact result iff the +oracle's power fits, and otherwise the correctly rounded value. `0^-n` refuses. + +### pow_decides_its_size_before_computing_and_honours_the_ceiling + +- `POW(1/3, 10^9)` returns at once, inexact. Its lower bound is past the ceiling, so + nothing is allocated. Without the size check the test does not finish. +- `(1/3)^6000` is computed, but is 9,510 bits and demotes. That is the ceiling's + exception. +- `1.05^28` is exact and `1.05^29` is not, the option (d) table's k = 29. +- Also pinned: `(-1.0)^(10^20)`, `0^0`, the scale of `1.50^2`, and that a non-integer + exponent is `powf`. + +### the_f64_conversion_agrees_with_parsing_the_decimal_text + +The generator spans **96-bit mantissas and scales 0–28**, as ADR 005 requires. The oracle +text is built digit by digit, not by `rust_decimal`'s `Display`, which is what the crate +itself calls. So the test does not compare the crate's path with itself. The review's +case `63294.607176721429540940386554` is pinned. + +**Injected in the source.** `num.rs::decimal_to_f64` was replaced with `rust_decimal`'s +`ToPrimitive::to_f64`, and three tests failed: +- this one, minimal case `-0.10806611601966961` at scale 17; +- `the_number_languages_gate_and_parse_agree`, on `92240.01133710762224`; +- `exact_mixed_comparison_is_transitive`. + +The third was not expected. The comparison's fast path decides by the rounded value, and +that is sound only if the rounding is correct. + +### the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_to_f64 + +The rival is `rust_decimal`'s `to_f64`, which is not correctly rounded (ADR 005, B2). The +test also asserts that the rival misses on the review's case, so the pinned case is known +to discriminate. + +### rust_decimal_to_f64_agrees_on_fifteen_digit_mantissas_so_a_narrow_generator_is_vacuous + +This test **passes the rival** on purpose. With mantissas of at most 15 digits, 4,096 +cases, `rust_decimal`'s `to_f64` agrees with the text every time, as ADR 005 says (200,000 +of 200,000). So a test whose generator stays in that range proves nothing about the +conversion. This test is why the generator above spans 96 bits, and it is what fails, and +should be re-read, if `rust_decimal` ever fixes its conversion. + +### a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval + +Reduced fractions with numerator and denominator widths from 20 to 126 bits, both signs: +the review's independent form. It also checks the product of two such fractions, which +reaches the bigint path and is stored. That covers `ratio_to_f64` on values wider than +`i128`. + +### a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval_catches_an_f64_division + +The rival is `n as f64 / d as f64`, which rounds three times. + +### exact_mixed_comparison_is_transitive + +The generator is a **crowd**: values crowded around one f64 `x`. +- `x` itself, and its two neighbours; +- the shortest decimal that spells `x`, and that decimal ± `k·10^-25`; +- a third of the shortest decimal. + +Every triple is checked for transitivity, and every pair for antisymmetry. Random values +would almost never put two exacts and an inexact on one f64, which is exactly where the +rival breaks. + +### exact_mixed_comparison_is_transitive_catches_comparing_through_f64 + +The rival is revision 1's rule: convert the exact operand to f64, then compare. It fails +as ADR 005 clause 6 says it does. `Exact(s) = Inexact(x)` and `Inexact(x) = Exact(s + +10^-25)`, while the two exacts differ. The marker is `comparison: not transitive`, so it +is the transitivity assertion that fails, not antisymmetry. + +### comparison_agrees_with_the_oracle_at_every_pairing_of_arms + +The differential half of comparison: the crowd, plus mixed random pools of decimals, +fractions and finite f64s, against `BigRational`'s order. NaN is unordered, and an +infinity orders past every exact value. + +### min_and_max_prefer_the_exact_value_and_do_not_depend_on_order + +Question 11's tie-break, plus the two the crate adds: the larger scale between equal +decimals, and IEEE's `-0.0` for `MIN` and `0.0` for `MAX`. Each pair is asserted in both +argument orders, the order-independence property the tie-breaks exist for. + +### the_accumulator_is_independent_of_order + +Random term lists, from every arm, are summed in one order and in a shuffled order. The +results must agree in **arm, value and scale**. The terms include fractions over a small +pool of denominators, so the exact grouped read is reached, and arbitrary fractions, so +the K cutoff is too. ADR 005 clause 8's seed case, `[0.30000000000000004, 1e12, −1e12]`, +is one of the generator's arms and is also pinned exactly. + +### the_accumulator_is_independent_of_order_catches_decimal_addition + +The rival is clause 8's own rival, a fold with `Decimal`'s `+`, demoting to the tower +where it overflows. On the seed case, one order rounds `0.30000000000000004 + 1e12` to +scale 16 and the other does not. + +### the_accumulator_retracts_exactly + +Inserting `xs` and `ys`, then retracting `ys` in reverse order, must equal a fresh sum of +`xs` in arm, value and scale. A group whose numerator returns to zero must be dropped, or +a retracted denominator would still count toward K. + +### the_accumulator_reads_the_exact_sum_or_its_stated_roundings + +This is the differential test of the read against the oracle. It is the only test that +states every branch. +- **No inexact term, at most K live groups.** The read is exact iff the exact total fits, + and is then that total. Otherwise it is the total's correctly rounded f64. +- **Inexact terms present.** The exact total is rounded once, then the f64s are added and + the result is rounded once more. +- **Past K groups.** Inexact. + +The oracle counts **live** groups, those whose summed numerator is nonzero. The first +version counted denominators ever seen, and that disagrees with the state-based count +whenever terms cancel. + +### more_than_k_denominators_read_inexact_even_when_the_total_fits + +`1/3 + 1/7 + … + 1/31` is nine distinct denominators. The exact total fits an `i128` +fraction, yet the read is inexact. That is question 13's second exception, the K cutoff +overriding representability. Eight of them read as an exact `Fraction`. The read past K is +asserted to be **each group rounded, then the total rounded once**. The first version +asserted one rounding of the exact total, and it failed: clause 8's maintained f64 +accumulator of group values rounds twice by design. + +### exact_sum_is_the_exact_weighted_total_rounded_once + +ADR 009's `ExactSum` against the oracle. Weights are drawn from small values, from the +whole `i64` range and up to 2^40. A whole-range weight exercises all three base-2^30 +digits, which ADR 009's chunk loop could not reach in reasonable time. Values include +subnormals. + +### division_decides_fit_directly + +ADR 005 clause 5's three seed cases, as both property inputs and pinned assertions: +- `1 / 2^40` and `1 / 5^30` are fractions; +- `1e-28 / 0.5` is the exact decimal `2e-28`. + +Random decimal pairs must also be an exact decimal iff the quotient fits one, an exact +fraction iff it fits only that, and otherwise inexact. The test also pins `AVG`'s example +(`70/3`), `1/3 × 3 = 1`, and division by an inexact `-0.0`. + +### division_decides_fit_directly_catches_multiplying_back + +The rival is multiplying back with **revision 6's** scale test (demote on the hint). It +fails on `1e-28 / 0.5`, as ADR 005 says. An experiment, not committed, ran multiplying +back with `Num`'s own `*`, which carries revision 7's recompute. It agreed with the direct +test on all 5,120 random cases and the seeds. The ADR's argument that multiplying back is +incomplete holds only under the earlier rule (`context/code/numtower.md`, "Division"). + +### the_number_languages_gate_and_parse_agree + +The generated lexemes have four sources: +- structured lexemes: an optional sign, up to 40 digits, an optional fraction, and an + optional exponent of up to 4 digits, so the empty parts give `1.`, `.5` and `.`; +- junk over the alphabet of digits, `eE+-._/aifnIN` and space; +- `n/d` lexemes; +- the named spellings: `inf`, `NaN`, `Infinity`, `1_000`, `+.5`, `e5`. + +For each lexeme, the parse succeeds iff `f64::from_str` reads a finite value. The exact +value parsed must round to the f64 `f64::from_str` reads. `is_number` and `parse_number` +must agree in both typings. + +### the_number_languages_gate_and_parse_agree_catches_from_str_exact + +The rival is `rust_decimal`'s `from_str_exact`. ADR 005 clause 3 names it as the parse +whose language differs from the gate's: it rejects `1.5e3` and accepts `1_000`. + +### the_number_language_seed_cases + +Every seed case ADR 005's migration step 1 lists: +- `1.5e3` is the exact `1500`; +- `1_000` is text; +- `-0.0` is the exact `0.0`, with no sign; +- `3/4` is text when the type is inferred, and `0.75` when it is forced; +- `1e999`, `-1e999`, `nan`, `inf` and `Infinity` are text; +- `1e-999` is an inexact zero; +- `1e-30` is the exact `1/10^30`; +- `0.12345678901234567890123456789` is an exact fraction, not a rounded decimal; +- `2^120` is an exact `n/1`; +- `1e-40` is inexact. + +### a_typed_value_and_the_same_value_computed_are_equally_exact + +For random decimal pairs (and the seed `1e-15 × 1e-15`), the exact product is spelled as +`e-` and parsed. The parse must have the arm and the value of the computed +product. An earlier draft built the text only when the reduced denominator was a power of +ten, and rejected most cases. A reduced decimal product's denominator is `2^a·5^b`, so the +text now scales by the smallest power of ten that makes the product an integer. + +### a_typed_value_and_the_same_value_computed_are_equally_exact_catches_demoting_on_the_hint + +The rival is revision 6's multiplication, which demotes whenever the returned scale drops. +`1e-15 × 1e-15` then becomes an f64, while the typed `1e-30` is the exact fraction. That +is the asymmetry revision 6's N2 was about. + +### a_recomputed_result_keeps_the_stored_scale_clause_4_names + +ADR 005 clause 4's example: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28, +not scale 0 and not 32. The test also pins that a common-path sum keeps its returned scale +(`12.50 + 1.5 = 14.00`), that `1000 + 0.333…` goes to the fraction arm, and that an exact +zero result has no sign. + +### an_intermediate_beyond_i128_comes_back_exact + +The point of question 13 (b). `a = 10^30/7`; `a × a` does not fit `i128`, yet `(a × a) / +a` through `Interim` is `a`, exactly. The same product stored through `Num` is inexact +and correctly rounded. + +### the_canonical_text_reads_back_as_the_same_number + +Every arm's `Display` text reads back through `from_canonical` as the same arm, value and +scale. The test pins the four spellings ADR 005 clause 11 tabulates, and that canonical +exact text never reads back as an inexact value. + +--- + +## `crates/numtower/src/sum.rs` + +### sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once + +The combine must return the **lcm** as its denominator, and the reduced result must equal +the `BigRational` sum. Group numerators are up to `i128 × i64`, and the denominators share +factors (3, 77, 2^20, 3^40), so the lcm and the product differ. + +### sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once_catches_the_product + +The rival combines over the product of the denominators. Its value is right, so only the +lcm assertion can see it. That is why the property asserts the denominator before the +value. diff --git a/crates/numtower/examples/bench.rs b/crates/numtower/examples/bench.rs index f328e4d9..809a3be0 100644 --- a/crates/numtower/examples/bench.rs +++ b/crates/numtower/examples/bench.rs @@ -1 +1,187 @@ -fn main() {} +//! ADR 005 migration step 1's benchmark: `cargo run --release -p numtower --example bench`. +//! A microbenchmark, not the evaluator; what it does and does not show is in `context/code/numtower.md`. + +use std::hint::black_box; +use std::time::Instant; + +use numtower::{Decimal, ExactSum, Num, NumSum, Rounding, Typing, parse_number}; + +struct SplitMix64(u64); + +impl SplitMix64 { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } +} + +/// Typed two-place values from 0.01 to 999,999.99, as the ADR's appendix draws them. +fn typed_texts(seed: u64, n: usize) -> Vec { + let mut rng = SplitMix64(seed); + (0..n) + .map(|_| { + let cents = 1 + rng.next() % 99_999_999; + format!("{}.{:02}", cents / 100, cents % 100) + }) + .collect() +} + +fn nums(texts: &[String]) -> Vec { + texts + .iter() + .map(|t| parse_number(t, Typing::Inferred).unwrap()) + .collect() +} + +/// The best of `runs`, in nanoseconds per item. +fn best(runs: usize, items: usize, mut f: impl FnMut()) -> f64 { + (0..runs) + .map(|_| { + let t = Instant::now(); + f(); + t.elapsed().as_nanos() as f64 / items as f64 + }) + .fold(f64::INFINITY, f64::min) +} + +fn sums() { + println!("SUM over typed two-place values, ns per element, best of 20"); + for n in [18_000, 90_000] { + let texts = typed_texts(1, n); + let floats: Vec = texts.iter().map(|t| t.parse().unwrap()).collect(); + let decimals: Vec = texts.iter().map(|t| t.parse().unwrap()).collect(); + let values = nums(&texts); + let f64_fold = best(20, n, || { + black_box(black_box(&floats).iter().sum::()); + }); + let decimal_fold = best(20, n, || { + let s = black_box(&decimals) + .iter() + .fold(Decimal::ZERO, |a, b| a.checked_add(*b).unwrap()); + black_box(s); + }); + let num_fold = best(20, n, || { + let zero = Num::exact(Decimal::ZERO); + black_box(black_box(&values).iter().fold(zero, |a, b| &a + b)); + }); + let accumulator = best(20, n, || { + let mut s = NumSum::new(); + black_box(&values).iter().for_each(|v| s.insert(v)); + black_box(s.read()); + }); + println!( + " {n:>6} cells: f64 {f64_fold:.2}, rust_decimal {decimal_fold:.2}, Num + fold {num_fold:.2}, NumSum {accumulator:.2}" + ); + } +} + +fn operations() { + println!("Per operation over 18,000 typed pairs, ns, best of 20"); + let (a, b) = ( + nums(&typed_texts(21, 18_000)), + nums(&typed_texts(22, 18_000)), + ); + let decimal = + |v: &[Num]| -> Vec { v.iter().map(|n| n.to_string().parse().unwrap()).collect() }; + let (da, db) = (decimal(&a), decimal(&b)); + let pairs = a.len(); + let rd_add = best(20, pairs, || { + da.iter().zip(&db).for_each(|(x, y)| { + black_box(x.checked_add(*y)); + }) + }); + let rd_mul = best(20, pairs, || { + da.iter().zip(&db).for_each(|(x, y)| { + black_box(x.checked_mul(*y)); + }) + }); + let rd_div = best(20, pairs, || { + da.iter().zip(&db).for_each(|(x, y)| { + black_box(x.checked_div(*y)); + }) + }); + let add = best(20, pairs, || { + a.iter().zip(&b).for_each(|(x, y)| drop(black_box(x + y))) + }); + let mul = best(20, pairs, || { + a.iter().zip(&b).for_each(|(x, y)| drop(black_box(x * y))) + }); + let div = best(20, pairs, || { + a.iter() + .zip(&b) + .for_each(|(x, y)| drop(black_box(x.checked_div(y)))) + }); + let margin = best(20, pairs, || { + a.iter() + .zip(&b) + .for_each(|(rev, cost)| drop(black_box((rev - cost).checked_div(rev)))); + }); + let to_f64 = best(20, pairs, || { + a.iter().for_each(|x| { + black_box(x.to_f64()); + }) + }); + let places = best(20, pairs, || { + a.iter() + .for_each(|x| drop(black_box(x.round_to_places(2, Rounding::HalfAwayFromZero)))); + }); + println!(" rust_decimal checked: add {rd_add:.1}, mul {rd_mul:.1}, div {rd_div:.1}"); + println!(" Num: add {add:.1}, mul {mul:.1}, div {div:.1}"); + println!( + " Num (Rev - Cost) / Rev {margin:.1}, to_f64 {to_f64:.1}, round_to_places(2) {places:.1}" + ); +} + +/// K groups of near-`i128` denominators: the ADR's worst case for the exact read. +fn grouped_read() { + println!("NumSum read over K distinct 126-bit denominators, us per read, best of 200"); + for k in [2usize, 8] { + let mut rng = SplitMix64(0x5EED); + let mut sum = NumSum::new(); + for _ in 0..k { + let den = ((rng.next() as i128) << 62 | rng.next() as i128 >> 2) | 1 | (1 << 125); + let num = (rng.next() as i128) << 60 ^ rng.next() as i128; + let term = Num::ratio(num, den).unwrap(); + sum.add(&term, i64::MAX); + sum.add(&term, i64::MAX); + } + assert_eq!(sum.groups(), k); + let read = best(200, 1, || drop(black_box(sum.read()))); + println!( + " K = {k}: {:.1} us (exact: {})", + read / 1000.0, + sum.read().is_exact() + ); + } +} + +fn exact_sum() { + println!("ExactSum (f64 superaccumulator), best of 20"); + let floats: Vec = typed_texts(1, 1_000_000) + .iter() + .map(|t| t.parse().unwrap()) + .collect(); + let mut acc = ExactSum::new(); + let insert = best(20, floats.len(), || { + acc = ExactSum::new(); + floats.iter().for_each(|x| acc.add(*x, 1)); + }); + let read = best(20, 1, || { + black_box(acc.value()); + }); + println!( + " {insert:.2} ns per insert, {:.2} us per read", + read / 1000.0 + ); +} + +fn main() { + println!("numtower bench (release); name the machine when you quote these"); + sums(); + operations(); + grouped_read(); + exact_sum(); +} diff --git a/crates/numtower/src/sum.rs b/crates/numtower/src/sum.rs index 5276a50f..65401366 100644 --- a/crates/numtower/src/sum.rs +++ b/crates/numtower/src/sum.rs @@ -165,23 +165,31 @@ impl NumSum { /// Adds `x` with a Z-set `weight`. Exact and associative: the order cannot matter. pub fn add(&mut self, x: &Num, weight: i64) { - match (x.kind(), x.exact_parts()) { - (NumRef::Inexact(f), _) => { + match x.kind() { + NumRef::Inexact(f) => { self.inexact.add(f, weight); self.inexact_weight += weight as i128; } - (_, Some((n, d))) => match at_scale_28(d) { - Some(unit) => self.add_scaled(n, unit, weight), - None => self.add_to_group(n, d, weight), + NumRef::Exact(d) => { + let unit = pow10(MAX_SCALE - d.scale()) as u128; + self.add_scaled(d.mantissa(), unit, weight); + } + NumRef::Fraction(r) => match at_scale_28(*r.denom()) { + Some(unit) => self.add_scaled(*r.numer(), unit, weight), + None => self.add_to_group(*r.numer(), *r.denom(), weight), }, - (_, None) => unreachable!("an exact value has parts"), } } fn add_scaled(&mut self, n: i128, unit: u128, weight: i64) { - let term = I320::from_i128(n) - .checked_mul_u128(unit) - .and_then(|t| t.checked_mul_u128(weight.unsigned_abs() as u128)) + let magnitude = weight.unsigned_abs() as u128; + let scaled = match unit.checked_mul(magnitude) { + Some(both) => I320::from_i128(n).checked_mul_u128(both), + None => I320::from_i128(n) + .checked_mul_u128(unit) + .and_then(|t| t.checked_mul_u128(magnitude)), + }; + let term = scaled .and_then(|t| if weight < 0 { t.checked_neg() } else { Some(t) }) .expect("a weighted term is below 2^284"); self.decimals = self diff --git a/crates/numtower/tests/properties.rs b/crates/numtower/tests/properties.rs index 68cc76e5..43043a46 100644 --- a/crates/numtower/tests/properties.rs +++ b/crates/numtower/tests/properties.rs @@ -219,6 +219,8 @@ fn pow_decides_its_size_before_computing_and_honours_the_ceiling() { assert!(dec("0").pow(&dec("-1")).is_err()); assert!(!dec("2").pow(&dec("0.5")).unwrap().is_exact()); assert!(same(&dec("2").pow(&dec("-2")).unwrap(), &dec("0.25"))); + assert!(same(&dec("1.50").pow(&dec("2.0")).unwrap(), &dec("2.2500"))); + assert!(same(&dec("0").pow(&dec("0")).unwrap(), &dec("1"))); } // --------------------------------------------------------------------------- diff --git a/nix/ci.nix b/nix/ci.nix index ceb4f384..f8a56de9 100644 --- a/nix/ci.nix +++ b/nix/ci.nix @@ -46,6 +46,8 @@ "improvise-wasm-client" "improvise-worker-server" "improvise-gp" + # ADR 005 clause 14: nothing depends on it yet, and improvise-core will. + "numtower" ]; pFlags = crates: lib.concatMapStringsSep " " (c: "-p ${c}") crates; diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index 3ae5941d..3e7f5d87 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -63,6 +63,7 @@ const SECTIONS = [ ["context/code/tui.md", "code/tui", "The TUI"], ["context/code/cli.md", "code/cli", "The headless CLI"], ["context/code/gp.md", "code/gp", "gp-tree and improvise-gp"], + ["context/code/numtower.md", "code/numtower", "numtower, the numeric tower"], ["context/code/kb-tool.md", "code/kb-tool", "The kb tool"], ["context/code/tooling.md", "code/tooling", "The repository's own checkers"], ], @@ -81,6 +82,7 @@ const SECTIONS = [ ["context/test-evidence/tui.md", "test-evidence/tui", "The TUI"], ["context/test-evidence/cli.md", "test-evidence/cli", "The headless CLI"], ["context/test-evidence/gp.md", "test-evidence/gp", "gp-tree and improvise-gp"], + ["context/test-evidence/numtower.md", "test-evidence/numtower", "numtower"], ["context/test-evidence/tooling.md", "test-evidence/tooling", "The repository's own gates"], ["context/test-evidence/layout-cost.md", "test-evidence/layout-cost", "Layout and keypress cost"], ["context/test-evidence/scale-bisect.md", "test-evidence/scale-bisect", "Scale bisect"], From 2e5b0e92347b2d2862fa1c300f67ee9ae18151ac Mon Sep 17 00:00:00 2001 From: "Claude (rules-merge-main)" Date: Thu, 8 Oct 2026 22:08:02 -0700 Subject: [PATCH 056/106] docs(wave): rule 9 gates every merge of main and scopes stacked PRs Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- .agents/skills/wave/agent-rules.md | 37 +++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/.agents/skills/wave/agent-rules.md b/.agents/skills/wave/agent-rules.md index fc6c1756..444059f8 100644 --- a/.agents/skills/wave/agent-rules.md +++ b/.agents/skills/wave/agent-rules.md @@ -102,17 +102,38 @@ not in a second copy. where each came from). Close the old branch rather than force-pushing it: `AGENTS.md`, "Shipping a wave". 9. **Merge main as soon as you can, and keep merging it.** The user: - "Agents should merge main asap" and, asked the point, "Reduces conflicts" - (2026-10-09). `git fetch gh`, then merge the new `gh/main` SHA (rule 7's - re-pin, done on purpose) at the start of a task, before every push, and - every time you resume. Main only gets further from your base, and a - conflict met early is small. Never `origin/main`, never rebase, never - squash (rule 8). Commit the merge through `scripts/agent-git`. + "Agents should merge main asap" and, when asked why, "Reduces conflicts" + (2026-10-09 UTC; bd memory `agents-merge-main-asap`, written by the + orchestrator; PR #335). Those two quotations are the user's words. The + timing below is the orchestrator's operationalisation of them. + - **When.** `git fetch gh`, then merge the new `gh/main` SHA (rule 7's + re-pin, done on purpose) at the start of a task, before the first review + request, before every push to a PR under review, and every time you + resume. Main only gets further from your base, and a conflict met early + is small. Never `origin/main`, never rebase, never squash (rule 8). + Commit the merge through `scripts/agent-git`. + - **After every merge of main, clean or not**, re-run the gates (see + "Gates") on the merged tree, not on each side, and diff the test names + against your pre-merge HEAD to see that none was lost. A clean merge is + the dangerous case: it can delete a test module or break a shared test + file with no conflict marker. + - **Re-measure your baseline** after a mid-task merge, and report it + against the merged SHA, so main's changes are not read as yours. - **Resolve a conflict by hand.** For each conflicted file, diff the result against both parents and the merge base, so a side's non-conflicting changes are not lost to a whole-file `--ours` or - `--theirs`. Check that no test name was lost. Then re-run the gates - (see "Gates") on the merged tree, not on each side. + `--theirs`. + - **A stacked PR merges its base branch, not `gh/main`.** GitHub diffs it + against that base, so merging a `gh/main` ahead of the base puts main's + commits in its diff (the #145/#148 symptom, which rule 3's numstat + cannot see). The orchestrator retargets it to main once the base lands. + - **The exception is narrow.** A durability push of WIP (see "Commit + early") need not wait for a merge of main that conflicts. If you cannot + resolve the conflict yet, push the WIP to a separate wip branch, never + to the PR. + - **Each merge-and-push moves the head SHA** an in-flight review is aimed + at. Reviewers re-fetch the tip before posting, but do not merge main + only to push a reply. - The mechanics are `AGENTS.md`, "Shipping a wave" (merge `gh/main`, never `origin/main`) and "Merging many PRs at once" (diff each parent; two clean changes to one test file can still fail together). Not restated here. From 74efa481964e00f04f6eb186179ab04fec057ed7 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Thu, 8 Oct 2026 22:38:52 -0700 Subject: [PATCH 057/106] docs(adr): fold the user's nine answers on 068e4130 into ADR 007 revision 9 (improvise-cq1q.1) Answers 37-45 in a ninth round of "What the user decided". Q24 and Q25 are answered, slugify's steps are rewritten (whitespace to _, strip, collapse runs of _, trim leading and trailing _ and -), groups get ids (overriding answer 9), and Q23 is restated around "a formula is a derived measure" with a worked example. Q23 stays open; Q26-Q34 are new. core.md section 20 notes that slugify predates the proposed rules. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 727 ++++++++++++++++++++------- context/code/core.md | 17 +- 2 files changed, 559 insertions(+), 185 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index bc22f671..0f6dfd17 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -11,11 +11,16 @@ comments on its head `4aed8c07` and a clarification in chat (answers 33 and 34), its two later inline comments, also on `4aed8c07` (answers 35 and 36), and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 reverses the matching order that answer 31 recorded, and answer 34 overrides the ADR's own reading of "space". Answers 35 and 36 make measures the normal formula -target and give them an id/display split, which reopens Q23. It asks three new +target and give them an id/display split, which reopens Q23. It asked three new questions, Q23 (how a formula-target measure carries its id, and how a formula line -spells its target), Q24 and Q25 (two details of slugify's whitespace rule), and -flags eleven places where it states a reading of its own that is not a user answer, -listed under Q23. · +spells its target), Q24 and Q25 (two details of slugify's whitespace rule). +The user's nine inline comments on head `068e4130` are answers 37–45 (the ninth +round). **They answer Q24 and Q25**, rewrite slugify's steps, **give groups ids +(overriding answer 9)**, reject two of the ADR's flagged readings (the `_Index`/`_Dim` +one and the legal `-` slug) and override a third (`collapsed:`), and ask for Q23 to be +stated more clearly, which it now is. **Q23 stays open.** The ninth round adds nine +questions, Q26–Q34. The places where the ADR states a reading of its own that is not +a user answer are listed at the end of "Still open". · **Decided by:** the user; see "What the user decided" · **Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and @@ -117,7 +122,8 @@ exception to 'every reference is an id'"): > "This seems right" -Groups get no id in this ADR. +Groups get no id in this ADR. **Overridden by answer 44** (ninth round): a group +name is an id, and the friendly name is separate. **10. Q6, per-entity versions.** On line 469, Q6 ("Recommendation: not in this ADR"): @@ -231,7 +237,8 @@ run of spaces collapse to one `_`?"): This decides three things: - a run of spaces collapses to one `_`, as recommended (a run of whitespace, by - answer 34); + answer 34; answer 41 makes each whitespace character a `_` first and then collapses + runs of `_`); - a duplicate slug gets a numeric suffix after a `-`; - **a file has ids on everything or on nothing.** A new-version file with a declaration missing its id is refused, like any file that cannot be resaved safely (answer 11). @@ -260,6 +267,8 @@ capitalised displays"): The ids are `_index`, `_dim` and `_measure`, and the displays stay `_Index`, `_Dim` and `_Measure`. A leading `_` is reserved for them. Clause 4 says how slugify keeps a generated id out of that space, and clause 5 says what happens to a hand-written one. +Answer 42 widens the reservation to "virtual categories and measures and other +entities", and answer 43 restates `_index` and `_dim`. **24. Q21, an old file naming an undeclared item.** On line 708, option (a) ("Declare the item at load"): @@ -354,6 +363,8 @@ That item has its own `ItemId`, the formula has its own `FormulaId`, and by answer 22 the two differ. A formula line names such a target by the item's id. This was chosen over the recommended alternative, which was to give every formula target the `_Measure` treatment (no real item, handled by its `FormulaId`). +The user's comment on this sentence is answer 37: a formula id that would collide +takes the counter's suffix. **31. A typed name that is both one thing's display and another thing's id** (improvise-m56i item 6): @@ -365,7 +376,8 @@ comment on this answer (answer 33): until autocomplete exists, a typed name is matched against ids first, then against displays.** What stands from answer 31 is that the client holds the resolution (answer 32), and that a display matching more than one thing is refused, with the refusal listing the candidates' ids. Where -answers 31 and 33 differ, answer 33 governs. +answers 31 and 33 differ, answer 33 governs. The user's comment on this sentence is +answer 38: ids "should always be preferred when there's ambiguity". **32. Which side holds clause 12's capture** (improvise-hexb item 2), as recorded: @@ -374,7 +386,8 @@ answers 31 and 33 differ, answer 33 governs. > projection; the wire carries only id-form text; the host never resolves a > display name." -Clause 12 is rewritten to match. +Clause 12 is rewritten to match. The user confirmed this on the PR and widened it +from the formula editor to every reference (answer 39). ### Eighth round: answers on revision 9 (head `4aed8c07`) @@ -415,7 +428,9 @@ This **overrides the ADR's own reading** (revision 9's, which was not a user ans "Space" in answers 18 and 21 means whitespace, and a run of whitespace becomes one `_`. The user did not enumerate which characters count, so the definition in clause 4 is **the ADR's own reading of "whitespace"**, flagged there. Answers 18 and 21 -keep their wording, which this reads as "space, meaning whitespace". +keep their wording, which this reads as "space, meaning whitespace". **Answer 41 +refines the mechanism:** each whitespace character becomes `_`, and runs of `_` are +collapsed afterwards, so a run of whitespace still ends as one `_`. **35. Measures get the id/display split too.** On line 1091, the end of Q23's option (c), "Always the target's id, giving a `_Measure` target an id of its own that is @@ -454,6 +469,145 @@ things in earlier text meet them, and none is resolved here: revision 9 wrote the dimension target as the main case and the `_Measure` target as the exception. +### Ninth round: answers on revision 9 (head `068e4130`) + +The user answered revision 9 at `068e4130` with nine inline comments on PR #326, +posted 2026-10-09 05:17:38Z–05:26:39Z. The GitHub API gives all nine the original +commit `068e4130`. Each is mapped to the line it was attached to on `068e4130`, +quoted verbatim, and given the comment id. Where the ADR reads more into an answer +than its words say, the text says it is the ADR's reading and asks a numbered +question. + +**37. An id names one thing, and a colliding formula id takes the counter.** Comment +4226862456, on line 356, the last sentence of answer 30 ("This was chosen over the +recommended alternative, which was to give every formula target the `_Measure` +treatment …"): + +> "An id always refers to the same thing and so must be globally unique: if a formula has an id that would conflict, increment the counter to disambiguate it. Formulas may refer to anything in a typed context: e.g. a formula might count the number of times a particular letter appears in a category value." + +This decides: +- **A formula id that would collide is suffixed from the counter**, like any other + generated id. This restates answers 22 and 28 for the formula case, and clause 1's + example already follows it: a formula `Tax` in `CustomCat` gets item id `tax`, then + formula id `tax-`. +- **Formulas may refer to any kind of thing**, and the context of the reference + gives its type. This is why ids are unique across kinds (answer 22): a bare id in + formula text must name one thing whatever its kind. + +The example (count a letter in a category value) has no function in today's formula +language. Whether it is in this ADR's scope, what a reference may name, and whether +"a category value" means an item's display or its id are **Q26**. + +**38. Ids win under ambiguity.** Comment 4226867495, on line 368, the end of the +paragraph under answer 31 ("a display matching more than one thing is refused … +Where answers 31 and 33 differ, answer 33 governs"): + +> "IDs are unambiguous so they should always be preferred when there's ambiguity. " + +This confirms answer 33's ids-first order. The display-versus-display refusal is +untouched: no id is involved there, so there is nothing to prefer. Whether "always" +makes ids-first permanent rather than interim (clause 12 calls it interim until an +autocomplete exists) is **Q27**. + +**39. Display names live only in the client.** Comment 4226873976, on line 377, +"Clause 12 is rewritten to match.", under answer 32: + +> "Yes, the client should be the only place that allows any reference to something by a human-friendly name and this sort of reference should be minimized to the UI layer and converted to IDs ASAP" + +This confirms answer 32 and **widens it from the formula editor to every +reference**: only a client accepts a human-friendly name, it does so in the UI layer +only, and it converts the name to an id as early as it can. Clause 2's "resolved to +ids at the input edge" and clause 6's "every operand naming an existing thing is an +id" already say this. Two places in the ADR resolve a display outside a client: the +old-version loader (clause 5) and headless `improvise cmd`/`script`. Whether they +are exceptions is **Q28**. + +**40. A formula is a derived measure.** Comment 4226895692, on line 1303, the end of +Q23 ("Quoting the target position …"): + +> "Formulas are a derived measure. I don't really understand the problem here, this whole section needs to be stated more clearly." + +**Q23 is rewritten** to state the problem in those terms, with a worked example and +a plain split between what is decided and what is open. It stays **open**: the +answer says what a formula *is*, and the ADR does not read it as choosing an option. +"Still open" says why. + +**41. Whitespace becomes `_` first, then runs of `_` collapse.** Comment 4226897815, +on line 1316, Q24's "neighbouring edge", where `Time \u{200B} Period` slugged to +`time__period`: + +> "The spaces should have been normalized to _ before collapsing runs of _s" + +This **answers Q24 with option (b)**. It changes slugify's step 2 and adds a +collapse step (clause 4, steps 2 and 4): +- each whitespace character becomes `_`; +- **runs of `_` collapse to one `_`**, whether they came from whitespace or were + already in the display. So `a_ b` → `a_b` and `a__b` → `a_b`; +- `Time \u{200B} Period` → `time_period`. + +That last result needs the collapse to run **after** step 3 strips the zero-width +space. The answer says the collapse comes after the normalisation, and the example it +is attached to gives one `_` only if it also comes after the strip. So clause 4 puts +it there. That placement is the ADR's reading, and **Q29** asks the user to confirm +it. This overrides revision 9's reading and the code's comment in +`context/code/core.md` §20 ("no extra underscore-collapsing step exists"). + +**42. A leading `_` is reserved; a trailing one is dropped.** Comment 4226901843, on +line 1324, Q25's option (c): + +> "A leading _ is reserved for virtual categories and measures and other entities. A trailing one should be dropped to: disambiguate with -#" + +This **answers Q25 with option (b)**: slugify drops a trailing `_` as it drops a +leading one, so `East ` → `east` and `East_` → `east`. The ADR reads "to:" as "too;": +drop it too, and if the shorter slug then collides, disambiguate it with `-` +(answers 27 and 28). The reservation of a leading `_` now covers "virtual categories +and measures and other entities", which is wider than answer 23's three categories. +The ADR reads the virtual views `_records` and `_drill` (clause 1) as among the +"other entities". What a virtual *measure* is, and that reading, are **Q30**. + +**43. `_Index` and `_Dim` have ids.** Comment 4226904733, on line 1330, the flagged +reading "the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1)": + +> "No, they have ids _index and _dim " + +This **rejects the reading as it was worded**, and restates answer 23: the categories' +ids are `_index` and `_dim`. The reading was about their *coordinates*, the +synthetic positions in records mode (`_Dim` carries a column index, gotcha 1), not +the categories. Whether those positions are ids in the document's one namespace, and +how they would be spelled so that `_index`'s `0` does not collide with a user's item +`0`, is **Q31**. Today the repository spells the virtual categories `_Index`, `_Dim` +and `_Measure`, as keys, on the wire and in `.improv` files. What the move to lower +case ids costs is **Q32**. + +**44. Groups get ids.** Comment 4226908008, on line 1332, the flagged reading +"`collapsed:` is `(CategoryId, group name)` (clause 2)": + +> "Group name should be an id for the group, not a friendly name " + +This **overrides answer 9** ("not now") and the reading it was attached to. A group +has an id, and its friendly name is a separate display. A `collapsed:` entry names a +group by its id. Clause 1's table, clause 2 and clause 5 are changed to match. How a +group's id is made and spelled, and what the wire carries, are **Q33**. The ADR +does not decide them. +- **Relation, not a dependency.** PR #336 (improvise-cq1q.12, branch + `adr007/display-2b`, open, not on `main`) gives `Group` a private display beside + its key. On the ADR's reading, under answer 44 that key becomes the group's id, as + a category's does (clause 2). Nothing here depends on #336 landing. + +**45. A slug of `-` is not an id; no id begins or ends with `-`.** Comment +4226910518, on line 1334, the flagged reading "'whitespace' means the Unicode +`White_Space` property … and a slug of only `-` is a legal id (clause 4)": + +> "A slug of - should be invalid as an id and there should be no leading or trailing -s" + +This **rejects the second half of that reading**. Slugify removes leading and +trailing `-`, so `-` and `---` leave nothing and take `-` (step 6), and +`-East-` → `east`. The first half of the bullet, the meaning of "whitespace", was not +addressed and stays the ADR's flagged reading. That a **hand-written** id with a +leading or trailing `-` is refused at load, as one with a leading `_` is, and how the +trims interleave (`-_x` → `x`), are the ADR's reading of "invalid as an id" and are +**Q34**. + ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. @@ -514,15 +668,19 @@ Twelve clauses. | formula | yes | the document, across every kind (answer 22) | `FormulaId` | | view | yes (the user's example) | the document, across every kind (answer 22) | `ViewId` | | record | `RecordHash`, now over ids (clause 8) | the document | unchanged shape | -| group | **no**, not in this ADR (answer 9) | | | +| group | **yes** (answer 44, overriding answer 9) | the document, across every kind (the ADR's reading of answers 22 and 37; Q33) | to be decided (Q33) | -**Ids are unique in the whole document** (answer 22). One namespace covers categories, -items, formulas and views, so an item id names one item without its category. A +**Ids are unique in the whole document** (answers 22 and 37: "An id always refers to +the same thing and so must be globally unique"). One namespace covers categories, +items, formulas and views, and on the ADR's reading groups too (Q33), so an item id +names one item without its category. A `CellKey` still pairs each item with its category, because the key needs the categories. An operation that creates an item still names the parent category. -**Virtual categories have reserved ids** (answer 23): `_index`, `_dim` and `_measure`, -with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** +**Virtual categories have reserved ids** (answers 23 and 43): `_index`, `_dim` and +`_measure`, with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin +with `_`**: a leading `_` "is reserved for virtual categories and measures and other +entities" (answer 42). Moving today's spellings to these ids is Q32. **Measures are the normal formula target** (answers 35 and 36), and **a measure has an id and a display**. A stored measure is an ordinary item. Revision 9 says a `_Measure` formula target has no item of its own and that its handle is its @@ -531,21 +689,27 @@ the normal case. On revision 9's reading, this is why the category panel's `_Mea `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). **A formula target outside `_Measure`, the rare case (answer 36), is a real item** (answer 30). It has its own `ItemId`, and its formula has a separate `FormulaId`. Generated ids follow clause 4, so a -formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-`. +formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-` +(answer 37: "if a formula has an id that would conflict, increment the counter"). The panel lists that row as `ItemHandle::Item`, like any other item. -**The virtual views have reserved ids too** (the ADR's own reading, not a user -answer): `_records` and `_drill`, with the displays `_Records` and `_Drill`. They -are never persisted, but they are addressed on the wire (`SwitchView`, -improvise-obtu). Answer 23 reserves a leading `_` for the virtual *categories*, -and this extends it to the virtual views, because a slugified `_Drill` would -otherwise be `drill`, which could collide with a user's view (improvise-p003 item 1). +**The virtual views have reserved ids too**: `_records` and `_drill`, with the +displays `_Records` and `_Drill`. They are never persisted, but they are addressed on +the wire (`SwitchView`, improvise-obtu). Revision 9 introduced this as its own +reading, extending answer 23's reservation from the virtual categories to the +virtual views, because a slugified `_Drill` would otherwise be `drill`, which could +collide with a user's view (improvise-p003 item 1). Answer 42 now reserves a leading +`_` for "virtual categories and measures and other entities". **That the virtual +views are among the "other entities" is the ADR's reading** (Q30). -**The ADR's reading, not a user answer: `_Index` and `_Dim` coordinates are -outside the id namespace.** They are synthetic record positions (`_Dim` carries a -column index, gotcha 1). They are never persisted and never typed, so they take no -part in answer 22's uniqueness, and `_index`'s `0` cannot collide with a user item -whose id is `0` (improvise-p003 item 2). +**The `_Index` and `_Dim` categories have ids `_index` and `_dim`** (answers 23 and +43). Revision 9 read their *coordinates*, the synthetic record positions (`_Dim` +carries a column index, gotcha 1), as outside the id namespace, so that `_index`'s +`0` could not collide with a user item whose id is `0` (improvise-p003 item 2). **The +user rejected that reading as worded** (answer 43, "No, they have ids _index and +_dim"). The ADR does not read the answer as deciding what the positions are: that +is **Q31**. Until it is answered, nothing in this ADR depends on the positions' +spelling, because they are never persisted and never typed. ### 2. Storage keys by id; `display` is the only place a human name lives @@ -566,10 +730,12 @@ Following answers 1 to 3: wrote this pair as `(CategoryId, ItemId)` while also putting a `FormulaId` in it (Codex on PR #326). - **View configuration:** the axis maps, page selections, and hidden and collapsed sets - are keyed by ids. The one exception is a collapsed entry, which names a group: - groups have no id (answer 9), so it is `(CategoryId, group name)`, and - `collapsed:` writes the category id, then the group name (the ADR's own - reading, not a user answer; improvise-p003 item 4). + are keyed by ids, with no exception. A collapsed entry names a group, and a + group name is an id (answer 44), so the entry is the category id and the group's + id. Revision 9 had made it `(CategoryId, group name)`, because answer 9 gave groups + no id (improvise-p003 item 4); answer 44 overrides both. Whether the category id + is still needed beside a group id that is unique in the document, and how the + line is spelled, are part of Q33. - **Formulas:** a formula's references are ids (clause 5). The formula panel renders them as display names (answer 19; clause 6). @@ -615,63 +781,88 @@ Following answers 8, 16 and 18, **every** id the app generates comes from the di name by **slugify**. That covers import, interactive creation, and assigning ids to an id-less `.improv` file at load (answer 16(a), "All id generation"). So a thing's id never depends on how it was made. The steps, in order (answer 18, "casefold, replace -space with `_`, strip"): +space with `_`, strip", as answers 34, 41, 42 and 45 refine it): 1. **Casefold, with Unicode full case folding** (answers 16(b), 18 and 26). Rust's `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped), so this needs a case-folding table or a small crate in `improvise-core`. It must pass the `wasm` gate, because that crate is in the browser bundle. -2. **Replace each run of whitespace with one `_`** (answers 18, 21 and 34; answer 18 - changed answer 8's `-`). A run is any unbroken sequence of whitespace - characters, whatever mix, so it becomes one `_`, never one per character. +2. **Replace each whitespace character with `_`** (answers 18, 34 and 41; answer 18 + changed answer 8's `-`). "The spaces should have been normalized to _ before + collapsing runs of _s" (answer 41), so this step makes one `_` per character, and + step 4 does the collapsing. 3. **Strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`** (answers 16(c) and 16(d)). This strips non-ASCII letters, so an id never needs quoting in an `.improv` reference (clause 5 notes where formula text differs). -4. **Remove any leading `_`** (answer 23: no non-virtual id may begin with `_`). The - answer states the constraint. Removing the underscores is this ADR's way of meeting - it, so `_Draft` → `draft`. -5. **If nothing is left, use `-`** (answer 16(e)), such as `category-1` or +4. **Collapse each run of `_` to one `_`** (answers 21 and 41). It does not matter + whether an underscore came from whitespace or was in the display already. **The + step comes after the strip, which is the ADR's reading** (Q29): it is the order in + which answer 41's example, `Time \u{200B} Period`, gives one `_`. +5. **Remove every leading and trailing `_` and `-`**, until the slug neither begins + nor ends with either. + - A leading `_` "is reserved for virtual categories and measures and other + entities" (answers 23 and 42). Removing it is this ADR's way of meeting that + constraint, so `_Draft` → `draft`. + - A trailing `_` "should be dropped" (answer 42). + - "There should be no leading or trailing -s" (answer 45). + - Repeating the removal until neither character is left at either end (`-_x` → + `x`) is the ADR's reading (Q34). +6. **If nothing is left, use `-`** (answer 16(e)), such as `category-1` or `item-4`. `n` comes from the same counter as every other suffix (answer 28), not - from the lowest free number. + from the lowest free number. This covers a display that is only `-` or only `_`, + because "a slug of - should be invalid as an id" (answer 45). -Examples: `East` → `east`, `Time Period` → `time_period`, `Coastal_East` → -`coastal_east`, `Q1-2024` → `q1-2024`, `Region(code)` → `regioncode`, `Café` → `caf`, -`Ü` → `category-`, `Straße` → `strasse` (answer 26). `Time Period` and -`Time_Period` slug alike, so whichever comes second gets `time_period-`. `East` in -`Region` and `East` in `Direction` slug alike too, and since ids are global (answer -22), the second is `east-`. Here `` is the counter's next value, for example -`east-1`. +Examples: `East` → `east`, `Time Period` → `time_period`, `Time Period` → +`time_period`, `Coastal_East` → `coastal_east`, `Q1-2024` → `q1-2024`, +`Region(code)` → `regioncode`, `Café` → `caf`, `Ü` → `category-`, `Straße` → +`strasse` (answer 26). By answer 41, `a_ b` → `a_b`, `a__b` → `a_b` and +`Time \u{200B} Period` → `time_period`. By answer 42, `East ` → `east` and `East_` → +`east`. By answer 45, `-East-` → `east`, and `-` and `---` → `-`. Step 4 +collapses only `_`, so a `-` between two spaces survives: `Q1 - 2024` → +`q1_-_2024`. The user has not said otherwise, and this ADR adds no rule for it. -**Two readings of the steps, the ADR's own and not user answers.** The first is -the ADR's reading of the word "whitespace", which answer 34 requires and does not -define. The second was chosen by the implementation (PR #318; `context/code/core.md` -§20): +`Time Period` and `Time_Period` slug alike, so whichever comes second gets +`time_period-`. So do `East`, `East ` and `East_`, and the second and third each +take a suffix ("disambiguate with -#", answer 42). `East` in `Region` and `East` in +`Direction` slug alike too, and since ids are global (answers 22 and 37), the second is +`east-`. Here `` is the counter's next value, for example `east-1`. The suffix +never puts a `-` at either end, so a suffixed id still meets step 5. + +**The ADR's own reading of the word "whitespace"**, which answer 34 requires and does +not define. It is not a user answer. Revision 9 had a second reading here, that a slug +of only `-` is a legal id. Answer 45 rejects it, so step 5 removes it. - **"Whitespace" means the Unicode `White_Space` property**, the set Rust's `char::is_whitespace` tests. That is U+0009–U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. It needs no table in `improvise-core`. A tab, a no-break space (U+00A0) and an ideographic space (U+3000) are whitespace, so each becomes `_` like a space does: `Time\tPeriod`, `Time\u{A0}Period` and `Time Period` all slug to `time_period`, - and `Time \t Period`, one run of three characters, does too. A zero-width space + and so does `Time \t Period`, whose three `_` collapse at step 4. A zero-width space (U+200B) and a byte-order mark (U+FEFF) are **not** `White_Space`, so step 3 strips them. The realistic case is a header pasted from a web page, where a no-break space stands for a space. The ADR chose this set because it is the standard definition and the user asked for "whitespace", not for a list. The user may prefer a narrower one (ASCII whitespace only, say), and that is a one-line change. - An underscore the display already had is still never merged with the one a run - makes (`a_ b` → `a__b`), because the user said *consecutive spaces* collapse, not - consecutive underscores. That is **Q24**. A trailing run slugs to a trailing - `_` (`East ` → `east_`), which is **Q25**. -- **A slug of only `-` is a legal bare id.** `-` and `---` slug to themselves. - Step 5's "if nothing is left" means the empty string only. The id alphabet - admits them, so a line such as `-: row` is legal but hard to read. + Revision 9 asked here whether `a_ b` keeps two underscores (Q24) and whether a + trailing run leaves a trailing `_` (Q25). Answers 41 and 42 settle both, and steps + 4 and 5 follow them. **What this changes in built code (a follow-up, not made here).** PR #318's -`slugify` (`crates/improvise-core/src/model/id.rs`, `underscore_space_runs`) tests -`c == ' '`, and `context/code/core.md` §20 states the U+0020-only reading as a -rule. So today `Time\tPeriod` slugs to `timeperiod`. Answer 34 makes that wrong, and -the implementation needs a ticket to test whitespace instead and to repair §20 and -its tests. Nothing calls `slugify` yet (§20), so no stored id is affected. +`slugify` (`crates/improvise-core/src/model/id.rs`) and `context/code/core.md` §20 +predate answers 34, 41, 42 and 45, and differ from steps 2–6 in five ways: +- `underscore_space_runs` tests `c == ' '`, so `Time\tPeriod` slugs to + `timeperiod` (answer 34); +- it collapses runs of spaces before the strip and never collapses `_`, so `a_ b` → + `a__b` and `Time \u{200B} Period` → `time__period` (answer 41); +- it removes only leading `_`, so `East ` → `east_` (answer 42); +- it never removes `-`, so `-` slugs to `-` and `-East-` to `-east-` (answer 45); +- `IdAllocator::claim` accepts `-`, `-x` and `x-`, which Q34 would refuse. + +The tests `slugify_keeps_an_underscore_the_display_already_had` and +`slugify_collapses_a_run_of_spaces_to_one_underscore` assert the superseded results. +Answers 34, 41, 42 and 45 are all folded into revision 9, which the user has not yet +approved, so the follow-up should wait for that approval. Nothing calls `slugify` +yet (§20), so no stored id is affected. The remaining rules: @@ -679,7 +870,7 @@ The remaining rules: version, or a CSV or JSON import) gets every id from slugify. A new-version file must carry an id on every declaration, and one that does not is refused (clause 5). So no file mixes hand-written and generated ids. -- **Any colliding id gets a suffix, `gensym`-style** (answers 21, 27 and 28). +- **Any colliding id gets a suffix, `gensym`-style** (answers 21, 27, 28, 37 and 42). - **When:** whenever a generated slug is already taken, whatever kind of thing holds it. That applies on import, when an id-less file loads, and when the host creates something in a session. An id that does not collide stays bare. @@ -734,7 +925,13 @@ parenthesised run that contains a capital, such as `Sales(Copy)`, does not match `id_tag`, and neither does an empty or trailing-space tag. In a new-version file that declaration has no id, so the load is refused (answer 21), and the error names the line. Revision 4 generated an id and warned. A hand-written id beginning with `_`, -other than the three virtual ids, is refused too (answer 23). +other than the reserved virtual ids, is refused too (answers 23 and 42). A +hand-written id that is `-`, or that begins or ends with `-`, is refused the same +way. That is the ADR's reading of answer 45 ("a slug of - should be invalid as an id +and there should be no leading or trailing -s") for an id the program did not +generate (Q34). Whether a hand-written id ending in `_` is refused is also Q34: +answer 42 drops a trailing `_` from a slug, and does not say whether a written id may +carry one. **Inside formula text an id may still need quoting.** The formula grammar's `bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So @@ -760,8 +957,11 @@ id_ref = { id_token } ``` `model_name` has no id, so it keeps `header_name`. `Initial View:` becomes a reference -and takes an `id_ref`. A group reference (`[Coastal]`) stays a name while groups have no -id (answer 9). +and takes an `id_ref`. **A group reference (`[Coastal]`, and the group in a +`collapsed:` line) becomes an id** (answer 44, overriding answer 9's "stays a name"). +The grammar above does not yet spell it, and neither does the group definition +(`> Coastal`), which becomes a declaration carrying a display and an id. Those +spellings are Q33. **An axis line is tried before `format:`, and a format that reads as an axis line is pipe-quoted** (improvise-le4p; the ADR's own reading, not a user answer). @@ -823,7 +1023,8 @@ property 1 must be stated over the parsed result, not over the generated text. **Which regime a file is read under is decided by its version line alone** (clause 10). In an old-version file, a header is the whole line, a reference resolves against -display names, and ids are assigned by slugify. In a new-version file, declarations +display names, and ids are assigned by slugify. That resolution happens in the loader, +not in a client, which answer 39 may or may not allow (Q28). In a new-version file, declarations carry ids and a reference is an id. **An old file naming an item it never declares** declares it at load (answer 24). This @@ -874,6 +1075,12 @@ pipe-quotes any display name that would otherwise read as display plus id: - `AddItem { category: CategoryId, display }`, `AddItemInGroup { category: CategoryId, display, group }` and `AddFormula { target_category: CategoryId, raw }`. + - Since groups have ids (answer 44), `AddItemInGroup`'s `group` names an existing + group by its id, or creates one by its display. Which, or both, is Q33. +- **The client is the only place a display name is accepted as a reference** + (answers 32 and 39). It converts the name to an id as soon as it can, in the UI + layer, and the wire never carries a display as a reference. The one display an + operand carries is the name of the thing being created. - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. So a create-then-act compound becomes **one command** carrying both halves. `create-and-switch-view` sends `[CreateView, SwitchView]` today and @@ -1064,22 +1271,29 @@ is the capture-at-open rule the cell editor already follows: `OpenedCell` in generation and the view at open, and the commit answers for those rather than for whatever the document holds by then. - **The client holds the mapping** (answer 32), as it holds `OpenedCell`. The TUI - and the browser both use the same resolver in `improvise-protocol`. + and the browser both use the same resolver in `improvise-protocol`. The user + confirmed this and widened it: the client "should be the only place that allows any + reference to something by a human-friendly name", and converts it "to IDs ASAP" + (answer 39). - The mapping covers every display name the panel showed in that formula's text, including the target's. - A name the user types that was **not** in the captured mapping is new to the edit. The client resolves it against its own id-bearing projection (clause 6) at - commit, in answer 33's order: **ids first, then displays**. A typed name equal to + commit, in answer 33's order: **ids first, then displays** ("IDs are unambiguous so + they should always be preferred when there's ambiguity", answer 38). A typed name equal to an id names that id's thing, even where it is also another thing's display. Otherwise, if exactly one thing in scope has that display, it names that thing. If a display matches more than one thing, the edit is refused and the refusal lists the candidates' ids. If the name does not resolve at all, the edit is refused and names it. An id is matched exactly: no case folding. -- **The ids-first order is interim** (answer 33). The long-term design separates two - layers. A view-layer autocomplete of typed display names turns text into ids - before it is committed, and the model layer refers to ids only. When it exists, - this resolver's display stage and the ids-first order go with it. This ADR does not - specify the autocomplete. +- **The ids-first order was stated as interim** (answer 33). The long-term design + separates two layers. A view-layer autocomplete of typed display names turns text + into ids before it is committed, and the model layer refers to ids only. Revision 9 + read that, when the autocomplete exists, this resolver's display stage and the + ids-first order go with it. **Answer 38 says ids "should always be preferred when + there's ambiguity"**, so the preference itself is not temporary. Whether it carries + into the autocomplete (an id match offered first, say), or ends with this resolver, + is Q27. This ADR does not specify the autocomplete. - **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are in the file's id form, and the host never resolves a display name (answer 32). The host parses ids, and one that no longer exists is @@ -1180,9 +1394,11 @@ fdk1 landed in PR #221, before this ADR. What exists on main: The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14, revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All -of those are answered.** After acceptance, three implementation questions were -answered on their tickets (answers 30–32), and the user's comments on revision 9 are -answers 33 and 34, and its two later comments are answers 35 and 36. Revision 9 adds Q23 to Q25, which are open. +of those are answered**, and answer 44 later overrode Q5's answer. After acceptance, +three implementation questions were answered on their tickets (answers 30–32). The +user's comments on revision 9 are answers 33–36 (eighth round) and 37–45 (ninth +round). Revision 9 added Q23 to Q25: answers 41 and 42 settle Q24 and Q25, and Q23 is +still open. The ninth round adds Q26 to Q34, which are open. | # | question | my recommendation | status | |---|---|---|---| @@ -1190,7 +1406,7 @@ answers 33 and 34, and its two later comments are answers 35 and 36. Revision 9 | 2 | Formula id placement | F1, `- Profit(3) = …` | **answered: F1** (answer 5) | | 3 | Headers in old files | first draft: accept the rare break; revision 1: gate on version | **answered**: the space is optional (answer 6), and old files keep the whole-line reading (answer 15, Q12) | | 4 | Do `## Data` keys carry ids? | no | **answered: yes, ids** (answer 7), reversing the recommendation | -| 5 | Group ids? | not now | **answered: not now** (answer 9) | +| 5 | Group ids? | not now | **answered: not now** (answer 9). **Overridden: a group name is an id** (answer 44); the mechanism is Q33 | | 6 | Per-entity versions? | not in this ADR | **answered: not in this ADR** (answer 10) | | 7 | `RecordHash` over ids? | no, keep names | **answered: over ids** (answer 3), reversing the recommendation | | 8 | Duplicate id: repair or refuse? | revision 2: refuse | **answered: refuse the load** (answer 11) | @@ -1200,23 +1416,35 @@ answers 33 and 34, and its two later comments are answers 35 and 36. Revision 9 | 12 | Old files and the new header rule | (a) no | **answered: (a), with the version marked unambiguously** (answer 15) | | 13 | Slugify details | all generation; keep case; keep `_` `-`; strip non-ASCII; `-` | **answered** (answer 16): all generation, **lower case only** (reversing the recommendation), keep both, strip, `-` | | 14 | Formula text: ids or display names? | ids in the file | **answered**: ids in the file (answer 3); the panel is Q16 | -| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-`, and "space" means whitespace (answer 34) | +| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-`, "space" means whitespace (answer 34), and each whitespace character becomes `_` before runs of `_` collapse (answer 41) | | 16 | What the formula panel shows | none | **answered: display names, target shown as `Profit (profit)`, ids where a name is ambiguous** (answer 19) | | 17 | Capitals in a hand-written id | (a) no | **answered: no** (answer 20) | -| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse** (a run of whitespace, answer 34), plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | +| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse** (a run of whitespace, answer 34; by collapsing runs of `_`, answer 41), plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | | 18(b) | What "casefold" means | full case folding | **answered: full Unicode case folding** (answer 26) | -| 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answer 22), stronger than (a) | -| 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23) | +| 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answers 22 and 37), stronger than (a) | +| 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23); the reservation covers "virtual categories and measures and other entities" (answer 42), and `_index`/`_dim` are restated (answer 43) | | 21 | An old file naming an undeclared item | (a) declare it at load | **answered: (a)** (answer 24) | | — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal | | 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) | -| — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation | -| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31). **Order reversed: ids first until there is autocomplete** (answer 33) | -| — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32) | -| — | Whether a measure has an id/display split, and which formula targets are normal (user comments on revision 9) | — | **answered in principle**: measures have the split (answer 35), and the normal target is a measure (answers 35, 36). The mechanism is Q23 | -| 23 | How a formula-target measure carries its id, and how a formula line spells its target (revision 9; reworded after answers 35 and 36) | none until the user chooses | **open** | -| 24 | Does `a_ b` slug to `a__b`? (answer 34) | (a) keep | **open** | -| 25 | Does a trailing whitespace run slug to a trailing `_`? | (a) keep | **open** | +| — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation; a formula id that collides takes the counter's suffix (answer 37) | +| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31). **Order reversed: ids first until there is autocomplete** (answer 33). Ids are always preferred under ambiguity (answer 38); whether that outlives the interim resolver is Q27 | +| — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32), widened to every reference by a human-friendly name (answer 39) | +| — | Whether a measure has an id/display split, and which formula targets are normal (user comments on revision 9) | — | **answered in principle**: measures have the split (answer 35), the normal target is a measure (answers 35, 36), and a formula is a derived measure (answer 40). The mechanism is Q23 | +| — | Revision 9's reading that `_Index`/`_Dim` coordinates are outside the id namespace | (the ADR's reading) | **rejected as worded** (answer 43); what the positions are is Q31 | +| — | Revision 9's reading that `collapsed:` names a group by name | (the ADR's reading) | **overridden: a group name is an id** (answer 44) | +| — | Revision 9's reading that a slug of only `-` is a legal id | (the ADR's reading) | **rejected: `-` is not an id, and no id begins or ends with `-`** (answer 45) | +| 23 | Is a formula's derived measure one thing with one id, or two things with two ids? (revision 9; reworded after answers 35 and 36; restated after answer 40) | none until the user chooses | **open** | +| 24 | Does `a_ b` slug to `a__b`? (answer 34) | (a) keep | **answered: (b), runs of `_` collapse**, so `a_ b` → `a_b` (answer 41), not the recommendation | +| 25 | Does a trailing whitespace run slug to a trailing `_`? | (a) keep | **answered: (b), a trailing `_` is dropped** (answer 42), not the recommendation | +| 26 | What may a formula refer to, and what is "a category value"? (answer 37) | (iii) only keep ids unique | **open** | +| 27 | Does ids-first outlive the interim resolver? (answer 38) | none | **open** | +| 28 | May the old-version loader and headless `cmd`/`script` resolve displays? (answer 39) | yes, both | **open** | +| 29 | Does step 4's collapse come after the strip? (answer 41) | (a) after | **open** | +| 30 | Which "other entities" share the reserved `_`? (answer 42) | the virtual views are among them | **open** | +| 31 | Are `_Index`'s and `_Dim`'s positions ids? (answer 43) | none | **open** | +| 32 | Moving the virtual categories to lower-case ids (answers 23 and 43) | as written there | **open** | +| 33 | How groups carry ids (answer 44) | none, beyond the readings stated | **open** | +| 34 | Hand-written ids with `-` or a trailing `_`, and how the trims repeat (answer 45) | refuse `-` at either end; repeat the trims | **open** | ## Still open @@ -1228,120 +1456,251 @@ review were left as tickets rather than ADR text: improvise-le4p, improvise-m56i improvise-p003. Revision 9 folds all three in, and improvise-m56i's two questions are answers 30 and 31. -**Revision 9 needs the user's approval, and it asks three questions**, Q23 to Q25. +**Revision 9 needs the user's approval.** It asked Q23 to Q25, and the ninth round +answered Q24 and Q25. Q23 and the ninth round's Q26 to Q34 are open. -**Q23. How does a formula-target measure carry its id, and how does a formula line -spell its target?** (Reworded after answers 35 and 36. Revision 9 asked only how -the line spelled its target, and it had the dimension case in view.) +**Q23. A formula is a derived measure. Is the measure and its formula one thing with +one id, or two things with two ids?** (Restated after answer 40, which asked for this +question to be stated more clearly. Revision 9 asked how a formula line spells its +target; after answers 35 and 36 it asked how a formula-target measure carries its id. +It is the same question, stated in the user's terms.) -The user's words: measures have an id and a display, and measures are the normal -formula target. What is open is the *mechanism*. A stored measure is already an item -with both. A measure created by a formula (`profit`) has, in revision 9, no item: -its display lives on the `Formula`, and its id is the formula's own id. Clause 5 -therefore spells the line `- Profit() = … [_measure]` (the display bare) -for the normal case, and `- () = … []` for the rare -dimension case, so the two cases put different kinds of thing in one position. +**What is decided.** +- A formula is a derived measure (answer 40), and a measure is the normal formula + target (answers 35 and 36). +- A measure has an id and a display (answer 35). +- Every reference is an id (answer 3), and no two ids in a document are equal + (answers 22 and 37). +- A formula carries an id in its line, after the target and before the `=` + (answer 5), and `ReplaceFormula` and `RemoveFormula` address a formula by it + (clause 6). +- The rare dimension target is a real item with a separate formula id (answer 30), + and a formula id that collides takes the counter's suffix (answer 37). -- **(a) Keep the handle: the target's id is the formula's id.** The display lives on - the `Formula`. The id is generated from the target's display, and with no item to - collide with it comes out bare: `Profit` → `profit`, not `profit-`. - `- Profit(profit) = revenue - cost [_measure]`, and a dimension target is - `- tax(tax-1) = … [customcat]`. References to the measure are `profit`. This is - the smallest change. Its cost is that the normal case writes a display in a - position that holds an id for the rare one. Under (a) a measure does have an id - and a display; whether that is the split the user asks for is the thing to decide. -- **(b) Always a display, then ids in tags.** Measure: `- Profit(profit) = … [_measure]`. - Dimension: `- Tax(tax-1)(tax) = … [customcat]`. The target position never mixes - kinds. A dimension line gains a second tag. -- **(c) A formula-target measure is a real `_Measure` item**, like a dimension target - under answer 30: its own `ItemId` and display, and a separate `FormulaId` - (`profit`, then `profit-`). The target position is the item id in every case: - `- profit(profit-1) = revenue - cost [_measure]`. It removes `ItemHandle::FormulaTarget`, - returns `CellKey` to `(CategoryId, ItemId)`, and ends the `CellKey` typing problem - Codex raised on this PR. It reverses clause 1's "no item of its own" and revision - 9's readings in clauses 1, 2, 5 and 8. It also needs the item and its display - written somewhere: today only non-formula `_Measure` items persist, and formula - targets are rebuilt from `## Formulas`. +**The worked example.** A document has two stored measures, `Revenue` and `Cost`. +Each is an item of `_Measure`, with ids `revenue` and `cost`, and each holds data. +The user adds `Profit = Revenue - Cost`. `Profit` holds no data of its own: its +column is computed by the formula, which is what makes it a derived measure. Two +kinds of thing must then name Profit by an id: +1. **Things that refer to the measure.** Another formula (`Margin = Profit / + Revenue`), the `CellKey` of each Profit cell, a `hidden:` line, and ADR 008's + cursor anchor. +2. **Things that refer to the formula.** `ReplaceFormula` and `RemoveFormula`, which + edit or delete the formula. -Answers 5 and 19 picture the measure case as `Profit(3)` and `Profit (profit)`. -Answer 19's panel picture fits (a), (b) and (c): each gives the measure a display and -an id, and under (c) the `profit` is the item id. Answer 5's file-line picture, -`- Profit(3) = …`, fits (a) and (b) exactly, and (c) only loosely, because (c) writes -the item id in the target position where answer 5 shows the display. +**The question is whether 1 and 2 use the same id.** -**No recommendation is made until the user chooses.** Revision 9 recommended (a), -written when the dimension case was the one in view. Any ranking of how well an option -fits answer 35 would be the ADR's own judgement and not the user's, since (a) and (b) -also give a measure an id and a display. (a) is the smallest change. +- **(a) One thing, one id.** The formula *is* the measure. Profit's id is `profit` + and its display is `Profit`, both held on the formula. `Margin = profit / revenue`, + each Profit cell's key holds `profit`, and `ReplaceFormula` addresses `profit`. + The file line is `- Profit(profit) = revenue - cost`. No `_Measure` item exists for + Profit, so in code the column's coordinate is `ItemHandle::FormulaTarget(profit)` + (clause 2). This is revision 9's model, and the smallest change. +- **(c) Two things, two ids.** The measure is a `_Measure` item with id `profit` and + display `Profit`. The formula that computes it is a separate thing, with id + `profit-1`, suffixed because `profit` is taken (answer 37). `Margin = profit / + revenue` and each Profit cell's key holds `profit`, while `ReplaceFormula` + addresses `profit-1`. The file line is `- profit(profit-1) = revenue - cost`. This + is answer 30's model applied to measures too, so a measure target and a dimension + target work the same way. +- **(b) is not a third model.** It is (a)'s model, with the rare dimension line + spelled `- Tax(tax-1)(tax) = … [customcat]` (display, formula id, item id), so that + the target position always holds a display. Under (a) and (c) the dimension line + is `- tax(tax-1) = … [customcat]`, with the item id in the target position. -Four further points the choice must settle, all the ADR's own reading and none a -user answer: -- **A formula targeting a stored measure of the same name.** `measure_item_names` - merges them today, so a stored `Profit` column and a formula `Profit = …` are one - measure. Under (a) and (b) that one column has two ids (the item's `profit` and the - formula's `profit-`), and the ADR must say which one keys and references use. - Under (c) it is the natural case, and the formula attaches to the existing item. -- **Removing a formula under (c).** `RemoveFormula` drops the formula and leaves its - `_Measure` item, as it leaves a dimension target's item today (answer 30). The - measure column then has no formula and possibly no data. +**Where (a) and (c) differ.** + +| | (a) one id (and (b)) | (c) two ids | +|---|---|---| +| A Profit cell's coordinate | the formula's id, as `ItemHandle::FormulaTarget` | an ordinary item id | +| `CellKey` | `(CategoryId, ItemHandle)` | `(CategoryId, ItemId)`, and `ItemHandle` goes | +| A stored `Profit` column and a formula `Profit = …`, merged into one measure today by `measure_item_names` | one column with two ids (the item's `profit`, the formula's `profit-1`), and the ADR must say which keys the cells | the formula attaches to the existing item | +| `RemoveFormula` on Profit | the measure goes too | the item `profit` stays, with no formula and maybe no data, as a dimension target's item stays today | +| What the file must newly write | nothing: the formula line declares the measure | the item `profit` and its display (today a formula target is not written as an item, and is rebuilt from `## Formulas`) | +| A dimension target (answer 30) | works differently from a measure target | works the same way | +| ADR 008 (PR #308), which anchors the cursor by item identity | needs `ItemHandle` | works as it is | +| ADR 009 (PR #309), which uses `find_item_category`'s `_Measure` fallback for formula targets | keeps it | loses it | + +Neither ADR 008 nor ADR 009 is edited here. + +**What is open.** The choice between (a) and (c) and, under (a), whether the +dimension line is spelled (a)'s way or (b)'s. **No recommendation is made.** Answer +40, "Formulas are a derived measure", reads most literally as (a), where the formula +*is* the measure. But (c) also models a derived measure, an item whose values a +formula derives, and the answer was a request for clarity rather than a choice. So +reading it as choosing (a) would be the ADR's judgement, not the user's. + +Smaller points, each the ADR's own reading and none a user answer: - **The `[_measure]` tag.** Today's writer omits the tag for a `_Measure` formula (`- {raw}`), and the user's own example, `profit = revenue - cost`, has none. The - examples above write it only to show the category. Whether the tag is omitted for - `_measure` is not decided here. -- **Other ADRs.** ADR 008 (PR #308) anchors the cursor by item identity, which under - (a) or (b) a formula-measure column does not have, so it would need `ItemHandle`. - ADR 009 (PR #309) relies on `find_item_category`'s `_Measure` fallback for formula - targets, which (c) would remove. Neither ADR is edited here. - -Two consequences, both the ADR's own reading, for whichever is chosen: -- **References to a formula target** are the id text of its handle: under (a) and - (b) the item id outside `_Measure` and the formula id for a `_Measure` target; - under (c) always the target's item id. + examples above omit it too. Whether the new writer keeps omitting it is not decided + here. +- **Answers 5 and 19** picture the measure case as `Profit(3)` (the file line) and + `Profit (profit)` (the panel). The panel picture fits every option. The file-line + picture, a display before the id, fits (a) exactly and (c) only loosely, because + (c) writes the item id where answer 5 shows the display. +- **References to a formula target** are the id text of its handle: under (a) the + item id outside `_Measure` and the formula id for a `_Measure` target; under (c) + always the target's item id. - **Quoting the target position.** Where a display sits in the target position - (a, b), a display that looks like `Name(x)` would read back as a display and an id, - so the writer pipe-quotes it: `- |Margin(%)|(margin) = …`. An item id never - contains a parenthesis, so only a display needs it. (c) has no such position. + ((a) for a measure, and (b)), a display that looks like `Name(x)` would read back + as a display and an id, so the writer pipe-quotes it: `- |Margin(%)|(margin) = …`. + An item id never contains a parenthesis, so only a display needs it. (c) has no + such position. -**Q24. Does `a_ b` slug to `a__b`?** Answer 34 collapses a run of whitespace to one -`_`, and the ADR applies the steps as written, so an underscore the display already -had is not merged with the one a run makes. The user did not say. The author's -lean, not the user's, is (a): it is what the user's wording ("consecutive spaces") -and PR #318 give. -- **(a) Keep it.** `a_ b` → `a__b`, and a deliberate `a__b` stays. -- **(b) Collapse any run of whitespace and `_` to one `_`.** `a_ b` → `a_b`, and - `a__b` → `a_b`. That changes ids for displays that hold a double underscore. +**Q24. Does `a_ b` slug to `a__b`?** **Answered by answer 41: no.** Revision 9's +options were (a) keep it, its lean, and (b) collapse any run of whitespace and `_` +to one `_`. Answer 41, "The spaces should have been normalized to _ before +collapsing runs of _s", is (b): `a_ b` → `a_b` and `a__b` → `a_b`, so ids change for +displays that hold a double underscore. Its example, `Time \u{200B} Period`, now slugs +to `time_period`. Clause 4's steps 2 and 4 state it, and where step 4 goes is Q29. -A neighbouring edge: step 2 runs before step 3 strips non-alphabet characters, so -`Time \u{200B} Period` (a zero-width space between two spaces) slugs to -`time__period`, two runs and not one. +**Q25. Does a trailing run slug to a trailing `_`?** **Answered by answer 42: no.** +Revision 9's options were (a) keep it, its lean, (b) remove a trailing `_` as a +leading one is removed, and (c) trim whitespace before step 2. Answer 42 is (b): "A +trailing one should be dropped", so `East ` → `east` and `East_` → `east`, and a +slug that then collides takes the suffix. Clause 4's step 5 states it. -**Q25. Does a trailing run slug to a trailing `_`?** Step 4 removes leading `_` and -nothing removes trailing ones, so `East ` → `east_`. The user did not say. The -author's lean, not the user's, is (a). -- **(a) Keep it.** It is what PR #318 does. -- **(b) Remove trailing `_` as step 4 removes leading `_`.** `East ` → `east`, and - also `East_` → `east`. -- **(c) Trim whitespace before step 2.** `East ` → `east`, and `East_` stays `east_`. +**Q26. What may a formula refer to, and what is "a category value"?** Answer 37 says +"Formulas may refer to anything in a typed context", with the example of a formula +counting "the number of times a particular letter appears in a category value". +Today a formula reference (`Expr::Ref`) names an item or a formula target, and +`find_item_category` finds its category. The answer does not say: +- (i) whether a bare id in formula text may also name a category, a view or a group, + with its kind given by where it appears (an aggregate's argument, a `WHERE` clause, + a function's parameter); +- (ii) whether the text a function like that reads is the item's display (`Café`) + or its id (`caf`). The two differ, because slugify folds case and strips; +- (iii) whether this ADR must specify any of it, or only keep ids unique so that it + can be added later. -Revision 9 states a reading of its own, which is not a user answer, in each of these -eleven places (three of them hold two readings). Each is flagged where it appears, and +The ADR's lean, not the user's, is (iii): the formula language has no string +function today, and answer 22 already makes a reference to any kind unambiguous. + +**Q27. Does ids-first outlive the interim resolver?** Answer 33 made ids-first the +order "before we have autocomplete". Answer 38 says ids "should always be preferred +when there's ambiguity". +- **(a) Yes.** Wherever typed text could name both an id and a display, the id wins, + and that includes the autocomplete's ranking (an id match offered first, say). + Clause 12's "interim" wording goes. +- **(b) No.** Once the autocomplete turns display text into ids, nothing is matched + and nothing is ambiguous, so answer 38 governs only until then. + +No recommendation. + +**Q28. Two places resolve a display outside a client.** Answer 39: "the client should +be the only place that allows any reference to something by a human-friendly name". +- **The old-version loader** resolves an old file's references against display + names (clause 5), because in an old file the name is the only identity there is. +- **Headless `improvise cmd` and `improvise script`** run command lines that name + things by display, with no client attached. + +The ADR's reading: both are allowed. The loader migrates a file and is not a person +referring to something. The CLI is the UI layer of its own invocation, and it +converts names to ids at its edge, as a client does. Is that right? + +**Q29. Does step 4's collapse come after the strip?** Clause 4 puts it after step 3. +That order is what makes answer 41's example, `Time \u{200B} Period`, give +`time_period`. Before the strip, the zero-width character keeps the two `_` apart, and +the result would still be `time__period`. +- **(a) After the strip**, as clause 4 is written. The ADR's lean. +- **(b) Before the strip.** + +**Q30. Which "other entities" share the reserved `_`?** Answer 42: "A leading _ is +reserved for virtual categories and measures and other entities." +- (i) No virtual measure exists today. Is one planned, or does "measures" mean the + `_measure` category? +- (ii) The ADR reads the virtual views, `_records` and `_drill` (clause 1), as among + the "other entities". Is that right? +- (iii) The ADR reads "dropped to: disambiguate with -#" as "dropped too; + disambiguate with -#": a slug that collides once its `_` is dropped takes the + suffix. Is that right? + +**Q31. Are the positions under `_Index` and `_Dim` ids?** Records mode gives each row +a synthetic coordinate under `_Index`, and each column one under `_Dim` (a column +index, gotcha 1). They are never persisted and never typed. Answer 43 says the two +categories have ids `_index` and `_dim`, as answer 23 said, but revision 9's reading +was about the positions, and the answer does not say what they are. +- **(a) Outside the id namespace**, revision 9's reading: a position `0` under + `_index` does not collide with a user's item `0`. +- **(b) Inside it**, spelled so that they cannot collide, such as `_index-0`. That + needs answer 42's reservation to cover them. +- **(c) Something else.** + +No recommendation. + +**Q32. Moving the virtual categories to lower-case ids.** Today `_Index`, `_Dim` and +`_Measure` are keys in the model, on the wire, in `ViewState` and in `.improv` files: +`web/demo.improv` has `_Measure: column`, `_Index: none`, `## Category: _Measure` +and `_Measure=Revenue`. Under answers 23 and 43 they become `_index`, `_dim` and +`_measure`, and the displays stay as they are. The ADR's reading of the migration: +- an old file's `_Measure` resolves to `_measure` through its display at load, like + any other name (clause 5). The first save writes `_measure: column` and + `_measure=revenue`; +- a new-version file must spell the id, and `_Measure` there is refused (answer 20); +- the virtual category's header carries its id like any other: + `## Category: _Measure(_measure)`; +- every string literal that names them in code, on the wire and in the browser client + moves in the re-key step (clause 11, step 3). + +Is that the intended migration? + +**Q33. How do groups carry ids?** Answer 44 decides that a group has an id and a +separate friendly name. It does not decide: +- (i) **Namespace.** Is a group id unique in the whole document, like every other id + (the ADR's reading of answers 22 and 37), or only within its category? +- (ii) **Generation.** Is it the slug of the display, with `group` as an empty slug's + `` and the shared counter for collisions (the ADR's reading of answer 16(a), + "All id generation")? +- (iii) **The file.** For example a definition `> Coastal(coastal)`, a membership + `- Coastal_East(coastal_east)[coastal]`, a parent group (improvise-felo.6's + hierarchies) named by id, and a `collapsed:` line holding the group id, with or + without its category id. +- (iv) **The wire.** Does `AddItemInGroup`'s `group` name an existing group by id, + create one by display, or either? Is there a `RenameGroup { group, to }`? +- (v) **The frame.** `AxisFrame`'s `FrameGroup` carries a group name today. PR #336 + (improvise-cq1q.12, open) adds a display beside it. + +No recommendation beyond the readings stated. + +**Q34. Hand-written ids, and how step 5's trims repeat.** Answer 45: "A slug of - +should be invalid as an id and there should be no leading or trailing -s". +- (i) Is a hand-written id that is `-`, or begins or ends with `-`, refused at load, + as one beginning with `_` is? The ADR's reading is yes (clause 5). +- (ii) Is a hand-written id ending in `_` refused too? Answer 42 drops a trailing `_` + from a slug, and does not say whether a written id may carry one. +- (iii) Step 5 repeats until neither end holds `_` or `-`, so `-_x` → `x` and `x_-` → + `x`. One pass of each would leave `_x`, which is reserved, or `x_`. The ADR's + reading is to repeat. + +**Where revision 9 states a reading of its own.** In each of these fourteen places it +states a reading that is not a user answer. Each is flagged where it appears, and approving the PR approves them: -- the virtual views' reserved ids `_records` and `_drill` (clause 1); -- the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1); +- the virtual views' reserved ids `_records` and `_drill`, now read as among answer + 42's "other entities" (clause 1; Q30); +- groups share the document's one id namespace (clause 1; Q33); - a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2; **pending Q23**); -- `collapsed:` is `(CategoryId, group name)` (clause 2); - "whitespace" means the Unicode `White_Space` property (replacing the U+0020 reading, - by answer 34), and a slug of only `-` is a legal id (clause 4); + by answer 34) (clause 4); +- step 4's collapse comes after the strip (clause 4; Q29); +- step 5's trims repeat until neither end holds `_` or `-` (clause 4; Q34); +- a hand-written id that is `-` or begins or ends with `-` is refused at load + (clause 5; Q34); - the axis-before-`format:` order and the quoting of a format that reads as an axis line (clause 5); - the formula-line shape, and a `_Measure` target's display living on the `Formula` - (clause 5; **reopened by answers 35 and 36, Q23**); + (clause 5; **pending Q23**); - CheckedBatch's created-record exemption is removed (clause 6); - a display equal to another thing's id counts as ambiguous in the formula panel (clause 6); - an id is matched exactly, with no case folding (clause 12); - the display stage of the typed-name resolver goes once autocomplete exists - (clause 12). + (clause 12; Q27, since answer 38); +- answer 42's "dropped to:" is read as "dropped too;" (ninth round; Q30). + +Three readings that revision 9 listed here are gone. The user rejected the +`_Index`/`_Dim` reading as worded (answer 43, Q31), gave groups ids, which overrides +the `collapsed:` reading (answer 44), and rejected the legal `-` slug (answer 45). ## Consequences @@ -1362,10 +1721,10 @@ approving the PR approves them: - `Formula::raw` is no longer verbatim: the file carries formula text over ids. - The formula panel is a second rendering of formula text, over display names, so the **client** must resolve display names typed there back to ids (answers 19, - 32 and 33), ids first until an autocomplete exists. The host sees only id-form - text. + 32, 33 and 39), ids first (answers 33 and 38). The host sees only id-form text. - Ids are lower case, so an old file's data keys change spelling on its first save. - ADR 002's prose needs a wording update when this lands. -**Not addressed.** Per-entity versions (Q6), group ids (Q5), and ids in `AxisFrame` and -`CellId::Derived`. +**Not addressed.** Per-entity versions (Q6), and ids in `AxisFrame` and +`CellId::Derived`. Group ids are now in scope (answer 44, overriding Q5's answer), +but how they are made, spelled and carried is Q33. diff --git a/context/code/core.md b/context/code/core.md index d17f45df..49e6c119 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1712,6 +1712,20 @@ slugs to `a__b`: the steps are applied as written, and no extra underscore-collapsing step exists. A trailing run survives, so `East ` slugs to `east_`. Only *leading* underscores are removed (step 4). +**ADR 007 revision 9 proposes different rules, and this code predates them.** +Revision 9 is not yet approved. Its answers 34, 41, 42 and 45 make clause 4: +- each whitespace character becomes `_`, not only U+0020; +- runs of `_` collapse after the strip, so `a_ b` and `a__b` both slug to `a_b` + and `Time \u{200B} Period` to `time_period`; +- leading and trailing `_` and `-` are removed, so `East ` → `east`, `-East-` → + `east`, and `-` alone takes `-`. + +This section, `slugify`, and the tests +`slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and +`slugify_keeps_an_underscore_the_display_already_had` describe the earlier rule +above. They change in a follow-up once revision 9 is approved. The ADR's clause 4 +lists every difference. + **The allocator records every id it admits and releases none.** That is the whole of retirement (clause 4, "within a session a removed id is never handed out again"). No release operation exists, so a removed id cannot be reissued @@ -1734,7 +1748,8 @@ the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading `_` (only `_index`, `_dim` and `_measure` may carry one, per answer 23, and the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other id may start with `_`"), and -duplicates. It never renames. Refusing is the loader's job, and the loader +duplicates. It accepts `-`, `-x` and `x-`. Revision 9's Q34 asks whether a +hand-written id like that must be refused (answer 45). It never renames. Refusing is the loader's job, and the loader must name both lines of a duplicate; `ClaimError` carries only the id, so the loader keeps the line numbers. The virtual views `_records` and `_drill` are not yet in `VIRTUAL_IDS`. improvise-p003 item 1 asks for them, and they arrive From 568f52165da976eba3db1b3d055386ab8c595352 Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 22:47:04 -0700 Subject: [PATCH 058/106] fix(numtower): inexact presence per value, POW sign from exact parity, i128::MIN numerators (improvise-59nw.1) Answers ChatGPT Codex's P1 and Copilot's two findings on PR #337; each has a regression test that failed at edbb50c4. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/numtower.md | 23 ++++++++-- context/test-evidence/numtower.md | 37 ++++++++++++++++ crates/numtower/src/arith.rs | 22 ++++++++-- crates/numtower/src/num.rs | 8 +--- crates/numtower/src/sum.rs | 11 +++-- crates/numtower/tests/oracle/mod.rs | 10 +++-- crates/numtower/tests/properties.rs | 66 +++++++++++++++++++++++++++++ 7 files changed, 156 insertions(+), 21 deletions(-) diff --git a/context/code/numtower.md b/context/code/numtower.md index cb794e83..74941fc0 100644 --- a/context/code/numtower.md +++ b/context/code/numtower.md @@ -34,7 +34,10 @@ bytes, as the ADR measured, and `a_num_is_24_bytes` pins it. **The normal form.** An exact zero has no sign. A fraction is reduced and has a positive denominator. A value whose reduced denominator is `2^p·5^q`, with `k = max(p, q) <= 28` and a mantissa that fits 96 bits, is always the `Decimal`, never a `Fraction`. So -`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. +`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. Every `i128` +is a valid numerator, `i128::MIN` included, because the arm is `Ratio`. That one +value cannot be negated in `i128`, so `Operand::parts` keeps it out of the `i128` fast +path and its arithmetic takes the bigint path (Copilot on PR #337). ## Arithmetic @@ -116,6 +119,10 @@ rule, because the ADR decided it and because it needs no second division. negative power is a division and has its minimal scale. - **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`, including one too large to compute. +- **A demoted power takes its sign from the exact exponent.** `powf` gets the base's + magnitude, and the sign is set from `base < 0 && n odd`. An exponent above 2^53 rounds + to an even f64, so handing `powf` the signed base made `(-2)^(2^53 + 1)` read `+inf` + (Copilot on PR #337). ### The ceiling @@ -192,16 +199,24 @@ of the multiset of terms and not of their order. | the decimal partition | every decimal, and every fraction whose denominator is `2^p·5^q` with `k <= 28`, as an integer at scale 28 | 320 bits | | the groups | per other reduced denominator, the summed numerator | 256 bits each, in a `BTreeMap` | | `group_floats` | an `ExactSum` of each group's correctly rounded value | ADR 009's | -| `inexact` | an `ExactSum` of the f64 terms, plus their net weight | ADR 009's | +| `inexact` | an `ExactSum` of the f64 terms | ADR 009's | +| `inexact_terms` | the net weight per f64 bit pattern, entries at zero dropped | a `BTreeMap` | -**The read.** It is exact when the net inexact weight is zero and at most K = 8 groups +**The read.** It is exact when no inexact term is present and at most K = 8 groups remain. The groups and the decimal partition are then combined **over the lcm of their denominators, and reduced once**, and the stored bound applies to the result. Past K the read is the f64 accumulators: `group_floats`, the decimal partition converted once, and the inexact terms. That reading rounds each group, then the total. With inexact terms and at most K groups, the exact total is converted once and added to the inexact terms. A group whose numerator returns to zero is dropped, so "how many groups" is a function of -state. The K cutoff overrides representability: nine denominators whose exact total fits +state. + +**"Is an inexact term present" is asked per value, never of the net weight.** The first +version kept one net count. Inserting `1.0` and retracting `2.0` leaves the count at zero +while the accumulator holds −1.0, so the read took the exact branch and dropped it +(ChatGPT Codex, P1 on PR #337). Presence is now a map from f64 bit pattern to net weight, +dropped at zero, the same rule the groups follow. It costs O(distinct inexact values), +the same order as the groups. The K cutoff overrides representability: nine denominators whose exact total fits still read inexact (`more_than_k_denominators_read_inexact_even_when_the_total_fits`). That is question 13's second exception, and the ADR's flagged reading. diff --git a/context/test-evidence/numtower.md b/context/test-evidence/numtower.md index 7938728d..705879b8 100644 --- a/context/test-evidence/numtower.md +++ b/context/test-evidence/numtower.md @@ -285,6 +285,43 @@ Every arm's `Display` text reads back through `from_canonical` as the same arm, scale. The test pins the four spellings ADR 005 clause 11 tabulates, and that canonical exact text never reads back as an inexact value. +### the_accumulator_keeps_inexact_terms_whose_weights_cancel + +ChatGPT Codex, P1 on PR #337, against `edbb50c4`. `NumSum` decided "is an inexact term +present" from the **net** inexact weight. Adding `1.0` with weight 1 and `2.0` with weight +−1 nets to zero while the accumulator holds −1.0. So `read()` took the exact branch and +returned the exact partition alone, dropping the inexact term silently. + +This test **failed at `edbb50c4`**, on its first assertion: the read of `[1.0 ×1, 2.0 ×−1, +0.5]` claimed to be exact. Two things made the bug invisible to the existing properties: +- every generated term was inserted with weight +1; +- the retraction test retracts the very values it inserted, so per-value presence and net + presence agree there by construction. + +The test also asserts that retracting both inexact terms returns the exact `0.5`, so the +fix cannot simply make every sum that ever saw an f64 inexact. + +### a_huge_odd_exponent_keeps_the_sign_of_a_negative_base + +Copilot on PR #337, against `edbb50c4`. When the size check demotes a power to `powf`, the +exponent becomes an f64. Above 2^53 it rounds to an even value, so `(-2)^(2^53 + 1)` +returned `+inf` where the sign must be negative, and the negative exponent gave `+0.0`. +This test **failed at `edbb50c4`** on its first assertion. It also checks a fraction base, +and that an even exponent stays positive, so a fix that always negates cannot pass. + +### i128_min_is_an_exact_fraction_numerator + +Copilot on PR #337, against `edbb50c4`. `Num::fit` rejected `i128::MIN` as a numerator +because `to_i128` filtered it for both halves, so `Num::ratio(i128::MIN, 1)` became inexact +although `Ratio` holds it. This test **failed at `edbb50c4`** on its first +assertion. The oracle's `fits` had the same symmetric-range assumption, so no property +could see the bug. Both are fixed, and `fraction()` now generates `i128::MIN / d`. + +Allowing the value exposes a second hazard: it cannot be negated in `i128`, and the old +`negate` computed `-n`, which panics in a debug build. So the test also negates +`i128::MIN / 3` (an inexact, correctly rounded result), takes it through `Interim` twice +and back, divides it, compares it and reads its canonical text. + --- ## `crates/numtower/src/sum.rs` diff --git a/crates/numtower/src/arith.rs b/crates/numtower/src/arith.rs index 18cf3174..c8e9dcaf 100644 --- a/crates/numtower/src/arith.rs +++ b/crates/numtower/src/arith.rs @@ -135,19 +135,21 @@ impl<'a> Operand<'a> { } } + /// The `i128` fast path's operands; `i128::MIN` cannot be negated, so it takes the bigint path. fn parts(self) -> Option<(i128, i128)> { match self { Operand::Dec(d) => Some((d.mantissa(), pow10(d.scale()))), - Operand::Small(n, d) => Some((n, d)), + Operand::Small(n, d) => (n != i128::MIN).then_some((n, d)), Operand::Big(_) | Operand::Float(_) => None, } } fn big(self) -> Option { match self { + Operand::Dec(d) => Some(BigQ::from_i128(d.mantissa(), pow10(d.scale()))), + Operand::Small(n, d) => Some(BigQ::from_i128(n, d)), Operand::Big(q) => Some(q.clone()), Operand::Float(_) => None, - exact => exact.parts().map(|(n, d)| BigQ::from_i128(n, d)), } } @@ -247,7 +249,9 @@ fn negate(a: Operand) -> Interim { match a { Operand::Dec(d) => Interim::from(Num::exact(-*d)), Operand::Float(x) => inexact(-x), - Operand::Small(n, d) => small_ratio(-n, d) + Operand::Small(n, d) => n + .checked_neg() + .and_then(|m| small_ratio(m, d)) .map(Interim::from) .unwrap_or_else(|| settle_wide(BigQ::from_i128(n, d).neg())), Operand::Big(q) => settle_wide(q.neg()), @@ -380,14 +384,24 @@ fn exact_power(a: Operand, n: &BigInt) -> Result { (false, negative, false) => { let lower = |x: &BigInt| BigUint::from(x.bits().saturating_sub(1)) * k; let too_big = lower(&q.num).max(lower(&q.den)) > BigUint::from(CEILING_BITS); + let odd_negative = q.num.is_negative() && k.is_odd(); match (too_big, k.to_u32()) { (false, Some(k)) => Ok(exact_integer_power(&q, k, negative, target)), - _ => Ok(inexact(a.float().powf(n.to_f64().unwrap_or(f64::INFINITY)))), + _ => Ok(inexact(signed_powf(a.float(), n, odd_negative))), } } } } +/// `powf` on the magnitude, the sign taken from the exact exponent's parity. +fn signed_powf(base: f64, n: &BigInt, odd_negative: bool) -> f64 { + let magnitude = base.abs().powf(n.to_f64().unwrap_or(f64::INFINITY)); + match odd_negative { + true => -magnitude, + false => magnitude, + } +} + fn exact_integer_power(q: &BigQ, k: u32, negative: bool, target: u32) -> Interim { let (num, den) = (q.num.pow(k), q.den.pow(k)); let value = match negative { diff --git a/crates/numtower/src/num.rs b/crates/numtower/src/num.rs index 50075814..1f0448f1 100644 --- a/crates/numtower/src/num.rs +++ b/crates/numtower/src/num.rs @@ -106,8 +106,8 @@ impl Num { /// `q` in an exact arm, at its minimal scale, if it fits one. pub(crate) fn fit(q: &BigQ) -> Option { decimal_of(q).or_else(|| { - let num = to_i128(&q.num)?; - let den = to_i128(&q.den)?; + let num = q.num.to_i128()?; + let den = q.den.to_i128()?; Some(Num::fraction(num, den)) }) } @@ -138,10 +138,6 @@ pub(crate) fn small_ratio(num: i128, den: i128) -> Option { }) } -fn to_i128(x: &BigInt) -> Option { - x.to_i128().filter(|v| *v != i128::MIN) -} - pub(crate) fn pow10(k: u32) -> i128 { 10i128.pow(k) } diff --git a/crates/numtower/src/sum.rs b/crates/numtower/src/sum.rs index 65401366..b5271c1a 100644 --- a/crates/numtower/src/sum.rs +++ b/crates/numtower/src/sum.rs @@ -147,7 +147,8 @@ pub struct NumSum { groups: BTreeMap, group_floats: ExactSum, inexact: ExactSum, - inexact_weight: i128, + /// Net weight per f64 bit pattern: a read is inexact while any term is present. + inexact_terms: BTreeMap, } impl NumSum { @@ -168,7 +169,11 @@ impl NumSum { match x.kind() { NumRef::Inexact(f) => { self.inexact.add(f, weight); - self.inexact_weight += weight as i128; + let net = self.inexact_terms.entry(f.to_bits()).or_insert(0); + *net += weight as i128; + if *net == 0 { + self.inexact_terms.remove(&f.to_bits()); + } } NumRef::Exact(d) => { let unit = pow10(MAX_SCALE - d.scale()) as u128; @@ -225,7 +230,7 @@ impl NumSum { /// One read: exact over at most K groups, else the f64 accumulators (clause 8). pub fn read(&self) -> Num { - match (self.inexact_weight != 0, self.groups.len() > K) { + match (!self.inexact_terms.is_empty(), self.groups.len() > K) { (false, false) => Num::settle(&self.exact_total()), (_, true) => { let mut acc = self.inexact.clone(); diff --git a/crates/numtower/tests/oracle/mod.rs b/crates/numtower/tests/oracle/mod.rs index 6acb8f5c..6218812b 100644 --- a/crates/numtower/tests/oracle/mod.rs +++ b/crates/numtower/tests/oracle/mod.rs @@ -30,8 +30,8 @@ pub fn fits(v: &BigRational) -> bool { let scaled = v * BigRational::from_integer(BigInt::from(10).pow(s)); scaled.is_integer() && scaled.to_integer().abs() < limit }); - let max = BigInt::from(i128::MAX); - decimal || (v.numer().abs() <= max && v.denom() <= &max) + let (min, max) = (BigInt::from(i128::MIN), BigInt::from(i128::MAX)); + decimal || (&min <= v.numer() && v.numer() <= &max && v.denom() <= &max) } /// Whether `f` is the round-to-nearest-even f64 of `v`: the review's interval form. @@ -127,9 +127,11 @@ fn of_width(bits: u32) -> impl Strategy { } pub fn fraction() -> impl Strategy { - (1u32..=126, 1u32..=126) + let widths = (1u32..=126, 1u32..=126) .prop_flat_map(|(a, b)| (of_width(a), of_width(b))) - .prop_map(|(n, d)| Num::ratio(n, d).unwrap()) + .prop_map(|(n, d)| Num::ratio(n, d).unwrap()); + let edge = (1i128..i128::MAX).prop_map(|d| Num::ratio(i128::MIN, d | 1).unwrap()); + prop_oneof![8 => widths, 1 => edge] } pub fn finite_f64() -> impl Strategy { diff --git a/crates/numtower/tests/properties.rs b/crates/numtower/tests/properties.rs index 43043a46..95afebe9 100644 --- a/crates/numtower/tests/properties.rs +++ b/crates/numtower/tests/properties.rs @@ -473,6 +473,72 @@ fn min_and_max_prefer_the_exact_value_and_do_not_depend_on_order() { assert!(same(Num::inexact(f64::NAN).min_of(&dec("2")), &dec("2"))); } +#[test] +fn the_accumulator_keeps_inexact_terms_whose_weights_cancel() { + let mut s = NumSum::new(); + s.add(&Num::inexact(1.0), 1); + s.add(&Num::inexact(2.0), -1); + s.insert(&dec("0.5")); + let read = s.read(); + assert!(!read.is_exact(), "an inexact term is still present: {read}"); + assert_eq!(read.to_f64(), -0.5); + s.add(&Num::inexact(1.0), -1); + s.add(&Num::inexact(2.0), 1); + assert!(same(&s.read(), &dec("0.5")), "every inexact term retracted"); +} + +#[test] +fn a_huge_odd_exponent_keeps_the_sign_of_a_negative_base() { + let odd = Num::exact(Decimal::from(9_007_199_254_740_993i64)); + let up = dec("-2").pow(&odd).unwrap(); + assert!(!up.is_exact() && up.to_f64() == f64::NEG_INFINITY, "{up}"); + let down = dec("-2").pow(&-&odd).unwrap(); + assert!( + !down.is_exact() && down.to_f64() == 0.0 && down.to_f64().is_sign_negative(), + "{down}" + ); + let third = Num::ratio(-1, 3).unwrap().pow(&odd).unwrap(); + assert!(third.to_f64().is_sign_negative(), "{third}"); + let even = &odd + &dec("1"); + assert_eq!(dec("-2").pow(&even).unwrap().to_f64(), f64::INFINITY); +} + +#[test] +fn i128_min_is_an_exact_fraction_numerator() { + let min = Num::ratio(i128::MIN, 1).unwrap(); + assert!( + matches!(min.kind(), NumRef::Fraction(r) if *r.numer() == i128::MIN), + "{min}" + ); + let third = Num::ratio(i128::MIN, 3).unwrap(); + assert!(matches!(third.kind(), NumRef::Fraction(_)), "{third}"); + let negated = -&third; + let want = -q(&third).unwrap(); + assert!( + !negated.is_exact() && is_nearest(negated.to_f64(), &want), + "{negated}" + ); + let back = + &(&Interim::from(third.clone()) * &Interim::from(dec("-1"))) * &Interim::from(dec("-1")); + assert!( + same(&back.store(), &third), + "through the wide path and back" + ); + let halved = third.checked_div(&dec("2")).unwrap(); + assert_eq!( + q(&halved), + Some(q(&third).unwrap() / BigRational::from_integer(BigInt::from(2))) + ); + assert_eq!( + min.cmp_exact(&Num::ratio(i128::MIN + 1, 1).unwrap()), + Some(Ordering::Less) + ); + assert_eq!( + Num::from_canonical(&min.to_string()).map(|n| same(&n, &min)), + Some(true) + ); +} + // --------------------------------------------------------------------------- // The accumulator is independent of order. From fa26ec1e417ab581cc7e725122246cfe9480e464 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Thu, 8 Oct 2026 23:09:04 -0700 Subject: [PATCH 059/106] docs(adr): condense ADR 007 to status, context, decision and open questions (improvise-cq1q.1) On the user's instruction the ADR no longer carries the decision process: no answer rounds, quotes, answer numbers or revision history. The accepted decisions and the proposed amendment are stated as design, with each amendment rule marked [Amendment] and a table of what it replaces. Settled questions are deleted; the eleven open ones are renumbered from 1 and written for a first-time reader. The verbatim record of every answer moves to a provenance comment on PR #326. Folds in the delta review of 74efa481: a group's identifier is an id, and whether it also has a display is now an open question; formulas referring to anything in a typed context is stated without deciding what that allows; the Decision's opening and Rejection::Gone include groups. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 2288 ++++++++------------------ context/code/core.md | 15 +- 2 files changed, 725 insertions(+), 1578 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 0f6dfd17..8f9ec480 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -1,1045 +1,374 @@ -# ADR 007 — Stable identities for categories, items, formulas and views - -**Status:** accepted (2026-09-25). The user approved revision 7. Revision 8 then -followed the adversarial review of that head: it adds clause 12, the ADR's own reading -of when a formula edit's names become ids, and corrects stale facts about main. The -user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17:10Z: -"looks good, approved"). -**Revision 9 is a proposed amendment, not yet approved.** It folds in the user's -answers on the implementation tickets (answers 30–32), the user's two inline -comments on its head `4aed8c07` and a clarification in chat (answers 33 and 34), -its two later inline comments, also on `4aed8c07` (answers 35 and 36), and clarifies text the reviews found unclear or stale (improvise-cq1q.1). Answer 33 -reverses the matching order that answer 31 recorded, and answer 34 overrides the -ADR's own reading of "space". Answers 35 and 36 make measures the normal formula -target and give them an id/display split, which reopens Q23. It asked three new -questions, Q23 (how a formula-target measure carries its id, and how a formula line -spells its target), Q24 and Q25 (two details of slugify's whitespace rule). -The user's nine inline comments on head `068e4130` are answers 37–45 (the ninth -round). **They answer Q24 and Q25**, rewrite slugify's steps, **give groups ids -(overriding answer 9)**, reject two of the ADR's flagged readings (the `_Index`/`_Dim` -one and the legal `-` slug) and override a third (`collapsed:`), and ask for Q23 to be -stated more clearly, which it now is. **Q23 stays open.** The ninth round adds nine -questions, Q26–Q34. The places where the ADR states a reading of its own that is not -a user answer are listed at the end of "Still open". · -**Decided by:** the user; see "What the user decided" · -**Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · -**Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and -improvise-fdk1's `RenameRecordCoordinate` (landed in PR #221) · -**Builds on:** ADR 001, ADR 002 · **Depended on:** improvise-tt22 (remove legacy JSON), -which landed in PR #224 - -**Numbering.** The ADR was first filed as 005, but improvise-l2b8 already claims ADR 005 -(exact decimal arithmetic) and improvise-ilvl claims 006. The review tickets filed -against the first draft still say "ADR 005" in their titles; they mean this document. - -**Direction from** the user, 2026-09-23, in three messages: - -> "there would be a lot of benefit to categories/formulas having stable identities, even if -> they were only transient: it would help pin editing operations and distinguish them from -> delete/remove" - -> "It's probably worth a file format change (in a opt-in way, that gets rectified on save if -> the input file was missing them)" - -> "Maybe something like `View: Default(default)` and the standard -> `View: |Default (default)|(default)` for quoting." - -## What the user decided - -The user answered the **first draft** (`context/adr/005-stable-identities.md` at -`9d97664c`) with inline comments on PR #210, and then posted one comment on the thread. -Each answer is quoted verbatim below, with where it was attached and what it overturned. -**Five of the first draft's positions were reversed.** The rest of this document is -rewritten to match. - -**1. Storage keys by id; the human name becomes a display field.** On first-draft -line 92, the clause "`CellKey`, `DataStore`, views and formula text **stay keyed by -name**": - -> "Shouldn't it be easy to replace this to key by ID? There's no type change so, at worst, -> it would be renaming field access. We could also add the name as something like -> `display` and reserve `name` or whatever it is currently for the id." - -**2. The point of ids is that keys use them.** On first-draft line 97, "ids do not make a -rename cheap. They make it *expressible*": - -> "This is a lot lower value if CellKeys, etc. don't refer to the new id" - -**3. Every reference uses the id.** On first-draft line 275, the heading of clause 8, -"ADR 002: `RecordHash` stays defined over names": - -> "No, anything referring to something with an id should refer to the id as the stable -> identity to avoid glitches." - -This answers Q7 by reversing it, and reverses clause 5's "references carry names" too. - -**4. Q1, the id's internal type.** On first-draft line 469, the recommendation to wrap "a -small string": - -> "We probably can reuse symbols as the internal type with interning?" - -Ids are strings, interned through the existing `SymbolTable`. - -**5. Q2, formula id placement.** On first-draft line 473, recommendation F1 -(`- Profit(3) = Revenue - Cost`): - -> "Yes, before the equal to the right of the name" - -**6. Q3, header syntax.** On first-draft line 480, the claim that "a header with a space -before its parenthesis (`Sales (copy)`) is unaffected": - -> "Space before the open paren is optional: document this edge case that might break -> existing docs." - -**7. Q4, data keys.** On first-draft line 485, the recommendation "no — names only": - -> "In Data, we should prefer ids that are guaranteed unique to names that might not be." - -**8. Import generates ids; legacy JSON goes.** Thread comment, 2026-09-23 06:51: - -> "For CSV or JSON import, the dimensions are to be treated as the human friendly name and -> never parsed for IDs: during import, ids should be generated by a slugify algorithm that -> strips non-alphanumeric characters and replaces spaces with hyphens(collapsing runs). -> The only supported native file format is .improv, cleanup any legacy JSON format." - -Legacy JSON removal is its own implementation ticket, **improvise-tt22**. This ADR -depended on it and did not implement it; it landed in PR #224. It also dissolves -improvise-65kf: with no legacy JSON to load, there are no numeric JSON ids to accept. - -**The answers change the ADR's centre of gravity.** The first draft put an id *beside* -the name as an index. The user's answers make the **id the key** and the name a -**display** string. So a rename becomes cheap: it changes one display field and moves no -key, no surrogate and no basis tier. The first draft had called that impossible. - -### Second round: answers on revision 2 - -The user answered revision 2 (`context/adr/007-stable-identities.md` at `5e46ce03`) -with thirteen inline comments on PR #210, posted 2026-09-23 15:03–15:11Z. Each answer -is mapped to the question **on the line it is attached to**, and quoted verbatim. - -**9. Q5, group ids.** On line 468, inside Q5 ("Recommendation: not now … the one -exception to 'every reference is an id'"): - -> "This seems right" - -Groups get no id in this ADR. **Overridden by answer 44** (ninth round): a group -name is an id, and the friendly name is separate. - -**10. Q6, per-entity versions.** On line 469, Q6 ("Recommendation: not in this ADR"): - -> "Agreed" - -**11. Q8, a duplicate or unknown id.** On line 473, the recommendation to refuse the -load: - -> "Yeah, don't load the file because you can't safely resave it" - -**12. Q9, the reading of "opt-in".** On line 476, the recommendation that an id-less -file is read forever and every save writes ids: - -> "Yes" - -**13. Q10, id reuse after a reload.** On line 485, inside option **(b)**, "Accept -reuse across a reload (recommended)": - -> "Yes, b" - -This is option (b), and (b) was the recommendation. Within a session, retirement still -prevents reuse. Across a save and reload, a removed name that comes back gets its old -slug back. - -**14. Q11, who assigns ids.** On line 492, inside option (a): - -> "The host assigns" - -**15. Q12, old-version files and the new header rule.** On line 502, inside option (a), -"No (recommended)": - -> "Yes, a as long as version is unambiguously marked" - -Old files are read as they always were, on the condition that the version is marked -unambiguously. Clause 9 says how the loader meets that condition. - -**16. Q13, slugify.** Five answers, one per item: -- **(a) where it applies.** On line 510: - > "All id generation" -- **(b) case.** On line 512, against the recommendation to keep case: - > "lower case only" -- **(c) `_` and `-`.** On line 515: - > "Yeah, keep both" -- **(d) non-ASCII letters.** On line 518: - > "Strip non lower alphanumeric or -_" -- **(e) the empty slug.** On line 520, the recommendation `-`: - > "Yes" - -So an id is lower case, and its alphabet is `a`–`z`, `0`–`9`, `-` and `_`. Clause 4 -states the rule. Q15 asked the user to confirm the order of its steps (answer 18). - -**17. Q14, formula text.** On line 529, the recommendation "ids in the file": - -> "Display in the formula panel but not as column/row/page headers." - -This does not contradict ids in the file, which answer 3 already requires. It answers -**where formula text is displayed**, and it could be read two ways. That was Q16, -which answer 19 settles. - -### Third round: answers on revision 3 - -The user answered revision 3 (`context/adr/007-stable-identities.md` at `18c82e1c`) -with three inline comments on PR #210, posted 2026-09-24 00:34–00:36Z. Each is mapped -to the question on the line it is attached to, and quoted verbatim. - -**18. Q15, the order of slugify's steps.** On line 571, Q15's recommendation "fold, then -strip": - -> "casefold, replace space with `_`, strip" - -The fold comes first, as recommended. The answer also **changes answer 8's middle -step**: a space becomes `_`, not `-`. Two details of that sentence are Q18. - -**19. Q16, what the formula panel shows.** On line 584, the paragraph that declined to -recommend a reading: - -> "It should be at least `Profit (profit) = Revenue - Cost` but if `Revenue` is -> ambiguous, you'll have to display the slugs. Working with slugs internally and making -> the friendly names view-layer is almost always the right move here." - -So the panel shows formula text over **display names**, with the target's id beside it -in parentheses. Where a display name in the text is ambiguous, the panel shows the id -instead. Ids are what the file and the model hold, and display names belong to the view -layer. This also settles answer 17: the thing displayed "in the formula panel but not as -column/row/page headers" is the target's **id**. Whether an id is enough to resolve an -ambiguous name is Q19. - -**20. Q17, capitals in a hand-written id.** On line 589, the question "May a -hand-written id contain capitals?": - -> "no." - -This is option (a). `id_token` is lower case only, and a header ending in a -parenthesised run that contains a capital loads as a whole-line display with a load -warning. The virtual categories are spelled with capitals today. That is Q20. -(Answer 21 later made a new-version file all-or-nothing, so that header is now refused -rather than warned about. See clause 5.) - -### Fourth round: answers on revision 4 - -The user answered revision 4 (`context/adr/007-stable-identities.md` at `024aa2d3`) -with five inline comments on PR #210, posted 2026-09-24 06:05–06:08Z. Each is mapped -to the question on the line it is attached to, and quoted verbatim. - -**21. Q18(a), runs of spaces, and the collision suffix.** On line 658, Q18(a) ("Does a -run of spaces collapse to one `_`?"): - -> "runs of spaces are collapsed to one `_` and then `-###` numeric suffix is added if -> duplicate (only when importing a file without IDs: a file must either have no IDs at -> all or always have IDs)" - -This decides three things: -- a run of spaces collapses to one `_`, as recommended (a run of whitespace, by - answer 34; answer 41 makes each whitespace character a `_` first and then collapses - runs of `_`); -- a duplicate slug gets a numeric suffix after a `-`; -- **a file has ids on everything or on nothing.** A new-version file with a declaration - missing its id is refused, like any file that cannot be resaved safely (answer 11). - So the earlier rule that a hand-written id wins over a generated one in the same file - is gone. - -The parenthesis also limits the suffix to generating ids for a file that has none. Two -details of that are Q22. Q18(b), what "casefold" means, was not answered in this round -(answer 26 settles it). - -**22. Q19, whether an id disambiguates formula text.** On line 678, option (a) ("Make -item ids unique in the document"): - -> "Ids have to be globally unique." - -This is stronger than option (a). **No two ids in a document are equal, whatever their -kind**, so a category, an item, a formula and a view never share one. An id in formula -text or on the wire therefore always names exactly one thing. - -**23. Q20, the virtual categories.** On line 691, option (a) ("Lower-case ids, -capitalised displays"): - -> "Virtual categories have IDs: we need to add a "slug" version such as `_index`, but -> no non-virtual id can start with `_`" - -The ids are `_index`, `_dim` and `_measure`, and the displays stay `_Index`, `_Dim` and -`_Measure`. A leading `_` is reserved for them. Clause 4 says how slugify keeps a -generated id out of that space, and clause 5 says what happens to a hand-written one. -Answer 42 widens the reservation to "virtual categories and measures and other -entities", and answer 43 restates `_index` and `_dim`. - -**24. Q21, an old file naming an undeclared item.** On line 708, option (a) ("Declare -the item at load"): - -> "yes, this is right." - -**25. The file version.** On line 731, the Consequences line "A new file version.": - -> "File version should be added explciitly if it's not already part of the format." - -It is already part of the format. The first line of every `.improv` file is a -mandatory, exact version literal (`version_line = { "v2025-04-09" ~ NEWLINE ~ … }` in -`improv.pest`), and the writer emits it. So the new format is marked by a **new -literal**, and nothing is inferred. Clause 10, new in this revision, states the rule in -one place. - -### Fifth round: answers on revision 5 - -The user answered revision 5 (`context/adr/007-stable-identities.md` at `e044c968`) -with three inline comments on PR #210, posted 2026-09-24 19:50–19:51Z. Each is mapped -to the question on the line it is attached to, and quoted verbatim. - -**26. Q18(b), what "casefold" means.** On line 766, the recommendation to take -"casefold" literally, as full case folding: - -> "full unicode" - -Slugify's first step is **Unicode full case folding**, so `Straße` → `strasse`. - -**27. Q22(a), how the suffix is spelled.** On line 771, the recommendation "a plain -number": - -> "Just a plane number." - -The suffix is `-` followed by a plain decimal number, with no padding. - -**28. Q22(b), collisions outside import.** On line 782, option (i), "Suffix it there -too": - -> "Any ambiguous id gets a suffix, if if the id is between different types of things -> identified. Don't bother implementing a strict order here: `east` `east-1` -> `east-2`, etc. just do what gensym does in CL: global increment-only counter appended -> when dedupe needed." - -This decides three things: -- **every** colliding id gets a suffix: on import, in a session, and across kinds (a - category and an item that slug alike); -- the number comes from **one counter for the document that only ever increments**, as - Common Lisp's `gensym` does, so it is not "the lowest free number"; -- the suffix is appended **only when a dedupe is needed**, so an id that does not - collide stays bare. - -Clause 4 is rewritten to match. The rules this replaces (lowest free number, and -suffixing only on import) were revision 5's. - -### Sixth round: answer on revision 6 - -The user replied at 2026-09-24 22:09Z to their own answer 28, on line 782 of -`e044c968`: - -**29. Hand-written ids are canonical.** - -> "You have to treat the file's hand-written ids as canonical, doing otherwise will -> corrupt the data section." - -This constrains answer 28. The `gensym` suffix applies only to an id the program is -**generating**, on import, when an id-less file loads, or when the host creates -something in a session. **An id written in a file is never renamed**, because every -data key and reference in that file spells it. Clause 4 says what follows: -- a generated id that would collide with a file's id takes the suffix, and the file's - id keeps its spelling; -- two identical hand-written ids in one file are refused at load, not repaired. - -### Seventh round: answers on the implementation tickets - -After the ADR was accepted, planning its implementation (epic improvise-cq1q) -raised three questions the ADR does not answer. They were answered on -2026-09-25, and the answers are recorded as comments on the tickets -(improvise-m56i and improvise-hexb). The tracker shows the repository owner's -account as the author of those comments, and agents write to it under the same -account, so authorship there does not show whose words they are. The user's -"Approved" on this PR is what confirms them. Answers 30 and 31 quote the option -chosen, as recorded, and the sentences after each are the ADR's own statement of -what it means. Answer 32 was recorded as one sentence and is quoted whole. - -**30. A formula target outside `_Measure`** (improvise-m56i item 2): - -> "Real item + separate formula id" - -`Model::add_formula` keeps creating a real item for a target outside `_Measure`. -That item has its own `ItemId`, the formula has its own `FormulaId`, and by -answer 22 the two differ. A formula line names such a target by the item's id. -This was chosen over the recommended alternative, which was to give every formula -target the `_Measure` treatment (no real item, handled by its `FormulaId`). -The user's comment on this sentence is answer 37: a formula id that would collide -takes the counter's suffix. - -**31. A typed name that is both one thing's display and another thing's id** -(improvise-m56i item 6): - -> "Display first; client holds it" - -The option is quoted as recorded. **The order it names was reversed by the user's own -comment on this answer (answer 33): until autocomplete exists, a typed name is -matched against ids first, then against displays.** What stands from answer 31 is -that the client holds the resolution (answer 32), and that a display matching more -than one thing is refused, with the refusal listing the candidates' ids. Where -answers 31 and 33 differ, answer 33 governs. The user's comment on this sentence is -answer 38: ids "should always be preferred when there's ambiguity". - -**32. Which side holds clause 12's capture** (improvise-hexb item 2), as recorded: - -> "USER DECISION 2026-09-25 (item 2): the CLIENT holds clause 12's display->id -> capture (like OpenedCell) and resolves names against its id-bearing -> projection; the wire carries only id-form text; the host never resolves a -> display name." - -Clause 12 is rewritten to match. The user confirmed this on the PR and widened it -from the formula editor to every reference (answer 39). - -### Eighth round: answers on revision 9 (head `4aed8c07`) - -The user answered revision 9 (`context/adr/007-stable-identities.md` at `4aed8c07`) -with four inline comments on PR #326: two posted 2026-10-04 02:48Z and 02:49Z, and -two posted 2026-10-05 20:22:52Z and 20:23:40Z. The GitHub API gives all four the -original commit `4aed8c07`. **All four are one round**: the user wrote the last two -before `e9659903`, which folded in the first two, existed, so they had not seen -answers 33 and 34 written up. The second comment was also clarified in chat (not on -the PR, relayed to the steward). Each is mapped to the line it was attached to on -`4aed8c07`, and quoted verbatim. - -**33. The order of answer 31.** On line 356, the sentence under answer 31, "A typed -name is matched against displays first, then against ids": - -> "This should be handled by separating the view layer autocomplete of typed display names from the model layer reference to IDs: but, before we have autocomplete, IDs are matched first." - -This decides two things: -- **The long-term design.** A view-layer autocomplete resolves typed display text to - ids. The model layer refers to ids only, and never matches a display. -- **The interim rule, until autocomplete exists.** A typed name is matched against - **ids first, then displays.** This reverses answer 31's "displays first". - -Clause 12 states the rule and what it means for a document where one thing's display -equals another's id. Nothing built yet implements either order (clause 11, step 7). - -**34. What "space" means in slugify.** On line 553, the bullet that read "Space" -as U+0020 only, so that a tab was stripped and `Time\tPeriod` slugged to -`timeperiod`: - -> "This shouldn’t matter because consecutive spaces should normalize to a single underscore" - -and, in chat: - -> "space should generally mean "whitespace"" - -This **overrides the ADR's own reading** (revision 9's, which was not a user answer). -"Space" in answers 18 and 21 means whitespace, and a run of whitespace becomes one -`_`. The user did not enumerate which characters count, so the definition in clause -4 is **the ADR's own reading of "whitespace"**, flagged there. Answers 18 and 21 -keep their wording, which this reads as "space, meaning whitespace". **Answer 41 -refines the mechanism:** each whitespace character becomes `_`, and runs of `_` are -collapsed afterwards, so a run of whitespace still ends as one `_`. - -**35. Measures get the id/display split too.** On line 1091, the end of Q23's -option (c), "Always the target's id, giving a `_Measure` target an id of its own that is -distinct from the formula's. This reverses clause 1's 'its handle is its formula's -`FormulaId`'": - -> "Measures should have ID/display split too. Measures are the normal case for formulas, anyways: something like `profit = revenue - cost`" - -**36. Dimensions are rarely formula targets.** Also on line 1091, 48 seconds later: - -> "Dimensions are mainly in formulas when you're calculating some kind of derived dimension, but usually dimensions should be relatively static and not interesting in formulas." - -These decide two things, and no more: -- **A measure has an id and a display**, as every other thing does. -- **The normal formula target is a measure** (`profit = revenue - cost`), and a - dimension (a non-`_Measure` category) target is the rare, derived case. - -They do **not** decide what a formula-target measure's id is, how its formula line -names it, or whether it is an item. That is **Q23**, reopened and reworded. Four -things in earlier text meet them, and none is resolved here: -- **Clause 1 and revision 9's flagged readings (clauses 1, 2, 5, 8)** say a `_Measure` - formula target "has no item of its own" and is handled by `FormulaTarget(FormulaId)`. - A stored measure (a non-formula `_Measure` item) is an ordinary item and already has - an id and a display. A measure that exists only because a formula targets it has - its display on the `Formula` and its id in the `FormulaId`. Whether that is the - split the user wants, or whether the target should be an item, is Q23. -- **Answer 3** ("every reference is an id") is about references. Clause 5 calls a - formula line a declaration, so the bare display in its target position is a - declaration's display and not a reference. Whether the normal case should still - spell an id there is part of Q23. -- **Answers 5 and 19** picture the measure case as a display plus one id - (`Profit(3)`, `Profit (profit)`), which fits an id/display split (answer 5's - file-line picture fits option (c) only loosely; see Q23). -- **Answer 30** and the ADR's worked example (`Tax` in `CustomCat`) are about the - *rare* case. The decision stands. The emphasis of the surrounding text changed: - revision 9 wrote the dimension target as the main case and the `_Measure` target as - the exception. - -### Ninth round: answers on revision 9 (head `068e4130`) - -The user answered revision 9 at `068e4130` with nine inline comments on PR #326, -posted 2026-10-09 05:17:38Z–05:26:39Z. The GitHub API gives all nine the original -commit `068e4130`. Each is mapped to the line it was attached to on `068e4130`, -quoted verbatim, and given the comment id. Where the ADR reads more into an answer -than its words say, the text says it is the ADR's reading and asks a numbered -question. - -**37. An id names one thing, and a colliding formula id takes the counter.** Comment -4226862456, on line 356, the last sentence of answer 30 ("This was chosen over the -recommended alternative, which was to give every formula target the `_Measure` -treatment …"): - -> "An id always refers to the same thing and so must be globally unique: if a formula has an id that would conflict, increment the counter to disambiguate it. Formulas may refer to anything in a typed context: e.g. a formula might count the number of times a particular letter appears in a category value." - -This decides: -- **A formula id that would collide is suffixed from the counter**, like any other - generated id. This restates answers 22 and 28 for the formula case, and clause 1's - example already follows it: a formula `Tax` in `CustomCat` gets item id `tax`, then - formula id `tax-`. -- **Formulas may refer to any kind of thing**, and the context of the reference - gives its type. This is why ids are unique across kinds (answer 22): a bare id in - formula text must name one thing whatever its kind. - -The example (count a letter in a category value) has no function in today's formula -language. Whether it is in this ADR's scope, what a reference may name, and whether -"a category value" means an item's display or its id are **Q26**. - -**38. Ids win under ambiguity.** Comment 4226867495, on line 368, the end of the -paragraph under answer 31 ("a display matching more than one thing is refused … -Where answers 31 and 33 differ, answer 33 governs"): - -> "IDs are unambiguous so they should always be preferred when there's ambiguity. " - -This confirms answer 33's ids-first order. The display-versus-display refusal is -untouched: no id is involved there, so there is nothing to prefer. Whether "always" -makes ids-first permanent rather than interim (clause 12 calls it interim until an -autocomplete exists) is **Q27**. - -**39. Display names live only in the client.** Comment 4226873976, on line 377, -"Clause 12 is rewritten to match.", under answer 32: - -> "Yes, the client should be the only place that allows any reference to something by a human-friendly name and this sort of reference should be minimized to the UI layer and converted to IDs ASAP" - -This confirms answer 32 and **widens it from the formula editor to every -reference**: only a client accepts a human-friendly name, it does so in the UI layer -only, and it converts the name to an id as early as it can. Clause 2's "resolved to -ids at the input edge" and clause 6's "every operand naming an existing thing is an -id" already say this. Two places in the ADR resolve a display outside a client: the -old-version loader (clause 5) and headless `improvise cmd`/`script`. Whether they -are exceptions is **Q28**. - -**40. A formula is a derived measure.** Comment 4226895692, on line 1303, the end of -Q23 ("Quoting the target position …"): - -> "Formulas are a derived measure. I don't really understand the problem here, this whole section needs to be stated more clearly." - -**Q23 is rewritten** to state the problem in those terms, with a worked example and -a plain split between what is decided and what is open. It stays **open**: the -answer says what a formula *is*, and the ADR does not read it as choosing an option. -"Still open" says why. - -**41. Whitespace becomes `_` first, then runs of `_` collapse.** Comment 4226897815, -on line 1316, Q24's "neighbouring edge", where `Time \u{200B} Period` slugged to -`time__period`: - -> "The spaces should have been normalized to _ before collapsing runs of _s" - -This **answers Q24 with option (b)**. It changes slugify's step 2 and adds a -collapse step (clause 4, steps 2 and 4): -- each whitespace character becomes `_`; -- **runs of `_` collapse to one `_`**, whether they came from whitespace or were - already in the display. So `a_ b` → `a_b` and `a__b` → `a_b`; -- `Time \u{200B} Period` → `time_period`. - -That last result needs the collapse to run **after** step 3 strips the zero-width -space. The answer says the collapse comes after the normalisation, and the example it -is attached to gives one `_` only if it also comes after the strip. So clause 4 puts -it there. That placement is the ADR's reading, and **Q29** asks the user to confirm -it. This overrides revision 9's reading and the code's comment in -`context/code/core.md` §20 ("no extra underscore-collapsing step exists"). - -**42. A leading `_` is reserved; a trailing one is dropped.** Comment 4226901843, on -line 1324, Q25's option (c): - -> "A leading _ is reserved for virtual categories and measures and other entities. A trailing one should be dropped to: disambiguate with -#" - -This **answers Q25 with option (b)**: slugify drops a trailing `_` as it drops a -leading one, so `East ` → `east` and `East_` → `east`. The ADR reads "to:" as "too;": -drop it too, and if the shorter slug then collides, disambiguate it with `-` -(answers 27 and 28). The reservation of a leading `_` now covers "virtual categories -and measures and other entities", which is wider than answer 23's three categories. -The ADR reads the virtual views `_records` and `_drill` (clause 1) as among the -"other entities". What a virtual *measure* is, and that reading, are **Q30**. - -**43. `_Index` and `_Dim` have ids.** Comment 4226904733, on line 1330, the flagged -reading "the `_Index`/`_Dim` coordinates are outside the id namespace (clause 1)": - -> "No, they have ids _index and _dim " - -This **rejects the reading as it was worded**, and restates answer 23: the categories' -ids are `_index` and `_dim`. The reading was about their *coordinates*, the -synthetic positions in records mode (`_Dim` carries a column index, gotcha 1), not -the categories. Whether those positions are ids in the document's one namespace, and -how they would be spelled so that `_index`'s `0` does not collide with a user's item -`0`, is **Q31**. Today the repository spells the virtual categories `_Index`, `_Dim` -and `_Measure`, as keys, on the wire and in `.improv` files. What the move to lower -case ids costs is **Q32**. - -**44. Groups get ids.** Comment 4226908008, on line 1332, the flagged reading -"`collapsed:` is `(CategoryId, group name)` (clause 2)": - -> "Group name should be an id for the group, not a friendly name " - -This **overrides answer 9** ("not now") and the reading it was attached to. A group -has an id, and its friendly name is a separate display. A `collapsed:` entry names a -group by its id. Clause 1's table, clause 2 and clause 5 are changed to match. How a -group's id is made and spelled, and what the wire carries, are **Q33**. The ADR -does not decide them. -- **Relation, not a dependency.** PR #336 (improvise-cq1q.12, branch - `adr007/display-2b`, open, not on `main`) gives `Group` a private display beside - its key. On the ADR's reading, under answer 44 that key becomes the group's id, as - a category's does (clause 2). Nothing here depends on #336 landing. - -**45. A slug of `-` is not an id; no id begins or ends with `-`.** Comment -4226910518, on line 1334, the flagged reading "'whitespace' means the Unicode -`White_Space` property … and a slug of only `-` is a legal id (clause 4)": - -> "A slug of - should be invalid as an id and there should be no leading or trailing -s" - -This **rejects the second half of that reading**. Slugify removes leading and -trailing `-`, so `-` and `---` leave nothing and take `-` (step 6), and -`-East-` → `east`. The first half of the bullet, the meaning of "whitespace", was not -addressed and stays the ADR's flagged reading. That a **hand-written** id with a -leading or trailing `-` is refused at load, as one with a leading `_` is, and how the -trims interleave (`-_x` → `x`), are the ADR's reading of "invalid as an id" and are -**Q34**. +# ADR 007 — Stable identities for categories, items, formulas, views and groups + +**Ticket:** improvise-sboz (implementation epic improvise-cq1q; this amendment +improvise-cq1q.1) · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's +interim `FormulaId` (PR #214) and improvise-fdk1's `RenameRecordCoordinate` +(PR #221) · **Builds on:** ADR 001, ADR 002 · Review tickets filed against an early +draft call this document "ADR 005". + +## Status + +**Accepted** on 2026-09-25 (PR #210). **An amendment is proposed and not yet +approved** (PR #326). The amendment's text is marked **[Amendment]** wherever it +appears. Everything without that mark is accepted and binding. + +Where the amendment replaces accepted text, the accepted rule binds until the +amendment is approved: + +| clause | accepted rule | the amendment proposes | +|---|---|---| +| 1 | groups have no id | a group's identifier is an id | +| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | the virtual views also have reserved `_` ids: `_records`, `_drill` | +| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` (pending open question 1) | +| 2, 5 | a collapsed entry and a group reference name a group by name | by its id | +| 4 | a run of U+0020 spaces becomes one `_`; only a leading `_` is removed | each whitespace character becomes `_`, runs of `_` collapse after the strip, and leading and trailing `_` and `-` are removed | +| 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | + +The amendment also adds rules where the accepted text says nothing: +- a measure has an id and a display; +- the axis-line/`format:` order; +- `ViewState` and `Workbook` key the active view by id; +- CheckedBatch's created-record exemption is removed; +- two branches of one file can merge into a file that will not load. + +Each is marked where it appears. Eleven questions are open (last section). ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. `CellKey` and `DataStore`'s interned symbols are names. Views, axis maps, page -selections, hidden and collapsed sets, and formula text all refer by name. The user's -first message names what that costs: +selections, hidden and collapsed sets, and formula text all refer by name. That costs +three things: +- **An edit cannot be told apart from a delete followed by an add.** No rename + operation exists for a category or an item. A rename is remove plus add, which loses + every cell (improvise-fz77). A formula is identified by `(target, target_category)`, + and `Model::add_formula` upserts on that pair. +- **An edit whose target was concurrently replaced cannot be told apart from an edit + of the replacement.** +- **A records-mode coordinate rename names its category and item by name** + (`RenameRecordCoordinate { record, category, item }`, `String` operands). -- **An edit cannot be told apart from a delete followed by an add.** No rename operation - exists for a category or an item. A rename is remove plus add, which loses every cell - (improvise-fz77). A formula is effectively identified by `(target, target_category)`, - and `Model::add_formula` *upserts* on that pair. improvise-km71 (PR #214) made formula - editing its own operation, with an interim id built from that pair (below). -- **An edit whose target was concurrently replaced cannot be told apart from an edit of - the replacement.** -- **A records-mode coordinate rename names its category and item by name.** - improvise-fdk1 (PR #221) landed `RenameRecordCoordinate { record, category, item }` - with `String` operands (below). - -**What already exists.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize` -aliases, assigned from monotone counters. **No production caller uses either value.** -`Category::add_item` returns the id on every call, and `Model::add_category` returns -one too. Tests read them (`category.rs`'s `add_item_returns_sequential_ids` and -`add_item_duplicate_returns_same_id`, and one in `types.rs`), but no production -caller does: `parse.rs` discards the value. Neither has ever been written to -`.improv`. `View` has no id. - -**`FormulaId` exists, as an interim.** km71 landed an opaque `FormulaId` whose private -fields are `(target, target_category)`, built only by `Formula::id()` -(`context/code/core.md` §18). It is on the wire in -`ModelCommand::ReplaceFormula { id, was, raw }`, in `FormulaSlot::Existing` and in -`FormulaEntry`. It was made opaque so that this ADR changes one struct rather than every -signature. It is still a name pair, so a formula whose target is renamed gets a -different `FormulaId`. `Model::replace_formula` therefore refuses to rename a target -outside `_Measure` until this ADR lands (improvise-kb1d). - -**Legacy JSON is gone.** improvise-tt22 landed in PR #224, so `.improv` is the only -native format. +**What exists already.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize` +aliases from monotone counters. No production caller reads them, they have never been +written to `.improv`, and `View` has no id. km71 landed an opaque interim `FormulaId` +over `(target, target_category)`, built only by `Formula::id()` (`context/code/core.md` +§18). It is on the wire in `ModelCommand::ReplaceFormula { id, was, raw }`, +`FormulaSlot::Existing` and `FormulaEntry`. Because it is still a name pair, +`Model::replace_formula` refuses to rename a target outside `_Measure` +(improvise-kb1d). Legacy JSON is gone (improvise-tt22, PR #224), so `.improv` is the +only native format. ## Decision -**Categories, items, formulas and views each have an id, and the id is their key.** A -category or item's current `name` field *becomes* the id (the user's suggestion), and the +**Categories, items, formulas and views each have an id, and so do groups +[Amendment]. The id is the key.** The current `name` field becomes the id, and the human name moves to a new `display` field. -- **Every reference is an id.** That covers `CellKey`s, axis maps, page selections, - hidden and collapsed sets, formula references, data keys, and wire operands that name - an existing thing. -- **Display names are for people.** They appear on screen, are typed into minibuffers, - and are resolved to ids at the input edge. - -Twelve clauses. +- **Every reference is an id**: `CellKey`s, axis maps, page selections, hidden and + collapsed sets, formula references, data keys, and every wire operand that names an + existing thing. A reference to the stable identity cannot glitch when something is + renamed or replaced. +- **Display names are for people.** They appear on screen, are typed into + minibuffers, and are turned into ids at the input edge. ### 1. What carries an identity -| kind | id | unique within | wire address | -|---|---|---|---| -| category | yes | the document, across every kind (answer 22) | `CategoryId` | -| item | yes | the document, across every kind (answer 22) | `ItemId` | -| formula | yes | the document, across every kind (answer 22) | `FormulaId` | -| view | yes (the user's example) | the document, across every kind (answer 22) | `ViewId` | -| record | `RecordHash`, now over ids (clause 8) | the document | unchanged shape | -| group | **yes** (answer 44, overriding answer 9) | the document, across every kind (the ADR's reading of answers 22 and 37; Q33) | to be decided (Q33) | +| kind | id | wire address | +|---|---|---| +| category | yes | `CategoryId` | +| item | yes | `ItemId` | +| formula | yes | `FormulaId` | +| view | yes | `ViewId` | +| group | **[Amendment]** yes (open questions 9 and 10) | open question 10 | +| record | `RecordHash`, over ids (clause 8) | unchanged shape | -**Ids are unique in the whole document** (answers 22 and 37: "An id always refers to -the same thing and so must be globally unique"). One namespace covers categories, -items, formulas and views, and on the ADR's reading groups too (Q33), so an item id -names one item without its category. A -`CellKey` still pairs each item with its category, because the key needs the -categories. An operation that creates an item still names the parent category. +**An id always names the same thing, so ids are unique in the whole document**, across +every kind. An item id names one item without its category. A `CellKey` still pairs +each item with its category, and an operation that creates an item still names the +parent category. A formula whose generated id would collide takes a suffix like any +other generated id (clause 4). A formula `Tax` in `CustomCat` gets item id `tax`, then +formula id `tax-`. **[Amendment]** Formulas may refer to anything in a typed +context. What exactly that allows is open question 2. -**Virtual categories have reserved ids** (answers 23 and 43): `_index`, `_dim` and -`_measure`, with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin -with `_`**: a leading `_` "is reserved for virtual categories and measures and other -entities" (answer 42). Moving today's spellings to these ids is Q32. -**Measures are the normal formula target** (answers 35 and 36), and **a measure -has an id and a display**. A stored measure is an ordinary item. Revision 9 says a -`_Measure` formula target has no item of its own and that its handle is its -formula's `FormulaId`; that is **under question again (Q23)**, because it is -the normal case. On revision 9's reading, this is why the category panel's `_Measure` rows need -`ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). **A formula -target outside `_Measure`, the rare case (answer 36), is a real item** (answer 30). It has its own `ItemId`, -and its formula has a separate `FormulaId`. Generated ids follow clause 4, so a -formula `Tax` in `CustomCat` gets item id `tax` and then formula id `tax-` -(answer 37: "if a formula has an id that would conflict, increment the counter"). -The panel lists that row as `ItemHandle::Item`, like any other item. +**Reserved ids.** The virtual categories have ids `_index`, `_dim` and `_measure`, +displayed `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** +**[Amendment]** A leading `_` is reserved for virtual categories, measures and other +virtual entities. The virtual views have ids `_records` and `_drill`, displayed +`_Records` and `_Drill`. They are never persisted, but they are addressed on the wire +(`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could collide +with a user's view. Which entities the reservation covers is open question 6. Whether +records mode's synthetic positions under `_Index` and `_Dim` are ids is open question 7. +Moving today's capitalised keys to these ids is open question 8. -**The virtual views have reserved ids too**: `_records` and `_drill`, with the -displays `_Records` and `_Drill`. They are never persisted, but they are addressed on -the wire (`SwitchView`, improvise-obtu). Revision 9 introduced this as its own -reading, extending answer 23's reservation from the virtual categories to the -virtual views, because a slugified `_Drill` would otherwise be `drill`, which could -collide with a user's view (improvise-p003 item 1). Answer 42 now reserves a leading -`_` for "virtual categories and measures and other entities". **That the virtual -views are among the "other entities" is the ADR's reading** (Q30). - -**The `_Index` and `_Dim` categories have ids `_index` and `_dim`** (answers 23 and -43). Revision 9 read their *coordinates*, the synthetic record positions (`_Dim` -carries a column index, gotcha 1), as outside the id namespace, so that `_index`'s -`0` could not collide with a user item whose id is `0` (improvise-p003 item 2). **The -user rejected that reading as worded** (answer 43, "No, they have ids _index and -_dim"). The ADR does not read the answer as deciding what the positions are: that -is **Q31**. Until it is answered, nothing in this ADR depends on the positions' -spelling, because they are never persisted and never typed. +**Formula targets.** **[Amendment]** A measure is the normal formula target +(`profit = revenue - cost`). A dimension (a category other than `_Measure`) is a rare +one, for a derived dimension. +- **A target outside `_Measure` is a real item**, with its own `ItemId`, and its + formula has a separate `FormulaId`. +- **A `_Measure` target has no item of its own**, and its handle is its formula's + `FormulaId`. So the category panel's `_Measure` rows use + `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). +- **[Amendment]** A measure has an id and a display, as everything else does. + Whether a formula-target measure keeps the formula's id or becomes an item with + its own is open question 1. The `ItemHandle` text in clauses 2, 5, 6, 8 and 11 + depends on its answer. ### 2. Storage keys by id; `display` is the only place a human name lives -Following answers 1 to 3: - **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id, and each `Category`, `Item` and `View` gains a `display: String`. -- **Cells and derived surrogates:** a `CellKey` is a sorted list of - `(CategoryId, ItemHandle)` pairs (the ADR's own reading of answer 30, not a - user answer), where `ItemHandle` is clause 6's - `{Item(ItemId), FormulaTarget(FormulaId)}`. Outside `_Measure` every coordinate is - an `Item`. A `_Measure` formula cell's coordinate is `FormulaTarget`, because - that target has no item (**pending Q23**: option (c) gives it an item, and this - reading falls). The key stays sorted, now **by category id**, so gotcha - 5's canonical form is kept with a different sort key. `DataStore` interns ids - exactly as it interns names today. Answer 22 makes every id unique across kinds, - so an id's text alone names one thing; the tag is there so that the types - never claim an `ItemId` the document does not hold. Revision 9's first draft - wrote this pair as `(CategoryId, ItemId)` while also putting a `FormulaId` in it - (Codex on PR #326). +- **Cells:** a `CellKey` is a sorted list of `(CategoryId, ItemId)` pairs. + **[Amendment]** The pairs are `(CategoryId, ItemHandle)`, so that a `_Measure` + formula cell's coordinate is `FormulaTarget` without claiming an `ItemId` the + document does not hold (pending open question 1). + - The key stays sorted, now **by category id**, so the canonical form (gotcha 5) + is kept with a different sort key. + - `DataStore` interns ids exactly as it interns names today. + - An id's text names one thing, because ids are unique across kinds. - **View configuration:** the axis maps, page selections, and hidden and collapsed sets - are keyed by ids, with no exception. A collapsed entry names a group, and a - group name is an id (answer 44), so the entry is the category id and the group's - id. Revision 9 had made it `(CategoryId, group name)`, because answer 9 gave groups - no id (improvise-p003 item 4); answer 44 overrides both. Whether the category id - is still needed beside a group id that is unique in the document, and how the - line is spelled, are part of Q33. -- **Formulas:** a formula's references are ids (clause 5). The formula panel renders - them as display names (answer 19; clause 6). + are keyed by ids. **[Amendment]** A collapsed entry names its group by the group's + id. Whether the category id stays beside it is part of open question 10. +- **Formulas:** a formula's references are ids (clause 5), and the formula panel + renders them as display names (clause 6). -**What a rename now costs:** one `display` write. It moves no `CellKey`, no `RecordHash` -and no `layout_version`, so an edit a client composed before the rename still lands after -it. That is the "avoid glitches" the user's rule is for. The grid's labels are display -strings, so the rename's effect must still dirty the grid and the panels -(`Dirty::of` gotcha 35), but no basis tier moves. +**A rename costs one `display` write.** It moves no `CellKey`, no `RecordHash` and no +`layout_version`, so an edit a client composed before the rename still lands after it. +The grid's labels are display strings, so a rename must still dirty the grid and the +panels (`Dirty::of`, gotcha 35), but no basis tier moves. -**What the migration costs, stated honestly.** Every API that takes a name today -(`Model::category(name)`, `Workbook::add_item(cat, item)`, `CellKey::new` callers, the -formula evaluator's `Expr::Ref` lookup, `View` methods) takes an id instead. The -display→id resolution moves to the places a human types, such as the minibuffer commits, -the formula parser's reference resolution, and import. The change is mechanical but wide, -and it is the bulk of the implementation. The user judged it cheap ("at worst, it would -be renaming field access"), and for the field that is right. The call sites are the -cost. +**The migration cost is the call sites.** Every API that takes a name today takes an +id instead: `Model::category(name)`, `Workbook::add_item(cat, item)`, `CellKey::new`'s +callers, the evaluator's `Expr::Ref` lookup and the `View` methods. Display→id +resolution moves to the places a human types. The field rename itself is cheap. The +change is mechanical but wide, and it is the bulk of the implementation. -**Display names stay unique within their scope**, as names are today. So a human-typed -display resolves to at most one id within a scope, and a rename to a display name -already in use is refused. Typed text is matched against ids before displays -(answer 33; clause 12). An item's scope is its category, so two categories may each have an item -displayed `Revenue`. Formula text crosses categories, which is where answer 19's -"ambiguous" arises. Their ids differ (answer 22), so the panel resolves the ambiguity -by showing the ids. +**Display names stay unique within their scope**, as names are today, and a rename to +a display already in use there is refused. An item's scope is its category, so two +categories may each have an item displayed `Revenue`. Formula text crosses +categories, so a display may be ambiguous there. The ids differ, so the formula panel +shows the ids where a display is ambiguous. ### 3. Ids are interned symbols -Following answer 4, an id is a **string**, and in memory it is a `Symbol` from the -existing `SymbolTable`: copyable, with integer comparison. Two consequences. - -- **A `Symbol` never crosses the wire or reaches the file.** It is an index into one - process's table. The wire and `.improv` carry the id's **text**, and each end interns - it on arrival. -- **`Rejection` stops being `Copy`**, because `Rejection::Gone` (clause 7) carries an id's - text. `basis.rs`'s `SaveFailed` carries no message *because* `Rejection` was `Copy`, so - that reason lapses. Whether `SaveFailed` now gains its message is the implementer's - call. +An id is a **string**, and in memory a `Symbol` from the existing `SymbolTable`: +copyable, compared as an integer. +- **A `Symbol` never crosses the wire or reaches the file.** It indexes one process's + table. The wire and `.improv` carry the id's text, and each end interns it on + arrival. +- **`Rejection` stops being `Copy`**, because `Rejection::Gone` (clause 7) carries an + id's text. `SaveFailed` carries no message only because `Rejection` was `Copy`. + Whether it gains one is the implementer's call. ### 4. How an id is made: slugify, one rule everywhere -Following answers 8, 16 and 18, **every** id the app generates comes from the display -name by **slugify**. That covers import, interactive creation, and assigning ids to an -id-less `.improv` file at load (answer 16(a), "All id generation"). So a thing's id -never depends on how it was made. The steps, in order (answer 18, "casefold, replace -space with `_`, strip", as answers 34, 41, 42 and 45 refine it): +**Every** id the app generates comes from the display name by **slugify**: on +import, on interactive creation, and when an id-less `.improv` file is loaded. So a +thing's id never depends on how it was made. An id is lower case, in the alphabet +`a`–`z`, `0`–`9`, `-` and `_`. -1. **Casefold, with Unicode full case folding** (answers 16(b), 18 and 26). Rust's - `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped), so this - needs a case-folding table or a small crate in `improvise-core`. It must pass the - `wasm` gate, because that crate is in the browser bundle. -2. **Replace each whitespace character with `_`** (answers 18, 34 and 41; answer 18 - changed answer 8's `-`). "The spaces should have been normalized to _ before - collapsing runs of _s" (answer 41), so this step makes one `_` per character, and - step 4 does the collapsing. -3. **Strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`** (answers 16(c) - and 16(d)). This strips non-ASCII letters, so an id never needs quoting in an - `.improv` reference (clause 5 notes where formula text differs). -4. **Collapse each run of `_` to one `_`** (answers 21 and 41). It does not matter - whether an underscore came from whitespace or was in the display already. **The - step comes after the strip, which is the ADR's reading** (Q29): it is the order in - which answer 41's example, `Time \u{200B} Period`, gives one `_`. -5. **Remove every leading and trailing `_` and `-`**, until the slug neither begins - nor ends with either. - - A leading `_` "is reserved for virtual categories and measures and other - entities" (answers 23 and 42). Removing it is this ADR's way of meeting that - constraint, so `_Draft` → `draft`. - - A trailing `_` "should be dropped" (answer 42). - - "There should be no leading or trailing -s" (answer 45). - - Repeating the removal until neither character is left at either end (`-_x` → - `x`) is the ADR's reading (Q34). -6. **If nothing is left, use `-`** (answer 16(e)), such as `category-1` or - `item-4`. `n` comes from the same counter as every other suffix (answer 28), not - from the lowest free number. This covers a display that is only `-` or only `_`, - because "a slug of - should be invalid as an id" (answer 45). +The steps, in order, as the amendment states them **[Amendment]**: +1. **Casefold, with Unicode full case folding**, so `Straße` → `strasse`. Rust's + `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped). The folding + table or crate must pass the `wasm` gate, because `improvise-core` is in the browser + bundle. +2. **Replace each whitespace character with `_`.** "Whitespace" is the Unicode + `White_Space` property (`char::is_whitespace`): U+0009–U+000D, U+0020, U+0085, + U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. + - A tab, a no-break space and an ideographic space each become `_`. + - A zero-width space (U+200B) and a byte-order mark (U+FEFF) are not whitespace, + and step 3 strips them. + - The realistic case is a header pasted from a web page, where a no-break space + stands for a space. +3. **Strip every character outside the id alphabet.** This strips non-ASCII letters, + so an id never needs quoting in an `.improv` reference. +4. **Collapse each run of `_` to one `_`**, whether the underscores came from + whitespace or were in the display. The step comes after the strip (open + question 5). +5. **Remove every leading and trailing `_` and `-`**, repeating until neither end + holds either (open question 11). + - A leading `_` is reserved (clause 1). + - A trailing `_` is dropped. + - An id never begins or ends with `-`. +6. **If nothing is left, use `-`**, such as `category-1` or `item-4`. `n` + comes from the shared counter below. A display of only `-`, `_` or non-ASCII letters + ends here, because `-` alone is not an id. -Examples: `East` → `east`, `Time Period` → `time_period`, `Time Period` → -`time_period`, `Coastal_East` → `coastal_east`, `Q1-2024` → `q1-2024`, -`Region(code)` → `regioncode`, `Café` → `caf`, `Ü` → `category-`, `Straße` → -`strasse` (answer 26). By answer 41, `a_ b` → `a_b`, `a__b` → `a_b` and -`Time \u{200B} Period` → `time_period`. By answer 42, `East ` → `east` and `East_` → -`east`. By answer 45, `-East-` → `east`, and `-` and `---` → `-`. Step 4 -collapses only `_`, so a `-` between two spaces survives: `Q1 - 2024` → -`q1_-_2024`. The user has not said otherwise, and this ADR adds no rule for it. +The accepted text has five steps instead: casefold; replace each **run of spaces** +with one `_`; strip; remove any **leading** `_`; `-` if empty. -`Time Period` and `Time_Period` slug alike, so whichever comes second gets -`time_period-`. So do `East`, `East ` and `East_`, and the second and third each -take a suffix ("disambiguate with -#", answer 42). `East` in `Region` and `East` in -`Direction` slug alike too, and since ids are global (answers 22 and 37), the second is -`east-`. Here `` is the counter's next value, for example `east-1`. The suffix -never puts a `-` at either end, so a suffixed id still meets step 5. +Examples under the amendment: -**The ADR's own reading of the word "whitespace"**, which answer 34 requires and does -not define. It is not a user answer. Revision 9 had a second reading here, that a slug -of only `-` is a legal id. Answer 45 rejects it, so step 5 removes it. -- **"Whitespace" means the Unicode `White_Space` property**, the set Rust's - `char::is_whitespace` tests. That is U+0009–U+000D, U+0020, U+0085, U+00A0, - U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. It needs no - table in `improvise-core`. A tab, a no-break space (U+00A0) and an ideographic - space (U+3000) are whitespace, so each becomes `_` like a space does: - `Time\tPeriod`, `Time\u{A0}Period` and `Time Period` all slug to `time_period`, - and so does `Time \t Period`, whose three `_` collapse at step 4. A zero-width space - (U+200B) and a byte-order mark (U+FEFF) are **not** `White_Space`, so step 3 - strips them. The realistic case is a header pasted from a web page, where a - no-break space stands for a space. The ADR chose this set because it is the standard - definition and the user asked for "whitespace", not for a list. The user may - prefer a narrower one (ASCII whitespace only, say), and that is a one-line change. - Revision 9 asked here whether `a_ b` keeps two underscores (Q24) and whether a - trailing run leaves a trailing `_` (Q25). Answers 41 and 42 settle both, and steps - 4 and 5 follow them. +| display | id | +|---|---| +| `East` | `east` | +| `Time Period`, `Time Period`, `Time\tPeriod`, `Time\u{A0}Period`, `Time \u{200B} Period` | `time_period` | +| `Coastal_East` | `coastal_east` | +| `a_ b`, `a__b` | `a_b` | +| `Q1-2024` | `q1-2024` | +| `Q1 - 2024` | `q1_-_2024` (only `_` runs collapse) | +| `East `, `East_`, `-East-` | `east` | +| `_Draft` | `draft` | +| `Region(code)` | `regioncode` | +| `Café` | `caf` | +| `Straße` | `strasse` | +| `Ü`, `-`, `---` | `-` | -**What this changes in built code (a follow-up, not made here).** PR #318's -`slugify` (`crates/improvise-core/src/model/id.rs`) and `context/code/core.md` §20 -predate answers 34, 41, 42 and 45, and differ from steps 2–6 in five ways: -- `underscore_space_runs` tests `c == ' '`, so `Time\tPeriod` slugs to - `timeperiod` (answer 34); -- it collapses runs of spaces before the strip and never collapses `_`, so `a_ b` → - `a__b` and `Time \u{200B} Period` → `time__period` (answer 41); -- it removes only leading `_`, so `East ` → `east_` (answer 42); -- it never removes `-`, so `-` slugs to `-` and `-East-` to `-east-` (answer 45); -- `IdAllocator::claim` accepts `-`, `-x` and `x-`, which Q34 would refuse. +**Collisions.** `Time Period` and `Time_Period` slug alike, and so do `East`, `East ` +and `East_`. Whichever comes later takes a suffix. `East` in `Region` and `East` in +`Direction` also slug alike, and ids are global, so the second is `east-`. +- **Any colliding generated id gets a suffix**, whatever kind of thing holds the + slug: on import, when an id-less file loads, and when the host creates something in + a session. An id that does not collide stays bare. +- **The suffix is `-` plus the next value of one increment-only counter per + document**, as a plain number (`east-1`, not `east-001`). The counter works like + Common Lisp's `gensym`: it never goes down and is never reused, so no order holds + among suffixes (`east-7` may follow `east` directly). A suffix ends in a digit, so a + suffixed id still meets step 5. +- **Retirement.** Within a session a removed id is never handed out again. The + counter is past any suffix it issued, and a removed bare slug stays reserved in + memory. +- **Determinism.** The counter advances in file order or creation order, never in + hash order (gotcha 18), so the same input gives the same ids. +- **The counter** lives in memory, starts at 1 when a document loads, and is not + written to the file. -The tests `slugify_keeps_an_underscore_the_display_already_had` and -`slugify_collapses_a_run_of_spaces_to_one_underscore` assert the superseded results. -Answers 34, 41, 42 and 45 are all folded into revision 9, which the user has not yet -approved, so the follow-up should wait for that approval. Nothing calls `slugify` -yet (§20), so no stored id is affected. +**Hand-written ids are canonical.** An id written in a file is never renamed, because +every data key and reference in that file spells it. So: +- **a file has ids on everything or on nothing.** An id-less file (an old version, + or a CSV or JSON import) gets every id from slugify. A new-version file must carry + an id on every declaration, and one that does not is refused (clause 5); +- the suffix only ever goes on the id being generated. When a generated candidate + `slug-` is taken by a file's id, the counter advances until a candidate is free; +- **two identical hand-written ids in one file are refused at load**, even when they + declare different kinds of thing. Renaming one would corrupt the data section, and a + file with a duplicate cannot be resaved safely. The error names both lines. -The remaining rules: +**Reuse across a save and reload is accepted.** Retired ids are not persisted. So +remove `East`, save, reload and add `East`, and the new item gets `east` again. An +in-flight command from before the reload is refused by the generation tier. -- **A file has ids on everything or on nothing** (answer 21). An id-less file (old - version, or a CSV or JSON import) gets every id from slugify. A new-version file must - carry an id on every declaration, and one that does not is refused (clause 5). So no - file mixes hand-written and generated ids. -- **Any colliding id gets a suffix, `gensym`-style** (answers 21, 27, 28, 37 and 42). - - **When:** whenever a generated slug is already taken, whatever kind of thing holds - it. That applies on import, when an id-less file loads, and when the host creates - something in a session. An id that does not collide stays bare. - - **What:** `-` and the next value of **one increment-only counter per document**, - as a plain number (`east-1`, not `east-001`). The counter never goes down and is - never reused, so no order is enforced among suffixes: `east-7` may follow `east` - directly. - - **Retirement:** within a session a removed id is never handed out again. The - counter has already moved past any suffix it issued, and a bare slug that was - removed is still reserved in memory. - - **Determinism:** the counter advances in file order or creation order, so the same - input gives the same ids, never hash order (gotcha 18). - - **Hand-written ids are canonical** (answer 29). The suffix only ever goes on the - id being generated. An id read from a new-version file keeps its spelling, even - when it looks like a slug or a suffixed one (`east-3`), because the file's data - keys and references spell it. So when a generated candidate `slug-` is already - taken by a file's id, the counter advances again until the candidate is free. - - **Two identical hand-written ids in one file are refused at load** (answers 11, - 22 and 29). By answer 22 they are a duplicate even when they declare different - kinds of thing, such as a category and an item. Renaming either one is exactly - what answer 29 forbids, and answer 11 already refuses a duplicate because such a - file "can't safely resave". The error names both lines (clause 5). This follows - from the answers, so it is not asked as a new question. - - **The counter** lives in memory and starts at 1 when a document loads. It is not - written to the file, since retired ids are not persisted (answer 13). That is - this ADR's reading. -- **Across a save and reload, reuse is accepted** (answer 13, Q10(b)). Retired ids are - not persisted. So remove `East`, save, reload and add `East`, and the new item gets - `east` again. An in-flight command from before the reload is refused by the - generation tier. The one remaining risk is an outside reference that outlives the - file, and none exists today. -- **Two branches of one file can merge into a file that will not load.** If each - branch adds a different thing whose id generates to the same slug, the merged - file declares that id twice. Its load is refused (answers 11 and 29), and the - error names both declarations, so the merge must be repaired by hand. Stable - identity across copies and branches of a file is not promised (improvise-8tk5). +**[Amendment] Two branches of one file can merge into a file that will not load.** If +each branch adds a different thing that generates the same slug, the merged file +declares that id twice. The load is refused and names both declarations, so the merge +is repaired by hand. Stable identity across copies and branches of a file is not +promised (improvise-8tk5). + +**Built code.** `crates/improvise-core/src/model/id.rs` (`slugify`, `IdAllocator`) +implements the accepted steps, and `context/code/core.md` §20 describes them. Once +the amendment is approved, a follow-up must: +- test whitespace, not only U+0020; +- collapse `_` after the strip; +- trim a trailing `_` and both ends' `-`; +- settle what `claim` refuses (open question 11); +- add the virtual views to `VIRTUAL_IDS`; +- add `IdKind::Group` if groups are generated (open question 10). + +Nothing calls `slugify` yet. ### 5. Persistence: ids on declarations *and* references -Following answers 3 and 7, the file now carries ids everywhere: -- **Declarations** (section headers, item lines, formula lines) carry the display name - and the id. -- **Every reference** carries the id: axis lines, `hidden:` and `collapsed:`, page - selections, `## Aggregation`, `Initial View:`, `## Data` keys, and names inside formula - text (answer 3). The formula panel shows display names instead (answer 19). +- **Declarations** (section headers, item lines, formula lines) carry the display and + the id. +- **Every reference** is the id: axis lines, `hidden:` and `collapsed:`, page + selections, `## Aggregation`, `Initial View:`, `## Data` keys (ids are guaranteed + unique, names might not be), and names inside formula text. -**Id alphabet.** Every id, generated or hand-written, is spelled only in `a`–`z`, -`0`–`9`, `_` and `-` (clause 4; answer 20). The grammar spells them **with ranges**, -because the generator drops `ASCII_ALPHANUMERIC` (improvise-rxz9). So an id never needs -pipe quoting in an `.improv` declaration or reference. A header ending in a -parenthesised run that contains a capital, such as `Sales(Copy)`, does not match -`id_tag`, and neither does an empty or trailing-space tag. In a new-version file that -declaration has no id, so the load is refused (answer 21), and the error names the -line. Revision 4 generated an id and warned. A hand-written id beginning with `_`, -other than the reserved virtual ids, is refused too (answers 23 and 42). A -hand-written id that is `-`, or that begins or ends with `-`, is refused the same -way. That is the ADR's reading of answer 45 ("a slug of - should be invalid as an id -and there should be no leading or trailing -s") for an id the program did not -generate (Q34). Whether a hand-written id ending in `_` is refused is also Q34: -answer 42 drops a trailing `_` from a slug, and does not say whether a written id may -carry one. +**The id alphabet.** Every id is spelled only in `a`–`z`, `0`–`9`, `_` and `-`. The +grammar spells it **with ranges**, because the generator drops `ASCII_ALPHANUMERIC` +(improvise-rxz9). So an id never needs pipe quoting in a declaration or reference. + +**What the loader refuses** in a new-version file: +- a declaration with no id. A header ending in a parenthesised run with a capital + (`Sales(Copy)`), an empty tag or a tag with a trailing space does not match + `id_tag`, so it has no id. The error names the line; +- a hand-written id beginning with `_`, other than a reserved one; +- **[Amendment]** a hand-written id that is `-` (open question 11 asks whether a + leading or trailing `-`, or a trailing `_`, is refused too). **Inside formula text an id may still need quoting.** The formula grammar's `bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So -the writer spells an id in formula text through `quote::quote_name`, as it spells a -name today. An id beginning with a digit (`2024`) or equal to a keyword (`sum`, `if`, -`where`, `is`) is written `|2024|` or `|sum|`. No new rule is needed, but "never needs -quoting" does not extend to formula text. +the writer spells an id in formula text through `quote::quote_name`. An id beginning +with a digit (`2024`) or equal to a keyword (`sum`, `if`, `where`, `is`) is written +`|2024|` or `|sum|`. ```pest id_token = @{ ('a'..'z' | '0'..'9' | "_" | "-")+ } -// The space is optional (answer 6). +// The space before the parenthesis is optional. id_tag = ${ " "* ~ "(" ~ id_token ~ ")" } -// Declarations: display name, then its id. The id is optional in the grammar so the -// loader, not the parser, refuses a missing one and can name the line (answer 21). +// Declarations: display, then id. The id is optional in the grammar so that the +// loader, not the parser, refuses a missing one and can name the line. header_decl = { pipe_quoted ~ id_tag ~ &NEWLINE | header_display ~ id_tag ~ &NEWLINE | line_name } header_display = @{ (!(id_tag ~ NEWLINE) ~ !NEWLINE ~ ANY)+ } item_decl = { name ~ id_tag? } grouped_item = { "- " ~ name ~ id_tag? ~ "[" ~ name ~ "]" ~ NEWLINE ~ blank_lines } -// References: an id, never a display name. +// References: an id, never a display. id_ref = { id_token } ``` -`model_name` has no id, so it keeps `header_name`. `Initial View:` becomes a reference -and takes an `id_ref`. **A group reference (`[Coastal]`, and the group in a -`collapsed:` line) becomes an id** (answer 44, overriding answer 9's "stays a name"). -The grammar above does not yet spell it, and neither does the group definition -(`> Coastal`), which becomes a declaration carrying a display and an id. Those -spellings are Q33. +`model_name` has no id and keeps `header_name`. `Initial View:` is a reference and +takes an `id_ref`. -**An axis line is tried before `format:`, and a format that reads as an axis line is -pipe-quoted** (improvise-le4p; the ADR's own reading, not a user answer). -`view_entry` is an ordered choice, and `format_line` takes the rest of the line, -so a category whose id is `format` would lose its axis line to the view's number -format. In the new-version arm, `axis_line` therefore comes before `format_line`. -That moves the collision to the other side. `set-format` stores its argument -verbatim and the writer emits `format: {number_format}`, so `:set-format none` -writes `format: none`, which would now read as the axis line of a category -`format`: the format is lost, or the id is unknown and the load is refused -(answer 11), so a document that saves would not load. +**Group references.** A group reference (`[Coastal]`) is a name. +**[Amendment]** It is the group's id. So is the group in a `collapsed:` line. The +group definition (`> Coastal`) declares the id. The spellings are open question 10. -The rule is that **the writer pipe-quotes a number format whenever the line would -also match `axis_line`**, that is, when the value is an axis keyword (`row`, -`column`, `page`, `filter` or `none`) alone or followed by `, `, or when it -begins with `|`. The new arm's `format_line` takes -`pipe_quoted ~ &NEWLINE | rest_of_line`, as `header_name` does (gotcha 34). So -`format: |none|` is the format `none`, and nothing is refused. An old-version -file's `format: none` is read as before, because the old arm keeps `format_line` -first, and it is written quoted on its first save. The alternatives were to refuse -such a format in `set-format`, which does nothing for a file that already holds -one, or in the writer, which leaves a document that cannot be saved. The id -`format` needs no quoting, and `id_ref` keeps no quoting arm. +**[Amendment] An axis line is tried before `format:`, and a format that reads as an +axis line is pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice, and +`format_line` takes the rest of the line. So a category whose id is `format` would +lose its axis line to the view's number format, and the new-version arm puts +`axis_line` first. +- **The collision then moves.** `set-format` stores its argument verbatim, so + `:set-format none` writes `format: none`, which would now read as the axis line of a + category `format`. +- **So the writer pipe-quotes a number format whenever the line would also match + `axis_line`**: an axis keyword (`row`, `column`, `page`, `filter`, `none`) alone or + followed by `, `, or a value beginning with `|`. The new arm's `format_line` is + `pipe_quoted ~ &NEWLINE | rest_of_line`, so `format: |none|` is the format `none`. +- **An old-version file's `format: none`** is read as before (its arm keeps + `format_line` first), and it is written quoted on its first save. +- **Required tests:** + - a category displayed `Format`, on a row axis, round-trips; + - a view whose number format is each axis keyword round-trips in a document where + that view also holds a category `format` on an axis. -Required tests: a category displayed `Format`, on a row axis, round-trips; and a -view whose number format is each axis keyword round-trips, in a document where -the same view also holds a category `format` on an axis. - -**A formula line** (improvise-m56i; answers 3, 5, 22 and 30). **Under question -again by answers 35 and 36, because the `_Measure` target is the normal case (Q23).** -The shape below is revision 9's. It is the ADR's -own reading, not a user answer, except that the formula id's place is answer 5's and -an item target's id is answer 30's. It is -`- () = []`: -- **The parenthesised id is the formula's own `FormulaId`** (answer 5's - placement). +**[Amendment] A formula line** is +`- () = []` (pending open +question 1): +- **The parenthesised id is the formula's `FormulaId`**, between the target and the + `=`. - **`` is the target item's id when the target is an item** (outside - `_Measure`, answer 30). For a `_Measure` target, which has no item, it is the - target's display. - - `Formula` carries that display, and a display rename of a `_Measure` target - writes it. Where that display lives is also the ADR's own reading. - - Whether one line should spell both cases this way is **Q23**, below, and so - are how an expression refers to a target and how a `_Measure` display is quoted. -- **The expression is over ids**, spelled through `quote::quote_name` as above. A - reference to a formula target is the id text of its `ItemHandle`: the item id - outside `_Measure`, and the formula id for a `_Measure` target. -- **A `WHERE`/`IS` filter value follows the same rule** (answer 3: every reference - is an id). It is the id text of an `ItemHandle` in the filtered category, bare or - quoted: an item id, or a formula id when the category is `_Measure` and the value - names a formula target. The formula panel shows it as a display. -- **A formula cell's `CellKey`** carries `ItemHandle::Item()` - outside `_Measure`. For a `_Measure` target it carries - `ItemHandle::FormulaTarget()` under `_measure` (clause 2). + `_Measure`). For a `_Measure` target, which has no item, it is the target's + display, held on the `Formula`. +- **The expression is over ids**, spelled through `quote::quote_name`. A reference to + a formula target is the id text of its `ItemHandle`: the item id outside `_Measure`, + the formula id for a `_Measure` target. +- **A `WHERE`/`IS` value follows the same rule**. It is the id text of an + `ItemHandle` in the filtered category, and the formula panel shows it as a display. +- **A display in the target position that looks like `Name(x)` is pipe-quoted**, as in + `- |Margin(%)|(margin) = …`. An item id never contains a parenthesis. -**`header_display` is guarded only by a lookahead**, which the generator skips (gotcha -34). So the generator may emit a display ending in something that later parses as an id. -Round-trip property 3 still holds, because the first parse settles the split. But +**`header_display` is guarded only by a lookahead**, which the generator skips +(gotcha 34). So the generator may emit a display ending in something that parses as an +id. Round-trip property 3 still holds, because the first parse settles the split, but property 1 must be stated over the parsed result, not over the generated text. -**Which regime a file is read under is decided by its version line alone** (clause -10). In an old-version file, a header is the whole line, a reference resolves against -display names, and ids are assigned by slugify. That resolution happens in the loader, -not in a client, which answer 39 may or may not allow (Q28). In a new-version file, declarations -carry ids and a reference is an id. +**The version line alone picks the regime** (clause 10). +- **In an old-version file** a header is the whole line, a reference resolves against + display names, and ids are assigned by slugify. +- **In a new-version file** declarations carry ids and a reference is an id. -**An old file naming an item it never declares** declares it at load (answer 24). This -happens today when a `## Data` line names an undeclared item (improvise-98pd, second -source). The item gets a slug like any other, and the first save writes it, so a file -that loads today keeps loading (the lesson of improvise-80rm). The NonItem work -(improvise-gk51, PR #265) projects such a cell as `CellId::NonItem`. From a loaded file -that case no longer arises. Ids do not settle 98pd's other two sources, `set-cell` and -a stale page selection: a page selection holds an item id, and removing the item leaves -it dangling exactly as a removed name does today. +**An old file naming an item it never declares** declares it at load. This happens +today when a `## Data` line names an undeclared item (improvise-98pd). The item gets a +slug like any other, and the first save writes it, so a file that loads today keeps +loading (improvise-80rm). Ids do not settle 98pd's other two sources, `set-cell` and a +stale page selection. A page selection holds an item id, and removing the item leaves +it dangling, as a removed name does today. -**Slugify folds case (answer 16(b)), so old data keys change spelling on first save.** -An old file's `Region=East` resolves at load through the display `East`. The first -save writes the id: `Region=East` becomes `region=east`. The data is the same, but a -`git diff` across that first save touches every data line. +**Old data keys change spelling on the first save.** Slugify folds case, so an old +file's `Region=East` resolves at load through the display `East` and is written +`region=east`. The data is the same, but a `git diff` across that save touches every +data line. **Round-trip properties:** 1. an id-less (old-version) file loads, gets ids, and is written in the new version on @@ -1047,137 +376,114 @@ save writes the id: `Region=East` becomes `region=east`. The data is the same, b 2. a new-version file loads with every id kept verbatim; 3. `format(parse(format(parse(x)))) == format(parse(x))`. -A **formula-line generator strategy** is required. The grammar walker emits no `=` and +A **formula-line generator strategy** is required. The grammar walker emits no `=`, and the walker drops such lines, so without its own strategy every formula property is vacuous (improvise-rxz9). -**A duplicate, unknown or missing id in a new-version file refuses the load** (answers -11, 21, 22 and 29; a duplicate across two kinds of thing counts, and a hand-written id -is never renamed to resolve one; answer 11: -"don't load the file because you can't safely resave it"). The error names the id, -and for a duplicate it names both declarations. Repairing silently is ruled out, -because references carry ids (answer 3): reassigning a duplicate would silently change -what its references point at. +**A duplicate, unknown or missing id in a new-version file refuses the load.** The +error names the id, and for a duplicate both declarations. A silent repair is ruled +out, because reassigning an id would silently change what its references point at. -**Legacy JSON is removed** (improvise-tt22), so it needs no migration here. - -**Import** (CSV, JSON) never parses ids out of dimension names (answer 8). A header -`Region(code)` is the display name `Region(code)`, and its id is its slug. The writer -pipe-quotes any display name that would otherwise read as display plus id: +**Import** (CSV, JSON) never parses ids out of dimension names. A header `Region(code)` +is the display `Region(code)`, and its id is its slug. The writer pipe-quotes any +display that would otherwise read as display plus id: `## Category: |Region(code)|(regioncode)`. -### 6. The wire: address by id, create by display name, learn ids from projections +### 6. The wire: address by id, create by display, learn ids from projections -- **The host assigns every id** (answer 14). A client never proposes one. +- **The host assigns every id.** A client never proposes one. - **Every operand naming an existing thing is an id.** The one operand naming the thing - being created is a display name, and the host slugifies it: + being created is a display, and the host slugifies it: - `AddCategory(display)` and `CreateView(display)`; - - `AddItem { category: CategoryId, display }`, - `AddItemInGroup { category: CategoryId, display, group }` and - `AddFormula { target_category: CategoryId, raw }`. - - Since groups have ids (answer 44), `AddItemInGroup`'s `group` names an existing - group by its id, or creates one by its display. Which, or both, is Q33. -- **The client is the only place a display name is accepted as a reference** - (answers 32 and 39). It converts the name to an id as soon as it can, in the UI - layer, and the wire never carries a display as a reference. The one display an - operand carries is the name of the thing being created. + - `AddItem { category: CategoryId, display }`; + - `AddItemInGroup { category: CategoryId, display, group }`. Whether `group` is an + existing group's id or a new group's display is open question 10 **[Amendment]**; + - `AddFormula { target_category: CategoryId, raw }`. +- **[Amendment] Only a client accepts a display name as a reference**, in its UI + layer, and it converts the name to an id as early as it can. The wire never carries + a display as a reference. Whether the old-version loader and headless `improvise + cmd`/`script` are exceptions is open question 4. - **A frame cannot address a thing it creates**, because it does not know the id the - host will assign. So a create-then-act compound becomes **one command** carrying - both halves. `create-and-switch-view` sends `[CreateView, SwitchView]` today and - becomes `CreateView { display, switch: true }`. Any other compound of this shape - gets the same treatment. -- **New operations are born with ids:** + host will assign. A create-then-act compound becomes **one command**: + `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes + `CreateView { display, switch: true }`. Any other compound of that shape gets the + same treatment. +- **New operations carry ids:** - `RenameCategory { category, to: display }`; - `RenameItem { category, item, to: display }`; - - `ReplaceFormula { id, was, raw }`, which km71 already landed with an interim id - (below); - - fdk1's `RenameRecordCoordinate`, which landed with name operands (below); + - `ReplaceFormula { id, was, raw }`; + - `RenameRecordCoordinate`; - the removes (`RemoveCategory`, `RemoveItem`, `RemoveFormula`, `DeleteView`). -- **Older variants** (`SetAxis`, `SetPageSelection`, `HideItem`, …) **must also move to - ids**. Their operands are references, and a reference is now an id. They may move one - whole variant at a time. That includes `SetCell { view, … }` and - `ClearCell { view, … }`, which improvise-wpxr (PR #281, merged) gave a view named by - its name, like every other view-scoped variant (improvise-lltt). +- **Older variants move to ids too** (`SetAxis`, `SetPageSelection`, `HideItem`, …, + including `SetCell { view, … }` and `ClearCell { view, … }`), one whole variant at a + time. - **Projections carry id and display together:** - - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }` - (**pending Q23**: under option (c) there is no `ItemHandle`); - - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows - (answer 19): the target's display name with its id in parentheses, then the - expression over display names, with an id wherever a display name is ambiguous - (ids are unique, answer 22). A display that equals another thing's id is - ambiguous in this sense (the ADR's own reading of answers 19 and 33, flagged - under Q23), so the panel never shows a bare name that the ids-first rule would - bind elsewhere. `stored` is the id-form text the file carries, and it is what - `ReplaceFormula`'s `was` is compared against. So a rename of `Revenue` elsewhere - changes what the panel shows but does not refuse an edit of `Profit`, which is - the "avoid glitches" of answer 3. What makes that hold for an edit already open - is clause 12; + - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }` (pending open + question 1); + - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows: + the target's display with its id in parentheses (`Profit (profit) = Revenue - + Cost`), then the expression over display names, with an id wherever a display is + ambiguous. **[Amendment]** A display equal to another thing's id counts as + ambiguous, so the panel never shows a bare name that the ids-first rule (clause 12) + would bind elsewhere. `stored` is the id-form text the file carries, and + `ReplaceFormula`'s `was` is compared against it. So a rename of `Revenue` elsewhere + changes what the panel shows, but does not refuse an edit of `Profit`; - `ViewSummary { id, display, … }`. `AxisFrame` labels stay display strings. They gain ids when a consumer needs them. -- **`ViewState` keys by id**: the cursors, `expanded_cats`, the view history - (`view_back_stack` / `view_forward_stack`) and `active_view` itself, as does - `Workbook::active_view` (improvise-p003 item 3). Answer 1 makes this natural - rather than a migration of its own. A view or category rename then orphans - nothing. -- **A `_Measure` formula-target row** (`ItemHandle::FormulaTarget(FormulaId)`; under - question again, Q23): - removing it means `RemoveFormula`, and renaming it means a display rename of - the formula's target (improvise-i0n7). -- **CheckedBatch's created-record exemption goes** (the ADR's own reading, not a - user answer; improvise-p003 item 5). It - exempts a record that neither resolves nor is claimed, for frames like - `[AddItem(East), SetCell(East)]`. Under this clause a frame cannot address what - it creates, so nothing produces such a frame any more. The wire step removes - the exemption rather than leaving dead reasoning in place. +- **`ViewState` keys by id**: the cursors, `expanded_cats` and the view history + (`view_back_stack` / `view_forward_stack`). **[Amendment]** So do `active_view` + itself and `Workbook::active_view` (improvise-p003). A rename then orphans nothing. +- **[Amendment] A `_Measure` formula-target row** (`ItemHandle::FormulaTarget`) is + removed by `RemoveFormula` and renamed by a display rename of the formula's target + (improvise-i0n7; pending open question 1). +- **[Amendment] CheckedBatch's created-record exemption is removed** (improvise-p003). + It exempts a record that neither resolves nor is claimed, for frames like + `[AddItem(East), SetCell(East)]`. A frame cannot address what it creates, so nothing + produces such a frame any more. ### 7. ADR 001: an id that no longer resolves is gone -A command naming an id that does not resolve is rejected before anything is applied, as -`Rejection::Gone(Named)`: -- `Named` is one of `Category`, `Item`, `Formula` or `View`, and carries the id's text. -- It is **structural**: force never creates structure. +A command naming an id that does not resolve is rejected before anything is applied, +as `Rejection::Gone(Named)`: +- `Named` is `Category`, `Item`, `Formula` or `View`, and carries the id's text. + **[Amendment]** It gains `Group`. +- The rejection is **structural**: force never creates structure. The check has two halves, like the record tier: - **Before the frame:** an exhaustive `ids_named` feeds `Checked::against`. - **Within the frame:** an **id arm in `broken_chain`** refuses `[RemoveFormula(f), ReplaceFormula(f)]` (improvise-gkyo). -Per-entity versions are not added (answer 10). +Per-entity versions are not added. -### 8. ADR 002: `RecordHash` is now defined over ids +### 8. ADR 002: `RecordHash` is defined over ids -Following answer 3, `RecordHash` hashes the explicit serialization of the sorted -`(CategoryId, ItemHandle)` pairs, spelled as their id text, the same function -over different strings. The text is enough: answer 22 makes ids unique across -kinds, so no `Item` and `FormulaTarget` can spell alike, and the hash stays -canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate -is never part of a record in the first place. **Pending Q23**: under option (c) no -`FormulaTarget` exists, and this paragraph reduces to ids over `ItemId`s. -- **ADR 002's decision is unchanged in substance.** It still hashes an explicit - serialization of a normalized record, with a fixed key and a width argued for. -- **Its prose will be out of date once this lands.** ADR 002 says `CellKey::new` - "sorts coordinates by category name", and a record is spelled in names. Both become - ids. ADR 002 is **not edited here**. Updating that wording belongs to the - implementation PR. +`RecordHash` hashes the explicit serialization of the sorted `(CategoryId, ItemId)` +text pairs: the same function over different strings. **[Amendment]** The pairs are +`(CategoryId, ItemHandle)` spelled as id text. Ids are unique across kinds, so the hash +stays canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate +is never part of a record (pending open question 1). -**What changes in behaviour:** a display rename no longer changes any `RecordHash` or -bumps `layout_version`, because `view_shape` hashes ids too. So a client's surrogate -survives a rename, and its in-flight edit still lands. A **coordinate** change, as in -fdk1 or a records-mode move, still produces a different record and a different hash, as -it should: it *is* a different record. +ADR 002's decision is unchanged in substance. Its prose ("sorts coordinates by +category name", records spelled in names) is out of date once this lands, and the +implementation PR updates it. -### 9. Headers that change meaning: the edge case answer 6 asks to document +**In behaviour:** a display rename changes no `RecordHash` and bumps no +`layout_version`, because `view_shape` hashes ids too. So a client's surrogate survives +a rename, and its in-flight edit lands. A **coordinate** change, as in fdk1 or a +records-mode move, produces a different record and a different hash, because it is a +different record. -Under answer 6, `header_decl` reads a trailing `(token)`, with or without a space before -it, as the id. **It applies only to new-version files** (answer 15, Q12(a)). An -old-version file is read as it always was, so its headers never split. In a new-version -file the writer never produces an ambiguous header, because it pipe-quotes any display -that would split. So the edge cases below arise **only when a new-version file is -written by hand**. The table shows what the header rule would do to lines that exist -in old files today. That is why old files do not get the rule. +### 9. Headers that change meaning -| header line | whole-line reading (today) | under `header_decl` | +In a new-version file, `header_decl` reads a trailing `(token)`, with or without a +space before it, as the id. **Old-version files never get this rule**, so their headers +never split. The writer pipe-quotes any display that would split, so the cases below +arise only in a new-version file written by hand. They show why old files keep the +whole-line reading. + +| header line | whole-line reading (old version) | under `header_decl` (new version) | |---|---|---| | `## View: Default(default)` | `Default(default)` | display `Default`, id `default` | | `## View: \|Default (default)\|(default)` | pipes literal | display `Default (default)`, id `default` | @@ -1186,545 +492,385 @@ in old files today. That is why old files do not get the rule. | `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** | | `## Category: Time Period` | `Time Period` | unchanged; id `time_period` generated | -**The documents that change meaning** are those with a view or category header ending in -a parenthesised run of letters, digits, `_` or `-`. **The app writes such headers -itself**: `improvise import` on a CSV with headers like `Region(code)` or `Size(cm)` -saves `## Category: Region(code)` today (improvise-80rm). Any view someone named -`Sales (copy)` does the same. +The app itself writes such headers: `improvise import` on a CSV with headers like +`Region(code)` saves `## Category: Region(code)` today (improvise-80rm). -**What it would cost if old files got the new rule** (the case answer 15 rules out): -- The category is renamed on load: display `Region`, id `code`. -- Every reference in that old file spells `Region(code)`, which resolves to neither the - display nor the id. -- So the axis lines and every data key for that category do not resolve. -- Today's loader would silently drop the axis lines and orphan the data, so nothing is - drawn. +If old files got the rule, that category would load as display `Region`, id `code`. +Every reference in the file spells `Region(code)`, which resolves to neither, so the +axis lines and data keys would not resolve, and nothing would be drawn. Because old +files keep the whole-line reading, none of this happens. On the first save the writer +pipe-quotes such a display: `## Category: |Region(code)|(regioncode)`. -**Decided (answer 15):** old-version files keep the whole-line reading, so none of -this happens to a file that exists today. On the first save the writer pipe-quotes -such a display, for example `## Category: |Region(code)|(regioncode)`, so the new file -cannot split it either. +### 10. The file version alone picks the regime -### 10. The file version is explicit, and it alone picks the regime - -Following answers 15 and 25: - **The version is already explicit.** The first line of every `.improv` file is a mandatory version line, matched as an **exact literal** - (`version_line = { "v2025-04-09" ~ NEWLINE ~ blank_lines }` in `improv.pest`), and - `format_md` writes it. Nothing is added to make the version explicit. What is added is - a second literal. -- **The new format gets a new literal.** The grammar's `file` rule gets one arm per - literal, and each arm uses its own header and reference rules. The new literal's - exact spelling is left to the implementation. -- **The regime is chosen only by the first line**, never inferred from what the body - looks like. So the old literal means no ids anywhere, and the new literal means ids - on every declaration (answer 21). -- **A file whose first line is neither literal is refused**, as it is today. -- **Every save writes the new literal** (answer 12). So an old file changes version on - its first save. + (`version_line = { "v2025-04-09" ~ NEWLINE ~ blank_lines }`), and `format_md` writes + it. +- **The new format gets a second literal.** The grammar's `file` rule gets one arm per + literal, each with its own header and reference rules. The literal's spelling is the + implementation's choice. +- **The regime is chosen only by the first line**, never inferred from the body. The + old literal means no ids anywhere. The new literal means ids on every declaration. +- **A file whose first line is neither literal is refused**, as today. +- **Every save writes the new literal**, so an old file changes version on its first + save. ### 11. Migration 1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224. -2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId`, - `ViewId`, and `ItemHandle`, which the re-key (step 3) needs for `CellKey` before - clause 6's step (5) uses it in projections (**pending Q23**: under option (c) - there is no `ItemHandle`). +2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId` and + `ViewId`. **[Amendment]** Also `ItemHandle`, which the re-key needs for `CellKey` + (pending open question 1). 3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with display→id resolution moved to every input edge. -4. **Persistence:** the new version literal (clause 10), the grammar in clause 5, the - loader and its refusals, slugify, the writer, and the round-trip properties with a - formula-line strategy. +4. **Persistence:** the new literal (clause 10), clause 5's grammar, the loader and its + refusals, slugify, the writer, and the round-trip properties with a formula-line + strategy. 5. **Projections and `ViewState`** (clause 6). 6. **Vocabulary:** the id-addressed variants, `Rejection::Gone`, `ids_named` and `broken_chain`'s id arm. -7. **Clause 12's capture:** the client-side display→id mapping taken when the - formula editor opens, and the resolver that turns typed names into id-form - text (answers 32 and 33). Neither the resolver nor any display-matching code - exists yet, so the interim order is built ids first from the start and nothing - has to be migrated. The autocomplete that retires it is not in this list. +7. **[Amendment] Clause 12's client-side capture and resolver.** No display-matching + code exists yet, so the resolver is built ids first from the start. -The implementation epic (improvise-cq1q) orders these steps to keep `main` -shippable after each one, which the list above does not: the display field -before the re-key, and projection ids before the re-key. That ordering is the -epic's, not a change to this list. +The implementation epic (improvise-cq1q) orders these steps to keep `main` shippable +after each one: the display field before the re-key, and projection ids before the +re-key. ### 12. A formula edit's names are fixed when the editor opens -**This clause is the ADR's own reading, not a user answer.** It closes a contradiction -the adversarial review of `4830fa30` found between clause 6 and the km71 section, which -is also ChatGPT Codex's P1 on `29ad14f4`. - -**The problem.** The formula panel shows display names (answer 19), and the user edits -that text. If the host maps those names to ids at **commit**, a concurrent rename breaks -the edit one of two ways: -- the name the user typed no longer resolves, so the edit is refused, which contradicts - clause 6's "does not refuse an edit of `Profit`"; +**The problem.** The formula panel shows display names, and the user edits that text. +If those names were mapped to ids at **commit**, a concurrent rename would break the +edit one of two ways: +- the typed name no longer resolves, and the edit is refused; - or the name now belongs to something else (`Revenue` renamed away, another item - renamed to `Revenue`), so the edit silently binds to a different thing. That is the - glitch answer 3 exists to prevent. + renamed to `Revenue`), and the edit silently binds to a different thing. -**The rule.** The display-name-to-id mapping is **captured when the formula editor -opens**, and the commit resolves the edited text through that captured mapping. This -is the capture-at-open rule the cell editor already follows: `OpenedCell` in -`improvise-protocol/src/client.rs` records the record, its version, the document -generation and the view at open, and the commit answers for those rather than for -whatever the document holds by then. -- **The client holds the mapping** (answer 32), as it holds `OpenedCell`. The TUI - and the browser both use the same resolver in `improvise-protocol`. The user - confirmed this and widened it: the client "should be the only place that allows any - reference to something by a human-friendly name", and converts it "to IDs ASAP" - (answer 39). -- The mapping covers every display name the panel showed in that formula's text, - including the target's. -- A name the user types that was **not** in the captured mapping is new to the - edit. The client resolves it against its own id-bearing projection (clause 6) at - commit, in answer 33's order: **ids first, then displays** ("IDs are unambiguous so - they should always be preferred when there's ambiguity", answer 38). A typed name equal to - an id names that id's thing, even where it is also another thing's display. - Otherwise, if exactly one thing in scope has that display, it names that thing. If - a display matches more than one thing, the edit is refused and the refusal lists - the candidates' ids. If the name does not resolve at all, the edit is refused and - names it. An id is matched exactly: no case folding. -- **The ids-first order was stated as interim** (answer 33). The long-term design - separates two layers. A view-layer autocomplete of typed display names turns text - into ids before it is committed, and the model layer refers to ids only. Revision 9 - read that, when the autocomplete exists, this resolver's display stage and the - ids-first order go with it. **Answer 38 says ids "should always be preferred when - there's ambiguity"**, so the preference itself is not temporary. Whether it carries - into the autocomplete (an id match offered first, say), or ends with this resolver, - is Q27. This ADR does not specify the autocomplete. -- **The wire carries only id-form text.** `ReplaceFormula.raw` and - `AddFormula.raw` are in the file's id form, and the host never resolves a display - name (answer 32). The host parses ids, and one that no longer exists is - `Rejection::Gone` (clause 7). -- `was` is still compared against the stored id-form text (clause 6), so a concurrent +**The rule.** The display→id mapping is **captured when the formula editor opens**, and +the commit resolves the edited text through it. This is the cell editor's +capture-at-open rule: `OpenedCell` (`improvise-protocol/src/client.rs`) records the +record, its version, the document generation and the view at open, and the commit +answers for those. +- The mapping covers every display the panel showed in that formula's text, including + the target's. +- `was` is compared against the stored id-form text (clause 6), so a concurrent *replace* of the formula is still seen. +- A captured id that no longer exists at commit is `Rejection::Gone` (clause 7). +- **A typed name that was not in the captured mapping** is resolved against the + document at commit, and refused, naming it, if it does not resolve. -**What the interim order means for a document where an item's display equals another -item's id.** Item A is displayed `East` and has id `east`. Item B is displayed `east`, -so slugify finds `east` taken and gives it `east-1` (clause 4). Typing `east` names -**A**, because the id wins. B is reached by typing `east-1`, or by autocomplete once -it exists. A user who typed `east` meaning B is bound to A without being told, which -is the cost of the interim order and the reason the panel shows B as `east-1`, not as -a bare `east` (clause 6). The captured mapping cannot disagree with the rule, since -it holds only names the panel showed bare, and the panel shows none that ids-first -would bind elsewhere. +**[Amendment] The client does the resolving:** +- **The client holds the mapping**, as it holds `OpenedCell`. The TUI and the browser + share one resolver in `improvise-protocol`. +- **A name new to the edit** is resolved by the client against its id-bearing + projection (clause 6), **ids first, then displays**. Ids are unambiguous, so they are + preferred wherever there is ambiguity. + - A typed name equal to an id names that id's thing, even where it is also another + thing's display. + - Otherwise a display that exactly one thing in scope has names that thing. + - A display matching several things is refused, and the refusal lists their ids. + - A name that resolves to nothing is refused, and the refusal names it. + - Ids are matched exactly, with no case folding. +- **The long-term design separates two layers.** A view-layer autocomplete turns + typed display text into ids before commit, and the model layer refers to ids only. + What becomes of the ids-first order once that exists is open question 3. This ADR + does not specify the autocomplete. +- **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are + in the file's id form, and the host never resolves a display. The host parses ids, + and one that no longer exists is `Rejection::Gone`. -**Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. -From another client, rename the item `Revenue` to `Sales` and rename a different -item to `Revenue`. Then edit something that is **not a name**: change the text to -`Revenue - Cost * 1` and commit it. The stored formula must reference the original -item's id, and the commit must not be refused. The test must fail under -commit-time resolution, which binds to the new `Revenue`, and it must fail under a -refusal on rename. Both are the rivals this clause rules out. +**[Amendment] What ids-first costs.** Suppose item A is displayed `East` with id `east`, +and item B is displayed `east`, so slugify gave it `east-1`. Typing `east` names **A**. +B is reached by typing `east-1`. A user who typed `east` meaning B is bound to A +without being told. That is why the panel shows B as `east-1` and never as a bare +`east`. -**Required test for the interim order** (answer 33). In a document with items A -(display `East`, id `east`) and B (display `east`, id `east-1`), type `east` into a -formula editor, in text not captured at open. The stored formula must reference A. -The test must fail under displays-first, which binds B. Typing `east-1` must bind B, -and typing a display that is nobody's id must still resolve by display. A fixture -where no display equals an id passes under either order and proves nothing. +**Required regression test.** +1. Open a formula editor on `Profit = Revenue - Cost`. +2. From another client, rename `Revenue` to `Sales`, and rename a different item to + `Revenue`. +3. Edit something that is not a name (`Revenue - Cost * 1`) and commit. -Revision 8's version committed the *unedited* text. A client that treats an -unchanged text as a no-op passes it without resolving any name, so the test could -not tell a correct capture from no capture at all (improvise-hexb item 1). +The stored formula must reference the original item's id, and the commit must not be +refused. The test must fail under commit-time resolution and under a refusal on +rename. Committing the *unedited* text proves nothing, because a client may treat it +as a no-op without resolving any name. -## What this means for you +**[Amendment] Required test for ids-first.** With items A (`East`, `east`) and B +(`east`, `east-1`), type `east` into an editor, in text not captured at open: +- it must bind A, and the test must fail under displays-first; +- `east-1` must bind B; +- a display that is nobody's id must still resolve by display. -### improvise-km71 — editing a formula (landed as an interim) +A fixture where no display equals an id proves nothing. -km71 landed in PR #214, before this ADR, and was built to be revised by it -(`context/code/core.md` §18): -- **What landed.** `ModelCommand::ReplaceFormula { id: FormulaId, was, raw }`. - `FormulaId` is opaque, with private fields `(target, target_category)`. - `Model::replace_formula` writes in place, keeping the position. It refuses a new - target that another formula owns, and it refuses any target change outside - `_Measure` (improvise-kb1d). `was` is the text the editor opened on. It makes a - concurrent *replace* visible, which even a stable id cannot do, so **`was` stays**. -- **What this ADR changes.** - - `FormulaId`'s fields become one id. The signatures that carry `FormulaId` do not - change, which is what making it opaque was for. - - A target change becomes a **display** rename of the target. The id does not move, - and nothing that references the target is rewritten. So kb1d's refusal outside - `_Measure` can be lifted, and so can the dangling-reference report core.md §18 - describes. - - The panel shows `Profit (profit) = Revenue - Cost` (answer 19). The user edits that - text. Its display names are resolved to ids through the mapping captured when the - editor opened (clause 12), not through the names the document holds at commit, and - `was` is compared against the stored id-form text (clause 6). The parenthesised id is not the user's to - change, since the host assigns ids (answer 14). An edit that changes it is refused. - This is the ADR's reading, not a user answer. -- **Rejections.** Another client's deletion gives `Rejection::Gone(Formula(id))`. A - deletion earlier in the same frame gives the broken-chain refusal. +## What this means for the tickets -### improvise-fz77 — renaming a category or item +**improvise-km71 (editing a formula), landed as an interim.** +`ReplaceFormula { id: FormulaId, was, raw }` writes in place. It refuses a new target +that another formula owns, and any target change outside `_Measure` (improvise-kb1d). +`was` makes a concurrent *replace* visible, which even a stable id cannot, so `was` +stays. -- **The operations.** `RenameCategory { category, to }` and - `RenameItem { category, item, to }`. The panel rows carry the ids. -- **It is now cheap** (answers 1 and 2): one `display` write. No `CellKey`, axis map, - page selection, hidden or collapsed set, or formula text is rewritten, because all of - them hold ids. No `RecordHash` or `layout_version` moves, so in-flight edits survive. -- **What it still owes:** - - **Uniqueness.** Refuse a display name already used in the same scope. - - **Dirtiness.** Mark the effect so the grid labels and panels are re-projected - (gotcha 35). The labels are display strings. - - **Tests.** Assert that the *labels* changed and the *surrogates did not*. That is - the property this design exists for. -- **It depends on step 3 of the migration**, the re-keying. Until then there is no cheap - rename to build on. +This ADR changes three things: +- **`FormulaId`'s fields become one id.** The signatures do not change. +- **A target change becomes a display rename of the target.** The id does not move, + so kb1d's refusal and core.md §18's dangling-reference report can be lifted. +- **The panel's text is resolved through clause 12's captured mapping.** The + parenthesised id is not the user's to change, so an edit that changes it is + refused. -### improvise-fdk1 — renaming a records coordinate +Rejections: +- Another client's deletion gives `Rejection::Gone(Formula(id))`. +- A deletion earlier in the same frame gives the broken-chain refusal. -fdk1 landed in PR #221, before this ADR. What exists on main: -- `ModelCommand::RenameRecordCoordinate { record: RecordHash, category: String, item: - String }`, which moves a whole record to a new coordinate and creates the item if need - be; -- `CellId::Coordinate { record: RecordHash, category: String }`, the surrogate a - records-mode coordinate cell carries. +**improvise-fz77 (renaming a category or item).** It adds `RenameCategory` and +`RenameItem`. A rename is one `display` write, and no key, set, formula text, +`RecordHash` or `layout_version` moves. It still owes three things: +- refuse a display already used in the scope; +- mark the effect so labels and panels re-project (gotcha 35); +- tests asserting that the *labels* changed and the *surrogates did not*. -**What this ADR changes**, as it does for km71's `FormulaId`: -- `category` becomes a `CategoryId`, in the command and in `CellId::Coordinate`. -- `item` becomes an `ItemRef`: `Existing(ItemId)` or `New(display)`. That is the - create-by-display half of clause 6, since the host assigns the new item's id. -- Unlike a display rename, this **changes a coordinate**, so the record's `CellKey`s and - `RecordHash` change. It is a different record, and that does not change. +It depends on migration step 3. -## Questions as asked, and where each now stands - -The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14, -revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All -of those are answered**, and answer 44 later overrode Q5's answer. After acceptance, -three implementation questions were answered on their tickets (answers 30–32). The -user's comments on revision 9 are answers 33–36 (eighth round) and 37–45 (ninth -round). Revision 9 added Q23 to Q25: answers 41 and 42 settle Q24 and Q25, and Q23 is -still open. The ninth round adds Q26 to Q34, which are open. - -| # | question | my recommendation | status | -|---|---|---|---| -| 1 | Integer or word ids? | first draft: words; revision 1: integers only | **answered**: interned symbols (answer 4), generated by slugify (answers 8, 16). Words. | -| 2 | Formula id placement | F1, `- Profit(3) = …` | **answered: F1** (answer 5) | -| 3 | Headers in old files | first draft: accept the rare break; revision 1: gate on version | **answered**: the space is optional (answer 6), and old files keep the whole-line reading (answer 15, Q12) | -| 4 | Do `## Data` keys carry ids? | no | **answered: yes, ids** (answer 7), reversing the recommendation | -| 5 | Group ids? | not now | **answered: not now** (answer 9). **Overridden: a group name is an id** (answer 44); the mechanism is Q33 | -| 6 | Per-entity versions? | not in this ADR | **answered: not in this ADR** (answer 10) | -| 7 | `RecordHash` over ids? | no, keep names | **answered: over ids** (answer 3), reversing the recommendation | -| 8 | Duplicate id: repair or refuse? | revision 2: refuse | **answered: refuse the load** (answer 11) | -| 9 | Reading of "opt-in" | read optional, write always | **answered: yes** (answer 12) | -| 10 | Id reuse after a reload | revision 2: (b) accept reuse | **answered: (b)** (answer 13) | -| 11 | Who assigns ids? | the host | **answered: the host** (answer 14) | -| 12 | Old files and the new header rule | (a) no | **answered: (a), with the version marked unambiguously** (answer 15) | -| 13 | Slugify details | all generation; keep case; keep `_` `-`; strip non-ASCII; `-` | **answered** (answer 16): all generation, **lower case only** (reversing the recommendation), keep both, strip, `-` | -| 14 | Formula text: ids or display names? | ids in the file | **answered**: ids in the file (answer 3); the panel is Q16 | -| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-`, "space" means whitespace (answer 34), and each whitespace character becomes `_` before runs of `_` collapse (answer 41) | -| 16 | What the formula panel shows | none | **answered: display names, target shown as `Profit (profit)`, ids where a name is ambiguous** (answer 19) | -| 17 | Capitals in a hand-written id | (a) no | **answered: no** (answer 20) | -| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse** (a run of whitespace, answer 34; by collapsing runs of `_`, answer 41), plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | -| 18(b) | What "casefold" means | full case folding | **answered: full Unicode case folding** (answer 26) | -| 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answers 22 and 37), stronger than (a) | -| 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23); the reservation covers "virtual categories and measures and other entities" (answer 42), and `_index`/`_dim` are restated (answer 43) | -| 21 | An old file naming an undeclared item | (a) declare it at load | **answered: (a)** (answer 24) | -| — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal | -| 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) | -| — | A formula target outside `_Measure` (improvise-m56i item 2) | no real item; a `FormulaTarget` handle | **answered: a real item and a separate formula id** (answer 30), reversing the recommendation; a formula id that collides takes the counter's suffix (answer 37) | -| — | A typed name matching one display and another id (improvise-m56i item 6) | display first, then id; refuse ambiguity | **answered: display first; the client holds it** (answer 31). **Order reversed: ids first until there is autocomplete** (answer 33). Ids are always preferred under ambiguity (answer 38); whether that outlives the interim resolver is Q27 | -| — | Which side holds clause 12's capture (improvise-hexb item 2) | the client | **answered: the client; id-form text on the wire** (answer 32), widened to every reference by a human-friendly name (answer 39) | -| — | Whether a measure has an id/display split, and which formula targets are normal (user comments on revision 9) | — | **answered in principle**: measures have the split (answer 35), the normal target is a measure (answers 35, 36), and a formula is a derived measure (answer 40). The mechanism is Q23 | -| — | Revision 9's reading that `_Index`/`_Dim` coordinates are outside the id namespace | (the ADR's reading) | **rejected as worded** (answer 43); what the positions are is Q31 | -| — | Revision 9's reading that `collapsed:` names a group by name | (the ADR's reading) | **overridden: a group name is an id** (answer 44) | -| — | Revision 9's reading that a slug of only `-` is a legal id | (the ADR's reading) | **rejected: `-` is not an id, and no id begins or ends with `-`** (answer 45) | -| 23 | Is a formula's derived measure one thing with one id, or two things with two ids? (revision 9; reworded after answers 35 and 36; restated after answer 40) | none until the user chooses | **open** | -| 24 | Does `a_ b` slug to `a__b`? (answer 34) | (a) keep | **answered: (b), runs of `_` collapse**, so `a_ b` → `a_b` (answer 41), not the recommendation | -| 25 | Does a trailing whitespace run slug to a trailing `_`? | (a) keep | **answered: (b), a trailing `_` is dropped** (answer 42), not the recommendation | -| 26 | What may a formula refer to, and what is "a category value"? (answer 37) | (iii) only keep ids unique | **open** | -| 27 | Does ids-first outlive the interim resolver? (answer 38) | none | **open** | -| 28 | May the old-version loader and headless `cmd`/`script` resolve displays? (answer 39) | yes, both | **open** | -| 29 | Does step 4's collapse come after the strip? (answer 41) | (a) after | **open** | -| 30 | Which "other entities" share the reserved `_`? (answer 42) | the virtual views are among them | **open** | -| 31 | Are `_Index`'s and `_Dim`'s positions ids? (answer 43) | none | **open** | -| 32 | Moving the virtual categories to lower-case ids (answers 23 and 43) | as written there | **open** | -| 33 | How groups carry ids (answer 44) | none, beyond the readings stated | **open** | -| 34 | Hand-written ids with `-` or a trailing `_`, and how the trims repeat (answer 45) | refuse `-` at either end; repeat the trims | **open** | - -## Still open - -Nothing is open among Q1–Q22, and answer 29 settles the one case it raises: two -identical hand-written ids are refused at load. **The user approved revision 7. -Revision 8 adds clause 12, which is the ADR's own reading, and the user approved it -("looks good, approved", 2026-09-25 00:17:10Z).** Three follow-ups from the same -review were left as tickets rather than ADR text: improvise-le4p, improvise-m56i and -improvise-p003. Revision 9 folds all three in, and improvise-m56i's two -questions are answers 30 and 31. - -**Revision 9 needs the user's approval.** It asked Q23 to Q25, and the ninth round -answered Q24 and Q25. Q23 and the ninth round's Q26 to Q34 are open. - -**Q23. A formula is a derived measure. Is the measure and its formula one thing with -one id, or two things with two ids?** (Restated after answer 40, which asked for this -question to be stated more clearly. Revision 9 asked how a formula line spells its -target; after answers 35 and 36 it asked how a formula-target measure carries its id. -It is the same question, stated in the user's terms.) - -**What is decided.** -- A formula is a derived measure (answer 40), and a measure is the normal formula - target (answers 35 and 36). -- A measure has an id and a display (answer 35). -- Every reference is an id (answer 3), and no two ids in a document are equal - (answers 22 and 37). -- A formula carries an id in its line, after the target and before the `=` - (answer 5), and `ReplaceFormula` and `RemoveFormula` address a formula by it - (clause 6). -- The rare dimension target is a real item with a separate formula id (answer 30), - and a formula id that collides takes the counter's suffix (answer 37). - -**The worked example.** A document has two stored measures, `Revenue` and `Cost`. -Each is an item of `_Measure`, with ids `revenue` and `cost`, and each holds data. -The user adds `Profit = Revenue - Cost`. `Profit` holds no data of its own: its -column is computed by the formula, which is what makes it a derived measure. Two -kinds of thing must then name Profit by an id: -1. **Things that refer to the measure.** Another formula (`Margin = Profit / - Revenue`), the `CellKey` of each Profit cell, a `hidden:` line, and ADR 008's - cursor anchor. -2. **Things that refer to the formula.** `ReplaceFormula` and `RemoveFormula`, which - edit or delete the formula. - -**The question is whether 1 and 2 use the same id.** - -- **(a) One thing, one id.** The formula *is* the measure. Profit's id is `profit` - and its display is `Profit`, both held on the formula. `Margin = profit / revenue`, - each Profit cell's key holds `profit`, and `ReplaceFormula` addresses `profit`. - The file line is `- Profit(profit) = revenue - cost`. No `_Measure` item exists for - Profit, so in code the column's coordinate is `ItemHandle::FormulaTarget(profit)` - (clause 2). This is revision 9's model, and the smallest change. -- **(c) Two things, two ids.** The measure is a `_Measure` item with id `profit` and - display `Profit`. The formula that computes it is a separate thing, with id - `profit-1`, suffixed because `profit` is taken (answer 37). `Margin = profit / - revenue` and each Profit cell's key holds `profit`, while `ReplaceFormula` - addresses `profit-1`. The file line is `- profit(profit-1) = revenue - cost`. This - is answer 30's model applied to measures too, so a measure target and a dimension - target work the same way. -- **(b) is not a third model.** It is (a)'s model, with the rare dimension line - spelled `- Tax(tax-1)(tax) = … [customcat]` (display, formula id, item id), so that - the target position always holds a display. Under (a) and (c) the dimension line - is `- tax(tax-1) = … [customcat]`, with the item id in the target position. - -**Where (a) and (c) differ.** - -| | (a) one id (and (b)) | (c) two ids | -|---|---|---| -| A Profit cell's coordinate | the formula's id, as `ItemHandle::FormulaTarget` | an ordinary item id | -| `CellKey` | `(CategoryId, ItemHandle)` | `(CategoryId, ItemId)`, and `ItemHandle` goes | -| A stored `Profit` column and a formula `Profit = …`, merged into one measure today by `measure_item_names` | one column with two ids (the item's `profit`, the formula's `profit-1`), and the ADR must say which keys the cells | the formula attaches to the existing item | -| `RemoveFormula` on Profit | the measure goes too | the item `profit` stays, with no formula and maybe no data, as a dimension target's item stays today | -| What the file must newly write | nothing: the formula line declares the measure | the item `profit` and its display (today a formula target is not written as an item, and is rebuilt from `## Formulas`) | -| A dimension target (answer 30) | works differently from a measure target | works the same way | -| ADR 008 (PR #308), which anchors the cursor by item identity | needs `ItemHandle` | works as it is | -| ADR 009 (PR #309), which uses `find_item_category`'s `_Measure` fallback for formula targets | keeps it | loses it | - -Neither ADR 008 nor ADR 009 is edited here. - -**What is open.** The choice between (a) and (c) and, under (a), whether the -dimension line is spelled (a)'s way or (b)'s. **No recommendation is made.** Answer -40, "Formulas are a derived measure", reads most literally as (a), where the formula -*is* the measure. But (c) also models a derived measure, an item whose values a -formula derives, and the answer was a request for clarity rather than a choice. So -reading it as choosing (a) would be the ADR's judgement, not the user's. - -Smaller points, each the ADR's own reading and none a user answer: -- **The `[_measure]` tag.** Today's writer omits the tag for a `_Measure` formula - (`- {raw}`), and the user's own example, `profit = revenue - cost`, has none. The - examples above omit it too. Whether the new writer keeps omitting it is not decided - here. -- **Answers 5 and 19** picture the measure case as `Profit(3)` (the file line) and - `Profit (profit)` (the panel). The panel picture fits every option. The file-line - picture, a display before the id, fits (a) exactly and (c) only loosely, because - (c) writes the item id where answer 5 shows the display. -- **References to a formula target** are the id text of its handle: under (a) the - item id outside `_Measure` and the formula id for a `_Measure` target; under (c) - always the target's item id. -- **Quoting the target position.** Where a display sits in the target position - ((a) for a measure, and (b)), a display that looks like `Name(x)` would read back - as a display and an id, so the writer pipe-quotes it: `- |Margin(%)|(margin) = …`. - An item id never contains a parenthesis, so only a display needs it. (c) has no - such position. - -**Q24. Does `a_ b` slug to `a__b`?** **Answered by answer 41: no.** Revision 9's -options were (a) keep it, its lean, and (b) collapse any run of whitespace and `_` -to one `_`. Answer 41, "The spaces should have been normalized to _ before -collapsing runs of _s", is (b): `a_ b` → `a_b` and `a__b` → `a_b`, so ids change for -displays that hold a double underscore. Its example, `Time \u{200B} Period`, now slugs -to `time_period`. Clause 4's steps 2 and 4 state it, and where step 4 goes is Q29. - -**Q25. Does a trailing run slug to a trailing `_`?** **Answered by answer 42: no.** -Revision 9's options were (a) keep it, its lean, (b) remove a trailing `_` as a -leading one is removed, and (c) trim whitespace before step 2. Answer 42 is (b): "A -trailing one should be dropped", so `East ` → `east` and `East_` → `east`, and a -slug that then collides takes the suffix. Clause 4's step 5 states it. - -**Q26. What may a formula refer to, and what is "a category value"?** Answer 37 says -"Formulas may refer to anything in a typed context", with the example of a formula -counting "the number of times a particular letter appears in a category value". -Today a formula reference (`Expr::Ref`) names an item or a formula target, and -`find_item_category` finds its category. The answer does not say: -- (i) whether a bare id in formula text may also name a category, a view or a group, - with its kind given by where it appears (an aggregate's argument, a `WHERE` clause, - a function's parameter); -- (ii) whether the text a function like that reads is the item's display (`Café`) - or its id (`caf`). The two differ, because slugify folds case and strips; -- (iii) whether this ADR must specify any of it, or only keep ids unique so that it - can be added later. - -The ADR's lean, not the user's, is (iii): the formula language has no string -function today, and answer 22 already makes a reference to any kind unambiguous. - -**Q27. Does ids-first outlive the interim resolver?** Answer 33 made ids-first the -order "before we have autocomplete". Answer 38 says ids "should always be preferred -when there's ambiguity". -- **(a) Yes.** Wherever typed text could name both an id and a display, the id wins, - and that includes the autocomplete's ranking (an id match offered first, say). - Clause 12's "interim" wording goes. -- **(b) No.** Once the autocomplete turns display text into ids, nothing is matched - and nothing is ambiguous, so answer 38 governs only until then. - -No recommendation. - -**Q28. Two places resolve a display outside a client.** Answer 39: "the client should -be the only place that allows any reference to something by a human-friendly name". -- **The old-version loader** resolves an old file's references against display - names (clause 5), because in an old file the name is the only identity there is. -- **Headless `improvise cmd` and `improvise script`** run command lines that name - things by display, with no client attached. - -The ADR's reading: both are allowed. The loader migrates a file and is not a person -referring to something. The CLI is the UI layer of its own invocation, and it -converts names to ids at its edge, as a client does. Is that right? - -**Q29. Does step 4's collapse come after the strip?** Clause 4 puts it after step 3. -That order is what makes answer 41's example, `Time \u{200B} Period`, give -`time_period`. Before the strip, the zero-width character keeps the two `_` apart, and -the result would still be `time__period`. -- **(a) After the strip**, as clause 4 is written. The ADR's lean. -- **(b) Before the strip.** - -**Q30. Which "other entities" share the reserved `_`?** Answer 42: "A leading _ is -reserved for virtual categories and measures and other entities." -- (i) No virtual measure exists today. Is one planned, or does "measures" mean the - `_measure` category? -- (ii) The ADR reads the virtual views, `_records` and `_drill` (clause 1), as among - the "other entities". Is that right? -- (iii) The ADR reads "dropped to: disambiguate with -#" as "dropped too; - disambiguate with -#": a slug that collides once its `_` is dropped takes the - suffix. Is that right? - -**Q31. Are the positions under `_Index` and `_Dim` ids?** Records mode gives each row -a synthetic coordinate under `_Index`, and each column one under `_Dim` (a column -index, gotcha 1). They are never persisted and never typed. Answer 43 says the two -categories have ids `_index` and `_dim`, as answer 23 said, but revision 9's reading -was about the positions, and the answer does not say what they are. -- **(a) Outside the id namespace**, revision 9's reading: a position `0` under - `_index` does not collide with a user's item `0`. -- **(b) Inside it**, spelled so that they cannot collide, such as `_index-0`. That - needs answer 42's reservation to cover them. -- **(c) Something else.** - -No recommendation. - -**Q32. Moving the virtual categories to lower-case ids.** Today `_Index`, `_Dim` and -`_Measure` are keys in the model, on the wire, in `ViewState` and in `.improv` files: -`web/demo.improv` has `_Measure: column`, `_Index: none`, `## Category: _Measure` -and `_Measure=Revenue`. Under answers 23 and 43 they become `_index`, `_dim` and -`_measure`, and the displays stay as they are. The ADR's reading of the migration: -- an old file's `_Measure` resolves to `_measure` through its display at load, like - any other name (clause 5). The first save writes `_measure: column` and - `_measure=revenue`; -- a new-version file must spell the id, and `_Measure` there is refused (answer 20); -- the virtual category's header carries its id like any other: - `## Category: _Measure(_measure)`; -- every string literal that names them in code, on the wire and in the browser client - moves in the re-key step (clause 11, step 3). - -Is that the intended migration? - -**Q33. How do groups carry ids?** Answer 44 decides that a group has an id and a -separate friendly name. It does not decide: -- (i) **Namespace.** Is a group id unique in the whole document, like every other id - (the ADR's reading of answers 22 and 37), or only within its category? -- (ii) **Generation.** Is it the slug of the display, with `group` as an empty slug's - `` and the shared counter for collisions (the ADR's reading of answer 16(a), - "All id generation")? -- (iii) **The file.** For example a definition `> Coastal(coastal)`, a membership - `- Coastal_East(coastal_east)[coastal]`, a parent group (improvise-felo.6's - hierarchies) named by id, and a `collapsed:` line holding the group id, with or - without its category id. -- (iv) **The wire.** Does `AddItemInGroup`'s `group` name an existing group by id, - create one by display, or either? Is there a `RenameGroup { group, to }`? -- (v) **The frame.** `AxisFrame`'s `FrameGroup` carries a group name today. PR #336 - (improvise-cq1q.12, open) adds a display beside it. - -No recommendation beyond the readings stated. - -**Q34. Hand-written ids, and how step 5's trims repeat.** Answer 45: "A slug of - -should be invalid as an id and there should be no leading or trailing -s". -- (i) Is a hand-written id that is `-`, or begins or ends with `-`, refused at load, - as one beginning with `_` is? The ADR's reading is yes (clause 5). -- (ii) Is a hand-written id ending in `_` refused too? Answer 42 drops a trailing `_` - from a slug, and does not say whether a written id may carry one. -- (iii) Step 5 repeats until neither end holds `_` or `-`, so `-_x` → `x` and `x_-` → - `x`. One pass of each would leave `_x`, which is reserved, or `x_`. The ADR's - reading is to repeat. - -**Where revision 9 states a reading of its own.** In each of these fourteen places it -states a reading that is not a user answer. Each is flagged where it appears, and -approving the PR approves them: -- the virtual views' reserved ids `_records` and `_drill`, now read as among answer - 42's "other entities" (clause 1; Q30); -- groups share the document's one id namespace (clause 1; Q33); -- a `CellKey` coordinate is `(CategoryId, ItemHandle)` (clause 2; **pending Q23**); -- "whitespace" means the Unicode `White_Space` property (replacing the U+0020 reading, - by answer 34) (clause 4); -- step 4's collapse comes after the strip (clause 4; Q29); -- step 5's trims repeat until neither end holds `_` or `-` (clause 4; Q34); -- a hand-written id that is `-` or begins or ends with `-` is refused at load - (clause 5; Q34); -- the axis-before-`format:` order and the quoting of a format that reads as an axis - line (clause 5); -- the formula-line shape, and a `_Measure` target's display living on the `Formula` - (clause 5; **pending Q23**); -- CheckedBatch's created-record exemption is removed (clause 6); -- a display equal to another thing's id counts as ambiguous in the formula panel - (clause 6); -- an id is matched exactly, with no case folding (clause 12); -- the display stage of the typed-name resolver goes once autocomplete exists - (clause 12; Q27, since answer 38); -- answer 42's "dropped to:" is read as "dropped too;" (ninth round; Q30). - -Three readings that revision 9 listed here are gone. The user rejected the -`_Index`/`_Dim` reading as worded (answer 43, Q31), gave groups ids, which overrides -the `collapsed:` reading (answer 44), and rejected the legal `-` slug (answer 45). +**improvise-fdk1 (renaming a records coordinate).** +`RenameRecordCoordinate { record, category: String, item: String }` and +`CellId::Coordinate { record, category: String }` exist on main. This ADR changes +three things: +- `category` becomes a `CategoryId`. +- `item` becomes `ItemRef::{Existing(ItemId), New(display)}`. +- The rename **changes a coordinate**, so the record's `CellKey`s and `RecordHash` + change. It is a different record. ## Consequences **Good.** - Edit, rename and replace are distinct, addressed operations. -- A rename is one display write: it moves no key, surrogate or basis tier, and in-flight - edits survive it. +- A rename is one display write, and in-flight edits survive it. - `## Data` keys are unique by construction. -- km71's `FormulaId` becomes a real id by changing one struct, and kb1d's refusal can be - lifted. +- km71's `FormulaId` becomes a real id by changing one struct. **Bad.** -- Re-keying storage is wide: every name-taking API moves to ids, and display→id - resolution moves to the input edges. -- A second version literal (clause 10), and every old file changes version on its first - save. +- Re-keying storage is wide. Every name-taking API moves to ids. +- Old files change version, and their data keys change case, on the first save. - `Rejection` loses `Copy`. -- `Formula::raw` is no longer verbatim: the file carries formula text over ids. -- The formula panel is a second rendering of formula text, over display names, so - the **client** must resolve display names typed there back to ids (answers 19, - 32, 33 and 39), ids first (answers 33 and 38). The host sees only id-form text. -- Ids are lower case, so an old file's data keys change spelling on its first save. -- ADR 002's prose needs a wording update when this lands. +- `Formula::raw` is no longer verbatim. The file carries formula text over ids. +- The formula panel is a second rendering of formula text, so display names typed + there must be resolved back to ids (by the client, under the amendment). +- ADR 002's prose needs a wording update. -**Not addressed.** Per-entity versions (Q6), and ids in `AxisFrame` and -`CellId::Derived`. Group ids are now in scope (answer 44, overriding Q5's answer), -but how they are made, spelled and carried is Q33. +**Not addressed.** Per-entity versions, ids in `AxisFrame` and `CellId::Derived`, and +(under the accepted text) group ids. + +## Open questions + +These are open, and nothing above depends on an answer except where it says so. + +### 1. Is a formula and the measure it derives one thing with one id, or two things with two ids? + +**The question.** A formula is a derived measure. Take `Revenue` and `Cost`, stored +measures with ids `revenue` and `cost`, and add `Profit = Revenue - Cost`. Profit holds +no data: the formula computes its column. Two kinds of thing must name Profit by an +id: +- things that refer to the **measure**: another formula (`Margin = Profit / Revenue`), + each Profit cell's `CellKey`, a `hidden:` line, and ADR 008's cursor anchor; +- things that refer to the **formula**: `ReplaceFormula` and `RemoveFormula`. + +Do they use the same id? + +**Why it matters.** It decides the `CellKey` type (clause 2), the formula line's +shape (clause 5), `RecordHash` (clause 8), the `_Measure` panel rows (clause 6) and +what `.improv` must write. + +**Options.** +- **(a) One id.** The formula *is* the measure. Profit is `profit` everywhere, with + its display on the formula, and the line is `- Profit(profit) = revenue - cost`. + - A Profit cell's coordinate is `ItemHandle::FormulaTarget(profit)`. + - The cost: a stored `Profit` column and a formula `Profit` (merged into one measure + today by `measure_item_names`) would have two ids, and one must be chosen to key + cells. + - `RemoveFormula` removes the measure. + - The rare dimension target (an item `tax` plus formula `tax-1`) works differently. + - ADR 008 needs `ItemHandle`. + - This is the smallest change from the text above. +- **(b)** is (a) with the dimension line spelled `- Tax(tax-1)(tax) = … [customcat]` + (display, formula id, item id), so that the target position always holds a display. +- **(c) Two ids.** Profit is a `_Measure` item `profit`, and its formula is `profit-1`. + The line is `- profit(profit-1) = revenue - cost`. + - `CellKey` returns to `(CategoryId, ItemId)` and `ItemHandle` goes. + - A same-named stored measure is simply the item the formula attaches to. + - Measure and dimension targets work alike. + - The cost: the file must newly write the item and its display (today a formula + target is rebuilt from `## Formulas`). + - `RemoveFormula` leaves an item with no formula and maybe no data. + - ADR 009's `_Measure` fallback in `find_item_category` goes. + +Also to settle: whether the writer keeps omitting the `[_measure]` tag on a `_Measure` +formula line, as today's writer does. + +**Recommendation:** none. Both (a) and (c) model a derived measure, and the choice is a +product judgement about whether a formula and its column are one thing. + +### 2. What may a formula refer to? + +**The question.** Formulas may refer to things in a typed context. An example is a +function counting how often a letter appears in a category value. Today a formula +reference (`Expr::Ref`) names an item or a formula target. Three things are open: +- (i) may a bare id in formula text also name a category, a view or a group, with + its kind given by where it appears? +- (ii) does a text function read an item's display (`Café`) or its id (`caf`)? They + differ, because slugify folds case and strips; +- (iii) must this ADR specify that, or only keep ids unique so it can be added later? + +**Why it matters.** It decides whether the formula grammar and evaluator change with +this ADR. + +**Author's recommendation:** (iii). The formula language has no string function +today, and global uniqueness already makes a reference to any kind unambiguous. + +### 3. Does ids-first matching outlive the interim resolver? + +**The question.** Clause 12's resolver matches typed text against ids before displays, +and ids are preferred wherever there is ambiguity. Once a view-layer autocomplete +turns display text into ids, does the preference carry into it? + +**Options.** +- **(a) Yes.** Wherever typed text could name both an id and a display, the id wins, + including in the autocomplete's ranking. +- **(b) No.** Once the autocomplete exists, nothing is matched at commit, so the rule + ends with the interim resolver. + +**Recommendation:** none. It matters only once an autocomplete is designed. + +### 4. May anything other than a client resolve a display name? + +**The question.** Only a client accepts a display as a reference (clause 6). Two +places would resolve displays outside one: +- the **old-version loader**, which resolves an old file's references by display + (clause 5), because in an old file the name is the only identity; +- **headless `improvise cmd` and `improvise script`**, which run command lines naming + things by display, with no client attached. + +**Why it matters.** If neither is allowed, old files cannot be migrated and the +headless CLI needs a client layer of its own. + +**Author's recommendation:** allow both. The loader is migrating a file, not a person +referring to something. The CLI is the UI layer of its own invocation and converts at +its edge, as a client does. + +### 5. Does the `_` collapse come after the strip? + +**The question.** Clause 4 collapses runs of `_` (step 4) after stripping characters +outside the alphabet (step 3). + +**Why it matters.** `Time \u{200B} Period` (a zero-width space between two spaces) +gives `time_period` only if the collapse comes after the strip. Before it, the +zero-width character still separates the two `_`, and the result is `time__period`. + +**Options:** (a) after the strip, as written; (b) before it. +**Author's recommendation:** (a). + +### 6. Which entities share the reserved leading `_`? + +**The question.** A leading `_` is reserved for virtual categories, measures and other +virtual entities. +- (i) No virtual measure exists today. Is one planned, or does "measures" mean the + `_measure` category? +- (ii) Are the virtual views, `_records` and `_drill` (clause 1), among the reserved + entities? + +**Why it matters.** It fixes `VIRTUAL_IDS` and what `claim` refuses. + +**Author's recommendation:** (ii) yes, as clause 1 is written. Otherwise `_Drill` +slugs to `drill` and can collide with a user's view. + +### 7. Are records mode's positions under `_Index` and `_Dim` ids? + +**The question.** Records mode gives each row a synthetic coordinate under `_Index`, +and each column one under `_Dim` (a column index, gotcha 1). They are never persisted +and never typed. + +**Options.** +- **(a)** Outside the id namespace: a position `0` under `_index` does not collide + with a user's item `0`. +- **(b)** Inside it, spelled so that it cannot collide (for example `_index-0`), under + the reservation of a leading `_`. +- **(c)** Something else. + +**Why it matters.** Under (b), uniqueness is checked for synthetic coordinates, and +`claim`'s exact match on reserved ids must change. + +**Recommendation:** none. + +### 8. How do the virtual categories move to their lower-case ids? + +**The question.** Today `_Index`, `_Dim` and `_Measure` are keys in the model, on the +wire, in `ViewState` and in `.improv` files. For example, `web/demo.improv` has +`_Measure: column`, `_Index: none`, `## Category: _Measure` and `_Measure=Revenue`. +Under clause 1 they become `_index`, `_dim` and `_measure`, with the displays +unchanged. The proposed migration: +- an old file's `_Measure` resolves to `_measure` through its display at load, like + any name, and the first save writes `_measure: column` and `_measure=revenue`; +- a new-version file must spell the id, and `_Measure` there is refused (ids are + lower case); +- the header becomes `## Category: _Measure(_measure)`; +- every string literal naming them in code, on the wire and in the browser client + moves in the re-key step. + +**Why it matters.** Every shipped document and much test code spells these names. + +**Author's recommendation:** the migration above. + +### 9. Does a group also carry a display name? + +**The question.** A group's identifier is an id, not a friendly name (clause 1). Does +a group also have a separate display, as categories, items and views do? + +**Why it matters.** +- If it does not, the grid and panels show the id: `Coastal East` would appear as + `coastal_east`. +- If it does, groups need a display rename like everything else, and the file + declares both. +- The code is already moving: PR #336 (improvise-cq1q.12, open) adds a private + display to `Group`, beside its key. + +**Options:** (a) yes, a display beside the id; (b) no, the id is all a group has. +**Author's recommendation:** (a), for consistency with every other id-bearing kind and +so that users see what they typed. + +### 10. How do groups carry ids? + +**The question.** A group's identifier is an id. Not yet decided: +- (i) **Namespace.** Is a group id unique in the whole document, like every other id, + or only within its category? +- (ii) **Generation.** Is it the slug of a display (if question 9 gives it one), with + `group` as an empty slug's `` and the shared counter for collisions? +- (iii) **The file.** For example a definition `> Coastal(coastal)`, a membership + `- Coastal_East(coastal_east)[coastal]`, and a parent group (group hierarchies, + improvise-felo.6) named by id. Does a `collapsed:` line hold the group id with or + without its category id? +- (iv) **The wire.** Does `AddItemInGroup`'s `group` name an existing group by id, + create one by display, or either? Is there a `RenameGroup { group, to }`? +- (v) **The frame.** `AxisFrame`'s `FrameGroup` carries a group name today. What does + it carry? + +**Why it matters.** Clauses 2, 5, 6 and 7 cannot be implemented for groups without +these answers. + +**Author's recommendation:** (i) document-wide, as every other id is; (ii) slugify, +like every generated id. No recommendation on (iii)–(v). + +### 11. What may a hand-written id look like at its ends, and do slugify's trims repeat? + +**The question.** No id may be `-`, and no id begins or ends with `-`. +- (i) Is a hand-written id that begins or ends with `-` refused at load, as one + beginning with `_` is? +- (ii) Slugify drops a trailing `_`. Is a hand-written id ending in `_` refused too? +- (iii) Clause 4's step 5 repeats until neither end holds `_` or `-`, so `-_x` → `x`. + One pass of each would leave `_x`, which is reserved, or `x_`. Is repeating right? + +**Why it matters.** It decides what `IdAllocator::claim` refuses. Today it accepts +`-`, `-x` and `x-`. + +**Author's recommendation:** (i) yes; (ii) yes, so that a hand-written id obeys the +same shape as a generated one; (iii) repeat. diff --git a/context/code/core.md b/context/code/core.md index 49e6c119..04cbdf89 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1691,7 +1691,8 @@ probability (`> ||[||]` is a self-parent), so `generated_file_parses` accepts a `slugify` and `IdAllocator` are the first piece of ADR 007. They are pure functions of the display and the allocator's own history. Nothing calls them yet: the creation sites move onto them in the re-key step (improvise-cq1q.4). -Each rule traces to a numbered answer in the ADR's "What the user decided". +Each rule traces to a numbered answer of the user's. The ADR no longer carries +them; they are listed, with comment ids, in PR #326's provenance comment. **Case folding is full Unicode folding, and it comes from `unicase`.** `str::to_lowercase` is not folding: `ß` lowercases to itself and is then @@ -1712,8 +1713,8 @@ slugs to `a__b`: the steps are applied as written, and no extra underscore-collapsing step exists. A trailing run survives, so `East ` slugs to `east_`. Only *leading* underscores are removed (step 4). -**ADR 007 revision 9 proposes different rules, and this code predates them.** -Revision 9 is not yet approved. Its answers 34, 41, 42 and 45 make clause 4: +**ADR 007's proposed amendment changes these rules, and this code predates it.** +The amendment (PR #326) is not yet approved. Its clause 4: - each whitespace character becomes `_`, not only U+0020; - runs of `_` collapse after the strip, so `a_ b` and `a__b` both slug to `a_b` and `Time \u{200B} Period` to `time_period`; @@ -1723,8 +1724,8 @@ Revision 9 is not yet approved. Its answers 34, 41, 42 and 45 make clause 4: This section, `slugify`, and the tests `slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and `slugify_keeps_an_underscore_the_display_already_had` describe the earlier rule -above. They change in a follow-up once revision 9 is approved. The ADR's clause 4 -lists every difference. +above. They change in a follow-up once the amendment is approved. The ADR's +clause 4 lists every difference. **The allocator records every id it admits and releases none.** That is the whole of retirement (clause 4, "within a session a removed id is never handed @@ -1748,8 +1749,8 @@ the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading `_` (only `_index`, `_dim` and `_measure` may carry one, per answer 23, and the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other id may start with `_`"), and -duplicates. It accepts `-`, `-x` and `x-`. Revision 9's Q34 asks whether a -hand-written id like that must be refused (answer 45). It never renames. Refusing is the loader's job, and the loader +duplicates. It accepts `-`, `-x` and `x-`. ADR 007's open question 11 asks +whether a hand-written id like that must be refused. It never renames. Refusing is the loader's job, and the loader must name both lines of a duplicate; `ClaimError` carries only the id, so the loader keeps the line numbers. The virtual views `_records` and `_drill` are not yet in `VIRTUAL_IDS`. improvise-p003 item 1 asks for them, and they arrive From b7add90830a1741cd27abe7252ebd937eba6a76a Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 23:09:57 -0700 Subject: [PATCH 060/106] docs(adr): ADR 009 condensed to problem, decision and open questions (improvise-felo.4) Rewritten on the user's instruction: settled questions are removed, revision and review history is removed, the decision is stated plainly, and only unsettled questions remain, renumbered 1-17 for a first-time reader. The record of the user's answers moves to a PR comment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 3275 ++++----------------- 1 file changed, 612 insertions(+), 2663 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 8d2e9bd5..e78b1398 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,2782 +1,731 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait -**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and -12** (see "What the user decided"). Question 11 and questions 13 to 28 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** -(PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in -the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details -ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 28. *Superseded:* until 2026-10-09 the -user had answered none of questions 1 to 11. · -**Ticket:** improvise-felo.4 · -**Decision from:** the user, 2026-09-25 (below) · -**Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8; PR #320 merged on 2026-10-09, and its own status line still reads proposed), ADR 008 (the -virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `c8be7720`, its revision 37) · -**Governed by:** `context/design-principles.md` §3 and §5 +**Status:** proposed. Only the user accepts an ADR. · +**Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own +epic) · +**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed +models) · +**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") · +**Governed by:** `context/design-principles.md` §3, §5 and §7 -**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 -is on main since PR #320 merged (2026-10-09), and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and -no open PR claimed it when this was written. +**Summary.** Formulas are evaluated incrementally by a small in-house implementation of +the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A +cell write becomes a delta. Aggregates are maintained rather than re-folded, and only +the formula cells whose inputs moved are re-evaluated, by the one existing evaluator. +Operator state is keyed by the **shape key** of a projection request, never by a client. +The same circuit serves the in-process TUI and remote clients. A remote client learns of +changes from one unaddressed notice per accepted batch, and it refetches what it shows. +Sums are exact, through an exact accumulator. -**Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are -dated, so re-derive them before citing this document elsewhere. **The 2026-10-05 -revision's new citations are at `gh/main` `50e1314d`**, and each says so. The older -citations were not re-derived, and some have moved. For example, `Host::mutating_frame` -takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and -`App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 37 of `context/adr/008-virtualized-grid.md` on PR #308, at `c8be7720`. Clauses 3, -5 and 8, which this ADR cites most, have the same numbers there as in revision 3. +File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive +them before relying on one. -## The decision this ADR writes down +## Problem and context -The user pointed at a teaching implementation of the DBSP paper, -[brurucy/dbsp-from-scratch](https://github.com/brurucy/dbsp-from-scratch/blob/master/dbsp_paper_walkthrough_implementation.ipynb), -as a way to evaluate formulas so that data can stream. The notebook builds, in order: +**Data that streams.** A live-backed model (ADR 003) is a source whose rows change +underneath the document. Today every change takes the path an edit takes, which +recomputes every formula cell. That is tolerable for one keystroke at a time and not for +a stream of changes. A change should cost in proportion to what it touches. -- abelian groups; -- streams, with `Delay` (z⁻¹), `Differentiate` and `Integrate`; -- Z-sets, which are maps from element to an integer weight; -- lifted and doubly-lifted joins, and a delta join that keeps less state; -- `distinct` through the `H` function over an integrated state; -- recursion by nested streams, shown on graph reachability. +**Today's evaluation is a from-scratch fold after every batch.** +- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls + `Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates + every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16` + (`types.rs:710`). +- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`, + `types.rs:848`, through `DataStore::matching_values`). +- `formula_cache` holds one aggregation shape at a time. A session on a view with another + None axis gets a **cloned model** (`rederived_for`, `project.rs:205`). +- A formula total that holds an aggregate walks a cross product + (`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`). +- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing + batch, and `document_content` (`app.rs:168`) after each `CellData` effect. +- Every mutating frame from a client clones the document for rollback + (`Host::mutating_frame`, `host.rs:224-225`). -The user then decided: +**What an incremental evaluator must not change.** Each of these is a place where it +could quietly disagree with today's code: +- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one + distinct value is that value, and anything else is `Error("mixed")`. +- **Circularity (gotcha 9).** A cell still moving after 16 passes is + `Error("circular")`, and so is an acyclic chain that needs all sixteen. +- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0 + guard. +- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration, + because f64 addition is not associative. +- **The tiers are observed (gotcha 40).** A write of the value already there bumps + nothing, and a rewritten record counts once. -> "I think doing our own implementation of the core of dbsp as a crate we can plugin is -> probably dieal [sic] here. Write the ADR and Update ADR 008 to assume the decisions in -> that one" -> -> — the user, 2026-09-25, recorded on improvise-felo.4 - -So the choice between Feldera's crate and our own is already made (see Alternatives for -what was checked). This ADR says what "the core of dbsp" is for improvise, where it -plugs in, and what it may not change. - -## What it is for - -**The primary benefit is incremental formula evaluation, so that data can stream.** -That is the user's own framing, both in the prompt that led to this ADR and in their -later question: "Isn't the benefit of adopting DBSP primarily incremental formula -updates?" Yes. When a cell changes, whether from a keystroke, a client's frame or a -live source (ADR 003, clause 10), only the aggregates that cell feeds are adjusted, -and only the formula cells whose inputs moved are re-evaluated. Nothing is recomputed -from scratch. The machinery is: - -- the aggregates formulas read are **maintained**, not re-folded (clause 4); -- an aggregate over a per-record expression such as `SUM(Revenue * Price)` is a - maintained operator too (clause 5a); -- the circuit decides **which** formula cells moved, and the one existing evaluator - recomputes only those, in dependency order (clauses 5 and 6). - -**The grid is the second consumer.** ADR 008 needs the same maintained state for other -purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, -and column totals over rows never scrolled to (clause 9). It gets them from the same -circuit, not from a second mechanism. Since ADR 008's view/host split, the grid's consumer -may be in another process, and clause 12 says how the same circuit serves both. - -## What ADR 008's review established for this ADR - -At its revisions 18 to 37, ADR 008 (PR #308) became the ADR of the **view/host split**, and the -user routed part of it here. This section quotes what the user said that bears on this ADR, -and says where each statement came from. -- **PR comments** are cited by their GitHub comment id and UTC time. The question numbers - are ADR 008's, as ADR 008 assigns them. -- **Fourteen chat statements**, C1 to C14 below, were given to the orchestrator and relayed - to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR - would confirm them. ADR 008 records the same statements, also as relayed and not on the - PR, but spread over several groups of its own. ADR 008's list also has six more (its - items 1 to 6: the five standing design statements, and the "q19 … q24" corrections). - This ADR relies on none of those six. This list is the one place this ADR - quotes them, and the clauses cite them by number. - -None of this answers this ADR's own questions 1 to 11. - -**Relayed in chat, on no PR** (all on 2026-10-05; quoted exactly as relayed): -- **C1.** "The DBSP machinery needs to be implemented in a network transparent way, but - the details of that go to ADR 009". Clause 12. -- **C2.** "Thinking about it a bit more, we might need to keep track of recent projections - + clients to stream aggregation results back to the UI correctly. But this should be as - stateless as possible". Clause 12. -- **C3.** "Q53 is cross product: I'm still ambivalent about which side of the wire pruning - should happen on". ADR 008 records it as its question 53 answered and its question 57 - open. Clause 12, question 19. -- **C4.** "But the top left should be as absolute as possible". Clause 12, question 20. -- **C5.** "After pruning we might have to stream rows and columns to attempt to fill out - the full grid". Clause 12, question 21. -- **C6.** "Pruning is just a filter applied to full columns a rows, right? Hide a - column/row in the grid if it's all null. That sounds like a UI side concern because - that's where the visible grid lives". Clause 12, question 22. -- **C7.** "The point of pruning is to make the displayed values more dense, so a value - being present outside the viewport doesn't matter." Clause 12, question 22. -- **C8.** "The problem pruning solves is visualizing sparse datasets: screens that are - mostly empty are useless and so you drop out categories from rows and columns until - every visible row or column has at least one value on screen." Clause 12, question 22. -- **C9.** "We should be able to optimize pruning on the right and down assumption of - growth (truncated at the bottom right corner of the axis space)". Clause 12; ADR 008's - question 67. -- **C10.** "The app is free to optimize fetching for up or left by just fetching the new - rows, until it finds one that has values in the viewport but, conceptually, scrolling up - or left would request a new viewport with a new top left corner. The requested top left - corner falling on a pruned row or column is an annoying edge case, maybe just fail the - fetch and retry?" Clause 12, question 22(d). -- **C11.** "Snap forward is going to cause issues with the viewport getting stuck". - Clause 12, question 22(d). -- **C12.** "The simplest might be some kind of binary search for a new top left corner: - increasing jumps of the top left until you get a hit and then move down/right until - You only have one new row column." Clause 12, question 22(d). -- **C13.** "I think we can minimize the amount of UI state on the backend by having an - operator of some kind responsible for both ends of the connection between the frontend - and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). - A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, - 14 and 28. -- **C14.** "And it asks as a sink/source for the DBSP graph part on the backend and a - source/sink for the client side" (relayed 2026-10-09, after C13). Quoted exactly as - sent. *The ADR reads "asks" as "acts"*, flagged. Clause 12, question 28. - -**Asked on ADR 008, and not yet answered:** -- "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, - 2026-10-03T03:15:38Z, on ADR 008's question 2). Clause 13 is this ADR's reading of it. - Question 12. - -**The read side and the write side** (#308, inline comments of 2026-10-05): -- Q43 (4189071865, 21:34:45Z): "yeah, the read side of the model is a projection of the - model's full space into a particular set of axes using a `project` verb, with - offset+dimensions. The write side is an apply function that can only take model - commands and applies them to the model using Redux-style Flux architecture." -- Q38 (4189007864, 21:27:55Z): "No, axes are a model concept because they define the - grid. Aggregation should always be done model-side for consistency." -- Q38 (4189027924, 21:30:08Z): "A grid request would look something like:" - ``` - Axes: Row(A,B,C), Col(D,E), Page(F), None(G,H) - Top-left (4,5) # relative to the cross-product of the axes in the category-native order (which might be arbitrary) - Dimensions (10,10) # so get ten rows and ten columns worth of data from the grid defined by the axes. - ``` -- Q33 (4188936570, 21:20:11Z): "The host shouldn't have to know anything about the - viewport: it's mostly a question of the parameters of the grid the user is actually - looking at, rather than a "viewport" strictly speaking." -- Q37 (4189000059, 21:27:04Z): "I don't think the host needs to no [sic] anything about - the UI: the initial load of the UI should just be the default view stored in the model - (or, in some cases, a particularly requested view). Sending UI state over the wire is - an architecture smell." -- Q42 (4189051110, 21:32:31Z): "The space scrolling happens in is defined by the axes, top - left corner offset and with/height. But how much of this space is actually materialized - and when fetches, etc. happens (and model values are cached) is client-policy driven." -- Q32 (4188917796, 21:18:17Z): "I can't think of any cursor command with model-side - effects." -- Q27 (4188816383, 21:07:43Z): "go with the recommendation: the ordered list of records is - a model-side concept." -- Q31 (4188911659, 21:17:39Z): "You have to distinguish view in the sense of UI and view - in the sense written into `.improv` files. P has to be a model-side effect because it - prunes data for a particular view. It is not a UI-only command." And (4188926877, - 21:19:10Z): "The only real issue here is that pruning shouldn't infect every client - here. It might need to be UI-only until a view is saved, and then the view definition - goes to the model." -- Q29 (4188887721, 21:15:14Z): "Prune should only be written if the user explicitly - unpruned the view." -- The cursor: "The cursor should never jump: it should stay where the user left it as much - as possible." (Q12, 4171501445, 2026-10-03T03:21:05Z). And "As far as possible, updates - should shift the grid being viewed to keep the cursor in the same visual place on the - screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). - -**Answered on ADR 008 on 2026-10-06** (#308 inline comments; question numbers are ADR -008's): -- Q33 (4191973298, 05:50:48Z): "The backend should only hold model-relevant information. - If we need to persist cursor position/last view or other UI concerns, that would go in - localStorage/cookies/fragment routing/etc." -- Q39 (4191981468, 05:52:14Z): "I think layout version is mostly a view fact here, we have - to separate this from model mutations (value editing, etc.)" -- Q46 (4191986207, 05:53:05Z): "go with an explicit saye-view command" [sic]. -- Q47 (4191992428, 05:53:56Z): "The transient UI state never is reflected in a .improv - file unless an there is an explicit decision to persist it as a saved view (or by - replacing an existing view)" [sic]. -- Q48 (4191999772, 05:55:06Z): "A .improv file has to have at least a default view - definition. UI state is always ephemeral unless explicitly saved: when a new View is - created through the UI, it must not be lost if the model confirms that it was saved." -- Q49 (4192005787, 05:56:07Z): "The only one of these that should be "global" is set - number format as far as I can tell: the other ones are all ephemeral UI state." -- Q50 (4192010101, 05:56:53Z): "Disabling pruning should only be persisted if the - resulting view is explicitly saved." -- Q51 (4192014974, 05:57:40Z): "page is a third axis, do the others need to actually - travel across the wire?" -- Q52 (4192016491, 05:57:59Z): "defer this to ADR 009". This is ADR 008's question - about what replaces the pushed projection stream, and it is this ADR's question 13. - -**Answers ADR 008 records that this ADR's text had not caught up with:** -- Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. -- Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases - where the column size has been stably smaller." And Q22 (4188309022, - 2026-10-05T20:15:12Z): "hysteresis as recommended." Clauses 9 and 13. -- Q6 (4171493762, 2026-10-03T03:18:12Z): "maintained incrementally". Clause 9. -- Q13, Q14 and Q15: "the recommendation is fine.", three comments (4171504439, - 4171505979 and 4171509012, 2026-10-03T03:22:16Z to 03:23:36Z), which ADR 008 records as - its answers to those three questions. Q13 sets the frame budget, and Q15 sends the - formulas the per-record map cannot express to the occupied contexts. Clauses 5a and 14. -- The store's memory (4171481179, 2026-10-03T03:13:14Z): "We have to be careful here to - minimize memory requirements of the store: offloading inactive regions to disk (maybe - mmap or similar), double-delta/gorilla/lz4 compression, etc." And Q17 (4188271710, - 2026-10-05T20:11:02Z): "Let's defer this as recommended." Clause 14. -- Jobs (4171458281, 2026-10-03T03:04:52Z): "We also have to have a way to - invalidate/cancel in-flight jobs efficiently and probably a way to throttle them (a - priority queue or similar) to avoid pinning the CPU." Then (4171694278, 04:30:31Z): - "priority queue for work stealing is also important here." Clause 7. - -**What this ADR reads from them.** These are the author's readings, and each one names -the clause it changes: -1. **Reads and writes go through `project` and `apply`.** A consumer reads the circuit's - outputs through `project`, and the circuit's inputs change only through `apply` of - model commands. So nothing here may depend on a host-held mirror of a client - (`Sessions`, `ClientSession`, `reduce_view` on the host), on the per-mirror diff - (`grid_updates`), or on resetting a client's cursor. Clauses 9 and 12. -2. **Aggregation stays on the model side.** No client runs an aggregate operator, so - network transparency concerns the circuit's outputs, and a live source's inputs. It - does not mean splitting operators across a wire. Clause 12. -3. **State is keyed by the request's shape, not by a view's name.** Axes are a model - concept, so operator state keyed by axes is model state, not UI state. But a `project` - request names its own axes, and an unsaved request may carry parameters no saved view - holds (ADR 008's flagged readings 1 and 3, its question 51). Clause 7, question 14. -4. **The record order belongs to the store.** The ordered list of records is the store's - (Q27). The circuit may consume it, and does not own it. Clause 9. -5. **The memory requirement covers this ADR too.** The store-memory requirement covers the - circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. - Clause 14, question 17. -6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's - questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 39, 42, - 43 and 46 to 52, quoted above. - Clauses 5a, 7, 9, 12, 13 and 14 follow them. -7. **View configuration is the client's until saved** (Q46 to Q50). Hiding, collapsing, - page selection, axis assignment and pruning are ephemeral UI state per client until an - explicit `SaveView`. Only number format is "global" (Q49). So operator state is keyed - by the request, never by a client's unsaved settings held on the host. Clause 7, - questions 23 and 24. -8. **No basis tier this ADR relies on is a view fact** (Q39), and the host keeps no - per-client UI state (Q33). Clauses 7 and 12, question 13. - -## Context - -### The primary case: data that streams - -A live-backed model (ADR 003) is a source whose rows change underneath the document. -Today the only way to reflect a change is the path every edit takes: recompute every -formula cell of the document (next subsection). That is tolerable when a human types -one cell at a time. It is not tolerable when a source delivers a stream of changes, -each touching a few records of a large table. Incremental evaluation makes the cost of -a change proportional to what the change touches, not to the document. - -### Evaluation today is a from-scratch fold, rerun after every batch - -- **Every batch recomputes every formula cell.** `App::apply_effects` - (`crates/improvise-command/src/ui/app.rs:1069` at `50e1314d`) ends in `rebuild_layout` - (`:926`). - That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). - It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), - and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` - (`:699`, `:1001-1022`). A cursor move and a cell write pay the same. -- **Raw aggregation scans the matching cells on every read.** `evaluate_unfolded` - (`types.rs:837-845`) folds `DataStore::matching_values` through - `aggregate_cell_values` (`:30-42`). `fold_stored` (`:918-930`) does the same for a - collapsed group, through `spanned_values` (`crates/improvise-core/src/model/cell.rs:445`). -- **`formula_cache` holds one aggregation shape at a time.** It is keyed by cell and - computed for one `none_cats` list. A session on another view with a different None - axis gets a *cloned model* recomputed for its own list (`rederived_for`, - `crates/improvise-command/src/ui/project.rs:195-210`). -- **A formula total with an aggregate inside walks a cross product.** `contexts_of` - (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every - combination of the spanned categories' items. ADR 008 clause 5 names this as the one - O(grid) path left in totals. -- **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes - every cell before and after a cell-writing batch. `document_content` - (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is - re-read after **each** `CellData` effect (`app.rs:1119-1125` at `50e1314d`). So the first is O(cells) - per batch and the second O(cells) per effect, whatever was written. -- **Every mutating frame from a client already clones the document.** - `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, - `app.rs:673` at `50e1314d`), for ADR 001 clause 4. That is O(cells) per frame on the network - path, whatever this ADR does. - -### What an incremental evaluator must not change - -These rules come from the repo map and `context/code/core.md`. Each one is a place an -incremental evaluator could quietly disagree with the eager one. - -- **Gotcha 7, text unanimity.** `aggregate_cell_values` checks the all-numeric branch - first. Otherwise one distinct value is that value, and anything else is - `Error("mixed")`. `Count` of a lone number is `1`, not the number. -- **Gotcha 9 and core.md §3, circularity.** A cell still moving after 16 passes is - written as `Error("circular")` (improvise-7iww). **A chain that needs all sixteen - passes is also reported circular**, because the fixed point can only confirm - convergence with a pass that changes nothing. The direct evaluator agrees because its - recursion depth is the same 16 (`types.rs:705-714`). -- **Gotcha 17, one epsilon.** `FLOAT_EQ_EPSILON = 1e-10` (`types.rs:14`) is shared by - `=`, `!=` and the div/0 guard, in both evaluators. -- **Gotcha 18, ordering is correctness.** A `HashSet` index is for lookup, never for - iteration. `SUM` and `AVG` drifted in the last bits when the store was walked in hash - order (improvise-shyt), because f64 addition is not associative. -- **Gotcha 40, the tiers are observed.** A write that lands the value already there - bumps nothing. The record set is collected into a *set*, so a rewritten record counts - once. +**The view/host split (ADR 008).** A client reads through `project(request)`, where a +request names axes, an absolute top-left and dimensions. It writes only through +`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys. +Aggregation is always done on the model side. A view's hiding, collapsing, page +selection, axis assignment and pruning are one client's ephemeral state until an +explicit save-view command. Only the number format is document-wide. ## Decision -**Formulas are evaluated incrementally. A change re-evaluates only the formula cells -whose inputs moved, over aggregates that are maintained rather than re-folded. The -machinery is a new, problem-agnostic workspace crate implementing the DBSP core. -improvise-core defines a trait for the derived state its evaluator reads. Today's -evaluator is one implementation of that trait, and a circuit built on the new crate is -the other. Both stay in the tree, and a differential test holds them equal.** Fourteen -clauses, plus clause 5a. It is numbered 5a so that no clause number or question number -cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. +### 1. The crate: `zset` -### 1. The crate: `zset` (the user's answer to question 1) +A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on +`std`, so it is wasm32-clean by construction. It contains: +- **`ZSet`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an + abelian group. Weights use checked arithmetic, and a zero weight is removed. The group + laws are property-tested. +- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per + source and returns one per sink. There is one clock, one thread and no scheduler. +- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id + property-tested); the linear `map`, `filter` and `flat_map`; `index_by`; + `join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear + aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a + multiset per group. +- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a + resumable `step(deadline)`, so a rebuild can run as a sliced job. -Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond -`std`, so it is wasm32-clean by construction. +Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one +clock (improvise applies batches in one total order); threads within one circuit (a host +with threads may run independent circuits on a pool); a query language. Spilling to disk +is not in the first version, but operator state sits behind one interface (a trace: look +up, read a range, apply a delta), so the store-memory work can page or compress it later. -**What it contains:** +### 2. The plug: a trait, and a seam crate -- **`ZSet`**: a map from `K: Ord + Clone` to a non-zero `i64` weight, with `add`, - `neg` and `zero`. It forms an abelian group, and its laws are property-tested as the - notebook's first cell does: commutativity, associativity, identity and inverse. - Weights use checked arithmetic, so an overflow is a panic, not a wrap. A weight that - reaches zero is removed, so equal Z-sets are equal as maps. -- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one input - delta per source and returns one output delta per sink. There is one clock, one - thread, and no scheduler. -- **Stream operators:** `delay` (z⁻¹), `integrate` (I) and `differentiate` (D), with - I∘D = D∘I = id tested as a property. -- **Linear operators, which are their own incremental form:** `map`, `filter` and - `flat_map`. -- **`index_by(key)`.** It turns a `ZSet` into an indexed Z-set `K → ZSet`, which - is the input shape for grouping and joins. -- **`join_indexed`.** The bilinear delta join: Δ(a ⋈ b) = Δa ⋈ Δb + I(a)₋₁ ⋈ Δb + - Δa ⋈ I(b)₋₁, over integrated indexes. It is needed for one purpose, clause 5's - cross-stem references. -- **`distinct`.** It is `H` over the integrated input, as in the notebook. The output - delta is non-empty exactly when an element's total weight crosses zero. -- **`aggregate(group, Aggregator)`.** Its output delta retracts the group's old value - and inserts its new one. The trait has two kinds of implementation: - - **linear:** `Count`, and `Sum` for any `G: AbelianGroup`. The state is one value - per group, and each delta element costs O(1). - - **integrated:** the aggregator keeps the group's integrated multiset, - `BTreeMap`, and reads its answer off it. `Min` and `Max` are the first and - last keys. Unanimity (clause 4) is "exactly one distinct key". Each delta element - costs O(log d), where d is the number of distinct values in the group. +improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the +formula result at a key, and the raw fold of the cells that match a partial key. It +takes the cell delta and the formula delta after each model effect. -**What it leaves out, deliberately:** +**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and +read through the trait. The circuit decides *which* formula cells to re-evaluate and +supplies their inputs. Formulas are not compiled into operators, because a compiled +`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the +defect. -- **Nested streams and least-fixed-point recursion** (the notebook's - `LiftedLiftedJoin`, `DeltaLiftedDeltaLiftedDistinct` and `IncrementalGraphReachability`). - improvise's circular semantics are not a least fixed point (clause 6). -- **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies - batches in one total order (ADR 001, "There is already a total order"). -- **Spilling to disk, compression, and threads, in the first version.** State lives in - memory, and a browser worker has one thread. **This was a non-goal and is now only a - deferral.** The user's store-memory requirement (ADR 008 clause 12) covers this state - too, and clause 14 says what the crate must not rule out. A host with threads may run - independent circuits on a pool (ADR 008 question 23's answer). One circuit's `step` - stays single-threaded. -- **A query language or planner.** Circuits are wired by hand in the seam crate - (clause 2). +**The incremental backend lives in a seam crate, `improvise-incremental`,** which +depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same +shape as `gp-tree` and `improvise-gp`. -**The weight type is `i64`.** Integer weights make `ZSet` an exact group, so -consolidation and retraction never drift. The drift problem is in the *values* being -summed, not in the weights (clause 8). +**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change, +as `Eager`. It stays as the oracle for the differential test while the circuit matures, +and for the two cases that still need it (clauses 5 and 10). The differential test is a +property test in the seam crate that applies one random edit sequence to both +backends and compares every drawn cell, total and occupancy count. The goal is that every +formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is +open question 15. -### 2. The plug: a trait in improvise-core, and a seam crate that implements it +### 3. The input: the cell delta -**improvise-core gains a trait for the derived state its evaluator reads.** The working -name is `Derived`. It replaces the two places where eager evaluation folds the store: - -- the formula result at a key, which is `formula_cache` today (`types.rs:370`); -- the raw fold of the cells matching a partial key under an aggregation, which is - `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). - -It takes two inputs, both applied after each model effect (clause 7): the cell delta, -and the formula delta. It also has a rebuild, for a new shape key (clause 7). - -**The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code -and read through the trait. The circuit decides *which* formula cells to re-evaluate -and supplies their inputs, and the existing per-cell evaluator computes the value. -Formulas are never compiled into operators. A compiled `IF`/`POW`/`WHERE` would be a -second implementation of evaluation, which design-principles §5 calls the defect -itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental totals. - -**Two implementations:** - -- **`Eager`** is today's code moved behind the trait, with no behaviour change: `apply` - records the delta, and the post-batch refresh is `recompute_formulas`. -- **`Incremental`** lives in a seam crate, working name `improvise-incremental`, which - depends on `zset` and `improvise-core`. This is the `gp-tree`/`improvise-gp` shape: - improvise-core never depends on `zset`, and it can be built and tested eager-only - (question 2). - -**Why keep both, for now.** `Eager` is the oracle for the differential test. It was -also to be the backend a query-only source keeps (clause 10). **The user: "Eager should -be deprecated", and "For now: the goal is to only have DBSP formulas"** (questions 8 and -6). So `Eager` stays only as long as the differential test needs an oracle, and what -replaces it for query-only sources and for clause 5a's non-record-local formulas is -question 26. *As first written* (superseded), this paragraph and question 6 kept `Eager` -permanently. The differential test is a property test -in the seam crate. It generates documents with groups, collapse, hidden items, formulas -(including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text -cells and non-finite numbers, applies the same random edit sequence to both backends, -and compares every drawn cell, every column total and every occupancy count. - -### 3. The input: a cell write is a retraction plus an insertion - -**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue, Seq)>` with -weights ±1, where `Seq` is the cell's store sequence (clause 5a).** `DataStore::set` and -`DataStore::remove` (`cell.rs:286`, `:325`) are the only places a cell changes, so the -delta is emitted there, each element carrying the sequence `s` of the cell it names: - -- a new cell is `+(k, v, s)`, with a fresh `s`; -- an overwrite of `v₀` with `v₁` is `−(k, v₀, s) + (k, v₁, s)`, the key keeping its `s`; +`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a +`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number +assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an +overwrite: +- a new cell is `+(k, v, s)`; +- an overwrite is `−(k, v₀, s) + (k, v₁, s)`; - a removal is `−(k, v₀, s)`; -- a remove-then-set within one effect is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. +- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. -**The `(key, value)` projection is derived, not a second stream.** The record tier -and any consumer that does not need order read the delta mapped through -`(k, v, s) ↦ (k, v)` and consolidated, a `ZSet<(CellKey, CellValue)>`. Only clause 5a's -stage reads `s`. Defining the canonical element without `s` would strip it, and a -same-value remove-then-set would cancel, defeating 5a. +**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their +strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an +equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a +change, and a NaN rewritten with the same bits is not. Multisets order values by +`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted +`Vec`, so deriving it is sound. -An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty -Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once -per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy -inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation -under the view/host split. The per-effect step stays as the circuit's own clock. What a -client counts is clause 12's publication number, which moves once per published batch -(T3). +**What the delta replaces.** The record tier reads the records named by the +consolidated `(key, value)` projection of the delta, instead of diffing two store scans. +The dirty flag's content half becomes "a non-empty projected delta, or a moved +formula-set fingerprint". The fingerprint is needed because formula edits move no cell, +and the worker persists on this flag. -**A delta key's identity is bit identity, the same identity the record tier uses -today.** Two elements have the same `(key, value)` part exactly when their keys are -equal and their values are equal *by bits*, and bit identity is stated over that part -only (the sequence is compared as the integer it is, which keeps an -equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and -`Error` by their strings. That is `hash_value` (`app.rs:246-262` at `50e1314d`), which -`record_fingerprints` already hashes. So the cell delta reproduces today's tier -exactly: +**The circuit steps once per model effect.** That keeps occupancy current inside the +effect loop. What a client counts is a separate publication number (clause 12). -- overwriting `0.0` with `-0.0` is a non-empty delta and bumps the record, as it does - today; -- overwriting a NaN with a NaN of the same bits is empty and bumps nothing, as today. +**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`, +the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a +rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set +fingerprint, taken today over `(raw, target, target_category)`, must then be taken over +the id form, or a rename reads as a formula change. -It is **not** `CellValue`'s derived `PartialEq`, under which `-0.0 == 0.0` and -`NaN != NaN`. `recompute_formulas` uses that one for its "did this cell move" test -(`types.rs:1010`), and it stays there. The ordering behind a multiset (clause 4) is -`f64::total_cmp`, whose equality *is* bit equality for every finite and non-finite -value. So the delta identity and the multiset ordering agree, and neither is -`PartialEq`. Step 3's tests pin both cases: signed zero, and a repeated same-bits -NaN. +### 4. Aggregates: linear where they can be -**This lines up with ADR 001's record tier with no extra rule.** The records a batch -moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys -in the batch's consolidated delta. So: +**The group** is the cell key minus the None categories, which is the partial key +`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the +same None list, which removes `rederived_for`'s model clone. -- a write of the value already there moves nothing, which is gotcha 40's rule; -- a record gained, lost or rewritten is named once, because a Z-set is already a set of - keys. +| answer | state per group | cost per delta element | +|---|---|---| +| `Count` | a count of values and a count of numbers | O(1) | +| `Sum` | an exact accumulator (clause 8) | O(1) | +| `Avg` | `Sum` / `Count` | O(1) | +| `Min`, `Max` | a multiset of the numbers | O(log d) | +| unanimity and `mixed` | a multiset of all values | O(log d) | -`bump_changed_records` can then read the delta instead of diffing two -`record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta -comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, -because it observes `view_shape` and not cells. Nothing in this ADR reads or bumps it. -The user calls it "mostly a view fact", to be separated from model mutations (ADR 008 -Q39). ADR 001's layout tier is to be split (ADR 008 question 68), and this ADR relies -only on `document_generation` and the record tier. +Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules +for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential +test generates signed zeros and non-finite values to pin this. Only measures whose +aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups, +they hold at most one entry per stored cell. -**The dirty flag's content half does not move onto the cell delta alone.** -`document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it -is re-read after every `CellData` effect (`app.rs:1119-1125` at `50e1314d`). `AddFormula`, -`RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, -`:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a -non-empty cell delta means dirty" would never save a formula edit in the browser. The -content half becomes: **a non-empty *projected* cell delta (the `(k, v)` projection, so a same-value -remove-then-set raises nothing), or a moved formula-set fingerprint**. -The fingerprint is the one clause 7 already reads after each model effect, so nothing -new is observed. Its cost is O(formulas), where `document_content` is O(cells) per -`CellData` effect. - -**`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` -(`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only -`PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by -`f64::total_cmp`. Clause 4 says where that differs from `==`. - -**Names and ADR 007 (added on 2026-10-05).** ADR 007 is accepted. It makes ids the keys -and names display-only, and in it a `CellKey` becomes sorted `(CategoryId, ItemId)` pairs. -The user said there: "This is a lot lower value if CellKeys, etc. don't refer to the new -id". Every key this ADR defines follows `CellKey`: the cell delta, `CellKey`'s `Ord`, the -stems, `record_of`, the group keys, the shape key's categories, and the output's tuple -identities (clause 12). So **under ADR 007 they are id-keyed with no change to this ADR, -and a rename emits no cell delta and rebuilds no operator.** It is one `display` write, -which the output carries as a relabelling, not as a retraction and an insertion. Two -places do not follow `CellKey`, and must move with ADR 007: -- **the formula fingerprint** (clause 7). It is taken over `(raw, target, - target_category)`, which are names today. Under ADR 007 it must be taken over the - formula's id-form, or a rename would read as a formula delta and re-evaluate every - affected target; -- **the cross-product order** (clause 12), which orders items by the category's own - order. That order is unchanged by a rename. - -Until ADR 007 lands, a rename is what it is today: a remove and an insert of every cell -under the old name, and this ADR's deltas say exactly that. - -### 4. Aggregates: linear where they can be, an integrated multiset where they cannot - -**The group** is the cell key with the view's `none_cats` removed. That is the partial -key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct -`none_cats` list, shared by every projection shape that has that list (clause 7). That replaces -`rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group -comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does -(`types.rs:717-722`). - -**What each group keeps:** - -| answer | state per group | cost per delta element | exact on f64? | -|---|---|---|---| -| `Count` | count of values, count of numbers | O(1) | yes (integers) | -| `Sum` | running sum (an exact accumulator under question 3's recommendation) | O(1) | **no** as an f64; **yes** as the accumulator (clause 8) | -| `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | -| `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | -| unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | - -Reading a group reproduces `aggregate_cell_values`'s branch order: - -- if the count of numbers equals the count of values, the numeric answer; -- else if the value multiset has exactly one distinct key, that value; -- else `Error("mixed")`. - -The empty group is absent. The formula-level `eval_agg` (`types.rs:55-87`) reads the -same state with its own emptiness rule, `0` for `Sum` and `Count`, at read time. - -**Memory.** Summed over all groups, the multisets hold at most one entry per stored -cell. So the state for one `none_cats` list is O(cells), on the order of the store it -summarizes. **Estimate, not measured:** a `BTreeMap` entry of about 40–60 bytes plus the -key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the measures -whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A -`Sum`-only document keeps one number per group. - -**The multiset's identity is clause 3's bit identity, and eager's *reading* is not.** A -multiset keyed by `total_cmp` holds the values the store holds, by bits, and that is -correct for the state. But eager *reads* a group through `PartialEq` and `f64::min`: -`NaN != NaN`, so two NaNs in one group read as `mixed` today, `-0.0` and `0.0` read as -one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eager's -rules explicitly and does not inherit `total_cmp`'s. The differential test generates -signed zeros and non-finite values (core.md §16) to pin it. - -#### Ordered aggregates: `LAST` and `FIRST` need state ordered by an item, not a value - -improvise-felo.7 asks for `LAST` and `FIRST` over a fold: the closing or opening value -of a stock price, say, across the months of a year. improvise-felo.8 asks for -semi-additive measures: `sum` across one category and `last` across another. Both are -open tickets with their own semantics to decide. What this ADR must say is whether its -per-group state can carry them. Today it cannot: the integrated state above is an -**unordered** multiset, keyed by the value itself. - -**The structure.** An ordered aggregate keeps, per group, a `BTreeMap` keyed by the -**ordering category's item order**, with the tie-break felo.7 decides. Each entry holds -the values at that position with their weights. `LAST` reads the entry at the greatest -key, and `FIRST` the least. - -- **It is the same shape as `Min`/`Max`.** Those also keep an integrated ordered map and - read an extreme; they order by the value, where `LAST`/`FIRST` order by the item's - position and read the value stored there. So one generic operator covers all four: - an integrated map keyed by `K`, read at an extreme. `Min`/`Max` instantiate it with - `K = value`, and `LAST`/`FIRST` with `K = (item position, tie-break)`. -- **Cost:** O(log d) per delta element, where d is the number of distinct positions in - the group, and one entry per contributing record. That is the same order as - `Min`/`Max`'s multisets. -- **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. *As first written*, this item - said a reorder of drawn items already rebuilds, through `view_shape`. **That is - superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a - reorder rebuilds nothing, and a state keyed by position would go stale. The position of - a **hidden** item matters only if `LAST`/`FIRST` must order values that come from - hidden items. There are three keys: - - **position**, which goes stale on a reorder now that nothing rebuilds; - - **stable item identity** (ADR 007), with the order looked up at read time, which - makes a reorder free, but then a read costs O(d); - - an **order label**, which compares in the current order and is not renumbered by an - insert elsewhere. Reads stay O(log d). A reorder relabels the moved items, and - re-keys every group entry stored under them, at O(entries under the moved items). - - This is question 11. -- **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, - `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a - delta (retract the old value, insert the new), and the outer consumes it. So the - composition is incremental at O(log d) per stage. Which stage is inner is felo.8's - semantic decision, because sum-of-last and last-of-sum differ. The circuit takes - either order. -- **Exactness.** `LAST`/`FIRST` return a stored value unchanged, so they are exact on - f64 like `Min`/`Max`. A semi-additive `sum` stage still needs question 3's answer. +**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures, +improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an +extreme, keyed by the ordering category's order instead of by value. A semi-additive +measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the +order is keyed is open question 2. ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it -**Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key -with the target category and each `none_cats` entry removed. That is -`distinct(map(project))` over the cell delta. A stem enters when its first cell arrives -and leaves when its last cell goes. +**Stems** are `distinct` over the cell delta, mapped to "key minus the target category +and the None categories". A formula counts at a stem only where its top-level `WHERE` +(if any) matches the stem's coordinate, or where raw data exists at stem + target. That +is exactly where `recompute_formulas` caches a value today, so pruning keeps its current +behaviour. -**The stem rule: a formula cell exists at a stem only where its value would be cached -today.** `recompute_formulas` caches a formula cell only when the cached evaluator -returns a value (`types.rs:1010-1014`). For a formula with a top-level `WHERE`, a stem -whose coordinate fails the filter falls back to `aggregate_raw` (`types.rs:1047-1055`), -which is absent when no raw data lies at the target key. So: +**A formula cell is re-evaluated when one of its inputs moved:** +- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`; +- for an `Agg` with no filter, the group at the stem itself; +- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat` + set to `Item`, which is another stem. This is the one join: stems indexed by "stem + minus `Cat`", joined with the aggregate's output delta. -- **a formula counts at a stem only where its top-level `WHERE` (if any) matches the - stem's coordinate, or where raw data exists at stem + target.** +The affected `(stem, formula)` pairs are ordered topologically over the formula graph and +handed to the existing evaluator. -This is a coordinate predicate plus one count lookup, never an evaluation. It is the -rule ADR 008 clause 3's occupancy uses for formula cells, and it keeps today's pruning. -The review of ADR 008 found the case that needs it: `Double = Revenue * 2 WHERE -Region = East` with data only at West is pruned today, and a rule that counted every -stem would have kept the column. +**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline: +1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it + retracts the record's previous snapshot and inserts the new one. The previous snapshot + is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. + So the stage reads the store and keeps no second copy of the values. +2. **A lifted map** to `(context key, order key, expr at record)`. The context key + removes the target and None categories, and at a fold or total the spanned categories, + mapping a collapsed member to its group. The order key is (context rank, the minimum + store sequence over the record's live cells), which is exactly the order eager + concatenates in. +3. **An inner `WHERE`** is a filter plus the join above. +4. **The group aggregate**, with an error channel kept beside it that reads the first + `Err` in order-key order, as eager does. -**A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula -`T` at stem `s` are: +No cross product is ever built. The map is sound only when `expr` reads the record's own +cells. Other formulas take a fallback that evaluates over the **occupied contexts** only, +the contexts some record projects to. Because `agg_values` evaluates only at each +context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one +edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over +zero occupied contexts it yields the empty value instead. Formulas targeting a category +other than `_Measure` stay on the eager path until their order key is defined. Open +question 8 covers both. -- for each `Ref(R)`, the group `s.with(cat(R), R)`. `cat(R)` is resolved once per - formula set through `find_item_category`, including its `_Measure` fallback for - formula targets (`types.rs:725-740`). Whether `R` is derived (`formula_for`) is also - decided per formula set, so a ref to another formula becomes an edge between formula - nodes, and a ref to raw data becomes an edge from an aggregate group. -- for an `Agg` with no filter, the group at `s` itself, because `eval_agg` pools over - `context.without(target_category)`. -- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at - `s.with(Cat, Item)`. That is a different stem, so a delta at `Cat = Item` reaches - every stem that differs only in `Cat`. **This is the one join**: stems indexed by - "stem minus `Cat`", joined with the aggregate's output delta. +### 6. Circular references -The affected `(stem, formula)` pairs are consolidated, ordered topologically over the -**formula graph**, and handed to the existing evaluator. Its cached path already reads -refs through `formula_cache` and falls back to raw aggregation (`types.rs:1066-1086`). -With the trait, both reads go to the maintained state. +DBSP's recursion computes a least fixed point, and improvise's circularity is not one: +`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs +forever. The required answer is `Error("circular")`. The first version therefore decides +circularity outside the crate: +- the formula graph is rebuilt when the formula set changes, and its strongly connected + components are found; +- formulas outside every cycle are evaluated once per step, in topological order; +- a formula in a cycle has its affected cells evaluated by today's bounded fixed point, + restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still + convergent where `x` is false; +- the sixteen-pass bound is reproduced, so the two backends agree in the differential + test. -**A formula's own `WHERE`** is decided per cell, as today. A cell whose coordinate does -not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own -group and nothing else. - -**An aggregate over an expression** is the one place a formula needs an expression -*inside* an operator. It has its own clause, 5a. - -### 5a. Aggregate over an expression: a lifted map feeding the group aggregate - -`SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per -record**, `Revenue * Price` at that record, and pools those values across the records -of a **context**. It is the one place a formula needs an expression *inside* an -operator. This clause first states exactly what today's evaluator computes, because the -incremental form must compute the same thing. - -**What eager computes** (`eval_agg`, `types.rs:55-87`, and `agg_values`, `:89-139`): - -- **Contexts.** The formula cell's key is the context at a stem. At a fold or a column - total, the contexts are the members `contexts_of` lists through `Expansion::members` - (`types.rs:201-213`, `:905-916`), the cross product of the spanned categories' covered - items. That is the one O(grid) path in totals (ADR 008 clause 5). -- **An inner `WHERE` rewrites the context.** A filter `Cat = Item` *replaces* the - context's `Cat` coordinate (`c.with(&f.category, &f.item)`, `types.rs:69`, and again at - `:116`), and the distinct rewritten contexts are pooled once each. So the stem - `Region=West`, evaluating `SUM(x WHERE Region=East)`, pools **East's** records. -- **Pooling.** For each context, `partial = context.without(target_category)` - (`types.rs:97`). The context's member records are those `member_records` finds under - `partial`, in the store's record order (a record sits where its first cell sits). - The expression is evaluated at each, and the per-context value lists are concatenated - **in context order** (`types.rs:73-77`). So the order is (context rank, record store - order): expansion-major at a fold or total, and plain store order at a stem. -- **Errors.** `agg_values` collects `Result`s (`types.rs:126-136`), and so does - `eval_agg` (`:76`). So the aggregate's value is the **first** `Err` in that order, - if any member errs. - -**The design.** Four stages, each an operator from clause 1: - -1. **An integrated per-record stage.** A cell delta does not carry a record: when only - `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged - `Revenue`, so the old mapped value cannot be computed from the delta alone. So the - first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each - live record's cells, as `ZSet<(measure, value)>` per record, **each cell with its - store sequence** (below). On each step it emits, for every record the consolidated - delta touches, including a cell the expression never reads: - - a retraction of the record's **previous** snapshot, read from the integrated state - before the step (`I(x)₋₁`); - - an insertion of its **new** snapshot. - - **Several writes to one record in one effect** consolidate before this stage (clause - 3 steps once per model effect), so the record is retracted once and inserted once, - at its net state. Writes in separate effects are separate steps. A record whose - snapshot is unchanged after consolidation emits nothing. -2. **A lifted map over record snapshots.** For each snapshot, the map produces one - entry per context the record belongs to: `(context key, order key, expr at record)`. - The value is a `Result`, so an error is a mapped value like any other. `map` is linear, - so a retracted snapshot retracts exactly the entries it once produced. - - **The context key** is the record projected onto the context's shape: the target - category removed and each `none_cats` category removed, per `none_cats` list, - exactly as clause 4 defines "the group". At a fold or total, a collapsed group's - member is also mapped to its group, and each spanned category is removed. That is - the same projection occupancy uses (ADR 008 clause 3). - - **The order key** is (context rank, record store order). The context rank is the - record's position in `Expansion::members` order, and it is 0 at a stem. - - **The record store order is the minimum store sequence over the record's *live* - cells**, recomputed every time the record is touched, and carried in the snapshot. - Eager orders records by their **current** earliest live cell in `DataStore::cells` - (`member_records`), and `shift_remove` changes that when a record's leading cell - goes. So the stage keeps a **per-cell** sequence, assigned when a key is first - inserted: - - **the sequence is assigned where `IndexMap` assigns position, at `DataStore`'s - own mutation sites**, and each cell-delta element carries it. An overwrite - (`set` on a live key) keeps the key's sequence, as `IndexMap::insert` does. A - `remove` followed by a `set` of the same key moves the key to the end, even - within one effect, and gets a fresh sequence. Because the element carries the - sequence, the two cases stay distinct after consolidation: an overwrite is - `−(k, v₀, s) + (k, v₁, s)`, and a remove-then-set is - `−(k, v₀, s) + (k, v₁, s′)`. A stage that inferred the sequence from a - consolidated `(key, value)` delta could not tell them apart. The differential - test's generator includes a same-key remove-then-set within one effect; - - the record tier (clause 3) projects the delta to `(key, value)` before it - consolidates, so moving a cell without changing its value still bumps nothing, - as today's fingerprint does; - - losing a record's leading cell changes its minimum, so the record is re-emitted - with its new order key. - - `IndexMap`'s relative order is exactly this sequence order among live cells. - `DataStore::sort_by_key` (`cell.rs:269`, on entry into records mode) reorders the - store, so it reassigns the sequences, and as an operator-shape change it rebuilds - this stage. - - The review demonstrated why a per-*record* sequence is not enough, against eager at - `57d385e1`. With `S = SUM(Rev * 1)` and store order `A.Other=5, B.Rev=1, - C.Rev=1e16, A.Rev=1, D.Rev=-1e16`, eager reads 2.0. Removing `A.Other`, a cell the - expression never reads, moves A behind C, and eager reads 0.0. A sequence fixed - when A first appeared would still read 2.0. -3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records - with `Cat = Item` pass the filter, and are indexed by their context key **without** - `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of - clause 5 (stems indexed by "stem minus `Cat`"). A filter alone would put East's - records in East's group and leave West's read empty; the join is what makes West's - stem read East's pool, as the rewrite does today. -4. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the - mapped `Ok` values, with `Avg` derived. **The error channel** is kept beside it: per - group, the set of erring entries, ordered by order key. The aggregate reads the - first of them if the set is non-empty, reproducing today's first-`Err`. - -Member records are never listed and no cross product is built. A context in today's -cross product that holds no records contributed nothing to the pool, so pooling only -the records that exist gives the same answer. - -**This is not compiling formulas into operators.** The operators are the record stage, -the map, the index and the aggregate. The expression inside the map is still evaluated -by `eval_expr` at the record, exactly as `agg_values` calls it today through `eval_at` -(`types.rs:127-134`). The circuit decides *which* records to re-evaluate (those the -delta names), and the one evaluator computes each value. Clause 2's rule stands: there -is one evaluator. - -**The boundary.** The map is sound only when `expr` reads the record's own cells, so -that its value at a record changes only when that record changes. An inner expression -that reads another record, holds a nested aggregate, or refers to a formula whose value -depends on other records, is not a function of one record. **Revised on 2026-10-05:** -such a formula is not maintained, and at a fold or a total it does **not** enumerate -today's eager contexts. Those are `Expansion::members`, the cross product (ADR 008 clause -5, a finding of Copilot's on #308). At a stem, eager pools one context and there is no -product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's -**store-driven fallback**, which evaluates the expression only over the **occupied -contexts**, the contexts some record projects to. The user accepted that recommendation, -as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as -eager's, with one edge.** `agg_values` evaluates the expression only at the -`member_records` of each context (`types.rs:47-53`, `:91-139`), so a context with no -record contributes no value, as the paragraph above says of the pipeline. **The edge:** -an inner bare `Expr::Ref` that names no item errors `ref:` at the top of -`agg_values`, for every context it is called with, records or not. Evaluated over -**zero** occupied contexts the call never happens, and the aggregate yields its empty -value instead of the error. ADR 008 revision 25 found this edge, and it is question 15's -whole difference. With at least one occupied context the error is the same. *The first -revision of 2026-10-05 said the fallback differs "wherever the expression is non-empty at -an empty context", and a later one said it never differs. Both were wrong.* Whether this -ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: -- one changes **one operand** of a two-operand expression (`Price` alone, under - `SUM(Revenue * Price)`); -- one **deletes a record's leading cell**, including a cell the expression never reads - (the `A.Other` case above), and checks both the value and which error is reported - first. - -**5a covers formulas whose target category is `_Measure`.** For a formula that targets -another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A record, -whose key `record_of` takes without `_Measure`, then belongs to the context only through -its cell under that measure, and eager orders it by *that* cell's position, not by the -record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. **At a fold -or a total that eager path is `Expansion::members`, so while it lasts it blocks ADR 008's -retirement of `Expansion::members` (its migration step 6) for these formulas.** Sending them to the -occupied-contexts fallback above as well is question 18. - -**What stays eager until step 5, and what is bit-equal before step 6.** -- **Before step 5** (formulas), every aggregate over an expression is evaluated as - today: through `Expansion::members` at a fold or a total, and over the one context at a - stem. -- **From step 5**, the record stage, the map and the index are maintained, so the cross - product is gone. `Count`, `Min` and `Max` over the mapped values, and the error - channel, are read from the maintained state. They are order-free, or ordered by the - order key, and bit-exact on f64 (clause 8). -- **`Sum` and `Avg` over the mapped values are re-folded** from the group's maintained - values in **order-key order** until step 6. That is (context rank, record store - order), which is exactly the order eager concatenates in, so the re-fold is bit-equal - to eager at a stem, a fold and a total. It costs O(group log group) per change, - against O(grid). -- **From step 6** (on question 3's answer), `Sum` is maintained like any other. Under - question 3's recommendation (c) the order key no longer matters for `Sum`, because - the exact accumulator is order-free. - -### 6. Circular references: DBSP's recursion is not our semantics - -**The user, on questions 4 and 5:** the sixteen-pass bound is kept "for testing purposes -but eventually remove it because DBSP should fix the cycle issue", and the component -fixed point "For validation, but if there's a more natural DBSP-native way we should plan -on that." So what follows is the **current** plan, kept for testing and validation, and a -DBSP-native treatment of cycles is to be planned. The argument below, that DBSP's least -fixed point is not improvise's circular semantics, is the author's. It is the open part, -question 25. - -DBSP handles recursion by computing a **least fixed point**: nested streams iterate a -monotone operator, and `distinct` keeps the lattice finite, so reachability converges. -improvise has no such guarantee: - -- `A = B + 1` with `B = A + 1` has no fixed point at all; -- `X = X + 1` over a stored `X = 10` climbs forever; -- the required answer is `Error("circular")`, not a limit or a non-terminating step. - -So the crate has no recursion operator, and circularity is decided *outside* it: - -- **The formula graph** (target → the targets its refs name) is rebuilt when the formula - set changes, and its strongly connected components are found (Tarjan). Formulas - outside every cycle are evaluated in topological order, each exactly once per step. -- **A formula in a cycle, a self-loop included, has its affected cells evaluated by - today's bounded fixed point, restricted to that component.** A formula-level cycle is - not always circular at every cell. `A = IF(x, B, 1)` with `B = A` converges wherever - `x` is false. The per-cell fixed point is the only thing that gets those cells right, - and reusing it keeps `Error("circular")` exactly as it is. -- **The depth bound is part of the semantics today.** An acyclic chain that needs all - sixteen passes is reported circular by both eager evaluators (core.md §3). A - topological evaluator would compute it correctly, and so would disagree with eager. - Question 4 asks whether to reproduce the bound. The recommendation is to reproduce - it, by carrying each formula's topological depth, so the differential test compares - like with like. Changing the rule would be its own decision, for both backends at - once. +This serves testing and validation. It is to be replaced by a DBSP-native treatment of +cycles, whose shape is open question 14. ### 7. The circuit follows the document per model effect -Some changes alter what the circuit computes, not only its input. They are observed -**inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same -position as the frame and dirty observations (`app.rs:1036-1084`). They are not the -post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). -That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 -states the same rule. +**Operator state is selected by a shape key.** The key is exactly what changes which +cells feed which operator: +- the categories of each axis, by level; +- the None categories; +- the filter keys: each filter category with its selected item, and the hidden items of + every category on None, Page or Filter. Those items drop cells from the fold; +- the page **categories**, which key a slice per page (open question 1); +- `measure_agg`, which is model state, never taken from a request. -- **A shape key, and read-time parameters (revised on 2026-10-05).** Operator state is - selected by the **shape key** of a `project` request, not by the name of a view - (question 14). The key is exactly what changes which cells feed which operator: - - the categories of each axis, by level; - - the None categories; - - the filter keys: each `Axis::Filter` category with its selected item, and the hidden - items of every category on None, Page or Filter, as ADR 008 clause 3 also says. Such - an item drops cells from the fold, so it filters. A hidden item of a **page** - category is a filter key, not a read-time parameter; - - the page **categories**, which key the slices under question 7's recommendation; - - `measure_agg`. +**Everything else is a read-time parameter**, applied when output is read: the hidden +items of an axis category, collapsed groups and group membership, the drawn order of +items, the page selection, and prune. A request carries its own key and parameters from +its client's UI state, and a saved view only supplies defaults. - Everything else a request carries is a **read-time parameter**, applied when the - output is read, without touching occupancy or cell operators: - - the hidden items of an axis category. For the column totals, the hidden row-axis - set is subtracted at read time (see "Hiding a row moves the column totals" below); - - the collapsed groups, and the group membership of axis items; - - the drawn order of items; - - the page **selection**, which picks a slice; - - prune, which under client-side pruning is not the host's at all (clause 12). +**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn +items, the collapsed groups, the page selection and `prune_empty`, all of which are +read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit +neither reads nor depends on that tier, which is a view fact. Only `document_generation` +and the record tier matter to this ADR. - **`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72` at `50e1314d`) - hashes each category's drawn items in order (hidden items excluded), its collapsed - groups, its page selection and `prune_empty`. Those are all read-time parameters except - the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, - `AddItem`, reorder and page step, which is what item grain and question 7 exist to - avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40). That tier - is a view fact by the user's Q39 answer, and the circuit does not depend on it. +**Item grain.** Occupancy and per-column aggregates are keyed by the **items** of the +axis categories, never by a position. So collapsing, expanding, hiding, showing, +reordering and adding an axis item change no operator: the visible axis is a remap. +ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a +rebuild. A collapsed group's value is combined from its members when read. That is exact +for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it +costs O(members) per read (not measured). The exceptions are column totals (below). - **Where the key and the parameters come from (revised for ADR 008's 2026-10-06 - answers).** Hiding, collapsing, the page selection, axis assignment and pruning are - ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number - format is "global" (Q49). So a request carries its own key and parameters, taken from - that client's UI state. **`measure_agg` is the exception:** it is model state - (`Model::measure_agg`, `types.rs:370` at `50e1314d`), part of every shape key but read - from the model, never taken from a request (ADR 008 Q51 says the same). A saved - `.improv` view only supplies the **defaults** a client - starts from, and the circuit never stores a client's unsaved settings. Number format is - display-only (gotcha 8) and never reaches the circuit, except through clause 13's - widths if those were adopted. Whether every read-time parameter must travel in the - request (the user's Q51: "do the others need to actually travel across the wire?") is - question 24. A hide and a collapse change model-side aggregates (the totals, and a - collapsed group's fold), so the author reads them as needing to travel. - Requests with the same key share one set of operators, whichever clients sent them. The - live keys are a **cache**: a key enters when a request names it, and it is evicted once - no recent request has named it **and no live topic subscription pins it**. Every - eviction is announced in the next notice header (clause 12). Elsewhere in this ADR, "a shape" means a - shape key. +**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job +runner (8 ms per normal operation). Deltas that arrive during the build are buffered and +added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is +O(delta) and is never sliced. Each step's occupancy output carries the count of tuples +whose count crossed zero, so ADR 008's ordered index can choose between inserting and +rebuilding. Measurements against the budget are stated as ratios against an unrelated +operation on the same run, not as bare numbers. - *As first written* (superseded twice): the operator shape was `view_shape` plus group - membership and `measure_agg`, and a view whose shape moved had its operators rebuilt - **before the frame observation that follows this effect**. The first revision of - 2026-10-05 still listed hidden items and collapsed groups in the shape, and still - deferred to `view_shape` for a saved view. That contradicted the item-grain rule below - and question 7, as the review of `3698c680` found. -- **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column - aggregates are keyed by the **items** of the axis categories, together with the filter - and page keys. They are never keyed by a position in a level sequence. So a collapse, - expand, hide or show of an **axis** category's items changes no occupancy or cell - operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item changes them, and they subtract - the hidden rows when read (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms - for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or - reordering items, changes no operator either: a new item tuple is just a new key, and - the order is a read-time parameter. Only a change of the **shape key** selects other - operators, which a new key builds as a job: - - a change of which categories are its levels or on None; - - a change of its filter keys, which includes hiding an item of a category on None, - Page or Filter; - - a change of its page categories; - - a change of `measure_agg`. +**Formula edits are an input.** After each model effect the host compares a fingerprint +of the formula set. If it moved, the formula delta is fed to the circuit, and the formula +graph is re-derived. A document swap (`install_workbook`, which bumps +`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and +no formula, so it touches no operator. - A collapsed group's value is a pooled fold over its members. With item grain it is - combined from the members' state when it is read. For `Count` that combine is exact. - For `Sum` it is exact only under question 3's (c), or under ADR 005's wide accumulator. - A sum of the members' f64 sums is not bit-equal to eager's one store-order fold - (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that - fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default, (A); question 23).** Today a - column total pools only the **drawn** rows. The total's evaluator excludes hidden items - through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` - evaluates over that memo (`:1066-1070`). The test - `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take - `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide - Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, - and no read-time parameter can take Feb back out of it. **Whose hide this is** (ADR 008, - Q46 to Q50): a hide is one client's ephemeral UI state until an explicit `SaveView`, - and never a model mutation. So the options below concern a request's **unsaved** hidden - set. A saved view's hidden set arrives only through `SaveView`, as part of the saved - definition a request starts from, and it is subtracted the same way. So: - - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** - item rebuilds the column totals of that shape, as a job. ADR 008 once said the same - (its finding 10). It no longer does: its clause 5, at revision 28, says totals are - **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's - question 23. Occupancy and cells stay read-time remaps. Collapse and - expand do not move a total, because it pools the members either way, which is what - the test above pins. A hide or show of a column-axis item hides or shows that - column's total. Under a **collapsed** column group it also changes the group column's - cells and total, because the fold memo skips hidden items on every category. That is a - read-time recombination of the members' item-grain state, and nothing is rebuilt. - - **(A):** keep the per-(column tuple, measure) total state over **all** rows, and - subtract the hidden rows when the total is read. Clause 4 keeps, per group (a row - tuple and a column tuple at item grain), a count of values, a count of numbers, a - running sum and, where `measure_agg` or text needs it, a multiset of values with a - count per key. The column's state is the sum of its groups' states. So at read time - the H cells of the hidden rows in that column are taken out as a **temporary - overlay**, never written back: - - `Count`, and the two counts behind the numeric-first branch: subtract. O(H). - - `Min` and `Max`: walk from the extreme key, skipping each key whose whole count is - hidden. At most H keys are skipped, so O(H log d), with d the distinct keys. - - The one-distinct-value text rule, and `mixed`: count the keys whose whole count is - hidden, and subtract that from the column's number of distinct keys. O(H log d). - Eager's own reading rules (NaN under `PartialEq`, clause 4) apply after the overlay, - unchanged. - - `Sum`: subtract the hidden rows' sums. That is exact only under question 3's (c), - or ADR 005's wide accumulator, because f64 subtraction is not exact. Before step 6 - a `Sum` total stays on today's path anyway. `Avg` is `Sum / Count`. +**The shape cache.** Live shape keys are a cache. A key enters when a request names it, +and it is evicted when no recent request names it (open question 4). Every eviction is +announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm +by every client reading it. Idle readers make that likely, since a shape that clients +only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job +queue. - So (A) costs **O(H log d) per read for every aggregate**. It is exact, and it keeps no - per-set state. *As first written* (superseded), this item said `Min`, `Max` and the - text rule cost O(rows) per read. That overstated it, as the review of `ad60ad2b` - found. The argument above is the author's and is not measured. +**Column totals subtract hidden rows at read time.** Today a column total pools only the +drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`), +pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a +per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept +over all rows, and a read subtracts the request's H hidden cells in that column as a +temporary overlay: +- `Count` subtracts; +- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden; +- the text rule counts the distinct keys left; +- `Sum` subtracts through the exact accumulator. - **(B) has a defect: the key is shared and the hide is not.** Operator state is shared - by every request with the same shape key (above), and an unsaved hide is a parameter of - one client's request. If client X hides Feb and client Y, on the same key, does not, - (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: - - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and - cells stay shared under the shape key. Totals are kept per (shape key, hidden - row-axis set). Each distinct hidden set in use costs at most one O(cells) rebuild, as - a job, and one set of per-column aggregates. The bound is an upper one: an entry can - be derived from the all-rows state by (A)'s subtraction, made once and kept, at O(H) - per column, with `Sum` exact only under question 3's (c). So (B′) is (A) memoised. - The totals cache multiplies by the number of distinct hidden sets that recent - requests name, and it is evicted like any key (question 14). - - **(A)**, as above, needs no per-set state at all. +That is O(H log d) per read for every aggregate, with no per-client state. The shape key +is shared across clients and a hide is one client's, so a total cannot be rebuilt per +hide. A hidden column item under a collapsed column group changes that group column's +cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression +totals need their entries indexed by row tuple as well, so a read costs O(hidden +records). The alternative is open question 12. - **Aggregates over an expression** (clause 5a) follow the same split. A 5a total's - entries are keyed by context, with the row categories removed. To subtract hidden rows - under (A), the 5a stage's entries must also be indexed by row tuple, at O(entries) more - state, so a read costs O(hidden records). Under (B′), a 5a total per hidden set is its - own map and aggregate stage, at O(records) to build. +### 8. Floating point: exactness comes from the representation - *The author's lean, re-evaluated after the corrected cost:* **(A)**. A read costs - O(H log d), and a hidden set is usually a handful of rows, so a read costs about as - much as (B′)'s O(1) in practice (not measured). (A) keeps no per-set state, which is - what C2's "as stateless as possible" asks, and it never rebuilds or multiplies the - cache. (B′) is kept as a **memo** on top of (A), for a hidden set so large that H log - d per read would not fit the frame budget. The earlier lean to (B′) rested on the - overstated O(rows) cost, and it is **superseded**. Plain (B) without the hidden set in - the key stays **withdrawn**. The user decides it in question 23. -- **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it - cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a - 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). - So a rebuild is a resumable `step(deadline)` over an owned snapshot of its input. The - deltas that arrive meanwhile are buffered and added at install, which Z-set addition - makes exact. A circuit `step` on a cell delta is O(delta), and it is never sliced. - Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the - install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and - priority queue the user required. This ADR only makes its rebuilds steppable and - cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user - ruled. Any measurement this ADR later reports against them is stated as a ratio - against an unrelated operation on the same run, as ADR 008's question 13 requires. -- **`SaveView`, and the default view (Q46 to Q48).** `SaveView` is a model command that - writes a view definition, acknowledged by the model, and a document always has at - least a default view. **Neither touches the circuit.** A saved definition moves no - cell and no formula, so it emits no cell delta and no formula delta. Shape keys come - from requests, so saving builds no operator. The next request that names the saved - view simply starts from its defaults. The acknowledgement is ADR 008's - (`ServerFrame::Accepted`, its question 69), not this ADR's. -- **A crossing count (ADR 008's second request).** Each step's occupancy output carries - the number of tuples whose count crossed zero. ADR 008's ordered index uses it to - choose between an insertion and a rebuild by sorting. -- **The formula set is an input, not a rebuild.** After each model effect the host - compares a fingerprint of the formula set, over (`raw`, `target`, - `target_category`), with its previous read. If it moved, the formula delta (−the old - formulas, +the new ones) is fed to the circuit as an explicit input. The formula - layer then re-derives the formula graph (clause 6) and re-evaluates the affected - targets at their stems. This is observed, not declared by effect type, as the tiers - are. It is what catches a `ReplaceFormula` that keeps its target, which moves no - `view_shape`. -- **A document swap rebuilds every circuit** from the new store. `install_workbook` - bumps `document_generation`, and the same after-each-effect read sees it. +A sum maintained by `+v` and `−v` in f64 drifts from the eager fold. +`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max, +with values of up to five decimal places: -Re-reading the shape keys costs O(categories) per live key per model effect, and the -formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a -new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by -`layout_version` after the loop, as today, and the circuit reads neither. +| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact | +|---|---|---|---| +| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 | +| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 | +| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 | -**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first -written* (superseded by item grain): `AddItem` moved the operator shape, so the view was -rebuilt with East in its sequences before `SetCell`'s delta arrived. **Under item grain, -`AddItem` moves no operator.** `SetCell`'s delta inserts the item tuple `East` as a new -key, and only the visible axis, which is derived, gains a row. The case needs no -ordering rule at all. +Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively, +against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager. +At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer +differently on the two paths. Eager itself is already more than 1e-10 from the exact +decimal sum. -**Recommended: a page selection is not a rebuild (question 7).** Under the recommended -answer, the shape key includes the page categories and not the page selection, so `[` and `]` read another slice -of state that already exists. The held-key path stays O(visible) (improvise-748f's -constraint). The slices cost nothing extra, because the state is still bounded by the -stored cells, however many pages there are. Filter coordinates still filter; only page -coordinates key the slice. If the user answers the other way, a page change is an -operator-shape change like any other. +**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer +wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so +that those can be retracted. Adding and retracting are integer additions, so the +accumulator is a true abelian group, and it rounds once, on read. A weight is split into +base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example: +- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles; +- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again + with weights up to 2⁴⁰; +- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to + 1.1 µs per read; +- it is 568 bytes per group, against 8 for an f64. -### 8. Floating point: retraction drifts, and exactness comes from the representation +`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0, +where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they +become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and +the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum` +0.1.6 rounds about a quarter of single negative values wrongly and cannot retract +non-finite values. -f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum -the eager fold computes. **Measured**, not asserted: -`cargo run --release -p improvise-core --example retraction_drift` -(`crates/improvise-core/examples/retraction_drift.rs`, added by this PR). It was run on -an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N -retract-and-insert cycles. It compares the sum against the eager fold, -`evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count -of 10⁻⁵ units rounded to f64 once. Each value is a signed whole number of 10⁻⁵ units: -up to 99,999 units, times a power of ten spread over 3 or 9 decades. So the smallest -values carry five decimal places, not the two a user usually types. Two-place input -would absorb differently, and it was **not** measured. +**Lossy operations happen at the edge.** Sums stay in the accumulator. Division, +non-integer `POW` and display rounding happen only where a value leaves the exact domain. +An exact value becomes an f64 through its decimal string, which is correctly rounded. -| store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | -|---|---|---|---|---| -| 1,000 cells | 3 decades | 6.4e-11 – 1.3e-10 (35–140 ulp) | in 1 of 2 seeds | ≤ 7.3e-12 | -| 18,000 cells | 3 decades | 1.7e-9 – 2.3e-9 (57–80 ulp) | yes | 1.5e-10 – 2.6e-10 | -| 18,000 cells | 9 decades | 2.3e-5 – 3.2e-4 (3–42 ulp) | yes | 2.4e-4 – 2.5e-4 | +**The accumulator does not make `=` against a typed decimal exact.** It sums the stored, +already-rounded f64s, so it can miss the typed total by up to about half an ulp of each +term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example: +`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's +exact decimals fix that, and they also cover per-cell arithmetic and typed values such as +money. -A fourth case holds 1e17, then adds 1,000 cells of 0.01, then retracts the 1e17. The -naive running sum reads **0.00**, and the eager fold and the exact sum both read -**10.00** (at `,.2`). The small values were absorbed and never came back. +### 9. What ADR 008 gets -A Neumaier-compensated running sum recovers that case exactly. In the random cases it -stays about as close to the exact sum as eager does, **but it is never bit-equal to -eager**, because eager folds in store order and is itself inexact. +- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple. + Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. + A fold row with a formula column but no stored cell under it, and the `Exact` case with + nothing on None, are **not** occupied, on purpose. +- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit + consumes it and never keeps a second order. +- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item + grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because + the host cannot compute a client's visible axis. +- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the + formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows). +- **Widths for text columns** (clause 13). -Four conclusions follow about a **floating-point** running sum: +### 10. Live sources -- **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by - any f64 running-sum technique, naive or compensated. So the differential test can - compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit, because they are - order-free, but it can compare an f64 `Sum` and `Avg` only with a tolerance. -- **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum - differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So - `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour - difference, not a display one. -- **This is already a bug in eager, which the measurement found in passing.** With 3 - decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal - sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed - can already be wrong today. That is recorded here for improvise-l2b8. -- **Only an exact representation dissolves the problem.** The next two subsections give - two: an exact accumulator for f64 sums, and ADR 005's decimals. - -#### Does DBSP itself help with floating point? No. - -The user asked whether rebuilding the evaluator on DBSP brings any correctness benefit -for floating point. On its own, **none**: - -- **Only the Z-set weights are exact.** They are integers (clause 1). The *values* being - summed are whatever type the aggregator holds. -- **An f64 `Sum` maintained by retraction is worse than eager, not better.** The table - above is the evidence: it drifts further from the exact sum the longer it runs, and - the absorption case loses the answer entirely. - -**The benefits come from two other places.** -- **The representation.** An exact accumulator (next subsection), or ADR 005's - decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian - group, provided both backends use the same accumulator. Under ADR 005 as merged that - holds **per partition**: the 320-bit decimal partition and each fraction group. A read - that combines more than K = 8 distinct non-decimal denominators is inexact by ADR 005's - cutoff. Decimals alone do not: ADR - 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at - its 28-digit bound, so the order of summation changes both the value and the - exactness. Then retraction is exact, incremental - equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent - `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes - with it. -- **The differential test** between `Eager` and `Incremental` (clause 2) is itself a - correctness gain. Two independent evaluations of every formula cell catch evaluator - bugs in either one, which nothing in the tree does today. - -What DBSP does contribute is **placement**. Every aggregate is a named operator with a -chosen algebra, so the circuit states exactly where values enter an exact domain and -where they leave it. That is the next subsection's principle. - -#### An exact accumulator for f64 sums, and deferring the lossy operations - -**A superaccumulator sums f64 values exactly.** It is a fixed-point integer wide enough -for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit integer shifted -by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's -xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. -Adding or retracting a value is integer addition, so the accumulator is a true abelian -group over the finite f64s. **Rounding happens once, when the value is read.** - -**The group claim needs one stated bound.** A consolidated Z-set carries weights, not -just ±1: equal mapped values consolidate, so one element can arrive with a large weight. -The accumulator therefore splits any weight into chunks of at most 2³⁰, and counts -*magnitude* between carry propagations rather than calls. Each limb then stays below -2⁶² between carries, for any `i64` weight. The first version counted calls, and review -found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is -`1.82e-304`. The example now checks that case, and it reads the exact answer. - -The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight -near `i64::MAX`. The version in `numtower` splits the weight into base-2³⁰ -digits instead and adds each as a shifted value. That is exact in the fixed-point -representation and O(1) per add. In practice a weight is bounded by a record count -anyway. Then: - -- `Sum` is the exact sum, rounded once. It is independent of order. -- `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded - if the division is done on the exact value; dividing the rounded sum rounds twice, - and step 6 must say which it does. -- `Min` and `Max` are already exact. -- Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. - -**Measured**, in the same example, by `ExactSum` (158 lines, including its rounding and the weight bound): - -- **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` - maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over - the current values. That includes the absorption case, which reads exactly 10.00. - Against the exact *decimal* sum, it differs at 1 checkpoint of 25, by 9.1e-13. That - gap is the rounding of the typed decimals into f64, not the sum. -- **It is correct.** It was checked (not committed) against Python's `math.fsum` on - 20,000 random cases. The cases include subnormals, values near the overflow bound, - and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the - 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. After - the weight bound, it was checked again, on 20,000 cases with weights up to 2⁴⁰ - against exact `Fraction` sums: 0 mismatches. The review's own oracle run agreed. -- **Cost, Apple M4 Max, best of 3 over 10M values.** The example prints per single - update. - - This revision's three runs agreed: 2.0 ns per insert or retract, against 1.5 ns - for a plain f64 add in the same black-boxed loop, and 0.58 µs per read. - - The previous revision's run on the same machine gave 3.2 against 2.3 ns and 1.1 µs, - and the review's run gave 3.6 against 2.7 ns. - - So an update costs roughly 1.3× a plain add. A read, which is the rounding paid - once per group read, costs 0.6–1.1 µs. - - **568 bytes per group**, against 8 for an f64. - - The ranges below are between runs on one kind of machine; read your own run. - - The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB - of accumulators. A sparse form that stores only the occupied span of limbs would be a - few limbs for typical data. **That is an estimate, not measured.** - -**No existing crate was usable.** `xsum` 0.1.6 (MIT, no dependencies, and it builds for -`wasm32-unknown-unknown`) was checked and rejected on three counts: - -- **It is wrong on a single negative value.** It rounds about a quarter of single - negative inputs to a neighbouring f64: 49,819 of 200,000 sampled values, all - negative, for example `-5890.88` read back as `-5890.880000000001`. -- It keeps one infinity and one NaN flag, not counts, so a non-finite value cannot be - retracted. -- Its accumulator is not `Clone`, and reading it takes `&mut self`. - -`accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not -evaluated. So the accumulator is about 150 lines of our own, with its own tests and a -differential test against an exact rational sum. **It lives in `numtower`, ADR 005's -numeric crate.** The user decided that on ADR 005 question 15, answering "(a)" on PR -#320 on 2026-09-25. This ADR's seam crate wraps it in a newtype for `zset`'s group -trait, so `zset` stays free of numbers. - -**Defer the lossy operations: the general principle.** Keep intermediate results in a -domain where the operation is exact, and apply the lossy operations at the edge: - -- sums go in the accumulator; -- decimal add, subtract and multiply are exact under ADR 005, within `rust_decimal`'s - 28-digit bound, which ADR 005 detects rather than silently rounds; -- division, `POW` with a non-integer exponent, and rounding for display happen only - where a value leaves the exact domain; -- **an exact value becomes an f64 through its decimal string and `str::parse::`**, - which is correctly rounded. `rust_decimal`'s `Decimal::to_f64` is not: ADR 005 - measured it disagreeing in 17,375 of 200,000 values. `ExactSum`'s own read is - correctly rounded by construction (clause 8's oracle checks). - -This is honest only about where the gain comes from. **The representation earns it, not -DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named -operator with a chosen algebra, so "where does rounding happen" has one answer per -operator rather than one per call site. - -#### What each exact domain covers - -| | exact accumulator | ADR 005 decimals | -|---|---|---| -| `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | -| per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | -| typed values such as money | no | yes | -| a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 as merged (option (d)): a three-level tower, `Decimal` / `Fraction(Box>)` / `Inexact(f64)`, in `numtower`. `rust_decimal` alone was measured at +15.4 KB gzipped, and `num-rational` with `num-bigint` at about +20.8 KB on their own; the combination is unprobed. `numtower` is not yet in the workspace (implementation epic improvise-59nw). Its sum keeps a 320-bit decimal partition and a 256-bit numerator per fraction group, and a read combines at most K = 8 groups (the user's final answer; K = 32 was superseded). The 40 bytes per summed group cover the decimal partition only | - -**The accumulator does not make `=` against a typed decimal right.** It sums the -*stored* f64s exactly, and each stored f64 is already the rounded form of what the user -typed. So the exact sum of the stored values, rounded once, can differ from the typed -total, rounded once, by up to about half an ulp of each term. That is the last bit of -the largest term, which is more than the last bit of the total when the terms cancel. -- **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, - rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not - `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result - missed the typed total in 21%, 16%, 7.5% and 2.5% of 2-, 3-, 10- and 100-term sums, - and in 0 of 8 seeds at 18k and 90k cells. On two-place values uniform in ±1,000.00, it - missed in 38–40% at every one of those term counts. -- **A spot check** of this ADR (not committed), on the ±1,000 values, agreed: about - 37–40% of 2- to 100-term sums missed. -- **This ADR's own example** missed in 1 of 25 large-sum checkpoints. - -So the miss rate depends on **how large the total is relative to its terms**, not on -the term count as such, nor on how large the values are. When the terms share a sign, the total -grows, and its last-bit step grows with it, so the terms' own rounding errors fall -below it and misses become rare. When the terms cancel, the total stays small, and -misses stay near 40%. - -**Where a miss is visible.** A miss is bounded by the **terms'** rounding, not by the -total's. Each stored term is off from its typed value by at most half an ulp of that -term, so the exact sum of the stored terms is off from the typed total by at most about -½·Σ ulp(term), plus the total's own half-ulp at the final rounding. So the misses are -last-bit **relative to the largest term**, not relative to the total. When terms cancel, -a miss can be far larger than the total's ulp. - -- **The large-term, small-total case (from ChatGPT Codex on this PR).** The terms are - `1000000000000000.10 + (-1000000000000000.00)`, and the typed total is 0.10. The - first term is stored as `1e15 + 0.125`, because the ulp at 1e15 is 0.125. So the - exact sum of the stored terms is **0.125**. The miss is 0.025, on a total of 0.1, and - `=` sees it. -- **The condition, stated on the terms.** `=` is an absolute `1e-10` (gotcha 17), so a - miss can reach it only when ½·Σ ulp(term) reaches `1e-10`. That needs at least one - term of about 2¹⁹ ≈ 5.2e5 or larger, or many terms just below it. It does not depend - on how large the total is. -- **The measured cases fit the condition.** On the ±1,000 values every term is far - below 2¹⁹, and the review counted 0 of 7,506, 0 of 7,901 and 0 of 7,963 misses - visible to `=` at 2, 10 and 100 terms. On positive values up to 999,999.99 the terms - themselves pass 2¹⁹, and most misses are visible. -- **Under ADR 005's epsilon-free `=`** (its question 3), every miss is visible, since - there is no tolerance at all. - -The misses show in the editor and to a raw comparison, and not in a display format -unless the terms are large enough, as in the case above. ADR 005's "once the magnitude -is above 2¹⁹" needed the same correction, to the terms' magnitude, and ADR 005 has -made it. Only exact decimal cells (ADR 005) make `=` against a typed total exact at -every magnitude. - -**So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. -The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this -ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting -for ADR 005**, provided `Eager` uses the same accumulator. That was question 3's option -(c), and **the user chose it**: "use the exact accumulator." (4226373188). - -### 9. What ADR 008 gets from it - -ADR 008 (revision 37 at `c8be7720`, PR #308) assumes these, under the shared names below. -**Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. -A consumer reads it through `project` (clause 12), and none of it is addressed to a -host-held mirror of a client. - -- **Occupancy** (ADR 008 clause 3). For each projection shape (clause 7), there is a count - per row tuple and per column tuple of cells that hold a value. The counts are kept at - **item grain**, and the visible axis is a remap of them. - - Stored cells pass the shape's filter. - - A hidden item of an axis category is masked only when the visible axis is derived. - - A collapsed group's members map to the group's tuple at read time. - - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` - holds a value: a top-level `WHERE` must match, or raw data must exist at the target. - - Page coordinates key a slice, subject to question 7. - - A tuple is on the axis exactly when its count is above zero. **ADR 008's question 1 is - answered: "go with occupied"** (#308, comment 4171484686). So the two cases where - today's pruning keeps a row with no stored cell under it are dropped on purpose: a fold - row with a formula column, and `Aggregation::Exact`. Clause 5's stem rule already drops - them, there is no "non-occupancy path" for them, and the differential test carves them - out by name. *As first written*, this item said this ADR "does not settle" that - question; that is superseded. Occupancy is integer and order-free, so it does not wait - for ADR 005. -- **The ordered record list** (ADR 008 clause 1, `Records`). The user: "the ordered list - of records is a model-side concept" (Q27). It is the store's order (gotcha 18, - `DataStore::live_records`), and this ADR does **not** maintain it as a per-shape - operator. Clause 5a's record stage keys records by `record_of` and orders them by the - store sequence, so it *consumes* the store's order and never defines a second one. How - a page and filter select from that list is ADR 008's question 55. It is not this ADR's - to answer, because this ADR keeps no per-view copy of the list. Whether records mode is - `project` over the virtual axes (ADR 008 question 54) changes nothing here: occupancy - over `_Index` and `_Dim` is the same operator as over any other axis. -- **The occupancy delta** (ADR 008 clause 8, which the split revises). `distinct` over - occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** - axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first - written*, its consumer was the per-view frame observation and its reset of each client's - cursor (gotcha 24). Under the split the host holds no cursor, and no client's read-time - parameters, so it cannot compute any client's **visible** axis. The delta is a - model-side output at item grain that keeps question 22's index current. Under question - 13's (b) it is **not pushed**. A client learns of a new or vanished row from the change - notice, recomputes its own visible axis, and shifts its grid so that the cursor keeps - its place (Q34). Only under (c) would a window's delta be pushed. -- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the - pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at - `50e1314d`), - not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. A hidden row leaves the total - and the key is shared across clients, so under question 23's default (A) a total - subtracts the request's hidden rows when it is read, at O(H log d) (clause 7, "Hiding a - row moves the column totals"). Clause 5 replaces `Expansion::members`. Totals - that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: - the exact accumulator (recommended) or ADR 005 (clause 8). -- **What ADR 008 no longer asks.** Its question 6 is answered ("maintained - incrementally"). Its questions 1 and 2 are answered there too: 1 is "go with occupied", - and 2 is (b), sticky per session, with hysteresis as the easing (its question 22). *As - first written*, this item said they "stay open there", which had gone stale. -- **Widths.** A width depends on formatted text, which is display-only (gotcha 8). Under - the user's answer to ADR 008's question 2, widths are taken over the rows realized and - held by the view layer. The user's question whether DBSP can maintain a maximum width is - answered in clause 13. - -### 10. ADR 003's live sources are delta streams, when they can be - -A live source that can report its changes, by polling and diffing, by triggers, or by a -change feed, produces a cell delta. That is the same input `DataStore` produces, and the -circuit cannot tell them apart. That is the "data can stream" half of the user's -prompt. - -**A source that can only answer aggregate queries stays on `Eager` for now,** with ADR -003's pushdown. The user has said "Eager should be deprecated" (question 8), so this is -an interim, and its replacement is question 26. There is a tension: ADR 003 wants data larger than memory, and every -integrated operator here keeps state proportional to its input (clause 4). So the -backend is chosen per document, which is what "pluggable" buys (question 8). A -read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. +A live source that can report its changes produces a cell delta, and the circuit cannot +tell it from `DataStore`'s. A source that can only answer aggregate queries stays on +`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated +state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so +the backend is chosen per document. ### 11. Constraints -- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, - `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, all three join the browser - bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds - them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). -- **Two seconds per test binary.** The crate's laws and its "incremental equals - from-scratch" properties run at the default 256 cases over small inputs. The - differential test lives in the seam crate's own binary, so its cost is visible on its - own. Its document sizes are bounded by the strategy, never by raising or lowering the - case count (design-principles §7). -- **Ordering determinism (gotcha 18).** A `ZSet` iterates in `K`'s `Ord` order, never - in hash order. A hash index is allowed for lookup only. Output deltas are - consolidated and sorted before they leave an operator. Nothing in the circuit decides - a display order: record order stays `DataStore`'s (`live_records`, `cell.rs:366`). - The determinism test builds **two** circuits independently from the same history and - compares them. One circuit compared with itself proves nothing (improvise-hzjo's - vacuity). -- **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:693` at `50e1314d`). **Decided (question 9, the user: "The recommendation is fine, I think"): the backend's state joins the snapshot.** - Applying the inverse delta is **not** a rollback, even with exact aggregators. It - restores integrated values, but `step(d)` then `step(−d)` still advances the circuit - and changes the history that `delay` and `differentiate` hold. So the next step after - a "rewind" can differ from the next step after nothing happened. A real rewind would - have to restore every stateful operator, with a test of observational equivalence - across all of them. This ADR does not propose one. +- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then + `zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm` + CI job, each checked with `cargo tree --edges normal`. +- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over + small inputs, and the differential test lives in the seam crate's own binary. +- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and + sorted, and nothing in the circuit decides a display order. The determinism test builds + **two** circuits independently from one history. +- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that + `Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the + integrated values but not the history `delay` and `differentiate` hold. The cost is a + second O(state) clone per mutating frame, and peak memory to match. The keyboard path + takes no snapshot. +- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence. - **The snapshot costs a second O(cells) clone per mutating frame.** - `Host::mutating_frame` already clones the workbook on every mutating frame - (`host.rs:224-225`, `app.rs:673` at `50e1314d`), and the integrated state is of the same order. So - on the network path a write is not O(delta): it is O(cells) for the snapshot today, - and roughly twice that with the circuit in it. The keyboard path (`apply_effects` - without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the - store-memory requirement (clause 14), that second clone is also peak memory, which a - rewind would avoid. Question 9 stays open, and is now weighed against clause 14 as - well. -- **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the - delta sequence, so replaying commands reproduces its state. +### 12. Delivery: one circuit, in process or across a wire -### 12. Network transparency: one circuit, consumed in process or across a wire +The DBSP machinery is **network-transparent**: the same operators and streams whether a +consumer is in process or across a wire. -**The requirement is the user's** (C1): "The DBSP machinery needs to be implemented in a -network transparent way, but the details of that go to ADR 009". **Everything -else in this clause is this ADR's design, and the author's choices are marked as such. -None of it is measured.** +**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their +change notices, plus the input deltas of a remote live source. The write side carries +model commands with their basis, never deltas. The circuit's internal streams stay on +the host, because aggregation is model-side. No UI state crosses. -**What crosses a wire, and what does not** (the author's reading). Two boundaries can be -a wire. -- **The output boundary** carries the results of `project` and their changes: a window's - cells, its axis occupancy, its totals, and ADR 008 clause 4's width facts. In the TUI - the consumer is in the same process. In the browser it is the page, across `postMessage` - from the worker or a socket from `improvise-ws-server` (gotcha 45). -- **The input boundary of a live source** (clause 10). A source that reports its changes - delivers a cell delta, possibly from another machine. The circuit cannot tell that - delta from one `DataStore` emits. +**A request has a shape key and a window.** The key selects operator state, shared by +every request from any client. The window (absolute top-left and dimensions) selects +part of the output and is never state. So a request costs O(visible). -Three things do not cross: -- **The write side carries no deltas.** A client changes the model only by `apply` of - model commands (the user, Q43), so traffic from a client to the host stays - `ModelCommand`s with their basis (ADR 001), as today. -- **The circuit's internal streams stay on the host.** The cell delta, the per-record - stage and the group aggregates never cross: aggregation is model-side (the user, Q38), - so no operator runs in a client. -- **No UI state crosses.** The host is told the request and nothing about a cursor, a - scroll or a key (Q32, Q33, Q37). +**Positions are absolute cross-product positions**, in each category's own item order, +and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when +occupancy changes. They do move when an item is inserted, removed or reordered, so: +- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute + position attached when sent; +- an item edit is sent to every client as an **axis change** ("inserted at index j of + level L"). A client re-derives its positions at O(levels) each and re-anchors its + top-left to the same tuple; +- resolving positions needs rank and select over each category's items, which is ADR + 008's order-statistic structure. -**What a projection request is to the circuit** (the author's answer to "is the result -keyed by request, by axes, or shared?"). A request has two parts. -- **Its shape key** (clause 7) selects the circuit's state. The key is shared by every - request, from any client, that has it. Its read-time parameters (hidden and collapsed - axis items, drawn order, page selection) are applied to that state's output. -- **Its window**, the top-left and the dimensions, selects a part of that state's output. - The window is never state. +**Pruning's model-side support.** Pruning hides a row or column that is empty within the +displayed values, so a value outside the window does not count. That differs from +today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed +window, one pass that drops the empty rows and columns is already the fixpoint. A fill +then adds rows and columns forward from the absolute top-left, until the requested +dimensions are met or the axes end. Growth is only rightward and downward. For the fill, +the model answers one **restricted successor query**: "the next k rows after absolute +position p with a value in at least one column of S", with S in the call, and the same +for columns. A per-column index of occupied rows answers it: a sorted list per column, +seeked to p and merged forward with a linear minimum over the |S| cursors, at +O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the +cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a +B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's +whole state is its request and its last absolute position, so a resume is a seek. +**Scrolling up or left is a new request with a new top-left.** The new top-left is found +by galloping backwards in doubling jumps until a probe hits, then narrowing until one new +row is gained. Each probe is a restricted successor query bounded by the old top-left. +That is about 14 to 28 probes at density 10⁻⁴, and no special answer is needed when a +top-left lands on an empty row. Which side of the wire runs the prune and the fill is +open question 9. -So maintained results are keyed by **shape**, and the only per-request work is -restricting a shape's output to a window. That is O(visible) per request and needs -nothing kept between requests. +**The wire properties**, which hold on each hop: +- **T1. Plain data.** Output elements are serializable values that name cells by tuples, + positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`. +- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`), + so a value change that only a NaN payload distinguishes cannot be told apart remotely. + Which identity outputs use is open question 5. +- **T3. A publication stamp.** Each accepted batch is published once, stamped + `(host instance, document_generation, publication)`. The publication number increments + once per published batch, so a multi-effect batch is one header, and a refused frame + consumes none. The host instance changes on restart. A client drops anything at or + below the number it has applied. **The refetch rule:** a client that sees a gap, a new + `document_generation` or host instance, its shape in an eviction list, or a notice it + cannot interpret, refetches. +- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative + to its stamp. +- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and + flags, under the latest stamp. Axis changes are positional, so they stay an ordered + list. A `document_generation` change inside the run voids the merge. A basis is never + sent ahead of the cells it describes (gotcha 43). +- **T6. One API.** The TUI calls `project` and receives the same types as a remote + client. A test runs one edit sequence through an in-process consumer and through a + serialize-and-deserialize round trip, and requires equal results. -**Positions are cross-product positions** (C3). A request's top-left is a position in the -**cross product** of its axes, in each category's own item order. That position is a -bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed -contexts, we can represent bigints as strings in JSON"). A cross-product position names -the same tuple whatever is occupied, so **an occupancy change never moves a request's -window**. It changes only which positions inside the window hold data. **Which side of -the wire prunes is open.** C3 says the user is "still ambivalent", and C6 says pruning -"sounds like a UI side concern". That is question 19. The rest of this clause is the -author's analysis, and it is written for **client-side pruning** first, because that is -the author's lean after C6 to C8. Where model-side pruning would differ, it says so. +**The change notice.** After each accepted batch the host emits one notice, addressed to +no client. It carries: +- the stamp; +- the axis changes; +- display relabels (`(id, new display)` under ADR 007); +- a flag when the formula set or `measure_agg` moved; +- the eviction list; +- for each live shape key, the **moved output keys**: cells, formula cells and + column-tuple totals, as the circuit computes them. -**"As absolute as possible", and what still moves a position** (C4). *The author's -analysis.* A cross-product position does not move when occupancy changes, pruning -included. **It does move when an item is inserted, removed or reordered** in a category's -own order. Positions are mixed-radix numbers over the items of each level, so inserting -item j at level L shifts every later item of L, and changes the radix that every outer -position is multiplied by. No position over a changing item list is fully absolute. -Three things follow: -- **Deltas are keyed by identity, and positions are attached when sent.** An output - element names its row and column by tuple identity: item ids under ADR 007, and names - until then (see "Names and ADR 007" in clause 3). The host attaches the absolute - position from the current item order when it sends the element. A delta keyed by - position alone would go stale with the next item insert. -- **An item edit is sent to every consumer as an axis change.** "Item inserted at index - j of level L", or removed, or moved. Under question 13's (b) it rides on the change - notice, which every consumer receives. Under (c) it goes to **every** subscriber of a - shape with - that category on an axis, wherever its window lies, because an insert before a - window's top-left shifts every position that subscriber holds. A client re-derives its - positions arithmetically, at O(levels) per position, and re-anchors its top-left to the - same tuple, as ADR 008 clause 11 anchors the cursor. Resending every row whose position - moved would cost O(rows after j). -- **The cost is rank and select over each category's items.** "The position of item x" - and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` - first, a B-tree when measured, its question 16). A tuple's position is O(levels × log - items). An index ordered by position must compare tuples by **current** item position, - not by a position stored at insert time. With item grain (clause 7) its keys are item - identities, so an item insert re-orders nothing in it, and a comparison goes through - the order-statistic lookup, at O(levels × log items). That is **not measured**. - Question 20. +Moved records alone would not do. From a moved record, a client cannot tell whether a +formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue +WHERE Region=East)` on page `Region=West` moves when an East record is written. A client +intersects the moved keys with its window by identity. A collapsed fold row has no +maintained key, so a moved member key goes through the client's own member-to-group +remap first. The client then refetches what moved, by `project` or by the successor +query. -**Pruning as a filter on the displayed values** (C6, C7). **This is a change from today's -code**, and it is recorded here, not decided. Today, `prune_empty` drops a row only when -it is empty across the **whole** grid, and prunes the columns against the same unpruned -grid (`layout.rs:752-757` at `50e1314d`). The user's intent, as ADR 008 reads it (its -questions 60 to 65), is that emptiness is judged over the **displayed** values, so a row -that holds a value only in a column outside the window is hidden. *The rest of this -paragraph is the author's analysis.* -- **What the model owes a client** is then not a pruned axis. It owes cross-product - windows (an absolute top-left and dimensions) and a **restricted successor query**: - "the next k rows after absolute position p that hold a value in at least one column of - the set S", and the same with rows and columns swapped. **S travels in the call**, - because only the client knows which columns it displays. A per-axis count cannot - answer it: a row can be occupied overall and still be empty in every column the client - shows. -- **Three ways to answer it**, all unmeasured: - - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then - test each candidate against S by lookup. The state is the per-axis counts clause 9 - keeps. The worst case scans many rows that are occupied overall but empty in S, at - |S| lookups each. - - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that - hold a value in it. The query seeks each of S's lists to p and merges them forward, - choosing the minimum by a linear scan over the |S| cursors: O(|S| × log n + k × |S|). - The state is O(occupied cells), the two-dimensional count clause 9 does not keep. It - is maintained from the cell delta and from formula stems (clause 5's stem rule decides - when a formula cell holds a value). A collapsed group's column merges its members' - lists when it is read. - - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, - tested against S as the scan walks forward. It has the same state as (ii), and it - scans like (i), with an exact test per row. +**Every client receives every notice**, and the host holds no per-client state for +delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists +exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a +refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, +`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and +`GridExtent` is derived by the client from category lists. In records mode the row +extent is the admitted record count, which the client fetches. Whether notices should be +filtered per client is open question 3. - The author leans to (ii): a window has tens of columns, so the merge is small, and it - never walks rows that are empty in the window. Its cost is a second O(occupied cells) - structure beside the store (clause 14). Index (ii) is a set of `(column, row)` pairs, - so it is itself a Z-set fed from the cell delta and from formula stems. A write costs - O(log n), and its delta, "row r gained or lost a value in column c", is what a - subscription restricted to S receives. -- **Under model-side pruning instead**, the model would keep a pruned axis per shape: ADR - 008 clause 3's ordered index over occupied tuples, with rank and select for "row i of - n". Pruning would be a request parameter (ADR 008's flagged reading 1), and the host - would judge "pruned". Everything below that mentions a host judging emptiness applies - only to this alternative. +**A connection operator, at the wire boundary.** One component owns both ends of a +connection. Its host half is a sink for the circuit's results and notices and a source of +requests and model commands. Its client half is the mirror image: a source of results to +the client, and a sink for the client's requests. It owns: +- serialization (T1, T2); +- stamps and the refetch rule (T3); +- coalescing (T5); +- keep-alive in both directions. A client that hears neither a header nor a ping within + the timeout treats the connection as dropped, so silence is never read as "no + batches"; +- session resumption. -**The fixpoint comes from the fill, not from one window** (C8). *The author reads C8* as -a fixpoint on the displayed grid: every visible row has a value among the visible -columns, and every visible column has one among the visible rows. C8 says "drop out -categories", which the author reads as dropping rows and columns, that is, items or -tuples. The user may mean dropping a whole category from an axis. ADR 008 flags the same -ambiguity (its question 66(a)), and it is the user's to settle. *As first written*, this -clause said that hiding a row can empty a column. **That was wrong.** A row dropped for -having no value among the window's columns contributes no value to any of those columns. -So on a fixed window, one pass that drops the rows and columns empty within the window -already is the fixpoint, which is unique (ADR 008's question 61 makes the same argument). -The iteration comes from the fill step, which brings in rows and columns from outside the -window (ADR 008 question 66). A client alternates three steps: -- it fetches a window; -- it prunes, in one pass; -- it fills, asking for the next rows past its last absolute row position with a value - in its displayed columns, and likewise for columns. +In process, both halves are function calls. That covers the TUI, and the browser's +`Worker`, which already owns one `Host` and one port (`worker.rs:14-15`). The host half +belongs in or above `improvise-incremental`, and the wire types in `improvise-protocol`. +`zset` stays free of serde. The client half is shared by every front end, beside +`improvise-protocol`'s `ClientState`, so that a new front end does not write it again. +Its remaining shape is open question 17. -The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from -the change notice which output keys moved, and re-asks. Under (c), a subscribed client receives index (ii)'s -deltas for its S, prunes again locally, and fills again only when a row or column has -gone. Whether the loop always terminates is **not proven**. The fill only moves forward -through absolute positions, so it adds finitely many candidates before the axis ends. A -client that bounds its fills per frame stays within ADR 008's frame budget whether or not -the loop has converged, and that bound is client policy (Q42). +### 13. Column widths for text -**Growth is only rightward and downward** (C9). *The author's analysis.* The fill only -grows away from the absolute top-left, and it stops at the end of each axis. So **the -fill** only ever asks successor queries: "the next row (or column) after p with a value -in S", bounded by the end of the axis. Scrolling up or left is not a reverse fill: it is -a **new request with a new absolute top-left** (C10), and it is the one place a backward -search appears (next paragraph). What forward-only growth simplifies: -- **The index options.** Each needs only a seek to p and forward iteration, so any - ordered structure with a lower-bound seek and a forward iterator suffices: the sorted - `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no - rank over **occupied tuples**, and no "row i of n", is needed. Rank and select over - **each category's items** stay, because a cross-product position is built from them - (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its - rank by that arithmetic. -- **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**, and to their moved keys in a question-13 (b) notice. A formula cell inside the - range moves when any of its inputs move, wherever those sit, so it is tested by the - moved output keys, never by the position of the record that moved. An item edit is an axis change and goes to every consumer (above). A change at - position q matters to a fill only if q lies between its top-left and its front. A fill - whose front is before q finds the change when it gets there. -- **Resumable fills.** A fill's whole state is its request and its last absolute position - on each axis. A resume is a seek past that position, so a lost stream, a reconnect or a - sliced job (ADR 008 clause 11) all resume the same way, with nothing else kept. If a - column's list changes during a merge, a re-seek from the last position is enough. An - item insert before the front shifts positions, and the axis-change message above lets - the client re-derive its front. -- **The corner truncates the work.** A fill is bounded by the requested dimensions and by - the bottom-right corner of the axis space. One round costs O(|S| × log n + k × |S|), - never a scan of empty space past the last occupied position. +The host maintains each text column's maximum display width incrementally. A maximum is +not invertible, so it is an integrated aggregate. Display widths are small integers, so +the state is a histogram from width to count, per (shape key, column): O(distinct widths), +at O(log d) per update. Its input is the delta of the column's text cells over every row, +which the circuit already produces for unfolded cells. Fold rows need the fold maintained +at the visible grain. A text width does not depend on the number format, so there is no +rebuild when the format changes. A client's hidden rows come out through clause 7's overlay. +Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized, +held by the view layer. A number's width would need formatting on the write path, and a +rebuild per format change. Which columns count as text is open question 16. -**A pruned answer as a stream** (C5). Under client-side pruning the fill is the client's, -and each step is one restricted successor query, so the host keeps no fill state at all. -Under model-side pruning, `project` itself would scan forward from the top-left and send -rows and columns as it finds them, as a sliced job resumable from `(request, fill -front)`. Either way, changes during a fill are identity-keyed deltas stamped by publication number -(T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an -ordinary delta with its absolute position, and the client keeps or drops the extra row by -its own policy (Q42). So a fill and a live update are one stream, which keeps the -in-process and the remote consumer the same (T6). Question 21. +### 14. Resident state -**Scrolling up or left, and a top-left in empty space** (C10 to C12). -- **The baseline is the user's galloping search** (C12; "the simplest might be"). *The - author's reading:* the client moves its top-left up (or left) in growing jumps until a - request hits, then narrows back until exactly one new row (or column) is gained. **Each - probe is a restricted successor query** from the candidate top-left t, with the client's - current S, bounded by its old top-left: "is there a row in [t, old top-left) with a - value in S, and which?" It is not a bare cross-product `project`, which would return - mostly empty rows. -- **What it requires:** the response reports the **actual** absolute position of every - row and column returned, which T1 already requires. Under client-side pruning a - top-left that lands on an empty row is no edge case for the host. The host returns the - cross-product window or the successor answer, and the client's own prune decides what - shows. -- **Cost** (arithmetic, not measured). Doubling jumps take about log₂(gap) probes, and a - binary narrowing about log₂(gap) more. At density 10^-4 the gap is about 10^4 rows, - so 14 + 14 ≈ 28 probes. It is often about 14, because a probe returns up to k rows with - their positions. If the hit's rows reach the old top-left, the client reads the answer - directly. Under concurrent writes neither phase can run forever: the gallop ends at the - top of the axis, and the narrowing is a bounded bisection. -- **Statelessness and transparency.** The host keeps nothing for the search, and the - in-process and remote consumers make the same calls (T6). The cost is round trips: about - 14 over a socket for one scroll step into a sparse stretch, and negligible in the TUI. -- **Mixed cases** (the author's). A diagonal scroll moves both axes. The client gallops - one axis at a time. It gallops rows against its current displayed columns, then - columns against the rows it now displays, then prunes once. When only the row axis's - new top-left is empty, only the row gallop does work. When both are empty, both do, in - turn. Whether rows-then-columns and columns-then-rows always give the same window is - **not proven**. ADR 008's question 61(b) notes that order matters for a one-sided - pass, and the order is client policy. -- **A predecessor query is an optional optimisation.** The model could answer "the - previous row before p with a value in S" in one call. It costs no new state under any - of the three options: a sorted `Vec` and a B-tree both seek an upper bound and iterate - in reverse at the same cost, option (ii)'s lists merge backwards as they merge - forwards, and options (i) and (iii) walk backwards with the same test. *The author's - lean:* offer it only once a measured round-trip cost shows that one call is worth a - second query direction. -- *Superseded:* earlier text in this clause leaned to offering the predecessor query - outright, and then called that lean "stronger" once options (a) and (c) below needed - it. The galloping baseline replaces both leans. Options (a) to (c) are kept below as - a record. +Beside the store, the circuit keeps: +- each cell's store sequence, for clause 5's per-record stage (one integer per cell, + kept in `DataStore`); +- the multisets for `Min`, `Max` and text, O(cells) of those measures; +- `ExactSum` at 568 bytes per summed group; +- occupancy per live shape, O(occupied tuples); +- the restricted-query index, O(occupied cells); +- the width histograms; +- the rollback snapshot per mutating frame. -**Superseded: the pruned top-left under model-side pruning.** These options assume the -host judges "pruned", which holds only under model-side pruning. They are kept so that -the user's objection is recorded with what it answered. -- (a) **Fail, then retry** (C10's tentative "maybe"). The error names the nearest - occupied successor and predecessor. -- (b) **Snap forward and report the actual top-left.** **The user objects** (C11). *The - author's reading of the reason:* a client scrolling up asks for a top-left just above - its window. If that row is empty, (b) snaps forward to the window's old top-left, so the - window never moves past an empty stretch. -- (c) **Snap in the scroll direction.** It needs the request to carry its direction. +The store itself is the integrated base that a rebuild reads, so it is never copied. +State is evicted with its shape, and it sits behind one trace interface, so the +store-memory work (offload to disk, compression) can page or compress it later. None of +these sizes is measured at ADR 008's fixture. Migration step 4 measures them. -Neither (a) nor (c) can get stuck. A retry under (a) goes to a named neighbour that is -strictly beyond the old top-left in the scroll direction, and occupied. A write that -empties it only delays the search, by at most the gap. (c) moves in the direction asked, -and stays put only at the end of the axis. At the top of the axis, (a) names only the -successor and reports that there is no predecessor, and (c) returns the first occupied -position and reports that it is the top. So neither bounces. An axis with nothing -occupied answers "empty axis". Both need the predecessor query. +### Relations to other ADRs -**What network transparency requires** (T1 to T6, the author's): -- **T1. Results are plain data.** Every element on an output stream is a value the - protocol crate can serialize, with no `Rc`, no borrowed store key and no closure. Facts - at `50e1314d`: `CellValue` is `Serialize`, with numbers going through `wire_f64` - (`crates/improvise-core/src/model/cell.rs:86-118`). `CellKey` is **not**: it derives only - `Debug, Clone, PartialEq, Eq, Hash` (`cell.rs:10`). So an output element names a cell by - the request's row and column tuples, or by cross-product positions. Where a write may - follow, it also carries ADR 002's surrogate (`CellId`). It never names a cell by - `CellKey`. -- **T2. One identity on both sides of the wire.** A consumer across a wire must - consolidate exactly as an in-process one does. **On NaN they would not.** Clause 3's - delta identity is bit identity, under which two NaNs with different payloads differ. But - the wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114` at `50e1314d`), and the - projection compares values with `same_on_the_wire` (`crates/improvise-protocol/src/render.rs:173-180` - at `50e1314d`; `context/code/protocol.md` gives the reason). So an output delta - `−(c, NaN₁) + (c, NaN₂)` is non-empty in process and consolidates to nothing after a - round trip. **Proposed:** output streams are consolidated under the wire's identity - (every NaN is one value, and every other number is its bits) before they leave the host. - The record tier keeps bit identity, as it does today. Question 15. -- **T3. Every published batch is stamped, by a publication sequence.** The stamp is - `(host instance, document_generation, publication)`. - - **The publication number** increments once per **published batch**, and that is - what a client counts. The circuit's internal `step` advances once per model effect - (clause 3), and it is never a client-visible gap detector. *As first written* - (superseded), the stamp carried `step`. A batch of two model effects then advanced - from n to n+2 under one header, and a client read the missing n+1 as a loss (found by - ChatGPT Codex on `70ababfe`). *The author's lean* is this separate sequence (i). The - alternatives: (ii) a header per model-effect step, which multiplies headers and - exposes a batch's internal steps, which ADR 001 treats as one atomic unit; or (iii) a - header that carries the previous published stamp, so a client can tell a - consolidated range from a loss, at a little more per header. - - **The host instance** is an id chosen when the host starts. The other two counters - live in memory and start again at their initial values after a restart. Without the - instance id, a reconnected client would drop the new host's notices as duplicates. - - ADR 001 already puts batches in one total order. A consumer that has applied - publication n, of the same instance and generation, drops anything at or below n, so - a duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot - answers. - - Under question 13's (b′), every client receives every batch's **header** (clause 12), - so the publication numbers it sees are contiguous, and a gap is always a real loss. - Contiguous numbers prove only that no header was lost, not that a key list was routed - to the client. That holds only because a subscription is registered atomically with - its fetch (clause 12). - - **The general rule:** a client that misses a notice, sees a gap, sees - `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. A changed host instance is such a - discontinuity, so a reconnect after a restart always refetches. The same rule holds - for a client under (a), (b) and (c). - - What is published is the **batch's** consolidated delta, and only once the batch is - accepted. A refused frame is restored (clause 11), publishes nothing, and consumes no - publication number. -- **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the - snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at - publication n: the window as a Z-set of current values, each with weight +1. Deltas - from publication n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current - result. `GridCache::folded(&cache.seed()) == *cache` states the same property for - today's bootstrap compaction (gotcha 32). -- **T5. Order holds across streams, and coalescing happens within a stream** (as stated, - for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client - merges pending notices into one, taking the union of their moved keys and flags and the - latest publication number, and refetches once. **Axis changes do not merge.** They are positional, - so they are kept as a list and applied in their original order. A - `document_generation` change inside the merged run voids the merge, because the old - keys belong to a document that is gone, and the client refetches from scratch). A slow - consumer's pending deltas on one stream can be **summed** into one delta. Z-set - addition is commutative and associative, so the sum is exact, and the queue is bounded - by the size of the result, not by the number of steps. When the summed delta outgrows - the snapshot, a snapshot replaces it. That is the backpressure, and nothing is dropped - silently. **Across streams, order stays load-bearing**, as gotchas 33 and 43 require: - the occupancy change comes before the cells it places, and the cells come before the - basis that describes them. This does not break gotcha 42's "never coalesce". That rule - is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple - identity, or by a stable cross-product position, is not positional in that sense. - **Coalescing is therefore per subscription, never per stream.** All of a - subscription's pending deltas, the basis included, are summed together up to one - publication number, - and sent as one batch in the fixed order. Coalescing one stream while another is sent as - it arrives would let a basis for publication n+2 overtake the cells it describes, which is - gotcha 43's failure. -- **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives - the same result and delta types as a remote client. Its transport is a function call. - That is what makes the requirement testable: run one edit sequence through two - consumers, one in process and one behind a serialize and deserialize round trip, and - require their integrated results to be equal under T2's identity. A TUI shortcut that - read operator state directly would be a second read path, which is the derivation-twice - defect (design-principles §5). - -Weights on the output are ±1, because a result is a set of current values. A consolidated -weight that does reach past 2^53 follows the user's bigint-as-string rule (Q36). - -**Where the wire form lives** (the author's). `zset` stays dependent on `std` alone (clause 1). -The wire types live in `improvise-protocol`, with the one envelope pair (gotcha 42). A -consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that boundary. -The alternative is a `serde` feature on `zset`. The author leans to the conversion, -because what crosses is the projected result, not the crate's own element type. - -**How stateless the host can be, and what replaces the pushed stream.** The user: "as -stateless as possible" (C2), and "The backend should only hold model-relevant -information" (ADR 008 Q33). ADR 008's question 52, what replaces the pushed projection -stream, was answered "defer this to ADR 009", so it is this ADR's question 13. The -options, in ADR 008's lettering: -- **(a) Never learns: refetch by policy.** The host keeps nothing and says nothing. A - client refetches when its own policy says to (Q42). A change made by another client - stays invisible until then. -- **(b) An unaddressed change notice** (Flux style). After each accepted batch, the - model emits one notice to every consumer, addressed to none. It carries: - - the stamp `(host instance, document_generation, publication)` (T3); - - each item edit, as an axis change (above); - - a flag when the formula set or `measure_agg` moved. Either may move every aggregate - without moving a record. `measure_agg` has no wire command today, and changes only - with a load, which moves `document_generation`; - - **what moved, as the model computes it** (the author's lean, below); - - **display relabels.** Under ADR 007 a rename is one `display` write, which moves no - key (clause 3), so no moved-key list can carry it. The notice carries it as `(id, - new display)`, and the client relabels what it holds without refetching. Before ADR - 007, a rename is a removal and an insertion of every cell under the old name, and it - arrives as moved keys. - - **Records alone are not enough** (found by the review of `dce3ab63`). A client could - test whether each moved record falls under its own axes and filters, but that misses - every formula whose inputs sit at another stem: an inner `WHERE` naming another item, - or a ref to an item of a page, filter or None category (clause 5's one join). For - example, a client on page `Region=West` shows `EastShare = Revenue / SUM(Revenue WHERE - Region=East)`. Another client writes an East record. The West client's test says "not - mine", and its cells stay stale until its own refetch policy fires. Two remedies: - - **The author's lean: the notice carries the moved keys the circuit already - computes.** Clause 5 finds the affected `(stem, formula)` pairs, and clause 4 the - moved group keys. That is the circuit's whole job. For each **live shape key**, which - is a model-side cache keyed by requests and never by a client, the notice lists the - moved output keys: cells, formula cells and column-tuple totals. A client intersects - them with its window by identity, so the model decides relevance and the client only - looks it up. **A collapsed fold row has no maintained output key under item grain** - (clause 7: it is combined at read time). So a moved member key goes through the - client's own member-to-group remap before it is matched against the window. The - client holds that remap, since collapse is its UI state. A notice then costs O(moved output keys across live shapes), not - O(records moved). - - **The alternative: a conservative client rule.** The notice carries records only, - and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to - a page, filter or None item, or an aggregate) treats **every** moved record as - relevant. The notice is small, but such windows refetch on every batch. - - Under either remedy, the client refetches what it needs by `project`, or by question - 22's successor query. A formula or `measure_agg` change means refetching the window. - The host keeps no per-client state. -- **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask - to be kept current. The host keeps the request and the last publication number it sent, then pushes - the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost - table costs a snapshot). But it is per-client state on the backend, and it holds a - window, which is a client's view parameter. -- **(d) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and - `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, - `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. - -**The trade between (b) and (c) is host state against wire traffic** (the author's; -corrected after the review of `5a8dc486`; nothing here is measured). -- **(b) keeps no per-client state, and pays in traffic.** Its one broadcast is still - delivered to all N clients, and each receives every live shape's moved keys. That is - O(N × all moved keys) on the wire per batch. In this ADR's primary case, a live source - streaming into a large table, every remote client would receive every moved key, most - of them for shapes it does not read. -- **(c) keeps per-client state, and saves traffic.** It holds the request and the last - publication number per subscriber, which is a client's window and so a view parameter on the host, - against Q33. In return it sends each client only its window's deltas. -- *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) - per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. - -**A middle form, (b′): one topic per shape key.** A client subscribes to the topic of -its own shape key. The host keeps O(1) per client: a topic subscription, which names a -shape key and holds no window, cursor or view. **A topic filters only the moved-key -lists; every batch's header goes to everyone** (found by the review of `e152b667`): -- **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the stamp (host instance, `document_generation`, publication; T3), - the axis changes, the display relabels, the formula and `measure_agg` flags, and the - eviction list (below). None of - these is per shape: an item insert shifts every shape with that category on an axis. - So a filter must not withhold them. Headers are also what keep T3's loss detection - working. Every client sees every stamp, so a gap is a real loss and never a quiet - batch. That costs O(N) small headers per batch. -- **The moved-key lists are filtered by shape.** A client receives only its own shape's - list. -- **Subscribing, first or after a switch of shape key.** *As first written* - (superseded), this said a client fetches at publication n and then subscribes from n, and - that the switch "leaves no gap". **That was wrong** (found by the review of `70ababfe`). - The host keeps no past moved keys. So if batch n+1 moves a key in the new shape before - the subscription is registered, the client gets header n+1, with contiguous stamps, - but not n+1's key list, and its window goes stale silently. A client's first - subscription has the same race. *The author's lean:* **(i) the fetch registers the - subscription atomically.** The request names its topic, and the host registers the - subscription in the same step that computes the fetch at n. Every later batch then - routes to it. The alternative, **(ii)**: the - subscription is acknowledged with the current publication number m, and the client refetches if - m > n. That costs a second round trip whenever a batch lands in between. The client - drops the old topic once the new fetch is in. -- **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a - shape once no recent request names it. A client still subscribed to that topic would - then receive no moved keys, which looks exactly like "nothing changed", and its window - would go stale. *The author's lean* combines two invariants: - - **(i) a live topic subscription pins its shape, and a subscription lives exactly as - long as its connection** (revised after the review of `85dc0ad8`). *As first - written* (superseded), subscriptions were leases renewed by requests or a - heartbeat. That let a client's own lease lapse silently while another client still - pinned the shape. The shape was then not evicted and nothing was announced, so the - client kept receiving contiguous headers but no key lists. Tying the subscription to - the connection (the socket, the worker's port, or the TUI's in-process call) removes - the heartbeat and the lapse together. A live connection keeps its subscriptions. A - dropped connection drops them. **The transport must detect a dead connection**: a - websocket ping with a timeout on the socket server, a closed port for the worker, and - nothing for the TUI, which is in process. **The ping runs both ways.** A client must - also detect a dead host, or it reads silence as "no batches" and keeps a stale - window. So a client that hears neither a header nor a ping within the timeout treats - its connection as dropped, and refetches on reconnect. A half-open socket that is never detected - would hold its pins forever. A client always knows when its connection dropped, - and it refetches on reconnect (T3's general rule). *The author's lean*, because it is - the simplest of the three. The alternatives were a per-client subscription epoch in - that client's header, or acknowledged renewals with a self-unsubscribe on a missed - acknowledgement. Each adds a mechanism. - - **(ii) every eviction is announced.** Any eviction, by ageing out or under - memory pressure (clause 14), puts the shape key in the next header's **eviction - list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. - Evicting a popular shape therefore causes a **refetch storm**. **Idle readers make - it likely under plain (b):** a shape that clients only watch, without requesting, - ages out, and every one of them refetches at once. That refetch is also what brings - the shape back. *The author's - options*, unmeasured: refuse to evict a shape with many live subscriptions - (hysteresis), stagger the refetches with a per-client jitter, or rate-limit - rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list - names only the shapes evicted since the previous header, so it does not grow without - bound. - - **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists - moved keys only for live shape keys, and a topic-less client pins nothing. So its shape - can be evicted silently. The header's eviction list reaches it too, and it refetches. - Only the "records plus a conservative rule" alternative of (b) escapes the hole, because - it lists no shape at all. Instead of an eviction list, a header could carry the set of - live shape keys, or a shape-cache generation that moves on every eviction. A client - whose key is missing from the live set would then re-request. Both say the same thing, - and the eviction list is smaller. - - The alternative, **(iii)**: every header states, for each shape it carries keys for, - that the shape was evaluated, and a subscribed shape missing from that statement means - a conservative refetch. It needs no pins, and it costs a list of evaluated shapes - per header. -- **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) - per batch. It saves nothing when every client reads one shape, and much when shapes - differ. It is still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** A subscription is per-connection state. - It holds no UI state, and the next request, which names its shape, rebuilds it. - The author reads it as delivery state, not UI state. **But the user said "The backend - should only hold model-relevant information"** (ADR 008 Q33), and whether a topic - subscription counts as that is the user's question, put in question 13. -- **Network transparency.** The topic is a filter applied when sending, so in-process - and remote consumers still read the same notices (T6). The TUI simply subscribes to its - own shape. - -**What each form costs in complexity, not only in traffic** (the author's; nothing -measured). The mechanism (b′) needed has grown review by review: topics, pins, -atomic registration, an eviction list, and a lease (now a connection tie). That growth -is itself a signal. -- **Plain (b): moved keys per live shape, broadcast to every client.** It keeps no - per-client state and needs no topics, pins or registration. It still needs the eviction - list, because a client's shape can be evicted. Its traffic is O(N × all moved keys) per - batch. **With a size cap**, a header whose key lists exceed a fixed size carries "too - many changes, refetch" instead of the lists. That bounds every notice, at the price of a - refetch by every client in a busy batch. -- **The records-only variant of (b).** It sends moved records and a formula flag, and - windows with cross-stem formulas refetch on every batch. It needs no shape lists, so it - needs no eviction list either. It is the simplest form, and its notices are the - smallest. It refetches the most. -- **(b′): (b) filtered by connection-tied topics.** It cuts traffic to the shapes each - client reads. The price is per-connection state, which is up against Q33, plus atomic - registration at fetch and pins on the shape cache. - -*The author's recommendation, revised again:* **plain (b) with a size cap**, keeping no -per-client state, with **(b′)** as the optimisation to adopt only if traffic is measured to -matter with many clients on different shapes. (c) stays the further optimisation for a -small window on a busy shape. (b′) is defensible against (b) with a cap only where -clients read many different shapes of one busy document, and nothing here shows that -case is common. The user may well prefer the simpler design. *As first written* -(superseded), the lean was (b′). - -**The user's answer to question 7 bears on this, and does not decide it.** "As much state -as possible should be maintained in the backend, unless it's UI-specific state", and "we -probably need to have a mix of caching and a pull model to expand the cached data". Plain -(b) keeps the maintained model state in the backend and has clients pull on a notice, -which fits both sentences. Whether a per-connection topic is backend state of the kind -the user means, or UI-specific, is still question 13's question to the user. - -*Earlier still* (superseded), this clause leaned to (c), which it then called (b). That -was before the user said the backend holds only model-relevant information. Under (b′), -the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is -keyed by requests and holds no client, plus the topic subscriptions, which hold one shape -key per client. - -**A connection operator (C13, the user's consideration; everything below is the -author's reading).** The user: "I think we can minimize the amount of UI state on the -backend by having an operator of some kind responsible for both ends of the connection -between the frontend and backend and handling keep alive / session resumption / etc". -- **What "operator" means, after C14.** C14 resolves most of C13's ambiguity: "And it - asks as a sink/source for the DBSP graph part on the backend and a source/sink for the - client side", with "asks" read as "acts" (flagged). *The author's reading:* the - operator is **shaped like the DBSP graph**. Toward the host it is a **sink** for the - graph's output streams (projection results and change notices) and a **source** of the - requests and model commands it feeds in. Toward the client it is a **source** of those - results and notices, and a **sink** for the client's requests and events. So the - connection between frontend and backend is itself a pair of sink/source endpoints, and - the operator owns both. What stays ambiguous is whether "both ends" means both halves of - that pair inside one component, or a component in each of the two processes. Question - 28. -- **Why this fits C1** ("network transparent"). The circuit's streams are the same - whether consumed in process or across a wire, and the operator is the **adapter at the - wire boundary**. It owns: - - the serialization of Z-set deltas, under T1 (`CellKey` is not `Serialize`, so the wire - form names tuples and ADR 002 surrogates) and T2 (the wire's NaN identity); - - the stamps and the gap rule (T3); - - snapshot-then-delta (T4); - - coalescing and backpressure (T5); - - session resumption. - - In process, both of its halves collapse to function calls, which is T6. -- **Where it sits among the crates** (the author's reading). The host-facing half - consumes the circuit's output types, so it belongs with the seam crate - `improvise-incremental` (question 2's answer) or right above it. The wire types it - serializes belong in `improvise-protocol`, beside the one envelope pair (gotcha 42). - `zset` stays dependent on `std` alone (question 1's answer), with no serde, because the - conversion happens in the operator. The **client-facing half** would be written again - by every front end unless it is shared, so by design-principles §5 it belongs in the - shared layer, as `improvise-protocol`'s client fold (`ClientState`, `client.rs`) - already is. Who owns keep-alive and resumption state, and whether the TUI, the browser - and a CLI share one client half, are question 28. -- **What it would hold, so that the host does not:** the connection; the keep-alive, with - the ping both ways; session resumption, meaning a reconnect with a token and then the - replay-or-refetch rules (T3's host instance and publication number); the topic - subscriptions and their pins ((b′)); and each client's last applied stamp. The host - would then answer `project` and `apply`, publish its unaddressed notices, and know - nothing about clients. The operator is the one place that does. -- **Where it could live, in today's code:** - - in process, for the TUI; - - beside the host in the browser worker: `improvise-worker-server`'s `Worker` already - owns one `Host` and one port (`worker.rs:14-15` at `f0fb6880`); - - as the socket server's front: `improvise-ws-server`'s `Registry` and transport are - the nearest thing today (`registry.rs:88-89`). - - Whether it is a new crate, a new process, or a role that existing code takes on is - question 28. -- **What it changes for question 13.** If the operator subscribes to the host's - unaddressed notices **on behalf of its clients** and fans them out, the host sends - O(operators) notices per batch, not O(clients). The per-client filtering of (b′), and - even the window restriction of (c), move into the operator. The wire between host and - operator then carries (b)'s broadcast, and the wire between operator and client carries - (b′)'s or (c)'s traffic. That changes the trade the review of `5a8dc486` caught, of host - state against wire traffic: with an operator, (b′) and (c) cost the **operator** state, - not the host. The costs are a new component, a second hop, and a place where a lost - notice can happen. When the operator is in-process or in the worker, the second hop is - a function call. None of this is measured. -- **What it changes for question 14.** Pins and subscriptions belong to the operator. It - re-requests the shapes its clients read, and that keeps them alive in the host's cache, - so the host's eviction rule (no recent request) holds unchanged. A host-side pin is no - longer needed. -- **The Q33 question.** If topic subscriptions live in the operator, the host holds none, - and a subscription is no longer "backend" state in the sense of the user's Q33 answer. - That holds only if the operator does not count as backend, which is part of question - 28. -- **Network transparency and "as stateless as possible".** T1 to T6 then hold on each - hop, and the operator must not change a result, only route it. The host becomes as - stateless toward clients as plain (b), and the per-client state is concentrated in one - component that can be restarted, because a client then refetches (T3's general rule). - In ADR 008's view/host split, the operator sits on the **host side of the wire, outside - the model**. It knows clients and holds no UI state beyond delivery state. -- **What it does not answer.** Question 13's choice, question 14's eviction policy, and - whether the operator is a new component at all. - -**What becomes of today's pieces under (b) and (b′)** (the author's): -- **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its - levels' item counts, which a client derives from the category lists it fetches. A - pruned axis has no global extent under client-side pruning (clause 12), and question - 22's successor query answers "is there more". **In records mode** the row extent is the - record count that the page and filter admit. That is a model fact, which the client must - **fetch**, because it cannot derive it. -- **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which - widget a change can reach, moves into the client's test of the notice. -- **The occupancy delta** is not pushed. It stays a model-side output that keeps question - 22's index current. A client learns of a new or vanished row from the notice's records, - and re-asks. - -A shape is kept while recent requests name it or a live topic subscription pins it. -Eviction drops its operators, is announced in the next header's eviction list, and the -next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has -not set (question 14). - -**What a client must hold to resume:** its request, the last publication number it applied, its window, -and its basis. The host holds nothing that a client needs back. - -**The basis.** ADR 001's basis is answered with the result it describes, and after it -(gotcha 43), built from the request's window, never from a mirror. `published_basis` -(`project.rs:585` at `50e1314d`) already takes the `GridCache` it describes, not a -cursor. Under the user's Q39 answer, only `document_generation` and the record tier are -model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). - -### 13. Incremental max width: the host maintains it for string-value columns - -The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" -**The user then decided** (question 12, 4226624890): "Yes, I think the host should -maintain the max width for string-value columns incrementally." So **the host maintains -each string-value column's maximum display width incrementally**. This clause is now -that direction. Its cost analysis is the author's, and none of it is measured. - -**Scope.** "String-value columns" is read, by the author, as columns whose cells hold -text values (`CellValue::Text`), as opposed to numbers formatted for display. Numbers' -widths depend on the number format, which is display-only, so they stay with ADR 008's -rule: (b), sticky per session, eased by hysteresis, over the rows realized, held by the -view layer (ADR 008 clause 4). Whether the user means this split, and what a column -holding both text and numbers does, is question 27. - -**For text, two costs below shrink.** A text value's display width does not depend on -`number_format`, so there is no rebuild when the format changes, and no formatting on -the write path, only a display-width measurement of the string. What remains is the -histogram per column, and the delta of text cells over the whole column, including fold -rows. - -**DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. -When the widest value is retracted, the new maximum cannot be computed from the old -maximum and the retraction. So it belongs to clause 4's **integrated** class, beside `Min` -and `Max`: each group keeps its integrated multiset and reads off the last key. A *top-k* -(keeping only the k widest) holds less state, but it is wrong once the top k have all been -retracted, and then it needs a rescan. So it is not proposed. - -**For widths the multiset is small.** The group is one column of a projection shape, -together with its number format, since a width is measured on formatted text. The values -are display widths, which are small integers. So the state is a histogram from width to -count, per column. It is O(distinct widths) per column, whatever the number of rows, and -an update costs O(log d), where d is the number of distinct widths. - -**The cost is upstream, in what feeds the histogram.** Its input is the display width of -**every evaluated cell in the column**. Of the three costs below, the second and third -apply **only to numbers**, which a host-maintained width would cover only if question 27 -extends the decision to them. For text, the first one remains: -- **A delta of evaluated cells over the whole column**, not a window. Clause 4's group - aggregates and clause 5's formula re-evaluation already produce an output delta for - every unfolded cell that moves, with its old value as the retraction. So unfolded rows - need no new state. **Fold rows are not covered.** Under item grain (clause 7) a - collapsed group's value is combined when it is read, not maintained, so its width would - need the fold maintained per column at the visible grain. That holds for text as well: - a fold row's text, under the unanimity rule, is a combined value. -- **A client's hidden rows** come out of the whole-column histogram the way they come out - of a total: subtracted at read time by question 23's (A) overlay, at O(H log d). -- **Numbers only: formatting every changed cell on the host**, on the write path: `format_f64` and a - display width for each changed cell. Each one is cheap, but it would be the first - display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value - back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But - today the host is asked for widths only over the rows requested (ADR 008 clause 4), and - this would widen that to whole columns. -- **Numbers only: a rebuild whenever the format changes.** `number_format` is a parameter of the view - definition, and a rebuild costs O(occupied rows) per column, as a job. - -**Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 -option (c), "size each column from its whole contents". The user had chosen (b) there, -and now chooses the maintained width for string-value columns. So the two rules split by -column kind, and ADR 008 should record that (question 27). *As first written* -(superseded), this paragraph recommended not maintaining widths "now", and asked the -user in question 12. The user answered yes. - -### 14. Resident state, and the store-memory requirement - -The user, on ADR 008: "We have to be careful here to minimize memory requirements of the -store: offloading inactive regions to disk (maybe mmap or similar), -double-delta/gorilla/lz4 compression, etc." (#308, comment 4171481179). And then, on ADR -008's question 17: "Let's defer this as recommended." (4188271710). So the store's memory -gets its own ADR, once this ADR's operator state has been measured at ADR 008's fixture. -**This clause withdraws the earlier non-goal** ("Persistence, spilling to disk, and -threads", clause 1), and it says what this ADR must not rule out. - -**What this ADR keeps resident, beside the store.** This is the author's inventory. The -orders come from the clauses and are not measured. - -| state | clause | order | -|---|---|---| -| the integrated base that a rebuild reads | 7 | O(cells), if it is kept as a copy | -| clause 5a's per-record stage | 5a | O(cells of the records an aggregate over an expression reads), as written | -| multisets for `Min`, `Max` and text | 4 | O(cells) of the measures that need them | -| `ExactSum`, per summed group | 8 | 568 bytes per group | -| occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | -| the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | -| the subscription table, only under question 13's (c) | 12 | O(request) per subscription | - -**Proposed by the author:** -- **The store is the integral, so it is not copied.** The integrated base a rebuild reads - is `DataStore` itself, which is I(cell delta) by definition. Clause 5a's per-record - stage keeps each record's cells only to compute the retraction `I(x)₋₁`. That equals - `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. So - the stage can read the store and keep no copy of the values. It still needs each cell's - store sequence (clause 5a), one integer per cell, which `DataStore` would keep beside - the value rather than in a second map. This is **not measured**, and the retraction of - the order key has to be checked against it (question 17). -- **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a - range, and apply a delta. The first implementation keeps it in memory, and the - store-memory ADR can page or compress it without changing any operator. -- **Shapes are evicted** unless a live topic subscription pins them (clause 12). So per-shape state is bounded by the shapes in - recent use, not by every saved view. -- **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, - `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and - O(groups), and the store-memory ADR has to count them. - -**The frame budget.** ADR 008 clause 11 sets it, and the user accepted the recommendation -(ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms -slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is -a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration -step 4 must. It reports both as ratios against an unrelated operation on the same run, -as ADR 008's question 13 requires, not as bare milliseconds. +- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its + `document_generation` and record tiers, not on `layout_version`. +- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace; + epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values. + Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a + 256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators. +- **ADR 006:** replay reproduces a circuit's state. +- **ADR 007:** keys become ids, so a rename is a relabel (clause 3). +- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the + occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire + prunes, and which parameters a request carries, are shared with questions 9 and 13 + below. ## Alternatives considered -**Feldera's `dbsp` crate.** What was verified on 2026-09-24: - -- The latest release is 0.354.0, a 4.8 MB compressed tarball, MIT OR Apache-2.0, - requiring Rust 1.96.1. -- It has 67 direct normal dependencies, none optional, and two features - (`backend-mode`, `default`). tokio, mimalloc, nix, zstd, libc and core_affinity are - all required. -- `cargo tree -e normal` resolves about 455 crates natively and 421 for - `wasm32-unknown-unknown`. For comparison, improvise-core's whole normal tree for wasm32 - is 25 crates, and improvise-command's is 68. -- `cargo check` succeeds natively. **`cargo check --target wasm32-unknown-unknown` - fails**: `mio` (tokio's) reports 48 errors. - -What was **not** verified: whether anything past `mio` would also fail. The build stops -at the first failure, so that is estimated, not measured. It is likely, because -mimalloc's sys crate and zstd build C code, and `libsqlite3-sys` already fails under this -repo's nix cc-wrapper (gotcha 41). Even if it built, it is a multi-threaded, -storage-backed query engine where improvise needs a single-threaded operator library. -The user's decision already excludes it. This records why that decision loses nothing. - -**Keep eager evaluation and memoize.** improvise-swre proposes keying `grid_of` on the -existing tiers, so an unchanged grid costs nothing. That is worth doing under either -backend. But every write still recomputes every formula and rescans the store, and a -second view with a different None axis still clones the model. It removes the cost of -*not* changing and leaves the cost of changing at O(cells). - -**An ad-hoc incremental cache.** Track dirty stems by hand: invalidate the groups a -write touches, and recompute them. It is cheaper to start. But each derived fact, such -as a group aggregate, occupancy, a column total, a stem or a cross-stem `WHERE`, needs -its own invalidation rule. Each rule is a second description of evaluation, free to -disagree with the first. The Z-set algebra gives **one** rule, linearity and -bilinearity, whose laws can be property-tested once in the crate. That is the same -argument as design-principles §5's "a derivation done twice is the defect". - -**Compile formulas into operators.** This is the fully DBSP-shaped answer. It is -rejected in clause 2, because it would be a second evaluator. +- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio, + mimalloc, zstd and others), about 421 crates for wasm32, and fails + `cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded, + storage-backed engine where improvise needs a single-threaded operator library. +- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only + the cost of not changing: every write still recomputes every formula. +- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule, + which is a second description of evaluation. Z-sets give one rule, linearity, whose laws + are tested once. +- **Compile formulas into operators.** A second evaluator (clause 2). ## Consequences and migration -**Order.** Each step is its own ticket and PR, with gates green, and none switches the -displayed path until the differential test holds. +**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches. +The store scans behind the record tier and the dirty flag go, and so do the per-view model +clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources +that can report changes plug in unchanged. The host keeps no per-client state, and one +delivery path serves every front end. -1. **The `zset` crate**, with its group laws, operator properties and determinism - test. Nothing in improvise uses it yet. -2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the - existing suite is the evidence. `CellKey` gains `Ord`. -3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero - and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half - becomes "a non-empty projected cell delta, or a moved formula-set fingerprint" (clause 3), with - a test that a formula edit alone still marks the document dirty. That removes the - O(cells) scans per cell write. The existing tier tests are the evidence. -4. **`Incremental`, for the order-free parts**, under the differential test: stems with - clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's shape key and read-time parameters. The formula delta arrives - only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in - step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the - frame observation) depend on this step.** -5. **Formulas**: the formula graph and the formula delta (clause 7), topological - re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that - replaces `Expansion::members`: the integrated per-record stage, the map keyed by - context and order key, the inner-`WHERE` join, and the error channel. Its tests - include a change to one operand of a two-operand expression, and the deletion of a - record's leading cell. Formula values that - depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR - 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals**, with the exact accumulator in both backends, - as the user decided on question 3. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), - a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs - question 3's (c) for that subtraction to be exact. -7. **Live sources as delta streams**, when ADR 003's source trait exists. -8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to - 22. It brings the output types in `improvise-protocol`, the publication stamp (T3), - snapshot-then-delta, question 22's restricted successor query, and the test with two - consumers (T6). **Required tests:** - - a multi-effect batch such as `[AddItem, SetCell]` publishes exactly one header, and - no client refetches; - - a dropped header **is** detected as a gap, and the client refetches; - - a batch that lands between a fetch and its subscription reaches the client's key - list (question 13's atomic registration); - - a reconnect after a host restart refetches, and drops nothing as a duplicate; - - a refused frame consumes no publication number; - - under (b′), a dropped connection drops its subscriptions, and the client refetches on - reconnect, even while another client still pins the shape. It depends on step 4 for occupancy. It replaces the per-mirror diff - only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, - the cost of a step and of a rebuild (clause 14). +**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that +must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs +568 bytes per group, and displayed sums change in their last bits. Circularity is decided +in two layers. The network path still clones per mutating frame. Every client receives +every notice. Every column-total read pays O(H log d) for the request's hidden rows. -**Tickets.** improvise-felo.4 is this ADR's record **only**. The user: "No, new epic and -felo.4 records ADR only" (question 10). So the steps above go to a **new implementation -epic**, which this revision does not file. improvise-swre stays independent and is useful under either backend. -improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is -already inexact beyond `FLOAT_EQ_EPSILON`. Under the user's answer to question 3 it is no -longer a prerequisite of step 6. - -**Good.** - -- On the keyboard path, a write's *evaluation* costs O(delta) plus the formula cells it - reaches, not O(cells). On the network path, the ADR 001 snapshot still costs O(cells) - per mutating frame (see "Bad"). -- The store scans behind the record tier and the dirty flag disappear. -- Per-view model clones for a different None axis disappear. -- ADR 008 gets occupancy, the occupancy delta and column totals without a second - evaluator. -- Live sources that can report changes plug in without touching the evaluator. -- One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client UI state under question 13's (b′), only one topic per client (none under plain (b)). Option (c) would add a subscription table that can be rebuilt - from the requests. - -**Bad.** - -- Two backends, kept equal by a differential test that must itself be kept honest. -- Integrated state is O(cells) in memory, per distinct None axis, in addition to the - store. -- An f64 `Sum` cannot be maintained on the displayed path at all. Maintaining it needs - an exact representation: the accumulator costs 568 bytes per group (clause 8), or - ADR 005's decimals. -- Under question 3's recommendation, `Eager`'s displayed sums change in the last bits, - because they become correctly rounded. That is a behaviour change, even though it is - a fix. -- Circularity is decided in two layers: the formula graph's components and the - per-cell bound. -- A mutating frame from a client already clones the workbook for rollback - (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) - clone per frame. Incremental evaluation does not make the network path O(delta). -- Under question 23's default (A), every column-total read pays O(H log d) to subtract - the request's hidden rows, and clause 5a's totals need their entries indexed by row - tuple as well. Occupancy and cells do not rebuild. -- The shape keys are re-read after every model effect: O(categories) per live key, a cost the - post-loop `view_shape` diff paid once per batch. -- The browser bundle grows by two small crates. -- Per-shape state lives only as long as requests name it. A shape that comes back after - eviction is rebuilt as a job, and the client waits through its pending state. -- The output identity (every NaN is one value) differs from the record tier's (bit - identity), as T2 explains. Two identities need stating everywhere a delta crosses. -- A position over a changing item list is never fully absolute. An item insert moves - every later position, and is sent as an axis change (clause 12). - -## What the user decided - -**On 2026-10-09 the user answered questions 1 to 10 and 12** in inline comments on #309, -in review 5465446805 (03:34:52Z to 04:32:13Z). They are quoted exactly below, double -spaces and the "is is" kept. Question 3's comment is anchored on an older head -(`d766605b`, line 1100), where that line is question 3's option (c). Each other comment -is mapped from its `original_line` on `6ad36437` to the question it closes. **The user -has not said "Approved"**, so the ADR stays proposed. Question 11 is unanswered. - -- **1.** (4226370667) "zset is is fine." The crate is `zset` (clause 1). -- **2.** (4226371183) "seam crate". `Incremental` lives in the seam crate - `improvise-incremental` (clause 2). -- **3.** (4226373188) "use the exact accumulator." Option (c): `ExactSum` for `Sum` and - `Avg` over f64 (clause 8, migration step 6). -- **4.** (4226588257) "Yeah, for testing purposes but eventually remove it because DBSP - should fix the cycle issue." The sixteen-pass bound is reproduced so that the - differential test compares like with like, and is to be removed later (clause 6; - question 25). -- **5.** (4226590267) "For validation, but if there's a more natural DBSP-native way we - should plan on that." The component fixed point is kept for validation, and a - DBSP-native treatment of cycles is to be planned (clause 6; question 25). -- **6.** (4226592632) "For now: the goal is to only have DBSP formulas". `Eager` stays - for now, and the goal is that formulas are evaluated by DBSP only (clause 2; - question 26). -- **7.** (4226599186) "As much state as possible should be maintained in the backend, - unless it's UI-specific state. As far as possible, state should be incrementally - recomputed using DBSP operators to minimize per-request recalculation." And - (4226616602) "However, this is intrinsically limited because the grid is infinite, so - we probably need to have a mix of caching and a pull model to expand the cached data." - *The author's reading:* this endorses question 7's slice, which keeps every page's - state maintained in the backend, and it states a principle that bears on questions 13 - and 14 (clause 12). -- **8.** (4226618377) "Eager should be deprecated." This **overrides** question 8's - recommendation, which kept query-only sources on `Eager` (clause 10; question 26). -- **9.** (4226622610) "The recommendation is fine, I think". The backend's state joins - the rollback snapshot, with no later switch to the inverse delta (clause 11). -- **10.** (4226621783) "No, new epic and felo.4 records ADR only". This **overrides** the - recommendation: improvise-felo.4 stays the record of this ADR, and the implementation - steps go to a new epic. Nothing is filed by this revision. -- **12.** (4226624890) "Yes, I think the host should maintain the max width for - string-value columns incrementally." This **overrides** the author's recommendation - ("no, not now"). Clause 13 is rewritten as the decided direction; its scope is - question 27. - -**What the answers change, and where they pull against earlier text** (the author's -readings, flagged): -- **`Eager` is deprecated, not permanent** (questions 6 and 8). Clause 2's "why keep - both", clause 10's query-only sources on `Eager`, and clause 5a's eager path for - formulas the per-record map cannot express all assumed a permanent `Eager`. They now - hold "for now", as the differential test's oracle. What replaces `Eager` for those two - uses is question 26. -- **"Only DBSP formulas"** (question 6) pulls against clause 2's rule that formulas are - never compiled into operators, and that the one existing evaluator computes each - value. The user's words may mean "every formula goes through the circuit", which - clause 2 already allows, or "formulas become operators". Question 26. -- **"DBSP should fix the cycle issue"** (question 4) pulls against clause 6's argument - that DBSP's least fixed point is not improvise's circular semantics. `X = X + 1` has no - fixed point. Question 25. -- **"As much state as possible ... in the backend"** (question 7) sits beside the - user's ADR 008 Q33 answer, "The backend should only hold model-relevant information". - The two agree for derived model state, such as the shape cache and maintained - aggregates, and both exclude UI state. "A mix of caching and a pull model to expand the - cached data" fits question 13's plain (b), where clients pull on a notice, and question - 14's shape cache. It does not decide question 13 or 14, and does not say whether a - per-connection topic counts as backend state. -- **String-value column widths are maintained by the host** (question 12). That - narrows, for those columns, ADR 008's decided width rule: (b), sticky per session with - hysteresis, over the rows realized. ADR 008 should record it. Question 27. +**Order.** Each step is a ticket and a PR in the implementation epic, and none switches +the displayed path until the differential test holds: +1. the `zset` crate, with its laws, operator properties and determinism test; +2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`; +3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a + signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty; +4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and + occupancy) under the differential test, with step and rebuild costs measured at ADR + 008's fixture. ADR 008's pruning depends on this step; +5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's + aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price` + and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members` + depends on this step; +6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row + overlay; +7. live sources as delta streams, once ADR 003's source trait exists; +8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one + header and no client refetches; a dropped header is detected; a reconnect after a host + restart refetches and drops nothing as a duplicate; a refused frame consumes no + publication number; and a dropped connection forces a refetch. ## Open questions -Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered -questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10-05, -2026-10-06 and 2026-10-09 revisions, and none is renumbered. +1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page + categories, every page's state is maintained and `[`/`]` only reads another slice. The + held key stays O(visible), and the state stays bounded by the stored cells. If it does + not, a page change builds operators like any other new key. *Recommendation:* the + slice, which clause 7 assumes. -1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? - *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not - carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. +2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder, + since a reorder rebuilds nothing. By item identity with the order looked up at read + time, which makes reads O(d). Or by an order label: a key that compares in the current + order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder + re-keys every entry stored under the moved items. *Recommendation:* the order label, + with only the drawn members ordered. Not measured. -2. *[Answered: a seam crate.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on - `zset` and `improvise-core`), or inside improvise-core? - *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and - `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. +3. **Should notices be filtered per client?** Broadcast (clause 12) keeps no per-client state, + but every client receives every live shape's moved keys. The options: + - **topics:** a client receives only its own shape key's key lists. Headers still go to + everyone, so the refetch rule keeps working. That costs one subscription per + connection, registered atomically with the fetch that names it, and pinning its + shape; + - **window push:** per-client subscriptions holding the window, sending only the + window's deltas. That costs the most host state. -3. *[Answered: (c), the exact accumulator.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums - until then. - - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in - the differential test. - - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by - **both** backends, so that incremental equals eager bit for bit. + Is a per-connection subscription "model-relevant information" the backend may hold, + or UI state it must not? It need not live in the host at all if the connection + operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt + topics only if measured traffic, with many clients on different shapes, justifies + them. - *Recommendation:* (c). `ExactSum` lives in `numtower`, wrapped by the seam crate, - as the user decided on ADR 005 question 15. It is measured bit-equal after a million - retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with - and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and - its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and - typed values, and it composes with (c): under ADR 005, a decimal sum is exact - through ADR 005's own wide accumulator (its question 10), provided both backends use - it, exactly as (c) requires for `ExactSum`. `rust_decimal`'s own `+` is not - associative at its bound, so decimals alone are not enough. **The costs to accept:** - - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. - - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it - would use the same accumulator. A whole-grid eager projection pays it once per - summed cell. - - Today's displayed sums change in their last bits, because they become correctly - rounded. - - **Signed zero:** the accumulator reads `[−0.0]` as `+0.0`, where the f64 `sum()` - that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - - `Avg`'s one rounding must be stated in step 6. +4. **How long does a shape key live?** A key is evicted once no recent request names it. + Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades + memory against refetch storms. If the connection operator re-requests the shapes its + clients read, that is a keep-alive by another name: one host `project` per shape per + eviction period. *Recommendation:* evict after a period with no request, set by + measurement, with hysteresis for shapes with many readers. -4. *[Answered: reproduce it for testing, and remove it eventually; question 25.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported - `circular` today (core.md §3). Should `Incremental` reproduce that? - *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to - change, change it in both backends at once, as its own decision. +5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs + with different payloads differ, but the wire spells every NaN `"nan"`. A remote client + would therefore see such a change as nothing. *Recommendation:* outputs consolidate + under the wire's identity (every NaN one value), and the record tier keeps bit + identity. -5. *[Answered: yes, for validation, and plan a DBSP-native way; question 25.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by - today's bounded fixed point, restricted to their component, rather than by any - operator in the crate? - *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the - cells a guarded cycle leaves convergent. The crate needs no recursion operator. +6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and + column aggregates by items, reports zero crossings per step, and slices rebuilds. That + makes collapse, hide, reorder and add item free, at the cost of recombining a + collapsed group's members on read. *Recommendation:* adopt all three, as written. -6. *[Answered: for now; the goal is DBSP formulas only; question 26.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? - *Recommendation:* yes, permanently. It is the differential test's oracle and the - backend for ADR 003's query-only sources. Deleting it would delete the test. +7. **Resident state.** Should the first version keep no second copy of the store (the + store is the integral), put operator state behind one trace interface, and measure + state at ADR 008's fixture before the store-memory work begins? *Recommendation:* + yes. -7. *[Answered in principle (the author's reading: the slice); see "What the user decided". Earlier note: ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so - switching page reads existing state, rather than rebuilding the view's operators per - page change? - *Recommendation:* the slice. The state stays bounded by the stored cells, and the - held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this - recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". +8. **Formulas the per-record map cannot express.** They take the occupied-contexts + fallback (clause 5). + - (a) Should `Eager` use the same enumeration, so the two backends agree? That changes + eager's answer only at the bare-ref edge, which could also be kept by checking the + ref once before enumerating. + - (b) Should formulas targeting a category other than `_Measure` take the same + fallback, so that nothing still needs `Expansion::members`? -8. *[Answered: "Eager should be deprecated", which overrides the recommendation; question 26.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, - with query-only sources kept on `Eager` and ADR 003's pushdown? - *Recommendation:* yes, chosen per document. Integrated state is proportional to the - source, which contradicts ADR 003's larger-than-memory goal for exactly the sources - that cannot report their changes anyway. + *Recommendation:* yes to both. -9. *[Answered: the recommendation, hedged "I think". Clause 14 also weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` - snapshot that `Host::mutating_frame` already takes? - *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying - `−d` restores integrated values but not the history `delay` and `differentiate` - hold, so it is not a rewind even with exact aggregators (clause 11). The cost is a - second O(cells) clone per mutating frame, beside the workbook clone that exists - today. A true rewind would need its own design and an observational-equivalence test - over every stateful operator. +9. **Which side of the wire prunes and fills?** On the client, the model offers only + windows and the restricted successor query, and the fill is the client's, with no host + state. On the model, prune is a request parameter, and the model keeps a pruned axis, + with rank and select for "row i of n", and runs the fill as a sliced job resumable + from (request, fill front). At a density of 10⁻⁴, a client without the successor query + would ship mostly empty windows. ADR 008 asks the same question. *Recommendation:* the + client. -10. *[Answered: no; a new epic, and felo.4 records the ADR only.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps - 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by - steps 4, 5 and 6? - *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are - how it is met. +10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas + by position instead goes stale on every item insert. Identity costs rank and select + per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as + clause 12 assumes. -11. *[Open. The recommendation was revised on 2026-10-05 by the author, because clause 7 - now makes the drawn order a read-time parameter; the user has answered neither - version.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and - improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering - category's item position, and rebuild on a reorder? Or should they key by stable item - identity (ADR 007), and look the order up at read time? - *Recommendation (revised):* neither as first stated. Key each entry by an **order - label**: a key that compares items in their current order and is not renumbered when - an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's - question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator - serves all four. An insert or a reorder relabels only the items that moved, and - rebuilds nothing. **But each relabelled item re-keys every group entry stored under - it**, so a reorder costs O(entries under the moved items), not O(moved items). This - is not measured. `LAST`/`FIRST` order only the values the fold - covers, which are the drawn members. - *As first written* (superseded): key by item position, relying on `view_shape`'s - rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn - order, so that rebuild does not happen, and a position key would go stale. +11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers + the restricted successor query. Per-axis counts can only skip rows that are empty + everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned + extent. *Recommendation:* keep them in that role. Not measured. -12. *[Answered: yes, for string-value columns; the recommendation below is overridden; question 27.]* **Incremental max width** (clause 13; the user's question on ADR 008). - Should the host maintain each column's whole-column maximum display width? That is - ADR 008's question 2 option (c), and its state would now be a small histogram per - column. *Recommendation (the author's):* no, not now. The user decided on (b) with - hysteresis, and under (b) a window's widths cost O(visible) and need no maintenance. - The maintained form also needs fold values maintained per column, and formatting on - the write path. Revisit it if a measured workload shows jitter that hysteresis does - not cure. +12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at + O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape + key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per + distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at + read time. Memoise per hidden set only if a measured hidden set is too large for the + frame budget. -13. *[New; now this ADR's to carry, by the user's "defer this to ADR 009" on ADR 008's - question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: - this ADR.** The options are: - - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the publication stamp, axis - changes, display relabels, a formula or `measure_agg` flag, and the moved output - keys per live shape key. The alternative is moved records plus a conservative rule - for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - - (b′) the same notices, delivered by topic. Every batch's header (stamp, - `document_generation`, axis changes, relabels, flags, eviction list) goes to every - client, with the host instance in its stamp. Only the moved-key lists are filtered - by shape key. A topic subscription lives exactly as long as its connection, and pins - its shape. It is registered atomically with the fetch that names its topic. It costs - O(N) headers per batch and per-connection state, which holds no UI state. **Question - for the user:** is a topic subscription "model-relevant information", in the sense - of your answer to ADR 008's Q33 ("The backend should only hold model-relevant - information")? (b′) depends on the answer. The user's consideration C13, a - connection operator that owns both ends, would move topic subscriptions out of the - host, so they stop being host state (clause 12, question 28); - - (c) push the model's data changes through soft-state subscriptions; - - (d) today's per-mirror push, which is obsolete. +13. **Which read-time parameters travel in a request?** The page selection travels, as a + third axis. Hidden items and collapsed groups change model-side totals and folds, so + without them a client could not get those from the model. Alternatively, a collapse + could be expressed as an axis at the group's level and a hide as a filter exclusion, + but both change the shape key, which costs a build per state. The drawn order, the + number format and prune change no aggregate. *Recommendation:* hidden items and + collapsed groups travel; the rest stay with the client. - *Recommendation (the author's, revised again):* **plain (b) with a size cap**, which - keeps no per-client state, with (b′) as a measured optimisation and (c) beyond it. The - simpler choice is the user's to prefer. (b′) buys traffic savings at the price of - connection-tied topics, pins, atomic registration and per-connection state against - Q33. The records-only variant of (b) is simpler still, and refetches more. Under every - option, `GridExtent` is derived by the client (in records mode it is fetched), `Dirty` - is retired with the mirrors, and the occupancy delta stays model-side, feeding - question 22's index. *Earlier* (superseded): (b), then (b′). - -14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state - selected by a request's **shape key** (axes by level, None categories, filter keys, - page categories, `measure_agg`), cached and evicted, with every other parameter - applied at read time (the author's)? Or is it kept per saved `.improv` view only, so - that a request with an unsaved key is served without incremental state? - *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while **and no live topic subscription pins it** (question 13's (b′)). An eviction - is announced in the next notice header, so a subscriber refetches and is never left - with a window that looks current and is not. The time is an implementation number. An unsaved collapse, reorder, page - step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key too. Under question 23's default (A) - its totals subtract the hidden rows at read time and add nothing to the cache. Under - (B′), it would add a **totals entry** keyed by (shape key, hidden row-axis set). Then - the cache would hold up to one entry per distinct hidden set that recent requests - name, and a client that toggles hides through many sets would pay to derive each new - entry. Only an - unsaved change of axes, filters or `measure_agg` builds a new key, as a job. - *Under C13* (the user's consideration, not decided), pins move to a connection - operator, which keeps shapes alive by re-requesting them. The host's rule is then only - "evict once no recent request names it" (clause 12). - -15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate - under the wire's identity, where every NaN is one value, while the record tier keeps - bit identity? *Recommendation:* yes. Otherwise a remote consumer and an in-process - one disagree when only a NaN's payload changes. - -16. *[New.]* **ADR 008's three requests** (ADR 008, "What revision 4 asks of ADR 009"): - occupancy and the per-column aggregates kept at item grain, a crossing count per - step, and rebuilds as steppable jobs. *Recommendation:* adopt all three, as clause 7 - is now written. This changes clause 7's list of what rebuilds, removes the rebuild - from gotcha 43's worked case, and undercuts question 11's premise. - -17. *[New.]* **Resident state** (clause 14). Should this ADR withdraw "spilling to disk" - as a non-goal, keep no second copy of the store (the store is the integral), and put - operator state behind one interface that the store-memory ADR can page? - *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the - order the user's answer to ADR 008's question 17 already sets. - -18. *[New; premise corrected on 2026-10-05.]* **Formulas the per-record map cannot - express** (clause 5a). ADR 008's question 15 sends them to a fallback over the - occupied contexts, never to `Expansion::members`; the user's answer was "the - recommendation is fine.". *As first written*, this question said the fallback changes - `Eager`'s answers wherever the expression is non-empty at an empty context. **The code - says it does not, except at one edge.** At `50e1314d`, `agg_values` evaluates the - expression only over the `member_records` of each context (`types.rs:91-139`; - `member_records` at `:47-53`), so a context that holds no record contributes no - value. The edge is an inner bare ref naming no item. It errors for every context it - is called with, so over zero occupied contexts it yields the empty value instead of - `ref:` (clause 5a; ADR 008 revision 25). And today's eager path - reaches `Expansion::members` only at a fold or a total (`contexts_of`, - `types.rs:916-925`). At a stem it pools one context. Two parts remain: - - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be - a cost change with bit-equal answers, except at the ref edge, since dropping empty - contexts keeps the order of the rest. The differential test then carves out only - that edge, by name. - - Should the formulas that target a category other than `_Measure` take the same - fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - - *Recommendation:* yes to both. The only change of behaviour is the ref edge, which - could also be kept by checking the ref once before enumerating. This was argued from - the code and not run. The differential test pins it. - -19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side - concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** - (clause 12). **Owner: ADR 008** (its question 57), because it is a question about the - view/host split. This question records what each answer asks of the circuit. On the - model side, the circuit keeps a pruned axis per shape key, with rank and select. On - the client side, it answers question 22's restricted successor query, and the client - prunes. *Recommendation (the author's), revised after C6 to C8:* the client side. The - model offers cross-product windows and the restricted successor query, so a sparse - grid still never ships empty space. *As first written*, this recommended the model - side; that is withdrawn. - -20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR**, for the - delta keys and the form of the top-left both, as ADR 008 revision 25 proposes. Its - question 58 points here. A - cross-product position stays fixed when occupancy changes, but it moves when an item - is inserted, removed or reordered. Should deltas be keyed by tuple identity, with - absolute positions attached when they are sent? And should an item edit be sent to - every subscriber of the key as an axis change, so that clients re-derive their - positions arithmetically? *Recommendation:* yes. It costs rank and select over each - category's items (ADR 008 clause 10), at O(levels × log items) per lookup, which is - not measured. It is compatible with ADR 008's question 58 recommendation, a tuple of - item ids as the primary form of the top-left. Item grain (clause 7) serves either - answer to whether hidden items belong to the absolute space, because hiding is a - remap. - -21. *[New.]* **How a pruned answer streams, and the fill step** (clause 12; C5). **Owner: - this ADR**, as ADR 008 revision 25 proposes. Its questions 59 and 66 point here. That - includes when a fill stops, and in what order rows and columns fill. *Recommendation:* under client-side pruning the - fill is the client's, one restricted successor query per step, and the host keeps no - fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, - so a fill and live updates are one stream. Under model-side pruning the same holds, - with the fill a sliced host job resumable from `(request, fill front)`. - -22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause - 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. - Four parts, with owners: - **Owner: this ADR** for the whole question, as ADR 008 revision 25 proposes (its - questions 61, 62 and 65 point here). Part (c) is the exception: it restates answers - ADR 008 records, so its text can only change there. - - (a) Should the - circuit offer a **restricted successor query**, "the next k rows after absolute - position p with a value in at least one column of S", with S in the call, and the - same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, - (ii) a column index of occupied rows, merged, or (iii) a row index of occupied - columns? Under C9 it only ever looks forward. - - (b) Should clause 9's whole-extent, per-axis occupancy be restated as index - (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it - only to skip. That skips only rows empty everywhere, not rows empty in S, so on its - own it does not answer (a). - - (c) Do ADR 008's question 1 ("go with occupied") and question 11 - ("prune empty should be the default") need restating? Question 1 would become the - rule for which cells of a window hold a value, and question 11 a client's display - default. - - (d) What happens - when scrolling up or left, and when a requested top-left lands on an empty row - (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It - probes with the restricted successor query, and it needs responses to report actual - positions. It costs about 14 to 28 probes at density 10^-4, by arithmetic. The - earlier fail, snap-forward and direction-snap options are kept there as superseded, - with the user's objection to snapping forward (C11). - - *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms - that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the - gallop as the baseline, and a predecessor query only as an optional optimisation, - offered once a measured round-trip cost justifies it. - -23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column - totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). Under ADR 008's 2026-10-06 - answers (Q46 to Q50), a hide is a client's UI parameter until an explicit `SaveView`. - So this question concerns the hidden set a request carries, and it assumes that set - travels (question 24). The options: - - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, - because the shape key is shared across clients and an unsaved hide belongs to one - client's request. - - **(A)** keep each total over all rows, and subtract the request's H hidden cells in - that column when it is read, as a temporary overlay. It costs O(H log d) per read - for every aggregate. `Count` subtracts, `Min` and `Max` skip fully hidden keys from - the extreme, and the text rule counts the distinct keys left. It keeps no per-set - state. `Sum` is exact only under question 3's (c) or ADR 005's accumulator. - Clause 5a's totals need their entries indexed by row tuple too. - - **(B′)** keep totals per (shape key, hidden row-axis set). Reads are O(1), but the - cache multiplies by the distinct hidden sets in use. Each entry costs at most one - O(cells) rebuild, or one O(H) derivation by (A)'s subtraction. That makes (B′) a - memo on (A). - - *Recommendation (the author's, re-evaluated on 2026-10-05):* **(A)**, with (B′) kept - only as a memo for a hidden set so large that its reads would not fit the frame - budget. It is the stateless option, which C2 asks for, and a typical hidden set is - small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on - an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of - `ad60ad2b` found. None of these costs is measured. - -24. *[New.]* **Which read-time parameters must travel in a request** (clause 7; the user - on ADR 008's Q51: "page is a third axis, do the others need to actually travel across - the wire?"). **Owner: ADR 008** (its question 51). This question states only what the - circuit needs. The page selection travels as the third axis. A **hide** changes - model-side aggregates (the column totals, and a collapsed group's fold), and so does a - **collapse** (a fold row's value). The user: "Aggregation should always be done - model-side for consistency" (Q38). So if hidden items and collapsed groups did not - travel, a client could not get its totals and folds from the model. It would have to - compute them itself, against Q38, or show totals over all rows. **Alternative forms**, - which ADR 008's Q51 names and which are closer to the user's lean that only the axes - travel: a **collapse** as an axis at the group's level (the request's row axis names - the group level instead of the items), and a **hide** as a filter-key exclusion. Both - are already in the shape key's vocabulary (clause 7), so either form gives the model - what it needs without a separate parameter. **But both change the shape key.** A hide - as a filter-key exclusion makes each distinct hidden set its own key, at an O(cells) - rebuild per set, and it loses question 23's (A), where every hidden set shares one - key and is subtracted at read time. A collapse as an axis level does the same for - each collapse state. The **drawn order**, - **number format** and **prune** (under client-side pruning) change no aggregate, and - need not travel. *Recommendation (the author's):* hidden items and collapsed groups - travel with the request; the drawn order, number format and prune stay with the - client. Not decided. - -25. *[New.]* **What "DBSP should fix the cycle issue" means** (clause 6; the user's - answers to questions 4 and 5). Clause 6 argues that DBSP's recursion computes a least - fixed point, which improvise's circularity is not: `X = X + 1` over a stored 10 has - no fixed point, and the required answer is `Error("circular")`. A DBSP-native - treatment could be: +14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and + validation. The options: - (a) a recursion operator bounded at sixteen iterations, which writes `circular` - where it has not converged, so the bound stays part of the semantics; - - (b) structural detection: every cell in a cycle of the formula graph is `circular`, - which changes today's answer for guarded cycles such as `A = IF(x, B, 1)`; + where it has not converged and keeps today's answers; + - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles + such as `A = IF(x, B, 1)`; - (c) a monotone reformulation, where one exists. - *Recommendation (the author's):* (a) for the first DBSP-native step, because it - keeps today's answers, and the differential test can then retire the per-cell fixed - point. Not decided. + *Recommendation:* (a) first. -26. *[New.]* **What replaces `Eager`, and what "only DBSP formulas" means** (clause 2; - the user's answers to questions 6 and 8). Two parts: - - (a) Does "only DBSP formulas" mean every formula is evaluated through the circuit, - with the one per-cell evaluator inside it as clause 2 has it? Or that formulas are - compiled into operators, which clause 2 rejects as a second evaluator? - - (b) Once `Eager` is gone, what serves a query-only live source (clause 10), and the - formulas the per-record map cannot express (clause 5a)? And what is the - differential test's oracle? +15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle, + query-only live sources, and the formulas of question 8 that stay eager. + *Recommendation:* a small from-scratch reference implementation, kept in the test + crate, as the oracle; a pushdown backend behind the same trait for query-only + sources; and the fallback of question 8 as a circuit stage. - *Recommendation (the author's):* (a) the first reading, which keeps one evaluator. - (b) the occupied-contexts fallback becomes a circuit stage. A query-only source gets - its own pushdown backend behind clause 2's trait, not `Eager`. The differential test's - oracle becomes a small, slow, from-scratch reference implementation kept in the test - crate only. Not decided. +16. **Which columns get a host-maintained width?** Columns whose cells are text, with + numbers keeping the sticky width with hysteresis? And what of a column that holds + both? *Recommendation:* text cells get the maintained width, and the view layer takes + the larger of that and its own numeric width. -27. *[New.]* **Which columns are "string-value columns"** (clause 13; the user's answer - to question 12). Is it columns whose cells are text, so that numeric columns keep ADR - 008's (b) with hysteresis? And what does a column holding both text and numbers do? - *Recommendation (the author's):* a column whose drawn cells include any text keeps a - maintained width for its text cells. The view layer takes the larger of that and its - held numeric width. ADR 008 should record the split. Not decided. +17. **What exactly is the connection operator?** (clause 12) + - (a) Do its "both ends" mean both halves in one component, or one component on each + side of the wire? + - (b) Is it a new crate or process, or a role for existing code (the browser + `Worker`, the socket server's `Registry` and transport, in-process code in the TUI)? + - (c) Who owns keep-alive and resumption state, and is that backend state? + - (d) Should one operator serve many connections and subscribe to the host on their + behalf, so that the host sends O(operators) per batch, not O(clients)? That saving + needs one operator per host process serving many clients, so per-connection state + then lives inside a shared operator. -28. *[New.]* **The connection "operator" of C13 and C14** (clause 12). The user: "an - operator of some kind responsible for both ends of the connection between the - frontend and backend and handling keep alive / session resumption / etc", and "And it - asks as a sink/source for the DBSP graph part on the backend and a source/sink for the - client side" ("asks" read as "acts", flagged). Part (a) is read as answered by C14. - - (a) *Read as answered by C14:* it is a DBSP-graph-shaped adapter, a sink and source - toward the graph and a source and sink toward the client, not a DBSP operator inside - the circuit. Still open: does "both ends" mean both halves of the sink/source pair - inside one component, or a component on each side of the wire? - - (b) Is it a new crate or process, or a role existing code takes on? Its host-facing - half could live with `improvise-incremental`, its wire types in - `improvise-protocol`, and its client-facing half beside `improvise-protocol`'s - client fold, shared by the TUI, the browser and any CLI (design-principles §5). The - browser's `Worker` (`worker.rs:14-15` at `f0fb6880`) and the socket server's - `Registry` and transport (`registry.rs:88-89`) are the nearest code today. - - (c) Who owns keep-alive and resumption state: the operator's host-facing half, its - client-facing half, or both? And does it count as "backend" in the sense of the - user's Q33 answer? - - (d) Does it subscribe to the host's notices on behalf of its clients and fan them - out, so that the host sends O(operators) per batch? - - *Recommendation (the author's):* (a) one component per connection that owns both - halves, with the client half as a shared library each front end links. (b) the - host-facing half in or above `improvise-incremental`, wire types in - `improvise-protocol`, and the client half shared beside `ClientState`. (c) resumption - state in the host-facing half, keyed by a token the client holds, and treated as - delivery state, not backend UI state. (d) yes. Then question 13's choice moves to the - operator–client hop, and the host sends plain (b). Not decided. + *Recommendation:* (a) one operator per host process: a host half shared by its + connections, and a client half in every front end. (b) a role of the existing + transport crates, not a new process. (c) the host half holds resumption state, keyed + by a token the client holds, as delivery state, not UI state. (d) yes, for the socket + server. The TUI and the browser worker each serve one client, so there the operator is + a function call. From 1b2650822059e1c91b2986d68074e1d381f1b0fd Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 23:19:12 -0700 Subject: [PATCH 061/106] docs(numtower): NumSum keeps five things, not four (improvise-59nw.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/numtower.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/context/code/numtower.md b/context/code/numtower.md index 74941fc0..da7e844c 100644 --- a/context/code/numtower.md +++ b/context/code/numtower.md @@ -191,7 +191,7 @@ that, and today's f64 path would print `-0.00`. ## Sums -`NumSum` keeps four things, and every one is an abelian group, so the state is a function +`NumSum` keeps five things, and every one is an abelian group, so the state is a function of the multiset of terms and not of their order. | part | holds | width | From e4f915423c5947db606e0422aa406c95134750be Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 23:20:39 -0700 Subject: [PATCH 062/106] docs(adr): ADR 009 leaves the operator's shape to its open question and points recommended clauses at theirs (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 55 ++++++++++++++--------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e78b1398..e2c239ac 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -252,8 +252,8 @@ read-time parameters here. It keeps its one job, bumping `layout_version`. The c neither reads nor depends on that tier, which is a view fact. Only `document_generation` and the record tier matter to this ADR. -**Item grain.** Occupancy and per-column aggregates are keyed by the **items** of the -axis categories, never by a position. So collapsing, expanding, hiding, showing, +**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates +are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing, reordering and adding an axis item change no operator: the visible axis is a remap. ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. A collapsed group's value is combined from its members when read. That is exact @@ -349,7 +349,8 @@ money. ### 9. What ADR 008 gets -- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple. +- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple + (kept as a pre-filter and as the unpruned extent; see open question 11). Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. A fold row with a formula column but no stored cell under it, and the `Exact` case with nothing on None, are **not** occupied, on purpose. @@ -403,7 +404,8 @@ part of the output and is never state. So a request costs O(visible). **Positions are absolute cross-product positions**, in each category's own item order, and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when -occupancy changes. They do move when an item is inserted, removed or reordered, so: +occupancy changes. They do move when an item is inserted, removed or reordered, so the +recommended design (open question 10) is: - output elements are keyed by tuple identity (item ids under ADR 007), with the absolute position attached when sent; - an item edit is sent to every client as an **axis change** ("inserted at index j of @@ -426,11 +428,13 @@ O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed fro cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's whole state is its request and its last absolute position, so a resume is a seek. -**Scrolling up or left is a new request with a new top-left.** The new top-left is found -by galloping backwards in doubling jumps until a probe hits, then narrowing until one new -row is gained. Each probe is a restricted successor query bounded by the old top-left. -That is about 14 to 28 probes at density 10⁻⁴, and no special answer is needed when a -top-left lands on an empty row. Which side of the wire runs the prune and the fill is +**Scrolling up or left is a new request with a new top-left.** How the client finds that +top-left when rows are pruned is the recommendation of open question 9: it gallops +backwards in doubling jumps until a probe hits, then narrows until one new row is gained. +Each probe is a restricted successor query bounded by the old top-left. That is about 14 +to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an +empty row. The alternatives there are a model-side predecessor query, and failing the +fetch and retrying. Which side of the wire runs the prune and the fill is open question 9. **The wire properties**, which hold on each hop: @@ -483,10 +487,10 @@ refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, extent is the admitted record count, which the client fetches. Whether notices should be filtered per client is open question 3. -**A connection operator, at the wire boundary.** One component owns both ends of a -connection. Its host half is a sink for the circuit's results and notices and a source of -requests and model commands. Its client half is the mirror image: a source of results to -the client, and a sink for the client's requests. It owns: +**A connection operator, at the wire boundary.** A connection operator sits at the wire +boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for +results and notices and a source of requests and model commands. Toward the client it is +the mirror image. It owns: - serialization (T1, T2); - stamps and the refetch rule (T3); - coalescing (T5); @@ -495,12 +499,11 @@ the client, and a sink for the client's requests. It owns: batches"; - session resumption. -In process, both halves are function calls. That covers the TUI, and the browser's -`Worker`, which already owns one `Host` and one port (`worker.rs:14-15`). The host half -belongs in or above `improvise-incremental`, and the wire types in `improvise-protocol`. -`zset` stays free of serde. The client half is shared by every front end, beside -`improvise-protocol`'s `ClientState`, so that a new front end does not write it again. -Its remaining shape is open question 17. +In process it is function calls, as in the TUI and in the browser's `Worker`, which +already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in +`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one +component per connection, or a host half shared per host process with a client half in +every front end), and who owns resumption state, are open question 17. ### 13. Column widths for text @@ -610,7 +613,10 @@ the displayed path until the differential test holds: re-keys every entry stored under the moved items. *Recommendation:* the order label, with only the drawn members ordered. Not measured. -3. **Should notices be filtered per client?** Broadcast (clause 12) keeps no per-client state, +3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed + stream. The rejected alternative is to send no notice at all and let each client + refetch on its own schedule: it keeps the host silent, but another client's change + stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state, but every client receives every live shape's moved keys. The options: - **topics:** a client receives only its own shape key's key lists. Headers still go to everyone, so the refetch rule keeps working. That costs one subscription per @@ -663,8 +669,13 @@ the displayed path until the differential test holds: state. On the model, prune is a request parameter, and the model keeps a pruned axis, with rank and select for "row i of n", and runs the fill as a sliced job resumable from (request, fill front). At a density of 10⁻⁴, a client without the successor query - would ship mostly empty windows. ADR 008 asks the same question. *Recommendation:* the - client. + would ship mostly empty windows. A second part: when scrolling up or left, how does the + client find a new top-left when rows are pruned? The options are galloping backwards + with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a + model-side predecessor query (one round trip, no new state, but a second query + direction), or failing the fetch and retrying with the nearest occupied row named. ADR + 008 asks the same question. *Recommendation:* the client prunes and fills, and it + gallops. 10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas by position instead goes stale on every item insert. Identity costs rank and select From 1190c80efe24d005209e1fcddf97c34327b1c38d Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Thu, 8 Oct 2026 23:22:01 -0700 Subject: [PATCH 063/106] docs(adr): tag every ADR 007 change to accepted text as [Amendment] (improvise-cq1q.1) The Status now says the tag is the complete list of proposed rules, and its table gains the regression test's step 3. Clause 12's test keeps the accepted "commit the unedited text" with the amendment's edit-a-non-name step tagged; the outside-_Measure real-item rule, the formula-collision example and the East/East_ collision sentences are tagged; the "exactly one id" sentence in clause 2 is restored; "other entities" loses the added "virtual"; the title drops "and groups". core.md section 20 and the id.rs test evidence cite ADR clauses instead of answer numbers. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 65 ++++++++++++++-------------- context/code/core.md | 20 +++++---- context/test-evidence/core.md | 15 ++++--- 3 files changed, 52 insertions(+), 48 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 8f9ec480..79c29a4a 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -1,4 +1,4 @@ -# ADR 007 — Stable identities for categories, items, formulas, views and groups +# ADR 007 — Stable identities for categories, items, formulas and views **Ticket:** improvise-sboz (implementation epic improvise-cq1q; this amendment improvise-cq1q.1) · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's @@ -9,8 +9,9 @@ draft call this document "ADR 005". ## Status **Accepted** on 2026-09-25 (PR #210). **An amendment is proposed and not yet -approved** (PR #326). The amendment's text is marked **[Amendment]** wherever it -appears. Everything without that mark is accepted and binding. +approved** (PR #326). **Every rule tagged [Amendment] is proposed; everything +without the tag is accepted and binding.** The tag is the complete list of what the +amendment adds. Where the amendment replaces accepted text, the accepted rule binds until the amendment is approved: @@ -23,15 +24,9 @@ amendment is approved: | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | a run of U+0020 spaces becomes one `_`; only a leading `_` is removed | each whitespace character becomes `_`, runs of `_` collapse after the strip, and leading and trailing `_` and `-` are removed | | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | +| 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | -The amendment also adds rules where the accepted text says nothing: -- a measure has an id and a display; -- the axis-line/`format:` order; -- `ViewState` and `Workbook` key the active view by id; -- CheckedBatch's created-record exemption is removed; -- two branches of one file can merge into a file that will not load. - -Each is marked where it appears. Eleven questions are open (last section). +Eleven questions are open (last section). ## Context @@ -84,15 +79,15 @@ human name moves to a new `display` field. **An id always names the same thing, so ids are unique in the whole document**, across every kind. An item id names one item without its category. A `CellKey` still pairs each item with its category, and an operation that creates an item still names the -parent category. A formula whose generated id would collide takes a suffix like any -other generated id (clause 4). A formula `Tax` in `CustomCat` gets item id `tax`, then -formula id `tax-`. **[Amendment]** Formulas may refer to anything in a typed -context. What exactly that allows is open question 2. +parent category. **[Amendment]** A formula whose generated id would collide takes a +suffix like any other generated id (clause 4): a formula `Tax` in `CustomCat` gets +item id `tax`, then formula id `tax-`. **[Amendment]** Formulas may refer to +anything in a typed context. What exactly that allows is open question 2. **Reserved ids.** The virtual categories have ids `_index`, `_dim` and `_measure`, displayed `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** **[Amendment]** A leading `_` is reserved for virtual categories, measures and other -virtual entities. The virtual views have ids `_records` and `_drill`, displayed +entities. The virtual views have ids `_records` and `_drill`, displayed `_Records` and `_Drill`. They are never persisted, but they are addressed on the wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could collide with a user's view. Which entities the reservation covers is open question 6. Whether @@ -102,8 +97,8 @@ Moving today's capitalised keys to these ids is open question 8. **Formula targets.** **[Amendment]** A measure is the normal formula target (`profit = revenue - cost`). A dimension (a category other than `_Measure`) is a rare one, for a derived dimension. -- **A target outside `_Measure` is a real item**, with its own `ItemId`, and its - formula has a separate `FormulaId`. +- **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, + and its formula has a separate `FormulaId`. - **A `_Measure` target has no item of its own**, and its handle is its formula's `FormulaId`. So the category panel's `_Measure` rows use `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). @@ -141,8 +136,9 @@ callers, the evaluator's `Expr::Ref` lookup and the `View` methods. Display→id resolution moves to the places a human types. The field rename itself is cheap. The change is mechanical but wide, and it is the bulk of the implementation. -**Display names stay unique within their scope**, as names are today, and a rename to -a display already in use there is refused. An item's scope is its category, so two +**Display names stay unique within their scope**, as names are today. So a +human-typed name resolves to exactly one id within its scope, and a rename to a +display already in use there is refused. An item's scope is its category, so two categories may each have an item displayed `Revenue`. Formula text crosses categories, so a display may be ambiguous there. The ids differ, so the formula panel shows the ids where a display is ambiguous. @@ -212,8 +208,9 @@ Examples under the amendment: | `Straße` | `strasse` | | `Ü`, `-`, `---` | `-` | -**Collisions.** `Time Period` and `Time_Period` slug alike, and so do `East`, `East ` -and `East_`. Whichever comes later takes a suffix. `East` in `Region` and `East` in +**Collisions.** `Time Period` and `Time_Period` slug alike, and whichever comes later +takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, which slug alike +only under the amendment's steps (under the accepted steps `East ` slugs to `east_`). `East` in `Region` and `East` in `Direction` also slug alike, and ids are global, so the second is `east-`. - **Any colliding generated id gets a suffix**, whatever kind of thing holds the slug: on import, when an id-less file loads, and when the host creates something in @@ -221,8 +218,8 @@ and `East_`. Whichever comes later takes a suffix. `East` in `Region` and `East` - **The suffix is `-` plus the next value of one increment-only counter per document**, as a plain number (`east-1`, not `east-001`). The counter works like Common Lisp's `gensym`: it never goes down and is never reused, so no order holds - among suffixes (`east-7` may follow `east` directly). A suffix ends in a digit, so a - suffixed id still meets step 5. + among suffixes (`east-7` may follow `east` directly). **[Amendment]** A suffix ends + in a digit, so a suffixed id still meets the amendment's step 5. - **Retirement.** Within a session a removed id is never handed out again. The counter is past any suffix it issued, and a removed bare slug stays reserved in memory. @@ -589,12 +586,16 @@ without being told. That is why the panel shows B as `east-1` and never as a bar 1. Open a formula editor on `Profit = Revenue - Cost`. 2. From another client, rename `Revenue` to `Sales`, and rename a different item to `Revenue`. -3. Edit something that is not a name (`Revenue - Cost * 1`) and commit. +3. Commit the unedited text. The stored formula must reference the original item's id, and the commit must not be refused. The test must fail under commit-time resolution and under a refusal on -rename. Committing the *unedited* text proves nothing, because a client may treat it -as a no-op without resolving any name. +rename. + +**[Amendment]** Step 3 becomes: edit something that is not a name +(`Revenue - Cost * 1`) and commit. Committing the unedited text cannot tell a correct +capture from no capture at all, because a client may treat unchanged text as a no-op +without resolving any name (improvise-hexb). **[Amendment] Required test for ids-first.** With items A (`East`, `east`) and B (`east`, `east-1`), type `east` into an editor, in text not captured at open: @@ -773,11 +774,11 @@ zero-width character still separates the two `_`, and the result is `time__perio ### 6. Which entities share the reserved leading `_`? **The question.** A leading `_` is reserved for virtual categories, measures and other -virtual entities. -- (i) No virtual measure exists today. Is one planned, or does "measures" mean the - `_measure` category? -- (ii) Are the virtual views, `_records` and `_drill` (clause 1), among the reserved - entities? +entities. +- (i) Does "measures" mean virtual measures (none exists today), the `_measure` + category, or something else? +- (ii) Which "other entities" are covered? In particular, are the virtual views, + `_records` and `_drill` (clause 1), among them? **Why it matters.** It fixes `VIRTUAL_IDS` and what `claim` refuses. diff --git a/context/code/core.md b/context/code/core.md index 04cbdf89..ffcee2e4 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1691,12 +1691,12 @@ probability (`> ||[||]` is a self-parent), so `generated_file_parses` accepts a `slugify` and `IdAllocator` are the first piece of ADR 007. They are pure functions of the display and the allocator's own history. Nothing calls them yet: the creation sites move onto them in the re-key step (improvise-cq1q.4). -Each rule traces to a numbered answer of the user's. The ADR no longer carries -them; they are listed, with comment ids, in PR #326's provenance comment. +Each rule traces to ADR 007's clause 4 (slugify, the counter, hand-written +ids) or clause 1 (reserved ids). **Case folding is full Unicode folding, and it comes from `unicase`.** `str::to_lowercase` is not folding: `ß` lowercases to itself and is then -stripped, so `Straße` would slug to `strae` (answer 26). The same goes for `ſ` +stripped, so `Straße` would slug to `strae` (clause 4, step 1: full case folding). The same goes for `ſ` (long s) and the `fi` ligature. `unicase` 2.9 is `no_std` and has no dependencies, so it passes the wasm gate. Its `to_folded_case` applies the C+F mappings of Unicode's CaseFolding.txt, and its own tests pin `Maße` → @@ -1705,8 +1705,8 @@ letter survives step 3. Deriving that ASCII-producing subset of the table by hand was rejected. It is a second copy of the Unicode data, and nothing would say when it drifted. -**A "space" is U+0020.** Answer 18 says "replace space with `_`", and answer 21 -says runs collapse. A tab or a no-break space is not a space here, so it is +**A "space" is U+0020.** The accepted clause 4 says "replace each run of spaces +with one `_`". A tab or a no-break space is not a space here, so it is stripped by step 3. Only a run of U+0020 becomes one `_`. An underscore the display already had is never merged with the one a space run makes. So `a_ b` slugs to `a__b`: the steps are applied as written, and no extra @@ -1732,7 +1732,8 @@ whole of retirement (clause 4, "within a session a removed id is never handed out again"). No release operation exists, so a removed id cannot be reissued by construction; no check refuses it. The counter starts at 1, is shared by every kind, and advances each time a suffix is tried, including a candidate -that turns out to be taken by a hand-written id (answers 28 and 29). So +that turns out to be taken by a hand-written id (clause 4: the `gensym` counter, and hand-written +ids are canonical). So `east-7` may follow `east` directly, and no order is enforced among suffixes. **The walk past taken candidates is a loop, never recursion.** A new-version file saved from a non-ASCII import holds `item-1` … `item-N`, and @@ -1741,16 +1742,17 @@ version recursed once per candidate, not in tail position, and aborted the process on a stack overflow: in debug at N ≈ 11,000, in release at 50,000. An abort, not a panic, so the TUI lost unsaved edits (Codex P2 on PR #318). An empty slug always takes a suffix (`category-1`), even when the bare kind -name happens to be free, because answer 16(e) spells the result +name happens to be free, because clause 4 spells the result `-`. **`claim` is the only door for a hand-written id.** It checks, in this order: the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading -`_` (only `_index`, `_dim` and `_measure` may carry one, per answer 23, and +`_` (only `_index`, `_dim` and `_measure` may carry one, per clause 1, and the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other id may start with `_`"), and duplicates. It accepts `-`, `-x` and `x-`. ADR 007's open question 11 asks -whether a hand-written id like that must be refused. It never renames. Refusing is the loader's job, and the loader +whether a hand-written id like that must be refused. It never renames. +Refusing is the loader's job, and the loader must name both lines of a duplicate; `ClaimError` carries only the id, so the loader keeps the line numbers. The virtual views `_records` and `_drill` are not yet in `VIRTUAL_IDS`. improvise-p003 item 1 asks for them, and they arrive diff --git a/context/test-evidence/core.md b/context/test-evidence/core.md index 354bceba..d4ae4c48 100644 --- a/context/test-evidence/core.md +++ b/context/test-evidence/core.md @@ -1935,14 +1935,14 @@ Rival: every space becomes `_`, with runs not collapsed. Only this test fails: ### model::id::tests::slugify_keeps_an_underscore_the_display_already_had -Rival: `.replace("__", "_")` after the space step, which reads answer 21 as -"no doubled underscores". Only this test fails: `left: "a_b"`. The rule +Rival: `.replace("__", "_")` after the space step, which reads the accepted +clause 4 as "no doubled underscores". Only this test fails: `left: "a_b"`. The rule collapses runs of *spaces*. An underscore that was in the display is kept. ### model::id::tests::the_suffix_is_the_next_counter_value_not_the_lowest_free_number -Rival: the lowest free `-` for each base, which revision 5 of the ADR -proposed and answer 28 replaced. Three tests fail: this one (`"west-1"` against +Rival: the lowest free `-` for each base, which an early draft of +ADR 007 proposed and clause 4's shared `gensym` counter replaced. Three tests fail: this one (`"west-1"` against `"west-2"`), `an_empty_slug_takes_its_kind_and_the_shared_counter` (`"item-1"`) and `a_slug_taken_by_another_kind_takes_a_suffix` (`"time_period-1"`). Only interleaving two bases separates one shared counter from per-base numbering, @@ -1952,8 +1952,9 @@ and this test is the one that does it on purpose. Rival: `claim` validates the id but does not record it. This test and `claim_refuses_a_duplicate_a_malformed_and_a_reserved_id` fail. The second -`East` is issued `east-1`, the file's own id, which is the collision answer 29 -forbids: "doing otherwise will corrupt the data section". +`East` is issued `east-1`, the file's own id, which is the collision clause 4 +forbids: hand-written ids are canonical, and renaming one would corrupt the +data section. ### model::id::tests::generated_ids_are_distinct_well_formed_and_never_reserved @@ -1990,7 +1991,7 @@ exempt any id that begins with a virtual id (`VIRTUAL_IDS.iter().any(|v| id.starts_with(v))`), all 11 earlier tests passed, because `claim_refuses_a_duplicate_a_malformed_and_a_reserved_id` tries only `_draft`. This test fails on that rival (`left: Ok(())` for `_indexer`). The match is -exact by the ADR's own wording: answer 23 says no other id may begin with `_`. +exact by the ADR's own wording: clause 1 says no other id may begin with `_`. ### model::id::tests::a_display_with_leading_spaces_never_slugs_into_the_reserved_space From 34ab0fc3d33309ea620219395c02047de6aa288e Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Thu, 8 Oct 2026 23:44:07 -0700 Subject: [PATCH 064/106] fix(numtower): past-K oracle and weighted NumSum properties, negative-power scale, ExactSum value equality (improvise-59nw.1) Answers the BLOCK on 302594bf (B1-B3) and its should-fix items: the past-K read and Z-set weights are now checked against a BigRational oracle (the review's F7, F8, F12, F14, F15 now fail by name), a negative power of a unit base has its minimal scale, ExactSum compares values, a forced n/d never reads as infinity, and Interim compares exactly and knows zero without storing. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/numtower.md | 31 ++- context/test-evidence/numtower.md | 148 ++++++++++- crates/numtower/src/arith.rs | 34 ++- crates/numtower/src/cmp.rs | 10 + crates/numtower/src/lang.rs | 10 +- crates/numtower/src/sum.rs | 15 +- crates/numtower/tests/properties.rs | 398 +++++++++++++++++++++++----- 7 files changed, 565 insertions(+), 81 deletions(-) diff --git a/context/code/numtower.md b/context/code/numtower.md index da7e844c..ea27d290 100644 --- a/context/code/numtower.md +++ b/context/code/numtower.md @@ -53,6 +53,17 @@ the stored bound. The operators on `&Num` are one-operation evaluations: each bu (`an_intermediate_beyond_i128_comes_back_exact`). That difference is (b)'s whole point. Step 3's evaluators should evaluate through `Interim` and store once per cell. +**So `Interim` compares, and knows zero, without storing.** `IF(a*a/a = a, …)` and the +division guard both run before the stored bound. If `=` had to store its operands first, +its answer would depend on how the formula is spelled, which is what question 13 (b) +rules out. `Interim::cmp_exact` is clause 6's exact order over every pairing: two stored +values go through `Num::cmp_exact`, and anything wide compares as a bigint rational, +against an f64 by its exact binary value. `Interim::is_zero` is the one zero predicate. A +wide value is never zero, because zero always fits a stored arm. The comparison through +storing is the rival (`an_interim_compares_exactly_without_storing_catches_storing_first`): +`a` and `a ± 1/(3·10^30)` store to one f64 and compare equal. The review of `302594bf` +asked for this or for a note naming it as a step-3 item; the crate does the first. + ### Contagion, and how many roundings a demotion has An inexact operand makes the result inexact. Each exact operand is converted by the @@ -116,7 +127,11 @@ rule, because the ADR decided it and because it needs no second division. ceiling's stated exception. - **The scale.** A positive power of a decimal takes clause 4's stored scale with nominal `s·n`, the scale repeated multiplication would give, so `1.50^2` is `2.2500`. A - negative power is a division and has its minimal scale. + negative power is a division and has its minimal scale, **unit bases included**: + `1.0^-2` is `1` and `(-1.0)^-3` is `-1`, as `2.0^-2` is `0.25`. Until the review of + `302594bf` the unit-base arm padded every power to `s·|n|`, so `1.0^-2` read `1.00` + (Copilot on PR #337). The power property now checks the scale of every decimal result + against this rule, with unit bases among its seeds. - **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`, including one too large to compute. - **A demoted power takes its sign from the exact exponent.** `powf` gets the base's @@ -171,6 +186,11 @@ checks that the two agree, and that the exact value read rounds to the f64 f64**. The ADR says `n/d` "is a fraction or, once normalized, a decimal", and does not say what a forced `n/d` too large for `i128` becomes. The crate applies clause 3's order to it, as it does to every other spelled number. This is a reading, and it is flagged. + **Except that the nearest f64 must be finite.** A forced `n/d` whose value rounds to + infinity (`n` of 400 digits, `/1`) is text, exactly as the gate makes `1e999` text. + Clause 3: "a finite lexeme must never read as infinity". One that underflows is the + inexact zero, as `1e-999` is. `is_number` and `parse_number` share the one helper, + `forced_fraction`, so the gate and the parse cannot disagree on it. **The canonical text is `Display`.** An inexact finite value is `#i` followed by Rust's `{:?}` spelling, which is the shortest that reads back (`#i1e300`, `#i0.1`). @@ -230,6 +250,15 @@ on PR #309. It has three changes: It counts NaN and the infinities, so they can be retracted. A count that is nonzero in either direction is treated as present. +**`ExactSum`'s `==` compares sums, not representations.** A derived equality compared the +raw limbs and the `pending` carry counter, so `0.5 + 0.5 != 1.0`, an insert then retract +was not equal to an empty sum, and the same weights added in another order compared +unequal (Copilot on PR #337). For a type whose whole point is order-freedom, and which +ADR 009's group newtype will wrap, that equality is a trap. It now compares the +propagated limbs, which are canonical (every limb but the top in `[0, 2^32)`), plus the +three special-value counts, and ignores `pending`. `NumSum` derives no equality: compare +its `read()`. + **For ADR 009's seam crate.** `NumSum` and `ExactSum` both have the shape a Z-set group needs: `add(&Num, i64)` / `add(f64, i64)`, `insert`, `retract`, `read`/`value`, `Clone`. `ExactSum` also has `absorb`, to merge two accumulators. Neither implements a group trait diff --git a/context/test-evidence/numtower.md b/context/test-evidence/numtower.md index 705879b8..451f5a0c 100644 --- a/context/test-evidence/numtower.md +++ b/context/test-evidence/numtower.md @@ -154,11 +154,25 @@ argument orders, the order-independence property the tie-breaks exist for. ### the_accumulator_is_independent_of_order -Random term lists, from every arm, are summed in one order and in a shuffled order. The -results must agree in **arm, value and scale**. The terms include fractions over a small -pool of denominators, so the exact grouped read is reached, and arbitrary fractions, so -the K cutoff is too. ADR 005 clause 8's seed case, `[0.30000000000000004, 1e12, −1e12]`, -is one of the generator's arms and is also pinned exactly. +Random **weighted** term lists, from every arm, are summed in one order and in a shuffled +order. The results must agree in **arm, value and scale**. The terms include fractions +over a pool of fourteen non-decimal denominators, so groups receive several terms and +lists often hold more than K of them, and arbitrary fractions too. ADR 005 clause 8's +seed case, `[0.30000000000000004, 1e12, −1e12]`, is one of the generator's arms and is +also pinned exactly. It caught the review's F12 (a group's old f64 not retracted), where +the stale values make the past-K read depend on the order of updates. + +**On weights** (the review of `302594bf`, B2). Until then every `NumSum` property called +`insert` and `retract` only, so every weight was ±1. That was the cause the answer to +Codex's P1 had itself named, and the generators had not changed. Two faults passed the +whole suite: F7 (the decimal partition ignored the weight's magnitude) and F8 (a group +multiplied by `weight.signum()`). Weights now come from `weight()`: +- the units 1, −1 and 0; +- −3 to 3; +- magnitudes up to 2^40; +- `±i64::MAX` and 2^30 − 1. + +`i64::MIN` is left out so a weight always negates. ### the_accumulator_is_independent_of_order_catches_decimal_addition @@ -168,23 +182,65 @@ scale 16 and the other does not. ### the_accumulator_retracts_exactly -Inserting `xs` and `ys`, then retracting `ys` in reverse order, must equal a fresh sum of -`xs` in arm, value and scale. A group whose numerator returns to zero must be dropped, or -a retracted denominator would still count toward K. +Adding weighted `xs` and `ys`, then adding each `y` again with its weight negated in +reverse order, must equal a fresh sum of `xs` in arm, value and scale. A group whose +numerator returns to zero must be dropped, or a retracted denominator would still count +toward K. With weights it catches F7 and F12 as well. ### the_accumulator_reads_the_exact_sum_or_its_stated_roundings -This is the differential test of the read against the oracle. It is the only test that -states every branch. +The differential test of the read against a `BigRational` oracle, over weighted terms. It +states the value of all three branches: - **No inexact term, at most K live groups.** The read is exact iff the exact total fits, and is then that total. Otherwise it is the total's correctly rounded f64. -- **Inexact terms present.** The exact total is rounded once, then the f64s are added and - the result is rounded once more. -- **Past K groups.** Inexact. +- **Inexact terms present.** The exact total is rounded once, then the weighted f64s are + added and the result is rounded once more. +- **Past K groups.** The nearest f64 of Σ round(live group) + round(decimal partition) + + Σ w·f. + +**Until the review of `302594bf` (B1) the past-K branch asserted only that the read was +inexact, and this entry claimed the test stated every branch. Both were wrong.** Three +faults passed the whole suite: +- F12: a group's old f64 was not retracted from `group_floats`; +- F14: the past-K read dropped the decimal partition; +- F15: the past-K read dropped every inexact term. + +The test now has the reviewer's oracle and 512 cases. Half the lists come from the +fourteen-denominator pool, up to 40 terms long, so groups receive several terms and K is +passed often. **It also counts how many cases reach each branch and fails if any branch +is reached fewer than 20 times**, so a generator change cannot silently starve one. The oracle counts **live** groups, those whose summed numerator is nonzero. The first version counted denominators ever seen, and that disagrees with the state-based count -whenever terms cancel. +whenever terms cancel. Inexact presence is likewise per bit pattern, by net weight. + +**Every fault in the review's script, re-run after the fix.** Each was injected into the +source and `cargo test -p numtower` was run on a confirmed rebuild. Each failed by name: + +| fault | failed | +|---|---| +| F1 decimal sum's scale filter dropped | `a_recomputed_result_keeps_the_stored_scale_clause_4_names`, `an_exact_result_is_never_a_rounded_one`, `contagion_is_total` | +| F2 `rust_decimal`'s `to_f64` | `exact_mixed_comparison_is_transitive`, `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `the_f64_conversion_agrees_with_parsing_the_decimal_text`, `the_number_languages_gate_and_parse_agree` | +| F3 mixed comparison through f64 | `exact_mixed_comparison_is_transitive`, `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `an_interim_compares_exactly_without_storing` | +| F4 one net inexact count | `the_accumulator_keeps_inexact_terms_whose_weights_cancel`, `the_accumulator_retracts_exactly`, this test | +| F5 product, not lcm | `sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once` | +| F6 `ExactSum` rounded per limb | `exact_sum_is_the_exact_weighted_total_rounded_once`, `more_than_k_denominators_read_inexact_even_when_the_total_fits`, `the_accumulator_keeps_inexact_terms_whose_weights_cancel`, this test | +| F7 decimal weight magnitude ignored | this test, `the_accumulator_retracts_exactly` | +| F8 group weight as its sign | this test | +| F9 decimal product's scale filter dropped | `a_typed_value_and_the_same_value_computed_are_equally_exact`, `an_exact_result_is_never_a_rounded_one`, `contagion_is_total` | +| F10 `ExactSum`'s low weight digit only | `exact_sum_is_the_exact_weighted_total_rounded_once`, this test | +| F11 `POW` sign lost | `a_huge_odd_exponent_keeps_the_sign_of_a_negative_base` | +| F12 old group f64 kept | this test, `the_accumulator_is_independent_of_order`, `the_accumulator_retracts_exactly` | +| F13 mixed tie not checked exactly | `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `exact_mixed_comparison_is_transitive`, `an_interim_compares_exactly_without_storing` | +| F14 past K, decimal partition dropped | this test | +| F15 past K, inexact terms dropped | this test | + +- **F13's text now occurs twice.** `Interim`'s comparison added a second copy of the + replaced line, so the review's script refuses to inject it. It was injected at its + original site, in `cmp.rs::mixed`. +- **The five former survivors were run twice and failed both times.** F12's set varied + between runs: `the_accumulator_retracts_exactly` failed in one run and not the other, + and this test failed in both. ### more_than_k_denominators_read_inexact_even_when_the_total_fits @@ -322,6 +378,70 @@ Allowing the value exposes a second hazard: it cannot be negated in `i128`, and `i128::MIN / 3` (an inexact, correctly rounded result), takes it through `Interim` twice and back, divides it, compares it and reads its canonical text. +### a_negative_power_has_its_minimal_scale + +Copilot on PR #337, and B3 of the review of `302594bf`. The unit-base arm of `POW` +padded every result to `s·|n|`, so `1.0^-2` was `1.00` and `(-1.0)^-3` was `-1.000`, +while `2.0^-2` was `0.25`. `context/code/numtower.md` already said "a negative power … +has its minimal scale", so the code contradicted the crate's stated rule. This test +**failed at `302594bf`** on its first assertion. It also pins that a positive power keeps +`s·n` (`(-1.0)^3 = -1.000`), so a fix that stripped every scale cannot pass. + +The broader guard is the scale assertion now in +`pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits`, which had checked +value and exactness but never scale. Its oracle, `power_scale`, is the rule restated: +- start from the minimal scale; +- pad toward `min(s·k, 28)` for a decimal base with `k > 0` only, while the mantissa + fits 96 bits. + +Unit bases (`1.0`, `-1.0`, `1.000`, `-1`, `0.00`) are among its seeds. It **failed at +`302594bf`** with the minimal case `1.0^-1 = 1.0`. The injection G1 (a negative power +keeps the base's scale) fails both tests. + +### exact_sums_are_equal_when_their_values_are + +Copilot on PR #337, and B3 of the review. `ExactSum` derived `PartialEq` over its raw +limbs and its `pending` carry counter. This test **failed at `302594bf`** on its first +assertion, `0.5 + 0.5 == 1.0`. It also asserts: +- insert then retract equals an empty sum; +- weights `(2^30 − 1, 2)` added in either order are equal; +- different values are unequal, and a counted NaN is part of the value, so an `==` that + always answers true cannot pass. + +Injection G2, equality back on the representation, fails it. + +### a_forced_fraction_never_reads_as_infinity + +A should-fix from the review of `302594bf`. A forced `n/d` with a 400-digit `n` read as +`Inexact(inf)`, though clause 3 says a finite lexeme never reads as infinity. This test +**failed at `302594bf`**. It also asserts: +- `is_number` agrees with the parse; +- a negative one is text too; +- an underflowing `1/9…9` is still the inexact zero, so a fix that refused every inexact + forced fraction cannot pass. + +Injection G3 fails it. + +### an_interim_compares_exactly_without_storing + +New API, from a should-fix in the review: `Interim` had no comparison and no zero test +while step 3 is told to evaluate through it. For two wide products `a`, it compares `a` +against: +- another wide product; +- `a ± 1/(3·10^30)`; +- `a` itself; +- the f64 nearest `a`, and that f64's upper neighbour. + +Each comparison is made in both directions, against the `BigRational` order. It also +asserts `a − a` is zero and `a` is not. Injection G4, storing before comparing, fails it, +and so do F3 and F13, because the stored arms delegate to `Num::cmp_exact`. + +### an_interim_compares_exactly_without_storing_catches_storing_first + +The rival is the comparison the review warned step 3 would otherwise have to make: store +both operands, then compare. `a` and `a ± 1/(3·10^30)` store to one f64, so the rival +calls them equal. + --- ## `crates/numtower/src/sum.rs` diff --git a/crates/numtower/src/arith.rs b/crates/numtower/src/arith.rs index c8e9dcaf..1b19a38b 100644 --- a/crates/numtower/src/arith.rs +++ b/crates/numtower/src/arith.rs @@ -69,6 +69,29 @@ impl Interim { pub fn pow(&self, o: &Interim) -> Result { power(self.operand(), o.operand()) } + + /// Exactly zero; a wide value never is, because zero fits a stored arm. + pub fn is_zero(&self) -> bool { + match &self.0 { + Inner::Stored(n) => n.is_zero(), + Inner::Wide(_) => false, + } + } + + /// Clause 6's exact order, before the stored bound; `None` only when either is NaN. + pub fn cmp_exact(&self, o: &Interim) -> Option { + match (&self.0, &o.0) { + (Inner::Stored(a), Inner::Stored(b)) => a.cmp_exact(b), + _ => match (self.operand().big(), o.operand().big()) { + (Some(a), Some(b)) => Some(a.cmp_value(&b)), + (Some(a), None) => crate::cmp::exact_against_f64(&a, o.operand().float()), + (None, Some(b)) => { + crate::cmp::exact_against_f64(&b, self.operand().float()).map(|c| c.reverse()) + } + (None, None) => self.operand().float().partial_cmp(&o.operand().float()), + }, + } + } } impl Num { @@ -364,10 +387,13 @@ fn exact_power(a: Operand, n: &BigInt) -> Result { _ => 0, }; let k = n.magnitude(); - let target = k - .to_u32() - .and_then(|k| k.checked_mul(base_scale)) - .map_or(MAX_SCALE, |s| s.min(MAX_SCALE)); + let target = match n.is_negative() { + true => 0, + false => k + .to_u32() + .and_then(|k| k.checked_mul(base_scale)) + .map_or(MAX_SCALE, |s| s.min(MAX_SCALE)), + }; let unit = q.den.is_one() && q.num.magnitude().is_one(); match (q.num.is_zero(), n.is_negative(), unit) { (_, _, _) if n.is_zero() => Ok(Interim::from(padded(1, 0, 0))), diff --git a/crates/numtower/src/cmp.rs b/crates/numtower/src/cmp.rs index 2466b017..81cac9b7 100644 --- a/crates/numtower/src/cmp.rs +++ b/crates/numtower/src/cmp.rs @@ -64,6 +64,16 @@ fn mixed(exact: &Num, x: f64) -> Option { } } +/// A wide exact value against an f64, with no fast path: the slow, total form of `mixed`. +pub(crate) fn exact_against_f64(q: &BigQ, x: f64) -> Option { + match (x.is_nan(), x.is_infinite(), x > 0.0) { + (true, _, _) => None, + (false, true, true) => Some(Ordering::Less), + (false, true, false) => Some(Ordering::Greater), + (false, false, _) => Some(q.cmp_value(&BigQ::from_f64(x))), + } +} + fn pick<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num { match (a.cmp_exact(b), is_nan(a), is_nan(b)) { (_, true, _) => b, diff --git a/crates/numtower/src/lang.rs b/crates/numtower/src/lang.rs index e2a7d666..b5d00960 100644 --- a/crates/numtower/src/lang.rs +++ b/crates/numtower/src/lang.rs @@ -19,18 +19,24 @@ pub enum Typing { /// The gate: the strings `f64::from_str` accepts and reads as a finite value. pub fn is_number(text: &str, typing: Typing) -> bool { - gate(text) || (typing == Typing::Forced && fraction_lexeme(text).is_some()) + gate(text) || (typing == Typing::Forced && forced_fraction(text).is_some()) } /// A number if the text is one, read exactly; `None` means the text is text. pub fn parse_number(text: &str, typing: Typing) -> Option { match (gate(text), typing) { (true, _) => Lexeme::read(text).map(|lx| lx.value(text)), - (false, Typing::Forced) => fraction_lexeme(text).map(|q| Num::settle(&q)), + (false, Typing::Forced) => forced_fraction(text), (false, Typing::Inferred) => None, } } +/// A forced `n/d`; one that rounds to infinity is text, as a finite lexeme never reads as one. +fn forced_fraction(text: &str) -> Option { + let n = Num::settle(&fraction_lexeme(text)?); + n.to_f64().is_finite().then_some(n) +} + fn gate(text: &str) -> bool { text.parse::().is_ok_and(f64::is_finite) } diff --git a/crates/numtower/src/sum.rs b/crates/numtower/src/sum.rs index b5271c1a..366edbb5 100644 --- a/crates/numtower/src/sum.rs +++ b/crates/numtower/src/sum.rs @@ -20,7 +20,7 @@ const LIMBS: usize = 70; const CHUNK: u64 = 1 << 30; /// An exact sum of weighted f64s, rounded once when read (ADR 009). -#[derive(Clone, Debug, PartialEq, Eq)] +#[derive(Clone, Debug)] pub struct ExactSum { limbs: [i64; LIMBS], pending: u64, @@ -29,6 +29,19 @@ pub struct ExactSum { neg_inf: i128, } +/// Equal sums, not equal representations: carries propagated, `pending` ignored. +impl PartialEq for ExactSum { + fn eq(&self, o: &ExactSum) -> bool { + let (mut mine, mut theirs) = (self.limbs, o.limbs); + propagate(&mut mine); + propagate(&mut theirs); + let counts = |s: &ExactSum| (s.nan, s.pos_inf, s.neg_inf); + mine == theirs && counts(self) == counts(o) + } +} + +impl Eq for ExactSum {} + impl Default for ExactSum { fn default() -> Self { ExactSum { diff --git a/crates/numtower/tests/properties.rs b/crates/numtower/tests/properties.rs index 95afebe9..b6b42e18 100644 --- a/crates/numtower/tests/properties.rs +++ b/crates/numtower/tests/properties.rs @@ -161,9 +161,30 @@ fn contagion_is_total_catches_an_integral_inexact_kept_exact() { ); } +/// The scale a decimal power is stored at: minimal, padded toward `min(s·k, 28)` for k > 0 only. +fn power_scale(base: &Num, k: i32, v: &BigRational) -> u32 { + let scaled = |s: u32| v * BigRational::from_integer(BigInt::from(10).pow(s)); + let fits_at = + |s: u32| scaled(s).is_integer() && scaled(s).to_integer().magnitude().bits() <= 96; + let minimal = (0..=28) + .find(|&s| scaled(s).is_integer()) + .expect("a decimal result"); + let target = match (base.kind(), k > 0) { + (NumRef::Exact(d), true) => (d.scale() * k as u32).min(28), + _ => 0, + }; + (minimal..=target.max(minimal)) + .rev() + .find(|&s| fits_at(s)) + .unwrap_or(minimal) +} + #[test] fn pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits() { - run((any_num(), -6i32..=6), 256, false, |(a, k)| { + let units = + prop::sample::select(vec!["1.0", "-1.0", "1.000", "-1", "0.00", "2.0"]).prop_map(dec); + let bases = prop_oneof![4 => any_num(), 1 => units]; + run((bases, -6i32..=6), 256, false, |(a, k)| { let r = a.pow(&Num::exact(Decimal::from(k))); match (q(&a).filter(|_| a.is_exact()), r) { (_, Err(_)) => prop_assert!(a.is_zero() && k < 0, "{a}^{k} refused"), @@ -179,6 +200,10 @@ fn pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits() { r.is_exact() || is_nearest(r.to_f64(), &want), "{a}^{k} = {r}" ); + if let NumRef::Exact(d) = r.kind() { + let scale = power_scale(&a, k, &want); + prop_assert_eq!(d.scale(), scale, "power scale: {}^{} = {}", a, k, r); + } } } Ok(()) @@ -186,6 +211,18 @@ fn pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits() { .unwrap(); } +#[test] +fn a_negative_power_has_its_minimal_scale() { + let at = |base: &str, k: i64| dec(base).pow(&Num::exact(Decimal::from(k))).unwrap(); + assert!(same(&at("1.0", -2), &dec("1")), "{}", at("1.0", -2)); + assert!(same(&at("-1.0", -3), &dec("-1")), "{}", at("-1.0", -3)); + assert!(same(&at("2.0", -2), &dec("0.25"))); + assert!( + same(&at("-1.0", 3), &dec("-1.000")), + "a positive power keeps s·n" + ); +} + #[test] fn pow_decides_its_size_before_computing_and_honours_the_ceiling() { let third = Num::ratio(1, 3).unwrap(); @@ -556,21 +593,75 @@ fn term() -> impl Strategy { ] } +/// Z-set weights: zero, unit, small, wide and both extremes (never `i64::MIN`, so it negates). +fn weight() -> impl Strategy { + prop_oneof![ + 3 => Just(1i64), + 1 => Just(-1i64), + 1 => Just(0i64), + 2 => -3i64..=3, + 2 => (-(1i64 << 40))..(1i64 << 40), + 1 => prop_oneof![Just(i64::MAX), Just(-i64::MAX), Just((1i64 << 30) - 1)], + ] +} + +/// Many non-decimal denominators, so a list often holds more than K groups, several terms each. +const POOL: [i128; 14] = [ + 3, + 7, + 11, + 12, + 13, + 17, + 19, + 23, + 29, + 31, + 37, + 49, + 1 << 40, + 3 * (1 << 33), +]; + +fn pooled_term() -> impl Strategy { + let pooled = (-1000i128..1000, prop::sample::select(POOL.to_vec())) + .prop_map(|(n, d)| Num::ratio(n, d).unwrap()); + prop_oneof![ + 3 => decimal(), + 4 => pooled, + 1 => fraction(), + 2 => finite_f64().prop_map(Num::inexact), + ] +} + +fn weighted(terms: impl Strategy) -> impl Strategy { + (terms, weight()) +} + fn sum_all(xs: &[Num]) -> Num { let mut s = NumSum::new(); xs.iter().for_each(|x| s.insert(x)); s.read() } -fn check_order_free(sum: &dyn Fn(&[Num]) -> Num, deterministic: bool) -> Result<(), String> { +fn sum_weighted(xs: &[(Num, i64)]) -> Num { + let mut s = NumSum::new(); + xs.iter().for_each(|(x, w)| s.add(x, *w)); + s.read() +} + +type WeightedSum<'a> = &'a dyn Fn(&[(Num, i64)]) -> Num; + +fn check_order_free(sum: WeightedSum, deterministic: bool) -> Result<(), String> { let seed = vec![ - dec("0.30000000000000004"), - dec("1000000000000"), - dec("-1000000000000"), + (dec("0.30000000000000004"), 1), + (dec("1000000000000"), 1), + (dec("-1000000000000"), 1), ]; let cases = prop_oneof![ 1 => Just(seed), - 4 => prop::collection::vec(term(), 0..24), + 4 => prop::collection::vec(weighted(term()), 0..24), + 2 => prop::collection::vec(weighted(pooled_term()), 0..40), ]; let shuffled = cases.prop_flat_map(|xs| (Just(xs.clone()), Just(xs).prop_shuffle())); run(shuffled, 256, deterministic, |(xs, ys)| { @@ -582,7 +673,7 @@ fn check_order_free(sum: &dyn Fn(&[Num]) -> Num, deterministic: bool) -> Result< #[test] fn the_accumulator_is_independent_of_order() { - check_order_free(&sum_all, false).unwrap(); + check_order_free(&sum_weighted, false).unwrap(); let seed = sum_all(&[ dec("0.30000000000000004"), dec("1000000000000"), @@ -592,13 +683,14 @@ fn the_accumulator_is_independent_of_order() { } /// Clause 8's rival: `Decimal`'s own `+`, demoting to f64 where it overflows. -fn decimal_fold(xs: &[Num]) -> Num { - xs.iter().fold(Num::exact(Decimal::ZERO), |acc, x| { +fn decimal_fold(xs: &[(Num, i64)]) -> Num { + xs.iter().fold(Num::exact(Decimal::ZERO), |acc, (x, w)| { + let x = x * &Num::exact(Decimal::from(*w)); match (acc.kind(), x.kind()) { (NumRef::Exact(a), NumRef::Exact(b)) => { - a.checked_add(*b).map_or_else(|| &acc + x, Num::exact) + a.checked_add(*b).map_or_else(|| &acc + &x, Num::exact) } - _ => &acc + x, + _ => &acc + &x, } }) } @@ -610,66 +702,117 @@ fn the_accumulator_is_independent_of_order_catches_decimal_addition() { #[test] fn the_accumulator_retracts_exactly() { - run( - ( - prop::collection::vec(term(), 0..16), - prop::collection::vec(term(), 0..16), - ), - 256, - false, - |(xs, ys)| { - let mut s = NumSum::new(); - xs.iter().chain(&ys).for_each(|x| s.insert(x)); - ys.iter().rev().for_each(|y| s.retract(y)); - let fresh = sum_all(&xs); - prop_assert!( - same(&s.read(), &fresh), - "{} after retraction, {} fresh", - s.read(), - fresh - ); - Ok(()) - }, - ) + let terms = || prop::collection::vec(weighted(pooled_term()), 0..20); + run((terms(), terms()), 256, false, |(xs, ys)| { + let mut s = NumSum::new(); + xs.iter().chain(&ys).for_each(|(x, w)| s.add(x, *w)); + ys.iter().rev().for_each(|(y, w)| s.add(y, -w)); + let fresh = sum_weighted(&xs); + prop_assert!( + same(&s.read(), &fresh), + "{} after retraction, {} fresh", + s.read(), + fresh + ); + Ok(()) + }) .unwrap(); } +/// The oracle's partition of a weighted list, exactly as clause 8 and option (d) state it. +struct Expected { + decimal_part: BigRational, + groups: std::collections::BTreeMap, + inexact: std::collections::BTreeMap, + inexact_total: BigRational, +} + +fn ratio_of(i: i128) -> BigRational { + BigRational::from_integer(BigInt::from(i)) +} + +fn rounded(v: &BigRational) -> BigRational { + BigRational::from_float(v.to_f64().unwrap()).unwrap() +} + +impl Expected { + fn of(xs: &[(Num, i64)]) -> Expected { + let zero = || ratio_of(0); + let mut e = Expected { + decimal_part: zero(), + groups: Default::default(), + inexact: Default::default(), + inexact_total: zero(), + }; + for (x, w) in xs { + let weighted = q(x).unwrap() * ratio_of(*w as i128); + match x.kind() { + NumRef::Inexact(f) => { + *e.inexact.entry(f.to_bits()).or_insert(0) += *w as i128; + e.inexact_total += weighted; + } + NumRef::Fraction(r) if !decimal_denominator(*r.denom()) => { + *e.groups.entry(*r.denom()).or_insert_with(zero) += weighted; + } + _ => e.decimal_part += weighted, + } + } + e + } + + fn live(&self) -> impl Iterator { + self.groups.values().filter(|v| !v.is_zero()) + } +} + #[test] fn the_accumulator_reads_the_exact_sum_or_its_stated_roundings() { - run(prop::collection::vec(term(), 0..24), 256, false, |xs| { - let got = sum_all(&xs); - let mut groups = std::collections::BTreeMap::::new(); - let mut exact = BigRational::from_integer(BigInt::from(0)); - let mut floats = Vec::new(); - for x in &xs { - match x.kind() { - NumRef::Inexact(f) => floats.push(f), - NumRef::Fraction(r) if !decimal_denominator(*r.denom()) => { - *groups.entry(*r.denom()).or_insert_with(BigRational::zero) += q(x).unwrap(); - exact += q(x).unwrap(); - } - _ => exact += q(x).unwrap(), - } - } - let live = groups.values().filter(|v| !v.is_zero()).count(); - match (floats.is_empty(), live <= numtower::K) { - (true, true) => { - prop_assert_eq!(got.is_exact(), fits(&exact), "{}", got); - prop_assert!(!got.is_exact() || q(&got) == Some(exact.clone()), "{got}"); - prop_assert!(got.is_exact() || is_nearest(got.to_f64(), &exact), "{got}"); - } + let arms = std::cell::Cell::new([0u32; 3]); + let lists = prop_oneof![ + prop::collection::vec(weighted(term()), 0..24), + prop::collection::vec(weighted(pooled_term()), 0..40), + ]; + run(lists, 512, false, |xs| { + let got = sum_weighted(&xs); + let e = Expected::of(&xs); + let present = e.inexact.values().any(|w| *w != 0); + let exact_total = e.live().fold(e.decimal_part.clone(), |s, g| s + g); + let mut seen = arms.get(); + match (present, e.live().count() <= numtower::K) { (false, true) => { - let mut total = BigRational::from_float(exact.to_f64().unwrap()).unwrap(); - floats - .iter() - .for_each(|f| total += BigRational::from_float(*f).unwrap()); + seen[0] += 1; + prop_assert_eq!(got.is_exact(), fits(&exact_total), "{}", got); + prop_assert!( + !got.is_exact() || q(&got) == Some(exact_total.clone()), + "{got}" + ); + prop_assert!( + got.is_exact() || is_nearest(got.to_f64(), &exact_total), + "{got}" + ); + } + (true, true) => { + seen[1] += 1; + let total = rounded(&exact_total) + &e.inexact_total; prop_assert!(!got.is_exact() && is_nearest(got.to_f64(), &total), "{got}"); } - (_, false) => prop_assert!(!got.is_exact(), "past K groups: {got}"), + (_, false) => { + seen[2] += 1; + let groups = e.live().map(rounded).fold(ratio_of(0), |s, g| s + g); + let total = groups + rounded(&e.decimal_part) + &e.inexact_total; + let near = is_nearest(got.to_f64(), &total); + prop_assert!(!got.is_exact() && near, "past K groups: {got} for {total}"); + } } + arms.set(seen); Ok(()) }) .unwrap(); + let seen = arms.get(); + assert!( + seen.iter().all(|n| *n >= 20), + "every branch must be reached often: {seen:?}" + ); } fn decimal_denominator(d: i128) -> bool { @@ -1085,3 +1228,140 @@ fn the_canonical_text_reads_back_as_the_same_number() { let ten35 = Num::ratio(10i128.pow(35), 1).unwrap(); assert_eq!(ten35.to_string(), format!("{}/1", 10i128.pow(35))); } + +#[test] +fn exact_sums_are_equal_when_their_values_are() { + let of = |terms: &[(f64, i64)]| { + let mut s = numtower::ExactSum::new(); + terms.iter().for_each(|(x, w)| s.add(*x, *w)); + s + }; + assert_eq!( + of(&[(0.5, 1), (0.5, 1)]), + of(&[(1.0, 1)]), + "0.5 + 0.5 = 1.0" + ); + assert_eq!( + of(&[(3.25, 1), (3.25, -1)]), + numtower::ExactSum::new(), + "insert then retract" + ); + let big = (1i64 << 30) - 1; + assert_eq!( + of(&[(1.5, big), (0.25, 2)]), + of(&[(0.25, 2), (1.5, big)]), + "order" + ); + assert_ne!(of(&[(1.0, 1)]), of(&[(1.0, 2)])); + assert_ne!( + of(&[(f64::NAN, 1)]), + of(&[]), + "a counted NaN is part of the value" + ); +} + +#[test] +fn a_forced_fraction_never_reads_as_infinity() { + let huge = format!("{}/1", "9".repeat(400)); + assert_eq!( + parse_number(&huge, Typing::Forced).map(|n| n.to_string()), + None + ); + assert!( + !is_number(&huge, Typing::Forced), + "the gate and the parse agree" + ); + let negative = format!("-{huge}"); + assert!(parse_number(&negative, Typing::Forced).is_none()); + let tiny = format!("1/{}", "9".repeat(400)); + let zero = parse_number(&tiny, Typing::Forced).unwrap(); + assert!( + !zero.is_exact() && zero.to_f64() == 0.0, + "an underflow is a finite zero, as 1e-999" + ); + assert!(is_number(&tiny, Typing::Forced)); +} + +type InterimCmp<'a> = &'a dyn Fn(&Interim, &Interim) -> Option; + +fn check_interim_order(cmp: InterimCmp, deterministic: bool) -> Result<(), String> { + let eps = Interim::from(Num::ratio(1, 3 * 10i128.pow(30)).unwrap()); + run( + ( + wide_fraction(), + wide_fraction(), + wide_fraction(), + wide_fraction(), + ), + 256, + deterministic, + |(x1, x2, y1, y2)| { + let a = &Interim::from(x1.clone()) * &Interim::from(x2.clone()); + let av = q(&x1).unwrap() * q(&x2).unwrap(); + let near = av.to_f64().unwrap(); + let eps_v = + BigRational::new(BigInt::from(1), BigInt::from(3) * BigInt::from(10).pow(30)); + let others = [ + ( + &Interim::from(y1.clone()) * &Interim::from(y2.clone()), + q(&y1).unwrap() * q(&y2).unwrap(), + ), + (&a + &eps, &av + &eps_v), + (&a - &eps, &av - &eps_v), + (a.clone(), av.clone()), + ( + Interim::from(Num::inexact(near)), + BigRational::from_float(near).unwrap(), + ), + ( + Interim::from(Num::inexact(near.next_up())), + BigRational::from_float(near.next_up()).unwrap(), + ), + ]; + for (b, bv) in &others { + prop_assert_eq!( + cmp(&a, b), + Some(av.cmp(bv)), + "interim order: {:?} vs {:?}", + a, + b + ); + prop_assert_eq!( + cmp(b, &a), + Some(bv.cmp(&av)), + "interim order: {:?} vs {:?}", + b, + a + ); + } + let twin = a.clone(); + prop_assert!( + !a.is_zero() && (&a - &twin).is_zero(), + "interim zero: {a:?}" + ); + Ok(()) + }, + ) +} + +#[test] +fn an_interim_compares_exactly_without_storing() { + check_interim_order(&Interim::cmp_exact, false).unwrap(); + assert_eq!( + Interim::from(Num::inexact(f64::NAN)).cmp_exact(&Interim::from(dec("1"))), + None + ); +} + +/// Storing first: what `=` would do if `Interim` could not compare. +fn store_then_compare(a: &Interim, b: &Interim) -> Option { + a.clone().store().cmp_exact(&b.clone().store()) +} + +#[test] +fn an_interim_compares_exactly_without_storing_catches_storing_first() { + assert_caught( + check_interim_order(&store_then_compare, true), + "interim order", + ); +} From f11d27377d785d054b7f48643429d2a8d11e68a7 Mon Sep 17 00:00:00 2001 From: "Claude (numtower)" Date: Fri, 9 Oct 2026 00:14:57 -0700 Subject: [PATCH 065/106] test(numtower): property tests run at the 256-case budget (improvise-59nw.1) Answers Copilot on PR #337 (context/design-principles.md section 7). Every fault in the review's table still fails by name at 256 cases. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/test-evidence/numtower.md | 21 +++++++++++++++------ crates/numtower/tests/properties.rs | 10 +++++----- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/context/test-evidence/numtower.md b/context/test-evidence/numtower.md index 451f5a0c..c57949a3 100644 --- a/context/test-evidence/numtower.md +++ b/context/test-evidence/numtower.md @@ -105,11 +105,18 @@ to discriminate. ### rust_decimal_to_f64_agrees_on_fifteen_digit_mantissas_so_a_narrow_generator_is_vacuous -This test **passes the rival** on purpose. With mantissas of at most 15 digits, 4,096 -cases, `rust_decimal`'s `to_f64` agrees with the text every time, as ADR 005 says (200,000 -of 200,000). So a test whose generator stays in that range proves nothing about the -conversion. This test is why the generator above spans 96 bits, and it is what fails, and -should be re-read, if `rust_decimal` ever fixes its conversion. +This test **passes the rival** on purpose. With mantissas of at most 15 digits, +`rust_decimal`'s `to_f64` agrees with the text every time, at this test's 256 cases as +in ADR 005's 200,000 of 200,000. So a test whose generator stays in that range proves +nothing about the conversion, and this test is why the generator above spans 96 bits. It +ran at 4,096 cases until Copilot pointed out, on PR #337, that the repository's budget is +256 (`context/design-principles.md` §7). The 200,000 is the ADR's measurement, and this +test only pins its direction. + +If `rust_decimal` ever fixes its conversion, this test still passes. What fails then is +`the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_to_f64`, +whose rival would agree. That is the signal to re-read both. An earlier version of this +entry named this test instead, which was wrong. ### a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval @@ -205,7 +212,9 @@ faults passed the whole suite: - F14: the past-K read dropped the decimal partition; - F15: the past-K read dropped every inexact term. -The test now has the reviewer's oracle and 512 cases. Half the lists come from the +The test now has the reviewer's oracle, at the repository's 256-case budget. 34ab0fc3 ran +it at 512, and Copilot pointed out the budget. At 256, every fault in the table below +still failed by name, and F7, F8, F12, F14 and F15 failed in four of four passes. Half the lists come from the fourteen-denominator pool, up to 40 terms long, so groups receive several terms and K is passed often. **It also counts how many cases reach each branch and fails if any branch is reached fewer than 20 times**, so a generator change cannot silently starve one. diff --git a/crates/numtower/tests/properties.rs b/crates/numtower/tests/properties.rs index b6b42e18..5c5a8e5e 100644 --- a/crates/numtower/tests/properties.rs +++ b/crates/numtower/tests/properties.rs @@ -93,7 +93,7 @@ fn an_exact_result_is_never_a_rounded_one_catches_a_silently_rounding_add() { fn check_contagion(add: BinOp, deterministic: bool) -> Result<(), String> { let seed = Just((dec("2"), Num::inexact(3.0))); let cases = prop_oneof![1 => seed, 8 => (any_num(), any_num())]; - run(cases, 512, deterministic, |(a, b)| { + run(cases, 256, deterministic, |(a, b)| { let results = [ ("+", add(&a, &b)), ("-", Some(&a - &b)), @@ -310,7 +310,7 @@ fn the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_ #[test] fn rust_decimal_to_f64_agrees_on_fifteen_digit_mantissas_so_a_narrow_generator_is_vacuous() { let narrow = 10i128.pow(15) - 1; - let result = run((-narrow..=narrow, 0u32..=15), 4096, true, |(m, s)| { + let result = run((-narrow..=narrow, 0u32..=15), 256, true, |(m, s)| { let d = Decimal::from_i128_with_scale(m, s); prop_assert_eq!( crate_to_f64(&d).to_bits(), @@ -772,7 +772,7 @@ fn the_accumulator_reads_the_exact_sum_or_its_stated_roundings() { prop::collection::vec(weighted(term()), 0..24), prop::collection::vec(weighted(pooled_term()), 0..40), ]; - run(lists, 512, false, |xs| { + run(lists, 256, false, |xs| { let got = sum_weighted(&xs); let e = Expected::of(&xs); let present = e.inexact.values().any(|w| *w != 0); @@ -1004,7 +1004,7 @@ fn check_language( parse: &dyn Fn(&str, Typing) -> Option, deterministic: bool, ) -> Result<(), String> { - run(lexeme(), 512, deterministic, |s| { + run(lexeme(), 256, deterministic, |s| { let gate = s.parse::().is_ok_and(f64::is_finite); let parsed = parse(&s, Typing::Inferred); prop_assert_eq!( @@ -1205,7 +1205,7 @@ fn an_intermediate_beyond_i128_comes_back_exact() { #[test] fn the_canonical_text_reads_back_as_the_same_number() { - run(any_num(), 512, false, |n| { + run(any_num(), 256, false, |n| { let text = n.to_string(); let back = Num::from_canonical(&text); prop_assert!( From 6e4678788785dee2b9c1c31c9e20d409b54e3643 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:01:10 -0700 Subject: [PATCH 066/106] docs(adr): settle ADR 007's eleven open questions and state the decisions (improvise-cq1q.1) The decisions are written into the clauses as plain design, tagged [Amendment]: a measure formula's id is the measure's id; formulas may refer to any id-bearing thing; ids always win, with the display as a preview; user interfaces accept display names and scripts use ids and display_for; slugify replaces every non-alphanumeric character with _, collapses _, trims _ at both ends and prefixes n before a digit; a hand-written id obeys the generated syntax; a leading _ is reserved for virtual ids, which are lower case and never written to .improv; records-mode positions are values; groups have a display and ordinary ids. All eleven questions are deleted. Five new ones are open: the - suffix in hand-written ids, ASCII "alphanumeric", how a file records the virtual categories, the dimension formula line and [_measure] tag, and which headless entry points are scripts. core.md section 20's note follows the new slug rules. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 539 ++++++++++++--------------- context/code/core.md | 15 +- 2 files changed, 247 insertions(+), 307 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 79c29a4a..36e719e0 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -18,15 +18,16 @@ amendment is approved: | clause | accepted rule | the amendment proposes | |---|---|---| -| 1 | groups have no id | a group's identifier is an id | -| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | the virtual views also have reserved `_` ids: `_records`, `_drill` | -| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` (pending open question 1) | +| 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | +| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; virtual ids are never written to `.improv` | +| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | -| 4 | a run of U+0020 spaces becomes one `_`; only a leading `_` is removed | each whitespace character becomes `_`, runs of `_` collapse after the strip, and leading and trailing `_` and `-` are removed | +| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every non-alphanumeric character becomes `_`; runs of `_` collapse; leading and trailing `_` are removed; `n` is prepended to a slug that starts with a digit | +| 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` and may begin with `_` only if reserved | a hand-written id obeys every syntax rule of a generated id | | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | | 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | -Eleven questions are open (last section). +Five questions are open (last section). ## Context @@ -73,55 +74,79 @@ human name moves to a new `display` field. | item | yes | `ItemId` | | formula | yes | `FormulaId` | | view | yes | `ViewId` | -| group | **[Amendment]** yes (open questions 9 and 10) | open question 10 | +| group | **[Amendment]** yes, with a display | `GroupId` | | record | `RecordHash`, over ids (clause 8) | unchanged shape | **An id always names the same thing, so ids are unique in the whole document**, across every kind. An item id names one item without its category. A `CellKey` still pairs each item with its category, and an operation that creates an item still names the -parent category. **[Amendment]** A formula whose generated id would collide takes a -suffix like any other generated id (clause 4): a formula `Tax` in `CustomCat` gets -item id `tax`, then formula id `tax-`. **[Amendment]** Formulas may refer to -anything in a typed context. What exactly that allows is open question 2. +parent category. + +**[Amendment] Groups.** A group has an id and a display, like a category or an item, +and its id behaves like every other id: it is unique in the document, generated by +slugify (`group` is an empty slug's ``), declared beside the display in the file, +referred to by id, created by display, and renamed by a display rename. In code, +PR #336 (improvise-cq1q.12, open) already adds a private display to `Group` beside its +key; under this amendment that key becomes the group's id. **Reserved ids.** The virtual categories have ids `_index`, `_dim` and `_measure`, displayed `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** -**[Amendment]** A leading `_` is reserved for virtual categories, measures and other -entities. The virtual views have ids `_records` and `_drill`, displayed -`_Records` and `_Drill`. They are never persisted, but they are addressed on the wire -(`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could collide -with a user's view. Which entities the reservation covers is open question 6. Whether -records mode's synthetic positions under `_Index` and `_Dim` are ids is open question 7. -Moving today's capitalised keys to these ids is open question 8. +**[Amendment]** A leading `_` is reserved for virtual ids in general. The virtual views +have ids `_records` and `_drill`, displayed `_Records` and `_Drill`; they are addressed +on the wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and +could collide with a user's view. Slugify never produces a leading `_` (clause 4), so +the reservation needs no check in the grammar. +- **[Amendment] Virtual ids are lower case and are never written to `.improv`.** The + code, the wire, `ViewState` and the browser client spell the virtual categories + `_Index`, `_Dim` and `_Measure` today, and move to the lower-case ids in the re-key + step (clause 11). How the file records what it records today under `_Measure`, + `_Index` and `_Dim` is open question 3. +- **[Amendment] Records mode's synthetic positions are values, not ids.** The row + coordinate under `_index` and the column coordinate under `_dim` (a column index, + gotcha 1) are values of those virtual categories. They are never persisted or typed, + take no part in id uniqueness, and cannot collide with a user's item. -**Formula targets.** **[Amendment]** A measure is the normal formula target +**Formulas and measures.** **[Amendment]** A measure is the normal formula target (`profit = revenue - cost`). A dimension (a category other than `_Measure`) is a rare one, for a derived dimension. -- **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, - and its formula has a separate `FormulaId`. - **A `_Measure` target has no item of its own**, and its handle is its formula's `FormulaId`. So the category panel's `_Measure` rows use `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). -- **[Amendment]** A measure has an id and a display, as everything else does. - Whether a formula-target measure keeps the formula's id or becomes an item with - its own is open question 1. The `ItemHandle` text in clauses 2, 5, 6, 8 and 11 - depends on its answer. +- **[Amendment] A formula is a kind of measure, so a `_Measure` formula's id is the + measure's id.** A measure has an id and a display, as everything else does. + `Profit = Revenue - Cost` is one thing, `profit`: another formula refers to it as + `profit`, each Profit cell's `CellKey` holds `profit` under `_measure`, and + `ReplaceFormula` and `RemoveFormula` address `profit`. A formula that derives a + measure already stored under that display takes the stored measure's id, so the + column has one id, not two. Removing the formula removes a measure that exists only + through it; a measure that also holds stored data stays, as a stored measure. +- **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, + and its formula has a separate `FormulaId`. A formula whose generated id would + collide takes a suffix like any other generated id (clause 4): a formula `Tax` in + `CustomCat` gets item id `tax`, then formula id `tax-`. +- **[Amendment] Formulas may refer to anything by id.** A function in a formula may + accept any id-bearing thing (a category, an item, a measure, a formula, a group), + referred to by its id. Whether a given function takes the thing itself or its value + (an item's display, say) is part of that function's signature. Global uniqueness + makes every such reference unambiguous, and a function may read any part of a record, + so nothing further is needed here. ### 2. Storage keys by id; `display` is the only place a human name lives - **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id, - and each `Category`, `Item` and `View` gains a `display: String`. + and each `Category`, `Item` and `View` gains a `display: String`. **[Amendment]** So + does `Group`. - **Cells:** a `CellKey` is a sorted list of `(CategoryId, ItemId)` pairs. **[Amendment]** The pairs are `(CategoryId, ItemHandle)`, so that a `_Measure` formula cell's coordinate is `FormulaTarget` without claiming an `ItemId` the - document does not hold (pending open question 1). + document does not hold. - The key stays sorted, now **by category id**, so the canonical form (gotcha 5) is kept with a different sort key. - `DataStore` interns ids exactly as it interns names today. - An id's text names one thing, because ids are unique across kinds. - **View configuration:** the axis maps, page selections, and hidden and collapsed sets are keyed by ids. **[Amendment]** A collapsed entry names its group by the group's - id. Whether the category id stays beside it is part of open question 10. + id. - **Formulas:** a formula's references are ids (clause 5), and the formula panel renders them as display names (clause 6). @@ -158,68 +183,68 @@ copyable, compared as an integer. **Every** id the app generates comes from the display name by **slugify**: on import, on interactive creation, and when an id-less `.improv` file is loaded. So a -thing's id never depends on how it was made. An id is lower case, in the alphabet -`a`–`z`, `0`–`9`, `-` and `_`. +thing's id never depends on how it was made. An id is lower case. -The steps, in order, as the amendment states them **[Amendment]**: +**The accepted steps:** +1. casefold, with Unicode full case folding; +2. replace each run of spaces with one `_`; +3. strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`; +4. remove any leading `_`; +5. if nothing is left, use `-`. + +**[Amendment] The steps become:** 1. **Casefold, with Unicode full case folding**, so `Straße` → `strasse`. Rust's - `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped). The folding - table or crate must pass the `wasm` gate, because `improvise-core` is in the browser - bundle. -2. **Replace each whitespace character with `_`.** "Whitespace" is the Unicode - `White_Space` property (`char::is_whitespace`): U+0009–U+000D, U+0020, U+0085, - U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F and U+3000. - - A tab, a no-break space and an ideographic space each become `_`. - - A zero-width space (U+200B) and a byte-order mark (U+FEFF) are not whitespace, - and step 3 strips them. - - The realistic case is a header pasted from a web page, where a no-break space - stands for a space. -3. **Strip every character outside the id alphabet.** This strips non-ASCII letters, - so an id never needs quoting in an `.improv` reference. -4. **Collapse each run of `_` to one `_`**, whether the underscores came from - whitespace or were in the display. The step comes after the strip (open - question 5). -5. **Remove every leading and trailing `_` and `-`**, repeating until neither end - holds either (open question 11). - - A leading `_` is reserved (clause 1). - - A trailing `_` is dropped. - - An id never begins or ends with `-`. + `str::to_lowercase` is not enough (`ß` stays `ß`). The folding table or crate must + pass the `wasm` gate, because `improvise-core` is in the browser bundle. +2. **Replace every character that is not alphanumeric with `_`.** That includes + spaces, tabs and every other whitespace, punctuation and `-`. Whether + "alphanumeric" means ASCII only is open question 2. +3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 + or were in the display. +4. **Remove every leading and trailing `_`.** A leading `_` is reserved for virtual + ids (clause 1), and a trailing one is dropped. +5. **If the slug starts with a digit, prepend `n`.** An id is never purely numeric + and never starts with a digit, so `2024` becomes `n2024`. 6. **If nothing is left, use `-`**, such as `category-1` or `item-4`. `n` - comes from the shared counter below. A display of only `-`, `_` or non-ASCII letters - ends here, because `-` alone is not an id. + comes from the shared counter below. -The accepted text has five steps instead: casefold; replace each **run of spaces** -with one `_`; strip; remove any **leading** `_`; `-` if empty. +So a generated id is a lower-case letter followed by letters, digits and single +underscores, never ending in `_`, plus, only when it collided, a `-` suffix; or +`-`. Whether a hand-written id may carry that suffix is open question 1. -Examples under the amendment: +Examples under the amendment, with "alphanumeric" read as ASCII: | display | id | |---|---| | `East` | `east` | -| `Time Period`, `Time Period`, `Time\tPeriod`, `Time\u{A0}Period`, `Time \u{200B} Period` | `time_period` | +| `Time Period`, `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` | | `Coastal_East` | `coastal_east` | | `a_ b`, `a__b` | `a_b` | -| `Q1-2024` | `q1-2024` | -| `Q1 - 2024` | `q1_-_2024` (only `_` runs collapse) | +| `Q1-2024`, `Q1 - 2024` | `q1_2024` | | `East `, `East_`, `-East-` | `east` | | `_Draft` | `draft` | -| `Region(code)` | `regioncode` | +| `Region(code)` | `region_code` | | `Café` | `caf` | | `Straße` | `strasse` | -| `Ü`, `-`, `---` | `-` | +| `2024`, `_2024` | `n2024` | +| `2024 Budget` | `n2024_budget` | +| `Ü`, `-`, `%` | `-` | + +Under the accepted steps, the ones that differ are `Q1-2024` → `q1-2024`, +`Q1 - 2024` → `q1_-_2024`, `East ` → `east_`, `a_ b` → `a__b`, `Region(code)` → +`regioncode`, `2024` → `2024` and `-` → `-`. **Collisions.** `Time Period` and `Time_Period` slug alike, and whichever comes later -takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, which slug alike -only under the amendment's steps (under the accepted steps `East ` slugs to `east_`). `East` in `Region` and `East` in -`Direction` also slug alike, and ids are global, so the second is `east-`. +takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, and `Q1-2024` and +`Q1 2024`, which slug alike only under the amendment's steps. `East` in `Region` and +`East` in `Direction` also slug alike, and ids are global, so the second is `east-`. - **Any colliding generated id gets a suffix**, whatever kind of thing holds the slug: on import, when an id-less file loads, and when the host creates something in a session. An id that does not collide stays bare. - **The suffix is `-` plus the next value of one increment-only counter per document**, as a plain number (`east-1`, not `east-001`). The counter works like Common Lisp's `gensym`: it never goes down and is never reused, so no order holds - among suffixes (`east-7` may follow `east` directly). **[Amendment]** A suffix ends - in a digit, so a suffixed id still meets the amendment's step 5. + among suffixes (`east-7` may follow `east` directly). - **Retirement.** Within a session a removed id is never handed out again. The counter is past any suffix it issued, and a removed bare slug stays reserved in memory. @@ -238,6 +263,9 @@ every data key and reference in that file spells it. So: - **two identical hand-written ids in one file are refused at load**, even when they declare different kinds of thing. Renaming one would corrupt the data section, and a file with a duplicate cannot be resaved safely. The error names both lines. +- **[Amendment] A hand-written id obeys every syntax rule of a generated id**: lower + case, never starting with `_` or a digit, never ending in `_`, no runs of `_`, and no + character slugify would replace. One that does not is refused at load (clause 5). **Reuse across a save and reload is accepted.** Retired ids are not persisted. So remove `East`, save, reload and add `East`, and the new item gets `east` again. An @@ -252,19 +280,18 @@ promised (improvise-8tk5). **Built code.** `crates/improvise-core/src/model/id.rs` (`slugify`, `IdAllocator`) implements the accepted steps, and `context/code/core.md` §20 describes them. Once the amendment is approved, a follow-up must: -- test whitespace, not only U+0020; -- collapse `_` after the strip; -- trim a trailing `_` and both ends' `-`; -- settle what `claim` refuses (open question 11); -- add the virtual views to `VIRTUAL_IDS`; -- add `IdKind::Group` if groups are generated (open question 10). +- replace every non-alphanumeric character, not only spaces, with `_`, and collapse + `_` runs; +- trim a trailing `_`, and prepend `n` before a leading digit; +- make `claim` refuse every hand-written id a generated one could not be; +- add the virtual views to `VIRTUAL_IDS`, and `IdKind::Group`. Nothing calls `slugify` yet. ### 5. Persistence: ids on declarations *and* references -- **Declarations** (section headers, item lines, formula lines) carry the display and - the id. +- **Declarations** (section headers, item lines, formula lines, and **[Amendment]** + group definitions) carry the display and the id. - **Every reference** is the id: axis lines, `hidden:` and `collapsed:`, page selections, `## Aggregation`, `Initial View:`, `## Data` keys (ids are guaranteed unique, names might not be), and names inside formula text. @@ -278,14 +305,17 @@ grammar spells it **with ranges**, because the generator drops `ASCII_ALPHANUMER (`Sales(Copy)`), an empty tag or a tag with a trailing space does not match `id_tag`, so it has no id. The error names the line; - a hand-written id beginning with `_`, other than a reserved one; -- **[Amendment]** a hand-written id that is `-` (open question 11 asks whether a - leading or trailing `-`, or a trailing `_`, is refused too). +- **[Amendment]** a hand-written id that breaks any syntax rule of a generated id + (clause 4). Virtual ids are never written to a file, so no hand-written id begins + with `_`. The grammar's `id_token` stays permissive, and the loader refuses, so the + error can name the line. **Inside formula text an id may still need quoting.** The formula grammar's `bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So -the writer spells an id in formula text through `quote::quote_name`. An id beginning -with a digit (`2024`) or equal to a keyword (`sum`, `if`, `where`, `is`) is written -`|2024|` or `|sum|`. +the writer spells an id in formula text through `quote::quote_name`. An id equal to a +keyword (`sum`, `if`, `where`, `is`) is written `|sum|`, and under the accepted rules +an id beginning with a digit (`2024`) is written `|2024|`. **[Amendment]** No id begins +with a digit, so only keyword ids need quoting. ```pest id_token = @{ ('a'..'z' | '0'..'9' | "_" | "-")+ } @@ -306,9 +336,10 @@ id_ref = { id_token } `model_name` has no id and keeps `header_name`. `Initial View:` is a reference and takes an `id_ref`. -**Group references.** A group reference (`[Coastal]`) is a name. -**[Amendment]** It is the group's id. So is the group in a `collapsed:` line. The -group definition (`> Coastal`) declares the id. The spellings are open question 10. +**Group references.** A group reference (`[Coastal]`) is a name. **[Amendment]** It is +the group's id, as is the group in a `collapsed:` line, and the group definition +(`> Coastal`) declares a display and an id like any other declaration +(`> Coastal(coastal)`). **[Amendment] An axis line is tried before `format:`, and a format that reads as an axis line is pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice, and @@ -330,20 +361,19 @@ lose its axis line to the view's number format, and the new-version arm puts that view also holds a category `format` on an axis. **[Amendment] A formula line** is -`- () = []` (pending open -question 1): +`- () = []`: - **The parenthesised id is the formula's `FormulaId`**, between the target and the - `=`. -- **`` is the target item's id when the target is an item** (outside - `_Measure`). For a `_Measure` target, which has no item, it is the target's - display, held on the `Formula`. + `=`. For a measure formula it is also the measure's id. +- **For a measure formula, `` is the measure's display**, held on the + `Formula`: `- Profit(profit) = revenue - cost`. How a dimension formula's line spells + its target, and whether the `[_measure]` tag is written, is open question 4. - **The expression is over ids**, spelled through `quote::quote_name`. A reference to a formula target is the id text of its `ItemHandle`: the item id outside `_Measure`, - the formula id for a `_Measure` target. + the formula's (and measure's) id for a `_Measure` target. - **A `WHERE`/`IS` value follows the same rule**. It is the id text of an `ItemHandle` in the filtered category, and the formula panel shows it as a display. - **A display in the target position that looks like `Name(x)` is pipe-quoted**, as in - `- |Margin(%)|(margin) = …`. An item id never contains a parenthesis. + `- |Margin(%)|(margin) = …`. An id never contains a parenthesis. **`header_display` is guarded only by a lookahead**, which the generator skips (gotcha 34). So the generator may emit a display ending in something that parses as an @@ -352,7 +382,8 @@ property 1 must be stated over the parsed result, not over the generated text. **The version line alone picks the regime** (clause 10). - **In an old-version file** a header is the whole line, a reference resolves against - display names, and ids are assigned by slugify. + display names, and ids are assigned by slugify. This resolution migrates a file + whose names were its only identity; it is not a reference by a person. - **In a new-version file** declarations carry ids and a reference is an id. **An old file naming an item it never declares** declares it at load. This happens @@ -384,7 +415,7 @@ out, because reassigning an id would silently change what its references point a **Import** (CSV, JSON) never parses ids out of dimension names. A header `Region(code)` is the display `Region(code)`, and its id is its slug. The writer pipe-quotes any display that would otherwise read as display plus id: -`## Category: |Region(code)|(regioncode)`. +`## Category: |Region(code)|(regioncode)` (**[Amendment]** `(region_code)`). ### 6. The wire: address by id, create by display, learn ids from projections @@ -393,13 +424,17 @@ display that would otherwise read as display plus id: being created is a display, and the host slugifies it: - `AddCategory(display)` and `CreateView(display)`; - `AddItem { category: CategoryId, display }`; - - `AddItemInGroup { category: CategoryId, display, group }`. Whether `group` is an - existing group's id or a new group's display is open question 10 **[Amendment]**; + - `AddItemInGroup { category: CategoryId, display, group }`. **[Amendment]** `group` + is the group's id, and a new group is created by its display under the same + create-by-display rule; - `AddFormula { target_category: CategoryId, raw }`. -- **[Amendment] Only a client accepts a display name as a reference**, in its UI - layer, and it converts the name to an id as early as it can. The wire never carries - a display as a reference. Whether the old-version loader and headless `improvise - cmd`/`script` are exceptions is open question 4. +- **[Amendment] Only a user interface accepts a display name as a reference**, and it + converts the name to an id as early as it can. The TUI and browser clients are user + interfaces, and so is the headless command line. The wire never carries a display as + a reference. +- **[Amendment] Scripts refer to things by id.** A script that names something by its + display fails. A script gets a display only through a dedicated function such as + `display_for(id)`. Which headless entry points count as scripts is open question 5. - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. A create-then-act compound becomes **one command**: `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes @@ -408,6 +443,7 @@ display that would otherwise read as display plus id: - **New operations carry ids:** - `RenameCategory { category, to: display }`; - `RenameItem { category, item, to: display }`; + - **[Amendment]** a group rename by group id and new display; - `ReplaceFormula { id, was, raw }`; - `RenameRecordCoordinate`; - the removes (`RemoveCategory`, `RemoveItem`, `RemoveFormula`, `DeleteView`). @@ -415,8 +451,7 @@ display that would otherwise read as display plus id: including `SetCell { view, … }` and `ClearCell { view, … }`), one whole variant at a time. - **Projections carry id and display together:** - - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }` (pending open - question 1); + - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }`; - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows: the target's display with its id in parentheses (`Profit (profit) = Revenue - Cost`), then the expression over display names, with an id wherever a display is @@ -425,15 +460,17 @@ display that would otherwise read as display plus id: would bind elsewhere. `stored` is the id-form text the file carries, and `ReplaceFormula`'s `was` is compared against it. So a rename of `Revenue` elsewhere changes what the panel shows, but does not refuse an edit of `Profit`; - - `ViewSummary { id, display, … }`. + - `ViewSummary { id, display, … }`; + - **[Amendment]** the grid frame's group entries (`FrameGroup`) carry the group's id + and display. `AxisFrame` labels stay display strings. They gain ids when a consumer needs them. - **`ViewState` keys by id**: the cursors, `expanded_cats` and the view history (`view_back_stack` / `view_forward_stack`). **[Amendment]** So do `active_view` itself and `Workbook::active_view` (improvise-p003). A rename then orphans nothing. - **[Amendment] A `_Measure` formula-target row** (`ItemHandle::FormulaTarget`) is - removed by `RemoveFormula` and renamed by a display rename of the formula's target - (improvise-i0n7; pending open question 1). + removed by `RemoveFormula` and renamed by a display rename of the measure + (improvise-i0n7). - **[Amendment] CheckedBatch's created-record exemption is removed** (improvise-p003). It exempts a record that neither resolves nor is claimed, for frames like `[AddItem(East), SetCell(East)]`. A frame cannot address what it creates, so nothing @@ -460,7 +497,7 @@ Per-entity versions are not added. text pairs: the same function over different strings. **[Amendment]** The pairs are `(CategoryId, ItemHandle)` spelled as id text. Ids are unique across kinds, so the hash stays canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate -is never part of a record (pending open question 1). +is never part of a record. ADR 002's decision is unchanged in substance. Its prose ("sorts coordinates by category name", records spelled in names) is out of date once this lands, and the @@ -486,7 +523,7 @@ whole-line reading. | `## View: \|Default (default)\|(default)` | pipes literal | display `Default (default)`, id `default` | | `## View: Sales (copy)` | `Sales (copy)` | **display `Sales`, id `copy`** | | `## Category: Region(code)` | `Region(code)` | **display `Region`, id `code`** | -| `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** | +| `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** (**[Amendment]** refused: an id may not start with a digit) | | `## Category: Time Period` | `Time Period` | unchanged; id `time_period` generated | The app itself writes such headers: `improvise import` on a CSV with headers like @@ -496,7 +533,8 @@ If old files got the rule, that category would load as display `Region`, id `cod Every reference in the file spells `Region(code)`, which resolves to neither, so the axis lines and data keys would not resolve, and nothing would be drawn. Because old files keep the whole-line reading, none of this happens. On the first save the writer -pipe-quotes such a display: `## Category: |Region(code)|(regioncode)`. +pipe-quotes such a display: `## Category: |Region(code)|(regioncode)` +(**[Amendment]** `(region_code)`). ### 10. The file version alone picks the regime @@ -517,10 +555,12 @@ pipe-quotes such a display: `## Category: |Region(code)|(regioncode)`. 1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224. 2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId` and - `ViewId`. **[Amendment]** Also `ItemHandle`, which the re-key needs for `CellKey` - (pending open question 1). + `ViewId`. **[Amendment]** Also `GroupId`, a display on `Group`, and `ItemHandle`, + which the re-key needs for `CellKey`. 3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with - display→id resolution moved to every input edge. + display→id resolution moved to every input edge. **[Amendment]** The virtual + categories' keys move from `_Index`, `_Dim` and `_Measure` to `_index`, `_dim` and + `_measure` here, in the model, on the wire, in `ViewState` and in the browser client. 4. **Persistence:** the new literal (clause 10), clause 5's grammar, the loader and its refusals, slugify, the writer, and the round-trip properties with a formula-line strategy. @@ -556,22 +596,22 @@ answers for those. - **A typed name that was not in the captured mapping** is resolved against the document at commit, and refused, naming it, if it does not resolve. -**[Amendment] The client does the resolving:** +**[Amendment] The client does the resolving, and ids always win:** - **The client holds the mapping**, as it holds `OpenedCell`. The TUI and the browser share one resolver in `improvise-protocol`. - **A name new to the edit** is resolved by the client against its id-bearing projection (clause 6), **ids first, then displays**. Ids are unambiguous, so they are - preferred wherever there is ambiguity. + preferred wherever there is ambiguity, now and once an autocomplete exists. - A typed name equal to an id names that id's thing, even where it is also another thing's display. - Otherwise a display that exactly one thing in scope has names that thing. - A display matching several things is refused, and the refusal lists their ids. - A name that resolves to nothing is refused, and the refusal names it. - Ids are matched exactly, with no case folding. -- **The long-term design separates two layers.** A view-layer autocomplete turns - typed display text into ids before commit, and the model layer refers to ids only. - What becomes of the ids-first order once that exists is open question 3. This ADR - does not specify the autocomplete. +- **Typing prefers ids and previews displays.** A view-layer autocomplete turns typed + text into ids before commit, preferring a match on an id, and shows the matched + thing's display as a preview. The model layer refers to ids only. This ADR does not + specify the autocomplete further. - **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are in the file's id form, and the host never resolves a display. The host parses ids, and one that no longer exists is `Rejection::Gone`. @@ -579,8 +619,8 @@ answers for those. **[Amendment] What ids-first costs.** Suppose item A is displayed `East` with id `east`, and item B is displayed `east`, so slugify gave it `east-1`. Typing `east` names **A**. B is reached by typing `east-1`. A user who typed `east` meaning B is bound to A -without being told. That is why the panel shows B as `east-1` and never as a bare -`east`. +without being told, unless the preview shows it. That is why the panel shows B as +`east-1` and never as a bare `east`. **Required regression test.** 1. Open a formula editor on `Profit = Revenue - Cost`. @@ -659,219 +699,118 @@ three things: - The formula panel is a second rendering of formula text, so display names typed there must be resolved back to ids (by the client, under the amendment). - ADR 002's prose needs a wording update. +- **[Amendment]** Scripts that name things by display break and must be rewritten + over ids. -**Not addressed.** Per-entity versions, ids in `AxisFrame` and `CellId::Derived`, and -(under the accepted text) group ids. +**Not addressed.** Per-entity versions, ids in `AxisFrame` labels and +`CellId::Derived`, and (under the accepted text) group ids. ## Open questions These are open, and nothing above depends on an answer except where it says so. -### 1. Is a formula and the measure it derives one thing with one id, or two things with two ids? +### 1. May a hand-written id carry the collision suffix? -**The question.** A formula is a derived measure. Take `Revenue` and `Cost`, stored -measures with ids `revenue` and `cost`, and add `Profit = Revenue - Cost`. Profit holds -no data: the formula computes its column. Two kinds of thing must name Profit by an -id: -- things that refer to the **measure**: another formula (`Margin = Profit / Revenue`), - each Profit cell's `CellKey`, a `hidden:` line, and ADR 008's cursor anchor; -- things that refer to the **formula**: `ReplaceFormula` and `RemoveFormula`. +**The question.** A hand-written id obeys every syntax rule of a generated id +(clause 4), and slugify turns `-` into `_`, so a slug never contains `-`. But a +generated id that collided carries a `-` suffix (`east-1`), and an empty slug +becomes `-` (`item-4`). Every save writes those ids into the file, so the +next load reads them as hand-written. -Do they use the same id? - -**Why it matters.** It decides the `CellKey` type (clause 2), the formula line's -shape (clause 5), `RecordHash` (clause 8), the `_Measure` panel rows (clause 6) and -what `.improv` must write. +**Why it matters.** If a hand-written id may never contain `-`, the app refuses to load +a file it saved itself. **Options.** -- **(a) One id.** The formula *is* the measure. Profit is `profit` everywhere, with - its display on the formula, and the line is `- Profit(profit) = revenue - cost`. - - A Profit cell's coordinate is `ItemHandle::FormulaTarget(profit)`. - - The cost: a stored `Profit` column and a formula `Profit` (merged into one measure - today by `measure_item_names`) would have two ids, and one must be chosen to key - cells. - - `RemoveFormula` removes the measure. - - The rare dimension target (an item `tax` plus formula `tax-1`) works differently. - - ADR 008 needs `ItemHandle`. - - This is the smallest change from the text above. -- **(b)** is (a) with the dimension line spelled `- Tax(tax-1)(tax) = … [customcat]` - (display, formula id, item id), so that the target position always holds a display. -- **(c) Two ids.** Profit is a `_Measure` item `profit`, and its formula is `profit-1`. - The line is `- profit(profit-1) = revenue - cost`. - - `CellKey` returns to `(CategoryId, ItemId)` and `ItemHandle` goes. - - A same-named stored measure is simply the item the formula attaches to. - - Measure and dimension targets work alike. - - The cost: the file must newly write the item and its display (today a formula - target is rebuilt from `## Formulas`). - - `RemoveFormula` leaves an item with no formula and maybe no data. - - ADR 009's `_Measure` fallback in `find_item_category` goes. +- **(a)** The syntax of an id is a slug, optionally followed by `-` and a number: + `[a-z][a-z0-9]*(_[a-z0-9]+)*` then optionally `-[0-9]+` (which also covers + `-`). A `-` anywhere else is refused. +- **(b)** Change the suffix to avoid `-`, for example `east_1`. That collides with a + display such as `East 1`, which also slugs to `east_1`, so the counter must advance + past it. -Also to settle: whether the writer keeps omitting the `[_measure]` tag on a `_Measure` -formula line, as today's writer does. +**Author's recommendation:** (a). It keeps the counter's suffix visibly distinct from +anything a display can slug to. -**Recommendation:** none. Both (a) and (c) model a derived measure, and the choice is a -product judgement about whether a formula and its column are one thing. +### 2. Does "alphanumeric" mean ASCII? -### 2. What may a formula refer to? - -**The question.** Formulas may refer to things in a typed context. An example is a -function counting how often a letter appears in a category value. Today a formula -reference (`Expr::Ref`) names an item or a formula target. Three things are open: -- (i) may a bare id in formula text also name a category, a view or a group, with - its kind given by where it appears? -- (ii) does a text function read an item's display (`Café`) or its id (`caf`)? They - differ, because slugify folds case and strips; -- (iii) must this ADR specify that, or only keep ids unique so it can be added later? - -**Why it matters.** It decides whether the formula grammar and evaluator change with -this ADR. - -**Author's recommendation:** (iii). The formula language has no string function -today, and global uniqueness already makes a reference to any kind unambiguous. - -### 3. Does ids-first matching outlive the interim resolver? - -**The question.** Clause 12's resolver matches typed text against ids before displays, -and ids are preferred wherever there is ambiguity. Once a view-layer autocomplete -turns display text into ids, does the preference carry into it? +**The question.** Slugify replaces every non-alphanumeric character with `_`. Unicode +counts `é`, `ü` and `日` as alphanumeric, but an id's alphabet is `a`–`z`, `0`–`9`, `_` +(and `-`). **Options.** -- **(a) Yes.** Wherever typed text could name both an id and a display, the id wins, - including in the autocomplete's ranking. -- **(b) No.** Once the autocomplete exists, nothing is matched at commit, so the rule - ends with the interim resolver. +- **(a) ASCII.** A non-ASCII letter becomes `_`: `Café` → `caf`, `Ü` → `-`, + `Crème Brûlée` → `cr_me_br_l_e`. +- **(b) Unicode, then drop what is outside the alphabet.** A non-ASCII letter + disappears without splitting the word: `Café` → `caf`, `Crème Brûlée` → + `crme_brle`. +- **(c) Unicode, then transliterate** (`é` → `e`). `Café` → `cafe`, `Crème Brûlée` → + `creme_brulee`. This needs a transliteration table that passes the `wasm` gate. -**Recommendation:** none. It matters only once an autocomplete is designed. +**Why it matters.** It decides every id generated from a non-English display. -### 4. May anything other than a client resolve a display name? +**Author's recommendation:** (a): it is the simplest reading of "non-alphanumeric +characters are replaced", and it needs no table. -**The question.** Only a client accepts a display as a reference (clause 6). Two -places would resolve displays outside one: -- the **old-version loader**, which resolves an old file's references by display - (clause 5), because in an old file the name is the only identity; -- **headless `improvise cmd` and `improvise script`**, which run command lines naming - things by display, with no client attached. +### 3. How does a file record the virtual categories, if virtual ids are never written? -**Why it matters.** If neither is allowed, old files cannot be migrated and the -headless CLI needs a client layer of its own. +**The question.** Virtual ids are never written to `.improv` (clause 1). But today's +files record three things under the virtual categories (`web/demo.improv` has all of +them): +- the virtual categories' axis placements, such as `_Measure: column` and + `_Index: none`; +- the stored measures, under `## Category: _Measure`; +- each data key's measure, as `_Measure=Revenue`. -**Author's recommendation:** allow both. The loader is migrating a file, not a person -referring to something. The CLI is the UI layer of its own invocation and converts at -its edge, as a client does. - -### 5. Does the `_` collapse come after the strip? - -**The question.** Clause 4 collapses runs of `_` (step 4) after stripping characters -outside the alphabet (step 3). - -**Why it matters.** `Time \u{200B} Period` (a zero-width space between two spaces) -gives `time_period` only if the collapse comes after the strip. Before it, the -zero-width character still separates the two `_`, and the result is `time__period`. - -**Options:** (a) after the strip, as written; (b) before it. -**Author's recommendation:** (a). - -### 6. Which entities share the reserved leading `_`? - -**The question.** A leading `_` is reserved for virtual categories, measures and other -entities. -- (i) Does "measures" mean virtual measures (none exists today), the `_measure` - category, or something else? -- (ii) Which "other entities" are covered? In particular, are the virtual views, - `_records` and `_drill` (clause 1), among them? - -**Why it matters.** It fixes `VIRTUAL_IDS` and what `claim` refuses. - -**Author's recommendation:** (ii) yes, as clause 1 is written. Otherwise `_Drill` -slugs to `drill` and can collide with a user's view. - -### 7. Are records mode's positions under `_Index` and `_Dim` ids? - -**The question.** Records mode gives each row a synthetic coordinate under `_Index`, -and each column one under `_Dim` (a column index, gotcha 1). They are never persisted -and never typed. +**Why it matters.** Every document with data has measures, so every file needs some +way to say which measure a cell holds and where `_Measure` sits. **Options.** -- **(a)** Outside the id namespace: a position `0` under `_index` does not collide - with a user's item `0`. -- **(b)** Inside it, spelled so that it cannot collide (for example `_index-0`), under - the reservation of a leading `_`. -- **(c)** Something else. +- **(a) Reserved keywords.** The file keeps fixed spellings for the virtual categories + (`_Measure`, `_Index`, `_Dim`, or new lower-case keywords). They are part of the + format, not ids, so no id is written. +- **(b) Write the virtual ids** (`_measure: column`, `_measure=revenue`), an exception + to "never written". +- **(c) Leave the virtual categories implicit.** Their axis placements take defaults, + the measures section has no category header, and a data key's measure is written + without its category. -**Why it matters.** Under (b), uniqueness is checked for synthetic coordinates, and -`claim`'s exact match on reserved ids must change. +**Author's recommendation:** (a). The file needs these facts, and a keyword says so +without turning a virtual id into file content. -**Recommendation:** none. +### 4. How does a dimension formula's line spell its target, and is the `[_measure]` tag written? -### 8. How do the virtual categories move to their lower-case ids? +**The question.** A measure formula's line is `- Profit(profit) = revenue - cost`: the +measure's display, then its id (clause 5). A dimension formula (the rare case) has an +item id and a separate formula id. Its line may be: +- **(a)** `- tax(tax-1) = … [customcat]`: the item id in the target position. The + target position then holds a display for a measure and an id for a dimension. +- **(b)** `- Tax(tax-1)(tax) = … [customcat]`: display, formula id, item id. The target + position always holds a display. -**The question.** Today `_Index`, `_Dim` and `_Measure` are keys in the model, on the -wire, in `ViewState` and in `.improv` files. For example, `web/demo.improv` has -`_Measure: column`, `_Index: none`, `## Category: _Measure` and `_Measure=Revenue`. -Under clause 1 they become `_index`, `_dim` and `_measure`, with the displays -unchanged. The proposed migration: -- an old file's `_Measure` resolves to `_measure` through its display at load, like - any name, and the first save writes `_measure: column` and `_measure=revenue`; -- a new-version file must spell the id, and `_Measure` there is refused (ids are - lower case); -- the header becomes `## Category: _Measure(_measure)`; -- every string literal naming them in code, on the wire and in the browser client - moves in the re-key step. +Today's writer omits the category tag for a `_Measure` formula. Should the new writer +keep omitting `[_measure]`? -**Why it matters.** Every shipped document and much test code spells these names. +**Why it matters.** It fixes the formula-line grammar and its generator strategy. -**Author's recommendation:** the migration above. +**Author's recommendation:** (b), so one position never holds two kinds of thing; +keep omitting `[_measure]`, which also keeps the virtual id out of the file +(question 3). -### 9. Does a group also carry a display name? +### 5. Which headless entry points are scripts? -**The question.** A group's identifier is an id, not a friendly name (clause 1). Does -a group also have a separate display, as categories, items and views do? +**The question.** User interfaces accept display names, and scripts refer to things by +id only (clause 6). The headless front ends are `improvise cmd ''`, which runs +one or more command lines given on the command line, and `improvise script `, +which runs a file of them. -**Why it matters.** -- If it does not, the grid and panels show the id: `Coastal East` would appear as - `coastal_east`. -- If it does, groups need a display rename like everything else, and the file - declares both. -- The code is already moving: PR #336 (improvise-cq1q.12, open) adds a private - display to `Group`, beside its key. +**Why it matters.** It decides whether `improvise cmd 'add-item Region East'`-style +invocations that name existing things by display keep working. -**Options:** (a) yes, a display beside the id; (b) no, the id is all a group has. -**Author's recommendation:** (a), for consistency with every other id-bearing kind and -so that users see what they typed. +**Options.** +- **(a)** `improvise cmd` is a user interface and resolves displays; a script file is + a script. +- **(b)** Both are scripts: only the TUI and browser clients resolve displays. -### 10. How do groups carry ids? - -**The question.** A group's identifier is an id. Not yet decided: -- (i) **Namespace.** Is a group id unique in the whole document, like every other id, - or only within its category? -- (ii) **Generation.** Is it the slug of a display (if question 9 gives it one), with - `group` as an empty slug's `` and the shared counter for collisions? -- (iii) **The file.** For example a definition `> Coastal(coastal)`, a membership - `- Coastal_East(coastal_east)[coastal]`, and a parent group (group hierarchies, - improvise-felo.6) named by id. Does a `collapsed:` line hold the group id with or - without its category id? -- (iv) **The wire.** Does `AddItemInGroup`'s `group` name an existing group by id, - create one by display, or either? Is there a `RenameGroup { group, to }`? -- (v) **The frame.** `AxisFrame`'s `FrameGroup` carries a group name today. What does - it carry? - -**Why it matters.** Clauses 2, 5, 6 and 7 cannot be implemented for groups without -these answers. - -**Author's recommendation:** (i) document-wide, as every other id is; (ii) slugify, -like every generated id. No recommendation on (iii)–(v). - -### 11. What may a hand-written id look like at its ends, and do slugify's trims repeat? - -**The question.** No id may be `-`, and no id begins or ends with `-`. -- (i) Is a hand-written id that begins or ends with `-` refused at load, as one - beginning with `_` is? -- (ii) Slugify drops a trailing `_`. Is a hand-written id ending in `_` refused too? -- (iii) Clause 4's step 5 repeats until neither end holds `_` or `-`, so `-_x` → `x`. - One pass of each would leave `_x`, which is reserved, or `x_`. Is repeating right? - -**Why it matters.** It decides what `IdAllocator::claim` refuses. Today it accepts -`-`, `-x` and `x-`. - -**Author's recommendation:** (i) yes; (ii) yes, so that a hand-written id obeys the -same shape as a generated one; (iii) repeat. +**Author's recommendation:** (a). A person types `improvise cmd` lines as they type +`:` lines in the TUI; a script file is a program that should not break when something +is renamed. diff --git a/context/code/core.md b/context/code/core.md index ffcee2e4..5969e0f3 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1715,11 +1715,12 @@ to `east_`. Only *leading* underscores are removed (step 4). **ADR 007's proposed amendment changes these rules, and this code predates it.** The amendment (PR #326) is not yet approved. Its clause 4: -- each whitespace character becomes `_`, not only U+0020; -- runs of `_` collapse after the strip, so `a_ b` and `a__b` both slug to `a_b` - and `Time \u{200B} Period` to `time_period`; -- leading and trailing `_` and `-` are removed, so `East ` → `east`, `-East-` → - `east`, and `-` alone takes `-`. +- every non-alphanumeric character becomes `_` (not only U+0020, and `-` too), + so `Q1-2024` slugs to `q1_2024` and `Region(code)` to `region_code`; +- runs of `_` collapse, so `a_ b` and `a__b` both slug to `a_b`; +- leading and trailing `_` are removed, so `East ` → `east` and `-East-` → `east`; +- `n` is prepended to a slug that starts with a digit, so `2024` → `n2024`; +- a hand-written id must obey every syntax rule of a generated one. This section, `slugify`, and the tests `slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and @@ -1750,8 +1751,8 @@ the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading `_` (only `_index`, `_dim` and `_measure` may carry one, per clause 1, and the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other id may start with `_`"), and -duplicates. It accepts `-`, `-x` and `x-`. ADR 007's open question 11 asks -whether a hand-written id like that must be refused. It never renames. +duplicates. It accepts `-`, `-x`, `x-` and `2024`, all of which the amendment +refuses (a hand-written id obeys the syntax of a generated one). It never renames. Refusing is the loader's job, and the loader must name both lines of a duplicate; `ClaimError` carries only the id, so the loader keeps the line numbers. The virtual views `_records` and `_drill` are From 2dfe088668720b6338f5fd32a58bbdacbc549a72 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:05:01 -0700 Subject: [PATCH 067/106] docs(adr): state that a hand-written id may carry the - suffix, and keep slugs in the id alphabet (improvise-cq1q.1) Codex's review of 6e467878. The suffix question followed from two decided rules (collisions take -; a hand-written id obeys the generated syntax), so clause 4 now states the id syntax and the question is deleted. Step 2 says the slug ends in a-z, 0-9 and _ whichever reading of "alphanumeric" is chosen. The remaining questions are renumbered 1-4. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 56 ++++++++++------------------ 1 file changed, 20 insertions(+), 36 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 36e719e0..a8f54484 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -27,7 +27,7 @@ amendment is approved: | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | | 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | -Five questions are open (last section). +Four questions are open (last section). ## Context @@ -100,7 +100,7 @@ the reservation needs no check in the grammar. code, the wire, `ViewState` and the browser client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, and move to the lower-case ids in the re-key step (clause 11). How the file records what it records today under `_Measure`, - `_Index` and `_Dim` is open question 3. + `_Index` and `_Dim` is open question 2. - **[Amendment] Records mode's synthetic positions are values, not ids.** The row coordinate under `_index` and the column coordinate under `_dim` (a column index, gotcha 1) are values of those virtual categories. They are never persisted or typed, @@ -198,7 +198,9 @@ thing's id never depends on how it was made. An id is lower case. pass the `wasm` gate, because `improvise-core` is in the browser bundle. 2. **Replace every character that is not alphanumeric with `_`.** That includes spaces, tabs and every other whitespace, punctuation and `-`. Whether - "alphanumeric" means ASCII only is open question 2. + "alphanumeric" means ASCII only is open question 1. Whichever it decides, the slug + ends up in `a`–`z`, `0`–`9` and `_`: a Unicode reading needs a further step that + drops or transliterates what is outside that alphabet, and question 1 picks it. 3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 or were in the display. 4. **Remove every leading and trailing `_`.** A leading `_` is reserved for virtual @@ -210,7 +212,9 @@ thing's id never depends on how it was made. An id is lower case. So a generated id is a lower-case letter followed by letters, digits and single underscores, never ending in `_`, plus, only when it collided, a `-` suffix; or -`-`. Whether a hand-written id may carry that suffix is open question 1. +`-`. The syntax of an id is therefore a slug, +`[a-z][a-z0-9]*(_[a-z0-9]+)*`, optionally followed by `-` and a number; a `-` anywhere +else never occurs. Examples under the amendment, with "alphanumeric" read as ASCII: @@ -263,9 +267,11 @@ every data key and reference in that file spells it. So: - **two identical hand-written ids in one file are refused at load**, even when they declare different kinds of thing. Renaming one would corrupt the data section, and a file with a duplicate cannot be resaved safely. The error names both lines. -- **[Amendment] A hand-written id obeys every syntax rule of a generated id**: lower - case, never starting with `_` or a digit, never ending in `_`, no runs of `_`, and no - character slugify would replace. One that does not is refused at load (clause 5). +- **[Amendment] A hand-written id obeys every syntax rule of a generated id**: a slug + (lower case, starting with a letter, single underscores, not ending in `_`), + optionally followed by the `-` suffix. So `east-1` and `item-4`, which every save + writes, load back; `-east`, `east-`, `2024` and `_draft` do not. One that breaks the + syntax is refused at load (clause 5). **Reuse across a save and reload is accepted.** Retired ids are not persisted. So remove `East`, save, reload and add `East`, and the new item gets `east` again. An @@ -366,7 +372,7 @@ lose its axis line to the view's number format, and the new-version arm puts `=`. For a measure formula it is also the measure's id. - **For a measure formula, `` is the measure's display**, held on the `Formula`: `- Profit(profit) = revenue - cost`. How a dimension formula's line spells - its target, and whether the `[_measure]` tag is written, is open question 4. + its target, and whether the `[_measure]` tag is written, is open question 3. - **The expression is over ids**, spelled through `quote::quote_name`. A reference to a formula target is the id text of its `ItemHandle`: the item id outside `_Measure`, the formula's (and measure's) id for a `_Measure` target. @@ -434,7 +440,7 @@ display that would otherwise read as display plus id: a reference. - **[Amendment] Scripts refer to things by id.** A script that names something by its display fails. A script gets a display only through a dedicated function such as - `display_for(id)`. Which headless entry points count as scripts is open question 5. + `display_for(id)`. Which headless entry points count as scripts is open question 4. - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. A create-then-act compound becomes **one command**: `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes @@ -709,29 +715,7 @@ three things: These are open, and nothing above depends on an answer except where it says so. -### 1. May a hand-written id carry the collision suffix? - -**The question.** A hand-written id obeys every syntax rule of a generated id -(clause 4), and slugify turns `-` into `_`, so a slug never contains `-`. But a -generated id that collided carries a `-` suffix (`east-1`), and an empty slug -becomes `-` (`item-4`). Every save writes those ids into the file, so the -next load reads them as hand-written. - -**Why it matters.** If a hand-written id may never contain `-`, the app refuses to load -a file it saved itself. - -**Options.** -- **(a)** The syntax of an id is a slug, optionally followed by `-` and a number: - `[a-z][a-z0-9]*(_[a-z0-9]+)*` then optionally `-[0-9]+` (which also covers - `-`). A `-` anywhere else is refused. -- **(b)** Change the suffix to avoid `-`, for example `east_1`. That collides with a - display such as `East 1`, which also slugs to `east_1`, so the counter must advance - past it. - -**Author's recommendation:** (a). It keeps the counter's suffix visibly distinct from -anything a display can slug to. - -### 2. Does "alphanumeric" mean ASCII? +### 1. Does "alphanumeric" mean ASCII? **The question.** Slugify replaces every non-alphanumeric character with `_`. Unicode counts `é`, `ü` and `日` as alphanumeric, but an id's alphabet is `a`–`z`, `0`–`9`, `_` @@ -751,7 +735,7 @@ counts `é`, `ü` and `日` as alphanumeric, but an id's alphabet is `a`–`z`, **Author's recommendation:** (a): it is the simplest reading of "non-alphanumeric characters are replaced", and it needs no table. -### 3. How does a file record the virtual categories, if virtual ids are never written? +### 2. How does a file record the virtual categories, if virtual ids are never written? **The question.** Virtual ids are never written to `.improv` (clause 1). But today's files record three things under the virtual categories (`web/demo.improv` has all of @@ -777,7 +761,7 @@ way to say which measure a cell holds and where `_Measure` sits. **Author's recommendation:** (a). The file needs these facts, and a keyword says so without turning a virtual id into file content. -### 4. How does a dimension formula's line spell its target, and is the `[_measure]` tag written? +### 3. How does a dimension formula's line spell its target, and is the `[_measure]` tag written? **The question.** A measure formula's line is `- Profit(profit) = revenue - cost`: the measure's display, then its id (clause 5). A dimension formula (the rare case) has an @@ -794,9 +778,9 @@ keep omitting `[_measure]`? **Author's recommendation:** (b), so one position never holds two kinds of thing; keep omitting `[_measure]`, which also keeps the virtual id out of the file -(question 3). +(question 2). -### 5. Which headless entry points are scripts? +### 4. Which headless entry points are scripts? **The question.** User interfaces accept display names, and scripts refer to things by id only (clause 6). The headless front ends are `improvise cmd ''`, which runs From 8405f5c18baa0ba357bc4929774a18cd38bb2bdf Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:18:17 -0700 Subject: [PATCH 068/106] docs(adr): fold the user's answers on ASCII ids, virtual categories and scripts into ADR 007 (improvise-cq1q.1) - Id alphanumerics are ASCII and include -: slugify maps everything outside a-z, 0-9 and - to _, collapses _, trims _ and - at both ends, prefixes n before a digit; the id syntax, examples and the suffix / hyphenated-slug collision rule follow. - A file never declares a virtual category and refers to one by its id. - Only CSV import and the .improv loaders read display names in input text; command-language text, from any source, names things by id. - The formula line is shown before and after; the dimension spelling and the [_measure] tag follow from the rules, so that question is deleted. - The Opus review of 2dfe0886: each measure is keyed by one handle, and the shared-display case moves to an open question; views join the things a formula may name. Open questions are now three: a formula and a stored measure with the same display, where stored measures are declared, and whether interactive typing keeps a display stage. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 321 +++++++++++++++------------ context/code/core.md | 12 +- 2 files changed, 188 insertions(+), 145 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index a8f54484..5f09f406 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -19,15 +19,16 @@ amendment is approved: | clause | accepted rule | the amendment proposes | |---|---|---| | 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | -| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; virtual ids are never written to `.improv` | +| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; a file never declares a virtual category and refers to one by its id (`_measure`) | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | -| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every non-alphanumeric character becomes `_`; runs of `_` collapse; leading and trailing `_` are removed; `n` is prepended to a slug that starts with a digit | +| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended to a slug that starts with a digit | | 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` and may begin with `_` only if reserved | a hand-written id obeys every syntax rule of a generated id | +| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders turn display names in input text into ids; command-language text, from any source, names things by id | | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | | 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | -Four questions are open (last section). +Three questions are open (last section). ## Context @@ -64,7 +65,8 @@ human name moves to a new `display` field. existing thing. A reference to the stable identity cannot glitch when something is renamed or replaced. - **Display names are for people.** They appear on screen, are typed into - minibuffers, and are turned into ids at the input edge. + minibuffers, and are turned into ids at the input edge. **[Amendment]** Clause 6 narrows + which readers turn display text into ids. ### 1. What carries an identity @@ -96,15 +98,21 @@ have ids `_records` and `_drill`, displayed `_Records` and `_Drill`; they are ad on the wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could collide with a user's view. Slugify never produces a leading `_` (clause 4), so the reservation needs no check in the grammar. -- **[Amendment] Virtual ids are lower case and are never written to `.improv`.** The - code, the wire, `ViewState` and the browser client spell the virtual categories - `_Index`, `_Dim` and `_Measure` today, and move to the lower-case ids in the re-key - step (clause 11). How the file records what it records today under `_Measure`, - `_Index` and `_Dim` is open question 2. +- **[Amendment] Virtual ids are lower case.** The code, the wire, `ViewState` and the + browser client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, + and move to the lower-case ids in the re-key step (clause 11). +- **[Amendment] A file never declares a virtual category; it refers to one by its + id.** The virtual categories exist in every document, so they are not required in + the file. Where a line refers to one, it spells the virtual id: + `_measure: column`, `_index: none`, `region=east, _measure=revenue = 1200`. An + old-version file's `_Measure` resolves through its display at load, like every name + in an old file (clause 5), and the first save writes `_measure`. Where the file + declares the stored measures themselves (`Revenue`, `Cost`) is open question 2. - **[Amendment] Records mode's synthetic positions are values, not ids.** The row - coordinate under `_index` and the column coordinate under `_dim` (a column index, - gotcha 1) are values of those virtual categories. They are never persisted or typed, - take no part in id uniqueness, and cannot collide with a user's item. + coordinate under `_index` is a value of that virtual category. So, by the same + reasoning, is the column coordinate under `_dim` (a column index, gotcha 1). They are + never persisted or typed, take no part in id uniqueness, and cannot collide with a + user's item. **Formulas and measures.** **[Amendment]** A measure is the normal formula target (`profit = revenue - cost`). A dimension (a category other than `_Measure`) is a rare @@ -112,21 +120,23 @@ one, for a derived dimension. - **A `_Measure` target has no item of its own**, and its handle is its formula's `FormulaId`. So the category panel's `_Measure` rows use `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). -- **[Amendment] A formula is a kind of measure, so a `_Measure` formula's id is the - measure's id.** A measure has an id and a display, as everything else does. - `Profit = Revenue - Cost` is one thing, `profit`: another formula refers to it as - `profit`, each Profit cell's `CellKey` holds `profit` under `_measure`, and - `ReplaceFormula` and `RemoveFormula` address `profit`. A formula that derives a - measure already stored under that display takes the stored measure's id, so the - column has one id, not two. Removing the formula removes a measure that exists only - through it; a measure that also holds stored data stays, as a stored measure. +- **[Amendment] A formula is a kind of measure, so there is no distinction between a + `_Measure` formula's id and its measure's id.** A measure has an id and a display, + as everything else does. `Profit = Revenue - Cost` is one thing, `profit`: another + formula refers to it as `profit`, each Profit cell's `CellKey` holds `profit` under + `_measure`, and `ReplaceFormula` and `RemoveFormula` address `profit`. +- **[Amendment] Each measure's cells are keyed by one handle.** A stored measure is + keyed `ItemHandle::Item(id)`. A measure that exists only through its formula is + keyed `ItemHandle::FormulaTarget(id)`. The id is the measure's either way, and no + column is keyed by both. What happens when a formula's measure and a stored measure + share a display, and what removing that formula does, is open question 1. - **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, and its formula has a separate `FormulaId`. A formula whose generated id would collide takes a suffix like any other generated id (clause 4): a formula `Tax` in `CustomCat` gets item id `tax`, then formula id `tax-`. - **[Amendment] Formulas may refer to anything by id.** A function in a formula may - accept any id-bearing thing (a category, an item, a measure, a formula, a group), - referred to by its id. Whether a given function takes the thing itself or its value + accept any id-bearing thing (a category, an item, a measure, a formula, a view, a + group), referred to by its id. Whether a given function takes the thing itself or its value (an item's display, say) is part of that function's signature. Global uniqueness makes every such reference unambiguous, and a function may read any part of a record, so nothing further is needed here. @@ -196,27 +206,27 @@ thing's id never depends on how it was made. An id is lower case. 1. **Casefold, with Unicode full case folding**, so `Straße` → `strasse`. Rust's `str::to_lowercase` is not enough (`ß` stays `ß`). The folding table or crate must pass the `wasm` gate, because `improvise-core` is in the browser bundle. -2. **Replace every character that is not alphanumeric with `_`.** That includes - spaces, tabs and every other whitespace, punctuation and `-`. Whether - "alphanumeric" means ASCII only is open question 1. Whichever it decides, the slug - ends up in `a`–`z`, `0`–`9` and `_`: a Unicode reading needs a further step that - drops or transliterates what is outside that alphabet, and question 1 picks it. +2. **Replace every character outside `a`–`z`, `0`–`9` and `-` with `_`.** The id's + alphanumerics are ASCII and include `-`. Every other character, including spaces, + tabs and other whitespace, punctuation and every non-ASCII letter, becomes `_`. 3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 or were in the display. -4. **Remove every leading and trailing `_`.** A leading `_` is reserved for virtual - ids (clause 1), and a trailing one is dropped. +4. **Remove every leading and trailing `_` and `-`**, repeating until neither end + holds either. A leading `_` is reserved for virtual ids (clause 1), a trailing `_` + is dropped, and no id begins or ends with `-`. 5. **If the slug starts with a digit, prepend `n`.** An id is never purely numeric and never starts with a digit, so `2024` becomes `n2024`. 6. **If nothing is left, use `-`**, such as `category-1` or `item-4`. `n` - comes from the shared counter below. + comes from the shared counter below. A display of only `-`, only `_` or only + non-ASCII letters ends here, because `-` alone is not an id. -So a generated id is a lower-case letter followed by letters, digits and single -underscores, never ending in `_`, plus, only when it collided, a `-` suffix; or -`-`. The syntax of an id is therefore a slug, -`[a-z][a-z0-9]*(_[a-z0-9]+)*`, optionally followed by `-` and a number; a `-` anywhere -else never occurs. +**The syntax of an id** is therefore: a lower-case ASCII letter, then any of `a`–`z`, +`0`–`9`, `-` and single underscores, never ending in `_` or `-`: +`[a-z]([a-z0-9-]|_(?=[a-z0-9-]))*`, not ending in `-`. Slugify's output already has +that shape, so slugifying an id gives the same id back. The collision suffix `-` +(below) keeps the shape too, because it ends in a digit. -Examples under the amendment, with "alphanumeric" read as ASCII: +Examples under the amendment: | display | id | |---|---| @@ -224,24 +234,40 @@ Examples under the amendment, with "alphanumeric" read as ASCII: | `Time Period`, `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` | | `Coastal_East` | `coastal_east` | | `a_ b`, `a__b` | `a_b` | -| `Q1-2024`, `Q1 - 2024` | `q1_2024` | +| `Q1-2024` | `q1-2024` | +| `Q1 - 2024` | `q1_-_2024` | +| `East-1` | `east-1` | | `East `, `East_`, `-East-` | `east` | | `_Draft` | `draft` | | `Region(code)` | `region_code` | | `Café` | `caf` | +| `Crème Brûlée` | `cr_me_br_l_e` | | `Straße` | `strasse` | | `2024`, `_2024` | `n2024` | -| `2024 Budget` | `n2024_budget` | +| `2024-Q1` | `n2024-q1` | | `Ü`, `-`, `%` | `-` | -Under the accepted steps, the ones that differ are `Q1-2024` → `q1-2024`, -`Q1 - 2024` → `q1_-_2024`, `East ` → `east_`, `a_ b` → `a__b`, `Region(code)` → -`regioncode`, `2024` → `2024` and `-` → `-`. +Under the accepted steps, the ones that differ are `East ` → `east_`, `a_ b` → +`a__b`, `Time\tPeriod` → `timeperiod`, `Time \u{200B} Period` → `time__period`, +`-East-` → `-east-`, +`Region(code)` → `regioncode`, `Crème Brûlée` → `crme_brle`, `2024` → `2024`, +`2024-Q1` → `2024-q1` and `-` → `-`. **Collisions.** `Time Period` and `Time_Period` slug alike, and whichever comes later -takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, and `Q1-2024` and -`Q1 2024`, which slug alike only under the amendment's steps. `East` in `Region` and -`East` in `Direction` also slug alike, and ids are global, so the second is `east-`. +takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, which slug alike +only under the amendment's steps. `East` in `Region` and `East` in `Direction` also +slug alike, and ids are global, so the second is `east-`. + +**[Amendment] A suffix and a hyphenated slug can look alike.** The display `East-1` +slugs to `east-1`, which is also what the counter's suffix may make of `east`. They +never collide, because every id a document holds is taken, whoever made it: +- if `east-1` is already held (as a slug or as a suffix), a display `East-1` that slugs + to it is a collision, and becomes `east-1-`; +- if the counter's candidate `east-` is already held, the counter advances until a + candidate is free. + +So an id says nothing about how it was made: `east-1` may be a suffixed `east` or the +slug of `East-1`. - **Any colliding generated id gets a suffix**, whatever kind of thing holds the slug: on import, when an id-less file loads, and when the host creates something in a session. An id that does not collide stays bare. @@ -267,11 +293,9 @@ every data key and reference in that file spells it. So: - **two identical hand-written ids in one file are refused at load**, even when they declare different kinds of thing. Renaming one would corrupt the data section, and a file with a duplicate cannot be resaved safely. The error names both lines. -- **[Amendment] A hand-written id obeys every syntax rule of a generated id**: a slug - (lower case, starting with a letter, single underscores, not ending in `_`), - optionally followed by the `-` suffix. So `east-1` and `item-4`, which every save - writes, load back; `-east`, `east-`, `2024` and `_draft` do not. One that breaks the - syntax is refused at load (clause 5). +- **[Amendment] A hand-written id obeys every syntax rule of a generated id** (the + syntax above). So `east-1`, `item-4` and `q1_-_2024` load; `-east`, `east-`, `2024`, + `a__b` and `_draft` are refused at load (clause 5). **Reuse across a save and reload is accepted.** Retired ids are not persisted. So remove `East`, save, reload and add `East`, and the new item gets `east` again. An @@ -286,9 +310,9 @@ promised (improvise-8tk5). **Built code.** `crates/improvise-core/src/model/id.rs` (`slugify`, `IdAllocator`) implements the accepted steps, and `context/code/core.md` §20 describes them. Once the amendment is approved, a follow-up must: -- replace every non-alphanumeric character, not only spaces, with `_`, and collapse - `_` runs; -- trim a trailing `_`, and prepend `n` before a leading digit; +- replace every character outside `a`–`z`, `0`–`9` and `-`, not only spaces, with + `_` instead of stripping it, and collapse `_` runs; +- trim a trailing `_` and both ends' `-`, and prepend `n` before a leading digit; - make `claim` refuse every hand-written id a generated one could not be; - add the virtual views to `VIRTUAL_IDS`, and `IdKind::Group`. @@ -312,16 +336,17 @@ grammar spells it **with ranges**, because the generator drops `ASCII_ALPHANUMER `id_tag`, so it has no id. The error names the line; - a hand-written id beginning with `_`, other than a reserved one; - **[Amendment]** a hand-written id that breaks any syntax rule of a generated id - (clause 4). Virtual ids are never written to a file, so no hand-written id begins - with `_`. The grammar's `id_token` stays permissive, and the loader refuses, so the - error can name the line. + (clause 4). A file never declares a virtual category, so no declared id begins with + `_`; a reference may still spell a virtual id such as `_measure`. The grammar's + `id_token` stays permissive, and the loader refuses, so the error can name the line. **Inside formula text an id may still need quoting.** The formula grammar's `bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So the writer spells an id in formula text through `quote::quote_name`. An id equal to a keyword (`sum`, `if`, `where`, `is`) is written `|sum|`, and under the accepted rules an id beginning with a digit (`2024`) is written `|2024|`. **[Amendment]** No id begins -with a digit, so only keyword ids need quoting. +with a digit, so only keyword ids need quoting. A `-` inside an id needs none, because +the formula grammar's bare identifier already admits `-` after its first character. ```pest id_token = @{ ('a'..'z' | '0'..'9' | "_" | "-")+ } @@ -366,16 +391,39 @@ lose its axis line to the view's number format, and the new-version arm puts - a view whose number format is each axis keyword round-trips in a document where that view also holds a category `format` on an axis. -**[Amendment] A formula line** is -`- () = []`: -- **The parenthesised id is the formula's `FormulaId`**, between the target and the - `=`. For a measure formula it is also the measure's id. -- **For a measure formula, `` is the measure's display**, held on the - `Formula`: `- Profit(profit) = revenue - cost`. How a dimension formula's line spells - its target, and whether the `[_measure]` tag is written, is open question 3. +**[Amendment] A formula line** declares the formula: its target, its id, and its +expression over ids. Today's `web/demo.improv` and the format's own dimension example +are written: + +``` +## Formulas +- Profit = Revenue - Cost +- Tax = Revenue * 0.1 [CustomCat] +``` + +Under the amendment the same two formulas are written: + +``` +## Formulas +- Profit(profit) = revenue - cost +- tax(tax-1) = revenue * 0.1 [customcat] +``` + +- **A measure formula** (no category tag) is the normal case. The formula is the + measure (clause 1), so the line is the measure's only declaration: `Profit` is its + display, and `(profit)` is the id it shares with its formula. No `[_measure]` tag is + written, because a file does not spell a virtual category where it is not needed + (clause 1); a line without a tag targets `_measure`. +- **A dimension formula** (a `[]` tag) targets a real item, already + declared with its display in its category's section (`## Category: CustomCat(customcat)` + lists `- Tax(tax)`). So the line names the target by its item id, `tax`, and + `(tax-1)` is the formula's own id. The display is not repeated, because a display + lives in one place. +- **So the loader can always tell the two apart**: with a tag the target is an item + id; without one it is a measure's display. - **The expression is over ids**, spelled through `quote::quote_name`. A reference to a formula target is the id text of its `ItemHandle`: the item id outside `_Measure`, - the formula's (and measure's) id for a `_Measure` target. + the measure's (and its formula's) id for a `_Measure` target. - **A `WHERE`/`IS` value follows the same rule**. It is the id text of an `ItemHandle` in the filtered category, and the formula panel shows it as a display. - **A display in the target position that looks like `Name(x)` is pipe-quoted**, as in @@ -434,13 +482,17 @@ display that would otherwise read as display plus id: is the group's id, and a new group is created by its display under the same create-by-display rule; - `AddFormula { target_category: CategoryId, raw }`. -- **[Amendment] Only a user interface accepts a display name as a reference**, and it - converts the name to an id as early as it can. The TUI and browser clients are user - interfaces, and so is the headless command line. The wire never carries a display as - a reference. -- **[Amendment] Scripts refer to things by id.** A script that names something by its - display fails. A script gets a display only through a dedicated function such as - `display_for(id)`. Which headless entry points count as scripts is open question 4. +- **[Amendment] Only two readers turn display names in input text into ids: CSV + import and the `.improv` loaders.** They read text people wrote, where names are + all there is. The wire never carries a display as a reference. +- **[Amendment] Command-language text names things by id.** Anything that uses the + command language as a scripting language is a script, wherever the text comes from + (a script file, `improvise cmd`, another program). A script that names something by + its display fails. A script gets a display only through a dedicated function such as + `display_for(id)`. +- **[Amendment] Interactive typing prefers ids and previews displays** (clause 12). + Whether an interactive input still resolves typed text that matches no id by + display is open question 3. - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. A create-then-act compound becomes **one command**: `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes @@ -610,8 +662,9 @@ answers for those. preferred wherever there is ambiguity, now and once an autocomplete exists. - A typed name equal to an id names that id's thing, even where it is also another thing's display. - - Otherwise a display that exactly one thing in scope has names that thing. - - A display matching several things is refused, and the refusal lists their ids. + - Otherwise a display that exactly one thing in scope has names that thing, and a + display matching several things is refused with a list of their ids. Whether this + display stage stays is open question 3. - A name that resolves to nothing is refused, and the refusal names it. - Ids are matched exactly, with no case folding. - **Typing prefers ids and previews displays.** A view-layer autocomplete turns typed @@ -647,7 +700,8 @@ without resolving any name (improvise-hexb). (`east`, `east-1`), type `east` into an editor, in text not captured at open: - it must bind A, and the test must fail under displays-first; - `east-1` must bind B; -- a display that is nobody's id must still resolve by display. +- a display that is nobody's id must still resolve by display (pending open + question 3). A fixture where no display equals an id proves nothing. @@ -715,86 +769,73 @@ three things: These are open, and nothing above depends on an answer except where it says so. -### 1. Does "alphanumeric" mean ASCII? +### 1. A formula and a stored measure with the same display: one measure, and what does removal do? -**The question.** Slugify replaces every non-alphanumeric character with `_`. Unicode -counts `é`, `ü` and `日` as alphanumeric, but an id's alphabet is `a`–`z`, `0`–`9`, `_` -(and `-`). +**The question.** A document stores a `Profit` measure: an item of `_measure` with id +`profit`, display `Profit`, holding data. The user then adds the formula +`Profit = Revenue - Cost`. A formula is a kind of measure with one id (clause 1), and +displays are unique within their scope (clause 2), so this cannot simply make a second +`Profit`. Today `measure_item_names` merges the two into one column. + +**Why it matters.** It decides which handle keys the column (`Item(profit)` or +`FormulaTarget(profit)`, clause 1), what `RemoveFormula` leaves behind, and whether +documents that rely on today's merge keep loading. **Options.** -- **(a) ASCII.** A non-ASCII letter becomes `_`: `Café` → `caf`, `Ü` → `-`, - `Crème Brûlée` → `cr_me_br_l_e`. -- **(b) Unicode, then drop what is outside the alphabet.** A non-ASCII letter - disappears without splitting the word: `Café` → `caf`, `Crème Brûlée` → - `crme_brle`. -- **(c) Unicode, then transliterate** (`é` → `e`). `Café` → `cafe`, `Crème Brûlée` → - `creme_brulee`. This needs a transliteration table that passes the `wasm` gate. +- **(a) The formula attaches to the stored measure.** One measure, id `profit`, keyed + `Item(profit)`; the formula computes it. Removing the formula leaves the stored + measure and its data. +- **(b) Refuse it.** Adding a formula whose display is a stored measure's is refused, + as any duplicate display is. Existing files that hold both must be repaired at load. +- **(c) The formula replaces the stored measure.** Its stored data is dropped or kept + hidden. -**Why it matters.** It decides every id generated from a non-English display. +**Author's recommendation:** (a). It keeps today's behaviour, so existing documents +load unchanged, and a measure stays one thing with one id. -**Author's recommendation:** (a): it is the simplest reading of "non-alphanumeric -characters are replaced", and it needs no table. +### 2. Where does a file declare the stored measures? -### 2. How does a file record the virtual categories, if virtual ids are never written? +**The question.** A file never declares a virtual category (clause 1). But the stored +measures, such as `Revenue` and `Cost`, are items of `_measure`, and each needs a +declaration that carries its display and id. Today they are listed under +`## Category: _Measure`: -**The question.** Virtual ids are never written to `.improv` (clause 1). But today's -files record three things under the virtual categories (`web/demo.improv` has all of -them): -- the virtual categories' axis placements, such as `_Measure: column` and - `_Index: none`; -- the stored measures, under `## Category: _Measure`; -- each data key's measure, as `_Measure=Revenue`. +``` +## Category: _Measure +- Revenue, Cost +``` -**Why it matters.** Every document with data has measures, so every file needs some -way to say which measure a cell holds and where `_Measure` sits. +**Why it matters.** Without a declaration a measure has no display or id in the file, +and a new-version file refuses any undeclared id (clause 5). **Options.** -- **(a) Reserved keywords.** The file keeps fixed spellings for the virtual categories - (`_Measure`, `_Index`, `_Dim`, or new lower-case keywords). They are part of the - format, not ids, so no id is written. -- **(b) Write the virtual ids** (`_measure: column`, `_measure=revenue`), an exception - to "never written". -- **(c) Leave the virtual categories implicit.** Their axis placements take defaults, - the measures section has no category header, and a data key's measure is written - without its category. +- **(a) Keep a section for the virtual category's items only**, spelled by its id: + `## Category: _measure` with `- Revenue(revenue), Cost(cost)`. The section declares + items, not the category. +- **(b) A dedicated section**, such as `## Measures`, with the same item lines. +- **(c) Declare a measure where it is first used**, in a data key. A data key holds + only an id, so the display would be lost. -**Author's recommendation:** (a). The file needs these facts, and a keyword says so -without turning a virtual id into file content. +**Author's recommendation:** (b). It names what the section holds, and keeps the +virtual category out of the file entirely. -### 3. How does a dimension formula's line spell its target, and is the `[_measure]` tag written? +### 3. Does interactive typing still resolve a display name that matches no id? -**The question.** A measure formula's line is `- Profit(profit) = revenue - cost`: the -measure's display, then its id (clause 5). A dimension formula (the rare case) has an -item id and a separate formula id. Its line may be: -- **(a)** `- tax(tax-1) = … [customcat]`: the item id in the target position. The - target position then holds a display for a measure and an id for a dimension. -- **(b)** `- Tax(tax-1)(tax) = … [customcat]`: display, formula id, item id. The target - position always holds a display. +**The question.** Only CSV import and the `.improv` loaders turn display names in input +text into ids (clause 6). Interactive typing (the TUI's minibuffers and `:` line, the +formula editor, the browser) prefers ids and shows the display as a preview +(clause 12). Clause 12's resolver also has a display stage: typed text that matches no +id but is exactly one thing's display names that thing. Does that stage stay? -Today's writer omits the category tag for a `_Measure` formula. Should the new writer -keep omitting `[_measure]`? - -**Why it matters.** It fixes the formula-line grammar and its generator strategy. - -**Author's recommendation:** (b), so one position never holds two kinds of thing; -keep omitting `[_measure]`, which also keeps the virtual id out of the file -(question 2). - -### 4. Which headless entry points are scripts? - -**The question.** User interfaces accept display names, and scripts refer to things by -id only (clause 6). The headless front ends are `improvise cmd ''`, which runs -one or more command lines given on the command line, and `improvise script `, -which runs a file of them. - -**Why it matters.** It decides whether `improvise cmd 'add-item Region East'`-style -invocations that name existing things by display keep working. +**Why it matters.** Without it, a person typing `Revenue` in the formula editor must +type or pick the id `revenue`. With it, a typed display still works, and interactive +input is a third place that reads display names. **Options.** -- **(a)** `improvise cmd` is a user interface and resolves displays; a script file is - a script. -- **(b)** Both are scripts: only the TUI and browser clients resolve displays. +- **(a) Keep the display stage** for interactive typing only. Command-language text + from scripts never gets it. +- **(b) Ids only.** Interactive inputs match ids; displays appear only as previews and + autocomplete suggestions, which insert the id. -**Author's recommendation:** (a). A person types `improvise cmd` lines as they type -`:` lines in the TUI; a script file is a program that should not break when something -is renamed. +**Recommendation:** none. "Prefer ids" leaves room for (a); "only import and the +loaders read display names" points to (b). diff --git a/context/code/core.md b/context/code/core.md index 5969e0f3..533b02b5 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1715,10 +1715,12 @@ to `east_`. Only *leading* underscores are removed (step 4). **ADR 007's proposed amendment changes these rules, and this code predates it.** The amendment (PR #326) is not yet approved. Its clause 4: -- every non-alphanumeric character becomes `_` (not only U+0020, and `-` too), - so `Q1-2024` slugs to `q1_2024` and `Region(code)` to `region_code`; +- every character outside `a`–`z`, `0`–`9` and `-` becomes `_` instead of being + stripped (not only U+0020), so `Region(code)` slugs to `region_code`, while + `Q1-2024` keeps its `-`; - runs of `_` collapse, so `a_ b` and `a__b` both slug to `a_b`; -- leading and trailing `_` are removed, so `East ` → `east` and `-East-` → `east`; +- leading and trailing `_` and `-` are removed, so `East ` → `east`, `-East-` → + `east`, and `-` alone takes `-`; - `n` is prepended to a slug that starts with a digit, so `2024` → `n2024`; - a hand-written id must obey every syntax rule of a generated one. @@ -1751,8 +1753,8 @@ the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading `_` (only `_index`, `_dim` and `_measure` may carry one, per clause 1, and the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other id may start with `_`"), and -duplicates. It accepts `-`, `-x`, `x-` and `2024`, all of which the amendment -refuses (a hand-written id obeys the syntax of a generated one). It never renames. +duplicates. It accepts `-`, `-x`, `x-`, `a__b` and `2024`, all of which the +amendment refuses (a hand-written id obeys the syntax of a generated one). It never renames. Refusing is the loader's job, and the loader must name both lines of a duplicate; `ClaimError` carries only the id, so the loader keeps the line numbers. The virtual views `_records` and `_drill` are From ccc675188662e92c930f77ca2ad95a5fa081aa88 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:20:20 -0700 Subject: [PATCH 069/106] docs(adr): say plainly that a formula line's category tag is an id and _measure is implicit (improvise-cq1q.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 5f09f406..1e6fb637 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -419,6 +419,8 @@ Under the amendment the same two formulas are written: lists `- Tax(tax)`). So the line names the target by its item id, `tax`, and `(tax-1)` is the formula's own id. The display is not repeated, because a display lives in one place. +- **The tag is always an id**, the target category's (`[customcat]`, never + `[CustomCat]`), and `_measure` is implicit: it is never written as a tag. - **So the loader can always tell the two apart**: with a tag the target is an item id; without one it is a measure's display. - **The expression is over ids**, spelled through `quote::quote_name`. A reference to From 209270542ccf61f3eee819276127dcea78dc4cef Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:38:06 -0700 Subject: [PATCH 070/106] docs(adr): a formula never merges into a stored measure; it takes profit- (improvise-cq1q.1) Clause 1 states it, with what removal leaves. Open question 1 is replaced by the residue: two measures then share a display, which clause 2's uniqueness rule forbids. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 52 ++++++++++++++++------------ 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 1e6fb637..56cad442 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -21,6 +21,7 @@ amendment is approved: | 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | | 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; a file never declares a virtual category and refers to one by its id (`_measure`) | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | +| 1 | (silent; today `measure_item_names` merges a formula into a stored measure of the same name) | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended to a slug that starts with a digit | | 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` and may begin with `_` only if reserved | a hand-written id obeys every syntax rule of a generated id | @@ -126,10 +127,16 @@ one, for a derived dimension. formula refers to it as `profit`, each Profit cell's `CellKey` holds `profit` under `_measure`, and `ReplaceFormula` and `RemoveFormula` address `profit`. - **[Amendment] Each measure's cells are keyed by one handle.** A stored measure is - keyed `ItemHandle::Item(id)`. A measure that exists only through its formula is - keyed `ItemHandle::FormulaTarget(id)`. The id is the measure's either way, and no - column is keyed by both. What happens when a formula's measure and a stored measure - share a display, and what removing that formula does, is open question 1. + keyed `ItemHandle::Item(id)`. A formula's measure is keyed + `ItemHandle::FormulaTarget(id)`. The id is the measure's either way, and no column is + keyed by both. +- **[Amendment] A formula never merges into a stored measure.** If a stored measure + `profit` (displayed `Profit`) exists and the user adds `Profit = Revenue - Cost`, + the new formula is a second measure with a fresh id, `profit-` from the shared + counter (clause 4), like any colliding generated id. The stored measure keeps + `profit` and its data. Removing the formula removes `profit-` and leaves `profit` + untouched. Both are displayed `Profit`, which clause 2's display-uniqueness rule + does not allow; open question 1 asks how that is resolved. - **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, and its formula has a separate `FormulaId`. A formula whose generated id would collide takes a suffix like any other generated id (clause 4): a formula `Tax` in @@ -176,7 +183,9 @@ human-typed name resolves to exactly one id within its scope, and a rename to a display already in use there is refused. An item's scope is its category, so two categories may each have an item displayed `Revenue`. Formula text crosses categories, so a display may be ambiguous there. The ids differ, so the formula panel -shows the ids where a display is ambiguous. +shows the ids where a display is ambiguous. **[Amendment]** A formula and a stored +measure may end up sharing a display (clause 1); whether that is an exception to this +rule is open question 1. ### 3. Ids are interned symbols @@ -771,29 +780,26 @@ three things: These are open, and nothing above depends on an answer except where it says so. -### 1. A formula and a stored measure with the same display: one measure, and what does removal do? +### 1. Two measures with the same display: is that allowed? -**The question.** A document stores a `Profit` measure: an item of `_measure` with id -`profit`, display `Profit`, holding data. The user then adds the formula -`Profit = Revenue - Cost`. A formula is a kind of measure with one id (clause 1), and -displays are unique within their scope (clause 2), so this cannot simply make a second -`Profit`. Today `measure_item_names` merges the two into one column. +**The question.** A formula never merges into a stored measure (clause 1): with a stored +`Profit` (id `profit`) in the document, adding the formula `Profit = Revenue - Cost` +creates a second measure, `profit-1`. Both are displayed `Profit`. But clause 2 keeps +displays unique within their scope, and both measures are in `_measure`'s scope. -**Why it matters.** It decides which handle keys the column (`Item(profit)` or -`FormulaTarget(profit)`, clause 1), what `RemoveFormula` leaves behind, and whether -documents that rely on today's merge keep loading. +**Why it matters.** Under the uniqueness rule, adding that formula would be refused, +which contradicts clause 1. **Options.** -- **(a) The formula attaches to the stored measure.** One measure, id `profit`, keyed - `Item(profit)`; the formula computes it. Removing the formula leaves the stored - measure and its data. -- **(b) Refuse it.** Adding a formula whose display is a stored measure's is refused, - as any duplicate display is. Existing files that hold both must be repaired at load. -- **(c) The formula replaces the stored measure.** Its stored data is dropped or kept - hidden. +- **(a) Allow duplicate displays among measures only.** The formula panel already + shows ids wherever a display is ambiguous (clause 6), and typing prefers ids + (clause 12), so the two stay distinguishable. +- **(b) Drop display uniqueness everywhere.** Ids carry identity, and every place that + shows displays already falls back to ids when they clash. +- **(c) Keep uniqueness and give the new formula a different display**, such as + `Profit 1`, chosen by the host. -**Author's recommendation:** (a). It keeps today's behaviour, so existing documents -load unchanged, and a measure stays one thing with one id. +**Author's recommendation:** (a). It is the smallest change that makes clause 1 hold. ### 2. Where does a file declare the stored measures? From 4778d9741b627f4f47a9caedc499d3f8ec100920 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:39:23 -0700 Subject: [PATCH 071/106] docs(adr): virtual categories and measures are defined by criteria, never declared (improvise-cq1q.1) Their ids and displays are hardcoded; an old file's _Measure names are recognised, not migrated as declarations. Open question 2 narrows to where a new-version file declares the stored measures. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 50 ++++++++++++++++------------ 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 56cad442..7b2fda5c 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -19,7 +19,7 @@ amendment is approved: | clause | accepted rule | the amendment proposes | |---|---|---| | 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | -| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; a file never declares a virtual category and refers to one by its id (`_measure`) | +| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; virtual categories and measures are defined by criteria with hardcoded ids and displays, are never declared in a file, and are referred to by id (`_measure`) | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 1 | (silent; today `measure_item_names` merges a formula into a stored measure of the same name) | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | @@ -102,13 +102,18 @@ the reservation needs no check in the grammar. - **[Amendment] Virtual ids are lower case.** The code, the wire, `ViewState` and the browser client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, and move to the lower-case ids in the re-key step (clause 11). -- **[Amendment] A file never declares a virtual category; it refers to one by its - id.** The virtual categories exist in every document, so they are not required in - the file. Where a line refers to one, it spells the virtual id: - `_measure: column`, `_index: none`, `region=east, _measure=revenue = 1200`. An - old-version file's `_Measure` resolves through its display at load, like every name - in an old file (clause 5), and the first save writes `_measure`. Where the file - declares the stored measures themselves (`Revenue`, `Cost`) is open question 2. +- **[Amendment] Virtual categories and virtual measures are defined by criteria, not + declared.** Each contains whatever in the document meets its criterion: `_measure` + holds the measures, and `_index` and `_dim` hold the record positions. Their ids and + displays are hardcoded in the application, so a file never declares one. Where a + line refers to one, it spells the virtual id: `_measure: column`, `_index: none`, + `region=east, _measure=revenue = 1200`. +- **[Amendment] An old file's virtual names are recognised, not migrated as + declarations.** The loader recognises `_Measure`, `_Index` and `_Dim` by their + hardcoded displays, and an old `## Category: _Measure` section is read for the + stored measures it lists, not as a declaration of `_measure`. The first save writes + `_measure` wherever the file refers to it. Where a new-version file declares the + stored measures themselves (`Revenue`, `Cost`) is open question 2. - **[Amendment] Records mode's synthetic positions are values, not ids.** The row coordinate under `_index` is a value of that virtual category. So, by the same reasoning, is the column coordinate under `_dim` (a column index, gotcha 1). They are @@ -803,29 +808,30 @@ which contradicts clause 1. ### 2. Where does a file declare the stored measures? -**The question.** A file never declares a virtual category (clause 1). But the stored -measures, such as `Revenue` and `Cost`, are items of `_measure`, and each needs a -declaration that carries its display and id. Today they are listed under -`## Category: _Measure`: +**The question.** `_measure` is never declared: it holds whatever meets its criterion, +and its id and display are hardcoded (clause 1). But the stored measures in it, such as +`Revenue` and `Cost`, are ordinary things with a user's display and a generated id, so +the file must declare each one. Today they are listed under the virtual category's own +header, which a new-version file no longer writes: ``` ## Category: _Measure - Revenue, Cost ``` -**Why it matters.** Without a declaration a measure has no display or id in the file, -and a new-version file refuses any undeclared id (clause 5). +**Why it matters.** Without a declaration a stored measure has no display or id in the +file, and a new-version file refuses any undeclared id (clause 5). The declaration is +also what puts a stored measure into `_measure`'s criterion. **Options.** -- **(a) Keep a section for the virtual category's items only**, spelled by its id: - `## Category: _measure` with `- Revenue(revenue), Cost(cost)`. The section declares - items, not the category. -- **(b) A dedicated section**, such as `## Measures`, with the same item lines. -- **(c) Declare a measure where it is first used**, in a data key. A data key holds - only an id, so the display would be lost. +- **(a) A dedicated section**, such as `## Measures`, with item lines + (`- Revenue(revenue), Cost(cost)`). Being declared there is what makes something a + stored measure. +- **(b) A marker on an ordinary item line**, wherever the measure is listed, for + example `- Revenue(revenue) [measure]`. -**Author's recommendation:** (b). It names what the section holds, and keeps the -virtual category out of the file entirely. +**Author's recommendation:** (a). It names what the section holds, and keeps the +virtual category's header out of the file. ### 3. Does interactive typing still resolve a display name that matches no id? From c7f8d0a3cde39b75d7a4a32cdba07bf45729aa16 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:40:36 -0700 Subject: [PATCH 072/106] docs(adr): delete ADR 007's remaining open questions (improvise-cq1q.1) The duplicate measure display follows from the profit- rule and is stated as its consequence; where stored measures are listed is the format's existing concern; interactive typing is UI behaviour, not identity. The Open questions section now says None. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 97 ++++------------------------ 1 file changed, 13 insertions(+), 84 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 7b2fda5c..8ca77b63 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -29,7 +29,7 @@ amendment is approved: | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | | 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | -Three questions are open (last section). +No questions are open. ## Context @@ -112,8 +112,8 @@ the reservation needs no check in the grammar. declarations.** The loader recognises `_Measure`, `_Index` and `_Dim` by their hardcoded displays, and an old `## Category: _Measure` section is read for the stored measures it lists, not as a declaration of `_measure`. The first save writes - `_measure` wherever the file refers to it. Where a new-version file declares the - stored measures themselves (`Revenue`, `Cost`) is open question 2. + `_measure` wherever the file refers to it. The stored measures themselves + (`Revenue`, `Cost`) are listed as the format lists them today, each with its id. - **[Amendment] Records mode's synthetic positions are values, not ids.** The row coordinate under `_index` is a value of that virtual category. So, by the same reasoning, is the column coordinate under `_dim` (a column index, gotcha 1). They are @@ -140,8 +140,10 @@ one, for a derived dimension. the new formula is a second measure with a fresh id, `profit-` from the shared counter (clause 4), like any colliding generated id. The stored measure keeps `profit` and its data. Removing the formula removes `profit-` and leaves `profit` - untouched. Both are displayed `Profit`, which clause 2's display-uniqueness rule - does not allow; open question 1 asks how that is resolved. + untouched. The formula keeps the display the user typed, so both are displayed + `Profit` and are told apart by id: the formula panel shows the ids where the display + is ambiguous (clause 6), and a typed `Profit` that matches both is refused with both + ids listed (clause 12). - **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, and its formula has a separate `FormulaId`. A formula whose generated id would collide takes a suffix like any other generated id (clause 4): a formula `Tax` in @@ -188,9 +190,9 @@ human-typed name resolves to exactly one id within its scope, and a rename to a display already in use there is refused. An item's scope is its category, so two categories may each have an item displayed `Revenue`. Formula text crosses categories, so a display may be ambiguous there. The ids differ, so the formula panel -shows the ids where a display is ambiguous. **[Amendment]** A formula and a stored -measure may end up sharing a display (clause 1); whether that is an exception to this -rule is open question 1. +shows the ids where a display is ambiguous. **[Amendment]** The one exception is a formula +whose measure shares a stored measure's display (clause 1): it is created anyway, and +the two are told apart by id. ### 3. Ids are interned symbols @@ -507,8 +509,6 @@ display that would otherwise read as display plus id: its display fails. A script gets a display only through a dedicated function such as `display_for(id)`. - **[Amendment] Interactive typing prefers ids and previews displays** (clause 12). - Whether an interactive input still resolves typed text that matches no id by - display is open question 3. - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. A create-then-act compound becomes **one command**: `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes @@ -679,8 +679,7 @@ answers for those. - A typed name equal to an id names that id's thing, even where it is also another thing's display. - Otherwise a display that exactly one thing in scope has names that thing, and a - display matching several things is refused with a list of their ids. Whether this - display stage stays is open question 3. + display matching several things is refused with a list of their ids. - A name that resolves to nothing is refused, and the refusal names it. - Ids are matched exactly, with no case folding. - **Typing prefers ids and previews displays.** A view-layer autocomplete turns typed @@ -716,8 +715,7 @@ without resolving any name (improvise-hexb). (`east`, `east-1`), type `east` into an editor, in text not captured at open: - it must bind A, and the test must fail under displays-first; - `east-1` must bind B; -- a display that is nobody's id must still resolve by display (pending open - question 3). +- a display that is nobody's id must still resolve by display. A fixture where no display equals an id proves nothing. @@ -783,73 +781,4 @@ three things: ## Open questions -These are open, and nothing above depends on an answer except where it says so. - -### 1. Two measures with the same display: is that allowed? - -**The question.** A formula never merges into a stored measure (clause 1): with a stored -`Profit` (id `profit`) in the document, adding the formula `Profit = Revenue - Cost` -creates a second measure, `profit-1`. Both are displayed `Profit`. But clause 2 keeps -displays unique within their scope, and both measures are in `_measure`'s scope. - -**Why it matters.** Under the uniqueness rule, adding that formula would be refused, -which contradicts clause 1. - -**Options.** -- **(a) Allow duplicate displays among measures only.** The formula panel already - shows ids wherever a display is ambiguous (clause 6), and typing prefers ids - (clause 12), so the two stay distinguishable. -- **(b) Drop display uniqueness everywhere.** Ids carry identity, and every place that - shows displays already falls back to ids when they clash. -- **(c) Keep uniqueness and give the new formula a different display**, such as - `Profit 1`, chosen by the host. - -**Author's recommendation:** (a). It is the smallest change that makes clause 1 hold. - -### 2. Where does a file declare the stored measures? - -**The question.** `_measure` is never declared: it holds whatever meets its criterion, -and its id and display are hardcoded (clause 1). But the stored measures in it, such as -`Revenue` and `Cost`, are ordinary things with a user's display and a generated id, so -the file must declare each one. Today they are listed under the virtual category's own -header, which a new-version file no longer writes: - -``` -## Category: _Measure -- Revenue, Cost -``` - -**Why it matters.** Without a declaration a stored measure has no display or id in the -file, and a new-version file refuses any undeclared id (clause 5). The declaration is -also what puts a stored measure into `_measure`'s criterion. - -**Options.** -- **(a) A dedicated section**, such as `## Measures`, with item lines - (`- Revenue(revenue), Cost(cost)`). Being declared there is what makes something a - stored measure. -- **(b) A marker on an ordinary item line**, wherever the measure is listed, for - example `- Revenue(revenue) [measure]`. - -**Author's recommendation:** (a). It names what the section holds, and keeps the -virtual category's header out of the file. - -### 3. Does interactive typing still resolve a display name that matches no id? - -**The question.** Only CSV import and the `.improv` loaders turn display names in input -text into ids (clause 6). Interactive typing (the TUI's minibuffers and `:` line, the -formula editor, the browser) prefers ids and shows the display as a preview -(clause 12). Clause 12's resolver also has a display stage: typed text that matches no -id but is exactly one thing's display names that thing. Does that stage stay? - -**Why it matters.** Without it, a person typing `Revenue` in the formula editor must -type or pick the id `revenue`. With it, a typed display still works, and interactive -input is a third place that reads display names. - -**Options.** -- **(a) Keep the display stage** for interactive typing only. Command-language text - from scripts never gets it. -- **(b) Ids only.** Interactive inputs match ids; displays appear only as previews and - autocomplete suggestions, which insert the id. - -**Recommendation:** none. "Prefer ids" leaves room for (a); "only import and the -loaders read display names" points to (b). +None. From 98db007e716bee6127cb7eeb90ea7186c58d5aec Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:41:00 -0700 Subject: [PATCH 073/106] docs(adr): interactive input matches a display or an id when the pick is unambiguous (improvise-cq1q.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 8ca77b63..7f86354c 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -508,7 +508,9 @@ display that would otherwise read as display plus id: (a script file, `improvise cmd`, another program). A script that names something by its display fails. A script gets a display only through a dedicated function such as `display_for(id)`. -- **[Amendment] Interactive typing prefers ids and previews displays** (clause 12). +- **[Amendment] Interactive input matches either a display or an id**, in the TUI + and the browser, as long as it is unambiguous which thing the user is picking. Ids + win when it is not, and the display is shown as a preview (clause 12). - **A frame cannot address a thing it creates**, because it does not know the id the host will assign. A create-then-act compound becomes **one command**: `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes @@ -670,7 +672,8 @@ answers for those. - **A typed name that was not in the captured mapping** is resolved against the document at commit, and refused, naming it, if it does not resolve. -**[Amendment] The client does the resolving, and ids always win:** +**[Amendment] The client does the resolving. Interactive input may match a display or +an id as long as the pick is unambiguous, and ids win when it is not:** - **The client holds the mapping**, as it holds `OpenedCell`. The TUI and the browser share one resolver in `improvise-protocol`. - **A name new to the edit** is resolved by the client against its id-bearing From e7d9ad0d2a4ad09d6c145c0ab4ccb59330989fd6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 09:42:23 -0700 Subject: [PATCH 074/106] docs(adr): ADR 009 states only what the decision changes (improvise-felo.4) Following the user's instruction, restated conventions and current behaviour are removed except where the decision conflicts with them, and the open questions are cut to the five design choices still open. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 920 ++++++---------------- 1 file changed, 252 insertions(+), 668 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e2c239ac..02310a7b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,742 +1,326 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait -**Status:** proposed. Only the user accepts an ADR. · +**Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed -models) · -**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") · -**Governed by:** `context/design-principles.md` §3, §5 and §7 +**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008 -**Summary.** Formulas are evaluated incrementally by a small in-house implementation of -the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A -cell write becomes a delta. Aggregates are maintained rather than re-folded, and only -the formula cells whose inputs moved are re-evaluated, by the one existing evaluator. -Operator state is keyed by the **shape key** of a projection request, never by a client. -The same circuit serves the in-process TUI and remote clients. A remote client learns of -changes from one unaddressed notice per accepted batch, and it refetches what it shows. -Sums are exact, through an exact accumulator. +## Problem -File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive -them before relying on one. - -## Problem and context - -**Data that streams.** A live-backed model (ADR 003) is a source whose rows change -underneath the document. Today every change takes the path an edit takes, which -recomputes every formula cell. That is tolerable for one keystroke at a time and not for -a stream of changes. A change should cost in proportion to what it touches. - -**Today's evaluation is a from-scratch fold after every batch.** -- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls - `Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates - every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16` - (`types.rs:710`). -- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`, - `types.rs:848`, through `DataStore::matching_values`). -- `formula_cache` holds one aggregation shape at a time. A session on a view with another - None axis gets a **cloned model** (`rederived_for`, `project.rs:205`). -- A formula total that holds an aggregate walks a cross product - (`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`). -- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing - batch, and `document_content` (`app.rs:168`) after each `CellData` effect. -- Every mutating frame from a client clones the document for rollback - (`Host::mutating_frame`, `host.rs:224-225`). - -**What an incremental evaluator must not change.** Each of these is a place where it -could quietly disagree with today's code: -- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one - distinct value is that value, and anything else is `Error("mixed")`. -- **Circularity (gotcha 9).** A cell still moving after 16 passes is - `Error("circular")`, and so is an acyclic chain that needs all sixteen. -- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0 - guard. -- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration, - because f64 addition is not associative. -- **The tiers are observed (gotcha 40).** A write of the value already there bumps - nothing, and a rewritten record counts once. - -**The view/host split (ADR 008).** A client reads through `project(request)`, where a -request names axes, an absolute top-left and dimensions. It writes only through -`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys. -Aggregation is always done on the model side. A view's hiding, collapsing, page -selection, axis assignment and pruning are one client's ephemeral state until an -explicit save-view command. Only the number format is document-wide. +Every change recomputes every formula cell (`Model::recompute_formulas`, called by +`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That +does not scale to a live source (ADR 003) streaming changes into a large table, nor to +ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without +building the grid. A change should cost in proportion to what it touches. ## Decision -### 1. The crate: `zset` +Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A +cell write becomes a delta, aggregates are maintained, and only the formula cells whose +inputs moved are re-evaluated by the existing evaluator. The incremental results must +equal today's eager results exactly. -A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on -`std`, so it is wasm32-clean by construction. It contains: -- **`ZSet`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an - abelian group. Weights use checked arithmetic, and a zero weight is removed. The group - laws are property-tested. -- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per - source and returns one per sink. There is one clock, one thread and no scheduler. -- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id - property-tested); the linear `map`, `filter` and `flat_map`; `index_by`; - `join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear - aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a - multiset per group. -- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a - resumable `step(deadline)`, so a rebuild can run as a sliced job. +### 1. The `zset` crate -Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one -clock (improvise applies batches in one total order); threads within one circuit (a host -with threads may run independent circuits on a pool); a query language. Spilling to disk -is not in the first version, but operator state sits behind one interface (a trace: look -up, read a range, apply a delta), so the store-memory work can page or compress it later. +A problem-agnostic workspace crate with no dependencies beyond `std` (so it is +wasm32-clean and serde-free). It provides: +- `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; +- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`, + `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, + `distinct` and `aggregate`; +- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it + can run as a sliced job; +- operator state behind one trace interface (look up, read a range, apply a delta), so + later store-memory work can page or compress it. -### 2. The plug: a trait, and a seam crate +There are no nested streams, no recursion operator, no timestamps beyond one clock, and +no query language. -improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the -formula result at a key, and the raw fold of the cells that match a partial key. It -takes the cell delta and the formula delta after each model effect. +### 2. The plug -**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and -read through the trait. The circuit decides *which* formula cells to re-evaluate and -supplies their inputs. Formulas are not compiled into operators, because a compiled -`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the -defect. +improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the +formula result at a key, and the raw fold of the cells that match a partial key. The +incremental backend implements it in a seam crate, `improvise-incremental`, which +depends on `zset` and improvise-core. improvise-core never depends on `zset`. -**The incremental backend lives in a seam crate, `improvise-incremental`,** which -depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same -shape as `gp-tree` and `improvise-gp`. +The circuit decides *which* formula cells to re-evaluate. The one existing evaluator +computes each value, so formulas are never compiled into operators. -**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change, -as `Eager`. It stays as the oracle for the differential test while the circuit matures, -and for the two cases that still need it (clauses 5 and 10). The differential test is a -property test in the seam crate that applies one random edit sequence to both -backends and compares every drawn cell, total and occupancy count. The goal is that every -formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is -open question 15. +**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. +It serves as the oracle of a differential property test, which applies one random edit +sequence to both backends and compares every drawn cell, total and occupancy count. It +is removed once its remaining uses are replaced (open question 3). -### 3. The input: the cell delta +### 3. The cell delta -`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a -`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number -assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an -overwrite: -- a new cell is `+(k, v, s)`; -- an overwrite is `−(k, v₀, s) + (k, v₁, s)`; -- a removal is `−(k, v₀, s)`; -- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. +`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`: +- `+(k, v, s)` for a new cell; +- `−(k, v₀, s) + (k, v₁, s)` for an overwrite; +- `−(k, v₀, s)` for a removal; +- a remove-then-set gets a fresh `s′`. -**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their -strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an -equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a -change, and a NaN rewritten with the same bits is not. Multisets order values by -`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted -`Vec`, so deriving it is sound. +`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an +overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite +consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect. -**What the delta replaces.** The record tier reads the records named by the -consolidated `(key, value)` projection of the delta, instead of diffing two store scans. -The dirty flag's content half becomes "a non-empty projected delta, or a moved -formula-set fingerprint". The fingerprint is needed because formula edits move no cell, -and the worker persists on this flag. +The record tier and the dirty flag read this delta instead of scanning the store. The +dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the +formula set moves, because formula edits move no cell. -**The circuit steps once per model effect.** That keeps occupancy current inside the -effect loop. What a client counts is a separate publication number (clause 12). +Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The +formula-set fingerprint must use the id form too, or a rename reads as a formula change. -**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`, -the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a -rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set -fingerprint, taken today over `(raw, target, target_category)`, must then be taken over -the id form, or a rename reads as a formula change. +### 4. Aggregates -### 4. Aggregates: linear where they can be +A group is the cell key minus the None categories. One aggregation circuit serves every +shape key with the same None list, which removes the per-session model clone. -**The group** is the cell key minus the None categories, which is the partial key -`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the -same None list, which removes `rederived_for`'s model clone. - -| answer | state per group | cost per delta element | +| aggregate | state per group | per delta element | |---|---|---| -| `Count` | a count of values and a count of numbers | O(1) | -| `Sum` | an exact accumulator (clause 8) | O(1) | -| `Avg` | `Sum` / `Count` | O(1) | -| `Min`, `Max` | a multiset of the numbers | O(log d) | -| unanimity and `mixed` | a multiset of all values | O(log d) | +| `Count` | count of values, count of numbers | O(1) | +| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) | +| `Min`, `Max`, text unanimity | a multiset of values | O(log d) | -Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules -for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential -test generates signed zeros and non-finite values to pin this. Only measures whose -aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups, -they hold at most one entry per stored cell. +A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and +semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the +ordering category's order. How that order is keyed is for those tickets. -**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures, -improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an -extreme, keyed by the ordering category's order instead of by value. A semi-additive -measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the -order is keyed is open question 2. +### 5. Formulas -### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it +**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None +categories". A formula counts at a stem where its top-level `WHERE` matches or raw data +exists at the target. -**Stems** are `distinct` over the cell delta, mapped to "key minus the target category -and the None categories". A formula counts at a stem only where its top-level `WHERE` -(if any) matches the stem's coordinate, or where raw data exists at stem + target. That -is exactly where `recompute_formulas` caches a value today, so pruning keeps its current -behaviour. +**Inputs.** A formula cell is re-evaluated when one of its inputs moved: +- a `Ref(R)`: the group at the stem with `R`'s category set to `R`; +- an `Agg`: the group at the stem; +- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set + to `Item`. This is the one join: stems indexed by "stem minus `Cat`". -**A formula cell is re-evaluated when one of its inputs moved:** -- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`; -- for an `Agg` with no filter, the group at the stem itself; -- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat` - set to `Item`, which is another stem. This is the one join: stems indexed by "stem - minus `Cat`", joined with the aggregate's output delta. +The affected pairs are evaluated in the formula graph's topological order. -The affected `(stem, formula)` pairs are ordered topologically over the formula graph and -handed to the existing evaluator. +**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: +1. A per-record stage retracts and reinserts each touched record. It reads the record's + previous cells as the store minus the delta, so it keeps no copy of the values. +2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum + `Seq` of the record's live cells), which is eager's concatenation order. +3. The group aggregate, with an error channel that reads the first `Err` in order. -**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline: -1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it - retracts the record's previous snapshot and inserts the new one. The previous snapshot - is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. - So the stage reads the store and keeps no second copy of the values. -2. **A lifted map** to `(context key, order key, expr at record)`. The context key - removes the target and None categories, and at a fold or total the spanned categories, - mapping a collapsed member to its group. The order key is (context rank, the minimum - store sequence over the record's live cells), which is exactly the order eager - concatenates in. -3. **An inner `WHERE`** is a filter plus the join above. -4. **The group aggregate**, with an error channel kept beside it that reads the first - `Err` in order-key order, as eager does. - -No cross product is ever built. The map is sound only when `expr` reads the record's own -cells. Other formulas take a fallback that evaluates over the **occupied contexts** only, -the contexts some record projects to. Because `agg_values` evaluates only at each -context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one -edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over -zero occupied contexts it yields the empty value instead. Formulas targeting a category -other than `_Measure` stay on the eager path until their order key is defined. Open -question 8 covers both. +No cross product is built. A formula whose expression reads other records is evaluated +over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is +unchanged, because a context with no records contributes nothing. One exception: an +inner bare ref that names no item errors at every context, so for it the ref is checked +before enumerating. This replaces `Expansion::members`. ### 6. Circular references -DBSP's recursion computes a least fixed point, and improvise's circularity is not one: -`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs -forever. The required answer is `Error("circular")`. The first version therefore decides -circularity outside the crate: -- the formula graph is rebuilt when the formula set changes, and its strongly connected - components are found; -- formulas outside every cycle are evaluated once per step, in topological order; -- a formula in a cycle has its affected cells evaluated by today's bounded fixed point, - restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still - convergent where `x` is false; -- the sixteen-pass bound is reproduced, so the two backends agree in the differential - test. +DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the +answer must be `Error("circular")`. The first version finds the formula graph's strongly +connected components. Acyclic formulas are evaluated once each, in topological order. A +component is evaluated with today's bounded fixed point, restricted to that component. +That reproduces the sixteen-pass bound for the differential test. It is to be replaced +by a DBSP-native treatment (open question 2). -This serves testing and validation. It is to be replaced by a DBSP-native treatment of -cycles, whose shape is open question 14. +### 7. Operator state follows the request's shape key -### 7. The circuit follows the document per model effect - -**Operator state is selected by a shape key.** The key is exactly what changes which -cells feed which operator: -- the categories of each axis, by level; +A projection request's **shape key** selects operator state, shared by every request +with the same key, from any client. The key is: +- the categories on each axis, by level; - the None categories; -- the filter keys: each filter category with its selected item, and the hidden items of - every category on None, Page or Filter. Those items drop cells from the fold; -- the page **categories**, which key a slice per page (open question 1); -- `measure_agg`, which is model state, never taken from a request. +- the filter keys: each filter category's item, and the hidden items of categories on + None, Page or Filter; +- the page categories, so each page is a maintained slice and paging rebuilds nothing; +- `measure_agg`. -**Everything else is a read-time parameter**, applied when output is read: the hidden -items of an axis category, collapsed groups and group membership, the drawn order of -items, the page selection, and prune. A request carries its own key and parameters from -its client's UI state, and a saved view only supplies defaults. +Everything else in a request is applied when output is read: hidden and collapsed items +on axis categories, item order, the page selection and prune. Occupancy and column +aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item +changes no operator. The visible axis is a remap. -**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn -items, the collapsed groups, the page selection and `prune_empty`, all of which are -read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit -neither reads nor depends on that tier, which is a view fact. Only `document_generation` -and the record tier matter to this ADR. +- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas + arriving during the build are buffered and added at install. +- **A formula edit** is a formula delta fed to the circuit. +- **A document swap** rebuilds everything. +- **Saving a view definition** touches no operator. +- **Live keys are a cache.** A key is evicted after a period in which no request names + it (the period is set by measurement), and every eviction is announced (clause 9). -**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates -are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing, -reordering and adding an axis item change no operator: the visible axis is a remap. -ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a -rebuild. A collapsed group's value is combined from its members when read. That is exact -for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it -costs O(members) per read (not measured). The exceptions are column totals (below). - -**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job -runner (8 ms per normal operation). Deltas that arrive during the build are buffered and -added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is -O(delta) and is never sliced. Each step's occupancy output carries the count of tuples -whose count crossed zero, so ADR 008's ordered index can choose between inserting and -rebuilding. Measurements against the budget are stated as ratios against an unrelated -operation on the same run, not as bare numbers. - -**Formula edits are an input.** After each model effect the host compares a fingerprint -of the formula set. If it moved, the formula delta is fed to the circuit, and the formula -graph is re-derived. A document swap (`install_workbook`, which bumps -`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and -no formula, so it touches no operator. - -**The shape cache.** Live shape keys are a cache. A key enters when a request names it, -and it is evicted when no recent request names it (open question 4). Every eviction is -announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm -by every client reading it. Idle readers make that likely, since a shape that clients -only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job -queue. - -**Column totals subtract hidden rows at read time.** Today a column total pools only the -drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`), -pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a -per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept -over all rows, and a read subtracts the request's H hidden cells in that column as a -temporary overlay: +**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row +leaves it. The shape key is shared and a hide is one client's, so totals are kept over +all rows. A read subtracts the request's H hidden cells as a temporary overlay: - `Count` subtracts; -- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden; -- the text rule counts the distinct keys left; -- `Sum` subtracts through the exact accumulator. +- `Min` and `Max` skip keys whose whole count is hidden; +- text counts the distinct keys left; +- `Sum` subtracts exactly through the accumulator. -That is O(H log d) per read for every aggregate, with no per-client state. The shape key -is shared across clients and a hide is one client's, so a total cannot be rebuilt per -hide. A hidden column item under a collapsed column group changes that group column's -cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression -totals need their entries indexed by row tuple as well, so a read costs O(hidden -records). The alternative is open question 12. +That is O(H log d) per read for every aggregate, with no per-client state. +Aggregate-over-expression totals index their entries by row tuple for the same +subtraction. -### 8. Floating point: exactness comes from the representation +### 8. Exact sums -A sum maintained by `+v` and `−v` in f64 drifts from the eager fold. -`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max, -with values of up to five decimal places: +An f64 sum maintained by `+v` and `−v` drifts. Measured by +`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million +cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer +differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both +backends: +- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which + makes it an abelian group that rounds once, on read; +- measured bit-equal to a fresh sum after a million cycles, and exact against rational + sums on 20,000 cases; +- 568 bytes per group, and about 1.3× the cost of an f64 add per update. -| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact | -|---|---|---|---| -| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 | -| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 | -| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 | +It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change +in their last bits, because they become correctly rounded, and a sum of `-0.0` reads +`+0.0`. -Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively, -against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager. -At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer -differently on the two paths. Eager itself is already more than 1e-10 from the exact -decimal sum. +### 9. Delivery -**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer -wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so -that those can be retracted. Adding and retracting are integer additions, so the -accumulator is a true abelian group, and it rounds once, on read. A weight is split into -base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example: -- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles; -- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again - with weights up to 2⁴⁰; -- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to - 1.1 µs per read; -- it is 568 bytes per group, against 8 for an f64. +The same circuit serves the in-process TUI and remote clients. Its internal streams never +cross a wire, because aggregation stays on the host. -`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0, -where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they -become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and -the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum` -0.1.6 rounds about a quarter of single negative values wrongly and cannot retract -non-finite values. +**A request** is a shape key plus a window, given as an absolute cross-product top-left +and dimensions. The window is never state, and a request costs O(visible). +- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings. +- **Output** is keyed by tuple identity, with the position attached when sent. +- **An item insert, removal or reorder** is sent as an axis change, and each client + re-derives its positions. -**Lossy operations happen at the edge.** Sums stay in the accumulator. Division, -non-integer `POW` and display rounding happen only where a value leaves the exact domain. -An exact value becomes an f64 through its decimal string, which is correctly rounded. +**Pruning support.** Pruning is window-relative: a row or column empty within the +displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model +answers one restricted successor query: "the next k rows after position p with a value +in a column of S", and the same for columns. A per-column sorted index of occupied rows +answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its +window, fills forward with that query, and finds a new top-left when scrolling up or left +by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether +pruning runs on the client and which parameters travel are ADR 008's questions. This +design supports either answer. -**The accumulator does not make `=` against a typed decimal exact.** It sums the stored, -already-rounded f64s, so it can miss the typed total by up to about half an ulp of each -term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example: -`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's -exact decimals fix that, and they also cover per-cell arithmetic and typed values such as -money. +**The change notice.** After each accepted batch the host emits one notice to every +client, addressed to none. It carries: +- the stamp `(host instance, document_generation, publication)`. The publication number + increments once per published batch, and the host instance changes on restart; +- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved; +- the shape keys evicted since the last notice; +- for each live shape key, the moved output keys (cells, formula cells and totals), as + the circuit computes them. Moved records alone would miss formulas with cross-stem + inputs. -### 9. What ADR 008 gets +A notice whose key lists exceed a fixed size carries "refetch" instead. -- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple - (kept as a pre-filter and as the unpruned extent; see open question 11). - Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. - A fold row with a formula column but no stored cell under it, and the `Exact` case with - nothing on None, are **not** occupied, on purpose. -- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit - consumes it and never keeps a second order. -- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item - grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because - the host cannot compute a client's visible axis. -- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the - formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows). -- **Widths for text columns** (clause 13). +**What a client does.** It intersects the moved keys with its window. A fold row has no +maintained key, so it maps member keys through its own remap first. It then refetches +what moved. It refetches everything when it sees a publication gap, a new +`document_generation` or host instance, its shape in the eviction list, or a notice it +cannot read. A slow client may merge pending notices: the union of keys, the latest +stamp, and axis changes kept in order. Outputs are consolidated under the wire's value +identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`. -### 10. Live sources +**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, +`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client +state. `GridExtent` is derived by the client, and in records mode the record count is +fetched. -A live source that can report its changes produces a cell delta, and the circuit cannot -tell it from `DataStore`'s. A source that can only answer aggregate queries stays on -`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated -state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so -the backend is chosen per document. +**A connection operator** at the wire boundary adapts the circuit to a connection. It is +a sink for results and notices and a source of requests toward the circuit, and the +mirror image toward the client. It owns serialization, the stamp rules, notice +coalescing, keep-alive in both directions, and session resumption. In process it is +function calls. Its shape is open question 4. -### 11. Constraints +### 10. Text column widths -- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then - `zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm` - CI job, each checked with `cargo tree --edges normal`. -- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over - small inputs, and the differential test lives in the seam crate's own binary. -- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and - sorted, and nothing in the circuit decides a display order. The determinism test builds - **two** circuits independently from one history. -- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that - `Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the - integrated values but not the history `delay` and `differentiate` hold. The cost is a - second O(state) clone per mutating frame, and peak memory to match. The keyboard path - takes no snapshot. -- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence. +The host maintains each text column's maximum display width as a histogram from width to +count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows +come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule. -### 12. Delivery: one circuit, in process or across a wire +### 11. Rollback and memory -The DBSP machinery is **network-transparent**: the same operators and streams whether a -consumer is in process or across a wire. +- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An + inverse delta is not a rewind, because `delay` and `differentiate` keep history. +- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, + `ExactSum` per summed group, occupancy per live shape, the successor-query index and + the width histograms. The store itself is the integrated base and is never copied. It + is measured at ADR 008's fixture before the store-memory work begins. -**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their -change notices, plus the input deltas of a remote live source. The write side carries -model commands with their basis, never deltas. The circuit's internal streams stay on -the host, because aggregation is model-side. No UI state crosses. +## Alternatives rejected -**A request has a shape key and a window.** The key selects operator state, shared by -every request from any client. The window (absolute top-left and dimensions) selects -part of the output and is never state. So a request costs O(visible). +- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it + fails to build for wasm32. +- **Compiling formulas into operators:** a second evaluator. +- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a + second description of evaluation. +- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic + only by holding per-client state on the host (open question 1). -**Positions are absolute cross-product positions**, in each category's own item order, -and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when -occupancy changes. They do move when an item is inserted, removed or reordered, so the -recommended design (open question 10) is: -- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute - position attached when sent; -- an item edit is sent to every client as an **axis change** ("inserted at index j of - level L"). A client re-derives its positions at O(levels) each and re-anchors its - top-left to the same tuple; -- resolving positions needs rank and select over each category's items, which is ADR - 008's order-statistic structure. +## Consequences -**Pruning's model-side support.** Pruning hides a row or column that is empty within the -displayed values, so a value outside the window does not count. That differs from -today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed -window, one pass that drops the empty rows and columns is already the fixpoint. A fill -then adds rows and columns forward from the absolute top-left, until the requested -dimensions are met or the axes end. Growth is only rightward and downward. For the fill, -the model answers one **restricted successor query**: "the next k rows after absolute -position p with a value in at least one column of S", with S in the call, and the same -for columns. A per-column index of occupied rows answers it: a sorted list per column, -seeked to p and merged forward with a linear minimum over the |S| cursors, at -O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the -cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a -B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's -whole state is its request and its last absolute position, so a resume is a seek. -**Scrolling up or left is a new request with a new top-left.** How the client finds that -top-left when rows are pruned is the recommendation of open question 9: it gallops -backwards in doubling jumps until a probe hits, then narrows until one new row is gained. -Each probe is a restricted successor query bounded by the old top-left. That is about 14 -to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an -empty row. The alternatives there are a model-side predecessor query, and failing the -fetch and retrying. Which side of the wire runs the prune and the fill is -open question 9. +**Good:** +- A write's evaluation costs O(delta) plus the formula cells it reaches. +- The store scans and the per-view model clones go. +- ADR 008 gets occupancy, totals and widths from one evaluator. +- Live sources that report changes plug in. +- The host holds no per-client state. -**The wire properties**, which hold on each hop: -- **T1. Plain data.** Output elements are serializable values that name cells by tuples, - positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`. -- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`), - so a value change that only a NaN payload distinguishes cannot be told apart remotely. - Which identity outputs use is open question 5. -- **T3. A publication stamp.** Each accepted batch is published once, stamped - `(host instance, document_generation, publication)`. The publication number increments - once per published batch, so a multi-effect batch is one header, and a refused frame - consumes none. The host instance changes on restart. A client drops anything at or - below the number it has applied. **The refetch rule:** a client that sees a gap, a new - `document_generation` or host instance, its shape in an eviction list, or a notice it - cannot interpret, refetches. -- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative - to its stamp. -- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and - flags, under the latest stamp. Axis changes are positional, so they stay an ordered - list. A `document_generation` change inside the run voids the merge. A basis is never - sent ahead of the cells it describes (gotcha 43). -- **T6. One API.** The TUI calls `project` and receives the same types as a remote - client. A test runs one edit sequence through an in-process consumer and through a - serialize-and-deserialize round trip, and requires equal results. +**Bad:** +- Two backends exist until `Eager` goes. +- Integrated state is O(cells) beside the store. +- `ExactSum` costs 568 bytes per summed group. +- Every client receives every notice. +- A mutating frame still clones the state for rollback. -**The change notice.** After each accepted batch the host emits one notice, addressed to -no client. It carries: -- the stamp; -- the axis changes; -- display relabels (`(id, new display)` under ADR 007); -- a flag when the formula set or `measure_agg` moved; -- the eviction list; -- for each live shape key, the **moved output keys**: cells, formula cells and - column-tuple totals, as the circuit computes them. +**Order** (each step is a ticket in the implementation epic, behind the differential +test): +1. `zset`; +2. the `Derived` trait with `Eager`; +3. the cell delta, the record tier and the dirty flag; +4. order-free aggregates and occupancy, measured at ADR 008's fixture; +5. formulas and the aggregate-over-expression pipeline; +6. `Sum`, `Avg` and totals through `ExactSum`; +7. live sources; +8. delivery. -Moved records alone would not do. From a moved record, a client cannot tell whether a -formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue -WHERE Region=East)` on page `Region=West` moves when an East record is written. A client -intersects the moved keys with its window by identity. A collapsed fold row has no -maintained key, so a moved member key goes through the client's own member-to-group -remap first. The client then refetches what moved, by `project` or by the successor -query. - -**Every client receives every notice**, and the host holds no per-client state for -delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists -exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a -refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, -`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and -`GridExtent` is derived by the client from category lists. In records mode the row -extent is the admitted record count, which the client fetches. Whether notices should be -filtered per client is open question 3. - -**A connection operator, at the wire boundary.** A connection operator sits at the wire -boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for -results and notices and a source of requests and model commands. Toward the client it is -the mirror image. It owns: -- serialization (T1, T2); -- stamps and the refetch rule (T3); -- coalescing (T5); -- keep-alive in both directions. A client that hears neither a header nor a ping within - the timeout treats the connection as dropped, so silence is never read as "no - batches"; -- session resumption. - -In process it is function calls, as in the TUI and in the browser's `Worker`, which -already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in -`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one -component per connection, or a host half shared per host process with a client half in -every front end), and who owns resumption state, are open question 17. - -### 13. Column widths for text - -The host maintains each text column's maximum display width incrementally. A maximum is -not invertible, so it is an integrated aggregate. Display widths are small integers, so -the state is a histogram from width to count, per (shape key, column): O(distinct widths), -at O(log d) per update. Its input is the delta of the column's text cells over every row, -which the circuit already produces for unfolded cells. Fold rows need the fold maintained -at the visible grain. A text width does not depend on the number format, so there is no -rebuild when the format changes. A client's hidden rows come out through clause 7's overlay. -Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized, -held by the view layer. A number's width would need formatting on the write path, and a -rebuild per format change. Which columns count as text is open question 16. - -### 14. Resident state - -Beside the store, the circuit keeps: -- each cell's store sequence, for clause 5's per-record stage (one integer per cell, - kept in `DataStore`); -- the multisets for `Min`, `Max` and text, O(cells) of those measures; -- `ExactSum` at 568 bytes per summed group; -- occupancy per live shape, O(occupied tuples); -- the restricted-query index, O(occupied cells); -- the width histograms; -- the rollback snapshot per mutating frame. - -The store itself is the integrated base that a rebuild reads, so it is never copied. -State is evicted with its shape, and it sits behind one trace interface, so the -store-memory work (offload to disk, compression) can page or compress it later. None of -these sizes is measured at ADR 008's fixture. Migration step 4 measures them. - -### Relations to other ADRs - -- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its - `document_generation` and record tiers, not on `layout_version`. -- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace; - epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values. - Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a - 256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators. -- **ADR 006:** replay reproduces a circuit's state. -- **ADR 007:** keys become ids, so a rename is a relabel (clause 3). -- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the - occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire - prunes, and which parameters a request carries, are shared with questions 9 and 13 - below. - -## Alternatives considered - -- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio, - mimalloc, zstd and others), about 421 crates for wasm32, and fails - `cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded, - storage-backed engine where improvise needs a single-threaded operator library. -- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only - the cost of not changing: every write still recomputes every formula. -- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule, - which is a second description of evaluation. Z-sets give one rule, linearity, whose laws - are tested once. -- **Compile formulas into operators.** A second evaluator (clause 2). - -## Consequences and migration - -**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches. -The store scans behind the record tier and the dirty flag go, and so do the per-view model -clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources -that can report changes plug in unchanged. The host keeps no per-client state, and one -delivery path serves every front end. - -**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that -must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs -568 bytes per group, and displayed sums change in their last bits. Circularity is decided -in two layers. The network path still clones per mutating frame. Every client receives -every notice. Every column-total read pays O(H log d) for the request's hidden rows. - -**Order.** Each step is a ticket and a PR in the implementation epic, and none switches -the displayed path until the differential test holds: -1. the `zset` crate, with its laws, operator properties and determinism test; -2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`; -3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a - signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty; -4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and - occupancy) under the differential test, with step and rebuild costs measured at ADR - 008's fixture. ADR 008's pruning depends on this step; -5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's - aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price` - and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members` - depends on this step; -6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row - overlay; -7. live sources as delta streams, once ADR 003's source trait exists; -8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one - header and no client refetches; a dropped header is detected; a reconnect after a host - restart refetches and drops nothing as a duplicate; a refused frame consumes no - publication number; and a dropped connection forces a refetch. +Step 8's required tests: +- `[AddItem, SetCell]` publishes one notice, and no client refetches; +- a dropped notice is detected; +- a reconnect after a host restart refetches; +- a refused frame consumes no publication number. ## Open questions -1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page - categories, every page's state is maintained and `[`/`]` only reads another slice. The - held key stays O(visible), and the state stays bounded by the stored cells. If it does - not, a page change builds operators like any other new key. *Recommendation:* the - slice, which clause 7 assumes. +1. **Should notices be filtered per client?** Clause 9 sends every client every live + shape's moved keys. That keeps the host free of per-client state, but traffic is + O(clients × all moved keys) per batch. Filtering by topic would send each connection + only its own shape's key lists. It needs one subscription per connection, which pins + its shape in the cache. Is such a subscription "model-relevant information" the + backend may hold? It could also live in the connection operator (question 4). + *Recommendation:* broadcast with the size cap. Add topics only if measured traffic + with many clients on different shapes justifies them. -2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder, - since a reorder rebuilds nothing. By item identity with the order looked up at read - time, which makes reads O(d). Or by an order label: a key that compares in the current - order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder - re-keys every entry stored under the moved items. *Recommendation:* the order label, - with only the drawn members ordered. Not measured. +2. **How should cycles be handled natively?** Clause 6's component fixed point is + interim. The options: + - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it + has not converged, which keeps today's answers; + - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles + such as `A = IF(x, B, 1)`; + - (c) a monotone reformulation, where one exists. -3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed - stream. The rejected alternative is to send no notice at all and let each client - refetch on its own schedule: it keeps the host silent, but another client's change - stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state, - but every client receives every live shape's moved keys. The options: - - **topics:** a client receives only its own shape key's key lists. Headers still go to - everyone, so the refetch rule keeps working. That costs one subscription per - connection, registered atomically with the fetch that names it, and pinning its - shape; - - **window push:** per-client subscriptions holding the window, sending only the - window's deltas. That costs the most host state. + *Recommendation:* (a). - Is a per-connection subscription "model-relevant information" the backend may hold, - or UI state it must not? It need not live in the host at all if the connection - operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt - topics only if measured traffic, with many clients on different shapes, justifies - them. +3. **What replaces `Eager`?** It still serves as the differential oracle and as the + backend for live sources that can only answer aggregate queries. *Recommendation:* a + small from-scratch reference implementation, kept in the test crate, as the oracle, + and a pushdown backend behind the same trait for query-only sources. -4. **How long does a shape key live?** A key is evicted once no recent request names it. - Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades - memory against refetch storms. If the connection operator re-requests the shapes its - clients read, that is a keep-alive by another name: one host `project` per shape per - eviction period. *Recommendation:* evict after a period with no request, set by - measurement, with hysteresis for shapes with many readers. +4. **What shape is the connection operator?** Either one component per connection, or + one per host process: a host half shared by all its connections, with a client half + in every front end. The second lets the host send one notice per operator rather than + per client, but holds per-connection state in the shared operator. Who owns + resumption state, and is it backend state? *Recommendation:* one per host process, + with resumption state in the host half keyed by a client-held token, treated as + delivery state rather than UI state. In the TUI and the browser worker, with one + client each, it is a function call. -5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs - with different payloads differ, but the wire spells every NaN `"nan"`. A remote client - would therefore see such a change as nothing. *Recommendation:* outputs consolidate - under the wire's identity (every NaN one value), and the record tier keeps bit - identity. - -6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and - column aggregates by items, reports zero crossings per step, and slices rebuilds. That - makes collapse, hide, reorder and add item free, at the cost of recombining a - collapsed group's members on read. *Recommendation:* adopt all three, as written. - -7. **Resident state.** Should the first version keep no second copy of the store (the - store is the integral), put operator state behind one trace interface, and measure - state at ADR 008's fixture before the store-memory work begins? *Recommendation:* - yes. - -8. **Formulas the per-record map cannot express.** They take the occupied-contexts - fallback (clause 5). - - (a) Should `Eager` use the same enumeration, so the two backends agree? That changes - eager's answer only at the bare-ref edge, which could also be kept by checking the - ref once before enumerating. - - (b) Should formulas targeting a category other than `_Measure` take the same - fallback, so that nothing still needs `Expansion::members`? - - *Recommendation:* yes to both. - -9. **Which side of the wire prunes and fills?** On the client, the model offers only - windows and the restricted successor query, and the fill is the client's, with no host - state. On the model, prune is a request parameter, and the model keeps a pruned axis, - with rank and select for "row i of n", and runs the fill as a sliced job resumable - from (request, fill front). At a density of 10⁻⁴, a client without the successor query - would ship mostly empty windows. A second part: when scrolling up or left, how does the - client find a new top-left when rows are pruned? The options are galloping backwards - with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a - model-side predecessor query (one round trip, no new state, but a second query - direction), or failing the fetch and retrying with the nearest occupied row named. ADR - 008 asks the same question. *Recommendation:* the client prunes and fills, and it - gallops. - -10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas - by position instead goes stale on every item insert. Identity costs rank and select - per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as - clause 12 assumes. - -11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers - the restricted successor query. Per-axis counts can only skip rows that are empty - everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned - extent. *Recommendation:* keep them in that role. Not measured. - -12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at - O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape - key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per - distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at - read time. Memoise per hidden set only if a measured hidden set is too large for the - frame budget. - -13. **Which read-time parameters travel in a request?** The page selection travels, as a - third axis. Hidden items and collapsed groups change model-side totals and folds, so - without them a client could not get those from the model. Alternatively, a collapse - could be expressed as an axis at the group's level and a hide as a filter exclusion, - but both change the shape key, which costs a build per state. The drawn order, the - number format and prune change no aggregate. *Recommendation:* hidden items and - collapsed groups travel; the rest stay with the client. - -14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and - validation. The options: - - (a) a recursion operator bounded at sixteen iterations, which writes `circular` - where it has not converged and keeps today's answers; - - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles - such as `A = IF(x, B, 1)`; - - (c) a monotone reformulation, where one exists. - - *Recommendation:* (a) first. - -15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle, - query-only live sources, and the formulas of question 8 that stay eager. - *Recommendation:* a small from-scratch reference implementation, kept in the test - crate, as the oracle; a pushdown backend behind the same trait for query-only - sources; and the fallback of question 8 as a circuit stage. - -16. **Which columns get a host-maintained width?** Columns whose cells are text, with - numbers keeping the sticky width with hysteresis? And what of a column that holds - both? *Recommendation:* text cells get the maintained width, and the view layer takes - the larger of that and its own numeric width. - -17. **What exactly is the connection operator?** (clause 12) - - (a) Do its "both ends" mean both halves in one component, or one component on each - side of the wire? - - (b) Is it a new crate or process, or a role for existing code (the browser - `Worker`, the socket server's `Registry` and transport, in-process code in the TUI)? - - (c) Who owns keep-alive and resumption state, and is that backend state? - - (d) Should one operator serve many connections and subscribe to the host on their - behalf, so that the host sends O(operators) per batch, not O(clients)? That saving - needs one operator per host process serving many clients, so per-connection state - then lives inside a shared operator. - - *Recommendation:* (a) one operator per host process: a host half shared by its - connections, and a client half in every front end. (b) a role of the existing - transport crates, not a new process. (c) the host half holds resumption state, keyed - by a token the client holds, as delivery state, not UI state. (d) yes, for the socket - server. The TUI and the browser worker each serve one client, so there the operator is - a function call. +5. **Which columns get a host-maintained width?** Columns whose cells are text? What + about a column holding both text and numbers? *Recommendation:* text cells get the + maintained width, and the view layer takes the larger of that and its own numeric + width. From 38058ec73f1198b6c3b8d14c0784dd70e6ff759c Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:43:16 -0700 Subject: [PATCH 075/106] docs(adr): move ADR 007's amendment to its own file, stating only what it changes (improvise-cq1q.1) On the user's instruction an ADR states only what its decision changes, and mentions an existing convention only where it conflicts. The accepted ADR 007 is restored to main's text plus a one-line pointer, so merging this PR keeps it; the amendment is context/adr/007-stable-identities- amendment-1.md (280 lines, from 784), with a table of the accepted rules it replaces. The doc site lists the new file, and core.md section 20 points at it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- .../adr/007-stable-identities-amendment-1.md | 280 ++++ context/adr/007-stable-identities.md | 1400 +++++++++-------- context/code/core.md | 5 +- scripts/build-doc-site.mjs | 1 + 4 files changed, 1050 insertions(+), 636 deletions(-) create mode 100644 context/adr/007-stable-identities-amendment-1.md diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md new file mode 100644 index 00000000..c4bf95a1 --- /dev/null +++ b/context/adr/007-stable-identities-amendment-1.md @@ -0,0 +1,280 @@ +# ADR 007, amendment 1 — Stable identities + +**Ticket:** improvise-sboz (this amendment improvise-cq1q.1; implementation epic +improvise-cq1q) · **Builds on:** ADR 001, ADR 002 + +## Status + +**ADR 007 is accepted** (PR #210, 2026-09-25), and `007-stable-identities.md` stands. **This +amendment is proposed and not yet approved** (PR #326). This file states only what the +amendment changes or adds, numbered by the accepted ADR's clauses; everything else in +the accepted ADR is still true. Where the amendment conflicts with an accepted rule or an existing +behaviour, the accepted rule binds until the amendment is approved: + +| clause | accepted rule or existing behaviour | the amendment | +|---|---|---| +| 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | +| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill` | +| 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays, never declared, and referred to by id (`_measure`) | +| 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | +| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | +| 2, 5 | a collapsed entry and a group reference name a group by name | by its id | +| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit | +| 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id | +| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders read display names in input text; command-language text names things by id | +| 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | +| 12 | the regression test commits the unedited text | the test edits a non-name token and commits | + +## Context + +Implementing the accepted ADR (epic improvise-cq1q) and using it raised points the +accepted text does not settle: whether groups and measures carry ids, what a virtual +category is in a file, exactly what slugify produces, which inputs may read display +names, and which side resolves names in a formula edit. This amendment settles them. + +## Decision + +### 1. What carries an identity + +**Groups have ids.** A group has an id and a display, like a category or an item, and +its id is an ordinary id: unique in the document, generated by slugify (`group` is an +empty slug's ``), declared beside its display, referred to by id, created by +display, and renamed by a display rename. Its wire address is `GroupId`. In code, +PR #336 (improvise-cq1q.12) already adds a display to `Group` beside its key; that key +becomes the group's id. + +**Virtual ids.** +- **A leading `_` is reserved for virtual ids in general.** The virtual views have ids + `_records` and `_drill`, displayed `_Records` and `_Drill`: they are addressed on the + wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could + collide with a user's view. Slugify never produces a leading `_`, so the grammar does + not check the reservation. +- **Virtual ids are lower case.** The code, the wire, `ViewState` and the browser + client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, and move + to `_index`, `_dim` and `_measure` in the re-key step. +- **Virtual categories and measures are defined by criteria, not declared.** Each + contains whatever in the document meets its criterion: `_measure` holds the measures, + and `_index` and `_dim` hold the record positions. Their ids and displays are + hardcoded in the application, so a file never declares one, and a line that refers to + one spells its id: `_measure: column`, `region=east, _measure=revenue = 1200`. +- **Conflict with old files:** they declare and name `_Measure`. The loader recognises + `_Measure`, `_Index` and `_Dim` by their hardcoded displays, reads an old + `## Category: _Measure` section only for the stored measures it lists, and the first + save writes `_measure`. +- **Records mode's synthetic positions are values, not ids.** The row coordinate under + `_index`, and by the same reasoning the column coordinate under `_dim`, are values of + those virtual categories. They take no part in id uniqueness and cannot collide with + a user's item. + +**Formulas and measures.** A measure is the normal formula target (`profit = revenue - +cost`); a dimension target is rare, for a derived dimension. +- **A formula is a kind of measure, so a `_Measure` formula's id is its measure's id.** + `Profit = Revenue - Cost` is one thing, `profit`: other formulas, each Profit cell's + `CellKey`, and `ReplaceFormula`/`RemoveFormula` all name `profit`. +- **Each measure's cells are keyed by one handle:** `ItemHandle::Item(id)` for a stored + measure, `ItemHandle::FormulaTarget(id)` for a formula's measure, never both. +- **A formula never merges into a stored measure.** With a stored `profit` (displayed + `Profit`), adding `Profit = Revenue - Cost` creates a second measure with a fresh id, + `profit-`, by the ordinary collision rule. The stored measure keeps `profit` and + its data; removing the formula removes only `profit-`. + - **Conflict with display uniqueness:** both are displayed `Profit` in `_measure`'s + scope. This is the one permitted case; the two are told apart by id (the formula + panel shows ids where a display is ambiguous, and a typed `Profit` matching both is + refused with both ids listed, clause 12). +- **A target outside `_Measure` is a real item** with its own `ItemId`, and its formula + has a separate `FormulaId`: a formula `Tax` in `CustomCat` gets item id `tax`, then + formula id `tax-`. +- **Formulas may refer to anything by id.** A function in a formula may accept any + id-bearing thing (category, item, measure, formula, view, group) by its id. Whether a + function takes the thing or its value (an item's display, say) is part of that + function's signature. + +### 2. Storage keys + +- **`CellKey` pairs are `(CategoryId, ItemHandle)`**, so a `_Measure` formula cell's + coordinate is `FormulaTarget` without claiming an `ItemId` the document does not + hold. +- **A collapsed entry names its group by the group's id**, and `Group` gains a + `display`. + +### 4. How an id is made + +**Slugify's steps:** +1. **Casefold**, with Unicode full case folding (`Straße` → `strasse`). +2. **Replace every character outside `a`–`z`, `0`–`9` and `-` with `_`.** Ids' + alphanumerics are ASCII and include `-`; whitespace, punctuation and every non-ASCII + letter become `_`. +3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 or + were in the display. +4. **Remove every leading and trailing `_` and `-`**, repeating until neither end holds + either. +5. **If the slug starts with a digit, prepend `n`**, so no id is purely numeric or + starts with a digit: `2024` → `n2024`. +6. **If nothing is left, use `-`** from the shared counter. A display of only + `-`, only `_` or only non-ASCII letters ends here. + +**The syntax of an id:** a lower-case ASCII letter, then `a`–`z`, `0`–`9`, `-` and +single underscores, never ending in `_` or `-`. Slugify's output has this shape, so +slugifying an id returns it unchanged, and a collision suffix `-` keeps the shape. + +| display | id | +|---|---| +| `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` | +| `a_ b`, `a__b` | `a_b` | +| `Q1 - 2024` | `q1_-_2024` | +| `East `, `East_`, `-East-` | `east` | +| `Region(code)` | `region_code` | +| `Crème Brûlée` | `cr_me_br_l_e` | +| `2024`, `_2024` | `n2024` | +| `2024-Q1` | `n2024-q1` | +| `Ü`, `-`, `%` | `-` | + +**A suffix and a hyphenated slug can look alike.** `East-1` slugs to `east-1`, which is +also what the counter may make of `east`. They never collide, because every id a +document holds is taken however it was made: a display slugging to a held `east-1` +becomes `east-1-`, and a held counter candidate is skipped. + +**Hand-written ids obey every syntax rule of a generated id.** `east-1`, `item-4` and +`q1_-_2024` load; `-east`, `east-`, `2024`, `a__b` and `_draft` are refused at load. + +**Two branches of one file can merge into a file that will not load**, if each adds a +different thing that generates the same slug: the merged file declares that id twice, +and the load is refused naming both. Stable identity across copies of a file is not +promised (improvise-8tk5). + +**Built code.** `crates/improvise-core/src/model/id.rs` implements the accepted steps +(`context/code/core.md` §20). After approval it must replace rather than strip, collapse +`_`, trim a trailing `_` and both ends' `-`, prepend `n`, make `claim` refuse what a +generated id could not be, and add the virtual views and `IdKind::Group`. + +### 5. Persistence + +- **The loader also refuses a hand-written id that breaks the syntax above**, naming + the line; the grammar's `id_token` stays permissive. A declared id never begins with + `_`, since a file never declares a virtual category. +- **Formula text quotes only keyword ids** (`|sum|`). No id starts with a digit, and + the formula grammar's bare identifier already admits `-`. +- **Groups:** a group reference (`[coastal]`) and the group in a `collapsed:` line are + ids, and a group definition declares a display and an id (`> Coastal(coastal)`). +- **A header such as `## Category: Budget (2024)`** in a new-version file gives the id + `2024`, which is refused. Import pipe-quotes `Region(code)` as + `## Category: |Region(code)|(region_code)`. + +**An axis line is tried before `format:`, and a format that reads as an axis line is +pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice and `format_line` +takes the rest of the line, so a category with id `format` would lose its axis line to +the number format; the new-version arm puts `axis_line` first. Then `:set-format none` +would write `format: none`, read as an axis line, so the writer pipe-quotes a format +that would also match `axis_line` (an axis keyword alone or followed by `, `, or a +value starting with `|`), and the new arm's `format_line` is +`pipe_quoted ~ &NEWLINE | rest_of_line`. An old file's `format: none` is read as before +and written quoted on first save. Required tests: a category displayed `Format` on a +row axis round-trips; a view whose format is each axis keyword round-trips beside a +category `format` on an axis. + +**A formula line.** Today: + +``` +## Formulas +- Profit = Revenue - Cost +- Tax = Revenue * 0.1 [CustomCat] +``` + +Under the amendment: + +``` +## Formulas +- Profit(profit) = revenue - cost +- tax(tax-1) = revenue * 0.1 [customcat] +``` + +- **A measure formula** has no tag: `_measure` is implicit and never written as a tag. + The line is the measure's only declaration: `Profit` is its display and `(profit)` + the id it shares with its formula. +- **A dimension formula** has a tag, which is always the target category's id. The + target is already declared with its display in its category, so the line names it by + item id (`tax`), and `(tax-1)` is the formula's own id. +- So the loader tells them apart by the tag. The expression is over ids; a reference to + a formula target, and a `WHERE`/`IS` value, is the id text of its `ItemHandle`. A + target display that looks like `Name(x)` is pipe-quoted: `- |Margin(%)|(margin) = …`. + +### 6. Display names and the wire + +- **Only CSV import and the `.improv` loaders turn display names in input text into + ids.** They read text people wrote. An old file's names are resolved there as a + migration. +- **Command-language text names things by id.** Anything that uses the command language + as a scripting language is a script, wherever the text comes from (a file, + `improvise cmd`, another program). A script naming something by display fails; it + gets a display only through a dedicated function such as `display_for(id)`. +- **Interactive input** (TUI, browser) matches either a display or an id, as long as it + is unambiguous which thing the user is picking. Ids win when it is not, and the + display is shown as a preview (clause 12). +- **Groups on the wire:** `AddItemInGroup`'s `group` is a group id, a new group is + created by display, a group is renamed by id and new display, and `FrameGroup` + carries the group's id and display. +- **The formula panel** treats a display equal to another thing's id as ambiguous, so it + never shows a bare name that ids-first would bind elsewhere. +- **`ViewState::active_view` and `Workbook::active_view`** are keyed by id + (improvise-p003). +- **A `_Measure` formula-target row** is removed by `RemoveFormula` and renamed by a + display rename of the measure (improvise-i0n7). +- **CheckedBatch's created-record exemption is removed** (improvise-p003). It exists for + frames like `[AddItem(East), SetCell(East)]`, and a frame cannot address what it + creates, so nothing produces one. + +### 7. Rejections + +`Rejection::Gone(Named)` gains `Named::Group`. + +### 8. `RecordHash` + +`RecordHash` hashes `(CategoryId, ItemHandle)` pairs spelled as id text. Ids are unique +across kinds, so it stays canonical; a record excludes `_measure`, so a `FormulaTarget` +is never part of one. + +### 11. Migration + +- Step 2 adds `GroupId`, a display on `Group`, and `ItemHandle`. +- Step 3 moves the virtual categories' keys to `_index`, `_dim` and `_measure`. +- A new step 7 builds clause 12's client-side capture and resolver, ids first from the + start. + +### 12. Who resolves names in a formula edit + +**The client resolves; the host never sees a display.** +- The client holds the display→id mapping captured when the editor opened, as it holds + `OpenedCell`. The TUI and the browser share one resolver in `improvise-protocol`. +- **A name new to the edit** is resolved by the client against its id-bearing + projection: a typed name equal to an id names that thing, even if it is also another + thing's display; otherwise a display exactly one thing has names it; a display + matching several things is refused with their ids; a name matching nothing is + refused. Ids are matched exactly. Ids win wherever there is ambiguity, now and once an + autocomplete exists; the autocomplete prefers id matches and previews the display. +- **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are + in id form; an id that no longer exists is `Rejection::Gone`. + +**What ids-first costs.** Item A is displayed `East` (id `east`) and item B is displayed +`east`, so it got `east-1`. Typing `east` names A; B is reached as `east-1`. The panel +therefore shows B as `east-1`, never as a bare `east`. + +**Tests.** +- The accepted regression test's step 3 becomes: edit something that is not a name + (`Revenue - Cost * 1`) and commit. Committing unedited text cannot tell a correct + capture from none, because a client may pass unchanged text as a no-op + (improvise-hexb). +- Ids-first: with A (`East`, `east`) and B (`east`, `east-1`), typing `east` in text not + captured at open must bind A (and fail under displays-first), `east-1` must bind B, + and a display that is nobody's id must still resolve by display. A fixture where no + display equals an id proves nothing. + +## Consequences + +- Scripts that name things by display break and must be rewritten over ids. +- Two measures can share a display; every place that shows displays falls back to ids. +- Old files' `_Measure` declarations and hyphen- or digit-led names change on first save. +- `id.rs` and its tests need the follow-up in clause 4. + +## Open questions + +None. diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index 7f86354c..e044be5c 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -1,474 +1,589 @@ # ADR 007 — Stable identities for categories, items, formulas and views -**Ticket:** improvise-sboz (implementation epic improvise-cq1q; this amendment -improvise-cq1q.1) · **Unblocks:** improvise-fz77 · **Revises:** improvise-km71's -interim `FormulaId` (PR #214) and improvise-fdk1's `RenameRecordCoordinate` -(PR #221) · **Builds on:** ADR 001, ADR 002 · Review tickets filed against an early -draft call this document "ADR 005". +**Amended:** an amendment is proposed in `007-stable-identities-amendment-1.md` +(PR #326, not yet approved). Until it is approved, this text binds. -## Status +**Status:** accepted (2026-09-25). The user approved revision 7. Revision 8 then +followed the adversarial review of that head: it adds clause 12, the ADR's own reading +of when a formula edit's names become ids, and corrects stale facts about main. The +user approved revision 8 on PR #210 (owner comment 5824468104, 2026-09-25 00:17:10Z: +"looks good, approved"). · +**Decided by:** the user; see "What the user decided" · +**Ticket:** improvise-sboz · **Unblocks:** improvise-fz77 · +**Revises:** improvise-km71's interim `FormulaId` (landed in PR #214) and +improvise-fdk1's `RenameRecordCoordinate` (landed in PR #221) · +**Builds on:** ADR 001, ADR 002 · **Depended on:** improvise-tt22 (remove legacy JSON), +which landed in PR #224 -**Accepted** on 2026-09-25 (PR #210). **An amendment is proposed and not yet -approved** (PR #326). **Every rule tagged [Amendment] is proposed; everything -without the tag is accepted and binding.** The tag is the complete list of what the -amendment adds. +**Numbering.** The ADR was first filed as 005, but improvise-l2b8 already claims ADR 005 +(exact decimal arithmetic) and improvise-ilvl claims 006. The review tickets filed +against the first draft still say "ADR 005" in their titles; they mean this document. -Where the amendment replaces accepted text, the accepted rule binds until the -amendment is approved: +**Direction from** the user, 2026-09-23, in three messages: -| clause | accepted rule | the amendment proposes | -|---|---|---| -| 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | -| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill`; virtual categories and measures are defined by criteria with hardcoded ids and displays, are never declared in a file, and are referred to by id (`_measure`) | -| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | -| 1 | (silent; today `measure_item_names` merges a formula into a stored measure of the same name) | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | -| 2, 5 | a collapsed entry and a group reference name a group by name | by its id | -| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended to a slug that starts with a digit | -| 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` and may begin with `_` only if reserved | a hand-written id obeys every syntax rule of a generated id | -| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders turn display names in input text into ids; command-language text, from any source, names things by id | -| 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | -| 12 | the regression test commits the **unedited** text after the concurrent renames | the test edits a non-name token and commits, because a client may pass unedited text as a no-op without resolving any name | +> "there would be a lot of benefit to categories/formulas having stable identities, even if +> they were only transient: it would help pin editing operations and distinguish them from +> delete/remove" -No questions are open. +> "It's probably worth a file format change (in a opt-in way, that gets rectified on save if +> the input file was missing them)" + +> "Maybe something like `View: Default(default)` and the standard +> `View: |Default (default)|(default)` for quoting." + +## What the user decided + +The user answered the **first draft** (`context/adr/005-stable-identities.md` at +`9d97664c`) with inline comments on PR #210, and then posted one comment on the thread. +Each answer is quoted verbatim below, with where it was attached and what it overturned. +**Five of the first draft's positions were reversed.** The rest of this document is +rewritten to match. + +**1. Storage keys by id; the human name becomes a display field.** On first-draft +line 92, the clause "`CellKey`, `DataStore`, views and formula text **stay keyed by +name**": + +> "Shouldn't it be easy to replace this to key by ID? There's no type change so, at worst, +> it would be renaming field access. We could also add the name as something like +> `display` and reserve `name` or whatever it is currently for the id." + +**2. The point of ids is that keys use them.** On first-draft line 97, "ids do not make a +rename cheap. They make it *expressible*": + +> "This is a lot lower value if CellKeys, etc. don't refer to the new id" + +**3. Every reference uses the id.** On first-draft line 275, the heading of clause 8, +"ADR 002: `RecordHash` stays defined over names": + +> "No, anything referring to something with an id should refer to the id as the stable +> identity to avoid glitches." + +This answers Q7 by reversing it, and reverses clause 5's "references carry names" too. + +**4. Q1, the id's internal type.** On first-draft line 469, the recommendation to wrap "a +small string": + +> "We probably can reuse symbols as the internal type with interning?" + +Ids are strings, interned through the existing `SymbolTable`. + +**5. Q2, formula id placement.** On first-draft line 473, recommendation F1 +(`- Profit(3) = Revenue - Cost`): + +> "Yes, before the equal to the right of the name" + +**6. Q3, header syntax.** On first-draft line 480, the claim that "a header with a space +before its parenthesis (`Sales (copy)`) is unaffected": + +> "Space before the open paren is optional: document this edge case that might break +> existing docs." + +**7. Q4, data keys.** On first-draft line 485, the recommendation "no — names only": + +> "In Data, we should prefer ids that are guaranteed unique to names that might not be." + +**8. Import generates ids; legacy JSON goes.** Thread comment, 2026-09-23 06:51: + +> "For CSV or JSON import, the dimensions are to be treated as the human friendly name and +> never parsed for IDs: during import, ids should be generated by a slugify algorithm that +> strips non-alphanumeric characters and replaces spaces with hyphens(collapsing runs). +> The only supported native file format is .improv, cleanup any legacy JSON format." + +Legacy JSON removal is its own implementation ticket, **improvise-tt22**. This ADR +depended on it and did not implement it; it landed in PR #224. It also dissolves +improvise-65kf: with no legacy JSON to load, there are no numeric JSON ids to accept. + +**The answers change the ADR's centre of gravity.** The first draft put an id *beside* +the name as an index. The user's answers make the **id the key** and the name a +**display** string. So a rename becomes cheap: it changes one display field and moves no +key, no surrogate and no basis tier. The first draft had called that impossible. + +### Second round: answers on revision 2 + +The user answered revision 2 (`context/adr/007-stable-identities.md` at `5e46ce03`) +with thirteen inline comments on PR #210, posted 2026-09-23 15:03–15:11Z. Each answer +is mapped to the question **on the line it is attached to**, and quoted verbatim. + +**9. Q5, group ids.** On line 468, inside Q5 ("Recommendation: not now … the one +exception to 'every reference is an id'"): + +> "This seems right" + +Groups get no id in this ADR. + +**10. Q6, per-entity versions.** On line 469, Q6 ("Recommendation: not in this ADR"): + +> "Agreed" + +**11. Q8, a duplicate or unknown id.** On line 473, the recommendation to refuse the +load: + +> "Yeah, don't load the file because you can't safely resave it" + +**12. Q9, the reading of "opt-in".** On line 476, the recommendation that an id-less +file is read forever and every save writes ids: + +> "Yes" + +**13. Q10, id reuse after a reload.** On line 485, inside option **(b)**, "Accept +reuse across a reload (recommended)": + +> "Yes, b" + +This is option (b), and (b) was the recommendation. Within a session, retirement still +prevents reuse. Across a save and reload, a removed name that comes back gets its old +slug back. + +**14. Q11, who assigns ids.** On line 492, inside option (a): + +> "The host assigns" + +**15. Q12, old-version files and the new header rule.** On line 502, inside option (a), +"No (recommended)": + +> "Yes, a as long as version is unambiguously marked" + +Old files are read as they always were, on the condition that the version is marked +unambiguously. Clause 9 says how the loader meets that condition. + +**16. Q13, slugify.** Five answers, one per item: +- **(a) where it applies.** On line 510: + > "All id generation" +- **(b) case.** On line 512, against the recommendation to keep case: + > "lower case only" +- **(c) `_` and `-`.** On line 515: + > "Yeah, keep both" +- **(d) non-ASCII letters.** On line 518: + > "Strip non lower alphanumeric or -_" +- **(e) the empty slug.** On line 520, the recommendation `-`: + > "Yes" + +So an id is lower case, and its alphabet is `a`–`z`, `0`–`9`, `-` and `_`. Clause 4 +states the rule. Q15 asked the user to confirm the order of its steps (answer 18). + +**17. Q14, formula text.** On line 529, the recommendation "ids in the file": + +> "Display in the formula panel but not as column/row/page headers." + +This does not contradict ids in the file, which answer 3 already requires. It answers +**where formula text is displayed**, and it could be read two ways. That was Q16, +which answer 19 settles. + +### Third round: answers on revision 3 + +The user answered revision 3 (`context/adr/007-stable-identities.md` at `18c82e1c`) +with three inline comments on PR #210, posted 2026-09-24 00:34–00:36Z. Each is mapped +to the question on the line it is attached to, and quoted verbatim. + +**18. Q15, the order of slugify's steps.** On line 571, Q15's recommendation "fold, then +strip": + +> "casefold, replace space with `_`, strip" + +The fold comes first, as recommended. The answer also **changes answer 8's middle +step**: a space becomes `_`, not `-`. Two details of that sentence are Q18. + +**19. Q16, what the formula panel shows.** On line 584, the paragraph that declined to +recommend a reading: + +> "It should be at least `Profit (profit) = Revenue - Cost` but if `Revenue` is +> ambiguous, you'll have to display the slugs. Working with slugs internally and making +> the friendly names view-layer is almost always the right move here." + +So the panel shows formula text over **display names**, with the target's id beside it +in parentheses. Where a display name in the text is ambiguous, the panel shows the id +instead. Ids are what the file and the model hold, and display names belong to the view +layer. This also settles answer 17: the thing displayed "in the formula panel but not as +column/row/page headers" is the target's **id**. Whether an id is enough to resolve an +ambiguous name is Q19. + +**20. Q17, capitals in a hand-written id.** On line 589, the question "May a +hand-written id contain capitals?": + +> "no." + +This is option (a). `id_token` is lower case only, and a header ending in a +parenthesised run that contains a capital loads as a whole-line display with a load +warning. The virtual categories are spelled with capitals today. That is Q20. +(Answer 21 later made a new-version file all-or-nothing, so that header is now refused +rather than warned about. See clause 5.) + +### Fourth round: answers on revision 4 + +The user answered revision 4 (`context/adr/007-stable-identities.md` at `024aa2d3`) +with five inline comments on PR #210, posted 2026-09-24 06:05–06:08Z. Each is mapped +to the question on the line it is attached to, and quoted verbatim. + +**21. Q18(a), runs of spaces, and the collision suffix.** On line 658, Q18(a) ("Does a +run of spaces collapse to one `_`?"): + +> "runs of spaces are collapsed to one `_` and then `-###` numeric suffix is added if +> duplicate (only when importing a file without IDs: a file must either have no IDs at +> all or always have IDs)" + +This decides three things: +- a run of spaces collapses to one `_`, as recommended; +- a duplicate slug gets a numeric suffix after a `-`; +- **a file has ids on everything or on nothing.** A new-version file with a declaration + missing its id is refused, like any file that cannot be resaved safely (answer 11). + So the earlier rule that a hand-written id wins over a generated one in the same file + is gone. + +The parenthesis also limits the suffix to generating ids for a file that has none. Two +details of that are Q22. Q18(b), what "casefold" means, was not answered in this round +(answer 26 settles it). + +**22. Q19, whether an id disambiguates formula text.** On line 678, option (a) ("Make +item ids unique in the document"): + +> "Ids have to be globally unique." + +This is stronger than option (a). **No two ids in a document are equal, whatever their +kind**, so a category, an item, a formula and a view never share one. An id in formula +text or on the wire therefore always names exactly one thing. + +**23. Q20, the virtual categories.** On line 691, option (a) ("Lower-case ids, +capitalised displays"): + +> "Virtual categories have IDs: we need to add a "slug" version such as `_index`, but +> no non-virtual id can start with `_`" + +The ids are `_index`, `_dim` and `_measure`, and the displays stay `_Index`, `_Dim` and +`_Measure`. A leading `_` is reserved for them. Clause 4 says how slugify keeps a +generated id out of that space, and clause 5 says what happens to a hand-written one. + +**24. Q21, an old file naming an undeclared item.** On line 708, option (a) ("Declare +the item at load"): + +> "yes, this is right." + +**25. The file version.** On line 731, the Consequences line "A new file version.": + +> "File version should be added explciitly if it's not already part of the format." + +It is already part of the format. The first line of every `.improv` file is a +mandatory, exact version literal (`version_line = { "v2025-04-09" ~ NEWLINE ~ … }` in +`improv.pest`), and the writer emits it. So the new format is marked by a **new +literal**, and nothing is inferred. Clause 10, new in this revision, states the rule in +one place. + +### Fifth round: answers on revision 5 + +The user answered revision 5 (`context/adr/007-stable-identities.md` at `e044c968`) +with three inline comments on PR #210, posted 2026-09-24 19:50–19:51Z. Each is mapped +to the question on the line it is attached to, and quoted verbatim. + +**26. Q18(b), what "casefold" means.** On line 766, the recommendation to take +"casefold" literally, as full case folding: + +> "full unicode" + +Slugify's first step is **Unicode full case folding**, so `Straße` → `strasse`. + +**27. Q22(a), how the suffix is spelled.** On line 771, the recommendation "a plain +number": + +> "Just a plane number." + +The suffix is `-` followed by a plain decimal number, with no padding. + +**28. Q22(b), collisions outside import.** On line 782, option (i), "Suffix it there +too": + +> "Any ambiguous id gets a suffix, if if the id is between different types of things +> identified. Don't bother implementing a strict order here: `east` `east-1` +> `east-2`, etc. just do what gensym does in CL: global increment-only counter appended +> when dedupe needed." + +This decides three things: +- **every** colliding id gets a suffix: on import, in a session, and across kinds (a + category and an item that slug alike); +- the number comes from **one counter for the document that only ever increments**, as + Common Lisp's `gensym` does, so it is not "the lowest free number"; +- the suffix is appended **only when a dedupe is needed**, so an id that does not + collide stays bare. + +Clause 4 is rewritten to match. The rules this replaces (lowest free number, and +suffixing only on import) were revision 5's. + +### Sixth round: answer on revision 6 + +The user replied at 2026-09-24 22:09Z to their own answer 28, on line 782 of +`e044c968`: + +**29. Hand-written ids are canonical.** + +> "You have to treat the file's hand-written ids as canonical, doing otherwise will +> corrupt the data section." + +This constrains answer 28. The `gensym` suffix applies only to an id the program is +**generating**, on import, when an id-less file loads, or when the host creates +something in a session. **An id written in a file is never renamed**, because every +data key and reference in that file spells it. Clause 4 says what follows: +- a generated id that would collide with a file's id takes the suffix, and the file's + id keeps its spelling; +- two identical hand-written ids in one file are refused at load, not repaired. ## Context **Today the name is the identity.** `Model::categories` is an `IndexMap` keyed by name. `CellKey` and `DataStore`'s interned symbols are names. Views, axis maps, page -selections, hidden and collapsed sets, and formula text all refer by name. That costs -three things: -- **An edit cannot be told apart from a delete followed by an add.** No rename - operation exists for a category or an item. A rename is remove plus add, which loses - every cell (improvise-fz77). A formula is identified by `(target, target_category)`, - and `Model::add_formula` upserts on that pair. -- **An edit whose target was concurrently replaced cannot be told apart from an edit - of the replacement.** -- **A records-mode coordinate rename names its category and item by name** - (`RenameRecordCoordinate { record, category, item }`, `String` operands). +selections, hidden and collapsed sets, and formula text all refer by name. The user's +first message names what that costs: -**What exists already.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize` -aliases from monotone counters. No production caller reads them, they have never been -written to `.improv`, and `View` has no id. km71 landed an opaque interim `FormulaId` -over `(target, target_category)`, built only by `Formula::id()` (`context/code/core.md` -§18). It is on the wire in `ModelCommand::ReplaceFormula { id, was, raw }`, -`FormulaSlot::Existing` and `FormulaEntry`. Because it is still a name pair, -`Model::replace_formula` refuses to rename a target outside `_Measure` -(improvise-kb1d). Legacy JSON is gone (improvise-tt22, PR #224), so `.improv` is the -only native format. +- **An edit cannot be told apart from a delete followed by an add.** No rename operation + exists for a category or an item. A rename is remove plus add, which loses every cell + (improvise-fz77). A formula is effectively identified by `(target, target_category)`, + and `Model::add_formula` *upserts* on that pair. improvise-km71 (PR #214) made formula + editing its own operation, with an interim id built from that pair (below). +- **An edit whose target was concurrently replaced cannot be told apart from an edit of + the replacement.** +- **A records-mode coordinate rename names its category and item by name.** + improvise-fdk1 (PR #221) landed `RenameRecordCoordinate { record, category, item }` + with `String` operands (below). + +**What already exists.** `Category.id: CategoryId` and `Item.id: ItemId` are `usize` +aliases, assigned from monotone counters. **No production caller uses either value.** +`Model::add_category` and `Category::add_item` return them on the idempotent path, and +nothing reads the returned value; the only reader is one test. Neither has ever been written to `.improv`. `View` has no +id. + +**`FormulaId` exists, as an interim.** km71 landed an opaque `FormulaId` whose private +fields are `(target, target_category)`, built only by `Formula::id()` +(`context/code/core.md` §18). It is on the wire in +`ModelCommand::ReplaceFormula { id, was, raw }`, in `FormulaSlot::Existing` and in +`FormulaEntry`. It was made opaque so that this ADR changes one struct rather than every +signature. It is still a name pair, so a formula whose target is renamed gets a +different `FormulaId`. `Model::replace_formula` therefore refuses to rename a target +outside `_Measure` until this ADR lands (improvise-kb1d). + +**Legacy JSON is gone.** improvise-tt22 landed in PR #224, so `.improv` is the only +native format. ## Decision -**Categories, items, formulas and views each have an id, and so do groups -[Amendment]. The id is the key.** The current `name` field becomes the id, and the +**Categories, items, formulas and views each have an id, and the id is their key.** A +category or item's current `name` field *becomes* the id (the user's suggestion), and the human name moves to a new `display` field. -- **Every reference is an id**: `CellKey`s, axis maps, page selections, hidden and - collapsed sets, formula references, data keys, and every wire operand that names an - existing thing. A reference to the stable identity cannot glitch when something is - renamed or replaced. -- **Display names are for people.** They appear on screen, are typed into - minibuffers, and are turned into ids at the input edge. **[Amendment]** Clause 6 narrows - which readers turn display text into ids. +- **Every reference is an id.** That covers `CellKey`s, axis maps, page selections, + hidden and collapsed sets, formula references, data keys, and wire operands that name + an existing thing. +- **Display names are for people.** They appear on screen, are typed into minibuffers, + and are resolved to ids at the input edge. + +Twelve clauses. ### 1. What carries an identity -| kind | id | wire address | -|---|---|---| -| category | yes | `CategoryId` | -| item | yes | `ItemId` | -| formula | yes | `FormulaId` | -| view | yes | `ViewId` | -| group | **[Amendment]** yes, with a display | `GroupId` | -| record | `RecordHash`, over ids (clause 8) | unchanged shape | +| kind | id | unique within | wire address | +|---|---|---|---| +| category | yes | the document, across every kind (answer 22) | `CategoryId` | +| item | yes | the document, across every kind (answer 22) | `ItemId` | +| formula | yes | the document, across every kind (answer 22) | `FormulaId` | +| view | yes (the user's example) | the document, across every kind (answer 22) | `ViewId` | +| record | `RecordHash`, now over ids (clause 8) | the document | unchanged shape | +| group | **no**, not in this ADR (answer 9) | | | -**An id always names the same thing, so ids are unique in the whole document**, across -every kind. An item id names one item without its category. A `CellKey` still pairs -each item with its category, and an operation that creates an item still names the -parent category. +**Ids are unique in the whole document** (answer 22). One namespace covers categories, +items, formulas and views, so an item id names one item without its category. A +`CellKey` still pairs each item with its category, because the key needs the +categories. An operation that creates an item still names the parent category. -**[Amendment] Groups.** A group has an id and a display, like a category or an item, -and its id behaves like every other id: it is unique in the document, generated by -slugify (`group` is an empty slug's ``), declared beside the display in the file, -referred to by id, created by display, and renamed by a display rename. In code, -PR #336 (improvise-cq1q.12, open) already adds a private display to `Group` beside its -key; under this amendment that key becomes the group's id. - -**Reserved ids.** The virtual categories have ids `_index`, `_dim` and `_measure`, -displayed `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** -**[Amendment]** A leading `_` is reserved for virtual ids in general. The virtual views -have ids `_records` and `_drill`, displayed `_Records` and `_Drill`; they are addressed -on the wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and -could collide with a user's view. Slugify never produces a leading `_` (clause 4), so -the reservation needs no check in the grammar. -- **[Amendment] Virtual ids are lower case.** The code, the wire, `ViewState` and the - browser client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, - and move to the lower-case ids in the re-key step (clause 11). -- **[Amendment] Virtual categories and virtual measures are defined by criteria, not - declared.** Each contains whatever in the document meets its criterion: `_measure` - holds the measures, and `_index` and `_dim` hold the record positions. Their ids and - displays are hardcoded in the application, so a file never declares one. Where a - line refers to one, it spells the virtual id: `_measure: column`, `_index: none`, - `region=east, _measure=revenue = 1200`. -- **[Amendment] An old file's virtual names are recognised, not migrated as - declarations.** The loader recognises `_Measure`, `_Index` and `_Dim` by their - hardcoded displays, and an old `## Category: _Measure` section is read for the - stored measures it lists, not as a declaration of `_measure`. The first save writes - `_measure` wherever the file refers to it. The stored measures themselves - (`Revenue`, `Cost`) are listed as the format lists them today, each with its id. -- **[Amendment] Records mode's synthetic positions are values, not ids.** The row - coordinate under `_index` is a value of that virtual category. So, by the same - reasoning, is the column coordinate under `_dim` (a column index, gotcha 1). They are - never persisted or typed, take no part in id uniqueness, and cannot collide with a - user's item. - -**Formulas and measures.** **[Amendment]** A measure is the normal formula target -(`profit = revenue - cost`). A dimension (a category other than `_Measure`) is a rare -one, for a derived dimension. -- **A `_Measure` target has no item of its own**, and its handle is its formula's - `FormulaId`. So the category panel's `_Measure` rows use - `ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). -- **[Amendment] A formula is a kind of measure, so there is no distinction between a - `_Measure` formula's id and its measure's id.** A measure has an id and a display, - as everything else does. `Profit = Revenue - Cost` is one thing, `profit`: another - formula refers to it as `profit`, each Profit cell's `CellKey` holds `profit` under - `_measure`, and `ReplaceFormula` and `RemoveFormula` address `profit`. -- **[Amendment] Each measure's cells are keyed by one handle.** A stored measure is - keyed `ItemHandle::Item(id)`. A formula's measure is keyed - `ItemHandle::FormulaTarget(id)`. The id is the measure's either way, and no column is - keyed by both. -- **[Amendment] A formula never merges into a stored measure.** If a stored measure - `profit` (displayed `Profit`) exists and the user adds `Profit = Revenue - Cost`, - the new formula is a second measure with a fresh id, `profit-` from the shared - counter (clause 4), like any colliding generated id. The stored measure keeps - `profit` and its data. Removing the formula removes `profit-` and leaves `profit` - untouched. The formula keeps the display the user typed, so both are displayed - `Profit` and are told apart by id: the formula panel shows the ids where the display - is ambiguous (clause 6), and a typed `Profit` that matches both is refused with both - ids listed (clause 12). -- **[Amendment] A target outside `_Measure` is a real item**, with its own `ItemId`, - and its formula has a separate `FormulaId`. A formula whose generated id would - collide takes a suffix like any other generated id (clause 4): a formula `Tax` in - `CustomCat` gets item id `tax`, then formula id `tax-`. -- **[Amendment] Formulas may refer to anything by id.** A function in a formula may - accept any id-bearing thing (a category, an item, a measure, a formula, a view, a - group), referred to by its id. Whether a given function takes the thing itself or its value - (an item's display, say) is part of that function's signature. Global uniqueness - makes every such reference unambiguous, and a function may read any part of a record, - so nothing further is needed here. +**Virtual categories have reserved ids** (answer 23): `_index`, `_dim` and `_measure`, +with the displays `_Index`, `_Dim` and `_Measure`. **No other id may begin with `_`.** +A `_Measure` formula target has no item of its own. Its handle is its +formula's `FormulaId`, which is why the category panel's `_Measure` rows need +`ItemHandle::{Item(ItemId), FormulaTarget(FormulaId)}` (clause 6). ### 2. Storage keys by id; `display` is the only place a human name lives +Following answers 1 to 3: - **Categories and views:** `Model::categories` and `Workbook::views` are keyed by id, - and each `Category`, `Item` and `View` gains a `display: String`. **[Amendment]** So - does `Group`. -- **Cells:** a `CellKey` is a sorted list of `(CategoryId, ItemId)` pairs. - **[Amendment]** The pairs are `(CategoryId, ItemHandle)`, so that a `_Measure` - formula cell's coordinate is `FormulaTarget` without claiming an `ItemId` the - document does not hold. - - The key stays sorted, now **by category id**, so the canonical form (gotcha 5) - is kept with a different sort key. - - `DataStore` interns ids exactly as it interns names today. - - An id's text names one thing, because ids are unique across kinds. + and each `Category`, `Item` and `View` gains a `display: String`. +- **Cells and derived surrogates:** a `CellKey` is a sorted list of + `(CategoryId, ItemId)` pairs. It stays sorted, now **by category id**, so gotcha 5's + canonical form is kept with a different sort key. `DataStore` interns ids exactly as it + interns names today. - **View configuration:** the axis maps, page selections, and hidden and collapsed sets - are keyed by ids. **[Amendment]** A collapsed entry names its group by the group's - id. -- **Formulas:** a formula's references are ids (clause 5), and the formula panel - renders them as display names (clause 6). + are keyed by ids. +- **Formulas:** a formula's references are ids (clause 5). The formula panel renders + them as display names (answer 19; clause 6). -**A rename costs one `display` write.** It moves no `CellKey`, no `RecordHash` and no -`layout_version`, so an edit a client composed before the rename still lands after it. -The grid's labels are display strings, so a rename must still dirty the grid and the -panels (`Dirty::of`, gotcha 35), but no basis tier moves. +**What a rename now costs:** one `display` write. It moves no `CellKey`, no `RecordHash` +and no `layout_version`, so an edit a client composed before the rename still lands after +it. That is the "avoid glitches" the user's rule is for. The grid's labels are display +strings, so the rename's effect must still dirty the grid and the panels +(`Dirty::of` gotcha 35), but no basis tier moves. -**The migration cost is the call sites.** Every API that takes a name today takes an -id instead: `Model::category(name)`, `Workbook::add_item(cat, item)`, `CellKey::new`'s -callers, the evaluator's `Expr::Ref` lookup and the `View` methods. Display→id -resolution moves to the places a human types. The field rename itself is cheap. The -change is mechanical but wide, and it is the bulk of the implementation. +**What the migration costs, stated honestly.** Every API that takes a name today +(`Model::category(name)`, `Workbook::add_item(cat, item)`, `CellKey::new` callers, the +formula evaluator's `Expr::Ref` lookup, `View` methods) takes an id instead. The +display→id resolution moves to the places a human types, such as the minibuffer commits, +the formula parser's reference resolution, and import. The change is mechanical but wide, +and it is the bulk of the implementation. The user judged it cheap ("at worst, it would +be renaming field access"), and for the field that is right. The call sites are the +cost. -**Display names stay unique within their scope**, as names are today. So a -human-typed name resolves to exactly one id within its scope, and a rename to a -display already in use there is refused. An item's scope is its category, so two -categories may each have an item displayed `Revenue`. Formula text crosses -categories, so a display may be ambiguous there. The ids differ, so the formula panel -shows the ids where a display is ambiguous. **[Amendment]** The one exception is a formula -whose measure shares a stored measure's display (clause 1): it is created anyway, and -the two are told apart by id. +**Display names stay unique within their scope**, as names are today. So a human-typed +name resolves to exactly one id, and a rename to a display name already in use is +refused. An item's scope is its category, so two categories may each have an item +displayed `Revenue`. Formula text crosses categories, which is where answer 19's +"ambiguous" arises. Their ids differ (answer 22), so the panel resolves the ambiguity +by showing the ids. ### 3. Ids are interned symbols -An id is a **string**, and in memory a `Symbol` from the existing `SymbolTable`: -copyable, compared as an integer. -- **A `Symbol` never crosses the wire or reaches the file.** It indexes one process's - table. The wire and `.improv` carry the id's text, and each end interns it on - arrival. -- **`Rejection` stops being `Copy`**, because `Rejection::Gone` (clause 7) carries an - id's text. `SaveFailed` carries no message only because `Rejection` was `Copy`. - Whether it gains one is the implementer's call. +Following answer 4, an id is a **string**, and in memory it is a `Symbol` from the +existing `SymbolTable`: copyable, with integer comparison. Two consequences. + +- **A `Symbol` never crosses the wire or reaches the file.** It is an index into one + process's table. The wire and `.improv` carry the id's **text**, and each end interns + it on arrival. +- **`Rejection` stops being `Copy`**, because `Rejection::Gone` (clause 7) carries an id's + text. `basis.rs`'s `SaveFailed` carries no message *because* `Rejection` was `Copy`, so + that reason lapses. Whether `SaveFailed` now gains its message is the implementer's + call. ### 4. How an id is made: slugify, one rule everywhere -**Every** id the app generates comes from the display name by **slugify**: on -import, on interactive creation, and when an id-less `.improv` file is loaded. So a -thing's id never depends on how it was made. An id is lower case. +Following answers 8, 16 and 18, **every** id the app generates comes from the display +name by **slugify**. That covers import, interactive creation, and assigning ids to an +id-less `.improv` file at load (answer 16(a), "All id generation"). So a thing's id +never depends on how it was made. The steps, in order (answer 18, "casefold, replace +space with `_`, strip"): -**The accepted steps:** -1. casefold, with Unicode full case folding; -2. replace each run of spaces with one `_`; -3. strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`; -4. remove any leading `_`; -5. if nothing is left, use `-`. +1. **Casefold, with Unicode full case folding** (answers 16(b), 18 and 26). Rust's + `str::to_lowercase` is not enough (`ß` stays `ß` and is then stripped), so this + needs a case-folding table or a small crate in `improvise-core`. It must pass the + `wasm` gate, because that crate is in the browser bundle. +2. **Replace each run of spaces with one `_`** (answers 18 and 21; answer 18 changed + answer 8's `-`). +3. **Strip every character that is not `a`–`z`, `0`–`9`, `-` or `_`** (answers 16(c) + and 16(d)). This strips non-ASCII letters, so an id never needs quoting in an + `.improv` reference (clause 5 notes where formula text differs). +4. **Remove any leading `_`** (answer 23: no non-virtual id may begin with `_`). The + answer states the constraint. Removing the underscores is this ADR's way of meeting + it, so `_Draft` → `draft`. +5. **If nothing is left, use `-`** (answer 16(e)), such as `category-1` or + `item-4`. `n` comes from the same counter as every other suffix (answer 28), not + from the lowest free number. -**[Amendment] The steps become:** -1. **Casefold, with Unicode full case folding**, so `Straße` → `strasse`. Rust's - `str::to_lowercase` is not enough (`ß` stays `ß`). The folding table or crate must - pass the `wasm` gate, because `improvise-core` is in the browser bundle. -2. **Replace every character outside `a`–`z`, `0`–`9` and `-` with `_`.** The id's - alphanumerics are ASCII and include `-`. Every other character, including spaces, - tabs and other whitespace, punctuation and every non-ASCII letter, becomes `_`. -3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 - or were in the display. -4. **Remove every leading and trailing `_` and `-`**, repeating until neither end - holds either. A leading `_` is reserved for virtual ids (clause 1), a trailing `_` - is dropped, and no id begins or ends with `-`. -5. **If the slug starts with a digit, prepend `n`.** An id is never purely numeric - and never starts with a digit, so `2024` becomes `n2024`. -6. **If nothing is left, use `-`**, such as `category-1` or `item-4`. `n` - comes from the shared counter below. A display of only `-`, only `_` or only - non-ASCII letters ends here, because `-` alone is not an id. +Examples: `East` → `east`, `Time Period` → `time_period`, `Coastal_East` → +`coastal_east`, `Q1-2024` → `q1-2024`, `Region(code)` → `regioncode`, `Café` → `caf`, +`Ü` → `category-`, `Straße` → `strasse` (answer 26). `Time Period` and +`Time_Period` slug alike, so whichever comes second gets `time_period-`. `East` in +`Region` and `East` in `Direction` slug alike too, and since ids are global (answer +22), the second is `east-`. Here `` is the counter's next value, for example +`east-1`. -**The syntax of an id** is therefore: a lower-case ASCII letter, then any of `a`–`z`, -`0`–`9`, `-` and single underscores, never ending in `_` or `-`: -`[a-z]([a-z0-9-]|_(?=[a-z0-9-]))*`, not ending in `-`. Slugify's output already has -that shape, so slugifying an id gives the same id back. The collision suffix `-` -(below) keeps the shape too, because it ends in a digit. +The remaining rules: -Examples under the amendment: - -| display | id | -|---|---| -| `East` | `east` | -| `Time Period`, `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` | -| `Coastal_East` | `coastal_east` | -| `a_ b`, `a__b` | `a_b` | -| `Q1-2024` | `q1-2024` | -| `Q1 - 2024` | `q1_-_2024` | -| `East-1` | `east-1` | -| `East `, `East_`, `-East-` | `east` | -| `_Draft` | `draft` | -| `Region(code)` | `region_code` | -| `Café` | `caf` | -| `Crème Brûlée` | `cr_me_br_l_e` | -| `Straße` | `strasse` | -| `2024`, `_2024` | `n2024` | -| `2024-Q1` | `n2024-q1` | -| `Ü`, `-`, `%` | `-` | - -Under the accepted steps, the ones that differ are `East ` → `east_`, `a_ b` → -`a__b`, `Time\tPeriod` → `timeperiod`, `Time \u{200B} Period` → `time__period`, -`-East-` → `-east-`, -`Region(code)` → `regioncode`, `Crème Brûlée` → `crme_brle`, `2024` → `2024`, -`2024-Q1` → `2024-q1` and `-` → `-`. - -**Collisions.** `Time Period` and `Time_Period` slug alike, and whichever comes later -takes a suffix. **[Amendment]** So do `East`, `East ` and `East_`, which slug alike -only under the amendment's steps. `East` in `Region` and `East` in `Direction` also -slug alike, and ids are global, so the second is `east-`. - -**[Amendment] A suffix and a hyphenated slug can look alike.** The display `East-1` -slugs to `east-1`, which is also what the counter's suffix may make of `east`. They -never collide, because every id a document holds is taken, whoever made it: -- if `east-1` is already held (as a slug or as a suffix), a display `East-1` that slugs - to it is a collision, and becomes `east-1-`; -- if the counter's candidate `east-` is already held, the counter advances until a - candidate is free. - -So an id says nothing about how it was made: `east-1` may be a suffixed `east` or the -slug of `East-1`. -- **Any colliding generated id gets a suffix**, whatever kind of thing holds the - slug: on import, when an id-less file loads, and when the host creates something in - a session. An id that does not collide stays bare. -- **The suffix is `-` plus the next value of one increment-only counter per - document**, as a plain number (`east-1`, not `east-001`). The counter works like - Common Lisp's `gensym`: it never goes down and is never reused, so no order holds - among suffixes (`east-7` may follow `east` directly). -- **Retirement.** Within a session a removed id is never handed out again. The - counter is past any suffix it issued, and a removed bare slug stays reserved in - memory. -- **Determinism.** The counter advances in file order or creation order, never in - hash order (gotcha 18), so the same input gives the same ids. -- **The counter** lives in memory, starts at 1 when a document loads, and is not - written to the file. - -**Hand-written ids are canonical.** An id written in a file is never renamed, because -every data key and reference in that file spells it. So: -- **a file has ids on everything or on nothing.** An id-less file (an old version, - or a CSV or JSON import) gets every id from slugify. A new-version file must carry - an id on every declaration, and one that does not is refused (clause 5); -- the suffix only ever goes on the id being generated. When a generated candidate - `slug-` is taken by a file's id, the counter advances until a candidate is free; -- **two identical hand-written ids in one file are refused at load**, even when they - declare different kinds of thing. Renaming one would corrupt the data section, and a - file with a duplicate cannot be resaved safely. The error names both lines. -- **[Amendment] A hand-written id obeys every syntax rule of a generated id** (the - syntax above). So `east-1`, `item-4` and `q1_-_2024` load; `-east`, `east-`, `2024`, - `a__b` and `_draft` are refused at load (clause 5). - -**Reuse across a save and reload is accepted.** Retired ids are not persisted. So -remove `East`, save, reload and add `East`, and the new item gets `east` again. An -in-flight command from before the reload is refused by the generation tier. - -**[Amendment] Two branches of one file can merge into a file that will not load.** If -each branch adds a different thing that generates the same slug, the merged file -declares that id twice. The load is refused and names both declarations, so the merge -is repaired by hand. Stable identity across copies and branches of a file is not -promised (improvise-8tk5). - -**Built code.** `crates/improvise-core/src/model/id.rs` (`slugify`, `IdAllocator`) -implements the accepted steps, and `context/code/core.md` §20 describes them. Once -the amendment is approved, a follow-up must: -- replace every character outside `a`–`z`, `0`–`9` and `-`, not only spaces, with - `_` instead of stripping it, and collapse `_` runs; -- trim a trailing `_` and both ends' `-`, and prepend `n` before a leading digit; -- make `claim` refuse every hand-written id a generated one could not be; -- add the virtual views to `VIRTUAL_IDS`, and `IdKind::Group`. - -Nothing calls `slugify` yet. +- **A file has ids on everything or on nothing** (answer 21). An id-less file (old + version, or a CSV or JSON import) gets every id from slugify. A new-version file must + carry an id on every declaration, and one that does not is refused (clause 5). So no + file mixes hand-written and generated ids. +- **Any colliding id gets a suffix, `gensym`-style** (answers 21, 27 and 28). + - **When:** whenever a generated slug is already taken, whatever kind of thing holds + it. That applies on import, when an id-less file loads, and when the host creates + something in a session. An id that does not collide stays bare. + - **What:** `-` and the next value of **one increment-only counter per document**, + as a plain number (`east-1`, not `east-001`). The counter never goes down and is + never reused, so no order is enforced among suffixes: `east-7` may follow `east` + directly. + - **Retirement:** within a session a removed id is never handed out again. The + counter has already moved past any suffix it issued, and a bare slug that was + removed is still reserved in memory. + - **Determinism:** the counter advances in file order or creation order, so the same + input gives the same ids, never hash order (gotcha 18). + - **Hand-written ids are canonical** (answer 29). The suffix only ever goes on the + id being generated. An id read from a new-version file keeps its spelling, even + when it looks like a slug or a suffixed one (`east-3`), because the file's data + keys and references spell it. So when a generated candidate `slug-` is already + taken by a file's id, the counter advances again until the candidate is free. + - **Two identical hand-written ids in one file are refused at load** (answers 11, + 22 and 29). By answer 22 they are a duplicate even when they declare different + kinds of thing, such as a category and an item. Renaming either one is exactly + what answer 29 forbids, and answer 11 already refuses a duplicate because such a + file "can't safely resave". The error names both lines (clause 5). This follows + from the answers, so it is not asked as a new question. + - **The counter** lives in memory and starts at 1 when a document loads. It is not + written to the file, since retired ids are not persisted (answer 13). That is + this ADR's reading. +- **Across a save and reload, reuse is accepted** (answer 13, Q10(b)). Retired ids are + not persisted. So remove `East`, save, reload and add `East`, and the new item gets + `east` again. An in-flight command from before the reload is refused by the + generation tier. The one remaining risk is an outside reference that outlives the + file, and none exists today. ### 5. Persistence: ids on declarations *and* references -- **Declarations** (section headers, item lines, formula lines, and **[Amendment]** - group definitions) carry the display and the id. -- **Every reference** is the id: axis lines, `hidden:` and `collapsed:`, page - selections, `## Aggregation`, `Initial View:`, `## Data` keys (ids are guaranteed - unique, names might not be), and names inside formula text. +Following answers 3 and 7, the file now carries ids everywhere: +- **Declarations** (section headers, item lines, formula lines) carry the display name + and the id. +- **Every reference** carries the id: axis lines, `hidden:` and `collapsed:`, page + selections, `## Aggregation`, `Initial View:`, `## Data` keys, and names inside formula + text (answer 3). The formula panel shows display names instead (answer 19). -**The id alphabet.** Every id is spelled only in `a`–`z`, `0`–`9`, `_` and `-`. The -grammar spells it **with ranges**, because the generator drops `ASCII_ALPHANUMERIC` -(improvise-rxz9). So an id never needs pipe quoting in a declaration or reference. - -**What the loader refuses** in a new-version file: -- a declaration with no id. A header ending in a parenthesised run with a capital - (`Sales(Copy)`), an empty tag or a tag with a trailing space does not match - `id_tag`, so it has no id. The error names the line; -- a hand-written id beginning with `_`, other than a reserved one; -- **[Amendment]** a hand-written id that breaks any syntax rule of a generated id - (clause 4). A file never declares a virtual category, so no declared id begins with - `_`; a reference may still spell a virtual id such as `_measure`. The grammar's - `id_token` stays permissive, and the loader refuses, so the error can name the line. +**Id alphabet.** Every id, generated or hand-written, is spelled only in `a`–`z`, +`0`–`9`, `_` and `-` (clause 4; answer 20). The grammar spells them **with ranges**, +because the generator drops `ASCII_ALPHANUMERIC` (improvise-rxz9). So an id never needs +pipe quoting in an `.improv` declaration or reference. A header ending in a +parenthesised run that contains a capital, such as `Sales(Copy)`, does not match +`id_tag`, and neither does an empty or trailing-space tag. In a new-version file that +declaration has no id, so the load is refused (answer 21), and the error names the +line. Revision 4 generated an id and warned. A hand-written id beginning with `_`, +other than the three virtual ids, is refused too (answer 23). **Inside formula text an id may still need quoting.** The formula grammar's `bare_ident` must begin with a letter or `_`, and keywords are guarded (gotcha 34). So -the writer spells an id in formula text through `quote::quote_name`. An id equal to a -keyword (`sum`, `if`, `where`, `is`) is written `|sum|`, and under the accepted rules -an id beginning with a digit (`2024`) is written `|2024|`. **[Amendment]** No id begins -with a digit, so only keyword ids need quoting. A `-` inside an id needs none, because -the formula grammar's bare identifier already admits `-` after its first character. +the writer spells an id in formula text through `quote::quote_name`, as it spells a +name today. An id beginning with a digit (`2024`) or equal to a keyword (`sum`, `if`, +`where`, `is`) is written `|2024|` or `|sum|`. No new rule is needed, but "never needs +quoting" does not extend to formula text. ```pest id_token = @{ ('a'..'z' | '0'..'9' | "_" | "-")+ } -// The space before the parenthesis is optional. +// The space is optional (answer 6). id_tag = ${ " "* ~ "(" ~ id_token ~ ")" } -// Declarations: display, then id. The id is optional in the grammar so that the -// loader, not the parser, refuses a missing one and can name the line. +// Declarations: display name, then its id. The id is optional in the grammar so the +// loader, not the parser, refuses a missing one and can name the line (answer 21). header_decl = { pipe_quoted ~ id_tag ~ &NEWLINE | header_display ~ id_tag ~ &NEWLINE | line_name } header_display = @{ (!(id_tag ~ NEWLINE) ~ !NEWLINE ~ ANY)+ } item_decl = { name ~ id_tag? } grouped_item = { "- " ~ name ~ id_tag? ~ "[" ~ name ~ "]" ~ NEWLINE ~ blank_lines } -// References: an id, never a display. +// References: an id, never a display name. id_ref = { id_token } ``` -`model_name` has no id and keeps `header_name`. `Initial View:` is a reference and -takes an `id_ref`. +`model_name` has no id, so it keeps `header_name`. `Initial View:` becomes a reference +and takes an `id_ref`. A group reference (`[Coastal]`) stays a name while groups have no +id (answer 9). -**Group references.** A group reference (`[Coastal]`) is a name. **[Amendment]** It is -the group's id, as is the group in a `collapsed:` line, and the group definition -(`> Coastal`) declares a display and an id like any other declaration -(`> Coastal(coastal)`). - -**[Amendment] An axis line is tried before `format:`, and a format that reads as an -axis line is pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice, and -`format_line` takes the rest of the line. So a category whose id is `format` would -lose its axis line to the view's number format, and the new-version arm puts -`axis_line` first. -- **The collision then moves.** `set-format` stores its argument verbatim, so - `:set-format none` writes `format: none`, which would now read as the axis line of a - category `format`. -- **So the writer pipe-quotes a number format whenever the line would also match - `axis_line`**: an axis keyword (`row`, `column`, `page`, `filter`, `none`) alone or - followed by `, `, or a value beginning with `|`. The new arm's `format_line` is - `pipe_quoted ~ &NEWLINE | rest_of_line`, so `format: |none|` is the format `none`. -- **An old-version file's `format: none`** is read as before (its arm keeps - `format_line` first), and it is written quoted on its first save. -- **Required tests:** - - a category displayed `Format`, on a row axis, round-trips; - - a view whose number format is each axis keyword round-trips in a document where - that view also holds a category `format` on an axis. - -**[Amendment] A formula line** declares the formula: its target, its id, and its -expression over ids. Today's `web/demo.improv` and the format's own dimension example -are written: - -``` -## Formulas -- Profit = Revenue - Cost -- Tax = Revenue * 0.1 [CustomCat] -``` - -Under the amendment the same two formulas are written: - -``` -## Formulas -- Profit(profit) = revenue - cost -- tax(tax-1) = revenue * 0.1 [customcat] -``` - -- **A measure formula** (no category tag) is the normal case. The formula is the - measure (clause 1), so the line is the measure's only declaration: `Profit` is its - display, and `(profit)` is the id it shares with its formula. No `[_measure]` tag is - written, because a file does not spell a virtual category where it is not needed - (clause 1); a line without a tag targets `_measure`. -- **A dimension formula** (a `[]` tag) targets a real item, already - declared with its display in its category's section (`## Category: CustomCat(customcat)` - lists `- Tax(tax)`). So the line names the target by its item id, `tax`, and - `(tax-1)` is the formula's own id. The display is not repeated, because a display - lives in one place. -- **The tag is always an id**, the target category's (`[customcat]`, never - `[CustomCat]`), and `_measure` is implicit: it is never written as a tag. -- **So the loader can always tell the two apart**: with a tag the target is an item - id; without one it is a measure's display. -- **The expression is over ids**, spelled through `quote::quote_name`. A reference to - a formula target is the id text of its `ItemHandle`: the item id outside `_Measure`, - the measure's (and its formula's) id for a `_Measure` target. -- **A `WHERE`/`IS` value follows the same rule**. It is the id text of an - `ItemHandle` in the filtered category, and the formula panel shows it as a display. -- **A display in the target position that looks like `Name(x)` is pipe-quoted**, as in - `- |Margin(%)|(margin) = …`. An id never contains a parenthesis. - -**`header_display` is guarded only by a lookahead**, which the generator skips -(gotcha 34). So the generator may emit a display ending in something that parses as an -id. Round-trip property 3 still holds, because the first parse settles the split, but +**`header_display` is guarded only by a lookahead**, which the generator skips (gotcha +34). So the generator may emit a display ending in something that later parses as an id. +Round-trip property 3 still holds, because the first parse settles the split. But property 1 must be stated over the parsed result, not over the generated text. -**The version line alone picks the regime** (clause 10). -- **In an old-version file** a header is the whole line, a reference resolves against - display names, and ids are assigned by slugify. This resolution migrates a file - whose names were its only identity; it is not a reference by a person. -- **In a new-version file** declarations carry ids and a reference is an id. +**Which regime a file is read under is decided by its version line alone** (clause +10). In an old-version file, a header is the whole line, a reference resolves against +display names, and ids are assigned by slugify. In a new-version file, declarations +carry ids and a reference is an id. -**An old file naming an item it never declares** declares it at load. This happens -today when a `## Data` line names an undeclared item (improvise-98pd). The item gets a -slug like any other, and the first save writes it, so a file that loads today keeps -loading (improvise-80rm). Ids do not settle 98pd's other two sources, `set-cell` and a -stale page selection. A page selection holds an item id, and removing the item leaves -it dangling, as a removed name does today. +**An old file naming an item it never declares** declares it at load (answer 24). This +happens today when a `## Data` line names an undeclared item (improvise-98pd, second +source). The item gets a slug like any other, and the first save writes it, so a file +that loads today keeps loading (the lesson of improvise-80rm). The NonItem work +(improvise-gk51, PR #265) projects such a cell as `CellId::NonItem`. From a loaded file +that case no longer arises. Ids do not settle 98pd's other two sources, `set-cell` and +a stale page selection: a page selection holds an item id, and removing the item leaves +it dangling exactly as a removed name does today. -**Old data keys change spelling on the first save.** Slugify folds case, so an old -file's `Region=East` resolves at load through the display `East` and is written -`region=east`. The data is the same, but a `git diff` across that save touches every -data line. +**Slugify folds case (answer 16(b)), so old data keys change spelling on first save.** +An old file's `Region=East` resolves at load through the display `East`. The first +save writes the id: `Region=East` becomes `region=east`. The data is the same, but a +`git diff` across that first save touches every data line. **Round-trip properties:** 1. an id-less (old-version) file loads, gets ids, and is written in the new version on @@ -476,312 +591,329 @@ data line. 2. a new-version file loads with every id kept verbatim; 3. `format(parse(format(parse(x)))) == format(parse(x))`. -A **formula-line generator strategy** is required. The grammar walker emits no `=`, and +A **formula-line generator strategy** is required. The grammar walker emits no `=` and the walker drops such lines, so without its own strategy every formula property is vacuous (improvise-rxz9). -**A duplicate, unknown or missing id in a new-version file refuses the load.** The -error names the id, and for a duplicate both declarations. A silent repair is ruled -out, because reassigning an id would silently change what its references point at. +**A duplicate, unknown or missing id in a new-version file refuses the load** (answers +11, 21, 22 and 29; a duplicate across two kinds of thing counts, and a hand-written id +is never renamed to resolve one; answer 11: +"don't load the file because you can't safely resave it"). The error names the id, +and for a duplicate it names both declarations. Repairing silently is ruled out, +because references carry ids (answer 3): reassigning a duplicate would silently change +what its references point at. -**Import** (CSV, JSON) never parses ids out of dimension names. A header `Region(code)` -is the display `Region(code)`, and its id is its slug. The writer pipe-quotes any -display that would otherwise read as display plus id: -`## Category: |Region(code)|(regioncode)` (**[Amendment]** `(region_code)`). +**Legacy JSON is removed** (improvise-tt22), so it needs no migration here. -### 6. The wire: address by id, create by display, learn ids from projections +**Import** (CSV, JSON) never parses ids out of dimension names (answer 8). A header +`Region(code)` is the display name `Region(code)`, and its id is its slug. The writer +pipe-quotes any display name that would otherwise read as display plus id: +`## Category: |Region(code)|(regioncode)`. -- **The host assigns every id.** A client never proposes one. +### 6. The wire: address by id, create by display name, learn ids from projections + +- **The host assigns every id** (answer 14). A client never proposes one. - **Every operand naming an existing thing is an id.** The one operand naming the thing - being created is a display, and the host slugifies it: + being created is a display name, and the host slugifies it: - `AddCategory(display)` and `CreateView(display)`; - - `AddItem { category: CategoryId, display }`; - - `AddItemInGroup { category: CategoryId, display, group }`. **[Amendment]** `group` - is the group's id, and a new group is created by its display under the same - create-by-display rule; - - `AddFormula { target_category: CategoryId, raw }`. -- **[Amendment] Only two readers turn display names in input text into ids: CSV - import and the `.improv` loaders.** They read text people wrote, where names are - all there is. The wire never carries a display as a reference. -- **[Amendment] Command-language text names things by id.** Anything that uses the - command language as a scripting language is a script, wherever the text comes from - (a script file, `improvise cmd`, another program). A script that names something by - its display fails. A script gets a display only through a dedicated function such as - `display_for(id)`. -- **[Amendment] Interactive input matches either a display or an id**, in the TUI - and the browser, as long as it is unambiguous which thing the user is picking. Ids - win when it is not, and the display is shown as a preview (clause 12). + - `AddItem { category: CategoryId, display }`, + `AddItemInGroup { category: CategoryId, display, group }` and + `AddFormula { target_category: CategoryId, raw }`. - **A frame cannot address a thing it creates**, because it does not know the id the - host will assign. A create-then-act compound becomes **one command**: - `create-and-switch-view` sends `[CreateView, SwitchView]` today and becomes - `CreateView { display, switch: true }`. Any other compound of that shape gets the - same treatment. -- **New operations carry ids:** + host will assign. So a create-then-act compound becomes **one command** carrying + both halves. `create-and-switch-view` sends `[CreateView, SwitchView]` today and + becomes `CreateView { display, switch: true }`. Any other compound of this shape + gets the same treatment. +- **New operations are born with ids:** - `RenameCategory { category, to: display }`; - `RenameItem { category, item, to: display }`; - - **[Amendment]** a group rename by group id and new display; - - `ReplaceFormula { id, was, raw }`; - - `RenameRecordCoordinate`; + - `ReplaceFormula { id, was, raw }`, which km71 already landed with an interim id + (below); + - fdk1's `RenameRecordCoordinate`, which landed with name operands (below); - the removes (`RemoveCategory`, `RemoveItem`, `RemoveFormula`, `DeleteView`). -- **Older variants move to ids too** (`SetAxis`, `SetPageSelection`, `HideItem`, …, - including `SetCell { view, … }` and `ClearCell { view, … }`), one whole variant at a - time. +- **Older variants** (`SetAxis`, `SetPageSelection`, `HideItem`, …) **must also move to + ids**. Their operands are references, and a reference is now an id. They may move one + whole variant at a time. That includes `SetCell { view, … }` and + `ClearCell { view, … }`, which improvise-wpxr (PR #281, merged) gave a view named by + its name, like every other view-scoped variant (improvise-lltt). - **Projections carry id and display together:** - `CategoryRow { id, display, items: Vec<(ItemHandle, String)>, axis }`; - - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows: - the target's display with its id in parentheses (`Profit (profit) = Revenue - - Cost`), then the expression over display names, with an id wherever a display is - ambiguous. **[Amendment]** A display equal to another thing's id counts as - ambiguous, so the panel never shows a bare name that the ids-first rule (clause 12) - would bind elsewhere. `stored` is the id-form text the file carries, and - `ReplaceFormula`'s `was` is compared against it. So a rename of `Revenue` elsewhere - changes what the panel shows, but does not refuse an edit of `Profit`; - - `ViewSummary { id, display, … }`; - - **[Amendment]** the grid frame's group entries (`FrameGroup`) carry the group's id - and display. + - `FormulaCache` rows `{ id, stored, display }`. `display` is what the panel shows + (answer 19): the target's display name with its id in parentheses, then the + expression over display names, with an id wherever a display name is ambiguous + (ids are unique, answer 22). `stored` is the id-form text the file carries, and it is what + `ReplaceFormula`'s `was` is compared against. So a rename of `Revenue` elsewhere + changes what the panel shows but does not refuse an edit of `Profit`, which is + the "avoid glitches" of answer 3. What makes that hold for an edit already open + is clause 12; + - `ViewSummary { id, display, … }`. `AxisFrame` labels stay display strings. They gain ids when a consumer needs them. -- **`ViewState` keys by id**: the cursors, `expanded_cats` and the view history - (`view_back_stack` / `view_forward_stack`). **[Amendment]** So do `active_view` - itself and `Workbook::active_view` (improvise-p003). A rename then orphans nothing. -- **[Amendment] A `_Measure` formula-target row** (`ItemHandle::FormulaTarget`) is - removed by `RemoveFormula` and renamed by a display rename of the measure - (improvise-i0n7). -- **[Amendment] CheckedBatch's created-record exemption is removed** (improvise-p003). - It exempts a record that neither resolves nor is claimed, for frames like - `[AddItem(East), SetCell(East)]`. A frame cannot address what it creates, so nothing - produces such a frame any more. +- **`ViewState` keys by id**: the cursors, `expanded_cats`, and the view history + (`view_back_stack` / `view_forward_stack`). Answer 1 makes this natural rather than a + migration of its own. A view or category rename then orphans nothing. ### 7. ADR 001: an id that no longer resolves is gone -A command naming an id that does not resolve is rejected before anything is applied, -as `Rejection::Gone(Named)`: -- `Named` is `Category`, `Item`, `Formula` or `View`, and carries the id's text. - **[Amendment]** It gains `Group`. -- The rejection is **structural**: force never creates structure. +A command naming an id that does not resolve is rejected before anything is applied, as +`Rejection::Gone(Named)`: +- `Named` is one of `Category`, `Item`, `Formula` or `View`, and carries the id's text. +- It is **structural**: force never creates structure. The check has two halves, like the record tier: - **Before the frame:** an exhaustive `ids_named` feeds `Checked::against`. - **Within the frame:** an **id arm in `broken_chain`** refuses `[RemoveFormula(f), ReplaceFormula(f)]` (improvise-gkyo). -Per-entity versions are not added. +Per-entity versions are not added (answer 10). -### 8. ADR 002: `RecordHash` is defined over ids +### 8. ADR 002: `RecordHash` is now defined over ids -`RecordHash` hashes the explicit serialization of the sorted `(CategoryId, ItemId)` -text pairs: the same function over different strings. **[Amendment]** The pairs are -`(CategoryId, ItemHandle)` spelled as id text. Ids are unique across kinds, so the hash -stays canonical. A record is the key minus `_measure`, so a `FormulaTarget` coordinate -is never part of a record. +Following answer 3, `RecordHash` hashes the explicit serialization of the sorted +`(CategoryId, ItemId)` text pairs, the same function over different strings. +- **ADR 002's decision is unchanged in substance.** It still hashes an explicit + serialization of a normalized record, with a fixed key and a width argued for. +- **Its prose will be out of date once this lands.** ADR 002 says `CellKey::new` + "sorts coordinates by category name", and a record is spelled in names. Both become + ids. ADR 002 is **not edited here**. Updating that wording belongs to the + implementation PR. -ADR 002's decision is unchanged in substance. Its prose ("sorts coordinates by -category name", records spelled in names) is out of date once this lands, and the -implementation PR updates it. +**What changes in behaviour:** a display rename no longer changes any `RecordHash` or +bumps `layout_version`, because `view_shape` hashes ids too. So a client's surrogate +survives a rename, and its in-flight edit still lands. A **coordinate** change, as in +fdk1 or a records-mode move, still produces a different record and a different hash, as +it should: it *is* a different record. -**In behaviour:** a display rename changes no `RecordHash` and bumps no -`layout_version`, because `view_shape` hashes ids too. So a client's surrogate survives -a rename, and its in-flight edit lands. A **coordinate** change, as in fdk1 or a -records-mode move, produces a different record and a different hash, because it is a -different record. +### 9. Headers that change meaning: the edge case answer 6 asks to document -### 9. Headers that change meaning +Under answer 6, `header_decl` reads a trailing `(token)`, with or without a space before +it, as the id. **It applies only to new-version files** (answer 15, Q12(a)). An +old-version file is read as it always was, so its headers never split. In a new-version +file the writer never produces an ambiguous header, because it pipe-quotes any display +that would split. So the edge cases below arise **only when a new-version file is +written by hand**. The table shows what the header rule would do to lines that exist +in old files today. That is why old files do not get the rule. -In a new-version file, `header_decl` reads a trailing `(token)`, with or without a -space before it, as the id. **Old-version files never get this rule**, so their headers -never split. The writer pipe-quotes any display that would split, so the cases below -arise only in a new-version file written by hand. They show why old files keep the -whole-line reading. - -| header line | whole-line reading (old version) | under `header_decl` (new version) | +| header line | whole-line reading (today) | under `header_decl` | |---|---|---| | `## View: Default(default)` | `Default(default)` | display `Default`, id `default` | | `## View: \|Default (default)\|(default)` | pipes literal | display `Default (default)`, id `default` | | `## View: Sales (copy)` | `Sales (copy)` | **display `Sales`, id `copy`** | | `## Category: Region(code)` | `Region(code)` | **display `Region`, id `code`** | -| `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** (**[Amendment]** refused: an id may not start with a digit) | +| `## Category: Budget (2024)` | `Budget (2024)` | **display `Budget`, id `2024`** | | `## Category: Time Period` | `Time Period` | unchanged; id `time_period` generated | -The app itself writes such headers: `improvise import` on a CSV with headers like -`Region(code)` saves `## Category: Region(code)` today (improvise-80rm). +**The documents that change meaning** are those with a view or category header ending in +a parenthesised run of letters, digits, `_` or `-`. **The app writes such headers +itself**: `improvise import` on a CSV with headers like `Region(code)` or `Size(cm)` +saves `## Category: Region(code)` today (improvise-80rm). Any view someone named +`Sales (copy)` does the same. -If old files got the rule, that category would load as display `Region`, id `code`. -Every reference in the file spells `Region(code)`, which resolves to neither, so the -axis lines and data keys would not resolve, and nothing would be drawn. Because old -files keep the whole-line reading, none of this happens. On the first save the writer -pipe-quotes such a display: `## Category: |Region(code)|(regioncode)` -(**[Amendment]** `(region_code)`). +**What it would cost if old files got the new rule** (the case answer 15 rules out): +- The category is renamed on load: display `Region`, id `code`. +- Every reference in that old file spells `Region(code)`, which resolves to neither the + display nor the id. +- So the axis lines and every data key for that category do not resolve. +- Today's loader would silently drop the axis lines and orphan the data, so nothing is + drawn. -### 10. The file version alone picks the regime +**Decided (answer 15):** old-version files keep the whole-line reading, so none of +this happens to a file that exists today. On the first save the writer pipe-quotes +such a display, for example `## Category: |Region(code)|(regioncode)`, so the new file +cannot split it either. +### 10. The file version is explicit, and it alone picks the regime + +Following answers 15 and 25: - **The version is already explicit.** The first line of every `.improv` file is a mandatory version line, matched as an **exact literal** - (`version_line = { "v2025-04-09" ~ NEWLINE ~ blank_lines }`), and `format_md` writes - it. -- **The new format gets a second literal.** The grammar's `file` rule gets one arm per - literal, each with its own header and reference rules. The literal's spelling is the - implementation's choice. -- **The regime is chosen only by the first line**, never inferred from the body. The - old literal means no ids anywhere. The new literal means ids on every declaration. -- **A file whose first line is neither literal is refused**, as today. -- **Every save writes the new literal**, so an old file changes version on its first - save. + (`version_line = { "v2025-04-09" ~ NEWLINE ~ blank_lines }` in `improv.pest`), and + `format_md` writes it. Nothing is added to make the version explicit. What is added is + a second literal. +- **The new format gets a new literal.** The grammar's `file` rule gets one arm per + literal, and each arm uses its own header and reference rules. The new literal's + exact spelling is left to the implementation. +- **The regime is chosen only by the first line**, never inferred from what the body + looks like. So the old literal means no ids anywhere, and the new literal means ids + on every declaration (answer 21). +- **A file whose first line is neither literal is refused**, as it is today. +- **Every save writes the new literal** (answer 12). So an old file changes version on + its first save. ### 11. Migration 1. **Remove legacy JSON** (improvise-tt22). Done, in PR #224. -2. **Types:** the id newtypes over `Symbol`, the `display` fields, `FormulaId` and - `ViewId`. **[Amendment]** Also `GroupId`, a display on `Group`, and `ItemHandle`, - which the re-key needs for `CellKey`. +2. **Types:** the id newtypes over `Symbol`, the `display` fields, and `FormulaId` and + `ViewId`. 3. **Re-key storage by id** (clause 2). This is the wide, mechanical step, with - display→id resolution moved to every input edge. **[Amendment]** The virtual - categories' keys move from `_Index`, `_Dim` and `_Measure` to `_index`, `_dim` and - `_measure` here, in the model, on the wire, in `ViewState` and in the browser client. -4. **Persistence:** the new literal (clause 10), clause 5's grammar, the loader and its - refusals, slugify, the writer, and the round-trip properties with a formula-line - strategy. + display→id resolution moved to every input edge. +4. **Persistence:** the new version literal (clause 10), the grammar in clause 5, the + loader and its refusals, slugify, the writer, and the round-trip properties with a + formula-line strategy. 5. **Projections and `ViewState`** (clause 6). 6. **Vocabulary:** the id-addressed variants, `Rejection::Gone`, `ids_named` and `broken_chain`'s id arm. -7. **[Amendment] Clause 12's client-side capture and resolver.** No display-matching - code exists yet, so the resolver is built ids first from the start. - -The implementation epic (improvise-cq1q) orders these steps to keep `main` shippable -after each one: the display field before the re-key, and projection ids before the -re-key. ### 12. A formula edit's names are fixed when the editor opens -**The problem.** The formula panel shows display names, and the user edits that text. -If those names were mapped to ids at **commit**, a concurrent rename would break the -edit one of two ways: -- the typed name no longer resolves, and the edit is refused; +**This clause is the ADR's own reading, not a user answer.** It closes a contradiction +the adversarial review of `4830fa30` found between clause 6 and the km71 section, which +is also ChatGPT Codex's P1 on `29ad14f4`. + +**The problem.** The formula panel shows display names (answer 19), and the user edits +that text. If the host maps those names to ids at **commit**, a concurrent rename breaks +the edit one of two ways: +- the name the user typed no longer resolves, so the edit is refused, which contradicts + clause 6's "does not refuse an edit of `Profit`"; - or the name now belongs to something else (`Revenue` renamed away, another item - renamed to `Revenue`), and the edit silently binds to a different thing. + renamed to `Revenue`), so the edit silently binds to a different thing. That is the + glitch answer 3 exists to prevent. -**The rule.** The display→id mapping is **captured when the formula editor opens**, and -the commit resolves the edited text through it. This is the cell editor's -capture-at-open rule: `OpenedCell` (`improvise-protocol/src/client.rs`) records the -record, its version, the document generation and the view at open, and the commit -answers for those. -- The mapping covers every display the panel showed in that formula's text, including - the target's. -- `was` is compared against the stored id-form text (clause 6), so a concurrent - *replace* of the formula is still seen. +**The rule.** The display-name-to-id mapping is **captured when the formula editor +opens**, and the commit resolves the edited text through that captured mapping. This +is the capture-at-open rule the cell editor already follows: `OpenedCell` in +`improvise-protocol/src/client.rs` records the record, its version, the document +generation and the view at open, and the commit answers for those rather than for +whatever the document holds by then. +- The mapping covers every display name the panel showed in that formula's text, + including the target's. +- A name the user types that was **not** in the captured mapping is new to the edit. It + is resolved against the document at commit, as any typed name is. If it does not + resolve, the edit is refused and says which name. - A captured id that no longer exists at commit is `Rejection::Gone` (clause 7). -- **A typed name that was not in the captured mapping** is resolved against the - document at commit, and refused, naming it, if it does not resolve. +- `was` is still compared against the stored id-form text (clause 6), so a concurrent + *replace* of the formula is still seen. -**[Amendment] The client does the resolving. Interactive input may match a display or -an id as long as the pick is unambiguous, and ids win when it is not:** -- **The client holds the mapping**, as it holds `OpenedCell`. The TUI and the browser - share one resolver in `improvise-protocol`. -- **A name new to the edit** is resolved by the client against its id-bearing - projection (clause 6), **ids first, then displays**. Ids are unambiguous, so they are - preferred wherever there is ambiguity, now and once an autocomplete exists. - - A typed name equal to an id names that id's thing, even where it is also another - thing's display. - - Otherwise a display that exactly one thing in scope has names that thing, and a - display matching several things is refused with a list of their ids. - - A name that resolves to nothing is refused, and the refusal names it. - - Ids are matched exactly, with no case folding. -- **Typing prefers ids and previews displays.** A view-layer autocomplete turns typed - text into ids before commit, preferring a match on an id, and shows the matched - thing's display as a preview. The model layer refers to ids only. This ADR does not - specify the autocomplete further. -- **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are - in the file's id form, and the host never resolves a display. The host parses ids, - and one that no longer exists is `Rejection::Gone`. +**Required regression test.** Open a formula editor on `Profit = Revenue - Cost`. From +another client, rename the item `Revenue` to `Sales` and rename a different item to +`Revenue`. Commit the unedited text. The formula must still reference the original +item's id, and the commit must not be refused. The test must fail under +commit-time resolution, which binds to the new `Revenue`, and it must fail under a +refusal on rename. Both are the rivals this clause rules out. -**[Amendment] What ids-first costs.** Suppose item A is displayed `East` with id `east`, -and item B is displayed `east`, so slugify gave it `east-1`. Typing `east` names **A**. -B is reached by typing `east-1`. A user who typed `east` meaning B is bound to A -without being told, unless the preview shows it. That is why the panel shows B as -`east-1` and never as a bare `east`. +## What this means for you -**Required regression test.** -1. Open a formula editor on `Profit = Revenue - Cost`. -2. From another client, rename `Revenue` to `Sales`, and rename a different item to - `Revenue`. -3. Commit the unedited text. +### improvise-km71 — editing a formula (landed as an interim) -The stored formula must reference the original item's id, and the commit must not be -refused. The test must fail under commit-time resolution and under a refusal on -rename. +km71 landed in PR #214, before this ADR, and was built to be revised by it +(`context/code/core.md` §18): +- **What landed.** `ModelCommand::ReplaceFormula { id: FormulaId, was, raw }`. + `FormulaId` is opaque, with private fields `(target, target_category)`. + `Model::replace_formula` writes in place, keeping the position. It refuses a new + target that another formula owns, and it refuses any target change outside + `_Measure` (improvise-kb1d). `was` is the text the editor opened on. It makes a + concurrent *replace* visible, which even a stable id cannot do, so **`was` stays**. +- **What this ADR changes.** + - `FormulaId`'s fields become one id. The signatures that carry `FormulaId` do not + change, which is what making it opaque was for. + - A target change becomes a **display** rename of the target. The id does not move, + and nothing that references the target is rewritten. So kb1d's refusal outside + `_Measure` can be lifted, and so can the dangling-reference report core.md §18 + describes. + - The panel shows `Profit (profit) = Revenue - Cost` (answer 19). The user edits that + text. Its display names are resolved to ids through the mapping captured when the + editor opened (clause 12), not through the names the document holds at commit, and + `was` is compared against the stored id-form text (clause 6). The parenthesised id is not the user's to + change, since the host assigns ids (answer 14). An edit that changes it is refused. + This is the ADR's reading, not a user answer. +- **Rejections.** Another client's deletion gives `Rejection::Gone(Formula(id))`. A + deletion earlier in the same frame gives the broken-chain refusal. -**[Amendment]** Step 3 becomes: edit something that is not a name -(`Revenue - Cost * 1`) and commit. Committing the unedited text cannot tell a correct -capture from no capture at all, because a client may treat unchanged text as a no-op -without resolving any name (improvise-hexb). +### improvise-fz77 — renaming a category or item -**[Amendment] Required test for ids-first.** With items A (`East`, `east`) and B -(`east`, `east-1`), type `east` into an editor, in text not captured at open: -- it must bind A, and the test must fail under displays-first; -- `east-1` must bind B; -- a display that is nobody's id must still resolve by display. +- **The operations.** `RenameCategory { category, to }` and + `RenameItem { category, item, to }`. The panel rows carry the ids. +- **It is now cheap** (answers 1 and 2): one `display` write. No `CellKey`, axis map, + page selection, hidden or collapsed set, or formula text is rewritten, because all of + them hold ids. No `RecordHash` or `layout_version` moves, so in-flight edits survive. +- **What it still owes:** + - **Uniqueness.** Refuse a display name already used in the same scope. + - **Dirtiness.** Mark the effect so the grid labels and panels are re-projected + (gotcha 35). The labels are display strings. + - **Tests.** Assert that the *labels* changed and the *surrogates did not*. That is + the property this design exists for. +- **It depends on step 3 of the migration**, the re-keying. Until then there is no cheap + rename to build on. -A fixture where no display equals an id proves nothing. +### improvise-fdk1 — renaming a records coordinate -## What this means for the tickets +fdk1 landed in PR #221, before this ADR. What exists on main: +- `ModelCommand::RenameRecordCoordinate { record: RecordHash, category: String, item: + String }`, which moves a whole record to a new coordinate and creates the item if need + be; +- `CellId::Coordinate { record: RecordHash, category: String }`, the surrogate a + records-mode coordinate cell carries. -**improvise-km71 (editing a formula), landed as an interim.** -`ReplaceFormula { id: FormulaId, was, raw }` writes in place. It refuses a new target -that another formula owns, and any target change outside `_Measure` (improvise-kb1d). -`was` makes a concurrent *replace* visible, which even a stable id cannot, so `was` -stays. +**What this ADR changes**, as it does for km71's `FormulaId`: +- `category` becomes a `CategoryId`, in the command and in `CellId::Coordinate`. +- `item` becomes an `ItemRef`: `Existing(ItemId)` or `New(display)`. That is the + create-by-display half of clause 6, since the host assigns the new item's id. +- Unlike a display rename, this **changes a coordinate**, so the record's `CellKey`s and + `RecordHash` change. It is a different record, and that does not change. -This ADR changes three things: -- **`FormulaId`'s fields become one id.** The signatures do not change. -- **A target change becomes a display rename of the target.** The id does not move, - so kb1d's refusal and core.md §18's dangling-reference report can be lifted. -- **The panel's text is resolved through clause 12's captured mapping.** The - parenthesised id is not the user's to change, so an edit that changes it is - refused. +## Questions as asked, and where each now stands -Rejections: -- Another client's deletion gives `Rejection::Gone(Formula(id))`. -- A deletion earlier in the same frame gives the broken-chain refusal. +The first draft asked Q1–Q9. Revision 1 added Q10 and Q11, revision 2 added Q12–Q14, +revision 3 added Q15–Q17, revision 4 added Q18–Q21 and revision 5 added Q22. **All +are answered.** -**improvise-fz77 (renaming a category or item).** It adds `RenameCategory` and -`RenameItem`. A rename is one `display` write, and no key, set, formula text, -`RecordHash` or `layout_version` moves. It still owes three things: -- refuse a display already used in the scope; -- mark the effect so labels and panels re-project (gotcha 35); -- tests asserting that the *labels* changed and the *surrogates did not*. +| # | question | my recommendation | status | +|---|---|---|---| +| 1 | Integer or word ids? | first draft: words; revision 1: integers only | **answered**: interned symbols (answer 4), generated by slugify (answers 8, 16). Words. | +| 2 | Formula id placement | F1, `- Profit(3) = …` | **answered: F1** (answer 5) | +| 3 | Headers in old files | first draft: accept the rare break; revision 1: gate on version | **answered**: the space is optional (answer 6), and old files keep the whole-line reading (answer 15, Q12) | +| 4 | Do `## Data` keys carry ids? | no | **answered: yes, ids** (answer 7), reversing the recommendation | +| 5 | Group ids? | not now | **answered: not now** (answer 9) | +| 6 | Per-entity versions? | not in this ADR | **answered: not in this ADR** (answer 10) | +| 7 | `RecordHash` over ids? | no, keep names | **answered: over ids** (answer 3), reversing the recommendation | +| 8 | Duplicate id: repair or refuse? | revision 2: refuse | **answered: refuse the load** (answer 11) | +| 9 | Reading of "opt-in" | read optional, write always | **answered: yes** (answer 12) | +| 10 | Id reuse after a reload | revision 2: (b) accept reuse | **answered: (b)** (answer 13) | +| 11 | Who assigns ids? | the host | **answered: the host** (answer 14) | +| 12 | Old files and the new header rule | (a) no | **answered: (a), with the version marked unambiguously** (answer 15) | +| 13 | Slugify details | all generation; keep case; keep `_` `-`; strip non-ASCII; `-` | **answered** (answer 16): all generation, **lower case only** (reversing the recommendation), keep both, strip, `-` | +| 14 | Formula text: ids or display names? | ids in the file | **answered**: ids in the file (answer 3); the panel is Q16 | +| 15 | Slugify order | fold, then strip | **answered: casefold, space → `_`, strip** (answer 18); the space character changed from `-` | +| 16 | What the formula panel shows | none | **answered: display names, target shown as `Profit (profit)`, ids where a name is ambiguous** (answer 19) | +| 17 | Capitals in a hand-written id | (a) no | **answered: no** (answer 20) | +| 18(a) | Does a run of spaces collapse? | collapse | **answered: collapse**, plus a numeric suffix on a duplicate, and ids on everything or nothing (answer 21) | +| 18(b) | What "casefold" means | full case folding | **answered: full Unicode case folding** (answer 26) | +| 19 | Does an id disambiguate formula text? | (a) item ids unique in the document | **answered: ids are globally unique** (answer 22), stronger than (a) | +| 20 | Virtual category ids | (a) lower-case ids, capitalised displays | **answered: (a)**, and no other id may begin with `_` (answer 23) | +| 21 | An old file naming an undeclared item | (a) declare it at load | **answered: (a)** (answer 24) | +| — | "A new file version" (a Consequences line) | — | **answered**: the version must be explicit (answer 25); it already is, and clause 10 adds a second literal | +| 22 | The suffix's spelling, and interactive collisions | (a) plain number; (b)(i) suffix there too | **answered**: a plain number (answer 27); any collision is suffixed, across kinds, from a `gensym`-style increment-only counter (answer 28), and only a generated id is ever suffixed, since a hand-written one is canonical (answer 29) | -It depends on migration step 3. +## Still open -**improvise-fdk1 (renaming a records coordinate).** -`RenameRecordCoordinate { record, category: String, item: String }` and -`CellId::Coordinate { record, category: String }` exist on main. This ADR changes -three things: -- `category` becomes a `CategoryId`. -- `item` becomes `ItemRef::{Existing(ItemId), New(display)}`. -- The rename **changes a coordinate**, so the record's `CellKey`s and `RecordHash` - change. It is a different record. +Nothing is open among Q1–Q22, and answer 29 settles the one case it raises: two +identical hand-written ids are refused at load. **The user approved revision 7. +Revision 8 adds clause 12, which is the ADR's own reading, and the user approved it +("looks good, approved", 2026-09-25 00:17:10Z).** Three follow-ups from the same +review stay as tickets, not ADR text: improvise-le4p, improvise-m56i and +improvise-p003. ## Consequences **Good.** - Edit, rename and replace are distinct, addressed operations. -- A rename is one display write, and in-flight edits survive it. +- A rename is one display write: it moves no key, surrogate or basis tier, and in-flight + edits survive it. - `## Data` keys are unique by construction. -- km71's `FormulaId` becomes a real id by changing one struct. +- km71's `FormulaId` becomes a real id by changing one struct, and kb1d's refusal can be + lifted. **Bad.** -- Re-keying storage is wide. Every name-taking API moves to ids. -- Old files change version, and their data keys change case, on the first save. +- Re-keying storage is wide: every name-taking API moves to ids, and display→id + resolution moves to the input edges. +- A second version literal (clause 10), and every old file changes version on its first + save. - `Rejection` loses `Copy`. -- `Formula::raw` is no longer verbatim. The file carries formula text over ids. -- The formula panel is a second rendering of formula text, so display names typed - there must be resolved back to ids (by the client, under the amendment). -- ADR 002's prose needs a wording update. -- **[Amendment]** Scripts that name things by display break and must be rewritten - over ids. +- `Formula::raw` is no longer verbatim: the file carries formula text over ids. +- The formula panel is a second rendering of formula text, over display names, so the + host must resolve display names typed there back to ids (answer 19). +- Ids are lower case, so an old file's data keys change spelling on its first save. +- ADR 002's prose needs a wording update when this lands. -**Not addressed.** Per-entity versions, ids in `AxisFrame` labels and -`CellId::Derived`, and (under the accepted text) group ids. - -## Open questions - -None. +**Not addressed.** Per-entity versions (Q6), group ids (Q5), and ids in `AxisFrame` and +`CellId::Derived`. diff --git a/context/code/core.md b/context/code/core.md index 533b02b5..4655a987 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1714,7 +1714,8 @@ underscore-collapsing step exists. A trailing run survives, so `East ` slugs to `east_`. Only *leading* underscores are removed (step 4). **ADR 007's proposed amendment changes these rules, and this code predates it.** -The amendment (PR #326) is not yet approved. Its clause 4: +The amendment (`context/adr/007-stable-identities-amendment-1.md`, PR #326) is not +yet approved. Its clause 4: - every character outside `a`–`z`, `0`–`9` and `-` becomes `_` instead of being stripped (not only U+0020), so `Region(code)` slugs to `region_code`, while `Q1-2024` keeps its `-`; @@ -1727,7 +1728,7 @@ The amendment (PR #326) is not yet approved. Its clause 4: This section, `slugify`, and the tests `slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and `slugify_keeps_an_underscore_the_display_already_had` describe the earlier rule -above. They change in a follow-up once the amendment is approved. The ADR's +above. They change in a follow-up once the amendment is approved. The amendment's clause 4 lists every difference. **The allocator records every id it admits and releases none.** That is the diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index 3ae5941d..55c91ba1 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -47,6 +47,7 @@ const SECTIONS = [ ["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"], ["context/adr/005-exact-decimal-arithmetic.md", "adr/005-exact-decimal-arithmetic", "ADR 005 — Exact decimal arithmetic (proposed)"], ["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"], + ["context/adr/007-stable-identities-amendment-1.md", "adr/007-stable-identities-amendment-1", "ADR 007, amendment 1 (proposed)"], ], }, { From 5923f039f7a1b02b0493a225cbada33aab02b51a Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 09:45:46 -0700 Subject: [PATCH 076/106] docs(adr): ADR 009 makes the client pruning recipe conditional and restores the dropped-connection test (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 02310a7b..cbb35871 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -190,11 +190,11 @@ and dimensions. The window is never state, and a request costs O(visible). displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model answers one restricted successor query: "the next k rows after position p with a value in a column of S", and the same for columns. A per-column sorted index of occupied rows -answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its -window, fills forward with that query, and finds a new top-left when scrolling up or left -by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether -pruning runs on the client and which parameters travel are ADR 008's questions. This -design supports either answer. +answers it in O(|S| log n + k|S|), holding O(occupied cells). Whether pruning runs on +the client and which parameters travel are ADR 008's questions, and this design supports +either answer. If the client prunes, it prunes its window, fills forward with that +query, and, when scrolling up or left, finds a new top-left by galloping backwards in +doubling jumps (about 14 to 28 probes at density 10⁻⁴). **The change notice.** After each accepted batch the host emits one notice to every client, addressed to none. It carries: @@ -283,7 +283,8 @@ Step 8's required tests: - `[AddItem, SetCell]` publishes one notice, and no client refetches; - a dropped notice is detected; - a reconnect after a host restart refetches; -- a refused frame consumes no publication number. +- a refused frame consumes no publication number; +- a dropped connection forces a refetch. ## Open questions From 1df5f99e356ea5abcd24a9960860863d4087d8e5 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:50:15 -0700 Subject: [PATCH 077/106] docs(adr): equal displays in a scope are shown with a gapless " #2" suffix in id order (improvise-cq1q.1) Replaces the "one permitted exception" to display uniqueness for a formula beside a stored measure of the same name: Profit and Profit #2. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- .../adr/007-stable-identities-amendment-1.md | 30 +++++++++++++++---- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index c4bf95a1..5930f4c4 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -17,6 +17,7 @@ behaviour, the accepted rule binds until the amendment is approved: | 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill` | | 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays, never declared, and referred to by id (`_measure`) | | 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | +| 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit | @@ -77,10 +78,8 @@ cost`); a dimension target is rare, for a derived dimension. `Profit`), adding `Profit = Revenue - Cost` creates a second measure with a fresh id, `profit-`, by the ordinary collision rule. The stored measure keeps `profit` and its data; removing the formula removes only `profit-`. - - **Conflict with display uniqueness:** both are displayed `Profit` in `_measure`'s - scope. This is the one permitted case; the two are told apart by id (the formula - panel shows ids where a display is ambiguous, and a typed `Profit` matching both is - refused with both ids listed, clause 12). + - Both are named `Profit` in `_measure`'s scope, so they are shown as `Profit` and + `Profit #2` (clause 2's suffix rule). - **A target outside `_Measure` is a real item** with its own `ItemId`, and its formula has a separate `FormulaId`: a formula `Tax` in `CustomCat` gets item id `tax`, then formula id `tax-`. @@ -96,6 +95,27 @@ cost`); a dimension target is rare, for a derived dimension. hold. - **A collapsed entry names its group by the group's id**, and `Group` gains a `display`. +- **Displays stay unique within a scope by suffixing.** The accepted text refuses a + display already in use in the scope; a formula's measure beside a stored measure of + the same name (clause 1) cannot be refused. So when several things in one scope have + the same display, they are shown with a suffix: the first keeps the bare display, and + the rest get ` #2`, ` #3`, … (a space, then `#` and the number) as a gapless + sequence in the order their ids sort. + - The order is the ids' string order, the order `CellKey` already sorts ids in. So + `profit`, `profit-1` and `profit-2` show as `Profit`, `Profit #2` and `Profit #3`, + and `profit-10` sorts before `profit-2`. + - Gapless: if the thing shown as `Profit #2` is removed, `Profit #3` becomes + `Profit #2`. + - The suffix is computed when a display is shown, from the things that share it, and + is never stored or written to a file. + - A suffixed display is a display like any other: interactive input that types + `Profit #2` picks that thing (clause 6), and the ids stay the unambiguous name. + - Old files cannot hold two things with one name in a scope, so loading one adds no + suffix. + - A display can also be literally `Profit #2`. Beside two things displayed `Profit`, + it shows the same text as the suffixed one; the two stay distinct by id, and typing + `Profit #2` then matches both, so the pick is ambiguous and is refused with both ids + listed (clause 12). ### 4. How an id is made @@ -271,7 +291,7 @@ therefore shows B as `east-1`, never as a bare `east`. ## Consequences - Scripts that name things by display break and must be rewritten over ids. -- Two measures can share a display; every place that shows displays falls back to ids. +- Two measures can share a display, and are shown as `Profit` and `Profit #2`. - Old files' `_Measure` declarations and hyphen- or digit-led names change on first save. - `id.rs` and its tests need the follow-up in clause 4. From 46c8a967bc6fee5a8bba0649757cb77e5afcfecd Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 09:52:14 -0700 Subject: [PATCH 078/106] docs(adr): a display suffix skips a number a literal display already holds (improvise-cq1q.1) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities-amendment-1.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index 5930f4c4..dc8ff825 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -112,10 +112,11 @@ cost`); a dimension target is rare, for a derived dimension. `Profit #2` picks that thing (clause 6), and the ids stay the unambiguous name. - Old files cannot hold two things with one name in a scope, so loading one adds no suffix. - - A display can also be literally `Profit #2`. Beside two things displayed `Profit`, - it shows the same text as the suffixed one; the two stay distinct by id, and typing - `Profit #2` then matches both, so the pick is ambiguous and is refused with both ids - listed (clause 12). + - A suffix never repeats a display already in the scope. If something is literally + displayed `Profit #2`, the shared `Profit`s take the next free numbers instead: + `Profit`, `Profit #3`, `Profit #4`, still in id order. "Gapless" counts only the + suffixes actually assigned, so a number taken by a literal display is skipped, + like an id collision candidate that is already held. ### 4. How an id is made From c10eca98e0f4d797f8207a1ab0355c5028e4065c Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 10:04:28 -0700 Subject: [PATCH 079/106] docs(adr): keep the _Measure section for stored measures, and mark ADR 007's amendment accepted (improvise-cq1q.1) The Opus review of 46c8a967: a new-version file had nowhere to declare its stored measures. The ## Category: _Measure section stays and lists them with displays and ids; its header declares nothing. Clause 6 says input text means files and command-language text, distinct from interactive typing. The sort note says plain string order, not numeric. The user approved the amendment on PR #326 (2026-10-09), so its Status, the accepted ADR's pointer, core.md section 20 and the doc-site title no longer say proposed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- .../adr/007-stable-identities-amendment-1.md | 34 ++++++++++--------- context/adr/007-stable-identities.md | 4 +-- context/code/core.md | 8 ++--- scripts/build-doc-site.mjs | 2 +- 4 files changed, 25 insertions(+), 23 deletions(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index dc8ff825..6f11596c 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -5,24 +5,23 @@ improvise-cq1q) · **Builds on:** ADR 001, ADR 002 ## Status -**ADR 007 is accepted** (PR #210, 2026-09-25), and `007-stable-identities.md` stands. **This -amendment is proposed and not yet approved** (PR #326). This file states only what the -amendment changes or adds, numbered by the accepted ADR's clauses; everything else in -the accepted ADR is still true. Where the amendment conflicts with an accepted rule or an existing -behaviour, the accepted rule binds until the amendment is approved: +**Accepted** (2026-10-09, PR #326). It amends ADR 007 (accepted 2026-09-25, PR #210, +`007-stable-identities.md`). This file states only what the amendment changes or +adds, numbered by the accepted ADR's clauses; everything else in ADR 007 is still +true. Where the amendment replaces an ADR 007 rule or an existing behaviour: | clause | accepted rule or existing behaviour | the amendment | |---|---|---| | 1 | groups have no id | a group has an id and a display, and its id behaves like every other id | | 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill` | -| 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays, never declared, and referred to by id (`_measure`) | +| 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays and referred to by id (`_measure`); the `## Category: _Measure` header lists the stored measures but declares nothing | | 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | | 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit | | 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id | -| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders read display names in input text; command-language text names things by id | +| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders read display names in files and command-language text, which names things by id; interactive input matches a display or an id | | 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text | | 12 | the regression test commits the unedited text | the test edits a non-name token and commits | @@ -58,10 +57,11 @@ becomes the group's id. and `_index` and `_dim` hold the record positions. Their ids and displays are hardcoded in the application, so a file never declares one, and a line that refers to one spells its id: `_measure: column`, `region=east, _measure=revenue = 1200`. -- **Conflict with old files:** they declare and name `_Measure`. The loader recognises - `_Measure`, `_Index` and `_Dim` by their hardcoded displays, reads an old - `## Category: _Measure` section only for the stored measures it lists, and the first - save writes `_measure`. +- **The stored measures are still listed in a `## Category: _Measure` section**, each + with its display and id (`- Revenue(revenue), Cost(cost)`). Its header names the + hardcoded virtual category and is not a declaration of it. The loader recognises + `_Measure`, `_Index` and `_Dim` by their hardcoded displays, and references are + written `_measure`. - **Records mode's synthetic positions are values, not ids.** The row coordinate under `_index`, and by the same reasoning the column coordinate under `_dim`, are values of those virtual categories. They take no part in id uniqueness and cannot collide with @@ -101,7 +101,8 @@ cost`); a dimension target is rare, for a derived dimension. the same display, they are shown with a suffix: the first keeps the bare display, and the rest get ` #2`, ` #3`, … (a space, then `#` and the number) as a gapless sequence in the order their ids sort. - - The order is the ids' string order, the order `CellKey` already sorts ids in. So + - The order is the ids' plain string order, the order `CellKey` already sorts ids + in, not the counter's numeric order. So `profit`, `profit-1` and `profit-2` show as `Profit`, `Profit #2` and `Profit #3`, and `profit-10` sorts before `profit-2`. - Gapless: if the thing shown as `Profit #2` is removed, `Profit #3` becomes @@ -164,7 +165,7 @@ and the load is refused naming both. Stable identity across copies of a file is promised (improvise-8tk5). **Built code.** `crates/improvise-core/src/model/id.rs` implements the accepted steps -(`context/code/core.md` §20). After approval it must replace rather than strip, collapse +(`context/code/core.md` §20). A follow-up must replace rather than strip, collapse `_`, trim a trailing `_` and both ends' `-`, prepend `n`, make `claim` refuse what a generated id could not be, and add the virtual views and `IdKind::Group`. @@ -222,13 +223,14 @@ Under the amendment: ### 6. Display names and the wire - **Only CSV import and the `.improv` loaders turn display names in input text into - ids.** They read text people wrote. An old file's names are resolved there as a - migration. + ids**, where input text means files and command-language text. They read text people + wrote. An old file's names are resolved there as a migration. - **Command-language text names things by id.** Anything that uses the command language as a scripting language is a script, wherever the text comes from (a file, `improvise cmd`, another program). A script naming something by display fails; it gets a display only through a dedicated function such as `display_for(id)`. -- **Interactive input** (TUI, browser) matches either a display or an id, as long as it +- **Interactive input** (typing in the TUI or the browser, which is not input text) + matches either a display or an id, as long as it is unambiguous which thing the user is picking. Ids win when it is not, and the display is shown as a preview (clause 12). - **Groups on the wire:** `AddItemInGroup`'s `group` is a group id, a new group is diff --git a/context/adr/007-stable-identities.md b/context/adr/007-stable-identities.md index e044be5c..7bf92117 100644 --- a/context/adr/007-stable-identities.md +++ b/context/adr/007-stable-identities.md @@ -1,7 +1,7 @@ # ADR 007 — Stable identities for categories, items, formulas and views -**Amended:** an amendment is proposed in `007-stable-identities-amendment-1.md` -(PR #326, not yet approved). Until it is approved, this text binds. +**Amended:** by `007-stable-identities-amendment-1.md` (accepted 2026-10-09, PR #326), +which states every rule it replaces. **Status:** accepted (2026-09-25). The user approved revision 7. Revision 8 then followed the adversarial review of that head: it adds clause 12, the ADR's own reading diff --git a/context/code/core.md b/context/code/core.md index 4655a987..c00deb72 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -1713,9 +1713,9 @@ slugs to `a__b`: the steps are applied as written, and no extra underscore-collapsing step exists. A trailing run survives, so `East ` slugs to `east_`. Only *leading* underscores are removed (step 4). -**ADR 007's proposed amendment changes these rules, and this code predates it.** -The amendment (`context/adr/007-stable-identities-amendment-1.md`, PR #326) is not -yet approved. Its clause 4: +**ADR 007's amendment changes these rules, and this code predates it.** The +amendment (`context/adr/007-stable-identities-amendment-1.md`, accepted 2026-10-09) +says in its clause 4: - every character outside `a`–`z`, `0`–`9` and `-` becomes `_` instead of being stripped (not only U+0020), so `Region(code)` slugs to `region_code`, while `Q1-2024` keeps its `-`; @@ -1728,7 +1728,7 @@ yet approved. Its clause 4: This section, `slugify`, and the tests `slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and `slugify_keeps_an_underscore_the_display_already_had` describe the earlier rule -above. They change in a follow-up once the amendment is approved. The amendment's +above. They change in a follow-up the amendment calls for. The amendment's clause 4 lists every difference. **The allocator records every id it admits and releases none.** That is the diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index 55c91ba1..1cf88745 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -47,7 +47,7 @@ const SECTIONS = [ ["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"], ["context/adr/005-exact-decimal-arithmetic.md", "adr/005-exact-decimal-arithmetic", "ADR 005 — Exact decimal arithmetic (proposed)"], ["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"], - ["context/adr/007-stable-identities-amendment-1.md", "adr/007-stable-identities-amendment-1", "ADR 007, amendment 1 (proposed)"], + ["context/adr/007-stable-identities-amendment-1.md", "adr/007-stable-identities-amendment-1", "ADR 007, amendment 1"], ], }, { From cb73ad3d55976ed413cd311a149beebf0ef5567c Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 10:06:14 -0700 Subject: [PATCH 080/106] docs(adr): the _Measure section lists every stored measure (improvise-cq1q.1) Formula measures stay declared by their ## Formulas lines. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities-amendment-1.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index 6f11596c..61a54de7 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -59,7 +59,10 @@ becomes the group's id. one spells its id: `_measure: column`, `region=east, _measure=revenue = 1200`. - **The stored measures are still listed in a `## Category: _Measure` section**, each with its display and id (`- Revenue(revenue), Cost(cost)`). Its header names the - hardcoded virtual category and is not a declaration of it. The loader recognises + hardcoded virtual category and is not a declaration of it. The section is complete: + it lists every stored measure, so a file never relies on its data lines to discover + one. A formula's measure is not listed there; it is declared by its `## Formulas` + line, which carries the id it shares with its formula. The loader recognises `_Measure`, `_Index` and `_Dim` by their hardcoded displays, and references are written `_measure`. - **Records mode's synthetic positions are values, not ids.** The row coordinate under From 12642e40b62f513171ca163cab436bb85dc07865 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 10:06:59 -0700 Subject: [PATCH 081/106] docs(adr): ids sort in natural order everywhere (improvise-cq1q.1) Defines natural order, applies it to the display suffix, and adds a Status row for the conflict with CellKey's String::cmp order: RecordHash values change and saved data lines reorder; the loader canonicalises. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities-amendment-1.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index 61a54de7..560a046c 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -18,6 +18,7 @@ true. Where the amendment replaces an ADR 007 rule or an existing behaviour: | 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-` beside a stored `profit` | | 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | +| 4 | ids sort by plain string order (`CellKey::new` sorts with `String::cmp`, and the writer orders data lines by their text) | ids sort in natural order. `CellKey`'s canonical coordinate order changes with it, so `RecordHash` values change (they are on the wire, not in files), and a saved file's data lines and their coordinates are written in the new order; the loader canonicalises keys, so the data read back is the same | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit | | 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id | @@ -104,10 +105,9 @@ cost`); a dimension target is rare, for a derived dimension. the same display, they are shown with a suffix: the first keeps the bare display, and the rest get ` #2`, ` #3`, … (a space, then `#` and the number) as a gapless sequence in the order their ids sort. - - The order is the ids' plain string order, the order `CellKey` already sorts ids - in, not the counter's numeric order. So - `profit`, `profit-1` and `profit-2` show as `Profit`, `Profit #2` and `Profit #3`, - and `profit-10` sorts before `profit-2`. + - The order is the ids' natural order (clause 4), so `profit`, `profit-1`, + `profit-2` and `profit-10` show as `Profit`, `Profit #2`, `Profit #3` and + `Profit #4`. - Gapless: if the thing shown as `Profit #2` is removed, `Profit #3` becomes `Profit #2`. - The suffix is computed when a display is shown, from the things that share it, and @@ -159,6 +159,12 @@ also what the counter may make of `east`. They never collide, because every id a document holds is taken however it was made: a display slugging to a held `east-1` becomes `east-1-`, and a held counter candidate is skipped. +**Ids are always sorted in natural order**, wherever ids are ordered. Split each id into +alternating runs of digits and non-digits; compare the runs pairwise, a digit run by its +numeric value and a non-digit run by byte order; when two digit runs have equal value, +the shorter (fewer leading zeros) sorts first; when one id runs out of runs first, it +sorts first. So `profit`, `profit-1`, `profit-2`, `profit-10`, and `a-7` before `a-007`. + **Hand-written ids obey every syntax rule of a generated id.** `east-1`, `item-4` and `q1_-_2024` load; `-east`, `east-`, `2024`, `a__b` and `_draft` are refused at load. From bc08a1e41ece185db0e7875376e2b59bafada663 Mon Sep 17 00:00:00 2001 From: "Claude (adr007-steward)" Date: Fri, 9 Oct 2026 10:26:20 -0700 Subject: [PATCH 082/106] docs(adr): list every place natural order changes, and define digit-run value (improvise-cq1q.1) Status rows for records-mode order and fold order, the writer's other sorted lines, GP fit row order, and two internal fingerprints. A digit run compares by length after stripping leading zeros, then by bytes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/007-stable-identities-amendment-1.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/context/adr/007-stable-identities-amendment-1.md b/context/adr/007-stable-identities-amendment-1.md index 560a046c..940b8e38 100644 --- a/context/adr/007-stable-identities-amendment-1.md +++ b/context/adr/007-stable-identities-amendment-1.md @@ -19,6 +19,10 @@ true. Where the amendment replaces an ADR 007 rule or an existing behaviour: | 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order | | 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` | | 4 | ids sort by plain string order (`CellKey::new` sorts with `String::cmp`, and the writer orders data lines by their text) | ids sort in natural order. `CellKey`'s canonical coordinate order changes with it, so `RecordHash` values change (they are on the wire, not in files), and a saved file's data lines and their coordinates are written in the new order; the loader canonicalises keys, so the data read back is the same | +| 4 | records mode string-sorts the store on entry (`DataStore::sort_by_key`, run by `SortData`) | records-mode rows come out in natural order, and so does the order `matching_positions` folds in (gotcha 18), so a `SUM`/`AVG` can change in the last bits | +| 4 | the writer string-sorts `hidden:`, `expanded(…)`, `collapsed:` and `## Aggregation` lines | those lines are written in natural order too | +| 4 | a GP fit string-sorts its records (`improvise-gp` `dataset.rs`) | the feature matrix's row order, and so a seeded fit's folds, follow natural order: a seeded fit can produce a different tree | +| 4 | `app.rs` view-shape fingerprints and `Model`'s `Debug` string-sort ids | they sort naturally too; nothing visible changes | | 2, 5 | a collapsed entry and a group reference name a group by name | by its id | | 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit | | 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id | @@ -160,10 +164,11 @@ document holds is taken however it was made: a display slugging to a held `east- becomes `east-1-`, and a held counter candidate is skipped. **Ids are always sorted in natural order**, wherever ids are ordered. Split each id into -alternating runs of digits and non-digits; compare the runs pairwise, a digit run by its -numeric value and a non-digit run by byte order; when two digit runs have equal value, -the shorter (fewer leading zeros) sorts first; when one id runs out of runs first, it -sorts first. So `profit`, `profit-1`, `profit-2`, `profit-10`, and `a-7` before `a-007`. +alternating runs of digits and non-digits; compare the runs pairwise, a non-digit run by +byte order and a digit run by value: strip its leading zeros, then the shorter run is +smaller, then compare bytes (so runs of any length work, with no fixed-width parse); when +two digit runs have equal value, the one with fewer leading zeros sorts first; when one +id runs out of runs first, it sorts first. So `profit`, `profit-1`, `profit-2`, `profit-10`, and `a-7` before `a-007`. **Hand-written ids obey every syntax rule of a generated id.** `east-1`, `item-4` and `q1_-_2024` load; `-east`, `east-`, `2024`, `a__b` and `_draft` are refused at load. From 1a7d2fbf85ca88a8751b4dadaac41b944d42d1aa Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 13:34:45 -0700 Subject: [PATCH 083/106] docs(adr): ADR 009 aligns its ADR 007 statements with the accepted amendment (improvise-felo.4) CellKey's order is the natural id order, a store reorder reassigns Seq, CellKey pairs are (CategoryId, ItemHandle), and display suffixes are computed by the client. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index cbb35871..9bc84d21 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -61,14 +61,18 @@ is removed once its remaining uses are replaced (open question 3). `Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite -consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect. +consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the +natural order of ids (ADR 007 amendment 1, clause 4). A store reorder, such as entering +records mode, reassigns `Seq` and rebuilds clause 5's per-record stage. The circuit steps +once per model effect. The record tier and the dirty flag read this delta instead of scanning the store. The dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the formula set moves, because formula edits move no cell. -Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The -formula-set fingerprint must use the id form too, or a rename reads as a formula change. +Under ADR 007 and its amendment, `CellKey` pairs are `(CategoryId, ItemHandle)`. So a +rename is a display write and emits no delta. The formula-set fingerprint must use the id +form too, or a rename reads as a formula change. ### 4. Aggregates @@ -200,7 +204,9 @@ doubling jumps (about 14 to 28 probes at density 10⁻⁴). client, addressed to none. It carries: - the stamp `(host instance, document_generation, publication)`. The publication number increments once per published batch, and the host instance changes on restart; -- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved; +- axis changes, display relabels as `(id, display)`, and a flag when the formula set or + `measure_agg` moved. A shared display's ` #2` suffix is never sent: the client computes + it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as the circuit computes them. Moved records alone would miss formulas with cross-stem From f0119a437d1115c24ff65f4187b973be79c5de31 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Fri, 9 Oct 2026 21:13:55 -0700 Subject: [PATCH 084/106] docs(context): condense repo-map to a digest; move gotchas and inventory to their own pages repo-map.md was 173 KB (~43k tokens), too long to be part of every new agent's context. The full gotcha text moves verbatim to context/gotchas.md and the per-file inventory (with the old Wasm/CLI/Testing sections) to context/file-inventory.md; the map keeps a one-or-two-line digest per gotcha under the same numbers, so every `gotcha N` citation still resolves. The doc-site manifest publishes the two new pages, and `gotchas` gets the same gotcha-N anchors as `repo-map`. Co-Authored-By: Claude Sonnet 5.5 --- context/code/doc-site.md | 14 +- context/file-inventory.md | 428 +++++++++++++++++++++++++ context/gotchas.md | 139 +++++++++ context/repo-map.md | 617 ++++++------------------------------- scripts/build-doc-site.mjs | 4 +- 5 files changed, 671 insertions(+), 531 deletions(-) create mode 100644 context/file-inventory.md create mode 100644 context/gotchas.md diff --git a/context/code/doc-site.md b/context/code/doc-site.md index cfd4b734..09a9eb01 100644 --- a/context/code/doc-site.md +++ b/context/code/doc-site.md @@ -43,7 +43,7 @@ Not published: will meet, and nobody visiting a website is about to read a PR thread. None of the three is referenced by any published document except as a row in -`repo-map.md`'s file inventory, which is inside a code fence, so leaving them +the file inventory (`file-inventory.md`), which is inside a code fence, so leaving them out creates no dangling link. That was checked rather than assumed. **And "not published" is now derived rather than assumed** (improvise-res9). @@ -233,11 +233,13 @@ recorded as a decision rather than left to be rediscovered as a bug. ## 6. Navigation -`repo-map.md` is one long document whose gotcha list is numbered 1 to 48, and -splitting it per gotcha was rejected: the gotchas cross-reference each other -constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a reader -following those wants a scroll, not 48 page loads. It stays one page, with an -id on every list item and a compact strip of numbered links under the heading. +`repo-map.md` carries a digest of the numbered gotchas and `gotchas.md` the long +form; the map was condensed because it was too long to put in every new agent's +context. Splitting per gotcha was rejected: the gotchas cross-reference each +other constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a +reader following those wants a scroll, not one page load per number. Each of the +two stays one page, with an id on every list item and a compact strip of +numbered links under the heading. The ids come from the list item's **own number**, not from a counter, so an inserted gotcha renumbers the anchors exactly as it renumbers the prose that cites them. diff --git a/context/file-inventory.md b/context/file-inventory.md new file mode 100644 index 00000000..6ba97b8b --- /dev/null +++ b/context/file-inventory.md @@ -0,0 +1,428 @@ +# File Inventory (the long form) + +The per-file listing that `context/repo-map.md` summarises. Moved here unchanged. + +Line counts are a **dated snapshot**, not a maintained fact: measured 2026-09-24 at `c1772ca9`, and **70 of 105 had drifted past 10%** in the three days since the 2026-09-21 refresh — eleven had doubled or more (`client.rs` 255 → 943, `lockstep.rs` 1126 → 4064). Treat them as "is this file big", never as a figure to cite; re-derive with `wc -l`. The `ws-server/` and `wasm-client/` entries use a shorthand prefix, so a refresh must map it to `crates/improvise-/{src,}/` by hand or they drift silently. Test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted. + +### `improvise-core` (`crates/improvise-core/src/`) +``` +model/types.rs 3560 / 112t Model, formula eval (fixed-point), CRUD +model/cell.rs 1358 / 47t CellKey (sorted), CellValue, DataStore (interned + index), + live_records (insertion order — gotcha 18) +model/record.rs 226 / 8t RecordHash / CellId surrogates: the opaque identities a + thin client names a cell by (ADR 002) +model/category.rs 310 / 9t Category, Item, Group, CategoryKind +model/symbol.rs 89 / 3t Symbol interning +view/layout.rs 2604 / 57t GridLayout, drill, records mode, records_columns; a private + per-axis position index (`row_pos`/`col_pos`, rebuilt by `reindex`) + so a data position resolves in O(1) (improvise-felo.13) +view/types.rs 817 / 43t View config (axes, pages, hidden, collapsed, format) +model/id.rs 290 slugify + IdAllocator: how a generated id is made (ADR 007 + clause 4, core.md §20). Nothing outside the file calls it yet +view/axis.rs 190 / 5t Axis enum + label/next methods (no colour: that is a Face) +workbook.rs 413 / 22t Workbook: Model + cross-view ops, item facade +format.rs 519 / 40t format_f64, parse_number_format (display only) +``` + +### `improvise-formula` (`crates/improvise-formula/src/`) +``` +parser.rs 1394 / 79t pest grammar + tokenizer → Formula AST +quote.rs 160 / 7t quote_name / is_bare_name / pipe_quote / unescape_pipe_body: + the one home, both halves (improvise-i2m5). improvise-gp + re-exports it; improvise-io calls it from + persistence/{format,parse}.rs +formula.pest 108 PEG grammar for formula text +ast.rs 130 Expr, BinOp, AggFunc, Formula, Filter +``` + +### `improvise-io` (`crates/improvise-io/src/`) +``` +storage.rs 334 / 13t Storage trait (read/write/list/delete), PhysicalFs, MemoryFs, + `impl Storage for Rc` (improvise-hmws), and + contract::storage_contract — public, not cfg(test), because + a browser must run that sequence (harness/) +opfs.rs 289 / 0t OpfsFs: Storage over the browser's Origin Private File + System. wasm32 only. Run by harness/, not by cargo test +persistence/improv.pest 179 PEG grammar — single source of truth +persistence/mod.rs 441 / 11t document lifecycle: save/load/gzip/legacy-JSON over Storage; + facade re-exporting format_md/parse_md/export_csv* +persistence/format.rs 401 / 11t format_md: Workbook → .improv text +persistence/parse.rs 1684 / 46t parse_md: pest tree-walk → Workbook (+ grammar generator) +persistence/export.rs 243 / 5t export_csv / render_csv. Builds its layout with + with_frozen_records, NOT new — `new` skips prune_empty + (improvise-iot6) +persistence/roundtrip_tests.rs + 1236 / 62t format_md ↔ parse_md characterization (tests only) +import/wizard.rs 1967 / 62t ImportPipeline + ImportWizard. `Plan::place` is the one + placement pass, shared by `collisions()` and the build so + the two cannot disagree; `DuplicatePolicy` / `Collision` + (improvise-cgqt) +import/analyzer.rs 434 / 17t Field kind detection (Category/Measure/Time/Skip); + analyze_records is first-seen order, not HashSet (gotcha 18) +import/csv_parser.rs 590 / 26t CSV parsing, multi-file merge +``` + +### `improvise-protocol` (`crates/improvise-protocol/src/`) +``` +lib.rs 619 / 2t ViewState, ViewCursor, ViewFrame, AppMode (17), Panel, + MinibufferConfig, MinibufColor, HELP_PAGE_COUNT +reduce.rs 749 / 10t Command + ModelCommand (the wire format), HelpPage, reduce_view +render.rs 2157 / 46t CellDisplay, WidgetCache trait + one cache per widget + (GridCache/CategoryTreeCache/FormulaCache/ViewListCache/ + TileBarCache), AxisFrame/FrameEntry/FrameGroup and + header_level_widths (the grid frame, gotcha 44), + GridCache::seed (the bootstrap compaction), Projection, + bootstrap-only Snapshot +keymap.rs 975 / 0t KeyPattern, Binding, Keymap, ModeKey (17), TextEntry, + PanelKeymap, Keymaps::default_keymaps/prefix_keymap — the + shared table both front ends look up. Tests live with + dispatch in improvise-command/src/command/keymap.rs +viewmodel.rs 170 / 5t flatten_cat_tree — derivations between a cache and a + renderer, called by both front ends (improvise-edp) +face.rs 682 Face (a role: Axis, WizardField, Mode*, MinibufferOf…), Theme, + AnsiSlot — appearance as data a front end applies (ADR 004, + protocol.md §10); draw.rs::face_style is the TUI's one translation +key.rs 96 / 3t KeyCode, KeyModifiers, KeyEvent (moved from command/) +frame.rs 156 / 3t ClientFrame/ServerFrame — the ONE envelope pair, both + deployments (improvise-xks6). Nothing else defines one. + ClientFrame::Save is a request, not a Command (gotcha 45) +basis.rs 257 / 6t Basis (the three tiers), Rejection, BasisCache — ADR 001 + layer 1. Basis has no Default and no public constructor +help.rs 1241 / 14t HelpPage content and Hint { keys: Vec<(KeyCode, KeyModifiers)>, + command, caption }. `compact` drops a chord by its MODIFIERS + (anything beyond shift), never by parsing a label it just + formatted (improvise-thpe) — the guard is the signature +resolve.rs 4883 / 88t the browser's key resolvers: a BrowserKey to the Commands it + composes, mirroring each Cmd line for line INCLUDING where + siblings disagree (gotcha 47). The largest file in the crate +client.rs 943 / 15t ClientState: ViewState + one cache per widget, and the fold; + in_flight, `opened` (OpenedCell), notify, settle_status + (gotcha 47). EditorSession is in lib.rs beside ViewState +``` + +### `gp-tree` / `improvise-gp` (`crates/gp-tree/`, `crates/improvise-gp/`) +``` +gp-tree/src/lib.rs 575 / 8t Tree, Spec, eval, simplify, render, breed, Population, + SplitMix64 (vendored seeded Rng; seed() keeps the seed so + a fit can be shown and recorded, pinned to the published + reference vectors) +gp-tree/src/data.rs 1000 / 21t raw string rows -> f32 feature vectors: per-column type + inference, one-hot, mean-imputation + col.missing flag, + ColumnSpec/encode_row, Dataset (incl. warnings), + Model = trees + plan + seed + FitConfig +gp-tree/src/fit.rs 961 / 32t the search: FitConfig (Default = the CLI's numbers), fit() + -> simplified champion, cv_regress/cv_classify, + permutation_importance, folds/standardize/mse. + Returns data; never prints +gp-tree/examples/fit_csv.rs 583 CLI over the above: flags, demo dataset, all printing +gp-tree/tests/scoring_known_answers.rs + 250 / 6t f32 bit patterns + a next_u64 witness: the evidence any + change to the scoring paths needs (gotcha 37c) +improvise-gp/src/lib.rs 978 / 25t formula_text / fitted_formula: a Tree as IF(...) formula + text, validated back through parse_formula. A threshold is + the SHORTEST decimal that reads back as the same f64 (not + the exact expansion), so a row on a split boundary routes + the same in f64 as in f32 eval; non-finite values panic +improvise-gp/src/dataset.rs 506 / 15t a View's records -> the feature matrix a fit consumes — + the improvise-side counterpart of preprocess. Records mode + is already a feature matrix (gotcha 1) +improvise-gp/src/fit.rs 1778 / 35t one fit end to end: a View's records in, an improvise + Formula and the evidence for believing it out. Not in a + command, because none of it is terminal-specific +``` + +### Command layer (`crates/improvise-command/src/command/`) +``` +cmd/core.rs 475 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers +cmd/registry.rs 903 / 13t default_registry() — all registrations +cmd/navigation.rs 507 / 12t Move, EnterAdvance, Page* +cmd/cell.rs 724 / 20t ClearCellCommand, YankCell, PasteCell, TransposeAxes, SaveCmd +cmd/commit.rs 1304 / 27t CommitAndAdvance, CommitBuffer(CommitKind), CommitExport, + CommitSave +cmd/effect_cmds.rs 519 / 6t effect_cmd! macro, 25+ simple wrappers, `unless_unknown` (gotcha 51) +cmd/fit.rs 293 / 8t fit-measure: evolve a tree over the active view and store it + as an ordinary `## Formulas` line — nothing GP-specific is + left in the document afterwards +cmd/grid.rs 988 / 22t ToggleGroupAtCursor, ViewNavigate, DrillIntoCell, TogglePruneEmpty +cmd/mode.rs 469 / 13t EnterMode, Quit, EditOrDrill, EnterTileSelect +cmd/panel.rs 936 / 24t Panel toggle/cycle/cursor, formula/category/view panels +cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode +cmd/text_buffer.rs 399 / 14t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand +cmd/tile.rs 170 / 6t MoveTileCursor, TileAxisOp +cmd/wizard.rs 244 / 6t HandleWizardKey — per-step import-wizard key tables +keymap.rs 1353 / 44t KeymapSet, Keymap::dispatch/dispatch_transient, effects_of, + SetTransientPrefix — what a thin client cannot have. The + table itself is improvise-protocol/src/keymap.rs +parse.rs 350 / 23t Script/command-line parser (prefix syntax) +``` + +### Semantic UI state (`crates/improvise-command/src/ui/`) +``` +ui/effect.rs 4020 / 99t Effect trait. 63 effects: 53 `impl Effect for` blocks, 9 + simple_effect! expansions, and SetTransientPrefix in + command/keymap.rs. The 10 outside the brace-match are all + ViewOnly; 28 of the 53 are Model(_) (gotcha 38, h3y7) +ui/app.rs 5558 / 134t App, ModelState, ViewState + dispatch methods, Panel, handle_key, dispatch_key, document_generation, autosave; apply_effects declares `dirty` for WholeWorkbook and observes it for CellData/LayoutGeometry (gotcha 45) +ui/project.rs 6383 / 118t Dirty, Viewport, grid_of/category_rows/formula_texts/ + view_summaries/tile_entries, grid_updates (the diff), + ClientSession + Sessions (connect/apply/rebase/project_all) +ui/reduce.rs 1217 / 24t Checked / CheckedBatch / apply_command / model_effects / + reduce_key — one door from a Command or a keypress to the + authoritative App. `reduce_full` is GONE (improvise-h3y7) +ui/host.rs 3410 / 67t Host: one App + one Sessions; handle, commands, project_all, + rebase. Lives here, not in ws-server (gotcha 45) +ui/cat_tree.rs 174 / 8t Category tree flattener for panel +ui/viewport.rs 259 compute_col_widths / compute_row_header_width / visible_cols + (pure geometry; the width rule itself is the protocol's); + grid_width, SIDE_PANEL_WIDTH, MIN_GRID_WIDTH (improvise-gtn7.1); + GridSummary / grid_summary / grid_window (improvise-felo.13) +``` + +### TUI rendering, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`) +``` +ui/grid.rs 905 / 14t GridWidget (ratatui): no &Model, no &View — the frame comes + off GridCache (gotcha 44). Its module doc is the best-written + statement of that rule in the repo +ui/help.rs 277 5-page help overlay (render only) +ui/import_wizard_ui.rs 829 / 9t Import wizard rendering. `Panel` is the one writer, and + every write is clipped to the panel's rows and width +ui/text.rs 169 truncate and wrap, both measured in terminal cells, never bytes +draw.rs 1122 / 20t TUI event loop, frame composition; `content_tests` draws a + whole frame to check the selection beside a side panel +main.rs 428 CLI entry (clap): open, import, cmd, script +# other ui/*.rs are small panel renderers — skip unless changing layout/style +tests/grid_frame_agreement.rs + 479 / 6t the two front ends draw the same grid frame (improvise-anr9) +tests/records_wide_smoke.rs 67 / 1t +tests/shipped_documents_load.rs 333 / 8t every TRACKED .improv/.improv.gz loads, and every record + it projects is writable. "Shipped" is read off `git ls-files`, + not a .gitignore reading (improvise-c0aj), and the walk's git + is scrubbed of every hostile GIT_* setting the caller may carry +tests/web_demo_model.rs 83 / 2t every visible cell in web/demo.improv is a Record +``` + +### Browser host and client (`crates/improvise-ws-server/src/`, `crates/improvise-wasm-client/src/`, `web/`) +``` +ws-server/registry.rs 616 / 14t Registry of Hosts keyed by DocumentId; ClientId {document, + session} — a SessionId is unique only within its document. + A document is never evicted while the server runs (ADR 001) +ws-server/host.rs 4 / 0t a re-export shim only. The session loop is + improvise-command/src/ui/host.rs (gotcha 45) +ws-server/origin.rs 132 / 6t OriginPolicy: which origins may open a socket. Loopback by + default, --allow-origin widens, absent allowed, two refused + (improvise-kc7n). The only decision transport::admitted makes +ws-server/transport.rs 330 / 4t tokio + tungstenite plumbing, plus the Origin check at the + handshake — a refused upgrade, not an ignored socket +ws-server/main.rs 190 / 5t CLI entry: serve a file on a port; --bind, --allow-origin +ws-server/tests/socket.rs 389 / 6t the one test that uses a real socket: handshake, JSON + framing, SessionId routing, which document a URL asked for +wasm-client/lib.rs 455 / 8t Client: the wasm-bindgen surface (init, on_message, on_frame, + viewport/set_viewport, render_model, set_cell_at, typed_cell) +wasm-client/keys.rs 187 / 5t BrowserKey ONLY — a KeyboardEvent flattened to an improvise + key. It does NOT resolve: `resolve` and KeyOutcome live in + improvise-protocol/src/resolve.rs, shared by both front ends +wasm-client/render.rs 1249 / 21t RenderModel: editable / selected / cat tree, decided here +wasm-client/socket.rs 547 / 13t the fold (fold_frame/handle_frame), the two compositions + and typed() text→CellValue. Defines no enums +# state.rs is GONE — ClientState moved to improvise-protocol/src/client.rs +wasm-client/tests/browser_round_trip.rs 2136 / 34t improvise-1ey's scope against a real Host +wasm-client/tests/lockstep.rs 4064 / 78t host and client agree on a keypress +web/app.js 816 string calls only, no protocol knowledge, no framework. + Run by harness/tests/shell.spec.js (Chromium; the standalone half on Firefox too) — see + gotcha 48 for what that does and does not reach +web/worker.js 86 three string calls, no protocol knowledge. RUN by + shell.spec.js's no-?ws path, with bindings.rs (gotcha 45) +web/shell.js 399 everything app.js DECIDES that is not painting: no DOM, + no browser API, one `node --test` away. Owns socketUrl, + gridPlan, frameCostOf, documentTitle — so a chunk of the + browser's render decisions is NOT in the file gotcha 42 + and gotcha 46 name. Reasoning: context/code/clients.md +web/serial.js 19 the one rule worker.js exists to keep: one frame at a + time, in arrival order (gotcha 48's async-listener race), + extracted so it is testable without worker globals +web/index.html, style.css the page; legibility only +web/build.sh 38 wasm-pack build → web/pkg (gitignored). Needs + wasm-opt = ["-O", "--enable-bulk-memory"] or the bundled + wasm-opt rejects LLVM's memory.copy +web/assemble.sh 112 assembles the deployable site (copies web/*.improv) +web/demo.improv 65 the document the site ships and seeds from. Every cell a + Record — asserted (gotcha 46) +``` + +### Browser worker host (`crates/improvise-worker-server/`) +``` +improvise-worker-server/src/worker.rs 765 / 17t Worker: one port, one client; handshake, Addressed + filter, persist trigger (model_state.dirty). Every decision, every test +improvise-worker-server/src/vfs.rs 348 / 8t documents/open: seeding is a Storage::write +improvise-worker-server/src/bindings.rs 185 / 0t wasm32 shim: OpfsFs awaits + postMessage. No Rust test + reaches it, but harness/tests/shell.spec.js DOES: the + no-?ws path runs start()/OpfsFs::open under Chromium + and Firefox. + Called NEVER RUN here for months: see gotcha 45 +improvise-worker-server/src/lib.rs 51 / 1t the module doc that states exactly what is and is not + covered — the one correct copy of that fact +improvise-worker-server/tests/post_message.rs 407 / 8t main-thread traffic, strings only +``` + +### Browser harness (`harness/`) — not a workspace member + +``` +harness/playwright.config.js 50 projects: chromium (every spec), firefox (opfs, plus + shell.spec's standalone half: grepInvert @mocked-peer), + webkit (opfs only, which skips on a runtime probe: no + OPFS). CI runs `--project=chromium --project=firefox` only: + nixpkgs' WebKit aborts inside the runner VMs (ci.yml) +harness/tests/shell.spec.js 345 the first execution of web/app.js. A change to web/*.js + belongs here (gotcha 48). Playwright serves harness/serve/, + which harness/build.sh ASSEMBLES: edit web/*.js, rebuild, + or the specs run the old page +harness/tests/opfs.spec.js 103 drives the OPFS contract page and reads back its result +harness/opfs-contract/src/lib.rs 470 runs improvise_io::storage::contract::storage_contract + against a real OpfsFs inside a real Web Worker +harness/www/opfs/{index.html,worker.js} the page and worker that host it +harness/build.sh 50 assembles harness/serve/ (web/assemble.sh, the fixtures, + and the contract crate's wasm-pack build) — gitignored +# gated by the `browser` job in .github/workflows/ci.yml, which can fail the build +``` + +### Examples +``` +examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta) +examples/pretty-print.rs Parse stdin, print formatted .improv +crates/improvise-core/examples/category_width_bench.rs + The same records at 12 and at 24 regular categories + (improvise-felo.1; context/code/core.md) +``` + +### Context docs +``` +.claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl +scripts/check-comment-runs.py The two-line comment cap AND --check-evidence. The flag is + ADDITIVE since improvise-ijql — every invocation that can + exit 0 answers the cap — guarded by a subprocess self-test, + the only level a dispatch bug is visible at. Reasoning: + context/code/tooling.md §2 +scripts/check-ci-graph.py pages-deploy must `needs` every job in ci.yml, and every + suite `checkers.sh --list` knows must run in some CI job + (improvise-dak5). It REFUSES what it cannot read rather than + parsing YAML: the values it accepts are an allowlist + (tooling.md). --self-test; run in the `test` job. README + states the rule and lists no jobs (improvise-x7cq) +scripts/checkers.sh The one entry point for every *-test.sh suite: scrubs the + env to an allowlist, runs under the bash that runs it (no + version floor since #297). `ci_suites` is the table of which + suites each CI job runs (`--ci `); ci.yml names none +scripts/pr-watch Waits on a PR's checks (the pr-watch skill). Exit 5: the PR + is already MERGED or CLOSED, so there is nothing to wait for +scripts/assemble-stamp.sh The cache-bust stamp gate web/assemble.sh calls, extracted + so its refusals can be exercised without a web/pkg +scripts/assemble-stamp-test.sh Its check_the_checker, run first in the `build` CI job +context/code/tooling.md The repository's own checkers: why an evidence heading's + names come from cargo rather than from the source, and + why a leaf name is not an identity +context/test-evidence/tooling.md Evidence for the gates in scripts/ and for the tests that + hold CI's shape +scripts/nix-seed-flake-inputs.sh Seeds locked flake inputs over git so `nix develop` works in cloud sessions +prolog-sessions/kb.pl Executable knowledge base: invariants, doctrine, decisions (CI-checked) +context/adr/007-stable-identities.md Stable identities: ids unique across the document are the + keys and names are display-only; gensym suffixes apply to + generated ids only, and hand-written ids are canonical. + Accepted 2026-09-25. (There is no ADR 005 or 006 file: + improvise-l2b8 and improvise-ilvl claim those numbers.) +context/adr/004-faces-and-themes.md Faces: appearance as UI-agnostic data a front end + applies, emacs-style. Accepted 2026-09-22. +context/adr/003-live-backed-models.md A model may be backed by a live query source — + improvise as a DB client. Accepted, NOT a priority. +context/adr/002-naming-a-cell.md How a client names a cell: an opaque surrogate on + CellDisplay, whose absence means the cell is an + aggregate. Records why _Index cannot serve. +context/adr/001-concurrent-edits.md Concurrent edits between clients: record-level atomicity, + optimistic basis check per batch, advisory reservations. + Records why a causal clock was rejected. +context/cloud-agents.md Cloud-session environment: tracker sync, bd pitfalls, what is blocked +context/design-principles.md Architectural principles & testing doctrine +context/plan.md Show HN launch plan +context/repo-map.md This file +docs/design-notes.md Product vision & non-goals +``` + +--- + +--- + +# Wasm, CLI, testing and dependencies (long form) + +The three repo-map sections of these names, as they stood before the map was condensed. + +## Wasm + +The browser bundle is `improvise-core`, `improvise-formula`, `improvise-io`, +`improvise-protocol`, `improvise-command`, `improvise-wasm-client` and +`improvise-worker-server`. All seven +`cargo check --target wasm32-unknown-unknown` clean with no feature flags +(improvise-ywd), gated by the `wasm` job in `.github/workflows/ci.yml`. That +job gates **eight** crates, not seven: `improvise-gp` was added (improvise-z5eg) +explicitly *not* as a bundle member but as a candidate, because staying +wasm-clean is free now and expensive to retrofit — so the bundle is seven and +the gate is eight, and neither number is the other one's typo. +**A feature that adds a dependency to a bundle crate must be checked with `cargo tree --edges normal`** — the default output includes **dev** edges, so a plain `cargo tree -p improvise-protocol --target wasm32-unknown-unknown` reports a dev-only `serde_json` as *present* and reads exactly like a leak. Check per-crate too, because the `wasm` CI job invokes each crate separately and feature unification cannot reach across that (improvise-26dm added a `test-support` feature on `improvise-protocol` on those terms; it is off in every shipped build). + +`improvise-ws-server` is the host and is native only — tokio, tungstenite and +`Box` — so `improvise-wasm-client` cannot depend on it in anger. It +no longer needs to: both ends `pub use` `improvise_protocol::frame` and neither +defines a pair (gotcha 42). The **dev**-dependency remains, for +`tests/browser_round_trip.rs`, which hands the client's own serialized frame to +a real `Host::handle`. The root `improvise` crate is +the terminal binary and does **not** build for wasm — it depends on +crossterm; that is correct, not a gap. + +`cargo check` passing is not the same as running. Two known runtime hazards +it cannot see: `Instant::now()` panics on wasm (hence `web-time` in +`improvise-command`), and any `std::fs` call compiles but fails in a browser +— improvise-fuj closed the effects that bypassed `Storage`; autosave and the headless final write still do (improvise-31oe.11). + +## CLI + +``` +improvise [model.improv] # open TUI (default) +improvise import data.csv [--no-wizard] [-o out] # import CSV/JSON +improvise cmd 'add-cat Region' -f model.improv # headless command(s) +improvise script setup.txt -f model.improv # run script file +``` + +Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`, `--split-date ` (a date column imports as ONE hierarchical category by default; this restores the flat split, and `--extract` still picks the components — improvise-8jk6.3), `--strict` (with `--no-wizard`, any skipped record is a refusal and nothing is written, where the default writes and warns with a count — improvise-8jk6.6), and `--on-duplicate refuse|sum|separate` (default `refuse`; applies with `--no-wizard`, where the wizard would otherwise ask — improvise-cgqt). A headless import also refuses a `--split-date` that names a non-date field and an `--axis` that names a category the import did not build (exit 1). + +--- + +## Testing — the short version + +Full guidance lives in `context/design-principles.md` §7. Quick reminders: + +- **Two numbers, not one** — the ambiguity between them is what let the old + "suite runs in <2s" drift: **no single test binary over 2 seconds**, and the + workspace wall clock is the sum. Measured at `63a5c13`: **1464 tests over 35 + binaries** (1 ignored), longest single binary **0.6–1.0s across two runs on one machine** — + the spread being its own argument for reading your own run. Excludes compilation, + and excludes `harness/` (Playwright, minutes, its own CI job). *Re-measure* rather + than re-assert — `cargo test --workspace` prints a `finished in` per binary, and + **the portable signal is the ratio, not the seconds**. **Not the ratio to the + median, though: it is degenerate.** Most binaries print `0.00s` at two decimals, + so the median is zero and every ratio to it is undefined (improvise-i5z6, 39 + binaries). Compare the slowest binary to the next few slowest instead: an outlier + is one that sits well clear of them *in every run*. That is how i5z6 was read — + `improvise-formula`'s lib longest of 39 in all three runs, at 0.91–1.95s against a + runner-up's 0.50–0.92s. Both outliers so far had one cause: a generator re-parsing + its pest grammar per case. `improvise-io` (improvise-8shu, 1.42s → 0.21s) and + `improvise-formula` (improvise-i5z6, 0.84–0.98s → 0.06–0.07s) each read it once through a + `OnceLock` now. A new grammar-walking strategy should do the same from the start. + The full argument is `context/design-principles.md` §7. +- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed. +- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability. +- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Record the bug in `context/test-evidence/`, **keyed by test name** — *not* in the test's doc comment, which the two-line comment cap forbids and `scripts/check-comment-runs.py` enforces. A heading may be module-qualified and must be when its leaf is not unique. +- Coverage target ~80% line/branch on logic code; skip ratatui render paths. + +--- + +## Key dependencies + +ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta. diff --git a/context/gotchas.md b/context/gotchas.md new file mode 100644 index 00000000..d98edc42 --- /dev/null +++ b/context/gotchas.md @@ -0,0 +1,139 @@ +# Gotchas (the long form) + +The numbered gotchas that `context/repo-map.md` summarises in a line or two each. +**The numbers are stable and are cited across the repo (`gotcha 24`, `repo-map gotcha 35`) +— a citation means the entry below.** Read the ones your diff touches; the repo map's +digest says which those are. Every entry is the original text, moved here unchanged. + +Dated claims and counts inside an entry are snapshots: recount rather than cite. + +1. **Records mode is wide format.** A row is one *coordinate tuple* — the cells sharing everything but `_Measure` — and the columns are `[coordinates…, **`+cat`**, raw measures…, `+`]`: a column per regular category, then the column that creates a **new category** (improvise-2zq, sitting where the column it creates will appear), then a column per raw measure, then the column that creates a new measure (improvise-c5v). Consequences: two measures on one coordinate are ONE row; editing a category column moves **every** measure cell in the row, not just the one under the cursor; `_Dim` in the synthetic key carries the column **index**, not a label, because a category and a measure may share a name — resolve it with `GridLayout::record_col` into a `RecordCol`. Formula targets are not columns (nothing to type into them). Drill pending edits are keyed by `(row, RecordCol)`. + **`prune_axis_items` takes a predicate, not a flag vector.** `prune_empty` hands it `impl Fn(usize) -> bool` over the *data* index (group headers consume no index). The domain is derived from `items` itself, so there is no parallel length a caller could get wrong — the earlier `&[bool]` + `debug_assert` pair could disagree between debug and release if a third caller ever sized `keep` from something other than `row_count()`/`col_count()` (improvise-n3wt). **It re-marks the survivors rather than filtering the markers** (improvise-e265): runs nest, so dropping an item can leave a header with nothing under it or a `GroupEnd` closing a run that was never reopened, and no per-marker filter can tell. `remark` instead emits, before each kept item, exactly the headers and `GroupEnd`s that make the output's open runs equal the input's at every level — each survivor keeps exactly the runs it was in. Reasoning: `context/code/core.md` §6. **`prune_empty` shares one `GridEval`** across the whole row×col emptiness test, and **a grid evaluates only through `GridLayout::evaluator`**, which builds the fold memo from that layout's own `none_cats` and hidden items — so no other grid's memo can reach its cells or totals (improvise-metf, improvise-opbo; the memo itself is improvise-h6z1, `core.md` §14). + **`GridLayout::new` does not prune; `with_frozen_records` does** — and that split has now produced **four** bugs: `view_frame`, `project::grid_in` (PR #78), and `persistence/export.rs::render_csv` (improvise-iot6), which wrote rows *and columns* the grid does not draw for as long as CSV export has existed. Every remaining `GridLayout::new` call site in the workspace is test code — verified — so a new production caller almost certainly wants `with_frozen_records(m, v, None)`. + +2. **Effects never touch `std::fs`.** Every document read or write goes through `App::storage: Box` (improvise-fuj, closed). Two writes still go around it: `App`'s autosave and the headless final write call the path-taking `persistence::save` (improvise-31oe.11, open). The host picks the backend: `App::new` supplies `PhysicalFs`, `App::with_storage` takes any other — the browser worker supplies OPFS (improvise-i34, closed), and tests use `MemoryFs`. `improvise-io` keeps path-taking wrappers (`save`, `load`, `export_csv`, `parse_csv_with_warnings`) for the native CLI, each delegating to a `_to`/`_from`/`_bytes` variant that takes the backend or the bytes. +3. **Every effect declares an `EffectKind`, and there is no default.** `ViewOnly` never reaches other clients; `Model(CellData | LayoutGeometry | WholeWorkbook)` does (improvise-mae). A new effect that forgets to classify itself fails to compile — that is deliberate, since the failure mode is silent: a mis-declared `ViewOnly` edit would simply never reach connected clients. Watch two traps: the *View* lives inside `model_state.workbook`, so "touches model_state" does NOT mean "is a model change" (axis/page/format are `LayoutGeometry`, cursor and scroll are `ViewOnly` — see improvise-8uz; and `SwitchView`, `ViewBack` and `ViewForward` moved from `LayoutGeometry` to `ViewOnly` with improvise-jytm, writing `view_state` and nothing else, which is what took the latter two off the wire — but a sound derivation is not a guard: only `SwitchView` had one, and review showed reclassifying either sibling back was silent across the whole suite); and `Save`/`SaveAs` touch `model_state` for bookkeeping only and are `ViewOnly`. `AddCategory`/`RemoveCategory` are the third trap and go the other way: they look like data edits, but both go through `Workbook`, whose `on_category_added`/`on_category_removed` rewrite **every** view's axis map, so they are `LayoutGeometry`. The two "an effect declaring X actually changes X" tests cannot separate them — a category change moves model data *and* view config, so it clears either bar; `structural_effects_declare_their_kind` pins the declaration itself. A third trap, now guarded: **`EffectKind` is too coarse to check a *mapping*.** `ui/reduce.rs::model_effects` sends each of the 26 `ModelCommand` variants (counted on `c1772ca9`; gotcha 38) to one effect, and every inverse pair — `HideItem`/`ShowItem`, `AddItem`/`RemoveItem`, `CreateView`/`DeleteView`, `SetCell`/`ClearCell` — declares the *same* kind, so swapping three of them at once left the suite at 918 passing while three wire messages did the exact inverse of what they said. The mapping is name-for-name by construction, so `every_model_command_names_the_effect_of_the_same_name` asserts that, reading the type name off the effect's derived `Debug` (`Effect` is a trait object, so `type_name` is out of reach). A variant whose effect must be named something else is a variant that stopped being derived — it should fail there and be argued about. Their reach is wider than the name suggests: `Model::remove_category` also drops every formula targeting the category, which is why `Dirty::of(LayoutGeometry)` reports `formulas: true` — see gotcha 35. +4. **Commands never mutate.** `&CmdContext` is read-only; return `Vec>`. If you want to touch `&mut App`, add or extend an Effect. +5. **`CellKey` is always sorted.** Use the constructors; equality and hashing rely on canonical form. +6. **No `Model::add_item`.** With a `Workbook` in hand use the facade: `wb.add_item("Region", "East")?` / `add_item_in_group` / `remove_item` (improvise-tvt). With only a `Model`, go through the category: `m.category_mut("Region").unwrap().add_item("East")`. +7. **Virtual categories.** `_Measure` items = explicit items ∪ formula targets. Use `measure_item_names()` / `effective_item_names("_Measure")`. `add_formula` does *not* add items to `_Measure`. Formula-target lookup also falls back into `_Measure` via `Model::find_item_category`. Cell aggregation goes through `apply_agg`, formula `SUM(…)`/`AVG(…)` through `eval_agg` — they differ on empty input (absent vs. A third rule joins them for **non-numeric** values: `aggregate_cell_values` reduces text by **unanimity** — one matching value, or several copies of the same value, is that value; differing values, or text among numbers, is `CellValue::Error("mixed")` (improvise-l4qx). The **all-numeric branch is checked first**, so `Count` of a lone number is still `1` and not the number — that ordering is the whole difference between this rule and "a single value passes through". What the rule forbids is *dropping*: `filter_map(as_f64)` used to, and since `_Index`/`_Dim` are on `Axis::None` in every default view that made **every text cell in every document render blank**, including the pipe-quoted text `.improv` carries as a first-class value. `evaluate_aggregated` and `aggregate_raw` share the helper, so a formula ref over mixed values now errors `"mixed"` rather than silently summing the numeric ones. `0` for Sum/Count, error for Avg/Min/Max). `Model::evaluate_aggregated` takes `Aggregation::{Exact, OverHidden}` rather than `none_cats: &[String]` (improvise-5jm5) — the list was only ever tested for emptiness, and the guard against that shape returning is the **signature**. `recompute_formulas`/`aggregate_raw` keep the list because they walk it. +8. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full `f64`. Never feed a rounded value back into eval. +9. **Formula eval is fixed-point.** `recompute_formulas(none_cats)` iterates until stable, bounded by `MAX_EVAL_DEPTH`; circular refs converge to `CellValue::Error("circular")`. `App::new` runs it before the first frame so formula cells render on startup. +10. **Drill into formula cells** strips the `_Measure` coordinate when it names a formula target, so `matching_cells` returns the raw records that feed the formula instead of returning empty. +11. **Minibuffer keymaps come from `TextEntry`.** The eight text-entry modes are built by the `TextEntry` builder in `improvise-protocol/src/keymap.rs` — seven by `TextEntry::new`, and `RecordsEditing` by `TextEntry::with_parent` off `Editing`'s map, which is why a count taken off the `::new` call sites reads seven. It binds the shared spine (AnyChar→`append-char `, Backspace→`pop-char `, Esc→`[clear-buffer , enter-mode ]`) and welds every commit key to its `clear-buffer` via `.commit(key, cmd, args)` (improvise-w3q). `TextEntry::new` takes the `AppMode` and reads `` off its `MinibufferConfig`, so the buffer the prompt names and the buffer the keys edit are the same string by construction (improvise-k8h) — the keymap never spells a buffer name. `.rebind` exists only for CommandMode's Backspace. `SearchMode` is hand-written on purpose — no commit key, and `exit-search-mode` owns the query. `EnterMode` clears the incoming mode's minibuffer through `write_buffer` too — building `SetBuffer` directly there was unreachable only because no shipped `AppMode` is keyed `"search"`. `MinibufferConfig::buffer_key` is a `String`, not `&'static str`: a borrowed-'static field makes `Deserialize` unimplementable, and every text-entry `AppMode` variant carries a `MinibufferConfig` (improvise-a2x). A buffer named at runtime is read with `cmd::core::read_buffer` and written with its mirror `write_buffer` — "search" is not in the buffers map but its own `ViewState` field, and those two functions are the only code that knows it (improvise-gdl). +12. **Panel keymaps come from `PanelKeymap`.** The three panel modes share a spine (Esc/Tab→Normal, arrows and j/k→`move-panel-cursor `, every panel's toggle key, `:`→command mode) built by `PanelKeymap::new(panel)`; the keys that act on the panel's contents arrive through `.bind`/`.bind_args`/`.delete` because they vary in shape, not just command name (improvise-4pg). `Panel::ALL`/`name()`/`key()` in `ui/app.rs` drive the toggle keys for Normal and all three panels, so a fourth panel reaches every keymap from one `ALL` entry. `name()` must spell what `parse_panel` accepts. +13. **Prefix keys are stored as an address, not a keymap.** `ViewState::transient_prefix` holds the `(KeyCode, KeyModifiers)` that opened a prefix; `KeymapSet::prefix_keymap` resolves it back to the sub-keymap, walking parents so `RecordsNormal` reaches Normal's `g`/`y`/`Z` (improvise-jc8) — that is the command-layer wrapper; the resolution itself is `Keymaps::prefix_keymap` in `improvise-protocol/src/keymap.rs`. `ViewState` must stay free of `improvise-command` types — it is destined for the protocol crate, which `improvise-command` will depend on. Resolution assumes prefixes are **flat**; `prefix_keymaps_are_flat` fails the build if a nested one is added. +14. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the mode; Normal + `search_mode=true` maps to `SearchMode`. Minibuffer modes bind Enter/Esc as `Binding::Sequence` so `clear-buffer` fires alongside commit/cancel. Hints are checked against the keymaps: `every_hint_key_resolves_in_its_mode` (`command/keymap.rs`) splits each `AppMode::hint_text()` and requires every key it names to resolve in that mode's map, because `only_quit_has_no_hint` asserts only that a hint is *present*. It checks which keys are named, not what they do. Its tokenizer knows the hint dialect explicitly (named keys, `Ctrl-`, `F/C/V` alternatives, character runs like `hjkl`/`[:]`, the em-dash prose tail, CommandMode's `:q` ex-commands) and panics on a token it cannot classify rather than skipping it. +15. **Registry names come from prototypes, not strings.** Every `register*` call reads the name off a prototype `Cmd` instance, so a typo is impossible (improvise-9cn) — so no `register*` call spells a name. Name literals do remain in `registry.rs`, and only tests guard them — the keymap test `every_bound_command_name_is_registered` for the `Move` literals (all bound; `registered_names_are_unique` covers them too), and for the four aliases `registry.rs::every_alias_resolves_to_a_registered_command` (a hand-copied list, not the table) plus the `parse.rs` tests `alias_add_cat_resolves_to_add_category`, `alias_formula_resolves_to_add_formula`, `alias_add_view_resolves_to_create_view` and `alias_q_bang_resolves_to_force_quit` — the `Move` prototypes carry `cmd_name: "move-selection"` / `"scroll-rows"` / `"page-scroll"` (the four `jump-*` names go through `reg_jump!`), and four `r.alias` calls spell an alias and its target (`add-cat`, `formula`, `add-view`, `q!`). What types can't reach: five commands name themselves off a field (`Move` off `cmd_name`, `ViewNavigate`, `TileAxisOp`, `ToggleGroupAtCursor`, `CommitAndAdvance`), so a prototype can name the wrong *variant* and still compile. That shows up as a **duplicate** registration, not a mismatch — the wrong-variant entry collides with its sibling and the sibling's name vanishes. `registered_names_are_unique` catches it; `every_bound_command_name_is_registered` catches the keymap side, and checks **both** halves of `dispatch`'s `registry.interactive(name, args, ctx).ok()?` — the name resolves *and* the args parse — because improvise-s13 moved `enter-mode ["export-prompt"]`/`["formula-edit"]` out of the nullary class, where a typo'd mode name compiles into a dead key. It builds the `CmdContext` in each keymap's own `AppMode`, since `commit-buffer item` reads `ctx.mode`. +16. **`effect_cmd!` macro** wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written `Cmd` once there's real decision logic. Its counterpart on the effect side is `simple_effect!` in `ui/effect.rs`, which declares a one-field newtype that writes one place on the `App`. Its **kind argument is mandatory** — `simple_effect!(Name(T) => |app| place, EffectKind::ViewOnly)` — even though every call site so far answers `ViewOnly`: hard-coding it reintroduced exactly the default gotcha 3 does without, since `$place` is an unconstrained expression and a one-line `active_view_mut().number_format` setter would have reported `ViewOnly` with the suite green (improvise-8fy). Exactly six wrappers return `unless_unknown(needs, effect)` rather than the bare effect (gotcha 51): `set-axis`, `set-page`, `toggle-group`, `hide-item`, `show-item` and `switch-view`. The rest return bare effects — `delete-view` (whose `DeleteView` effect refuses an unknown view itself), `add-item`, `add-items`, `set-cell`, `create-view` — and `remove-formula` checks in `execute`. Likewise, mode entry has no per-mode shim commands — bind `enter-mode ` (improvise-s13); the name is decoded by `parse_mode_name` in `registry.rs`, and `EnterMode` clears the mode's minibuffer on the way in. +17. **Float equality inside formula eval has one epsilon, and the div/0 guard shares it.** `FLOAT_EQ_EPSILON = 1e-10` in `model/types.rs` is used by `=`/`!=` **and** by the division-by-zero guard (`rv.abs() < FLOAT_EQ_EPSILON`), in both `eval_expr` and its aggregated twin — six uses across the two evaluators (a div/0 guard plus the `Eq` and `Ne` arms in each), one constant. This gotcha previously recorded the opposite (`=`/`!=` epsilon, div/0 exact) and was stale; the mix is gone. Why an epsilon at all and why this one: `context/code/core.md` §4. +18. **`IndexMap`** preserves insertion order for categories, views, and items. Persistence relies on this — and so, since improvise-shyt, do `DataStore::matching_cells`/`matching_values`: the secondary index **narrows** the candidates, then `matching_positions` resolves each to its position in the primary `IndexMap`, sorts, and reads back. **An index whose values are `HashSet`s is for O(1) lookup, never for iteration.** Walking it gave a different order in every process, which was not tidiness. Records-mode rows reordered between sessions; `SUM`/`AVG` drifted in the last bits because `f64` addition is not associative (`Min`/`Max`/`Count` and the text-unanimity rule of gotcha 7 are order-free, so they were never affected); and worst, the **drill snapshot** froze its record list in hash order while `drill_state.pending_edits` are keyed by *row index* (gotcha 1), so the same drill opened in a second process addressed **different records** — silent misdirected writes, and nothing in the tracker knew. Measured at 18,000 cells: +20% worst case on `matching_values`, and `matching_cells` 7% *faster*, sorted positions resolving `CellKey`s with better locality. `DataStore::live_records` was the open choice and is now decided (improvise-2yc2): it walks `cells` and projects insertion order onto records, deliberately paying O(cells) rather than O(records), because `cells`' order is already the answer `matching_positions` gives and a second answer to "what order is this store in" is the disease. It is derived per call rather than memoised on `RecordEntry`, since a stored position is a copy and a record deleted and re-created would hold its old one. Note what is *not* on offer: an unordered return — every iterator has an order, and a type that merely disclaims one still hands the caller hash order, which is the guard living one level away from the hazard and the shape this whole class of bug returns through. **The second real instance was `import::analyzer::analyze_records`** (improvise-7yx), which built its distinct values through a `HashSet`: `build_model` adds items in exactly that order and `IndexMap` preserves it, so the *same CSV imported twice* produced two different `.improv` files and two different row orders. Now first-seen order, pinned by a five-iteration test because each `HashSet` gets its own `RandomState`. Also answered here since: an **empty `partial`** matches every cell vacuously (improvise-hkt6) — it used to index `sets[0]` and panic, unreachable only because both callers guarded. **A group's fold reads through the same rule.** `Model::fold_stored` folds a collapsed group's stored values once, over every record the group covers, through `DataStore::spanned_values` → `spanned_positions` (improvise-yb4b) — one scan, not a lookup per member. `spanned_positions` chooses **by index-set size** between narrowing once by the base key and narrowing once per covered item of the smallest span, whichever visits fewer (improvise-jq0m), and **both arms return store order** — sorted positions, as `matching_positions` does. That is load-bearing, not tidy: a fold that returned hash or per-item order would reintroduce the `SUM` drift above, and the choice between the arms would then be visible in the last bits of a total. **A partial as wide as the widest stored cell needs no narrowing** (improvise-felo.13): `DataStore` keeps `arities` (how many stored cells have each width), and `exact_positions` answers a strictly-sorted partial at least that wide with one probe of `cells`, since only an equal cell can match; anything else falls through to the index. The answer is the same sorted positions. +19. **Commit paths for cell edits** split across `commit_cell_value` → `commit_regular_cell_value` / `commit_plain_records_edit` in `command/cmd/commit.rs`. Keep the synthetic-records branch in sync with the plain branch when changing behavior. **Which cells refuse an editor is one function, asked at both ends of an edit**: `cmd::core::uneditable(model, layout, row, col)` is the shared decline for **opening** an editor (`EnterEditAtCursor`) and for **committing** one (`CmdContext::committable_cell_key`) — a fold or other aggregate, a formula's result, or a `CellId::NonItem` (improvise-enz6, improvise-gk51). It reads the cell's `CellId` through `project::cell_id`, the same derivation the projection uses, and the browser's `resolve::declined_at` matches on the projected `CellId`, so the two front ends refuse the same cells. Asking only at open is the bug enz6 was: Enter-to-advance moved onto a collapsed group's row and committed a write to the group coordinate. **`CellId::NonItem { category, item }` is a stored value whose coordinate names no item of its category** — records mode can draw one, and no write can address it, so it is projected as its own variant rather than as a `Record` that the host then refuses (gk51 was exactly that mismatch). Its one wording is `improvise_protocol::non_item_refusal`. +20. **Minibuffer commits go through one command.** `commit-buffer ` (`CommitBuffer { kind: CommitKind }`) owns buffer→trim→bail-on-empty→dirty→status for all three; the varying parts are methods on `CommitKind`. The add effect is emitted **last** so its failure message wins over the command's optimistic status (improvise-6t3). `CommitKind::Item` carries its target category, resolved from `AppMode::ItemAdd` by `parse_commit_kind` at registry time — so `commit-buffer item` is not constructible from a script line. +21. **An abort stops the chain, never the bookkeeping — and ordering is what separates them.** Three effects set `abort_effects`, and they are not all saves: `Save` and `SaveAs` on a failed write or a missing path (improvise-8tk — `SaveAs` was the arm left out of the original fix, so `:w /bad/path . q` quit on a failed write), and `AddCategory` when `Model::add_category` refuses, because what follows assumes the category exists — from the `+cat` column (improvise-2zq) the rest of the batch would advance the cursor and re-open an editor on a column that was never created. A `Binding::Sequence` flattens into one effect vec, so the abort carries across sequence steps; that is a property of the *flattening*, and it is why improvise-0hb's proposal to apply effects between steps was rejected — `apply_effects` resets the flag per batch, so step 2 in a fresh batch would land the `Quit` after a failed `Save`, silently reopening a P1. `a_sequence_whose_first_step_aborts_does_not_run_the_second` is the guard. **The rule the aborts taught, which is the part to carry:** an abort at the *tail* of a batch also ate the mode restoration and the buffer clearing, stranding the user in `CommandMode`/`Editing`/`SavePrompt` — where `draw_bottom_bar` renders the minibuffer *instead of* the status line, so the error the effect had just written was never drawn and the operation appeared to do nothing at all. So **tidy-up goes at the head, where an abort cannot reach it**: `ExecuteCommand` fronts its batch with `ReturnToNormalUnlessMoved`, `CommitAndAdvance` with the mode change, `CommitSave` with the mode change (found by Copilot on PR #27). Ordering answers it without giving any effect a "runs anyway" flag, and it does not weaken gotcha 31's observed-not-predicted mode settling: whatever a later effect sets, it sets *after*, so the mode a `:` line ends in is unchanged in every case — the only difference is what happens when the batch never reaches the end. **An abort is not a rollback**; that is the hole clause 4 leaves (gotcha 47). +22. **The import wizard's back path is a history stack, not a reverse table.** `advance` pushes the step it leaves onto `ImportWizard::history`; `previous` pops it, so back always retraces the path actually taken through the conditional transitions (improvise-907). Preconditions live in `validation_error()`, checked on the step where the user can still fix them — never deferred to `build_model`. `NameModel` is terminal for `advance`: Enter there calls `build()`, which parks the workbook in `built` on `Done`, and `AcceptWizardModel` takes it from there. Going back from `Done` discards `built`. **`ResolveDuplicates` appears only when `ImportPipeline::collisions()` is non-empty**: `after_fields` routes there from `ReviewProposals` or `ConfigureDates`, so it comes after the field and date steps and before `DefineFormulas`, with `NameDuplicateCategory` between them when the answer is Separate. Enter there is `WizardOp::ChooseDuplicate` (sum, keep separate, or go back). `ImportPipeline::on_duplicate` defaults to `DuplicatePolicy::Refuse`, so a path that never asked — the headless `--no-wizard` import included — errors listing the collisions rather than summing them. **A collision is a cell written by two records** (a tuple *and* a measure), not a shared tuple: records with disjoint measures on one tuple write different cells and collide with nothing (improvise-9bit). Reasoning: `context/code/formula-and-io.md`, "Colliding records are a question, not a default". +23. **`app.layout` goes stale mid-batch.** `apply_effects` rebuilds only *after* the whole batch, so an effect that reads the layout after an earlier effect changed the grid needs a `RebuildLayout` effect in front of it — the command decides the order (see `EnterEditAtCursorCmd`, `CommitAndAdvance`). Read the cursor's cell text via `App::display_value_at_cursor()`; don't build a `CmdContext` inside `apply` (it reads `layout`, the memoised `grid_geometry()` and the cursor, and a stale layout there is the bug). **Since improvise-felo.13 the rebuild is skipped when nothing it reads has moved**: `App::refresh_layout` rebuilds only when `layout_stale` (any `EffectKind::Model(_)` in the batch, or `install_workbook`), the drawn view's name, or the drill snapshot's pointer (`LayoutBasis`) changed, so a cursor key rebuilds nothing, and the column widths and row-header width are measured once per layout (`App::grid_geometry`, dropped with the layout). Two of the three inputs are **observed** (the view name and the drill pointer); `layout_stale` is **declared** from `EffectKind::Model(_)` (`touches_model`), so gotcha 3's classification hole — a `ViewOnly` effect that changes what the layout reads — now also leaves `app.layout` stale, where before felo.13 every batch rebuilt. It is still not a hand-kept list of effects. A writer that bypasses effects and changes what the layout reads must call `rebuild_layout()` itself. Measurements: `context/test-evidence/layout-cost.md`. + +24. **The cursor is per-client and keyed by view.** `ViewState::cursors: HashMap` holds `selected`/`row_offset`/`col_offset`, keyed by view name; an absent key means the origin. Reach it through `App::cursor()` (a `Copy` snapshot), `App::cursor_mut()`, and `App::reset_cursor()` — never off `View`, which no longer has the fields (improvise-8uz). Two consequences. **Changing the grid's shape must reset the cursor explicitly**: `View::transpose_axes`/`cycle_axis` used to do it themselves, and now `TransposeAxes`/`CycleAxis` call `app.reset_cursor()` — a new geometry effect that renumbers rows or columns has to do the same. **Replacing the document must clear the map**: `LoadModel` and `AcceptWizardModel` call `cursors.clear()`, because the incoming workbook's view names have nothing to do with the outgoing one's. That clearing replaces `Workbook::normalize_view_state`, which is gone — a `View` can no longer carry a stale offset. The same clearing owes the **session mirrors**: `install_workbook` bumps `App::document_generation`, and `ClientSession::project` reseeds any mirror seeded at an earlier one (improvise-cqq). An effect cannot reach `Sessions`, so the generation is the mechanism and `Sessions::rebase` is only the client-facing half. The same is true of a **geometry** change, and it owes the mirrors *both halves*: `App::cursor_epoch` is bumped by `reset_cursor` and watched by `Sessions::adopt_cursor_epoch`, which resets every mirror **and** returns `Broadcast::ToAll(Command::ResetCursor)` (improvise-gvnb). Doing only the first is **strictly worse than doing nothing** — the diff then believes every client already holds origin-positioned content and sends nothing, so they stay stale *permanently* rather than until their next scroll. The trigger is the **epoch moving**, not the command variant, for the reason gotcha 39 watches `document_generation` instead of matching on `LoadModel`: which effects renumber the grid is a per-effect decision, so a table of variants would be a second description of it, free to disagree. What is broadcast is an **invalidation, never a position** — another client's transpose must not move where I am looking (that is UI and it is mine, improvise-g0f), only declare that where I was looking has stopped meaning anything. **Since improvise-ni6k no effect calls `reset_cursor` at all — the cursor is invalidated by OBSERVING the grid's frame.** `apply_effects` diffs the frame after every model effect, **inside the loop**, because a batch that reframes and then places the cursor loses the placement otherwise (four records-mode tests pin that position). There is no list of effects anywhere, so the mechanism is total over `EffectKind::Model(_)` by construction — a genuinely stronger shape than a hand-kept table, and the reason the old "a geometry effect that forgets `reset_cursor` is invisible in both directions" hazard is closed. + + **Two arms, and the asymmetry is the whole design (improvise-mcq4).** `LayoutGeometry` hashes only the **active** view — it is the held-key path, and other views cannot reframe there since `view_mut()` writes only the active view, with the category and item effects already covered by `view_shapes`. `CellData` hashes **every** view, because a cell write can prune a row out of a view the host is not on; narrowing that arm to one view measured as *free* and silently lost a multi-client invalidation, which is gotcha 24's own strictly-worse-than-nothing case. Cost of the wide arm: 0% on the held-key path, +5.7% on an 18k two-view `SetCell`, linear in view count, accepted. State the absolute as well as the ratio: the observation roughly doubles held-key latency on large pruned documents (3.04 → 5.69 ms at 18k) and 56 ms at 90,000 cells is 3.4 frames. + + **Totality is conditional on classification**, and nothing makes classification total: a `ViewOnly` effect that renumbers the grid is invisible to all of this (gotcha 3's hole, unchanged). And two known gaps are filed rather than fixed — a geometry effect immediately after a mid-batch `SwitchView` skips the check entirely, because there is no basis for the new view yet (improvise-9c9o; under-reporting, which gotcha 35 says is the expensive direction), and `Model(WholeWorkbook)` matches neither arm, which is benign only because all three such effects go through `install_workbook` → `rebase` → a fresh `Snapshot`. + +25. **`ViewState` and its field types are `Serialize`/`Deserialize`.** It is the wire payload of the client Snapshot (improvise-q08), so `AppMode`, `MinibufferConfig`, `MinibufColor`, `ViewFrame`, `KeyCode` and `KeyModifiers` all derive both (improvise-a2x), and `improvise-command` depends on `serde` for it. The constraint this puts on new state: **no borrowed-`'static` fields anywhere ViewState can reach** — `Deserialize` cannot produce one, and the error surfaces on every enclosing variant rather than at the offending field. `view_state_survives_a_json_round_trip` guards it. `ViewState` and `Snapshot` also derive `Clone` and `PartialEq`, so the bootstrap message can be compared in a round-trip test. + +26. **Axis presentation lives on `Axis`; its colour is a `Face`.** Four vocabularies hang off the enum in `view/axis.rs` — `plain_label()` ("Row"/"Col"/"Page"/"None"), `glyph()` (which `Display` appends, giving "Row ↕"), `short_label()` (three columns for the tile bar; `None` is "·", *not* a truncation), and `initial()` (one letter for the view panel; `None` is empty). Plus `next()` for the Row→Column→Page→None rotation. **`Axis` carries no colour**: `AxisColor` and `Axis::color()` are gone. A colour is a `Face::Axis(axis)` in `improvise-protocol/src/face.rs` (ADR 004, `context/code/protocol.md` §10), resolved by the `Theme`, and `draw.rs::face_style` is the one place a face becomes a ratatui `Style` — the tile bar and the category panel both call it, so neither keeps a table. `improvise-core` still names no ratatui type. Not covered here: `persistence/format.rs` maps `Axis` to the on-disk keywords ("row"/"column"/"page"), which is a format concern and stays in the io layer. + +27. **`Axis::Filter` constrains without being navigable.** Drilling fixes its coordinates on `Filter`, not `Page` (improvise-dz8): both restrict which records are visible, but only `Page` is something the user may move or page through. The split shows up as two different questions, and picking the wrong one is the trap: **"what data is visible" is `View::filtering_cats()`** (Page ∪ Filter — used by `GridLayout` and by `add-record-row` when it builds a key from the current constraints), while **"what can the user act on" is `categories_on(Axis::Page)`** (used by `[`/`]` page cycling) or `View::tileable_cats()` (used by all three tile-select sites). `Axis::next()` maps `Filter` to itself and no navigable axis rotates into it, so tile select cannot hand one out. The `.improv` grammar accepts `filter` and round-trips a selection with it — a Filter axis only ever occurs in the never-persisted `_Drill` view, but the alternative was writing it out as some other axis. **What a virtual view must *be* is model-side and has one home** (improvise-gd28): `view::{records_spine, records_steps, drill_steps}` return `Vec` — pure data naming no `Effect`, no `App` and no front end — and `cmd/grid.rs::steps_into_active_view` is the whole of *how the TUI gets there*, twenty lines a browser answers its own way. Before the split, the drill's filter placement and the records spine were spelled out twice in `cmd/grid.rs` with nothing keeping them in agreement. `DrillState` deliberately did **not** move: it holds an `Rc` and its `pending_edits` are uncommitted writes, so gotcha 30's session-local tag already places it. Reasoning: `context/code/core.md` §11. + +28. **The browser backend is sync on purpose, with an async seam.** `OpfsFs` (`improvise-io/src/opfs.rs`, wasm32 only) implements the same synchronous `Storage` trait as `PhysicalFs`, because `FileSystemFileHandle.createSyncAccessHandle()` yields a handle whose read/write/truncate/flush **are** blocking — available only inside a dedicated Web Worker, which is where improvise-9x6 puts the app. So `Storage` never became async. What is still async splits out into four explicit methods the worker awaits from its message loop: `open()`, `attach(path)` (must precede any read/write of that document — acquiring a handle is async even though using it is not), `refresh()` (repopulates the index `list` answers from), and `commit_deletes()`. Consequence: **`delete` is visible immediately but lands in OPFS at the end of the batch** — it drops the name from the index and queues a `removeEntry`. Handles are exclusive per file, so `attach` closes any handle it replaces and `detach` releases the lock. `storage_contract` in `storage.rs` is the behaviour every backend owes; it runs against both host backends and is the written-down spec `OpfsFs` must satisfy in a browser, where the harness cannot reach. **And the premise is dedicated-worker-only, which is narrower than this paragraph used to read.** `createSyncAccessHandle` is spec'd `[Exposed=DedicatedWorker]`, and measured in a module `SharedWorker` on Chromium and Firefox it is **absent** — not rejected, not flagged, not transient: `getDirectory()` and `getFileHandle()` both succeed and the method is simply not on the object (improvise-12ha). So `OpfsFs` cannot be hosted by a `SharedWorker` as written, and "a browser worker" in this paragraph means a *dedicated* one everywhere it appears. `OpfsFs::open`'s own guard does not enforce that — it `dyn_into`s `WorkerGlobalScope`, which a `SharedWorkerGlobalScope` **satisfies**, so a shared worker sails past a check whose message names the right requirement and fails later as a raw `TypeError` deep in the glue (improvise-7gip). The consequence for multi-tab is that the machinery everyone reaches for first is the storage seam, not `Sessions`: `Sessions`, the `Addressed` filter and the one `frame` pair are all ready, and none of that helps, because the no is entirely on the handle. + +29. **"Which view is active" is stored twice, and the copies are now allowed to differ.** `Workbook::active_view` is the document's **default** — what a freshly connected client opens, and what `.improv` writes as `Initial View`. `ViewState::active_view` is **authoritative for that client**, and it is what keys `ViewState::cursors`. The second copy exists because `reduce_view` runs on a client that holds no `Workbook` (improvise-cqi); the *split* is improvise-jytm answering improvise-g0f, on the user's ruling that **the model is shared and the UI is not**. Two clients may sit on different views of one document. Three things follow. **`App::resolve_view` is the one answer** to what a view name names in this document, falling back to the default so a slice naming a deleted view cannot panic in `rebuild_layout`; `view()`, `view_mut()`, `view_name()`, `project::grid_in` and `ClientSession::view_in` all go through it, and a second copy of that fallback was the first thing review found. **The guard was replaced, not deleted** — `the_two_active_view_names_never_diverge` became `the_documents_default_view_outlives_this_clients_switches` plus `a_fresh_session_opens_the_documents_default_view`, because a deliberate split with no guard is how one silently regresses into the other. And **a deleted view is the case the split creates**: `DeleteView` can take away the view another client is on, so `Sessions::adopt_deleted_views` moves that mirror to the default and tells it, and `effect::DeleteView` does the same for the host — observed ("does this mirror still name a view the document has"), never predicted from the command variant. **Testing any of it needs a bystander that can be moved**: both audience tests first sat theirs on the default, where a wrongly-addressed relay folds to a no-op, and review demonstrated the vacuity by injecting `ToAll` twice. One gap remains: `Initial View` no longer follows the TUI user (improvise-azhb). The other — a view-scoped `ModelCommand` naming no view — is **closed** (improvise-lltt), and this paragraph described it as an open gap for a day after: every view-scoped variant carries `view: String` and `model_effects` maps it to `ViewTarget::Named`, so a connected client's `SetAxis` lands on the view it names. `ViewTarget::Active` resolves through `App::view_mut()` and is therefore the *host's* view — but **the wire never constructs one**, which is what closed it. Follow mode is deliberately not foreclosed. + +30. **Two tests gate `ViewState` membership.** `ViewState` is the *portable* session slice — the one thing serialized into the Snapshot a newly-connected client gets (improvise-q08). A field may live there only if it is **portable** (plain data from the allowlisted crates, no `Rc`/`Arc`) and **snapshot-safe** (a fresh client inheriting the value is correct). `wizard` fails both — `ImportWizard` is an `improvise-io` type, and a half-open dialog is not something to inherit — and `drill_state` fails both — it holds an `Rc`, and `pending_edits` are uncommitted writes belonging to whoever staged them. Both live on `App` under the **session-local** residue tag (improvise-h0b). The `ViewState: Send` static assertion in `app.rs`'s tests is the compile-time proxy for portability: `Rc` is not `Send`, so a re-added shared pointer fails the build (the `Deserialize` derive from gotcha 25 now rejects it too, serde's `rc` feature being off). `Send` does *not* catch the origin-crate or snapshot-safety halves — check those by hand. + +31. **`:` commands settle their mode by observation, not prediction.** `ExecuteCommand` ends every batch with `ReturnToNormalUnlessMoved(ctx.mode)`, which drops to `Normal` only if the mode is still what the batch started in. There is no `Effect::changes_mode()` any more: it was a per-type constant, and `ViewBack`/`ViewForward`/`StartImportWizard` swap the mode only on some runs, so no up-front answer was right — `:import` opened the wizard and was snapped straight back out (improvise-toc). A new effect that changes the mode needs no annotation; just don't return the batch's starting mode from a success path. + +32. **There is no aggregate render cache — each widget folds its own stream.** `improvise-protocol::render` holds one cache type per widget (`GridCache`, `CategoryTreeCache`, `FormulaCache`, `ViewListCache`, `TileBarCache`), each implementing `WidgetCache` with its own `Update` type, and a `Projection` is one such update addressed to one widget (improvise-rkeq). `reduce_view` folds only `ViewState` now — it no longer takes a cache, and `Command::Project` passes through it untouched. Two consequences. **The overlap between caches is deliberate**: the tile bar and the category panel both carry `(category, axis)` and are updated by separate projections, so neither is woken by the other's changes; do not "fix" it by sharing a variant. **Bootstrap is a *compaction*, not a replay**: `Snapshot { view, seed: Vec }` seeds a new client with a batch of the same projections the steady stream carries, so there is no whole-state type that could disagree with the incremental one. The seed is the server's *current* state, not the head of the history — `GridCache::seed()` builds it, and the property that earns it is `GridCache::folded(&cache.seed()) == *cache`. Nothing is defined for the help overlay (its content is static and `help_page` is already in `ViewState`) or for the import wizard, which now has a shape and **deliberately no projection**: `WizardCache` is built by `project::wizard_cache` and drawn locally, because a half-open dialogue is addressed to precisely the session that opened it, never to every client (improvise-764 — the rendering-side twin of the argument that removed `AcceptWizardModel` from `ModelCommand`, gotcha 38). **A cache without a `Projection` is the shape session-local residue takes when it still has to be drawn by two different front ends.** It lives in `improvise-protocol` rather than `improvise-command` for the `flatten_cat_tree` reason: when the wasm client grows a wizard it must draw *these* rows, not a second derivation. + +33. **The grid's labels frame its cells, and `apply` enforces it — the fold is not commutative.** Three consequences in `improvise-protocol/src/render.rs`. (a) A `GridUpdate::Cells` that overtakes the `RowFrame`/`ColFrame` naming its positions is **dropped**, because `fit_to_labels` truncates to the labels it has; a client must not reorder the stream, and `a_cell_update_that_overtakes_its_labels_is_lost` pins it. (b) A label change that *renames* a position blanks the cell under it: `fit_to_labels` preserves by POSITION, which is only right where the label did not change, so `apply` calls `invalidate_relabelled` itself rather than trusting a producer to send `GridUpdate::Invalidated` first. `Invalidated` therefore now means only "contents stale, frame unchanged". (c) **every arm that touches labels *or* widths re-fits** — `col_widths` is re-fitted with the columns — a new column is drawn at its label's own character count until `ColWidths` catches up — because a renderer zips widths with labels and a length mismatch indexes off the end. Fitting on the label arms alone is what let a late `ColWidths` put the mismatch straight back — a short update keeps each unheard-of column at its label's width, and a long one is dropped as having overtaken the frame that names it, exactly as (a) holds `Cells`. `GridCache` also carries `totals` and `page_coords` (improvise-n10). `totals` is framed by `col_frame` on the same terms as the cells and widths and re-fits with them, with one asymmetry: an **empty** strip means "this grid has no totals" (records mode, or an empty grid) and is therefore *not* padded, where a non-empty one is — padding it would draw a blank total row under a records grid that has none. `Invalidated` blanks the totals too: they are contents, and a stale sum under fresh cells is the one number a user takes at face value. `page_coords` is framed by nothing (it names categories, not positions) and survives a reshape. Note the fold tests: asserting `fold(prefix)+fold(rest) == fold(whole)` is vacuous (it is the definition of a left fold), so the tests that earn their place are the order-sensitivity one and the compaction-equivalence one — see `doctrine(fold_tests_must_test_order_not_the_axiom)`. + +34. **A formula target is unquoted, and pipe quoting has one home.** `parse_formula` strips the pipes off a target exactly as it does off an `Expr::Ref` — the target names a `_Measure` item, so a target that kept its delimiters made `add_formula` create an item literally named `|Total Revenue|`: a ghost column the user's real item never matched, and `Expr::Ref("Total Revenue")` never resolved to. Both the TUI formula minibuffer (`AddFormula`) and `.improv` load go through that one call, so both were affected. The writer emits `Formula::raw`, not `target`, so file text is unaffected either way — which is exactly why the round-trip tests never saw it, and why the fix repairs existing files on next load with no migration. Pipe quoting has exactly **one home**: `improvise_formula::quote`, owning both halves — `quote_name`/`is_bare_name`, `pipe_quote` (unconditional, for a `.improv` text cell that must not read back as a number) and `unescape_pipe_body`. `improvise-gp` re-exports it, `improvise-io` calls it from `persistence/{format,parse}.rs`, `parser.rs::unquote_pipe` is a shim over it, and io's own copy is **deleted** (improvise-i2m5). **The keyword guard is canonical — user-confirmed, so do not revert it as over-caution**: an item literally named `SUM` is written `|SUM|`. Bare keywords do parse, but only because `agg_call` needs a `(` and PEG backtracking falls through to `ref_expr` — rule ordering, not a stated rule — and a `## Formulas` line carries formula text verbatim, so a name `format_md` writes is read back by `parse_formula`: one namespace, one spelling rule. One thing the guard does not cover, and one that used to be an exception and no longer is. **A section header takes its name as the whole line, totally** (improvise-pczv): `header_name = { pipe_quoted ~ &NEWLINE | line_name }` — the pipe-quoted arm needs the lookahead, or `|A|B` would parse as `A` and silently drop `B`, and `line_name` catches everything else, so a header name can no longer fail to parse. Four rules share it (`model_name`, `initial_view`, `category_section`, `view_section`), which is why it is a rule and not four copies. Note a header is still *written* raw where a reference is quoted — the grammar accepts both arms, so the quoting asymmetry between a header and a reference is a writer convention, not a format one. And the two former *unquoting* copies differed only on a trailing lone backslash, unreachable from the grammar, where io returned it and `parser.rs` panicked — the surviving `unescape_pipe_body` is total. Two `## Category:` sections naming one category are **one** category, concatenated by `ParsedFile::push_category` in **pass 1** (improvise-349m) — merging in pass 2 was not idempotent, because `build_category` creates its own section's item-groups before its own `>` lines while `format_md` writes every item before every `>` line. No grammar change was needed: this is the tree-walker, not the format, and duplicate headers stay legal. And **`WHERE` now carries `IS`'s word boundary** — both are compound-atomic with `sp+` (improvise-r08u), so `SUM(Rev WHEREAS x = y)` is a syntax error rather than a `WHERE` on a category `AS`. The guard is `sp+` and **not** the `!bare_char` lookahead the ticket proposed, because the grammar-walking generator reads the grammar through `pest_meta` and **skips predicates**, emitting `WHEREa=""` — a rule whose only guard is a lookahead is a rule the generator can violate. It costs one spelling: a category may no longer abut the keyword. + +35. **`Dirty` over-reports on purpose, and the asymmetry is the reason.** `Dirty::of` maps an `EffectKind` to the widgets that kind can reach, and the two directions do not cost the same: over-reporting pays one redundant `Projection`, under-reporting leaves a client's cache wrong **forever**, with nothing in the stream to correct it. `LayoutGeometry` therefore includes `formulas`, because `RemoveCategory` wears it and `Model::remove_category` ends by dropping every formula targeting the category — a client's formula panel otherwise kept showing a formula the document no longer had. It now coincides with `Dirty::EVERYTHING` and is still spelled out, because the two are derived from different effects and need not stay equal. The derivation now lives in `context/code/command.md`, and the doc comment on `Dirty::of` points there rather than restating it; the reason `LayoutGeometry` is spelled out rather than written as `EVERYTHING` is a two-line comment inside the match body. It went stale once already — claiming `AddCategory` was `CellData` after it was reclassified — so reclassifying an effect means re-reading the prose, not just the comment. + +36. **A client's command has one door, and the door takes the `App`.** `Sessions::apply_checked(app, id, Checked<'_>)` is the only way a connected client's `Command` reaches the server — `Host` calls *that* (improvise-ahdf). `Sessions::apply` is not `pub`, delegates to `apply_checked` after `Checked::against`, and has no production caller: its call sites are tests, which is why it takes a raw `&Command`. `ClientSession::fold` and the mirror's `ViewState` are private, so a transport cannot reach the mirror at all. `apply_switch` now **takes a `SessionId`** — it is addressed to exactly one session, which is the opposite of what this gotcha used to say — moves only that session's mirror, and dirties nothing. The one thing it still asks the document is whether the view exists, because a mirror keys `ViewState::cursors` and is read by `grid_in`. The audience is conditional: an accepted switch relays **nothing** (the asker folded it, and no other client's view moves), a refused one is `Broadcast::To(asker, …)` — improvise-6c6q, where `ToOthers` told every client but the one that was wrong. `adopt_active_view` and `Broadcast::ToOthers` are **deleted**; the variants are `ToAll` and `To(SessionId, _)`, the second because the deleted-view case needs a one-session audience that is *not* the asker. The paragraph's warning about the "what do the others hear" helper survives and is why the 6c6q test drives the asker's own `ViewState` — that helper is itself deleted. What survives unchanged: the model is shared and the UI is not, and `Broadcast` is a `Vec` because one command can owe two relays with different audiences. **The trap this paragraph used to describe is gone with `ToOthers`, and the lesson it taught is not.** It was copying `ToOthers` from `apply_switch` into `apply_model`, invisible from the bystander's side because `adopt_active_view` moved the asker's mirror either way — so the drift sat between the server's picture of the asker and the asker's own slice, where a test written through a "what do the others hear" helper looks away by construction. improvise-6c6q closed the mirror-image hole on the refused path. Two rules survive the mechanism: **assert the `Broadcast` itself**, since the audience is the message rather than something to infer from folded state, and **seat a bystander somewhere a wrong relay can move it from** — a bystander already on the relayed value cannot see a wrongly *widened* audience either, because the wrong relay folds to a no-op. Review demonstrated that vacuity twice on PR #48. A `Vec` rather than an `Option` because one command can owe two relays with different audiences, and an `Option` would have had to drop one silently. Everything else is that client's alone and never touches the `App`'s own view slice: **the model is shared, the UI is not.** Folding `SwitchView` into the mirror alone was the bug — `viewport()` then read the cursor under the new view's key and the next `project()` built that client a grid out of the *other* view. + +37. **`gp-tree` returns data and never prints, and a fit takes two things to reproduce.** Two rules with one root — the crate is a library a TUI, a wasm worker or a CLI may each host. (a) A decision the user should hear about comes back as **data**: `Dataset::warnings` carries the columns `preprocess` dropped for being entirely missing (improvise-d44s), and `fit.rs` hands back pooled `(prediction, actual)` pairs and a score per feature rather than tables. `examples/fit_csv.rs` owns every `println!`, and that is the only reason its output stayed byte-identical across these moves — adding an `eprintln!` to either module is the regression. (b) **The seed is necessary and not sufficient**: `data::Model` records `seed: Option` *and* `config: Option`, because the same seed under a different population/generations/elite/penalty lands on a different tree (improvise-ksst). Both are `Option` for one reason — every `FitConfig` value and the seed `0` are legal, so nothing can double as "unknown". Note `#[serde(default)]` on an `Option` field is **documentation, not the mechanism**: serde defaults it anyway, so deleting the attribute fails no test, and a backward-compatibility test that only deletes it is not discriminating. Aim such a test at an absent field loading as the *default* instead of as unknown. The GA loop itself is `fit::fit` (improvise-0ai); `breed` remains one fitness-agnostic generation for callers whose fitness is not "predict this column". (c) **A scoring function takes one slice of rows.** `Row` pairs a feature vector with what it predicts, so the parallel `x`/`y` lengths that `zip` used to truncate silently cannot be built (improvise-dkqn, `doctrine(an_invariant_is_made_unrepresentable_not_validated)` — the same answer `prune_axis_items` got, and the reason no `Result` entered five signatures). The pairing is made at the boundary that *builds* the data: `preprocess` walks one row index and the rows live inside `Target`, because a `Dataset` carrying `rows` and `target` separately just moves the zip up one level into every caller. The empty row set is one shape, answered once in `mean_over`. Changing any of this needs **bit-level** evidence: `tests/scoring_known_answers.rs` pins every path as f32 bit patterns plus a `next_u64` witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. (d) **A fit is storable only when the document can *name* its features.** `preprocess` invents `Region=North`, `Cost.missing`, `when.year`; those are **encoding artefacts, not items of the document**, and `Expr::Ref` resolves through `Model::find_item_category` — so a tree splitting on one emitted a well-formed formula whose every cell was `Error("ref:…")`, silently (improvise-8xhy). `fitted_formula` therefore takes the `&Model` and refuses, listing what it could not resolve; `formula_text` still spells it, being the tree's own text. **The check asks the document, never the name's shape**, because a measure literally named `Region=North` is a legal document and the two are indistinguishable as strings — which also means `.missing` and the date parts are covered with no second copy of `preprocess`'s expansion rules to drift from. **This is an interim** (improvise-4zl7): a `WHERE` predicate needs a *scalar coordinate predicate* the formula language does not have — `Filter` gates a formula or an aggregate, it is not an `Expr` — would still leave `.missing` and the date parts, and must first answer what such a predicate means where `context.get(cat)` is `None`, since `Formula::filter` falls through to `aggregate_raw` and a categorical fit would silently degrade in every view that aggregates over the category. **improvise-zv8t closed the categorical half of the accident, and the close cost a document.** `fit_measure` handed `preprocess` an `Overrides::default()`, so it never said which header columns are document categories *although the very next line told `Features::of`* — and `preprocess` infers `Numeric → Date → Categorical` from the values, so a `Year` category with items `2024`/`2025` was a *numeric column* and a category of ISO dates a *date column*, both refused with `Year IS 2024` spellable all along. It now forces the same `header[..category_columns]` slice to `Forced::Categorical`. Two things follow. The **date decline is narrower than it read** — it covers a genuine date *measure*, never a category misread as one. And the forcing **widens the encoding**: a hundred-item numeric category one-hots past `MAX_FEATURES` and refuses a fit that previously succeeded by never splitting on it (improvise-e8no — and **not** to be closed by making the forcing conditional on cardinality, which is inference by another name and makes the *encoding* depend on a budget rather than on what the document says). The `Artefact` arms stay: `crate::dataset` is `pub` and takes the caller's `Overrides`, so the roles are still reachable. **Forcing by name while budgeting by index defeats the budget** — a category and a measure may legally share a name, and `no_ambiguous_column_names` ran *after* `preprocess`, so the expansion the preflight exists to prevent was allocated and only then refused; `as_categories` now returns `Result`, so the ordering is a compile error rather than a convention. The test that missed all this is the shape to remember: `improvise_eval` **fabricates** a `_Measure` item per feature name, so it asserted the emitter and `gp_tree::eval` agree against a model that could not exist. It is kept, with its scope now stated, and the property it could not state is asserted separately against a real document. + +38. **The wire vocabulary is derived twice, by the same recipe.** `Command`'s view half is the `EffectKind::ViewOnly` effects targeting the portable slice (`decision(improvise-cqi, command_set_derived_not_invented)`); its model half, `ModelCommand`, is the `EffectKind::Model(..)` effects — **28** of them, brace-matched on `c1772ca9` (2026-09-24) — minus `ApplyAndClearDrill` and `AcceptWizardModel`, leaving **26** variants (improvise-kmgs). The two added since this paragraph counted 26 and 24 are `ReplaceFormula` (improvise-km71) and `RenameRecordCoordinate` (improvise-fdk1); `SwitchView`, `ViewBack` and `ViewForward` left the model half with improvise-jytm. Every count in the rest of this paragraph is dated, so recount rather than cite it. **Completeness is now checked, not counted**: `ui/reduce/vocabulary_is_complete.rs` requires every effect to be on the wire under its own name or listed in `NOT_ON_THE_WIRE` with a verdict — `Command`, `Frame`, `Off` with its argument, or `Gap` naming the ticket that owns it (`SaveAs` and `ExportCsv` are gaps, improvise-2mqz). It is the forward twin of `vocabulary_is_composed.rs`, and it is what the enumeration below lacked. `SwitchView` no longer needs subtracting separately: it is not a model effect any more. `ViewBack`/`ViewForward` left for a reason this paragraph **stated wrongly until 2026-09-21**: it said they went for the *same* reason as the other two exclusions, session-local residue. That is false, and the falseness mattered. `ViewState::view_back_stack` **is in `ViewState`**, which is the portable slice — so by the view half's own rule (`decision(improvise-cqi)`: the `ViewOnly` effects targeting the portable slice) they were always `Command`s and were simply **missing**. The cost of the mis-statement was a live divergence nobody could see: `effect::SwitchView` pushes the leaving frame and clears the forward stack, while `reduce_view`'s arm set `active_view` **and nothing else** — so **no client ever had a back history at all**, and that, not the drill commit the `declined()` comment blames, is why `view-back` was dead in a browser (improvise-gd28, PR #115). **The lesson is about derivations, not about these two variants**: a derivation stated as a rule is only as good as the last time somebody enumerated it *against* the rule. This one had been checked for correctness — every listed variant belongs — and never for **completeness**. Note the obvious `awk` extraction reports **30**: it picks up `HelpPageSet`, which is `ViewOnly`, because that is the last `pub struct` before `#[cfg(test)]` and the test module mentions `ModelChange::` — brace-match the `impl Effect` blocks instead (`doctrine(derive_then_verify)`). Three things follow. **A `ModelCommand` does not mutate**: `ui/reduce.rs::model_effects` maps it back to the effect it came from and `apply_command` runs that through `App::apply_effects`, so effects stay the only mutation site and the `Dirty` is the effect's own `EffectKind`, never a second classification beside gotcha 35's table. **Two model effects are deliberately off the wire.** `ApplyAndClearDrill` and `AcceptWizardModel` were once kept on the grounds that the *command* is portable even where the operand is not; **that was the wrong criterion** (improvise-3x3u). A `Command::Model` is routed with no `SessionId`, which is right exactly when the change is addressed to *no one* session — and these are addressed to precisely one, never the sender: their operands are `drill_state.pending_edits` and the built wizard workbook, session-local residue (gotcha 30) private to whoever staged them (improvise-tom1). Unaddressed, B's `ApplyAndClearDrill` commits A's uncommitted typing and B's `AcceptWizardModel` swaps the document out of a dialogue B never opened — both demonstrated against `Sessions::apply` before removal. So **28** − these two = **26** on `c1772ca9` (it was 26 − 2 = 24 before km71 and fdk1), and the older spelling of this sum (`29 − SwitchView − these two = 26`) was itself a survival of the same staleness this paragraph documents: it subtracted `SwitchView` and forgot its two siblings, so it contradicted the count at the head of the paragraph and the enum’s own 24 variants (found by ChatGPT Codex on PR #116). The effects are untouched and still reached by the key path. Not permanent: multi-client drill wants the *addressed* form, which is where ADR 001 layer 2 goes anyway. Guarded by `no_model_command_commits_another_clients_session_local_state`, plus a companion asserting both effects still exist and are still model changes so the first cannot pass by naming nothing. **`Sessions::apply_checked` routes the model half to `apply_model`**, which relays `adopt_cursor_epoch`'s `ResetCursor` and `adopt_deleted_views`'s per-client switch; a document-replacing `LoadModel` bumps the generation and lands in the rebase arm. The former claim here — that `ViewBack`, `ViewForward` and `LoadModel` all end in `App::switch_to_view` so every mirror takes the accepted active view — was wrong on all three clauses after improvise-jytm and is deleted. A keymap lookup still cannot return a `Command`, and not for want of vocabulary: a binding names an *intent* resolved against `CmdContext` (`move-cursor down`) while a `Command` is the *resolved mutation* (`SetSelected { row, col }`) — improvise-gxi's AC 4 asks for something the split forbids. + +39. **`Applied` is an enum because a document swap is not a diff.** `Sessions::apply_checked` returns `Applied::Diff { dirty, broadcast }`, `Applied::Rebased { snapshots }` or — since ADR 001 layer 1 — `Applied::Rejected { reason }`, and the enum *is* the mechanism: a host that wants the `Dirty` must match, and matching names the case where a `Snapshot` is owed. The server's mirrors self-heal — `ClientSession::project` reseeds on the `document_generation` bump — but `ViewState::cursors` lives on the **client**, keyed by the outgoing document's view names (gotcha 24), and only a fresh `Snapshot` replaces it. `Command::Model` put the three `WholeWorkbook` mutations on the wire (gotcha 38), so the host that must send the snapshots is no longer the host that asked. `apply_model` therefore calls `rebase` itself rather than instructing the host to, exactly as `apply_switch` adopts the accepted view itself. **The trigger is the generation moving, not the command variant**: a `LoadModel` that cannot read its path leaves the document alone, and reseeding on the variant would cost a client its cursors for a *failed* load — `a_load_that_fails_is_not_a_rebase` guards that arm. A `rebase: bool` beside `dirty` was rejected, because the host that forgets to call `rebase` is the host that forgets to read the flag. The **third** case is a *normal outcome*, not an error, and it is in the enum for the same reason the other two are: a host that wants the `Dirty` must match, and matching names it. `ServerMsg::Rejected` is the matching frame, addressed to **the asker alone** — nothing happened, so nobody else is owed anything — and the client keeps it as a `Rejection` rather than a sentence, because force needs to know *which* record and *which* kind (gotcha 43). + **Fixture warning.** When testing a document swap, the incoming document's active view must have the **same name** as the outgoing one. A rename makes `apply_model` broadcast a `SwitchView`; the client folds it, and the stale cursor is then read under a key nothing has written — right *by accident*, with the bug invisible. `other_document()` renames, and a first version of this test passed because of it; `a_document_shaped_differently_under_the_same_view_name()` keeps the name and changes only the shape, so there is nothing to hide behind. + +40. **A batch's basis is three tiers, and only two of them can be read off the effect.** `App` carries `document_generation` (bumped by `install_workbook`), `layout_version` (per *view shape*) and `record_versions` (per *record*) — ADR 001's stale-basis check, and shared document state, so never on `ViewState` (gotcha 30). **The unit is the record**: `record_of(key)` is the coordinate tuple minus `_Measure` (`record_row_identity`), because renaming a category coordinate in wide records mode moves every measure cell in the row (gotcha 1) — a cell-level unit is violated by one legal edit, so the unit must be at least as coarse as the coarsest legal write. **Only the `Dirty` is read off the `EffectKind`; both lower tiers are *observed* around the loop** — the source says so at `apply_effects` and this paragraph said the opposite until improvise-ilvl's audit. The `EffectKind` decides only *whether to look*: `touches_model` arms the shape snapshot, `writes_cells` the record fingerprints. `record_versions` **must** be observed, because an `EffectKind` says *that* a batch writes cells, never *which*; `layout_version` is observed for a better reason than necessity — `observe_reshape` diffs `view_shapes` before and after and bumps only where a view's shape actually moved (improvise-pdvm/nynk), so it is a **fact about the document** rather than a claim about the batch. Two consequences the declared form did not have. An **idempotent** `set-axis` bumps nothing, and batching two real changes into one line or two gives the same `2`. And the gotcha 21 argument is subsumed rather than restated: an aborted batch moved no shape, so there is nothing to not-claim — which is why observing once **after** the loop is correct where accumulating `Dirty` inside it is. **`view_shape` hashes what is drawn, not occupancy** (its doc comment: "Drawn, not configured") — axes, the drawn item names minus hidden ones, folds, page selections and the `prune_empty` *flag* — so an occupancy write does **not** bump `layout_version`: a `CellData`-only batch that prunes a row out of a view moves the cursor epoch (the frame observation) and not `layout_version`, as `context/code/command.md` records (the question that raised was improvise-6nsd, closed). The arming is wider than the tier (`touches_model` takes the shape snapshot), but a bump needs a shape hash to have moved. Three further consequences. A write landing the value already there **bumps nothing**, which is correct — no basis was invalidated. A `LayoutGeometry` effect that drops cells **will** bump their records when it shares a batch with a `CellData` one: an over-bump costing at most an unnecessary rejection, the same direction `Dirty::of` is deliberately sloppy in. And only *raw* data is fingerprinted — formula results live in `Model::formula_cache`, and a record whose derived value moved because someone edited elsewhere has not itself been written. The diff collects into a **set**: chaining `before.keys()` with `after.keys()` counts any rewritten record twice, a bug that survived one test run. + +41. **`Storage` is not a VFS, and that closes the SQLite question.** The trait is whole-document `read`/`write` bytes (gotcha 2, `invariant(storage_is_two_ops)`) — no offsets, seeking, handles or locking — so an `sqlite3_vfs`, which needs `xRead(offset,len)`/`xWrite`/`xTruncate`/`xLock`, has nothing to bind to. SQLite *can* still sit behind it, as an in-memory database serialized to a blob (`rusqlite` `serialize`/`deserialize_read_exact`, proven with a probe before declining — declining on a capability that actually exists would have been the wrong decline). It is worth nothing: measured on 18,000 cells the SQLite container is **22% larger** than `.improv`, **4.7× larger gzipped**, ~**160×** larger at the floor (24 KB against ~150 bytes), and still rewritten whole on every save, so the one dimension SQLite normally wins — incremental page writes — is unavailable. improvise-9ix was declined on that evidence. Two things generalise past SQLite. **A second serialization format is a second implementation of the grammar**, obliged to track `improv.pest` forever; and **its own round-trip test cannot detect drift from `.improv`**, because it never compares the two — that missing cross-format equality test is what made the ticket look cheap. If a structured-query story is ever wanted it is a one-way `export_sqlite` beside `export_csv`, native-only and feature-gated: `libsqlite3-sys`' C build fails under this repo's nix cc-wrapper for `wasm32-unknown-unknown`, which would break the `wasm` CI gate the day it landed. + +42. **The browser client's render decisions are made in Rust, and the frames it speaks are nobody's but the protocol's.** `improvise-wasm-client/src/render.rs` builds a `RenderModel` in which every question with a *silent* wrong answer is already answered, because the DOM half (`web/app.js`) is untested JavaScript and a wrong answer there shows up as a wrong pixel rather than a failing test. Three such answers. **`editable` is a fact about the `CellId` variant** — `Record` only, per ADR 002 — and the trap is the third case: `None` means "no projected cell", **not** "aggregate", so conflating them either hides a cell or offers a write the type system exists to make unconstructible. **The selection is placed by reconciling the scroll offset**, because `GridCache` is viewport-indexed while `ViewState::selected` is absolute. `render.rs` does it by *adding* — `r + cursor.row_offset == sel_row` — rather than subtracting, which is why there is no underflow path and why a selection above the window simply never matches. Omitting the offset entirely looks right until the first scroll, so the test that earns its place uses **nonzero and unequal** offsets: at `selected=(5,1)`, `row_offset=4`, `col_offset=1` it separates "reconciles" from "ignores" *and* from "swaps the two offsets". **The category panel is `viewmodel::flatten_cat_tree`**, never a second walk of `CategoryTreeCache` — that second walk was improvise-edp. The page is four string calls: `Client::viewport` (the handshake — the `Snapshot` seeds the *visible* cells, so one built at extent zero is an empty grid and a resize is a fan-out with nothing dirty), `on_frame` (one frame, one fold, arrival order — gotcha 33 holds *inside* a batch, so never coalesce), `render_model`, `typed_cell`. + + **One envelope pair, two deployments (improvise-xks6).** `improvise_protocol::frame` owns `ClientFrame { Viewport, Commands(Vec<_>), Save, Key }` and `ServerFrame { Snapshot(Box<_>), Commands(Vec<_>), Rejected, Accepted }` (as of `c1772ca9`; this line named a singleton `Command` and no `Save`/`Accepted` for some time after both changed); `improvise-ws-server` and `improvise-wasm-client` each `pub use` them and **neither defines one**. There used to be **three** pairs for two deployments — the socket's server side, a hand-kept *mirror* of it in the browser bundle, and a third written for the worker (improvise-tm6) before the projection architecture landed. The question that had to be answered before any of it moved was whether the **worker** and the **socket** want the same vocabulary, and the answer is **yes**: every difference was a *date* rather than a decision, and the test applied to each was whether the **argument** for it names the thing they differ in. None did. `Viewport` is deployment-independent — the extent is not in `ViewState` and no effect produces it, so no derivation reaches it, and a worker hosting the same `Sessions` owes `set_extent` the same answer; it is the handshake on both because a `Snapshot` seeds the *visible* cells. `Key` is deployment-independent too — a binding names an *intent* resolved against a `CmdContext` a thin client does not hold (gotcha 38), which says nothing about what the bytes travel over; the socket lacked it because key forwarding was never built on it (improvise-bpq2), and `Host::handle` now answers `ClientFrame::Key` with an explicit empty vec that **says so**, where the frame used to just fail to parse in `next_client_msg`. And `Commands(Vec<_>)` batches where a singleton cannot, because a fan-out must arrive whole and in sequence (gotcha 33) — a property of the **fold**, not of the transport; an in-process `postMessage` interleaves exactly as a socket does. The worker pair was not merely different but **incoherent by omission**: written before ADR 001, it could carry `Command::Model { basis, .. }` and had no `Rejected`, so a client speaking it composed a claim that could never be answered. That incoherence — not the difference — is the tell for staleness. + + **The mirror is gone, and so are the two tests that held it honest**; that deletion was the ticket's measure of success. `the_socket_framing_matches_the_server` round-tripped hand-listed fixtures through the other crate's types to catch a **rename**; `every_server_variant_is_one_this_crate_mirrors` matched exhaustively to catch a **variant added on the server and never mirrored**. With one definition both are **type identity**: a rename is a compile error at every call site in both crates, and the un-mirrored variant is not a constructible state. That is stronger than either test, and it is the shape to insist on — **if a consolidation leaves one of its compatibility tests alive, the consolidation is not finished.** What survives is `the_retired_spellings_do_not_parse` in `frame.rs`, aimed at the one thing still possible (the retired `Bootstrap`/`Command` spellings returning as serde **aliases**, which would be two homes again wearing one type), and it keeps the injected-fault discipline this paragraph earned the hard way: **assert the mutated input is still well-formed** before feeding it in, and **assert the error kind** — serde's `unknown variant` — not merely that an error occurred, because a rewrite that also breaks the JSON makes the check pass against an enum that really does have the variant. End-to-end compatibility is now `tests/browser_round_trip.rs`, which hands the wasm client's own serialized frame to a real `Host::handle`; `improvise-ws-server` stays a **dev**-dependency of the browser bundle for exactly that. + + **What the harness does and does not reach is gotcha 48's subject** — `web/app.js` is no longer unrun (`harness/tests/shell.spec.js` executes it under Chromium, and its standalone half under Firefox), improvise-hwpi is closed, and the quiet-failure and never-reads-a-key claims that used to stand here went with it. What remains unexercised is named there: the grouped and multi-level DOM, the CSS and the `ResizeObserver` extent arithmetic. What a careful read *did* establish about the last of these: the observer cannot feed back into itself (`#grid` is a flex child with `overflow: auto`, so `replaceChildren` cannot resize it), and `ROW_PX`/`COL_PX` under-estimate, so the worst case is a row of over-fetch. What *is* verified end to end is the Rust half: `tests/browser_round_trip.rs` drives improvise-1ey's stated scope against a real `Host` over a real `App` touching the client only through the four string calls. One deployment note that is not a bug: `examples/demo.improv` leaves `Customer`/`Date` on the None axis, so **every** cell is an aggregate and nothing is editable — correct behaviour, bad demo. Pick a model with no regular category on None. + +43. **A batch carries its basis, and the server had to publish it first.** `Command::Model { basis, cmd }` is a **struct variant** so the basis cannot be forgotten, and `Sessions::apply_model` checks it *before* applying anything — a stale batch is never applied, not even partially (improvise-xwi6/fkfg, ADR 001 layer 1). Four things to know. **`Basis` has no `Default` and no public constructor**: a zero-valued basis is not "no claim", it is the claim that *the document is fresh*, which in a new `App` is true — so a default would be a forgeable bypass rather than an absence. `BasisCache::claim` is the only door a client has and it returns `None` until the server has published one. Note what that does and does not buy: `Basis` derives `Deserialize` because it must cross the wire, and its fields are public, so **a hand-rolled client can state any tiers it likes**. This is an *anti-staleness coordination check, not a security boundary* — `RecordUnstated` closes **omission**, not **invention**, and nothing here tries to close invention, because a client that forges a current basis is indistinguishable from one genuinely caught up. "The only constructor" is a statement about clients written against this crate, which is still worth having: it makes the honest path the easy one. **Which records are checked is the server's decision** — `reduce.rs::records_named`, an exhaustive match with one or-pattern for the variants that name no record, so a new record-writing variant fails the build. Reading the list off the *client's* claim would be a **force by omission**: the record tier skipped, with no flag to remove and nothing in the message saying so. A claim that omits a record the command writes gets `Rejection::RecordUnstated`, a case of its own because "you lost a race" and "you were never in one" are different things to tell a user. **`Rejection` keeps structural and data staleness apart**, which is the whole reason ADR 002 put a maintained `RecordHash` index in `DataStore`: `RecordGone` means nothing resolves and there is nothing to force (**force never creates structure**), `RecordMoved` means it resolves at another version and force is one keystroke — which is also why the wasm client stores the `Rejection` itself rather than collapsing it into `status_msg`, since a string cannot carry which record or which kind. **And the ADR omitted the publication.** It said a batch carries the basis it was composed against and never said how a client learns one; nothing in the projection stream carried any of the three tiers, so the check as written was satisfiable only by forging. `Projection::Basis` + `BasisCache` close that — the server publishes the two document tiers plus the counters for the records *this client's grid names*, diffed against what that client was last told and seeded in the bootstrap — and it is emitted **after** the grid updates in a batch, because the counters describe the cells those updates carry: a basis arriving first would let a client claim a version for a value it had not been shown, the mirror image of gotcha 33. That ordering was documented as load-bearing in three places and **held up only by the order of two statements** until `a_published_basis_follows_the_grid_updates_it_describes` pinned it (proved by injecting the swap) — the same job `a_cell_update_that_overtakes_its_labels_is_lost` does for the mirror-image rule. Two things are unchecked and deliberately so: `Command::SwitchView` carries no basis (it sits outside `Command::Model`; the workbook refuses a view it does not have, so it cannot corrupt, but it is a hole in the rule), and `apply_command` does not check at all — **the check is a property of the session door, not of the mutation path**, which is right because a host holding the `App` is the truth rather than a claimant. That second one **was** a door rather than an exemption, and needed a type to stop being one (improvise-ondj): both took a wire `Command` and their `Command::Model { cmd, .. }` arm **discarded the basis with `..`**, while `ui/reduce.rs`'s own module doc pointed every future host — the browser worker included — at exactly those two names. **The struct variant closed omission *from the message*; it did nothing to close ignoring *on arrival*, which is the half a struct variant is usually reached for.** So both now take `Checked<'_>`, a private-field newtype over a `&Command` whose only constructor is `Checked::against(&App, &Command) -> Result<_, Rejection>` — also the only caller of `stale_basis`. `Sessions::apply_checked` receives the proof and `Checked::against` is answered **once, before the dispatch**, so `apply_switch`/`apply_model` receive the proof rather than each remembering to ask: an invariant made unrepresentable rather than validated, the same answer `prune_axis_items` and `gp-tree`'s `Row` got. Visibility was the weaker option and was rejected for a concrete reason — `pub(crate)` closes only the out-of-crate path, and `reduce_full` had **no production caller anywhere in the workspace**, so it would have become dead code. It has since been **deleted** — which is the argument’s own conclusion arriving, not a contradiction of it; the surviving sibling is `reduce_key`. This paragraph named it in the present tense for some time after, so read every function name here as a claim to check. **The regression guard is therefore the signature**: the test that reproduced the bug no longer compiles, and what remains asserts the constructor refuses and that refusing leaves the document untouched. One residual hole, documented on the type and filed: a `Checked` does not name *which* `App` it was checked against, so a multi-document host could in principle check one and apply to another. That is also why `ui::reduce::caught_up(&App, cmd)` is not a bypass: it takes `&App`, so nothing reachable from a wire can call it. **Since improvise-955n the frame, not the command, is what is answered — and only two of the three tiers hoist.** `CheckedBatch::against` runs before anything in a `ClientFrame::Commands` is applied. `document_generation` and `layout_version` describe the *document*, so answering them once for the frame is exactly right, and a first version hoisted all three — which was a lost update: the record tier asks "is the client's picture of **this record** current", and which records are at stake is a property of the *command* (`records_named`). Skipping it for later commands let `[SetCell(North), SetCell(South)]` overwrite a South somebody else had moved, silently, with the frame reported accepted, and invisible to the other two tiers because a `SetCell` is `CellData` and moves neither. So every mutation answers for its own records **against the frame's basis** — the one the first basis-carrying command stated, never each command's own field, or a later command could claim a fresher picture than the frame was composed under. The one exemption is stated over a *property* rather than a list of commands: a record that **neither resolves in the document nor is claimed by the basis** is one the frame is about to create (`[AddItem(East), SetCell(East)]`, which `BasisCache::claim` cannot possibly claim for), and only that combination is exempt; a record that does not resolve but *is* claimed is still `RecordGone`. That the `Basis` inside a *later* `Command::Model` is ignored rather than compared is an honest interim: clause 1 says the **frame** states one basis, and moving the field from `Command::Model` to `ClientFrame::Commands` would make that unrepresentable-otherwise instead of a convention — a protocol change that belongs with clause 4. Two things worth keeping from the rejected design: **arity is not a property of a variant** (a future `Command::RunScript` is one variant performing unboundedly many mutations), so no count over the vocabulary can enforce "one mutation per frame"; and **the transaction boundary is not the frame**, because one command may already be a chain, which is why clause 4's mechanism belongs at `App::apply_effects`. + +44. **The TUI grid draws from its cache, and since improvise-anr9 the cache frames it too.** `GridWidget` holds no `&Model` and no `&View`: cell text, aggregate-ness (`CellId::Aggregate`, ADR 002), column widths, the page bar, the totals strip and now the **frame** all come off `GridCache`. The TUI still builds that cache over the **whole** grid (offsets zero), so a cache position is an absolute position — the browser's `Viewport` is a real window, the TUI's is not, and the two clients therefore use it differently; column totals **must** be projected rather than client-derived, because the sum runs over every row *including the ones a viewport scrolled past*. The frame is an `AxisFrame` per axis — `cats` (the categories, outermost first) plus `entries: Vec` interleaving `Group(FrameGroup)` headers and `GroupEnd { level }` markers with `Item { tuple, collapsed }` label *tuples*, exactly as `GridLayout::row_items` interleaves them. That replaced a `Vec` built by `GridLayout::row_label`, which slash-joined the tuple — the exact rendering `two_row_categories_produce_cross_product_labels` says must not appear — so the browser drew `Food/Alice` where the TUI drew two header columns, with no groups and no collapse indicators. It is **one** type and not a `labels` plus a `levels` beside it: two answers to "what are the labels" is the disease (improvise-edp), and this is the second cure, which cost a *type* change rather than an addition. Three things to know before touching it. **`leaf_labels()` is not "the labels"** — it is the innermost label, what a one-line summary calls a row by; a draw loop wants `header_labels(first_rendered)`, `groups()` and `shares_ancestry(sel)`, each of which is one pass, because `item`/`label`/`shows_level`/`group_of` scan `entries` from the start and asking them per position is quadratic in the frame *and grows with the TUI's absolute scroll offset* (found three times on anr9; the per-position forms survive for one-off lookups and say so). **The width rule has one home**, `improvise_protocol::header_level_widths` — widest label per level, plus a gap, clamped — reached by `AxisFrame::header_widths` and by `ui::viewport::compute_row_header_width`, which borrows level tuples off a `GridLayout` rather than building a frame on the keypress path. **A `cats` change re-identifies every position**: a cell's identity is `cats` zipped with its tuple, so `invalidate_relabelled` blanks the whole axis when the categories move, not merely the positions whose tuples differ, and blanks the **totals** on a recategorization of *either* axis — a column total is a sum over the rows, so re-identifying the rows leaves a number answering coordinates the grid no longer has (gotcha 33(b): the rule lives in `apply`, not in a producer's memory). What is still **not** on the cache is `record_col`, and that is correct: it keys pending drill edits and is session-local (gotcha 1). `body_from(idx)` takes an argument only because the two front ends scroll differently — a browser's frame is already windowed and passes `0`, the TUI's covers the whole grid and passes its row offset — not because the rule differs. `group_of` answers group **membership**, and since improvise-e265 membership is **the innermost open run**: `FrameGroup` and `FrameEntry::GroupEnd` each carry a `level` (an index into `cats`), runs **nest**, a header at level L replaces L and closes every deeper run, and a `GroupEnd { level }` closes L and deeper. Before e265 it was the nearest preceding header unless a level-less `GroupEnd` intervened (improvise-gd83, PR #77), which could not express one run inside another. `GroupEnd` draws nothing and exists only so membership is not inherited. `AxisFrame` deserialises through `UncheckedAxisFrame` and **refuses** a `level >= cats.len()` — a level naming no category is a malformed frame, not a clamp. And `FrameEntry::Item` is `{ tuple, collapsed }`: `collapsed` is the **layout's own fold flag**, read by `AxisFrame::folds`, so a client asks the frame whether a position folds rather than re-deriving it from the groups (improvise-4k1j). Both of the gaps this sentence used to list are **closed**, and it named them as open for a day after: `group_runs` coalesces by **header identity** (`std::ptr::eq`), not by name, so two runs under two headers equal as values are two runs (improvise-h0iv); and `AxisFrame::first` carries **no** `serde(default)` — its doc comment says why, since zero is a claim and not an absence (improvise-1wl0). `doctrine(a_stale_list_of_what_is_missing_is_worse_than_no_list)` is about exactly this paragraph, twice now. And `project::grid_of` used to evaluate every cell per frame where the widget evaluated only the visible ones; since improvise-felo.13 the host's own layout keeps a `GridSummary` (whole-grid widths, column formats and totals, `ui/viewport.rs::grid_summary`, held in `App::grid_geometry()`), and `grid_in` evaluates only a `grid_window` over it. The browser's window therefore costs its own cells. A session on another view, or a rederived model, still takes the whole-grid `grid_body` path. `grid_window` equals `grid_body` where the summary is current. Since improvise-jytm `grid_in` may additionally **derive a `GridLayout`** for a session on a view the host is not on; the host's layout is reused where the names agree, which is the common case and the only one that can carry a drill snapshot. + + **`RecordHash` is document-scoped; `AggregateHash` is layout-scoped and cannot be otherwise** (improvise-gd28). A `RecordHash` resolves through `DataStore`'s maintained index — but **only when the command names a view the document has** (improvise-wpxr; Codex P2 on #281). `SetCell`/`ClearCell` carry `view`, and the lookup is **strict**: a view the document lacks drew nothing, stored or not, so it is `Rejection::RecordGone` rather than a `resolve_view` fallback. An **empty** record — one the store holds no cell for, which the grid still drew — resolves against **that named view's layout**, the only place it exists. `RenameRecordCoordinate` is the exception and resolves in the **store alone**, since only a record holding cells can be renamed: `protocol::named_records` answers each command's records as `NamedRecord::Drawn { view, record }` or `NamedRecord::Stored(record)`, one exhaustive match. Every one of these questions goes through `App::grids()`, a `Grids` borrow that memoises **one derived `GridLayout` per view for as long as the borrow lives** (improvise-kimw) — hold one across a check, never across a mutation, since the layouts it holds describe the model it borrowed. An aggregate has no existence outside the layout that derives it, so there is no index to fall back to and none can be built — `App::resolve_aggregate` therefore answers only for the **host's** active view, which since improvise-jytm is not the client's. **That, and not ADR 001 layer 2, is what blocks browser drill.** The ticket's blocking argument was right about the *commit* (`ApplyAndClearDrill` is addressed to one session, gotcha 38) and wrong about the *entry*: entering a drill needs no commit, and `CreateView`/`SetAxis`/`SetPageSelection` are already unaddressed `ModelCommand`s that name their view. The gap is one level down from where it was looked for — **the mutation and its operand have different scopes**: the mutation configures the shared `_Drill` and is addressed to nobody, while the operand exists only inside one session's layout. So the fix is not a new mechanism but the `view: String` convention every other view-scoped variant already follows, resolved against a layout derived for that name (improvise-obtu). Two things travel with it: editing *inside* a drill grid from a browser does not follow from it, because `edit_or_drill` declines records-mode editing out loud before reaching a cell (improvise-fdk1, since closed, is the separate records-coordinate rename on the wire — `RenameRecordCoordinate`, gotcha 38 — and did not unblock drill editing); and closing it makes improvise-t219 live, `_Drill` being one shared view in `Workbook::views` that two clients drilling would contend for — the same masking relationship improvise-6uva has with improvise-up8g. Guarded by `an_aggregate_a_session_was_shown_does_not_resolve_against_the_hosts_view`, whose **control** is the load-bearing half — the same surrogate resolves once the host is moved onto that view, so a blanket `None` cannot pass. + +45. **One session loop, two transports — and the browser saves itself.** `Host` is `App` + `Sessions` + `handle(id, ClientFrame) -> Vec` and lives in `improvise_command::ui::host`, **not** in `improvise-ws-server`, which re-exports it. It moved for improvise-9x6: the worker hosts the same authoritative `App` over a `MessagePort` and cannot depend on tokio, and a second copy would have been exactly the mirror improvise-xks6 had just deleted for the frames — a drifting `Applied`/`Broadcast` match is invisible until a client sees it. `registry` stayed on the server, because a worker hosts **one** document. What the worker adds is only what a `MessagePort` differs in: **one client by construction** (the `Addressed` filter is honoured anyway, against a future `SharedWorker`), and **a handshake with no close** — `ClientFrame::Viewport` must still come first, but a pre-handshake frame is *dropped* rather than closing a connection that does not exist. + + **Saving is a frame, and dirtiness is read off the classification the effect already carries.** `effect::Save` is `ViewOnly` (gotcha 3) — correctly, since it writes *this host's* `file_path` through *this host's* `Storage` — so the derivation that builds `ModelCommand` from the model effects cannot name it, and for a while that meant no client on any transport could ask its host to save. `ClientFrame::Save` closes it (improvise-368h): a **request** carrying no path, because a client cannot see the host's storage, answered `ServerFrame::Rejected(NoDocumentPath | SaveFailed)` to the asker alone and **silently on success** — nothing moved, so no widget is dirty. It is a frame and not a `Command` for one concrete reason: as a `Command` it put a runtime-fallible operation inside a batch `ClientFrame::Commands` promises is all-or-nothing, so `[Save, SetCell]` reported a failed write and then applied the edit. It also fails the test every `Command` passes — it mutates no shared state and no client's slice, so `reduce_view` has nothing to fold and `Checked` has no basis to answer for. **And `MarkDirty` is gone.** It was a separate effect the *command* layer composed beside each mutation — reasonable on the key path, unreachable from anywhere else, because improvise-kmgs derived `ModelCommand` from the effects **one at a time** and so put `SetCell` on the wire without its companion (improvise-gv9r): a derivation correct per element and wrong for a set, whose symptom was that "save if dirty" never fired. `App::apply_effects` now raises the flag from the effect's `EffectKind`, which makes omitting it unrepresentable rather than validated — **declared** for `WholeWorkbook` (a replaced document differs from the file its path names, so a `LoadModel` that cannot read its path still raises it, costing one redundant write) and **observed** for `CellData` and `LayoutGeometry`, where `document_content`/`document_config` fingerprint the document before and after, so a write that leaves it as it was raises nothing (improvise-748f). The observation is read inside the loop, and `Save` clears the flag in loop order, so a batch that edits and then saves ends clean. The worker's persist trigger is therefore the **flag**, not a table of command variants: an observation rather than a second description of which commands change the document, for the reason `cursor_epoch` beats matching on `CycleAxis` (gotcha 24) and `document_generation` beats matching on `LoadModel` (gotcha 39). It is also strictly tighter — a refused batch raises nothing, and neither does a write that changes nothing, so the "was anything rejected" guard went with it. A browser still never *sends* `ClientFrame::Save`, and that is the answer rather than a gap: there is no file the user owns and no Ctrl-S semantics, the document *is* the OPFS entry, so the worker persists on its own behalf through the one existing `Save` effect. The cost that remains: `Storage` is whole-document (gotcha 41), so OPFS rewrites the file per accepted model command. + + **The async seam does not survive being boxed.** `OpfsFs::open`/`attach`/`refresh`/`commit_deletes` are on the struct, not on `Storage`. `App::storage` is a `Box` and a trait object can name none of the four, so **handing the backend to the `App` puts the async seam permanently out of reach**; the host must hold the concrete backend *and* give the `App` a `Storage`, which means sharing one. That is `impl Storage for Rc`, and it lives in **`improvise-io`** rather than in each worker (improvise-hmws): `impl Storage for Rc` is two foreign types in any other crate, so improvise-9x6 wrote a forwarding newtype that every future worker would have written again. `Rc` and not `Arc` because the one caller is a Web Worker and wasm32 has no other thread. **The flat root is no longer load-bearing**: `OpfsFs::list` joins `dir` like the other two backends, so `storage_contract` takes `dir` as a parameter and the three agree at *every* directory rather than only at `Path::new("")` where joining is the identity (improvise-vhcz). What the index holds is **documents, not entries**, and the split between its three writers is the rule: `attach` indexes nothing (what it creates is zero bytes), `Storage::write` indexes the name and un-queues any pending delete of it, and `refresh` rebuilds from OPFS skipping zero-byte entries — counting an entry it cannot measure as a document, since the one cause of that is another tab holding the exclusive handle (improvise-vvl3). + + **What no test here can see** has shrunk, and precisely. `harness/` (improvise-vvl3/xf1) runs `improvise_io::storage::contract::storage_contract` against a real `OpfsFs` inside a real dedicated Web Worker under Chromium and Firefox, and the `browser` job in CI can fail the build — so **`OpfsFs` is no longer compile-checked-and-never-run**, and two bugs fell out of its first execution. That is why `storage_contract` stopped being a `#[cfg(test)]` helper and became a public function returning `Result`: a browser must run *that* sequence, not a second transcription of it, and a failure has to travel out of a worker as a string. Still unseen: **Safari** (Playwright's WebKit has no OPFS, so xf1's criterion is met for two engines of three; `opfs.spec.js` skips on a runtime probe and an unexpected skip on any other engine is a failure). **`bindings.rs` is NOT in that list, and said it was for months.** `shell.spec.js`'s "the standalone deployment" loads the app with no `?ws=`, which spawns the worker, which imports the glue, which calls `bindings.rs::start` and `OpfsFs::open` — and the spec then asserts a grid draws, so the whole chain has run in CI since improvise-vvl3 (Chromium, and Firefox since 2026-09-20). The claim survived in **five** places (this line, the File Inventory, two `kb.pl` facts and improvise-bck's grooming note) because each was written before the harness landed and none was re-read after; the spec's own comment says the chain plainly. It is the reason to distrust a "never run" written in prose: nothing checks it, and `kb.pl`'s CI gate checks consistency rather than truth. What was already verified stays: `tests/post_message.rs` drives a real `Worker` over a real `App` on nothing but the JSON a `MessagePort` carries. Build note unchanged: the worker bundle needs `--enable-nontrapping-float-to-int` **as well as** `--enable-bulk-memory`, because it carries the whole `App` and so chrono's date arithmetic. + +46. **The page picks a peer, and the demo was built against an assertion.** `web/app.js` holds two transports and no protocol knowledge: `?ws=` selects `improvise-ws-server`, its absence spawns `new Worker('./worker.js?doc=&seed=', { type: "module" })` (improvise-djm). `type: "module"` is mandatory — `worker.js` `import`s the bundle — and there is **no `MessageChannel`**: a dedicated worker's own `postMessage`/`onmessage` *is* the port, so the ticket's channel wording was a date rather than a decision. The websocket path is **kept**, because the ws-server is how a TUI-hosted document is shared and nothing else in the browser can reach it; the cost is contained because the two differ only in *where strings come from* — handshake, fold, repaint and keyboard are all above the seam — and **the presence of the parameter is the whole selection mechanism**, a second knob being a second thing to get wrong. `#status` composes **two** facts, the transport's and the document's, because a single source let a repaint silently erase a connection error. **A worker that rejects in `start()` fires no `error` on the main thread** — `worker.js` catches its own — so the guard is a **first-frame watchdog**, deliberately *not* per-transport: "the peer never answered the handshake" is the one failure that means the same thing over a socket and over a port, and a check aimed at each peer's own failure modes would have missed exactly this one. + + **The demo is a model built to satisfy a check, not an axis flipped and hoped for.** `examples/demo.improv` leaves `Customer`/`Date` on None, so every cell is an `Aggregate` and nothing is editable — and the naive fix does not work either, since `Product × Region` by `Date_Month × Measure` still aggregates. `web/demo.improv` puts **exactly one category on each visible axis** (`Product` row, `_Measure` column, `Quarter` page, nothing regular on None) and carries `Profit = Revenue - Cost`, so the payoff is on one screen with no keystroke. **Settability is asked of `grid_of` as a `CellId` variant**, never re-derived from the axes — a second derivation could agree with itself while disagreeing with `render.rs::editable`, which is the whole point of ADR 002 — and the assertion was proved discriminating by pointing it at `examples/demo.improv`, where it names the aggregate at (0,0). **One category per axis is a property of the demo, not of the format**: when this was written `GridCache` slash-joined a multi-level tuple, so a second category on an axis put that known gap on the front page — improvise-anr9 has since closed it (gotcha 44), so what remains is the demo's own simplicity rather than a rendering limit. `web/*.improv` is un-ignored in `.gitignore` (`*.improv` is ignored repo-wide) and the CI assemble step copies it: leaving it out is an empty grid, not a 404 the page could report. One wart it now puts on the front page: a formula cell **is** a `Record`, so it is offered as editable and a write to it reverts on recompute — pre-existing and matching the TUI, newly visible. + +47. **Keys become commands in the browser, and the server must not know the keymap.** The user's ruling on improvise-bpq2, following from "the model is shared, the UI is not" — and it buys two clients different bindings over one document. What made it urgent is **not** latency. `a_key_resolved_on_the_host_moves_state_no_client_can_hear` shows nothing in the projection stream carries the host's own `ViewState`, so host-side resolution **diverges**; and `Host::handle` answers `ClientFrame::Key` with an explicit empty vec, so **an upstream key is a black hole, not a slow path** — every `KeyOutcome::Upstream` names a binding that does nothing at all in a browser. That test still holds and was not flipped: bpq2 closes the divergence by *avoiding* it. **The method is the ticket's** — enumerate the **bindings**, not the commands (`doctrine(binding_dump_diff)`) — and where a client cannot compose the `Command`, **enrich the projection** rather than moving resolution back. + + Three things that cost. **`GridExtent` on `GridCache`** — the whole grid, of which the labels are one window — because a viewport-indexed cache cannot tell "the last visible row" from "the last row". It is deliberately **not framed by the labels** (`fit_to_labels` leaves it alone) and its batch position is not load-bearing the way `RowFrame`-before-`Cells` is, since it addresses no position; the producer reads `GridLayout::row_count()`, and reading it off the projected ranges is **invisible in the TUI** (which builds at offsets zero, gotcha 44) and is exactly the browser bug. **The window is the client's own fact**, recorded by `Client::set_viewport` from what it announced — *not derivable*, since the row frame's own length is the window **clipped to the grid** and so differs precisely at the bottom edge where the scroll arithmetic matters. `Client::viewport` (associated, no recording) still exists and is the one that quietly turns cursor keys off; `web/app.js` does call `set_viewport`. **`ClientFrame::Commands(Vec)`** replaces the singleton, because one cursor key is `SetSelected` plus the offsets `viewport_effects` returns and a host that folds the selection but not the offset projects the wrong window. It is **answered whole, then applied whole** (improvise-955n): `CheckedBatch::against` checks the frame before anything runs, so there is no longer a moment at which part of a frame has landed and the rest has been refused, and `Applied::Rejected` is unreachable through `Host::commands` — still matched, because a host that wants the `Dirty` must name every case (gotcha 39). The contract is four clauses and **all four are now delivered** — clause 4 by improvise-yqpd on 2026-09-21, whose mechanism is `App::snapshot`/`restore` plus a `Sessions` clone taken in `Host::mutating_frame` and put back by `undo_frame`, deliberately *not* inside `apply_effects`, which runs on every keypress. **This paragraph said it was undelivered for a day after it landed**, and an orchestrator brief repeated that from here, so read the clause list as a claim to check. A delivered clause 4 is also what makes the CLIENT's rollback well-defined (improvise-nxah): a refusal means the host applied none of the frame and restored every mirror, so a client keeping its optimistic fold is the only thing left out of step. What rolls back there is `ViewState`, never a cache — a `Command::Model` is a no-op in `reduce_view`, so the optimistic fold never reached one. The old text, kept because the hazard it names is still the right shape to fear: one basis per frame, checked once, applied in order with each command seeing the last one's state, and all-or-nothing. was that the fourth was open because `App::apply_effects` aborts without undoing (gotcha 21 — an abort is not a rollback), so `[RemoveItem("North"), SetCell(the North record)]` would land the removal, resolve the write to nothing, and report the frame accepted having applied half of itself — partial *and* silent. That is what `yqpd` closed, and it closed it **outside** `apply_effects` for the reason the old text implies: the abort lives on the keypress path and must stay cheap. The old note that it was *unreachable* from any client here, the only multi-command producer being `keys::commit`, still reads correctly as a caution — **reachability is a reason not to rush a fix and not a reason not to notice one**, the lost update `CheckedBatch` fixed having had the same reachability argument and none of the documentation, which is how it shipped. + + Two corrections to the ticket's own gap list, both worth more than the code. **Panel cursor movement was never a gap**: improvise-edp had already made `viewmodel::flatten_cat_tree` the one home both ends call, so it is the *shared* derivation, not a second one — worth re-checking the "known gaps" in other old tickets against what has landed since. And the "unmodified characters only" rule was **mis-scoped, not wrong**: it was about `KeyPattern::AnyChar`, which is bound only in the text-entry modes, so `Ctrl-d` in Normal has an exact binding and mirroring it is a faithful subset. Each resolver mirrors its `Cmd` **line for line, including where siblings disagree** — `Move` and `MoveTileCursor` emit even when they clamp, `MovePanelCursor` returns nothing when the cursor does not move. Tidying that into one rule would make the client a second derivation, which is what `tests/lockstep.rs` exists to rule out. What still needs the host is **one list**, `resolve::NEEDS_THE_HOST` in `improvise-protocol/src/resolve.rs` (improvise-uln5) — this line named a list in `keys.rs` and a test called `the_normal_mode_keys_that_still_need_the_host_are_these` for some time after both had gone. Three tests hold it, one per direction: `the_command_names_that_still_need_the_host_are_these` (no **false** entry: each name is bound and really is silent), `a_help_row_the_browser_cannot_run_is_drawn_marked` in `wasm-client/render.rs` (the help overlay **marks** those rows, improvise-uln5), and `every_silent_binding_is_on_the_list` (every binding of **every mode a browser can reach** acts, declines out loud, or is on the list — improvise-oyaa, improvise-s2gb). The last replaced a check read off the help rows, which could not see a binding no help row names. Closing a gap therefore means deleting its entry, and the first test fails until you do. + + **What the browser client decides before a frame leaves it** (all in `improvise-protocol`, so both ends share it). Five rules, each bought by a bug: + - **Every client cache read goes through `projected_at`**, which subtracts the frame's own origin (`AxisFrame::first`) before indexing. A position outside the window the cache holds is **unknown**, never the cell at the same cache index (improvise-7mro) — once the client scrolled ahead of its cache, an index read answered a different cell's value. `unprojected` names that state, and a cell op on it waits rather than guessing. + - **The keyboard commit writes `ClientState::opened`**, not the cursor. `OpenedCell` captures the record, the `OpenedTarget` (`Value { measure }` or `Coordinate { category }`), the record's version, the `document_generation` and the view **at open**, so a commit lands on the cell the editor opened on however the cursor moved since (improvise-8xq0), and claims the basis it was **opened** against rather than one read at commit, which silently overwrote a value another client wrote while the editor was open (improvise-xz2z). A retry after a crossing `LayoutMoved` must match the **whole** `OpenedTarget`, not the record alone, or it wrote whatever cell was under the cursor by then (improvise-8phw for values, improvise-9ww8 for renames). + - **`refused_here` exempts a captured editor** (improvise-3h9w): with `opened` set, `commit-cell-edit` is answered by the capture, so the cursor's current cell cannot refuse a commit aimed elsewhere. + - **A decline that must stop a binding is `Step::Declined`**, not a `SetStatus` beside the binding's other steps: Enter's welded `clear-buffer` otherwise still ran and threw away the typed text (improvise-joed), and a touched editor over a declined cell went `Upstream` with no word at all (improvise-ik6l). + - **`ClientState::notify` queues a client-side notice at the tail of `in_flight`**, as if it were a sent frame, so a rebase that rebuilds the view from `settled` and replays `in_flight` replays the notice too instead of erasing it (improvise-n0pb). `settle_status` runs on **every** path that rebuilds the view (bootstrap, accept, refuse, relay), and assigns a message the editor open as it arrives, so an old message loses its owner once that editor closes (improvise-0dbk). `EditorSession`, the key it is owned by, lives in `improvise-protocol/src/lib.rs` beside `ViewState` so both front ends ask it (improvise-su4u, improvise-1x9h). + +48. **The browser JavaScript is argued *and now partly executed*, and the two rules it can break are both about *when*.** `web/app.js` and `web/worker.js` were syntax-checked, `.d.ts`-checked for call arity and read — never run — and every bug in them was found by reading. `harness/tests/shell.spec.js` is the first execution of either: it runs `app.js` under Chromium against a mocked peer, and it exists because PR #21 declined to add regression tests for its own close-listener fix on the ground that there was nowhere to put them. That decline is only honest because the assertions live there now, and **a change to `web/*.js` belongs in that spec directory** rather than in a second ad-hoc surface beside it. Playwright rather than a headless-DOM shim because what goes wrong in these files is a real Worker, a real WebSocket, real OPFS and real event ordering, none of which jsdom has. Two classes of bug still follow from *when*, both found by review. **An `async` `message` listener does not serialize.** `worker.js`'s `handle()` awaits `server.flush()`, so the event loop may start the next message while an earlier one is suspended, interleaving two `on_message`/`flush` pairs and letting a later frame's replies **overtake** an earlier frame's — which gotcha 33 answers by *dropping* the overtaken update, not by replaying it. Every frame is therefore chained onto **one tail promise** and the listener is deliberately not `async`; that also turns "frames that arrived before `start()` finished" into an ordinary case of the ordering rule instead of a drain loop, which had the same race twice over because `server` was assigned *before* the loop ran. **A watchdog must be disarmed, not pre-empted.** The handshake watchdog exists for one condition — the peer never answered — and every *loud* failure (worker `error`/`messageerror`, socket `error`) has already written something more precise; the timer then fired 8s later and replaced it, so `setShellFailure` clears it. Its neighbour, the socket's `close` listener overwriting a precise `error` with "disconnected", **is fixed** (improvise-taxx, PR #21) — and the ticket's own proposed mechanism was wrong: `readyState` cannot discriminate, an `opened` flag is the only thing that can. PR #21's *other* half is the one that keeps getting lost: it routes **both** close arms through `setShellFailure`, which disarms the watchdog, so a genuine mid-session disconnect is **not** overwritten eight seconds later. A bug report said it was, measured it honestly against a pre-#21 tree, and then "confirmed" it survived #21 by simulating the `opened` flag **and not the routing** — a simulated fix that is not the fix you named reproduces a bug the real code does not have, and reads exactly like a confirmation. The ticket was filed and closed invalid; the property is now asserted by `a socket that opened and then dropped reads as disconnected, and stays that way`, which runs past the 8s window, rather than argued in a comment that was wrong twice. + + **What the harness still cannot see**, and why the argued-not-tested discipline stays: the grouped and multi-level DOM, the CSS, and the `ResizeObserver` extent arithmetic. What a careful read established about the last of these — the observer cannot feed back into itself (`#grid` is a flex child with `overflow: auto`, so `replaceChildren` cannot resize it) and `ROW_PX`/`COL_PX` under-estimate, so the worst case is a row of over-fetch — is still a read and not a test. + + **A third thing no test here can see, which is why the docs must be read as code.** Three comments in one range said "not built yet" about work that had landed *in that same range* — `README.md`'s browser section, `app.js`'s keyboard note, and `Host::handle`'s `Key` arm — each because a follow-up commit updated **one copy of an argument and not its siblings**. When a fact is stated in more than one place, changing it is a grep, not an edit. + + **An evidence heading may be module-qualified, and must be when its leaf is not unique.** `scripts/check-comment-runs.py --check-evidence` takes its Rust names from `cargo test --workspace -- --list` minus the `--ignored` twin, as **full module paths**, and **refuses** an ambiguous leaf naming every candidate rather than warning. Six leaf names collide in this workspace. Note neither half suffices alone: rename one of a colliding pair and the leaf is unique again and resolves silently to the survivor, so qualification is the fix and the refusal is only what forces someone to write it (improvise-rm4s). The residual, filed: `--list` does not name the binary, so two crates sharing a module path are indistinguishable (improvise-sxsf) — **measured latent** (1444 names, zero collisions) and the proposed per-package fix **declined twice over**: 18× the cost (45.2s against 2.4s, and `--check-evidence` needs the `--ignored` twin, so 22 cargo invocations), and *a package is not a binary* — `improvise` alone has five integration test binaries beside its lib, and an integration test's top-level `fn` lists with **no module path at all** (60 of 1444 names are bare), so `-p improvise` prefixes all six identically. Per-**target** is the only shape that would close it. + +49. **A websocket handshake is exempt from the same-origin policy, and that is a server-side problem.** There is no CORS preflight, so any page a user visits can open a socket to `improvise-ws-server` *from that user's own allowed address* and read and rewrite every document it holds — cross-site websocket hijacking (improvise-kc7n). **No network control can see it**: an IP allow-list, a private network and loopback binding all answer "may this **client** reach the endpoint", and the honest answer is yes; only `Origin` says what is **driving** the client. **Loopback is the least protected case, not the safest** — no proxy in front of it, and loopback is not a boundary a browser enforces. `src/origin.rs` is the whole decision and `transport::admitted` its one caller; it refuses the **upgrade** with a 403 rather than accepting a socket and ignoring it, and `serve` **takes** the policy so a host cannot forget to apply it. Three arms are decided rather than defaulted: **loopback allowed** (the shipped local recipe serves its page from `http://localhost:8080`, so the one case worth optimising for must need no flag), **absent allowed** (no browser omits `Origin`, so absent is a non-browser client an address check already answers — and refusing it breaks `tests/socket.rs`, the repo's own instrument), and **two refused** (`HeaderMap::get` is first-wins, so a proxy that *appends* rather than sets leaves two, and answering from the first lets the second be anything). The deployed Pages origin is deliberately **not** hard-coded: it can only reach a plain-`ws://` server through a TLS proxy anyway, so one flag is an honest cost on an already multi-step path where a baked-in default would be a permanent hole for every other deployment. The residual hole is **stated rather than guarded** — a proxy that *strips* `Origin` turns the check off, which is why the proxy's own allow-list stays in `README.md` as defence in depth. **None of this is authentication**: it answers what page is driving a client, never who is connecting, and improvise-kpnr is still open. It correctly does **not** move to `improvise-command` beside `Host` (gotcha 45) — a `MessagePort` has no handshake and no headers, and a worker and its page are the same origin by construction, so this is genuinely transport-only and not a mirror waiting to drift. + + +50. **A composed TUI row measures in cells and names which side yields.** `draw.rs::fill_line` takes a `Keeps`, because the three rows that share a line genuinely disagree about it: the title bar keeps the **right** (the `?:help :q quit` escape hatch outranks an unbounded document name — improvise-t14d), while `minibuffer_line` and `draw_status` keep the **left**, which is what they already did, since right-priority there would let a long status message eat the text the user is editing. So the ticket's "give the right segment priority" was correct only for the title bar, and a global rule would have been the wrong shape. Widths go through `ui::text::truncate`, never `str::len` — and the ticket framed *that* half wrongly too: the title's own `·` separator is two bytes and one cell, so a byte count mis-padded **every** document, not only one with a non-ASCII name. + + **One key table on screen, and it is the one derived from the keymap** (improvise-rr8n). `TileBar` now advertises no keys at all: its literal read `Ctrl+↑↓←→ to move tiles` where those keys are bound to `enter-tile-select`, and the derived status line is a strict superset (`Space`, `h`/`l`, `Tab` appear only there). The ticket asked for the opposite — rewrite the literal, suppress the derived line — and was inverted deliberately, because suppressing a derived superset to preserve a literal that had already rotted keeps the stale table and deletes information. The **import wizard goes the other way**, and the asymmetry is the rule rather than an exception: its status line *is* derived, but from the **mode** keymap while the real keys are per-step inside `HandleWizardKey`, so there the derived table is the one that lies and `draw_status` suppresses the fallback for that one mode. **Keep whichever table is derived from the thing it describes.** Two blind spots make this class invisible: `every_hint_key_resolves_in_its_mode` **cannot see a string literal in a widget** (the same gap improvise-wk18 names for `help.rs`), and the widget's own characterization test renders at **80 columns**, where the hint does not fit and is never drawn — so the one test pinning that output was structurally incapable of seeing either hint. + + **A scroll is measured where it is drawn** (improvise-gtn7.1). The keypress path counted `visible_cols` from `term_width` while `draw_content` gave the grid `term_width` minus the 32-column side panel, so with any panel open the cursor scrolled to a column never drawn. Now `viewport::grid_width(term_width, side_panel_open)` is the one answer, reached through `App::grid_width`, and both `draw_content` and `App::cmd_context` read it; `App::side_panel_open` is the one answer to whether a panel took its column. The tile bar had the same shape: `first_tile` chose its scroll against a width that began one or two columns before where the tiles were drawn, and both now start at `tiles_left(start)`. The grid bug was latent from `b5418f2e`, when `compute_visible_cols` took `term_width`, and was exposed by `50e4a372`, which opened the category panel by default. Still open: columns of mixed width, where a `visible_cols` count taken at the old offset is applied at the new one (improvise-gtn7.2). + + +51. **The command language reads pipes, refuses surplus, and checks names when it applies (improvise-31oe.2, 31oe.3).** Reasoning: `context/code/cli.md` §14 and §15. (a) A `:` argument is read through `improvise_formula::quote`, so `create-view |By area|` names `By area`; a **surplus argument is refused** rather than ignored, and `cmd::core::only_when_omitted` gates the *cursor fallbacks* of `clear-cell`, `yank`, `paste` and `drill-into-cell` so a coordinate typed by the user is never overridden by the cursor. (b) `set-axis`, `set-page`, `toggle-group`, `hide-item`, `show-item` and `switch-view` return `RequireKnown(Needs)` (`effect.rs`) ahead of the real effect, built by `effect_cmds.rs::unless_unknown`. It is `ViewOnly`, writes nothing, is **off the wire** (listed in `vocabulary_is_complete.rs`) and checks the **document at apply time** — a `:` line builds every segment before any runs, so a check in `execute` refused `create-view Second . switch-view Second` — then refuses and aborts the batch (gotcha 21). (c) A refused name therefore stops the rest of a `:` line, while headless `improvise cmd` rebuilds the context per segment and runs each as its own batch (improvise-31oe.14, open). + +52. **A date column imports as one hierarchical category, and the import refuses what it cannot keep (improvise-8jk6.3, 8jk6.6, d0n6).** Reasoning: `context/code/formula-and-io.md` ("Import: …") and `cli.md` §13. The default `DateShape` nests Year > Quarter > Month (the chosen levels) in ONE category named after the column and opens folded to its years; `DateShape::Split` is the old flat split, chosen by `--split-date ` headless or `s` on the wizard's date step. `Plan::spelling_clash` refuses a raw, unparsed date spelled like a generated item or group (`2024`, `2024-01`) because the two would silently merge — asked at the wizard's ConfigureDates step **and** again at build time, so a headless import cannot reach it either. Headless, `--strict` turns a partial skip into a refusal before the build, and a folder whose CSVs are all header-only (no records) is refused as an import that kept no record. + +--- diff --git a/context/repo-map.md b/context/repo-map.md index b2888ace..b967a21f 100644 --- a/context/repo-map.md +++ b/context/repo-map.md @@ -156,513 +156,101 @@ Region=West, Measure=Revenue = |pending| # pipe-quoted text --- -## Gotchas (read before editing) +## Gotchas (digest — read before editing) -1. **Records mode is wide format.** A row is one *coordinate tuple* — the cells sharing everything but `_Measure` — and the columns are `[coordinates…, **`+cat`**, raw measures…, `+`]`: a column per regular category, then the column that creates a **new category** (improvise-2zq, sitting where the column it creates will appear), then a column per raw measure, then the column that creates a new measure (improvise-c5v). Consequences: two measures on one coordinate are ONE row; editing a category column moves **every** measure cell in the row, not just the one under the cursor; `_Dim` in the synthetic key carries the column **index**, not a label, because a category and a measure may share a name — resolve it with `GridLayout::record_col` into a `RecordCol`. Formula targets are not columns (nothing to type into them). Drill pending edits are keyed by `(row, RecordCol)`. - **`prune_axis_items` takes a predicate, not a flag vector.** `prune_empty` hands it `impl Fn(usize) -> bool` over the *data* index (group headers consume no index). The domain is derived from `items` itself, so there is no parallel length a caller could get wrong — the earlier `&[bool]` + `debug_assert` pair could disagree between debug and release if a third caller ever sized `keep` from something other than `row_count()`/`col_count()` (improvise-n3wt). **It re-marks the survivors rather than filtering the markers** (improvise-e265): runs nest, so dropping an item can leave a header with nothing under it or a `GroupEnd` closing a run that was never reopened, and no per-marker filter can tell. `remark` instead emits, before each kept item, exactly the headers and `GroupEnd`s that make the output's open runs equal the input's at every level — each survivor keeps exactly the runs it was in. Reasoning: `context/code/core.md` §6. **`prune_empty` shares one `GridEval`** across the whole row×col emptiness test, and **a grid evaluates only through `GridLayout::evaluator`**, which builds the fold memo from that layout's own `none_cats` and hidden items — so no other grid's memo can reach its cells or totals (improvise-metf, improvise-opbo; the memo itself is improvise-h6z1, `core.md` §14). - **`GridLayout::new` does not prune; `with_frozen_records` does** — and that split has now produced **four** bugs: `view_frame`, `project::grid_in` (PR #78), and `persistence/export.rs::render_csv` (improvise-iot6), which wrote rows *and columns* the grid does not draw for as long as CSV export has existed. Every remaining `GridLayout::new` call site in the workspace is test code — verified — so a new production caller almost certainly wants `with_frozen_records(m, v, None)`. +One or two lines each: the rule, and the trap. **The numbers are stable and cited across the repo; a +citation of `gotcha N` means entry N in `context/gotchas.md`, which holds the full argument, +the incidents and the ticket ids.** Read the long form of every gotcha your diff touches. Counts and +"closed/open" claims in the long form are dated snapshots. -2. **Effects never touch `std::fs`.** Every document read or write goes through `App::storage: Box` (improvise-fuj, closed). Two writes still go around it: `App`'s autosave and the headless final write call the path-taking `persistence::save` (improvise-31oe.11, open). The host picks the backend: `App::new` supplies `PhysicalFs`, `App::with_storage` takes any other — the browser worker supplies OPFS (improvise-i34, closed), and tests use `MemoryFs`. `improvise-io` keeps path-taking wrappers (`save`, `load`, `export_csv`, `parse_csv_with_warnings`) for the native CLI, each delegating to a `_to`/`_from`/`_bytes` variant that takes the backend or the bytes. -3. **Every effect declares an `EffectKind`, and there is no default.** `ViewOnly` never reaches other clients; `Model(CellData | LayoutGeometry | WholeWorkbook)` does (improvise-mae). A new effect that forgets to classify itself fails to compile — that is deliberate, since the failure mode is silent: a mis-declared `ViewOnly` edit would simply never reach connected clients. Watch two traps: the *View* lives inside `model_state.workbook`, so "touches model_state" does NOT mean "is a model change" (axis/page/format are `LayoutGeometry`, cursor and scroll are `ViewOnly` — see improvise-8uz; and `SwitchView`, `ViewBack` and `ViewForward` moved from `LayoutGeometry` to `ViewOnly` with improvise-jytm, writing `view_state` and nothing else, which is what took the latter two off the wire — but a sound derivation is not a guard: only `SwitchView` had one, and review showed reclassifying either sibling back was silent across the whole suite); and `Save`/`SaveAs` touch `model_state` for bookkeeping only and are `ViewOnly`. `AddCategory`/`RemoveCategory` are the third trap and go the other way: they look like data edits, but both go through `Workbook`, whose `on_category_added`/`on_category_removed` rewrite **every** view's axis map, so they are `LayoutGeometry`. The two "an effect declaring X actually changes X" tests cannot separate them — a category change moves model data *and* view config, so it clears either bar; `structural_effects_declare_their_kind` pins the declaration itself. A third trap, now guarded: **`EffectKind` is too coarse to check a *mapping*.** `ui/reduce.rs::model_effects` sends each of the 26 `ModelCommand` variants (counted on `c1772ca9`; gotcha 38) to one effect, and every inverse pair — `HideItem`/`ShowItem`, `AddItem`/`RemoveItem`, `CreateView`/`DeleteView`, `SetCell`/`ClearCell` — declares the *same* kind, so swapping three of them at once left the suite at 918 passing while three wire messages did the exact inverse of what they said. The mapping is name-for-name by construction, so `every_model_command_names_the_effect_of_the_same_name` asserts that, reading the type name off the effect's derived `Debug` (`Effect` is a trait object, so `type_name` is out of reach). A variant whose effect must be named something else is a variant that stopped being derived — it should fail there and be argued about. Their reach is wider than the name suggests: `Model::remove_category` also drops every formula targeting the category, which is why `Dirty::of(LayoutGeometry)` reports `formulas: true` — see gotcha 35. -4. **Commands never mutate.** `&CmdContext` is read-only; return `Vec>`. If you want to touch `&mut App`, add or extend an Effect. -5. **`CellKey` is always sorted.** Use the constructors; equality and hashing rely on canonical form. -6. **No `Model::add_item`.** With a `Workbook` in hand use the facade: `wb.add_item("Region", "East")?` / `add_item_in_group` / `remove_item` (improvise-tvt). With only a `Model`, go through the category: `m.category_mut("Region").unwrap().add_item("East")`. -7. **Virtual categories.** `_Measure` items = explicit items ∪ formula targets. Use `measure_item_names()` / `effective_item_names("_Measure")`. `add_formula` does *not* add items to `_Measure`. Formula-target lookup also falls back into `_Measure` via `Model::find_item_category`. Cell aggregation goes through `apply_agg`, formula `SUM(…)`/`AVG(…)` through `eval_agg` — they differ on empty input (absent vs. A third rule joins them for **non-numeric** values: `aggregate_cell_values` reduces text by **unanimity** — one matching value, or several copies of the same value, is that value; differing values, or text among numbers, is `CellValue::Error("mixed")` (improvise-l4qx). The **all-numeric branch is checked first**, so `Count` of a lone number is still `1` and not the number — that ordering is the whole difference between this rule and "a single value passes through". What the rule forbids is *dropping*: `filter_map(as_f64)` used to, and since `_Index`/`_Dim` are on `Axis::None` in every default view that made **every text cell in every document render blank**, including the pipe-quoted text `.improv` carries as a first-class value. `evaluate_aggregated` and `aggregate_raw` share the helper, so a formula ref over mixed values now errors `"mixed"` rather than silently summing the numeric ones. `0` for Sum/Count, error for Avg/Min/Max). `Model::evaluate_aggregated` takes `Aggregation::{Exact, OverHidden}` rather than `none_cats: &[String]` (improvise-5jm5) — the list was only ever tested for emptiness, and the guard against that shape returning is the **signature**. `recompute_formulas`/`aggregate_raw` keep the list because they walk it. -8. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full `f64`. Never feed a rounded value back into eval. -9. **Formula eval is fixed-point.** `recompute_formulas(none_cats)` iterates until stable, bounded by `MAX_EVAL_DEPTH`; circular refs converge to `CellValue::Error("circular")`. `App::new` runs it before the first frame so formula cells render on startup. -10. **Drill into formula cells** strips the `_Measure` coordinate when it names a formula target, so `matching_cells` returns the raw records that feed the formula instead of returning empty. -11. **Minibuffer keymaps come from `TextEntry`.** The eight text-entry modes are built by the `TextEntry` builder in `improvise-protocol/src/keymap.rs` — seven by `TextEntry::new`, and `RecordsEditing` by `TextEntry::with_parent` off `Editing`'s map, which is why a count taken off the `::new` call sites reads seven. It binds the shared spine (AnyChar→`append-char `, Backspace→`pop-char `, Esc→`[clear-buffer , enter-mode ]`) and welds every commit key to its `clear-buffer` via `.commit(key, cmd, args)` (improvise-w3q). `TextEntry::new` takes the `AppMode` and reads `` off its `MinibufferConfig`, so the buffer the prompt names and the buffer the keys edit are the same string by construction (improvise-k8h) — the keymap never spells a buffer name. `.rebind` exists only for CommandMode's Backspace. `SearchMode` is hand-written on purpose — no commit key, and `exit-search-mode` owns the query. `EnterMode` clears the incoming mode's minibuffer through `write_buffer` too — building `SetBuffer` directly there was unreachable only because no shipped `AppMode` is keyed `"search"`. `MinibufferConfig::buffer_key` is a `String`, not `&'static str`: a borrowed-'static field makes `Deserialize` unimplementable, and every text-entry `AppMode` variant carries a `MinibufferConfig` (improvise-a2x). A buffer named at runtime is read with `cmd::core::read_buffer` and written with its mirror `write_buffer` — "search" is not in the buffers map but its own `ViewState` field, and those two functions are the only code that knows it (improvise-gdl). -12. **Panel keymaps come from `PanelKeymap`.** The three panel modes share a spine (Esc/Tab→Normal, arrows and j/k→`move-panel-cursor `, every panel's toggle key, `:`→command mode) built by `PanelKeymap::new(panel)`; the keys that act on the panel's contents arrive through `.bind`/`.bind_args`/`.delete` because they vary in shape, not just command name (improvise-4pg). `Panel::ALL`/`name()`/`key()` in `ui/app.rs` drive the toggle keys for Normal and all three panels, so a fourth panel reaches every keymap from one `ALL` entry. `name()` must spell what `parse_panel` accepts. -13. **Prefix keys are stored as an address, not a keymap.** `ViewState::transient_prefix` holds the `(KeyCode, KeyModifiers)` that opened a prefix; `KeymapSet::prefix_keymap` resolves it back to the sub-keymap, walking parents so `RecordsNormal` reaches Normal's `g`/`y`/`Z` (improvise-jc8) — that is the command-layer wrapper; the resolution itself is `Keymaps::prefix_keymap` in `improvise-protocol/src/keymap.rs`. `ViewState` must stay free of `improvise-command` types — it is destined for the protocol crate, which `improvise-command` will depend on. Resolution assumes prefixes are **flat**; `prefix_keymaps_are_flat` fails the build if a nested one is added. -14. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the mode; Normal + `search_mode=true` maps to `SearchMode`. Minibuffer modes bind Enter/Esc as `Binding::Sequence` so `clear-buffer` fires alongside commit/cancel. Hints are checked against the keymaps: `every_hint_key_resolves_in_its_mode` (`command/keymap.rs`) splits each `AppMode::hint_text()` and requires every key it names to resolve in that mode's map, because `only_quit_has_no_hint` asserts only that a hint is *present*. It checks which keys are named, not what they do. Its tokenizer knows the hint dialect explicitly (named keys, `Ctrl-`, `F/C/V` alternatives, character runs like `hjkl`/`[:]`, the em-dash prose tail, CommandMode's `:q` ex-commands) and panics on a token it cannot classify rather than skipping it. -15. **Registry names come from prototypes, not strings.** Every `register*` call reads the name off a prototype `Cmd` instance, so a typo is impossible (improvise-9cn) — so no `register*` call spells a name. Name literals do remain in `registry.rs`, and only tests guard them — the keymap test `every_bound_command_name_is_registered` for the `Move` literals (all bound; `registered_names_are_unique` covers them too), and for the four aliases `registry.rs::every_alias_resolves_to_a_registered_command` (a hand-copied list, not the table) plus the `parse.rs` tests `alias_add_cat_resolves_to_add_category`, `alias_formula_resolves_to_add_formula`, `alias_add_view_resolves_to_create_view` and `alias_q_bang_resolves_to_force_quit` — the `Move` prototypes carry `cmd_name: "move-selection"` / `"scroll-rows"` / `"page-scroll"` (the four `jump-*` names go through `reg_jump!`), and four `r.alias` calls spell an alias and its target (`add-cat`, `formula`, `add-view`, `q!`). What types can't reach: five commands name themselves off a field (`Move` off `cmd_name`, `ViewNavigate`, `TileAxisOp`, `ToggleGroupAtCursor`, `CommitAndAdvance`), so a prototype can name the wrong *variant* and still compile. That shows up as a **duplicate** registration, not a mismatch — the wrong-variant entry collides with its sibling and the sibling's name vanishes. `registered_names_are_unique` catches it; `every_bound_command_name_is_registered` catches the keymap side, and checks **both** halves of `dispatch`'s `registry.interactive(name, args, ctx).ok()?` — the name resolves *and* the args parse — because improvise-s13 moved `enter-mode ["export-prompt"]`/`["formula-edit"]` out of the nullary class, where a typo'd mode name compiles into a dead key. It builds the `CmdContext` in each keymap's own `AppMode`, since `commit-buffer item` reads `ctx.mode`. -16. **`effect_cmd!` macro** wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written `Cmd` once there's real decision logic. Its counterpart on the effect side is `simple_effect!` in `ui/effect.rs`, which declares a one-field newtype that writes one place on the `App`. Its **kind argument is mandatory** — `simple_effect!(Name(T) => |app| place, EffectKind::ViewOnly)` — even though every call site so far answers `ViewOnly`: hard-coding it reintroduced exactly the default gotcha 3 does without, since `$place` is an unconstrained expression and a one-line `active_view_mut().number_format` setter would have reported `ViewOnly` with the suite green (improvise-8fy). Exactly six wrappers return `unless_unknown(needs, effect)` rather than the bare effect (gotcha 51): `set-axis`, `set-page`, `toggle-group`, `hide-item`, `show-item` and `switch-view`. The rest return bare effects — `delete-view` (whose `DeleteView` effect refuses an unknown view itself), `add-item`, `add-items`, `set-cell`, `create-view` — and `remove-formula` checks in `execute`. Likewise, mode entry has no per-mode shim commands — bind `enter-mode ` (improvise-s13); the name is decoded by `parse_mode_name` in `registry.rs`, and `EnterMode` clears the mode's minibuffer on the way in. -17. **Float equality inside formula eval has one epsilon, and the div/0 guard shares it.** `FLOAT_EQ_EPSILON = 1e-10` in `model/types.rs` is used by `=`/`!=` **and** by the division-by-zero guard (`rv.abs() < FLOAT_EQ_EPSILON`), in both `eval_expr` and its aggregated twin — six uses across the two evaluators (a div/0 guard plus the `Eq` and `Ne` arms in each), one constant. This gotcha previously recorded the opposite (`=`/`!=` epsilon, div/0 exact) and was stale; the mix is gone. Why an epsilon at all and why this one: `context/code/core.md` §4. -18. **`IndexMap`** preserves insertion order for categories, views, and items. Persistence relies on this — and so, since improvise-shyt, do `DataStore::matching_cells`/`matching_values`: the secondary index **narrows** the candidates, then `matching_positions` resolves each to its position in the primary `IndexMap`, sorts, and reads back. **An index whose values are `HashSet`s is for O(1) lookup, never for iteration.** Walking it gave a different order in every process, which was not tidiness. Records-mode rows reordered between sessions; `SUM`/`AVG` drifted in the last bits because `f64` addition is not associative (`Min`/`Max`/`Count` and the text-unanimity rule of gotcha 7 are order-free, so they were never affected); and worst, the **drill snapshot** froze its record list in hash order while `drill_state.pending_edits` are keyed by *row index* (gotcha 1), so the same drill opened in a second process addressed **different records** — silent misdirected writes, and nothing in the tracker knew. Measured at 18,000 cells: +20% worst case on `matching_values`, and `matching_cells` 7% *faster*, sorted positions resolving `CellKey`s with better locality. `DataStore::live_records` was the open choice and is now decided (improvise-2yc2): it walks `cells` and projects insertion order onto records, deliberately paying O(cells) rather than O(records), because `cells`' order is already the answer `matching_positions` gives and a second answer to "what order is this store in" is the disease. It is derived per call rather than memoised on `RecordEntry`, since a stored position is a copy and a record deleted and re-created would hold its old one. Note what is *not* on offer: an unordered return — every iterator has an order, and a type that merely disclaims one still hands the caller hash order, which is the guard living one level away from the hazard and the shape this whole class of bug returns through. **The second real instance was `import::analyzer::analyze_records`** (improvise-7yx), which built its distinct values through a `HashSet`: `build_model` adds items in exactly that order and `IndexMap` preserves it, so the *same CSV imported twice* produced two different `.improv` files and two different row orders. Now first-seen order, pinned by a five-iteration test because each `HashSet` gets its own `RandomState`. Also answered here since: an **empty `partial`** matches every cell vacuously (improvise-hkt6) — it used to index `sets[0]` and panic, unreachable only because both callers guarded. **A group's fold reads through the same rule.** `Model::fold_stored` folds a collapsed group's stored values once, over every record the group covers, through `DataStore::spanned_values` → `spanned_positions` (improvise-yb4b) — one scan, not a lookup per member. `spanned_positions` chooses **by index-set size** between narrowing once by the base key and narrowing once per covered item of the smallest span, whichever visits fewer (improvise-jq0m), and **both arms return store order** — sorted positions, as `matching_positions` does. That is load-bearing, not tidy: a fold that returned hash or per-item order would reintroduce the `SUM` drift above, and the choice between the arms would then be visible in the last bits of a total. **A partial as wide as the widest stored cell needs no narrowing** (improvise-felo.13): `DataStore` keeps `arities` (how many stored cells have each width), and `exact_positions` answers a strictly-sorted partial at least that wide with one probe of `cells`, since only an equal cell can match; anything else falls through to the index. The answer is the same sorted positions. -19. **Commit paths for cell edits** split across `commit_cell_value` → `commit_regular_cell_value` / `commit_plain_records_edit` in `command/cmd/commit.rs`. Keep the synthetic-records branch in sync with the plain branch when changing behavior. **Which cells refuse an editor is one function, asked at both ends of an edit**: `cmd::core::uneditable(model, layout, row, col)` is the shared decline for **opening** an editor (`EnterEditAtCursor`) and for **committing** one (`CmdContext::committable_cell_key`) — a fold or other aggregate, a formula's result, or a `CellId::NonItem` (improvise-enz6, improvise-gk51). It reads the cell's `CellId` through `project::cell_id`, the same derivation the projection uses, and the browser's `resolve::declined_at` matches on the projected `CellId`, so the two front ends refuse the same cells. Asking only at open is the bug enz6 was: Enter-to-advance moved onto a collapsed group's row and committed a write to the group coordinate. **`CellId::NonItem { category, item }` is a stored value whose coordinate names no item of its category** — records mode can draw one, and no write can address it, so it is projected as its own variant rather than as a `Record` that the host then refuses (gk51 was exactly that mismatch). Its one wording is `improvise_protocol::non_item_refusal`. -20. **Minibuffer commits go through one command.** `commit-buffer ` (`CommitBuffer { kind: CommitKind }`) owns buffer→trim→bail-on-empty→dirty→status for all three; the varying parts are methods on `CommitKind`. The add effect is emitted **last** so its failure message wins over the command's optimistic status (improvise-6t3). `CommitKind::Item` carries its target category, resolved from `AppMode::ItemAdd` by `parse_commit_kind` at registry time — so `commit-buffer item` is not constructible from a script line. -21. **An abort stops the chain, never the bookkeeping — and ordering is what separates them.** Three effects set `abort_effects`, and they are not all saves: `Save` and `SaveAs` on a failed write or a missing path (improvise-8tk — `SaveAs` was the arm left out of the original fix, so `:w /bad/path . q` quit on a failed write), and `AddCategory` when `Model::add_category` refuses, because what follows assumes the category exists — from the `+cat` column (improvise-2zq) the rest of the batch would advance the cursor and re-open an editor on a column that was never created. A `Binding::Sequence` flattens into one effect vec, so the abort carries across sequence steps; that is a property of the *flattening*, and it is why improvise-0hb's proposal to apply effects between steps was rejected — `apply_effects` resets the flag per batch, so step 2 in a fresh batch would land the `Quit` after a failed `Save`, silently reopening a P1. `a_sequence_whose_first_step_aborts_does_not_run_the_second` is the guard. **The rule the aborts taught, which is the part to carry:** an abort at the *tail* of a batch also ate the mode restoration and the buffer clearing, stranding the user in `CommandMode`/`Editing`/`SavePrompt` — where `draw_bottom_bar` renders the minibuffer *instead of* the status line, so the error the effect had just written was never drawn and the operation appeared to do nothing at all. So **tidy-up goes at the head, where an abort cannot reach it**: `ExecuteCommand` fronts its batch with `ReturnToNormalUnlessMoved`, `CommitAndAdvance` with the mode change, `CommitSave` with the mode change (found by Copilot on PR #27). Ordering answers it without giving any effect a "runs anyway" flag, and it does not weaken gotcha 31's observed-not-predicted mode settling: whatever a later effect sets, it sets *after*, so the mode a `:` line ends in is unchanged in every case — the only difference is what happens when the batch never reaches the end. **An abort is not a rollback**; that is the hole clause 4 leaves (gotcha 47). -22. **The import wizard's back path is a history stack, not a reverse table.** `advance` pushes the step it leaves onto `ImportWizard::history`; `previous` pops it, so back always retraces the path actually taken through the conditional transitions (improvise-907). Preconditions live in `validation_error()`, checked on the step where the user can still fix them — never deferred to `build_model`. `NameModel` is terminal for `advance`: Enter there calls `build()`, which parks the workbook in `built` on `Done`, and `AcceptWizardModel` takes it from there. Going back from `Done` discards `built`. **`ResolveDuplicates` appears only when `ImportPipeline::collisions()` is non-empty**: `after_fields` routes there from `ReviewProposals` or `ConfigureDates`, so it comes after the field and date steps and before `DefineFormulas`, with `NameDuplicateCategory` between them when the answer is Separate. Enter there is `WizardOp::ChooseDuplicate` (sum, keep separate, or go back). `ImportPipeline::on_duplicate` defaults to `DuplicatePolicy::Refuse`, so a path that never asked — the headless `--no-wizard` import included — errors listing the collisions rather than summing them. **A collision is a cell written by two records** (a tuple *and* a measure), not a shared tuple: records with disjoint measures on one tuple write different cells and collide with nothing (improvise-9bit). Reasoning: `context/code/formula-and-io.md`, "Colliding records are a question, not a default". -23. **`app.layout` goes stale mid-batch.** `apply_effects` rebuilds only *after* the whole batch, so an effect that reads the layout after an earlier effect changed the grid needs a `RebuildLayout` effect in front of it — the command decides the order (see `EnterEditAtCursorCmd`, `CommitAndAdvance`). Read the cursor's cell text via `App::display_value_at_cursor()`; don't build a `CmdContext` inside `apply` (it reads `layout`, the memoised `grid_geometry()` and the cursor, and a stale layout there is the bug). **Since improvise-felo.13 the rebuild is skipped when nothing it reads has moved**: `App::refresh_layout` rebuilds only when `layout_stale` (any `EffectKind::Model(_)` in the batch, or `install_workbook`), the drawn view's name, or the drill snapshot's pointer (`LayoutBasis`) changed, so a cursor key rebuilds nothing, and the column widths and row-header width are measured once per layout (`App::grid_geometry`, dropped with the layout). Two of the three inputs are **observed** (the view name and the drill pointer); `layout_stale` is **declared** from `EffectKind::Model(_)` (`touches_model`), so gotcha 3's classification hole — a `ViewOnly` effect that changes what the layout reads — now also leaves `app.layout` stale, where before felo.13 every batch rebuilt. It is still not a hand-kept list of effects. A writer that bypasses effects and changes what the layout reads must call `rebuild_layout()` itself. Measurements: `context/test-evidence/layout-cost.md`. - -24. **The cursor is per-client and keyed by view.** `ViewState::cursors: HashMap` holds `selected`/`row_offset`/`col_offset`, keyed by view name; an absent key means the origin. Reach it through `App::cursor()` (a `Copy` snapshot), `App::cursor_mut()`, and `App::reset_cursor()` — never off `View`, which no longer has the fields (improvise-8uz). Two consequences. **Changing the grid's shape must reset the cursor explicitly**: `View::transpose_axes`/`cycle_axis` used to do it themselves, and now `TransposeAxes`/`CycleAxis` call `app.reset_cursor()` — a new geometry effect that renumbers rows or columns has to do the same. **Replacing the document must clear the map**: `LoadModel` and `AcceptWizardModel` call `cursors.clear()`, because the incoming workbook's view names have nothing to do with the outgoing one's. That clearing replaces `Workbook::normalize_view_state`, which is gone — a `View` can no longer carry a stale offset. The same clearing owes the **session mirrors**: `install_workbook` bumps `App::document_generation`, and `ClientSession::project` reseeds any mirror seeded at an earlier one (improvise-cqq). An effect cannot reach `Sessions`, so the generation is the mechanism and `Sessions::rebase` is only the client-facing half. The same is true of a **geometry** change, and it owes the mirrors *both halves*: `App::cursor_epoch` is bumped by `reset_cursor` and watched by `Sessions::adopt_cursor_epoch`, which resets every mirror **and** returns `Broadcast::ToAll(Command::ResetCursor)` (improvise-gvnb). Doing only the first is **strictly worse than doing nothing** — the diff then believes every client already holds origin-positioned content and sends nothing, so they stay stale *permanently* rather than until their next scroll. The trigger is the **epoch moving**, not the command variant, for the reason gotcha 39 watches `document_generation` instead of matching on `LoadModel`: which effects renumber the grid is a per-effect decision, so a table of variants would be a second description of it, free to disagree. What is broadcast is an **invalidation, never a position** — another client's transpose must not move where I am looking (that is UI and it is mine, improvise-g0f), only declare that where I was looking has stopped meaning anything. **Since improvise-ni6k no effect calls `reset_cursor` at all — the cursor is invalidated by OBSERVING the grid's frame.** `apply_effects` diffs the frame after every model effect, **inside the loop**, because a batch that reframes and then places the cursor loses the placement otherwise (four records-mode tests pin that position). There is no list of effects anywhere, so the mechanism is total over `EffectKind::Model(_)` by construction — a genuinely stronger shape than a hand-kept table, and the reason the old "a geometry effect that forgets `reset_cursor` is invisible in both directions" hazard is closed. - - **Two arms, and the asymmetry is the whole design (improvise-mcq4).** `LayoutGeometry` hashes only the **active** view — it is the held-key path, and other views cannot reframe there since `view_mut()` writes only the active view, with the category and item effects already covered by `view_shapes`. `CellData` hashes **every** view, because a cell write can prune a row out of a view the host is not on; narrowing that arm to one view measured as *free* and silently lost a multi-client invalidation, which is gotcha 24's own strictly-worse-than-nothing case. Cost of the wide arm: 0% on the held-key path, +5.7% on an 18k two-view `SetCell`, linear in view count, accepted. State the absolute as well as the ratio: the observation roughly doubles held-key latency on large pruned documents (3.04 → 5.69 ms at 18k) and 56 ms at 90,000 cells is 3.4 frames. - - **Totality is conditional on classification**, and nothing makes classification total: a `ViewOnly` effect that renumbers the grid is invisible to all of this (gotcha 3's hole, unchanged). And two known gaps are filed rather than fixed — a geometry effect immediately after a mid-batch `SwitchView` skips the check entirely, because there is no basis for the new view yet (improvise-9c9o; under-reporting, which gotcha 35 says is the expensive direction), and `Model(WholeWorkbook)` matches neither arm, which is benign only because all three such effects go through `install_workbook` → `rebase` → a fresh `Snapshot`. - -25. **`ViewState` and its field types are `Serialize`/`Deserialize`.** It is the wire payload of the client Snapshot (improvise-q08), so `AppMode`, `MinibufferConfig`, `MinibufColor`, `ViewFrame`, `KeyCode` and `KeyModifiers` all derive both (improvise-a2x), and `improvise-command` depends on `serde` for it. The constraint this puts on new state: **no borrowed-`'static` fields anywhere ViewState can reach** — `Deserialize` cannot produce one, and the error surfaces on every enclosing variant rather than at the offending field. `view_state_survives_a_json_round_trip` guards it. `ViewState` and `Snapshot` also derive `Clone` and `PartialEq`, so the bootstrap message can be compared in a round-trip test. - -26. **Axis presentation lives on `Axis`; its colour is a `Face`.** Four vocabularies hang off the enum in `view/axis.rs` — `plain_label()` ("Row"/"Col"/"Page"/"None"), `glyph()` (which `Display` appends, giving "Row ↕"), `short_label()` (three columns for the tile bar; `None` is "·", *not* a truncation), and `initial()` (one letter for the view panel; `None` is empty). Plus `next()` for the Row→Column→Page→None rotation. **`Axis` carries no colour**: `AxisColor` and `Axis::color()` are gone. A colour is a `Face::Axis(axis)` in `improvise-protocol/src/face.rs` (ADR 004, `context/code/protocol.md` §10), resolved by the `Theme`, and `draw.rs::face_style` is the one place a face becomes a ratatui `Style` — the tile bar and the category panel both call it, so neither keeps a table. `improvise-core` still names no ratatui type. Not covered here: `persistence/format.rs` maps `Axis` to the on-disk keywords ("row"/"column"/"page"), which is a format concern and stays in the io layer. - -27. **`Axis::Filter` constrains without being navigable.** Drilling fixes its coordinates on `Filter`, not `Page` (improvise-dz8): both restrict which records are visible, but only `Page` is something the user may move or page through. The split shows up as two different questions, and picking the wrong one is the trap: **"what data is visible" is `View::filtering_cats()`** (Page ∪ Filter — used by `GridLayout` and by `add-record-row` when it builds a key from the current constraints), while **"what can the user act on" is `categories_on(Axis::Page)`** (used by `[`/`]` page cycling) or `View::tileable_cats()` (used by all three tile-select sites). `Axis::next()` maps `Filter` to itself and no navigable axis rotates into it, so tile select cannot hand one out. The `.improv` grammar accepts `filter` and round-trips a selection with it — a Filter axis only ever occurs in the never-persisted `_Drill` view, but the alternative was writing it out as some other axis. **What a virtual view must *be* is model-side and has one home** (improvise-gd28): `view::{records_spine, records_steps, drill_steps}` return `Vec` — pure data naming no `Effect`, no `App` and no front end — and `cmd/grid.rs::steps_into_active_view` is the whole of *how the TUI gets there*, twenty lines a browser answers its own way. Before the split, the drill's filter placement and the records spine were spelled out twice in `cmd/grid.rs` with nothing keeping them in agreement. `DrillState` deliberately did **not** move: it holds an `Rc` and its `pending_edits` are uncommitted writes, so gotcha 30's session-local tag already places it. Reasoning: `context/code/core.md` §11. - -28. **The browser backend is sync on purpose, with an async seam.** `OpfsFs` (`improvise-io/src/opfs.rs`, wasm32 only) implements the same synchronous `Storage` trait as `PhysicalFs`, because `FileSystemFileHandle.createSyncAccessHandle()` yields a handle whose read/write/truncate/flush **are** blocking — available only inside a dedicated Web Worker, which is where improvise-9x6 puts the app. So `Storage` never became async. What is still async splits out into four explicit methods the worker awaits from its message loop: `open()`, `attach(path)` (must precede any read/write of that document — acquiring a handle is async even though using it is not), `refresh()` (repopulates the index `list` answers from), and `commit_deletes()`. Consequence: **`delete` is visible immediately but lands in OPFS at the end of the batch** — it drops the name from the index and queues a `removeEntry`. Handles are exclusive per file, so `attach` closes any handle it replaces and `detach` releases the lock. `storage_contract` in `storage.rs` is the behaviour every backend owes; it runs against both host backends and is the written-down spec `OpfsFs` must satisfy in a browser, where the harness cannot reach. **And the premise is dedicated-worker-only, which is narrower than this paragraph used to read.** `createSyncAccessHandle` is spec'd `[Exposed=DedicatedWorker]`, and measured in a module `SharedWorker` on Chromium and Firefox it is **absent** — not rejected, not flagged, not transient: `getDirectory()` and `getFileHandle()` both succeed and the method is simply not on the object (improvise-12ha). So `OpfsFs` cannot be hosted by a `SharedWorker` as written, and "a browser worker" in this paragraph means a *dedicated* one everywhere it appears. `OpfsFs::open`'s own guard does not enforce that — it `dyn_into`s `WorkerGlobalScope`, which a `SharedWorkerGlobalScope` **satisfies**, so a shared worker sails past a check whose message names the right requirement and fails later as a raw `TypeError` deep in the glue (improvise-7gip). The consequence for multi-tab is that the machinery everyone reaches for first is the storage seam, not `Sessions`: `Sessions`, the `Addressed` filter and the one `frame` pair are all ready, and none of that helps, because the no is entirely on the handle. - -29. **"Which view is active" is stored twice, and the copies are now allowed to differ.** `Workbook::active_view` is the document's **default** — what a freshly connected client opens, and what `.improv` writes as `Initial View`. `ViewState::active_view` is **authoritative for that client**, and it is what keys `ViewState::cursors`. The second copy exists because `reduce_view` runs on a client that holds no `Workbook` (improvise-cqi); the *split* is improvise-jytm answering improvise-g0f, on the user's ruling that **the model is shared and the UI is not**. Two clients may sit on different views of one document. Three things follow. **`App::resolve_view` is the one answer** to what a view name names in this document, falling back to the default so a slice naming a deleted view cannot panic in `rebuild_layout`; `view()`, `view_mut()`, `view_name()`, `project::grid_in` and `ClientSession::view_in` all go through it, and a second copy of that fallback was the first thing review found. **The guard was replaced, not deleted** — `the_two_active_view_names_never_diverge` became `the_documents_default_view_outlives_this_clients_switches` plus `a_fresh_session_opens_the_documents_default_view`, because a deliberate split with no guard is how one silently regresses into the other. And **a deleted view is the case the split creates**: `DeleteView` can take away the view another client is on, so `Sessions::adopt_deleted_views` moves that mirror to the default and tells it, and `effect::DeleteView` does the same for the host — observed ("does this mirror still name a view the document has"), never predicted from the command variant. **Testing any of it needs a bystander that can be moved**: both audience tests first sat theirs on the default, where a wrongly-addressed relay folds to a no-op, and review demonstrated the vacuity by injecting `ToAll` twice. One gap remains: `Initial View` no longer follows the TUI user (improvise-azhb). The other — a view-scoped `ModelCommand` naming no view — is **closed** (improvise-lltt), and this paragraph described it as an open gap for a day after: every view-scoped variant carries `view: String` and `model_effects` maps it to `ViewTarget::Named`, so a connected client's `SetAxis` lands on the view it names. `ViewTarget::Active` resolves through `App::view_mut()` and is therefore the *host's* view — but **the wire never constructs one**, which is what closed it. Follow mode is deliberately not foreclosed. - -30. **Two tests gate `ViewState` membership.** `ViewState` is the *portable* session slice — the one thing serialized into the Snapshot a newly-connected client gets (improvise-q08). A field may live there only if it is **portable** (plain data from the allowlisted crates, no `Rc`/`Arc`) and **snapshot-safe** (a fresh client inheriting the value is correct). `wizard` fails both — `ImportWizard` is an `improvise-io` type, and a half-open dialog is not something to inherit — and `drill_state` fails both — it holds an `Rc`, and `pending_edits` are uncommitted writes belonging to whoever staged them. Both live on `App` under the **session-local** residue tag (improvise-h0b). The `ViewState: Send` static assertion in `app.rs`'s tests is the compile-time proxy for portability: `Rc` is not `Send`, so a re-added shared pointer fails the build (the `Deserialize` derive from gotcha 25 now rejects it too, serde's `rc` feature being off). `Send` does *not* catch the origin-crate or snapshot-safety halves — check those by hand. - -31. **`:` commands settle their mode by observation, not prediction.** `ExecuteCommand` ends every batch with `ReturnToNormalUnlessMoved(ctx.mode)`, which drops to `Normal` only if the mode is still what the batch started in. There is no `Effect::changes_mode()` any more: it was a per-type constant, and `ViewBack`/`ViewForward`/`StartImportWizard` swap the mode only on some runs, so no up-front answer was right — `:import` opened the wizard and was snapped straight back out (improvise-toc). A new effect that changes the mode needs no annotation; just don't return the batch's starting mode from a success path. - -32. **There is no aggregate render cache — each widget folds its own stream.** `improvise-protocol::render` holds one cache type per widget (`GridCache`, `CategoryTreeCache`, `FormulaCache`, `ViewListCache`, `TileBarCache`), each implementing `WidgetCache` with its own `Update` type, and a `Projection` is one such update addressed to one widget (improvise-rkeq). `reduce_view` folds only `ViewState` now — it no longer takes a cache, and `Command::Project` passes through it untouched. Two consequences. **The overlap between caches is deliberate**: the tile bar and the category panel both carry `(category, axis)` and are updated by separate projections, so neither is woken by the other's changes; do not "fix" it by sharing a variant. **Bootstrap is a *compaction*, not a replay**: `Snapshot { view, seed: Vec }` seeds a new client with a batch of the same projections the steady stream carries, so there is no whole-state type that could disagree with the incremental one. The seed is the server's *current* state, not the head of the history — `GridCache::seed()` builds it, and the property that earns it is `GridCache::folded(&cache.seed()) == *cache`. Nothing is defined for the help overlay (its content is static and `help_page` is already in `ViewState`) or for the import wizard, which now has a shape and **deliberately no projection**: `WizardCache` is built by `project::wizard_cache` and drawn locally, because a half-open dialogue is addressed to precisely the session that opened it, never to every client (improvise-764 — the rendering-side twin of the argument that removed `AcceptWizardModel` from `ModelCommand`, gotcha 38). **A cache without a `Projection` is the shape session-local residue takes when it still has to be drawn by two different front ends.** It lives in `improvise-protocol` rather than `improvise-command` for the `flatten_cat_tree` reason: when the wasm client grows a wizard it must draw *these* rows, not a second derivation. - -33. **The grid's labels frame its cells, and `apply` enforces it — the fold is not commutative.** Three consequences in `improvise-protocol/src/render.rs`. (a) A `GridUpdate::Cells` that overtakes the `RowFrame`/`ColFrame` naming its positions is **dropped**, because `fit_to_labels` truncates to the labels it has; a client must not reorder the stream, and `a_cell_update_that_overtakes_its_labels_is_lost` pins it. (b) A label change that *renames* a position blanks the cell under it: `fit_to_labels` preserves by POSITION, which is only right where the label did not change, so `apply` calls `invalidate_relabelled` itself rather than trusting a producer to send `GridUpdate::Invalidated` first. `Invalidated` therefore now means only "contents stale, frame unchanged". (c) **every arm that touches labels *or* widths re-fits** — `col_widths` is re-fitted with the columns — a new column is drawn at its label's own character count until `ColWidths` catches up — because a renderer zips widths with labels and a length mismatch indexes off the end. Fitting on the label arms alone is what let a late `ColWidths` put the mismatch straight back — a short update keeps each unheard-of column at its label's width, and a long one is dropped as having overtaken the frame that names it, exactly as (a) holds `Cells`. `GridCache` also carries `totals` and `page_coords` (improvise-n10). `totals` is framed by `col_frame` on the same terms as the cells and widths and re-fits with them, with one asymmetry: an **empty** strip means "this grid has no totals" (records mode, or an empty grid) and is therefore *not* padded, where a non-empty one is — padding it would draw a blank total row under a records grid that has none. `Invalidated` blanks the totals too: they are contents, and a stale sum under fresh cells is the one number a user takes at face value. `page_coords` is framed by nothing (it names categories, not positions) and survives a reshape. Note the fold tests: asserting `fold(prefix)+fold(rest) == fold(whole)` is vacuous (it is the definition of a left fold), so the tests that earn their place are the order-sensitivity one and the compaction-equivalence one — see `doctrine(fold_tests_must_test_order_not_the_axiom)`. - -34. **A formula target is unquoted, and pipe quoting has one home.** `parse_formula` strips the pipes off a target exactly as it does off an `Expr::Ref` — the target names a `_Measure` item, so a target that kept its delimiters made `add_formula` create an item literally named `|Total Revenue|`: a ghost column the user's real item never matched, and `Expr::Ref("Total Revenue")` never resolved to. Both the TUI formula minibuffer (`AddFormula`) and `.improv` load go through that one call, so both were affected. The writer emits `Formula::raw`, not `target`, so file text is unaffected either way — which is exactly why the round-trip tests never saw it, and why the fix repairs existing files on next load with no migration. Pipe quoting has exactly **one home**: `improvise_formula::quote`, owning both halves — `quote_name`/`is_bare_name`, `pipe_quote` (unconditional, for a `.improv` text cell that must not read back as a number) and `unescape_pipe_body`. `improvise-gp` re-exports it, `improvise-io` calls it from `persistence/{format,parse}.rs`, `parser.rs::unquote_pipe` is a shim over it, and io's own copy is **deleted** (improvise-i2m5). **The keyword guard is canonical — user-confirmed, so do not revert it as over-caution**: an item literally named `SUM` is written `|SUM|`. Bare keywords do parse, but only because `agg_call` needs a `(` and PEG backtracking falls through to `ref_expr` — rule ordering, not a stated rule — and a `## Formulas` line carries formula text verbatim, so a name `format_md` writes is read back by `parse_formula`: one namespace, one spelling rule. One thing the guard does not cover, and one that used to be an exception and no longer is. **A section header takes its name as the whole line, totally** (improvise-pczv): `header_name = { pipe_quoted ~ &NEWLINE | line_name }` — the pipe-quoted arm needs the lookahead, or `|A|B` would parse as `A` and silently drop `B`, and `line_name` catches everything else, so a header name can no longer fail to parse. Four rules share it (`model_name`, `initial_view`, `category_section`, `view_section`), which is why it is a rule and not four copies. Note a header is still *written* raw where a reference is quoted — the grammar accepts both arms, so the quoting asymmetry between a header and a reference is a writer convention, not a format one. And the two former *unquoting* copies differed only on a trailing lone backslash, unreachable from the grammar, where io returned it and `parser.rs` panicked — the surviving `unescape_pipe_body` is total. Two `## Category:` sections naming one category are **one** category, concatenated by `ParsedFile::push_category` in **pass 1** (improvise-349m) — merging in pass 2 was not idempotent, because `build_category` creates its own section's item-groups before its own `>` lines while `format_md` writes every item before every `>` line. No grammar change was needed: this is the tree-walker, not the format, and duplicate headers stay legal. And **`WHERE` now carries `IS`'s word boundary** — both are compound-atomic with `sp+` (improvise-r08u), so `SUM(Rev WHEREAS x = y)` is a syntax error rather than a `WHERE` on a category `AS`. The guard is `sp+` and **not** the `!bare_char` lookahead the ticket proposed, because the grammar-walking generator reads the grammar through `pest_meta` and **skips predicates**, emitting `WHEREa=""` — a rule whose only guard is a lookahead is a rule the generator can violate. It costs one spelling: a category may no longer abut the keyword. - -35. **`Dirty` over-reports on purpose, and the asymmetry is the reason.** `Dirty::of` maps an `EffectKind` to the widgets that kind can reach, and the two directions do not cost the same: over-reporting pays one redundant `Projection`, under-reporting leaves a client's cache wrong **forever**, with nothing in the stream to correct it. `LayoutGeometry` therefore includes `formulas`, because `RemoveCategory` wears it and `Model::remove_category` ends by dropping every formula targeting the category — a client's formula panel otherwise kept showing a formula the document no longer had. It now coincides with `Dirty::EVERYTHING` and is still spelled out, because the two are derived from different effects and need not stay equal. The derivation now lives in `context/code/command.md`, and the doc comment on `Dirty::of` points there rather than restating it; the reason `LayoutGeometry` is spelled out rather than written as `EVERYTHING` is a two-line comment inside the match body. It went stale once already — claiming `AddCategory` was `CellData` after it was reclassified — so reclassifying an effect means re-reading the prose, not just the comment. - -36. **A client's command has one door, and the door takes the `App`.** `Sessions::apply_checked(app, id, Checked<'_>)` is the only way a connected client's `Command` reaches the server — `Host` calls *that* (improvise-ahdf). `Sessions::apply` is not `pub`, delegates to `apply_checked` after `Checked::against`, and has no production caller: its call sites are tests, which is why it takes a raw `&Command`. `ClientSession::fold` and the mirror's `ViewState` are private, so a transport cannot reach the mirror at all. `apply_switch` now **takes a `SessionId`** — it is addressed to exactly one session, which is the opposite of what this gotcha used to say — moves only that session's mirror, and dirties nothing. The one thing it still asks the document is whether the view exists, because a mirror keys `ViewState::cursors` and is read by `grid_in`. The audience is conditional: an accepted switch relays **nothing** (the asker folded it, and no other client's view moves), a refused one is `Broadcast::To(asker, …)` — improvise-6c6q, where `ToOthers` told every client but the one that was wrong. `adopt_active_view` and `Broadcast::ToOthers` are **deleted**; the variants are `ToAll` and `To(SessionId, _)`, the second because the deleted-view case needs a one-session audience that is *not* the asker. The paragraph's warning about the "what do the others hear" helper survives and is why the 6c6q test drives the asker's own `ViewState` — that helper is itself deleted. What survives unchanged: the model is shared and the UI is not, and `Broadcast` is a `Vec` because one command can owe two relays with different audiences. **The trap this paragraph used to describe is gone with `ToOthers`, and the lesson it taught is not.** It was copying `ToOthers` from `apply_switch` into `apply_model`, invisible from the bystander's side because `adopt_active_view` moved the asker's mirror either way — so the drift sat between the server's picture of the asker and the asker's own slice, where a test written through a "what do the others hear" helper looks away by construction. improvise-6c6q closed the mirror-image hole on the refused path. Two rules survive the mechanism: **assert the `Broadcast` itself**, since the audience is the message rather than something to infer from folded state, and **seat a bystander somewhere a wrong relay can move it from** — a bystander already on the relayed value cannot see a wrongly *widened* audience either, because the wrong relay folds to a no-op. Review demonstrated that vacuity twice on PR #48. A `Vec` rather than an `Option` because one command can owe two relays with different audiences, and an `Option` would have had to drop one silently. Everything else is that client's alone and never touches the `App`'s own view slice: **the model is shared, the UI is not.** Folding `SwitchView` into the mirror alone was the bug — `viewport()` then read the cursor under the new view's key and the next `project()` built that client a grid out of the *other* view. - -37. **`gp-tree` returns data and never prints, and a fit takes two things to reproduce.** Two rules with one root — the crate is a library a TUI, a wasm worker or a CLI may each host. (a) A decision the user should hear about comes back as **data**: `Dataset::warnings` carries the columns `preprocess` dropped for being entirely missing (improvise-d44s), and `fit.rs` hands back pooled `(prediction, actual)` pairs and a score per feature rather than tables. `examples/fit_csv.rs` owns every `println!`, and that is the only reason its output stayed byte-identical across these moves — adding an `eprintln!` to either module is the regression. (b) **The seed is necessary and not sufficient**: `data::Model` records `seed: Option` *and* `config: Option`, because the same seed under a different population/generations/elite/penalty lands on a different tree (improvise-ksst). Both are `Option` for one reason — every `FitConfig` value and the seed `0` are legal, so nothing can double as "unknown". Note `#[serde(default)]` on an `Option` field is **documentation, not the mechanism**: serde defaults it anyway, so deleting the attribute fails no test, and a backward-compatibility test that only deletes it is not discriminating. Aim such a test at an absent field loading as the *default* instead of as unknown. The GA loop itself is `fit::fit` (improvise-0ai); `breed` remains one fitness-agnostic generation for callers whose fitness is not "predict this column". (c) **A scoring function takes one slice of rows.** `Row` pairs a feature vector with what it predicts, so the parallel `x`/`y` lengths that `zip` used to truncate silently cannot be built (improvise-dkqn, `doctrine(an_invariant_is_made_unrepresentable_not_validated)` — the same answer `prune_axis_items` got, and the reason no `Result` entered five signatures). The pairing is made at the boundary that *builds* the data: `preprocess` walks one row index and the rows live inside `Target`, because a `Dataset` carrying `rows` and `target` separately just moves the zip up one level into every caller. The empty row set is one shape, answered once in `mean_over`. Changing any of this needs **bit-level** evidence: `tests/scoring_known_answers.rs` pins every path as f32 bit patterns plus a `next_u64` witness that catches a changed RNG call order, because the CLI prints three or four decimals and byte-identical output alone cannot see a moved accumulation. (d) **A fit is storable only when the document can *name* its features.** `preprocess` invents `Region=North`, `Cost.missing`, `when.year`; those are **encoding artefacts, not items of the document**, and `Expr::Ref` resolves through `Model::find_item_category` — so a tree splitting on one emitted a well-formed formula whose every cell was `Error("ref:…")`, silently (improvise-8xhy). `fitted_formula` therefore takes the `&Model` and refuses, listing what it could not resolve; `formula_text` still spells it, being the tree's own text. **The check asks the document, never the name's shape**, because a measure literally named `Region=North` is a legal document and the two are indistinguishable as strings — which also means `.missing` and the date parts are covered with no second copy of `preprocess`'s expansion rules to drift from. **This is an interim** (improvise-4zl7): a `WHERE` predicate needs a *scalar coordinate predicate* the formula language does not have — `Filter` gates a formula or an aggregate, it is not an `Expr` — would still leave `.missing` and the date parts, and must first answer what such a predicate means where `context.get(cat)` is `None`, since `Formula::filter` falls through to `aggregate_raw` and a categorical fit would silently degrade in every view that aggregates over the category. **improvise-zv8t closed the categorical half of the accident, and the close cost a document.** `fit_measure` handed `preprocess` an `Overrides::default()`, so it never said which header columns are document categories *although the very next line told `Features::of`* — and `preprocess` infers `Numeric → Date → Categorical` from the values, so a `Year` category with items `2024`/`2025` was a *numeric column* and a category of ISO dates a *date column*, both refused with `Year IS 2024` spellable all along. It now forces the same `header[..category_columns]` slice to `Forced::Categorical`. Two things follow. The **date decline is narrower than it read** — it covers a genuine date *measure*, never a category misread as one. And the forcing **widens the encoding**: a hundred-item numeric category one-hots past `MAX_FEATURES` and refuses a fit that previously succeeded by never splitting on it (improvise-e8no — and **not** to be closed by making the forcing conditional on cardinality, which is inference by another name and makes the *encoding* depend on a budget rather than on what the document says). The `Artefact` arms stay: `crate::dataset` is `pub` and takes the caller's `Overrides`, so the roles are still reachable. **Forcing by name while budgeting by index defeats the budget** — a category and a measure may legally share a name, and `no_ambiguous_column_names` ran *after* `preprocess`, so the expansion the preflight exists to prevent was allocated and only then refused; `as_categories` now returns `Result`, so the ordering is a compile error rather than a convention. The test that missed all this is the shape to remember: `improvise_eval` **fabricates** a `_Measure` item per feature name, so it asserted the emitter and `gp_tree::eval` agree against a model that could not exist. It is kept, with its scope now stated, and the property it could not state is asserted separately against a real document. - -38. **The wire vocabulary is derived twice, by the same recipe.** `Command`'s view half is the `EffectKind::ViewOnly` effects targeting the portable slice (`decision(improvise-cqi, command_set_derived_not_invented)`); its model half, `ModelCommand`, is the `EffectKind::Model(..)` effects — **28** of them, brace-matched on `c1772ca9` (2026-09-24) — minus `ApplyAndClearDrill` and `AcceptWizardModel`, leaving **26** variants (improvise-kmgs). The two added since this paragraph counted 26 and 24 are `ReplaceFormula` (improvise-km71) and `RenameRecordCoordinate` (improvise-fdk1); `SwitchView`, `ViewBack` and `ViewForward` left the model half with improvise-jytm. Every count in the rest of this paragraph is dated, so recount rather than cite it. **Completeness is now checked, not counted**: `ui/reduce/vocabulary_is_complete.rs` requires every effect to be on the wire under its own name or listed in `NOT_ON_THE_WIRE` with a verdict — `Command`, `Frame`, `Off` with its argument, or `Gap` naming the ticket that owns it (`SaveAs` and `ExportCsv` are gaps, improvise-2mqz). It is the forward twin of `vocabulary_is_composed.rs`, and it is what the enumeration below lacked. `SwitchView` no longer needs subtracting separately: it is not a model effect any more. `ViewBack`/`ViewForward` left for a reason this paragraph **stated wrongly until 2026-09-21**: it said they went for the *same* reason as the other two exclusions, session-local residue. That is false, and the falseness mattered. `ViewState::view_back_stack` **is in `ViewState`**, which is the portable slice — so by the view half's own rule (`decision(improvise-cqi)`: the `ViewOnly` effects targeting the portable slice) they were always `Command`s and were simply **missing**. The cost of the mis-statement was a live divergence nobody could see: `effect::SwitchView` pushes the leaving frame and clears the forward stack, while `reduce_view`'s arm set `active_view` **and nothing else** — so **no client ever had a back history at all**, and that, not the drill commit the `declined()` comment blames, is why `view-back` was dead in a browser (improvise-gd28, PR #115). **The lesson is about derivations, not about these two variants**: a derivation stated as a rule is only as good as the last time somebody enumerated it *against* the rule. This one had been checked for correctness — every listed variant belongs — and never for **completeness**. Note the obvious `awk` extraction reports **30**: it picks up `HelpPageSet`, which is `ViewOnly`, because that is the last `pub struct` before `#[cfg(test)]` and the test module mentions `ModelChange::` — brace-match the `impl Effect` blocks instead (`doctrine(derive_then_verify)`). Three things follow. **A `ModelCommand` does not mutate**: `ui/reduce.rs::model_effects` maps it back to the effect it came from and `apply_command` runs that through `App::apply_effects`, so effects stay the only mutation site and the `Dirty` is the effect's own `EffectKind`, never a second classification beside gotcha 35's table. **Two model effects are deliberately off the wire.** `ApplyAndClearDrill` and `AcceptWizardModel` were once kept on the grounds that the *command* is portable even where the operand is not; **that was the wrong criterion** (improvise-3x3u). A `Command::Model` is routed with no `SessionId`, which is right exactly when the change is addressed to *no one* session — and these are addressed to precisely one, never the sender: their operands are `drill_state.pending_edits` and the built wizard workbook, session-local residue (gotcha 30) private to whoever staged them (improvise-tom1). Unaddressed, B's `ApplyAndClearDrill` commits A's uncommitted typing and B's `AcceptWizardModel` swaps the document out of a dialogue B never opened — both demonstrated against `Sessions::apply` before removal. So **28** − these two = **26** on `c1772ca9` (it was 26 − 2 = 24 before km71 and fdk1), and the older spelling of this sum (`29 − SwitchView − these two = 26`) was itself a survival of the same staleness this paragraph documents: it subtracted `SwitchView` and forgot its two siblings, so it contradicted the count at the head of the paragraph and the enum’s own 24 variants (found by ChatGPT Codex on PR #116). The effects are untouched and still reached by the key path. Not permanent: multi-client drill wants the *addressed* form, which is where ADR 001 layer 2 goes anyway. Guarded by `no_model_command_commits_another_clients_session_local_state`, plus a companion asserting both effects still exist and are still model changes so the first cannot pass by naming nothing. **`Sessions::apply_checked` routes the model half to `apply_model`**, which relays `adopt_cursor_epoch`'s `ResetCursor` and `adopt_deleted_views`'s per-client switch; a document-replacing `LoadModel` bumps the generation and lands in the rebase arm. The former claim here — that `ViewBack`, `ViewForward` and `LoadModel` all end in `App::switch_to_view` so every mirror takes the accepted active view — was wrong on all three clauses after improvise-jytm and is deleted. A keymap lookup still cannot return a `Command`, and not for want of vocabulary: a binding names an *intent* resolved against `CmdContext` (`move-cursor down`) while a `Command` is the *resolved mutation* (`SetSelected { row, col }`) — improvise-gxi's AC 4 asks for something the split forbids. - -39. **`Applied` is an enum because a document swap is not a diff.** `Sessions::apply_checked` returns `Applied::Diff { dirty, broadcast }`, `Applied::Rebased { snapshots }` or — since ADR 001 layer 1 — `Applied::Rejected { reason }`, and the enum *is* the mechanism: a host that wants the `Dirty` must match, and matching names the case where a `Snapshot` is owed. The server's mirrors self-heal — `ClientSession::project` reseeds on the `document_generation` bump — but `ViewState::cursors` lives on the **client**, keyed by the outgoing document's view names (gotcha 24), and only a fresh `Snapshot` replaces it. `Command::Model` put the three `WholeWorkbook` mutations on the wire (gotcha 38), so the host that must send the snapshots is no longer the host that asked. `apply_model` therefore calls `rebase` itself rather than instructing the host to, exactly as `apply_switch` adopts the accepted view itself. **The trigger is the generation moving, not the command variant**: a `LoadModel` that cannot read its path leaves the document alone, and reseeding on the variant would cost a client its cursors for a *failed* load — `a_load_that_fails_is_not_a_rebase` guards that arm. A `rebase: bool` beside `dirty` was rejected, because the host that forgets to call `rebase` is the host that forgets to read the flag. The **third** case is a *normal outcome*, not an error, and it is in the enum for the same reason the other two are: a host that wants the `Dirty` must match, and matching names it. `ServerMsg::Rejected` is the matching frame, addressed to **the asker alone** — nothing happened, so nobody else is owed anything — and the client keeps it as a `Rejection` rather than a sentence, because force needs to know *which* record and *which* kind (gotcha 43). - **Fixture warning.** When testing a document swap, the incoming document's active view must have the **same name** as the outgoing one. A rename makes `apply_model` broadcast a `SwitchView`; the client folds it, and the stale cursor is then read under a key nothing has written — right *by accident*, with the bug invisible. `other_document()` renames, and a first version of this test passed because of it; `a_document_shaped_differently_under_the_same_view_name()` keeps the name and changes only the shape, so there is nothing to hide behind. - -40. **A batch's basis is three tiers, and only two of them can be read off the effect.** `App` carries `document_generation` (bumped by `install_workbook`), `layout_version` (per *view shape*) and `record_versions` (per *record*) — ADR 001's stale-basis check, and shared document state, so never on `ViewState` (gotcha 30). **The unit is the record**: `record_of(key)` is the coordinate tuple minus `_Measure` (`record_row_identity`), because renaming a category coordinate in wide records mode moves every measure cell in the row (gotcha 1) — a cell-level unit is violated by one legal edit, so the unit must be at least as coarse as the coarsest legal write. **Only the `Dirty` is read off the `EffectKind`; both lower tiers are *observed* around the loop** — the source says so at `apply_effects` and this paragraph said the opposite until improvise-ilvl's audit. The `EffectKind` decides only *whether to look*: `touches_model` arms the shape snapshot, `writes_cells` the record fingerprints. `record_versions` **must** be observed, because an `EffectKind` says *that* a batch writes cells, never *which*; `layout_version` is observed for a better reason than necessity — `observe_reshape` diffs `view_shapes` before and after and bumps only where a view's shape actually moved (improvise-pdvm/nynk), so it is a **fact about the document** rather than a claim about the batch. Two consequences the declared form did not have. An **idempotent** `set-axis` bumps nothing, and batching two real changes into one line or two gives the same `2`. And the gotcha 21 argument is subsumed rather than restated: an aborted batch moved no shape, so there is nothing to not-claim — which is why observing once **after** the loop is correct where accumulating `Dirty` inside it is. **`view_shape` hashes what is drawn, not occupancy** (its doc comment: "Drawn, not configured") — axes, the drawn item names minus hidden ones, folds, page selections and the `prune_empty` *flag* — so an occupancy write does **not** bump `layout_version`: a `CellData`-only batch that prunes a row out of a view moves the cursor epoch (the frame observation) and not `layout_version`, as `context/code/command.md` records (the question that raised was improvise-6nsd, closed). The arming is wider than the tier (`touches_model` takes the shape snapshot), but a bump needs a shape hash to have moved. Three further consequences. A write landing the value already there **bumps nothing**, which is correct — no basis was invalidated. A `LayoutGeometry` effect that drops cells **will** bump their records when it shares a batch with a `CellData` one: an over-bump costing at most an unnecessary rejection, the same direction `Dirty::of` is deliberately sloppy in. And only *raw* data is fingerprinted — formula results live in `Model::formula_cache`, and a record whose derived value moved because someone edited elsewhere has not itself been written. The diff collects into a **set**: chaining `before.keys()` with `after.keys()` counts any rewritten record twice, a bug that survived one test run. - -41. **`Storage` is not a VFS, and that closes the SQLite question.** The trait is whole-document `read`/`write` bytes (gotcha 2, `invariant(storage_is_two_ops)`) — no offsets, seeking, handles or locking — so an `sqlite3_vfs`, which needs `xRead(offset,len)`/`xWrite`/`xTruncate`/`xLock`, has nothing to bind to. SQLite *can* still sit behind it, as an in-memory database serialized to a blob (`rusqlite` `serialize`/`deserialize_read_exact`, proven with a probe before declining — declining on a capability that actually exists would have been the wrong decline). It is worth nothing: measured on 18,000 cells the SQLite container is **22% larger** than `.improv`, **4.7× larger gzipped**, ~**160×** larger at the floor (24 KB against ~150 bytes), and still rewritten whole on every save, so the one dimension SQLite normally wins — incremental page writes — is unavailable. improvise-9ix was declined on that evidence. Two things generalise past SQLite. **A second serialization format is a second implementation of the grammar**, obliged to track `improv.pest` forever; and **its own round-trip test cannot detect drift from `.improv`**, because it never compares the two — that missing cross-format equality test is what made the ticket look cheap. If a structured-query story is ever wanted it is a one-way `export_sqlite` beside `export_csv`, native-only and feature-gated: `libsqlite3-sys`' C build fails under this repo's nix cc-wrapper for `wasm32-unknown-unknown`, which would break the `wasm` CI gate the day it landed. - -42. **The browser client's render decisions are made in Rust, and the frames it speaks are nobody's but the protocol's.** `improvise-wasm-client/src/render.rs` builds a `RenderModel` in which every question with a *silent* wrong answer is already answered, because the DOM half (`web/app.js`) is untested JavaScript and a wrong answer there shows up as a wrong pixel rather than a failing test. Three such answers. **`editable` is a fact about the `CellId` variant** — `Record` only, per ADR 002 — and the trap is the third case: `None` means "no projected cell", **not** "aggregate", so conflating them either hides a cell or offers a write the type system exists to make unconstructible. **The selection is placed by reconciling the scroll offset**, because `GridCache` is viewport-indexed while `ViewState::selected` is absolute. `render.rs` does it by *adding* — `r + cursor.row_offset == sel_row` — rather than subtracting, which is why there is no underflow path and why a selection above the window simply never matches. Omitting the offset entirely looks right until the first scroll, so the test that earns its place uses **nonzero and unequal** offsets: at `selected=(5,1)`, `row_offset=4`, `col_offset=1` it separates "reconciles" from "ignores" *and* from "swaps the two offsets". **The category panel is `viewmodel::flatten_cat_tree`**, never a second walk of `CategoryTreeCache` — that second walk was improvise-edp. The page is four string calls: `Client::viewport` (the handshake — the `Snapshot` seeds the *visible* cells, so one built at extent zero is an empty grid and a resize is a fan-out with nothing dirty), `on_frame` (one frame, one fold, arrival order — gotcha 33 holds *inside* a batch, so never coalesce), `render_model`, `typed_cell`. - - **One envelope pair, two deployments (improvise-xks6).** `improvise_protocol::frame` owns `ClientFrame { Viewport, Commands(Vec<_>), Save, Key }` and `ServerFrame { Snapshot(Box<_>), Commands(Vec<_>), Rejected, Accepted }` (as of `c1772ca9`; this line named a singleton `Command` and no `Save`/`Accepted` for some time after both changed); `improvise-ws-server` and `improvise-wasm-client` each `pub use` them and **neither defines one**. There used to be **three** pairs for two deployments — the socket's server side, a hand-kept *mirror* of it in the browser bundle, and a third written for the worker (improvise-tm6) before the projection architecture landed. The question that had to be answered before any of it moved was whether the **worker** and the **socket** want the same vocabulary, and the answer is **yes**: every difference was a *date* rather than a decision, and the test applied to each was whether the **argument** for it names the thing they differ in. None did. `Viewport` is deployment-independent — the extent is not in `ViewState` and no effect produces it, so no derivation reaches it, and a worker hosting the same `Sessions` owes `set_extent` the same answer; it is the handshake on both because a `Snapshot` seeds the *visible* cells. `Key` is deployment-independent too — a binding names an *intent* resolved against a `CmdContext` a thin client does not hold (gotcha 38), which says nothing about what the bytes travel over; the socket lacked it because key forwarding was never built on it (improvise-bpq2), and `Host::handle` now answers `ClientFrame::Key` with an explicit empty vec that **says so**, where the frame used to just fail to parse in `next_client_msg`. And `Commands(Vec<_>)` batches where a singleton cannot, because a fan-out must arrive whole and in sequence (gotcha 33) — a property of the **fold**, not of the transport; an in-process `postMessage` interleaves exactly as a socket does. The worker pair was not merely different but **incoherent by omission**: written before ADR 001, it could carry `Command::Model { basis, .. }` and had no `Rejected`, so a client speaking it composed a claim that could never be answered. That incoherence — not the difference — is the tell for staleness. - - **The mirror is gone, and so are the two tests that held it honest**; that deletion was the ticket's measure of success. `the_socket_framing_matches_the_server` round-tripped hand-listed fixtures through the other crate's types to catch a **rename**; `every_server_variant_is_one_this_crate_mirrors` matched exhaustively to catch a **variant added on the server and never mirrored**. With one definition both are **type identity**: a rename is a compile error at every call site in both crates, and the un-mirrored variant is not a constructible state. That is stronger than either test, and it is the shape to insist on — **if a consolidation leaves one of its compatibility tests alive, the consolidation is not finished.** What survives is `the_retired_spellings_do_not_parse` in `frame.rs`, aimed at the one thing still possible (the retired `Bootstrap`/`Command` spellings returning as serde **aliases**, which would be two homes again wearing one type), and it keeps the injected-fault discipline this paragraph earned the hard way: **assert the mutated input is still well-formed** before feeding it in, and **assert the error kind** — serde's `unknown variant` — not merely that an error occurred, because a rewrite that also breaks the JSON makes the check pass against an enum that really does have the variant. End-to-end compatibility is now `tests/browser_round_trip.rs`, which hands the wasm client's own serialized frame to a real `Host::handle`; `improvise-ws-server` stays a **dev**-dependency of the browser bundle for exactly that. - - **What the harness does and does not reach is gotcha 48's subject** — `web/app.js` is no longer unrun (`harness/tests/shell.spec.js` executes it under Chromium, and its standalone half under Firefox), improvise-hwpi is closed, and the quiet-failure and never-reads-a-key claims that used to stand here went with it. What remains unexercised is named there: the grouped and multi-level DOM, the CSS and the `ResizeObserver` extent arithmetic. What a careful read *did* establish about the last of these: the observer cannot feed back into itself (`#grid` is a flex child with `overflow: auto`, so `replaceChildren` cannot resize it), and `ROW_PX`/`COL_PX` under-estimate, so the worst case is a row of over-fetch. What *is* verified end to end is the Rust half: `tests/browser_round_trip.rs` drives improvise-1ey's stated scope against a real `Host` over a real `App` touching the client only through the four string calls. One deployment note that is not a bug: `examples/demo.improv` leaves `Customer`/`Date` on the None axis, so **every** cell is an aggregate and nothing is editable — correct behaviour, bad demo. Pick a model with no regular category on None. - -43. **A batch carries its basis, and the server had to publish it first.** `Command::Model { basis, cmd }` is a **struct variant** so the basis cannot be forgotten, and `Sessions::apply_model` checks it *before* applying anything — a stale batch is never applied, not even partially (improvise-xwi6/fkfg, ADR 001 layer 1). Four things to know. **`Basis` has no `Default` and no public constructor**: a zero-valued basis is not "no claim", it is the claim that *the document is fresh*, which in a new `App` is true — so a default would be a forgeable bypass rather than an absence. `BasisCache::claim` is the only door a client has and it returns `None` until the server has published one. Note what that does and does not buy: `Basis` derives `Deserialize` because it must cross the wire, and its fields are public, so **a hand-rolled client can state any tiers it likes**. This is an *anti-staleness coordination check, not a security boundary* — `RecordUnstated` closes **omission**, not **invention**, and nothing here tries to close invention, because a client that forges a current basis is indistinguishable from one genuinely caught up. "The only constructor" is a statement about clients written against this crate, which is still worth having: it makes the honest path the easy one. **Which records are checked is the server's decision** — `reduce.rs::records_named`, an exhaustive match with one or-pattern for the variants that name no record, so a new record-writing variant fails the build. Reading the list off the *client's* claim would be a **force by omission**: the record tier skipped, with no flag to remove and nothing in the message saying so. A claim that omits a record the command writes gets `Rejection::RecordUnstated`, a case of its own because "you lost a race" and "you were never in one" are different things to tell a user. **`Rejection` keeps structural and data staleness apart**, which is the whole reason ADR 002 put a maintained `RecordHash` index in `DataStore`: `RecordGone` means nothing resolves and there is nothing to force (**force never creates structure**), `RecordMoved` means it resolves at another version and force is one keystroke — which is also why the wasm client stores the `Rejection` itself rather than collapsing it into `status_msg`, since a string cannot carry which record or which kind. **And the ADR omitted the publication.** It said a batch carries the basis it was composed against and never said how a client learns one; nothing in the projection stream carried any of the three tiers, so the check as written was satisfiable only by forging. `Projection::Basis` + `BasisCache` close that — the server publishes the two document tiers plus the counters for the records *this client's grid names*, diffed against what that client was last told and seeded in the bootstrap — and it is emitted **after** the grid updates in a batch, because the counters describe the cells those updates carry: a basis arriving first would let a client claim a version for a value it had not been shown, the mirror image of gotcha 33. That ordering was documented as load-bearing in three places and **held up only by the order of two statements** until `a_published_basis_follows_the_grid_updates_it_describes` pinned it (proved by injecting the swap) — the same job `a_cell_update_that_overtakes_its_labels_is_lost` does for the mirror-image rule. Two things are unchecked and deliberately so: `Command::SwitchView` carries no basis (it sits outside `Command::Model`; the workbook refuses a view it does not have, so it cannot corrupt, but it is a hole in the rule), and `apply_command` does not check at all — **the check is a property of the session door, not of the mutation path**, which is right because a host holding the `App` is the truth rather than a claimant. That second one **was** a door rather than an exemption, and needed a type to stop being one (improvise-ondj): both took a wire `Command` and their `Command::Model { cmd, .. }` arm **discarded the basis with `..`**, while `ui/reduce.rs`'s own module doc pointed every future host — the browser worker included — at exactly those two names. **The struct variant closed omission *from the message*; it did nothing to close ignoring *on arrival*, which is the half a struct variant is usually reached for.** So both now take `Checked<'_>`, a private-field newtype over a `&Command` whose only constructor is `Checked::against(&App, &Command) -> Result<_, Rejection>` — also the only caller of `stale_basis`. `Sessions::apply_checked` receives the proof and `Checked::against` is answered **once, before the dispatch**, so `apply_switch`/`apply_model` receive the proof rather than each remembering to ask: an invariant made unrepresentable rather than validated, the same answer `prune_axis_items` and `gp-tree`'s `Row` got. Visibility was the weaker option and was rejected for a concrete reason — `pub(crate)` closes only the out-of-crate path, and `reduce_full` had **no production caller anywhere in the workspace**, so it would have become dead code. It has since been **deleted** — which is the argument’s own conclusion arriving, not a contradiction of it; the surviving sibling is `reduce_key`. This paragraph named it in the present tense for some time after, so read every function name here as a claim to check. **The regression guard is therefore the signature**: the test that reproduced the bug no longer compiles, and what remains asserts the constructor refuses and that refusing leaves the document untouched. One residual hole, documented on the type and filed: a `Checked` does not name *which* `App` it was checked against, so a multi-document host could in principle check one and apply to another. That is also why `ui::reduce::caught_up(&App, cmd)` is not a bypass: it takes `&App`, so nothing reachable from a wire can call it. **Since improvise-955n the frame, not the command, is what is answered — and only two of the three tiers hoist.** `CheckedBatch::against` runs before anything in a `ClientFrame::Commands` is applied. `document_generation` and `layout_version` describe the *document*, so answering them once for the frame is exactly right, and a first version hoisted all three — which was a lost update: the record tier asks "is the client's picture of **this record** current", and which records are at stake is a property of the *command* (`records_named`). Skipping it for later commands let `[SetCell(North), SetCell(South)]` overwrite a South somebody else had moved, silently, with the frame reported accepted, and invisible to the other two tiers because a `SetCell` is `CellData` and moves neither. So every mutation answers for its own records **against the frame's basis** — the one the first basis-carrying command stated, never each command's own field, or a later command could claim a fresher picture than the frame was composed under. The one exemption is stated over a *property* rather than a list of commands: a record that **neither resolves in the document nor is claimed by the basis** is one the frame is about to create (`[AddItem(East), SetCell(East)]`, which `BasisCache::claim` cannot possibly claim for), and only that combination is exempt; a record that does not resolve but *is* claimed is still `RecordGone`. That the `Basis` inside a *later* `Command::Model` is ignored rather than compared is an honest interim: clause 1 says the **frame** states one basis, and moving the field from `Command::Model` to `ClientFrame::Commands` would make that unrepresentable-otherwise instead of a convention — a protocol change that belongs with clause 4. Two things worth keeping from the rejected design: **arity is not a property of a variant** (a future `Command::RunScript` is one variant performing unboundedly many mutations), so no count over the vocabulary can enforce "one mutation per frame"; and **the transaction boundary is not the frame**, because one command may already be a chain, which is why clause 4's mechanism belongs at `App::apply_effects`. - -44. **The TUI grid draws from its cache, and since improvise-anr9 the cache frames it too.** `GridWidget` holds no `&Model` and no `&View`: cell text, aggregate-ness (`CellId::Aggregate`, ADR 002), column widths, the page bar, the totals strip and now the **frame** all come off `GridCache`. The TUI still builds that cache over the **whole** grid (offsets zero), so a cache position is an absolute position — the browser's `Viewport` is a real window, the TUI's is not, and the two clients therefore use it differently; column totals **must** be projected rather than client-derived, because the sum runs over every row *including the ones a viewport scrolled past*. The frame is an `AxisFrame` per axis — `cats` (the categories, outermost first) plus `entries: Vec` interleaving `Group(FrameGroup)` headers and `GroupEnd { level }` markers with `Item { tuple, collapsed }` label *tuples*, exactly as `GridLayout::row_items` interleaves them. That replaced a `Vec` built by `GridLayout::row_label`, which slash-joined the tuple — the exact rendering `two_row_categories_produce_cross_product_labels` says must not appear — so the browser drew `Food/Alice` where the TUI drew two header columns, with no groups and no collapse indicators. It is **one** type and not a `labels` plus a `levels` beside it: two answers to "what are the labels" is the disease (improvise-edp), and this is the second cure, which cost a *type* change rather than an addition. Three things to know before touching it. **`leaf_labels()` is not "the labels"** — it is the innermost label, what a one-line summary calls a row by; a draw loop wants `header_labels(first_rendered)`, `groups()` and `shares_ancestry(sel)`, each of which is one pass, because `item`/`label`/`shows_level`/`group_of` scan `entries` from the start and asking them per position is quadratic in the frame *and grows with the TUI's absolute scroll offset* (found three times on anr9; the per-position forms survive for one-off lookups and say so). **The width rule has one home**, `improvise_protocol::header_level_widths` — widest label per level, plus a gap, clamped — reached by `AxisFrame::header_widths` and by `ui::viewport::compute_row_header_width`, which borrows level tuples off a `GridLayout` rather than building a frame on the keypress path. **A `cats` change re-identifies every position**: a cell's identity is `cats` zipped with its tuple, so `invalidate_relabelled` blanks the whole axis when the categories move, not merely the positions whose tuples differ, and blanks the **totals** on a recategorization of *either* axis — a column total is a sum over the rows, so re-identifying the rows leaves a number answering coordinates the grid no longer has (gotcha 33(b): the rule lives in `apply`, not in a producer's memory). What is still **not** on the cache is `record_col`, and that is correct: it keys pending drill edits and is session-local (gotcha 1). `body_from(idx)` takes an argument only because the two front ends scroll differently — a browser's frame is already windowed and passes `0`, the TUI's covers the whole grid and passes its row offset — not because the rule differs. `group_of` answers group **membership**, and since improvise-e265 membership is **the innermost open run**: `FrameGroup` and `FrameEntry::GroupEnd` each carry a `level` (an index into `cats`), runs **nest**, a header at level L replaces L and closes every deeper run, and a `GroupEnd { level }` closes L and deeper. Before e265 it was the nearest preceding header unless a level-less `GroupEnd` intervened (improvise-gd83, PR #77), which could not express one run inside another. `GroupEnd` draws nothing and exists only so membership is not inherited. `AxisFrame` deserialises through `UncheckedAxisFrame` and **refuses** a `level >= cats.len()` — a level naming no category is a malformed frame, not a clamp. And `FrameEntry::Item` is `{ tuple, collapsed }`: `collapsed` is the **layout's own fold flag**, read by `AxisFrame::folds`, so a client asks the frame whether a position folds rather than re-deriving it from the groups (improvise-4k1j). Both of the gaps this sentence used to list are **closed**, and it named them as open for a day after: `group_runs` coalesces by **header identity** (`std::ptr::eq`), not by name, so two runs under two headers equal as values are two runs (improvise-h0iv); and `AxisFrame::first` carries **no** `serde(default)` — its doc comment says why, since zero is a claim and not an absence (improvise-1wl0). `doctrine(a_stale_list_of_what_is_missing_is_worse_than_no_list)` is about exactly this paragraph, twice now. And `project::grid_of` used to evaluate every cell per frame where the widget evaluated only the visible ones; since improvise-felo.13 the host's own layout keeps a `GridSummary` (whole-grid widths, column formats and totals, `ui/viewport.rs::grid_summary`, held in `App::grid_geometry()`), and `grid_in` evaluates only a `grid_window` over it. The browser's window therefore costs its own cells. A session on another view, or a rederived model, still takes the whole-grid `grid_body` path. `grid_window` equals `grid_body` where the summary is current. Since improvise-jytm `grid_in` may additionally **derive a `GridLayout`** for a session on a view the host is not on; the host's layout is reused where the names agree, which is the common case and the only one that can carry a drill snapshot. - - **`RecordHash` is document-scoped; `AggregateHash` is layout-scoped and cannot be otherwise** (improvise-gd28). A `RecordHash` resolves through `DataStore`'s maintained index — but **only when the command names a view the document has** (improvise-wpxr; Codex P2 on #281). `SetCell`/`ClearCell` carry `view`, and the lookup is **strict**: a view the document lacks drew nothing, stored or not, so it is `Rejection::RecordGone` rather than a `resolve_view` fallback. An **empty** record — one the store holds no cell for, which the grid still drew — resolves against **that named view's layout**, the only place it exists. `RenameRecordCoordinate` is the exception and resolves in the **store alone**, since only a record holding cells can be renamed: `protocol::named_records` answers each command's records as `NamedRecord::Drawn { view, record }` or `NamedRecord::Stored(record)`, one exhaustive match. Every one of these questions goes through `App::grids()`, a `Grids` borrow that memoises **one derived `GridLayout` per view for as long as the borrow lives** (improvise-kimw) — hold one across a check, never across a mutation, since the layouts it holds describe the model it borrowed. An aggregate has no existence outside the layout that derives it, so there is no index to fall back to and none can be built — `App::resolve_aggregate` therefore answers only for the **host's** active view, which since improvise-jytm is not the client's. **That, and not ADR 001 layer 2, is what blocks browser drill.** The ticket's blocking argument was right about the *commit* (`ApplyAndClearDrill` is addressed to one session, gotcha 38) and wrong about the *entry*: entering a drill needs no commit, and `CreateView`/`SetAxis`/`SetPageSelection` are already unaddressed `ModelCommand`s that name their view. The gap is one level down from where it was looked for — **the mutation and its operand have different scopes**: the mutation configures the shared `_Drill` and is addressed to nobody, while the operand exists only inside one session's layout. So the fix is not a new mechanism but the `view: String` convention every other view-scoped variant already follows, resolved against a layout derived for that name (improvise-obtu). Two things travel with it: editing *inside* a drill grid from a browser does not follow from it, because `edit_or_drill` declines records-mode editing out loud before reaching a cell (improvise-fdk1, since closed, is the separate records-coordinate rename on the wire — `RenameRecordCoordinate`, gotcha 38 — and did not unblock drill editing); and closing it makes improvise-t219 live, `_Drill` being one shared view in `Workbook::views` that two clients drilling would contend for — the same masking relationship improvise-6uva has with improvise-up8g. Guarded by `an_aggregate_a_session_was_shown_does_not_resolve_against_the_hosts_view`, whose **control** is the load-bearing half — the same surrogate resolves once the host is moved onto that view, so a blanket `None` cannot pass. - -45. **One session loop, two transports — and the browser saves itself.** `Host` is `App` + `Sessions` + `handle(id, ClientFrame) -> Vec` and lives in `improvise_command::ui::host`, **not** in `improvise-ws-server`, which re-exports it. It moved for improvise-9x6: the worker hosts the same authoritative `App` over a `MessagePort` and cannot depend on tokio, and a second copy would have been exactly the mirror improvise-xks6 had just deleted for the frames — a drifting `Applied`/`Broadcast` match is invisible until a client sees it. `registry` stayed on the server, because a worker hosts **one** document. What the worker adds is only what a `MessagePort` differs in: **one client by construction** (the `Addressed` filter is honoured anyway, against a future `SharedWorker`), and **a handshake with no close** — `ClientFrame::Viewport` must still come first, but a pre-handshake frame is *dropped* rather than closing a connection that does not exist. - - **Saving is a frame, and dirtiness is read off the classification the effect already carries.** `effect::Save` is `ViewOnly` (gotcha 3) — correctly, since it writes *this host's* `file_path` through *this host's* `Storage` — so the derivation that builds `ModelCommand` from the model effects cannot name it, and for a while that meant no client on any transport could ask its host to save. `ClientFrame::Save` closes it (improvise-368h): a **request** carrying no path, because a client cannot see the host's storage, answered `ServerFrame::Rejected(NoDocumentPath | SaveFailed)` to the asker alone and **silently on success** — nothing moved, so no widget is dirty. It is a frame and not a `Command` for one concrete reason: as a `Command` it put a runtime-fallible operation inside a batch `ClientFrame::Commands` promises is all-or-nothing, so `[Save, SetCell]` reported a failed write and then applied the edit. It also fails the test every `Command` passes — it mutates no shared state and no client's slice, so `reduce_view` has nothing to fold and `Checked` has no basis to answer for. **And `MarkDirty` is gone.** It was a separate effect the *command* layer composed beside each mutation — reasonable on the key path, unreachable from anywhere else, because improvise-kmgs derived `ModelCommand` from the effects **one at a time** and so put `SetCell` on the wire without its companion (improvise-gv9r): a derivation correct per element and wrong for a set, whose symptom was that "save if dirty" never fired. `App::apply_effects` now raises the flag from the effect's `EffectKind`, which makes omitting it unrepresentable rather than validated — **declared** for `WholeWorkbook` (a replaced document differs from the file its path names, so a `LoadModel` that cannot read its path still raises it, costing one redundant write) and **observed** for `CellData` and `LayoutGeometry`, where `document_content`/`document_config` fingerprint the document before and after, so a write that leaves it as it was raises nothing (improvise-748f). The observation is read inside the loop, and `Save` clears the flag in loop order, so a batch that edits and then saves ends clean. The worker's persist trigger is therefore the **flag**, not a table of command variants: an observation rather than a second description of which commands change the document, for the reason `cursor_epoch` beats matching on `CycleAxis` (gotcha 24) and `document_generation` beats matching on `LoadModel` (gotcha 39). It is also strictly tighter — a refused batch raises nothing, and neither does a write that changes nothing, so the "was anything rejected" guard went with it. A browser still never *sends* `ClientFrame::Save`, and that is the answer rather than a gap: there is no file the user owns and no Ctrl-S semantics, the document *is* the OPFS entry, so the worker persists on its own behalf through the one existing `Save` effect. The cost that remains: `Storage` is whole-document (gotcha 41), so OPFS rewrites the file per accepted model command. - - **The async seam does not survive being boxed.** `OpfsFs::open`/`attach`/`refresh`/`commit_deletes` are on the struct, not on `Storage`. `App::storage` is a `Box` and a trait object can name none of the four, so **handing the backend to the `App` puts the async seam permanently out of reach**; the host must hold the concrete backend *and* give the `App` a `Storage`, which means sharing one. That is `impl Storage for Rc`, and it lives in **`improvise-io`** rather than in each worker (improvise-hmws): `impl Storage for Rc` is two foreign types in any other crate, so improvise-9x6 wrote a forwarding newtype that every future worker would have written again. `Rc` and not `Arc` because the one caller is a Web Worker and wasm32 has no other thread. **The flat root is no longer load-bearing**: `OpfsFs::list` joins `dir` like the other two backends, so `storage_contract` takes `dir` as a parameter and the three agree at *every* directory rather than only at `Path::new("")` where joining is the identity (improvise-vhcz). What the index holds is **documents, not entries**, and the split between its three writers is the rule: `attach` indexes nothing (what it creates is zero bytes), `Storage::write` indexes the name and un-queues any pending delete of it, and `refresh` rebuilds from OPFS skipping zero-byte entries — counting an entry it cannot measure as a document, since the one cause of that is another tab holding the exclusive handle (improvise-vvl3). - - **What no test here can see** has shrunk, and precisely. `harness/` (improvise-vvl3/xf1) runs `improvise_io::storage::contract::storage_contract` against a real `OpfsFs` inside a real dedicated Web Worker under Chromium and Firefox, and the `browser` job in CI can fail the build — so **`OpfsFs` is no longer compile-checked-and-never-run**, and two bugs fell out of its first execution. That is why `storage_contract` stopped being a `#[cfg(test)]` helper and became a public function returning `Result`: a browser must run *that* sequence, not a second transcription of it, and a failure has to travel out of a worker as a string. Still unseen: **Safari** (Playwright's WebKit has no OPFS, so xf1's criterion is met for two engines of three; `opfs.spec.js` skips on a runtime probe and an unexpected skip on any other engine is a failure). **`bindings.rs` is NOT in that list, and said it was for months.** `shell.spec.js`'s "the standalone deployment" loads the app with no `?ws=`, which spawns the worker, which imports the glue, which calls `bindings.rs::start` and `OpfsFs::open` — and the spec then asserts a grid draws, so the whole chain has run in CI since improvise-vvl3 (Chromium, and Firefox since 2026-09-20). The claim survived in **five** places (this line, the File Inventory, two `kb.pl` facts and improvise-bck's grooming note) because each was written before the harness landed and none was re-read after; the spec's own comment says the chain plainly. It is the reason to distrust a "never run" written in prose: nothing checks it, and `kb.pl`'s CI gate checks consistency rather than truth. What was already verified stays: `tests/post_message.rs` drives a real `Worker` over a real `App` on nothing but the JSON a `MessagePort` carries. Build note unchanged: the worker bundle needs `--enable-nontrapping-float-to-int` **as well as** `--enable-bulk-memory`, because it carries the whole `App` and so chrono's date arithmetic. - -46. **The page picks a peer, and the demo was built against an assertion.** `web/app.js` holds two transports and no protocol knowledge: `?ws=` selects `improvise-ws-server`, its absence spawns `new Worker('./worker.js?doc=&seed=', { type: "module" })` (improvise-djm). `type: "module"` is mandatory — `worker.js` `import`s the bundle — and there is **no `MessageChannel`**: a dedicated worker's own `postMessage`/`onmessage` *is* the port, so the ticket's channel wording was a date rather than a decision. The websocket path is **kept**, because the ws-server is how a TUI-hosted document is shared and nothing else in the browser can reach it; the cost is contained because the two differ only in *where strings come from* — handshake, fold, repaint and keyboard are all above the seam — and **the presence of the parameter is the whole selection mechanism**, a second knob being a second thing to get wrong. `#status` composes **two** facts, the transport's and the document's, because a single source let a repaint silently erase a connection error. **A worker that rejects in `start()` fires no `error` on the main thread** — `worker.js` catches its own — so the guard is a **first-frame watchdog**, deliberately *not* per-transport: "the peer never answered the handshake" is the one failure that means the same thing over a socket and over a port, and a check aimed at each peer's own failure modes would have missed exactly this one. - - **The demo is a model built to satisfy a check, not an axis flipped and hoped for.** `examples/demo.improv` leaves `Customer`/`Date` on None, so every cell is an `Aggregate` and nothing is editable — and the naive fix does not work either, since `Product × Region` by `Date_Month × Measure` still aggregates. `web/demo.improv` puts **exactly one category on each visible axis** (`Product` row, `_Measure` column, `Quarter` page, nothing regular on None) and carries `Profit = Revenue - Cost`, so the payoff is on one screen with no keystroke. **Settability is asked of `grid_of` as a `CellId` variant**, never re-derived from the axes — a second derivation could agree with itself while disagreeing with `render.rs::editable`, which is the whole point of ADR 002 — and the assertion was proved discriminating by pointing it at `examples/demo.improv`, where it names the aggregate at (0,0). **One category per axis is a property of the demo, not of the format**: when this was written `GridCache` slash-joined a multi-level tuple, so a second category on an axis put that known gap on the front page — improvise-anr9 has since closed it (gotcha 44), so what remains is the demo's own simplicity rather than a rendering limit. `web/*.improv` is un-ignored in `.gitignore` (`*.improv` is ignored repo-wide) and the CI assemble step copies it: leaving it out is an empty grid, not a 404 the page could report. One wart it now puts on the front page: a formula cell **is** a `Record`, so it is offered as editable and a write to it reverts on recompute — pre-existing and matching the TUI, newly visible. - -47. **Keys become commands in the browser, and the server must not know the keymap.** The user's ruling on improvise-bpq2, following from "the model is shared, the UI is not" — and it buys two clients different bindings over one document. What made it urgent is **not** latency. `a_key_resolved_on_the_host_moves_state_no_client_can_hear` shows nothing in the projection stream carries the host's own `ViewState`, so host-side resolution **diverges**; and `Host::handle` answers `ClientFrame::Key` with an explicit empty vec, so **an upstream key is a black hole, not a slow path** — every `KeyOutcome::Upstream` names a binding that does nothing at all in a browser. That test still holds and was not flipped: bpq2 closes the divergence by *avoiding* it. **The method is the ticket's** — enumerate the **bindings**, not the commands (`doctrine(binding_dump_diff)`) — and where a client cannot compose the `Command`, **enrich the projection** rather than moving resolution back. - - Three things that cost. **`GridExtent` on `GridCache`** — the whole grid, of which the labels are one window — because a viewport-indexed cache cannot tell "the last visible row" from "the last row". It is deliberately **not framed by the labels** (`fit_to_labels` leaves it alone) and its batch position is not load-bearing the way `RowFrame`-before-`Cells` is, since it addresses no position; the producer reads `GridLayout::row_count()`, and reading it off the projected ranges is **invisible in the TUI** (which builds at offsets zero, gotcha 44) and is exactly the browser bug. **The window is the client's own fact**, recorded by `Client::set_viewport` from what it announced — *not derivable*, since the row frame's own length is the window **clipped to the grid** and so differs precisely at the bottom edge where the scroll arithmetic matters. `Client::viewport` (associated, no recording) still exists and is the one that quietly turns cursor keys off; `web/app.js` does call `set_viewport`. **`ClientFrame::Commands(Vec)`** replaces the singleton, because one cursor key is `SetSelected` plus the offsets `viewport_effects` returns and a host that folds the selection but not the offset projects the wrong window. It is **answered whole, then applied whole** (improvise-955n): `CheckedBatch::against` checks the frame before anything runs, so there is no longer a moment at which part of a frame has landed and the rest has been refused, and `Applied::Rejected` is unreachable through `Host::commands` — still matched, because a host that wants the `Dirty` must name every case (gotcha 39). The contract is four clauses and **all four are now delivered** — clause 4 by improvise-yqpd on 2026-09-21, whose mechanism is `App::snapshot`/`restore` plus a `Sessions` clone taken in `Host::mutating_frame` and put back by `undo_frame`, deliberately *not* inside `apply_effects`, which runs on every keypress. **This paragraph said it was undelivered for a day after it landed**, and an orchestrator brief repeated that from here, so read the clause list as a claim to check. A delivered clause 4 is also what makes the CLIENT's rollback well-defined (improvise-nxah): a refusal means the host applied none of the frame and restored every mirror, so a client keeping its optimistic fold is the only thing left out of step. What rolls back there is `ViewState`, never a cache — a `Command::Model` is a no-op in `reduce_view`, so the optimistic fold never reached one. The old text, kept because the hazard it names is still the right shape to fear: one basis per frame, checked once, applied in order with each command seeing the last one's state, and all-or-nothing. was that the fourth was open because `App::apply_effects` aborts without undoing (gotcha 21 — an abort is not a rollback), so `[RemoveItem("North"), SetCell(the North record)]` would land the removal, resolve the write to nothing, and report the frame accepted having applied half of itself — partial *and* silent. That is what `yqpd` closed, and it closed it **outside** `apply_effects` for the reason the old text implies: the abort lives on the keypress path and must stay cheap. The old note that it was *unreachable* from any client here, the only multi-command producer being `keys::commit`, still reads correctly as a caution — **reachability is a reason not to rush a fix and not a reason not to notice one**, the lost update `CheckedBatch` fixed having had the same reachability argument and none of the documentation, which is how it shipped. - - Two corrections to the ticket's own gap list, both worth more than the code. **Panel cursor movement was never a gap**: improvise-edp had already made `viewmodel::flatten_cat_tree` the one home both ends call, so it is the *shared* derivation, not a second one — worth re-checking the "known gaps" in other old tickets against what has landed since. And the "unmodified characters only" rule was **mis-scoped, not wrong**: it was about `KeyPattern::AnyChar`, which is bound only in the text-entry modes, so `Ctrl-d` in Normal has an exact binding and mirroring it is a faithful subset. Each resolver mirrors its `Cmd` **line for line, including where siblings disagree** — `Move` and `MoveTileCursor` emit even when they clamp, `MovePanelCursor` returns nothing when the cursor does not move. Tidying that into one rule would make the client a second derivation, which is what `tests/lockstep.rs` exists to rule out. What still needs the host is **one list**, `resolve::NEEDS_THE_HOST` in `improvise-protocol/src/resolve.rs` (improvise-uln5) — this line named a list in `keys.rs` and a test called `the_normal_mode_keys_that_still_need_the_host_are_these` for some time after both had gone. Three tests hold it, one per direction: `the_command_names_that_still_need_the_host_are_these` (no **false** entry: each name is bound and really is silent), `a_help_row_the_browser_cannot_run_is_drawn_marked` in `wasm-client/render.rs` (the help overlay **marks** those rows, improvise-uln5), and `every_silent_binding_is_on_the_list` (every binding of **every mode a browser can reach** acts, declines out loud, or is on the list — improvise-oyaa, improvise-s2gb). The last replaced a check read off the help rows, which could not see a binding no help row names. Closing a gap therefore means deleting its entry, and the first test fails until you do. - - **What the browser client decides before a frame leaves it** (all in `improvise-protocol`, so both ends share it). Five rules, each bought by a bug: - - **Every client cache read goes through `projected_at`**, which subtracts the frame's own origin (`AxisFrame::first`) before indexing. A position outside the window the cache holds is **unknown**, never the cell at the same cache index (improvise-7mro) — once the client scrolled ahead of its cache, an index read answered a different cell's value. `unprojected` names that state, and a cell op on it waits rather than guessing. - - **The keyboard commit writes `ClientState::opened`**, not the cursor. `OpenedCell` captures the record, the `OpenedTarget` (`Value { measure }` or `Coordinate { category }`), the record's version, the `document_generation` and the view **at open**, so a commit lands on the cell the editor opened on however the cursor moved since (improvise-8xq0), and claims the basis it was **opened** against rather than one read at commit, which silently overwrote a value another client wrote while the editor was open (improvise-xz2z). A retry after a crossing `LayoutMoved` must match the **whole** `OpenedTarget`, not the record alone, or it wrote whatever cell was under the cursor by then (improvise-8phw for values, improvise-9ww8 for renames). - - **`refused_here` exempts a captured editor** (improvise-3h9w): with `opened` set, `commit-cell-edit` is answered by the capture, so the cursor's current cell cannot refuse a commit aimed elsewhere. - - **A decline that must stop a binding is `Step::Declined`**, not a `SetStatus` beside the binding's other steps: Enter's welded `clear-buffer` otherwise still ran and threw away the typed text (improvise-joed), and a touched editor over a declined cell went `Upstream` with no word at all (improvise-ik6l). - - **`ClientState::notify` queues a client-side notice at the tail of `in_flight`**, as if it were a sent frame, so a rebase that rebuilds the view from `settled` and replays `in_flight` replays the notice too instead of erasing it (improvise-n0pb). `settle_status` runs on **every** path that rebuilds the view (bootstrap, accept, refuse, relay), and assigns a message the editor open as it arrives, so an old message loses its owner once that editor closes (improvise-0dbk). `EditorSession`, the key it is owned by, lives in `improvise-protocol/src/lib.rs` beside `ViewState` so both front ends ask it (improvise-su4u, improvise-1x9h). - -48. **The browser JavaScript is argued *and now partly executed*, and the two rules it can break are both about *when*.** `web/app.js` and `web/worker.js` were syntax-checked, `.d.ts`-checked for call arity and read — never run — and every bug in them was found by reading. `harness/tests/shell.spec.js` is the first execution of either: it runs `app.js` under Chromium against a mocked peer, and it exists because PR #21 declined to add regression tests for its own close-listener fix on the ground that there was nowhere to put them. That decline is only honest because the assertions live there now, and **a change to `web/*.js` belongs in that spec directory** rather than in a second ad-hoc surface beside it. Playwright rather than a headless-DOM shim because what goes wrong in these files is a real Worker, a real WebSocket, real OPFS and real event ordering, none of which jsdom has. Two classes of bug still follow from *when*, both found by review. **An `async` `message` listener does not serialize.** `worker.js`'s `handle()` awaits `server.flush()`, so the event loop may start the next message while an earlier one is suspended, interleaving two `on_message`/`flush` pairs and letting a later frame's replies **overtake** an earlier frame's — which gotcha 33 answers by *dropping* the overtaken update, not by replaying it. Every frame is therefore chained onto **one tail promise** and the listener is deliberately not `async`; that also turns "frames that arrived before `start()` finished" into an ordinary case of the ordering rule instead of a drain loop, which had the same race twice over because `server` was assigned *before* the loop ran. **A watchdog must be disarmed, not pre-empted.** The handshake watchdog exists for one condition — the peer never answered — and every *loud* failure (worker `error`/`messageerror`, socket `error`) has already written something more precise; the timer then fired 8s later and replaced it, so `setShellFailure` clears it. Its neighbour, the socket's `close` listener overwriting a precise `error` with "disconnected", **is fixed** (improvise-taxx, PR #21) — and the ticket's own proposed mechanism was wrong: `readyState` cannot discriminate, an `opened` flag is the only thing that can. PR #21's *other* half is the one that keeps getting lost: it routes **both** close arms through `setShellFailure`, which disarms the watchdog, so a genuine mid-session disconnect is **not** overwritten eight seconds later. A bug report said it was, measured it honestly against a pre-#21 tree, and then "confirmed" it survived #21 by simulating the `opened` flag **and not the routing** — a simulated fix that is not the fix you named reproduces a bug the real code does not have, and reads exactly like a confirmation. The ticket was filed and closed invalid; the property is now asserted by `a socket that opened and then dropped reads as disconnected, and stays that way`, which runs past the 8s window, rather than argued in a comment that was wrong twice. - - **What the harness still cannot see**, and why the argued-not-tested discipline stays: the grouped and multi-level DOM, the CSS, and the `ResizeObserver` extent arithmetic. What a careful read established about the last of these — the observer cannot feed back into itself (`#grid` is a flex child with `overflow: auto`, so `replaceChildren` cannot resize it) and `ROW_PX`/`COL_PX` under-estimate, so the worst case is a row of over-fetch — is still a read and not a test. - - **A third thing no test here can see, which is why the docs must be read as code.** Three comments in one range said "not built yet" about work that had landed *in that same range* — `README.md`'s browser section, `app.js`'s keyboard note, and `Host::handle`'s `Key` arm — each because a follow-up commit updated **one copy of an argument and not its siblings**. When a fact is stated in more than one place, changing it is a grep, not an edit. - - **An evidence heading may be module-qualified, and must be when its leaf is not unique.** `scripts/check-comment-runs.py --check-evidence` takes its Rust names from `cargo test --workspace -- --list` minus the `--ignored` twin, as **full module paths**, and **refuses** an ambiguous leaf naming every candidate rather than warning. Six leaf names collide in this workspace. Note neither half suffices alone: rename one of a colliding pair and the leaf is unique again and resolves silently to the survivor, so qualification is the fix and the refusal is only what forces someone to write it (improvise-rm4s). The residual, filed: `--list` does not name the binary, so two crates sharing a module path are indistinguishable (improvise-sxsf) — **measured latent** (1444 names, zero collisions) and the proposed per-package fix **declined twice over**: 18× the cost (45.2s against 2.4s, and `--check-evidence` needs the `--ignored` twin, so 22 cargo invocations), and *a package is not a binary* — `improvise` alone has five integration test binaries beside its lib, and an integration test's top-level `fn` lists with **no module path at all** (60 of 1444 names are bare), so `-p improvise` prefixes all six identically. Per-**target** is the only shape that would close it. - -49. **A websocket handshake is exempt from the same-origin policy, and that is a server-side problem.** There is no CORS preflight, so any page a user visits can open a socket to `improvise-ws-server` *from that user's own allowed address* and read and rewrite every document it holds — cross-site websocket hijacking (improvise-kc7n). **No network control can see it**: an IP allow-list, a private network and loopback binding all answer "may this **client** reach the endpoint", and the honest answer is yes; only `Origin` says what is **driving** the client. **Loopback is the least protected case, not the safest** — no proxy in front of it, and loopback is not a boundary a browser enforces. `src/origin.rs` is the whole decision and `transport::admitted` its one caller; it refuses the **upgrade** with a 403 rather than accepting a socket and ignoring it, and `serve` **takes** the policy so a host cannot forget to apply it. Three arms are decided rather than defaulted: **loopback allowed** (the shipped local recipe serves its page from `http://localhost:8080`, so the one case worth optimising for must need no flag), **absent allowed** (no browser omits `Origin`, so absent is a non-browser client an address check already answers — and refusing it breaks `tests/socket.rs`, the repo's own instrument), and **two refused** (`HeaderMap::get` is first-wins, so a proxy that *appends* rather than sets leaves two, and answering from the first lets the second be anything). The deployed Pages origin is deliberately **not** hard-coded: it can only reach a plain-`ws://` server through a TLS proxy anyway, so one flag is an honest cost on an already multi-step path where a baked-in default would be a permanent hole for every other deployment. The residual hole is **stated rather than guarded** — a proxy that *strips* `Origin` turns the check off, which is why the proxy's own allow-list stays in `README.md` as defence in depth. **None of this is authentication**: it answers what page is driving a client, never who is connecting, and improvise-kpnr is still open. It correctly does **not** move to `improvise-command` beside `Host` (gotcha 45) — a `MessagePort` has no handshake and no headers, and a worker and its page are the same origin by construction, so this is genuinely transport-only and not a mirror waiting to drift. - - -50. **A composed TUI row measures in cells and names which side yields.** `draw.rs::fill_line` takes a `Keeps`, because the three rows that share a line genuinely disagree about it: the title bar keeps the **right** (the `?:help :q quit` escape hatch outranks an unbounded document name — improvise-t14d), while `minibuffer_line` and `draw_status` keep the **left**, which is what they already did, since right-priority there would let a long status message eat the text the user is editing. So the ticket's "give the right segment priority" was correct only for the title bar, and a global rule would have been the wrong shape. Widths go through `ui::text::truncate`, never `str::len` — and the ticket framed *that* half wrongly too: the title's own `·` separator is two bytes and one cell, so a byte count mis-padded **every** document, not only one with a non-ASCII name. - - **One key table on screen, and it is the one derived from the keymap** (improvise-rr8n). `TileBar` now advertises no keys at all: its literal read `Ctrl+↑↓←→ to move tiles` where those keys are bound to `enter-tile-select`, and the derived status line is a strict superset (`Space`, `h`/`l`, `Tab` appear only there). The ticket asked for the opposite — rewrite the literal, suppress the derived line — and was inverted deliberately, because suppressing a derived superset to preserve a literal that had already rotted keeps the stale table and deletes information. The **import wizard goes the other way**, and the asymmetry is the rule rather than an exception: its status line *is* derived, but from the **mode** keymap while the real keys are per-step inside `HandleWizardKey`, so there the derived table is the one that lies and `draw_status` suppresses the fallback for that one mode. **Keep whichever table is derived from the thing it describes.** Two blind spots make this class invisible: `every_hint_key_resolves_in_its_mode` **cannot see a string literal in a widget** (the same gap improvise-wk18 names for `help.rs`), and the widget's own characterization test renders at **80 columns**, where the hint does not fit and is never drawn — so the one test pinning that output was structurally incapable of seeing either hint. - - **A scroll is measured where it is drawn** (improvise-gtn7.1). The keypress path counted `visible_cols` from `term_width` while `draw_content` gave the grid `term_width` minus the 32-column side panel, so with any panel open the cursor scrolled to a column never drawn. Now `viewport::grid_width(term_width, side_panel_open)` is the one answer, reached through `App::grid_width`, and both `draw_content` and `App::cmd_context` read it; `App::side_panel_open` is the one answer to whether a panel took its column. The tile bar had the same shape: `first_tile` chose its scroll against a width that began one or two columns before where the tiles were drawn, and both now start at `tiles_left(start)`. The grid bug was latent from `b5418f2e`, when `compute_visible_cols` took `term_width`, and was exposed by `50e4a372`, which opened the category panel by default. Still open: columns of mixed width, where a `visible_cols` count taken at the old offset is applied at the new one (improvise-gtn7.2). - - -51. **The command language reads pipes, refuses surplus, and checks names when it applies (improvise-31oe.2, 31oe.3).** Reasoning: `context/code/cli.md` §14 and §15. (a) A `:` argument is read through `improvise_formula::quote`, so `create-view |By area|` names `By area`; a **surplus argument is refused** rather than ignored, and `cmd::core::only_when_omitted` gates the *cursor fallbacks* of `clear-cell`, `yank`, `paste` and `drill-into-cell` so a coordinate typed by the user is never overridden by the cursor. (b) `set-axis`, `set-page`, `toggle-group`, `hide-item`, `show-item` and `switch-view` return `RequireKnown(Needs)` (`effect.rs`) ahead of the real effect, built by `effect_cmds.rs::unless_unknown`. It is `ViewOnly`, writes nothing, is **off the wire** (listed in `vocabulary_is_complete.rs`) and checks the **document at apply time** — a `:` line builds every segment before any runs, so a check in `execute` refused `create-view Second . switch-view Second` — then refuses and aborts the batch (gotcha 21). (c) A refused name therefore stops the rest of a `:` line, while headless `improvise cmd` rebuilds the context per segment and runs each as its own batch (improvise-31oe.14, open). - -52. **A date column imports as one hierarchical category, and the import refuses what it cannot keep (improvise-8jk6.3, 8jk6.6, d0n6).** Reasoning: `context/code/formula-and-io.md` ("Import: …") and `cli.md` §13. The default `DateShape` nests Year > Quarter > Month (the chosen levels) in ONE category named after the column and opens folded to its years; `DateShape::Split` is the old flat split, chosen by `--split-date ` headless or `s` on the wizard's date step. `Plan::spelling_clash` refuses a raw, unparsed date spelled like a generated item or group (`2024`, `2024-01`) because the two would silently merge — asked at the wizard's ConfigureDates step **and** again at build time, so a headless import cannot reach it either. Headless, `--strict` turns a partial skip into a refusal before the build, and a folder whose CSVs are all header-only (no records) is refused as an import that kept no record. +1. **Records mode is wide format.** A row is a coordinate tuple; columns are `[coords…, +cat, raw measures…, +]`. Editing a category column moves every measure cell in the row. `_Dim` carries a column *index*, resolved by `GridLayout::record_col`. `GridLayout::new` does **not** prune; `with_frozen_records(m, v, None)` does — production code wants the latter. `prune_axis_items` takes a predicate and re-marks survivors. +2. **Effects never touch `std::fs`.** Documents go through `App::storage: Box`. Autosave and the headless final write still bypass it (improvise-31oe.11). +3. **Every effect declares an `EffectKind`; there is no default.** `ViewOnly` never reaches other clients; `Model(CellData | LayoutGeometry | WholeWorkbook)` does. The view lives inside the workbook, so axis/page/format are `LayoutGeometry` and cursor/scroll/`SwitchView` are `ViewOnly`; `Add/RemoveCategory` are `LayoutGeometry`. `EffectKind` is too coarse to check the `ModelCommand` mapping — `every_model_command_names_the_effect_of_the_same_name` does. +4. **Commands never mutate.** `&CmdContext` is read-only; return effects. +5. **`CellKey` is always sorted.** Use the constructors. +6. **No `Model::add_item`.** Use the `Workbook` facade (`wb.add_item`) or go through the category. +7. **Virtual categories.** `_Measure` items = explicit ∪ formula targets (`measure_item_names()`). Cell aggregation (`apply_agg`) and formula `SUM/AVG` (`eval_agg`) differ on empty input; non-numeric values reduce by unanimity, else `Error("mixed")` — never dropped. `evaluate_aggregated` takes `Aggregation::{Exact, OverHidden}`. +8. **Display rounding is view-only.** `format_f64` only in rendering; never feed a rounded value back into eval. +9. **Formula eval is fixed-point.** `recompute_formulas` iterates to stability; cycles become `Error("circular")`. +10. **Drilling a formula cell** strips the `_Measure` coordinate so the raw feeding records come back. +11. **Minibuffer keymaps come from `TextEntry`** (eight text-entry modes); the buffer a prompt names and the keys edit are one string by construction. Read/write a runtime-named buffer with `read_buffer`/`write_buffer`. `SearchMode` is hand-written. +12. **Panel keymaps come from `PanelKeymap`**; `Panel::ALL`/`name()`/`key()` drive the toggle keys. +13. **A prefix key is stored as an address** (`ViewState::transient_prefix`), resolved by `Keymaps::prefix_keymap`; prefixes must stay flat (`prefix_keymaps_are_flat`). `ViewState` stays free of `improvise-command` types. +14. **Keybindings are per-mode** (`ModeKey::from_app_mode`); minibuffer Enter/Esc are `Binding::Sequence`. `every_hint_key_resolves_in_its_mode` checks which keys a hint names, not what they do. +15. **Registry names come from prototypes, not strings** — except the `Move` literals and four aliases, guarded by tests; a wrong-variant prototype shows up as a *duplicate* registration. +16. **`effect_cmd!` / `simple_effect!`** wrap one effect; `simple_effect!`'s kind argument is mandatory. Six wrappers return `unless_unknown(...)` (gotcha 51). Mode entry is `enter-mode `; no per-mode shim commands. +17. **One float epsilon** (`FLOAT_EQ_EPSILON = 1e-10`) for `=`/`!=` *and* the div/0 guard, in both evaluators. +18. **Order is data.** `IndexMap` order is persisted; an index of `HashSet`s is for lookup, never iteration (`DataStore::matching_*` sort to store order; `analyze_records` is first-seen order). A group's fold reads through the same rule (`spanned_positions`). +19. **Cell-edit commit paths split** (`commit_regular_cell_value` / `commit_plain_records_edit`); keep both in sync. `cmd::core::uneditable` is the one decline, asked at open *and* commit. `CellId::NonItem` is a stored value naming no item. +20. **Minibuffer commits go through `commit-buffer `**; the add effect is emitted last so its failure message wins. +21. **An abort stops the chain, never the bookkeeping.** `Save`/`SaveAs`/`AddCategory` set `abort_effects`; tidy-up (mode restore, buffer clear) goes at the *head* of a batch. An abort is not a rollback (gotcha 47). +22. **The import wizard's back path is a history stack.** Preconditions live in `validation_error()`; `ResolveDuplicates` appears only when `collisions()` is non-empty; headless defaults to `DuplicatePolicy::Refuse`. A collision is a cell written by two records. +23. **`app.layout` goes stale mid-batch** — put a `RebuildLayout` effect before any effect that reads the layout after a grid change. The rebuild is skipped when nothing it reads moved (felo.13); a writer bypassing effects must call `rebuild_layout()`. +24. **The cursor is per-client, keyed by view** (`ViewState::cursors`; use `App::cursor()/cursor_mut()`). Replacing the document clears the map and bumps `document_generation`. Cursor invalidation is *observed* from the grid frame (`cursor_epoch`), not called by effects; two arms (`LayoutGeometry` hashes the active view, `CellData` every view). +25. **`ViewState` and its field types derive `Serialize`/`Deserialize`** — no borrowed-`'static` fields anywhere it reaches. +26. **Axis presentation lives on `Axis`; colour is a `Face`** resolved by the `Theme`; `draw.rs::face_style` is the one translation to ratatui. +27. **`Axis::Filter` constrains without being navigable.** Visible data = `filtering_cats()`; user-actionable = `categories_on(Page)` / `tileable_cats()`. What a virtual view must be is model-side (`records_steps`, `drill_steps`). +28. **The browser backend is sync on purpose, with an async seam** (`OpfsFs`; `open/attach/refresh/commit_deletes`). It needs a *dedicated* worker — `createSyncAccessHandle` is absent in a `SharedWorker`. +29. **"Active view" is stored twice and may differ:** `Workbook::active_view` is the default, `ViewState::active_view` is authoritative per client. `App::resolve_view` is the one fallback; `Sessions::adopt_deleted_views` handles a deleted view. +30. **Two tests gate `ViewState` membership:** portable (no `Rc`/`Arc`) and snapshot-safe. `wizard` and `drill_state` live on `App` as session-local residue. +31. **`:` commands settle their mode by observation** (`ReturnToNormalUnlessMoved`), not by a per-effect constant. +32. **There is no aggregate render cache.** One cache per widget (`WidgetCache`), a `Projection` is one update to one widget; bootstrap is a *compaction* (`Snapshot { view, seed }`). The wizard has a cache but deliberately no projection. +33. **The grid's labels frame its cells; the fold is not commutative.** A `Cells` update that overtakes its frame is dropped; `apply` invalidates relabelled positions itself; every arm touching labels or widths re-fits. Don't coalesce a batch. +34. **A formula target is unquoted; pipe quoting has one home** (`improvise_formula::quote`). The keyword guard is canonical. `header_name` is total; duplicate `## Category:` sections are one category; `WHERE` needs `sp+`. +35. **`Dirty` over-reports on purpose** — a redundant projection is cheap, a missed one leaves a client wrong forever. `LayoutGeometry` includes `formulas`. +36. **A client's command has one door:** `Sessions::apply_checked(app, id, Checked)`. Assert the `Broadcast` itself, and seat a bystander where a wrong relay can move it. +37. **`gp-tree` returns data and never prints; a fit needs seed *and* `FitConfig`.** A scorer takes `Row` slices. `fitted_formula` refuses features the document cannot name; categories are forced `Categorical` (zv8t). +38. **The wire vocabulary is derived twice** (`Command` view half, `ModelCommand` model half). Completeness is checked (`vocabulary_is_complete.rs`). `ApplyAndClearDrill` and `AcceptWizardModel` are deliberately off the wire (session-local operands). Recount, don't cite. +39. **`Applied` is an enum (`Diff | Rebased | Rejected`)** because a document swap is not a diff; the trigger is the generation moving, not the variant. A swap fixture must keep the view *name*. +40. **A batch's basis is three tiers** (`document_generation`, `layout_version`, `record_versions`); the unit is the record; lower tiers are *observed*, not read off the effect. +41. **`Storage` is not a VFS;** SQLite was declined on measurement (a second serialization is a second grammar). +42. **Browser render decisions are made in Rust** (`RenderModel`: `editable` from the `CellId` variant, selection reconciled with offsets, `flatten_cat_tree`). `improvise_protocol::frame` is the one envelope pair for both deployments. +43. **A batch carries its basis, and the server publishes it** (`Projection::Basis`). `Basis` has no `Default` or public constructor; `Checked`/`CheckedBatch` make "ignoring the basis" unrepresentable; `Rejection` keeps `RecordGone` / `RecordMoved` / `RecordUnstated` apart. Anti-staleness, not security. +44. **The TUI grid draws from `GridCache`**, which frames it too (`AxisFrame`; width rule `header_level_widths`). Use the one-pass accessors, not per-position scans. `RecordHash` is document-scoped, `AggregateHash` layout-scoped. +45. **One session loop (`Host`), two transports.** Saving is a request frame (`ClientFrame::Save`); dirtiness is raised from the `EffectKind`, observed for data. `impl Storage for Rc` lives in `improvise-io`. `harness/` runs `OpfsFs` for real; Safari is unseen. +46. **The page picks a peer:** `?ws=` → socket, else a module `Worker`. `#status` composes two facts; the first-frame watchdog is per-peer-agnostic. The demo model is built so every cell is a `Record`. +47. **Keys become commands in the browser;** the server must not know the keymap. `GridExtent`, `set_viewport`, `Commands(Vec)`, `resolve::NEEDS_THE_HOST` and the five client rules (`projected_at`, `opened`, `refused_here`, `Step::Declined`, `notify`). +48. **The browser JS is partly executed** (`harness/tests/shell.spec.js`); changes to `web/*.js` belong there. Serialize frames on one tail promise; disarm, don't pre-empt, a watchdog. Evidence headings may be module-qualified (and must be when the leaf collides). +49. **A websocket handshake is exempt from same-origin policy** — `src/origin.rs` refuses the upgrade (loopback and absent allowed, two refused). Not authentication. +50. **A composed TUI row measures in cells and names which side yields** (`fill_line` + `Keeps`). Keep whichever key table is *derived from what it describes*. Scrolls are measured where drawn (`grid_width`). +51. **The command language reads pipes, refuses surplus, and checks names when applied** (`only_when_omitted`; `RequireKnown`/`unless_unknown`). +52. **A date column imports as one hierarchical category;** `--split-date` restores flat. `Plan::spelling_clash` refuses what would silently merge; `--strict` refuses partial skips. --- -## File Inventory +## File Map -Line counts are a **dated snapshot**, not a maintained fact: measured 2026-09-24 at `c1772ca9`, and **70 of 105 had drifted past 10%** in the three days since the 2026-09-21 refresh — eleven had doubled or more (`client.rs` 255 → 943, `lockstep.rs` 1126 → 4064). Treat them as "is this file big", never as a figure to cite; re-derive with `wc -l`. The `ws-server/` and `wasm-client/` entries use a shorthand prefix, so a refresh must map it to `crates/improvise-/{src,}/` by hand or they drift silently. Test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted. +Per-crate entry points. Per-file notes, test counts and a dated line-count snapshot are in +`context/file-inventory.md`; re-derive sizes with `wc -l`. -### `improvise-core` (`crates/improvise-core/src/`) -``` -model/types.rs 3560 / 112t Model, formula eval (fixed-point), CRUD -model/cell.rs 1358 / 47t CellKey (sorted), CellValue, DataStore (interned + index), - live_records (insertion order — gotcha 18) -model/record.rs 226 / 8t RecordHash / CellId surrogates: the opaque identities a - thin client names a cell by (ADR 002) -model/category.rs 310 / 9t Category, Item, Group, CategoryKind -model/symbol.rs 89 / 3t Symbol interning -view/layout.rs 2604 / 57t GridLayout, drill, records mode, records_columns; a private - per-axis position index (`row_pos`/`col_pos`, rebuilt by `reindex`) - so a data position resolves in O(1) (improvise-felo.13) -view/types.rs 817 / 43t View config (axes, pages, hidden, collapsed, format) -model/id.rs 290 slugify + IdAllocator: how a generated id is made (ADR 007 - clause 4, core.md §20). Nothing outside the file calls it yet -view/axis.rs 190 / 5t Axis enum + label/next methods (no colour: that is a Face) -workbook.rs 413 / 22t Workbook: Model + cross-view ops, item facade -format.rs 519 / 40t format_f64, parse_number_format (display only) -``` - -### `improvise-formula` (`crates/improvise-formula/src/`) -``` -parser.rs 1394 / 79t pest grammar + tokenizer → Formula AST -quote.rs 160 / 7t quote_name / is_bare_name / pipe_quote / unescape_pipe_body: - the one home, both halves (improvise-i2m5). improvise-gp - re-exports it; improvise-io calls it from - persistence/{format,parse}.rs -formula.pest 108 PEG grammar for formula text -ast.rs 130 Expr, BinOp, AggFunc, Formula, Filter -``` - -### `improvise-io` (`crates/improvise-io/src/`) -``` -storage.rs 334 / 13t Storage trait (read/write/list/delete), PhysicalFs, MemoryFs, - `impl Storage for Rc` (improvise-hmws), and - contract::storage_contract — public, not cfg(test), because - a browser must run that sequence (harness/) -opfs.rs 289 / 0t OpfsFs: Storage over the browser's Origin Private File - System. wasm32 only. Run by harness/, not by cargo test -persistence/improv.pest 179 PEG grammar — single source of truth -persistence/mod.rs 441 / 11t document lifecycle: save/load/gzip/legacy-JSON over Storage; - facade re-exporting format_md/parse_md/export_csv* -persistence/format.rs 401 / 11t format_md: Workbook → .improv text -persistence/parse.rs 1684 / 46t parse_md: pest tree-walk → Workbook (+ grammar generator) -persistence/export.rs 243 / 5t export_csv / render_csv. Builds its layout with - with_frozen_records, NOT new — `new` skips prune_empty - (improvise-iot6) -persistence/roundtrip_tests.rs - 1236 / 62t format_md ↔ parse_md characterization (tests only) -import/wizard.rs 1967 / 62t ImportPipeline + ImportWizard. `Plan::place` is the one - placement pass, shared by `collisions()` and the build so - the two cannot disagree; `DuplicatePolicy` / `Collision` - (improvise-cgqt) -import/analyzer.rs 434 / 17t Field kind detection (Category/Measure/Time/Skip); - analyze_records is first-seen order, not HashSet (gotcha 18) -import/csv_parser.rs 590 / 26t CSV parsing, multi-file merge -``` - -### `improvise-protocol` (`crates/improvise-protocol/src/`) -``` -lib.rs 619 / 2t ViewState, ViewCursor, ViewFrame, AppMode (17), Panel, - MinibufferConfig, MinibufColor, HELP_PAGE_COUNT -reduce.rs 749 / 10t Command + ModelCommand (the wire format), HelpPage, reduce_view -render.rs 2157 / 46t CellDisplay, WidgetCache trait + one cache per widget - (GridCache/CategoryTreeCache/FormulaCache/ViewListCache/ - TileBarCache), AxisFrame/FrameEntry/FrameGroup and - header_level_widths (the grid frame, gotcha 44), - GridCache::seed (the bootstrap compaction), Projection, - bootstrap-only Snapshot -keymap.rs 975 / 0t KeyPattern, Binding, Keymap, ModeKey (17), TextEntry, - PanelKeymap, Keymaps::default_keymaps/prefix_keymap — the - shared table both front ends look up. Tests live with - dispatch in improvise-command/src/command/keymap.rs -viewmodel.rs 170 / 5t flatten_cat_tree — derivations between a cache and a - renderer, called by both front ends (improvise-edp) -face.rs 682 Face (a role: Axis, WizardField, Mode*, MinibufferOf…), Theme, - AnsiSlot — appearance as data a front end applies (ADR 004, - protocol.md §10); draw.rs::face_style is the TUI's one translation -key.rs 96 / 3t KeyCode, KeyModifiers, KeyEvent (moved from command/) -frame.rs 156 / 3t ClientFrame/ServerFrame — the ONE envelope pair, both - deployments (improvise-xks6). Nothing else defines one. - ClientFrame::Save is a request, not a Command (gotcha 45) -basis.rs 257 / 6t Basis (the three tiers), Rejection, BasisCache — ADR 001 - layer 1. Basis has no Default and no public constructor -help.rs 1241 / 14t HelpPage content and Hint { keys: Vec<(KeyCode, KeyModifiers)>, - command, caption }. `compact` drops a chord by its MODIFIERS - (anything beyond shift), never by parsing a label it just - formatted (improvise-thpe) — the guard is the signature -resolve.rs 4883 / 88t the browser's key resolvers: a BrowserKey to the Commands it - composes, mirroring each Cmd line for line INCLUDING where - siblings disagree (gotcha 47). The largest file in the crate -client.rs 943 / 15t ClientState: ViewState + one cache per widget, and the fold; - in_flight, `opened` (OpenedCell), notify, settle_status - (gotcha 47). EditorSession is in lib.rs beside ViewState -``` - -### `gp-tree` / `improvise-gp` (`crates/gp-tree/`, `crates/improvise-gp/`) -``` -gp-tree/src/lib.rs 575 / 8t Tree, Spec, eval, simplify, render, breed, Population, - SplitMix64 (vendored seeded Rng; seed() keeps the seed so - a fit can be shown and recorded, pinned to the published - reference vectors) -gp-tree/src/data.rs 1000 / 21t raw string rows -> f32 feature vectors: per-column type - inference, one-hot, mean-imputation + col.missing flag, - ColumnSpec/encode_row, Dataset (incl. warnings), - Model = trees + plan + seed + FitConfig -gp-tree/src/fit.rs 961 / 32t the search: FitConfig (Default = the CLI's numbers), fit() - -> simplified champion, cv_regress/cv_classify, - permutation_importance, folds/standardize/mse. - Returns data; never prints -gp-tree/examples/fit_csv.rs 583 CLI over the above: flags, demo dataset, all printing -gp-tree/tests/scoring_known_answers.rs - 250 / 6t f32 bit patterns + a next_u64 witness: the evidence any - change to the scoring paths needs (gotcha 37c) -improvise-gp/src/lib.rs 978 / 25t formula_text / fitted_formula: a Tree as IF(...) formula - text, validated back through parse_formula. A threshold is - the SHORTEST decimal that reads back as the same f64 (not - the exact expansion), so a row on a split boundary routes - the same in f64 as in f32 eval; non-finite values panic -improvise-gp/src/dataset.rs 506 / 15t a View's records -> the feature matrix a fit consumes — - the improvise-side counterpart of preprocess. Records mode - is already a feature matrix (gotcha 1) -improvise-gp/src/fit.rs 1778 / 35t one fit end to end: a View's records in, an improvise - Formula and the evidence for believing it out. Not in a - command, because none of it is terminal-specific -``` - -### Command layer (`crates/improvise-command/src/command/`) -``` -cmd/core.rs 475 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers -cmd/registry.rs 903 / 13t default_registry() — all registrations -cmd/navigation.rs 507 / 12t Move, EnterAdvance, Page* -cmd/cell.rs 724 / 20t ClearCellCommand, YankCell, PasteCell, TransposeAxes, SaveCmd -cmd/commit.rs 1304 / 27t CommitAndAdvance, CommitBuffer(CommitKind), CommitExport, - CommitSave -cmd/effect_cmds.rs 519 / 6t effect_cmd! macro, 25+ simple wrappers, `unless_unknown` (gotcha 51) -cmd/fit.rs 293 / 8t fit-measure: evolve a tree over the active view and store it - as an ordinary `## Formulas` line — nothing GP-specific is - left in the document afterwards -cmd/grid.rs 988 / 22t ToggleGroupAtCursor, ViewNavigate, DrillIntoCell, TogglePruneEmpty -cmd/mode.rs 469 / 13t EnterMode, Quit, EditOrDrill, EnterTileSelect -cmd/panel.rs 936 / 24t Panel toggle/cycle/cursor, formula/category/view panels -cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode -cmd/text_buffer.rs 399 / 14t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand -cmd/tile.rs 170 / 6t MoveTileCursor, TileAxisOp -cmd/wizard.rs 244 / 6t HandleWizardKey — per-step import-wizard key tables -keymap.rs 1353 / 44t KeymapSet, Keymap::dispatch/dispatch_transient, effects_of, - SetTransientPrefix — what a thin client cannot have. The - table itself is improvise-protocol/src/keymap.rs -parse.rs 350 / 23t Script/command-line parser (prefix syntax) -``` - -### Semantic UI state (`crates/improvise-command/src/ui/`) -``` -ui/effect.rs 4020 / 99t Effect trait. 63 effects: 53 `impl Effect for` blocks, 9 - simple_effect! expansions, and SetTransientPrefix in - command/keymap.rs. The 10 outside the brace-match are all - ViewOnly; 28 of the 53 are Model(_) (gotcha 38, h3y7) -ui/app.rs 5558 / 134t App, ModelState, ViewState + dispatch methods, Panel, handle_key, dispatch_key, document_generation, autosave; apply_effects declares `dirty` for WholeWorkbook and observes it for CellData/LayoutGeometry (gotcha 45) -ui/project.rs 6383 / 118t Dirty, Viewport, grid_of/category_rows/formula_texts/ - view_summaries/tile_entries, grid_updates (the diff), - ClientSession + Sessions (connect/apply/rebase/project_all) -ui/reduce.rs 1217 / 24t Checked / CheckedBatch / apply_command / model_effects / - reduce_key — one door from a Command or a keypress to the - authoritative App. `reduce_full` is GONE (improvise-h3y7) -ui/host.rs 3410 / 67t Host: one App + one Sessions; handle, commands, project_all, - rebase. Lives here, not in ws-server (gotcha 45) -ui/cat_tree.rs 174 / 8t Category tree flattener for panel -ui/viewport.rs 259 compute_col_widths / compute_row_header_width / visible_cols - (pure geometry; the width rule itself is the protocol's); - grid_width, SIDE_PANEL_WIDTH, MIN_GRID_WIDTH (improvise-gtn7.1); - GridSummary / grid_summary / grid_window (improvise-felo.13) -``` - -### TUI rendering, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`) -``` -ui/grid.rs 905 / 14t GridWidget (ratatui): no &Model, no &View — the frame comes - off GridCache (gotcha 44). Its module doc is the best-written - statement of that rule in the repo -ui/help.rs 277 5-page help overlay (render only) -ui/import_wizard_ui.rs 829 / 9t Import wizard rendering. `Panel` is the one writer, and - every write is clipped to the panel's rows and width -ui/text.rs 169 truncate and wrap, both measured in terminal cells, never bytes -draw.rs 1122 / 20t TUI event loop, frame composition; `content_tests` draws a - whole frame to check the selection beside a side panel -main.rs 428 CLI entry (clap): open, import, cmd, script -# other ui/*.rs are small panel renderers — skip unless changing layout/style -tests/grid_frame_agreement.rs - 479 / 6t the two front ends draw the same grid frame (improvise-anr9) -tests/records_wide_smoke.rs 67 / 1t -tests/shipped_documents_load.rs 333 / 8t every TRACKED .improv/.improv.gz loads, and every record - it projects is writable. "Shipped" is read off `git ls-files`, - not a .gitignore reading (improvise-c0aj), and the walk's git - is scrubbed of every hostile GIT_* setting the caller may carry -tests/web_demo_model.rs 83 / 2t every visible cell in web/demo.improv is a Record -``` - -### Browser host and client (`crates/improvise-ws-server/src/`, `crates/improvise-wasm-client/src/`, `web/`) -``` -ws-server/registry.rs 616 / 14t Registry of Hosts keyed by DocumentId; ClientId {document, - session} — a SessionId is unique only within its document. - A document is never evicted while the server runs (ADR 001) -ws-server/host.rs 4 / 0t a re-export shim only. The session loop is - improvise-command/src/ui/host.rs (gotcha 45) -ws-server/origin.rs 132 / 6t OriginPolicy: which origins may open a socket. Loopback by - default, --allow-origin widens, absent allowed, two refused - (improvise-kc7n). The only decision transport::admitted makes -ws-server/transport.rs 330 / 4t tokio + tungstenite plumbing, plus the Origin check at the - handshake — a refused upgrade, not an ignored socket -ws-server/main.rs 190 / 5t CLI entry: serve a file on a port; --bind, --allow-origin -ws-server/tests/socket.rs 389 / 6t the one test that uses a real socket: handshake, JSON - framing, SessionId routing, which document a URL asked for -wasm-client/lib.rs 455 / 8t Client: the wasm-bindgen surface (init, on_message, on_frame, - viewport/set_viewport, render_model, set_cell_at, typed_cell) -wasm-client/keys.rs 187 / 5t BrowserKey ONLY — a KeyboardEvent flattened to an improvise - key. It does NOT resolve: `resolve` and KeyOutcome live in - improvise-protocol/src/resolve.rs, shared by both front ends -wasm-client/render.rs 1249 / 21t RenderModel: editable / selected / cat tree, decided here -wasm-client/socket.rs 547 / 13t the fold (fold_frame/handle_frame), the two compositions - and typed() text→CellValue. Defines no enums -# state.rs is GONE — ClientState moved to improvise-protocol/src/client.rs -wasm-client/tests/browser_round_trip.rs 2136 / 34t improvise-1ey's scope against a real Host -wasm-client/tests/lockstep.rs 4064 / 78t host and client agree on a keypress -web/app.js 816 string calls only, no protocol knowledge, no framework. - Run by harness/tests/shell.spec.js (Chromium; the standalone half on Firefox too) — see - gotcha 48 for what that does and does not reach -web/worker.js 86 three string calls, no protocol knowledge. RUN by - shell.spec.js's no-?ws path, with bindings.rs (gotcha 45) -web/shell.js 399 everything app.js DECIDES that is not painting: no DOM, - no browser API, one `node --test` away. Owns socketUrl, - gridPlan, frameCostOf, documentTitle — so a chunk of the - browser's render decisions is NOT in the file gotcha 42 - and gotcha 46 name. Reasoning: context/code/clients.md -web/serial.js 19 the one rule worker.js exists to keep: one frame at a - time, in arrival order (gotcha 48's async-listener race), - extracted so it is testable without worker globals -web/index.html, style.css the page; legibility only -web/build.sh 38 wasm-pack build → web/pkg (gitignored). Needs - wasm-opt = ["-O", "--enable-bulk-memory"] or the bundled - wasm-opt rejects LLVM's memory.copy -web/assemble.sh 112 assembles the deployable site (copies web/*.improv) -web/demo.improv 65 the document the site ships and seeds from. Every cell a - Record — asserted (gotcha 46) -``` - -### Browser worker host (`crates/improvise-worker-server/`) -``` -improvise-worker-server/src/worker.rs 765 / 17t Worker: one port, one client; handshake, Addressed - filter, persist trigger (model_state.dirty). Every decision, every test -improvise-worker-server/src/vfs.rs 348 / 8t documents/open: seeding is a Storage::write -improvise-worker-server/src/bindings.rs 185 / 0t wasm32 shim: OpfsFs awaits + postMessage. No Rust test - reaches it, but harness/tests/shell.spec.js DOES: the - no-?ws path runs start()/OpfsFs::open under Chromium - and Firefox. - Called NEVER RUN here for months: see gotcha 45 -improvise-worker-server/src/lib.rs 51 / 1t the module doc that states exactly what is and is not - covered — the one correct copy of that fact -improvise-worker-server/tests/post_message.rs 407 / 8t main-thread traffic, strings only -``` - -### Browser harness (`harness/`) — not a workspace member - -``` -harness/playwright.config.js 50 projects: chromium (every spec), firefox (opfs, plus - shell.spec's standalone half: grepInvert @mocked-peer), - webkit (opfs only, which skips on a runtime probe: no - OPFS). CI runs `--project=chromium --project=firefox` only: - nixpkgs' WebKit aborts inside the runner VMs (ci.yml) -harness/tests/shell.spec.js 345 the first execution of web/app.js. A change to web/*.js - belongs here (gotcha 48). Playwright serves harness/serve/, - which harness/build.sh ASSEMBLES: edit web/*.js, rebuild, - or the specs run the old page -harness/tests/opfs.spec.js 103 drives the OPFS contract page and reads back its result -harness/opfs-contract/src/lib.rs 470 runs improvise_io::storage::contract::storage_contract - against a real OpfsFs inside a real Web Worker -harness/www/opfs/{index.html,worker.js} the page and worker that host it -harness/build.sh 50 assembles harness/serve/ (web/assemble.sh, the fixtures, - and the contract crate's wasm-pack build) — gitignored -# gated by the `browser` job in .github/workflows/ci.yml, which can fail the build -``` - -### Examples -``` -examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta) -examples/pretty-print.rs Parse stdin, print formatted .improv -crates/improvise-core/examples/category_width_bench.rs - The same records at 12 and at 24 regular categories - (improvise-felo.1; context/code/core.md) -``` - -### Context docs -``` -.claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl -scripts/check-comment-runs.py The two-line comment cap AND --check-evidence. The flag is - ADDITIVE since improvise-ijql — every invocation that can - exit 0 answers the cap — guarded by a subprocess self-test, - the only level a dispatch bug is visible at. Reasoning: - context/code/tooling.md §2 -scripts/check-ci-graph.py pages-deploy must `needs` every job in ci.yml, and every - suite `checkers.sh --list` knows must run in some CI job - (improvise-dak5). It REFUSES what it cannot read rather than - parsing YAML: the values it accepts are an allowlist - (tooling.md). --self-test; run in the `test` job. README - states the rule and lists no jobs (improvise-x7cq) -scripts/checkers.sh The one entry point for every *-test.sh suite: scrubs the - env to an allowlist, runs under the bash that runs it (no - version floor since #297). `ci_suites` is the table of which - suites each CI job runs (`--ci `); ci.yml names none -scripts/pr-watch Waits on a PR's checks (the pr-watch skill). Exit 5: the PR - is already MERGED or CLOSED, so there is nothing to wait for -scripts/assemble-stamp.sh The cache-bust stamp gate web/assemble.sh calls, extracted - so its refusals can be exercised without a web/pkg -scripts/assemble-stamp-test.sh Its check_the_checker, run first in the `build` CI job -context/code/tooling.md The repository's own checkers: why an evidence heading's - names come from cargo rather than from the source, and - why a leaf name is not an identity -context/test-evidence/tooling.md Evidence for the gates in scripts/ and for the tests that - hold CI's shape -scripts/nix-seed-flake-inputs.sh Seeds locked flake inputs over git so `nix develop` works in cloud sessions -prolog-sessions/kb.pl Executable knowledge base: invariants, doctrine, decisions (CI-checked) -context/adr/007-stable-identities.md Stable identities: ids unique across the document are the - keys and names are display-only; gensym suffixes apply to - generated ids only, and hand-written ids are canonical. - Accepted 2026-09-25. (There is no ADR 005 or 006 file: - improvise-l2b8 and improvise-ilvl claim those numbers.) -context/adr/004-faces-and-themes.md Faces: appearance as UI-agnostic data a front end - applies, emacs-style. Accepted 2026-09-22. -context/adr/003-live-backed-models.md A model may be backed by a live query source — - improvise as a DB client. Accepted, NOT a priority. -context/adr/002-naming-a-cell.md How a client names a cell: an opaque surrogate on - CellDisplay, whose absence means the cell is an - aggregate. Records why _Index cannot serve. -context/adr/001-concurrent-edits.md Concurrent edits between clients: record-level atomicity, - optimistic basis check per batch, advisory reservations. - Records why a causal clock was rejected. -context/cloud-agents.md Cloud-session environment: tracker sync, bd pitfalls, what is blocked -context/design-principles.md Architectural principles & testing doctrine -context/plan.md Show HN launch plan -context/repo-map.md This file -docs/design-notes.md Product vision & non-goals -``` +| Area | Where | Entry points | +|---|---|---| +| Core model | `crates/improvise-core/src/` | `model/{types,cell,record,category,id}.rs`, `view/{layout,types,axis}.rs`, `workbook.rs`, `format.rs` | +| Formula | `crates/improvise-formula/src/` | `parser.rs`, `formula.pest`, `ast.rs`, `quote.rs` | +| I/O | `crates/improvise-io/src/` | `storage.rs`, `opfs.rs`, `persistence/{improv.pest,format,parse,export,mod}.rs`, `import/{wizard,analyzer,csv_parser}.rs` | +| Protocol | `crates/improvise-protocol/src/` | `lib.rs` (ViewState), `reduce.rs`, `render.rs`, `keymap.rs`, `face.rs`, `frame.rs`, `basis.rs`, `resolve.rs`, `client.rs`, `viewmodel.rs`, `help.rs` | +| Command layer | `crates/improvise-command/src/command/` | `cmd/{core,registry,navigation,cell,commit,effect_cmds,fit,grid,mode,panel,search,text_buffer,tile,wizard}.rs`, `keymap.rs`, `parse.rs` | +| Semantic UI state | `crates/improvise-command/src/ui/` | `effect.rs`, `app.rs`, `project.rs`, `reduce.rs`, `host.rs`, `viewport.rs`, `cat_tree.rs` | +| TUI + CLI | `src/` | `draw.rs`, `main.rs`, `ui/{grid,help,import_wizard_ui,text}.rs`; integration tests in `tests/` | +| GP | `crates/{gp-tree,improvise-gp}/` | `gp-tree/src/{lib,data,fit}.rs`, `improvise-gp/src/{lib,dataset,fit}.rs` | +| Socket host | `crates/improvise-ws-server/src/` | `registry.rs`, `origin.rs`, `transport.rs`, `main.rs`; `host.rs` is a re-export shim | +| Worker host | `crates/improvise-worker-server/src/` | `worker.rs`, `vfs.rs`, `bindings.rs` (wasm32 shim) | +| Browser client | `crates/improvise-wasm-client/src/`, `web/` | `lib.rs`, `keys.rs`, `render.rs`, `socket.rs`; `web/{app,shell,worker,serial}.js`, `index.html`, `demo.improv`, `build.sh`, `assemble.sh` | +| Browser harness | `harness/` | Playwright (`tests/shell.spec.js`, `tests/opfs.spec.js`), `opfs-contract/`, `build.sh` — not a workspace member; its own CI job | +| Examples | `examples/`, `crates/*/examples/` | `gen-grammar.rs`, `pretty-print.rs`, `category_width_bench.rs`, `fit_csv.rs` | +| Scripts | `scripts/` | `checkers.sh` (every `*-test.sh` suite), `check-comment-runs.py`, `check-ci-graph.py`, `bd-audit-ids`, `kb`, `pr-watch`, `pr-reviews`, `build-doc-site.mjs` | +| Context docs | `context/` | `design-principles.md`, `gotchas.md`, `file-inventory.md`, `code/.md` (arguments), `test-evidence/` (bug evidence by test name), `adr/` (001–004, 007), `cloud-agents.md`; `prolog-sessions/kb.pl` is the KB; `docs/design-notes.md` is product vision | --- ## Wasm -The browser bundle is `improvise-core`, `improvise-formula`, `improvise-io`, -`improvise-protocol`, `improvise-command`, `improvise-wasm-client` and -`improvise-worker-server`. All seven -`cargo check --target wasm32-unknown-unknown` clean with no feature flags -(improvise-ywd), gated by the `wasm` job in `.github/workflows/ci.yml`. That -job gates **eight** crates, not seven: `improvise-gp` was added (improvise-z5eg) -explicitly *not* as a bundle member but as a candidate, because staying -wasm-clean is free now and expensive to retrofit — so the bundle is seven and -the gate is eight, and neither number is the other one's typo. -**A feature that adds a dependency to a bundle crate must be checked with `cargo tree --edges normal`** — the default output includes **dev** edges, so a plain `cargo tree -p improvise-protocol --target wasm32-unknown-unknown` reports a dev-only `serde_json` as *present* and reads exactly like a leak. Check per-crate too, because the `wasm` CI job invokes each crate separately and feature unification cannot reach across that (improvise-26dm added a `test-support` feature on `improvise-protocol` on those terms; it is off in every shipped build). - -`improvise-ws-server` is the host and is native only — tokio, tungstenite and -`Box` — so `improvise-wasm-client` cannot depend on it in anger. It -no longer needs to: both ends `pub use` `improvise_protocol::frame` and neither -defines a pair (gotcha 42). The **dev**-dependency remains, for -`tests/browser_round_trip.rs`, which hands the client's own serialized frame to -a real `Host::handle`. The root `improvise` crate is -the terminal binary and does **not** build for wasm — it depends on -crossterm; that is correct, not a gap. - -`cargo check` passing is not the same as running. Two known runtime hazards -it cannot see: `Instant::now()` panics on wasm (hence `web-time` in -`improvise-command`), and any `std::fs` call compiles but fails in a browser -— improvise-fuj closed the effects that bypassed `Storage`; autosave and the headless final write still do (improvise-31oe.11). +The browser bundle is `improvise-{core,formula,io,protocol,command,wasm-client,worker-server}`; all seven +`cargo check --target wasm32-unknown-unknown` clean with no feature flags, and the `wasm` CI job also gates +`improvise-gp` (a candidate, not a member). `improvise-ws-server` and the root crate are native-only. +Check new bundle dependencies with `cargo tree --edges normal` (the default output includes dev edges), +per crate. `cargo check` cannot see runtime hazards: `Instant::now()` panics on wasm (`web-time`), and +`std::fs` compiles but fails in a browser. Long form: `context/file-inventory.md`, appendix. ## CLI @@ -673,39 +261,20 @@ improvise cmd 'add-cat Region' -f model.improv # headless command(s) improvise script setup.txt -f model.improv # run script file ``` -Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`, `--split-date ` (a date column imports as ONE hierarchical category by default; this restores the flat split, and `--extract` still picks the components — improvise-8jk6.3), `--strict` (with `--no-wizard`, any skipped record is a refusal and nothing is written, where the default writes and warns with a count — improvise-8jk6.6), and `--on-duplicate refuse|sum|separate` (default `refuse`; applies with `--no-wizard`, where the wizard would otherwise ask — improvise-cgqt). A headless import also refuses a `--split-date` that names a non-date field and an `--axis` that names a category the import did not build (exit 1). - ---- +Import flags: `--category --measure --time --skip --extract --axis --formula --name`, `--split-date ` +(flat date split; default is one hierarchical category), `--strict` (refuse any skipped record), and +`--on-duplicate refuse|sum|separate` (default `refuse`). Details: `context/code/cli.md`, gotchas 51–52. ## Testing — the short version -Full guidance lives in `context/design-principles.md` §7. Quick reminders: +Full guidance: `context/design-principles.md` §7. -- **Two numbers, not one** — the ambiguity between them is what let the old - "suite runs in <2s" drift: **no single test binary over 2 seconds**, and the - workspace wall clock is the sum. Measured at `63a5c13`: **1464 tests over 35 - binaries** (1 ignored), longest single binary **0.6–1.0s across two runs on one machine** — - the spread being its own argument for reading your own run. Excludes compilation, - and excludes `harness/` (Playwright, minutes, its own CI job). *Re-measure* rather - than re-assert — `cargo test --workspace` prints a `finished in` per binary, and - **the portable signal is the ratio, not the seconds**. **Not the ratio to the - median, though: it is degenerate.** Most binaries print `0.00s` at two decimals, - so the median is zero and every ratio to it is undefined (improvise-i5z6, 39 - binaries). Compare the slowest binary to the next few slowest instead: an outlier - is one that sits well clear of them *in every run*. That is how i5z6 was read — - `improvise-formula`'s lib longest of 39 in all three runs, at 0.91–1.95s against a - runner-up's 0.50–0.92s. Both outliers so far had one cause: a generator re-parsing - its pest grammar per case. `improvise-io` (improvise-8shu, 1.42s → 0.21s) and - `improvise-formula` (improvise-i5z6, 0.84–0.98s → 0.06–0.07s) each read it once through a - `OnceLock` now. A new grammar-walking strategy should do the same from the start. - The full argument is `context/design-principles.md` §7. -- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed. -- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability. -- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Record the bug in `context/test-evidence/`, **keyed by test name** — *not* in the test's doc comment, which the two-line comment cap forbids and `scripts/check-comment-runs.py` enforces. A heading may be module-qualified and must be when its leaf is not unique. -- Coverage target ~80% line/branch on logic code; skip ratatui render paths. - ---- +- **No single test binary over 2 seconds**; the workspace wall clock is the sum. Re-measure with `cargo test --workspace` (a `finished in` per binary) and compare the slowest binary to the next few — a ratio, not seconds, and not a ratio to the median (it is zero). +- Commands: build a `CmdContext`, call `execute`, assert on the returned effects. No terminal. +- Property tests (`proptest`, default 256 cases) for invariants and round-trips; a grammar-walking strategy must read its pest grammar once (`OnceLock`). +- **Bug-fix workflow:** write the failing test first. Record the bug in `context/test-evidence/`, keyed by test name — not in a doc comment (two-line cap, `scripts/check-comment-runs.py`). +- Coverage target ~80% on logic code; skip ratatui render paths. ## Key dependencies -ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta. +ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest, flate2, csv, enum_dispatch. Dev: proptest, tempfile, pest_meta. diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index cede9610..2a464b56 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -34,6 +34,8 @@ const SECTIONS = [ ["docs/design-notes.md", "design-notes", "Design notes"], ["context/design-principles.md", "design-principles", "Design principles"], ["context/repo-map.md", "repo-map", "Repository map"], + ["context/gotchas.md", "gotchas", "Gotchas (long form)"], + ["context/file-inventory.md", "file-inventory", "File inventory (long form)"], ], }, { @@ -613,7 +615,7 @@ for (const section of SECTIONS) { for (const [src, slug, title] of section.docs) { const depth = slug.split("/").length - 1; const ctx = { depth, cited, unresolved, anchors: new Set() }; - if (slug === "repo-map") ctx.listItemId = (n) => (n >= 1 && n <= 200 ? `gotcha-${n}` : null); + if (slug === "repo-map" || slug === "gotchas") ctx.listItemId = (n) => (n >= 1 && n <= 200 ? `gotcha-${n}` : null); const md = readFileSync(join(root, src), "utf8"); const { html, toc } = renderMarkdown(md, ctx); From f3a54d34e076b5916c7520e25b58ddc9587dd861 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:18:55 -0700 Subject: [PATCH 085/106] docs(adr): ADR 009 makes every operator edge network-transparent, over up to two hops and request/response transports (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 56 +++++++++++++++-------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 9bc84d21..33e43cfe 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -180,8 +180,34 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery -The same circuit serves the in-process TUI and remote clients. Its internal streams never -cross a wire, because aggregation stays on the host. +**Every edge may cross a network.** An edge, meaning the stream from one operator to the +next, may be in-process or remote, and a path may cross up to two network hops: browser ⇄ +gateway ⇄ host, or host ⇄ worker ⇄ client. No edge assumes a function call or shared +memory. An operator that sits at a wire boundary is just one implementation of an edge, +an **edge adapter**. So every edge has the same semantics: +- **A wire form.** Every element type that flows on an edge has a serializable form, + defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays + free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its + tuple of ids, its position or an ADR 002 surrogate. +- **Stamps compose end to end.** The host stamps each published batch (below), and an + intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up + downstream as a gap. +- **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces + exactly: deltas are summed, and notices merged. +- **No long-lived stream assumed.** An edge may be carried as a series of requests and + responses, for example several HTTP requests on one connection, as well as a socket. + So each edge keeps an ordered log, every request can be retried, and a resync from a + publication stamp is idempotent. +- **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is + alive while its next message or request succeeds. Each hop keeps alive in both + directions. A side that hears nothing within the timeout, or whose request fails, + treats the hop as dropped. The downstream side then resyncs: it refetches through the + path, and the host answers from current state. + +Aggregation stays on the model side (ADR 008), so the operators that aggregate run on +the host. Which other edges cross a wire is a deployment choice. The engine's +invalidation signal, which ADR 006's replay trigger relies on, is the change notice and +the refetch rule below, followed from a displayed cell's key back to the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left and dimensions. The window is never state, and a request costs O(visible). @@ -227,11 +253,12 @@ identity, so every NaN counts as one value, because the wire spells every NaN `" state. `GridExtent` is derived by the client, and in records mode the record count is fetched. -**A connection operator** at the wire boundary adapts the circuit to a connection. It is -a sink for results and notices and a source of requests toward the circuit, and the -mirror image toward the client. It owns serialization, the stamp rules, notice -coalescing, keep-alive in both directions, and session resumption. In process it is -function calls. Its shape is open question 4. +**Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for +results and notices and a source of requests. Toward the client it is the mirror image. +Each hop has an adapter at each end, and each end keeps only that hop's state: the last +stamp it applied and its liveness timer. A gateway hop (browser ⇄ gateway ⇄ host) may +serve many clients behind one upstream connection, so the host sends one notice per +gateway rather than one per client. In process, an edge is a function call. ### 10. Text column widths @@ -290,7 +317,7 @@ Step 8's required tests: - a dropped notice is detected; - a reconnect after a host restart refetches; - a refused frame consumes no publication number; -- a dropped connection forces a refetch. +- a dropped hop forces a refetch. ## Open questions @@ -299,7 +326,7 @@ Step 8's required tests: O(clients × all moved keys) per batch. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? It could also live in the connection operator (question 4). + backend may hold? On a two-hop path the topics could instead live in a gateway. *Recommendation:* broadcast with the size cap. Add topics only if measured traffic with many clients on different shapes justifies them. @@ -318,16 +345,7 @@ Step 8's required tests: small from-scratch reference implementation, kept in the test crate, as the oracle, and a pushdown backend behind the same trait for query-only sources. -4. **What shape is the connection operator?** Either one component per connection, or - one per host process: a host half shared by all its connections, with a client half - in every front end. The second lets the host send one notice per operator rather than - per client, but holds per-connection state in the shared operator. Who owns - resumption state, and is it backend state? *Recommendation:* one per host process, - with resumption state in the host half keyed by a client-held token, treated as - delivery state rather than UI state. In the TUI and the browser worker, with one - client each, it is a function call. - -5. **Which columns get a host-maintained width?** Columns whose cells are text? What +4. **Which columns get a host-maintained width?** Columns whose cells are text? What about a column holding both text and numbers? *Recommendation:* text cells get the maintained width, and the view layer takes the larger of that and its own numeric width. From 5e94255c21d86013e83b7c2271d0693b869a6101 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:22:50 -0700 Subject: [PATCH 086/106] docs(adr): ADR 009 requires a lossless edge wire form with round-trip tests (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 33e43cfe..44aeadcb 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -189,6 +189,22 @@ an **edge adapter**. So every edge has the same semantics: defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its tuple of ids, its position or an ADR 002 surrogate. +- **The wire form is lossless.** Nothing is lost in transport, numeric precision above + all: + - a number never crosses as a JSON or JS double. A `numtower` `Num` crosses as its + canonical text (ADR 005). An f64 crosses as its exact bits, which keeps `-0.0` and + NaN payloads. That conflicts with today's `wire_f64` and with `numtower`'s `nan` + token, which both drop the payload, so the edge carries bits for every non-finite + value; + - integers wider than 2⁵³, including positions and counts, cross as strings; + - text crosses byte-exact, with no normalisation; + - the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs + a sum gets the host's exact read or rebuilds it from the exact deltas. + + The stamp rule detects lost messages, but nothing would detect a lossy encoding. So + every element type has a round-trip property test, where encoding then decoding is + the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and + `i128`-wide fractions. - **Stamps compose end to end.** The host stamps each published batch (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. @@ -245,8 +261,8 @@ maintained key, so it maps member keys through its own remap first. It then refe what moved. It refetches everything when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, or a notice it cannot read. A slow client may merge pending notices: the union of keys, the latest -stamp, and axis changes kept in order. Outputs are consolidated under the wire's value -identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`. +stamp, and axis changes kept in order. Because the wire is lossless, outputs keep the +circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client From 353352f89d62e596f9c92dd46ce0cb715d7b7421 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:23:49 -0700 Subject: [PATCH 087/106] docs(adr): ADR 009 gives merged notices a covered range, one shared notice log, and a clock per process (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 29 ++++++++++++++--------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 44aeadcb..99dee2da 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -3,7 +3,7 @@ **Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008 +**Relates to:** ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 ## Problem @@ -25,7 +25,8 @@ equal today's eager results exactly. A problem-agnostic workspace crate with no dependencies beyond `std` (so it is wasm32-clean and serde-free). It provides: - `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; -- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`, +- a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`, + `differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, `distinct` and `aggregate`; - resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it @@ -208,12 +209,18 @@ an **edge adapter**. So every edge has the same semantics: - **Stamps compose end to end.** The host stamps each published batch (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. +- **One clock per process.** Each process runs its own synchronous sub-circuit. Across a + hop, the stamped batches are the clock. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces - exactly: deltas are summed, and notices merged. + exactly: deltas are summed, and notices merged. A merged notice carries the range of + publications it covers, `(from, to]`, so a downstream side sees no gap where nothing + was lost. - **No long-lived stream assumed.** An edge may be carried as a series of requests and responses, for example several HTTP requests on one connection, as well as a socket. - So each edge keeps an ordered log, every request can be retried, and a resync from a - publication stamp is idempotent. + Every request can be retried, and a resync from a publication stamp is idempotent. A + client asks for "notices since n", and the host answers from **one shared, bounded log + of published notices**: a single log, not one per edge or client. A request older than + the log's start is answered "refetch". - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, @@ -221,7 +228,7 @@ an **edge adapter**. So every edge has the same semantics: path, and the host answers from current state. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the host. Which other edges cross a wire is a deployment choice. The engine's +the model side, which may itself span a hop. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger relies on, is the change notice and the refetch rule below, followed from a displayed cell's key back to the model. @@ -260,13 +267,13 @@ A notice whose key lists exceed a fixed size carries "refetch" instead. maintained key, so it maps member keys through its own remap first. It then refetches what moved. It refetches everything when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, or a notice it -cannot read. A slow client may merge pending notices: the union of keys, the latest -stamp, and axis changes kept in order. Because the wire is lossless, outputs keep the +cannot read. A slow client may merge pending notices: the union of keys, the covered +range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state. `GridExtent` is derived by the client, and in records mode the record count is +state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is fetched. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for @@ -287,8 +294,8 @@ come out through clause 7's overlay. Numeric widths stay with ADR 008's view-lay - **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An inverse delta is not a rewind, because `delay` and `differentiate` keep history. - **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, - `ExactSum` per summed group, occupancy per live shape, the successor-query index and - the width histograms. The store itself is the integrated base and is never copied. It + `ExactSum` per summed group, occupancy per live shape, the successor-query index, the + width histograms and the bounded notice log. The store itself is the integrated base and is never copied. It is measured at ADR 008's fixture before the store-memory work begins. ## Alternatives rejected From 731a04ecd560917b25a11026f6b9adde42056f58 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:26:59 -0700 Subject: [PATCH 088/106] docs(adr): ADR 009 answers stale state with an engine correction operation (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 39 +++++++++++++++-------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 99dee2da..4de8bf20 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -220,17 +220,26 @@ an **edge adapter**. So every edge has the same semantics: Every request can be retried, and a resync from a publication stamp is idempotent. A client asks for "notices since n", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client. A request older than - the log's start is answered "refetch". + the log's start is answered with a correction (below). - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, - treats the hop as dropped. The downstream side then resyncs: it refetches through the - path, and the host answers from current state. + treats the hop as dropped. The downstream side then asks for a correction from its + last stamp. + +- **Correction is an engine operation.** The engine offers, per shape key, a + **correction** computed from a client's last stamp. Where the shared log still covers + that stamp, it is the net delta since then. Otherwise it is a replacing delta: the + current state for the client's window, with whatever the client held retracted. A + full refetch is the case with an empty basis. A correction is idempotent, resumable + by stamp, and written in the same lossless delta form as a notice, so a client, a + gateway or a worker applies it with the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the model side, which may itself span a hop. Which other edges cross a wire is a deployment choice. The engine's -invalidation signal, which ADR 006's replay trigger relies on, is the change notice and -the refetch rule below, followed from a displayed cell's key back to the model. +the model side, which may itself span a hop. Which other edges cross a wire is a +deployment choice. The engine's invalidation signal, which ADR 006's replay trigger +relies on, is a correction trigger below, followed from a displayed cell's key back to +the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left and dimensions. The window is never state, and a request costs O(visible). @@ -261,13 +270,13 @@ client, addressed to none. It carries: the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. -A notice whose key lists exceed a fixed size carries "refetch" instead. +A notice whose key lists exceed a fixed size carries "correct" instead. **What a client does.** It intersects the moved keys with its window. A fold row has no -maintained key, so it maps member keys through its own remap first. It then refetches -what moved. It refetches everything when it sees a publication gap, a new -`document_generation` or host instance, its shape in the eviction list, or a notice it -cannot read. A slow client may merge pending notices: the union of keys, the covered +maintained key, so it maps member keys through its own remap first. It then fetches what +moved. It asks for a correction when it sees a publication gap, a new +`document_generation` or host instance, its shape in the eviction list, a notice it +cannot read, or a dropped hop. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. @@ -336,11 +345,13 @@ test): 8. delivery. Step 8's required tests: -- `[AddItem, SetCell]` publishes one notice, and no client refetches; +- `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; - a dropped notice is detected; -- a reconnect after a host restart refetches; +- a reconnect after a host restart asks for a correction; +- a corrected state equals the state of a client that was never stale (a property + test over random edit sequences and losses); - a refused frame consumes no publication number; -- a dropped hop forces a refetch. +- a dropped hop triggers a correction. ## Open questions From 2bea189c6743c3ad04a1f7448160498d4b357998 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:28:49 -0700 Subject: [PATCH 089/106] docs(adr): ADR 009 states correction as the Z-set difference target minus held (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 27 +++++++++++++++-------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 4de8bf20..0687f53e 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -227,13 +227,21 @@ an **edge adapter**. So every edge has the same semantics: treats the hop as dropped. The downstream side then asks for a correction from its last stamp. -- **Correction is an engine operation.** The engine offers, per shape key, a - **correction** computed from a client's last stamp. Where the shared log still covers - that stamp, it is the net delta since then. Otherwise it is a replacing delta: the - current state for the client's window, with whatever the client held retracted. A - full refetch is the case with an empty basis. A correction is idempotent, resumable - by stamp, and written in the same lossless delta form as a notice, so a client, a - gateway or a worker applies it with the same code. +- **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` + is the current state for the client's window and `held` is what the client holds. + Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so + the group structure of `ZSet` provides it, with no separate machinery. The client + sends its last stamp: + - where the shared log covers that stamp, the host sends the sum of the logged deltas + since then; + - otherwise the host sends `target`, and the client, which knows `held`, applies + `target − held`; + - a full refetch is `target − ∅`. + + A correction is labelled with the publication range it covers, so it is applied only + at the stamp it starts from, which makes it idempotent and resumable. It uses the same + lossless delta form as a notice, so a client, a gateway or a worker applies it with the + same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -348,8 +356,9 @@ Step 8's required tests: - `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; - a dropped notice is detected; - a reconnect after a host restart asks for a correction; -- a corrected state equals the state of a client that was never stale (a property - test over random edit sequences and losses); +- a corrected state equals the target, `held + (target − held) = target`, and + correcting a state that is already correct is the zero Z-set (property tests over + random edit sequences and losses); - a refused frame consumes no publication number; - a dropped hop triggers a correction. From 7ea88606e28fba9f8741fd8a37e119e2c77867c1 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:29:21 -0700 Subject: [PATCH 090/106] docs(adr): ADR 009 names the client's position a watermark (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0687f53e..ad507a64 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -225,21 +225,24 @@ an **edge adapter**. So every edge has the same semantics: alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, treats the hop as dropped. The downstream side then asks for a correction from its - last stamp. + watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` is the current state for the client's window and `held` is what the client holds. Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so - the group structure of `ZSet` provides it, with no separate machinery. The client - sends its last stamp: - - where the shared log covers that stamp, the host sends the sum of the logged deltas - since then; + the group structure of `ZSet` provides it, with no separate machinery. The client's + position is a **watermark**: the publication number up to which it has applied the + stream, so everything at or before it is applied. There is one publication sequence, + so a watermark is one number, held per shape key the client reads. The host and any + gateway keep no watermarks: a client sends its own with each request. + - where the shared log covers the watermark, the host sends the sum of the logged + deltas after it; - otherwise the host sends `target`, and the client, which knows `held`, applies `target − held`; - a full refetch is `target − ∅`. A correction is labelled with the publication range it covers, so it is applied only - at the stamp it starts from, which makes it idempotent and resumable. It uses the same + at the watermark it starts from, which makes it idempotent and resumable. It uses the same lossless delta form as a notice, so a client, a gateway or a worker applies it with the same code. @@ -295,8 +298,8 @@ fetched. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for results and notices and a source of requests. Toward the client it is the mirror image. -Each hop has an adapter at each end, and each end keeps only that hop's state: the last -stamp it applied and its liveness timer. A gateway hop (browser ⇄ gateway ⇄ host) may +Each hop has an adapter at each end, and each end keeps only that hop's state: its +liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ host) may serve many clients behind one upstream connection, so the host sends one notice per gateway rather than one per client. In process, an edge is a function call. From 5239d3b95a54b70a045dc4a0625f7953b5ace290 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:31:39 -0700 Subject: [PATCH 091/106] docs(adr): ADR 009 aligns with ADRs 005, 006, 007 and 008 after the cross-read (improvise-felo.4) Folds use the natural CellKey order and Seq is gone; no view action reorders the store; rollback joins the per-command snapshot; a notice per committed command, with a view-definitions flag; the non-finite wire change to ADR 005 is stated; the watermark is the whole stamp and corrections are restricted to the window; deltas cross a hop only after commit; the totals overlay is conditional on ADR 008; this ADR owns the text-width rule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 141 ++++++++++++---------- 1 file changed, 78 insertions(+), 63 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ad507a64..794658c7 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -3,7 +3,8 @@ **Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Relates to:** ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 +**Relates to:** ADR 001 (the basis a write claims), ADR 002 (surrogates on the wire), +ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 ## Problem @@ -26,9 +27,8 @@ A problem-agnostic workspace crate with no dependencies beyond `std` (so it is wasm32-clean and serde-free). It provides: - `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; - a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`, - `differentiate`, - `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, - `distinct` and `aggregate`; + `differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join + `join_indexed`, `distinct` and `aggregate`; - resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it can run as a sliced job; - operator state behind one trace interface (look up, read a range, apply a delta), so @@ -54,18 +54,15 @@ is removed once its remaining uses are replaced (open question 3). ### 3. The cell delta -`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`: -- `+(k, v, s)` for a new cell; -- `−(k, v₀, s) + (k, v₁, s)` for an overwrite; -- `−(k, v₀, s)` for a removal; -- a remove-then-set gets a fresh `s′`. +`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue)>`: +- `+(k, v)` for a new cell; +- `−(k, v₀) + (k, v₁)` for an overwrite; +- `−(k, v₀)` for a removal. -`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an -overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite +Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the -natural order of ids (ADR 007 amendment 1, clause 4). A store reorder, such as entering -records mode, reassigns `Seq` and rebuilds clause 5's per-record stage. The circuit steps -once per model effect. +natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The +circuit steps once per model effect. The record tier and the dirty flag read this delta instead of scanning the store. The dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the @@ -107,8 +104,8 @@ The affected pairs are evaluated in the formula graph's topological order. **An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: 1. A per-record stage retracts and reinserts each touched record. It reads the record's previous cells as the store minus the delta, so it keeps no copy of the values. -2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum - `Seq` of the record's live cells), which is eager's concatenation order. +2. A map produces `(context, order key, expr)`. The order key is (context rank, record + key in `CellKey` order), which is eager's concatenation order. 3. The group aggregate, with an error channel that reads the first `Err` in order. No cross product is built. A formula whose expression reads other records is evaluated @@ -150,9 +147,11 @@ changes no operator. The visible axis is a remap. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). -**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row -leaves it. The shape key is shared and a hide is one client's, so totals are kept over -all rows. A read subtracts the request's H hidden cells as a temporary overlay: +**Column totals and hidden rows.** This holds if hidden items travel in a request, which +is ADR 008's open question. If they do not, totals are over all rows. A total pools only +the drawn rows, so a hidden row leaves it. The shape key is shared and a hide is one +client's, so totals are kept over all rows. A read subtracts the request's H hidden cells +as a temporary overlay: - `Count` subtracts; - `Min` and `Max` skip keys whose whole count is hidden; - text counts the distinct keys left; @@ -192,12 +191,13 @@ an **edge adapter**. So every edge has the same semantics: tuple of ids, its position or an ADR 002 surrogate. - **The wire form is lossless.** Nothing is lost in transport, numeric precision above all: - - a number never crosses as a JSON or JS double. A `numtower` `Num` crosses as its - canonical text (ADR 005). An f64 crosses as its exact bits, which keeps `-0.0` and - NaN payloads. That conflicts with today's `wire_f64` and with `numtower`'s `nan` - token, which both drop the payload, so the edge carries bits for every non-finite - value; - - integers wider than 2⁵³, including positions and counts, cross as strings; + - a value never crosses as a JSON or JS double. A `numtower` `Num` crosses as its + canonical text (ADR 005), and an inexact finite value keeps its `#i` spelling, which + is already bit-exact, `-0.0` included. A non-finite value crosses as its bits, which + keeps NaN payloads. **That changes accepted ADR 005 clause 11**, whose wire spells + non-finite values as `inf`, `-inf` and `nan`, and it needs approval with this ADR; + - positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as + strings; - text crosses byte-exact, with no normalisation; - the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs a sum gets the host's exact read or rebuilds it from the exact deltas. @@ -206,11 +206,12 @@ an **edge adapter**. So every edge has the same semantics: every element type has a round-trip property test, where encoding then decoding is the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and `i128`-wide fractions. -- **Stamps compose end to end.** The host stamps each published batch (below), and an +- **Stamps compose end to end.** The host stamps each committed command (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. - **One clock per process.** Each process runs its own synchronous sub-circuit. Across a - hop, the stamped batches are the clock. + hop, the stamped commands are the clock. A delta crosses a hop only after its command + commits, so a remote operator never sees a delta that is later rolled back. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces exactly: deltas are summed, and notices merged. A merged notice carries the range of publications it covers, `(from, to]`, so a downstream side sees no gap where nothing @@ -219,8 +220,9 @@ an **edge adapter**. So every edge has the same semantics: responses, for example several HTTP requests on one connection, as well as a socket. Every request can be retried, and a resync from a publication stamp is idempotent. A client asks for "notices since n", and the host answers from **one shared, bounded log - of published notices**: a single log, not one per edge or client. A request older than - the log's start is answered with a correction (below). + of published notices**: a single log, not one per edge or client, holding the notices + themselves (keys and flags, not values). A request older than the log's start is + answered with a correction (below). - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, @@ -231,20 +233,23 @@ an **edge adapter**. So every edge has the same semantics: is the current state for the client's window and `held` is what the client holds. Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so the group structure of `ZSet` provides it, with no separate machinery. The client's - position is a **watermark**: the publication number up to which it has applied the - stream, so everything at or before it is applied. There is one publication sequence, - so a watermark is one number, held per shape key the client reads. The host and any + position is a **watermark**: the whole stamp `(host instance, document_generation, + publication)` up to which it has applied the stream, so everything at or before it is + applied. A watermark from another host instance or generation is behind every current + stamp. The client holds one per shape key it reads. The host and any gateway keep no watermarks: a client sends its own with each request. - - where the shared log covers the watermark, the host sends the sum of the logged - deltas after it; - - otherwise the host sends `target`, and the client, which knows `held`, applies + - where the shared log covers the watermark, the host sends the current values of the + keys the log lists as moved since then, restricted to the client's window, and the + client applies `target − held` over those keys. The log holds keys and flags; values + are always read from the model's current state, so nothing per client is kept; + - otherwise the host sends `target` for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. - A correction is labelled with the publication range it covers, so it is applied only - at the watermark it starts from, which makes it idempotent and resumable. It uses the same - lossless delta form as a notice, so a client, a gateway or a worker applies it with the - same code. + A correction is labelled with the publication range it covers, so it is applied only at + the watermark it starts from, which makes it idempotent and resumable. Its elements are + `(key, value)` pairs with weights, in the lossless wire form, so a client, a gateway or a + worker applies it with the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -269,12 +274,14 @@ either answer. If the client prunes, it prunes its window, fills forward with th query, and, when scrolling up or left, finds a new top-left by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). -**The change notice.** After each accepted batch the host emits one notice to every -client, addressed to none. It carries: +**The change notice.** Each command commits on its own (ADR 006), and after each +committed command the host emits one notice to every client, addressed to none. A frame +whose later command fails keeps its earlier commands, and their notices, published. A +notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number - increments once per published batch, and the host instance changes on restart; -- axis changes, display relabels as `(id, display)`, and a flag when the formula set or - `measure_agg` moved. A shared display's ` #2` suffix is never sent: the client computes + increments once per committed command, and the host instance changes on restart; +- axis changes, display relabels as `(id, display)`, and a flag when the formula set, + `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as @@ -287,35 +294,41 @@ A notice whose key lists exceed a fixed size carries "correct" instead. maintained key, so it maps member keys through its own remap first. It then fetches what moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it -cannot read, or a dropped hop. A slow client may merge pending notices: the union of keys, the covered -range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the -circuit's bit identity end to end. +cannot read, or a dropped hop. A slow client may merge pending notices: the union of +keys, the covered range of publications, and axis changes kept in order. Because the +wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is -fetched. +state, only the shared notice log. `GridExtent` is derived by the client, and in records +mode the record count is fetched. `Projection` also published the ADR 001 basis a write +claims. Whether `project` returns it instead, and what replaces the layout tier, are +ADR 008's questions. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for results and notices and a source of requests. Toward the client it is the mirror image. Each hop has an adapter at each end, and each end keeps only that hop's state: its -liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ host) may -serve many clients behind one upstream connection, so the host sends one notice per -gateway rather than one per client. In process, an edge is a function call. +liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ +host) may serve many clients behind one upstream connection, so the host sends one notice +per gateway rather than one per client. In process, an edge is a function call. ### 10. Text column widths -The host maintains each text column's maximum display width as a histogram from width to -count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows -come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule. +This ADR owns the rule for text columns. The host maintains each text column's maximum +display width as a histogram from width to count, per (shape key, column), fed by the +delta of the column's text cells. Hidden rows come out through clause 7's overlay, where +that applies. Numeric widths stay with ADR 008's view-layer rule. ### 11. Rollback and memory -- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An - inverse delta is not a rewind, because `delay` and `differentiate` keep history. -- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, - `ExactSum` per summed group, occupancy per live shape, the successor-query index, the - width histograms and the bounded notice log. The store itself is the integrated base and is never copied. It +- **Rollback.** The circuit's state joins ADR 006's per-command snapshot, which is taken + before a command's effects and restored if the command fails. An inverse delta is not + a rewind, because `delay` and `differentiate` keep history. The requirement is that + this snapshot does not cost O(cells) per command against ADR 008's 8 ms budget, and + that writes do not block reads. How is for the implementation epic. +- **Memory.** Resident state beside the store is the multisets, `ExactSum` per summed + group, occupancy per live shape, the successor-query index, the width histograms and + the bounded notice log. The store itself is the integrated base and is never copied. It is measured at ADR 008's fixture before the store-memory work begins. ## Alternatives rejected @@ -342,7 +355,7 @@ come out through clause 7's overlay. Numeric widths stay with ADR 008's view-lay - Integrated state is O(cells) beside the store. - `ExactSum` costs 568 bytes per summed group. - Every client receives every notice. -- A mutating frame still clones the state for rollback. +- Each command snapshots the circuit's state for rollback. **Order** (each step is a ticket in the implementation epic, behind the differential test): @@ -356,13 +369,15 @@ test): 8. delivery. Step 8's required tests: -- `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; +- a frame `[AddItem, SetCell]` publishes one notice per committed command, numbered + consecutively, and no client asks for a correction; - a dropped notice is detected; - a reconnect after a host restart asks for a correction; - a corrected state equals the target, `held + (target − held) = target`, and correcting a state that is already correct is the zero Z-set (property tests over random edit sequences and losses); -- a refused frame consumes no publication number; +- a refused command consumes no publication number, and the commands before it in its + frame stay published; - a dropped hop triggers a correction. ## Open questions From 13b3dff35e09095ad1e8126c4c6cbf02004e5550 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:35:35 -0700 Subject: [PATCH 092/106] docs(adr): ADR 009 reads only committed state and sends a whole window when a range moved positions (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 +++++++++++++---------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 794658c7..c854b1aa 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -140,18 +140,19 @@ aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or ad changes no operator. The visible axis is a remap. - **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas - arriving during the build are buffered and added at install. + arriving during the build are buffered and added at install. The finished result is the + response to the request that asked for it, not a notice. - **A formula edit** is a formula delta fed to the circuit. - **A document swap** rebuilds everything. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). -**Column totals and hidden rows.** This holds if hidden items travel in a request, which -is ADR 008's open question. If they do not, totals are over all rows. A total pools only -the drawn rows, so a hidden row leaves it. The shape key is shared and a hide is one -client's, so totals are kept over all rows. A read subtracts the request's H hidden cells -as a temporary overlay: +**Column totals and hidden rows.** Totals are kept over all rows, because the shape key +is shared and a hide is one client's. Whether hidden items travel in a request is ADR +008's open question. If they do not, a total is read as it is kept. If they do, a total +must leave the hidden rows out, so a read subtracts the request's H hidden cells as a +temporary overlay: - `Count` subtracts; - `Min` and `Max` skip keys whose whole count is hidden; - text counts the distinct keys left; @@ -218,8 +219,8 @@ an **edge adapter**. So every edge has the same semantics: was lost. - **No long-lived stream assumed.** An edge may be carried as a series of requests and responses, for example several HTTP requests on one connection, as well as a socket. - Every request can be retried, and a resync from a publication stamp is idempotent. A - client asks for "notices since n", and the host answers from **one shared, bounded log + Every request can be retried, and a resync from a watermark is idempotent. A client + asks for "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). @@ -238,18 +239,23 @@ an **edge adapter**. So every edge has the same semantics: applied. A watermark from another host instance or generation is behind every current stamp. The client holds one per shape key it reads. The host and any gateway keep no watermarks: a client sends its own with each request. - - where the shared log covers the watermark, the host sends the current values of the - keys the log lists as moved since then, restricted to the client's window, and the - client applies `target − held` over those keys. The log holds keys and flags; values - are always read from the model's current state, so nothing per client is kept; - - otherwise the host sends `target` for the whole window, and the client applies - `target − held`; + - where the shared log covers the watermark and the covered range holds only moved + keys, the host sends the current values of those keys, restricted to the client's + window, and the client applies `target − held` over them. The log holds keys and + flags, not values, so nothing per client is kept; + - where the covered range holds an axis change, a relabel or a flag, which position + and labels depend on, the host sends `target` for the whole window; + - where the log no longer covers the watermark, the host sends `target` for the whole + window, and the client applies `target − held`; - a full refetch is `target − ∅`. - A correction is labelled with the publication range it covers, so it is applied only at - the watermark it starts from, which makes it idempotent and resumable. Its elements are - `(key, value)` pairs with weights, in the lossless wire form, so a client, a gateway or a - worker applies it with the same code. + **Every read sees only the last committed state**, whether it is a correction, a + fetch or `project`, and it is labelled with that state's stamp. A value a failed + command wrote and then rolled back is never read. A correction is labelled with the + publication range it covers, so it is applied only at the watermark it starts from, + which makes it idempotent and resumable. Its elements are `(key, value)` pairs with + weights, in the lossless wire form, so a client, a gateway or a worker applies it with + the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -384,7 +390,7 @@ Step 8's required tests: 1. **Should notices be filtered per client?** Clause 9 sends every client every live shape's moved keys. That keeps the host free of per-client state, but traffic is - O(clients × all moved keys) per batch. Filtering by topic would send each connection + O(clients × all moved keys) per committed command. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the backend may hold? On a two-hop path the topics could instead live in a gateway. From cc113c732fd45d14c1c02a5fcdf1b28a3f680b18 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Fri, 9 Oct 2026 21:42:24 -0700 Subject: [PATCH 093/106] docs(context): answer the review of the repo-map split - repo-stats reads the gotcha list from context/gotchas.md (the digest in repo-map.md no longer measures the doc-growth signal) and reports it as `docs.gotchas`; the test fixture and its derived expectations follow. - AGENTS.md and the wave agent-rules protect gotchas.md and file-inventory.md like the repo map they were split from. - doc-site.md's publication table lists the two new pages. - file-inventory.md's Context docs rows describe the three files correctly. - the code-review skill points at gotchas.md for the long form. Co-Authored-By: Claude Sonnet 5.5 --- .agents/skills/code-review/SKILL.md | 2 +- .agents/skills/wave/agent-rules.md | 3 ++- AGENTS.md | 3 ++- context/code/doc-site.md | 2 +- context/file-inventory.md | 4 +++- scripts/repo-stats | 15 ++++++--------- scripts/repo-stats-test.sh | 21 ++++++++++----------- 7 files changed, 25 insertions(+), 25 deletions(-) diff --git a/.agents/skills/code-review/SKILL.md b/.agents/skills/code-review/SKILL.md index 1594097a..b4687084 100644 --- a/.agents/skills/code-review/SKILL.md +++ b/.agents/skills/code-review/SKILL.md @@ -25,7 +25,7 @@ them here would let the copy drift: | Source | What it holds | |---|---| | `context/design-principles.md` | The "why" — functional-first, polymorphism over conditionals, correctness by construction, layering, §7 testing doctrine | -| `context/repo-map.md` | The "where", plus **numbered gotchas** — read the ones your diff touches | +| `context/repo-map.md` | The "where", plus a **numbered gotcha digest** (the full text is `context/gotchas.md`) — read the long form of the ones your diff touches | | `prolog-sessions/kb.pl` | Executable knowledge: `invariant/2`, `doctrine/2`, `forbidden/2`, `decision/3`, and prior `work_done/3` | | `AGENTS.md` | Working rules: bug-first tests, `Option`/`Result`/`Box` at the edges, history reconciled only by merge, a new conventional commit by default (amend only an unpushed HEAD, for a small fix), and nothing that needs a force push | | `.agents/skills/wave/agent-rules.md`, "GitHub" | The reviewer's rule: what to read before your first comment, how to answer each existing review, and what every review you post must state. Read it before reviewing a PR | diff --git a/.agents/skills/wave/agent-rules.md b/.agents/skills/wave/agent-rules.md index 444059f8..16840d3b 100644 --- a/.agents/skills/wave/agent-rules.md +++ b/.agents/skills/wave/agent-rules.md @@ -341,7 +341,8 @@ out on purpose: ## Files you must not write `.beads/**` (never commit `issues.jsonl`, never `git checkout` it), -`prolog-sessions/kb.pl`, `context/repo-map.md`: `AGENTS.md`, "Who owns what". +`prolog-sessions/kb.pl`, `context/repo-map.md`, `context/gotchas.md`, +`context/file-inventory.md`: `AGENTS.md`, "Who owns what". Also `AGENTS.md`, unless your brief grants you a named section. Report proposed text for any of these in your hand-back. diff --git a/AGENTS.md b/AGENTS.md index b5c5e18a..845b1507 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -160,8 +160,9 @@ badly: | `.beads/` (incl. `issues.jsonl`) | bd owns it. See "Never hand-edit" below. | | `prolog-sessions/kb.pl` | Written by `scripts/kb` only. See `context/code/kb-tool.md`. | | `context/repo-map.md` | Every agent wants to add a gotcha; every wave would conflict. | +| `context/gotchas.md`, `context/file-inventory.md` | The long forms the map digests; same conflict, same owner. | -Wave agents **read** all three freely (`bd show`, `bd ready`, grep `kb.pl`) and +Wave agents **read** all of them freely (`bd show`, `bd ready`, grep `kb.pl`) and **write** none of them. They report proposed repo-map lines and `kb.pl` facts as *text* in their hand-back, and the orchestrator applies them. Repo-map lines are batched at the end of the wave, after the review, because they conflict. diff --git a/context/code/doc-site.md b/context/code/doc-site.md index 09a9eb01..5a364b4c 100644 --- a/context/code/doc-site.md +++ b/context/code/doc-site.md @@ -18,7 +18,7 @@ Published, in four sections: | Section | Contents | Why | |---|---|---| -| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md` | What the project is for, the doctrine, and the single best explanation of how it is built. | +| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md`, `context/gotchas.md`, `context/file-inventory.md` | What the project is for, the doctrine, and the single best explanation of how it is built. The map is a digest; the last two are its long forms, published so a `gotcha N` citation and the per-file listing have somewhere to land. | | Decision records | `context/adr/*.md` | An ADR is written for an outside reader by construction — it states the options that were rejected, which is the part that is worth reading and the part that is nowhere else. | | The arguments, by area | `context/code/*.md` | This is the reasoning that used to live in code comments and was moved out under the two-line cap (design-principles §6). It is the most valuable prose in the repository and it is addressed to exactly the reader a documentation site has. | | Test evidence | `context/test-evidence/*.md` | One entry per test whose narrative is longer than its code. | diff --git a/context/file-inventory.md b/context/file-inventory.md index 6ba97b8b..aa56a4f2 100644 --- a/context/file-inventory.md +++ b/context/file-inventory.md @@ -342,7 +342,9 @@ context/adr/001-concurrent-edits.md Concurrent edits between clients: record-le context/cloud-agents.md Cloud-session environment: tracker sync, bd pitfalls, what is blocked context/design-principles.md Architectural principles & testing doctrine context/plan.md Show HN launch plan -context/repo-map.md This file +context/repo-map.md The map: crates, how-to-find-things, the gotcha digest, a file map +context/gotchas.md The long form of every numbered gotcha (the digest is in repo-map.md) +context/file-inventory.md This file: the per-file listing the repo map summarises docs/design-notes.md Product vision & non-goals ``` diff --git a/scripts/repo-stats b/scripts/repo-stats index 6444d2a8..f96ab557 100755 --- a/scripts/repo-stats +++ b/scripts/repo-stats @@ -189,12 +189,9 @@ def read_rows(path): def gotchas(text): - """(number, lines, chars) for each numbered item of repo-map's Gotchas section.""" - out, cur, inside = [], None, False + """(number, lines, chars) for each numbered item of context/gotchas.md.""" + out, cur, inside = [], None, True for line in text.split("\n"): - if line.startswith("## "): - inside = line.startswith("## Gotchas") - continue if inside and line.startswith("---"): inside = False if not inside: @@ -214,13 +211,13 @@ def doc_sizes(repo, sha): "context").split("\n") if p.endswith(".md")] show = lambda p: git(repo, "show", f"{sha}:{p}") lines = {p: len(show(p).splitlines()) for p in sorted(files)} - gs = gotchas(show("context/repo-map.md")) if "context/repo-map.md" in files else [] + gs = gotchas(show("context/gotchas.md")) if "context/gotchas.md" in files else [] longest = max(gs, key=lambda g: g[2], default=None) evidence = {p: sum(1 for l in show(p).split("\n") if re.match(r"^#{3,4} ", l)) for p in sorted(files) if p.startswith("context/test-evidence/")} return { "context_lines": lines, - "repo_map": {"gotchas": len(gs), "longest": longest and + "gotchas": {"count": len(gs), "longest": longest and {"number": longest[0], "lines": longest[1], "chars": longest[2]}}, "test_evidence": {"entries": sum(evidence.values()), "per_file": evidence}, } @@ -270,9 +267,9 @@ def text(r, top): out += ["", TITLES[t]] + table(r["tables"][t]) out += ["", "Doc sizes (lines)"] out += [f" {n:>6} {path}" for path, n in r["docs"]["context_lines"].items()] - rm, te = r["docs"]["repo_map"], r["docs"]["test_evidence"] + rm, te = r["docs"]["gotchas"], r["docs"]["test_evidence"] lg = rm["longest"] - out.append(f" repo-map gotchas: {rm['gotchas']}" + ( + out.append(f" gotchas: {rm['count']}" + ( f"; longest is {lg['number']} ({lg['lines']} lines, {lg['chars']} chars)" if lg else "")) out.append(f" test-evidence entries: {te['entries']}") diff --git a/scripts/repo-stats-test.sh b/scripts/repo-stats-test.sh index 2e138dbd..65fbe5e2 100755 --- a/scripts/repo-stats-test.sh +++ b/scripts/repo-stats-test.sh @@ -76,10 +76,9 @@ put context/code/x.md 8 x3 mkdir -p "$repo/context/test-evidence" printf '### t_one\ntext\n#### t_two\n## not an entry\n### t_three\n' \ > "$repo/context/test-evidence/y.md" -cat > "$repo/context/repo-map.md" <<'EOF' -# Map - -## Gotchas (read before editing) +printf '# Map\n' > "$repo/context/repo-map.md" +cat > "$repo/context/gotchas.md" <<'EOF' +# Gotchas 1. **One.** short. 2. **Two.** a much longer gotcha, with plenty of words in it to be the longest one @@ -136,25 +135,25 @@ if run "$work/stats" text; then row "docs/*.md is other docs" by-area "other docs" '["2","0","0.000"]' row "context/code" by-area "context/code" '["8","0","0.000"]' row "context/test-evidence" by-area "context/test-evidence" '["5","0","0.000"]' - row "repo-map is its own area" by-area "repo-map" '["13","0","0.000"]' - row "other context" by-area "other context" '["2","0","0.000"]' + row "repo-map is its own area" by-area "repo-map" '["1","0","0.000"]' + row "other context" by-area "other context" '["13","0","0.000"]' row "kb.pl" by-area "kb.pl" '["5","0","0.000"]' row "scripts/CI takes .github, and a rename is a delete and an add" \ by-area "scripts/CI" '["15","7","0.467"]' row "web and harness, and its deletion ratio" by-area "web and harness" '["14","9","0.643"]' row "a lock file, .beads/ and site/ are generated" by-area "generated" '["14","0","0.000"]' row "the pre-window README counts nowhere" by-area "other" '["4","0","0.000"]' - row "prose is the context areas, other docs and agent docs" by-kind prose '["43","6","0.140"]' + row "prose is the context areas, other docs and agent docs" by-kind prose '["42","6","0.143"]' row "code is rust, scripts/CI and web" by-kind code '["82","45","0.549"]' - row "prose to code is prose over code" by-kind ProseToCode '["0.524","0.133",""]' + row "prose to code is prose over code" by-kind ProseToCode '["0.512","0.133",""]' is "weeks are columns, first-seen first" '.tables["area-by-week"][0][1:]' '["2026-W24","2026-W23"]' row "an area is split by ISO week" area-by-week "web and harness" '["3","11"]' is "every day's churn.csv total equals code-maat's abs-churn" \ '[.tables["by-day"][1:][]|.[1]==.[2]]|all' 'true' - row "a day's total is every file changed that day" by-day 2026-06-04 '["72","72"]' - is "gotchas end with their section" '.docs.repo_map.gotchas' '3' + row "a day's total is every file changed that day" by-day 2026-06-04 '["71","71"]' + is "gotchas end with their section" '.docs.gotchas.count' '3' is "the longest gotcha is measured over its lines" \ - '.docs.repo_map.longest|[.number,.lines]' '[2,2]' + '.docs.gotchas.longest|[.number,.lines]' '[2,2]' is "test-evidence entries are ### and #### headings" '.docs.test_evidence.entries' '3' is "coupling finds the 15 pairs of six files, and web/app.js with scripts/foo.sh" \ '.coupling|length' '16' From e720f7ca509f0643981765ff6b9b182fb0a1bc25 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:11:37 -0700 Subject: [PATCH 094/106] docs(adr): ADR 009 makes the formula language the query language and states the incremental principle (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 37 ++++++++++++++++------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index c854b1aa..0e14cdd1 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -21,6 +21,10 @@ cell write becomes a delta, aggregates are maintained, and only the formula cell inputs moved are re-evaluated by the existing evaluator. The incremental results must equal today's eager results exactly. +**Principle: everything is as incremental as possible**, so the work done per model +mutation is minimised. Where this ADR does work that grows with more than the +mutation, it says so and names the incremental path that normally applies. + ### 1. The `zset` crate A problem-agnostic workspace crate with no dependencies beyond `std` (so it is @@ -34,8 +38,11 @@ wasm32-clean and serde-free). It provides: - operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it. -There are no nested streams, no recursion operator, no timestamps beyond one clock, and -no query language. +There are no nested streams, no recursion operator and no timestamps beyond one clock. +**The formula language is the query language.** What a request asks of the model beyond +its axes and window (filters, aggregation, derived values) is written in formula text +over ids (ADR 007). It goes through the same evaluator and circuit as a document +formula, so there is no second query language. ### 2. The plug @@ -85,7 +92,9 @@ shape key with the same None list, which removes the per-session model clone. A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the -ordering category's order. How that order is keyed is for those tickets. +ordering category's order. Those tickets choose the key. It must keep a reorder or an +insert incremental: only the entries under the moved items change, and nothing is +rebuilt. ### 5. Formulas @@ -118,7 +127,8 @@ before enumerating. This replaces `Expansion::members`. DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the answer must be `Error("circular")`. The first version finds the formula graph's strongly -connected components. Acyclic formulas are evaluated once each, in topological order. A +connected components, recomputed when the formula set changes; how to keep that +incremental belongs with open question 2. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced by a DBSP-native treatment (open question 2). @@ -143,7 +153,8 @@ changes no operator. The visible axis is a remap. arriving during the build are buffered and added at install. The finished result is the response to the request that asked for it, not a notice. - **A formula edit** is a formula delta fed to the circuit. -- **A document swap** rebuilds everything. +- **A document swap** rebuilds everything, and a new key's first build is a full pass. + These are the only rebuilds; every other change is a delta. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). @@ -243,10 +254,11 @@ an **edge adapter**. So every edge has the same semantics: keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and flags, not values, so nothing per client is kept; - - where the covered range holds an axis change, a relabel or a flag, which position - and labels depend on, the host sends `target` for the whole window; - - where the log no longer covers the watermark, the host sends `target` for the whole - window, and the client applies `target − held`; + - where the covered range also holds axis changes, relabels or flags, which positions + and labels depend on, the correction carries them too, in order, as a merged notice + does, so the client re-derives positions incrementally; + - only where the log no longer covers the watermark does the host fall back to `target` + for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. **Every read sees only the last committed state**, whether it is a correction, a @@ -294,7 +306,8 @@ notice carries: the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. -A notice whose key lists exceed a fixed size carries "correct" instead. +A notice whose key lists exceed a fixed size carries "correct" instead. That is a +fallback, and it costs every reader a correction. **What a client does.** It intersects the moved keys with its window. A fold row has no maintained key, so it maps member keys through its own remap first. It then fetches what @@ -405,7 +418,9 @@ Step 8's required tests: such as `A = IF(x, B, 1)`; - (c) a monotone reformulation, where one exists. - *Recommendation:* (a). + Separately, the strongly connected components are recomputed whenever the formula set + changes. Should they be maintained incrementally too? *Recommendation:* (a), and + recompute the components only for the formulas an edit touches. 3. **What replaces `Eager`?** It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. *Recommendation:* a From 63ee13289bbdbfb781a348e0bf6246a50678b40b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:22:19 -0700 Subject: [PATCH 095/106] docs(adr): ADR 009 makes an edge a session rather than a two-hop path, and its clock causal (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 65 ++++++++++++----------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0e14cdd1..0568dab0 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -157,7 +157,8 @@ changes no operator. The visible axis is a remap. These are the only rebuilds; every other change is a delta. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names - it (the period is set by measurement), and every eviction is announced (clause 9). + it. The period is a local duration on the host, set by measurement, and never compared + with another machine's clock. Every eviction is announced (clause 9). **Column totals and hidden rows.** Totals are kept over all rows, because the shape key is shared and a hide is one client's. Whether hidden items travel in a request is ADR @@ -193,8 +194,7 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery **Every edge may cross a network.** An edge, meaning the stream from one operator to the -next, may be in-process or remote, and a path may cross up to two network hops: browser ⇄ -gateway ⇄ host, or host ⇄ worker ⇄ client. No edge assumes a function call or shared +next, may be in-process or remote. No edge assumes a function call or shared memory. An operator that sits at a wire boundary is just one implementation of an edge, an **edge adapter**. So every edge has the same semantics: - **A wire form.** Every element type that flows on an edge has a serializable form, @@ -218,27 +218,30 @@ an **edge adapter**. So every edge has the same semantics: every element type has a round-trip property test, where encoding then decoding is the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and `i128`-wide fractions. -- **Stamps compose end to end.** The host stamps each committed command (below), and an - intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up - downstream as a gap. -- **One clock per process.** Each process runs its own synchronous sub-circuit. Across a - hop, the stamped commands are the clock. A delta crosses a hop only after its command +- **The clock is causal, never a wall clock.** Ordering, staleness and loss detection + use only the stamp sequence (below), a logical clock the host advances once per + committed command. Each process runs its own synchronous sub-circuit, and across an + edge the stamped commands are the clock. No decision compares wall-clock times across + machines, so clock skew cannot affect it. A delta crosses a wire only after its command commits, so a remote operator never sees a delta that is later rolled back. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces exactly: deltas are summed, and notices merged. A merged notice carries the range of - publications it covers, `(from, to]`, so a downstream side sees no gap where nothing - was lost. -- **No long-lived stream assumed.** An edge may be carried as a series of requests and - responses, for example several HTTP requests on one connection, as well as a socket. - Every request can be retried, and a resync from a watermark is idempotent. A client + publications it covers, `(from, to]`, so the client that merges notices sees no gap + where nothing was lost. +- **An edge is a session, not a connection.** An edge may be one long-lived connection, + or a sequence of requests carrying the same session token or cookie, for example + several HTTP requests. The token only lets a client's requests be recognised as one + edge: the host keeps no per-client state, so what continues across connections is the + client's own watermark and shape keys. Every request can be retried, and a resync from + a watermark is idempotent. A client asks for "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). -- **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is - alive while its next message or request succeeds. Each hop keeps alive in both - directions. A side that hears nothing within the timeout, or whose request fails, - treats the hop as dropped. The downstream side then asks for a correction from its +- **Failure.** An edge is alive while its next message or request succeeds, and each end + keeps alive in both directions. Timeouts are local monotonic durations, never compared + across machines. A side that hears nothing within its timeout, or whose request fails, + treats the connection as dropped. The client then asks for a correction from its watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` @@ -248,8 +251,8 @@ an **edge adapter**. So every edge has the same semantics: position is a **watermark**: the whole stamp `(host instance, document_generation, publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current - stamp. The client holds one per shape key it reads. The host and any - gateway keep no watermarks: a client sends its own with each request. + stamp. The client holds one per shape key it reads. The host keeps no watermarks: a + client sends its own with each request. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and @@ -266,12 +269,11 @@ an **edge adapter**. So every edge has the same semantics: command wrote and then rolled back is never read. A correction is labelled with the publication range it covers, so it is applied only at the watermark it starts from, which makes it idempotent and resumable. Its elements are `(key, value)` pairs with - weights, in the lossless wire form, so a client, a gateway or a worker applies it with - the same code. + weights, in the lossless wire form, so a client and a worker apply it with the same + code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the model side, which may itself span a hop. Which other edges cross a wire is a -deployment choice. The engine's invalidation signal, which ADR 006's replay trigger +the model side. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger relies on, is a correction trigger below, followed from a displayed cell's key back to the model. @@ -313,7 +315,7 @@ fallback, and it costs every reader a correction. maintained key, so it maps member keys through its own remap first. It then fetches what moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it -cannot read, or a dropped hop. A slow client may merge pending notices: the union of +cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. @@ -324,12 +326,10 @@ mode the record count is fetched. `Projection` also published the ADR 001 basis claims. Whether `project` returns it instead, and what replaces the layout tier, are ADR 008's questions. -**Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for -results and notices and a source of requests. Toward the client it is the mirror image. -Each hop has an adapter at each end, and each end keeps only that hop's state: its -liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ -host) may serve many clients behind one upstream connection, so the host sends one notice -per gateway rather than one per client. In process, an edge is a function call. +**Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the +circuit, the client's adapter is a sink for results and notices and a source of requests. +Toward the client it is the mirror image. Each end keeps only its timeout, and the +client's end keeps the watermark. In process, an edge is a function call. ### 10. Text column widths @@ -397,7 +397,8 @@ Step 8's required tests: random edit sequences and losses); - a refused command consumes no publication number, and the commands before it in its frame stay published; -- a dropped hop triggers a correction. +- a dropped connection or a failed request triggers a correction, and a later request + with the same session resumes from the watermark. ## Open questions @@ -406,7 +407,7 @@ Step 8's required tests: O(clients × all moved keys) per committed command. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? On a two-hop path the topics could instead live in a gateway. + backend may hold? *Recommendation:* broadcast with the size cap. Add topics only if measured traffic with many clients on different shapes justifies them. From 16f40314fd7fc88fa5cd451981915be60e722bcf Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Fri, 9 Oct 2026 22:23:59 -0700 Subject: [PATCH 096/106] docs(context): act on the adversarial review of the repo-map split - AGENTS.md: "Three files" is now "These files"; the table lists five. - Digests 43 and 45 regain the traps the review found missing. - protocol.md and formula-and-io.md cite file-inventory.md for the Wasm appendix and the inventory, which moved there. Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 2 +- context/code/formula-and-io.md | 2 +- context/code/protocol.md | 2 +- context/repo-map.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 845b1507..8f0431f9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,7 +151,7 @@ that costs to get right. Everything in it was learned by getting it wrong first. ### Who owns what -Three files are **orchestrator-only**, and the rule exists because they are +These files are **orchestrator-only**, and the rule exists because they are guaranteed conflicts — every agent has a reason to touch them and they merge badly: diff --git a/context/code/formula-and-io.md b/context/code/formula-and-io.md index 9a8b88ad..7c7da5d8 100644 --- a/context/code/formula-and-io.md +++ b/context/code/formula-and-io.md @@ -24,7 +24,7 @@ over `unescape_pipe_body`. Before improvise-i2m5 there were **three** copies and they had drifted, over the keyword rule below. **`improvise-io` no longer has one** — its -`persistence/quoting.rs` is deleted. `context/repo-map.md`'s file inventory +`persistence/quoting.rs` is deleted. `context/file-inventory.md` kept claiming io "still has its own unguarded copy" long after the deletion — it was corrected in wave 1, but any *other* doc or comment still saying so is wrong, and that is the shape this particular fact goes stale in. diff --git a/context/code/protocol.md b/context/code/protocol.md index 30f4ec7d..e9b8bd2c 100644 --- a/context/code/protocol.md +++ b/context/code/protocol.md @@ -902,7 +902,7 @@ The feature is on for **every workspace crate with a normal dependency on serde_json** (improvise-dz8c). Each crate's own manifest names it, because each build is made from one crate alone: the `wasm` CI job and `wasm-pack` invoke per crate, and feature unification cannot reach across separate invocations -(`context/repo-map.md`, `## Wasm`). Enabling it in one crate and relying on the +(`context/file-inventory.md`, appendix "Wasm"). Enabling it in one crate and relying on the others to inherit it works in `cargo test` and nowhere else. **Why "can parse" and not "parses a frame".** The first version of the rule diff --git a/context/repo-map.md b/context/repo-map.md index b967a21f..7a64ddf9 100644 --- a/context/repo-map.md +++ b/context/repo-map.md @@ -205,9 +205,9 @@ the incidents and the ticket ids.** Read the long form of every gotcha your diff 40. **A batch's basis is three tiers** (`document_generation`, `layout_version`, `record_versions`); the unit is the record; lower tiers are *observed*, not read off the effect. 41. **`Storage` is not a VFS;** SQLite was declined on measurement (a second serialization is a second grammar). 42. **Browser render decisions are made in Rust** (`RenderModel`: `editable` from the `CellId` variant, selection reconciled with offsets, `flatten_cat_tree`). `improvise_protocol::frame` is the one envelope pair for both deployments. -43. **A batch carries its basis, and the server publishes it** (`Projection::Basis`). `Basis` has no `Default` or public constructor; `Checked`/`CheckedBatch` make "ignoring the basis" unrepresentable; `Rejection` keeps `RecordGone` / `RecordMoved` / `RecordUnstated` apart. Anti-staleness, not security. +43. **A batch carries its basis, and the server publishes it** (`Projection::Basis`). `Basis` has no `Default` or public constructor; `Checked`/`CheckedBatch` make "ignoring the basis" unrepresentable; `Rejection` keeps `RecordGone` / `RecordMoved` / `RecordUnstated` apart. Anti-staleness, not security. The check is a property of the session door (`apply_command` does not check), and only two of the three tiers hoist per frame — the record tier is asked per command (`records_named`) against the frame's basis. Read gotcha 43 before touching it. 44. **The TUI grid draws from `GridCache`**, which frames it too (`AxisFrame`; width rule `header_level_widths`). Use the one-pass accessors, not per-position scans. `RecordHash` is document-scoped, `AggregateHash` layout-scoped. -45. **One session loop (`Host`), two transports.** Saving is a request frame (`ClientFrame::Save`); dirtiness is raised from the `EffectKind`, observed for data. `impl Storage for Rc` lives in `improvise-io`. `harness/` runs `OpfsFs` for real; Safari is unseen. +45. **One session loop (`Host`), two transports.** Saving is a request frame (`ClientFrame::Save`); dirtiness is *declared* for `WholeWorkbook` and *observed* for `CellData`/`LayoutGeometry` (`MarkDirty` is gone). `impl Storage for Rc` lives in `improvise-io`. `harness/` runs `OpfsFs` for real; Safari is unseen. 46. **The page picks a peer:** `?ws=` → socket, else a module `Worker`. `#status` composes two facts; the first-frame watchdog is per-peer-agnostic. The demo model is built so every cell is a `Record`. 47. **Keys become commands in the browser;** the server must not know the keymap. `GridExtent`, `set_viewport`, `Commands(Vec)`, `resolve::NEEDS_THE_HOST` and the five client rules (`projected_at`, `opened`, `refused_here`, `Step::Declined`, `notify`). 48. **The browser JS is partly executed** (`harness/tests/shell.spec.js`); changes to `web/*.js` belong there. Serialize frames on one tail promise; disarm, don't pre-empt, a watchdog. Evidence headings may be module-qualified (and must be when the leaf collides). From 978bd4c8c9bbe1b6050a13c5c8f75b9119dcd2ff Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:25:22 -0700 Subject: [PATCH 097/106] docs(adr): ADR 009 makes the session token transport only and keys shapes by the parsed query (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 ++++++++++++----------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0568dab0..6099ec8b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -144,7 +144,8 @@ with the same key, from any client. The key is: - the page categories, so each page is a maintained slice and paging rebuilds nothing; - `measure_agg`. -Everything else in a request is applied when output is read: hidden and collapsed items +The key uses the canonical parsed id form of the request's query text, so two spellings +of one query share their operator state. Everything else in a request is applied when output is read: hidden and collapsed items on axis categories, item order, the page selection and prune. Occupancy and column aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item changes no operator. The visible axis is a remap. @@ -230,19 +231,19 @@ an **edge adapter**. So every edge has the same semantics: where nothing was lost. - **An edge is a session, not a connection.** An edge may be one long-lived connection, or a sequence of requests carrying the same session token or cookie, for example - several HTTP requests. The token only lets a client's requests be recognised as one - edge: the host keeps no per-client state, so what continues across connections is the - client's own watermark and shape keys. Every request can be retried, and a resync from - a watermark is idempotent. A client - asks for "notices after my watermark", and the host answers from **one shared, bounded log + several HTTP requests. The host decides nothing from the token: it is transport, and + any later authentication is out of scope. What continues across connections is the + client's own watermark and shape keys, which it sends with each request. Every request + can be retried, and a resync from a watermark is idempotent. A client asks for + "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). -- **Failure.** An edge is alive while its next message or request succeeds, and each end - keeps alive in both directions. Timeouts are local monotonic durations, never compared - across machines. A side that hears nothing within its timeout, or whose request fails, - treats the connection as dropped. The client then asks for a correction from its - watermark. +- **Failure.** Timeouts are local monotonic durations, never compared across machines. + On a long-lived connection, both ends keep alive, and a host-side timeout belongs to + that open connection and is released with it. On a sequence of requests, only the + client detects failure: a request that fails or times out. In either case the client + then asks for a correction from its watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` is the current state for the client's window and `held` is what the client holds. @@ -260,7 +261,8 @@ an **edge adapter**. So every edge has the same semantics: - where the covered range also holds axis changes, relabels or flags, which positions and labels depend on, the correction carries them too, in order, as a merged notice does, so the client re-derives positions incrementally; - - only where the log no longer covers the watermark does the host fall back to `target` + - only where the log no longer covers the watermark, or the covered range holds an + oversized notice that lists no keys, does the host fall back to `target` for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. @@ -299,7 +301,8 @@ committed command the host emits one notice to every client, addressed to none. whose later command fails keeps its earlier commands, and their notices, published. A notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number - increments once per committed command, and the host instance changes on restart; + increments once per committed command. The host instance changes on restart; it is a + random id, never a wall-clock time; - axis changes, display relabels as `(id, display)`, and a flag when the formula set, `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes it from the displays in scope; @@ -328,8 +331,8 @@ ADR 008's questions. **Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the circuit, the client's adapter is a sink for results and notices and a source of requests. -Toward the client it is the mirror image. Each end keeps only its timeout, and the -client's end keeps the watermark. In process, an edge is a function call. +Toward the client it is the mirror image. The client's end keeps the watermark. In +process, an edge is a function call. ### 10. Text column widths From 6bf899c9ec81c64b66b60ce22245085ff27f78b6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:26:21 -0700 Subject: [PATCH 098/106] docs(adr): ADR 009 makes per-client filtering an operator in the graph (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 41 +++++++++++------------ 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 6099ec8b..834b849f 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -57,7 +57,7 @@ computes each value, so formulas are never compiled into operators. **`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. It serves as the oracle of a differential property test, which applies one random edit sequence to both backends and compares every drawn cell, total and occupancy count. It -is removed once its remaining uses are replaced (open question 3). +is removed once its remaining uses are replaced (open question 2). ### 3. The cell delta @@ -128,10 +128,10 @@ before enumerating. This replaces `Expansion::members`. DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the answer must be `Error("circular")`. The first version finds the formula graph's strongly connected components, recomputed when the formula set changes; how to keep that -incremental belongs with open question 2. Acyclic formulas are evaluated once each, in topological order. A +incremental belongs with open question 1. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced -by a DBSP-native treatment (open question 2). +by a DBSP-native treatment (open question 1). ### 7. Operator state follows the request's shape key @@ -297,20 +297,29 @@ query, and, when scrolling up or left, finds a new top-left by galloping backwar doubling jumps (about 14 to 28 probes at density 10⁻⁴). **The change notice.** Each command commits on its own (ADR 006), and after each -committed command the host emits one notice to every client, addressed to none. A frame +committed command the host emits one notice, addressed to no client. A frame whose later command fails keeps its earlier commands, and their notices, published. A notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number increments once per committed command. The host instance changes on restart; it is a random id, never a wall-clock time; - axis changes, display relabels as `(id, display)`, and a flag when the formula set, - `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes - it from the displays in scope; + `measure_agg`, the view definitions or the default view moved. A shared display's + ` #2` suffix is never sent: the client computes it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. +**A client receives only what it needs.** Filtering is part of the DBSP graph, not a +separate mechanism. A client either pulls just the state it needs, or pushes a filter to +the host's side of its connection: a filter operator on its shape key and window, written +in the query language, through which its notices pass. A pushed filter belongs to the open +connection, like that connection's timeout, and is released with it. On a sequence of +requests the client sends it with each request, so the host keeps no per-client state +beyond an open connection. While a connection's filter is open, it counts as a request +naming its shape, so its shape is not evicted. + A notice whose key lists exceed a fixed size carries "correct" instead. That is a fallback, and it costs every reader a correction. @@ -360,8 +369,8 @@ that applies. Numeric widths stay with ADR 008's view-layer rule. - **Compiling formulas into operators:** a second evaluator. - **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a second description of evaluation. -- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic - only by holding per-client state on the host (open question 1). +- **A topic or subscription mechanism beside the graph:** filtering is an operator in + the graph (clause 9). ## Consequences @@ -376,7 +385,6 @@ that applies. Numeric widths stay with ADR 008's view-layer rule. - Two backends exist until `Eager` goes. - Integrated state is O(cells) beside the store. - `ExactSum` costs 568 bytes per summed group. -- Every client receives every notice. - Each command snapshots the circuit's state for rollback. **Order** (each step is a ticket in the implementation epic, behind the differential @@ -405,16 +413,7 @@ Step 8's required tests: ## Open questions -1. **Should notices be filtered per client?** Clause 9 sends every client every live - shape's moved keys. That keeps the host free of per-client state, but traffic is - O(clients × all moved keys) per committed command. Filtering by topic would send each connection - only its own shape's key lists. It needs one subscription per connection, which pins - its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? - *Recommendation:* broadcast with the size cap. Add topics only if measured traffic - with many clients on different shapes justifies them. - -2. **How should cycles be handled natively?** Clause 6's component fixed point is +1. **How should cycles be handled natively?** Clause 6's component fixed point is interim. The options: - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it has not converged, which keeps today's answers; @@ -426,12 +425,12 @@ Step 8's required tests: changes. Should they be maintained incrementally too? *Recommendation:* (a), and recompute the components only for the formulas an edit touches. -3. **What replaces `Eager`?** It still serves as the differential oracle and as the +2. **What replaces `Eager`?** It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. *Recommendation:* a small from-scratch reference implementation, kept in the test crate, as the oracle, and a pushdown backend behind the same trait for query-only sources. -4. **Which columns get a host-maintained width?** Columns whose cells are text? What +3. **Which columns get a host-maintained width?** Columns whose cells are text? What about a column holding both text and numbers? *Recommendation:* text cells get the maintained width, and the view layer takes the larger of that and its own numeric width. From 81f1b5e13a4e6a3e1e4e90279cf0eaa37125b323 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:28:52 -0700 Subject: [PATCH 099/106] docs(adr): ADR 009 settles incremental cycles, a temporary Eager, and per-column width hints (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 62 ++++++++++------------- 1 file changed, 26 insertions(+), 36 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 834b849f..25573031 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -54,10 +54,12 @@ depends on `zset` and improvise-core. improvise-core never depends on `zset`. The circuit decides *which* formula cells to re-evaluate. The one existing evaluator computes each value, so formulas are never compiled into operators. -**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. -It serves as the oracle of a differential property test, which applies one random edit -sequence to both backends and compares every drawn cell, total and occupancy count. It -is removed once its remaining uses are replaced (open question 2). +**`Eager` is temporary.** Today's code moves behind the trait as `Eager`, unchanged. It +exists only to show discrepancies between the new evaluator and the old one: a +differential property test applies one random edit sequence to both backends and +compares every drawn cell, total and occupancy count. Eventually it is deleted, and +nothing depends on it surviving. Long-term tests compare against exact references, such +as `ExactSum` against rational sums, not against `Eager`. ### 3. The cell delta @@ -126,12 +128,13 @@ before enumerating. This replaces `Expansion::members`. ### 6. Circular references DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the -answer must be `Error("circular")`. The first version finds the formula graph's strongly -connected components, recomputed when the formula set changes; how to keep that -incremental belongs with open question 1. Acyclic formulas are evaluated once each, in topological order. A -component is evaluated with today's bounded fixed point, restricted to that component. -That reproduces the sixteen-pass bound for the differential test. It is to be replaced -by a DBSP-native treatment (open question 1). +answer must be `Error("circular")`. The formula graph's strongly connected components +are maintained incrementally as formulas change. Acyclic formulas are evaluated once +each, in topological order. A cyclic component is maintained incrementally too: a change +re-iterates only the cells it reaches. A component that does not converge within the +existing bound of sixteen passes (`MAX_EVAL_DEPTH`) propagates `Error("circular")`, as +today. An optimisation that speeds convergence is allowed only where it is exact: it +gives the same result as the plain iteration, and a test compares the two. ### 7. Operator state follows the request's shape key @@ -343,12 +346,12 @@ circuit, the client's adapter is a sink for results and notices and a source of Toward the client it is the mirror image. The client's end keeps the watermark. In process, an edge is a function call. -### 10. Text column widths +### 10. Column width hints -This ADR owns the rule for text columns. The host maintains each text column's maximum -display width as a histogram from width to count, per (shape key, column), fed by the -delta of the column's text cells. Hidden rows come out through clause 7's overlay, where -that applies. Numeric widths stay with ADR 008's view-layer rule. +Every column incrementally maintains a width hint, which optimises grid sizing and +reduces jitter. The hint is an integrated aggregate: a histogram from display width to +count, so a retraction is O(log d). Which end of the connection maintains it is open +question 1. ### 11. Rollback and memory @@ -413,24 +416,11 @@ Step 8's required tests: ## Open questions -1. **How should cycles be handled natively?** Clause 6's component fixed point is - interim. The options: - - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it - has not converged, which keeps today's answers; - - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles - such as `A = IF(x, B, 1)`; - - (c) a monotone reformulation, where one exists. - - Separately, the strongly connected components are recomputed whenever the formula set - changes. Should they be maintained incrementally too? *Recommendation:* (a), and - recompute the components only for the formulas an edit touches. - -2. **What replaces `Eager`?** It still serves as the differential oracle and as the - backend for live sources that can only answer aggregate queries. *Recommendation:* a - small from-scratch reference implementation, kept in the test crate, as the oracle, - and a pushdown backend behind the same trait for query-only sources. - -3. **Which columns get a host-maintained width?** Columns whose cells are text? What - about a column holding both text and numbers? *Recommendation:* text cells get the - maintained width, and the view layer takes the larger of that and its own numeric - width. +1. **Which end of the connection maintains a column's width hint?** The client end knows + which values are potentially in its viewport, so it can maintain the hint over the + values it holds, fed by the deltas it already receives. The host end can maintain it + over a whole column, through the graph, including values the client has never held. + It would then need the request's hidden rows (clause 7's overlay) and, for numbers, the + number format. *Recommendation:* the client end, over the values it holds. The host + computes a whole-column hint only where a client asks for one, as a filter operator + on its connection (clause 9). From b8a0b6a846f14e079b66040e7fb33991bd771d19 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:31:45 -0700 Subject: [PATCH 100/106] docs(adr): ADR 009 keeps the window out of a pushed filter and names query pushdown sources (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 25573031..aab7b93d 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -51,6 +51,10 @@ formula result at a key, and the raw fold of the cells that match a partial key. incremental backend implements it in a seam crate, `improvise-incremental`, which depends on `zset` and improvise-core. improvise-core never depends on `zset`. +A live source (ADR 003) that reports its changes feeds cell deltas like `DataStore`. A +source that answers aggregate queries but reports no changes implements the derived-value +operator directly, by query pushdown, with no change feed. + The circuit decides *which* formula cells to re-evaluate. The one existing evaluator computes each value, so formulas are never compiled into operators. @@ -316,8 +320,10 @@ notice carries: **A client receives only what it needs.** Filtering is part of the DBSP graph, not a separate mechanism. A client either pulls just the state it needs, or pushes a filter to -the host's side of its connection: a filter operator on its shape key and window, written -in the query language, through which its notices pass. A pushed filter belongs to the open +the host's side of its connection: a filter operator on its shape's query, written in the +query language, through which its notices pass. The filter never names the window, which +stays the client's: the client intersects the moved keys of a notice or correction with +its own window. A pushed filter belongs to the open connection, like that connection's timeout, and is released with it. On a sequence of requests the client sends it with each request, so the host keeps no per-client state beyond an open connection. While a connection's filter is open, it counts as a request From 14430b5628ba9f8072d40db0a6db7d6de79763c6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:32:37 -0700 Subject: [PATCH 101/106] docs(adr): ADR 009 requires self-contained operators and leaves width-hint placement to scheduling (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index aab7b93d..76611600 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -38,6 +38,11 @@ wasm32-clean and serde-free). It provides: - operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it. +**Operators are self-contained.** An operator's state and behaviour depend only on its +inputs and its own state, never on a shared in-process global or on another operator's +internals. That keeps the graph distributable: in principle the model could run across a +cluster of machines. Nothing more is designed for that here. + There are no nested streams, no recursion operator and no timestamps beyond one clock. **The formula language is the query language.** What a request asks of the model beyond its axes and window (filters, aggregation, derived values) is written in formula text @@ -356,8 +361,8 @@ process, an edge is a function call. Every column incrementally maintains a width hint, which optimises grid sizing and reduces jitter. The hint is an integrated aggregate: a histogram from display width to -count, so a retraction is O(log d). Which end of the connection maintains it is open -question 1. +count, so a retraction is O(log d). Where in the graph the hint is computed is a +scheduling choice, made as an optimisation later. ### 11. Rollback and memory @@ -422,11 +427,4 @@ Step 8's required tests: ## Open questions -1. **Which end of the connection maintains a column's width hint?** The client end knows - which values are potentially in its viewport, so it can maintain the hint over the - values it holds, fed by the deltas it already receives. The host end can maintain it - over a whole column, through the graph, including values the client has never held. - It would then need the request's hidden rows (clause 7's overlay) and, for numbers, the - number format. *Recommendation:* the client end, over the values it holds. The host - computes a whole-column hint only where a client asks for one, as a filter operator - on its connection (clause 9). +None. From b26c92faf7e128b0a8197b259713af57fa395de6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:35:01 -0700 Subject: [PATCH 102/106] docs(adr): ADR 009 states the model and view stores as conceptual boxes, not a network boundary (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 +++++++++++++---------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 76611600..e08ba4b7 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -183,7 +183,8 @@ temporary overlay: - text counts the distinct keys left; - `Sum` subtracts exactly through the accumulator. -That is O(H log d) per read for every aggregate, with no per-client state. +That is O(H log d) per read for every aggregate, with no view-store state in the model +store. Aggregate-over-expression totals index their entries by row tuple for the same subtraction. @@ -206,6 +207,11 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery +The **model store** (the document model and the DBSP graph) and the **view store** (the +cursor, the window and unsaved view definitions) are conceptual boxes, not a network +boundary. Where a piece of state physically lives (a client, the host or a worker) is +an implementation detail. The model store's decisions depend only on the model. + **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared memory. An operator that sits at a wire boundary is just one implementation of an edge, @@ -264,12 +270,12 @@ an **edge adapter**. So every edge has the same semantics: position is a **watermark**: the whole stamp `(host instance, document_generation, publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current - stamp. The client holds one per shape key it reads. The host keeps no watermarks: a - client sends its own with each request. + stamp. The client holds one per shape key it reads. A watermark is view-store state: + the model store keeps none, and a client sends its own with each request. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and - flags, not values, so nothing per client is kept; + flags, not values, so the model store keeps no view-store state; - where the covered range also holds axis changes, relabels or flags, which positions and labels depend on, the correction carries them too, in order, as a merged notice does, so the client re-derives positions incrementally; @@ -292,7 +298,7 @@ relies on, is a correction trigger below, followed from a displayed cell's key b the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left -and dimensions. The window is never state, and a request costs O(visible). +and dimensions. The window is view-store state, and a request costs O(visible). - **Positions** are bigints where they exceed 2⁵³, sent as JSON strings. - **Output** is keyed by tuple identity, with the position attached when sent. - **An item insert, removal or reorder** is sent as an axis change, and each client @@ -326,12 +332,11 @@ notice carries: **A client receives only what it needs.** Filtering is part of the DBSP graph, not a separate mechanism. A client either pulls just the state it needs, or pushes a filter to the host's side of its connection: a filter operator on its shape's query, written in the -query language, through which its notices pass. The filter never names the window, which -stays the client's: the client intersects the moved keys of a notice or correction with -its own window. A pushed filter belongs to the open -connection, like that connection's timeout, and is released with it. On a sequence of -requests the client sends it with each request, so the host keeps no per-client state -beyond an open connection. While a connection's filter is open, it counts as a request +query language, through which its notices pass. Whether the filter also carries the +client's window is an optimisation, since the model store's decisions depend only on the +model. A pushed filter belongs to the open connection, like that connection's timeout, +and is released with it. On a sequence of requests the client sends it with each +request. While a connection's filter is open, it counts as a request naming its shape, so its shape is not evicted. A notice whose key lists exceed a fixed size carries "correct" instead. That is a @@ -346,8 +351,8 @@ keys, the covered range of publications, and axis changes kept in order. Because wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, -`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state, only the shared notice log. `GridExtent` is derived by the client, and in records +`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no +view-store state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is fetched. `Projection` also published the ADR 001 basis a write claims. Whether `project` returns it instead, and what replaces the layout tier, are ADR 008's questions. @@ -393,7 +398,7 @@ scheduling choice, made as an optimisation later. - The store scans and the per-view model clones go. - ADR 008 gets occupancy, totals and widths from one evaluator. - Live sources that report changes plug in. -- The host holds no per-client state. +- The model store holds no view-store state. **Bad:** - Two backends exist until `Eager` goes. From d61f5d74e6e8a7e23c3586d92d7fbe72e61748c0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:36:04 -0700 Subject: [PATCH 103/106] docs(adr): ADR 009 permits disposable server-side twins of client state at an edge (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e08ba4b7..1ab6966a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -210,7 +210,11 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r The **model store** (the document model and the DBSP graph) and the **view store** (the cursor, the window and unsaved view definitions) are conceptual boxes, not a network boundary. Where a piece of state physically lives (a client, the host or a worker) is -an implementation detail. The model store's decisions depend only on the model. +an implementation detail. The model store's decisions depend only on the model. A +server-side twin of a client's state, such as its filter, watermark or window, may be +held at the serving end of an edge where that makes the graph genuinely simpler. It is +then part of the edge, not of the model store, and it is disposable: losing it forces a +correction from the client's watermark, never wrong data. **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared @@ -258,8 +262,8 @@ an **edge adapter**. So every edge has the same semantics: themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). - **Failure.** Timeouts are local monotonic durations, never compared across machines. - On a long-lived connection, both ends keep alive, and a host-side timeout belongs to - that open connection and is released with it. On a sequence of requests, only the + On a long-lived connection, both ends keep alive, and a host-side timeout is tied to + that open connection. On a sequence of requests, only the client detects failure: a request that fails or times out. In either case the client then asks for a correction from its watermark. @@ -271,7 +275,8 @@ an **edge adapter**. So every edge has the same semantics: publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current stamp. The client holds one per shape key it reads. A watermark is view-store state: - the model store keeps none, and a client sends its own with each request. + the model store keeps none, so a client sends its own with each request unless a twin + at the serving end holds it. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and @@ -334,9 +339,8 @@ separate mechanism. A client either pulls just the state it needs, or pushes a f the host's side of its connection: a filter operator on its shape's query, written in the query language, through which its notices pass. Whether the filter also carries the client's window is an optimisation, since the model store's decisions depend only on the -model. A pushed filter belongs to the open connection, like that connection's timeout, -and is released with it. On a sequence of requests the client sends it with each -request. While a connection's filter is open, it counts as a request +model. A pushed filter is held by the open connection, or by a twin. Otherwise the +client sends it with each request. While a connection's filter is open, it counts as a request naming its shape, so its shape is not evicted. A notice whose key lists exceed a fixed size carries "correct" instead. That is a From a9ee6f00d0fe7fb1506a40b62a4ac0bbe8d83181 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:38:43 -0700 Subject: [PATCH 104/106] docs(adr): ADR 009 states twin-loss recovery and the store as an integrate trace (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1ab6966a..36031e02 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -123,7 +123,9 @@ The affected pairs are evaluated in the formula graph's topological order. **An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: 1. A per-record stage retracts and reinserts each touched record. It reads the record's - previous cells as the store minus the delta, so it keeps no copy of the values. + previous cells as the store minus the delta, so it keeps no copy of the values. The + store is the `integrate` operator's trace over the cell delta, read as an input of + the stage, so the stage stays self-contained. 2. A map produces `(context, order key, expr)`. The order key is (context rank, record key in `CellKey` order), which is eager's concatenation order. 3. The group aggregate, with an error channel that reads the first `Err` in order. @@ -213,8 +215,9 @@ boundary. Where a piece of state physically lives (a client, the host or a worke an implementation detail. The model store's decisions depend only on the model. A server-side twin of a client's state, such as its filter, watermark or window, may be held at the serving end of an edge where that makes the graph genuinely simpler. It is -then part of the edge, not of the model store, and it is disposable: losing it forces a -correction from the client's watermark, never wrong data. +then part of the edge, not of the model store, and it is disposable: losing it never +gives wrong data. A client that keeps its own watermark recovers with a correction from +it. A client that relied on the twin for its watermark recovers with a full refetch. **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared From bb578d71e3e50f74c883e9861fce0f14db6e1d12 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:44:30 -0700 Subject: [PATCH 105/106] docs(adr): ADR 009 values compare by float rules, NaN canonical NaN payloads are not preserved: every NaN is one canonical value in the delta and crosses as ADR 005's `nan` token, so ADR 009 no longer changes ADR 005 clause 11. Zeros follow IEEE: -0.0 equals +0.0, and a zero sum takes the sign IEEE gives the same addends. The round-trip property test asserts a NaN comes back as the canonical NaN. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 29 +++++++++++++---------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 36031e02..a791116a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -77,8 +77,11 @@ as `ExactSum` against rational sums, not against `Eager`. - `−(k, v₀) + (k, v₁)` for an overwrite; - `−(k, v₀)` for a removal. -Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite -consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the +Values compare by normal float rules, except that every NaN is one canonical value: +`-0.0` equals `+0.0`, and an equal-value overwrite consolidates to nothing. NaN payloads +are not preserved, so nothing can be smuggled through them. Today's record tier compares +bits, so overwriting `0.0` with `-0.0` stops counting as a change. Multisets order values +by `f64::total_cmp` after mapping every NaN to the canonical one and `-0.0` to `+0.0`. `CellKey` gains `Ord`, which is its canonical order: the natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The circuit steps once per model effect. @@ -203,9 +206,10 @@ backends: sums on 20,000 cases; - 568 bytes per group, and about 1.3× the cost of an f64 add per update. -It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change -in their last bits, because they become correctly rounded, and a sum of `-0.0` reads -`+0.0`. +The sign of a zero sum follows IEEE rules for the same addends: it is `-0.0` only if +every addend is `-0.0`, and an exact cancellation gives `+0.0`. It lives in ADR 005's +`numtower` crate. **User-visible change:** displayed sums change in their last bits, +because they become correctly rounded. ### 9. Delivery @@ -229,11 +233,10 @@ an **edge adapter**. So every edge has the same semantics: tuple of ids, its position or an ADR 002 surrogate. - **The wire form is lossless.** Nothing is lost in transport, numeric precision above all: - - a value never crosses as a JSON or JS double. A `numtower` `Num` crosses as its - canonical text (ADR 005), and an inexact finite value keeps its `#i` spelling, which - is already bit-exact, `-0.0` included. A non-finite value crosses as its bits, which - keeps NaN payloads. **That changes accepted ADR 005 clause 11**, whose wire spells - non-finite values as `inf`, `-inf` and `nan`, and it needs approval with this ADR; + - a value never crosses as a JSON or JS double. It crosses in ADR 005's canonical text: + an inexact finite value as `#i` with its shortest round-trip spelling, which keeps + the sign of zero, infinities as `inf` and `-inf`, and every NaN as the one token + `nan`; - positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as strings; - text crosses byte-exact, with no normalisation; @@ -242,8 +245,8 @@ an **edge adapter**. So every edge has the same semantics: The stamp rule detects lost messages, but nothing would detect a lossy encoding. So every element type has a round-trip property test, where encoding then decoding is - the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and - `i128`-wide fractions. + the identity for every value, including `-0.0`, infinities, 28-digit decimals and + `i128`-wide fractions. A NaN round-trips as the canonical NaN. - **The clock is causal, never a wall clock.** Ordering, staleness and loss detection use only the stamp sequence (below), a logical clock the host advances once per committed command. Each process runs its own synchronous sub-circuit, and across an @@ -355,7 +358,7 @@ moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the -wire is lossless, outputs keep the circuit's bit identity end to end. +wire is lossless, outputs keep the circuit's value identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no From 40792ee03e2af92f7f7af73238c66adbea982b0d Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:45:44 -0700 Subject: [PATCH 106/106] docs(adr): ADR 009 requires a zero-sign property test for sums Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index a791116a..68135713 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -439,6 +439,8 @@ Step 8's required tests: frame stay published; - a dropped connection or a failed request triggers a correction, and a later request with the same session resumes from the watermark. +- a zero sum has the sign IEEE gives the same addends: `-0.0` only if every addend is + `-0.0`, and `+0.0` for an exact cancellation (a property test against IEEE). ## Open questions