From 322875b30affcf751b8a43148d6d3694a026e8d3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:14:37 -0700 Subject: [PATCH 01/67] docs(adr): ADR 009 incremental evaluation (proposed) (improvise-felo.4) An in-house DBSP core as a problem-agnostic crate, plugged in behind a trait in improvise-core, with the eager evaluator kept as the oracle. Adds examples/retraction_drift.rs, which measures f64 retraction drift against the eager fold and the exact decimal sum. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 612 ++++++++++++++++++ .../examples/retraction_drift.rs | 191 ++++++ scripts/build-doc-site.mjs | 1 + 3 files changed, 804 insertions(+) create mode 100644 context/adr/009-incremental-evaluation.md create mode 100644 crates/improvise-core/examples/retraction_drift.rs diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md new file mode 100644 index 00000000..b61fe0ad --- /dev/null +++ b/context/adr/009-incremental-evaluation.md @@ -0,0 +1,612 @@ +# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait + +**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end +are theirs to answer on the PR. · +**Ticket:** improvise-felo.4 · +**Decision from:** the user, 2026-09-25 (below) · +**Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · +**Relates to:** ADR 005 (exact decimals, improvise-l2b8, no file yet), ADR 008 (the +virtualized grid, PR #308, which is revised to assume this ADR) · +**Governed by:** `context/design-principles.md` §3 and §5 + +**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl and have +no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +no open PR claimed it when this was written. + +**Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are +dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means +revision 2 of `context/adr/008-virtualized-grid.md` on PR #308. + +## The decision this ADR writes down + +The user pointed at a teaching implementation of the DBSP paper, +[brurucy/dbsp-from-scratch](https://github.com/brurucy/dbsp-from-scratch/blob/master/dbsp_paper_walkthrough_implementation.ipynb), +as a way to evaluate formulas so that data can stream. The notebook builds, in order: + +- abelian groups; +- streams, with `Delay` (z⁻¹), `Differentiate` and `Integrate`; +- Z-sets, which are maps from element to an integer weight; +- lifted and doubly-lifted joins, and a delta join that keeps less state; +- `distinct` through the `H` function over an integrated state; +- recursion by nested streams, shown on graph reachability. + +The user then decided: + +> "I think doing our own implementation of the core of dbsp as a crate we can plugin is +> probably dieal [sic] here. Write the ADR and Update ADR 008 to assume the decisions in +> that one" +> +> — the user, 2026-09-25, recorded on improvise-felo.4 + +So the choice between Feldera's crate and our own is already made (see Alternatives for +what was checked). This ADR says what "the core of dbsp" is for improvise, where it +plugs in, and what it may not change. + +## Context + +### Evaluation today is a from-scratch fold, rerun after every batch + +- **Every batch recomputes every formula cell.** `App::apply_effects` + (`crates/improvise-command/src/ui/app.rs:989`) ends in `rebuild_layout` (`:880-891`). + That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). + It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), + and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` + (`:699`, `:1001-1022`). A cursor move and a cell write pay the same. +- **Raw aggregation scans the matching cells on every read.** `evaluate_unfolded` + (`types.rs:837-845`) folds `DataStore::matching_values` through + `aggregate_cell_values` (`:30-42`). `fold_stored` (`:918-930`) does the same for a + collapsed group, through `spanned_values` (`crates/improvise-core/src/model/cell.rs:445`). +- **`formula_cache` holds one aggregation shape at a time.** It is keyed by cell and + computed for one `none_cats` list. A session on another view with a different None + axis gets a *cloned model* recomputed for its own list (`rederived_for`, + `crates/improvise-command/src/ui/project.rs:195-210`). +- **A formula total with an aggregate inside walks a cross product.** `contexts_of` + (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every + combination of the spanned categories' items. ADR 008 clause 5 names this as the one + O(grid) path left in totals. +- **Two of the record tiers are observed by scanning the store.** `record_fingerprints` + (`app.rs:254-264`) hashes every cell before and after a cell-writing batch, and + `document_content` (`app.rs:165-178`) does the same for the dirty flag. Both are + O(cells) per batch, whatever the batch wrote. + +### What an incremental evaluator must not change + +These rules come from the repo map and `context/code/core.md`. Each one is a place an +incremental evaluator could quietly disagree with the eager one. + +- **Gotcha 7, text unanimity.** `aggregate_cell_values` checks the all-numeric branch + first. Otherwise one distinct value is that value, and anything else is + `Error("mixed")`. `Count` of a lone number is `1`, not the number. +- **Gotcha 9 and core.md §3, circularity.** A cell still moving after 16 passes is + written as `Error("circular")` (improvise-7iww). **A chain that needs all sixteen + passes is also reported circular**, because the fixed point can only confirm + convergence with a pass that changes nothing. The direct evaluator agrees because its + recursion depth is the same 16 (`types.rs:705-714`). +- **Gotcha 17, one epsilon.** `FLOAT_EQ_EPSILON = 1e-10` (`types.rs:14`) is shared by + `=`, `!=` and the div/0 guard, in both evaluators. +- **Gotcha 18, ordering is correctness.** A `HashSet` index is for lookup, never for + iteration. `SUM` and `AVG` drifted in the last bits when the store was walked in hash + order (improvise-shyt), because f64 addition is not associative. +- **Gotcha 40, the tiers are observed.** A write that lands the value already there + bumps nothing. The record set is collected into a *set*, so a rewritten record counts + once. + +## Decision + +**A new, problem-agnostic workspace crate implements the DBSP core. improvise-core +defines a trait for the derived state it reads. Today's evaluator is one implementation +of that trait, and a circuit built on the new crate is the other. Both stay in the tree, +and a differential test holds them equal.** Eleven clauses. + +### 1. The crate: `zset` (name is question 1) + +Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond +`std`, so it is wasm32-clean by construction. + +**What it contains:** + +- **`ZSet`**: a map from `K: Ord + Clone` to a non-zero `i64` weight, with `add`, + `neg` and `zero`. It forms an abelian group, and its laws are property-tested as the + notebook's first cell does: commutativity, associativity, identity and inverse. + Weights use checked arithmetic, so an overflow is a panic, not a wrap. A weight that + reaches zero is removed, so equal Z-sets are equal as maps. +- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one input + delta per source and returns one output delta per sink. There is one clock, one + thread, and no scheduler. +- **Stream operators:** `delay` (z⁻¹), `integrate` (I) and `differentiate` (D), with + I∘D = D∘I = id tested as a property. +- **Linear operators, which are their own incremental form:** `map`, `filter` and + `flat_map`. +- **`index_by(key)`.** It turns a `ZSet` into an indexed Z-set `K → ZSet`, which + is the input shape for grouping and joins. +- **`join_indexed`.** The bilinear delta join: Δ(a ⋈ b) = Δa ⋈ Δb + I(a)₋₁ ⋈ Δb + + Δa ⋈ I(b)₋₁, over integrated indexes. It is needed for one purpose, clause 5's + cross-stem references. +- **`distinct`.** It is `H` over the integrated input, as in the notebook. The output + delta is non-empty exactly when an element's total weight crosses zero. +- **`aggregate(group, Aggregator)`.** Its output delta retracts the group's old value + and inserts its new one. The trait has two kinds of implementation: + - **linear:** `Count`, and `Sum` for any `G: AbelianGroup`. The state is one value + per group, and each delta element costs O(1). + - **integrated:** the aggregator keeps the group's integrated multiset, + `BTreeMap`, and reads its answer off it. `Min` and `Max` are the first and + last keys. Unanimity (clause 4) is "exactly one distinct key". Each delta element + costs O(log d), where d is the number of distinct values in the group. + +**What it leaves out, deliberately:** + +- **Nested streams and least-fixed-point recursion** (the notebook's + `LiftedLiftedJoin`, `DeltaLiftedDeltaLiftedDistinct` and `IncrementalGraphReachability`). + improvise's circular semantics are not a least fixed point (clause 6). +- **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies + batches in one total order (ADR 001, "There is already a total order"). +- **Persistence, spilling to disk, and threads.** State lives in memory, and a browser + worker has one thread. +- **A query language or planner.** Circuits are wired by hand in the seam crate + (clause 2). + +**The weight type is `i64`.** Integer weights make `ZSet` an exact group, so +consolidation and retraction never drift. The drift problem is in the *values* being +summed, not in the weights (clause 8). + +### 2. The plug: a trait in improvise-core, and a seam crate that implements it + +**improvise-core gains a trait for the derived state its evaluator reads.** The working +name is `Derived`. It replaces the two places where eager evaluation folds the store: + +- the formula result at a key, which is `formula_cache` today (`types.rs:370`); +- the raw fold of the cells matching a partial key under an aggregation, which is + `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). + +It has one mutation, `apply(&mut self, delta: &CellDelta)`, plus a rebuild for the +changes that are not cell deltas (clause 7). + +**The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code +and read through the trait. The circuit decides *which* formula cells to re-evaluate +and supplies their inputs, and the existing per-cell evaluator computes the value. +Formulas are never compiled into operators. A compiled `IF`/`POW`/`WHERE` would be a +second implementation of evaluation, which design-principles §5 calls the defect +itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental totals. + +**Two implementations:** + +- **`Eager`** is today's code moved behind the trait, with no behaviour change: `apply` + records the delta, and the post-batch refresh is `recompute_formulas`. +- **`Incremental`** lives in a seam crate, working name `improvise-incremental`, which + depends on `zset` and `improvise-core`. This is the `gp-tree`/`improvise-gp` shape: + improvise-core never depends on `zset`, and it can be built and tested eager-only + (question 2). + +**Why keep both.** `Eager` is the oracle for the differential test. It is also the +backend a query-only source keeps (clause 10). The differential test is a property test +in the seam crate. It generates documents with groups, collapse, hidden items, formulas +(including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text +cells and non-finite numbers, applies the same random edit sequence to both backends, +and compares every drawn cell, every column total and every occupancy count. + +### 3. The input: a cell write is a retraction plus an insertion + +**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue)>` with weights +±1.** `DataStore::set` and `DataStore::remove` (`cell.rs:286`, `:325`) are the only +places a cell changes, so the delta is emitted there: + +- a new cell is `+(k, v)`; +- an overwrite of `v₀` with `v₁` is `−(k, v₀) + (k, v₁)`; +- a removal is `−(k, v₀)`. + +An overwrite with an equal value consolidates to the empty Z-set. The circuit steps once +per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy +inside the effect loop (`app.rs:1073-1084`). + +**This lines up with ADR 001's record tier with no extra rule.** The records a batch +moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys +in the batch's consolidated delta. So: + +- a write of the value already there moves nothing, which is gotcha 40's rule; +- a record gained, lost or rewritten is named once, because a Z-set is already a set of + keys. + +`bump_changed_records` can then read the delta instead of diffing two +`record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta +comes from the mutation sites, not from an `EffectKind`. The same holds for the content +half of the dirty flag, where a non-empty delta means dirty. `layout_version` is +unchanged, because it observes `view_shape` and not cells. + +**`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` +(`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only +`PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by +`f64::total_cmp`. Clause 4 says where that differs from `==`. + +### 4. Aggregates: linear where they can be, an integrated multiset where they cannot + +**The group** is the cell key with the view's `none_cats` removed. That is the partial +key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct +`none_cats` list, shared by every view that has that list. That replaces +`rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group +comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does +(`types.rs:717-722`). + +**What each group keeps:** + +| answer | state per group | cost per delta element | exact on f64? | +|---|---|---|---| +| `Count` | count of values, count of numbers | O(1) | yes (integers) | +| `Sum` | running sum | O(1) | **no** (clause 8) | +| `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | +| `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | +| unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | + +Reading a group reproduces `aggregate_cell_values`'s branch order: + +- if the count of numbers equals the count of values, the numeric answer; +- else if the value multiset has exactly one distinct key, that value; +- else `Error("mixed")`. + +The empty group is absent. The formula-level `eval_agg` (`types.rs:55-87`) reads the +same state with its own emptiness rule, `0` for `Sum` and `Count`, at read time. + +**Memory.** Summed over all groups, the multisets hold at most one entry per stored +cell. So the state for one `none_cats` list is O(cells), on the order of the store it +summarizes. **Estimate, not measured:** a `BTreeMap` entry of about 40–60 bytes plus the +key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the measures +whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A +`Sum`-only document keeps one number per group. + +**`total_cmp` is not `==`, and the differential test must say so.** `NaN != NaN`, so +today two NaNs in one group read as `mixed`, and `f64::min` skips a NaN. A multiset keyed +by `total_cmp` would call two NaNs one value and would order NaN at an end. The test +generates non-finite values (core.md §16), and the aggregator matches eager +behaviour explicitly rather than inheriting `total_cmp`'s. + +### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it + +**Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key +with the target category and each `none_cats` entry removed. That is +`distinct(map(project))` over the cell delta. A stem enters when its first cell arrives +and leaves when its last cell goes. + +**A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula +`T` at stem `s` are: + +- for each `Ref(R)`, the group `s.with(cat(R), R)`. `cat(R)` is resolved once per + formula set through `find_item_category`, including its `_Measure` fallback for + formula targets (`types.rs:725-740`). Whether `R` is derived (`formula_for`) is also + decided per formula set, so a ref to another formula becomes an edge between formula + nodes, and a ref to raw data becomes an edge from an aggregate group. +- for an `Agg` with no filter, the group at `s` itself, because `eval_agg` pools over + `context.without(target_category)`. +- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at + `s.with(Cat, Item)`. That is a different stem, so a delta at `Cat = Item` reaches + every stem that differs only in `Cat`. **This is the one join**: stems indexed by + "stem minus `Cat`", joined with the aggregate's output delta. + +The affected `(stem, formula)` pairs are consolidated, ordered topologically over the +**formula graph**, and handed to the existing evaluator. Its cached path already reads +refs through `formula_cache` and falls back to raw aggregation (`types.rs:1066-1086`). +With the trait, both reads go to the maintained state. + +**A formula's own `WHERE`** is decided per cell, as today. A cell whose coordinate does +not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own +group and nothing else. + +**An aggregate over an expression replaces `Expansion::members`.** `SUM(expr)` pools +`expr` evaluated at each member record (`agg_values`, `types.rs:89-139`). Incrementally, +that is `map(record ↦ expr at record)`, then `index_by(base key)`, then `Sum`. The +member records never have to be listed. **The boundary:** the map is sound only when +`expr` reads the record's own cells. An inner expression that reads another record, or +holds a nested aggregate, stays on the eager path for that formula, and the +differential test pins the boundary. Until `Sum` is exact, this path carries clause 8's +caveat. ADR 008 clause 5 adopts the same boundary. + +### 6. Circular references: DBSP's recursion is not our semantics + +DBSP handles recursion by computing a **least fixed point**: nested streams iterate a +monotone operator, and `distinct` keeps the lattice finite, so reachability converges. +improvise has no such guarantee: + +- `A = B + 1` with `B = A + 1` has no fixed point at all; +- `X = X + 1` over a stored `X = 10` climbs forever; +- the required answer is `Error("circular")`, not a limit or a non-terminating step. + +So the crate has no recursion operator, and circularity is decided *outside* it: + +- **The formula graph** (target → the targets its refs name) is rebuilt when the formula + set changes, and its strongly connected components are found (Tarjan). Formulas + outside every cycle are evaluated in topological order, each exactly once per step. +- **A formula in a cycle, a self-loop included, has its affected cells evaluated by + today's bounded fixed point, restricted to that component.** A formula-level cycle is + not always circular at every cell. `A = IF(x, B, 1)` with `B = A` converges wherever + `x` is false. The per-cell fixed point is the only thing that gets those cells right, + and reusing it keeps `Error("circular")` exactly as it is. +- **The depth bound is part of the semantics today.** An acyclic chain that needs all + sixteen passes is reported circular by both eager evaluators (core.md §3). A + topological evaluator would compute it correctly, and so would disagree with eager. + Question 4 asks whether to reproduce the bound. The recommendation is to reproduce + it, by carrying each formula's topological depth, so the differential test compares + like with like. Changing the rule would be its own decision, for both backends at + once. + +### 7. What is not a cell delta rebuilds, once + +Some changes alter the circuit's *shape*, not its input: + +- the formula set: add, remove or replace; +- `measure_agg`; +- category and item structure: add or remove a category, remove an item, and group + membership; +- a view's axes, hidden items and collapsed groups, which change `view_shape`; +- a document swap, `document_generation`. + +Each rebuilds only the affected operators, from the integrated base Z-set. That costs +O(cells), once per change, and is never repeated per keypress. **The trigger is the same +observation the tiers use**: a `view_shape` diff (`app.rs:40-72`) rebuilds that view's +operators, the same diff that bumps `layout_version`. So there is one signal and not two. + +**A page selection is not a rebuild.** The group key includes the page categories, so +`[` and `]` read another slice of state that already exists. The held-key path stays +O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the +state is still bounded by the stored cells, however many pages there are (question 7). + +### 8. Floating point: retraction drifts, and exactness is ADR 005's + +f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum +the eager fold computes. **Measured**, not asserted: +`cargo run --release -p improvise-core --example retraction_drift` +(`crates/improvise-core/examples/retraction_drift.rs`, added by this PR). It was run on +an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N +retract-and-insert cycles. It compares the sum against the eager fold, +`evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count +of 10⁻⁵ units rounded to f64 once. Values are signed two-place decimals spread over 3 or +9 decades. + +| store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | +|---|---|---|---|---| +| 1,000 cells | 3 decades | 6.4e-11 – 1.3e-10 (35–140 ulp) | in 1 of 2 seeds | ≤ 7.3e-12 | +| 18,000 cells | 3 decades | 1.7e-9 – 2.3e-9 (57–80 ulp) | yes | 1.5e-10 – 2.6e-10 | +| 18,000 cells | 9 decades | 2.3e-5 – 3.2e-4 (3–42 ulp) | yes | 2.4e-4 – 2.5e-4 | + +A fourth case holds 1e17, then adds 1,000 cells of 0.01, then retracts the 1e17. The +naive running sum reads **0.00**, and the eager fold and the exact sum both read +**10.00** (at `,.2`). The small values were absorbed and never came back. + +A Neumaier-compensated running sum recovers that case exactly. In the random cases it +stays about as close to the exact sum as eager does, **but it is never bit-equal to +eager**, because eager folds in store order and is itself inexact. + +Four conclusions follow: + +- **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by + any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, + unanimity and occupancy bit for bit today, because they are order-free. It can compare + `Sum` and `Avg` only with a tolerance. +- **The gap reaches the epsilon.** In most 18k-cell cases the incremental sum and the + eager sum differ by more than `FLOAT_EQ_EPSILON`. So `IF(SUM(..) = x, ..)` can answer + differently on the two paths. That is a behaviour difference, not a display one. +- **This is already a bug in eager, which the measurement found in passing.** The eager + 18k-cell sum is 1.5e-10 to 5.8e-10 from the exact decimal sum, so `=` against a + decimal a user typed can already be wrong today. That supports ADR 005 on its own + merits, and it is recorded here for improvise-l2b8. +- **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true + `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's + whole class), and the differential test is bit-exact everywhere. + +**So ADR 005 is a prerequisite for the maintained `Sum`/`Avg` on the displayed path, but +not for the rest of this ADR** (question 3). Occupancy, `Count`, `Min`, `Max`, +unanimity, stems and invalidation can all land before it. Until ADR 005 lands, a `Sum` +or `Avg` the grid displays, including a column total that holds one, is read from the +eager fold. The maintained sum runs only inside the differential test, under a stated +tolerance. If the user wants maintained sums before ADR 005, the fallback is a +compensated sum that re-folds a group from its values when cancellation is detected. +The measurement above is the evidence for and against that fallback. + +### 9. What ADR 008 gets from it + +ADR 008 (revision 2, PR #308) assumes these, under the shared names below. + +- **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and + per column tuple of cells that hold a value. Stored cells pass the view's filter, with + hidden items dropped, and a collapsed group's members map to the group's tuple. + Formula cells count at their stems, which is today's `formula_cache` stem set. A tuple + is on the axis iff its count is above zero. **This ADR does not settle ADR 008 + question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those + cells are values with no stored cell under them. The stem definition drops them, and + if the user rules otherwise, those views fall back to a non-occupancy path. Occupancy + is integer and order-free, so it does not wait for ADR 005. +- **The occupancy delta** (ADR 008 clause 8). `distinct` over occupancy emits a + non-empty delta exactly when a tuple enters or leaves an axis. A reframe is a + non-empty occupancy delta. The two-arm asymmetry of the frame observation (gotcha 24, + improvise-mcq4) is unchanged, but each arm costs O(delta) instead of a rebuilt grid + per view. +- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over + the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), + not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) + aggregate, and the formula layer re-evaluates only the columns whose inputs moved, + through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that + hold a `Sum` or `Avg` wait for ADR 005 (clause 8). +- **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, + or maintain) is struck there as decided by the user through this ADR. Its questions 1 + and 2 stay open there. +- **What it does not get.** Column widths depend on formatted text, which is + display-only (gotcha 8), so they are not maintained here. They stay ADR 008 question 2. + +### 10. ADR 003's live sources are delta streams, when they can be + +A live source that can report its changes, by polling and diffing, by triggers, or by a +change feed, produces a cell delta. That is the same input `DataStore` produces, and the +circuit cannot tell them apart. That is the "data can stream" half of the user's +prompt. + +**A source that can only answer aggregate queries stays on `Eager`,** with ADR 003's +pushdown. There is a tension: ADR 003 wants data larger than memory, and every +integrated operator here keeps state proportional to its input (clause 4). So the +backend is chosen per document, which is what "pluggable" buys (question 8). A +read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. + +### 11. Constraints + +- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset` and + `improvise-core`. If improvise-command selects the backend, both join the browser + bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds + them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). +- **Two seconds per test binary.** The crate's laws and its "incremental equals + from-scratch" properties run at the default 256 cases over small inputs. The + differential test lives in the seam crate's own binary, so its cost is visible on its + own. Its document sizes are bounded by the strategy, never by raising or lowering the + case count (design-principles §7). +- **Ordering determinism (gotcha 18).** A `ZSet` iterates in `K`'s `Ord` order, never + in hash order. A hash index is allowed for lookup only. Output deltas are + consolidated and sorted before they leave an operator. Nothing in the circuit decides + a display order: record order stays `DataStore`'s (`live_records`, `cell.rs:366`). + The determinism test builds **two** circuits independently from the same history and + compares them. One circuit compared with itself proves nothing (improvise-hzjo's + vacuity). +- **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` + (`app.rs:647`). The backend's state is part of the snapshot. Undoing by applying the + inverse delta is exact for integer weights and order-free aggregators, but not for an + f64 `Sum`, so the snapshot is kept until clause 8's prerequisite lands (question 9). +- **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the + delta sequence, so replaying commands reproduces its state. + +## Alternatives considered + +**Feldera's `dbsp` crate.** What was verified on 2026-09-24: + +- The latest release is 0.354.0, a 4.8 MB compressed tarball, MIT OR Apache-2.0, + requiring Rust 1.96.1. +- It has 67 direct normal dependencies, none optional, and two features + (`backend-mode`, `default`). tokio, mimalloc, nix, zstd, libc and core_affinity are + all required. +- `cargo tree -e normal` resolves about 455 crates natively and 421 for + `wasm32-unknown-unknown`. For comparison, improvise-core's whole normal tree for wasm32 + is 25 crates, and improvise-command's is 68. +- `cargo check` succeeds natively. **`cargo check --target wasm32-unknown-unknown` + fails**: `mio` (tokio's) reports 48 errors. + +What was **not** verified: whether anything past `mio` would also fail. The build stops +at the first failure, so that is estimated, not measured. It is likely, because +mimalloc's sys crate and zstd build C code, and `libsqlite3-sys` already fails under this +repo's nix cc-wrapper (gotcha 41). Even if it built, it is a multi-threaded, +storage-backed query engine where improvise needs a single-threaded operator library. +The user's decision already excludes it. This records why that decision loses nothing. + +**Keep eager evaluation and memoize.** improvise-swre proposes keying `grid_of` on the +existing tiers, so an unchanged grid costs nothing. That is worth doing under either +backend. But every write still recomputes every formula and rescans the store, and a +second view with a different None axis still clones the model. It removes the cost of +*not* changing and leaves the cost of changing at O(cells). + +**An ad-hoc incremental cache.** Track dirty stems by hand: invalidate the groups a +write touches, and recompute them. It is cheaper to start. But each derived fact, such +as a group aggregate, occupancy, a column total, a stem or a cross-stem `WHERE`, needs +its own invalidation rule. Each rule is a second description of evaluation, free to +disagree with the first. The Z-set algebra gives **one** rule, linearity and +bilinearity, whose laws can be property-tested once in the crate. That is the same +argument as design-principles §5's "a derivation done twice is the defect". + +**Compile formulas into operators.** This is the fully DBSP-shaped answer. It is +rejected in clause 2, because it would be a second evaluator. + +## Consequences and migration + +**Order.** Each step is its own ticket and PR, with gates green, and none switches the +displayed path until the differential test holds. + +1. **The `zset` crate**, with its group laws, operator properties and determinism + test. Nothing in improvise uses it yet. +2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the + existing suite is the evidence. `CellKey` gains `Ord`. +3. **The cell delta from `DataStore`.** `bump_changed_records` and the content half of + the dirty flag move onto it. That removes two O(cells) scans per cell-writing batch. + The existing tier tests are the evidence. +4. **`Incremental`, for the order-free parts**: stems, `Count`, `Min`, `Max`, + unanimity, occupancy and the occupancy delta, under the differential test. **ADR 008 + clauses 3 and 8 (pruning and the frame observation) depend on this step.** +5. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's fallback). + **ADR 008 clause 5 depends on this step.** +6. **Formulas**: the formula graph, topological re-evaluation, the component fixed + point, and the aggregate-over-expression map that replaces `Expansion::members`. +7. **Live sources as delta streams**, when ADR 003's source trait exists. + +**Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is +question 10. improvise-swre stays independent and is useful under either backend. +improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause 8. + +**Good.** + +- A write costs O(delta) plus the formula cells it reaches, not O(cells). +- The store scans behind the record tier and the dirty flag disappear. +- Per-view model clones for a different None axis disappear. +- ADR 008 gets occupancy, the occupancy delta and column totals without a second + evaluator. +- Live sources that can report changes plug in without touching the evaluator. + +**Bad.** + +- Two backends, kept equal by a differential test that must itself be kept honest. +- Integrated state is O(cells) in memory, per distinct None axis, in addition to the + store. +- `Sum` and `Avg` cannot be maintained on the displayed path until ADR 005 lands, and + the measurement says so. +- Circularity is decided in two layers: the formula graph's components and the + per-cell bound. +- The browser bundle grows by two small crates. + +## Open questions + +Each question has a recommendation. The user's answers will be folded in under "What +the user decided", as ADRs 004 and 007 did. + +1. **The crate's name.** `zset`, `dbsp-core`, or `improvise-dbsp`? + *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not + carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. + +2. **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on + `zset` and `improvise-core`), or inside improvise-core? + *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and + `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. + +3. **Floating point.** Should ADR 005 (exact decimals) be a prerequisite for maintaining + `Sum` and `Avg` on the displayed path, with the eager fold kept for them until then? + Or should a compensated f64 sum, re-folded on detected cancellation, ship first with + a tolerance in the differential test? + *Recommendation:* ADR 005 as the prerequisite. The measured drift reaches + `FLOAT_EQ_EPSILON`, so `=` could answer differently on the two paths. No f64 technique + makes the differential test bit-exact. Everything else in this ADR lands without it. + +4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported + `circular` today (core.md §3). Should `Incremental` reproduce that? + *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to + change, change it in both backends at once, as its own decision. + +5. **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by + today's bounded fixed point, restricted to their component, rather than by any + operator in the crate? + *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the + cells a guarded cycle leaves convergent. The crate needs no recursion operator. + +6. **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? + *Recommendation:* yes, permanently. It is the differential test's oracle and the + backend for ADR 003's query-only sources. Deleting it would delete the test. + +7. **Page selection as a slice.** Should the group key include the page categories, so + switching page reads existing state, rather than rebuilding the view's operators per + page change? + *Recommendation:* the slice. The state stays bounded by the stored cells, and the + held-key path (`[`/`]`) stays O(visible). ADR 008 revision 2 assumes this. + +8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, + with query-only sources kept on `Eager` and ADR 003's pushdown? + *Recommendation:* yes, chosen per document. Integrated state is proportional to the + source, which contradicts ADR 003's larger-than-memory goal for exactly the sources + that cannot report their changes anyway. + +9. **Rollback.** For ADR 001 clause 4, should the backend's state be snapshotted with + the `App`, or undone by applying the inverse delta? + *Recommendation:* snapshot until ADR 005 lands, then the inverse delta. The inverse + is exact only when every aggregator is. + +10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps + 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by + steps 4 and 5? + *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are + how it is met. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs new file mode 100644 index 00000000..9e627378 --- /dev/null +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -0,0 +1,191 @@ +//! How far a SUM maintained by insert/retract deltas drifts from the eager fold, +//! in f64. ADR 009 (improvise-felo.4), "Floating point". + +use improvise_core::format::format_f64; +use improvise_core::model::cell::{CellKey, CellValue}; +use improvise_core::model::types::{Aggregation, Model}; + +const MEASURE: &str = "_Measure"; + +/// SplitMix64: seeded, so a run is reproducible and a second seed is a new sample. +struct Rng(u64); + +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + fn below(&mut self, n: u64) -> u64 { + self.next() % n + } +} + +/// A user-entered decimal as an exact count of 1e-5 units, its magnitude spread +/// over `decades` decades. The f64 the store holds is `typed(units)`. +fn decimal(rng: &mut Rng, decades: u32) -> i64 { + let scale = 10i64.pow(rng.below(u64::from(decades)) as u32); + let cents = rng.below(100_000) as i64; + let sign = if rng.below(4) == 0 { -1 } else { 1 }; + sign * cents * scale +} + +fn typed(units: i64) -> f64 { + units as f64 / 1e5 +} + +/// The exact decimal sum, rounded to f64 once: what ADR 005's arithmetic would show. +fn exact(units: &[i64]) -> f64 { + units.iter().map(|&u| i128::from(u)).sum::() as f64 / 1e5 +} + +fn key(row: usize) -> CellKey { + CellKey::new(vec![ + ("Row".to_string(), format!("r{row}")), + (MEASURE.to_string(), "Amount".to_string()), + ]) +} + +/// `cells` rows of one measure; the eager SUM is the one the grid draws. +fn document(rng: &mut Rng, cells: usize, decades: u32) -> (Model, Vec) { + let mut m = Model::new("drift"); + m.add_category("Row").expect("under the limit"); + let values: Vec = (0..cells).map(|_| decimal(rng, decades)).collect(); + for (row, v) in values.iter().enumerate() { + m.category_mut("Row") + .expect("just added") + .add_item(format!("r{row}")); + m.set_cell(key(row), CellValue::Number(typed(*v))); + } + (m, values) +} + +fn eager(m: &Model) -> f64 { + let total = CellKey::new(vec![(MEASURE.to_string(), "Amount".to_string())]); + m.evaluate_aggregated_f64(&total, Aggregation::OverHidden) +} + +/// Neumaier's compensated running sum: the cheap mitigation, measured beside the naive one. +#[derive(Default)] +struct Compensated { + sum: f64, + carry: f64, +} + +impl Compensated { + fn add(&mut self, x: f64) { + let t = self.sum + x; + self.carry += match self.sum.abs() >= x.abs() { + true => (self.sum - t) + x, + false => (x - t) + self.sum, + }; + self.sum = t; + } + + fn value(&self) -> f64 { + self.sum + self.carry + } +} + +fn ulps(a: f64, b: f64) -> u64 { + (a.to_bits() as i64).abs_diff(b.to_bits() as i64) +} + +/// Each running sum against the eager fold (the differential oracle) and against the +/// exact decimal sum. `flip`: the gap reaches `FLOAT_EQ_EPSILON`, so `=` can answer differently. +fn report(label: &str, cycles: usize, exact: f64, eager: f64, naive: f64, comp: f64) { + let shown = |v: f64| format_f64(v, true, 2); + println!( + "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e}", + (naive - eager).abs(), + ulps(naive, eager), + (comp - eager).abs(), + ulps(comp, eager), + (naive - eager).abs() >= 1e-10, + (eager - exact).abs(), + (naive - exact).abs(), + (comp - exact).abs(), + ); + if shown(naive) != shown(eager) || shown(eager) != shown(exact) { + println!( + " shown at ,.2: naive {} eager {} exact {}", + shown(naive), + shown(eager), + shown(exact) + ); + } +} + +/// Each cycle retracts one live value and inserts a fresh one at the same key. +fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) { + let mut rng = Rng(seed); + let (mut m, mut values) = document(&mut rng, cells, decades); + let mut naive = eager(&m); + let mut comp = Compensated::default(); + comp.add(naive); + let mut done = 0; + for &at in checkpoints { + while done < at { + let row = rng.below(cells as u64) as usize; + let new = decimal(&mut rng, decades); + naive = naive - typed(values[row]) + typed(new); + comp.add(-typed(values[row])); + comp.add(typed(new)); + values[row] = new; + m.set_cell(key(row), CellValue::Number(typed(new))); + done += 1; + } + let label = format!("{cells} cells, {decades} dec"); + report(&label, at, exact(&values), eager(&m), naive, comp.value()); + } +} + +/// A large value arrives, small ones accumulate under it, then it is retracted. +fn absorbed_by_a_large_value() { + let mut m = Model::new("absorb"); + m.add_category("Row").expect("under the limit"); + m.category_mut("Row").expect("just added").add_item("r0"); + let mut naive = 0.0; + let mut comp = Compensated::default(); + m.set_cell(key(0), CellValue::Number(1e17)); + naive += 1e17; + comp.add(1e17); + for row in 1..=1000 { + m.category_mut("Row") + .expect("just added") + .add_item(format!("r{row}")); + m.set_cell(key(row), CellValue::Number(0.01)); + naive += 0.01; + comp.add(0.01); + } + m.clear_cell(&key(0)); + naive -= 1e17; + comp.add(-1e17); + report("1e17, 1000 x 0.01, -1e17", 1001, 10.0, eager(&m), naive, comp.value()); +} + +fn main() { + println!( + "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10}", + "case", + "cycles", + "naive-eag", + "ulp", + "comp-eag", + "ulp", + "flip", + "eag-exact", + "nai-exact", + "cmp-exact" + ); + let checkpoints = [1_000, 10_000, 100_000, 1_000_000]; + for seed in [1, 2] { + random_updates(seed, 1_000, 3, &checkpoints); + random_updates(seed, 18_000, 3, &checkpoints); + random_updates(seed, 18_000, 9, &checkpoints); + } + absorbed_by_a_large_value(); +} diff --git a/scripts/build-doc-site.mjs b/scripts/build-doc-site.mjs index f9bab7b3..a2967458 100644 --- a/scripts/build-doc-site.mjs +++ b/scripts/build-doc-site.mjs @@ -46,6 +46,7 @@ const SECTIONS = [ ["context/adr/003-live-backed-models.md", "adr/003-live-backed-models", "ADR 003 — Live-backed models"], ["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"], ["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"], + ["context/adr/009-incremental-evaluation.md", "adr/009-incremental-evaluation", "ADR 009 — Incremental evaluation (proposed)"], ], }, { From 5e60e59acd5b9df028699b25f1acf8c4157c7c11 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:15:47 -0700 Subject: [PATCH 02/67] docs(adr): state ADR 009's drift counts per checkpoint; rustfmt the drift example (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 ++++++++------ crates/improvise-core/examples/retraction_drift.rs | 9 ++++++++- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b61fe0ad..802addc3 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -379,12 +379,14 @@ Four conclusions follow: any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit today, because they are order-free. It can compare `Sum` and `Avg` only with a tolerance. -- **The gap reaches the epsilon.** In most 18k-cell cases the incremental sum and the - eager sum differ by more than `FLOAT_EQ_EPSILON`. So `IF(SUM(..) = x, ..)` can answer - differently on the two paths. That is a behaviour difference, not a display one. -- **This is already a bug in eager, which the measurement found in passing.** The eager - 18k-cell sum is 1.5e-10 to 5.8e-10 from the exact decimal sum, so `=` against a - decimal a user typed can already be wrong today. That supports ADR 005 on its own +- **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum + differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So + `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour + difference, not a display one. +- **This is already a bug in eager, which the measurement found in passing.** With 3 + decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal + sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed + can already be wrong today. That supports ADR 005 on its own merits, and it is recorded here for improvise-l2b8. - **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 9e627378..92dae6e5 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -164,7 +164,14 @@ fn absorbed_by_a_large_value() { m.clear_cell(&key(0)); naive -= 1e17; comp.add(-1e17); - report("1e17, 1000 x 0.01, -1e17", 1001, 10.0, eager(&m), naive, comp.value()); + report( + "1e17, 1000 x 0.01, -1e17", + 1001, + 10.0, + eager(&m), + naive, + comp.value(), + ); } fn main() { From 9be37faf79471208c06df90b1e3e4d1c0d8d50dc Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Thu, 24 Sep 2026 20:17:24 -0700 Subject: [PATCH 03/67] docs(adr): name ADR 009's crate options without an id-shaped token (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 802addc3..edcb5ac3 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -558,7 +558,7 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. -1. **The crate's name.** `zset`, `dbsp-core`, or `improvise-dbsp`? +1. **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. From c456be2d5400ea25c8fc1b2a273b68ac54099839 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 20:45:59 -0700 Subject: [PATCH 04/67] docs(adr): ADR 009 follows the document per model effect; answer the joint review (improvise-felo.4) - clause 7: the operator shape and the formula-set fingerprint are re-read after each model effect inside the loop. Formula edits are an explicit input, and a document swap rebuilds everything. Stated in the same words as ADR 008 revision 3. - clause 3: a delta key's identity is bit identity, matching hash_value. The dirty flag keeps a formula fingerprint, so formula edits still persist. - clause 5: the stem rule for a formula with a top-level WHERE. - clause 11 and question 9: rollback is a snapshot, with its second O(cells) clone stated. The inverse delta is not a rewind. - migration: formulas become step 5, and Sum/Avg and column totals step 6. - the drift example's values are five-place, not two-place. Clause 7 and question 7 are worded as recommendations. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 221 ++++++++++++++++------ 1 file changed, 166 insertions(+), 55 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index edcb5ac3..ffee8dd0 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -15,7 +15,7 @@ no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means -revision 2 of `context/adr/008-virtualized-grid.md` on PR #308. +revision 3 of `context/adr/008-virtualized-grid.md` on PR #308. ## The decision this ADR writes down @@ -64,10 +64,15 @@ plugs in, and what it may not change. (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every combination of the spanned categories' items. ADR 008 clause 5 names this as the one O(grid) path left in totals. -- **Two of the record tiers are observed by scanning the store.** `record_fingerprints` - (`app.rs:254-264`) hashes every cell before and after a cell-writing batch, and - `document_content` (`app.rs:165-178`) does the same for the dirty flag. Both are - O(cells) per batch, whatever the batch wrote. +- **Two observations scan the store.** `record_fingerprints` (`app.rs:254-264`) hashes + every cell before and after a cell-writing batch. `document_content` + (`app.rs:165-180`) hashes every cell and every formula for the dirty flag, and it is + re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) + per batch and the second O(cells) per effect, whatever was written. +- **Every mutating frame from a client already clones the document.** + `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:162`, + `app.rs:627-629`), for ADR 001 clause 4. That is O(cells) per frame on the network + path, whatever this ADR does. ### What an incremental evaluator must not change @@ -158,8 +163,8 @@ name is `Derived`. It replaces the two places where eager evaluation folds the s - the raw fold of the cells matching a partial key under an aggregation, which is `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). -It has one mutation, `apply(&mut self, delta: &CellDelta)`, plus a rebuild for the -changes that are not cell deltas (clause 7). +It takes two inputs, both applied after each model effect (clause 7): the cell delta, +and the formula delta. It also has a rebuild, for an operator shape that moved. **The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code and read through the trait. The circuit decides *which* formula cells to re-evaluate @@ -198,6 +203,25 @@ An overwrite with an equal value consolidates to the empty Z-set. The circuit st per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). +**A delta key's identity is bit identity, the same identity the record tier uses +today.** Two `(CellKey, CellValue)` pairs are one element exactly when their keys are +equal and their values are equal *by bits*: `Number` by `f64::to_bits`, `Text` and +`Error` by their strings. That is `hash_value` (`app.rs:233-250`), which +`record_fingerprints` already hashes. So the cell delta reproduces today's tier +exactly: + +- overwriting `0.0` with `-0.0` is a non-empty delta and bumps the record, as it does + today; +- overwriting a NaN with a NaN of the same bits is empty and bumps nothing, as today. + +It is **not** `CellValue`'s derived `PartialEq`, under which `-0.0 == 0.0` and +`NaN != NaN`. `recompute_formulas` uses that one for its "did this cell move" test +(`types.rs:1010`), and it stays there. The ordering behind a multiset (clause 4) is +`f64::total_cmp`, whose equality *is* bit equality for every finite and non-finite +value. So the delta identity and the multiset ordering agree, and neither is +`PartialEq`. Step 3's tests pin both cases: signed zero, and a repeated same-bits +NaN. + **This lines up with ADR 001's record tier with no extra rule.** The records a batch moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys in the batch's consolidated delta. So: @@ -208,9 +232,19 @@ in the batch's consolidated delta. So: `bump_changed_records` can then read the delta instead of diffing two `record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta -comes from the mutation sites, not from an `EffectKind`. The same holds for the content -half of the dirty flag, where a non-empty delta means dirty. `layout_version` is -unchanged, because it observes `view_shape` and not cells. +comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, +because it observes `view_shape` and not cells. + +**The dirty flag's content half does not move onto the cell delta alone.** +`document_content` (`app.rs:165-180`) hashes the formulas as well as the cells, and it +is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, +`RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, +`:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a +non-empty cell delta means dirty" would never save a formula edit in the browser. The +content half becomes: **a non-empty cell delta, or a moved formula-set fingerprint**. +The fingerprint is the one clause 7 already reads after each model effect, so nothing +new is observed. Its cost is O(formulas), where `document_content` is O(cells) per +`CellData` effect. **`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` (`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only @@ -252,11 +286,13 @@ key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the me whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A `Sum`-only document keeps one number per group. -**`total_cmp` is not `==`, and the differential test must say so.** `NaN != NaN`, so -today two NaNs in one group read as `mixed`, and `f64::min` skips a NaN. A multiset keyed -by `total_cmp` would call two NaNs one value and would order NaN at an end. The test -generates non-finite values (core.md §16), and the aggregator matches eager -behaviour explicitly rather than inheriting `total_cmp`'s. +**The multiset's identity is clause 3's bit identity, and eager's *reading* is not.** A +multiset keyed by `total_cmp` holds the values the store holds, by bits, and that is +correct for the state. But eager *reads* a group through `PartialEq` and `f64::min`: +`NaN != NaN`, so two NaNs in one group read as `mixed` today, `-0.0` and `0.0` read as +one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eager's +rules explicitly and does not inherit `total_cmp`'s. The differential test generates +signed zeros and non-finite values (core.md §16) to pin it. ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it @@ -265,6 +301,21 @@ with the target category and each `none_cats` entry removed. That is `distinct(map(project))` over the cell delta. A stem enters when its first cell arrives and leaves when its last cell goes. +**The stem rule: a formula cell exists at a stem only where its value would be cached +today.** `recompute_formulas` caches a formula cell only when the cached evaluator +returns a value (`types.rs:1010-1014`). For a formula with a top-level `WHERE`, a stem +whose coordinate fails the filter falls back to `aggregate_raw` (`types.rs:1047-1055`), +which is absent when no raw data lies at the target key. So: + +- **a formula counts at a stem only where its top-level `WHERE` (if any) matches the + stem's coordinate, or where raw data exists at stem + target.** + +This is a coordinate predicate plus one count lookup, never an evaluation. It is the +rule ADR 008 clause 3's occupancy uses for formula cells, and it keeps today's pruning. +The review of ADR 008 found the case that needs it: `Double = Revenue * 2 WHERE +Region = East` with data only at West is pruned today, and a rule that counted every +stem would have kept the column. + **A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula `T` at stem `s` are: @@ -326,26 +377,51 @@ So the crate has no recursion operator, and circularity is decided *outside* it: like with like. Changing the rule would be its own decision, for both backends at once. -### 7. What is not a cell delta rebuilds, once +### 7. The circuit follows the document per model effect -Some changes alter the circuit's *shape*, not its input: +Some changes alter what the circuit computes, not only its input. They are observed +**inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same +position as the frame and dirty observations (`app.rs:1036-1084`). They are not the +post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). +That diff comes too late, and it cannot see most of these changes. ADR 008 states this +clause in the same words. -- the formula set: add, remove or replace; -- `measure_agg`; -- category and item structure: add or remove a category, remove an item, and group - membership; -- a view's axes, hidden items and collapsed groups, which change `view_shape`; -- a document swap, `document_generation`. +- **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), + plus the group membership of every drawn item, collapsed or not, plus + `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is + unchanged. A view whose operator shape moved has its operators rebuilt from the + integrated base Z-set **before the next effect's cell delta is applied**. +- **The formula set is an input, not a rebuild.** After each model effect the host + compares a fingerprint of the formula set, over (`raw`, `target`, + `target_category`), with its previous read. If it moved, the formula delta (−the old + formulas, +the new ones) is fed to the circuit as an explicit input. The formula + layer then re-derives the formula graph (clause 6) and re-evaluates the affected + targets at their stems. This is observed, not declared by effect type, as the tiers + are. It is what catches a `ReplaceFormula` that keeps its target, which moves no + `view_shape`. +- **A document swap rebuilds every circuit** from the new store. `install_workbook` + bumps `document_generation`, and the same after-each-effect read sees it. -Each rebuilds only the affected operators, from the integrated base Z-set. That costs -O(cells), once per change, and is never repeated per keypress. **The trigger is the same -observation the tiers use**: a `view_shape` diff (`app.rs:40-72`) rebuilds that view's -operators, the same diff that bumps `layout_version`. So there is one signal and not two. +Re-reading the operator shape costs O(items) per view per model effect, and the formula +fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs +O(cells), once per actual reshape. The same `view_shape` function is therefore read at +two moments: by the circuit after each effect, and by `layout_version` after the loop. +They answer different questions, and neither replaces the other. -**A page selection is not a rebuild.** The group key includes the page categories, so -`[` and `]` read another slice of state that already exists. The held-key path stays -O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the -state is still bounded by the stored cells, however many pages there are (question 7). +**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** `AddItem` +moves the operator shape, so the view is rebuilt with East in its sequences. Only then +does `SetCell`'s delta arrive, at operators that already know East. So the in-loop +`CellData` frame observation reads occupancy for the new shape. The `LayoutGeometry` +arm (`app.rs:1062-1071`) reads the frame immediately after each geometry effect, and it +reads a rebuilt occupancy for the same reason. + +**Recommended: a page selection is not a rebuild (question 7).** Under the recommended +answer, the group key includes the page categories, so `[` and `]` read another slice +of state that already exists. The held-key path stays O(visible) (improvise-748f's +constraint). The slices cost nothing extra, because the state is still bounded by the +stored cells, however many pages there are. Filter coordinates still filter; only page +coordinates key the slice. If the user answers the other way, a page change is an +operator-shape change like any other. ### 8. Floating point: retraction drifts, and exactness is ADR 005's @@ -356,8 +432,10 @@ the eager fold computes. **Measured**, not asserted: an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N retract-and-insert cycles. It compares the sum against the eager fold, `evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count -of 10⁻⁵ units rounded to f64 once. Values are signed two-place decimals spread over 3 or -9 decades. +of 10⁻⁵ units rounded to f64 once. Each value is a signed whole number of 10⁻⁵ units: +up to 99,999 units, times a power of ten spread over 3 or 9 decades. So the smallest +values carry five decimal places, not the two a user usually types. Two-place input +would absorb differently, and it was **not** measured. | store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | |---|---|---|---|---| @@ -403,12 +481,15 @@ The measurement above is the evidence for and against that fallback. ### 9. What ADR 008 gets from it -ADR 008 (revision 2, PR #308) assumes these, under the shared names below. +ADR 008 (revision 3, PR #308) assumes these, under the shared names below. - **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and per column tuple of cells that hold a value. Stored cells pass the view's filter, with hidden items dropped, and a collapsed group's members map to the group's tuple. - Formula cells count at their stems, which is today's `formula_cache` stem set. A tuple + Formula cells count by clause 5's stem rule, which is when today's `formula_cache` + holds a value: a top-level `WHERE` must match, or raw data must exist at the target. + The operators are rebuilt per clause 7, so a batch that reshapes and then writes feeds + its write to the new shape. Page coordinates key a slice, subject to question 7. A tuple is on the axis iff its count is above zero. **This ADR does not settle ADR 008 question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those cells are values with no stored cell under them. The stem definition drops them, and @@ -463,9 +544,20 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:647`). The backend's state is part of the snapshot. Undoing by applying the - inverse delta is exact for integer weights and order-free aggregators, but not for an - f64 `Sum`, so the snapshot is kept until clause 8's prerequisite lands (question 9). + (`app.rs:647`). **Recommended (question 9): the backend's state joins the snapshot.** + Applying the inverse delta is **not** a rollback, even with exact aggregators. It + restores integrated values, but `step(d)` then `step(−d)` still advances the circuit + and changes the history that `delay` and `differentiate` hold. So the next step after + a "rewind" can differ from the next step after nothing happened. A real rewind would + have to restore every stateful operator, with a test of observational equivalence + across all of them. This ADR does not propose one. + + **The snapshot costs a second O(cells) clone per mutating frame.** + `Host::mutating_frame` already clones the workbook on every mutating frame + (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So + on the network path a write is not O(delta): it is O(cells) for the snapshot today, + and roughly twice that with the circuit in it. The keyboard path (`apply_effects` + without a `Host`) takes no snapshot and keeps the O(delta) cost. - **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the delta sequence, so replaying commands reproduces its state. @@ -517,16 +609,23 @@ displayed path until the differential test holds. test. Nothing in improvise uses it yet. 2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the existing suite is the evidence. `CellKey` gains `Ord`. -3. **The cell delta from `DataStore`.** `bump_changed_records` and the content half of - the dirty flag move onto it. That removes two O(cells) scans per cell-writing batch. - The existing tier tests are the evidence. -4. **`Incremental`, for the order-free parts**: stems, `Count`, `Min`, `Max`, - unanimity, occupancy and the occupancy delta, under the differential test. **ADR 008 - clauses 3 and 8 (pruning and the frame observation) depend on this step.** -5. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's fallback). - **ADR 008 clause 5 depends on this step.** -6. **Formulas**: the formula graph, topological re-evaluation, the component fixed - point, and the aggregate-over-expression map that replaces `Expansion::members`. +3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero + and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half + becomes "a non-empty cell delta, or a moved formula-set fingerprint" (clause 3), with + a test that a formula edit alone still marks the document dirty. That removes the + O(cells) scans per cell write. The existing tier tests are the evidence. +4. **`Incremental`, for the order-free parts**, under the differential test: stems with + clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy + delta, all following clause 7's per-effect operator shape. **ADR 008 clauses 3 and 8 + (pruning and the frame observation) depend on this step.** +5. **Formulas**: the formula graph and the formula delta (clause 7), topological + re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression + map that replaces `Expansion::members` (clause 5). Formula values that depend on a + `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of + `Expansion::members` depends on this step.** +6. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's + fallback). A column total is a formula over the pooled fold, so it needs step 5's + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is @@ -535,7 +634,9 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause **Good.** -- A write costs O(delta) plus the formula cells it reaches, not O(cells). +- On the keyboard path, a write's *evaluation* costs O(delta) plus the formula cells it + reaches, not O(cells). On the network path, the ADR 001 snapshot still costs O(cells) + per mutating frame (see "Bad"). - The store scans behind the record tier and the dirty flag disappear. - Per-view model clones for a different None axis disappear. - ADR 008 gets occupancy, the occupancy delta and column totals without a second @@ -551,6 +652,11 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause the measurement says so. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. +- A mutating frame from a client already clones the workbook for rollback + (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) + clone per frame. Incremental evaluation does not make the network path O(delta). +- The operator shape is re-read after every model effect: O(items) per view, a cost the + post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. ## Open questions @@ -594,7 +700,8 @@ the user decided", as ADRs 004 and 007 did. switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the - held-key path (`[`/`]`) stays O(visible). ADR 008 revision 2 assumes this. + held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this + recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". 8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? @@ -602,13 +709,17 @@ the user decided", as ADRs 004 and 007 did. source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. **Rollback.** For ADR 001 clause 4, should the backend's state be snapshotted with - the `App`, or undone by applying the inverse delta? - *Recommendation:* snapshot until ADR 005 lands, then the inverse delta. The inverse - is exact only when every aggregator is. +9. **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` + snapshot that `Host::mutating_frame` already takes? + *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying + `−d` restores integrated values but not the history `delay` and `differentiate` + hold, so it is not a rewind even with exact aggregators (clause 11). The cost is a + second O(cells) clone per mutating frame, beside the workbook clone that exists + today. A true rewind would need its own design and an observational-equivalence test + over every stateful operator. 10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by - steps 4 and 5? + steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. From 0ab3ac7dbf5186f848400459dfd3bea9674513a1 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 21:27:56 -0700 Subject: [PATCH 05/67] docs(adr): ADR 009 rebuilds before the frame observation, and before step 5 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ffee8dd0..5c26445a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -383,14 +383,16 @@ Some changes alter what the circuit computes, not only its input. They are obser **inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same position as the frame and dirty observations (`app.rs:1036-1084`). They are not the post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). -That diff comes too late, and it cannot see most of these changes. ADR 008 states this -clause in the same words. +That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 +states the same rule. - **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), plus the group membership of every drawn item, collapsed or not, plus `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is unchanged. A view whose operator shape moved has its operators rebuilt from the - integrated base Z-set **before the next effect's cell delta is applied**. + integrated base Z-set **before the frame observation that follows this effect**. + The `LayoutGeometry` frame arm (`app.rs:1062-1071`) reads occupancy right after the + effect that moved the shape, so the next effect's cell delta is too late. - **The formula set is an input, not a rebuild.** After each model effect the host compares a fingerprint of the formula set, over (`raw`, `target`, `target_category`), with its previous read. If it moved, the formula delta (−the old @@ -616,8 +618,10 @@ displayed path until the differential test holds. O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's per-effect operator shape. **ADR 008 clauses 3 and 8 - (pruning and the frame observation) depend on this step.** + delta, all following clause 7's per-effect operator shape. The formula delta arrives + only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in + step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the + frame observation) depend on this step.** 5. **Formulas**: the formula graph and the formula delta (clause 7), topological re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression map that replaces `Expansion::members` (clause 5). Formula values that depend on a From a2836f44f57fec90caf299bd9650d10d2fd6e81e Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 21:53:29 -0700 Subject: [PATCH 06/67] docs(adr): ADR 009 leads with incremental formulas; exact f64 sums; ordered aggregates (improvise-felo.4) - Reframed: incremental formula evaluation for streamed data (ADR 003) is the primary benefit, and ADR 008's grid is the second consumer. - New clause 5a: aggregate over an expression, as a lifted map feeding the group aggregate, still evaluated by eval_expr. - Clause 4: ordered state for LAST/FIRST (felo.7) and semi-additive measures (felo.8). New question 11. - Clause 8: DBSP alone gives no floating-point benefit. An exact superaccumulator makes f64 SUM a true group. It is measured in retraction_drift: bit-equal after 1M cycles, about 3.2 ns per update, 568 bytes per group. It is correct against math.fsum on 20,000 cases, and xsum 0.1.6 is rejected. The principle of deferring lossy operations, and a table of what each exact domain covers. Question 3 gains option (c) and recommends it. No clause or question number moved. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 330 +++++++++++++++--- .../examples/retraction_drift.rs | 284 +++++++++++++-- 2 files changed, 544 insertions(+), 70 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 5c26445a..5e9d4aa8 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -42,8 +42,38 @@ So the choice between Feldera's crate and our own is already made (see Alternati what was checked). This ADR says what "the core of dbsp" is for improvise, where it plugs in, and what it may not change. +## What it is for + +**The primary benefit is incremental formula evaluation, so that data can stream.** +That is the user's own framing, both in the prompt that led to this ADR and in their +later question: "Isn't the benefit of adopting DBSP primarily incremental formula +updates?" Yes. When a cell changes, whether from a keystroke, a client's frame or a +live source (ADR 003, clause 10), only the aggregates that cell feeds are adjusted, +and only the formula cells whose inputs moved are re-evaluated. Nothing is recomputed +from scratch. The machinery is: + +- the aggregates formulas read are **maintained**, not re-folded (clause 4); +- an aggregate over a per-record expression such as `SUM(Revenue * Price)` is a + maintained operator too (clause 5a); +- the circuit decides **which** formula cells moved, and the one existing evaluator + recomputes only those, in dependency order (clauses 5 and 6). + +**The grid is the second consumer.** ADR 008 needs the same maintained state for other +purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, +and column totals over rows never scrolled to (clause 9). It gets them from the same +circuit, not from a second mechanism. + ## Context +### The primary case: data that streams + +A live-backed model (ADR 003) is a source whose rows change underneath the document. +Today the only way to reflect a change is the path every edit takes: recompute every +formula cell of the document (next subsection). That is tolerable when a human types +one cell at a time. It is not tolerable when a source delivers a stream of changes, +each touching a few records of a large table. Incremental evaluation makes the cost of +a change proportional to what the change touches, not to the document. + ### Evaluation today is a from-scratch fold, rerun after every batch - **Every batch recomputes every formula cell.** `App::apply_effects` @@ -98,10 +128,14 @@ incremental evaluator could quietly disagree with the eager one. ## Decision -**A new, problem-agnostic workspace crate implements the DBSP core. improvise-core -defines a trait for the derived state it reads. Today's evaluator is one implementation -of that trait, and a circuit built on the new crate is the other. Both stay in the tree, -and a differential test holds them equal.** Eleven clauses. +**Formulas are evaluated incrementally. A change re-evaluates only the formula cells +whose inputs moved, over aggregates that are maintained rather than re-folded. The +machinery is a new, problem-agnostic workspace crate implementing the DBSP core. +improvise-core defines a trait for the derived state its evaluator reads. Today's +evaluator is one implementation of that trait, and a circuit built on the new crate is +the other. Both stay in the tree, and a differential test holds them equal.** Eleven +clauses, plus clause 5a. It is numbered 5a so that no clause number or question number +cited elsewhere moves. ### 1. The crate: `zset` (name is question 1) @@ -265,7 +299,7 @@ comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does | answer | state per group | cost per delta element | exact on f64? | |---|---|---|---| | `Count` | count of values, count of numbers | O(1) | yes (integers) | -| `Sum` | running sum | O(1) | **no** (clause 8) | +| `Sum` | running sum (an exact accumulator under question 3's recommendation) | O(1) | **no** as an f64; **yes** as the accumulator (clause 8) | | `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | | `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | | unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | @@ -294,6 +328,43 @@ one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eag rules explicitly and does not inherit `total_cmp`'s. The differential test generates signed zeros and non-finite values (core.md §16) to pin it. +#### Ordered aggregates: `LAST` and `FIRST` need state ordered by an item, not a value + +improvise-felo.7 asks for `LAST` and `FIRST` over a fold: the closing or opening value +of a stock price, say, across the months of a year. improvise-felo.8 asks for +semi-additive measures: `sum` across one category and `last` across another. Both are +open tickets with their own semantics to decide. What this ADR must say is whether its +per-group state can carry them. Today it cannot: the integrated state above is an +**unordered** multiset, keyed by the value itself. + +**The structure.** An ordered aggregate keeps, per group, a `BTreeMap` keyed by the +**ordering category's item order**, with the tie-break felo.7 decides. Each entry holds +the values at that position with their weights. `LAST` reads the entry at the greatest +key, and `FIRST` the least. + +- **It is the same shape as `Min`/`Max`.** Those also keep an integrated ordered map and + read an extreme; they order by the value, where `LAST`/`FIRST` order by the item's + position and read the value stored there. So one generic operator covers all four: + an integrated map keyed by `K`, read at an extreme. `Min`/`Max` instantiate it with + `K = value`, and `LAST`/`FIRST` with `K = (item position, tie-break)`. +- **Cost:** O(log d) per delta element, where d is the number of distinct positions in + the group, and one entry per contributing record. That is the same order as + `Min`/`Max`'s multisets. +- **What moves the order.** Adding an item in the middle of the ordering category, or + reordering its items, changes every stored position. So the ordering category's + item order must be part of the operator shape (clause 7), and a change to it rebuilds + the ordered aggregators that use it. The alternative is to key by a stable item + identity (ADR 007) and look the order up at read time. That makes a reorder free, but + a read then costs O(d). This is question 11. +- **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, + `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a + delta (retract the old value, insert the new), and the outer consumes it. So the + composition is incremental at O(log d) per stage. Which stage is inner is felo.8's + semantic decision, because sum-of-last and last-of-sum differ. The circuit takes + either order. +- **Exactness.** `LAST`/`FIRST` return a stored value unchanged, so they are exact on + f64 like `Min`/`Max`. A semi-additive `sum` stage still needs question 3's answer. + ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it **Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key @@ -340,14 +411,61 @@ With the trait, both reads go to the maintained state. not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own group and nothing else. -**An aggregate over an expression replaces `Expansion::members`.** `SUM(expr)` pools -`expr` evaluated at each member record (`agg_values`, `types.rs:89-139`). Incrementally, -that is `map(record ↦ expr at record)`, then `index_by(base key)`, then `Sum`. The -member records never have to be listed. **The boundary:** the map is sound only when -`expr` reads the record's own cells. An inner expression that reads another record, or -holds a nested aggregate, stays on the eager path for that formula, and the -differential test pins the boundary. Until `Sum` is exact, this path carries clause 8's -caveat. ADR 008 clause 5 adopts the same boundary. +**An aggregate over an expression** is the one place a formula needs an expression +*inside* an operator. It has its own clause, 5a. + +### 5a. Aggregate over an expression: a lifted map feeding the group aggregate + +`SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per +record**, `Revenue * Price` at that record, and pools those values across the records +of a group. Today `agg_values` (`types.rs:89-139`) lists the member records of each +context and calls the record-level evaluator on each (`member_records`, then +`eval_at`). At a fold or a column total, the contexts come from `Expansion::members` +(`types.rs:201-213`), which materializes the cross product of the spanned categories' +items. That is the one O(grid) path in totals (ADR 008 clause 5). + +**The design.** It is a three-stage pipeline over the record stream, and each stage is +an operator from clause 1: + +1. **A lifted map over the per-record expression.** For each record in the cell + delta's `record_of` set, the map produces `(base key, expr at record)`. A record + whose cells changed is retracted at its old value and inserted at its new one, so + the map's output is itself a delta. `map` is linear, so this is its own incremental + form. +2. **`index_by(base key)`.** The base key is the record's key with the target category + removed, which is what `eval_agg` pools over (`types.rs:55-87`). An inner `WHERE` + (`SUM(expr WHERE Cat = Item)`) is a `filter` before the index. It is the same + rewrite `eval_agg` applies to its contexts (`types.rs:66-72`). +3. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the + mapped values, with `Avg` derived. + +Member records never have to be listed, and no cross product is ever built. A context +in today's cross product that holds no records contributed nothing to the pool, so +pooling only the records that exist is the same answer. + +**This is not compiling formulas into operators.** The operator is the map; the +expression inside it is still evaluated by `eval_expr` at the record, exactly as +`agg_values` calls it today. The circuit decides *which* records to re-evaluate (those +the delta names), and the one evaluator computes each value. Clause 2's rule stands: +there is one evaluator. + +**The boundary.** The map is sound only when `expr` reads the record's own cells, so +that its value at a record changes only when that record changes. An inner expression +that reads another record, holds a nested aggregate, or refers to a formula whose value +depends on other records, is not a function of one record. That formula stays on the +eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the +same boundary. + +**What stays eager until step 5, and until step 6.** +- Before step 5 (formulas), every aggregate over an expression is evaluated as today, + through `Expansion::members`. +- From step 5, the map and the index are maintained, so the cross product is gone. + `Count`, `Min` and `Max` over the mapped values are read from the maintained state, + since they are order-free and bit-exact on f64 (clause 8). +- `Sum` and `Avg` over the mapped values are **re-folded** from the group's maintained + values, in the store's record order, until step 6. That keeps them bit-equal to + eager, at a cost of O(group) per change rather than O(grid). +- From step 6 (on question 3's answer), `Sum` is maintained like any other. ### 6. Circular references: DBSP's recursion is not our semantics @@ -425,7 +543,7 @@ stored cells, however many pages there are. Filter coordinates still filter; onl coordinates key the slice. If the user answers the other way, a page change is an operator-shape change like any other. -### 8. Floating point: retraction drifts, and exactness is ADR 005's +### 8. Floating point: retraction drifts, and exactness comes from the representation f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum the eager fold computes. **Measured**, not asserted: @@ -453,12 +571,12 @@ A Neumaier-compensated running sum recovers that case exactly. In the random cas stays about as close to the exact sum as eager does, **but it is never bit-equal to eager**, because eager folds in store order and is itself inexact. -Four conclusions follow: +Four conclusions follow about a **floating-point** running sum: - **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by - any running-sum technique. The differential test can compare `Count`, `Min`, `Max`, - unanimity and occupancy bit for bit today, because they are order-free. It can compare - `Sum` and `Avg` only with a tolerance. + any f64 running-sum technique, naive or compensated. So the differential test can + compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit, because they are + order-free, but it can compare an f64 `Sum` and `Avg` only with a tolerance. - **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour @@ -466,20 +584,115 @@ Four conclusions follow: - **This is already a bug in eager, which the measurement found in passing.** With 3 decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed - can already be wrong today. That supports ADR 005 on its own - merits, and it is recorded here for improvise-l2b8. -- **Exact decimals dissolve both problems.** Under ADR 005, `Sum` is a true - `AbelianGroup`, retraction is exact, the order of folding stops mattering (gotcha 18's - whole class), and the differential test is bit-exact everywhere. + can already be wrong today. That is recorded here for improvise-l2b8. +- **Only an exact representation dissolves the problem.** The next two subsections give + two: an exact accumulator for f64 sums, and ADR 005's decimals. -**So ADR 005 is a prerequisite for the maintained `Sum`/`Avg` on the displayed path, but -not for the rest of this ADR** (question 3). Occupancy, `Count`, `Min`, `Max`, -unanimity, stems and invalidation can all land before it. Until ADR 005 lands, a `Sum` -or `Avg` the grid displays, including a column total that holds one, is read from the -eager fold. The maintained sum runs only inside the differential test, under a stated -tolerance. If the user wants maintained sums before ADR 005, the fallback is a -compensated sum that re-folds a group from its values when cancellation is detected. -The measurement above is the evidence for and against that fallback. +#### Does DBSP itself help with floating point? No. + +The user asked whether rebuilding the evaluator on DBSP brings any correctness benefit +for floating point. On its own, **none**: + +- **Only the Z-set weights are exact.** They are integers (clause 1). The *values* being + summed are whatever type the aggregator holds. +- **An f64 `Sum` maintained by retraction is worse than eager, not better.** The table + above is the evidence: it drifts further from the exact sum the longer it runs, and + the absorption case loses the answer entirely. + +**The benefits come from two other places.** +- **The representation.** An exact accumulator (next subsection) or exact decimals + (ADR 005) make `Sum` a true abelian group. Then retraction is exact, incremental + equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent + `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes + with it. +- **The differential test** between `Eager` and `Incremental` (clause 2) is itself a + correctness gain. Two independent evaluations of every formula cell catch evaluator + bugs in either one, which nothing in the tree does today. + +What DBSP does contribute is **placement**. Every aggregate is a named operator with a +chosen algebra, so the circuit states exactly where values enter an exact domain and +where they leave it. That is the next subsection's principle. + +#### An exact accumulator for f64 sums, and deferring the lossy operations + +**A superaccumulator sums f64 values exactly.** It is a fixed-point integer wide enough +for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit integer shifted +by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's +xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. +Adding or retracting a value is integer addition, so the accumulator is a true abelian +group over the finite f64s. **Rounding happens once, when the value is read.** Then: + +- `Sum` is the exact sum, rounded once. It is independent of order. +- `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded + if the division is done on the exact value; dividing the rounded sum rounds twice, + and step 6 must say which it does. +- `Min` and `Max` are already exact. +- Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. + +**Measured**, in the same example, by `ExactSum` (147 lines, including its rounding): + +- **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` + maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over + the current values. That includes the absorption case, which reads exactly 10.00. + Against the exact *decimal* sum, it differs at 1 checkpoint of 25, by 9.1e-13. That + gap is the rounding of the typed decimals into f64, not the sum. +- **It is correct.** It was checked (not committed) against Python's `math.fsum` on + 20,000 random cases. The cases include subnormals, values near the overflow bound, + and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the + 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. +- **Cost, Apple M4 Max, best of 3 over 10M values:** + - about 3.2 ns per insert or retract, against about 2.3 ns for a plain f64 add in the + same loop; + - about 1.1 µs per read, which is the rounding, paid once per group read; + - **568 bytes per group**, against 8 for an f64. + + The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB + of accumulators. A sparse form that stores only the occupied span of limbs would be a + few limbs for typical data. **That is an estimate, not measured.** + +**No existing crate was usable.** `xsum` 0.1.6 (MIT, no dependencies, and it builds for +`wasm32-unknown-unknown`) was checked and rejected on three counts: + +- **It is wrong on a single negative value.** It rounds about a quarter of single + negative inputs to a neighbouring f64: 49,819 of 200,000 sampled values, all + negative, for example `-5890.88` read back as `-5890.880000000001`. +- It keeps one infinity and one NaN flag, not counts, so a non-finite value cannot be + retracted. +- Its accumulator is not `Clone`, and reading it takes `&mut self`. + +`accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not +evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and +a differential test against an exact rational sum. + +**Defer the lossy operations: the general principle.** Keep intermediate results in a +domain where the operation is exact, and apply the lossy operations at the edge: + +- sums go in the accumulator; +- decimal add, subtract and multiply are exact under ADR 005; +- division, `POW` with a non-integer exponent, and rounding for display happen only + where a value leaves the exact domain. + +This is honest only about where the gain comes from. **The representation earns it, not +DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named +operator with a chosen algebra, so "where does rounding happen" has one answer per +operator rather than one per call site. + +#### What each exact domain covers + +| | exact accumulator | ADR 005 decimals | +|---|---|---| +| `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | +| per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | +| typed values such as money | no | yes | +| a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | +| cost | 568 bytes per group, about 3.2 ns per update | bignum weight in the wasm bundle (improvise-l2b8's question 3) | + +**So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. +The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this +ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting +for ADR 005**, provided `Eager` uses the same accumulator. That is question 3's new +option, and its recommendation. ### 9. What ADR 008 gets from it @@ -507,7 +720,8 @@ ADR 008 (revision 3, PR #308) assumes these, under the shared names below. not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate, and the formula layer re-evaluates only the columns whose inputs moved, through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that - hold a `Sum` or `Avg` wait for ADR 005 (clause 8). + hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the + exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, or maintain) is struck there as decided by the user through this ADR. Its questions 1 and 2 stay open there. @@ -627,14 +841,16 @@ displayed path until the differential test holds. map that replaces `Expansion::members` (clause 5). Formula values that depend on a `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals, after ADR 005** (or on question 3's - fallback). A column total is a formula over the pooled fold, so it needs step 5's +6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact + accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is question 10. improvise-swre stays independent and is useful under either backend. -improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause 8. +improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is +already inexact beyond `FLOAT_EQ_EPSILON`. Under question 3's recommendation it is no +longer a prerequisite of step 6. **Good.** @@ -652,8 +868,12 @@ improvise-l2b8 (ADR 005) gains a dependant, and a measured argument, from clause - Two backends, kept equal by a differential test that must itself be kept honest. - Integrated state is O(cells) in memory, per distinct None axis, in addition to the store. -- `Sum` and `Avg` cannot be maintained on the displayed path until ADR 005 lands, and - the measurement says so. +- An f64 `Sum` cannot be maintained on the displayed path at all. Maintaining it needs + an exact representation: the accumulator costs 568 bytes per group (clause 8), or + ADR 005's decimals. +- Under question 3's recommendation, `Eager`'s displayed sums change in the last bits, + because they become correctly rounded. That is a behaviour change, even though it is + a fix. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. - A mutating frame from a client already clones the workbook for rollback @@ -677,13 +897,22 @@ the user decided", as ADRs 004 and 007 did. *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. **Floating point.** Should ADR 005 (exact decimals) be a prerequisite for maintaining - `Sum` and `Avg` on the displayed path, with the eager fold kept for them until then? - Or should a compensated f64 sum, re-folded on detected cancellation, ship first with - a tolerance in the differential test? - *Recommendation:* ADR 005 as the prerequisite. The measured drift reaches - `FLOAT_EQ_EPSILON`, so `=` could answer differently on the two paths. No f64 technique - makes the differential test bit-exact. Everything else in this ADR lands without it. +3. **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? + - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums + until then. + - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in + the differential test. + - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by + **both** backends, so that incremental equals eager bit for bit. + + *Recommendation:* (c), in `zset`. It is measured bit-equal after a million + retract-and-insert cycles, and exact against `math.fsum` on 20,000 cases. It costs + about 3.2 ns per update and 568 bytes per group, and it does not wait for ADR 005. + (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for + per-cell arithmetic and typed values, and it composes with (c): under ADR 005 a + decimal sum is exact on its own. The cost to accept is that today's displayed sums + change in their last bits, because they become correctly rounded, and that + `Avg`'s one rounding must be stated in step 6. 4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? @@ -727,3 +956,16 @@ the user decided", as ADRs 004 and 007 did. steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. + +11. **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). + Should `LAST`/`FIRST` key their per-group state by the ordering category's item + position, with that category's item order added to the operator shape so that a + reorder rebuilds them? Or should they key by stable item identity (ADR 007), and + look the order up at read time? + *Recommendation:* item position, with the order in the operator shape. Reads stay + O(log d), like `Min`/`Max`, and the same generic operator serves all four. An item + reorder is a rare structural edit, and clause 7 already rebuilds on those. This + extends the operator shape's `measure_agg` part to cover "and the item order of + each ordering category". ADR 008 clause 3 lists the parts as `view_shape`, group + membership and `measure_agg`. This reading changes what the `measure_agg` part + covers, not the list, so ADR 008's text and cross-references still hold. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 92dae6e5..26487b9e 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -1,5 +1,8 @@ //! How far a SUM maintained by insert/retract deltas drifts from the eager fold, -//! in f64. ADR 009 (improvise-felo.4), "Floating point". +//! in f64, and what an exact accumulator costs. ADR 009 (improvise-felo.4), clause 8. + +use std::hint::black_box; +use std::time::Instant; use improvise_core::format::format_f64; use improvise_core::model::cell::{CellKey, CellValue}; @@ -90,31 +93,196 @@ impl Compensated { } } +const LIMB_BITS: u32 = 32; +/// Every finite f64 is `mant * 2^(shift - 1074)` with `shift <= 2045` and `mant < 2^53`. +const LIMBS: usize = 67; + +/// An exact sum of f64s: a signed fixed-point integer in units of 2^-1074, wide enough +/// for every finite f64, with non-finite values counted so they can be retracted too. +#[derive(Clone)] +struct ExactSum { + limbs: [i64; LIMBS], + pending: u32, + nan: i64, + pos_inf: i64, + neg_inf: i64, +} + +impl ExactSum { + fn new() -> Self { + ExactSum { + limbs: [0; LIMBS], + pending: 0, + nan: 0, + pos_inf: 0, + neg_inf: 0, + } + } + + /// Add `x` with `weight` +1 (insert) or -1 (retract). Exact: no rounding happens here. + fn add(&mut self, x: f64, weight: i64) { + match (x.is_nan(), x.is_infinite()) { + (true, _) => self.nan += weight, + (false, true) if x > 0.0 => self.pos_inf += weight, + (false, true) => self.neg_inf += weight, + (false, false) => self.add_finite(x, weight), + } + } + + fn add_finite(&mut self, x: f64, weight: i64) { + let bits = x.to_bits(); + let biased = ((bits >> 52) & 0x7ff) as u32; + let fraction = bits & ((1 << 52) - 1); + let (mant, shift) = match biased { + 0 => (fraction, 0), + e => (fraction | (1 << 52), e - 1), + }; + let sign = if x.is_sign_negative() { + -weight + } else { + weight + }; + let wide = u128::from(mant) << (shift % LIMB_BITS); + let k = (shift / LIMB_BITS) as usize; + for i in 0..3 { + let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64; + self.limbs[k + i] += sign * part; + } + self.pending += 1; + if self.pending == 1 << 30 { + propagate(&mut self.limbs); + self.pending = 0; + } + } + + /// The exact sum rounded to the nearest f64, ties to even: the one rounding. + fn value(&self) -> f64 { + if self.nan > 0 || (self.pos_inf > 0 && self.neg_inf > 0) { + return f64::NAN; + } + if self.pos_inf > 0 { + return f64::INFINITY; + } + if self.neg_inf > 0 { + return f64::NEG_INFINITY; + } + let mut limbs = self.limbs; + propagate(&mut limbs); + let negative = limbs[LIMBS - 1] < 0; + if negative { + limbs.iter_mut().for_each(|l| *l = -*l); + propagate(&mut limbs); + } + let magnitude = round_to_f64(&digits(&limbs)); + match negative { + true => -magnitude, + false => magnitude, + } + } +} + +/// Carry each limb into the next, leaving every limb but the top in [0, 2^32). +fn propagate(limbs: &mut [i64; LIMBS]) { + for k in 0..LIMBS - 1 { + let carry = limbs[k] >> LIMB_BITS; + limbs[k] -= carry << LIMB_BITS; + limbs[k + 1] += carry; + } +} + +/// A non-negative normalized accumulator as 32-bit digits, the top limb split out. +fn digits(limbs: &[i64; LIMBS]) -> Vec { + let mut out: Vec = limbs.iter().map(|&l| l as u64).collect(); + let mut top = out.pop().unwrap_or(0); + loop { + out.push(top & 0xffff_ffff); + top >>= LIMB_BITS; + if top == 0 { + break; + } + } + out +} + +fn bit(d: &[u64], pos: usize) -> u64 { + (d[pos / 32] >> (pos % 32)) & 1 +} + +/// Round an exact multiple of 2^-1074 to the nearest f64, ties to even. +fn round_to_f64(d: &[u64]) -> f64 { + let Some(k) = d.iter().rposition(|&x| x != 0) else { + return 0.0; + }; + let msb = 32 * k + 63 - d[k].leading_zeros() as usize; + if msb < 53 { + let int = d[0] | (d[1] << 32); + return int as f64 * f64::from_bits(1); + } + let mut mant = (0..53).fold(0u64, |m, i| m | (bit(d, msb - 52 + i) << i)); + let round = bit(d, msb - 53) == 1; + let sticky = (0..msb - 53).any(|p| bit(d, p) == 1); + let mut top = msb; + if round && (sticky || mant & 1 == 1) { + mant += 1; + if mant == 1 << 53 { + mant >>= 1; + top += 1; + } + } + let exp = top as i64 - 52 - 1074; + if exp > 971 { + return f64::INFINITY; + } + let scale = match exp >= -1022 { + true => f64::from_bits(((exp + 1023) as u64) << 52), + false => f64::from_bits(1u64 << (exp + 1074)), + }; + mant as f64 * scale +} + +fn exact_over(values: &[i64]) -> f64 { + let mut acc = ExactSum::new(); + values.iter().for_each(|&v| acc.add(typed(v), 1)); + acc.value() +} + fn ulps(a: f64, b: f64) -> u64 { (a.to_bits() as i64).abs_diff(b.to_bits() as i64) } +struct Sums { + exact: f64, + eager: f64, + naive: f64, + comp: f64, + xs_maintained: f64, + xs_fresh: f64, +} + /// Each running sum against the eager fold (the differential oracle) and against the /// exact decimal sum. `flip`: the gap reaches `FLOAT_EQ_EPSILON`, so `=` can answer differently. -fn report(label: &str, cycles: usize, exact: f64, eager: f64, naive: f64, comp: f64) { +fn report(label: &str, cycles: usize, s: &Sums) { let shown = |v: f64| format_f64(v, true, 2); println!( - "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e}", - (naive - eager).abs(), - ulps(naive, eager), - (comp - eager).abs(), - ulps(comp, eager), - (naive - eager).abs() >= 1e-10, - (eager - exact).abs(), - (naive - exact).abs(), - (comp - exact).abs(), + "{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e} {:>7} {:>10.2e}", + (s.naive - s.eager).abs(), + ulps(s.naive, s.eager), + (s.comp - s.eager).abs(), + ulps(s.comp, s.eager), + (s.naive - s.eager).abs() >= 1e-10, + (s.eager - s.exact).abs(), + (s.naive - s.exact).abs(), + (s.comp - s.exact).abs(), + s.xs_maintained.to_bits() == s.xs_fresh.to_bits(), + (s.xs_maintained - s.exact).abs(), ); - if shown(naive) != shown(eager) || shown(eager) != shown(exact) { + if shown(s.naive) != shown(s.eager) || shown(s.eager) != shown(s.exact) { println!( - " shown at ,.2: naive {} eager {} exact {}", - shown(naive), - shown(eager), - shown(exact) + " shown at ,.2: naive {} eager {} exact {} maintained-exact-sum {}", + shown(s.naive), + shown(s.eager), + shown(s.exact), + shown(s.xs_maintained) ); } } @@ -126,6 +294,8 @@ fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) let mut naive = eager(&m); let mut comp = Compensated::default(); comp.add(naive); + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| xs.add(typed(v), 1)); let mut done = 0; for &at in checkpoints { while done < at { @@ -134,12 +304,21 @@ fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) naive = naive - typed(values[row]) + typed(new); comp.add(-typed(values[row])); comp.add(typed(new)); + xs.add(typed(values[row]), -1); + xs.add(typed(new), 1); values[row] = new; m.set_cell(key(row), CellValue::Number(typed(new))); done += 1; } - let label = format!("{cells} cells, {decades} dec"); - report(&label, at, exact(&values), eager(&m), naive, comp.value()); + let sums = Sums { + exact: exact(&values), + eager: eager(&m), + naive, + comp: comp.value(), + xs_maintained: xs.value(), + xs_fresh: exact_over(&values), + }; + report(&format!("{cells} cells, {decades} dec"), at, &sums); } } @@ -150,9 +329,11 @@ fn absorbed_by_a_large_value() { m.category_mut("Row").expect("just added").add_item("r0"); let mut naive = 0.0; let mut comp = Compensated::default(); + let mut xs = ExactSum::new(); m.set_cell(key(0), CellValue::Number(1e17)); naive += 1e17; comp.add(1e17); + xs.add(1e17, 1); for row in 1..=1000 { m.category_mut("Row") .expect("just added") @@ -160,23 +341,71 @@ fn absorbed_by_a_large_value() { m.set_cell(key(row), CellValue::Number(0.01)); naive += 0.01; comp.add(0.01); + xs.add(0.01, 1); } m.clear_cell(&key(0)); naive -= 1e17; comp.add(-1e17); - report( - "1e17, 1000 x 0.01, -1e17", - 1001, - 10.0, - eager(&m), + xs.add(1e17, -1); + let mut fresh = ExactSum::new(); + (0..1000).for_each(|_| fresh.add(0.01, 1)); + let sums = Sums { + exact: 10.0, + eager: eager(&m), naive, - comp.value(), + comp: comp.value(), + xs_maintained: xs.value(), + xs_fresh: fresh.value(), + }; + report("1e17, 1000 x 0.01, -1e17", 1001, &sums); +} + +/// Nanoseconds per operation over `n` values, best of three runs. +fn per_op(n: usize, mut op: impl FnMut() -> f64) -> f64 { + (0..3) + .map(|_| { + let t = Instant::now(); + black_box(op()); + t.elapsed().as_secs_f64() * 1e9 / n as f64 + }) + .fold(f64::INFINITY, f64::min) +} + +fn cost() { + let n = 10_000_000; + let mut rng = Rng(7); + let values: Vec = (0..n).map(|_| typed(decimal(&mut rng, 9))).collect(); + let plain = per_op(n, || { + values + .iter() + .fold(0.0, |s, &v| black_box(s + v) - black_box(v) + v) + }); + let exact_add = per_op(n, || { + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| { + xs.add(black_box(v), 1); + xs.add(black_box(v), -1); + xs.add(v, 1); + }); + xs.limbs[0] as f64 + }); + let mut xs = ExactSum::new(); + values.iter().for_each(|&v| xs.add(v, 1)); + let read = per_op(1, || xs.value()); + println!(); + println!("cost, {n} values, best of 3 (per op = one insert + one retract + one insert):"); + println!(" plain f64 add/sub {plain:>8.2} ns per op"); + println!(" exact accumulator {exact_add:>8.2} ns per op"); + println!(" exact read (round) {read:>8.0} ns per read"); + println!( + " state per group {} bytes, against 8 for an f64", + std::mem::size_of::() ); } fn main() { println!( - "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10}", + "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10} {:>7} {:>10}", "case", "cycles", "naive-eag", @@ -186,7 +415,9 @@ fn main() { "flip", "eag-exact", "nai-exact", - "cmp-exact" + "cmp-exact", + "xs=fresh", + "xs-exact" ); let checkpoints = [1_000, 10_000, 100_000, 1_000_000]; for seed in [1, 2] { @@ -195,4 +426,5 @@ fn main() { random_updates(seed, 18_000, 9, &checkpoints); } absorbed_by_a_large_value(); + cost(); } From 25ebf641a92299ade54f25be2892de88020cb645 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 22:13:53 -0700 Subject: [PATCH 07/67] docs(adr): ADR 009 clause 5a computes today's aggregate; bound ExactSum's weights (improvise-felo.4) - 5a: group by context (the stem projection, plus the fold/total projection), not by record key. The inner WHERE is a filter plus clause 5's join. There is an integrated per-record stage before the map, and a rule for several writes to one record. The order key is (context rank, record store order), so the pre-step-6 re-fold is bit-equal to eager at folds and totals. The first-Err error channel is kept. - ExactSum splits weights into chunks of 2^30 and counts magnitude between carries. The example checks weight 2^33, and 20,000 weighted cases match exact fractions. The example prints the cost per update. - Q3(c) states the signed-zero difference and Eager's per-read cost. Q11 is narrowed: drawn-item order is already in view_shape. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 215 ++++++++++++------ .../examples/retraction_drift.rs | 54 +++-- 2 files changed, 190 insertions(+), 79 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 5e9d4aa8..d85c62ab 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -351,9 +351,11 @@ key, and `FIRST` the least. the group, and one entry per contributing record. That is the same order as `Min`/`Max`'s multisets. - **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. So the ordering category's - item order must be part of the operator shape (clause 7), and a change to it rebuilds - the ordered aggregators that use it. The alternative is to key by a stable item + reordering its items, changes every stored position. **Most of that is already in the + operator shape**: `view_shape` hashes each category's *drawn* items, in order + (`app.rs:47-52`), so a reorder of drawn items already rebuilds. What it does not see + is the position of a **hidden** item, which matters only if `LAST`/`FIRST` must order + values that come from hidden items. The alternative is to key by a stable item identity (ADR 007) and look the order up at read time. That makes a reorder free, but a read then costs O(d). This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, @@ -418,54 +420,108 @@ group and nothing else. `SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per record**, `Revenue * Price` at that record, and pools those values across the records -of a group. Today `agg_values` (`types.rs:89-139`) lists the member records of each -context and calls the record-level evaluator on each (`member_records`, then -`eval_at`). At a fold or a column total, the contexts come from `Expansion::members` -(`types.rs:201-213`), which materializes the cross product of the spanned categories' -items. That is the one O(grid) path in totals (ADR 008 clause 5). +of a **context**. It is the one place a formula needs an expression *inside* an +operator. This clause first states exactly what today's evaluator computes, because the +incremental form must compute the same thing. -**The design.** It is a three-stage pipeline over the record stream, and each stage is -an operator from clause 1: +**What eager computes** (`eval_agg`, `types.rs:55-87`, and `agg_values`, `:89-139`): -1. **A lifted map over the per-record expression.** For each record in the cell - delta's `record_of` set, the map produces `(base key, expr at record)`. A record - whose cells changed is retracted at its old value and inserted at its new one, so - the map's output is itself a delta. `map` is linear, so this is its own incremental - form. -2. **`index_by(base key)`.** The base key is the record's key with the target category - removed, which is what `eval_agg` pools over (`types.rs:55-87`). An inner `WHERE` - (`SUM(expr WHERE Cat = Item)`) is a `filter` before the index. It is the same - rewrite `eval_agg` applies to its contexts (`types.rs:66-72`). -3. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the - mapped values, with `Avg` derived. +- **Contexts.** The formula cell's key is the context at a stem. At a fold or a column + total, the contexts are the members `contexts_of` lists through `Expansion::members` + (`types.rs:201-213`, `:905-916`), the cross product of the spanned categories' covered + items. That is the one O(grid) path in totals (ADR 008 clause 5). +- **An inner `WHERE` rewrites the context.** A filter `Cat = Item` *replaces* the + context's `Cat` coordinate (`c.with(&f.category, &f.item)`, `types.rs:69`, and again at + `:116`), and the distinct rewritten contexts are pooled once each. So the stem + `Region=West`, evaluating `SUM(x WHERE Region=East)`, pools **East's** records. +- **Pooling.** For each context, `partial = context.without(target_category)` + (`types.rs:97`). The context's member records are those `member_records` finds under + `partial`, in the store's record order (a record sits where its first cell sits). + The expression is evaluated at each, and the per-context value lists are concatenated + **in context order** (`types.rs:73-77`). So the order is (context rank, record store + order): expansion-major at a fold or total, and plain store order at a stem. +- **Errors.** `agg_values` collects `Result`s (`types.rs:126-136`), and so does + `eval_agg` (`:76`). So the aggregate's value is the **first** `Err` in that order, + if any member errs. -Member records never have to be listed, and no cross product is ever built. A context -in today's cross product that holds no records contributed nothing to the pool, so -pooling only the records that exist is the same answer. +**The design.** Four stages, each an operator from clause 1: -**This is not compiling formulas into operators.** The operator is the map; the -expression inside it is still evaluated by `eval_expr` at the record, exactly as -`agg_values` calls it today. The circuit decides *which* records to re-evaluate (those -the delta names), and the one evaluator computes each value. Clause 2's rule stands: -there is one evaluator. +1. **An integrated per-record stage.** A cell delta does not carry a record: when only + `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged + `Revenue`, so the old mapped value cannot be computed from the delta alone. So the + first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each + live record's cells, as `ZSet<(measure, value)>` per record. On each step it emits, + for every record the consolidated delta touches: + - a retraction of the record's **previous** snapshot, read from the integrated state + before the step (`I(x)₋₁`); + - an insertion of its **new** snapshot. + + **Several writes to one record in one effect** consolidate before this stage (clause + 3 steps once per model effect), so the record is retracted once and inserted once, + at its net state. Writes in separate effects are separate steps. A record whose + snapshot is unchanged after consolidation emits nothing. +2. **A lifted map over record snapshots.** For each snapshot, the map produces one + entry per context the record belongs to: `(context key, order key, expr at record)`. + The value is a `Result`, so an error is a mapped value like any other. `map` is linear, + so a retracted snapshot retracts exactly the entries it once produced. + - **The context key** is the record projected onto the context's shape: the target + category removed and each `none_cats` category removed, per `none_cats` list, + exactly as clause 4 defines "the group". At a fold or total, a collapsed group's + member is also mapped to its group, and each spanned category is removed. That is + the same projection occupancy uses (ADR 008 clause 3). + - **The order key** is (context rank, record store order). The context rank is the + record's position in `Expansion::members` order, and it is 0 at a stem. The record + store order is the insertion sequence of the record's first live cell. `DataStore` + does not keep one today, so this stage assigns it; `IndexMap`'s relative order is + exactly first-insertion order among live cells. `DataStore::sort_by_key` + (`cell.rs:269`, on entry into records mode) reorders the store, so it is an + operator-shape change and rebuilds this stage. +3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records + with `Cat = Item` pass the filter, and are indexed by their context key **without** + `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of + clause 5 (stems indexed by "stem minus `Cat`"). A filter alone would put East's + records in East's group and leave West's read empty; the join is what makes West's + stem read East's pool, as the rewrite does today. +4. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the + mapped `Ok` values, with `Avg` derived. **The error channel** is kept beside it: per + group, the set of erring entries, ordered by order key. The aggregate reads the + first of them if the set is non-empty, reproducing today's first-`Err`. + +Member records are never listed and no cross product is built. A context in today's +cross product that holds no records contributed nothing to the pool, so pooling only +the records that exist gives the same answer. + +**This is not compiling formulas into operators.** The operators are the record stage, +the map, the index and the aggregate. The expression inside the map is still evaluated +by `eval_expr` at the record, exactly as `agg_values` calls it today through `eval_at` +(`types.rs:127-134`). The circuit decides *which* records to re-evaluate (those the +delta names), and the one evaluator computes each value. Clause 2's rule stands: there +is one evaluator. **The boundary.** The map is sound only when `expr` reads the record's own cells, so that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. That formula stays on the eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. +same boundary. The regression that pins the record stage changes **one operand** of a +two-operand expression (`Price` alone, under `SUM(Revenue * Price)`) and compares with +eager. -**What stays eager until step 5, and until step 6.** -- Before step 5 (formulas), every aggregate over an expression is evaluated as today, - through `Expansion::members`. -- From step 5, the map and the index are maintained, so the cross product is gone. - `Count`, `Min` and `Max` over the mapped values are read from the maintained state, - since they are order-free and bit-exact on f64 (clause 8). -- `Sum` and `Avg` over the mapped values are **re-folded** from the group's maintained - values, in the store's record order, until step 6. That keeps them bit-equal to - eager, at a cost of O(group) per change rather than O(grid). -- From step 6 (on question 3's answer), `Sum` is maintained like any other. +**What stays eager until step 5, and what is bit-equal before step 6.** +- **Before step 5** (formulas), every aggregate over an expression is evaluated as + today, through `Expansion::members`. +- **From step 5**, the record stage, the map and the index are maintained, so the cross + product is gone. `Count`, `Min` and `Max` over the mapped values, and the error + channel, are read from the maintained state. They are order-free, or ordered by the + order key, and bit-exact on f64 (clause 8). +- **`Sum` and `Avg` over the mapped values are re-folded** from the group's maintained + values in **order-key order** until step 6. That is (context rank, record store + order), which is exactly the order eager concatenates in, so the re-fold is bit-equal + to eager at a stem, a fold and a total. It costs O(group log group) per change, + against O(grid). +- **From step 6** (on question 3's answer), `Sum` is maintained like any other. Under + question 3's recommendation (c) the order key no longer matters for `Sum`, because + the exact accumulator is order-free. ### 6. Circular references: DBSP's recursion is not our semantics @@ -620,7 +676,15 @@ for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit i by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. Adding or retracting a value is integer addition, so the accumulator is a true abelian -group over the finite f64s. **Rounding happens once, when the value is read.** Then: +group over the finite f64s. **Rounding happens once, when the value is read.** + +**The group claim needs one stated bound.** A consolidated Z-set carries weights, not +just ±1: equal mapped values consolidate, so one element can arrive with a large weight. +The accumulator therefore splits any weight into chunks of at most 2³⁰, and counts +*magnitude* between carry propagations rather than calls. Each limb then stays below +2⁶² between carries, for any `i64` weight. The first version counted calls, and review +found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is +`1.82e-304`. The example now checks that case, and it reads the exact answer. Then: - `Sum` is the exact sum, rounded once. It is independent of order. - `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded @@ -629,7 +693,7 @@ group over the finite f64s. **Rounding happens once, when the value is read.** T - `Min` and `Max` are already exact. - Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. -**Measured**, in the same example, by `ExactSum` (147 lines, including its rounding): +**Measured**, in the same example, by `ExactSum` (158 lines, including its rounding and the weight bound): - **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over @@ -639,13 +703,21 @@ group over the finite f64s. **Rounding happens once, when the value is read.** T - **It is correct.** It was checked (not committed) against Python's `math.fsum` on 20,000 random cases. The cases include subnormals, values near the overflow bound, and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the - 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. -- **Cost, Apple M4 Max, best of 3 over 10M values:** - - about 3.2 ns per insert or retract, against about 2.3 ns for a plain f64 add in the - same loop; - - about 1.1 µs per read, which is the rounding, paid once per group read; + 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. After + the weight bound, it was checked again, on 20,000 cases with weights up to 2⁴⁰ + against exact `Fraction` sums: 0 mismatches. The review's own oracle run agreed. +- **Cost, Apple M4 Max, best of 3 over 10M values.** The example prints per single + update. + - This revision's three runs agreed: 2.0 ns per insert or retract, against 1.5 ns + for a plain f64 add in the same black-boxed loop, and 0.58 µs per read. + - The previous revision's run on the same machine gave 3.2 against 2.3 ns and 1.1 µs, + and the review's run gave 3.6 against 2.7 ns. + - So an update costs roughly 1.3× a plain add. A read, which is the rounding paid + once per group read, costs 0.6–1.1 µs. - **568 bytes per group**, against 8 for an f64. + The ranges below are between runs on one kind of machine; read your own run. + The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB of accumulators. A sparse form that stores only the occupied span of limbs would be a few limbs for typical data. **That is an estimate, not measured.** @@ -686,7 +758,7 @@ operator rather than one per call site. | `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, about 3.2 ns per update | bignum weight in the wasm bundle (improvise-l2b8's question 3) | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | bignum weight in the wasm bundle (improvise-l2b8's question 3) | **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -837,10 +909,12 @@ displayed path until the differential test holds. step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the frame observation) depend on this step.** 5. **Formulas**: the formula graph and the formula delta (clause 7), topological - re-evaluation, the component fixed point (clause 6), and the aggregate-over-expression - map that replaces `Expansion::members` (clause 5). Formula values that depend on a - `Sum` or `Avg` read it from the eager fold until step 6. **ADR 008's retirement of - `Expansion::members` depends on this step.** + re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that + replaces `Expansion::members`: the integrated per-record stage, the map keyed by + context and order key, the inner-`WHERE` join, and the error channel. Its tests + include a change to one operand of a two-operand expression. Formula values that + depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR + 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** @@ -906,13 +980,20 @@ the user decided", as ADRs 004 and 007 did. **both** backends, so that incremental equals eager bit for bit. *Recommendation:* (c), in `zset`. It is measured bit-equal after a million - retract-and-insert cycles, and exact against `math.fsum` on 20,000 cases. It costs - about 3.2 ns per update and 568 bytes per group, and it does not wait for ADR 005. - (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for - per-cell arithmetic and typed values, and it composes with (c): under ADR 005 a - decimal sum is exact on its own. The cost to accept is that today's displayed sums - change in their last bits, because they become correctly rounded, and that - `Avg`'s one rounding must be stated in step 6. + retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and + its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and + typed values, and it composes with (c): under ADR 005 a decimal sum is exact on its + own. **The costs to accept:** + - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. + - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it + would use the same accumulator. A whole-grid eager projection pays it once per + summed cell. + - Today's displayed sums change in their last bits, because they become correctly + rounded. + - **Signed zero:** the accumulator reads `[−0.0]` as `+0.0`, where the f64 `sum()` + that `eval_agg` uses (`types.rs:79`) gives `−0.0`. + - `Avg`'s one rounding must be stated in step 6. 4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? @@ -962,10 +1043,12 @@ the user decided", as ADRs 004 and 007 did. position, with that category's item order added to the operator shape so that a reorder rebuilds them? Or should they key by stable item identity (ADR 007), and look the order up at read time? - *Recommendation:* item position, with the order in the operator shape. Reads stay - O(log d), like `Min`/`Max`, and the same generic operator serves all four. An item - reorder is a rare structural edit, and clause 7 already rebuilds on those. This - extends the operator shape's `measure_agg` part to cover "and the item order of - each ordering category". ADR 008 clause 3 lists the parts as `view_shape`, group - membership and `measure_agg`. This reading changes what the `measure_agg` part - covers, not the list, so ADR 008's text and cross-references still hold. + *Recommendation:* item position. Reads stay O(log d), like `Min`/`Max`, and the + same generic operator serves all four. The order of **drawn** items is already in + `view_shape` (`app.rs:47-52`), so a reorder of drawn items already rebuilds, and + nothing new is needed for them. The one open part is **hidden** items, whose + positions `view_shape` does not hash. Recommended: `LAST`/`FIRST` order only the + values the fold covers, which are the drawn members. Then the operator shape needs + no widening at all, and ADR 008's text and cross-references hold unchanged. If the + user wants hidden items ordered too, the `measure_agg` part must also cover their + positions. diff --git a/crates/improvise-core/examples/retraction_drift.rs b/crates/improvise-core/examples/retraction_drift.rs index 26487b9e..407a7ab5 100644 --- a/crates/improvise-core/examples/retraction_drift.rs +++ b/crates/improvise-core/examples/retraction_drift.rs @@ -96,13 +96,15 @@ impl Compensated { const LIMB_BITS: u32 = 32; /// Every finite f64 is `mant * 2^(shift - 1074)` with `shift <= 2045` and `mant < 2^53`. const LIMBS: usize = 67; +/// The most weight one limb absorbs between carries: each add moves a limb by < 2^32. +const CHUNK: i64 = 1 << 30; /// An exact sum of f64s: a signed fixed-point integer in units of 2^-1074, wide enough /// for every finite f64, with non-finite values counted so they can be retracted too. #[derive(Clone)] struct ExactSum { limbs: [i64; LIMBS], - pending: u32, + pending: u64, nan: i64, pos_inf: i64, neg_inf: i64, @@ -119,17 +121,30 @@ impl ExactSum { } } - /// Add `x` with `weight` +1 (insert) or -1 (retract). Exact: no rounding happens here. + /// Add `x` with any `weight` (a consolidated Z-set weight). Exact: no rounding here. fn add(&mut self, x: f64, weight: i64) { match (x.is_nan(), x.is_infinite()) { (true, _) => self.nan += weight, (false, true) if x > 0.0 => self.pos_inf += weight, (false, true) => self.neg_inf += weight, - (false, false) => self.add_finite(x, weight), + (false, false) => { + let mut left = weight; + while left != 0 { + let chunk = left.clamp(-CHUNK, CHUNK); + self.add_finite(x, chunk); + left -= chunk; + } + } } } + /// Bounded so each limb stays under 2^62: `|weight| <= CHUNK` and `pending` + /// counts magnitude, not calls. fn add_finite(&mut self, x: f64, weight: i64) { + if self.pending + weight.unsigned_abs() > CHUNK as u64 { + propagate(&mut self.limbs); + self.pending = 0; + } let bits = x.to_bits(); let biased = ((bits >> 52) & 0x7ff) as u32; let fraction = bits & ((1 << 52) - 1); @@ -148,11 +163,7 @@ impl ExactSum { let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64; self.limbs[k + i] += sign * part; } - self.pending += 1; - if self.pending == 1 << 30 { - propagate(&mut self.limbs); - self.pending = 0; - } + self.pending += weight.unsigned_abs(); } /// The exact sum rounded to the nearest f64, ties to even: the one rounding. @@ -375,12 +386,13 @@ fn cost() { let n = 10_000_000; let mut rng = Rng(7); let values: Vec = (0..n).map(|_| typed(decimal(&mut rng, 9))).collect(); - let plain = per_op(n, || { + let updates = 3 * n; + let plain = per_op(updates, || { values .iter() .fold(0.0, |s, &v| black_box(s + v) - black_box(v) + v) }); - let exact_add = per_op(n, || { + let exact_add = per_op(updates, || { let mut xs = ExactSum::new(); values.iter().for_each(|&v| { xs.add(black_box(v), 1); @@ -393,9 +405,9 @@ fn cost() { values.iter().for_each(|&v| xs.add(v, 1)); let read = per_op(1, || xs.value()); println!(); - println!("cost, {n} values, best of 3 (per op = one insert + one retract + one insert):"); - println!(" plain f64 add/sub {plain:>8.2} ns per op"); - println!(" exact accumulator {exact_add:>8.2} ns per op"); + println!("cost, {n} values x (insert, retract, insert), best of 3, per single update:"); + println!(" plain f64 add/sub {plain:>8.2} ns per update"); + println!(" exact accumulator {exact_add:>8.2} ns per update"); println!(" exact read (round) {read:>8.0} ns per read"); println!( " state per group {} bytes, against 8 for an f64", @@ -403,6 +415,21 @@ fn cost() { ); } +/// A consolidated Z-set weight can be large; the accumulator must stay exact at 2^33. +fn large_weight() { + let x = f64::from_bits((1u64 << 32) - 1); + let mut xs = ExactSum::new(); + xs.add(x, 1 << 33); + let want = ((1u64 << 32) - 1) as f64 * 2f64.powi(33) * f64::from_bits(1); + println!(); + println!( + "large weight: (2^32-1) x 2^-1074 at weight 2^33 reads {:e}, exact {:e}, equal {}", + xs.value(), + want, + xs.value().to_bits() == want.to_bits() + ); +} + fn main() { println!( "{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10} {:>7} {:>10}", @@ -426,5 +453,6 @@ fn main() { random_updates(seed, 18_000, 9, &checkpoints); } absorbed_by_a_large_value(); + large_weight(); cost(); } From 8ebcf889bdcb3f6976565870f4c4db90ef15d849 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Thu, 24 Sep 2026 23:02:00 -0700 Subject: [PATCH 08/67] docs(adr): ADR 009 matches ADR 005 revision 2 on exact decimal sums (improvise-felo.4) A decimal Sum is a group only through ADR 005's wide accumulator, used by both backends. rust_decimal's own + is not associative at its 28-digit bound. ExactSum does not make = against a typed decimal right: it can miss in the last bit. The decimal cost cell now uses ADR 005's measurement, and exact-to-f64 conversion goes through the decimal string. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 ++++++++++++++++++----- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index d85c62ab..b246be9b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -5,7 +5,7 @@ are theirs to answer on the PR. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8, no file yet), ADR 008 (the +**Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid, PR #308, which is revised to assume this ADR) · **Governed by:** `context/design-principles.md` §3 and §5 @@ -656,8 +656,12 @@ for floating point. On its own, **none**: the absorption case loses the answer entirely. **The benefits come from two other places.** -- **The representation.** An exact accumulator (next subsection) or exact decimals - (ADR 005) make `Sum` a true abelian group. Then retraction is exact, incremental +- **The representation.** An exact accumulator (next subsection), or ADR 005's + decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian + group, provided both backends use the same accumulator. Decimals alone do not: ADR + 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at + its 28-digit bound, so the order of summation changes both the value and the + exactness. Then retraction is exact, incremental equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes with it. @@ -740,9 +744,14 @@ a differential test against an exact rational sum. domain where the operation is exact, and apply the lossy operations at the edge: - sums go in the accumulator; -- decimal add, subtract and multiply are exact under ADR 005; +- decimal add, subtract and multiply are exact under ADR 005, within `rust_decimal`'s + 28-digit bound, which ADR 005 detects rather than silently rounds; - division, `POW` with a non-integer exponent, and rounding for display happen only - where a value leaves the exact domain. + where a value leaves the exact domain; +- **an exact value becomes an f64 through its decimal string and `str::parse::`**, + which is correctly rounded. `rust_decimal`'s `Decimal::to_f64` is not: ADR 005 + measured it disagreeing in 17,375 of 200,000 values. `ExactSum`'s own read is + correctly rounded by construction (clause 8's oracle checks). This is honest only about where the gain comes from. **The representation earns it, not DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named @@ -755,10 +764,25 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only for sums | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | **no**, in general: it sums the stored f64s exactly, and those are already rounded, so the result can miss the typed total in the last bit (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | bignum weight in the wasm bundle (improvise-l2b8's question 3) | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | + +**The accumulator does not make `=` against a typed decimal right.** It sums the +*stored* f64s exactly, and each stored f64 is already the rounded form of what the user +typed. So the exact sum of the stored values, rounded once, can differ in the last bit +from the typed total, rounded once. +- **ADR 005 revision 2 measured it** with this ADR's `ExactSum` as the stand-in, on + typed two-place values. The result missed the typed total in 21%, 16%, 7.5% and 2.5% + of 2-, 3-, 10- and 100-term sums, and in 0 of 8 seeds at 18k and 90k cells. +- **A spot check** (not committed) with two-place values up to ±1,000 missed in about + 38–40% of 2- to 100-term sums. +- **This ADR's own example** missed in 1 of 25 large-sum checkpoints. + +So the miss rate depends on the values' magnitudes, not only on the term count. The +misses are last-bit: they are visible to `=` and in the editor, not in a display +format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -983,8 +1007,10 @@ the user decided", as ADRs 004 and 007 did. retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and - typed values, and it composes with (c): under ADR 005 a decimal sum is exact on its - own. **The costs to accept:** + typed values, and it composes with (c): under ADR 005, a decimal sum is exact + through ADR 005's own wide accumulator (its question 10), provided both backends use + it, exactly as (c) requires for `ExactSum`. `rust_decimal`'s own `+` is not + associative at its bound, so decimals alone are not enough. **The costs to accept:** - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it would use the same accumulator. A whole-grid eager projection pays it once per From 09df35dffbe722a2ee775c24382fefbef45f3fb0 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:01:06 -0700 Subject: [PATCH 09/67] docs(adr): ADR 009 names ADR 005's stand-in correctly; the miss rate follows the total's size (improvise-felo.4) ADR 005 measured with the exact rational of the stored f64s, rounded once, not with ExactSum itself. The typed-total miss rate depends on how large the total is relative to its terms, not on the values' magnitudes. Both of ADR 005's distributions are cited. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b246be9b..3ed26294 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -773,14 +773,21 @@ operator rather than one per call site. *stored* f64s exactly, and each stored f64 is already the rounded form of what the user typed. So the exact sum of the stored values, rounded once, can differ in the last bit from the typed total, rounded once. -- **ADR 005 revision 2 measured it** with this ADR's `ExactSum` as the stand-in, on - typed two-place values. The result missed the typed total in 21%, 16%, 7.5% and 2.5% - of 2-, 3-, 10- and 100-term sums, and in 0 of 8 seeds at 18k and 90k cells. -- **A spot check** (not committed) with two-place values up to ±1,000 missed in about - 38–40% of 2- to 100-term sums. +- **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, + rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not + `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result + missed the typed total in 21%, 16%, 7.5% and 2.5% of 2-, 3-, 10- and 100-term sums, + and in 0 of 8 seeds at 18k and 90k cells. On two-place values uniform in ±1,000.00, it + missed in 38–40% at every one of those term counts. +- **A spot check** of this ADR (not committed), on the ±1,000 values, agreed: about + 37–40% of 2- to 100-term sums missed. - **This ADR's own example** missed in 1 of 25 large-sum checkpoints. -So the miss rate depends on the values' magnitudes, not only on the term count. The +So the miss rate depends on **how large the total is relative to its terms**, not on +the term count or on how large the values are. When the terms share a sign, the total +grows, and its last-bit step grows with it, so the terms' own rounding errors fall +below it and misses become rare. When the terms cancel, the total stays small, and +misses stay near 40%. The misses are last-bit: they are visible to `=` and in the editor, not in a display format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. From 284ac5b3e30281bb52b905493619d5a034d7bfac Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:06:54 -0700 Subject: [PATCH 10/67] docs(adr): ADR 009 hedges ExactSum's home on ADR 005 question 15 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 3ed26294..1012dd2c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -738,7 +738,9 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. +a differential test against an exact rational sum. Subject to ADR 005 question 15, +`ExactSum` may live in `numtower` instead and be wrapped by the seam crate, so that +`zset` stays free of numbers. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -1010,7 +1012,8 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c), in `zset`. It is measured bit-equal after a million + *Recommendation:* (c), in `zset`. Subject to ADR 005 question 15, `ExactSum` may + live in `numtower` instead and be wrapped by the seam crate. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and From b2c125918c024eae1ade3b2e234a3691418d895a Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 10:54:23 -0700 Subject: [PATCH 11/67] docs(adr): ADR 009 orders records by their current leading cell; qualify "visible to =" (improvise-felo.4) - 5a: a per-cell store sequence. A record's order key is the minimum over its live cells, carried in the snapshot, so deleting a leading cell re-emits the record as eager's shift_remove reorders it. A delete-leading-cell regression is added. Formulas with a non-_Measure target stay eager until 5a defines their projection. - A last-bit miss against a typed total is visible to = only above about 2^19, because = is an absolute 1e-10. The table row and paragraph now match ADR 005. - "not on the term count as such"; the numbering note says ADR 005 is on #320; ExactSum's home defers to ADR 005 question 15; the chunk loop's cost is noted. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 100 ++++++++++++++++------ 1 file changed, 74 insertions(+), 26 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1012dd2c..d60aec48 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -9,8 +9,8 @@ are theirs to answer on the PR. · virtualized grid, PR #308, which is revised to assume this ADR) · **Governed by:** `context/design-principles.md` §3 and §5 -**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl and have -no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 +is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are @@ -450,8 +450,9 @@ incremental form must compute the same thing. `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged `Revenue`, so the old mapped value cannot be computed from the delta alone. So the first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each - live record's cells, as `ZSet<(measure, value)>` per record. On each step it emits, - for every record the consolidated delta touches: + live record's cells, as `ZSet<(measure, value)>` per record, **each cell with its + store sequence** (below). On each step it emits, for every record the consolidated + delta touches, including a cell the expression never reads: - a retraction of the record's **previous** snapshot, read from the integrated state before the step (`I(x)₋₁`); - an insertion of its **new** snapshot. @@ -470,12 +471,30 @@ incremental form must compute the same thing. member is also mapped to its group, and each spanned category is removed. That is the same projection occupancy uses (ADR 008 clause 3). - **The order key** is (context rank, record store order). The context rank is the - record's position in `Expansion::members` order, and it is 0 at a stem. The record - store order is the insertion sequence of the record's first live cell. `DataStore` - does not keep one today, so this stage assigns it; `IndexMap`'s relative order is - exactly first-insertion order among live cells. `DataStore::sort_by_key` - (`cell.rs:269`, on entry into records mode) reorders the store, so it is an - operator-shape change and rebuilds this stage. + record's position in `Expansion::members` order, and it is 0 at a stem. + - **The record store order is the minimum store sequence over the record's *live* + cells**, recomputed every time the record is touched, and carried in the snapshot. + Eager orders records by their **current** earliest live cell in `DataStore::cells` + (`member_records`), and `shift_remove` changes that when a record's leading cell + goes. So the stage keeps a **per-cell** sequence, assigned when a key is first + inserted: + - a same-key retract and insert within one step is an in-place overwrite, so the + cell keeps its sequence, as `IndexMap::insert` does; + - a key inserted again after a removal in an earlier step gets a fresh sequence, at + the end, as `IndexMap` puts it; + - losing a record's leading cell changes its minimum, so the record is re-emitted + with its new order key. + + `IndexMap`'s relative order is exactly this sequence order among live cells. + `DataStore::sort_by_key` (`cell.rs:269`, on entry into records mode) reorders the + store, so it reassigns the sequences, and as an operator-shape change it rebuilds + this stage. + + The review demonstrated why a per-*record* sequence is not enough, against eager at + `57d385e1`. With `S = SUM(Rev * 1)` and store order `A.Other=5, B.Rev=1, + C.Rev=1e16, A.Rev=1, D.Rev=-1e16`, eager reads 2.0. Removing `A.Other`, a cell the + expression never reads, moves A behind C, and eager reads 0.0. A sequence fixed + when A first appeared would still read 2.0. 3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records with `Cat = Item` pass the filter, and are indexed by their context key **without** `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of @@ -503,9 +522,19 @@ that its value at a record changes only when that record changes. An inner expre that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. That formula stays on the eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. The regression that pins the record stage changes **one operand** of a -two-operand expression (`Price` alone, under `SUM(Revenue * Price)`) and compares with -eager. +same boundary. Two regressions pin the record stage against eager: +- one changes **one operand** of a two-operand expression (`Price` alone, under + `SUM(Revenue * Price)`); +- one **deletes a record's leading cell**, including a cell the expression never reads + (the `A.Other` case above), and checks both the value and which error is reported + first. + +**5a covers formulas whose target category is `_Measure`.** For a formula that targets +another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A record, +whose key `record_of` takes without `_Measure`, then belongs to the context only through +its cell under that measure, and eager orders it by *that* cell's position, not by the +record's first live cell. Until 5a defines that projection and its order key, such +formulas stay on the eager path, and the differential test pins that they do. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as @@ -688,7 +717,13 @@ The accumulator therefore splits any weight into chunks of at most 2³⁰, and c *magnitude* between carry propagations rather than calls. Each limb then stays below 2⁶² between carries, for any `i64` weight. The first version counted calls, and review found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is -`1.82e-304`. The example now checks that case, and it reads the exact answer. Then: +`1.82e-304`. The example now checks that case, and it reads the exact answer. + +The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight +near `i64::MAX`. The version in `zset` (or `numtower`) splits the weight into base-2³⁰ +digits instead and adds each as a shifted value. That is exact in the fixed-point +representation and O(1) per add. In practice a weight is bounded by a record count +anyway. Then: - `Sum` is the exact sum, rounded once. It is independent of order. - `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded @@ -738,9 +773,10 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. Subject to ADR 005 question 15, -`ExactSum` may live in `numtower` instead and be wrapped by the seam crate, so that -`zset` stays free of numbers. +a differential test against an exact rational sum. **Where it lives is ADR 005 +question 15's decision** (recommended there: a separate `numtower` crate, wrapped by +this ADR's seam crate so that `zset` stays free of numbers). This ADR defers to that +answer. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -766,7 +802,7 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | **no**, in general: it sums the stored f64s exactly, and those are already rounded, so the result can miss the typed total in the last bit (see below) | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | yes below about 5e5, no above: it sums the stored, already-rounded f64s exactly, so it can miss the typed total in the last bit, and `=`'s absolute 1e-10 absorbs that only while the total is below about 2¹⁹ (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | | cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | @@ -786,12 +822,23 @@ from the typed total, rounded once. - **This ADR's own example** missed in 1 of 25 large-sum checkpoints. So the miss rate depends on **how large the total is relative to its terms**, not on -the term count or on how large the values are. When the terms share a sign, the total +the term count as such, nor on how large the values are. When the terms share a sign, the total grows, and its last-bit step grows with it, so the terms' own rounding errors fall below it and misses become rare. When the terms cancel, the total stays small, and -misses stay near 40%. The -misses are last-bit: they are visible to `=` and in the editor, not in a display -format. Only exact decimal cells (ADR 005) make `=` against a typed total exact. +misses stay near 40%. + +**Where a miss is visible.** The misses are last-bit, so they show in the editor and +to a raw comparison, not in a display format. **They are visible to `=` only once the +total is above about 2¹⁹ ≈ 5.2e5**, because `=` is an absolute `1e-10` (gotcha 17), +and one ulp reaches it only at that magnitude. The same holds if ADR 005's epsilon-free +`=` is adopted (its question 3), since that has no tolerance at all. So: +- on the ±1,000 values, where misses are most frequent, the review counted 0 of 7,506, + 0 of 7,901 and 0 of 7,963 misses visible to `=` at 2, 10 and 100 terms; +- on positive values up to 999,999.99, most misses are visible, because the totals are + large. + +ADR 005 states the same condition. Only exact decimal cells (ADR 005) make `=` against +a typed total exact at every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this @@ -945,7 +992,8 @@ displayed path until the differential test holds. re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that replaces `Expansion::members`: the integrated per-record stage, the map keyed by context and order key, the inner-`WHERE` join, and the error channel. Its tests - include a change to one operand of a two-operand expression. Formula values that + include a change to one operand of a two-operand expression, and the deletion of a + record's leading cell. Formula values that depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact @@ -1012,9 +1060,9 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c), in `zset`. Subject to ADR 005 question 15, `ExactSum` may - live in `numtower` instead and be wrapped by the seam crate. It is measured bit-equal after a million - retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + *Recommendation:* (c). Where `ExactSum` lives is ADR 005 question 15's decision + (recommended there: `numtower`, wrapped by the seam crate), and this ADR defers to + it. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and typed values, and it composes with (c): under ADR 005, a decimal sum is exact From 4973845fda79f818da7a4ef6266b7760c89cc6f2 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 12:03:43 -0700 Subject: [PATCH 12/67] docs(adr): ADR 009 states =-visibility on the terms, not the total; sequences come from the store (improvise-felo.4) - A miss against a typed total is bounded by the terms' rounding, about half an ulp of each term, so it is last-bit relative to the largest term. It can reach = only when the terms reach about 2^19. The large-term, small-total case (1e15 + 0.10 - 1e15 reads 0.125) is added. The epsilon-free = shows every miss. - 5a: cell sequences are assigned at DataStore's mutation sites and carried in each delta element, so a remove-then-set of one key within an effect moves it to the end, as IndexMap does. The generator includes that case. The record tier projects to (key, value). Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 62 ++++++++++++++++------- 1 file changed, 44 insertions(+), 18 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index d60aec48..b4e6ccdd 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -478,10 +478,19 @@ incremental form must compute the same thing. (`member_records`), and `shift_remove` changes that when a record's leading cell goes. So the stage keeps a **per-cell** sequence, assigned when a key is first inserted: - - a same-key retract and insert within one step is an in-place overwrite, so the - cell keeps its sequence, as `IndexMap::insert` does; - - a key inserted again after a removal in an earlier step gets a fresh sequence, at - the end, as `IndexMap` puts it; + - **the sequence is assigned where `IndexMap` assigns position, at `DataStore`'s + own mutation sites**, and each cell-delta element carries it. An overwrite + (`set` on a live key) keeps the key's sequence, as `IndexMap::insert` does. A + `remove` followed by a `set` of the same key moves the key to the end, even + within one effect, and gets a fresh sequence. Because the element carries the + sequence, the two cases stay distinct after consolidation: an overwrite is + `−(k, v₀, s) + (k, v₁, s)`, and a remove-then-set is + `−(k, v₀, s) + (k, v₁, s′)`. A stage that inferred the sequence from a + consolidated `(key, value)` delta could not tell them apart. The differential + test's generator includes a same-key remove-then-set within one effect; + - the record tier (clause 3) projects the delta to `(key, value)` before it + consolidates, so moving a cell without changing its value still bumps nothing, + as today's fingerprint does; - losing a record's leading cell changes its minimum, so the record is re-emitted with its new order key. @@ -802,15 +811,16 @@ operator rather than one per call site. |---|---|---| | `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | | per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | yes below about 5e5, no above: it sums the stored, already-rounded f64s exactly, so it can miss the typed total in the last bit, and `=`'s absolute 1e-10 absorbs that only while the total is below about 2¹⁹ (see below) | yes, exactly | +| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | | cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | **The accumulator does not make `=` against a typed decimal right.** It sums the *stored* f64s exactly, and each stored f64 is already the rounded form of what the user -typed. So the exact sum of the stored values, rounded once, can differ in the last bit -from the typed total, rounded once. +typed. So the exact sum of the stored values, rounded once, can differ from the typed +total, rounded once, by up to about half an ulp of each term. That is the last bit of +the largest term, which is more than the last bit of the total when the terms cancel. - **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result @@ -827,18 +837,34 @@ grows, and its last-bit step grows with it, so the terms' own rounding errors fa below it and misses become rare. When the terms cancel, the total stays small, and misses stay near 40%. -**Where a miss is visible.** The misses are last-bit, so they show in the editor and -to a raw comparison, not in a display format. **They are visible to `=` only once the -total is above about 2¹⁹ ≈ 5.2e5**, because `=` is an absolute `1e-10` (gotcha 17), -and one ulp reaches it only at that magnitude. The same holds if ADR 005's epsilon-free -`=` is adopted (its question 3), since that has no tolerance at all. So: -- on the ±1,000 values, where misses are most frequent, the review counted 0 of 7,506, - 0 of 7,901 and 0 of 7,963 misses visible to `=` at 2, 10 and 100 terms; -- on positive values up to 999,999.99, most misses are visible, because the totals are - large. +**Where a miss is visible.** A miss is bounded by the **terms'** rounding, not by the +total's. Each stored term is off from its typed value by at most half an ulp of that +term, so the exact sum of the stored terms is off from the typed total by at most about +½·Σ ulp(term), plus the total's own half-ulp at the final rounding. So the misses are +last-bit **relative to the largest term**, not relative to the total. When terms cancel, +a miss can be far larger than the total's ulp. -ADR 005 states the same condition. Only exact decimal cells (ADR 005) make `=` against -a typed total exact at every magnitude. +- **The large-term, small-total case (from ChatGPT Codex on this PR).** The terms are + `1000000000000000.10 + (-1000000000000000.00)`, and the typed total is 0.10. The + first term is stored as `1e15 + 0.125`, because the ulp at 1e15 is 0.125. So the + exact sum of the stored terms is **0.125**. The miss is 0.025, on a total of 0.1, and + `=` sees it. +- **The condition, stated on the terms.** `=` is an absolute `1e-10` (gotcha 17), so a + miss can reach it only when ½·Σ ulp(term) reaches `1e-10`. That needs at least one + term of about 2¹⁹ ≈ 5.2e5 or larger, or many terms just below it. It does not depend + on how large the total is. +- **The measured cases fit the condition.** On the ±1,000 values every term is far + below 2¹⁹, and the review counted 0 of 7,506, 0 of 7,901 and 0 of 7,963 misses + visible to `=` at 2, 10 and 100 terms. On positive values up to 999,999.99 the terms + themselves pass 2¹⁹, and most misses are visible. +- **Under ADR 005's epsilon-free `=`** (its question 3), every miss is visible, since + there is no tolerance at all. + +The misses show in the editor and to a raw comparison, and not in a display format +unless the terms are large enough, as in the case above. ADR 005's "once the magnitude +is above 2¹⁹" needs the same correction, to the terms' magnitude, and its author has +been told. Only exact decimal cells (ADR 005) make `=` against a typed total exact at +every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this From 8d520d038815f731f8f579ade3d534348e66d0a0 Mon Sep 17 00:00:00 2001 From: "Claude (agent-felo4)" Date: Fri, 25 Sep 2026 12:12:00 -0700 Subject: [PATCH 13/67] docs(adr): ExactSum lives in numtower, as the user decided on ADR 005 question 15 (improvise-felo.4) The user answered (a) on PR #320. ADR 009's seam crate wraps ExactSum for zset's trait, so zset stays free of numbers. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b4e6ccdd..fe3efbd4 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -729,7 +729,7 @@ found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer `1.82e-304`. The example now checks that case, and it reads the exact answer. The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight -near `i64::MAX`. The version in `zset` (or `numtower`) splits the weight into base-2³⁰ +near `i64::MAX`. The version in `numtower` splits the weight into base-2³⁰ digits instead and adds each as a shifted value. That is exact in the fixed-point representation and O(1) per add. In practice a weight is bounded by a record count anyway. Then: @@ -781,11 +781,11 @@ anyway. Then: - Its accumulator is not `Clone`, and reading it takes `&mut self`. `accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not -evaluated. So the accumulator is about 150 lines inside `zset`, with its own tests and -a differential test against an exact rational sum. **Where it lives is ADR 005 -question 15's decision** (recommended there: a separate `numtower` crate, wrapped by -this ADR's seam crate so that `zset` stays free of numbers). This ADR defers to that -answer. +evaluated. So the accumulator is about 150 lines of our own, with its own tests and a +differential test against an exact rational sum. **It lives in `numtower`, ADR 005's +numeric crate.** The user decided that on ADR 005 question 15, answering "(a)" on PR +#320 on 2026-09-25. This ADR's seam crate wraps it in a newtype for `zset`'s group +trait, so `zset` stays free of numbers. **Defer the lossy operations: the general principle.** Keep intermediate results in a domain where the operation is exact, and apply the lossy operations at the edge: @@ -921,8 +921,8 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 11. Constraints -- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset` and - `improvise-core`. If improvise-command selects the backend, both join the browser +- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, + `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, both join the browser bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). - **Two seconds per test binary.** The crate's laws and its "incremental equals @@ -1086,9 +1086,9 @@ the user decided", as ADRs 004 and 007 did. - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by **both** backends, so that incremental equals eager bit for bit. - *Recommendation:* (c). Where `ExactSum` lives is ADR 005 question 15's decision - (recommended there: `numtower`, wrapped by the seam crate), and this ADR defers to - it. It is measured bit-equal after a million retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with + *Recommendation:* (c). `ExactSum` lives in `numtower`, wrapped by the seam crate, + as the user decided on ADR 005 question 15. It is measured bit-equal after a million + retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and typed values, and it composes with (c): under ADR 005, a decimal sum is exact From d766605b304e293a7177644b33c1208417895472 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-fix)" Date: Fri, 2 Oct 2026 15:51:27 -0700 Subject: [PATCH 14/67] docs(adr): the cell delta carries the store sequence; two stale statements fixed (improvise-felo.4) Clause 3 defines the canonical element with Seq, derives the (key, value) projection for the record tier, and states bit identity over that part only. Clause 11 names all three crates; the 2^19 note is past tense. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++-------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index fe3efbd4..711c4e36 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -225,21 +225,32 @@ and compares every drawn cell, every column total and every occupancy count. ### 3. The input: a cell write is a retraction plus an insertion -**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue)>` with weights -±1.** `DataStore::set` and `DataStore::remove` (`cell.rs:286`, `:325`) are the only -places a cell changes, so the delta is emitted there: +**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue, Seq)>` with +weights ±1, where `Seq` is the cell's store sequence (clause 5a).** `DataStore::set` and +`DataStore::remove` (`cell.rs:286`, `:325`) are the only places a cell changes, so the +delta is emitted there, each element carrying the sequence `s` of the cell it names: -- a new cell is `+(k, v)`; -- an overwrite of `v₀` with `v₁` is `−(k, v₀) + (k, v₁)`; -- a removal is `−(k, v₀)`. +- a new cell is `+(k, v, s)`, with a fresh `s`; +- an overwrite of `v₀` with `v₁` is `−(k, v₀, s) + (k, v₁, s)`, the key keeping its `s`; +- a removal is `−(k, v₀, s)`; +- a remove-then-set within one effect is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. -An overwrite with an equal value consolidates to the empty Z-set. The circuit steps once +**The `(key, value)` projection is derived, not a second stream.** The record tier +and any consumer that does not need order read the delta mapped through +`(k, v, s) ↦ (k, v)` and consolidated, a `ZSet<(CellKey, CellValue)>`. Only clause 5a's +stage reads `s`. Defining the canonical element without `s` would strip it, and a +same-value remove-then-set would cancel, defeating 5a. + +An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty +Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). **A delta key's identity is bit identity, the same identity the record tier uses -today.** Two `(CellKey, CellValue)` pairs are one element exactly when their keys are -equal and their values are equal *by bits*: `Number` by `f64::to_bits`, `Text` and +today.** Two elements have the same `(key, value)` part exactly when their keys are +equal and their values are equal *by bits*, and bit identity is stated over that part +only (the sequence is compared as the integer it is, which keeps an +equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and `Error` by their strings. That is `hash_value` (`app.rs:233-250`), which `record_fingerprints` already hashes. So the cell delta reproduces today's tier exactly: @@ -862,8 +873,8 @@ a miss can be far larger than the total's ulp. The misses show in the editor and to a raw comparison, and not in a display format unless the terms are large enough, as in the case above. ADR 005's "once the magnitude -is above 2¹⁹" needs the same correction, to the terms' magnitude, and its author has -been told. Only exact decimal cells (ADR 005) make `=` against a typed total exact at +is above 2¹⁹" needed the same correction, to the terms' magnitude, and ADR 005 has +made it. Only exact decimal cells (ADR 005) make `=` against a typed total exact at every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. @@ -922,7 +933,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 11. Constraints - **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, - `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, both join the browser + `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, all three join the browser bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). - **Two seconds per test binary.** The crate's laws and its "incremental equals From 7878decbaa3ccdf00eb5af3869597a4ac13d0166 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-fix2)" Date: Fri, 2 Oct 2026 17:12:57 -0700 Subject: [PATCH 15/67] docs(adr): dirty flag reads the projected cell delta (improvise-felo.4) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 711c4e36..f76ff26c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -286,7 +286,8 @@ is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a non-empty cell delta means dirty" would never save a formula edit in the browser. The -content half becomes: **a non-empty cell delta, or a moved formula-set fingerprint**. +content half becomes: **a non-empty *projected* cell delta (the `(k, v)` projection, so a same-value +remove-then-set raises nothing), or a moved formula-set fingerprint**. The fingerprint is the one clause 7 already reads after each model effect, so nothing new is observed. Its cost is O(formulas), where `document_content` is O(cells) per `CellData` effect. @@ -1016,7 +1017,7 @@ displayed path until the differential test holds. existing suite is the evidence. `CellKey` gains `Ord`. 3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half - becomes "a non-empty cell delta, or a moved formula-set fingerprint" (clause 3), with + becomes "a non-empty projected cell delta, or a moved formula-set fingerprint" (clause 3), with a test that a formula edit alone still marks the document dirty. That removes the O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with From a9bcb816e9d3d44c40982fa41b6ac723a1126fd0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:54:30 -0700 Subject: [PATCH 16/67] docs(adr): ADR 009 revised to fit ADR 008's view/host split (improvise-felo.4) Adds the user's ADR 008 statements that bear on this ADR, clauses 12-14 (network transparency, incremental max width, resident state), keys operator state by projection shape at item grain, retires the per-mirror delivery assumptions, and appends questions 12-21. Questions 1-11 stay open. Reads ADR 008 at 69f0a7c6. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 786 ++++++++++++++++++++-- 1 file changed, 714 insertions(+), 72 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index f76ff26c..eaa01619 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,12 +1,19 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. · +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 19** +(PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in +the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details +ADR 008 routes here. They are in the section "What ADR 008's review established for this +ADR", in clauses 12 to 14, and in questions 12 to 18. **The user has answered none of +this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline +comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the -virtualized grid, PR #308, which is revised to assume this ADR) · +virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision +reads it at `69f0a7c6`, its revision 19) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -14,8 +21,13 @@ is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #3 no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are -dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means -revision 3 of `context/adr/008-virtualized-grid.md` on PR #308. +dated, so re-derive them before citing this document elsewhere. **The 2026-10-05 +revision's new citations are at `gh/main` `50e1314d`**, and each says so. The older +citations were not re-derived, and some have moved. For example, `Host::mutating_frame` +takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and +`App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means +revision 19 of `context/adr/008-virtualized-grid.md` on PR #308, at `69f0a7c6`. Clauses 3, +5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -61,7 +73,123 @@ from scratch. The machinery is: **The grid is the second consumer.** ADR 008 needs the same maintained state for other purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, and column totals over rows never scrolled to (clause 9). It gets them from the same -circuit, not from a second mechanism. +circuit, not from a second mechanism. Since ADR 008's view/host split, the grid's consumer +may be in another process, and clause 12 says how the same circuit serves both. + +## What ADR 008's review established for this ADR + +At its revisions 18 and 19, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +user routed part of it here. This section quotes what the user said that bears on this ADR, +and says where each statement came from. +- **PR comments** are cited by their GitHub comment id and UTC time. The question numbers + are ADR 008's, as ADR 008 assigns them. +- **The two chat statements** were given to the orchestrator and relayed to this ADR. They + are on no PR, and ADR 008 records them the same way. The user's "Approved" on this ADR + would confirm them. + +None of this answers this ADR's own questions 1 to 11. + +**Routed here, in chat:** +- "The DBSP machinery needs to be implemented in a network transparent way, but the + details of that go to ADR 009" (chat, relayed on 2026-10-05). Clause 12. +- "Thinking about it a bit more, we might need to keep track of recent projections + + clients to stream aggregation results back to the UI correctly. But this should be as + stateless as possible" (chat, relayed on 2026-10-05). Clause 12. +- On ADR 008's question 53: "Q53 is cross product: I'm still ambivalent about which side + of the wire pruning should happen on" (chat, relayed on 2026-10-05). So a request's + top-left is a cross-product position, and which side prunes is open. ADR 008 records it + at revision 19 as its question 53 answered and its question 57 open. Clause 12, + question 19. +- "But the top left should be as absolute as possible" (chat, relayed on 2026-10-05). + Clause 12, question 20 (ADR 008's question 58). +- "After pruning we might have to stream rows and columns to attempt to fill out the full + grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). + +**Asked on ADR 008, and not yet answered:** +- "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, + 2026-10-03T03:15:38Z, on ADR 008's question 2). Clause 13 is this ADR's reading of it. + Question 12. + +**The read side and the write side** (#308, inline comments of 2026-10-05): +- Q43 (4189071865, 21:34:45Z): "yeah, the read side of the model is a projection of the + model's full space into a particular set of axes using a `project` verb, with + offset+dimensions. The write side is an apply function that can only take model + commands and applies them to the model using Redux-style Flux architecture." +- Q38 (4189007864, 21:27:55Z): "No, axes are a model concept because they define the + grid. Aggregation should always be done model-side for consistency." +- Q38 (4189027924, 21:30:08Z): "A grid request would look something like:" + ``` + Axes: Row(A,B,C), Col(D,E), Page(F), None(G,H) + Top-left (4,5) # relative to the cross-product of the axes in the category-native order (which might be arbitrary) + Dimensions (10,10) # so get ten rows and ten columns worth of data from the grid defined by the axes. + ``` +- Q33 (4188936570, 21:20:11Z): "The host shouldn't have to know anything about the + viewport: it's mostly a question of the parameters of the grid the user is actually + looking at, rather than a "viewport" strictly speaking." +- Q37 (4189000059, 21:27:04Z): "I don't think the host needs to no [sic] anything about + the UI: the initial load of the UI should just be the default view stored in the model + (or, in some cases, a particularly requested view). Sending UI state over the wire is + an architecture smell." +- Q42 (4189051110, 21:32:31Z): "The space scrolling happens in is defined by the axes, top + left corner offset and with/height. But how much of this space is actually materialized + and when fetches, etc. happens (and model values are cached) is client-policy driven." +- Q32 (4188917796, 21:18:17Z): "I can't think of any cursor command with model-side + effects." +- Q27 (4188816383, 21:07:43Z): "go with the recommendation: the ordered list of records is + a model-side concept." +- Q31 (4188911659, 21:17:39Z): "You have to distinguish view in the sense of UI and view + in the sense written into `.improv` files. P has to be a model-side effect because it + prunes data for a particular view. It is not a UI-only command." And (4188926877, + 21:19:10Z): "The only real issue here is that pruning shouldn't infect every client + here. It might need to be UI-only until a view is saved, and then the view definition + goes to the model." +- Q29 (4188887721, 21:15:14Z): "Prune should only be written if the user explicitly + unpruned the view." +- The cursor: "The cursor should never jump: it should stay where the user left it as much + as possible." (Q12, 4171501445, 2026-10-03T03:21:05Z). And "As far as possible, updates + should shift the grid being viewed to keep the cursor in the same visual place on the + screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). + +**Answers ADR 008 records that this ADR's text had not caught up with:** +- Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. +- Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases + where the column size has been stably smaller." And Q22 (4188309022, + 2026-10-05T20:15:12Z): "hysteresis as recommended." Clauses 9 and 13. +- Q6 (4171493762, 2026-10-03T03:18:12Z): "maintained incrementally". Clause 9. +- Q13, Q14 and Q15: "the recommendation is fine.", three comments (4171504439, + 4171505979 and 4171509012, 2026-10-03T03:22:16Z to 03:23:36Z), which ADR 008 records as + its answers to those three questions. Q13 sets the frame budget, and Q15 sends the + formulas the per-record map cannot express to the occupied contexts. Clauses 5a and 14. +- The store's memory (4171481179, 2026-10-03T03:13:14Z): "We have to be careful here to + minimize memory requirements of the store: offloading inactive regions to disk (maybe + mmap or similar), double-delta/gorilla/lz4 compression, etc." And Q17 (4188271710, + 2026-10-05T20:11:02Z): "Let's defer this as recommended." Clause 14. +- Jobs (4171458281, 2026-10-03T03:04:52Z): "We also have to have a way to + invalidate/cancel in-flight jobs efficiently and probably a way to throttle them (a + priority queue or similar) to avoid pinning the CPU." Then (4171694278, 04:30:31Z): + "priority queue for work stealing is also important here." Clause 7. + +**What this ADR reads from them.** These are the author's readings, and each one names +the clause it changes: +1. **Reads and writes go through `project` and `apply`.** A consumer reads the circuit's + outputs through `project`, and the circuit's inputs change only through `apply` of + model commands. So nothing here may depend on a host-held mirror of a client + (`Sessions`, `ClientSession`, `reduce_view` on the host), on the per-mirror diff + (`grid_updates`), or on resetting a client's cursor. Clauses 9 and 12. +2. **Aggregation stays on the model side.** No client runs an aggregate operator, so + network transparency concerns the circuit's outputs, and a live source's inputs. It + does not mean splitting operators across a wire. Clause 12. +3. **State is keyed by the request's shape, not by a view's name.** Axes are a model + concept, so operator state keyed by axes is model state, not UI state. But a `project` + request names its own axes, and an unsaved request may carry parameters no saved view + holds (ADR 008's flagged reading 1, its question 51). Clause 7, question 14. +4. **The record order belongs to the store.** The ordered list of records is the store's + (Q27). The circuit may consume it, and does not own it. Clause 9. +5. **The memory requirement covers this ADR too.** The store-memory requirement covers the + circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. + Clause 14, question 17. +6. **Five of ADR 008's questions are answered.** Those are questions 1, 2, 6, 13 and 15, + and clauses 5a, 9, 13 and 14 now follow the answers. ## Context @@ -133,9 +261,9 @@ whose inputs moved, over aggregates that are maintained rather than re-folded. T machinery is a new, problem-agnostic workspace crate implementing the DBSP core. improvise-core defines a trait for the derived state its evaluator reads. Today's evaluator is one implementation of that trait, and a circuit built on the new crate is -the other. Both stay in the tree, and a differential test holds them equal.** Eleven +the other. Both stay in the tree, and a differential test holds them equal.** Fourteen clauses, plus clause 5a. It is numbered 5a so that no clause number or question number -cited elsewhere moves. +cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. ### 1. The crate: `zset` (name is question 1) @@ -179,8 +307,12 @@ Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** bey improvise's circular semantics are not a least fixed point (clause 6). - **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies batches in one total order (ADR 001, "There is already a total order"). -- **Persistence, spilling to disk, and threads.** State lives in memory, and a browser - worker has one thread. +- **Spilling to disk, compression, and threads, in the first version.** State lives in + memory, and a browser worker has one thread. **This was a non-goal and is now only a + deferral.** The user's store-memory requirement (ADR 008 clause 12) covers this state + too, and clause 14 says what the crate must not rule out. A host with threads may run + independent circuits on a pool (ADR 008 question 23's answer). One circuit's `step` + stays single-threaded. - **A query language or planner.** Circuits are wired by hand in the seam crate (clause 2). @@ -244,7 +376,9 @@ same-value remove-then-set would cancel, defeating 5a. An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy -inside the effect loop (`app.rs:1073-1084`). +inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation +under the view/host split. The per-effect step stays, because each step is what clause +12 stamps. **A delta key's identity is bit identity, the same identity the record tier uses today.** Two elements have the same `(key, value)` part exactly when their keys are @@ -301,7 +435,7 @@ new is observed. Its cost is O(formulas), where `document_content` is O(cells) p **The group** is the cell key with the view's `none_cats` removed. That is the partial key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct -`none_cats` list, shared by every view that has that list. That replaces +`none_cats` list, shared by every projection shape that has that list (clause 7). That replaces `rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does (`types.rs:717-722`). @@ -541,9 +675,16 @@ is one evaluator. **The boundary.** The map is sound only when `expr` reads the record's own cells, so that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value -depends on other records, is not a function of one record. That formula stays on the -eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the -same boundary. Two regressions pin the record stage against eager: +depends on other records, is not a function of one record. **Revised on 2026-10-05:** +such a formula does **not** stay on today's eager path, because that path is +`Expansion::members`, the cross product (ADR 008 clause 5, a finding of Copilot's on +#308). It takes ADR 008's **store-driven fallback**, which evaluates the expression only +over the **occupied contexts**, the contexts some record projects to. The user accepted +that recommendation, as ADR 008's question 15 ("the recommendation is fine."). It differs +from today's eager answer wherever the expression is non-empty at a context that holds no +record, and ADR 008 carves those cases out of its differential test by name. Whether this +ADR's `Eager` adopts the same rule, so that its own differential test needs no carve-out, +is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -555,7 +696,10 @@ another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A r whose key `record_of` takes without `_Measure`, then belongs to the context only through its cell under that measure, and eager orders it by *that* cell's position, not by the record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. +formulas stay on the eager path, and the differential test pins that they do. **That +eager path is `Expansion::members`, so while it lasts it blocks ADR 008's retirement of +`Expansion::members` (its migration step 6) for these formulas.** Sending them to the +occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as @@ -610,13 +754,58 @@ post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1 That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 states the same rule. -- **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`), - plus the group membership of every drawn item, collapsed or not, plus - `measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is - unchanged. A view whose operator shape moved has its operators rebuilt from the - integrated base Z-set **before the frame observation that follows this effect**. - The `LayoutGeometry` frame arm (`app.rs:1062-1071`) reads occupancy right after the - effect that moved the shape, so the next effect's cell delta is too late. +- **Operator shape, per projection shape (revised on 2026-10-05).** Operator state is + keyed by the **operator shape of a `project` request**, not by the name of a view + (question 14). The shape is made of: + - the categories of each axis, by level; + - the None categories; + - the filter and page keys; + - the hidden items and collapsed groups the request carries; + - `measure_agg`. + + A saved `.improv` view supplies these to a request that names it. An unsaved request + carries its own, for example an unsaved prune or hide (ADR 008's flagged reading 1, its + question 51). Requests with the same shape share one set of operators, whichever + clients sent them. The live shapes are a **cache**. A shape enters when a request names + it, and it is evicted once no recent request has named it (clause 12). For a saved + view the shape still includes what `view_shape` (`app.rs:40-72` at `57d385e1`) hashes, + plus the group membership of every drawn item and `measure_agg`. `view_shape` itself is + not widened, so what bumps `layout_version` is unchanged. + + *As first written* (superseded): a view whose operator shape moved had its operators + rebuilt **before the frame observation that follows this effect**. That rule served the + per-view frame observation, which ADR 008 clause 8 revises under the split. The three + rules below replace it. +- **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column + aggregates are keyed by the **items** of the axis categories, together with the filter + and page keys. They are never keyed by a position in a level sequence. So a collapse, + expand, hide or show of an **axis** category's items changes no operator. The visible + axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms + for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item changes no + operator either, because a new item tuple is just a new key. Three things still + rebuild a shape: + - a change of which categories are its levels; + - a change of its filter or page keys, which includes hiding an item of a category that + is not on an axis; + - a change of `measure_agg`. + + A collapsed group's value is a pooled fold over its members. With item grain it is + combined from the members' state when it is read. For `Sum` and `Count` that combine is + linear. For the integrated aggregates it costs O(members) per read, and whether that + fits the frame budget at a 1,000-member group is **not measured**. +- **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it + cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a + 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). + So a rebuild is a resumable `step(deadline)` over an owned snapshot of its input. The + deltas that arrive meanwhile are buffered and added at install, which Z-set addition + makes exact. A circuit `step` on a cell delta is O(delta), and it is never sliced. + Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the + install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and + priority queue the user required. This ADR only makes its rebuilds steppable and + cancellable between slices. +- **A crossing count (ADR 008's second request).** Each step's occupancy output carries + the number of tuples whose count crossed zero. ADR 008's ordered index uses it to + choose between an insertion and a rebuild by sorting. - **The formula set is an input, not a rebuild.** After each model effect the host compares a fingerprint of the formula set, over (`raw`, `target`, `target_category`), with its previous read. If it moved, the formula delta (−the old @@ -628,18 +817,18 @@ states the same rule. - **A document swap rebuilds every circuit** from the new store. `install_workbook` bumps `document_generation`, and the same after-each-effect read sees it. -Re-reading the operator shape costs O(items) per view per model effect, and the formula +Re-reading the operator shape costs O(items) per live shape per model effect, and the formula fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs O(cells), once per actual reshape. The same `view_shape` function is therefore read at two moments: by the circuit after each effect, and by `layout_version` after the loop. They answer different questions, and neither replaces the other. -**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** `AddItem` -moves the operator shape, so the view is rebuilt with East in its sequences. Only then -does `SetCell`'s delta arrive, at operators that already know East. So the in-loop -`CellData` frame observation reads occupancy for the new shape. The `LayoutGeometry` -arm (`app.rs:1062-1071`) reads the frame immediately after each geometry effect, and it -reads a rebuilt occupancy for the same reason. +**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first +written* (superseded by item grain): `AddItem` moved the operator shape, so the view was +rebuilt with East in its sequences before `SetCell`'s delta arrived. **Under item grain, +`AddItem` moves no operator.** `SetCell`'s delta inserts the item tuple `East` as a new +key, and only the visible axis, which is derived, gains a row. The case needs no +ordering rule at all. **Recommended: a page selection is not a rebuild (question 7).** Under the recommended answer, the group key includes the page categories, so `[` and `]` read another slice @@ -886,37 +1075,61 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 3, PR #308) assumes these, under the shared names below. +ADR 008 (revision 19 at `69f0a7c6`, PR #308) assumes these, under the shared names below. +**Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. +A consumer reads it through `project` (clause 12), and none of it is addressed to a +host-held mirror of a client. -- **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and - per column tuple of cells that hold a value. Stored cells pass the view's filter, with - hidden items dropped, and a collapsed group's members map to the group's tuple. - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` - holds a value: a top-level `WHERE` must match, or raw data must exist at the target. - The operators are rebuilt per clause 7, so a batch that reshapes and then writes feeds - its write to the new shape. Page coordinates key a slice, subject to question 7. A tuple - is on the axis iff its count is above zero. **This ADR does not settle ADR 008 - question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those - cells are values with no stored cell under them. The stem definition drops them, and - if the user rules otherwise, those views fall back to a non-occupancy path. Occupancy - is integer and order-free, so it does not wait for ADR 005. -- **The occupancy delta** (ADR 008 clause 8). `distinct` over occupancy emits a - non-empty delta exactly when a tuple enters or leaves an axis. A reframe is a - non-empty occupancy delta. The two-arm asymmetry of the frame observation (gotcha 24, - improvise-mcq4) is unchanged, but each arm costs O(delta) instead of a rebuilt grid - per view. -- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over - the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), +- **Occupancy** (ADR 008 clause 3). For each projection shape (clause 7), there is a count + per row tuple and per column tuple of cells that hold a value. The counts are kept at + **item grain**, and the visible axis is a remap of them. + - Stored cells pass the shape's filter. + - A hidden item of an axis category is masked only when the visible axis is derived. + - A collapsed group's members map to the group's tuple at read time. + - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` + holds a value: a top-level `WHERE` must match, or raw data must exist at the target. + - Page coordinates key a slice, subject to question 7. + + A tuple is on the axis exactly when its count is above zero. **ADR 008's question 1 is + answered: "go with occupied"** (#308, comment 4171484686). So the two cases where + today's pruning keeps a row with no stored cell under it are dropped on purpose: a fold + row with a formula column, and `Aggregation::Exact`. Clause 5's stem rule already drops + them, there is no "non-occupancy path" for them, and the differential test carves them + out by name. *As first written*, this item said this ADR "does not settle" that + question; that is superseded. Occupancy is integer and order-free, so it does not wait + for ADR 005. +- **The ordered record list** (ADR 008 clause 1, `Records`). The user: "the ordered list + of records is a model-side concept" (Q27). It is the store's order (gotcha 18, + `DataStore::live_records`), and this ADR does **not** maintain it as a per-shape + operator. Clause 5a's record stage keys records by `record_of` and orders them by the + store sequence, so it *consumes* the store's order and never defines a second one. How + a page and filter select from that list is ADR 008's question 55. It is not this ADR's + to answer, because this ADR keeps no per-view copy of the list. Whether records mode is + `project` over the virtual axes (ADR 008 question 54) changes nothing here: occupancy + over `_Index` and `_Dim` is the same operator as over any other axis. +- **The occupancy delta** (ADR 008 clause 8, which the split revises). `distinct` over + occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** + axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first + written*, its consumer was the per-view frame observation and its reset of each client's + cursor (gotcha 24). Under the split the host holds no cursor. So the delta is a model + fact, published on clause 12's delivery stream to the consumers of that shape, and a + client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the + host pushes it is question 13, which is ADR 008's question 52 seen from this side. +- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the + pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate, and the formula layer re-evaluates only the columns whose inputs moved, - through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that - hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the - exact accumulator (recommended) or ADR 005 (clause 8). -- **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize, - or maintain) is struck there as decided by the user through this ADR. Its questions 1 - and 2 stay open there. -- **What it does not get.** Column widths depend on formatted text, which is - display-only (gotcha 8), so they are not maintained here. They stay ADR 008 question 2. + aggregate at item grain, and the formula layer re-evaluates only the columns whose + inputs moved, through clause 2's trait. Clause 5 replaces `Expansion::members`. Totals + that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: + the exact accumulator (recommended) or ADR 005 (clause 8). +- **What ADR 008 no longer asks.** Its question 6 is answered ("maintained + incrementally"). Its questions 1 and 2 are answered there too: 1 is "go with occupied", + and 2 is (b), sticky per session, with hysteresis as the easing (its question 22). *As + first written*, this item said they "stay open there", which had gone stale. +- **Widths.** A width depends on formatted text, which is display-only (gotcha 8). Under + the user's answer to ADR 008's question 2, widths are taken over the rows realized and + held by the view layer. The user's question whether DBSP can maintain a maximum width is + answered in clause 13. ### 10. ADR 003's live sources are delta streams, when they can be @@ -963,10 +1176,336 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So on the network path a write is not O(delta): it is O(cells) for the snapshot today, and roughly twice that with the circuit in it. The keyboard path (`apply_effects` - without a `Host`) takes no snapshot and keeps the O(delta) cost. + without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the + store-memory requirement (clause 14), that second clone is also peak memory, which a + rewind would avoid. Question 9 stays open, and is now weighed against clause 14 as + well. - **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the delta sequence, so replaying commands reproduces its state. +### 12. Network transparency: one circuit, consumed in process or across a wire + +**The requirement is the user's:** "The DBSP machinery needs to be implemented in a +network transparent way, but the details of that go to ADR 009" (chat). **Everything +else in this clause is this ADR's design, and the author's choices are marked as such. +None of it is measured.** + +**What crosses a wire, and what does not** (the author's reading). Two boundaries can be +a wire. +- **The output boundary** carries the results of `project` and their changes: a window's + cells, its axis occupancy, its totals, and ADR 008 clause 4's width facts. In the TUI + the consumer is in the same process. In the browser it is the page, across `postMessage` + from the worker or a socket from `improvise-ws-server` (gotcha 45). +- **The input boundary of a live source** (clause 10). A source that reports its changes + delivers a cell delta, possibly from another machine. The circuit cannot tell that + delta from one `DataStore` emits. + +Three things do not cross: +- **The write side carries no deltas.** A client changes the model only by `apply` of + model commands (the user, Q43), so traffic from a client to the host stays + `ModelCommand`s with their basis (ADR 001), as today. +- **The circuit's internal streams stay on the host.** The cell delta, the per-record + stage and the group aggregates never cross: aggregation is model-side (the user, Q38), + so no operator runs in a client. +- **No UI state crosses.** The host is told the request and nothing about a cursor, a + scroll or a key (Q32, Q33, Q37). + +**What a projection request is to the circuit** (the author's answer to "is the result +keyed by request, by axes, or shared?"). A request has two parts. +- **Its operator shape** (clause 7) selects the circuit's state. The shape is shared by + every request, from any client, that has it. +- **Its window**, the top-left and the dimensions, selects a part of that state's output. + The window is never state. + +So maintained results are keyed by **shape**, and the only per-request work is +restricting a shape's output to a window. That is O(visible) per request and needs +nothing kept between requests. + +**Positions are cross-product positions.** The user answered ADR 008's question 53 in +chat: "Q53 is cross product: I'm still ambivalent about which side of the wire pruning +should happen on" (relayed on 2026-10-05; on no PR). So a request's top-left is a position +in the **cross product** of its axes, in each category's own item order. That position is +a bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed +contexts, we can represent bigints as strings in JSON"). A cross-product position names +the same tuple whatever is occupied, so **an occupancy change never moves a request's +window**. It changes only which positions inside the window hold data. **Which side of +the wire prunes is open** (the user is "still ambivalent"), and is question 19. The +circuit must serve both sides, and what it must expose in each case is the author's +analysis: +- **Pruning on the model side.** `project` returns occupied rows and columns. The circuit + must then answer two order-statistic queries over a shape's visible occupancy, from a + cross-product position: + - the next k occupied tuples at or after a position, so a pruned window can start where + the request points; + - the rank of a tuple among the occupied ones, and the occupied tuple of a given rank, + so a client can scroll a pruned axis by rows and show "row i of n". + + That is ADR 008's ordered index over occupied tuples (its clause 3), the sorted `Vec` + first and the order-statistic B-tree when measured (its question 16), fed by this ADR's + occupancy delta and crossing count. Under this case a client sees only occupied rows, + and its scroll space is the pruned one, built on top of the cross-product offset. +- **Pruning on the client side.** `project` returns the cross-product window, empty + positions included, together with a **skip structure** the client can query: for each + axis, the occupied runs that intersect the window and the next occupied position past + each edge. The circuit then has to expose the same order-statistic queries, but as a + model fact a client can request, not as a re-indexing the model performs on its behalf. + The cost is that a sparse grid at 3 × 1,000 (10^9 positions, about 10^5 occupied in ADR + 008's fixture) sends mostly empty windows unless the client asks for skips first. + +Both cases need the same maintained structure. They differ only in who calls it, and in +whether a `project` result is already pruned. Prune would be a request parameter in the +first case (ADR 008's flagged reading 1) and a client policy in the second. + +**"As absolute as possible", and what still moves a position.** The user, in chat: "But +the top left should be as absolute as possible" (relayed on 2026-10-05; on no PR). *The +rest of this paragraph is the author's analysis.* A cross-product position does not move +when occupancy changes, pruning included. **It does move when an item is inserted, +removed or reordered** in a category's own order. Positions are mixed-radix numbers over +the items of each level, so inserting item j at level L shifts every later item of L, and +changes the radix that every outer position is multiplied by. No position over a +changing item list is fully absolute. Three things follow: +- **Deltas are keyed by identity, and positions are attached when sent.** An output + element names its row and column by tuple identity: item ids under ADR 007, and names + until then. The host attaches the absolute position from the current item order when it + sends the element. A delta keyed by position alone would go stale with the next item + insert. +- **An item edit is sent as an axis change, not as moved rows.** "Item inserted at index + j of level L", or removed, or moved. A client re-derives every position it holds + arithmetically, at O(levels) per position, and its top-left can be re-anchored to the + same tuple. That is the cursor's identity rule from ADR 008 clause 11 applied to the + window. Resending every row whose position moved would cost O(rows after j). +- **The cost is rank and select over each category's items.** "The position of item x" + and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` + first, a B-tree when measured, its question 16). A tuple's position is O(levels × log + items). The ordered occupancy index must compare tuples by **current** item position, + not by a position stored at insert time. With item grain (clause 7) its keys are item + identities, so an item insert re-orders nothing in it, and comparison goes through the + order-statistic lookup, at O(levels × log items) per comparison. That is **not + measured**. Question 20. + +**A pruned answer may arrive as a stream.** The user, in chat: "After pruning we might +have to stream rows and columns to attempt to fill out the full grid" (relayed on +2026-10-05; on no PR). *The author's analysis:* under model-side pruning, `project` scans +forward from the absolute top-left over each axis's occupancy until it has the requested +number of occupied rows and columns, or reaches the end of the axis. It sends the rows +and columns it finds as it finds them. What the circuit must support: +- **Skipping empty space in O(log n + k).** "The next k occupied tuples at or after a + position" over the ordered occupancy index. It must never walk empty cross-product + space, which at 3 × 1,000 is mostly empty. +- **The absolute position of every row and column sent**, so that a client can place it, + and a later delta can name it, without counting. +- **Bounded work per slice.** The fill is a job under ADR 008 clause 11's runner when k + is large, and it can be resumed from the last absolute position it sent. So the only + state a fill keeps is `(request, fill front)`, which is "as stateless as possible" in + the user's words. A lost fill restarts from the client's last known position. +- **Changes during a fill are identity-keyed deltas, stamped by step** (T3). Suppose a + row becomes occupied between the top-left and the fill front after it was passed. The + subscriber receives it as an ordinary occupancy delta carrying its absolute position, + so its window now holds one row more than asked, and it drops or keeps the extra row by + client policy (Q42). A row that becomes occupied ahead of the front is simply found by + the fill. Each chunk of the fill is a delta to the window's Z-set, stamped like any + other. So a fill and a live update are one stream, which keeps the in-process and the + remote consumer the same (T6). + +- **Rows and columns fill independently** (the author's, on ADR 008's question 59(c)). + This ADR's occupancy is kept **per axis**: a count per row tuple and a count per + column tuple (clause 9). That matches today's `prune_empty`, which drops a row only + when it is empty across the whole grid, and prunes the columns against the same + unpruned grid (`layout.rs:752-757` at `50e1314d`). So a row occupied only in a column that falls + outside the window is still an occupied row, and the row fill never waits on the + column fill. A joint rule (a row occupied *within the requested columns*) would need a + two-dimensional count per (row tuple, column tuple). That is O(occupied cells) more + state, and it would change today's pruning, so it is not proposed. + +Whether a pruned response is streamed or returned whole, and what a client sees while a +fill is incomplete, is question 21. + +**What network transparency requires** (T1 to T6, the author's): +- **T1. Results are plain data.** Every element on an output stream is a value the + protocol crate can serialize, with no `Rc`, no borrowed store key and no closure. Facts + at `50e1314d`: `CellValue` is `Serialize`, with numbers going through `wire_f64` + (`crates/improvise-core/src/model/cell.rs:86-118`). `CellKey` is **not**: it derives only + `Debug, Clone, PartialEq, Eq, Hash` (`cell.rs:10`). So an output element names a cell by + the request's row and column tuples, or by cross-product positions. Where a write may + follow, it also carries ADR 002's surrogate (`CellId`). It never names a cell by + `CellKey`. +- **T2. One identity on both sides of the wire.** A consumer across a wire must + consolidate exactly as an in-process one does. **On NaN they would not.** Clause 3's + delta identity is bit identity, under which two NaNs with different payloads differ. But + the wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114` at `50e1314d`), and the + projection compares values with `same_on_the_wire` (`crates/improvise-protocol/src/render.rs:173-180` + at `50e1314d`; `context/code/protocol.md` gives the reason). So an output delta + `−(c, NaN₁) + (c, NaN₂)` is non-empty in process and consolidates to nothing after a + round trip. **Proposed:** output streams are consolidated under the wire's identity + (every NaN is one value, and every other number is its bits) before they leave the host. + The record tier keeps bit identity, as it does today. Question 15. +- **T3. Every output change is stamped.** A delta carries the step that computed it, as + `(document_generation, step)`. The circuit steps once per model effect (clause 3), and + ADR 001 already puts batches in one total order. A consumer that has applied step n + drops anything at or below n, so a duplicate does no harm, and it treats a gap as a + loss, which a fresh snapshot answers. What is published is the **batch's** consolidated + delta, and only once the batch is accepted. A refused frame is restored (clause 11) and + publishes nothing. +- **T4. A snapshot, then deltas.** A subscription starts with the integrated result at + step n: the window as a Z-set of current values, each with weight +1. Deltas from step + n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current + result. `GridCache::folded(&cache.seed()) == *cache` states the same property for + today's bootstrap compaction (gotcha 32). +- **T5. Order holds across streams, and coalescing happens within a stream.** A slow + consumer's pending deltas on one stream can be **summed** into one delta. Z-set + addition is commutative and associative, so the sum is exact, and the queue is bounded + by the size of the result, not by the number of steps. When the summed delta outgrows + the snapshot, a snapshot replaces it. That is the backpressure, and nothing is dropped + silently. **Across streams, order stays load-bearing**, as gotchas 33 and 43 require: + the occupancy change comes before the cells it places, and the cells come before the + basis that describes them. This does not break gotcha 42's "never coalesce". That rule + is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple + identity, or by a stable cross-product position, is not positional in that sense. +- **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives + the same result and delta types as a remote client. Its transport is a function call. + That is what makes the requirement testable: run one edit sequence through two + consumers, one in process and one behind a serialize and deserialize round trip, and + require their integrated results to be equal under T2's identity. A TUI shortcut that + read operator state directly would be a second read path, which is the derivation-twice + defect (design-principles §5). + +Weights on the output are ±1, because a result is a set of current values. A consolidated +weight that does reach past 2^53 follows the user's bigint-as-string rule (Q36). + +**Where the wire form lives** (the author's). `zset` stays dependent on `std` alone (clause 1). +The wire types live in `improvise-protocol`, with the one envelope pair (gotcha 42). A +consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that boundary. +The alternative is a `serde` feature on `zset`. The author leans to the conversion, +because what crosses is the projected result, not the crate's own element type. + +**How stateless the host can be.** The user: "as stateless as possible". These are the +options of question 13, and ADR 008's question 52 asks the push half of the same thing. +- **(a) Pure fetch.** The host keeps nothing per client. A client calls + `project(request)`. It learns its result may be stale from a change notice addressed to + no one: the accepted batch's step. It then refetches when its own policy says to (Q42). + Each refetch costs O(visible). +- **(b) Soft-state subscriptions** (the author's leaning). A request may ask to be kept + current. For each subscription the host keeps the request and the last step it sent, + and nothing else: no copy of the window, no cursor and no viewport. After each accepted + batch it reads the delta of each subscribed shape, restricts it to the request's window, + and sends it under T3 to T5. **The table is a cache.** If it is lost (a restart, an + eviction, a reconnect), a client resumes by resending its request and the last step it + applied, and the host answers with a snapshot. The host could instead keep a short ring + of past deltas per shape; the author's lean is no ring at first. So the user's "recent + projections + clients" are this table plus clause 7's shape cache, and **both can be + rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). +- **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and + `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, + `:822` and `:981` at `50e1314d`). It keeps each client's `ViewState` and grid on the host + and diffs against them. ADR 008 lists it as obsolete under the split, and this ADR does + not propose it. + +A shape is kept while some subscription or recent request names it. Eviction drops its +operators, and the next request rebuilds them as a job (clause 7). How long "recent" lasts +is a cost the user has not set (question 14). + +**What a client must hold to resume:** its request, the last step it applied, its +integrated window, and its basis (ADR 001). The host holds nothing that a client needs +back. + +**The basis.** ADR 001's basis is published with the result it describes, and after it +(gotcha 43). The host builds it from the request's window (ADR 008, lens row 18), not from +a mirror. `published_basis` (`project.rs:585` at `50e1314d`) already takes the `GridCache` +it describes, not a cursor. + +### 13. Incremental max width (the user's question on ADR 008) + +The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" +**This clause is the author's analysis, and none of it is measured.** The user's decision +on widths stands as ADR 008 records it: (b), sticky per session, eased by hysteresis, +taken over the rows realized and held by the view layer (ADR 008 clause 4). + +**DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. +When the widest value is retracted, the new maximum cannot be computed from the old +maximum and the retraction. So it belongs to clause 4's **integrated** class, beside `Min` +and `Max`: each group keeps its integrated multiset and reads off the last key. A *top-k* +(keeping only the k widest) holds less state, but it is wrong once the top k have all been +retracted, and then it needs a rescan. So it is not proposed. + +**For widths the multiset is small.** The group is one column of a projection shape, +together with its number format, since a width is measured on formatted text. The values +are display widths, which are small integers. So the state is a histogram from width to +count, per column. It is O(distinct widths) per column, whatever the number of rows, and +an update costs O(log d), where d is the number of distinct widths. + +**The cost is upstream, in what feeds the histogram.** Its input is the display width of +**every evaluated cell in the column**, under the request's number format. That needs +three things: +- **A delta of evaluated cells over the whole column**, not a window. Clause 4's group + aggregates and clause 5's formula re-evaluation already produce an output delta for + every unfolded cell that moves, with its old value as the retraction. So unfolded rows + need no new state. **Fold rows are not covered.** Under item grain (clause 7) a + collapsed group's value is combined when it is read, not maintained, so its width would + need the fold maintained per column at the visible grain. +- **Formatting every changed cell on the host**, on the write path: `format_f64` and a + display width for each changed cell. Each one is cheap, but it would be the first + display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value + back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But + today the host is asked for widths only over the rows requested (ADR 008 clause 4), and + this would widen that to whole columns. +- **A rebuild whenever the format changes.** `number_format` is a parameter of the view + definition, and a rebuild costs O(occupied rows) per column, as a job. + +**What it would buy, set against what the user decided.** A maintained whole-column width +is ADR 008's question 2 option (c), "size each column from its whole contents", which the +user did **not** choose. Under (b) a window's widths are an O(visible) computation over +cells the window already evaluates, so maintaining them incrementally adds nothing. So +the real question is whether the user wants (c) after all, now that its state is cheap. It +would remove the jitter at its source and make hysteresis unnecessary, at the upstream +cost above. That is question 12. + +### 14. Resident state, and the store-memory requirement + +The user, on ADR 008: "We have to be careful here to minimize memory requirements of the +store: offloading inactive regions to disk (maybe mmap or similar), +double-delta/gorilla/lz4 compression, etc." (#308, comment 4171481179). And then, on ADR +008's question 17: "Let's defer this as recommended." (4188271710). So the store's memory +gets its own ADR, once this ADR's operator state has been measured at ADR 008's fixture. +**This clause withdraws the earlier non-goal** ("Persistence, spilling to disk, and +threads", clause 1), and it says what this ADR must not rule out. + +**What this ADR keeps resident, beside the store.** This is the author's inventory. The +orders come from the clauses and are not measured. + +| state | clause | order | +|---|---|---| +| the integrated base that a rebuild reads | 7 | O(cells), if it is kept as a copy | +| clause 5a's per-record stage | 5a | O(cells of the records an aggregate over an expression reads), as written | +| multisets for `Min`, `Max` and text | 4 | O(cells) of the measures that need them | +| `ExactSum`, per summed group | 8 | 568 bytes per group | +| occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | +| the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | +| the subscription table | 12 | O(request) per subscription | + +**Proposed by the author:** +- **The store is the integral, so it is not copied.** The integrated base a rebuild reads + is `DataStore` itself, which is I(cell delta) by definition. Clause 5a's per-record + stage keeps each record's cells only to compute the retraction `I(x)₋₁`. That equals + `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. So + the stage can read the store and keep no copy of the values. It still needs each cell's + store sequence (clause 5a), one integer per cell, which `DataStore` would keep beside + the value rather than in a second map. This is **not measured**, and the retraction of + the order key has to be checked against it (question 17). +- **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a + range, and apply a delta. The first implementation keeps it in memory, and the + store-memory ADR can page or compress it without changing any operator. +- **Shapes are evicted** (clause 12). So per-shape state is bounded by the shapes in + recent use, not by every saved view. +- **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, + `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and + O(groups), and the store-memory ADR has to count them. + +**The frame budget.** ADR 008 clause 11 sets it, and the user accepted the recommendation +(ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms +slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is +a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration +step 4 must. + ## Alternatives considered **Feldera's `dbsp` crate.** What was verified on 2026-09-24: @@ -1038,6 +1577,12 @@ displayed path until the differential test holds. accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. +8. **The delivery stream** (clause 12), on the answers to questions 13, 15, 19, 20 and + 21. It brings the output types in `improvise-protocol`, the step stamp, + snapshot-then-delta, the order-statistic queries over occupancy, and the test with two + consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff + only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, + the cost of a step and of a rebuild (clause 14). **Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is question 10. improvise-swre stays independent and is useful under either backend. @@ -1055,6 +1600,9 @@ longer a prerequisite of step 6. - ADR 008 gets occupancy, the occupancy delta and column totals without a second evaluator. - Live sources that can report changes plug in without touching the evaluator. +- One delivery path serves the TUI in process and every remote client (clause 12), and + the host keeps no per-client UI state: only a subscription table that can be rebuilt + from the requests. **Bad.** @@ -1072,25 +1620,33 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- The operator shape is re-read after every model effect: O(items) per view, a cost the +- The operator shape is re-read after every model effect: O(items) per live shape, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. +- Per-shape state lives only as long as requests name it. A shape that comes back after + eviction is rebuilt as a job, and the client waits through its pending state. +- The output identity (every NaN is one value) differs from the record tier's (bit + identity), as T2 explains. Two identities need stating everywhere a delta crosses. +- A position over a changing item list is never fully absolute. An item insert moves + every later position, and is sent as an axis change (clause 12). ## Open questions Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. +the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered +none of questions 1 to 11 on #309.** Questions 12 to 21 were added by the 2026-10-05 +revision, and none is renumbered. -1. **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? +1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. -2. **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on +2. *[Open.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on `zset` and `improvise-core`), or inside improvise-core? *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? +3. *[Open.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums until then. - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in @@ -1117,35 +1673,35 @@ the user decided", as ADRs 004 and 007 did. that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - `Avg`'s one rounding must be stated in step 6. -4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported +4. *[Open.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to change, change it in both backends at once, as its own decision. -5. **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by +5. *[Open.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by today's bounded fixed point, restricted to their component, rather than by any operator in the crate? *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the cells a guarded cycle leaves convergent. The crate needs no recursion operator. -6. **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? +6. *[Open.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. **Page selection as a slice.** Should the group key include the page categories, so +7. *[Open. ADR 008's flagged reading 1 makes the page selection a parameter of the request, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". -8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas, +8. *[Open.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? *Recommendation:* yes, chosen per document. Integrated state is proportional to the source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` +9. *[Open. Clause 14 now weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` snapshot that `Host::mutating_frame` already takes? *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying `−d` restores integrated values but not the history `delay` and `differentiate` @@ -1154,13 +1710,13 @@ the user decided", as ADRs 004 and 007 did. today. A true rewind would need its own design and an observational-equivalence test over every stateful operator. -10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps +10. *[Open.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. -11. **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). +11. *[Open, and affected by question 16. Under item grain a reorder of an axis category's items rebuilds nothing, so the recommendation's premise (key by position and rely on `view_shape`'s rebuild) no longer holds. Keying by item identity and reading the order at read time, which clause 12 already needs (question 20), becomes the natural form. The recommendation is left as written until the user answers both.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering category's item position, with that category's item order added to the operator shape so that a reorder rebuilds them? Or should they key by stable item identity (ADR 007), and @@ -1174,3 +1730,89 @@ the user decided", as ADRs 004 and 007 did. no widening at all, and ADR 008's text and cross-references hold unchanged. If the user wants hidden items ordered too, the `measure_agg` part must also cover their positions. + +12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). + Should the host maintain each column's whole-column maximum display width? That is + ADR 008's question 2 option (c), and its state would now be a small histogram per + column. *Recommendation (the author's):* no, not now. The user decided on (b) with + hysteresis, and under (b) a window's widths cost O(visible) and need no maintenance. + The maintained form also needs fold values maintained per column, and formatting on + the write path. Revisit it if a measured workload shows jitter that hysteresis does + not cure. + +13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per + client** (clause 12; ADR 008's question 52 asks the push half). *Recommendation (the + author's):* (b), soft-state subscriptions that can be rebuilt from the request and the + last step, with no copy of a client's window on the host. When its table is lost, (b) + falls back to (a). (c) is obsolete under ADR 008's split. One of the two ADRs should + hold this question, and the other should point to it. + +14. *[New.]* **What keys operator state, and how long a shape lives** (clause 7). Is it + keyed per projection shape, cached and evicted (the author's)? Or per saved `.improv` + view only, so that a request whose shape is unsaved (an unsaved prune or hide, ADR + 008's question 51) is served without incremental state? *Recommendation:* per shape, + cached, and evicted once no request has named it for a while. The time is an + implementation number. An unsaved shape is then as incremental as a saved one, which + keeps "pruning shouldn't infect every client" from costing an eager path. + +15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate + under the wire's identity, where every NaN is one value, while the record tier keeps + bit identity? *Recommendation:* yes. Otherwise a remote consumer and an in-process + one disagree when only a NaN's payload changes. + +16. *[New.]* **ADR 008's three requests** (ADR 008, "What revision 4 asks of ADR 009"): + occupancy and the per-column aggregates kept at item grain, a crossing count per + step, and rebuilds as steppable jobs. *Recommendation:* adopt all three, as clause 7 + is now written. This changes clause 7's list of what rebuilds, removes the rebuild + from gotcha 43's worked case, and undercuts question 11's premise. + +17. *[New.]* **Resident state** (clause 14). Should this ADR withdraw "spilling to disk" + as a non-goal, keep no second copy of the store (the store is the integral), and put + operator state behind one interface that the store-memory ADR can page? + *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the + order the user's answer to ADR 008's question 17 already sets. + +18. *[New.]* **Formulas the per-record map cannot express** (clause 5a). ADR 008's + question 15 sends them to a fallback over the occupied contexts, never to + `Expansion::members`; the user's answer was "the recommendation is fine.". Two parts: + - Does this ADR's `Eager` adopt the same rule, so that `Eager` and `Incremental` + agree and this ADR's differential test needs no carve-out? + - Do the formulas that target a category other than `_Measure` take the same + fallback, so that they do not block ADR 008's retirement of `Expansion::members`? + + *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's + question 1 and question 15 cases, as a stated change of behaviour. + +19. *[New. The user is "still ambivalent".]* **Which side of the wire prunes** (clause 12; + the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, + and one ADR should hold it). On the model side, `project` + returns occupied rows and columns, and the circuit answers next-occupied and + rank/select queries for it. On the client side, `project` returns the cross-product + window together with a skip structure, and the client asks for the queries itself. + *Recommendation (the author's):* the model side, with prune as a request parameter. + A sparse grid at 3 × 1,000 then never ships empty space, and a client stays thin. + Either way, the circuit maintains the same structure (clause 12). + +20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left + should be as absolute as possible"; ADR 008's question 58 asks the same from the + request side). A cross-product position stays fixed when + occupancy changes, but it moves when an item is inserted, removed or reordered. Should + deltas be keyed by tuple identity, with absolute positions attached when they are + sent, and should an item edit be sent as an axis change, so that clients re-derive + their positions arithmetically? *Recommendation:* yes. It costs rank and select over + each category's items (ADR 008 clause 10), and comparisons in the occupancy index at + O(levels × log items) each, which is not measured. It is compatible with ADR 008's + question 58 recommendation, which makes a tuple of item ids the primary form of the + top-left. Whether hidden items belong to the absolute space is ADR 008's question 58. + Under item grain (clause 7) this ADR serves either answer, because hiding is a remap. + +21. *[New.]* **A pruned answer as a stream** (clause 12; the user: "After pruning we + might have to stream rows and columns to attempt to fill out the full grid"). Should + `project` under pruning fill forward from the absolute top-left, as a sliced job that + can be resumed from its fill front, and send each found row and column as a stamped + delta? Should a row that becomes occupied behind the front arrive as an ordinary + occupancy delta, and leave the window one row larger than asked, for the client's + policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one + stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 + asks the same, and asks in its part (c) whether rows or columns fill first. Clause 12's + answer is that the two fill independently (the author's). From 4851eec8a6174f50eacf1335c6ea8fc0d5f73dea Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:56:02 -0700 Subject: [PATCH 17/67] docs(adr): ADR 009 records window-relative pruning and the restricted skip query (improvise-felo.4) The user's chat statements make pruning a filter over displayed values. Clause 12 now names the query a client's fill loop needs and three ways to answer it; question 19's recommendation is revised and question 22 added. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 107 ++++++++++++++++++---- 1 file changed, 88 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index eaa01619..bd29138a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -5,7 +5,7 @@ are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision (PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 18. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -104,6 +104,11 @@ None of this answers this ADR's own questions 1 to 11. Clause 12, question 20 (ADR 008's question 58). - "After pruning we might have to stream rows and columns to attempt to fill out the full grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). +- "Pruning is just a filter applied to full columns a rows, right? Hide a column/row in + the grid if it's all null. That sounds like a UI side concern because that's where the + visible grid lives" and "The point of pruning is to make the displayed values more + dense, so a value being present outside the viewport doesn't matter." (chat, relayed on + 2026-10-05). Clause 12, question 22. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1307,15 +1312,55 @@ and columns it finds as it finds them. What the circuit must support: other. So a fill and a live update are one stream, which keeps the in-process and the remote consumer the same (T6). -- **Rows and columns fill independently** (the author's, on ADR 008's question 59(c)). - This ADR's occupancy is kept **per axis**: a count per row tuple and a count per - column tuple (clause 9). That matches today's `prune_empty`, which drops a row only - when it is empty across the whole grid, and prunes the columns against the same - unpruned grid (`layout.rs:752-757` at `50e1314d`). So a row occupied only in a column that falls - outside the window is still an occupied row, and the row fill never waits on the - column fill. A joint rule (a row occupied *within the requested columns*) would need a - two-dimensional count per (row tuple, column tuple). That is O(occupied cells) more - state, and it would change today's pruning, so it is not proposed. +**Pruning is a filter on the values displayed (the user, later in chat).** "Pruning is +just a filter applied to full columns a rows, right? Hide a column/row in the grid if it's +all null. That sounds like a UI side concern because that's where the visible grid lives" +and "The point of pruning is to make the displayed values more dense, so a value being +present outside the viewport doesn't matter." (both relayed on 2026-10-05; on no PR). +**This is a change from today's code**, and it is recorded here, not decided. Today, +`prune_empty` drops a row only when it is empty across the **whole** grid, and prunes the +columns against the same unpruned grid (`layout.rs:752-757` at `50e1314d`). In the user's +words, emptiness is judged over the **displayed** values, so a row that holds a value only +in a column outside the window is hidden. *The rest of this paragraph is the author's +analysis.* +- **What the model then owes a client** is not a pruned axis. It owes windows (an + absolute top-left and dimensions) and a **restricted skip query** for the client's fill + loop: "the next k rows, from absolute position p, that hold a value in at least one of + these columns", and the same with rows and columns swapped. The question is about a set + of columns, not about the whole grid, so a per-axis count cannot answer it. A row can be + occupied overall and still be empty in every column the client shows. +- **Three ways to answer the query**, all unmeasured: + - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then + test each candidate row against the column set by lookup. The state is the per-axis + counts clause 9 already keeps. The worst case scans many rows that are occupied + overall but empty in the column set, at O(|columns|) lookups each. + - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that + hold a value in it. The query is a k-way merge of |columns| sorted lists from p, at + O(|columns| × log n + k × |columns|). The state is O(occupied cells at grid grain), + the two-dimensional count clause 9 does not keep. It is maintained from the cell delta + and from formula stems (clause 5's stem rule decides when a formula cell holds a + value). A collapsed group's column merges its members' lists when it is read. + - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, + intersected with the column set as the fill walks forward. It has the same state as + (ii), and its scan behaves like (i), with an exact test per row. + + The author leans to (ii): a window has tens of columns, so the merge is small, and it + never walks rows that are empty in the window. Its cost is a second O(occupied cells) + structure, beside the store (clause 14). +- **The filter has a fixpoint.** Rows are judged against the displayed columns, and the + columns against the displayed rows. Hiding a row can empty a column, and hiding that + column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records + it, in its questions 60 and later); the skip query only has to be cheap enough to call + once per iteration. +- **What this changes in this ADR.** If pruning is a client filter over the window, + nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The + pruned axis, its ordered index and "row i of n" are not needed either, because a + window-relative filter has no global n. What remains model-side is **which cells hold a + value**, including formula stems, which only the model can evaluate. ADR 008's question + 1 ("go with occupied") is about that, and it still applies, to the cells of a window. Its + question 11 ("prune empty should be the default") becomes a default for a client's + display. Whether clause 9's occupancy should be restated as the index of (ii), and + whether ADR 008 clause 3's `Occupied` axis survives, is question 22. Whether a pruned response is streamed or returned whole, and what a client sees while a fill is incomplete, is question 21. @@ -1577,8 +1622,8 @@ displayed path until the differential test holds. accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** 7. **Live sources as delta streams**, when ADR 003's source trait exists. -8. **The delivery stream** (clause 12), on the answers to questions 13, 15, 19, 20 and - 21. It brings the output types in `improvise-protocol`, the step stamp, +8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to + 22. It brings the output types in `improvise-protocol`, the step stamp, snapshot-then-delta, the order-statistic queries over occupancy, and the test with two consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, @@ -1634,7 +1679,7 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 21 were added by the 2026-10-05 +none of questions 1 to 11 on #309.** Questions 12 to 22 were added by the 2026-10-05 revision, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -1783,15 +1828,16 @@ revision, and none is renumbered. *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's question 1 and question 15 cases, as a stated change of behaviour. -19. *[New. The user is "still ambivalent".]* **Which side of the wire prunes** (clause 12; +19. *[New. The user was "still ambivalent", then said in chat that pruning "sounds like a UI side concern"; see question 22. Not confirmed as a decision.]* **Which side of the wire prunes** (clause 12; the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, and one ADR should hold it). On the model side, `project` returns occupied rows and columns, and the circuit answers next-occupied and rank/select queries for it. On the client side, `project` returns the cross-product window together with a skip structure, and the client asks for the queries itself. - *Recommendation (the author's):* the model side, with prune as a request parameter. - A sparse grid at 3 × 1,000 then never ships empty space, and a client stays thin. - Either way, the circuit maintains the same structure (clause 12). + *Recommendation (the author's), revised after the user's later chat:* the client + side, as the user leans. The model offers windows and question 22's restricted skip + query, so a sparse grid still never ships empty space. *As first written*, this + recommended the model side; that is withdrawn. 20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left should be as absolute as possible"; ADR 008's question 58 asks the same from the @@ -1814,5 +1860,28 @@ revision, and none is renumbered. occupancy delta, and leave the window one row larger than asked, for the client's policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 - asks the same, and asks in its part (c) whether rows or columns fill first. Clause 12's - answer is that the two fill independently (the author's). + asks the same, and asks in its part (c) whether rows or columns fill first. Under the + user's later chat (question 22), the fill loop is the **client's**: it calls the + restricted skip query once per step of its row and column fixpoint. The host then + keeps no fill state at all, which is more stateless than the sliced host job this + question first described. + +22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause + 12; the user, in chat: "Pruning is just a filter applied to full columns a rows ... + That sounds like a UI side concern" and "a value being present outside the viewport + doesn't matter"). This changes today's semantics, which prune against the whole + grid. Three parts: + - (a) Should the circuit offer a **restricted skip query**, "the next k rows from + absolute position p with a value in at least one of these columns", and the same for + columns? How should it answer: (i) per-axis occupancy as a pre-filter, (ii) a column + index of occupied rows, merged, or (iii) a row index of occupied columns? + - (b) Should clause 9's whole-extent, per-axis occupancy be **restated** as that + index, and ADR 008 clause 3's `Occupied` axis and its "row i of n" be dropped, + since a window-relative filter has no global count? + - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty + should be the default") need restating? Question 1 would become the rule for which + cells of a window hold a value, and question 11 a client's display default. + + *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms + that pruning is the client's. (c) restate both in those terms, in ADR 008, since they + are ADR 008's answers. From 09938f0064f2d55e8b075c68c367537b0310b074 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:56:49 -0700 Subject: [PATCH 18/67] docs(adr): ADR 009 analyses the client's fetch, prune and fill loop (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index bd29138a..8e2a12ea 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1352,6 +1352,38 @@ analysis.* column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records it, in its questions 60 and later); the skip query only has to be cheap enough to call once per iteration. +- **The loop the user describes, and what makes it converge cheaply.** The user, in chat: + "The problem pruning solves is visualizing sparse datasets: screens that are mostly + empty are useless and so you drop out categories from rows and columns until every + visible row or column has at least one value on screen." (relayed on 2026-10-05; on no + PR). The criterion is a fixpoint on the displayed grid. *The author's analysis:* a client + alternates three steps. It fetches a window. It prunes, removing every row with no + value in the visible columns and every column with no value in the visible rows, until + nothing changes. It fills, asking for the next rows (or columns) past its last absolute + position that have a value in the columns (or rows) still visible. Three things the + circuit must answer for that loop: + - **"Any value in row r among columns S", and its batched form**, the next k such rows + from p. With index (ii) the batched form costs O(|S| × log n + k × |S|) per call, keeps + no state per request, and is recomputed for each request. That is the "as stateless + as possible" side of it. Without an index, each candidate row costs |S| store lookups, + and a sparse dataset puts most candidates in empty space. That is exactly the case + pruning exists for, so the index is not optional at 3 × 1,000. + - **Maintenance is incremental and cheap per write.** A cell that starts or stops + holding a value inserts or removes one entry in one column's list, at O(log n). Index + (ii) is a set of `(column, row)` pairs, so it is itself a Z-set fed from the cell delta + and from formula stems. Its delta is "row r gained or lost a value in column c". + - **The answers are friendly to deltas.** A client subscribed to a window can be sent + just those `(row, column)` changes that fall in its window's columns (clause 12's + subscription, restricted to S). It re-runs its own prune step locally, and it fills + again only when a row or column has been removed. So the host never re-runs a + client's fixpoint. + + Whether the loop always terminates is **not proven**. Within a fixed candidate set, + iterated removal reaches the unique greatest fixpoint, as a 1-core of the bipartite + row-column graph does. The fill only moves forward through absolute positions, so it + adds finitely many candidates before the axis ends. A client that bounds its fills per + frame stays within ADR 008's frame budget whether or not the loop has converged. That + bound is a client policy (Q42). - **What this changes in this ADR.** If pruning is a client filter over the window, nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The pruned axis, its ordered index and "row i of n" are not needed either, because a @@ -1882,6 +1914,9 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. + The user's later statement makes the criterion a fixpoint on the displayed grid, + reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is + what each fill step calls. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From 530ec2941bf4d0e8b84893c55aea4145fd5e1434 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:57:34 -0700 Subject: [PATCH 19/67] docs(adr): ADR 009 reads ADR 008 revision 20 and cross-references its questions 60-65 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 8e2a12ea..dc461009 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 19** -(PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 20** +(PR #308, read at `b0dc277c`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `69f0a7c6`, its revision 19) · +reads it at `b0dc277c`, its revision 20) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 19 of `context/adr/008-virtualized-grid.md` on PR #308, at `69f0a7c6`. Clauses 3, +revision 20 of `context/adr/008-virtualized-grid.md` on PR #308, at `b0dc277c`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 and 19, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 20, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1080,7 +1080,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 19 at `69f0a7c6`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 20 at `b0dc277c`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1379,7 +1379,9 @@ analysis.* client's fixpoint. Whether the loop always terminates is **not proven**. Within a fixed candidate set, - iterated removal reaches the unique greatest fixpoint, as a 1-core of the bipartite + iterated removal reaches the greatest fixpoint, which is unique (the union of any two + sets with the property has it too). ADR 008's question 61 says a fixpoint "may be + non-unique"; that holds for other readings, not for this one. Iterated removal reaches it, as computing the 1-core of the bipartite row-column graph does. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A client that bounds its fills per frame stays within ADR 008's frame budget whether or not the loop has converged. That @@ -1914,6 +1916,12 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. + ADR 008 revision 20 asks the same from its side: its question 62 (the skip query), + question 65 (whether "occupied" stays whole-extent) and question 61 (one pass or a + fixpoint). Its question 65 recommends keeping per-axis, whole-extent occupancy and + using it only to skip. Under (i) above, that skips only rows that are empty + everywhere, not rows that are empty in the displayed columns. So it does not answer + the restricted query by itself. The user's later statement makes the criterion a fixpoint on the displayed grid, reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is what each fill step calls. From 037f795473133258e2b326ae11a2e379f63b2082 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 14:59:09 -0700 Subject: [PATCH 20/67] docs(adr): ADR 009 corrects the single-window fixpoint claim and reads ADR 008 revision 21 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 ++++++++++++----------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index dc461009..48f2f643 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 20** -(PR #308, read at `b0dc277c`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 21** +(PR #308, read at `3b70a0f1`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `b0dc277c`, its revision 20) · +reads it at `3b70a0f1`, its revision 21) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 20 of `context/adr/008-virtualized-grid.md` on PR #308, at `b0dc277c`. Clauses 3, +revision 21 of `context/adr/008-virtualized-grid.md` on PR #308, at `3b70a0f1`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 20, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 21, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1080,7 +1080,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 20 at `b0dc277c`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 21 at `3b70a0f1`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1347,11 +1347,13 @@ analysis.* The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) structure, beside the store (clause 14). -- **The filter has a fixpoint.** Rows are judged against the displayed columns, and the - columns against the displayed rows. Hiding a row can empty a column, and hiding that - column can empty another row. Iterating to a fixpoint is the client's (ADR 008 records - it, in its questions 60 and later); the skip query only has to be cheap enough to call - once per iteration. +- **The fixpoint comes from the fill, not from one window.** *As first written*, this + item said that hiding a row can empty a column. **That was wrong.** A row dropped for + having no value among the window's columns contributes no value to any of those + columns. So on a fixed window, one pass that drops the rows and columns empty within + the window already is the fixpoint (ADR 008 revision 21, question 61, makes the same + argument). The iteration comes from the fill step, which brings in rows and columns from + outside the window. ADR 008's question 66 records that loop. - **The loop the user describes, and what makes it converge cheaply.** The user, in chat: "The problem pruning solves is visualizing sparse datasets: screens that are mostly empty are useless and so you drop out categories from rows and columns until every @@ -1380,9 +1382,8 @@ analysis.* Whether the loop always terminates is **not proven**. Within a fixed candidate set, iterated removal reaches the greatest fixpoint, which is unique (the union of any two - sets with the property has it too). ADR 008's question 61 says a fixpoint "may be - non-unique"; that holds for other readings, not for this one. Iterated removal reaches it, as computing the 1-core of the bipartite - row-column graph does. The fill only moves forward through absolute positions, so it + sets with the property has it too, as ADR 008 revision 21's question 61 also says). On + a fixed window that takes one pass. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A client that bounds its fills per frame stays within ADR 008's frame budget whether or not the loop has converged. That bound is a client policy (Q42). @@ -1916,9 +1917,9 @@ revision, and none is renumbered. should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. - ADR 008 revision 20 asks the same from its side: its question 62 (the skip query), - question 65 (whether "occupied" stays whole-extent) and question 61 (one pass or a - fixpoint). Its question 65 recommends keeping per-axis, whole-extent occupancy and + ADR 008 revision 21 asks the same from its side: its question 62 (the skip query), + question 65 (whether "occupied" stays whole-extent), question 61 (one pass or a + fixpoint) and question 66 (the fill loop). Its question 65 recommends keeping per-axis, whole-extent occupancy and using it only to skip. Under (i) above, that skips only rows that are empty everywhere, not rows that are empty in the displayed columns. So it does not answer the restricted query by itself. From 6305fa3bc7de606bfd2bb30763b1d718c15b3035 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:05:38 -0700 Subject: [PATCH 21/67] docs(adr): ADR 009 analyses forward-only, right-and-down pruning growth (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 35 +++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 48f2f643..4d749786 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1347,6 +1347,38 @@ analysis.* The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) structure, beside the store (clause 14). +- **Growth is only rightward and downward (the user, in chat).** "We should be able to + optimize pruning on the right and down assumption of growth (truncated at the bottom + right corner of the axis space)" (relayed on 2026-10-05; on no PR). *The rest of this + item is the author's analysis.* The fill loop only grows away from the absolute + top-left, and it stops at the end of each axis. So every skip query is a one-directional + **successor** query: "the next row (or column) after position p with a value among this + set", bounded by the end of the axis. Pruning never needs a predecessor or a reverse + scan. Scrolling up or left is not a reverse fill. It is a **new request with a new + absolute top-left**, filled forward from there, and nothing from the old request has to + be walked backwards. What that simplifies: + - **The index options.** (ii)'s per-column sorted lists need only a seek (a lower + bound at p) followed by forward iteration. A k-way merge of forward cursors keeps + O(|S|) state and never steps back. Option (i)'s pre-filter and option (iii)'s per-row + test also walk forward only. So any ordered structure with a lower-bound seek and a + forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 + question 16). No order statistics (rank, or "row i of n") are needed for pruning. + They remain only for a client that wants a scrollbar over an unpruned axis, where + rank is arithmetic over the cross product anyway. + - **Maintenance.** An insert or removal at position q affects only fills whose front is + already past q, and only within their window. A fill whose front is before q finds the + change when it gets there. So a subscription needs a delta only for positions between + its top-left and its front, which is a range test against two absolute positions. + - **Resumable streams.** A fill's whole state is its request and its last absolute + position on each axis. A resume is a seek past that position, so a lost stream, a + reconnect or a sliced job (ADR 008 clause 11) all resume the same way, with nothing + else kept. Positions are stable under occupancy changes because they are + cross-product positions. An item insert before the front shifts them, which is + question 20's axis-change message: the client re-derives its front arithmetically. + - **The corner truncates the work.** The fill is bounded by the requested dimensions + and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × + |S|) per round, never a scan of empty space past the last occupied position. + - **The fixpoint comes from the fill, not from one window.** *As first written*, this item said that hiding a row can empty a column. **That was wrong.** A row dropped for having no value among the window's columns contributes no value to any of those @@ -1926,6 +1958,9 @@ revision, and none is renumbered. The user's later statement makes the criterion a fixpoint on the displayed grid, reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is what each fill step calls. + Under the user's right-and-down growth (clause 12), the query is a forward successor + query only, and so are all three options; no rank or reverse scan is needed for + pruning. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From de3b2bc3234c4ecf2cc559086be6aa42b7b57089 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:08:52 -0700 Subject: [PATCH 22/67] docs(adr): ADR 009 analyses predecessor search and a top-left on a pruned row (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 4d749786..c311be43 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1379,6 +1379,58 @@ analysis.* and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × |S|) per round, never a scan of empty space past the last occupied position. +- **Scrolling up or left, and a top-left that lands on a pruned row (the user, in + chat).** "The app is free to optimize fetching for up or left by just fetching the new + rows, until it finds one that has values in the viewport but, conceptually, scrolling up + or left would request a new viewport with a new top left corner. The requested top left + corner falling on a pruned row or column is an annoying edge case, maybe just fail the + fetch and retry?" (relayed on 2026-10-05; on no PR). The user's "maybe" is tentative, + and nothing here is decided. *The rest of this item is the author's analysis.* + - **The optimisation is a predecessor search.** Fetching only the new rows above the + window, and scanning upward until one has a value among the displayed columns, asks + "the previous row before p with a value in S". The forward-only queries above do not + serve it directly. There are two ways to serve it: + - **The client does it with forward requests.** It guesses an earlier top-left, + requests forward from there, and widens the guess if nothing qualifies. Each guess + costs one forward query, and the number of guesses grows with the gap. At the + fixture's density of 10^-4 a blind step of tens of rows usually finds nothing, so + it needs widening, about log₂(gap) guesses if each guess doubles the step. + - **The model offers a predecessor query.** It is cheap under every option. A sorted + `Vec` and a B-tree both give an upper-bound seek and a reverse iterator at the same + cost as the forward ones, O(log n) and then O(1) per step. Option (ii)'s per-column + sorted lists merge backwards exactly as they merge forwards. Options (i) and (iii) + walk backwards with the same per-row test. So offering it adds no state. It adds + only a second direction to the query surface, which the conceptual model (always + fill forward from a new top-left) does not need. *Author's lean:* offer it, because + it costs no state and saves the client a guessing loop. The user's to confirm. + - **The edge case: a top-left on a pruned (empty) row or column.** Listed neutrally: + - (a) **Fail, then retry** (the user's tentative "maybe"). The model answers with an + error. If the error names the nearest occupied successor (and predecessor), the + retry is one round trip. If it names nothing, the client must guess, as above. + - (b) **Snap forward, and report the actual top-left.** The model starts the fill at + the next occupied position at or after the requested one, and returns that + position in the response. No extra round trip. The client sees its window start + somewhere other than it asked. + - (c) **Snap toward the scroll direction.** Like (b), but snap backward when the + request came from an upward scroll. That needs the request to say which direction + it came from, which the request sketch does not carry. + + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the + ordered index, O(log n), in either direction, so (a) with named neighbours and (b) + cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap + to the next occupied row is on the order of 10^4 cross-product rows, so a client that + guesses blind under (a) would need about log₂(10^4) ≈ 14 doubling retries. **Unpruned + space is not an edge case at all**: a top-left on an empty position is answered with + empty cells. + - **How it fits identity-keyed deltas and resumable streams.** Under (b) and under (a) + with named neighbours, the response's actual top-left is an absolute position, and + the tuple there is an identity. So the stream's resumable state stays `(request, + actual top-left, last position)`, and a later delta names rows by identity whichever + top-left was asked for. A top-left whose row is pruned after the stream started is the + same case in time. A delta reports the row's removal, the client's window keeps its + absolute top-left, and the next fill skips past the empty position. That is (b)'s + behaviour without a new request. + - **The fixpoint comes from the fill, not from one window.** *As first written*, this item said that hiding a row can empty a column. **That was wrong.** A row dropped for having no value among the window's columns contributes no value to any of those @@ -1961,6 +2013,11 @@ revision, and none is renumbered. Under the user's right-and-down growth (clause 12), the query is a forward successor query only, and so are all three options; no rank or reverse scan is needed for pruning. + The user's later chat adds a client-side optimisation for scrolling up or left (a + predecessor search, which every option serves as cheaply as a successor search). It + also raises the edge case of a top-left that lands on a pruned row: fail and retry, + snap forward, or snap toward the scroll direction (clause 12). That edge case is + part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they are ADR 008's answers. From 885e8eb4693e9d99f9e79fd65f4846c503ce067c Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:10:01 -0700 Subject: [PATCH 23/67] docs(adr): ADR 009 records the objection to snap-forward and argues the alternatives cannot stick (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 32 ++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index c311be43..099361e9 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1415,6 +1415,35 @@ analysis.* request came from an upward scroll. That needs the request to say which direction it came from, which the request sketch does not carry. + **The user objects to (b)** (chat, relayed on 2026-10-05; on no PR): "Snap forward is + going to cause issues with the viewport getting stuck". *The author's reading of the + reason:* a client scrolling up asks for a top-left just above its window. If that + position is empty, (b) snaps forward to the first occupied position, which is the + window's old top-left. So the window never moves past an empty stretch above it. (b) + is therefore set aside, and (a) and (c) remain. **Both need the predecessor query**: + (a) to name the nearest occupied position before the request, and (c) to snap backward. + So the author's lean to offer a predecessor query is stronger now, and it is still the + author's. + + **Neither (a) nor (c) can get stuck, and neither oscillates** (the author's argument): + - **(a) moves the client.** The error names the nearest occupied predecessor and + successor. A client scrolling up retries at the predecessor, which is strictly + before its old top-left, so each scroll step makes progress. A client scrolling + down retries at the successor, which is strictly after. Neither retry can land on an + empty position, because both are occupied by definition. One retry suffices, unless + a write empties that position in between; then the next error names new neighbours, + still in the same direction. + - **(c) moves in the direction asked.** It snaps backward for an upward request and + forward for a downward one, so the actual top-left moves in the direction of the + scroll, or stays put only when there is nothing further that way. + - **At the bounds.** At the top of the axis there is no predecessor. Under (a) the + error names the successor only, and says there is no predecessor. Under (c) an + upward snap with nothing above it returns the first occupied position (the top of + the occupied axis) and says it is the top. Either way the client learns it is at + the edge and stops scrolling, so it does not bounce between "fail" and "retry". + The bottom-right corner is symmetric. An axis with **no** occupied position + answers "empty axis" under both, and the window is empty. + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the ordered index, O(log n), in either direction, so (a) with named neighbours and (b) cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap @@ -2016,7 +2045,8 @@ revision, and none is renumbered. The user's later chat adds a client-side optimisation for scrolling up or left (a predecessor search, which every option serves as cheaply as a successor search). It also raises the edge case of a top-left that lands on a pruned row: fail and retry, - snap forward, or snap toward the scroll direction (clause 12). That edge case is + snap forward (which the user objects to: "the viewport getting stuck"), or snap toward + the scroll direction (clause 12). That edge case is part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they From 3698c68025e42edc5593275bb22a11aa28f9d3a5 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:12:18 -0700 Subject: [PATCH 24/67] docs(adr): ADR 009 records the user's galloping top-left search and revises the predecessor lean (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 ++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 099361e9..79255763 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1444,6 +1444,44 @@ analysis.* The bottom-right corner is symmetric. An axis with **no** occupied position answers "empty axis" under both, and the window is empty. + **The user's simpler procedure** (chat, relayed on 2026-10-05; on no PR), quoted + exactly as sent: "The simplest might be some kind of binary search for a new top left + corner: increasing jumps of the top left until you get a hit and then move down/right + until + You only have one new row column." *The author's reading:* the procedure belongs to the + client and uses forward requests only. The client moves its requested top-left up (or + left) in growing jumps until a request returns an occupied row before its old + top-left, then narrows back down (or right) until exactly one new row (or column) is + gained. A pruned top-left is no longer an edge case at all: the response simply starts + at a later position than the one asked for. *The rest is the author's analysis.* + - **What it requires of the response:** the **actual** absolute position of every + row and column returned, the first one included. The client compares those + positions with its old top-left, so this is T1 again, and it already holds under + clause 12. Nothing else is needed: no error, no snap rule, no direction in the + request, and no predecessor query. + - **Cost.** With doubling jumps, the gallop takes about log₂(gap) requests. A plain + binary search over the last interval takes about log₂(gap) more. At density 10^-4 + the expected gap is about 10^4 rows, so that is 14 + 14 ≈ 28 forward requests, + against one predecessor request. The refinement is often much cheaper. A forward + response holds k occupied rows with their positions, so if the hit request's k rows + reach the old top-left, the client reads the last row before it directly, and the + search ends after the gallop, about 14 requests. Refinement is needed only when + more than k occupied rows lie between the hit and the old top-left. These figures + are arithmetic, not measured. + - **Statelessness and network transparency.** The host keeps nothing for the search. + Every request is an ordinary `project` from an absolute top-left, so the in-process + and remote consumers do exactly the same thing (T6). The cost is round trips: about + 14 for a remote client and about 14 function calls for the TUI. For the TUI that is + nothing. Over a socket it is about 14 latencies for one scroll step into a sparse + stretch. + - **The author's lean, revised.** Take the user's procedure as the **baseline**. It + makes (a), (b) and (c) unnecessary, because a pruned top-left just returns a later + first position. Keep a model-side predecessor query only as an **optional + optimisation** for a remote client. It costs no state (see above) and collapses + the gallop into one request. Offer it once a measured round-trip cost shows it is + worth a second query direction. *As first written*, this item leaned to offering it + outright; that lean is withdrawn. + Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the ordered index, O(log n), in either direction, so (a) with named neighbours and (b) cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap @@ -2046,7 +2084,11 @@ revision, and none is renumbered. predecessor search, which every option serves as cheaply as a successor search). It also raises the edge case of a top-left that lands on a pruned row: fail and retry, snap forward (which the user objects to: "the viewport getting stuck"), or snap toward - the scroll direction (clause 12). That edge case is + the scroll direction (clause 12). The user has since suggested a simpler client procedure: + gallop the top-left with forward requests until one hits, then narrow. A pruned + top-left then needs no special answer, provided responses report actual positions. + The author now takes that as the baseline, and keeps a predecessor query only as an + optional optimisation. That edge case is part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms that pruning is the client's. (c) restate both in those terms, in ADR 008, since they From 71a56234792f5963bbbb22d19ecae9e3c9b702aa Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:30:55 -0700 Subject: [PATCH 25/67] docs(adr): ADR 009 answers the review of 3698c680 (improvise-felo.4) B1: clause 7 splits a shape key from read-time parameters and says view_shape is not the key; clause 4, Q7 and Q11 follow. B2: one numbered list of the twelve relayed chat statements. B3: withdrawn leans marked superseded; clause 12's pruning text rewritten for the gallop baseline. Also: clause 5a's fallback is answer-preserving (agg_values pools only member_records), ADR 007 keys, T5 per-subscription coalescing, item edits reach every subscriber, fold Sum exactness, owners for questions shared with ADR 008, re-derived citations. Reads ADR 008 at 3180327b. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 964 ++++++++++------------ 1 file changed, 454 insertions(+), 510 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 79255763..3fc94606 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 21** -(PR #308, read at `3b70a0f1`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 24** +(PR #308, read at `3180327b`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `3b70a0f1`, its revision 21) · +reads it at `3180327b`, its revision 24) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 21 of `context/adr/008-virtualized-grid.md` on PR #308, at `3b70a0f1`. Clauses 3, +revision 24 of `context/adr/008-virtualized-grid.md` on PR #308, at `3180327b`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,37 +78,52 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 21, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 24, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **The two chat statements** were given to the orchestrator and relayed to this ADR. They - are on no PR, and ADR 008 records them the same way. The user's "Approved" on this ADR - would confirm them. +- **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed + to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR + would confirm them. ADR 008 records the same statements, also as relayed and not on the + PR, but spread over several groups of its own. This list is the one place this ADR + quotes them, and the clauses cite them by number. None of this answers this ADR's own questions 1 to 11. -**Routed here, in chat:** -- "The DBSP machinery needs to be implemented in a network transparent way, but the - details of that go to ADR 009" (chat, relayed on 2026-10-05). Clause 12. -- "Thinking about it a bit more, we might need to keep track of recent projections + - clients to stream aggregation results back to the UI correctly. But this should be as - stateless as possible" (chat, relayed on 2026-10-05). Clause 12. -- On ADR 008's question 53: "Q53 is cross product: I'm still ambivalent about which side - of the wire pruning should happen on" (chat, relayed on 2026-10-05). So a request's - top-left is a cross-product position, and which side prunes is open. ADR 008 records it - at revision 19 as its question 53 answered and its question 57 open. Clause 12, - question 19. -- "But the top left should be as absolute as possible" (chat, relayed on 2026-10-05). - Clause 12, question 20 (ADR 008's question 58). -- "After pruning we might have to stream rows and columns to attempt to fill out the full - grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59). -- "Pruning is just a filter applied to full columns a rows, right? Hide a column/row in - the grid if it's all null. That sounds like a UI side concern because that's where the - visible grid lives" and "The point of pruning is to make the displayed values more - dense, so a value being present outside the viewport doesn't matter." (chat, relayed on - 2026-10-05). Clause 12, question 22. +**Relayed in chat, on no PR** (all on 2026-10-05; quoted exactly as relayed): +- **C1.** "The DBSP machinery needs to be implemented in a network transparent way, but + the details of that go to ADR 009". Clause 12. +- **C2.** "Thinking about it a bit more, we might need to keep track of recent projections + + clients to stream aggregation results back to the UI correctly. But this should be as + stateless as possible". Clause 12. +- **C3.** "Q53 is cross product: I'm still ambivalent about which side of the wire pruning + should happen on". ADR 008 records it as its question 53 answered and its question 57 + open. Clause 12, question 19. +- **C4.** "But the top left should be as absolute as possible". Clause 12, question 20. +- **C5.** "After pruning we might have to stream rows and columns to attempt to fill out + the full grid". Clause 12, question 21. +- **C6.** "Pruning is just a filter applied to full columns a rows, right? Hide a + column/row in the grid if it's all null. That sounds like a UI side concern because + that's where the visible grid lives". Clause 12, question 22. +- **C7.** "The point of pruning is to make the displayed values more dense, so a value + being present outside the viewport doesn't matter." Clause 12, question 22. +- **C8.** "The problem pruning solves is visualizing sparse datasets: screens that are + mostly empty are useless and so you drop out categories from rows and columns until + every visible row or column has at least one value on screen." Clause 12, question 22. +- **C9.** "We should be able to optimize pruning on the right and down assumption of + growth (truncated at the bottom right corner of the axis space)". Clause 12; ADR 008's + question 67. +- **C10.** "The app is free to optimize fetching for up or left by just fetching the new + rows, until it finds one that has values in the viewport but, conceptually, scrolling up + or left would request a new viewport with a new top left corner. The requested top left + corner falling on a pruned row or column is an annoying edge case, maybe just fail the + fetch and retry?" Clause 12, question 22(d). +- **C11.** "Snap forward is going to cause issues with the viewport getting stuck". + Clause 12, question 22(d). +- **C12.** "The simplest might be some kind of binary search for a new top left corner: + increasing jumps of the top left until you get a hit and then move down/right until + You only have one new row column." Clause 12, question 22(d). **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -187,14 +202,16 @@ the clause it changes: 3. **State is keyed by the request's shape, not by a view's name.** Axes are a model concept, so operator state keyed by axes is model state, not UI state. But a `project` request names its own axes, and an unsaved request may carry parameters no saved view - holds (ADR 008's flagged reading 1, its question 51). Clause 7, question 14. + holds (ADR 008's flagged readings 1 and 3, its question 51). Clause 7, question 14. 4. **The record order belongs to the store.** The ordered list of records is the store's (Q27). The circuit may consume it, and does not own it. Clause 9. 5. **The memory requirement covers this ADR too.** The store-memory requirement covers the circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. Clause 14, question 17. -6. **Five of ADR 008's questions are answered.** Those are questions 1, 2, 6, 13 and 15, - and clauses 5a, 9, 13 and 14 now follow the answers. +6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's + questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 42 and + 43, quoted above. + Clauses 5a, 7, 9, 12, 13 and 14 follow them. ## Context @@ -210,7 +227,8 @@ a change proportional to what the change touches, not to the document. ### Evaluation today is a from-scratch fold, rerun after every batch - **Every batch recomputes every formula cell.** `App::apply_effects` - (`crates/improvise-command/src/ui/app.rs:989`) ends in `rebuild_layout` (`:880-891`). + (`crates/improvise-command/src/ui/app.rs:1069` at `50e1314d`) ends in `rebuild_layout` + (`:926`). That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` @@ -227,14 +245,14 @@ a change proportional to what the change touches, not to the document. (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every combination of the spanned categories' items. ADR 008 clause 5 names this as the one O(grid) path left in totals. -- **Two observations scan the store.** `record_fingerprints` (`app.rs:254-264`) hashes +- **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes every cell before and after a cell-writing batch. `document_content` - (`app.rs:165-180`) hashes every cell and every formula for the dirty flag, and it is + (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) per batch and the second O(cells) per effect, whatever was written. - **Every mutating frame from a client already clones the document.** - `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:162`, - `app.rs:627-629`), for ADR 001 clause 4. That is O(cells) per frame on the network + `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, + `app.rs:673` at `50e1314d`), for ADR 001 clause 4. That is O(cells) per frame on the network path, whatever this ADR does. ### What an incremental evaluator must not change @@ -335,7 +353,7 @@ name is `Derived`. It replaces the two places where eager evaluation folds the s `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). It takes two inputs, both applied after each model effect (clause 7): the cell delta, -and the formula delta. It also has a rebuild, for an operator shape that moved. +and the formula delta. It also has a rebuild, for a new shape key (clause 7). **The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code and read through the trait. The circuit decides *which* formula cells to re-evaluate @@ -390,7 +408,7 @@ today.** Two elements have the same `(key, value)` part exactly when their keys equal and their values are equal *by bits*, and bit identity is stated over that part only (the sequence is compared as the integer it is, which keeps an equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and -`Error` by their strings. That is `hash_value` (`app.rs:233-250`), which +`Error` by their strings. That is `hash_value` (`app.rs:246-262` at `50e1314d`), which `record_fingerprints` already hashes. So the cell delta reproduces today's tier exactly: @@ -415,12 +433,12 @@ in the batch's consolidated delta. So: keys. `bump_changed_records` can then read the delta instead of diffing two -`record_fingerprints` scans (`app.rs:254-264`). The tier is still *observed*: the delta +`record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, because it observes `view_shape` and not cells. **The dirty flag's content half does not move onto the cell delta alone.** -`document_content` (`app.rs:165-180`) hashes the formulas as well as the cells, and it +`document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a @@ -436,6 +454,25 @@ new is observed. Its cost is O(formulas), where `document_content` is O(cells) p `PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by `f64::total_cmp`. Clause 4 says where that differs from `==`. +**Names and ADR 007 (added on 2026-10-05).** ADR 007 is accepted. It makes ids the keys +and names display-only, and in it a `CellKey` becomes sorted `(CategoryId, ItemId)` pairs. +The user said there: "This is a lot lower value if CellKeys, etc. don't refer to the new +id". Every key this ADR defines follows `CellKey`: the cell delta, `CellKey`'s `Ord`, the +stems, `record_of`, the group keys, the shape key's categories, and the output's tuple +identities (clause 12). So **under ADR 007 they are id-keyed with no change to this ADR, +and a rename emits no cell delta and rebuilds no operator.** It is one `display` write, +which the output carries as a relabelling, not as a retraction and an insertion. Two +places do not follow `CellKey`, and must move with ADR 007: +- **the formula fingerprint** (clause 7). It is taken over `(raw, target, + target_category)`, which are names today. Under ADR 007 it must be taken over the + formula's id-form, or a rename would read as a formula delta and re-evaluate every + affected target; +- **the cross-product order** (clause 12), which orders items by the category's own + order. That order is unchanged by a rename. + +Until ADR 007 lands, a rename is what it is today: a remove and an insert of every cell +under the old name, and this ADR's deltas say exactly that. + ### 4. Aggregates: linear where they can be, an integrated multiset where they cannot **The group** is the cell key with the view's `none_cats` removed. That is the partial @@ -502,11 +539,12 @@ key, and `FIRST` the least. the group, and one entry per contributing record. That is the same order as `Min`/`Max`'s multisets. - **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. **Most of that is already in the - operator shape**: `view_shape` hashes each category's *drawn* items, in order - (`app.rs:47-52`), so a reorder of drawn items already rebuilds. What it does not see - is the position of a **hidden** item, which matters only if `LAST`/`FIRST` must order - values that come from hidden items. The alternative is to key by a stable item + reordering its items, changes every stored position. *As first written*, this item + said a reorder of drawn items already rebuilds, through `view_shape`. **That is + superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a + reorder rebuilds nothing, and a state keyed by position would go stale. The position of + a **hidden** item matters only if `LAST`/`FIRST` must order values that come from + hidden items. The alternative is to key by a stable item identity (ADR 007) and look the order up at read time. That makes a reorder free, but a read then costs O(d). This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, @@ -681,15 +719,20 @@ is one evaluator. that its value at a record changes only when that record changes. An inner expression that reads another record, holds a nested aggregate, or refers to a formula whose value depends on other records, is not a function of one record. **Revised on 2026-10-05:** -such a formula does **not** stay on today's eager path, because that path is -`Expansion::members`, the cross product (ADR 008 clause 5, a finding of Copilot's on -#308). It takes ADR 008's **store-driven fallback**, which evaluates the expression only -over the **occupied contexts**, the contexts some record projects to. The user accepted -that recommendation, as ADR 008's question 15 ("the recommendation is fine."). It differs -from today's eager answer wherever the expression is non-empty at a context that holds no -record, and ADR 008 carves those cases out of its differential test by name. Whether this -ADR's `Eager` adopts the same rule, so that its own differential test needs no carve-out, -is question 18. Two regressions pin the record stage against eager: +such a formula is not maintained, and at a fold or a total it does **not** enumerate +today's eager contexts. Those are `Expansion::members`, the cross product (ADR 008 clause +5, a finding of Copilot's on #308). At a stem, eager pools one context and there is no +product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's +**store-driven fallback**, which evaluates the expression only over the **occupied +contexts**, the contexts some record projects to. The user accepted that recommendation, +as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as +eager's**, as the paragraph above says of the pipeline. `agg_values` evaluates the +expression only at the `member_records` of each context (`types.rs:47-53`, `:91-139`), +so a context with no record contributes no value and no error, whatever the expression. +The fallback changes the cost and not the answer. *The first revision of 2026-10-05 said +it "differs from today's eager answer" wherever the expression is non-empty at an empty +context. That was wrong, and ADR 008's question 15 carries the same premise.* Whether +this ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -701,9 +744,9 @@ another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A r whose key `record_of` takes without `_Measure`, then belongs to the context only through its cell under that measure, and eager orders it by *that* cell's position, not by the record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. **That -eager path is `Expansion::members`, so while it lasts it blocks ADR 008's retirement of -`Expansion::members` (its migration step 6) for these formulas.** Sending them to the +formulas stay on the eager path, and the differential test pins that they do. **At a fold +or a total that eager path is `Expansion::members`, so while it lasts it blocks ADR 008's +retirement of `Expansion::members` (its migration step 6) for these formulas.** Sending them to the occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** @@ -759,44 +802,67 @@ post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1 That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 states the same rule. -- **Operator shape, per projection shape (revised on 2026-10-05).** Operator state is - keyed by the **operator shape of a `project` request**, not by the name of a view - (question 14). The shape is made of: +- **A shape key, and read-time parameters (revised on 2026-10-05).** Operator state is + selected by the **shape key** of a `project` request, not by the name of a view + (question 14). The key is exactly what changes which cells feed which operator: - the categories of each axis, by level; - the None categories; - - the filter and page keys; - - the hidden items and collapsed groups the request carries; + - the filter keys: each `Axis::Filter` category with its selected item, and the hidden + items of every category that is **not** on an axis (such an item drops cells from the + fold, so it filters); + - the page **categories**, which key the slices under question 7's recommendation; - `measure_agg`. - A saved `.improv` view supplies these to a request that names it. An unsaved request - carries its own, for example an unsaved prune or hide (ADR 008's flagged reading 1, its - question 51). Requests with the same shape share one set of operators, whichever - clients sent them. The live shapes are a **cache**. A shape enters when a request names - it, and it is evicted once no recent request has named it (clause 12). For a saved - view the shape still includes what `view_shape` (`app.rs:40-72` at `57d385e1`) hashes, - plus the group membership of every drawn item and `measure_agg`. `view_shape` itself is - not widened, so what bumps `layout_version` is unchanged. + Everything else a request carries is a **read-time parameter**, applied when the + output is read, without touching operator state: + - the hidden items of an axis category; + - the collapsed groups, and the group membership of axis items; + - the drawn order of items; + - the page **selection**, which picks a slice; + - prune, which under client-side pruning is not the host's at all (clause 12). - *As first written* (superseded): a view whose operator shape moved had its operators - rebuilt **before the frame observation that follows this effect**. That rule served the - per-view frame observation, which ADR 008 clause 8 revises under the split. The three - rules below replace it. + **`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72` at `50e1314d`) + hashes each category's drawn items in order (hidden items excluded), its collapsed + groups, its page selection and `prune_empty`. Those are all read-time parameters except + the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, + `AddItem`, reorder and page step, which is what item grain and question 7 exist to + avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40), and it is + unchanged. + + A saved `.improv` view supplies the key and the parameters to a request that names it. + An unsaved request carries its own (ADR 008's flagged readings 1 and 3, its question + 51). + Requests with the same key share one set of operators, whichever clients sent them. The + live keys are a **cache**: a key enters when a request names it, and it is evicted once + no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a + shape key. + + *As first written* (superseded twice): the operator shape was `view_shape` plus group + membership and `measure_agg`, and a view whose shape moved had its operators rebuilt + **before the frame observation that follows this effect**. The first revision of + 2026-10-05 still listed hidden items and collapsed groups in the shape, and still + deferred to `view_shape` for a saved view. That contradicted the item-grain rule below + and question 7, as the review of `3698c680` found. - **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column aggregates are keyed by the **items** of the axis categories, together with the filter and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no operator. The visible axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms - for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item changes no - operator either, because a new item tuple is just a new key. Three things still - rebuild a shape: - - a change of which categories are its levels; - - a change of its filter or page keys, which includes hiding an item of a category that - is not on an axis; + for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or + reordering items, changes no operator either: a new item tuple is just a new key, and + the order is a read-time parameter. Only a change of the **shape key** selects other + operators, which a new key builds as a job: + - a change of which categories are its levels or on None; + - a change of its filter keys, which includes hiding an item of a category that is not + on an axis; + - a change of its page categories; - a change of `measure_agg`. A collapsed group's value is a pooled fold over its members. With item grain it is - combined from the members' state when it is read. For `Sum` and `Count` that combine is - linear. For the integrated aggregates it costs O(members) per read, and whether that + combined from the members' state when it is read. For `Count` that combine is exact. + For `Sum` it is exact only under question 3's (c), or under ADR 005's wide accumulator. + A sum of the members' f64 sums is not bit-equal to eager's one store-order fold + (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a @@ -807,7 +873,9 @@ states the same rule. Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and priority queue the user required. This ADR only makes its rebuilds steppable and - cancellable between slices. + cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user + ruled. Any measurement this ADR later reports against them is stated as a ratio + against an unrelated operation on the same run, as ADR 008's question 13 requires. - **A crossing count (ADR 008's second request).** Each step's occupancy output carries the number of tuples whose count crossed zero. ADR 008's ordered index uses it to choose between an insertion and a rebuild by sorting. @@ -822,11 +890,10 @@ states the same rule. - **A document swap rebuilds every circuit** from the new store. `install_workbook` bumps `document_generation`, and the same after-each-effect read sees it. -Re-reading the operator shape costs O(items) per live shape per model effect, and the formula -fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs -O(cells), once per actual reshape. The same `view_shape` function is therefore read at -two moments: by the circuit after each effect, and by `layout_version` after the loop. -They answer different questions, and neither replaces the other. +Re-reading the shape keys costs O(categories) per live key per model effect, and the +formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a +new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by +`layout_version` after the loop, as today. **The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first written* (superseded by item grain): `AddItem` moved the operator shape, so the view was @@ -836,7 +903,7 @@ key, and only the visible axis, which is derived, gains a row. The case needs no ordering rule at all. **Recommended: a page selection is not a rebuild (question 7).** Under the recommended -answer, the group key includes the page categories, so `[` and `]` read another slice +answer, the shape key includes the page categories and not the page selection, so `[` and `]` read another slice of state that already exists. The held-key path stays O(visible) (improvise-748f's constraint). The slices cost nothing extra, because the state is still bounded by the stored cells, however many pages there are. Filter coordinates still filter; only page @@ -1080,7 +1147,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 21 at `3b70a0f1`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 24 at `3180327b`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1168,7 +1235,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:647`). **Recommended (question 9): the backend's state joins the snapshot.** + (`app.rs:693` at `50e1314d`). **Recommended (question 9): the backend's state joins the snapshot.** Applying the inverse delta is **not** a rollback, even with exact aggregators. It restores integrated values, but `step(d)` then `step(−d)` still advances the circuit and changes the history that `delay` and `differentiate` hold. So the next step after @@ -1178,7 +1245,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n **The snapshot costs a second O(cells) clone per mutating frame.** `Host::mutating_frame` already clones the workbook on every mutating frame - (`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So + (`host.rs:224-225`, `app.rs:673` at `50e1314d`), and the integrated state is of the same order. So on the network path a write is not O(delta): it is O(cells) for the snapshot today, and roughly twice that with the circuit in it. The keyboard path (`apply_effects` without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the @@ -1190,8 +1257,8 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n ### 12. Network transparency: one circuit, consumed in process or across a wire -**The requirement is the user's:** "The DBSP machinery needs to be implemented in a -network transparent way, but the details of that go to ADR 009" (chat). **Everything +**The requirement is the user's** (C1): "The DBSP machinery needs to be implemented in a +network transparent way, but the details of that go to ADR 009". **Everything else in this clause is this ADR's design, and the author's choices are marked as such. None of it is measured.** @@ -1217,8 +1284,9 @@ Three things do not cross: **What a projection request is to the circuit** (the author's answer to "is the result keyed by request, by axes, or shared?"). A request has two parts. -- **Its operator shape** (clause 7) selects the circuit's state. The shape is shared by - every request, from any client, that has it. +- **Its shape key** (clause 7) selects the circuit's state. The key is shared by every + request, from any client, that has it. Its read-time parameters (hidden and collapsed + axis items, drawn order, page selection) are applied to that state's output. - **Its window**, the top-left and the dimensions, selects a part of that state's output. The window is never state. @@ -1226,330 +1294,205 @@ So maintained results are keyed by **shape**, and the only per-request work is restricting a shape's output to a window. That is O(visible) per request and needs nothing kept between requests. -**Positions are cross-product positions.** The user answered ADR 008's question 53 in -chat: "Q53 is cross product: I'm still ambivalent about which side of the wire pruning -should happen on" (relayed on 2026-10-05; on no PR). So a request's top-left is a position -in the **cross product** of its axes, in each category's own item order. That position is -a bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed +**Positions are cross-product positions** (C3). A request's top-left is a position in the +**cross product** of its axes, in each category's own item order. That position is a +bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed contexts, we can represent bigints as strings in JSON"). A cross-product position names the same tuple whatever is occupied, so **an occupancy change never moves a request's window**. It changes only which positions inside the window hold data. **Which side of -the wire prunes is open** (the user is "still ambivalent"), and is question 19. The -circuit must serve both sides, and what it must expose in each case is the author's -analysis: -- **Pruning on the model side.** `project` returns occupied rows and columns. The circuit - must then answer two order-statistic queries over a shape's visible occupancy, from a - cross-product position: - - the next k occupied tuples at or after a position, so a pruned window can start where - the request points; - - the rank of a tuple among the occupied ones, and the occupied tuple of a given rank, - so a client can scroll a pruned axis by rows and show "row i of n". +the wire prunes is open.** C3 says the user is "still ambivalent", and C6 says pruning +"sounds like a UI side concern". That is question 19. The rest of this clause is the +author's analysis, and it is written for **client-side pruning** first, because that is +the author's lean after C6 to C8. Where model-side pruning would differ, it says so. - That is ADR 008's ordered index over occupied tuples (its clause 3), the sorted `Vec` - first and the order-statistic B-tree when measured (its question 16), fed by this ADR's - occupancy delta and crossing count. Under this case a client sees only occupied rows, - and its scroll space is the pruned one, built on top of the cross-product offset. -- **Pruning on the client side.** `project` returns the cross-product window, empty - positions included, together with a **skip structure** the client can query: for each - axis, the occupied runs that intersect the window and the next occupied position past - each edge. The circuit then has to expose the same order-statistic queries, but as a - model fact a client can request, not as a re-indexing the model performs on its behalf. - The cost is that a sparse grid at 3 × 1,000 (10^9 positions, about 10^5 occupied in ADR - 008's fixture) sends mostly empty windows unless the client asks for skips first. - -Both cases need the same maintained structure. They differ only in who calls it, and in -whether a `project` result is already pruned. Prune would be a request parameter in the -first case (ADR 008's flagged reading 1) and a client policy in the second. - -**"As absolute as possible", and what still moves a position.** The user, in chat: "But -the top left should be as absolute as possible" (relayed on 2026-10-05; on no PR). *The -rest of this paragraph is the author's analysis.* A cross-product position does not move -when occupancy changes, pruning included. **It does move when an item is inserted, -removed or reordered** in a category's own order. Positions are mixed-radix numbers over -the items of each level, so inserting item j at level L shifts every later item of L, and -changes the radix that every outer position is multiplied by. No position over a -changing item list is fully absolute. Three things follow: +**"As absolute as possible", and what still moves a position** (C4). *The author's +analysis.* A cross-product position does not move when occupancy changes, pruning +included. **It does move when an item is inserted, removed or reordered** in a category's +own order. Positions are mixed-radix numbers over the items of each level, so inserting +item j at level L shifts every later item of L, and changes the radix that every outer +position is multiplied by. No position over a changing item list is fully absolute. +Three things follow: - **Deltas are keyed by identity, and positions are attached when sent.** An output element names its row and column by tuple identity: item ids under ADR 007, and names - until then. The host attaches the absolute position from the current item order when it - sends the element. A delta keyed by position alone would go stale with the next item - insert. -- **An item edit is sent as an axis change, not as moved rows.** "Item inserted at index - j of level L", or removed, or moved. A client re-derives every position it holds - arithmetically, at O(levels) per position, and its top-left can be re-anchored to the - same tuple. That is the cursor's identity rule from ADR 008 clause 11 applied to the - window. Resending every row whose position moved would cost O(rows after j). + until then (see "Names and ADR 007" in clause 3). The host attaches the absolute + position from the current item order when it sends the element. A delta keyed by + position alone would go stale with the next item insert. +- **An item edit is sent to every subscriber as an axis change.** "Item inserted at index + j of level L", or removed, or moved. It goes to **every** subscriber of a shape with + that category on an axis, wherever its window lies, because an insert before a + window's top-left shifts every position that subscriber holds. A client re-derives its + positions arithmetically, at O(levels) per position, and re-anchors its top-left to the + same tuple, as ADR 008 clause 11 anchors the cursor. Resending every row whose position + moved would cost O(rows after j). - **The cost is rank and select over each category's items.** "The position of item x" and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` first, a B-tree when measured, its question 16). A tuple's position is O(levels × log - items). The ordered occupancy index must compare tuples by **current** item position, + items). An index ordered by position must compare tuples by **current** item position, not by a position stored at insert time. With item grain (clause 7) its keys are item - identities, so an item insert re-orders nothing in it, and comparison goes through the - order-statistic lookup, at O(levels × log items) per comparison. That is **not - measured**. Question 20. + identities, so an item insert re-orders nothing in it, and a comparison goes through + the order-statistic lookup, at O(levels × log items). That is **not measured**. + Question 20. -**A pruned answer may arrive as a stream.** The user, in chat: "After pruning we might -have to stream rows and columns to attempt to fill out the full grid" (relayed on -2026-10-05; on no PR). *The author's analysis:* under model-side pruning, `project` scans -forward from the absolute top-left over each axis's occupancy until it has the requested -number of occupied rows and columns, or reaches the end of the axis. It sends the rows -and columns it finds as it finds them. What the circuit must support: -- **Skipping empty space in O(log n + k).** "The next k occupied tuples at or after a - position" over the ordered occupancy index. It must never walk empty cross-product - space, which at 3 × 1,000 is mostly empty. -- **The absolute position of every row and column sent**, so that a client can place it, - and a later delta can name it, without counting. -- **Bounded work per slice.** The fill is a job under ADR 008 clause 11's runner when k - is large, and it can be resumed from the last absolute position it sent. So the only - state a fill keeps is `(request, fill front)`, which is "as stateless as possible" in - the user's words. A lost fill restarts from the client's last known position. -- **Changes during a fill are identity-keyed deltas, stamped by step** (T3). Suppose a - row becomes occupied between the top-left and the fill front after it was passed. The - subscriber receives it as an ordinary occupancy delta carrying its absolute position, - so its window now holds one row more than asked, and it drops or keeps the extra row by - client policy (Q42). A row that becomes occupied ahead of the front is simply found by - the fill. Each chunk of the fill is a delta to the window's Z-set, stamped like any - other. So a fill and a live update are one stream, which keeps the in-process and the - remote consumer the same (T6). - -**Pruning is a filter on the values displayed (the user, later in chat).** "Pruning is -just a filter applied to full columns a rows, right? Hide a column/row in the grid if it's -all null. That sounds like a UI side concern because that's where the visible grid lives" -and "The point of pruning is to make the displayed values more dense, so a value being -present outside the viewport doesn't matter." (both relayed on 2026-10-05; on no PR). -**This is a change from today's code**, and it is recorded here, not decided. Today, -`prune_empty` drops a row only when it is empty across the **whole** grid, and prunes the -columns against the same unpruned grid (`layout.rs:752-757` at `50e1314d`). In the user's -words, emptiness is judged over the **displayed** values, so a row that holds a value only -in a column outside the window is hidden. *The rest of this paragraph is the author's -analysis.* -- **What the model then owes a client** is not a pruned axis. It owes windows (an - absolute top-left and dimensions) and a **restricted skip query** for the client's fill - loop: "the next k rows, from absolute position p, that hold a value in at least one of - these columns", and the same with rows and columns swapped. The question is about a set - of columns, not about the whole grid, so a per-axis count cannot answer it. A row can be - occupied overall and still be empty in every column the client shows. -- **Three ways to answer the query**, all unmeasured: +**Pruning as a filter on the displayed values** (C6, C7). **This is a change from today's +code**, and it is recorded here, not decided. Today, `prune_empty` drops a row only when +it is empty across the **whole** grid, and prunes the columns against the same unpruned +grid (`layout.rs:752-757` at `50e1314d`). The user's intent, as ADR 008 reads it (its +questions 60 to 65), is that emptiness is judged over the **displayed** values, so a row +that holds a value only in a column outside the window is hidden. *The rest of this +paragraph is the author's analysis.* +- **What the model owes a client** is then not a pruned axis. It owes cross-product + windows (an absolute top-left and dimensions) and a **restricted successor query**: + "the next k rows after absolute position p that hold a value in at least one column of + the set S", and the same with rows and columns swapped. **S travels in the call**, + because only the client knows which columns it displays. A per-axis count cannot + answer it: a row can be occupied overall and still be empty in every column the client + shows. +- **Three ways to answer it**, all unmeasured: - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then - test each candidate row against the column set by lookup. The state is the per-axis - counts clause 9 already keeps. The worst case scans many rows that are occupied - overall but empty in the column set, at O(|columns|) lookups each. + test each candidate against S by lookup. The state is the per-axis counts clause 9 + keeps. The worst case scans many rows that are occupied overall but empty in S, at + |S| lookups each. - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that - hold a value in it. The query is a k-way merge of |columns| sorted lists from p, at - O(|columns| × log n + k × |columns|). The state is O(occupied cells at grid grain), - the two-dimensional count clause 9 does not keep. It is maintained from the cell delta - and from formula stems (clause 5's stem rule decides when a formula cell holds a - value). A collapsed group's column merges its members' lists when it is read. + hold a value in it. The query seeks each of S's lists to p and merges them forward, + choosing the minimum by a linear scan over the |S| cursors: O(|S| × log n + k × |S|). + The state is O(occupied cells), the two-dimensional count clause 9 does not keep. It + is maintained from the cell delta and from formula stems (clause 5's stem rule decides + when a formula cell holds a value). A collapsed group's column merges its members' + lists when it is read. - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, - intersected with the column set as the fill walks forward. It has the same state as - (ii), and its scan behaves like (i), with an exact test per row. + tested against S as the scan walks forward. It has the same state as (ii), and it + scans like (i), with an exact test per row. The author leans to (ii): a window has tens of columns, so the merge is small, and it never walks rows that are empty in the window. Its cost is a second O(occupied cells) - structure, beside the store (clause 14). -- **Growth is only rightward and downward (the user, in chat).** "We should be able to - optimize pruning on the right and down assumption of growth (truncated at the bottom - right corner of the axis space)" (relayed on 2026-10-05; on no PR). *The rest of this - item is the author's analysis.* The fill loop only grows away from the absolute - top-left, and it stops at the end of each axis. So every skip query is a one-directional - **successor** query: "the next row (or column) after position p with a value among this - set", bounded by the end of the axis. Pruning never needs a predecessor or a reverse - scan. Scrolling up or left is not a reverse fill. It is a **new request with a new - absolute top-left**, filled forward from there, and nothing from the old request has to - be walked backwards. What that simplifies: - - **The index options.** (ii)'s per-column sorted lists need only a seek (a lower - bound at p) followed by forward iteration. A k-way merge of forward cursors keeps - O(|S|) state and never steps back. Option (i)'s pre-filter and option (iii)'s per-row - test also walk forward only. So any ordered structure with a lower-bound seek and a - forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 - question 16). No order statistics (rank, or "row i of n") are needed for pruning. - They remain only for a client that wants a scrollbar over an unpruned axis, where - rank is arithmetic over the cross product anyway. - - **Maintenance.** An insert or removal at position q affects only fills whose front is - already past q, and only within their window. A fill whose front is before q finds the - change when it gets there. So a subscription needs a delta only for positions between - its top-left and its front, which is a range test against two absolute positions. - - **Resumable streams.** A fill's whole state is its request and its last absolute - position on each axis. A resume is a seek past that position, so a lost stream, a - reconnect or a sliced job (ADR 008 clause 11) all resume the same way, with nothing - else kept. Positions are stable under occupancy changes because they are - cross-product positions. An item insert before the front shifts them, which is - question 20's axis-change message: the client re-derives its front arithmetically. - - **The corner truncates the work.** The fill is bounded by the requested dimensions - and by the bottom-right corner of the axis space, so its cost is O(|S| × log n + k × - |S|) per round, never a scan of empty space past the last occupied position. + structure beside the store (clause 14). Index (ii) is a set of `(column, row)` pairs, + so it is itself a Z-set fed from the cell delta and from formula stems. A write costs + O(log n), and its delta, "row r gained or lost a value in column c", is what a + subscription restricted to S receives. +- **Under model-side pruning instead**, the model would keep a pruned axis per shape: ADR + 008 clause 3's ordered index over occupied tuples, with rank and select for "row i of + n". Pruning would be a request parameter (ADR 008's flagged reading 1), and the host + would judge "pruned". Everything below that mentions a host judging emptiness applies + only to this alternative. -- **Scrolling up or left, and a top-left that lands on a pruned row (the user, in - chat).** "The app is free to optimize fetching for up or left by just fetching the new - rows, until it finds one that has values in the viewport but, conceptually, scrolling up - or left would request a new viewport with a new top left corner. The requested top left - corner falling on a pruned row or column is an annoying edge case, maybe just fail the - fetch and retry?" (relayed on 2026-10-05; on no PR). The user's "maybe" is tentative, - and nothing here is decided. *The rest of this item is the author's analysis.* - - **The optimisation is a predecessor search.** Fetching only the new rows above the - window, and scanning upward until one has a value among the displayed columns, asks - "the previous row before p with a value in S". The forward-only queries above do not - serve it directly. There are two ways to serve it: - - **The client does it with forward requests.** It guesses an earlier top-left, - requests forward from there, and widens the guess if nothing qualifies. Each guess - costs one forward query, and the number of guesses grows with the gap. At the - fixture's density of 10^-4 a blind step of tens of rows usually finds nothing, so - it needs widening, about log₂(gap) guesses if each guess doubles the step. - - **The model offers a predecessor query.** It is cheap under every option. A sorted - `Vec` and a B-tree both give an upper-bound seek and a reverse iterator at the same - cost as the forward ones, O(log n) and then O(1) per step. Option (ii)'s per-column - sorted lists merge backwards exactly as they merge forwards. Options (i) and (iii) - walk backwards with the same per-row test. So offering it adds no state. It adds - only a second direction to the query surface, which the conceptual model (always - fill forward from a new top-left) does not need. *Author's lean:* offer it, because - it costs no state and saves the client a guessing loop. The user's to confirm. - - **The edge case: a top-left on a pruned (empty) row or column.** Listed neutrally: - - (a) **Fail, then retry** (the user's tentative "maybe"). The model answers with an - error. If the error names the nearest occupied successor (and predecessor), the - retry is one round trip. If it names nothing, the client must guess, as above. - - (b) **Snap forward, and report the actual top-left.** The model starts the fill at - the next occupied position at or after the requested one, and returns that - position in the response. No extra round trip. The client sees its window start - somewhere other than it asked. - - (c) **Snap toward the scroll direction.** Like (b), but snap backward when the - request came from an upward scroll. That needs the request to say which direction - it came from, which the request sketch does not carry. +**The fixpoint comes from the fill, not from one window** (C8). *The author reads C8* as +a fixpoint on the displayed grid: every visible row has a value among the visible +columns, and every visible column has one among the visible rows. C8 says "drop out +categories", which the author reads as dropping rows and columns, that is, items or +tuples. The user may mean dropping a whole category from an axis. ADR 008 flags the same +ambiguity (its question 66(a)), and it is the user's to settle. *As first written*, this +clause said that hiding a row can empty a column. **That was wrong.** A row dropped for +having no value among the window's columns contributes no value to any of those columns. +So on a fixed window, one pass that drops the rows and columns empty within the window +already is the fixpoint, which is unique (ADR 008's question 61 makes the same argument). +The iteration comes from the fill step, which brings in rows and columns from outside the +window (ADR 008 question 66). A client alternates three steps: +- it fetches a window; +- it prunes, in one pass; +- it fills, asking for the next rows past its last absolute row position with a value + in its displayed columns, and likewise for columns. - **The user objects to (b)** (chat, relayed on 2026-10-05; on no PR): "Snap forward is - going to cause issues with the viewport getting stuck". *The author's reading of the - reason:* a client scrolling up asks for a top-left just above its window. If that - position is empty, (b) snaps forward to the first occupied position, which is the - window's old top-left. So the window never moves past an empty stretch above it. (b) - is therefore set aside, and (a) and (c) remain. **Both need the predecessor query**: - (a) to name the nearest occupied position before the request, and (c) to snap backward. - So the author's lean to offer a predecessor query is stronger now, and it is still the - author's. +The model never re-runs a client's fixpoint. A subscribed client receives index (ii)'s +deltas for its S, prunes again locally, and fills again only when a row or column has +gone. Whether the loop always terminates is **not proven**. The fill only moves forward +through absolute positions, so it adds finitely many candidates before the axis ends. A +client that bounds its fills per frame stays within ADR 008's frame budget whether or not +the loop has converged, and that bound is client policy (Q42). - **Neither (a) nor (c) can get stuck, and neither oscillates** (the author's argument): - - **(a) moves the client.** The error names the nearest occupied predecessor and - successor. A client scrolling up retries at the predecessor, which is strictly - before its old top-left, so each scroll step makes progress. A client scrolling - down retries at the successor, which is strictly after. Neither retry can land on an - empty position, because both are occupied by definition. One retry suffices, unless - a write empties that position in between; then the next error names new neighbours, - still in the same direction. - - **(c) moves in the direction asked.** It snaps backward for an upward request and - forward for a downward one, so the actual top-left moves in the direction of the - scroll, or stays put only when there is nothing further that way. - - **At the bounds.** At the top of the axis there is no predecessor. Under (a) the - error names the successor only, and says there is no predecessor. Under (c) an - upward snap with nothing above it returns the first occupied position (the top of - the occupied axis) and says it is the top. Either way the client learns it is at - the edge and stops scrolling, so it does not bounce between "fail" and "retry". - The bottom-right corner is symmetric. An axis with **no** occupied position - answers "empty axis" under both, and the window is empty. +**Growth is only rightward and downward** (C9). *The author's analysis.* The fill only +grows away from the absolute top-left, and it stops at the end of each axis. So **the +fill** only ever asks successor queries: "the next row (or column) after p with a value +in S", bounded by the end of the axis. Scrolling up or left is not a reverse fill: it is +a **new request with a new absolute top-left** (C10), and it is the one place a backward +search appears (next paragraph). What forward-only growth simplifies: +- **The index options.** Each needs only a seek to p and forward iteration, so any + ordered structure with a lower-bound seek and a forward iterator suffices: the sorted + `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no + rank or "row i of n" is needed. A scrollbar over the unpruned cross product gets its + rank by arithmetic. +- **Maintenance.** The range test that follows applies to **occupancy and cell deltas + only**. An item edit is an axis change and goes to every subscriber (above). A change at + position q matters to a fill only if q lies between its top-left and its front. A fill + whose front is before q finds the change when it gets there. +- **Resumable fills.** A fill's whole state is its request and its last absolute position + on each axis. A resume is a seek past that position, so a lost stream, a reconnect or a + sliced job (ADR 008 clause 11) all resume the same way, with nothing else kept. If a + column's list changes during a merge, a re-seek from the last position is enough. An + item insert before the front shifts positions, and the axis-change message above lets + the client re-derive its front. +- **The corner truncates the work.** A fill is bounded by the requested dimensions and by + the bottom-right corner of the axis space. One round costs O(|S| × log n + k × |S|), + never a scan of empty space past the last occupied position. - **The user's simpler procedure** (chat, relayed on 2026-10-05; on no PR), quoted - exactly as sent: "The simplest might be some kind of binary search for a new top left - corner: increasing jumps of the top left until you get a hit and then move down/right - until - You only have one new row column." *The author's reading:* the procedure belongs to the - client and uses forward requests only. The client moves its requested top-left up (or - left) in growing jumps until a request returns an occupied row before its old - top-left, then narrows back down (or right) until exactly one new row (or column) is - gained. A pruned top-left is no longer an edge case at all: the response simply starts - at a later position than the one asked for. *The rest is the author's analysis.* - - **What it requires of the response:** the **actual** absolute position of every - row and column returned, the first one included. The client compares those - positions with its old top-left, so this is T1 again, and it already holds under - clause 12. Nothing else is needed: no error, no snap rule, no direction in the - request, and no predecessor query. - - **Cost.** With doubling jumps, the gallop takes about log₂(gap) requests. A plain - binary search over the last interval takes about log₂(gap) more. At density 10^-4 - the expected gap is about 10^4 rows, so that is 14 + 14 ≈ 28 forward requests, - against one predecessor request. The refinement is often much cheaper. A forward - response holds k occupied rows with their positions, so if the hit request's k rows - reach the old top-left, the client reads the last row before it directly, and the - search ends after the gallop, about 14 requests. Refinement is needed only when - more than k occupied rows lie between the hit and the old top-left. These figures - are arithmetic, not measured. - - **Statelessness and network transparency.** The host keeps nothing for the search. - Every request is an ordinary `project` from an absolute top-left, so the in-process - and remote consumers do exactly the same thing (T6). The cost is round trips: about - 14 for a remote client and about 14 function calls for the TUI. For the TUI that is - nothing. Over a socket it is about 14 latencies for one scroll step into a sparse - stretch. - - **The author's lean, revised.** Take the user's procedure as the **baseline**. It - makes (a), (b) and (c) unnecessary, because a pruned top-left just returns a later - first position. Keep a model-side predecessor query only as an **optional - optimisation** for a remote client. It costs no state (see above) and collapses - the gallop into one request. Offer it once a measured round-trip cost shows it is - worth a second query direction. *As first written*, this item leaned to offering it - outright; that lean is withdrawn. +**A pruned answer as a stream** (C5). Under client-side pruning the fill is the client's, +and each step is one restricted successor query, so the host keeps no fill state at all. +Under model-side pruning, `project` itself would scan forward from the top-left and send +rows and columns as it finds them, as a sliced job resumable from `(request, fill +front)`. Either way, changes during a fill are identity-keyed deltas stamped by step +(T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an +ordinary delta with its absolute position, and the client keeps or drops the extra row by +its own policy (Q42). So a fill and a live update are one stream, which keeps the +in-process and the remote consumer the same (T6). Question 21. - Cost at density 10^-4: finding the nearest occupied neighbour is one seek in the - ordered index, O(log n), in either direction, so (a) with named neighbours and (b) - cost the same on the host. (a) adds a round trip, and (b) adds none. The expected gap - to the next occupied row is on the order of 10^4 cross-product rows, so a client that - guesses blind under (a) would need about log₂(10^4) ≈ 14 doubling retries. **Unpruned - space is not an edge case at all**: a top-left on an empty position is answered with - empty cells. - - **How it fits identity-keyed deltas and resumable streams.** Under (b) and under (a) - with named neighbours, the response's actual top-left is an absolute position, and - the tuple there is an identity. So the stream's resumable state stays `(request, - actual top-left, last position)`, and a later delta names rows by identity whichever - top-left was asked for. A top-left whose row is pruned after the stream started is the - same case in time. A delta reports the row's removal, the client's window keeps its - absolute top-left, and the next fill skips past the empty position. That is (b)'s - behaviour without a new request. +**Scrolling up or left, and a top-left in empty space** (C10 to C12). +- **The baseline is the user's galloping search** (C12; "the simplest might be"). *The + author's reading:* the client moves its top-left up (or left) in growing jumps until a + request hits, then narrows back until exactly one new row (or column) is gained. **Each + probe is a restricted successor query** from the candidate top-left t, with the client's + current S, bounded by its old top-left: "is there a row in [t, old top-left) with a + value in S, and which?" It is not a bare cross-product `project`, which would return + mostly empty rows. +- **What it requires:** the response reports the **actual** absolute position of every + row and column returned, which T1 already requires. Under client-side pruning a + top-left that lands on an empty row is no edge case for the host. The host returns the + cross-product window or the successor answer, and the client's own prune decides what + shows. +- **Cost** (arithmetic, not measured). Doubling jumps take about log₂(gap) probes, and a + binary narrowing about log₂(gap) more. At density 10^-4 the gap is about 10^4 rows, + so 14 + 14 ≈ 28 probes. It is often about 14, because a probe returns up to k rows with + their positions. If the hit's rows reach the old top-left, the client reads the answer + directly. Under concurrent writes neither phase can run forever: the gallop ends at the + top of the axis, and the narrowing is a bounded bisection. +- **Statelessness and transparency.** The host keeps nothing for the search, and the + in-process and remote consumers make the same calls (T6). The cost is round trips: about + 14 over a socket for one scroll step into a sparse stretch, and negligible in the TUI. +- **Mixed cases** (the author's). A diagonal scroll moves both axes. The client gallops + one axis at a time. It gallops rows against its current displayed columns, then + columns against the rows it now displays, then prunes once. When only the row axis's + new top-left is empty, only the row gallop does work. When both are empty, both do, in + turn. Whether rows-then-columns and columns-then-rows always give the same window is + **not proven**. ADR 008's question 61(b) notes that order matters for a one-sided + pass, and the order is client policy. +- **A predecessor query is an optional optimisation.** The model could answer "the + previous row before p with a value in S" in one call. It costs no new state under any + of the three options: a sorted `Vec` and a B-tree both seek an upper bound and iterate + in reverse at the same cost, option (ii)'s lists merge backwards as they merge + forwards, and options (i) and (iii) walk backwards with the same test. *The author's + lean:* offer it only once a measured round-trip cost shows that one call is worth a + second query direction. +- *Superseded:* earlier text in this clause leaned to offering the predecessor query + outright, and then called that lean "stronger" once options (a) and (c) below needed + it. The galloping baseline replaces both leans. Options (a) to (c) are kept below as + a record. -- **The fixpoint comes from the fill, not from one window.** *As first written*, this - item said that hiding a row can empty a column. **That was wrong.** A row dropped for - having no value among the window's columns contributes no value to any of those - columns. So on a fixed window, one pass that drops the rows and columns empty within - the window already is the fixpoint (ADR 008 revision 21, question 61, makes the same - argument). The iteration comes from the fill step, which brings in rows and columns from - outside the window. ADR 008's question 66 records that loop. -- **The loop the user describes, and what makes it converge cheaply.** The user, in chat: - "The problem pruning solves is visualizing sparse datasets: screens that are mostly - empty are useless and so you drop out categories from rows and columns until every - visible row or column has at least one value on screen." (relayed on 2026-10-05; on no - PR). The criterion is a fixpoint on the displayed grid. *The author's analysis:* a client - alternates three steps. It fetches a window. It prunes, removing every row with no - value in the visible columns and every column with no value in the visible rows, until - nothing changes. It fills, asking for the next rows (or columns) past its last absolute - position that have a value in the columns (or rows) still visible. Three things the - circuit must answer for that loop: - - **"Any value in row r among columns S", and its batched form**, the next k such rows - from p. With index (ii) the batched form costs O(|S| × log n + k × |S|) per call, keeps - no state per request, and is recomputed for each request. That is the "as stateless - as possible" side of it. Without an index, each candidate row costs |S| store lookups, - and a sparse dataset puts most candidates in empty space. That is exactly the case - pruning exists for, so the index is not optional at 3 × 1,000. - - **Maintenance is incremental and cheap per write.** A cell that starts or stops - holding a value inserts or removes one entry in one column's list, at O(log n). Index - (ii) is a set of `(column, row)` pairs, so it is itself a Z-set fed from the cell delta - and from formula stems. Its delta is "row r gained or lost a value in column c". - - **The answers are friendly to deltas.** A client subscribed to a window can be sent - just those `(row, column)` changes that fall in its window's columns (clause 12's - subscription, restricted to S). It re-runs its own prune step locally, and it fills - again only when a row or column has been removed. So the host never re-runs a - client's fixpoint. +**Superseded: the pruned top-left under model-side pruning.** These options assume the +host judges "pruned", which holds only under model-side pruning. They are kept so that +the user's objection is recorded with what it answered. +- (a) **Fail, then retry** (C10's tentative "maybe"). The error names the nearest + occupied successor and predecessor. +- (b) **Snap forward and report the actual top-left.** **The user objects** (C11). *The + author's reading of the reason:* a client scrolling up asks for a top-left just above + its window. If that row is empty, (b) snaps forward to the window's old top-left, so the + window never moves past an empty stretch. +- (c) **Snap in the scroll direction.** It needs the request to carry its direction. - Whether the loop always terminates is **not proven**. Within a fixed candidate set, - iterated removal reaches the greatest fixpoint, which is unique (the union of any two - sets with the property has it too, as ADR 008 revision 21's question 61 also says). On - a fixed window that takes one pass. The fill only moves forward through absolute positions, so it - adds finitely many candidates before the axis ends. A client that bounds its fills per - frame stays within ADR 008's frame budget whether or not the loop has converged. That - bound is a client policy (Q42). -- **What this changes in this ADR.** If pruning is a client filter over the window, - nothing needs the whole-extent, per-axis occupancy of clause 9 *for pruning's sake*. The - pruned axis, its ordered index and "row i of n" are not needed either, because a - window-relative filter has no global n. What remains model-side is **which cells hold a - value**, including formula stems, which only the model can evaluate. ADR 008's question - 1 ("go with occupied") is about that, and it still applies, to the cells of a window. Its - question 11 ("prune empty should be the default") becomes a default for a client's - display. Whether clause 9's occupancy should be restated as the index of (ii), and - whether ADR 008 clause 3's `Occupied` axis survives, is question 22. - -Whether a pruned response is streamed or returned whole, and what a client sees while a -fill is incomplete, is question 21. +Neither (a) nor (c) can get stuck. A retry under (a) goes to a named neighbour that is +strictly beyond the old top-left in the scroll direction, and occupied. A write that +empties it only delays the search, by at most the gap. (c) moves in the direction asked, +and stays put only at the end of the axis. At the top of the axis, (a) names only the +successor and reports that there is no predecessor, and (c) returns the first occupied +position and reports that it is the top. So neither bounces. An axis with nothing +occupied answers "empty axis". Both need the predecessor query. **What network transparency requires** (T1 to T6, the author's): - **T1. Results are plain data.** Every element on an output stream is a value the @@ -1592,6 +1535,11 @@ fill is incomplete, is question 21. basis that describes them. This does not break gotcha 42's "never coalesce". That rule is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple identity, or by a stable cross-product position, is not positional in that sense. + **Coalescing is therefore per subscription, never per stream.** All of a + subscription's pending deltas, the basis included, are summed together up to one step, + and sent as one batch in the fixed order. Coalescing one stream while another is sent as + it arrives would let a basis for step n+2 overtake the cells it describes, which is + gotcha 43's failure. - **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives the same result and delta types as a remote client. Its transport is a function call. That is what makes the requirement testable: run one edit sequence through two @@ -1609,7 +1557,7 @@ consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that bo The alternative is a `serde` feature on `zset`. The author leans to the conversion, because what crosses is the projected result, not the crate's own element type. -**How stateless the host can be.** The user: "as stateless as possible". These are the +**How stateless the host can be.** The user: "as stateless as possible" (C2). These are the options of question 13, and ADR 008's question 52 asks the push half of the same thing. - **(a) Pure fetch.** The host keeps nothing per client. A client calls `project(request)`. It learns its result may be stale from a change notice addressed to @@ -1623,7 +1571,7 @@ options of question 13, and ADR 008's question 52 asks the push half of the same eviction, a reconnect), a client resumes by resending its request and the last step it applied, and the host answers with a snapshot. The host could instead keep a short ring of past deltas per shape; the author's lean is no ring at first. So the user's "recent - projections + clients" are this table plus clause 7's shape cache, and **both can be + projections + clients" (C2) are this table plus clause 7's shape cache, and **both can be rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). - **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, @@ -1735,7 +1683,8 @@ orders come from the clauses and are not measured. (ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration -step 4 must. +step 4 must. It reports both as ratios against an unrelated operation on the same run, +as ADR 008's question 13 requires, not as bare milliseconds. ## Alternatives considered @@ -1792,7 +1741,7 @@ displayed path until the differential test holds. O(cells) scans per cell write. The existing tier tests are the evidence. 4. **`Incremental`, for the order-free parts**, under the differential test: stems with clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's per-effect operator shape. The formula delta arrives + delta, all following clause 7's shape key and read-time parameters. The formula delta arrives only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the frame observation) depend on this step.** @@ -1849,9 +1798,9 @@ longer a prerequisite of step 6. - Circularity is decided in two layers: the formula graph's components and the per-cell bound. - A mutating frame from a client already clones the workbook for rollback - (`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells) + (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- The operator shape is re-read after every model effect: O(items) per live shape, a cost the +- The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. - Per-shape state lives only as long as requests name it. A shape that comes back after @@ -1919,7 +1868,7 @@ revision, and none is renumbered. *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. *[Open. ADR 008's flagged reading 1 makes the page selection a parameter of the request, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so +7. *[Open. ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the @@ -1947,20 +1896,22 @@ revision, and none is renumbered. *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are how it is met. -11. *[Open, and affected by question 16. Under item grain a reorder of an axis category's items rebuilds nothing, so the recommendation's premise (key by position and rely on `view_shape`'s rebuild) no longer holds. Keying by item identity and reading the order at read time, which clause 12 already needs (question 20), becomes the natural form. The recommendation is left as written until the user answers both.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8). - Should `LAST`/`FIRST` key their per-group state by the ordering category's item - position, with that category's item order added to the operator shape so that a - reorder rebuilds them? Or should they key by stable item identity (ADR 007), and - look the order up at read time? - *Recommendation:* item position. Reads stay O(log d), like `Min`/`Max`, and the - same generic operator serves all four. The order of **drawn** items is already in - `view_shape` (`app.rs:47-52`), so a reorder of drawn items already rebuilds, and - nothing new is needed for them. The one open part is **hidden** items, whose - positions `view_shape` does not hash. Recommended: `LAST`/`FIRST` order only the - values the fold covers, which are the drawn members. Then the operator shape needs - no widening at all, and ADR 008's text and cross-references hold unchanged. If the - user wants hidden items ordered too, the `measure_agg` part must also cover their - positions. +11. *[Open. The recommendation was revised on 2026-10-05 by the author, because clause 7 + now makes the drawn order a read-time parameter; the user has answered neither + version.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and + improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering + category's item position, and rebuild on a reorder? Or should they key by stable item + identity (ADR 007), and look the order up at read time? + *Recommendation (revised):* neither as first stated. Key each entry by an **order + label**: a key that compares items in their current order and is not renumbered when + an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's + question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator + serves all four. An insert or a reorder relabels only the items that moved, and + rebuilds nothing. This is not measured. `LAST`/`FIRST` order only the values the fold + covers, which are the drawn members. + *As first written* (superseded): key by item position, relying on `view_shape`'s + rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn + order, so that rebuild does not happen, and a position key would go stale. 12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). Should the host maintain each column's whole-column maximum display width? That is @@ -1972,19 +1923,21 @@ revision, and none is renumbered. not cure. 13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per - client** (clause 12; ADR 008's question 52 asks the push half). *Recommendation (the - author's):* (b), soft-state subscriptions that can be rebuilt from the request and the - last step, with no copy of a client's window on the host. When its table is lost, (b) - falls back to (a). (c) is obsolete under ADR 008's split. One of the two ADRs should - hold this question, and the other should point to it. + client** (clause 12). **Owner: this ADR.** ADR 008's question 52 asks the push half, + and should point here. *Recommendation (the author's):* (b), soft-state + subscriptions that can be rebuilt from the request and the last step, with no copy of + a client's window on the host. When its table is lost, (b) falls back to (a). (c) is + obsolete under ADR 008's split. -14. *[New.]* **What keys operator state, and how long a shape lives** (clause 7). Is it - keyed per projection shape, cached and evicted (the author's)? Or per saved `.improv` - view only, so that a request whose shape is unsaved (an unsaved prune or hide, ADR - 008's question 51) is served without incremental state? *Recommendation:* per shape, - cached, and evicted once no request has named it for a while. The time is an - implementation number. An unsaved shape is then as incremental as a saved one, which - keeps "pruning shouldn't infect every client" from costing an eager path. +14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state + selected by a request's **shape key** (axes by level, None categories, filter keys, + page categories, `measure_agg`), cached and evicted, with every other parameter + applied at read time (the author's)? Or is it kept per saved `.improv` view only, so + that a request with an unsaved key is served without incremental state? + *Recommendation:* per shape key, cached, and evicted once no request has named it for + a while. The time is an implementation number. An unsaved hide, collapse, reorder, + page step or prune then shares the saved view's key and costs nothing. Only an + unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2003,93 +1956,84 @@ revision, and none is renumbered. *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the order the user's answer to ADR 008's question 17 already sets. -18. *[New.]* **Formulas the per-record map cannot express** (clause 5a). ADR 008's - question 15 sends them to a fallback over the occupied contexts, never to - `Expansion::members`; the user's answer was "the recommendation is fine.". Two parts: - - Does this ADR's `Eager` adopt the same rule, so that `Eager` and `Incremental` - agree and this ADR's differential test needs no carve-out? - - Do the formulas that target a category other than `_Measure` take the same +18. *[New; premise corrected on 2026-10-05.]* **Formulas the per-record map cannot + express** (clause 5a). ADR 008's question 15 sends them to a fallback over the + occupied contexts, never to `Expansion::members`; the user's answer was "the + recommendation is fine.". *As first written*, this question said the fallback changes + `Eager`'s answers. **The code says it does not.** At `50e1314d`, `agg_values` evaluates + the expression only over `member_records` of each context (`types.rs:91-139`; `member_records` at `:47-53`). So a + context that holds no record contributes no value, and no error, whatever the + expression is. The fallback changes the cost, not the answer. And today's eager path + reaches `Expansion::members` only at a fold or a total (`contexts_of`, + `types.rs:916-925`). At a stem it pools one context. Two parts remain: + - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be + a cost change with bit-equal answers, since dropping empty contexts keeps the order + of the rest, and the differential test then needs no carve-out. + - Should the formulas that target a category other than `_Measure` take the same fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - *Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's - question 1 and question 15 cases, as a stated change of behaviour. + *Recommendation:* yes to both, with no change of behaviour. This was argued from the + code and not run. The differential test pins it. -19. *[New. The user was "still ambivalent", then said in chat that pruning "sounds like a UI side concern"; see question 22. Not confirmed as a decision.]* **Which side of the wire prunes** (clause 12; - the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same, - and one ADR should hold it). On the model side, `project` - returns occupied rows and columns, and the circuit answers next-occupied and - rank/select queries for it. On the client side, `project` returns the cross-product - window together with a skip structure, and the client asks for the queries itself. - *Recommendation (the author's), revised after the user's later chat:* the client - side, as the user leans. The model offers windows and question 22's restricted skip - query, so a sparse grid still never ships empty space. *As first written*, this - recommended the model side; that is withdrawn. +19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side + concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** + (clause 12). **Owner: ADR 008** (its question 57), because it is a question about the + view/host split. This question records what each answer asks of the circuit. On the + model side, the circuit keeps a pruned axis per shape key, with rank and select. On + the client side, it answers question 22's restricted successor query, and the client + prunes. *Recommendation (the author's), revised after C6 to C8:* the client side. The + model offers cross-product windows and the restricted successor query, so a sparse + grid still never ships empty space. *As first written*, this recommended the model + side; that is withdrawn. -20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left - should be as absolute as possible"; ADR 008's question 58 asks the same from the - request side). A cross-product position stays fixed when - occupancy changes, but it moves when an item is inserted, removed or reordered. Should - deltas be keyed by tuple identity, with absolute positions attached when they are - sent, and should an item edit be sent as an axis change, so that clients re-derive - their positions arithmetically? *Recommendation:* yes. It costs rank and select over - each category's items (ADR 008 clause 10), and comparisons in the occupancy index at - O(levels × log items) each, which is not measured. It is compatible with ADR 008's - question 58 recommendation, which makes a tuple of item ids the primary form of the - top-left. Whether hidden items belong to the absolute space is ADR 008's question 58. - Under item grain (clause 7) this ADR serves either answer, because hiding is a remap. +20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR for the + delta keys; ADR 008 (its question 58) for the form of the request's top-left.** A + cross-product position stays fixed when occupancy changes, but it moves when an item + is inserted, removed or reordered. Should deltas be keyed by tuple identity, with + absolute positions attached when they are sent? And should an item edit be sent to + every subscriber of the key as an axis change, so that clients re-derive their + positions arithmetically? *Recommendation:* yes. It costs rank and select over each + category's items (ADR 008 clause 10), at O(levels × log items) per lookup, which is + not measured. It is compatible with ADR 008's question 58 recommendation, a tuple of + item ids as the primary form of the top-left. Item grain (clause 7) serves either + answer to whether hidden items belong to the absolute space, because hiding is a + remap. -21. *[New.]* **A pruned answer as a stream** (clause 12; the user: "After pruning we - might have to stream rows and columns to attempt to fill out the full grid"). Should - `project` under pruning fill forward from the absolute top-left, as a sliced job that - can be resumed from its fill front, and send each found row and column as a stamped - delta? Should a row that becomes occupied behind the front arrive as an ordinary - occupancy delta, and leave the window one row larger than asked, for the client's - policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one - stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59 - asks the same, and asks in its part (c) whether rows or columns fill first. Under the - user's later chat (question 22), the fill loop is the **client's**: it calls the - restricted skip query once per step of its row and column fixpoint. The host then - keeps no fill state at all, which is more stateless than the sliced host job this - question first described. +21. *[New.]* **How a pruned answer streams** (clause 12; C5). **Owner: ADR 008** (its + question 59) for the semantics: when a fill stops, and in what order rows and columns + fill. **This ADR** owns the delivery. *Recommendation:* under client-side pruning the + fill is the client's, one restricted successor query per step, and the host keeps no + fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, + so a fill and live updates are one stream. Under model-side pruning the same holds, + with the fill a sliced host job resumable from `(request, fill front)`. 22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause - 12; the user, in chat: "Pruning is just a filter applied to full columns a rows ... - That sounds like a UI side concern" and "a value being present outside the viewport - doesn't matter"). This changes today's semantics, which prune against the whole - grid. Three parts: - - (a) Should the circuit offer a **restricted skip query**, "the next k rows from - absolute position p with a value in at least one of these columns", and the same for - columns? How should it answer: (i) per-axis occupancy as a pre-filter, (ii) a column - index of occupied rows, merged, or (iii) a row index of occupied columns? - - (b) Should clause 9's whole-extent, per-axis occupancy be **restated** as that - index, and ADR 008 clause 3's `Occupied` axis and its "row i of n" be dropped, - since a window-relative filter has no global count? - - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty - should be the default") need restating? Question 1 would become the rule for which - cells of a window hold a value, and question 11 a client's display default. + 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. + Four parts, with owners: + - (a) **Owner: this ADR** (ADR 008's question 62 should point here). Should the + circuit offer a **restricted successor query**, "the next k rows after absolute + position p with a value in at least one column of S", with S in the call, and the + same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, + (ii) a column index of occupied rows, merged, or (iii) a row index of occupied + columns? Under C9 it only ever looks forward. + - (b) **Owner: this ADR for clause 9; ADR 008 (its question 65) for its `Occupied` + axis.** Should clause 9's whole-extent, per-axis occupancy be restated as index + (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it + only to skip. That skips only rows empty everywhere, not rows empty in S, so on its + own it does not answer (a). + - (c) **Owner: ADR 008.** Do its question 1 ("go with occupied") and question 11 + ("prune empty should be the default") need restating? Question 1 would become the + rule for which cells of a window hold a value, and question 11 a client's display + default. + - (d) **Owner: ADR 008 for the semantics; this ADR records the costs.** What happens + when scrolling up or left, and when a requested top-left lands on an empty row + (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It + probes with the restricted successor query, and it needs responses to report actual + positions. It costs about 14 to 28 probes at density 10^-4, by arithmetic. The + earlier fail, snap-forward and direction-snap options are kept there as superseded, + with the user's objection to snapping forward (C11). - ADR 008 revision 21 asks the same from its side: its question 62 (the skip query), - question 65 (whether "occupied" stays whole-extent), question 61 (one pass or a - fixpoint) and question 66 (the fill loop). Its question 65 recommends keeping per-axis, whole-extent occupancy and - using it only to skip. Under (i) above, that skips only rows that are empty - everywhere, not rows that are empty in the displayed columns. So it does not answer - the restricted query by itself. - The user's later statement makes the criterion a fixpoint on the displayed grid, - reached by a client loop of fetch, prune and fill (clause 12). The query in (a) is - what each fill step calls. - Under the user's right-and-down growth (clause 12), the query is a forward successor - query only, and so are all three options; no rank or reverse scan is needed for - pruning. - The user's later chat adds a client-side optimisation for scrolling up or left (a - predecessor search, which every option serves as cheaply as a successor search). It - also raises the edge case of a top-left that lands on a pruned row: fail and retry, - snap forward (which the user objects to: "the viewport getting stuck"), or snap toward - the scroll direction (clause 12). The user has since suggested a simpler client procedure: - gallop the top-left with forward requests until one hits, then narrow. A pruned - top-left then needs no special answer, provided responses report actual positions. - The author now takes that as the baseline, and keeps a predecessor query only as an - optional optimisation. That edge case is - part (d) of this question. *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms - that pruning is the client's. (c) restate both in those terms, in ADR 008, since they - are ADR 008's answers. + that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the + gallop as the baseline, and a predecessor query only as an optional optimisation, + offered once a measured round-trip cost justifies it. From 255e3d6d96ada6e95a451c23fb5933db00f7abcd Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:32:39 -0700 Subject: [PATCH 26/67] docs(adr): ADR 009 reads ADR 008 revision 25: the ref edge in clause 5a and the proposed question owners (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 80 +++++++++++++---------- 1 file changed, 47 insertions(+), 33 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 3fc94606..6720b9a2 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 24** -(PR #308, read at `3180327b`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 25** +(PR #308, read at `5c208f54`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `3180327b`, its revision 24) · +reads it at `5c208f54`, its revision 25) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 24 of `context/adr/008-virtualized-grid.md` on PR #308, at `3180327b`. Clauses 3, +revision 25 of `context/adr/008-virtualized-grid.md` on PR #308, at `5c208f54`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 24, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 25, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -726,13 +726,17 @@ product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's **store-driven fallback**, which evaluates the expression only over the **occupied contexts**, the contexts some record projects to. The user accepted that recommendation, as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as -eager's**, as the paragraph above says of the pipeline. `agg_values` evaluates the -expression only at the `member_records` of each context (`types.rs:47-53`, `:91-139`), -so a context with no record contributes no value and no error, whatever the expression. -The fallback changes the cost and not the answer. *The first revision of 2026-10-05 said -it "differs from today's eager answer" wherever the expression is non-empty at an empty -context. That was wrong, and ADR 008's question 15 carries the same premise.* Whether -this ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: +eager's, with one edge.** `agg_values` evaluates the expression only at the +`member_records` of each context (`types.rs:47-53`, `:91-139`), so a context with no +record contributes no value, as the paragraph above says of the pipeline. **The edge:** +an inner bare `Expr::Ref` that names no item errors `ref:` at the top of +`agg_values`, for every context it is called with, records or not. Evaluated over +**zero** occupied contexts the call never happens, and the aggregate yields its empty +value instead of the error. ADR 008 revision 25 found this edge, and it is question 15's +whole difference. With at least one occupied context the error is the same. *The first +revision of 2026-10-05 said the fallback differs "wherever the expression is non-empty at +an empty context", and a later one said it never differs. Both were wrong.* Whether this +ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: - one changes **one operand** of a two-operand expression (`Price` alone, under `SUM(Revenue * Price)`); - one **deletes a record's leading cell**, including a cell the expression never reads @@ -1147,7 +1151,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 24 at `3180327b`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 25 at `5c208f54`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1408,8 +1412,10 @@ search appears (next paragraph). What forward-only growth simplifies: - **The index options.** Each needs only a seek to p and forward iteration, so any ordered structure with a lower-bound seek and a forward iterator suffices: the sorted `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no - rank or "row i of n" is needed. A scrollbar over the unpruned cross product gets its - rank by arithmetic. + rank over **occupied tuples**, and no "row i of n", is needed. Rank and select over + **each category's items** stay, because a cross-product position is built from them + (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its + rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas only**. An item edit is an axis change and goes to every subscriber (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill @@ -1960,20 +1966,25 @@ revision, and none is renumbered. express** (clause 5a). ADR 008's question 15 sends them to a fallback over the occupied contexts, never to `Expansion::members`; the user's answer was "the recommendation is fine.". *As first written*, this question said the fallback changes - `Eager`'s answers. **The code says it does not.** At `50e1314d`, `agg_values` evaluates - the expression only over `member_records` of each context (`types.rs:91-139`; `member_records` at `:47-53`). So a - context that holds no record contributes no value, and no error, whatever the - expression is. The fallback changes the cost, not the answer. And today's eager path + `Eager`'s answers wherever the expression is non-empty at an empty context. **The code + says it does not, except at one edge.** At `50e1314d`, `agg_values` evaluates the + expression only over the `member_records` of each context (`types.rs:91-139`; + `member_records` at `:47-53`), so a context that holds no record contributes no + value. The edge is an inner bare ref naming no item. It errors for every context it + is called with, so over zero occupied contexts it yields the empty value instead of + `ref:` (clause 5a; ADR 008 revision 25). And today's eager path reaches `Expansion::members` only at a fold or a total (`contexts_of`, `types.rs:916-925`). At a stem it pools one context. Two parts remain: - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be - a cost change with bit-equal answers, since dropping empty contexts keeps the order - of the rest, and the differential test then needs no carve-out. + a cost change with bit-equal answers, except at the ref edge, since dropping empty + contexts keeps the order of the rest. The differential test then carves out only + that edge, by name. - Should the formulas that target a category other than `_Measure` take the same fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - *Recommendation:* yes to both, with no change of behaviour. This was argued from the - code and not run. The differential test pins it. + *Recommendation:* yes to both. The only change of behaviour is the ref edge, which + could also be kept by checking the ref once before enumerating. This was argued from + the code and not run. The differential test pins it. 19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** @@ -1986,8 +1997,9 @@ revision, and none is renumbered. grid still never ships empty space. *As first written*, this recommended the model side; that is withdrawn. -20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR for the - delta keys; ADR 008 (its question 58) for the form of the request's top-left.** A +20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR**, for the + delta keys and the form of the top-left both, as ADR 008 revision 25 proposes. Its + question 58 points here. A cross-product position stays fixed when occupancy changes, but it moves when an item is inserted, removed or reordered. Should deltas be keyed by tuple identity, with absolute positions attached when they are sent? And should an item edit be sent to @@ -1999,9 +2011,9 @@ revision, and none is renumbered. answer to whether hidden items belong to the absolute space, because hiding is a remap. -21. *[New.]* **How a pruned answer streams** (clause 12; C5). **Owner: ADR 008** (its - question 59) for the semantics: when a fill stops, and in what order rows and columns - fill. **This ADR** owns the delivery. *Recommendation:* under client-side pruning the +21. *[New.]* **How a pruned answer streams, and the fill step** (clause 12; C5). **Owner: + this ADR**, as ADR 008 revision 25 proposes. Its questions 59 and 66 point here. That + includes when a fill stops, and in what order rows and columns fill. *Recommendation:* under client-side pruning the fill is the client's, one restricted successor query per step, and the host keeps no fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, so a fill and live updates are one stream. Under model-side pruning the same holds, @@ -2010,22 +2022,24 @@ revision, and none is renumbered. 22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. Four parts, with owners: - - (a) **Owner: this ADR** (ADR 008's question 62 should point here). Should the + **Owner: this ADR** for the whole question, as ADR 008 revision 25 proposes (its + questions 61, 62 and 65 point here). Part (c) is the exception: it restates answers + ADR 008 records, so its text can only change there. + - (a) Should the circuit offer a **restricted successor query**, "the next k rows after absolute position p with a value in at least one column of S", with S in the call, and the same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, (ii) a column index of occupied rows, merged, or (iii) a row index of occupied columns? Under C9 it only ever looks forward. - - (b) **Owner: this ADR for clause 9; ADR 008 (its question 65) for its `Occupied` - axis.** Should clause 9's whole-extent, per-axis occupancy be restated as index + - (b) Should clause 9's whole-extent, per-axis occupancy be restated as index (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it only to skip. That skips only rows empty everywhere, not rows empty in S, so on its own it does not answer (a). - - (c) **Owner: ADR 008.** Do its question 1 ("go with occupied") and question 11 + - (c) Do ADR 008's question 1 ("go with occupied") and question 11 ("prune empty should be the default") need restating? Question 1 would become the rule for which cells of a window hold a value, and question 11 a client's display default. - - (d) **Owner: ADR 008 for the semantics; this ADR records the costs.** What happens + - (d) What happens when scrolling up or left, and when a requested top-left lands on an empty row (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It probes with the restricted successor query, and it needs responses to report actual From 15cb987a938149260a980916442baa1d7217b32b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:39:32 -0700 Subject: [PATCH 27/67] docs(adr): ADR 009 rebuilds column totals on a row hide, and fixes the review's nits (improvise-felo.4) Today a column total pools only the drawn rows, so a hide changes it; clause 7 now rebuilds totals on a row-axis hide or show by default (B), with read-time subtraction (A) as question 23. Hidden page-category items are filter keys; step 8, the stem path, the order label's cost, the ADR 008 chat-list note and two citations are corrected. Reads ADR 008 at 99322d0e. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 119 ++++++++++++++++------ 1 file changed, 88 insertions(+), 31 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 6720b9a2..e034ecae 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,11 +1,11 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 25** -(PR #308, read at `5c208f54`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 26** +(PR #308, read at `99322d0e`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 22. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `5c208f54`, its revision 25) · +reads it at `99322d0e`, its revision 26) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 25 of `context/adr/008-virtualized-grid.md` on PR #308, at `5c208f54`. Clauses 3, +revision 26 of `context/adr/008-virtualized-grid.md` on PR #308, at `99322d0e`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 25, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 26, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -86,7 +86,9 @@ and says where each statement came from. - **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the - PR, but spread over several groups of its own. This list is the one place this ADR + PR, but spread over several groups of its own. ADR 008's list also has six more (its + items 1 to 6: the five standing design statements, and the "q19 … q24" corrections). + This ADR relies on none of those six. This list is the one place this ADR quotes them, and the clauses cite them by number. None of this answers this ADR's own questions 1 to 11. @@ -248,7 +250,7 @@ a change proportional to what the change touches, not to the document. - **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes every cell before and after a cell-writing batch. `document_content` (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is - re-read after **each** `CellData` effect (`app.rs:1039-1045`). So the first is O(cells) + re-read after **each** `CellData` effect (`app.rs:1119-1125` at `50e1314d`). So the first is O(cells) per batch and the second O(cells) per effect, whatever was written. - **Every mutating frame from a client already clones the document.** `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, @@ -439,7 +441,7 @@ because it observes `view_shape` and not cells. **The dirty flag's content half does not move onto the cell delta alone.** `document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it -is re-read after every `CellData` effect (`app.rs:1039-1045`). `AddFormula`, +is re-read after every `CellData` effect (`app.rs:1119-1125` at `50e1314d`). `AddFormula`, `RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, `:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a non-empty cell delta means dirty" would never save a formula edit in the browser. The @@ -544,9 +546,15 @@ key, and `FIRST` the least. superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a reorder rebuilds nothing, and a state keyed by position would go stale. The position of a **hidden** item matters only if `LAST`/`FIRST` must order values that come from - hidden items. The alternative is to key by a stable item - identity (ADR 007) and look the order up at read time. That makes a reorder free, but - a read then costs O(d). This is question 11. + hidden items. There are three keys: + - **position**, which goes stale on a reorder now that nothing rebuilds; + - **stable item identity** (ADR 007), with the order looked up at read time, which + makes a reorder free, but then a read costs O(d); + - an **order label**, which compares in the current order and is not renumbered by an + insert elsewhere. Reads stay O(log d). A reorder relabels the moved items, and + re-keys every group entry stored under them, at O(entries under the moved items). + + This is question 11. - **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a delta (retract the old value, insert the new), and the outer consumes it. So the @@ -755,7 +763,8 @@ occupied-contexts fallback above as well is question 18. **What stays eager until step 5, and what is bit-equal before step 6.** - **Before step 5** (formulas), every aggregate over an expression is evaluated as - today, through `Expansion::members`. + today: through `Expansion::members` at a fold or a total, and over the one context at a + stem. - **From step 5**, the record stage, the map and the index are maintained, so the cross product is gone. `Count`, `Min` and `Max` over the mapped values, and the error channel, are read from the maintained state. They are order-free, or ordered by the @@ -812,14 +821,16 @@ states the same rule. - the categories of each axis, by level; - the None categories; - the filter keys: each `Axis::Filter` category with its selected item, and the hidden - items of every category that is **not** on an axis (such an item drops cells from the - fold, so it filters); + items of every category on None, Page or Filter, as ADR 008 clause 3 also says. Such + an item drops cells from the fold, so it filters. A hidden item of a **page** + category is a filter key, not a read-time parameter; - the page **categories**, which key the slices under question 7's recommendation; - `measure_agg`. Everything else a request carries is a **read-time parameter**, applied when the - output is read, without touching operator state: - - the hidden items of an axis category; + output is read, without touching occupancy or cell operators: + - the hidden items of an axis category, which also rebuild the column totals when they + are on the row axis (see "Hiding a row moves the column totals" below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -850,15 +861,16 @@ states the same rule. - **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column aggregates are keyed by the **items** of the axis categories, together with the filter and page keys. They are never keyed by a position in a level sequence. So a collapse, - expand, hide or show of an **axis** category's items changes no operator. The visible - axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms + expand, hide or show of an **axis** category's items changes no occupancy or cell + operator, and the visible axis is a remap of the item-grain state. **The exception is + the column totals:** a hide or show of a row-axis item rebuilds them (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other operators, which a new key builds as a job: - a change of which categories are its levels or on None; - - a change of its filter keys, which includes hiding an item of a category that is not - on an axis; + - a change of its filter keys, which includes hiding an item of a category on None, + Page or Filter; - a change of its page categories; - a change of `measure_agg`. @@ -868,6 +880,30 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. +- **Hiding a row moves the column totals (the author's default; question 23).** Today a + column total pools only the **drawn** rows. The total's evaluator excludes hidden items + through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` + evaluates over that memo (`:1066-1070`). The test + `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take + `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide + Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, + and no read-time parameter can take Feb back out of it. So: + - **(B), the default:** a hide or show of a **row-axis** item rebuilds the column + totals of that shape, as a job. ADR 008 says the same ("totals still rebuild on hide + and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and + expand do not move a total, because it pools the members either way, which is what + the test above pins. A hide or show of a column-axis item only hides or shows that + column's total. + - **(A), the open alternative:** build each total from clause 4's (row tuple, column + tuple) group state, and drop the hidden rows when the total is read. For `Sum` and + `Count` that is O(hidden rows) per read: subtract the hidden rows' state. That is + exact only under question 3's (c), or under ADR 005's wide accumulator, because + f64 subtraction is not exact. For `Min`, `Max` and the one-distinct-value text rule + it is O(rows) per read, a recombination of every visible row's state. A hide then + costs nothing to install, and reads cost more. + + The author takes (B), because it matches the code and ADR 008, and it is the smaller + change. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1151,7 +1187,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 25 at `5c208f54`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 26 at `99322d0e`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1194,8 +1230,10 @@ host-held mirror of a client. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate at item grain, and the formula layer re-evaluates only the columns whose - inputs moved, through clause 2's trait. Clause 5 replaces `Expansion::members`. Totals + aggregate at item grain over the drawn rows, and the formula layer re-evaluates only + the columns whose inputs moved, through clause 2's trait. A hide or show of a row-axis + item rebuilds the totals (clause 7, "Hiding a row moves the column totals"; question + 23). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1761,11 +1799,12 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** The totals rebuild on a hide or + show of a row-axis item (clause 7), unless the user takes question 23's (A). 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, - snapshot-then-delta, the order-statistic queries over occupancy, and the test with two + snapshot-then-delta, question 22's restricted successor query, and the test with two consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -1806,6 +1845,8 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). +- A hide or show of a row-axis item rebuilds the column totals of that shape, as a job, + under question 23's default (clause 7). Occupancy and cells do not rebuild. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -1820,7 +1861,7 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 22 were added by the 2026-10-05 +none of questions 1 to 11 on #309.** Questions 12 to 23 were added by the 2026-10-05 revision, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -1913,7 +1954,9 @@ revision, and none is renumbered. an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator serves all four. An insert or a reorder relabels only the items that moved, and - rebuilds nothing. This is not measured. `LAST`/`FIRST` order only the values the fold + rebuilds nothing. **But each relabelled item re-keys every group entry stored under + it**, so a reorder costs O(entries under the moved items), not O(moved items). This + is not measured. `LAST`/`FIRST` order only the values the fold covers, which are the drawn members. *As first written* (superseded): key by item position, relying on `view_shape`'s rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn @@ -1941,9 +1984,11 @@ revision, and none is renumbered. applied at read time (the author's)? Or is it kept per saved `.improv` view only, so that a request with an unsaved key is served without incremental state? *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while. The time is an implementation number. An unsaved hide, collapse, reorder, - page step or prune then shares the saved view's key and costs nothing. Only an - unsaved change of axes, filters or `measure_agg` builds a new key, as a job. + a while. The time is an implementation number. An unsaved collapse, reorder, page + step or prune then shares the saved view's key and builds no operator. An unsaved + hide or show of a row-axis item shares the key too, but it rebuilds that request's + column totals (question 23's default). Only an unsaved change of axes, filters or + `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2051,3 +2096,15 @@ revision, and none is renumbered. that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the gallop as the baseline, and a predecessor query only as an optional optimisation, offered once a measured round-trip cost justifies it. + +23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column + totals"). Today a column total pools only the drawn rows, so hiding a row changes it + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) + aggregate cannot remove a hidden row at read time. Should a hide or show of a + row-axis item **(B)** rebuild that shape's column totals as a job? Or **(A)** should + totals be read from the (row tuple, column tuple) group state, dropping the hidden rows + when read? (A) costs O(hidden rows) per read for `Sum` and `Count`, which is exact only + under question 3's (c) or ADR 005's accumulator. For `Min`, `Max` and the text rule it + costs O(rows) per read. *Recommendation (the author's):* (B). It matches today's code + and ADR 008's finding 10, and it is the smaller change. (A) is the alternative if a + measured hide-heavy workload shows the rebuild is too slow. From 405d433732f34662cea0c6c5a9efd82a706e5a10 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:41:30 -0700 Subject: [PATCH 28/67] docs(adr): ADR 009 keys column totals by the hidden row set, because the shape key is shared (improvise-felo.4) Plain (B) is withdrawn: two clients on one shape key with different hidden rows would share one wrong total. The author now leans to (B'), totals per (shape key, hidden row-axis set), with (A) and a mix still open in question 23; question 14 states the cache cost. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 91 ++++++++++++++++------- 1 file changed, 64 insertions(+), 27 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e034ecae..9525ae13 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -829,8 +829,9 @@ states the same rule. Everything else a request carries is a **read-time parameter**, applied when the output is read, without touching occupancy or cell operators: - - the hidden items of an axis category, which also rebuild the column totals when they - are on the row axis (see "Hiding a row moves the column totals" below); + - the hidden items of an axis category. For the column totals only, the hidden + row-axis set selects a totals entry (see "Hiding a row moves the column totals" + below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -863,7 +864,8 @@ states the same rule. and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no occupancy or cell operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item rebuilds them (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms + the column totals:** a hide or show of a row-axis item selects or builds a totals entry + (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other @@ -880,7 +882,7 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default; question 23).** Today a +- **Hiding a row moves the column totals (the author's default, (B′); question 23).** Today a column total pools only the **drawn** rows. The total's evaluator excludes hidden items through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` evaluates over that memo (`:1066-1070`). The test @@ -888,8 +890,8 @@ states the same rule. `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, and no read-time parameter can take Feb back out of it. So: - - **(B), the default:** a hide or show of a **row-axis** item rebuilds the column - totals of that shape, as a job. ADR 008 says the same ("totals still rebuild on hide + - **(B):** a hide or show of a **row-axis** item rebuilds the column totals of that + shape, as a job. ADR 008 says the same ("totals still rebuild on hide and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what the test above pins. A hide or show of a column-axis item only hides or shows that @@ -902,8 +904,27 @@ states the same rule. it is O(rows) per read, a recombination of every visible row's state. A hide then costs nothing to install, and reads cost more. - The author takes (B), because it matches the code and ADR 008, and it is the smaller - change. The user decides it in question 23. + **(B) has a defect: the key is shared and the hide is not.** Operator state is shared + by every request with the same shape key (above), and an unsaved hide is a parameter of + one client's request. If client X hides Feb and client Y, on the same key, does not, + (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: + - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and + cells stay shared under the shape key. Totals are kept per (shape key, hidden + row-axis set). Each distinct hidden set in use costs one O(cells) rebuild, as a job, + and one set of per-column aggregates. The totals cache multiplies by the number of + distinct hidden sets that recent requests name, and it is evicted like any key + (question 14). + - **(A)**, as above, needs no per-set state at all. + + *The author's lean, re-evaluated:* **(B′)**. Reads stay O(1) per total. And the common + case shares one entry: clients reading a saved view carry its saved hidden set. An + unsaved hide costs that one client a job and a cache entry, and nobody else anything. + (A) is stateless and installs a hide for free. But its `Min`, `Max` and text totals cost + O(rows) per read, per column drawn, which at 100k rows and tens of columns is unlikely + to fit the frame budget (not measured). And its `Sum` waits for question 3's (c). So + (A) is better only for `Sum` and `Count` once question 3's (c) lands, and a mix (A for + those, B′ for the rest) is possible. Plain (B) without the hidden set in the key is + **withdrawn**. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1228,12 +1249,13 @@ host-held mirror of a client. client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the host pushes it is question 13, which is ADR 008's question 52 seen from this side. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the - pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`), + pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at + `50e1314d`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. A hide or show of a row-axis - item rebuilds the totals (clause 7, "Hiding a row moves the column totals"; question - 23). Clause 5 replaces `Expansion::members`. Totals + the columns whose inputs moved, through clause 2's trait. Totals are kept per (shape + key, hidden row-axis set), because a hidden row leaves the total and the key is shared + across clients (clause 7, "Hiding a row moves the column totals"; question 23). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1799,8 +1821,8 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** The totals rebuild on a hide or - show of a row-axis item (clause 7), unless the user takes question 23's (A). + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Totals are kept per (shape key, + hidden row-axis set) under question 23's (B′) (clause 7), unless the user takes (A). 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, @@ -1845,8 +1867,9 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- A hide or show of a row-axis item rebuilds the column totals of that shape, as a job, - under question 23's default (clause 7). Occupancy and cells do not rebuild. +- A hide or show of a row-axis item builds a totals entry per (shape key, hidden + row-axis set), as a job, under question 23's (B′) (clause 7). Occupancy and cells do + not rebuild. The totals cache can hold one entry per distinct hidden set in use. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -1986,9 +2009,12 @@ revision, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key too, but it rebuilds that request's - column totals (question 23's default). Only an unsaved change of axes, filters or - `measure_agg` builds a new key, as a job. + hide or show of a row-axis item shares the key for occupancy and cells. Under + question 23's (B′), it adds a **totals entry** keyed by (shape key, hidden row-axis + set), which is built as a job and evicted like a key. So the totals cache holds up to + one entry per distinct hidden set that recent requests name, not one per key, and a + client that toggles hides through many sets costs a rebuild for each new set. Only an + unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2100,11 +2126,22 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) - aggregate cannot remove a hidden row at read time. Should a hide or show of a - row-axis item **(B)** rebuild that shape's column totals as a job? Or **(A)** should - totals be read from the (row tuple, column tuple) group state, dropping the hidden rows - when read? (A) costs O(hidden rows) per read for `Sum` and `Count`, which is exact only - under question 3's (c) or ADR 005's accumulator. For `Min`, `Max` and the text rule it - costs O(rows) per read. *Recommendation (the author's):* (B). It matches today's code - and ADR 008's finding 10, and it is the smaller change. (A) is the alternative if a - measured hide-heavy workload shows the rebuild is too slow. + aggregate cannot remove a hidden row at read time. The options: + - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, + because the shape key is shared across clients and an unsaved hide belongs to one + client's request. Two clients on one key with different hidden rows would get one + wrong total. + - **(B′)** make the hidden row-axis set part of the key **for totals only**. Each + distinct set in use costs one O(cells) rebuild, as a job, and one set of per-column + aggregates. The cache multiplies by the number of distinct sets (question 14). + - **(A)** read each total from the (row tuple, column tuple) group state, and drop the + hidden rows when it is read. It keeps no per-set state. It costs O(hidden rows) per + read for `Sum` and `Count`, exact only under question 3's (c) or ADR 005's + accumulator, and O(rows) per read for `Min`, `Max` and the text rule. + + *Recommendation (the author's, re-evaluated):* (B′). Reads stay O(1). Clients on a + saved view share its saved hidden set, so the common case is one entry, and an unsaved + hide costs only its own client a job and a cache entry. (A)'s O(rows) reads for `Min` + and `Max` are unlikely to fit the frame budget at ADR 008's fixture (not measured). A + mix is open: (A) for `Sum` and `Count` once question 3's (c) lands, and (B′) for the + rest. From ad60ad2b7ee31e81d6c40d6753ea633cf69568bb Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:45:28 -0700 Subject: [PATCH 29/67] docs(adr): ADR 009 marks (B) withdrawn where it first appears and states ADR 008 revision 28's totals rule (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 9525ae13..770e8d1c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -890,9 +890,11 @@ states the same rule. `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, and no read-time parameter can take Feb back out of it. So: - - **(B):** a hide or show of a **row-axis** item rebuilds the column totals of that - shape, as a job. ADR 008 says the same ("totals still rebuild on hide - and show", its finding 10). Occupancy and cells stay read-time remaps. Collapse and + - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** + item rebuilds the column totals of that shape, as a job. ADR 008 once said the same + (its finding 10). It no longer does: its clause 5, at revision 28, says totals are + **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's + question 23. Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what the test above pins. A hide or show of a column-axis item only hides or shows that column's total. From 0cf16b043a9dc2aa168d743f94ab63354e260618 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:46:25 -0700 Subject: [PATCH 30/67] docs(adr): ADR 009 reads ADR 008 at revision 29 (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 770e8d1c..b082cfb2 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 26** -(PR #308, read at `99322d0e`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 29** +(PR #308, read at `0edd6f2a`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `99322d0e`, its revision 26) · +reads it at `0edd6f2a`, its revision 29) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 26 of `context/adr/008-virtualized-grid.md` on PR #308, at `99322d0e`. Clauses 3, +revision 29 of `context/adr/008-virtualized-grid.md` on PR #308, at `0edd6f2a`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 26, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 29, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1210,7 +1210,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 26 at `99322d0e`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 29 at `0edd6f2a`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. From 0b066d46cb9ca9aeb9f0bbc6ac89136518c2b7be Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 15:47:59 -0700 Subject: [PATCH 31/67] docs(adr): ADR 009 corrects option (A)'s total cost to O(H log d) and leans to (A) (improvise-felo.4) A total over all rows can subtract the hidden rows' groups at read time for every aggregate (Min/Max skip fully hidden keys; the text rule counts distinct keys left), so (A) is stateless and cheap; (B') becomes a memo on it. Also: a hidden column item under a collapsed column group, clause 5a totals under both options, and (B')'s cost stated as an upper bound. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 143 +++++++++++++--------- 1 file changed, 85 insertions(+), 58 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index b082cfb2..1f904f47 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -829,9 +829,8 @@ states the same rule. Everything else a request carries is a **read-time parameter**, applied when the output is read, without touching occupancy or cell operators: - - the hidden items of an axis category. For the column totals only, the hidden - row-axis set selects a totals entry (see "Hiding a row moves the column totals" - below); + - the hidden items of an axis category. For the column totals, the hidden row-axis + set is subtracted at read time (see "Hiding a row moves the column totals" below); - the collapsed groups, and the group membership of axis items; - the drawn order of items; - the page **selection**, which picks a slice; @@ -864,8 +863,8 @@ states the same rule. and page keys. They are never keyed by a position in a level sequence. So a collapse, expand, hide or show of an **axis** category's items changes no occupancy or cell operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item selects or builds a totals entry - (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms + the column totals:** a hide or show of a row-axis item changes them, and they subtract + the hidden rows when read (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or reordering items, changes no operator either: a new item tuple is just a new key, and the order is a read-time parameter. Only a change of the **shape key** selects other @@ -882,7 +881,7 @@ states the same rule. A sum of the members' f64 sums is not bit-equal to eager's one store-order fold (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default, (B′); question 23).** Today a +- **Hiding a row moves the column totals (the author's default, (A); question 23).** Today a column total pools only the **drawn** rows. The total's evaluator excludes hidden items through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` evaluates over that memo (`:1066-1070`). The test @@ -896,15 +895,32 @@ states the same rule. **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's question 23. Occupancy and cells stay read-time remaps. Collapse and expand do not move a total, because it pools the members either way, which is what - the test above pins. A hide or show of a column-axis item only hides or shows that - column's total. - - **(A), the open alternative:** build each total from clause 4's (row tuple, column - tuple) group state, and drop the hidden rows when the total is read. For `Sum` and - `Count` that is O(hidden rows) per read: subtract the hidden rows' state. That is - exact only under question 3's (c), or under ADR 005's wide accumulator, because - f64 subtraction is not exact. For `Min`, `Max` and the one-distinct-value text rule - it is O(rows) per read, a recombination of every visible row's state. A hide then - costs nothing to install, and reads cost more. + the test above pins. A hide or show of a column-axis item hides or shows that + column's total. Under a **collapsed** column group it also changes the group column's + cells and total, because the fold memo skips hidden items on every category. That is a + read-time recombination of the members' item-grain state, and nothing is rebuilt. + - **(A):** keep the per-(column tuple, measure) total state over **all** rows, and + subtract the hidden rows when the total is read. Clause 4 keeps, per group (a row + tuple and a column tuple at item grain), a count of values, a count of numbers, a + running sum and, where `measure_agg` or text needs it, a multiset of values with a + count per key. The column's state is the sum of its groups' states. So at read time + the H cells of the hidden rows in that column are taken out as a **temporary + overlay**, never written back: + - `Count`, and the two counts behind the numeric-first branch: subtract. O(H). + - `Min` and `Max`: walk from the extreme key, skipping each key whose whole count is + hidden. At most H keys are skipped, so O(H log d), with d the distinct keys. + - The one-distinct-value text rule, and `mixed`: count the keys whose whole count is + hidden, and subtract that from the column's number of distinct keys. O(H log d). + Eager's own reading rules (NaN under `PartialEq`, clause 4) apply after the overlay, + unchanged. + - `Sum`: subtract the hidden rows' sums. That is exact only under question 3's (c), + or ADR 005's wide accumulator, because f64 subtraction is not exact. Before step 6 + a `Sum` total stays on today's path anyway. `Avg` is `Sum / Count`. + + So (A) costs **O(H log d) per read for every aggregate**. It is exact, and it keeps no + per-set state. *As first written* (superseded), this item said `Min`, `Max` and the + text rule cost O(rows) per read. That overstated it, as the review of `ad60ad2b` + found. The argument above is the author's and is not measured. **(B) has a defect: the key is shared and the hide is not.** Operator state is shared by every request with the same shape key (above), and an unsaved hide is a parameter of @@ -912,21 +928,28 @@ states the same rule. (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and cells stay shared under the shape key. Totals are kept per (shape key, hidden - row-axis set). Each distinct hidden set in use costs one O(cells) rebuild, as a job, - and one set of per-column aggregates. The totals cache multiplies by the number of - distinct hidden sets that recent requests name, and it is evicted like any key - (question 14). + row-axis set). Each distinct hidden set in use costs at most one O(cells) rebuild, as + a job, and one set of per-column aggregates. The bound is an upper one: an entry can + be derived from the all-rows state by (A)'s subtraction, made once and kept, at O(H) + per column, with `Sum` exact only under question 3's (c). So (B′) is (A) memoised. + The totals cache multiplies by the number of distinct hidden sets that recent + requests name, and it is evicted like any key (question 14). - **(A)**, as above, needs no per-set state at all. - *The author's lean, re-evaluated:* **(B′)**. Reads stay O(1) per total. And the common - case shares one entry: clients reading a saved view carry its saved hidden set. An - unsaved hide costs that one client a job and a cache entry, and nobody else anything. - (A) is stateless and installs a hide for free. But its `Min`, `Max` and text totals cost - O(rows) per read, per column drawn, which at 100k rows and tens of columns is unlikely - to fit the frame budget (not measured). And its `Sum` waits for question 3's (c). So - (A) is better only for `Sum` and `Count` once question 3's (c) lands, and a mix (A for - those, B′ for the rest) is possible. Plain (B) without the hidden set in the key is - **withdrawn**. The user decides it in question 23. + **Aggregates over an expression** (clause 5a) follow the same split. A 5a total's + entries are keyed by context, with the row categories removed. To subtract hidden rows + under (A), the 5a stage's entries must also be indexed by row tuple, at O(entries) more + state, so a read costs O(hidden records). Under (B′), a 5a total per hidden set is its + own map and aggregate stage, at O(records) to build. + + *The author's lean, re-evaluated after the corrected cost:* **(A)**. A read costs + O(H log d), and a hidden set is usually a handful of rows, so a read costs about as + much as (B′)'s O(1) in practice (not measured). (A) keeps no per-set state, which is + what C2's "as stateless as possible" asks, and it never rebuilds or multiplies the + cache. (B′) is kept as a **memo** on top of (A), for a hidden set so large that H log + d per read would not fit the frame budget. The earlier lean to (B′) rested on the + overstated O(rows) cost, and it is **superseded**. Plain (B) without the hidden set in + the key stays **withdrawn**. The user decides it in question 23. - **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). @@ -1255,9 +1278,10 @@ host-held mirror of a client. `50e1314d`), not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. Totals are kept per (shape - key, hidden row-axis set), because a hidden row leaves the total and the key is shared - across clients (clause 7, "Hiding a row moves the column totals"; question 23). Clause 5 replaces `Expansion::members`. Totals + the columns whose inputs moved, through clause 2's trait. A hidden row leaves the total + and the key is shared across clients, so under question 23's default (A) a total + subtracts the request's hidden rows when it is read, at O(H log d) (clause 7, "Hiding a + row moves the column totals"). Clause 5 replaces `Expansion::members`. Totals that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the exact accumulator (recommended) or ADR 005 (clause 8). - **What ADR 008 no longer asks.** Its question 6 is answered ("maintained @@ -1823,8 +1847,9 @@ displayed path until the differential test holds. 008's retirement of `Expansion::members` depends on this step.** 6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Totals are kept per (shape key, - hidden row-axis set) under question 23's (B′) (clause 7), unless the user takes (A). + re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), + a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs + question 3's (c) for that subtraction to be exact. 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to 22. It brings the output types in `improvise-protocol`, the step stamp, @@ -1869,9 +1894,9 @@ longer a prerequisite of step 6. - A mutating frame from a client already clones the workbook for rollback (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) clone per frame. Incremental evaluation does not make the network path O(delta). -- A hide or show of a row-axis item builds a totals entry per (shape key, hidden - row-axis set), as a job, under question 23's (B′) (clause 7). Occupancy and cells do - not rebuild. The totals cache can hold one entry per distinct hidden set in use. +- Under question 23's default (A), every column-total read pays O(H log d) to subtract + the request's hidden rows, and clause 5a's totals need their entries indexed by row + tuple as well. Occupancy and cells do not rebuild. - The shape keys are re-read after every model effect: O(categories) per live key, a cost the post-loop `view_shape` diff paid once per batch. - The browser bundle grows by two small crates. @@ -2011,11 +2036,12 @@ revision, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key for occupancy and cells. Under - question 23's (B′), it adds a **totals entry** keyed by (shape key, hidden row-axis - set), which is built as a job and evicted like a key. So the totals cache holds up to - one entry per distinct hidden set that recent requests name, not one per key, and a - client that toggles hides through many sets costs a rebuild for each new set. Only an + hide or show of a row-axis item shares the key too. Under question 23's default (A) + its totals subtract the hidden rows at read time and add nothing to the cache. Under + (B′), it would add a **totals entry** keyed by (shape key, hidden row-axis set). Then + the cache would hold up to one entry per distinct hidden set that recent requests + name, and a client that toggles hides through many sets would pay to derive each new + entry. Only an unsaved change of axes, filters or `measure_agg` builds a new key, as a job. 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate @@ -2127,23 +2153,24 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). A per-(column tuple, measure) - aggregate cannot remove a hidden row at read time. The options: + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). The options: - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, because the shape key is shared across clients and an unsaved hide belongs to one - client's request. Two clients on one key with different hidden rows would get one - wrong total. - - **(B′)** make the hidden row-axis set part of the key **for totals only**. Each - distinct set in use costs one O(cells) rebuild, as a job, and one set of per-column - aggregates. The cache multiplies by the number of distinct sets (question 14). - - **(A)** read each total from the (row tuple, column tuple) group state, and drop the - hidden rows when it is read. It keeps no per-set state. It costs O(hidden rows) per - read for `Sum` and `Count`, exact only under question 3's (c) or ADR 005's - accumulator, and O(rows) per read for `Min`, `Max` and the text rule. + client's request. + - **(A)** keep each total over all rows, and subtract the request's H hidden cells in + that column when it is read, as a temporary overlay. It costs O(H log d) per read + for every aggregate. `Count` subtracts, `Min` and `Max` skip fully hidden keys from + the extreme, and the text rule counts the distinct keys left. It keeps no per-set + state. `Sum` is exact only under question 3's (c) or ADR 005's accumulator. + Clause 5a's totals need their entries indexed by row tuple too. + - **(B′)** keep totals per (shape key, hidden row-axis set). Reads are O(1), but the + cache multiplies by the distinct hidden sets in use. Each entry costs at most one + O(cells) rebuild, or one O(H) derivation by (A)'s subtraction. That makes (B′) a + memo on (A). - *Recommendation (the author's, re-evaluated):* (B′). Reads stay O(1). Clients on a - saved view share its saved hidden set, so the common case is one entry, and an unsaved - hide costs only its own client a job and a cache entry. (A)'s O(rows) reads for `Min` - and `Max` are unlikely to fit the frame budget at ADR 008's fixture (not measured). A - mix is open: (A) for `Sum` and `Count` once question 3's (c) lands, and (B′) for the - rest. + *Recommendation (the author's, re-evaluated on 2026-10-05):* **(A)**, with (B′) kept + only as a memo for a hidden set so large that its reads would not fit the frame + budget. It is the stateless option, which C2 asks for, and a typical hidden set is + small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on + an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of + `ad60ad2b` found. None of these costs is measured. From dce3ab63c725290450d56223bfa2b886b1924195 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:05:11 -0700 Subject: [PATCH 32/67] docs(adr): ADR 009 folds ADR 008's 2026-10-06 answers: client-side view state, no layout tier, delivery by unaddressed notice (improvise-felo.4) Quotes Q33, Q39 and Q46-Q52 with ids. Request parameters come from a client's ephemeral UI state and saved views only supply defaults; SaveView touches no operator; the circuit relies on no view-fact tier. Q13 (now this ADR's, per "defer this to ADR 009") leans to an unaddressed change notice, with the fates of GridExtent, Dirty and the occupancy delta. Q23 concerns unsaved hides; Q24 asks which parameters must travel. Reads ADR 008 at 5b29335a. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 233 ++++++++++++++++------ 1 file changed, 170 insertions(+), 63 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1f904f47..980ff33c 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,11 +1,11 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 29** -(PR #308, read at `0edd6f2a`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 32** +(PR #308, read at `5b29335a`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 23. **The user has answered none of +ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline comments are Copilot's two and our replies. · **Ticket:** improvise-felo.4 · @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `0edd6f2a`, its revision 29) · +reads it at `5b29335a`, its revision 32) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 29 of `context/adr/008-virtualized-grid.md` on PR #308, at `0edd6f2a`. Clauses 3, +revision 32 of `context/adr/008-virtualized-grid.md` on PR #308, at `5b29335a`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 29, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 32, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -172,6 +172,29 @@ None of this answers this ADR's own questions 1 to 11. should shift the grid being viewed to keep the cursor in the same visual place on the screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). +**Answered on ADR 008 on 2026-10-06** (#308 inline comments; question numbers are ADR +008's): +- Q33 (4191973298, 05:50:48Z): "The backend should only hold model-relevant information. + If we need to persist cursor position/last view or other UI concerns, that would go in + localStorage/cookies/fragment routing/etc." +- Q39 (4191981468, 05:52:14Z): "I think layout version is mostly a view fact here, we have + to separate this from model mutations (value editing, etc.)" +- Q46 (4191986207, 05:53:05Z): "go with an explicit saye-view command" [sic]. +- Q47 (4191992428, 05:53:56Z): "The transient UI state never is reflected in a .improv + file unless an there is an explicit decision to persist it as a saved view (or by + replacing an existing view)" [sic]. +- Q48 (4191999772, 05:55:06Z): "A .improv file has to have at least a default view + definition. UI state is always ephemeral unless explicitly saved: when a new View is + created through the UI, it must not be lost if the model confirms that it was saved." +- Q49 (4192005787, 05:56:07Z): "The only one of these that should be "global" is set + number format as far as I can tell: the other ones are all ephemeral UI state." +- Q50 (4192010101, 05:56:53Z): "Disabling pruning should only be persisted if the + resulting view is explicitly saved." +- Q51 (4192014974, 05:57:40Z): "page is a third axis, do the others need to actually + travel across the wire?" +- Q52 (4192016491, 05:57:59Z): "defer this to ADR 009". This is ADR 008's question + about what replaces the pushed projection stream, and it is this ADR's question 13. + **Answers ADR 008 records that this ADR's text had not caught up with:** - Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. - Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases @@ -211,9 +234,16 @@ the clause it changes: circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. Clause 14, question 17. 6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's - questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 42 and - 43, quoted above. + questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 39, 42, + 43 and 46 to 52, quoted above. Clauses 5a, 7, 9, 12, 13 and 14 follow them. +7. **View configuration is the client's until saved** (Q46 to Q50). Hiding, collapsing, + page selection, axis assignment and pruning are ephemeral UI state per client until an + explicit `SaveView`. Only number format is "global" (Q49). So operator state is keyed + by the request, never by a client's unsaved settings held on the host. Clause 7, + questions 23 and 24. +8. **No basis tier this ADR relies on is a view fact** (Q39), and the host keeps no + per-client UI state (Q33). Clauses 7 and 12, question 13. ## Context @@ -437,7 +467,10 @@ in the batch's consolidated delta. So: `bump_changed_records` can then read the delta instead of diffing two `record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, -because it observes `view_shape` and not cells. +because it observes `view_shape` and not cells. Nothing in this ADR reads or bumps it. +The user calls it "mostly a view fact", to be separated from model mutations (ADR 008 +Q39). ADR 001's layout tier is to be split (ADR 008 question 68), and this ADR relies +only on `document_generation` and the record tier. **The dirty flag's content half does not move onto the cell delta alone.** `document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it @@ -841,12 +874,20 @@ states the same rule. groups, its page selection and `prune_empty`. Those are all read-time parameters except the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, `AddItem`, reorder and page step, which is what item grain and question 7 exist to - avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40), and it is - unchanged. + avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40). That tier + is a view fact by the user's Q39 answer, and the circuit does not depend on it. - A saved `.improv` view supplies the key and the parameters to a request that names it. - An unsaved request carries its own (ADR 008's flagged readings 1 and 3, its question - 51). + **Where the key and the parameters come from (revised for ADR 008's 2026-10-06 + answers).** Hiding, collapsing, the page selection, axis assignment and pruning are + ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number + format is "global" (Q49). So a request carries its own key and parameters, taken from + that client's UI state. A saved `.improv` view only supplies the **defaults** a client + starts from, and the circuit never stores a client's unsaved settings. Number format is + display-only (gotcha 8) and never reaches the circuit, except through clause 13's + widths if those were adopted. Whether every read-time parameter must travel in the + request (the user's Q51: "do the others need to actually travel across the wire?") is + question 24. A hide and a collapse change model-side aggregates (the totals, and a + collapsed group's fold), so the author reads them as needing to travel. Requests with the same key share one set of operators, whichever clients sent them. The live keys are a **cache**: a key enters when a request names it, and it is evicted once no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a @@ -888,7 +929,11 @@ states the same rule. `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, - and no read-time parameter can take Feb back out of it. So: + and no read-time parameter can take Feb back out of it. **Whose hide this is** (ADR 008, + Q46 to Q50): a hide is one client's ephemeral UI state until an explicit `SaveView`, + and never a model mutation. So the options below concern a request's **unsaved** hidden + set. A saved view's hidden set arrives only through `SaveView`, as part of the saved + definition a request starts from, and it is subtracted the same way. So: - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** item rebuilds the column totals of that shape, as a job. ADR 008 once said the same (its finding 10). It no longer does: its clause 5, at revision 28, says totals are @@ -962,6 +1007,13 @@ states the same rule. cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user ruled. Any measurement this ADR later reports against them is stated as a ratio against an unrelated operation on the same run, as ADR 008's question 13 requires. +- **`SaveView`, and the default view (Q46 to Q48).** `SaveView` is a model command that + writes a view definition, acknowledged by the model, and a document always has at + least a default view. **Neither touches the circuit.** A saved definition moves no + cell and no formula, so it emits no cell delta and no formula delta. Shape keys come + from requests, so saving builds no operator. The next request that names the saved + view simply starts from its defaults. The acknowledgement is ADR 008's + (`ServerFrame::Accepted`, its question 69), not this ADR's. - **A crossing count (ADR 008's second request).** Each step's occupancy output carries the number of tuples whose count crossed zero. ADR 008's ordered index uses it to choose between an insertion and a rebuild by sorting. @@ -979,7 +1031,7 @@ states the same rule. Re-reading the shape keys costs O(categories) per live key per model effect, and the formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by -`layout_version` after the loop, as today. +`layout_version` after the loop, as today, and the circuit reads neither. **The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first written* (superseded by item grain): `AddItem` moved the operator shape, so the view was @@ -1233,7 +1285,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 29 at `0edd6f2a`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 32 at `5b29335a`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1407,8 +1459,10 @@ Three things follow: until then (see "Names and ADR 007" in clause 3). The host attaches the absolute position from the current item order when it sends the element. A delta keyed by position alone would go stale with the next item insert. -- **An item edit is sent to every subscriber as an axis change.** "Item inserted at index - j of level L", or removed, or moved. It goes to **every** subscriber of a shape with +- **An item edit is sent to every consumer as an axis change.** "Item inserted at index + j of level L", or removed, or moved. Under question 13's (b) it rides on the change + notice, which every consumer receives. Under (c) it goes to **every** subscriber of a + shape with that category on an axis, wherever its window lies, because an insert before a window's top-left shifts every position that subscriber holds. A client re-derives its positions arithmetically, at O(levels) per position, and re-anchors its top-left to the @@ -1482,7 +1536,8 @@ window (ADR 008 question 66). A client alternates three steps: - it fills, asking for the next rows past its last absolute row position with a value in its displayed columns, and likewise for columns. -The model never re-runs a client's fixpoint. A subscribed client receives index (ii)'s +The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from +the change notice which records moved, and re-asks. Under (c), a subscribed client receives index (ii)'s deltas for its S, prunes again locally, and fills again only when a row or column has gone. Whether the loop always terminates is **not proven**. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A @@ -1503,7 +1558,7 @@ search appears (next paragraph). What forward-only growth simplifies: (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**. An item edit is an axis change and goes to every subscriber (above). A change at + only**. An item edit is an axis change and goes to every consumer (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill whose front is before q finds the change when it gets there. - **Resumable fills.** A fill's whole state is its request and its last absolute position @@ -1612,7 +1667,8 @@ occupied answers "empty axis". Both need the predecessor query. loss, which a fresh snapshot answers. What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. -- **T4. A snapshot, then deltas.** A subscription starts with the integrated result at +- **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the + snapshot and the notices are the deltas' stamps). A subscription starts with the integrated result at step n: the window as a Z-set of current values, each with weight +1. Deltas from step n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for @@ -1649,40 +1705,66 @@ consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that bo The alternative is a `serde` feature on `zset`. The author leans to the conversion, because what crosses is the projected result, not the crate's own element type. -**How stateless the host can be.** The user: "as stateless as possible" (C2). These are the -options of question 13, and ADR 008's question 52 asks the push half of the same thing. -- **(a) Pure fetch.** The host keeps nothing per client. A client calls - `project(request)`. It learns its result may be stale from a change notice addressed to - no one: the accepted batch's step. It then refetches when its own policy says to (Q42). - Each refetch costs O(visible). -- **(b) Soft-state subscriptions** (the author's leaning). A request may ask to be kept - current. For each subscription the host keeps the request and the last step it sent, - and nothing else: no copy of the window, no cursor and no viewport. After each accepted - batch it reads the delta of each subscribed shape, restricts it to the request's window, - and sends it under T3 to T5. **The table is a cache.** If it is lost (a restart, an - eviction, a reconnect), a client resumes by resending its request and the last step it - applied, and the host answers with a snapshot. The host could instead keep a short ring - of past deltas per shape; the author's lean is no ring at first. So the user's "recent - projections + clients" (C2) are this table plus clause 7's shape cache, and **both can be - rebuilt from the requests**. A subscription costs O(request) in memory, not O(window). -- **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and +**How stateless the host can be, and what replaces the pushed stream.** The user: "as +stateless as possible" (C2), and "The backend should only hold model-relevant +information" (ADR 008 Q33). ADR 008's question 52, what replaces the pushed projection +stream, was answered "defer this to ADR 009", so it is this ADR's question 13. The +options, in ADR 008's lettering: +- **(a) Never learns: refetch by policy.** The host keeps nothing and says nothing. A + client refetches when its own policy says to (Q42). A change made by another client + stays invisible until then. +- **(b) An unaddressed change notice** (Flux style). After each accepted batch, the + model emits one notice to every consumer, addressed to none. It carries the step stamp + `(document_generation, step)` (T3), the identities of the records the batch moved (the + record tier, clause 3), each item edit as an axis change (above), and a flag when the + formula set moved (clause 7). Each client + decides for itself whether its window is affected. A moved record whose coordinates + fall under the client's axes and filters can change a cell or a total in its window, so + the client refetches by `project`, or by question 22's successor query. A formula change + means refetching. The host keeps no per-client state. A notice costs O(records moved), + and every client pays a test per notice. +- **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask + to be kept current. The host keeps the request and the last step it sent, then pushes + the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost + table costs a snapshot). But it is per-client state on the backend, and it holds a + window, which is a client's view parameter. +- **(d) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, - `:822` and `:981` at `50e1314d`). It keeps each client's `ViewState` and grid on the host - and diffs against them. ADR 008 lists it as obsolete under the split, and this ADR does - not propose it. + `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -A shape is kept while some subscription or recent request names it. Eviction drops its -operators, and the next request rebuilds them as a job (clause 7). How long "recent" lasts -is a cost the user has not set (question 14). +*The author's recommendation, revised for Q33:* **(b)**. It is the only option that keeps +a client informed while the host holds nothing per client. It is also the most network +transparent: one broadcast stream of model events, consumed the same way in process (the +TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would +otherwise refetch too much, if a measured round-trip cost justifies keeping per-client +state. *As first written* (superseded), this clause leaned to (c), which it then called +(b). That was before the user said the backend holds only model-relevant information. +Under (b) the user's "recent projections + clients" (C2) shrinks to clause 7's shape +cache, which is keyed by requests and holds no client. -**What a client must hold to resume:** its request, the last step it applied, its -integrated window, and its basis (ADR 001). The host holds nothing that a client needs -back. +**What becomes of today's pieces under (b)** (the author's): +- **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its + levels' item counts, which a client derives from the category lists it fetches. A + pruned axis has no global extent under client-side pruning (clause 12), and question + 22's successor query answers "is there more". +- **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which + widget a change can reach, moves into the client's test of the notice. +- **The occupancy delta** is not pushed. It stays a model-side output that keeps question + 22's index current. A client learns of a new or vanished row from the notice's records, + and re-asks. -**The basis.** ADR 001's basis is published with the result it describes, and after it -(gotcha 43). The host builds it from the request's window (ADR 008, lens row 18), not from -a mirror. `published_basis` (`project.rs:585` at `50e1314d`) already takes the `GridCache` -it describes, not a cursor. +A shape is kept while recent requests name it. Eviction drops its operators, and the next +request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has +not set (question 14). + +**What a client must hold to resume:** its request, the last step it applied, its window, +and its basis. The host holds nothing that a client needs back. + +**The basis.** ADR 001's basis is answered with the result it describes, and after it +(gotcha 43), built from the request's window, never from a mirror. `published_basis` +(`project.rs:585` at `50e1314d`) already takes the `GridCache` it describes, not a +cursor. Under the user's Q39 answer, only `document_generation` and the record tier are +model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). ### 13. Incremental max width (the user's question on ADR 008) @@ -1751,7 +1833,7 @@ orders come from the clauses and are not measured. | `ExactSum`, per summed group | 8 | 568 bytes per group | | occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | | the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | -| the subscription table | 12 | O(request) per subscription | +| the subscription table, only under question 13's (c) | 12 | O(request) per subscription | **Proposed by the author:** - **The store is the integral, so it is not copied.** The integrated base a rebuild reads @@ -1875,7 +1957,7 @@ longer a prerequisite of step 6. evaluator. - Live sources that can report changes plug in without touching the evaluator. - One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client UI state: only a subscription table that can be rebuilt + the host keeps no per-client state under question 13's (b). Option (c) would add only a subscription table that can be rebuilt from the requests. **Bad.** @@ -1911,8 +1993,8 @@ longer a prerequisite of step 6. Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 23 were added by the 2026-10-05 -revision, and none is renumbered. +none of questions 1 to 11 on #309.** Questions 12 to 24 were added by the 2026-10-05 and 2026-10-06 +revisions, and none is renumbered. 1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not @@ -2021,12 +2103,20 @@ revision, and none is renumbered. the write path. Revisit it if a measured workload shows jitter that hysteresis does not cure. -13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per - client** (clause 12). **Owner: this ADR.** ADR 008's question 52 asks the push half, - and should point here. *Recommendation (the author's):* (b), soft-state - subscriptions that can be rebuilt from the request and the last step, with no copy of - a client's window on the host. When its table is lost, (b) falls back to (a). (c) is - obsolete under ADR 008's split. +13. *[New; now this ADR's to carry, by the user's "defer this to ADR 009" on ADR 008's + question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: + this ADR.** The options are: + - (a) never learn, and refetch by policy; + - (b) an unaddressed change notice per accepted batch, carrying the step stamp, the + moved records, axis changes and a formula flag, after which each client decides; + - (c) push the model's data changes through soft-state subscriptions; + - (d) today's per-mirror push, which is obsolete. + + *Recommendation (the author's, revised for Q33):* (b). The host keeps nothing per + client, and in-process and remote consumers read one stream. (c) is an optimisation + to adopt only if measured round trips justify per-client state. Under (b), + `GridExtent` is derived by the client, `Dirty` is retired with the mirrors, and the + occupancy delta stays model-side, feeding question 22's index. 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2153,7 +2243,10 @@ revision, and none is renumbered. 23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). The options: + (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). Under ADR 008's 2026-10-06 + answers (Q46 to Q50), a hide is a client's UI parameter until an explicit `SaveView`. + So this question concerns the hidden set a request carries, and it assumes that set + travels (question 24). The options: - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, because the shape key is shared across clients and an unsaved hide belongs to one client's request. @@ -2174,3 +2267,17 @@ revision, and none is renumbered. small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of `ad60ad2b` found. None of these costs is measured. + +24. *[New.]* **Which read-time parameters must travel in a request** (clause 7; the user + on ADR 008's Q51: "page is a third axis, do the others need to actually travel across + the wire?"). **Owner: ADR 008** (its question 51). This question states only what the + circuit needs. The page selection travels as the third axis. A **hide** changes + model-side aggregates (the column totals, and a collapsed group's fold), and so does a + **collapse** (a fold row's value). The user: "Aggregation should always be done + model-side for consistency" (Q38). So if hidden items and collapsed groups did not + travel, a client could not get its totals and folds from the model. It would have to + compute them itself, against Q38, or show totals over all rows. The **drawn order**, + **number format** and **prune** (under client-side pruning) change no aggregate, and + need not travel. *Recommendation (the author's):* hidden items and collapsed groups + travel with the request; the drawn order, number format and prune stay with the + client. Not decided. From 5a8dc486ca14e1aaf112e81f27411c0d2b5ae907 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:11:03 -0700 Subject: [PATCH 33/67] docs(adr): ADR 009's change notice carries moved output keys, not records alone (improvise-felo.4) A record test misses formulas whose inputs sit at another stem; the notice now carries the moved keys per live shape (the author's lean), with a conservative client rule as the alternative, and a formula or measure_agg flag. Also: the occupancy delta is not pushed under (b), T4 and T5 are scoped, measure_agg is model state, Q24 names ADR 008's alternative forms, and records-mode extent is fetched. Reads ADR 008 at 9df2e478. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 105 +++++++++++++++------- 1 file changed, 75 insertions(+), 30 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 980ff33c..ac524b74 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 32** -(PR #308, read at `5b29335a`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 33** +(PR #308, read at `9df2e478`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `5b29335a`, its revision 32) · +reads it at `9df2e478`, its revision 33) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 32 of `context/adr/008-virtualized-grid.md` on PR #308, at `5b29335a`. Clauses 3, +revision 33 of `context/adr/008-virtualized-grid.md` on PR #308, at `9df2e478`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 32, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 33, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -881,7 +881,10 @@ states the same rule. answers).** Hiding, collapsing, the page selection, axis assignment and pruning are ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number format is "global" (Q49). So a request carries its own key and parameters, taken from - that client's UI state. A saved `.improv` view only supplies the **defaults** a client + that client's UI state. **`measure_agg` is the exception:** it is model state + (`Model::measure_agg`, `types.rs:370` at `50e1314d`), part of every shape key but read + from the model, never taken from a request (ADR 008 Q51 says the same). A saved + `.improv` view only supplies the **defaults** a client starts from, and the circuit never stores a client's unsaved settings. Number format is display-only (gotcha 8) and never reaches the circuit, except through clause 13's widths if those were adopted. Whether every read-time parameter must travel in the @@ -1285,7 +1288,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 32 at `5b29335a`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 33 at `9df2e478`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1321,10 +1324,12 @@ host-held mirror of a client. occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first written*, its consumer was the per-view frame observation and its reset of each client's - cursor (gotcha 24). Under the split the host holds no cursor. So the delta is a model - fact, published on clause 12's delivery stream to the consumers of that shape, and a - client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the - host pushes it is question 13, which is ADR 008's question 52 seen from this side. + cursor (gotcha 24). Under the split the host holds no cursor, and no client's read-time + parameters, so it cannot compute any client's **visible** axis. The delta is a + model-side output at item grain that keeps question 22's index current. Under question + 13's (b) it is **not pushed**. A client learns of a new or vanished row from the change + notice, recomputes its own visible axis, and shifts its grid so that the cursor keeps + its place (Q34). Only under (c) would a window's delta be pushed. - **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at `50e1314d`), @@ -1537,7 +1542,7 @@ window (ADR 008 question 66). A client alternates three steps: in its displayed columns, and likewise for columns. The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from -the change notice which records moved, and re-asks. Under (c), a subscribed client receives index (ii)'s +the change notice which output keys moved, and re-asks. Under (c), a subscribed client receives index (ii)'s deltas for its S, prunes again locally, and fills again only when a row or column has gone. Whether the loop always terminates is **not proven**. The fill only moves forward through absolute positions, so it adds finitely many candidates before the axis ends. A @@ -1558,7 +1563,9 @@ search appears (next paragraph). What forward-only growth simplifies: (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its rank by that arithmetic. - **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**. An item edit is an axis change and goes to every consumer (above). A change at + only**, and to their moved keys in a question-13 (b) notice. A formula cell inside the + range moves when any of its inputs move, wherever those sit, so it is tested by the + moved output keys, never by the position of the record that moved. An item edit is an axis change and goes to every consumer (above). A change at position q matters to a fill only if q lies between its top-left and its front. A fill whose front is before q finds the change when it gets there. - **Resumable fills.** A fill's whole state is its request and its last absolute position @@ -1668,12 +1675,15 @@ occupied answers "empty axis". Both need the predecessor query. delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the - snapshot and the notices are the deltas' stamps). A subscription starts with the integrated result at + snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at step n: the window as a Z-set of current values, each with weight +1. Deltas from step n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for today's bootstrap compaction (gotcha 32). -- **T5. Order holds across streams, and coalescing happens within a stream.** A slow +- **T5. Order holds across streams, and coalescing happens within a stream** (as stated, + for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client + merges pending notices into one, taking the union of their moved keys and the latest + stamp, and refetches once). A slow consumer's pending deltas on one stream can be **summed** into one delta. Z-set addition is commutative and associative, so the sum is exact, and the queue is bounded by the size of the result, not by the number of steps. When the summed delta outgrows @@ -1714,15 +1724,37 @@ options, in ADR 008's lettering: client refetches when its own policy says to (Q42). A change made by another client stays invisible until then. - **(b) An unaddressed change notice** (Flux style). After each accepted batch, the - model emits one notice to every consumer, addressed to none. It carries the step stamp - `(document_generation, step)` (T3), the identities of the records the batch moved (the - record tier, clause 3), each item edit as an axis change (above), and a flag when the - formula set moved (clause 7). Each client - decides for itself whether its window is affected. A moved record whose coordinates - fall under the client's axes and filters can change a cell or a total in its window, so - the client refetches by `project`, or by question 22's successor query. A formula change - means refetching. The host keeps no per-client state. A notice costs O(records moved), - and every client pays a test per notice. + model emits one notice to every consumer, addressed to none. It carries: + - the step stamp `(document_generation, step)` (T3); + - each item edit, as an axis change (above); + - a flag when the formula set or `measure_agg` moved. Either may move every aggregate + without moving a record. `measure_agg` has no wire command today, and changes only + with a load, which moves `document_generation`; + - **what moved, as the model computes it** (the author's lean, below). + + **Records alone are not enough** (found by the review of `dce3ab63`). A client could + test whether each moved record falls under its own axes and filters, but that misses + every formula whose inputs sit at another stem: an inner `WHERE` naming another item, + or a ref to an item of a page, filter or None category (clause 5's one join). For + example, a client on page `Region=West` shows `EastShare = Revenue / SUM(Revenue WHERE + Region=East)`. Another client writes an East record. The West client's test says "not + mine", and its cells stay stale until its own refetch policy fires. Two remedies: + - **The author's lean: the notice carries the moved keys the circuit already + computes.** Clause 5 finds the affected `(stem, formula)` pairs, and clause 4 the + moved group keys. That is the circuit's whole job. For each **live shape key**, which + is a model-side cache keyed by requests and never by a client, the notice lists the + moved output keys: cells, formula cells and column-tuple totals. A client intersects + them with its window by identity, so the model decides relevance and the client only + looks it up. A notice then costs O(moved output keys across live shapes), not + O(records moved). + - **The alternative: a conservative client rule.** The notice carries records only, + and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to + a page, filter or None item, or an aggregate) treats **every** moved record as + relevant. The notice is small, but such windows refetch on every batch. + + Under either remedy, the client refetches what it needs by `project`, or by question + 22's successor query. A formula or `measure_agg` change means refetching the window. + The host keeps no per-client state. - **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask to be kept current. The host keeps the request and the last step it sent, then pushes the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost @@ -1732,8 +1764,12 @@ options, in ADR 008's lettering: `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -*The author's recommendation, revised for Q33:* **(b)**. It is the only option that keeps -a client informed while the host holds nothing per client. It is also the most network +*The author's recommendation, revised for Q33:* **(b)**, with moved keys per live shape. +It is the only option that keeps a client informed while the host holds nothing per +client. **The bigger notice does not change the choice against (c).** (b)'s notice lists +keys per live shape, one broadcast for all clients. (c) pushes values per window, and so +needs per-client state. The key lists cost O(moved keys) per batch, where (c)'s deltas +cost the same per subscriber. It is also the most network transparent: one broadcast stream of model events, consumed the same way in process (the TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would otherwise refetch too much, if a measured round-trip cost justifies keeping per-client @@ -1746,7 +1782,9 @@ cache, which is keyed by requests and holds no client. - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A pruned axis has no global extent under client-side pruning (clause 12), and question - 22's successor query answers "is there more". + 22's successor query answers "is there more". **In records mode** the row extent is the + record count that the page and filter admit. That is a model fact, which the client must + **fetch**, because it cannot derive it. - **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which widget a change can reach, moves into the client's test of the notice. - **The occupancy delta** is not pushed. It stays a model-side output that keeps question @@ -2107,8 +2145,10 @@ revisions, and none is renumbered. question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: this ADR.** The options are: - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the step stamp, the - moved records, axis changes and a formula flag, after which each client decides; + - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis + changes, a formula or `measure_agg` flag, and the moved output keys per live shape + key (the author's lean). The alternative is moved records plus a conservative rule + for windows with cross-stem formulas; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2276,7 +2316,12 @@ revisions, and none is renumbered. **collapse** (a fold row's value). The user: "Aggregation should always be done model-side for consistency" (Q38). So if hidden items and collapsed groups did not travel, a client could not get its totals and folds from the model. It would have to - compute them itself, against Q38, or show totals over all rows. The **drawn order**, + compute them itself, against Q38, or show totals over all rows. **Alternative forms**, + which ADR 008's Q51 names and which are closer to the user's lean that only the axes + travel: a **collapse** as an axis at the group's level (the request's row axis names + the group level instead of the items), and a **hide** as a filter-key exclusion. Both + are already in the shape key's vocabulary (clause 7), so either form gives the model + what it needs without a separate parameter. The **drawn order**, **number format** and **prune** (under client-side pruning) change no aggregate, and need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the From e152b667fc3043bac964fb2e44946502c406a707 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:15:38 -0700 Subject: [PATCH 34/67] docs(adr): ADR 009 states delivery as host state against wire traffic and leans to one topic per shape (improvise-felo.4) Corrects the (b)/(c) comparison, adds (b'): notices by shape-key topic, O(1) delivery state per client. Also: fold rows match through the client's member-to-group remap, axis changes do not coalesce, display relabels ride the notice under ADR 007, and Q24's alternative forms change the shape key. Reads ADR 008 at 385784e7. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 112 +++++++++++++++------- 1 file changed, 78 insertions(+), 34 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ac524b74..00167007 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 33** -(PR #308, read at `9df2e478`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 34** +(PR #308, read at `385784e7`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `9df2e478`, its revision 33) · +reads it at `385784e7`, its revision 34) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 33 of `context/adr/008-virtualized-grid.md` on PR #308, at `9df2e478`. Clauses 3, +revision 34 of `context/adr/008-virtualized-grid.md` on PR #308, at `385784e7`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 33, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 34, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1288,7 +1288,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 33 at `9df2e478`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 34 at `385784e7`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1682,8 +1682,11 @@ occupied answers "empty axis". Both need the predecessor query. today's bootstrap compaction (gotcha 32). - **T5. Order holds across streams, and coalescing happens within a stream** (as stated, for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client - merges pending notices into one, taking the union of their moved keys and the latest - stamp, and refetches once). A slow + merges pending notices into one, taking the union of their moved keys and flags and the + latest stamp, and refetches once. **Axis changes do not merge.** They are positional, + so they are kept as a list and applied in their original order. A + `document_generation` change inside the merged run voids the merge, because the old + keys belong to a document that is gone, and the client refetches from scratch). A slow consumer's pending deltas on one stream can be **summed** into one delta. Z-set addition is commutative and associative, so the sum is exact, and the queue is bounded by the size of the result, not by the number of steps. When the summed delta outgrows @@ -1730,7 +1733,12 @@ options, in ADR 008's lettering: - a flag when the formula set or `measure_agg` moved. Either may move every aggregate without moving a record. `measure_agg` has no wire command today, and changes only with a load, which moves `document_generation`; - - **what moved, as the model computes it** (the author's lean, below). + - **what moved, as the model computes it** (the author's lean, below); + - **display relabels.** Under ADR 007 a rename is one `display` write, which moves no + key (clause 3), so no moved-key list can carry it. The notice carries it as `(id, + new display)`, and the client relabels what it holds without refetching. Before ADR + 007, a rename is a removal and an insertion of every cell under the old name, and it + arrives as moved keys. **Records alone are not enough** (found by the review of `dce3ab63`). A client could test whether each moved record falls under its own axes and filters, but that misses @@ -1745,7 +1753,10 @@ options, in ADR 008's lettering: is a model-side cache keyed by requests and never by a client, the notice lists the moved output keys: cells, formula cells and column-tuple totals. A client intersects them with its window by identity, so the model decides relevance and the client only - looks it up. A notice then costs O(moved output keys across live shapes), not + looks it up. **A collapsed fold row has no maintained output key under item grain** + (clause 7: it is combined at read time). So a moved member key goes through the + client's own member-to-group remap before it is matched against the window. The + client holds that remap, since collapse is its UI state. A notice then costs O(moved output keys across live shapes), not O(records moved). - **The alternative: a conservative client rule.** The notice carries records only, and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to @@ -1764,21 +1775,46 @@ options, in ADR 008's lettering: `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. -*The author's recommendation, revised for Q33:* **(b)**, with moved keys per live shape. -It is the only option that keeps a client informed while the host holds nothing per -client. **The bigger notice does not change the choice against (c).** (b)'s notice lists -keys per live shape, one broadcast for all clients. (c) pushes values per window, and so -needs per-client state. The key lists cost O(moved keys) per batch, where (c)'s deltas -cost the same per subscriber. It is also the most network -transparent: one broadcast stream of model events, consumed the same way in process (the -TUI) and remote (T6). (c) is kept as an optimisation for a remote client that would -otherwise refetch too much, if a measured round-trip cost justifies keeping per-client -state. *As first written* (superseded), this clause leaned to (c), which it then called -(b). That was before the user said the backend holds only model-relevant information. -Under (b) the user's "recent projections + clients" (C2) shrinks to clause 7's shape -cache, which is keyed by requests and holds no client. +**The trade between (b) and (c) is host state against wire traffic** (the author's; +corrected after the review of `5a8dc486`; nothing here is measured). +- **(b) keeps no per-client state, and pays in traffic.** Its one broadcast is still + delivered to all N clients, and each receives every live shape's moved keys. That is + O(N × all moved keys) on the wire per batch. In this ADR's primary case, a live source + streaming into a large table, every remote client would receive every moved key, most + of them for shapes it does not read. +- **(c) keeps per-client state, and saves traffic.** It holds the request and the last + step per subscriber, which is a client's window and so a view parameter on the host, + against Q33. In return it sends each client only its window's deltas. +- *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) + per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. -**What becomes of today's pieces under (b)** (the author's): +**A middle form, (b′): one topic per shape key.** A client subscribes to the topic of +its own shape key, and receives notices only for that shape. The host keeps O(1) per +client: a topic subscription, which names a shape key and holds no window, cursor or +view. It is the same kind of delivery state a socket connection already is. The costs: +- **Traffic.** O(Σ over clients of the moved keys of that client's shape) per batch. It + saves nothing when every client reads one shape, and much when shapes differ. It is + still not restricted to the window; only (c) does that. +- **"As stateless as possible" (C2) and Q33.** It is per-client state, but it holds no UI + state, and it can be rebuilt by the next request, which names its shape. The author + reads it as delivery state, not UI state, but the user has not said so. +- **Network transparency.** The topic is a filter applied when sending, so in-process + and remote consumers still read the same notices (T6). The TUI simply subscribes to its + own shape. + +*The author's recommendation, revised:* **(b′)**. It keeps the host free of UI state, +while cutting (b)'s traffic to the shapes each client reads. A client that names no +topic receives the full broadcast, which is (b). (c) stays an optimisation for a client +whose window is a small part of a busy shape, if measured traffic justifies holding a +window on the host. All of this is unmeasured. + +*Earlier still* (superseded), this clause leaned to (c), which it then called (b). That +was before the user said the backend holds only model-relevant information. Under (b′), +the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is +keyed by requests and holds no client, plus the topic subscriptions, which hold one shape +key per client. + +**What becomes of today's pieces under (b) and (b′)** (the author's): - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A pruned axis has no global extent under client-side pruning (clause 12), and question @@ -1995,7 +2031,7 @@ longer a prerequisite of step 6. evaluator. - Live sources that can report changes plug in without touching the evaluator. - One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client state under question 13's (b). Option (c) would add only a subscription table that can be rebuilt + the host keeps no per-client UI state under question 13's (b′), only one topic per client (none under plain (b)). Option (c) would add a subscription table that can be rebuilt from the requests. **Bad.** @@ -2146,17 +2182,21 @@ revisions, and none is renumbered. this ADR.** The options are: - (a) never learn, and refetch by policy; - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis - changes, a formula or `measure_agg` flag, and the moved output keys per live shape - key (the author's lean). The alternative is moved records plus a conservative rule - for windows with cross-stem formulas; + changes, display relabels, a formula or `measure_agg` flag, and the moved output + keys per live shape key. The alternative is moved records plus a conservative rule + for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); + - (b′) the same notices, delivered by topic: one topic per shape key, with O(1) + per-client state that holds no UI state; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. - *Recommendation (the author's, revised for Q33):* (b). The host keeps nothing per - client, and in-process and remote consumers read one stream. (c) is an optimisation - to adopt only if measured round trips justify per-client state. Under (b), - `GridExtent` is derived by the client, `Dirty` is retired with the mirrors, and the - occupancy delta stays model-side, feeding question 22's index. + *Recommendation (the author's, revised):* (b′). It holds no UI state on the host and + cuts (b)'s traffic to the shapes each client reads. (b) is what a client that names no + topic gets. (c) is an optimisation if measured traffic justifies holding a window. + Under all three, `GridExtent` is derived by the client (in records mode it is + fetched), `Dirty` is retired with the mirrors, and the occupancy delta stays + model-side, feeding question 22's index. *As first written*, this recommended plain + (b); the trade it skipped is wire traffic. 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2321,7 +2361,11 @@ revisions, and none is renumbered. travel: a **collapse** as an axis at the group's level (the request's row axis names the group level instead of the items), and a **hide** as a filter-key exclusion. Both are already in the shape key's vocabulary (clause 7), so either form gives the model - what it needs without a separate parameter. The **drawn order**, + what it needs without a separate parameter. **But both change the shape key.** A hide + as a filter-key exclusion makes each distinct hidden set its own key, at an O(cells) + rebuild per set, and it loses question 23's (A), where every hidden set shares one + key and is subtracted at read time. A collapse as an axis level does the same for + each collapse state. The **drawn order**, **number format** and **prune** (under client-side pruning) change no aggregate, and need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the From 8ee0d78c64d7c676c703ebf8f133d174d0a256f5 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:19:00 -0700 Subject: [PATCH 35/67] docs(adr): ADR 009 sends every batch header to every topic and pins subscribed shapes (improvise-felo.4) Answers the Opus review's B8 and ChatGPT Codex's cache-lifecycle P on e152b667: headers (stamp, generation, axis changes, relabels, flags, eviction list) reach every client so T3's gap rule holds; only moved-key lists are filtered by shape; a topic subscription is a lease that pins its shape and evictions are announced; the general refetch rule covers gaps, restarts and evictions; the shape-key switch rule is stated; Q13 puts the Q33 question about topic state to the user. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 81 ++++++++++++++++++----- 1 file changed, 64 insertions(+), 17 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 00167007..e02328c1 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -893,7 +893,8 @@ states the same rule. collapsed group's fold), so the author reads them as needing to travel. Requests with the same key share one set of operators, whichever clients sent them. The live keys are a **cache**: a key enters when a request names it, and it is evicted once - no recent request has named it (clause 12). Elsewhere in this ADR, "a shape" means a + no recent request has named it **and no live topic subscription pins it**. Every + eviction is announced in the next notice header (clause 12). Elsewhere in this ADR, "a shape" means a shape key. *As first written* (superseded twice): the operator shape was `view_shape` plus group @@ -1671,7 +1672,13 @@ occupied answers "empty axis". Both need the predecessor query. `(document_generation, step)`. The circuit steps once per model effect (clause 3), and ADR 001 already puts batches in one total order. A consumer that has applied step n drops anything at or below n, so a duplicate does no harm, and it treats a gap as a - loss, which a fresh snapshot answers. What is published is the **batch's** consolidated + loss, which a fresh snapshot answers. Under question 13's (b′), every client receives + every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is + always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees + `document_generation` change, finds its shape in an eviction list, or receives a + notice it cannot interpret, **refetches**. That covers a host restart: the stamp + sequence starts again, which is a discontinuity, and a client holding a cached window + across it refetches. The same rule holds for a client under (a), (b) and (c). What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the @@ -1789,15 +1796,45 @@ corrected after the review of `5a8dc486`; nothing here is measured). per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. **A middle form, (b′): one topic per shape key.** A client subscribes to the topic of -its own shape key, and receives notices only for that shape. The host keeps O(1) per -client: a topic subscription, which names a shape key and holds no window, cursor or -view. It is the same kind of delivery state a socket connection already is. The costs: -- **Traffic.** O(Σ over clients of the moved keys of that client's shape) per batch. It - saves nothing when every client reads one shape, and much when shapes differ. It is - still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** It is per-client state, but it holds no UI - state, and it can be rebuilt by the next request, which names its shape. The author - reads it as delivery state, not UI state, but the user has not said so. +its own shape key. The host keeps O(1) per client: a topic subscription, which names a +shape key and holds no window, cursor or view. **A topic filters only the moved-key +lists; every batch's header goes to everyone** (found by the review of `e152b667`): +- **The header goes to every topic, and to every topic-less client, for every accepted + batch.** It holds the step stamp, `document_generation`, the axis changes, the display + relabels, the formula and `measure_agg` flags, and the eviction list (below). None of + these is per shape: an item insert shifts every shape with that category on an axis. + So a filter must not withhold them. Headers are also what keep T3's loss detection + working. Every client sees every stamp, so a gap is a real loss and never a quiet + batch. That costs O(N) small headers per batch. +- **The moved-key lists are filtered by shape.** A client receives only its own shape's + list. +- **Switching shape key.** A client that changes its request's shape fetches under the + new key, which returns the stamp n the fetch was computed at. It subscribes to the new + topic from n on and drops the old topic. Headers never stopped arriving, so the switch + leaves no gap. +- **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a + shape once no recent request names it. A client still subscribed to that topic would + then receive no moved keys, which looks exactly like "nothing changed", and its window + would go stale. *The author's lean* combines two invariants: + - **(i) a live topic subscription pins its shape.** Subscriptions are leases, renewed + by the client's requests or a heartbeat. Host state grows with the live + subscriptions, and a client that vanishes is released when its lease expires. + - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under + memory pressure (clause 14), puts the shape key in the next header's **eviction + list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + + The alternative, **(iii)**: every header states, for each shape it carries keys for, + that the shape was evaluated, and a subscribed shape missing from that statement means + a conservative refetch. It needs no leases, and it costs a list of evaluated shapes + per header. +- **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) + per batch. It saves nothing when every client reads one shape, and much when shapes + differ. It is still not restricted to the window; only (c) does that. +- **"As stateless as possible" (C2) and Q33.** A subscription is per-client state, with + a lease. It holds no UI state, and the next request, which names its shape, rebuilds it. + The author reads it as delivery state, not UI state. **But the user said "The backend + should only hold model-relevant information"** (ADR 008 Q33), and whether a topic + subscription counts as that is the user's question, put in question 13. - **Network transparency.** The topic is a filter applied when sending, so in-process and remote consumers still read the same notices (T6). The TUI simply subscribes to its own shape. @@ -1827,8 +1864,9 @@ key per client. 22's index current. A client learns of a new or vanished row from the notice's records, and re-asks. -A shape is kept while recent requests name it. Eviction drops its operators, and the next -request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has +A shape is kept while recent requests name it or a live topic subscription pins it. +Eviction drops its operators, is announced in the next header's eviction list, and the +next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has not set (question 14). **What a client must hold to resume:** its request, the last step it applied, its window, @@ -1921,7 +1959,7 @@ orders come from the clauses and are not measured. - **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a range, and apply a delta. The first implementation keeps it in memory, and the store-memory ADR can page or compress it without changing any operator. -- **Shapes are evicted** (clause 12). So per-shape state is bounded by the shapes in +- **Shapes are evicted** unless a live topic subscription pins them (clause 12). So per-shape state is bounded by the shapes in recent use, not by every saved view. - **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and @@ -2185,8 +2223,14 @@ revisions, and none is renumbered. changes, display relabels, a formula or `measure_agg` flag, and the moved output keys per live shape key. The alternative is moved records plus a conservative rule for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - - (b′) the same notices, delivered by topic: one topic per shape key, with O(1) - per-client state that holds no UI state; + - (b′) the same notices, delivered by topic. Every batch's header (stamp, + `document_generation`, axis changes, relabels, flags, eviction list) goes to every + client, and only the moved-key lists are filtered by shape key. A topic subscription + is a lease that pins its shape, and any eviction is announced in the header. It costs + O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question + for the user:** is a topic subscription "model-relevant information", in the sense + of your answer to ADR 008's Q33 ("The backend should only hold model-relevant + information")? (b′) depends on the answer; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2204,7 +2248,10 @@ revisions, and none is renumbered. applied at read time (the author's)? Or is it kept per saved `.improv` view only, so that a request with an unsaved key is served without incremental state? *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while. The time is an implementation number. An unsaved collapse, reorder, page + a while **and no live topic subscription pins it** (question 13's (b′)). An eviction + is announced in the next notice header, so a subscriber refetches and is never left + with a window that looks current and is not. The time and the lease length are + implementation numbers. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved hide or show of a row-axis item shares the key too. Under question 23's default (A) its totals subtract the hidden rows at read time and add nothing to the cache. Under From 70ababfe0c6c0f5616ba5103df586e70f9224cd4 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:19:46 -0700 Subject: [PATCH 36/67] docs(adr): ADR 009 adds a host instance to the stamp and covers plain (b) by the eviction header (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 ++++++++++++++++------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e02328c1..286ddabc 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1669,16 +1669,20 @@ occupied answers "empty axis". Both need the predecessor query. (every NaN is one value, and every other number is its bits) before they leave the host. The record tier keeps bit identity, as it does today. Question 15. - **T3. Every output change is stamped.** A delta carries the step that computed it, as - `(document_generation, step)`. The circuit steps once per model effect (clause 3), and - ADR 001 already puts batches in one total order. A consumer that has applied step n - drops anything at or below n, so a duplicate does no harm, and it treats a gap as a + `(host instance, document_generation, step)`. The **host instance** is an id chosen + when the host starts. The other two counters live in memory and start again at their + initial values after a restart, so without the instance id a reconnected client would + drop the new host's notices as duplicates. The circuit steps once per model effect + (clause 3), and ADR 001 already puts batches in one total order. A consumer that has + applied step n of the same instance and generation drops anything at or below n, so a + duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot answers. Under question 13's (b′), every client receives every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. That covers a host restart: the stamp - sequence starts again, which is a discontinuity, and a client holding a cached window - across it refetches. The same rule holds for a client under (a), (b) and (c). What is published is the **batch's** consolidated + notice it cannot interpret, **refetches**. A changed host instance is such a + discontinuity, so a reconnect after a restart always refetches. The same rule holds for + a client under (a), (b) and (c). What is published is the **batch's** consolidated delta, and only once the batch is accepted. A refused frame is restored (clause 11) and publishes nothing. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the @@ -1800,8 +1804,9 @@ its own shape key. The host keeps O(1) per client: a topic subscription, which n shape key and holds no window, cursor or view. **A topic filters only the moved-key lists; every batch's header goes to everyone** (found by the review of `e152b667`): - **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the step stamp, `document_generation`, the axis changes, the display - relabels, the formula and `measure_agg` flags, and the eviction list (below). None of + batch.** It holds the step stamp (with the host instance, T3), `document_generation`, + the axis changes, the display relabels, the formula and `measure_agg` flags, and the + eviction list (below). None of these is per shape: an item insert shifts every shape with that category on an axis. So a filter must not withhold them. Headers are also what keep T3's loss detection working. Every client sees every stamp, so a gap is a real loss and never a quiet @@ -1823,6 +1828,15 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists + moved keys only for live shape keys, and a topic-less client pins nothing. So its shape + can be evicted silently. The header's eviction list reaches it too, and it refetches. + Only the "records plus a conservative rule" alternative of (b) escapes the hole, because + it lists no shape at all. Instead of an eviction list, a header could carry the set of + live shape keys, or a shape-cache generation that moves on every eviction. A client + whose key is missing from the live set would then re-request. Both say the same thing, + and the eviction list is smaller. + The alternative, **(iii)**: every header states, for each shape it carries keys for, that the shape was evaluated, and a subscribed shape missing from that statement means a conservative refetch. It needs no leases, and it costs a list of evaluated shapes @@ -2225,7 +2239,8 @@ revisions, and none is renumbered. for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - (b′) the same notices, delivered by topic. Every batch's header (stamp, `document_generation`, axis changes, relabels, flags, eviction list) goes to every - client, and only the moved-key lists are filtered by shape key. A topic subscription + client, with the host instance in its stamp. Only the moved-key lists are filtered + by shape key. A topic subscription is a lease that pins its shape, and any eviction is announced in the header. It costs O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense From 85dc0ad8ba8c789182995ec3bef0ef2e6f319e8c Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:23:30 -0700 Subject: [PATCH 37/67] docs(adr): ADR 009 registers subscriptions atomically and stamps batches by a publication number (improvise-felo.4) B9: a fetch registers its topic in the same step, so no batch can fall between them; ack-and-recheck is the alternative; refetch-storm options and the lease heartbeat's cost are stated. Codex on 70ababfe: the stamp counts published batches, not per-effect circuit steps, so a multi-effect batch is one header; required tests added to step 8. Reads ADR 008 at 1c974bf7. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 121 +++++++++++++++------- 1 file changed, 81 insertions(+), 40 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 286ddabc..29ffee43 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 34** -(PR #308, read at `385784e7`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 35** +(PR #308, read at `1c974bf7`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `385784e7`, its revision 34) · +reads it at `1c974bf7`, its revision 35) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 34 of `context/adr/008-virtualized-grid.md` on PR #308, at `385784e7`. Clauses 3, +revision 35 of `context/adr/008-virtualized-grid.md` on PR #308, at `1c974bf7`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 34, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 35, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -432,8 +432,9 @@ An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to th Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation -under the view/host split. The per-effect step stays, because each step is what clause -12 stamps. +under the view/host split. The per-effect step stays as the circuit's own clock. What a +client counts is clause 12's publication number, which moves once per published batch +(T3). **A delta key's identity is bit identity, the same identity the record tier uses today.** Two elements have the same `(key, value)` part exactly when their keys are @@ -1289,7 +1290,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 34 at `385784e7`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 35 at `1c974bf7`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1668,33 +1669,48 @@ occupied answers "empty axis". Both need the predecessor query. round trip. **Proposed:** output streams are consolidated under the wire's identity (every NaN is one value, and every other number is its bits) before they leave the host. The record tier keeps bit identity, as it does today. Question 15. -- **T3. Every output change is stamped.** A delta carries the step that computed it, as - `(host instance, document_generation, step)`. The **host instance** is an id chosen - when the host starts. The other two counters live in memory and start again at their - initial values after a restart, so without the instance id a reconnected client would - drop the new host's notices as duplicates. The circuit steps once per model effect - (clause 3), and ADR 001 already puts batches in one total order. A consumer that has - applied step n of the same instance and generation drops anything at or below n, so a - duplicate does no harm, and it treats a gap as a - loss, which a fresh snapshot answers. Under question 13's (b′), every client receives - every batch's **header** (clause 12), so the stamps it sees are contiguous and a gap is - always a real loss. **The general rule:** a client that misses a notice, sees a gap, sees - `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. A changed host instance is such a - discontinuity, so a reconnect after a restart always refetches. The same rule holds for - a client under (a), (b) and (c). What is published is the **batch's** consolidated - delta, and only once the batch is accepted. A refused frame is restored (clause 11) and - publishes nothing. +- **T3. Every published batch is stamped, by a publication sequence.** The stamp is + `(host instance, document_generation, publication)`. + - **The publication number** increments once per **published batch**, and that is + what a client counts. The circuit's internal `step` advances once per model effect + (clause 3), and it is never a client-visible gap detector. *As first written* + (superseded), the stamp carried `step`. A batch of two model effects then advanced + from n to n+2 under one header, and a client read the missing n+1 as a loss (found by + ChatGPT Codex on `70ababfe`). *The author's lean* is this separate sequence (i). The + alternatives: (ii) a header per model-effect step, which multiplies headers and + exposes a batch's internal steps, which ADR 001 treats as one atomic unit; or (iii) a + header that carries the previous published stamp, so a client can tell a + consolidated range from a loss, at a little more per header. + - **The host instance** is an id chosen when the host starts. The other two counters + live in memory and start again at their initial values after a restart. Without the + instance id, a reconnected client would drop the new host's notices as duplicates. + - ADR 001 already puts batches in one total order. A consumer that has applied + publication n, of the same instance and generation, drops anything at or below n, so + a duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot + answers. + - Under question 13's (b′), every client receives every batch's **header** (clause 12), + so the publication numbers it sees are contiguous, and a gap is always a real loss. + Contiguous numbers prove only that no header was lost, not that a key list was routed + to the client. That holds only because a subscription is registered atomically with + its fetch (clause 12). + - **The general rule:** a client that misses a notice, sees a gap, sees + `document_generation` change, finds its shape in an eviction list, or receives a + notice it cannot interpret, **refetches**. A changed host instance is such a + discontinuity, so a reconnect after a restart always refetches. The same rule holds + for a client under (a), (b) and (c). + - What is published is the **batch's** consolidated delta, and only once the batch is + accepted. A refused frame is restored (clause 11), publishes nothing, and consumes no + publication number. - **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at - step n: the window as a Z-set of current values, each with weight +1. Deltas from step - n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current + publication n: the window as a Z-set of current values, each with weight +1. Deltas + from publication n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current result. `GridCache::folded(&cache.seed()) == *cache` states the same property for today's bootstrap compaction (gotcha 32). - **T5. Order holds across streams, and coalescing happens within a stream** (as stated, for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client merges pending notices into one, taking the union of their moved keys and flags and the - latest stamp, and refetches once. **Axis changes do not merge.** They are positional, + latest publication number, and refetches once. **Axis changes do not merge.** They are positional, so they are kept as a list and applied in their original order. A `document_generation` change inside the merged run voids the merge, because the old keys belong to a document that is gone, and the client refetches from scratch). A slow @@ -1708,9 +1724,10 @@ occupied answers "empty axis". Both need the predecessor query. is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple identity, or by a stable cross-product position, is not positional in that sense. **Coalescing is therefore per subscription, never per stream.** All of a - subscription's pending deltas, the basis included, are summed together up to one step, + subscription's pending deltas, the basis included, are summed together up to one + publication number, and sent as one batch in the fixed order. Coalescing one stream while another is sent as - it arrives would let a basis for step n+2 overtake the cells it describes, which is + it arrives would let a basis for publication n+2 overtake the cells it describes, which is gotcha 43's failure. - **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives the same result and delta types as a remote client. Its transport is a function call. @@ -1739,7 +1756,7 @@ options, in ADR 008's lettering: stays invisible until then. - **(b) An unaddressed change notice** (Flux style). After each accepted batch, the model emits one notice to every consumer, addressed to none. It carries: - - the step stamp `(document_generation, step)` (T3); + - the stamp `(host instance, document_generation, publication)` (T3); - each item edit, as an axis change (above); - a flag when the formula set or `measure_agg` moved. Either may move every aggregate without moving a record. `measure_agg` has no wire command today, and changes only @@ -1804,7 +1821,7 @@ its own shape key. The host keeps O(1) per client: a topic subscription, which n shape key and holds no window, cursor or view. **A topic filters only the moved-key lists; every batch's header goes to everyone** (found by the review of `e152b667`): - **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the step stamp (with the host instance, T3), `document_generation`, + batch.** It holds the stamp (host instance, `document_generation`, publication; T3), the axis changes, the display relabels, the formula and `measure_agg` flags, and the eviction list (below). None of these is per shape: an item insert shifts every shape with that category on an axis. @@ -1813,10 +1830,19 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 batch. That costs O(N) small headers per batch. - **The moved-key lists are filtered by shape.** A client receives only its own shape's list. -- **Switching shape key.** A client that changes its request's shape fetches under the - new key, which returns the stamp n the fetch was computed at. It subscribes to the new - topic from n on and drops the old topic. Headers never stopped arriving, so the switch - leaves no gap. +- **Subscribing, first or after a switch of shape key.** *As first written* + (superseded), this said a client fetches at publication n and then subscribes from n, and + that the switch "leaves no gap". **That was wrong** (found by the review of `70ababfe`). + The host keeps no past moved keys. So if batch n+1 moves a key in the new shape before + the subscription is registered, the client gets header n+1, with contiguous stamps, + but not n+1's key list, and its window goes stale silently. A client's first + subscription has the same race. *The author's lean:* **(i) the fetch registers the + subscription atomically.** The request names its topic, and the host registers the + lease in the same step that computes the fetch at n. Every later batch then routes to + it, and this fits "leases renewed by requests". The alternative, **(ii)**: the + subscription is acknowledged with the current publication number m, and the client refetches if + m > n. That costs a second round trip whenever a batch lands in between. The client + drops the old topic once the new fetch is in. - **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a shape once no recent request names it. A client still subscribed to that topic would then receive no moved keys, which looks exactly like "nothing changed", and its window @@ -1827,6 +1853,12 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + Evicting a popular shape therefore causes a **refetch storm**. *The author's + options*, unmeasured: refuse to evict a shape with many live subscriptions + (hysteresis), stagger the refetches with a per-client jitter, or rate-limit + rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list + names only the shapes evicted since the previous header, so it does not grow without + bound. **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists moved keys only for live shape keys, and a topic-less client pins nothing. So its shape @@ -2060,9 +2092,15 @@ displayed path until the differential test holds. question 3's (c) for that subtraction to be exact. 7. **Live sources as delta streams**, when ADR 003's source trait exists. 8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to - 22. It brings the output types in `improvise-protocol`, the step stamp, + 22. It brings the output types in `improvise-protocol`, the publication stamp (T3), snapshot-then-delta, question 22's restricted successor query, and the test with two - consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff + consumers (T6). **Required tests:** + - a multi-effect batch such as `[AddItem, SetCell]` publishes exactly one header, and + no client refetches; + - a dropped header **is** detected as a gap, and the client refetches; + - a batch that lands between a fetch and its subscription reaches the client's key + list (question 13's atomic registration); + - a reconnect after a host restart refetches, and drops nothing as a duplicate. It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -2233,7 +2271,7 @@ revisions, and none is renumbered. question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: this ADR.** The options are: - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the step stamp, axis + - (b) an unaddressed change notice per accepted batch, carrying the publication stamp, axis changes, display relabels, a formula or `measure_agg` flag, and the moved output keys per live shape key. The alternative is moved records plus a conservative rule for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); @@ -2242,7 +2280,10 @@ revisions, and none is renumbered. client, with the host instance in its stamp. Only the moved-key lists are filtered by shape key. A topic subscription is a lease that pins its shape, and any eviction is announced in the header. It costs - O(N) headers per batch and O(1) per-client state, which holds no UI state. **Question + O(N) headers per batch, O(1) per-client state (which holds no UI state), and a + lease heartbeat, which is a second small message stream per client, between + requests. A subscription is registered atomically with the fetch that names its + topic, so no batch can fall between them. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant information")? (b′) depends on the answer; From 6ad3643790f4da64fbae06c325c816fd24323950 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Mon, 5 Oct 2026 23:27:19 -0700 Subject: [PATCH 38/67] docs(adr): ADR 009 ties topic subscriptions to connections and leans to plain (b) with a size cap (improvise-felo.4) B10: a lease could lapse silently while another client pinned the shape; a subscription now lives exactly as long as its connection, which drops the heartbeat. Q13 compares the forms in complexity as well as traffic, and the author's lean returns to plain (b) with a size cap, with (b') as a measured optimisation. Client-visible step uses now say publication; two more required tests. Reads ADR 008 at a25fea81. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 112 ++++++++++++++-------- 1 file changed, 71 insertions(+), 41 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 29ffee43..48421fcc 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,8 +1,8 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 35** -(PR #308, read at `1c974bf7`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 36** +(PR #308, read at `a25fea81`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of @@ -13,7 +13,7 @@ comments are Copilot's two and our replies. · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · **Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `1c974bf7`, its revision 35) · +reads it at `a25fea81`, its revision 36) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 35 of `context/adr/008-virtualized-grid.md` on PR #308, at `1c974bf7`. Clauses 3, +revision 36 of `context/adr/008-virtualized-grid.md` on PR #308, at `a25fea81`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 35, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 36, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -1290,7 +1290,7 @@ option, and its recommendation. ### 9. What ADR 008 gets from it -ADR 008 (revision 35 at `1c974bf7`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 36 at `a25fea81`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1584,7 +1584,7 @@ search appears (next paragraph). What forward-only growth simplifies: and each step is one restricted successor query, so the host keeps no fill state at all. Under model-side pruning, `project` itself would scan forward from the top-left and send rows and columns as it finds them, as a sliced job resumable from `(request, fill -front)`. Either way, changes during a fill are identity-keyed deltas stamped by step +front)`. Either way, changes during a fill are identity-keyed deltas stamped by publication number (T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an ordinary delta with its absolute position, and the client keeps or drops the extra row by its own policy (Q42). So a fill and a live update are one stream, which keeps the @@ -1795,7 +1795,7 @@ options, in ADR 008's lettering: 22's successor query. A formula or `measure_agg` change means refetching the window. The host keeps no per-client state. - **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask - to be kept current. The host keeps the request and the last step it sent, then pushes + to be kept current. The host keeps the request and the last publication number it sent, then pushes the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost table costs a snapshot). But it is per-client state on the backend, and it holds a window, which is a client's view parameter. @@ -1811,7 +1811,7 @@ corrected after the review of `5a8dc486`; nothing here is measured). streaming into a large table, every remote client would receive every moved key, most of them for shapes it does not read. - **(c) keeps per-client state, and saves traffic.** It holds the request and the last - step per subscriber, which is a client's window and so a view parameter on the host, + publication number per subscriber, which is a client's window and so a view parameter on the host, against Q33. In return it sends each client only its window's deltas. - *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. @@ -1838,8 +1838,8 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 but not n+1's key list, and its window goes stale silently. A client's first subscription has the same race. *The author's lean:* **(i) the fetch registers the subscription atomically.** The request names its topic, and the host registers the - lease in the same step that computes the fetch at n. Every later batch then routes to - it, and this fits "leases renewed by requests". The alternative, **(ii)**: the + subscription in the same step that computes the fetch at n. Every later batch then + routes to it. The alternative, **(ii)**: the subscription is acknowledged with the current publication number m, and the client refetches if m > n. That costs a second round trip whenever a batch lands in between. The client drops the old topic once the new fetch is in. @@ -1847,12 +1847,22 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 shape once no recent request names it. A client still subscribed to that topic would then receive no moved keys, which looks exactly like "nothing changed", and its window would go stale. *The author's lean* combines two invariants: - - **(i) a live topic subscription pins its shape.** Subscriptions are leases, renewed - by the client's requests or a heartbeat. Host state grows with the live - subscriptions, and a client that vanishes is released when its lease expires. - - **(ii) every eviction is announced.** A lease expiry, or a forced eviction under + - **(i) a live topic subscription pins its shape, and a subscription lives exactly as + long as its connection** (revised after the review of `85dc0ad8`). *As first + written* (superseded), subscriptions were leases renewed by requests or a + heartbeat. That let a client's own lease lapse silently while another client still + pinned the shape. The shape was then not evicted and nothing was announced, so the + client kept receiving contiguous headers but no key lists. Tying the subscription to + the connection (the socket, the worker's port, or the TUI's in-process call) removes + the heartbeat and the lapse together. A live connection keeps its subscriptions. A + dropped connection drops them. A client always knows when its connection dropped, + and it refetches on reconnect (T3's general rule). *The author's lean*, because it is + the simplest of the three. The alternatives were a per-client subscription epoch in + that client's header, or acknowledged renewals with a self-unsubscribe on a missed + acknowledgement. Each adds a mechanism. + - **(ii) every eviction is announced.** Any eviction, by ageing out or under memory pressure (clause 14), puts the shape key in the next header's **eviction - list**, and every subscriber to it refetches. The refetch rebuilds the shape, as a job. + list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. Evicting a popular shape therefore causes a **refetch storm**. *The author's options*, unmeasured: refuse to evict a shape with many live subscriptions (hysteresis), stagger the refetches with a per-client jitter, or rate-limit @@ -1871,13 +1881,13 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 The alternative, **(iii)**: every header states, for each shape it carries keys for, that the shape was evaluated, and a subscribed shape missing from that statement means - a conservative refetch. It needs no leases, and it costs a list of evaluated shapes + a conservative refetch. It needs no pins, and it costs a list of evaluated shapes per header. - **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) per batch. It saves nothing when every client reads one shape, and much when shapes differ. It is still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** A subscription is per-client state, with - a lease. It holds no UI state, and the next request, which names its shape, rebuilds it. +- **"As stateless as possible" (C2) and Q33.** A subscription is per-connection state. + It holds no UI state, and the next request, which names its shape, rebuilds it. The author reads it as delivery state, not UI state. **But the user said "The backend should only hold model-relevant information"** (ADR 008 Q33), and whether a topic subscription counts as that is the user's question, put in question 13. @@ -1885,11 +1895,31 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 and remote consumers still read the same notices (T6). The TUI simply subscribes to its own shape. -*The author's recommendation, revised:* **(b′)**. It keeps the host free of UI state, -while cutting (b)'s traffic to the shapes each client reads. A client that names no -topic receives the full broadcast, which is (b). (c) stays an optimisation for a client -whose window is a small part of a busy shape, if measured traffic justifies holding a -window on the host. All of this is unmeasured. +**What each form costs in complexity, not only in traffic** (the author's; nothing +measured). The mechanism (b′) needed has grown review by review: topics, pins, +atomic registration, an eviction list, and a lease (now a connection tie). That growth +is itself a signal. +- **Plain (b): moved keys per live shape, broadcast to every client.** It keeps no + per-client state and needs no topics, pins or registration. It still needs the eviction + list, because a client's shape can be evicted. Its traffic is O(N × all moved keys) per + batch. **With a size cap**, a header whose key lists exceed a fixed size carries "too + many changes, refetch" instead of the lists. That bounds every notice, at the price of a + refetch by every client in a busy batch. +- **The records-only variant of (b).** It sends moved records and a formula flag, and + windows with cross-stem formulas refetch on every batch. It needs no shape lists, so it + needs no eviction list either. It is the simplest form, and its notices are the + smallest. It refetches the most. +- **(b′): (b) filtered by connection-tied topics.** It cuts traffic to the shapes each + client reads. The price is per-connection state, which is up against Q33, plus atomic + registration at fetch and pins on the shape cache. + +*The author's recommendation, revised again:* **plain (b) with a size cap**, keeping no +per-client state, with **(b′)** as the optimisation to adopt only if traffic is measured to +matter with many clients on different shapes. (c) stays the further optimisation for a +small window on a busy shape. (b′) is defensible against (b) with a cap only where +clients read many different shapes of one busy document, and nothing here shows that +case is common. The user may well prefer the simpler design. *As first written* +(superseded), the lean was (b′). *Earlier still* (superseded), this clause leaned to (c), which it then called (b). That was before the user said the backend holds only model-relevant information. Under (b′), @@ -1915,7 +1945,7 @@ Eviction drops its operators, is announced in the next header's eviction list, a next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has not set (question 14). -**What a client must hold to resume:** its request, the last step it applied, its window, +**What a client must hold to resume:** its request, the last publication number it applied, its window, and its basis. The host holds nothing that a client needs back. **The basis.** ADR 001's basis is answered with the result it describes, and after it @@ -2100,7 +2130,10 @@ displayed path until the differential test holds. - a dropped header **is** detected as a gap, and the client refetches; - a batch that lands between a fetch and its subscription reaches the client's key list (question 13's atomic registration); - - a reconnect after a host restart refetches, and drops nothing as a duplicate. It depends on step 4 for occupancy. It replaces the per-mirror diff + - a reconnect after a host restart refetches, and drops nothing as a duplicate; + - a refused frame consumes no publication number; + - under (b′), a dropped connection drops its subscriptions, and the client refetches on + reconnect, even while another client still pins the shape. It depends on step 4 for occupancy. It replaces the per-mirror diff only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). @@ -2278,25 +2311,23 @@ revisions, and none is renumbered. - (b′) the same notices, delivered by topic. Every batch's header (stamp, `document_generation`, axis changes, relabels, flags, eviction list) goes to every client, with the host instance in its stamp. Only the moved-key lists are filtered - by shape key. A topic subscription - is a lease that pins its shape, and any eviction is announced in the header. It costs - O(N) headers per batch, O(1) per-client state (which holds no UI state), and a - lease heartbeat, which is a second small message stream per client, between - requests. A subscription is registered atomically with the fetch that names its - topic, so no batch can fall between them. **Question + by shape key. A topic subscription lives exactly as long as its connection, and pins + its shape. It is registered atomically with the fetch that names its topic. It costs + O(N) headers per batch and per-connection state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant information")? (b′) depends on the answer; - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. - *Recommendation (the author's, revised):* (b′). It holds no UI state on the host and - cuts (b)'s traffic to the shapes each client reads. (b) is what a client that names no - topic gets. (c) is an optimisation if measured traffic justifies holding a window. - Under all three, `GridExtent` is derived by the client (in records mode it is - fetched), `Dirty` is retired with the mirrors, and the occupancy delta stays - model-side, feeding question 22's index. *As first written*, this recommended plain - (b); the trade it skipped is wire traffic. + *Recommendation (the author's, revised again):* **plain (b) with a size cap**, which + keeps no per-client state, with (b′) as a measured optimisation and (c) beyond it. The + simpler choice is the user's to prefer. (b′) buys traffic savings at the price of + connection-tied topics, pins, atomic registration and per-connection state against + Q33. The records-only variant of (b) is simpler still, and refetches more. Under every + option, `GridExtent` is derived by the client (in records mode it is fetched), `Dirty` + is retired with the mirrors, and the occupancy delta stays model-side, feeding + question 22's index. *Earlier* (superseded): (b), then (b′). 14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state selected by a request's **shape key** (axes by level, None categories, filter keys, @@ -2306,8 +2337,7 @@ revisions, and none is renumbered. *Recommendation:* per shape key, cached, and evicted once no request has named it for a while **and no live topic subscription pins it** (question 13's (b′)). An eviction is announced in the next notice header, so a subscriber refetches and is never left - with a window that looks current and is not. The time and the lease length are - implementation numbers. An unsaved collapse, reorder, page + with a window that looks current and is not. The time is an implementation number. An unsaved collapse, reorder, page step or prune then shares the saved view's key and builds no operator. An unsaved hide or show of a row-axis item shares the key too. Under question 23's default (A) its totals subtract the hidden rows at read time and add nothing to the cache. Under From e6a52790554398307fe2aa4e33354579d342833f Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:36:41 -0700 Subject: [PATCH 39/67] docs(adr): ADR 009 folds the user's answers to questions 1-10 and 12 (improvise-felo.4) Adds "What the user decided" with the eleven comments of review 5465446805, verbatim. zset, the seam crate and the exact accumulator are decided; Eager is deprecated and kept for now; the cycle bound and component fixed point stay for testing with a DBSP-native plan; the rollback snapshot stands; felo.4 records the ADR only and a new epic takes the steps; clause 13 becomes the decided direction for string-value columns. New questions 25-27 hold the residue. Also: dead- connection detection and the idle-reader eviction storm. Reads ADR 008 at c8be7720; merges gh/main f0fb6880. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 249 +++++++++++++++++----- 1 file changed, 198 insertions(+), 51 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 48421fcc..05e14ef8 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,23 +1,23 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 36** -(PR #308, read at `a25fea81`, which the user is still reviewing). The revision brings in +are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and +12** (see "What the user decided"). Question 11 and questions 13 to 27 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** +(PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 24. **The user has answered none of -this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline -comments are Copilot's two and our replies. · +ADR", in clauses 12 to 14, and in questions 12 to 27. *Superseded:* until 2026-10-09 the +user had answered none of questions 1 to 11. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · **Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the +**Relates to:** ADR 005 (exact decimals, improvise-l2b8; PR #320 merged on 2026-10-09, and its own status line still reads proposed), ADR 008 (the virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `a25fea81`, its revision 36) · +reads it at `c8be7720`, its revision 37) · **Governed by:** `context/design-principles.md` §3 and §5 **Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 -is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and +is on main since PR #320 merged (2026-10-09), and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and no open PR claimed it when this was written. **Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are @@ -26,7 +26,7 @@ revision's new citations are at `gh/main` `50e1314d`**, and each says so. The ol citations were not re-derived, and some have moved. For example, `Host::mutating_frame` takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and `App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 36 of `context/adr/008-virtualized-grid.md` on PR #308, at `a25fea81`. Clauses 3, +revision 37 of `context/adr/008-virtualized-grid.md` on PR #308, at `c8be7720`. Clauses 3, 5 and 8, which this ADR cites most, have the same numbers there as in revision 3. ## The decision this ADR writes down @@ -78,7 +78,7 @@ may be in another process, and clause 12 says how the same circuit serves both. ## What ADR 008's review established for this ADR -At its revisions 18 to 36, ADR 008 (PR #308) became the ADR of the **view/host split**, and the +At its revisions 18 to 37, ADR 008 (PR #308) became the ADR of the **view/host split**, and the user routed part of it here. This section quotes what the user said that bears on this ADR, and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers @@ -320,7 +320,7 @@ the other. Both stay in the tree, and a differential test holds them equal.** Fo clauses, plus clause 5a. It is numbered 5a so that no clause number or question number cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. -### 1. The crate: `zset` (name is question 1) +### 1. The crate: `zset` (the user's answer to question 1) Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond `std`, so it is wasm32-clean by construction. @@ -403,8 +403,13 @@ itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental improvise-core never depends on `zset`, and it can be built and tested eager-only (question 2). -**Why keep both.** `Eager` is the oracle for the differential test. It is also the -backend a query-only source keeps (clause 10). The differential test is a property test +**Why keep both, for now.** `Eager` is the oracle for the differential test. It was +also to be the backend a query-only source keeps (clause 10). **The user: "Eager should +be deprecated", and "For now: the goal is to only have DBSP formulas"** (questions 8 and +6). So `Eager` stays only as long as the differential test needs an oracle, and what +replaces it for query-only sources and for clause 5a's non-record-local formulas is +question 26. *As first written* (superseded), this paragraph and question 6 kept `Eager` +permanently. The differential test is a property test in the seam crate. It generates documents with groups, collapse, hidden items, formulas (including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text cells and non-finite numbers, applies the same random edit sequence to both backends, @@ -814,6 +819,14 @@ occupied-contexts fallback above as well is question 18. ### 6. Circular references: DBSP's recursion is not our semantics +**The user, on questions 4 and 5:** the sixteen-pass bound is kept "for testing purposes +but eventually remove it because DBSP should fix the cycle issue", and the component +fixed point "For validation, but if there's a more natural DBSP-native way we should plan +on that." So what follows is the **current** plan, kept for testing and validation, and a +DBSP-native treatment of cycles is to be planned. The argument below, that DBSP's least +fixed point is not improvise's circular semantics, is the author's. It is the open part, +question 25. + DBSP handles recursion by computing a **least fixed point**: nested streams iterate a monotone operator, and `distinct` keeps the lattice finite, so reachability converges. improvise has no such guarantee: @@ -1285,12 +1298,12 @@ every magnitude. **So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting -for ADR 005**, provided `Eager` uses the same accumulator. That is question 3's new -option, and its recommendation. +for ADR 005**, provided `Eager` uses the same accumulator. That was question 3's option +(c), and **the user chose it**: "use the exact accumulator." (4226373188). ### 9. What ADR 008 gets from it -ADR 008 (revision 36 at `a25fea81`, PR #308) assumes these, under the shared names below. +ADR 008 (revision 37 at `c8be7720`, PR #308) assumes these, under the shared names below. **Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. A consumer reads it through `project` (clause 12), and none of it is addressed to a host-held mirror of a client. @@ -1359,8 +1372,9 @@ change feed, produces a cell delta. That is the same input `DataStore` produces, circuit cannot tell them apart. That is the "data can stream" half of the user's prompt. -**A source that can only answer aggregate queries stays on `Eager`,** with ADR 003's -pushdown. There is a tension: ADR 003 wants data larger than memory, and every +**A source that can only answer aggregate queries stays on `Eager` for now,** with ADR +003's pushdown. The user has said "Eager should be deprecated" (question 8), so this is +an interim, and its replacement is question 26. There is a tension: ADR 003 wants data larger than memory, and every integrated operator here keeps state proportional to its input (clause 4). So the backend is chosen per document, which is what "pluggable" buys (question 8). A read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. @@ -1384,7 +1398,7 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n compares them. One circuit compared with itself proves nothing (improvise-hzjo's vacuity). - **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:693` at `50e1314d`). **Recommended (question 9): the backend's state joins the snapshot.** + (`app.rs:693` at `50e1314d`). **Decided (question 9, the user: "The recommendation is fine, I think"): the backend's state joins the snapshot.** Applying the inverse delta is **not** a rollback, even with exact aggregators. It restores integrated values, but `step(d)` then `step(−d)` still advances the circuit and changes the history that `delay` and `differentiate` hold. So the next step after @@ -1855,7 +1869,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 client kept receiving contiguous headers but no key lists. Tying the subscription to the connection (the socket, the worker's port, or the TUI's in-process call) removes the heartbeat and the lapse together. A live connection keeps its subscriptions. A - dropped connection drops them. A client always knows when its connection dropped, + dropped connection drops them. **The transport must detect a dead connection**: a + websocket ping with a timeout on the socket server, a closed port for the worker, and + nothing for the TUI, which is in process. A half-open socket that is never detected + would hold its pins forever. A client always knows when its connection dropped, and it refetches on reconnect (T3's general rule). *The author's lean*, because it is the simplest of the three. The alternatives were a per-client subscription epoch in that client's header, or acknowledged renewals with a self-unsubscribe on a missed @@ -1863,7 +1880,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 - **(ii) every eviction is announced.** Any eviction, by ageing out or under memory pressure (clause 14), puts the shape key in the next header's **eviction list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. - Evicting a popular shape therefore causes a **refetch storm**. *The author's + Evicting a popular shape therefore causes a **refetch storm**. **Idle readers make + it likely under plain (b):** a shape that clients only watch, without requesting, + ages out, and every one of them refetches at once. That refetch is also what brings + the shape back. *The author's options*, unmeasured: refuse to evict a shape with many live subscriptions (hysteresis), stagger the refetches with a per-client jitter, or rate-limit rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list @@ -1921,6 +1941,13 @@ clients read many different shapes of one busy document, and nothing here shows case is common. The user may well prefer the simpler design. *As first written* (superseded), the lean was (b′). +**The user's answer to question 7 bears on this, and does not decide it.** "As much state +as possible should be maintained in the backend, unless it's UI-specific state", and "we +probably need to have a mix of caching and a pull model to expand the cached data". Plain +(b) keeps the maintained model state in the backend and has clients pull on a notice, +which fits both sentences. Whether a per-connection topic is backend state of the kind +the user means, or UI-specific, is still question 13's question to the user. + *Earlier still* (superseded), this clause leaned to (c), which it then called (b). That was before the user said the backend holds only model-relevant information. Under (b′), the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is @@ -1954,12 +1981,26 @@ and its basis. The host holds nothing that a client needs back. cursor. Under the user's Q39 answer, only `document_generation` and the record tier are model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). -### 13. Incremental max width (the user's question on ADR 008) +### 13. Incremental max width: the host maintains it for string-value columns The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" -**This clause is the author's analysis, and none of it is measured.** The user's decision -on widths stands as ADR 008 records it: (b), sticky per session, eased by hysteresis, -taken over the rows realized and held by the view layer (ADR 008 clause 4). +**The user then decided** (question 12, 4226624890): "Yes, I think the host should +maintain the max width for string-value columns incrementally." So **the host maintains +each string-value column's maximum display width incrementally**. This clause is now +that direction. Its cost analysis is the author's, and none of it is measured. + +**Scope.** "String-value columns" is read, by the author, as columns whose cells hold +text values (`CellValue::Text`), as opposed to numbers formatted for display. Numbers' +widths depend on the number format, which is display-only, so they stay with ADR 008's +rule: (b), sticky per session, eased by hysteresis, over the rows realized, held by the +view layer (ADR 008 clause 4). Whether the user means this split, and what a column +holding both text and numbers does, is question 27. + +**For text, two costs below shrink.** A text value's display width does not depend on +`number_format`, so there is no rebuild when the format changes, and no formatting on +the write path, only a display-width measurement of the string. What remains is the +histogram per column, and the delta of text cells over the whole column, including fold +rows. **DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. When the widest value is retracted, the new maximum cannot be computed from the old @@ -1992,13 +2033,12 @@ three things: - **A rebuild whenever the format changes.** `number_format` is a parameter of the view definition, and a rebuild costs O(occupied rows) per column, as a job. -**What it would buy, set against what the user decided.** A maintained whole-column width -is ADR 008's question 2 option (c), "size each column from its whole contents", which the -user did **not** choose. Under (b) a window's widths are an O(visible) computation over -cells the window already evaluates, so maintaining them incrementally adds nothing. So -the real question is whether the user wants (c) after all, now that its state is cheap. It -would remove the jitter at its source and make hysteresis unnecessary, at the upstream -cost above. That is question 12. +**Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 +option (c), "size each column from its whole contents". The user had chosen (b) there, +and now chooses the maintained width for string-value columns. So the two rules split by +column kind, and ADR 008 should record that (question 27). *As first written* +(superseded), this paragraph recommended not maintaining widths "now", and asked the +user in question 12. The user answered yes. ### 14. Resident state, and the store-memory requirement @@ -2115,8 +2155,8 @@ displayed path until the differential test holds. record's leading cell. Formula values that depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals**, on question 3's answer: the exact - accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's +6. **`Sum` and `Avg`, and column totals**, with the exact accumulator in both backends, + as the user decided on question 3. A column total is a formula over the pooled fold, so it needs step 5's re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs question 3's (c) for that subtraction to be exact. @@ -2137,10 +2177,11 @@ displayed path until the differential test holds. only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, the cost of a step and of a rebuild (clause 14). -**Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is -question 10. improvise-swre stays independent and is useful under either backend. +**Tickets.** improvise-felo.4 is this ADR's record **only**. The user: "No, new epic and +felo.4 records ADR only" (question 10). So the steps above go to a **new implementation +epic**, which this revision does not file. improvise-swre stays independent and is useful under either backend. improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is -already inexact beyond `FLOAT_EQ_EPSILON`. Under question 3's recommendation it is no +already inexact beyond `FLOAT_EQ_EPSILON`. Under the user's answer to question 3 it is no longer a prerequisite of step 6. **Good.** @@ -2186,23 +2227,92 @@ longer a prerequisite of step 6. - A position over a changing item list is never fully absolute. An item insert moves every later position, and is sent as an axis change (clause 12). +## What the user decided + +**On 2026-10-09 the user answered questions 1 to 10 and 12** in inline comments on #309, +in review 5465446805 (03:34:52Z to 04:32:13Z). They are quoted exactly below, double +spaces and the "is is" kept. Question 3's comment is anchored on an older head +(`d766605b`, line 1100), where that line is question 3's option (c). Each other comment +is mapped from its `original_line` on `6ad36437` to the question it closes. **The user +has not said "Approved"**, so the ADR stays proposed. Question 11 is unanswered. + +- **1.** (4226370667) "zset is is fine." The crate is `zset` (clause 1). +- **2.** (4226371183) "seam crate". `Incremental` lives in the seam crate + `improvise-incremental` (clause 2). +- **3.** (4226373188) "use the exact accumulator." Option (c): `ExactSum` for `Sum` and + `Avg` over f64 (clause 8, migration step 6). +- **4.** (4226588257) "Yeah, for testing purposes but eventually remove it because DBSP + should fix the cycle issue." The sixteen-pass bound is reproduced so that the + differential test compares like with like, and is to be removed later (clause 6; + question 25). +- **5.** (4226590267) "For validation, but if there's a more natural DBSP-native way we + should plan on that." The component fixed point is kept for validation, and a + DBSP-native treatment of cycles is to be planned (clause 6; question 25). +- **6.** (4226592632) "For now: the goal is to only have DBSP formulas". `Eager` stays + for now, and the goal is that formulas are evaluated by DBSP only (clause 2; + question 26). +- **7.** (4226599186) "As much state as possible should be maintained in the backend, + unless it's UI-specific state. As far as possible, state should be incrementally + recomputed using DBSP operators to minimize per-request recalculation." And + (4226616602) "However, this is intrinsically limited because the grid is infinite, so + we probably need to have a mix of caching and a pull model to expand the cached data." + *The author's reading:* this endorses question 7's slice, which keeps every page's + state maintained in the backend, and it states a principle that bears on questions 13 + and 14 (clause 12). +- **8.** (4226618377) "Eager should be deprecated." This **overrides** question 8's + recommendation, which kept query-only sources on `Eager` (clause 10; question 26). +- **9.** (4226622610) "The recommendation is fine, I think". The backend's state joins + the rollback snapshot, with no later switch to the inverse delta (clause 11). +- **10.** (4226621783) "No, new epic and felo.4 records ADR only". This **overrides** the + recommendation: improvise-felo.4 stays the record of this ADR, and the implementation + steps go to a new epic. Nothing is filed by this revision. +- **12.** (4226624890) "Yes, I think the host should maintain the max width for + string-value columns incrementally." This **overrides** the author's recommendation + ("no, not now"). Clause 13 is rewritten as the decided direction; its scope is + question 27. + +**What the answers change, and where they pull against earlier text** (the author's +readings, flagged): +- **`Eager` is deprecated, not permanent** (questions 6 and 8). Clause 2's "why keep + both", clause 10's query-only sources on `Eager`, and clause 5a's eager path for + formulas the per-record map cannot express all assumed a permanent `Eager`. They now + hold "for now", as the differential test's oracle. What replaces `Eager` for those two + uses is question 26. +- **"Only DBSP formulas"** (question 6) pulls against clause 2's rule that formulas are + never compiled into operators, and that the one existing evaluator computes each + value. The user's words may mean "every formula goes through the circuit", which + clause 2 already allows, or "formulas become operators". Question 26. +- **"DBSP should fix the cycle issue"** (question 4) pulls against clause 6's argument + that DBSP's least fixed point is not improvise's circular semantics. `X = X + 1` has no + fixed point. Question 25. +- **"As much state as possible ... in the backend"** (question 7) sits beside the + user's ADR 008 Q33 answer, "The backend should only hold model-relevant information". + The two agree for derived model state, such as the shape cache and maintained + aggregates, and both exclude UI state. "A mix of caching and a pull model to expand the + cached data" fits question 13's plain (b), where clients pull on a notice, and question + 14's shape cache. It does not decide question 13 or 14, and does not say whether a + per-connection topic counts as backend state. +- **String-value column widths are maintained by the host** (question 12). That + narrows, for those columns, ADR 008's decided width rule: (b), sticky per session with + hysteresis, over the rows realized. ADR 008 should record it. Question 27. + ## Open questions Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered -none of questions 1 to 11 on #309.** Questions 12 to 24 were added by the 2026-10-05 and 2026-10-06 -revisions, and none is renumbered. +the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered +questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10-05, +2026-10-06 and 2026-10-09 revisions, and none is renumbered. -1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? +1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. -2. *[Open.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on +2. *[Answered: a seam crate.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on `zset` and `improvise-core`), or inside improvise-core? *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. -3. *[Open.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? +3. *[Answered: (c), the exact accumulator.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums until then. - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in @@ -2229,35 +2339,35 @@ revisions, and none is renumbered. that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - `Avg`'s one rounding must be stated in step 6. -4. *[Open.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported +4. *[Answered: reproduce it for testing, and remove it eventually; question 25.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported `circular` today (core.md §3). Should `Incremental` reproduce that? *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to change, change it in both backends at once, as its own decision. -5. *[Open.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by +5. *[Answered: yes, for validation, and plan a DBSP-native way; question 25.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by today's bounded fixed point, restricted to their component, rather than by any operator in the crate? *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the cells a guarded cycle leaves convergent. The crate needs no recursion operator. -6. *[Open.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? +6. *[Answered: for now; the goal is DBSP formulas only; question 26.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? *Recommendation:* yes, permanently. It is the differential test's oracle and the backend for ADR 003's query-only sources. Deleting it would delete the test. -7. *[Open. ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so +7. *[Answered in principle (the author's reading: the slice); see "What the user decided". Earlier note: ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so switching page reads existing state, rather than rebuilding the view's operators per page change? *Recommendation:* the slice. The state stays bounded by the stored cells, and the held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". -8. *[Open.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, +8. *[Answered: "Eager should be deprecated", which overrides the recommendation; question 26.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, with query-only sources kept on `Eager` and ADR 003's pushdown? *Recommendation:* yes, chosen per document. Integrated state is proportional to the source, which contradicts ADR 003's larger-than-memory goal for exactly the sources that cannot report their changes anyway. -9. *[Open. Clause 14 now weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` +9. *[Answered: the recommendation, hedged "I think". Clause 14 also weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` snapshot that `Host::mutating_frame` already takes? *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying `−d` restores integrated values but not the history `delay` and `differentiate` @@ -2266,7 +2376,7 @@ revisions, and none is renumbered. today. A true rewind would need its own design and an observational-equivalence test over every stateful operator. -10. *[Open.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps +10. *[Answered: no; a new epic, and felo.4 records the ADR only.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by steps 4, 5 and 6? *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are @@ -2291,7 +2401,7 @@ revisions, and none is renumbered. rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn order, so that rebuild does not happen, and a position key would go stale. -12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008). +12. *[Answered: yes, for string-value columns; the recommendation below is overridden; question 27.]* **Incremental max width** (clause 13; the user's question on ADR 008). Should the host maintain each column's whole-column maximum display width? That is ADR 008's question 2 option (c), and its state would now be a small histogram per column. *Recommendation (the author's):* no, not now. The user decided on (b) with @@ -2503,3 +2613,40 @@ revisions, and none is renumbered. need not travel. *Recommendation (the author's):* hidden items and collapsed groups travel with the request; the drawn order, number format and prune stay with the client. Not decided. + +25. *[New.]* **What "DBSP should fix the cycle issue" means** (clause 6; the user's + answers to questions 4 and 5). Clause 6 argues that DBSP's recursion computes a least + fixed point, which improvise's circularity is not: `X = X + 1` over a stored 10 has + no fixed point, and the required answer is `Error("circular")`. A DBSP-native + treatment could be: + - (a) a recursion operator bounded at sixteen iterations, which writes `circular` + where it has not converged, so the bound stays part of the semantics; + - (b) structural detection: every cell in a cycle of the formula graph is `circular`, + which changes today's answer for guarded cycles such as `A = IF(x, B, 1)`; + - (c) a monotone reformulation, where one exists. + + *Recommendation (the author's):* (a) for the first DBSP-native step, because it + keeps today's answers, and the differential test can then retire the per-cell fixed + point. Not decided. + +26. *[New.]* **What replaces `Eager`, and what "only DBSP formulas" means** (clause 2; + the user's answers to questions 6 and 8). Two parts: + - (a) Does "only DBSP formulas" mean every formula is evaluated through the circuit, + with the one per-cell evaluator inside it as clause 2 has it? Or that formulas are + compiled into operators, which clause 2 rejects as a second evaluator? + - (b) Once `Eager` is gone, what serves a query-only live source (clause 10), and the + formulas the per-record map cannot express (clause 5a)? And what is the + differential test's oracle? + + *Recommendation (the author's):* (a) the first reading, which keeps one evaluator. + (b) the occupied-contexts fallback becomes a circuit stage. A query-only source gets + its own pushdown backend behind clause 2's trait, not `Eager`. The differential test's + oracle becomes a small, slow, from-scratch reference implementation kept in the test + crate only. Not decided. + +27. *[New.]* **Which columns are "string-value columns"** (clause 13; the user's answer + to question 12). Is it columns whose cells are text, so that numeric columns keep ADR + 008's (b) with hysteresis? And what does a column holding both text and numbers do? + *Recommendation (the author's):* a column whose drawn cells include any text keeps a + maintained width for its text cells. The view layer takes the larger of that and its + held numeric width. ADR 008 should record the split. Not decided. From 3335ace76e3be98f3a0ed65d996f09f8ca3668e0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:42:22 -0700 Subject: [PATCH 40/67] docs(adr): ADR 009 scopes clause 13's costs to numbers, restates ADR 005 as merged, pings both ways (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 26 ++++++++++++++++------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 05e14ef8..f1d2c49d 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1125,7 +1125,10 @@ for floating point. On its own, **none**: **The benefits come from two other places.** - **The representation.** An exact accumulator (next subsection), or ADR 005's decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian - group, provided both backends use the same accumulator. Decimals alone do not: ADR + group, provided both backends use the same accumulator. Under ADR 005 as merged that + holds **per partition**: the 320-bit decimal partition and each fraction group. A read + that combines more than K = 8 distinct non-decimal denominators is inexact by ADR 005's + cutoff. Decimals alone do not: ADR 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at its 28-digit bound, so the order of summation changes both the value and the exactness. Then retraction is exact, incremental @@ -1243,7 +1246,7 @@ operator rather than one per call site. | `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | | typed values such as money | no | yes | | a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 (PR #320): `rust_decimal`, not a bignum, measured at +15.4 KB gzipped in the wasm bundle, plus 40 bytes per summed group for its wide accumulator | +| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 as merged (option (d)): a three-level tower, `Decimal` / `Fraction(Box>)` / `Inexact(f64)`, in `numtower`. `rust_decimal` alone was measured at +15.4 KB gzipped, and `num-rational` with `num-bigint` at about +20.8 KB on their own; the combination is unprobed. `numtower` is not yet in the workspace (implementation epic improvise-59nw). Its sum keeps a 320-bit decimal partition and a 256-bit numerator per fraction group, and a read combines at most K = 8 groups (the user's final answer; K = 32 was superseded). The 40 bytes per summed group cover the decimal partition only | **The accumulator does not make `=` against a typed decimal right.** It sums the *stored* f64s exactly, and each stored f64 is already the rounded form of what the user @@ -1871,7 +1874,10 @@ lists; every batch's header goes to everyone** (found by the review of `e152b667 the heartbeat and the lapse together. A live connection keeps its subscriptions. A dropped connection drops them. **The transport must detect a dead connection**: a websocket ping with a timeout on the socket server, a closed port for the worker, and - nothing for the TUI, which is in process. A half-open socket that is never detected + nothing for the TUI, which is in process. **The ping runs both ways.** A client must + also detect a dead host, or it reads silence as "no batches" and keeps a stale + window. So a client that hears neither a header nor a ping within the timeout treats + its connection as dropped, and refetches on reconnect. A half-open socket that is never detected would hold its pins forever. A client always knows when its connection dropped, and it refetches on reconnect (T3's general rule). *The author's lean*, because it is the simplest of the three. The alternatives were a per-client subscription epoch in @@ -2016,21 +2022,25 @@ count, per column. It is O(distinct widths) per column, whatever the number of r an update costs O(log d), where d is the number of distinct widths. **The cost is upstream, in what feeds the histogram.** Its input is the display width of -**every evaluated cell in the column**, under the request's number format. That needs -three things: +**every evaluated cell in the column**. Of the three costs below, the second and third +apply **only to numbers**, which a host-maintained width would cover only if question 27 +extends the decision to them. For text, the first one remains: - **A delta of evaluated cells over the whole column**, not a window. Clause 4's group aggregates and clause 5's formula re-evaluation already produce an output delta for every unfolded cell that moves, with its old value as the retraction. So unfolded rows need no new state. **Fold rows are not covered.** Under item grain (clause 7) a collapsed group's value is combined when it is read, not maintained, so its width would - need the fold maintained per column at the visible grain. -- **Formatting every changed cell on the host**, on the write path: `format_f64` and a + need the fold maintained per column at the visible grain. That holds for text as well: + a fold row's text, under the unanimity rule, is a combined value. +- **A client's hidden rows** come out of the whole-column histogram the way they come out + of a total: subtracted at read time by question 23's (A) overlay, at O(H log d). +- **Numbers only: formatting every changed cell on the host**, on the write path: `format_f64` and a display width for each changed cell. Each one is cheap, but it would be the first display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But today the host is asked for widths only over the rows requested (ADR 008 clause 4), and this would widen that to whole columns. -- **A rebuild whenever the format changes.** `number_format` is a parameter of the view +- **Numbers only: a rebuild whenever the format changes.** `number_format` is a parameter of the view definition, and a rebuild costs O(occupied rows) per column, as a job. **Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 From 0dc881f934de451df714be6e13b8cea8fef6545b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:58:58 -0700 Subject: [PATCH 41/67] docs(adr): ADR 009 records the user's connection-operator consideration (C13) and its bearing on delivery (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 87 +++++++++++++++++++++-- 1 file changed, 82 insertions(+), 5 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index f1d2c49d..89f939da 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -2,11 +2,11 @@ **Status:** proposed. Only the user accepts an ADR. The numbered questions at the end are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and -12** (see "What the user decided"). Question 11 and questions 13 to 27 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** +12** (see "What the user decided"). Question 11 and questions 13 to 28 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** (PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 27. *Superseded:* until 2026-10-09 the +ADR", in clauses 12 to 14, and in questions 12 to 28. *Superseded:* until 2026-10-09 the user had answered none of questions 1 to 11. · **Ticket:** improvise-felo.4 · **Decision from:** the user, 2026-09-25 (below) · @@ -83,7 +83,7 @@ user routed part of it here. This section quotes what the user said that bears o and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **Twelve chat statements**, C1 to C12 below, were given to the orchestrator and relayed +- **Thirteen chat statements**, C1 to C13 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the PR, but spread over several groups of its own. ADR 008's list also has six more (its @@ -126,6 +126,11 @@ None of this answers this ADR's own questions 1 to 11. - **C12.** "The simplest might be some kind of binary search for a new top left corner: increasing jumps of the top left until you get a hit and then move down/right until You only have one new row column." Clause 12, question 22(d). +- **C13.** "I think we can minimize the amount of UI state on the backend by having an + operator of some kind responsible for both ends of the connection between the frontend + and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). + A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, + 14 and 28. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1960,6 +1965,55 @@ the user's "recent projections + clients" (C2) becomes clause 7's shape cache, w keyed by requests and holds no client, plus the topic subscriptions, which hold one shape key per client. +**A connection operator (C13, the user's consideration; everything below is the +author's reading).** The user: "I think we can minimize the amount of UI state on the +backend by having an operator of some kind responsible for both ends of the connection +between the frontend and backend and handling keep alive / session resumption / etc". +- **What "operator" means is ambiguous.** It is read here **not** as a DBSP operator, + but as a session or connection manager that sits between a frontend and the host and + owns both ends of the link: a gateway, or an actor. Question 28. +- **What it would hold, so that the host does not:** the connection; the keep-alive, with + the ping both ways; session resumption, meaning a reconnect with a token and then the + replay-or-refetch rules (T3's host instance and publication number); the topic + subscriptions and their pins ((b′)); and each client's last applied stamp. The host + would then answer `project` and `apply`, publish its unaddressed notices, and know + nothing about clients. The operator is the one place that does. +- **Where it could live, in today's code:** + - in process, for the TUI; + - beside the host in the browser worker: `improvise-worker-server`'s `Worker` already + owns one `Host` and one port (`worker.rs:14-15` at `f0fb6880`); + - as the socket server's front: `improvise-ws-server`'s `Registry` and transport are + the nearest thing today (`registry.rs:88-89`). + + Whether it is a new crate, a new process, or a role that existing code takes on is + question 28. +- **What it changes for question 13.** If the operator subscribes to the host's + unaddressed notices **on behalf of its clients** and fans them out, the host sends + O(operators) notices per batch, not O(clients). The per-client filtering of (b′), and + even the window restriction of (c), move into the operator. The wire between host and + operator then carries (b)'s broadcast, and the wire between operator and client carries + (b′)'s or (c)'s traffic. That changes the trade the review of `5a8dc486` caught, of host + state against wire traffic: with an operator, (b′) and (c) cost the **operator** state, + not the host. The costs are a new component, a second hop, and a place where a lost + notice can happen. When the operator is in-process or in the worker, the second hop is + a function call. None of this is measured. +- **What it changes for question 14.** Pins and subscriptions belong to the operator. It + re-requests the shapes its clients read, and that keeps them alive in the host's cache, + so the host's eviction rule (no recent request) holds unchanged. A host-side pin is no + longer needed. +- **The Q33 question.** If topic subscriptions live in the operator, the host holds none, + and a subscription is no longer "backend" state in the sense of the user's Q33 answer. + That holds only if the operator does not count as backend, which is part of question + 28. +- **Network transparency and "as stateless as possible".** T1 to T6 then hold on each + hop, and the operator must not change a result, only route it. The host becomes as + stateless toward clients as plain (b), and the per-client state is concentrated in one + component that can be restarted, because a client then refetches (T3's general rule). + In ADR 008's view/host split, the operator sits on the **host side of the wire, outside + the model**. It knows clients and holds no UI state beyond delivery state. +- **What it does not answer.** Question 13's choice, question 14's eviction policy, and + whether the operator is a new component at all. + **What becomes of today's pieces under (b) and (b′)** (the author's): - **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its levels' item counts, which a client derives from the category lists it fetches. A @@ -2310,7 +2364,7 @@ readings, flagged): Each question has a recommendation. The user's answers will be folded in under "What the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered -questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10-05, +questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10-05, 2026-10-06 and 2026-10-09 revisions, and none is renumbered. 1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? @@ -2436,7 +2490,9 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 O(N) headers per batch and per-connection state, which holds no UI state. **Question for the user:** is a topic subscription "model-relevant information", in the sense of your answer to ADR 008's Q33 ("The backend should only hold model-relevant - information")? (b′) depends on the answer; + information")? (b′) depends on the answer. The user's consideration C13, a + connection operator that owns both ends, would move topic subscriptions out of the + host, so they stop being host state (clause 12, question 28); - (c) push the model's data changes through soft-state subscriptions; - (d) today's per-mirror push, which is obsolete. @@ -2466,6 +2522,9 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 name, and a client that toggles hides through many sets would pay to derive each new entry. Only an unsaved change of axes, filters or `measure_agg` builds a new key, as a job. + *Under C13* (the user's consideration, not decided), pins move to a connection + operator, which keeps shapes alive by re-requesting them. The host's rule is then only + "evict once no recent request names it" (clause 12). 15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate under the wire's identity, where every NaN is one value, while the record tier keeps @@ -2660,3 +2719,21 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 27 were added by the 2026-10 *Recommendation (the author's):* a column whose drawn cells include any text keeps a maintained width for its text cells. The view layer takes the larger of that and its held numeric width. ADR 008 should record the split. Not decided. + +28. *[New.]* **The connection "operator" of C13** (clause 12). The user: "an operator of + some kind responsible for both ends of the connection between the frontend and + backend and handling keep alive / session resumption / etc". Four parts: + - (a) Is "operator" a session or connection manager (a gateway or actor that owns + both ends of the link), as the author reads it, and not a DBSP operator? + - (b) Is it a new crate or process, or a role that existing code takes on: + `improvise-worker-server`'s `Worker` in the browser, `improvise-ws-server`'s + `Registry` and transport for sockets, and in-process code in the TUI? + - (c) Does it count as "backend" in the sense of the user's Q33 answer, so that + topic subscriptions held there are or are not backend state? + - (d) Does it subscribe to the host's notices on behalf of its clients and fan them + out, so that the host sends O(operators) per batch? + + *Recommendation (the author's):* (a) yes. (b) a role of the existing transport + crates, not a new process, so the TUI and the worker get it with no extra hop. (c) + not backend: it is delivery and holds no UI state. (d) yes. With that, question 13's + choice moves to the operator–client hop, and the host sends plain (b). Not decided. From 74eb5aaaadc4d062681362bbc9abf4aa5256d6e6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 21:59:29 -0700 Subject: [PATCH 42/67] docs(adr): ADR 009 reads C14: the operator is a sink/source adapter at the wire boundary (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 79 +++++++++++++++++------ 1 file changed, 61 insertions(+), 18 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 89f939da..8d2e9bd5 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -83,7 +83,7 @@ user routed part of it here. This section quotes what the user said that bears o and says where each statement came from. - **PR comments** are cited by their GitHub comment id and UTC time. The question numbers are ADR 008's, as ADR 008 assigns them. -- **Thirteen chat statements**, C1 to C13 below, were given to the orchestrator and relayed +- **Fourteen chat statements**, C1 to C14 below, were given to the orchestrator and relayed to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR would confirm them. ADR 008 records the same statements, also as relayed and not on the PR, but spread over several groups of its own. ADR 008's list also has six more (its @@ -131,6 +131,9 @@ None of this answers this ADR's own questions 1 to 11. and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, 14 and 28. +- **C14.** "And it asks as a sink/source for the DBSP graph part on the backend and a + source/sink for the client side" (relayed 2026-10-09, after C13). Quoted exactly as + sent. *The ADR reads "asks" as "acts"*, flagged. Clause 12, question 28. **Asked on ADR 008, and not yet answered:** - "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, @@ -1969,9 +1972,38 @@ key per client. author's reading).** The user: "I think we can minimize the amount of UI state on the backend by having an operator of some kind responsible for both ends of the connection between the frontend and backend and handling keep alive / session resumption / etc". -- **What "operator" means is ambiguous.** It is read here **not** as a DBSP operator, - but as a session or connection manager that sits between a frontend and the host and - owns both ends of the link: a gateway, or an actor. Question 28. +- **What "operator" means, after C14.** C14 resolves most of C13's ambiguity: "And it + asks as a sink/source for the DBSP graph part on the backend and a source/sink for the + client side", with "asks" read as "acts" (flagged). *The author's reading:* the + operator is **shaped like the DBSP graph**. Toward the host it is a **sink** for the + graph's output streams (projection results and change notices) and a **source** of the + requests and model commands it feeds in. Toward the client it is a **source** of those + results and notices, and a **sink** for the client's requests and events. So the + connection between frontend and backend is itself a pair of sink/source endpoints, and + the operator owns both. What stays ambiguous is whether "both ends" means both halves of + that pair inside one component, or a component in each of the two processes. Question + 28. +- **Why this fits C1** ("network transparent"). The circuit's streams are the same + whether consumed in process or across a wire, and the operator is the **adapter at the + wire boundary**. It owns: + - the serialization of Z-set deltas, under T1 (`CellKey` is not `Serialize`, so the wire + form names tuples and ADR 002 surrogates) and T2 (the wire's NaN identity); + - the stamps and the gap rule (T3); + - snapshot-then-delta (T4); + - coalescing and backpressure (T5); + - session resumption. + + In process, both of its halves collapse to function calls, which is T6. +- **Where it sits among the crates** (the author's reading). The host-facing half + consumes the circuit's output types, so it belongs with the seam crate + `improvise-incremental` (question 2's answer) or right above it. The wire types it + serializes belong in `improvise-protocol`, beside the one envelope pair (gotcha 42). + `zset` stays dependent on `std` alone (question 1's answer), with no serde, because the + conversion happens in the operator. The **client-facing half** would be written again + by every front end unless it is shared, so by design-principles §5 it belongs in the + shared layer, as `improvise-protocol`'s client fold (`ClientState`, `client.rs`) + already is. Who owns keep-alive and resumption state, and whether the TUI, the browser + and a CLI share one client half, are question 28. - **What it would hold, so that the host does not:** the connection; the keep-alive, with the ping both ways; session resumption, meaning a reconnect with a token and then the replay-or-refetch rules (T3's host instance and publication number); the topic @@ -2720,20 +2752,31 @@ questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10 maintained width for its text cells. The view layer takes the larger of that and its held numeric width. ADR 008 should record the split. Not decided. -28. *[New.]* **The connection "operator" of C13** (clause 12). The user: "an operator of - some kind responsible for both ends of the connection between the frontend and - backend and handling keep alive / session resumption / etc". Four parts: - - (a) Is "operator" a session or connection manager (a gateway or actor that owns - both ends of the link), as the author reads it, and not a DBSP operator? - - (b) Is it a new crate or process, or a role that existing code takes on: - `improvise-worker-server`'s `Worker` in the browser, `improvise-ws-server`'s - `Registry` and transport for sockets, and in-process code in the TUI? - - (c) Does it count as "backend" in the sense of the user's Q33 answer, so that - topic subscriptions held there are or are not backend state? +28. *[New.]* **The connection "operator" of C13 and C14** (clause 12). The user: "an + operator of some kind responsible for both ends of the connection between the + frontend and backend and handling keep alive / session resumption / etc", and "And it + asks as a sink/source for the DBSP graph part on the backend and a source/sink for the + client side" ("asks" read as "acts", flagged). Part (a) is read as answered by C14. + - (a) *Read as answered by C14:* it is a DBSP-graph-shaped adapter, a sink and source + toward the graph and a source and sink toward the client, not a DBSP operator inside + the circuit. Still open: does "both ends" mean both halves of the sink/source pair + inside one component, or a component on each side of the wire? + - (b) Is it a new crate or process, or a role existing code takes on? Its host-facing + half could live with `improvise-incremental`, its wire types in + `improvise-protocol`, and its client-facing half beside `improvise-protocol`'s + client fold, shared by the TUI, the browser and any CLI (design-principles §5). The + browser's `Worker` (`worker.rs:14-15` at `f0fb6880`) and the socket server's + `Registry` and transport (`registry.rs:88-89`) are the nearest code today. + - (c) Who owns keep-alive and resumption state: the operator's host-facing half, its + client-facing half, or both? And does it count as "backend" in the sense of the + user's Q33 answer? - (d) Does it subscribe to the host's notices on behalf of its clients and fan them out, so that the host sends O(operators) per batch? - *Recommendation (the author's):* (a) yes. (b) a role of the existing transport - crates, not a new process, so the TUI and the worker get it with no extra hop. (c) - not backend: it is delivery and holds no UI state. (d) yes. With that, question 13's - choice moves to the operator–client hop, and the host sends plain (b). Not decided. + *Recommendation (the author's):* (a) one component per connection that owns both + halves, with the client half as a shared library each front end links. (b) the + host-facing half in or above `improvise-incremental`, wire types in + `improvise-protocol`, and the client half shared beside `ClientState`. (c) resumption + state in the host-facing half, keyed by a token the client holds, and treated as + delivery state, not backend UI state. (d) yes. Then question 13's choice moves to the + operator–client hop, and the host sends plain (b). Not decided. From b7add90830a1741cd27abe7252ebd937eba6a76a Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 23:09:57 -0700 Subject: [PATCH 43/67] docs(adr): ADR 009 condensed to problem, decision and open questions (improvise-felo.4) Rewritten on the user's instruction: settled questions are removed, revision and review history is removed, the decision is stated plainly, and only unsettled questions remain, renumbered 1-17 for a first-time reader. The record of the user's answers moves to a PR comment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 3275 ++++----------------- 1 file changed, 612 insertions(+), 2663 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 8d2e9bd5..e78b1398 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,2782 +1,731 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait -**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end -are theirs to answer on the PR. **On 2026-10-09 the user answered questions 1 to 10 and -12** (see "What the user decided"). Question 11 and questions 13 to 28 are open. **Revised on 2026-10-05 to fit ADR 008 revision 37** -(PR #308, read at `c8be7720`, which the user is still reviewing). The revision brings in -the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details -ADR 008 routes here. They are in the section "What ADR 008's review established for this -ADR", in clauses 12 to 14, and in questions 12 to 28. *Superseded:* until 2026-10-09 the -user had answered none of questions 1 to 11. · -**Ticket:** improvise-felo.4 · -**Decision from:** the user, 2026-09-25 (below) · -**Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) · -**Relates to:** ADR 005 (exact decimals, improvise-l2b8; PR #320 merged on 2026-10-09, and its own status line still reads proposed), ADR 008 (the -virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision -reads it at `c8be7720`, its revision 37) · -**Governed by:** `context/design-principles.md` §3 and §5 +**Status:** proposed. Only the user accepts an ADR. · +**Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own +epic) · +**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed +models) · +**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") · +**Governed by:** `context/design-principles.md` §3, §5 and §7 -**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005 -is on main since PR #320 merged (2026-10-09), and 006 has no file yet. 007 is merged, and 008 is PR #308. 009 is the next free number. No bead and -no open PR claimed it when this was written. +**Summary.** Formulas are evaluated incrementally by a small in-house implementation of +the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A +cell write becomes a delta. Aggregates are maintained rather than re-folded, and only +the formula cells whose inputs moved are re-evaluated, by the one existing evaluator. +Operator state is keyed by the **shape key** of a projection request, never by a client. +The same circuit serves the in-process TUI and remote clients. A remote client learns of +changes from one unaddressed notice per accepted batch, and it refetches what it shows. +Sums are exact, through an exact accumulator. -**Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are -dated, so re-derive them before citing this document elsewhere. **The 2026-10-05 -revision's new citations are at `gh/main` `50e1314d`**, and each says so. The older -citations were not re-derived, and some have moved. For example, `Host::mutating_frame` -takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and -`App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means -revision 37 of `context/adr/008-virtualized-grid.md` on PR #308, at `c8be7720`. Clauses 3, -5 and 8, which this ADR cites most, have the same numbers there as in revision 3. +File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive +them before relying on one. -## The decision this ADR writes down +## Problem and context -The user pointed at a teaching implementation of the DBSP paper, -[brurucy/dbsp-from-scratch](https://github.com/brurucy/dbsp-from-scratch/blob/master/dbsp_paper_walkthrough_implementation.ipynb), -as a way to evaluate formulas so that data can stream. The notebook builds, in order: +**Data that streams.** A live-backed model (ADR 003) is a source whose rows change +underneath the document. Today every change takes the path an edit takes, which +recomputes every formula cell. That is tolerable for one keystroke at a time and not for +a stream of changes. A change should cost in proportion to what it touches. -- abelian groups; -- streams, with `Delay` (z⁻¹), `Differentiate` and `Integrate`; -- Z-sets, which are maps from element to an integer weight; -- lifted and doubly-lifted joins, and a delta join that keeps less state; -- `distinct` through the `H` function over an integrated state; -- recursion by nested streams, shown on graph reachability. +**Today's evaluation is a from-scratch fold after every batch.** +- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls + `Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates + every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16` + (`types.rs:710`). +- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`, + `types.rs:848`, through `DataStore::matching_values`). +- `formula_cache` holds one aggregation shape at a time. A session on a view with another + None axis gets a **cloned model** (`rederived_for`, `project.rs:205`). +- A formula total that holds an aggregate walks a cross product + (`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`). +- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing + batch, and `document_content` (`app.rs:168`) after each `CellData` effect. +- Every mutating frame from a client clones the document for rollback + (`Host::mutating_frame`, `host.rs:224-225`). -The user then decided: +**What an incremental evaluator must not change.** Each of these is a place where it +could quietly disagree with today's code: +- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one + distinct value is that value, and anything else is `Error("mixed")`. +- **Circularity (gotcha 9).** A cell still moving after 16 passes is + `Error("circular")`, and so is an acyclic chain that needs all sixteen. +- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0 + guard. +- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration, + because f64 addition is not associative. +- **The tiers are observed (gotcha 40).** A write of the value already there bumps + nothing, and a rewritten record counts once. -> "I think doing our own implementation of the core of dbsp as a crate we can plugin is -> probably dieal [sic] here. Write the ADR and Update ADR 008 to assume the decisions in -> that one" -> -> — the user, 2026-09-25, recorded on improvise-felo.4 - -So the choice between Feldera's crate and our own is already made (see Alternatives for -what was checked). This ADR says what "the core of dbsp" is for improvise, where it -plugs in, and what it may not change. - -## What it is for - -**The primary benefit is incremental formula evaluation, so that data can stream.** -That is the user's own framing, both in the prompt that led to this ADR and in their -later question: "Isn't the benefit of adopting DBSP primarily incremental formula -updates?" Yes. When a cell changes, whether from a keystroke, a client's frame or a -live source (ADR 003, clause 10), only the aggregates that cell feeds are adjusted, -and only the formula cells whose inputs moved are re-evaluated. Nothing is recomputed -from scratch. The machinery is: - -- the aggregates formulas read are **maintained**, not re-folded (clause 4); -- an aggregate over a per-record expression such as `SUM(Revenue * Price)` is a - maintained operator too (clause 5a); -- the circuit decides **which** formula cells moved, and the one existing evaluator - recomputes only those, in dependency order (clauses 5 and 6). - -**The grid is the second consumer.** ADR 008 needs the same maintained state for other -purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation, -and column totals over rows never scrolled to (clause 9). It gets them from the same -circuit, not from a second mechanism. Since ADR 008's view/host split, the grid's consumer -may be in another process, and clause 12 says how the same circuit serves both. - -## What ADR 008's review established for this ADR - -At its revisions 18 to 37, ADR 008 (PR #308) became the ADR of the **view/host split**, and the -user routed part of it here. This section quotes what the user said that bears on this ADR, -and says where each statement came from. -- **PR comments** are cited by their GitHub comment id and UTC time. The question numbers - are ADR 008's, as ADR 008 assigns them. -- **Fourteen chat statements**, C1 to C14 below, were given to the orchestrator and relayed - to this ADR on 2026-10-05. **They are on no PR.** The user's "Approved" on this ADR - would confirm them. ADR 008 records the same statements, also as relayed and not on the - PR, but spread over several groups of its own. ADR 008's list also has six more (its - items 1 to 6: the five standing design statements, and the "q19 … q24" corrections). - This ADR relies on none of those six. This list is the one place this ADR - quotes them, and the clauses cite them by number. - -None of this answers this ADR's own questions 1 to 11. - -**Relayed in chat, on no PR** (all on 2026-10-05; quoted exactly as relayed): -- **C1.** "The DBSP machinery needs to be implemented in a network transparent way, but - the details of that go to ADR 009". Clause 12. -- **C2.** "Thinking about it a bit more, we might need to keep track of recent projections - + clients to stream aggregation results back to the UI correctly. But this should be as - stateless as possible". Clause 12. -- **C3.** "Q53 is cross product: I'm still ambivalent about which side of the wire pruning - should happen on". ADR 008 records it as its question 53 answered and its question 57 - open. Clause 12, question 19. -- **C4.** "But the top left should be as absolute as possible". Clause 12, question 20. -- **C5.** "After pruning we might have to stream rows and columns to attempt to fill out - the full grid". Clause 12, question 21. -- **C6.** "Pruning is just a filter applied to full columns a rows, right? Hide a - column/row in the grid if it's all null. That sounds like a UI side concern because - that's where the visible grid lives". Clause 12, question 22. -- **C7.** "The point of pruning is to make the displayed values more dense, so a value - being present outside the viewport doesn't matter." Clause 12, question 22. -- **C8.** "The problem pruning solves is visualizing sparse datasets: screens that are - mostly empty are useless and so you drop out categories from rows and columns until - every visible row or column has at least one value on screen." Clause 12, question 22. -- **C9.** "We should be able to optimize pruning on the right and down assumption of - growth (truncated at the bottom right corner of the axis space)". Clause 12; ADR 008's - question 67. -- **C10.** "The app is free to optimize fetching for up or left by just fetching the new - rows, until it finds one that has values in the viewport but, conceptually, scrolling up - or left would request a new viewport with a new top left corner. The requested top left - corner falling on a pruned row or column is an annoying edge case, maybe just fail the - fetch and retry?" Clause 12, question 22(d). -- **C11.** "Snap forward is going to cause issues with the viewport getting stuck". - Clause 12, question 22(d). -- **C12.** "The simplest might be some kind of binary search for a new top left corner: - increasing jumps of the top left until you get a hit and then move down/right until - You only have one new row column." Clause 12, question 22(d). -- **C13.** "I think we can minimize the amount of UI state on the backend by having an - operator of some kind responsible for both ends of the connection between the frontend - and backend and handling keep alive / session resumption / etc" (relayed 2026-10-09). - A **consideration**, not a decision. Clause 12, "A connection operator"; questions 13, - 14 and 28. -- **C14.** "And it asks as a sink/source for the DBSP graph part on the backend and a - source/sink for the client side" (relayed 2026-10-09, after C13). Quoted exactly as - sent. *The ADR reads "asks" as "acts"*, flagged. Clause 12, question 28. - -**Asked on ADR 008, and not yet answered:** -- "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459, - 2026-10-03T03:15:38Z, on ADR 008's question 2). Clause 13 is this ADR's reading of it. - Question 12. - -**The read side and the write side** (#308, inline comments of 2026-10-05): -- Q43 (4189071865, 21:34:45Z): "yeah, the read side of the model is a projection of the - model's full space into a particular set of axes using a `project` verb, with - offset+dimensions. The write side is an apply function that can only take model - commands and applies them to the model using Redux-style Flux architecture." -- Q38 (4189007864, 21:27:55Z): "No, axes are a model concept because they define the - grid. Aggregation should always be done model-side for consistency." -- Q38 (4189027924, 21:30:08Z): "A grid request would look something like:" - ``` - Axes: Row(A,B,C), Col(D,E), Page(F), None(G,H) - Top-left (4,5) # relative to the cross-product of the axes in the category-native order (which might be arbitrary) - Dimensions (10,10) # so get ten rows and ten columns worth of data from the grid defined by the axes. - ``` -- Q33 (4188936570, 21:20:11Z): "The host shouldn't have to know anything about the - viewport: it's mostly a question of the parameters of the grid the user is actually - looking at, rather than a "viewport" strictly speaking." -- Q37 (4189000059, 21:27:04Z): "I don't think the host needs to no [sic] anything about - the UI: the initial load of the UI should just be the default view stored in the model - (or, in some cases, a particularly requested view). Sending UI state over the wire is - an architecture smell." -- Q42 (4189051110, 21:32:31Z): "The space scrolling happens in is defined by the axes, top - left corner offset and with/height. But how much of this space is actually materialized - and when fetches, etc. happens (and model values are cached) is client-policy driven." -- Q32 (4188917796, 21:18:17Z): "I can't think of any cursor command with model-side - effects." -- Q27 (4188816383, 21:07:43Z): "go with the recommendation: the ordered list of records is - a model-side concept." -- Q31 (4188911659, 21:17:39Z): "You have to distinguish view in the sense of UI and view - in the sense written into `.improv` files. P has to be a model-side effect because it - prunes data for a particular view. It is not a UI-only command." And (4188926877, - 21:19:10Z): "The only real issue here is that pruning shouldn't infect every client - here. It might need to be UI-only until a view is saved, and then the view definition - goes to the model." -- Q29 (4188887721, 21:15:14Z): "Prune should only be written if the user explicitly - unpruned the view." -- The cursor: "The cursor should never jump: it should stay where the user left it as much - as possible." (Q12, 4171501445, 2026-10-03T03:21:05Z). And "As far as possible, updates - should shift the grid being viewed to keep the cursor in the same visual place on the - screen." (Q34, 4188946818, 2026-10-05T21:21:18Z). - -**Answered on ADR 008 on 2026-10-06** (#308 inline comments; question numbers are ADR -008's): -- Q33 (4191973298, 05:50:48Z): "The backend should only hold model-relevant information. - If we need to persist cursor position/last view or other UI concerns, that would go in - localStorage/cookies/fragment routing/etc." -- Q39 (4191981468, 05:52:14Z): "I think layout version is mostly a view fact here, we have - to separate this from model mutations (value editing, etc.)" -- Q46 (4191986207, 05:53:05Z): "go with an explicit saye-view command" [sic]. -- Q47 (4191992428, 05:53:56Z): "The transient UI state never is reflected in a .improv - file unless an there is an explicit decision to persist it as a saved view (or by - replacing an existing view)" [sic]. -- Q48 (4191999772, 05:55:06Z): "A .improv file has to have at least a default view - definition. UI state is always ephemeral unless explicitly saved: when a new View is - created through the UI, it must not be lost if the model confirms that it was saved." -- Q49 (4192005787, 05:56:07Z): "The only one of these that should be "global" is set - number format as far as I can tell: the other ones are all ephemeral UI state." -- Q50 (4192010101, 05:56:53Z): "Disabling pruning should only be persisted if the - resulting view is explicitly saved." -- Q51 (4192014974, 05:57:40Z): "page is a third axis, do the others need to actually - travel across the wire?" -- Q52 (4192016491, 05:57:59Z): "defer this to ADR 009". This is ADR 008's question - about what replaces the pushed projection stream, and it is this ADR's question 13. - -**Answers ADR 008 records that this ADR's text had not caught up with:** -- Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9. -- Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases - where the column size has been stably smaller." And Q22 (4188309022, - 2026-10-05T20:15:12Z): "hysteresis as recommended." Clauses 9 and 13. -- Q6 (4171493762, 2026-10-03T03:18:12Z): "maintained incrementally". Clause 9. -- Q13, Q14 and Q15: "the recommendation is fine.", three comments (4171504439, - 4171505979 and 4171509012, 2026-10-03T03:22:16Z to 03:23:36Z), which ADR 008 records as - its answers to those three questions. Q13 sets the frame budget, and Q15 sends the - formulas the per-record map cannot express to the occupied contexts. Clauses 5a and 14. -- The store's memory (4171481179, 2026-10-03T03:13:14Z): "We have to be careful here to - minimize memory requirements of the store: offloading inactive regions to disk (maybe - mmap or similar), double-delta/gorilla/lz4 compression, etc." And Q17 (4188271710, - 2026-10-05T20:11:02Z): "Let's defer this as recommended." Clause 14. -- Jobs (4171458281, 2026-10-03T03:04:52Z): "We also have to have a way to - invalidate/cancel in-flight jobs efficiently and probably a way to throttle them (a - priority queue or similar) to avoid pinning the CPU." Then (4171694278, 04:30:31Z): - "priority queue for work stealing is also important here." Clause 7. - -**What this ADR reads from them.** These are the author's readings, and each one names -the clause it changes: -1. **Reads and writes go through `project` and `apply`.** A consumer reads the circuit's - outputs through `project`, and the circuit's inputs change only through `apply` of - model commands. So nothing here may depend on a host-held mirror of a client - (`Sessions`, `ClientSession`, `reduce_view` on the host), on the per-mirror diff - (`grid_updates`), or on resetting a client's cursor. Clauses 9 and 12. -2. **Aggregation stays on the model side.** No client runs an aggregate operator, so - network transparency concerns the circuit's outputs, and a live source's inputs. It - does not mean splitting operators across a wire. Clause 12. -3. **State is keyed by the request's shape, not by a view's name.** Axes are a model - concept, so operator state keyed by axes is model state, not UI state. But a `project` - request names its own axes, and an unsaved request may carry parameters no saved view - holds (ADR 008's flagged readings 1 and 3, its question 51). Clause 7, question 14. -4. **The record order belongs to the store.** The ordered list of records is the store's - (Q27). The circuit may consume it, and does not own it. Clause 9. -5. **The memory requirement covers this ADR too.** The store-memory requirement covers the - circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn. - Clause 14, question 17. -6. **ADR 008's answers this ADR now follows.** It relies on the answers to ADR 008's - questions 1, 2, 6, 12, 13, 14, 15, 17, 22, 27, 29, 31, 32, 33, 34, 37, 38, 39, 42, - 43 and 46 to 52, quoted above. - Clauses 5a, 7, 9, 12, 13 and 14 follow them. -7. **View configuration is the client's until saved** (Q46 to Q50). Hiding, collapsing, - page selection, axis assignment and pruning are ephemeral UI state per client until an - explicit `SaveView`. Only number format is "global" (Q49). So operator state is keyed - by the request, never by a client's unsaved settings held on the host. Clause 7, - questions 23 and 24. -8. **No basis tier this ADR relies on is a view fact** (Q39), and the host keeps no - per-client UI state (Q33). Clauses 7 and 12, question 13. - -## Context - -### The primary case: data that streams - -A live-backed model (ADR 003) is a source whose rows change underneath the document. -Today the only way to reflect a change is the path every edit takes: recompute every -formula cell of the document (next subsection). That is tolerable when a human types -one cell at a time. It is not tolerable when a source delivers a stream of changes, -each touching a few records of a large table. Incremental evaluation makes the cost of -a change proportional to what the change touches, not to the document. - -### Evaluation today is a from-scratch fold, rerun after every batch - -- **Every batch recomputes every formula cell.** `App::apply_effects` - (`crates/improvise-command/src/ui/app.rs:1069` at `50e1314d`) ends in `rebuild_layout` - (`:926`). - That calls `Model::recompute_formulas` (`crates/improvise-core/src/model/types.rs:971-1030`). - It clears `formula_cache`, rebuilds the stem set from every stored cell (`:986-997`), - and iterates `stem × formula` to a fixed point bounded by `MAX_EVAL_DEPTH = 16` - (`:699`, `:1001-1022`). A cursor move and a cell write pay the same. -- **Raw aggregation scans the matching cells on every read.** `evaluate_unfolded` - (`types.rs:837-845`) folds `DataStore::matching_values` through - `aggregate_cell_values` (`:30-42`). `fold_stored` (`:918-930`) does the same for a - collapsed group, through `spanned_values` (`crates/improvise-core/src/model/cell.rs:445`). -- **`formula_cache` holds one aggregation shape at a time.** It is keyed by cell and - computed for one `none_cats` list. A session on another view with a different None - axis gets a *cloned model* recomputed for its own list (`rederived_for`, - `crates/improvise-command/src/ui/project.rs:195-210`). -- **A formula total with an aggregate inside walks a cross product.** `contexts_of` - (`types.rs:905-916`) calls `Expansion::members` (`:201-213`), which materializes every - combination of the spanned categories' items. ADR 008 clause 5 names this as the one - O(grid) path left in totals. -- **Two observations scan the store.** `record_fingerprints` (`app.rs:266-276` at `50e1314d`) hashes - every cell before and after a cell-writing batch. `document_content` - (`app.rs:168-182`) hashes every cell and every formula for the dirty flag, and it is - re-read after **each** `CellData` effect (`app.rs:1119-1125` at `50e1314d`). So the first is O(cells) - per batch and the second O(cells) per effect, whatever was written. -- **Every mutating frame from a client already clones the document.** - `Host::mutating_frame` takes `App::snapshot` before applying anything (`host.rs:224-225`, - `app.rs:673` at `50e1314d`), for ADR 001 clause 4. That is O(cells) per frame on the network - path, whatever this ADR does. - -### What an incremental evaluator must not change - -These rules come from the repo map and `context/code/core.md`. Each one is a place an -incremental evaluator could quietly disagree with the eager one. - -- **Gotcha 7, text unanimity.** `aggregate_cell_values` checks the all-numeric branch - first. Otherwise one distinct value is that value, and anything else is - `Error("mixed")`. `Count` of a lone number is `1`, not the number. -- **Gotcha 9 and core.md §3, circularity.** A cell still moving after 16 passes is - written as `Error("circular")` (improvise-7iww). **A chain that needs all sixteen - passes is also reported circular**, because the fixed point can only confirm - convergence with a pass that changes nothing. The direct evaluator agrees because its - recursion depth is the same 16 (`types.rs:705-714`). -- **Gotcha 17, one epsilon.** `FLOAT_EQ_EPSILON = 1e-10` (`types.rs:14`) is shared by - `=`, `!=` and the div/0 guard, in both evaluators. -- **Gotcha 18, ordering is correctness.** A `HashSet` index is for lookup, never for - iteration. `SUM` and `AVG` drifted in the last bits when the store was walked in hash - order (improvise-shyt), because f64 addition is not associative. -- **Gotcha 40, the tiers are observed.** A write that lands the value already there - bumps nothing. The record set is collected into a *set*, so a rewritten record counts - once. +**The view/host split (ADR 008).** A client reads through `project(request)`, where a +request names axes, an absolute top-left and dimensions. It writes only through +`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys. +Aggregation is always done on the model side. A view's hiding, collapsing, page +selection, axis assignment and pruning are one client's ephemeral state until an +explicit save-view command. Only the number format is document-wide. ## Decision -**Formulas are evaluated incrementally. A change re-evaluates only the formula cells -whose inputs moved, over aggregates that are maintained rather than re-folded. The -machinery is a new, problem-agnostic workspace crate implementing the DBSP core. -improvise-core defines a trait for the derived state its evaluator reads. Today's -evaluator is one implementation of that trait, and a circuit built on the new crate is -the other. Both stay in the tree, and a differential test holds them equal.** Fourteen -clauses, plus clause 5a. It is numbered 5a so that no clause number or question number -cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14. +### 1. The crate: `zset` -### 1. The crate: `zset` (the user's answer to question 1) +A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on +`std`, so it is wasm32-clean by construction. It contains: +- **`ZSet`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an + abelian group. Weights use checked arithmetic, and a zero weight is removed. The group + laws are property-tested. +- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per + source and returns one per sink. There is one clock, one thread and no scheduler. +- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id + property-tested); the linear `map`, `filter` and `flat_map`; `index_by`; + `join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear + aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a + multiset per group. +- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a + resumable `step(deadline)`, so a rebuild can run as a sliced job. -Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** beyond -`std`, so it is wasm32-clean by construction. +Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one +clock (improvise applies batches in one total order); threads within one circuit (a host +with threads may run independent circuits on a pool); a query language. Spilling to disk +is not in the first version, but operator state sits behind one interface (a trace: look +up, read a range, apply a delta), so the store-memory work can page or compress it later. -**What it contains:** +### 2. The plug: a trait, and a seam crate -- **`ZSet`**: a map from `K: Ord + Clone` to a non-zero `i64` weight, with `add`, - `neg` and `zero`. It forms an abelian group, and its laws are property-tested as the - notebook's first cell does: commutativity, associativity, identity and inverse. - Weights use checked arithmetic, so an overflow is a panic, not a wrap. A weight that - reaches zero is removed, so equal Z-sets are equal as maps. -- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one input - delta per source and returns one output delta per sink. There is one clock, one - thread, and no scheduler. -- **Stream operators:** `delay` (z⁻¹), `integrate` (I) and `differentiate` (D), with - I∘D = D∘I = id tested as a property. -- **Linear operators, which are their own incremental form:** `map`, `filter` and - `flat_map`. -- **`index_by(key)`.** It turns a `ZSet` into an indexed Z-set `K → ZSet`, which - is the input shape for grouping and joins. -- **`join_indexed`.** The bilinear delta join: Δ(a ⋈ b) = Δa ⋈ Δb + I(a)₋₁ ⋈ Δb + - Δa ⋈ I(b)₋₁, over integrated indexes. It is needed for one purpose, clause 5's - cross-stem references. -- **`distinct`.** It is `H` over the integrated input, as in the notebook. The output - delta is non-empty exactly when an element's total weight crosses zero. -- **`aggregate(group, Aggregator)`.** Its output delta retracts the group's old value - and inserts its new one. The trait has two kinds of implementation: - - **linear:** `Count`, and `Sum` for any `G: AbelianGroup`. The state is one value - per group, and each delta element costs O(1). - - **integrated:** the aggregator keeps the group's integrated multiset, - `BTreeMap`, and reads its answer off it. `Min` and `Max` are the first and - last keys. Unanimity (clause 4) is "exactly one distinct key". Each delta element - costs O(log d), where d is the number of distinct values in the group. +improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the +formula result at a key, and the raw fold of the cells that match a partial key. It +takes the cell delta and the formula delta after each model effect. -**What it leaves out, deliberately:** +**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and +read through the trait. The circuit decides *which* formula cells to re-evaluate and +supplies their inputs. Formulas are not compiled into operators, because a compiled +`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the +defect. -- **Nested streams and least-fixed-point recursion** (the notebook's - `LiftedLiftedJoin`, `DeltaLiftedDeltaLiftedDistinct` and `IncrementalGraphReachability`). - improvise's circular semantics are not a least fixed point (clause 6). -- **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies - batches in one total order (ADR 001, "There is already a total order"). -- **Spilling to disk, compression, and threads, in the first version.** State lives in - memory, and a browser worker has one thread. **This was a non-goal and is now only a - deferral.** The user's store-memory requirement (ADR 008 clause 12) covers this state - too, and clause 14 says what the crate must not rule out. A host with threads may run - independent circuits on a pool (ADR 008 question 23's answer). One circuit's `step` - stays single-threaded. -- **A query language or planner.** Circuits are wired by hand in the seam crate - (clause 2). +**The incremental backend lives in a seam crate, `improvise-incremental`,** which +depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same +shape as `gp-tree` and `improvise-gp`. -**The weight type is `i64`.** Integer weights make `ZSet` an exact group, so -consolidation and retraction never drift. The drift problem is in the *values* being -summed, not in the weights (clause 8). +**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change, +as `Eager`. It stays as the oracle for the differential test while the circuit matures, +and for the two cases that still need it (clauses 5 and 10). The differential test is a +property test in the seam crate that applies one random edit sequence to both +backends and compares every drawn cell, total and occupancy count. The goal is that every +formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is +open question 15. -### 2. The plug: a trait in improvise-core, and a seam crate that implements it +### 3. The input: the cell delta -**improvise-core gains a trait for the derived state its evaluator reads.** The working -name is `Derived`. It replaces the two places where eager evaluation folds the store: - -- the formula result at a key, which is `formula_cache` today (`types.rs:370`); -- the raw fold of the cells matching a partial key under an aggregation, which is - `aggregate_cell_values(matching_values(..))` today (`types.rs:844`, `:1191`). - -It takes two inputs, both applied after each model effect (clause 7): the cell delta, -and the formula delta. It also has a rebuild, for a new shape key (clause 7). - -**The formula evaluator stays one.** `eval_expr` and `eval_expr_cached` keep their code -and read through the trait. The circuit decides *which* formula cells to re-evaluate -and supplies their inputs, and the existing per-cell evaluator computes the value. -Formulas are never compiled into operators. A compiled `IF`/`POW`/`WHERE` would be a -second implementation of evaluation, which design-principles §5 calls the defect -itself. It is also the reason ADR 008 revision 1 gave for rejecting incremental totals. - -**Two implementations:** - -- **`Eager`** is today's code moved behind the trait, with no behaviour change: `apply` - records the delta, and the post-batch refresh is `recompute_formulas`. -- **`Incremental`** lives in a seam crate, working name `improvise-incremental`, which - depends on `zset` and `improvise-core`. This is the `gp-tree`/`improvise-gp` shape: - improvise-core never depends on `zset`, and it can be built and tested eager-only - (question 2). - -**Why keep both, for now.** `Eager` is the oracle for the differential test. It was -also to be the backend a query-only source keeps (clause 10). **The user: "Eager should -be deprecated", and "For now: the goal is to only have DBSP formulas"** (questions 8 and -6). So `Eager` stays only as long as the differential test needs an oracle, and what -replaces it for query-only sources and for clause 5a's non-record-local formulas is -question 26. *As first written* (superseded), this paragraph and question 6 kept `Eager` -permanently. The differential test is a property test -in the seam crate. It generates documents with groups, collapse, hidden items, formulas -(including `WHERE`, aggregates inside formulas, chains and cycles), page filters, text -cells and non-finite numbers, applies the same random edit sequence to both backends, -and compares every drawn cell, every column total and every occupancy count. - -### 3. The input: a cell write is a retraction plus an insertion - -**The input stream is the *cell delta*: a `ZSet<(CellKey, CellValue, Seq)>` with -weights ±1, where `Seq` is the cell's store sequence (clause 5a).** `DataStore::set` and -`DataStore::remove` (`cell.rs:286`, `:325`) are the only places a cell changes, so the -delta is emitted there, each element carrying the sequence `s` of the cell it names: - -- a new cell is `+(k, v, s)`, with a fresh `s`; -- an overwrite of `v₀` with `v₁` is `−(k, v₀, s) + (k, v₁, s)`, the key keeping its `s`; +`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a +`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number +assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an +overwrite: +- a new cell is `+(k, v, s)`; +- an overwrite is `−(k, v₀, s) + (k, v₁, s)`; - a removal is `−(k, v₀, s)`; -- a remove-then-set within one effect is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. +- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. -**The `(key, value)` projection is derived, not a second stream.** The record tier -and any consumer that does not need order read the delta mapped through -`(k, v, s) ↦ (k, v)` and consolidated, a `ZSet<(CellKey, CellValue)>`. Only clause 5a's -stage reads `s`. Defining the canonical element without `s` would strip it, and a -same-value remove-then-set would cancel, defeating 5a. +**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their +strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an +equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a +change, and a NaN rewritten with the same bits is not. Multisets order values by +`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted +`Vec`, so deriving it is sound. -An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty -Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once -per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy -inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation -under the view/host split. The per-effect step stays as the circuit's own clock. What a -client counts is clause 12's publication number, which moves once per published batch -(T3). +**What the delta replaces.** The record tier reads the records named by the +consolidated `(key, value)` projection of the delta, instead of diffing two store scans. +The dirty flag's content half becomes "a non-empty projected delta, or a moved +formula-set fingerprint". The fingerprint is needed because formula edits move no cell, +and the worker persists on this flag. -**A delta key's identity is bit identity, the same identity the record tier uses -today.** Two elements have the same `(key, value)` part exactly when their keys are -equal and their values are equal *by bits*, and bit identity is stated over that part -only (the sequence is compared as the integer it is, which keeps an -equal-value overwrite empty): `Number` by `f64::to_bits`, `Text` and -`Error` by their strings. That is `hash_value` (`app.rs:246-262` at `50e1314d`), which -`record_fingerprints` already hashes. So the cell delta reproduces today's tier -exactly: +**The circuit steps once per model effect.** That keeps occupancy current inside the +effect loop. What a client counts is a separate publication number (clause 12). -- overwriting `0.0` with `-0.0` is a non-empty delta and bumps the record, as it does - today; -- overwriting a NaN with a NaN of the same bits is empty and bumps nothing, as today. +**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`, +the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a +rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set +fingerprint, taken today over `(raw, target, target_category)`, must then be taken over +the id form, or a rename reads as a formula change. -It is **not** `CellValue`'s derived `PartialEq`, under which `-0.0 == 0.0` and -`NaN != NaN`. `recompute_formulas` uses that one for its "did this cell move" test -(`types.rs:1010`), and it stays there. The ordering behind a multiset (clause 4) is -`f64::total_cmp`, whose equality *is* bit equality for every finite and non-finite -value. So the delta identity and the multiset ordering agree, and neither is -`PartialEq`. Step 3's tests pin both cases: signed zero, and a repeated same-bits -NaN. +### 4. Aggregates: linear where they can be -**This lines up with ADR 001's record tier with no extra rule.** The records a batch -moved are the `record_of` (`crates/improvise-core/src/model/record.rs:14`) of the keys -in the batch's consolidated delta. So: +**The group** is the cell key minus the None categories, which is the partial key +`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the +same None list, which removes `rederived_for`'s model clone. -- a write of the value already there moves nothing, which is gotcha 40's rule; -- a record gained, lost or rewritten is named once, because a Z-set is already a set of - keys. +| answer | state per group | cost per delta element | +|---|---|---| +| `Count` | a count of values and a count of numbers | O(1) | +| `Sum` | an exact accumulator (clause 8) | O(1) | +| `Avg` | `Sum` / `Count` | O(1) | +| `Min`, `Max` | a multiset of the numbers | O(log d) | +| unanimity and `mixed` | a multiset of all values | O(log d) | -`bump_changed_records` can then read the delta instead of diffing two -`record_fingerprints` scans (`app.rs:266-276` at `50e1314d`). The tier is still *observed*: the delta -comes from the mutation sites, not from an `EffectKind`. `layout_version` is unchanged, -because it observes `view_shape` and not cells. Nothing in this ADR reads or bumps it. -The user calls it "mostly a view fact", to be separated from model mutations (ADR 008 -Q39). ADR 001's layout tier is to be split (ADR 008 question 68), and this ADR relies -only on `document_generation` and the record tier. +Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules +for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential +test generates signed zeros and non-finite values to pin this. Only measures whose +aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups, +they hold at most one entry per stored cell. -**The dirty flag's content half does not move onto the cell delta alone.** -`document_content` (`app.rs:168-182` at `50e1314d`) hashes the formulas as well as the cells, and it -is re-read after every `CellData` effect (`app.rs:1119-1125` at `50e1314d`). `AddFormula`, -`RemoveFormula` and `ReplaceFormula` are `CellData` effects (`effect.rs:230`, `:247`, -`:268`) that emit no cell delta. The worker persists on this flag (gotcha 45). So "a -non-empty cell delta means dirty" would never save a formula edit in the browser. The -content half becomes: **a non-empty *projected* cell delta (the `(k, v)` projection, so a same-value -remove-then-set raises nothing), or a moved formula-set fingerprint**. -The fingerprint is the one clause 7 already reads after each model effect, so nothing -new is observed. Its cost is O(formulas), where `document_content` is O(cells) per -`CellData` effect. - -**`CellKey` gains `Ord`.** It is a sorted `Vec<(String, String)>` -(`cell.rs:10-11`), so deriving `Ord` is sound. `CellValue` holds an `f64` and is only -`PartialEq` (`cell.rs:116`), so multiset keys use a wrapper ordered by -`f64::total_cmp`. Clause 4 says where that differs from `==`. - -**Names and ADR 007 (added on 2026-10-05).** ADR 007 is accepted. It makes ids the keys -and names display-only, and in it a `CellKey` becomes sorted `(CategoryId, ItemId)` pairs. -The user said there: "This is a lot lower value if CellKeys, etc. don't refer to the new -id". Every key this ADR defines follows `CellKey`: the cell delta, `CellKey`'s `Ord`, the -stems, `record_of`, the group keys, the shape key's categories, and the output's tuple -identities (clause 12). So **under ADR 007 they are id-keyed with no change to this ADR, -and a rename emits no cell delta and rebuilds no operator.** It is one `display` write, -which the output carries as a relabelling, not as a retraction and an insertion. Two -places do not follow `CellKey`, and must move with ADR 007: -- **the formula fingerprint** (clause 7). It is taken over `(raw, target, - target_category)`, which are names today. Under ADR 007 it must be taken over the - formula's id-form, or a rename would read as a formula delta and re-evaluate every - affected target; -- **the cross-product order** (clause 12), which orders items by the category's own - order. That order is unchanged by a rename. - -Until ADR 007 lands, a rename is what it is today: a remove and an insert of every cell -under the old name, and this ADR's deltas say exactly that. - -### 4. Aggregates: linear where they can be, an integrated multiset where they cannot - -**The group** is the cell key with the view's `none_cats` removed. That is the partial -key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct -`none_cats` list, shared by every projection shape that has that list (clause 7). That replaces -`rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group -comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does -(`types.rs:717-722`). - -**What each group keeps:** - -| answer | state per group | cost per delta element | exact on f64? | -|---|---|---|---| -| `Count` | count of values, count of numbers | O(1) | yes (integers) | -| `Sum` | running sum (an exact accumulator under question 3's recommendation) | O(1) | **no** as an f64; **yes** as the accumulator (clause 8) | -| `Avg` | derived: `Sum / Count` | O(1) | no, through `Sum` | -| `Min`, `Max` | multiset of the numbers | O(log d) | yes (order-free) | -| unanimity and `mixed` | multiset of all values | O(log d) | yes (order-free) | - -Reading a group reproduces `aggregate_cell_values`'s branch order: - -- if the count of numbers equals the count of values, the numeric answer; -- else if the value multiset has exactly one distinct key, that value; -- else `Error("mixed")`. - -The empty group is absent. The formula-level `eval_agg` (`types.rs:55-87`) reads the -same state with its own emptiness rule, `0` for `Sum` and `Count`, at read time. - -**Memory.** Summed over all groups, the multisets hold at most one entry per stored -cell. So the state for one `none_cats` list is O(cells), on the order of the store it -summarizes. **Estimate, not measured:** a `BTreeMap` entry of about 40–60 bytes plus the -key's size gives a few megabytes at the 190k cells ADR 008 measured. Only the measures -whose `measure_agg` is `Min`, `Max`, or that hold text, need the multisets. A -`Sum`-only document keeps one number per group. - -**The multiset's identity is clause 3's bit identity, and eager's *reading* is not.** A -multiset keyed by `total_cmp` holds the values the store holds, by bits, and that is -correct for the state. But eager *reads* a group through `PartialEq` and `f64::min`: -`NaN != NaN`, so two NaNs in one group read as `mixed` today, `-0.0` and `0.0` read as -one value, and `f64::min` skips a NaN. So the aggregator's *read* reproduces eager's -rules explicitly and does not inherit `total_cmp`'s. The differential test generates -signed zeros and non-finite values (core.md §16) to pin it. - -#### Ordered aggregates: `LAST` and `FIRST` need state ordered by an item, not a value - -improvise-felo.7 asks for `LAST` and `FIRST` over a fold: the closing or opening value -of a stock price, say, across the months of a year. improvise-felo.8 asks for -semi-additive measures: `sum` across one category and `last` across another. Both are -open tickets with their own semantics to decide. What this ADR must say is whether its -per-group state can carry them. Today it cannot: the integrated state above is an -**unordered** multiset, keyed by the value itself. - -**The structure.** An ordered aggregate keeps, per group, a `BTreeMap` keyed by the -**ordering category's item order**, with the tie-break felo.7 decides. Each entry holds -the values at that position with their weights. `LAST` reads the entry at the greatest -key, and `FIRST` the least. - -- **It is the same shape as `Min`/`Max`.** Those also keep an integrated ordered map and - read an extreme; they order by the value, where `LAST`/`FIRST` order by the item's - position and read the value stored there. So one generic operator covers all four: - an integrated map keyed by `K`, read at an extreme. `Min`/`Max` instantiate it with - `K = value`, and `LAST`/`FIRST` with `K = (item position, tie-break)`. -- **Cost:** O(log d) per delta element, where d is the number of distinct positions in - the group, and one entry per contributing record. That is the same order as - `Min`/`Max`'s multisets. -- **What moves the order.** Adding an item in the middle of the ordering category, or - reordering its items, changes every stored position. *As first written*, this item - said a reorder of drawn items already rebuilds, through `view_shape`. **That is - superseded:** under clause 7's shape key the drawn order is a read-time parameter, so a - reorder rebuilds nothing, and a state keyed by position would go stale. The position of - a **hidden** item matters only if `LAST`/`FIRST` must order values that come from - hidden items. There are three keys: - - **position**, which goes stale on a reorder now that nothing rebuilds; - - **stable item identity** (ADR 007), with the order looked up at read time, which - makes a reorder free, but then a read costs O(d); - - an **order label**, which compares in the current order and is not renumbered by an - insert elsewhere. Reads stay O(log d). A reorder relabels the moved items, and - re-keys every group entry stored under them, at O(entries under the moved items). - - This is question 11. -- **Semi-additive measures (felo.8) are aggregates of aggregates.** For example, - `last` across `Date` and then `sum` across `Account`. The inner aggregate emits a - delta (retract the old value, insert the new), and the outer consumes it. So the - composition is incremental at O(log d) per stage. Which stage is inner is felo.8's - semantic decision, because sum-of-last and last-of-sum differ. The circuit takes - either order. -- **Exactness.** `LAST`/`FIRST` return a stored value unchanged, so they are exact on - f64 like `Min`/`Max`. A semi-additive `sum` stage still needs question 3's answer. +**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures, +improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an +extreme, keyed by the ordering category's order instead of by value. A semi-additive +measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the +order is keyed is open question 2. ### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it -**Stems are a `distinct`.** Today's stem set (`types.rs:986-997`) is every stored key -with the target category and each `none_cats` entry removed. That is -`distinct(map(project))` over the cell delta. A stem enters when its first cell arrives -and leaves when its last cell goes. +**Stems** are `distinct` over the cell delta, mapped to "key minus the target category +and the None categories". A formula counts at a stem only where its top-level `WHERE` +(if any) matches the stem's coordinate, or where raw data exists at stem + target. That +is exactly where `recompute_formulas` caches a value today, so pruning keeps its current +behaviour. -**The stem rule: a formula cell exists at a stem only where its value would be cached -today.** `recompute_formulas` caches a formula cell only when the cached evaluator -returns a value (`types.rs:1010-1014`). For a formula with a top-level `WHERE`, a stem -whose coordinate fails the filter falls back to `aggregate_raw` (`types.rs:1047-1055`), -which is absent when no raw data lies at the target key. So: +**A formula cell is re-evaluated when one of its inputs moved:** +- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`; +- for an `Agg` with no filter, the group at the stem itself; +- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat` + set to `Item`, which is another stem. This is the one join: stems indexed by "stem + minus `Cat`", joined with the aggregate's output delta. -- **a formula counts at a stem only where its top-level `WHERE` (if any) matches the - stem's coordinate, or where raw data exists at stem + target.** +The affected `(stem, formula)` pairs are ordered topologically over the formula graph and +handed to the existing evaluator. -This is a coordinate predicate plus one count lookup, never an evaluation. It is the -rule ADR 008 clause 3's occupancy uses for formula cells, and it keeps today's pruning. -The review of ADR 008 found the case that needs it: `Double = Revenue * 2 WHERE -Region = East` with data only at West is pruned today, and a rule that counted every -stem would have kept the column. +**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline: +1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it + retracts the record's previous snapshot and inserts the new one. The previous snapshot + is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. + So the stage reads the store and keeps no second copy of the values. +2. **A lifted map** to `(context key, order key, expr at record)`. The context key + removes the target and None categories, and at a fold or total the spanned categories, + mapping a collapsed member to its group. The order key is (context rank, the minimum + store sequence over the record's live cells), which is exactly the order eager + concatenates in. +3. **An inner `WHERE`** is a filter plus the join above. +4. **The group aggregate**, with an error channel kept beside it that reads the first + `Err` in order-key order, as eager does. -**A formula cell is re-evaluated when one of its inputs moved.** The inputs of formula -`T` at stem `s` are: +No cross product is ever built. The map is sound only when `expr` reads the record's own +cells. Other formulas take a fallback that evaluates over the **occupied contexts** only, +the contexts some record projects to. Because `agg_values` evaluates only at each +context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one +edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over +zero occupied contexts it yields the empty value instead. Formulas targeting a category +other than `_Measure` stay on the eager path until their order key is defined. Open +question 8 covers both. -- for each `Ref(R)`, the group `s.with(cat(R), R)`. `cat(R)` is resolved once per - formula set through `find_item_category`, including its `_Measure` fallback for - formula targets (`types.rs:725-740`). Whether `R` is derived (`formula_for`) is also - decided per formula set, so a ref to another formula becomes an edge between formula - nodes, and a ref to raw data becomes an edge from an aggregate group. -- for an `Agg` with no filter, the group at `s` itself, because `eval_agg` pools over - `context.without(target_category)`. -- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at - `s.with(Cat, Item)`. That is a different stem, so a delta at `Cat = Item` reaches - every stem that differs only in `Cat`. **This is the one join**: stems indexed by - "stem minus `Cat`", joined with the aggregate's output delta. +### 6. Circular references -The affected `(stem, formula)` pairs are consolidated, ordered topologically over the -**formula graph**, and handed to the existing evaluator. Its cached path already reads -refs through `formula_cache` and falls back to raw aggregation (`types.rs:1066-1086`). -With the trait, both reads go to the maintained state. +DBSP's recursion computes a least fixed point, and improvise's circularity is not one: +`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs +forever. The required answer is `Error("circular")`. The first version therefore decides +circularity outside the crate: +- the formula graph is rebuilt when the formula set changes, and its strongly connected + components are found; +- formulas outside every cycle are evaluated once per step, in topological order; +- a formula in a cycle has its affected cells evaluated by today's bounded fixed point, + restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still + convergent where `x` is false; +- the sixteen-pass bound is reproduced, so the two backends agree in the differential + test. -**A formula's own `WHERE`** is decided per cell, as today. A cell whose coordinate does -not match falls back to raw aggregation (`types.rs:1047-1055`), so its input is its own -group and nothing else. - -**An aggregate over an expression** is the one place a formula needs an expression -*inside* an operator. It has its own clause, 5a. - -### 5a. Aggregate over an expression: a lifted map feeding the group aggregate - -`SUM(Revenue * Price)` does not sum a stored measure. It computes a value **per -record**, `Revenue * Price` at that record, and pools those values across the records -of a **context**. It is the one place a formula needs an expression *inside* an -operator. This clause first states exactly what today's evaluator computes, because the -incremental form must compute the same thing. - -**What eager computes** (`eval_agg`, `types.rs:55-87`, and `agg_values`, `:89-139`): - -- **Contexts.** The formula cell's key is the context at a stem. At a fold or a column - total, the contexts are the members `contexts_of` lists through `Expansion::members` - (`types.rs:201-213`, `:905-916`), the cross product of the spanned categories' covered - items. That is the one O(grid) path in totals (ADR 008 clause 5). -- **An inner `WHERE` rewrites the context.** A filter `Cat = Item` *replaces* the - context's `Cat` coordinate (`c.with(&f.category, &f.item)`, `types.rs:69`, and again at - `:116`), and the distinct rewritten contexts are pooled once each. So the stem - `Region=West`, evaluating `SUM(x WHERE Region=East)`, pools **East's** records. -- **Pooling.** For each context, `partial = context.without(target_category)` - (`types.rs:97`). The context's member records are those `member_records` finds under - `partial`, in the store's record order (a record sits where its first cell sits). - The expression is evaluated at each, and the per-context value lists are concatenated - **in context order** (`types.rs:73-77`). So the order is (context rank, record store - order): expansion-major at a fold or total, and plain store order at a stem. -- **Errors.** `agg_values` collects `Result`s (`types.rs:126-136`), and so does - `eval_agg` (`:76`). So the aggregate's value is the **first** `Err` in that order, - if any member errs. - -**The design.** Four stages, each an operator from clause 1: - -1. **An integrated per-record stage.** A cell delta does not carry a record: when only - `Price` changes, the delta holds `−Price₀` and `+Price₁` but not the unchanged - `Revenue`, so the old mapped value cannot be computed from the delta alone. So the - first stage is `index_by(record_of)` over the cell delta, integrated. It keeps each - live record's cells, as `ZSet<(measure, value)>` per record, **each cell with its - store sequence** (below). On each step it emits, for every record the consolidated - delta touches, including a cell the expression never reads: - - a retraction of the record's **previous** snapshot, read from the integrated state - before the step (`I(x)₋₁`); - - an insertion of its **new** snapshot. - - **Several writes to one record in one effect** consolidate before this stage (clause - 3 steps once per model effect), so the record is retracted once and inserted once, - at its net state. Writes in separate effects are separate steps. A record whose - snapshot is unchanged after consolidation emits nothing. -2. **A lifted map over record snapshots.** For each snapshot, the map produces one - entry per context the record belongs to: `(context key, order key, expr at record)`. - The value is a `Result`, so an error is a mapped value like any other. `map` is linear, - so a retracted snapshot retracts exactly the entries it once produced. - - **The context key** is the record projected onto the context's shape: the target - category removed and each `none_cats` category removed, per `none_cats` list, - exactly as clause 4 defines "the group". At a fold or total, a collapsed group's - member is also mapped to its group, and each spanned category is removed. That is - the same projection occupancy uses (ADR 008 clause 3). - - **The order key** is (context rank, record store order). The context rank is the - record's position in `Expansion::members` order, and it is 0 at a stem. - - **The record store order is the minimum store sequence over the record's *live* - cells**, recomputed every time the record is touched, and carried in the snapshot. - Eager orders records by their **current** earliest live cell in `DataStore::cells` - (`member_records`), and `shift_remove` changes that when a record's leading cell - goes. So the stage keeps a **per-cell** sequence, assigned when a key is first - inserted: - - **the sequence is assigned where `IndexMap` assigns position, at `DataStore`'s - own mutation sites**, and each cell-delta element carries it. An overwrite - (`set` on a live key) keeps the key's sequence, as `IndexMap::insert` does. A - `remove` followed by a `set` of the same key moves the key to the end, even - within one effect, and gets a fresh sequence. Because the element carries the - sequence, the two cases stay distinct after consolidation: an overwrite is - `−(k, v₀, s) + (k, v₁, s)`, and a remove-then-set is - `−(k, v₀, s) + (k, v₁, s′)`. A stage that inferred the sequence from a - consolidated `(key, value)` delta could not tell them apart. The differential - test's generator includes a same-key remove-then-set within one effect; - - the record tier (clause 3) projects the delta to `(key, value)` before it - consolidates, so moving a cell without changing its value still bumps nothing, - as today's fingerprint does; - - losing a record's leading cell changes its minimum, so the record is re-emitted - with its new order key. - - `IndexMap`'s relative order is exactly this sequence order among live cells. - `DataStore::sort_by_key` (`cell.rs:269`, on entry into records mode) reorders the - store, so it reassigns the sequences, and as an operator-shape change it rebuilds - this stage. - - The review demonstrated why a per-*record* sequence is not enough, against eager at - `57d385e1`. With `S = SUM(Rev * 1)` and store order `A.Other=5, B.Rev=1, - C.Rev=1e16, A.Rev=1, D.Rev=-1e16`, eager reads 2.0. Removing `A.Other`, a cell the - expression never reads, moves A behind C, and eager reads 0.0. A sequence fixed - when A first appeared would still read 2.0. -3. **An inner `WHERE` is a filter plus clause 5's join, not a filter alone.** Records - with `Cat = Item` pass the filter, and are indexed by their context key **without** - `Cat`. A stem `s` then reads the group at `s` without `Cat`, through the one join of - clause 5 (stems indexed by "stem minus `Cat`"). A filter alone would put East's - records in East's group and leave West's read empty; the join is what makes West's - stem read East's pool, as the rewrite does today. -4. **The group aggregate** from clause 4: `Count`, `Sum`, `Min` or `Max` over the - mapped `Ok` values, with `Avg` derived. **The error channel** is kept beside it: per - group, the set of erring entries, ordered by order key. The aggregate reads the - first of them if the set is non-empty, reproducing today's first-`Err`. - -Member records are never listed and no cross product is built. A context in today's -cross product that holds no records contributed nothing to the pool, so pooling only -the records that exist gives the same answer. - -**This is not compiling formulas into operators.** The operators are the record stage, -the map, the index and the aggregate. The expression inside the map is still evaluated -by `eval_expr` at the record, exactly as `agg_values` calls it today through `eval_at` -(`types.rs:127-134`). The circuit decides *which* records to re-evaluate (those the -delta names), and the one evaluator computes each value. Clause 2's rule stands: there -is one evaluator. - -**The boundary.** The map is sound only when `expr` reads the record's own cells, so -that its value at a record changes only when that record changes. An inner expression -that reads another record, holds a nested aggregate, or refers to a formula whose value -depends on other records, is not a function of one record. **Revised on 2026-10-05:** -such a formula is not maintained, and at a fold or a total it does **not** enumerate -today's eager contexts. Those are `Expansion::members`, the cross product (ADR 008 clause -5, a finding of Copilot's on #308). At a stem, eager pools one context and there is no -product (`contexts_of`, `types.rs:916-925` at `50e1314d`). It takes ADR 008's -**store-driven fallback**, which evaluates the expression only over the **occupied -contexts**, the contexts some record projects to. The user accepted that recommendation, -as ADR 008's question 15 ("the recommendation is fine."). **The answer is the same as -eager's, with one edge.** `agg_values` evaluates the expression only at the -`member_records` of each context (`types.rs:47-53`, `:91-139`), so a context with no -record contributes no value, as the paragraph above says of the pipeline. **The edge:** -an inner bare `Expr::Ref` that names no item errors `ref:` at the top of -`agg_values`, for every context it is called with, records or not. Evaluated over -**zero** occupied contexts the call never happens, and the aggregate yields its empty -value instead of the error. ADR 008 revision 25 found this edge, and it is question 15's -whole difference. With at least one occupied context the error is the same. *The first -revision of 2026-10-05 said the fallback differs "wherever the expression is non-empty at -an empty context", and a later one said it never differs. Both were wrong.* Whether this -ADR's `Eager` uses the same enumeration is question 18. Two regressions pin the record stage against eager: -- one changes **one operand** of a two-operand expression (`Price` alone, under - `SUM(Revenue * Price)`); -- one **deletes a record's leading cell**, including a cell the expression never reads - (the `A.Other` case above), and checks both the value and which error is reported - first. - -**5a covers formulas whose target category is `_Measure`.** For a formula that targets -another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A record, -whose key `record_of` takes without `_Measure`, then belongs to the context only through -its cell under that measure, and eager orders it by *that* cell's position, not by the -record's first live cell. Until 5a defines that projection and its order key, such -formulas stay on the eager path, and the differential test pins that they do. **At a fold -or a total that eager path is `Expansion::members`, so while it lasts it blocks ADR 008's -retirement of `Expansion::members` (its migration step 6) for these formulas.** Sending them to the -occupied-contexts fallback above as well is question 18. - -**What stays eager until step 5, and what is bit-equal before step 6.** -- **Before step 5** (formulas), every aggregate over an expression is evaluated as - today: through `Expansion::members` at a fold or a total, and over the one context at a - stem. -- **From step 5**, the record stage, the map and the index are maintained, so the cross - product is gone. `Count`, `Min` and `Max` over the mapped values, and the error - channel, are read from the maintained state. They are order-free, or ordered by the - order key, and bit-exact on f64 (clause 8). -- **`Sum` and `Avg` over the mapped values are re-folded** from the group's maintained - values in **order-key order** until step 6. That is (context rank, record store - order), which is exactly the order eager concatenates in, so the re-fold is bit-equal - to eager at a stem, a fold and a total. It costs O(group log group) per change, - against O(grid). -- **From step 6** (on question 3's answer), `Sum` is maintained like any other. Under - question 3's recommendation (c) the order key no longer matters for `Sum`, because - the exact accumulator is order-free. - -### 6. Circular references: DBSP's recursion is not our semantics - -**The user, on questions 4 and 5:** the sixteen-pass bound is kept "for testing purposes -but eventually remove it because DBSP should fix the cycle issue", and the component -fixed point "For validation, but if there's a more natural DBSP-native way we should plan -on that." So what follows is the **current** plan, kept for testing and validation, and a -DBSP-native treatment of cycles is to be planned. The argument below, that DBSP's least -fixed point is not improvise's circular semantics, is the author's. It is the open part, -question 25. - -DBSP handles recursion by computing a **least fixed point**: nested streams iterate a -monotone operator, and `distinct` keeps the lattice finite, so reachability converges. -improvise has no such guarantee: - -- `A = B + 1` with `B = A + 1` has no fixed point at all; -- `X = X + 1` over a stored `X = 10` climbs forever; -- the required answer is `Error("circular")`, not a limit or a non-terminating step. - -So the crate has no recursion operator, and circularity is decided *outside* it: - -- **The formula graph** (target → the targets its refs name) is rebuilt when the formula - set changes, and its strongly connected components are found (Tarjan). Formulas - outside every cycle are evaluated in topological order, each exactly once per step. -- **A formula in a cycle, a self-loop included, has its affected cells evaluated by - today's bounded fixed point, restricted to that component.** A formula-level cycle is - not always circular at every cell. `A = IF(x, B, 1)` with `B = A` converges wherever - `x` is false. The per-cell fixed point is the only thing that gets those cells right, - and reusing it keeps `Error("circular")` exactly as it is. -- **The depth bound is part of the semantics today.** An acyclic chain that needs all - sixteen passes is reported circular by both eager evaluators (core.md §3). A - topological evaluator would compute it correctly, and so would disagree with eager. - Question 4 asks whether to reproduce the bound. The recommendation is to reproduce - it, by carrying each formula's topological depth, so the differential test compares - like with like. Changing the rule would be its own decision, for both backends at - once. +This serves testing and validation. It is to be replaced by a DBSP-native treatment of +cycles, whose shape is open question 14. ### 7. The circuit follows the document per model effect -Some changes alter what the circuit computes, not only its input. They are observed -**inside** `apply_effects`, after **every** `EffectKind::Model(_)` effect: the same -position as the frame and dirty observations (`app.rs:1036-1084`). They are not the -post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1124`). -That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3 -states the same rule. +**Operator state is selected by a shape key.** The key is exactly what changes which +cells feed which operator: +- the categories of each axis, by level; +- the None categories; +- the filter keys: each filter category with its selected item, and the hidden items of + every category on None, Page or Filter. Those items drop cells from the fold; +- the page **categories**, which key a slice per page (open question 1); +- `measure_agg`, which is model state, never taken from a request. -- **A shape key, and read-time parameters (revised on 2026-10-05).** Operator state is - selected by the **shape key** of a `project` request, not by the name of a view - (question 14). The key is exactly what changes which cells feed which operator: - - the categories of each axis, by level; - - the None categories; - - the filter keys: each `Axis::Filter` category with its selected item, and the hidden - items of every category on None, Page or Filter, as ADR 008 clause 3 also says. Such - an item drops cells from the fold, so it filters. A hidden item of a **page** - category is a filter key, not a read-time parameter; - - the page **categories**, which key the slices under question 7's recommendation; - - `measure_agg`. +**Everything else is a read-time parameter**, applied when output is read: the hidden +items of an axis category, collapsed groups and group membership, the drawn order of +items, the page selection, and prune. A request carries its own key and parameters from +its client's UI state, and a saved view only supplies defaults. - Everything else a request carries is a **read-time parameter**, applied when the - output is read, without touching occupancy or cell operators: - - the hidden items of an axis category. For the column totals, the hidden row-axis - set is subtracted at read time (see "Hiding a row moves the column totals" below); - - the collapsed groups, and the group membership of axis items; - - the drawn order of items; - - the page **selection**, which picks a slice; - - prune, which under client-side pruning is not the host's at all (clause 12). +**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn +items, the collapsed groups, the page selection and `prune_empty`, all of which are +read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit +neither reads nor depends on that tier, which is a view fact. Only `document_generation` +and the record tier matter to this ADR. - **`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72` at `50e1314d`) - hashes each category's drawn items in order (hidden items excluded), its collapsed - groups, its page selection and `prune_empty`. Those are all read-time parameters except - the axis assignment. A key built from `view_shape` would rebuild on every hide, collapse, - `AddItem`, reorder and page step, which is what item grain and question 7 exist to - avoid. `view_shape` keeps its one job, bumping `layout_version` (gotcha 40). That tier - is a view fact by the user's Q39 answer, and the circuit does not depend on it. +**Item grain.** Occupancy and per-column aggregates are keyed by the **items** of the +axis categories, never by a position. So collapsing, expanding, hiding, showing, +reordering and adding an axis item change no operator: the visible axis is a remap. +ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a +rebuild. A collapsed group's value is combined from its members when read. That is exact +for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it +costs O(members) per read (not measured). The exceptions are column totals (below). - **Where the key and the parameters come from (revised for ADR 008's 2026-10-06 - answers).** Hiding, collapsing, the page selection, axis assignment and pruning are - ephemeral UI state of one client until an explicit `SaveView` (Q46 to Q50). Only number - format is "global" (Q49). So a request carries its own key and parameters, taken from - that client's UI state. **`measure_agg` is the exception:** it is model state - (`Model::measure_agg`, `types.rs:370` at `50e1314d`), part of every shape key but read - from the model, never taken from a request (ADR 008 Q51 says the same). A saved - `.improv` view only supplies the **defaults** a client - starts from, and the circuit never stores a client's unsaved settings. Number format is - display-only (gotcha 8) and never reaches the circuit, except through clause 13's - widths if those were adopted. Whether every read-time parameter must travel in the - request (the user's Q51: "do the others need to actually travel across the wire?") is - question 24. A hide and a collapse change model-side aggregates (the totals, and a - collapsed group's fold), so the author reads them as needing to travel. - Requests with the same key share one set of operators, whichever clients sent them. The - live keys are a **cache**: a key enters when a request names it, and it is evicted once - no recent request has named it **and no live topic subscription pins it**. Every - eviction is announced in the next notice header (clause 12). Elsewhere in this ADR, "a shape" means a - shape key. +**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job +runner (8 ms per normal operation). Deltas that arrive during the build are buffered and +added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is +O(delta) and is never sliced. Each step's occupancy output carries the count of tuples +whose count crossed zero, so ADR 008's ordered index can choose between inserting and +rebuilding. Measurements against the budget are stated as ratios against an unrelated +operation on the same run, not as bare numbers. - *As first written* (superseded twice): the operator shape was `view_shape` plus group - membership and `measure_agg`, and a view whose shape moved had its operators rebuilt - **before the frame observation that follows this effect**. The first revision of - 2026-10-05 still listed hidden items and collapsed groups in the shape, and still - deferred to `view_shape` for a saved view. That contradicted the item-grain rule below - and question 7, as the review of `3698c680` found. -- **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column - aggregates are keyed by the **items** of the axis categories, together with the filter - and page keys. They are never keyed by a position in a level sequence. So a collapse, - expand, hide or show of an **axis** category's items changes no occupancy or cell - operator, and the visible axis is a remap of the item-grain state. **The exception is - the column totals:** a hide or show of a row-axis item changes them, and they subtract - the hidden rows when read (next bullet). ADR 008 clause 3 measured that remap at 2.3 ms - for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item, or - reordering items, changes no operator either: a new item tuple is just a new key, and - the order is a read-time parameter. Only a change of the **shape key** selects other - operators, which a new key builds as a job: - - a change of which categories are its levels or on None; - - a change of its filter keys, which includes hiding an item of a category on None, - Page or Filter; - - a change of its page categories; - - a change of `measure_agg`. +**Formula edits are an input.** After each model effect the host compares a fingerprint +of the formula set. If it moved, the formula delta is fed to the circuit, and the formula +graph is re-derived. A document swap (`install_workbook`, which bumps +`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and +no formula, so it touches no operator. - A collapsed group's value is a pooled fold over its members. With item grain it is - combined from the members' state when it is read. For `Count` that combine is exact. - For `Sum` it is exact only under question 3's (c), or under ADR 005's wide accumulator. - A sum of the members' f64 sums is not bit-equal to eager's one store-order fold - (`fold_stored`), so before step 6 a fold's `Sum` stays on eager's path. For the integrated aggregates it costs O(members) per read, and whether that - fits the frame budget at a 1,000-member group is **not measured**. -- **Hiding a row moves the column totals (the author's default, (A); question 23).** Today a - column total pools only the **drawn** rows. The total's evaluator excludes hidden items - through its fold memo (`layout.rs:797-803` at `50e1314d`), and `column_total` - evaluates over that memo (`:1066-1070`). The test - `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was` pins it. Take - `Month` {Jan 3, Feb 4, Apr 100} on Row and `Type` on Column: the total is 107. Hide - Feb, and today's total is 103. A per-(column tuple, measure) aggregate still holds 107, - and no read-time parameter can take Feb back out of it. **Whose hide this is** (ADR 008, - Q46 to Q50): a hide is one client's ephemeral UI state until an explicit `SaveView`, - and never a model mutation. So the options below concern a request's **unsaved** hidden - set. A saved view's hidden set arrives only through `SaveView`, as part of the saved - definition a request starts from, and it is subtracted the same way. So: - - **(B), withdrawn** (see "(B) has a defect" below): a hide or show of a **row-axis** - item rebuilds the column totals of that shape, as a job. ADR 008 once said the same - (its finding 10). It no longer does: its clause 5, at revision 28, says totals are - **not** rebuilt per shape key on a bare hide, and that the choice follows this ADR's - question 23. Occupancy and cells stay read-time remaps. Collapse and - expand do not move a total, because it pools the members either way, which is what - the test above pins. A hide or show of a column-axis item hides or shows that - column's total. Under a **collapsed** column group it also changes the group column's - cells and total, because the fold memo skips hidden items on every category. That is a - read-time recombination of the members' item-grain state, and nothing is rebuilt. - - **(A):** keep the per-(column tuple, measure) total state over **all** rows, and - subtract the hidden rows when the total is read. Clause 4 keeps, per group (a row - tuple and a column tuple at item grain), a count of values, a count of numbers, a - running sum and, where `measure_agg` or text needs it, a multiset of values with a - count per key. The column's state is the sum of its groups' states. So at read time - the H cells of the hidden rows in that column are taken out as a **temporary - overlay**, never written back: - - `Count`, and the two counts behind the numeric-first branch: subtract. O(H). - - `Min` and `Max`: walk from the extreme key, skipping each key whose whole count is - hidden. At most H keys are skipped, so O(H log d), with d the distinct keys. - - The one-distinct-value text rule, and `mixed`: count the keys whose whole count is - hidden, and subtract that from the column's number of distinct keys. O(H log d). - Eager's own reading rules (NaN under `PartialEq`, clause 4) apply after the overlay, - unchanged. - - `Sum`: subtract the hidden rows' sums. That is exact only under question 3's (c), - or ADR 005's wide accumulator, because f64 subtraction is not exact. Before step 6 - a `Sum` total stays on today's path anyway. `Avg` is `Sum / Count`. +**The shape cache.** Live shape keys are a cache. A key enters when a request names it, +and it is evicted when no recent request names it (open question 4). Every eviction is +announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm +by every client reading it. Idle readers make that likely, since a shape that clients +only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job +queue. - So (A) costs **O(H log d) per read for every aggregate**. It is exact, and it keeps no - per-set state. *As first written* (superseded), this item said `Min`, `Max` and the - text rule cost O(rows) per read. That overstated it, as the review of `ad60ad2b` - found. The argument above is the author's and is not measured. +**Column totals subtract hidden rows at read time.** Today a column total pools only the +drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`), +pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a +per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept +over all rows, and a read subtracts the request's H hidden cells in that column as a +temporary overlay: +- `Count` subtracts; +- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden; +- the text rule counts the distinct keys left; +- `Sum` subtracts through the exact accumulator. - **(B) has a defect: the key is shared and the hide is not.** Operator state is shared - by every request with the same shape key (above), and an unsaved hide is a parameter of - one client's request. If client X hides Feb and client Y, on the same key, does not, - (B)'s rebuild serves one of them a wrong total. So (B) only works in one of two forms: - - **(B′) the hidden row-axis set is part of the key, for totals only.** Occupancy and - cells stay shared under the shape key. Totals are kept per (shape key, hidden - row-axis set). Each distinct hidden set in use costs at most one O(cells) rebuild, as - a job, and one set of per-column aggregates. The bound is an upper one: an entry can - be derived from the all-rows state by (A)'s subtraction, made once and kept, at O(H) - per column, with `Sum` exact only under question 3's (c). So (B′) is (A) memoised. - The totals cache multiplies by the number of distinct hidden sets that recent - requests name, and it is evicted like any key (question 14). - - **(A)**, as above, needs no per-set state at all. +That is O(H log d) per read for every aggregate, with no per-client state. The shape key +is shared across clients and a hide is one client's, so a total cannot be rebuilt per +hide. A hidden column item under a collapsed column group changes that group column's +cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression +totals need their entries indexed by row tuple as well, so a read costs O(hidden +records). The alternative is open question 12. - **Aggregates over an expression** (clause 5a) follow the same split. A 5a total's - entries are keyed by context, with the row categories removed. To subtract hidden rows - under (A), the 5a stage's entries must also be indexed by row tuple, at O(entries) more - state, so a read costs O(hidden records). Under (B′), a 5a total per hidden set is its - own map and aggregate stage, at O(records) to build. +### 8. Floating point: exactness comes from the representation - *The author's lean, re-evaluated after the corrected cost:* **(A)**. A read costs - O(H log d), and a hidden set is usually a handful of rows, so a read costs about as - much as (B′)'s O(1) in practice (not measured). (A) keeps no per-set state, which is - what C2's "as stateless as possible" asks, and it never rebuilds or multiplies the - cache. (B′) is kept as a **memo** on top of (A), for a hidden set so large that H log - d per read would not fit the frame budget. The earlier lean to (B′) rested on the - overstated O(rows) cost, and it is **superseded**. Plain (B) without the hidden set in - the key stays **withdrawn**. The user decides it in question 23. -- **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it - cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a - 2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13). - So a rebuild is a resumable `step(deadline)` over an owned snapshot of its input. The - deltas that arrive meanwhile are buffered and added at install, which Z-set addition - makes exact. A circuit `step` on a cell delta is O(delta), and it is never sliced. - Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the - install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and - priority queue the user required. This ADR only makes its rebuilds steppable and - cancellable between slices. The 8 ms and 2 ms are the budget's own numbers, which the user - ruled. Any measurement this ADR later reports against them is stated as a ratio - against an unrelated operation on the same run, as ADR 008's question 13 requires. -- **`SaveView`, and the default view (Q46 to Q48).** `SaveView` is a model command that - writes a view definition, acknowledged by the model, and a document always has at - least a default view. **Neither touches the circuit.** A saved definition moves no - cell and no formula, so it emits no cell delta and no formula delta. Shape keys come - from requests, so saving builds no operator. The next request that names the saved - view simply starts from its defaults. The acknowledgement is ADR 008's - (`ServerFrame::Accepted`, its question 69), not this ADR's. -- **A crossing count (ADR 008's second request).** Each step's occupancy output carries - the number of tuples whose count crossed zero. ADR 008's ordered index uses it to - choose between an insertion and a rebuild by sorting. -- **The formula set is an input, not a rebuild.** After each model effect the host - compares a fingerprint of the formula set, over (`raw`, `target`, - `target_category`), with its previous read. If it moved, the formula delta (−the old - formulas, +the new ones) is fed to the circuit as an explicit input. The formula - layer then re-derives the formula graph (clause 6) and re-evaluates the affected - targets at their stems. This is observed, not declared by effect type, as the tiers - are. It is what catches a `ReplaceFormula` that keeps its target, which moves no - `view_shape`. -- **A document swap rebuilds every circuit** from the new store. `install_workbook` - bumps `document_generation`, and the same after-each-effect read sees it. +A sum maintained by `+v` and `−v` in f64 drifts from the eager fold. +`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max, +with values of up to five decimal places: -Re-reading the shape keys costs O(categories) per live key per model effect, and the -formula fingerprint O(formulas). Neither grows with the cells or the grid. Building a -new key's operators costs O(cells), once per key, as a job. `view_shape` is read only by -`layout_version` after the loop, as today, and the circuit reads neither. +| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact | +|---|---|---|---| +| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 | +| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 | +| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 | -**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first -written* (superseded by item grain): `AddItem` moved the operator shape, so the view was -rebuilt with East in its sequences before `SetCell`'s delta arrived. **Under item grain, -`AddItem` moves no operator.** `SetCell`'s delta inserts the item tuple `East` as a new -key, and only the visible axis, which is derived, gains a row. The case needs no -ordering rule at all. +Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively, +against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager. +At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer +differently on the two paths. Eager itself is already more than 1e-10 from the exact +decimal sum. -**Recommended: a page selection is not a rebuild (question 7).** Under the recommended -answer, the shape key includes the page categories and not the page selection, so `[` and `]` read another slice -of state that already exists. The held-key path stays O(visible) (improvise-748f's -constraint). The slices cost nothing extra, because the state is still bounded by the -stored cells, however many pages there are. Filter coordinates still filter; only page -coordinates key the slice. If the user answers the other way, a page change is an -operator-shape change like any other. +**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer +wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so +that those can be retracted. Adding and retracting are integer additions, so the +accumulator is a true abelian group, and it rounds once, on read. A weight is split into +base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example: +- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles; +- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again + with weights up to 2⁴⁰; +- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to + 1.1 µs per read; +- it is 568 bytes per group, against 8 for an f64. -### 8. Floating point: retraction drifts, and exactness comes from the representation +`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0, +where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they +become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and +the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum` +0.1.6 rounds about a quarter of single negative values wrongly and cannot retract +non-finite values. -f64 addition is not associative, so a sum maintained by `+v` and `−v` is not the sum -the eager fold computes. **Measured**, not asserted: -`cargo run --release -p improvise-core --example retraction_drift` -(`crates/improvise-core/examples/retraction_drift.rs`, added by this PR). It was run on -an Apple M4 Max, with seeds 1 and 2. The example keeps a running sum through N -retract-and-insert cycles. It compares the sum against the eager fold, -`evaluate_aggregated_f64`, and against the exact decimal sum, taken as an integer count -of 10⁻⁵ units rounded to f64 once. Each value is a signed whole number of 10⁻⁵ units: -up to 99,999 units, times a power of ten spread over 3 or 9 decades. So the smallest -values carry five decimal places, not the two a user usually types. Two-place input -would absorb differently, and it was **not** measured. +**Lossy operations happen at the edge.** Sums stay in the accumulator. Division, +non-integer `POW` and display rounding happen only where a value leaves the exact domain. +An exact value becomes an f64 through its decimal string, which is correctly rounded. -| store | magnitudes | after 1M cycles: naive vs eager | ≥ `FLOAT_EQ_EPSILON`? | eager vs exact | -|---|---|---|---|---| -| 1,000 cells | 3 decades | 6.4e-11 – 1.3e-10 (35–140 ulp) | in 1 of 2 seeds | ≤ 7.3e-12 | -| 18,000 cells | 3 decades | 1.7e-9 – 2.3e-9 (57–80 ulp) | yes | 1.5e-10 – 2.6e-10 | -| 18,000 cells | 9 decades | 2.3e-5 – 3.2e-4 (3–42 ulp) | yes | 2.4e-4 – 2.5e-4 | +**The accumulator does not make `=` against a typed decimal exact.** It sums the stored, +already-rounded f64s, so it can miss the typed total by up to about half an ulp of each +term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example: +`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's +exact decimals fix that, and they also cover per-cell arithmetic and typed values such as +money. -A fourth case holds 1e17, then adds 1,000 cells of 0.01, then retracts the 1e17. The -naive running sum reads **0.00**, and the eager fold and the exact sum both read -**10.00** (at `,.2`). The small values were absorbed and never came back. +### 9. What ADR 008 gets -A Neumaier-compensated running sum recovers that case exactly. In the random cases it -stays about as close to the exact sum as eager does, **but it is never bit-equal to -eager**, because eager folds in store order and is itself inexact. +- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple. + Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. + A fold row with a formula column but no stored cell under it, and the `Exact` case with + nothing on None, are **not** occupied, on purpose. +- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit + consumes it and never keeps a second order. +- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item + grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because + the host cannot compute a client's visible axis. +- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the + formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows). +- **Widths for text columns** (clause 13). -Four conclusions follow about a **floating-point** running sum: +### 10. Live sources -- **Bit-equality between an incremental and an eager f64 `Sum` is not attainable** by - any f64 running-sum technique, naive or compensated. So the differential test can - compare `Count`, `Min`, `Max`, unanimity and occupancy bit for bit, because they are - order-free, but it can compare an f64 `Sum` and `Avg` only with a tolerance. -- **The gap reaches the epsilon.** At 18k cells, the incremental sum and the eager sum - differ by at least `FLOAT_EQ_EPSILON` at 13 of the 16 checkpoints. So - `IF(SUM(..) = x, ..)` can answer differently on the two paths. That is a behaviour - difference, not a display one. -- **This is already a bug in eager, which the measurement found in passing.** With 3 - decades of magnitude, the eager 18k-cell sum is up to 5.8e-10 from the exact decimal - sum, and above 1e-10 at 7 of its 8 checkpoints. So `=` against a decimal a user typed - can already be wrong today. That is recorded here for improvise-l2b8. -- **Only an exact representation dissolves the problem.** The next two subsections give - two: an exact accumulator for f64 sums, and ADR 005's decimals. - -#### Does DBSP itself help with floating point? No. - -The user asked whether rebuilding the evaluator on DBSP brings any correctness benefit -for floating point. On its own, **none**: - -- **Only the Z-set weights are exact.** They are integers (clause 1). The *values* being - summed are whatever type the aggregator holds. -- **An f64 `Sum` maintained by retraction is worse than eager, not better.** The table - above is the evidence: it drifts further from the exact sum the longer it runs, and - the absorption case loses the answer entirely. - -**The benefits come from two other places.** -- **The representation.** An exact accumulator (next subsection), or ADR 005's - decimals summed through ADR 005's own wide accumulator, make `Sum` a true abelian - group, provided both backends use the same accumulator. Under ADR 005 as merged that - holds **per partition**: the 320-bit decimal partition and each fraction group. A read - that combines more than K = 8 distinct non-decimal denominators is inexact by ADR 005's - cutoff. Decimals alone do not: ADR - 005 revision 2 (PR #320) shows that `rust_decimal`'s own `+` is not associative at - its 28-digit bound, so the order of summation changes both the value and the - exactness. Then retraction is exact, incremental - equals eager bit for bit, summation order stops mattering (gotcha 18's order-dependent - `SUM` goes away as a class), and today's eager error beyond `FLOAT_EQ_EPSILON` goes - with it. -- **The differential test** between `Eager` and `Incremental` (clause 2) is itself a - correctness gain. Two independent evaluations of every formula cell catch evaluator - bugs in either one, which nothing in the tree does today. - -What DBSP does contribute is **placement**. Every aggregate is a named operator with a -chosen algebra, so the circuit states exactly where values enter an exact domain and -where they leave it. That is the next subsection's principle. - -#### An exact accumulator for f64 sums, and deferring the lossy operations - -**A superaccumulator sums f64 values exactly.** It is a fixed-point integer wide enough -for the whole f64 range: in units of 2⁻¹⁰⁷⁴, a finite f64 is a 53-bit integer shifted -by at most 2045 bits. This is the idea behind Kulisch accumulators and Radford Neal's -xsum, and Shewchuk's expansions (behind Python's `math.fsum`) are a related technique. -Adding or retracting a value is integer addition, so the accumulator is a true abelian -group over the finite f64s. **Rounding happens once, when the value is read.** - -**The group claim needs one stated bound.** A consolidated Z-set carries weights, not -just ±1: equal mapped values consolidate, so one element can arrive with a large weight. -The accumulator therefore splits any weight into chunks of at most 2³⁰, and counts -*magnitude* between carry propagations rather than calls. Each limb then stays below -2⁶² between carries, for any `i64` weight. The first version counted calls, and review -found it silently wrong at weight 2³³: it read `−4.24e-314` where the answer is -`1.82e-304`. The example now checks that case, and it reads the exact answer. - -The example's chunk loop is exact but costs O(|w| / 2³⁰), which is seconds for a weight -near `i64::MAX`. The version in `numtower` splits the weight into base-2³⁰ -digits instead and adds each as a shifted value. That is exact in the fixed-point -representation and O(1) per add. In practice a weight is bounded by a record count -anyway. Then: - -- `Sum` is the exact sum, rounded once. It is independent of order. -- `Avg` is the exact sum divided by the count, rounded once. It is correctly rounded - if the division is done on the exact value; dividing the rounded sum rounds twice, - and step 6 must say which it does. -- `Min` and `Max` are already exact. -- Non-finite values are counted (NaN, +∞, −∞), so they can be retracted too. - -**Measured**, in the same example, by `ExactSum` (158 lines, including its rounding and the weight bound): - -- **Maintained equals fresh, bit for bit.** At all 25 checkpoints above, an `ExactSum` - maintained through a million retract-and-insert cycles equals a fresh `ExactSum` over - the current values. That includes the absorption case, which reads exactly 10.00. - Against the exact *decimal* sum, it differs at 1 checkpoint of 25, by 9.1e-13. That - gap is the rounding of the typed decimals into f64, not the sum. -- **It is correct.** It was checked (not committed) against Python's `math.fsum` on - 20,000 random cases. The cases include subnormals, values near the overflow bound, - and an insert-then-retract of an equal-sized noise set. There were 0 mismatches; the - 1,596 cases where `fsum` overflows were checked against exact `Fraction` sums. After - the weight bound, it was checked again, on 20,000 cases with weights up to 2⁴⁰ - against exact `Fraction` sums: 0 mismatches. The review's own oracle run agreed. -- **Cost, Apple M4 Max, best of 3 over 10M values.** The example prints per single - update. - - This revision's three runs agreed: 2.0 ns per insert or retract, against 1.5 ns - for a plain f64 add in the same black-boxed loop, and 0.58 µs per read. - - The previous revision's run on the same machine gave 3.2 against 2.3 ns and 1.1 µs, - and the review's run gave 3.6 against 2.7 ns. - - So an update costs roughly 1.3× a plain add. A read, which is the rounding paid - once per group read, costs 0.6–1.1 µs. - - **568 bytes per group**, against 8 for an f64. - - The ranges below are between runs on one kind of machine; read your own run. - - The size is the real cost. A document with 100,000 `Sum` groups would hold about 57 MB - of accumulators. A sparse form that stores only the occupied span of limbs would be a - few limbs for typical data. **That is an estimate, not measured.** - -**No existing crate was usable.** `xsum` 0.1.6 (MIT, no dependencies, and it builds for -`wasm32-unknown-unknown`) was checked and rejected on three counts: - -- **It is wrong on a single negative value.** It rounds about a quarter of single - negative inputs to a neighbouring f64: 49,819 of 200,000 sampled values, all - negative, for example `-5890.88` read back as `-5890.880000000001`. -- It keeps one infinity and one NaN flag, not counts, so a non-finite value cannot be - retracted. -- Its accumulator is not `Clone`, and reading it takes `&mut self`. - -`accurate` 0.4.1 has four required dependencies, plus optional `rayon`. `fsum` 0.1.4 was found and not -evaluated. So the accumulator is about 150 lines of our own, with its own tests and a -differential test against an exact rational sum. **It lives in `numtower`, ADR 005's -numeric crate.** The user decided that on ADR 005 question 15, answering "(a)" on PR -#320 on 2026-09-25. This ADR's seam crate wraps it in a newtype for `zset`'s group -trait, so `zset` stays free of numbers. - -**Defer the lossy operations: the general principle.** Keep intermediate results in a -domain where the operation is exact, and apply the lossy operations at the edge: - -- sums go in the accumulator; -- decimal add, subtract and multiply are exact under ADR 005, within `rust_decimal`'s - 28-digit bound, which ADR 005 detects rather than silently rounds; -- division, `POW` with a non-integer exponent, and rounding for display happen only - where a value leaves the exact domain; -- **an exact value becomes an f64 through its decimal string and `str::parse::`**, - which is correctly rounded. `rust_decimal`'s `Decimal::to_f64` is not: ADR 005 - measured it disagreeing in 17,375 of 200,000 values. `ExactSum`'s own read is - correctly rounded by construction (clause 8's oracle checks). - -This is honest only about where the gain comes from. **The representation earns it, not -DBSP.** What the circuit adds is that the edges are explicit: an aggregate is a named -operator with a chosen algebra, so "where does rounding happen" has one answer per -operator rather than one per call site. - -#### What each exact domain covers - -| | exact accumulator | ADR 005 decimals | -|---|---|---| -| `SUM`/`AVG` over stored f64 cells | yes: exact sum, one rounding | yes, if cells are decimals | -| per-cell arithmetic (`Revenue - Cost`, `* 1.1`) | **no**: still f64 | yes | -| `=` against a typed decimal (`FLOAT_EQ_EPSILON`) | only while the **terms** are small: it sums the stored, already-rounded f64s exactly, so it can miss the typed total by up to about half an ulp of each term, and `=`'s absolute 1e-10 absorbs that only while the terms are below about 2¹⁹. A large-term, small-total sum misses visibly (see below) | yes, exactly | -| typed values such as money | no | yes | -| a value's own rounding when typed (`0.1`) | no: the f64 is already inexact | yes | -| cost | 568 bytes per group, 2.0–3.6 ns per update, 0.6–1.1 µs per read | per ADR 005 as merged (option (d)): a three-level tower, `Decimal` / `Fraction(Box>)` / `Inexact(f64)`, in `numtower`. `rust_decimal` alone was measured at +15.4 KB gzipped, and `num-rational` with `num-bigint` at about +20.8 KB on their own; the combination is unprobed. `numtower` is not yet in the workspace (implementation epic improvise-59nw). Its sum keeps a 320-bit decimal partition and a 256-bit numerator per fraction group, and a read combines at most K = 8 groups (the user's final answer; K = 32 was superseded). The 40 bytes per summed group cover the decimal partition only | - -**The accumulator does not make `=` against a typed decimal right.** It sums the -*stored* f64s exactly, and each stored f64 is already the rounded form of what the user -typed. So the exact sum of the stored values, rounded once, can differ from the typed -total, rounded once, by up to about half an ulp of each term. That is the last bit of -the largest term, which is more than the last bit of the total when the terms cancel. -- **ADR 005 measured it** (PR #320) using the exact rational of the stored f64s, - rounded once, as its stand-in. That is the value `ExactSum` computes, but it is not - `ExactSum` itself. On positive two-place values from 0.01 to 999,999.99, the result - missed the typed total in 21%, 16%, 7.5% and 2.5% of 2-, 3-, 10- and 100-term sums, - and in 0 of 8 seeds at 18k and 90k cells. On two-place values uniform in ±1,000.00, it - missed in 38–40% at every one of those term counts. -- **A spot check** of this ADR (not committed), on the ±1,000 values, agreed: about - 37–40% of 2- to 100-term sums missed. -- **This ADR's own example** missed in 1 of 25 large-sum checkpoints. - -So the miss rate depends on **how large the total is relative to its terms**, not on -the term count as such, nor on how large the values are. When the terms share a sign, the total -grows, and its last-bit step grows with it, so the terms' own rounding errors fall -below it and misses become rare. When the terms cancel, the total stays small, and -misses stay near 40%. - -**Where a miss is visible.** A miss is bounded by the **terms'** rounding, not by the -total's. Each stored term is off from its typed value by at most half an ulp of that -term, so the exact sum of the stored terms is off from the typed total by at most about -½·Σ ulp(term), plus the total's own half-ulp at the final rounding. So the misses are -last-bit **relative to the largest term**, not relative to the total. When terms cancel, -a miss can be far larger than the total's ulp. - -- **The large-term, small-total case (from ChatGPT Codex on this PR).** The terms are - `1000000000000000.10 + (-1000000000000000.00)`, and the typed total is 0.10. The - first term is stored as `1e15 + 0.125`, because the ulp at 1e15 is 0.125. So the - exact sum of the stored terms is **0.125**. The miss is 0.025, on a total of 0.1, and - `=` sees it. -- **The condition, stated on the terms.** `=` is an absolute `1e-10` (gotcha 17), so a - miss can reach it only when ½·Σ ulp(term) reaches `1e-10`. That needs at least one - term of about 2¹⁹ ≈ 5.2e5 or larger, or many terms just below it. It does not depend - on how large the total is. -- **The measured cases fit the condition.** On the ±1,000 values every term is far - below 2¹⁹, and the review counted 0 of 7,506, 0 of 7,901 and 0 of 7,963 misses - visible to `=` at 2, 10 and 100 terms. On positive values up to 999,999.99 the terms - themselves pass 2¹⁹, and most misses are visible. -- **Under ADR 005's epsilon-free `=`** (its question 3), every miss is visible, since - there is no tolerance at all. - -The misses show in the editor and to a raw comparison, and not in a display format -unless the terms are large enough, as in the case above. ADR 005's "once the magnitude -is above 2¹⁹" needed the same correction, to the terms' magnitude, and ADR 005 has -made it. Only exact decimal cells (ADR 005) make `=` against a typed total exact at -every magnitude. - -**So ADR 005 still matters**, for per-cell arithmetic and for typed values like money. -The accumulator covers only `SUM` and `AVG` over f64. But that is exactly the part this -ADR needs: it is what makes the maintained `Sum` bit-equal to eager, **without waiting -for ADR 005**, provided `Eager` uses the same accumulator. That was question 3's option -(c), and **the user chose it**: "use the exact accumulator." (4226373188). - -### 9. What ADR 008 gets from it - -ADR 008 (revision 37 at `c8be7720`, PR #308) assumes these, under the shared names below. -**Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact. -A consumer reads it through `project` (clause 12), and none of it is addressed to a -host-held mirror of a client. - -- **Occupancy** (ADR 008 clause 3). For each projection shape (clause 7), there is a count - per row tuple and per column tuple of cells that hold a value. The counts are kept at - **item grain**, and the visible axis is a remap of them. - - Stored cells pass the shape's filter. - - A hidden item of an axis category is masked only when the visible axis is derived. - - A collapsed group's members map to the group's tuple at read time. - - Formula cells count by clause 5's stem rule, which is when today's `formula_cache` - holds a value: a top-level `WHERE` must match, or raw data must exist at the target. - - Page coordinates key a slice, subject to question 7. - - A tuple is on the axis exactly when its count is above zero. **ADR 008's question 1 is - answered: "go with occupied"** (#308, comment 4171484686). So the two cases where - today's pruning keeps a row with no stored cell under it are dropped on purpose: a fold - row with a formula column, and `Aggregation::Exact`. Clause 5's stem rule already drops - them, there is no "non-occupancy path" for them, and the differential test carves them - out by name. *As first written*, this item said this ADR "does not settle" that - question; that is superseded. Occupancy is integer and order-free, so it does not wait - for ADR 005. -- **The ordered record list** (ADR 008 clause 1, `Records`). The user: "the ordered list - of records is a model-side concept" (Q27). It is the store's order (gotcha 18, - `DataStore::live_records`), and this ADR does **not** maintain it as a per-shape - operator. Clause 5a's record stage keys records by `record_of` and orders them by the - store sequence, so it *consumes* the store's order and never defines a second one. How - a page and filter select from that list is ADR 008's question 55. It is not this ADR's - to answer, because this ADR keeps no per-view copy of the list. Whether records mode is - `project` over the virtual axes (ADR 008 question 54) changes nothing here: occupancy - over `_Index` and `_Dim` is the same operator as over any other axis. -- **The occupancy delta** (ADR 008 clause 8, which the split revises). `distinct` over - occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible** - axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first - written*, its consumer was the per-view frame observation and its reset of each client's - cursor (gotcha 24). Under the split the host holds no cursor, and no client's read-time - parameters, so it cannot compute any client's **visible** axis. The delta is a - model-side output at item grain that keeps question 22's index current. Under question - 13's (b) it is **not pushed**. A client learns of a new or vanished row from the change - notice, recomputes its own visible axis, and shifts its grid so that the cursor keeps - its place (Q34). Only under (c) would a window's delta be pushed. -- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the - pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:1066-1070` at - `50e1314d`), - not a sum of row values. So the circuit keeps a per-(column tuple, raw measure) - aggregate at item grain over the drawn rows, and the formula layer re-evaluates only - the columns whose inputs moved, through clause 2's trait. A hidden row leaves the total - and the key is shared across clients, so under question 23's default (A) a total - subtracts the request's hidden rows when it is read, at O(H log d) (clause 7, "Hiding a - row moves the column totals"). Clause 5 replaces `Expansion::members`. Totals - that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: - the exact accumulator (recommended) or ADR 005 (clause 8). -- **What ADR 008 no longer asks.** Its question 6 is answered ("maintained - incrementally"). Its questions 1 and 2 are answered there too: 1 is "go with occupied", - and 2 is (b), sticky per session, with hysteresis as the easing (its question 22). *As - first written*, this item said they "stay open there", which had gone stale. -- **Widths.** A width depends on formatted text, which is display-only (gotcha 8). Under - the user's answer to ADR 008's question 2, widths are taken over the rows realized and - held by the view layer. The user's question whether DBSP can maintain a maximum width is - answered in clause 13. - -### 10. ADR 003's live sources are delta streams, when they can be - -A live source that can report its changes, by polling and diffing, by triggers, or by a -change feed, produces a cell delta. That is the same input `DataStore` produces, and the -circuit cannot tell them apart. That is the "data can stream" half of the user's -prompt. - -**A source that can only answer aggregate queries stays on `Eager` for now,** with ADR -003's pushdown. The user has said "Eager should be deprecated" (question 8), so this is -an interim, and its replacement is question 26. There is a tension: ADR 003 wants data larger than memory, and every -integrated operator here keeps state proportional to its input (clause 4). So the -backend is chosen per document, which is what "pluggable" buys (question 8). A -read-only live source also sidesteps ADR 001's record tier, as ADR 003 already notes. +A live source that can report its changes produces a cell delta, and the circuit cannot +tell it from `DataStore`'s. A source that can only answer aggregate queries stays on +`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated +state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so +the backend is chosen per document. ### 11. Constraints -- **wasm32.** `zset` has no dependencies. The seam crate adds only `zset`, - `improvise-core` and ADR 005's `numtower`, for `ExactSum`. If improvise-command selects the backend, all three join the browser - bundle, so they join the `wasm` CI job's per-crate list in the same PR that adds - them. Check with `cargo tree --edges normal`, per crate (repo map, "Wasm"). -- **Two seconds per test binary.** The crate's laws and its "incremental equals - from-scratch" properties run at the default 256 cases over small inputs. The - differential test lives in the seam crate's own binary, so its cost is visible on its - own. Its document sizes are bounded by the strategy, never by raising or lowering the - case count (design-principles §7). -- **Ordering determinism (gotcha 18).** A `ZSet` iterates in `K`'s `Ord` order, never - in hash order. A hash index is allowed for lookup only. Output deltas are - consolidated and sorted before they leave an operator. Nothing in the circuit decides - a display order: record order stays `DataStore`'s (`live_records`, `cell.rs:366`). - The determinism test builds **two** circuits independently from the same history and - compares them. One circuit compared with itself proves nothing (improvise-hzjo's - vacuity). -- **ADR 001 clause 4, rollback.** A refused frame is put back by `App::restore` - (`app.rs:693` at `50e1314d`). **Decided (question 9, the user: "The recommendation is fine, I think"): the backend's state joins the snapshot.** - Applying the inverse delta is **not** a rollback, even with exact aggregators. It - restores integrated values, but `step(d)` then `step(−d)` still advances the circuit - and changes the history that `delay` and `differentiate` hold. So the next step after - a "rewind" can differ from the next step after nothing happened. A real rewind would - have to restore every stateful operator, with a test of observational equivalence - across all of them. This ADR does not propose one. +- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then + `zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm` + CI job, each checked with `cargo tree --edges normal`. +- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over + small inputs, and the differential test lives in the seam crate's own binary. +- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and + sorted, and nothing in the circuit decides a display order. The determinism test builds + **two** circuits independently from one history. +- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that + `Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the + integrated values but not the history `delay` and `differentiate` hold. The cost is a + second O(state) clone per mutating frame, and peak memory to match. The keyboard path + takes no snapshot. +- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence. - **The snapshot costs a second O(cells) clone per mutating frame.** - `Host::mutating_frame` already clones the workbook on every mutating frame - (`host.rs:224-225`, `app.rs:673` at `50e1314d`), and the integrated state is of the same order. So - on the network path a write is not O(delta): it is O(cells) for the snapshot today, - and roughly twice that with the circuit in it. The keyboard path (`apply_effects` - without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the - store-memory requirement (clause 14), that second clone is also peak memory, which a - rewind would avoid. Question 9 stays open, and is now weighed against clause 14 as - well. -- **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the - delta sequence, so replaying commands reproduces its state. +### 12. Delivery: one circuit, in process or across a wire -### 12. Network transparency: one circuit, consumed in process or across a wire +The DBSP machinery is **network-transparent**: the same operators and streams whether a +consumer is in process or across a wire. -**The requirement is the user's** (C1): "The DBSP machinery needs to be implemented in a -network transparent way, but the details of that go to ADR 009". **Everything -else in this clause is this ADR's design, and the author's choices are marked as such. -None of it is measured.** +**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their +change notices, plus the input deltas of a remote live source. The write side carries +model commands with their basis, never deltas. The circuit's internal streams stay on +the host, because aggregation is model-side. No UI state crosses. -**What crosses a wire, and what does not** (the author's reading). Two boundaries can be -a wire. -- **The output boundary** carries the results of `project` and their changes: a window's - cells, its axis occupancy, its totals, and ADR 008 clause 4's width facts. In the TUI - the consumer is in the same process. In the browser it is the page, across `postMessage` - from the worker or a socket from `improvise-ws-server` (gotcha 45). -- **The input boundary of a live source** (clause 10). A source that reports its changes - delivers a cell delta, possibly from another machine. The circuit cannot tell that - delta from one `DataStore` emits. +**A request has a shape key and a window.** The key selects operator state, shared by +every request from any client. The window (absolute top-left and dimensions) selects +part of the output and is never state. So a request costs O(visible). -Three things do not cross: -- **The write side carries no deltas.** A client changes the model only by `apply` of - model commands (the user, Q43), so traffic from a client to the host stays - `ModelCommand`s with their basis (ADR 001), as today. -- **The circuit's internal streams stay on the host.** The cell delta, the per-record - stage and the group aggregates never cross: aggregation is model-side (the user, Q38), - so no operator runs in a client. -- **No UI state crosses.** The host is told the request and nothing about a cursor, a - scroll or a key (Q32, Q33, Q37). +**Positions are absolute cross-product positions**, in each category's own item order, +and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when +occupancy changes. They do move when an item is inserted, removed or reordered, so: +- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute + position attached when sent; +- an item edit is sent to every client as an **axis change** ("inserted at index j of + level L"). A client re-derives its positions at O(levels) each and re-anchors its + top-left to the same tuple; +- resolving positions needs rank and select over each category's items, which is ADR + 008's order-statistic structure. -**What a projection request is to the circuit** (the author's answer to "is the result -keyed by request, by axes, or shared?"). A request has two parts. -- **Its shape key** (clause 7) selects the circuit's state. The key is shared by every - request, from any client, that has it. Its read-time parameters (hidden and collapsed - axis items, drawn order, page selection) are applied to that state's output. -- **Its window**, the top-left and the dimensions, selects a part of that state's output. - The window is never state. +**Pruning's model-side support.** Pruning hides a row or column that is empty within the +displayed values, so a value outside the window does not count. That differs from +today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed +window, one pass that drops the empty rows and columns is already the fixpoint. A fill +then adds rows and columns forward from the absolute top-left, until the requested +dimensions are met or the axes end. Growth is only rightward and downward. For the fill, +the model answers one **restricted successor query**: "the next k rows after absolute +position p with a value in at least one column of S", with S in the call, and the same +for columns. A per-column index of occupied rows answers it: a sorted list per column, +seeked to p and merged forward with a linear minimum over the |S| cursors, at +O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the +cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a +B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's +whole state is its request and its last absolute position, so a resume is a seek. +**Scrolling up or left is a new request with a new top-left.** The new top-left is found +by galloping backwards in doubling jumps until a probe hits, then narrowing until one new +row is gained. Each probe is a restricted successor query bounded by the old top-left. +That is about 14 to 28 probes at density 10⁻⁴, and no special answer is needed when a +top-left lands on an empty row. Which side of the wire runs the prune and the fill is +open question 9. -So maintained results are keyed by **shape**, and the only per-request work is -restricting a shape's output to a window. That is O(visible) per request and needs -nothing kept between requests. +**The wire properties**, which hold on each hop: +- **T1. Plain data.** Output elements are serializable values that name cells by tuples, + positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`. +- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`), + so a value change that only a NaN payload distinguishes cannot be told apart remotely. + Which identity outputs use is open question 5. +- **T3. A publication stamp.** Each accepted batch is published once, stamped + `(host instance, document_generation, publication)`. The publication number increments + once per published batch, so a multi-effect batch is one header, and a refused frame + consumes none. The host instance changes on restart. A client drops anything at or + below the number it has applied. **The refetch rule:** a client that sees a gap, a new + `document_generation` or host instance, its shape in an eviction list, or a notice it + cannot interpret, refetches. +- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative + to its stamp. +- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and + flags, under the latest stamp. Axis changes are positional, so they stay an ordered + list. A `document_generation` change inside the run voids the merge. A basis is never + sent ahead of the cells it describes (gotcha 43). +- **T6. One API.** The TUI calls `project` and receives the same types as a remote + client. A test runs one edit sequence through an in-process consumer and through a + serialize-and-deserialize round trip, and requires equal results. -**Positions are cross-product positions** (C3). A request's top-left is a position in the -**cross product** of its axes, in each category's own item order. That position is a -bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed -contexts, we can represent bigints as strings in JSON"). A cross-product position names -the same tuple whatever is occupied, so **an occupancy change never moves a request's -window**. It changes only which positions inside the window hold data. **Which side of -the wire prunes is open.** C3 says the user is "still ambivalent", and C6 says pruning -"sounds like a UI side concern". That is question 19. The rest of this clause is the -author's analysis, and it is written for **client-side pruning** first, because that is -the author's lean after C6 to C8. Where model-side pruning would differ, it says so. +**The change notice.** After each accepted batch the host emits one notice, addressed to +no client. It carries: +- the stamp; +- the axis changes; +- display relabels (`(id, new display)` under ADR 007); +- a flag when the formula set or `measure_agg` moved; +- the eviction list; +- for each live shape key, the **moved output keys**: cells, formula cells and + column-tuple totals, as the circuit computes them. -**"As absolute as possible", and what still moves a position** (C4). *The author's -analysis.* A cross-product position does not move when occupancy changes, pruning -included. **It does move when an item is inserted, removed or reordered** in a category's -own order. Positions are mixed-radix numbers over the items of each level, so inserting -item j at level L shifts every later item of L, and changes the radix that every outer -position is multiplied by. No position over a changing item list is fully absolute. -Three things follow: -- **Deltas are keyed by identity, and positions are attached when sent.** An output - element names its row and column by tuple identity: item ids under ADR 007, and names - until then (see "Names and ADR 007" in clause 3). The host attaches the absolute - position from the current item order when it sends the element. A delta keyed by - position alone would go stale with the next item insert. -- **An item edit is sent to every consumer as an axis change.** "Item inserted at index - j of level L", or removed, or moved. Under question 13's (b) it rides on the change - notice, which every consumer receives. Under (c) it goes to **every** subscriber of a - shape with - that category on an axis, wherever its window lies, because an insert before a - window's top-left shifts every position that subscriber holds. A client re-derives its - positions arithmetically, at O(levels) per position, and re-anchors its top-left to the - same tuple, as ADR 008 clause 11 anchors the cursor. Resending every row whose position - moved would cost O(rows after j). -- **The cost is rank and select over each category's items.** "The position of item x" - and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec` - first, a B-tree when measured, its question 16). A tuple's position is O(levels × log - items). An index ordered by position must compare tuples by **current** item position, - not by a position stored at insert time. With item grain (clause 7) its keys are item - identities, so an item insert re-orders nothing in it, and a comparison goes through - the order-statistic lookup, at O(levels × log items). That is **not measured**. - Question 20. +Moved records alone would not do. From a moved record, a client cannot tell whether a +formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue +WHERE Region=East)` on page `Region=West` moves when an East record is written. A client +intersects the moved keys with its window by identity. A collapsed fold row has no +maintained key, so a moved member key goes through the client's own member-to-group +remap first. The client then refetches what moved, by `project` or by the successor +query. -**Pruning as a filter on the displayed values** (C6, C7). **This is a change from today's -code**, and it is recorded here, not decided. Today, `prune_empty` drops a row only when -it is empty across the **whole** grid, and prunes the columns against the same unpruned -grid (`layout.rs:752-757` at `50e1314d`). The user's intent, as ADR 008 reads it (its -questions 60 to 65), is that emptiness is judged over the **displayed** values, so a row -that holds a value only in a column outside the window is hidden. *The rest of this -paragraph is the author's analysis.* -- **What the model owes a client** is then not a pruned axis. It owes cross-product - windows (an absolute top-left and dimensions) and a **restricted successor query**: - "the next k rows after absolute position p that hold a value in at least one column of - the set S", and the same with rows and columns swapped. **S travels in the call**, - because only the client knows which columns it displays. A per-axis count cannot - answer it: a row can be occupied overall and still be empty in every column the client - shows. -- **Three ways to answer it**, all unmeasured: - - (i) **Per-axis occupancy as a pre-filter.** Skip rows that are empty overall, then - test each candidate against S by lookup. The state is the per-axis counts clause 9 - keeps. The worst case scans many rows that are occupied overall but empty in S, at - |S| lookups each. - - (ii) **A column index of occupied rows**: for each column tuple, the sorted rows that - hold a value in it. The query seeks each of S's lists to p and merges them forward, - choosing the minimum by a linear scan over the |S| cursors: O(|S| × log n + k × |S|). - The state is O(occupied cells), the two-dimensional count clause 9 does not keep. It - is maintained from the cell delta and from formula stems (clause 5's stem rule decides - when a formula cell holds a value). A collapsed group's column merges its members' - lists when it is read. - - (iii) **A row index of occupied columns**: for each row tuple, its occupied columns, - tested against S as the scan walks forward. It has the same state as (ii), and it - scans like (i), with an exact test per row. +**Every client receives every notice**, and the host holds no per-client state for +delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists +exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a +refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, +`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and +`GridExtent` is derived by the client from category lists. In records mode the row +extent is the admitted record count, which the client fetches. Whether notices should be +filtered per client is open question 3. - The author leans to (ii): a window has tens of columns, so the merge is small, and it - never walks rows that are empty in the window. Its cost is a second O(occupied cells) - structure beside the store (clause 14). Index (ii) is a set of `(column, row)` pairs, - so it is itself a Z-set fed from the cell delta and from formula stems. A write costs - O(log n), and its delta, "row r gained or lost a value in column c", is what a - subscription restricted to S receives. -- **Under model-side pruning instead**, the model would keep a pruned axis per shape: ADR - 008 clause 3's ordered index over occupied tuples, with rank and select for "row i of - n". Pruning would be a request parameter (ADR 008's flagged reading 1), and the host - would judge "pruned". Everything below that mentions a host judging emptiness applies - only to this alternative. +**A connection operator, at the wire boundary.** One component owns both ends of a +connection. Its host half is a sink for the circuit's results and notices and a source of +requests and model commands. Its client half is the mirror image: a source of results to +the client, and a sink for the client's requests. It owns: +- serialization (T1, T2); +- stamps and the refetch rule (T3); +- coalescing (T5); +- keep-alive in both directions. A client that hears neither a header nor a ping within + the timeout treats the connection as dropped, so silence is never read as "no + batches"; +- session resumption. -**The fixpoint comes from the fill, not from one window** (C8). *The author reads C8* as -a fixpoint on the displayed grid: every visible row has a value among the visible -columns, and every visible column has one among the visible rows. C8 says "drop out -categories", which the author reads as dropping rows and columns, that is, items or -tuples. The user may mean dropping a whole category from an axis. ADR 008 flags the same -ambiguity (its question 66(a)), and it is the user's to settle. *As first written*, this -clause said that hiding a row can empty a column. **That was wrong.** A row dropped for -having no value among the window's columns contributes no value to any of those columns. -So on a fixed window, one pass that drops the rows and columns empty within the window -already is the fixpoint, which is unique (ADR 008's question 61 makes the same argument). -The iteration comes from the fill step, which brings in rows and columns from outside the -window (ADR 008 question 66). A client alternates three steps: -- it fetches a window; -- it prunes, in one pass; -- it fills, asking for the next rows past its last absolute row position with a value - in its displayed columns, and likewise for columns. +In process, both halves are function calls. That covers the TUI, and the browser's +`Worker`, which already owns one `Host` and one port (`worker.rs:14-15`). The host half +belongs in or above `improvise-incremental`, and the wire types in `improvise-protocol`. +`zset` stays free of serde. The client half is shared by every front end, beside +`improvise-protocol`'s `ClientState`, so that a new front end does not write it again. +Its remaining shape is open question 17. -The model never re-runs a client's fixpoint. Under question 13's (b) a client learns from -the change notice which output keys moved, and re-asks. Under (c), a subscribed client receives index (ii)'s -deltas for its S, prunes again locally, and fills again only when a row or column has -gone. Whether the loop always terminates is **not proven**. The fill only moves forward -through absolute positions, so it adds finitely many candidates before the axis ends. A -client that bounds its fills per frame stays within ADR 008's frame budget whether or not -the loop has converged, and that bound is client policy (Q42). +### 13. Column widths for text -**Growth is only rightward and downward** (C9). *The author's analysis.* The fill only -grows away from the absolute top-left, and it stops at the end of each axis. So **the -fill** only ever asks successor queries: "the next row (or column) after p with a value -in S", bounded by the end of the axis. Scrolling up or left is not a reverse fill: it is -a **new request with a new absolute top-left** (C10), and it is the one place a backward -search appears (next paragraph). What forward-only growth simplifies: -- **The index options.** Each needs only a seek to p and forward iteration, so any - ordered structure with a lower-bound seek and a forward iterator suffices: the sorted - `Vec`, or a B-tree when measured (ADR 008 question 16). Under client-side pruning no - rank over **occupied tuples**, and no "row i of n", is needed. Rank and select over - **each category's items** stay, because a cross-product position is built from them - (above, and ADR 008 revision 25). A scrollbar over the unpruned cross product gets its - rank by that arithmetic. -- **Maintenance.** The range test that follows applies to **occupancy and cell deltas - only**, and to their moved keys in a question-13 (b) notice. A formula cell inside the - range moves when any of its inputs move, wherever those sit, so it is tested by the - moved output keys, never by the position of the record that moved. An item edit is an axis change and goes to every consumer (above). A change at - position q matters to a fill only if q lies between its top-left and its front. A fill - whose front is before q finds the change when it gets there. -- **Resumable fills.** A fill's whole state is its request and its last absolute position - on each axis. A resume is a seek past that position, so a lost stream, a reconnect or a - sliced job (ADR 008 clause 11) all resume the same way, with nothing else kept. If a - column's list changes during a merge, a re-seek from the last position is enough. An - item insert before the front shifts positions, and the axis-change message above lets - the client re-derive its front. -- **The corner truncates the work.** A fill is bounded by the requested dimensions and by - the bottom-right corner of the axis space. One round costs O(|S| × log n + k × |S|), - never a scan of empty space past the last occupied position. +The host maintains each text column's maximum display width incrementally. A maximum is +not invertible, so it is an integrated aggregate. Display widths are small integers, so +the state is a histogram from width to count, per (shape key, column): O(distinct widths), +at O(log d) per update. Its input is the delta of the column's text cells over every row, +which the circuit already produces for unfolded cells. Fold rows need the fold maintained +at the visible grain. A text width does not depend on the number format, so there is no +rebuild when the format changes. A client's hidden rows come out through clause 7's overlay. +Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized, +held by the view layer. A number's width would need formatting on the write path, and a +rebuild per format change. Which columns count as text is open question 16. -**A pruned answer as a stream** (C5). Under client-side pruning the fill is the client's, -and each step is one restricted successor query, so the host keeps no fill state at all. -Under model-side pruning, `project` itself would scan forward from the top-left and send -rows and columns as it finds them, as a sliced job resumable from `(request, fill -front)`. Either way, changes during a fill are identity-keyed deltas stamped by publication number -(T3). Suppose a row becomes occupied in S after the fill has passed it. It arrives as an -ordinary delta with its absolute position, and the client keeps or drops the extra row by -its own policy (Q42). So a fill and a live update are one stream, which keeps the -in-process and the remote consumer the same (T6). Question 21. +### 14. Resident state -**Scrolling up or left, and a top-left in empty space** (C10 to C12). -- **The baseline is the user's galloping search** (C12; "the simplest might be"). *The - author's reading:* the client moves its top-left up (or left) in growing jumps until a - request hits, then narrows back until exactly one new row (or column) is gained. **Each - probe is a restricted successor query** from the candidate top-left t, with the client's - current S, bounded by its old top-left: "is there a row in [t, old top-left) with a - value in S, and which?" It is not a bare cross-product `project`, which would return - mostly empty rows. -- **What it requires:** the response reports the **actual** absolute position of every - row and column returned, which T1 already requires. Under client-side pruning a - top-left that lands on an empty row is no edge case for the host. The host returns the - cross-product window or the successor answer, and the client's own prune decides what - shows. -- **Cost** (arithmetic, not measured). Doubling jumps take about log₂(gap) probes, and a - binary narrowing about log₂(gap) more. At density 10^-4 the gap is about 10^4 rows, - so 14 + 14 ≈ 28 probes. It is often about 14, because a probe returns up to k rows with - their positions. If the hit's rows reach the old top-left, the client reads the answer - directly. Under concurrent writes neither phase can run forever: the gallop ends at the - top of the axis, and the narrowing is a bounded bisection. -- **Statelessness and transparency.** The host keeps nothing for the search, and the - in-process and remote consumers make the same calls (T6). The cost is round trips: about - 14 over a socket for one scroll step into a sparse stretch, and negligible in the TUI. -- **Mixed cases** (the author's). A diagonal scroll moves both axes. The client gallops - one axis at a time. It gallops rows against its current displayed columns, then - columns against the rows it now displays, then prunes once. When only the row axis's - new top-left is empty, only the row gallop does work. When both are empty, both do, in - turn. Whether rows-then-columns and columns-then-rows always give the same window is - **not proven**. ADR 008's question 61(b) notes that order matters for a one-sided - pass, and the order is client policy. -- **A predecessor query is an optional optimisation.** The model could answer "the - previous row before p with a value in S" in one call. It costs no new state under any - of the three options: a sorted `Vec` and a B-tree both seek an upper bound and iterate - in reverse at the same cost, option (ii)'s lists merge backwards as they merge - forwards, and options (i) and (iii) walk backwards with the same test. *The author's - lean:* offer it only once a measured round-trip cost shows that one call is worth a - second query direction. -- *Superseded:* earlier text in this clause leaned to offering the predecessor query - outright, and then called that lean "stronger" once options (a) and (c) below needed - it. The galloping baseline replaces both leans. Options (a) to (c) are kept below as - a record. +Beside the store, the circuit keeps: +- each cell's store sequence, for clause 5's per-record stage (one integer per cell, + kept in `DataStore`); +- the multisets for `Min`, `Max` and text, O(cells) of those measures; +- `ExactSum` at 568 bytes per summed group; +- occupancy per live shape, O(occupied tuples); +- the restricted-query index, O(occupied cells); +- the width histograms; +- the rollback snapshot per mutating frame. -**Superseded: the pruned top-left under model-side pruning.** These options assume the -host judges "pruned", which holds only under model-side pruning. They are kept so that -the user's objection is recorded with what it answered. -- (a) **Fail, then retry** (C10's tentative "maybe"). The error names the nearest - occupied successor and predecessor. -- (b) **Snap forward and report the actual top-left.** **The user objects** (C11). *The - author's reading of the reason:* a client scrolling up asks for a top-left just above - its window. If that row is empty, (b) snaps forward to the window's old top-left, so the - window never moves past an empty stretch. -- (c) **Snap in the scroll direction.** It needs the request to carry its direction. +The store itself is the integrated base that a rebuild reads, so it is never copied. +State is evicted with its shape, and it sits behind one trace interface, so the +store-memory work (offload to disk, compression) can page or compress it later. None of +these sizes is measured at ADR 008's fixture. Migration step 4 measures them. -Neither (a) nor (c) can get stuck. A retry under (a) goes to a named neighbour that is -strictly beyond the old top-left in the scroll direction, and occupied. A write that -empties it only delays the search, by at most the gap. (c) moves in the direction asked, -and stays put only at the end of the axis. At the top of the axis, (a) names only the -successor and reports that there is no predecessor, and (c) returns the first occupied -position and reports that it is the top. So neither bounces. An axis with nothing -occupied answers "empty axis". Both need the predecessor query. +### Relations to other ADRs -**What network transparency requires** (T1 to T6, the author's): -- **T1. Results are plain data.** Every element on an output stream is a value the - protocol crate can serialize, with no `Rc`, no borrowed store key and no closure. Facts - at `50e1314d`: `CellValue` is `Serialize`, with numbers going through `wire_f64` - (`crates/improvise-core/src/model/cell.rs:86-118`). `CellKey` is **not**: it derives only - `Debug, Clone, PartialEq, Eq, Hash` (`cell.rs:10`). So an output element names a cell by - the request's row and column tuples, or by cross-product positions. Where a write may - follow, it also carries ADR 002's surrogate (`CellId`). It never names a cell by - `CellKey`. -- **T2. One identity on both sides of the wire.** A consumer across a wire must - consolidate exactly as an in-process one does. **On NaN they would not.** Clause 3's - delta identity is bit identity, under which two NaNs with different payloads differ. But - the wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114` at `50e1314d`), and the - projection compares values with `same_on_the_wire` (`crates/improvise-protocol/src/render.rs:173-180` - at `50e1314d`; `context/code/protocol.md` gives the reason). So an output delta - `−(c, NaN₁) + (c, NaN₂)` is non-empty in process and consolidates to nothing after a - round trip. **Proposed:** output streams are consolidated under the wire's identity - (every NaN is one value, and every other number is its bits) before they leave the host. - The record tier keeps bit identity, as it does today. Question 15. -- **T3. Every published batch is stamped, by a publication sequence.** The stamp is - `(host instance, document_generation, publication)`. - - **The publication number** increments once per **published batch**, and that is - what a client counts. The circuit's internal `step` advances once per model effect - (clause 3), and it is never a client-visible gap detector. *As first written* - (superseded), the stamp carried `step`. A batch of two model effects then advanced - from n to n+2 under one header, and a client read the missing n+1 as a loss (found by - ChatGPT Codex on `70ababfe`). *The author's lean* is this separate sequence (i). The - alternatives: (ii) a header per model-effect step, which multiplies headers and - exposes a batch's internal steps, which ADR 001 treats as one atomic unit; or (iii) a - header that carries the previous published stamp, so a client can tell a - consolidated range from a loss, at a little more per header. - - **The host instance** is an id chosen when the host starts. The other two counters - live in memory and start again at their initial values after a restart. Without the - instance id, a reconnected client would drop the new host's notices as duplicates. - - ADR 001 already puts batches in one total order. A consumer that has applied - publication n, of the same instance and generation, drops anything at or below n, so - a duplicate does no harm, and it treats a gap as a loss, which a fresh snapshot - answers. - - Under question 13's (b′), every client receives every batch's **header** (clause 12), - so the publication numbers it sees are contiguous, and a gap is always a real loss. - Contiguous numbers prove only that no header was lost, not that a key list was routed - to the client. That holds only because a subscription is registered atomically with - its fetch (clause 12). - - **The general rule:** a client that misses a notice, sees a gap, sees - `document_generation` change, finds its shape in an eviction list, or receives a - notice it cannot interpret, **refetches**. A changed host instance is such a - discontinuity, so a reconnect after a restart always refetches. The same rule holds - for a client under (a), (b) and (c). - - What is published is the **batch's** consolidated delta, and only once the batch is - accepted. A refused frame is restored (clause 11), publishes nothing, and consumes no - publication number. -- **T4. A snapshot, then deltas** (for question 13's (c); under (b) a fetch is the - snapshot, and the notices carry the stamps and the moved keys). A subscription starts with the integrated result at - publication n: the window as a Z-set of current values, each with weight +1. Deltas - from publication n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current - result. `GridCache::folded(&cache.seed()) == *cache` states the same property for - today's bootstrap compaction (gotcha 32). -- **T5. Order holds across streams, and coalescing happens within a stream** (as stated, - for question 13's (c). Under (b) the equivalent is coalescing notices: a slow client - merges pending notices into one, taking the union of their moved keys and flags and the - latest publication number, and refetches once. **Axis changes do not merge.** They are positional, - so they are kept as a list and applied in their original order. A - `document_generation` change inside the merged run voids the merge, because the old - keys belong to a document that is gone, and the client refetches from scratch). A slow - consumer's pending deltas on one stream can be **summed** into one delta. Z-set - addition is commutative and associative, so the sum is exact, and the queue is bounded - by the size of the result, not by the number of steps. When the summed delta outgrows - the snapshot, a snapshot replaces it. That is the backpressure, and nothing is dropped - silently. **Across streams, order stays load-bearing**, as gotchas 33 and 43 require: - the occupancy change comes before the cells it places, and the cells come before the - basis that describes them. This does not break gotcha 42's "never coalesce". That rule - is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple - identity, or by a stable cross-product position, is not positional in that sense. - **Coalescing is therefore per subscription, never per stream.** All of a - subscription's pending deltas, the basis included, are summed together up to one - publication number, - and sent as one batch in the fixed order. Coalescing one stream while another is sent as - it arrives would let a basis for publication n+2 overtake the cells it describes, which is - gotcha 43's failure. -- **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives - the same result and delta types as a remote client. Its transport is a function call. - That is what makes the requirement testable: run one edit sequence through two - consumers, one in process and one behind a serialize and deserialize round trip, and - require their integrated results to be equal under T2's identity. A TUI shortcut that - read operator state directly would be a second read path, which is the derivation-twice - defect (design-principles §5). - -Weights on the output are ±1, because a result is a set of current values. A consolidated -weight that does reach past 2^53 follows the user's bigint-as-string rule (Q36). - -**Where the wire form lives** (the author's). `zset` stays dependent on `std` alone (clause 1). -The wire types live in `improvise-protocol`, with the one envelope pair (gotcha 42). A -consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that boundary. -The alternative is a `serde` feature on `zset`. The author leans to the conversion, -because what crosses is the projected result, not the crate's own element type. - -**How stateless the host can be, and what replaces the pushed stream.** The user: "as -stateless as possible" (C2), and "The backend should only hold model-relevant -information" (ADR 008 Q33). ADR 008's question 52, what replaces the pushed projection -stream, was answered "defer this to ADR 009", so it is this ADR's question 13. The -options, in ADR 008's lettering: -- **(a) Never learns: refetch by policy.** The host keeps nothing and says nothing. A - client refetches when its own policy says to (Q42). A change made by another client - stays invisible until then. -- **(b) An unaddressed change notice** (Flux style). After each accepted batch, the - model emits one notice to every consumer, addressed to none. It carries: - - the stamp `(host instance, document_generation, publication)` (T3); - - each item edit, as an axis change (above); - - a flag when the formula set or `measure_agg` moved. Either may move every aggregate - without moving a record. `measure_agg` has no wire command today, and changes only - with a load, which moves `document_generation`; - - **what moved, as the model computes it** (the author's lean, below); - - **display relabels.** Under ADR 007 a rename is one `display` write, which moves no - key (clause 3), so no moved-key list can carry it. The notice carries it as `(id, - new display)`, and the client relabels what it holds without refetching. Before ADR - 007, a rename is a removal and an insertion of every cell under the old name, and it - arrives as moved keys. - - **Records alone are not enough** (found by the review of `dce3ab63`). A client could - test whether each moved record falls under its own axes and filters, but that misses - every formula whose inputs sit at another stem: an inner `WHERE` naming another item, - or a ref to an item of a page, filter or None category (clause 5's one join). For - example, a client on page `Region=West` shows `EastShare = Revenue / SUM(Revenue WHERE - Region=East)`. Another client writes an East record. The West client's test says "not - mine", and its cells stay stale until its own refetch policy fires. Two remedies: - - **The author's lean: the notice carries the moved keys the circuit already - computes.** Clause 5 finds the affected `(stem, formula)` pairs, and clause 4 the - moved group keys. That is the circuit's whole job. For each **live shape key**, which - is a model-side cache keyed by requests and never by a client, the notice lists the - moved output keys: cells, formula cells and column-tuple totals. A client intersects - them with its window by identity, so the model decides relevance and the client only - looks it up. **A collapsed fold row has no maintained output key under item grain** - (clause 7: it is combined at read time). So a moved member key goes through the - client's own member-to-group remap before it is matched against the window. The - client holds that remap, since collapse is its UI state. A notice then costs O(moved output keys across live shapes), not - O(records moved). - - **The alternative: a conservative client rule.** The notice carries records only, - and a window that holds any formula with cross-stem inputs (an inner `WHERE`, a ref to - a page, filter or None item, or an aggregate) treats **every** moved record as - relevant. The notice is small, but such windows refetch on every batch. - - Under either remedy, the client refetches what it needs by `project`, or by question - 22's successor query. A formula or `measure_agg` change means refetching the window. - The host keeps no per-client state. -- **(c) Push the model's data changes**, as soft-state subscriptions. A request may ask - to be kept current. The host keeps the request and the last publication number it sent, then pushes - the window's deltas under T3 to T5. The table can be rebuilt from the requests (a lost - table costs a snapshot). But it is per-client state on the backend, and it holds a - window, which is a client's view parameter. -- **(d) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and - `grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`, - `:822` and `:981` at `50e1314d`). It is obsolete under the split, and is not proposed. - -**The trade between (b) and (c) is host state against wire traffic** (the author's; -corrected after the review of `5a8dc486`; nothing here is measured). -- **(b) keeps no per-client state, and pays in traffic.** Its one broadcast is still - delivered to all N clients, and each receives every live shape's moved keys. That is - O(N × all moved keys) on the wire per batch. In this ADR's primary case, a live source - streaming into a large table, every remote client would receive every moved key, most - of them for shapes it does not read. -- **(c) keeps per-client state, and saves traffic.** It holds the request and the last - publication number per subscriber, which is a client's window and so a view parameter on the host, - against Q33. In return it sends each client only its window's deltas. -- *As first written* (superseded), this paragraph said (b)'s key lists cost O(moved keys) - per batch while (c)'s cost the same per subscriber. That had the wire cost backwards. - -**A middle form, (b′): one topic per shape key.** A client subscribes to the topic of -its own shape key. The host keeps O(1) per client: a topic subscription, which names a -shape key and holds no window, cursor or view. **A topic filters only the moved-key -lists; every batch's header goes to everyone** (found by the review of `e152b667`): -- **The header goes to every topic, and to every topic-less client, for every accepted - batch.** It holds the stamp (host instance, `document_generation`, publication; T3), - the axis changes, the display relabels, the formula and `measure_agg` flags, and the - eviction list (below). None of - these is per shape: an item insert shifts every shape with that category on an axis. - So a filter must not withhold them. Headers are also what keep T3's loss detection - working. Every client sees every stamp, so a gap is a real loss and never a quiet - batch. That costs O(N) small headers per batch. -- **The moved-key lists are filtered by shape.** A client receives only its own shape's - list. -- **Subscribing, first or after a switch of shape key.** *As first written* - (superseded), this said a client fetches at publication n and then subscribes from n, and - that the switch "leaves no gap". **That was wrong** (found by the review of `70ababfe`). - The host keeps no past moved keys. So if batch n+1 moves a key in the new shape before - the subscription is registered, the client gets header n+1, with contiguous stamps, - but not n+1's key list, and its window goes stale silently. A client's first - subscription has the same race. *The author's lean:* **(i) the fetch registers the - subscription atomically.** The request names its topic, and the host registers the - subscription in the same step that computes the fetch at n. Every later batch then - routes to it. The alternative, **(ii)**: the - subscription is acknowledged with the current publication number m, and the client refetches if - m > n. That costs a second round trip whenever a batch lands in between. The client - drops the old topic once the new fetch is in. -- **The cache lifecycle** (found by ChatGPT Codex on `e152b667`). Clause 7 evicts a - shape once no recent request names it. A client still subscribed to that topic would - then receive no moved keys, which looks exactly like "nothing changed", and its window - would go stale. *The author's lean* combines two invariants: - - **(i) a live topic subscription pins its shape, and a subscription lives exactly as - long as its connection** (revised after the review of `85dc0ad8`). *As first - written* (superseded), subscriptions were leases renewed by requests or a - heartbeat. That let a client's own lease lapse silently while another client still - pinned the shape. The shape was then not evicted and nothing was announced, so the - client kept receiving contiguous headers but no key lists. Tying the subscription to - the connection (the socket, the worker's port, or the TUI's in-process call) removes - the heartbeat and the lapse together. A live connection keeps its subscriptions. A - dropped connection drops them. **The transport must detect a dead connection**: a - websocket ping with a timeout on the socket server, a closed port for the worker, and - nothing for the TUI, which is in process. **The ping runs both ways.** A client must - also detect a dead host, or it reads silence as "no batches" and keeps a stale - window. So a client that hears neither a header nor a ping within the timeout treats - its connection as dropped, and refetches on reconnect. A half-open socket that is never detected - would hold its pins forever. A client always knows when its connection dropped, - and it refetches on reconnect (T3's general rule). *The author's lean*, because it is - the simplest of the three. The alternatives were a per-client subscription epoch in - that client's header, or acknowledged renewals with a self-unsubscribe on a missed - acknowledgement. Each adds a mechanism. - - **(ii) every eviction is announced.** Any eviction, by ageing out or under - memory pressure (clause 14), puts the shape key in the next header's **eviction - list**, and every client reading that shape (subscribed or not) refetches. The refetch rebuilds the shape, as a job. - Evicting a popular shape therefore causes a **refetch storm**. **Idle readers make - it likely under plain (b):** a shape that clients only watch, without requesting, - ages out, and every one of them refetches at once. That refetch is also what brings - the shape back. *The author's - options*, unmeasured: refuse to evict a shape with many live subscriptions - (hysteresis), stagger the refetches with a per-client jitter, or rate-limit - rebuild-triggering fetches through ADR 008 clause 11's job queue. The eviction list - names only the shapes evicted since the previous header, so it does not grow without - bound. - - **Plain (b) has the same hole, and the same fix closes it.** Under (b) a notice lists - moved keys only for live shape keys, and a topic-less client pins nothing. So its shape - can be evicted silently. The header's eviction list reaches it too, and it refetches. - Only the "records plus a conservative rule" alternative of (b) escapes the hole, because - it lists no shape at all. Instead of an eviction list, a header could carry the set of - live shape keys, or a shape-cache generation that moves on every eviction. A client - whose key is missing from the live set would then re-request. Both say the same thing, - and the eviction list is smaller. - - The alternative, **(iii)**: every header states, for each shape it carries keys for, - that the shape was evaluated, and a subscribed shape missing from that statement means - a conservative refetch. It needs no pins, and it costs a list of evaluated shapes - per header. -- **Traffic.** O(N headers + Σ over clients of the moved keys of that client's shape) - per batch. It saves nothing when every client reads one shape, and much when shapes - differ. It is still not restricted to the window; only (c) does that. -- **"As stateless as possible" (C2) and Q33.** A subscription is per-connection state. - It holds no UI state, and the next request, which names its shape, rebuilds it. - The author reads it as delivery state, not UI state. **But the user said "The backend - should only hold model-relevant information"** (ADR 008 Q33), and whether a topic - subscription counts as that is the user's question, put in question 13. -- **Network transparency.** The topic is a filter applied when sending, so in-process - and remote consumers still read the same notices (T6). The TUI simply subscribes to its - own shape. - -**What each form costs in complexity, not only in traffic** (the author's; nothing -measured). The mechanism (b′) needed has grown review by review: topics, pins, -atomic registration, an eviction list, and a lease (now a connection tie). That growth -is itself a signal. -- **Plain (b): moved keys per live shape, broadcast to every client.** It keeps no - per-client state and needs no topics, pins or registration. It still needs the eviction - list, because a client's shape can be evicted. Its traffic is O(N × all moved keys) per - batch. **With a size cap**, a header whose key lists exceed a fixed size carries "too - many changes, refetch" instead of the lists. That bounds every notice, at the price of a - refetch by every client in a busy batch. -- **The records-only variant of (b).** It sends moved records and a formula flag, and - windows with cross-stem formulas refetch on every batch. It needs no shape lists, so it - needs no eviction list either. It is the simplest form, and its notices are the - smallest. It refetches the most. -- **(b′): (b) filtered by connection-tied topics.** It cuts traffic to the shapes each - client reads. The price is per-connection state, which is up against Q33, plus atomic - registration at fetch and pins on the shape cache. - -*The author's recommendation, revised again:* **plain (b) with a size cap**, keeping no -per-client state, with **(b′)** as the optimisation to adopt only if traffic is measured to -matter with many clients on different shapes. (c) stays the further optimisation for a -small window on a busy shape. (b′) is defensible against (b) with a cap only where -clients read many different shapes of one busy document, and nothing here shows that -case is common. The user may well prefer the simpler design. *As first written* -(superseded), the lean was (b′). - -**The user's answer to question 7 bears on this, and does not decide it.** "As much state -as possible should be maintained in the backend, unless it's UI-specific state", and "we -probably need to have a mix of caching and a pull model to expand the cached data". Plain -(b) keeps the maintained model state in the backend and has clients pull on a notice, -which fits both sentences. Whether a per-connection topic is backend state of the kind -the user means, or UI-specific, is still question 13's question to the user. - -*Earlier still* (superseded), this clause leaned to (c), which it then called (b). That -was before the user said the backend holds only model-relevant information. Under (b′), -the user's "recent projections + clients" (C2) becomes clause 7's shape cache, which is -keyed by requests and holds no client, plus the topic subscriptions, which hold one shape -key per client. - -**A connection operator (C13, the user's consideration; everything below is the -author's reading).** The user: "I think we can minimize the amount of UI state on the -backend by having an operator of some kind responsible for both ends of the connection -between the frontend and backend and handling keep alive / session resumption / etc". -- **What "operator" means, after C14.** C14 resolves most of C13's ambiguity: "And it - asks as a sink/source for the DBSP graph part on the backend and a source/sink for the - client side", with "asks" read as "acts" (flagged). *The author's reading:* the - operator is **shaped like the DBSP graph**. Toward the host it is a **sink** for the - graph's output streams (projection results and change notices) and a **source** of the - requests and model commands it feeds in. Toward the client it is a **source** of those - results and notices, and a **sink** for the client's requests and events. So the - connection between frontend and backend is itself a pair of sink/source endpoints, and - the operator owns both. What stays ambiguous is whether "both ends" means both halves of - that pair inside one component, or a component in each of the two processes. Question - 28. -- **Why this fits C1** ("network transparent"). The circuit's streams are the same - whether consumed in process or across a wire, and the operator is the **adapter at the - wire boundary**. It owns: - - the serialization of Z-set deltas, under T1 (`CellKey` is not `Serialize`, so the wire - form names tuples and ADR 002 surrogates) and T2 (the wire's NaN identity); - - the stamps and the gap rule (T3); - - snapshot-then-delta (T4); - - coalescing and backpressure (T5); - - session resumption. - - In process, both of its halves collapse to function calls, which is T6. -- **Where it sits among the crates** (the author's reading). The host-facing half - consumes the circuit's output types, so it belongs with the seam crate - `improvise-incremental` (question 2's answer) or right above it. The wire types it - serializes belong in `improvise-protocol`, beside the one envelope pair (gotcha 42). - `zset` stays dependent on `std` alone (question 1's answer), with no serde, because the - conversion happens in the operator. The **client-facing half** would be written again - by every front end unless it is shared, so by design-principles §5 it belongs in the - shared layer, as `improvise-protocol`'s client fold (`ClientState`, `client.rs`) - already is. Who owns keep-alive and resumption state, and whether the TUI, the browser - and a CLI share one client half, are question 28. -- **What it would hold, so that the host does not:** the connection; the keep-alive, with - the ping both ways; session resumption, meaning a reconnect with a token and then the - replay-or-refetch rules (T3's host instance and publication number); the topic - subscriptions and their pins ((b′)); and each client's last applied stamp. The host - would then answer `project` and `apply`, publish its unaddressed notices, and know - nothing about clients. The operator is the one place that does. -- **Where it could live, in today's code:** - - in process, for the TUI; - - beside the host in the browser worker: `improvise-worker-server`'s `Worker` already - owns one `Host` and one port (`worker.rs:14-15` at `f0fb6880`); - - as the socket server's front: `improvise-ws-server`'s `Registry` and transport are - the nearest thing today (`registry.rs:88-89`). - - Whether it is a new crate, a new process, or a role that existing code takes on is - question 28. -- **What it changes for question 13.** If the operator subscribes to the host's - unaddressed notices **on behalf of its clients** and fans them out, the host sends - O(operators) notices per batch, not O(clients). The per-client filtering of (b′), and - even the window restriction of (c), move into the operator. The wire between host and - operator then carries (b)'s broadcast, and the wire between operator and client carries - (b′)'s or (c)'s traffic. That changes the trade the review of `5a8dc486` caught, of host - state against wire traffic: with an operator, (b′) and (c) cost the **operator** state, - not the host. The costs are a new component, a second hop, and a place where a lost - notice can happen. When the operator is in-process or in the worker, the second hop is - a function call. None of this is measured. -- **What it changes for question 14.** Pins and subscriptions belong to the operator. It - re-requests the shapes its clients read, and that keeps them alive in the host's cache, - so the host's eviction rule (no recent request) holds unchanged. A host-side pin is no - longer needed. -- **The Q33 question.** If topic subscriptions live in the operator, the host holds none, - and a subscription is no longer "backend" state in the sense of the user's Q33 answer. - That holds only if the operator does not count as backend, which is part of question - 28. -- **Network transparency and "as stateless as possible".** T1 to T6 then hold on each - hop, and the operator must not change a result, only route it. The host becomes as - stateless toward clients as plain (b), and the per-client state is concentrated in one - component that can be restarted, because a client then refetches (T3's general rule). - In ADR 008's view/host split, the operator sits on the **host side of the wire, outside - the model**. It knows clients and holds no UI state beyond delivery state. -- **What it does not answer.** Question 13's choice, question 14's eviction policy, and - whether the operator is a new component at all. - -**What becomes of today's pieces under (b) and (b′)** (the author's): -- **`GridExtent`** is not pushed. A cross-product axis's extent is the product of its - levels' item counts, which a client derives from the category lists it fetches. A - pruned axis has no global extent under client-side pruning (clause 12), and question - 22's successor query answers "is there more". **In records mode** the row extent is the - record count that the page and filter admit. That is a model fact, which the client must - **fetch**, because it cannot derive it. -- **`Dirty`** goes with the per-mirror projection it drove. Its job, deciding which - widget a change can reach, moves into the client's test of the notice. -- **The occupancy delta** is not pushed. It stays a model-side output that keeps question - 22's index current. A client learns of a new or vanished row from the notice's records, - and re-asks. - -A shape is kept while recent requests name it or a live topic subscription pins it. -Eviction drops its operators, is announced in the next header's eviction list, and the -next request rebuilds them as a job (clause 7). How long "recent" lasts is a cost the user has -not set (question 14). - -**What a client must hold to resume:** its request, the last publication number it applied, its window, -and its basis. The host holds nothing that a client needs back. - -**The basis.** ADR 001's basis is answered with the result it describes, and after it -(gotcha 43), built from the request's window, never from a mirror. `published_basis` -(`project.rs:585` at `50e1314d`) already takes the `GridCache` it describes, not a -cursor. Under the user's Q39 answer, only `document_generation` and the record tier are -model facts. This ADR does not rely on `layout_version` (ADR 008 question 68). - -### 13. Incremental max width: the host maintains it for string-value columns - -The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?" -**The user then decided** (question 12, 4226624890): "Yes, I think the host should -maintain the max width for string-value columns incrementally." So **the host maintains -each string-value column's maximum display width incrementally**. This clause is now -that direction. Its cost analysis is the author's, and none of it is measured. - -**Scope.** "String-value columns" is read, by the author, as columns whose cells hold -text values (`CellValue::Text`), as opposed to numbers formatted for display. Numbers' -widths depend on the number format, which is display-only, so they stay with ADR 008's -rule: (b), sticky per session, eased by hysteresis, over the rows realized, held by the -view layer (ADR 008 clause 4). Whether the user means this split, and what a column -holding both text and numbers does, is question 27. - -**For text, two costs below shrink.** A text value's display width does not depend on -`number_format`, so there is no rebuild when the format changes, and no formatting on -the write path, only a display-width measurement of the string. What remains is the -histogram per column, and the delta of text cells over the whole column, including fold -rows. - -**DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse. -When the widest value is retracted, the new maximum cannot be computed from the old -maximum and the retraction. So it belongs to clause 4's **integrated** class, beside `Min` -and `Max`: each group keeps its integrated multiset and reads off the last key. A *top-k* -(keeping only the k widest) holds less state, but it is wrong once the top k have all been -retracted, and then it needs a rescan. So it is not proposed. - -**For widths the multiset is small.** The group is one column of a projection shape, -together with its number format, since a width is measured on formatted text. The values -are display widths, which are small integers. So the state is a histogram from width to -count, per column. It is O(distinct widths) per column, whatever the number of rows, and -an update costs O(log d), where d is the number of distinct widths. - -**The cost is upstream, in what feeds the histogram.** Its input is the display width of -**every evaluated cell in the column**. Of the three costs below, the second and third -apply **only to numbers**, which a host-maintained width would cover only if question 27 -extends the decision to them. For text, the first one remains: -- **A delta of evaluated cells over the whole column**, not a window. Clause 4's group - aggregates and clause 5's formula re-evaluation already produce an output delta for - every unfolded cell that moves, with its old value as the retraction. So unfolded rows - need no new state. **Fold rows are not covered.** Under item grain (clause 7) a - collapsed group's value is combined when it is read, not maintained, so its width would - need the fold maintained per column at the visible grain. That holds for text as well: - a fold row's text, under the unanimity rule, is a combined value. -- **A client's hidden rows** come out of the whole-column histogram the way they come out - of a total: subtracted at read time by question 23's (A) overlay, at O(H log d). -- **Numbers only: formatting every changed cell on the host**, on the write path: `format_f64` and a - display width for each changed cell. Each one is cheap, but it would be the first - display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value - back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But - today the host is asked for widths only over the rows requested (ADR 008 clause 4), and - this would widen that to whole columns. -- **Numbers only: a rebuild whenever the format changes.** `number_format` is a parameter of the view - definition, and a rebuild costs O(occupied rows) per column, as a job. - -**Against ADR 008's width rule.** A maintained whole-column width is ADR 008's question 2 -option (c), "size each column from its whole contents". The user had chosen (b) there, -and now chooses the maintained width for string-value columns. So the two rules split by -column kind, and ADR 008 should record that (question 27). *As first written* -(superseded), this paragraph recommended not maintaining widths "now", and asked the -user in question 12. The user answered yes. - -### 14. Resident state, and the store-memory requirement - -The user, on ADR 008: "We have to be careful here to minimize memory requirements of the -store: offloading inactive regions to disk (maybe mmap or similar), -double-delta/gorilla/lz4 compression, etc." (#308, comment 4171481179). And then, on ADR -008's question 17: "Let's defer this as recommended." (4188271710). So the store's memory -gets its own ADR, once this ADR's operator state has been measured at ADR 008's fixture. -**This clause withdraws the earlier non-goal** ("Persistence, spilling to disk, and -threads", clause 1), and it says what this ADR must not rule out. - -**What this ADR keeps resident, beside the store.** This is the author's inventory. The -orders come from the clauses and are not measured. - -| state | clause | order | -|---|---|---| -| the integrated base that a rebuild reads | 7 | O(cells), if it is kept as a copy | -| clause 5a's per-record stage | 5a | O(cells of the records an aggregate over an expression reads), as written | -| multisets for `Min`, `Max` and text | 4 | O(cells) of the measures that need them | -| `ExactSum`, per summed group | 8 | 568 bytes per group | -| occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape | -| the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) | -| the subscription table, only under question 13's (c) | 12 | O(request) per subscription | - -**Proposed by the author:** -- **The store is the integral, so it is not copied.** The integrated base a rebuild reads - is `DataStore` itself, which is I(cell delta) by definition. Clause 5a's per-record - stage keeps each record's cells only to compute the retraction `I(x)₋₁`. That equals - `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. So - the stage can read the store and keep no copy of the values. It still needs each cell's - store sequence (clause 5a), one integer per cell, which `DataStore` would keep beside - the value rather than in a second map. This is **not measured**, and the retraction of - the order key has to be checked against it (question 17). -- **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a - range, and apply a delta. The first implementation keeps it in memory, and the - store-memory ADR can page or compress it without changing any operator. -- **Shapes are evicted** unless a live topic subscription pins them (clause 12). So per-shape state is bounded by the shapes in - recent use, not by every saved view. -- **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`, - `Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and - O(groups), and the store-memory ADR has to count them. - -**The frame budget.** ADR 008 clause 11 sets it, and the user accepted the recommendation -(ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms -slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is -a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration -step 4 must. It reports both as ratios against an unrelated operation on the same run, -as ADR 008's question 13 requires, not as bare milliseconds. +- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its + `document_generation` and record tiers, not on `layout_version`. +- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace; + epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values. + Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a + 256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators. +- **ADR 006:** replay reproduces a circuit's state. +- **ADR 007:** keys become ids, so a rename is a relabel (clause 3). +- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the + occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire + prunes, and which parameters a request carries, are shared with questions 9 and 13 + below. ## Alternatives considered -**Feldera's `dbsp` crate.** What was verified on 2026-09-24: - -- The latest release is 0.354.0, a 4.8 MB compressed tarball, MIT OR Apache-2.0, - requiring Rust 1.96.1. -- It has 67 direct normal dependencies, none optional, and two features - (`backend-mode`, `default`). tokio, mimalloc, nix, zstd, libc and core_affinity are - all required. -- `cargo tree -e normal` resolves about 455 crates natively and 421 for - `wasm32-unknown-unknown`. For comparison, improvise-core's whole normal tree for wasm32 - is 25 crates, and improvise-command's is 68. -- `cargo check` succeeds natively. **`cargo check --target wasm32-unknown-unknown` - fails**: `mio` (tokio's) reports 48 errors. - -What was **not** verified: whether anything past `mio` would also fail. The build stops -at the first failure, so that is estimated, not measured. It is likely, because -mimalloc's sys crate and zstd build C code, and `libsqlite3-sys` already fails under this -repo's nix cc-wrapper (gotcha 41). Even if it built, it is a multi-threaded, -storage-backed query engine where improvise needs a single-threaded operator library. -The user's decision already excludes it. This records why that decision loses nothing. - -**Keep eager evaluation and memoize.** improvise-swre proposes keying `grid_of` on the -existing tiers, so an unchanged grid costs nothing. That is worth doing under either -backend. But every write still recomputes every formula and rescans the store, and a -second view with a different None axis still clones the model. It removes the cost of -*not* changing and leaves the cost of changing at O(cells). - -**An ad-hoc incremental cache.** Track dirty stems by hand: invalidate the groups a -write touches, and recompute them. It is cheaper to start. But each derived fact, such -as a group aggregate, occupancy, a column total, a stem or a cross-stem `WHERE`, needs -its own invalidation rule. Each rule is a second description of evaluation, free to -disagree with the first. The Z-set algebra gives **one** rule, linearity and -bilinearity, whose laws can be property-tested once in the crate. That is the same -argument as design-principles §5's "a derivation done twice is the defect". - -**Compile formulas into operators.** This is the fully DBSP-shaped answer. It is -rejected in clause 2, because it would be a second evaluator. +- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio, + mimalloc, zstd and others), about 421 crates for wasm32, and fails + `cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded, + storage-backed engine where improvise needs a single-threaded operator library. +- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only + the cost of not changing: every write still recomputes every formula. +- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule, + which is a second description of evaluation. Z-sets give one rule, linearity, whose laws + are tested once. +- **Compile formulas into operators.** A second evaluator (clause 2). ## Consequences and migration -**Order.** Each step is its own ticket and PR, with gates green, and none switches the -displayed path until the differential test holds. +**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches. +The store scans behind the record tier and the dirty flag go, and so do the per-view model +clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources +that can report changes plug in unchanged. The host keeps no per-client state, and one +delivery path serves every front end. -1. **The `zset` crate**, with its group laws, operator properties and determinism - test. Nothing in improvise uses it yet. -2. **The `Derived` trait, with `Eager` behind it.** No behaviour change, and the - existing suite is the evidence. `CellKey` gains `Ord`. -3. **The cell delta from `DataStore`**, with clause 3's bit identity and its signed-zero - and NaN tests. `bump_changed_records` moves onto it. The dirty flag's content half - becomes "a non-empty projected cell delta, or a moved formula-set fingerprint" (clause 3), with - a test that a formula edit alone still marks the document dirty. That removes the - O(cells) scans per cell write. The existing tier tests are the evidence. -4. **`Incremental`, for the order-free parts**, under the differential test: stems with - clause 5's stem rule, `Count`, `Min`, `Max`, unanimity, occupancy and the occupancy - delta, all following clause 7's shape key and read-time parameters. The formula delta arrives - only in step 5, so **until step 5, a moved formula-set fingerprint (introduced in - step 3) rebuilds the affected views.** **ADR 008 clauses 3 and 8 (pruning and the - frame observation) depend on this step.** -5. **Formulas**: the formula graph and the formula delta (clause 7), topological - re-evaluation, the component fixed point (clause 6), and clause 5a's pipeline that - replaces `Expansion::members`: the integrated per-record stage, the map keyed by - context and order key, the inner-`WHERE` join, and the error channel. Its tests - include a change to one operand of a two-operand expression, and the deletion of a - record's leading cell. Formula values that - depend on a `Sum` or `Avg` are re-folded in order-key order until step 6. **ADR - 008's retirement of `Expansion::members` depends on this step.** -6. **`Sum` and `Avg`, and column totals**, with the exact accumulator in both backends, - as the user decided on question 3. A column total is a formula over the pooled fold, so it needs step 5's - re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.** Under question 23's default (A), - a total subtracts the request's hidden rows when it is read (clause 7). `Sum` needs - question 3's (c) for that subtraction to be exact. -7. **Live sources as delta streams**, when ADR 003's source trait exists. -8. **The delivery stream** (clause 12), on the answers to questions 13, 15 and 19 to - 22. It brings the output types in `improvise-protocol`, the publication stamp (T3), - snapshot-then-delta, question 22's restricted successor query, and the test with two - consumers (T6). **Required tests:** - - a multi-effect batch such as `[AddItem, SetCell]` publishes exactly one header, and - no client refetches; - - a dropped header **is** detected as a gap, and the client refetches; - - a batch that lands between a fetch and its subscription reaches the client's key - list (question 13's atomic registration); - - a reconnect after a host restart refetches, and drops nothing as a duplicate; - - a refused frame consumes no publication number; - - under (b′), a dropped connection drops its subscriptions, and the client refetches on - reconnect, even while another client still pins the shape. It depends on step 4 for occupancy. It replaces the per-mirror diff - only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture, - the cost of a step and of a rebuild (clause 14). +**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that +must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs +568 bytes per group, and displayed sums change in their last bits. Circularity is decided +in two layers. The network path still clones per mutating frame. Every client receives +every notice. Every column-total read pays O(H log d) for the request's hidden rows. -**Tickets.** improvise-felo.4 is this ADR's record **only**. The user: "No, new epic and -felo.4 records ADR only" (question 10). So the steps above go to a **new implementation -epic**, which this revision does not file. improvise-swre stays independent and is useful under either backend. -improvise-l2b8 (ADR 005) gains a measured argument from clause 8: eager `SUM` is -already inexact beyond `FLOAT_EQ_EPSILON`. Under the user's answer to question 3 it is no -longer a prerequisite of step 6. - -**Good.** - -- On the keyboard path, a write's *evaluation* costs O(delta) plus the formula cells it - reaches, not O(cells). On the network path, the ADR 001 snapshot still costs O(cells) - per mutating frame (see "Bad"). -- The store scans behind the record tier and the dirty flag disappear. -- Per-view model clones for a different None axis disappear. -- ADR 008 gets occupancy, the occupancy delta and column totals without a second - evaluator. -- Live sources that can report changes plug in without touching the evaluator. -- One delivery path serves the TUI in process and every remote client (clause 12), and - the host keeps no per-client UI state under question 13's (b′), only one topic per client (none under plain (b)). Option (c) would add a subscription table that can be rebuilt - from the requests. - -**Bad.** - -- Two backends, kept equal by a differential test that must itself be kept honest. -- Integrated state is O(cells) in memory, per distinct None axis, in addition to the - store. -- An f64 `Sum` cannot be maintained on the displayed path at all. Maintaining it needs - an exact representation: the accumulator costs 568 bytes per group (clause 8), or - ADR 005's decimals. -- Under question 3's recommendation, `Eager`'s displayed sums change in the last bits, - because they become correctly rounded. That is a behaviour change, even though it is - a fix. -- Circularity is decided in two layers: the formula graph's components and the - per-cell bound. -- A mutating frame from a client already clones the workbook for rollback - (`host.rs:224-225` at `50e1314d`). Snapshotting the circuit too (question 9) adds a second O(cells) - clone per frame. Incremental evaluation does not make the network path O(delta). -- Under question 23's default (A), every column-total read pays O(H log d) to subtract - the request's hidden rows, and clause 5a's totals need their entries indexed by row - tuple as well. Occupancy and cells do not rebuild. -- The shape keys are re-read after every model effect: O(categories) per live key, a cost the - post-loop `view_shape` diff paid once per batch. -- The browser bundle grows by two small crates. -- Per-shape state lives only as long as requests name it. A shape that comes back after - eviction is rebuilt as a job, and the client waits through its pending state. -- The output identity (every NaN is one value) differs from the record tier's (bit - identity), as T2 explains. Two identities need stating everywhere a delta crosses. -- A position over a changing item list is never fully absolute. An item insert moves - every later position, and is sent as an axis change (clause 12). - -## What the user decided - -**On 2026-10-09 the user answered questions 1 to 10 and 12** in inline comments on #309, -in review 5465446805 (03:34:52Z to 04:32:13Z). They are quoted exactly below, double -spaces and the "is is" kept. Question 3's comment is anchored on an older head -(`d766605b`, line 1100), where that line is question 3's option (c). Each other comment -is mapped from its `original_line` on `6ad36437` to the question it closes. **The user -has not said "Approved"**, so the ADR stays proposed. Question 11 is unanswered. - -- **1.** (4226370667) "zset is is fine." The crate is `zset` (clause 1). -- **2.** (4226371183) "seam crate". `Incremental` lives in the seam crate - `improvise-incremental` (clause 2). -- **3.** (4226373188) "use the exact accumulator." Option (c): `ExactSum` for `Sum` and - `Avg` over f64 (clause 8, migration step 6). -- **4.** (4226588257) "Yeah, for testing purposes but eventually remove it because DBSP - should fix the cycle issue." The sixteen-pass bound is reproduced so that the - differential test compares like with like, and is to be removed later (clause 6; - question 25). -- **5.** (4226590267) "For validation, but if there's a more natural DBSP-native way we - should plan on that." The component fixed point is kept for validation, and a - DBSP-native treatment of cycles is to be planned (clause 6; question 25). -- **6.** (4226592632) "For now: the goal is to only have DBSP formulas". `Eager` stays - for now, and the goal is that formulas are evaluated by DBSP only (clause 2; - question 26). -- **7.** (4226599186) "As much state as possible should be maintained in the backend, - unless it's UI-specific state. As far as possible, state should be incrementally - recomputed using DBSP operators to minimize per-request recalculation." And - (4226616602) "However, this is intrinsically limited because the grid is infinite, so - we probably need to have a mix of caching and a pull model to expand the cached data." - *The author's reading:* this endorses question 7's slice, which keeps every page's - state maintained in the backend, and it states a principle that bears on questions 13 - and 14 (clause 12). -- **8.** (4226618377) "Eager should be deprecated." This **overrides** question 8's - recommendation, which kept query-only sources on `Eager` (clause 10; question 26). -- **9.** (4226622610) "The recommendation is fine, I think". The backend's state joins - the rollback snapshot, with no later switch to the inverse delta (clause 11). -- **10.** (4226621783) "No, new epic and felo.4 records ADR only". This **overrides** the - recommendation: improvise-felo.4 stays the record of this ADR, and the implementation - steps go to a new epic. Nothing is filed by this revision. -- **12.** (4226624890) "Yes, I think the host should maintain the max width for - string-value columns incrementally." This **overrides** the author's recommendation - ("no, not now"). Clause 13 is rewritten as the decided direction; its scope is - question 27. - -**What the answers change, and where they pull against earlier text** (the author's -readings, flagged): -- **`Eager` is deprecated, not permanent** (questions 6 and 8). Clause 2's "why keep - both", clause 10's query-only sources on `Eager`, and clause 5a's eager path for - formulas the per-record map cannot express all assumed a permanent `Eager`. They now - hold "for now", as the differential test's oracle. What replaces `Eager` for those two - uses is question 26. -- **"Only DBSP formulas"** (question 6) pulls against clause 2's rule that formulas are - never compiled into operators, and that the one existing evaluator computes each - value. The user's words may mean "every formula goes through the circuit", which - clause 2 already allows, or "formulas become operators". Question 26. -- **"DBSP should fix the cycle issue"** (question 4) pulls against clause 6's argument - that DBSP's least fixed point is not improvise's circular semantics. `X = X + 1` has no - fixed point. Question 25. -- **"As much state as possible ... in the backend"** (question 7) sits beside the - user's ADR 008 Q33 answer, "The backend should only hold model-relevant information". - The two agree for derived model state, such as the shape cache and maintained - aggregates, and both exclude UI state. "A mix of caching and a pull model to expand the - cached data" fits question 13's plain (b), where clients pull on a notice, and question - 14's shape cache. It does not decide question 13 or 14, and does not say whether a - per-connection topic counts as backend state. -- **String-value column widths are maintained by the host** (question 12). That - narrows, for those columns, ADR 008's decided width rule: (b), sticky per session with - hysteresis, over the rows realized. ADR 008 should record it. Question 27. +**Order.** Each step is a ticket and a PR in the implementation epic, and none switches +the displayed path until the differential test holds: +1. the `zset` crate, with its laws, operator properties and determinism test; +2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`; +3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a + signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty; +4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and + occupancy) under the differential test, with step and rebuild costs measured at ADR + 008's fixture. ADR 008's pruning depends on this step; +5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's + aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price` + and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members` + depends on this step; +6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row + overlay; +7. live sources as delta streams, once ADR 003's source trait exists; +8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one + header and no client refetches; a dropped header is detected; a reconnect after a host + restart refetches and drops nothing as a duplicate; a refused frame consumes no + publication number; and a dropped connection forces a refetch. ## Open questions -Each question has a recommendation. The user's answers will be folded in under "What -the user decided", as ADRs 004 and 007 did. **Status on 2026-10-09: the user has answered -questions 1 to 10 and 12 on #309.** Questions 12 to 28 were added by the 2026-10-05, -2026-10-06 and 2026-10-09 revisions, and none is renumbered. +1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page + categories, every page's state is maintained and `[`/`]` only reads another slice. The + held key stays O(visible), and the state stays bounded by the stored cells. If it does + not, a page change builds operators like any other new key. *Recommendation:* the + slice, which clause 7 assumes. -1. *[Answered: `zset`; see "What the user decided".]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix? - *Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not - carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`. +2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder, + since a reorder rebuilds nothing. By item identity with the order looked up at read + time, which makes reads O(d). Or by an order label: a key that compares in the current + order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder + re-keys every entry stored under the moved items. *Recommendation:* the order label, + with only the drawn members ordered. Not measured. -2. *[Answered: a seam crate.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on - `zset` and `improvise-core`), or inside improvise-core? - *Recommendation:* a seam crate. improvise-core then never depends on `zset`, and - `Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape. +3. **Should notices be filtered per client?** Broadcast (clause 12) keeps no per-client state, + but every client receives every live shape's moved keys. The options: + - **topics:** a client receives only its own shape key's key lists. Headers still go to + everyone, so the refetch rule keeps working. That costs one subscription per + connection, registered atomically with the fetch that names it, and pinning its + shape; + - **window push:** per-client subscriptions holding the window, sending only the + window's deltas. That costs the most host state. -3. *[Answered: (c), the exact accumulator.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display? - - (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums - until then. - - (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in - the differential test. - - (c) **An exact accumulator** (clause 8) for `Sum` and `Avg` over f64, used by - **both** backends, so that incremental equals eager bit for bit. + Is a per-connection subscription "model-relevant information" the backend may hold, + or UI state it must not? It need not live in the host at all if the connection + operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt + topics only if measured traffic, with many clients on different shapes, justifies + them. - *Recommendation:* (c). `ExactSum` lives in `numtower`, wrapped by the seam crate, - as the user decided on ADR 005 question 15. It is measured bit-equal after a million - retract-and-insert cycles, and exact against an exact oracle on 20,000 cases, with - and without large weights. It does not wait for ADR 005. (b) is never bit-exact, and - its gap reaches `FLOAT_EQ_EPSILON`. (a) remains right for per-cell arithmetic and - typed values, and it composes with (c): under ADR 005, a decimal sum is exact - through ADR 005's own wide accumulator (its question 10), provided both backends use - it, exactly as (c) requires for `ExactSum`. `rust_decimal`'s own `+` is not - associative at its bound, so decimals alone are not enough. **The costs to accept:** - - 568 bytes per group, and 2.0–3.6 ns per update against 1.5–2.7 ns for a plain add. - - **`Eager` pays the read on every aggregate evaluation**, 0.6–1.1 µs, since it - would use the same accumulator. A whole-grid eager projection pays it once per - summed cell. - - Today's displayed sums change in their last bits, because they become correctly - rounded. - - **Signed zero:** the accumulator reads `[−0.0]` as `+0.0`, where the f64 `sum()` - that `eval_agg` uses (`types.rs:79`) gives `−0.0`. - - `Avg`'s one rounding must be stated in step 6. +4. **How long does a shape key live?** A key is evicted once no recent request names it. + Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades + memory against refetch storms. If the connection operator re-requests the shapes its + clients read, that is a keep-alive by another name: one host `project` per shape per + eviction period. *Recommendation:* evict after a period with no request, set by + measurement, with hysteresis for shapes with many readers. -4. *[Answered: reproduce it for testing, and remove it eventually; question 25.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported - `circular` today (core.md §3). Should `Incremental` reproduce that? - *Recommendation:* reproduce it, so the two backends are comparable. If the bound is to - change, change it in both backends at once, as its own decision. +5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs + with different payloads differ, but the wire spells every NaN `"nan"`. A remote client + would therefore see such a change as nothing. *Recommendation:* outputs consolidate + under the wire's identity (every NaN one value), and the record tier keeps bit + identity. -5. *[Answered: yes, for validation, and plan a DBSP-native way; question 25.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by - today's bounded fixed point, restricted to their component, rather than by any - operator in the crate? - *Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the - cells a guarded cycle leaves convergent. The crate needs no recursion operator. +6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and + column aggregates by items, reports zero crossings per step, and slices rebuilds. That + makes collapse, hide, reorder and add item free, at the cost of recombining a + collapsed group's members on read. *Recommendation:* adopt all three, as written. -6. *[Answered: for now; the goal is DBSP formulas only; question 26.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay? - *Recommendation:* yes, permanently. It is the differential test's oracle and the - backend for ADR 003's query-only sources. Deleting it would delete the test. +7. **Resident state.** Should the first version keep no second copy of the store (the + store is the integral), put operator state behind one trace interface, and measure + state at ADR 008's fixture before the store-memory work begins? *Recommendation:* + yes. -7. *[Answered in principle (the author's reading: the slice); see "What the user decided". Earlier note: ADR 008's flagged reading 3 counts the page selection as part of the view definition, and its flagged reading 1 sends such parameters with the request. Clause 7 now keys state by the page categories and reads the selection at read time, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so - switching page reads existing state, rather than rebuilding the view's operators per - page change? - *Recommendation:* the slice. The state stays bounded by the stored cells, and the - held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this - recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7". +8. **Formulas the per-record map cannot express.** They take the occupied-contexts + fallback (clause 5). + - (a) Should `Eager` use the same enumeration, so the two backends agree? That changes + eager's answer only at the bare-ref edge, which could also be kept by checking the + ref once before enumerating. + - (b) Should formulas targeting a category other than `_Measure` take the same + fallback, so that nothing still needs `Expansion::members`? -8. *[Answered: "Eager should be deprecated", which overrides the recommendation; question 26.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas, - with query-only sources kept on `Eager` and ADR 003's pushdown? - *Recommendation:* yes, chosen per document. Integrated state is proportional to the - source, which contradicts ADR 003's larger-than-memory goal for exactly the sources - that cannot report their changes anyway. + *Recommendation:* yes to both. -9. *[Answered: the recommendation, hedged "I think". Clause 14 also weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App` - snapshot that `Host::mutating_frame` already takes? - *Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying - `−d` restores integrated values but not the history `delay` and `differentiate` - hold, so it is not a rewind even with exact aggregators (clause 11). The cost is a - second O(cells) clone per mutating frame, beside the workbook clone that exists - today. A true rewind would need its own design and an observational-equivalence test - over every stateful operator. +9. **Which side of the wire prunes and fills?** On the client, the model offers only + windows and the restricted successor query, and the fill is the client's, with no host + state. On the model, prune is a request parameter, and the model keeps a pruned axis, + with rank and select for "row i of n", and runs the fill as a sliced job resumable + from (request, fill front). At a density of 10⁻⁴, a client without the successor query + would ship mostly empty windows. ADR 008 asks the same question. *Recommendation:* the + client. -10. *[Answered: no; a new epic, and felo.4 records the ADR only.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps - 1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by - steps 4, 5 and 6? - *Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are - how it is met. +10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas + by position instead goes stale on every item insert. Identity costs rank and select + per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as + clause 12 assumes. -11. *[Open. The recommendation was revised on 2026-10-05 by the author, because clause 7 - now makes the drawn order a read-time parameter; the user has answered neither - version.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and - improvise-felo.8). Should `LAST`/`FIRST` key their per-group state by the ordering - category's item position, and rebuild on a reorder? Or should they key by stable item - identity (ADR 007), and look the order up at read time? - *Recommendation (revised):* neither as first stated. Key each entry by an **order - label**: a key that compares items in their current order and is not renumbered when - an item is inserted elsewhere (an order-maintenance label, the "order key" of ADR 008's - question 58). Reads stay O(log d), like `Min`/`Max`, and the same generic operator - serves all four. An insert or a reorder relabels only the items that moved, and - rebuilds nothing. **But each relabelled item re-keys every group entry stored under - it**, so a reorder costs O(entries under the moved items), not O(moved items). This - is not measured. `LAST`/`FIRST` order only the values the fold - covers, which are the drawn members. - *As first written* (superseded): key by item position, relying on `view_shape`'s - rebuild when drawn items reorder. Clause 7's shape key no longer includes the drawn - order, so that rebuild does not happen, and a position key would go stale. +11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers + the restricted successor query. Per-axis counts can only skip rows that are empty + everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned + extent. *Recommendation:* keep them in that role. Not measured. -12. *[Answered: yes, for string-value columns; the recommendation below is overridden; question 27.]* **Incremental max width** (clause 13; the user's question on ADR 008). - Should the host maintain each column's whole-column maximum display width? That is - ADR 008's question 2 option (c), and its state would now be a small histogram per - column. *Recommendation (the author's):* no, not now. The user decided on (b) with - hysteresis, and under (b) a window's widths cost O(visible) and need no maintenance. - The maintained form also needs fold values maintained per column, and formatting on - the write path. Revisit it if a measured workload shows jitter that hysteresis does - not cure. +12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at + O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape + key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per + distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at + read time. Memoise per hidden set only if a measured hidden set is too large for the + frame budget. -13. *[New; now this ADR's to carry, by the user's "defer this to ADR 009" on ADR 008's - question 52.]* **Delivery: what replaces the pushed stream** (clause 12). **Owner: - this ADR.** The options are: - - (a) never learn, and refetch by policy; - - (b) an unaddressed change notice per accepted batch, carrying the publication stamp, axis - changes, display relabels, a formula or `measure_agg` flag, and the moved output - keys per live shape key. The alternative is moved records plus a conservative rule - for windows with cross-stem formulas. Its cost is traffic, O(N × all moved keys); - - (b′) the same notices, delivered by topic. Every batch's header (stamp, - `document_generation`, axis changes, relabels, flags, eviction list) goes to every - client, with the host instance in its stamp. Only the moved-key lists are filtered - by shape key. A topic subscription lives exactly as long as its connection, and pins - its shape. It is registered atomically with the fetch that names its topic. It costs - O(N) headers per batch and per-connection state, which holds no UI state. **Question - for the user:** is a topic subscription "model-relevant information", in the sense - of your answer to ADR 008's Q33 ("The backend should only hold model-relevant - information")? (b′) depends on the answer. The user's consideration C13, a - connection operator that owns both ends, would move topic subscriptions out of the - host, so they stop being host state (clause 12, question 28); - - (c) push the model's data changes through soft-state subscriptions; - - (d) today's per-mirror push, which is obsolete. +13. **Which read-time parameters travel in a request?** The page selection travels, as a + third axis. Hidden items and collapsed groups change model-side totals and folds, so + without them a client could not get those from the model. Alternatively, a collapse + could be expressed as an axis at the group's level and a hide as a filter exclusion, + but both change the shape key, which costs a build per state. The drawn order, the + number format and prune change no aggregate. *Recommendation:* hidden items and + collapsed groups travel; the rest stay with the client. - *Recommendation (the author's, revised again):* **plain (b) with a size cap**, which - keeps no per-client state, with (b′) as a measured optimisation and (c) beyond it. The - simpler choice is the user's to prefer. (b′) buys traffic savings at the price of - connection-tied topics, pins, atomic registration and per-connection state against - Q33. The records-only variant of (b) is simpler still, and refetches more. Under every - option, `GridExtent` is derived by the client (in records mode it is fetched), `Dirty` - is retired with the mirrors, and the occupancy delta stays model-side, feeding - question 22's index. *Earlier* (superseded): (b), then (b′). - -14. *[New.]* **What keys operator state, and how long a key lives** (clause 7). Is state - selected by a request's **shape key** (axes by level, None categories, filter keys, - page categories, `measure_agg`), cached and evicted, with every other parameter - applied at read time (the author's)? Or is it kept per saved `.improv` view only, so - that a request with an unsaved key is served without incremental state? - *Recommendation:* per shape key, cached, and evicted once no request has named it for - a while **and no live topic subscription pins it** (question 13's (b′)). An eviction - is announced in the next notice header, so a subscriber refetches and is never left - with a window that looks current and is not. The time is an implementation number. An unsaved collapse, reorder, page - step or prune then shares the saved view's key and builds no operator. An unsaved - hide or show of a row-axis item shares the key too. Under question 23's default (A) - its totals subtract the hidden rows at read time and add nothing to the cache. Under - (B′), it would add a **totals entry** keyed by (shape key, hidden row-axis set). Then - the cache would hold up to one entry per distinct hidden set that recent requests - name, and a client that toggles hides through many sets would pay to derive each new - entry. Only an - unsaved change of axes, filters or `measure_agg` builds a new key, as a job. - *Under C13* (the user's consideration, not decided), pins move to a connection - operator, which keeps shapes alive by re-requesting them. The host's rule is then only - "evict once no recent request names it" (clause 12). - -15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate - under the wire's identity, where every NaN is one value, while the record tier keeps - bit identity? *Recommendation:* yes. Otherwise a remote consumer and an in-process - one disagree when only a NaN's payload changes. - -16. *[New.]* **ADR 008's three requests** (ADR 008, "What revision 4 asks of ADR 009"): - occupancy and the per-column aggregates kept at item grain, a crossing count per - step, and rebuilds as steppable jobs. *Recommendation:* adopt all three, as clause 7 - is now written. This changes clause 7's list of what rebuilds, removes the rebuild - from gotcha 43's worked case, and undercuts question 11's premise. - -17. *[New.]* **Resident state** (clause 14). Should this ADR withdraw "spilling to disk" - as a non-goal, keep no second copy of the store (the store is the integral), and put - operator state behind one interface that the store-memory ADR can page? - *Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the - order the user's answer to ADR 008's question 17 already sets. - -18. *[New; premise corrected on 2026-10-05.]* **Formulas the per-record map cannot - express** (clause 5a). ADR 008's question 15 sends them to a fallback over the - occupied contexts, never to `Expansion::members`; the user's answer was "the - recommendation is fine.". *As first written*, this question said the fallback changes - `Eager`'s answers wherever the expression is non-empty at an empty context. **The code - says it does not, except at one edge.** At `50e1314d`, `agg_values` evaluates the - expression only over the `member_records` of each context (`types.rs:91-139`; - `member_records` at `:47-53`), so a context that holds no record contributes no - value. The edge is an inner bare ref naming no item. It errors for every context it - is called with, so over zero occupied contexts it yields the empty value instead of - `ref:` (clause 5a; ADR 008 revision 25). And today's eager path - reaches `Expansion::members` only at a fold or a total (`contexts_of`, - `types.rs:916-925`). At a stem it pools one context. Two parts remain: - - Should this ADR's `Eager` adopt the occupied-contexts enumeration too? It would be - a cost change with bit-equal answers, except at the ref edge, since dropping empty - contexts keeps the order of the rest. The differential test then carves out only - that edge, by name. - - Should the formulas that target a category other than `_Measure` take the same - fallback, so that they do not block ADR 008's retirement of `Expansion::members`? - - *Recommendation:* yes to both. The only change of behaviour is the ref edge, which - could also be kept by checking the ref once before enumerating. This was argued from - the code and not run. The differential test pins it. - -19. *[New. The user was "still ambivalent" (C3), then said pruning "sounds like a UI side - concern" (C6). Not confirmed as a decision.]* **Which side of the wire prunes** - (clause 12). **Owner: ADR 008** (its question 57), because it is a question about the - view/host split. This question records what each answer asks of the circuit. On the - model side, the circuit keeps a pruned axis per shape key, with rank and select. On - the client side, it answers question 22's restricted successor query, and the client - prunes. *Recommendation (the author's), revised after C6 to C8:* the client side. The - model offers cross-product windows and the restricted successor query, so a sparse - grid still never ships empty space. *As first written*, this recommended the model - side; that is withdrawn. - -20. *[New.]* **Delta keys and item edits** (clause 12; C4). **Owner: this ADR**, for the - delta keys and the form of the top-left both, as ADR 008 revision 25 proposes. Its - question 58 points here. A - cross-product position stays fixed when occupancy changes, but it moves when an item - is inserted, removed or reordered. Should deltas be keyed by tuple identity, with - absolute positions attached when they are sent? And should an item edit be sent to - every subscriber of the key as an axis change, so that clients re-derive their - positions arithmetically? *Recommendation:* yes. It costs rank and select over each - category's items (ADR 008 clause 10), at O(levels × log items) per lookup, which is - not measured. It is compatible with ADR 008's question 58 recommendation, a tuple of - item ids as the primary form of the top-left. Item grain (clause 7) serves either - answer to whether hidden items belong to the absolute space, because hiding is a - remap. - -21. *[New.]* **How a pruned answer streams, and the fill step** (clause 12; C5). **Owner: - this ADR**, as ADR 008 revision 25 proposes. Its questions 59 and 66 point here. That - includes when a fill stops, and in what order rows and columns fill. *Recommendation:* under client-side pruning the - fill is the client's, one restricted successor query per step, and the host keeps no - fill state. Changes during a fill arrive as ordinary identity-keyed, stamped deltas, - so a fill and live updates are one stream. Under model-side pruning the same holds, - with the fill a sliced host job resumable from `(request, fill front)`. - -22. *[New.]* **Pruning over the displayed values, and what the model must answer** (clause - 12; C6, C7 and C8). This changes today's semantics, which prune against the whole grid. - Four parts, with owners: - **Owner: this ADR** for the whole question, as ADR 008 revision 25 proposes (its - questions 61, 62 and 65 point here). Part (c) is the exception: it restates answers - ADR 008 records, so its text can only change there. - - (a) Should the - circuit offer a **restricted successor query**, "the next k rows after absolute - position p with a value in at least one column of S", with S in the call, and the - same for columns? Which index answers it: (i) per-axis occupancy as a pre-filter, - (ii) a column index of occupied rows, merged, or (iii) a row index of occupied - columns? Under C9 it only ever looks forward. - - (b) Should clause 9's whole-extent, per-axis occupancy be restated as index - (ii)? ADR 008's question 65 recommends keeping whole-extent occupancy and using it - only to skip. That skips only rows empty everywhere, not rows empty in S, so on its - own it does not answer (a). - - (c) Do ADR 008's question 1 ("go with occupied") and question 11 - ("prune empty should be the default") need restating? Question 1 would become the - rule for which cells of a window hold a value, and question 11 a client's display - default. - - (d) What happens - when scrolling up or left, and when a requested top-left lands on an empty row - (C10 to C12)? Clause 12 takes the user's galloping search as the baseline. It - probes with the restricted successor query, and it needs responses to report actual - positions. It costs about 14 to 28 probes at density 10^-4, by arithmetic. The - earlier fail, snap-forward and direction-snap options are kept there as superseded, - with the user's objection to snapping forward (C11). - - *Recommendation (the author's):* (a) yes, with (ii). (b) yes, once the user confirms - that pruning is the client's. (c) restate both in those terms, in ADR 008. (d) the - gallop as the baseline, and a predecessor query only as an optional optimisation, - offered once a measured round-trip cost justifies it. - -23. *[New.]* **Column totals under a hide** (clause 7, "Hiding a row moves the column - totals"). Today a column total pools only the drawn rows, so hiding a row changes it - (`layout.rs:797-803` and `:1066-1070` at `50e1314d`). Under ADR 008's 2026-10-06 - answers (Q46 to Q50), a hide is a client's UI parameter until an explicit `SaveView`. - So this question concerns the hidden set a request carries, and it assumes that set - travels (question 24). The options: - - **(B)** rebuild the shape's totals on a row-axis hide or show. This is **withdrawn**, - because the shape key is shared across clients and an unsaved hide belongs to one - client's request. - - **(A)** keep each total over all rows, and subtract the request's H hidden cells in - that column when it is read, as a temporary overlay. It costs O(H log d) per read - for every aggregate. `Count` subtracts, `Min` and `Max` skip fully hidden keys from - the extreme, and the text rule counts the distinct keys left. It keeps no per-set - state. `Sum` is exact only under question 3's (c) or ADR 005's accumulator. - Clause 5a's totals need their entries indexed by row tuple too. - - **(B′)** keep totals per (shape key, hidden row-axis set). Reads are O(1), but the - cache multiplies by the distinct hidden sets in use. Each entry costs at most one - O(cells) rebuild, or one O(H) derivation by (A)'s subtraction. That makes (B′) a - memo on (A). - - *Recommendation (the author's, re-evaluated on 2026-10-05):* **(A)**, with (B′) kept - only as a memo for a hidden set so large that its reads would not fit the frame - budget. It is the stateless option, which C2 asks for, and a typical hidden set is - small. *As first written* (superseded): (B), and then (B′). The (B′) lean rested on - an O(rows) read cost for `Min` and `Max` that overstated (A), as the review of - `ad60ad2b` found. None of these costs is measured. - -24. *[New.]* **Which read-time parameters must travel in a request** (clause 7; the user - on ADR 008's Q51: "page is a third axis, do the others need to actually travel across - the wire?"). **Owner: ADR 008** (its question 51). This question states only what the - circuit needs. The page selection travels as the third axis. A **hide** changes - model-side aggregates (the column totals, and a collapsed group's fold), and so does a - **collapse** (a fold row's value). The user: "Aggregation should always be done - model-side for consistency" (Q38). So if hidden items and collapsed groups did not - travel, a client could not get its totals and folds from the model. It would have to - compute them itself, against Q38, or show totals over all rows. **Alternative forms**, - which ADR 008's Q51 names and which are closer to the user's lean that only the axes - travel: a **collapse** as an axis at the group's level (the request's row axis names - the group level instead of the items), and a **hide** as a filter-key exclusion. Both - are already in the shape key's vocabulary (clause 7), so either form gives the model - what it needs without a separate parameter. **But both change the shape key.** A hide - as a filter-key exclusion makes each distinct hidden set its own key, at an O(cells) - rebuild per set, and it loses question 23's (A), where every hidden set shares one - key and is subtracted at read time. A collapse as an axis level does the same for - each collapse state. The **drawn order**, - **number format** and **prune** (under client-side pruning) change no aggregate, and - need not travel. *Recommendation (the author's):* hidden items and collapsed groups - travel with the request; the drawn order, number format and prune stay with the - client. Not decided. - -25. *[New.]* **What "DBSP should fix the cycle issue" means** (clause 6; the user's - answers to questions 4 and 5). Clause 6 argues that DBSP's recursion computes a least - fixed point, which improvise's circularity is not: `X = X + 1` over a stored 10 has - no fixed point, and the required answer is `Error("circular")`. A DBSP-native - treatment could be: +14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and + validation. The options: - (a) a recursion operator bounded at sixteen iterations, which writes `circular` - where it has not converged, so the bound stays part of the semantics; - - (b) structural detection: every cell in a cycle of the formula graph is `circular`, - which changes today's answer for guarded cycles such as `A = IF(x, B, 1)`; + where it has not converged and keeps today's answers; + - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles + such as `A = IF(x, B, 1)`; - (c) a monotone reformulation, where one exists. - *Recommendation (the author's):* (a) for the first DBSP-native step, because it - keeps today's answers, and the differential test can then retire the per-cell fixed - point. Not decided. + *Recommendation:* (a) first. -26. *[New.]* **What replaces `Eager`, and what "only DBSP formulas" means** (clause 2; - the user's answers to questions 6 and 8). Two parts: - - (a) Does "only DBSP formulas" mean every formula is evaluated through the circuit, - with the one per-cell evaluator inside it as clause 2 has it? Or that formulas are - compiled into operators, which clause 2 rejects as a second evaluator? - - (b) Once `Eager` is gone, what serves a query-only live source (clause 10), and the - formulas the per-record map cannot express (clause 5a)? And what is the - differential test's oracle? +15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle, + query-only live sources, and the formulas of question 8 that stay eager. + *Recommendation:* a small from-scratch reference implementation, kept in the test + crate, as the oracle; a pushdown backend behind the same trait for query-only + sources; and the fallback of question 8 as a circuit stage. - *Recommendation (the author's):* (a) the first reading, which keeps one evaluator. - (b) the occupied-contexts fallback becomes a circuit stage. A query-only source gets - its own pushdown backend behind clause 2's trait, not `Eager`. The differential test's - oracle becomes a small, slow, from-scratch reference implementation kept in the test - crate only. Not decided. +16. **Which columns get a host-maintained width?** Columns whose cells are text, with + numbers keeping the sticky width with hysteresis? And what of a column that holds + both? *Recommendation:* text cells get the maintained width, and the view layer takes + the larger of that and its own numeric width. -27. *[New.]* **Which columns are "string-value columns"** (clause 13; the user's answer - to question 12). Is it columns whose cells are text, so that numeric columns keep ADR - 008's (b) with hysteresis? And what does a column holding both text and numbers do? - *Recommendation (the author's):* a column whose drawn cells include any text keeps a - maintained width for its text cells. The view layer takes the larger of that and its - held numeric width. ADR 008 should record the split. Not decided. +17. **What exactly is the connection operator?** (clause 12) + - (a) Do its "both ends" mean both halves in one component, or one component on each + side of the wire? + - (b) Is it a new crate or process, or a role for existing code (the browser + `Worker`, the socket server's `Registry` and transport, in-process code in the TUI)? + - (c) Who owns keep-alive and resumption state, and is that backend state? + - (d) Should one operator serve many connections and subscribe to the host on their + behalf, so that the host sends O(operators) per batch, not O(clients)? That saving + needs one operator per host process serving many clients, so per-connection state + then lives inside a shared operator. -28. *[New.]* **The connection "operator" of C13 and C14** (clause 12). The user: "an - operator of some kind responsible for both ends of the connection between the - frontend and backend and handling keep alive / session resumption / etc", and "And it - asks as a sink/source for the DBSP graph part on the backend and a source/sink for the - client side" ("asks" read as "acts", flagged). Part (a) is read as answered by C14. - - (a) *Read as answered by C14:* it is a DBSP-graph-shaped adapter, a sink and source - toward the graph and a source and sink toward the client, not a DBSP operator inside - the circuit. Still open: does "both ends" mean both halves of the sink/source pair - inside one component, or a component on each side of the wire? - - (b) Is it a new crate or process, or a role existing code takes on? Its host-facing - half could live with `improvise-incremental`, its wire types in - `improvise-protocol`, and its client-facing half beside `improvise-protocol`'s - client fold, shared by the TUI, the browser and any CLI (design-principles §5). The - browser's `Worker` (`worker.rs:14-15` at `f0fb6880`) and the socket server's - `Registry` and transport (`registry.rs:88-89`) are the nearest code today. - - (c) Who owns keep-alive and resumption state: the operator's host-facing half, its - client-facing half, or both? And does it count as "backend" in the sense of the - user's Q33 answer? - - (d) Does it subscribe to the host's notices on behalf of its clients and fan them - out, so that the host sends O(operators) per batch? - - *Recommendation (the author's):* (a) one component per connection that owns both - halves, with the client half as a shared library each front end links. (b) the - host-facing half in or above `improvise-incremental`, wire types in - `improvise-protocol`, and the client half shared beside `ClientState`. (c) resumption - state in the host-facing half, keyed by a token the client holds, and treated as - delivery state, not backend UI state. (d) yes. Then question 13's choice moves to the - operator–client hop, and the host sends plain (b). Not decided. + *Recommendation:* (a) one operator per host process: a host half shared by its + connections, and a client half in every front end. (b) a role of the existing + transport crates, not a new process. (c) the host half holds resumption state, keyed + by a token the client holds, as delivery state, not UI state. (d) yes, for the socket + server. The TUI and the browser worker each serve one client, so there the operator is + a function call. From e4f915423c5947db606e0422aa406c95134750be Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Thu, 8 Oct 2026 23:20:39 -0700 Subject: [PATCH 44/67] docs(adr): ADR 009 leaves the operator's shape to its open question and points recommended clauses at theirs (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 55 ++++++++++++++--------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e78b1398..e2c239ac 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -252,8 +252,8 @@ read-time parameters here. It keeps its one job, bumping `layout_version`. The c neither reads nor depends on that tier, which is a view fact. Only `document_generation` and the record tier matter to this ADR. -**Item grain.** Occupancy and per-column aggregates are keyed by the **items** of the -axis categories, never by a position. So collapsing, expanding, hiding, showing, +**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates +are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing, reordering and adding an axis item change no operator: the visible axis is a remap. ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a rebuild. A collapsed group's value is combined from its members when read. That is exact @@ -349,7 +349,8 @@ money. ### 9. What ADR 008 gets -- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple. +- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple + (kept as a pre-filter and as the unpruned extent; see open question 11). Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. A fold row with a formula column but no stored cell under it, and the `Exact` case with nothing on None, are **not** occupied, on purpose. @@ -403,7 +404,8 @@ part of the output and is never state. So a request costs O(visible). **Positions are absolute cross-product positions**, in each category's own item order, and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when -occupancy changes. They do move when an item is inserted, removed or reordered, so: +occupancy changes. They do move when an item is inserted, removed or reordered, so the +recommended design (open question 10) is: - output elements are keyed by tuple identity (item ids under ADR 007), with the absolute position attached when sent; - an item edit is sent to every client as an **axis change** ("inserted at index j of @@ -426,11 +428,13 @@ O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed fro cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's whole state is its request and its last absolute position, so a resume is a seek. -**Scrolling up or left is a new request with a new top-left.** The new top-left is found -by galloping backwards in doubling jumps until a probe hits, then narrowing until one new -row is gained. Each probe is a restricted successor query bounded by the old top-left. -That is about 14 to 28 probes at density 10⁻⁴, and no special answer is needed when a -top-left lands on an empty row. Which side of the wire runs the prune and the fill is +**Scrolling up or left is a new request with a new top-left.** How the client finds that +top-left when rows are pruned is the recommendation of open question 9: it gallops +backwards in doubling jumps until a probe hits, then narrows until one new row is gained. +Each probe is a restricted successor query bounded by the old top-left. That is about 14 +to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an +empty row. The alternatives there are a model-side predecessor query, and failing the +fetch and retrying. Which side of the wire runs the prune and the fill is open question 9. **The wire properties**, which hold on each hop: @@ -483,10 +487,10 @@ refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, extent is the admitted record count, which the client fetches. Whether notices should be filtered per client is open question 3. -**A connection operator, at the wire boundary.** One component owns both ends of a -connection. Its host half is a sink for the circuit's results and notices and a source of -requests and model commands. Its client half is the mirror image: a source of results to -the client, and a sink for the client's requests. It owns: +**A connection operator, at the wire boundary.** A connection operator sits at the wire +boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for +results and notices and a source of requests and model commands. Toward the client it is +the mirror image. It owns: - serialization (T1, T2); - stamps and the refetch rule (T3); - coalescing (T5); @@ -495,12 +499,11 @@ the client, and a sink for the client's requests. It owns: batches"; - session resumption. -In process, both halves are function calls. That covers the TUI, and the browser's -`Worker`, which already owns one `Host` and one port (`worker.rs:14-15`). The host half -belongs in or above `improvise-incremental`, and the wire types in `improvise-protocol`. -`zset` stays free of serde. The client half is shared by every front end, beside -`improvise-protocol`'s `ClientState`, so that a new front end does not write it again. -Its remaining shape is open question 17. +In process it is function calls, as in the TUI and in the browser's `Worker`, which +already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in +`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one +component per connection, or a host half shared per host process with a client half in +every front end), and who owns resumption state, are open question 17. ### 13. Column widths for text @@ -610,7 +613,10 @@ the displayed path until the differential test holds: re-keys every entry stored under the moved items. *Recommendation:* the order label, with only the drawn members ordered. Not measured. -3. **Should notices be filtered per client?** Broadcast (clause 12) keeps no per-client state, +3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed + stream. The rejected alternative is to send no notice at all and let each client + refetch on its own schedule: it keeps the host silent, but another client's change + stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state, but every client receives every live shape's moved keys. The options: - **topics:** a client receives only its own shape key's key lists. Headers still go to everyone, so the refetch rule keeps working. That costs one subscription per @@ -663,8 +669,13 @@ the displayed path until the differential test holds: state. On the model, prune is a request parameter, and the model keeps a pruned axis, with rank and select for "row i of n", and runs the fill as a sliced job resumable from (request, fill front). At a density of 10⁻⁴, a client without the successor query - would ship mostly empty windows. ADR 008 asks the same question. *Recommendation:* the - client. + would ship mostly empty windows. A second part: when scrolling up or left, how does the + client find a new top-left when rows are pruned? The options are galloping backwards + with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a + model-side predecessor query (one round trip, no new state, but a second query + direction), or failing the fetch and retrying with the nearest occupied row named. ADR + 008 asks the same question. *Recommendation:* the client prunes and fills, and it + gallops. 10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas by position instead goes stale on every item insert. Identity costs rank and select From e7d9ad0d2a4ad09d6c145c0ab4ccb59330989fd6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 09:42:23 -0700 Subject: [PATCH 45/67] docs(adr): ADR 009 states only what the decision changes (improvise-felo.4) Following the user's instruction, restated conventions and current behaviour are removed except where the decision conflicts with them, and the open questions are cut to the five design choices still open. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 920 ++++++---------------- 1 file changed, 252 insertions(+), 668 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e2c239ac..02310a7b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,742 +1,326 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait -**Status:** proposed. Only the user accepts an ADR. · +**Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed -models) · -**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") · -**Governed by:** `context/design-principles.md` §3, §5 and §7 +**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008 -**Summary.** Formulas are evaluated incrementally by a small in-house implementation of -the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A -cell write becomes a delta. Aggregates are maintained rather than re-folded, and only -the formula cells whose inputs moved are re-evaluated, by the one existing evaluator. -Operator state is keyed by the **shape key** of a projection request, never by a client. -The same circuit serves the in-process TUI and remote clients. A remote client learns of -changes from one unaddressed notice per accepted batch, and it refetches what it shows. -Sums are exact, through an exact accumulator. +## Problem -File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive -them before relying on one. - -## Problem and context - -**Data that streams.** A live-backed model (ADR 003) is a source whose rows change -underneath the document. Today every change takes the path an edit takes, which -recomputes every formula cell. That is tolerable for one keystroke at a time and not for -a stream of changes. A change should cost in proportion to what it touches. - -**Today's evaluation is a from-scratch fold after every batch.** -- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls - `Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates - every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16` - (`types.rs:710`). -- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`, - `types.rs:848`, through `DataStore::matching_values`). -- `formula_cache` holds one aggregation shape at a time. A session on a view with another - None axis gets a **cloned model** (`rederived_for`, `project.rs:205`). -- A formula total that holds an aggregate walks a cross product - (`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`). -- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing - batch, and `document_content` (`app.rs:168`) after each `CellData` effect. -- Every mutating frame from a client clones the document for rollback - (`Host::mutating_frame`, `host.rs:224-225`). - -**What an incremental evaluator must not change.** Each of these is a place where it -could quietly disagree with today's code: -- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one - distinct value is that value, and anything else is `Error("mixed")`. -- **Circularity (gotcha 9).** A cell still moving after 16 passes is - `Error("circular")`, and so is an acyclic chain that needs all sixteen. -- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0 - guard. -- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration, - because f64 addition is not associative. -- **The tiers are observed (gotcha 40).** A write of the value already there bumps - nothing, and a rewritten record counts once. - -**The view/host split (ADR 008).** A client reads through `project(request)`, where a -request names axes, an absolute top-left and dimensions. It writes only through -`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys. -Aggregation is always done on the model side. A view's hiding, collapsing, page -selection, axis assignment and pruning are one client's ephemeral state until an -explicit save-view command. Only the number format is document-wide. +Every change recomputes every formula cell (`Model::recompute_formulas`, called by +`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That +does not scale to a live source (ADR 003) streaming changes into a large table, nor to +ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without +building the grid. A change should cost in proportion to what it touches. ## Decision -### 1. The crate: `zset` +Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A +cell write becomes a delta, aggregates are maintained, and only the formula cells whose +inputs moved are re-evaluated by the existing evaluator. The incremental results must +equal today's eager results exactly. -A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on -`std`, so it is wasm32-clean by construction. It contains: -- **`ZSet`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an - abelian group. Weights use checked arithmetic, and a zero weight is removed. The group - laws are property-tested. -- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per - source and returns one per sink. There is one clock, one thread and no scheduler. -- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id - property-tested); the linear `map`, `filter` and `flat_map`; `index_by`; - `join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear - aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a - multiset per group. -- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a - resumable `step(deadline)`, so a rebuild can run as a sliced job. +### 1. The `zset` crate -Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one -clock (improvise applies batches in one total order); threads within one circuit (a host -with threads may run independent circuits on a pool); a query language. Spilling to disk -is not in the first version, but operator state sits behind one interface (a trace: look -up, read a range, apply a delta), so the store-memory work can page or compress it later. +A problem-agnostic workspace crate with no dependencies beyond `std` (so it is +wasm32-clean and serde-free). It provides: +- `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; +- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`, + `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, + `distinct` and `aggregate`; +- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it + can run as a sliced job; +- operator state behind one trace interface (look up, read a range, apply a delta), so + later store-memory work can page or compress it. -### 2. The plug: a trait, and a seam crate +There are no nested streams, no recursion operator, no timestamps beyond one clock, and +no query language. -improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the -formula result at a key, and the raw fold of the cells that match a partial key. It -takes the cell delta and the formula delta after each model effect. +### 2. The plug -**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and -read through the trait. The circuit decides *which* formula cells to re-evaluate and -supplies their inputs. Formulas are not compiled into operators, because a compiled -`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the -defect. +improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the +formula result at a key, and the raw fold of the cells that match a partial key. The +incremental backend implements it in a seam crate, `improvise-incremental`, which +depends on `zset` and improvise-core. improvise-core never depends on `zset`. -**The incremental backend lives in a seam crate, `improvise-incremental`,** which -depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same -shape as `gp-tree` and `improvise-gp`. +The circuit decides *which* formula cells to re-evaluate. The one existing evaluator +computes each value, so formulas are never compiled into operators. -**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change, -as `Eager`. It stays as the oracle for the differential test while the circuit matures, -and for the two cases that still need it (clauses 5 and 10). The differential test is a -property test in the seam crate that applies one random edit sequence to both -backends and compares every drawn cell, total and occupancy count. The goal is that every -formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is -open question 15. +**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. +It serves as the oracle of a differential property test, which applies one random edit +sequence to both backends and compares every drawn cell, total and occupancy count. It +is removed once its remaining uses are replaced (open question 3). -### 3. The input: the cell delta +### 3. The cell delta -`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a -`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number -assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an -overwrite: -- a new cell is `+(k, v, s)`; -- an overwrite is `−(k, v₀, s) + (k, v₁, s)`; -- a removal is `−(k, v₀, s)`; -- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. +`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`: +- `+(k, v, s)` for a new cell; +- `−(k, v₀, s) + (k, v₁, s)` for an overwrite; +- `−(k, v₀, s)` for a removal; +- a remove-then-set gets a fresh `s′`. -**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their -strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an -equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a -change, and a NaN rewritten with the same bits is not. Multisets order values by -`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted -`Vec`, so deriving it is sound. +`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an +overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite +consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect. -**What the delta replaces.** The record tier reads the records named by the -consolidated `(key, value)` projection of the delta, instead of diffing two store scans. -The dirty flag's content half becomes "a non-empty projected delta, or a moved -formula-set fingerprint". The fingerprint is needed because formula edits move no cell, -and the worker persists on this flag. +The record tier and the dirty flag read this delta instead of scanning the store. The +dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the +formula set moves, because formula edits move no cell. -**The circuit steps once per model effect.** That keeps occupancy current inside the -effect loop. What a client counts is a separate publication number (clause 12). +Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The +formula-set fingerprint must use the id form too, or a rename reads as a formula change. -**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`, -the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a -rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set -fingerprint, taken today over `(raw, target, target_category)`, must then be taken over -the id form, or a rename reads as a formula change. +### 4. Aggregates -### 4. Aggregates: linear where they can be +A group is the cell key minus the None categories. One aggregation circuit serves every +shape key with the same None list, which removes the per-session model clone. -**The group** is the cell key minus the None categories, which is the partial key -`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the -same None list, which removes `rederived_for`'s model clone. - -| answer | state per group | cost per delta element | +| aggregate | state per group | per delta element | |---|---|---| -| `Count` | a count of values and a count of numbers | O(1) | -| `Sum` | an exact accumulator (clause 8) | O(1) | -| `Avg` | `Sum` / `Count` | O(1) | -| `Min`, `Max` | a multiset of the numbers | O(log d) | -| unanimity and `mixed` | a multiset of all values | O(log d) | +| `Count` | count of values, count of numbers | O(1) | +| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) | +| `Min`, `Max`, text unanimity | a multiset of values | O(log d) | -Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules -for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential -test generates signed zeros and non-finite values to pin this. Only measures whose -aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups, -they hold at most one entry per stored cell. +A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and +semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the +ordering category's order. How that order is keyed is for those tickets. -**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures, -improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an -extreme, keyed by the ordering category's order instead of by value. A semi-additive -measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the -order is keyed is open question 2. +### 5. Formulas -### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it +**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None +categories". A formula counts at a stem where its top-level `WHERE` matches or raw data +exists at the target. -**Stems** are `distinct` over the cell delta, mapped to "key minus the target category -and the None categories". A formula counts at a stem only where its top-level `WHERE` -(if any) matches the stem's coordinate, or where raw data exists at stem + target. That -is exactly where `recompute_formulas` caches a value today, so pruning keeps its current -behaviour. +**Inputs.** A formula cell is re-evaluated when one of its inputs moved: +- a `Ref(R)`: the group at the stem with `R`'s category set to `R`; +- an `Agg`: the group at the stem; +- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set + to `Item`. This is the one join: stems indexed by "stem minus `Cat`". -**A formula cell is re-evaluated when one of its inputs moved:** -- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`; -- for an `Agg` with no filter, the group at the stem itself; -- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat` - set to `Item`, which is another stem. This is the one join: stems indexed by "stem - minus `Cat`", joined with the aggregate's output delta. +The affected pairs are evaluated in the formula graph's topological order. -The affected `(stem, formula)` pairs are ordered topologically over the formula graph and -handed to the existing evaluator. +**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: +1. A per-record stage retracts and reinserts each touched record. It reads the record's + previous cells as the store minus the delta, so it keeps no copy of the values. +2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum + `Seq` of the record's live cells), which is eager's concatenation order. +3. The group aggregate, with an error channel that reads the first `Err` in order. -**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline: -1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it - retracts the record's previous snapshot and inserts the new one. The previous snapshot - is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. - So the stage reads the store and keeps no second copy of the values. -2. **A lifted map** to `(context key, order key, expr at record)`. The context key - removes the target and None categories, and at a fold or total the spanned categories, - mapping a collapsed member to its group. The order key is (context rank, the minimum - store sequence over the record's live cells), which is exactly the order eager - concatenates in. -3. **An inner `WHERE`** is a filter plus the join above. -4. **The group aggregate**, with an error channel kept beside it that reads the first - `Err` in order-key order, as eager does. - -No cross product is ever built. The map is sound only when `expr` reads the record's own -cells. Other formulas take a fallback that evaluates over the **occupied contexts** only, -the contexts some record projects to. Because `agg_values` evaluates only at each -context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one -edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over -zero occupied contexts it yields the empty value instead. Formulas targeting a category -other than `_Measure` stay on the eager path until their order key is defined. Open -question 8 covers both. +No cross product is built. A formula whose expression reads other records is evaluated +over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is +unchanged, because a context with no records contributes nothing. One exception: an +inner bare ref that names no item errors at every context, so for it the ref is checked +before enumerating. This replaces `Expansion::members`. ### 6. Circular references -DBSP's recursion computes a least fixed point, and improvise's circularity is not one: -`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs -forever. The required answer is `Error("circular")`. The first version therefore decides -circularity outside the crate: -- the formula graph is rebuilt when the formula set changes, and its strongly connected - components are found; -- formulas outside every cycle are evaluated once per step, in topological order; -- a formula in a cycle has its affected cells evaluated by today's bounded fixed point, - restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still - convergent where `x` is false; -- the sixteen-pass bound is reproduced, so the two backends agree in the differential - test. +DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the +answer must be `Error("circular")`. The first version finds the formula graph's strongly +connected components. Acyclic formulas are evaluated once each, in topological order. A +component is evaluated with today's bounded fixed point, restricted to that component. +That reproduces the sixteen-pass bound for the differential test. It is to be replaced +by a DBSP-native treatment (open question 2). -This serves testing and validation. It is to be replaced by a DBSP-native treatment of -cycles, whose shape is open question 14. +### 7. Operator state follows the request's shape key -### 7. The circuit follows the document per model effect - -**Operator state is selected by a shape key.** The key is exactly what changes which -cells feed which operator: -- the categories of each axis, by level; +A projection request's **shape key** selects operator state, shared by every request +with the same key, from any client. The key is: +- the categories on each axis, by level; - the None categories; -- the filter keys: each filter category with its selected item, and the hidden items of - every category on None, Page or Filter. Those items drop cells from the fold; -- the page **categories**, which key a slice per page (open question 1); -- `measure_agg`, which is model state, never taken from a request. +- the filter keys: each filter category's item, and the hidden items of categories on + None, Page or Filter; +- the page categories, so each page is a maintained slice and paging rebuilds nothing; +- `measure_agg`. -**Everything else is a read-time parameter**, applied when output is read: the hidden -items of an axis category, collapsed groups and group membership, the drawn order of -items, the page selection, and prune. A request carries its own key and parameters from -its client's UI state, and a saved view only supplies defaults. +Everything else in a request is applied when output is read: hidden and collapsed items +on axis categories, item order, the page selection and prune. Occupancy and column +aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item +changes no operator. The visible axis is a remap. -**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn -items, the collapsed groups, the page selection and `prune_empty`, all of which are -read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit -neither reads nor depends on that tier, which is a view fact. Only `document_generation` -and the record tier matter to this ADR. +- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas + arriving during the build are buffered and added at install. +- **A formula edit** is a formula delta fed to the circuit. +- **A document swap** rebuilds everything. +- **Saving a view definition** touches no operator. +- **Live keys are a cache.** A key is evicted after a period in which no request names + it (the period is set by measurement), and every eviction is announced (clause 9). -**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates -are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing, -reordering and adding an axis item change no operator: the visible axis is a remap. -ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a -rebuild. A collapsed group's value is combined from its members when read. That is exact -for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it -costs O(members) per read (not measured). The exceptions are column totals (below). - -**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job -runner (8 ms per normal operation). Deltas that arrive during the build are buffered and -added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is -O(delta) and is never sliced. Each step's occupancy output carries the count of tuples -whose count crossed zero, so ADR 008's ordered index can choose between inserting and -rebuilding. Measurements against the budget are stated as ratios against an unrelated -operation on the same run, not as bare numbers. - -**Formula edits are an input.** After each model effect the host compares a fingerprint -of the formula set. If it moved, the formula delta is fed to the circuit, and the formula -graph is re-derived. A document swap (`install_workbook`, which bumps -`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and -no formula, so it touches no operator. - -**The shape cache.** Live shape keys are a cache. A key enters when a request names it, -and it is evicted when no recent request names it (open question 4). Every eviction is -announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm -by every client reading it. Idle readers make that likely, since a shape that clients -only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job -queue. - -**Column totals subtract hidden rows at read time.** Today a column total pools only the -drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`), -pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a -per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept -over all rows, and a read subtracts the request's H hidden cells in that column as a -temporary overlay: +**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row +leaves it. The shape key is shared and a hide is one client's, so totals are kept over +all rows. A read subtracts the request's H hidden cells as a temporary overlay: - `Count` subtracts; -- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden; -- the text rule counts the distinct keys left; -- `Sum` subtracts through the exact accumulator. +- `Min` and `Max` skip keys whose whole count is hidden; +- text counts the distinct keys left; +- `Sum` subtracts exactly through the accumulator. -That is O(H log d) per read for every aggregate, with no per-client state. The shape key -is shared across clients and a hide is one client's, so a total cannot be rebuilt per -hide. A hidden column item under a collapsed column group changes that group column's -cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression -totals need their entries indexed by row tuple as well, so a read costs O(hidden -records). The alternative is open question 12. +That is O(H log d) per read for every aggregate, with no per-client state. +Aggregate-over-expression totals index their entries by row tuple for the same +subtraction. -### 8. Floating point: exactness comes from the representation +### 8. Exact sums -A sum maintained by `+v` and `−v` in f64 drifts from the eager fold. -`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max, -with values of up to five decimal places: +An f64 sum maintained by `+v` and `−v` drifts. Measured by +`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million +cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer +differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both +backends: +- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which + makes it an abelian group that rounds once, on read; +- measured bit-equal to a fresh sum after a million cycles, and exact against rational + sums on 20,000 cases; +- 568 bytes per group, and about 1.3× the cost of an f64 add per update. -| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact | -|---|---|---|---| -| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 | -| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 | -| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 | +It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change +in their last bits, because they become correctly rounded, and a sum of `-0.0` reads +`+0.0`. -Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively, -against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager. -At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer -differently on the two paths. Eager itself is already more than 1e-10 from the exact -decimal sum. +### 9. Delivery -**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer -wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so -that those can be retracted. Adding and retracting are integer additions, so the -accumulator is a true abelian group, and it rounds once, on read. A weight is split into -base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example: -- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles; -- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again - with weights up to 2⁴⁰; -- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to - 1.1 µs per read; -- it is 568 bytes per group, against 8 for an f64. +The same circuit serves the in-process TUI and remote clients. Its internal streams never +cross a wire, because aggregation stays on the host. -`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0, -where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they -become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and -the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum` -0.1.6 rounds about a quarter of single negative values wrongly and cannot retract -non-finite values. +**A request** is a shape key plus a window, given as an absolute cross-product top-left +and dimensions. The window is never state, and a request costs O(visible). +- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings. +- **Output** is keyed by tuple identity, with the position attached when sent. +- **An item insert, removal or reorder** is sent as an axis change, and each client + re-derives its positions. -**Lossy operations happen at the edge.** Sums stay in the accumulator. Division, -non-integer `POW` and display rounding happen only where a value leaves the exact domain. -An exact value becomes an f64 through its decimal string, which is correctly rounded. +**Pruning support.** Pruning is window-relative: a row or column empty within the +displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model +answers one restricted successor query: "the next k rows after position p with a value +in a column of S", and the same for columns. A per-column sorted index of occupied rows +answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its +window, fills forward with that query, and finds a new top-left when scrolling up or left +by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether +pruning runs on the client and which parameters travel are ADR 008's questions. This +design supports either answer. -**The accumulator does not make `=` against a typed decimal exact.** It sums the stored, -already-rounded f64s, so it can miss the typed total by up to about half an ulp of each -term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example: -`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's -exact decimals fix that, and they also cover per-cell arithmetic and typed values such as -money. +**The change notice.** After each accepted batch the host emits one notice to every +client, addressed to none. It carries: +- the stamp `(host instance, document_generation, publication)`. The publication number + increments once per published batch, and the host instance changes on restart; +- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved; +- the shape keys evicted since the last notice; +- for each live shape key, the moved output keys (cells, formula cells and totals), as + the circuit computes them. Moved records alone would miss formulas with cross-stem + inputs. -### 9. What ADR 008 gets +A notice whose key lists exceed a fixed size carries "refetch" instead. -- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple - (kept as a pre-filter and as the unpruned extent; see open question 11). - Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. - A fold row with a formula column but no stored cell under it, and the `Exact` case with - nothing on None, are **not** occupied, on purpose. -- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit - consumes it and never keeps a second order. -- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item - grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because - the host cannot compute a client's visible axis. -- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the - formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows). -- **Widths for text columns** (clause 13). +**What a client does.** It intersects the moved keys with its window. A fold row has no +maintained key, so it maps member keys through its own remap first. It then refetches +what moved. It refetches everything when it sees a publication gap, a new +`document_generation` or host instance, its shape in the eviction list, or a notice it +cannot read. A slow client may merge pending notices: the union of keys, the latest +stamp, and axis changes kept in order. Outputs are consolidated under the wire's value +identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`. -### 10. Live sources +**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, +`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client +state. `GridExtent` is derived by the client, and in records mode the record count is +fetched. -A live source that can report its changes produces a cell delta, and the circuit cannot -tell it from `DataStore`'s. A source that can only answer aggregate queries stays on -`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated -state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so -the backend is chosen per document. +**A connection operator** at the wire boundary adapts the circuit to a connection. It is +a sink for results and notices and a source of requests toward the circuit, and the +mirror image toward the client. It owns serialization, the stamp rules, notice +coalescing, keep-alive in both directions, and session resumption. In process it is +function calls. Its shape is open question 4. -### 11. Constraints +### 10. Text column widths -- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then - `zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm` - CI job, each checked with `cargo tree --edges normal`. -- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over - small inputs, and the differential test lives in the seam crate's own binary. -- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and - sorted, and nothing in the circuit decides a display order. The determinism test builds - **two** circuits independently from one history. -- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that - `Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the - integrated values but not the history `delay` and `differentiate` hold. The cost is a - second O(state) clone per mutating frame, and peak memory to match. The keyboard path - takes no snapshot. -- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence. +The host maintains each text column's maximum display width as a histogram from width to +count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows +come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule. -### 12. Delivery: one circuit, in process or across a wire +### 11. Rollback and memory -The DBSP machinery is **network-transparent**: the same operators and streams whether a -consumer is in process or across a wire. +- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An + inverse delta is not a rewind, because `delay` and `differentiate` keep history. +- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, + `ExactSum` per summed group, occupancy per live shape, the successor-query index and + the width histograms. The store itself is the integrated base and is never copied. It + is measured at ADR 008's fixture before the store-memory work begins. -**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their -change notices, plus the input deltas of a remote live source. The write side carries -model commands with their basis, never deltas. The circuit's internal streams stay on -the host, because aggregation is model-side. No UI state crosses. +## Alternatives rejected -**A request has a shape key and a window.** The key selects operator state, shared by -every request from any client. The window (absolute top-left and dimensions) selects -part of the output and is never state. So a request costs O(visible). +- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it + fails to build for wasm32. +- **Compiling formulas into operators:** a second evaluator. +- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a + second description of evaluation. +- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic + only by holding per-client state on the host (open question 1). -**Positions are absolute cross-product positions**, in each category's own item order, -and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when -occupancy changes. They do move when an item is inserted, removed or reordered, so the -recommended design (open question 10) is: -- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute - position attached when sent; -- an item edit is sent to every client as an **axis change** ("inserted at index j of - level L"). A client re-derives its positions at O(levels) each and re-anchors its - top-left to the same tuple; -- resolving positions needs rank and select over each category's items, which is ADR - 008's order-statistic structure. +## Consequences -**Pruning's model-side support.** Pruning hides a row or column that is empty within the -displayed values, so a value outside the window does not count. That differs from -today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed -window, one pass that drops the empty rows and columns is already the fixpoint. A fill -then adds rows and columns forward from the absolute top-left, until the requested -dimensions are met or the axes end. Growth is only rightward and downward. For the fill, -the model answers one **restricted successor query**: "the next k rows after absolute -position p with a value in at least one column of S", with S in the call, and the same -for columns. A per-column index of occupied rows answers it: a sorted list per column, -seeked to p and merged forward with a linear minimum over the |S| cursors, at -O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the -cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a -B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's -whole state is its request and its last absolute position, so a resume is a seek. -**Scrolling up or left is a new request with a new top-left.** How the client finds that -top-left when rows are pruned is the recommendation of open question 9: it gallops -backwards in doubling jumps until a probe hits, then narrows until one new row is gained. -Each probe is a restricted successor query bounded by the old top-left. That is about 14 -to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an -empty row. The alternatives there are a model-side predecessor query, and failing the -fetch and retrying. Which side of the wire runs the prune and the fill is -open question 9. +**Good:** +- A write's evaluation costs O(delta) plus the formula cells it reaches. +- The store scans and the per-view model clones go. +- ADR 008 gets occupancy, totals and widths from one evaluator. +- Live sources that report changes plug in. +- The host holds no per-client state. -**The wire properties**, which hold on each hop: -- **T1. Plain data.** Output elements are serializable values that name cells by tuples, - positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`. -- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`), - so a value change that only a NaN payload distinguishes cannot be told apart remotely. - Which identity outputs use is open question 5. -- **T3. A publication stamp.** Each accepted batch is published once, stamped - `(host instance, document_generation, publication)`. The publication number increments - once per published batch, so a multi-effect batch is one header, and a refused frame - consumes none. The host instance changes on restart. A client drops anything at or - below the number it has applied. **The refetch rule:** a client that sees a gap, a new - `document_generation` or host instance, its shape in an eviction list, or a notice it - cannot interpret, refetches. -- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative - to its stamp. -- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and - flags, under the latest stamp. Axis changes are positional, so they stay an ordered - list. A `document_generation` change inside the run voids the merge. A basis is never - sent ahead of the cells it describes (gotcha 43). -- **T6. One API.** The TUI calls `project` and receives the same types as a remote - client. A test runs one edit sequence through an in-process consumer and through a - serialize-and-deserialize round trip, and requires equal results. +**Bad:** +- Two backends exist until `Eager` goes. +- Integrated state is O(cells) beside the store. +- `ExactSum` costs 568 bytes per summed group. +- Every client receives every notice. +- A mutating frame still clones the state for rollback. -**The change notice.** After each accepted batch the host emits one notice, addressed to -no client. It carries: -- the stamp; -- the axis changes; -- display relabels (`(id, new display)` under ADR 007); -- a flag when the formula set or `measure_agg` moved; -- the eviction list; -- for each live shape key, the **moved output keys**: cells, formula cells and - column-tuple totals, as the circuit computes them. +**Order** (each step is a ticket in the implementation epic, behind the differential +test): +1. `zset`; +2. the `Derived` trait with `Eager`; +3. the cell delta, the record tier and the dirty flag; +4. order-free aggregates and occupancy, measured at ADR 008's fixture; +5. formulas and the aggregate-over-expression pipeline; +6. `Sum`, `Avg` and totals through `ExactSum`; +7. live sources; +8. delivery. -Moved records alone would not do. From a moved record, a client cannot tell whether a -formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue -WHERE Region=East)` on page `Region=West` moves when an East record is written. A client -intersects the moved keys with its window by identity. A collapsed fold row has no -maintained key, so a moved member key goes through the client's own member-to-group -remap first. The client then refetches what moved, by `project` or by the successor -query. - -**Every client receives every notice**, and the host holds no per-client state for -delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists -exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a -refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, -`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and -`GridExtent` is derived by the client from category lists. In records mode the row -extent is the admitted record count, which the client fetches. Whether notices should be -filtered per client is open question 3. - -**A connection operator, at the wire boundary.** A connection operator sits at the wire -boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for -results and notices and a source of requests and model commands. Toward the client it is -the mirror image. It owns: -- serialization (T1, T2); -- stamps and the refetch rule (T3); -- coalescing (T5); -- keep-alive in both directions. A client that hears neither a header nor a ping within - the timeout treats the connection as dropped, so silence is never read as "no - batches"; -- session resumption. - -In process it is function calls, as in the TUI and in the browser's `Worker`, which -already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in -`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one -component per connection, or a host half shared per host process with a client half in -every front end), and who owns resumption state, are open question 17. - -### 13. Column widths for text - -The host maintains each text column's maximum display width incrementally. A maximum is -not invertible, so it is an integrated aggregate. Display widths are small integers, so -the state is a histogram from width to count, per (shape key, column): O(distinct widths), -at O(log d) per update. Its input is the delta of the column's text cells over every row, -which the circuit already produces for unfolded cells. Fold rows need the fold maintained -at the visible grain. A text width does not depend on the number format, so there is no -rebuild when the format changes. A client's hidden rows come out through clause 7's overlay. -Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized, -held by the view layer. A number's width would need formatting on the write path, and a -rebuild per format change. Which columns count as text is open question 16. - -### 14. Resident state - -Beside the store, the circuit keeps: -- each cell's store sequence, for clause 5's per-record stage (one integer per cell, - kept in `DataStore`); -- the multisets for `Min`, `Max` and text, O(cells) of those measures; -- `ExactSum` at 568 bytes per summed group; -- occupancy per live shape, O(occupied tuples); -- the restricted-query index, O(occupied cells); -- the width histograms; -- the rollback snapshot per mutating frame. - -The store itself is the integrated base that a rebuild reads, so it is never copied. -State is evicted with its shape, and it sits behind one trace interface, so the -store-memory work (offload to disk, compression) can page or compress it later. None of -these sizes is measured at ADR 008's fixture. Migration step 4 measures them. - -### Relations to other ADRs - -- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its - `document_generation` and record tiers, not on `layout_version`. -- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace; - epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values. - Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a - 256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators. -- **ADR 006:** replay reproduces a circuit's state. -- **ADR 007:** keys become ids, so a rename is a relabel (clause 3). -- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the - occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire - prunes, and which parameters a request carries, are shared with questions 9 and 13 - below. - -## Alternatives considered - -- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio, - mimalloc, zstd and others), about 421 crates for wasm32, and fails - `cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded, - storage-backed engine where improvise needs a single-threaded operator library. -- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only - the cost of not changing: every write still recomputes every formula. -- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule, - which is a second description of evaluation. Z-sets give one rule, linearity, whose laws - are tested once. -- **Compile formulas into operators.** A second evaluator (clause 2). - -## Consequences and migration - -**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches. -The store scans behind the record tier and the dirty flag go, and so do the per-view model -clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources -that can report changes plug in unchanged. The host keeps no per-client state, and one -delivery path serves every front end. - -**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that -must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs -568 bytes per group, and displayed sums change in their last bits. Circularity is decided -in two layers. The network path still clones per mutating frame. Every client receives -every notice. Every column-total read pays O(H log d) for the request's hidden rows. - -**Order.** Each step is a ticket and a PR in the implementation epic, and none switches -the displayed path until the differential test holds: -1. the `zset` crate, with its laws, operator properties and determinism test; -2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`; -3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a - signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty; -4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and - occupancy) under the differential test, with step and rebuild costs measured at ADR - 008's fixture. ADR 008's pruning depends on this step; -5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's - aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price` - and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members` - depends on this step; -6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row - overlay; -7. live sources as delta streams, once ADR 003's source trait exists; -8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one - header and no client refetches; a dropped header is detected; a reconnect after a host - restart refetches and drops nothing as a duplicate; a refused frame consumes no - publication number; and a dropped connection forces a refetch. +Step 8's required tests: +- `[AddItem, SetCell]` publishes one notice, and no client refetches; +- a dropped notice is detected; +- a reconnect after a host restart refetches; +- a refused frame consumes no publication number. ## Open questions -1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page - categories, every page's state is maintained and `[`/`]` only reads another slice. The - held key stays O(visible), and the state stays bounded by the stored cells. If it does - not, a page change builds operators like any other new key. *Recommendation:* the - slice, which clause 7 assumes. +1. **Should notices be filtered per client?** Clause 9 sends every client every live + shape's moved keys. That keeps the host free of per-client state, but traffic is + O(clients × all moved keys) per batch. Filtering by topic would send each connection + only its own shape's key lists. It needs one subscription per connection, which pins + its shape in the cache. Is such a subscription "model-relevant information" the + backend may hold? It could also live in the connection operator (question 4). + *Recommendation:* broadcast with the size cap. Add topics only if measured traffic + with many clients on different shapes justifies them. -2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder, - since a reorder rebuilds nothing. By item identity with the order looked up at read - time, which makes reads O(d). Or by an order label: a key that compares in the current - order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder - re-keys every entry stored under the moved items. *Recommendation:* the order label, - with only the drawn members ordered. Not measured. +2. **How should cycles be handled natively?** Clause 6's component fixed point is + interim. The options: + - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it + has not converged, which keeps today's answers; + - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles + such as `A = IF(x, B, 1)`; + - (c) a monotone reformulation, where one exists. -3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed - stream. The rejected alternative is to send no notice at all and let each client - refetch on its own schedule: it keeps the host silent, but another client's change - stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state, - but every client receives every live shape's moved keys. The options: - - **topics:** a client receives only its own shape key's key lists. Headers still go to - everyone, so the refetch rule keeps working. That costs one subscription per - connection, registered atomically with the fetch that names it, and pinning its - shape; - - **window push:** per-client subscriptions holding the window, sending only the - window's deltas. That costs the most host state. + *Recommendation:* (a). - Is a per-connection subscription "model-relevant information" the backend may hold, - or UI state it must not? It need not live in the host at all if the connection - operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt - topics only if measured traffic, with many clients on different shapes, justifies - them. +3. **What replaces `Eager`?** It still serves as the differential oracle and as the + backend for live sources that can only answer aggregate queries. *Recommendation:* a + small from-scratch reference implementation, kept in the test crate, as the oracle, + and a pushdown backend behind the same trait for query-only sources. -4. **How long does a shape key live?** A key is evicted once no recent request names it. - Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades - memory against refetch storms. If the connection operator re-requests the shapes its - clients read, that is a keep-alive by another name: one host `project` per shape per - eviction period. *Recommendation:* evict after a period with no request, set by - measurement, with hysteresis for shapes with many readers. +4. **What shape is the connection operator?** Either one component per connection, or + one per host process: a host half shared by all its connections, with a client half + in every front end. The second lets the host send one notice per operator rather than + per client, but holds per-connection state in the shared operator. Who owns + resumption state, and is it backend state? *Recommendation:* one per host process, + with resumption state in the host half keyed by a client-held token, treated as + delivery state rather than UI state. In the TUI and the browser worker, with one + client each, it is a function call. -5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs - with different payloads differ, but the wire spells every NaN `"nan"`. A remote client - would therefore see such a change as nothing. *Recommendation:* outputs consolidate - under the wire's identity (every NaN one value), and the record tier keeps bit - identity. - -6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and - column aggregates by items, reports zero crossings per step, and slices rebuilds. That - makes collapse, hide, reorder and add item free, at the cost of recombining a - collapsed group's members on read. *Recommendation:* adopt all three, as written. - -7. **Resident state.** Should the first version keep no second copy of the store (the - store is the integral), put operator state behind one trace interface, and measure - state at ADR 008's fixture before the store-memory work begins? *Recommendation:* - yes. - -8. **Formulas the per-record map cannot express.** They take the occupied-contexts - fallback (clause 5). - - (a) Should `Eager` use the same enumeration, so the two backends agree? That changes - eager's answer only at the bare-ref edge, which could also be kept by checking the - ref once before enumerating. - - (b) Should formulas targeting a category other than `_Measure` take the same - fallback, so that nothing still needs `Expansion::members`? - - *Recommendation:* yes to both. - -9. **Which side of the wire prunes and fills?** On the client, the model offers only - windows and the restricted successor query, and the fill is the client's, with no host - state. On the model, prune is a request parameter, and the model keeps a pruned axis, - with rank and select for "row i of n", and runs the fill as a sliced job resumable - from (request, fill front). At a density of 10⁻⁴, a client without the successor query - would ship mostly empty windows. A second part: when scrolling up or left, how does the - client find a new top-left when rows are pruned? The options are galloping backwards - with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a - model-side predecessor query (one round trip, no new state, but a second query - direction), or failing the fetch and retrying with the nearest occupied row named. ADR - 008 asks the same question. *Recommendation:* the client prunes and fills, and it - gallops. - -10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas - by position instead goes stale on every item insert. Identity costs rank and select - per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as - clause 12 assumes. - -11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers - the restricted successor query. Per-axis counts can only skip rows that are empty - everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned - extent. *Recommendation:* keep them in that role. Not measured. - -12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at - O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape - key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per - distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at - read time. Memoise per hidden set only if a measured hidden set is too large for the - frame budget. - -13. **Which read-time parameters travel in a request?** The page selection travels, as a - third axis. Hidden items and collapsed groups change model-side totals and folds, so - without them a client could not get those from the model. Alternatively, a collapse - could be expressed as an axis at the group's level and a hide as a filter exclusion, - but both change the shape key, which costs a build per state. The drawn order, the - number format and prune change no aggregate. *Recommendation:* hidden items and - collapsed groups travel; the rest stay with the client. - -14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and - validation. The options: - - (a) a recursion operator bounded at sixteen iterations, which writes `circular` - where it has not converged and keeps today's answers; - - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles - such as `A = IF(x, B, 1)`; - - (c) a monotone reformulation, where one exists. - - *Recommendation:* (a) first. - -15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle, - query-only live sources, and the formulas of question 8 that stay eager. - *Recommendation:* a small from-scratch reference implementation, kept in the test - crate, as the oracle; a pushdown backend behind the same trait for query-only - sources; and the fallback of question 8 as a circuit stage. - -16. **Which columns get a host-maintained width?** Columns whose cells are text, with - numbers keeping the sticky width with hysteresis? And what of a column that holds - both? *Recommendation:* text cells get the maintained width, and the view layer takes - the larger of that and its own numeric width. - -17. **What exactly is the connection operator?** (clause 12) - - (a) Do its "both ends" mean both halves in one component, or one component on each - side of the wire? - - (b) Is it a new crate or process, or a role for existing code (the browser - `Worker`, the socket server's `Registry` and transport, in-process code in the TUI)? - - (c) Who owns keep-alive and resumption state, and is that backend state? - - (d) Should one operator serve many connections and subscribe to the host on their - behalf, so that the host sends O(operators) per batch, not O(clients)? That saving - needs one operator per host process serving many clients, so per-connection state - then lives inside a shared operator. - - *Recommendation:* (a) one operator per host process: a host half shared by its - connections, and a client half in every front end. (b) a role of the existing - transport crates, not a new process. (c) the host half holds resumption state, keyed - by a token the client holds, as delivery state, not UI state. (d) yes, for the socket - server. The TUI and the browser worker each serve one client, so there the operator is - a function call. +5. **Which columns get a host-maintained width?** Columns whose cells are text? What + about a column holding both text and numbers? *Recommendation:* text cells get the + maintained width, and the view layer takes the larger of that and its own numeric + width. From 5923f039f7a1b02b0493a225cbada33aab02b51a Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 09:45:46 -0700 Subject: [PATCH 46/67] docs(adr): ADR 009 makes the client pruning recipe conditional and restores the dropped-connection test (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 02310a7b..cbb35871 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -190,11 +190,11 @@ and dimensions. The window is never state, and a request costs O(visible). displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model answers one restricted successor query: "the next k rows after position p with a value in a column of S", and the same for columns. A per-column sorted index of occupied rows -answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its -window, fills forward with that query, and finds a new top-left when scrolling up or left -by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether -pruning runs on the client and which parameters travel are ADR 008's questions. This -design supports either answer. +answers it in O(|S| log n + k|S|), holding O(occupied cells). Whether pruning runs on +the client and which parameters travel are ADR 008's questions, and this design supports +either answer. If the client prunes, it prunes its window, fills forward with that +query, and, when scrolling up or left, finds a new top-left by galloping backwards in +doubling jumps (about 14 to 28 probes at density 10⁻⁴). **The change notice.** After each accepted batch the host emits one notice to every client, addressed to none. It carries: @@ -283,7 +283,8 @@ Step 8's required tests: - `[AddItem, SetCell]` publishes one notice, and no client refetches; - a dropped notice is detected; - a reconnect after a host restart refetches; -- a refused frame consumes no publication number. +- a refused frame consumes no publication number; +- a dropped connection forces a refetch. ## Open questions From 1a7d2fbf85ca88a8751b4dadaac41b944d42d1aa Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 13:34:45 -0700 Subject: [PATCH 47/67] docs(adr): ADR 009 aligns its ADR 007 statements with the accepted amendment (improvise-felo.4) CellKey's order is the natural id order, a store reorder reassigns Seq, CellKey pairs are (CategoryId, ItemHandle), and display suffixes are computed by the client. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index cbb35871..9bc84d21 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -61,14 +61,18 @@ is removed once its remaining uses are replaced (open question 3). `Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite -consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect. +consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the +natural order of ids (ADR 007 amendment 1, clause 4). A store reorder, such as entering +records mode, reassigns `Seq` and rebuilds clause 5's per-record stage. The circuit steps +once per model effect. The record tier and the dirty flag read this delta instead of scanning the store. The dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the formula set moves, because formula edits move no cell. -Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The -formula-set fingerprint must use the id form too, or a rename reads as a formula change. +Under ADR 007 and its amendment, `CellKey` pairs are `(CategoryId, ItemHandle)`. So a +rename is a display write and emits no delta. The formula-set fingerprint must use the id +form too, or a rename reads as a formula change. ### 4. Aggregates @@ -200,7 +204,9 @@ doubling jumps (about 14 to 28 probes at density 10⁻⁴). client, addressed to none. It carries: - the stamp `(host instance, document_generation, publication)`. The publication number increments once per published batch, and the host instance changes on restart; -- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved; +- axis changes, display relabels as `(id, display)`, and a flag when the formula set or + `measure_agg` moved. A shared display's ` #2` suffix is never sent: the client computes + it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as the circuit computes them. Moved records alone would miss formulas with cross-stem From f3a54d34e076b5916c7520e25b58ddc9587dd861 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:18:55 -0700 Subject: [PATCH 48/67] docs(adr): ADR 009 makes every operator edge network-transparent, over up to two hops and request/response transports (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 56 +++++++++++++++-------- 1 file changed, 37 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 9bc84d21..33e43cfe 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -180,8 +180,34 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery -The same circuit serves the in-process TUI and remote clients. Its internal streams never -cross a wire, because aggregation stays on the host. +**Every edge may cross a network.** An edge, meaning the stream from one operator to the +next, may be in-process or remote, and a path may cross up to two network hops: browser ⇄ +gateway ⇄ host, or host ⇄ worker ⇄ client. No edge assumes a function call or shared +memory. An operator that sits at a wire boundary is just one implementation of an edge, +an **edge adapter**. So every edge has the same semantics: +- **A wire form.** Every element type that flows on an edge has a serializable form, + defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays + free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its + tuple of ids, its position or an ADR 002 surrogate. +- **Stamps compose end to end.** The host stamps each published batch (below), and an + intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up + downstream as a gap. +- **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces + exactly: deltas are summed, and notices merged. +- **No long-lived stream assumed.** An edge may be carried as a series of requests and + responses, for example several HTTP requests on one connection, as well as a socket. + So each edge keeps an ordered log, every request can be retried, and a resync from a + publication stamp is idempotent. +- **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is + alive while its next message or request succeeds. Each hop keeps alive in both + directions. A side that hears nothing within the timeout, or whose request fails, + treats the hop as dropped. The downstream side then resyncs: it refetches through the + path, and the host answers from current state. + +Aggregation stays on the model side (ADR 008), so the operators that aggregate run on +the host. Which other edges cross a wire is a deployment choice. The engine's +invalidation signal, which ADR 006's replay trigger relies on, is the change notice and +the refetch rule below, followed from a displayed cell's key back to the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left and dimensions. The window is never state, and a request costs O(visible). @@ -227,11 +253,12 @@ identity, so every NaN counts as one value, because the wire spells every NaN `" state. `GridExtent` is derived by the client, and in records mode the record count is fetched. -**A connection operator** at the wire boundary adapts the circuit to a connection. It is -a sink for results and notices and a source of requests toward the circuit, and the -mirror image toward the client. It owns serialization, the stamp rules, notice -coalescing, keep-alive in both directions, and session resumption. In process it is -function calls. Its shape is open question 4. +**Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for +results and notices and a source of requests. Toward the client it is the mirror image. +Each hop has an adapter at each end, and each end keeps only that hop's state: the last +stamp it applied and its liveness timer. A gateway hop (browser ⇄ gateway ⇄ host) may +serve many clients behind one upstream connection, so the host sends one notice per +gateway rather than one per client. In process, an edge is a function call. ### 10. Text column widths @@ -290,7 +317,7 @@ Step 8's required tests: - a dropped notice is detected; - a reconnect after a host restart refetches; - a refused frame consumes no publication number; -- a dropped connection forces a refetch. +- a dropped hop forces a refetch. ## Open questions @@ -299,7 +326,7 @@ Step 8's required tests: O(clients × all moved keys) per batch. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? It could also live in the connection operator (question 4). + backend may hold? On a two-hop path the topics could instead live in a gateway. *Recommendation:* broadcast with the size cap. Add topics only if measured traffic with many clients on different shapes justifies them. @@ -318,16 +345,7 @@ Step 8's required tests: small from-scratch reference implementation, kept in the test crate, as the oracle, and a pushdown backend behind the same trait for query-only sources. -4. **What shape is the connection operator?** Either one component per connection, or - one per host process: a host half shared by all its connections, with a client half - in every front end. The second lets the host send one notice per operator rather than - per client, but holds per-connection state in the shared operator. Who owns - resumption state, and is it backend state? *Recommendation:* one per host process, - with resumption state in the host half keyed by a client-held token, treated as - delivery state rather than UI state. In the TUI and the browser worker, with one - client each, it is a function call. - -5. **Which columns get a host-maintained width?** Columns whose cells are text? What +4. **Which columns get a host-maintained width?** Columns whose cells are text? What about a column holding both text and numbers? *Recommendation:* text cells get the maintained width, and the view layer takes the larger of that and its own numeric width. From 5e94255c21d86013e83b7c2271d0693b869a6101 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:22:50 -0700 Subject: [PATCH 49/67] docs(adr): ADR 009 requires a lossless edge wire form with round-trip tests (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 33e43cfe..44aeadcb 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -189,6 +189,22 @@ an **edge adapter**. So every edge has the same semantics: defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its tuple of ids, its position or an ADR 002 surrogate. +- **The wire form is lossless.** Nothing is lost in transport, numeric precision above + all: + - a number never crosses as a JSON or JS double. A `numtower` `Num` crosses as its + canonical text (ADR 005). An f64 crosses as its exact bits, which keeps `-0.0` and + NaN payloads. That conflicts with today's `wire_f64` and with `numtower`'s `nan` + token, which both drop the payload, so the edge carries bits for every non-finite + value; + - integers wider than 2⁵³, including positions and counts, cross as strings; + - text crosses byte-exact, with no normalisation; + - the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs + a sum gets the host's exact read or rebuilds it from the exact deltas. + + The stamp rule detects lost messages, but nothing would detect a lossy encoding. So + every element type has a round-trip property test, where encoding then decoding is + the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and + `i128`-wide fractions. - **Stamps compose end to end.** The host stamps each published batch (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. @@ -245,8 +261,8 @@ maintained key, so it maps member keys through its own remap first. It then refe what moved. It refetches everything when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, or a notice it cannot read. A slow client may merge pending notices: the union of keys, the latest -stamp, and axis changes kept in order. Outputs are consolidated under the wire's value -identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`. +stamp, and axis changes kept in order. Because the wire is lossless, outputs keep the +circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client From 353352f89d62e596f9c92dd46ce0cb715d7b7421 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:23:49 -0700 Subject: [PATCH 50/67] docs(adr): ADR 009 gives merged notices a covered range, one shared notice log, and a clock per process (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 29 ++++++++++++++--------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 44aeadcb..99dee2da 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -3,7 +3,7 @@ **Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008 +**Relates to:** ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 ## Problem @@ -25,7 +25,8 @@ equal today's eager results exactly. A problem-agnostic workspace crate with no dependencies beyond `std` (so it is wasm32-clean and serde-free). It provides: - `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; -- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`, +- a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`, + `differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, `distinct` and `aggregate`; - resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it @@ -208,12 +209,18 @@ an **edge adapter**. So every edge has the same semantics: - **Stamps compose end to end.** The host stamps each published batch (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. +- **One clock per process.** Each process runs its own synchronous sub-circuit. Across a + hop, the stamped batches are the clock. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces - exactly: deltas are summed, and notices merged. + exactly: deltas are summed, and notices merged. A merged notice carries the range of + publications it covers, `(from, to]`, so a downstream side sees no gap where nothing + was lost. - **No long-lived stream assumed.** An edge may be carried as a series of requests and responses, for example several HTTP requests on one connection, as well as a socket. - So each edge keeps an ordered log, every request can be retried, and a resync from a - publication stamp is idempotent. + Every request can be retried, and a resync from a publication stamp is idempotent. A + client asks for "notices since n", and the host answers from **one shared, bounded log + of published notices**: a single log, not one per edge or client. A request older than + the log's start is answered "refetch". - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, @@ -221,7 +228,7 @@ an **edge adapter**. So every edge has the same semantics: path, and the host answers from current state. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the host. Which other edges cross a wire is a deployment choice. The engine's +the model side, which may itself span a hop. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger relies on, is the change notice and the refetch rule below, followed from a displayed cell's key back to the model. @@ -260,13 +267,13 @@ A notice whose key lists exceed a fixed size carries "refetch" instead. maintained key, so it maps member keys through its own remap first. It then refetches what moved. It refetches everything when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, or a notice it -cannot read. A slow client may merge pending notices: the union of keys, the latest -stamp, and axis changes kept in order. Because the wire is lossless, outputs keep the +cannot read. A slow client may merge pending notices: the union of keys, the covered +range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state. `GridExtent` is derived by the client, and in records mode the record count is +state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is fetched. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for @@ -287,8 +294,8 @@ come out through clause 7's overlay. Numeric widths stay with ADR 008's view-lay - **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An inverse delta is not a rewind, because `delay` and `differentiate` keep history. - **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, - `ExactSum` per summed group, occupancy per live shape, the successor-query index and - the width histograms. The store itself is the integrated base and is never copied. It + `ExactSum` per summed group, occupancy per live shape, the successor-query index, the + width histograms and the bounded notice log. The store itself is the integrated base and is never copied. It is measured at ADR 008's fixture before the store-memory work begins. ## Alternatives rejected From 731a04ecd560917b25a11026f6b9adde42056f58 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:26:59 -0700 Subject: [PATCH 51/67] docs(adr): ADR 009 answers stale state with an engine correction operation (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 39 +++++++++++++++-------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 99dee2da..4de8bf20 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -220,17 +220,26 @@ an **edge adapter**. So every edge has the same semantics: Every request can be retried, and a resync from a publication stamp is idempotent. A client asks for "notices since n", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client. A request older than - the log's start is answered "refetch". + the log's start is answered with a correction (below). - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, - treats the hop as dropped. The downstream side then resyncs: it refetches through the - path, and the host answers from current state. + treats the hop as dropped. The downstream side then asks for a correction from its + last stamp. + +- **Correction is an engine operation.** The engine offers, per shape key, a + **correction** computed from a client's last stamp. Where the shared log still covers + that stamp, it is the net delta since then. Otherwise it is a replacing delta: the + current state for the client's window, with whatever the client held retracted. A + full refetch is the case with an empty basis. A correction is idempotent, resumable + by stamp, and written in the same lossless delta form as a notice, so a client, a + gateway or a worker applies it with the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the model side, which may itself span a hop. Which other edges cross a wire is a deployment choice. The engine's -invalidation signal, which ADR 006's replay trigger relies on, is the change notice and -the refetch rule below, followed from a displayed cell's key back to the model. +the model side, which may itself span a hop. Which other edges cross a wire is a +deployment choice. The engine's invalidation signal, which ADR 006's replay trigger +relies on, is a correction trigger below, followed from a displayed cell's key back to +the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left and dimensions. The window is never state, and a request costs O(visible). @@ -261,13 +270,13 @@ client, addressed to none. It carries: the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. -A notice whose key lists exceed a fixed size carries "refetch" instead. +A notice whose key lists exceed a fixed size carries "correct" instead. **What a client does.** It intersects the moved keys with its window. A fold row has no -maintained key, so it maps member keys through its own remap first. It then refetches -what moved. It refetches everything when it sees a publication gap, a new -`document_generation` or host instance, its shape in the eviction list, or a notice it -cannot read. A slow client may merge pending notices: the union of keys, the covered +maintained key, so it maps member keys through its own remap first. It then fetches what +moved. It asks for a correction when it sees a publication gap, a new +`document_generation` or host instance, its shape in the eviction list, a notice it +cannot read, or a dropped hop. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. @@ -336,11 +345,13 @@ test): 8. delivery. Step 8's required tests: -- `[AddItem, SetCell]` publishes one notice, and no client refetches; +- `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; - a dropped notice is detected; -- a reconnect after a host restart refetches; +- a reconnect after a host restart asks for a correction; +- a corrected state equals the state of a client that was never stale (a property + test over random edit sequences and losses); - a refused frame consumes no publication number; -- a dropped hop forces a refetch. +- a dropped hop triggers a correction. ## Open questions From 2bea189c6743c3ad04a1f7448160498d4b357998 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:28:49 -0700 Subject: [PATCH 52/67] docs(adr): ADR 009 states correction as the Z-set difference target minus held (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 27 +++++++++++++++-------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 4de8bf20..0687f53e 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -227,13 +227,21 @@ an **edge adapter**. So every edge has the same semantics: treats the hop as dropped. The downstream side then asks for a correction from its last stamp. -- **Correction is an engine operation.** The engine offers, per shape key, a - **correction** computed from a client's last stamp. Where the shared log still covers - that stamp, it is the net delta since then. Otherwise it is a replacing delta: the - current state for the client's window, with whatever the client held retracted. A - full refetch is the case with an empty basis. A correction is idempotent, resumable - by stamp, and written in the same lossless delta form as a notice, so a client, a - gateway or a worker applies it with the same code. +- **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` + is the current state for the client's window and `held` is what the client holds. + Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so + the group structure of `ZSet` provides it, with no separate machinery. The client + sends its last stamp: + - where the shared log covers that stamp, the host sends the sum of the logged deltas + since then; + - otherwise the host sends `target`, and the client, which knows `held`, applies + `target − held`; + - a full refetch is `target − ∅`. + + A correction is labelled with the publication range it covers, so it is applied only + at the stamp it starts from, which makes it idempotent and resumable. It uses the same + lossless delta form as a notice, so a client, a gateway or a worker applies it with the + same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -348,8 +356,9 @@ Step 8's required tests: - `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; - a dropped notice is detected; - a reconnect after a host restart asks for a correction; -- a corrected state equals the state of a client that was never stale (a property - test over random edit sequences and losses); +- a corrected state equals the target, `held + (target − held) = target`, and + correcting a state that is already correct is the zero Z-set (property tests over + random edit sequences and losses); - a refused frame consumes no publication number; - a dropped hop triggers a correction. From 7ea88606e28fba9f8741fd8a37e119e2c77867c1 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:29:21 -0700 Subject: [PATCH 53/67] docs(adr): ADR 009 names the client's position a watermark (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0687f53e..ad507a64 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -225,21 +225,24 @@ an **edge adapter**. So every edge has the same semantics: alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, treats the hop as dropped. The downstream side then asks for a correction from its - last stamp. + watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` is the current state for the client's window and `held` is what the client holds. Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so - the group structure of `ZSet` provides it, with no separate machinery. The client - sends its last stamp: - - where the shared log covers that stamp, the host sends the sum of the logged deltas - since then; + the group structure of `ZSet` provides it, with no separate machinery. The client's + position is a **watermark**: the publication number up to which it has applied the + stream, so everything at or before it is applied. There is one publication sequence, + so a watermark is one number, held per shape key the client reads. The host and any + gateway keep no watermarks: a client sends its own with each request. + - where the shared log covers the watermark, the host sends the sum of the logged + deltas after it; - otherwise the host sends `target`, and the client, which knows `held`, applies `target − held`; - a full refetch is `target − ∅`. A correction is labelled with the publication range it covers, so it is applied only - at the stamp it starts from, which makes it idempotent and resumable. It uses the same + at the watermark it starts from, which makes it idempotent and resumable. It uses the same lossless delta form as a notice, so a client, a gateway or a worker applies it with the same code. @@ -295,8 +298,8 @@ fetched. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for results and notices and a source of requests. Toward the client it is the mirror image. -Each hop has an adapter at each end, and each end keeps only that hop's state: the last -stamp it applied and its liveness timer. A gateway hop (browser ⇄ gateway ⇄ host) may +Each hop has an adapter at each end, and each end keeps only that hop's state: its +liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ host) may serve many clients behind one upstream connection, so the host sends one notice per gateway rather than one per client. In process, an edge is a function call. From 5239d3b95a54b70a045dc4a0625f7953b5ace290 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:31:39 -0700 Subject: [PATCH 54/67] docs(adr): ADR 009 aligns with ADRs 005, 006, 007 and 008 after the cross-read (improvise-felo.4) Folds use the natural CellKey order and Seq is gone; no view action reorders the store; rollback joins the per-command snapshot; a notice per committed command, with a view-definitions flag; the non-finite wire change to ADR 005 is stated; the watermark is the whole stamp and corrections are restricted to the window; deltas cross a hop only after commit; the totals overlay is conditional on ADR 008; this ADR owns the text-width rule. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 141 ++++++++++++---------- 1 file changed, 78 insertions(+), 63 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index ad507a64..794658c7 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -3,7 +3,8 @@ **Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Relates to:** ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 +**Relates to:** ADR 001 (the basis a write claims), ADR 002 (surrogates on the wire), +ADR 003, ADR 005, ADR 006, ADR 007, ADR 008 ## Problem @@ -26,9 +27,8 @@ A problem-agnostic workspace crate with no dependencies beyond `std` (so it is wasm32-clean and serde-free). It provides: - `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; - a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`, - `differentiate`, - `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, - `distinct` and `aggregate`; + `differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join + `join_indexed`, `distinct` and `aggregate`; - resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it can run as a sliced job; - operator state behind one trace interface (look up, read a range, apply a delta), so @@ -54,18 +54,15 @@ is removed once its remaining uses are replaced (open question 3). ### 3. The cell delta -`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`: -- `+(k, v, s)` for a new cell; -- `−(k, v₀, s) + (k, v₁, s)` for an overwrite; -- `−(k, v₀, s)` for a removal; -- a remove-then-set gets a fresh `s′`. +`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue)>`: +- `+(k, v)` for a new cell; +- `−(k, v₀) + (k, v₁)` for an overwrite; +- `−(k, v₀)` for a removal. -`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an -overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite +Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the -natural order of ids (ADR 007 amendment 1, clause 4). A store reorder, such as entering -records mode, reassigns `Seq` and rebuilds clause 5's per-record stage. The circuit steps -once per model effect. +natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The +circuit steps once per model effect. The record tier and the dirty flag read this delta instead of scanning the store. The dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the @@ -107,8 +104,8 @@ The affected pairs are evaluated in the formula graph's topological order. **An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: 1. A per-record stage retracts and reinserts each touched record. It reads the record's previous cells as the store minus the delta, so it keeps no copy of the values. -2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum - `Seq` of the record's live cells), which is eager's concatenation order. +2. A map produces `(context, order key, expr)`. The order key is (context rank, record + key in `CellKey` order), which is eager's concatenation order. 3. The group aggregate, with an error channel that reads the first `Err` in order. No cross product is built. A formula whose expression reads other records is evaluated @@ -150,9 +147,11 @@ changes no operator. The visible axis is a remap. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). -**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row -leaves it. The shape key is shared and a hide is one client's, so totals are kept over -all rows. A read subtracts the request's H hidden cells as a temporary overlay: +**Column totals and hidden rows.** This holds if hidden items travel in a request, which +is ADR 008's open question. If they do not, totals are over all rows. A total pools only +the drawn rows, so a hidden row leaves it. The shape key is shared and a hide is one +client's, so totals are kept over all rows. A read subtracts the request's H hidden cells +as a temporary overlay: - `Count` subtracts; - `Min` and `Max` skip keys whose whole count is hidden; - text counts the distinct keys left; @@ -192,12 +191,13 @@ an **edge adapter**. So every edge has the same semantics: tuple of ids, its position or an ADR 002 surrogate. - **The wire form is lossless.** Nothing is lost in transport, numeric precision above all: - - a number never crosses as a JSON or JS double. A `numtower` `Num` crosses as its - canonical text (ADR 005). An f64 crosses as its exact bits, which keeps `-0.0` and - NaN payloads. That conflicts with today's `wire_f64` and with `numtower`'s `nan` - token, which both drop the payload, so the edge carries bits for every non-finite - value; - - integers wider than 2⁵³, including positions and counts, cross as strings; + - a value never crosses as a JSON or JS double. A `numtower` `Num` crosses as its + canonical text (ADR 005), and an inexact finite value keeps its `#i` spelling, which + is already bit-exact, `-0.0` included. A non-finite value crosses as its bits, which + keeps NaN payloads. **That changes accepted ADR 005 clause 11**, whose wire spells + non-finite values as `inf`, `-inf` and `nan`, and it needs approval with this ADR; + - positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as + strings; - text crosses byte-exact, with no normalisation; - the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs a sum gets the host's exact read or rebuilds it from the exact deltas. @@ -206,11 +206,12 @@ an **edge adapter**. So every edge has the same semantics: every element type has a round-trip property test, where encoding then decoding is the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and `i128`-wide fractions. -- **Stamps compose end to end.** The host stamps each published batch (below), and an +- **Stamps compose end to end.** The host stamps each committed command (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap. - **One clock per process.** Each process runs its own synchronous sub-circuit. Across a - hop, the stamped batches are the clock. + hop, the stamped commands are the clock. A delta crosses a hop only after its command + commits, so a remote operator never sees a delta that is later rolled back. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces exactly: deltas are summed, and notices merged. A merged notice carries the range of publications it covers, `(from, to]`, so a downstream side sees no gap where nothing @@ -219,8 +220,9 @@ an **edge adapter**. So every edge has the same semantics: responses, for example several HTTP requests on one connection, as well as a socket. Every request can be retried, and a resync from a publication stamp is idempotent. A client asks for "notices since n", and the host answers from **one shared, bounded log - of published notices**: a single log, not one per edge or client. A request older than - the log's start is answered with a correction (below). + of published notices**: a single log, not one per edge or client, holding the notices + themselves (keys and flags, not values). A request older than the log's start is + answered with a correction (below). - **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, @@ -231,20 +233,23 @@ an **edge adapter**. So every edge has the same semantics: is the current state for the client's window and `held` is what the client holds. Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so the group structure of `ZSet` provides it, with no separate machinery. The client's - position is a **watermark**: the publication number up to which it has applied the - stream, so everything at or before it is applied. There is one publication sequence, - so a watermark is one number, held per shape key the client reads. The host and any + position is a **watermark**: the whole stamp `(host instance, document_generation, + publication)` up to which it has applied the stream, so everything at or before it is + applied. A watermark from another host instance or generation is behind every current + stamp. The client holds one per shape key it reads. The host and any gateway keep no watermarks: a client sends its own with each request. - - where the shared log covers the watermark, the host sends the sum of the logged - deltas after it; - - otherwise the host sends `target`, and the client, which knows `held`, applies + - where the shared log covers the watermark, the host sends the current values of the + keys the log lists as moved since then, restricted to the client's window, and the + client applies `target − held` over those keys. The log holds keys and flags; values + are always read from the model's current state, so nothing per client is kept; + - otherwise the host sends `target` for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. - A correction is labelled with the publication range it covers, so it is applied only - at the watermark it starts from, which makes it idempotent and resumable. It uses the same - lossless delta form as a notice, so a client, a gateway or a worker applies it with the - same code. + A correction is labelled with the publication range it covers, so it is applied only at + the watermark it starts from, which makes it idempotent and resumable. Its elements are + `(key, value)` pairs with weights, in the lossless wire form, so a client, a gateway or a + worker applies it with the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -269,12 +274,14 @@ either answer. If the client prunes, it prunes its window, fills forward with th query, and, when scrolling up or left, finds a new top-left by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). -**The change notice.** After each accepted batch the host emits one notice to every -client, addressed to none. It carries: +**The change notice.** Each command commits on its own (ADR 006), and after each +committed command the host emits one notice to every client, addressed to none. A frame +whose later command fails keeps its earlier commands, and their notices, published. A +notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number - increments once per published batch, and the host instance changes on restart; -- axis changes, display relabels as `(id, display)`, and a flag when the formula set or - `measure_agg` moved. A shared display's ` #2` suffix is never sent: the client computes + increments once per committed command, and the host instance changes on restart; +- axis changes, display relabels as `(id, display)`, and a flag when the formula set, + `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as @@ -287,35 +294,41 @@ A notice whose key lists exceed a fixed size carries "correct" instead. maintained key, so it maps member keys through its own remap first. It then fetches what moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it -cannot read, or a dropped hop. A slow client may merge pending notices: the union of keys, the covered -range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the -circuit's bit identity end to end. +cannot read, or a dropped hop. A slow client may merge pending notices: the union of +keys, the covered range of publications, and axis changes kept in order. Because the +wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is -fetched. +state, only the shared notice log. `GridExtent` is derived by the client, and in records +mode the record count is fetched. `Projection` also published the ADR 001 basis a write +claims. Whether `project` returns it instead, and what replaces the layout tier, are +ADR 008's questions. **Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for results and notices and a source of requests. Toward the client it is the mirror image. Each hop has an adapter at each end, and each end keeps only that hop's state: its -liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ host) may -serve many clients behind one upstream connection, so the host sends one notice per -gateway rather than one per client. In process, an edge is a function call. +liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ +host) may serve many clients behind one upstream connection, so the host sends one notice +per gateway rather than one per client. In process, an edge is a function call. ### 10. Text column widths -The host maintains each text column's maximum display width as a histogram from width to -count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows -come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule. +This ADR owns the rule for text columns. The host maintains each text column's maximum +display width as a histogram from width to count, per (shape key, column), fed by the +delta of the column's text cells. Hidden rows come out through clause 7's overlay, where +that applies. Numeric widths stay with ADR 008's view-layer rule. ### 11. Rollback and memory -- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An - inverse delta is not a rewind, because `delay` and `differentiate` keep history. -- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, - `ExactSum` per summed group, occupancy per live shape, the successor-query index, the - width histograms and the bounded notice log. The store itself is the integrated base and is never copied. It +- **Rollback.** The circuit's state joins ADR 006's per-command snapshot, which is taken + before a command's effects and restored if the command fails. An inverse delta is not + a rewind, because `delay` and `differentiate` keep history. The requirement is that + this snapshot does not cost O(cells) per command against ADR 008's 8 ms budget, and + that writes do not block reads. How is for the implementation epic. +- **Memory.** Resident state beside the store is the multisets, `ExactSum` per summed + group, occupancy per live shape, the successor-query index, the width histograms and + the bounded notice log. The store itself is the integrated base and is never copied. It is measured at ADR 008's fixture before the store-memory work begins. ## Alternatives rejected @@ -342,7 +355,7 @@ come out through clause 7's overlay. Numeric widths stay with ADR 008's view-lay - Integrated state is O(cells) beside the store. - `ExactSum` costs 568 bytes per summed group. - Every client receives every notice. -- A mutating frame still clones the state for rollback. +- Each command snapshots the circuit's state for rollback. **Order** (each step is a ticket in the implementation epic, behind the differential test): @@ -356,13 +369,15 @@ test): 8. delivery. Step 8's required tests: -- `[AddItem, SetCell]` publishes one notice, and no client asks for a correction; +- a frame `[AddItem, SetCell]` publishes one notice per committed command, numbered + consecutively, and no client asks for a correction; - a dropped notice is detected; - a reconnect after a host restart asks for a correction; - a corrected state equals the target, `held + (target − held) = target`, and correcting a state that is already correct is the zero Z-set (property tests over random edit sequences and losses); -- a refused frame consumes no publication number; +- a refused command consumes no publication number, and the commands before it in its + frame stay published; - a dropped hop triggers a correction. ## Open questions From 13b3dff35e09095ad1e8126c4c6cbf02004e5550 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 21:35:35 -0700 Subject: [PATCH 55/67] docs(adr): ADR 009 reads only committed state and sends a whole window when a range moved positions (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 44 +++++++++++++---------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 794658c7..c854b1aa 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -140,18 +140,19 @@ aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or ad changes no operator. The visible axis is a remap. - **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas - arriving during the build are buffered and added at install. + arriving during the build are buffered and added at install. The finished result is the + response to the request that asked for it, not a notice. - **A formula edit** is a formula delta fed to the circuit. - **A document swap** rebuilds everything. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). -**Column totals and hidden rows.** This holds if hidden items travel in a request, which -is ADR 008's open question. If they do not, totals are over all rows. A total pools only -the drawn rows, so a hidden row leaves it. The shape key is shared and a hide is one -client's, so totals are kept over all rows. A read subtracts the request's H hidden cells -as a temporary overlay: +**Column totals and hidden rows.** Totals are kept over all rows, because the shape key +is shared and a hide is one client's. Whether hidden items travel in a request is ADR +008's open question. If they do not, a total is read as it is kept. If they do, a total +must leave the hidden rows out, so a read subtracts the request's H hidden cells as a +temporary overlay: - `Count` subtracts; - `Min` and `Max` skip keys whose whole count is hidden; - text counts the distinct keys left; @@ -218,8 +219,8 @@ an **edge adapter**. So every edge has the same semantics: was lost. - **No long-lived stream assumed.** An edge may be carried as a series of requests and responses, for example several HTTP requests on one connection, as well as a socket. - Every request can be retried, and a resync from a publication stamp is idempotent. A - client asks for "notices since n", and the host answers from **one shared, bounded log + Every request can be retried, and a resync from a watermark is idempotent. A client + asks for "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). @@ -238,18 +239,23 @@ an **edge adapter**. So every edge has the same semantics: applied. A watermark from another host instance or generation is behind every current stamp. The client holds one per shape key it reads. The host and any gateway keep no watermarks: a client sends its own with each request. - - where the shared log covers the watermark, the host sends the current values of the - keys the log lists as moved since then, restricted to the client's window, and the - client applies `target − held` over those keys. The log holds keys and flags; values - are always read from the model's current state, so nothing per client is kept; - - otherwise the host sends `target` for the whole window, and the client applies - `target − held`; + - where the shared log covers the watermark and the covered range holds only moved + keys, the host sends the current values of those keys, restricted to the client's + window, and the client applies `target − held` over them. The log holds keys and + flags, not values, so nothing per client is kept; + - where the covered range holds an axis change, a relabel or a flag, which position + and labels depend on, the host sends `target` for the whole window; + - where the log no longer covers the watermark, the host sends `target` for the whole + window, and the client applies `target − held`; - a full refetch is `target − ∅`. - A correction is labelled with the publication range it covers, so it is applied only at - the watermark it starts from, which makes it idempotent and resumable. Its elements are - `(key, value)` pairs with weights, in the lossless wire form, so a client, a gateway or a - worker applies it with the same code. + **Every read sees only the last committed state**, whether it is a correction, a + fetch or `project`, and it is labelled with that state's stamp. A value a failed + command wrote and then rolled back is never read. A correction is labelled with the + publication range it covers, so it is applied only at the watermark it starts from, + which makes it idempotent and resumable. Its elements are `(key, value)` pairs with + weights, in the lossless wire form, so a client, a gateway or a worker applies it with + the same code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a @@ -384,7 +390,7 @@ Step 8's required tests: 1. **Should notices be filtered per client?** Clause 9 sends every client every live shape's moved keys. That keeps the host free of per-client state, but traffic is - O(clients × all moved keys) per batch. Filtering by topic would send each connection + O(clients × all moved keys) per committed command. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the backend may hold? On a two-hop path the topics could instead live in a gateway. From e720f7ca509f0643981765ff6b9b182fb0a1bc25 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:11:37 -0700 Subject: [PATCH 56/67] docs(adr): ADR 009 makes the formula language the query language and states the incremental principle (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 37 ++++++++++++++++------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index c854b1aa..0e14cdd1 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -21,6 +21,10 @@ cell write becomes a delta, aggregates are maintained, and only the formula cell inputs moved are re-evaluated by the existing evaluator. The incremental results must equal today's eager results exactly. +**Principle: everything is as incremental as possible**, so the work done per model +mutation is minimised. Where this ADR does work that grows with more than the +mutation, it says so and names the incremental path that normally applies. + ### 1. The `zset` crate A problem-agnostic workspace crate with no dependencies beyond `std` (so it is @@ -34,8 +38,11 @@ wasm32-clean and serde-free). It provides: - operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it. -There are no nested streams, no recursion operator, no timestamps beyond one clock, and -no query language. +There are no nested streams, no recursion operator and no timestamps beyond one clock. +**The formula language is the query language.** What a request asks of the model beyond +its axes and window (filters, aggregation, derived values) is written in formula text +over ids (ADR 007). It goes through the same evaluator and circuit as a document +formula, so there is no second query language. ### 2. The plug @@ -85,7 +92,9 @@ shape key with the same None list, which removes the per-session model clone. A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the -ordering category's order. How that order is keyed is for those tickets. +ordering category's order. Those tickets choose the key. It must keep a reorder or an +insert incremental: only the entries under the moved items change, and nothing is +rebuilt. ### 5. Formulas @@ -118,7 +127,8 @@ before enumerating. This replaces `Expansion::members`. DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the answer must be `Error("circular")`. The first version finds the formula graph's strongly -connected components. Acyclic formulas are evaluated once each, in topological order. A +connected components, recomputed when the formula set changes; how to keep that +incremental belongs with open question 2. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced by a DBSP-native treatment (open question 2). @@ -143,7 +153,8 @@ changes no operator. The visible axis is a remap. arriving during the build are buffered and added at install. The finished result is the response to the request that asked for it, not a notice. - **A formula edit** is a formula delta fed to the circuit. -- **A document swap** rebuilds everything. +- **A document swap** rebuilds everything, and a new key's first build is a full pass. + These are the only rebuilds; every other change is a delta. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). @@ -243,10 +254,11 @@ an **edge adapter**. So every edge has the same semantics: keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and flags, not values, so nothing per client is kept; - - where the covered range holds an axis change, a relabel or a flag, which position - and labels depend on, the host sends `target` for the whole window; - - where the log no longer covers the watermark, the host sends `target` for the whole - window, and the client applies `target − held`; + - where the covered range also holds axis changes, relabels or flags, which positions + and labels depend on, the correction carries them too, in order, as a merged notice + does, so the client re-derives positions incrementally; + - only where the log no longer covers the watermark does the host fall back to `target` + for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. **Every read sees only the last committed state**, whether it is a correction, a @@ -294,7 +306,8 @@ notice carries: the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. -A notice whose key lists exceed a fixed size carries "correct" instead. +A notice whose key lists exceed a fixed size carries "correct" instead. That is a +fallback, and it costs every reader a correction. **What a client does.** It intersects the moved keys with its window. A fold row has no maintained key, so it maps member keys through its own remap first. It then fetches what @@ -405,7 +418,9 @@ Step 8's required tests: such as `A = IF(x, B, 1)`; - (c) a monotone reformulation, where one exists. - *Recommendation:* (a). + Separately, the strongly connected components are recomputed whenever the formula set + changes. Should they be maintained incrementally too? *Recommendation:* (a), and + recompute the components only for the formulas an edit touches. 3. **What replaces `Eager`?** It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. *Recommendation:* a From 63ee13289bbdbfb781a348e0bf6246a50678b40b Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:22:19 -0700 Subject: [PATCH 57/67] docs(adr): ADR 009 makes an edge a session rather than a two-hop path, and its clock causal (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 65 ++++++++++++----------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0e14cdd1..0568dab0 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -157,7 +157,8 @@ changes no operator. The visible axis is a remap. These are the only rebuilds; every other change is a delta. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names - it (the period is set by measurement), and every eviction is announced (clause 9). + it. The period is a local duration on the host, set by measurement, and never compared + with another machine's clock. Every eviction is announced (clause 9). **Column totals and hidden rows.** Totals are kept over all rows, because the shape key is shared and a hide is one client's. Whether hidden items travel in a request is ADR @@ -193,8 +194,7 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery **Every edge may cross a network.** An edge, meaning the stream from one operator to the -next, may be in-process or remote, and a path may cross up to two network hops: browser ⇄ -gateway ⇄ host, or host ⇄ worker ⇄ client. No edge assumes a function call or shared +next, may be in-process or remote. No edge assumes a function call or shared memory. An operator that sits at a wire boundary is just one implementation of an edge, an **edge adapter**. So every edge has the same semantics: - **A wire form.** Every element type that flows on an edge has a serializable form, @@ -218,27 +218,30 @@ an **edge adapter**. So every edge has the same semantics: every element type has a round-trip property test, where encoding then decoding is the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and `i128`-wide fractions. -- **Stamps compose end to end.** The host stamps each committed command (below), and an - intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up - downstream as a gap. -- **One clock per process.** Each process runs its own synchronous sub-circuit. Across a - hop, the stamped commands are the clock. A delta crosses a hop only after its command +- **The clock is causal, never a wall clock.** Ordering, staleness and loss detection + use only the stamp sequence (below), a logical clock the host advances once per + committed command. Each process runs its own synchronous sub-circuit, and across an + edge the stamped commands are the clock. No decision compares wall-clock times across + machines, so clock skew cannot affect it. A delta crosses a wire only after its command commits, so a remote operator never sees a delta that is later rolled back. - **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces exactly: deltas are summed, and notices merged. A merged notice carries the range of - publications it covers, `(from, to]`, so a downstream side sees no gap where nothing - was lost. -- **No long-lived stream assumed.** An edge may be carried as a series of requests and - responses, for example several HTTP requests on one connection, as well as a socket. - Every request can be retried, and a resync from a watermark is idempotent. A client + publications it covers, `(from, to]`, so the client that merges notices sees no gap + where nothing was lost. +- **An edge is a session, not a connection.** An edge may be one long-lived connection, + or a sequence of requests carrying the same session token or cookie, for example + several HTTP requests. The token only lets a client's requests be recognised as one + edge: the host keeps no per-client state, so what continues across connections is the + client's own watermark and shape keys. Every request can be retried, and a resync from + a watermark is idempotent. A client asks for "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). -- **Failure is local to a hop.** A hop may be a socket or a series of requests, and it is - alive while its next message or request succeeds. Each hop keeps alive in both - directions. A side that hears nothing within the timeout, or whose request fails, - treats the hop as dropped. The downstream side then asks for a correction from its +- **Failure.** An edge is alive while its next message or request succeeds, and each end + keeps alive in both directions. Timeouts are local monotonic durations, never compared + across machines. A side that hears nothing within its timeout, or whose request fails, + treats the connection as dropped. The client then asks for a correction from its watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` @@ -248,8 +251,8 @@ an **edge adapter**. So every edge has the same semantics: position is a **watermark**: the whole stamp `(host instance, document_generation, publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current - stamp. The client holds one per shape key it reads. The host and any - gateway keep no watermarks: a client sends its own with each request. + stamp. The client holds one per shape key it reads. The host keeps no watermarks: a + client sends its own with each request. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and @@ -266,12 +269,11 @@ an **edge adapter**. So every edge has the same semantics: command wrote and then rolled back is never read. A correction is labelled with the publication range it covers, so it is applied only at the watermark it starts from, which makes it idempotent and resumable. Its elements are `(key, value)` pairs with - weights, in the lossless wire form, so a client, a gateway or a worker applies it with - the same code. + weights, in the lossless wire form, so a client and a worker apply it with the same + code. Aggregation stays on the model side (ADR 008), so the operators that aggregate run on -the model side, which may itself span a hop. Which other edges cross a wire is a -deployment choice. The engine's invalidation signal, which ADR 006's replay trigger +the model side. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger relies on, is a correction trigger below, followed from a displayed cell's key back to the model. @@ -313,7 +315,7 @@ fallback, and it costs every reader a correction. maintained key, so it maps member keys through its own remap first. It then fetches what moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it -cannot read, or a dropped hop. A slow client may merge pending notices: the union of +cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end. @@ -324,12 +326,10 @@ mode the record count is fetched. `Projection` also published the ADR 001 basis claims. Whether `project` returns it instead, and what replaces the layout tier, are ADR 008's questions. -**Edge adapters at the client.** Toward the circuit, a client's adapter is a sink for -results and notices and a source of requests. Toward the client it is the mirror image. -Each hop has an adapter at each end, and each end keeps only that hop's state: its -liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ -host) may serve many clients behind one upstream connection, so the host sends one notice -per gateway rather than one per client. In process, an edge is a function call. +**Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the +circuit, the client's adapter is a sink for results and notices and a source of requests. +Toward the client it is the mirror image. Each end keeps only its timeout, and the +client's end keeps the watermark. In process, an edge is a function call. ### 10. Text column widths @@ -397,7 +397,8 @@ Step 8's required tests: random edit sequences and losses); - a refused command consumes no publication number, and the commands before it in its frame stay published; -- a dropped hop triggers a correction. +- a dropped connection or a failed request triggers a correction, and a later request + with the same session resumes from the watermark. ## Open questions @@ -406,7 +407,7 @@ Step 8's required tests: O(clients × all moved keys) per committed command. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? On a two-hop path the topics could instead live in a gateway. + backend may hold? *Recommendation:* broadcast with the size cap. Add topics only if measured traffic with many clients on different shapes justifies them. From 978bd4c8c9bbe1b6050a13c5c8f75b9119dcd2ff Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:25:22 -0700 Subject: [PATCH 58/67] docs(adr): ADR 009 makes the session token transport only and keys shapes by the parsed query (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 ++++++++++++----------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 0568dab0..6099ec8b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -144,7 +144,8 @@ with the same key, from any client. The key is: - the page categories, so each page is a maintained slice and paging rebuilds nothing; - `measure_agg`. -Everything else in a request is applied when output is read: hidden and collapsed items +The key uses the canonical parsed id form of the request's query text, so two spellings +of one query share their operator state. Everything else in a request is applied when output is read: hidden and collapsed items on axis categories, item order, the page selection and prune. Occupancy and column aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item changes no operator. The visible axis is a remap. @@ -230,19 +231,19 @@ an **edge adapter**. So every edge has the same semantics: where nothing was lost. - **An edge is a session, not a connection.** An edge may be one long-lived connection, or a sequence of requests carrying the same session token or cookie, for example - several HTTP requests. The token only lets a client's requests be recognised as one - edge: the host keeps no per-client state, so what continues across connections is the - client's own watermark and shape keys. Every request can be retried, and a resync from - a watermark is idempotent. A client - asks for "notices after my watermark", and the host answers from **one shared, bounded log + several HTTP requests. The host decides nothing from the token: it is transport, and + any later authentication is out of scope. What continues across connections is the + client's own watermark and shape keys, which it sends with each request. Every request + can be retried, and a resync from a watermark is idempotent. A client asks for + "notices after my watermark", and the host answers from **one shared, bounded log of published notices**: a single log, not one per edge or client, holding the notices themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). -- **Failure.** An edge is alive while its next message or request succeeds, and each end - keeps alive in both directions. Timeouts are local monotonic durations, never compared - across machines. A side that hears nothing within its timeout, or whose request fails, - treats the connection as dropped. The client then asks for a correction from its - watermark. +- **Failure.** Timeouts are local monotonic durations, never compared across machines. + On a long-lived connection, both ends keep alive, and a host-side timeout belongs to + that open connection and is released with it. On a sequence of requests, only the + client detects failure: a request that fails or times out. In either case the client + then asks for a correction from its watermark. - **Correction is Z-set arithmetic.** A correction is `target − held`, where `target` is the current state for the client's window and `held` is what the client holds. @@ -260,7 +261,8 @@ an **edge adapter**. So every edge has the same semantics: - where the covered range also holds axis changes, relabels or flags, which positions and labels depend on, the correction carries them too, in order, as a merged notice does, so the client re-derives positions incrementally; - - only where the log no longer covers the watermark does the host fall back to `target` + - only where the log no longer covers the watermark, or the covered range holds an + oversized notice that lists no keys, does the host fall back to `target` for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. @@ -299,7 +301,8 @@ committed command the host emits one notice to every client, addressed to none. whose later command fails keeps its earlier commands, and their notices, published. A notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number - increments once per committed command, and the host instance changes on restart; + increments once per committed command. The host instance changes on restart; it is a + random id, never a wall-clock time; - axis changes, display relabels as `(id, display)`, and a flag when the formula set, `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes it from the displays in scope; @@ -328,8 +331,8 @@ ADR 008's questions. **Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the circuit, the client's adapter is a sink for results and notices and a source of requests. -Toward the client it is the mirror image. Each end keeps only its timeout, and the -client's end keeps the watermark. In process, an edge is a function call. +Toward the client it is the mirror image. The client's end keeps the watermark. In +process, an edge is a function call. ### 10. Text column widths From 6bf899c9ec81c64b66b60ce22245085ff27f78b6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:26:21 -0700 Subject: [PATCH 59/67] docs(adr): ADR 009 makes per-client filtering an operator in the graph (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 41 +++++++++++------------ 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 6099ec8b..834b849f 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -57,7 +57,7 @@ computes each value, so formulas are never compiled into operators. **`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. It serves as the oracle of a differential property test, which applies one random edit sequence to both backends and compares every drawn cell, total and occupancy count. It -is removed once its remaining uses are replaced (open question 3). +is removed once its remaining uses are replaced (open question 2). ### 3. The cell delta @@ -128,10 +128,10 @@ before enumerating. This replaces `Expansion::members`. DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the answer must be `Error("circular")`. The first version finds the formula graph's strongly connected components, recomputed when the formula set changes; how to keep that -incremental belongs with open question 2. Acyclic formulas are evaluated once each, in topological order. A +incremental belongs with open question 1. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced -by a DBSP-native treatment (open question 2). +by a DBSP-native treatment (open question 1). ### 7. Operator state follows the request's shape key @@ -297,20 +297,29 @@ query, and, when scrolling up or left, finds a new top-left by galloping backwar doubling jumps (about 14 to 28 probes at density 10⁻⁴). **The change notice.** Each command commits on its own (ADR 006), and after each -committed command the host emits one notice to every client, addressed to none. A frame +committed command the host emits one notice, addressed to no client. A frame whose later command fails keeps its earlier commands, and their notices, published. A notice carries: - the stamp `(host instance, document_generation, publication)`. The publication number increments once per committed command. The host instance changes on restart; it is a random id, never a wall-clock time; - axis changes, display relabels as `(id, display)`, and a flag when the formula set, - `measure_agg`, the view definitions or the default view moved. A shared display's ` #2` suffix is never sent: the client computes - it from the displays in scope; + `measure_agg`, the view definitions or the default view moved. A shared display's + ` #2` suffix is never sent: the client computes it from the displays in scope; - the shape keys evicted since the last notice; - for each live shape key, the moved output keys (cells, formula cells and totals), as the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. +**A client receives only what it needs.** Filtering is part of the DBSP graph, not a +separate mechanism. A client either pulls just the state it needs, or pushes a filter to +the host's side of its connection: a filter operator on its shape key and window, written +in the query language, through which its notices pass. A pushed filter belongs to the open +connection, like that connection's timeout, and is released with it. On a sequence of +requests the client sends it with each request, so the host keeps no per-client state +beyond an open connection. While a connection's filter is open, it counts as a request +naming its shape, so its shape is not evicted. + A notice whose key lists exceed a fixed size carries "correct" instead. That is a fallback, and it costs every reader a correction. @@ -360,8 +369,8 @@ that applies. Numeric widths stay with ADR 008's view-layer rule. - **Compiling formulas into operators:** a second evaluator. - **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a second description of evaluation. -- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic - only by holding per-client state on the host (open question 1). +- **A topic or subscription mechanism beside the graph:** filtering is an operator in + the graph (clause 9). ## Consequences @@ -376,7 +385,6 @@ that applies. Numeric widths stay with ADR 008's view-layer rule. - Two backends exist until `Eager` goes. - Integrated state is O(cells) beside the store. - `ExactSum` costs 568 bytes per summed group. -- Every client receives every notice. - Each command snapshots the circuit's state for rollback. **Order** (each step is a ticket in the implementation epic, behind the differential @@ -405,16 +413,7 @@ Step 8's required tests: ## Open questions -1. **Should notices be filtered per client?** Clause 9 sends every client every live - shape's moved keys. That keeps the host free of per-client state, but traffic is - O(clients × all moved keys) per committed command. Filtering by topic would send each connection - only its own shape's key lists. It needs one subscription per connection, which pins - its shape in the cache. Is such a subscription "model-relevant information" the - backend may hold? - *Recommendation:* broadcast with the size cap. Add topics only if measured traffic - with many clients on different shapes justifies them. - -2. **How should cycles be handled natively?** Clause 6's component fixed point is +1. **How should cycles be handled natively?** Clause 6's component fixed point is interim. The options: - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it has not converged, which keeps today's answers; @@ -426,12 +425,12 @@ Step 8's required tests: changes. Should they be maintained incrementally too? *Recommendation:* (a), and recompute the components only for the formulas an edit touches. -3. **What replaces `Eager`?** It still serves as the differential oracle and as the +2. **What replaces `Eager`?** It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. *Recommendation:* a small from-scratch reference implementation, kept in the test crate, as the oracle, and a pushdown backend behind the same trait for query-only sources. -4. **Which columns get a host-maintained width?** Columns whose cells are text? What +3. **Which columns get a host-maintained width?** Columns whose cells are text? What about a column holding both text and numbers? *Recommendation:* text cells get the maintained width, and the view layer takes the larger of that and its own numeric width. From 81f1b5e13a4e6a3e1e4e90279cf0eaa37125b323 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:28:52 -0700 Subject: [PATCH 60/67] docs(adr): ADR 009 settles incremental cycles, a temporary Eager, and per-column width hints (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 62 ++++++++++------------- 1 file changed, 26 insertions(+), 36 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 834b849f..25573031 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -54,10 +54,12 @@ depends on `zset` and improvise-core. improvise-core never depends on `zset`. The circuit decides *which* formula cells to re-evaluate. The one existing evaluator computes each value, so formulas are never compiled into operators. -**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. -It serves as the oracle of a differential property test, which applies one random edit -sequence to both backends and compares every drawn cell, total and occupancy count. It -is removed once its remaining uses are replaced (open question 2). +**`Eager` is temporary.** Today's code moves behind the trait as `Eager`, unchanged. It +exists only to show discrepancies between the new evaluator and the old one: a +differential property test applies one random edit sequence to both backends and +compares every drawn cell, total and occupancy count. Eventually it is deleted, and +nothing depends on it surviving. Long-term tests compare against exact references, such +as `ExactSum` against rational sums, not against `Eager`. ### 3. The cell delta @@ -126,12 +128,13 @@ before enumerating. This replaces `Expansion::members`. ### 6. Circular references DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the -answer must be `Error("circular")`. The first version finds the formula graph's strongly -connected components, recomputed when the formula set changes; how to keep that -incremental belongs with open question 1. Acyclic formulas are evaluated once each, in topological order. A -component is evaluated with today's bounded fixed point, restricted to that component. -That reproduces the sixteen-pass bound for the differential test. It is to be replaced -by a DBSP-native treatment (open question 1). +answer must be `Error("circular")`. The formula graph's strongly connected components +are maintained incrementally as formulas change. Acyclic formulas are evaluated once +each, in topological order. A cyclic component is maintained incrementally too: a change +re-iterates only the cells it reaches. A component that does not converge within the +existing bound of sixteen passes (`MAX_EVAL_DEPTH`) propagates `Error("circular")`, as +today. An optimisation that speeds convergence is allowed only where it is exact: it +gives the same result as the plain iteration, and a test compares the two. ### 7. Operator state follows the request's shape key @@ -343,12 +346,12 @@ circuit, the client's adapter is a sink for results and notices and a source of Toward the client it is the mirror image. The client's end keeps the watermark. In process, an edge is a function call. -### 10. Text column widths +### 10. Column width hints -This ADR owns the rule for text columns. The host maintains each text column's maximum -display width as a histogram from width to count, per (shape key, column), fed by the -delta of the column's text cells. Hidden rows come out through clause 7's overlay, where -that applies. Numeric widths stay with ADR 008's view-layer rule. +Every column incrementally maintains a width hint, which optimises grid sizing and +reduces jitter. The hint is an integrated aggregate: a histogram from display width to +count, so a retraction is O(log d). Which end of the connection maintains it is open +question 1. ### 11. Rollback and memory @@ -413,24 +416,11 @@ Step 8's required tests: ## Open questions -1. **How should cycles be handled natively?** Clause 6's component fixed point is - interim. The options: - - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it - has not converged, which keeps today's answers; - - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles - such as `A = IF(x, B, 1)`; - - (c) a monotone reformulation, where one exists. - - Separately, the strongly connected components are recomputed whenever the formula set - changes. Should they be maintained incrementally too? *Recommendation:* (a), and - recompute the components only for the formulas an edit touches. - -2. **What replaces `Eager`?** It still serves as the differential oracle and as the - backend for live sources that can only answer aggregate queries. *Recommendation:* a - small from-scratch reference implementation, kept in the test crate, as the oracle, - and a pushdown backend behind the same trait for query-only sources. - -3. **Which columns get a host-maintained width?** Columns whose cells are text? What - about a column holding both text and numbers? *Recommendation:* text cells get the - maintained width, and the view layer takes the larger of that and its own numeric - width. +1. **Which end of the connection maintains a column's width hint?** The client end knows + which values are potentially in its viewport, so it can maintain the hint over the + values it holds, fed by the deltas it already receives. The host end can maintain it + over a whole column, through the graph, including values the client has never held. + It would then need the request's hidden rows (clause 7's overlay) and, for numbers, the + number format. *Recommendation:* the client end, over the values it holds. The host + computes a whole-column hint only where a client asks for one, as a filter operator + on its connection (clause 9). From b8a0b6a846f14e079b66040e7fb33991bd771d19 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:31:45 -0700 Subject: [PATCH 61/67] docs(adr): ADR 009 keeps the window out of a pushed filter and names query pushdown sources (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 25573031..aab7b93d 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -51,6 +51,10 @@ formula result at a key, and the raw fold of the cells that match a partial key. incremental backend implements it in a seam crate, `improvise-incremental`, which depends on `zset` and improvise-core. improvise-core never depends on `zset`. +A live source (ADR 003) that reports its changes feeds cell deltas like `DataStore`. A +source that answers aggregate queries but reports no changes implements the derived-value +operator directly, by query pushdown, with no change feed. + The circuit decides *which* formula cells to re-evaluate. The one existing evaluator computes each value, so formulas are never compiled into operators. @@ -316,8 +320,10 @@ notice carries: **A client receives only what it needs.** Filtering is part of the DBSP graph, not a separate mechanism. A client either pulls just the state it needs, or pushes a filter to -the host's side of its connection: a filter operator on its shape key and window, written -in the query language, through which its notices pass. A pushed filter belongs to the open +the host's side of its connection: a filter operator on its shape's query, written in the +query language, through which its notices pass. The filter never names the window, which +stays the client's: the client intersects the moved keys of a notice or correction with +its own window. A pushed filter belongs to the open connection, like that connection's timeout, and is released with it. On a sequence of requests the client sends it with each request, so the host keeps no per-client state beyond an open connection. While a connection's filter is open, it counts as a request From 14430b5628ba9f8072d40db0a6db7d6de79763c6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:32:37 -0700 Subject: [PATCH 62/67] docs(adr): ADR 009 requires self-contained operators and leaves width-hint placement to scheduling (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index aab7b93d..76611600 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -38,6 +38,11 @@ wasm32-clean and serde-free). It provides: - operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it. +**Operators are self-contained.** An operator's state and behaviour depend only on its +inputs and its own state, never on a shared in-process global or on another operator's +internals. That keeps the graph distributable: in principle the model could run across a +cluster of machines. Nothing more is designed for that here. + There are no nested streams, no recursion operator and no timestamps beyond one clock. **The formula language is the query language.** What a request asks of the model beyond its axes and window (filters, aggregation, derived values) is written in formula text @@ -356,8 +361,8 @@ process, an edge is a function call. Every column incrementally maintains a width hint, which optimises grid sizing and reduces jitter. The hint is an integrated aggregate: a histogram from display width to -count, so a retraction is O(log d). Which end of the connection maintains it is open -question 1. +count, so a retraction is O(log d). Where in the graph the hint is computed is a +scheduling choice, made as an optimisation later. ### 11. Rollback and memory @@ -422,11 +427,4 @@ Step 8's required tests: ## Open questions -1. **Which end of the connection maintains a column's width hint?** The client end knows - which values are potentially in its viewport, so it can maintain the hint over the - values it holds, fed by the deltas it already receives. The host end can maintain it - over a whole column, through the graph, including values the client has never held. - It would then need the request's hidden rows (clause 7's overlay) and, for numbers, the - number format. *Recommendation:* the client end, over the values it holds. The host - computes a whole-column hint only where a client asks for one, as a filter operator - on its connection (clause 9). +None. From b26c92faf7e128b0a8197b259713af57fa395de6 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:35:01 -0700 Subject: [PATCH 63/67] docs(adr): ADR 009 states the model and view stores as conceptual boxes, not a network boundary (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 33 +++++++++++++---------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 76611600..e08ba4b7 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -183,7 +183,8 @@ temporary overlay: - text counts the distinct keys left; - `Sum` subtracts exactly through the accumulator. -That is O(H log d) per read for every aggregate, with no per-client state. +That is O(H log d) per read for every aggregate, with no view-store state in the model +store. Aggregate-over-expression totals index their entries by row tuple for the same subtraction. @@ -206,6 +207,11 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r ### 9. Delivery +The **model store** (the document model and the DBSP graph) and the **view store** (the +cursor, the window and unsaved view definitions) are conceptual boxes, not a network +boundary. Where a piece of state physically lives (a client, the host or a worker) is +an implementation detail. The model store's decisions depend only on the model. + **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared memory. An operator that sits at a wire boundary is just one implementation of an edge, @@ -264,12 +270,12 @@ an **edge adapter**. So every edge has the same semantics: position is a **watermark**: the whole stamp `(host instance, document_generation, publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current - stamp. The client holds one per shape key it reads. The host keeps no watermarks: a - client sends its own with each request. + stamp. The client holds one per shape key it reads. A watermark is view-store state: + the model store keeps none, and a client sends its own with each request. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and - flags, not values, so nothing per client is kept; + flags, not values, so the model store keeps no view-store state; - where the covered range also holds axis changes, relabels or flags, which positions and labels depend on, the correction carries them too, in order, as a merged notice does, so the client re-derives positions incrementally; @@ -292,7 +298,7 @@ relies on, is a correction trigger below, followed from a displayed cell's key b the model. **A request** is a shape key plus a window, given as an absolute cross-product top-left -and dimensions. The window is never state, and a request costs O(visible). +and dimensions. The window is view-store state, and a request costs O(visible). - **Positions** are bigints where they exceed 2⁵³, sent as JSON strings. - **Output** is keyed by tuple identity, with the position attached when sent. - **An item insert, removal or reorder** is sent as an axis change, and each client @@ -326,12 +332,11 @@ notice carries: **A client receives only what it needs.** Filtering is part of the DBSP graph, not a separate mechanism. A client either pulls just the state it needs, or pushes a filter to the host's side of its connection: a filter operator on its shape's query, written in the -query language, through which its notices pass. The filter never names the window, which -stays the client's: the client intersects the moved keys of a notice or correction with -its own window. A pushed filter belongs to the open -connection, like that connection's timeout, and is released with it. On a sequence of -requests the client sends it with each request, so the host keeps no per-client state -beyond an open connection. While a connection's filter is open, it counts as a request +query language, through which its notices pass. Whether the filter also carries the +client's window is an optimisation, since the model store's decisions depend only on the +model. A pushed filter belongs to the open connection, like that connection's timeout, +and is released with it. On a sequence of requests the client sends it with each +request. While a connection's filter is open, it counts as a request naming its shape, so its shape is not evicted. A notice whose key lists exceed a fixed size carries "correct" instead. That is a @@ -346,8 +351,8 @@ keys, the covered range of publications, and axis changes kept in order. Because wire is lossless, outputs keep the circuit's bit identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, -`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client -state, only the shared notice log. `GridExtent` is derived by the client, and in records +`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no +view-store state, only the shared notice log. `GridExtent` is derived by the client, and in records mode the record count is fetched. `Projection` also published the ADR 001 basis a write claims. Whether `project` returns it instead, and what replaces the layout tier, are ADR 008's questions. @@ -393,7 +398,7 @@ scheduling choice, made as an optimisation later. - The store scans and the per-view model clones go. - ADR 008 gets occupancy, totals and widths from one evaluator. - Live sources that report changes plug in. -- The host holds no per-client state. +- The model store holds no view-store state. **Bad:** - Two backends exist until `Eager` goes. From d61f5d74e6e8a7e23c3586d92d7fbe72e61748c0 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:36:04 -0700 Subject: [PATCH 64/67] docs(adr): ADR 009 permits disposable server-side twins of client state at an edge (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e08ba4b7..1ab6966a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -210,7 +210,11 @@ in their last bits, because they become correctly rounded, and a sum of `-0.0` r The **model store** (the document model and the DBSP graph) and the **view store** (the cursor, the window and unsaved view definitions) are conceptual boxes, not a network boundary. Where a piece of state physically lives (a client, the host or a worker) is -an implementation detail. The model store's decisions depend only on the model. +an implementation detail. The model store's decisions depend only on the model. A +server-side twin of a client's state, such as its filter, watermark or window, may be +held at the serving end of an edge where that makes the graph genuinely simpler. It is +then part of the edge, not of the model store, and it is disposable: losing it forces a +correction from the client's watermark, never wrong data. **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared @@ -258,8 +262,8 @@ an **edge adapter**. So every edge has the same semantics: themselves (keys and flags, not values). A request older than the log's start is answered with a correction (below). - **Failure.** Timeouts are local monotonic durations, never compared across machines. - On a long-lived connection, both ends keep alive, and a host-side timeout belongs to - that open connection and is released with it. On a sequence of requests, only the + On a long-lived connection, both ends keep alive, and a host-side timeout is tied to + that open connection. On a sequence of requests, only the client detects failure: a request that fails or times out. In either case the client then asks for a correction from its watermark. @@ -271,7 +275,8 @@ an **edge adapter**. So every edge has the same semantics: publication)` up to which it has applied the stream, so everything at or before it is applied. A watermark from another host instance or generation is behind every current stamp. The client holds one per shape key it reads. A watermark is view-store state: - the model store keeps none, and a client sends its own with each request. + the model store keeps none, so a client sends its own with each request unless a twin + at the serving end holds it. - where the shared log covers the watermark and the covered range holds only moved keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and @@ -334,9 +339,8 @@ separate mechanism. A client either pulls just the state it needs, or pushes a f the host's side of its connection: a filter operator on its shape's query, written in the query language, through which its notices pass. Whether the filter also carries the client's window is an optimisation, since the model store's decisions depend only on the -model. A pushed filter belongs to the open connection, like that connection's timeout, -and is released with it. On a sequence of requests the client sends it with each -request. While a connection's filter is open, it counts as a request +model. A pushed filter is held by the open connection, or by a twin. Otherwise the +client sends it with each request. While a connection's filter is open, it counts as a request naming its shape, so its shape is not evicted. A notice whose key lists exceed a fixed size carries "correct" instead. That is a From a9ee6f00d0fe7fb1506a40b62a4ac0bbe8d83181 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:38:43 -0700 Subject: [PATCH 65/67] docs(adr): ADR 009 states twin-loss recovery and the store as an integrate trace (improvise-felo.4) Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 1ab6966a..36031e02 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -123,7 +123,9 @@ The affected pairs are evaluated in the formula graph's topological order. **An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: 1. A per-record stage retracts and reinserts each touched record. It reads the record's - previous cells as the store minus the delta, so it keeps no copy of the values. + previous cells as the store minus the delta, so it keeps no copy of the values. The + store is the `integrate` operator's trace over the cell delta, read as an input of + the stage, so the stage stays self-contained. 2. A map produces `(context, order key, expr)`. The order key is (context rank, record key in `CellKey` order), which is eager's concatenation order. 3. The group aggregate, with an error channel that reads the first `Err` in order. @@ -213,8 +215,9 @@ boundary. Where a piece of state physically lives (a client, the host or a worke an implementation detail. The model store's decisions depend only on the model. A server-side twin of a client's state, such as its filter, watermark or window, may be held at the serving end of an edge where that makes the graph genuinely simpler. It is -then part of the edge, not of the model store, and it is disposable: losing it forces a -correction from the client's watermark, never wrong data. +then part of the edge, not of the model store, and it is disposable: losing it never +gives wrong data. A client that keeps its own watermark recovers with a correction from +it. A client that relied on the twin for its watermark recovers with a full refetch. **Every edge may cross a network.** An edge, meaning the stream from one operator to the next, may be in-process or remote. No edge assumes a function call or shared From bb578d71e3e50f74c883e9861fce0f14db6e1d12 Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:44:30 -0700 Subject: [PATCH 66/67] docs(adr): ADR 009 values compare by float rules, NaN canonical NaN payloads are not preserved: every NaN is one canonical value in the delta and crosses as ADR 005's `nan` token, so ADR 009 no longer changes ADR 005 clause 11. Zeros follow IEEE: -0.0 equals +0.0, and a zero sum takes the sign IEEE gives the same addends. The round-trip property test asserts a NaN comes back as the canonical NaN. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 29 +++++++++++++---------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index 36031e02..a791116a 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -77,8 +77,11 @@ as `ExactSum` against rational sums, not against `Eager`. - `−(k, v₀) + (k, v₁)` for an overwrite; - `−(k, v₀)` for a removal. -Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite -consolidates to nothing. `CellKey` gains `Ord`, which is its canonical order: the +Values compare by normal float rules, except that every NaN is one canonical value: +`-0.0` equals `+0.0`, and an equal-value overwrite consolidates to nothing. NaN payloads +are not preserved, so nothing can be smuggled through them. Today's record tier compares +bits, so overwriting `0.0` with `-0.0` stops counting as a change. Multisets order values +by `f64::total_cmp` after mapping every NaN to the canonical one and `-0.0` to `+0.0`. `CellKey` gains `Ord`, which is its canonical order: the natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The circuit steps once per model effect. @@ -203,9 +206,10 @@ backends: sums on 20,000 cases; - 568 bytes per group, and about 1.3× the cost of an f64 add per update. -It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change -in their last bits, because they become correctly rounded, and a sum of `-0.0` reads -`+0.0`. +The sign of a zero sum follows IEEE rules for the same addends: it is `-0.0` only if +every addend is `-0.0`, and an exact cancellation gives `+0.0`. It lives in ADR 005's +`numtower` crate. **User-visible change:** displayed sums change in their last bits, +because they become correctly rounded. ### 9. Delivery @@ -229,11 +233,10 @@ an **edge adapter**. So every edge has the same semantics: tuple of ids, its position or an ADR 002 surrogate. - **The wire form is lossless.** Nothing is lost in transport, numeric precision above all: - - a value never crosses as a JSON or JS double. A `numtower` `Num` crosses as its - canonical text (ADR 005), and an inexact finite value keeps its `#i` spelling, which - is already bit-exact, `-0.0` included. A non-finite value crosses as its bits, which - keeps NaN payloads. **That changes accepted ADR 005 clause 11**, whose wire spells - non-finite values as `inf`, `-inf` and `nan`, and it needs approval with this ADR; + - a value never crosses as a JSON or JS double. It crosses in ADR 005's canonical text: + an inexact finite value as `#i` with its shortest round-trip spelling, which keeps + the sign of zero, infinities as `inf` and `-inf`, and every NaN as the one token + `nan`; - positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as strings; - text crosses byte-exact, with no normalisation; @@ -242,8 +245,8 @@ an **edge adapter**. So every edge has the same semantics: The stamp rule detects lost messages, but nothing would detect a lossy encoding. So every element type has a round-trip property test, where encoding then decoding is - the identity for every value, including `-0.0`, NaN payloads, 28-digit decimals and - `i128`-wide fractions. + the identity for every value, including `-0.0`, infinities, 28-digit decimals and + `i128`-wide fractions. A NaN round-trips as the canonical NaN. - **The clock is causal, never a wall clock.** Ordering, staleness and loss detection use only the stamp sequence (below), a logical clock the host advances once per committed command. Each process runs its own synchronous sub-circuit, and across an @@ -355,7 +358,7 @@ moved. It asks for a correction when it sees a publication gap, a new `document_generation` or host instance, its shape in the eviction list, a notice it cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the -wire is lossless, outputs keep the circuit's bit identity end to end. +wire is lossless, outputs keep the circuit's value identity end to end. **What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, `ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no From 40792ee03e2af92f7f7af73238c66adbea982b0d Mon Sep 17 00:00:00 2001 From: "Claude (adr9-steward)" Date: Fri, 9 Oct 2026 22:45:44 -0700 Subject: [PATCH 67/67] docs(adr): ADR 009 requires a zero-sign property test for sums Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/adr/009-incremental-evaluation.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index a791116a..68135713 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -439,6 +439,8 @@ Step 8's required tests: frame stay published; - a dropped connection or a failed request triggers a correction, and a later request with the same session resumes from the watermark. +- a zero sum has the sign IEEE gives the same addends: `-0.0` only if every addend is + `-0.0`, and `+0.0` for an exact cancellation (a property test against IEEE). ## Open questions