diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index e2c239ac..02310a7b 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -1,742 +1,326 @@ # ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait -**Status:** proposed. Only the user accepts an ADR. · +**Status:** proposed. · **Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · -**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed -models) · -**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") · -**Governed by:** `context/design-principles.md` §3, §5 and §7 +**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008 -**Summary.** Formulas are evaluated incrementally by a small in-house implementation of -the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A -cell write becomes a delta. Aggregates are maintained rather than re-folded, and only -the formula cells whose inputs moved are re-evaluated, by the one existing evaluator. -Operator state is keyed by the **shape key** of a projection request, never by a client. -The same circuit serves the in-process TUI and remote clients. A remote client learns of -changes from one unaddressed notice per accepted batch, and it refetches what it shows. -Sums are exact, through an exact accumulator. +## Problem -File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive -them before relying on one. - -## Problem and context - -**Data that streams.** A live-backed model (ADR 003) is a source whose rows change -underneath the document. Today every change takes the path an edit takes, which -recomputes every formula cell. That is tolerable for one keystroke at a time and not for -a stream of changes. A change should cost in proportion to what it touches. - -**Today's evaluation is a from-scratch fold after every batch.** -- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls - `Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates - every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16` - (`types.rs:710`). -- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`, - `types.rs:848`, through `DataStore::matching_values`). -- `formula_cache` holds one aggregation shape at a time. A session on a view with another - None axis gets a **cloned model** (`rederived_for`, `project.rs:205`). -- A formula total that holds an aggregate walks a cross product - (`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`). -- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing - batch, and `document_content` (`app.rs:168`) after each `CellData` effect. -- Every mutating frame from a client clones the document for rollback - (`Host::mutating_frame`, `host.rs:224-225`). - -**What an incremental evaluator must not change.** Each of these is a place where it -could quietly disagree with today's code: -- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one - distinct value is that value, and anything else is `Error("mixed")`. -- **Circularity (gotcha 9).** A cell still moving after 16 passes is - `Error("circular")`, and so is an acyclic chain that needs all sixteen. -- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0 - guard. -- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration, - because f64 addition is not associative. -- **The tiers are observed (gotcha 40).** A write of the value already there bumps - nothing, and a rewritten record counts once. - -**The view/host split (ADR 008).** A client reads through `project(request)`, where a -request names axes, an absolute top-left and dimensions. It writes only through -`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys. -Aggregation is always done on the model side. A view's hiding, collapsing, page -selection, axis assignment and pruning are one client's ephemeral state until an -explicit save-view command. Only the number format is document-wide. +Every change recomputes every formula cell (`Model::recompute_formulas`, called by +`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That +does not scale to a live source (ADR 003) streaming changes into a large table, nor to +ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without +building the grid. A change should cost in proportion to what it touches. ## Decision -### 1. The crate: `zset` +Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A +cell write becomes a delta, aggregates are maintained, and only the formula cells whose +inputs moved are re-evaluated by the existing evaluator. The incremental results must +equal today's eager results exactly. -A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on -`std`, so it is wasm32-clean by construction. It contains: -- **`ZSet`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an - abelian group. Weights use checked arithmetic, and a zero weight is removed. The group - laws are property-tested. -- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per - source and returns one per sink. There is one clock, one thread and no scheduler. -- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id - property-tested); the linear `map`, `filter` and `flat_map`; `index_by`; - `join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear - aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a - multiset per group. -- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a - resumable `step(deadline)`, so a rebuild can run as a sliced job. +### 1. The `zset` crate -Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one -clock (improvise applies batches in one total order); threads within one circuit (a host -with threads may run independent circuits on a pool); a query language. Spilling to disk -is not in the first version, but operator state sits behind one interface (a trace: look -up, read a range, apply a delta), so the store-memory work can page or compress it later. +A problem-agnostic workspace crate with no dependencies beyond `std` (so it is +wasm32-clean and serde-free). It provides: +- `ZSet`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group; +- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`, + `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`, + `distinct` and `aggregate`; +- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it + can run as a sliced job; +- operator state behind one trace interface (look up, read a range, apply a delta), so + later store-memory work can page or compress it. -### 2. The plug: a trait, and a seam crate +There are no nested streams, no recursion operator, no timestamps beyond one clock, and +no query language. -improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the -formula result at a key, and the raw fold of the cells that match a partial key. It -takes the cell delta and the formula delta after each model effect. +### 2. The plug -**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and -read through the trait. The circuit decides *which* formula cells to re-evaluate and -supplies their inputs. Formulas are not compiled into operators, because a compiled -`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the -defect. +improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the +formula result at a key, and the raw fold of the cells that match a partial key. The +incremental backend implements it in a seam crate, `improvise-incremental`, which +depends on `zset` and improvise-core. improvise-core never depends on `zset`. -**The incremental backend lives in a seam crate, `improvise-incremental`,** which -depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same -shape as `gp-tree` and `improvise-gp`. +The circuit decides *which* formula cells to re-evaluate. The one existing evaluator +computes each value, so formulas are never compiled into operators. -**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change, -as `Eager`. It stays as the oracle for the differential test while the circuit matures, -and for the two cases that still need it (clauses 5 and 10). The differential test is a -property test in the seam crate that applies one random edit sequence to both -backends and compares every drawn cell, total and occupancy count. The goal is that every -formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is -open question 15. +**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged. +It serves as the oracle of a differential property test, which applies one random edit +sequence to both backends and compares every drawn cell, total and occupancy count. It +is removed once its remaining uses are replaced (open question 3). -### 3. The input: the cell delta +### 3. The cell delta -`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a -`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number -assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an -overwrite: -- a new cell is `+(k, v, s)`; -- an overwrite is `−(k, v₀, s) + (k, v₁, s)`; -- a removal is `−(k, v₀, s)`; -- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`. +`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`: +- `+(k, v, s)` for a new cell; +- `−(k, v₀, s) + (k, v₁, s)` for an overwrite; +- `−(k, v₀, s)` for a removal; +- a remove-then-set gets a fresh `s′`. -**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their -strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an -equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a -change, and a NaN rewritten with the same bits is not. Multisets order values by -`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted -`Vec`, so deriving it is sound. +`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an +overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite +consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect. -**What the delta replaces.** The record tier reads the records named by the -consolidated `(key, value)` projection of the delta, instead of diffing two store scans. -The dirty flag's content half becomes "a non-empty projected delta, or a moved -formula-set fingerprint". The fingerprint is needed because formula edits move no cell, -and the worker persists on this flag. +The record tier and the dirty flag read this delta instead of scanning the store. The +dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the +formula set moves, because formula edits move no cell. -**The circuit steps once per model effect.** That keeps occupancy current inside the -effect loop. What a client counts is a separate publication number (clause 12). +Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The +formula-set fingerprint must use the id form too, or a rename reads as a formula change. -**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`, -the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a -rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set -fingerprint, taken today over `(raw, target, target_category)`, must then be taken over -the id form, or a rename reads as a formula change. +### 4. Aggregates -### 4. Aggregates: linear where they can be +A group is the cell key minus the None categories. One aggregation circuit serves every +shape key with the same None list, which removes the per-session model clone. -**The group** is the cell key minus the None categories, which is the partial key -`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the -same None list, which removes `rederived_for`'s model clone. - -| answer | state per group | cost per delta element | +| aggregate | state per group | per delta element | |---|---|---| -| `Count` | a count of values and a count of numbers | O(1) | -| `Sum` | an exact accumulator (clause 8) | O(1) | -| `Avg` | `Sum` / `Count` | O(1) | -| `Min`, `Max` | a multiset of the numbers | O(log d) | -| unanimity and `mixed` | a multiset of all values | O(log d) | +| `Count` | count of values, count of numbers | O(1) | +| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) | +| `Min`, `Max`, text unanimity | a multiset of values | O(log d) | -Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules -for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential -test generates signed zeros and non-finite values to pin this. Only measures whose -aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups, -they hold at most one entry per stored cell. +A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and +semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the +ordering category's order. How that order is keyed is for those tickets. -**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures, -improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an -extreme, keyed by the ordering category's order instead of by value. A semi-additive -measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the -order is keyed is open question 2. +### 5. Formulas -### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it +**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None +categories". A formula counts at a stem where its top-level `WHERE` matches or raw data +exists at the target. -**Stems** are `distinct` over the cell delta, mapped to "key minus the target category -and the None categories". A formula counts at a stem only where its top-level `WHERE` -(if any) matches the stem's coordinate, or where raw data exists at stem + target. That -is exactly where `recompute_formulas` caches a value today, so pruning keeps its current -behaviour. +**Inputs.** A formula cell is re-evaluated when one of its inputs moved: +- a `Ref(R)`: the group at the stem with `R`'s category set to `R`; +- an `Agg`: the group at the stem; +- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set + to `Item`. This is the one join: stems indexed by "stem minus `Cat`". -**A formula cell is re-evaluated when one of its inputs moved:** -- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`; -- for an `Agg` with no filter, the group at the stem itself; -- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat` - set to `Item`, which is another stem. This is the one join: stems indexed by "stem - minus `Cat`", joined with the aggregate's output delta. +The affected pairs are evaluated in the formula graph's topological order. -The affected `(stem, formula)` pairs are ordered topologically over the formula graph and -handed to the existing evaluator. +**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline: +1. A per-record stage retracts and reinserts each touched record. It reads the record's + previous cells as the store minus the delta, so it keeps no copy of the values. +2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum + `Seq` of the record's live cells), which is eager's concatenation order. +3. The group aggregate, with an error channel that reads the first `Err` in order. -**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline: -1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it - retracts the record's previous snapshot and inserts the new one. The previous snapshot - is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. - So the stage reads the store and keeps no second copy of the values. -2. **A lifted map** to `(context key, order key, expr at record)`. The context key - removes the target and None categories, and at a fold or total the spanned categories, - mapping a collapsed member to its group. The order key is (context rank, the minimum - store sequence over the record's live cells), which is exactly the order eager - concatenates in. -3. **An inner `WHERE`** is a filter plus the join above. -4. **The group aggregate**, with an error channel kept beside it that reads the first - `Err` in order-key order, as eager does. - -No cross product is ever built. The map is sound only when `expr` reads the record's own -cells. Other formulas take a fallback that evaluates over the **occupied contexts** only, -the contexts some record projects to. Because `agg_values` evaluates only at each -context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one -edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over -zero occupied contexts it yields the empty value instead. Formulas targeting a category -other than `_Measure` stay on the eager path until their order key is defined. Open -question 8 covers both. +No cross product is built. A formula whose expression reads other records is evaluated +over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is +unchanged, because a context with no records contributes nothing. One exception: an +inner bare ref that names no item errors at every context, so for it the ref is checked +before enumerating. This replaces `Expansion::members`. ### 6. Circular references -DBSP's recursion computes a least fixed point, and improvise's circularity is not one: -`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs -forever. The required answer is `Error("circular")`. The first version therefore decides -circularity outside the crate: -- the formula graph is rebuilt when the formula set changes, and its strongly connected - components are found; -- formulas outside every cycle are evaluated once per step, in topological order; -- a formula in a cycle has its affected cells evaluated by today's bounded fixed point, - restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still - convergent where `x` is false; -- the sixteen-pass bound is reproduced, so the two backends agree in the differential - test. +DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the +answer must be `Error("circular")`. The first version finds the formula graph's strongly +connected components. Acyclic formulas are evaluated once each, in topological order. A +component is evaluated with today's bounded fixed point, restricted to that component. +That reproduces the sixteen-pass bound for the differential test. It is to be replaced +by a DBSP-native treatment (open question 2). -This serves testing and validation. It is to be replaced by a DBSP-native treatment of -cycles, whose shape is open question 14. +### 7. Operator state follows the request's shape key -### 7. The circuit follows the document per model effect - -**Operator state is selected by a shape key.** The key is exactly what changes which -cells feed which operator: -- the categories of each axis, by level; +A projection request's **shape key** selects operator state, shared by every request +with the same key, from any client. The key is: +- the categories on each axis, by level; - the None categories; -- the filter keys: each filter category with its selected item, and the hidden items of - every category on None, Page or Filter. Those items drop cells from the fold; -- the page **categories**, which key a slice per page (open question 1); -- `measure_agg`, which is model state, never taken from a request. +- the filter keys: each filter category's item, and the hidden items of categories on + None, Page or Filter; +- the page categories, so each page is a maintained slice and paging rebuilds nothing; +- `measure_agg`. -**Everything else is a read-time parameter**, applied when output is read: the hidden -items of an axis category, collapsed groups and group membership, the drawn order of -items, the page selection, and prune. A request carries its own key and parameters from -its client's UI state, and a saved view only supplies defaults. +Everything else in a request is applied when output is read: hidden and collapsed items +on axis categories, item order, the page selection and prune. Occupancy and column +aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item +changes no operator. The visible axis is a remap. -**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn -items, the collapsed groups, the page selection and `prune_empty`, all of which are -read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit -neither reads nor depends on that tier, which is a view fact. Only `document_generation` -and the record tier matter to this ADR. +- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas + arriving during the build are buffered and added at install. +- **A formula edit** is a formula delta fed to the circuit. +- **A document swap** rebuilds everything. +- **Saving a view definition** touches no operator. +- **Live keys are a cache.** A key is evicted after a period in which no request names + it (the period is set by measurement), and every eviction is announced (clause 9). -**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates -are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing, -reordering and adding an axis item change no operator: the visible axis is a remap. -ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a -rebuild. A collapsed group's value is combined from its members when read. That is exact -for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it -costs O(members) per read (not measured). The exceptions are column totals (below). - -**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job -runner (8 ms per normal operation). Deltas that arrive during the build are buffered and -added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is -O(delta) and is never sliced. Each step's occupancy output carries the count of tuples -whose count crossed zero, so ADR 008's ordered index can choose between inserting and -rebuilding. Measurements against the budget are stated as ratios against an unrelated -operation on the same run, not as bare numbers. - -**Formula edits are an input.** After each model effect the host compares a fingerprint -of the formula set. If it moved, the formula delta is fed to the circuit, and the formula -graph is re-derived. A document swap (`install_workbook`, which bumps -`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and -no formula, so it touches no operator. - -**The shape cache.** Live shape keys are a cache. A key enters when a request names it, -and it is evicted when no recent request names it (open question 4). Every eviction is -announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm -by every client reading it. Idle readers make that likely, since a shape that clients -only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job -queue. - -**Column totals subtract hidden rows at read time.** Today a column total pools only the -drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`), -pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a -per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept -over all rows, and a read subtracts the request's H hidden cells in that column as a -temporary overlay: +**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row +leaves it. The shape key is shared and a hide is one client's, so totals are kept over +all rows. A read subtracts the request's H hidden cells as a temporary overlay: - `Count` subtracts; -- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden; -- the text rule counts the distinct keys left; -- `Sum` subtracts through the exact accumulator. +- `Min` and `Max` skip keys whose whole count is hidden; +- text counts the distinct keys left; +- `Sum` subtracts exactly through the accumulator. -That is O(H log d) per read for every aggregate, with no per-client state. The shape key -is shared across clients and a hide is one client's, so a total cannot be rebuilt per -hide. A hidden column item under a collapsed column group changes that group column's -cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression -totals need their entries indexed by row tuple as well, so a read costs O(hidden -records). The alternative is open question 12. +That is O(H log d) per read for every aggregate, with no per-client state. +Aggregate-over-expression totals index their entries by row tuple for the same +subtraction. -### 8. Floating point: exactness comes from the representation +### 8. Exact sums -A sum maintained by `+v` and `−v` in f64 drifts from the eager fold. -`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max, -with values of up to five decimal places: +An f64 sum maintained by `+v` and `−v` drifts. Measured by +`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million +cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer +differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both +backends: +- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which + makes it an abelian group that rounds once, on read; +- measured bit-equal to a fresh sum after a million cycles, and exact against rational + sums on 20,000 cases; +- 568 bytes per group, and about 1.3× the cost of an f64 add per update. -| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact | -|---|---|---|---| -| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 | -| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 | -| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 | +It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change +in their last bits, because they become correctly rounded, and a sum of `-0.0` reads +`+0.0`. -Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively, -against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager. -At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer -differently on the two paths. Eager itself is already more than 1e-10 from the exact -decimal sum. +### 9. Delivery -**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer -wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so -that those can be retracted. Adding and retracting are integer additions, so the -accumulator is a true abelian group, and it rounds once, on read. A weight is split into -base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example: -- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles; -- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again - with weights up to 2⁴⁰; -- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to - 1.1 µs per read; -- it is 568 bytes per group, against 8 for an f64. +The same circuit serves the in-process TUI and remote clients. Its internal streams never +cross a wire, because aggregation stays on the host. -`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0, -where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they -become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and -the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum` -0.1.6 rounds about a quarter of single negative values wrongly and cannot retract -non-finite values. +**A request** is a shape key plus a window, given as an absolute cross-product top-left +and dimensions. The window is never state, and a request costs O(visible). +- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings. +- **Output** is keyed by tuple identity, with the position attached when sent. +- **An item insert, removal or reorder** is sent as an axis change, and each client + re-derives its positions. -**Lossy operations happen at the edge.** Sums stay in the accumulator. Division, -non-integer `POW` and display rounding happen only where a value leaves the exact domain. -An exact value becomes an f64 through its decimal string, which is correctly rounded. +**Pruning support.** Pruning is window-relative: a row or column empty within the +displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model +answers one restricted successor query: "the next k rows after position p with a value +in a column of S", and the same for columns. A per-column sorted index of occupied rows +answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its +window, fills forward with that query, and finds a new top-left when scrolling up or left +by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether +pruning runs on the client and which parameters travel are ADR 008's questions. This +design supports either answer. -**The accumulator does not make `=` against a typed decimal exact.** It sums the stored, -already-rounded f64s, so it can miss the typed total by up to about half an ulp of each -term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example: -`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's -exact decimals fix that, and they also cover per-cell arithmetic and typed values such as -money. +**The change notice.** After each accepted batch the host emits one notice to every +client, addressed to none. It carries: +- the stamp `(host instance, document_generation, publication)`. The publication number + increments once per published batch, and the host instance changes on restart; +- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved; +- the shape keys evicted since the last notice; +- for each live shape key, the moved output keys (cells, formula cells and totals), as + the circuit computes them. Moved records alone would miss formulas with cross-stem + inputs. -### 9. What ADR 008 gets +A notice whose key lists exceed a fixed size carries "refetch" instead. -- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple - (kept as a pre-filter and as the unpruned extent; see open question 11). - Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero. - A fold row with a formula column but no stored cell under it, and the `Exact` case with - nothing on None, are **not** occupied, on purpose. -- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit - consumes it and never keeps a second order. -- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item - grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because - the host cannot compute a client's visible axis. -- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the - formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows). -- **Widths for text columns** (clause 13). +**What a client does.** It intersects the moved keys with its window. A fold row has no +maintained key, so it maps member keys through its own remap first. It then refetches +what moved. It refetches everything when it sees a publication gap, a new +`document_generation` or host instance, its shape in the eviction list, or a notice it +cannot read. A slow client may merge pending notices: the union of keys, the latest +stamp, and axis changes kept in order. Outputs are consolidated under the wire's value +identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`. -### 10. Live sources +**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`, +`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client +state. `GridExtent` is derived by the client, and in records mode the record count is +fetched. -A live source that can report its changes produces a cell delta, and the circuit cannot -tell it from `DataStore`'s. A source that can only answer aggregate queries stays on -`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated -state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so -the backend is chosen per document. +**A connection operator** at the wire boundary adapts the circuit to a connection. It is +a sink for results and notices and a source of requests toward the circuit, and the +mirror image toward the client. It owns serialization, the stamp rules, notice +coalescing, keep-alive in both directions, and session resumption. In process it is +function calls. Its shape is open question 4. -### 11. Constraints +### 10. Text column widths -- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then - `zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm` - CI job, each checked with `cargo tree --edges normal`. -- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over - small inputs, and the differential test lives in the seam crate's own binary. -- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and - sorted, and nothing in the circuit decides a display order. The determinism test builds - **two** circuits independently from one history. -- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that - `Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the - integrated values but not the history `delay` and `differentiate` hold. The cost is a - second O(state) clone per mutating frame, and peak memory to match. The keyboard path - takes no snapshot. -- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence. +The host maintains each text column's maximum display width as a histogram from width to +count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows +come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule. -### 12. Delivery: one circuit, in process or across a wire +### 11. Rollback and memory -The DBSP machinery is **network-transparent**: the same operators and streams whether a -consumer is in process or across a wire. +- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An + inverse delta is not a rewind, because `delay` and `differentiate` keep history. +- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets, + `ExactSum` per summed group, occupancy per live shape, the successor-query index and + the width histograms. The store itself is the integrated base and is never copied. It + is measured at ADR 008's fixture before the store-memory work begins. -**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their -change notices, plus the input deltas of a remote live source. The write side carries -model commands with their basis, never deltas. The circuit's internal streams stay on -the host, because aggregation is model-side. No UI state crosses. +## Alternatives rejected -**A request has a shape key and a window.** The key selects operator state, shared by -every request from any client. The window (absolute top-left and dimensions) selects -part of the output and is never state. So a request costs O(visible). +- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it + fails to build for wasm32. +- **Compiling formulas into operators:** a second evaluator. +- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a + second description of evaluation. +- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic + only by holding per-client state on the host (open question 1). -**Positions are absolute cross-product positions**, in each category's own item order, -and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when -occupancy changes. They do move when an item is inserted, removed or reordered, so the -recommended design (open question 10) is: -- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute - position attached when sent; -- an item edit is sent to every client as an **axis change** ("inserted at index j of - level L"). A client re-derives its positions at O(levels) each and re-anchors its - top-left to the same tuple; -- resolving positions needs rank and select over each category's items, which is ADR - 008's order-statistic structure. +## Consequences -**Pruning's model-side support.** Pruning hides a row or column that is empty within the -displayed values, so a value outside the window does not count. That differs from -today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed -window, one pass that drops the empty rows and columns is already the fixpoint. A fill -then adds rows and columns forward from the absolute top-left, until the requested -dimensions are met or the axes end. Growth is only rightward and downward. For the fill, -the model answers one **restricted successor query**: "the next k rows after absolute -position p with a value in at least one column of S", with S in the call, and the same -for columns. A per-column index of occupied rows answers it: a sorted list per column, -seeked to p and merged forward with a linear minimum over the |S| cursors, at -O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the -cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a -B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's -whole state is its request and its last absolute position, so a resume is a seek. -**Scrolling up or left is a new request with a new top-left.** How the client finds that -top-left when rows are pruned is the recommendation of open question 9: it gallops -backwards in doubling jumps until a probe hits, then narrows until one new row is gained. -Each probe is a restricted successor query bounded by the old top-left. That is about 14 -to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an -empty row. The alternatives there are a model-side predecessor query, and failing the -fetch and retrying. Which side of the wire runs the prune and the fill is -open question 9. +**Good:** +- A write's evaluation costs O(delta) plus the formula cells it reaches. +- The store scans and the per-view model clones go. +- ADR 008 gets occupancy, totals and widths from one evaluator. +- Live sources that report changes plug in. +- The host holds no per-client state. -**The wire properties**, which hold on each hop: -- **T1. Plain data.** Output elements are serializable values that name cells by tuples, - positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`. -- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`), - so a value change that only a NaN payload distinguishes cannot be told apart remotely. - Which identity outputs use is open question 5. -- **T3. A publication stamp.** Each accepted batch is published once, stamped - `(host instance, document_generation, publication)`. The publication number increments - once per published batch, so a multi-effect batch is one header, and a refused frame - consumes none. The host instance changes on restart. A client drops anything at or - below the number it has applied. **The refetch rule:** a client that sees a gap, a new - `document_generation` or host instance, its shape in an eviction list, or a notice it - cannot interpret, refetches. -- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative - to its stamp. -- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and - flags, under the latest stamp. Axis changes are positional, so they stay an ordered - list. A `document_generation` change inside the run voids the merge. A basis is never - sent ahead of the cells it describes (gotcha 43). -- **T6. One API.** The TUI calls `project` and receives the same types as a remote - client. A test runs one edit sequence through an in-process consumer and through a - serialize-and-deserialize round trip, and requires equal results. +**Bad:** +- Two backends exist until `Eager` goes. +- Integrated state is O(cells) beside the store. +- `ExactSum` costs 568 bytes per summed group. +- Every client receives every notice. +- A mutating frame still clones the state for rollback. -**The change notice.** After each accepted batch the host emits one notice, addressed to -no client. It carries: -- the stamp; -- the axis changes; -- display relabels (`(id, new display)` under ADR 007); -- a flag when the formula set or `measure_agg` moved; -- the eviction list; -- for each live shape key, the **moved output keys**: cells, formula cells and - column-tuple totals, as the circuit computes them. +**Order** (each step is a ticket in the implementation epic, behind the differential +test): +1. `zset`; +2. the `Derived` trait with `Eager`; +3. the cell delta, the record tier and the dirty flag; +4. order-free aggregates and occupancy, measured at ADR 008's fixture; +5. formulas and the aggregate-over-expression pipeline; +6. `Sum`, `Avg` and totals through `ExactSum`; +7. live sources; +8. delivery. -Moved records alone would not do. From a moved record, a client cannot tell whether a -formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue -WHERE Region=East)` on page `Region=West` moves when an East record is written. A client -intersects the moved keys with its window by identity. A collapsed fold row has no -maintained key, so a moved member key goes through the client's own member-to-group -remap first. The client then refetches what moved, by `project` or by the successor -query. - -**Every client receives every notice**, and the host holds no per-client state for -delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists -exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a -refetch by every client in a busy batch. Today's per-mirror push (`Sessions`, -`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and -`GridExtent` is derived by the client from category lists. In records mode the row -extent is the admitted record count, which the client fetches. Whether notices should be -filtered per client is open question 3. - -**A connection operator, at the wire boundary.** A connection operator sits at the wire -boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for -results and notices and a source of requests and model commands. Toward the client it is -the mirror image. It owns: -- serialization (T1, T2); -- stamps and the refetch rule (T3); -- coalescing (T5); -- keep-alive in both directions. A client that hears neither a header nor a ping within - the timeout treats the connection as dropped, so silence is never read as "no - batches"; -- session resumption. - -In process it is function calls, as in the TUI and in the browser's `Worker`, which -already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in -`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one -component per connection, or a host half shared per host process with a client half in -every front end), and who owns resumption state, are open question 17. - -### 13. Column widths for text - -The host maintains each text column's maximum display width incrementally. A maximum is -not invertible, so it is an integrated aggregate. Display widths are small integers, so -the state is a histogram from width to count, per (shape key, column): O(distinct widths), -at O(log d) per update. Its input is the delta of the column's text cells over every row, -which the circuit already produces for unfolded cells. Fold rows need the fold maintained -at the visible grain. A text width does not depend on the number format, so there is no -rebuild when the format changes. A client's hidden rows come out through clause 7's overlay. -Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized, -held by the view layer. A number's width would need formatting on the write path, and a -rebuild per format change. Which columns count as text is open question 16. - -### 14. Resident state - -Beside the store, the circuit keeps: -- each cell's store sequence, for clause 5's per-record stage (one integer per cell, - kept in `DataStore`); -- the multisets for `Min`, `Max` and text, O(cells) of those measures; -- `ExactSum` at 568 bytes per summed group; -- occupancy per live shape, O(occupied tuples); -- the restricted-query index, O(occupied cells); -- the width histograms; -- the rollback snapshot per mutating frame. - -The store itself is the integrated base that a rebuild reads, so it is never copied. -State is evicted with its shape, and it sits behind one trace interface, so the -store-memory work (offload to disk, compression) can page or compress it later. None of -these sizes is measured at ADR 008's fixture. Migration step 4 measures them. - -### Relations to other ADRs - -- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its - `document_generation` and record tiers, not on `layout_version`. -- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace; - epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values. - Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a - 256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators. -- **ADR 006:** replay reproduces a circuit's state. -- **ADR 007:** keys become ids, so a rename is a relabel (clause 3). -- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the - occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire - prunes, and which parameters a request carries, are shared with questions 9 and 13 - below. - -## Alternatives considered - -- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio, - mimalloc, zstd and others), about 421 crates for wasm32, and fails - `cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded, - storage-backed engine where improvise needs a single-threaded operator library. -- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only - the cost of not changing: every write still recomputes every formula. -- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule, - which is a second description of evaluation. Z-sets give one rule, linearity, whose laws - are tested once. -- **Compile formulas into operators.** A second evaluator (clause 2). - -## Consequences and migration - -**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches. -The store scans behind the record tier and the dirty flag go, and so do the per-view model -clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources -that can report changes plug in unchanged. The host keeps no per-client state, and one -delivery path serves every front end. - -**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that -must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs -568 bytes per group, and displayed sums change in their last bits. Circularity is decided -in two layers. The network path still clones per mutating frame. Every client receives -every notice. Every column-total read pays O(H log d) for the request's hidden rows. - -**Order.** Each step is a ticket and a PR in the implementation epic, and none switches -the displayed path until the differential test holds: -1. the `zset` crate, with its laws, operator properties and determinism test; -2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`; -3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a - signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty; -4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and - occupancy) under the differential test, with step and rebuild costs measured at ADR - 008's fixture. ADR 008's pruning depends on this step; -5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's - aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price` - and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members` - depends on this step; -6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row - overlay; -7. live sources as delta streams, once ADR 003's source trait exists; -8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one - header and no client refetches; a dropped header is detected; a reconnect after a host - restart refetches and drops nothing as a duplicate; a refused frame consumes no - publication number; and a dropped connection forces a refetch. +Step 8's required tests: +- `[AddItem, SetCell]` publishes one notice, and no client refetches; +- a dropped notice is detected; +- a reconnect after a host restart refetches; +- a refused frame consumes no publication number. ## Open questions -1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page - categories, every page's state is maintained and `[`/`]` only reads another slice. The - held key stays O(visible), and the state stays bounded by the stored cells. If it does - not, a page change builds operators like any other new key. *Recommendation:* the - slice, which clause 7 assumes. +1. **Should notices be filtered per client?** Clause 9 sends every client every live + shape's moved keys. That keeps the host free of per-client state, but traffic is + O(clients × all moved keys) per batch. Filtering by topic would send each connection + only its own shape's key lists. It needs one subscription per connection, which pins + its shape in the cache. Is such a subscription "model-relevant information" the + backend may hold? It could also live in the connection operator (question 4). + *Recommendation:* broadcast with the size cap. Add topics only if measured traffic + with many clients on different shapes justifies them. -2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder, - since a reorder rebuilds nothing. By item identity with the order looked up at read - time, which makes reads O(d). Or by an order label: a key that compares in the current - order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder - re-keys every entry stored under the moved items. *Recommendation:* the order label, - with only the drawn members ordered. Not measured. +2. **How should cycles be handled natively?** Clause 6's component fixed point is + interim. The options: + - (a) a recursion operator bounded at sixteen iterations, writing `circular` where it + has not converged, which keeps today's answers; + - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles + such as `A = IF(x, B, 1)`; + - (c) a monotone reformulation, where one exists. -3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed - stream. The rejected alternative is to send no notice at all and let each client - refetch on its own schedule: it keeps the host silent, but another client's change - stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state, - but every client receives every live shape's moved keys. The options: - - **topics:** a client receives only its own shape key's key lists. Headers still go to - everyone, so the refetch rule keeps working. That costs one subscription per - connection, registered atomically with the fetch that names it, and pinning its - shape; - - **window push:** per-client subscriptions holding the window, sending only the - window's deltas. That costs the most host state. + *Recommendation:* (a). - Is a per-connection subscription "model-relevant information" the backend may hold, - or UI state it must not? It need not live in the host at all if the connection - operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt - topics only if measured traffic, with many clients on different shapes, justifies - them. +3. **What replaces `Eager`?** It still serves as the differential oracle and as the + backend for live sources that can only answer aggregate queries. *Recommendation:* a + small from-scratch reference implementation, kept in the test crate, as the oracle, + and a pushdown backend behind the same trait for query-only sources. -4. **How long does a shape key live?** A key is evicted once no recent request names it. - Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades - memory against refetch storms. If the connection operator re-requests the shapes its - clients read, that is a keep-alive by another name: one host `project` per shape per - eviction period. *Recommendation:* evict after a period with no request, set by - measurement, with hysteresis for shapes with many readers. +4. **What shape is the connection operator?** Either one component per connection, or + one per host process: a host half shared by all its connections, with a client half + in every front end. The second lets the host send one notice per operator rather than + per client, but holds per-connection state in the shared operator. Who owns + resumption state, and is it backend state? *Recommendation:* one per host process, + with resumption state in the host half keyed by a client-held token, treated as + delivery state rather than UI state. In the TUI and the browser worker, with one + client each, it is a function call. -5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs - with different payloads differ, but the wire spells every NaN `"nan"`. A remote client - would therefore see such a change as nothing. *Recommendation:* outputs consolidate - under the wire's identity (every NaN one value), and the record tier keeps bit - identity. - -6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and - column aggregates by items, reports zero crossings per step, and slices rebuilds. That - makes collapse, hide, reorder and add item free, at the cost of recombining a - collapsed group's members on read. *Recommendation:* adopt all three, as written. - -7. **Resident state.** Should the first version keep no second copy of the store (the - store is the integral), put operator state behind one trace interface, and measure - state at ADR 008's fixture before the store-memory work begins? *Recommendation:* - yes. - -8. **Formulas the per-record map cannot express.** They take the occupied-contexts - fallback (clause 5). - - (a) Should `Eager` use the same enumeration, so the two backends agree? That changes - eager's answer only at the bare-ref edge, which could also be kept by checking the - ref once before enumerating. - - (b) Should formulas targeting a category other than `_Measure` take the same - fallback, so that nothing still needs `Expansion::members`? - - *Recommendation:* yes to both. - -9. **Which side of the wire prunes and fills?** On the client, the model offers only - windows and the restricted successor query, and the fill is the client's, with no host - state. On the model, prune is a request parameter, and the model keeps a pruned axis, - with rank and select for "row i of n", and runs the fill as a sliced job resumable - from (request, fill front). At a density of 10⁻⁴, a client without the successor query - would ship mostly empty windows. A second part: when scrolling up or left, how does the - client find a new top-left when rows are pruned? The options are galloping backwards - with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a - model-side predecessor query (one round trip, no new state, but a second query - direction), or failing the fetch and retrying with the nearest occupied row named. ADR - 008 asks the same question. *Recommendation:* the client prunes and fills, and it - gallops. - -10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas - by position instead goes stale on every item insert. Identity costs rank and select - per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as - clause 12 assumes. - -11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers - the restricted successor query. Per-axis counts can only skip rows that are empty - everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned - extent. *Recommendation:* keep them in that role. Not measured. - -12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at - O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape - key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per - distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at - read time. Memoise per hidden set only if a measured hidden set is too large for the - frame budget. - -13. **Which read-time parameters travel in a request?** The page selection travels, as a - third axis. Hidden items and collapsed groups change model-side totals and folds, so - without them a client could not get those from the model. Alternatively, a collapse - could be expressed as an axis at the group's level and a hide as a filter exclusion, - but both change the shape key, which costs a build per state. The drawn order, the - number format and prune change no aggregate. *Recommendation:* hidden items and - collapsed groups travel; the rest stay with the client. - -14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and - validation. The options: - - (a) a recursion operator bounded at sixteen iterations, which writes `circular` - where it has not converged and keeps today's answers; - - (b) marking every cell in a graph cycle `circular`, which changes guarded cycles - such as `A = IF(x, B, 1)`; - - (c) a monotone reformulation, where one exists. - - *Recommendation:* (a) first. - -15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle, - query-only live sources, and the formulas of question 8 that stay eager. - *Recommendation:* a small from-scratch reference implementation, kept in the test - crate, as the oracle; a pushdown backend behind the same trait for query-only - sources; and the fallback of question 8 as a circuit stage. - -16. **Which columns get a host-maintained width?** Columns whose cells are text, with - numbers keeping the sticky width with hysteresis? And what of a column that holds - both? *Recommendation:* text cells get the maintained width, and the view layer takes - the larger of that and its own numeric width. - -17. **What exactly is the connection operator?** (clause 12) - - (a) Do its "both ends" mean both halves in one component, or one component on each - side of the wire? - - (b) Is it a new crate or process, or a role for existing code (the browser - `Worker`, the socket server's `Registry` and transport, in-process code in the TUI)? - - (c) Who owns keep-alive and resumption state, and is that backend state? - - (d) Should one operator serve many connections and subscribe to the host on their - behalf, so that the host sends O(operators) per batch, not O(clients)? That saving - needs one operator per host process serving many clients, so per-connection state - then lives inside a shared operator. - - *Recommendation:* (a) one operator per host process: a host half shared by its - connections, and a client half in every front end. (b) a role of the existing - transport crates, not a new process. (c) the host half holds resumption state, keyed - by a token the client holds, as delivery state, not UI state. (d) yes, for the socket - server. The TUI and the browser worker each serve one client, so there the operator is - a function call. +5. **Which columns get a host-maintained width?** Columns whose cells are text? What + about a column holding both text and numbers? *Recommendation:* text cells get the + maintained width, and the view layer takes the larger of that and its own numeric + width.