docs(adr): ADR 009 states only what the decision changes (improvise-felo.4)
Following the user's instruction, restated conventions and current behaviour are removed except where the decision conflicts with them, and the open questions are cut to the five design choices still open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (adr9-steward)
committed by
Edward Langley
Edward Langley
parent
e4f915423c
commit
e7d9ad0d2a
@@ -1,742 +1,326 @@
|
||||
# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait
|
||||
|
||||
**Status:** proposed. Only the user accepts an ADR. ·
|
||||
**Status:** proposed. ·
|
||||
**Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own
|
||||
epic) ·
|
||||
**Builds on:** ADR 001 (basis tiers), ADR 002 (naming a cell), ADR 003 (live-backed
|
||||
models) ·
|
||||
**Relates to:** ADR 005, ADR 007, ADR 008 (see "Relations to other ADRs") ·
|
||||
**Governed by:** `context/design-principles.md` §3, §5 and §7
|
||||
**Relates to:** ADR 003, ADR 005, ADR 007, ADR 008
|
||||
|
||||
**Summary.** Formulas are evaluated incrementally by a small in-house implementation of
|
||||
the DBSP core: Z-sets, streams and a synchronous circuit, in a crate named `zset`. A
|
||||
cell write becomes a delta. Aggregates are maintained rather than re-folded, and only
|
||||
the formula cells whose inputs moved are re-evaluated, by the one existing evaluator.
|
||||
Operator state is keyed by the **shape key** of a projection request, never by a client.
|
||||
The same circuit serves the in-process TUI and remote clients. A remote client learns of
|
||||
changes from one unaddressed notice per accepted batch, and it refetches what it shows.
|
||||
Sums are exact, through an exact accumulator.
|
||||
## Problem
|
||||
|
||||
File and line citations are at `gh/main` `50e1314d`. Line numbers drift, so re-derive
|
||||
them before relying on one.
|
||||
|
||||
## Problem and context
|
||||
|
||||
**Data that streams.** A live-backed model (ADR 003) is a source whose rows change
|
||||
underneath the document. Today every change takes the path an edit takes, which
|
||||
recomputes every formula cell. That is tolerable for one keystroke at a time and not for
|
||||
a stream of changes. A change should cost in proportion to what it touches.
|
||||
|
||||
**Today's evaluation is a from-scratch fold after every batch.**
|
||||
- `App::apply_effects` (`app.rs:1069`) ends in `rebuild_layout`, which calls
|
||||
`Model::recompute_formulas` (`types.rs:982`). That clears `formula_cache` and iterates
|
||||
every stem against every formula to a fixed point, bounded by `MAX_EVAL_DEPTH = 16`
|
||||
(`types.rs:710`).
|
||||
- Raw aggregation rescans the matching cells on every read (`evaluate_unfolded`,
|
||||
`types.rs:848`, through `DataStore::matching_values`).
|
||||
- `formula_cache` holds one aggregation shape at a time. A session on a view with another
|
||||
None axis gets a **cloned model** (`rederived_for`, `project.rs:205`).
|
||||
- A formula total that holds an aggregate walks a cross product
|
||||
(`contexts_of` → `Expansion::members`, `types.rs:916`, `:203`).
|
||||
- Two observations scan the store: `record_fingerprints` (`app.rs:266`) per cell-writing
|
||||
batch, and `document_content` (`app.rs:168`) after each `CellData` effect.
|
||||
- Every mutating frame from a client clones the document for rollback
|
||||
(`Host::mutating_frame`, `host.rs:224-225`).
|
||||
|
||||
**What an incremental evaluator must not change.** Each of these is a place where it
|
||||
could quietly disagree with today's code:
|
||||
- **Text unanimity (gotcha 7).** The all-numeric branch is checked first. Otherwise one
|
||||
distinct value is that value, and anything else is `Error("mixed")`.
|
||||
- **Circularity (gotcha 9).** A cell still moving after 16 passes is
|
||||
`Error("circular")`, and so is an acyclic chain that needs all sixteen.
|
||||
- **One epsilon (gotcha 17).** `FLOAT_EQ_EPSILON = 1e-10` serves `=`, `!=` and the div/0
|
||||
guard.
|
||||
- **Order is correctness (gotcha 18).** A hash index is for lookup, never for iteration,
|
||||
because f64 addition is not associative.
|
||||
- **The tiers are observed (gotcha 40).** A write of the value already there bumps
|
||||
nothing, and a rewritten record counts once.
|
||||
|
||||
**The view/host split (ADR 008).** A client reads through `project(request)`, where a
|
||||
request names axes, an absolute top-left and dimensions. It writes only through
|
||||
`apply(model command)`. The host keeps no UI state: no cursor, no viewport, no keys.
|
||||
Aggregation is always done on the model side. A view's hiding, collapsing, page
|
||||
selection, axis assignment and pruning are one client's ephemeral state until an
|
||||
explicit save-view command. Only the number format is document-wide.
|
||||
Every change recomputes every formula cell (`Model::recompute_formulas`, called by
|
||||
`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That
|
||||
does not scale to a live source (ADR 003) streaming changes into a large table, nor to
|
||||
ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without
|
||||
building the grid. A change should cost in proportion to what it touches.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. The crate: `zset`
|
||||
Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A
|
||||
cell write becomes a delta, aggregates are maintained, and only the formula cells whose
|
||||
inputs moved are re-evaluated by the existing evaluator. The incremental results must
|
||||
equal today's eager results exactly.
|
||||
|
||||
A workspace crate that knows nothing about improvise, like `gp-tree`, and depends only on
|
||||
`std`, so it is wasm32-clean by construction. It contains:
|
||||
- **`ZSet<K>`**, a map from `K: Ord + Clone` to a non-zero `i64` weight, forming an
|
||||
abelian group. Weights use checked arithmetic, and a zero weight is removed. The group
|
||||
laws are property-tested.
|
||||
- **A synchronous `Circuit`.** Operators are wired once, and `step` takes one delta per
|
||||
source and returns one per sink. There is one clock, one thread and no scheduler.
|
||||
- **Operators:** `delay`, `integrate` and `differentiate` (with I∘D = D∘I = id
|
||||
property-tested); the linear `map`, `filter` and `flat_map`; `index_by`;
|
||||
`join_indexed`, the bilinear delta join; `distinct`; and `aggregate`, with linear
|
||||
aggregators (`Count`, `Sum` over any abelian group) and integrated ones that keep a
|
||||
multiset per group.
|
||||
- **Steppable rebuilds.** Building an operator from an owned snapshot of its input is a
|
||||
resumable `step(deadline)`, so a rebuild can run as a sliced job.
|
||||
### 1. The `zset` crate
|
||||
|
||||
Left out: nested streams, recursion and least fixed points (clause 6); timestamps beyond one
|
||||
clock (improvise applies batches in one total order); threads within one circuit (a host
|
||||
with threads may run independent circuits on a pool); a query language. Spilling to disk
|
||||
is not in the first version, but operator state sits behind one interface (a trace: look
|
||||
up, read a range, apply a delta), so the store-memory work can page or compress it later.
|
||||
A problem-agnostic workspace crate with no dependencies beyond `std` (so it is
|
||||
wasm32-clean and serde-free). It provides:
|
||||
- `ZSet<K>`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group;
|
||||
- a synchronous, single-threaded `Circuit`, with `delay`, `integrate`, `differentiate`,
|
||||
`map`, `filter`, `flat_map`, `index_by`, the bilinear delta join `join_indexed`,
|
||||
`distinct` and `aggregate`;
|
||||
- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it
|
||||
can run as a sliced job;
|
||||
- operator state behind one trace interface (look up, read a range, apply a delta), so
|
||||
later store-memory work can page or compress it.
|
||||
|
||||
### 2. The plug: a trait, and a seam crate
|
||||
There are no nested streams, no recursion operator, no timestamps beyond one clock, and
|
||||
no query language.
|
||||
|
||||
improvise-core gains a trait, `Derived`, for the two things its evaluator reads: the
|
||||
formula result at a key, and the raw fold of the cells that match a partial key. It
|
||||
takes the cell delta and the formula delta after each model effect.
|
||||
### 2. The plug
|
||||
|
||||
**There is one formula evaluator.** `eval_expr` and its cached twin keep their code and
|
||||
read through the trait. The circuit decides *which* formula cells to re-evaluate and
|
||||
supplies their inputs. Formulas are not compiled into operators, because a compiled
|
||||
`IF`/`POW`/`WHERE` would be a second evaluator, which design-principles §5 calls the
|
||||
defect.
|
||||
improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the
|
||||
formula result at a key, and the raw fold of the cells that match a partial key. The
|
||||
incremental backend implements it in a seam crate, `improvise-incremental`, which
|
||||
depends on `zset` and improvise-core. improvise-core never depends on `zset`.
|
||||
|
||||
**The incremental backend lives in a seam crate, `improvise-incremental`,** which
|
||||
depends on `zset` and improvise-core. So improvise-core never depends on `zset`, the same
|
||||
shape as `gp-tree` and `improvise-gp`.
|
||||
The circuit decides *which* formula cells to re-evaluate. The one existing evaluator
|
||||
computes each value, so formulas are never compiled into operators.
|
||||
|
||||
**`Eager` is deprecated.** Today's code moves behind the trait with no behaviour change,
|
||||
as `Eager`. It stays as the oracle for the differential test while the circuit matures,
|
||||
and for the two cases that still need it (clauses 5 and 10). The differential test is a
|
||||
property test in the seam crate that applies one random edit sequence to both
|
||||
backends and compares every drawn cell, total and occupancy count. The goal is that every
|
||||
formula is evaluated through the circuit. What replaces `Eager` for its remaining uses is
|
||||
open question 15.
|
||||
**`Eager` is deprecated.** Today's code moves behind the trait as `Eager`, unchanged.
|
||||
It serves as the oracle of a differential property test, which applies one random edit
|
||||
sequence to both backends and compares every drawn cell, total and occupancy count. It
|
||||
is removed once its remaining uses are replaced (open question 3).
|
||||
|
||||
### 3. The input: the cell delta
|
||||
### 3. The cell delta
|
||||
|
||||
`DataStore::set` and `DataStore::remove` are the only places a cell changes. They emit a
|
||||
`ZSet<(CellKey, CellValue, Seq)>`, where `Seq` is the cell's store sequence: a number
|
||||
assigned when the key is inserted, in the same order `IndexMap` keeps, and kept by an
|
||||
overwrite:
|
||||
- a new cell is `+(k, v, s)`;
|
||||
- an overwrite is `−(k, v₀, s) + (k, v₁, s)`;
|
||||
- a removal is `−(k, v₀, s)`;
|
||||
- a remove-then-set is `−(k, v₀, s) + (k, v₁, s′)`, with a fresh `s′`.
|
||||
`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue, Seq)>`:
|
||||
- `+(k, v, s)` for a new cell;
|
||||
- `−(k, v₀, s) + (k, v₁, s)` for an overwrite;
|
||||
- `−(k, v₀, s)` for a removal;
|
||||
- a remove-then-set gets a fresh `s′`.
|
||||
|
||||
**Identity is bit identity**: `Number` by `f64::to_bits`, and `Text` and `Error` by their
|
||||
strings. That is `hash_value` (`app.rs:246`), which the record tier already hashes. So an
|
||||
equal-value overwrite consolidates to nothing and bumps nothing, `0.0` → `-0.0` is a
|
||||
change, and a NaN rewritten with the same bits is not. Multisets order values by
|
||||
`f64::total_cmp`, whose equality is bit equality. `CellKey` gains `Ord`. It is a sorted
|
||||
`Vec`, so deriving it is sound.
|
||||
`Seq` is a per-cell sequence assigned at insertion, in `IndexMap` order, and kept by an
|
||||
overwrite. Value identity is bit identity (`f64::to_bits`), so an equal-value overwrite
|
||||
consolidates to nothing. `CellKey` gains `Ord`. The circuit steps once per model effect.
|
||||
|
||||
**What the delta replaces.** The record tier reads the records named by the
|
||||
consolidated `(key, value)` projection of the delta, instead of diffing two store scans.
|
||||
The dirty flag's content half becomes "a non-empty projected delta, or a moved
|
||||
formula-set fingerprint". The fingerprint is needed because formula edits move no cell,
|
||||
and the worker persists on this flag.
|
||||
The record tier and the dirty flag read this delta instead of scanning the store. The
|
||||
dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the
|
||||
formula set moves, because formula edits move no cell.
|
||||
|
||||
**The circuit steps once per model effect.** That keeps occupancy current inside the
|
||||
effect loop. What a client counts is a separate publication number (clause 12).
|
||||
Under ADR 007, `CellKey` becomes id-based. A rename then emits no delta. The
|
||||
formula-set fingerprint must use the id form too, or a rename reads as a formula change.
|
||||
|
||||
**Keys and ADR 007.** Every key here follows `CellKey`: the delta, the stems, `record_of`,
|
||||
the group keys and the output's tuples. When `CellKey` becomes id-based under ADR 007, a
|
||||
rename emits no cell delta and rebuilds nothing. It is a display relabel. The formula-set
|
||||
fingerprint, taken today over `(raw, target, target_category)`, must then be taken over
|
||||
the id form, or a rename reads as a formula change.
|
||||
### 4. Aggregates
|
||||
|
||||
### 4. Aggregates: linear where they can be
|
||||
A group is the cell key minus the None categories. One aggregation circuit serves every
|
||||
shape key with the same None list, which removes the per-session model clone.
|
||||
|
||||
**The group** is the cell key minus the None categories, which is the partial key
|
||||
`evaluate_unfolded` folds over. One aggregation circuit serves every shape key with the
|
||||
same None list, which removes `rederived_for`'s model clone.
|
||||
|
||||
| answer | state per group | cost per delta element |
|
||||
| aggregate | state per group | per delta element |
|
||||
|---|---|---|
|
||||
| `Count` | a count of values and a count of numbers | O(1) |
|
||||
| `Sum` | an exact accumulator (clause 8) | O(1) |
|
||||
| `Avg` | `Sum` / `Count` | O(1) |
|
||||
| `Min`, `Max` | a multiset of the numbers | O(log d) |
|
||||
| unanimity and `mixed` | a multiset of all values | O(log d) |
|
||||
| `Count` | count of values, count of numbers | O(1) |
|
||||
| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) |
|
||||
| `Min`, `Max`, text unanimity | a multiset of values | O(log d) |
|
||||
|
||||
Reading a group reproduces `aggregate_cell_values`' branch order, with eager's own rules
|
||||
for NaN and signed zero (`PartialEq`, `f64::min`), not `total_cmp`'s. The differential
|
||||
test generates signed zeros and non-finite values to pin this. Only measures whose
|
||||
aggregator is `Min` or `Max`, or that hold text, need multisets. Summed over all groups,
|
||||
they hold at most one entry per stored cell.
|
||||
A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and
|
||||
semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the
|
||||
ordering category's order. How that order is keyed is for those tickets.
|
||||
|
||||
**Ordered aggregates** (`LAST`/`FIRST`, improvise-felo.7, and semi-additive measures,
|
||||
improvise-felo.8) are the same shape as `Min`/`Max`: an integrated map read at an
|
||||
extreme, keyed by the ordering category's order instead of by value. A semi-additive
|
||||
measure is an aggregate of aggregates, and it composes at O(log d) per stage. How the
|
||||
order is keyed is open question 2.
|
||||
### 5. Formulas
|
||||
|
||||
### 5. Formulas: the circuit finds what moved, and the evaluator recomputes it
|
||||
**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None
|
||||
categories". A formula counts at a stem where its top-level `WHERE` matches or raw data
|
||||
exists at the target.
|
||||
|
||||
**Stems** are `distinct` over the cell delta, mapped to "key minus the target category
|
||||
and the None categories". A formula counts at a stem only where its top-level `WHERE`
|
||||
(if any) matches the stem's coordinate, or where raw data exists at stem + target. That
|
||||
is exactly where `recompute_formulas` caches a value today, so pruning keeps its current
|
||||
behaviour.
|
||||
**Inputs.** A formula cell is re-evaluated when one of its inputs moved:
|
||||
- a `Ref(R)`: the group at the stem with `R`'s category set to `R`;
|
||||
- an `Agg`: the group at the stem;
|
||||
- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set
|
||||
to `Item`. This is the one join: stems indexed by "stem minus `Cat`".
|
||||
|
||||
**A formula cell is re-evaluated when one of its inputs moved:**
|
||||
- for a `Ref(R)`, the group at the stem with `R`'s category set to `R`;
|
||||
- for an `Agg` with no filter, the group at the stem itself;
|
||||
- for an `Agg` whose inner `WHERE` names `Cat = Item`, the group at the stem with `Cat`
|
||||
set to `Item`, which is another stem. This is the one join: stems indexed by "stem
|
||||
minus `Cat`", joined with the aggregate's output delta.
|
||||
The affected pairs are evaluated in the formula graph's topological order.
|
||||
|
||||
The affected `(stem, formula)` pairs are ordered topologically over the formula graph and
|
||||
handed to the existing evaluator.
|
||||
**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline:
|
||||
1. A per-record stage retracts and reinserts each touched record. It reads the record's
|
||||
previous cells as the store minus the delta, so it keeps no copy of the values.
|
||||
2. A map produces `(context, order key, expr)`. The order key is (context rank, minimum
|
||||
`Seq` of the record's live cells), which is eager's concatenation order.
|
||||
3. The group aggregate, with an error channel that reads the first `Err` in order.
|
||||
|
||||
**An aggregate over an expression** (`SUM(Revenue * Price)`) is a four-stage pipeline:
|
||||
1. **A per-record stage**, keyed by `record_of`. For each record a delta touches, it
|
||||
retracts the record's previous snapshot and inserts the new one. The previous snapshot
|
||||
is `I(x) − Δx`: the record's cells in the store after the step, minus the step's delta.
|
||||
So the stage reads the store and keeps no second copy of the values.
|
||||
2. **A lifted map** to `(context key, order key, expr at record)`. The context key
|
||||
removes the target and None categories, and at a fold or total the spanned categories,
|
||||
mapping a collapsed member to its group. The order key is (context rank, the minimum
|
||||
store sequence over the record's live cells), which is exactly the order eager
|
||||
concatenates in.
|
||||
3. **An inner `WHERE`** is a filter plus the join above.
|
||||
4. **The group aggregate**, with an error channel kept beside it that reads the first
|
||||
`Err` in order-key order, as eager does.
|
||||
|
||||
No cross product is ever built. The map is sound only when `expr` reads the record's own
|
||||
cells. Other formulas take a fallback that evaluates over the **occupied contexts** only,
|
||||
the contexts some record projects to. Because `agg_values` evaluates only at each
|
||||
context's `member_records` (`types.rs:91-139`), that gives eager's answer, with one
|
||||
edge: an inner bare `Ref` that names no item errors `ref:` for every context, and over
|
||||
zero occupied contexts it yields the empty value instead. Formulas targeting a category
|
||||
other than `_Measure` stay on the eager path until their order key is defined. Open
|
||||
question 8 covers both.
|
||||
No cross product is built. A formula whose expression reads other records is evaluated
|
||||
over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is
|
||||
unchanged, because a context with no records contributes nothing. One exception: an
|
||||
inner bare ref that names no item errors at every context, so for it the ref is checked
|
||||
before enumerating. This replaces `Expansion::members`.
|
||||
|
||||
### 6. Circular references
|
||||
|
||||
DBSP's recursion computes a least fixed point, and improvise's circularity is not one:
|
||||
`A = B + 1` with `B = A + 1` has no fixed point, and `X = X + 1` over a stored 10 climbs
|
||||
forever. The required answer is `Error("circular")`. The first version therefore decides
|
||||
circularity outside the crate:
|
||||
- the formula graph is rebuilt when the formula set changes, and its strongly connected
|
||||
components are found;
|
||||
- formulas outside every cycle are evaluated once per step, in topological order;
|
||||
- a formula in a cycle has its affected cells evaluated by today's bounded fixed point,
|
||||
restricted to its component. A guarded cycle such as `A = IF(x, B, 1)` is then still
|
||||
convergent where `x` is false;
|
||||
- the sixteen-pass bound is reproduced, so the two backends agree in the differential
|
||||
test.
|
||||
DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the
|
||||
answer must be `Error("circular")`. The first version finds the formula graph's strongly
|
||||
connected components. Acyclic formulas are evaluated once each, in topological order. A
|
||||
component is evaluated with today's bounded fixed point, restricted to that component.
|
||||
That reproduces the sixteen-pass bound for the differential test. It is to be replaced
|
||||
by a DBSP-native treatment (open question 2).
|
||||
|
||||
This serves testing and validation. It is to be replaced by a DBSP-native treatment of
|
||||
cycles, whose shape is open question 14.
|
||||
### 7. Operator state follows the request's shape key
|
||||
|
||||
### 7. The circuit follows the document per model effect
|
||||
|
||||
**Operator state is selected by a shape key.** The key is exactly what changes which
|
||||
cells feed which operator:
|
||||
- the categories of each axis, by level;
|
||||
A projection request's **shape key** selects operator state, shared by every request
|
||||
with the same key, from any client. The key is:
|
||||
- the categories on each axis, by level;
|
||||
- the None categories;
|
||||
- the filter keys: each filter category with its selected item, and the hidden items of
|
||||
every category on None, Page or Filter. Those items drop cells from the fold;
|
||||
- the page **categories**, which key a slice per page (open question 1);
|
||||
- `measure_agg`, which is model state, never taken from a request.
|
||||
- the filter keys: each filter category's item, and the hidden items of categories on
|
||||
None, Page or Filter;
|
||||
- the page categories, so each page is a maintained slice and paging rebuilds nothing;
|
||||
- `measure_agg`.
|
||||
|
||||
**Everything else is a read-time parameter**, applied when output is read: the hidden
|
||||
items of an axis category, collapsed groups and group membership, the drawn order of
|
||||
items, the page selection, and prune. A request carries its own key and parameters from
|
||||
its client's UI state, and a saved view only supplies defaults.
|
||||
Everything else in a request is applied when output is read: hidden and collapsed items
|
||||
on axis categories, item order, the page selection and prune. Occupancy and column
|
||||
aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item
|
||||
changes no operator. The visible axis is a remap.
|
||||
|
||||
**`view_shape` is not the shape key.** `view_shape` (`app.rs:40-72`) hashes the drawn
|
||||
items, the collapsed groups, the page selection and `prune_empty`, all of which are
|
||||
read-time parameters here. It keeps its one job, bumping `layout_version`. The circuit
|
||||
neither reads nor depends on that tier, which is a view fact. Only `document_generation`
|
||||
and the record tier matter to this ADR.
|
||||
- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas
|
||||
arriving during the build are buffered and added at install.
|
||||
- **A formula edit** is a formula delta fed to the circuit.
|
||||
- **A document swap** rebuilds everything.
|
||||
- **Saving a view definition** touches no operator.
|
||||
- **Live keys are a cache.** A key is evicted after a period in which no request names
|
||||
it (the period is set by measurement), and every eviction is announced (clause 9).
|
||||
|
||||
**Item grain** (recommended; see open question 6). Occupancy and per-column aggregates
|
||||
are keyed by the **items** of the axis categories, never by a position. So collapsing, expanding, hiding, showing,
|
||||
reordering and adding an axis item change no operator: the visible axis is a remap.
|
||||
ADR 008 measured the remap at 2.3 ms for 100k occupied tuples, against 6 to 45 ms for a
|
||||
rebuild. A collapsed group's value is combined from its members when read. That is exact
|
||||
for `Count`, and for `Sum` under the exact accumulator; for the integrated aggregates it
|
||||
costs O(members) per read (not measured). The exceptions are column totals (below).
|
||||
|
||||
**Only a new shape key builds operators**, as a job in 2 ms slices under ADR 008's job
|
||||
runner (8 ms per normal operation). Deltas that arrive during the build are buffered and
|
||||
added at install, which Z-set addition makes exact. A circuit `step` on a cell delta is
|
||||
O(delta) and is never sliced. Each step's occupancy output carries the count of tuples
|
||||
whose count crossed zero, so ADR 008's ordered index can choose between inserting and
|
||||
rebuilding. Measurements against the budget are stated as ratios against an unrelated
|
||||
operation on the same run, not as bare numbers.
|
||||
|
||||
**Formula edits are an input.** After each model effect the host compares a fingerprint
|
||||
of the formula set. If it moved, the formula delta is fed to the circuit, and the formula
|
||||
graph is re-derived. A document swap (`install_workbook`, which bumps
|
||||
`document_generation`) rebuilds every circuit. Saving a view definition moves no cell and
|
||||
no formula, so it touches no operator.
|
||||
|
||||
**The shape cache.** Live shape keys are a cache. A key enters when a request names it,
|
||||
and it is evicted when no recent request names it (open question 4). Every eviction is
|
||||
announced in the next notice header (clause 12). Evicting a popular shape causes a refetch storm
|
||||
by every client reading it. Idle readers make that likely, since a shape that clients
|
||||
only watch ages out. Mitigations are hysteresis, jitter, or rate-limiting through the job
|
||||
queue.
|
||||
|
||||
**Column totals subtract hidden rows at read time.** Today a column total pools only the
|
||||
drawn rows: the evaluator's fold memo skips hidden items (`layout.rs:797-803`, `:1066`),
|
||||
pinned by `collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was`. So a
|
||||
per-(column, measure) aggregate over all rows cannot answer by itself. Each total is kept
|
||||
over all rows, and a read subtracts the request's H hidden cells in that column as a
|
||||
temporary overlay:
|
||||
**Column totals and hidden rows.** A total pools only the drawn rows, so a hidden row
|
||||
leaves it. The shape key is shared and a hide is one client's, so totals are kept over
|
||||
all rows. A read subtracts the request's H hidden cells as a temporary overlay:
|
||||
- `Count` subtracts;
|
||||
- `Min` and `Max` walk from the extreme, skipping keys whose whole count is hidden;
|
||||
- the text rule counts the distinct keys left;
|
||||
- `Sum` subtracts through the exact accumulator.
|
||||
- `Min` and `Max` skip keys whose whole count is hidden;
|
||||
- text counts the distinct keys left;
|
||||
- `Sum` subtracts exactly through the accumulator.
|
||||
|
||||
That is O(H log d) per read for every aggregate, with no per-client state. The shape key
|
||||
is shared across clients and a hide is one client's, so a total cannot be rebuilt per
|
||||
hide. A hidden column item under a collapsed column group changes that group column's
|
||||
cells and total, which is a read-time recombination. Clause 5's aggregate-over-expression
|
||||
totals need their entries indexed by row tuple as well, so a read costs O(hidden
|
||||
records). The alternative is open question 12.
|
||||
That is O(H log d) per read for every aggregate, with no per-client state.
|
||||
Aggregate-over-expression totals index their entries by row tuple for the same
|
||||
subtraction.
|
||||
|
||||
### 8. Floating point: exactness comes from the representation
|
||||
### 8. Exact sums
|
||||
|
||||
A sum maintained by `+v` and `−v` in f64 drifts from the eager fold.
|
||||
`crates/improvise-core/examples/retraction_drift.rs` measured it on an Apple M4 Max,
|
||||
with values of up to five decimal places:
|
||||
An f64 sum maintained by `+v` and `−v` drifts. Measured by
|
||||
`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million
|
||||
cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer
|
||||
differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both
|
||||
backends:
|
||||
- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which
|
||||
makes it an abelian group that rounds once, on read;
|
||||
- measured bit-equal to a fresh sum after a million cycles, and exact against rational
|
||||
sums on 20,000 cases;
|
||||
- 568 bytes per group, and about 1.3× the cost of an f64 add per update.
|
||||
|
||||
| store | magnitudes | after 1M cycles: naive vs eager | eager vs exact |
|
||||
|---|---|---|---|
|
||||
| 1,000 cells | 3 decades | 6.4e-11 to 1.3e-10 | ≤ 7.3e-12 |
|
||||
| 18,000 cells | 3 decades | 1.7e-9 to 2.3e-9 | 1.5e-10 to 2.6e-10 |
|
||||
| 18,000 cells | 9 decades | 2.3e-5 to 3.2e-4 | 2.4e-4 to 2.5e-4 |
|
||||
It lives in ADR 005's `numtower` crate. **User-visible changes:** displayed sums change
|
||||
in their last bits, because they become correctly rounded, and a sum of `-0.0` reads
|
||||
`+0.0`.
|
||||
|
||||
Holding 1e17, adding 1,000 values of 0.01, then retracting the 1e17 reads 0.00 naively,
|
||||
against 10.00 for the eager and exact sums. A compensated sum is never bit-equal to eager.
|
||||
At 18k cells the gap reaches `FLOAT_EQ_EPSILON`, so `IF(SUM(..) = x, ..)` could answer
|
||||
differently on the two paths. Eager itself is already more than 1e-10 from the exact
|
||||
decimal sum.
|
||||
### 9. Delivery
|
||||
|
||||
**Sums use an exact accumulator, in both backends.** `ExactSum` is a fixed-point integer
|
||||
wide enough for every finite f64 (units of 2⁻¹⁰⁷⁴), with counts of NaN, +∞ and −∞ so
|
||||
that those can be retracted. Adding and retracting are integer additions, so the
|
||||
accumulator is a true abelian group, and it rounds once, on read. A weight is split into
|
||||
base-2³⁰ digits, so any `i64` weight is exact at O(1). Measured in the same example:
|
||||
- maintained equals fresh, bit for bit, at all 25 checkpoints after a million cycles;
|
||||
- it agrees with exact rational sums on 20,000 random cases (0 mismatches), and again
|
||||
with weights up to 2⁴⁰;
|
||||
- it costs 2.0 to 3.6 ns per update, against 1.5 to 2.7 ns for a plain add, and 0.6 to
|
||||
1.1 µs per read;
|
||||
- it is 568 bytes per group, against 8 for an f64.
|
||||
The same circuit serves the in-process TUI and remote clients. Its internal streams never
|
||||
cross a wire, because aggregation stays on the host.
|
||||
|
||||
`Avg` is the exact sum divided by the count, rounded once. A signed zero reads as +0.0,
|
||||
where f64 `sum()` gives −0.0. Displayed sums change in their last bits, because they
|
||||
become correctly rounded. `ExactSum` lives in `numtower`, ADR 005's numeric crate, and
|
||||
the seam crate wraps it for `zset`'s group trait. No existing crate was usable: `xsum`
|
||||
0.1.6 rounds about a quarter of single negative values wrongly and cannot retract
|
||||
non-finite values.
|
||||
**A request** is a shape key plus a window, given as an absolute cross-product top-left
|
||||
and dimensions. The window is never state, and a request costs O(visible).
|
||||
- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings.
|
||||
- **Output** is keyed by tuple identity, with the position attached when sent.
|
||||
- **An item insert, removal or reorder** is sent as an axis change, and each client
|
||||
re-derives its positions.
|
||||
|
||||
**Lossy operations happen at the edge.** Sums stay in the accumulator. Division,
|
||||
non-integer `POW` and display rounding happen only where a value leaves the exact domain.
|
||||
An exact value becomes an f64 through its decimal string, which is correctly rounded.
|
||||
**Pruning support.** Pruning is window-relative: a row or column empty within the
|
||||
displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model
|
||||
answers one restricted successor query: "the next k rows after position p with a value
|
||||
in a column of S", and the same for columns. A per-column sorted index of occupied rows
|
||||
answers it in O(|S| log n + k|S|), holding O(occupied cells). The client prunes its
|
||||
window, fills forward with that query, and finds a new top-left when scrolling up or left
|
||||
by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴). Whether
|
||||
pruning runs on the client and which parameters travel are ADR 008's questions. This
|
||||
design supports either answer.
|
||||
|
||||
**The accumulator does not make `=` against a typed decimal exact.** It sums the stored,
|
||||
already-rounded f64s, so it can miss the typed total by up to about half an ulp of each
|
||||
term. That miss reaches the absolute `1e-10` of `=` once terms reach about 2¹⁹. Example:
|
||||
`1000000000000000.10 + (-1000000000000000.00)` sums to 0.125, not 0.10. Only ADR 005's
|
||||
exact decimals fix that, and they also cover per-cell arithmetic and typed values such as
|
||||
money.
|
||||
**The change notice.** After each accepted batch the host emits one notice to every
|
||||
client, addressed to none. It carries:
|
||||
- the stamp `(host instance, document_generation, publication)`. The publication number
|
||||
increments once per published batch, and the host instance changes on restart;
|
||||
- axis changes, display relabels, and a flag when the formula set or `measure_agg` moved;
|
||||
- the shape keys evicted since the last notice;
|
||||
- for each live shape key, the moved output keys (cells, formula cells and totals), as
|
||||
the circuit computes them. Moved records alone would miss formulas with cross-stem
|
||||
inputs.
|
||||
|
||||
### 9. What ADR 008 gets
|
||||
A notice whose key lists exceed a fixed size carries "refetch" instead.
|
||||
|
||||
- **Occupancy**, per shape key at item grain: a count per row tuple and per column tuple
|
||||
(kept as a pre-filter and as the unpruned extent; see open question 11).
|
||||
Formula cells count by clause 5's stem rule. A tuple is occupied when its count is above zero.
|
||||
A fold row with a formula column but no stored cell under it, and the `Exact` case with
|
||||
nothing on None, are **not** occupied, on purpose.
|
||||
- **The ordered record list** stays the store's (`DataStore::live_records`). The circuit
|
||||
consumes it and never keeps a second order.
|
||||
- **The occupancy delta**, `distinct` over occupancy, is a model-side output at item
|
||||
grain. It keeps the pruning index current (clause 12) and is not pushed to clients, because
|
||||
the host cannot compute a client's visible axis.
|
||||
- **Column totals**, maintained per (column tuple, raw measure) at item grain, with the
|
||||
formula layer re-evaluating only the columns whose inputs moved (clause 7 for hidden rows).
|
||||
- **Widths for text columns** (clause 13).
|
||||
**What a client does.** It intersects the moved keys with its window. A fold row has no
|
||||
maintained key, so it maps member keys through its own remap first. It then refetches
|
||||
what moved. It refetches everything when it sees a publication gap, a new
|
||||
`document_generation` or host instance, its shape in the eviction list, or a notice it
|
||||
cannot read. A slow client may merge pending notices: the union of keys, the latest
|
||||
stamp, and axis changes kept in order. Outputs are consolidated under the wire's value
|
||||
identity, so every NaN counts as one value, because the wire spells every NaN `"nan"`.
|
||||
|
||||
### 10. Live sources
|
||||
**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`,
|
||||
`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The host keeps no per-client
|
||||
state. `GridExtent` is derived by the client, and in records mode the record count is
|
||||
fetched.
|
||||
|
||||
A live source that can report its changes produces a cell delta, and the circuit cannot
|
||||
tell it from `DataStore`'s. A source that can only answer aggregate queries stays on
|
||||
`Eager`'s pushdown in the first version. Its replacement is open question 15. Integrated
|
||||
state grows with its input, which conflicts with ADR 003's larger-than-memory goal, so
|
||||
the backend is chosen per document.
|
||||
**A connection operator** at the wire boundary adapts the circuit to a connection. It is
|
||||
a sink for results and notices and a source of requests toward the circuit, and the
|
||||
mirror image toward the client. It owns serialization, the stamp rules, notice
|
||||
coalescing, keep-alive in both directions, and session resumption. In process it is
|
||||
function calls. Its shape is open question 4.
|
||||
|
||||
### 11. Constraints
|
||||
### 10. Text column widths
|
||||
|
||||
- **wasm32.** `zset` has no dependencies. If the command layer selects the backend, then
|
||||
`zset`, `improvise-incremental` and `numtower` join the browser bundle and the `wasm`
|
||||
CI job, each checked with `cargo tree --edges normal`.
|
||||
- **Two seconds per test binary.** The crate's laws and properties run at 256 cases over
|
||||
small inputs, and the differential test lives in the seam crate's own binary.
|
||||
- **Determinism.** A `ZSet` iterates in `Ord` order, output deltas are consolidated and
|
||||
sorted, and nothing in the circuit decides a display order. The determinism test builds
|
||||
**two** circuits independently from one history.
|
||||
- **Rollback (ADR 001).** The circuit's state joins the `App` snapshot that
|
||||
`Host::mutating_frame` takes. An inverse delta is not a rewind: it restores the
|
||||
integrated values but not the history `delay` and `differentiate` hold. The cost is a
|
||||
second O(state) clone per mutating frame, and peak memory to match. The keyboard path
|
||||
takes no snapshot.
|
||||
- **Replay (ADR 006).** A circuit is a deterministic function of its delta sequence.
|
||||
The host maintains each text column's maximum display width as a histogram from width to
|
||||
count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows
|
||||
come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule.
|
||||
|
||||
### 12. Delivery: one circuit, in process or across a wire
|
||||
### 11. Rollback and memory
|
||||
|
||||
The DBSP machinery is **network-transparent**: the same operators and streams whether a
|
||||
consumer is in process or across a wire.
|
||||
- **Rollback.** The circuit's state joins the snapshot `Host::mutating_frame` takes. An
|
||||
inverse delta is not a rewind, because `delay` and `differentiate` keep history.
|
||||
- **Memory.** Resident state beside the store is the per-cell `Seq`, the multisets,
|
||||
`ExactSum` per summed group, occupancy per live shape, the successor-query index and
|
||||
the width histograms. The store itself is the integrated base and is never copied. It
|
||||
is measured at ADR 008's fixture before the store-memory work begins.
|
||||
|
||||
**What crosses a wire.** The outputs of `project` (cells, totals, width facts) and their
|
||||
change notices, plus the input deltas of a remote live source. The write side carries
|
||||
model commands with their basis, never deltas. The circuit's internal streams stay on
|
||||
the host, because aggregation is model-side. No UI state crosses.
|
||||
## Alternatives rejected
|
||||
|
||||
**A request has a shape key and a window.** The key selects operator state, shared by
|
||||
every request from any client. The window (absolute top-left and dimensions) selects
|
||||
part of the output and is never state. So a request costs O(visible).
|
||||
- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it
|
||||
fails to build for wasm32.
|
||||
- **Compiling formulas into operators:** a second evaluator.
|
||||
- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a
|
||||
second description of evaluation.
|
||||
- **Per-client filtering of notices, or pushed per-window deltas:** they save traffic
|
||||
only by holding per-client state on the host (open question 1).
|
||||
|
||||
**Positions are absolute cross-product positions**, in each category's own item order,
|
||||
and a bigint where they do not fit 2⁵³ (sent as a JSON string). They do not move when
|
||||
occupancy changes. They do move when an item is inserted, removed or reordered, so the
|
||||
recommended design (open question 10) is:
|
||||
- output elements are keyed by tuple identity (item ids under ADR 007), with the absolute
|
||||
position attached when sent;
|
||||
- an item edit is sent to every client as an **axis change** ("inserted at index j of
|
||||
level L"). A client re-derives its positions at O(levels) each and re-anchors its
|
||||
top-left to the same tuple;
|
||||
- resolving positions needs rank and select over each category's items, which is ADR
|
||||
008's order-statistic structure.
|
||||
## Consequences
|
||||
|
||||
**Pruning's model-side support.** Pruning hides a row or column that is empty within the
|
||||
displayed values, so a value outside the window does not count. That differs from
|
||||
today's `prune_empty`, which judges the whole grid (`layout.rs:752-757`). On a fixed
|
||||
window, one pass that drops the empty rows and columns is already the fixpoint. A fill
|
||||
then adds rows and columns forward from the absolute top-left, until the requested
|
||||
dimensions are met or the axes end. Growth is only rightward and downward. For the fill,
|
||||
the model answers one **restricted successor query**: "the next k rows after absolute
|
||||
position p with a value in at least one column of S", with S in the call, and the same
|
||||
for columns. A per-column index of occupied rows answers it: a sorted list per column,
|
||||
seeked to p and merged forward with a linear minimum over the |S| cursors, at
|
||||
O(|S| log n + k|S|). The index holds O(occupied cells) beside the store, fed from the
|
||||
cell delta and the stem rule. The query only looks forward, so a sorted `Vec` or a
|
||||
B-tree with a lower-bound seek suffices, with no rank over occupied tuples. A fill's
|
||||
whole state is its request and its last absolute position, so a resume is a seek.
|
||||
**Scrolling up or left is a new request with a new top-left.** How the client finds that
|
||||
top-left when rows are pruned is the recommendation of open question 9: it gallops
|
||||
backwards in doubling jumps until a probe hits, then narrows until one new row is gained.
|
||||
Each probe is a restricted successor query bounded by the old top-left. That is about 14
|
||||
to 28 probes at density 10⁻⁴, and no special answer is needed when a top-left lands on an
|
||||
empty row. The alternatives there are a model-side predecessor query, and failing the
|
||||
fetch and retrying. Which side of the wire runs the prune and the fill is
|
||||
open question 9.
|
||||
**Good:**
|
||||
- A write's evaluation costs O(delta) plus the formula cells it reaches.
|
||||
- The store scans and the per-view model clones go.
|
||||
- ADR 008 gets occupancy, totals and widths from one evaluator.
|
||||
- Live sources that report changes plug in.
|
||||
- The host holds no per-client state.
|
||||
|
||||
**The wire properties**, which hold on each hop:
|
||||
- **T1. Plain data.** Output elements are serializable values that name cells by tuples,
|
||||
positions and ADR 002 surrogates, never by `CellKey`, which is not `Serialize`.
|
||||
- **T2. One identity.** The wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114`),
|
||||
so a value change that only a NaN payload distinguishes cannot be told apart remotely.
|
||||
Which identity outputs use is open question 5.
|
||||
- **T3. A publication stamp.** Each accepted batch is published once, stamped
|
||||
`(host instance, document_generation, publication)`. The publication number increments
|
||||
once per published batch, so a multi-effect batch is one header, and a refused frame
|
||||
consumes none. The host instance changes on restart. A client drops anything at or
|
||||
below the number it has applied. **The refetch rule:** a client that sees a gap, a new
|
||||
`document_generation` or host instance, its shape in an eviction list, or a notice it
|
||||
cannot interpret, refetches.
|
||||
- **T4. Snapshot, then deltas.** A fetch is the snapshot, and later notices are relative
|
||||
to its stamp.
|
||||
- **T5. Coalescing.** A slow client merges pending notices: the union of moved keys and
|
||||
flags, under the latest stamp. Axis changes are positional, so they stay an ordered
|
||||
list. A `document_generation` change inside the run voids the merge. A basis is never
|
||||
sent ahead of the cells it describes (gotcha 43).
|
||||
- **T6. One API.** The TUI calls `project` and receives the same types as a remote
|
||||
client. A test runs one edit sequence through an in-process consumer and through a
|
||||
serialize-and-deserialize round trip, and requires equal results.
|
||||
**Bad:**
|
||||
- Two backends exist until `Eager` goes.
|
||||
- Integrated state is O(cells) beside the store.
|
||||
- `ExactSum` costs 568 bytes per summed group.
|
||||
- Every client receives every notice.
|
||||
- A mutating frame still clones the state for rollback.
|
||||
|
||||
**The change notice.** After each accepted batch the host emits one notice, addressed to
|
||||
no client. It carries:
|
||||
- the stamp;
|
||||
- the axis changes;
|
||||
- display relabels (`(id, new display)` under ADR 007);
|
||||
- a flag when the formula set or `measure_agg` moved;
|
||||
- the eviction list;
|
||||
- for each live shape key, the **moved output keys**: cells, formula cells and
|
||||
column-tuple totals, as the circuit computes them.
|
||||
**Order** (each step is a ticket in the implementation epic, behind the differential
|
||||
test):
|
||||
1. `zset`;
|
||||
2. the `Derived` trait with `Eager`;
|
||||
3. the cell delta, the record tier and the dirty flag;
|
||||
4. order-free aggregates and occupancy, measured at ADR 008's fixture;
|
||||
5. formulas and the aggregate-over-expression pipeline;
|
||||
6. `Sum`, `Avg` and totals through `ExactSum`;
|
||||
7. live sources;
|
||||
8. delivery.
|
||||
|
||||
Moved records alone would not do. From a moved record, a client cannot tell whether a
|
||||
formula with a cross-stem input moved. For example, `EastShare = Revenue / SUM(Revenue
|
||||
WHERE Region=East)` on page `Region=West` moves when an East record is written. A client
|
||||
intersects the moved keys with its window by identity. A collapsed fold row has no
|
||||
maintained key, so a moved member key goes through the client's own member-to-group
|
||||
remap first. The client then refetches what moved, by `project` or by the successor
|
||||
query.
|
||||
|
||||
**Every client receives every notice**, and the host holds no per-client state for
|
||||
delivery. The traffic is O(clients × all moved keys) per batch. A notice whose key lists
|
||||
exceed a fixed size carries "refetch" instead, which bounds every notice at the cost of a
|
||||
refetch by every client in a busy batch. Today's per-mirror push (`Sessions`,
|
||||
`ClientSession`, `grid_updates`, `Projection`, `GridUpdate`, `Dirty`) goes, and
|
||||
`GridExtent` is derived by the client from category lists. In records mode the row
|
||||
extent is the admitted record count, which the client fetches. Whether notices should be
|
||||
filtered per client is open question 3.
|
||||
|
||||
**A connection operator, at the wire boundary.** A connection operator sits at the wire
|
||||
boundary as a sink/source adapter for the circuit. Toward the circuit it is a sink for
|
||||
results and notices and a source of requests and model commands. Toward the client it is
|
||||
the mirror image. It owns:
|
||||
- serialization (T1, T2);
|
||||
- stamps and the refetch rule (T3);
|
||||
- coalescing (T5);
|
||||
- keep-alive in both directions. A client that hears neither a header nor a ping within
|
||||
the timeout treats the connection as dropped, so silence is never read as "no
|
||||
batches";
|
||||
- session resumption.
|
||||
|
||||
In process it is function calls, as in the TUI and in the browser's `Worker`, which
|
||||
already owns one `Host` and one port (`worker.rs:14-15`). The wire types belong in
|
||||
`improvise-protocol`, and `zset` stays free of serde. Where its halves live (one
|
||||
component per connection, or a host half shared per host process with a client half in
|
||||
every front end), and who owns resumption state, are open question 17.
|
||||
|
||||
### 13. Column widths for text
|
||||
|
||||
The host maintains each text column's maximum display width incrementally. A maximum is
|
||||
not invertible, so it is an integrated aggregate. Display widths are small integers, so
|
||||
the state is a histogram from width to count, per (shape key, column): O(distinct widths),
|
||||
at O(log d) per update. Its input is the delta of the column's text cells over every row,
|
||||
which the circuit already produces for unfolded cells. Fold rows need the fold maintained
|
||||
at the visible grain. A text width does not depend on the number format, so there is no
|
||||
rebuild when the format changes. A client's hidden rows come out through clause 7's overlay.
|
||||
Numbers keep ADR 008's rule: sticky per session with hysteresis, over the rows realized,
|
||||
held by the view layer. A number's width would need formatting on the write path, and a
|
||||
rebuild per format change. Which columns count as text is open question 16.
|
||||
|
||||
### 14. Resident state
|
||||
|
||||
Beside the store, the circuit keeps:
|
||||
- each cell's store sequence, for clause 5's per-record stage (one integer per cell,
|
||||
kept in `DataStore`);
|
||||
- the multisets for `Min`, `Max` and text, O(cells) of those measures;
|
||||
- `ExactSum` at 568 bytes per summed group;
|
||||
- occupancy per live shape, O(occupied tuples);
|
||||
- the restricted-query index, O(occupied cells);
|
||||
- the width histograms;
|
||||
- the rollback snapshot per mutating frame.
|
||||
|
||||
The store itself is the integrated base that a rebuild reads, so it is never copied.
|
||||
State is evicted with its shape, and it sits behind one trace interface, so the
|
||||
store-memory work (offload to disk, compression) can page or compress it later. None of
|
||||
these sizes is measured at ADR 008's fixture. Migration step 4 measures them.
|
||||
|
||||
### Relations to other ADRs
|
||||
|
||||
- **ADR 001:** the basis check on writes is unchanged. This ADR relies on its
|
||||
`document_generation` and record tiers, not on `layout_version`.
|
||||
- **ADR 005:** sums use `ExactSum` in its `numtower` crate (not yet in the workspace;
|
||||
epic improvise-59nw). Its exact decimals cover per-cell arithmetic and typed values.
|
||||
Its own decimal accumulator is exact per partition (a 320-bit decimal partition and a
|
||||
256-bit numerator per fraction group), up to K = 8 distinct non-decimal denominators.
|
||||
- **ADR 006:** replay reproduces a circuit's state.
|
||||
- **ADR 007:** keys become ids, so a rename is a relabel (clause 3).
|
||||
- **ADR 008:** the virtualized grid and the view/host split consume occupancy, the
|
||||
occupancy delta, totals and widths from this circuit (clause 9). Which side of the wire
|
||||
prunes, and which parameters a request carries, are shared with questions 9 and 13
|
||||
below.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Feldera's `dbsp` crate.** 0.354.0 has 67 required direct dependencies (tokio,
|
||||
mimalloc, zstd and others), about 421 crates for wasm32, and fails
|
||||
`cargo check --target wasm32-unknown-unknown` at `mio`. It is a multi-threaded,
|
||||
storage-backed engine where improvise needs a single-threaded operator library.
|
||||
- **Keep eager and memoize** (improvise-swre). Worth doing either way, but it removes only
|
||||
the cost of not changing: every write still recomputes every formula.
|
||||
- **An ad-hoc incremental cache.** Each derived fact needs its own invalidation rule,
|
||||
which is a second description of evaluation. Z-sets give one rule, linearity, whose laws
|
||||
are tested once.
|
||||
- **Compile formulas into operators.** A second evaluator (clause 2).
|
||||
|
||||
## Consequences and migration
|
||||
|
||||
**Good.** A keyboard write's evaluation costs O(delta) plus the formula cells it reaches.
|
||||
The store scans behind the record tier and the dirty flag go, and so do the per-view model
|
||||
clones. ADR 008 gets occupancy, totals and widths without a second evaluator. Live sources
|
||||
that can report changes plug in unchanged. The host keeps no per-client state, and one
|
||||
delivery path serves every front end.
|
||||
|
||||
**Bad.** Two backends exist until `Eager` goes, kept equal by a differential test that
|
||||
must itself be kept honest. Integrated state is O(cells) beside the store. `ExactSum` costs
|
||||
568 bytes per group, and displayed sums change in their last bits. Circularity is decided
|
||||
in two layers. The network path still clones per mutating frame. Every client receives
|
||||
every notice. Every column-total read pays O(H log d) for the request's hidden rows.
|
||||
|
||||
**Order.** Each step is a ticket and a PR in the implementation epic, and none switches
|
||||
the displayed path until the differential test holds:
|
||||
1. the `zset` crate, with its laws, operator properties and determinism test;
|
||||
2. the `Derived` trait with `Eager` behind it (no behaviour change), and `CellKey: Ord`;
|
||||
3. the cell delta, with the record tier and the dirty flag moved onto it, tested with a
|
||||
signed zero, a same-bits NaN, and a formula edit that alone marks the document dirty;
|
||||
4. `Incremental` for the order-free parts (stems, `Count`, `Min`, `Max`, unanimity and
|
||||
occupancy) under the differential test, with step and rebuild costs measured at ADR
|
||||
008's fixture. ADR 008's pruning depends on this step;
|
||||
5. formulas: the graph, topological re-evaluation, the component fixed point, and clause 5's
|
||||
aggregate-over-expression pipeline, tested by changing one operand of `Revenue * Price`
|
||||
and by deleting a record's leading cell. ADR 008's retirement of `Expansion::members`
|
||||
depends on this step;
|
||||
6. `Sum`, `Avg` and column totals through `ExactSum` in both backends, with the hidden-row
|
||||
overlay;
|
||||
7. live sources as delta streams, once ADR 003's source trait exists;
|
||||
8. delivery (clause 12). Its required tests: a batch of `[AddItem, SetCell]` publishes one
|
||||
header and no client refetches; a dropped header is detected; a reconnect after a host
|
||||
restart refetches and drops nothing as a duplicate; a refused frame consumes no
|
||||
publication number; and a dropped connection forces a refetch.
|
||||
Step 8's required tests:
|
||||
- `[AddItem, SetCell]` publishes one notice, and no client refetches;
|
||||
- a dropped notice is detected;
|
||||
- a reconnect after a host restart refetches;
|
||||
- a refused frame consumes no publication number.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Does a page selection read a slice, or rebuild?** If the shape key includes the page
|
||||
categories, every page's state is maintained and `[`/`]` only reads another slice. The
|
||||
held key stays O(visible), and the state stays bounded by the stored cells. If it does
|
||||
not, a page change builds operators like any other new key. *Recommendation:* the
|
||||
slice, which clause 7 assumes.
|
||||
1. **Should notices be filtered per client?** Clause 9 sends every client every live
|
||||
shape's moved keys. That keeps the host free of per-client state, but traffic is
|
||||
O(clients × all moved keys) per batch. Filtering by topic would send each connection
|
||||
only its own shape's key lists. It needs one subscription per connection, which pins
|
||||
its shape in the cache. Is such a subscription "model-relevant information" the
|
||||
backend may hold? It could also live in the connection operator (question 4).
|
||||
*Recommendation:* broadcast with the size cap. Add topics only if measured traffic
|
||||
with many clients on different shapes justifies them.
|
||||
|
||||
2. **How are `LAST`/`FIRST` keyed?** By item position, which goes stale on a reorder,
|
||||
since a reorder rebuilds nothing. By item identity with the order looked up at read
|
||||
time, which makes reads O(d). Or by an order label: a key that compares in the current
|
||||
order and is not renumbered by an insert elsewhere. Reads stay O(log d), and a reorder
|
||||
re-keys every entry stored under the moved items. *Recommendation:* the order label,
|
||||
with only the drawn members ordered. Not measured.
|
||||
2. **How should cycles be handled natively?** Clause 6's component fixed point is
|
||||
interim. The options:
|
||||
- (a) a recursion operator bounded at sixteen iterations, writing `circular` where it
|
||||
has not converged, which keeps today's answers;
|
||||
- (b) marking every cell in a graph cycle `circular`, which changes guarded cycles
|
||||
such as `A = IF(x, B, 1)`;
|
||||
- (c) a monotone reformulation, where one exists.
|
||||
|
||||
3. **Should notices be filtered per client?** Clause 12's notice replaces the pushed
|
||||
stream. The rejected alternative is to send no notice at all and let each client
|
||||
refetch on its own schedule: it keeps the host silent, but another client's change
|
||||
stays invisible until the next refetch. Broadcast (clause 12) keeps no per-client state,
|
||||
but every client receives every live shape's moved keys. The options:
|
||||
- **topics:** a client receives only its own shape key's key lists. Headers still go to
|
||||
everyone, so the refetch rule keeps working. That costs one subscription per
|
||||
connection, registered atomically with the fetch that names it, and pinning its
|
||||
shape;
|
||||
- **window push:** per-client subscriptions holding the window, sending only the
|
||||
window's deltas. That costs the most host state.
|
||||
*Recommendation:* (a).
|
||||
|
||||
Is a per-connection subscription "model-relevant information" the backend may hold,
|
||||
or UI state it must not? It need not live in the host at all if the connection
|
||||
operator holds it (question 17). *Recommendation:* broadcast with the size cap. Adopt
|
||||
topics only if measured traffic, with many clients on different shapes, justifies
|
||||
them.
|
||||
3. **What replaces `Eager`?** It still serves as the differential oracle and as the
|
||||
backend for live sources that can only answer aggregate queries. *Recommendation:* a
|
||||
small from-scratch reference implementation, kept in the test crate, as the oracle,
|
||||
and a pushdown backend behind the same trait for query-only sources.
|
||||
|
||||
4. **How long does a shape key live?** A key is evicted once no recent request names it.
|
||||
Each eviction is announced and makes its readers refetch and rebuild. "Recent" trades
|
||||
memory against refetch storms. If the connection operator re-requests the shapes its
|
||||
clients read, that is a keep-alive by another name: one host `project` per shape per
|
||||
eviction period. *Recommendation:* evict after a period with no request, set by
|
||||
measurement, with hysteresis for shapes with many readers.
|
||||
4. **What shape is the connection operator?** Either one component per connection, or
|
||||
one per host process: a host half shared by all its connections, with a client half
|
||||
in every front end. The second lets the host send one notice per operator rather than
|
||||
per client, but holds per-connection state in the shared operator. Who owns
|
||||
resumption state, and is it backend state? *Recommendation:* one per host process,
|
||||
with resumption state in the host half keyed by a client-held token, treated as
|
||||
delivery state rather than UI state. In the TUI and the browser worker, with one
|
||||
client each, it is a function call.
|
||||
|
||||
5. **Which identity do outputs use?** The circuit compares values by bits, so two NaNs
|
||||
with different payloads differ, but the wire spells every NaN `"nan"`. A remote client
|
||||
would therefore see such a change as nothing. *Recommendation:* outputs consolidate
|
||||
under the wire's identity (every NaN one value), and the record tier keeps bit
|
||||
identity.
|
||||
|
||||
6. **Item grain, crossing counts and steppable rebuilds.** The design keys occupancy and
|
||||
column aggregates by items, reports zero crossings per step, and slices rebuilds. That
|
||||
makes collapse, hide, reorder and add item free, at the cost of recombining a
|
||||
collapsed group's members on read. *Recommendation:* adopt all three, as written.
|
||||
|
||||
7. **Resident state.** Should the first version keep no second copy of the store (the
|
||||
store is the integral), put operator state behind one trace interface, and measure
|
||||
state at ADR 008's fixture before the store-memory work begins? *Recommendation:*
|
||||
yes.
|
||||
|
||||
8. **Formulas the per-record map cannot express.** They take the occupied-contexts
|
||||
fallback (clause 5).
|
||||
- (a) Should `Eager` use the same enumeration, so the two backends agree? That changes
|
||||
eager's answer only at the bare-ref edge, which could also be kept by checking the
|
||||
ref once before enumerating.
|
||||
- (b) Should formulas targeting a category other than `_Measure` take the same
|
||||
fallback, so that nothing still needs `Expansion::members`?
|
||||
|
||||
*Recommendation:* yes to both.
|
||||
|
||||
9. **Which side of the wire prunes and fills?** On the client, the model offers only
|
||||
windows and the restricted successor query, and the fill is the client's, with no host
|
||||
state. On the model, prune is a request parameter, and the model keeps a pruned axis,
|
||||
with rank and select for "row i of n", and runs the fill as a sliced job resumable
|
||||
from (request, fill front). At a density of 10⁻⁴, a client without the successor query
|
||||
would ship mostly empty windows. A second part: when scrolling up or left, how does the
|
||||
client find a new top-left when rows are pruned? The options are galloping backwards
|
||||
with successor probes (about 14 to 28 probes at density 10⁻⁴, with no host state), a
|
||||
model-side predecessor query (one round trip, no new state, but a second query
|
||||
direction), or failing the fetch and retrying with the nearest occupied row named. ADR
|
||||
008 asks the same question. *Recommendation:* the client prunes and fills, and it
|
||||
gallops.
|
||||
|
||||
10. **Are deltas keyed by identity, with item edits sent as axis changes?** Keying deltas
|
||||
by position instead goes stale on every item insert. Identity costs rank and select
|
||||
per lookup, at O(levels × log items) (not measured). *Recommendation:* identity, as
|
||||
clause 12 assumes.
|
||||
|
||||
11. **Should whole-extent per-axis occupancy be kept?** The per-column index (clause 12) answers
|
||||
the restricted successor query. Per-axis counts can only skip rows that are empty
|
||||
everywhere, not rows empty in S, so they serve as a pre-filter and as the unpruned
|
||||
extent. *Recommendation:* keep them in that role. Not measured.
|
||||
|
||||
12. **Column totals under a hide.** The design subtracts the hidden rows at read time, at
|
||||
O(H log d), with no state per hidden set (clause 7). The alternative keeps totals per (shape
|
||||
key, hidden set): O(1) reads, with one O(cells) build (or an O(H) derivation) per
|
||||
distinct hidden set in use, and a cache that multiplies. *Recommendation:* subtract at
|
||||
read time. Memoise per hidden set only if a measured hidden set is too large for the
|
||||
frame budget.
|
||||
|
||||
13. **Which read-time parameters travel in a request?** The page selection travels, as a
|
||||
third axis. Hidden items and collapsed groups change model-side totals and folds, so
|
||||
without them a client could not get those from the model. Alternatively, a collapse
|
||||
could be expressed as an axis at the group's level and a hide as a filter exclusion,
|
||||
but both change the shape key, which costs a build per state. The drawn order, the
|
||||
number format and prune change no aggregate. *Recommendation:* hidden items and
|
||||
collapsed groups travel; the rest stay with the client.
|
||||
|
||||
14. **How are cycles handled natively?** The bounded fixed point (clause 6) is for testing and
|
||||
validation. The options:
|
||||
- (a) a recursion operator bounded at sixteen iterations, which writes `circular`
|
||||
where it has not converged and keeps today's answers;
|
||||
- (b) marking every cell in a graph cycle `circular`, which changes guarded cycles
|
||||
such as `A = IF(x, B, 1)`;
|
||||
- (c) a monotone reformulation, where one exists.
|
||||
|
||||
*Recommendation:* (a) first.
|
||||
|
||||
15. **What replaces `Eager`?** Its remaining uses are the differential test's oracle,
|
||||
query-only live sources, and the formulas of question 8 that stay eager.
|
||||
*Recommendation:* a small from-scratch reference implementation, kept in the test
|
||||
crate, as the oracle; a pushdown backend behind the same trait for query-only
|
||||
sources; and the fallback of question 8 as a circuit stage.
|
||||
|
||||
16. **Which columns get a host-maintained width?** Columns whose cells are text, with
|
||||
numbers keeping the sticky width with hysteresis? And what of a column that holds
|
||||
both? *Recommendation:* text cells get the maintained width, and the view layer takes
|
||||
the larger of that and its own numeric width.
|
||||
|
||||
17. **What exactly is the connection operator?** (clause 12)
|
||||
- (a) Do its "both ends" mean both halves in one component, or one component on each
|
||||
side of the wire?
|
||||
- (b) Is it a new crate or process, or a role for existing code (the browser
|
||||
`Worker`, the socket server's `Registry` and transport, in-process code in the TUI)?
|
||||
- (c) Who owns keep-alive and resumption state, and is that backend state?
|
||||
- (d) Should one operator serve many connections and subscribe to the host on their
|
||||
behalf, so that the host sends O(operators) per batch, not O(clients)? That saving
|
||||
needs one operator per host process serving many clients, so per-connection state
|
||||
then lives inside a shared operator.
|
||||
|
||||
*Recommendation:* (a) one operator per host process: a host half shared by its
|
||||
connections, and a client half in every front end. (b) a role of the existing
|
||||
transport crates, not a new process. (c) the host half holds resumption state, keyed
|
||||
by a token the client holds, as delivery state, not UI state. (d) yes, for the socket
|
||||
server. The TUI and the browser worker each serve one client, so there the operator is
|
||||
a function call.
|
||||
5. **Which columns get a host-maintained width?** Columns whose cells are text? What
|
||||
about a column holding both text and numbers? *Recommendation:* text cells get the
|
||||
maintained width, and the view layer takes the larger of that and its own numeric
|
||||
width.
|
||||
|
||||
Reference in New Issue
Block a user