fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(adr): ADR 009 revised to fit ADR 008's view/host split (improvise-felo.4)

Browse Source 
Adds the user's ADR 008 statements that bear on this ADR, clauses 12-14
(network transparency, incremental max width, resident state), keys
operator state by projection shape at item grain, retires the per-mirror
delivery assumptions, and appends questions 12-21. Questions 1-11 stay
open. Reads ADR 008 at 69f0a7c6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (adr9-steward)
2026-10-05 14:54:30 -07:00
committed by Edward Langley
parent 502373c025
commit a9bcb816e9
1 changed files with 714 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
context/adr/009-incremental-evaluation.md
+714 -72
View File
@@ -1,12 +1,19 @@
# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait
**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end
are theirs to answer on the PR. ·
are theirs to answer on the PR. **Revised on 2026-10-05 to fit ADR 008 revision 19**
(PR #308, read at `69f0a7c6`, which the user is still reviewing). The revision brings in
the view/host split, the user's answers on ADR 008 that bear on this ADR, and the details
ADR 008 routes here. They are in the section "What ADR 008's review established for this
ADR", in clauses 12 to 14, and in questions 12 to 18. **The user has answered none of
this ADR's questions 1 to 11.** That was checked on #309 on 2026-10-05: its only inline
comments are Copilot's two and our replies. ·
**Ticket:** improvise-felo.4 ·
**Decision from:** the user, 2026-09-25 (below) ·
**Builds on:** ADR 001 (the record tiers), ADR 003 (live-backed models) ·
**Relates to:** ADR 005 (exact decimals, improvise-l2b8, proposed on PR #320), ADR 008 (the
virtualized grid, PR #308, which is revised to assume this ADR) ·
virtualized grid and the view/host split, PR #308, which assumes this ADR; this revision
reads it at `69f0a7c6`, its revision 19) ·
**Governed by:** `context/design-principles.md` §3 and §5
**Numbering.** ADR 005 and 006 are claimed by improvise-l2b8 and improvise-ilvl. ADR 005
@@ -14,8 +21,13 @@ is proposed on PR #320, and 006 has no file yet. 007 is merged, and 008 is PR #3
no open PR claimed it when this was written.
**Citations.** Every `file:line` below is at `gh/main` `57d385e1`. Line numbers are
dated, so re-derive them before citing this document elsewhere. "ADR 008 clause N" means
revision 3 of `context/adr/008-virtualized-grid.md` on PR #308.
dated, so re-derive them before citing this document elsewhere. **The 2026-10-05
revision's new citations are at `gh/main` `50e1314d`**, and each says so. The older
citations were not re-derived, and some have moved. For example, `Host::mutating_frame`
takes its snapshot at `host.rs:224-225` at `50e1314d`, not `:162`, and
`App::apply_effects` starts at `app.rs:1069`, not `:989`. "ADR 008 clause N" means
revision 19 of `context/adr/008-virtualized-grid.md` on PR #308, at `69f0a7c6`. Clauses 3,
5 and 8, which this ADR cites most, have the same numbers there as in revision 3.
## The decision this ADR writes down
@@ -61,7 +73,123 @@ from scratch. The machinery is:
**The grid is the second consumer.** ADR 008 needs the same maintained state for other
purposes: occupancy for `prune_empty`, the occupancy delta for its frame observation,
and column totals over rows never scrolled to (clause 9). It gets them from the same
circuit, not from a second mechanism.
circuit, not from a second mechanism. Since ADR 008's view/host split, the grid's consumer
may be in another process, and clause 12 says how the same circuit serves both.
## What ADR 008's review established for this ADR
At its revisions 18 and 19, ADR 008 (PR #308) became the ADR of the **view/host split**, and the
user routed part of it here. This section quotes what the user said that bears on this ADR,
and says where each statement came from.
- **PR comments** are cited by their GitHub comment id and UTC time. The question numbers
are ADR 008's, as ADR 008 assigns them.
- **The two chat statements** were given to the orchestrator and relayed to this ADR. They
are on no PR, and ADR 008 records them the same way. The user's "Approved" on this ADR
would confirm them.
None of this answers this ADR's own questions 1 to 11.
**Routed here, in chat:**
- "The DBSP machinery needs to be implemented in a network transparent way, but the
details of that go to ADR 009" (chat, relayed on 2026-10-05). Clause 12.
- "Thinking about it a bit more, we might need to keep track of recent projections +
clients to stream aggregation results back to the UI correctly. But this should be as
stateless as possible" (chat, relayed on 2026-10-05). Clause 12.
- On ADR 008's question 53: "Q53 is cross product: I'm still ambivalent about which side
of the wire pruning should happen on" (chat, relayed on 2026-10-05). So a request's
top-left is a cross-product position, and which side prunes is open. ADR 008 records it
at revision 19 as its question 53 answered and its question 57 open. Clause 12,
question 19.
- "But the top left should be as absolute as possible" (chat, relayed on 2026-10-05).
Clause 12, question 20 (ADR 008's question 58).
- "After pruning we might have to stream rows and columns to attempt to fill out the full
grid" (chat, relayed on 2026-10-05). Clause 12, question 21 (ADR 008's question 59).
**Asked on ADR 008, and not yet answered:**
- "Maybe dbsp can give us incremental max width?" (#308, comment 4171487459,
2026-10-03T03:15:38Z, on ADR 008's question 2). Clause 13 is this ADR's reading of it.
Question 12.
**The read side and the write side** (#308, inline comments of 2026-10-05):
- Q43 (4189071865, 21:34:45Z): "yeah, the read side of the model is a projection of the
model's full space into a particular set of axes using a `project` verb, with
offset+dimensions. The write side is an apply function that can only take model
commands and applies them to the model using Redux-style Flux architecture."
- Q38 (4189007864, 21:27:55Z): "No, axes are a model concept because they define the
grid. Aggregation should always be done model-side for consistency."
- Q38 (4189027924, 21:30:08Z): "A grid request would look something like:"
```
Axes: Row(A,B,C), Col(D,E), Page(F), None(G,H)
Top-left (4,5) # relative to the cross-product of the axes in the category-native order (which might be arbitrary)
Dimensions (10,10) # so get ten rows and ten columns worth of data from the grid defined by the axes.
```
- Q33 (4188936570, 21:20:11Z): "The host shouldn't have to know anything about the
viewport: it's mostly a question of the parameters of the grid the user is actually
looking at, rather than a "viewport" strictly speaking."
- Q37 (4189000059, 21:27:04Z): "I don't think the host needs to no [sic] anything about
the UI: the initial load of the UI should just be the default view stored in the model
(or, in some cases, a particularly requested view). Sending UI state over the wire is
an architecture smell."
- Q42 (4189051110, 21:32:31Z): "The space scrolling happens in is defined by the axes, top
left corner offset and with/height. But how much of this space is actually materialized
and when fetches, etc. happens (and model values are cached) is client-policy driven."
- Q32 (4188917796, 21:18:17Z): "I can't think of any cursor command with model-side
effects."
- Q27 (4188816383, 21:07:43Z): "go with the recommendation: the ordered list of records is
a model-side concept."
- Q31 (4188911659, 21:17:39Z): "You have to distinguish view in the sense of UI and view
in the sense written into `.improv` files. P has to be a model-side effect because it
prunes data for a particular view. It is not a UI-only command." And (4188926877,
21:19:10Z): "The only real issue here is that pruning shouldn't infect every client
here. It might need to be UI-only until a view is saved, and then the view definition
goes to the model."
- Q29 (4188887721, 21:15:14Z): "Prune should only be written if the user explicitly
unpruned the view."
- The cursor: "The cursor should never jump: it should stay where the user left it as much
as possible." (Q12, 4171501445, 2026-10-03T03:21:05Z). And "As far as possible, updates
should shift the grid being viewed to keep the cursor in the same visual place on the
screen." (Q34, 4188946818, 2026-10-05T21:21:18Z).
**Answers ADR 008 records that this ADR's text had not caught up with:**
- Q1 (4171484686, 2026-10-03T03:14:34Z): "go with occupied". Clause 9.
- Q2 (4171486541, 03:15:19Z): "do b, but maybe with some easing into narrower in cases
where the column size has been stably smaller." And Q22 (4188309022,
2026-10-05T20:15:12Z): "hysteresis as recommended." Clauses 9 and 13.
- Q6 (4171493762, 2026-10-03T03:18:12Z): "maintained incrementally". Clause 9.
- Q13, Q14 and Q15: "the recommendation is fine.", three comments (4171504439,
4171505979 and 4171509012, 2026-10-03T03:22:16Z to 03:23:36Z), which ADR 008 records as
its answers to those three questions. Q13 sets the frame budget, and Q15 sends the
formulas the per-record map cannot express to the occupied contexts. Clauses 5a and 14.
- The store's memory (4171481179, 2026-10-03T03:13:14Z): "We have to be careful here to
minimize memory requirements of the store: offloading inactive regions to disk (maybe
mmap or similar), double-delta/gorilla/lz4 compression, etc." And Q17 (4188271710,
2026-10-05T20:11:02Z): "Let's defer this as recommended." Clause 14.
- Jobs (4171458281, 2026-10-03T03:04:52Z): "We also have to have a way to
invalidate/cancel in-flight jobs efficiently and probably a way to throttle them (a
priority queue or similar) to avoid pinning the CPU." Then (4171694278, 04:30:31Z):
"priority queue for work stealing is also important here." Clause 7.
**What this ADR reads from them.** These are the author's readings, and each one names
the clause it changes:
1. **Reads and writes go through `project` and `apply`.** A consumer reads the circuit's
outputs through `project`, and the circuit's inputs change only through `apply` of
model commands. So nothing here may depend on a host-held mirror of a client
(`Sessions`, `ClientSession`, `reduce_view` on the host), on the per-mirror diff
(`grid_updates`), or on resetting a client's cursor. Clauses 9 and 12.
2. **Aggregation stays on the model side.** No client runs an aggregate operator, so
network transparency concerns the circuit's outputs, and a live source's inputs. It
does not mean splitting operators across a wire. Clause 12.
3. **State is keyed by the request's shape, not by a view's name.** Axes are a model
concept, so operator state keyed by axes is model state, not UI state. But a `project`
request names its own axes, and an unsaved request may carry parameters no saved view
holds (ADR 008's flagged reading 1, its question 51). Clause 7, question 14.
4. **The record order belongs to the store.** The ordered list of records is the store's
(Q27). The circuit may consume it, and does not own it. Clause 9.
5. **The memory requirement covers this ADR too.** The store-memory requirement covers the
circuit's resident state as well, so the non-goal "spilling to disk" is withdrawn.
Clause 14, question 17.
6. **Five of ADR 008's questions are answered.** Those are questions 1, 2, 6, 13 and 15,
and clauses 5a, 9, 13 and 14 now follow the answers.
## Context
@@ -133,9 +261,9 @@ whose inputs moved, over aggregates that are maintained rather than re-folded. T
machinery is a new, problem-agnostic workspace crate implementing the DBSP core.
improvise-core defines a trait for the derived state its evaluator reads. Today's
evaluator is one implementation of that trait, and a circuit built on the new crate is
the other. Both stay in the tree, and a differential test holds them equal.** Eleven
the other. Both stay in the tree, and a differential test holds them equal.** Fourteen
clauses, plus clause 5a. It is numbered 5a so that no clause number or question number
cited elsewhere moves.
cited elsewhere moves. The 2026-10-05 revision added clauses 12 to 14.
### 1. The crate: `zset` (name is question 1)
@@ -179,8 +307,12 @@ Like `gp-tree`, it knows nothing about improvise. It has **no dependencies** bey
improvise's circular semantics are not a least fixed point (clause 6).
- **Timestamps beyond one clock, lattices, and out-of-order input.** improvise applies
batches in one total order (ADR 001, "There is already a total order").
- **Persistence, spilling to disk, and threads.** State lives in memory, and a browser
worker has one thread.
- **Spilling to disk, compression, and threads, in the first version.** State lives in
memory, and a browser worker has one thread. **This was a non-goal and is now only a
deferral.** The user's store-memory requirement (ADR 008 clause 12) covers this state
too, and clause 14 says what the crate must not rule out. A host with threads may run
independent circuits on a pool (ADR 008 question 23's answer). One circuit's `step`
stays single-threaded.
- **A query language or planner.** Circuits are wired by hand in the seam crate
(clause 2).
@@ -244,7 +376,9 @@ same-value remove-then-set would cancel, defeating 5a.
An overwrite with an equal value, `−(k, v, s) + (k, v, s)`, consolidates to the empty
Z-set. A remove-then-set of an equal value does not: `s ≠ s′`, which is the point. The circuit steps once
per model effect, as ADR 008 clause 3 requires: the frame observation reads occupancy
inside the effect loop (`app.rs:1073-1084`).
inside the effect loop (`app.rs:1073-1084`). ADR 008 clause 8 revises that observation
under the view/host split. The per-effect step stays, because each step is what clause
12 stamps.
**A delta key's identity is bit identity, the same identity the record tier uses
today.** Two elements have the same `(key, value)` part exactly when their keys are
@@ -301,7 +435,7 @@ new is observed. Its cost is O(formulas), where `document_content` is O(cells) p
**The group** is the cell key with the view's `none_cats` removed. That is the partial
key `evaluate_unfolded` folds over. There is one aggregation circuit per distinct
`none_cats` list, shared by every view that has that list. That replaces
`none_cats` list, shared by every projection shape that has that list (clause 7). That replaces
`rederived_for`'s model clone (`project.rs:195-210`). **The aggregator** for a group
comes from its `_Measure` coordinate through `measure_agg`, as `agg_for` does
(`types.rs:717-722`).
@@ -541,9 +675,16 @@ is one evaluator.
**The boundary.** The map is sound only when `expr` reads the record's own cells, so
that its value at a record changes only when that record changes. An inner expression
that reads another record, holds a nested aggregate, or refers to a formula whose value
depends on other records, is not a function of one record. That formula stays on the
eager path, and the differential test pins the boundary. ADR 008 clause 5 adopts the
same boundary. Two regressions pin the record stage against eager:
depends on other records, is not a function of one record. **Revised on 2026-10-05:**
such a formula does **not** stay on today's eager path, because that path is
`Expansion::members`, the cross product (ADR 008 clause 5, a finding of Copilot's on
#308). It takes ADR 008's **store-driven fallback**, which evaluates the expression only
over the **occupied contexts**, the contexts some record projects to. The user accepted
that recommendation, as ADR 008's question 15 ("the recommendation is fine."). It differs
from today's eager answer wherever the expression is non-empty at a context that holds no
record, and ADR 008 carves those cases out of its differential test by name. Whether this
ADR's `Eager` adopts the same rule, so that its own differential test needs no carve-out,
is question 18. Two regressions pin the record stage against eager:
- one changes **one operand** of a two-operand expression (`Price` alone, under
`SUM(Revenue * Price)`);
- one **deletes a record's leading cell**, including a cell the expression never reads
@@ -555,7 +696,10 @@ another category (`[CustomCat]`), the context keeps a `_Measure` coordinate. A r
whose key `record_of` takes without `_Measure`, then belongs to the context only through
its cell under that measure, and eager orders it by *that* cell's position, not by the
record's first live cell. Until 5a defines that projection and its order key, such
formulas stay on the eager path, and the differential test pins that they do.
formulas stay on the eager path, and the differential test pins that they do. **That
eager path is `Expansion::members`, so while it lasts it blocks ADR 008's retirement of
`Expansion::members` (its migration step 6) for these formulas.** Sending them to the
occupied-contexts fallback above as well is question 18.
**What stays eager until step 5, and what is bit-equal before step 6.**
- **Before step 5** (formulas), every aggregate over an expression is evaluated as
@@ -610,13 +754,58 @@ post-loop `view_shape` diff that bumps `layout_version` (`app.rs:1102`, `:1116-1
That diff comes too late, and it cannot see most of these changes. ADR 008 clause 3
states the same rule.
- **Operator shape, per view.** The operator shape is `view_shape` (`app.rs:40-72`),
plus the group membership of every drawn item, collapsed or not, plus
`measure_agg`. `view_shape` itself is not widened, so what bumps `layout_version` is
unchanged. A view whose operator shape moved has its operators rebuilt from the
integrated base Z-set **before the frame observation that follows this effect**.
The `LayoutGeometry` frame arm (`app.rs:1062-1071`) reads occupancy right after the
effect that moved the shape, so the next effect's cell delta is too late.
- **Operator shape, per projection shape (revised on 2026-10-05).** Operator state is
keyed by the **operator shape of a `project` request**, not by the name of a view
(question 14). The shape is made of:
- the categories of each axis, by level;
- the None categories;
- the filter and page keys;
- the hidden items and collapsed groups the request carries;
- `measure_agg`.
A saved `.improv` view supplies these to a request that names it. An unsaved request
carries its own, for example an unsaved prune or hide (ADR 008's flagged reading 1, its
question 51). Requests with the same shape share one set of operators, whichever
clients sent them. The live shapes are a **cache**. A shape enters when a request names
it, and it is evicted once no recent request has named it (clause 12). For a saved
view the shape still includes what `view_shape` (`app.rs:40-72` at `57d385e1`) hashes,
plus the group membership of every drawn item and `measure_agg`. `view_shape` itself is
not widened, so what bumps `layout_version` is unchanged.
*As first written* (superseded): a view whose operator shape moved had its operators
rebuilt **before the frame observation that follows this effect**. That rule served the
per-view frame observation, which ADR 008 clause 8 revises under the split. The three
rules below replace it.
- **Item grain (ADR 008's first request; question 16).** Occupancy and the per-column
aggregates are keyed by the **items** of the axis categories, together with the filter
and page keys. They are never keyed by a position in a level sequence. So a collapse,
expand, hide or show of an **axis** category's items changes no operator. The visible
axis is a remap of the item-grain state. ADR 008 clause 3 measured that remap at 2.3 ms
for 100k occupied tuples, against 6 to 45 ms for a rebuild. Adding an item changes no
operator either, because a new item tuple is just a new key. Three things still
rebuild a shape:
- a change of which categories are its levels;
- a change of its filter or page keys, which includes hiding an item of a category that
is not on an axis;
- a change of `measure_agg`.
A collapsed group's value is a pooled fold over its members. With item grain it is
combined from the members' state when it is read. For `Sum` and `Count` that combine is
linear. For the integrated aggregates it costs O(members) per read, and whether that
fits the frame budget at a 1,000-member group is **not measured**.
- **A rebuild is a job (ADR 008's third request).** A rebuild costs O(cells), so it
cannot run on the hot path. ADR 008 clause 11 sets 8 ms for a normal UI operation and a
2 ms slice for a job, and the user accepted that recommendation (ADR 008 question 13).
So a rebuild is a resumable `step(deadline)` over an owned snapshot of its input. The
deltas that arrive meanwhile are buffered and added at install, which Z-set addition
makes exact. A circuit `step` on a cell delta is O(delta), and it is never sliced.
Several things are ADR 008 clause 11's, not this ADR's: what a pending shape shows, the
install as a `Model(LayoutGeometry)` effect, and the cancellation, throttling and
priority queue the user required. This ADR only makes its rebuilds steppable and
cancellable between slices.
- **A crossing count (ADR 008's second request).** Each step's occupancy output carries
the number of tuples whose count crossed zero. ADR 008's ordered index uses it to
choose between an insertion and a rebuild by sorting.
- **The formula set is an input, not a rebuild.** After each model effect the host
compares a fingerprint of the formula set, over (`raw`, `target`,
`target_category`), with its previous read. If it moved, the formula delta (−the old
@@ -628,18 +817,18 @@ states the same rule.
- **A document swap rebuilds every circuit** from the new store. `install_workbook`
bumps `document_generation`, and the same after-each-effect read sees it.
Re-reading the operator shape costs O(items) per view per model effect, and the formula
Re-reading the operator shape costs O(items) per live shape per model effect, and the formula
fingerprint O(formulas). Neither grows with the cells or the grid. A rebuild costs
O(cells), once per actual reshape. The same `view_shape` function is therefore read at
two moments: by the circuit after each effect, and by `layout_version` after the loop.
They answer different questions, and neither replaces the other.
**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** `AddItem`
moves the operator shape, so the view is rebuilt with East in its sequences. Only then
does `SetCell`'s delta arrive, at operators that already know East. So the in-loop
`CellData` frame observation reads occupancy for the new shape. The `LayoutGeometry`
arm (`app.rs:1062-1071`) reads the frame immediately after each geometry effect, and it
reads a rebuilt occupancy for the same reason.
**The worked case is gotcha 43's frame, `[AddItem(East), SetCell(East)]`.** *As first
written* (superseded by item grain): `AddItem` moved the operator shape, so the view was
rebuilt with East in its sequences before `SetCell`'s delta arrived. **Under item grain,
`AddItem` moves no operator.** `SetCell`'s delta inserts the item tuple `East` as a new
key, and only the visible axis, which is derived, gains a row. The case needs no
ordering rule at all.
**Recommended: a page selection is not a rebuild (question 7).** Under the recommended
answer, the group key includes the page categories, so `[` and `]` read another slice
@@ -886,37 +1075,61 @@ option, and its recommendation.
### 9. What ADR 008 gets from it
ADR 008 (revision 3, PR #308) assumes these, under the shared names below.
ADR 008 (revision 19 at `69f0a7c6`, PR #308) assumes these, under the shared names below.
**Revised on 2026-10-05 for the view/host split.** Every item below is a model-side fact.
A consumer reads it through `project` (clause 12), and none of it is addressed to a
host-held mirror of a client.
- **Occupancy** (ADR 008 clause 3). For each view, there is a count per row tuple and
per column tuple of cells that hold a value. Stored cells pass the view's filter, with
hidden items dropped, and a collapsed group's members map to the group's tuple.
Formula cells count by clause 5's stem rule, which is when today's `formula_cache`
holds a value: a top-level `WHERE` must match, or raw data must exist at the target.
The operators are rebuilt per clause 7, so a batch that reshapes and then writes feeds
its write to the new shape. Page coordinates key a slice, subject to question 7. A tuple
is on the axis iff its count is above zero. **This ADR does not settle ADR 008
question 1 (both cases: fold rows with a formula column, and `Exact`).** Today those
cells are values with no stored cell under them. The stem definition drops them, and
if the user rules otherwise, those views fall back to a non-occupancy path. Occupancy
is integer and order-free, so it does not wait for ADR 005.
- **The occupancy delta** (ADR 008 clause 8). `distinct` over occupancy emits a
non-empty delta exactly when a tuple enters or leaves an axis. A reframe is a
non-empty occupancy delta. The two-arm asymmetry of the frame observation (gotcha 24,
improvise-mcq4) is unchanged, but each arm costs O(delta) instead of a rebuilt grid
per view.
- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over
the pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`),
- **Occupancy** (ADR 008 clause 3). For each projection shape (clause 7), there is a count
per row tuple and per column tuple of cells that hold a value. The counts are kept at
**item grain**, and the visible axis is a remap of them.
- Stored cells pass the shape's filter.
- A hidden item of an axis category is masked only when the visible axis is derived.
- A collapsed group's members map to the group's tuple at read time.
- Formula cells count by clause 5's stem rule, which is when today's `formula_cache`
holds a value: a top-level `WHERE` must match, or raw data must exist at the target.
- Page coordinates key a slice, subject to question 7.
A tuple is on the axis exactly when its count is above zero. **ADR 008's question 1 is
answered: "go with occupied"** (#308, comment 4171484686). So the two cases where
today's pruning keeps a row with no stored cell under it are dropped on purpose: a fold
row with a formula column, and `Aggregation::Exact`. Clause 5's stem rule already drops
them, there is no "non-occupancy path" for them, and the differential test carves them
out by name. *As first written*, this item said this ADR "does not settle" that
question; that is superseded. Occupancy is integer and order-free, so it does not wait
for ADR 005.
- **The ordered record list** (ADR 008 clause 1, `Records`). The user: "the ordered list
of records is a model-side concept" (Q27). It is the store's order (gotcha 18,
`DataStore::live_records`), and this ADR does **not** maintain it as a per-shape
operator. Clause 5a's record stage keys records by `record_of` and orders them by the
store sequence, so it *consumes* the store's order and never defines a second one. How
a page and filter select from that list is ADR 008's question 55. It is not this ADR's
to answer, because this ADR keeps no per-view copy of the list. Whether records mode is
`project` over the virtual axes (ADR 008 question 54) changes nothing here: occupancy
over `_Index` and `_Dim` is the same operator as over any other axis.
- **The occupancy delta** (ADR 008 clause 8, which the split revises). `distinct` over
occupancy emits a non-empty delta exactly when a tuple enters or leaves the **visible**
axis (ADR 008 clause 3, "A write or removal on a collapsed or hidden axis"). *As first
written*, its consumer was the per-view frame observation and its reset of each client's
cursor (gotcha 24). Under the split the host holds no cursor. So the delta is a model
fact, published on clause 12's delivery stream to the consumers of that shape, and a
client uses it to shift its grid so that the cursor keeps its place (Q34). Whether the
host pushes it is question 13, which is ADR 008's question 52 seen from this side.
- **Column totals** (ADR 008 clause 5). A column total is the formula evaluated over the
pooled fold (`GridEval::column_total`, `crates/improvise-core/src/view/layout.rs:976-980`),
not a sum of row values. So the circuit keeps a per-(column tuple, raw measure)
aggregate, and the formula layer re-evaluates only the columns whose inputs moved,
through clause 2's trait. `Expansion::members` is replaced by clause 5. Totals that
hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer: the
exact accumulator (recommended) or ADR 005 (clause 8).
- **What ADR 008 therefore stops asking.** Its question 6 (totals: query and memoize,
or maintain) is struck there as decided by the user through this ADR. Its questions 1
and 2 stay open there.
- **What it does not get.** Column widths depend on formatted text, which is
display-only (gotcha 8), so they are not maintained here. They stay ADR 008 question 2.
aggregate at item grain, and the formula layer re-evaluates only the columns whose
inputs moved, through clause 2's trait. Clause 5 replaces `Expansion::members`. Totals
that hold a `Sum` or `Avg` wait for step 6, whose prerequisite is question 3's answer:
the exact accumulator (recommended) or ADR 005 (clause 8).
- **What ADR 008 no longer asks.** Its question 6 is answered ("maintained
incrementally"). Its questions 1 and 2 are answered there too: 1 is "go with occupied",
and 2 is (b), sticky per session, with hysteresis as the easing (its question 22). *As
first written*, this item said they "stay open there", which had gone stale.
- **Widths.** A width depends on formatted text, which is display-only (gotcha 8). Under
the user's answer to ADR 008's question 2, widths are taken over the rows realized and
held by the view layer. The user's question whether DBSP can maintain a maximum width is
answered in clause 13.
### 10. ADR 003's live sources are delta streams, when they can be
@@ -963,10 +1176,336 @@ read-only live source also sidesteps ADR 001's record tier, as ADR 003 already n
(`host.rs:162`, `app.rs:627-629`), and the integrated state is of the same order. So
on the network path a write is not O(delta): it is O(cells) for the snapshot today,
and roughly twice that with the circuit in it. The keyboard path (`apply_effects`
without a `Host`) takes no snapshot and keeps the O(delta) cost.
without a `Host`) takes no snapshot and keeps the O(delta) cost. Under the
store-memory requirement (clause 14), that second clone is also peak memory, which a
rewind would avoid. Question 9 stays open, and is now weighed against clause 14 as
well.
- **ADR 006, replay (improvise-ilvl).** A circuit is a deterministic function of the
delta sequence, so replaying commands reproduces its state.
### 12. Network transparency: one circuit, consumed in process or across a wire
**The requirement is the user's:** "The DBSP machinery needs to be implemented in a
network transparent way, but the details of that go to ADR 009" (chat). **Everything
else in this clause is this ADR's design, and the author's choices are marked as such.
None of it is measured.**
**What crosses a wire, and what does not** (the author's reading). Two boundaries can be
a wire.
- **The output boundary** carries the results of `project` and their changes: a window's
cells, its axis occupancy, its totals, and ADR 008 clause 4's width facts. In the TUI
the consumer is in the same process. In the browser it is the page, across `postMessage`
from the worker or a socket from `improvise-ws-server` (gotcha 45).
- **The input boundary of a live source** (clause 10). A source that reports its changes
delivers a cell delta, possibly from another machine. The circuit cannot tell that
delta from one `DataStore` emits.
Three things do not cross:
- **The write side carries no deltas.** A client changes the model only by `apply` of
model commands (the user, Q43), so traffic from a client to the host stays
`ModelCommand`s with their basis (ADR 001), as today.
- **The circuit's internal streams stay on the host.** The cell delta, the per-record
stage and the group aggregates never cross: aggregation is model-side (the user, Q38),
so no operator runs in a client.
- **No UI state crosses.** The host is told the request and nothing about a cursor, a
scroll or a key (Q32, Q33, Q37).
**What a projection request is to the circuit** (the author's answer to "is the result
keyed by request, by axes, or shared?"). A request has two parts.
- **Its operator shape** (clause 7) selects the circuit's state. The shape is shared by
every request, from any client, that has it.
- **Its window**, the top-left and the dimensions, selects a part of that state's output.
The window is never state.
So maintained results are keyed by **shape**, and the only per-request work is
restricting a shape's output to a window. That is O(visible) per request and needs
nothing kept between requests.
**Positions are cross-product positions.** The user answered ADR 008's question 53 in
chat: "Q53 is cross product: I'm still ambivalent about which side of the wire pruning
should happen on" (relayed on 2026-10-05; on no PR). So a request's top-left is a position
in the **cross product** of its axes, in each category's own item order. That position is
a bigint where it does not fit a machine word (ADR 008 questions 28 and 36: "in typed
contexts, we can represent bigints as strings in JSON"). A cross-product position names
the same tuple whatever is occupied, so **an occupancy change never moves a request's
window**. It changes only which positions inside the window hold data. **Which side of
the wire prunes is open** (the user is "still ambivalent"), and is question 19. The
circuit must serve both sides, and what it must expose in each case is the author's
analysis:
- **Pruning on the model side.** `project` returns occupied rows and columns. The circuit
must then answer two order-statistic queries over a shape's visible occupancy, from a
cross-product position:
- the next k occupied tuples at or after a position, so a pruned window can start where
the request points;
- the rank of a tuple among the occupied ones, and the occupied tuple of a given rank,
so a client can scroll a pruned axis by rows and show "row i of n".
That is ADR 008's ordered index over occupied tuples (its clause 3), the sorted `Vec`
first and the order-statistic B-tree when measured (its question 16), fed by this ADR's
occupancy delta and crossing count. Under this case a client sees only occupied rows,
and its scroll space is the pruned one, built on top of the cross-product offset.
- **Pruning on the client side.** `project` returns the cross-product window, empty
positions included, together with a **skip structure** the client can query: for each
axis, the occupied runs that intersect the window and the next occupied position past
each edge. The circuit then has to expose the same order-statistic queries, but as a
model fact a client can request, not as a re-indexing the model performs on its behalf.
The cost is that a sparse grid at 3 × 1,000 (10^9 positions, about 10^5 occupied in ADR
008's fixture) sends mostly empty windows unless the client asks for skips first.
Both cases need the same maintained structure. They differ only in who calls it, and in
whether a `project` result is already pruned. Prune would be a request parameter in the
first case (ADR 008's flagged reading 1) and a client policy in the second.
**"As absolute as possible", and what still moves a position.** The user, in chat: "But
the top left should be as absolute as possible" (relayed on 2026-10-05; on no PR). *The
rest of this paragraph is the author's analysis.* A cross-product position does not move
when occupancy changes, pruning included. **It does move when an item is inserted,
removed or reordered** in a category's own order. Positions are mixed-radix numbers over
the items of each level, so inserting item j at level L shifts every later item of L, and
changes the radix that every outer position is multiplied by. No position over a
changing item list is fully absolute. Three things follow:
- **Deltas are keyed by identity, and positions are attached when sent.** An output
element names its row and column by tuple identity: item ids under ADR 007, and names
until then. The host attaches the absolute position from the current item order when it
sends the element. A delta keyed by position alone would go stale with the next item
insert.
- **An item edit is sent as an axis change, not as moved rows.** "Item inserted at index
j of level L", or removed, or moved. A client re-derives every position it holds
arithmetically, at O(levels) per position, and its top-left can be re-anchored to the
same tuple. That is the cursor's identity rule from ADR 008 clause 11 applied to the
window. Resending every row whose position moved would cost O(rows after j).
- **The cost is rank and select over each category's items.** "The position of item x"
and "the item at position i" are ADR 008 clause 10's order-statistic structure (a `Vec`
first, a B-tree when measured, its question 16). A tuple's position is O(levels × log
items). The ordered occupancy index must compare tuples by **current** item position,
not by a position stored at insert time. With item grain (clause 7) its keys are item
identities, so an item insert re-orders nothing in it, and comparison goes through the
order-statistic lookup, at O(levels × log items) per comparison. That is **not
measured**. Question 20.
**A pruned answer may arrive as a stream.** The user, in chat: "After pruning we might
have to stream rows and columns to attempt to fill out the full grid" (relayed on
2026-10-05; on no PR). *The author's analysis:* under model-side pruning, `project` scans
forward from the absolute top-left over each axis's occupancy until it has the requested
number of occupied rows and columns, or reaches the end of the axis. It sends the rows
and columns it finds as it finds them. What the circuit must support:
- **Skipping empty space in O(log n + k).** "The next k occupied tuples at or after a
position" over the ordered occupancy index. It must never walk empty cross-product
space, which at 3 × 1,000 is mostly empty.
- **The absolute position of every row and column sent**, so that a client can place it,
and a later delta can name it, without counting.
- **Bounded work per slice.** The fill is a job under ADR 008 clause 11's runner when k
is large, and it can be resumed from the last absolute position it sent. So the only
state a fill keeps is `(request, fill front)`, which is "as stateless as possible" in
the user's words. A lost fill restarts from the client's last known position.
- **Changes during a fill are identity-keyed deltas, stamped by step** (T3). Suppose a
row becomes occupied between the top-left and the fill front after it was passed. The
subscriber receives it as an ordinary occupancy delta carrying its absolute position,
so its window now holds one row more than asked, and it drops or keeps the extra row by
client policy (Q42). A row that becomes occupied ahead of the front is simply found by
the fill. Each chunk of the fill is a delta to the window's Z-set, stamped like any
other. So a fill and a live update are one stream, which keeps the in-process and the
remote consumer the same (T6).
- **Rows and columns fill independently** (the author's, on ADR 008's question 59(c)).
This ADR's occupancy is kept **per axis**: a count per row tuple and a count per
column tuple (clause 9). That matches today's `prune_empty`, which drops a row only
when it is empty across the whole grid, and prunes the columns against the same
unpruned grid (`layout.rs:752-757` at `50e1314d`). So a row occupied only in a column that falls
outside the window is still an occupied row, and the row fill never waits on the
column fill. A joint rule (a row occupied *within the requested columns*) would need a
two-dimensional count per (row tuple, column tuple). That is O(occupied cells) more
state, and it would change today's pruning, so it is not proposed.
Whether a pruned response is streamed or returned whole, and what a client sees while a
fill is incomplete, is question 21.
**What network transparency requires** (T1 to T6, the author's):
- **T1. Results are plain data.** Every element on an output stream is a value the
protocol crate can serialize, with no `Rc`, no borrowed store key and no closure. Facts
at `50e1314d`: `CellValue` is `Serialize`, with numbers going through `wire_f64`
(`crates/improvise-core/src/model/cell.rs:86-118`). `CellKey` is **not**: it derives only
`Debug, Clone, PartialEq, Eq, Hash` (`cell.rs:10`). So an output element names a cell by
the request's row and column tuples, or by cross-product positions. Where a write may
follow, it also carries ADR 002's surrogate (`CellId`). It never names a cell by
`CellKey`.
- **T2. One identity on both sides of the wire.** A consumer across a wire must
consolidate exactly as an in-process one does. **On NaN they would not.** Clause 3's
delta identity is bit identity, under which two NaNs with different payloads differ. But
the wire spells every NaN `"nan"` (`wire_f64`, `cell.rs:86-114` at `50e1314d`), and the
projection compares values with `same_on_the_wire` (`crates/improvise-protocol/src/render.rs:173-180`
at `50e1314d`; `context/code/protocol.md` gives the reason). So an output delta
`−(c, NaN₁) + (c, NaN₂)` is non-empty in process and consolidates to nothing after a
round trip. **Proposed:** output streams are consolidated under the wire's identity
(every NaN is one value, and every other number is its bits) before they leave the host.
The record tier keeps bit identity, as it does today. Question 15.
- **T3. Every output change is stamped.** A delta carries the step that computed it, as
`(document_generation, step)`. The circuit steps once per model effect (clause 3), and
ADR 001 already puts batches in one total order. A consumer that has applied step n
drops anything at or below n, so a duplicate does no harm, and it treats a gap as a
loss, which a fresh snapshot answers. What is published is the **batch's** consolidated
delta, and only once the batch is accepted. A refused frame is restored (clause 11) and
publishes nothing.
- **T4. A snapshot, then deltas.** A subscription starts with the integrated result at
step n: the window as a Z-set of current values, each with weight +1. Deltas from step
n+1 follow. Because I∘D = id (clause 1), the snapshot plus the deltas equals the current
result. `GridCache::folded(&cache.seed()) == *cache` states the same property for
today's bootstrap compaction (gotcha 32).
- **T5. Order holds across streams, and coalescing happens within a stream.** A slow
consumer's pending deltas on one stream can be **summed** into one delta. Z-set
addition is commutative and associative, so the sum is exact, and the queue is bounded
by the size of the result, not by the number of steps. When the summed delta outgrows
the snapshot, a snapshot replaces it. That is the backpressure, and nothing is dropped
silently. **Across streams, order stays load-bearing**, as gotchas 33 and 43 require:
the occupancy change comes before the cells it places, and the cells come before the
basis that describes them. This does not break gotcha 42's "never coalesce". That rule
is about positional `GridUpdate`s, whose fold depends on order. A delta keyed by tuple
identity, or by a stable cross-product position, is not positional in that sense.
- **T6. The in-process consumer uses the same API.** The TUI calls `project` and receives
the same result and delta types as a remote client. Its transport is a function call.
That is what makes the requirement testable: run one edit sequence through two
consumers, one in process and one behind a serialize and deserialize round trip, and
require their integrated results to be equal under T2's identity. A TUI shortcut that
read operator state directly would be a second read path, which is the derivation-twice
defect (design-principles §5).
Weights on the output are ±1, because a result is a set of current values. A consolidated
weight that does reach past 2^53 follows the user's bigint-as-string rule (Q36).
**Where the wire form lives** (the author's). `zset` stays dependent on `std` alone (clause 1).
The wire types live in `improvise-protocol`, with the one envelope pair (gotcha 42). A
consolidated `ZSet` is converted to a sorted `Vec<(element, weight)>` at that boundary.
The alternative is a `serde` feature on `zset`. The author leans to the conversion,
because what crosses is the projected result, not the crate's own element type.
**How stateless the host can be.** The user: "as stateless as possible". These are the
options of question 13, and ADR 008's question 52 asks the push half of the same thing.
- **(a) Pure fetch.** The host keeps nothing per client. A client calls
`project(request)`. It learns its result may be stale from a change notice addressed to
no one: the accepted batch's step. It then refetches when its own policy says to (Q42).
Each refetch costs O(visible).
- **(b) Soft-state subscriptions** (the author's leaning). A request may ask to be kept
current. For each subscription the host keeps the request and the last step it sent,
and nothing else: no copy of the window, no cursor and no viewport. After each accepted
batch it reads the delta of each subscribed shape, restricts it to the request's window,
and sends it under T3 to T5. **The table is a cache.** If it is lost (a restart, an
eviction, a reconnect), a client resumes by resending its request and the last step it
applied, and the host answers with a snapshot. The host could instead keep a short ring
of past deltas per shape; the author's lean is no ring at first. So the user's "recent
projections + clients" are this table plus clause 7's shape cache, and **both can be
rebuilt from the requests**. A subscription costs O(request) in memory, not O(window).
- **(c) Today's push to a mirror per client.** That is `Sessions`, `ClientSession`, and
`grid_updates` with `Projection` and `GridUpdate` (`crates/improvise-command/src/ui/project.rs:771`,
`:822` and `:981` at `50e1314d`). It keeps each client's `ViewState` and grid on the host
and diffs against them. ADR 008 lists it as obsolete under the split, and this ADR does
not propose it.
A shape is kept while some subscription or recent request names it. Eviction drops its
operators, and the next request rebuilds them as a job (clause 7). How long "recent" lasts
is a cost the user has not set (question 14).
**What a client must hold to resume:** its request, the last step it applied, its
integrated window, and its basis (ADR 001). The host holds nothing that a client needs
back.
**The basis.** ADR 001's basis is published with the result it describes, and after it
(gotcha 43). The host builds it from the request's window (ADR 008, lens row 18), not from
a mirror. `published_basis` (`project.rs:585` at `50e1314d`) already takes the `GridCache`
it describes, not a cursor.
### 13. Incremental max width (the user's question on ADR 008)
The user asked, on ADR 008's question 2: "Maybe dbsp can give us incremental max width?"
**This clause is the author's analysis, and none of it is measured.** The user's decision
on widths stands as ADR 008 records it: (b), sticky per session, eased by hysteresis,
taken over the rows realized and held by the view layer (ADR 008 clause 4).
**DBSP can maintain a maximum, but not as a linear aggregate.** A maximum has no inverse.
When the widest value is retracted, the new maximum cannot be computed from the old
maximum and the retraction. So it belongs to clause 4's **integrated** class, beside `Min`
and `Max`: each group keeps its integrated multiset and reads off the last key. A *top-k*
(keeping only the k widest) holds less state, but it is wrong once the top k have all been
retracted, and then it needs a rescan. So it is not proposed.
**For widths the multiset is small.** The group is one column of a projection shape,
together with its number format, since a width is measured on formatted text. The values
are display widths, which are small integers. So the state is a histogram from width to
count, per column. It is O(distinct widths) per column, whatever the number of rows, and
an update costs O(log d), where d is the number of distinct widths.
**The cost is upstream, in what feeds the histogram.** Its input is the display width of
**every evaluated cell in the column**, under the request's number format. That needs
three things:
- **A delta of evaluated cells over the whole column**, not a window. Clause 4's group
aggregates and clause 5's formula re-evaluation already produce an output delta for
every unfolded cell that moves, with its old value as the retraction. So unfolded rows
need no new state. **Fold rows are not covered.** Under item grain (clause 7) a
collapsed group's value is combined when it is read, not maintained, so its width would
need the fold maintained per column at the visible grain.
- **Formatting every changed cell on the host**, on the write path: `format_f64` and a
display width for each changed cell. Each one is cheap, but it would be the first
display-side quantity the circuit maintains. Gotcha 8 forbids feeding a rounded value
back into evaluation, and a width sink feeds nothing back, so the gotcha holds. But
today the host is asked for widths only over the rows requested (ADR 008 clause 4), and
this would widen that to whole columns.
- **A rebuild whenever the format changes.** `number_format` is a parameter of the view
definition, and a rebuild costs O(occupied rows) per column, as a job.
**What it would buy, set against what the user decided.** A maintained whole-column width
is ADR 008's question 2 option (c), "size each column from its whole contents", which the
user did **not** choose. Under (b) a window's widths are an O(visible) computation over
cells the window already evaluates, so maintaining them incrementally adds nothing. So
the real question is whether the user wants (c) after all, now that its state is cheap. It
would remove the jitter at its source and make hysteresis unnecessary, at the upstream
cost above. That is question 12.
### 14. Resident state, and the store-memory requirement
The user, on ADR 008: "We have to be careful here to minimize memory requirements of the
store: offloading inactive regions to disk (maybe mmap or similar),
double-delta/gorilla/lz4 compression, etc." (#308, comment 4171481179). And then, on ADR
008's question 17: "Let's defer this as recommended." (4188271710). So the store's memory
gets its own ADR, once this ADR's operator state has been measured at ADR 008's fixture.
**This clause withdraws the earlier non-goal** ("Persistence, spilling to disk, and
threads", clause 1), and it says what this ADR must not rule out.
**What this ADR keeps resident, beside the store.** This is the author's inventory. The
orders come from the clauses and are not measured.
| state | clause | order |
|---|---|---|
| the integrated base that a rebuild reads | 7 | O(cells), if it is kept as a copy |
| clause 5a's per-record stage | 5a | O(cells of the records an aggregate over an expression reads), as written |
| multisets for `Min`, `Max` and text | 4 | O(cells) of the measures that need them |
| `ExactSum`, per summed group | 8 | 568 bytes per group |
| occupancy, per live shape, at item grain | 7, 9 | O(occupied tuples) per shape |
| the circuit's rollback snapshot | 11 | a second O(state) clone per mutating frame (question 9) |
| the subscription table | 12 | O(request) per subscription |
**Proposed by the author:**
- **The store is the integral, so it is not copied.** The integrated base a rebuild reads
is `DataStore` itself, which is I(cell delta) by definition. Clause 5a's per-record
stage keeps each record's cells only to compute the retraction `I(x)₋₁`. That equals
`I(x) − Δx`: the record's cells in the store after the step, minus the step's delta. So
the stage can read the store and keep no copy of the values. It still needs each cell's
store sequence (clause 5a), one integer per cell, which `DataStore` would keep beside
the value rather than in a second map. This is **not measured**, and the retraction of
the order key has to be checked against it (question 17).
- **Operator state sits behind one interface.** DBSP calls it a trace: look up, read a
range, and apply a delta. The first implementation keeps it in memory, and the
store-memory ADR can page or compress it without changing any operator.
- **Shapes are evicted** (clause 12). So per-shape state is bounded by the shapes in
recent use, not by every saved view.
- **What does not shrink.** The multisets and `ExactSum` are the price of exact `Min`,
`Max` and unanimity, and of an exact maintained `Sum` (clause 8). They are O(cells) and
O(groups), and the store-memory ADR has to count them.
**The frame budget.** ADR 008 clause 11 sets it, and the user accepted the recommendation
(ADR 008 question 13): a normal UI operation fits in 8 ms or becomes a job, run in 2 ms
slices. A circuit step is O(delta) and runs on the hot path. A rebuild is O(cells) and is
a job (clause 7). This ADR has measured neither at ADR 008's fixture yet, and migration
step 4 must.
## Alternatives considered
**Feldera's `dbsp` crate.** What was verified on 2026-09-24:
@@ -1038,6 +1577,12 @@ displayed path until the differential test holds.
accumulator in both backends (recommended), or after ADR 005. A column total is a formula over the pooled fold, so it needs step 5's
re-evaluation. **ADR 008 clause 5 (column totals) depends on this step.**
7. **Live sources as delta streams**, when ADR 003's source trait exists.
8. **The delivery stream** (clause 12), on the answers to questions 13, 15, 19, 20 and
21. It brings the output types in `improvise-protocol`, the step stamp,
snapshot-then-delta, the order-statistic queries over occupancy, and the test with two
consumers (T6). It depends on step 4 for occupancy. It replaces the per-mirror diff
only when ADR 008's view/host split lands. Step 4 also measures, at ADR 008's fixture,
the cost of a step and of a rebuild (clause 14).
**Tickets.** improvise-felo.4 is this ADR. The steps above are not yet filed, which is
question 10. improvise-swre stays independent and is useful under either backend.
@@ -1055,6 +1600,9 @@ longer a prerequisite of step 6.
- ADR 008 gets occupancy, the occupancy delta and column totals without a second
evaluator.
- Live sources that can report changes plug in without touching the evaluator.
- One delivery path serves the TUI in process and every remote client (clause 12), and
the host keeps no per-client UI state: only a subscription table that can be rebuilt
from the requests.
**Bad.**
@@ -1072,25 +1620,33 @@ longer a prerequisite of step 6.
- A mutating frame from a client already clones the workbook for rollback
(`host.rs:162`). Snapshotting the circuit too (question 9) adds a second O(cells)
clone per frame. Incremental evaluation does not make the network path O(delta).
- The operator shape is re-read after every model effect: O(items) per view, a cost the
- The operator shape is re-read after every model effect: O(items) per live shape, a cost the
post-loop `view_shape` diff paid once per batch.
- The browser bundle grows by two small crates.
- Per-shape state lives only as long as requests name it. A shape that comes back after
eviction is rebuilt as a job, and the client waits through its pending state.
- The output identity (every NaN is one value) differs from the record tier's (bit
identity), as T2 explains. Two identities need stating everywhere a delta crosses.
- A position over a changing item list is never fully absolute. An item insert moves
every later position, and is sent as an axis change (clause 12).
## Open questions
Each question has a recommendation. The user's answers will be folded in under "What
the user decided", as ADRs 004 and 007 did.
the user decided", as ADRs 004 and 007 did. **Status on 2026-10-05: the user has answered
none of questions 1 to 11 on #309.** Questions 12 to 21 were added by the 2026-10-05
revision, and none is renumbered.
1. **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix?
1. *[Open.]* **The crate's name.** `zset`, `dbsp-core`, or a name with the `improvise-` prefix?
*Recommendation:* `zset`. It is problem-agnostic, like `gp-tree`, so it should not
carry improvise's name. `dbsp-core` invites confusion with Feldera's `dbsp`.
2. **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on
2. *[Open.]* **Where `Incremental` lives.** In a seam crate (`improvise-incremental`, depending on
`zset` and `improvise-core`), or inside improvise-core?
*Recommendation:* a seam crate. improvise-core then never depends on `zset`, and
`Eager` stays buildable and testable alone. It is the `gp-tree`/`improvise-gp` shape.
3. **Floating point.** What makes a maintained `Sum` and `Avg` safe to display?
3. *[Open.]* **Floating point.** What makes a maintained `Sum` and `Avg` safe to display?
- (a) ADR 005 (exact decimals) as a prerequisite, with the eager fold kept for sums
until then.
- (b) A compensated f64 sum, re-folded on detected cancellation, with a tolerance in
@@ -1117,35 +1673,35 @@ the user decided", as ADRs 004 and 007 did.
that `eval_agg` uses (`types.rs:79`) gives `−0.0`.
- `Avg`'s one rounding must be stated in step 6.
4. **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported
4. *[Open.]* **The sixteen-pass bound.** An acyclic chain needing all sixteen passes is reported
`circular` today (core.md §3). Should `Incremental` reproduce that?
*Recommendation:* reproduce it, so the two backends are comparable. If the bound is to
change, change it in both backends at once, as its own decision.
5. **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by
5. *[Open.]* **Formula cycles.** Should formulas in a cycle of the formula graph be evaluated by
today's bounded fixed point, restricted to their component, rather than by any
operator in the crate?
*Recommendation:* yes. It keeps `Error("circular")` exactly as it is, including the
cells a guarded cycle leaves convergent. The crate needs no recursion operator.
6. **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay?
6. *[Open.]* **Keep `Eager` permanently?** Once `Incremental` is the default, should `Eager` stay?
*Recommendation:* yes, permanently. It is the differential test's oracle and the
backend for ADR 003's query-only sources. Deleting it would delete the test.
7. **Page selection as a slice.** Should the group key include the page categories, so
7. *[Open. ADR 008's flagged reading 1 makes the page selection a parameter of the request, which is this question's slice seen from the request side.]* **Page selection as a slice.** Should the group key include the page categories, so
switching page reads existing state, rather than rebuilding the view's operators per
page change?
*Recommendation:* the slice. The state stays bounded by the stored cells, and the
held-key path (`[`/`]`) stays O(visible). Clause 7 is written to this
recommendation, and ADR 008 clause 3 says it is "subject to ADR 009 question 7".
8. **Live sources.** Should `Incremental` serve only sources that can emit change deltas,
8. *[Open.]* **Live sources.** Should `Incremental` serve only sources that can emit change deltas,
with query-only sources kept on `Eager` and ADR 003's pushdown?
*Recommendation:* yes, chosen per document. Integrated state is proportional to the
source, which contradicts ADR 003's larger-than-memory goal for exactly the sources
that cannot report their changes anyway.
9. **Rollback.** For ADR 001 clause 4, should the backend's state join the `App`
9. *[Open. Clause 14 now weighs the second clone against the store-memory requirement.]* **Rollback.** For ADR 001 clause 4, should the backend's state join the `App`
snapshot that `Host::mutating_frame` already takes?
*Recommendation:* yes, snapshot, with no later switch to the inverse delta. Applying
`−d` restores integrated values but not the history `delay` and `differentiate`
@@ -1154,13 +1710,13 @@ the user decided", as ADRs 004 and 007 did.
today. A true rewind would need its own design and an observational-equivalence test
over every stateful operator.
10. **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps
10. *[Open.]* **Ticket structure.** Should improvise-felo.4 become the epic for clause-order steps
1–7, with one child bead per step, and ADR 008's dependent steps linked as blocked by
steps 4, 5 and 6?
*Recommendation:* yes. The user's decision is recorded on felo.4, and the steps are
how it is met.
11. **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8).
11. *[Open, and affected by question 16. Under item grain a reorder of an axis category's items rebuilds nothing, so the recommendation's premise (key by position and rely on `view_shape`'s rebuild) no longer holds. Keying by item identity and reading the order at read time, which clause 12 already needs (question 20), becomes the natural form. The recommendation is left as written until the user answers both.]* **Ordered aggregates' key** (clause 4, for improvise-felo.7 and improvise-felo.8).
Should `LAST`/`FIRST` key their per-group state by the ordering category's item
position, with that category's item order added to the operator shape so that a
reorder rebuilds them? Or should they key by stable item identity (ADR 007), and
@@ -1174,3 +1730,89 @@ the user decided", as ADRs 004 and 007 did.
no widening at all, and ADR 008's text and cross-references hold unchanged. If the
user wants hidden items ordered too, the `measure_agg` part must also cover their
positions.
12. *[New.]* **Incremental max width** (clause 13; the user's question on ADR 008).
Should the host maintain each column's whole-column maximum display width? That is
ADR 008's question 2 option (c), and its state would now be a small histogram per
column. *Recommendation (the author's):* no, not now. The user decided on (b) with
hysteresis, and under (b) a window's widths cost O(visible) and need no maintenance.
The maintained form also needs fold values maintained per column, and formatting on
the write path. Revisit it if a measured workload shows jitter that hysteresis does
not cure.
13. *[New.]* **Delivery: fetch, soft-state subscriptions, or push to a mirror per
client** (clause 12; ADR 008's question 52 asks the push half). *Recommendation (the
author's):* (b), soft-state subscriptions that can be rebuilt from the request and the
last step, with no copy of a client's window on the host. When its table is lost, (b)
falls back to (a). (c) is obsolete under ADR 008's split. One of the two ADRs should
hold this question, and the other should point to it.
14. *[New.]* **What keys operator state, and how long a shape lives** (clause 7). Is it
keyed per projection shape, cached and evicted (the author's)? Or per saved `.improv`
view only, so that a request whose shape is unsaved (an unsaved prune or hide, ADR
008's question 51) is served without incremental state? *Recommendation:* per shape,
cached, and evicted once no request has named it for a while. The time is an
implementation number. An unsaved shape is then as incremental as a saved one, which
keeps "pruning shouldn't infect every client" from costing an eager path.
15. *[New.]* **The output identity** (clause 12, T2). Should output streams consolidate
under the wire's identity, where every NaN is one value, while the record tier keeps
bit identity? *Recommendation:* yes. Otherwise a remote consumer and an in-process
one disagree when only a NaN's payload changes.
16. *[New.]* **ADR 008's three requests** (ADR 008, "What revision 4 asks of ADR 009"):
occupancy and the per-column aggregates kept at item grain, a crossing count per
step, and rebuilds as steppable jobs. *Recommendation:* adopt all three, as clause 7
is now written. This changes clause 7's list of what rebuilds, removes the rebuild
from gotcha 43's worked case, and undercuts question 11's premise.
17. *[New.]* **Resident state** (clause 14). Should this ADR withdraw "spilling to disk"
as a non-goal, keep no second copy of the store (the store is the integral), and put
operator state behind one interface that the store-memory ADR can page?
*Recommendation:* yes. The measurement at ADR 008's fixture comes first, which is the
order the user's answer to ADR 008's question 17 already sets.
18. *[New.]* **Formulas the per-record map cannot express** (clause 5a). ADR 008's
question 15 sends them to a fallback over the occupied contexts, never to
`Expansion::members`; the user's answer was "the recommendation is fine.". Two parts:
- Does this ADR's `Eager` adopt the same rule, so that `Eager` and `Incremental`
agree and this ADR's differential test needs no carve-out?
- Do the formulas that target a category other than `_Measure` take the same
fallback, so that they do not block ADR 008's retirement of `Expansion::members`?
*Recommendation:* yes to both. `Eager`'s answers then change in exactly ADR 008's
question 1 and question 15 cases, as a stated change of behaviour.
19. *[New. The user is "still ambivalent".]* **Which side of the wire prunes** (clause 12;
the user's chat answer to ADR 008's question 53; ADR 008's question 57 asks the same,
and one ADR should hold it). On the model side, `project`
returns occupied rows and columns, and the circuit answers next-occupied and
rank/select queries for it. On the client side, `project` returns the cross-product
window together with a skip structure, and the client asks for the queries itself.
*Recommendation (the author's):* the model side, with prune as a request parameter.
A sparse grid at 3 × 1,000 then never ships empty space, and a client stays thin.
Either way, the circuit maintains the same structure (clause 12).
20. *[New.]* **Absolute positions under item edits** (clause 12; the user: "the top left
should be as absolute as possible"; ADR 008's question 58 asks the same from the
request side). A cross-product position stays fixed when
occupancy changes, but it moves when an item is inserted, removed or reordered. Should
deltas be keyed by tuple identity, with absolute positions attached when they are
sent, and should an item edit be sent as an axis change, so that clients re-derive
their positions arithmetically? *Recommendation:* yes. It costs rank and select over
each category's items (ADR 008 clause 10), and comparisons in the occupancy index at
O(levels × log items) each, which is not measured. It is compatible with ADR 008's
question 58 recommendation, which makes a tuple of item ids the primary form of the
top-left. Whether hidden items belong to the absolute space is ADR 008's question 58.
Under item grain (clause 7) this ADR serves either answer, because hiding is a remap.
21. *[New.]* **A pruned answer as a stream** (clause 12; the user: "After pruning we
might have to stream rows and columns to attempt to fill out the full grid"). Should
`project` under pruning fill forward from the absolute top-left, as a sliced job that
can be resumed from its fill front, and send each found row and column as a stamped
delta? Should a row that becomes occupied behind the front arrive as an ordinary
occupancy delta, and leave the window one row larger than asked, for the client's
policy to trim? *Recommendation:* yes. It keeps the fill and live updates as one
stream, and the host's state per fill is `(request, fill front)`. ADR 008's question 59
asks the same, and asks in its part (c) whether rows or columns fill first. Clause 12's
answer is that the two fill independently (the author's).