improvise/context/adr/009-incremental-evaluation.md

ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait

Status: proposed. · Ticket: improvise-felo.4 (the record of this ADR; implementation goes to its own epic) · Relates to: ADR 003, ADR 005, ADR 006, ADR 007, ADR 008

Problem

Every change recomputes every formula cell (Model::recompute_formulas, called by rebuild_layout after each batch), and raw aggregates are re-folded on every read. That does not scale to a live source (ADR 003) streaming changes into a large table, nor to ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without building the grid. A change should cost in proportion to what it touches.

Decision

Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A cell write becomes a delta, aggregates are maintained, and only the formula cells whose inputs moved are re-evaluated by the existing evaluator. The incremental results must equal today's eager results exactly.

1. The zset crate

A problem-agnostic workspace crate with no dependencies beyond std (so it is wasm32-clean and serde-free). It provides:

• ZSet<K>, a map from K: Ord to a non-zero i64 weight, which is an abelian group;
• a synchronous, single-threaded Circuit per process, with delay, integrate, differentiate, map, filter, flat_map, index_by, the bilinear delta join join_indexed, distinct and aggregate;
• resumable rebuilds: building an operator from a snapshot is a step(deadline), so it can run as a sliced job;
• operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it.

There are no nested streams, no recursion operator, no timestamps beyond one clock, and no query language.

2. The plug

improvise-core gains a trait, Derived, for the two things the evaluator reads: the formula result at a key, and the raw fold of the cells that match a partial key. The incremental backend implements it in a seam crate, improvise-incremental, which depends on zset and improvise-core. improvise-core never depends on zset.

The circuit decides which formula cells to re-evaluate. The one existing evaluator computes each value, so formulas are never compiled into operators.

Eager is deprecated. Today's code moves behind the trait as Eager, unchanged. It serves as the oracle of a differential property test, which applies one random edit sequence to both backends and compares every drawn cell, total and occupancy count. It is removed once its remaining uses are replaced (open question 3).

3. The cell delta

DataStore::set and DataStore::remove emit a ZSet<(CellKey, CellValue, Seq)>:

• +(k, v, s) for a new cell;
• −(k, v₀, s) + (k, v₁, s) for an overwrite;
• −(k, v₀, s) for a removal;
• a remove-then-set gets a fresh s′.

Seq is a per-cell sequence assigned at insertion, in IndexMap order, and kept by an overwrite. Value identity is bit identity (f64::to_bits), so an equal-value overwrite consolidates to nothing. CellKey gains Ord, which is its canonical order: the natural order of ids (ADR 007 amendment 1, clause 4). A store reorder, such as entering records mode, reassigns Seq and rebuilds clause 5's per-record stage. The circuit steps once per model effect.

The record tier and the dirty flag read this delta instead of scanning the store. The dirty flag fires on a non-empty (key, value) delta, or when a fingerprint of the formula set moves, because formula edits move no cell.

Under ADR 007 and its amendment, CellKey pairs are (CategoryId, ItemHandle). So a rename is a display write and emits no delta. The formula-set fingerprint must use the id form too, or a rename reads as a formula change.

4. Aggregates

A group is the cell key minus the None categories. One aggregation circuit serves every shape key with the same None list, which removes the per-session model clone.

aggregate	state per group	per delta element
Count	count of values, count of numbers	O(1)
Sum, Avg	an exact accumulator (clause 8), and the count	O(1)
Min, Max, text unanimity	a multiset of values	O(log d)

A group is read with eager's rules, including NaN and signed zero. LAST/FIRST and semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the ordering category's order. How that order is keyed is for those tickets.

5. Formulas

Stems are distinct over the delta, keyed by "cell key minus the target and None categories". A formula counts at a stem where its top-level WHERE matches or raw data exists at the target.

Inputs. A formula cell is re-evaluated when one of its inputs moved:

• a Ref(R): the group at the stem with R's category set to R;
• an Agg: the group at the stem;
• an Agg whose inner WHERE names Cat = Item: the group at the stem with Cat set to Item. This is the one join: stems indexed by "stem minus Cat".

The affected pairs are evaluated in the formula graph's topological order.

An aggregate over an expression (SUM(Revenue * Price)) is a pipeline:

1. A per-record stage retracts and reinserts each touched record. It reads the record's previous cells as the store minus the delta, so it keeps no copy of the values.
2. A map produces (context, order key, expr). The order key is (context rank, minimum Seq of the record's live cells), which is eager's concatenation order.
3. The group aggregate, with an error channel that reads the first Err in order.

No cross product is built. A formula whose expression reads other records is evaluated over the occupied contexts only, and Eager uses the same enumeration. The answer is unchanged, because a context with no records contributes nothing. One exception: an inner bare ref that names no item errors at every context, so for it the ref is checked before enumerating. This replaces Expansion::members.

6. Circular references

DBSP's least fixed point is not improvise's circularity: X = X + 1 has none, and the answer must be Error("circular"). The first version finds the formula graph's strongly connected components. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced by a DBSP-native treatment (open question 2).

7. Operator state follows the request's shape key

A projection request's shape key selects operator state, shared by every request with the same key, from any client. The key is:

• the categories on each axis, by level;
• the None categories;
• the filter keys: each filter category's item, and the hidden items of categories on None, Page or Filter;
• the page categories, so each page is a maintained slice and paging rebuilds nothing;
• measure_agg.

Everything else in a request is applied when output is read: hidden and collapsed items on axis categories, item order, the page selection and prune. Occupancy and column aggregates are keyed by axis items, so a collapse, hide, show, reorder or added item changes no operator. The visible axis is a remap.

• A new key builds its operators as a sliced job under ADR 008's job runner. Deltas arriving during the build are buffered and added at install.
• A formula edit is a formula delta fed to the circuit.
• A document swap rebuilds everything.
• Saving a view definition touches no operator.
• Live keys are a cache. A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9).

Column totals and hidden rows. A total pools only the drawn rows, so a hidden row leaves it. The shape key is shared and a hide is one client's, so totals are kept over all rows. A read subtracts the request's H hidden cells as a temporary overlay:

• Count subtracts;
• Min and Max skip keys whose whole count is hidden;
• text counts the distinct keys left;
• Sum subtracts exactly through the accumulator.

That is O(H log d) per read for every aggregate, with no per-client state. Aggregate-over-expression totals index their entries by row tuple for the same subtraction.

8. Exact sums

An f64 sum maintained by +v and −v drifts. Measured by crates/improvise-core/examples/retraction_drift.rs, at 18,000 cells after a million cycles, it is 1.7e-9 from eager, past FLOAT_EQ_EPSILON, so = could answer differently on the two paths. So every Sum and Avg uses ExactSum, in both backends:

• a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which makes it an abelian group that rounds once, on read;
• measured bit-equal to a fresh sum after a million cycles, and exact against rational sums on 20,000 cases;
• 568 bytes per group, and about 1.3× the cost of an f64 add per update.

It lives in ADR 005's numtower crate. User-visible changes: displayed sums change in their last bits, because they become correctly rounded, and a sum of -0.0 reads +0.0.

9. Delivery

Every edge may cross a network. An edge, meaning the stream from one operator to the next, may be in-process or remote, and a path may cross up to two network hops: browser ⇄ gateway ⇄ host, or host ⇄ worker ⇄ client. No edge assumes a function call or shared memory. An operator that sits at a wire boundary is just one implementation of an edge, an edge adapter. So every edge has the same semantics:

• A wire form. Every element type that flows on an edge has a serializable form, defined in improvise-protocol. The adapter converts to and from it, and zset stays free of serde. CellKey is not Serialize, so on the wire a cell is named by its tuple of ids, its position or an ADR 002 surrogate.

• The wire form is lossless. Nothing is lost in transport, numeric precision above all:

  • a number never crosses as a JSON or JS double. A numtower Num crosses as its canonical text (ADR 005). An f64 crosses as its exact bits, which keeps -0.0 and NaN payloads. That conflicts with today's wire_f64 and with numtower's nan token, which both drop the payload, so the edge carries bits for every non-finite value;
  • integers wider than 2⁵³, including positions and counts, cross as strings;
  • text crosses byte-exact, with no normalisation;
  • the delta is the unit that crosses, never ExactSum's limbs. A receiver that needs a sum gets the host's exact read or rebuilds it from the exact deltas.

  The stamp rule detects lost messages, but nothing would detect a lossy encoding. So every element type has a round-trip property test, where encoding then decoding is the identity for every value, including -0.0, NaN payloads, 28-digit decimals and i128-wide fractions.

• Stamps compose end to end. The host stamps each published batch (below), and an intermediate edge forwards the stamp unchanged. So a loss anywhere on the path shows up downstream as a gap.

• One clock per process. Each process runs its own synchronous sub-circuit. Across a hop, the stamped batches are the clock.

• Ordering and backpressure. An edge delivers in order. A slow edge coalesces exactly: deltas are summed, and notices merged. A merged notice carries the range of publications it covers, (from, to], so a downstream side sees no gap where nothing was lost.

• No long-lived stream assumed. An edge may be carried as a series of requests and responses, for example several HTTP requests on one connection, as well as a socket. Every request can be retried, and a resync from a publication stamp is idempotent. A client asks for "notices since n", and the host answers from one shared, bounded log of published notices: a single log, not one per edge or client. A request older than the log's start is answered with a correction (below).

• Failure is local to a hop. A hop may be a socket or a series of requests, and it is alive while its next message or request succeeds. Each hop keeps alive in both directions. A side that hears nothing within the timeout, or whose request fails, treats the hop as dropped. The downstream side then asks for a correction from its watermark.

• Correction is Z-set arithmetic. A correction is target − held, where target is the current state for the client's window and held is what the client holds. Applying it with ordinary Z-set addition gives held + (target − held) = target, so the group structure of ZSet provides it, with no separate machinery. The client's position is a watermark: the publication number up to which it has applied the stream, so everything at or before it is applied. There is one publication sequence, so a watermark is one number, held per shape key the client reads. The host and any gateway keep no watermarks: a client sends its own with each request.

  • where the shared log covers the watermark, the host sends the sum of the logged deltas after it;
  • otherwise the host sends target, and the client, which knows held, applies target − held;
  • a full refetch is target − ∅.

  A correction is labelled with the publication range it covers, so it is applied only at the watermark it starts from, which makes it idempotent and resumable. It uses the same lossless delta form as a notice, so a client, a gateway or a worker applies it with the same code.

Aggregation stays on the model side (ADR 008), so the operators that aggregate run on the model side, which may itself span a hop. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger relies on, is a correction trigger below, followed from a displayed cell's key back to the model.

A request is a shape key plus a window, given as an absolute cross-product top-left and dimensions. The window is never state, and a request costs O(visible).

• Positions are bigints where they exceed 2⁵³, sent as JSON strings.
• Output is keyed by tuple identity, with the position attached when sent.
• An item insert, removal or reorder is sent as an axis change, and each client re-derives its positions.

Pruning support. Pruning is window-relative: a row or column empty within the displayed values is hidden. This replaces today's whole-grid prune_empty. The model answers one restricted successor query: "the next k rows after position p with a value in a column of S", and the same for columns. A per-column sorted index of occupied rows answers it in O(|S| log n + k|S|), holding O(occupied cells). Whether pruning runs on the client and which parameters travel are ADR 008's questions, and this design supports either answer. If the client prunes, it prunes its window, fills forward with that query, and, when scrolling up or left, finds a new top-left by galloping backwards in doubling jumps (about 14 to 28 probes at density 10⁻⁴).

The change notice. After each accepted batch the host emits one notice to every client, addressed to none. It carries:

• the stamp (host instance, document_generation, publication). The publication number increments once per published batch, and the host instance changes on restart;
• axis changes, display relabels as (id, display), and a flag when the formula set or measure_agg moved. A shared display's #2 suffix is never sent: the client computes it from the displays in scope;
• the shape keys evicted since the last notice;
• for each live shape key, the moved output keys (cells, formula cells and totals), as the circuit computes them. Moved records alone would miss formulas with cross-stem inputs.

A notice whose key lists exceed a fixed size carries "correct" instead.

What a client does. It intersects the moved keys with its window. A fold row has no maintained key, so it maps member keys through its own remap first. It then fetches what moved. It asks for a correction when it sees a publication gap, a new document_generation or host instance, its shape in the eviction list, a notice it cannot read, or a dropped hop. A slow client may merge pending notices: the union of keys, the covered range of publications, and axis changes kept in order. Because the wire is lossless, outputs keep the circuit's bit identity end to end.

What this replaces. Today's per-session mirrors and pushed diffs (Sessions, ClientSession, grid_updates, Projection, Dirty). The host keeps no per-client state, only the shared notice log. GridExtent is derived by the client, and in records mode the record count is fetched.

Edge adapters at the client. Toward the circuit, a client's adapter is a sink for results and notices and a source of requests. Toward the client it is the mirror image. Each hop has an adapter at each end, and each end keeps only that hop's state: its liveness timer, and at the client end the watermark. A gateway hop (browser ⇄ gateway ⇄ host) may serve many clients behind one upstream connection, so the host sends one notice per gateway rather than one per client. In process, an edge is a function call.

10. Text column widths

The host maintains each text column's maximum display width as a histogram from width to count, per (shape key, column), fed by the delta of the column's text cells. Hidden rows come out through clause 7's overlay. Numeric widths stay with ADR 008's view-layer rule.

11. Rollback and memory

• Rollback. The circuit's state joins the snapshot Host::mutating_frame takes. An inverse delta is not a rewind, because delay and differentiate keep history.
• Memory. Resident state beside the store is the per-cell Seq, the multisets, ExactSum per summed group, occupancy per live shape, the successor-query index, the width histograms and the bounded notice log. The store itself is the integrated base and is never copied. It is measured at ADR 008's fixture before the store-memory work begins.

Alternatives rejected

• Feldera's dbsp crate: 67 required dependencies, including tokio and zstd, and it fails to build for wasm32.
• Compiling formulas into operators: a second evaluator.
• An ad-hoc dirty-tracking cache: one invalidation rule per derived fact, each a second description of evaluation.
• Per-client filtering of notices, or pushed per-window deltas: they save traffic only by holding per-client state on the host (open question 1).

Consequences

Good:

• A write's evaluation costs O(delta) plus the formula cells it reaches.
• The store scans and the per-view model clones go.
• ADR 008 gets occupancy, totals and widths from one evaluator.
• Live sources that report changes plug in.
• The host holds no per-client state.

Bad:

• Two backends exist until Eager goes.
• Integrated state is O(cells) beside the store.
• ExactSum costs 568 bytes per summed group.
• Every client receives every notice.
• A mutating frame still clones the state for rollback.

Order (each step is a ticket in the implementation epic, behind the differential test):

1. zset;
2. the Derived trait with Eager;
3. the cell delta, the record tier and the dirty flag;
4. order-free aggregates and occupancy, measured at ADR 008's fixture;
5. formulas and the aggregate-over-expression pipeline;
6. Sum, Avg and totals through ExactSum;
7. live sources;
8. delivery.

Step 8's required tests:

• [AddItem, SetCell] publishes one notice, and no client asks for a correction;
• a dropped notice is detected;
• a reconnect after a host restart asks for a correction;
• a corrected state equals the target, held + (target − held) = target, and correcting a state that is already correct is the zero Z-set (property tests over random edit sequences and losses);
• a refused frame consumes no publication number;
• a dropped hop triggers a correction.

Open questions

1. Should notices be filtered per client? Clause 9 sends every client every live shape's moved keys. That keeps the host free of per-client state, but traffic is O(clients × all moved keys) per batch. Filtering by topic would send each connection only its own shape's key lists. It needs one subscription per connection, which pins its shape in the cache. Is such a subscription "model-relevant information" the backend may hold? On a two-hop path the topics could instead live in a gateway. Recommendation: broadcast with the size cap. Add topics only if measured traffic with many clients on different shapes justifies them.

2. How should cycles be handled natively? Clause 6's component fixed point is interim. The options:

   • (a) a recursion operator bounded at sixteen iterations, writing circular where it has not converged, which keeps today's answers;
   • (b) marking every cell in a graph cycle circular, which changes guarded cycles such as A = IF(x, B, 1);
   • (c) a monotone reformulation, where one exists.

   Recommendation: (a).

3. What replaces Eager? It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. Recommendation: a small from-scratch reference implementation, kept in the test crate, as the oracle, and a pushdown backend behind the same trait for query-only sources.

4. Which columns get a host-maintained width? Columns whose cells are text? What about a column holding both text and numbers? Recommendation: text cells get the maintained width, and the view layer takes the larger of that and its own numeric width.