diff --git a/context/adr/009-incremental-evaluation.md b/context/adr/009-incremental-evaluation.md index c854b1aa..0e14cdd1 100644 --- a/context/adr/009-incremental-evaluation.md +++ b/context/adr/009-incremental-evaluation.md @@ -21,6 +21,10 @@ cell write becomes a delta, aggregates are maintained, and only the formula cell inputs moved are re-evaluated by the existing evaluator. The incremental results must equal today's eager results exactly. +**Principle: everything is as incremental as possible**, so the work done per model +mutation is minimised. Where this ADR does work that grows with more than the +mutation, it says so and names the incremental path that normally applies. + ### 1. The `zset` crate A problem-agnostic workspace crate with no dependencies beyond `std` (so it is @@ -34,8 +38,11 @@ wasm32-clean and serde-free). It provides: - operator state behind one trace interface (look up, read a range, apply a delta), so later store-memory work can page or compress it. -There are no nested streams, no recursion operator, no timestamps beyond one clock, and -no query language. +There are no nested streams, no recursion operator and no timestamps beyond one clock. +**The formula language is the query language.** What a request asks of the model beyond +its axes and window (filters, aggregation, derived values) is written in formula text +over ids (ADR 007). It goes through the same evaluator and circuit as a document +formula, so there is no second query language. ### 2. The plug @@ -85,7 +92,9 @@ shape key with the same None list, which removes the per-session model clone. A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the -ordering category's order. How that order is keyed is for those tickets. +ordering category's order. Those tickets choose the key. It must keep a reorder or an +insert incremental: only the entries under the moved items change, and nothing is +rebuilt. ### 5. Formulas @@ -118,7 +127,8 @@ before enumerating. This replaces `Expansion::members`. DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the answer must be `Error("circular")`. The first version finds the formula graph's strongly -connected components. Acyclic formulas are evaluated once each, in topological order. A +connected components, recomputed when the formula set changes; how to keep that +incremental belongs with open question 2. Acyclic formulas are evaluated once each, in topological order. A component is evaluated with today's bounded fixed point, restricted to that component. That reproduces the sixteen-pass bound for the differential test. It is to be replaced by a DBSP-native treatment (open question 2). @@ -143,7 +153,8 @@ changes no operator. The visible axis is a remap. arriving during the build are buffered and added at install. The finished result is the response to the request that asked for it, not a notice. - **A formula edit** is a formula delta fed to the circuit. -- **A document swap** rebuilds everything. +- **A document swap** rebuilds everything, and a new key's first build is a full pass. + These are the only rebuilds; every other change is a delta. - **Saving a view definition** touches no operator. - **Live keys are a cache.** A key is evicted after a period in which no request names it (the period is set by measurement), and every eviction is announced (clause 9). @@ -243,10 +254,11 @@ an **edge adapter**. So every edge has the same semantics: keys, the host sends the current values of those keys, restricted to the client's window, and the client applies `target − held` over them. The log holds keys and flags, not values, so nothing per client is kept; - - where the covered range holds an axis change, a relabel or a flag, which position - and labels depend on, the host sends `target` for the whole window; - - where the log no longer covers the watermark, the host sends `target` for the whole - window, and the client applies `target − held`; + - where the covered range also holds axis changes, relabels or flags, which positions + and labels depend on, the correction carries them too, in order, as a merged notice + does, so the client re-derives positions incrementally; + - only where the log no longer covers the watermark does the host fall back to `target` + for the whole window, and the client applies `target − held`; - a full refetch is `target − ∅`. **Every read sees only the last committed state**, whether it is a correction, a @@ -294,7 +306,8 @@ notice carries: the circuit computes them. Moved records alone would miss formulas with cross-stem inputs. -A notice whose key lists exceed a fixed size carries "correct" instead. +A notice whose key lists exceed a fixed size carries "correct" instead. That is a +fallback, and it costs every reader a correction. **What a client does.** It intersects the moved keys with its window. A fold row has no maintained key, so it maps member keys through its own remap first. It then fetches what @@ -405,7 +418,9 @@ Step 8's required tests: such as `A = IF(x, B, 1)`; - (c) a monotone reformulation, where one exists. - *Recommendation:* (a). + Separately, the strongly connected components are recomputed whenever the formula set + changes. Should they be maintained incrementally too? *Recommendation:* (a), and + recompute the components only for the formulas an edit touches. 3. **What replaces `Eager`?** It still serves as the differential oracle and as the backend for live sources that can only answer aggregate queries. *Recommendation:* a