diff --git a/context/file-inventory.md b/context/file-inventory.md index 3da13bdf..2ccfcb23 100644 --- a/context/file-inventory.md +++ b/context/file-inventory.md @@ -294,6 +294,21 @@ crates/improvise-core/examples/category_width_bench.rs (improvise-felo.1; context/code/core.md) ``` +### Sources, incremental evaluation and circuits (`crates/improvise-core/src/model/`, `crates/{improvise-incremental,zset,zset-ffi,numtower}/`) +``` +improvise-core/src/model/source.rs CellSource, ChangeSource, Select, Summary/Numbers/Agreement, BadParts, SharedSource; TableSource (test-support); Model::over +improvise-core/src/model/over_tests.rs tests of a model over a source +improvise-incremental/src/lib.rs Derived trait and re-exports + reading.rs, delta.rs, eager.rs Reading, CellDelta/DeltaValue/cell_delta/moved_records, Eager (the oracle) + group.rs, shapes.rs, formulas.rs maintained groups (ExactSum), shapes per category set, formulas by stem + incremental.rs, edit.rs, fold.rs Incremental (over/sync/apply_changes), Edit, the fold + pushdown.rs, reads.rs Pushdown over a query-answering source; reads shared with Eager + harness.rs differential, probes, Maintained + sched.rs, staged.rs Executor/Unit/Pool (Pool native-only); the aggregate/grid example on two units +zset/src/{circuit,ops,trace,edge}.rs Circuit (arena, Send not Sync, begin/commit/rollback), operators, Trace, edges (EdgeSink/EdgeSource/Pipe/EdgeCodec/Stamp) +zset-ffi/src/{lib,engine}.rs the C ABI; include/zset_ffi.h (zs_circuit_begin/commit/rollback, ZS_ERR_TRANSACTION, ZS_ERR_INTERRUPTED) +``` + ### Context docs ``` .claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl @@ -317,6 +332,9 @@ scripts/pr-watch Waits on a PR's checks (the pr-watch skill). scripts/assemble-stamp.sh The cache-bust stamp gate web/assemble.sh calls, extracted so its refusals can be exercised without a web/pkg scripts/assemble-stamp-test.sh Its check_the_checker, run first in the `build` CI job +context/code/source.md CellSource, Model::over, Summary, ChangeSource (ADR 009 step 7) +context/code/improvise-incremental.md The Derived plug: steps 3-7, Eager's math, Scheduling +context/code/zset.md Circuit, transactions, threads, edges context/code/tooling.md The repository's own checkers: why an evidence heading's names come from cargo rather than from the source, and why a leaf name is not an identity diff --git a/context/gotchas.md b/context/gotchas.md index f6692978..f47177d3 100644 --- a/context/gotchas.md +++ b/context/gotchas.md @@ -138,4 +138,16 @@ Dated claims and counts inside an entry are snapshots: recount rather than cite. 53. **Delivery: a watermark is a `Stamp`; a correction is `target - held`, applied only at the watermark it starts from.** `through` is not `at` (the watermark advances only to `through`), a refused frame restores `Delivery` with the app snapshot, and a lost last notice is found only by `poll`. Axis changes refetch the window until ADR 008's render model. The argument is `context/code/delivery.md` (sections 3, 4 and 6); a refused frame restoring `Delivery` is §6, Rollback. + **Livelocks (improvise-z2vr).** A notice from another epoch only sets `stale`; `seen` moves only through an answer (`apply_answer`, shared by `sync` and `poll`), because ignoring one kind of foreign notice after another left the live host's old notices looking new. `Notice::cut_for` limits a notice to one shape and notices cut for different shapes do not merge. Long form `context/code/delivery.md`, sections 3 and 4. + +54. **Sources and exactness:** one evaluator (`Model`'s) over any `CellSource`; a `Model::over` source is read-only in every build, and a `ChangeSource` has one drainer (its cells are as of its last drain). Eager's math is fixed (`settled` compares by bits with every NaN one value; Min/Max by `total_cmp`; NaN-only Min/Max is NaN; a stable NaN is not `circular`). An exactness test needs an input that is inexact in floats. Stem order is a known evaluator defect (improvise-bw4p.30). `context/code/source.md`, `improvise-incremental.md`. + +55. **zset circuits are ids, not shared pointers.** `Stream`/`Input`/`Output` are `Copy` ids used through the circuit (`current`/`push`/`take`); a closure cannot read a stream during a step. Rollback replays newest first, `take` an output only after `commit`, and a panicking step outside a transaction leaves the circuit unusable. `context/code/zset.md`, "Threads" and "Transactions". + +56. **An edge joins by correction only.** A receiver joins an incarnation only by the answer to its latest request (a token per receiver incarnation, seeded fresh by `EdgeSource::new`); the host's timeout must exceed the round trip. `context/code/zset.md`, "Edges". + +57. **A scheduler turn runs outside the lock.** A panicking turn is caught and its unit buried (dropped inside `catch_unwind`), so `wait_idle` does not hang on a dead unit; a panic payload whose own drop panics is still open (improvise-bw4p.40). `context/code/improvise-incremental.md`, "Scheduling". + +58. **Tooling notes:** `Cargo.nix` has drifted from a fresh crate2nix run (improvise-pqf1). `proptest-regressions/` is gitignored, but zset commits its seeds (`crates/zset/tests/*.proptest-regressions`, a file beside the test, which that pattern does not match). `scripts/build-doc-site.mjs` lists every `context/` doc by hand: register a new one there. CI's test derivation sets `CARGO_PROFILE_DEV_DEBUG = "0"` (`nix/ci.nix`). + --- diff --git a/context/repo-map.md b/context/repo-map.md index 57efb2a9..694fdf27 100644 --- a/context/repo-map.md +++ b/context/repo-map.md @@ -3,17 +3,17 @@ Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture. Apache-2.0. Root binary+lib `improvise`; sub-crates under `crates/`: -- `improvise-core` — `Model`, `View`, `Workbook`, number formatting. No UI/IO. +- `improvise-core` — `Model`, `View`, `Workbook`, number formatting, and `model/source.rs` (`CellSource`: where raw cells are read from; `Model::over`). Depends on `improvise-formula` + `numtower` (exact sums). No UI/IO. `context/code/source.md`. - `improvise-formula` — formula parser, AST, `parse_formula`. - `improvise-io` — `.improv` save/load, CSV/JSON import, `Storage` backends. No UI/commands. - `improvise-protocol` — the wire vocabulary and the portable session slice: `Command`, `Projection`, `ViewState`, the per-widget caches, `reduce_view`, key types, and `delivery/{wire,notice,exchange,client}.rs` (notices, watermarks, corrections). Depends on `improvise-core`, `zset`, `numtower` + serde. No IO, no command layer, no terminal. - `improvise-command` — command/effect layer + semantic `App` state (app.rs, effect.rs, cat_tree.rs, viewport.rs, delivery.rs). Depends on `improvise-protocol`, `improvise-gp`, `zset`, `improvise-incremental`. No ratatui/crossterm. -- `numtower` — exact decimal arithmetic tower (`Num`, canonical text), ADR 005. Pure Rust, no workspace deps; `improvise-protocol` uses it for the wire form. Long form `context/code/numtower.md`. -- `zset` — std-only Z-sets (non-zero checked `i64` weights, panic on overflow) plus `Circuit`, operators, `Trace`, `Rebuild` (ADR 009 clause 1). No dependencies. `context/code/zset.md`. +- `numtower` — exact decimal arithmetic tower (`Num`, canonical text), ADR 005. Pure Rust, no workspace deps; `improvise-protocol` uses it for the wire form and `improvise-core` for exact sums. Long form `context/code/numtower.md`. +- `zset` — std-only Z-sets (non-zero checked `i64` weights, panic on overflow) plus `Circuit` (`Send`, not `Sync`; transactions `begin`/`commit`/`rollback`), operators, `Trace`, `Rebuild`, and `edge.rs` (stamped, coalescing delta edges between circuits). No dependencies. `context/code/zset.md`. - `zset-ffi` — C ABI over `zset` (`cdylib`/`staticlib`/`rlib`) for CFFI; `include/zset_ffi.h`. Native only, **not** in `wasmCrates`. `context/code/zset-ffi.md`. -- `improvise-incremental` — the `Derived` trait, `Reading`, `Eager`, `differential_in`/`drawn_probes`, and `cell_delta`/`DeltaValue`/`moved_records` (the store's `CellChange` log to a `ZSet`). Depends on `improvise-core` + `zset`; `core` never depends on `zset`. `context/code/improvise-incremental.md`. +- `improvise-incremental` — the `Derived` trait, `Reading`, `Eager`, `differential_in`/`drawn_probes`, and `cell_delta`/`DeltaValue`/`moved_records` (the store's `CellChange` log to a `ZSet`), `Incremental` (maintained groups/shapes, formulas by stem, `ExactSum`; `over`/`sync`/`apply_changes` on a `ChangeSource`), `Pushdown` (a query-answering source), and `sched.rs`/`staged.rs` (an `Executor` of `Unit`s; `Pool` is native-only). Depends on `improvise-core`, `zset` + `numtower`; `core` never depends on `zset`. `context/code/improvise-incremental.md`. - `gp-tree` — problem-agnostic genetic-programming regression trees (`Tree`, `Spec`, `breed`, `simplify`, `render`). Knows nothing about improvise. -- `improvise-gp` — the one seam between the two: emits a fitted `gp_tree::Tree` as improvise formula text (`formula_text` / `fitted_formula`). Depends on `gp-tree`, `improvise-formula` **and `improvise-core`** — `dataset.rs` builds a feature matrix from a `View`'s records (improvise-4s8) and `fitted_formula` takes the `&Model` it must resolve against (improvise-8xhy). Nothing depends on it yet. +- `improvise-gp` — the one seam between the two: emits a fitted `gp_tree::Tree` as improvise formula text (`formula_text` / `fitted_formula`). Depends on `gp-tree`, `improvise-formula` **and `improvise-core`** — `dataset.rs` builds a feature matrix from a `View`'s records (improvise-4s8) and `fitted_formula` takes the `&Model` it must resolve against (improvise-8xhy). `improvise-command` depends on it. - `improvise-ws-server` — the host: tokio + tungstenite, a `Registry` of `Host`s keyed by `DocumentId` (multi-document, ADR 001), `Sessions` fan-out, and the socket framing (`wire.rs`). Native only. - `improvise-worker-server` — the *other* host: the same `App`, `Sessions` and `Host` as the socket server, behind `self.onmessage`/`self.postMessage` with OPFS underneath. wasm32 only in anger; `worker.rs`/`vfs.rs` are ordinary Rust and are where every decision and every test is. Nothing depends on it. - `improvise-wasm-client` — the browser client: the fold, the key classifier, the socket framing it mirrors (`socket.rs`), and the render model the DOM paints (`render.rs`). Non-dev deps are `improvise-protocol` + serde + wasm-bindgen and nothing else. The page itself is `web/`. @@ -49,6 +49,10 @@ Architectural intent lives in `context/design-principles.md` — read that for t | Change import-wizard keys/steps | `command/cmd/wizard.rs` (keys) + `import/wizard.rs` (steps) | | Change frame layout | `draw.rs` → `draw()` | | Change an incremental operator or the circuit | `crates/zset/src/{ops,circuit}.rs` | +| Change where raw cells are read from | `improvise-core/src/model/source.rs` (`CellSource`, `Model::over`); `context/code/source.md` | +| Change an incremental aggregate/formula | `improvise-incremental/src/{group,shapes,formulas}.rs` (`Incremental`); reads shared with `Eager`/`Pushdown` in `reads.rs` | +| Run sub-circuits on threads | `improvise-incremental/src/sched.rs` (`Executor`, `Unit`, `Pool`) + `staged.rs` | +| Change the edge protocol between circuits | `crates/zset/src/edge.rs` + `context/code/zset.md`, "Edges" | | Change the C ABI | `crates/zset-ffi/src/lib.rs` + `crates/zset-ffi/include/zset_ffi.h` | | Change the wire form of a value or delta | `improvise-protocol/src/delivery/wire.rs` | | Change what the host publishes/answers | `improvise-command/src/ui/delivery.rs` | @@ -225,7 +229,12 @@ the incidents and the ticket ids.** Read the long form of every gotcha your diff 50. **A composed TUI row measures in cells and names which side yields** (`fill_line` + `Keeps`). Keep whichever key table is *derived from what it describes*. Scrolls are measured where drawn (`grid_width`). 51. **The command language reads pipes, refuses surplus, and checks names when applied** (`only_when_omitted`; `RequireKnown`/`unless_unknown`). 52. **A date column imports as one hierarchical category;** `--split-date` restores flat. `Plan::spelling_clash` refuses what would silently merge; `--strict` refuses partial skips. -53. **Delivery: a watermark is a `Stamp`; a correction is `target - held`, applied only at the watermark it starts from.** `through` is not `at` (the watermark advances only to `through`), a refused frame restores `Delivery` with the app snapshot, and a lost last notice is found only by `poll`. Axis changes refetch the window until ADR 008's render model. Long form `context/code/delivery.md`. +53. **Delivery: a watermark is a `Stamp`; a correction is `target - held`, applied only at the watermark it starts from.** `through` is not `at` (the watermark advances only to `through`), a refused frame restores `Delivery` with the app snapshot, and a lost last notice is found only by `poll`. Axis changes refetch the window until ADR 008's render model. A notice from a foreign epoch only marks the shape stale (`seen` moves only through `apply_answer`), and notices cut for different shapes (`cut_for`) never merge. Long form `context/code/delivery.md`. +54. **Sources and exactness:** one evaluator (`Model`'s) over any `CellSource`; a `Model::over` source is read-only in every build, and a `ChangeSource` has one drainer (its cells are as of its last drain). Eager's math is fixed (`settled` compares by bits with every NaN one value; Min/Max by `total_cmp`; NaN-only Min/Max is NaN; a stable NaN is not `circular`). An exactness test needs an input that is inexact in floats. Stem order is a known evaluator defect (improvise-bw4p.30). `context/code/source.md`, `improvise-incremental.md`. +55. **zset circuits are ids, not shared pointers.** `Stream`/`Input`/`Output` are `Copy` ids used through the circuit (`current`/`push`/`take`); a closure cannot read a stream during a step. Rollback replays newest first, `take` an output only after `commit`, and a panicking step outside a transaction leaves the circuit unusable. `context/code/zset.md`, "Threads" and "Transactions". +56. **An edge joins by correction only.** A receiver joins an incarnation only by the answer to its latest request (a token per receiver incarnation, seeded fresh by `EdgeSource::new`); the host's timeout must exceed the round trip. `context/code/zset.md`, "Edges". +57. **A scheduler turn runs outside the lock.** A panicking turn is caught and its unit buried (dropped inside `catch_unwind`), so `wait_idle` does not hang on a dead unit; a panic payload whose own drop panics is still open (improvise-bw4p.40). `context/code/improvise-incremental.md`, "Scheduling". +58. **Tooling notes:** `Cargo.nix` has drifted from a fresh crate2nix run (improvise-pqf1). `proptest-regressions/` is gitignored, but zset commits its seeds (`crates/zset/tests/*.proptest-regressions`, a file beside the test, which that pattern does not match). `scripts/build-doc-site.mjs` lists every `context/` doc by hand: register a new one there. CI's test derivation sets `CARGO_PROFILE_DEV_DEBUG = "0"` (`nix/ci.nix`). --- diff --git a/prolog-sessions/kb.pl b/prolog-sessions/kb.pl index da838702..aca028e3 100644 --- a/prolog-sessions/kb.pl +++ b/prolog-sessions/kb.pl @@ -10,9 +10,9 @@ crate(improvise_ws_server,'crates/improvise-ws-server','the socket host: tokio + crate(improvise_wasm_client,'crates/improvise-wasm-client','the browser client: the fold, the key classifier, and the render model the DOM paints. Non-dev deps are improvise-protocol + serde + wasm-bindgen and nothing else.'). crate(improvise_worker_server,'crates/improvise-worker-server','the other host: the same App, Sessions and Host as the socket server, behind self.onmessage/self.postMessage with OPFS underneath. worker.rs and vfs.rs are ordinary Rust and hold every decision and every test; bindings.rs is the wasm32 shim, run by harness/tests/shell.spec.js under Chromium and by no Rust test.'). crate(numtower,'crates/numtower','exact decimal arithmetic tower (ADR 005): Num, canonical text; no workspace deps. Used by improvise-protocol for the wire form.'). -crate(zset,'crates/zset','std-only Z-sets (checked i64 weights) plus Circuit, operators, Trace and Rebuild (ADR 009 clause 1); no dependencies.'). crate(zset_ffi,'crates/zset-ffi','C ABI over zset (cdylib, staticlib, rlib; include/zset_ffi.h) for CFFI. native_only: not in nix/ci.nix wasmCrates.'). -crate(improvise_incremental,'crates/improvise-incremental','the Derived trait, Reading, Eager, differential_in/drawn_probes and cell_delta/DeltaValue; plugs zset in behind a trait (ADR 009).'). +crate(zset,'crates/zset','std-only Z-sets (checked i64 weights) plus Circuit (Send, not Sync; transactions), operators, Trace, Rebuild and edge.rs (stamped delta edges between circuits); no dependencies.'). +crate(improvise_incremental,'crates/improvise-incremental','the Derived trait, Reading, Eager, Incremental (groups, formulas by stem, ExactSum), Pushdown, cell_delta/DeltaValue, and sched.rs/staged.rs (Executor of Units, native-only Pool); plugs zset in behind a trait (ADR 009).'). layer(model,improvise_core). layer(view,improvise_core). layer(formula,improvise_formula). @@ -592,6 +592,15 @@ decision('improvise-felo.4',resumable_deadline_predicate,'The resumable step tak decision('improvise-felo.4',eager_occupancy,'Eager::occupancy counts the visible stored members of a collapsed group, and formula-target keys are not asked.'). decision('improvise-z2vr',delivery_axis_refetch,'A delivery axis change makes the correction a window refetch until ADR 008 supplies the render model (context/code/delivery.md section 3).'). decision('improvise-felo.4',dirty_flag_is_delta_plus,'The document dirty flag is each CellData effect own cell delta (drained after every such effect, so ordering with Save holds) plus reorders, formulas and config; writes cancelling across effects still leave it set. The record tier alone uses the consolidated batch delta (moved_records). record_fingerprints and document_content are gone.'). +decision('improvise-bw4p',zset_send_by_arena,'Circuit is Send and not Sync with no lock: it owns every edge slot, input pending delta and sink sum in one Vec> arena, nodes hold typed ids, and Stream/Input/Output are Copy ids used through the circuit (current/push/take; a foreign handle panics). A closure can therefore not read a stream during a step. Cost about 7-9% per tick, measured and kept (context/code/zset.md, Threads). Not Sync is pinned by a_circuit_is_not_sync.'). +decision('improvise-bw4p',multithreading_is_scheduling,'User ruling 2026-10-11: a sub-circuit stays synchronous and single-threaded, the edge (zset::edge) is the only channel that crosses threads, and a small executor (improvise-incremental sched.rs) decides which thread runs which unit. Native threads only: Pool and the blocking calls are cfg(not(wasm32)). A ruling recorded in context/code, not an ADR amendment.'). +decision('improvise-bw4p',edge_join_by_correction,'A zset edge receiver joins a sender incarnation only by the answer to its latest request: a token per receiver incarnation (EdgeSource::new seeds it from a fresh base; restart keeps the counter), echoed in the Correction. A frame from an unjoined epoch only marks stale. Two earlier designs, joining by a first delta with from == 0 and a left list of incarnations, were wrong. The host timeout (connection_lost) must exceed the round trip (context/code/zset.md, Edges).'). +decision('improvise-bw4p',sched_worker_survives_any_turn_panic,'A unit turn runs outside the executor lock. Its panic is caught and the unit buried (never run or kicked again, recorded in Executor::panics), and a dead unit is dropped inside catch_unwind so a Drop that panics cannot kill the worker. wait_idle does not hang on a dead unit. OPEN: a panic_any payload whose own Drop panics is dropped outside catch_unwind (improvise-bw4p.40).'). +decision('improvise-bw4p.5',zset_rollback_is_an_undo_log,'Circuit::begin/commit/rollback keep an undo log of what the steps touch, not a snapshot; rollback replays newest first so no undo leaves i64; a Rebuild is bound to its circuit and a rollback that took its chunks interrupts it for good (the circuit counts undone steps); take an Output only after commit. A step that panics outside a transaction leaves the circuit unusable. zset-ffi exposes zs_circuit_begin/commit/rollback with ZS_ERR_TRANSACTION and ZS_ERR_INTERRUPTED.'). +decision('improvise-rnak',a_change_source_shows_state_as_of_its_last_drain,'A ChangeSource shows its cells as of its last drain_changes, so a snapshot read now plus changes drained later counts no change twice or not at all. A feed has one drainer: two backends over one feed split the changes (test a_feed_has_one_drainer). Model::over(source) is read-only in every build (assert, not debug_assert) and the evaluator is still the single Model one.'). +decision('improvise-rnak',an_exactness_test_needs_an_inexact_input,'A test that a sum or merge is exact must feed values whose float sum is order-dependent or inexact; with exactly representable inputs a float implementation passes (found reviewing Summary merges). Summary::from_parts refuses states no sequence of values could produce rather than repairing them.'). +decision('improvise-bw4p.20',eager_math_fixed,'User ruling 2026-10-10: do not reproduce math bugs in Eager. Min/Max order by total_cmp over non-NaN values (-0.0 below +0.0, platform independent); Min/Max over only NaN is NaN; the formula fixed point compares with model::types::settled (bits, every NaN one value) so a stable NaN is not circular and a zero sign flip is a change; empty SUM is +0.0. Min/Max skip NaN while Sum/Avg propagate it (kept, a product question).'). +decision('improvise-bw4p.30',stem_order_is_a_known_evaluator_defect,'recompute_formulas updates its cache in place in stem order and stops at 16 passes, so two sources holding the same cells in different orders can disagree. This is an evaluator defect, not a source contract: distinct_without returns first-seen order and a source with no order of its own should list stems in a stable one until it is fixed.'). wire_carries(command,both_directions,'user intent up, projection commands down'). wire_carries(snapshot,on_connect_only,'improvise-q08: full ViewState + render cache filled from current viewport'). wire_carries(command,both_directions,'user intent client->server, projections server->client'). @@ -660,6 +669,8 @@ crate_depends_on(improvise_command,zset). crate_depends_on(improvise_command,improvise_incremental). crate_depends_on(improvise_gp,improvise_core). crate_depends_on(improvise_command,improvise_gp). +crate_depends_on(improvise_core,numtower). +crate_depends_on(improvise_incremental,numtower). depends_transitively(A,B):-depends_via(A,B,[A]). depends_via(A,B,C):-crate_depends_on(A,B). depends_via(A,B,C):-crate_depends_on(A,D), \+memberchk(D,C),depends_via(D,B,[D|C]).