The server already serialises every write — one `App`, one `apply_effects` —
so a total order exists and this is not about ordering. What was missing is
noticing that a client composed an edit against a picture of the document
that has since moved: the stale-basis shape of gotcha 24 and gotcha 39, one
level up.
`Command::Model` becomes a struct variant carrying a `Basis` — the three
tiers of ADR 001's table, `document_generation`, `layout_version`, and the
write counters of the records the batch names. `Sessions::apply` checks it
before applying anything and returns `Applied::Rejected { reason }`, a third
case beside `Diff` and `Rebased` so a host that wants the `Dirty` has to name
it (gotcha 39's argument). `ServerMsg` grows the matching third frame,
addressed to the asker alone: nothing happened, so nobody else is owed
anything, and the asker already holds what is current.
The tiering is the economy. A `CellData` batch is refused when the *layout*
moved even though no record it names was written, because the coordinates its
key hangs off changed underneath it — A deleting an item while B commits an
edit keyed by it. That is corruption rather than a lost keystroke, it is the
class a cell-level lock would not have covered, and it falls out of the tiers
with no lock anywhere.
`Rejection` keeps structural and data staleness apart, which is what ADR
002's maintained `RecordHash` index in `DataStore` exists for: `RecordGone`
means the record does not resolve and there is nothing to write to, so
nothing coherent to force; `RecordMoved` means it resolves at another
version, where force is one keystroke. Collapsing them is the failure a
regenerate-from-the-layout scheme would have had.
**No bypass, by construction.** `Basis` has no `Default` and no public
constructor — `BasisCache::claim` is the only door and it withholds until the
server has published one, so a zero-valued forgery is not expressible. Which
records are checked is `records_named`, the server's own exhaustive read of
the command, not whatever the client chose to declare: a client that omitted
its record would otherwise have a force by omission, with no flag to remove.
Force stays what the ADR says it is — a re-composition against the current
basis — and `re_composing_against_the_republished_basis_is_accepted` walks
that path with nothing that skips the check.
WHERE THE ADR WAS WRONG: it says "a batch carries the basis it was composed
against" and never says how a client comes to know one. Nothing in the
projection stream carried any of the three tiers, so the check as specified
was satisfiable only by forging. `Projection::Basis` + `BasisCache` close
that: the server publishes the basis for the records a client's own grid
names, diffed against what that client was last told, seeded in the
bootstrap. The publication is not in ADR 001's implementation order and
should be.
Tests, per `doctrine(check_the_checker)` — a check nobody has watched reject
is indistinguishable from one that always accepts, and the converse: every
tier appears twice, rejecting and accepting. Fixtures hold every identifier
constant and move only the tier under test, and each asserts that the *other*
tiers did not move, so a rejection cannot be a renamed key or the wrong tier
firing (`doctrine(a_rename_can_hide_a_stale_key)`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Gotcha 18 gains matching_cells/matching_values and the reason the old hash
walk was not a tidiness issue -- the drill snapshot froze its records in that
order while pending_edits are keyed by row index, so a second process
addressed different records.
Gotcha 37 gains (d): a fit is storable only when the document can name its
features, the check asks the document rather than the name's shape, and this
is an interim because remedy 3 needs a scalar coordinate predicate the
formula language does not have.
Also corrects improvise-gp's dependency line, stale since improvise-4s8.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Two bugs found incidentally by other agents and worked around rather than
fixed. Both are cases where a green suite was not evidence.
shyt: matching_cells/matching_values walked the secondary index's HashSet,
so records-mode row order differed between processes -- and worse, the drill
snapshot froze its record list in that order while pending_edits are keyed by
ROW INDEX, so the same drill in a second process addressed different records.
f64 is not associative, so SUM/AVG drifted in the last bits too.
8xhy: preprocess one-hots into Region=North, which find_item_category cannot
resolve, so every cell of a categorical fit was Error(ref:...). Interim
remedy: fitted_formula takes the &Model and refuses. It asks the document,
never the name's shape -- a measure literally named Region=North is legal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`preprocess` invents feature names -- `Region=North` for a one-hot,
`Cost.missing` for a missing indicator, `when.year` for a date part. Those are
encoding artefacts, not items of anyone's document, so the emitted `Expr::Ref`
failed `Model::find_item_category` and every cell of the fitted measure came
back `Error("ref:Region=North")`. `fitted_formula` said `Ok` throughout. So
"fit a tree and get a working formula" held only for a fit over numeric
measures, and nothing said so.
`fitted_formula` now takes the `&Model` the formula is to be stored in and
refuses a tree whose splits it cannot resolve. That is the same contract the
function already had -- its doc called the parse "the emitter's own contract
check, so a name or threshold this crate cannot spell surfaces here rather
than as a broken formula in the user's file" -- widened by one word, from
spell to resolve. `formula_text` is untouched: it is the tree's own text and
there is nothing else for it to write.
The check asks the *document*, never the name's shape. A measure literally
named `Region=North` is a legal document and is accepted; the two are
indistinguishable as strings and only the document can tell them apart.
Asking the real precondition also covers `.missing`, the date parts, and any
future sub-feature naming scheme with no second copy of `preprocess`'s
expansion rules to drift from.
THIS IS AN INTERIM, and labelled as one in the function's doc. It is the
smallest of the ticket's three remedies and is honest about what works rather
than making more work. The other two were costed. Materialising one-hots as
real `_Measure` items pollutes the document with artefacts the user never
asked for. Teaching the emitter improvise's `WHERE` vocabulary needs a
*scalar coordinate predicate*, which the formula language does not have:
`Filter` is a formula-level or aggregate-level gate (`Formula::filter`,
`Expr::Agg`'s third field), not an expression, so a categorical split needs a
new `Expr` variant, a grammar production, arms in both evaluators, and an
answer to what the predicate means in a cell that aggregates over the
category -- where `context.get("Region")` is `None` and the formula would
silently fall through to `aggregate_raw`. And it would close only the
categorical third of the defect, leaving `.missing` and the date parts exactly
where they are. Full costing in the report on improvise-8xhy.
On the test that was lying. `improvise_evaluation_matches_gp_tree_eval` could
not see any of this because its `improvise_eval` helper fabricates a
`_Measure` item for every feature name, including `outlook=rainy` -- it
asserts the emitter and `gp_tree::eval` agree against a model that could not
exist. The test is kept, because quoting and threshold routing are real things
it does check, and its doc now states its scope. The property it could not
state is asserted separately against a real document: if `fitted_formula` says
`Ok`, no cell of the target is a reference error. The characterization test
`a_split_on_a_one_hot_feature_does_not_evaluate` is flipped to
`a_split_on_a_one_hot_feature_is_refused`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Most work here is done by waves of agents in worktrees, and every rule in
this section was learned by breaking it: the three orchestrator-only files
and why, that kb.pl additions must be batched because loadProgram replaces a
whole predicate, that tickets sharing a file belong to one agent, that a
worktree does not reliably branch from the orchestrator's HEAD, that direnv
status reports 0 for allowed, and that the code review's characteristic find
here is a test that passes while the thing it names is broken.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`GridWidget` no longer holds a `&Model`. Every value it paints — each
cell's text, whether the cell is an aggregate, the column widths, the page
bar and the totals strip — now arrives already decided on `GridCache`. A
widget that cannot reach the model cannot quietly answer a question
differently from the way the browser client answers it, and that
divergence is the whole thing projecting exists to prevent (improvise-edp
was one instance of it).
Three derivations moved rather than being deleted:
- **Column totals.** The widget summed every row of each visible column
through `Model::evaluate_aggregated_f64`. `GridCache::totals` now
carries the strip and `project::column_totals` computes it. It is
projected rather than left to the client because the sum runs over
*every* row of the grid, including the rows a viewport has scrolled
past — a client holding only its window could not compute it. An
**empty** strip means "no totals" (records mode, or an empty grid),
which is a different statement from a strip of blanks, and it is the
cache's answer to a question the widget used to ask `is_records_mode`
twice.
- **The page bar.** `GridCache::page_coords`, off `layout.page_coords`.
- **The italic-aggregate rule.** The widget walked `none_cats` against
`Model::category(..).kind.is_regular()`. That fact is already on the
projection as `CellId::Aggregate` (ADR 002) — decided once, by the side
that holds the model — so the widget now reads it per cell.
Both new fields are framed by `col_labels` exactly as the cells and widths
are, and `fit_to_labels` re-fits the totals with them (gotcha 33c): a
renderer zips the strip with the columns, and a length mismatch indexes
off the end. `Invalidated` blanks the totals too, because they are
contents and a stale sum under fresh cells is the one number a user would
take at face value. Both get a `GridUpdate` variant, a `seed()` entry and
a `grid_updates` diff arm, so bootstrap stays a compaction of the same
stream (gotcha 32).
What deliberately did NOT move, and is the finding worth carrying forward:
`GridCache` carries *flat* labels — `GridLayout::row_label` joins a
multi-level tuple with `/` — while the TUI draws one header row per level
with repeat suppression, plus inline group headers with a collapse
indicator. Neither the tuples nor the groups exist in the protocol, so
`GridWidget` still takes `&GridLayout` + `&View` for the frame alone. That
is a gap in the cache, not a design: the browser grid has no groups and no
multi-level headers today, and the widget's doc comment says so where the
next reader will find it. `drill_state` also stays off the cache, for the
opposite reason — it is session-local residue (gotcha 30).
Behaviour is pinned rather than argued: `the_whole_frame_is_unchanged`
asserts the exact buffer text of a grid with row groups, column groups, a
page axis, multi-level headers and a total row, written against the old
path before the move and unchanged after it
(`doctrine(characterization_survives_the_move)`). A `contains` could not
have seen this — repeat suppression, indicator placement, column alignment
and adaptive row-header width are all positional.
The TUI builds the cache over the whole grid (offsets zero), so a position
in it is an absolute position and the widget's scrolling arithmetic is
untouched. One consequence to watch: `grid_of` now evaluates every cell
per frame where the widget evaluated only the visible ones. The totals
loop was already O(rows x visible cols), and `compute_col_widths` was
already whole-grid, so this widens an existing cost rather than
introducing one — but it is a real widening on a wide grid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`TransposeAxes` and `CycleAxis` reset the cursor because renumbering the
grid makes the old indices name different cells (gotcha 24), but the reset
only reached the authoritative `App`'s own view slice. Every `ClientSession`
keeps its own cursor in its mirrored `ViewState`, and nothing reset those, so
a client was left indexing a grid that had changed shape under it —
`Viewport::window` clips, so what it actually saw was a blank grid until it
happened to scroll. Reachable remotely since the model half of the wire
vocabulary landed (improvise-kmgs).
Three pieces, and the first two are worthless apart. `Command::ResetCursor`
is the wire variant, derived from the `App::reset_cursor` write the two
geometry effects make on the portable slice. `Sessions::adopt_cursor_epoch`
resets every mirror *and* returns the broadcast that tells the clients:
resetting only the mirrors would leave the diff believing each client already
holds origin-positioned content, so nothing would be sent and every client
would stay stale permanently rather than until its next scroll — a partial
fix strictly worse than none. And the trigger is `App::cursor_epoch` moving,
not the command variant, for the reason gotcha 39 watches
`document_generation` instead of matching on `LoadModel`: which effects
renumber the grid is a per-effect decision, so a table of variants here would
be a second description of it, free to disagree.
The audience had to become part of the message. A `SwitchView` broadcast
corrects state the asker folded optimistically and must not go back to it; a
cursor invalidation is a consequence the asker is owed on the same terms as
everyone else, because a `Command::Model` is the one kind a client cannot
fold optimistically. So `Applied::Diff::broadcast` is now
`Vec<Broadcast>` over `Broadcast::{ToOthers, ToAll}` — a `Vec` because one
command can owe two relays with different audiences, and an `Option` would
have had to drop one silently.
What is not broadcast is a position. Another client's transpose must not move
where I am looking — that is UI and it is mine (improvise-g0f) — it may only
declare that where I was looking has stopped meaning anything, which is a
fact about the shared geometry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`DataStore::matching_cells` and `matching_values` walked the secondary
index's `HashSet`, so their order was the hash order: different between
processes, and different between two stores holding the same cells. That
defeated the reason `DataStore::cells` is an `IndexMap` in the first place
-- its own doc comment says "so records mode can display rows in the order
they were entered" (gotcha 18) -- and made any `f64` fold over the result
reproducible only within one run.
The index exists for O(1) *lookup*, not for iteration, so it stays as what
narrows the candidates and the primary map becomes what orders them: the new
`matching_positions` resolves each surviving candidate to its `IndexMap`
position with one hash lookup and sorts the positions. The shape stays
O(smallest set x |partial|) with O(m log m) added over the *matches*, never a
scan of the store.
Measured on 18,000 cells (release, 2000 iterations/case), before -> after:
matching_values 1 match 6.9us -> 7.3us
matching_values 300 matches 9.0us -> 10.8us
matching_values 3600 147us -> 165us
matching_cells 300 matches 37.7us -> 35.0us
matching_cells 3600 535us -> 500us
Worst case +20% on the value-only path; `matching_cells` is slightly *faster*,
sorted positions resolving CellKeys with better locality.
Reached beyond records mode: the drill snapshot (`cmd/grid.rs`) froze its
record list in the same unstable order, and `drill_state.pending_edits` are
keyed by row index (gotcha 1), so the same drill opened twice could key
staged edits to different records. No change needed there -- it inherits.
The tests are built the way the instability shows. Asserting a literal order
would only pin what this machine's hasher did today (doctrine
check_the_checker), so one test compares against `iter_cells` -- the primary
map's own order -- and one builds the same logical cells by two opposite
insertion histories and requires each to list its own.
`improvise-gp::dataset` sorts its rows by coordinate tuple as a workaround for
this; the sort is kept but is no longer what buys stability, and its two
comments saying otherwise are corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Three corrections. The selection is reconciled by ADDING the offset to the
render index, not subtracting it from the selection -- which is why there is
no underflow path; and the test that earns its place uses nonzero, unequal
offsets so it separates 'reconciles' from 'ignores' and from 'swaps'.
The mirrored framing is kept honest by TWO tests: the round trip catches a
rename, and an exhaustive match over the server's enums catches an addition
the hand-written fixture list would miss. And the check-the-checker was
itself broken -- it injected malformed JSON, so is_err() held for a syntax
error and would have held against an enum that really did have the variant.
Plus what a careful read of web/app.js established and what it could not:
filed as improvise-jm5v.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`a_frame_named_the_other_envelopes_way_does_not_parse` exists to prove that
`the_socket_framing_matches_the_server` is discriminating: mirroring
`improvise-ws-server`'s two wire enums is only safe if a *renamed* variant
fails a test. The injected fault was not a rename.
ours.replacen("Commands\":[", "Command\":", 1)
rewrites `{"Commands":[{"SetStatus":"saved"}]}` to
`{"Command":{"SetStatus":"saved"}]}` — the opening bracket is deleted with
the variant name and the trailing `]` is left behind, so the string is not
JSON at all. `is_err()` then held for a *syntax* error, and would have held
identically against a `ServerFrame` that really did carry a `Command`
variant. Proven by adding one: with `ServerFrame::Command(Vec<Command>)` in
the enum the old assertion still passed, because malformed JSON fails to
parse whatever variants exist. The checker was checking serde_json's
tokenizer, which is exactly `doctrine(check_the_checker)`'s failure mode
wearing the name of its cure.
The rename is now asserted well-formed before it is fed in, and the failure
is asserted to be `unknown variant` rather than merely a failure — the same
correction applied to the `Bootstrap` case, which was right by luck. Under
the injected variant the corrected test fails.
Second gap, same test: the round trip enumerates variants **by hand**, so a
variant added on the server side and never mirrored here fails nothing.
`every_server_variant_is_one_this_crate_mirrors` matches exhaustively over
`ClientMsg` and `ServerMsg`, so that is a compile error in this crate rather
than a runtime surprise in a browser.
1091 passing, clippy and fmt clean, and `cargo check -p improvise-wasm-client
--target wasm32-unknown-unknown` still clean — the guard lives in a test
module and `improvise-ws-server` stays a dev-dependency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Neither improvise-ws-server nor improvise-wasm-client appeared anywhere in
the map, which is a gap in the two newest crates and the two the browser MVP
is made of. Adds both to the crate list, the how-to-find table and the file
inventory, plus web/.
Gotcha 42 records why the client's render decisions are in Rust (the DOM half
is untested JavaScript), the three answers that would be silently wrong in JS,
and why socket.rs duplicates ws-server's enums -- with the note that there are
three envelopes for two deployments and no Key variant on the socket.
Also: gotcha 34's 'section headers are unquoted on both sides' clause was
made false by improvise-pczv and is replaced by what header_name now does;
the Wasm section said four crates and now says six.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
The last blocker of improvise-1ey. Render decisions live in Rust under test
(render.rs) because the DOM half is untested JavaScript; the ws-server framing
is mirrored in socket.rs and compatibility-tested against the server's own
types, because a native crate cannot be a wasm dependency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
The last blocker of improvise-1ey. Mostly assembly: `Client` already had the
fold, the key classifier and `set_cell`, `CellDisplay::id` already said what a
cell is (ADR 002), and `flatten_cat_tree` already owned the category panel's
flattening. What was missing was the three joints between them.
`render.rs` is the one that matters. The DOM half is JavaScript and JavaScript
here is untested, so every question with a *silent* wrong answer is answered
in Rust under test and crosses as decided data: `editable` is a fact about the
`CellId` variant (Record only — an Aggregate is a drill target and `Drill` is
not on the wire yet, and `None` is "no projected cell", not "aggregate"), the
selection is placed by subtracting the scroll offset because `GridCache` is
viewport-indexed while the cursor is not, and the category panel is
`flatten_cat_tree` rather than a second walk of the cache.
`socket.rs` frames what `improvise-ws-server` actually speaks, which is not
what `wire.rs` speaks: it has `Viewport` (the handshake — a snapshot built at
extent zero is an empty grid), it batches `Commands`, and it has no `Key`
variant. The two enums are duplicated because a native crate cannot be a wasm
dependency; `improvise-ws-server` is a dev-dependency so the compatibility is
checked through its own types, and a companion test proves the check is
discriminating on the variant names that would drift.
`web/` is the shell, and it is thin on purpose: four string calls in and out,
no protocol knowledge, no framework.
Gates: 1047 passing (was 1031), clippy and fmt clean, and
`cargo check -p improvise-wasm-client --target wasm32-unknown-unknown` clean.
Verified end to end headlessly — the wasm-pack artifact driven from node over
a real websocket to a real `improvise-ws-server`: 1,200 read, "98765" typed,
"98,765" projected back. The DOM painting itself is unverified; there is no
headless browser here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Decision: `.improv` stores the emitted formula and no serialized
`gp_tree::data::Model`. `persistence/improv.pest` is explicitly confirmed
unchanged. Rationale on the crate docs.
Against a blob: it is a second copy of what the file already carries -- the
tree IS the formula -- with nothing keeping the two in agreement when a user
edits the formula or a cell it was fitted on, and a round-trip test over the
blob cannot see the drift (gotcha 41's second half; its first half, "a second
implementation of the grammar", does not transfer to an opaque production).
And `Tree`/`FitConfig` are gp-tree types, so improvise-io would gain a
gp-tree dependency -- the inversion this crate exists to prevent.
What the blob was for does not arise here: the encoding plan exists so
`fit_csv --predict` can score an untyped CSV row, while improvise's new row is
a record in the same document that `recompute_formulas` scores through the
stored formula. Seed + config are not written either -- gp-tree already
records them, and nothing reads them until the fit command (improvise-vx3)
exists to decide how provenance is worded.
Three tests carry the reasoning, and one of them refutes the ticket's premise.
"The emitted formula is enough to recompute existing cells" holds only for a
fit over numeric measures: a one-hot feature is named `Region=North`, an
encoding artefact that is not an item of the document, so `Expr::Ref` fails to
resolve and every cell of the fitted measure is `Error("ref:Region=North")`.
`col.missing` and the date sub-features are in the same position. Recorded as
a characterization test; the remedy belongs to improvise-vx3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
`## Category:`, `## View:`, `# ` and `Initial View:` never went through a
quoting rule on either side: the grammar read `rest_of_line` and the writer
emitted the name raw. That is coherent for a whole-line token, and it is why
round-trips worked — but it is not total, and the gap is reachable.
An RFC 4180 CSV may quote a header field containing a newline. Nothing on the
import path forbids it, so `"Region\nName",Amount` imports cleanly into a
category of that name, `format_md` writes it across two lines, and the saved
document cannot be reopened. Leading or trailing whitespace is quieter and
worse: the reader trims, so the category returns renamed while every data
coordinate still names the untrimmed original — the cells are orphaned with
no error. Both are demonstrated by tests that fail before this change.
The answer keeps the whole-line contract rather than routing headers through
`quote_name`. The header carries its name as the whole line — that is why
`# Q3 Budget` needs no pipes where a reference to the same name does — and
`quote_header` pipe-quotes only the names the line cannot carry: the empty
name, edge whitespace, an embedded line break, and a name already spelled
`|…|`. Each clause is one thing the reader does. `quote_name` was rejected
because it would also quote every ordinary spaced title, rewriting the header
of every existing document for no correctness gain.
Grammar first: `header_name = { pipe_quoted ~ &NEWLINE | line_name }`. The
`&NEWLINE` decides the one ambiguous line — `## Category: |a| and |b|` is a
whole-line name, not a quoted one — and `line_name` is the old `rest_of_line`
branch, so a header written by an older build still loads unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
A `Host` was always one authoritative `App`, one `Sessions` and the clients
attached to them, and `Sessions` never owned the `App` -- so multi-document
is a registry above `Host` and nothing below it changed. `improvise-command`
and `improvise-protocol` are untouched.
The one thing a `Host` could not supply is a server-wide address: `SessionId`
is minted per `Sessions`, so every document's first client is `SessionId(0)`.
`ClientId { document, session }` is that address, and `Registry::route` is
the single place a `SessionId` acquires the document it came out of -- which
makes cross-document delivery unrepresentable rather than checked. A
broadcast and an `Applied::Rebased` fan-out therefore stop at the document
boundary by construction (gotchas 36, 39).
Lifecycle. Documents are opened at startup from the files named on the
command line; a client cannot ask the server to open a path, because
choosing which paths a client may open is access control and that is out of
scope. A document is *not* closed when its last session leaves: eviction
would lose unsaved edits, and would rewind `document_generation` and
`record_versions` to zero so a reconnecting client's stale basis would be
accepted as current (ADR 001).
A client names its document in the URL path -- `ws://host:port/quarterly` --
so `ClientMsg` is unchanged and a single-document server still answers `/`.
A name nothing serves is closed with a websocket close frame carrying the
reason; no `ServerMsg::Error` variant was invented for it.
Tests: 14 in `registry.rs` (socket-free), 4 in `main.rs`, 1 in `transport.rs`
and a second socket test for the URL rule. Each isolation and lifecycle claim
has a fault injected beside it (`doctrine(check_the_checker)`): a same-document
bystander that does hear the edit, a second document that has the same view
name to fold, a failed load that rebases nobody, and a freshly opened document
that has lost both the edit and the basis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Records mode is already a feature matrix: a row is one coordinate tuple and
each measure is its own column (gotcha 1). `dataset::feature_table` enumerates
the same rows against the user's own view and lays them out as the string
table `gp_tree::data::preprocess` reads; `dataset::dataset` runs it through
preprocess so one-hot expansion, imputation and the feature ranges stay in
gp-tree.
Four selection rules settled here, each recorded on the module:
* rows are scoped by `View::filtering_cats()` (Page union Filter), not by
`categories_on(Axis::Page)` -- gotcha 27's trap, and the one that fails
silently.
* hidden items scope the rows and the feature columns ("fit what I can see");
an explicitly named target is exempt.
* every regular category is a feature whatever axis it sits on, including
Axis::None, because a coordinate is part of what makes a record distinct --
dropping it leaves two rows with equal feature vectors and different
targets. Label categories are excluded, as in records mode, by not being
`is_regular`.
* no formula-derived value enters the matrix at all: a formula target is not a
feature, and is refused *as* the target. One rule covers both halves of the
ticket's "the target, or a formula depending on it", with no dependency
analysis to get wrong.
Records are sorted by coordinate tuple: `DataStore::matching_cells` walks a
HashSet, so an unsorted table would make a seeded fit reproducible only within
one process. Numbers reach the table via `f64::to_string`, never `format_f64`
(gotcha 8). Nothing prints; skipped records come back in `warnings` (gotcha
37).
improvise-gp gains improvise-core as a real dependency (it was already a
dev-dependency).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
A git worktree does not load the flake pin: its .envrc is unallowed, so a
process there falls through to whatever bd the profile carries. Today the
two agree, which is why nothing has broken; nothing enforces that they
keep agreeing.
The failure mode is silent — a mutating bd of the wrong version empties
the dependency graph in .beads/issues.jsonl and reports success. Section 0
of the session-start hook now reads the pinned version out of flake.nix
(never a second copy, which would rot when the pin moves), compares it to
`bd version`, and prints a banner and exits 2 on any disagreement. An
unreadable pin or an unreadable version fails the same way: an assertion
that cannot answer is not a pass.
It blocks nothing — reading with a skewed bd is safe — and it runs before
the CLAUDE_CODE_REMOTE gate so local worktrees, where the problem lives,
are covered. The cloud path re-checks after installing bd and still
completes its setup before the non-zero exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
improvise-vjms: `CategoryTreeCache` carries `_Index`/`_Dim`/`_Measure`
alongside the regular categories, so the TUI panel and a browser panel
render the same rows from one source. `category_rows` drops the
`regular_category_names()` filter; its doc comment now says why, and the
cache's stale "the client does not know the `_`-prefixed convention" is
deleted rather than left to mislead.
The trap `improvise-ete` was closed for: `_Measure`'s items are derived —
explicit items ∪ formula targets (gotcha 7) — so membership goes through
`effective_item_names`. A raw `category("_Measure").items` compiles, looks
right, and drops every formula target; the new characterization test is
the only one in the suite that notices (verified by injection).
improvise-edp: the ViewModel layer the ticket specced is mostly not there
to build — after improvise-35e each widget already takes one cache plus
`&AppMode`, and a per-widget cache (gotcha 32) is already render-shaped.
What is real is the category panel's *flattening*, which mixes the cache
with per-client `ViewState::expanded_cats`. It moves to
`improvise_protocol::viewmodel`, not `improvise-command`, because
improvise-bpq2 resolves `move-panel-cursor` client-side and the browser
client depends on the protocol crate alone. The entry carries its axis, so
`CategoryContent` now takes a cache and an expanded set — no `&Model`, no
`&View`. That is the fourth and last panel off the model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
improvise-l4qx. `evaluate_aggregated` and `aggregate_raw` both reduced a
cell's matching values with `filter_map(as_f64)`, so a `CellValue::Text`
aggregated to nothing and rendered blank. `_Index` and `_Dim` sit on
`Axis::None` in every default view, so the aggregating path is the normal
case: every text cell in every document was invisible, including the
pipe-quoted text `.improv` documents treat as a first-class value.
Text now aggregates by unanimity. One matching value, or several copies
of the same value, reduce to that value — reducing equal things is the
identity. Differing values, or text among numbers, have no defined
reduction and become `Error("mixed")`, which the user sees. Emptiness
keeps `apply_agg`'s answer (absent, not zero); the explicit error for an
undefined reduction follows `eval_agg`'s.
The all-numeric branch is checked first and is unchanged, so `Count` of a
lone number is still 1 rather than the number — the case that rules out
"pass a single value through" as the rule. Both call sites now share one
helper, as `apply_agg` already required of this family (improvise-kdp).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
User ruling: for many use cases editing capabilities would be a downside. Two
simplifications follow. It dissolves the ADR 001 tension by scoping rather than
amendment — ADR 001 governs concurrent edits and a read-only model has none, so
what remains is projection freshness, a smaller and different question. And the
machinery already exists: ADR 002 makes SetCell take a CellId::Record, so a
source projecting none makes a write unconstructible with no runtime check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
The composition improvise-q08 found a thin client could not do. The client
holds no Model and cannot build a CellKey; it does not need to, because the
surrogate on the cell it is already drawing is the address, echoed back
unread.
Client::set_cell(row, col, value) returns None on three distinct shapes the
cache already carries: off the grid, no projected cell (id: None), or an
aggregate -- and the last is not a check, it is the `let CellId::Record(..)
= .. else` failing to bind. Nothing is folded locally: the new value comes
back as a GridUpdate from the host, which is the side that formats it.
set_cell_at takes the CellValue as JSON, matching the crate's rule that
everything crossing the boundary is JSON -- what a typed string *means* is
the host's decision and is not re-implemented here.
Also adds the records-mode half of the emission tests: a records cell must
be named by RecordRow::coords and never by the synthetic (_Index, _Dim) key,
whose positions are exactly what ADR 002 rejected _Index for; and a
coordinate column carries no id, because editing it renames a coordinate
across the whole row (gotcha 1), which is not a SetCell.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Accepted but explicitly not a priority: the goal is the OPTION of using
improvise as a DB client. Written down now so the option stays open, since
several decisions in flight could foreclose it by accident.
The consequence worth the ADR: a live-backed document splits into our
configuration and their data, which .improv does not distinguish — and read
through the DB-client framing that is the natural shape, a saved query plus a
pivot configuration, i.e. a report definition. Also flags that an external
writer voids a guarantee ADR 001 currently states.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
ADR 002's spine, end to end.
CellId::{Record(RecordCell), Aggregate(AggregateHash)} rides on
CellDisplay::id, filled in by grid_of for the same reason `numeric` is: the
server is the side that knows. The kind of identity is the permission --
SetCell/ClearCell take a RecordCell and nothing else, so a client cannot
compose a write against an aggregate, and an aggregate id is still an id, so
Drill has something to address.
The fork is not invented: `cell_id` asks the question EditOrDrill already
asks -- a regular category on the None axis -- and records mode projects
RecordRow::coords rather than the synthetic (_Index, _Dim) key, whose
positions are exactly what ADR 002 rejected _Index for.
`id: None` means "no projected cell here" (padding, Invalidated, or a
records-mode category column, which renames a coordinate across the row and
is a different intent from SetCell). It is NOT "aggregate" -- that is
Some(CellId::Aggregate(..)).
model_effects now takes the App, because resolving a surrogate is a lookup
against authoritative state. Resolution is the index first (every record
that holds data, on screen or not) and the layout second (a cell with no
value yet is named by the grid and by nothing in the store -- which is the
whole of improvise-1ey). An unresolvable surrogate applies nothing.
record_versions moves off App into the DataStore beside the index: still
shared document state, now one lookup for both questions.
The socket integration test now composes its SetCell from the id the server
projected, which is the only handle a browser has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
ADR 002's surrogate, and the reverse index a server resolves it through.
The hash is specified rather than inherited: SipHash-2-4 under a constant
key over an explicit length-prefixed byte serialization of the sorted
(category, item) pairs. Not DefaultHasher/RandomState (randomly seeded per
process), not #[derive(Hash)] (no stability guarantee). 128 bits, argued in
the doc comment: a collision is a silent write to the wrong record and no
later check catches it, while the cost of the wider hash is a bounded 8
bytes per visible cell. The wire form is hex-as-a-string, because a JSON
number above 2^53 comes back changed in a JavaScript host.
RecordHash::of normalizes, so a cell key and its record hash alike -- the
_Measure-drop is a property of the constructor, not a rule callers follow.
The index unifies ADR 002's resolution with ADR 001's record_versions:
RecordHash -> RecordEntry { record, cells, version }, one lookup answering
both "which record do you mean?" and "is your basis stale?". A zero cell
count is a tombstone rather than a removal, so a delete-and-recreate cannot
hand a stale basis its own version number back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
improvise-9ix declined on measurements: 22% larger than .improv, 4.7x gzipped,
160x at the floor, and still rewritten whole on every save. The generalisable
half is that a second serialization format duplicates the grammar and its own
round-trip test cannot detect drift from the first — the missing cross-format
equality test is what made the ticket look cheap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
An aggregate is not the absence of an identity but a different kind of one, so
Option<CellId> became CellId::{Record, Aggregate}. That makes an invalid Drill
unconstructible too, not just an invalid SetCell, and it closes the drill
addressing question the Option shape had left open. Hover previews deferred.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Decided: hash the record (coord tuple minus _Measure), not the cell — it is
already ADR 001's atomicity unit and what record_versions is keyed by, so one
identity serves addressing and staleness. Resolution is a maintained index in
DataStore, which already interns names and already keeps a (Symbol,Symbol)
secondary index that makes invalidation tractable.
Records the three ways this breaks silently: a per-process-seeded hasher, a
derived Hash with no stability guarantee, and a width chosen for convenience
rather than for the consequence of a collision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
An opaque surrogate on CellDisplay whose absence means the cell is an
aggregate, so a client cannot compose SetCell for something unsettable.
Records why _Index cannot serve: VirtualIndex is positional over the matching
set, so a stale one names the wrong record silently — but what it points at,
record identity, is ADR 001's unit, and one surrogate can serve both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH
Edward Langley