docs(repo-map): the index narrows and the primary map orders; a fit names only what the document holds

Gotcha 18 gains matching_cells/matching_values and the reason the old hash
walk was not a tidiness issue -- the drill snapshot froze its records in that
order while pending_edits are keyed by row index, so a second process
addressed different records.

Gotcha 37 gains (d): a fit is storable only when the document can name its
features, the check asks the document rather than the name's shape, and this
is an interim because remedy 3 needs a scalar coordinate predicate the
formula language does not have.

Also corrects improvise-gp's dependency line, stale since improvise-4s8.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EoW5BB87ZZdm96SvBTWawH

.beads/issues.jsonl

@@ -26,7 +26,7 @@
 {"_type":"issue","id":"improvise-hwpi","title":"The browser shell fails quietly, and never reads a key","description":"Found by the improvise-cr3 code review (2026-09-19) by READING web/app.js. There is no browser and no headless DOM in this environment, so none of this is observed -- it is what the code says will happen, and it is filed rather than fixed because each item is a judgement call about the shell rather than a defect in the Rust.\n\nFOUR THINGS, in descending order of how much they would cost a person at a demo:\n\n1. NO KEYBOARD AT ALL. #grid carries tabindex=\"0\" but the page registers no keydown handler, so Client::on_key_event -- and the whole keys.rs classifier and the lockstep machinery behind it -- is UNREACHED from the browser. Consistent with improvise-bpq2 (ClientMsg has no Key variant server-side) but larger than it: even a LOCAL key, one that needs no round trip, does nothing. improvise-1ey acceptance (3) is \"arrow keys move the cursor locally with no round-trip\", and nothing in the page can do it.\n\n2. A MALFORMED FRAME IS QUIET. client.on_frame(event.data) returns Result\u003c_, JsError\u003e, which wasm-bindgen turns into a THROWN EXCEPTION. It is thrown inside the message listener with no try, so it surfaces as an uncaught error in the devtools console and the following repaint() is skipped. Nothing in the UI says so: the grid simply stops updating. A try/catch writing to el.status would make it visible.\n\n3. NO error LISTENER ON THE SOCKET. A connection that never opens leaves #status blank -- only close writes \"disconnected\". So \"server not running\" and \"still connecting\" look the same.\n\n4. close() RUNS TWICE ON COMMIT, benignly. Enter -\u003e close(true) -\u003e td.replaceChildren(original) removes the focused \u003cinput\u003e -\u003e blur fires -\u003e close(false). editing is already null so it is harmless (one extra repaint), but there is no re-entry guard and only a real browser would show it.\n\nNOTE WHAT MAKES THIS TICKET DIFFERENT: items 2-4 are the kind of thing a test would catch instantly and we have no way to run one. Before fixing them, consider whether the right first move is a headless DOM in CI (jsdom or playwright) -- otherwise the fixes are as unverified as the bugs. improvise-xf1 (verify OpfsFs against a real browser) wants the same harness.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:40:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:40:36Z","dependency_count":0,"dependent_count":0,"comment_count":0}
 {"_type":"issue","id":"improvise-zv4m","title":"improvise-protocol should own the websocket framing, so both ends stop mirroring it","description":"Raised by improvise-cr3 (2026-09-19) as the thing to push back on, and filed rather than fixed there.\n\nTODAY: improvise-wasm-client/src/socket.rs declares ClientFrame/ServerFrame, which MIRROR improvise-ws-server/src/wire.rs ClientMsg/ServerMsg field for field. The client cannot depend on the server crate -- tokio, tungstenite, Box\u003cdyn Storage\u003e, all native -- so the duplication was the only way to speak the protocol from wasm.\n\nWHAT HOLDS THE LINE MEANWHILE: improvise-ws-server is a DEV-dependency of the client (the same argument the crate already makes for improvise-command), and the_socket_framing_matches_the_server round-trips every variant THROUGH THE SERVERS OWN TYPES in both directions; a_frame_named_the_other_envelopes_way_does_not_parse is its check-the-checker. So a drift is caught -- but only by a test whose job is to notice, rather than by a shared definition that cannot drift.\n\nTHE FIX: move the framing into improvise-protocol, which both ends already depend on and which is the crate whose charter is exactly this -- \"the wire vocabulary and the portable session slice\". Then the mirror, the dev-dependency and both compatibility tests all delete.\n\nNOTE THE THIRD ENVELOPE while you are in here: improvise-wasm-client/src/wire.rs has ClientMessage/ServerMessage framing the WORKER deployment (improvise-tm6). Three envelopes for two deployments is worth resolving in one pass -- decide whether the worker and the socket want the same vocabulary before moving either.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:37:50Z","closed_at":"2026-09-20T00:37:50Z","close_reason":"Duplicate of improvise-xks6 ('One home for the client/server frame envelopes'), filed minutes apart from the improvise-cr3 hand-back without spotting the existing ticket. Closing this one; xks6 is the survivor and its description should absorb two facts this one carried:\n\n * WHAT HOLDS THE LINE MEANWHILE: improvise-ws-server is a DEV-dependency of improvise-wasm-client, and the_socket_framing_matches_the_server round-trips every variant THROUGH THE SERVERS OWN TYPES in both directions, with a_frame_named_the_other_envelopes_way_does_not_parse as its check-the-checker. The drift is caught by a test whose job is to notice, rather than prevented by a shared definition.\n * THERE ARE THREE ENVELOPES, NOT TWO: improvise-wasm-client/src/wire.rs frames the WORKER deployment (improvise-tm6) separately again. Decide whether the worker and the socket want the same vocabulary before moving either, or the move produces two homes instead of one.\n\n(See also memory agent-issue-drift-pattern -- search before filing.)","dependency_count":0,"dependent_count":0,"comment_count":0}
 {"_type":"issue","id":"improvise-8xhy","title":"[bug] A fit over a categorical feature emits an unevaluable formula","description":"Found during improvise-7na (2026-09-19), verified by probe, and pinned as a characterization test rather than fixed.\n\ngp_tree::data::preprocess one-hots a categorical column into a feature named Region=North. That is an ENCODING ARTEFACT, not an item of the document, so the emitted formula Expr::Ref fails Model::find_item_category and EVERY CELL of the fitted measure evaluates to Error(\"ref:Region=North\"). col.missing and the col.year / col.month date sub-features are in exactly the same position.\n\nSo \"fit a tree and get a working formula\" holds ONLY for a fit over numeric measures.\n\nTHE TEST THAT SHOULD HAVE CAUGHT IT WAS LYING: improvise_evaluation_matches_gp_tree_eval cannot see this, because its improvise_eval helper FABRICATES a _Measure item for every feature name, including outlook=rainy. It asserts the emitter and gp_tree::eval agree, which they do -- against a model that could not exist. Worth remembering as a shape: a helper that builds the world the code wants is not a test of whether that world exists.\n\nPinned as a_split_on_a_one_hot_feature_does_not_evaluate -- characterization, flip it when fixed.\n\nTHREE REMEDIES, pick deliberately: restrict fits to numeric features (smallest, and honest about what works); materialise one-hot columns as real _Measure items (makes the formula evaluable but pollutes the document with encoding artefacts); or teach the emitter improvise WHERE vocabulary so a categorical split becomes a real predicate over the category (largest, and the only one that produces a formula a human would have written). The third is probably right and should be costed before the first is shipped as permanent.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:35:05Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:35:05Z","dependency_count":0,"dependent_count":0,"comment_count":0}
-{"_type":"issue","id":"improvise-shyt","title":"[bug] Records-mode row order is not stable across processes","description":"Found during improvise-4s8 (2026-09-19). A real latent bug in improvise-core, out of that ticket scope, worked around there by sorting.\n\nDataStore::matching_cells returns cells in HashSet iteration order, so RecordRow::group -- and therefore RECORDS-MODE ROW ORDER IN THE LIVE UI -- is not stable across processes. The same document opened twice can list its records in different orders.\n\nWHY IT MATTERS BEYOND TIDINESS:\n * gotcha 18 says IndexMap insertion order is load-bearing for persistence, and DataStore::cells is IndexMap precisely so \"records mode can display rows in the order they were entered\" (its own doc comment). The SECONDARY INDEX walk defeats that.\n * f32 accumulation is order-dependent, so anything summing over matching_cells is reproducible only within one process. improvise-4s8 had to sort before building a feature matrix or a SEEDED FIT would not reproduce across runs -- see the known-answer vectors in gp-tree, which exist because that class of drift is silent.\n * A user watching rows reorder between sessions has no way to tell whether the document changed.\n\nFIX: make matching_cells (or its callers that care) yield in the primary IndexMap order rather than the index HashSet order. Note the index exists for O(1) LOOKUP, not for iteration, so the fix is probably to intersect the index result back against the insertion-ordered primary rather than to change the index.\n\nTEST IT THE WAY THE INSTABILITY ACTUALLY SHOWS: a single-process test cannot distinguish \"sorted\" from \"hashed that way today\". Build the same document twice with different insertion histories that produce the same set, or assert directly against the IndexMap order.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:35:04Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:35:04Z","dependency_count":0,"dependent_count":0,"comment_count":0}
+{"_type":"issue","id":"improvise-shyt","title":"[bug] Records-mode row order is not stable across processes","description":"Found during improvise-4s8 (2026-09-19). A real latent bug in improvise-core, out of that ticket scope, worked around there by sorting.\n\nDataStore::matching_cells returns cells in HashSet iteration order, so RecordRow::group -- and therefore RECORDS-MODE ROW ORDER IN THE LIVE UI -- is not stable across processes. The same document opened twice can list its records in different orders.\n\nWHY IT MATTERS BEYOND TIDINESS:\n * gotcha 18 says IndexMap insertion order is load-bearing for persistence, and DataStore::cells is IndexMap precisely so \"records mode can display rows in the order they were entered\" (its own doc comment). The SECONDARY INDEX walk defeats that.\n * f32 accumulation is order-dependent, so anything summing over matching_cells is reproducible only within one process. improvise-4s8 had to sort before building a feature matrix or a SEEDED FIT would not reproduce across runs -- see the known-answer vectors in gp-tree, which exist because that class of drift is silent.\n * A user watching rows reorder between sessions has no way to tell whether the document changed.\n\nFIX: make matching_cells (or its callers that care) yield in the primary IndexMap order rather than the index HashSet order. Note the index exists for O(1) LOOKUP, not for iteration, so the fix is probably to intersect the index result back against the insertion-ordered primary rather than to change the index.\n\nTEST IT THE WAY THE INSTABILITY ACTUALLY SHOWS: a single-process test cannot distinguish \"sorted\" from \"hashed that way today\". Build the same document twice with different insertion histories that produce the same set, or assert directly against the IndexMap order.","status":"closed","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:35:04Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:45:20Z","closed_at":"2026-09-20T00:45:20Z","close_reason":"Fixed. A private DataStore::matching_positions replaces the duplicated prelude in matching_values and matching_cells: the secondary index still NARROWS the candidates -- that is what it is for -- but each survivor is resolved to its position in the primary IndexMap with get_index_of, positions are sorted, results read back with get_index. The index does lookup, the primary map does ordering.\n\nTHE BLAST RADIUS WAS LARGER THAN FILED -- five callers, and the one nobody knew about is worse than the headline:\n * view/layout.rs::group_records -- records-mode row order in the live UI (the filed case).\n * command/cmd/grid.rs::DrillIntoCell -- THE DRILL SNAPSHOT froze its record list in hash order while drill_state.pending_edits are keyed by ROW INDEX (gotcha 1). So the same drill opened in a second process keyed staged edits to DIFFERENT RECORDS: silent misdirected writes, not a cosmetic reorder. Never filed; inherits the fix.\n * eval_agg, evaluate and aggregate_raw fold f64 over the result, and f64 addition is not associative, so a formula's value could differ in the last bits between two openings of the same document. Min/Max/Count are order-free, and so is the text-unanimity rule (gotcha 7).\n * improvise-gp dataset.rs::visible_records was already sorting as a workaround (improvise-4s8). The sort is KEPT -- removing it would change the fit's row order and gp-tree's known-answer vectors are bit-level -- but its comments claimed the sort is what buys stability, which this fix makes false, so they now say it is a choice of CANONICAL order: a fit's rows should be a property of the data, not of the typing history.\n\nMEASURED, release, 18,000 cells, 2000 iterations: worst case +20% on matching_values (a sort plus one hash lookup per candidate); matching_cells is 7% FASTER, because sorted positions resolve CellKeys with better locality. Nothing became O(store) -- the alternatives (an ordered index value type, or scanning the primary) would have moved the cost onto every set or onto every query.\n\nTHE TEST SHAPE IS THE POINT: a single-process test cannot tell 'sorted' from 'hashed that way today on this machine'. So the order is compared against iter_cells -- the primary map's own order -- rather than a literal, and the_same_cells_entered_in_opposite_orders_list_in_opposite_orders builds the same logical store by two different histories and requires each to list its own.","dependency_count":0,"dependent_count":0,"comment_count":0}
 {"_type":"issue","id":"improvise-xks6","title":"One home for the client/server frame envelopes","description":"Two crates now define the same frames independently, which is a live divergence risk:\n  improvise-wasm-client::wire  -- ClientMessage / ServerMessage\n  improvise-ws-server::wire    -- ClientMsg / ServerMsg\nThe ws-server agent asked for one version to be landed and will delete its wire.rs and re-export; the change on its side is two type names.\n\nPUT THEM IN improvise-protocol. Both sides already depend on it, and this is the one thing there that is FRAMING rather than vocabulary -- the protocol says what a Command is, but not what distinguishes one JSON blob from another on the wire.\n\nTHE SHAPE, as the ws-server landed it and the client should adopt:\n  client -\u003e server:  Viewport { rows: usize, cols: usize } | Command(Command)\n  server -\u003e client:  Snapshot(Box\u003cSnapshot\u003e) | Commands(Vec\u003cCommand\u003e)\nNotes that matter: Snapshot is BOXED because clippy large_enum_variant fires otherwise and a Snapshot is sent once per connection while Commands is every frame. Server-\u003eclient is ONE shape carrying both projections (as Command::Project) and a peer broadcast, because on the wire they are the same thing and a client folds both -- a separate Projections variant was rejected for that reason.\n\nViewport is the only addition to the shared vocabulary and it is derived, not invented: the extent is the one session input no Command carries by construction (gotcha 40).","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T23:46:23Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:46:23Z","dependency_count":0,"dependent_count":0,"comment_count":0}
 {"_type":"issue","id":"improvise-l4qx","title":"[bug] A text cell is invisible whenever any category is on the None axis","description":"Found during improvise-q08 (2026-09-19), PRE-EXISTING, and it is the NORMAL case rather than an edge one.\n\nModel::evaluate_aggregated takes the aggregating path when none_cats is non-empty, and that path does filter_map(as_f64) -- so a CellValue::Text aggregates to NOTHING and never becomes a CellDisplay. _Index and _Dim sit on Axis::None in a default view, so none_cats is non-empty almost always.\n\nWHY IT MATTERS: .improv has pipe-quoted text cells as a FIRST-CLASS FEATURE -- the format doc gives Region=West, Measure=Revenue = |pending| as an example -- and they do not render. A user can write a text value, save it, load it and not see it.\n\nThe q08 agent hit this while building a fixture, worked around it there, and noted it in the round-trip test comment. Costing a test iteration is how it surfaced; nothing in the suite covers it.\n\nWrite the failing test first: a text cell in a default view (so _Index/_Dim are on None) must render its text. Then decide what aggregation of text should MEAN -- filter_map(as_f64) silently dropping it is clearly wrong, but \"show the single value when there is exactly one, and something explicit when there are several\" is a product decision, not an obvious fix. Note gotcha 7: apply_agg and eval_agg already differ on empty input, so this family has precedent for being decided rather than defaulted.","status":"closed","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T23:46:09Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:24:05Z","closed_at":"2026-09-20T00:24:05Z","close_reason":"FIXED. aggregate_cell_values in model/types.rs, shared by evaluate_aggregated and aggregate_raw (which had byte-identical bodies precisely so they could not drift -- improvise-kdp).\n\nTHE RULE IS UNANIMITY, and the ORDER is load-bearing:\n  empty            -\u003e None (absent, unchanged -- follows apply_agg)\n  ALL NUMERIC      -\u003e apply_agg -\u003e Number (unchanged; checked FIRST)\n  all values equal -\u003e that value\n  otherwise        -\u003e Error(\"mixed\")\n\nReducing several copies of one value is the identity, so that reduction has an answer even when the values are not numbers -- which covers the case the ticket is actually about. Two different words, or a word among numbers, have no reduction, and gotcha 7 already has precedent for saying so out loud (eval_agg returns Err(\"empty\") for AVG/MIN/MAX over nothing rather than inventing a value). So this is consistent with BOTH existing rules: emptiness follows apply_agg (absent, not zero), undefined-reduction follows eval_agg (explicit error, not silence).\n\nTHE ORDERING IS THE WHOLE DIFFERENCE from the ticket own first suggestion. \"Show the single value when exactly one matches\" taken literally BREAKS Count, whose entire point is to not be the value. The all-numeric branch is tested first, so Count of a lone number is still 1. Proven by injection: swapping the branch order failed exactly one test, a_lone_number_still_counts_as_one_rather_than_passing_through, which is why that guard exists.\n\nREJECTED: \"show the first\" (depends on DataStore hash iteration order, so the displayed value would be arbitrary AND look authoritative while hiding the others -- worse than the blank); \"show a count\" (changes the type of the answer; a count rendered under a Sum measure is indistinguishable from real data); and making the display path differ from the formula path.\n\nBEHAVIOUR CHANGE TO KNOW: a formula ref over MIXED values now errors \"mixed\" instead of \"ref:name\". Still an error, but a truer one, and it no longer silently sums the numbers while dropping the text. Text-valued refs still error exactly as before. Fixed-point convergence untouched.\n\nFailing test first: a_text_cell_renders_in_a_default_view, which also asserts !none_cats.is_empty() to confirm the premise that a DEFAULT view really does put _Index/_Dim on None -- the normal path, not an edge one. 1024 tests pass; three faults injected, all fired.","dependency_count":0,"dependent_count":0,"comment_count":0}
 {"_type":"issue","id":"improvise-vjms","title":"Decide whether a category panel shows the virtual categories","description":"Found by improvise-35e (2026-09-19). A GENUINE BEHAVIOUR DIVERGENCE, which is why that ticket stopped rather than migrating the widget.\n\nThe TUI category panel builds from cat_tree::build_cat_tree, which walks Model::category_names() and therefore draws _Index, _Dim and _Measure. CategoryRow/category_rows carry only regular_category_names(), and CategoryTreeCache own doc says \"Only the categories a user may see -- the client does not know the _-prefixed virtual-category convention\".\n\nSo migrating the category panel to the cache would SILENTLY DROP THREE ROWS from the TUI panel. That is a behaviour change, not a relocation, so doctrine(characterization_survives_the_move) says stop and ask.\n\nTHE DECISION IS improvise-edp: either the browser panel grows the virtual categories, or the TUI panel loses them. Note gotcha 7 -- _Measure items are explicit items UNION formula targets, so a client showing _Measure needs measure_item_names semantics, not a raw item list; that is part of the cost of the first option.\n\nWhichever way it goes, the widget then also needs a tree flattener over CategoryTreeCache -- today build_cat_tree takes \u0026Model.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T23:40:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:06:54Z","closed_at":"2026-09-20T00:06:54Z","close_reason":"DECIDED (user, 2026-09-19): the TUI keeps the virtual categories AND the browser gets them too. CategoryTreeCache grows to carry _Index, _Dim and _Measure, so both panels render the same thing and the cache stays the single source for what a category panel shows.\n\nWHAT THIS COSTS, and it is the part to get right: the client now has to be told about the _-prefixed convention, which the caches doc currently says it does not know. In particular _MEASURE ITEMS ARE DERIVED, not a raw list -- gotcha 7: they are the explicit items UNION the formula targets, and add_formula does NOT add items to _Measure. So category_tree_of must ship membership computed through measure_item_names()/effective_item_names(), never category(\"_Measure\").items, or the panel will silently omit every formula target. That is the same trap improvise-ete was closed for proposing.\n\nConsequences to carry into improvise-edp:\n * CategoryTreeCache and its doc comment change -- the \"the client does not know the _-prefixed virtual-category convention\" line is now false and must go.\n * build_cat_tree can then migrate off \u0026Model, which is what unblocks the category panel and therefore edp.\n * A characterization test should pin that the cache carries all seven rows for a default workbook (4 regular + 3 virtual) and that _Measure includes a formula target that is not an explicit item -- one fixture that would fail under the raw-list implementation.","dependency_count":0,"dependent_count":0,"comment_count":0}