From 353bf243b4653e5b638e20f6d2645ca0d246740a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:35:11 -0700 Subject: [PATCH 01/72] fix(keymap): delete the dead F1 binding, and pin the surviving one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit F1 was bound on two consecutive lines: first to a bare `enter-mode` with no args, then to `enter-mode ["help"]`. `Keymap::bind` inserts into a HashMap, so the second wins and the first can never take effect — and had it been the survivor it would have failed `require_args` at dispatch, making F1 do nothing at all. The ticket was filed rather than done because deleting one line of a duplicated pair risks deleting the wrong one. Two pieces of evidence say `enter-mode ["help"]` is intended: the binding dump resolves F1 to it today, and `?` on the very next line carries the same binding. Evidence: the `IMPROVISE_BINDING_DUMP` dump is byte-identical across the deletion (4021 resolved lookups, unchanged). The new `f1_and_question_mark_both_open_help` makes that a standing guard rather than a one-off diff; it was shown to be discriminating by restoring the deleted line and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/keymap.rs | 21 +++++++++++++++++++ crates/improvise-protocol/src/keymap.rs | 1 - 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/crates/improvise-command/src/command/keymap.rs b/crates/improvise-command/src/command/keymap.rs index ba21c2bc..318d73e2 100644 --- a/crates/improvise-command/src/command/keymap.rs +++ b/crates/improvise-command/src/command/keymap.rs @@ -385,6 +385,27 @@ mod tests { } } + /// improvise-th7m: `F1` was bound twice on consecutive lines — first to a + /// bare `enter-mode` with no args, then to `enter-mode ["help"]`. The + /// first was dead, because `Keymap::bind` *inserts* and the later binding + /// wins; had it ever been the surviving one it would have failed + /// `require_args` at dispatch and made `F1` do nothing. + /// + /// The risk in deleting a line of a duplicated pair is deleting the wrong + /// one, so this pins which binding `F1` is meant to carry: the same one + /// `?` carries, which is what identified it as the intended survivor. + #[test] + fn f1_and_question_mark_both_open_help() { + let ks = KeymapSet::default_keymaps(); + for key in [KeyCode::F(1), KeyCode::Char('?')] { + assert_eq!( + bound(&ks, &ModeKey::Normal, key), + ("enter-mode".to_string(), vec!["help".to_string()]), + "{key:?} should open the help overlay" + ); + } + } + #[test] fn editing_mode_has_any_char_and_esc() { let ks = KeymapSet::default_keymaps(); diff --git a/crates/improvise-protocol/src/keymap.rs b/crates/improvise-protocol/src/keymap.rs index 413d5db1..cc14abb1 100644 --- a/crates/improvise-protocol/src/keymap.rs +++ b/crates/improvise-protocol/src/keymap.rs @@ -570,7 +570,6 @@ impl Keymaps { ); normal.bind(KeyCode::Char('/'), none, "search"); normal.bind(KeyCode::Char('s'), ctrl, "save"); - normal.bind(KeyCode::F(1), none, "enter-mode"); normal.bind_args(KeyCode::F(1), none, "enter-mode", vec!["help".into()]); normal.bind_args(KeyCode::Char('?'), none, "enter-mode", vec!["help".into()]); From 1c2361b45a4fae6e56f423f1aad5b3f248ca89a3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:36:12 -0700 Subject: [PATCH 02/72] fix(persistence): delete bank-info.improv, and pin why it could not be read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-odbc named two readings of the parse failure at `- JORDAN MARTINEZ` and said to decide by asking whether improvise can WRITE that line today. It cannot: `item_list`'s `name` is `pipe_quoted | bare_name`, `bare_name` is `[A-Za-z_][A-Za-z0-9_-]*`, and `quote_name` mirrors it character for character, so a spaced item name is emitted `|JORDAN MARTINEZ|`. So the reader and the writer agree and the round-trip property is not incomplete — the file was stale. It was stale by hand, not by drift. c48a5cd migrated it to v2025-04-09 by editing it in place: the `Date` items were pipe-quoted and the `Entity` ones were not, and every line of its `## Data` section names a category `Measure` where the file declares `_Measure`. It has not been loadable since April, and nothing loaded it, so nothing said so. Deleted rather than regenerated: there is no generator for it — it is hand-written fake bank data — so "regenerating" would mean inventing a canonical form and checking in 1200 lines whose only property is that they parse, which the generator-driven proptests already establish over a far wider input space. What is kept is the evidence and the gap the incident actually exposed: - `an_item_name_the_item_line_cannot_carry_round_trips` puts the file's own item names through `format_md`/`parse_md` and asserts the spaced ones come back quoted and the bare-legal one stays bare. - `a_bare_item_name_with_a_space_is_rejected` asserts the reader really does refuse the unquoted line, with the quoted form parsing as the precondition. Without it the first test would be consistent with the reader accepting both spellings, which is not what happened. - `tests/shipped_documents_load.rs` closes "a sample document exercised by nothing": it walks the two directories `.gitignore` un-ignores and loads every `.improv` in them. A directory walk, not a list of names, because a list is a second place to remember. Proved discriminating by dropping the offending line into `examples/` and watching it fail. Unrelated, reported not fixed: `foo.improv` in the repo root has no version line at all and is equally unloadable. Co-Authored-By: Claude Opus 5 (1M context) --- bank-info.improv | 1186 ----------------- .../src/persistence/roundtrip_tests.rs | 77 ++ tests/shipped_documents_load.rs | 53 + 3 files changed, 130 insertions(+), 1186 deletions(-) delete mode 100644 bank-info.improv create mode 100644 tests/shipped_documents_load.rs diff --git a/bank-info.improv b/bank-info.improv deleted file mode 100644 index 63d6766c..00000000 --- a/bank-info.improv +++ /dev/null @@ -1,1186 +0,0 @@ -v2025-04-09 -# Imported Model - -## Category: Category -- transfers -- investments -- other -- charity -- income -- utilities -- credit-card-payment -- taxes -- dining -- null - -## Category: Date -- |07/18/2025| -- |11/21/2025| -- |05/29/2025| -- |03/03/2025| -- |02/20/2025| -- |03/13/2026| -- |12/22/2025| -- |10/29/2025| -- |01/06/2026| -- |06/02/2025| -- |02/21/2025| -- |02/10/2025| -- |12/16/2024| -- |03/10/2025| -- |02/27/2025| -- |06/03/2025| -- |10/21/2025| -- |03/31/2026| -- |04/30/2025| -- |12/15/2025| -- |12/05/2025| -- |03/06/2026| -- |08/01/2025| -- |11/25/2025| -- |03/16/2026| -- |11/12/2025| -- |03/31/2025| -- |12/29/2025| -- |02/19/2025| -- |08/18/2025| -- |02/18/2026| -- |05/20/2025| -- |03/19/2026| -- |04/25/2025| -- |03/03/2026| -- |12/16/2025| -- |09/09/2025| -- |08/05/2025| -- |08/26/2025| -- |07/08/2025| -- |07/07/2025| -- |03/30/2026| -- |11/18/2025| -- |12/17/2025| -- |09/17/2025| -- |04/16/2025| -- |12/19/2025| -- |01/16/2025| -- |10/08/2025| -- |01/30/2026| -- |09/03/2025| -- |10/01/2025| -- |01/28/2026| -- |02/18/2025| -- |10/17/2025| -- |09/02/2025| -- |10/07/2025| -- |02/28/2025| -- |12/26/2025| -- |11/13/2025| -- |03/24/2026| -- |06/20/2025| -- |03/06/2025| -- |03/12/2025| -- |10/15/2025| -- |12/27/2024| -- |01/02/2025| -- |08/28/2025| -- |10/27/2025| -- |10/20/2025| -- |01/15/2026| -- |05/01/2025| -- |12/10/2024| -- |02/24/2025| -- |03/28/2025| -- |12/05/2024| -- |12/30/2024| -- |06/09/2025| -- |12/03/2024| -- |04/15/2025| -- |11/20/2025| -- |02/11/2026| -- |11/14/2025| -- |03/04/2025| -- |01/15/2025| -- |03/25/2025| -- |02/25/2025| -- |10/16/2025| -- |01/21/2026| -- |12/31/2024| -- |02/20/2026| -- |01/12/2026| -- |03/23/2026| -- |03/18/2025| -- |12/04/2025| -- |07/22/2025| -- |06/24/2025| -- |05/05/2025| -- |11/17/2025| -- |01/29/2025| -- |01/17/2025| -- |04/28/2025| -- |05/19/2025| -- |02/26/2025| -- |12/03/2025| -- |03/05/2026| -- |06/12/2025| -- |08/14/2025| -- |11/10/2025| -- |02/26/2026| -- |06/10/2025| -- |03/24/2025| -- |02/17/2026| -- |02/13/2026| -- |12/31/2025| -- |12/10/2025| -- |11/07/2025| -- |04/02/2025| -- |01/31/2025| -- |02/25/2026| -- |01/28/2025| -- |12/11/2025| -- |07/31/2025| -- |10/24/2025| -- |08/27/2025| -- |02/12/2025| -- |03/18/2026| -- |09/15/2025| -- |04/11/2025| -- |02/05/2025| -- |10/14/2025| -- |03/12/2026| -- |12/12/2025| -- |07/29/2025| -- |12/13/2024| -- |09/11/2025| -- |01/26/2026| -- |09/08/2025| -- |05/23/2025| -- |04/14/2025| -- |01/03/2025| -- |02/13/2025| -- |01/13/2026| -- |01/02/2026| -- |03/25/2026| -- |11/04/2025| -- |04/22/2025| -- |07/14/2025| -- |07/24/2025| -- |06/30/2025| -- |01/13/2025| -- |01/08/2026| -- |12/30/2025| -- |01/07/2025| -- |12/02/2025| -- |04/01/2025| -- |02/27/2026| -- |02/02/2026| -- |01/14/2025| -- |09/30/2025| -- |12/26/2024| -- |06/18/2025| -- |05/06/2025| -- |03/27/2025| -- |11/03/2025| -- |12/09/2024| -- |03/27/2026| -- |08/19/2025| -- |02/03/2026| -- |12/23/2025| -- |12/18/2025| -- |08/25/2025| -- |07/03/2025| -- |06/06/2025| -- |03/17/2025| -- |10/10/2025| -- |03/11/2025| -- |03/04/2026| -- |12/08/2025| -- |03/10/2026| -- |01/27/2026| -- |01/05/2026| -- |06/16/2025| -- |05/13/2025| -- |02/04/2026| -- |03/02/2026| -- |04/17/2025| -- |07/15/2025| -- |06/04/2025| -- |04/21/2025| -- |02/04/2025| -- |10/30/2025| -- |07/17/2025| -- |06/23/2025| -- |07/02/2025| -- |09/10/2025| -- |01/20/2026| -- |09/12/2025| -- |05/09/2025| -- |12/20/2024| -- |02/06/2026| -- |08/04/2025| -- |12/02/2024| -- |07/16/2025| -- |06/05/2025| -- |07/01/2025| -- |09/23/2025| -- |12/17/2024| -- |02/23/2026| -- |02/09/2026| -- |01/22/2026| -- |05/02/2025| -- |01/29/2026| -- |08/29/2025| -- |02/11/2025| -- |02/03/2025| -- |04/08/2025| -- |12/24/2025| -- |03/17/2026| -- |06/17/2025| -- |01/22/2025| -- |03/09/2026| -- |09/22/2025| -- |05/16/2025| -- |05/12/2025| -- |02/19/2026| -- |12/12/2024| -- |09/16/2025| -- |04/24/2025| -- |07/28/2025| -- |10/28/2025| -- |02/05/2026| -- |02/24/2026| -- |01/09/2026| -- |01/14/2026| -- |05/27/2025| -- |03/26/2026| -- |12/06/2024| -- |06/13/2025| -- |03/11/2026| -- |01/23/2025| -- |04/07/2025| -- |10/09/2025| -- |08/12/2025| -- |11/24/2025| -- |02/12/2026| -- |10/31/2025| -- |05/30/2025| -- |09/26/2025| -- |08/11/2025| -- |06/25/2025| -- |11/19/2025| -- |03/14/2025| -- |10/06/2025| -- |12/01/2025| -- |06/26/2025| -- |08/13/2025| -- |05/28/2025| -- |09/29/2025| -- |06/27/2025| -- |12/23/2024| -- |02/10/2026| -- |04/29/2025| -- |01/27/2025| -- |12/09/2025| -- |01/23/2026| -- |12/24/2024| -- |01/07/2026| -- |02/14/2025| -- |08/15/2025| -- |02/06/2025| -- |11/28/2025| -- |01/16/2026| -- |03/20/2026| - -## Category: Entity -- JORDAN MARTINEZ -- Robinhood -- ROBINHOOD -- SANTAPAULATRAIL -- CHECK # 445 -- Holy Trinity Parish -- Kestrel Systems -- SoCalGas -- CHECK # 444 -- MARGARET MARTINEZ -- JPMORGAN CHASE -- ATM -- JM.RIVERA JORDAN MARTINEZ -- Westbrook University -- CHECK # 443 -- Morgan Stanley -- Vanguard -- Chase -- Apple Card Payments -- Apple Card -- AMERICAN EXPRESS -- CHECK # 440 -- HARRIS INC -- AMEX -- TREASURY DIRECT -- SO CAL EDISON CO -- CHECK # 441 -- American Express -- WAY2SAVE SAVINGS -- CHECK # 436 -- CHECK # 439 -- APPLECARD GSBANK PAYMENT -- CHECK # 438 -- CHECK -- Jordan Martinez -- CHECK # 434 -- JPMorgan Chase -- CHECK # 425 -- CHECK # 432 -- XXXXXXX8402 -- Apple Cash -- Infinite Loop -- INFINITELOOP -- Unknown -- MOBILE DEPOSIT -- IRS -- FRANCHISE TAX BO -- null -- Caroline Martinez -- CHECK # 429 -- HORIZON INDEX FUNDS -- Starbucks -- CARDMEMBER SERV WEB PYMT -- Specs #2410 -- CHECK # 428 -- CHECK # 420 -- CITI -- CHECK # 426 -- CHECK # 427 -- AMEX EPAYMENT -- MKWLSNG45 -- MARTINEZ CAROLINE -- MSPBNA -- CHECK # 423 -- ***3715 -- CHECK # 424 -- VENTURACNTYTAXCA -- DEPOSITED OR CASHED CHECK # 414 -- CHECK # 422 -- CHECK # 421 -- CHECK # 419 -- CHECK # 418 -- UNKNOWN -- RSLI -- DIANE HOLLOWAY -- MARTINEZ J WAY2SAVE SAVINGS -- APPLE CASH -- CHECK # 412 -- CHECK # 416 -- ATM WITHDRAWAL -- CHECK # 413 -- APPLECARD -- InfiniteLoop -- CHECK # 411 -- CHECK # 409 -- CR47182906 -- CHECK # 410 -- CITI CARD ONLINE PAYMENT -- CHECK # 408 -- Treasury Direct -- CHECK # 407 -- CHECK # 401 -- APPLE CASH BANK XFER -- CHECK # 403 -- None -- CITI AUTOPAY PAYMENT -- HARRIS EMILY -- VENMO -- CHECK # 496 -- Unknown Entity -- Credit Card Payment -- CHECK # 495 -- SO CAL GAS -- BookDepot -- NextDNS -- SARAH PETERSON -- CHECK # 494 -- PETERSON SARAH -- CHECK # 493 -- CHECK # 492 -- "Citi -- CHECK # 491 -- CHECK # 490 -- Cardmember Service Web Payment -- CHECK # 489 -- FRANCHISE TAX BO PAYMENTS -- LINDA REYNOLDS -- CHECK # 487 -- CHECK # 488 -- CHECK # 486 -- CHECK # 485 -- ST OF CA DMV -- CHECK # 484 -- CHECK # 483 -- CHECK # 482 -- ZELLE TO M THOMAS -- CHECK # 481 -- CHECK # 480 -- CHECK # 477 -- VIRAGE -- Cardmember Serv -- CHECK # 479 -- CHECK # 478 -- CHECK # 675 -- APA TREAS -- BLUE RIDGE SUPPLY -- MERIDIAN GOODS LLC -- CHECK # 673 -- CHECK # 672 -- CHECK # 670 -- CHECK # 671 -- INTEREST PAYMENT -- EVERYDAY CHECKING -- SAVE AS YOU GO -- transfer to new account # ending in 5917 - -## Category: File -- checking-merged -- savings-merged - -## Category: Date_Year -- 2026 -- 2025 -- 2024 - -## Category: Date_Month -- 2026-03 -- 2026-02 -- 2026-01 -- 2025-12 -- 2025-11 -- 2025-10 -- 2025-09 -- 2025-08 -- 2025-07 -- 2025-06 -- 2025-05 -- 2025-04 -- 2025-03 -- 2025-02 -- 2025-01 -- 2024-12 - -## Category: Date_Quarter -- 2026-Q1 -- 2025-Q4 -- 2025-Q3 -- 2025-Q2 -- 2025-Q1 -- 2024-Q4 - -## Category: _Measure -- Amount - -## Data -Category=charity, Date=01/17/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -271 -Category=charity, Date=01/20/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Westbrook University, File=checking-merged, Measure=Amount = -179 -Category=charity, Date=01/27/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -433 -Category=charity, Date=01/28/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -417 -Category=charity, Date=02/17/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Westbrook University, File=checking-merged, Measure=Amount = -285 -Category=charity, Date=02/18/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -276 -Category=charity, Date=02/27/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -618 -Category=charity, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -376 -Category=charity, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -1430 -Category=charity, Date=03/17/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -179 -Category=charity, Date=03/17/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Westbrook University, File=checking-merged, Measure=Amount = -208 -Category=charity, Date=03/27/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -502 -Category=charity, Date=03/27/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -358 -Category=charity, Date=04/17/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -205 -Category=charity, Date=04/29/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -545 -Category=charity, Date=05/19/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -257 -Category=charity, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -416 -Category=charity, Date=06/17/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -263 -Category=charity, Date=06/27/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -593 -Category=charity, Date=07/17/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -176 -Category=charity, Date=07/29/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -592 -Category=charity, Date=08/18/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -280 -Category=charity, Date=08/27/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -452 -Category=charity, Date=09/17/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -198 -Category=charity, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -637 -Category=charity, Date=10/17/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -225 -Category=charity, Date=10/28/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -378 -Category=charity, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -190 -Category=charity, Date=11/28/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -604 -Category=charity, Date=12/17/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Westbrook University, File=checking-merged, Measure=Amount = -266 -Category=charity, Date=12/17/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -296 -Category=charity, Date=12/26/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -1138 -Category=charity, Date=12/26/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Westbrook University, File=checking-merged, Measure=Amount = -1027 -Category=charity, Date=12/27/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -642 -Category=charity, Date=12/29/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Holy Trinity Parish, File=checking-merged, Measure=Amount = -464 -Category=credit-card-payment, Date=01/02/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -961.8700 -Category=credit-card-payment, Date=01/02/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -778.2900 -Category=credit-card-payment, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=American Express, File=checking-merged, Measure=Amount = -7773.0800 -Category=credit-card-payment, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -2966.1800 -Category=credit-card-payment, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Chase, File=checking-merged, Measure=Amount = -809.8600 -Category=credit-card-payment, Date=01/07/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -15070.9100 -Category=credit-card-payment, Date=01/12/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -605.2500 -Category=credit-card-payment, Date=01/13/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -29 -Category=credit-card-payment, Date=01/13/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1462.4300 -Category=credit-card-payment, Date=01/13/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -175.0700 -Category=credit-card-payment, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -196.5600 -Category=credit-card-payment, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -632.8000 -Category=credit-card-payment, Date=01/27/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -1450.9700 -Category=credit-card-payment, Date=01/29/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Cardmember Serv, File=checking-merged, Measure=Amount = -11.4500 -Category=credit-card-payment, Date=01/29/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -11763.5700 -Category=credit-card-payment, Date=01/29/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -11.7200 -Category=credit-card-payment, Date=02/06/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=AMEX, File=checking-merged, Measure=Amount = -1201.8500 -Category=credit-card-payment, Date=02/09/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -715.6400 -Category=credit-card-payment, Date=02/10/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -2112.5300 -Category=credit-card-payment, Date=02/12/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -27 -Category=credit-card-payment, Date=02/12/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -37 -Category=credit-card-payment, Date=02/12/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Chase, File=checking-merged, Measure=Amount = -158.1200 -Category=credit-card-payment, Date=02/13/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -393.4400 -Category=credit-card-payment, Date=02/13/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -334.9000 -Category=credit-card-payment, Date=02/19/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -1919.1100 -Category=credit-card-payment, Date=02/20/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -2587 -Category=credit-card-payment, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -901.8600 -Category=credit-card-payment, Date=02/24/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -357.7100 -Category=credit-card-payment, Date=02/24/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -1859.3900 -Category=credit-card-payment, Date=02/24/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -505.9900 -Category=credit-card-payment, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=American Express, File=checking-merged, Measure=Amount = -1537.6100 -Category=credit-card-payment, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Chase, File=checking-merged, Measure=Amount = -1356.0700 -Category=credit-card-payment, Date=03/03/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CARDMEMBER SERV WEB PYMT, File=checking-merged, Measure=Amount = -7.6200 -Category=credit-card-payment, Date=03/06/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=AMEX, File=checking-merged, Measure=Amount = -1384.3500 -Category=credit-card-payment, Date=03/06/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=AMEX, File=checking-merged, Measure=Amount = -2145.6700 -Category=credit-card-payment, Date=03/09/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -687.6900 -Category=credit-card-payment, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -6275.8800 -Category=credit-card-payment, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -1195.0800 -Category=credit-card-payment, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -632.5900 -Category=credit-card-payment, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -6414.1600 -Category=credit-card-payment, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1569.1400 -Category=credit-card-payment, Date=03/10/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -6337.5700 -Category=credit-card-payment, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -2723.8700 -Category=credit-card-payment, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Card, File=checking-merged, Measure=Amount = -1302.4800 -Category=credit-card-payment, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Chase, File=checking-merged, Measure=Amount = -2841.0400 -Category=credit-card-payment, Date=03/31/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Credit Card Payment, File=checking-merged, Measure=Amount = -10.9800 -Category=credit-card-payment, Date=04/07/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=AMEX EPAYMENT, File=checking-merged, Measure=Amount = -1252.5300 -Category=credit-card-payment, Date=04/07/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1758.9200 -Category=credit-card-payment, Date=04/11/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -1.3600 -Category=credit-card-payment, Date=04/14/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -23 -Category=credit-card-payment, Date=04/14/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -36 -Category=credit-card-payment, Date=04/14/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -413.9400 -Category=credit-card-payment, Date=04/21/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1683.5400 -Category=credit-card-payment, Date=04/24/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -240.5400 -Category=credit-card-payment, Date=04/28/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -2048.6000 -Category=credit-card-payment, Date=04/28/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -450.1800 -Category=credit-card-payment, Date=04/28/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -1518.5300 -Category=credit-card-payment, Date=04/29/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Cardmember Service Web Payment, File=checking-merged, Measure=Amount = -12.4500 -Category=credit-card-payment, Date=05/05/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -1595.7800 -Category=credit-card-payment, Date=05/12/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=AMEX, File=checking-merged, Measure=Amount = -111 -Category=credit-card-payment, Date=05/12/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -23 -Category=credit-card-payment, Date=05/13/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -712.3600 -Category=credit-card-payment, Date=05/19/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1703.2100 -Category=credit-card-payment, Date=05/27/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity="Citi, File=checking-merged, Measure=Amount = -704.1100 -Category=credit-card-payment, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -2515.4500 -Category=credit-card-payment, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -1859.8300 -Category=credit-card-payment, Date=05/29/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -21401.1500 -Category=credit-card-payment, Date=05/29/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=null, File=checking-merged, Measure=Amount = -11.7400 -Category=credit-card-payment, Date=06/02/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1711.0900 -Category=credit-card-payment, Date=06/12/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -22 -Category=credit-card-payment, Date=06/13/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -829.3900 -Category=credit-card-payment, Date=06/13/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CITI CARD ONLINE PAYMENT, File=checking-merged, Measure=Amount = -903.6100 -Category=credit-card-payment, Date=06/13/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -753.6800 -Category=credit-card-payment, Date=06/13/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -406.1100 -Category=credit-card-payment, Date=06/16/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -2129.7600 -Category=credit-card-payment, Date=06/25/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -12825.3000 -Category=credit-card-payment, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Credit Card Payment, File=checking-merged, Measure=Amount = -9.0900 -Category=credit-card-payment, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -1674.5600 -Category=credit-card-payment, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -20 -Category=credit-card-payment, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -2319.3100 -Category=credit-card-payment, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -731.3500 -Category=credit-card-payment, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -162.3700 -Category=credit-card-payment, Date=07/24/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CITI AUTOPAY PAYMENT, File=checking-merged, Measure=Amount = -143.9300 -Category=credit-card-payment, Date=07/28/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -2131 -Category=credit-card-payment, Date=07/29/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -178 -Category=credit-card-payment, Date=07/29/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=None, File=checking-merged, Measure=Amount = -12.2600 -Category=credit-card-payment, Date=08/12/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -64.3600 -Category=credit-card-payment, Date=08/12/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -29 -Category=credit-card-payment, Date=08/13/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -2310.0900 -Category=credit-card-payment, Date=08/13/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -245.6500 -Category=credit-card-payment, Date=08/14/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -137.5500 -Category=credit-card-payment, Date=08/15/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -481.2300 -Category=credit-card-payment, Date=08/15/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -1714.2800 -Category=credit-card-payment, Date=08/15/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -1090.5600 -Category=credit-card-payment, Date=08/25/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -997.5300 -Category=credit-card-payment, Date=08/29/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -11346.2400 -Category=credit-card-payment, Date=08/29/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -438.4200 -Category=credit-card-payment, Date=08/29/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -12.9900 -Category=credit-card-payment, Date=09/02/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card, File=checking-merged, Measure=Amount = -3871.0100 -Category=credit-card-payment, Date=09/08/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -625.1300 -Category=credit-card-payment, Date=09/11/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -17537.2500 -Category=credit-card-payment, Date=09/11/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CITI CARD ONLINE PAYMENT, File=checking-merged, Measure=Amount = -235.8700 -Category=credit-card-payment, Date=09/12/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -716.9000 -Category=credit-card-payment, Date=09/22/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -964.6700 -Category=credit-card-payment, Date=09/26/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -4730.2000 -Category=credit-card-payment, Date=09/26/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -2341.4200 -Category=credit-card-payment, Date=09/26/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -719.8500 -Category=credit-card-payment, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=APPLECARD, File=checking-merged, Measure=Amount = -9.1000 -Category=credit-card-payment, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -837.3500 -Category=credit-card-payment, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -793.5900 -Category=credit-card-payment, Date=10/14/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -21 -Category=credit-card-payment, Date=10/14/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -67.3700 -Category=credit-card-payment, Date=10/20/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -6828 -Category=credit-card-payment, Date=10/20/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -682.7900 -Category=credit-card-payment, Date=10/21/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -645.1300 -Category=credit-card-payment, Date=10/21/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -1390.7700 -Category=credit-card-payment, Date=10/24/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -5359.3000 -Category=credit-card-payment, Date=10/27/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -78.6100 -Category=credit-card-payment, Date=10/29/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CARDMEMBER SERV WEB PYMT, File=checking-merged, Measure=Amount = -10.7000 -Category=credit-card-payment, Date=11/03/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -642.7700 -Category=credit-card-payment, Date=11/12/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -21 -Category=credit-card-payment, Date=11/13/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -56.3200 -Category=credit-card-payment, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -451.9300 -Category=credit-card-payment, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -1729.5800 -Category=credit-card-payment, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -617.8200 -Category=credit-card-payment, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -906.9700 -Category=credit-card-payment, Date=12/01/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=***3715, File=checking-merged, Measure=Amount = -13.7100 -Category=credit-card-payment, Date=12/01/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -696.6500 -Category=credit-card-payment, Date=12/02/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -2081.1500 -Category=credit-card-payment, Date=12/06/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=American Express, File=checking-merged, Measure=Amount = -1042.8000 -Category=credit-card-payment, Date=12/08/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=AMEX EPAYMENT, File=checking-merged, Measure=Amount = -471.3700 -Category=credit-card-payment, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Apple Card, File=checking-merged, Measure=Amount = -54 -Category=credit-card-payment, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Chase, File=checking-merged, Measure=Amount = -39 -Category=credit-card-payment, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -0.9100 -Category=credit-card-payment, Date=12/12/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -116.7700 -Category=credit-card-payment, Date=12/12/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Chase, File=checking-merged, Measure=Amount = -11.7300 -Category=credit-card-payment, Date=12/13/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -13377.1200 -Category=credit-card-payment, Date=12/13/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=American Express, File=checking-merged, Measure=Amount = -433.5000 -Category=credit-card-payment, Date=12/13/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -355.0400 -Category=credit-card-payment, Date=12/13/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -949.6100 -Category=credit-card-payment, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=American Express, File=checking-merged, Measure=Amount = -6582.2800 -Category=credit-card-payment, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CITI, File=checking-merged, Measure=Amount = -198.7400 -Category=credit-card-payment, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -797.5400 -Category=credit-card-payment, Date=12/16/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -5957.7900 -Category=credit-card-payment, Date=12/16/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Chase, File=checking-merged, Measure=Amount = -3636.9700 -Category=credit-card-payment, Date=12/16/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -2355.4000 -Category=credit-card-payment, Date=12/24/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Apple Card Payments, File=checking-merged, Measure=Amount = -289.4100 -Category=credit-card-payment, Date=12/29/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CARDMEMBER SERV WEB PYMT, File=checking-merged, Measure=Amount = -16.4000 -Category=credit-card-payment, Date=12/29/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -631.5100 -Category=credit-card-payment, Date=12/30/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=AMERICAN EXPRESS, File=checking-merged, Measure=Amount = -7533 -Category=credit-card-payment, Date=12/30/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Credit Card Payment, File=checking-merged, Measure=Amount = -9.5700 -Category=credit-card-payment, Date=12/30/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -2037.2300 -Category=dining, Date=04/28/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -8 -Category=dining, Date=05/27/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -8 -Category=dining, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -10 -Category=dining, Date=08/11/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -10 -Category=dining, Date=08/13/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -9 -Category=dining, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -8 -Category=dining, Date=12/22/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -13 -Category=dining, Date=12/23/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=MERIDIAN GOODS LLC, File=checking-merged, Measure=Amount = -18 -Category=dining, Date=12/29/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Starbucks, File=checking-merged, Measure=Amount = -12 -Category=dining, Date=12/30/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Starbucks, File=checking-merged, Measure=Amount = -10 -Category=income, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 2730.9400 -Category=income, Date=01/03/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4570.2200 -Category=income, Date=01/15/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=APA TREAS, File=checking-merged, Measure=Amount = 0.2000 -Category=income, Date=01/16/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4791.4000 -Category=income, Date=01/17/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4416.3400 -Category=income, Date=01/20/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=MOBILE DEPOSIT, File=checking-merged, Measure=Amount = 302 -Category=income, Date=01/30/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.3000 -Category=income, Date=01/30/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3919.1300 -Category=income, Date=01/31/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.5600 -Category=income, Date=01/31/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 2726.4000 -Category=income, Date=01/31/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=MOBILE DEPOSIT, File=checking-merged, Measure=Amount = 235 -Category=income, Date=02/13/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3467.6800 -Category=income, Date=02/14/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4540.7400 -Category=income, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.2800 -Category=income, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4497.7300 -Category=income, Date=02/28/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.5900 -Category=income, Date=02/28/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3422.1800 -Category=income, Date=03/10/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 26125.0800 -Category=income, Date=03/10/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 32930.2500 -Category=income, Date=03/13/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3864.3300 -Category=income, Date=03/14/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3977.5500 -Category=income, Date=03/27/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 11889.6500 -Category=income, Date=03/28/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3087.7600 -Category=income, Date=03/31/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.8700 -Category=income, Date=03/31/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.3400 -Category=income, Date=04/11/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3605.7400 -Category=income, Date=04/25/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4275.0900 -Category=income, Date=04/30/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.4500 -Category=income, Date=05/09/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3909.3800 -Category=income, Date=05/23/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3802.5500 -Category=income, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=MOBILE DEPOSIT, File=checking-merged, Measure=Amount = 242 -Category=income, Date=05/30/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.4800 -Category=income, Date=06/06/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4962.7100 -Category=income, Date=06/10/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 18989.3700 -Category=income, Date=06/20/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3154.9800 -Category=income, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.7200 -Category=income, Date=07/03/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4296.5700 -Category=income, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=MOBILE DEPOSIT, File=checking-merged, Measure=Amount = 60.1600 -Category=income, Date=07/18/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3001.1600 -Category=income, Date=07/31/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.6300 -Category=income, Date=08/01/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3295.8400 -Category=income, Date=08/15/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5251.6100 -Category=income, Date=08/29/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.5300 -Category=income, Date=08/29/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5497.8500 -Category=income, Date=09/10/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 26658.9800 -Category=income, Date=09/12/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5424.4300 -Category=income, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CR47182906, File=checking-merged, Measure=Amount = 7.2500 -Category=income, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -165 -Category=income, Date=09/26/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 6040.0700 -Category=income, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=MOBILE DEPOSIT, File=checking-merged, Measure=Amount = 157.7700 -Category=income, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.4500 -Category=income, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 110 -Category=income, Date=10/10/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 4704.9400 -Category=income, Date=10/16/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=RSLI, File=checking-merged, Measure=Amount = 1512 -Category=income, Date=10/24/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 6154.4900 -Category=income, Date=10/31/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.3300 -Category=income, Date=11/07/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5212.7200 -Category=income, Date=11/21/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 8167.5200 -Category=income, Date=11/28/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.5100 -Category=income, Date=12/05/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3322.3800 -Category=income, Date=12/05/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 3618.3500 -Category=income, Date=12/10/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 16448.2600 -Category=income, Date=12/10/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=HARRIS INC, File=checking-merged, Measure=Amount = 31983.4600 -Category=income, Date=12/19/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5239.1500 -Category=income, Date=12/20/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Kestrel Systems, File=checking-merged, Measure=Amount = 5228.9600 -Category=income, Date=12/31/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.6100 -Category=income, Date=12/31/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=INTEREST PAYMENT, File=savings-merged, Measure=Amount = 0.2900 -Category=investments, Date=01/02/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=HORIZON INDEX FUNDS, File=checking-merged, Measure=Amount = 15118.1800 -Category=investments, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=HORIZON INDEX FUNDS, File=checking-merged, Measure=Amount = 9825.5600 -Category=investments, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -43 -Category=investments, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -258 -Category=investments, Date=01/05/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -109 -Category=investments, Date=01/06/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -35 -Category=investments, Date=01/06/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -119 -Category=investments, Date=01/07/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -211 -Category=investments, Date=01/07/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -110 -Category=investments, Date=01/08/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -51 -Category=investments, Date=01/09/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -47 -Category=investments, Date=01/12/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -77 -Category=investments, Date=01/13/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -38 -Category=investments, Date=01/13/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -103 -Category=investments, Date=01/14/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -207 -Category=investments, Date=01/14/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -106 -Category=investments, Date=01/15/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -45 -Category=investments, Date=01/16/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -164 -Category=investments, Date=01/16/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -65 -Category=investments, Date=01/20/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -52 -Category=investments, Date=01/21/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -50 -Category=investments, Date=01/21/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -124 -Category=investments, Date=01/22/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -290 -Category=investments, Date=01/22/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -30 -Category=investments, Date=01/23/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -57 -Category=investments, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -87 -Category=investments, Date=01/27/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -35 -Category=investments, Date=01/27/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -116 -Category=investments, Date=01/28/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -260 -Category=investments, Date=01/28/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -86 -Category=investments, Date=01/29/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -114 -Category=investments, Date=01/30/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -103 -Category=investments, Date=02/02/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -96 -Category=investments, Date=02/02/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -141 -Category=investments, Date=02/03/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -149 -Category=investments, Date=02/03/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -61 -Category=investments, Date=02/03/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -124 -Category=investments, Date=02/04/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -247 -Category=investments, Date=02/04/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -120 -Category=investments, Date=02/05/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -105 -Category=investments, Date=02/06/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -79 -Category=investments, Date=02/09/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -78 -Category=investments, Date=02/10/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -44 -Category=investments, Date=02/10/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -124 -Category=investments, Date=02/11/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -283 -Category=investments, Date=02/11/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -122 -Category=investments, Date=02/12/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -124 -Category=investments, Date=02/13/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -83 -Category=investments, Date=02/17/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -85 -Category=investments, Date=02/18/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -152 -Category=investments, Date=02/18/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -58 -Category=investments, Date=02/18/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -123 -Category=investments, Date=02/19/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -227 -Category=investments, Date=02/19/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -107 -Category=investments, Date=02/20/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -32 -Category=investments, Date=02/20/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -126 -Category=investments, Date=02/21/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -146 -Category=investments, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -129 -Category=investments, Date=02/24/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -59 -Category=investments, Date=02/24/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -61 -Category=investments, Date=02/24/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -71 -Category=investments, Date=02/25/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=02/25/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -90 -Category=investments, Date=02/26/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -126 -Category=investments, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -118 -Category=investments, Date=02/28/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -119 -Category=investments, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -172 -Category=investments, Date=03/03/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -234 -Category=investments, Date=03/03/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -153 -Category=investments, Date=03/03/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -61 -Category=investments, Date=03/03/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -122 -Category=investments, Date=03/04/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -200 -Category=investments, Date=03/04/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -119 -Category=investments, Date=03/05/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -98 -Category=investments, Date=03/06/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -88 -Category=investments, Date=03/09/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -118 -Category=investments, Date=03/10/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -42 -Category=investments, Date=03/10/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -71 -Category=investments, Date=03/11/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -196 -Category=investments, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -90 -Category=investments, Date=03/12/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -122 -Category=investments, Date=03/13/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -128 -Category=investments, Date=03/16/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -87 -Category=investments, Date=03/16/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Vanguard, File=checking-merged, Measure=Amount = -174 -Category=investments, Date=03/17/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -188 -Category=investments, Date=03/17/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -64 -Category=investments, Date=03/17/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -102 -Category=investments, Date=03/18/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -303 -Category=investments, Date=03/18/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -77 -Category=investments, Date=03/19/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -128 -Category=investments, Date=03/20/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -81 -Category=investments, Date=03/23/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -128 -Category=investments, Date=03/24/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -43 -Category=investments, Date=03/24/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -76 -Category=investments, Date=03/25/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -231 -Category=investments, Date=03/25/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -114 -Category=investments, Date=03/26/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -89 -Category=investments, Date=03/27/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -106 -Category=investments, Date=03/30/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -101 -Category=investments, Date=03/31/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -47 -Category=investments, Date=03/31/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Robinhood, File=checking-merged, Measure=Amount = -105 -Category=investments, Date=04/01/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -205 -Category=investments, Date=04/01/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -225 -Category=investments, Date=04/02/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -351 -Category=investments, Date=04/08/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -301 -Category=investments, Date=04/08/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Vanguard, File=checking-merged, Measure=Amount = -1 -Category=investments, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -272 -Category=investments, Date=04/16/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -229 -Category=investments, Date=04/22/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -265 -Category=investments, Date=04/29/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = 459 -Category=investments, Date=04/29/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -178 -Category=investments, Date=05/01/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -220 -Category=investments, Date=05/06/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -216 -Category=investments, Date=05/13/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -213 -Category=investments, Date=05/16/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -242 -Category=investments, Date=05/20/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -272 -Category=investments, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = 4525.6500 -Category=investments, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -213 -Category=investments, Date=06/02/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -189 -Category=investments, Date=06/03/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -226 -Category=investments, Date=06/10/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -211 -Category=investments, Date=06/17/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -186 -Category=investments, Date=06/24/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -229 -Category=investments, Date=07/01/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -303 -Category=investments, Date=07/02/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -553 -Category=investments, Date=07/08/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -298 -Category=investments, Date=07/15/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -257 -Category=investments, Date=07/16/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -176 -Category=investments, Date=07/22/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -247 -Category=investments, Date=07/29/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -168 -Category=investments, Date=08/01/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -174 -Category=investments, Date=08/04/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -479 -Category=investments, Date=08/05/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -252 -Category=investments, Date=08/12/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -262 -Category=investments, Date=08/13/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = 1468 -Category=investments, Date=08/18/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -193 -Category=investments, Date=08/19/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -199 -Category=investments, Date=08/26/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -236 -Category=investments, Date=08/28/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = 11153.8000 -Category=investments, Date=09/02/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Treasury Direct, File=checking-merged, Measure=Amount = -236 -Category=investments, Date=09/03/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -192 -Category=investments, Date=09/09/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -180 -Category=investments, Date=09/15/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -1009 -Category=investments, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -54 -Category=investments, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -216 -Category=investments, Date=09/23/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -60 -Category=investments, Date=09/23/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -276 -Category=investments, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -55 -Category=investments, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -200 -Category=investments, Date=10/01/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -164 -Category=investments, Date=10/07/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -36 -Category=investments, Date=10/07/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -203 -Category=investments, Date=10/08/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = 1773 -Category=investments, Date=10/15/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -60 -Category=investments, Date=10/15/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -26 -Category=investments, Date=10/16/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -190 -Category=investments, Date=10/20/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -2156 -Category=investments, Date=10/21/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -41 -Category=investments, Date=10/21/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -268 -Category=investments, Date=10/28/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -50 -Category=investments, Date=10/28/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -203 -Category=investments, Date=11/03/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -219 -Category=investments, Date=11/04/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -35 -Category=investments, Date=11/04/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -276 -Category=investments, Date=11/10/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -70 -Category=investments, Date=11/12/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -38 -Category=investments, Date=11/12/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -229 -Category=investments, Date=11/13/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -32 -Category=investments, Date=11/14/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -76 -Category=investments, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -111 -Category=investments, Date=11/18/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -37 -Category=investments, Date=11/18/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -204 -Category=investments, Date=11/19/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -2418 -Category=investments, Date=11/20/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -39 -Category=investments, Date=11/21/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -71 -Category=investments, Date=11/24/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -77 -Category=investments, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -65 -Category=investments, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -5 -Category=investments, Date=11/28/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=12/01/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -222 -Category=investments, Date=12/01/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -214 -Category=investments, Date=12/02/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=12/02/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -201 -Category=investments, Date=12/02/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -60 -Category=investments, Date=12/02/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -5 -Category=investments, Date=12/03/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=12/03/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=MSPBNA, File=checking-merged, Measure=Amount = 11986 -Category=investments, Date=12/04/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -39 -Category=investments, Date=12/05/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -51 -Category=investments, Date=12/08/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -51 -Category=investments, Date=12/09/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -54 -Category=investments, Date=12/09/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -6 -Category=investments, Date=12/10/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -243 -Category=investments, Date=12/10/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -5 -Category=investments, Date=12/11/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -4 -Category=investments, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -30 -Category=investments, Date=12/12/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -52 -Category=investments, Date=12/13/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -104 -Category=investments, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -89 -Category=investments, Date=12/16/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -3072 -Category=investments, Date=12/16/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -157 -Category=investments, Date=12/16/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -42 -Category=investments, Date=12/16/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -112 -Category=investments, Date=12/17/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -270 -Category=investments, Date=12/17/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -74 -Category=investments, Date=12/18/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -38 -Category=investments, Date=12/19/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -62 -Category=investments, Date=12/22/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -57 -Category=investments, Date=12/23/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -41 -Category=investments, Date=12/23/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -95 -Category=investments, Date=12/24/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -298 -Category=investments, Date=12/24/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -105 -Category=investments, Date=12/26/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -112 -Category=investments, Date=12/29/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -121 -Category=investments, Date=12/30/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ROBINHOOD, File=checking-merged, Measure=Amount = -58 -Category=investments, Date=12/30/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -93 -Category=investments, Date=12/31/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Robinhood, File=checking-merged, Measure=Amount = -169 -Category=investments, Date=12/31/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Robinhood, File=checking-merged, Measure=Amount = -91 -Category=null, Date=03/11/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=null, File=savings-merged, Measure=Amount = -2304 -Category=null, Date=03/31/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=null, File=savings-merged, Measure=Amount = -1.3300 -Category=null, Date=05/28/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=null, File=savings-merged, Measure=Amount = 3816 -Category=null, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=null, File=savings-merged, Measure=Amount = -5479.7900 -Category=null, Date=08/13/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=null, File=savings-merged, Measure=Amount = 1723 -Category=other, Date=01/02/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -206 -Category=other, Date=01/05/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 429, File=checking-merged, Measure=Amount = -53 -Category=other, Date=01/12/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Unknown, File=checking-merged, Measure=Amount = -42 -Category=other, Date=01/12/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=null, File=checking-merged, Measure=Amount = -42 -Category=other, Date=01/15/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 675, File=checking-merged, Measure=Amount = -48 -Category=other, Date=01/20/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Unknown, File=checking-merged, Measure=Amount = -36 -Category=other, Date=01/23/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=ATM WITHDRAWAL, File=checking-merged, Measure=Amount = -180 -Category=other, Date=01/23/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 478, File=checking-merged, Measure=Amount = -55 -Category=other, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 425, File=checking-merged, Measure=Amount = -306 -Category=other, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 432, File=checking-merged, Measure=Amount = -260 -Category=other, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 434, File=checking-merged, Measure=Amount = -49 -Category=other, Date=01/28/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 479, File=checking-merged, Measure=Amount = -39 -Category=other, Date=01/31/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=VIRAGE, File=checking-merged, Measure=Amount = -316 -Category=other, Date=02/02/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK, File=checking-merged, Measure=Amount = -36 -Category=other, Date=02/05/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 477, File=checking-merged, Measure=Amount = -688 -Category=other, Date=02/05/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 438, File=checking-merged, Measure=Amount = -52 -Category=other, Date=02/06/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 480, File=checking-merged, Measure=Amount = -36 -Category=other, Date=02/10/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 481, File=checking-merged, Measure=Amount = -39 -Category=other, Date=02/11/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=ZELLE TO M THOMAS, File=checking-merged, Measure=Amount = -220 -Category=other, Date=02/12/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=APPLECARD GSBANK PAYMENT, File=checking-merged, Measure=Amount = -857.0200 -Category=other, Date=02/18/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 482, File=checking-merged, Measure=Amount = -46 -Category=other, Date=02/18/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 439, File=checking-merged, Measure=Amount = -41 -Category=other, Date=02/19/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 483, File=checking-merged, Measure=Amount = -659 -Category=other, Date=02/19/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 436, File=checking-merged, Measure=Amount = -1680 -Category=other, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -186 -Category=other, Date=02/24/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 484, File=checking-merged, Measure=Amount = -58 -Category=other, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 441, File=checking-merged, Measure=Amount = -277 -Category=other, Date=03/04/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=ST OF CA DMV, File=checking-merged, Measure=Amount = -399 -Category=other, Date=03/10/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 440, File=checking-merged, Measure=Amount = -36 -Category=other, Date=03/12/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 485, File=checking-merged, Measure=Amount = -51 -Category=other, Date=03/16/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 443, File=checking-merged, Measure=Amount = -65 -Category=other, Date=03/17/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 486, File=checking-merged, Measure=Amount = -45 -Category=other, Date=03/23/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=ATM, File=checking-merged, Measure=Amount = -436 -Category=other, Date=03/23/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 444, File=checking-merged, Measure=Amount = -58 -Category=other, Date=03/30/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=CHECK # 445, File=checking-merged, Measure=Amount = -55 -Category=other, Date=03/30/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SANTAPAULATRAIL, File=checking-merged, Measure=Amount = -115 -Category=other, Date=03/31/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=CHECK # 488, File=checking-merged, Measure=Amount = -63 -Category=other, Date=04/02/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 487, File=checking-merged, Measure=Amount = -602 -Category=other, Date=04/16/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 489, File=checking-merged, Measure=Amount = -36 -Category=other, Date=05/02/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 490, File=checking-merged, Measure=Amount = -582 -Category=other, Date=05/20/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 491, File=checking-merged, Measure=Amount = -39 -Category=other, Date=05/27/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 492, File=checking-merged, Measure=Amount = -55 -Category=other, Date=06/04/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=ATM, File=checking-merged, Measure=Amount = -312 -Category=other, Date=06/05/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 493, File=checking-merged, Measure=Amount = -42 -Category=other, Date=06/09/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 494, File=checking-merged, Measure=Amount = -56 -Category=other, Date=06/09/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=PETERSON SARAH, File=checking-merged, Measure=Amount = -70 -Category=other, Date=06/12/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SARAH PETERSON, File=checking-merged, Measure=Amount = -88 -Category=other, Date=06/13/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=NextDNS, File=checking-merged, Measure=Amount = -21.1800 -Category=other, Date=06/16/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -230 -Category=other, Date=06/18/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=BookDepot, File=checking-merged, Measure=Amount = -198.8300 -Category=other, Date=06/23/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -298 -Category=other, Date=06/26/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK # 495, File=checking-merged, Measure=Amount = -64 -Category=other, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=CHECK, File=checking-merged, Measure=Amount = -38 -Category=other, Date=07/03/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Unknown Entity, File=checking-merged, Measure=Amount = -121.5200 -Category=other, Date=07/07/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 496, File=checking-merged, Measure=Amount = -44 -Category=other, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=HARRIS EMILY, File=checking-merged, Measure=Amount = -420 -Category=other, Date=07/28/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Unknown, File=checking-merged, Measure=Amount = -64 -Category=other, Date=08/04/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Unknown, File=checking-merged, Measure=Amount = -47 -Category=other, Date=08/11/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 403, File=checking-merged, Measure=Amount = -56 -Category=other, Date=08/18/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 401, File=checking-merged, Measure=Amount = -936 -Category=other, Date=08/27/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 407, File=checking-merged, Measure=Amount = -474 -Category=other, Date=09/03/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 408, File=checking-merged, Measure=Amount = -54 -Category=other, Date=09/15/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 410, File=checking-merged, Measure=Amount = -38 -Category=other, Date=09/16/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CR47182906, File=checking-merged, Measure=Amount = 0.0100 -Category=other, Date=09/22/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 409, File=checking-merged, Measure=Amount = -61 -Category=other, Date=09/22/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 411, File=checking-merged, Measure=Amount = -53 -Category=other, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=InfiniteLoop, File=checking-merged, Measure=Amount = -39 -Category=other, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=CHECK # 413, File=checking-merged, Measure=Amount = -65 -Category=other, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=ATM WITHDRAWAL, File=checking-merged, Measure=Amount = -468 -Category=other, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 416, File=checking-merged, Measure=Amount = -45 -Category=other, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK, File=checking-merged, Measure=Amount = -105 -Category=other, Date=10/07/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 412, File=checking-merged, Measure=Amount = -300 -Category=other, Date=10/07/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = 155 -Category=other, Date=10/14/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=DIANE HOLLOWAY, File=checking-merged, Measure=Amount = -20 -Category=other, Date=10/16/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=null, File=checking-merged, Measure=Amount = -60 -Category=other, Date=10/17/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=RSLI, File=checking-merged, Measure=Amount = 35802 -Category=other, Date=10/28/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=UNKNOWN, File=checking-merged, Measure=Amount = -38 -Category=other, Date=10/30/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 418, File=checking-merged, Measure=Amount = -415 -Category=other, Date=11/03/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 419, File=checking-merged, Measure=Amount = -54 -Category=other, Date=11/10/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 421, File=checking-merged, Measure=Amount = -60 -Category=other, Date=11/17/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 422, File=checking-merged, Measure=Amount = -56 -Category=other, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 424, File=checking-merged, Measure=Amount = -48 -Category=other, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=DEPOSITED OR CASHED CHECK # 414, File=checking-merged, Measure=Amount = -34 -Category=other, Date=12/01/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 423, File=checking-merged, Measure=Amount = -64 -Category=other, Date=12/02/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=CHECK # 671, File=checking-merged, Measure=Amount = -43 -Category=other, Date=12/08/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 427, File=checking-merged, Measure=Amount = -59 -Category=other, Date=12/08/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=MARTINEZ CAROLINE, File=checking-merged, Measure=Amount = -594 -Category=other, Date=12/08/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=MKWLSNG45, File=checking-merged, Measure=Amount = -31 -Category=other, Date=12/09/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=CHECK # 670, File=checking-merged, Measure=Amount = -904 -Category=other, Date=12/10/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 426, File=checking-merged, Measure=Amount = -28 -Category=other, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=CHECK # 672, File=checking-merged, Measure=Amount = -43 -Category=other, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 420, File=checking-merged, Measure=Amount = -242 -Category=other, Date=12/15/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=CHECK # 428, File=checking-merged, Measure=Amount = -60 -Category=other, Date=12/20/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=CHECK # 673, File=checking-merged, Measure=Amount = -38 -Category=other, Date=12/22/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Specs #2410, File=checking-merged, Measure=Amount = -716.0800 -Category=other, Date=12/23/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=BLUE RIDGE SUPPLY, File=checking-merged, Measure=Amount = -42 -Category=other, Date=12/23/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=Unknown, File=checking-merged, Measure=Amount = -49 -Category=taxes, Date=01/15/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=FRANCHISE TAX BO, File=checking-merged, Measure=Amount = -2343 -Category=taxes, Date=01/15/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=IRS, File=checking-merged, Measure=Amount = -3786 -Category=taxes, Date=01/15/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=FRANCHISE TAX BO, File=checking-merged, Measure=Amount = -2724 -Category=taxes, Date=01/15/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=IRS, File=checking-merged, Measure=Amount = -2427 -Category=taxes, Date=04/11/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=VENTURACNTYTAXCA, File=checking-merged, Measure=Amount = -3190.4200 -Category=taxes, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=FRANCHISE TAX BO PAYMENTS, File=checking-merged, Measure=Amount = -2652 -Category=taxes, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=FRANCHISE TAX BO, File=checking-merged, Measure=Amount = -3011 -Category=taxes, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=LINDA REYNOLDS, File=checking-merged, Measure=Amount = -663 -Category=taxes, Date=04/16/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=IRS, File=checking-merged, Measure=Amount = -3079 -Category=taxes, Date=06/16/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=FRANCHISE TAX BO, File=checking-merged, Measure=Amount = -2597 -Category=taxes, Date=06/16/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=IRS, File=checking-merged, Measure=Amount = -2118 -Category=taxes, Date=09/15/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=IRS, File=checking-merged, Measure=Amount = -3807 -Category=taxes, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=VENTURACNTYTAXCA, File=checking-merged, Measure=Amount = -2907.4100 -Category=taxes, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=VENTURACNTYTAXCA, File=checking-merged, Measure=Amount = -4365.8000 -Category=transfers, Date=01/02/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -135 -Category=transfers, Date=01/12/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Caroline Martinez, File=checking-merged, Measure=Amount = 693 -Category=transfers, Date=01/21/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -111 -Category=transfers, Date=01/23/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Apple Cash, File=checking-merged, Measure=Amount = -109 -Category=transfers, Date=01/23/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=INFINITELOOP, File=checking-merged, Measure=Amount = -99 -Category=transfers, Date=01/23/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Infinite Loop, File=checking-merged, Measure=Amount = -59 -Category=transfers, Date=01/23/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -3 -Category=transfers, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 3 -Category=transfers, Date=01/30/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -112 -Category=transfers, Date=02/02/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -44 -Category=transfers, Date=02/06/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -104 -Category=transfers, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = -1968 -Category=transfers, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 2037 -Category=transfers, Date=02/25/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = 5342.8800 -Category=transfers, Date=02/26/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = 9250 -Category=transfers, Date=02/27/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = 1881 -Category=transfers, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -56 -Category=transfers, Date=03/09/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -129 -Category=transfers, Date=03/16/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = -136 -Category=transfers, Date=03/17/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JM.RIVERA JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -23 -Category=transfers, Date=03/23/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -642.0600 -Category=transfers, Date=03/23/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=MARGARET MARTINEZ, File=checking-merged, Measure=Amount = -91.7500 -Category=transfers, Date=03/24/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=JPMorgan Chase, File=checking-merged, Measure=Amount = -1709.9900 -Category=transfers, Date=03/31/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -58 -Category=transfers, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = -8111 -Category=transfers, Date=04/15/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 8298 -Category=transfers, Date=05/27/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = 24737.6000 -Category=transfers, Date=06/12/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = 20426 -Category=transfers, Date=06/12/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=MARTINEZ J WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = -23147 -Category=transfers, Date=06/25/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = -6089 -Category=transfers, Date=06/25/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 5219 -Category=transfers, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -38 -Category=transfers, Date=07/01/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=TREASURY DIRECT, File=checking-merged, Measure=Amount = -163 -Category=transfers, Date=07/01/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=transfer to new account # ending in 5917, File=savings-merged, Measure=Amount = -16730 -Category=transfers, Date=07/14/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=VENMO, File=checking-merged, Measure=Amount = 518 -Category=transfers, Date=07/28/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = -1417 -Category=transfers, Date=07/28/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -849.0100 -Category=transfers, Date=07/28/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=MARTINEZ J WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 1729 -Category=transfers, Date=08/11/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1068.7200 -Category=transfers, Date=08/14/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=APPLE CASH BANK XFER, File=checking-merged, Measure=Amount = 527.0300 -Category=transfers, Date=08/26/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=Morgan Stanley, File=checking-merged, Measure=Amount = 8607.1900 -Category=transfers, Date=09/08/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -72.9100 -Category=transfers, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=InfiniteLoop, File=checking-merged, Measure=Amount = -61 -Category=transfers, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -2 -Category=transfers, Date=09/30/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 2 -Category=transfers, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=APPLE CASH, File=checking-merged, Measure=Amount = -52 -Category=transfers, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=INFINITELOOP, File=checking-merged, Measure=Amount = -87 -Category=transfers, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -88.3400 -Category=transfers, Date=10/06/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -2 -Category=transfers, Date=10/07/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 1 -Category=transfers, Date=10/08/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=APPLE CASH, File=checking-merged, Measure=Amount = -89 -Category=transfers, Date=10/08/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -1 -Category=transfers, Date=10/09/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 1 -Category=transfers, Date=10/14/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = 18808 -Category=transfers, Date=10/14/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=MARTINEZ J WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 22333 -Category=transfers, Date=11/18/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = -5388 -Category=transfers, Date=11/18/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = 3661 -Category=transfers, Date=11/20/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=JORDAN MARTINEZ, File=checking-merged, Measure=Amount = -79 -Category=transfers, Date=11/20/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = 10.3700 -Category=transfers, Date=11/25/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=Jordan Martinez, File=checking-merged, Measure=Amount = -0.9700 -Category=transfers, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=EVERYDAY CHECKING, File=savings-merged, Measure=Amount = 9074 -Category=transfers, Date=12/12/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=MARTINEZ J WAY2SAVE SAVINGS, File=checking-merged, Measure=Amount = -5398 -Category=transfers, Date=12/16/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=JPMORGAN CHASE, File=checking-merged, Measure=Amount = -1939.5600 -Category=transfers, Date=12/22/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -1 -Category=transfers, Date=12/23/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=XXXXXXX8402, File=checking-merged, Measure=Amount = -1 -Category=transfers, Date=12/23/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 1 -Category=transfers, Date=12/24/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=SAVE AS YOU GO, File=savings-merged, Measure=Amount = 1 -Category=utilities, Date=01/23/2025, Date_Month=2025-01, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -269.4200 -Category=utilities, Date=01/26/2026, Date_Month=2026-01, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SoCalGas, File=checking-merged, Measure=Amount = -238.0100 -Category=utilities, Date=02/02/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -45.0600 -Category=utilities, Date=02/05/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -159.5900 -Category=utilities, Date=02/21/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -225.6600 -Category=utilities, Date=02/23/2026, Date_Month=2026-02, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SoCalGas, File=checking-merged, Measure=Amount = -203.1100 -Category=utilities, Date=02/26/2025, Date_Month=2025-02, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -128.3200 -Category=utilities, Date=03/02/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -36.5000 -Category=utilities, Date=03/24/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -404.7400 -Category=utilities, Date=03/24/2026, Date_Month=2026-03, Date_Quarter=2026-Q1, Date_Year=2026, Entity=SoCalGas, File=checking-merged, Measure=Amount = -168.0900 -Category=utilities, Date=03/31/2025, Date_Month=2025-03, Date_Quarter=2025-Q1, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -774.0100 -Category=utilities, Date=04/22/2025, Date_Month=2025-04, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -329.0100 -Category=utilities, Date=05/20/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -184.7000 -Category=utilities, Date=05/29/2025, Date_Month=2025-05, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -43.9500 -Category=utilities, Date=06/23/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -64.8500 -Category=utilities, Date=06/30/2025, Date_Month=2025-06, Date_Quarter=2025-Q2, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -30.0600 -Category=utilities, Date=07/22/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -35.8400 -Category=utilities, Date=07/31/2025, Date_Month=2025-07, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -63.3000 -Category=utilities, Date=08/19/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -36.5800 -Category=utilities, Date=08/27/2025, Date_Month=2025-08, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -101.0600 -Category=utilities, Date=09/22/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -44.7300 -Category=utilities, Date=09/29/2025, Date_Month=2025-09, Date_Quarter=2025-Q3, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -197.4500 -Category=utilities, Date=10/21/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -43.0300 -Category=utilities, Date=10/29/2025, Date_Month=2025-10, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -509.9300 -Category=utilities, Date=11/21/2025, Date_Month=2025-11, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -96.0100 -Category=utilities, Date=12/02/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -58.0800 -Category=utilities, Date=12/24/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=SO CAL GAS, File=checking-merged, Measure=Amount = -209.2200 -Category=utilities, Date=12/26/2024, Date_Month=2024-12, Date_Quarter=2024-Q4, Date_Year=2024, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -127.7000 -Category=utilities, Date=12/26/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SoCalGas, File=checking-merged, Measure=Amount = -251.0800 -Category=utilities, Date=12/30/2025, Date_Month=2025-12, Date_Quarter=2025-Q4, Date_Year=2025, Entity=SO CAL EDISON CO, File=checking-merged, Measure=Amount = -37.7600 - -## View: Default -_Index: none -_Dim: none -Category: row -Date: none -Entity: none -File: page -Date_Year: none -Date_Month: none -Date_Quarter: column -Measure: page -format: ,.0 diff --git a/crates/improvise-io/src/persistence/roundtrip_tests.rs b/crates/improvise-io/src/persistence/roundtrip_tests.rs index 62711bfe..566d80d8 100644 --- a/crates/improvise-io/src/persistence/roundtrip_tests.rs +++ b/crates/improvise-io/src/persistence/roundtrip_tests.rs @@ -886,3 +886,80 @@ fn a_csv_header_with_an_embedded_newline_survives_a_save() { loaded.model.categories.keys().collect::>() ); } + +/// improvise-odbc: `bank-info.improv` in the repo root failed to load at +/// `- JORDAN MARTINEZ` (line 293) with `expected pipe_quoted`. The ticket +/// named two readings — a stale file, or a writer/reader gap in which +/// improvise emits an item line the grammar rejects — and said to decide by +/// asking whether improvise can *write* that line today. +/// +/// It cannot. `item_list`'s `name` is `pipe_quoted | bare_name`, and +/// `bare_name` is `[A-Za-z_][A-Za-z0-9_-]*` with no space; `quote_name` +/// mirrors it character for character, so every one of these item names is +/// emitted pipe-quoted and read straight back. The file was hand-edited in +/// c48a5cd (the v2025-04-09 migration quoted the `Date` items and left the +/// `Entity` ones bare) and never round-tripped, so the reading is "stale +/// file", and the standing finding is that nothing exercised it. +/// +/// This is the shape that finding turns into: the item names from the file +/// that motivated the ticket, asserted through the writer and back. +#[test] +fn an_item_name_the_item_line_cannot_carry_round_trips() { + let names = [ + "JORDAN MARTINEZ", + "Holy Trinity Parish", + "CHECK # 445", + "02/14/2025", + "SANTAPAULATRAIL", + ]; + let mut m = Workbook::new("Bank"); + m.add_category("Entity").unwrap(); + for n in names { + m.add_item("Entity", n).unwrap(); + } + + let text = format_md(&m); + assert!( + text.contains("- |JORDAN MARTINEZ|"), + "a spaced item name must be written pipe-quoted, got:\n{text}" + ); + assert!( + text.contains("SANTAPAULATRAIL") && !text.contains("|SANTAPAULATRAIL|"), + "a bare-legal item name must stay bare, got:\n{text}" + ); + + let loaded = parse_md(&text).unwrap_or_else(|e| panic!("{e}\n{text}")); + let got: Vec<&str> = loaded + .model + .category("Entity") + .unwrap() + .items + .values() + .map(|i| i.name.as_str()) + .collect(); + for n in names { + assert!(got.contains(&n), "{n:?} missing from {got:?}\n{text}"); + } +} + +/// The other half of the same finding: the exact line the file carries is +/// *not* something the reader accepts, so the file really was broken rather +/// than the error message being misleading. Asserting the failure keeps the +/// test above honest — without it, "the writer quotes it" would be consistent +/// with the reader also accepting the unquoted form. +#[test] +fn a_bare_item_name_with_a_space_is_rejected() { + let text = "v2025-04-09\n# Bank\n\n## Category: Entity\n- JORDAN MARTINEZ\n"; + let err = parse_md(text).expect_err("a bare spaced item name must not parse"); + let msg = err.to_string(); + assert!( + msg.contains("JORDAN MARTINEZ"), + "the error should point at the offending line, got: {msg}" + ); + + let quoted = "v2025-04-09\n# Bank\n\n## Category: Entity\n- |JORDAN MARTINEZ|\n"; + assert!( + parse_md(quoted).is_ok(), + "precondition: the same line quoted must parse" + ); +} diff --git a/tests/shipped_documents_load.rs b/tests/shipped_documents_load.rs new file mode 100644 index 00000000..64a045f2 --- /dev/null +++ b/tests/shipped_documents_load.rs @@ -0,0 +1,53 @@ +//! Every `.improv` document this repository ships must load (improvise-odbc). +//! +//! The finding behind this test: `bank-info.improv` sat in the repo root for +//! five months unable to parse — hand-edited through the v2025-04-09 format +//! change rather than round-tripped, so its item lines carried bare names the +//! grammar cannot spell (`- JORDAN MARTINEZ`) and its data lines named a +//! category `Measure` the file declares as `_Measure`. Nothing loaded it, so +//! nothing said so. +//! +//! The grammar itself is covered far better than a fixture can manage, by the +//! generator-driven proptests in `improvise-io`. What *those* cannot see is a +//! checked-in document drifting out of the format, because they generate their +//! inputs. This test is aimed at exactly that gap, and it is deliberately a +//! directory walk rather than a list of names: a list is a second place to +//! remember, and the file that gets forgotten is the one nobody added to it. +//! +//! The scope is the two directories `.gitignore` un-ignores — `examples/` and +//! `web/`. Everything else matching `*.improv` is somebody's working file. + +use std::path::{Path, PathBuf}; + +fn shipped_documents() -> Vec { + let root = Path::new(env!("CARGO_MANIFEST_DIR")); + let mut found: Vec = ["examples", "web"] + .iter() + .flat_map(|dir| std::fs::read_dir(root.join(dir)).expect("a shipped-document directory")) + .map(|entry| entry.expect("a readable directory entry").path()) + .filter(|p| p.extension().is_some_and(|e| e == "improv")) + .collect(); + found.sort(); + found +} + +#[test] +fn every_shipped_document_loads() { + let documents = shipped_documents(); + assert!( + documents.len() >= 2, + "precondition: the walk must find the documents that exist, got {documents:?}" + ); + + for path in documents { + let workbook = improvise::persistence::load(&path) + .unwrap_or_else(|e| panic!("{} does not load: {e}", path.display())); + assert!( + workbook.views.contains_key(&workbook.active_view), + "{} loads but its initial view {:?} is not one of {:?}", + path.display(), + workbook.active_view, + workbook.views.keys().collect::>() + ); + } +} From 18189c1dee341a5bdecb5a58a22d0acdd73475cc Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:36:30 -0700 Subject: [PATCH 03/72] refactor(wasm-client): one answer to what a typed string means `Client::set_cell_at` and `socket.rs::typed` gave opposite answers to the same question, one level below the envelopes improvise-xks6 had just consolidated. `set_cell_at`'s doc said classifying a typed string "is the host's decision, not one to re-implement here" and took a serialized `CellValue`; `typed`'s said the opposite and `typed_cell` takes a plain string. Both composed the same `ClientFrame`, and they stayed apart only because nothing called both. The `set_cell_at` claim is false on every deployment. `ModelCommand::SetCell` carries a `CellValue` on the wire, so no host ever sees the string. The ticket's escape hatch -- that improvise-9x6's `App` in the browser might make the pre-classified form right for the worker -- does not exist: that `App` sits behind the frame boundary and is handed a value, not a string. So `set_cell_at` and the `set_cell` it wrapped are deleted, not re-documented. `set_cell` could not express `ClearCell`, which makes it a strictly weaker duplicate of `typed_cell_frame` rather than a peer, and `the_json_wrapper_composes_the_same_message` was precisely the compatibility test doctrine says a finished consolidation does not keep. Evidence: 1189 -> 1184. The five deleted tests are the two wrapper tests (aimed at deleted code) and three None-shape tests already asserted against `typed_cell_frame` in socket.rs. The fourth None shape -- ADR 001's absent basis -- had no twin, so `a_client_with_no_basis_composes_nothing` was retargeted rather than dropped, and proved discriminating by injecting a published `Projection::Basis` into its fixture and watching it flip to Some. Three doc comments elsewhere named the deleted function (keys.rs, state.rs, and socket.rs's own None-shape test); all three re-point, per gotcha 48's rule that changing a stated fact is a grep rather than an edit. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-wasm-client/src/keys.rs | 2 +- crates/improvise-wasm-client/src/lib.rs | 167 +-------------------- crates/improvise-wasm-client/src/socket.rs | 42 +++++- crates/improvise-wasm-client/src/state.rs | 2 +- 4 files changed, 46 insertions(+), 167 deletions(-) diff --git a/crates/improvise-wasm-client/src/keys.rs b/crates/improvise-wasm-client/src/keys.rs index 371ab885..47467f93 100644 --- a/crates/improvise-wasm-client/src/keys.rs +++ b/crates/improvise-wasm-client/src/keys.rs @@ -1080,7 +1080,7 @@ mod tests { /// `t` is a document mutation, so it carries a basis — and until one has /// been published there is nothing truthful to claim, which is the - /// fourth `None` shape `Client::set_cell` already has (ADR 001). + /// fourth `None` shape `Client::typed_cell_frame` already has (ADR 001). #[test] fn transpose_needs_a_published_basis() { let state = in_mode(AppMode::Normal); diff --git a/crates/improvise-wasm-client/src/lib.rs b/crates/improvise-wasm-client/src/lib.rs index 7c81cb05..fe7b25bf 100644 --- a/crates/improvise-wasm-client/src/lib.rs +++ b/crates/improvise-wasm-client/src/lib.rs @@ -61,9 +61,7 @@ pub use keys::{BrowserKey, KeyOutcome, resolve}; pub use render::{RenderCatRow, RenderCell, RenderModel, RenderRow}; pub use state::ClientState; -use improvise_protocol::{ - CellId, CellValue, ClientFrame, Command, ModelCommand, RecordCell, Snapshot, -}; +use improvise_protocol::{CellId, ClientFrame, RecordCell, Snapshot}; use wasm_bindgen::prelude::*; /// The handle a JS bootstrap holds for the lifetime of the tab. @@ -131,20 +129,6 @@ impl Client { self.state.revision } - /// Compose a write to the cell at a viewport position. - /// - /// `value_json` is a serialized `CellValue`; everything crossing this - /// boundary is JSON, and *what* a typed string means — a number or text - /// — is the host's decision, not one to re-implement here. Returns the - /// serialized [`ClientFrame`], or `None` when the position holds - /// nothing this client may write to. - #[wasm_bindgen] - pub fn set_cell_at(&self, row: usize, col: usize, value_json: &str) -> Option { - let value: CellValue = serde_json::from_str(value_json).ok()?; - let message = self.set_cell(row, col, value)?; - serde_json::to_string(&message).ok() - } - // ── the transport surface (improvise-cr3) ─────────────────────────── // // Three string calls, and they are the whole harness a page needs: send @@ -254,42 +238,6 @@ impl Client { self.state.bootstrap(snapshot); } - /// Compose a write to the cell at a viewport position, addressed by the - /// surrogate the server projected there (ADR 002). - /// - /// This is the composition `improvise-q08` found a thin client could not - /// do. It holds no `Model`, so it cannot build a `CellKey` — but it does - /// not need to: the id on the cell it is already drawing *is* the - /// address, and it is echoed back unread. - /// - /// `None` on three shapes, and they are three different facts the cache - /// already carries: the position is outside the grid, the position holds - /// no settable cell (`id: None` — padding, or a records-mode column that - /// renames a coordinate rather than writing one), or the cell is an - /// **aggregate**, which `SetCell` cannot be composed for because - /// [`ModelCommand::SetCell`] does not take that kind of id. The last is - /// the one worth noting: it is not a check this function performs, it is - /// the pattern match failing to bind. - /// - /// Unlike [`Client::key`]'s locally-resolved commands, nothing is folded - /// here. The new value comes back as a `GridUpdate::Cells` from the host, - /// which is the side that formats it; guessing it locally is how the two - /// ends drift apart. - /// A **fourth** shape returns `None`, and it is ADR 001's: this client - /// has not yet been told the document's basis, so it has nothing - /// truthful to say about what it composed against. `BasisCache::claim` - /// is the only constructor there is, and it withholds rather than - /// invents — a forged basis would be the check deleted with extra steps. - /// In practice a client that has folded its bootstrap always has one. - pub fn set_cell(&self, row: usize, col: usize, value: CellValue) -> Option { - let cell = self.record_cell_at(row, col)?; - let basis = self.state.basis.claim([cell.record])?; - Some(ClientFrame::command(Command::Model { - basis, - cmd: ModelCommand::SetCell { cell, value }, - })) - } - /// Classify a keypress, folding it locally when this client can answer it. pub fn key(&mut self, browser: &BrowserKey) -> Option { let event = browser.to_key_event()?; @@ -321,62 +269,17 @@ mod tests { serde_json::from_str(&format!("\"{hex}\"")).expect("a surrogate is hex") } - fn client_showing(id: Option) -> Client { - let mut cell = improvise_protocol::CellDisplay::number("1200"); - cell.id = id; - let mut client = init(); - client.bootstrap(Snapshot { - view: ViewState::default(), - seed: vec![ - Projection::Grid(GridUpdate::RowLabels(vec!["North".into()])), - Projection::Grid(GridUpdate::ColLabels(vec!["Revenue".into()])), - Projection::Grid(GridUpdate::Cells(vec![((0, 0), cell)])), - Projection::Basis(published_basis()), - ], - }); - client - } - - /// What a server publishes for the one record this fixture's grid names - /// (ADR 001). A client that was never sent one composes nothing at all, - /// which `a_client_with_no_basis_composes_nothing` pins. - fn published_basis() -> improvise_protocol::Basis { - improvise_protocol::basis::published( - 0, - 0, - [(surrogate("fe4c8e5e9cbd578e0d388f4b2b5a7879"), 0)] - .into_iter() - .collect(), - ) - } - - /// The composition improvise-q08 found a thin client could not do. The - /// id goes back out **unchanged**: it is a token to echo, so the - /// assertion is equality with what arrived, not a re-derivation. - #[test] - fn a_client_composes_a_write_from_the_id_it_was_given() { - let id = improvise_protocol::RecordCell { - record: surrogate("fe4c8e5e9cbd578e0d388f4b2b5a7879"), - measure: Some("Revenue".into()), - }; - let client = client_showing(Some(CellId::Record(id.clone()))); - assert_eq!( - client.set_cell(0, 0, CellValue::Number(42.0)), - Some(ClientFrame::command(Command::Model { - basis: published_basis(), - cmd: ModelCommand::SetCell { - cell: id, - value: CellValue::Number(42.0), - }, - })) - ); - } - /// The fourth `None`, and the one ADR 001 adds: a client that has not /// been told the document's basis has nothing truthful to say about what /// it composed against, so it composes nothing. That is the property /// that makes the basis check unforgeable from a browser — /// `BasisCache::claim` is the only constructor, and it withholds. + /// + /// The other three `None` shapes — an aggregate, `id: None`, and a + /// position outside the grid — are pinned against the same composition in + /// `socket.rs`'s `typing_into_a_cell_that_is_not_a_record_composes_nothing`. + /// They were asserted twice, once against each of two compositions, until + /// improvise-r4lo left only one. #[test] fn a_client_with_no_basis_composes_nothing() { let id = improvise_protocol::RecordCell { @@ -394,61 +297,7 @@ mod tests { Projection::Grid(GridUpdate::Cells(vec![((0, 0), cell)])), ], }); - assert_eq!(client.set_cell(0, 0, CellValue::Number(42.0)), None); - } - - /// The point of the enum: an aggregate carries an identity, and it is - /// still not one `SetCell` accepts. Nothing here checks anything — the - /// `let CellId::Record(..) = ... else` simply does not bind. - #[test] - fn a_client_cannot_compose_a_write_for_an_aggregate() { - let hash: improvise_protocol::AggregateHash = - serde_json::from_str("\"5b05bbcd2364099f68a49dbf9099e075\"").expect("hex"); - let client = client_showing(Some(CellId::Aggregate(hash))); - assert_eq!(client.set_cell(0, 0, CellValue::Number(42.0)), None); - } - - /// `id: None` is not "aggregate" — it is "no projected cell here", and a - /// position holding one is not writable either. - #[test] - fn a_cell_the_server_named_nothing_for_is_not_settable() { - let client = client_showing(None); - assert_eq!(client.set_cell(0, 0, CellValue::Number(42.0)), None); - assert_eq!( - client.set_cell(9, 9, CellValue::Number(42.0)), - None, - "and neither is a position outside the grid" - ); - } - - /// The JS-facing wrapper is the same path with JSON at both ends, so a - /// browser needs no structural bindings. - #[test] - fn the_json_wrapper_composes_the_same_message() { - let id = improvise_protocol::RecordCell { - record: surrogate("fe4c8e5e9cbd578e0d388f4b2b5a7879"), - measure: Some("Revenue".into()), - }; - let client = client_showing(Some(CellId::Record(id))); - let json = client - .set_cell_at(0, 0, "{\"Number\":7.5}") - .expect("the wrapper must compose"); - assert_eq!( - serde_json::from_str::(&json).expect("round trips"), - client.set_cell(0, 0, CellValue::Number(7.5)).unwrap() - ); - } - - /// A value that is not a `CellValue` composes nothing rather than - /// panicking across the wasm boundary. - #[test] - fn the_json_wrapper_refuses_a_value_it_cannot_read() { - let id = improvise_protocol::RecordCell { - record: surrogate("fe4c8e5e9cbd578e0d388f4b2b5a7879"), - measure: None, - }; - let client = client_showing(Some(CellId::Record(id))); - assert_eq!(client.set_cell_at(0, 0, "not json"), None); + assert_eq!(client.typed_cell_frame(0, 0, "42"), None); } /// The end-to-end shape a JS harness drives: bootstrap, key, message. diff --git a/crates/improvise-wasm-client/src/socket.rs b/crates/improvise-wasm-client/src/socket.rs index 27dd49e2..103d82ba 100644 --- a/crates/improvise-wasm-client/src/socket.rs +++ b/crates/improvise-wasm-client/src/socket.rs @@ -65,6 +65,11 @@ fn rejection_message(rejection: &Rejection) -> String { /// Rust rather than in the page's JavaScript is what makes it testable and /// keeps one spelling of the rule per language boundary instead of two. /// +/// This is the **only** answer the client gives to "what does a typed string +/// mean" (improvise-r4lo). It holds on every deployment, including the worker: +/// a host's `App` is behind the frame boundary and receives a `CellValue`, so +/// no host anywhere ever sees the string to classify it. +/// /// `None` means "clear this cell", which is a different command, not a /// different value — hence the `Option` rather than a `CellValue::Text("")`. fn typed(text: &str) -> Option { @@ -137,12 +142,37 @@ impl Client { } /// Compose the write a user made by typing `text` into the cell at a - /// viewport position. + /// viewport position — the **one** composition of a cell write there is + /// (improvise-r4lo). /// - /// `None` on exactly the shapes [`Client::set_cell`] refuses — outside - /// the grid, no projected cell, an aggregate, which `SetCell` cannot be - /// composed for because it does not take that kind of id (ADR 002), or - /// no basis yet (ADR 001). + /// This is the composition `improvise-q08` found a thin client could not + /// do. It holds no `Model`, so it cannot build a `CellKey` — but it does + /// not need to: the id on the cell it is already drawing *is* the + /// address, and it is echoed back unread (ADR 002). + /// + /// It takes the user's **string**, not a `CellValue`, and [`typed`] above + /// says why: the classification is unavoidably this side of the wire, + /// because `ModelCommand::SetCell` already carries a `CellValue` by the + /// time any host sees it. That is true of the worker deployment too — the + /// `App` there sits *behind* the frame boundary and is handed a value, not + /// a string — so there is no deployment for which a pre-classified entry + /// point would be the right one. A sibling taking a serialized `CellValue` + /// used to sit beside this and claim the opposite; it was deleted rather + /// than re-documented, because it could not express `ClearCell` and was + /// therefore a strictly weaker duplicate of this. + /// + /// `None` on four shapes, each a different fact the client already holds: + /// the position is outside the grid; the position holds no settable cell + /// (`id: None` — padding, or a records-mode column that renames a + /// coordinate rather than writing one); the cell is an **aggregate**, + /// which is the pattern failing to bind rather than a check (ADR 002); + /// or this client has not been told the document's basis and so has + /// nothing truthful to say about what it composed against (ADR 001 — + /// `BasisCache::claim` withholds rather than invents). + /// + /// Nothing is folded here. The new value comes back as a + /// `GridUpdate::Cells` from the host, which is the side that formats it; + /// guessing it locally is how the two ends drift apart. pub fn typed_cell_frame(&self, row: usize, col: usize, text: &str) -> Option { let cell = self.record_cell_at(row, col)?; let basis = self.state().basis.claim([cell.record])?; @@ -366,7 +396,7 @@ mod tests { } /// Typing into a cell the client may not write is no frame at all, for - /// the same reason `set_cell` composes nothing: the id is of the wrong + /// the same reason any write composes nothing: the id is of the wrong /// kind, or there is no id. `Drill` is not on the wire yet, so an /// aggregate is simply not editable here. #[test] diff --git a/crates/improvise-wasm-client/src/state.rs b/crates/improvise-wasm-client/src/state.rs index e977e358..8459f30f 100644 --- a/crates/improvise-wasm-client/src/state.rs +++ b/crates/improvise-wasm-client/src/state.rs @@ -36,7 +36,7 @@ pub struct ClientState { pub tiles: TileBarCache, /// The picture of the document this client may claim to have composed /// against (ADR 001). Not drawn; folded like any other projection, and - /// read by `Client::set_cell` when it composes a write. + /// read by `Client::typed_cell_frame` when it composes a write. #[serde(skip)] pub basis: BasisCache, /// The last basis rejection this client was told about, if any. From 0ba8d9a7408526d861e64f1e9093cb9ecdfde583 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:06 -0700 Subject: [PATCH 04/72] refactor(io): make storage_contract a function, not a test helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract in `storage.rs` is described everywhere as "the behaviour every backend owes", and gotcha 28 calls it the written-down spec `OpfsFs` must satisfy in a browser. It could not be: it was `#[cfg(test)]` in this file, so the one backend that most needed it was the one backend that could never be handed to it. Moving it into `pub mod contract` is what lets a browser harness run THIS sequence rather than a second transcription of it — a second copy of a contract is the disease this codebase keeps curing (doctrine: a_derivation_done_twice_is_the_defect). Two shape changes fall out. It returns `anyhow::Result<()>` rather than asserting, because a failure has to travel out of a Web Worker as a string; the two host tests `.unwrap()` it and get the same message. And it now ends by deleting `b.improv` and re-checking `list` is empty, so a caller can run it twice against a persistent backend — which a browser one is. No behaviour change: 1189 passing before and after. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/storage.rs | 101 +++++++++++++++++++---------- 1 file changed, 68 insertions(+), 33 deletions(-) diff --git a/crates/improvise-io/src/storage.rs b/crates/improvise-io/src/storage.rs index 12cfbf52..727eead4 100644 --- a/crates/improvise-io/src/storage.rs +++ b/crates/improvise-io/src/storage.rs @@ -30,6 +30,71 @@ pub trait Storage { fn delete(&self, path: &Path) -> Result<()>; } +/// The behaviour every backend owes, as a function rather than as a test. +/// +/// It reads like a test helper and it was one — `#[cfg(test)]` in this file, +/// run against `MemoryFs` and `PhysicalFs` so they could not drift apart. The +/// one backend that most needed it could never be handed to it: `OpfsFs` is +/// `cfg(target_arch = "wasm32")` and its sync access handles exist in no host +/// harness, so the contract described it only in a doc comment. Moving it out +/// of `cfg(test)` is what lets a browser run **this** sequence rather than a +/// second transcription of it (improvise-vvl3/xf1) — a second copy of a +/// contract is the disease this codebase keeps curing. +/// +/// It returns `Result` rather than asserting, because a failure has to travel +/// out of a Web Worker as a string; the host tests `.unwrap()` it and get the +/// same message. +pub mod contract { + use std::path::Path; + + use anyhow::{Result, ensure}; + + use super::Storage; + + /// Write / read / replace / list-sorted / delete / read-fails / + /// delete-twice-errors / shorter-write-truncates, at `dir`. + /// + /// `dir` is a parameter and not a constant because the three backends + /// must agree at **every** `dir`, not only at the flat root the browser + /// happens to use (improvise-vhcz). A backend whose `list` answered in a + /// different path shape would pass at `Path::new("")` — where joining is + /// the identity — and fail here at anything else. + pub fn storage_contract(fs: &dyn Storage, dir: &Path) -> Result<()> { + let a = dir.join("a.improv"); + let b = dir.join("b.improv"); + + ensure!(fs.list(dir)?.is_empty(), "starts empty"); + + fs.write(&a, b"first")?; + ensure!(fs.read(&a)? == b"first", "reads back what was written"); + ensure!(fs.list(dir)? == vec![a.clone()], "lists the one document"); + + fs.write(&a, b"second")?; + ensure!(fs.read(&a)? == b"second", "write replaces"); + ensure!(fs.list(dir)? == vec![a.clone()], "no duplicate entry"); + + fs.write(&b, b"other")?; + ensure!(fs.list(dir)? == vec![a.clone(), b.clone()], "sorted"); + + fs.delete(&a)?; + ensure!(fs.read(&a).is_err(), "deleted document is unreadable"); + ensure!( + fs.list(dir)? == vec![b.clone()], + "delete drops it from list" + ); + ensure!(fs.delete(&a).is_err(), "deleting twice is an error"); + + // A shorter document must not leave the tail of a longer one. + fs.write(&b, b"0123456789")?; + fs.write(&b, b"xy")?; + ensure!(fs.read(&b)? == b"xy", "a shorter write truncates"); + + fs.delete(&b)?; + ensure!(fs.list(dir)?.is_empty(), "ends as it started"); + Ok(()) + } +} + /// The real filesystem. What the native build wires up. #[derive(Debug, Default, Clone, Copy)] pub struct PhysicalFs; @@ -140,47 +205,17 @@ impl Storage for MemoryFs { mod tests { use super::*; - /// The behaviour every backend owes, exercised against each host - /// backend so they cannot drift apart. `OpfsFs` has to satisfy the same - /// sequence in a browser, where this harness cannot reach it — so this - /// doubles as the written-down contract for it (improvise-i34). - fn storage_contract(fs: &dyn Storage, dir: &Path) { - let a = dir.join("a.improv"); - let b = dir.join("b.improv"); - - assert!(fs.list(dir).unwrap().is_empty(), "starts empty"); - - fs.write(&a, b"first").unwrap(); - assert_eq!(fs.read(&a).unwrap(), b"first"); - assert_eq!(fs.list(dir).unwrap(), vec![a.clone()]); - - fs.write(&a, b"second").unwrap(); - assert_eq!(fs.read(&a).unwrap(), b"second", "write replaces"); - assert_eq!(fs.list(dir).unwrap(), vec![a.clone()], "no duplicate entry"); - - fs.write(&b, b"other").unwrap(); - assert_eq!(fs.list(dir).unwrap(), vec![a.clone(), b.clone()], "sorted"); - - fs.delete(&a).unwrap(); - assert!(fs.read(&a).is_err(), "deleted document is unreadable"); - assert_eq!(fs.list(dir).unwrap(), vec![b.clone()]); - assert!(fs.delete(&a).is_err(), "deleting twice is an error"); - - // A shorter document must not leave the tail of a longer one. - fs.write(&b, b"0123456789").unwrap(); - fs.write(&b, b"xy").unwrap(); - assert_eq!(fs.read(&b).unwrap(), b"xy"); - } + use super::contract::storage_contract; #[test] fn memory_fs_satisfies_the_storage_contract() { - storage_contract(&MemoryFs::new(), Path::new("docs")); + storage_contract(&MemoryFs::new(), Path::new("docs")).unwrap(); } #[test] fn physical_fs_satisfies_the_storage_contract() { let dir = tempfile::tempdir().unwrap(); - storage_contract(&PhysicalFs, dir.path()); + storage_contract(&PhysicalFs, dir.path()).unwrap(); } #[test] From 92a3d3e179d7ec3e735b39fb0f67e2f6679ff9e1 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:28 -0700 Subject: [PATCH 05/72] feat(io): a shared backend is a backend, so the async seam survives boxing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-hmws. `OpfsFs` keeps `open`/`attach`/`refresh`/`commit_deletes` off the `Storage` trait so the trait can stay synchronous — gotcha 28, and not negotiable, because `createSyncAccessHandle` really does yield a blocking handle. But `App::storage` is a `Box`, so handing the backend to the `App` puts all four permanently out of reach: the caller has to hold the concrete backend *and* give the `App` a `Storage`, which means sharing one. The ticket offered two fixes — ship a forwarding newtype from `improvise-io`, or have `OpfsFs` hand out an attachable handle. Neither is needed, and the reason is worth stating: `impl Storage for Rc` is refused in `improvise-worker-server` because *there* both types are foreign. In `improvise-io` the trait is local, so the impl is simply allowed. It is blanket — `impl Storage for Rc` — because nothing about it is OPFS-shaped; it is the general statement that a shared handle to a backend is usable as that backend. `Rc` and not `Arc` because the one caller is a Web Worker and wasm32 has no other threads. So `bindings::Shared` and its `_shim_is_a_storage` canary are deleted rather than moved: there is no newtype to ship and no second copy to keep. The seam now costs an `Rc::clone`. Evidence: `a_shared_backend_is_a_backend_and_a_clone_is_the_same_one`. The regression guard is the *signature* — it does not compile without the impl — so what the body asserts is the part a hand-written newtype could still get wrong: that the clone the `App` holds and the handle the message loop keeps are one backend. Proved discriminating by making the forwarded `write` a no-op and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/storage.rs | 65 +++++++++++++++++++ .../improvise-worker-server/src/bindings.rs | 55 ++++------------ 2 files changed, 78 insertions(+), 42 deletions(-) diff --git a/crates/improvise-io/src/storage.rs b/crates/improvise-io/src/storage.rs index 727eead4..867a2aa5 100644 --- a/crates/improvise-io/src/storage.rs +++ b/crates/improvise-io/src/storage.rs @@ -95,6 +95,42 @@ pub mod contract { } } +/// A shared backend is a backend. +/// +/// This exists because of a seam that only a browser has, and it lives here +/// because of the orphan rule (improvise-hmws). `OpfsFs` keeps four genuinely +/// async operations — `open`, `attach`, `refresh`, `commit_deletes` — off the +/// trait, deliberately, so that [`Storage`] can stay synchronous (gotcha 28). +/// But `App::storage` is a `Box`, and a trait object can name +/// none of the four: **handing the backend to the `App` puts the async seam +/// permanently out of reach**. The caller therefore needs to hold the concrete +/// backend *and* give the `App` a `Storage`, which means sharing one. +/// +/// `impl Storage for Rc` is refused in any *other* crate — two foreign +/// types — so improvise-9x6 wrote a forwarding newtype in the worker, and +/// every future worker would have written it again. Here the trait is local, +/// so the impl is allowed, and it is blanket rather than `OpfsFs`-specific +/// because nothing about it is OPFS-shaped: it is the general statement that a +/// shared handle to a backend is usable as that backend. +/// +/// `Rc` and not `Arc`: the one caller is a Web Worker, which is a single +/// thread, and `wasm32-unknown-unknown` has no others. An `Arc` impl would be +/// speculative — add it when something needs it. +impl Storage for std::rc::Rc { + fn read(&self, path: &Path) -> Result> { + (**self).read(path) + } + fn write(&self, path: &Path, bytes: &[u8]) -> Result<()> { + (**self).write(path, bytes) + } + fn list(&self, dir: &Path) -> Result> { + (**self).list(dir) + } + fn delete(&self, path: &Path) -> Result<()> { + (**self).delete(path) + } +} + /// The real filesystem. What the native build wires up. #[derive(Debug, Default, Clone, Copy)] pub struct PhysicalFs; @@ -218,6 +254,35 @@ mod tests { storage_contract(&PhysicalFs, dir.path()).unwrap(); } + /// improvise-hmws: a worker must hold the concrete backend (for the four + /// async methods the trait does not carry) *and* hand the `App` a + /// `Storage`. Before the blanket impl above, `Rc` was not a + /// `Storage` and could not be made one outside this crate — the orphan + /// rule — so `improvise-worker-server` wrote a forwarding newtype, and so + /// would every worker after it. + /// + /// The guard is the **signature**: this does not compile without the + /// impl. What it asserts beyond compiling is the part a newtype could + /// still get wrong — that the clone the `App` holds and the handle the + /// message loop keeps are one backend, not two. + #[test] + fn a_shared_backend_is_a_backend_and_a_clone_is_the_same_one() { + use std::rc::Rc; + + let fs = Rc::new(MemoryFs::new()); + storage_contract(&Rc::clone(&fs), Path::new("docs")).unwrap(); + + let boxed: Box = Box::new(Rc::clone(&fs)); + boxed + .write(Path::new("shared.improv"), b"through the box") + .unwrap(); + assert_eq!( + fs.read(Path::new("shared.improv")).unwrap(), + b"through the box", + "the App's copy and the loop's copy must be one backend" + ); + } + #[test] fn memory_fs_round_trips_bytes() { let fs = MemoryFs::new(); diff --git a/crates/improvise-worker-server/src/bindings.rs b/crates/improvise-worker-server/src/bindings.rs index 7339132f..3a37a440 100644 --- a/crates/improvise-worker-server/src/bindings.rs +++ b/crates/improvise-worker-server/src/bindings.rs @@ -21,11 +21,15 @@ //! `App::storage` is a `Box`, and the trait has none of those four //! methods, so **handing `OpfsFs` to the `App` puts the async seam out of //! reach**. A boxed backend can be read and written and never attached again. -//! The fix is local and small — keep an `Rc` for the async half and -//! give the `App` a newtype over a clone of it — but it cannot live in -//! `improvise-io`'s callers generically, because `impl Storage for Rc` -//! is two foreign types and the orphan rule refuses it. Hence [`Shared`]. -//! Reported as a VFS gap rather than routed around silently. +//! So the worker keeps an `Rc` for the async half and gives the `App` a +//! clone of it for the sync half. +//! +//! That clone is a `Storage` because `improvise-io` says so +//! (`impl Storage for Rc`, improvise-hmws). This module used to +//! carry a forwarding newtype, because `impl Storage for Rc` *here* is +//! two foreign types and the orphan rule refuses it — and so would every +//! worker written after this one. In `improvise-io` the trait is local and the +//! impl is allowed, so the seam costs a `Rc::clone` and nothing else. //! //! # `attach` before anything //! @@ -36,42 +40,17 @@ //! that is the seam being explicit rather than a lazy `attach` hidden inside a //! synchronous call, which could not await anything anyway. -use std::path::{Path, PathBuf}; +use std::path::PathBuf; use std::rc::Rc; use anyhow::Result; use improvise_command::ui::app::App; use improvise_io::opfs::OpfsFs; -use improvise_io::storage::Storage; use wasm_bindgen::prelude::*; use crate::vfs::{self, Opening}; use crate::worker::Worker; -/// An `OpfsFs` the worker and the `App` both hold. -/// -/// The `App` needs a `Box` while the message loop needs the -/// backend's four async methods, and the trait carries neither the methods nor -/// a way to downcast back. A local newtype over the shared pointer is the -/// smallest thing that gives both; `Rc` rather than `Arc` because a worker is -/// one thread and `wasm32-unknown-unknown` has no others. -struct Shared(Rc); - -impl Storage for Shared { - fn read(&self, path: &Path) -> Result> { - self.0.read(path) - } - fn write(&self, path: &Path, bytes: &[u8]) -> Result<()> { - self.0.write(path, bytes) - } - fn list(&self, dir: &Path) -> Result> { - self.0.list(dir) - } - fn delete(&self, path: &Path) -> Result<()> { - self.0.delete(path) - } -} - /// The worker's handle for the lifetime of the page. #[wasm_bindgen] pub struct WorkerServer { @@ -121,9 +100,9 @@ async fn boot(document: &str, seed: Option<&[u8]>) -> Result { Some(bytes) => Opening::named(path.clone()).with_seed(bytes), None => Opening::named(path.clone()), }; - let workbook = vfs::open(&Shared(Rc::clone(&storage)), &opening)?; + let workbook = vfs::open(&*storage, &opening)?; - let mut app = App::with_storage(workbook, Some(path), Box::new(Shared(Rc::clone(&storage)))); + let mut app = App::with_storage(workbook, Some(path), Box::new(Rc::clone(&storage))); app.rebuild_layout(); Ok(WorkerServer { worker: Worker::new(app), @@ -159,7 +138,7 @@ impl WorkerServer { /// (improvise-i34): there is no OS file manager behind this. #[wasm_bindgen] pub fn documents(&self) -> Result, JsError> { - vfs::documents(&Shared(Rc::clone(&self.storage))) + vfs::documents(&*self.storage) .map(|paths| { paths .iter() @@ -175,11 +154,3 @@ impl WorkerServer { self.storage.refresh().await.map_err(report) } } - -/// Assert the shim is reachable at all. Not a test of behaviour — there is -/// nothing here to test without a browser — but it does keep the module from -/// rotting out of the wasm build unnoticed. -#[allow(dead_code)] -fn _shim_is_a_storage(fs: Rc) -> Box { - Box::new(Shared(fs)) -} From 6c7b728c123e65a56b335d6893355a1922395932 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:30 -0700 Subject: [PATCH 06/72] fix(import): proposed items come back in first-seen order, not hash order MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-7yx item 2 asked for the ambiguous-date heuristic to be pinned and for the doc-comment to say whether its answer is right. Pinning it turned up a worse bug one line away. `analyze_records` collected each string column's distinct values into a `HashSet` and then iterated it. That is gotcha 18 in a second place — an index whose values are hashed is for lookup, never for iteration — and it reaches further here than tidiness, because `build_model` adds items in exactly that order, `IndexMap` keeps insertion order and persistence relies on it. The same CSV imported twice produced two different `.improv` files and two different row orders on screen. A probe saw five different orders in five runs of one process, each `HashSet` having its own `RandomState`. It also reached the date heuristic, which is the symptom a user could not have explained: `detect_date_format` tests only the first ten samples, so *which* ten it saw varied, and a column whose eleventh value is the one ruling out `%m/%d/%Y` detected a different format from run to run — every date in it off by the difference between an American and a European reading. The fix is a filtered `insert` over the values in order: the set still deduplicates and is no longer iterated. First-seen order is the records' own order, which is the one a user can predict. Four tests, all watched failing first (the two determinism ones fail on the parent commit; the two heuristic ones pass there and pin behaviour): - `the_distinct_values_are_in_first_seen_order` — asserts the exact sequence, five times, so a hash order cannot pass by luck. - `the_detected_date_format_does_not_depend_on_the_hash_seed` — thirteen values, twelve ambiguous and one day-first-only, so before the fix the answer flipped with roughly even odds. - `an_ambiguous_date_is_read_american_first` pins the first-match answer for `01/02/2025`-style input, with the day-first reading asserted to parse too so that it is a statement about ambiguity and not about one format fitting. The doc-comment says plainly that this silently mis-reads a European CSV and that what makes it tolerable is the wizard showing the user the detected format. - `a_day_first_only_value_settles_the_ambiguity` is the discriminating other arm: where the data can tell the readings apart, it does. The ten-sample truncation is unchanged and is still a real limit; it is now a deterministic one. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/import/analyzer.rs | 115 ++++++++++++++++++++- 1 file changed, 113 insertions(+), 2 deletions(-) diff --git a/crates/improvise-io/src/import/analyzer.rs b/crates/improvise-io/src/import/analyzer.rs index 1db61462..19255d83 100644 --- a/crates/improvise-io/src/import/analyzer.rs +++ b/crates/improvise-io/src/import/analyzer.rs @@ -133,8 +133,17 @@ pub fn analyze_records(records: &[Value]) -> Vec { } if all_string { - let distinct: HashSet<&str> = values.iter().filter_map(|v| v.as_str()).collect(); - let distinct_vec: Vec = distinct.into_iter().map(String::from).collect(); + // First-seen order, not hash order: `build_model` adds items in + // this order and `IndexMap` keeps it, so a hash order made the + // same CSV import to a different document every time (gotcha + // 18). The set is here to deduplicate and never to iterate. + let mut seen: HashSet<&str> = HashSet::new(); + let distinct_vec: Vec = values + .iter() + .filter_map(|v| v.as_str()) + .filter(|s| seen.insert(s)) + .map(String::from) + .collect(); let n = distinct_vec.len(); // Try chrono-based date detection @@ -289,4 +298,106 @@ mod tests { assert_eq!(date_prop.kind, FieldKind::TimeCategory); assert_eq!(date_prop.date_format, Some("%m/%d/%Y".to_string())); } + + /// `01/02/2025` and `02/03/2025` parse under `%m/%d/%Y` and under + /// `%d/%m/%Y`, and nothing in the data can tell them apart. The heuristic + /// is first-match over `DATE_FORMATS`, so the American reading wins. + /// + /// That is *a* defensible answer and not obviously the right one — a + /// European CSV imports with its days and months swapped, silently, and + /// the derived `_Month`/`_Quarter` categories inherit the error. What + /// makes it tolerable is that the wizard shows the detected format and + /// lets the user change it, which is the same argument + /// `detect_short_year_format` already records. It is pinned here so that + /// changing the answer is a decision somebody makes rather than a side + /// effect of reordering the table. + #[test] + fn an_ambiguous_date_is_read_american_first() { + let samples = vec!["01/02/2025", "02/03/2025"]; + assert_eq!(detect_date_format(&samples), Some("%m/%d/%Y".to_string())); + + // The precondition that makes this a statement about ambiguity rather + // than about one format simply fitting: the other reading fits too. + for s in &samples { + assert!( + NaiveDate::parse_from_str(s, "%d/%m/%Y").is_ok(), + "{s} must also parse day-first, or there is no ambiguity to resolve" + ); + } + } + + /// Where the two readings genuinely *can* be told apart, the unambiguous + /// value decides — first match still, but `%m/%d/%Y` now fails. + #[test] + fn a_day_first_only_value_settles_the_ambiguity() { + let samples = vec!["01/02/2025", "25/12/2025"]; + assert_eq!(detect_date_format(&samples), Some("%d/%m/%Y".to_string())); + } + + /// improvise-7yx, found while pinning the above: `analyze_records` built + /// its distinct values by collecting into a `HashSet` and iterating it, so + /// the order was a different one in every process. This is gotcha 18's bug + /// in a second place, and it reaches further here than a tidy-up: + /// `build_model` adds items in exactly this order, `IndexMap` preserves + /// insertion order, and persistence relies on it — so the *same CSV* + /// imported twice produced two different `.improv` files and two different + /// row orders on screen. + /// + /// The answer is first-seen order: deterministic, and the order the + /// records themselves are in, which is the one a user can predict. + /// + /// Five runs is enough to see it because each `HashSet` gets its own + /// `RandomState`; the pre-fix code produced five different orders here. + #[test] + fn the_distinct_values_are_in_first_seen_order() { + let vals = [ + "hotel", "alpha", "golf", "bravo", "foxtrot", "charlie", "echo", "delta", + ]; + let records: Vec = vals + .iter() + .map(|v| serde_json::json!({"Name": v, "Amount": 1})) + .collect(); + + for _ in 0..5 { + let proposals = analyze_records(&records); + let name = proposals.iter().find(|p| p.field == "Name").unwrap(); + assert_eq!( + name.distinct_values, vals, + "the proposed items must be the records' own order" + ); + } + } + + /// The consequence of the same bug that a user would have found hardest to + /// explain. `detect_date_format` tests only the first ten samples, so when + /// the sample order was a hash order, *which* ten it saw varied — and with + /// it the detected format, whenever the eleventh-and-later values are the + /// ones that rule a format out. + /// + /// Here twelve values are ambiguous and one (`25/12/2025`) is day-first + /// only. In hash order it lands inside the first ten about half the time, + /// so the column imported as `%d/%m/%Y` on some runs and `%m/%d/%Y` on + /// others — every date in it off by the difference. In first-seen order it + /// is last, deliberately: the answer is `%m/%d/%Y` *stably*, which is the + /// property being asserted. The truncation itself is unchanged and is + /// still a real limit. + #[test] + fn the_detected_date_format_does_not_depend_on_the_hash_seed() { + let mut vals: Vec = (1..=12).map(|d| format!("01/{d:02}/2025")).collect(); + vals.push("25/12/2025".to_string()); + assert!( + vals.len() > 10, + "precondition: more values than detect_date_format looks at" + ); + let records: Vec = vals + .iter() + .map(|v| serde_json::json!({"When": v})) + .collect(); + + for _ in 0..5 { + let proposals = analyze_records(&records); + let when = proposals.iter().find(|p| p.field == "When").unwrap(); + assert_eq!(when.date_format, Some("%m/%d/%Y".to_string())); + } + } } From 37d06fcc13504a7aa42562bb90fee0105c9b9ccb Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:35 -0700 Subject: [PATCH 07/72] fix(opfs): list answers in the same path shape as every other backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-vhcz. `OpfsFs::list` returned bare entry names where `MemoryFs` and `PhysicalFs` return `dir`-joined ones — paths a caller can hand straight back to `read`. At `Path::new("")` the two shapes coincide, and the flat root is the only `dir` the worker ever passes, so the disagreement was invisible: a worker that organised documents under `docs/` would have got a **full picker in every host test and an empty one in a browser**, with nothing saying why. The ticket asked which of the doc and the impl was wrong. It is the impl. The trait already says "backends that are not trees treat `dir` as a name prefix", which is exactly what this backend needs to do and was not doing — it ignored `dir` on the way out as well as on the way in. `dir` stays decoration rather than becoming a filter, because the backend is flat by construction: `entry_name` reduces every path to its file name, and an OPFS entry name may not contain a separator at all, so `docs/a.improv` and `a.improv` are the same entry. Joining makes `list` agree with `read`/`write`/`delete` about which names exist; filtering would make it disagree. `vfs.rs`'s "the root is flat" section is rewritten rather than deleted: the flat root is still load-bearing, but for the collapse of `dir`, not for a `list` shape that no longer differs. No host test can demonstrate this — `OpfsFs` is `cfg(target_arch = "wasm32")` and has never been run anywhere. The regression guard is the browser harness in the next commit, which runs `storage_contract` against `OpfsFs` at `docs/` as well as at the root; reverting this line fails it with `lists the one document`. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/opfs.rs | 21 ++++++++++++++-- crates/improvise-worker-server/src/vfs.rs | 30 +++++++++++++---------- 2 files changed, 36 insertions(+), 15 deletions(-) diff --git a/crates/improvise-io/src/opfs.rs b/crates/improvise-io/src/opfs.rs index 37afa2e3..b59e0033 100644 --- a/crates/improvise-io/src/opfs.rs +++ b/crates/improvise-io/src/opfs.rs @@ -232,8 +232,25 @@ impl Storage for OpfsFs { }) } - fn list(&self, _dir: &Path) -> Result> { - Ok(self.index.borrow().iter().map(PathBuf::from).collect()) + /// Entries in the origin root, presented under `dir`. + /// + /// The join is not cosmetic (improvise-vhcz). `MemoryFs` and `PhysicalFs` + /// answer `list` with `dir`-joined paths — paths a caller can hand + /// straight back to `read` — and this used to answer with bare names. At + /// `Path::new("")` the two shapes coincide, which is the only `dir` the + /// worker ever passes, so the disagreement was invisible: a worker that + /// organised documents under `docs/` would have got a **full picker in + /// every host test and an empty one in a browser**, and nothing would have + /// said why. + /// + /// `dir` is decoration rather than a filter because this backend is flat + /// by construction: [`entry_name`] reduces every path to its file name, so + /// `docs/a.improv` and `a.improv` are the same OPFS entry, and an entry + /// name may not contain a separator at all. That is the trait's own + /// "backends that are not trees treat `dir` as a name prefix" — the doc + /// was right and the impl was wrong. + fn list(&self, dir: &Path) -> Result> { + Ok(self.index.borrow().iter().map(|n| dir.join(n)).collect()) } fn delete(&self, path: &Path) -> Result<()> { diff --git a/crates/improvise-worker-server/src/vfs.rs b/crates/improvise-worker-server/src/vfs.rs index b6d41ee5..cf23a76c 100644 --- a/crates/improvise-worker-server/src/vfs.rs +++ b/crates/improvise-worker-server/src/vfs.rs @@ -41,13 +41,18 @@ //! //! [`root`] is `Path::new("")`, so a document name is a bare `demo.improv` //! whose `parent()` is the root. `OpfsFs` flattens by construction — it stores -//! one entry per document in the origin's root directory and its `list` -//! returns bare names — so the flat root is the one `dir` at which `OpfsFs` -//! and `MemoryFs` answer `list` identically, and therefore the one at which -//! `storage_contract` describes both. See `the_flat_root_is_where_the_backends -//! _agree`, which is the closest a host-side test can get to exercising the -//! browser backend (improvise-xf1 is deferred, so `OpfsFs` remains -//! compile-checked and not browser-tested — gotcha 28). +//! one entry per document in the origin's root directory, and an OPFS entry +//! name may not contain a separator at all — so `docs/a.improv` and +//! `a.improv` are the *same* entry there and different ones in `MemoryFs`. +//! The flat root is the one `dir` at which that collapse cannot be observed. +//! +//! It used to be load-bearing for a second, sharper reason: `OpfsFs::list` +//! answered with bare names where the host backends answer with `dir`-joined +//! ones, so the flat root was the only `dir` at which `list` agreed at all. +//! That is fixed (improvise-vhcz) and the contract now describes all three +//! everywhere; `the_flat_root_is_where_the_backends_agree` below pins the +//! shape this worker relies on, and `harness/` runs the contract itself +//! against `OpfsFs` in a real browser (improvise-vvl3). use std::path::{Path, PathBuf}; @@ -272,11 +277,10 @@ mod tests { ); } - /// `documents` is `Storage::list` at [`root`], and the flat root is the - /// one `dir` at which `OpfsFs` — which stores bare names in the origin - /// root — and the host backends answer alike. Nothing here can reach a - /// browser (improvise-xf1 is deferred), so this pins the *shape* the - /// worker relies on rather than the backend. + /// `documents` is `Storage::list` at [`root`], and a flat root is what + /// keeps `OpfsFs`'s collapse of `dir` unobservable: it stores one entry + /// per document in the origin root. This pins the *shape* the worker + /// relies on; the backend itself is exercised by `harness/`. #[test] fn the_flat_root_is_where_the_backends_agree() { let fs = MemoryFs::new(); @@ -285,7 +289,7 @@ mod tests { assert_eq!( documents(&fs).unwrap(), vec![PathBuf::from("a.improv"), PathBuf::from("b.improv")], - "sorted bare names, exactly what OpfsFs::list returns" + "sorted names under the flat root, exactly what OpfsFs::list answers" ); assert_eq!( Path::new("a.improv").parent(), From d3fe4aa0414e3c530b6390f99ee6d8d5ef48c59c Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:53 -0700 Subject: [PATCH 08/72] fix(web): a socket that never opened did not disconnect The `close` listener wrote "disconnected" unconditionally, so a connection that failed loudly -- the `error` listener having already written "cannot reach -- is improvise-ws-server running?" -- was immediately followed by a close event replacing it with the vaguer of the two facts. The user saw the least informative thing that happened. This is the watchdog bug's family and deliberately NOT its fix. The watchdog is for silence only, so it is disarmed; suppressing `close` outright would hide a genuine mid-session disconnect, which is the one thing that listener is really for. So the two cases are split rather than one being guarded. A CORRECTION TO THE TICKET'S SUGGESTED MECHANISM: it proposed reading `readyState` at close time, and that cannot work. `readyState` is already CLOSED by the time a close listener runs, whichever happened, and the CloseEvent cannot separate them either -- a refused connection and a mid-session network drop are both code 1006 with `wasClean: false`. The only discriminator is whether `open` ever fired, which the page must remember for itself. Hence the `opened` flag. Both arms go through `setShellFailure`, because a drop before the first frame leaves the watchdog armed and "disconnected" is more precise than the guess it would fire 8s later. The header comment stating the `#status` vocabulary is updated in the same commit: that fact is stated twice, and gotcha 48 says changing it is a grep. VERIFICATION IS READING, and that is the whole budget. There is no browser and no headless DOM here, so this is `node --check` (proved discriminating by appending an unbalanced paren and watching it fail) plus confirmation that every call app.js makes is in the generated .d.ts at the right arity: on_frame/1, on_key_event/5, render_model/0, set_viewport/2, typed_cell/3. Both wasm bundles still build. The listener itself has never been executed. Co-Authored-By: Claude Opus 5 (1M context) --- web/app.js | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/web/app.js b/web/app.js index 03fee4d1..3be540c1 100644 --- a/web/app.js +++ b/web/app.js @@ -39,6 +39,9 @@ // Neither is reachable silently. Every transport writes what it is doing to // `#status` — "starting", "connected", "cannot reach …", "disconnected" — so // a peer that never arrives reads differently from one that went away. +// "disconnected" means exactly the second: a socket that opened and then +// closed. A socket that never opened keeps saying "cannot reach …", because +// the close event there is the tail of the failure and not news. import init, { Client } from "./pkg/improvise_wasm_client.js"; @@ -191,8 +194,17 @@ function workerTransport() { /** A websocket to `improvise-ws-server`: how a TUI-hosted document is shared. */ function socketTransport() { const socket = new WebSocket(WS_URL); + const unreachable = `cannot reach ${WS_URL} — is improvise-ws-server running?`; + // Did this socket ever open? The page has to record that itself, because + // nothing on the close event carries it: `readyState` is already CLOSED by + // the time a close listener runs, whichever happened, and a refused + // connection and a mid-session network drop are BOTH code 1006 with + // `wasClean: false`. So "never connected" and "was connected and went away" + // are distinguishable only by remembering. + let opened = false; const ready = new Promise((resolve) => { socket.addEventListener("open", () => { + opened = true; setShellStatus(`connected · ${WS_URL}`); resolve(); }); @@ -200,10 +212,23 @@ function socketTransport() { socket.addEventListener("message", (event) => onFrame(event.data)); // Without this, a connection that never opens left #status blank and // "server not running" read exactly like "still connecting" (improvise-hwpi). - socket.addEventListener("error", () => - setShellFailure(`cannot reach ${WS_URL} — is improvise-ws-server running?`), + socket.addEventListener("error", () => setShellFailure(unreachable)); + // A socket that never opened did not "disconnect" — it failed to connect, + // and the close event is the tail of that failure rather than a second, + // vaguer thing that happened (improvise-taxx). Writing "disconnected" here + // unconditionally replaced the `error` listener's precise message with the + // least informative of the two facts. + // + // Same family as the watchdog bug and NOT the same fix: the watchdog is + // disarmed because it is for silence only, whereas suppressing `close` + // outright would hide a genuine mid-session disconnect, which is the one + // thing this listener is really for. Hence the split rather than a guard. + // Both arms go through `setShellFailure`: a drop before the first frame + // leaves the watchdog armed, and "disconnected" is more precise than the + // guess it would fire 8s later. + socket.addEventListener("close", () => + setShellFailure(opened ? "disconnected" : unreachable), ); - socket.addEventListener("close", () => setShellStatus("disconnected")); setShellStatus(`connecting to ${WS_URL}…`); return { send: (text) => { From c669d124c336ce40da2a5957cdcb9edde27101d1 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:37:54 -0700 Subject: [PATCH 09/72] fix(core): live_records answers in the store's own order, not the hash's MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-2yc2. `DataStore::live_records` walked the `records: HashMap`, so it yielded a different order in every process and — as the new test demonstrates — in every instance, `RandomState` keying each fresh map differently. That is the defect improvise-shyt had just been fixed for, caught before a caller could inherit it: shyt reordered records-mode rows between sessions, drifted `SUM`/`AVG` in the last `f64` bits, and froze a drill snapshot against row indices that named different records next time. It had no callers, so the ticket asked for a decision rather than a fix. The decision is the store's existing order — a record sits where its first cell sits in the primary `IndexMap` — because `matching_positions` already answers "what order is this store in" that way, and a second, different order would be two answers to one question. It is derived from `cells` on each call rather than memoised on `RecordEntry`, since a stored position is a copy that a deleted-and-recreated record would hold while its cells moved. The rejected alternative is worth recording: "return something whose type admits no order". Every iterator has an order, and a type that merely disclaims one still hands the caller hash order — the guard living one level away from the hazard, which is how this class of bug returns. Cost moves from O(records) to O(cells); with no caller to benchmark, that is the defensible default, and the doc says what a hot-path caller should argue. Evidence: `live_records_come_back_in_the_stores_own_insertion_order` pins the order against an interleaved fixture (a record's second measure must not move it), `two_stores_built_alike_agree_on_record_order` is the cross-process half made visible in one process, and `a_live_records_surrogate_is_the_one_that_resolves_it` holds the pair together. The first two were run against the unfixed code and both failed. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-core/src/model/cell.rs | 125 +++++++++++++++++++++++- 1 file changed, 120 insertions(+), 5 deletions(-) diff --git a/crates/improvise-core/src/model/cell.rs b/crates/improvise-core/src/model/cell.rs index d928fcc1..b0efd12e 100644 --- a/crates/improvise-core/src/model/cell.rs +++ b/crates/improvise-core/src/model/cell.rs @@ -335,12 +335,46 @@ impl DataStore { } } - /// Every record that still holds at least one cell, with its surrogate. + /// Every record that still holds at least one cell, with its surrogate, + /// **in the store's own order**: a record sits where its first cell + /// sits in the primary `IndexMap`. + /// + /// The order is chosen, not inherited (improvise-2yc2). Walking + /// `records` would have yielded hash order, which differs between + /// processes *and* between two instances in one process — the defect + /// gotcha 18 records for `matching_cells`, where it reordered + /// records-mode rows between sessions, drifted `SUM`/`AVG` in the last + /// `f64` bits, and froze a drill snapshot against row indices that + /// named different records next time. This function had no callers, so + /// the order was still free to decide rather than to discover. + /// + /// What it is decided *to* matters less than that it is the store's + /// existing one. `cells`' insertion order is already the answer + /// `matching_positions` gives, so projecting it onto records leaves one + /// answer to "what order is this store in" rather than two that are + /// free to disagree. It is derived from `cells` on each call rather + /// than memoised on [`RecordEntry`], for the same reason: a stored + /// position is a copy, and a record deleted and re-created would hold + /// its old one while its cells moved to the end. + /// + /// Note what is *not* on offer: an unordered return. Every iterator has + /// an order, and a type that merely disclaims one still hands the + /// caller hash order — the guard living one level away from the hazard, + /// which is the shape this whole class of bug returns through. + /// + /// Cost is O(cells) rather than O(records); with no caller to benchmark + /// against, correctness is the defensible default, and a hot-path + /// caller can argue for a memoised position with numbers. pub fn live_records(&self) -> impl Iterator { - self.records - .iter() - .filter(|(_, entry)| entry.is_live()) - .map(|(hash, entry)| (*hash, entry.record())) + let mut seen: HashSet = HashSet::new(); + self.cells.keys().filter_map(move |ikey| { + let hash = RecordHash::of(&self.to_cell_key(ikey)); + seen.insert(hash) + .then(|| self.records.get(&hash)) + .flatten() + .filter(|entry| entry.is_live()) + .map(|entry| (hash, entry.record())) + }) } /// Positions in the primary `cells` map of every cell whose coordinates @@ -380,6 +414,7 @@ impl DataStore { } sets.sort_by_key(|s| s.len()); + let first = sets[0]; let rest = &sets[1..]; @@ -685,6 +720,7 @@ mod data_store { (0..REGIONS.len()).map(|n| n as f64).collect::>() ); } + } #[cfg(test)] @@ -1015,6 +1051,85 @@ mod record_index { assert_eq!(store.resolve_record(stranger), None); } + /// improvise-2yc2. `live_records` walked the `records: HashMap`, so it + /// yielded a different order in every process and in every instance — + /// the defect improvise-shyt had just been fixed for, caught here + /// before a caller could inherit it. + /// + /// The order decided on is the store's own: a record sits where its + /// first cell sits in the primary `IndexMap`. That is not a new rule, + /// it is the one gotcha 18 already settled for `matching_cells` and + /// `matching_values`, projected onto records — a second, different + /// order would be two answers to one question. + /// + /// The fixture interleaves the measures deliberately: a record's + /// position is where it *first* appeared, so East's second measure + /// arriving after West must not move East behind it. + #[test] + fn live_records_come_back_in_the_stores_own_insertion_order() { + let mut store = DataStore::new(); + let east = key(&[("Region", "East")]); + let west = key(&[("Region", "West")]); + let north = key(&[("Region", "North")]); + for (record, measure, value) in [ + (&east, "Revenue", 1200.0), + (&west, "Revenue", 900.0), + (&east, "Cost", 400.0), + (&north, "Revenue", 700.0), + (&west, "Cost", 300.0), + ] { + let mut coords = record.0.clone(); + coords.push(("_Measure".to_string(), measure.to_string())); + store.set(CellKey::new(coords), CellValue::Number(value)); + } + + let order: Vec<&CellKey> = store.live_records().map(|(_, record)| record).collect(); + assert_eq!( + order, + vec![&east, &west, &north], + "a record sits where its first cell sits" + ); + } + + /// The cross-process half of the same decision, stated as something a + /// single-process test can see: `RandomState` keys each fresh `HashMap` + /// differently, so two stores built identically disagreed under the + /// hash walk. Nine records, because two small maps can agree by luck. + #[test] + fn two_stores_built_alike_agree_on_record_order() { + let build = || { + let mut store = DataStore::new(); + for n in 0..9 { + store.set( + key(&[("Region", &format!("R{n}")), ("_Measure", "Revenue")]), + CellValue::Number(f64::from(n)), + ); + } + store + }; + let (one, other) = (build(), build()); + let names = |store: &DataStore| -> Vec { + store + .live_records() + .map(|(_, record)| record.0[0].1.clone()) + .collect() + }; + assert_eq!(names(&one), names(&other)); + } + + /// The pair `live_records` yields is not two facts that could drift + /// apart: the surrogate is the record's own, so it is the one that + /// resolves it back. + #[test] + fn a_live_records_surrogate_is_the_one_that_resolves_it() { + let store = store(); + assert_eq!(store.live_records().count(), 2); + for (hash, record) in store.live_records() { + assert_eq!(hash, RecordHash::of(record)); + assert_eq!(store.resolve_record(hash), Some(record)); + } + } + /// The index is derived state, so it must come back with the data /// rather than being written alongside it. #[test] From e8296162acb42b9f007627ec34572a5a57cda314 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:38:22 -0700 Subject: [PATCH 10/72] fix: three totality gaps, each one refactor away from being reachable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-hkt6. Three findings from the 2026-09-19 wave review. None is a bug today; each is a guard that has drifted away from the thing it guards, which is how this class of bug comes back. (a) `DataStore::matching_positions` indexed `sets[0]` and panicked on an empty partial. Both public callers guard with `if partial.is_empty()`, so it was unreachable — with the guard one level away from the indexing. The fix is not `vec![]`: an empty partial constrains nothing, so *every* cell matches, and answering "none" down here would have made deleting a caller's fast path silently flip "all" to "none". The callers keep theirs as a fast path, and the test asserts the two doors agree rather than merely that nothing panicked. (b) `GridCache::invalidate_relabelled` blanked the cells under a renamed column and left its **total** standing. Gotcha 33(b)'s own argument says preserve-by-position is right only where the label did not change, and `Invalidated` already blanks the totals because a stale sum under fresh cells is the one number a user takes at face value. Unreachable from `grid_updates`, which diffs the totals and sends them in the same batch — which is exactly the point: 33(b) exists to stop `apply` trusting a producer to remember. Rows are the negative arm and stay untouched: a total spans every row, so no one row frames it. (c) `improvise_gp::mark_features` indexed `used[*feat]` bare where `expr_text` uses `.get().expect(...)` for the same precondition. Worse than inconsistent: `fitted_formula` runs `mark_features` first, so the legible message was the one nobody ever saw. Evidence, each proved by injecting the fault and watching it fail: `an_empty_partial_matches_every_cell_from_either_door` (index out of bounds, len 0 index 0), `a_relabelled_column_drops_its_total` with `a_relabelled_row_leaves_the_totals_alone` as the negative arm that stops "blank on any relabel" passing, and `a_tree_indexing_past_the_spec_says_so`, which asserts the message rather than the panic since an out-of-bounds index panics too. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-core/src/model/cell.rs | 58 ++++++++++++++++++--- crates/improvise-gp/src/lib.rs | 26 +++++++++- crates/improvise-protocol/src/render.rs | 69 +++++++++++++++++++++++++ 3 files changed, 144 insertions(+), 9 deletions(-) diff --git a/crates/improvise-core/src/model/cell.rs b/crates/improvise-core/src/model/cell.rs index b0efd12e..f26d47ea 100644 --- a/crates/improvise-core/src/model/cell.rs +++ b/crates/improvise-core/src/model/cell.rs @@ -393,6 +393,18 @@ impl DataStore { /// position with one hash lookup and the positions are sorted. That keeps /// the shape O(smallest set × |partial|) and adds O(m log m) over the /// *matches*, not a scan of the store. + /// + /// **An empty `partial` matches every cell**, vacuously, and this is + /// where that is answered (improvise-hkt6). It used to index `sets[0]` + /// and panic; both callers happened to guard, so it was unreachable — + /// but the guard lived one level away from the indexing, which is how + /// this class of bug returns. Note the guards could not simply be + /// pushed down as `vec![]`: "no coordinates constrain anything" means + /// *all* positions, and answering "none" here would have made removing + /// a caller's fast path change behaviour silently. The callers keep + /// theirs as a fast path, not as a guard, and + /// `an_empty_partial_matches_every_cell_from_either_door` holds the two + /// answers together. fn matching_positions(&self, partial: &[(String, String)]) -> Vec { // Intern the partial key (lookup only, no new symbols) let interned_partial: Vec<(Symbol, Symbol)> = partial @@ -415,14 +427,15 @@ impl DataStore { sets.sort_by_key(|s| s.len()); - let first = sets[0]; - let rest = &sets[1..]; - - let mut positions: Vec = first - .iter() - .filter(|ikey| rest.iter().all(|s| s.contains(*ikey))) - .filter_map(|ikey| self.cells.get_index_of(ikey)) - .collect(); + let mut positions: Vec = match sets.split_first() { + Some((first, rest)) => first + .iter() + .filter(|ikey| rest.iter().all(|s| s.contains(*ikey))) + .filter_map(|ikey| self.cells.get_index_of(ikey)) + .collect(), + // Nothing narrows, so nothing is excluded. + None => (0..self.cells.len()).collect(), + }; positions.sort_unstable(); positions } @@ -721,6 +734,35 @@ mod data_store { ); } + /// improvise-hkt6. `matching_positions` indexed `sets[0]` and panicked + /// on an empty partial. Both public callers guard, so it was + /// unreachable — but a guard one level away from the indexing is how + /// this returns, so the input the guards prevent is fed in directly + /// here. + /// + /// It asserts the *answer*, not merely the absence of a panic: an + /// empty partial constrains nothing, so every cell matches, and that + /// must be the same answer the callers' fast paths give. Returning + /// `vec![]` would also have been total and would have made deleting a + /// fast path silently flip "all" to "none". + #[test] + fn an_empty_partial_matches_every_cell_from_either_door() { + let store = store_of(®IONS); + let every = (0..REGIONS.len()).collect::>(); + assert_eq!(store.matching_positions(&[]), every, "one cell per region"); + + let through_positions: Vec<&CellValue> = store + .matching_positions(&[]) + .into_iter() + .filter_map(|n| store.cells.get_index(n)) + .map(|(_, value)| value) + .collect(); + assert_eq!( + through_positions, + store.matching_values(&[]), + "the fast path and the general path must agree" + ); + } } #[cfg(test)] diff --git a/crates/improvise-gp/src/lib.rs b/crates/improvise-gp/src/lib.rs index c3886eaa..6be66f02 100644 --- a/crates/improvise-gp/src/lib.rs +++ b/crates/improvise-gp/src/lib.rs @@ -177,9 +177,15 @@ fn unresolvable_features(tree: &Tree, spec: &Spec, model: &Model) -> Vec .collect() } +/// The precondition is `expr_text`'s, and it is spelled the same way here +/// on purpose (improvise-hkt6): a bare `used[*feat]` fails with +/// `index out of bounds`, which names neither the tree nor the spec. Two +/// sites, one failure, one sentence. fn mark_features(tree: &Tree, used: &mut [bool]) { if let Tree::Split { feat, lo, hi, .. } = tree { - used[*feat] = true; + *used + .get_mut(*feat) + .expect("tree indexes a feature the spec does not name") = true; mark_features(lo, used); mark_features(hi, used); } @@ -734,6 +740,24 @@ mod tests { assert!(err.contains("numeric measures"), "{err}"); } + /// improvise-hkt6. A tree whose `feat` runs past the spec violates + /// the same precondition in two places, and the two used to answer + /// differently: `expr_text` said what was wrong, `mark_features` + /// said `index out of bounds`. Whichever ran first decided what the + /// caller was told, and `fitted_formula` runs `mark_features` + /// first — so the legible message was the one nobody ever saw. + /// + /// Driven through `fitted_formula` because both private functions + /// are reached that way; the message is asserted, not merely the + /// panic, since an out-of-bounds index panics too. + #[test] + #[should_panic(expected = "tree indexes a feature the spec does not name")] + fn a_tree_indexing_past_the_spec_says_so() { + let spec = spec_of(&[(0.0, 200.0)], &["Cost"]); + let tree = split(7, 50.0, leaf(1.0), leaf(2.0)); + let _ = fitted_formula(&tree, &spec, TARGET, MEASURE, &document()); + } + /// The same refusal for the other two artefact families the ticket /// names alongside the one-hots. Nothing here enumerates them: the /// check asks the document whether the name resolves, which is the diff --git a/crates/improvise-protocol/src/render.rs b/crates/improvise-protocol/src/render.rs index 7e8e8633..4d3810ee 100644 --- a/crates/improvise-protocol/src/render.rs +++ b/crates/improvise-protocol/src/render.rs @@ -305,6 +305,18 @@ impl GridCache { /// Blank every cell framed by a label that is not the one that was /// there before, since the value under it answered the old label. + /// + /// A relabelled *column* drops its total on the same argument + /// (improvise-hkt6): the total under it is the sum of the old column, + /// and `Invalidated` already blanks the totals for exactly this reason + /// — a stale sum under fresh cells is the one number a user takes at + /// face value. Rows do not, because a total spans every row and is not + /// framed by any one of them. + /// + /// Unreachable from today's producer, which diffs + /// `mirror.totals != desired.totals` and emits `Totals` after + /// `ColLabels` in the same batch. That is precisely why it is fixed: + /// gotcha 33(b) exists to stop `apply` trusting a producer to remember. fn invalidate_relabelled(&mut self, old: &[String], axis: LabelAxis) { let new = match axis { LabelAxis::Row => &self.row_labels, @@ -326,6 +338,9 @@ impl GridCache { *cell = CellDisplay::default(); } } + if let Some(total) = self.totals.get_mut(i) { + *total = CellDisplay::default(); + } } } } @@ -976,6 +991,60 @@ mod tests { ); } + /// improvise-hkt6. `invalidate_relabelled` blanked the cells under a + /// renamed column and left its **total** standing, so a client that + /// heard `ColLabels` without a `Totals` behind it drew the old + /// column's sum under the new column's blanks. + /// + /// Unreachable from `grid_updates`, which always sends the changed + /// totals in the same batch — which is the point: gotcha 33(b) is the + /// rule that `apply` does not trust a producer to remember, and the + /// only way to test it is to send the update the producer never sends + /// alone. + /// + /// Three columns and a rename in the *middle*, so "blanked the right + /// one" is separable from "blanked all of them". + #[test] + fn a_relabelled_column_drops_its_total() { + let mut grid = GridCache::folded(&[ + GridUpdate::RowLabels(vec!["r0".into()]), + GridUpdate::ColLabels(vec!["A".into(), "B".into(), "C".into()]), + GridUpdate::Totals(vec![ + CellDisplay::number("10"), + CellDisplay::number("20"), + CellDisplay::number("30"), + ]), + ]); + grid.apply(&GridUpdate::ColLabels(vec![ + "A".into(), + "Bee".into(), + "C".into(), + ])); + + let texts: Vec<&str> = grid.totals.iter().map(|t| t.text.as_str()).collect(); + assert_eq!( + texts, + vec!["10", "", "30"], + "only the renamed column's sum answered a label that is gone" + ); + } + + /// The row half of the same rule is the *negative* arm, and it has to + /// stay: a total spans every row, so no single row frames it. Without + /// this, "blank the totals on any relabel" would pass the test above. + #[test] + fn a_relabelled_row_leaves_the_totals_alone() { + let mut grid = GridCache::folded(&[ + GridUpdate::RowLabels(vec!["r0".into(), "r1".into()]), + GridUpdate::ColLabels(vec!["A".into(), "B".into()]), + GridUpdate::Totals(vec![CellDisplay::number("10"), CellDisplay::number("20")]), + ]); + grid.apply(&GridUpdate::RowLabels(vec!["r0".into(), "renamed".into()])); + + let texts: Vec<&str> = grid.totals.iter().map(|t| t.text.as_str()).collect(); + assert_eq!(texts, vec!["10", "20"]); + } + /// The same rule on the other axis, and the case a bare label shrink /// gets right by accident: only the *relabelled* positions are blanked, /// so an unchanged row keeps what the server already sent for it. From d120596a10aa6e7ffdcb326f84a2e48e882f1f17 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:38:28 -0700 Subject: [PATCH 11/72] docs: the supported way to reach a TUI-hosted document over https Per the user's ruling on improvise-o8fa: document the reverse-proxy path properly, implement no TLS. The existing text already named the symptom correctly -- an https origin refuses `ws://` as mixed content, `improvise-ws-server` has no TLS, so `wss://` is unobtainable from it -- but it gestured at "a reverse proxy or a tunnel" and stopped, which is not something a reader can follow. So README gains a worked Caddy example (Caddyfile, both commands, and the `?ws=` value to reload with), the nginx difference that actually bites (`proxy_http_version 1.1` plus the Upgrade/Connection headers, without which the handshake is answered as an ordinary request), the cloudflared/ngrok one-liners, and the note that `--bind` should be left at loopback behind a proxy on the same host. It also says the cost rather than papering it: the documented path to sharing involves a second piece of software, which is a poor answer for a tool whose selling point is being one binary. And it says what that cost buys against -- the standalone worker deployment is the point of the Pages site and needs no server at all, and `http://localhost` pairing needs none of this either. Native `wss://` is recorded as DECLINED WITH REASON rather than left as a gap. The substance is the certificate, not rustls: a self-signed cert makes the browser refuse the socket with no usable error, because a websocket handshake has no interstitial to click through. A real cert or mkcert are the only pleasant options and both are a second piece of software too -- so the choice is not one binary versus two, it is two either way, with one option additionally shipping a TLS stack in a pivot-table binary. web/index.html is the second home for the same fact and is updated to point at the worked example rather than restating it (gotcha 48: a fact stated twice is a grep). Both were checked for text the correction left stale. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 75 +++++++++++++++++++++++++++++++++++++++++++++++--- web/index.html | 10 +++++-- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 79e5e003..1a0035ed 100644 --- a/README.md +++ b/README.md @@ -216,10 +216,77 @@ address as mixed content. An explicit `ws://` from an https page is upgraded rather than passed through — passing it through can only produce a failure that reads like "the server is not running". -So the local pairing above works as written. Reaching a server from the -**deployed** https page means terminating TLS in front of it (a reverse proxy -or a tunnel): `improvise-ws-server` speaks plain `ws://` only, and native -`wss://` is not implemented. +So the local pairing above works as written, and it is the case worth +optimising for: `http://localhost` is not a secure-context problem, so nothing +below is needed to develop, to demo locally, or to share a TUI-hosted document +with another tab on the same machine. + +### Sharing a TUI-hosted document with a remote browser + +Reaching `improvise-ws-server` from the **deployed** https page means +terminating TLS in front of it. `improvise-ws-server` speaks plain `ws://` +only — there is no rustls, no native-tls, nothing in the crate or its +dependencies — so `wss://` is not obtainable from the binary itself, and this +is the supported way to get it. + +**Say the cost first, because it is real.** This means the documented path to +sharing involves a second piece of software, which is a poor answer for a tool +whose selling point is that it is one binary. It is the answer anyway, for a +reason given under "Why not native `wss://`" below. If you only want *a* +document in *a* browser, you do not need any of this — that is the standalone +worker deployment, it is what the Pages site is for, and it needs no server +process at all. + +A worked example with [Caddy](https://caddyserver.com/), which obtains and +renews the certificate itself. Point a DNS name at the host, then: + +```caddyfile +# Caddyfile +improvise.example.com { + reverse_proxy 127.0.0.1:8787 +} +``` + +```sh +improvise-ws-server bank-info.improv --port 8787 # still plain ws, still loopback +caddy run # terminates TLS, proxies to it +``` + +Then, from , reload with +`?ws=improvise.example.com`. Caddy preserves the request path, so the +multi-document form works too: serve several files and name one with +`?ws=improvise.example.com/bank-info`. + +Caddy needs no websocket-specific configuration — `reverse_proxy` passes the +`Upgrade` handshake through. nginx does: it needs `proxy_http_version 1.1` and +the `Upgrade`/`Connection` headers set explicitly, or the handshake is +answered as an ordinary request and the socket never opens. + +For a one-off share with no DNS name and no certificate of your own, +`cloudflared tunnel --url http://127.0.0.1:8787` or `ngrok http 8787` each +print an https hostname that works the same way — pass the hostname to `?ws=` +without a scheme. + +Note the server binds `127.0.0.1` by default and `--bind` is the only way to +widen that. Behind a proxy on the same host you should *not* widen it: the +proxy reaches loopback, and leaving the default means the unencrypted port is +not reachable from anywhere else. + +### Why not native `wss://` + +Declined deliberately rather than merely unimplemented (improvise-o8fa). Adding +rustls is the easy half; the certificate is the hard one, and it is what makes +the feature worse than the proxy rather than better. A self-signed certificate +makes the browser refuse the socket outright — a websocket handshake has no +interstitial to click through, so the failure arrives as a bare connection +error with no usable indication that a certificate was the reason. That leaves +a real certificate or an explicitly trusted local CA (`mkcert`) as the only +pleasant options, and both of those are a second piece of software too. So the +choice is not "one binary versus two", it is "two either way, and one of the +options additionally ships a TLS stack and a certificate-loading surface in a +binary whose job is a pivot table". If sharing a TUI-hosted document to a +remote browser ever becomes a common case rather than an occasional one, this +is worth revisiting on that evidence. Caveat worth stating plainly: **no line of the browser JavaScript has ever run.** There is no browser and no headless DOM in the development environment, diff --git a/web/index.html b/web/index.html index 197a2128..03bde6fb 100644 --- a/web/index.html +++ b/web/index.html @@ -65,9 +65,13 @@ wss://, because this page is served over https and an https origin refuses a ws:// address as mixed content. Note that improvise-ws-server speaks plain - ws:// only, so reaching it from here means terminating TLS - in front of it. Served over http://localhost the same page - defaults to ws:// and pairs with it directly. + ws:// only, so reaching it from this page means + terminating TLS in front of it with a reverse proxy or a tunnel — the + README has a worked Caddy example. Served over + http://localhost the same page defaults to + ws:// and pairs with it directly, with nothing in between; + that is the case the pairing is really for. Sharing a document is what + the server is for — not running one is what everything above is for.

From 9881d11509cecc67fe3c2b5056b3f9e845d75af7 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:39:12 -0700 Subject: [PATCH 12/72] test(io): the adversarial CSV and corrupt-gzip cases improvise-7yx named MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Items 3 and 5 of improvise-7yx. Items 1 and 4 were already closed by improvise-jc8 and improvise-k8i respectively, as the 2026-09-19 grooming note records; item 2 landed in the previous commit. CSV (`csv_parser.rs`). A header name becomes a category name and survives into the `.improv` file, so a byte-order mark or a carriage return riding along in one is silent corruption rather than cosmetic. All four inputs already behave correctly, so each test carries a control that proves it can see the fault it is named for: - `a_leading_utf8_bom_is_not_part_of_the_first_header` asserts the input really begins `EF BB BF`, then that the key is `Region`. Its control puts a BOM in the *second* header, where it is not leading, is not stripped, and does show up in the key. - `crlf_line_endings_leave_no_carriage_return_in_the_data` asserts clean values and a numeric `Revenue` — a `1\r` would import every row as text. Its control is that a *lone* CR is itself a record terminator to this reader, splitting one row into two, so a mishandled CRLF would be visible. - `an_empty_file_is_no_records_and_no_warnings` and `a_header_with_no_rows_is_no_records` — the parser's job is to say truthfully that there was nothing there; refusing to build a model from nothing is the wizard's. - `a_single_column_file_yields_one_field_per_record` also asserts no warning, because a one-field row must not read as a short row. Gzip (`persistence/mod.rs`). `save_and_load_roundtrip_gzip` covered only the happy path. The two ways a `.gz` really arrives damaged are kept apart, because "you lost the tail" and "you cannot trust any of it" are different things: a truncated document is `ErrorKind::UnexpectedEof`, a flipped byte fails the checksum as `ErrorKind::InvalidInput`. Asserting the kind rather than the bare failure is what separates "the decoder refused damaged input" from "the loader stopped decompressing and choked on the compressed bytes as text", and the two tests assert *different* kinds through one helper, so neither can be vacuous. The helper asserts the corruption changed the bytes and left the gzip magic intact, so the failure is decompression and not misrouting. One line of non-test change, and the reason it is here: the gzip branch of `load_from` used a bare `?` where the UTF-8 branch two lines below carries `with_context`, so a user opening a corrupt `.gz` saw `incomplete deflate stream` and no filename. It now names the document, which is what the two tests assert alongside the kind. Proved discriminating by replacing the context with a constant string and watching both tests fail. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/import/csv_parser.rs | 103 +++++++++++++++++++ crates/improvise-io/src/persistence/mod.rs | 79 +++++++++++++- 2 files changed, 181 insertions(+), 1 deletion(-) diff --git a/crates/improvise-io/src/import/csv_parser.rs b/crates/improvise-io/src/import/csv_parser.rs index 2d216e63..d86081d8 100644 --- a/crates/improvise-io/src/import/csv_parser.rs +++ b/crates/improvise-io/src/import/csv_parser.rs @@ -359,6 +359,109 @@ mod tests { assert!(warnings.is_empty()); } + // ── Adversarial inputs (improvise-7yx item 3) ───────────────────── + // + // A header name becomes a *category name* and survives into the `.improv` + // file, so anything that rides along invisibly in one — a byte-order mark, + // a carriage return — is silent corruption rather than cosmetic. These + // four inputs are what a spreadsheet actually exports. + + /// Excel writes a UTF-8 BOM. The `csv` crate strips a leading one, so the + /// first header is `Region` and not `\u{feff}Region`. + /// + /// The control is what makes this an assertion rather than a tautology: a + /// BOM in the *second* header is not leading, is not stripped, and shows + /// up in the key. So the check can see a BOM where there is one. + #[test] + fn a_leading_utf8_bom_is_not_part_of_the_first_header() { + let input = "\u{feff}Region,Revenue\nEast,1\n"; + assert_eq!( + &input.as_bytes()[..3], + b"\xef\xbb\xbf", + "precondition: the input really carries a BOM" + ); + + let (records, warnings) = parse_csv_bytes_with_warnings(input.as_bytes()).unwrap(); + assert!(warnings.is_empty()); + assert_eq!(records.len(), 1); + let keys: Vec<&String> = records[0].as_object().unwrap().keys().collect(); + assert_eq!(keys, ["Region", "Revenue"]); + + let (control, _) = + parse_csv_bytes_with_warnings("Region,\u{feff}Revenue\nEast,1\n".as_bytes()).unwrap(); + assert!( + control[0].as_object().unwrap().contains_key("\u{feff}Revenue"), + "control: a non-leading BOM survives, so the assertion above can see one" + ); + } + + /// Windows line endings. The `\r` belongs to the terminator, not to the + /// last field of the row — a `Revenue` of `1\r` would be a string, not a + /// number, and every row would import as text. + /// + /// The control: a *lone* `\r` is itself a record terminator to this + /// reader, so `Ea\rst` splits one row into two. That is what a stray + /// carriage return does when it is not part of a CRLF pair, and it is why + /// "two records, clean values" below is a real observation. + #[test] + fn crlf_line_endings_leave_no_carriage_return_in_the_data() { + let (records, warnings) = + parse_csv_bytes_with_warnings(b"Region,Revenue\r\nEast,1\r\nWest,2\r\n").unwrap(); + assert!(warnings.is_empty()); + assert_eq!(records.len(), 2); + assert_eq!(records[0]["Region"], Value::String("East".to_string())); + assert_eq!( + records[0]["Revenue"], + Value::Number(serde_json::Number::from(1)) + ); + assert_eq!(records[1]["Region"], Value::String("West".to_string())); + + let (control, _) = parse_csv_bytes_with_warnings(b"Region,Revenue\nEa\rst,1\n").unwrap(); + assert_eq!( + control.len(), + 2, + "control: a lone CR terminates a record here, so a mishandled CRLF would show" + ); + } + + /// An empty file is no records and no complaint. The wizard's own + /// preconditions are what refuse to build a model from nothing + /// (`wizard.rs::validation_error`); the parser's job is to say truthfully + /// that there was nothing there. + #[test] + fn an_empty_file_is_no_records_and_no_warnings() { + let (records, warnings) = parse_csv_bytes_with_warnings(b"").unwrap(); + assert!(records.is_empty()); + assert!(warnings.is_empty()); + } + + /// A header with no rows under it is the same answer, and is the case a + /// user actually hits — an export filtered down to nothing. + #[test] + fn a_header_with_no_rows_is_no_records() { + let (records, warnings) = parse_csv_bytes_with_warnings(b"Region,Revenue\n").unwrap(); + assert!(records.is_empty()); + assert!(warnings.is_empty()); + } + + /// One column is the degenerate shape for everything downstream that + /// splits a row into coordinates and a measure: there is one field, it is + /// present, and it is not mistaken for a short row. + #[test] + fn a_single_column_file_yields_one_field_per_record() { + let (records, warnings) = parse_csv_bytes_with_warnings(b"Region\nEast\nWest\n").unwrap(); + assert!( + warnings.is_empty(), + "a full one-column row is not a short row: {warnings:?}" + ); + assert_eq!(records.len(), 2); + for (record, expected) in records.iter().zip(["East", "West"]) { + let map = record.as_object().unwrap(); + assert_eq!(map.len(), 1); + assert_eq!(map["Region"], Value::String(expected.to_string())); + } + } + #[test] fn parse_checking_csv_format() { // Simulates the format of /Users/edwlan/Downloads/Checking1.csv diff --git a/crates/improvise-io/src/persistence/mod.rs b/crates/improvise-io/src/persistence/mod.rs index 997aabbb..9978f933 100644 --- a/crates/improvise-io/src/persistence/mod.rs +++ b/crates/improvise-io/src/persistence/mod.rs @@ -88,7 +88,9 @@ pub fn load_from(storage: &dyn Storage, path: &Path) -> Result { let bytes = storage.read(path)?; let text = if is_gzip(path) { let mut s = String::new(); - GzDecoder::new(BufReader::new(&bytes[..])).read_to_string(&mut s)?; + GzDecoder::new(BufReader::new(&bytes[..])) + .read_to_string(&mut s) + .with_context(|| format!("{} is not readable gzip", path.display()))?; s } else { String::from_utf8(bytes).with_context(|| format!("{} is not UTF-8", path.display()))? @@ -245,6 +247,81 @@ mod tests { ); } + // ── gzip corruption (improvise-7yx item 5) ────────────────────────── + // + // `save_and_load_roundtrip_gzip` above covers only the happy path. The + // two ways a `.gz` document actually arrives damaged are a truncated + // write (the disk filled, or the process died mid-save) and a flipped + // byte, and they are worth keeping apart: the first is *incomplete* and + // the second is *wrong*, which is the difference between "you lost the + // tail" and "you cannot trust any of it". + + /// Writes `good` gzip bytes, hands `corrupt` to the loader in their place + /// and returns the error — after asserting that the corruption left the + /// bytes still looking like the gzip document `load_from` will route to + /// the decoder, so the failure is decompression and not misrouting. + fn load_corrupt_gzip(corrupt: impl Fn(&mut Vec)) -> anyhow::Error { + use crate::storage::{MemoryFs, Storage}; + use std::path::Path; + + let fs = MemoryFs::new(); + let path = Path::new("model.improv.gz"); + super::save_to(&fs, &storage_fixture(), path).unwrap(); + + let mut bytes = fs.read(path).unwrap(); + let before = bytes.clone(); + corrupt(&mut bytes); + assert_ne!(bytes, before, "the corruption must change the bytes"); + assert_eq!( + &bytes[..2], + &[0x1f, 0x8b], + "precondition: still a gzip document by its magic bytes" + ); + + fs.write(path, &bytes).unwrap(); + super::load_from(&fs, path).expect_err("corrupt gzip must not load") + } + + /// A `.gz` document cut short surfaces as an incomplete stream, named as + /// `UnexpectedEof`. The kind is asserted rather than merely the failure, + /// because "an error occurred" would also pass if the loader had stopped + /// decompressing altogether and choked on the compressed bytes as text. + #[test] + fn a_truncated_gzip_document_fails_as_an_unexpected_eof() { + let err = load_corrupt_gzip(|bytes| { + let half = bytes.len() / 2; + bytes.truncate(half); + }); + let io = err + .downcast_ref::() + .unwrap_or_else(|| panic!("expected an io::Error, got {err:?}")); + assert_eq!(io.kind(), std::io::ErrorKind::UnexpectedEof); + assert!( + err.to_string().contains("model.improv.gz"), + "the message must name the document, got {err}" + ); + } + + /// A flipped byte fails the gzip checksum instead — a different kind, + /// `InvalidInput`, and that distinction is the reason to test both. A + /// single test against one corruption could not tell "the decoder + /// refuses damaged input" from "the decoder refuses short input". + #[test] + fn a_gzip_document_with_a_flipped_byte_fails_its_checksum() { + let err = load_corrupt_gzip(|bytes| { + let mid = bytes.len() / 2; + bytes[mid] ^= 0xff; + }); + let io = err + .downcast_ref::() + .unwrap_or_else(|| panic!("expected an io::Error, got {err:?}")); + assert_eq!(io.kind(), std::io::ErrorKind::InvalidInput); + assert!( + err.to_string().contains("model.improv.gz"), + "the message must name the document, got {err}" + ); + } + // ── autosave_path ─────────────────────────────────────────────────── #[test] From d215af7f23c40773ff62663740c4a1f4b1e386ed Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:39:15 -0700 Subject: [PATCH 13/72] test(core): the live_records order test needs two histories, not two calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review finding on this branch's own work, against `doctrine(an_order_test_needs_two_histories)`: a single-process test cannot tell "ordered correctly" from "hashed that way today, on this machine". The first version built two *identical* stores and required them to agree, which is the weaker shape — it can pass by luck under the bug. Replaced with the doctrine's form, which the sibling `the_same_cells_entered_in_opposite_orders_list_in_opposite_orders` already uses for `matching_cells`: the same logical records built by opposite insertion histories, each required to list its own, so one result is the reverse of the other. Under the hash walk the two orders were unrelated to both histories and to each other, and it fails on every run rather than usually — checked three times against the reinstated walk. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-core/src/model/cell.rs | 33 ++++++++++++++++--------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/crates/improvise-core/src/model/cell.rs b/crates/improvise-core/src/model/cell.rs index f26d47ea..a56e44e7 100644 --- a/crates/improvise-core/src/model/cell.rs +++ b/crates/improvise-core/src/model/cell.rs @@ -1133,30 +1133,41 @@ mod record_index { ); } - /// The cross-process half of the same decision, stated as something a - /// single-process test can see: `RandomState` keys each fresh `HashMap` - /// differently, so two stores built identically disagreed under the - /// hash walk. Nine records, because two small maps can agree by luck. + /// `doctrine(an_order_test_needs_two_histories)`: one history cannot + /// tell "ordered" from "hashed that way today, on this machine". Two + /// stores holding the *same* logical records, built by opposite + /// histories, must each list its own — so one result is the reverse of + /// the other, where a hash walk gave two orders unrelated to both + /// histories and to each other. + /// + /// This also happens to be the only visible trace of the cross-process + /// half: `RandomState` keys each fresh `HashMap` differently, so the + /// hash walk disagreed even between two stores in one process. #[test] - fn two_stores_built_alike_agree_on_record_order() { - let build = || { + fn the_same_records_entered_in_opposite_orders_list_in_opposite_orders() { + let regions = ["North", "South", "East", "West", "Central", "Coastal"]; + let store_of = |order: &[&str]| { let mut store = DataStore::new(); - for n in 0..9 { + for (n, region) in order.iter().enumerate() { store.set( - key(&[("Region", &format!("R{n}")), ("_Measure", "Revenue")]), - CellValue::Number(f64::from(n)), + key(&[("Region", region), ("_Measure", "Revenue")]), + CellValue::Number(n as f64), ); } store }; - let (one, other) = (build(), build()); let names = |store: &DataStore| -> Vec { store .live_records() .map(|(_, record)| record.0[0].1.clone()) .collect() }; - assert_eq!(names(&one), names(&other)); + + let mut backward = regions; + backward.reverse(); + let mut reversed = names(&store_of(&backward)); + reversed.reverse(); + assert_eq!(names(&store_of(®ions)), reversed); } /// The pair `live_records` yields is not two facts that could drift From 3d7fdf9f0131c48c15c515075d78e33246a2784f Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:39:19 -0700 Subject: [PATCH 14/72] feat(keymap): a save-as prompt, so a pathless document has somewhere to go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-8tk made ZZ/:wq refuse to quit when the save fails, which is correct and unhelpful: with no file path the user is left holding "No file path — use :w " and has to know to type a path inline. There was no save-as prompt to send them to. `AppMode::ExportPrompt` is not one — `CommitExport` writes CSV, so binding it here would have written a CSV file under whatever .improv name the user typed. This is the ticket's preferred option (a): a `SavePrompt` minibuffer mode built by the `TextEntry` builder, committing to `SaveAs` — `SaveAs` and not `Save`, because the prompt exists precisely when there is no path for `Save` to use, and because `SaveAs` adopts the path so the next Ctrl-S works. Bound to Ctrl-W in Normal, beside Ctrl-E's export prompt, and reachable as `:enter-mode save-prompt`. The status message now names it. `ZZ` is deliberately NOT routed into the prompt automatically; see the hand-back for the reasoning. The abort of improvise-8tk is untouched. Evidence. The binding dump diff is purely additive — one Normal binding (inherited by RecordsNormal) and the new mode's own keymap; no existing line moved. The mode is carried by the existing meta-tests rather than by new bespoke ones, and both were proved to reach it by injection: every_hint_key_resolves_in_its_mode fails when the new hint names an unbound Tab, and every_bound_command_name_is_registered fails when the commit name is typo'd. commit_save_writes_the_model_to_the_typed_path pins SaveAs-not-ExportCsv, and commit_save_on_an_empty_path_writes_nothing pins the empty-path bail. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/commit.rs | 80 +++++++++++++++++++ .../src/command/cmd/registry.rs | 1 + .../improvise-command/src/command/keymap.rs | 5 +- crates/improvise-command/src/ui/effect.rs | 2 +- crates/improvise-protocol/src/keymap.rs | 20 +++++ crates/improvise-protocol/src/lib.rs | 22 ++++- 6 files changed, 127 insertions(+), 3 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/commit.rs b/crates/improvise-command/src/command/cmd/commit.rs index 6dc55a1a..2421954b 100644 --- a/crates/improvise-command/src/command/cmd/commit.rs +++ b/crates/improvise-command/src/command/cmd/commit.rs @@ -202,6 +202,55 @@ mod tests { assert!(dbg.contains("ExportCsv"), "Expected ExportCsv, got: {dbg}"); assert!(dbg.contains("Normal"), "Expected Normal mode, got: {dbg}"); } + + /// improvise-lae: the save prompt has to commit to `SaveAs` and not to + /// `ExportCsv`, which is what makes it a separate mode from + /// `ExportPrompt` rather than a second binding onto it — `ExportCsv` + /// would write CSV under whatever `.improv` name the user typed. + /// + /// `SaveAs` and not `Save`, because `Save` writes the path the document + /// already has, and the whole reason this prompt exists is that there + /// isn't one. + #[test] + fn commit_save_writes_the_model_to_the_typed_path() { + let m = two_cat_model(); + let layout = make_layout(&m); + let reg = make_registry(); + let mut bufs = HashMap::new(); + bufs.insert("save".to_string(), "/tmp/typed.improv".to_string()); + let mut ctx = make_ctx(&m, &layout, ®); + ctx.buffers = &bufs; + let dbg = effects_debug(&CommitSave.execute(&ctx)); + assert!( + dbg.contains("SaveAs(\"/tmp/typed.improv\")"), + "Expected SaveAs at the typed path, got: {dbg}" + ); + assert!( + !dbg.contains("ExportCsv"), + "the save prompt must not export CSV, got: {dbg}" + ); + assert!(dbg.contains("Normal"), "Expected Normal mode, got: {dbg}"); + } + + /// An empty prompt must not reach `SaveAs`: `SaveAs(PathBuf::from(""))` + /// asks the backend to write to `""`, and the user then reads whatever + /// I/O error that produces instead of "you typed nothing". + #[test] + fn commit_save_on_an_empty_path_writes_nothing() { + let m = two_cat_model(); + let layout = make_layout(&m); + let reg = make_registry(); + let mut bufs = HashMap::new(); + bufs.insert("save".to_string(), " ".to_string()); + let mut ctx = make_ctx(&m, &layout, ®); + ctx.buffers = &bufs; + let dbg = effects_debug(&CommitSave.execute(&ctx)); + assert!( + !dbg.contains("SaveAs"), + "an empty path must not reach SaveAs, got: {dbg}" + ); + assert!(dbg.contains("SetStatus"), "Expected a status, got: {dbg}"); + } } // ── Commit commands (mode-specific buffer consumers) ──────────────────────── @@ -546,6 +595,37 @@ impl Cmd for CommitBuffer { } } +/// Commit a save path from the save buffer. +/// +/// The sibling of [`CommitExport`], and deliberately not the same command: +/// the export prompt writes CSV, this one writes the *model* through +/// `SaveAs`, which also adopts the path so a later plain `Save` has one +/// (improvise-lae). +/// +/// An empty path bails rather than reaching `SaveAs`, which would otherwise +/// ask the storage backend to write to `""` and report its error instead of +/// the one the user can act on. +#[derive(Debug)] +pub struct CommitSave; +impl Cmd for CommitSave { + fn name(&self) -> &'static str { + "commit-save" + } + fn execute(&self, ctx: &CmdContext) -> Vec> { + let path = read_buffer(ctx, "save").trim().to_string(); + if path.is_empty() { + return vec![ + effect::set_status("No path given — not saved"), + effect::change_mode(AppMode::Normal), + ]; + } + vec![ + Box::new(effect::SaveAs(std::path::PathBuf::from(path))), + effect::change_mode(AppMode::Normal), + ] + } +} + /// Commit an export from the export buffer. #[derive(Debug)] pub struct CommitExport; diff --git a/crates/improvise-command/src/command/cmd/registry.rs b/crates/improvise-command/src/command/cmd/registry.rs index f0e9e05b..47e51e0c 100644 --- a/crates/improvise-command/src/command/cmd/registry.rs +++ b/crates/improvise-command/src/command/cmd/registry.rs @@ -600,6 +600,7 @@ pub fn default_registry() -> CmdRegistry { }, ); r.register_nullary(|| Box::new(CommitExport)); + r.register_nullary(|| Box::new(CommitSave)); r.register_nullary(|| Box::new(ExecuteCommand)); // ── Wizard ─────────────────────────────────────────────────────────── diff --git a/crates/improvise-command/src/command/keymap.rs b/crates/improvise-command/src/command/keymap.rs index 318d73e2..d1e09bf6 100644 --- a/crates/improvise-command/src/command/keymap.rs +++ b/crates/improvise-command/src/command/keymap.rs @@ -603,6 +603,7 @@ mod tests { ModeKey::CategoryAdd => AppMode::category_add(), ModeKey::ItemAdd => AppMode::item_add("Type".to_string()), ModeKey::ExportPrompt => AppMode::export_prompt(), + ModeKey::SavePrompt => AppMode::save_prompt(), ModeKey::CommandMode => AppMode::command_mode(), ModeKey::ImportWizard => AppMode::ImportWizard, ModeKey::RecordsNormal => AppMode::RecordsNormal, @@ -805,6 +806,7 @@ mod tests { (AppMode::category_add(), COMMIT_STAY), (AppMode::item_add("Region".into()), COMMIT_STAY), (AppMode::export_prompt(), COMMIT_LEAVE), + (AppMode::save_prompt(), COMMIT_LEAVE), (AppMode::command_mode(), COMMIT_LEAVE), ] } @@ -1023,7 +1025,7 @@ mod binding_dump { #[cfg(test)] mod tests_support { use super::*; - pub const ALL_MODES: [ModeKey; 16] = [ + pub const ALL_MODES: [ModeKey; 17] = [ ModeKey::Normal, ModeKey::Help, ModeKey::FormulaPanel, @@ -1035,6 +1037,7 @@ mod tests_support { ModeKey::CategoryAdd, ModeKey::ItemAdd, ModeKey::ExportPrompt, + ModeKey::SavePrompt, ModeKey::CommandMode, ModeKey::SearchMode, ModeKey::ImportWizard, diff --git a/crates/improvise-command/src/ui/effect.rs b/crates/improvise-command/src/ui/effect.rs index 59c2d80f..73743287 100644 --- a/crates/improvise-command/src/ui/effect.rs +++ b/crates/improvise-command/src/ui/effect.rs @@ -769,7 +769,7 @@ fn save_to_current_path( state: &crate::ui::app::ModelState, ) -> Result { state.file_path.as_ref().map_or_else( - || Err("No file path — use :w ".to_string()), + || Err("No file path — Ctrl-W to name one, or :w ".to_string()), |path| { crate::persistence::save_to(storage, &state.workbook, path) .map(|()| format!("Saved to {}", path.display())) diff --git a/crates/improvise-protocol/src/keymap.rs b/crates/improvise-protocol/src/keymap.rs index cc14abb1..d6a7dc58 100644 --- a/crates/improvise-protocol/src/keymap.rs +++ b/crates/improvise-protocol/src/keymap.rs @@ -89,6 +89,7 @@ pub enum ModeKey { CategoryAdd, ItemAdd, ExportPrompt, + SavePrompt, CommandMode, SearchMode, ImportWizard, @@ -111,6 +112,7 @@ impl ModeKey { AppMode::CategoryAdd { .. } => Some(ModeKey::CategoryAdd), AppMode::ItemAdd { .. } => Some(ModeKey::ItemAdd), AppMode::ExportPrompt { .. } => Some(ModeKey::ExportPrompt), + AppMode::SavePrompt { .. } => Some(ModeKey::SavePrompt), AppMode::CommandMode { .. } => Some(ModeKey::CommandMode), AppMode::ImportWizard => Some(ModeKey::ImportWizard), AppMode::RecordsNormal => Some(ModeKey::RecordsNormal), @@ -618,6 +620,15 @@ impl Keymaps { "enter-mode", vec!["export-prompt".into()], ); + // Ctrl-S saves to the known path; Ctrl-W asks for one. The pair a + // user with no file path needs, and what the "No file path" status + // message points at (improvise-lae). + normal.bind_args( + KeyCode::Char('w'), + ctrl, + "enter-mode", + vec!["save-prompt".into()], + ); // Search / category add normal.bind_args( @@ -907,6 +918,15 @@ impl Keymaps { ), ); + set.insert( + ModeKey::SavePrompt, + Arc::new( + TextEntry::new(&AppMode::save_prompt(), "normal") + .commit(KeyCode::Enter, "commit-save", vec![]) + .finish(), + ), + ); + set.insert( ModeKey::CommandMode, Arc::new( diff --git a/crates/improvise-protocol/src/lib.rs b/crates/improvise-protocol/src/lib.rs index c33ca2e7..72a937de 100644 --- a/crates/improvise-protocol/src/lib.rs +++ b/crates/improvise-protocol/src/lib.rs @@ -175,6 +175,11 @@ pub enum AppMode { ExportPrompt { minibuf: MinibufferConfig, }, + /// Ask for a path to write the *model* to. The sibling of `ExportPrompt`, + /// which writes CSV; this one commits to `SaveAs` (improvise-lae). + SavePrompt { + minibuf: MinibufferConfig, + }, /// Vim-style `:` command line CommandMode { minibuf: MinibufferConfig, @@ -223,6 +228,7 @@ impl AppMode { "records-editing" => Some(Self::records_editing()), "formula-edit" => Some(Self::formula_edit()), "export-prompt" => Some(Self::export_prompt()), + "save-prompt" => Some(Self::save_prompt()), _ => None, } } @@ -241,6 +247,7 @@ impl AppMode { Self::TileSelect => "TILES", Self::ImportWizard => "IMPORT", Self::ExportPrompt { .. } => "EXPORT", + Self::SavePrompt { .. } => "SAVE AS", Self::CommandMode { .. } => "COMMAND", Self::Help => "HELP", Self::Quit => "QUIT", @@ -278,6 +285,7 @@ impl AppMode { Self::ImportWizard => "Space:toggle c:cycle Enter:next Esc:cancel", Self::Help => "h/l:pages q/Esc:close", Self::ExportPrompt { .. } => "Enter:export Esc:cancel — type a file path", + Self::SavePrompt { .. } => "Enter:save Esc:cancel — type a file path", Self::Quit => "", } } @@ -297,6 +305,7 @@ impl AppMode { | Self::TileSelect | Self::ImportWizard | Self::ExportPrompt { .. } + | Self::SavePrompt { .. } | Self::CommandMode { .. } | Self::Help | Self::Quit @@ -314,7 +323,8 @@ impl AppMode { | Self::CommandMode { minibuf, .. } | Self::CategoryAdd { minibuf, .. } | Self::ItemAdd { minibuf, .. } - | Self::ExportPrompt { minibuf, .. } => Some(minibuf), + | Self::ExportPrompt { minibuf, .. } + | Self::SavePrompt { minibuf, .. } => Some(minibuf), _ => None, } } @@ -400,6 +410,16 @@ impl AppMode { }, } } + + pub fn save_prompt() -> Self { + Self::SavePrompt { + minibuf: MinibufferConfig { + buffer_key: "save".into(), + prompt: "save as: ".into(), + color: MinibufColor::Yellow, + }, + } + } } /// UI session-state slice: mode, cursors, panels, buffers, navigation stacks, From 0116a82386554a5a22c1dff529f68351e8f236dc Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:39:52 -0700 Subject: [PATCH 15/72] test(mode): pin `:wq`, the surface improvise-git would have deleted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-git asked for `SaveAndQuit` to be deleted and `ZZ` bound to a `Binding::Sequence [w, force-quit]`, on the repo's "do not name a command AAndB" principle. Declined, with evidence rather than opinion. `wq` is not only a keybinding. It is a registered command name, so `:wq` resolves through the registry, and `Quit`'s own unsaved-changes message advertises it: "Use :q! to force quit or :wq to save+quit". A keymap Sequence is not reachable from the `:` line — `ExecuteCommand` parses names through `CmdRegistry` — so deleting the command deletes the ex-command with it. Removing the registration and running the new test yields exactly that: ":wq — Unknown command: wq". And keeping the command while also adding the Sequence would be worse than doing nothing: it leaves two independently-written spellings of "save then quit", which is the duplication this codebase keeps curing. Today one definition serves both surfaces. The ticket's premise is incomplete in the same way improvise-aaw's turned out to be. No behaviour changes here. What lands is the missing characterization test — `:wq` had no direct test at all — plus a doc comment recording why the command stays, so the next reader does not re-derive this. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/cmd/mode.rs | 9 ++++++ .../src/command/cmd/text_buffer.rs | 29 +++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/crates/improvise-command/src/command/cmd/mode.rs b/crates/improvise-command/src/command/cmd/mode.rs index f194bb42..711bcc8b 100644 --- a/crates/improvise-command/src/command/cmd/mode.rs +++ b/crates/improvise-command/src/command/cmd/mode.rs @@ -276,6 +276,15 @@ impl Cmd for Quit { /// Save then quit. The `Quit` only lands if the `Save` succeeds — a failed /// save aborts the effect chain, so `wq`/`ZZ` leaves the user in the app /// with the error visible rather than discarding the model (improvise-8tk). +/// +/// **This looks like an `AAndB` command and is not one** (improvise-git). +/// `wq` is the *user's* name for the composition, not an invented internal +/// one: it is registered, so `:wq` reaches it, and `Quit`'s unsaved-changes +/// message advertises that spelling. A keymap `Binding::Sequence` cannot be +/// reached from the `:` line, so replacing this with one would either delete +/// the ex-command or leave two independently-written spellings of "save then +/// quit". One definition behind two surfaces is what this already is. +/// Pinned by `text_buffer::tests::colon_wq_saves_then_quits`. #[derive(Debug)] pub struct SaveAndQuit; impl Cmd for SaveAndQuit { diff --git a/crates/improvise-command/src/command/cmd/text_buffer.rs b/crates/improvise-command/src/command/cmd/text_buffer.rs index fb2018fe..88f3d910 100644 --- a/crates/improvise-command/src/command/cmd/text_buffer.rs +++ b/crates/improvise-command/src/command/cmd/text_buffer.rs @@ -94,6 +94,35 @@ mod tests { ); } + /// improvise-git proposed deleting the `SaveAndQuit` command and binding + /// `ZZ` to a `Binding::Sequence [w, force-quit]` instead — the repo's + /// "do not name a command AAndB" principle. + /// + /// That premise is incomplete, and this test is why. `wq` is not only a + /// keybinding: it is a **registered command name**, so `:wq` reaches it + /// through the registry, and `Quit`'s own unsaved-changes message + /// advertises it ("Use :q! to force quit or :wq to save+quit"). A keymap + /// `Sequence` is not reachable from the `:` line, so deleting the command + /// deletes the ex-command with it. + /// + /// Keeping the command *and* adding the Sequence would leave two + /// independently-written spellings of "save then quit", which is the + /// duplication this codebase keeps curing rather than creating. So this + /// pins the surface, and the conversion is declined on it. + #[test] + fn colon_wq_saves_then_quits() { + let m = two_cat_model(); + let layout = make_layout(&m); + let reg = make_registry(); + let mut bufs = HashMap::new(); + bufs.insert("command".to_string(), "wq".to_string()); + let mut ctx = make_ctx(&m, &layout, ®); + ctx.buffers = &bufs; + let dbg = effects_debug(&ExecuteCommand.execute(&ctx)); + assert!(dbg.contains("Save"), "`:wq` must save, got: {dbg}"); + assert!(dbg.contains("Quit"), "`:wq` must then quit, got: {dbg}"); + } + /// improvise-6sq: `ExecuteCommand` appends `change_mode(Normal)` unless /// some effect reports `changes_mode()`. `EnterEditAtCursor` sets /// `view_state.mode` but did not override it, so From fa40abe93291b85c2e965549e455a82994a9e2e3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:39:52 -0700 Subject: [PATCH 16/72] docs(wasm-client): say what the classification duplication actually costs Review of the previous commit found its own doc overstating. It said "no host anywhere ever sees the string to classify it" two paragraphs after saying this is the same split `commit_regular_cell_value` makes -- and the TUI host does classify its own local keystrokes there. The precise claim is about the WIRE: `ModelCommand::SetCell` carries a `CellValue`, so nothing that arrives over a frame is ever a string. A host's own keyboard is a different path. Which exposes the part worth recording rather than smoothing over. The rule genuinely lives in two places, because `improvise-command` is deliberately not a dependency of this crate, and `a_typed_string_is_classified_as_the_host_classifies_it` states that obligation without discharging it -- it asserts three outcomes by hand instead of comparing with the host's function, so the two can drift and stay green. That is a_consolidation_that_keeps_its_compatibility_test_is_unfinished still open one layer below what improvise-r4lo closed. Named in the doc and filed, not fixed here: the real answer is a shared split in `improvise-protocol`, which both crates already depend on, and that moves code in a crate this ticket does not own. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-wasm-client/src/socket.rs | 29 ++++++++++++++++++---- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/crates/improvise-wasm-client/src/socket.rs b/crates/improvise-wasm-client/src/socket.rs index 103d82ba..f5ff6a5d 100644 --- a/crates/improvise-wasm-client/src/socket.rs +++ b/crates/improvise-wasm-client/src/socket.rs @@ -65,10 +65,29 @@ fn rejection_message(rejection: &Rejection) -> String { /// Rust rather than in the page's JavaScript is what makes it testable and /// keeps one spelling of the rule per language boundary instead of two. /// -/// This is the **only** answer the client gives to "what does a typed string -/// mean" (improvise-r4lo). It holds on every deployment, including the worker: -/// a host's `App` is behind the frame boundary and receives a `CellValue`, so -/// no host anywhere ever sees the string to classify it. +/// This is the **only** answer the *client* gives to "what does a typed string +/// mean" (improvise-r4lo), and it holds on every deployment, including the +/// worker: a host's `App` sits behind the frame boundary, and what crosses that +/// boundary is `ModelCommand::SetCell { value: CellValue }`. So no host ever +/// classifies a string that arrived *over the wire* — there is none to +/// classify by the time it gets there. +/// +/// Note the careful half of that. A host does classify its own **local** +/// keystrokes, in `commit_regular_cell_value`, which is why this is a +/// duplicate of that split rather than a call to it: `improvise-command` is +/// deliberately not a dependency of this crate (see `Cargo.toml` — reaching +/// it would drag `App`, `Workbook` and persistence into the browser bundle). +/// +/// So this rule genuinely lives in two places, and +/// `a_typed_string_is_classified_as_the_host_classifies_it` states the +/// obligation without discharging it: it asserts the three outcomes by hand +/// rather than comparing with the host's function, so the two could drift and +/// stay green. That is +/// `doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished)` +/// still open, one layer below what improvise-r4lo closed — the real fix is a +/// shared split in `improvise-protocol`, which both crates already depend on. +/// Filed rather than done here, because it moves code in a crate this ticket +/// does not own. /// /// `None` means "clear this cell", which is a different command, not a /// different value — hence the `Option` rather than a `CellValue::Text("")`. @@ -153,7 +172,7 @@ impl Client { /// It takes the user's **string**, not a `CellValue`, and [`typed`] above /// says why: the classification is unavoidably this side of the wire, /// because `ModelCommand::SetCell` already carries a `CellValue` by the - /// time any host sees it. That is true of the worker deployment too — the + /// time any host sees it *over the wire*. That is true of the worker too — the /// `App` there sits *behind* the frame boundary and is handed a value, not /// a string — so there is no deployment for which a pre-classified entry /// point would be the right one. A sibling taking a serialized `CellValue` From c342124ebfa1954368560e5b79a6d9ee47e76635 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:40:03 -0700 Subject: [PATCH 17/72] test(wizard): pin WizardStep::Preview, the one step nothing drew MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-g6s1. `every_step_draws_what_it_drew_before` walks the wizard from wherever `ImportWizard::new` puts it, and neither of its fixtures — a flat array of records, and a document with two candidate arrays — can land on `Preview`. Nothing advances *into* Preview either: it is only ever an entry step, chosen when the document needs no path selection and parses to zero records. So it was the single arm where a change to `wizard_cache` could alter what is drawn with nothing failing, while every other step was pinned byte for byte. An empty array is the smallest document that reaches it. The frame is asserted exactly, like its siblings, and the step is asserted as a precondition so the fixture cannot quietly stop reaching Preview and keep passing. Proved discriminating by perturbing `project::wizard_cache` — uppercasing `preview_summary` — and watching it fail. (The first perturbation tried, `trim_end`, was a no-op on this summary and passed; that is worth knowing about this kind of check.) The fixture also puts a wart on the record: with no records to take keys from, the summary ends `Sample keys:` with nothing after the colon. Pinned as it is rather than quietly fixed, so the next change to it is deliberate. Co-Authored-By: Claude Opus 5 (1M context) --- src/ui/import_wizard_ui.rs | 52 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index d2d3e203..8266837f 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -586,6 +586,58 @@ mod tests { assert_eq!(render(&w), ARRAY_PATH_STEP); } + /// `WizardStep::Preview` is the one step `every_step_draws_what_it_drew_ + /// before` never reaches (improvise-g6s1). Both fixtures there start at + /// `ReviewProposals` or `SelectArrayPath`, and nothing advances *into* + /// Preview — it is only ever an entry step, chosen by `ImportWizard::new` + /// when the document needs no path selection and parses to zero records. + /// + /// That left it the single arm where a change to `wizard_cache` could + /// alter what is drawn with nothing failing, which is the same hole the + /// other steps were pinned to close. An empty array is the smallest + /// document that lands there. + /// + /// It also puts on the record a wart the step has today: the summary ends + /// `Sample keys:` with nothing after the colon, because there are no + /// records to take keys from. Pinned as it is rather than quietly fixed — + /// the point of a characterization fixture is that the next change to it + /// is deliberate. + #[test] + fn the_preview_step_draws_an_empty_document() { + let w = ImportWizard::new(json!([])); + assert!( + matches!(w.step, WizardStep::Preview), + "precondition: an empty array must enter at Preview, got {:?}", + w.step + ); + assert_eq!(render(&w), PREVIEW_STEP); + } + + const PREVIEW_STEP: &str = concat!( + "┌ Import Wizard — Preview ─────────────────────────────────────────────────────┐\n", + "│Array of 0 records. Sample keys: │\n", + "│ │\n", + "│Press Enter to continue… │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "│ │\n", + "└──────────────────────────────────────────────────────────────────────────────┘", + ); + const ARRAY_PATH_STEP: &str = concat!( "┌ Import Wizard — Select Array ────────────────────────────────────────────────┐\n", "│Select the path containing records: │\n", From ece655115d18a0fb27a10ca37a9e555ae2f197a9 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:40:29 -0700 Subject: [PATCH 18/72] test(panel): make the view-name test earn its name, and decline aaw on it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-aaw asked for `CreateAndSwitchView` to become a keymap `Binding::Sequence [create-view X, switch-view X, enter-mode normal]`. It cannot: the new view's name is derived from context — `format!("View {}", ctx.workbook.views.len() + 1)` — and a Sequence carries only literal args, so it would hard-code one name and the second `n` in the view panel would collide with the first. The derivation is the decision that makes this a command rather than a composition. The test that should have said so did not. `create_and_switch_view_names_ incrementally` asserted that `CreateView`, `SwitchView` and `Normal` appeared and never looked at the name — the repo's characteristic defect, a test that passes while the thing it names is unchecked. A hard-coded name would have kept it green, which is exactly the change under discussion. It now asserts the name at two different view counts, because one count cannot separate "derived from the workbook" from "constant". Proved discriminating by replacing the derivation with the literal `"View 2"` a Sequence would carry, and watching the second arm fail. No behaviour change. The doc comment records why the command stays. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/panel.rs | 48 ++++++++++++++++--- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/panel.rs b/crates/improvise-command/src/command/cmd/panel.rs index 10aebfce..3ab2863f 100644 --- a/crates/improvise-command/src/command/cmd/panel.rs +++ b/crates/improvise-command/src/command/cmd/panel.rs @@ -301,26 +301,48 @@ mod tests { assert!(effects.is_empty()); } + /// improvise-aaw asked for this command to be replaced by a keymap + /// `Binding::Sequence [create-view X, switch-view X, enter-mode normal]`. + /// It cannot be: a `Sequence` carries **literal** args, and the name here + /// is computed from `ctx` — so a Sequence would hard-code one name and + /// the second `n` in the view panel would collide with the first. + /// + /// The test was called `..._names_incrementally` and asserted only that + /// `CreateView`/`SwitchView`/`Normal` appeared — it never looked at the + /// name, which is the one thing that makes this command a command and + /// not a composition. Now it does, at two different view counts, because + /// a single count cannot tell "derived from the workbook" from "constant". #[test] fn create_and_switch_view_names_incrementally() { - let m = two_cat_model(); + let mut m = two_cat_model(); let layout = make_layout(&m); let reg = make_registry(); let ctx = make_ctx(&m, &layout, ®); - let effects = CreateAndSwitchView.execute(&ctx); - let dbg = effects_debug(&effects); + let dbg = effects_debug(&CreateAndSwitchView.execute(&ctx)); assert!( - dbg.contains("CreateView"), - "Expected CreateView, got: {dbg}" + dbg.contains("CreateView(\"View 2\")"), + "Expected the name derived from the one existing view, got: {dbg}" ); assert!( - dbg.contains("SwitchView"), - "Expected SwitchView, got: {dbg}" + dbg.contains("SwitchView(\"View 2\")"), + "Expected a switch to the view just created, got: {dbg}" ); assert!( dbg.contains("Normal"), "Expected return to Normal, got: {dbg}" ); + drop(ctx); + + // A second view in the workbook must move the name. A literal arg in + // a keymap Sequence could not, which is why aaw was declined. + m.create_view("View 2"); + let layout = make_layout(&m); + let ctx = make_ctx(&m, &layout, ®); + let dbg = effects_debug(&CreateAndSwitchView.execute(&ctx)); + assert!( + dbg.contains("CreateView(\"View 3\")"), + "the name must follow the workbook, got: {dbg}" + ); } #[test] @@ -642,6 +664,18 @@ impl Cmd for SwitchViewAtCursor { } /// Create a new view, switch to it, and return to Normal mode. +/// +/// **Not a shim, despite the name** (improvise-aaw). The new view's name is +/// derived from `ctx` — `format!("View {}", views.len() + 1)` — and a keymap +/// `Binding::Sequence` can only carry literal args, so the proposed +/// `[create-view X, switch-view X, enter-mode normal]` would hard-code one +/// name and the second `n` in the view panel would collide with the first. +/// The derivation is the decision that makes this a command. +/// +/// The ticket's two fallbacks are both behaviour changes wanting a product +/// decision, not refactors: prompting for a name is a new feature, and +/// making `create-view` switch by default changes `:create-view` for scripts +/// and duplicates what `grid.rs` does by hand for `_Drill`/`_Records`. #[derive(Debug)] pub struct CreateAndSwitchView; impl Cmd for CreateAndSwitchView { From a60daf5807029a41b102cf3b5f1ee08e410e7b7e Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:40:57 -0700 Subject: [PATCH 19/72] fix(wizard): clear the bottom line before writing it, and repin the frame MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-q0vb, and the frame change IS the commit rather than a side effect of one — which is the whole reason it was declined on PR #12 and filed instead. The bug. The wizard's bottom row has two writers: each step's key hint, and the message that overrides it while present, written in that order to the same `y`. `Buffer::set_string` writes exactly as many cells as its string has and leaves the rest of the row alone, so a message shorter than the hint it replaced left the hint's tail sitting beside it. The reproduction was already in the suite as an *expectation*: `FORMULA_STEP` asserted `something went wrongcel`, the `cel` being the end of `Esc: cancel`. The fix is one helper, `set_bottom_line`, that truncates to the panel width and then pads to it, and all five writers of that row now go through it. Padding there rather than at each of the four hint sites is deliberate: the defect is not that one writer forgot to clear the row, it is that the row had five writers and none of them owned clearing it. The helper also gives the four hints the truncation only the message had. Expectations changed, and why each new one is right — there is exactly one: FORMULA_STEP, the bottom row: was `│something went wrongcel │` now `│something went wrong │` The message is `something went wrong`. `cel` was never part of it and was never meant to be on screen; it is three characters of a hint the message had replaced. The new expectation is the message and nothing else. Nothing else moved. `EVERY_STEP` (five frames), `ARRAY_PATH_STEP` and `PREVIEW_STEP` are byte-identical, because padding with spaces is invisible to `buf_text`, which trims each row — so the only place the change can show is where it erases something, which is the bug and only the bug. That the other seven frames did not move is the evidence this is a fix and not a rewrite of the characterization the migration PR was protecting. Added `a_message_shorter_than_the_hint_leaves_none_of_it`, which names the rule rather than a frame, on `ReviewProposals` (the longest hint in the wizard), with both preconditions asserted: that the step really draws that hint, and that the message really is shorter than it. Both it and `FORMULA_STEP` were watched failing with the padding removed. Co-Authored-By: Claude Opus 5 (1M context) --- src/ui/import_wizard_ui.rs | 74 ++++++++++++++++++++++++++++++++++---- 1 file changed, 68 insertions(+), 6 deletions(-) diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index 8266837f..98695dd0 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -174,9 +174,11 @@ impl<'a> Widget for ImportWizardWidget<'a> { y += 1; } let hint_y = inner.y + inner.height - 1; - buf.set_string( + set_bottom_line( + buf, x, hint_y, + w, "Enter: next Space: toggle c: cycle kind \u{2190} back Esc: cancel", Style::default().fg(Color::DarkGray), ); @@ -233,9 +235,11 @@ impl<'a> Widget for ImportWizardWidget<'a> { } } let hint_y = inner.y + inner.height - 1; - buf.set_string( + set_bottom_line( + buf, x, hint_y, + w, "Space: toggle Enter: next \u{2190} back Esc: cancel", Style::default().fg(Color::DarkGray), ); @@ -317,7 +321,7 @@ impl<'a> Widget for ImportWizardWidget<'a> { } else { "n: new formula d: delete Enter: next \u{2190} back Esc: cancel" }; - buf.set_string(x, hint_y, hint, Style::default().fg(Color::DarkGray)); + set_bottom_line(buf, x, hint_y, w, hint, Style::default().fg(Color::DarkGray)); } WizardStepKind::NameModel => { buf.set_string(x, y, "Model name:", Style::default().fg(Color::Yellow)); @@ -369,9 +373,11 @@ impl<'a> Widget for ImportWizardWidget<'a> { } } let hint_y = inner.y + inner.height - 1; - buf.set_string( + set_bottom_line( + buf, x, hint_y, + w, "Enter: accept \u{2190}/Esc: back", Style::default().fg(Color::DarkGray), ); @@ -382,11 +388,29 @@ impl<'a> Widget for ImportWizardWidget<'a> { // line on every step, overriding the key hint while present. if let Some(msg) = &self.wizard.message { let msg_y = inner.y + inner.height - 1; - buf.set_string(x, msg_y, truncate(msg, w), Style::default().fg(Color::Red)); + set_bottom_line(buf, x, msg_y, w, msg, Style::default().fg(Color::Red)); } } } +/// Write the wizard's bottom line, padded to the panel width. +/// +/// That row has two writers — each step's key hint, and the wizard message +/// that overrides it while present — and they write in that order to the same +/// `y`. `Buffer::set_string` writes exactly as many cells as the string has +/// and leaves the rest alone, so a message shorter than the hint it replaced +/// left the hint's tail sitting beside it: `something went wrongcel`, the +/// `cel` from `Esc: cancel` (improvise-q0vb). +/// +/// Padding here rather than at each site is the point. The bug is not that +/// one of the five writers forgot; it is that the row had five writers and no +/// one of them owned clearing it. +fn set_bottom_line(buf: &mut Buffer, x: u16, y: u16, w: usize, text: &str, style: Style) { + let text = truncate(text, w); + let pad = w.saturating_sub(text.chars().count()); + buf.set_string(x, y, format!("{text}{:pad$}", "", pad = pad), style); +} + fn truncate(s: &str, max: usize) -> String { if s.len() <= max { s.to_string() @@ -573,6 +597,39 @@ mod tests { assert_eq!(render(&w), FORMULA_STEP); } + /// improvise-q0vb. The wizard's bottom row has two writers — the step's + /// key hint and the message that overrides it — and `set_string` writes + /// only as many cells as its string has. A message shorter than the hint + /// therefore left the hint's tail beside it, and the artifact was pinned + /// as correct output in `FORMULA_STEP`: `something went wrongcel`. + /// + /// Asserted here on `ReviewProposals`, whose hint is the longest in the + /// wizard, so the tail it would leave is unmistakable. The precondition + /// asserts the hint really is longer than the message — without it the + /// test would pass on a step where there was nothing to leave behind. + #[test] + fn a_message_shorter_than_the_hint_leaves_none_of_it() { + const HINT: &str = "Enter: next Space: toggle c: cycle kind ← back Esc: cancel"; + let mut w = wizard(); + assert!(matches!(w.step, WizardStep::ReviewProposals)); + assert!( + render(&w).lines().any(|l| l.contains(HINT)), + "precondition: this step draws the long hint" + ); + + w.message = Some("short".to_string()); + let drawn = render(&w); + let bottom = drawn.lines().nth_back(1).expect("the row above the border"); + assert_eq!( + bottom.trim_end_matches([' ', '│']).trim_start_matches('│'), + "short" + ); + assert!( + "short".len() < HINT.len(), + "precondition: the message is shorter than what it replaces" + ); + } + /// The two steps the flat-array fixture skips. `SelectArrayPath` is /// reached by a document with more than one candidate array, which is /// also the only step that draws `pipeline.array_paths`. @@ -812,7 +869,12 @@ mod tests { "│ │\n", "│ │\n", "│ │\n", - "│something went wrongcel │\n", + // improvise-q0vb: this used to read `something went wrongcel` — the + // `cel` was the tail of `Esc: cancel`, left behind because the message + // was written over a longer hint without clearing the rest of the row. + // The frame changing here is the point of that commit, not a side + // effect: it is the only one of the eight pinned frames that moves. + "│something went wrong │\n", "└──────────────────────────────────────────────────────────────────────────────┘", ); } From a63a5bf34330c9848016899f7c481af3a2832bb3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:42:02 -0700 Subject: [PATCH 20/72] style: cargo fmt Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/import/csv_parser.rs | 5 ++++- src/ui/import_wizard_ui.rs | 9 ++++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/crates/improvise-io/src/import/csv_parser.rs b/crates/improvise-io/src/import/csv_parser.rs index d86081d8..a377000f 100644 --- a/crates/improvise-io/src/import/csv_parser.rs +++ b/crates/improvise-io/src/import/csv_parser.rs @@ -390,7 +390,10 @@ mod tests { let (control, _) = parse_csv_bytes_with_warnings("Region,\u{feff}Revenue\nEast,1\n".as_bytes()).unwrap(); assert!( - control[0].as_object().unwrap().contains_key("\u{feff}Revenue"), + control[0] + .as_object() + .unwrap() + .contains_key("\u{feff}Revenue"), "control: a non-leading BOM survives, so the assertion above can see one" ); } diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index 98695dd0..1c679e21 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -321,7 +321,14 @@ impl<'a> Widget for ImportWizardWidget<'a> { } else { "n: new formula d: delete Enter: next \u{2190} back Esc: cancel" }; - set_bottom_line(buf, x, hint_y, w, hint, Style::default().fg(Color::DarkGray)); + set_bottom_line( + buf, + x, + hint_y, + w, + hint, + Style::default().fg(Color::DarkGray), + ); } WizardStepKind::NameModel => { buf.set_string(x, y, "Model name:", Style::default().fg(Color::Yellow)); From 460fb17a9113cc9a2b89c76e08284d1fa1331f6a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:42:11 -0700 Subject: [PATCH 21/72] fix(mode): stop records mode talking about drilling on an empty grid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Entering records mode puts every regular category on `Axis::None`, so `EditOrDrill`'s aggregation check is true unless the cursor sits on a synthetic record. On an empty records grid there is none, so `i` took the drill branch, found no cell key, and reported "cannot drill — no cell at cursor" — a pivot-mode mechanism named in a mode whose model is "a table I type into", with `o` one key away that does add a row. Option (a) of the two the ticket offers. Option (b) — make `i` behave like `o` there — was declined: the divergence between `i` and `o` on an empty records grid is exactly what improvise-6sq exists to record, and is why `edit-or-drill` and `enter-edit-at-cursor` are two commands. The outcome (no editor where there is nothing to edit) was never the bug; the wording was. Which sentence to use is read off `edit_mode` — the argument the *keymap* supplies — and not off `ctx.mode`, so the command stays as mode-agnostic as its doc comment claims and no runtime `is_records()` check returns (improvise-4ju). Evidence. `i_and_o_diverge_on_an_empty_records_grid` keeps its load-bearing assertion (`i` does not enter editing) and its wording assertion now names what must *not* appear; it was watched failing on the old message before the fix. The new `the_empty_grid_message_is_not_the_records_one_in_pivot_mode` is the negative arm — without it, "say the records sentence everywhere" passes, and would promise `o` in a mode where `o` adds no row. Proved discriminating by collapsing the two sentences into one and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/cmd/mode.rs | 26 +++++++- crates/improvise-command/src/ui/app.rs | 59 ++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/mode.rs b/crates/improvise-command/src/command/cmd/mode.rs index 711bcc8b..1f1609d6 100644 --- a/crates/improvise-command/src/command/cmd/mode.rs +++ b/crates/improvise-command/src/command/cmd/mode.rs @@ -310,6 +310,30 @@ impl Cmd for SaveAndQuit { pub struct EditOrDrill { pub edit_mode: AppMode, } +impl EditOrDrill { + /// What to say when there is no cell under the cursor at all. + /// + /// improvise-ct5: this used to say "cannot drill — no cell at cursor" in + /// both modes. Drilling is a pivot-mode concept, and in records mode — + /// where every regular category sits on `Axis::None`, so this branch is + /// the one an empty grid always takes — the user's model is "a table I + /// type into", with `o` one key away to add a row. Naming drilling there + /// described a mechanism the user is not using instead of the key that + /// would help. + /// + /// The choice reads `edit_mode` — the argument the *keymap* supplies — + /// rather than asking `ctx` what mode it is in, so the command stays as + /// mode-agnostic as its doc comment claims and no runtime `is_records()` + /// check comes back (improvise-4ju). + fn nothing_at_cursor(&self) -> &'static str { + if self.edit_mode.is_records() { + "No record at cursor — press o to add a row" + } else { + "No cell at cursor" + } + } +} + impl Cmd for EditOrDrill { fn name(&self) -> &'static str { "edit-or-drill" @@ -330,7 +354,7 @@ impl Cmd for EditOrDrill { let is_aggregated = !is_synthetic && regular_none; if is_aggregated { let Some(key) = ctx.cell_key().clone() else { - return vec![effect::set_status("cannot drill — no cell at cursor")]; + return vec![effect::set_status(self.nothing_at_cursor())]; }; return DrillIntoCell { key }.execute(ctx); } diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 7765798b..d34b0aa6 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1399,6 +1399,17 @@ mod tests { /// there is none, so `i` reports "cannot drill". `o` must still open an /// edit there (see `add_record_row_in_empty_records_view_creates_first_row`), /// so routing it through the same drill check would break it. + /// + /// improvise-ct5 changed the *message* and deliberately not the + /// divergence. The outcome — `i` opens no editor where there is nothing + /// to edit — is right, and `o` one key away is the affordance for adding + /// a row. What was wrong is that the message said "cannot drill", a + /// pivot-mode concept in a mode whose model is "a table I type into". + /// Making `i` behave like `o` here (the ticket's option (b)) would + /// collapse exactly the divergence improvise-6sq exists to record, so + /// the assertion below still pins that `i` does not enter editing; only + /// the wording assertion moved, and it now also names what must *not* + /// appear. #[test] fn i_and_o_diverge_on_an_empty_records_grid() { let mut wb = Workbook::new("T"); @@ -1418,7 +1429,53 @@ mod tests { "i must not enter editing when there is no cell to edit" ); assert!( - app.view_state.status_msg.contains("cannot drill"), + !app.view_state.status_msg.to_lowercase().contains("drill"), + "records mode must not talk about drilling, got status {:?}", + app.view_state.status_msg + ); + assert!( + app.view_state.status_msg.contains("press o to add a row"), + "the message should name the key that does add a row, got {:?}", + app.view_state.status_msg + ); + } + + /// The negative arm of improvise-ct5, and the reason the wording is + /// keyed off the mode the keymap supplies rather than reworded once. + /// + /// Without this, "say the records sentence everywhere" passes the test + /// above — and it would promise `o`, which adds no row in pivot mode, + /// trading one misleading message for another. + /// + /// The fixture is an empty pivot grid that still counts as aggregated: + /// `Region` on Row with no items makes `cell_key()` `None`, and `Extra` + /// cycled to `Axis::None` is the regular category that makes + /// `EditOrDrill` take the drill branch. + #[test] + fn the_empty_grid_message_is_not_the_records_one_in_pivot_mode() { + let mut wb = Workbook::new("T"); + wb.add_category("Region").unwrap(); + wb.add_category("Extra").unwrap(); + wb.active_view_mut() + .set_axis("Extra", crate::view::Axis::None); + let mut app = App::new(wb, None); + app.rebuild_layout(); + assert_eq!(app.layout.row_count(), 0, "setup: empty pivot grid"); + + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + assert_eq!( + app.view_state.mode, + AppMode::Normal, + "i must not enter editing when there is no cell to edit" + ); + assert!( + !app.view_state.status_msg.contains("add a row"), + "`o` adds no row in pivot mode; it must not be offered, got {:?}", + app.view_state.status_msg + ); + assert!( + app.view_state.status_msg.contains("No cell at cursor"), "got status {:?}", app.view_state.status_msg ); From 84e6c5aa49d6db5c52fece7f1a2213684e7f4c58 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:42:53 -0700 Subject: [PATCH 22/72] style(wizard): spell the bottom-line padding plainly Review pass on the branch: `format!("{text}{:pad$}", "", pad = pad)` said the same thing as a `repeat`, less obviously. No behaviour change; the eight characterization frames are unmoved. Co-Authored-By: Claude Opus 5 (1M context) --- src/ui/import_wizard_ui.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index 1c679e21..94fc5104 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -414,8 +414,8 @@ impl<'a> Widget for ImportWizardWidget<'a> { /// one of them owned clearing it. fn set_bottom_line(buf: &mut Buffer, x: u16, y: u16, w: usize, text: &str, style: Style) { let text = truncate(text, w); - let pad = w.saturating_sub(text.chars().count()); - buf.set_string(x, y, format!("{text}{:pad$}", "", pad = pad), style); + let pad = " ".repeat(w.saturating_sub(text.chars().count())); + buf.set_string(x, y, format!("{text}{pad}"), style); } fn truncate(s: &str, max: usize) -> String { From 16471d7defbb21e1b7ac55199adccd08f772a37b Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:43:41 -0700 Subject: [PATCH 23/72] docs: the proxy is the open door, not the closed one (Copilot review) Both Copilot findings on PR #21 applied. The first is the serious one and it is a defect in documentation this PR itself introduced. `improvise-ws-server` has NO authentication -- a deliberate MVP limit its own module doc states -- so anyone who can open a socket to it can rewrite every document it serves. Loopback binding is the only thing standing between that and the network, and terminating TLS in front of it REMOVES EXACTLY THAT PROTECTION. The previous text not only omitted the warning, it actively reassured the reader: the `--bind` note explained that keeping loopback means the port "is not reachable from anywhere else", which is true of the port and false of the deployment, because the proxy is public. TLS buys confidentiality, not access control, and the two read alike in a recipe. So the warning leads the section, the Caddyfile carries `basic_auth` with a comment saying it is not optional, the `--bind` note now says what it does and does not buy, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing by default -- an unguessable URL is not a control. Second finding: the watchdog comment was ambiguous in a file whose only verification is reading, which makes an ambiguous comment a real defect. It said a drop before the first frame "leaves the watchdog armed", meaning the state on ARRIVAL; it reads as a claim about the state after the call, which is the opposite of what `setShellFailure` does. Reworded to say the call disarms, why the opened path needs that (the watchdog is armed only after `transport.ready` resolves, i.e. on `open`), and that the never-opened path makes it a no-op. No open ticket covers server auth -- improvise-j31x is the multi-document registry and is closed -- so nothing is cited here and one is proposed in the hand-back rather than an ID invented. 1184 passing, unchanged. `node --check` clean. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 31 ++++++++++++++++++++++++++++--- web/app.js | 10 +++++++--- 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1a0035ed..ebfe73f3 100644 --- a/README.md +++ b/README.md @@ -237,12 +237,33 @@ document in *a* browser, you do not need any of this — that is the standalone worker deployment, it is what the Pages site is for, and it needs no server process at all. +**Read this before following any of it.** `improvise-ws-server` has **no +authentication of any kind**, and that is a deliberate MVP limit rather than an +oversight — its own module doc says so. Anyone who can open a socket to it can +**rewrite every document it serves**. Loopback binding is the only thing +currently standing between that and the network, and *terminating TLS in front +of it removes exactly that protection*: the proxy is public, so the endpoint is +public, and `--bind 127.0.0.1` protects the port while doing nothing about the +proxy. TLS gets you confidentiality, not access control. + +So put the access control in the proxy, and treat it as required rather than +optional — HTTP basic auth, mTLS, an IP allow-list, or a tunnel with its own +authentication (`cloudflared` can enforce Cloudflare Access; a bare +`--url` tunnel enforces nothing and is a public URL). Caddy's `basic_auth` +directive is the smallest version. Share only documents you would be willing +to let an anonymous visitor edit, and take the tunnel down when you are done. + A worked example with [Caddy](https://caddyserver.com/), which obtains and renews the certificate itself. Point a DNS name at the host, then: ```caddyfile # Caddyfile improvise.example.com { + # NOT optional — see above. Without this the document is world-writable. + # Generate the hash with: caddy hash-password + basic_auth { + editor $2a$14$...replace-with-your-own-hash... + } reverse_proxy 127.0.0.1:8787 } ``` @@ -265,12 +286,16 @@ answered as an ordinary request and the socket never opens. For a one-off share with no DNS name and no certificate of your own, `cloudflared tunnel --url http://127.0.0.1:8787` or `ngrok http 8787` each print an https hostname that works the same way — pass the hostname to `?ws=` -without a scheme. +without a scheme. Both print a **public** URL and neither authenticates +anything by default, so add the tunnel's own auth (`ngrok http --basic-auth`, +or a named Cloudflare tunnel behind Access) rather than relying on the URL +being unguessable. Note the server binds `127.0.0.1` by default and `--bind` is the only way to widen that. Behind a proxy on the same host you should *not* widen it: the -proxy reaches loopback, and leaving the default means the unencrypted port is -not reachable from anywhere else. +proxy reaches loopback, and leaving the default keeps the **unencrypted** port +off the network. That is worth doing and is not access control — the proxy is +still the open door, which is why the auth above belongs there. ### Why not native `wss://` diff --git a/web/app.js b/web/app.js index 3be540c1..728a9ef2 100644 --- a/web/app.js +++ b/web/app.js @@ -223,9 +223,13 @@ function socketTransport() { // disarmed because it is for silence only, whereas suppressing `close` // outright would hide a genuine mid-session disconnect, which is the one // thing this listener is really for. Hence the split rather than a guard. - // Both arms go through `setShellFailure`: a drop before the first frame - // leaves the watchdog armed, and "disconnected" is more precise than the - // guess it would fire 8s later. + // Both arms go through `setShellFailure`, which DISARMS the watchdog. That + // matters on the opened path: the watchdog is armed after `transport.ready` + // resolves — i.e. on `open` — so a socket that opened and then dropped + // before its first frame reaches here with the timer still pending, and + // "disconnected" is more precise than the guess it would otherwise fire 8s + // later. On the never-opened path there is no timer yet and the clear is a + // no-op. Neither arm leaves one armed. socket.addEventListener("close", () => setShellFailure(opened ? "disconnected" : unreachable), ); From d13fe32b534a80e8c6b700be033fcc1550965048 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:44:24 -0700 Subject: [PATCH 24/72] fix(project): diff the totals against the frame the labels will leave MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's finding on PR 20, and it is a real one this branch introduced. hkt6(b) made `apply` blank the total under a **relabelled** column. But `grid_updates` compared `mirror.totals` against `desired.totals`, so a column renamed while its total stayed the same number emitted `ColLabels` and no `Totals`: the client blanked its own total, and `ClientSession::project_diff` then assigns `self.grid = desired`, leaving the server believing the number was still on screen. A client cache wrong **forever**, with nothing in the stream to correct it — exactly the direction gotcha 35 keeps `Dirty::of` deliberately sloppy to avoid. The fix is the one the cells already use, and which this function's own doc comment already argues for: compare against the frame the label updates will leave rather than the mirror they change. **Only the totals.** Moving `col_widths` to the same comparison was tried and reverted: folding the labels re-fits the widths too, so it suppressed the `ColWidths` that `a_client_on_a_socket_is_seeded_and_gets_its_edit_back` asserts a reframe carries. That update is redundant under the fold, but the assertion is gotcha 33(c)'s standing evidence and not this ticket's to spend — filing it rather than spending it. Evidence: `a_renamed_column_whose_total_did_not_change_still_gets_its_total_back` asserts the fold of the updates equals `desired` — the shape `a_reshaped_grid_folds_to_the_desired_state` uses, because "a Totals update was emitted" would pass for a producer emitting the wrong one. Proved by injecting the `mirror.totals` comparison and watching the client end at `["10", "", "30"]` against `["10", "20", "30"]`. Strays outside this unit's declared files (`crates/improvise-command/src/ui/ project.rs`) — reported in the hand-back. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-command/src/ui/project.rs | 88 ++++++++++++++++++++-- 1 file changed, 81 insertions(+), 7 deletions(-) diff --git a/crates/improvise-command/src/ui/project.rs b/crates/improvise-command/src/ui/project.rs index 3ea84702..b0d87ff5 100644 --- a/crates/improvise-command/src/ui/project.rs +++ b/crates/improvise-command/src/ui/project.rs @@ -623,17 +623,34 @@ pub fn grid_updates(mirror: &GridCache, desired: &GridCache) -> Vec if mirror.col_widths != desired.col_widths { updates.push(GridUpdate::ColWidths(desired.col_widths.clone())); } - if mirror.totals != desired.totals { - updates.push(GridUpdate::Totals(desired.totals.clone())); - } - if mirror.page_coords != desired.page_coords { - updates.push(GridUpdate::PageCoords(desired.page_coords.clone())); - } - + // The **totals** are diffed against the frame the label updates will + // leave, not against the mirror they are about to change — the same + // reason the cells below are (improvise-hkt6, found by Copilot). `apply` + // blanks the total under a *relabelled* column (gotcha 33(b)), so + // comparing against `mirror` missed exactly the case where the label + // moved and the number did not: no `Totals` went out, the client + // blanked its own, and `ClientSession::project_diff`'s + // `self.grid = desired` left the server believing it was still there. + // A client cache wrong **forever**, which is the direction `Dirty::of` + // is deliberately sloppy to avoid (gotcha 35). + // + // Only the totals. The widths above stay on `mirror`: folding the + // labels re-fits them too, so diffing widths against the fold would + // suppress the `ColWidths` that `a_client_on_a_socket_is_seeded_and_ + // gets_its_edit_back` asserts a reframe carries — redundant, but it is + // gotcha 33(c)'s standing evidence and not this ticket's to spend. let mut framed = mirror.clone(); for update in &updates { framed.apply(update); } + if framed.totals != desired.totals { + let totals = GridUpdate::Totals(desired.totals.clone()); + framed.apply(&totals); + updates.push(totals); + } + if mirror.page_coords != desired.page_coords { + updates.push(GridUpdate::PageCoords(desired.page_coords.clone())); + } let mut changes = Vec::new(); for (row, cells) in desired.cells.iter().enumerate() { @@ -1598,6 +1615,63 @@ mod tests { } } + /// improvise-hkt6, found by Copilot on PR 20. The diff must carry the + /// mirror to `desired`, and it stopped doing so for one shape: a column + /// **renamed** while its total stayed the same number. + /// + /// `apply` blanks the total under a relabelled column (gotcha 33(b)), + /// so the client's total went blank; but the diff compared + /// `mirror.totals` against `desired.totals`, found them equal, and sent + /// no `Totals`. `ClientSession::project_diff` then assigns + /// `self.grid = desired`, so the server believed the number was still + /// on screen — a cache wrong **forever**, with nothing in the stream to + /// correct it, which is precisely the direction gotcha 35 keeps + /// `Dirty::of` sloppy to avoid. + /// + /// The assertion is the one `a_reshaped_grid_folds_to_the_desired_state` + /// makes — fold the updates onto the mirror and require the result to be + /// `desired` — because "a `Totals` update was emitted" would pass for a + /// producer that emitted the wrong totals. + /// + /// The fixture renames the **middle** column and keeps every total + /// equal, so it separates "re-sends what the relabel blanked" from + /// "re-sends whenever the labels move at all". + #[test] + fn a_renamed_column_whose_total_did_not_change_still_gets_its_total_back() { + let totals = || { + vec![ + CellDisplay::number("10"), + CellDisplay::number("20"), + CellDisplay::number("30"), + ] + }; + let mirror = GridCache { + row_labels: vec!["r0".into()], + col_labels: vec!["A".into(), "B".into(), "C".into()], + col_widths: vec![8, 8, 8], + cells: vec![vec![ + CellDisplay::number("1"), + CellDisplay::number("2"), + CellDisplay::number("3"), + ]], + totals: totals(), + ..GridCache::default() + }; + let desired = GridCache { + col_labels: vec!["A".into(), "Bee".into(), "C".into()], + ..mirror.clone() + }; + + let mut client = mirror.clone(); + for update in &grid_updates(&mirror, &desired) { + client.apply(update); + } + assert_eq!( + client, desired, + "the updates must carry the mirror to desired, totals included" + ); + } + /// A second view whose axes are the first one's transposed, so the two /// grids cannot be confused for each other: "Default" rows are regions, /// "Second" rows are measures. From 6a2b46be5bf3a623f387a7c296e13ecfb060f5c5 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:44:42 -0700 Subject: [PATCH 25/72] docs(keymap): the improvise-0hb design pass, and the guard it needs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-0hb asked for `Binding::Sequence` steps to be transactional at command level: apply effects so far between steps, rebuild the layout, build a fresh `CmdContext`, dispatch the next. Acceptance was "design doc first". This is the design doc, and the answer is defer — with the enumeration as evidence, and with the regression guard the work would need, so deferring is safe rather than merely cheap. Three findings, all checkable. 1. It would silently reopen improvise-8tk, a P1 data loss. `apply_effects` resets `abort_effects` at entry and again when it breaks, because the flag is per batch. Applying between steps dispatches step 2 into a fresh batch with the flag cleared, so the `Quit` after a failed `Save` lands. Gotcha 21's claim that the abort carries across Sequence steps is true *because* the Sequence flattens into one batch — the property 0hb removes. That claim was also untested: no shipped Sequence has a first step that can abort, and `ZZ` is a prefix resolving to the single `wq` command, so the existing test covers the within-a-command case only. 2. Almost nothing would benefit. Of the shipped Sequences, all but one are `TextEntry`'s `[, clear-buffer]` or `[clear-buffer, enter-mode]`, whose second step reads nothing from ctx. The one real composition, records `o`, already works — `EnterEditAtCursorCmd` fronts a `RebuildLayout` and the effect re-reads the layout. Every Esc and commit key would pay a layout rebuild and a `CmdContext` build (which recomputes column widths) for that. 3. The motivating rewrite is not even dispatchable. improvise-git's `[(w, []), (force-quit, [])]` is a dead key: `w` is `register_pure`, whose interactive constructor rejects an empty arg list, so `run` returns None and the binding produces nothing. The nullary spelling is `save`. Found by writing the test, not by reading. What lands: the argument on `effects_of`, where the next person to propose this will read it, and `a_sequence_whose_first_step_aborts_does_not_run_the_second` — the first test to drive a Sequence whose first step aborts. Proved discriminating by making the abort per-effect instead of per-batch, which is what 0hb would do, and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/keymap.rs | 45 +++++++++++++++++ crates/improvise-command/src/ui/app.rs | 50 +++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/crates/improvise-command/src/command/keymap.rs b/crates/improvise-command/src/command/keymap.rs index d1e09bf6..7efc0243 100644 --- a/crates/improvise-command/src/command/keymap.rs +++ b/crates/improvise-command/src/command/keymap.rs @@ -54,6 +54,51 @@ fn run( /// Turn a binding into effects. The one place a keymap lookup becomes a /// mutation, and the reason `Binding` itself can be registry-free data. +/// +/// # Why a `Sequence` flattens, and why it stays that way (improvise-0hb) +/// +/// Every step is executed against the **same pre-dispatch `ctx`**, and the +/// effects are concatenated into one vec that `App::apply_effects` runs as +/// one batch. So a later *command* cannot observe an earlier one's effects — +/// only a later *effect* can. improvise-0hb proposed closing that: apply +/// effects so far between steps, rebuild the layout, build a fresh +/// `CmdContext`, dispatch the next. The design pass says **no**, on three +/// findings. +/// +/// **It would silently reopen improvise-8tk, a P1 data loss.** +/// `apply_effects` resets `abort_effects` at entry *and* again when it +/// breaks, because the flag is per batch. Applying between steps therefore +/// dispatches step 2 into a fresh batch with the flag cleared, so the `Quit` +/// after a failed `Save` would land — the exact silent discard 8tk fixed. +/// Gotcha 21's claim that the abort "carries across sequence steps" is true +/// *because* of this flattening, not in spite of it. It was also untested +/// until now: no shipped `Sequence` has a first step that can abort, and +/// `ZZ` is a prefix onto the single `wq` command. +/// `app.rs::a_sequence_whose_first_step_aborts_does_not_run_the_second` is +/// the guard, and it fails when the abort is made per-step. +/// +/// **Almost nothing would benefit.** Of the shipped `Sequence` bindings, +/// all but one come from `TextEntry` and are either `[, clear-buffer +/// ]` or `[clear-buffer , enter-mode ]` — `clear-buffer` and +/// `enter-mode` take their operand as a literal arg and read nothing from +/// `ctx`, so a refreshed `ctx` would change no outcome. The single genuine +/// composition is records `o` = `[add-record-row, enter-edit-at-cursor]`, +/// and it already works, because `EnterEditAtCursorCmd` fronts its effect +/// with `RebuildLayout` and the effect re-reads the layout itself (gotcha +/// 23). Every Esc and every commit keypress would pay a layout rebuild and +/// a `CmdContext` construction — which recomputes column widths — to buy +/// that nothing. +/// +/// **And the ticket's motivating rewrite does not typecheck as a binding +/// anyway.** improvise-git's `[(w, []), (force-quit, [])]` cannot dispatch: +/// `w` is `register_pure`, whose interactive constructor rejects an empty +/// arg list, so `run` returns `None` and the whole binding is dead. The +/// nullary spelling is `save`. `every_bound_command_name_is_registered` +/// checks exactly this (gotcha 15) and would have caught it. +/// +/// If command-level transactionality is ever wanted, it wants its own +/// campaign: the abort must become a property of the *sequence* rather than +/// of the batch before any step is applied separately. fn effects_of( binding: &Binding, registry: &CmdRegistry, diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index d34b0aa6..8b85c7b5 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1636,6 +1636,56 @@ mod tests { ); } + /// Gotcha 21 claims a failed `Save` aborts across `Binding::Sequence` + /// steps as well as within one command's effect list. Nothing tested it: + /// no shipped `Sequence` has a first step that can abort, and `ZZ` is a + /// *prefix* resolving to the single `wq` command, so + /// `prefix_z_then_z_reaches_save_and_quit` exercises the within-a-command + /// case only. The claim was structurally true and unguarded. + /// + /// It is true because `Keymap::dispatch` flattens a `Sequence` into ONE + /// effect vec and `App::apply_effects` runs that as one batch, breaking + /// on `abort_effects`. It is *load-bearing*, because it is the reason a + /// `[w, force-quit]` Sequence would be safe — the argument the + /// improvise-git grooming note unblocked on. + /// + /// And it is exactly what improvise-0hb would remove. Command-level + /// transactionality means applying effects *between* steps, and + /// `apply_effects` resets `abort_effects` at entry and again on abort, so + /// step 2 would be dispatched into a fresh batch with the flag cleared — + /// the `Quit` after a failed save would land, silently reopening the + /// improvise-8tk P1 data loss. This is the guard that would catch it. + #[test] + fn a_sequence_whose_first_step_aborts_does_not_run_the_second() { + use improvise_protocol::keymap::Keymap; + + // No file path, so `w` fails and sets `abort_effects`. + let mut app = App::new(Workbook::new("T"), None); + let key = KeyCode::Char('x'); + let mods = KeyModifiers::NONE; + let mut km = Keymap::new(); + km.bind_seq(key, mods, vec![("save", vec![]), ("force-quit", vec![])]); + + let effects = { + let ctx = app.cmd_context(key, mods); + app.keymap_set.dispatch_transient(&km, &ctx, key, mods) + } + .expect("both steps resolve in the registry"); + app.apply_effects(effects); + + assert!( + app.view_state.status_msg.contains("No file path"), + "setup: the save must actually have failed, got {:?}", + app.view_state.status_msg + ); + assert_ne!( + app.view_state.mode, + AppMode::Quit, + "the abort must carry into the next Sequence step, not just the \ + next effect of the same command" + ); + } + /// A prefix is consumed whether or not the next key matches, so a stray /// key after `g` does nothing rather than being treated as a fresh /// normal-mode binding. From 09a4e39ae3db3a936f7fa87b8d070bf2ae2084fc Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:44:51 -0700 Subject: [PATCH 26/72] fix(opfs): list names documents, not the empty files attach creates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by running `storage_contract` against `OpfsFs` for the first time (improvise-vvl3). It failed on its very first assertion — `starts empty` — and the reason is a semantic divergence nobody had had to state: `Storage::list` promises "**Document** names directly inside `dir`", and `OpfsFs` was answering with *entries*. `attach` must create the file to obtain a sync access handle for a document that does not exist yet, and it indexed what it created. So a zero-byte file — a document that does not exist — was listed as one. That is the improvise-9x6 browser failure read from the other end: a fresh origin listed `demo.improv` before anything had been written to it, the seed was skipped as "already there", and `load_from` got an empty buffer. improvise-9x6 worked around it in `vfs::holds_a_document`, which reads each candidate back; the wart it could not reach is the **document picker**, which `vfs::documents` feeds straight from `list` — a browser user would be offered an entry that cannot be opened. So the index now means "documents": - `attach` acquires the handle and indexes nothing. The file it creates is not yet a document. - `write` indexes the name, because a written document is the first thing there is to list. It also un-queues any pending delete of that name — otherwise delete-then-write would be listed and readable and then removed from OPFS by `commit_deletes` at the end of the batch, which is the one ordering gotcha 28's deferral can get wrong. - `refresh` skips zero-byte entries. The size comes off the sync access handle when this instance holds one and only otherwise through `getFile()`, because a file under an exclusive sync access handle is not reliably readable by the promise-based API. `vfs::holds_a_document`'s read-back is left alone: it is now belt and braces rather than the mechanism, and removing it is a separate argument. Evidence, in the harness that comes with the next commit: `an_attached_but_unwritten_entry_is_not_a_document` asserts both halves — that `attach` does not list, and that a *second* `OpfsFs` instance sharing no index does not list it either, so the filter has to be in `refresh` and not only in the omission from `attach`. And `storage_contract_at_""`/`_at_"docs"` now pass, which they did not before: that failure is how this was found. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/Cargo.toml | 2 + crates/improvise-io/src/opfs.rs | 71 +++++++++++++++++++++++++++++---- 2 files changed, 65 insertions(+), 8 deletions(-) diff --git a/crates/improvise-io/Cargo.toml b/crates/improvise-io/Cargo.toml index cabe80d7..7bb41d31 100644 --- a/crates/improvise-io/Cargo.toml +++ b/crates/improvise-io/Cargo.toml @@ -29,6 +29,8 @@ wasm-bindgen-futures = "0.4" [target.'cfg(target_arch = "wasm32")'.dependencies.web-sys] version = "0.3" features = [ + "Blob", + "File", "FileSystemDirectoryHandle", "FileSystemFileHandle", "FileSystemGetFileOptions", diff --git a/crates/improvise-io/src/opfs.rs b/crates/improvise-io/src/opfs.rs index b59e0033..d2ace80e 100644 --- a/crates/improvise-io/src/opfs.rs +++ b/crates/improvise-io/src/opfs.rs @@ -98,6 +98,17 @@ impl OpfsFs { /// Acquire the sync access handle for one document, creating the file /// if it is not there. Must be awaited before `read` or `write` names /// that document. + /// + /// **It does not index what it created**, and that is the whole of + /// improvise-9x6's first browser failure read the other way round. The + /// file it creates is zero bytes, and a zero-byte entry is not a + /// document — `Storage::list` promises *document* names. Indexing here + /// made a fresh origin list `demo.improv` before anything had been + /// written to it, so the seed was skipped as "already there" and + /// `load_from` got an empty buffer. `vfs::holds_a_document` works around + /// that by reading each candidate back; this removes the need. + /// [`Self::write`] is what puts a name in the index, because a written + /// document is the first thing there is to list. pub async fn attach(&self, path: &Path) -> Result<()> { let name = entry_name(path)?; if let Some(old) = self.handles.borrow_mut().remove(&name) { @@ -115,12 +126,7 @@ impl OpfsFs { .map_err(|e| js_err(&format!("createSyncAccessHandle({name})"), e))? .dyn_into::() .map_err(|_| anyhow!("createSyncAccessHandle({name}) did not yield a handle"))?; - self.handles.borrow_mut().insert(name.clone(), handle); - let mut index = self.index.borrow_mut(); - if !index.contains(&name) { - index.push(name); - index.sort(); - } + self.handles.borrow_mut().insert(name, handle); Ok(()) } @@ -135,6 +141,12 @@ impl OpfsFs { } /// Re-read the directory listing that `list` answers from. + /// + /// Zero-byte entries are skipped: `Storage::list` promises *document* + /// names, and [`Self::attach`] creates an empty file for a document that + /// does not exist yet. Listing those made a browser offer an entry that + /// cannot be opened (improvise-vvl3 found it by running + /// `storage_contract` here for the first time). pub async fn refresh(&self) -> Result<()> { let mut names = Vec::new(); let iter = js_sys::Reflect::get(&self.root, &"keys".into()) @@ -164,7 +176,9 @@ impl OpfsFs { .ok() .and_then(|v| v.as_string()) { - names.push(name); + if self.entry_size(&name).await? > 0.0 { + names.push(name); + } } } names.sort(); @@ -186,6 +200,32 @@ impl OpfsFs { Ok(()) } + /// How many bytes one entry holds. + /// + /// Read off the sync access handle when this instance has one, and only + /// otherwise through `getFile()`: a file under an exclusive sync access + /// handle is not reliably readable by the promise-based API, and + /// `refresh` is called *after* `attach` in the harness as well as before + /// it in the worker. + async fn entry_size(&self, name: &str) -> Result { + if let Some(handle) = self.handles.borrow().get(name) { + return handle + .get_size() + .map_err(|e| js_err(&format!("getSize({name})"), e)); + } + let file = JsFuture::from(self.root.get_file_handle(name)) + .await + .map_err(|e| js_err(&format!("getFileHandle({name})"), e))? + .dyn_into::() + .map_err(|_| anyhow!("getFileHandle({name}) did not yield a file handle"))?; + Ok(JsFuture::from(file.get_file()) + .await + .map_err(|e| js_err(&format!("getFile({name})"), e))? + .dyn_into::() + .map_err(|_| anyhow!("getFile({name}) did not yield a file"))? + .size()) + } + fn with_handle( &self, path: &Path, @@ -229,7 +269,22 @@ impl Storage for OpfsFs { .map_err(|e| js_err("write()", e))?; handle.flush().map_err(|e| js_err("flush()", e))?; Ok(()) - }) + })?; + // A written document is the first thing there is to list. `attach` + // deliberately does not do this — see its doc comment. + let name = entry_name(path)?; + // Writing a name back un-queues its removal. Otherwise a delete + // followed by a write of the same document would be listed and + // readable, and then removed from OPFS by `commit_deletes` at the + // end of the batch — the one ordering gotcha 28's deferral can get + // wrong. + self.pending_deletes.borrow_mut().retain(|n| *n != name); + let mut index = self.index.borrow_mut(); + if !index.contains(&name) { + index.push(name); + index.sort(); + } + Ok(()) } /// Entries in the origin root, presented under `dir`. From 7e991360dfc0e80d9c0a86fa63d51359d8613940 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:45:03 -0700 Subject: [PATCH 27/72] test(harness): a headless browser, and the first execution of OpfsFs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-vvl3, absorbing improvise-xf1. The condition the user set when deferring browser testing — "WE HAVE TO HAVE SOMETHING TO TEST" — has been met: after improvise-9x6, djm, hwpi and the demo model there is a complete self-contained application whose entire untested surface is what a harness covers. `python3 -m http.server` plus one Playwright project, in `harness/`: harness/opfs-contract/ a wasm crate that runs `improvise_io::storage::contract::storage_contract` against `OpfsFs` — the shared function, not a transcription of it harness/www/opfs/ the page and the dedicated Web Worker it runs in; `createSyncAccessHandle` exists nowhere else harness/tests/ two specs harness/build.sh site + harness bundle -> harness/serve It lives at the repo root rather than under `web/`, because `web/` is the *source of what ships* and every file in it is copied to the site. The harness serves the **assembled** tree (`web/assemble.sh`, extracted from the CI workflow in this commit so there is one answer to "how is the site arranged" rather than two) plus its own `/opfs/` — a harness that runs against a different arrangement of files is testing something nobody deploys. `harness/opfs-contract` is deliberately not a workspace member: it only builds for wasm32, so a member would break `cargo test --workspace`. What this runs that nothing has ever run: - `storage_contract` against `OpfsFs`, at `""` **and** at `"docs"`. Two bugs fell straight out — improvise-vhcz's `list` path shape, whose fix the `"docs"` run guards (reverting it fails with `lists the one document`), and the zero-byte entry fix in the previous commit, which this found by failing on `starts empty`. - The `keys()` async-iterator walk in `refresh`, hand-rolled through `js_sys::Reflect` and named by xf1 as the least certain code in the file. Three entries written out of order, so one passing by accident cannot carry it. - `FileSystemReadWriteOptions`' `at` on read and write: a long write then a short one, which is where an append or a non-zero start shows up. - `commit_deletes`, both halves — queued but still in OPFS before, gone after — and **across a reload**, which is the only way to ask whether OPFS persisted. That reload is also improvise-bck's persistence criterion at the storage layer. - improvise-hwpi's three one-assertion shell tests, which that ticket said it could not write: status after a bad frame, status with no peer, one repaint on commit. Plus the first-frame watchdog, which was found by reading and only because someone went looking. Check the checker, throughout, because a harness that passes against a stub — or that silently skips because the worker never started — is exactly this repo's characteristic defect wearing a green tick: - `run_canary` deliberately fails, through the *same* reporting path every real check uses, and the spec asserts it is reported as a failure with its message intact. - The spec asserts the report contains exactly the expected check names, in order, so a battery that quietly shrinks is a failure. - Every shell assertion was injected and watched to fail: removing the bad-frame catch, the watchdog's `setTimeout`, the socket `error` listener and the `transport.send(frame)` on commit each fails its own test and only its own. - The commit test types `20000` and asserts `20,000`, so a page that echoed what was typed instead of drawing what the host returned would fail. One live bug is pinned rather than fixed. `a peer that cannot be reached` asserts the *history* of `#status`, because improvise-m18f is real and reproduced here: the socket's `close` listener overwrites the precise "cannot reach" with "disconnected". That is filed and deliberately not the same fix as the watchdog's (gotcha 48), so the test asserts both halves and will speak up whichever one moves. CI gets a `browser` job that can fail the build, and `pages-deploy` now needs it. Playwright's Chromium download is ~80 MB and well under a minute on ubuntu-latest, next to the two wasm-pack builds the job already needs — so "impractical in this CI" does not arise. A harness nobody runs is worse than none, because it reads as coverage. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 108 +++--- harness/.gitignore | 8 + harness/build.sh | 27 ++ harness/opfs-contract/Cargo.lock | 621 +++++++++++++++++++++++++++++++ harness/opfs-contract/Cargo.toml | 30 ++ harness/opfs-contract/src/lib.rs | 307 +++++++++++++++ harness/package-lock.json | 76 ++++ harness/package.json | 13 + harness/playwright.config.js | 30 ++ harness/tests/opfs.spec.js | 73 ++++ harness/tests/shell.spec.js | 115 ++++++ harness/www/opfs/index.html | 48 +++ harness/www/opfs/worker.js | 30 ++ web/assemble.sh | 59 +++ 14 files changed, 1501 insertions(+), 44 deletions(-) create mode 100644 harness/.gitignore create mode 100755 harness/build.sh create mode 100644 harness/opfs-contract/Cargo.lock create mode 100644 harness/opfs-contract/Cargo.toml create mode 100644 harness/opfs-contract/src/lib.rs create mode 100644 harness/package-lock.json create mode 100644 harness/package.json create mode 100644 harness/playwright.config.js create mode 100644 harness/tests/opfs.spec.js create mode 100644 harness/tests/shell.spec.js create mode 100644 harness/www/opfs/index.html create mode 100644 harness/www/opfs/worker.js create mode 100755 web/assemble.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4e0de4c4..0b926ac7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -74,15 +74,68 @@ jobs: nix shell nixpkgs#trealla --command tpl prolog-sessions/kb.pl \ -g "(kb_consistent -> halt(0) ; (kb_report, halt(1)))" + # --------------------------------------------------------------------------- + # The browser harness (improvise-vvl3, absorbing improvise-xf1). + # + # This is the only job that RUNS any of the browser half. `OpfsFs`, + # `web/app.js` and `web/worker.js` were previously compile-checked, + # syntax-checked and read — never executed — so every bug in them was found + # by reading or by a person pressing keys, neither of which is repeatable. + # + # It can fail the build, deliberately: a harness nobody runs is worse than + # none, because it reads as coverage. Playwright's browser download is ~80 MB + # and takes well under a minute on ubuntu-latest, which is affordable next to + # the two wasm-pack builds this job already needs. + browser: + name: Browser harness (OPFS contract + shell) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + targets: wasm32-unknown-unknown + - uses: Swatinem/rust-cache@v2 + - uses: actions/setup-node@v4 + with: + node-version: 22 + - name: Install wasm-pack + env: + WASM_PACK_VERSION: v0.15.0 + run: | + set -eu + url="https://github.com/rustwasm/wasm-pack/releases/download/${WASM_PACK_VERSION}/wasm-pack-${WASM_PACK_VERSION}-x86_64-unknown-linux-musl.tar.gz" + curl -sSfL "$url" | tar -xz --strip-components=1 -C /usr/local/bin \ + "wasm-pack-${WASM_PACK_VERSION}-x86_64-unknown-linux-musl/wasm-pack" + wasm-pack --version + - name: Install the harness + working-directory: harness + run: npm ci + - name: Install Chromium + working-directory: harness + run: npx playwright install --with-deps chromium + # Builds both site bundles, the harness's own wasm bundle, and assembles + # `harness/serve` — the same tree `pages-build` publishes, plus /opfs/. + - name: Build the harness + run: ./harness/build.sh + - name: Run the harness + working-directory: harness + run: npx playwright test + - if: failure() + uses: actions/upload-artifact@v4 + with: + name: playwright-traces + path: harness/test-results + retention-days: 7 + # --------------------------------------------------------------------------- # GitHub Pages delivery (improvise-d31). # # Two jobs on purpose. `pages-build` runs on every push and pull request, so # a change that breaks `web/build.sh` or the bundle fails on the PR that made # it — the build is a gate. `pages-deploy` runs only on a push to main and - # `needs` the two check jobs above, so nothing that failed `cargo test` or the - # wasm32 gate can reach the published site. The deploy depends on those checks - # rather than repeating them (improvise-d31 AC 2). + # `needs` the check jobs above, so nothing that failed `cargo test`, the + # wasm32 gate or the browser harness can reach the published site. The deploy + # depends on those checks rather than repeating them (improvise-d31 AC 2). # # Not path-filtered, deliberately. The bundles are `improvise-wasm-client` and # `improvise-worker-server` plus, between them, most of the workspace — the @@ -123,47 +176,14 @@ jobs: - name: Build the wasm bundle run: ./web/build.sh - # `web/` is the source directory: it also holds build.sh and .gitignore, - # which have no business being served. Assemble what ships. + # `web/assemble.sh`, not an inline copy of it: the browser harness job + # below assembles the same tree, and a harness that runs against a + # different arrangement of files is testing something nobody deploys. - name: Assemble the site - run: | - set -eu - mkdir -p site - cp web/index.html web/app.js web/style.css web/worker.js site/ - # The document the page opens on a browser that has never seen - # one. web/worker.js fetches it by the ?seed= URL and hands the - # bytes to Rust, which writes them through the VFS — so leaving - # it out is an empty grid, not a 404 the page could report. - cp web/demo.improv site/ - cp -rf web/pkg site/pkg - cp -rf web/worker-pkg site/worker-pkg - rm -f site/pkg/.gitignore site/pkg/package.json - rm -f site/worker-pkg/.gitignore site/worker-pkg/package.json - # Cache-bust the two generated bundles. Pages serves everything - # `max-age=600` and the wasm-pack filenames carry no content hash, - # so for ten minutes after a deploy a returning visitor runs the - # PREVIOUS build -- which during improvise-9x6 debugging made a - # shipped fix look like it had not shipped, twice. The specifiers - # are static import strings, so stamping the copies in site/ is - # enough; the tracked sources are left alone and a local - # `http.server` is unaffected. - # Both halves, because a fresh glue file does NOT pull a fresh - # .wasm: the glue locates it with - # `new URL('..._bg.wasm', import.meta.url)`, and a URL constructed - # against a base does not inherit the base's query string. Stamping - # only the import specifier would refresh the wrapper and keep the - # stale binary -- the failure mode this is here to end. - sed -i "s|\./pkg/improvise_wasm_client\.js|&?v=${GITHUB_SHA}|" site/app.js - sed -i "s|\./worker-pkg/improvise_worker_server\.js|&?v=${GITHUB_SHA}|" site/worker.js - sed -i "s|'improvise_wasm_client_bg\.wasm'|'improvise_wasm_client_bg.wasm?v=${GITHUB_SHA}'|" \ - site/pkg/improvise_wasm_client.js - sed -i "s|'improvise_worker_server_bg\.wasm'|'improvise_worker_server_bg.wasm?v=${GITHUB_SHA}'|" \ - site/worker-pkg/improvise_worker_server.js - grep -n "?v=" site/app.js site/worker.js \ - site/pkg/improvise_wasm_client.js \ - site/worker-pkg/improvise_worker_server.js - find site -type f | sort - du -sh site + run: ./web/assemble.sh site "$GITHUB_SHA" + + - run: find site -type f | sort + - run: du -sh site - uses: actions/upload-pages-artifact@v3 with: @@ -171,7 +191,7 @@ jobs: pages-deploy: name: Deploy to GitHub Pages - needs: [test, wasm, pages-build] + needs: [test, wasm, browser, pages-build] if: github.event_name == 'push' && github.ref == 'refs/heads/main' runs-on: ubuntu-latest permissions: diff --git a/harness/.gitignore b/harness/.gitignore new file mode 100644 index 00000000..be0359f7 --- /dev/null +++ b/harness/.gitignore @@ -0,0 +1,8 @@ +# Built by build.sh; never committed. +serve/ +www/opfs/pkg/ + +# Playwright +node_modules/ +test-results/ +playwright-report/ diff --git a/harness/build.sh b/harness/build.sh new file mode 100755 index 00000000..87044493 --- /dev/null +++ b/harness/build.sh @@ -0,0 +1,27 @@ +#!/bin/sh +# Build everything the browser harness serves, into harness/serve/. +# +# harness/serve/ the assembled site, exactly as `web/assemble.sh` +# produces it for GitHub Pages +# harness/serve/opfs/ this harness's own page, worker and wasm bundle +# +# One server root, so Playwright runs one `python3 -m http.server`. The site +# half is the *deployed* arrangement rather than `web/` itself, because a +# harness that runs against a different tree from the one that ships is +# testing something nobody deploys (improvise-vvl3). +set -eu +root=$(cd "$(dirname "$0")/.." && pwd) +serve="$root/harness/serve" + +"$root/web/build.sh" >/dev/null +"$root/web/assemble.sh" "$serve" harness >/dev/null + +wasm-pack build "$root/harness/opfs-contract" \ + --target web --out-dir "$root/harness/www/opfs/pkg" + +mkdir -p "$serve/opfs" +cp -f "$root/harness/www/opfs/index.html" "$root/harness/www/opfs/worker.js" "$serve/opfs/" +cp -rf "$root/harness/www/opfs/pkg" "$serve/opfs/pkg" +rm -f "$serve/opfs/pkg/.gitignore" "$serve/opfs/pkg/package.json" + +echo "harness/serve ready" diff --git a/harness/opfs-contract/Cargo.lock b/harness/opfs-contract/Cargo.lock new file mode 100644 index 00000000..8365df49 --- /dev/null +++ b/harness/opfs-contract/Cargo.lock @@ -0,0 +1,621 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "android_system_properties" +version = "0.1.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc" +dependencies = [ + "libc", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "cc" +version = "1.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "54413ede23c2daf518f35156dfde027feb2374004d63bd497f983c8db9c0e313" +dependencies = [ + "find-msvc-tools", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + +[[package]] +name = "chrono" +version = "0.4.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" +dependencies = [ + "iana-time-zone", + "js-sys", + "num-traits", + "serde", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "core-foundation-sys" +version = "0.8.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" + +[[package]] +name = "crc32fast" +version = "1.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "csv" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" +dependencies = [ + "csv-core", + "itoa", + "ryu", + "serde_core", +] + +[[package]] +name = "csv-core" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" +dependencies = [ + "memchr", +] + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "find-msvc-tools" +version = "0.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef25905e51abafe4dcea6c15fec58c57b601cdbd0ee53d22ea1d3016c587d39b" + +[[package]] +name = "flate2" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb" +dependencies = [ + "crc32fast", + "miniz_oxide", + "zlib-rs", +] + +[[package]] +name = "futures-core" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" + +[[package]] +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "iana-time-zone" +version = "0.1.65" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" +dependencies = [ + "android_system_properties", + "core-foundation-sys", + "iana-time-zone-haiku", + "js-sys", + "log", + "wasm-bindgen", + "windows-core", +] + +[[package]] +name = "iana-time-zone-haiku" +version = "0.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" +dependencies = [ + "cc", +] + +[[package]] +name = "improvise-core" +version = "0.1.0-rc2" +dependencies = [ + "anyhow", + "improvise-formula", + "indexmap", + "serde", + "siphasher", +] + +[[package]] +name = "improvise-formula" +version = "0.1.0-rc2" +dependencies = [ + "anyhow", + "pest", + "pest_derive", + "serde", +] + +[[package]] +name = "improvise-io" +version = "0.1.0-rc2" +dependencies = [ + "anyhow", + "chrono", + "csv", + "flate2", + "improvise-core", + "improvise-formula", + "indexmap", + "js-sys", + "pest", + "pest_derive", + "serde", + "serde_json", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "indexmap" +version = "2.14.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" +dependencies = [ + "equivalent", + "hashbrown", + "serde", + "serde_core", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "js-sys" +version = "0.3.105" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "log" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "miniz_oxide" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opfs-contract" +version = "0.0.0" +dependencies = [ + "anyhow", + "improvise-io", + "js-sys", + "serde", + "serde_json", + "wasm-bindgen", + "wasm-bindgen-futures", +] + +[[package]] +name = "pest" +version = "2.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d45aeb61b4bf818e12d4205f2466f8c4748f85f4fce0146d1c03d69d753f0ad" +dependencies = [ + "memchr", + "ucd-trie", +] + +[[package]] +name = "pest_derive" +version = "2.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89cc5a242e25ed4e7704d0be240f2cfbe20a8c27e7e252d94835be93d92dc39f" +dependencies = [ + "pest", + "pest_generator", +] + +[[package]] +name = "pest_generator" +version = "2.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7abf21475cc3820fe4b2ca2dc2142902f67a02189f3b5b3a229f4febc01a43e5" +dependencies = [ + "pest", + "pest_meta", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pest_meta" +version = "2.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adba4db388f687393c18c51348d44a41d870ca9df71a2c98172ea3035dc6936e" +dependencies = [ + "pest", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea" + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "ucd-trie" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" + +[[package]] +name = "unicode-ident" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954" + +[[package]] +name = "wasm-bindgen" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.78" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ef4c5d3d2cdf5c54f4231181768f5510842e350db025faf1f7163b1030ed928" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 3.0.6", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-sys" +version = "0.3.105" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9fbddc4a036f00ec4f18c83445bd3115cb306a91da554919a099d9222fe4a7f8" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-core" +version = "0.62.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" +dependencies = [ + "windows-implement", + "windows-interface", + "windows-link", + "windows-result", + "windows-strings", +] + +[[package]] +name = "windows-implement" +version = "0.60.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-interface" +version = "0.59.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-result" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" +dependencies = [ + "windows-link", +] + +[[package]] +name = "windows-strings" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" +dependencies = [ + "windows-link", +] + +[[package]] +name = "zlib-rs" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112" + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/harness/opfs-contract/Cargo.toml b/harness/opfs-contract/Cargo.toml new file mode 100644 index 00000000..03807337 --- /dev/null +++ b/harness/opfs-contract/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "opfs-contract" +version = "0.0.0" +edition = "2024" +description = "Runs improvise-io's storage_contract against OpfsFs in a real browser" +license = "Apache-2.0" +publish = false + +# Deliberately NOT a workspace member. It builds only for wasm32 — it names +# `improvise_io::opfs`, which is `cfg(target_arch = "wasm32")` — so a member +# would break `cargo test --workspace` and `cargo clippy --workspace`. An empty +# `[workspace]` table makes it its own root, which is what keeps it out without +# editing the root manifest (and without a Cargo.toml conflict with every other +# agent). +[workspace] + +[lib] +crate-type = ["cdylib", "rlib"] + +[dependencies] +improvise-io = { path = "../../crates/improvise-io" } +anyhow = "1" +js-sys = "0.3" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +wasm-bindgen = "0.2" +wasm-bindgen-futures = "0.4" + +[package.metadata.wasm-pack.profile.release] +wasm-opt = ["-O", "--enable-bulk-memory", "--enable-nontrapping-float-to-int"] diff --git a/harness/opfs-contract/src/lib.rs b/harness/opfs-contract/src/lib.rs new file mode 100644 index 00000000..ea49d613 --- /dev/null +++ b/harness/opfs-contract/src/lib.rs @@ -0,0 +1,307 @@ +//! Run `improvise_io::storage::contract::storage_contract` against `OpfsFs`, +//! inside a dedicated Web Worker, in a real browser. +//! +//! `OpfsFs` has been compile-checked and never run (gotcha 45). This is the +//! instrument that changes that (improvise-vvl3, absorbing improvise-xf1). It +//! is a *harness* crate, not part of the application: it ships in no bundle +//! and is not a workspace member. +//! +//! # Why a worker +//! +//! `FileSystemFileHandle.createSyncAccessHandle()` exists only in a dedicated +//! Web Worker, and it is the whole reason `Storage` could stay synchronous +//! (gotcha 28). A check that ran on the main thread would fail for a reason +//! that has nothing to do with `OpfsFs`. +//! +//! # Why a report rather than a panic +//! +//! Every check runs and its outcome is reported, so a failure names *which* +//! step and the rest still run. More importantly it is what lets the spec +//! check the checker: Playwright asserts the report contains exactly the +//! checks it expects, so a worker that never started, or a build that dropped +//! a check, is a failure rather than a silent pass. + +use std::path::Path; + +use anyhow::{Result, anyhow, ensure}; +use improvise_io::opfs::OpfsFs; +use improvise_io::storage::Storage; +use improvise_io::storage::contract::storage_contract; +use serde::Serialize; +use wasm_bindgen::prelude::*; + +/// One named check and what happened to it. +#[derive(Serialize)] +struct Check { + name: String, + ok: bool, + detail: String, +} + +#[derive(Default)] +struct Report(Vec); + +impl Report { + async fn run(&mut self, name: &str, f: F) + where + F: AsyncFnOnce() -> Result, + { + let (ok, detail) = match f().await { + Ok(detail) => (true, detail), + Err(e) => (false, format!("{e:#}")), + }; + self.0.push(Check { + name: name.to_string(), + ok, + detail, + }); + } + + fn finish(self) -> String { + serde_json::to_string(&self.0).expect("a Vec serializes") + } +} + +/// Remove every entry in the origin root, so a run does not inherit the last +/// one's leftovers. +/// +/// Uses only the backend's own vocabulary — `refresh`, `delete`, +/// `commit_deletes` — rather than reaching for `removeEntry` directly, so a +/// broken `delete` shows up here as well as in the contract. +async fn wipe(fs: &OpfsFs) -> Result { + fs.refresh().await?; + let names = fs.list(Path::new(""))?; + for name in &names { + fs.delete(name)?; + } + fs.commit_deletes().await?; + fs.refresh().await?; + ensure!( + fs.list(Path::new(""))?.is_empty(), + "wipe left {:?} behind", + fs.list(Path::new(""))? + ); + Ok(names.len()) +} + +/// The whole battery. Returns a JSON array of `Check`. +#[wasm_bindgen] +pub async fn run_checks() -> Result { + let mut report = Report::default(); + + // `open()` is the first thing that can fail, and everything after it needs + // the handle, so it is not a `Check` — a failure here is a thrown error. + let fs = OpfsFs::open() + .await + .map_err(|e| JsError::new(&format!("OpfsFs::open(): {e:#}")))?; + wipe(&fs) + .await + .map_err(|e| JsError::new(&format!("wipe: {e:#}")))?; + + report + .run("open_and_refresh_see_an_empty_origin_root", async || { + fs.refresh().await?; + let listed = fs.list(Path::new(""))?; + ensure!(listed.is_empty(), "a wiped root listed {listed:?}"); + Ok("empty".into()) + }) + .await; + + // `attach` must create the file to obtain a sync access handle for a + // document that does not exist yet — and that created file is zero bytes, + // which is NOT a document. `Storage::list` promises document names, so an + // attached-but-unwritten name must not be listed. Indexing it is exactly + // the improvise-9x6 browser failure (the seed was skipped as "already + // there" and `load_from` got an empty buffer). + report + .run("an_attached_but_unwritten_entry_is_not_a_document", async || { + fs.attach(Path::new("attached.improv")).await?; + let listed = fs.list(Path::new(""))?; + ensure!(listed.is_empty(), "attach listed {listed:?} before anything was written"); + + // ...and a second instance, which shares no index, must not see + // it either: the filter has to be in `refresh`, not only in the + // omission from `attach`. + let other = OpfsFs::open().await?; + other.refresh().await?; + let seen = other.list(Path::new(""))?; + ensure!(seen.is_empty(), "refresh() listed the empty entry: {seen:?}"); + + // The handle is real, though: writing through it works, and then + // it IS a document. + fs.write(Path::new("attached.improv"), b"now it is one")?; + ensure!( + fs.list(Path::new(""))? == vec![std::path::PathBuf::from("attached.improv")], + "a written document was not listed" + ); + Ok("created, unlisted until written".into()) + }) + .await; + + // The hand-rolled `keys()` async-iterator walk in `refresh` is the least + // certain code in opfs.rs (improvise-xf1 named it). One entry could pass + // by accident; several, out of order, cannot. + report + .run("refresh_walks_every_key_and_sorts_them", async || { + for name in ["c.improv", "a.improv", "b.improv"] { + fs.attach(Path::new(name)).await?; + fs.write(Path::new(name), name.as_bytes())?; + } + let fresh = OpfsFs::open().await?; + fresh.refresh().await?; + let listed = fresh.list(Path::new(""))?; + let names: Vec = listed.iter().map(|p| p.display().to_string()).collect(); + ensure!( + names == ["a.improv", "attached.improv", "b.improv", "c.improv"], + "refresh() saw {names:?}" + ); + Ok(format!("{names:?}")) + }) + .await; + + // `FileSystemReadWriteOptions`' `at` handling, the other thing xf1 called + // out. A `write` that appended instead of overwriting, or a `read` that + // started anywhere but 0, is invisible on a single short write and obvious + // across a long one followed by a short one. + report + .run("read_and_write_are_absolute_at_offset_zero", async || { + let p = Path::new("offsets.improv"); + fs.attach(p).await?; + fs.write(p, b"0123456789ABCDEF")?; + ensure!(fs.read(p)? == b"0123456789ABCDEF", "long write did not round-trip"); + fs.write(p, b"xy")?; + let short = fs.read(p)?; + ensure!( + short == b"xy", + "a shorter write left {:?} behind — truncate or `at` is wrong", + String::from_utf8_lossy(&short) + ); + Ok("absolute".into()) + }) + .await; + + // The point of the whole exercise. `attach` is hoisted out because + // acquiring a handle is the async half and `Storage::write` cannot await; + // the contract itself is the shared function, not a transcription of it. + for dir in ["", "docs"] { + report + .run(&format!("storage_contract_at_{:?}", dir), async || { + wipe(&fs).await?; + let d = Path::new(dir); + fs.attach(&d.join("a.improv")).await?; + fs.attach(&d.join("b.improv")).await?; + storage_contract(&fs, d)?; + Ok("passed".into()) + }) + .await; + } + + // Gotcha 28: a delete is visible to `read` and `list` at once and lands in + // OPFS at the end of the batch. Both halves, because either one alone + // would pass against a backend that got the timing exactly backwards. + report + .run("a_delete_is_visible_at_once_and_lands_on_commit", async || { + wipe(&fs).await?; + let p = Path::new("doomed.improv"); + fs.attach(p).await?; + fs.write(p, b"bytes")?; + + fs.delete(p)?; + ensure!(fs.read(p).is_err(), "a deleted document was still readable"); + ensure!(fs.list(Path::new(""))?.is_empty(), "a deleted document was still listed"); + + let before = OpfsFs::open().await?; + before.refresh().await?; + ensure!( + before.list(Path::new(""))?.iter().any(|q| q == p), + "the entry left OPFS before commit_deletes — the queue is not a queue" + ); + + fs.commit_deletes().await?; + let after = OpfsFs::open().await?; + after.refresh().await?; + ensure!( + !after.list(Path::new(""))?.iter().any(|q| q == p), + "commit_deletes did not remove the entry" + ); + Ok("queued then removed".into()) + }) + .await; + + Ok(report.finish()) +} + +/// Write the state the reload check expects, then hand control back. +/// +/// Split from [`after_reload`] because the one thing no in-page check can +/// establish is that OPFS *persisted* — and a reload is the only way to ask. +#[wasm_bindgen] +pub async fn before_reload() -> Result { + inner_before().await.map_err(js) +} + +async fn inner_before() -> Result { + let fs = OpfsFs::open().await?; + wipe(&fs).await?; + + let survivor = Path::new("survivor.improv"); + fs.attach(survivor).await?; + fs.write(survivor, b"persisted across a reload")?; + + let doomed = Path::new("doomed.improv"); + fs.attach(doomed).await?; + fs.write(doomed, b"should not survive")?; + fs.delete(doomed)?; + fs.commit_deletes().await?; + + Ok("staged".into()) +} + +/// Assert what [`before_reload`] left is still there, in a fresh page. +#[wasm_bindgen] +pub async fn after_reload() -> Result { + inner_after().await.map_err(js) +} + +async fn inner_after() -> Result { + let fs = OpfsFs::open().await?; + fs.refresh().await?; + let listed = fs.list(Path::new(""))?; + let names: Vec = listed.iter().map(|p| p.display().to_string()).collect(); + ensure!( + names == ["survivor.improv"], + "after a reload the origin held {names:?}" + ); + + let survivor = Path::new("survivor.improv"); + fs.attach(survivor).await?; + let bytes = fs.read(survivor)?; + ensure!( + bytes == b"persisted across a reload", + "the surviving document read back as {:?}", + String::from_utf8_lossy(&bytes) + ); + Ok("survived".into()) +} + +fn js(e: anyhow::Error) -> JsError { + JsError::new(&format!("{e:#}")) +} + +/// Deliberately fails, so the harness can prove it can see a failure. +/// +/// A green run against a stub, or one that silently skipped because the worker +/// never started, is exactly the check-the-checker failure this repo has a +/// doctrine about. The spec runs this and asserts it is reported as a failure +/// with its message intact, which exercises the *same* reporting path every +/// real check uses. +#[wasm_bindgen] +pub async fn run_canary() -> String { + let mut report = Report::default(); + report + .run("the_canary_must_fail", async || { + Err(anyhow!("the harness can see a failure")) + }) + .await; + report.finish() +} diff --git a/harness/package-lock.json b/harness/package-lock.json new file mode 100644 index 00000000..2904809a --- /dev/null +++ b/harness/package-lock.json @@ -0,0 +1,76 @@ +{ + "name": "improvise-harness", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "improvise-harness", + "devDependencies": { + "@playwright/test": "1.56.1" + } + }, + "node_modules/@playwright/test": { + "version": "1.56.1", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.56.1.tgz", + "integrity": "sha512-vSMYtL/zOcFpvJCW71Q/OEGQb7KYBPAdKh35WNSkaZA75JlAO8ED8UN6GUNTm3drWomcbcqRPFqQbLae8yBTdg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.56.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/playwright": { + "version": "1.56.1", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.56.1.tgz", + "integrity": "sha512-aFi5B0WovBHTEvpM3DzXTUaeN6eN0qWnTkKx4NQaH4Wvcmc153PdaY2UBdSYKaGYw+UyWXSVyxDUg5DoPEttjw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.56.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.56.1", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.56.1.tgz", + "integrity": "sha512-hutraynyn31F+Bifme+Ps9Vq59hKuUCz7H1kDOcBs+2oGguKkWTU50bBWrtz34OUWmIwpBTWDxaRPXrIXkgvmQ==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + } + } +} diff --git a/harness/package.json b/harness/package.json new file mode 100644 index 00000000..2923ee0f --- /dev/null +++ b/harness/package.json @@ -0,0 +1,13 @@ +{ + "name": "improvise-harness", + "private": true, + "type": "module", + "description": "The first instrument pointed at the browser half of improvise (improvise-vvl3)", + "scripts": { + "build": "./build.sh", + "test": "playwright test" + }, + "devDependencies": { + "@playwright/test": "1.56.1" + } +} diff --git a/harness/playwright.config.js b/harness/playwright.config.js new file mode 100644 index 00000000..b526269c --- /dev/null +++ b/harness/playwright.config.js @@ -0,0 +1,30 @@ +import { defineConfig, devices } from "@playwright/test"; + +// `python3 -m http.server` and one spec directory. Deliberately small +// (improvise-vvl3): the point is to be able to run `storage_contract` against +// `OpfsFs` and to read `#status` out of a real page, not to grow a second +// test framework beside `cargo test`. +// +// The server root is `harness/serve`, built by `build.sh` — the assembled +// site plus this harness's own page. A module script will not load from +// `file://`, so an http server is not optional. +export default defineConfig({ + testDir: "./tests", + fullyParallel: false, + workers: 1, + forbidOnly: !!process.env.CI, + retries: 0, + reporter: process.env.CI ? [["github"], ["list"]] : "list", + timeout: 60_000, + use: { + baseURL: "http://127.0.0.1:8099", + trace: "retain-on-failure", + }, + projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], + webServer: { + command: "python3 -m http.server 8099 --bind 127.0.0.1 --directory serve", + url: "http://127.0.0.1:8099/index.html", + reuseExistingServer: !process.env.CI, + timeout: 30_000, + }, +}); diff --git a/harness/tests/opfs.spec.js b/harness/tests/opfs.spec.js new file mode 100644 index 00000000..f8506931 --- /dev/null +++ b/harness/tests/opfs.spec.js @@ -0,0 +1,73 @@ +// `storage_contract` against `OpfsFs`, in a browser (improvise-vvl3/xf1). +// +// `OpfsFs` has been compile-checked and never run. Everything found in a +// browser so far was found by a person pressing keys, which is not a +// repeatable check. This is. +// +// The checks themselves are in Rust (harness/opfs-contract) because that is +// where `OpfsFs` is and where the contract is; this file drives them and reads +// the report. Assertions here are about *the report*, deliberately, so a +// worker that never started or a build that dropped a check fails rather than +// passing quietly. + +import { test, expect } from "@playwright/test"; + +/** Run one entry point in the harness worker. */ +async function run(page, cmd) { + return JSON.parse(await page.evaluate((c) => window.opfs(c), cmd)); +} + +/** The checks the report must contain, in order. Named here as well as in + * Rust on purpose: a check deleted on one side has to be deleted on both, + * which is what stops the battery quietly shrinking. */ +const EXPECTED = [ + "open_and_refresh_see_an_empty_origin_root", + "an_attached_but_unwritten_entry_is_not_a_document", + "refresh_walks_every_key_and_sorts_them", + "read_and_write_are_absolute_at_offset_zero", + 'storage_contract_at_""', + 'storage_contract_at_"docs"', + "a_delete_is_visible_at_once_and_lands_on_commit", +]; + +test.describe("OpfsFs in a real browser", () => { + test.beforeEach(async ({ page }) => { + await page.goto("/opfs/index.html"); + }); + + // Check the checker, and do it first: a green run against a worker that + // never started is the failure this repo has a doctrine about. This drives + // the *same* reporting path every real check uses. + test("the harness can see a failure", async ({ page }) => { + const report = await run(page, "run_canary"); + expect(report).toHaveLength(1); + expect(report[0].name).toBe("the_canary_must_fail"); + expect(report[0].ok).toBe(false); + expect(report[0].detail).toContain("the harness can see a failure"); + }); + + test("every check in the battery runs and passes", async ({ page }) => { + const report = await run(page, "run_checks"); + + // The battery is complete before any of it is believed. + expect(report.map((c) => c.name)).toEqual(EXPECTED); + + const failed = report.filter((c) => !c.ok); + expect( + failed, + `failing checks:\n${failed.map((c) => ` ${c.name}: ${c.detail}`).join("\n")}`, + ).toEqual([]); + }); + + // The one thing no in-page check can establish. gotcha 28: a delete is + // visible immediately and lands in OPFS at the end of the batch — so + // "commit_deletes actually removed it" is only answered by a fresh page. + // This is also the storage half of improvise-bck's persistence criterion. + test("what OPFS held survives a reload, and a committed delete does not", async ({ + page, + }) => { + expect(await page.evaluate(() => window.opfs("before_reload"))).toBe("staged"); + await page.reload(); + expect(await page.evaluate(() => window.opfs("after_reload"))).toBe("survived"); + }); +}); diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js new file mode 100644 index 00000000..b9d3ca17 --- /dev/null +++ b/harness/tests/shell.spec.js @@ -0,0 +1,115 @@ +// The browser shell: the three assertions improvise-hwpi could not write. +// +// That ticket fixed four things and said of itself that the fixes were "as +// unverified as the bugs", because three of them are one-assertion tests that +// needed a browser. These are those three, plus the watchdog — which was found +// by reading, and only because someone went looking. +// +// `web/app.js` and `web/worker.js` have never been rendered or run (gotcha 48). +// Everything below is the first execution of either. + +import { test, expect } from "@playwright/test"; + +const status = (page) => page.locator("#status"); + +test.describe("the shell reports what its peer is doing", () => { + // improvise-hwpi: `on_frame` returns a Result, which wasm-bindgen throws. + // Thrown in a bare listener it reached the devtools console and nothing + // else: the grid simply stopped updating with nothing on screen saying why. + // + // The fault is injected at the *peer*, not in the page — a mocked socket + // that answers the handshake with a frame the client cannot fold — so this + // exercises the real `on_frame` and the real catch. + test("a frame the client cannot fold says so on screen", async ({ page }) => { + await page.routeWebSocket("ws://127.0.0.1:8787/", (ws) => { + ws.onMessage(() => ws.send("{\"NotAFrame\":{}}")); + }); + await page.goto("/index.html?ws=127.0.0.1:8787"); + await expect(status(page)).toContainText("bad frame", { timeout: 15_000 }); + }); + + // improvise-hwpi: without an `error` listener, a connection that never + // opened left #status blank, and "the server is not running" read exactly + // like "still connecting". Port 9 (discard) refuses immediately. + // + // It is asserted over the *history* of #status rather than its final value, + // because of improvise-m18f, which this reproduces: the socket's `close` + // listener fires after `error` and overwrites the precise message with + // "disconnected". That one is filed and deliberately not fixed here — + // suppressing `close` could hide a genuine mid-session disconnect, so it is + // not the same fix as the watchdog's (gotcha 48). Asserting both halves + // pins hwpi's fix AND m18f's bug, so whichever moves, this speaks up. + test("a peer that cannot be reached is named, not left blank", async ({ page }) => { + await page.addInitScript(() => { + window.__statusLog = []; + new MutationObserver(() => { + const el = document.getElementById("status"); + const text = el?.textContent ?? ""; + if (text && window.__statusLog.at(-1) !== text) window.__statusLog.push(text); + }).observe(document, { childList: true, subtree: true, characterData: true }); + }); + await page.goto("/index.html?ws=127.0.0.1:9"); + await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); + + const log = await page.evaluate(() => window.__statusLog); + expect(log.some((t) => t.includes("cannot reach")), `#status said: ${JSON.stringify(log)}`) + .toBe(true); + // improvise-m18f, as it stands today. Delete this line when it is fixed — + // it is here so the fix cannot land silently. + expect(log.at(-1)).toContain("disconnected"); + }); + + // The first-frame watchdog (gotcha 46), found by reading. The condition is + // one no per-transport check would catch: the peer accepted the connection + // and then said nothing. The mock accepts and never answers. + test("a peer that accepts and then says nothing trips the watchdog", async ({ + page, + }) => { + await page.routeWebSocket("ws://127.0.0.1:8788/", () => {}); + await page.goto("/index.html?ws=127.0.0.1:8788"); + await expect(status(page)).toContainText("sent no document", { timeout: 20_000 }); + }); +}); + +test.describe("the standalone deployment", () => { + // No `?ws=`, so `web/app.js` spawns the in-page worker over OPFS: the whole + // application, with no server process. If this draws a grid, then + // worker.js, bindings.rs, OpfsFs, vfs::open and the seed fetch all ran. + test("the page draws the demo document out of the worker", async ({ page }) => { + await page.goto("/index.html"); + await expect(page.locator("#grid table")).toBeVisible({ timeout: 20_000 }); + await expect(status(page)).toContainText("in-page worker"); + // web/demo.improv: Product on row, _Measure on column (gotcha 46). + await expect(page.locator("#grid th", { hasText: "Revenue" })).toBeVisible(); + await expect(page.locator("#grid td.editable").first()).toBeVisible(); + }); + + // improvise-hwpi's third fix: Enter removes the focused , which + // fires `blur`, which called `close` a second time — harmless before, but + // only by accident. The observable is that the commit takes effect exactly + // once and the value on screen is the SERVER's, not the string typed. + // + // `20000` is typed and `20,000` is asserted: the number format is applied + // by the host and relayed back, so a page that simply echoed what was typed + // would show `20000` and fail. + test("committing an edit repaints once, with the server's value", async ({ + page, + }) => { + await page.goto("/index.html?doc=commit-test.improv"); + await expect(page.locator("#grid table")).toBeVisible({ timeout: 20_000 }); + + const cell = page.locator("#grid td.editable").first(); + const before = await cell.textContent(); + await cell.click(); + const input = cell.locator("input"); + await expect(input).toBeFocused(); + await input.fill("20000"); + await input.press("Enter"); + + await expect(cell).toHaveText("20,000", { timeout: 10_000 }); + expect(before).not.toBe("20,000"); + // One editor, closed once: a second `close` would reopen nothing, but a + // stray input left in the DOM is what the guard exists to prevent. + await expect(page.locator("#grid input")).toHaveCount(0); + }); +}); diff --git a/harness/www/opfs/index.html b/harness/www/opfs/index.html new file mode 100644 index 00000000..e23436c5 --- /dev/null +++ b/harness/www/opfs/index.html @@ -0,0 +1,48 @@ + + + + + improvise — OPFS contract harness + + + +

OPFS contract harness

+ 
idle
+ + + diff --git a/harness/www/opfs/worker.js b/harness/www/opfs/worker.js new file mode 100644 index 00000000..71b7d31a --- /dev/null +++ b/harness/www/opfs/worker.js @@ -0,0 +1,30 @@ +// The dedicated Web Worker the OPFS contract runs inside. +// +// `createSyncAccessHandle` exists only here — that is the whole reason +// `Storage` could stay synchronous (gotcha 28), and the reason this file +// exists rather than the page calling the bundle directly. +// +// Every decision is in Rust (harness/opfs-contract). This moves strings. + +import init, { + run_checks, + run_canary, + before_reload, + after_reload, +} from "./pkg/opfs_contract.js"; + +const ready = init(); + +const ENTRY = { run_checks, run_canary, before_reload, after_reload }; + +self.onmessage = async (event) => { + const { id, cmd } = event.data; + try { + await ready; + const fn = ENTRY[cmd]; + if (!fn) throw new Error(`no such entry point: ${cmd}`); + self.postMessage({ id, ok: true, value: await fn() }); + } catch (err) { + self.postMessage({ id, ok: false, error: String(err?.message ?? err) }); + } +}; diff --git a/web/assemble.sh b/web/assemble.sh new file mode 100755 index 00000000..0b4572f1 --- /dev/null +++ b/web/assemble.sh @@ -0,0 +1,59 @@ +#!/bin/sh +# Assemble the static site from `web/` into `site/`. +# +# `web/` is the *source* directory: it also holds build.sh, assemble.sh and +# .gitignore, which have no business being served. This copies what ships. +# +# Extracted from the CI workflow so that the Pages job and the browser harness +# assemble the same tree — a harness that runs against a different arrangement +# of files is testing something nobody deploys (improvise-vvl3). Same reason +# `web/build.sh` exists rather than two copies of the wasm-pack invocation. +# +# Usage: web/assemble.sh [OUT_DIR] [CACHE_BUST_TOKEN] +# +# The token cache-busts the two generated bundles. Pages serves everything +# `max-age=600` and the wasm-pack filenames carry no content hash, so for ten +# minutes after a deploy a returning visitor runs the PREVIOUS build — which +# during improvise-9x6 debugging made a shipped fix look like it had not +# shipped, twice. CI passes the commit SHA; a local run defaults to `dev`. +# +# Both halves are stamped, because a fresh glue file does NOT pull a fresh +# .wasm: the glue locates it with `new URL('..._bg.wasm', import.meta.url)`, +# and a URL constructed against a base does not inherit the base's query +# string. Stamping only the import specifier would refresh the wrapper and +# keep the stale binary — the failure mode this is here to end. +set -eu + +root=$(cd "$(dirname "$0")/.." && pwd) +out=${1:-"$root/site"} +token=${2:-dev} + +test -d "$root/web/pkg" || { echo "web/pkg missing — run web/build.sh first" >&2; exit 1; } +test -d "$root/web/worker-pkg" || { echo "web/worker-pkg missing — run web/build.sh first" >&2; exit 1; } + +rm -rf "$out" +mkdir -p "$out" +cp -f "$root/web/index.html" "$root/web/app.js" "$root/web/style.css" "$root/web/worker.js" "$out/" +# The document the page opens on a browser that has never seen one. +# web/worker.js fetches it by the ?seed= URL and hands the bytes to Rust, +# which writes them through the VFS — so leaving it out is an empty grid, not +# a 404 the page could report. +cp -f "$root/web/demo.improv" "$out/" +cp -rf "$root/web/pkg" "$out/pkg" +cp -rf "$root/web/worker-pkg" "$out/worker-pkg" +rm -f "$out/pkg/.gitignore" "$out/pkg/package.json" +rm -f "$out/worker-pkg/.gitignore" "$out/worker-pkg/package.json" + +# BSD sed (macOS) needs an argument to -i; GNU sed must not be given one. +sedi() { sed "$1" "$2" > "$2.tmp" && mv -f "$2.tmp" "$2"; } + +sedi "s|\./pkg/improvise_wasm_client\.js|&?v=${token}|" "$out/app.js" +sedi "s|\./worker-pkg/improvise_worker_server\.js|&?v=${token}|" "$out/worker.js" +sedi "s|'improvise_wasm_client_bg\.wasm'|'improvise_wasm_client_bg.wasm?v=${token}'|" \ + "$out/pkg/improvise_wasm_client.js" +sedi "s|'improvise_worker_server_bg\.wasm'|'improvise_worker_server_bg.wasm?v=${token}'|" \ + "$out/worker-pkg/improvise_worker_server.js" + +grep -n "?v=" "$out/app.js" "$out/worker.js" \ + "$out/pkg/improvise_wasm_client.js" \ + "$out/worker-pkg/improvise_worker_server.js" From 9c6de5754277356016932866552f9ca49ee577a5 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:45:14 -0700 Subject: [PATCH 28/72] feat(gp-tree): denormalize a champion, and hand back quality as data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three additions in `fit.rs`, all needed by a consumer that cannot print. `denormalize` is the inverse of `standardize` applied to a tree. A fit runs against a standardized target, so its leaf scores are in standard deviations; `fit_csv` closes that by *printing* `≈ score × std + mean` beside the tree, which a caller writing the tree into a document has nowhere to do. The splits are deliberately untouched — `standardize` rescales only the target, so a threshold is already in its feature's units and rescaling it would be the bug. `RegressionQuality` / `regression_quality` and `importance_ranking` move the arithmetic `report_regression_cv` and `report_importance` were doing inline into the library, and the CLI now prints from them. This is the one-home rule rather than a convenience: a second consumer computing R² against the predict-the-mean baseline would be a second derivation of the number a user decides whether to believe the rule by. The baseline is part of the type for the same reason — a fit no better than the mean is not a weak rule, it is a rule a user will believe. `gp-tree` still returns data and still never prints (gotcha 37a). Evidence. The CLI's own output is **byte-identical** before and after on both paths — the built-in classification demo and a regression CSV — and that check was proved discriminating by injecting `baseline_mse: ss_res` and watching the diff fire on exactly the baseline line. Nine unit tests cover the new functions, including the negative-R² case (reported, not clamped, because "learned nothing" and "actively misleading" must not read alike), the constant target, and the empty pool. Co-Authored-By: Claude Opus 5 (1M context) --- crates/gp-tree/examples/fit_csv.rs | 38 ++--- crates/gp-tree/src/fit.rs | 221 +++++++++++++++++++++++++++++ 2 files changed, 232 insertions(+), 27 deletions(-) diff --git a/crates/gp-tree/examples/fit_csv.rs b/crates/gp-tree/examples/fit_csv.rs index e0742844..11927a71 100644 --- a/crates/gp-tree/examples/fit_csv.rs +++ b/crates/gp-tree/examples/fit_csv.rs @@ -83,8 +83,8 @@ use gp_tree::data::{ read_csv, }; use gp_tree::fit::{ - FitConfig, Row, argmax, cv_classify, cv_regress, fit, misclass_rate, one_vs_rest, - permutation_importance, standardize, + FitConfig, RegressionQuality, Row, argmax, cv_classify, cv_regress, fit, importance_ranking, + misclass_rate, one_vs_rest, permutation_importance, regression_quality, standardize, }; use gp_tree::{Spec, SplitMix64, Tree, eval, is_valid, render, size}; @@ -137,16 +137,8 @@ fn confidence(scores: &[f32]) -> f32 { /// Print the most important features (by permutation importance), as a share of /// total importance with a little bar. Zero-importance features are omitted. fn report_importance(names: &[String], imp: &[f32]) { - let total: f32 = imp.iter().sum::().max(1e-9); - let mut order: Vec = (0..imp.len()).collect(); - order.sort_by(|&a, &b| { - imp[b] - .partial_cmp(&imp[a]) - .unwrap_or(std::cmp::Ordering::Equal) - }); println!("feature importance (permutation):"); - for &j in order.iter().take(12).filter(|&&j| imp[j] > 0.0) { - let share = 100.0 * imp[j] / total; + for (j, share) in importance_ranking(imp).into_iter().take(12) { println!( " {:<24} {share:>5.1}% {}", names[j], @@ -157,22 +149,14 @@ fn report_importance(names: &[String], imp: &[f32]) { /// Print pooled-CV regression metrics against the predict-the-mean baseline. fn report_regression_cv(pooled: &[(f32, f32)]) { - let n = pooled.len() as f32; - let ybar = pooled.iter().map(|&(_, a)| a).sum::() / n; - let ss_res: f32 = pooled.iter().map(|&(p, a)| (p - a).powi(2)).sum(); - let ss_tot: f32 = pooled.iter().map(|&(_, a)| (a - ybar).powi(2)).sum(); - let mse = ss_res / n; - let baseline = ss_tot / n; // MSE of always predicting the mean - let r2 = if ss_tot > 0.0 { - 1.0 - ss_res / ss_tot - } else { - 0.0 - }; - println!( - " held-out MSE {mse:.4} RMSE {:.4} R\u{00b2} {r2:.3}", - mse.sqrt() - ); - println!(" baseline (predict mean) MSE {baseline:.4}"); + let RegressionQuality { + mse, + rmse, + r2, + baseline_mse, + } = regression_quality(pooled); + println!(" held-out MSE {mse:.4} RMSE {rmse:.4} R\u{00b2} {r2:.3}"); + println!(" baseline (predict mean) MSE {baseline_mse:.4}"); } /// Print pooled-CV classification metrics: accuracy vs the majority-class diff --git a/crates/gp-tree/src/fit.rs b/crates/gp-tree/src/fit.rs index 24b7a484..bdb1a7eb 100644 --- a/crates/gp-tree/src/fit.rs +++ b/crates/gp-tree/src/fit.rs @@ -254,6 +254,102 @@ pub fn standardize(rows: &[Row]) -> (Vec>, f32, f32) { (normalized, mean, std) } +/// The inverse of [`standardize`], applied to a tree rather than to rows. +/// +/// A fit runs against a standardized target, so its leaf scores are in +/// standard deviations and a caller reading them as the user's units is out by +/// an affine map. `fit_csv` closes that by *printing* the map beside the tree; +/// a caller that has to hand the tree to something else — improvise writes it +/// into a document as a formula — has nowhere to print it, so the map has to be +/// folded into the leaves instead. +/// +/// Splits are untouched, and that is the whole reason this is sound: +/// [`standardize`] rescales only the target, so a threshold is already in the +/// feature's own units and re-scaling it would be the bug. +pub fn denormalize(t: &Tree, mean: f32, std: f32) -> Tree { + match t { + Tree::Leaf(s) => Tree::Leaf(s * std + mean), + Tree::Split { + feat, + thresh, + lo, + hi, + } => Tree::Split { + feat: *feat, + thresh: *thresh, + lo: Box::new(denormalize(lo, mean, std)), + hi: Box::new(denormalize(hi, mean, std)), + }, + } +} + +/// How well a set of pooled `(prediction, actual)` pairs did, **and what doing +/// nothing would have scored**. +/// +/// The baseline is the point. A tree with a plausible-looking RMSE that is no +/// better than always predicting the mean is not a weak rule, it is *no rule* — +/// and it is worse than no rule at all, because it is one a user will believe. +/// So the comparison is part of the answer rather than something a caller may +/// forget to compute. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct RegressionQuality { + /// Held-out mean squared error. + pub mse: f32, + /// Its square root, in the target's own units. + pub rmse: f32, + /// `1 - ss_res/ss_tot`: the share of the target's variance explained. + /// Zero when the target does not vary, and **negative** when the fit does + /// worse than the mean — which is a real outcome, not a clamped one. + pub r2: f32, + /// The mean squared error of always predicting the target's mean. + pub baseline_mse: f32, +} + +/// [`RegressionQuality`] over pooled cross-validation pairs. +/// +/// Empty pairs give all-zero, via [`mean_over`]: `cv_regress` pools nothing +/// below two folds, and "no evidence" reads the same as "explained nothing". +pub fn regression_quality(pooled: &[(f32, f32)]) -> RegressionQuality { + let n = pooled.len(); + let ybar = mean_over(pooled.iter().map(|&(_, a)| a).sum::(), n); + let ss_res: f32 = pooled.iter().map(|&(p, a)| (p - a).powi(2)).sum(); + let ss_tot: f32 = pooled.iter().map(|&(_, a)| (a - ybar).powi(2)).sum(); + let mse = mean_over(ss_res, n); + RegressionQuality { + mse, + rmse: mse.sqrt(), + r2: if ss_tot > 0.0 { + 1.0 - ss_res / ss_tot + } else { + 0.0 + }, + baseline_mse: mean_over(ss_tot, n), + } +} + +/// [`permutation_importance`] scores turned into a ranking: `(feature index, +/// percentage share)`, most important first, with the features that moved the +/// error not at all dropped. +/// +/// The share is of the total importance, so it reads as "this feature carried +/// 40% of what the model knows". A total of zero — every feature irrelevant, or +/// a tree that is one leaf — gives an empty ranking rather than a division by +/// zero. +pub fn importance_ranking(imp: &[f32]) -> Vec<(usize, f32)> { + let total: f32 = imp.iter().sum::().max(1e-9); + let mut order: Vec = (0..imp.len()).collect(); + order.sort_by(|&a, &b| { + imp[b] + .partial_cmp(&imp[a]) + .unwrap_or(std::cmp::Ordering::Equal) + }); + order + .into_iter() + .filter(|&j| imp[j] > 0.0) + .map(|j| (j, 100.0 * imp[j] / total)) + .collect() +} + /// Split `0..n` into `k` folds after a deterministic shuffle, so held-out rows /// are a representative sample rather than a contiguous block. /// @@ -418,6 +514,131 @@ pub fn permutation_importance( imp } +#[cfg(test)] +mod quality_tests { + use super::*; + + /// The baseline is the whole point of the type. A "prediction" that is + /// always the target's mean explains nothing, and the numbers have to say + /// so: R² exactly zero, and the fit's MSE equal to the baseline's. + #[test] + fn predicting_the_mean_scores_zero_and_ties_the_baseline() { + let actual = [1.0f32, 2.0, 3.0, 4.0]; + let mean = 2.5; + let q = regression_quality(&actual.map(|a| (mean, a))); + assert_eq!(q.r2, 0.0); + assert_eq!(q.mse, q.baseline_mse); + assert_eq!(q.rmse, q.mse.sqrt()); + } + + /// A perfect fit is R² 1 and zero error, while the baseline still carries + /// the target's own variance — so the two really are measuring different + /// things rather than both reading off the predictions. + #[test] + fn a_perfect_fit_is_one_and_the_baseline_keeps_the_variance() { + let q = regression_quality(&[(1.0, 1.0), (2.0, 2.0), (3.0, 3.0), (4.0, 4.0)]); + assert_eq!(q.r2, 1.0); + assert_eq!(q.mse, 0.0); + assert!(q.baseline_mse > 1.0, "{q:?}"); + } + + /// Worse than the mean is a real outcome and is reported as one. Clamping + /// it at zero would make "learned nothing" and "actively misleading" read + /// alike, which is the failure this whole report exists to prevent. + #[test] + fn a_fit_worse_than_the_mean_reports_a_negative_r2() { + let q = regression_quality(&[(4.0, 1.0), (1.0, 4.0)]); + assert!(q.r2 < 0.0, "{q:?}"); + assert!(q.mse > q.baseline_mse, "{q:?}"); + } + + /// A target that never varies has no variance to explain, so R² is defined + /// as zero rather than as a division by zero. + #[test] + fn a_constant_target_explains_nothing_rather_than_dividing_by_zero() { + let q = regression_quality(&[(7.0, 7.0), (7.0, 7.0)]); + assert_eq!(q.r2, 0.0); + assert_eq!(q.baseline_mse, 0.0); + } + + /// `cv_regress` pools nothing below two folds, so the caller can be handed + /// an empty slice. All-zero, and no NaN — the `mean_over` rule. + #[test] + fn no_pairs_is_all_zero_and_never_nan() { + let q = regression_quality(&[]); + assert_eq!(q.mse, 0.0); + assert_eq!(q.rmse, 0.0); + assert_eq!(q.r2, 0.0); + assert_eq!(q.baseline_mse, 0.0); + } + + /// Shares are of the total, most important first, and a feature the model + /// does not use at all is left out rather than listed at 0%. + #[test] + fn the_ranking_is_ordered_shares_with_the_unused_features_dropped() { + let ranked = importance_ranking(&[1.0, 0.0, 3.0]); + assert_eq!(ranked.len(), 2, "the zero-importance feature is listed"); + assert_eq!(ranked[0].0, 2); + assert_eq!(ranked[1].0, 0); + assert!((ranked[0].1 - 75.0).abs() < 1e-3, "{ranked:?}"); + assert!((ranked[1].1 - 25.0).abs() < 1e-3, "{ranked:?}"); + } + + /// Every feature irrelevant — or a champion that is a single leaf, which + /// scores every shuffle identically — is an empty ranking, not a division + /// by zero. + #[test] + fn nothing_important_ranks_nothing() { + assert!(importance_ranking(&[0.0, 0.0]).is_empty()); + } + + /// `denormalize` is the inverse of the affine map `standardize` applied, + /// so a tree fitted on standardized targets predicts in the user's own + /// units — which is the only form that can be written into a document. + /// The *splits* must not move: `standardize` rescales the target only, so + /// a threshold is already in its feature's units. + #[test] + fn denormalize_rescales_the_leaves_and_leaves_the_splits_alone() { + let t = Tree::Split { + feat: 0, + thresh: 0.5, + lo: Box::new(Tree::Leaf(-1.0)), + hi: Box::new(Tree::Leaf(2.0)), + }; + let back = denormalize(&t, 100.0, 10.0); + assert_eq!(eval(&back, &[0.0]), 90.0); + assert_eq!(eval(&back, &[1.0]), 120.0); + let Tree::Split { thresh, .. } = back else { + panic!("denormalize turned a split into a leaf"); + }; + assert_eq!(thresh, 0.5, "the threshold was rescaled with the leaves"); + } + + /// The round trip that earns the function: standardize a large-magnitude + /// target, fit *something* over it, denormalize, and the tree predicts the + /// original units. Done against a hand-built tree rather than a search, so + /// it pins the arithmetic and not the GA. + #[test] + fn a_tree_fitted_on_standardized_rows_predicts_original_units_after_denormalize() { + let rows: Vec> = [1000.0f32, 2000.0, 3000.0, 4000.0] + .iter() + .enumerate() + .map(|(i, &t)| Row::new(vec![i as f32], t)) + .collect(); + let (normalized, mean, std) = standardize(&rows); + // The exact rule, expressed in standardized space. + let standardized_tree = Tree::Split { + feat: 0, + thresh: 1.5, + lo: Box::new(Tree::Leaf(normalized[0].target)), + hi: Box::new(Tree::Leaf(normalized[3].target)), + }; + let real = denormalize(&standardized_tree, mean, std); + assert!((eval(&real, &[0.0]) - 1000.0).abs() < 0.5); + assert!((eval(&real, &[3.0]) - 4000.0).abs() < 0.5); + } +} + #[cfg(test)] mod tests { use super::*; From e46429ecfce88a0eef91b4ab02515631d7926cb5 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:45:23 -0700 Subject: [PATCH 29/72] =?UTF-8?q?feat(fit):=20fit-measure=20=E2=80=94=20a?= =?UTF-8?q?=20fitted=20tree=20as=20an=20ordinary=20improvise=20formula?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-szrt / improvise-vx3. `improvise-gp` was a bridge with nothing crossing it; this is the crossing. `improvise_gp::fit::fit_measure` is the sequence between the two halves that already existed: `dataset` (a View's records as a feature matrix) and `fitted_formula` (a champion as formula text the document can resolve). It lives in `improvise-gp` rather than in the command because none of it is terminal-specific — a browser fitting the same measure must run exactly these steps in exactly this order or it gets a different tree from the same seed. `command/cmd/fit.rs` is the user-facing surface: `fit-measure [] []`, returning `SetStatus` + `MarkDirty` + `AddFormula`. After it there is nothing GP-specific in the document — the rule is a `## Formulas` line, recomputed by the fixed-point evaluator, editable in the formula panel, drillable, saved with no grammar change. **Where the fit runs, and why.** In `execute`, on the command side. A fit is decision logic — a pure function of (Model, View, target, seed, FitConfig) that mutates nothing — so it belongs where decision logic belongs; putting an expensive stochastic search inside `Effect::apply` would mix it into the one place the app mutates. That buys purity, not asynchrony, so vx3's AC 4 is met on its "bounded" arm: `FitRequest::default` is a smaller search than the CLI's, **measured** at 6ms / 12ms / 105ms over 12 / 200 / 2,000 records in release. **Three things the sequence had to get right**, each of which fails quietly: * The fit is standardized and the formula must not be. Written down untouched, a champion predicts standard deviations — a well-formed formula evaluating to ±5 where the user's data is in the thousands, and nothing downstream can see it. `denormalize` folds the affine map back into the leaves first. * The rule is stored as a *new* measure (` (fit)`), never as the target itself: gotcha 7 makes a formula target and an explicit item one item, so the cells the fit was learned from would stop being visible. * `Quality` is an enum. `cv_regress` pools nothing below two folds and scoring an empty pool gives R² exactly 0 — "nobody checked" reading as "this rule is worth nothing", which is the loudest thing a fit can say. **A bug this found, fixed with its own failing test first.** `_Measure` is a filtering category, and `Workbook::new`'s default view puts it on `Axis::Page` — so `dataset::filter_coords` pinned it to one measure and the feature matrix collapsed. On a document whose target is `_Measure`'s first item every feature column came back blank; otherwise the table was empty and the error blamed the data ("no visible record has a Revenue value"). `_Measure` is the coordinate the columns are spread across, so paging it is not "which records am I looking at", it is "throw away every column". Every existing test in that module built its view with `View::new` + `on_category_added`, under which an underscore category lands on `Axis::None` — the fixture was building the one world the code worked in. The two new regression tests compare row *contents*, because asserting the header and the row count passes with the bug present. Evidence, and the faults injected to earn it. Dropping the `denormalize` call fails `the_fitted_formula_predicts_in_the_documents_own_units` and nothing else. Both `_Measure`-on-Page tests were watched failing before the one-line fix. Every new test asserts against a real document — never a helper that fabricates a `_Measure` item per feature name, which is what let improvise-8xhy hide. 1216 passing, up 27 from 1189; suite still well under 2s. Also fixes the refusal message, which a `rustfmt` pass over its multi-line literal had left with runs of ten spaces mid-sentence — invisible in the source and to every test that only asked whether a substring was present, and visible to precisely the user it exists for. Co-Authored-By: Claude Opus 5 (1M context) --- Cargo.lock | 1 + crates/improvise-command/Cargo.toml | 1 + .../improvise-command/src/command/cmd/fit.rs | 291 +++++++++ .../improvise-command/src/command/cmd/mod.rs | 1 + .../src/command/cmd/registry.rs | 2 + crates/improvise-gp/src/dataset.rs | 64 +- crates/improvise-gp/src/fit.rs | 616 ++++++++++++++++++ crates/improvise-gp/src/lib.rs | 10 +- 8 files changed, 982 insertions(+), 4 deletions(-) create mode 100644 crates/improvise-command/src/command/cmd/fit.rs create mode 100644 crates/improvise-gp/src/fit.rs diff --git a/Cargo.lock b/Cargo.lock index 18ddea3d..7688312c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -862,6 +862,7 @@ dependencies = [ "anyhow", "improvise-core", "improvise-formula", + "improvise-gp", "improvise-io", "improvise-protocol", "serde", diff --git a/crates/improvise-command/Cargo.toml b/crates/improvise-command/Cargo.toml index 4c28cc02..4fc83907 100644 --- a/crates/improvise-command/Cargo.toml +++ b/crates/improvise-command/Cargo.toml @@ -11,6 +11,7 @@ improvise-core = { path = "../improvise-core" } improvise-protocol = { path = "../improvise-protocol" } improvise-formula = { path = "../improvise-formula" } improvise-io = { path = "../improvise-io" } +improvise-gp = { path = "../improvise-gp" } anyhow = "1" serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/crates/improvise-command/src/command/cmd/fit.rs b/crates/improvise-command/src/command/cmd/fit.rs new file mode 100644 index 00000000..a762385a --- /dev/null +++ b/crates/improvise-command/src/command/cmd/fit.rs @@ -0,0 +1,291 @@ +//! `fit-measure`: evolve a decision tree over what the active view shows and +//! store it as an ordinary improvise formula. +//! +//! After this command there is nothing GP-specific left in the document — the +//! rule is a `## Formulas` line like any other, recomputed by the fixed-point +//! evaluator, editable in the formula panel, drillable, and saved with no new +//! grammar (improvise-21d's enabling fact, and improvise-7na's answer). +//! +//! # The fit runs inside `execute`, and that is the architecture rather than a +//! shortcut +//! +//! A `Cmd` is pure over `&CmdContext` and returns effects; an `Effect` is the +//! only mutation site. A fit is **decision logic** — a pure function of +//! `(Model, View, target, seed, FitConfig)` that mutates nothing — so it +//! belongs on the command side, and what the effects carry is its *result*. +//! Putting the search inside an `Effect::apply` would have mixed an expensive +//! stochastic computation into the one place the app mutates, which is the +//! split's whole point. +//! +//! What this does **not** buy is asynchrony. The command runs on the thread +//! that draws the next frame, so a fit is bounded rather than cancellable: +//! [`FitRequest::default`] is a deliberately smaller search than the CLI's +//! (improvise-vx3's AC 4 asks for "cancellable **or** bounded"). Moving it off +//! the main loop is a change to `draw.rs`'s event loop, not to this file, and +//! nothing here forecloses it — the fit is already a pure function that could +//! be handed to a worker and have its `FitOutcome` posted back as the same +//! effects. +//! +//! # What a refusal is for +//! +//! `improvise_gp::fit_measure` fails rather than storing something misleading +//! — most sharply when the champion splits on an encoding artefact like +//! `Region=North`, which is not an item of anyone's document, so every cell of +//! the stored measure would be `Error("ref:…")` (improvise-8xhy, repo-map +//! gotcha 37d). This command is where a user actually meets that restriction, +//! so the refusal goes to the status line **whole**: it names the features that +//! could not be resolved, because "the fit failed" is not something a user can +//! act on and "it split on Region, which is a category rather than a measure" +//! is. + +use improvise_gp::fit::{FitRequest, fit_measure, fitted_measure_name}; + +use crate::command::cmd::core::{Cmd, CmdContext}; +use crate::ui::effect::{self, Effect}; + +/// Fit `target` from the other measures and categories the view shows, storing +/// the rule as the measure `into`. +#[derive(Debug)] +pub struct FitMeasure { + pub target: String, + pub into: String, + pub request: FitRequest, +} + +impl FitMeasure { + /// A nameless instance for `register_pure`, which reads a command's name + /// off a prototype rather than off a string literal (gotcha 15). + pub fn prototype() -> Self { + Self { + target: String::new(), + into: String::new(), + request: FitRequest::default(), + } + } + + /// `fit-measure [] []`. + /// + /// The seed is an argument rather than a clock read for the reason the CLI + /// fixes its own default: two fits of the same document should agree, and a + /// user who wants a different search asks for one. It is reported back on + /// the status line so the fit that landed can be asked for again. + pub fn parse(args: &[String]) -> Result, String> { + let [target, rest @ ..] = args else { + return Err("fit-measure requires a target measure".into()); + }; + let (into, seed) = match rest { + [] => (fitted_measure_name(target), None), + [into] => (into.clone(), None), + [into, seed] => ( + into.clone(), + Some( + seed.parse::() + .map_err(|e| format!("fit-measure: {seed:?} is not a seed: {e}"))?, + ), + ), + _ => { + return Err(format!( + "fit-measure takes 1-3 arguments, got {}", + args.len() + )); + } + }; + Ok(Box::new(FitMeasure { + target: target.clone(), + into, + request: FitRequest { + seed: seed.unwrap_or(FitRequest::default().seed), + ..FitRequest::default() + }, + })) + } +} + +impl Cmd for FitMeasure { + fn name(&self) -> &'static str { + "fit-measure" + } + + fn execute(&self, ctx: &CmdContext) -> Vec> { + match fit_measure(ctx.model, ctx.view, &self.target, &self.into, &self.request) { + // The report first and `AddFormula` last, so a formula the emitter + // spelled but the parser will not take leaves *its* message on + // screen rather than this command's optimistic one (gotcha 20). + Ok(outcome) => vec![ + effect::set_status(outcome.status_line()), + Box::new(effect::MarkDirty), + Box::new(effect::AddFormula { + raw: outcome.formula.raw.clone(), + target_category: outcome.formula.target_category.clone(), + }), + ], + Err(e) => vec![effect::set_status(format!("fit-measure: {e}"))], + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::command::cmd::test_helpers::*; + use crate::model::category::CategoryKind; + use crate::model::cell::{CellKey, CellValue}; + use crate::view::Axis; + + const MEASURE: &str = "_Measure"; + const REGION: &str = "Region"; + const TAG: &str = "Tag"; + + /// A document a tree can learn exactly — `Score` steps at `Cost` 50 — in + /// the units a real document uses, and with `Region` single-valued so its + /// one-hot has the degenerate range `(1, 1)` that `simplify` prunes. The + /// champion therefore cannot reach an unresolvable feature and the success + /// path is deterministic rather than seed-dependent; `Tag` is a `Label` + /// category, which `dataset` keeps out of the features while still making + /// the twelve records distinct. See `improvise_gp::fit`'s own fixture. + fn workbook() -> crate::workbook::Workbook { + let mut wb = crate::workbook::Workbook::new("sales"); + wb.add_category(REGION).expect("first category"); + wb.add_item(REGION, "North").expect("Region exists"); + wb.add_category(TAG).expect("second category"); + wb.model.category_mut(TAG).expect("just added").kind = CategoryKind::Label; + for i in 0..12u32 { + wb.add_item(TAG, format!("r{i}")).expect("Tag exists"); + } + let measure = wb + .model + .category_mut(MEASURE) + .expect("_Measure always exists"); + for item in ["Cost", "Score"] { + measure.add_item(item); + } + for i in 0..12u32 { + let cost = 10.0 * f64::from(i); + wb.model.set_cell(cell(i, "Cost"), CellValue::Number(cost)); + wb.model.set_cell( + cell(i, "Score"), + CellValue::Number(if cost > 50.0 { 4000.0 } else { 1000.0 }), + ); + } + wb.active_view_mut().set_axis(REGION, Axis::Row); + wb + } + + fn cell(i: u32, measure: &str) -> CellKey { + CellKey::new(vec![ + (REGION.to_string(), "North".to_string()), + (TAG.to_string(), format!("r{i}")), + (MEASURE.to_string(), measure.to_string()), + ]) + } + + fn cmd(args: &[&str]) -> Box { + FitMeasure::parse(&args.iter().map(|s| s.to_string()).collect::>()) + .expect("the arguments parse") + } + + /// Run `fit-measure` the way the app does: build the read-only context + /// from a workbook, execute, and apply the effects to a real `App`. + fn run(args: &[&str]) -> crate::ui::app::App { + let wb = workbook(); + let layout = make_layout(&wb); + let registry = make_registry(); + let effects = cmd(args).execute(&make_ctx(&wb, &layout, ®istry)); + let mut app = crate::ui::app::App::new(workbook(), None); + app.apply_effects(effects); + app + } + + /// The end-to-end claim of improvise-vx3, asserted against a real + /// document: a command fits, the formula lands through the ordinary + /// `add_formula` path, and the new measure evaluates to numbers. + #[test] + fn the_command_installs_a_formula_the_document_evaluates() { + let app = run(&["Score"]); + let model = &app.model_state.workbook.model; + assert!( + model.formulas().iter().any(|f| f.target == "Score (fit)"), + "no fitted formula was stored: {:?} / {:?}", + model.formulas().iter().map(|f| &f.raw).collect::>(), + app.view_state.status_msg, + ); + for i in 0..12u32 { + match model.evaluate(&cell(i, "Score (fit)")) { + Some(CellValue::Number(_)) => {} + other => panic!("record r{i} came back as {other:?}"), + } + } + } + + /// Adding a rule is a change to the document, so the command marks it + /// dirty — otherwise the fit is lost on the next quit without a prompt. + #[test] + fn a_fit_marks_the_document_dirty() { + assert!( + !crate::ui::app::App::new(workbook(), None).model_state.dirty, + "the fixture starts clean" + ); + assert!(run(&["Score"]).model_state.dirty); + } + + /// improvise-vx3 is emphatic that the rule alone is not the deliverable: + /// a tidy tree that generalizes no better than the mean is worse than + /// useless. So the status line carries the held-out comparison, what the + /// rule leans on, and the seed that reproduces it. + #[test] + fn the_status_line_reports_quality_against_the_baseline() { + let app = run(&["Score"]); + let status = &app.view_state.status_msg; + for expected in ["R\u{b2}", "baseline", "leans on", "seed"] { + assert!( + status.contains(expected), + "{expected:?} missing from {status:?}" + ); + } + } + + /// A refusal reaches the user *whole*. "fit failed" is not actionable; + /// naming the measure that does not exist is (improvise-8xhy's argument, + /// arriving at the place a user meets it). + #[test] + fn a_refusal_names_what_went_wrong_and_stores_nothing() { + let wb = workbook(); + let layout = make_layout(&wb); + let registry = make_registry(); + let effects = cmd(&["Margin"]).execute(&make_ctx(&wb, &layout, ®istry)); + assert_eq!(effects.len(), 1, "a refused fit must emit no model change"); + let mut app = crate::ui::app::App::new(workbook(), None); + app.apply_effects(effects); + assert!( + app.view_state + .status_msg + .contains("not a measure holding data"), + "{:?}", + app.view_state.status_msg + ); + assert!(app.model_state.workbook.model.formulas().is_empty()); + } + + /// gotcha 37b: the seed is half of what reproduces a fit, so it is an + /// argument — and a different one really does search differently. + #[test] + fn the_seed_argument_changes_the_search() { + let wb = workbook(); + let layout = make_layout(&wb); + let registry = make_registry(); + let ctx = make_ctx(&wb, &layout, ®istry); + assert_ne!( + effects_debug(&cmd(&["Score", "A", "1"]).execute(&ctx)), + effects_debug(&cmd(&["Score", "A", "99"]).execute(&ctx)), + "two seeds produced the identical batch of effects" + ); + } + + #[test] + fn a_target_is_required_and_a_bad_seed_is_named() { + assert!(FitMeasure::parse(&[]).is_err()); + let err = FitMeasure::parse(&["Score".into(), "A".into(), "later".into()]) + .expect_err("\"later\" is not a seed"); + assert!(err.contains("is not a seed"), "{err}"); + } +} diff --git a/crates/improvise-command/src/command/cmd/mod.rs b/crates/improvise-command/src/command/cmd/mod.rs index 6bdd6d9f..0000d0fd 100644 --- a/crates/improvise-command/src/command/cmd/mod.rs +++ b/crates/improvise-command/src/command/cmd/mod.rs @@ -2,6 +2,7 @@ pub mod cell; pub mod commit; pub mod core; pub mod effect_cmds; +pub mod fit; pub mod grid; pub mod mode; pub mod navigation; diff --git a/crates/improvise-command/src/command/cmd/registry.rs b/crates/improvise-command/src/command/cmd/registry.rs index f0e9e05b..31fc13c1 100644 --- a/crates/improvise-command/src/command/cmd/registry.rs +++ b/crates/improvise-command/src/command/cmd/registry.rs @@ -15,6 +15,7 @@ use super::cell::*; use super::commit::*; use super::core::*; use super::effect_cmds::*; +use super::fit::FitMeasure; use super::grid::*; use super::mode::*; use super::navigation::*; @@ -54,6 +55,7 @@ pub fn default_registry() -> CmdRegistry { }, ); r.register_pure(&AddFormulaCmd(vec![]), AddFormulaCmd::parse); + r.register_pure(&FitMeasure::prototype(), FitMeasure::parse); r.register_pure(&RemoveFormulaCmd(vec![]), RemoveFormulaCmd::parse); r.register_pure(&CreateViewCmd(vec![]), CreateViewCmd::parse); r.register_pure(&DeleteViewCmd(vec![]), DeleteViewCmd::parse); diff --git a/crates/improvise-gp/src/dataset.rs b/crates/improvise-gp/src/dataset.rs index ceae60c3..f11cfb26 100644 --- a/crates/improvise-gp/src/dataset.rs +++ b/crates/improvise-gp/src/dataset.rs @@ -113,12 +113,23 @@ fn cell_text(value: &CellValue) -> String { } } -/// The coordinates every visible record shares, exactly as `GridLayout::new` -/// derives them: one selection per filtering category, defaulting to its first -/// item when the view names none. +/// The coordinates every visible record shares — one selection per filtering +/// category, defaulting to its first item when the view names none, exactly as +/// `GridLayout::new` derives them, **except** that `_Measure` never +/// constrains. +/// +/// That exception is not a special case, it is the difference between a grid +/// and a feature matrix. `_Measure` is the coordinate the *columns* are spread +/// across, so pinning it to one item is not "which records am I looking at" but +/// "throw away every column except this one" — and `Workbook::new`'s default +/// view puts it on `Axis::Page`, so that was the behaviour on a freshly opened +/// document. Either the target's own column survived and every feature came +/// back blank, or, on a document whose target is not `_Measure`'s first item, +/// the table was empty and the error blamed the data. fn filter_coords(model: &Model, view: &View) -> Vec<(String, String)> { view.filtering_cats() .into_iter() + .filter(|cat| !CategoryKind::of_name(cat).is_measure()) .filter_map(|cat| { let selected = view .page_selection(cat) @@ -395,6 +406,53 @@ mod tests { ); } + /// Regression (improvise-vx3): **`_Measure` is a filtering category and + /// must not filter a fit.** `Workbook::new`'s default view puts it on + /// `Axis::Page`, so `filtering_cats()` named it, `filter_coords` pinned it + /// to the first measure, and every "record" then carried exactly one + /// measure cell — never the target. The whole table came back as "no + /// visible record has a Revenue value to learn from" on a document plainly + /// full of them, which reads as a data problem rather than as an axis one. + /// + /// The rule that fixes it is not a special case: a fit's target *and* its + /// features are measures, spread across the `_Measure` coordinate, so + /// paging to one is not "which records am I looking at" but "throw away + /// every column". Page ∪ Filter still scopes the rows in every other + /// category, which is what the test below this one pins. + /// + /// Every test in this module used `View::new` + `on_category_added`, under + /// which a leading-underscore category lands on `Axis::None` — so the + /// fixture was quietly building the one world the code worked in, and no + /// test here could see the default a real `Workbook` hands out. + #[test] + fn a_measure_on_the_page_axis_does_not_filter_away_the_training_rows() { + let (model, mut view) = fixture(); + let unfiltered = table(&model, &view); + view.set_axis(MEASURE_CATEGORY, Axis::Page); + let paged = feature_table(&model, &view, "Revenue") + .expect("paging the measure axis is not a reason to have no data"); + // The *contents*, not the shape. Pinning only the header and the row + // count passes on this fixture even with the bug present: `_Measure`'s + // first item is `Revenue`, so every row still has a target and only + // the feature columns come back blank — the shape survives while the + // data does not, which is exactly the silent failure to guard against. + assert_eq!(paged, unfiltered); + } + + /// The other half, and the sharper one: which measure `filter_coords` + /// would have pinned to is the *first* item, so on a document whose target + /// is not first the table came back empty and the error blamed the data — + /// "no visible record has a Revenue value to learn from", on a document + /// plainly full of them. + #[test] + fn a_paged_measure_that_is_not_the_target_does_not_empty_the_table() { + let (model, mut view) = fixture(); + let unfiltered = table(&model, &view); + view.set_axis(MEASURE_CATEGORY, Axis::Page); + view.set_page_selection(MEASURE_CATEGORY, "Cost"); + assert_eq!(table(&model, &view), unfiltered); + } + /// gotcha 27: what is *visible* is Page ∪ Filter. Scoping by /// `categories_on(Axis::Page)` would fit against rows a drill has /// already excluded, and say nothing about it. diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs new file mode 100644 index 00000000..344c2670 --- /dev/null +++ b/crates/improvise-gp/src/fit.rs @@ -0,0 +1,616 @@ +//! One fit, end to end: a [`View`]'s records in, an improvise [`Formula`] and +//! the evidence for believing it out. +//! +//! [`dataset`] already turns what a view shows into the feature matrix a search +//! consumes, and [`crate::fitted_formula`] already turns a champion into +//! formula text the document can store. This module is the sequence between +//! them, and it exists here rather than in a command because none of it is +//! terminal-specific: a browser fitting the same measure over the same document +//! must run exactly these steps in exactly this order, or it gets a different +//! tree from the same seed. +//! +//! **Nothing here prints** (repo-map gotcha 37a). The quality numbers, the +//! importance ranking and `preprocess`'s dropped columns all come back as data +//! on [`FitOutcome`], and deciding they become a status line, a panel or a +//! stderr note is the host's business. [`FitOutcome::status_line`] is a +//! *rendering* of that data offered so a second front end does not compose its +//! own — the same argument that put `flatten_cat_tree` in the shared layer — +//! and it is a returned `String`, never a `println!`. +//! +//! # Three things the sequence has to get right +//! +//! **A fit is standardized and the formula must not be.** The crate's leaf +//! scores live in a ±5 range, so a target in the thousands cannot be reached by +//! leaf mutation and [`gp_tree::fit::standardize`] is not optional. Its +//! consequence is that the champion predicts *standard deviations*: `fit_csv` +//! answers that by printing `≈ score × std + mean` beside the tree, which a +//! document has nowhere to put. So the affine map is folded back into the +//! leaves with [`gp_tree::fit::denormalize`] before a single character of +//! formula text is spelled. Omitting it emits a perfectly well-formed formula +//! whose every cell is out by an affine map — wrong in the quiet way, since +//! small numbers in the right shape look like an answer. +//! +//! **The seed alone does not reproduce a fit** (gotcha 37b), so +//! [`FitOutcome`] carries the [`FitConfig`] beside it. Every stochastic step +//! draws from one generator in a fixed order — cross-validate, then fit, then +//! explain — so the pair really is sufficient. +//! +//! **Cross-validating nothing is not the same as explaining nothing.** +//! [`gp_tree::fit::cv_regress`] pools no pairs below two folds, and scoring an +//! empty pool gives an R² of exactly zero — which reads as "this rule is worth +//! nothing" when what happened is that nobody checked. [`Quality`] is an enum +//! so the two cannot be confused, and a host that wants the numbers has to name +//! the case where there are none. + +use gp_tree::data::{Overrides, Target}; +use gp_tree::fit::{ + FitConfig, RegressionQuality, cv_regress, denormalize, fit, importance_ranking, mse, + permutation_importance, regression_quality, standardize, +}; +use gp_tree::{Spec, SplitMix64}; +use improvise_core::model::Model; +use improvise_core::view::View; +use improvise_formula::Formula; + +use crate::dataset::dataset; + +/// What to ask of a fit. Everything a run's outcome depends on, so that +/// `(FitRequest, document)` reproduces it. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct FitRequest { + /// The seed for every stochastic step. + pub seed: u64, + /// The search's knobs — necessary beside the seed, not implied by it. + pub config: FitConfig, + /// Cross-validation folds. Below two, no held-out evidence is produced and + /// [`Quality::Unmeasured`] says so. + pub cv_folds: usize, + /// Shuffles per feature when measuring permutation importance. + pub importance_repeats: usize, +} + +impl Default for FitRequest { + /// Interactive defaults: bounded hard enough to run inside a keystroke. + /// + /// `FitConfig::default()` is the CLI's budget — 80 members over 600 + /// generations — which is the right answer for a batch tool and the wrong + /// one for a synchronous command, because a `Cmd` here runs on the thread + /// that draws the next frame. These numbers are a deliberate quality/ + /// latency trade, not a tuned optimum; a caller wanting the CLI's search + /// passes `FitConfig::default()`. + /// + /// **Measured**, release build, on a document of the shape this module's + /// tests use: **6ms** at 12 records, **12ms** at 200, **105ms** at 2,000 — + /// the whole sequence, five-fold CV included. So it is a pause rather than + /// a freeze at any document size in this repo's own benchmarks (18,000 + /// cells), and roughly linear in records beyond that. There is no timing + /// *test*: a wall-clock assertion is flaky on shared CI and would fail for + /// a reason that is not this code's. Re-measure after changing these. + fn default() -> Self { + Self { + seed: 0x0F17_C0FF_EE00_D00D, + config: FitConfig { + pop: 40, + generations: 120, + ..FitConfig::default() + }, + cv_folds: 5, + importance_repeats: 3, + } + } +} + +/// Held-out evidence, or the statement that there is none. +/// +/// The two are different things to tell a user and an `Option` would have made +/// them one: `Quality::Measured` carrying an all-zero [`RegressionQuality`] +/// means "this tree is no better than the mean", which is the loudest thing a +/// fit can say, and it must not be reachable by simply not having asked. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Quality { + /// Pooled `k`-fold cross-validation against the predict-the-mean baseline. + Measured { folds: usize, of: RegressionQuality }, + /// Fewer rows than folds asked for: nothing was held out, so nothing is + /// claimed. + Unmeasured { rows: usize }, +} + +/// A fitted rule and the evidence for it. +#[derive(Clone, Debug)] +pub struct FitOutcome { + /// The champion as an improvise formula, already parsed and already + /// checked against the document it is to be stored in. + pub formula: Formula, + /// How the rule did on data it was not fitted to. + pub quality: Quality, + /// `(feature name, percentage share)`, most important first, features the + /// rule does not use omitted. + pub importance: Vec<(String, f32)>, + /// Records and columns the selection dropped — `preprocess`'s notices and + /// the feature table's, in that order. + pub warnings: Vec, + /// The seed this fit ran under, and… + pub seed: u64, + /// …the knobs it ran under. Both, because neither alone reproduces it. + pub config: FitConfig, +} + +impl FitOutcome { + /// One line naming the rule's worth, what it leans on, and anything the + /// selection dropped — in that order, because a user who reads only the + /// first clause has read the part that decides whether to believe it. + pub fn status_line(&self) -> String { + let quality = match self.quality { + Quality::Measured { folds, of } => format!( + "R\u{b2} {:.2}, RMSE {:.3} vs baseline {:.3} ({folds}-fold CV)", + of.r2, + of.rmse, + of.baseline_mse.sqrt(), + ), + Quality::Unmeasured { rows } => { + format!("no held-out check \u{2014} only {rows} record(s)") + } + }; + let top: Vec = self + .importance + .iter() + .take(3) + .map(|(name, share)| format!("{name} {share:.0}%")) + .collect(); + let leans = if top.is_empty() { + "a constant \u{2014} no feature matters".to_string() + } else { + format!("leans on {}", top.join(", ")) + }; + let tail = if self.warnings.is_empty() { + String::new() + } else { + format!(" \u{2014} {}", self.warnings.join("; ")) + }; + format!( + "{} = fitted tree: {quality}; {leans}; seed {:#x}{tail}", + self.formula.target, self.seed + ) + } +} + +/// The measure a fit of `target` is stored under, unless a caller names +/// another: the fit is a *second, derived* column beside the data it explains. +/// +/// Storing it under `target` itself would make the formula shadow the very +/// cells it was fitted to — gotcha 7's `_Measure` items are explicit items ∪ +/// formula targets, so the two would be one item with a computed value, and +/// the user's typed data would stop being visible. A fit is a rule *about* the +/// data, never a replacement for it. +pub fn fitted_measure_name(target: &str) -> String { + format!("{target} (fit)") +} + +/// Fit `target` from everything else the `view` shows, storing the rule as the +/// measure `into`, and hand back the rule with the evidence for it. +/// +/// Fails, rather than storing something misleading, when: the selection cannot +/// produce a table at all (see [`crate::feature_table`]); `into` is a measure +/// that already holds data; the target is categorical, which a single +/// regression tree does not express; or the champion splits on a feature the +/// document cannot resolve (improvise-8xhy, and the message names them). +pub fn fit_measure( + model: &Model, + view: &View, + target: &str, + into: &str, + request: &FitRequest, +) -> anyhow::Result { + anyhow::ensure!( + into != target, + "a fit of {target:?} cannot be stored as {target:?} — the formula would shadow the \ + cells it was fitted to. Name a different measure" + ); + anyhow::ensure!( + !model.measure_item_names().iter().any(|m| m == into), + "{into:?} is already a measure; a formula on it would replace what is there. \ + Name a measure that does not exist yet" + ); + let data = dataset(model, view, target, &Overrides::default())?; + let rows = regression_rows(data.target, target)?; + + let names: Vec<&str> = data.feature_names.iter().map(String::as_str).collect(); + let spec = Spec { + ranges: &data.ranges, + names: &names, + }; + let mut rng = SplitMix64::new(request.seed); + + // Held-out evidence first, and from the same generator, so the sequence + // reproduces as a whole rather than step by step. + let folds = request.cv_folds.min(rows.len()); + let quality = match folds >= 2 { + true => Quality::Measured { + folds, + of: regression_quality(&cv_regress(&rows, &spec, &request.config, folds, &mut rng)), + }, + false => Quality::Unmeasured { rows: rows.len() }, + }; + + // Then the rule itself, over every row. Standardized to fit, de-normalized + // to be written down — see this module's header. + let (normalized, mean, std) = standardize(&rows); + let champion = denormalize( + &fit(&normalized, &spec, &request.config, &mut rng), + mean, + std, + ); + + let importance = permutation_importance( + |shuffled| mse(&champion, shuffled), + &rows, + names.len(), + request.importance_repeats, + &mut rng, + ); + + Ok(FitOutcome { + formula: crate::fitted_formula( + &champion, + &spec, + into, + improvise_core::model::category::MEASURE_CATEGORY, + model, + )?, + quality, + importance: importance_ranking(&importance) + .into_iter() + .map(|(j, share)| (data.feature_names[j].clone(), share)) + .collect(), + warnings: data.warnings, + seed: request.seed, + config: request.config, + }) +} + +/// The regression rows, or a refusal naming why a categorical target is not a +/// formula. +/// +/// `preprocess` infers classification the moment a target cell does not parse +/// as a number, and a classification fit is one tree *per class* plus an +/// `argmax` — three things improvise formulas have none of. Refusing here is +/// the same choice `fitted_formula` makes for an unresolvable feature: fail +/// where the formula would have been written, rather than store something that +/// looks like an answer. +fn regression_rows( + target_data: Target, + target: &str, +) -> anyhow::Result>> { + match target_data { + Target::Regress(rows) => Ok(rows), + Target::Classify { labels, .. } => anyhow::bail!( + "{target:?} holds text, so a fit over it is a classification into {labels:?} — \ + that is one tree per class chosen by argmax, which improvise formulas cannot \ + express. Fit a numeric measure instead" + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use improvise_core::model::category::CategoryKind; + use improvise_core::model::cell::{CellKey, CellValue}; + use improvise_core::view::Axis; + + const MEASURE: &str = "_Measure"; + const REGION: &str = "Region"; + const TAG: &str = "Tag"; + const FIT_INTO: &str = "Score (fit)"; + + /// A document with one regular category, one feature measure and one + /// target measure standing in a rule a tree can express exactly: `Score` + /// steps at `Cost` 50, and the numbers are in the **thousands** so a fit + /// that forgets to de-normalize is visibly wrong rather than subtly so. + /// + /// `Region` carries a single item on purpose. `preprocess` one-hots a + /// categorical column, and a one-hot that is constant across every record + /// has the degenerate range `(1, 1)`, which `simplify` prunes — so the + /// champion *cannot* split on an unresolvable feature and the success path + /// is deterministic rather than "whatever this seed happened to pick". + /// `Tag` is what makes the twelve records distinct, and it is a `Label` + /// category so `dataset` excludes it from the features (it would otherwise + /// one-hot into one column per row: a perfect memoriser). + fn document() -> (Model, View) { + let mut model = Model::new("sales"); + for cat in [REGION, TAG] { + model.add_category(cat).expect("under the category limit"); + } + model + .category_mut(REGION) + .expect("just added") + .add_item("North"); + model.category_mut(TAG).expect("just added").kind = CategoryKind::Label; + for i in 0..12u32 { + model + .category_mut(TAG) + .expect("just added") + .add_item(format!("r{i}")); + } + let measure = model.category_mut(MEASURE).expect("_Measure always exists"); + for item in ["Cost", "Score"] { + measure.add_item(item); + } + for i in 0..12u32 { + let cost = 10.0 * f64::from(i); + model.set_cell(cell(&format!("r{i}"), "Cost"), CellValue::Number(cost)); + model.set_cell( + cell(&format!("r{i}"), "Score"), + CellValue::Number(if cost > 50.0 { 4000.0 } else { 1000.0 }), + ); + } + let mut view = View::new("Default"); + for cat in model.category_names() { + view.on_category_added(cat); + } + view.set_axis(REGION, Axis::Row); + (model, view) + } + + fn cell(tag: &str, measure: &str) -> CellKey { + CellKey::new(vec![ + (REGION.to_string(), "North".to_string()), + (TAG.to_string(), tag.to_string()), + (MEASURE.to_string(), measure.to_string()), + ]) + } + + fn request() -> FitRequest { + FitRequest { + config: FitConfig { + pop: 24, + generations: 40, + ..FitRequest::default().config + }, + ..FitRequest::default() + } + } + + /// The whole point, asserted against a real document: the fit's formula + /// goes in through the ordinary `add_formula` path and every cell of the + /// new measure evaluates to a number — not to `Error("ref:…")`, which is + /// what improvise-8xhy was. + #[test] + fn a_fitted_formula_evaluates_in_the_document_it_was_fitted_on() { + let (mut model, view) = document(); + let outcome = + fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands"); + model.add_formula(outcome.formula.clone()); + model.recompute_formulas(&[]); + for i in 0..12u32 { + match model.evaluate(&cell(&format!("r{i}"), &outcome.formula.target)) { + Some(CellValue::Number(_)) => {} + other => panic!("record r{i} came back as {other:?}"), + } + } + } + + /// The de-normalization regression, and the reason it needs a target in + /// the thousands. `fit` runs against a standardized target, so a champion + /// written down untouched predicts standard deviations: a well-formed + /// formula, parsing cleanly, evaluating to ±5 where the user's data is + /// 1000–4000. Nothing downstream can see that — it is a number in a cell. + /// + /// Injecting the fault (dropping the `denormalize` call in `fit_measure`) + /// fails this and only this test. + #[test] + fn the_fitted_formula_predicts_in_the_documents_own_units() { + let (mut model, view) = document(); + let outcome = + fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands"); + model.add_formula(outcome.formula.clone()); + model.recompute_formulas(&[]); + for i in 0..12u32 { + let Some(CellValue::Number(n)) = model.evaluate(&cell(&format!("r{i}"), FIT_INTO)) + else { + continue; + }; + assert!( + (500.0..=5000.0).contains(&n), + "record r{i} predicted {n}, which is standardized units, not the \ + document's 1000–4000" + ); + } + // …and the same claim read off the formula text, so the assertion does + // not depend on the target item having been resolvable above. + let numbers = leaf_numbers(&outcome.formula.expr); + assert!(!numbers.is_empty(), "the fit emitted no leaf at all"); + assert!( + numbers.iter().any(|n| *n > 500.0), + "every leaf is {numbers:?} — standardized units, not the document's 1000–4000" + ); + } + + /// Every `Number` literal in the emitted expression, leaves and thresholds + /// alike. A threshold here is a `Cost` (0–110) and a leaf is a `Score` + /// (1000–4000), so "some literal exceeds 500" can only be a leaf. + fn leaf_numbers(expr: &improvise_formula::Expr) -> Vec { + use improvise_formula::Expr; + match expr { + Expr::Number(n) => vec![*n], + Expr::If(c, t, e) => [c, t, e].iter().flat_map(|e| leaf_numbers(e)).collect(), + Expr::BinOp(_, l, r) => [l, r].iter().flat_map(|e| leaf_numbers(e)).collect(), + Expr::UnaryMinus(e) => leaf_numbers(e), + Expr::Ref(_) | Expr::Agg(..) => Vec::new(), + } + } + + /// gotcha 37b: the seed alone does not reproduce a fit, so the outcome + /// carries the config beside it — and the pair really is sufficient. + #[test] + fn the_same_seed_and_config_reproduce_the_same_rule() { + let (model, view) = document(); + let a = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); + let b = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); + assert_eq!(a.formula.raw, b.formula.raw); + assert_eq!(a.seed, b.seed); + assert_eq!(a.config, b.config); + } + + /// And the other half: a different *config* under the same seed searches + /// differently, which is why recording the seed alone would not do. + #[test] + fn the_same_seed_under_a_different_config_is_a_different_search() { + let (model, view) = document(); + let base = request(); + let wider = FitRequest { + config: FitConfig { + generations: base.config.generations * 3, + ..base.config + }, + ..base + }; + assert_eq!(base.seed, wider.seed); + assert_ne!( + fit_measure(&model, &view, "Score", FIT_INTO, &base) + .expect("fits") + .formula + .raw, + fit_measure(&model, &view, "Score", FIT_INTO, &wider) + .expect("fits") + .formula + .raw, + "the same seed gave the same rule under different knobs, so recording \ + the seed alone would have been enough after all" + ); + } + + /// A fit is a rule *about* the data, never a replacement for it. Stored + /// under its own target the formula would shadow the very cells it was + /// fitted to (gotcha 7: `_Measure` items are explicit items ∪ formula + /// targets, so the two would be one item), and the user's typed numbers + /// would simply stop being visible. + #[test] + fn a_fit_stored_under_its_own_target_is_refused() { + let (model, view) = document(); + let err = + fit_measure(&model, &view, "Score", "Score", &request()).expect_err("self-shadowing"); + assert!(err.to_string().contains("shadow"), "{err}"); + } + + /// The same hazard one step removed: any existing measure would be + /// replaced by the rule, so the name has to be a new one. + #[test] + fn a_fit_stored_over_an_existing_measure_is_refused() { + let (model, view) = document(); + let err = + fit_measure(&model, &view, "Score", "Cost", &request()).expect_err("Cost holds data"); + assert!(err.to_string().contains("already a measure"), "{err}"); + } + + /// improvise-8xhy reaching a real user, and the reason improvise-4zl7 + /// exists: with a categorical feature carrying the signal, the champion + /// splits on a one-hot — `Region=North`, an encoding artefact no document + /// has an item for — and the whole fit is refused *by name* rather than + /// written in as a measure whose every cell is `Error("ref:…")`. + /// + /// Note what this costs in practice: which features a champion picks is a + /// property of the search, so on a document with several categories the + /// same command refuses or succeeds depending on the seed. + #[test] + fn a_fit_that_needs_a_category_is_refused_naming_the_feature() { + let (mut model, view) = document(); + // Region is the only thing left that can explain Score. + model + .category_mut(REGION) + .expect("Region exists") + .add_item("South"); + for i in 0..12u32 { + model.clear_cell(&cell(&format!("r{i}"), "Cost")); + } + for (i, region) in (0..12u32).map(|i| (i, if i < 6 { "North" } else { "South" })) { + model.set_cell( + CellKey::new(vec![ + (REGION.to_string(), region.to_string()), + (TAG.to_string(), format!("r{i}")), + (MEASURE.to_string(), "Score".to_string()), + ]), + CellValue::Number(if region == "North" { 1000.0 } else { 4000.0 }), + ); + } + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err("the champion can only split on a one-hot"); + let msg = err.to_string(); + assert!( + msg.contains("Region=North") || msg.contains("Region=South"), + "{msg}" + ); + assert!( + msg.contains("not an item") || msg.contains("not items"), + "{msg}" + ); + // The refusal is read off a one-line status bar, so it has to *be* one + // line. A `rustfmt` pass over the multi-line literal that spells it had + // left runs of ten spaces mid-sentence — invisible in the source and in + // every test that only asked whether a substring was present, and + // visible to precisely the user this message exists for. + assert!(!msg.contains('\n'), "{msg:?}"); + assert!(!msg.contains(" "), "{msg:?}"); + } + + /// A text-valued target is a *classification*, which is k trees and an + /// argmax — not something an improvise formula can hold. Refused by name, + /// at the point the formula would have been written. + #[test] + fn a_text_target_is_refused_as_a_classification() { + let (mut model, view) = document(); + for i in 0..12u32 { + model.set_cell( + cell(&format!("r{i}"), "Score"), + CellValue::Text(if i % 2 == 0 { "up" } else { "down" }.into()), + ); + } + let err = + fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect_err("text target"); + assert!(err.to_string().contains("classification"), "{err}"); + } + + /// Cross-validating nothing is not "explained nothing". With fewer records + /// than folds asked for, the outcome says so rather than reporting an R² + /// of zero that a user would read as a verdict on the rule. + #[test] + fn too_few_records_to_cross_validate_says_so_rather_than_scoring_zero() { + let (mut model, view) = document(); + for i in 1..12u32 { + for measure in ["Cost", "Score"] { + model.clear_cell(&cell(&format!("r{i}"), measure)); + } + } + let outcome = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); + assert!( + matches!(outcome.quality, Quality::Unmeasured { rows: 1 }), + "{:?}", + outcome.quality + ); + assert!( + outcome.status_line().contains("no held-out check"), + "{}", + outcome.status_line() + ); + } + + /// The status line is one line and leads with the verdict, because a user + /// who reads only its first clause has read the part that decides whether + /// to believe the rule at all. + #[test] + fn the_status_line_leads_with_the_verdict_and_names_the_seed() { + let (model, view) = document(); + let outcome = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); + let line = outcome.status_line(); + assert!(!line.contains('\n'), "{line}"); + assert!( + line.starts_with(&format!("{FIT_INTO} = fitted tree: R")), + "{line}" + ); + assert!(line.contains("baseline"), "{line}"); + assert!( + line.contains(&format!("{:#x}", outcome.seed)), + "the seed is what reproduces the fit: {line}" + ); + } +} diff --git a/crates/improvise-gp/src/lib.rs b/crates/improvise-gp/src/lib.rs index c3886eaa..c3069ab8 100644 --- a/crates/improvise-gp/src/lib.rs +++ b/crates/improvise-gp/src/lib.rs @@ -84,8 +84,10 @@ //! it looks and what it would and would not cover. pub mod dataset; +pub mod fit; pub use dataset::{FeatureTable, dataset, feature_table}; +pub use fit::{FitOutcome, FitRequest, Quality, fit_measure}; use gp_tree::{Spec, Tree}; use improvise_core::model::Model; @@ -153,7 +155,13 @@ pub fn fitted_formula( let unresolvable = unresolvable_features(tree, spec, model); anyhow::ensure!( unresolvable.is_empty(), - "the fitted tree splits on {unresolvable:?}, which {} not {} of this document — a formula refers to its inputs by name, so every cell of {target:?} would be a reference error. Encoding artefacts (`Cat=Item` one-hots, `col.missing` flags, date parts) cannot be written as a formula: fit over numeric measures for now", + // One line, because it is read off a status bar. A `rustfmt` pass over + // the previous multi-line literal left runs of ten spaces mid-sentence + // in the message a user actually sees. + "the fitted tree splits on {unresolvable:?}, which {} not {} of this document \ + — a formula refers to its inputs by name, so every cell of {target:?} would be \ + a reference error. Encoding artefacts (`Cat=Item` one-hots, `col.missing` flags, \ + date parts) cannot be written as a formula: fit over numeric measures for now", if unresolvable.len() == 1 { "is" } else { "are" }, if unresolvable.len() == 1 { "an item" From f129305e3140438164b8ba0c98bf22fffdd7eb09 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:45:42 -0700 Subject: [PATCH 30/72] test(wasm-client): cover the new save prompt, and correct a false comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adding `AppMode::SavePrompt` extended nothing in the browser client, because `keys.rs::text_entry_modes` is a hand-written list and the new mode was simply absent from it. Adding the line and watching the two tests that consume it — typing resolves locally, backspace pops locally — pass unchanged confirms the shared keymap already carries the mode into the browser with no client-side change, which is the point of the keymap move. The comment on that helper claimed a new mode "shows up here as a missing line rather than as untested behaviour". It does not; nothing checks the list against `AppMode`, and `save_prompt` was silently uncovered until added by hand. There is a second hand-written copy in improvise-command's `text_entry_modes_with_exits`. The comment now says what is true, and deriving both from one place is reported for filing rather than done here — it is outside this unit's boundary. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-wasm-client/src/keys.rs | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/crates/improvise-wasm-client/src/keys.rs b/crates/improvise-wasm-client/src/keys.rs index 371ab885..f3333795 100644 --- a/crates/improvise-wasm-client/src/keys.rs +++ b/crates/improvise-wasm-client/src/keys.rs @@ -726,8 +726,16 @@ mod tests { KeyEvent::new(code, KeyModifiers::NONE) } - /// Every text-entry mode, built by its own constructor so a new one shows - /// up here as a missing line rather than as untested behaviour. + /// Every text-entry mode, built by its own constructor. + /// + /// The comment here used to claim a new mode "shows up as a missing line + /// rather than as untested behaviour". It does not: this is a hand-written + /// list and nothing checks it against `AppMode`, so a new text-entry mode + /// is simply absent and silently uncovered — which is what happened to + /// `save_prompt` (improvise-lae) until it was added by hand. There is a + /// second hand-written copy in `improvise-command`'s + /// `keymap::tests::text_entry_modes_with_exits`. Deriving both from one + /// place is filed, not done here. fn text_entry_modes() -> Vec { vec![ AppMode::editing(), @@ -737,6 +745,7 @@ mod tests { AppMode::category_add(), AppMode::item_add("Region".into()), AppMode::export_prompt(), + AppMode::save_prompt(), ] } From 29962b113faf5c1d2d0acc00c1fd39e7fe14610c Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:46:19 -0700 Subject: [PATCH 31/72] test(harness): name the root check for what it asserts, and lint the crate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `open_and_refresh_see_an_empty_origin_root` claimed more than it checked: since the previous commit an empty *entry* is not a document, so `list` being empty means "no documents", not "no entries". A check whose name overstates it is the shape this repo keeps finding — a green tick against a claim nobody verified. Renamed, and `wipe`'s doc now says out loud that it cannot clear a zero-byte entry, and why that costs nothing. `harness/opfs-contract` is not a workspace member, so the `test` job's workspace-wide `cargo fmt --check` and clippy never reach it. The browser job now runs both against it — it was, in fact, unformatted. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 7 ++ harness/opfs-contract/src/lib.rs | 131 +++++++++++++++++++------------ harness/tests/opfs.spec.js | 2 +- 3 files changed, 87 insertions(+), 53 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b926ac7..e2131409 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -107,6 +107,13 @@ jobs: curl -sSfL "$url" | tar -xz --strip-components=1 -C /usr/local/bin \ "wasm-pack-${WASM_PACK_VERSION}-x86_64-unknown-linux-musl/wasm-pack" wasm-pack --version + # Not a workspace member (it only builds for wasm32), so the `test` job's + # workspace-wide fmt and clippy do not reach it. + - name: Lint the harness crate + working-directory: harness/opfs-contract + run: | + cargo fmt --all -- --check + cargo clippy --target wasm32-unknown-unknown --all-targets -- -D warnings - name: Install the harness working-directory: harness run: npm ci diff --git a/harness/opfs-contract/src/lib.rs b/harness/opfs-contract/src/lib.rs index ea49d613..542e338e 100644 --- a/harness/opfs-contract/src/lib.rs +++ b/harness/opfs-contract/src/lib.rs @@ -62,12 +62,18 @@ impl Report { } } -/// Remove every entry in the origin root, so a run does not inherit the last -/// one's leftovers. +/// Remove every *document* in the origin root, so a run does not inherit the +/// last one's leftovers. /// /// Uses only the backend's own vocabulary — `refresh`, `delete`, /// `commit_deletes` — rather than reaching for `removeEntry` directly, so a /// broken `delete` shows up here as well as in the contract. +/// +/// It therefore cannot clear a zero-byte entry, because `list` no longer +/// names one. That is deliberate and costs nothing: `attach` recreates and +/// `write` fills whatever a previous run left behind, and a leftover empty +/// entry is invisible to every assertion here for the same reason it is +/// invisible to a document picker. async fn wipe(fs: &OpfsFs) -> Result { fs.refresh().await?; let names = fs.list(Path::new(""))?; @@ -99,12 +105,15 @@ pub async fn run_checks() -> Result { .map_err(|e| JsError::new(&format!("wipe: {e:#}")))?; report - .run("open_and_refresh_see_an_empty_origin_root", async || { - fs.refresh().await?; - let listed = fs.list(Path::new(""))?; - ensure!(listed.is_empty(), "a wiped root listed {listed:?}"); - Ok("empty".into()) - }) + .run( + "open_and_refresh_see_an_origin_root_holding_no_documents", + async || { + fs.refresh().await?; + let listed = fs.list(Path::new(""))?; + ensure!(listed.is_empty(), "a wiped root listed {listed:?}"); + Ok("empty".into()) + }, + ) .await; // `attach` must create the file to obtain a sync access handle for a @@ -114,28 +123,37 @@ pub async fn run_checks() -> Result { // the improvise-9x6 browser failure (the seed was skipped as "already // there" and `load_from` got an empty buffer). report - .run("an_attached_but_unwritten_entry_is_not_a_document", async || { - fs.attach(Path::new("attached.improv")).await?; - let listed = fs.list(Path::new(""))?; - ensure!(listed.is_empty(), "attach listed {listed:?} before anything was written"); + .run( + "an_attached_but_unwritten_entry_is_not_a_document", + async || { + fs.attach(Path::new("attached.improv")).await?; + let listed = fs.list(Path::new(""))?; + ensure!( + listed.is_empty(), + "attach listed {listed:?} before anything was written" + ); - // ...and a second instance, which shares no index, must not see - // it either: the filter has to be in `refresh`, not only in the - // omission from `attach`. - let other = OpfsFs::open().await?; - other.refresh().await?; - let seen = other.list(Path::new(""))?; - ensure!(seen.is_empty(), "refresh() listed the empty entry: {seen:?}"); + // ...and a second instance, which shares no index, must not see + // it either: the filter has to be in `refresh`, not only in the + // omission from `attach`. + let other = OpfsFs::open().await?; + other.refresh().await?; + let seen = other.list(Path::new(""))?; + ensure!( + seen.is_empty(), + "refresh() listed the empty entry: {seen:?}" + ); - // The handle is real, though: writing through it works, and then - // it IS a document. - fs.write(Path::new("attached.improv"), b"now it is one")?; - ensure!( - fs.list(Path::new(""))? == vec![std::path::PathBuf::from("attached.improv")], - "a written document was not listed" - ); - Ok("created, unlisted until written".into()) - }) + // The handle is real, though: writing through it works, and then + // it IS a document. + fs.write(Path::new("attached.improv"), b"now it is one")?; + ensure!( + fs.list(Path::new(""))? == vec![std::path::PathBuf::from("attached.improv")], + "a written document was not listed" + ); + Ok("created, unlisted until written".into()) + }, + ) .await; // The hand-rolled `keys()` async-iterator walk in `refresh` is the least @@ -168,7 +186,10 @@ pub async fn run_checks() -> Result { let p = Path::new("offsets.improv"); fs.attach(p).await?; fs.write(p, b"0123456789ABCDEF")?; - ensure!(fs.read(p)? == b"0123456789ABCDEF", "long write did not round-trip"); + ensure!( + fs.read(p)? == b"0123456789ABCDEF", + "long write did not round-trip" + ); fs.write(p, b"xy")?; let short = fs.read(p)?; ensure!( @@ -200,32 +221,38 @@ pub async fn run_checks() -> Result { // OPFS at the end of the batch. Both halves, because either one alone // would pass against a backend that got the timing exactly backwards. report - .run("a_delete_is_visible_at_once_and_lands_on_commit", async || { - wipe(&fs).await?; - let p = Path::new("doomed.improv"); - fs.attach(p).await?; - fs.write(p, b"bytes")?; + .run( + "a_delete_is_visible_at_once_and_lands_on_commit", + async || { + wipe(&fs).await?; + let p = Path::new("doomed.improv"); + fs.attach(p).await?; + fs.write(p, b"bytes")?; - fs.delete(p)?; - ensure!(fs.read(p).is_err(), "a deleted document was still readable"); - ensure!(fs.list(Path::new(""))?.is_empty(), "a deleted document was still listed"); + fs.delete(p)?; + ensure!(fs.read(p).is_err(), "a deleted document was still readable"); + ensure!( + fs.list(Path::new(""))?.is_empty(), + "a deleted document was still listed" + ); - let before = OpfsFs::open().await?; - before.refresh().await?; - ensure!( - before.list(Path::new(""))?.iter().any(|q| q == p), - "the entry left OPFS before commit_deletes — the queue is not a queue" - ); + let before = OpfsFs::open().await?; + before.refresh().await?; + ensure!( + before.list(Path::new(""))?.iter().any(|q| q == p), + "the entry left OPFS before commit_deletes — the queue is not a queue" + ); - fs.commit_deletes().await?; - let after = OpfsFs::open().await?; - after.refresh().await?; - ensure!( - !after.list(Path::new(""))?.iter().any(|q| q == p), - "commit_deletes did not remove the entry" - ); - Ok("queued then removed".into()) - }) + fs.commit_deletes().await?; + let after = OpfsFs::open().await?; + after.refresh().await?; + ensure!( + !after.list(Path::new(""))?.iter().any(|q| q == p), + "commit_deletes did not remove the entry" + ); + Ok("queued then removed".into()) + }, + ) .await; Ok(report.finish()) diff --git a/harness/tests/opfs.spec.js b/harness/tests/opfs.spec.js index f8506931..f5719bab 100644 --- a/harness/tests/opfs.spec.js +++ b/harness/tests/opfs.spec.js @@ -21,7 +21,7 @@ async function run(page, cmd) { * Rust on purpose: a check deleted on one side has to be deleted on both, * which is what stops the battery quietly shrinking. */ const EXPECTED = [ - "open_and_refresh_see_an_empty_origin_root", + "open_and_refresh_see_an_origin_root_holding_no_documents", "an_attached_but_unwritten_entry_is_not_a_document", "refresh_walks_every_key_and_sorts_them", "read_and_write_are_absolute_at_offset_zero", From f0f400d29ac01ee212253eeb2fd90f5e85eed0a1 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:46:53 -0700 Subject: [PATCH 32/72] =?UTF-8?q?fix(effect):=20SaveAs=20must=20abort=20th?= =?UTF-8?q?e=20batch=20too=20=E2=80=94=20the=20other=20arm=20of=20improvis?= =?UTF-8?q?e-8tk?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found reviewing improvise-lae. improvise-8tk fixed one arm of a rule that has two: a failed `Save` sets `abort_effects` so the `Quit` chained after it never lands, but `SaveAs` only wrote "Save error:" to the status bar and let the batch continue. Three tests guard the `Save` arm and none guarded this one — `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)`, the same shape that doctrine was written for. It is reachable today without any of this branch's work: the `:` line composes on ` . ` and `ExecuteCommand` concatenates the commands' effects into one batch, so `:w /bad/path . q` quits and discards the model on a failed write. That is improvise-8tk's silent data loss on the other path. It also matters more now, because improvise-lae's save prompt commits to `SaveAs`. With the abort, a failed write skips both the `ChangeMode` `CommitSave` emits and the `clear-buffer` the keymap welds after it, so the user stays in the prompt with the path they typed still there to correct, and the error on screen. Without it they were dropped to Normal with an empty buffer. Evidence: `failed_save_as_aborts_the_chain` was watched failing on the old code; `successful_save_as_lets_the_chain_continue` is the positive arm, without which "always abort" would pass and every `:w ` would swallow what followed it — it passed before the change, so it is not circular. `ctrl_w_prompts_for_a_path_and_a_failed_write_keeps_the_prompt_open` drives the whole prompt through real keypresses. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-command/src/ui/app.rs | 50 +++++++++++++++++ crates/improvise-command/src/ui/effect.rs | 66 +++++++++++++++++++++++ 2 files changed, 116 insertions(+) diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 8b85c7b5..70e8467b 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1615,6 +1615,56 @@ mod tests { ); } + /// improvise-lae end to end: `Ctrl-W`, a path, `Enter`. + /// + /// The failing case is the interesting one, and it is what `SaveAs` + /// gaining `Save`'s abort buys the prompt. `CommitSave` emits + /// `[SaveAs, ChangeMode(Normal)]` and the keymap welds a trailing + /// `clear-buffer`, so an aborting `SaveAs` skips both: the user is left + /// in the prompt, with the path they typed still there to correct, and + /// the error on screen. Without the abort they would have been dropped + /// back to Normal with an error and an empty buffer. + #[test] + fn ctrl_w_prompts_for_a_path_and_a_failed_write_keeps_the_prompt_open() { + let mut app = App::new(Workbook::new("T"), None); + app.handle_key(KeyEvent::new(KeyCode::Char('w'), KeyModifiers::CONTROL)) + .unwrap(); + assert!( + matches!(app.view_state.mode, AppMode::SavePrompt { .. }), + "Ctrl-W should open the save prompt, got {:?}", + app.view_state.mode + ); + + let path = "/nonexistent-dir-improvise-lae/m.improv"; + for c in path.chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + assert_eq!( + app.view_state.buffers.get("save").map(String::as_str), + Some(path), + "typing must reach the buffer the prompt names" + ); + + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + assert!( + app.view_state.status_msg.starts_with("Save error:"), + "got {:?}", + app.view_state.status_msg + ); + assert!( + matches!(app.view_state.mode, AppMode::SavePrompt { .. }), + "a failed write must leave the prompt open, got {:?}", + app.view_state.mode + ); + assert_eq!( + app.view_state.buffers.get("save").map(String::as_str), + Some(path), + "the path the user typed must survive for them to correct" + ); + } + /// `ZZ` is save-and-quit. With no file path the save fails and aborts the /// chain (improvise-8tk), which also proves the second Z reached `wq`. #[test] diff --git a/crates/improvise-command/src/ui/effect.rs b/crates/improvise-command/src/ui/effect.rs index 73743287..50e44a86 100644 --- a/crates/improvise-command/src/ui/effect.rs +++ b/crates/improvise-command/src/ui/effect.rs @@ -788,8 +788,12 @@ impl Effect for SaveAs { app.model_state.dirty = false; app.view_state.status_msg = format!("Saved to {}", self.0.display()); } + // The same abort `Save` owes (improvise-8tk). `SaveAs` is the + // other arm of that rule and was left out of the original fix, + // so `:w /bad/path . q` quit on a failed write. Err(e) => { app.view_state.status_msg = format!("Save error: {e}"); + app.abort_effects = true; } } } @@ -1961,6 +1965,68 @@ mod tests { ); } + /// BUG (improvise-lae, found reviewing it): improvise-8tk fixed **one + /// arm** of a rule with two. `Save` aborts the batch when the write + /// fails; `SaveAs` reported the error to the status bar and let the + /// batch continue, which is the same silent discard 8tk closed — the + /// shape `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` + /// names, and the three tests above are evidence about `Save` only. + /// + /// It is reachable today: the `:` line composes with ` . `, so + /// `:w /bad/path . q` builds exactly this batch, and + /// `ExecuteCommand` concatenates the two commands' effects into one. + /// improvise-lae's new save prompt also commits to `SaveAs`, so the + /// arm was about to carry more traffic. + #[test] + fn failed_save_as_aborts_the_chain() { + let mut app = test_app(); + app.model_state.dirty = true; + app.apply_effects(vec![ + Box::new(SaveAs(PathBuf::from( + "/nonexistent-dir-improvise-lae/model.improv", + ))), + Box::new(ChangeMode(AppMode::Quit)), + ]); + assert_ne!( + app.view_state.mode, + AppMode::Quit, + "a failed save-as must not let the trailing Quit apply" + ); + assert!( + app.model_state.dirty, + "a failed save-as must leave the model dirty" + ); + assert!( + app.view_state.status_msg.starts_with("Save error:"), + "expected the save error to stay on screen, got: {}", + app.view_state.status_msg + ); + } + + /// The positive arm: a `SaveAs` that succeeds must not abort, or every + /// `:w ` would swallow whatever followed it. Without this, + /// "always abort" passes the test above. + #[test] + fn successful_save_as_lets_the_chain_continue() { + let dir = std::env::temp_dir().join("improvise-lae-save-as-ok"); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("model.improv"); + let mut app = test_app(); + app.model_state.dirty = true; + app.apply_effects(vec![ + Box::new(SaveAs(path.clone())), + Box::new(ChangeMode(AppMode::Quit)), + ]); + assert_eq!(app.view_state.mode, AppMode::Quit); + assert!(!app.model_state.dirty); + assert_eq!( + app.model_state.file_path.as_deref(), + Some(path.as_path()), + "a successful save-as adopts the path" + ); + let _ = std::fs::remove_file(&path); + } + /// A successful save must not abort — `wq` still quits. #[test] fn successful_save_lets_the_chain_continue() { From b929765d0d2759c8bfb6769cf784d9c8aa02755d Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:47:33 -0700 Subject: [PATCH 33/72] =?UTF-8?q?docs:=20say=20what=20the=20abort=20does?= =?UTF-8?q?=20not=20buy=20=E2=80=94=20the=20error=20is=20recorded,=20not?= =?UTF-8?q?=20drawn?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of the previous commit. Two doc comments claimed the failed save leaves "the error on screen". It does not: `draw_bottom_bar` renders the minibuffer *instead of* the status line, so any `status_msg` written while a prompt is open is recorded and never drawn. That is pre-existing and wider than this branch — a failed `commit-buffer category` in `CategoryAdd` is invisible for the same reason, and improvise-6t3 deliberately ordered its effects so that failure message would win over the optimistic one. Reported for filing rather than fixed: the bottom bar is one line, and giving it both the prompt and a status is a UI decision, not a bug fix. The data-loss half of the previous commit stands unchanged. What is corrected here is only the claim about what the user sees — a false statement in a doc comment is the thing gotcha 48 is about. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-command/src/ui/app.rs | 15 ++++++++++++--- crates/improvise-command/src/ui/effect.rs | 6 ++++++ 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 70e8467b..8290d9a5 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1621,9 +1621,18 @@ mod tests { /// gaining `Save`'s abort buys the prompt. `CommitSave` emits /// `[SaveAs, ChangeMode(Normal)]` and the keymap welds a trailing /// `clear-buffer`, so an aborting `SaveAs` skips both: the user is left - /// in the prompt, with the path they typed still there to correct, and - /// the error on screen. Without the abort they would have been dropped - /// back to Normal with an error and an empty buffer. + /// in the prompt, with the path they typed still there to correct. + /// Without the abort they would have been dropped back to Normal with an + /// empty buffer. + /// + /// One caveat this test can see and the TUI cannot: `draw_bottom_bar` + /// renders the minibuffer *instead of* the status line, so the error is + /// recorded in `status_msg` and not drawn while the prompt is open. That + /// is pre-existing and wider than this prompt — a failed + /// `commit-buffer category` in `CategoryAdd`, which improvise-6t3 + /// deliberately ordered so its failure message would win, is invisible + /// for the same reason. Reported, not fixed here: the bottom bar is one + /// line and giving it both is a UI decision. #[test] fn ctrl_w_prompts_for_a_path_and_a_failed_write_keeps_the_prompt_open() { let mut app = App::new(Workbook::new("T"), None); diff --git a/crates/improvise-command/src/ui/effect.rs b/crates/improvise-command/src/ui/effect.rs index 50e44a86..357d123f 100644 --- a/crates/improvise-command/src/ui/effect.rs +++ b/crates/improvise-command/src/ui/effect.rs @@ -1977,6 +1977,12 @@ mod tests { /// `ExecuteCommand` concatenates the two commands' effects into one. /// improvise-lae's new save prompt also commits to `SaveAs`, so the /// arm was about to carry more traffic. + /// + /// Note what the abort does and does not buy the prompt: the typed path + /// survives for the user to correct, but the message does not reach the + /// screen, because `draw_bottom_bar` draws the minibuffer instead of the + /// status line while a prompt is open. That is pre-existing and reported + /// separately; the data-loss half is what this fixes. #[test] fn failed_save_as_aborts_the_chain() { let mut app = test_app(); From 8ed32dd98f1de5d43bde839e3e80794c283a182a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:48:00 -0700 Subject: [PATCH 34/72] fix(opfs): an entry refresh cannot measure counts as a document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by reading the change in the previous commit rather than by running it, which is the honest provenance: the harness runs one tab. `refresh` now asks each entry's size so a zero-byte one is not offered as a document, and it reads that size through `getFile()` for any entry this instance has not attached. A file under an exclusive sync access handle is not reliably readable by the promise-based API — so **a second tab of the same origin**, holding a handle on a document this tab has not attached, could make `getFile()` throw and take the whole `refresh` down with it. A second tab breaking the first one's document picker is a worse bug than the one being fixed, and it is one no single-tab harness can see. So an entry whose size cannot be read counts as a document. It is the only answer that is wrong in the cheap direction: an entry somebody else has open is certainly a document, dropping it would hide one that exists, and failing the refresh would lose the whole listing. Over-reporting costs one unopenable row — the same direction `Dirty::of` is deliberately sloppy in, and for the same reason. `entry_size` returns `Option` rather than `Result` to say this in the signature: there is no error to propagate, only a measurement that was not available. Not covered by the harness, and said so rather than quietly: Playwright drives one page, so the two-tab path is argued, not run. Filed. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/opfs.rs | 48 ++++++++++++++++++++------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/crates/improvise-io/src/opfs.rs b/crates/improvise-io/src/opfs.rs index d2ace80e..26782aa4 100644 --- a/crates/improvise-io/src/opfs.rs +++ b/crates/improvise-io/src/opfs.rs @@ -176,7 +176,7 @@ impl OpfsFs { .ok() .and_then(|v| v.as_string()) { - if self.entry_size(&name).await? > 0.0 { + if self.holds_bytes(&name).await { names.push(name); } } @@ -200,30 +200,42 @@ impl OpfsFs { Ok(()) } - /// How many bytes one entry holds. + /// Whether one entry holds anything — i.e. whether it is a document. /// /// Read off the sync access handle when this instance has one, and only - /// otherwise through `getFile()`: a file under an exclusive sync access - /// handle is not reliably readable by the promise-based API, and - /// `refresh` is called *after* `attach` in the harness as well as before - /// it in the worker. - async fn entry_size(&self, name: &str) -> Result { + /// otherwise through `getFile()`, because a file under an exclusive sync + /// access handle is not reliably readable by the promise-based API. + /// + /// **An entry it cannot measure counts as a document.** The one way that + /// happens is another context of the same origin holding the exclusive + /// handle — two tabs of the same site — and an entry somebody else has + /// open is certainly a document. The alternative is worse in both + /// directions: failing the whole `refresh` would make a second tab break + /// the first one's document picker, and dropping the entry would hide a + /// document that exists. Over-reporting costs one unopenable row, the + /// same direction `Dirty::of` is deliberately sloppy in. + async fn holds_bytes(&self, name: &str) -> bool { if let Some(handle) = self.handles.borrow().get(name) { - return handle - .get_size() - .map_err(|e| js_err(&format!("getSize({name})"), e)); + return handle.get_size().is_ok_and(|size| size > 0.0); } + self.entry_size(name).await.is_none_or(|size| size > 0.0) + } + + /// The size of an unattached entry, or `None` if it cannot be read. + async fn entry_size(&self, name: &str) -> Option { let file = JsFuture::from(self.root.get_file_handle(name)) .await - .map_err(|e| js_err(&format!("getFileHandle({name})"), e))? + .ok()? .dyn_into::() - .map_err(|_| anyhow!("getFileHandle({name}) did not yield a file handle"))?; - Ok(JsFuture::from(file.get_file()) - .await - .map_err(|e| js_err(&format!("getFile({name})"), e))? - .dyn_into::() - .map_err(|_| anyhow!("getFile({name}) did not yield a file"))? - .size()) + .ok()?; + Some( + JsFuture::from(file.get_file()) + .await + .ok()? + .dyn_into::() + .ok()? + .size(), + ) } fn with_handle( From 239f1f0b805501d24d7307188b90d427a508bfcb Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:48:12 -0700 Subject: [PATCH 35/72] test: the shipped-document walk must not skip a gzipped one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review of improvise-odbc's own guard. The walk filtered on `Path::extension() == "improv"`, and `extension()` of `demo.improv.gz` is `gz` — so a gzipped shipped document would have been skipped silently, which is exactly the "exercised by nothing" hole the test exists to close, put back one line inside the fix for it. Matched on the full name instead. Proved both arms: a valid `.improv.gz` dropped into `examples/` passes, and a truncated one fails. (The truncation also exercises the `with_context` added two commits ago, from the other side.) Co-Authored-By: Claude Opus 5 (1M context) --- tests/shipped_documents_load.rs | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/shipped_documents_load.rs b/tests/shipped_documents_load.rs index 64a045f2..c544e0fa 100644 --- a/tests/shipped_documents_load.rs +++ b/tests/shipped_documents_load.rs @@ -25,7 +25,13 @@ fn shipped_documents() -> Vec { .iter() .flat_map(|dir| std::fs::read_dir(root.join(dir)).expect("a shipped-document directory")) .map(|entry| entry.expect("a readable directory entry").path()) - .filter(|p| p.extension().is_some_and(|e| e == "improv")) + // `.improv.gz` is the same document gzipped, and `extension()` calls + // it `gz` — so matching on the extension alone would skip it and put + // back the very hole this test exists to close. + .filter(|p| { + p.to_str() + .is_some_and(|s| s.ends_with(".improv") || s.ends_with(".improv.gz")) + }) .collect(); found.sort(); found From ee65cb2815292fa103462313ecea27a6bf50ac31 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:48:47 -0700 Subject: [PATCH 36/72] ci(browser): name clippy and rustfmt, which this job actually uses The browser job lints harness/opfs-contract because it is not a workspace member and the `test` job's workspace-wide fmt and clippy cannot reach it. It was relying on the toolchain action's defaults for the two components it runs. Named, like the `test` job does. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e2131409..b67732f0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -94,6 +94,9 @@ jobs: - uses: dtolnay/rust-toolchain@stable with: targets: wasm32-unknown-unknown + # Named rather than assumed: this job lints harness/opfs-contract, + # which the `test` job's workspace-wide fmt and clippy cannot reach. + components: clippy, rustfmt - uses: Swatinem/rust-cache@v2 - uses: actions/setup-node@v4 with: From 0af4087b16059e4fc01b3912eb6350f928ee9bb6 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:48:48 -0700 Subject: [PATCH 37/72] test(core): the delete-and-recreate case the ordering doc claims but nothing tested MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two of Copilot's second-pass findings on PR 20, both correct. `live_records`' doc says the order is derived from `cells` on each call *rather than* memoised on `RecordEntry`, and gives the reason: the entry is a tombstone that survives deletion (so a re-created record cannot hand a stale basis its own version back), which is exactly what would leave a memoised position available to be wrong. Nothing tested that. As Copilot put it, a memoising implementation passes all three of the new `live_records` tests while keeping a re-created record ahead of records that outlived it — a claim in a doc comment that no check could see, which is this repo's characteristic defect. `a_recreated_record_takes_its_new_place_not_its_old_one` asserts `[West, East]` after East is removed and re-created. Proved by injecting a memoised first-sighting position: it lands on `[East, West]`, precisely the shape named. Also refreshes the `invalidate_relabelled` doc, which the `grid_updates` fix earlier in this PR made stale within the same branch — it still said the producer diffs `mirror.totals`. Rewritten as what the blanking *obliges* of a producer, which is the durable statement and the one that does not go stale the next time the diff moves. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-core/src/model/cell.rs | 38 +++++++++++++++++++++++++ crates/improvise-protocol/src/render.rs | 12 +++++--- 2 files changed, 46 insertions(+), 4 deletions(-) diff --git a/crates/improvise-core/src/model/cell.rs b/crates/improvise-core/src/model/cell.rs index a56e44e7..87d89a23 100644 --- a/crates/improvise-core/src/model/cell.rs +++ b/crates/improvise-core/src/model/cell.rs @@ -1170,6 +1170,44 @@ mod record_index { assert_eq!(names(&store_of(®ions)), reversed); } + /// The claim `live_records`' doc makes about *why* it derives the + /// order from `cells` rather than memoising a position on + /// [`RecordEntry`] — and which nothing else here could see (Copilot, + /// PR 20). A memoised position would pass all three tests above and + /// still put a re-created record back at its **original** place, while + /// its cells sit at the end of the store. + /// + /// The entry survives the deletion on purpose — it is a tombstone, so + /// a re-created record cannot hand a stale basis its own version back + /// (see [`RecordEntry::cells`]) — which is exactly what makes the + /// memoised position available to be wrong. + #[test] + fn a_recreated_record_takes_its_new_place_not_its_old_one() { + let mut store = DataStore::new(); + let east = key(&[("Region", "East"), ("_Measure", "Revenue")]); + let west = key(&[("Region", "West"), ("_Measure", "Revenue")]); + store.set(east.clone(), CellValue::Number(1.0)); + store.set(west.clone(), CellValue::Number(2.0)); + + let names = |store: &DataStore| -> Vec { + store + .live_records() + .map(|(_, record)| record.0[0].1.clone()) + .collect() + }; + assert_eq!(names(&store), vec!["East", "West"]); + + store.remove(&east); + assert_eq!(names(&store), vec!["West"], "a dead record is not listed"); + + store.set(east, CellValue::Number(3.0)); + assert_eq!( + names(&store), + vec!["West", "East"], + "a re-created record sits where its new cell sits, not where its old one did" + ); + } + /// The pair `live_records` yields is not two facts that could drift /// apart: the surrogate is the record's own, so it is the one that /// resolves it back. diff --git a/crates/improvise-protocol/src/render.rs b/crates/improvise-protocol/src/render.rs index 4d3810ee..44f9c4ea 100644 --- a/crates/improvise-protocol/src/render.rs +++ b/crates/improvise-protocol/src/render.rs @@ -313,10 +313,14 @@ impl GridCache { /// face value. Rows do not, because a total spans every row and is not /// framed by any one of them. /// - /// Unreachable from today's producer, which diffs - /// `mirror.totals != desired.totals` and emits `Totals` after - /// `ColLabels` in the same batch. That is precisely why it is fixed: - /// gotcha 33(b) exists to stop `apply` trusting a producer to remember. + /// This is what obliges the producer to diff the totals against the + /// frame the label updates will leave rather than against the mirror + /// they change: a column renamed while its total stayed the same + /// number is blanked here, and a producer comparing `mirror.totals` + /// would find them equal and send nothing. `grid_updates` compares + /// `framed.totals` for exactly that reason. The blanking stays anyway, + /// because gotcha 33(b) exists to stop `apply` trusting a producer to + /// remember — the rule lives on both sides or on neither. fn invalidate_relabelled(&mut self, old: &[String], axis: LabelAxis) { let new = match axis { LabelAxis::Row => &self.row_labels, From 8ff1f7ea8a40faff7fb01d2640869801f0af6425 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:49:10 -0700 Subject: [PATCH 38/72] docs(opfs): the module doc said refresh was the only writer of the index MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It is not, since the previous three commits: `write` indexes a name and un-queues its pending delete, `attach` deliberately indexes nothing, and `refresh` filters. The old sentence — "list answers from the index refresh populated" — was one copy of an argument that had moved. gotcha 48's third rule: when a fact is stated in more than one place, changing it is a grep, not an edit. This is the sibling the three fixes missed. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/opfs.rs | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/crates/improvise-io/src/opfs.rs b/crates/improvise-io/src/opfs.rs index 26782aa4..7ff4c915 100644 --- a/crates/improvise-io/src/opfs.rs +++ b/crates/improvise-io/src/opfs.rs @@ -20,13 +20,26 @@ //! [`OpfsFs::refresh`], [`OpfsFs::commit_deletes`]. The worker calls //! these from its message loop, where awaiting is free. //! - **sync, via the trait**: `read`/`write` use an already-attached -//! handle; `list` answers from the index `refresh` populated; `delete` -//! drops the document from the index and queues the removal for -//! `commit_deletes`. +//! handle; `list` answers from an index; `delete` drops the document from +//! that index and queues the removal for `commit_deletes`. //! //! A `delete` is therefore visible to `read` and `list` immediately and //! lands in OPFS at the end of the batch. That ordering is the price of //! keeping the trait synchronous, and it is invisible to a single worker. +//! +//! # The index holds documents, not entries +//! +//! `Storage::list` promises *document* names, and the two stopped being the +//! same thing the moment `attach` had to create a file to get a handle for a +//! document that does not exist yet (improvise-vvl3). So three writers, and +//! the split between them is the whole rule: +//! +//! - [`OpfsFs::attach`] indexes **nothing**. What it creates is zero bytes. +//! - [`Storage::write`] indexes the name, and un-queues any pending delete +//! of it. A written document is the first thing there is to list. +//! - [`OpfsFs::refresh`] rebuilds the index from OPFS, skipping zero-byte +//! entries — and counting an entry it cannot measure as a document, since +//! the one cause of that is another tab holding the exclusive handle. use std::cell::RefCell; use std::collections::HashMap; From 5c1fefb4f132476471bce14428c3a2b94ee99bab Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:49:13 -0700 Subject: [PATCH 39/72] docs: an access control the browser cannot satisfy is not a control Copilot's second pass, confirmed: the basic_auth recipe from the previous commit CANNOT WORK. web/app.js opens `new WebSocket(WS_URL)` -- one argument, a URL -- and the browser WebSocket API cannot set request headers, while a WebSocket handshake shows no login prompt. So Caddy basic_auth in front of it is a 401 the page can neither answer nor explain. That is worse than the unauthenticated recipe it replaced, because it READS AS PROTECTION. The first round's lesson was that documentation instructing an action carries the action's consequences; this round is sharper and is the one worth keeping: A RECIPE IS CODE, and it needs the same "would this actually run" test the Rust gets. Neither my review pass nor the fix asked it of a Caddyfile. Note it is the same fact that already killed native wss:// in this same document -- a WS handshake has no interstitial, which is why a self-signed cert is unusable. The argument was two sections away and did not transfer. So the section now sorts controls by whether a browser can satisfy them at all, since that is the property that decides it: * basic auth and client-cert mTLS: named as NOT working, with the reason. * IP allow-list: the worked example, via Caddy's documented remote_ip/abort. Needs no credential from the page. * private network (Tailscale/WireGuard): named; needs no credential either. * cloudflared + Cloudflare Access: named and explicitly NOT written out as a recipe, because it turns on a session cookie from a prior interactive login and third-party-cookie rules, which cannot be verified without a browser. Documenting a flow I cannot test is the trap improvise-hwpi named. `ngrok http --basic-auth` carried the identical bug and is corrected to --cidr-allow, which is the same address check. Says plainly that a URL-borne token is what would make this work browser-side -- a query parameter being the one credential `new WebSocket(url)` can carry -- and that the server has no such capability: improvise-kpnr (ID read back). And states that the Caddyfile has not been executed here, on the same terms as the existing browser-JS caveat. 1184 passing, unchanged; no code touched. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 63 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ebfe73f3..614301dd 100644 --- a/README.md +++ b/README.md @@ -246,12 +246,36 @@ of it removes exactly that protection*: the proxy is public, so the endpoint is public, and `--bind 127.0.0.1` protects the port while doing nothing about the proxy. TLS gets you confidentiality, not access control. -So put the access control in the proxy, and treat it as required rather than -optional — HTTP basic auth, mTLS, an IP allow-list, or a tunnel with its own -authentication (`cloudflared` can enforce Cloudflare Access; a bare -`--url` tunnel enforces nothing and is a public URL). Caddy's `basic_auth` -directive is the smallest version. Share only documents you would be willing -to let an anonymous visitor edit, and take the tunnel down when you are done. +**And the control has to be one a browser can actually satisfy**, which rules +out most of the obvious ones. The page opens the socket as +`new WebSocket(url)` — one argument, a URL. The browser WebSocket API **cannot +set request headers**, and a WebSocket handshake shows **no login prompt**, so +HTTP basic auth in front of it is not a locked door, it is a **401 the page +cannot answer and cannot explain** — the same reason a self-signed certificate +is unusable here (see "Why not native `wss://`"). Client-certificate mTLS has +the same problem: the selection UI browsers show for a top-level navigation is +not something a cross-origin socket handshake can rely on. + +So the honest controls are the ones that need **no credential from the page**: + +- **An IP allow-list in the proxy** — the cheapest, and the worked example + below. It is about where the client is, so the page sends nothing. +- **A private network** (Tailscale, WireGuard). The peer is only reachable by + machines already on it; again the page sends nothing. +- **An identity-aware tunnel** (`cloudflared` with Cloudflare Access). This one + works by a session **cookie** set after an interactive login on the proxy's + own hostname, so it needs the user to visit that hostname first, and it is + subject to the browser's third-party-cookie rules. Plausible and *not + verified here* — there is no browser in this repository — so it is named, not + written out as a recipe. + +What would make this genuinely easy is a **token in the URL**, because a query +parameter is the one credential `new WebSocket(url)` can carry. The server has +no such capability today, and whether it should grow one — or whether "auth is +the proxy's job" becomes a stated design position — is **improvise-kpnr**. + +Until then: share only documents you would be willing to let anyone on the +other side of that control edit, and take the tunnel down when you are done. A worked example with [Caddy](https://caddyserver.com/), which obtains and renews the certificate itself. Point a DNS name at the host, then: @@ -259,11 +283,12 @@ renews the certificate itself. Point a DNS name at the host, then: ```caddyfile # Caddyfile improvise.example.com { - # NOT optional — see above. Without this the document is world-writable. - # Generate the hash with: caddy hash-password - basic_auth { - editor $2a$14$...replace-with-your-own-hash... - } + # NOT optional — see above. Without it the document is world-writable. + # An allow-list, not a password: the browser cannot send a credential on a + # WebSocket handshake, so this is checked against the client's address. + @denied not remote_ip 203.0.113.4 198.51.100.0/24 + abort @denied + reverse_proxy 127.0.0.1:8787 } ``` @@ -273,6 +298,13 @@ improvise-ws-server bank-info.improv --port 8787 # still plain ws, still loopb caddy run # terminates TLS, proxies to it ``` +On the same terms as the browser caveat below: **this configuration has not +been executed in this repository** — there is no Caddy and no browser here. It +is the form Caddy documents for `remote_ip`, and the reasoning about what a +browser can satisfy is checked against `web/app.js`, which opens +`new WebSocket(url)` and can therefore send no header. Verify the deny actually +denies before trusting it with a document. + Then, from , reload with `?ws=improvise.example.com`. Caddy preserves the request path, so the multi-document form works too: serve several files and name one with @@ -287,9 +319,12 @@ For a one-off share with no DNS name and no certificate of your own, `cloudflared tunnel --url http://127.0.0.1:8787` or `ngrok http 8787` each print an https hostname that works the same way — pass the hostname to `?ws=` without a scheme. Both print a **public** URL and neither authenticates -anything by default, so add the tunnel's own auth (`ngrok http --basic-auth`, -or a named Cloudflare tunnel behind Access) rather than relying on the URL -being unguessable. +anything by default — and note that `ngrok http --basic-auth` does **not** fix +that here, for the reason above: the page cannot answer a basic-auth challenge +on a socket handshake. `ngrok http --cidr-allow` is the form that works, being +the same address check as the Caddy example. An unguessable URL is not a +control either. If you cannot put an address check in front of it, do not +expose it. Note the server binds `127.0.0.1` by default and `--bind` is the only way to widen that. Behind a proxy on the same host you should *not* widen it: the From 365f27bbd30ac87f5f9c177bf4c47b68547004be Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:50:33 -0700 Subject: [PATCH 40/72] =?UTF-8?q?fix(fit):=20work=20the=20code-review=20ba?= =?UTF-8?q?cklog=20=E2=80=94=20two=20vacuous=20tests=20and=20a=20hidden=20?= =?UTF-8?q?axis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review's two HIGH findings were both this repo's signature defect, and one of them is a straight own goal. **`the_seed_argument_changes_the_search` proved nothing.** It compared `effects_debug` over the whole batch, and `status_line` ends with `seed {:#x}` — so the two strings differed *because the seed was printed*, and `fit_measure` could have ignored `request.seed` entirely with the test green. It now asserts on the `AddFormula` debug alone: two fields, the formula text and the target category, neither of which the seed reaches. (A substring guard was tried and is wrong — `1` and `99` occur inside ordinary thresholds.) **The new `improvise-gp` fixture re-committed the bug the same commit fixed.** `document()` built its view with `View::new` + `on_category_added`, under which `_Measure` lands on `Axis::None` — so every `fit_measure` test ran under the one layout that hides the axis bug, while `Workbook::new` hands out `_Measure: Page`. The fixture now sets `Page`, and that is checkable: injecting the `filter_coords` fault back in used to fail two `dataset` tests and now fails **six**, the whole `fit::tests` suite included. Also from the review: * `Quality::Measured` at exactly two rows was garbage wearing the label — two folds of one row, each tree fit on a single constant. `folds` is now capped at `rows / 2`, so every fold and every training complement holds at least two. The confusion `Quality` exists to prevent was moved from one row to two, not removed. * `the_same_seed_under_a_different_config_is_a_different_search` varied `generations` over an exactly-learnable fixture, which can converge to the same optimum and fail one day for a reason that is not a regression. It varies `pop`, which changes the RNG draw count per generation. * `a_fit_that_needs_a_category_is_refused_naming_the_feature` cleared `Cost` but not `Score`, leaving `Region=North` carrying *both* target values — so the one-hot the test is about explained nothing. It now clears both. * `the_fitted_formula_predicts_in_the_documents_own_units` had a `let … else { continue }`, so a loop that skipped every record asserted nothing. It panics: "no cell evaluated" *is* the improvise-8xhy failure. * `FitOutcome::warnings` documented the splice order backwards. * `fitted_measure_name` joins the crate-root re-exports, and the one consumer uses that door for all three. And the layering cost the review asked to be measured rather than assumed. `improvise-command` is in the worker bundle and the path from `App::new` to this command's prototype is live, so `gp-tree` links in whether a browser invokes it or not. Built with `wasm-pack --release`: `improvise_worker_server_bg.wasm` 1,341,130 → 1,420,470 bytes (+79 KB, +5.9%), 498 → 531 KB gzipped (+6.5%). `improvise-wasm-client` does not depend on `improvise-command` and is unchanged. Accepted and written down on the module, with the feature-gate named as the lever if it ever needs shrinking. 1216 passing, unchanged; clippy, fmt and the wasm gate clean; the CLI's output still byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/cmd/fit.rs | 49 ++++++++++++++++-- crates/improvise-gp/src/fit.rs | 51 +++++++++++++++---- crates/improvise-gp/src/lib.rs | 2 +- 3 files changed, 87 insertions(+), 15 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/fit.rs b/crates/improvise-command/src/command/cmd/fit.rs index a762385a..cd58b10f 100644 --- a/crates/improvise-command/src/command/cmd/fit.rs +++ b/crates/improvise-command/src/command/cmd/fit.rs @@ -37,8 +37,28 @@ //! could not be resolved, because "the fit failed" is not something a user can //! act on and "it split on Region, which is a category rather than a measure" //! is. +//! +//! # What registering it costs the browser, measured +//! +//! `improvise-command` is in the **worker** bundle, and the path from +//! `App::new` → `default_keymaps` → `default_registry` → this command's +//! prototype is live, so `gp-tree` links into it whether or not a browser ever +//! invokes `fit-measure`. `kb.pl` names keeping GP out of the wasm bundle as +//! `improvise-gp`'s reason to exist, so this is a cost to state rather than to +//! discover: built with `wasm-pack --release`, `improvise_worker_server_bg.wasm` +//! goes **1,341,130 → 1,420,470 bytes** (+79 KB, +5.9%), or **498 → 531 KB +//! gzipped** (+6.5%). `improvise-wasm-client` — the main-thread bundle — does +//! not depend on `improvise-command` at all and is **unchanged**. +//! +//! Accepted rather than feature-gated, for two reasons. The worker already +//! carries the whole `App`, persistence and the import wizard, so this is a +//! 6% addition to something an order of magnitude larger; and a browser fit is +//! a *wanted* end state, not dead weight — `improvise_gp::fit` exists where it +//! does precisely so a second host can run the same sequence. If the bundle +//! ever needs shrinking, a default-on cargo feature that `web/build.sh` turns +//! off is the lever, and this paragraph is the number to beat. -use improvise_gp::fit::{FitRequest, fit_measure, fitted_measure_name}; +use improvise_gp::{FitRequest, fit_measure, fitted_measure_name}; use crate::command::cmd::core::{Cmd, CmdContext}; use crate::ui::effect::{self, Effect}; @@ -268,16 +288,37 @@ mod tests { /// gotcha 37b: the seed is half of what reproduces a fit, so it is an /// argument — and a different one really does search differently. + /// + /// It asserts on the **rule**, never on the batch's debug string. The + /// first version compared `effects_debug` over the whole vec, which was + /// vacuous: `status_line` prints `seed {:#x}`, so the two strings differ + /// because the seed was *printed*, and `fit_measure` could have ignored + /// `request.seed` entirely with the test still green + /// (`doctrine(check_the_checker)`). + /// + /// `AddFormula` has exactly two fields — the formula text and the target + /// category — and the seed reaches neither, so its `Debug` cannot differ + /// for the reason the batch's did. (A substring guard was tried and is + /// wrong: the seeds `1` and `99` occur inside perfectly ordinary + /// thresholds.) #[test] fn the_seed_argument_changes_the_search() { let wb = workbook(); let layout = make_layout(&wb); let registry = make_registry(); let ctx = make_ctx(&wb, &layout, ®istry); + let rule = |seed: &str| { + cmd(&["Score", "A", seed]) + .execute(&ctx) + .iter() + .map(|e| format!("{e:?}")) + .find(|d| d.starts_with("AddFormula")) + .expect("a successful fit adds a formula") + }; assert_ne!( - effects_debug(&cmd(&["Score", "A", "1"]).execute(&ctx)), - effects_debug(&cmd(&["Score", "A", "99"]).execute(&ctx)), - "two seeds produced the identical batch of effects" + rule("1"), + rule("99"), + "two seeds produced the identical rule" ); } diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index 344c2670..10954bc9 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -62,7 +62,8 @@ pub struct FitRequest { pub seed: u64, /// The search's knobs — necessary beside the seed, not implied by it. pub config: FitConfig, - /// Cross-validation folds. Below two, no held-out evidence is produced and + /// Cross-validation folds, capped at half the record count so no fold is + /// a single row. Below two, no held-out evidence is produced and /// [`Quality::Unmeasured`] says so. pub cv_folds: usize, /// Shuffles per feature when measuring permutation importance. @@ -126,8 +127,8 @@ pub struct FitOutcome { /// `(feature name, percentage share)`, most important first, features the /// rule does not use omitted. pub importance: Vec<(String, f32)>, - /// Records and columns the selection dropped — `preprocess`'s notices and - /// the feature table's, in that order. + /// Records and columns the selection dropped — the feature table's notices + /// first, then `preprocess`'s, as [`crate::dataset::dataset`] splices them. pub warnings: Vec, /// The seed this fit ran under, and… pub seed: u64, @@ -223,7 +224,15 @@ pub fn fit_measure( // Held-out evidence first, and from the same generator, so the sequence // reproduces as a whole rather than step by step. - let folds = request.cv_folds.min(rows.len()); + // + // The cap is `rows / 2`, not `rows`. `fit_csv` clamps to the row count, + // which lets two rows become two folds of one: each tree is then fit on a + // *single* row — a constant — and the numbers that come back mean nothing + // while wearing the `Measured` label. That is the confusion `Quality` + // exists to prevent, moved from one row to two rather than removed. This + // floor guarantees every fold, and every training complement, holds at + // least two rows. + let folds = request.cv_folds.min(rows.len() / 2); let quality = match folds >= 2 { true => Quality::Measured { folds, @@ -349,6 +358,13 @@ mod tests { view.on_category_added(cat); } view.set_axis(REGION, Axis::Row); + // `on_category_added` sends a leading-underscore category to + // `Axis::None`, but `Workbook::new` — the constructor a real document + // comes from — leaves `_Measure` on `Axis::Page`. Every test in + // `dataset.rs` used the first, and the second is the layout under + // which the whole feature matrix used to collapse. Fitting under the + // real default is not an extra case, it is the only honest one. + view.set_axis(MEASURE, Axis::Page); (model, view) } @@ -406,9 +422,12 @@ mod tests { model.add_formula(outcome.formula.clone()); model.recompute_formulas(&[]); for i in 0..12u32 { + // `panic`, not `continue`: a loop that skips every record asserts + // nothing, and "no cell evaluated" is itself the improvise-8xhy + // failure this whole path exists to prevent. let Some(CellValue::Number(n)) = model.evaluate(&cell(&format!("r{i}"), FIT_INTO)) else { - continue; + panic!("record r{i} has no numeric value for the fitted measure"); }; assert!( (500.0..=5000.0).contains(&n), @@ -416,8 +435,8 @@ mod tests { document's 1000–4000" ); } - // …and the same claim read off the formula text, so the assertion does - // not depend on the target item having been resolvable above. + // …and the same claim read off the formula text, which does not depend + // on evaluation at all. let numbers = leaf_numbers(&outcome.formula.expr); assert!(!numbers.is_empty(), "the fit emitted no leaf at all"); assert!( @@ -454,13 +473,19 @@ mod tests { /// And the other half: a different *config* under the same seed searches /// differently, which is why recording the seed alone would not do. + /// + /// The knob varied is `pop`, not `generations`. More generations over an + /// exactly-learnable fixture can land on the same optimum and `simplify` + /// normalises it, so that version of the test would fail one day for a + /// reason that is not a regression. `pop` changes how many RNG draws each + /// generation makes, so the two searches diverge from the first one. #[test] fn the_same_seed_under_a_different_config_is_a_different_search() { let (model, view) = document(); let base = request(); let wider = FitRequest { config: FitConfig { - generations: base.config.generations * 3, + pop: base.config.pop * 3, ..base.config }, ..base @@ -515,13 +540,19 @@ mod tests { #[test] fn a_fit_that_needs_a_category_is_refused_naming_the_feature() { let (mut model, view) = document(); - // Region is the only thing left that can explain Score. model .category_mut(REGION) .expect("Region exists") .add_item("South"); + // Region is the *only* thing left that can explain Score, so every + // record the fixture wrote at Region=North has to go — clearing only + // `Cost` would leave North carrying both target values and make the + // one-hot explain nothing, which is not the fixture this comment + // describes. for i in 0..12u32 { - model.clear_cell(&cell(&format!("r{i}"), "Cost")); + for measure in ["Cost", "Score"] { + model.clear_cell(&cell(&format!("r{i}"), measure)); + } } for (i, region) in (0..12u32).map(|i| (i, if i < 6 { "North" } else { "South" })) { model.set_cell( diff --git a/crates/improvise-gp/src/lib.rs b/crates/improvise-gp/src/lib.rs index c3069ab8..b875d3df 100644 --- a/crates/improvise-gp/src/lib.rs +++ b/crates/improvise-gp/src/lib.rs @@ -87,7 +87,7 @@ pub mod dataset; pub mod fit; pub use dataset::{FeatureTable, dataset, feature_table}; -pub use fit::{FitOutcome, FitRequest, Quality, fit_measure}; +pub use fit::{FitOutcome, FitRequest, Quality, fit_measure, fitted_measure_name}; use gp_tree::{Spec, Tree}; use improvise_core::model::Model; From 394f60e84b5980cbb5c4c4ae3bcfad3800af7c65 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:52:16 -0700 Subject: [PATCH 41/72] =?UTF-8?q?fix:=20both=20Copilot=20findings=20on=20P?= =?UTF-8?q?R=20#22=20=E2=80=94=20gzip=20diagnosis,=20and=20padding=20in=20?= =?UTF-8?q?cells?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both correct, both applied, both watched failing with the fault injected. **A valid gzip holding invalid UTF-8 was blamed on gzip.** `read_to_string` decodes as it decompresses, so the `with_context` two commits ago — added because the gzip branch named no file where the UTF-8 branch did — ended up putting `is not readable gzip` on a document that had decompressed perfectly well. Decompression and decoding are two failures with two answers, so they now have two contexts: `read_to_end` under the gzip context, then one `String::from_utf8` under the UTF-8 context, shared with the uncompressed branch. That also leaves one answer to "is this UTF-8" instead of one per branch. `a_valid_gzip_of_invalid_utf8_is_reported_as_not_utf8` asserts the UTF-8 diagnosis *and* that the message does not blame gzip, with the payload asserted non-UTF-8 as the precondition — without it the test would also pass on a document that merely failed to decompress, which the other two tests own. Restoring the fused `read_to_string` fails this test and only this test; the `UnexpectedEof` and `InvalidInput` kinds are unchanged. **The bottom line was padded in characters, not terminal cells.** `Buffer` lays a string out by display width. A message carrying a wide character therefore wrote past the panel and over its right border, and a combining mark — zero cells — under-padded and left a sliver of the hint. Measured with `UnicodeWidthStr::width`, as `src/ui/grid.rs` already does. `truncate` still cuts by bytes, which is deliberate and documented there; bytes are never fewer than cells, so the two together cannot overrun. `the_bottom_line_is_padded_in_cells_not_characters` drives both characters — six CJK and three combining acutes — and asserts through the thing that breaks: the row still ends in the panel border, the row no longer contains `cancel`, and the panel is still 22 rows. Reverting to `chars().count()` fails it. Both findings were about a case the eight ASCII characterization frames cannot reach, which is exactly why they were worth having. None of those frames moved. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/persistence/mod.rs | 67 +++++++++++++++++++--- src/ui/import_wizard_ui.rs | 50 +++++++++++++++- 2 files changed, 109 insertions(+), 8 deletions(-) diff --git a/crates/improvise-io/src/persistence/mod.rs b/crates/improvise-io/src/persistence/mod.rs index 9978f933..49e0722e 100644 --- a/crates/improvise-io/src/persistence/mod.rs +++ b/crates/improvise-io/src/persistence/mod.rs @@ -85,16 +85,24 @@ pub fn save_to(storage: &dyn Storage, workbook: &Workbook, path: &Path) -> Resul /// Load a workbook from `path` in `storage`. pub fn load_from(storage: &dyn Storage, path: &Path) -> Result { - let bytes = storage.read(path)?; - let text = if is_gzip(path) { - let mut s = String::new(); - GzDecoder::new(BufReader::new(&bytes[..])) - .read_to_string(&mut s) + let stored = storage.read(path)?; + // Decompression and UTF-8 decoding are two failures with two answers, so + // they get two contexts. `read_to_string` would have fused them, and a + // valid gzip stream holding invalid UTF-8 would then have been reported as + // unreadable gzip — a wrong diagnosis of a document that decompressed + // perfectly well. Decoding once, after, also leaves one answer to "is this + // UTF-8" rather than one per branch. + let bytes = if is_gzip(path) { + let mut out = Vec::new(); + GzDecoder::new(BufReader::new(&stored[..])) + .read_to_end(&mut out) .with_context(|| format!("{} is not readable gzip", path.display()))?; - s + out } else { - String::from_utf8(bytes).with_context(|| format!("{} is not UTF-8", path.display()))? + stored }; + let text = + String::from_utf8(bytes).with_context(|| format!("{} is not UTF-8", path.display()))?; if text.trim_start().starts_with('{') { serde_json::from_str(&text).context("Failed to deserialize workbook") } else { @@ -322,6 +330,51 @@ mod tests { ); } + /// The third failure, and the one that made the first two worth keeping + /// apart in the code as well as in the tests (Copilot, PR #22). A gzip + /// stream can be perfectly well-formed and still hold bytes that are not + /// UTF-8; `read_to_string` decodes as it decompresses, so a single context + /// around it called that document unreadable gzip — a wrong diagnosis of a + /// document that decompressed without complaint. + /// + /// Decompression and decoding are separated, so this reports what the + /// uncompressed path would report for the same bytes. The precondition + /// asserts the payload really is invalid UTF-8; without it the test would + /// also pass on a document that merely failed to decompress, which is the + /// case the other two tests already own. + #[test] + fn a_valid_gzip_of_invalid_utf8_is_reported_as_not_utf8() { + use crate::storage::{MemoryFs, Storage}; + use flate2::{Compression, write::GzEncoder}; + use std::io::Write; + use std::path::Path; + + let payload = b"v2025-04-09\n# \xff\xfe\n"; + assert!( + String::from_utf8(payload.to_vec()).is_err(), + "precondition: the payload is not UTF-8" + ); + + let mut enc = GzEncoder::new(Vec::new(), Compression::default()); + enc.write_all(payload).unwrap(); + let bytes = enc.finish().unwrap(); + + let fs = MemoryFs::new(); + let path = Path::new("model.improv.gz"); + fs.write(path, &bytes).unwrap(); + + let err = super::load_from(&fs, path).expect_err("invalid UTF-8 must not load"); + let msg = err.to_string(); + assert!( + msg.contains("is not UTF-8"), + "expected the UTF-8 diagnosis, got: {msg}" + ); + assert!( + !msg.contains("gzip"), + "the gzip stream was fine; blaming it is the bug: {msg}" + ); + } + // ── autosave_path ─────────────────────────────────────────────────── #[test] diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index 94fc5104..0c8557ac 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -5,6 +5,8 @@ use ratatui::{ widgets::{Block, Borders, Clear, Widget}, }; +use unicode_width::UnicodeWidthStr; + use crate::protocol::{WizardCache, WizardFieldKind, WizardStepKind}; /// The import wizard, drawn from a [`WizardCache`] (improvise-764). @@ -412,9 +414,18 @@ impl<'a> Widget for ImportWizardWidget<'a> { /// Padding here rather than at each site is the point. The bug is not that /// one of the five writers forgot; it is that the row had five writers and no /// one of them owned clearing it. +/// +/// The padding is measured in **terminal cells**, not characters. `Buffer` +/// lays a string out by display width, so a char count over-pads a message +/// carrying a wide character — enough to run past the panel and overwrite the +/// border — and under-pads nothing, which is how a combining mark would have +/// left a sliver of the hint behind. `src/ui/grid.rs` measures the same way. +/// `truncate` above still cuts by bytes, which is deliberate and documented +/// there; bytes are never fewer than cells, so the two together cannot +/// overrun. fn set_bottom_line(buf: &mut Buffer, x: u16, y: u16, w: usize, text: &str, style: Style) { let text = truncate(text, w); - let pad = " ".repeat(w.saturating_sub(text.chars().count())); + let pad = " ".repeat(w.saturating_sub(UnicodeWidthStr::width(text.as_str()))); buf.set_string(x, y, format!("{text}{pad}"), style); } @@ -637,6 +648,43 @@ mod tests { ); } + /// The padding is in terminal cells, not characters (Copilot, PR #22). + /// `Buffer` lays a string out by display width, so a message carrying a + /// wide character padded by `chars().count()` writes past the panel and + /// over its right border — and a combining mark, which is zero cells, + /// under-pads and leaves a sliver of the hint behind. + /// + /// Both are asserted through the border, which is the thing that breaks: + /// every row of a correctly drawn panel ends in `│`, and the row must not + /// contain what it replaced. `render` draws into an 80×22 area, so the + /// border is real and not an artefact of the assertion. + #[test] + fn the_bottom_line_is_padded_in_cells_not_characters() { + for message in [ + "\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}", // six wide chars, twelve cells + "e\u{0301}e\u{0301}e\u{0301}", // combining acutes, zero cells each + ] { + let mut w = wizard(); + w.message = Some(message.to_string()); + let drawn = render(&w); + let bottom = drawn.lines().nth_back(1).expect("the row above the border"); + + assert!( + bottom.ends_with('│'), + "{message:?} pushed the panel border off the row: {bottom:?}" + ); + assert!( + !bottom.contains("cancel"), + "{message:?} left the hint behind: {bottom:?}" + ); + assert_eq!( + drawn.lines().count(), + 22, + "{message:?} changed the panel's height" + ); + } + } + /// The two steps the flat-array fixture skips. `SelectArrayPath` is /// reached by a document with more than one candidate array, which is /// also the only step that draws `pipeline.array_paths`. From 5bfa747942ff6561cbbb456bcdbf200a344d4f4e Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:54:13 -0700 Subject: [PATCH 42/72] fix(assemble): verify each cache-bust stamp on its own, and rename a test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two Copilot findings on PR #24, one applied as asked and one applied only in part, with the measurement that says why. **The combined `grep` was a real hole.** `grep -n "?v=" a b c d` succeeds when *any one* file matches, so under `set -e` three failed substitutions still produced a "successful" assembly. Those greps are the only guard against the generated wasm-pack glue changing shape — which is the exact failure this cache-busting exists to end, twice, during improvise-9x6 — so each stamped reference is now checked on its own and names itself when it is the one missing. Inherited from the inline CI step rather than written here, but it became mine when it became a script. Proved by pointing one `sed` at a pattern that cannot match: the old form exited 0, the new one exits 1 naming the file. **The repaint count was the right objection to the wrong remedy.** Copilot asked for "count the relevant grid mutations/repaints and prove the assertion by removing the guard". Done, and it does not discriminate: with `if (closed) return` removed from `edit`'s `close`, `close` still runs exactly twice, `typed_cell` is still called exactly once, `#grid` still takes exactly two childList mutations, and the final text is identical. The only difference is one extra *entry* into `repaint()` that produces no DOM change — because the second close is `close(false)`, so it does not re-send, and everything else it does is idempotent. Which is exactly what improvise-hwpi meant by "harmless before, but only by accident". Counting `repaint()` entries would need a test hook inside `web/app.js`: a page that knows it is being tested. So the half of the finding that stands is the **name**, which claimed more than the body checked — the same defect as `open_and_refresh_see_an_empty_origin_root` two commits ago. Renamed to `committing an edit sends one frame and shows the host's value`, and the measurement is recorded in the comment so nobody repeats it. Co-Authored-By: Claude Opus 5 (1M context) --- harness/tests/shell.spec.js | 33 +++++++++++++++++++++++---------- web/assemble.sh | 19 ++++++++++++++++--- 2 files changed, 39 insertions(+), 13 deletions(-) diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js index b9d3ca17..9c054fa5 100644 --- a/harness/tests/shell.spec.js +++ b/harness/tests/shell.spec.js @@ -84,15 +84,26 @@ test.describe("the standalone deployment", () => { await expect(page.locator("#grid td.editable").first()).toBeVisible(); }); - // improvise-hwpi's third fix: Enter removes the focused , which - // fires `blur`, which called `close` a second time — harmless before, but - // only by accident. The observable is that the commit takes effect exactly - // once and the value on screen is the SERVER's, not the string typed. + // A commit round-trips through the host, and what lands on screen is the + // HOST's value rather than the string that was typed. `20000` is typed and + // `20,000` asserted: the number format is applied by the host and relayed + // back, so a page that echoed the input would fail. // - // `20000` is typed and `20,000` is asserted: the number format is applied - // by the host and relayed back, so a page that simply echoed what was typed - // would show `20000` and fail. - test("committing an edit repaints once, with the server's value", async ({ + // This is deliberately NOT named for improvise-hwpi's third fix (the + // `closed` re-entry guard in `edit`), which Copilot asked for on PR #24. + // That guard is **not observable from the DOM**, measured rather than + // assumed: with it removed, `close` still runs exactly twice, the frame is + // still sent exactly once, `#grid` still takes exactly two childList + // mutations, and the final text is identical. The only difference is one + // extra *entry* into `repaint()` that produces no output — because the + // second close is `close(false)`, so it does not re-send, and everything + // else it does is idempotent. That is precisely why hwpi called it + // "harmless before, but only by accident". + // + // Counting repaint() entries would need a test hook inside `web/app.js`, + // which would be a page that knows it is being tested. So the name says + // what is checked and this comment says what is not. + test("committing an edit sends one frame and shows the host's value", async ({ page, }) => { await page.goto("/index.html?doc=commit-test.improv"); @@ -108,8 +119,10 @@ test.describe("the standalone deployment", () => { await expect(cell).toHaveText("20,000", { timeout: 10_000 }); expect(before).not.toBe("20,000"); - // One editor, closed once: a second `close` would reopen nothing, but a - // stray input left in the DOM is what the guard exists to prevent. + // The editor is gone and did not come back. This is the half of the + // re-entry guard that IS observable: whatever `close` ran, no stray + // is left behind and the host's value is not overwritten by the + // `original` text the second close writes before repainting. await expect(page.locator("#grid input")).toHaveCount(0); }); }); diff --git a/web/assemble.sh b/web/assemble.sh index 0b4572f1..8a84bc4f 100755 --- a/web/assemble.sh +++ b/web/assemble.sh @@ -54,6 +54,19 @@ sedi "s|'improvise_wasm_client_bg\.wasm'|'improvise_wasm_client_bg.wasm?v=${toke sedi "s|'improvise_worker_server_bg\.wasm'|'improvise_worker_server_bg.wasm?v=${token}'|" \ "$out/worker-pkg/improvise_worker_server.js" -grep -n "?v=" "$out/app.js" "$out/worker.js" \ - "$out/pkg/improvise_wasm_client.js" \ - "$out/worker-pkg/improvise_worker_server.js" +# One `grep` over all four files succeeds when ANY one of them matches, so +# three failed substitutions could still produce a "successful" assembly under +# `set -e` (found by Copilot on PR #24). These checks are the only guard +# against the generated glue changing shape, so each stamped reference is +# verified on its own and names itself when it is the one missing. +stamped() { + grep -q "$2" "$1" || { + echo "assemble: $1 is missing the cache-bust stamp: $2" >&2 + exit 1 + } + grep -n "$2" "$1" +} +stamped "$out/app.js" "pkg/improvise_wasm_client.js?v=${token}" +stamped "$out/worker.js" "worker-pkg/improvise_worker_server.js?v=${token}" +stamped "$out/pkg/improvise_wasm_client.js" "improvise_wasm_client_bg.wasm?v=${token}" +stamped "$out/worker-pkg/improvise_worker_server.js" "improvise_worker_server_bg.wasm?v=${token}" From afa4a1792167115bd814a038ca72d4d9b89215a3 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:54:55 -0700 Subject: [PATCH 43/72] =?UTF-8?q?fix:=20two=20Copilot=20findings=20on=20th?= =?UTF-8?q?e=20save=20prompt=20=E2=80=94=20a=20dead=20room=20and=20a=20hid?= =?UTF-8?q?den=20error?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both correct, both applied, and both wider than the line they were raised on. **A prompt a browser cannot commit is never entered.** `enter-mode` is trivially resolvable in the wasm client, so `Ctrl-W` opened `SavePrompt` there — and Enter then resolved `[commit-save, clear-buffer]`, whose first step has no arm in `keys::step`, went `Upstream`, and was dropped by `Host::handle`'s empty vec. The user types a path into a prompt that swallows it. `doctrine(an_upstream_key_is_a_black_hole)` already says an upstream key does nothing; what this adds is that a *doorway* to such a key is worse than the key, because the door costs one inert keypress and the room costs work the user believes was done. `Ctrl-E` into `ExportPrompt` had the identical trap and predates this branch, so both are fixed: fixing one arm of a rule with two is the shape `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` names. Neither is an omission to close later — `Save`, `SaveAs` and `ExportCsv` are all `ViewOnly`, so no `ModelCommand` names them and no client on any transport can ask a host to write a file (gotcha 45). Named as a predicate, `strands_the_user`, rather than matched at the call site (`doctrine(two_questions_one_enum)`): "can this client finish what entering the mode starts" is a different question from "is this a text-entry mode". **And the TUI reported a failed save where nothing draws it.** `SaveAs` aborts the batch on failure, so with `CommitSave` emitting `[SaveAs, ChangeMode(Normal)]` the app stayed in `SavePrompt` — where `draw_bottom_bar` renders the minibuffer *instead of* the status line, so the "Save error:" set a statement earlier was never drawn. The save appeared to do nothing at all. Leading with the mode change puts the error in `Normal`, where the bottom row is the status bar. It costs the typed path, which the welded `clear-buffer` takes on success anyway and `EnterMode` clears on the next `Ctrl-W` regardless (gotcha 11). Evidence, both injected. `a_prompt_this_client_cannot_commit_is_never_entered` fails on both arms when `strands_the_user` returns false, and carries a negative arm (`:` still opens the command line) so "refuse every mode" cannot pass it. The end-to-end test now asserts the mode *left* the prompt — the opposite of what it asserted an hour ago, deliberately — and fails when the two effects are swapped back. Copilot's third finding, that `context/repo-map.md` and `prolog-sessions/kb.pl` still say 16 AppMode variants and seven text-entry modes, is correct and is **not** applied here: both files are orchestrator-only in this wave. The exact edits are in the hand-back. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/commit.rs | 19 +++- crates/improvise-command/src/ui/app.rs | 46 +++++---- crates/improvise-wasm-client/src/keys.rs | 98 ++++++++++++++++++- 3 files changed, 137 insertions(+), 26 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/commit.rs b/crates/improvise-command/src/command/cmd/commit.rs index 2421954b..4a44a22f 100644 --- a/crates/improvise-command/src/command/cmd/commit.rs +++ b/crates/improvise-command/src/command/cmd/commit.rs @@ -619,9 +619,26 @@ impl Cmd for CommitSave { effect::change_mode(AppMode::Normal), ]; } + // The mode change comes **first**, and that ordering is the whole + // reason the user sees a failed save (found by Copilot on PR #27). + // + // `SaveAs` aborts the batch when the write fails (improvise-8tk's + // other arm), so anything after it is skipped. With the order + // reversed the batch stopped while still in `SavePrompt` — and + // `draw_bottom_bar` renders the minibuffer *instead of* the status + // line, so the "Save error:" set by the effect was never drawn. The + // save appeared to do nothing at all. + // + // Leading with the mode change means the error lands in `Normal`, + // where the status bar is what the bottom row draws. It costs the + // typed path, which the welded `clear-buffer` would have taken on a + // success anyway and which `EnterMode` clears on the next Ctrl-W + // regardless (gotcha 11) — so nothing durable is lost, and the + // convention that the side effect runs last and its message wins is + // the one gotcha 20 already states. vec![ - Box::new(effect::SaveAs(std::path::PathBuf::from(path))), effect::change_mode(AppMode::Normal), + Box::new(effect::SaveAs(std::path::PathBuf::from(path))), ] } } diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 8290d9a5..01cbb048 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1617,24 +1617,21 @@ mod tests { /// improvise-lae end to end: `Ctrl-W`, a path, `Enter`. /// - /// The failing case is the interesting one, and it is what `SaveAs` - /// gaining `Save`'s abort buys the prompt. `CommitSave` emits - /// `[SaveAs, ChangeMode(Normal)]` and the keymap welds a trailing - /// `clear-buffer`, so an aborting `SaveAs` skips both: the user is left - /// in the prompt, with the path they typed still there to correct. - /// Without the abort they would have been dropped back to Normal with an - /// empty buffer. + /// The failing case is the interesting one, and it pins an **ordering** + /// (found by Copilot on PR #27). `SaveAs` aborts the batch on a failed + /// write, so whatever `CommitSave` emits after it is skipped. A first + /// version emitted `[SaveAs, ChangeMode(Normal)]`, which left the app in + /// `SavePrompt` — and `draw_bottom_bar` renders the minibuffer *instead + /// of* the status line, so the "Save error:" the effect had just set was + /// never drawn. The save appeared to do nothing whatsoever. /// - /// One caveat this test can see and the TUI cannot: `draw_bottom_bar` - /// renders the minibuffer *instead of* the status line, so the error is - /// recorded in `status_msg` and not drawn while the prompt is open. That - /// is pre-existing and wider than this prompt — a failed - /// `commit-buffer category` in `CategoryAdd`, which improvise-6t3 - /// deliberately ordered so its failure message would win, is invisible - /// for the same reason. Reported, not fixed here: the bottom bar is one - /// line and giving it both is a UI decision. + /// Leading with the mode change puts the error in `Normal`, where the + /// bottom row is the status bar. This test therefore asserts the mode + /// *left* the prompt, which is the opposite of what it asserted before — + /// deliberately, and it is the only thing keeping the two effects in + /// that order. #[test] - fn ctrl_w_prompts_for_a_path_and_a_failed_write_keeps_the_prompt_open() { + fn ctrl_w_prompts_for_a_path_and_a_failed_write_reports_it_where_it_shows() { let mut app = App::new(Workbook::new("T"), None); app.handle_key(KeyEvent::new(KeyCode::Char('w'), KeyModifiers::CONTROL)) .unwrap(); @@ -1662,15 +1659,16 @@ mod tests { "got {:?}", app.view_state.status_msg ); - assert!( - matches!(app.view_state.mode, AppMode::SavePrompt { .. }), - "a failed write must leave the prompt open, got {:?}", - app.view_state.mode - ); assert_eq!( - app.view_state.buffers.get("save").map(String::as_str), - Some(path), - "the path the user typed must survive for them to correct" + app.view_state.mode, + AppMode::Normal, + "the error must be reported in a mode whose bottom row is the \ + status bar, not behind a minibuffer that hides it" + ); + assert!( + app.view_state.mode.minibuffer().is_none(), + "setup for the assertion above: only a mode with no minibuffer \ + draws the status line at all" ); } diff --git a/crates/improvise-wasm-client/src/keys.rs b/crates/improvise-wasm-client/src/keys.rs index f3333795..f0b3f019 100644 --- a/crates/improvise-wasm-client/src/keys.rs +++ b/crates/improvise-wasm-client/src/keys.rs @@ -316,7 +316,10 @@ fn step(state: &ClientState, event: KeyEvent, name: &str, args: &[String]) -> Op // ── mode entry ─────────────────────────────────────────────── "enter-mode" => { let mode = AppMode::from_name(args.first()?)?; - Some(entered(mode)) + match strands_the_user(&mode) { + true => None, + false => Some(entered(mode)), + } } // ── panels ─────────────────────────────────────────────────── @@ -427,6 +430,41 @@ fn write_buffer(name: &str, value: String) -> Command { } } +/// A prompt this client can *enter* but cannot *commit*. +/// +/// `enter-mode` is trivially resolvable here, and that is the trap: the user +/// reaches the prompt, types a path, presses Enter — and the commit resolves +/// to nothing, because `Host::handle` answers a `Key` frame with an empty +/// vec (`doctrine(an_upstream_key_is_a_black_hole)`). A dead room is worse +/// than a locked door: the door costs one inert key, the room swallows work +/// the user believes was done. +/// +/// Both prompts are host-only **by construction**, not by omission. +/// `effect::Save`, `SaveAs` and `ExportCsv` are all `EffectKind::ViewOnly`, +/// so the derivation that builds `ModelCommand` from the model effects +/// cannot name them — no client on any transport can ask its host to write a +/// file (gotcha 45). Closing this is not a resolver to add; it is a wire +/// operation that does not exist. In a browser it is also close to +/// meaningless: the worker persists on every `touches_document`, so the +/// document *is* the OPFS entry and there is no Ctrl-S semantics to offer. +/// +/// Named as a property rather than matched at the call site +/// (`doctrine(two_questions_one_enum)`): "can this client finish what +/// entering the mode starts" is not the same question as "is this a +/// text-entry mode", and a third such prompt belongs here. +/// +/// Found by Copilot on PR #27 against the newly added `SavePrompt`. +/// `ExportPrompt` had the identical trap already and is fixed with it — +/// fixing one arm of a rule with two is the shape +/// `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` +/// names. +fn strands_the_user(mode: &AppMode) -> bool { + matches!( + mode, + AppMode::SavePrompt { .. } | AppMode::ExportPrompt { .. } + ) +} + /// `EnterMode::execute`: clear the incoming mode's minibuffer, then change. fn entered(mode: AppMode) -> Vec { let mut commands = Vec::new(); @@ -1013,6 +1051,17 @@ mod tests { KeyEvent::new(KeyCode::Char('q'), KeyModifiers::CONTROL), "force-quit", ), + // The two doors this client refuses rather than opens — + // `enter-mode` itself is resolvable, the commit behind it is + // not. See `strands_the_user`. + ( + KeyEvent::new(KeyCode::Char('w'), KeyModifiers::CONTROL), + "enter-mode save-prompt (commit-save is the host's)", + ), + ( + KeyEvent::new(KeyCode::Char('e'), KeyModifiers::CONTROL), + "enter-mode export-prompt (commit-export is the host's)", + ), ] { assert_eq!( resolve(&state, key), @@ -1024,6 +1073,53 @@ mod tests { } } + /// The refusal above is only half the property. `Upstream` says the key + /// produced no `Command`; it does not say the client stayed put, and the + /// bug being guarded is precisely *entering a room with no exit*. + /// + /// Found by Copilot on PR #27. Without `strands_the_user`, `Ctrl-W` + /// resolved `enter-mode save-prompt` locally — `ChangeMode(SavePrompt)`, + /// a perfectly good `Command` — and the user then typed a path into a + /// prompt whose Enter goes nowhere. `Ctrl-E` did the same into + /// `ExportPrompt` and had done since before this branch. + /// + /// The negative arm matters as much: refusing *every* `enter-mode` would + /// pass the assertions above, so a mode the client can finish is checked + /// here too. + #[test] + fn a_prompt_this_client_cannot_commit_is_never_entered() { + let state = in_mode(AppMode::Normal); + for (key, mode) in [ + ( + KeyEvent::new(KeyCode::Char('w'), KeyModifiers::CONTROL), + "save-prompt", + ), + ( + KeyEvent::new(KeyCode::Char('e'), KeyModifiers::CONTROL), + "export-prompt", + ), + ] { + assert_eq!( + resolve(&state, key), + KeyOutcome::Upstream(key), + "{mode} must not be entered by a client that cannot commit it" + ); + } + + // ...but an `enter-mode` this client can finish is still resolved + // locally, or the fix above would be "refuse every mode". `:` is the + // sharpest negative: it is another minibuffer prompt, so it goes + // through the same `entered()` path and differs only in whether the + // commit behind it is reachable. + let entered = resolve(&state, ev(KeyCode::Char(':'))); + assert!( + matches!(&entered, KeyOutcome::Local(cmds) + if cmds.iter().any(|c| matches!( + c, Command::ChangeMode(AppMode::CommandMode { .. })))), + ": must still open the command line locally, got {entered:?}" + ); + } + /// The four keys an independent browser smoke test found inert, plus the /// two the editing path needs. Each asserts the *shape* of what is now /// composed, because "resolves locally" alone would pass on a wrong From 4abcdaaa64535281418489ebcabcf15480725a13 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:55:01 -0700 Subject: [PATCH 44/72] chore(beads): close improvise-2yc2 and improvise-hkt6, file three follow-ups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closed by PR #20 (WU-K), with the reasoning on each bead rather than in a commit nobody will read again. Filed from WU-I's hand-back: improvise-kpnr improvise-ws-server has no authentication, and the reverse-proxy path the README now documents removes the loopback binding that was its only control. improvise-j7v6 the typed-string classification lives in two crates, and its guard test states the obligation in its name while asserting three outcomes by hand. Filed from WU-K's hand-back: improvise-44ad ColWidths is redundant under the label fold — tried and reverted, because the test it would rewrite is gotcha 33(c)'s standing evidence. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 87 ++++++++++++++++++++++++--------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 4d9e28ff..3001b18f 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,9 +1,9 @@ {"_type":"issue","id":"improvise-0yq8","title":"Move ClientState and keys.rs::step into improvise-protocol","description":"THE SINGLE HIGHEST-VALUE CONSOLIDATION LEFT, named by the agent that did the keymap move and measured rather than guessed.\n\ndesign-principles 5 (user ruling 2026-09-20): minimise the code a new UI framework needs; centralise logic in a redux-like store so a new front end DISPLAYS STATE rather than reimplementing behaviours. The test is: would a third front end have to write this again?\n\nkeys.rs::step is 99 LINES OF \"WHAT THIS COMMAND DOES\" and the answer is YES. It depends on NOTHING but ViewState + the protocol widget caches + the announced window -- every one of which is ALREADY A PROTOCOL TYPE. So it is in the wrong crate, and the only reason it is there is history.\n\nAFTER THIS MOVE A NEW FRONT END NEEDS EXACTLY TWO THINGS: browser-event (or terminal-event, or whatever) -\u003e KeyEvent, and painting. That is the whole of design-principles 5 made true rather than aspirational.\n\nWHY IT WAS NOT DONE IN THE SAME PASS: it is a crate move on top of a crate move (the keymap had just moved), and half-doing it would leave two homes. Deliberate, not forgotten.\n\nMEASURED CONTEXT from that pass: keys.rs grew 264 -\u003e 401 non-comment lines while GAINING the whole editing path, search typing, panels, transpose and tile select -- and the key TABLE left the crate entirely. web/app.js, the only genuinely framework-bound file, grew 344 -\u003e 359 while gaining a live editor and a hint line. So the trend is right; this ticket is what finishes it.\n\nNOTE THE SECOND-CONSUMER ARGUMENT: the more of the front end is SHARED, the more of it a second consumer can AUDIT. CellDisplay::edit_text exists because the browser asked a question the TUI had answered privately and the two answers differed -- see doctrine(being_the_second_consumer_audits_the_abstraction).","status":"open","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-oai9","title":"[bug] The worker's errors reach the console with their cause stripped","description":"FOUND BY THE FIRST REAL BROWSER RUN of the deployed page (2026-09-20), and it is the reason that run produced no actionable information.\n\nThe console said:\n\n improvise worker: failed to start\n Error: Cannot open demo.improv\n\n\"Cannot open demo.improv\" is the CONTEXT vfs::open attaches with with_context. anyhow Display prints ONLY THE OUTERMOST CONTEXT, so bindings::start mapping through e.to_string() kept the label and threw away the cause -- the Storage failure or the parse failure that actually happened.\n\nWHY THIS IS THE WORST POSSIBLE PLACE FOR IT: bindings.rs is cfg(target_arch = wasm32) and is the one module in the crate that NO TEST IN THE REPOSITORY CAN REACH. A devtools console is the only instrument pointed at it. So the single code path with no test coverage was also the one discarding its own diagnostics.\n\nFIXED: a report() helper rendering with the alternate form {:#}, which walks the chain joining with \": \". It lives in lib.rs rather than bindings.rs SO IT CAN BE TESTED AT ALL -- bindings is wasm-gated. Five call sites (start, on_message, flush, documents, refresh).\n\nTEST WRITTEN AGAINST THE OLD RENDERING FIRST: a_reported_error_carries_its_cause_not_just_its_label asserts that anyhow to_string() IS just the label (pinning the trap, so the helper stops being needed if that ever changes) and that report() contains BOTH the label and the cause.\n\nTHIS DOES NOT FIX THE UNDERLYING FAILURE -- it makes the next reload name it. Redeploy and read the console again.","status":"open","priority":1,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T03:55:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:55:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-anr9","title":"Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once","description":"Two tickets turn out to be ONE piece of work, noticed by improvise-bpq2 while walking the bindings.\n\nimprovise-s1xw (multi-level grid labels -- the browser shows Food/Alice where the TUI shows two columns) and the bindings H / z / gz need THE SAME FIVE FACTS, which gotcha 44 already enumerates as the grid frame the cache does not carry:\n row_cats / col_cats -- how many header levels\n row_items -- multi-level label TUPLES and inline group headers\n col_group_for -- column group spans\n is_group_collapsed -- the collapse indicator\n(record_col is the sixth and is CORRECT to keep off: session-local, keys pending drill edits.)\n\nSo:\n * hide-selected-row-item (H) needs to know WHICH CATEGORY a row label belongs to -- impossible while row_labels slash-joins a tuple.\n * toggle-group / gz need group membership and collapsed state.\n * And the browser renders the slash-joined form, which the TUI own test asserts must not appear.\n\nDOING THEM SEPARATELY MEANS TOUCHING row_labels TWICE. It is a co-ordinated type change either way -- improvise-wasm-client/src/render.rs consumes both row_labels and col_labels -- so the cheap moment is once.\n\nDO NOT add row_levels beside row_labels. Two answers to \"what are the labels\" is the disease (improvise-edp), and bpq2 has just spent a wave demonstrating what one shared derivation buys.\n\nSupersedes/absorbs improvise-s1xw -- decide which id survives.","notes":"USER RULING 2026-09-20: 'Most of the view functionality should be a wasm blob that's shared between the TUI and the web UI' -- clarified as ONE SHARED CRATE, not the TUI loading wasm at runtime. The view derivations live in improvise-protocol; the browser gets them as wasm, the TUI links them natively, and ONLY THE PAINTING differs (ratatui vs DOM).\n\nTHIS TICKET IS NOW THE LAST STEP OF THAT, which is why it is P1.\n\nMeasured 2026-09-20: every other widget already takes a shared cache and no \u0026Model -- FormulaContent(\u0026FormulaCache), ViewContent(\u0026ViewListCache), TileBar(\u0026TileBarCache), CategoryContent(\u0026CategoryTreeCache), WizardWidget(\u0026WizardCache), HelpWidget(usize), WhichKeyWidget(\u0026[..]). THE REMAINING DUPLICATION IS ONE FILE: src/ui/grid.rs still derives 14 things straight off layout./view., and those are exactly the six frame facts in gotcha 44.\n\nSo closing this ticket is what makes 'most of the view functionality shared' TRUE rather than nearly true, and it fixes the Food/Alice divergence and unblocks H/z/gz as the same change.\n\nSEQUENCING NOTE: this touches improvise-protocol/src/render.rs, src/ui/grid.rs AND improvise-wasm-client/src/render.rs together -- it is a co-ordinated type change on row_labels/col_labels, not an additive one. Do not start it while another agent is in improvise-protocol; the client-side keymap work (improvise-bpq2 successor) is there now.","status":"open","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:25:36Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-vvl3","title":"A headless browser harness: run storage_contract and the shell assertions under Playwright","description":"THE CONDITION THAT DEFERRED improvise-xf1 HAS BEEN MET. The user deferred browser testing on 2026-09-20 with: \"I am happy just putting off the browser test for now: WE HAVE TO HAVE SOMETHING TO TEST.\" After improvise-9x6, djm, hwpi and the demo model, we do -- a complete self-contained application whose ENTIRE untested surface is the part a harness would cover.\n\nTHE EVIDENCE THAT IT HAS FLIPPED, not just an argument that it would be nice:\n * Three of improvise-hwpi four fixes are ONE-ASSERTION tests (status text after a bad frame, status text with no peer, one repaint on commit). That ticket itself said \"consider whether the right first move is a headless DOM -- otherwise the fixes are as unverified as the bugs.\" The fixes are now shipped, so that sentence is true of them.\n * The first-frame watchdog was found BY READING, and only because someone went looking for it. The next one will not be.\n * OPFS is the specific thing MemoryFs cannot stand in for, and it is now on the critical path of the DEFAULT deployment rather than a future one. storage_contract in storage.rs exists and is written down precisely so it can be run against OpfsFs in a browser; NOTHING BUT A BROWSER CAN RUN IT.\n * Persistence across reload is the one improvise-bck acceptance criterion no test in this repo can structurally reach.\n\nSCOPE, deliberately small: python3 -m http.server --directory site, plus one Playwright spec, reusing the assemble step CI already performs. Two parts -- (a) storage_contract against OpfsFs, which is the thing improvise-xf1 is actually about; (b) three shell assertions from hwpi.\n\nShould block improvise-bck sign-off. Un-defer or close improvise-xf1 into this, whichever the user prefers -- xf1 predates the worker and is narrower than what is now needed.","status":"open","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:52Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:29:52Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-gv9r","title":"[bug] A wire model command never marks the document dirty","description":"Found by improvise-9x6 (2026-09-20) the hard way: the worker first persist trigger was `if dirty`, and it SILENTLY NEVER FIRED.\n\nreduce::model_effects maps ModelCommand::SetCell to effect::SetCell, which writes the cell and does NOT touch model_state.dirty. Dirtiness is raised by a SEPARATE MarkDirty effect that only the command/key path emits -- so on every transport, a client edit leaves the authoritative document reporting CLEAN.\n\nTwo consequences, and the second is the one that will bite someone:\n * \"Save if dirty\" is a trigger that never fires. Any host built on it loses every client edit, silently.\n * Anything else that reads model_state.dirty about a client-edited document gets the wrong answer -- an unsaved-changes prompt, an autosave, a UI marker.\n\nWHY IT HAPPENED: MarkDirty being a separate effect is deliberate and reasonable on the key path, where a command composes the pair. The derivation that turned effects into ModelCommands (improvise-kmgs) took the effects ONE AT A TIME, so the pairing was lost -- the wire has SetCell without its MarkDirty companion. That is the interesting shape: a derivation that is correct per element and wrong for a set.\n\nCANDIDATE FIXES, and this wants deciding alongside improvise-9x6 sibling ticket about Save: make model_effects emit the MarkDirty companion where the key path does; or fold dirtiness into the effects that change the document so it cannot be forgotten (the unrepresentable-not-validated answer, and probably right); or decide that dirty is a host-local concept the wire has no opinion on, and then say so where someone will read it.","status":"open","priority":1,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:10:13Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-368h","title":"[bug] No client on any transport can ask its host to save","description":"Found by improvise-9x6 (2026-09-20) while building the worker persist trigger, and it affects improvise-ws-server too -- which has shipped without anyone noticing.\n\neffect::Save is EffectKind::ViewOnly, and CORRECTLY so: it writes to THIS HOST file path and clears THIS HOST dirty flag, mutating no document state. But the wire vocabulary is DERIVED from the model effects (gotcha 38), so the derivation cannot reach it. There is no Command::Save and no ModelCommand::Save.\n\nConsequence: a connected client can edit the document and can never cause it to be written. On the socket deployment that means a ws-server session edits in memory and the file on disk never changes unless a human is at a TUI that is not there.\n\nTHIS IS A PROTOCOL DECISION, NOT A PATCH. Two coherent answers:\n * ADD Command::Save. But Save is genuinely ViewOnly and genuinely per-host -- a client asking a host to write to a path the client cannot see is a different thing from a client mutating the document, and it needs an answer for \"which path\" and \"what if there is none\" (improvise-lae already exists for the no-path case).\n * SAY EXPLICITLY THAT HOSTS OWN PERSISTENCE, and make each host trigger on its own terms. That is what improvise-9x6 did for the worker, and in a browser it is arguably RIGHT rather than a workaround: there is no file the user owns and no Ctrl-S semantics -- the document IS the OPFS entry. But then ws-server needs its own answer, and today it has none.\n\nDecide which. The gap is that neither was chosen, so the socket server silently does nothing.","status":"open","priority":1,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:10:11Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-oai9","title":"[bug] The worker's errors reach the console with their cause stripped","description":"FOUND BY THE FIRST REAL BROWSER RUN of the deployed page (2026-09-20), and it is the reason that run produced no actionable information.\n\nThe console said:\n\n improvise worker: failed to start\n Error: Cannot open demo.improv\n\n\"Cannot open demo.improv\" is the CONTEXT vfs::open attaches with with_context. anyhow Display prints ONLY THE OUTERMOST CONTEXT, so bindings::start mapping through e.to_string() kept the label and threw away the cause -- the Storage failure or the parse failure that actually happened.\n\nWHY THIS IS THE WORST POSSIBLE PLACE FOR IT: bindings.rs is cfg(target_arch = wasm32) and is the one module in the crate that NO TEST IN THE REPOSITORY CAN REACH. A devtools console is the only instrument pointed at it. So the single code path with no test coverage was also the one discarding its own diagnostics.\n\nFIXED: a report() helper rendering with the alternate form {:#}, which walks the chain joining with \": \". It lives in lib.rs rather than bindings.rs SO IT CAN BE TESTED AT ALL -- bindings is wasm-gated. Five call sites (start, on_message, flush, documents, refresh).\n\nTEST WRITTEN AGAINST THE OLD RENDERING FIRST: a_reported_error_carries_its_cause_not_just_its_label asserts that anyhow to_string() IS just the label (pinning the trap, so the helper stops being needed if that ever changes) and that report() contains BOTH the label and the cause.\n\nTHIS DOES NOT FIX THE UNDERLYING FAILURE -- it makes the next reload name it. Redeploy and read the console again.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:55:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-anr9","title":"Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once","description":"Two tickets turn out to be ONE piece of work, noticed by improvise-bpq2 while walking the bindings.\n\nimprovise-s1xw (multi-level grid labels -- the browser shows Food/Alice where the TUI shows two columns) and the bindings H / z / gz need THE SAME FIVE FACTS, which gotcha 44 already enumerates as the grid frame the cache does not carry:\n row_cats / col_cats -- how many header levels\n row_items -- multi-level label TUPLES and inline group headers\n col_group_for -- column group spans\n is_group_collapsed -- the collapse indicator\n(record_col is the sixth and is CORRECT to keep off: session-local, keys pending drill edits.)\n\nSo:\n * hide-selected-row-item (H) needs to know WHICH CATEGORY a row label belongs to -- impossible while row_labels slash-joins a tuple.\n * toggle-group / gz need group membership and collapsed state.\n * And the browser renders the slash-joined form, which the TUI own test asserts must not appear.\n\nDOING THEM SEPARATELY MEANS TOUCHING row_labels TWICE. It is a co-ordinated type change either way -- improvise-wasm-client/src/render.rs consumes both row_labels and col_labels -- so the cheap moment is once.\n\nDO NOT add row_levels beside row_labels. Two answers to \"what are the labels\" is the disease (improvise-edp), and bpq2 has just spent a wave demonstrating what one shared derivation buys.\n\nSupersedes/absorbs improvise-s1xw -- decide which id survives.","notes":"USER RULING 2026-09-20: 'Most of the view functionality should be a wasm blob that's shared between the TUI and the web UI' -- clarified as ONE SHARED CRATE, not the TUI loading wasm at runtime. The view derivations live in improvise-protocol; the browser gets them as wasm, the TUI links them natively, and ONLY THE PAINTING differs (ratatui vs DOM).\n\nTHIS TICKET IS NOW THE LAST STEP OF THAT, which is why it is P1.\n\nMeasured 2026-09-20: every other widget already takes a shared cache and no \u0026Model -- FormulaContent(\u0026FormulaCache), ViewContent(\u0026ViewListCache), TileBar(\u0026TileBarCache), CategoryContent(\u0026CategoryTreeCache), WizardWidget(\u0026WizardCache), HelpWidget(usize), WhichKeyWidget(\u0026[..]). THE REMAINING DUPLICATION IS ONE FILE: src/ui/grid.rs still derives 14 things straight off layout./view., and those are exactly the six frame facts in gotcha 44.\n\nSo closing this ticket is what makes 'most of the view functionality shared' TRUE rather than nearly true, and it fixes the Food/Alice divergence and unblocks H/z/gz as the same change.\n\nSEQUENCING NOTE: this touches improvise-protocol/src/render.rs, src/ui/grid.rs AND improvise-wasm-client/src/render.rs together -- it is a co-ordinated type change on row_labels/col_labels, not an additive one. Do not start it while another agent is in improvise-protocol; the client-side keymap work (improvise-bpq2 successor) is there now.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:26Z","started_at":"2026-09-20T05:32:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-vvl3","title":"A headless browser harness: run storage_contract and the shell assertions under Playwright","description":"THE CONDITION THAT DEFERRED improvise-xf1 HAS BEEN MET. The user deferred browser testing on 2026-09-20 with: \"I am happy just putting off the browser test for now: WE HAVE TO HAVE SOMETHING TO TEST.\" After improvise-9x6, djm, hwpi and the demo model, we do -- a complete self-contained application whose ENTIRE untested surface is the part a harness would cover.\n\nTHE EVIDENCE THAT IT HAS FLIPPED, not just an argument that it would be nice:\n * Three of improvise-hwpi four fixes are ONE-ASSERTION tests (status text after a bad frame, status text with no peer, one repaint on commit). That ticket itself said \"consider whether the right first move is a headless DOM -- otherwise the fixes are as unverified as the bugs.\" The fixes are now shipped, so that sentence is true of them.\n * The first-frame watchdog was found BY READING, and only because someone went looking for it. The next one will not be.\n * OPFS is the specific thing MemoryFs cannot stand in for, and it is now on the critical path of the DEFAULT deployment rather than a future one. storage_contract in storage.rs exists and is written down precisely so it can be run against OpfsFs in a browser; NOTHING BUT A BROWSER CAN RUN IT.\n * Persistence across reload is the one improvise-bck acceptance criterion no test in this repo can structurally reach.\n\nSCOPE, deliberately small: python3 -m http.server --directory site, plus one Playwright spec, reusing the assemble step CI already performs. Two parts -- (a) storage_contract against OpfsFs, which is the thing improvise-xf1 is actually about; (b) three shell assertions from hwpi.\n\nShould block improvise-bck sign-off. Un-defer or close improvise-xf1 into this, whichever the user prefers -- xf1 predates the worker and is narrower than what is now needed.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:52Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-gv9r","title":"[bug] A wire model command never marks the document dirty","description":"Found by improvise-9x6 (2026-09-20) the hard way: the worker first persist trigger was `if dirty`, and it SILENTLY NEVER FIRED.\n\nreduce::model_effects maps ModelCommand::SetCell to effect::SetCell, which writes the cell and does NOT touch model_state.dirty. Dirtiness is raised by a SEPARATE MarkDirty effect that only the command/key path emits -- so on every transport, a client edit leaves the authoritative document reporting CLEAN.\n\nTwo consequences, and the second is the one that will bite someone:\n * \"Save if dirty\" is a trigger that never fires. Any host built on it loses every client edit, silently.\n * Anything else that reads model_state.dirty about a client-edited document gets the wrong answer -- an unsaved-changes prompt, an autosave, a UI marker.\n\nWHY IT HAPPENED: MarkDirty being a separate effect is deliberate and reasonable on the key path, where a command composes the pair. The derivation that turned effects into ModelCommands (improvise-kmgs) took the effects ONE AT A TIME, so the pairing was lost -- the wire has SetCell without its MarkDirty companion. That is the interesting shape: a derivation that is correct per element and wrong for a set.\n\nCANDIDATE FIXES, and this wants deciding alongside improvise-9x6 sibling ticket about Save: make model_effects emit the MarkDirty companion where the key path does; or fold dirtiness into the effects that change the document so it cannot be forgotten (the unrepresentable-not-validated answer, and probably right); or decide that dirty is a host-local concept the wire has no opinion on, and then say so where someone will read it.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:28Z","started_at":"2026-09-20T05:32:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-368h","title":"[bug] No client on any transport can ask its host to save","description":"Found by improvise-9x6 (2026-09-20) while building the worker persist trigger, and it affects improvise-ws-server too -- which has shipped without anyone noticing.\n\neffect::Save is EffectKind::ViewOnly, and CORRECTLY so: it writes to THIS HOST file path and clears THIS HOST dirty flag, mutating no document state. But the wire vocabulary is DERIVED from the model effects (gotcha 38), so the derivation cannot reach it. There is no Command::Save and no ModelCommand::Save.\n\nConsequence: a connected client can edit the document and can never cause it to be written. On the socket deployment that means a ws-server session edits in memory and the file on disk never changes unless a human is at a TUI that is not there.\n\nTHIS IS A PROTOCOL DECISION, NOT A PATCH. Two coherent answers:\n * ADD Command::Save. But Save is genuinely ViewOnly and genuinely per-host -- a client asking a host to write to a path the client cannot see is a different thing from a client mutating the document, and it needs an answer for \"which path\" and \"what if there is none\" (improvise-lae already exists for the no-path case).\n * SAY EXPLICITLY THAT HOSTS OWN PERSISTENCE, and make each host trigger on its own terms. That is what improvise-9x6 did for the worker, and in a browser it is arguably RIGHT rather than a workaround: there is no file the user owns and no Ctrl-S semantics -- the document IS the OPFS entry. But then ws-server needs its own answer, and today it has none.\n\nDecide which. The gap is that neither was chosen, so the socket server silently does nothing.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:28Z","started_at":"2026-09-20T05:32:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ondj","title":"[bug] reduce_full/apply_command apply a Command::Model without the basis check","description":"Found by the code review of ADR 001 layer 1 (2026-09-19). CONFIRMED, not live today, and it should be closed BEFORE the browser worker lands.\n\nui/reduce.rs is documented as \"the one place a command is applied to the authoritative state\" and \"Every deployment mode -- the native TUI, a websocket server, a browser worker -- reaches the model through the functions here\". Both pub fn apply_command and pub fn reduce_full accept a \u0026Command including Command::Model, and:\n\n Command::Model { cmd, .. } =\u003e { // the basis is DISCARDED by `..`\n let effects = model_effects(app, cmd);\n app.apply_effects(effects)\n }\n\nThe check lives ONLY in Sessions::apply_model.\n\nWHY THIS IS THE INTERESTING SHAPE: making Command::Model a struct variant made the basis impossible to OMIT FROM THE MESSAGE. It did nothing to make it impossible to IGNORE ON ARRIVAL -- and `..` in a pattern is exactly the syntax for ignoring it, with no compile error. So the design got half of what a struct variant is usually reached for.\n\nNothing in-tree routes a client command through reduce_full (only tests), so nothing is broken now. But improvise-9x6 (the browser worker) is precisely the host that will reach for the function the module doc points it at, and it would apply client-supplied model commands unchecked.\n\nREMEDIES, pick deliberately:\n * Have apply_command take a ModelCommand from a private caller and expose only Sessions::apply for a wire Command -- makes the door unreachable rather than guarded.\n * Or have reduce_full return a Rejected-shaped result for Command::Model.\n * At absolute minimum, a compile-visible note AND a test that names the door, so the next person meets it.\n\nThe first is the one that matches this repo: invariants are made unrepresentable, not validated (doctrine(an_invariant_is_made_unrepresentable_not_validated)).","status":"closed","priority":1,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:54:29Z","closed_at":"2026-09-20T01:54:29Z","close_reason":"Fixed as option 1 -- make the door unreachable -- but as a TYPE rather than as visibility.\n\napply_command and reduce_full now take Checked\u003c'a\u003e, a private-field newtype over a \u0026Command whose ONLY constructor is Checked::against(\u0026App, \u0026Command) -\u003e Result\u003cSelf, Rejection\u003e, which is also the only caller of stale_basis. So there is no longer a spelling of 'apply this batch and ignore what it claims': the that discarded the basis is now a Rejection the caller must handle. Sessions::apply answers ONCE, before the dispatch, and hands the proof to apply_switch/apply_model -- so neither has to remember to ask, and apply_model lost its basis/model parameters entirely.\n\nWHY NOT pub(crate), which is what the ADR literally suggested: it works, but reduce_full then has NO non-test caller anywhere in the workspace, so it becomes a dead_code warning in the non-test build, and the only ways out are deleting a documented host API or cfg(test)-gating it. The newtype keeps both functions public and useful to a future host -- including the browser worker, which now gets a door that WORKS and is safe -- while making the unchecked path unconstructible. It also holds INSIDE the crate, which visibility does not: a second in-crate mutation path cannot be written without the check either.\n\nEVERYTHING THE TICKET SAID MUST SURVIVE, DOES: the check is still a property of the session door and not of every mutation (the key path, model_effects and App::apply_effects are untouched, and a host's own ModelCommand still needs no basis); caught_up(\u0026App, cmd) is unchanged and still not a bypass; the 10 basis_check tests are untouched and all pass.\n\nTHE REGRESSION GUARD IS THE SIGNATURE. The demonstration test (committed first, failed on the old code) NO LONGER COMPILES after the fix, which is the better outcome. What survives asserts that Checked::against returns Err(Rejection::LayoutMoved) and the document is untouched, with a doc-comment saying in prose that the signature is the real guard. Check-the-checker: making against() never refuse fails 8 tests -- the new one plus 7 of the 10 basis_check ones.\n\nNATIVE CLI EXERCISED FOR REAL, not merely compiled: improvise cmd and improvise script both land their changes on disk. They go through the keymap/registry path and never touched reduce_full.\n\nTHE THING THE TICKET DID NOT PREDICT, and it is the reason this defect could sit unnoticed: reduce_full HAS NO PRODUCTION CALLER IN THE ENTIRE WORKSPACE, and apply_command's view-slice arm is unreached outside tests. Every host reaches the model through Sessions::apply or reduce_key. So ui/reduce.rs's 'the signature the three hosts share' is ASPIRATIONAL -- a door pointed at by documentation and used by nobody, which is exactly the shape that hides a defect until the next host arrives. Filed as a successor.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-xks6","title":"One home for the client/server frame envelopes","description":"Two crates now define the same frames independently, which is a live divergence risk:\n improvise-wasm-client::wire -- ClientMessage / ServerMessage\n improvise-ws-server::wire -- ClientMsg / ServerMsg\nThe ws-server agent asked for one version to be landed and will delete its wire.rs and re-export; the change on its side is two type names.\n\nPUT THEM IN improvise-protocol. Both sides already depend on it, and this is the one thing there that is FRAMING rather than vocabulary -- the protocol says what a Command is, but not what distinguishes one JSON blob from another on the wire.\n\nTHE SHAPE, as the ws-server landed it and the client should adopt:\n client -\u003e server: Viewport { rows: usize, cols: usize } | Command(Command)\n server -\u003e client: Snapshot(Box\u003cSnapshot\u003e) | Commands(Vec\u003cCommand\u003e)\nNotes that matter: Snapshot is BOXED because clippy large_enum_variant fires otherwise and a Snapshot is sent once per connection while Commands is every frame. Server-\u003eclient is ONE shape carrying both projections (as Command::Project) and a peer broadcast, because on the wire they are the same thing and a client folds both -- a separate Projections variant was rejected for that reason.\n\nViewport is the only addition to the shared vocabulary and it is derived, not invented: the extent is the one session input no Command carries by construction (gotcha 40).","notes":"RAISED TO P1 AND SEQUENCED AHEAD OF improvise-9x6 (2026-09-20).\n\nThere are now THREE envelope vocabularies for TWO deployments:\n * improvise-ws-server::wire -- ClientMsg/ServerMsg, the socket, server side\n * improvise-wasm-client::socket -- ClientFrame/ServerFrame, the same socket MIRRORED, because a native crate cannot be a wasm dependency\n * improvise-wasm-client::wire -- ClientMessage/ServerMessage, the WORKER deployment (improvise-tm6), currently unused by anything\n\nimprovise-9x6 gives the worker its own Host. If it lands before this ticket, it will speak a FOURTH arrangement and consolidation gets harder rather than easier -- so this comes first, or is at least DECIDED first.\n\nThe answer is almost certainly improvise-protocol owning the framing, since both ends already depend on it and its charter is exactly 'the wire vocabulary'. The open question this ticket must actually answer is whether the WORKER and the SOCKET want the SAME vocabulary. They differ today in ways that may be essential or may be accident: the socket has Viewport and no Key; the worker envelope was written before the projection architecture landed. Decide that before moving either, or the move produces two homes instead of one.\n\nThe mirror is currently held honest by two tests (gotcha 42) -- the_socket_framing_matches_the_server and every_server_variant_is_one_this_crate_mirrors. Both DELETE when this lands, and that is the measure of success.","status":"closed","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T23:46:23Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:52:29Z","closed_at":"2026-09-20T01:52:29Z","close_reason":"Done. Three envelope vocabularies for two deployments become ONE pair: improvise-protocol::frame owns ClientFrame { Viewport, Command, Key } and ServerFrame { Snapshot, Commands(Vec\u003c_\u003e), Rejected }; improvise-ws-server and improvise-wasm-client each pub use them and neither defines one. Both wire.rs files DELETED.\n\nTHE QUESTION THE TICKET SAID TO DECIDE FIRST -- do the worker and the socket want the same vocabulary -- IS ANSWERED YES, and the test applied to each difference was whether the ARGUMENT for it names the thing they differ in. None did:\n * Viewport is deployment-independent. Its justification is that the extent is a fact about a terminal or a browser window, is not in ViewState, and no effect produces it, so no derivation reaches it. Nothing transport-shaped. Its handshake role is a property of GridCache::seed (a Snapshot seeds the VISIBLE cells), not of sockets. A worker hosting the same Sessions owes set_extent the same answer.\n * Key is deployment-independent. Its justification (gotcha 38) is that a binding names an INTENT resolved against a CmdContext a thin client does not hold -- which says nothing about what the bytes travel over. Decisive evidence: the wasm client ALREADY PRODUCES a Key frame on both paths and tests/lockstep.rs round-trips it. So the socket's 'no Key' was never a statement about the socket, it was an unfinished feature (improvise-bpq2) that the separate envelope made invisible.\n * Commands(Vec\u003c_\u003e) vs Command(_): the batching is a property of the FOLD, not the transport -- a fan-out must arrive whole and in sequence because a cell update that overtakes its labels is DROPPED (gotcha 33), and an in-process postMessage interleaves exactly as a socket does. Commands(vec![c]) expresses everything the singleton could; the reverse is false.\n\nAND THE WORKER PAIR WAS INCOHERENT, NOT MERELY DIFFERENT -- the strongest evidence its differences were staleness. Written before ADR 001, it could carry Command::Model { basis, .. } and had NO Rejected variant, so a client speaking it composed a claim that could never be answered. It was missing a variant its own payload type demands.\n\nA FIND THE TICKET DID NOT PREDICT: the worker envelope was NOT 'unused by anything'. It was the return type of Client::set_cell/set_cell_at/key/on_key_event and the input to apply/handle_message/on_message. The wasm client was carrying TWO PARALLEL FOLD SURFACES OVER ONE STATE, identical except that the worker one had no Rejected arm. Once the envelopes unified they were literal duplicates, so the worker trio is deleted; web/app.js calls neither.\n\nBOTH MIRROR TESTS DELETED, which was the stated measure of success. What they checked is now TYPE IDENTITY: a rename is a compile error at every call site in both crates, and 'a variant added on one side and not the other' is not a constructible state. Strictly stronger than a hand-written fixture list. The check-the-checker MOVES rather than dies -- the_retired_spellings_do_not_parse keeps the injected-fault discipline and aims it at the one thing still possible, the retired Bootstrap/Command spellings returning as serde ALIASES, which would be two homes again wearing one type. End-to-end compatibility is now browser_round_trip.rs handing the wasm client's own serialized frame to a real Host::handle.\n\nHost::handle gains an explicit ClientFrame::Key =\u003e Vec::new() arm. Behaviour unchanged -- the frame previously failed to parse and was skipped -- but the gap is now A LINE OF CODE WITH improvise-bpq2 ON IT rather than a variant that does not exist.\n\n1134 -\u003e 1132 passing (5 deleted, 3 added), fully accounted for. Clippy, fmt, wasm32 all clean.\n\nWHAT improvise-9x6 CAN NOW ASSUME: nothing to define and no fourth arrangement to invent; first inbound frame is ClientFrame::Viewport for the same reason; ClientFrame::Key is already on the wire and already produced by the client, and a worker holds the authoritative App so it can resolve it through reduce_key where the socket cannot -- the worker may close bpq2's mechanism first; ServerFrame::Commands must be delivered whole and in order across a MessagePort; ServerFrame::Rejected must be answered, since ADR 001 is not socket-only.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-bpq2","title":"Client-side key resolution: what must the projections carry?","description":"USER RULING (2026-09-19): keys are converted into commands IN THE BROWSER; the keymap is a client-only concept the server must not know. Follows from the standing principle -- the model is shared, the UI is not, and a keymap is UI. It also buys something real: two clients may have DIFFERENT bindings over the same document, which is impossible if the server owns the keymap.\n\nThis inverts improvise-gsw conclusion (which had keys travelling upstream unresolved) and closes the keypress-door ticket. The work is now to make client-side resolution POSSIBLE.\n\nTHE PROBLEM TO SOLVE. A binding names an INTENT resolved against a CmdContext -- Model, GridLayout, cursor, yanked cell, expanded cats, mode. A Command is the RESOLVED MUTATION, e.g. SetSelected { row, col }. A thin client holds ViewState plus the widget caches. So for each binding ask: can a client holding only that state compose the Command? Where the answer is no, the fix is to ENRICH THE PROJECTION, not to move resolution to the server.\n\nKNOWN GAPS, from the gsw agent own analysis of what it could not resolve:\n * CURSOR MOVEMENT -- GridCache is indexed by position WITHIN THE VIEWPORT, so a client cannot distinguish \"last visible row\" from \"last row\" and cannot clamp. It needs the total row/col count, or an explicit \"can I move further\" fact.\n * PANEL CURSOR MOVEMENT -- move-panel-cursor walks a flattening that depends on expanded_cats. Either the cache carries the flattened list, or the client re-derives it -- and re-deriving is how two ends drift.\n * MODIFIED CHARACTERS -- KeyPattern::AnyChar matches a Char under ANY modifiers on the host. A client mirroring that is fine now that the client owns the keymap outright; the old constraint (local resolution may only ever be a strict subset) DISSOLVES once there is no host keymap to be a subset of.\n\nMETHOD: enumerate the bindings, not the commands -- the keymap is the thing being made client-side, so walk it. doctrine(binding_dump_diff) already says most bindings have no direct test, so a green suite is not evidence here either.\n\nALSO IN SCOPE: move ClientMessage/ServerMessage into improvise-protocol. The vocabulary is there and the framing that says which a JSON blob is, is not; the ws-server will want the same envelopes and two crates should not define them.\n\nBlocks improvise-1ey.","notes":"\n--- THE COMPOSABILITY LIST (improvise-q08 agent, 2026-09-19) ---\nWhich Commands a thin client holding ViewState + the five widget caches CANNOT compose. This is the real content of this ticket.\n\nHARD BLOCKERS:\n1. SetCell { key: CellKey } and ClearCell(CellKey) -- THE BIG ONE, and it blocks\n the most basic action there is. A client sees a grid of LABEL STRINGS indexed\n by viewport position. A CellKey is the full sorted coordinate tuple: every Row\n and Column category, plus every Page AND Filter selection (gotcha 27). Nothing\n in GridCache says which category a row label belongs to; nested row axes\n flatten several categories into one label; the page/filter constraints are not\n projected at all; and records mode encodes a COLUMN INDEX in _Dim (gotcha 1).\n There is no client-side derivation. NOTE THIS BLOCKS improvise-1ey DIRECTLY --\n its stated scope is \"type a number into a cell and see the value update\".\n2. ToggleGroup { category, group } and AddItemInGroup -- CategoryRow.items is a\n flat Vec\u003cString\u003e; groups, group membership and collapsed state are not\n projected. The server does the cat_tree flattening.\n3. RemoveFormula { target, target_category } -- FormulaCache is Vec\u003cString\u003e of RAW\n text. The client would have to re-parse to get target (dragging\n improvise-formula into a thin client), and target_category is simply absent, so\n a formula written [CustomCat] cannot be addressed at all.\n4. LoadModel(PathBuf) / ImportJsonHeadless -- paths name documents in the HOSTs\n Storage (gotcha 2) and nothing projects a listing, so a client cannot compose a\n path it has reason to believe exists.\n5. ApplyAndClearDrill / AcceptWizardModel -- nullary so constructible, but nothing\n projects drill or wizard state, so a client cannot know when either is\n meaningful. (Both are already off the wire per improvise-3x3u.)\n\nCOMPOSABLE BUT TRAPPED:\n* SetSelected { row, col } is ABSOLUTE; GridCache is VIEWPORT-RELATIVE. The client\n must add row_offset/col_offset from its own ViewState -- it has them, but the two\n index spaces are not distinguished by any type and getting it wrong is silent.\n* The grids TOTAL row/column count is not projected, so a client cannot clamp its\n cursor or size a scrollbar.\n\nFINE TODAY: SetAxis/CycleAxis/TransposeAxes/TogglePruneEmpty/SortData/\nCleanEmptyRecords, AddCategory/RemoveCategory, AddItem/RemoveItem/HideItem/\nShowItem/SetPageSelection WHEN DRIVEN FROM THE CATEGORY PANEL (not from the grid\ncursor -- same reason as 1), CreateView/DeleteView/SwitchView/ViewBack/ViewForward,\nAddFormula, SetNumberFormat (though the CURRENT format is in no cache, so it cannot\nbe pre-filled), and the whole view half.\n\nSUGGESTED ORDER: item 1 first and alone -- it is what 1ey needs and it is a\nprojection design question (how does a client name a cell?), not a plumbing one.\n\n--- ADR 002 ACCEPTED (user, 2026-09-19): see context/adr/002-naming-a-cell.md ---\nA projected cell carries an OPAQUE SURROGATE KEY, and its ABSENCE means the\ncell is an aggregate:\n CellDisplay { text, numeric, error, key: Option\u003cCellId\u003e }\nSome -\u003e a stored cell this client may set, named by a token it echoes back.\nNone -\u003e an aggregate: drill, do not set.\n\nWHY THE NULLABILITY IS THE POINT: a client cannot compose SetCell for an\naggregate because there is NO KEY TO PUT IN IT. That is make-invalid-states-\nunrepresentable (design-principles 3), strictly stronger than a server-side\nrejection. It also mirrors a fork the model ALREADY makes -- EditOrDrill in\ncmd/mode.rs branches on exactly this -- so the wire projects an existing\ndistinction rather than inventing one. And it follows the precedent\nCellDisplay::numeric set, whose doc says the server decides \"because it is the\nside that knows\".\n\n_INDEX WAS CONSIDERED AND REJECTED, and the reason is worth keeping: right\nontology, unsafe representation. CategoryKind::VirtualIndex is documented as\n\"Items are 0, 1, ... N where N = number of matching cells\" -- POSITIONAL over\nthe currently matching set, not an identity. _Index = \"3\" names a different\nrecord after a filter change, an insert or a delete, and a different one again\nin another view. A client echoing a stale _Index writes to the WRONG CELL,\nSILENTLY -- worse than a surrogate that fails to resolve, and the same hazard\nclass as viewport-relative-vs-absolute and the stale cursor under a renamed key.\n\nBUT KEEP WHAT IT POINTS AT: the surrogate should name the RECORD, not the cell.\nA record is the coord tuple minus _Measure (record_row_identity), which is\nalready ADR 001 unit of atomicity and already what record_versions is keyed by.\nAddress a cell as (record surrogate, measure) and ONE IDENTITY serves both\naddressing and staleness, instead of each ADR inventing its own.\n\nOPEN, for the implementer to decide with evidence: hash vs interned id (the\ncodebase interns Symbols already); how the server resolves a surrogate back --\nnote that REGENERATING FROM THE CURRENT LAYOUT means a stale surrogate simply\nfails to resolve, which IS ADR 001 stale-basis rejection at cell granularity,\nso the two compose; and what DRILL addresses with, which this does not answer.\n\n--- SURROGATE DECIDED (user, 2026-09-19): hash of the normalized record, maintained as an index ---\nNOT a hash of the cell -- of the RECORD (the coord tuple minus _Measure), which\nis already ADR 001 unit of atomicity and already what record_versions is keyed\nby. A cell is (record hash, measure).\n\nWHY: it makes stale edits cheap to identify, three ways at once. The surrogate\na client already holds IS the lookup key for its own version, so record_versions\nbecomes HashMap\u003cRecordHash, u64\u003e -- fixed size, no allocation, no CellKey\nreconstruction on the hot path. A basis is a list of (record hash, version)\npairs: compact on the wire, O(1) to check. And a record that no longer exists\nsimply fails to resolve, so that whole class of staleness needs no version\ncomparison at all.\n\nIt is well-defined because CellKey ALREADY has a canonical form -- CellKey::new\nsorts by category name and equality/hashing rely on it (gotcha 5). Normalized\nmeans that form with _Measure removed; the normalization is a property the type\nguarantees, not a new convention.\n\nRESOLUTION IS A MAINTAINED INDEX, IN DataStore -- a hash is one-way, and\nregenerating from the layout would be O(visible cells) per resolve AND would\nonly cover the visible window, while a client may hold a surrogate for a record\nthat scrolled away. DataStore is the right home: it already interns every name\ninto a SymbolTable and already maintains HashMap\u003c(Symbol,Symbol), HashSet\u003cInternedKey\u003e\u003e\nfor hot-path lookups. The RecordHash index is the same shape, updated at the same\npoints, and that existing (Symbol,Symbol) index is what makes INVALIDATION\ntractable -- when a category or item disappears, the records mentioning it are\nfindable without a scan.\n\nTWO THINGS THAT FALL OUT: the index and record_versions want to be ONE structure\n(RecordHash -\u003e (record, version)) since both are keyed the same and invalidated\ntogether, giving both ADRs their answer in one lookup. And structural vs data\nstaleness separate cleanly: does not resolve = the record is GONE (reject,\nnothing to write to); resolves with a differing version = the record CHANGED\n(reject as stale, offer force). Regenerating from the layout would have\nconflated those, because a record scrolled out of view would look identical to\none that no longer exists.\n\nTHREE SILENT-BREAKAGE TRAPS, each producing a WRONG WRITE rather than an error:\n * NOT DefaultHasher / RandomState -- randomly seeded PER PROCESS, so hashes\n would differ across a restart and between processes.\n * NOT #[derive(Hash)] as the definition -- Rust guarantees no stability across\n compiler versions or platforms. Hash an explicit BYTE SERIALIZATION of the\n sorted (category, item) pairs.\n * Pick the WIDTH BY CONSEQUENCE: a collision is a silent write to the wrong\n record, which is exactly why _Index was rejected. 64 bits hits birthday risk\n near 2^32 records; 128 bits costs 8 more bytes per VISIBLE cell (~8KB on a\n 50x20 viewport). Argue it, do not default to u64 for convenience.\n\n--- CellId IS A TWO-VARIANT ENUM (user, 2026-09-19), not Option ---\n enum CellId { Record(RecordHash), Aggregate(AggregateHash) }\nSetCell takes a Record and nothing else; Drill takes an Aggregate and nothing\nelse. An aggregate surrogate is a hash of its COORDINATE CONSTRAINT -- the row\nand column coordinates plus page/filter selections -- so it costs one hash per\ncell, the same width and the same DataStore index as a record.\n\nWHY NOT Option\u003cCellId\u003e with None meaning aggregate: an aggregate is NOT the\nabsence of an identity, it is a different KIND of identity. Option made SetCell\nsafe and left Drill with nothing to address, which is exactly how drill\naddressing became an open question. Two variants close it by construction and\nmake BOTH operations unrepresentable-if-wrong rather than just one.\n\nDEFERRED, filed separately: shipping a bounded keyset per aggregate for hover\npreviews. The unbounded version is untenable -- the payload would scale with the\nDATA while the rest of a projection scales with the VIEWPORT, ~16MB to render\n1000 cells over a million-record document -- and drill-on-demand works without\nit.\n\n--- ADR 002 SPINE LANDED (2026-09-19). What remains here is the KEY RESOLUTION half. ---\nMerged: RecordHash/AggregateHash, the record index in DataStore, CellId on\nCellDisplay, SetCell/ClearCell taking a surrogate, and Client::set_cell /\nset_cell_at in the wasm client. 1017 tests, 16 faults injected and all fired.\n\nSO A CLIENT CAN NOW COMPOSE A WRITE -- improvise-cr3 can call set_cell_at\ntoday. What is NOT done, and is this ticket own subject: WIRING IT TO A\nKEYSTROKE. The wasm client still forwards editing keys upstream unresolved,\nwhich is exactly the thing the user ruling (keymap is client-only) says must\nchange. The composability list earlier in these notes is still the map; item 1\n(SetCell) is now answered, items 2-5 are not.\n\nDECISIONS THE IMPLEMENTER MADE THAT ARE WORTH KNOWING:\n * The wire form of a surrogate is a 32-char HEX STRING, not a number. serde_json\n writes u128 as a bare number and a JavaScript host parses every JSON number as\n f64, so anything above 2^53 comes back CHANGED. This trap was not in the ADR.\n * CellId::Record carries RecordCell { record: RecordHash, measure: Option\u003cString\u003e },\n not a bare hash. The ADR says both \"hash the record\" and \"a cell is (record\n hash, measure)\"; both cannot hold if the hash covers the measure, because then\n record_versions could not be keyed by it -- which is the reason the ADR chose\n the record over the cell. Carrying the measure beside the hash satisfies both.\n * id: None is NOT \"aggregate\" (that is Some(CellId::Aggregate)). It means no\n projected cell: a padded/invalidated GridCache slot, or a records-mode\n COORDINATE column, which renames across the whole row (gotcha 1) and is a\n different intent from SetCell.\n * A zero cell count is a TOMBSTONE, not a removal -- dropping the entry would let\n a delete-and-recreate reset the version to 0, so a client holding version 0\n against the OLD record would be told its basis is current and its write\n applied. resolve_record filters on liveness; record_version reads through.\n * Resolution is INDEX FIRST, LAYOUT SECOND. The index covers every record holding\n data on screen or not; the layout answers cells with NO DATA YET -- and typing\n into an empty cell IS improvise-1ey.","status":"closed","priority":1,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:42:49Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:36:27Z","closed_at":"2026-09-20T02:36:27Z","close_reason":"Done for the cursor half, with a named list of what still needs the host so closing the rest has to come here and say so.\n\nTHE FINDING THAT REFRAMED THE TICKET: Host::handle answers ClientFrame::Key with an EXPLICIT EMPTY VEC, so an upstream key is a BLACK HOLE, not a slow path. Combined with the improvise-9x6 finding (nothing in the projection stream carries the host's ViewState, so host-side resolution DIVERGES), every KeyOutcome::Upstream was a binding that does nothing at all in a browser. That turns the ticket's measure into simply 'how much of the keymap can the client answer'.\n\nMETHOD WAS THE TICKET'S: walked KeymapSet::default_keymaps() -- all 16 modes, ~170 bindings -- via a throwaway dump test, per doctrine(binding_dump_diff). Resolved client-side now: hjkl/arrows, 0/$/Home/End, G, PageUp/Down, Ctrl-d/Ctrl-u, panel cursor, tile cursor. ~15 more need NO enrichment and were deliberately left rather than half-done.\n\nTHREE PROTOCOL ADDITIONS, each argued as the smallest thing:\n * GridExtent { rows, cols } via GridUpdate::Extent -- the ticket's gap 1. Deliberately NOT framed by the labels: it is the size of the grid the labels are a window ONTO, so fit_to_labels leaves it alone and its batch position is not load-bearing (it addresses no position, so it can neither overtake nor be overtaken). The producer reads GridLayout::row_count()/col_count(), never the projected ranges -- reading it off the ranges is INVISIBLE IN THE TUI, which builds at offsets zero, and is exactly the browser bug.\n * ClientFrame::Command -\u003e Commands(Vec\u003cCommand\u003e). FORCED: one cursor key is SetSelected plus up to two scroll offsets, and a host that folds the selection but not the offset projects the wrong window. Same argument ServerFrame::Commands already carried.\n * ClientState::window (session-local) + Client::set_viewport. NOT DERIVABLE: row_labels.len() is the window CLIPPED to the grid, which equals the window everywhere except the bottom edge -- exactly where the scroll arithmetic matters. Nothing added to ViewState; both gating tests hold; transient_prefix untouched.\n\nTWO CORRECTIONS TO THE TICKET'S OWN GAP LIST:\n * Panel cursor movement WAS NEVER A GAP. improvise-edp already made viewmodel::flatten_cat_tree the one home both ends call, so calling it is the SHARED derivation, not a second one. Nobody had noticed. Worth checking other old 'known gaps' against what has landed since.\n * The unmodified-characters rule was MIS-SCOPED, not wrong. It was about KeyPattern::AnyChar, which is bound ONLY in the text-entry modes; in Normal there is no such fallback, so Ctrl-d has an exact binding and mirroring it is a faithful subset. Kept where it was earned, dropped where it was never true.\n\na_key_resolved_on_the_host_moves_state_no_client_can_hear STILL HOLDS and was not flipped -- it states a true property of host-side resolution, which bpq2 now AVOIDS by construction rather than contradicts.\n\nEach resolver mirrors its Cmd LINE FOR LINE, including where siblings disagree about a no-op move (Move and MoveTileCursor emit anyway; MovePanelCursor does not). Unifying them would make the client a second derivation, which is what lockstep testing exists to rule out.\n\nSIX INJECTED FAULTS, ALL FIRED. The five new lockstep tests build the client's cache with the server's OWN grid_of, drive client and real reduce_key over the same keypresses, and compare ViewState. nothing_in_a_non_text_mode_resolves_locally became FALSE and was replaced by a named list of the Normal-mode keys that still need the host.\n\nSTILL NEEDS THE HOST, truthfully: all prefix keys (transient_prefix is an address, not a keymap -- gotcha 13, left intact); save/wq/commit-export (Storage by definition); : execute-command; the wizard's Any binding (WizardCache has no Projection by design); CommandMode Backspace; enter-advance (its AbortChain is a fact about the effect chain, not a cursor); records/drill toggles (session-local residue).\n\nOrchestrator applied the one line the agent flagged and could not make: web/app.js now calls client.set_viewport rather than the associated Client.viewport, without which the client never learns its window and every cursor key vanishes upstream.\n\n1172 passing, clippy/fmt clean, wasm32 clean on all seven bundle crates.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -36,15 +36,16 @@ {"_type":"issue","id":"improvise-km8","title":"1.2 Fix Cargo.toml metadata","description":"Add missing package fields: description, repository, homepage, documentation, readme, keywords, categories, license. Description: 'Terminal pivot-table modeling in the spirit of Lotus Improv'. Keywords: tui, pivot, spreadsheet, data, improv. Categories: command-line-utilities, visualization.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:48Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:20:17Z","close_reason":"Metadata already present: description, license, repository, homepage, readme, keywords, categories all populated.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-902","title":"1.1 Audit and remove or update context/SPEC.md","description":"Compare context/SPEC.md against actual code. If spec contradicts code significantly, delete it. Optionally move salvageable conceptual content to docs/design-notes.md with a staleness disclaimer. Do not try to bring spec into sync.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:46Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:10:05Z","close_reason":"Audited SPEC.md against code. Found minor inaccuracies (storage internals, wizard step count, search mode representation) but no major architectural contradictions. Deleted SPEC.md since it was largely redundant with repo-map.md and design-principles.md. Salvaged product vision and non-goals into docs/design-notes.md with staleness disclaimer.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-tik","title":"Phase 1: Repository hygiene","description":"Raise the quality floor of what a stranger sees when they land on the repo. Subtasks cover spec audit, Cargo.toml metadata, publish-readiness, CSV audit, and example files.","status":"closed","priority":1,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:45Z","created_by":"Edward Langley","updated_at":"2026-04-09T07:22:34Z","closed_at":"2026-04-09T07:22:34Z","close_reason":"All 5 subtasks complete: SPEC.md audited, Cargo.toml metadata fixed, publish dry-run clean, CSV audit done, example files created.","dependencies":[{"issue_id":"improvise-tik","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T23:37:31Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-902","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-bv1","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-km8","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-n1h","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-p1ym","title":"Project the page-axis item lists so [ and ] work in a browser","description":"The one key pair an independent smoke test found inert that the shared keymap did NOT fix, and it is a genuine projection gap rather than a resolution gap.\n\nGridCache::page_coords carries (category, SELECTED ITEM). PageNext/PagePrev need THE ITEM LIST TO CYCLE THROUGH, which nothing projects. So [ and ] resolve to a command name the client holds but cannot compose an argument for, and they reach Host::handle empty vec -- doctrine(an_upstream_key_is_a_black_hole), so they do nothing at all.\n\nCONFIRMED BY SMOKE TEST 2026-09-20 against the live page: \"[ / ] did not cycle Quarter (never saw Q2 ~17,100)\". The demo model puts Quarter on the page axis specifically so paging is a thing a visitor would try.\n\nTHE AGENT WAS TOLD NOT TO FORCE IT and did not -- a truthful \"these four keys work, [ / ] needs X\" is worth more than five half-working keys.\n\nSCOPE: extend page_coords (or add a sibling) to carry the item list per page-axis category. Note gotcha 27 -- \"what data is visible\" is View::filtering_cats() (Page UNION Filter) while \"what the user can page\" is categories_on(Axis::Page). This needs the SECOND one; projecting the first would let a client try to cycle a Filter axis, which by design it must not.\n\nWhile you are there: check whether the item list should be the EFFECTIVE items (hidden items excluded) rather than all of them, since [ and ] in the TUI cycle what the user can actually select.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"open","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:53:54Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-955n","title":"[bug] Host::commands applies a batch partially before rejecting","description":"Found by Copilot on PR #13 (the retro review of the unreviewed batch), DECLINED there with reasons on the thread, and filed rather than patched.\n\nADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a TYPE (Checked) to make the unchecked path unconstructible. But Checked is PER-COMMAND: Checked::against answers for one Command against the App AS IT STANDS. A ClientFrame::Commands is a SEQUENCE of them, and Host::commands applies in order and abandons at the first rejection. So a frame carrying SwitchView followed by a stale Command::Model has already applied the SwitchView.\n\nSO ADR 001 GUARANTEE IS CURRENTLY TRUE OF A SINGLE COMMAND AND FALSE OF A BATCH.\n\nA SECOND-ORDER EFFECT NOBODY HAD WRITTEN DOWN, found while declining it: Worker::on_frame skips persist() when ANY frame in the answer is Rejected. So a batch that DID apply a SwitchView before being refused leaves Workbook::active_view moved in memory and NOT WRITTEN BACK -- the document on disk and the document in the worker disagree, silently.\n\nWHY IT WAS DECLINED RATHER THAN PATCHED: closing it means the basis check moving from the COMMAND to the BATCH -- every command answered before any is applied. That is an ADR 001 design change with a real question inside it: WHAT DOES A BATCH BASIS MEAN WHEN ITS LATER COMMANDS WERE COMPOSED AGAINST ITS EARLIER ONES EFFECTS? A client resolving one keypress produces SetSelected plus offsets computed from the selection it just set. Answer that before writing code.\n\nREACHABILITY IS BOUNDED TODAY: no client in this workspace composes a batch containing more than one basis-carrying Command::Model -- the batches improvise-bpq2 produces are view commands, which carry no basis. Do this before the second tranche of client-side resolution creates a reason to batch two model commands.\n\nSupersedes the earlier ticket filed from bpq2 if there is one -- dedupe.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:35:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-08wb","title":"[bug] A refused ClientFrame::Commands keeps the commands before the refusal","description":"Found by improvise-bpq2 (2026-09-20), stated on the variant, and currently UNREACHABLE from any client in this workspace -- filed so it is a decision rather than a discovery.\n\nClientFrame::Commands(Vec\u003cCommand\u003e) batches, because one cursor key resolved client-side is SetSelected plus the offsets viewport_effects returns, and a host that folds the selection but not the offset projects the wrong window. Host::commands applies IN ORDER and ABANDONS AT THE FIRST REJECTION.\n\nTHAT LEAVES A HOLE ADR 001 DOES NOT HAVE. ADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a type (Checked) to make the unchecked path unconstructible. But Checked is per-COMMAND, and a ClientFrame::Commands is a sequence of them, so a frame whose third command is refused has already applied the first two.\n\nWHY IT IS NOT LIVE: no client in this workspace composes a Commands frame containing more than one basis-carrying Command::Model -- the batches bpq2 produces are view commands (SetSelected + offsets), which carry no basis.\n\nTHE FIX IS TO MOVE THE BASIS CHECK TO THE BATCH: check every Command::Model in the frame before applying any of them. That is the shape ADR 001 already describes for an effect batch, one level up, and it would make Host::commands match the guarantee the ADR states.\n\nDo this before a client has reason to batch two model commands -- which the second tranche of client-side resolution (the ~15 bindings bpq2 left) may well create.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:36:42Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-955n","title":"[bug] Host::commands applies a batch partially before rejecting","description":"Found by Copilot on PR #13 (the retro review of the unreviewed batch), DECLINED there with reasons on the thread, and filed rather than patched.\n\nADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a TYPE (Checked) to make the unchecked path unconstructible. But Checked is PER-COMMAND: Checked::against answers for one Command against the App AS IT STANDS. A ClientFrame::Commands is a SEQUENCE of them, and Host::commands applies in order and abandons at the first rejection. So a frame carrying SwitchView followed by a stale Command::Model has already applied the SwitchView.\n\nSO ADR 001 GUARANTEE IS CURRENTLY TRUE OF A SINGLE COMMAND AND FALSE OF A BATCH.\n\nA SECOND-ORDER EFFECT NOBODY HAD WRITTEN DOWN, found while declining it: Worker::on_frame skips persist() when ANY frame in the answer is Rejected. So a batch that DID apply a SwitchView before being refused leaves Workbook::active_view moved in memory and NOT WRITTEN BACK -- the document on disk and the document in the worker disagree, silently.\n\nWHY IT WAS DECLINED RATHER THAN PATCHED: closing it means the basis check moving from the COMMAND to the BATCH -- every command answered before any is applied. That is an ADR 001 design change with a real question inside it: WHAT DOES A BATCH BASIS MEAN WHEN ITS LATER COMMANDS WERE COMPOSED AGAINST ITS EARLIER ONES EFFECTS? A client resolving one keypress produces SetSelected plus offsets computed from the selection it just set. Answer that before writing code.\n\nREACHABILITY IS BOUNDED TODAY: no client in this workspace composes a batch containing more than one basis-carrying Command::Model -- the batches improvise-bpq2 produces are view commands, which carry no basis. Do this before the second tranche of client-side resolution creates a reason to batch two model commands.\n\nSupersedes the earlier ticket filed from bpq2 if there is one -- dedupe.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:29Z","started_at":"2026-09-20T05:32:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-08wb","title":"[bug] A refused ClientFrame::Commands keeps the commands before the refusal","description":"Found by improvise-bpq2 (2026-09-20), stated on the variant, and currently UNREACHABLE from any client in this workspace -- filed so it is a decision rather than a discovery.\n\nClientFrame::Commands(Vec\u003cCommand\u003e) batches, because one cursor key resolved client-side is SetSelected plus the offsets viewport_effects returns, and a host that folds the selection but not the offset projects the wrong window. Host::commands applies IN ORDER and ABANDONS AT THE FIRST REJECTION.\n\nTHAT LEAVES A HOLE ADR 001 DOES NOT HAVE. ADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a type (Checked) to make the unchecked path unconstructible. But Checked is per-COMMAND, and a ClientFrame::Commands is a sequence of them, so a frame whose third command is refused has already applied the first two.\n\nWHY IT IS NOT LIVE: no client in this workspace composes a Commands frame containing more than one basis-carrying Command::Model -- the batches bpq2 produces are view commands (SetSelected + offsets), which carry no basis.\n\nTHE FIX IS TO MOVE THE BASIS CHECK TO THE BATCH: check every Command::Model in the frame before applying any of them. That is the shape ADR 001 already describes for an effect batch, one level up, and it would make Host::commands match the guarantee the ADR states.\n\nDo this before a client has reason to batch two model commands -- which the second tranche of client-side resolution (the ~15 bindings bpq2 left) may well create.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:29Z","started_at":"2026-09-20T05:32:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ir7l","title":"[bug] A formula cell is offered as editable and silently reverts","description":"Pre-existing and consistent with the TUI, but improvise-bck just put it on the DEMO FRONT PAGE, which changes who meets it: the first thing a stranger sees is Profit beside Revenue and Cost.\n\nA formula target cell projects CellId::Record, so render.rs::editable says true, so the browser offers an editor. Type into Profit and the write lands -- then recompute_formulas overwrites it on the next pass and the value REVERTS with no explanation.\n\nWHY IT IS NOT SIMPLY A BUG IN editable: a formula target IS a stored coordinate; the cell is a record. What makes it unwritable is not its identity but that a FORMULA OWNS IT. So the fix is a fact the projection does not currently carry -- something like \"this cell is derived\" -- and it belongs beside numeric/error on CellDisplay, which already exists precisely because the SERVER is the side that knows.\n\nCHECK THE TUI BEFORE CHANGING ANYTHING: it has the same behaviour, so whatever is decided should land in both, or the divergence becomes the new bug (gotcha 44 is what that looks like). And note ADR 002 deliberately made settability a fact about the CellId VARIANT -- adding a second, independent \"but not really\" flag is exactly the two-answers-to-one-question shape this codebase keeps curing. Consider instead whether a formula-owned cell should project a different CellId at all.","notes":"VERIFIED CONCRETELY during the PR #13 review (2026-09-20), and the diagnosis is sharper than when this was filed.\n\nTHE ROOT CAUSE: project::cell_id decides Record vs Aggregate purely on whether a regular category sits on Axis::None. IT NEVER ASKS WHETHER THE _Measure COORDINATE NAMES A FORMULA TARGET. So a formula cell is a Record, render.rs::editable says true, and the browser offers an editor.\n\nMEASURED AGAINST THE SHIPPED DEMO:\n col 2 label=\"Profit\" is_record=true text=\"12,200\"\n after applying SetCell(key, Number(999.0)), Profit reads \"12,200\"\n\nThe write is ACCEPTED AND STORED. Model::evaluate_aggregated checks formula_cache BEFORE raw data (model/types.rs:409), so the typed value is shadowed forever with nothing telling the user. Not a revert -- a permanent silent shadow.\n\nAND THE TEST THAT COVERS THIS AREA IS LYING BY ITS NAME. every_visible_cell_in_the_web_demo_is_settable PASSES while its name is FALSE for the Profit column -- a third of the landing-page grid. It was proved discriminating against the one thing it CAN see (the aggregate at (0,0) in examples/demo.improv) and is structurally blind to this. That is exactly the shape AGENTS.md calls this repo's characteristic find, arriving in the test written to prevent it.\n\nTHE DECISION IS STILL THE DECISION, and ADR 002 is where it goes:\n * TEACH cell_id ABOUT FORMULA TARGETS -- changes what EVERY client calls a record, and record identity is also ADR 001's unit of atomicity, so this reaches the basis check too.\n * OR GIVE editable A SECOND INPUT -- but that is the conflation gotcha 42 exists to forbid, and ADR 002 deliberately made settability a fact about the CellId VARIANT precisely so there could not be a second answer.\n\nThe first is probably right and is not small. Until it lands, the demo's front page teaches a visitor that typing into Profit does nothing.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:51Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:35:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-vhcz","title":"[bug] OpfsFs::list disagrees with the other backends on path shape","description":"Found by improvise-9x6 (2026-09-20) and currently PAPERED OVER by always passing the flat root.\n\nOpfsFs::list returns BARE NAMES; MemoryFs and PhysicalFs return dir-JOINED ones. So Path::new(\"\") is the ONE dir at which all three agree, and the only one at which storage_contract actually describes OpfsFs.\n\nTHE TRAP: a worker that organised documents under docs/ would get an EMPTY PICKER in a browser and a FULL ONE in every test. The harness cannot see the disagreement because it never runs OpfsFs -- improvise-xf1 is deferred -- so this is precisely the class of defect the missing browser harness exists to catch, discovered by reading instead.\n\nTWO FIXES:\n * Make OpfsFs::list join dir, matching the other two. Straightforward, and makes storage_contract describe all three at every dir.\n * Or document the flat root as PART OF THE CONTRACT and assert it in storage_contract, so the constraint is stated rather than discovered.\n\nThe first is better unless there is a reason OPFS cannot -- note gotcha 28 says backends that are not trees treat dir as a NAME PREFIX, which is already the trait doc contemplating this, so check whether the doc or the impl is the thing that is wrong.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:14Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:10:14Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-hmws","title":"Storage cannot expose an async seam through Box\u003cdyn Storage\u003e","description":"Found by improvise-9x6 (2026-09-20). Every worker host will hit this, so it is worth solving once rather than in each.\n\nOpfsFs::open / attach / refresh / commit_deletes are on the STRUCT, not on the Storage trait -- deliberately, per gotcha 28: Storage stays synchronous because createSyncAccessHandle yields a blocking handle, and only four operations are genuinely async. But App::storage is a Box\u003cdyn Storage\u003e, so HANDING THE BACKEND TO THE App PUTS ALL FOUR PERMANENTLY OUT OF REACH. The App owns the only copy and the trait object cannot name them.\n\nAND THE OBVIOUS FIX IS REFUSED BY THE COMPILER: impl Storage for Rc\u003cOpfsFs\u003e is two foreign types, so the orphan rule forbids it. improvise-9x6 worked around it with a local newtype (bindings::Shared) holding an Rc\u003cOpfsFs\u003e and forwarding the four sync methods. That works but every worker must write it again.\n\nTWO CANDIDATE FIXES:\n * Ship the shared wrapper FROM improvise-io, so the newtype exists once where OpfsFs does and the orphan rule is satisfied.\n * Or have OpfsFs hand out an attachable handle -- separate the thing the App holds from the thing the message loop awaits, which is arguably the more honest modelling since they have different lifetimes and different call sites.\n\nPrefer whichever keeps Storage synchronous; that part is not negotiable (gotcha 28 is the argument).","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:10:13Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-vhcz","title":"[bug] OpfsFs::list disagrees with the other backends on path shape","description":"Found by improvise-9x6 (2026-09-20) and currently PAPERED OVER by always passing the flat root.\n\nOpfsFs::list returns BARE NAMES; MemoryFs and PhysicalFs return dir-JOINED ones. So Path::new(\"\") is the ONE dir at which all three agree, and the only one at which storage_contract actually describes OpfsFs.\n\nTHE TRAP: a worker that organised documents under docs/ would get an EMPTY PICKER in a browser and a FULL ONE in every test. The harness cannot see the disagreement because it never runs OpfsFs -- improvise-xf1 is deferred -- so this is precisely the class of defect the missing browser harness exists to catch, discovered by reading instead.\n\nTWO FIXES:\n * Make OpfsFs::list join dir, matching the other two. Straightforward, and makes storage_contract describe all three at every dir.\n * Or document the flat root as PART OF THE CONTRACT and assert it in storage_contract, so the constraint is stated rather than discovered.\n\nThe first is better unless there is a reason OPFS cannot -- note gotcha 28 says backends that are not trees treat dir as a NAME PREFIX, which is already the trait doc contemplating this, so check whether the doc or the impl is the thing that is wrong.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:14Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:36Z","started_at":"2026-09-20T05:32:36Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-hmws","title":"Storage cannot expose an async seam through Box\u003cdyn Storage\u003e","description":"Found by improvise-9x6 (2026-09-20). Every worker host will hit this, so it is worth solving once rather than in each.\n\nOpfsFs::open / attach / refresh / commit_deletes are on the STRUCT, not on the Storage trait -- deliberately, per gotcha 28: Storage stays synchronous because createSyncAccessHandle yields a blocking handle, and only four operations are genuinely async. But App::storage is a Box\u003cdyn Storage\u003e, so HANDING THE BACKEND TO THE App PUTS ALL FOUR PERMANENTLY OUT OF REACH. The App owns the only copy and the trait object cannot name them.\n\nAND THE OBVIOUS FIX IS REFUSED BY THE COMPILER: impl Storage for Rc\u003cOpfsFs\u003e is two foreign types, so the orphan rule forbids it. improvise-9x6 worked around it with a local newtype (bindings::Shared) holding an Rc\u003cOpfsFs\u003e and forwarding the four sync methods. That works but every worker must write it again.\n\nTWO CANDIDATE FIXES:\n * Ship the shared wrapper FROM improvise-io, so the newtype exists once where OpfsFs does and the orphan rule is satisfied.\n * Or have OpfsFs hand out an attachable handle -- separate the thing the App holds from the thing the message loop awaits, which is arguably the more honest modelling since they have different lifetimes and different call sites.\n\nPrefer whichever keeps Storage synchronous; that part is not negotiable (gotcha 28 is the argument).","status":"in_progress","priority":2,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:37Z","started_at":"2026-09-20T05:32:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-9mo9","title":"[bug] Geometry effects that renumber the grid but never reset the cursor","description":"Found by the code review of improvise-gvnb (2026-09-19). A PRE-EXISTING gap that gvnb inherited rather than introduced -- but gvnb is what makes it matter, because the epoch is now what tells every client their cursor is invalid.\n\nApp::reset_cursor bumps cursor_epoch; Sessions::adopt_cursor_epoch watches the COUNTER and broadcasts Command::ResetCursor to all. Nothing declares which effects renumber the grid -- deliberately, per gotcha 31 -- which means a geometry effect that forgets reset_cursor is INVISIBLE IN BOTH DIRECTIONS: the host own cursor stays stale AND no client is told.\n\nToday only TransposeAxes and CycleAxis call it. These also renumber rows or columns and do not:\n SetAxis, HideItem, ShowItem, ToggleGroup, TogglePruneEmpty, AddCategory, RemoveCategory\n (plus AddItem/RemoveItem, which are CellData rather than LayoutGeometry)\n\nSetAxis IS THE SHARPEST CASE: CycleAxis is literally SetAxis to Axis::next() and resets; SetAxis does not.\n\nWHY THE REVIEW DID NOT JUST PATCH IT: SetAxis is used by DrillIntoCell (cmd/grid.rs:365-471) to build the _Drill view, and resetting there could discard the user position in the view they drilled FROM, depending on effect order. That is a decision about effect ordering and intended UX, not a patch.\n\nKEEP THE NEGATIVE ARM. a_model_command_that_does_not_reshape_the_grid_leaves_cursors_alone uses SetNumberFormat -- a LayoutGeometry effect that must NOT reset. Without it, \"reset on every model command\" would pass the positive test. Whatever you do here, do not lose it.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:15Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:05:15Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-s1xw","title":"Multi-level grid labels on the wire -- the browser shows Food/Alice where the TUI shows two columns","description":"THE LAST THING KEEPING GridLayout INSIDE GridWidget, and a LIVE DIVERGENCE today, not a hypothetical. Found closing improvise-n10.\n\nGridCache::row_labels / col_labels are Vec\u003cString\u003e produced by GridLayout::row_label, which JOINS A MULTI-LEVEL TUPLE WITH A SLASH. That is exactly the rendering src/ui/grid.rs::two_row_categories_produce_cross_product_labels asserts must NOT appear (\"slash-joined labels should be gone\"). So the protocol carries a form of the labels THE TUI ITSELF CONSIDERS WRONG, and crates/improvise-wasm-client/src/render.rs renders precisely that form.\n\nConsequence: the browser grid has no multi-level headers, no repeat suppression, no groups and no collapse indicators, and shows Food/Alice where the TUI shows two header columns.\n\nSIX FACTS THE CACHE DOES NOT CARRY (exhaustive as of improvise-n10, by grep over the non-test body of src/ui/grid.rs):\n 1. number of header levels -- layout.row_cats / col_cats\n 2. multi-level label TUPLES -- layout.row_items (AxisEntry::DataItem(Vec\u003cString\u003e))\n 3. inline GROUP HEADER rows -- layout.row_items (AxisEntry::GroupHeader)\n 4. column GROUP SPANS -- layout.col_group_for(ci)\n 5. collapse indicator (down/right triangle) -- view.is_group_collapsed(cat, group)\n 6. records-mode column identity -- layout.record_col(ci). THIS ONE IS CORRECT TO KEEP OFF: it keys pending drill edits and is session-local by design.\n\nSo 1-5 are the gap.\n\nTWO CONSTRAINTS ON THE FIX:\n * IT IS A CO-ORDINATED CHANGE, NOT AN ADDITIVE ONE. row_labels/col_labels change TYPE, and improvise-wasm-client/src/render.rs consumes both fields, so the client updates in the same change.\n * DO NOT ADD row_levels BESIDE row_labels. Two answers to \"what are the labels\" is the disease this whole projection architecture exists to cure -- see improvise-edp, where the category panel was walked twice, two different ways.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:33Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-s1xw","title":"Multi-level grid labels on the wire -- the browser shows Food/Alice where the TUI shows two columns","description":"THE LAST THING KEEPING GridLayout INSIDE GridWidget, and a LIVE DIVERGENCE today, not a hypothetical. Found closing improvise-n10.\n\nGridCache::row_labels / col_labels are Vec\u003cString\u003e produced by GridLayout::row_label, which JOINS A MULTI-LEVEL TUPLE WITH A SLASH. That is exactly the rendering src/ui/grid.rs::two_row_categories_produce_cross_product_labels asserts must NOT appear (\"slash-joined labels should be gone\"). So the protocol carries a form of the labels THE TUI ITSELF CONSIDERS WRONG, and crates/improvise-wasm-client/src/render.rs renders precisely that form.\n\nConsequence: the browser grid has no multi-level headers, no repeat suppression, no groups and no collapse indicators, and shows Food/Alice where the TUI shows two header columns.\n\nSIX FACTS THE CACHE DOES NOT CARRY (exhaustive as of improvise-n10, by grep over the non-test body of src/ui/grid.rs):\n 1. number of header levels -- layout.row_cats / col_cats\n 2. multi-level label TUPLES -- layout.row_items (AxisEntry::DataItem(Vec\u003cString\u003e))\n 3. inline GROUP HEADER rows -- layout.row_items (AxisEntry::GroupHeader)\n 4. column GROUP SPANS -- layout.col_group_for(ci)\n 5. collapse indicator (down/right triangle) -- view.is_group_collapsed(cat, group)\n 6. records-mode column identity -- layout.record_col(ci). THIS ONE IS CORRECT TO KEEP OFF: it keys pending drill edits and is session-local by design.\n\nSo 1-5 are the gap.\n\nTWO CONSTRAINTS ON THE FIX:\n * IT IS A CO-ORDINATED CHANGE, NOT AN ADDITIVE ONE. row_labels/col_labels change TYPE, and improvise-wasm-client/src/render.rs consumes both fields, so the client updates in the same change.\n * DO NOT ADD row_levels BESIDE row_labels. Two answers to \"what are the labels\" is the disease this whole projection architecture exists to cure -- see improvise-edp, where the category panel was walked twice, two different ways.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:27Z","started_at":"2026-09-20T05:32:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-hwpi","title":"The browser shell fails quietly, and never reads a key","description":"Found by the improvise-cr3 code review (2026-09-19) by READING web/app.js. There is no browser and no headless DOM in this environment, so none of this is observed -- it is what the code says will happen, and it is filed rather than fixed because each item is a judgement call about the shell rather than a defect in the Rust.\n\nFOUR THINGS, in descending order of how much they would cost a person at a demo:\n\n1. NO KEYBOARD AT ALL. #grid carries tabindex=\"0\" but the page registers no keydown handler, so Client::on_key_event -- and the whole keys.rs classifier and the lockstep machinery behind it -- is UNREACHED from the browser. Consistent with improvise-bpq2 (ClientMsg has no Key variant server-side) but larger than it: even a LOCAL key, one that needs no round trip, does nothing. improvise-1ey acceptance (3) is \"arrow keys move the cursor locally with no round-trip\", and nothing in the page can do it.\n\n2. A MALFORMED FRAME IS QUIET. client.on_frame(event.data) returns Result\u003c_, JsError\u003e, which wasm-bindgen turns into a THROWN EXCEPTION. It is thrown inside the message listener with no try, so it surfaces as an uncaught error in the devtools console and the following repaint() is skipped. Nothing in the UI says so: the grid simply stops updating. A try/catch writing to el.status would make it visible.\n\n3. NO error LISTENER ON THE SOCKET. A connection that never opens leaves #status blank -- only close writes \"disconnected\". So \"server not running\" and \"still connecting\" look the same.\n\n4. close() RUNS TWICE ON COMMIT, benignly. Enter -\u003e close(true) -\u003e td.replaceChildren(original) removes the focused \u003cinput\u003e -\u003e blur fires -\u003e close(false). editing is already null so it is harmless (one extra repaint), but there is no re-entry guard and only a real browser would show it.\n\nNOTE WHAT MAKES THIS TICKET DIFFERENT: items 2-4 are the kind of thing a test would catch instantly and we have no way to run one. Before fixing them, consider whether the right first move is a headless DOM in CI (jsdom or playwright) -- otherwise the fixes are as unverified as the bugs. improvise-xf1 (verify OpfsFs against a real browser) wants the same harness.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:40:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:29:50Z","closed_at":"2026-09-20T02:29:50Z","close_reason":"All four items done, plus a fifth the ticket could not have predicted.\n\n1. KEYDOWN WIRED -- a window listener feeding Client::on_key_event. preventDefault fires only when a frame was produced and no Ctrl/Alt is held, so browser shortcuts stay the browser's; metaKey is skipped entirely, matching BrowserKey::to_key_event dropping Meta so Cmd-R cannot arrive as a bare r. The inline editor stopPropagation keeps it out of the way, with an if (editing) return as belt and braces.\n2. MALFORMED FRAME -- try/catch around on_frame writing to #status, plus console.error with the offending frame. render_model() is guarded the same way; it also returns a Result.\n3. SOCKET error LISTENER -- 'cannot reach ... -- is improvise-ws-server running?', so that no longer reads like 'still connecting'.\n4. close() RE-ENTRY GUARD -- a closed flag in edit().\n\nTHE FIFTH, AND IT IS THE INTERESTING ONE: a worker whose start() REJECTS is caught INSIDE worker.js, so it fires NO error event on the main thread. The page would have sat on 'in-page worker - demo.improv' over an empty grid forever -- THIS TICKET'S EXACT DISEASE, REBUILT IN THE NEW DEPLOYMENT ON DAY ONE. The guard is a first-frame watchdog (8s, cleared by the first frame), and it deliberately covers BOTH transports rather than each peer's own failure modes, because 'the peer never answered the handshake' is the one condition that means the same thing over a socket and over a port -- a check aimed per-peer would have missed exactly this one.\n\nKEYS ARE INERT AND THAT IS STATED, NOT HIDDEN. Host::handle answers ClientFrame::Key with an empty vec; improvise-bpq2 is the other half.\n\nSTILL UNRUN -- and this ticket predicted that would be the problem: 'consider whether the right first move is a headless DOM, otherwise the fixes are as unverified as the bugs.' That sentence is now true OF THESE FIXES. Three of the four are one-assertion Playwright tests. See the harness ticket.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-zv4m","title":"improvise-protocol should own the websocket framing, so both ends stop mirroring it","description":"Raised by improvise-cr3 (2026-09-19) as the thing to push back on, and filed rather than fixed there.\n\nTODAY: improvise-wasm-client/src/socket.rs declares ClientFrame/ServerFrame, which MIRROR improvise-ws-server/src/wire.rs ClientMsg/ServerMsg field for field. The client cannot depend on the server crate -- tokio, tungstenite, Box\u003cdyn Storage\u003e, all native -- so the duplication was the only way to speak the protocol from wasm.\n\nWHAT HOLDS THE LINE MEANWHILE: improvise-ws-server is a DEV-dependency of the client (the same argument the crate already makes for improvise-command), and the_socket_framing_matches_the_server round-trips every variant THROUGH THE SERVERS OWN TYPES in both directions; a_frame_named_the_other_envelopes_way_does_not_parse is its check-the-checker. So a drift is caught -- but only by a test whose job is to notice, rather than by a shared definition that cannot drift.\n\nTHE FIX: move the framing into improvise-protocol, which both ends already depend on and which is the crate whose charter is exactly this -- \"the wire vocabulary and the portable session slice\". Then the mirror, the dev-dependency and both compatibility tests all delete.\n\nNOTE THE THIRD ENVELOPE while you are in here: improvise-wasm-client/src/wire.rs has ClientMessage/ServerMessage framing the WORKER deployment (improvise-tm6). Three envelopes for two deployments is worth resolving in one pass -- decide whether the worker and the socket want the same vocabulary before moving either.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:37:50Z","closed_at":"2026-09-20T00:37:50Z","close_reason":"Duplicate of improvise-xks6 ('One home for the client/server frame envelopes'), filed minutes apart from the improvise-cr3 hand-back without spotting the existing ticket. Closing this one; xks6 is the survivor and its description should absorb two facts this one carried:\n\n * WHAT HOLDS THE LINE MEANWHILE: improvise-ws-server is a DEV-dependency of improvise-wasm-client, and the_socket_framing_matches_the_server round-trips every variant THROUGH THE SERVERS OWN TYPES in both directions, with a_frame_named_the_other_envelopes_way_does_not_parse as its check-the-checker. The drift is caught by a test whose job is to notice, rather than prevented by a shared definition.\n * THERE ARE THREE ENVELOPES, NOT TWO: improvise-wasm-client/src/wire.rs frames the WORKER deployment (improvise-tm6) separately again. Decide whether the worker and the socket want the same vocabulary before moving either, or the move produces two homes instead of one.\n\n(See also memory agent-issue-drift-pattern -- search before filing.)","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-8xhy","title":"[bug] A fit over a categorical feature emits an unevaluable formula","description":"Found during improvise-7na (2026-09-19), verified by probe, and pinned as a characterization test rather than fixed.\n\ngp_tree::data::preprocess one-hots a categorical column into a feature named Region=North. That is an ENCODING ARTEFACT, not an item of the document, so the emitted formula Expr::Ref fails Model::find_item_category and EVERY CELL of the fitted measure evaluates to Error(\"ref:Region=North\"). col.missing and the col.year / col.month date sub-features are in exactly the same position.\n\nSo \"fit a tree and get a working formula\" holds ONLY for a fit over numeric measures.\n\nTHE TEST THAT SHOULD HAVE CAUGHT IT WAS LYING: improvise_evaluation_matches_gp_tree_eval cannot see this, because its improvise_eval helper FABRICATES a _Measure item for every feature name, including outlook=rainy. It asserts the emitter and gp_tree::eval agree, which they do -- against a model that could not exist. Worth remembering as a shape: a helper that builds the world the code wants is not a test of whether that world exists.\n\nPinned as a_split_on_a_one_hot_feature_does_not_evaluate -- characterization, flip it when fixed.\n\nTHREE REMEDIES, pick deliberately: restrict fits to numeric features (smallest, and honest about what works); materialise one-hot columns as real _Measure items (makes the formula evaluable but pollutes the document with encoding artefacts); or teach the emitter improvise WHERE vocabulary so a categorical split becomes a real predicate over the category (largest, and the only one that produces a formula a human would have written). The third is probably right and should be costed before the first is shipped as permanent.","status":"closed","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:35:05Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:45:21Z","closed_at":"2026-09-20T00:45:21Z","close_reason":"Fixed as REMEDY 1, explicitly labelled interim. fitted_formula now takes the \u0026Model the formula is to be stored in and refuses a tree whose splits it cannot resolve, listing them. formula_text is untouched -- it is the tree's own text and there is nothing else it could write. Nothing outside the crate called fitted_formula.\n\nWHY THE EMITTER AND NOT THE DATASET LAYER: dropping categorical columns before the fit is smaller, but it throws away real signal (permutation importance, and any future remedy-3 emitter) and makes the crate's halves disagree -- one-hot encoding is gp-tree's whole job. The defect is that the EMITTER spells a formula the document cannot evaluate.\n\nWHY THE CHECK ASKS THE DOCUMENT, NOT THE NAME'S SHAPE: refusing names containing '=' or matching .missing would be a syntactic proxy and a wrong one, because a measure LITERALLY NAMED Region=North is a legal document and must be accepted. The two are indistinguishable as strings; only the document can tell them apart. So it calls Model::find_item_category -- the exact precondition Expr::Ref has -- which covers .missing, the date parts, and any future sub-feature naming scheme with no second copy of preprocess's expansion rules to drift from. Both directions pinned.\n\nTHE LYING TEST IS KEPT, NOT DELETED. Quoting and threshold routing are real things improvise_evaluation_matches_gp_tree_eval checks, and outlook=rainy earns its place there as an awkward NAME TO SPELL. What changed is that it and its helper now state their scope -- a test of arithmetic and quoting against a STIPULATED model, unable to see whether such a model could exist. The property it could not state is asserted separately against a real document: a_formula_the_emitter_accepted_evaluates_everywhere -- if fitted_formula says Ok, no cell of the target is a ref: error. That test and the refusal test are each other's check; without the accepting case the refusal could pass by refusing everything. The characterization is flipped to a_split_on_a_one_hot_feature_is_refused.\n\nREMEDY 3 IS STILL RIGHT AND IS NOT THIS TICKET -- successor filed. It is much larger than it looked: improvise has NO SCALAR COORDINATE PREDICATE (Filter gates a formula or an aggregate, it is not an Expr), it would still leave col.missing and the date parts uncovered, and it must first answer what such a predicate means where context.get(cat) is None.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -71,7 +72,7 @@ {"_type":"issue","id":"improvise-rkeq","title":"Decide RenderCache's shape: grid-only wire type vs per-widget sub-caches","description":"improvise-cqi landed RenderCache in improvise-protocol as a FLAT, grid-only type: row_labels, col_labels, col_widths, cells: Vec\u003cVec\u003cCellDisplay\u003e\u003e, documented as 'the client's copy of the visible grid', indexed by viewport position and rebuilt by folding projection commands. Six open tickets were written against a different object of the same name: a fat cache holding one sub-cache per widget, selected by a SubscriptionSet — improvise-35e (RenderCache::from_app, 'grid: Option\u003cGridCache\u003e, category_tree: Option\u003cCategoryTreeCache\u003e, formulas, views, tile_bar, wizard, help'), improvise-n10 ('takes \u0026GridCache'), improvise-pca ('its relevant sub-cache slice'), improvise-764 ('a WizardCache'), improvise-jb3 and improvise-edp. Neither is wrong; they are two objects with one name, and nothing in the tree will complain until someone writes from_app.","design":"Three shapes, none picked. (A) Grow the protocol type: RenderCache gains Option\u003cSubCache\u003e fields. Cost: the wire payload carries widget-shaped data a browser DOM renderer does not want, and Snapshot grows. (B) Keep the protocol RenderCache grid-only and give the native migration its own container — say WidgetCaches { grid: RenderCache, category_tree: ..., ... } — so the wire type is one field of it. The six migration tickets then say WidgetCaches where they say RenderCache. (C) Per-widget projection commands instead of a container: each widget subscribes and folds its own cache, no aggregate type at all. Whichever wins, improvise-cqq (projection emission) has to agree with it, because that is what produces the updates a cache folds.","acceptance_criteria":"(1) One of the three shapes is chosen and recorded here. (2) improvise-35e, n10, pca, 764, jb3 and edp are updated to name the chosen type. (3) If the protocol type changes shape, improvise-cqq's description is updated to emit for it.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T19:49:24Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:29:33Z","closed_at":"2026-09-19T20:29:33Z","close_reason":"DECIDED (user, 2026-09-19): shape (C) with per-widget caches. RenderCache is per-widget -- each widget owns its own cache type and folds its own projection-command stream; there is no flat grid-only wire type and no fat aggregate container. State synchronizes by events, not snapshots; the single exception is the initial state transfer, where a new client gets a Snapshot to seed from and then follows the event stream. All six migration tickets (35e, n10, pca, 764, jb3, edp) and the projection-emission ticket (cqq) were annotated with the decision.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-2ou","title":"test","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T19:43:00Z","created_by":"Edward Langley","updated_at":"2026-09-19T19:43:06Z","closed_at":"2026-09-19T19:43:06Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-g0f","title":"Decide whether two clients may view different views","description":"improvise-cqi added ViewState::active_view alongside Workbook::active_view. Today the host keeps them identical (App::switch_to_view, App::install_workbook), so behaviour is unchanged for a single client.\n\nThe open question is whether they are ALLOWED to diverge. If yes, 'which view am I looking at' becomes per-client the same way the cursor did in improvise-8uz, and Workbook::active_view degrades to the document's saved starting view. If no, SwitchView must be a broadcast command every client applies.\n\nThis matters for improvise-cqq (projection emission): the server computes each client's viewport from that client's view, and if views can differ then the projections differ per client too — which is a materially bigger server.\n\nNot urgent: the single-client TUI is correct either way, and the guard test the_two_active_view_names_never_diverge pins the current answer (they may not).","acceptance_criteria":"A decision recorded on the ticket, and either the divergence is permitted with a test showing two ViewStates on different views, or SwitchView is documented as a broadcast command.","notes":"GROOMED 2026-09-19: raised P3 -\u003e P2. It blocks improvise-cqq (P2), and a lower-priority blocker makes bd ready unusable in priority order (the priority-non-decreasing rule from the 2026-08-30 groom).","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-01T09:02:42Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:00:50Z","closed_at":"2026-09-19T21:00:50Z","close_reason":"DECIDED (user, 2026-09-19): YES -- two clients may view different views. Workbook::active_view becomes the DOCUMENT DEFAULT (what a fresh client opens); ViewState::active_view becomes authoritative PER CLIENT. Decided at the point it acquired teeth: improvise-cqq's per-client ClientSession makes divergence expressible for the first time. ViewState::cursors is already keyed by view name, so the cursor half needs no change. Implementation work filed separately; gotcha 29's lockstep invariant and its guard test the_two_active_view_names_never_diverge must be replaced by a deliberate split rather than deleted.","dependency_count":0,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"improvise-xf1","title":"Verify OpfsFs against a real browser","description":"improvise-i34 landed OpfsFs (crates/improvise-io/src/opfs.rs) compile-checked for wasm32 only — there is no browser in the dev environment.\n\nWhat to verify, in a dedicated Web Worker:\n- OpfsFs::open() gets the origin root\n- attach() acquires a sync access handle, creating the file\n- the storage_contract sequence in storage.rs passes: write/read/replace/list-sorted/delete/read-fails/delete-twice-errors/shorter-write-truncates\n- refresh() lists what attach created\n- commit_deletes() actually removes entries (reload the page and confirm)\n\nKnown API constraints already handled: handles are exclusive per file (attach closes what it replaces, detach releases), and sync access handles are worker-only. Safari 16.4+ has synchronous handle methods, so no IndexedDB fallback.\n\nMost likely failure points: the keys() async-iterator walk in refresh() is hand-rolled through js_sys::Reflect and is the least certain code in the file; and FileSystemReadWriteOptions 'at' handling on read/write.","acceptance_criteria":"The storage_contract sequence passes against OpfsFs in Chrome, Firefox and Safari, run from a dedicated Web Worker.","notes":"OPFS IS NO LONGER UNVERIFIED -- the happy path ran, in a real browser, on 2026-09-20.\n\nAn independent smoke test of https://fiddlerwoaroof.github.io/improvise/ (Safari) confirmed: the worker booted, the document seeded from demo.improv, an edit to Espresso Revenue (18,400 -\u003e 19,000) COMMITTED AND RECALCULATED Profit to 12,800, and THE EDIT SURVIVED A RELOAD. That is create, attach, write, read and persist through OpfsFs -- the first real execution of a backend that had only ever been compile-checked.\n\nSo the headline risk this ticket carried is retired: OpfsFs is not fundamentally broken, and gotcha 28's sync-access-handle design holds in practice.\n\nWHAT THIS DOES NOT COVER, and why the ticket stays open at P2 rather than closing:\n * storage_contract has still never been RUN against OpfsFs. The happy path working is not the contract passing -- list semantics, delete/commit_deletes, the flat-root disagreement (improvise-2f4y), and re-attach after close are all untested.\n * One real OpfsFs-related bug was already found and fixed by reading rather than testing (attach creates and indexes the entry, so vfs::open skipped the seed -- see improvise-9x6 and the vfs test that stands in for attach). A harness would have caught it in seconds and it cost a deploy cycle instead.\n * Everything found in the browser so far was found BY A PERSON PRESSING KEYS. That is not a repeatable check.\n\nSo the argument shifts: this is no longer 'does OPFS work at all', it is 'run storage_contract against the one backend that owes it'. Fold into the Playwright harness ticket rather than treating it as a separate browser investigation.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-01T06:10:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:29:44Z","dependency_count":0,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"improvise-xf1","title":"Verify OpfsFs against a real browser","description":"improvise-i34 landed OpfsFs (crates/improvise-io/src/opfs.rs) compile-checked for wasm32 only — there is no browser in the dev environment.\n\nWhat to verify, in a dedicated Web Worker:\n- OpfsFs::open() gets the origin root\n- attach() acquires a sync access handle, creating the file\n- the storage_contract sequence in storage.rs passes: write/read/replace/list-sorted/delete/read-fails/delete-twice-errors/shorter-write-truncates\n- refresh() lists what attach created\n- commit_deletes() actually removes entries (reload the page and confirm)\n\nKnown API constraints already handled: handles are exclusive per file (attach closes what it replaces, detach releases), and sync access handles are worker-only. Safari 16.4+ has synchronous handle methods, so no IndexedDB fallback.\n\nMost likely failure points: the keys() async-iterator walk in refresh() is hand-rolled through js_sys::Reflect and is the least certain code in the file; and FileSystemReadWriteOptions 'at' handling on read/write.","acceptance_criteria":"The storage_contract sequence passes against OpfsFs in Chrome, Firefox and Safari, run from a dedicated Web Worker.","notes":"OPFS IS NO LONGER UNVERIFIED -- the happy path ran, in a real browser, on 2026-09-20.\n\nAn independent smoke test of https://fiddlerwoaroof.github.io/improvise/ (Safari) confirmed: the worker booted, the document seeded from demo.improv, an edit to Espresso Revenue (18,400 -\u003e 19,000) COMMITTED AND RECALCULATED Profit to 12,800, and THE EDIT SURVIVED A RELOAD. That is create, attach, write, read and persist through OpfsFs -- the first real execution of a backend that had only ever been compile-checked.\n\nSo the headline risk this ticket carried is retired: OpfsFs is not fundamentally broken, and gotcha 28's sync-access-handle design holds in practice.\n\nWHAT THIS DOES NOT COVER, and why the ticket stays open at P2 rather than closing:\n * storage_contract has still never been RUN against OpfsFs. The happy path working is not the contract passing -- list semantics, delete/commit_deletes, the flat-root disagreement (improvise-2f4y), and re-attach after close are all untested.\n * One real OpfsFs-related bug was already found and fixed by reading rather than testing (attach creates and indexes the entry, so vfs::open skipped the seed -- see improvise-9x6 and the vfs test that stands in for attach). A harness would have caught it in seconds and it cost a deploy cycle instead.\n * Everything found in the browser so far was found BY A PERSON PRESSING KEYS. That is not a repeatable check.\n\nSo the argument shifts: this is no longer 'does OPFS work at all', it is 'run storage_contract against the one backend that owes it'. Fold into the Playwright harness ticket rather than treating it as a separate browser investigation.","status":"in_progress","priority":2,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-01T06:10:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:36Z","started_at":"2026-09-20T05:32:36Z","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-a2x","title":"Derive Serialize/Deserialize for the ViewState field types","description":"improvise-h0b evicted the two fields that could never be serialized (wizard: an improvise-io type; drill_state: holds an Rc). What remains is that ViewState's own field types still derive no Serialize, so it cannot actually be put on the wire.\n\nNeeded: AppMode, MinibufferConfig, ViewFrame (crates/improvise-command/src/ui/app.rs) and KeyCode, KeyModifiers (crates/improvise-command/src/command/key.rs). All five are plain data. CellValue already derives it.\n\nAlso requires adding serde with the derive feature to improvise-command's [dependencies] — today it depends only on serde_json.\n\nDeliberately NOT done in h0b: nothing serializes ViewState yet, so the derives would have no consumer. They land when improvise-cqi extracts the protocol crate and there is a Snapshot to build.","acceptance_criteria":"ViewState (or its successor in improvise-protocol) derives Serialize + Deserialize and a roundtrip test passes.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-08-30T22:56:09Z","created_by":"Edward Langley","updated_at":"2026-09-01T04:05:38Z","started_at":"2026-09-01T04:01:54Z","closed_at":"2026-09-01T04:05:38Z","close_reason":"ViewState + AppMode, MinibufferConfig, MinibufColor, ViewFrame, KeyCode, KeyModifiers all derive Serialize/Deserialize; serde added to improvise-command. Real blocker was unanticipated: MinibufferConfig.buffer_key was \u0026'static str, which Deserialize cannot produce, and rustc reported it on the five enclosing AppMode variants rather than the field. buffer_key and TextEntry::buffer are now Strings; rejected Cow (public-API plumbing) and a BufferKey enum (drags the buffers map key, script-parser args, and the search buffer). Guarded by view_state_survives_a_json_round_trip. Unblocks improvise-cqi.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-h0b","title":"Evict wizard and drill_state from ViewState (bootstrap-snapshot hygiene)","description":"Follow-up to improvise-jc8, which removed the hard cycle (Arc\u003cKeymap\u003e). Two fields remain in ViewState that should not be there:\n\n wizard: Option\u003cImportWizard\u003e -- improvise-io type; acyclic, but outside improvise-cqi AC 2's allowlist (core + formula + serde)\n drill_state: Option\u003cDrillState\u003e -- staged, uncommitted edits\n\nNeither is a dependency cycle, so this is not urgent the way jc8 was. The argument is bootstrap hygiene: per improvise-q08 the ONLY point ViewState is serialized is the Snapshot sent when a client connects. A fresh client should not inherit a half-open import wizard, and must not inherit another session's staged drill edits — those are uncommitted writes belonging to whoever staged them.\n\nShape (mirroring jc8): a host-side struct holds the session-local pieces alongside the portable ViewState.\n\n pub struct HostState {\n pub view: ViewState, // portable, serializable\n pub wizard: Option\u003cImportWizard\u003e,\n pub drill_state: Option\u003cDrillState\u003e,\n }\n\nAlso still needed before ViewState can be serialized at all: AppMode and DrillState derive no Serialize today (DrillState only matters if it stays).\n\nDo this before improvise-cqi extracts the protocol crate, or cqi inherits the problem.","notes":"PRIORITY RAISED P3-\u003eP2 (groomed 2026-08-30): this gates P2 work (improvise-cqi improvise-pro ), so leaving it P3 made 'bd ready' unworkable in priority order — the P2 cannot be reached without it. Priority should be non-decreasing along dependency edges. Revert if the intent was that the downstream P2 is genuinely not wanted yet.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-08-30T03:48:29Z","created_by":"Edward Langley","updated_at":"2026-08-30T22:58:17Z","started_at":"2026-08-30T22:53:35Z","closed_at":"2026-08-30T22:58:17Z","close_reason":"wizard and drill_state moved from ViewState to App under a documented session-local residue tag. Deviated from the ticket's HostState sketch: App already is the host struct with a residue taxonomy, so this adds a fifth tag rather than a wrapper layer (jc8 set the same precedent). Demonstrated with a ViewState: Send assertion that failed to compile first. The ticket's note about Serialize derives is split out as improvise-a2x, now blocking cqi.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-jc8","title":"Evict host-only state from ViewState: flip the transient-keymap Arc","description":"Prerequisite for improvise-cqi. ViewState (crates/improvise-command/src/ui/app.rs:381) holds transient_keymap: Option\u003cArc\u003cKeymap\u003e\u003e. Keymap lives in improvise-command, which will depend on improvise-protocol once improvise-gxi puts reduce_full there taking \u0026Command — so a ViewState in the protocol crate carrying a Keymap is a hard dependency cycle, independent of serde.\n\nFLIP THE ARC. The transient keymap is never arbitrary: it is always the sub-keymap of a Binding::Prefix at a specific (mode, key, mods), set during dispatch and consumed on the very next keypress. There are exactly three, all flat, all on the normal keymap (g, y, Z at keymap.rs:634-642); RecordsNormal reaches them through parent fallback. So ViewState can hold the ADDRESS rather than the keymap:\n\n - transient_keymap: Option\u003cArc\u003cKeymap\u003e\u003e\n + transient_prefix: Option\u003c(KeyCode, KeyModifiers)\u003e\n\nand let KeymapSet — which already owns every keymap — resolve it. Both consumers can: App::handle_key has keymap_set and the mode; src/draw.rs:238 only needs binding_hints() for the which-key popup. Nothing between set and consume changes the mode, because a Prefix binding emits SetTransientKeymap and nothing else.\n\nGuard the assumption: resolution looks the prefix up in the CURRENT MODE's keymap, which is only correct while prefixes are flat. If a nested prefix is ever added (a Prefix binding inside a sub-keymap), this breaks. Encode that as a test rather than a comment.\n\nSCOPE: this ticket is the Arc flip only. wizard: Option\u003cImportWizard\u003e and drill_state: Option\u003cDrillState\u003e also do not belong in a ViewState that gets serialized — improvise-q08 sends one bootstrap Snapshot on connect, and a fresh client should not inherit a half-open import wizard or another session's staged drill edits — but neither is a dependency cycle, so they are follow-ups.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-08-30T03:44:37Z","created_by":"Edward Langley","updated_at":"2026-08-30T03:48:16Z","started_at":"2026-08-30T03:44:42Z","closed_at":"2026-08-30T03:48:16Z","close_reason":"ViewState no longer holds Arc\u003cKeymap\u003e. It stores transient_prefix: Option\u003c(KeyCode, KeyModifiers)\u003e — the key that opened the prefix — and KeymapSet::prefix_keymap resolves it back to the sub-keymap. The Arc lives in KeymapSet, which already owned every keymap.\n\nThe motivation is the dependency graph, not serde: Keymap is in improvise-command, which will depend on improvise-protocol once improvise-gxi puts reduce_full there taking \u0026Command. A ViewState in protocol carrying a Keymap is a hard cycle. That blocker is now gone.\n\nCharacterization tests written BEFORE the change, all passing on the old code first, so the refactor had something to preserve — prefix keys had almost no coverage (one Keymap-level unit test):\n- prefix_g_then_g_jumps_to_first_row\n- prefix_z_then_z_reaches_save_and_quit (ZZ with no file path, which also confirms improvise-8tk's abort still fires)\n- unmatched_key_after_prefix_is_swallowed (a prefix is consumed whether or not the next key matches, so a stray key does not act as a fresh normal binding)\n- prefix_keys_reach_through_the_records_parent_keymap (RecordsNormal defines no prefixes; it inherits them)\n\nTwo new guards for the assumption the flip introduces:\n- prefix_keymaps_are_flat — resolution looks the prefix up in the CURRENT MODE's keymap, which is only correct while prefixes are flat. A nested Prefix would be resolved where it does not exist and the second half of the sequence would silently do nothing. Verified it bites by adding a nested prefix under y: 'Normal: prefix Key(Char('y')) contains nested prefixes [Key(Char('n'))]'.\n- prefix_resolves_through_parent_keymaps — pins that resolution walks parents, and that a plain command binding is not a prefix.\n\nSTILL IN ViewState, both out of scope here and neither a cycle: wizard: Option\u003cImportWizard\u003e (improvise-io is acyclic but outside cqi's AC 2 allowlist) and drill_state: Option\u003cDrillState\u003e. Both should still leave before cqi, for a different reason recorded on that ticket — improvise-q08 sends one bootstrap Snapshot on connect, and a fresh client should inherit neither a half-open import wizard nor another session's staged drill edits.","dependency_count":0,"dependent_count":1,"comment_count":0} @@ -107,14 +108,14 @@ {"_type":"issue","id":"improvise-60z","title":"Virtual views _Records and _Drill should not be persisted to .improv files","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-15T11:13:29Z","created_by":"Edward Langley","updated_at":"2026-07-31T16:04:38Z","started_at":"2026-07-31T07:26:39Z","closed_at":"2026-07-31T16:04:38Z","close_reason":"format_md skips views where View::is_virtual(); Initial View falls back to the first real view; parse_md drops stale virtual views so older files self-heal. Three regression tests in persistence.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-c5v","title":"Improve records view data entry by making each measure a normal value column (wide format instead of synthetic Value column)","description":"Current records view uses long format with synthetic 'Value' column. Editing values requires being on the 'Value' column; editing measure column renames the coordinate. This is awkward for typical data entry. Switch to wide format: one column per measure with direct value editing. Update build_records_mode, records_display, cell_key, resolve_display, DrillState pending edits, tests, and related effects. Red-green-refactor cycle. Extends/ reopens improvise-rbv.","status":"closed","priority":2,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-15T10:34:46Z","created_by":"Edward Langley","updated_at":"2026-08-29T19:29:18Z","started_at":"2026-08-29T19:13:28Z","closed_at":"2026-08-29T19:29:18Z","close_reason":"Records mode switched from long format (synthetic 'Value' column + a _Measure column) to wide format: a row is one coordinate tuple, each raw measure is its own editable column, and a trailing '+' column creates new measures.\n\nDerived the case matrix in Prolog rather than guessing it, which surfaced the load-bearing case the description does not mention: in wide format a row GROUPS one cell per measure, so editing a category column must move every one of those cells. The long-format code moved only the cell under the cursor, which would have torn rows in half — half the measures under the old coordinate, half under the new. Both edit paths (plain records and staged drill edits) implement it; both are tested.\n\nDesign decisions:\n- _Dim in the synthetic CellKey now carries the column INDEX, not a label. A category and a measure item can share a name, so a label cannot say which column an edit landed on. Resolved through GridLayout::record_col -\u003e RecordCol {Category, Measure, NewMeasure}. This rippled through synthetic_record_info, CmdContext::synthetic_record_at_cursor, DrillState.pending_edits (now keyed by (row, RecordCol), which also removes ApplyAndClearDrill's dependency on the layout still being the drill layout), and the grid renderer.\n- Measure columns are the raw _Measure items only. Formula targets are computed, so there is nothing to type into them.\n- The '+' column was the user's call over two alternatives, because a blank model has no measures and wide format would otherwise leave nowhere to enter a number — that is the onboarding path pinned by the old edit_record_row_in_blank_model_persists_value test. Adding a measure is a schema change, so it applies immediately even inside a drill rather than being staged.\n\nVerified end-to-end, not just by unit tests: tests/records_wide_smoke.rs imports a real CSV, loads the .improv, and asserts the records grid is two rows with both measures side by side. My first two expectations there were wrong (measure order and row order come from the file/store, not the source CSV) — the code was right both times.\n\nSix existing tests encoded long-format behavior and were rewritten, not deleted: the two-row helper now uses two Region items rather than two measures on one, and the blank-model test drives the new '+' flow.\n\nAlso removed prune_empty's records branch, which was unreachable behind the function's own early return; filed improvise-2yk in case the early return is itself the bug.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-edp","title":"Establish ViewModel layer: derived data between RenderCache and widgets","description":"Add a classical MVVM ViewModel layer between the RenderCache (wire-format data sent from server) and the widgets (pure renderers). The ViewModel is computed from RenderCache + ViewState + client-local concerns (terminal size, fonts, theme), memoized on state change, and consumed by widgets. Decouples widgets entirely from cache shape — they only know about the ViewModel they consume. Shared concept across native ratatui and browser DOM rendering; both consume viewmodels rather than caches directly.","design":"For each rendering surface, a pair of types: (1) SurfaceCache — the data the server sends over the wire (raw cells, labels, indices). (2) SurfaceViewModel — the render-ready derived data (styled cell display with highlighting, pre-computed visible row range, column positions in terminal coordinates, cursor-region flags). Paired with a pure function fn compute_surface_viewmodel(cache: \u0026SurfaceCache, view: \u0026ViewState, render_env: \u0026RenderEnv) -\u003e SurfaceViewModel. render_env holds client-local concerns: terminal dimensions for native, window dimensions + device pixel ratio for browser, color theme, unicode width handling. Memoization: recompute viewmodel when cache or view state changes; skip when only render_env changes trivially. Widgets read \u0026SurfaceViewModel — entirely decoupled from cache shape. Benefits: (a) widgets are unit-testable with hand-crafted viewmodels, (b) viewmodel logic is testable without any rendering, (c) ratatui and DOM renderers can share the same viewmodel derivation code, (d) memoization prevents redundant recomputation.","acceptance_criteria":"(1) At least one pair (GridCache, GridViewModel) defined with compute function. (2) GridWidget and DOM grid renderer both consume GridViewModel. (3) Unit tests for viewmodel derivation covering edge cases (empty grid, wide cells, highlights). (4) Memoization hook so recomputation is skipped when neither cache nor view state has changed.","notes":"Sits between improvise-35e (cache adapter) and the widget migration issues (improvise-n10, -pca, -jb3, -764). Each widget migration issue should read its viewmodel, not its cache directly. Shared with the browser epic: improvise-cr3 (DOM renderer) should also consume viewmodels. This is the refinement the user asked for — 'the viewmodel is probably a good idea as well'.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.\n\n--- UNBLOCKED: improvise-vjms decided (user, 2026-09-19) ---\nThe category panel question is answered: the TUI KEEPS the virtual categories\nand the BROWSER GETS THEM TOO. CategoryTreeCache grows to carry _Index, _Dim\nand _Measure so both panels render the same thing from one source.\n\nTHE TRAP TO AVOID when implementing: _Measure items are DERIVED, not stored --\ngotcha 7 says they are the explicit items UNION the formula targets, and\nadd_formula does not add items to _Measure. So category_tree_of must compute\nmembership through measure_item_names()/effective_item_names(), NEVER through\ncategory(\"_Measure\").items, or the panel silently omits every formula target.\nimprovise-ete was closed for proposing exactly that raw-list getter.\n\nAlso: CategoryTreeCache doc comment says \"the client does not know the\n_-prefixed virtual-category convention\" -- that is now false and must be\nremoved rather than left to mislead.\n\nThen build_cat_tree can migrate off \u0026Model (it takes \u0026Model today), which is\nwhat actually unblocks the category panel and this ticket. improvise-35e\nmigrated the other three panels already; this is the fourth.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:53:04Z","created_by":"spot","updated_at":"2026-09-20T00:25:56Z","closed_at":"2026-09-20T00:25:56Z","close_reason":"PREMISE LARGELY FALSE; the one real piece was built and the rest should not be.\n\nThe ticket asked for a SurfaceViewModel per SurfaceCache plus compute_*(cache, view, render_env). Evidence it does not apply: (a) gotcha 32 already removed the aggregate RenderCache the ticket names, and (b) after improvise-35e the widget seam is ALREADY one cache plus \u0026AppMode -- FormulaContent::new(\u0026FormulaCache, \u0026AppMode), ViewContent::new(\u0026ViewListCache, \u0026str). A per-widget cache built by project.rs is already the render-ready shape: CellDisplay carries formatted strings, GridCache carries col_widths. There is no cache-shape to decouple a widget from. render_env is likewise handled -- the terminal extent enters through ClientSession::set_extent, and inventing a Command for it is forbidden by decision(improvise-cqi, command_set_derived_not_invented).\n\nWHAT WAS REAL AND GOT BUILT: the category panel flattening, which genuinely sits between a cache and a renderer because it mixes CategoryTreeCache with per-client ViewState::expanded_cats. It lives in improvise-protocol::viewmodel (CatTreeEntry + flatten_cat_tree), NOT improvise-command, because improvise-bpq2 cannot reach improvise-command -- the wasm client Cargo.toml excludes it deliberately -- and move-panel-cursor walks this flattening. Both expanded_cats and cat_panel_cursor already live in protocol ViewState, so the derivation inputs and its consumer index are all in that crate. Putting it in improvise-command would have guaranteed bpq2 re-derives it client-side and the two ends drift.\n\nALSO DELIVERED: improvise-vjms. CategoryTreeCache now carries every category; the stale \"the client does not know the _-prefixed convention\" doc is gone; CatTreeEntry carries its Axis so CategoryContent takes a cache and an expanded set and no View -- the FOURTH AND LAST panel off \u0026Model. It also removes a latent panic: the widget called view.axis_of(name), which panics for a category the view has never seen.\n\nA LIVE SHIPPED BUG WAS FIXED, not a hypothetical one: build_cat_tree took \u0026Model and read cat.items.len()/ordered_item_names() -- the RAW list -- so the TUI category panel has been showing _Measure with the WRONG ITEM COUNT and OMITTING EVERY FORMULA TARGET, on main, today. That is the improvise-ete failure mode already shipped. Injecting it back fails exactly ONE test against 1023 green.\n\nNOT BUILT AND NOT WORTH A TICKET AS SPECCED: viewmodels for grid/formula/view/tile-bar. 1031 tests pass.","dependencies":[{"issue_id":"improvise-edp","depends_on_id":"improvise-35e","type":"blocks","created_at":"2026-04-14T00:53:35Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":6,"comment_count":0} -{"_type":"issue","id":"improvise-e0u","title":"Split native binary into in-process client + server, delete \u0026App adapter","description":"The architectural payoff: once every widget consumes RenderCache + ViewState, restructure the native binary so the ratatui UI is a proper client of an in-process server. Main loop has two halves: server holds the full App + projection layer + VFS persistence; client holds ViewState + RenderCache + ratatui widgets. Commands flow through a direct-call channel (zero-serialization in-process transport). Delete the temporary RenderCache::from_app adapter.","design":"Main loop restructure: fn main() spawns (a) a server loop that owns an App + projection layer + VFS storage (via improvise-6mq Storage trait backed by PhysicalFS for native), and (b) a client loop that owns a ViewState + RenderCache and renders ratatui. Communication via a tokio mpsc channel (or plain std::sync::mpsc since native is single-threaded) — no serialization because both sides hold the same Command enum in memory. Client captures keys → resolves via keymap → sends Command upstream → server calls reduce_full → projection commands come back down → client applies them via reduce_view → widgets redraw. Importantly, this is exactly the same message flow as the browser modes; the transport is just a function call. Delete crates/improvise-command/src/ui/app.rs's RenderCache::from_app adapter (issue 2). Native TUI's App now lives on the server side and is never read directly by widgets.","acceptance_criteria":"(1) Native binary has distinct client and server halves with a Command channel between them. (2) Ratatui widgets never read \u0026App. (3) RenderCache::from_app adapter is deleted. (4) All existing tests + integration tests pass. (5) Performance: frame latency unchanged from current native TUI (the projection work is now on the server side, the rendering work on the client side, but it's all the same process). (6) A structural lint: nothing in src/ui/ can import from src/model/ or src/command/ (only from improvise-protocol's cache types).","notes":"Terminal node of the required work in this epic. Depends on issues 1 (reduce_full), 2 (adapter), 3-6 (all widget migrations). After this lands, the native TUI and the browser modes share an identical architecture — only the transport and persistence backend differ.","status":"open","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:50:54Z","created_by":"spot","updated_at":"2026-08-14T00:44:33Z","dependencies":[{"issue_id":"improvise-e0u","depends_on_id":"improvise-35e","type":"blocks","created_at":"2026-04-14T00:53:39Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-764","type":"blocks","created_at":"2026-04-14T00:53:42Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:38Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-jb3","type":"blocks","created_at":"2026-04-14T00:53:41Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-n10","type":"blocks","created_at":"2026-04-14T00:53:40Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-pca","type":"blocks","created_at":"2026-04-14T00:53:40Z","created_by":"spot","metadata":"{}"}],"dependency_count":6,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"improvise-e0u","title":"Split native binary into in-process client + server, delete \u0026App adapter","description":"The architectural payoff: once every widget consumes RenderCache + ViewState, restructure the native binary so the ratatui UI is a proper client of an in-process server. Main loop has two halves: server holds the full App + projection layer + VFS persistence; client holds ViewState + RenderCache + ratatui widgets. Commands flow through a direct-call channel (zero-serialization in-process transport). Delete the temporary RenderCache::from_app adapter.","design":"Main loop restructure: fn main() spawns (a) a server loop that owns an App + projection layer + VFS storage (via improvise-6mq Storage trait backed by PhysicalFS for native), and (b) a client loop that owns a ViewState + RenderCache and renders ratatui. Communication via a tokio mpsc channel (or plain std::sync::mpsc since native is single-threaded) — no serialization because both sides hold the same Command enum in memory. Client captures keys → resolves via keymap → sends Command upstream → server calls reduce_full → projection commands come back down → client applies them via reduce_view → widgets redraw. Importantly, this is exactly the same message flow as the browser modes; the transport is just a function call. Delete crates/improvise-command/src/ui/app.rs's RenderCache::from_app adapter (issue 2). Native TUI's App now lives on the server side and is never read directly by widgets.","acceptance_criteria":"(1) Native binary has distinct client and server halves with a Command channel between them. (2) Ratatui widgets never read \u0026App. (3) RenderCache::from_app adapter is deleted. (4) All existing tests + integration tests pass. (5) Performance: frame latency unchanged from current native TUI (the projection work is now on the server side, the rendering work on the client side, but it's all the same process). (6) A structural lint: nothing in src/ui/ can import from src/model/ or src/command/ (only from improvise-protocol's cache types).","notes":"Terminal node of the required work in this epic. Depends on issues 1 (reduce_full), 2 (adapter), 3-6 (all widget migrations). After this lands, the native TUI and the browser modes share an identical architecture — only the transport and persistence backend differ.\nDEFERRED 2026-09-19 by user ruling during the whole-backlog campaign.\n\nDeferred as a campaign of its own, not as a rejection. e0u restructures the native binary around an in-process client/server split and deletes the RenderCache::from_app adapter -- it touches draw.rs, every widget, App and the projection layer, so it would conflict with essentially every other work unit in the backlog and would have to go last with everything rebasing onto it.\n\nThe cheap members of its cluster ARE being done now: improvise-2ud (CmdContext borrows the slices), improvise-drg (narrow Effect::apply where one-slice), improvise-s17o (decide reduce_full's fate), improvise-i5tk (benchmark the whole-grid projection first). Those are the parts that do not require the split.\n\nUn-defer when the browser/protocol backlog has settled and e0u can have a wave to itself.","status":"deferred","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:50:54Z","created_by":"spot","updated_at":"2026-09-20T05:31:01Z","dependencies":[{"issue_id":"improvise-e0u","depends_on_id":"improvise-35e","type":"blocks","created_at":"2026-04-14T00:53:39Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-764","type":"blocks","created_at":"2026-04-14T00:53:42Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:38Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-jb3","type":"blocks","created_at":"2026-04-14T00:53:41Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-n10","type":"blocks","created_at":"2026-04-14T00:53:40Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-e0u","depends_on_id":"improvise-pca","type":"blocks","created_at":"2026-04-14T00:53:40Z","created_by":"spot","metadata":"{}"}],"dependency_count":6,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-764","title":"Migrate import wizard widget and state to consume RenderCache","description":"Rewrite ui/import_wizard_ui.rs (347 lines) to consume a WizardCache instead of the ImportWizard struct directly. Also decide where the import wizard state itself lives in the new crate layout — probably needs to stay in improvise-command (not improvise-io) so it can be referenced from App without creating a dep cycle.","design":"WizardCache mirrors the ImportWizard state: current step, per-field decisions, preview rows, current validation errors. The widget reads cache + ViewState and draws. The ImportWizard state machine stays wherever it ends up living after the crate-split epic step 3 resolves its placement — either improvise-command (portable state machine) or improvise-io (tangled with CSV parsing). The separation between wizard state (session/view-ish) and import pipeline (pure data processing) is worth getting right because it affects the worker-server's ability to run imports.","acceptance_criteria":"(1) import_wizard_ui.rs no longer reads the wizard state directly — takes WizardCache. (2) ImportWizard is split into state + pipeline; state lives in an appropriate crate that doesn't cycle. (3) Native TUI wizard rendering unchanged. (4) Unit tests with fixture WizardCaches for each step.","notes":"Depends on improvise-edp (ViewModel layer). Wizard widget consumes WizardViewModel derived from WizardCache + ViewState. The wizard state machine itself (step tracking, field decisions) lives in the ImportWizard struct on the server side; the cache is a snapshot of its renderable state; the viewmodel adds any styling/layout derivation the widget needs.\nPRIORITY RAISED P3-\u003eP2 (groomed 2026-08-30): this gates P2 work (improvise-ltq improvise-e0u ), so leaving it P3 made 'bd ready' unworkable in priority order — the P2 cannot be reached without it. Priority should be non-decreasing along dependency edges. Revert if the intent was that the downstream P2 is genuinely not wanted yet.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:50:47Z","created_by":"spot","updated_at":"2026-09-20T00:58:17Z","closed_at":"2026-09-20T00:58:17Z","close_reason":"Done, and the ticket's open design question is answered.\n\nThe widget is rewritten against a new WizardCache (+ WizardStepKind, WizardFieldKind, WizardProposal, WizardDateField) in improvise-protocol::render, built by project::wizard_cache / wizard_of. ImportWizard, ImportPipeline, FieldKind, DateComponent and Workbook all stop at the projection.\n\nTHE ANSWER TO 'WHERE DOES THE WIZARD STATE LIVE': THERE IS DELIBERATELY NO Projection::Wizard. A Projection is addressed to a widget on EVERY client, which is correct exactly when the change is addressed to NO ONE session. A half-open import dialogue belongs to precisely the session that opened it. This follows settled doctrine rather than re-deciding it -- ImportWizard is on App and not ViewState (gotcha 30), and AcceptWizardModel was cut from ModelCommand for exactly this reason (gotcha 38 / improvise-tom1). Broadcasting the wizard would re-create that bug on the RENDERING side: every client drawing one client's dialogue.\n\nSo: a cache WITHOUT a projection is the shape session-local residue takes when it still has to be drawn by two different front ends. This resolves the open question gotcha 32 parks on this ticket.\n\nThe type still lives in improvise-protocol rather than improvise-command for the flatten_cat_tree reason: the wasm client reaches only the protocol, and when it grows a wizard it must draw THESE rows, not a second derivation.\n\nMulti-client import wants the ADDRESSED form, which is ADR 001 layer 2.\n\nONE GAP: WizardStep::Preview is unreachable from both fixtures -- it needs a document that parses to zero records without needing path selection. Filed.","dependencies":[{"issue_id":"improvise-764","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:38Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-jb3","title":"Migrate help and which_key widgets to consume RenderCache + ViewState","description":"Rewrite the small static-content widgets: ui/help.rs (5-page help overlay, 617 lines but mostly static text) and ui/which_key.rs (prefix-key hint popup). Lowest-risk widgets in the migration because their content is almost entirely derived from static data plus a small amount of ViewState (help_page index, active transient_keymap).","design":"HelpCache: largely static; just the page-index state and the content pages (constant). Could even be stateless if the renderer reads help_page directly from ViewState. WhichKeyCache: derived from the active transient_keymap — a flat list of (key, binding description) entries. Each widget becomes a pure render function taking its cache + ViewState for minimal dynamic bits.","acceptance_criteria":"(1) help.rs and which_key.rs no longer import App. (2) Unit-testable with fixture caches (or no cache for help). (3) Native TUI rendering unchanged.","notes":"Depends on improvise-edp (ViewModel layer). Help widget consumes HelpViewModel (derived from the static help content + current help_page); which_key consumes WhichKeyViewModel (derived from the active transient_keymap). Low-complexity widgets, but go through the viewmodel layer for consistency with the rest of the migration.\nPRIORITY RAISED P3-\u003eP2 (groomed 2026-08-30): this gates P2 work (improvise-e0u improvise-ltq ), so leaving it P3 made 'bd ready' unworkable in priority order — the P2 cannot be reached without it. Priority should be non-decreasing along dependency edges. Revert if the intent was that the downstream P2 is genuinely not wanted yet.\nSTALE ASSUMPTION (groomed 2026-08-30): this design refers three times to 'the active transient_keymap' as a ViewState field. That field is gone (improvise-jc8). ViewState now holds transient_prefix: Option\u003c(KeyCode, KeyModifiers)\u003e — the key that OPENED the prefix — and the sub-keymap is resolved on demand via KeymapSet::prefix_keymap(mode, key, mods), which walks parent keymaps.\n\nConsequence for the WhichKeyCache described here: it cannot be derived from ViewState alone any more. It needs ViewState (for the armed prefix) plus the keymap set (to resolve it and call binding_hints()). App::armed_prefix_keymap() does exactly that today and is the natural source. This matters because the whole point of jb3 is that widgets stop reading \u0026App — so decide whether the cache is populated server-side from the resolved keymap, or whether the hint list itself is what crosses to the client. The latter is probably right: hints are (label, command-name) pairs, which are portable, whereas a Keymap is not.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-14T07:50:23Z","created_by":"spot","updated_at":"2026-09-20T00:58:16Z","closed_at":"2026-09-20T00:58:16Z","close_reason":"The widget half was already done, and the ticket's own premise was stale -- its grooming note said so and was right.\n\nHelpWidget::new(help_page: usize) and WhichKeyWidget::new(\u0026[(String, \u0026'static str)]) were NEVER \u0026App-coupled; the git history shows no migration because none was needed. The which-key derivation happens in draw.rs, not in the widget: App::armed_prefix_keymap() + Keymap::binding_hints(). The widget is already a pure renderer of the hint list.\n\nWHAT REMAINS IS A PROTOCOL GAP, NOT A WIDGET PROBLEM, and it is filed as its own ticket: ViewState::transient_prefix is portable, but THE KEYMAP SET IS NOT ON THE CLIENT, so a browser which-key cannot be derived from ViewState alone. Either the resolved hint list crosses (a projection or a ViewState field) or the client gets the keymaps. That is a real decision and it was never this ticket's.","dependencies":[{"issue_id":"improvise-jb3","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:37Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-pca","title":"Migrate panel widgets (category/formula/view/tile_bar) to consume RenderCache","description":"Rewrite the smaller ratatui panel widgets so each one takes its relevant sub-cache slice instead of \u0026App. Targets: ui/category_panel.rs + cat_tree.rs, ui/formula_panel.rs, ui/view_panel.rs, ui/tile_bar.rs, ui/panel.rs (generic frame). Each becomes a pure renderer of its own cache structure.","design":"Four sub-cache types: CategoryTreeCache (flattened cat/item/group tree with expand state, current cursor index), FormulaListCache (raw strings + target info), ViewListCache (names + active flag), TileBarCache (per-category axis assignment). Each widget's render takes \u0026SubCache and \u0026ViewState (for mode / cursor). Widget signatures: fn draw_category_panel(frame, area, cache: \u0026CategoryTreeCache, view: \u0026ViewState); same shape for the others. The flattening logic that's currently in ui/cat_tree.rs (build_cat_tree) moves into cache construction rather than happening at render time. Handcrafted fixture caches for unit tests. Panels become entirely render-only; no App dependency. Shared with the browser/standalone clients later if they ever add panel rendering (post-MVP for those).","acceptance_criteria":"(1) Each of the four panel widgets no longer imports App. (2) Each is unit-testable with a fixture sub-cache. (3) Native TUI renders identically to before. (4) Sub-cache types live in improvise-protocol so they can be serialized for remote clients later.","notes":"Depends on improvise-edp (ViewModel layer). Each panel widget consumes its own sub-viewmodel (CategoryTreeViewModel, FormulaListViewModel, ViewListViewModel, TileBarViewModel), computed from the corresponding sub-cache + ViewState. Panels become pure renderers of viewmodels; all flattening, ordering, and highlight logic lives in the compute_*_viewmodel functions.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:49:06Z","created_by":"spot","updated_at":"2026-09-20T00:58:16Z","closed_at":"2026-09-20T00:58:16Z","close_reason":"Already done -- delivered by improvise-35e (92fd3a8) and improvise-vjms (986af9e), both already on main. No work was needed and none was invented.\n\nEVIDENCE: CategoryContent::new(\u0026CategoryTreeCache, \u0026HashSet\u003cString\u003e), FormulaContent::new(\u0026FormulaCache, \u0026AppMode), ViewContent::new(\u0026ViewListCache, \u0026str), TileBar::new(\u0026TileBarCache, \u0026AppMode, usize). None takes \u0026App, \u0026Model, \u0026View or \u0026Workbook outside its test module.\n\nONE RESIDUAL DERIVATION, filed separately rather than fixed: ViewContent::axis_summary hard-codes !cat.starts_with('_') to filter virtual categories out of the axis summary, while ViewSummary::axes carries EVERY category. That is the same convention improvise-vjms deliberately pushed onto the client for the category panel, still applied locally by the view panel. Not a bug today -- there is no browser view panel -- but it is a second, divergent answer waiting.","dependencies":[{"issue_id":"improvise-pca","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:36Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-n10","title":"Migrate GridWidget to consume RenderCache + ViewState (not \u0026App)","description":"Rewrite ui/grid.rs so that GridWidget::render takes \u0026GridCache + \u0026ViewState instead of reaching into \u0026App. The grid is the largest and most important widget — 1036 lines, mixes layout calc + ratatui drawing + direct App reads. The migration separates 'what to draw' (data in GridCache) from 'how to draw it' (ratatui primitives). Also aligns with improvise-cr3 (browser DOM renderer), which renders the same GridCache shape to DOM — share the GridCache type between them.","design":"pub struct GridCache { pub col_widths: Vec\u003cu16\u003e, pub row_labels: Vec\u003cString\u003e, pub col_labels: Vec\u003cString\u003e, pub cells: Vec\u003cVec\u003cCellDisplay\u003e\u003e, pub cursor: (usize, usize), pub highlights: Vec\u003cHighlight\u003e, pub mode_indicator: ModeIndicator, ... }. GridWidget becomes: fn draw_grid(frame: \u0026mut Frame, area: Rect, cache: \u0026GridCache, view: \u0026ViewState). All current dynamic calculations (col width from content, layout metrics, records mode detection) move to GridCache construction in the projection layer, not at render time. Widget is pure drawing. The temporary RenderCache::from_app adapter (issue 2) populates GridCache from App+View+GridLayout; later the projection layer computes it server-side for remote clients. During this migration, the widget is tested against a handcrafted GridCache fixture — no App needed. Share GridCache type with improvise-cr3 by placing it in improvise-protocol.","acceptance_criteria":"(1) GridWidget no longer imports App or reads anything outside GridCache + ViewState. (2) Widget is unit-testable with a fixture GridCache. (3) Native TUI renders identically to before for every .improv file in tests/. (4) GridCache type is shared with improvise-cr3 (both rasterize the same data structure). (5) Performance: per-frame render time unchanged (the projection layer work happens outside the widget, not inside).","notes":"Depends on issue improvise-edp (ViewModel layer) — the grid widget consumes GridViewModel (client-computed from GridCache + ViewState + render env), not GridCache directly. Share design work with improvise-cr3 (browser DOM renderer) — both target the same GridViewModel type, differing only in the rendering backend.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:49:03Z","created_by":"spot","updated_at":"2026-09-20T00:58:15Z","closed_at":"2026-09-20T00:58:15Z","close_reason":"Done, partially -- and the part left undone is the finding.\n\nGridWidget no longer holds a \u0026Model. Three derivations moved:\n * COLUMN TOTALS -\u003e GridCache::totals + project::column_totals. The widget was summing every row of each visible column through evaluate_aggregated_f64. This MUST be projected, not client-derived: the sum runs over every row INCLUDING THE ONES A VIEWPORT SCROLLED PAST, so a client holding only its window cannot compute it. An EMPTY strip means 'no totals' (records mode / empty grid) -- deliberately not a row of blanks, and fit_to_labels pads it only once non-empty. Invalidated blanks the totals: they are contents, and a stale sum under fresh cells is the one number a user takes at face value.\n * PAGE BAR -\u003e GridCache::page_coords, framed by nothing (it names categories, not positions) so it survives a reshape.\n * THE ITALIC-AGGREGATE RULE -\u003e read off CellId::Aggregate (ADR 002), which already carried the fact. The widget had been walking none_cats x Model::category(..).kind.is_regular() -- a second, hand-rolled copy of what project::cell_id decides.\n\nNOT DONE, DELIBERATELY: THE GRID'S FRAME. Six facts still come off GridLayout + View -- header level count (row_cats/col_cats), multi-level label TUPLES and inline GROUP HEADERS (row_items), column group spans (col_group_for), the collapse indicator (is_group_collapsed), and record_col (session-local by design, since it keys pending drill edits -- that one is CORRECT to keep off).\n\nTHE HEADLINE, AND IT IS A LIVE DIVERGENCE TODAY: GridCache::row_labels is GridLayout::row_label, which JOINS A MULTI-LEVEL TUPLE WITH '/'. That is exactly the rendering src/ui/grid.rs::two_row_categories_produce_cross_product_labels asserts must NOT appear. So the protocol carries a form of the labels the TUI considers wrong, and the wasm client renders precisely that form: the browser grid shows Food/Alice where the TUI shows two header columns, with no multi-level headers, no repeat suppression, no groups and no collapse indicators. Successor ticket filed -- it is a co-ordinated change (row_labels/col_labels change TYPE and improvise-wasm-client consumes them), and an additive row_levels beside row_labels is the wrong fix, because two answers to 'what are the labels' is the disease.\n\nUNVERIFIED: project::grid_of now evaluates every cell per frame where GridWidget evaluated only the visible ones. The totals loop was already O(rows x visible cols) and compute_col_widths was already whole-grid, so this WIDENS an existing cost rather than introducing one -- but on a wide grid it is a real widening and there is no benchmark. The fix would be to give the TUI a real viewport, which needs the width/window chicken-and-egg untangled first.","dependencies":[{"issue_id":"improvise-n10","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:36Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-35e","title":"Temporary adapter: RenderCache::from_app bridge for incremental widget migration","description":"Build a function that constructs a complete RenderCache from an \u0026App on every frame. This is a temporary bridge that lets ratatui widgets migrate from '\u0026App direct reads' to 'RenderCache consumer' one at a time, without breaking the build. Each migrated widget reads from the adapter's output; unmigrated widgets keep reading \u0026App. When every widget has been migrated and the binary split is complete, this adapter is deleted.","design":"pub fn RenderCache::from_app(app: \u0026App, subs: SubscriptionSet) -\u003e RenderCache. Walks the full App state and populates every subscribed-to sub-cache: grid cells/labels/widths, category tree, formula list, view list, tile bar axes, wizard state, etc. SubscriptionSet is the full set initially (native TUI subscribes to everything); later issues refine it per client. Called on every frame from run_tui before drawing. Unperformant by construction (rebuilds everything per frame) but correct, and it's throwaway code. Cache shape: pub struct RenderCache { grid: Option\u003cGridCache\u003e, category_tree: Option\u003cCategoryTreeCache\u003e, formulas: Option\u003cFormulaListCache\u003e, views: Option\u003cViewListCache\u003e, tile_bar: Option\u003cTileBarCache\u003e, wizard: Option\u003cWizardCache\u003e, help: Option\u003cHelpCache\u003e, ... }. Each field is Some if subscribed and populated from App data.","acceptance_criteria":"(1) RenderCache struct exists with optional sub-caches for every rendering surface. (2) from_app populates all subscribed sub-caches from current App state. (3) run_tui calls from_app once per frame before drawing. (4) Tests demonstrate the cache contents match what the corresponding widgets would read from App directly.","notes":"Throwaway code explicitly marked as such. Deleted after issue 7 (binary split) lands. Depends on issue 1 (reduce_full) only because it needs to know what the cache shape should be, which is determined by what widgets read.\nGROOMED 2026-09-19: the RenderCache that actually landed in improvise-cqi is grid-only (row_labels/col_labels/col_widths/cells) — not the 'grid + category_tree + formulas + views + tile_bar + wizard + help' shape this ticket's design describes. Blocked on improvise-rkeq until the shape is decided; writing from_app against the wrong one is the expensive mistake.\n--- DECIDED 2026-09-19 (user) ---\nRenderCache is PER-WIDGET, and state synchronizes by EVENTS, not snapshots.\n\nThis is shape (C) from improvise-rkeq, with per-widget caches:\n * No flat grid-only wire type and no fat aggregate container. Each widget\n owns its own cache type (GridCache, CategoryTreeCache, FormulaCache,\n ViewCache, TileBarCache, WizardCache, HelpCache).\n * Each widget subscribes to and FOLDS its own stream of projection\n Commands. There is no aggregate RenderCache on the wire.\n * The ONE exception is the initial state transfer: a newly connected\n client gets a Snapshot to seed from, then follows the event stream.\n Snapshot stays bootstrap-only (wire_carries(snapshot,bootstrap_only)).\n\nConsequence for this ticket: read \"RenderCache\" as \"that widget's own cache\ntype\", and expect updates to arrive as folded projection commands rather\nthan as a rebuilt snapshot.","status":"closed","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-14T07:48:38Z","created_by":"spot","updated_at":"2026-09-19T23:40:22Z","closed_at":"2026-09-19T23:40:22Z","close_reason":"PREMISE CHECK FIRST, and the ticket as written is DEAD: RenderCache::from_app(app, subs) -\u003e RenderCache cannot be written because that aggregate does not exist and will not (improvise-rkeq, gotcha 32). It was not written. The builders it asked for already existed and were already pub -- grid_of, category_rows, formula_texts, view_summaries, tile_entries -- and Dirty already answers the SubscriptionSet, so no parallel type was invented.\n\nTWO THINGS WERE GENUINELY MISSING, both real:\n1. THE ROOT CRATE COULD NOT NAME A CACHE TYPE. improvise had no improvise-protocol dependency and improvise-command re-exported core/formula/io but NOT protocol, so no ratatui widget in src/ could write fn new(cache: \u0026FormulaCache). That was the hard blocker and nothing complained because nobody had tried. Fixed by re-exporting the protocol through improvise-command (and the root crate), so a host names the caches through the crate that builds them and the root takes no new dependency.\n2. THE BUILDERS RETURNED PAYLOADS, NOT CACHES. Added category_tree_of/formulas_of/view_list_of/tile_bar_of, each one line over the existing payload fn so there is no second derivation. document_projections and the tests server_truth now both derive FROM the caches, so what a widget holds and what a client folds cannot disagree.\n\nThree widgets migrated off \u0026Model/\u0026View: formula panel, view panel, tile bar. No aggregate type and no adapter type -- a caller wanting several caches calls several builders, and no widget names the seam, so deleting the temporary half is deleting call sites.\n\n6 tests, each an EQUIVALENCE between the cache and what the widget used to read off the model (plus pinned render output where drawing has layout logic). Three faults injected into the builders, all fired. 970 workspace tests pass.","dependencies":[{"issue_id":"improvise-35e","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:34Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-35e","depends_on_id":"improvise-rkeq","type":"blocks","created_at":"2026-09-19T12:49:24Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-gxi","title":"Define unified reduce_full(\u0026mut App, \u0026Command) -\u003e Vec\u003cCommand\u003e","description":"Extract the single shared reducer function that all three deployment modes use to apply a Command to the authoritative state. Replaces today's App::handle_key → Cmd::execute → Effect::apply chain with a named function that takes a Command (data, not trait object), runs the command through the existing registry + effect pipeline, and returns any outbound projection commands. Load-bearing for the entire unified architecture — native TUI, ws-server, and worker-server all import this same function.","design":"Signature: pub fn reduce_full(app: \u0026mut App, cmd: \u0026Command) -\u003e Vec\u003cCommand\u003e. Internally: (1) resolve the Command to a Cmd trait object via the registry (or match directly on the Command enum), (2) build a CmdContext from the current App state, (3) run execute() to produce effects, (4) apply each effect to App, (5) walk the effect list and compute outbound projection Commands for each subscribing client (empty if no subscribers). The function lives in improvise-command (which requires App to move there too — see epic notes). Command enum is defined in improvise-protocol (improvise-cqi). The native TUI's existing App::handle_key becomes a thin wrapper: resolve the key to a Command via the keymap, call reduce_full, drop the returned projections (single-session native has no remote subscribers). The ws-server and worker-server call reduce_full directly on received Commands and dispatch the returned projections back over their respective transports.","acceptance_criteria":"(1) pub fn reduce_full exists in improvise-command. (2) App::handle_key is rewritten to resolve key → Command via keymap, then call reduce_full, then drop projections. (3) All existing native TUI behavior is preserved — integration and unit tests pass unchanged. (4) Keymap lookup returns Command values (data) rather than Box\u003cdyn Cmd\u003e trait objects. (5) CmdRegistry's dispatch API becomes 'dispatch_command(cmd: \u0026Command, ctx: \u0026CmdContext) -\u003e Vec\u003cEffect\u003e' or equivalent. (6) A unit test demonstrates reduce_full applied to a Command produces the same final App state as the old path.","notes":"Depends on improvise-cqi (Command enum defined in improvise-protocol) and on App having moved to improvise-command (a crate-epic step 5 detail that needs updating — see improvise-3mm notes). Cross-epic: improvise-cqq (browser projection layer) formally depends on this issue since it needs reduce_full to exist.\nGROOMED 2026-09-19 — PREREQUISITES DISSOLVED. The notes said this depends on \"App having moved to improvise-command (a crate-epic step 5 detail that needs updating — see improvise-3mm notes)\". improvise-3mm is closed and App already lives at crates/improvise-command/src/ui/app.rs, so that prerequisite is satisfied, not pending. improvise-cqi is closed too: Command, ViewState and reduce_view are in improvise-protocol, and improvise-command depends on it. Nothing blocks this now — it is in bd ready.\n\n--- PARTIALLY LANDED 2026-09-19; ACs 4-5 BLOCKED, deliberately not forced ---\nDONE: reduce_full(\u0026mut App, \u0026Command) -\u003e Vec\u003cCommand\u003e (AC 1), apply_command\n(the inner half a server uses to drive Sessions::project_all), reduce_key,\nand App::handle_key is now a wrapper over App::dispatch_key(key) -\u003e Dirty\n(AC 2 in spirit). All existing behaviour preserved (ACs 3, 6).\n\nNOT DONE, and not doable as written: ACs 4 and 5 ask that keymap lookup\nreturn Command values and that CmdRegistry gain dispatch_command(\u0026Command).\nThe Command enum's client-\u003eserver half is exactly the ViewOnly portable-slice\nmutations -- cqi derived it from those effects and nothing else -- so it has\nno SetCell, AddCategory, SetAxis or Save. A keymap cannot return a Command for\nmost bindings, and reduce_full over a Command cannot drive a model mutation.\nForcing it would mean inventing variants ahead of the mutations, which\ndecision(improvise-cqi, command_set_derived_not_invented) and\ndoctrine(derive_the_enum_from_the_effects) both forbid.\n\nBLOCKED ON: the new ticket to derive the model half of Command from the 29\nEffectKind::Model effects. Until then reduce_full and reduce_key are two\ndoors and ui/reduce.rs's module doc explains why.\n\n--- AC 4 IS UNREACHABLE BY DESIGN, not for want of vocabulary (2026-09-19) ---\nimprovise-kmgs landed ModelCommand, so the \"the vocabulary is missing\" reason\nis gone. AC 4 still cannot be met, and the sharper reason is worth recording:\n\na keymap binding names an INTENT, resolved against CmdContext -- \"move-cursor\ndown\". A Command is the RESOLVED MUTATION -- SetSelected { row, col } -- which\nonly exists once the cursor and layout have been read. So a keymap LOOKUP can\nnever return a Command; only dispatch, after CmdContext, can. That is a\nproperty of the Cmd/Command split itself, not a gap.\n\nRECOMMEND: rewrite AC 4 to \"dispatch (not lookup) yields a Command where the\nbinding resolves to a model mutation\", or drop it. Do not force the original\nwording.\n\nAC 5 is satisfied in substance: model_effects(\u0026ModelCommand) -\u003e Vec\u003cBox\u003cdyn\nEffect\u003e\u003e in ui/reduce.rs is the \"or equivalent\". It was deliberately NOT hung\noff CmdRegistry as dispatch_command, because the mapping needs neither the\nregistry nor a CmdContext and a method ignoring \u0026self would misdescribe it.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:48:36Z","created_by":"spot","updated_at":"2026-09-19T23:29:48Z","started_at":"2026-09-19T21:15:06Z","closed_at":"2026-09-19T23:29:48Z","close_reason":"CLOSED, re-scoped: ACs 1-3 and 6 delivered; ACs 4-5 resolved as written-against-a-false-premise.\n\nDELIVERED (wave 2, merged): reduce_full(\u0026mut App, \u0026Command) -\u003e Vec\u003cCommand\u003e, apply_command (the inner half a server uses to drive Sessions::project_all), reduce_key, and App::handle_key as a wrapper over App::dispatch_key(key) -\u003e Dirty. All existing behaviour preserved.\n\nAC 5 satisfied in substance: model_effects(\u0026ModelCommand) -\u003e Vec\u003cBox\u003cdyn Effect\u003e\u003e in ui/reduce.rs is the \"or equivalent\". Deliberately NOT hung off CmdRegistry as dispatch_command, because the mapping needs neither the registry nor a CmdContext and a method ignoring \u0026self would misdescribe it.\n\nAC 4 IS UNREACHABLE BY DESIGN, and the reason is sharper than \"the vocabulary is missing\" (improvise-kmgs supplied the vocabulary and it is still unreachable): a keymap binding names an INTENT resolved against CmdContext -- \"move-cursor down\" -- while a Command is the RESOLVED MUTATION -- SetSelected { row, col } -- which only exists once the cursor and layout have been read. So a keymap LOOKUP can never return a Command; only dispatch, after CmdContext, can. That is a property of the Cmd/Command split itself. Forcing it would mean either inventing intent-shaped Command variants (which decision(improvise-cqi, command_set_derived_not_invented) forbids) or dissolving the split.\n\nClosing rather than leaving open: it blocks improvise-cqq and the 35e -\u003e edp -\u003e cr3 chain to improvise-1ey, which is now the priority, and there is no remaining work here that a future ticket would not state better. If dispatch-yields-a-Command is wanted, file it as its own ticket against the real question.","dependencies":[{"issue_id":"improvise-gxi","depends_on_id":"improvise-cqi","type":"blocks","created_at":"2026-08-29T12:37:17Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":5,"comment_count":0} -{"_type":"issue","id":"improvise-ltq","title":"Epic: Native TUI as in-process client of the unified server architecture","description":"Restructure the native binary so the ratatui UI is a client of an in-process server, exactly the way the DOM renderer is a client of the service worker in the standalone deployment and of the ws-server in the network deployment. One binary process, two halves: a 'server' holding the full App + projection layer, and a 'client' holding ViewState + render cache + ratatui widgets. Commands flow through a direct in-process channel (same shape as websocket/postMessage protocols, zero serialization overhead). The server emits projection commands back to the client's cache. Result: the ratatui path is architecturally identical to the browser paths — same reduce_full, same Command vocabulary, same cache-based rendering. Widgets stop reading \u0026App directly.","design":"Four deployment modes all share one architecture: (1) native TUI = in-process client + server with direct-call transport, (2) network thin-client = browser main thread + ws-server with websocket transport, (3) standalone web = browser main thread + worker with postMessage transport, (4) hybrid = in-process native client + server PLUS remote ws-server subscriber = native TUI with remote browser observer. Under this epic, mode (1) gets restructured to match (2) and (3). The shared pieces: reduce_full, projection layer with per-client subscriptions, Command vocabulary, render cache shape. The difference per mode is only the transport + which clients subscribe to which projection kinds. Native TUI subscribes to every projection kind because it renders everything (grid, panels, help, wizard); browser thin-client MVP subscribes to grid only. Staging: introduce a temporary RenderCache::from_app adapter, migrate widgets one at a time with the adapter bridging, split the binary and delete the adapter at the end. Every widget migration is individually landable.","notes":"Implications for other epics: (A) improvise-cqq (browser epic projection layer) gains a prerequisite on issue 1 of this epic (unified reduce_full). (B) improvise-3mm (crate-epic step 5) needs its design updated to put App in improvise-command, not improvise-tui, so reduce_full can live alongside App without pulling ratatui into ws-server or worker-server. (C) improvise-cr3 (browser DOM renderer) and issue 3 of this epic (ratatui grid cache consumer) share the RenderCache grid shape and can share the GridViewModel type — coordinate the two to avoid duplication. Stretch issues cover hybrid mode (native TUI + remote browser subscriber) and native undo/replay via command logging.\nGROOMED 2026-09-19: implication (B) in the description is done — improvise-3mm closed with App in improvise-command, which is where this epic wanted it. Implication (A) is recorded as a real edge (cqq depends on gxi), so it no longer needs to live in prose.","status":"open","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:48:34Z","created_by":"spot","updated_at":"2026-09-19T19:49:04Z","dependencies":[{"issue_id":"improvise-ltq","depends_on_id":"improvise-35e","type":"blocks","created_at":"2026-04-14T00:53:49Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-764","type":"blocks","created_at":"2026-04-14T00:53:52Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-e0u","type":"blocks","created_at":"2026-04-14T00:53:52Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:49Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:48Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-jb3","type":"blocks","created_at":"2026-04-14T00:53:51Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-n10","type":"blocks","created_at":"2026-04-14T00:53:50Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-pca","type":"blocks","created_at":"2026-04-14T00:53:50Z","created_by":"spot","metadata":"{}"}],"dependency_count":8,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-ltq","title":"Epic: Native TUI as in-process client of the unified server architecture","description":"Restructure the native binary so the ratatui UI is a client of an in-process server, exactly the way the DOM renderer is a client of the service worker in the standalone deployment and of the ws-server in the network deployment. One binary process, two halves: a 'server' holding the full App + projection layer, and a 'client' holding ViewState + render cache + ratatui widgets. Commands flow through a direct in-process channel (same shape as websocket/postMessage protocols, zero serialization overhead). The server emits projection commands back to the client's cache. Result: the ratatui path is architecturally identical to the browser paths — same reduce_full, same Command vocabulary, same cache-based rendering. Widgets stop reading \u0026App directly.","design":"Four deployment modes all share one architecture: (1) native TUI = in-process client + server with direct-call transport, (2) network thin-client = browser main thread + ws-server with websocket transport, (3) standalone web = browser main thread + worker with postMessage transport, (4) hybrid = in-process native client + server PLUS remote ws-server subscriber = native TUI with remote browser observer. Under this epic, mode (1) gets restructured to match (2) and (3). The shared pieces: reduce_full, projection layer with per-client subscriptions, Command vocabulary, render cache shape. The difference per mode is only the transport + which clients subscribe to which projection kinds. Native TUI subscribes to every projection kind because it renders everything (grid, panels, help, wizard); browser thin-client MVP subscribes to grid only. Staging: introduce a temporary RenderCache::from_app adapter, migrate widgets one at a time with the adapter bridging, split the binary and delete the adapter at the end. Every widget migration is individually landable.","notes":"Implications for other epics: (A) improvise-cqq (browser epic projection layer) gains a prerequisite on issue 1 of this epic (unified reduce_full). (B) improvise-3mm (crate-epic step 5) needs its design updated to put App in improvise-command, not improvise-tui, so reduce_full can live alongside App without pulling ratatui into ws-server or worker-server. (C) improvise-cr3 (browser DOM renderer) and issue 3 of this epic (ratatui grid cache consumer) share the RenderCache grid shape and can share the GridViewModel type — coordinate the two to avoid duplication. Stretch issues cover hybrid mode (native TUI + remote browser subscriber) and native undo/replay via command logging.\nGROOMED 2026-09-19: implication (B) in the description is done — improvise-3mm closed with App in improvise-command, which is where this epic wanted it. Implication (A) is recorded as a real edge (cqq depends on gxi), so it no longer needs to live in prose.\nDEFERRED 2026-09-19: the epic closes when improvise-e0u closes, and e0u is deferred by user ruling this session. All of ltq's other children are already closed. Nothing to do here until e0u gets its own campaign.","status":"deferred","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:48:34Z","created_by":"spot","updated_at":"2026-09-20T05:31:01Z","dependencies":[{"issue_id":"improvise-ltq","depends_on_id":"improvise-35e","type":"blocks","created_at":"2026-04-14T00:53:49Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-764","type":"blocks","created_at":"2026-04-14T00:53:52Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-e0u","type":"blocks","created_at":"2026-04-14T00:53:52Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:49Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:48Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-jb3","type":"blocks","created_at":"2026-04-14T00:53:51Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-n10","type":"blocks","created_at":"2026-04-14T00:53:50Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-ltq","depends_on_id":"improvise-pca","type":"blocks","created_at":"2026-04-14T00:53:50Z","created_by":"spot","metadata":"{}"}],"dependency_count":8,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i34","title":"OPFS-backed Storage implementation for wasm target","description":"Implement the Storage trait (or vfs backend) against the browser's Origin Private File System. Provides read/write/list operations from wasm that work across Chrome, Firefox, and Safari. Falls back to an in-memory or IndexedDB-backed implementation if OPFS is unavailable in the target environment.","design":"Use web-sys to access navigator.storage.getDirectory() and the File System Access API. OPFS is async-only, so either (a) make the Storage trait async-compatible (preferred if we switch to an async trait in the abstraction issue) or (b) wrap OPFS calls in a queued executor so the sync trait API can block on completion via a yield loop. Option (a) is cleaner; the abstraction issue (improvise-6mq) should probably commit to async up front if OPFS is the target. Directory layout: one root directory under OPFS for improvise, with subdirectories per purpose (autosave, user files, session state). File picker integration: when the user says 'open a file' in the browser, use the native file picker and either copy the file into OPFS or open it directly via the File System Access API's handle-based API. Fallback: if OPFS is unavailable (older Safari, some privacy modes), provide an IndexedDB-backed implementation using the idb crate — exposes the same trait.","acceptance_criteria":"(1) Storage implementation for wasm target exists, keyed off a compile-time target check. (2) Round-trips: write bytes, read bytes, list files, delete file. (3) Manual test: load a .improv file via file picker, edit in the DOM renderer, save to OPFS, reload page, file is still there. (4) Unit tests using a memory-backed stub where OPFS is unavailable (CI).","notes":"Depends on improvise-6mq (Storage abstraction must exist first). May reveal a need to change the abstraction from sync to async, in which case update 6mq before starting this.\nUnblocked by improvise-6mq (Storage trait landed 2026-08-29: crates/improvise-io/src/storage.rs, two methods read/write on opaque document paths, PhysicalFs + MemoryFs). NOTE: still not startable in the current dev environment — no wasm32-unknown-unknown std in the Nix toolchain and no rustup. See improvise-ywd notes.\nToolchain unblocked 2026-08-30: wasm32-unknown-unknown added to the flake (improvise-ywd), and all four bundle crates check clean. See improvise-fuj first — the two effects that still call std::fs directly need a decision on how a wasm build injects its Storage backend, which is the same decision this ticket needs.\nSYNC-VS-ASYNC RESOLVED (2026-08-30) — the abstraction does NOT need to become async, so improvise-6mq stands as built.\n\nThe DESIGN here says 'OPFS is async-only' and the NOTES say this 'may reveal a need to change the abstraction from sync to async, in which case update 6mq before starting'. That premise is incomplete. OPFS exposes two APIs:\n\n - the promise-based one (getDirectory, getFileHandle, createWritable) — async\n - FileSystemFileHandle.createSyncAccessHandle() -\u003e FileSystemSyncAccessHandle,\n whose read/write/truncate/flush/close are SYNCHRONOUS, and which is available\n ONLY inside a dedicated Web Worker. It exists precisely so wasm can do\n blocking file I/O; it is the mechanism SQLite's WASM build uses for its OPFS\n VFS.\n\nimprovise-9x6 puts 'the full App + command pipeline + projection layer + VFS persistence inside a browser worker'. So the persistence layer never runs on the main thread, and the sync path is available to it.\n\nShape that falls out:\n - ASYNC, once, at worker init(): navigator.storage.getDirectory(), getFileHandle(), createSyncAccessHandle().\n - SYNC, thereafter: the Storage impl holds the access handles; read/write are ordinary blocking calls, satisfying the existing two-method trait unchanged.\n\nThis avoids an invasive async migration through persistence::save/load and every caller, which is the outcome 6mq's Option-B choice was already hedging toward.\n\nCAVEAT: asserted from API knowledge, not tested — there is no browser here. Confirm createSyncAccessHandle availability against the target browsers (it is the weakest on Safari) before relying on it. The IndexedDB fallback this ticket already calls for is the hedge, and IndexedDB IS async-only — so the fallback, not OPFS, is what would force the async question. Decide whether the fallback is in scope before starting.\n\nPREREQUISITE: improvise-fuj — how a wasm build injects its Storage instance. App holds no Storage today; persistence::save/load hardwire PhysicalFs, and two effects bypass the trait entirely. That plumbing is host-verifiable and should land first.","status":"closed","priority":2,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:34:08Z","created_by":"spot","updated_at":"2026-09-01T06:11:09Z","started_at":"2026-09-01T06:06:19Z","closed_at":"2026-09-01T06:11:09Z","close_reason":"Storage gained list/delete (PhysicalFs + MemoryFs, shared storage_contract test). OpfsFs added for wasm32 using sync access handles, keeping the trait synchronous — createSyncAccessHandle's read/write/truncate/flush/close are blocking, worker-only, which suits improvise-9x6. Async work is four explicit methods: open/attach/refresh/commit_deletes. delete is deferred (index updated immediately, removeEntry on commit). No IndexedDB fallback per user decision — verified Safari 16.4+ has sync handle methods, so it isn't needed. AC(1)(2)(4) met; AC(3) is a browser test that cannot run here, tracked as improvise-xf1 and blocking improvise-9x6.","dependencies":[{"issue_id":"improvise-i34","depends_on_id":"improvise-6mq","type":"blocks","created_at":"2026-04-14T00:40:10Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-i34","depends_on_id":"improvise-fuj","type":"blocks","created_at":"2026-08-30T09:57:13Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ywd","title":"Wasm compatibility audit and fixes across core/formula/command/io","description":"Verify every crate that belongs in the wasm bundle builds for wasm32-unknown-unknown and runs correctly. Fix anything that doesn't: feature-gate wall-clock time, swap Instant for web_time, enable chrono wasmbind, isolate dirs crate usage, verify pest/serde/indexmap/flate2/enum_dispatch all compile. Ship a cargo-check invocation per crate as a CI gate.","design":"Systematic audit: for each crate (improvise-core, improvise-formula, improvise-command, improvise-io-persistence), run cargo check --target wasm32-unknown-unknown and fix what breaks. Known hazards: (1) dirs crate — not wasm-compatible, isolate behind a trait function. (2) chrono::Local::now() — needs wasmbind feature, used in import analyzer for date parsing. (3) std::time::Instant — swap for web_time crate which falls back to js performance.now() in wasm. (4) std::fs in persistence — handled by the VFS abstraction issue. (5) std::env and std::process — probably none in core, verify. (6) flate2 — verify rust_backend feature works in wasm (should). Strategy: don't feature-gate everything individually, instead do the minimal isolation needed so each crate's public API is portable, and let the downstream bundle crate (improvise-web-standalone) opt in to what it actually needs.","acceptance_criteria":"(1) cargo check --target wasm32-unknown-unknown -p improvise-core passes. (2) Same for improvise-formula, improvise-command, and improvise-io (or improvise-persistence if split). (3) No new feature flags required to get these to build — the core path is wasm-compatible by default. (4) CI job added that runs the wasm checks on every push.","notes":"Depends on the full crate split (epics 1-5) being done — the audit needs each crate to exist independently. Does not depend on the VFS abstraction issue (std::fs concerns are handled there). These two can proceed in parallel.\nTOOLCHAIN-BLOCKED in this dev environment (checked 2026-08-29). The deliverable is 'cargo check --target wasm32-unknown-unknown' per crate plus a CI gate, but the Nix toolchain here ships no wasm32 std and there is no rustup to add it:\n\n $ cargo check -p improvise-core --target wasm32-unknown-unknown\n error[E0463]: can't find crate for 'std'\n note: the wasm32-unknown-unknown target may not be installed\n $ which rustup -\u003e not found\n\nSource changes could be written speculatively (Instant -\u003e web_time, chrono wasmbind, isolating dirs) but none could be VERIFIED, which is the whole point of an audit. Adding wasm32-unknown-unknown to the flake's rust toolchain is the actual first step of this ticket.\n\nSame constraint hits improvise-cqi acceptance criterion 5 ('builds for both host target and wasm32-unknown-unknown').","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:33:49Z","created_by":"spot","updated_at":"2026-08-30T16:51:24Z","started_at":"2026-08-30T16:47:28Z","closed_at":"2026-08-30T16:51:24Z","close_reason":"AC 1-3 turned out to need NO source changes; the blocker was the toolchain, exactly as the notes I left on 2026-08-29 predicted.\n\nAdded targets = [\"wasm32-unknown-unknown\"] to the flake's rustToolchain (flake.nix:26) and verified wasm32 std lands in the sysroot. Then all four bundle crates check clean, first try, with no feature flags:\n\n improvise-core ok\n improvise-formula ok\n improvise-io ok\n improvise-command ok\n\nThat satisfies AC 1, AC 2 and AC 3 (AC 3 explicitly: 'no new feature flags required — the core path is wasm-compatible by default'). The root improvise crate does NOT build for wasm because crossterm does not; that is correct — it is the terminal binary, not part of the bundle — so the CI job checks the four bundle crates by name rather than the workspace.\n\nAC 4: .github/workflows/ci.yml adds a wasm job plus a host job (fmt, clippy -D warnings, test). There was no test/lint CI before this — only cargo-dist's release.yml and the Claude bots. Verified clippy -D warnings and fmt pass locally so the job is not red on arrival, and validated the workflow YAML parses.\n\nTHE AUDIT'S REAL VALUE WAS THE HAZARD LIST, because a cargo check gate cannot catch anything that merely panics at runtime. Going through the DESIGN section's named hazards one by one:\n\n (1) dirs NON-ISSUE — declared at root Cargo.toml:36, used nowhere (already noted on improvise-v9w)\n (2) chrono Local::now NON-ISSUE — never called\n (3) std::time::Instant REAL — App::last_autosave; Instant::now() panics on wasm32-unknown-unknown yet compiles fine. FIXED: improvise-command now uses web-time, which re-exports std::time off-wasm and performance.now() on it. Verified both targets build and all 14 native suites pass.\n (4) std::fs REAL, NOT FIXED — StartImportWizard (effect.rs:931) and LoadModel (effect.rs:1052) still call std::fs::read_to_string directly, bypassing the Storage trait from improvise-6mq. Compiles on wasm, fails in a browser. Filed as improvise-fuj, because the fix needs a decision on how a wasm build injects its Storage backend — which also determines how improvise-i34 plugs in, so it should not be guessed at here.\n (5) std::env/process NON-ISSUE — std::env appears only in a test\n (6) flate2 NON-ISSUE — compiles for wasm as-is\n\nCaveat worth stating plainly: 'compiles for wasm32' is all that has been verified. Nothing here has been RUN on a wasm host — there is no browser or wasm runtime in this environment — so the web-time fix is verified by dual-target compilation and unchanged native behaviour, not by execution. Actual runtime verification arrives with improvise-i34 / improvise-tm6.","dependencies":[{"issue_id":"improvise-ywd","depends_on_id":"improvise-3mm","type":"blocks","created_at":"2026-04-14T00:40:09Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-6mq","title":"Introduce Storage/VFS abstraction in persistence layer","description":"Refactor persistence/mod.rs to go through a Storage trait instead of std::fs directly. Native build wires PhysicalFS; later issues wire OPFS for wasm. Behavior-preserving refactor — .improv files still save and load identically on native. Foundation for the standalone web deployment and for alternative formats (sqlite, parquet) that can target any backend.","design":"Option A: adopt the vfs crate directly (vfs::VfsPath, vfs::PhysicalFS). Pros: batteries included, filesystem-path-based API, open-source implementations. Cons: sync-only; wasm backends may need async. Option B: define our own narrow Storage trait with read_bytes/write_bytes/list/stat methods, sync for native and async-wrapped for wasm. Pros: tailored to our needs. Cons: more code to maintain. Recommend starting with Option A (vfs crate) and falling back to Option B if OPFS's async-only nature forces the issue. Refactor persistence::{save_md, load_md, save_json, load_json} to take \u0026dyn Storage (or VfsPath) as the target parameter. Autosave path resolution (currently uses the dirs crate) goes behind the abstraction too — native resolves to a dirs path, wasm resolves to a fixed OPFS path.","acceptance_criteria":"(1) Persistence functions take a storage parameter rather than reading/writing files directly. (2) Native build wires PhysicalFS (or equivalent) and all existing persistence tests pass unchanged. (3) dirs crate usage is isolated behind a function that can be stubbed for wasm. (4) Round-trip tests still pass. (5) A memory-backed test fixture exists for unit testing persistence without touching disk.","notes":"Depends on crate-split epic step 3 (improvise-8zh — improvise-io extraction) so the refactor happens in the right crate. Valuable independently of wasm: enables testing persistence without tempfile, simplifies fuzzing.","status":"closed","priority":2,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:33:47Z","created_by":"spot","updated_at":"2026-08-29T19:40:58Z","started_at":"2026-08-29T19:37:37Z","closed_at":"2026-08-29T19:40:58Z","close_reason":"persistence::save/load now go through a Storage trait; save/load remain thin PhysicalFs wrappers so every caller and all 83 persistence tests are untouched (AC 2).\n\nCHOSE OPTION B OVER THE RECOMMENDED OPTION A. The design section recommends starting with the vfs crate. I measured the actual surface first: persistence touches the filesystem in exactly two places — write the whole document in save, read the whole document in load. autosave_path is pure path manipulation with no I/O. That does not justify a dependency plus a VfsPath abstraction. The ticket's own con for Option A is decisive: it is sync-only while OPFS, the actual wasm target, is async — adopting A would mean doing B later anyway. The trait is two methods:\n\n fn read(\u0026self, path: \u0026Path) -\u003e Result\u003cVec\u003cu8\u003e\u003e\n fn write(\u0026self, path: \u0026Path, bytes: \u0026[u8]) -\u003e Result\u003c()\u003e\n\nPaths are opaque document names, not a directory tree, so an OPFS or map-backed backend implements it without pretending to have directories, metadata or seeking.\n\nAC 3 IS VACUOUS: 'dirs crate usage is isolated behind a function that can be stubbed for wasm' — there is no dirs usage. dirs = \"6\" is declared at root Cargo.toml:36 and referenced nowhere under src/ or crates/*/src/. It is simply an unused dependency; noted on improvise-v9w, which already covers removing unused root deps.\n\nGzip moved from streaming File+GzEncoder to encode-then-write. The .gz suffix still selects it, which makes gzip a property of the document name rather than of the backend. physical_and_memory_backends_write_identical_bytes pins that swapping backends does not change the bytes.\n\nAC 5 (memory-backed fixture): MemoryFs, plus three persistence tests that round-trip a workbook with categories, items and data through it — no tempfile, no disk. That is the shape a browser backend plugs into.\n\nVerified beyond unit tests: import to .improv, headless add-cat against it, and the .gz path all still work through the CLI.","dependencies":[{"issue_id":"improvise-6mq","depends_on_id":"improvise-8zh","type":"blocks","created_at":"2026-04-14T00:40:08Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":5,"comment_count":0} @@ -149,21 +150,25 @@ {"_type":"issue","id":"improvise-pby","title":"2.2 Write the README","description":"Replace existing README.md with new one under 250 lines. Sections in order: Title, one-sentence pitch, inline demo GIF, why this exists (Lotus Improv reference), quick start, key bindings, installation (Nix/crates.io/prebuilt - no Homebrew), codebase overview, expectations disclaimer, license. No badges, TOC, or contributing guide.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:32Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:02:52Z","close_reason":"README written with all 10 required sections, committed. License updated to Apache-2.0 per user request.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ihv","title":"2.1 Add Nix tooling for asciinema, VHS, and cargo-dist","description":"Modify flake.nix to add pkgs.asciinema and pkgs.vhs to dev shell nativeBuildInputs. Add cargo-dist if packaged in nixpkgs. Add runtime deps for VHS (ttyd, ffmpeg) if needed. Verify tools work via nix develop --command. Add nix run app or shell script for demo recording workflow.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:30Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:30:19Z","close_reason":"asciinema, vhs, cargo-dist added to flake.nix dev shell and verified. cargo-dist wrapper enables 'cargo dist' subcommand. scripts/record-demo.sh added.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-yk4","title":"Phase 2: README and demo artifacts","description":"The main launch work. README is 80% of the launch, demo artifacts are 20%. Covers Nix tooling for asciinema/VHS, writing the README, creating demo GIF and asciinema casts, and verifying all artifacts.","status":"closed","priority":2,"issue_type":"feature","assignee":"spot","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:49Z","created_by":"Edward Langley","updated_at":"2026-04-14T07:47:43Z","closed_at":"2026-04-14T07:47:43Z","close_reason":"All 5 subtasks closed; artifacts verified present and tracked in git (README.md, docs/demo.gif, docs/demo.tape, docs/casts/*.cast).","dependencies":[{"issue_id":"improvise-yk4","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T23:37:37Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-ihv","type":"blocks","created_at":"2026-04-08T23:37:34Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-odx","type":"blocks","created_at":"2026-04-08T23:37:35Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-pby","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-j7v6","title":"The typed-string classification lives in two crates, and its guard test asserts by hand","description":"Named by improvise-r4lo (PR #21) while deleting the OTHER duplication, and deliberately left rather than folded in -- r4lo was about two answers on the CLIENT side, and deleting Client::set_cell/set_cell_at settled that. This is the duplication one level out, and it crosses a crate boundary.\n\nwasm-client/src/socket.rs::typed classifies a user-typed string into a CellValue. So does improvise-command's commit_regular_cell_value, on the host's own key path. Two implementations of \"what does this string mean\", in two crates, with nothing holding them together.\n\nBOTH ARE LEGITIMATE, which is why this is not simply r4lo again. The browser has no App to ask and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; a host typing locally has an App and classifies there. The defect is that the RULE is written twice, not that it is applied in two places.\n\nTHE TEST THAT LOOKS LIKE A GUARD AND IS NOT: a_typed_string_is_classified_as_the_host_classifies_it states the obligation IN ITS NAME and does not discharge it -- it asserts three outcomes BY HAND rather than comparing against the host's function. So the two can drift and the test stays green, which is this repository's characteristic defect (doctrine(check_the_checker)): a test that passes while the thing it names is broken.\n\nTHE FIX: one classification function in improvise-protocol, which both crates already depend on, and both call sites calling it. Then the test either disappears (type identity replaces it) or becomes a real comparison. Note doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished) -- if the shared function lands and the hand-written three-outcome test survives beside it, the consolidation is not done.\n\nWATCH ONE THING WHILE MOVING IT: the classification must stay consistent with what the .improv format considers a number versus pipe-quoted text (gotcha 34, and format.rs::pipe_quote exists precisely so a text cell cannot read back as a number). A shared classifier that disagrees with the persistence layer would be a third answer wearing the badge of the first.\n","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rucf.3","title":"Projection freshness for a source an external writer can change","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent.\n\nADR 001 governs concurrent EDITS. A live-backed model is read-only, so it has none and the basis check does not apply to it -- the guarantee is not voided, it is scoped to documents improvise owns, which is what it always meant.\n\nWHAT REMAINS IS A DIFFERENT AND SMALLER QUESTION, and this bead is it: an EXTERNAL WRITER changes what clients SEE. So a live-backed projection needs a freshness story -- polled, triggered, or explicitly stale-until-refreshed. That is projection freshness, not conflict resolution, and conflating the two is the trap.\n\nNOTE WHAT CHANGES ABOUT THE PROJECTION ARCHITECTURE: today's diff (ui/project.rs::grid_updates) compares a desired state derived from the App against each client's mirror. With a live source, \"the desired state\" involves latency -- a HashMap lookup becomes a query -- so the VIEWPORT-BOUNDED PROJECTION STOPS BEING AN OPTIMISATION AND BECOMES LOAD-BEARING. improvise-i5tk (narrow the TUI's GridCache to its viewport, benchmark first) is the related bead and its answer may be forced by this one.\n\nSHARED WITH THE SQLITE SOURCE CHILD: the RecordHash -\u003e record index becomes a cache needing invalidation rather than a structure maintained alongside writes. One invalidation story, not two.\n\nEXPLICITLY OUT OF SCOPE: anything resembling a lease, a lock, or a reservation. Those are ADR 001 layer 2 and they are about edits.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:35Z","dependencies":[{"issue_id":"improvise-rucf.3","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rucf.2","title":"SQLite source implementation: aggregate pushdown over a user-supplied query, native-only and feature-gated","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent.\n\nThe SQLite implementation of the source trait: aggregate pushdown against a user-supplied query.\n\nWHICH ROWS: a user-supplied SELECT, with a bare table name as sugar over it. A query is strictly more useful (joins, filters, computed columns) and no harder to consume -- a result set is a result set.\n\nTYPE MAPPING, and it is BETTER INPUT THAN CSV rather than merely different: a SQLite column carries a declared type, where import/analyzer.rs::detect_date_format currently guesses a date by pattern-matching text. INTEGER/REAL are measure candidates, TEXT a category candidate, BLOB skip. NULL is ABSENCE, not the string \"null\" (improvise-ych). Dates may be TEXT, INTEGER (epoch) or REAL (julian) -- DETECT, DO NOT ASSUME. SQLite is dynamically typed per value, so both the declared-type and the sampled path are real and the classifier should take typed input where it has it.\n\nWHERE THE BOUNDARY GOES: forbidden(improvise_core, io). The source TRAIT belongs in improvise-core; this implementation belongs in improvise-io, FEATURE-GATED AND NATIVE-ONLY. libsqlite3-sys does not build for wasm32-unknown-unknown under this repo's nix cc-wrapper, and improvise-io is in the wasm bundle, so the wasm CI job must keep passing untouched. VERIFY THE GATE, do not assume it -- the improvise-9ix agent established the wasm failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating protects the browser bundle rather than working around an impossibility.\n\nNO WRITES, and not as a runtime refusal: per ADR 003 writes are not in the trait a query source implements, and a source projecting no CellId::Record makes a client write unconstructible.\n\nINTERNING AND IDENTITY: DataStore interns names and ADR 002 hashes the coordinate tuple. Both still work -- coordinates come from the query -- but the RecordHash -\u003e record index that resolves a CellId becomes A CACHE WITH AN INVALIDATION STORY rather than a structure maintained alongside writes. That invalidation is shared with the freshness child; do not solve it twice.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:32Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:34Z","dependencies":[{"issue_id":"improvise-rucf.2","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rucf.1","title":"The .improv source specification: a live-backed document is a saved query plus a pivot configuration","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent; the direction is accepted but not a priority.\n\nA live-backed document splits into OUR configuration and THEIR data, and .improv does not currently distinguish them. Today a file holds categories, views, formulas AND a ## Data section. A live-backed model's categories and cell values belong to the source; the views, formulas and axis assignments belong to improvise.\n\nSo the format grows a SOURCE SPECIFICATION where a live model would otherwise have data: a connection, a query, and the column-to-field mapping. Categories become DERIVED (SELECT DISTINCT) rather than declared.\n\nWHY THIS IS THE FIRST CHILD: persistence/improv.pest is the single source of truth (design-principles 3) and the parser is a tree-walker over it, so the grammar changes before the walker. The grammar-walking test generator reads the grammar AST via pest_meta, so a new production needs generator coverage or the round-trip proptests silently stop describing the format.\n\nTHE FRAMING THAT MAKES THIS FEEL RIGHT RATHER THAN LIKE A COMPLICATION, from the ADR: a live-backed .improv is A SAVED QUERY PLUS A PIVOT CONFIGURATION -- which is exactly what a saved report is in any BI tool. The document stops being \"my data\" and becomes \"my question\", portable between databases in a way a data file never is.\n\nOPEN: whether a document may be PARTLY live (some categories declared, some derived) or whether live and owned are exclusive per document. Exclusive is almost certainly right and much cheaper; say so deliberately rather than leaving it implied.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:33Z","dependencies":[{"issue_id":"improvise-rucf.1","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:31Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ktv1","title":"Move the help content into the shared layer so a browser can show help","description":"? and F1 now enter AppMode::Help in the browser -- the shared keymap delivered that for free -- BUT THE BROWSER HAS NO HELP OVERLAY, so the mode changes and nothing appears. src/ui/help.rs is ~600 lines of ratatui and nothing shared carries its content.\n\nTHE INTERIM, already shipped: the page draws AppMode::hint_text() in a #hint element, so each mode states its own keys. That is genuinely useful and is not a placeholder to rip out -- but it is one line, not five pages.\n\nWHY IT WAS DONE THAT WAY RATHER THAN SPECIAL-CASING: the alternative was to make the client refuse \"enter-mode help\", which is exactly the hand-written exception the keymap consolidation had just deleted. Declining to re-add one was right.\n\nTHE REAL FIX is the same shape as every other consolidation this week (design-principles 5): the help CONTENT is data -- pages of (key, description) -- and it belongs in improvise-protocol where both front ends read it. The TUI keeps its ratatui LAYOUT; the browser writes its own DOM; neither restates what the keys are.\n\nNOTE THE OPPORTUNITY: help content that lives beside the keymap can be DERIVED from it rather than written twice, which would close a drift nobody currently checks -- there is no test that the help text and the bindings agree, and doctrine(binding_dump_diff) exists because bindings have almost no direct tests. A help page generated from Keymaps::default_keymaps() cannot go stale.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-e24v","title":"Second tranche of client-side key resolution: ~15 bindings needing no enrichment","description":"improvise-bpq2 closed the cursor half and DELIBERATELY STOPPED rather than half-doing a second tranche. These need NOTHING added to the protocol -- the operands are already in ViewState or a widget cache -- so they are the cheapest remaining keys, and each one currently does NOTHING in a browser (Host::handle answers ClientFrame::Key with an empty vec).\n\n * enter-mode \u003cx\u003e (: ? F1 C F V Esc Tab in most modes) -\u003e ChangeMode\n * toggle-panel-visibility / toggle-panel-and-focus -\u003e SetPanelOpen + ChangeMode\n * cycle-panel-focus\n * help-page-next / help-page-prev\n * transpose, toggle-prune-empty, view-back\n * switch-view-at-cursor / delete-view-at-cursor / create-and-switch-view (ViewListCache is flat)\n * commit-buffer category|formula|item (all operands in ViewState)\n * commit-cell-edit (ADR 002 CellId + the buffer)\n * clear-cell / paste (CellId + ViewState::yanked)\n * set-axis-for-tile / cycle-axis-for-tile\n\nALSO IN REACH, and bpq2 listed them under \"needs enrichment\" before establishing they do not: filter-to-item (Enter), open-item-add-at-cursor (a/o), delete-category-at-cursor (d), cycle-axis-at-cursor (Space) -- CatTreeEntry already carries the name and the axis. Each is a distinct Command shape, which is why they were left.\n\nMETHOD IS SETTLED, follow it: mirror each Cmd LINE FOR LINE including where siblings disagree, and add a lockstep test that drives the client and the real reduce_key over the same keypresses and compares ViewState. Do not tidy sibling commands into one rule -- that makes the client a second derivation, which is what lockstep testing exists to rule out.\n\nAnd update the_normal_mode_keys_that_still_need_the_host_are_these as you go: it is a named list precisely so closing one has to come there and say so.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:36:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-s17o","title":"reduce_full and apply_command's view-slice arm have no production caller","description":"Found while fixing improvise-ondj (2026-09-20), and it is the reason that defect could sit unnoticed.\n\nui/reduce.rs module doc says: \"the one place a command is applied to the authoritative state\" and \"Every deployment mode -- the native TUI, a websocket server, a browser worker -- reaches the model through the functions here.\"\n\nTHAT IS ASPIRATIONAL. reduce_full has NO PRODUCTION CALLER ANYWHERE IN THE WORKSPACE. Every host reaches the model through Sessions::apply or reduce_key. Nothing exercised reduce_full Command::Model arm, so nothing could notice it skipped the basis check -- a door advertised by documentation and walked by nobody is exactly the shape that hides a defect until the next host arrives.\n\nSECOND HALF, same shape: apply_command other =\u003e reduce_view(\u0026mut app.view_state, other) folds a wire VIEW command into the AUTHORITATIVE App own slice. Nothing in production does that -- Sessions::apply folds view commands into the CLIENT MIRROR instead. So view_slice_commands_land_in_the_authoritative_view_state is testing a path no host walks, and passing tells you nothing about any deployment.\n\nTHE DECISION: are these a host API worth keeping, or test-only scaffolding wearing a module doc? If the former, HAVE A HOST USE ONE -- improvise-9x6 (the browser worker) is the natural candidate and is being built now, so this is decidable immediately rather than in principle. If the latter, delete them and correct the module doc, which is currently making a promise the code does not keep.\n\nDo not just delete the test: decide what the function is for first.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:54:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-r4lo","title":"Two answers to what a typed string means, and they are adjacent","description":"Noticed during improvise-xks6 (2026-09-20) and left alone, because the envelopes were that ticket. Searched first: nothing open covers it.\n\nTWO CONTRADICTORY DOC COMMENTS, now one level below the envelopes they used to hide behind:\n\n * Client::set_cell_at says classifying a typed string \"is the HOST decision, not one to re-implement here\", and takes a SERIALIZED CellValue.\n * socket.rs::typed says the OPPOSITE -- \"Putting it in Rust rather than in the page JavaScript is what makes it testable\" -- and typed_cell takes a PLAIN STRING.\n\nBoth now return the same ClientFrame, so this is the same disease improvise-xks6 just cured, one level down: TWO ANSWERS TO ONE QUESTION, kept apart only by the fact that nothing called both.\n\nWHICH ONE IS RIGHT: typed_cell is the one web/app.js calls and the one repo-map gotcha 42 blesses -- the browser has no App to ask, and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; putting it in Rust rather than JS is what makes it testable and keeps one spelling per language boundary. set_cell_at is UNREFERENCED by the page.\n\nSo the likely answer is to delete set_cell_at and fix the doc, but check first whether anything else (a test, the lockstep harness, a future worker bootstrap) wants the pre-classified form -- a host that HAS an App genuinely can classify, and improvise-9x6 puts an App in the browser, which may make set_cell_at the right entry point for the WORKER deployment and typed_cell right for the socket one. If so the fix is to say that in both doc comments rather than to delete either. Decide it, do not leave two claims standing.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:52:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:52:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-r4lo","title":"Two answers to what a typed string means, and they are adjacent","description":"Noticed during improvise-xks6 (2026-09-20) and left alone, because the envelopes were that ticket. Searched first: nothing open covers it.\n\nTWO CONTRADICTORY DOC COMMENTS, now one level below the envelopes they used to hide behind:\n\n * Client::set_cell_at says classifying a typed string \"is the HOST decision, not one to re-implement here\", and takes a SERIALIZED CellValue.\n * socket.rs::typed says the OPPOSITE -- \"Putting it in Rust rather than in the page JavaScript is what makes it testable\" -- and typed_cell takes a PLAIN STRING.\n\nBoth now return the same ClientFrame, so this is the same disease improvise-xks6 just cured, one level down: TWO ANSWERS TO ONE QUESTION, kept apart only by the fact that nothing called both.\n\nWHICH ONE IS RIGHT: typed_cell is the one web/app.js calls and the one repo-map gotcha 42 blesses -- the browser has no App to ask, and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; putting it in Rust rather than JS is what makes it testable and keeps one spelling per language boundary. set_cell_at is UNREFERENCED by the page.\n\nSo the likely answer is to delete set_cell_at and fix the doc, but check first whether anything else (a test, the lockstep harness, a future worker bootstrap) wants the pre-classified form -- a host that HAS an App genuinely can classify, and improvise-9x6 puts an App in the browser, which may make set_cell_at the right entry point for the WORKER deployment and typed_cell right for the socket one. If so the fix is to say that in both doc comments rather than to delete either. Decide it, do not leave two claims standing.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:52:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i5tk","title":"Narrow the TUI's GridCache to its viewport -- benchmark first","description":"Disclosed as a caveat of PR #12 rather than discovered by it: it is in the PR body, in the aa7a6d3 commit message, and in repo-map gotcha 44. Copilot raised it independently on round 2.\n\nproject::grid_of evaluates EVERY CELL PER FRAME, where GridWidget used to evaluate only the visible ones.\n\nBE PRECISE ABOUT WHAT CHANGED: this is a WIDENING, not an introduction. compute_col_widths was already whole-grid and the totals loop was already O(rows x visible cols). And it is UNBENCHMARKED IN BOTH DIRECTIONS -- \"this can dominate TUI rendering\" is exactly as unevidenced as \"it is fine\" would be. Do not fix it on the strength of the reasoning.\n\nSO ACCEPTANCE CRITERION #1 IS A BENCHMARK. The fix and the evidence that it was needed are the same work.\n\nWHAT THE FIX COSTS, if the benchmark says do it: teaching GridWidget that a cache position is RELATIVE, which is the browser client arrangement (gotcha 44 -- the TUI builds at offsets zero deliberately, and that is what let the scrolling arithmetic stay untouched through improvise-n10). So this is not a narrowing of a loop, it is a change to what a cache index MEANS in the TUI, and every scroll site is in scope.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:47Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:26:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-6c6q","title":"[bug] apply_switch relays a refused switch to everyone but the asker","description":"Found by the code review of ADR 001 layer 1 (2026-09-19). Pre-existing -- the pre-wave broadcast: Option\u003cCommand\u003e was explicitly \"every OTHER client\" -- and the doc overclaims.\n\napply_switch doc says: \"a client that optimistically folded a name the document turned down is corrected by the same message that tells the others to follow.\" But it relays Broadcast::ToOthers, so THE ASKER IS NOT SENT THE CORRECTION. If the workbook refuses the switch (the view does not exist), the asker slice keeps the asked-for name while its mirror holds the accepted one -- gotcha 24 mismatch, and gotcha 29 says the two copies must move together.\n\nUNLIKE the apply_model case the review fixed, a blanket ToAll is WRONG here: the asker really did fold optimistically on the accepted path, so ToAll would double-apply on the common path.\n\nTHE HONEST FIX IS CONDITIONAL: ToAll when accepted != asked, ToOthers otherwise. Which is to say the audience is a function of whether the asker \"folded the right thing\", not of the command variant -- a slightly sharper version of doctrine(the_audience_follows_the_optimistic_fold).\n\nCheap. The reason it is P3 and not higher is that switching to a view that does not exist is not a thing a well-behaved client does; it is reachable through the : command line.","notes":"ALSO FIX THE DOC COMMENT, and treat it as the more dangerous half. apply_switch's doc (project.rs:~714, verbatim on main) actively claims 'a client that optimistically folded a name the document turned down is corrected by the same message that tells the others to follow.' It is not -- the relay is ToOthers. A comment that says the hole is closed is worse than the hole, because the next reader stops looking.\n\nCopilot surfaced this independently on PR #12 and its suggested shape is right and cheap: apply_switch ALREADY RECEIVES the asked-for name -- it takes the whole cmd, which IS Command::SwitchView(asked) -- it simply never destructures it. So the conditional audience (ToAll when accepted != asked, ToOthers otherwise) needs no new parameter.\n\nTHE REAL COST OF THIS TICKET IS THE REGRESSION TEST, NOT THE CHANGE. It must drive THE ASKER'S OWN SLICE, not the to_others helper -- that helper looks away from the asker by construction, and that is exactly how the sibling bug in apply_model (fixed in 9cb16ce) shipped green.","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:15Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:26:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-z754","title":"Document name, dirty flag and file path have no home a client can read","description":"draw_title reads workbook.model.name, model_state.dirty and model_state.file_path. NONE IS PROJECTED. improvise-wasm-client RenderModel carries mode, status and view_name but not these, so A BROWSER TITLE BAR CANNOT SHOW THE DOCUMENT NAME OR THE DIRTY MARKER.\n\nFound closing improvise-n10, alongside the grid-frame gaps.\n\nWORTH DECIDING RATHER THAN JUST ADDING: dirty is a fact about the document that every client shares, and name/path likewise -- so these want a projection or a ViewState field, not a per-widget cache. But note file_path is also the thing Save uses and a browser has no filesystem path in the same sense (gotcha 2, gotcha 28), so \"what a remote client should be told about where the document lives\" is a real question and not obviously \"the path\".","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-1125","title":"Row-header width is computed per frame in the widget","description":"src/ui/grid.rs::render_grid computes sub_widths / row_header_width by walking layout.row_items per level. GridCache carries col_widths but NO row-header width.\n\nIts sibling compute_col_widths already moved into improvise-command::ui::viewport; this one did not. So a browser client sizing its row header will invent a second rule -- the exact shape of improvise-edp (the category panel walked twice) and of the multi-level-labels divergence.\n\nSmall, and it pairs naturally with the multi-level labels ticket, because both are facts about the grid FRAME rather than its contents, and the frame is what improvise-n10 deliberately left behind.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:34Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-1125","title":"Row-header width is computed per frame in the widget","description":"src/ui/grid.rs::render_grid computes sub_widths / row_header_width by walking layout.row_items per level. GridCache carries col_widths but NO row-header width.\n\nIts sibling compute_col_widths already moved into improvise-command::ui::viewport; this one did not. So a browser client sizing its row header will invent a second rule -- the exact shape of improvise-edp (the category panel walked twice) and of the multi-level-labels divergence.\n\nSmall, and it pairs naturally with the multi-level labels ticket, because both are facts about the grid FRAME rather than its contents, and the frame is what improvise-n10 deliberately left behind.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:34Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:27Z","started_at":"2026-09-20T05:32:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-x7sw","title":"Which-key hints have no protocol home (the open half of improvise-jb3)","description":"improvise-jb3 closed with its WIDGET half already done -- WhichKeyWidget::new(\u0026[(String, \u0026static str)]) was never \u0026App-coupled. This is the half that was never that ticket.\n\nThe derivation happens in draw.rs: App::armed_prefix_keymap() -\u003e Keymap::binding_hints(). ViewState::transient_prefix IS portable (gotcha 13 stores a prefix as an ADDRESS, not a keymap, precisely so ViewState stays free of improvise-command types) -- but THE KEYMAP SET IS NOT ON THE CLIENT, so a browser which-key cannot be derived from ViewState alone.\n\nTWO ANSWERS, and the choice is the ticket:\n * THE RESOLVED HINT LIST CROSSES -- a Projection, or a ViewState field. Cheap, and it keeps the keymap a client-only concept, which is the stated direction (\"keys should be converted into commands in the browser: the keymap should not be known server side at all\"). But it puts a UI string list on the wire, which is what improvise-k099 is already unhappy about for AppMode::hint_text/label.\n * THE CLIENT GETS THE KEYMAPS. Consistent with keys being a client concept, and the wasm client already has keys.rs -- but the hint list is currently derived from the SERVER-side KeymapSet, so this means the client owns the keymaps outright and the server stops having them. Larger, and probably where this is heading anyway.\n\nNote the second reading is the one that fits improvise-bpq2. Decide them together.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:33Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-2yc2","title":"DataStore::live_records walks a HashMap -- decide its order before writing the caller","description":"The next instance of improvise-shyt, found while fixing it, and deliberately LEFT ALONE rather than guessed at.\n\nDataStore::live_records walks the records: HashMap, so it yields a different order in every process -- the same defect that made records-mode rows reorder between sessions, misdirected drill staged edits, and drifted SUM/AVG in the last f64 bits.\n\nIt differs from shyt in one way that matters: IT HAS NO CALLERS YET. So this is not a bug report, it is \"decide the order deliberately before something depends on it\". The ADR 002 surrogate path will be the caller -- the RecordHash -\u003e RecordEntry index is what live_records walks -- and whoever writes it should pick an order on purpose rather than inherit a hash walk and then discover it, which is how shyt happened.\n\nTHE ANSWER SHYT LANDED ON, for reference: the index narrows, the primary IndexMap orders (matching_positions resolves each candidate with get_index_of, sorts, reads back with get_index). Measured at 18,000 cells that cost at worst +20% on one path and was 7% FASTER on the other. Whether that is the right answer here depends on what the surrogate path wants; a hash-keyed index may legitimately want no order at all, in which case say so and return something whose type admits it.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:45:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-4zl7","title":"Teach the emitter a categorical split (remedy 3 for improvise-8xhy)","description":"Successor to improvise-8xhy, which shipped remedy 1 (refuse at emit) as an explicit interim. This is the remedy that produces a formula A HUMAN WOULD HAVE WRITTEN, and it is the right end state -- but it was costed during 8xhy and it is much larger than it looks.\n\nANSWER THIS FIRST, BEFORE TOUCHING THE GRAMMAR. A fitted measure cell is evaluated at a CellKey context. In any view that aggregates over the split category -- Region on Page or None -- context.get(\"Region\") is None, and today Formula::filter falls straight through to aggregate_raw. So a categorical fit would SILENTLY DEGRADE to something else in exactly the views a pivot-table user spends most of their time in. That is a product decision about what a fit MEANS for an aggregated cell, not an emitter detail.\n\nSCOPE, once that is answered:\n * Expr::Is(Filter) evaluating to 1.0/0.0. This keeps the f64-only evaluator and avoids introducing string comparison into eval_expr. The primitive already exists in the right shape -- types.rs reads context.get(\u0026filter.category), the current cell own coordinate.\n * A grammar production plus tokenizer handling in the 1109-line parser.rs, with quoting rules for the category and item inside it (gotcha 34 -- one home, improvise_formula::quote).\n * Arms in BOTH evaluators: eval_expr and eval_expr_cached.\n * Coverage in the grammar-walking generator.\n * Thread Dataset::columns (the ColumnSpec/ColumnKind plan) into the emitter, so feature index -\u003e role is read off preprocess own plan. Do NOT reconstruct it: Spec::names is \u0026[\u0026str], and reconstructing index-\u003erole is a second implementation of preprocess expansion order, exactly the drift risk gotcha 41 names in another key.\n * .improv is free -- the writer emits Formula::raw.\n\nIT CLOSES ONLY A THIRD OF 8xhy EITHER WAY. A WHERE predicate covers the one-hots. col.missing means \"this record has no value for measure X\", a different vocabulary. The date sub-features (when.year, when.month) need date arithmetic improvise does not have at all. Those stay refused, or get their own tickets.\n\nPAIRS WITH improvise-vx3 (the fit-measure command), which is where a user would actually learn about the restriction.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:45:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-odbc","title":"[bug] bank-info.improv in the repo root does not parse","description":"Incidental finding from improvise-cr3 (2026-09-19), outside that ticket scope, unreproduced beyond the one load.\n\n Parse error at line 293, column 3: - JORDAN MARTINEZ (expected pipe_quoted)\n\nTWO READINGS, and they want different actions:\n * The file is STALE -- written before a grammar change -- in which case delete it or regenerate it, and the finding is that a sample file in the repo root is not exercised by anything.\n * The GRAMMAR HAS A GAP -- some name shape a writer emits that the reader rejects. That is the serious reading, because save/load round-trip is a property test (persistence proptests, parse(format(parse(generate())))) and a gap that survives it is a gap the GENERATOR does not reach.\n\nDecide which by checking whether improvise can WRITE that line today. If it can, the round-trip property is incomplete and the generator is the thing to fix, not the file.","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:37:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-2yc2","title":"DataStore::live_records walks a HashMap -- decide its order before writing the caller","description":"The next instance of improvise-shyt, found while fixing it, and deliberately LEFT ALONE rather than guessed at.\n\nDataStore::live_records walks the records: HashMap, so it yields a different order in every process -- the same defect that made records-mode rows reorder between sessions, misdirected drill staged edits, and drifted SUM/AVG in the last f64 bits.\n\nIt differs from shyt in one way that matters: IT HAS NO CALLERS YET. So this is not a bug report, it is \"decide the order deliberately before something depends on it\". The ADR 002 surrogate path will be the caller -- the RecordHash -\u003e RecordEntry index is what live_records walks -- and whoever writes it should pick an order on purpose rather than inherit a hash walk and then discover it, which is how shyt happened.\n\nTHE ANSWER SHYT LANDED ON, for reference: the index narrows, the primary IndexMap orders (matching_positions resolves each candidate with get_index_of, sorts, reads back with get_index). Measured at 18,000 cells that cost at worst +20% on one path and was 7% FASTER on the other. Whether that is the right answer here depends on what the surrogate path wants; a hash-keyed index may legitimately want no order at all, in which case say so and return something whose type admits it.","status":"closed","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:43Z","started_at":"2026-09-20T05:32:37Z","closed_at":"2026-09-20T05:54:43Z","close_reason":"Closed by PR #20 (WU-K). DECIDED: live_records answers in the STORE'S OWN ORDER -- it walks the primary IndexMap's cells and dedupes by RecordHash, so a record sits where its FIRST cell sits. That is the order matching_positions already gives (gotcha 18), projected onto records, rather than a second answer to \"what order is this store in\".\n\nDerived on each call rather than memoised on RecordEntry, because a stored position is a copy and the tombstone a deleted record leaves would keep it while its cells moved to the end. That case is now tested (a_recreated_record_takes_its_new_place_not_its_old_one) -- and it was found because Copilot noticed the doc comment ASSERTED the derive-don't-memoise choice mattered for delete-and-recreate while nothing tested it; a memoising implementation passed all three other tests.\n\nTHE TICKET'S ALTERNATIVE WAS REJECTED WITH A REASON WORTH KEEPING: \"a hash-keyed index may want no order at all -- return something whose type admits it\" cannot be built. Every iterator has an order, and a type that merely DISCLAIMS one still hands the caller hash order. Disclaiming is not making unrepresentable.\n\nCost moves O(records) -\u003e O(cells). With no caller to benchmark, correctness is the defensible default and the doc says what a hot-path caller should bring.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-4zl7","title":"Teach the emitter a categorical split (remedy 3 for improvise-8xhy)","description":"Successor to improvise-8xhy, which shipped remedy 1 (refuse at emit) as an explicit interim. This is the remedy that produces a formula A HUMAN WOULD HAVE WRITTEN, and it is the right end state -- but it was costed during 8xhy and it is much larger than it looks.\n\nANSWER THIS FIRST, BEFORE TOUCHING THE GRAMMAR. A fitted measure cell is evaluated at a CellKey context. In any view that aggregates over the split category -- Region on Page or None -- context.get(\"Region\") is None, and today Formula::filter falls straight through to aggregate_raw. So a categorical fit would SILENTLY DEGRADE to something else in exactly the views a pivot-table user spends most of their time in. That is a product decision about what a fit MEANS for an aggregated cell, not an emitter detail.\n\nSCOPE, once that is answered:\n * Expr::Is(Filter) evaluating to 1.0/0.0. This keeps the f64-only evaluator and avoids introducing string comparison into eval_expr. The primitive already exists in the right shape -- types.rs reads context.get(\u0026filter.category), the current cell own coordinate.\n * A grammar production plus tokenizer handling in the 1109-line parser.rs, with quoting rules for the category and item inside it (gotcha 34 -- one home, improvise_formula::quote).\n * Arms in BOTH evaluators: eval_expr and eval_expr_cached.\n * Coverage in the grammar-walking generator.\n * Thread Dataset::columns (the ColumnSpec/ColumnKind plan) into the emitter, so feature index -\u003e role is read off preprocess own plan. Do NOT reconstruct it: Spec::names is \u0026[\u0026str], and reconstructing index-\u003erole is a second implementation of preprocess expansion order, exactly the drift risk gotcha 41 names in another key.\n * .improv is free -- the writer emits Formula::raw.\n\nIT CLOSES ONLY A THIRD OF 8xhy EITHER WAY. A WHERE predicate covers the one-hots. col.missing means \"this record has no value for measure X\", a different vocabulary. The date sub-features (when.year, when.month) need date arithmetic improvise does not have at all. Those stay refused, or get their own tickets.\n\nPAIRS WITH improvise-vx3 (the fit-measure command), which is where a user would actually learn about the restriction.","status":"in_progress","priority":3,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:41Z","started_at":"2026-09-20T05:32:41Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-odbc","title":"[bug] bank-info.improv in the repo root does not parse","description":"Incidental finding from improvise-cr3 (2026-09-19), outside that ticket scope, unreproduced beyond the one load.\n\n Parse error at line 293, column 3: - JORDAN MARTINEZ (expected pipe_quoted)\n\nTWO READINGS, and they want different actions:\n * The file is STALE -- written before a grammar change -- in which case delete it or regenerate it, and the finding is that a sample file in the repo root is not exercised by anything.\n * The GRAMMAR HAS A GAP -- some name shape a writer emits that the reader rejects. That is the serious reading, because save/load round-trip is a property test (persistence proptests, parse(format(parse(generate())))) and a gap that survives it is a gap the GENERATOR does not reach.\n\nDecide which by checking whether improvise can WRITE that line today. If it can, the round-trip property is incomplete and the generator is the thing to fix, not the file.","status":"in_progress","priority":3,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:40Z","started_at":"2026-09-20T05:32:40Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-gd28","title":"Drill over the wire needs the addressed command form","description":"From the ADR 002 spine landing (2026-09-19). The RESOLVER exists and is tested -- App::resolve_aggregate, layout-only by necessity since an aggregate has no existence outside the layout that derives it -- and CellId::Aggregate is emitted. What is missing is the command.\n\nModelCommand::Drill was deliberately NOT added. Drilling creates the _Drill view and writes drill_state, which gotchas 30 and 38 rule session-local and therefore unaddressable by a NULLARY Command -- exactly the reason ApplyAndClearDrill was removed from the wire in improvise-3x3u.\n\nSo a wire Drill needs the ADDRESSED form: the variant carries a SessionId and the staging map moves off the single App onto the session. That is ADR 001 layer 2 (context/adr/001-concurrent-edits.md), so this is blocked on it rather than being independent work.\n\nThe seam is ready: emit, resolve and the type-level guarantee (Drill takes an Aggregate and nothing else) are all in place and tested.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:18:34Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-qddo","title":"Make DataStore an implementation of a source trait, not the implementation","description":"The one piece of ADR 003 worth doing regardless of when or whether live-backing happens. See context/adr/003-live-backed-models.md.\n\nTODAY Model holds a DataStore directly, and a Workbook loads ENTIRE -- there is no partial-load path anywhere in the tree (verified while declining improvise-9ix). So \"where cell values come from\" is not a seam; it is an assumption spread across GridLayout, evaluation and persistence.\n\nTHE WORK: introduce a source trait in improvise-core and make the current in-RAM DataStore AN implementation of it rather than THE implementation. No behaviour change -- doctrine(characterization_survives_the_move) applies, and the existing tests must carry across unchanged in meaning.\n\nWHY IT IS WORTH DOING ON ITS OWN, before any decision about SQLite:\n * it is the seam every later option needs -- live-backing, import-once, or a test double;\n * it makes \"the model is fully in RAM\" an explicit property of one implementation rather than an invisible assumption of the whole crate;\n * done as a pure refactor it is cheap and reviewable, whereas doing it AS PART OF adding SQLite mixes a large refactor with a large feature and makes both unreviewable.\n\nCONSTRAINT: forbidden(improvise_core, io) -- the TRAIT belongs in improvise-core, any IO-touching implementation belongs in improvise-io. Do not let the trait signature drag IO types into core; if it needs Result, that is a boundary decision worth arguing rather than defaulting (AGENTS.md keeps Option/Result at the edges).\n\nWATCH: aggregation is the real interface. apply_agg computes Sum/Avg/Min/Max/Count over a matching cell set, and GridLayout asks for a VALUE, not for the rows behind it -- so the trait should be shaped around \"give me the aggregate for this coordinate set\", not around \"give me the cells\". Shaping it around cells would foreclose SQL pushdown, which is the whole point of the eventual live source.","notes":"\n--- READ-ONLY DECIDED (user, 2026-09-19), and it shapes the trait ---\nA live-backed model is READ-ONLY, and not reluctantly: for many use cases\nediting capabilities would be a DOWNSIDE. Pointing a pivot tool at a production\ndatabase and being unable to write to it is a feature.\n\nSO DO NOT PUT WRITES IN THE TRAIT. If writing were a trait method, a read-only\nsource would have to fail at RUNTIME for something it structurally cannot\nserve. Keep reads -- and crucially AGGREGATES, see the note above about shaping\nit around \"give me the aggregate for this coordinate set\" -- in the trait every\nsource implements, and let writing be a capability only the owned in-RAM\nimplementation has. Then read-only is UNREPRESENTABLE rather than enforced,\nwhich is the same discipline ADR 002 CellId uses and the same design-principles\n3 rule.\n\nRELATED CONVERGENCE WORTH KNOWING: ADR 002 already makes SetCell take a\nCellId::Record and nothing else, so a live-backed model that projects no Record\nids makes a client-side write unconstructible with no runtime check and no\n\"this model is read-only\" dialog. The settability distinction built for\naggregates carries read-only sources for free -- so the client half of this may\nalready be done.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:16:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:57Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-rucf","title":"Read arbitrary SQLite databases as a data source","description":"USER REQUIREMENT (2026-09-19): improvise must be able to handle ARBITRARY SQLite databases as models -- someone points it at a database they already have and pivots over it.\n\nTHIS IS NOT improvise-9ix AND ITS DECLINE DOES NOT BEAR ON IT. 9ix proposed SQLite as improvise OWN PERSISTENCE FORMAT -- writing our model semantics into a second container -- and was declined because that is a second implementation of the pest grammar with a silent-drift failure mode, and because it measured 22% larger than .improv and 4.7x larger gzipped. Reading SOMEONE ELSE database has none of those properties: no round-trip obligation, no grammar duplication, and the value is real (pivot over data you already have). The one thing that DOES carry over is the dependency constraint, below.\n\nTHE SEAM IS CLEAN AND ALREADY EXISTS. import/csv_parser.rs produces Vec\u003cserde_json::Value\u003e (rows as objects) and import/analyzer.rs::analyze_records(\u0026[Value]) -\u003e Vec\u003cFieldProposal\u003e does the Category/Measure/Time/Skip classification from there. A SQLite source slots in beside csv_parser.rs and needs only to produce Vec\u003cValue\u003e. ImportPipeline and the wizard then work unchanged.\n\nA GENUINE IMPROVEMENT OVER CSV, worth exploiting rather than ignoring: the analyzer currently infers field kinds from STRING SAMPLES -- detect_date_format guesses a date by pattern-matching text. A SQLite column has a DECLARED TYPE. So the classifier should take typed input where it has it and fall back to sampling only where it does not (SQLite being dynamically typed per-value, both cases are real). That is a better analyzer, not just a new source.\n\nDESIGN QUESTIONS TO SETTLE, none of them hard but all of them real:\n * WHICH ROWS? A table name, or a user-supplied SELECT? A query is strictly more useful (joins, filters, computed columns) and no harder to consume -- a result set is a result set. Prefer it, and let a bare table name be sugar.\n * MULTIPLE TABLES: answered by the above if the input is a query.\n * TYPE MAPPING: INTEGER/REAL -\u003e measure candidates, TEXT -\u003e category candidates, NULL -\u003e the missing path (see improvise-ych: null is ABSENCE, not the string \"null\"), BLOB -\u003e skip. Dates in SQLite may be TEXT, INTEGER (epoch) or REAL (julian) -- detect, do not assume.\n * READ-ONLY for now. Writing edits back to the source database is a separate and much larger question; say so in the code rather than leaving it implied.\n\nTHE CONSTRAINT THAT CARRIES OVER FROM 9ix: improvise-io IS IN THE WASM BUNDLE (.github/workflows/ci.yml gates it), and libsqlite3-sys C build fails for wasm32-unknown-unknown under this repo nix cc-wrapper. So this must be FEATURE-GATED AND NATIVE-ONLY, with the wasm job still passing untouched -- or the browser path uses a different SQLite (sqlite-wasm-rs / OPFS), which is a second decision and should be its own ticket. Verify the gate rather than assuming it.\n\nOPEN, and it changes the size of this ticket by an order of magnitude -- see the note being added: whether this is IMPORT-ONCE (read the database, build a Model in RAM) or a LIVE-BACKED model (queries go to SQLite as the user pivots).","notes":"\n--- SUPERSEDED IN FRAMING by ADR 003 (user, 2026-09-19) ---\nThe user chose LIVE-BACKED over import-once, and the goal is stated as: have\nTHE OPTION of essentially using improvise as a DB CLIENT. NOT A PRIORITY --\nnothing here outranks improvise-1ey. Re-prioritised P2 -\u003e P3.\n\nSo this ticket read-only import framing is not the target shape. See\ncontext/adr/003-live-backed-models.md. The import-once seam it described\n(csv_parser -\u003e Vec\u003cValue\u003e -\u003e analyze_records) is still accurate and still\nuseful, but as ONE implementation of a source rather than as the answer.\n\nWHAT SURVIVES UNCHANGED and is worth keeping from this ticket: the type-mapping\nnotes (SQLite columns carry declared types, which is BETTER input than CSV\nstring sampling -- detect_date_format currently guesses a date by pattern-\nmatching text; dates may be TEXT, INTEGER epoch or REAL julian, so detect\nrather than assume; NULL is ABSENCE not the string \"null\", per improvise-ych),\nand the wasm constraint (improvise-io is in the bundle, libsqlite3-sys does not\nbuild for wasm32 under this repo nix cc-wrapper, so any SQLite dependency is\nfeature-gated and native-only).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:14:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:16:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-qddo","title":"Make DataStore an implementation of a source trait, not the implementation","description":"REWRITTEN 2026-09-19 to match context/adr/003-live-backed-models.md, and DEFERRED by user ruling in the same session, together with improvise-rucf and improvise-mjy7. The ADR calls this \"worth doing first regardless of the eventual shape\"; the user's ruling is that the whole direction waits, so it waits as a unit rather than landing a seam with nothing on the other side of it.\n\nTHE WORK IS UNCHANGED: introduce a source trait in improvise-core and make the current in-RAM DataStore AN implementation of it rather than THE implementation. No behaviour change -- doctrine(characterization_survives_the_move) applies and the existing tests must carry across unchanged in meaning.\n\nTODAY Model holds a DataStore directly and a Workbook loads ENTIRE; there is no partial-load path anywhere in the tree (verified while declining improvise-9ix). So \"where cell values come from\" is not a seam, it is an assumption spread across GridLayout, evaluation and persistence.\n\nWHAT ADR 003 ADDS TO THE ORIGINAL FRAMING, and it is the part worth re-reading before starting:\n\n * AGGREGATION IS THE INTERFACE, AND IT IS ALREADY SQL. apply_agg computes Sum/Avg/Min/Max/Count over a matching cell set -- exactly SQL's aggregates -- and GridLayout asks for a VALUE, not for the rows behind it. Shape the trait around \"give me the aggregate for this coordinate set\", never around \"give me the cells\": shaping it around cells forecloses pushdown, which is the entire point of the eventual live source.\n\n * WRITES MUST NOT BE IN THE TRAIT. This is new since the original text and it is a real constraint. If writing were in the trait a query source implements, a read-only source would have to FAIL AT RUNTIME for a method it structurally cannot serve. Keep reads and aggregates in the trait every source implements; let writing be a capability only the owned in-RAM implementation has. Read-only then becomes UNREPRESENTABLE rather than enforced -- the same discipline as CellId (ADR 002) and doctrine(an_invariant_is_made_unrepresentable_not_validated).\n\n * FORMULAS DO NOT MOVE. Evaluation is fixed-point and derived values live in Model::formula_cache, not in DataStore, so the split is clean: raw data is queried, derived data stays in RAM.\n\nCONSTRAINT: forbidden(improvise_core, io). The TRAIT belongs in improvise-core; any IO-touching implementation belongs in improvise-io. Do not let the trait signature drag IO types into core; if it needs Result, that is a boundary decision worth arguing rather than defaulting (AGENTS.md keeps Option/Result at the edges).\n\nWHY IT IS STILL WORTH DOING ON ITS OWN WHEN IT IS UN-DEFERRED: it is the seam every later option needs -- live-backing, a test double, or the import-once path ADR 003 rejected as the ONLY answer but did not forbid as A path; it makes \"the model is fully in RAM\" an explicit property of one implementation rather than an invisible assumption of the whole crate; and done as a pure refactor it is cheap and reviewable, where doing it AS PART OF adding a SQLite source mixes a large refactor with a large feature and makes both unreviewable.\n","notes":"\n--- READ-ONLY DECIDED (user, 2026-09-19), and it shapes the trait ---\nA live-backed model is READ-ONLY, and not reluctantly: for many use cases\nediting capabilities would be a DOWNSIDE. Pointing a pivot tool at a production\ndatabase and being unable to write to it is a feature.\n\nSO DO NOT PUT WRITES IN THE TRAIT. If writing were a trait method, a read-only\nsource would have to fail at RUNTIME for something it structurally cannot\nserve. Keep reads -- and crucially AGGREGATES, see the note above about shaping\nit around \"give me the aggregate for this coordinate set\" -- in the trait every\nsource implements, and let writing be a capability only the owned in-RAM\nimplementation has. Then read-only is UNREPRESENTABLE rather than enforced,\nwhich is the same discipline ADR 002 CellId uses and the same design-principles\n3 rule.\n\nRELATED CONVERGENCE WORTH KNOWING: ADR 002 already makes SetCell take a\nCellId::Record and nothing else, so a live-backed model that projects no Record\nids makes a client-side write unconstructible with no runtime check and no\n\"this model is read-only\" dialog. The settability distinction built for\naggregates carries read-only sources for free -- so the client half of this may\nalready be done.","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:16:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rucf","title":"Read arbitrary SQLite databases as a data source","description":"SUPERSEDED BY context/adr/003-live-backed-models.md, and rewritten 2026-09-19 to match it. The original text framed this as an open choice between import-once and live-backed, and said \"READ-ONLY for now\" as though that were a scoping convenience. ADR 003 decided both, and in the opposite direction from what this ticket assumed.\n\nWHAT THE ADR DECIDED, and this ticket now means:\n\n * LIVE-BACKED, not import-once. Import-once was explicitly REJECTED as the only answer -- it makes the model a snapshot, so source changes are invisible until re-import and the data must fit in memory. The model is a view over the source and queries go to it as the user pivots.\n * READ-ONLY IS A FEATURE, not a limitation accepted reluctantly. Pointing a pivot tool at a production database and being unable to write to it is the point. No write-through, no overlay, no reconciliation.\n * THEREFORE: Model becomes a FACADE OVER A DATA SOURCE, of which today's in-RAM DataStore is one implementation and a SQL query engine is another. That is a change to what Model IS, not to where its bytes come from.\n\nSTATUS: NOT A PRIORITY, per the ADR's own header. Deferred 2026-09-19 during the whole-backlog campaign, by user ruling, together with improvise-qddo and improvise-mjy7. Nothing here outranks the browser work. It is written down so the option stays open, because several decisions made recently could foreclose it by accident.\n\nTHE THREE THINGS THAT MAKE IT TRACTABLE, from the ADR, which should shape the design rather than be discovered late:\n 1. Aggregation is ALREADY the interface and it is ALREADY SQL. apply_agg computes Sum/Avg/Min/Max/Count over a matching cell set; those are exactly SQL's aggregates. GridLayout asks for a VALUE, not for the rows behind it. The pushdown is not a translation layer to be invented.\n 2. Formula results are ALREADY separate from stored data. Evaluation is fixed-point and derived values live in Model::formula_cache, not in DataStore. Raw data is queried; derived data stays in RAM. Formulas do not become SQL.\n 3. Clients are ALREADY insulated. A client sees CellDisplays and never learns where a value came from, so live-backing and the browser deployment compose rather than compete. A browser never needs SQLite at all.\n\nTHE CONSEQUENCE NOBODY ASKED FOR, and the reason this is three tickets rather than one: a live-backed document splits into OUR configuration and THEIR data, and .improv does not distinguish them. The format grows a SOURCE SPECIFICATION where a live model would otherwise have a ## Data section, and categories become derived (SELECT DISTINCT) rather than declared. improv.pest is the single source of truth, so the grammar comes first.\n\nCHILDREN (all filed and deferred alongside this one):\n * the .improv source specification and the grammar change\n * the SQLite source implementation, native-only and feature-gated\n * projection freshness for a source an external writer can change\n\nWHAT ADR 001 DOES AND DOES NOT OWE THIS: a read-only model has no concurrent EDITS, so the basis check simply does not apply -- the guarantee is scoped to documents improvise owns, which is what it always meant. What remains is a different and smaller question, which is the freshness child.\n\nTHE MACHINERY TO EXPRESS READ-ONLY ALREADY EXISTS, built for something else: ADR 002 makes SetCell take a CellId::Record and nothing else, so a source that projects no Record ids makes a write UNCONSTRUCTIBLE by the client -- no runtime check, no error path, no read-only dialog. One design consequence for the source trait follows: writes must NOT be in the trait a query source implements, or a read-only source has to fail at runtime for a method it structurally cannot serve.\n\nTHE DEPENDENCY CONSTRAINT THAT CARRIES OVER FROM improvise-9ix: improvise-io is in the wasm bundle (the wasm job in .github/workflows/ci.yml gates it) and libsqlite3-sys does not build for wasm32-unknown-unknown under this repo's nix cc-wrapper. Native-only and feature-gated, with the wasm job passing untouched. Verify the gate rather than assuming it.\n\nWHAT THIS IS NOT: improvise-9ix, which proposed SQLite as improvise's OWN PERSISTENCE FORMAT and was declined on measurements (22% larger than .improv, 4.7x gzipped, a second implementation of the pest grammar). Reading someone else's database has no round-trip obligation and no grammar duplication.\n","notes":"\n--- SUPERSEDED IN FRAMING by ADR 003 (user, 2026-09-19) ---\nThe user chose LIVE-BACKED over import-once, and the goal is stated as: have\nTHE OPTION of essentially using improvise as a DB CLIENT. NOT A PRIORITY --\nnothing here outranks improvise-1ey. Re-prioritised P2 -\u003e P3.\n\nSo this ticket read-only import framing is not the target shape. See\ncontext/adr/003-live-backed-models.md. The import-once seam it described\n(csv_parser -\u003e Vec\u003cValue\u003e -\u003e analyze_records) is still accurate and still\nuseful, but as ONE implementation of a source rather than as the answer.\n\nWHAT SURVIVES UNCHANGED and is worth keeping from this ticket: the type-mapping\nnotes (SQLite columns carry declared types, which is BETTER input than CSV\nstring sampling -- detect_date_format currently guesses a date by pattern-\nmatching text; dates may be TEXT, INTEGER epoch or REAL julian, so detect\nrather than assume; NULL is ABSENCE not the string \"null\", per improvise-ych),\nand the wasm constraint (improvise-io is in the bundle, libsqlite3-sys does not\nbuild for wasm32 under this repo nix cc-wrapper, so any SQLite dependency is\nfeature-gated and native-only).","status":"deferred","priority":3,"issue_type":"epic","owner":"el-github@elangley.org","created_at":"2026-09-20T00:14:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:18Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i9lt","title":"View::axis_of panics for a category the view has not seen","description":"Noticed during improvise-35e (2026-09-19), reported not fixed per the mention-do-not-fix rule for pre-existing issues.\n\nView::axis_of(cat) panics for a category the view has never seen, and View::tileable_cats admits exactly such categories -- it filters on try_axis_of(c).is_none_or(...). So the pairing is a live panic waiting for the right view state.\n\nThe old TileBar called axis_of over that list. The migrated widget cannot panic there any more, because tile_entries uses try_axis_of(...).unwrap_or(Axis::None) -- so the reachable path is gone and no failing test was written, since the old path no longer exists to fail.\n\nWorth a guard anyway: either make axis_of total (returning Axis::None like tile_entries does) or make it private and force callers through try_axis_of. A panicking accessor over a list built to include the panicking case is a trap for the next caller, and the next caller will not have the migration to protect them.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T23:40:38Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:25:57Z","closed_at":"2026-09-20T00:25:57Z","close_reason":"The reachable path is gone. improvise-edp migrated CategoryContent to take (\u0026CategoryTreeCache, \u0026HashSet\u003cString\u003e) instead of \u0026View, and the cache is built with try_axis_of(...).unwrap_or(Axis::None) -- so the widget that called view.axis_of(name) over a list that admits unseen categories no longer exists. Same resolution as the TileBar path in improvise-35e.\n\nView::axis_of itself is unchanged and still panics for a category the view has not seen. Left deliberately: it is a pre-existing API sharpness with no remaining caller that can reach it, and the mention-do-not-fix rule applies. If it acquires a new caller, make it total (returning Axis::None like the cache does) or make it private and force callers through try_axis_of -- a panicking accessor over a list built to include the panicking case is a trap for whoever arrives next.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-j31x","title":"Server hosts several documents: a registry above Sessions","description":"DEFERRED by the user (2026-09-19) -- recorded so nothing forecloses it, not to be built now.\n\nimprovise is a multi-document app and the server should be able to host several separate models. The shape:\n server -\u003e many documents\n document -\u003e one App + one Sessions + many ClientSessions\nThat is NOT one App per session (what improvise-q08 ticket text says, written before the session model landed) and NOT one App total.\n\nNOTHING THAT HAS LANDED NEEDS TO CHANGE. Sessions does not own the App -- connect(\u0026mut self, app: \u0026App, ..), apply(\u0026mut self, app: \u0026mut App, ..) and project_all(\u0026mut self, app: \u0026App, ..) all take it as a parameter. So Sessions is already document-agnostic and a Sessions instance is implicitly bound to one document generation sequence, which is right: document_generation lives on the App. Multi-document is a REGISTRY ABOVE Sessions mapping a document id to its (App, Sessions) pair.\n\nWHY IT MATTERS BEYOND TIDINESS: sessions editing independent models are trivially non-conflicting, so ADR 001 concurrency design scopes per document FOR FREE -- the basis version tiers live on the App and each document has its own. A server that shared one App across documents would throw that away.\n\nWHAT TO AVOID IN THE MEANTIME: improvise-q08 must not hard-code a single App into a static or a lazily-initialised singleton. The document should be a PARAMETER even while there is only one, and a connection should be attachable to a document even if the MVP answers that from a CLI argument naming one .improv file.\n\nScope when picked up: the registry, document lifecycle (open/close/last-session-leaves), and how a client names the document it wants. Auth and access control are explicitly NOT in scope here -- file them separately if wanted.","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:32:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:33:15Z","closed_at":"2026-09-20T00:33:15Z","close_reason":"Registry landed in crates/improvise-ws-server/src/registry.rs. 1051 tests pass.\n\nIT CAUGHT A REAL BUG BY VERIFYING THE SEAM INSTEAD OF TRUSTING IT. My brief and the q08 notes both said nothing below the registry needs to change. Nearly true -- but Sessions::next_id is a PER-Sessions counter (confirmed: project.rs holds next_id on Sessions, not globally), so EVERY documents first client is SessionId(0), and transport.rs keyed its outs: HashMap\u003cSessionId, _\u003e on it. A registry that kept that key would have delivered document Bs frames to document As client the moment two documents each had one client.\n\nThe server-wide address is the pair, ClientId { document, session }. Registry::route(document, Vec\u003cAddressed\u003e) is the ONLY place a SessionId acquires a document -- the one it came out of, since that is the only one in scope there -- so cross-document delivery of a projection, a SwitchView broadcast (gotcha 36) or an Applied::Rebased fan-out (gotcha 39) is UNREPRESENTABLE rather than checked. The test names the hazard: an_edit_is_projected_only_to_the_edited_documents_clients asserts on_alpha.session == on_beta.session FIRST.\n\nLIFECYCLE, and the non-obvious half. Open at STARTUP from the files named on the command line -- open-on-request was rejected because deciding which paths a client may open IS access control, which this ticket put out of scope; serving only what was named leaves the question unanswered rather than answered badly. NOT closed when the last session leaves: eviction loses unsaved edits, and worse, document_generation and record_versions start at ZERO in a fresh App (gotcha 40), so a client reconnecting with a STALE basis would have it accepted as current -- a hole in the exact mechanism ADR 001 is building, bought for a memory saving nobody asked for. Registry::is_idle is the hook a future policy asks from; nothing evicts.\n\nA client names its document in the URL PATH (ws://host:port/quarterly), which leaves wire.rs COMPLETELY unchanged so the wasm-client frames keep working; / resolves to the only document when there is exactly one, and with two is ConnectError::Ambiguous rather than a coin toss. A refusal is a websocket CLOSE FRAME carrying the reason -- no ServerMsg::Error variant invented, because that is a connection fact and wire.rs only grows variants the mutations demand.\n\nEvery required claim has a counter-test beside it so it cannot pass vacuously -- e.g. the broadcast-isolation fixture gives BOTH documents a view named Second, so betas silence is the routing and not a name it would have rejected anyway.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-2wi8","title":"ADR-001 e: a forced overwrite is visible to the client it overwrote","description":"ADR 001 layer 1, step e. Depends on step c.\n\nWithout this, force is silent data loss with a confirmation step in front of it. The overwritten client edit was already committed, so this is a NOTIFICATION rather than a conflict -- but it must exist, or \"someone else won\" is indistinguishable from \"my edit never happened\".\n\nDesign note: the notification is about the loser OWN prior edit, so it is per-client and therefore UI -- the model is shared, the UI is not. Do not broadcast it as document state.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:27:34Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:27:34Z","dependencies":[{"issue_id":"improvise-2wi8","depends_on_id":"improvise-3al1","type":"blocks","created_at":"2026-09-19T23:27:47Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} @@ -183,20 +188,20 @@ {"_type":"issue","id":"improvise-ki7c","title":"EnterMode writes a runtime-named buffer without going through write_buffer","description":"From the wave-1 review (2026-09-19). Latent, not live.\n\nimprovise-gdl made cmd::core::write_buffer(name, value) the single accessor for a RUNTIME-NAMED buffer, so that the 'search' name routes to SetSearchQuery instead of a dead map entry. But cmd/mode.rs's EnterMode still builds effect::SetBuffer { name: mb.buffer_key.to_string(), .. } directly. Every other remaining direct SetBuffer uses a LITERAL name ('command', 'export', 'edit') that is statically not 'search', so EnterMode is the one site that writes a name it does not know at compile time and bypasses the router.\n\nUnreachable today because no AppMode carries buffer_key: \"search\". Concrete failure when that changes: the natural follow-up to 'SearchMode is Normal + a flag' (gotcha 11/14) is a real AppMode::Search { minibuf: ... buffer_key: \"search\" }. EnterMode would then clear buffers[\"search\"] while append-char/pop-char read and write view_state.search_query -- so entering search mode would leave the PREVIOUS query on screen. Exactly the desync gdl closed, reopened from the other side.\n\nFIX: one line -- effects.push(write_buffer(\u0026mb.buffer_key, String::new())) in EnterMode.","notes":"\n--- FIXED, AWAITING MERGE (2026-09-19) ---\nThe fix is on branch wave-1-review-fixes and is in PR #9\n(https://github.com/fiddlerwoaroof/improvise/pull/9), which is open pending\nCopilot review. Moved to in_progress rather than left open so nobody picks\nthis up again -- see the agent-issue-drift-pattern memory. Close it when #9\nmerges, not before.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T20:42:38Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:50:35Z","started_at":"2026-09-19T21:15:03Z","closed_at":"2026-09-19T21:50:35Z","close_reason":"Merged to main in PR #9 as e1acac691d70992b5dfcce9f11ce2c2b50fa1c29 (2026-09-19). Verified a real merge commit with both parents, not a squash, so all 18 branch commits are reachable on main per AGENTS.md's preference for merges. Copilot's second review on that PR: approval recommended, findings none. All eight non-skipped CI checks green, including claude-review. Synced to origin and gitea.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-odi4","title":"Nothing ties AppMode::hint_text to the actual keymap","description":"From the wave-1 review (2026-09-19). only_quit_has_no_hint asserts a hint is PRESENT; nothing asserts it is ACCURATE. Rebinding a mode's keys leaves its hint silently lying to the user -- the same class of silent drift that doctrine(binding_dump_diff) exists for.\n\nConcretely: improvise-1c9 added 'Enter:export Esc:cancel — type a file path' to ExportPrompt. The reviewer confirmed it matches the real keymap today (Enter =\u003e SEQ[commit-export, clear-buffer export], Esc =\u003e SEQ[clear-buffer export, enter-mode normal]). Nothing keeps that true.\n\nFIX: for each mode, split its hint on whitespace and for every 'Key:verb' token assert keymap.lookup(key, NONE).is_some() in that mode. Prove it fires by rebinding or removing a key named in a hint.\n\nCosmetic while there: every other hint uses two spaces around the em dash; the new ExportPrompt one uses one.","notes":"\n--- FIXED, AWAITING MERGE (2026-09-19) ---\nThe fix is on branch wave-1-review-fixes and is in PR #9\n(https://github.com/fiddlerwoaroof/improvise/pull/9), which is open pending\nCopilot review. Moved to in_progress rather than left open so nobody picks\nthis up again -- see the agent-issue-drift-pattern memory. Close it when #9\nmerges, not before.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T20:42:37Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:50:35Z","started_at":"2026-09-19T21:15:03Z","closed_at":"2026-09-19T21:50:35Z","close_reason":"Merged to main in PR #9 as e1acac691d70992b5dfcce9f11ce2c2b50fa1c29 (2026-09-19). Verified a real merge commit with both parents, not a squash, so all 18 branch commits are reachable on main per AGENTS.md's preference for merges. Copilot's second review on that PR: approval recommended, findings none. All eight non-skipped CI checks green, including claude-review. Synced to origin and gitea.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ksst","title":"A gp-tree fit records its seed but not its GA config","description":"Follow-up from improvise-lzn (2026-09-19). data::Model now records the seed, so a fit looks reproducible, but the GA configuration (population, generations, elite count, penalty) is not recorded with it. A saved model is therefore only reproducible if the caller independently remembers the config -- and nothing says so, which is the trap: the seed's presence implies a completeness that is not there.\n\nEither record the config alongside the seed in data::Model, or document at the seed field that it is necessary but not sufficient.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:38:55Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:23:03Z","closed_at":"2026-09-19T21:23:03Z","close_reason":"Model::config: Option\u003cFitConfig\u003e recorded alongside the seed -- the preferred option, record rather than document. Both stay Option for the same reason: every FitConfig value is legal, so none can stand for unknown, exactly as 0 is a legal seed. The seed doc now says it is necessary but not sufficient and points at config. FINDING worth keeping: the first injection -- deleting serde(default) from config -- did NOT fail any test, because serde already treats an Option field as optional. That attribute is documentation, not the mechanism, and the PRE-EXISTING a_model_saved_before_seeds_still_loads added earlier this session is equally non-discriminating against the fault it appears to guard. The injection was re-aimed at faults that matter: an absent config loading as the DEFAULT rather than as unknown (caught), and one FitConfig field silently not persisted (caught). Per populate_before_roundtrip the fixture differs from Default in every field, and the round trip asserts the loaded config is not the default, so a silent substitution cannot pass.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-szrt","title":"Wire improvise-gp into improvise (nothing depends on it yet)","description":"Follow-up from improvise-izk (2026-09-19). crates/improvise-gp exists and is tested but has no consumers -- it is a bridge with nothing crossing it. The next step is a user-facing path: fit a tree from the current model/view and install the emitted formula, presumably as a command in the Cmd/Effect layer.\n\nDesign constraints when doing it: the Cmd must stay pure and return effects (the fit itself is expensive, so decide whether it belongs in an Effect or off the main loop), and the emitted formula must go through the same add_formula path as a hand-typed one so _Measure target handling stays identical.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:38:55Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:38:55Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-szrt","title":"Wire improvise-gp into improvise (nothing depends on it yet)","description":"Follow-up from improvise-izk (2026-09-19). crates/improvise-gp exists and is tested but has no consumers -- it is a bridge with nothing crossing it. The next step is a user-facing path: fit a tree from the current model/view and install the emitted formula, presumably as a command in the Cmd/Effect layer.\n\nDesign constraints when doing it: the Cmd must stay pure and return effects (the fit itself is expensive, so decide whether it belongs in an Effect or off the main loop), and the emitted formula must go through the same add_formula path as a hand-typed one so _Measure target handling stays identical.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-19T20:38:55Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:40Z","started_at":"2026-09-20T05:32:40Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-d44s","title":"gp_tree::data::preprocess writes to stderr from library code","description":"Follow-up from improvise-0yr (2026-09-19). preprocess() eprintln!s '(column X is entirely missing — skipped)'. That was correct in examples/fit_csv.rs but it is now library code, and a library deciding to write to the user's stderr is a layering problem -- the caller cannot suppress, redirect or localize it.\n\nLeft in place deliberately during 0yr so the extraction stayed byte-identical (the baselines it was verified against include that output). Fix by returning the skipped columns as data -- a warnings vec on Dataset, mirroring how improvise-io's build_model_with_warnings already surfaces import warnings rather than printing them.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:38:54Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:22:56Z","closed_at":"2026-09-19T21:22:56Z","close_reason":"Dataset gains pub warnings: Vec\u003cString\u003e; preprocess pushes the skipped-column notice instead of printing it, and fit_csv.rs prints the vec at the same point so the CLI output does not move. Mirrors improvise-io build_model_with_warnings, which the ticket named as the shape -- a Warning enum was rejected because design-principles enums-over-strings is about state, not human-readable notices. Bug-first: the test failed to COMPILE before the fix (no field warnings on Dataset, 5 sites). Fault injected: replaced the push with a discarded format!, so the column is still dropped but silently -- an_entirely_missing_column_is_skipped_and_reported_as_data failed (left 0, right 1) while 24 others stayed green. CLI output verified byte-identical across 9 scenarios.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-z5eg","title":"Gate improvise-gp in the CI wasm job","description":"Follow-up from improvise-izk (2026-09-19). The new crates/improvise-gp is not in the wasm job in .github/workflows/ci.yml. Its dependency set (gp-tree + improvise-formula) should be wasm-clean, but 'should be' is exactly what the job exists to stop us asserting -- see invariant(compile_check_is_not_runtime_proof) and the ywd precedent. Add it to the job and confirm, or record why it is deliberately excluded.","status":"closed","priority":3,"issue_type":"chore","owner":"el-github@elangley.org","created_at":"2026-09-19T20:38:54Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:15:51Z","closed_at":"2026-09-19T21:15:51Z","close_reason":"Added improvise-gp to the wasm job in .github/workflows/ci.yml. Verified by compiling first rather than asserting: cargo check -p improvise-gp --target wasm32-unknown-unknown finishes clean -- the review that filed this could not verify it (no wasm target installed there) and marked it SUSPECTED. Gated with a comment saying it is NOT in the browser bundle (nothing depends on it yet) but is a candidate -- fitting a tree in the browser -- and that staying wasm-clean is free now and expensive to retrofit. That answers the ticket's 'or record why it is deliberately excluded' branch by choosing the other one.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-heqk","title":"Lift parse_md's inner helpers to module scope in persistence/parse.rs","description":"Follow-up from improvise-nr5 (2026-09-19), explicitly out of scope there because nr5 was a pure relocation and this is a real restructuring.\n\nAfter the split, crates/improvise-io/src/persistence/parse.rs is 1378 lines, of which ~950 are the four pre-existing test modules. The remaining parse_md is ~300 lines of nested tree-walk with its helpers (next, first_str, extract_name, extract_name_pair) declared INSIDE the function body, so the parser cannot be reviewed section by section.\n\nLift those helpers and the per-section walkers to module scope. The pest grammar stays the single source of truth and must not change; the characterization_survives_the_move doctrine applies -- the 97 persistence tests must carry across unchanged in meaning.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:36:45Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:21:37Z","closed_at":"2026-09-19T21:21:37Z","close_reason":"Lifted to module scope: next, first_str, extract_name, extract_name_pair, and the PCategory/PView collectors; added one walker per top-level grammar section (walk_category_section, walk_formulas_section, walk_data_section, walk_view_section and friends) plus parse_tree and section_title. parse_md is now a three-line composition: build_workbook(walk_file(parse_tree(text)?)?). improv.pest untouched.\n\nBehaviour preservation proven in the strongest form available: only lines 1-332 were replaced, and the four test modules are BYTE-IDENTICAL afterwards (diff of the old tail against the new differs by one leading blank line). No test needed a new assertion because no test line changed at all. invariant(indexmap_order) holds -- PCategory::items is still a Vec replayed in file order, now documented on the field. No tests added deliberately: adding them would have muddied the byte-identical proof.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-3qz2","title":"Cloud-session tooling: session-start hook, code-review skill, nix input seeding","description":"Retroactive record of PR #7 (merged into main as f01a07a on 2026-09-19, branch claude/cloud-agent-tooling). Cloud sessions could not reach the tracker or build the project: nix develop needs flake inputs that are not fetchable there, and a session had no way to pick up bd state at start. Filed during the 2026-09-19 groom because the work landed with no issue of its own.","design":"Four artifacts: .claude/hooks/session-start.sh (bd prime at session start, with a tracker swap that has to be safe on every failure path), .claude/skills/code-review/SKILL.md (review skill grounded in context/design-principles.md + kb.pl; renamed from design-review so Copilot picks it up), scripts/nix-seed-flake-inputs.sh (seeds locked flake inputs over git so nix develop works in a cloud session without touching the flake), context/cloud-agents.md (what the cloud environment can and cannot reach). Five hardening commits followed review: pin the bd installer, stop hiding jq failures, guard mktemp, if/else for the cargo build check (SC2015), and a documented deliberate SC2016.","notes":"The durable lesson is in kb.pl as doctrine(force_the_failure) (84c3c70): four unchecked statuses on that branch each produced a usable-looking value on failure — bd init leaving an empty database, mv landing the new tracker inside the old directory, jq's status swallowed by a process substitution, and a failed mktemp turning $TMP/$NAME into an absolute path at the filesystem root. Reading the failure paths found none of them; stubbing mv and mktemp to fail did.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T19:49:34Z","created_by":"Edward Langley","updated_at":"2026-09-19T19:49:34Z","closed_at":"2026-09-19T19:49:34Z","close_reason":"Merged as f01a07a (PR #7, plus #8 review follow-up). Filed retroactively 2026-09-19 during the groom; the work predates the ticket.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-sef","title":"Audit and repair prolog-sessions/kb.pl, add a maintained consistency check","description":"Retroactive record of work already landed (a406b52, 3a58d3c, 3697171, 2026-08-30/31). kb.pl had been appended to every session without anyone checking that it loads or reasons correctly; it needed an audit, and then a mechanical check so the same drift cannot accumulate again.","design":"Two unsound rules: may_depend_on/2 compared a crate against a concern and left its second argument unconstrained (so may_depend_on(improvise_core, banana) succeeded), and before_fixing/2's A=C guard bound a fresh variable and constrained nothing. Replaced with concern/1 + may_use/2 + crate_depends_on/2 facts read from the Cargo.tomls, and illegal_dependency/2 for inverted layering. The MCP runtime's own predicates (sleep, js_eval, future, await_any) had been captured into the file by a saveSession and were removed. kb_report/0 and kb_consistent/0 enumerate kb_error/1 over four classes: domain integrity, layering, rule soundness (a rule must REJECT a sentinel from no declared domain), and functional keys. Wired into CI as a third job.","notes":"Proving the checker fires found two bugs in the checker itself: depends_transitively/2 looped forever on the very input illegal_dependency/2 exists to detect (now carries a visited set), and trealla rejects the unparenthesized :- discontiguous form that the MCP server had silently tolerated. Every check class was verified by injecting the fault it targets. 3697171 additionally recorded the backlog-grooming method as kb facts: cycle detection, priority inversion, stale file:line citations and dissolved premises are mechanically checkable; prose mentions of other issues are not (of 21 one-way mentions, 2 were real dependencies).","status":"closed","priority":3,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-09-19T19:34:39Z","created_by":"Edward Langley","updated_at":"2026-09-19T19:34:47Z","closed_at":"2026-09-19T19:34:43Z","close_reason":"Landed in a406b52 (rule repair + shim removal), 3a58d3c (kb_report/kb_consistent + CI job, with every check class fault-injected), 3697171 (backlog-grooming method recorded). Filed retroactively 2026-09-19 while reconciling commits against the tracker; the work itself predates the ticket.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-fuj","title":"Two effects read files directly instead of going through Storage","description":"Found during the wasm audit (improvise-ywd). improvise-6mq put persistence behind a Storage trait, but two effects still call std::fs directly and so bypass it:\n\n crates/improvise-command/src/ui/effect.rs:931 StartImportWizard -\u003e std::fs::read_to_string\n crates/improvise-command/src/ui/effect.rs:1052 LoadModel -\u003e std::fs::read_to_string\n\nBoth COMPILE for wasm32-unknown-unknown — std::fs exists on that target — but they fail at runtime in a browser, which is exactly the class of bug a cargo check gate cannot catch. That is why they are filed rather than fixed under the check-based acceptance criteria of improvise-ywd.\n\nLoadModel is the clearer case: it duplicates what persistence::load already does, and persistence::load_from(\u0026dyn Storage, path) now exists. StartImportWizard reads a JSON file to seed the wizard; it wants the same treatment.\n\nBlocked in practice on how a wasm build gets its Storage instance — App has no Storage today, the free functions persistence::save/load hardwire PhysicalFs. Deciding where the backend is injected (App field? a parameter on the effect? a thread-local?) is the real design question, and it also determines how improvise-i34 (OPFS backend) plugs in.","status":"closed","priority":3,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-08-30T16:50:43Z","created_by":"Edward Langley","updated_at":"2026-08-30T17:01:09Z","started_at":"2026-08-30T16:57:21Z","closed_at":"2026-08-30T17:01:09Z","close_reason":"Wider than filed. The two effects I named were the visible symptom; the actual gap was that App had no storage backend at all, so Save, SaveAs, ExportCsv, LoadModel and CSV import ALL hardwired the filesystem through persistence::save/load/export_csv and csv_parser::parse_csv.\n\nINJECTION POINT: App::storage: Box\u003cdyn Storage\u003e, chosen by the host at construction. App::new still supplies PhysicalFs so every existing caller and all 14 suites are untouched; App::with_storage takes any other backend. That is the plumbing improvise-i34 and improvise-9x6 need — a worker constructs App with an OPFS backend and nothing else changes.\n\nConsidered the alternatives named on the ticket: a parameter on each effect spreads the backend through every signature for something that is per-session config, and a thread-local hides a dependency that App is the natural owner of. App already owns the other host config (keymap_set, terminal size).\n\nimprovise-io keeps its path-taking wrappers so the native CLI is unchanged, each delegating to a backend- or bytes-taking variant:\n save -\u003e save_to, load -\u003e load_from (from improvise-6mq)\n export_csv -\u003e export_csv_to, via a new render_csv that produces bytes\n parse_csv_with_warnings -\u003e parse_csv_bytes_with_warnings\n\nRESULT: production code in improvise-command now contains no std::fs at all — only tests do.\n\nThe test is the deliverable, not the refactor: the_document_surface_runs_on_a_non_filesystem_backend drives SaveAs, ExportCsv, LoadModel and StartImportWizard over a MemoryFs with no disk touched, and inspects the backend afterwards to confirm the bytes landed there. Any of those effects reaching for std::fs fails it. That is the property a browser backend depends on, verified now rather than discovered later in wasm.\n\nAlso verified: both targets still check clean (native + wasm32 for improvise-io and improvise-command), and the real CLI still imports, runs a headless command, and exports CSV.","dependency_count":0,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"improvise-lae","title":"ZZ / :wq with no file path should prompt for a path instead of only erroring","description":"Follow-up to improvise-8tk. That fix stops ZZ/:wq from quitting when the save fails: with no file path the user now stays in the app with 'No file path — use :w \u003cpath\u003e' in the status bar. That is correct but not helpful — the vim reflex is ZZ, and the user has to know to type ':w \u003cpath\u003e' next.\n\nimprovise-8tk's fix sketch suggested opening the export prompt, but AppMode::ExportPrompt commits to ExportCsv (command/cmd/commit.rs::CommitExport) — it writes CSV, not a .improv model. There is no save-as prompt mode; ':w \u003cpath\u003e' goes straight through SaveOrSaveAs (command/cmd/effect_cmds.rs:390).\n\nOptions: (a) add a SavePrompt minibuffer mode built with the TextEntry builder in keymap.rs, committing to SaveAs; (b) make SaveAndQuit emit a mode change to that prompt when ctx has no file path — but CmdContext does not currently expose file_path, so that needs a context field.\n\nPrefer (a) plus a keymap binding; decide separately whether ZZ should route into it automatically.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-29T17:34:38Z","created_by":"Edward Langley","updated_at":"2026-08-29T17:34:38Z","dependency_count":0,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"improvise-lae","title":"ZZ / :wq with no file path should prompt for a path instead of only erroring","description":"Follow-up to improvise-8tk. That fix stops ZZ/:wq from quitting when the save fails: with no file path the user now stays in the app with 'No file path — use :w \u003cpath\u003e' in the status bar. That is correct but not helpful — the vim reflex is ZZ, and the user has to know to type ':w \u003cpath\u003e' next.\n\nimprovise-8tk's fix sketch suggested opening the export prompt, but AppMode::ExportPrompt commits to ExportCsv (command/cmd/commit.rs::CommitExport) — it writes CSV, not a .improv model. There is no save-as prompt mode; ':w \u003cpath\u003e' goes straight through SaveOrSaveAs (command/cmd/effect_cmds.rs:390).\n\nOptions: (a) add a SavePrompt minibuffer mode built with the TextEntry builder in keymap.rs, committing to SaveAs; (b) make SaveAndQuit emit a mode change to that prompt when ctx has no file path — but CmdContext does not currently expose file_path, so that needs a context field.\n\nPrefer (a) plus a keymap binding; decide separately whether ZZ should route into it automatically.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-08-29T17:34:38Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-ai7","title":"Collapse the row/col duplication in view/layout.rs","description":"Supersedes improvise-1wu and improvise-669. Both are the same defect — a row function and a col function that are near-identical — inside the same ~120-line region of crates/improvise-core/src/view/layout.rs, so they are one editing pass, not two.\n\n(1) from improvise-1wu — row_label (layout.rs:327) and col_label (layout.rs:342) are character-identical except for which items list they read. Extract fn axis_label(items: \u0026[AxisEntry], idx: usize) -\u003e String and call it from both. ~14 LOC.\n\n(2) from improvise-669 — the row prune loop (~layout.rs:249) and the col prune loop (~layout.rs:271) both walk a keep_row/keep_col bool vector while tracking data_idx separately from iteration order. The deep review flagged the data_idx tracking as brittle. Extract fn prune_axis_items(items: \u0026mut Vec\u003c_\u003e, keep: \u0026[bool]) so the invariant (data_idx == keep.len() at loop exit) is asserted in one place instead of zero. ~20 LOC. This is the half with actual correctness content: today a bug in the row prune can hide while the col prune stays correct.\n\nWHY TOGETHER: same file, same region, same row/col-symmetry smell, and the same test surface — layout tests would be re-read once instead of twice. Neither is big enough to justify its own session.\n\nDo (2) first; it is the one that can hide a bug. (1) is cosmetic.\n\nNOT included: improvise-rml (consolidate Axis display into methods on Axis) stays separate — different file (view/axis.rs plus the UI widgets) and a different concern (display strings and colors, not layout arithmetic).","notes":"STALE REF (groomed 2026-08-30, line numbers shifted by this session's work): view/layout.rs:249/271/327/342 -\u003e ALL FOUR are stale after improvise-c5v rewrote records mode; 249 and 271 now land inside the new records_columns(). The row/col duplication this ticket is about now lives in prune_empty (crates/improvise-core/src/view/layout.rs:321), which still filters row_items and col_items through two near-identical blocks. Re-verify the scope before starting: c5v deleted the unreachable records branch that used to sit at the end of that function.\nGROOMED 2026-09-19: prune_empty is layout.rs:323 now (the 2026-08-30 note said 321). Confirmed the c5v cleanup: the unreachable records tail is gone (improvise-ede closed today), so the remaining duplication is the row_items/col_items pair inside prune_empty plus row_label/col_label.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-14T01:11:07Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:33:46Z","closed_at":"2026-09-19T20:33:46Z","close_reason":"Collapsed the row/col duplication in view/layout.rs into four private free functions over \u0026[AxisEntry]: prune_axis_items (the half with correctness content -- the two hand-tracked data_idx loops in prune_empty now share one debug_assert on the parallel-length invariant, with bounds behaviour preserved via keep.get().unwrap_or(false)), axis_label, data_items (an iterator that replaced six copies of the DataItem filter), and data_index_to_visual. Net -46 lines. Deliberately left asymmetric: records mode entirely (rows are coordinate tuples, columns are measures plus the + column -- real asymmetry, not duplication), the has_value/keep_row/keep_col derivations, row_group_for/col_group_for, and the GridLayout row/col field pairing (unifying would be a speculative abstraction churning every consumer). No test changed meaning; 805 workspace tests pass, clippy clean.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-djn","title":"AddCategory effect silently swallows failures","description":"ui/effect.rs AddCategory does 'let _ = app.model_state.workbook.add_category(\u0026self.0)'. Duplicate names and the 12-regular-category limit are both dropped on the floor, so 'a' in the category panel with 12 categories already present reports 'Added category \"X\"' and adds nothing.\n\nAddItem and AddFormula already write their error to view_state.status_msg, and since improvise-6t3 the commit command emits its optimistic status BEFORE the add effect precisely so the add gets the last word. AddCategory just needs to do the same:\n\n if let Err(e) = app.model_state.workbook.add_category(\u0026self.0) {\n app.view_state.status_msg = e.to_string();\n }\n\nFound while unifying the commit commands (improvise-6t3). Needs a test that applies AddCategory over a full model and asserts the status bar reports the limit.","notes":"GROOMED 2026-09-19: still true; the site is crates/improvise-command/src/ui/effect.rs:71-74 (let _ = app.model_state.workbook.add_category(\u0026self.0)). Noticed while verifying: that effect declares EffectKind::Model(ModelChange::CellData), though adding a category changes the grid geometry — worth a look when fixing this, but not part of this ticket.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-08-01T23:22:54Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:35:55Z","closed_at":"2026-09-19T20:35:55Z","close_reason":"Bug confirmed by a failing test written first: add_category_past_the_limit_reports_the_failure fills to MAX_CATEGORIES and asserts the 13th reports the limit. It failed on exactly the status-bar assertion -- the category was correctly rejected, only the message was lost. One-line fix as the ticket specified; test kept as a regression guard with the bug in its doc-comment.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-toc","title":"Effect::changes_mode() is a static flag but ViewBack/ViewForward/StartImportWizard change the mode conditionally","description":"Effect::changes_mode() is a per-type constant, and its only consumer is ExecuteCommand: 'return to Normal unless a command already changed mode' (command/cmd/text_buffer.rs).\n\nThree effects mutate view_state.mode only on some runs, so no static answer is right:\n- ViewBack / ViewForward (ui/effect.rs) restore frame.mode only when the back/forward stack is non-empty.\n- StartImportWizard (ui/effect.rs) sets AppMode::ImportWizard only when the file parses.\n\nReporting false (today) means the success path is wrong: ':import data.csv' opens the wizard and is then snapped back to Normal, leaving the wizard drawn with the Normal keymap bound. Same for ':view-back' when there IS history. Reporting true would make the failure path wrong: a failed ':import' would strand the user in CommandMode.\n\nEnterEditAtCursor had the same symptom but is unconditional, so it was fixed by overriding changes_mode() (improvise-6sq).\n\nOptions:\n(a) have these effects set an app-level 'mode was changed' flag during apply, and let ExecuteCommand decide after applying rather than before — but ExecuteCommand is a pure Cmd and cannot observe apply.\n(b) split the conditional effects into their decision (in the Cmd, where the stack/file state is visible) and an unconditional ChangeMode effect. ViewBack's stack is on App though, not CmdContext.\n(c) give ExecuteCommand a trailing 'return to Normal if still in CommandMode' effect instead of the compile-time check — an effect CAN read app state at apply time, which is where this decision actually belongs.\n\n(c) looks cleanest and would let changes_mode() be deleted entirely.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-08-01T20:29:19Z","created_by":"Edward Langley","updated_at":"2026-09-19T19:48:50Z","closed_at":"2026-09-19T19:48:50Z","close_reason":"Fixed on the cloud-agent branch merged as f01a07a: a341717 \"settle : command mode by observation, not prediction\" plus e8dcdb1 (Copilot review follow-up). Effect::changes_mode() is gone; ExecuteCommand now ends every batch with ReturnToNormalUnlessMoved(ctx.mode) (effect.rs:556), which drops to Normal only if the mode is still what the batch started in — so ViewBack/ViewForward/StartImportWizard need no up-front answer. repo-map gotcha 31 documents it. Closed during the 2026-09-19 groom; the commits landed without closing the ticket.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-1c9","title":"ExportPrompt mode shows no status-bar hint","description":"AppMode::hint_text() returns \"\" for AppMode::ExportPrompt, so when the user hits the export prompt the status bar goes blank instead of telling them what to type (a path) or how to cancel (Esc).\n\nFound while making hint_text exhaustive in improvise-2hi: the old match had a '_ =\u003e \"\"' arm that hid the gap. The exhaustive match now names ExportPrompt explicitly, and app.rs::only_export_prompt_and_quit_have_no_hint pins the current behavior — update that test when adding the hint.\n\nQuit is also blank, but correctly so: it is never rendered.\n\nFix: give ExportPrompt a hint along the lines of 'Enter:export Esc:cancel — type a file path'.","notes":"GROOMED 2026-09-19: still true, new home — AppMode and hint_text moved to improvise-protocol in improvise-cqi. The empty arm is crates/improvise-protocol/src/lib.rs:218 (ExportPrompt and Quit share it; Quit having no hint is fine).","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-08-01T19:17:44Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:35:55Z","closed_at":"2026-09-19T20:35:55Z","close_reason":"Bug confirmed by a failing characterization test: only_export_prompt_and_quit_have_no_hint became only_quit_has_no_hint and failed with left: [EXPORT, QUIT]. Added the hint in improvise-protocol's AppMode::hint_text: 'Enter:export Esc:cancel — type a file path', with keys matching the actual ExportPrompt keymap (TextEntry + .commit(Enter, commit-export)). Quit keeps its blank arm and the doc-comment now says why.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7na","title":"Decide whether fitted gp-tree models are persisted alongside the formula","description":"The emitted formula is enough to recompute existing cells, so the .improv file needs nothing extra for the common case. But scoring NEW rows — the --predict path in fit_csv — needs the encoding plan (which columns are categorical, their training-set distinct values, imputation means), which the formula text does not carry.\n\nDecide: (a) formula only, and refitting is the way to score new data; (b) store the gp_tree Model as a sidecar/JSON blob; (c) store the seed + config so the fit is reproducible without storing the tree. Option (c) is cheap and matches the format's plain-text character, but only works if the training data is unchanged. This is a format decision, so settle it before the fit command ships and starts writing files.","acceptance_criteria":"A decision is recorded with its rationale, and the .improv grammar either changes or is explicitly confirmed unchanged.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:21Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:34:50Z","closed_at":"2026-09-20T00:34:50Z","close_reason":"DECIDED: (a) formula only. improv.pest explicitly confirmed unchanged; no code, three tests, recorded on improvise-gp crate docs.\n\nAGAINST THE BLOB, gotcha 41 SECOND half transfers and decides it: a blob is a second copy of what the file already carries -- the tree IS the formula -- with nothing keeping the two in agreement after a hand-edited formula or an edited cell, and a round-trip test over the blob cannot see that drift because it never compares the two. Gotcha 41 FIRST half does NOT transfer (an opaque production implements no grammar), and saying which half applies is the useful part. Plus Tree/FitConfig are gp-tree types, so improvise-io would gain a gp-tree dependency and .improv would track a third crate serde layout forever.\n\nWHAT THE BLOB WAS FOR DOES NOT ARISE: the encoding plan exists so fit_csv --predict can score a row arriving as untyped CSV text; improvise new row is a record in the same document, scored through the stored formula by recompute_formulas. Demonstrated by a_record_added_after_the_fit_is_scored_by_the_formula_alone, which adds a REGION THE FIT NEVER SAW and gets a correct score.\n\nSeed + config not written either: gp-tree records them already (gotcha 37b) and nothing reads them until a fit command exists -- provenance is that ticket call, alongside how it words the fit to the user.\n\nTHE TICKET PREMISE WAS FALSE AND THAT IS THE LARGER FINDING. \"The emitted formula is enough to recompute existing cells\" holds ONLY for a fit over numeric measures. preprocess one-hots a categorical column into a feature named Region=North -- an encoding artefact, not an item of the document -- so Expr::Ref fails find_item_category and EVERY CELL of the fitted measure is Error(\"ref:Region=North\"). col.missing and the col.year date family are in the same position. Pinned as a_split_on_a_one_hot_feature_does_not_evaluate (characterization; flip it when fixed).","dependencies":[{"issue_id":"improvise-7na","depends_on_id":"improvise-izk","type":"blocks","created_at":"2026-08-01T10:26:23Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"improvise-vx3","title":"fit-measure command: fit a tree and add it as a formula","description":"The user-facing surface. A command (:fit-measure \u003ctarget\u003e or similar) that extracts features from the active view, runs the fit, emits the champion as formula text and adds it as a new measure — after which it is an ordinary improvise formula: recomputed by the fixed-point evaluator, editable in the formula panel, saved in the ## Formulas section, drillable.\n\nReport quality alongside it, not just the rule: cross-validated R²/RMSE against the predict-the-mean baseline, and the permutation feature-importance ranking. A decision tree that looks tidy but generalizes no better than the mean is worse than useless — it is a rule the user will believe. Per the command/effect split, the fit belongs in a Cmd (pure over CmdContext) with effects that add the formula and set status.","acceptance_criteria":"(1) A command fits and installs the formula. (2) The formula survives save/load and recomputes. (3) Fit quality vs baseline is reported. (4) Fitting is cancellable or bounded — a GA run must not freeze the TUI.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:21Z","created_by":"Edward Langley","updated_at":"2026-08-01T17:25:21Z","dependencies":[{"issue_id":"improvise-vx3","depends_on_id":"improvise-0ai","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-4s8","type":"blocks","created_at":"2026-08-01T10:26:22Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-izk","type":"blocks","created_at":"2026-08-01T10:26:20Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-lzn","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":4,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"improvise-vx3","title":"fit-measure command: fit a tree and add it as a formula","description":"The user-facing surface. A command (:fit-measure \u003ctarget\u003e or similar) that extracts features from the active view, runs the fit, emits the champion as formula text and adds it as a new measure — after which it is an ordinary improvise formula: recomputed by the fixed-point evaluator, editable in the formula panel, saved in the ## Formulas section, drillable.\n\nReport quality alongside it, not just the rule: cross-validated R²/RMSE against the predict-the-mean baseline, and the permutation feature-importance ranking. A decision tree that looks tidy but generalizes no better than the mean is worse than useless — it is a rule the user will believe. Per the command/effect split, the fit belongs in a Cmd (pure over CmdContext) with effects that add the formula and set status.","acceptance_criteria":"(1) A command fits and installs the formula. (2) The formula survives save/load and recomputes. (3) Fit quality vs baseline is reported. (4) Fitting is cancellable or bounded — a GA run must not freeze the TUI.","status":"in_progress","priority":3,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:21Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:41Z","started_at":"2026-09-20T05:32:41Z","dependencies":[{"issue_id":"improvise-vx3","depends_on_id":"improvise-0ai","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-4s8","type":"blocks","created_at":"2026-08-01T10:26:22Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-izk","type":"blocks","created_at":"2026-08-01T10:26:20Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-lzn","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":4,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-4s8","title":"Extract a gp-tree feature matrix from a View's records","description":"Turn the active view's data into gp-tree's fixed-width f32 feature vectors: pick a target measure, use the remaining measures as numeric features and the regular categories as categorical features (one-hot, via the preprocessing promoted out of fit_csv). One row per distinct combination of the non-measure coordinates — which is what records mode already enumerates, so GridLayout's records path is the natural source rather than a new traversal.\n\nOpen questions worth settling here: whether hidden items and page filters should scope the training rows (the user's mental model is probably 'fit what I can see'), whether formula-derived measures may be used as features (they can, but a feature that already depends on the target makes the fit trivially perfect and useless — needs a guard), and what to do with Label categories, which are high-cardinality per-row text and would one-hot into thousands of columns.","acceptance_criteria":"(1) A function maps Model + View + target measure to (feature matrix, targets, Spec with names and ranges). (2) Spec ranges are supersets of the actual values — gp-tree's simplify prunes reachable branches otherwise, silently changing outputs. (3) A feature that is the target, or a formula depending on it, is rejected or excluded. (4) Tests over a small fixture model.","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:34:49Z","closed_at":"2026-09-20T00:34:49Z","close_reason":"crates/improvise-gp/src/dataset.rs: feature_table / dataset turn a View visible records into the string table gp_tree::data::preprocess reads. THE SHAPE WAS NOT INVENTED -- records mode ALREADY IS a feature matrix (gotcha 1), a row is one coordinate tuple and each measure a column -- so it enumerates the same rows against the USER OWN view rather than through GridLayout, which would need _Index/_Dim on the axes and therefore a different view from the one being looked at.\n\nFOUR SELECTION RULES, each a place a wrong answer is silent:\n * rows scoped by View::filtering_cats() (Page UNION Filter), NOT categories_on(Axis::Page) -- gotcha 27, and the Page answer quietly fits against rows a drill excluded;\n * hidden items scope rows AND feature columns (\"fit what I can see\"), an explicitly named target exempt;\n * every regular category is a feature WHATEVER AXIS it sits on including Axis::None, because a coordinate is part of what makes a record distinct -- drop a None category and two rows get identical feature vectors with different targets; Label categories excluded by not being is_regular, as in records_columns, since per-row text one-hots into a perfect memoriser;\n * NO FORMULA-DERIVED VALUE ENTERS AT ALL -- a formula target is not a feature and is refused AS the target, one rule covering both halves with no dependency analysis. Membership read off model.formulas(), never off absence from the explicit _Measure item list, because a name can be BOTH.\n\nTWO INJECTIONS FOUND WEAK TESTS AND CHANGED THE WORK, which is the discipline paying out: the formula-feature injection first PASSED because the fixture Profit was only implied by its formula and so never reached the explicit item list, making the filter dead code from the test point of view -- the fixture now lists Profit as an explicit item AS WELL AS a formula target, the only configuration in which that rule is observable. And dropping the record sort left records_come_out_in_coordinate_order green (2 rows), so that test was DELETED rather than kept vacuous, with the ordering claim moved to the exact-table test doc comment plus an honest note that a single-process test cannot distinguish sorted from hashed-that-way-today.\n\nRecords are sorted by coordinate tuple because matching_cells walks a HashSet and f32 accumulation is order-dependent. Numbers reach the table via f64::to_string, never format_f64 (gotcha 8). A record with no target value is dropped with a warning -- keeping it flips preprocess whole target to CLASSIFICATION on one unparseable cell. 13 tests.","dependencies":[{"issue_id":"improvise-4s8","depends_on_id":"improvise-0yr","type":"blocks","created_at":"2026-08-01T10:26:10Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-lzn","title":"Seeded Rng implementation for gp-tree fits in improvise","description":"gp_tree::Rng is a trait the caller implements, deliberately, so evolution is reproducible and the crate pulls in no RNG. improvise has no PRNG dependency today. Decide between a small vendored deterministic generator (xorshift/PCG, ~20 lines, no new dep) and taking on rand. Reproducibility is not optional here: a fit writes a formula into the user's file, so re-running the same fit on the same data must produce the same rule, and a bug report must be reconstructible from the seed. Whatever is chosen, the seed should be visible to the user and recorded with the fit.","acceptance_criteria":"(1) An Rng impl exists for improvise fits. (2) Same seed + same data + same config yields an identical formula, asserted by a test. (3) The seed is surfaced, not hidden.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-01T17:24:34Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:38:43Z","closed_at":"2026-09-19T20:38:43Z","close_reason":"Vendored SplitMix64 as gp_tree::SplitMix64 beside the Rng trait rather than adding a rand dependency -- the same generator was already written out three times, and the requirement is reproducibility, not statistical quality. Seed is surfaced, not hidden: SplitMix64::seed() keeps the starting value, fit_csv takes --seed (decimal or 0x hex) and prints it on every fit, and --save records it in data::Model::seed as #[serde(default)] so earlier models still load (verified by --predict against a pre-change baseline). Determinism tests pair the_same_seed_fits_the_same_formula with a_different_seed_fits_a_different_formula, so 'reproducible' cannot be satisfied by a search that never moved.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-0ai","title":"gp-tree: expose a fit/evolve entry point in the library","description":"The library exposes the GP operators — random_tree, mutate_at, crossover, subtree_at, graft_at, simplify, plus Population, GaConfig and breed — but no loop that drives them. The search itself lives in examples/fit_csv.rs, so every consumer reimplements generation-stepping, elitism, parsimony penalty and stopping. Add a fit entry point taking a dataset, a GaConfig and an Rng, returning the champion (simplified). Keep the k-fold cross-validation and permutation feature-importance reporting available too — those are what make the output trustworthy rather than just plausible.","acceptance_criteria":"(1) A public fit function evolves a champion from data + config + Rng. (2) fit_csv uses it. (3) Same seed and config produce the same tree.","notes":"GROOMED 2026-09-19 — STALE REF: examples/fit_csv.rs does not exist at that path; it is crates/gp-tree/examples/fit_csv.rs (gp-tree is a workspace member, not the root crate).","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:24:33Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:23:04Z","closed_at":"2026-09-19T21:23:04Z","close_reason":"New crates/gp-tree/src/fit.rs (425 lines, 7 tests): FitConfig with Default = the CLI historical numbers, fit() returning the simplified champion across restarts, plus cv_regress/cv_classify/folds/standardize/permutation_importance/mse. Nothing in the module prints -- CV returns pooled prediction/actual pairs and importance a score per feature; tables stay in the CLI, the same boundary d44s drew. fit_csv.rs drops 787 to 598 lines. Premise verified before designing: the generation loop really was Trainer::fit in the example with pop/gens as function-local consts. Fault injected: the generation loop bounded to zero, i.e. a fit returning its seed tree unevolved. The two DETERMINISM tests stayed GREEN -- they would have been fooled, which is why they are not sufficient -- while a_fit_improves_on_the_tree_it_started_from and the_config_changes_the_fit_at_a_fixed_seed failed. Reproducibility verified end to end: same seeds produce the same fits after promoting the loop, predictions and saved model included, so RNG call order is provably unchanged.","dependencies":[{"issue_id":"improvise-0ai","depends_on_id":"improvise-0yr","type":"blocks","created_at":"2026-08-01T10:26:10Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} @@ -220,18 +225,18 @@ {"_type":"issue","id":"improvise-8fy","title":"Introduce simple_effect! macro for single-field setter effects","description":"crates/improvise-command/src/ui/effect.rs has ~15 simple setter effects (SetSelected, SetRowOffset, SetColOffset, SetYanked, SetSearchQuery, SetSearchMode, SetStatus, etc.) that each implement the same 4-line apply: pub struct SetX(pub T); impl Effect { fn apply(\u0026self, app: \u0026mut App) { app.field = self.0.clone(); } }.\\n\\nAbstraction: simple_effect!(SetRowOffset, usize, row_offset) macro generates struct + Effect impl.\\n\\nCorrectness win: provides a single hook for cross-cutting concerns. If layout-affecting setters need to mark dirty or rebuild layout, one macro body covers all of them. Currently each setter independently decides whether to mark_dirty(), and silent omissions are plausible. ~50 LOC.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:54:14Z","created_by":"Ed L","updated_at":"2026-09-19T20:35:54Z","closed_at":"2026-09-19T20:35:54Z","close_reason":"simple_effect! macro added beside the Effect trait; eight single-field ViewOnly setters declared through it (-42 lines net). The macro HARD-CODES EffectKind::ViewOnly by design: an effect other clients must see is not 'simple' and keeps a hand-written impl, which is how gotcha 3's no-default-kind rule survives a macro at all. SetSelected (two fields) and SetNumberFormat (a Model change) stay hand-written. Existing effect tests already cover all eight, including view_only_effects_leave_shared_state_alone.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-1wu","title":"Extract axis label helper in view/layout.rs","description":"crates/improvise-core/crates/improvise-core/src/view/layout.rs:323-336 (row_label) and :338-351 (col_label) are character-identical except for which items list they read. Small syntactic duplication but trivial to consolidate.\\n\\nFix: fn axis_label(items: \u0026[AxisEntry], idx: usize) -\u003e String, call from both. ~14 LOC.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:54:13Z","created_by":"Ed L","updated_at":"2026-08-14T01:11:07Z","closed_at":"2026-08-14T01:11:07Z","dependencies":[{"issue_id":"improvise-1wu","depends_on_id":"improvise-ai7","type":"supersedes","created_at":"2026-08-13T18:11:07Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-669","title":"Extract axis pruning helper in view/layout.rs","description":"crates/improvise-core/crates/improvise-core/src/view/layout.rs:245-262 (rows) and :265-284 (cols) contain near-identical pruning loops that track data_idx separately from iteration order against a keep_row/keep_col bool vector. The data_idx tracking was flagged as brittle in the deep review.\\n\\nAbstraction: fn prune_axis_items(items: \u0026mut Vec\u003cAxisEntry\u003e, keep: \u0026[bool]) -\u003e Vec\u003cAxisEntry\u003e\\n\\nCorrectness win: centralizes the invariant (one place to assert data_idx == keep.len() at loop exit). A bug in the row prune can't hide while the col prune stays correct. ~20 LOC collapsed.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:54:12Z","created_by":"Ed L","updated_at":"2026-08-14T01:11:08Z","closed_at":"2026-08-14T01:11:08Z","dependencies":[{"issue_id":"improvise-669","depends_on_id":"improvise-ai7","type":"supersedes","created_at":"2026-08-13T18:11:08Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-aaw","title":"Replace CreateAndSwitchView with Binding::Sequence","description":"command/cmd/panel.rs:546-560 bundles create-view + switch-view + change-mode. Both primitives already exist via effect_cmds.\\n\\nFix: bind Sequence [(create-view nix-shell-env), (switch-view nix-shell-env), (enter-mode normal)] in the keymap. If the auto-generated view name is load-bearing, either thread through a minibuffer (matches other add flows) or keep a single create-view primitive that both creates and switches by default.\\n\\n~12 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nGROOMED 2026-09-19: still true; CreateAndSwitchView is cmd/panel.rs:658, not 546-560.\n\n--- INVESTIGATED 2026-09-19, NOT DONE: premise false ---\nCreateAndSwitchView is NOT a shim. It computes the new view's name from\ncontext -- format!(\"View {}\", ctx.workbook.views.len() + 1) -- and a static\nBinding::Sequence can only carry literal args, so [create-view X, switch-view X,\nenter-mode normal] would hard-code one name and produce duplicates on the\nsecond 'n'. The auto-generated name is load-bearing, so the command has a real\ndecision and is not replaceable by composition as written.\n\nThe ticket's own two fallbacks are both BEHAVIOUR changes needing a product\ndecision, not refactors:\n (a) thread the name through a new minibuffer mode -- that is a feature, and\n it changes what 'n' does in the view panel;\n (b) make the create-view primitive also switch -- that changes :create-view\n for scripts and duplicates what grid.rs already does by hand for the\n _Drill/_Records views.\nNeither was forced. RE-SCOPE this ticket to 'should creating a view from the\npanel prompt for a name?' or close it won't-fix.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:54Z","created_by":"Ed L","updated_at":"2026-09-19T20:36:35Z","dependencies":[{"issue_id":"improvise-aaw","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:23Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-git","title":"Replace SaveAndQuit with Binding::Sequence in keymap","description":"command/cmd/mode.rs:232-241 is an 8-line command chaining Save + ChangeMode(Quit). Both primitives already exist.\\n\\nFix: delete the struct; bind via Sequence [(w, []), (force-quit, [])] in the keymap.\\n\\nTrivial application of the 'if the keybinding is A then B, the binding should be a Sequence, not a command named AAndB' principle.\\n\\n~8 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nUNBLOCKED (groomed 2026-08-30): improvise-8tk is closed, and its fix removes this ticket's stated objection rather than merely satisfying it.\n\nThe blocker was recorded as: 'a Sequence has no way to stop between steps, so converting first would cement the ZZ/:wq data-loss bug into the keymap'. That is no longer true. Keymap::dispatch flattens a Binding::Sequence into ONE effect vec applied by a single App::apply_effects call, and the Save effect now sets abort_effects when the save fails — so the abort carries across sequence steps exactly as it does within a command's own effect list. A [w, force-quit] Sequence would stop after a failed save.\n\nVerified by ui/effect.rs::tests::save_without_file_path_aborts_the_chain and failed_save_to_path_aborts_the_chain, plus ui/app.rs::tests::prefix_z_then_z_reaches_save_and_quit, which drives the real ZZ path.\n\nSo the conversion is now safe. Still check improvise-lae first: it may change what ZZ does when there is no file path, which would change what the Sequence should contain.\nGROOMED 2026-09-19: the \"still check improvise-lae first\" sentence in this ticket is a real ordering constraint, now recorded as a dependency edge rather than prose. Stale ref: SaveAndQuit is cmd/mode.rs:243 (cited 232-241).","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:48Z","created_by":"Ed L","updated_at":"2026-09-19T19:47:59Z","dependencies":[{"issue_id":"improvise-git","depends_on_id":"improvise-8tk","type":"blocks","created_at":"2026-08-13T18:08:05Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-lae","type":"blocks","created_at":"2026-09-19T12:47:58Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:24Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-aaw","title":"Replace CreateAndSwitchView with Binding::Sequence","description":"command/cmd/panel.rs:546-560 bundles create-view + switch-view + change-mode. Both primitives already exist via effect_cmds.\\n\\nFix: bind Sequence [(create-view nix-shell-env), (switch-view nix-shell-env), (enter-mode normal)] in the keymap. If the auto-generated view name is load-bearing, either thread through a minibuffer (matches other add flows) or keep a single create-view primitive that both creates and switches by default.\\n\\n~12 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nGROOMED 2026-09-19: still true; CreateAndSwitchView is cmd/panel.rs:658, not 546-560.\n\n--- INVESTIGATED 2026-09-19, NOT DONE: premise false ---\nCreateAndSwitchView is NOT a shim. It computes the new view's name from\ncontext -- format!(\"View {}\", ctx.workbook.views.len() + 1) -- and a static\nBinding::Sequence can only carry literal args, so [create-view X, switch-view X,\nenter-mode normal] would hard-code one name and produce duplicates on the\nsecond 'n'. The auto-generated name is load-bearing, so the command has a real\ndecision and is not replaceable by composition as written.\n\nThe ticket's own two fallbacks are both BEHAVIOUR changes needing a product\ndecision, not refactors:\n (a) thread the name through a new minibuffer mode -- that is a feature, and\n it changes what 'n' does in the view panel;\n (b) make the create-view primitive also switch -- that changes :create-view\n for scripts and duplicates what grid.rs already does by hand for the\n _Drill/_Records views.\nNeither was forced. RE-SCOPE this ticket to 'should creating a view from the\npanel prompt for a name?' or close it won't-fix.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:54Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:31Z","started_at":"2026-09-20T05:32:31Z","dependencies":[{"issue_id":"improvise-aaw","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:23Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-git","title":"Replace SaveAndQuit with Binding::Sequence in keymap","description":"command/cmd/mode.rs:232-241 is an 8-line command chaining Save + ChangeMode(Quit). Both primitives already exist.\\n\\nFix: delete the struct; bind via Sequence [(w, []), (force-quit, [])] in the keymap.\\n\\nTrivial application of the 'if the keybinding is A then B, the binding should be a Sequence, not a command named AAndB' principle.\\n\\n~8 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nUNBLOCKED (groomed 2026-08-30): improvise-8tk is closed, and its fix removes this ticket's stated objection rather than merely satisfying it.\n\nThe blocker was recorded as: 'a Sequence has no way to stop between steps, so converting first would cement the ZZ/:wq data-loss bug into the keymap'. That is no longer true. Keymap::dispatch flattens a Binding::Sequence into ONE effect vec applied by a single App::apply_effects call, and the Save effect now sets abort_effects when the save fails — so the abort carries across sequence steps exactly as it does within a command's own effect list. A [w, force-quit] Sequence would stop after a failed save.\n\nVerified by ui/effect.rs::tests::save_without_file_path_aborts_the_chain and failed_save_to_path_aborts_the_chain, plus ui/app.rs::tests::prefix_z_then_z_reaches_save_and_quit, which drives the real ZZ path.\n\nSo the conversion is now safe. Still check improvise-lae first: it may change what ZZ does when there is no file path, which would change what the Sequence should contain.\nGROOMED 2026-09-19: the \"still check improvise-lae first\" sentence in this ticket is a real ordering constraint, now recorded as a dependency edge rather than prose. Stale ref: SaveAndQuit is cmd/mode.rs:243 (cited 232-241).","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:48Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:31Z","started_at":"2026-09-20T05:32:31Z","dependencies":[{"issue_id":"improvise-git","depends_on_id":"improvise-8tk","type":"blocks","created_at":"2026-08-13T18:08:05Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-lae","type":"blocks","created_at":"2026-09-19T12:47:58Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:24Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-s13","title":"Delete mode-entry shim commands; use enter-mode with arg","description":"command/cmd/mode.rs:311-319 has EnterExportPrompt and similar shims (EnterSearchMode, EnterTileSelect, EnterFormulaEdit) that are each ~9-line wrappers around change_mode(AppMode::foo()). The generic EnterMode already exists with parse_mode_name() in the registry.\\n\\nFix: delete the shims; bind 'enter-mode' with the mode name as an arg in the keymap.\\n\\nCorrectness win: one code path for mode transitions. A new invariant on mode enter (e.g., 'always rebuild layout') needs to be added in one place instead of N shims.\\n\\n~40 LOC across all shims.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nGROOMED 2026-09-19: still true; the shims have moved apart — EnterExportPrompt cmd/mode.rs:326, EnterSearchMode :338, EnterTileSelect :353, and EnterFormulaEdit is in cmd/panel.rs:465, not mode.rs.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:44Z","created_by":"Ed L","updated_at":"2026-09-19T20:35:53Z","closed_at":"2026-09-19T20:35:53Z","close_reason":"Deleted the two genuine shims: EnterExportPrompt and EnterFormulaEdit, rebinding Ctrl-e to 'enter-mode export-prompt' and formula-panel a/n/o to 'enter-mode formula-edit'. Premise was half false -- the ticket named four. EnterSearchMode changes no mode at all (SearchMode is Normal + a flag, not an AppMode) and EnterTileSelect guards on category count and seeds SetTileCatIdx(0); both are real commands and were left alone. Side effect, a fix not a regression: EnterMode clears the target mode's minibuffer, which the shims did not, though EnterFormulaEdit's doc-comment already claimed it did.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-7yx","title":"Close test gaps flagged by deep review","description":"Deep review identified several test-coverage gaps worth closing:\\n\\n1. Keymap fallback chain: command/keymap.rs has 22 tests but no case for 'transient keymap exists but key does not match' (app.rs:396-406).\\n2. Ambiguous date detection: import/analyzer.rs:49-77 first-match heuristic not tested against ['01/02/2025', '02/03/2025'] style ambiguity.\\n3. CSV adversarial inputs: no tests for BOM, CRLF, empty file, single-column file.\\n4. Records missing a category field mid-import: wizard.rs:160-229 has the valid=false/break path but it is not covered by build_model_cells_match_source_data or similar.\\n5. Gzip corruption error path: save_and_load_roundtrip_gzip tests happy path only; add a test that load() on a truncated .gz file surfaces an error.\\n\\nLow-effort, high-value additions. Batch into a single test-coverage PR.","notes":"GROOMED 2026-09-19 — two of the five items are now covered; three remain. (1) Keymap fallback COVERED: ui/app.rs::tests::unmatched_key_after_prefix_is_swallowed, added by improvise-jc8 when the transient keymap became a prefix address. (4) Records missing a category mid-import COVERED: wizard.rs::build_model_record_with_missing_category_value_skipped and build_model_warns_about_records_skipped_for_missing_category. STILL OPEN: (2) ambiguous date detection — analyzer.rs has detect_iso_date_format / detect_us_date_format / detect_no_date_format and no case where two readings both parse; (3) CSV adversarial — rfc4180 quoting and short rows ARE tested; BOM, CRLF, empty file and single-column file are not; (5) gzip corruption — only gzip_round_trips_through_memory_storage and save_and_load_roundtrip_gzip, both happy-path. Also all paths in the description predate the crate split: command/keymap.rs -\u003e crates/improvise-command/src/command/keymap.rs, import/* and persistence/* -\u003e crates/improvise-io/src/.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:19:07Z","created_by":"Ed L","updated_at":"2026-09-19T19:49:03Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-7yx","title":"Close test gaps flagged by deep review","description":"Deep review identified several test-coverage gaps worth closing:\\n\\n1. Keymap fallback chain: command/keymap.rs has 22 tests but no case for 'transient keymap exists but key does not match' (app.rs:396-406).\\n2. Ambiguous date detection: import/analyzer.rs:49-77 first-match heuristic not tested against ['01/02/2025', '02/03/2025'] style ambiguity.\\n3. CSV adversarial inputs: no tests for BOM, CRLF, empty file, single-column file.\\n4. Records missing a category field mid-import: wizard.rs:160-229 has the valid=false/break path but it is not covered by build_model_cells_match_source_data or similar.\\n5. Gzip corruption error path: save_and_load_roundtrip_gzip tests happy path only; add a test that load() on a truncated .gz file surfaces an error.\\n\\nLow-effort, high-value additions. Batch into a single test-coverage PR.","notes":"GROOMED 2026-09-19 — two of the five items are now covered; three remain. (1) Keymap fallback COVERED: ui/app.rs::tests::unmatched_key_after_prefix_is_swallowed, added by improvise-jc8 when the transient keymap became a prefix address. (4) Records missing a category mid-import COVERED: wizard.rs::build_model_record_with_missing_category_value_skipped and build_model_warns_about_records_skipped_for_missing_category. STILL OPEN: (2) ambiguous date detection — analyzer.rs has detect_iso_date_format / detect_us_date_format / detect_no_date_format and no case where two readings both parse; (3) CSV adversarial — rfc4180 quoting and short rows ARE tested; BOM, CRLF, empty file and single-column file are not; (5) gzip corruption — only gzip_round_trips_through_memory_storage and save_and_load_roundtrip_gzip, both happy-path. Also all paths in the description predate the crate split: command/keymap.rs -\u003e crates/improvise-command/src/command/keymap.rs, import/* and persistence/* -\u003e crates/improvise-io/src/.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:19:07Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:38Z","started_at":"2026-09-20T05:32:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-61f","title":"Add invariant test for command registry name mapping","description":"crates/improvise-command/src/command/cmd/registry.rs is 586 lines with 0 tests. Each registration manually duplicates the command name as a string alongside the Cmd::name() implementation on the struct. Nothing verifies they match. Fix: add a test that iterates all registered entries and asserts registry_name == command.name() after construction. Low effort, locks down a class of typo bugs.","notes":"Groomed 2026-06-09: complementary to improvise-9cn (register_cmd! macro). The macro prevents future drift at compile/registration time; this test catches drift that predates or escapes the macro. Doing 9cn first likely shrinks this to a trivial registry-walk test — sequence them together.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:55Z","created_by":"Ed L","updated_at":"2026-08-29T19:10:27Z","closed_at":"2026-08-29T19:10:27Z","close_reason":"Delivered as part of improvise-9cn. See that issue for why the invariant as specified here ('assert registry_name == command.name() for all entries') is necessary but NOT sufficient: a wrong-variant prototype collides with its sibling rather than producing a mismatch, so the roundtrip test passes while a command name vanishes from the registry. Shipped alongside registered_names_are_unique, which is the check that actually reaches that class, plus a keymap-side test that every bound command name resolves. All four verified by fault injection.","dependencies":[{"issue_id":"improvise-61f","depends_on_id":"improvise-9cn","type":"blocks","created_at":"2026-04-16T11:54:34Z","created_by":"Ed L","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-nr5","title":"Split persistence/mod.rs into focused sub-modules","description":"crates/improvise-io/crates/improvise-io/src/persistence/mod.rs is 2402 lines mixing four cohesive units: pipe quoting (lines ~20-82), format_md (~155-282), parse_md (~285-580), export_csv (~581-630). Split into persistence/{quoting,format,parse,export}.rs behind the existing pub facade; no public-API change. Improves navigability and makes adding new format features easier to review.","notes":"GROOMED 2026-09-19 — STALE REFS. Path was written twice: crates/improvise-io/crates/improvise-io/src/persistence/mod.rs -\u003e crates/improvise-io/src/persistence/mod.rs. Size is 2783 lines now, not 2402, and every line range in the description is stale. Current boundaries by symbol (prefer these; they survive edits): quoting = is_bare_name/escape_pipe/unescape_pipe/quote_name/pipe_quote (24-86) plus format_number/parse_number_cell; entry points save/load/save_to/load_from/autosave_path (128-178); format_md (197); parse_md (330); coord_str (630); export_csv/export_csv_to/render_csv (639-700). Tests start at line 706 (#[cfg(test)]) and are roughly three quarters of the file — the split should move each function with its tests.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:52Z","created_by":"Ed L","updated_at":"2026-09-19T20:36:35Z","closed_at":"2026-09-19T20:36:35Z","close_reason":"Split persistence/mod.rs (2783 lines) into six files: mod.rs (259, document lifecycle + facade + test_helpers), quoting.rs (63, the five pipe-quoting helpers as pub(super)), format.rs (350), parse.rs (1378, incl. the four existing test modules moved whole), export.rs (156), roundtrip_tests.rs (635). Two deviations from the ticket sketch, both deliberate: format_number/parse_number_cell went to their own direction's file rather than into quoting.rs, leaving quoting.rs a real shared leaf; and the 34 tests that call BOTH format_md and parse_md got their own file rather than being arbitrarily assigned. Pure relocation verified body-by-body: every function body identical modulo indentation, only signature-level pub(super)/pub changes; persistence test count 97 before and after; 815 workspace tests pass; wasm, clippy and fmt clean. Grammar untouched.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-t8s","title":"View::axis_of panics on missing category","description":"crates/improvise-core/src/view/types.rs:118 uses expect('axis_of called for category not registered with this view'). The invariant is that Workbook::add_category registers every category with every view, but the panic surface is fragile for new callers. Fix: either audit all callers and document the invariant clearly, or change signature to fn axis_of(\u0026self, cat: \u0026str) -\u003e Option\u003cAxis\u003e and let callers handle None.","status":"closed","priority":3,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:48Z","created_by":"Ed L","updated_at":"2026-06-10T04:46:47Z","closed_at":"2026-06-10T04:46:47Z","close_reason":"Fixed (test-first where applicable): records_display returns None for OOB columns (f04fe51); try_axis_of added + axis_of invariant documented + cycle_axis no longer panics on unregistered categories (f04fe51); scrolled header suppression fixed via pure show_sublabel helper for both row and column headers (df9a02b); CyclePanelFocus cycles from current panel with wrap (9e02245); AGENTS.md merge target corrected (9634c95).","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-oaq","title":"Redundant file re-read in ImportJsonHeadless effect","description":"src/ui/effect.rs:846 chains 'serde_json::from_str(\u0026std::fs::read_to_string(\u0026self.path).unwrap_or_default()).unwrap_or(serde_json::Value::Array(records.clone()))'. This re-reads and re-parses the JSON file after records were already parsed earlier (lines 799-812). Dead code path from earlier refactor. Fix: remove the redundant read, use the already-parsed records array.","status":"closed","priority":3,"issue_type":"bug","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:45Z","created_by":"Ed L","updated_at":"2026-06-10T04:46:39Z","closed_at":"2026-06-10T04:46:39Z","close_reason":"Fixed in f0b9227/77a5124 (test-first): short CSV rows padded with explicit nulls + per-column warnings threaded to status line (stderr on CLI path); wizard pre-checks 12-category limit at advance with actionable message rendered in UI; ImportJsonHeadless parses the file exactly once via extracted json_import_records helper.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-mzv","title":"Import wizard does not pre-check 12-category limit","description":"crates/improvise-io/src/import/wizard.rs:84-150 build_model() fails when \u003e12 categories are proposed, but the ImportWizard UI does not pre-check before allowing the user to confirm. A 20-column CSV imported as 20 categories fails late with 'Category limit exceeded' after the user commits. Fix: validate category count in the UI step and require the user to mark some as Measure/Skip before proceeding. Also add a test for the \u003e12 case.","status":"closed","priority":3,"issue_type":"bug","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:44Z","created_by":"Ed L","updated_at":"2026-06-10T04:46:38Z","closed_at":"2026-06-10T04:46:38Z","close_reason":"Fixed in f0b9227/77a5124 (test-first): short CSV rows padded with explicit nulls + per-column warnings threaded to status line (stderr on CLI path); wizard pre-checks 12-category limit at advance with actionable message rendered in UI; ImportJsonHeadless parses the file exactly once via extracted json_import_records helper.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-0hb","title":"Make Sequence bindings transactional at command level","description":"Currently Keymap::dispatch (crates/improvise-command/src/command/keymap.rs:261-267) calls each Sequence step's execute(ctx) with the same pre-dispatch ctx. Effects accumulate in order and apply atomically as a batch via App::apply_effects, but later commands in a Sequence cannot observe earlier commands' effects through ctx — only effects-after-effects can see prior mutations.\n\nFor example, the records 'o' sequence [add-record-row, enter-edit-at-cursor records-editing] works only because the EnterEditAtCursor effect itself calls app.rebuild_layout() before reading display_value. Effects can self-heal mid-batch, but commands cannot.\n\nTrue command-level transactionality would require, between Sequence steps: apply effects-so-far, rebuild layout, build a fresh ctx, then dispatch the next command. This would change the semantics of every existing Sequence and is worth its own design pass.\n\nAcceptance: design doc + implementation; existing Sequences keep working; new tests prove a later step sees prior steps' model mutations.","notes":"Surfaced while parameterizing EnterEditAtCursor / CommitAndAdvance to remove is_records() runtime checks (improvise-4ju).\nGROOMED 2026-09-19: still true; Keymap::dispatch is keymap.rs:255-272 now (cited 261-267) and the Sequence arm still feeds every step the same pre-dispatch ctx.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-16T05:45:49Z","created_by":"fido","updated_at":"2026-09-19T19:49:14Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-m91","title":"(Stretch) Command log + native undo/replay","description":"Once Commands are the universal unit of intent, log them per-session and support undo/redo via inverse commands or periodic snapshots. Enabled by the unified reducer; valuable for all three deployment modes but especially natural in the native TUI. Also enables debugging and reproducing bugs: a command log is a deterministic replay.","design":"Two approaches: (1) Inverse commands — each command definition includes how to undo it; undo-stack walks the log in reverse applying inverses. Clean but requires per-command inverse logic. (2) Snapshot-and-replay — periodic snapshots of ModelState + current log index; undo rolls back to nearest snapshot and replays up to target index. Simpler, works for any command, costs memory + replay time. Start with (2) for simplicity. The log is per-session (ClientSession.command_log) and survives only within a session. Persistence to disk as a .improv-journal companion file is a further enhancement. Replay use case: load a file, enable log recording, hit a bug, save the log, bug-report-as-replay.","acceptance_criteria":"(1) Every dispatched Command is logged in the ClientSession (or equivalent server-side state). (2) An Undo Command pops the log and restores state via snapshot+replay. (3) Redo re-applies. (4) Native TUI bindings for Ctrl-Z / Ctrl-Shift-Z. (5) Replay test: record a sequence, reset, replay, assert final state equals record-end state.","notes":"Stretch goal — valuable across all deployment modes but not part of the core architectural unification. Depends on issue improvise-gxi (reduce_full) and ideally on a persistence mechanism (improvise-6mq VFS abstraction) if logs should survive restart.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:53:02Z","created_by":"spot","updated_at":"2026-07-30T23:06:32Z","dependencies":[{"issue_id":"improvise-m91","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:44Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-uq7","title":"(Stretch) Hybrid mode: native TUI with remote browser observer","description":"Once the native binary has the in-process client/server split (issue improvise-e0u), allow a second subscriber to attach via a local ws-server thread. Result: a developer running the native TUI can share their session with a teammate in a browser, or have their own browser tab observing while they work in the terminal. Both subscribers receive projection commands from the same in-process server; commands from either side flow through the same reduce_full.","design":"Native binary gains a --serve PORT flag (or similar) that, when set, spawns a ws-server thread alongside the native client. The ws-server is exactly improvise-q08's server logic, attached to the same in-process App as the native client. Two ClientSessions: one for the local ratatui client (zero-transport), one (or more) for remote websocket clients. Each has its own ViewState + render cache + subscription set. The projection layer fans out effect-driven updates to every session. The two subscribers see independent cursor/scroll/mode state (because view state is per-session) but the same model state (because model is shared). This is the deployment mode that fully justifies per-session view state tracking on the server.","acceptance_criteria":"(1) Native binary has an optional --serve flag that spawns a ws-server in the same process. (2) A remote browser can connect and see the native user's current session (shared model, independent view). (3) Edits from either side show up on the other after a round trip. (4) Graceful disconnect/reconnect of remote clients while native keeps running.","notes":"Stretch goal — not required for the core unification. Depends on issue e0u (binary split done) and improvise-q08 (ws-server exists). Orthogonal to the browser and standalone epics. The architectural investment in per-session view state and subscription-based projections is what makes this possible with minimal additional code.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:52:57Z","created_by":"spot","updated_at":"2026-07-30T23:06:32Z","dependencies":[{"issue_id":"improvise-uq7","depends_on_id":"improvise-e0u","type":"blocks","created_at":"2026-04-14T00:53:42Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-uq7","depends_on_id":"improvise-q08","type":"blocks","created_at":"2026-04-14T00:53:43Z","created_by":"spot","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-0hb","title":"Make Sequence bindings transactional at command level","description":"Currently Keymap::dispatch (crates/improvise-command/src/command/keymap.rs:261-267) calls each Sequence step's execute(ctx) with the same pre-dispatch ctx. Effects accumulate in order and apply atomically as a batch via App::apply_effects, but later commands in a Sequence cannot observe earlier commands' effects through ctx — only effects-after-effects can see prior mutations.\n\nFor example, the records 'o' sequence [add-record-row, enter-edit-at-cursor records-editing] works only because the EnterEditAtCursor effect itself calls app.rebuild_layout() before reading display_value. Effects can self-heal mid-batch, but commands cannot.\n\nTrue command-level transactionality would require, between Sequence steps: apply effects-so-far, rebuild layout, build a fresh ctx, then dispatch the next command. This would change the semantics of every existing Sequence and is worth its own design pass.\n\nAcceptance: design doc + implementation; existing Sequences keep working; new tests prove a later step sees prior steps' model mutations.","notes":"Surfaced while parameterizing EnterEditAtCursor / CommitAndAdvance to remove is_records() runtime checks (improvise-4ju).\nGROOMED 2026-09-19: still true; Keymap::dispatch is keymap.rs:255-272 now (cited 261-267) and the Sequence arm still feeds every step the same pre-dispatch ctx.","status":"in_progress","priority":3,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T05:45:49Z","created_by":"fido","updated_at":"2026-09-20T05:32:32Z","started_at":"2026-09-20T05:32:32Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-m91","title":"(Stretch) Command log + native undo/replay","description":"Once Commands are the universal unit of intent, log them per-session and support undo/redo via inverse commands or periodic snapshots. Enabled by the unified reducer; valuable for all three deployment modes but especially natural in the native TUI. Also enables debugging and reproducing bugs: a command log is a deterministic replay.","design":"Two approaches: (1) Inverse commands — each command definition includes how to undo it; undo-stack walks the log in reverse applying inverses. Clean but requires per-command inverse logic. (2) Snapshot-and-replay — periodic snapshots of ModelState + current log index; undo rolls back to nearest snapshot and replays up to target index. Simpler, works for any command, costs memory + replay time. Start with (2) for simplicity. The log is per-session (ClientSession.command_log) and survives only within a session. Persistence to disk as a .improv-journal companion file is a further enhancement. Replay use case: load a file, enable log recording, hit a bug, save the log, bug-report-as-replay.","acceptance_criteria":"(1) Every dispatched Command is logged in the ClientSession (or equivalent server-side state). (2) An Undo Command pops the log and restores state via snapshot+replay. (3) Redo re-applies. (4) Native TUI bindings for Ctrl-Z / Ctrl-Shift-Z. (5) Replay test: record a sequence, reset, replay, assert final state equals record-end state.","notes":"Stretch goal — valuable across all deployment modes but not part of the core architectural unification. Depends on issue improvise-gxi (reduce_full) and ideally on a persistence mechanism (improvise-6mq VFS abstraction) if logs should survive restart.\nDEFERRED 2026-09-19 by user ruling. Stretch item, and its natural home is the e0u restructuring (also deferred): a command log is most useful once Commands are genuinely the universal unit of intent through one reducer, which is what e0u finishes.","status":"deferred","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:53:02Z","created_by":"spot","updated_at":"2026-09-20T05:31:02Z","dependencies":[{"issue_id":"improvise-m91","depends_on_id":"improvise-gxi","type":"blocks","created_at":"2026-04-14T00:53:44Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-uq7","title":"(Stretch) Hybrid mode: native TUI with remote browser observer","description":"Once the native binary has the in-process client/server split (issue improvise-e0u), allow a second subscriber to attach via a local ws-server thread. Result: a developer running the native TUI can share their session with a teammate in a browser, or have their own browser tab observing while they work in the terminal. Both subscribers receive projection commands from the same in-process server; commands from either side flow through the same reduce_full.","design":"Native binary gains a --serve PORT flag (or similar) that, when set, spawns a ws-server thread alongside the native client. The ws-server is exactly improvise-q08's server logic, attached to the same in-process App as the native client. Two ClientSessions: one for the local ratatui client (zero-transport), one (or more) for remote websocket clients. Each has its own ViewState + render cache + subscription set. The projection layer fans out effect-driven updates to every session. The two subscribers see independent cursor/scroll/mode state (because view state is per-session) but the same model state (because model is shared). This is the deployment mode that fully justifies per-session view state tracking on the server.","acceptance_criteria":"(1) Native binary has an optional --serve flag that spawns a ws-server in the same process. (2) A remote browser can connect and see the native user's current session (shared model, independent view). (3) Edits from either side show up on the other after a round trip. (4) Graceful disconnect/reconnect of remote clients while native keeps running.","notes":"Stretch goal — not required for the core unification. Depends on issue e0u (binary split done) and improvise-q08 (ws-server exists). Orthogonal to the browser and standalone epics. The architectural investment in per-session view state and subscription-based projections is what makes this possible with minimal additional code.\nDEFERRED 2026-09-19: blocked by improvise-e0u, which was deferred by user ruling this session. Hybrid mode needs the in-process client/server split before a second subscriber can attach.","status":"deferred","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:52:57Z","created_by":"spot","updated_at":"2026-09-20T05:31:02Z","dependencies":[{"issue_id":"improvise-uq7","depends_on_id":"improvise-e0u","type":"blocks","created_at":"2026-04-14T00:53:42Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-uq7","depends_on_id":"improvise-q08","type":"blocks","created_at":"2026-04-14T00:53:43Z","created_by":"spot","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-9ix","title":"SQLite persistence format (alternative save/restore)","description":"Implement .sqlite as an alternative file format alongside .improv. Tables for categories, items, cells, formulas, views. Works over any VFS backend — native filesystem, OPFS, etc. — because the underlying storage layer is abstracted. Enables structured queries over model state and provides a more scalable container than plaintext for very large models. Orthogonal to the core standalone deployment; filed as a backlog follow-on since the MVP doesn't need it.","design":"Use rusqlite with the 'bundled' feature so sqlite itself compiles into the wasm bundle. Schema: one row per cell keyed by (coord_hash, category_coords...); one row per category with kind + ordering; one row per item with group membership; one row per formula with raw + target. Views stored as JSON or as structured rows. Persistence API mirrors .improv: save_sqlite(storage, path) / load_sqlite(storage, path). Rusqlite on wasm may need custom VFS hookup — investigate sqlite-wasm-rs or similar for browser-friendly builds. Round-trip tests: model→sqlite→model must be identity.","acceptance_criteria":"(1) save_sqlite and load_sqlite functions exist and round-trip cleanly. (2) rusqlite links against the VFS abstraction correctly on both native and wasm. (3) Unit tests for schema, round-trip, and edge cases. (4) File manager UI (if any) offers .sqlite as a save format option.","notes":"Depends on improvise-6mq (VFS abstraction). Independent of the standalone deployment critical path — can be tackled any time after the Storage trait lands. Filed as backlog (P3).","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:39:41Z","created_by":"spot","updated_at":"2026-09-20T00:08:35Z","closed_at":"2026-09-20T00:08:35Z","close_reason":"DECLINED with measurements, not intuition. No code written; the tree was left untouched.\n\nALL THREE OF THE TICKET CLAIMS WERE TESTED:\n\n(1) \"Works over any VFS backend because the storage layer is abstracted\" -- FALSE. improvise-6mq did not deliver a VFS. Storage is four whole-document operations (read/write bytes, list, delete) with no offsets, seeking, handles or locking; invariant(storage_is_two_ops) states it as doctrine. An sqlite3_vfs needs xRead(offset,len)/xWrite/xTruncate/xLock and only the first two could even be faked -- by rewriting the whole database on every page access.\nBUT the agent PROVED SQLite can still fit before declining: rusqlite with features=[bundled,serialize] round-trips a database through bytes with zero filesystem contact (probe test passed; a one-row DB serializes to 8192 bytes). So the decline rests on VALUE, not on imagined plumbing trouble -- \"declining on a capability that actually exists would have been the wrong decline\".\n\n(2) \"A more scalable container than plaintext for very large models\" -- FALSE, MEASURED on an 18,000-cell model using the ticket own schema, VACUUMed:\n big.improv 1,280,846 bytes\n big.sqlite 1,568,768 +22%\n big.improv.gz 136,264\n big.sqlite.gz 642,897 4.7x LARGER\nFloor case: a 5-table schema holding 3 rows is 24,576 bytes against roughly 150 bytes of .improv -- about 160x. And because writes go through Storage::write(whole bytes), every save still rewrites the entire document, so the one dimension where SQLite normally wins -- incremental page writes -- is unavailable.\n\n(3) \"Enables structured queries over model state\" -- NOT DELIVERED by this design. The artifact is a closed blob; querying means deserializing the whole database into RAM beside the Model that is already fully in RAM. Anything you could ask it you can already ask DataStore::matching_values/matching_cells over interned, indexed data.\n\nALSO: cargo check --target wasm32-unknown-unknown fails in libsqlite3-sys C build. The agent was precise that this is the NIX CC-WRAPPER injecting -fzero-call-used-regs, not proof bundled SQLite can never target wasm -- but it is real evidence the repo wasm CI gate would break the day this landed.\n\nTHE COST THE TICKET DOES NOT PRICE: .improv is grammar-defined and the pest grammar is the single source of truth. A SQLite writer/reader is a SECOND IMPLEMENTATION OF THE WHOLE FORMAT -- groups, formula target categories, hidden items, collapsed groups, number formats, page/filter selections, IndexMap insertion order, the _Measure explicit-vs-formula-target split, virtual-view suppression -- which must track every future grammar change forever. And the ticket own acceptance test (model-\u003esqlite-\u003emodel identity) CANNOT DETECT DRIFT from .improv, because it never compares the two. That missing cross-format test is what made the ticket look cheap.\n\nFormat dispatch and naming were both checked and are NOT the problem: save_to/load_from are the only chokepoints, SQLite 16-byte header is a cleaner discriminator than the existing leading-brace JSON sniff, and .improv.sqlite would follow invariant(gzip_is_a_name_property).\n\nSALVAGE FILED SEPARATELY: the one claim with standing is INTEROP -- opening a model in sqlite3 or DB Browser. That is EXPORT, not persistence.","dependencies":[{"issue_id":"improvise-9ix","depends_on_id":"improvise-6mq","type":"blocks","created_at":"2026-04-14T00:40:34Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-avy","title":"Formula tokenizer: support quoted identifiers for ambiguous names","description":"The formula tokenizer currently uses heuristics to handle multi-word identifiers (greedy space consumption with keyword/operator break rules). This is fragile — we just fixed a bug where WHERE inside aggregates was consumed as part of an identifier.\n\nA more robust approach: support SQL/CL-style quoted identifiers. Two candidate syntaxes:\n- SQL style: double-quotes for identifiers, e.g. \"Total Revenue\" = \"Base Revenue\" + Bonus\n- CL style: pipe-delimited, e.g. |Total Revenue| = |Base Revenue| + Bonus\n\nSQL double-quoting is natural since the formula syntax already uses quotes for WHERE filter string literals. Disambiguation: in expression position, a quoted string becomes Token::Ident; in WHERE value position, it becomes Token::Str (parser already distinguishes these contexts).\n\nThis would let us simplify the tokenizer's space-handling heuristics and cleanly support category/item names that collide with keywords (WHERE, SUM, IF, etc.).","status":"closed","priority":3,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T06:45:27Z","created_by":"Edward Langley","updated_at":"2026-04-09T06:48:41Z","closed_at":"2026-04-09T06:48:41Z","close_reason":"Implemented pipe-quoted identifiers |...| in the formula tokenizer. Pipes produce Token::Ident, work in expressions, aggregates, and WHERE clauses. split_where and parse_where also updated. 6 new tests.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-0zf","title":"Bug: WHERE inside aggregate parens broken by greedy identifier tokenizer","description":"The tokenizer treats multi-word identifiers greedily — 'Revenue WHERE' becomes a single token. This means SUM(Revenue WHERE Region=East) doesn't parse correctly. The WHERE keyword is consumed as part of the identifier. Top-level WHERE (outside parens) works because split_where handles it before tokenization. Fix: either stop allowing spaces in identifiers inside parens, or detect WHERE as a keyword break point in the tokenizer.","status":"closed","priority":3,"issue_type":"bug","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T06:06:15Z","created_by":"Edward Langley","updated_at":"2026-04-09T06:38:08Z","closed_at":"2026-04-09T06:38:08Z","close_reason":"Fixed tokenizer to break multi-word identifiers at keywords (WHERE, SUM, AVG, MIN, MAX, COUNT, IF). Two-pronged: (1) break when current identifier IS a keyword, (2) break when next word IS a keyword. SUM(Revenue WHERE Region=East) now parses correctly.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -241,26 +246,26 @@ {"_type":"issue","id":"improvise-l36","title":"3.2 Publish to crates.io","description":"After cargo publish --dry-run is clean and user confirms, run cargo publish. Verify crate appears at crates.io/crates/improvise and cargo install improvise works.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:58Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-l36","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T21:09:29Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-l36","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-11a","title":"3.1 Configure cargo dist","description":"Run cargo dist init targeting x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin. Skip Windows and musl. Commit generated .github/workflows/release.yml and Cargo.toml additions. Test with v0.1.0-rc1 tag, verify builds, delete rc tag.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:55Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-11T07:49:57Z","close_reason":"Closed","dependencies":[{"issue_id":"improvise-11a","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-0s6","title":"Phase 3: Distribution","description":"Configure cargo dist, publish to crates.io, tag v0.1.0 release. Produces prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:51Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-0s6","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T23:37:39Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-3tj","type":"blocks","created_at":"2026-04-08T23:37:41Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-l36","type":"blocks","created_at":"2026-04-08T23:37:40Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:35:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:54:31Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:26:46Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:05:16Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-g6s1","title":"WizardStep::Preview has no characterization fixture","description":"The one import-wizard step improvise-764 could not pin. It needs a document that parses to ZERO RECORDS without needing path selection, and neither existing fixture reaches it.\n\nSmall, and worth doing while the WizardCache shape is fresh: every other step is pinned at buffer level, so Preview is the single arm where a change to wizard_cache could alter what is drawn with nothing failing.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:36Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:38Z","started_at":"2026-09-20T05:32:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-g6s1","title":"WizardStep::Preview has no characterization fixture","description":"The one import-wizard step improvise-764 could not pin. It needs a document that parses to ZERO RECORDS without needing path selection, and neither existing fixture reaches it.\n\nSmall, and worth doing while the WizardCache shape is fresh: every other step is pinned at buffer level, so Preview is the single arm where a change to wizard_cache could alter what is drawn with nothing failing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-k69y","title":"[bug] draw_status reads the document's active view, not the client's","description":"src/draw.rs::draw_status uses app.model_state.workbook.active_view; ViewState::active_view is the CLIENT copy (gotcha 29).\n\nHARMLESS TODAY -- they are in lockstep and the_two_active_view_names_never_diverge guards it. WRONG THE DAY improvise-jytm LANDS, which is exactly the ticket that makes them diverge on purpose (the document keeps the DEFAULT, the client is authoritative). On that day the TUI badge names the wrong view.\n\nOne line. Left unfixed because it is a behaviour change and belongs with jytm rather than ahead of it. Note the wasm client already reads the correct one (self.view.active_view), so the TUI is the outlier.","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-poia","title":"The records-mode + column has no projected identity","description":"From the ADR 002 spine landing (2026-09-19). In records mode the trailing + (NewMeasure) column creates a new measure on the row (gotcha 1). It carries no CellId, so creating a measure over the wire has no address.\n\nThat is correct for now rather than an oversight -- a new measure is not a write to an existing cell, so it is not a SetCell and should not borrow one identity. But it means a browser client cannot do something the TUI can.\n\nDecide what it should be: a distinct command carrying the record surrogate plus the new measure name (derived from the effect that does it today, per doctrine(derive_the_enum_from_the_effects)), or a client-side flow that adds the measure and then writes the cell as two commands. The second is probably right -- it composes from operations that already exist -- but check whether the TUI treats it as one atomic action first, because splitting an atomic action across two commands is how a half-applied state appears.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:18:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-mjy7","title":"Export a model as SQLite for interop (one-way, native-only)","description":"The salvageable part of improvise-9ix, which was declined as a PERSISTENCE format (see its close reason for the measurements).\n\nThe one claim with real standing is INTEROP: a user opening their model in sqlite3 or DB Browser to run their own queries. That is EXPORT, not persistence -- ONE-WAY, no round-trip obligation, and therefore none of the grammar-duplication risk that killed 9ix. A second FORMAT must track every future change to improv.pest forever; a second VIEW need not, because nothing loads it back.\n\nIt sits naturally beside export_csv/export_csv_to in crates/improvise-io/src/persistence/export.rs.\n\nTHE ONE REAL COST: the rusqlite dependency. cargo check --target wasm32-unknown-unknown fails in libsqlite3-sys C build under this repo nix cc-wrapper, so this must be NATIVE-ONLY and FEATURE-GATED, and the wasm CI job must keep passing untouched. Verify that rather than assume it -- the 9ix agent was careful to note the wasm failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating is about protecting the browser bundle, not about an impossibility.\n\nScope: choose a schema that is USEFUL TO QUERY rather than one that mirrors internal storage -- a reader wants cells joined to their coordinates, not an interned key blob. Do not add a reader.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:08:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:08:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-zavn","title":"Drill previews on hover: a bounded record keyset per aggregate","description":"DEFERRED by the user (2026-09-19) -- a good idea, recorded so the shape is not lost and so nobody builds the UNBOUNDED version by accident.\n\nTHE IDEA: show the underlying rows when the user hovers an aggregate cell, without a round trip.\n\nTHE TRAP THIS AVOIDS. Projecting the full keyset for every aggregate is untenable and the cost runs OPPOSITE to the value. A stored cell has fixed coordinates so it lands in exactly one displayed cell -- the keysets PARTITION rather than multiply -- but that still makes the payload scale with N (the data) while everything else in a projection scales with the VIEWPORT. Aggregation is precisely what makes a large document fit a screen, so shipping the keysets undoes that compression. Worked numbers for a 1000-cell viewport at u128: ~16KB at one record per cell (where the cell would be settable anyway and no keyset is needed), ~160KB at ten, and ~16MB over a million-record document -- to render a thousand cells. Per client, re-diffed as data changes. And the cheap cases are the ones where drilling barely matters.\n\nTHE SHAPE THAT WORKS: ship at most k record hashes per aggregate PLUS A COUNT, so the payload is viewport-proportional again, and fetch the full set on demand when the user actually drills. k is a product decision -- enough rows to make a hover useful, e.g. 3 to 5.\n\nPREREQUISITE: ADR 002 (context/adr/002-naming-a-cell.md). An aggregate already carries CellId::Aggregate(AggregateHash), so drill-on-demand works without this ticket; this is purely the latency optimisation for hover. Do not conflate the two -- drilling must stay correct without any preview.\n\nNote the preview hashes are RECORD hashes, so they resolve through the same DataStore index and obey the same staleness rules: a preview row whose record has since gone simply fails to resolve.","notes":"\n--- BLOOM FILTER IDEA (user, 2026-09-19): right tolerance, different question ---\nProposal was a Bloom filter instead of an explicit keyset, on the grounds that\nfalse positives are fine given a deterministic aggregate id to ask\nauthoritatively. The tolerance argument is right; the direction needs\ncorrecting.\n\nA BLOOM FILTER CANNOT ENUMERATE. A hover preview needs ROWS -- identities and\nvalues -- while a filter answers \"is X probably in this set?\" for an X you\nalready hold. The client holds aggregates, not the underlying records, so it has\nno candidates to test. So this does not serve previews directly.\n\nWHERE IT FITS INSTEAD, and it fits well: the REVERSE question, \"which visible\naggregates does record R feed?\" A client staging an edit to R tests R against\neach visible cell filter and marks the hits pending -- greyed, spinner -- with\nno round trip. A false positive marks one extra cell pending; it never misses\none. That asymmetry is exactly gotcha 35 rule: over-reporting costs work,\nunder-reporting is the bug, which is why Dirty::of is a deliberate\nover-approximation. Same shape.\n\nSIZE CORRECTION THAT MAKES IT CHEAP: a conventionally-sized filter still scales\nwith N -- roughly 9.6 bits per element at 1% FP, so ~1.2KB for a cell\naggregating 1000 records, and the payload again scales with the DATA. FIX THE\nWIDTH instead: a fixed 64- or 128-bit filter per cell is viewport-proportional\nby construction. Small aggregates are precise; large ones SATURATE and answer\n\"maybe\" to everything, degrading to mark-everything -- the safe direction, and\nthe same failure the design already tolerates.\n\nThe deterministic aggregate id is what makes the sloppiness safe: the\nauthoritative answer is one Drill(AggregateHash) away, so the filter is never\nload-bearing for correctness. Same structure as ADR 001 advisory reservations,\nwhere correctness never rests on the advisory layer.\n\nSO THIS TICKET NOW HOLDS TWO SEPARABLE IDEAS -- split them when picked up:\n (a) hover previews: needs a BOUNDED EXPLICIT keyset (k hashes + a count),\n because a preview needs rows;\n (b) optimistic edit feedback: a FIXED-WIDTH Bloom filter per aggregate,\n answering which visible cells a pending record edit affects.\n(b) belongs with ADR 001 layer 2 rather than here.","status":"deferred","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:59:53Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:30:50Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-mjy7","title":"Export a model as SQLite for interop (one-way, native-only)","description":"DEFERRED 2026-09-19 by user ruling during the whole-backlog campaign, together with improvise-qddo and improvise-rucf, and rewritten to say where it sits relative to ADR 003.\n\nTHE CLAIM IS STILL GOOD AND IS UNAFFECTED BY THE ADR. This is EXPORT for interop -- a user opening their model in sqlite3 or DB Browser to run their own queries. One-way, no round-trip obligation, and therefore none of the grammar-duplication risk that killed improvise-9ix: a second FORMAT must track every future change to improv.pest forever, a second VIEW need not, because nothing loads it back. It sits naturally beside export_csv/export_csv_to in crates/improvise-io/src/persistence/export.rs.\n\nWHY IT IS DEFERRED ANYWAY: it is the one bead in the SQLite cluster with no dependency on ADR 003, but it carries the same cost as the cluster's real work -- a rusqlite dependency and a wasm-gate verification -- for a capability nobody has asked for concretely. The user's ruling was to defer the cluster as a unit. Un-defer on a concrete interop request, or alongside improvise-rucf.2 (the SQLite source), where the dependency is being added anyway and the incremental cost of an exporter is small.\n\nNOTE THE DIRECTION IS OPPOSITE TO ADR 003'S. ADR 003 is about READING someone else's database as a live source. This is about WRITING ours out as a convenience. They share only the rusqlite dependency and the gating constraint; do not let one justify the other.\n\nTHE ONE REAL COST: cargo check --target wasm32-unknown-unknown fails in the libsqlite3-sys C build under this repo's nix cc-wrapper, so this must be NATIVE-ONLY AND FEATURE-GATED with the wasm CI job passing untouched. VERIFY that rather than assume it -- the improvise-9ix agent was careful to note the failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating is about protecting the browser bundle, not about an impossibility.\n\nSCOPE WHEN IT LANDS: choose a schema that is USEFUL TO QUERY rather than one that mirrors internal storage -- a reader wants cells joined to their coordinates, not an interned key blob. Do not add a reader; that is improvise-rucf's job and it has a different answer.\n","status":"deferred","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:08:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:48Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-zavn","title":"Drill previews on hover: a bounded record keyset per aggregate","description":"DEFERRED by the user (2026-09-19) -- a good idea, recorded so the shape is not lost and so nobody builds the UNBOUNDED version by accident.\n\nTHE IDEA: show the underlying rows when the user hovers an aggregate cell, without a round trip.\n\nTHE TRAP THIS AVOIDS. Projecting the full keyset for every aggregate is untenable and the cost runs OPPOSITE to the value. A stored cell has fixed coordinates so it lands in exactly one displayed cell -- the keysets PARTITION rather than multiply -- but that still makes the payload scale with N (the data) while everything else in a projection scales with the VIEWPORT. Aggregation is precisely what makes a large document fit a screen, so shipping the keysets undoes that compression. Worked numbers for a 1000-cell viewport at u128: ~16KB at one record per cell (where the cell would be settable anyway and no keyset is needed), ~160KB at ten, and ~16MB over a million-record document -- to render a thousand cells. Per client, re-diffed as data changes. And the cheap cases are the ones where drilling barely matters.\n\nTHE SHAPE THAT WORKS: ship at most k record hashes per aggregate PLUS A COUNT, so the payload is viewport-proportional again, and fetch the full set on demand when the user actually drills. k is a product decision -- enough rows to make a hover useful, e.g. 3 to 5.\n\nPREREQUISITE: ADR 002 (context/adr/002-naming-a-cell.md). An aggregate already carries CellId::Aggregate(AggregateHash), so drill-on-demand works without this ticket; this is purely the latency optimisation for hover. Do not conflate the two -- drilling must stay correct without any preview.\n\nNote the preview hashes are RECORD hashes, so they resolve through the same DataStore index and obey the same staleness rules: a preview row whose record has since gone simply fails to resolve.","notes":"\n--- BLOOM FILTER IDEA (user, 2026-09-19): right tolerance, different question ---\nProposal was a Bloom filter instead of an explicit keyset, on the grounds that\nfalse positives are fine given a deterministic aggregate id to ask\nauthoritatively. The tolerance argument is right; the direction needs\ncorrecting.\n\nA BLOOM FILTER CANNOT ENUMERATE. A hover preview needs ROWS -- identities and\nvalues -- while a filter answers \"is X probably in this set?\" for an X you\nalready hold. The client holds aggregates, not the underlying records, so it has\nno candidates to test. So this does not serve previews directly.\n\nWHERE IT FITS INSTEAD, and it fits well: the REVERSE question, \"which visible\naggregates does record R feed?\" A client staging an edit to R tests R against\neach visible cell filter and marks the hits pending -- greyed, spinner -- with\nno round trip. A false positive marks one extra cell pending; it never misses\none. That asymmetry is exactly gotcha 35 rule: over-reporting costs work,\nunder-reporting is the bug, which is why Dirty::of is a deliberate\nover-approximation. Same shape.\n\nSIZE CORRECTION THAT MAKES IT CHEAP: a conventionally-sized filter still scales\nwith N -- roughly 9.6 bits per element at 1% FP, so ~1.2KB for a cell\naggregating 1000 records, and the payload again scales with the DATA. FIX THE\nWIDTH instead: a fixed 64- or 128-bit filter per cell is viewport-proportional\nby construction. Small aggregates are precise; large ones SATURATE and answer\n\"maybe\" to everything, degrading to mark-everything -- the safe direction, and\nthe same failure the design already tolerates.\n\nThe deterministic aggregate id is what makes the sloppiness safe: the\nauthoritative answer is one Drill(AggregateHash) away, so the filter is never\nload-bearing for correctness. Same structure as ADR 001 advisory reservations,\nwhere correctness never rests on the advisory layer.\n\nSO THIS TICKET NOW HOLDS TWO SEPARABLE IDEAS -- split them when picked up:\n (a) hover previews: needs a BOUNDED EXPLICIT keyset (k hashes + a count),\n because a preview needs rows;\n (b) optimistic edit feedback: a FIXED-WIDTH Bloom filter per aggregate,\n answering which visible cells a pending record edit affects.\n(b) belongs with ADR 001 layer 2 rather than here.\nDEFERRAL RE-AFFIRMED 2026-09-19 during the whole-backlog campaign, status set to deferred so it stops appearing as ready work.\n\nThe reason it is recorded rather than closed is unchanged and is the important part: drilling is already correct without it (CellId::Aggregate resolves on demand), and the trap this bead exists to prevent is someone building the UNBOUNDED version -- projecting every aggregate's full keyset, whose payload scales with the data while every other projection scales with the viewport. The shape that works is at most k record hashes plus a count.","status":"deferred","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-19T23:59:53Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:03Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ut7p","title":"improvise-gp fit_from_gen0 is a third copy of the GA loop","description":"Noted during improvise-0ai (2026-09-19), deliberately not changed. The generation loop now lives in gp_tree::fit::fit. improvise-gp test helper fit_from_gen0 is still its own loop, kept because it also returns the GENERATION-0 best, which the library fit does not expose -- and that is exactly what breeding_improves_on_generation_zero needs in order to prove the search moved rather than merely differed. So the duplication is load-bearing today. Two honest options: expose the generation-0 champion from gp_tree::fit (a fit result carrying the starting cost, or fit_with_progress) and rewrite the helper onto it, or leave it and document why. Prefer the first -- did the search improve on where it started is a question any caller of a GA wants answerable, not only a test.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T21:23:05Z","created_by":"Edward Langley","updated_at":"2026-09-19T23:40:24Z","closed_at":"2026-09-19T23:40:24Z","close_reason":"Took the preferred option. gp_tree::fit gains Fit { tree, cost, start, start_cost } and fit_detailed; fit is now fit_detailed(..).tree. start is the best member of GENERATION 0 of the restart that produced tree -- the stronger claim, and the one improvise-gp needed. improvise-gp fit_from_gen0 is gone, replaced by seeded_fit over fit_detailed, so the GA loop has one home.\n\nLoop shape preserved deliberately: generation i is still what is scored at the top of iteration i, the sort is still stable so member 0 wins its own ties and generation 0 champion equals the seed ancestor whenever nothing beat it -- that is what makes the restructure behaviour-preserving rather than merely plausible. breed runs the same number of times with the same arguments; the extra scoring pass draws nothing from the RNG.\n\nREPRODUCIBILITY EVIDENCE RE-ESTABLISHED, not assumed: all six known-answer vectors pass unchanged and the nine-scenario fit_csv harness is byte-identical including model.json, so every recorded seed still names the same tree. Two faults injected, both fired.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-0vh1","title":"number_text emits grammar-rejected 'inf' and panics on NaN","description":"From the wave-1 gp-tree review (2026-09-19). Both are guarded by documented preconditions, hence P4.\n\nimprovise-gp/src/lib.rs:124. number_text(f32::INFINITY): at p=0 it yields \"inf\", and \"inf\".parse::\u003cf64\u003e() succeeds, so the round-trip loop ACCEPTS it and emits 'inf' -- which the formula grammar rejects (number has no such form). fitted_formula then returns Err, which is fine, but a bare formula_text caller gets silently unparseable text. number_text(f32::NAN) never matches (NaN != NaN) and falls through to unreachable!(), i.e. panics.\n\nAlso a doc imprecision worth correcting while there (lib.rs:22-25 and 114-123): the comment says it emits 'the exact decimal value of the f32'. It emits the shortest decimal that ROUND-TRIPS to the same f64 -- which is exactly what the routing argument needs, and is a weaker and more accurate claim. Verified: over 20,000 random f32 bit patterns plus the extremes, every value found a width inside 0..=149, worst observed width 60, none emitted e/E/inf/NaN, all parsed and round-tripped.","notes":"\n--- FIXED, AWAITING MERGE (2026-09-19) ---\nThe fix is on branch wave-1-review-fixes and is in PR #9\n(https://github.com/fiddlerwoaroof/improvise/pull/9), which is open pending\nCopilot review. Moved to in_progress rather than left open so nobody picks\nthis up again -- see the agent-issue-drift-pattern memory. Close it when #9\nmerges, not before.","status":"closed","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T20:43:13Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:50:38Z","started_at":"2026-09-19T21:15:05Z","closed_at":"2026-09-19T21:50:38Z","close_reason":"Merged to main in PR #9 as e1acac691d70992b5dfcce9f11ce2c2b50fa1c29 (2026-09-19). Verified a real merge commit with both parents, not a squash, so all 18 branch commits are reachable on main per AGENTS.md's preference for merges. Copilot's second review on that PR: approval recommended, findings none. All eight non-skipped CI checks green, including claude-review. Synced to origin and gitea.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-k099","title":"AppMode::hint_text/label put terminal UI strings on a serde wire enum","description":"From the wave-1 review (2026-09-19). Pre-existing, introduced by improvise-2hi, not by this wave -- filed so it is tracked rather than rediscovered.\n\ngotcha 26's principle is that wire tokens are not UI labels. But AppMode::hint_text and AppMode::label are English, terminal-specific, keybinding-naming UI strings hanging off an enum that is serialized into the client Snapshot. A non-terminal or localized client cannot use them and has to map AppMode itself anyway.\n\nThe established fix shape in this repo is the AxisColor/MinibufColor one: keep a semantic value in the portable crate and let the render layer resolve it (invariant(semantic_palette_at_the_core_boundary)). Decide whether hints are worth the same treatment or whether they should simply move out of the protocol crate.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:42:38Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:42:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-n3wt","title":"prune_axis_items: keep-slice length invariant is only a debug_assert","description":"From the wave-1 review (2026-09-19). NOT a defect today -- filed so the latent risk is recorded rather than rediscovered.\n\nimprovise-ai7 replaced the old guard 'data_idx \u003c rc \u0026\u0026 keep_row[data_idx]' with 'keep.get(data_idx).copied().unwrap_or(false)' plus a debug_assert_eq!(data_idx, keep.len()). The reviewer confirmed the lengths cannot diverge today: keep_row/keep_col are built as (0..rc)/(0..cc) collections, rc is row_count() which is itself data_items(\u0026row_items).count(), nothing mutates the axis vectors between building keep and consuming it, and the fn is private with exactly two call sites. The debug_assert can never fire, and cargo test --workspace --release (assert compiled out) is green.\n\nThe risk is what happens IF a third caller ever derives keep from something other than row_count()/col_count(), because debug_assert is compiled out in release and the two builds then differ in OPPOSITE directions from the old code:\n * keep too SHORT: old code panicked (loud, index out of bounds); new code silently drops the item in release -- a crash becomes silent data loss from the grid.\n * keep too LONG: old dropped every item at index \u003e= rc; new keeps them if flagged.\n\nFix by making the count unsupplied-able rather than asserted -- e.g. have prune_axis_items derive keep's length from items itself, or take a newtype that can only be built from the same vector. Failing that, a doc-comment on the fn naming the invariant and both failure directions.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:41:44Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:52:13Z","closed_at":"2026-09-19T20:52:13Z","close_reason":"Fixed structurally rather than by assertion: prune_axis_items now takes impl Fn(usize) -\u003e bool over the DATA index instead of a \u0026[bool] plus a debug_assert. The function no longer receives a length at all -- the index set is generated by walking items -- so there is no parallel quantity a caller can derive wrongly, and debug and release cannot diverge because both the assertion and the unwrap_or fallback are gone. Both failure directions the ticket named become unrepresentable rather than detected. prune_empty now consults the has_value grid directly and the two Vec\u003cbool\u003e allocations disappear; the eager-to-lazy change is safe because pruning row_items touches neither col_items nor has_value, so the column pass still reads the original grid with the original rc (reasoning left in a comment at the call site). Two tests added, including prune_axis_items_indexes_data_positions_not_visual_ones, which pins the contract the old debug_assert only gestured at -- a group header must not shift the numbering the predicate sees. Records mode untouched.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-th7m","title":"Dead F(1) binding: bound to bare 'enter-mode' then immediately rebound","description":"Noticed while working improvise-s13 (2026-09-19), not touched per AGENTS.md (mention dead code, do not delete unasked).\n\nIn crates/improvise-command/src/command/keymap.rs, F(1) is bound to a bare \"enter-mode\" with NO args -- which would fail require_args if it were ever dispatched -- and is then rebound on the very next line with [\"help\"]. The first line is dead: it can never take effect.\n\nTrivial deletion, but worth confirming the second binding is the intended one rather than deleting the wrong line.","status":"open","priority":4,"issue_type":"chore","owner":"el-github@elangley.org","created_at":"2026-09-19T20:36:45Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:36:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-2zq","title":"Records mode has no inline way to add a category, unlike the + column for measures","description":"improvise-c5v gave records mode a trailing '+' column: type a name, the measure is created, and it becomes a real column you can immediately type values into — without leaving the grid. There is no equivalent for categories, which is an asymmetry the wide-format work introduced.\n\nWHAT EXISTS TODAY (all leave the grid):\n- N in normal or records mode -\u003e search-or-category-add (command/cmd/search.rs:172). Emits SetPanelOpen{Category} + enter-mode category-add, so it opens the side panel and switches mode. Records-normal inherits it from the normal keymap (keymap.rs:645 builds rn with_parent(normal)).\n- n inside the category panel -\u003e the same CategoryAdd mode.\n- :add-cat Region (alias for add-category).\n\nNone of them set a coordinate on the row you were standing on, so after adding 'Region' you still have to navigate to that column and type the value.\n\nWHY IT LOOKS TRACTABLE:\n- GridLayout::records_columns filters on cat.kind.is_regular() and does NOT require the category to have items, so a freshly added empty category appears as a records column immediately.\n- CellKey::with(cat, item) ADDS a coordinate when absent, so the existing rename_record_coordinate path already handles a row that has no coordinate for the new category — it will move every measure cell in the row onto the new coordinate, which is the correct semantics.\n\nSo the shape could mirror '+': a second trailing column (say '+cat') whose commit runs add-category and then, optionally, sets that coordinate for the current row.\n\nOPEN QUESTIONS, decide before building:\n1. Two trailing columns ('+' for measure, '+cat' for category) may be more clutter than it is worth. An alternative is one '+' column that disambiguates by prefix, or a key rather than a column.\n2. Adding a category is more consequential than adding a measure: every existing record lacks a coordinate for it, so the column is blank everywhere and MAX_CATEGORIES (12) applies. Adding a measure only adds a blank column. Worth deciding whether inline creation is even desirable, or whether the panel is the right home for a schema change of that weight.\n3. Should creating the category also set its value on the current row in one commit, or just create the column?\n\nFiled after the user asked whether such a bead existed; it did not.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-29T19:32:40Z","created_by":"Edward Langley","updated_at":"2026-08-29T19:33:23Z","dependencies":[{"issue_id":"improvise-2zq","depends_on_id":"improvise-c5v","type":"blocks","created_at":"2026-08-29T12:32:45Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-th7m","title":"Dead F(1) binding: bound to bare 'enter-mode' then immediately rebound","description":"Noticed while working improvise-s13 (2026-09-19), not touched per AGENTS.md (mention dead code, do not delete unasked).\n\nIn crates/improvise-command/src/command/keymap.rs, F(1) is bound to a bare \"enter-mode\" with NO args -- which would fail require_args if it were ever dispatched -- and is then rebound on the very next line with [\"help\"]. The first line is dead: it can never take effect.\n\nTrivial deletion, but worth confirming the second binding is the intended one rather than deleting the wrong line.","status":"in_progress","priority":4,"issue_type":"chore","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-19T20:36:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:32Z","started_at":"2026-09-20T05:32:32Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-2zq","title":"Records mode has no inline way to add a category, unlike the + column for measures","description":"improvise-c5v gave records mode a trailing '+' column: type a name, the measure is created, and it becomes a real column you can immediately type values into — without leaving the grid. There is no equivalent for categories, which is an asymmetry the wide-format work introduced.\n\nWHAT EXISTS TODAY (all leave the grid):\n- N in normal or records mode -\u003e search-or-category-add (command/cmd/search.rs:172). Emits SetPanelOpen{Category} + enter-mode category-add, so it opens the side panel and switches mode. Records-normal inherits it from the normal keymap (keymap.rs:645 builds rn with_parent(normal)).\n- n inside the category panel -\u003e the same CategoryAdd mode.\n- :add-cat Region (alias for add-category).\n\nNone of them set a coordinate on the row you were standing on, so after adding 'Region' you still have to navigate to that column and type the value.\n\nWHY IT LOOKS TRACTABLE:\n- GridLayout::records_columns filters on cat.kind.is_regular() and does NOT require the category to have items, so a freshly added empty category appears as a records column immediately.\n- CellKey::with(cat, item) ADDS a coordinate when absent, so the existing rename_record_coordinate path already handles a row that has no coordinate for the new category — it will move every measure cell in the row onto the new coordinate, which is the correct semantics.\n\nSo the shape could mirror '+': a second trailing column (say '+cat') whose commit runs add-category and then, optionally, sets that coordinate for the current row.\n\nOPEN QUESTIONS, decide before building:\n1. Two trailing columns ('+' for measure, '+cat' for category) may be more clutter than it is worth. An alternative is one '+' column that disambiguates by prefix, or a key rather than a column.\n2. Adding a category is more consequential than adding a measure: every existing record lacks a coordinate for it, so the column is blank everywhere and MAX_CATEGORIES (12) applies. Adding a measure only adds a blank column. Worth deciding whether inline creation is even desirable, or whether the panel is the right home for a schema change of that weight.\n3. Should creating the category also set its value on the current row in one commit, or just create the column?\n\nFiled after the user asked whether such a bead existed; it did not.","status":"in_progress","priority":4,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-08-29T19:32:40Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:33Z","started_at":"2026-09-20T05:32:33Z","dependencies":[{"issue_id":"improvise-2zq","depends_on_id":"improvise-c5v","type":"blocks","created_at":"2026-08-29T12:32:45Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ede","title":"prune_empty's records branch was unreachable dead code","description":"Noticed while doing improvise-c5v. GridLayout::prune_empty starts with 'if self.is_records_mode() { return; }', which guarantees self.records is None for the rest of the function — but the tail carried an 'if let Some(records) = \u0026self.records' branch that re-indexed row_items and rebuilt the records vec. It could never run.\n\nI removed it rather than leaving it, because the RecordRow type change made it stop compiling, so it became my mess to clean rather than pre-existing dead code to leave alone (AGENTS.md).\n\nFiling this only as a record of the removal, in case the intent was that records mode SHOULD prune empty rows and the early return is the actual bug. If so, that is a separate behavior question: today records mode deliberately shows every row ('the user drilled in to see all the raw data', per the doc comment on prune_empty). Close this if the early return is intended.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-29T19:29:02Z","created_by":"Edward Langley","updated_at":"2026-09-19T19:48:04Z","closed_at":"2026-09-19T19:48:04Z","close_reason":"Answered by the code the ticket points at. prune_empty carries the doc comment \"In records mode every column is shown (the user drilled in to see all the raw data)\", and repo-map gotcha 1 says the same: a records row is one coordinate tuple and every raw record is meant to be visible. So the early return is intended and the removed branch was correctly dead. Groomed 2026-09-19 per the ticket own instruction (\"Close this if the early return is intended\"). Reopen if P in records mode doing nothing is itself worth a UX ticket.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-ct5","title":"i on an empty records grid reports 'cannot drill', leaking a pivot-mode concept","description":"In records mode every regular category sits on Axis::None (ToggleRecordsMode, command/cmd/grid.rs:432), so EditOrDrill's regular_none is always true and the edit branch depends entirely on is_synthetic. On an empty records grid there is no synthetic record at the cursor, so i takes the drill branch, finds no cell key, and sets 'cannot drill — no cell at cursor'.\n\nThe outcome (nothing to edit) is defensible, but the message talks about drilling — a pivot-mode concept the user is not in. In records mode the user's model is 'a table I type into', and o one key away does add a row and open an edit.\n\nOptions: (a) reword the message for records mode; (b) make i on an empty records grid behave like o. (b) is more helpful but makes i mode-dependent, which cuts against edit-or-drill being mode-agnostic — decide deliberately.\n\nPinned by ui/app.rs::tests::i_and_o_diverge_on_an_empty_records_grid, which exists to record that i and o diverge here (see improvise-6sq). Changing this behavior means updating that test; read its doc-comment first — the divergence itself is load-bearing.","notes":"STALE REF (groomed 2026-08-30, line numbers shifted by this session's work): command/cmd/grid.rs:432 -\u003e the axis loop moved to crates/improvise-command/src/command/cmd/grid.rs:443 (the '!CategoryKind::of_name(name).is_index_or_dim()' loop in ToggleRecordsMode).","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-08-29T17:52:53Z","created_by":"Edward Langley","updated_at":"2026-08-30T17:06:17Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-ct5","title":"i on an empty records grid reports 'cannot drill', leaking a pivot-mode concept","description":"In records mode every regular category sits on Axis::None (ToggleRecordsMode, command/cmd/grid.rs:432), so EditOrDrill's regular_none is always true and the edit branch depends entirely on is_synthetic. On an empty records grid there is no synthetic record at the cursor, so i takes the drill branch, finds no cell key, and sets 'cannot drill — no cell at cursor'.\n\nThe outcome (nothing to edit) is defensible, but the message talks about drilling — a pivot-mode concept the user is not in. In records mode the user's model is 'a table I type into', and o one key away does add a row and open an edit.\n\nOptions: (a) reword the message for records mode; (b) make i on an empty records grid behave like o. (b) is more helpful but makes i mode-dependent, which cuts against edit-or-drill being mode-agnostic — decide deliberately.\n\nPinned by ui/app.rs::tests::i_and_o_diverge_on_an_empty_records_grid, which exists to record that i and o diverge here (see improvise-6sq). Changing this behavior means updating that test; read its doc-comment first — the divergence itself is load-bearing.","notes":"STALE REF (groomed 2026-08-30, line numbers shifted by this session's work): command/cmd/grid.rs:432 -\u003e the axis loop moved to crates/improvise-command/src/command/cmd/grid.rs:443 (the '!CategoryKind::of_name(name).is_index_or_dim()' loop in ToggleRecordsMode).","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-08-29T17:52:53Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:33Z","started_at":"2026-09-20T05:32:33Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ont","title":"Migrate existing callers of workbook.views.get() to Workbook::active_view","description":"Workbook already exposes active_view() and active_view_mut() (crates/improvise-core/crates/improvise-core/src/workbook.rs:87-97) but many callers still do workbook.views.get(\u0026workbook.active_view).unwrap().X or wb.views.get('view_name').unwrap().axis_of(...).\\n\\n11 lookup sites identified:\\n- crates/improvise-core/crates/improvise-core/src/workbook.rs:167-168, 176-177 (within workbook.rs itself — these may be the implementations and be fine)\\n- crates/improvise-core/crates/improvise-core/src/model/types.rs:1868, 1871, 1873, 1886, 1890 (tests)\\n- crates/improvise-io/crates/improvise-io/src/persistence/mod.rs:943, 946, 1165\\n\\nFix: audit each site; migrate to Workbook::active_view() / Workbook::view(name) / add Workbook::view_axis(view_name, cat_name) -\u003e Option\u003cAxis\u003e to absorb the common chain.\\n\\nMostly a cleanup task but reduces future refactor exposure. ~15 LOC.","notes":"Groomed 2026-08-13 — re-verified against the tree; the premise has largely evaporated.\n\nCurrent state: 14 `views.get(` call sites remain, and 13 of them are inside #[cfg(test)] modules (workbook.rs, model/types.rs, persistence/mod.rs). Test code asserting on a NAMED view is not the chain this ticket wants to remove — `active_view()` is the wrong tool there, because the whole point of those assertions is to look at a view that is not active.\n\nThe single production site is src/ui/view_panel.rs:31:\n let Some(view) = self.workbook.views.get(view_name) else { ... }\nThat takes a dynamic view name from the panel list, so active_view() does not apply to it either. At most it wants a `Workbook::view(name) -\u003e Option\u003c\u0026View\u003e` accessor, which is a 3-line change with no correctness content.\n\nThe ticket's original count of '11 lookup sites' predates the improvise-tvt item-facade work, which removed most of the interesting ones. What is left is not worth a P3 slot. Dropped to P4.\n\nRECOMMEND: either close as wont-do, or rescope to exactly 'add Workbook::view(name) and use it in view_panel.rs' — which is small enough to fold into whatever next touches that file.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T19:02:37Z","created_by":"Ed L","updated_at":"2026-08-30T17:07:07Z","closed_at":"2026-08-30T17:07:07Z","close_reason":"OBSOLETE — the pattern this ticket targets no longer exists (verified 2026-08-30 while grooming).\n\nThe description says 'many callers still do workbook.views.get(\u0026workbook.active_view).unwrap().X'. A grep for 'views.get(\u0026' across crates/ and src/ now returns ZERO matches; the migration to Workbook::active_view() has already happened, presumably in the crate-split work.\n\nOf the 14 remaining '.views.get(' call sites, 13 are in test modules (workbook.rs, persistence/mod.rs, model/types.rs — all asserting on a specific named view, which is the correct API for that) and exactly one is production code:\n\n src/ui/view_panel.rs:31 self.workbook.views.get(view_name)\n\nThat one looks up a view BY NAME from the panel's own list, so active_view() does not apply to it. There is nothing left to migrate.\n\nNot carried forward: the ticket also floated adding Workbook::view_axis(view_name, cat_name) to absorb the .unwrap().axis_of(...) chain. That chain now appears only in tests, so the absorption would buy little; file it fresh if it ever earns its keep.\n\nAlso noted while here: the description's paths were malformed — 'crates/improvise-core/crates/improvise-core/src/workbook.rs' — with the crate prefix doubled throughout, which is why they resolved to nothing.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ete","title":"Add Model/Category getter methods to absorb .unwrap().items/groups chains","description":"Several callers reach through Model and Category to get item names, groups, or counts — each chain includes an .unwrap() that panics if the category is missing:\\n\\n- persistence/mod.rs:1635-1639, 2006-2007: model.category(name).unwrap().items.values().map(|i| i.name.clone()).collect() — 4 sites\\n- persistence/mod.rs:831: \u0026m2.model.category('Month').unwrap().groups — 1 site\\n- crates/improvise-command/src/ui/cat_tree.rs:32: cat.map(|c| c.items.len()) — item count\\n\\nAdd:\\n- Model::item_names(cat: \u0026str) -\u003e Option\u003cVec\u003cString\u003e\u003e\\n- Model::group_names(cat: \u0026str) -\u003e Option\u003cVec\u003cString\u003e\u003e\\n- Category::item_count() -\u003e usize\\n\\nCorrectness win: (a) prevents panics if the category lookup fails; (b) hides the items/groups IndexMap/HashMap from callers — internal-rep changes don't cascade; (c) the Option return type forces explicit handling.\\n\\n~15 LOC plus cleaner call sites.","notes":"GROOMED 2026-09-19 — DISSOLVED PREMISE, mostly. The four persistence/mod.rs sites this ticket was built on (model.category(name).unwrap().items.values()...) no longer exist in production code; grep for unwrap().items / unwrap().groups now hits only test modules (persistence/mod.rs is #[cfg(test)] from line 706 on, workbook.rs likewise). The stated correctness win — \"prevents panics if the category lookup fails\" — is therefore void: there is no production panic left to prevent. What survives is encapsulation in test code plus cat_tree.rs:32 (cat.map(|c| c.items.len()).unwrap_or(0)), which already handles the missing category. P3 -\u003e P4 on that basis; close it if the encapsulation alone is not worth 15 LOC.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T19:02:34Z","created_by":"Ed L","updated_at":"2026-09-19T20:52:14Z","closed_at":"2026-09-19T20:52:14Z","close_reason":"OBSOLETE -- premise dissolved, verified item by item rather than accepted from the groom's note. (1) Model::item_names would DUPLICATE the existing Model::effective_item_names, and in a worse form: the ticket's version reading category(name).items would silently omit formula targets for _Measure, which is exactly the trap gotcha 7 warns about; the existing one routes _Measure through measure_item_names and returns Vec rather than Option\u003cVec\u003e, which is also the right call under 'keep Option at the boundaries'. (2) Model::group_names has no caller inside improvise-core -- the site the ticket cites now lives in an improvise-io test. (3) Category::item_count has no in-scope caller either; the live one is improvise-command's cat_tree.rs, which already handles the missing category.\n\nThe ticket's stated correctness win -- 'prevents panics if the category lookup fails' -- is void: every unwrap().items / unwrap().groups chain in the repo is now inside a #[cfg(test)] module. The single production reader of cat.items.values() is persistence/format.rs, the serializer legitimately walking items to write them, which is not an unwrap chain and not something a getter should hide from a formatter.\n\nIf the encapsulation is still wanted for improvise-command's cat_tree.rs, that is one method for whoever owns that crate, not the three proposed here.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-a5q","title":"Parquet columnar export (read-only)","description":"One-way export to Apache Parquet for interop with data tools (pandas, Arrow, BI tools). Columnar layout maps naturally to improvise's category/measure/cell structure: each category becomes a column of the key space, each measure a value column. Read-back not required for MVP; this is primarily a data-out path. Orthogonal to the core standalone deployment; filed as a backlog follow-on.","design":"Use arrow-rs + parquet crates (or polars as a simpler wrapper). Model shape → Arrow RecordBatch: one row per cell, columns are (category_1, category_2, ..., measure_name, value). Write the batch to a parquet file via the VFS. Read-back (parquet → model) is a stretch goal — most users will export once and consume the file elsewhere. Browser build: parquet + arrow crates are heavy; may push the wasm bundle size significantly. Consider making parquet export a separate optional wasm chunk loaded on demand.","acceptance_criteria":"(1) Export function writes a valid parquet file from the current model. (2) File opens correctly in pandas / pyarrow and reproduces the improv data. (3) Works over VFS (native filesystem + OPFS).","notes":"Depends on improvise-6mq (VFS abstraction). Very low priority — a nice data-science handoff story but not required for core functionality. Filed as P4 backlog.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:39:46Z","created_by":"spot","updated_at":"2026-07-30T23:06:32Z","dependencies":[{"issue_id":"improvise-a5q","depends_on_id":"improvise-6mq","type":"blocks","created_at":"2026-04-14T00:40:35Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-a5q","title":"Parquet columnar export (read-only)","description":"One-way export to Apache Parquet for interop with data tools (pandas, Arrow, BI tools). Columnar layout maps naturally to improvise's category/measure/cell structure: each category becomes a column of the key space, each measure a value column. Read-back not required for MVP; this is primarily a data-out path. Orthogonal to the core standalone deployment; filed as a backlog follow-on.","design":"Use arrow-rs + parquet crates (or polars as a simpler wrapper). Model shape → Arrow RecordBatch: one row per cell, columns are (category_1, category_2, ..., measure_name, value). Write the batch to a parquet file via the VFS. Read-back (parquet → model) is a stretch goal — most users will export once and consume the file elsewhere. Browser build: parquet + arrow crates are heavy; may push the wasm bundle size significantly. Consider making parquet export a separate optional wasm chunk loaded on demand.","acceptance_criteria":"(1) Export function writes a valid parquet file from the current model. (2) File opens correctly in pandas / pyarrow and reproduces the improv data. (3) Works over VFS (native filesystem + OPFS).","notes":"Depends on improvise-6mq (VFS abstraction). Very low priority — a nice data-science handoff story but not required for core functionality. Filed as P4 backlog.\nDEFERRED 2026-09-19 by user ruling during the whole-backlog campaign. Orthogonal data-out path with no consumer asking for it; it sits behind the ADR 003 data-source direction (improvise-rucf), which is itself filed not-a-priority. Revisit if an interop request is concrete.","status":"deferred","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:39:46Z","created_by":"spot","updated_at":"2026-09-20T05:31:03Z","dependencies":[{"issue_id":"improvise-a5q","depends_on_id":"improvise-6mq","type":"blocks","created_at":"2026-04-14T00:40:35Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-3gy","title":"4.2 Enable GitHub Pages","description":"Enable Pages in GitHub repo settings: source=main branch, folder=/docs. Verify site is live at the GitHub Pages URL.","notes":"CORRECTION (2026-09-20): this ticket's text says source=main branch, folder=/docs. That is the WRONG SETTING for the pipeline that landed in improvise-d31.\n\nThe correct setting is Settings -\u003e Pages -\u003e Source: GITHUB ACTIONS. The workflow uploads a Pages artifact from pages-build and deploys it with deploy-pages@v4; a branch/folder source would serve the repo instead and silently ignore the workflow.\n\nThe workflow attempts enablement itself (configure-pages@v5, enablement: true), so this may need no human at all -- verified as NOT currently enabled (the Pages API 404s). If the run fails at configure-pages, note that default_workflow_permissions on this repo is 'read'; the job declares permissions: {pages: write, id-token: write} explicitly, which should grant above the default, but it is the first thing to check on a 403.","status":"closed","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:08:10Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:20:51Z","closed_at":"2026-09-20T04:20:51Z","close_reason":"Done. GitHub Pages is enabled and serving: https://fiddlerwoaroof.github.io/improvise/ renders the demo grid from the in-page worker, verified by the user in Safari 27 on 2026-09-20.\n\nNote the ticket's own text was WRONG about the setting and was corrected before this closed: it said source = main branch, folder = /docs. The pipeline improvise-d31 built uploads a Pages ARTIFACT and deploys it with deploy-pages@v4, so the correct source is GITHUB ACTIONS. configure-pages@v5 with enablement: true turned it on without a human touching the setting, which answers the open question about whether the workflow token may.","dependencies":[{"issue_id":"improvise-3gy","depends_on_id":"improvise-e61","type":"blocks","created_at":"2026-04-08T21:09:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-e61","title":"4.1 Create docs/index.html landing page","description":"Vanilla HTML, single file, under 200 lines. Dark background, monospace headings. Embed asciinema-player from jsdelivr CDN. Sections: title/tagline, pivot cast, drill cast, formulas cast, import cast, install commands + GitHub link. Player config: rows 30, cols 100, theme monokai, autoPlay false, loop true.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:08:07Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-e61","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T21:09:31Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-kh8","title":"Phase 4: Landing page (optional)","description":"Create docs/index.html with embedded asciinema casts and enable GitHub Pages. Optional but recommended for launch.","status":"open","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:53Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-kh8","depends_on_id":"improvise-3gy","type":"blocks","created_at":"2026-04-08T23:37:42Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-kh8","depends_on_id":"improvise-e61","type":"blocks","created_at":"2026-04-08T23:37:41Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} From 3de90074a96d5f887de295e02f7d16d30e35c49b Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:57:20 -0700 Subject: [PATCH 45/72] fix(fit): four Copilot findings, including a reachable panic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All six findings on PR #25 are correct. Four are fixed here; two are declined with reasons and documented on `fit_measure` rather than left silent. **A legal document could panic the command.** `preprocess` drops an entirely missing column, so a document with no regular category and one empty measure encodes to *zero* features — and `random_tree` then draws `rng.below(0)`, which `SplitMix64` answers with `0`, and `sample_thresh` indexes `spec.ranges[0]`. Refused now, and the refusal carries the warnings naming what was dropped. Proved by disabling the guard and watching `index out of bounds: the len is 0 but the index is 0` at `gp-tree/src/lib.rs:218`. **Nothing bounded the feature *width*.** `FitConfig` bounds the search; a regular category with one distinct value per record one-hots to one column per record, and permutation importance is `repeats × rows × features` evaluations — quadratic in records, on the thread that draws the next frame. The 6/12/105 ms benchmark this PR quoted does *not* cover that shape, because its fixture marks the row-identity category as a `Label`. `MAX_FEATURES = 64` refuses, and names `Label` as the fix. The test had to grow past 64 records to be honest: at twelve the document is refused anyway, by `fitted_formula` noticing `Tag=r0` — a different rule, which would have made the test pass with no guard at all. **`FitOutcome` did not meet its own reproducibility contract.** It carried `seed` + `config`, which is the pair gotcha 37b names — and that pair is sufficient for `gp_tree::data::Model`, whose other knobs are not in the loop. Here they are: cross-validation draws from the *same* generator `fit` then draws from, so `cv_folds` moves the champion and not merely the evidence. It carries the whole `FitRequest` now, with a test that changing only `cv_folds` changes the rule. **The status line advertised a seed the parser rejected.** `status_line` prints `{:#x}` and tells the user that reproduces the fit; `parse` took decimal only, so the one instruction it gives failed on the value it had just shown. The test round-trips through `status_line` rather than a hand-written `"0x…"`, because a literal would still pass if the printing changed — which is the half that breaks. **Declined, both documented on `fit_measure` and to be filed.** A mean-imputed feature is read *raw* by the stored formula, so a record missing that value evaluates to something other than what the fit predicted for it; closing that means choosing between refusing any feature with a missing cell — which would refuse most real documents, imputation being why `preprocess` imputes — and an `IFMISSING`-shaped primitive the formula language does not have. That is a product decision of the same kind as improvise-4zl7. And a boundary value can route differently in the formula than in the fit, because training narrows `f64` to `f32` while the formula compares the original: pre-existing in the emitter (improvise-izk), and the fix would rewrite the bit-level characterization tests that are the only evidence the routing argument holds. 1220 passing (+4); clippy, fmt and the wasm gate clean; the CLI's output still byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/cmd/fit.rs | 57 ++++- crates/improvise-gp/src/fit.rs | 213 ++++++++++++++++-- 2 files changed, 246 insertions(+), 24 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/fit.rs b/crates/improvise-command/src/command/cmd/fit.rs index cd58b10f..973c537a 100644 --- a/crates/improvise-command/src/command/cmd/fit.rs +++ b/crates/improvise-command/src/command/cmd/fit.rs @@ -63,6 +63,19 @@ use improvise_gp::{FitRequest, fit_measure, fitted_measure_name}; use crate::command::cmd::core::{Cmd, CmdContext}; use crate::ui::effect::{self, Effect}; +/// A seed as the status line spells it (`0x…`, either case) or in decimal. +/// +/// The hex arm is not a convenience: `FitOutcome::status_line` prints `{:#x}` +/// and tells the user that is what reproduces the fit, so accepting only +/// decimal made the advertised round trip fail on the advertised value. +fn parse_seed(text: &str) -> Result { + let parsed = match text.strip_prefix("0x").or_else(|| text.strip_prefix("0X")) { + Some(hex) => u64::from_str_radix(hex, 16), + None => text.parse::(), + }; + parsed.map_err(|e| format!("fit-measure: {text:?} is not a seed: {e}")) +} + /// Fit `target` from the other measures and categories the view shows, storing /// the rule as the measure `into`. #[derive(Debug)] @@ -88,7 +101,11 @@ impl FitMeasure { /// The seed is an argument rather than a clock read for the reason the CLI /// fixes its own default: two fits of the same document should agree, and a /// user who wants a different search asks for one. It is reported back on - /// the status line so the fit that landed can be asked for again. + /// the status line so the fit that landed can be asked for again — **which + /// is why it is read in the form it is printed**, hexadecimal with a `0x` + /// prefix, as well as decimal. `status_line` spells it `{:#x}`, so a + /// decimal-only parser made the one instruction the status line gives a + /// user fail on the value the status line just showed them. pub fn parse(args: &[String]) -> Result, String> { let [target, rest @ ..] = args else { return Err("fit-measure requires a target measure".into()); @@ -96,13 +113,7 @@ impl FitMeasure { let (into, seed) = match rest { [] => (fitted_measure_name(target), None), [into] => (into.clone(), None), - [into, seed] => ( - into.clone(), - Some( - seed.parse::() - .map_err(|e| format!("fit-measure: {seed:?} is not a seed: {e}"))?, - ), - ), + [into, seed] => (into.clone(), Some(parse_seed(seed)?)), _ => { return Err(format!( "fit-measure takes 1-3 arguments, got {}", @@ -329,4 +340,34 @@ mod tests { .expect_err("\"later\" is not a seed"); assert!(err.contains("is not a seed"), "{err}"); } + + /// Copilot on PR #25. The status line advertises the seed as `{:#x}` and + /// tells the user that is what reproduces the fit, so the one instruction + /// it gives has to work on the value it just showed them — a decimal-only + /// parser made copying it back an error. + /// + /// Asserted as a *round trip through the status line*, not against a + /// hand-written `"0x…"`: a literal would still pass if `status_line` + /// changed how it spells the number, which is the half that breaks. + #[test] + fn the_seed_the_status_line_prints_is_a_seed_this_command_accepts() { + let app = run(&["Score"]); + let printed = app + .view_state + .status_msg + .split_whitespace() + .last() + .expect("the status line ends with the seed") + .to_string(); + assert!(printed.starts_with("0x"), "{printed:?}"); + assert_eq!( + parse_seed(&printed), + Ok(FitRequest::default().seed), + "the status line's own seed did not parse back" + ); + // …and the ordinary decimal form still works, in both directions. + assert_eq!(parse_seed("99"), Ok(99)); + assert_eq!(parse_seed("0X1f"), Ok(31)); + assert!(parse_seed("0xnope").is_err()); + } } diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index 10954bc9..6020311e 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -30,10 +30,13 @@ //! whose every cell is out by an affine map — wrong in the quiet way, since //! small numbers in the right shape look like an answer. //! -//! **The seed alone does not reproduce a fit** (gotcha 37b), so -//! [`FitOutcome`] carries the [`FitConfig`] beside it. Every stochastic step -//! draws from one generator in a fixed order — cross-validate, then fit, then -//! explain — so the pair really is sufficient. +//! **The seed alone does not reproduce a fit** (gotcha 37b) — and neither does +//! the seed plus the [`FitConfig`], which is the pair that rule names. Every +//! stochastic step draws from **one** generator in a fixed order — +//! cross-validate, then fit, then explain — so `cv_folds` decides how far that +//! generator has advanced before `fit` draws from it, and therefore changes the +//! *champion* and not merely the evidence for it. [`FitOutcome`] carries the +//! whole [`FitRequest`] rather than two of its four fields. //! //! **Cross-validating nothing is not the same as explaining nothing.** //! [`gp_tree::fit::cv_regress`] pools no pairs below two folds, and scoring an @@ -54,6 +57,17 @@ use improvise_formula::Formula; use crate::dataset::dataset; +/// The widest encoded feature matrix `fit_measure` will search synchronously. +/// +/// A budget, not a modelling opinion: `FitConfig` bounds the *search*, and +/// nothing bounded the *width*, which `preprocess` can multiply without limit +/// (one column per distinct value of a categorical column). Permutation +/// importance alone is `repeats × rows × features` tree evaluations. 64 is +/// comfortably above any hand-built pivot table — this repo's own 18,000-cell +/// benchmark encodes far fewer — and comfortably below the one-column-per-row +/// shape that is really a mis-typed category. +pub const MAX_FEATURES: usize = 64; + /// What to ask of a fit. Everything a run's outcome depends on, so that /// `(FitRequest, document)` reproduces it. #[derive(Clone, Copy, Debug, PartialEq)] @@ -130,10 +144,24 @@ pub struct FitOutcome { /// Records and columns the selection dropped — the feature table's notices /// first, then `preprocess`'s, as [`crate::dataset::dataset`] splices them. pub warnings: Vec, - /// The seed this fit ran under, and… - pub seed: u64, - /// …the knobs it ran under. Both, because neither alone reproduces it. - pub config: FitConfig, + /// **The whole request**, because reproducing a fit needs all of it. + /// + /// This started as `seed` plus `config`, which is the pair gotcha 37b + /// names — and that pair is what `gp_tree::data::Model` records, for a CLI + /// whose other knobs are not in the loop. Here they are: `cv_folds` + /// decides how far the *shared* generator advances before `fit` draws from + /// it, so it changes the champion and not merely the evidence, and + /// `importance_repeats` changes the ranking. Keeping two of four fields + /// documented the contract without meeting it. + pub request: FitRequest, +} + +impl FitOutcome { + /// The seed this fit ran under. Shorthand for `self.request.seed`, kept + /// because it is what a user is shown and what they type back. + pub fn seed(&self) -> u64 { + self.request.seed + } } impl FitOutcome { @@ -170,7 +198,8 @@ impl FitOutcome { }; format!( "{} = fitted tree: {quality}; {leans}; seed {:#x}{tail}", - self.formula.target, self.seed + self.formula.target, + self.seed() ) } } @@ -193,8 +222,35 @@ pub fn fitted_measure_name(target: &str) -> String { /// Fails, rather than storing something misleading, when: the selection cannot /// produce a table at all (see [`crate::feature_table`]); `into` is a measure /// that already holds data; the target is categorical, which a single -/// regression tree does not express; or the champion splits on a feature the -/// document cannot resolve (improvise-8xhy, and the message names them). +/// regression tree does not express; the encoded matrix is empty or wider than +/// [`MAX_FEATURES`]; or the champion splits on a feature the document cannot +/// resolve (improvise-8xhy, and the message names them). +/// +/// # Two known gaps, both filed rather than fixed here +/// +/// **A mean-imputed feature is read raw by the formula.** `preprocess` fills a +/// missing numeric cell with the column mean, so a champion may split on a +/// value the training rows had and the *document* does not. The stored formula +/// reads the document, so that record evaluates to something other than what +/// the fit predicted for it — quietly, since the measure name resolves and only +/// the cell is absent. `fitted_formula`'s check asks whether the document can +/// *name* the feature, which is a different question from whether every record +/// has one. Closing it means choosing between refusing any feature with a +/// missing cell — which would refuse most real documents, imputation being why +/// `preprocess` imputes — and teaching the formula language an +/// `IFMISSING`-shaped primitive. That is a product decision of the same kind as +/// improvise-4zl7, not a line to change. +/// +/// **A boundary value can route differently in the formula than in the fit.** +/// Training narrows each `f64` cell to `f32`; the stored formula compares the +/// original `f64` against the threshold. `number_text` guarantees the threshold +/// reads back as the same `f64` the `f32` widens to, so the *threshold* is +/// exact — but a cell value just below a threshold that `f32` rounds *to* it +/// takes `hi` in `gp_tree::eval` and `lo` in formula evaluation. It is +/// pre-existing in the emitter (improvise-izk) rather than new here, and the +/// fix — one numeric representation end to end — would rewrite the bit-level +/// characterization tests that are the only evidence the routing argument holds +/// at all. pub fn fit_measure( model: &Model, view: &View, @@ -215,6 +271,40 @@ pub fn fit_measure( let data = dataset(model, view, target, &Overrides::default())?; let rows = regression_rows(data.target, target)?; + // Two guards on the *encoded* width, which is a different number from the + // one `feature_table` checked: `preprocess` both drops columns (entirely + // missing) and multiplies them (one-hot). + // + // Empty is a **panic**, not a poor fit: `random_tree` draws `rng.below(0)`, + // which is `0` by `SplitMix64`'s own answer to the empty range, and + // `sample_thresh` then indexes `spec.ranges[0]`. So a document whose only + // candidate feature is entirely missing crashes the command for any seed + // that does not open on a leaf. + anyhow::ensure!( + !data.feature_names.is_empty(), + "nothing left to fit {target:?} against once the empty columns were dropped{}", + match data.warnings.is_empty() { + true => String::new(), + false => format!(" — {}", data.warnings.join("; ")), + } + ); + // And wide is a *freeze*. A regular category with one distinct value per + // record one-hots to one column per record, and permutation importance is + // O(repeats × rows × features) evaluations — quadratic in records, on the + // thread that draws the next frame. The bound this module's header quotes + // was measured on a document whose row-identity category is a `Label`, so + // it does not cover this shape and refusing is honest where a silent + // multi-second freeze is not. `Label` is the fix a user wants, and the + // message says so. + anyhow::ensure!( + data.feature_names.len() <= MAX_FEATURES, + "fitting {target:?} would encode {} features from {} records, which is too wide to \ + search on the UI thread. A category with a distinct value per record becomes one \ + column per record — mark such a category as a Label so it is not a feature", + data.feature_names.len(), + rows.len(), + ); + let names: Vec<&str> = data.feature_names.iter().map(String::as_str).collect(); let spec = Spec { ranges: &data.ranges, @@ -272,8 +362,7 @@ pub fn fit_measure( .map(|(j, share)| (data.feature_names[j].clone(), share)) .collect(), warnings: data.warnings, - seed: request.seed, - config: request.config, + request: *request, }) } @@ -467,8 +556,100 @@ mod tests { let a = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); let b = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits"); assert_eq!(a.formula.raw, b.formula.raw); - assert_eq!(a.seed, b.seed); - assert_eq!(a.config, b.config); + assert_eq!(a.request, b.request); + } + + /// Copilot on PR #25, and it is right: the outcome used to carry `seed` + /// plus `config`, which is the pair gotcha 37b names — and that pair is + /// not sufficient *here*. Cross-validation draws from the same generator + /// the fit then draws from, so changing `cv_folds` alone moves the + /// champion. Recording two of four fields documented the reproducibility + /// contract without meeting it. + #[test] + fn the_fold_count_moves_the_champion_and_so_is_recorded() { + let (model, view) = document(); + let base = request(); + let fewer = FitRequest { + cv_folds: 2, + ..base + }; + assert_eq!(base.seed, fewer.seed); + assert_eq!(base.config, fewer.config); + let a = fit_measure(&model, &view, "Score", FIT_INTO, &base).expect("fits"); + let b = fit_measure(&model, &view, "Score", FIT_INTO, &fewer).expect("fits"); + assert_ne!( + a.formula.raw, b.formula.raw, + "cv_folds did not reach the champion, so recording seed + config \ + would have been sufficient after all" + ); + assert_eq!(a.request, base, "the outcome must carry what produced it"); + assert_eq!(b.request, fewer); + } + + /// Copilot on PR #25. `preprocess` drops a column that is entirely + /// missing, so a document whose only candidate feature is empty encodes to + /// **zero** features — and `random_tree` then draws `rng.below(0)`, which + /// `SplitMix64` answers with `0`, and `sample_thresh` indexes + /// `spec.ranges[0]`. A panic, on a legal document, for any seed that does + /// not happen to open on a leaf. It is refused, and the refusal carries + /// the warnings naming what was dropped. + #[test] + fn a_document_with_no_usable_feature_is_refused_rather_than_panicking() { + let (mut model, mut view) = document(); + // Both halves are needed to reach zero encoded features, and finding + // that out is the point: clearing `Cost` alone leaves `Region`'s + // one-hot standing, so the fit returns a constant leaf rather than + // crashing. The panic needs *every* candidate column gone — no regular + // category to one-hot, and the one measure entirely missing. + model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Label; + view.set_axis(REGION, Axis::None); + for i in 0..12u32 { + model.clear_cell(&cell(&format!("r{i}"), "Cost")); + } + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err("every feature column is empty"); + let msg = err.to_string(); + assert!(msg.contains("nothing left to fit"), "{msg}"); + assert!( + msg.contains("Cost"), + "the refusal must name what was dropped: {msg}" + ); + } + + /// Copilot on PR #25. `FitConfig` bounds the *search*; nothing bounded the + /// *width*. A regular category with one distinct value per record one-hots + /// to one column per record, and permutation importance is + /// `repeats × rows × features` evaluations — quadratic in records, on the + /// thread that draws the next frame. The module header's measured bound + /// does not cover this shape, because its fixture marks the row-identity + /// category as a `Label`; this is the same fixture with that one word + /// removed, which is exactly the mistake a user makes. + #[test] + fn a_category_with_a_distinct_value_per_record_is_refused_not_searched() { + let (mut model, view) = document(); + model.category_mut(TAG).expect("Tag exists").kind = CategoryKind::Regular; + // Past `MAX_FEATURES`, which the 12-record fixture is not — and the + // refusal has to be the *width* one. At twelve records this document + // is refused too, but by `fitted_formula` noticing the champion split + // on `Tag=r0`: a different rule, which would have made this test pass + // without the guard existing at all. + for i in 12..(MAX_FEATURES as u32 + 12) { + let tag = format!("r{i}"); + model + .category_mut(TAG) + .expect("Tag exists") + .add_item(tag.clone()); + model.set_cell(cell(&tag, "Cost"), CellValue::Number(f64::from(i))); + model.set_cell(cell(&tag, "Score"), CellValue::Number(1000.0)); + } + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err("one one-hot column per record is not a fit"); + let msg = err.to_string(); + assert!(msg.contains("too wide"), "{msg}"); + assert!( + msg.contains("Label"), + "the message must name the fix: {msg}" + ); } /// And the other half: a different *config* under the same seed searches @@ -640,7 +821,7 @@ mod tests { ); assert!(line.contains("baseline"), "{line}"); assert!( - line.contains(&format!("{:#x}", outcome.seed)), + line.contains(&format!("{:#x}", outcome.seed())), "the seed is what reproduces the fit: {line}" ); } From f4600eb2772d2d64609f36a58a93e338009606c9 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 22:57:33 -0700 Subject: [PATCH 46/72] docs: stop giving ngrok a recipe I have not run Copilot's third pass, on the flag I introduced LAST round while fixing the flag I introduced the round before. Two consecutive rounds of writing an access control into the README without running it is exactly the doctrine committed in 5c1fefb, so this does not make it three by swapping in a traffic-policy file I have also not run. CHECKED AGAINST NGROK'S OWN DOCS RATHER THAN TAKEN: * ngrok.com/docs/agent/cli -- `--cidr-allow` is still ACCEPTED, marked deprecated with "use traffic policy instead". So Copilot's stated mechanism, "no longer accepts --cidr-allow, fails at startup", is OVERSTATED: it is deprecated, not removed. The substance (do not document it) stands; the reason is obsolescence, not breakage. * ngrok.com/docs/traffic-policy/actions/restrict-ips -- the action really is `restrict-ips`, supplied via `--traffic-policy-file`. Knowing both spellings is still not knowing that a policy file I compose here would deny anything, so the section now names the MECHANISM and points at the documentation instead of offering a command line. A recipe that is right today and silently wrong at the next major version is worse than a pointer, and the spelling here has already changed once. The same test applied to every other command line in the section, which is the half worth more than the ngrok fix: * `improvise-ws-server bank-info.improv --port 8787` -- checked against clap in ws-server/main.rs: `files: Vec` positional, `--port` long. OK. * the Caddyfile `@denied not remote_ip ...` / `abort @denied` -- checked against caddyserver.com/docs/caddyfile/matchers, which uses that exact idiom. OK, and still marked as not executed here. * `./web/build.sh`, `python3 -m http.server --directory web`, `cargo run -p improvise-ws-server -- examples/demo.improv` -- paths and targets exist. OK. Also merged main, which brings the beads export in 4abcdaa. improvise-kpnr was genuinely dangling in this diff before that and now resolves; the AGENTS.md audit over README, repo-map, the ADRs and index.html reports 73 cited IDs against 271 in the export with ONE miss -- improvise-m18f in repo-map gotcha 48, which is orchestrator-owned and already reported. 1193 passing (1184 + PR #20's), clippy and fmt clean. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 614301dd..e56213fb 100644 --- a/README.md +++ b/README.md @@ -319,12 +319,20 @@ For a one-off share with no DNS name and no certificate of your own, `cloudflared tunnel --url http://127.0.0.1:8787` or `ngrok http 8787` each print an https hostname that works the same way — pass the hostname to `?ws=` without a scheme. Both print a **public** URL and neither authenticates -anything by default — and note that `ngrok http --basic-auth` does **not** fix -that here, for the reason above: the page cannot answer a basic-auth challenge -on a socket handshake. `ngrok http --cidr-allow` is the form that works, being -the same address check as the Caddy example. An unguessable URL is not a -control either. If you cannot put an address check in front of it, do not -expose it. +anything by default, and an unguessable URL is not a control. + +Two things carry over from above. `ngrok http --basic-auth` does **not** help +here — the page cannot answer a basic-auth challenge on a socket handshake — +and what you want instead is the same **address check** the Caddy example +uses. But no command line for it is given here on purpose: ngrok has moved IP +restriction from the (now deprecated) `--cidr-allow` flag to a Traffic Policy +`restrict-ips` action passed with `--traffic-policy-file`, and cloudflared's +equivalent is a product feature of its own. Those spellings have already +changed once, none of them has been run in this repository, and a recipe that +is correct today and silently wrong at the next major version is worse than a +pointer — so: **consult their current documentation, and confirm the +restriction denies before you trust it with a document.** If you cannot put an +address check in front of the tunnel, do not expose it. Note the server binds `127.0.0.1` by default and `--bind` is the only way to widen that. Behind a proxy on the same host you should *not* widen it: the From 9a93c388a4ff304e44d88e8915623bcc8999dae9 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:01:30 -0700 Subject: [PATCH 47/72] test(wizard): assert the cell padding against a sentinel, not against the hint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's third finding on PR #22, and it is right about the thing this repo cares most about: my combining-mark case passed for the wrong reason. Drawing through the widget cannot see the *under*-padding fault at all. The hint runs first and pads the row itself, so the cells a combining-mark message fails to reach hold spaces, not anything visible — and "does the hint survive" is satisfied either way. Worse, the check looked as though it had been proved, because the loop panicked on the wide-character case before it ever reached the combining one. That is `doctrine(an_injected_fault_must_be_the_fault_you_named)` from the other end: the injection fired, on a different arm than the one being claimed. Replaced with a direct test of `set_bottom_line` over a row pre-filled with `X`, two cells wider than the panel so the border is a real cell rather than an artefact of the assertion. Every interior cell must be the text or a space; the two border cells must still be `X`. Both faults then show up as a surviving `X` — one inside the panel, one outside it: wide, chars().count(): ["X","广"," ","度",…," "," "] border eaten combining, same: ["X","é","é","é"," "×12,"X","X","X","X"] 3 uncleared Each case was watched failing **on its own**, with the other removed, so neither is hiding behind the other's panic. `short` is kept as the ASCII arm that must stay green throughout. Co-Authored-By: Claude Opus 5 (1M context) --- src/ui/import_wizard_ui.rs | 66 ++++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/src/ui/import_wizard_ui.rs b/src/ui/import_wizard_ui.rs index 0c8557ac..a5391ee7 100644 --- a/src/ui/import_wizard_ui.rs +++ b/src/ui/import_wizard_ui.rs @@ -649,38 +649,50 @@ mod tests { } /// The padding is in terminal cells, not characters (Copilot, PR #22). - /// `Buffer` lays a string out by display width, so a message carrying a - /// wide character padded by `chars().count()` writes past the panel and - /// over its right border — and a combining mark, which is zero cells, - /// under-pads and leaves a sliver of the hint behind. + /// `Buffer` lays a string out by display width, so a `chars().count()` + /// pad is wrong in both directions: a wide character counts one and + /// occupies two, so the helper runs past the panel and over its right + /// border; a combining mark counts one and occupies none, so the helper + /// stops short and leaves cells it was supposed to clear. /// - /// Both are asserted through the border, which is the thing that breaks: - /// every row of a correctly drawn panel ends in `│`, and the row must not - /// contain what it replaced. `render` draws into an 80×22 area, so the - /// border is real and not an artefact of the assertion. + /// **Asserted against a sentinel**, which is the correction Copilot made + /// to a first version of this test. Drawing through the widget cannot see + /// the under-padding case at all: the hint that ran first had *already* + /// filled the row with its own padding, so the cells the combining + /// message failed to reach held spaces rather than anything visible, and + /// a "does the hint survive" assertion passed either way. (The first + /// version also looked like it had been proved, because the loop panicked + /// on the wide-character case before ever reaching the combining one — + /// `doctrine(an_injected_fault_must_be_the_fault_you_named)`, from the + /// other end.) + /// + /// So this fills the row with `X` first and asks the helper directly: + /// every interior cell must be the text or a space, and the two cells + /// standing in for the panel border must still be `X`. Both faults are + /// then visible as a surviving `X` — one inside, one outside. #[test] - fn the_bottom_line_is_padded_in_cells_not_characters() { - for message in [ - "\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}", // six wide chars, twelve cells - "e\u{0301}e\u{0301}e\u{0301}", // combining acutes, zero cells each + fn set_bottom_line_clears_every_cell_it_covers_and_no_more() { + const W: usize = 18; + for text in [ + "\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}\u{5e7f}\u{5ea6}", // 6 chars, 12 cells + "e\u{0301}e\u{0301}e\u{0301}", // 6 chars, 3 cells + "short", // 5 chars, 5 cells ] { - let mut w = wizard(); - w.message = Some(message.to_string()); - let drawn = render(&w); - let bottom = drawn.lines().nth_back(1).expect("the row above the border"); + let mut buf = Buffer::empty(Rect::new(0, 0, W as u16 + 2, 1)); + buf.set_string(0, 0, "X".repeat(W + 2), Style::default()); - assert!( - bottom.ends_with('│'), - "{message:?} pushed the panel border off the row: {bottom:?}" - ); - assert!( - !bottom.contains("cancel"), - "{message:?} left the hint behind: {bottom:?}" - ); + set_bottom_line(&mut buf, 1, 0, W, text, Style::default()); + + let row: Vec<&str> = (0..W as u16 + 2).map(|i| buf[(i, 0)].symbol()).collect(); + assert_eq!(row[0], "X", "{text:?} wrote left of the panel: {row:?}"); assert_eq!( - drawn.lines().count(), - 22, - "{message:?} changed the panel's height" + row[W + 1], + "X", + "{text:?} overran the panel's right border: {row:?}" + ); + assert!( + !row[1..=W].contains(&"X"), + "{text:?} left a cell it should have cleared: {row:?}" ); } } From 1c98aa0929bb3d331bbb822ae0b0f37961fd2e5d Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:03:01 -0700 Subject: [PATCH 48/72] docs: ambient credentials versus challenge-response, which is the real rule Copilot's fourth pass, and it caught this text CONTRADICTING ITSELF three paragraphs apart -- in prose I wrote in a single commit while fixing the previous round's contradiction. The Caddyfile comment said "the browser cannot send a credential on a WebSocket handshake". That is FALSE, and the same section already said so: the Cloudflare Access bullet explains that Access works by a session COOKIE the handshake carries. A WebSocket handshake is an HTTP request; the browser attaches ambient credentials to it automatically. The true rule, which is sharper than the one it replaces and explains both halves instead of one: AMBIENT credentials are carried -- cookies for that host above all, which is why an identity-aware proxy works at all. CHALLENGE-RESPONSE cannot be satisfied -- the page can set no header (there is no Authorization argument to `new WebSocket`) and can answer no prompt (a handshake surfaces no login dialog and fails to script as a bare connection error). So basic auth is still ruled out, for a REASON rather than by an over-broad invariant, and Access is no longer an unexplained exception to it. Two consequences carried through the rest of the section, on the grep discipline the last round earned: * mTLS is no longer asserted to fail. The certificate IS offered at the TLS layer rather than by the page; only the selection UI is doubtful, and that is now marked as not verified here rather than stated. * "a query parameter is the one credential new WebSocket can carry" restated the same over-broad claim in miniature, and becomes "the one place the page can put a credential DELIBERATELY -- everything else it sends is ambient." 1193 passing, unchanged; no code touched. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 40 +++++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index e56213fb..c81d9745 100644 --- a/README.md +++ b/README.md @@ -248,13 +248,24 @@ proxy. TLS gets you confidentiality, not access control. **And the control has to be one a browser can actually satisfy**, which rules out most of the obvious ones. The page opens the socket as -`new WebSocket(url)` — one argument, a URL. The browser WebSocket API **cannot -set request headers**, and a WebSocket handshake shows **no login prompt**, so -HTTP basic auth in front of it is not a locked door, it is a **401 the page -cannot answer and cannot explain** — the same reason a self-signed certificate -is unusable here (see "Why not native `wss://`"). Client-certificate mTLS has -the same problem: the selection UI browsers show for a top-level navigation is -not something a cross-origin socket handshake can rely on. +`new WebSocket(url)` — one argument, a URL — and the distinction that decides +everything here is **ambient credentials versus challenge–response**: + +- The handshake **is** an HTTP request, so the browser attaches what it + attaches *automatically* — cookies for that host above all. That is why an + identity-aware proxy can work at all. +- What the page cannot do is **set a header** (there is no `Authorization` + argument to `new WebSocket`) or **answer a prompt** (a handshake surfaces no + login dialog, and its failure reaches script as a bare connection error). + +So **HTTP basic auth is not a locked door here, it is a 401 the page can +neither answer nor explain** — the same shape as the self-signed certificate +that rules out native `wss://` below. Anything challenge-shaped fails for that +reason; anything ambient may work. Client-certificate mTLS sits awkwardly +between the two — the certificate is offered at the TLS layer rather than by +the page, but the selection UI browsers show is a top-level-navigation +affordance, and whether a cross-origin socket handshake gets it is **not +something verified here**. So the honest controls are the ones that need **no credential from the page**: @@ -262,15 +273,18 @@ So the honest controls are the ones that need **no credential from the page**: below. It is about where the client is, so the page sends nothing. - **A private network** (Tailscale, WireGuard). The peer is only reachable by machines already on it; again the page sends nothing. -- **An identity-aware tunnel** (`cloudflared` with Cloudflare Access). This one +- **An identity-aware tunnel** (`cloudflared` with Cloudflare Access) — the + *ambient* case above, and the reason that distinction is drawn at all. It works by a session **cookie** set after an interactive login on the proxy's - own hostname, so it needs the user to visit that hostname first, and it is + own hostname, which the handshake then carries by itself; so it needs the + user to visit that hostname first, and it is subject to the browser's third-party-cookie rules. Plausible and *not verified here* — there is no browser in this repository — so it is named, not written out as a recipe. -What would make this genuinely easy is a **token in the URL**, because a query -parameter is the one credential `new WebSocket(url)` can carry. The server has +What would make this genuinely easy is a **token in the URL**, because the URL +is the one place the page can put a credential *deliberately* — everything else +it sends is ambient. The server has no such capability today, and whether it should grow one — or whether "auth is the proxy's job" becomes a stated design position — is **improvise-kpnr**. @@ -284,8 +298,8 @@ renews the certificate itself. Point a DNS name at the host, then: # Caddyfile improvise.example.com { # NOT optional — see above. Without it the document is world-writable. - # An allow-list, not a password: the browser cannot send a credential on a - # WebSocket handshake, so this is checked against the client's address. + # An allow-list rather than a password, because the page cannot answer a + # challenge on a socket handshake: this is checked against the address. @denied not remote_ip 203.0.113.4 198.51.100.0/24 abort @denied From c23f7db3f4b7303e143075bed5e24dbbd456f41a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:03:05 -0700 Subject: [PATCH 49/72] fix(cmd): a `:` line's tidy-up runs where an abort cannot reach it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot round 2, and it is a regression this branch amplified. A failed `:w ` set `abort_effects` (f0f400d), which ate `ExecuteCommand`'s trailing `ReturnToNormalUnlessMoved` along with the rest of the batch. The app stayed in `CommandMode` with the stale command line drawn — and `draw_bottom_bar` renders the minibuffer *instead of* the status line, so the "Save error:" set one effect earlier was invisible. A failed save looked exactly like a key that did nothing. My own hand-back documented that mechanism and I did not check my own change against it. The `Save` arm (`:w` with no path) has behaved this way since improvise-8tk, so only the `SaveAs` arm is new; the test covers both. **The sharper rule, which gotcha 21 does not currently state.** The question is not "should the batch abort" — it must, or `:w bad . q` quits and discards the model, which is the whole of improvise-8tk. The question is what an abort owes the effects that were only ever going to tidy up. The answer is **ordering, not classification**: a batch puts its bookkeeping in front, where an abort cannot reach it. No effect gains a "runs anyway" flag, nothing needs a second classification beside `EffectKind`, and the rule generalises to any command composing bookkeeping with something that can fail — it is the same move `CommitSave` already makes with its `ChangeMode`. Running the restoration at the head does not weaken improvise-toc's "decide by observation" property: whatever a later effect sets, it sets *after* this, so the mode a `:` line ends in is unchanged in every existing case. `colon_enter_edit_at_cursor_is_not_snapped_back_to_normal` passes untouched. The only difference is what happens when the batch never reaches its end, which is the bug. Evidence: `a_failed_colon_write_lands_in_normal_where_its_error_is_drawn` drives both arms through real keypresses and was watched failing on the old order — it fails first on the pre-existing `Save` arm. `a_failed_colon_write_still_refuses_to_quit_after_it` is the other half of the rule and proves the fix did not buy visibility at the price of improvise-8tk. Also lands a characterization test for Copilot's second finding, which is declined in this PR with evidence — see the next paragraph and the PR thread. `Ctrl-W` from a records grid does return to plain `Normal` while the layout is still records, but so do `:` and `Ctrl-E`, and always have: the exit is a constant baked into `TextEntry::new(.., "normal")` and into each commit command. `every_minibuffer_mode_entered_from_records_returns_to_plain_normal` records all three. Fixing one arm is the shape `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` names, and fixing the class means text-entry modes carrying their origin — a design change wider than this PR. The binding *table* is not involved: an `IMPROVISE_BINDING_DUMP` diff shows `RecordsNormal` gains `Ctrl-W` and loses nothing, so `with_parent` inheritance is intact. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/text_buffer.rs | 33 +++- crates/improvise-command/src/ui/app.rs | 145 ++++++++++++++++++ 2 files changed, 170 insertions(+), 8 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/text_buffer.rs b/crates/improvise-command/src/command/cmd/text_buffer.rs index 88f3d910..a6dad806 100644 --- a/crates/improvise-command/src/command/cmd/text_buffer.rs +++ b/crates/improvise-command/src/command/cmd/text_buffer.rs @@ -393,17 +393,34 @@ impl Cmd for ExecuteCommand { match crate::command::parse::parse_line_with(ctx.registry, &raw) { Ok(cmds) => { - let mut effects: Vec> = Vec::new(); + // Leave the command line **first**, before anything that can + // fail runs. + // + // Still decided at apply time rather than here, because + // ViewBack, ViewForward and StartImportWizard move the mode + // only on some runs and no up-front answer is right + // (improvise-toc). Running it at the head does not weaken + // that: whatever a later effect sets, it sets *after* this, + // so the mode a `:` line ends in is unchanged in every case + // — the only difference is what happens when the batch never + // reaches the end. + // + // Which is the point. A failed `Save`/`SaveAs` sets + // `abort_effects` (gotcha 21), and an abort at the tail ate + // this restoration too, stranding the user in `CommandMode` + // with the stale command line drawn — where + // `draw_bottom_bar` renders the minibuffer *instead of* the + // status line, so the error the effect had just set was + // invisible. The abort must stop the *chain* (`:w bad . q` + // must not quit) and must not stop the *bookkeeping*, and + // ordering answers that without giving any effect a "runs + // anyway" flag: tidy-up goes where an abort cannot reach it. + let mut effects: Vec> = vec![Box::new( + effect::ReturnToNormalUnlessMoved(ctx.mode.clone()), + )]; for cmd in cmds { effects.extend(cmd.execute(ctx)); } - // Return to Normal unless one of the effects above moves the - // mode. Decided at apply time, not here: ViewBack, - // ViewForward and StartImportWizard change it only on some - // runs, so no up-front answer is right (improvise-toc). - effects.push(Box::new(effect::ReturnToNormalUnlessMoved( - ctx.mode.clone(), - ))); effects } Err(msg) => { diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 01cbb048..e75c67ac 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1672,6 +1672,151 @@ mod tests { ); } + /// Copilot round 2 reported that `Ctrl-W` from a records grid returns to + /// `Normal`, losing the records keymap while the layout is still a + /// records layout. It is right, and it is a **class**: every minibuffer + /// mode reachable from `RecordsNormal` exits to `Normal`, because the + /// exit is a constant baked into `TextEntry::new(..., "normal")` and + /// into each commit command. + /// + /// The point of this test is that the class **predates** `SavePrompt`: + /// `:` and `Ctrl-E` do exactly the same thing and always have. That is + /// why the fix is not "make `SavePrompt` remember its origin" — that + /// would leave two of the three arms broken, which is the shape + /// `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` + /// names. + /// + /// The binding *table* is not involved and was checked rather than + /// reasoned about: an `IMPROVISE_BINDING_DUMP` diff shows + /// `RecordsNormal` gains `Ctrl-W` and loses nothing, so `with_parent` + /// inheritance (gotcha 13) is intact. What is lost is the runtime mode. + #[test] + fn every_minibuffer_mode_entered_from_records_returns_to_plain_normal() { + for key in [ + KeyEvent::new(KeyCode::Char(':'), KeyModifiers::NONE), + KeyEvent::new(KeyCode::Char('e'), KeyModifiers::CONTROL), + KeyEvent::new(KeyCode::Char('w'), KeyModifiers::CONTROL), + ] { + let mut wb = Workbook::new("T"); + wb.add_category("Region").unwrap(); + wb.add_item("Region", "East").unwrap(); + let mut app = App::new(wb, None); + app.handle_key(KeyEvent::new(KeyCode::Char('R'), KeyModifiers::NONE)) + .unwrap(); + assert_eq!( + app.view_state.mode, + AppMode::RecordsNormal, + "setup: R enters records mode" + ); + + app.handle_key(key).unwrap(); + assert!( + app.view_state.mode.minibuffer().is_some(), + "setup: {:?} opens a minibuffer, got {:?}", + key.code, + app.view_state.mode + ); + + app.handle_key(KeyEvent::new(KeyCode::Esc, KeyModifiers::NONE)) + .unwrap(); + assert_eq!( + app.view_state.mode, + AppMode::Normal, + "{:?} cancels back to plain Normal, not RecordsNormal — the \ + layout is still a records layout, so this is the bug, and \ + it is the same for all three keys", + key.code + ); + assert!( + app.layout.is_records_mode(), + "...and the layout really is still records, which is what \ + makes the mode wrong rather than merely different" + ); + } + } + + /// A failing `:` command must not strand the user in `CommandMode`. + /// + /// Found by Copilot on PR #27, and it is the sharper form of gotcha 21. + /// The abort is *supposed* to stop the chain — `:w /bad . q` must not + /// quit. What it must not also stop is the **bookkeeping**: the mode + /// restoration that `ExecuteCommand` appends. Aborting before that ran + /// left the app in `CommandMode` with the stale command line still + /// drawn, and `draw_bottom_bar` renders the minibuffer *instead of* the + /// status line — so the "Save error:" set one effect earlier was + /// invisible. A failed save looked exactly like a key that did nothing. + /// + /// The answer is **ordering, not classification**: a batch puts its + /// tidy-up where an abort cannot reach it, which is in front. No effect + /// needs a new "runs anyway" flag, and the rule generalises to any + /// command composing bookkeeping with something that can fail. + /// + /// Both arms are here because they fail in different places: no path at + /// all is `Save`'s guard (aborting since improvise-8tk, so this arm was + /// broken before this branch), and a bad path is `SaveAs`'s write + /// (aborting since f0f400d on this branch). + #[test] + fn a_failed_colon_write_lands_in_normal_where_its_error_is_drawn() { + for (line, expect) in [ + ("w", "No file path"), + ("w /nonexistent-dir-improvise-lae/m.improv", "Save error:"), + ] { + let mut app = App::new(Workbook::new("T"), None); + app.handle_key(KeyEvent::new(KeyCode::Char(':'), KeyModifiers::NONE)) + .unwrap(); + for c in line.chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + + assert!( + app.view_state.status_msg.contains(expect), + ":{line} should report {expect:?}, got {:?}", + app.view_state.status_msg + ); + assert_eq!( + app.view_state.mode, + AppMode::Normal, + ":{line} must leave the command line, or the status it just \ + set is hidden behind the minibuffer" + ); + } + } + + /// The other half of the same rule: the abort must still stop the + /// *chain*. `:w . q` is improvise-8tk's data loss spelled on one + /// `:` line, and moving the mode restoration to the front must not have + /// bought the fix above at the price of that one. + #[test] + fn a_failed_colon_write_still_refuses_to_quit_after_it() { + let mut app = App::new(Workbook::new("T"), None); + app.view_state.buffers.insert( + "command".to_string(), + "w /nonexistent-dir-improvise-lae/m.improv . q".to_string(), + ); + app.handle_key(KeyEvent::new(KeyCode::Char(':'), KeyModifiers::NONE)) + .unwrap(); + // `:` clears the buffer on entry (gotcha 11), so type it instead. + for c in "w /nonexistent-dir-improvise-lae/m.improv . q".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + assert_ne!( + app.view_state.mode, + AppMode::Quit, + "a failed write must still abort the rest of the `:` line" + ); + assert!( + app.view_state.status_msg.starts_with("Save error:"), + "got {:?}", + app.view_state.status_msg + ); + } + /// `ZZ` is save-and-quit. With no file path the save fails and aborts the /// chain (improvise-8tk), which also proves the second Z reached `wq`. #[test] From fb507b98557254fcff39537957f9a04a28ea587a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:03:34 -0700 Subject: [PATCH 50/72] fix(opfs): a directory is not a document refresh could not measure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's second round on PR #24, and it found the hole in the previous commit's own argument. That commit said an entry `refresh` cannot measure counts as a document, "because the one cause is another tab holding the exclusive handle". There is a second cause, and I missed it: **`keys()` yields directories as well as files**, and `getFileHandle` on a directory rejects with `TypeMismatchError`. So a directory would have been listed as a document, and one can never be attached or read — the exact picker bug the zero-byte filter exists to remove, reintroduced by the fix for it. The two failures now get opposite answers, which is the whole of `holds_bytes`: - **`getFileHandle` fails** — the name is not a file. A directory, or an entry removed between `keys()` and here. Not a document; skip it. - **`getFile` fails on a handle `getFileHandle` already returned** — the exclusive-lock case, a second tab of the same origin. That one still fails open, for the reasons the previous commit gives. Argued *and* tested this time. `a_directory_entry_is_not_a_document` in the harness makes a real directory in OPFS — `OpfsFs` has no directory vocabulary at all, so the harness reaches `getDirectoryHandle` itself — and asserts it is not listed, with a real file written beside it as the precondition, so the check cannot pass by the directory never having been created or by `keys()` walking nothing. Proved discriminating by flipping the `None` arm back to `true`: it fails with `refresh() listed a directory as a document: ["a-folder"]`. Adding it also exposed that three checks were reading the leftovers of the check before them — the battery broke the moment one was inserted in the middle. Every check now wipes first, so the order of the battery is not load-bearing. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/opfs.rs | 65 +++++++++++++++--------- harness/opfs-contract/Cargo.lock | 1 + harness/opfs-contract/Cargo.toml | 15 ++++++ harness/opfs-contract/src/lib.rs | 85 +++++++++++++++++++++++++++++++- harness/tests/opfs.spec.js | 1 + 5 files changed, 142 insertions(+), 25 deletions(-) diff --git a/crates/improvise-io/src/opfs.rs b/crates/improvise-io/src/opfs.rs index 7ff4c915..070b2bb5 100644 --- a/crates/improvise-io/src/opfs.rs +++ b/crates/improvise-io/src/opfs.rs @@ -72,6 +72,20 @@ fn entry_name(path: &Path) -> Result { .ok_or_else(|| anyhow!("{} is not a usable document name", path.display())) } +/// The size of a file, or `None` if it cannot be read — which, on a handle +/// `getFileHandle` already returned, means somebody else holds the exclusive +/// sync access handle. See [`OpfsFs::holds_bytes`]. +async fn file_size(file: &web_sys::FileSystemFileHandle) -> Option { + Some( + JsFuture::from(file.get_file()) + .await + .ok()? + .dyn_into::() + .ok()? + .size(), + ) +} + pub struct OpfsFs { root: FileSystemDirectoryHandle, /// Sync access handles for attached documents. Exclusive per file: @@ -213,42 +227,45 @@ impl OpfsFs { Ok(()) } - /// Whether one entry holds anything — i.e. whether it is a document. + /// Whether one entry is a document with content in it. /// /// Read off the sync access handle when this instance has one, and only - /// otherwise through `getFile()`, because a file under an exclusive sync - /// access handle is not reliably readable by the promise-based API. + /// otherwise through the promise-based API, because a file under an + /// exclusive sync access handle is not reliably readable that way. /// - /// **An entry it cannot measure counts as a document.** The one way that - /// happens is another context of the same origin holding the exclusive - /// handle — two tabs of the same site — and an entry somebody else has - /// open is certainly a document. The alternative is worse in both - /// directions: failing the whole `refresh` would make a second tab break - /// the first one's document picker, and dropping the entry would hide a - /// document that exists. Over-reporting costs one unopenable row, the - /// same direction `Dirty::of` is deliberately sloppy in. + /// Two ways the measurement can fail, and they get opposite answers — + /// which is the whole of the function: + /// + /// - **`getFileHandle` fails**: the name is not a file. `keys()` yields + /// directories as well as files, and a directory rejects with + /// `TypeMismatchError`; an entry removed between `keys()` and here + /// rejects too. Neither is a document, and neither can ever be + /// `attach`ed or `read`, so listing one would be the same picker bug + /// this filter exists to remove (found by Copilot on PR #24). + /// - **`getFile` fails on a real file handle**: another context of the + /// same origin holds the exclusive sync access handle — two tabs of the + /// same site. An entry somebody else has open is certainly a document, + /// so this one **fails open**. Dropping it would hide a document that + /// exists, and failing the whole `refresh` would let a second tab break + /// the first one's picker. Over-reporting costs one row, the same + /// direction `Dirty::of` is deliberately sloppy in. async fn holds_bytes(&self, name: &str) -> bool { if let Some(handle) = self.handles.borrow().get(name) { return handle.get_size().is_ok_and(|size| size > 0.0); } - self.entry_size(name).await.is_none_or(|size| size > 0.0) + match self.file_handle(name).await { + None => false, + Some(file) => file_size(&file).await.is_none_or(|size| size > 0.0), + } } - /// The size of an unattached entry, or `None` if it cannot be read. - async fn entry_size(&self, name: &str) -> Option { - let file = JsFuture::from(self.root.get_file_handle(name)) + /// The file handle for `name`, or `None` if it does not name a file. + async fn file_handle(&self, name: &str) -> Option { + JsFuture::from(self.root.get_file_handle(name)) .await .ok()? .dyn_into::() - .ok()?; - Some( - JsFuture::from(file.get_file()) - .await - .ok()? - .dyn_into::() - .ok()? - .size(), - ) + .ok() } fn with_handle( diff --git a/harness/opfs-contract/Cargo.lock b/harness/opfs-contract/Cargo.lock index 8365df49..08a283d2 100644 --- a/harness/opfs-contract/Cargo.lock +++ b/harness/opfs-contract/Cargo.lock @@ -303,6 +303,7 @@ dependencies = [ "serde_json", "wasm-bindgen", "wasm-bindgen-futures", + "web-sys", ] [[package]] diff --git a/harness/opfs-contract/Cargo.toml b/harness/opfs-contract/Cargo.toml index 03807337..d08cfbe8 100644 --- a/harness/opfs-contract/Cargo.toml +++ b/harness/opfs-contract/Cargo.toml @@ -26,5 +26,20 @@ serde_json = "1" wasm-bindgen = "0.2" wasm-bindgen-futures = "0.4" +# Only in order to CREATE a directory in OPFS, which `OpfsFs` never does. +# `keys()` yields directories alongside files and one must not be listed as a +# document; without this the fix for that would be argued rather than tested. +[dependencies.web-sys] +version = "0.3" +features = [ + "FileSystemDirectoryHandle", + "FileSystemGetDirectoryOptions", + "FileSystemRemoveOptions", + "Navigator", + "StorageManager", + "WorkerGlobalScope", + "WorkerNavigator", +] + [package.metadata.wasm-pack.profile.release] wasm-opt = ["-O", "--enable-bulk-memory", "--enable-nontrapping-float-to-int"] diff --git a/harness/opfs-contract/src/lib.rs b/harness/opfs-contract/src/lib.rs index 542e338e..78731d4a 100644 --- a/harness/opfs-contract/src/lib.rs +++ b/harness/opfs-contract/src/lib.rs @@ -28,7 +28,9 @@ use improvise_io::opfs::OpfsFs; use improvise_io::storage::Storage; use improvise_io::storage::contract::storage_contract; use serde::Serialize; +use wasm_bindgen::JsCast; use wasm_bindgen::prelude::*; +use wasm_bindgen_futures::JsFuture; /// One named check and what happened to it. #[derive(Serialize)] @@ -62,6 +64,44 @@ impl Report { } } +/// The origin's root directory, reached the way `OpfsFs::open` reaches it. +/// +/// Only the harness needs this. `OpfsFs` has no directory vocabulary at all — +/// it stores one flat entry per document — so making a directory, to check +/// that `refresh` skips one, has to happen outside it. +async fn opfs_root() -> Result { + let scope = js_sys::global() + .dyn_into::() + .map_err(|_| anyhow!("not in a dedicated Web Worker"))?; + JsFuture::from(scope.navigator().storage().get_directory()) + .await + .map_err(|e| anyhow!("getDirectory(): {e:?}"))? + .dyn_into::() + .map_err(|_| anyhow!("getDirectory() did not yield a directory handle")) +} + +async fn make_directory(name: &str) -> Result<()> { + let opts = web_sys::FileSystemGetDirectoryOptions::new(); + opts.set_create(true); + JsFuture::from( + opfs_root() + .await? + .get_directory_handle_with_options(name, &opts), + ) + .await + .map_err(|e| anyhow!("getDirectoryHandle({name}): {e:?}"))?; + Ok(()) +} + +async fn drop_directory(name: &str) -> Result<()> { + let opts = web_sys::FileSystemRemoveOptions::new(); + opts.set_recursive(true); + JsFuture::from(opfs_root().await?.remove_entry_with_options(name, &opts)) + .await + .map_err(|e| anyhow!("removeEntry({name}): {e:?}"))?; + Ok(()) +} + /// Remove every *document* in the origin root, so a run does not inherit the /// last one's leftovers. /// @@ -126,6 +166,7 @@ pub async fn run_checks() -> Result { .run( "an_attached_but_unwritten_entry_is_not_a_document", async || { + wipe(&fs).await?; fs.attach(Path::new("attached.improv")).await?; let listed = fs.list(Path::new(""))?; ensure!( @@ -156,11 +197,52 @@ pub async fn run_checks() -> Result { ) .await; + // `keys()` yields DIRECTORIES as well as files, and `getFileHandle` on a + // directory rejects with `TypeMismatchError`. Treating that as "could not + // measure, so assume a document" would list an entry that can never be + // attached or read — the same picker bug the zero-byte filter removes + // (found by Copilot on PR #24). `OpfsFs` never creates a directory, so + // the harness makes one itself. + report + .run("a_directory_entry_is_not_a_document", async || { + wipe(&fs).await?; + make_directory("a-folder").await?; + + let fresh = OpfsFs::open().await?; + fresh.refresh().await?; + let listed = fresh.list(Path::new(""))?; + ensure!( + listed.is_empty(), + "refresh() listed a directory as a document: {listed:?}" + ); + + // The precondition, so this cannot pass by the directory never + // having been created: a real file beside it IS listed, which + // also says `keys()` walked the root at all. + fs.attach(Path::new("beside.improv")).await?; + fs.write(Path::new("beside.improv"), b"a real document")?; + let again = OpfsFs::open().await?; + again.refresh().await?; + let seen = again.list(Path::new(""))?; + ensure!( + seen == vec![std::path::PathBuf::from("beside.improv")], + "the file beside the directory was not listed alone: {seen:?}" + ); + + drop_directory("a-folder").await?; + Ok("skipped".into()) + }) + .await; + // The hand-rolled `keys()` async-iterator walk in `refresh` is the least // certain code in opfs.rs (improvise-xf1 named it). One entry could pass // by accident; several, out of order, cannot. report .run("refresh_walks_every_key_and_sorts_them", async || { + // Wipes first, like every other check. A battery whose steps + // depend on the leftovers of the step before breaks when one is + // inserted, which is exactly what the directory check above did. + wipe(&fs).await?; for name in ["c.improv", "a.improv", "b.improv"] { fs.attach(Path::new(name)).await?; fs.write(Path::new(name), name.as_bytes())?; @@ -170,7 +252,7 @@ pub async fn run_checks() -> Result { let listed = fresh.list(Path::new(""))?; let names: Vec = listed.iter().map(|p| p.display().to_string()).collect(); ensure!( - names == ["a.improv", "attached.improv", "b.improv", "c.improv"], + names == ["a.improv", "b.improv", "c.improv"], "refresh() saw {names:?}" ); Ok(format!("{names:?}")) @@ -183,6 +265,7 @@ pub async fn run_checks() -> Result { // across a long one followed by a short one. report .run("read_and_write_are_absolute_at_offset_zero", async || { + wipe(&fs).await?; let p = Path::new("offsets.improv"); fs.attach(p).await?; fs.write(p, b"0123456789ABCDEF")?; diff --git a/harness/tests/opfs.spec.js b/harness/tests/opfs.spec.js index f5719bab..2fe6be7e 100644 --- a/harness/tests/opfs.spec.js +++ b/harness/tests/opfs.spec.js @@ -23,6 +23,7 @@ async function run(page, cmd) { const EXPECTED = [ "open_and_refresh_see_an_origin_root_holding_no_documents", "an_attached_but_unwritten_entry_is_not_a_document", + "a_directory_entry_is_not_a_document", "refresh_walks_every_key_and_sorts_them", "read_and_write_are_absolute_at_offset_zero", 'storage_contract_at_""', From aa1534e0783027ba260c4b752862affcef123be4 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:04:38 -0700 Subject: [PATCH 51/72] chore(beads): file five follow-ups surfaced by WU-G MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All from PR #27's hand-back, searched against the open backlog first and none of them duplicates. improvise-mai9 P2 — a status message written while a minibuffer is open is never drawn. draw_bottom_bar renders the minibuffer INSTEAD of the status line, so improvise-6t3 carefully ordered a failure message to win a race to a surface nobody can see. Copilot then found the regression this causes, in the same PR that documented the mechanism. improvise-z05r P3 — CommitExport does not bail on an empty path and ExportCsv does not abort the batch: the export arms of what PR #27 fixed for save. Third instance of the same asymmetry, so the structural fix may beat the third patch. improvise-g9jb P3 — the text-entry mode list is hand-written in two crates and checked against AppMode in neither, which now decides whether a browser refuses to open a prompt it cannot commit. improvise-zx8t P4 — draw.rs::mode_style matches AppMode with a wildcard, so the repo map's "the compiler names every site" promise holds in five places and not the sixth. improvise-rw3a P3 — the open half of improvise-lae: whether ZZ routes into the new save prompt, and what ZZ means once it is open. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 3001b18f..91a343bc 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -36,6 +36,7 @@ {"_type":"issue","id":"improvise-km8","title":"1.2 Fix Cargo.toml metadata","description":"Add missing package fields: description, repository, homepage, documentation, readme, keywords, categories, license. Description: 'Terminal pivot-table modeling in the spirit of Lotus Improv'. Keywords: tui, pivot, spreadsheet, data, improv. Categories: command-line-utilities, visualization.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:48Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:20:17Z","close_reason":"Metadata already present: description, license, repository, homepage, readme, keywords, categories all populated.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-902","title":"1.1 Audit and remove or update context/SPEC.md","description":"Compare context/SPEC.md against actual code. If spec contradicts code significantly, delete it. Optionally move salvageable conceptual content to docs/design-notes.md with a staleness disclaimer. Do not try to bring spec into sync.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:46Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:10:05Z","close_reason":"Audited SPEC.md against code. Found minor inaccuracies (storage internals, wizard step count, search mode representation) but no major architectural contradictions. Deleted SPEC.md since it was largely redundant with repo-map.md and design-principles.md. Salvaged product vision and non-goals into docs/design-notes.md with staleness disclaimer.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-tik","title":"Phase 1: Repository hygiene","description":"Raise the quality floor of what a stranger sees when they land on the repo. Subtasks cover spec audit, Cargo.toml metadata, publish-readiness, CSV audit, and example files.","status":"closed","priority":1,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:45Z","created_by":"Edward Langley","updated_at":"2026-04-09T07:22:34Z","closed_at":"2026-04-09T07:22:34Z","close_reason":"All 5 subtasks complete: SPEC.md audited, Cargo.toml metadata fixed, publish dry-run clean, CSV audit done, example files created.","dependencies":[{"issue_id":"improvise-tik","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T23:37:31Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-902","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-bv1","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-km8","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-n1h","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-mai9","title":"A status message written while a minibuffer is open is never drawn","description":"Found by WU-G while building improvise-lae (PR #27), and CONFIRMED THE HARD WAY in the same PR: Copilot then found a regression that this mechanism causes, which the agent had itself written down an hour earlier and not checked its own change against.\n\nTHE MECHANISM: src/draw.rs::draw_bottom_bar renders the minibuffer INSTEAD OF the status line. So a status_msg written while a minibuffer is open is never drawn at all -- not truncated, not overwritten, simply never rendered.\n\nWHO IS AFFECTED TODAY:\n * CategoryAdd / ItemAdd: a failed `commit-buffer category` is INVISIBLE. Worse, improvise-6t3 deliberately ordered that command's effects so the add effect is emitted LAST, specifically so its failure message would WIN over the command's optimistic status. It wins a race to a surface nobody can see.\n * A failed `:w \u003cpath\u003e` from CommandMode, which is the regression Copilot found on PR #27: the abort stops the batch before ReturnToNormalUnlessMoved and clear-buffer run, so the app stays in CommandMode with the stale command in the minibuffer and the new error hidden behind it.\n\nTHE WORKAROUND ALREADY SHIPPED, and it is a workaround: CommitSave emits [ChangeMode(Normal), SaveAs] so the error lands when the bottom row is the status bar again. That works for one command and does not generalise -- every other text-entry commit has the same hole.\n\nTWO CANDIDATE FIXES, and the choice is the ticket:\n * Draw both: give the minibuffer and the status line separate rows, or render the status into the minibuffer row when a message is pending. Cheap to describe, and it changes frame geometry, which is pinned by exact-frame characterization tests (see improvise-q0vb for what that costs).\n * Make it unrepresentable: a status message set while a minibuffer is open is a bug in the CALLER, so make the caller unable to do it -- e.g. commit paths always leave the mode before reporting. That is what CommitSave now does by hand, promoted to a rule.\n\nPREFER WHICHEVER MAKES THE FAILURE LOUD. The current state is the quiet kind: the code sets a message, the test asserts the message was set, and the user never sees it. A test asserting `app.status_msg == \"...\"` passes in every one of these cases, which is why this survived improvise-6t3's careful effect ordering.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:20Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-p1ym","title":"Project the page-axis item lists so [ and ] work in a browser","description":"The one key pair an independent smoke test found inert that the shared keymap did NOT fix, and it is a genuine projection gap rather than a resolution gap.\n\nGridCache::page_coords carries (category, SELECTED ITEM). PageNext/PagePrev need THE ITEM LIST TO CYCLE THROUGH, which nothing projects. So [ and ] resolve to a command name the client holds but cannot compose an argument for, and they reach Host::handle empty vec -- doctrine(an_upstream_key_is_a_black_hole), so they do nothing at all.\n\nCONFIRMED BY SMOKE TEST 2026-09-20 against the live page: \"[ / ] did not cycle Quarter (never saw Q2 ~17,100)\". The demo model puts Quarter on the page axis specifically so paging is a thing a visitor would try.\n\nTHE AGENT WAS TOLD NOT TO FORCE IT and did not -- a truthful \"these four keys work, [ / ] needs X\" is worth more than five half-working keys.\n\nSCOPE: extend page_coords (or add a sibling) to carry the item list per page-axis category. Note gotcha 27 -- \"what data is visible\" is View::filtering_cats() (Page UNION Filter) while \"what the user can page\" is categories_on(Axis::Page). This needs the SECOND one; projecting the first would let a client try to cycle a Filter axis, which by design it must not.\n\nWhile you are there: check whether the item list should be the EFFECTIVE items (hidden items excluded) rather than all of them, since [ and ] in the TUI cycle what the user can actually select.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -150,6 +151,7 @@ {"_type":"issue","id":"improvise-pby","title":"2.2 Write the README","description":"Replace existing README.md with new one under 250 lines. Sections in order: Title, one-sentence pitch, inline demo GIF, why this exists (Lotus Improv reference), quick start, key bindings, installation (Nix/crates.io/prebuilt - no Homebrew), codebase overview, expectations disclaimer, license. No badges, TOC, or contributing guide.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:32Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:02:52Z","close_reason":"README written with all 10 required sections, committed. License updated to Apache-2.0 per user request.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ihv","title":"2.1 Add Nix tooling for asciinema, VHS, and cargo-dist","description":"Modify flake.nix to add pkgs.asciinema and pkgs.vhs to dev shell nativeBuildInputs. Add cargo-dist if packaged in nixpkgs. Add runtime deps for VHS (ttyd, ffmpeg) if needed. Verify tools work via nix develop --command. Add nix run app or shell script for demo recording workflow.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:30Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:30:19Z","close_reason":"asciinema, vhs, cargo-dist added to flake.nix dev shell and verified. cargo-dist wrapper enables 'cargo dist' subcommand. scripts/record-demo.sh added.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-yk4","title":"Phase 2: README and demo artifacts","description":"The main launch work. README is 80% of the launch, demo artifacts are 20%. Covers Nix tooling for asciinema/VHS, writing the README, creating demo GIF and asciinema casts, and verifying all artifacts.","status":"closed","priority":2,"issue_type":"feature","assignee":"spot","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:49Z","created_by":"Edward Langley","updated_at":"2026-04-14T07:47:43Z","closed_at":"2026-04-14T07:47:43Z","close_reason":"All 5 subtasks closed; artifacts verified present and tracked in git (README.md, docs/demo.gif, docs/demo.tape, docs/casts/*.cast).","dependencies":[{"issue_id":"improvise-yk4","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T23:37:37Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-ihv","type":"blocks","created_at":"2026-04-08T23:37:34Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-odx","type":"blocks","created_at":"2026-04-08T23:37:35Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-pby","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-44ad","title":"ColWidths is redundant under the label fold, and the test that would change is gotcha 33(c)'s evidence","description":"Found while closing improvise-hkt6 (PR #20, WU-K), tried, and DELIBERATELY REVERTED rather than landed -- filed so it is a decision rather than a rediscovery.\n\nhkt6(b) obliged grid_updates to diff `totals` against the LABEL-FOLDED cache rather than against the mirror (see gotcha 33). The symmetric move for `col_widths` looks obviously right and is not free: it BREAKS `a_client_on_a_socket_is_seeded_and_gets_its_edit_back`.\n\nWHY, and it is a real property rather than a brittle assertion: folding the labels RE-FITS the widths (gotcha 33(c) -- every arm that touches labels or widths re-fits, because a renderer zips widths with labels and a length mismatch indexes off the end). So under a fold-relative diff the ColWidths that a reframe currently carries becomes REDUNDANT and is suppressed. The client is correct either way; what changes is whether the reframe still SENDS the widths.\n\nTHE REASON IT WAS NOT SPENT IN PR #20: that assertion is gotcha 33(c)'s standing evidence -- it is the test that catches \"a late ColWidths put the mismatch straight back\". Rewriting it inside a PR whose subject was three totality gaps would have destroyed the evidence for a rule while fixing something else, which is exactly the shape improvise-q0vb was declined for on PR #12. Two right changes, different costs.\n\nTHE DECISION TO MAKE:\n * Leave widths diffed against the mirror. Costs one redundant ColWidths per reframe -- the same direction Dirty::of is deliberately sloppy in (gotcha 35: over-report pays one redundant Projection, under-report leaves a client wrong forever). Defensible and arguably correct on that principle alone.\n * Or move widths to the fold and REPLACE the socket assertion with one that still pins 33(c)'s property -- that a client never holds widths of a different length from its labels -- rather than one that pins the message count. That is the better test either way, and writing it is most of the work.\n\nPREFER THE SECOND ONLY IF the replacement test is written first and shown to fail against a deliberately late ColWidths. Otherwise prefer the first and say so in gotcha 33, so the asymmetry between totals and widths is STATED rather than discovered by the next person who notices it.\n","status":"open","priority":3,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:54:56Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:56Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-j7v6","title":"The typed-string classification lives in two crates, and its guard test asserts by hand","description":"Named by improvise-r4lo (PR #21) while deleting the OTHER duplication, and deliberately left rather than folded in -- r4lo was about two answers on the CLIENT side, and deleting Client::set_cell/set_cell_at settled that. This is the duplication one level out, and it crosses a crate boundary.\n\nwasm-client/src/socket.rs::typed classifies a user-typed string into a CellValue. So does improvise-command's commit_regular_cell_value, on the host's own key path. Two implementations of \"what does this string mean\", in two crates, with nothing holding them together.\n\nBOTH ARE LEGITIMATE, which is why this is not simply r4lo again. The browser has no App to ask and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; a host typing locally has an App and classifies there. The defect is that the RULE is written twice, not that it is applied in two places.\n\nTHE TEST THAT LOOKS LIKE A GUARD AND IS NOT: a_typed_string_is_classified_as_the_host_classifies_it states the obligation IN ITS NAME and does not discharge it -- it asserts three outcomes BY HAND rather than comparing against the host's function. So the two can drift and the test stays green, which is this repository's characteristic defect (doctrine(check_the_checker)): a test that passes while the thing it names is broken.\n\nTHE FIX: one classification function in improvise-protocol, which both crates already depend on, and both call sites calling it. Then the test either disappears (type identity replaces it) or becomes a real comparison. Note doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished) -- if the shared function lands and the hand-written three-outcome test survives beside it, the consolidation is not done.\n\nWATCH ONE THING WHILE MOVING IT: the classification must stay consistent with what the .improv format considers a number versus pipe-quoted text (gotcha 34, and format.rs::pipe_quote exists precisely so a text cell cannot read back as a number). A shared classifier that disagrees with the persistence layer would be a third answer wearing the badge of the first.\n","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-rucf.3","title":"Projection freshness for a source an external writer can change","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent.\n\nADR 001 governs concurrent EDITS. A live-backed model is read-only, so it has none and the basis check does not apply to it -- the guarantee is not voided, it is scoped to documents improvise owns, which is what it always meant.\n\nWHAT REMAINS IS A DIFFERENT AND SMALLER QUESTION, and this bead is it: an EXTERNAL WRITER changes what clients SEE. So a live-backed projection needs a freshness story -- polled, triggered, or explicitly stale-until-refreshed. That is projection freshness, not conflict resolution, and conflating the two is the trap.\n\nNOTE WHAT CHANGES ABOUT THE PROJECTION ARCHITECTURE: today's diff (ui/project.rs::grid_updates) compares a desired state derived from the App against each client's mirror. With a live source, \"the desired state\" involves latency -- a HashMap lookup becomes a query -- so the VIEWPORT-BOUNDED PROJECTION STOPS BEING AN OPTIMISATION AND BECOMES LOAD-BEARING. improvise-i5tk (narrow the TUI's GridCache to its viewport, benchmark first) is the related bead and its answer may be forced by this one.\n\nSHARED WITH THE SQLITE SOURCE CHILD: the RecordHash -\u003e record index becomes a cache needing invalidation rather than a structure maintained alongside writes. One invalidation story, not two.\n\nEXPLICITLY OUT OF SCOPE: anything resembling a lease, a lock, or a reservation. Those are ADR 001 layer 2 and they are about edits.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:35Z","dependencies":[{"issue_id":"improvise-rucf.3","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-rucf.2","title":"SQLite source implementation: aggregate pushdown over a user-supplied query, native-only and feature-gated","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent.\n\nThe SQLite implementation of the source trait: aggregate pushdown against a user-supplied query.\n\nWHICH ROWS: a user-supplied SELECT, with a bare table name as sugar over it. A query is strictly more useful (joins, filters, computed columns) and no harder to consume -- a result set is a result set.\n\nTYPE MAPPING, and it is BETTER INPUT THAN CSV rather than merely different: a SQLite column carries a declared type, where import/analyzer.rs::detect_date_format currently guesses a date by pattern-matching text. INTEGER/REAL are measure candidates, TEXT a category candidate, BLOB skip. NULL is ABSENCE, not the string \"null\" (improvise-ych). Dates may be TEXT, INTEGER (epoch) or REAL (julian) -- DETECT, DO NOT ASSUME. SQLite is dynamically typed per value, so both the declared-type and the sampled path are real and the classifier should take typed input where it has it.\n\nWHERE THE BOUNDARY GOES: forbidden(improvise_core, io). The source TRAIT belongs in improvise-core; this implementation belongs in improvise-io, FEATURE-GATED AND NATIVE-ONLY. libsqlite3-sys does not build for wasm32-unknown-unknown under this repo's nix cc-wrapper, and improvise-io is in the wasm bundle, so the wasm CI job must keep passing untouched. VERIFY THE GATE, do not assume it -- the improvise-9ix agent established the wasm failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating protects the browser bundle rather than working around an impossibility.\n\nNO WRITES, and not as a runtime refusal: per ADR 003 writes are not in the trait a query source implements, and a source projecting no CellId::Record makes a client write unconstructible.\n\nINTERNING AND IDENTITY: DataStore interns names and ADR 002 hashes the coordinate tuple. Both still work -- coordinates come from the query -- but the RecordHash -\u003e record index that resolves a CellId becomes A CACHE WITH AN INVALIDATION STORY rather than a structure maintained alongside writes. That invalidation is shared with the freshness child; do not solve it twice.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:32Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:34Z","dependencies":[{"issue_id":"improvise-rucf.2","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -249,7 +251,7 @@ {"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:38Z","started_at":"2026-09-20T05:32:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"closed","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:44Z","started_at":"2026-09-20T05:32:38Z","closed_at":"2026-09-20T05:54:44Z","close_reason":"Closed by PR #20 (WU-K). All three gaps fixed, each fault-injected.\n\n(a) matching_positions total-ised via split_first. The fix is NOT vec![]: an empty partial constrains NOTHING, so EVERY cell matches -- answering \"none\" one level down would have made deleting a caller's fast path silently flip \"all\" to \"none\". The callers' partial.is_empty() branches are fast paths, not guards, and the test asserts the two doors agree.\n\n(b) invalidate_relabelled now blanks a relabelled COLUMN's total. Rows are the negative arm and stay untouched -- a total spans every row, so no one row frames it, and that arm is what stops \"blank on any relabel\" passing.\n\n(c) mark_features uses .get_mut().expect(...), matching expr_text.\n\nTHE THING THE TICKET DID NOT PREDICT, and the reason this bead was not independent of its own framing: (b) was filed as \"unreachable in practice, fix it anyway because gotcha 33(b) exists to stop apply trusting producers\". Right about the rule, wrong about the cost. Adding a blanking rule on the CONSUMER side changes what every PRODUCER owes, and grid_updates did not pay it: it diffed mirror.totals against desired.totals, so a column renamed while its total stayed the same NUMBER emitted ColLabels and no Totals -- the client blanked its own total and project_diff's self.grid = desired left the server believing it was still there. A client cache wrong forever, gotcha 35's exact failure direction. Found by Copilot. Fixed by diffing totals against the label-folded frame.\n\nWHEN YOU MAKE apply STOP TRUSTING A PRODUCER, GO READ THE PRODUCER.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-g6s1","title":"WizardStep::Preview has no characterization fixture","description":"The one import-wizard step improvise-764 could not pin. It needs a document that parses to ZERO RECORDS without needing path selection, and neither existing fixture reaches it.\n\nSmall, and worth doing while the WizardCache shape is fresh: every other step is pinned at buffer level, so Preview is the single arm where a change to wizard_cache could alter what is drawn with nothing failing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-k69y","title":"[bug] draw_status reads the document's active view, not the client's","description":"src/draw.rs::draw_status uses app.model_state.workbook.active_view; ViewState::active_view is the CLIENT copy (gotcha 29).\n\nHARMLESS TODAY -- they are in lockstep and the_two_active_view_names_never_diverge guards it. WRONG THE DAY improvise-jytm LANDS, which is exactly the ticket that makes them diverge on purpose (the document keeps the DEFAULT, the client is authoritative). On that day the TUI badge names the wrong view.\n\nOne line. Left unfixed because it is a behaviour change and belongs with jytm rather than ahead of it. Note the wasm client already reads the correct one (self.view.active_view), so the TUI is the outlier.","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-poia","title":"The records-mode + column has no projected identity","description":"From the ADR 002 spine landing (2026-09-19). In records mode the trailing + (NewMeasure) column creates a new measure on the row (gotcha 1). It carries no CellId, so creating a measure over the wire has no address.\n\nThat is correct for now rather than an oversight -- a new measure is not a write to an existing cell, so it is not a SetCell and should not borrow one identity. But it means a browser client cannot do something the TUI can.\n\nDecide what it should be: a distinct command carrying the record surrogate plus the new measure name (derived from the effect that does it today, per doctrine(derive_the_enum_from_the_effects)), or a client-side flow that adds the measure and then writes the cell as two commands. The second is probably right -- it composes from operations that already exist -- but check whether the TUI treats it as one atomic action first, because splitting an atomic action across two commands is how a half-applied state appears.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:18:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} From 8173b87723e66a708911937913678193234dfd2e Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:06:13 -0700 Subject: [PATCH 52/72] fix(view): a virtual on an axis is a configuration, not a vacancy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-2zq step 1. Naming a category in records mode ejected the user from the grid they were standing in, and it is a live bug independent of the inline-column work it blocks. **What the old rule was.** `View::on_category_added` seats a new regular category on the first axis no *regular* category holds — Row, then Column, then Page — and, to get there, **bumped any virtual off that axis** on the assumption that a virtual sitting on Row or Column was a placeholder. It never is, and the assumption cost two live bugs: * `_Index` on Row together with `_Dim` on Column *is* records mode. The only production writers of those axes are `ToggleRecordsMode` and `DrillIntoCell`, so a virtual is on Row only when the view is a records or drill view. Bumping `_Index` therefore turned `N`, a name, `Enter` into a silent return to a pivot. * `_Measure` on Column is an ordinary user configuration — it is what `web/demo.improv` ships — and adding a second regular category kicked it off the axis, making every measure vanish from the grid. **What the new rule is, including for a non-records view**, since this changes how *any* view seats a category and not just the records one: * A virtual is **never moved**. That is the whole fix; both bugs above are the same mistake pointed at different virtuals. * A **records view** seats the new category on `Axis::None`. That is where records mode keeps every other regular category, and `GridLayout::records_columns` filters on `kind.is_regular()` and never looks at an axis, so the column appears immediately either way — `None` is the answer that also leaves the view a records view. * A **pivot view** is unchanged in the case that matters: Row, then Column, then Page, counting only regular occupancy. The one observable difference is that a new category now *joins* a virtual already on the target axis, nesting a header, instead of evicting it. `View::is_records_layout()` is new and is the **one** definition of "is this a records view" — `GridLayout` now asks it instead of re-deriving the same `_Index`-on-Row/`_Dim`-on-Column test, because two answers to one question is the disease this repo keeps curing (improvise-edp) and `on_category_added` had to ask it too. **Evidence, and the probe that shaped the fix.** Before designing it I broke `on_category_added` in two directions, as the method that has been paying off all wave. Removing the virtual bump entirely fails **zero** tests; seating every category on `None` fails **twelve**. So the auto-assignment is load-bearing and well covered, while the bump was completely unguarded — which is how it came to be wrong about two different virtuals with nothing noticing. `naming_a_category_in_records_mode_does_not_eject_the_user` drives `R`, `N`, a name, `Enter` through real keypresses, because that is how the bug was found. Two unit tests cover the rule directly, and `a_pivot_view_still_seats_categories_row_then_column_then_page` is the negative arm — without it "always seat on None" passes everything else. All four were run against the restored old rule: the three positives fail and the negative arm passes, which is exactly the split being claimed. Note for improvise-9mo9 (gotcha 24, another agent's): this does not change whether `AddCategory` renumbers the grid — it still does, and it still does not call `reset_cursor`. It changes only which axis the new category lands on. `AddCategory` keeps its `LayoutGeometry` kind, which is right for the same reason as before: `Workbook::add_category` rewrites every view's axis map. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-command/src/ui/app.rs | 64 +++++++++ crates/improvise-core/src/view/layout.rs | 11 +- crates/improvise-core/src/view/types.rs | 166 +++++++++++++++++------ 3 files changed, 194 insertions(+), 47 deletions(-) diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index e75c67ac..15b8a52f 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1391,6 +1391,70 @@ mod tests { ); } + /// improvise-2zq step 1: naming a category in records mode used to eject + /// the user from the grid they were standing in. + /// + /// Records mode is `_Index` on Row and `_Dim` on Column, with every + /// regular category on `Axis::None`. `View::on_category_added` seated a + /// new regular category on the first axis no *regular* category held — + /// Row — and **bumped the virtual out of the way** to get there. So `N`, + /// a name, `Enter` moved `_Index` off Row, records detection stopped + /// matching, and the grid silently became a pivot. + /// + /// Driven through the real keys rather than through `on_category_added`, + /// because that is how the bug was found and it is what makes this + /// evidence rather than a restatement of the fix. + #[test] + fn naming_a_category_in_records_mode_does_not_eject_the_user() { + let mut wb = Workbook::new("T"); + wb.add_category("Region").unwrap(); + wb.add_item("Region", "East").unwrap(); + let mut app = App::new(wb, None); + app.handle_key(KeyEvent::new(KeyCode::Char('R'), KeyModifiers::NONE)) + .unwrap(); + assert!(app.layout.is_records_mode(), "setup: in records mode"); + + app.handle_key(KeyEvent::new(KeyCode::Char('N'), KeyModifiers::NONE)) + .unwrap(); + for c in "Product".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + app.rebuild_layout(); + + assert!( + app.model_state.workbook.model.category("Product").is_some(), + "setup: the category really was created" + ); + assert!( + app.layout.is_records_mode(), + "naming a category must not leave records mode; axes are {:?}", + app.model_state + .workbook + .active_view() + .category_axes + .iter() + .collect::>() + ); + assert_eq!( + app.model_state.workbook.active_view().axis_of("Product"), + crate::view::Axis::None, + "records mode keeps every regular category on None, and \ + `records_columns` picks it up as a column from `is_regular` \ + alone — never from an axis" + ); + let cols: Vec = (0..app.layout.col_count()) + .map(|i| app.layout.col_label(i)) + .collect(); + assert!( + cols.iter().any(|c| c == "Product"), + "the new category must appear as a records column immediately, \ + got {cols:?}" + ); + } + /// improvise-6sq: `i` and `o` want opposite behavior on an empty records /// grid, which is why `edit-or-drill` and `enter-edit-at-cursor` cannot /// be merged into one command. Entering records mode puts every regular diff --git a/crates/improvise-core/src/view/layout.rs b/crates/improvise-core/src/view/layout.rs index 9cf5c0ea..138b9af7 100644 --- a/crates/improvise-core/src/view/layout.rs +++ b/crates/improvise-core/src/view/layout.rs @@ -1,7 +1,7 @@ use std::rc::Rc; use crate::model::Model; -use crate::model::category::{CategoryKind, DIM_CATEGORY, INDEX_CATEGORY, MEASURE_CATEGORY}; +use crate::model::category::{DIM_CATEGORY, INDEX_CATEGORY, MEASURE_CATEGORY}; use crate::model::cell::{CellKey, CellValue}; use crate::view::{Axis, View}; @@ -241,11 +241,10 @@ impl GridLayout { }) .collect(); - // Detect records mode: _Index on Row and _Dim on Col - let is_records_mode = row_cats.iter().any(|c| CategoryKind::of_name(c).is_index()) - && col_cats.iter().any(|c| CategoryKind::of_name(c).is_dim()); - - if is_records_mode { + // Records mode is a property of the view's axes, and `View` owns the + // one definition of it (improvise-2zq): `on_category_added` must ask + // the same question, and two derivations would drift. + if view.is_records_layout() { Self::build_records_mode(model, view, page_coords, none_cats) } else { let row_items = cross_product(model, view, &row_cats); diff --git a/crates/improvise-core/src/view/types.rs b/crates/improvise-core/src/view/types.rs index d927d637..307b4dee 100644 --- a/crates/improvise-core/src/view/types.rs +++ b/crates/improvise-core/src/view/types.rs @@ -3,6 +3,7 @@ use serde::{Deserialize, Serialize}; use std::collections::{HashMap, HashSet}; use super::axis::Axis; +use crate::model::category::CategoryKind; fn default_prune() -> bool { true @@ -56,52 +57,70 @@ impl View { } } + /// True when this view's axes spell **records mode**: `_Index` on Row and + /// `_Dim` on Column. + /// + /// The one definition — `GridLayout` asks this rather than re-deriving it, + /// because two answers to "is this a records view" is the disease this + /// codebase keeps curing (improvise-edp). It is a property of the *view's + /// axes* alone, which is why it lives here and not on the layout. + pub fn is_records_layout(&self) -> bool { + self.categories_on(Axis::Row) + .iter() + .any(|c| CategoryKind::of_name(c).is_index()) + && self + .categories_on(Axis::Column) + .iter() + .any(|c| CategoryKind::of_name(c).is_dim()) + } + + /// Seat a newly added category on an axis. + /// + /// Virtual (underscore) categories default to `Axis::None`. A regular + /// category takes the first slot no *regular* category holds — Row, then + /// Column, then Page. + /// + /// # A virtual on an axis is a configuration, not a vacancy (improvise-2zq) + /// + /// This used to **bump** a virtual off Row or Column to make room, on the + /// assumption that a virtual sitting there was a placeholder. It never is, + /// and the assumption cost two live bugs: + /// + /// - `_Index` on Row together with `_Dim` on Column *is* records mode, and + /// the only production writers of those axes are the records and drill + /// setups. Bumping `_Index` therefore **ejected the user from the grid + /// they were standing in**: naming a category with `N` in records mode + /// silently returned them to a pivot. + /// - `_Measure` on Column is an ordinary user configuration (it is what + /// `web/demo.improv` ships). Bumping it made the measures vanish from + /// the grid the moment a second regular category was added. + /// + /// So a virtual is never moved. A records view seats the new category on + /// `Axis::None`, which is where records mode keeps **every** regular + /// category and where `GridLayout::records_columns` — which filters on + /// `kind.is_regular()` and never looks at an axis — picks it up as a + /// column immediately. Elsewhere the new category simply joins whatever + /// virtual is already there, which nests a header rather than destroying + /// one. + /// + /// The Row → Column → Page progression for regular categories is + /// unchanged, and `a_pivot_view_still_seats_categories_row_then_column_then_page` + /// is the negative arm that says so — without it, "always seat on None" + /// would pass every assertion about records mode. pub fn on_category_added(&mut self, cat_name: &str) { if !self.category_axes.contains_key(cat_name) { - // Virtual/underscore categories default to Axis::None. - // Regular categories auto-assign: first → Row, second → Column, rest → Page. - // If a virtual currently holds Row or Column and a regular category needs - // the slot, bump the virtual to None. - let axis = if cat_name.starts_with('_') { + let axis = if cat_name.starts_with('_') || self.is_records_layout() { Axis::None } else { - let regular_rows: Vec = self - .categories_on(Axis::Row) - .into_iter() - .filter(|c| !c.starts_with('_')) - .map(String::from) - .collect(); - let regular_cols: Vec = self - .categories_on(Axis::Column) - .into_iter() - .filter(|c| !c.starts_with('_')) - .map(String::from) - .collect(); - if regular_rows.is_empty() { - // Bump any virtual on Row to None - let bump: Vec = self - .categories_on(Axis::Row) + let holds_a_regular = |axis: Axis| { + self.categories_on(axis) .into_iter() - .filter(|c| c.starts_with('_')) - .map(String::from) - .collect(); - for c in bump { - self.category_axes.insert(c, Axis::None); - } - Axis::Row - } else if regular_cols.is_empty() { - let bump: Vec = self - .categories_on(Axis::Column) - .into_iter() - .filter(|c| c.starts_with('_')) - .map(String::from) - .collect(); - for c in bump { - self.category_axes.insert(c, Axis::None); - } - Axis::Column - } else { - Axis::Page + .any(|c| !c.starts_with('_')) + }; + match (holds_a_regular(Axis::Row), holds_a_regular(Axis::Column)) { + (false, _) => Axis::Row, + (true, false) => Axis::Column, + (true, true) => Axis::Page, } }; self.category_axes.insert(cat_name.to_string(), axis); @@ -278,6 +297,71 @@ mod tests { v } + /// improvise-2zq, the **negative arm**. The fix makes a records view seat + /// a new category on `None`; without this, "always seat on None" would + /// pass every assertion about records mode while quietly deleting the + /// auto-assignment `on_category_added` exists for. + /// + /// Probing confirmed this arm is the load-bearing one: seating every + /// category on `None` fails 12 tests, while removing the virtual bump + /// fails none at all — which is how the bump came to be wrong in two + /// directions with nothing noticing. + #[test] + fn a_pivot_view_still_seats_categories_row_then_column_then_page() { + let v = view_with_cats(&["A", "B", "C", "D"]); + assert_eq!(v.axis_of("A"), Axis::Row, "first regular → Row"); + assert_eq!(v.axis_of("B"), Axis::Column, "second regular → Column"); + assert_eq!(v.axis_of("C"), Axis::Page, "the rest → Page"); + assert_eq!(v.axis_of("D"), Axis::Page); + } + + /// A records view seats a new regular category on `None`, where records + /// mode keeps all of them — and, crucially, leaves `_Index`/`_Dim` alone, + /// so the view is still a records view afterwards. + #[test] + fn a_records_view_seats_a_new_category_on_none_and_stays_a_records_view() { + let mut v = view_with_cats(&["_Index", "_Dim", "Region"]); + v.set_axis("_Index", Axis::Row); + v.set_axis("_Dim", Axis::Column); + v.set_axis("Region", Axis::None); + assert!(v.is_records_layout(), "setup"); + + v.on_category_added("Product"); + assert_eq!(v.axis_of("Product"), Axis::None); + assert_eq!(v.axis_of("_Index"), Axis::Row, "_Index must not be bumped"); + assert_eq!(v.axis_of("_Dim"), Axis::Column, "_Dim must not be bumped"); + assert!(v.is_records_layout(), "still a records view"); + } + + /// The second thing the bump broke, and the reason the fix is about + /// virtuals in general rather than about records mode: `_Measure` on + /// Column is an ordinary configuration — it is what `web/demo.improv` + /// ships — and adding a second regular category used to kick it off the + /// axis, making every measure vanish from the grid. + /// + /// This view is deliberately **not** a records layout, so it exercises + /// the pivot path rather than the early return above. + #[test] + fn a_measure_on_column_is_not_bumped_off_it() { + let mut v = view_with_cats(&["_Measure", "Region"]); + v.set_axis("_Measure", Axis::Column); + assert_eq!(v.axis_of("Region"), Axis::Row, "setup"); + assert!(!v.is_records_layout(), "setup: a pivot view, not records"); + + v.on_category_added("Product"); + assert_eq!( + v.axis_of("_Measure"), + Axis::Column, + "the user's measure axis must survive a new category" + ); + assert_eq!( + v.axis_of("Product"), + Axis::Column, + "the new category joins Column, nesting a header rather than \ + destroying one — Row already holds a regular category" + ); + } + #[test] fn first_category_assigned_to_row() { let v = view_with_cats(&["Region"]); From ae199d431a55da253d47ac3a57bf0365fef724e0 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:08:24 -0700 Subject: [PATCH 53/72] docs: an address check cannot see a cross-site socket Copilot's fifth pass, and the first finding in this PR that is a genuine security hole rather than a wrong sentence. Verified: there is NO reference to Origin anywhere in crates/improvise-ws-server -- the server accepts every handshake. A WebSocket is EXEMPT FROM THE SAME-ORIGIN POLICY; there is no CORS preflight on a handshake. So any hostile page an allowed user visits can open a socket to the endpoint FROM THAT USER'S OWN ALLOWED ADDRESS and read and edit the document. The IP allow-list I recommended last round is precisely the control that cannot see this, because by then the request really does come from an allowed client. Only the Origin header separates them. This is the same shape as the previous three rounds and it is worth naming as such: I reasoned about WHO MAY REACH the endpoint and not about WHAT MAY DRIVE the browser that reaches it -- one more conclusion carried without its reasoning. "Who may connect" is two questions, and the recipe answered one. So: * the prose states the two questions and says an Origin allow-list is needed IN ADDITION TO, never instead of, whichever control is chosen; * the Caddyfile carries both checks, with the second commented as explicitly not redundant with the first; * the `header` matcher syntax is checked against caddyserver.com/docs/ caddyfile/matchers (`header [ ...]`, negatable inside a named matcher) rather than written from memory -- the round-3 rule; * and the LOCAL pairing paragraph is corrected too, because the same fact applies there and I had called it unreservedly fine: loopback is not a boundary a browser enforces, so any page open in that browser can reach ws://127.0.0.1:8787 while the server runs. The server-side half -- ws-server should check Origin itself, since a proxy is not present in the local case at all -- is reported for filing rather than built here. It is a change to a crate this unit does not own. 1193 passing, unchanged; no code touched. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c81d9745..bc57a846 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,13 @@ optimising for: `http://localhost` is not a secure-context problem, so nothing below is needed to develop, to demo locally, or to share a TUI-hosted document with another tab on the same machine. +One caveat even there, for the reason spelled out below: a WebSocket is exempt +from the same-origin policy, and loopback is not a boundary a browser enforces +— so while the server is running, **any** page open in that browser can connect +to `ws://127.0.0.1:8787` and read or edit the document. Nothing in the server +checks `Origin`. That is tolerable for a local session you start and stop, and +it is the reason not to leave one running unattended. + ### Sharing a TUI-hosted document with a remote browser Reaching `improvise-ws-server` from the **deployed** https page means @@ -267,10 +274,21 @@ the page, but the selection UI browsers show is a top-level-navigation affordance, and whether a cross-origin socket handshake gets it is **not something verified here**. +**And "who may connect" is two questions, not one.** A WebSocket is **exempt +from the same-origin policy** — there is no CORS preflight on a handshake — so +any hostile page an allowed user visits can open a socket to your endpoint +*from that user's own allowed address*, and read and write the document. An +address check cannot see that, because by then the request genuinely does come +from an allowed client. Only the **`Origin` header** distinguishes them, and +`improvise-ws-server` **never looks at it** — it accepts every handshake. So a +proxy needs an Origin allow-list **in addition to**, never instead of, +whichever control below you pick. The worked example carries both. + So the honest controls are the ones that need **no credential from the page**: - **An IP allow-list in the proxy** — the cheapest, and the worked example - below. It is about where the client is, so the page sends nothing. + below. It is about where the client is, so the page sends nothing. On its + own it answers only the first of the two questions above. - **A private network** (Tailscale, WireGuard). The peer is only reachable by machines already on it; again the page sends nothing. - **An identity-aware tunnel** (`cloudflared` with Cloudflare Access) — the @@ -303,6 +321,14 @@ improvise.example.com { @denied not remote_ip 203.0.113.4 198.51.100.0/24 abort @denied + # ALSO not optional, and it is NOT redundant with the address check. + # A WebSocket is exempt from the same-origin policy, so any hostile page + # an allowed user visits can open this socket FROM THAT ALLOWED ADDRESS + # and read and edit the document. The address check cannot see that; only + # the Origin can. Name the exact origins the page is served from. + @badorigin not header Origin https://fiddlerwoaroof.github.io + abort @badorigin + reverse_proxy 127.0.0.1:8787 } ``` From 61984ec46a40086a325ff38338b7334c0edad118 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:08:35 -0700 Subject: [PATCH 54/72] docs(ws-server): its usage example named the document this branch deleted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot, PR #22, third pass — a dangling reference, and one my own deletion created, so it is mine to clean up (AGENTS.md §3) even though the file is outside this unit's stated boundary. Flagged in the hand-back. `improvise-ws-server`'s module doc opens with a copy-pasteable command line and a URL path derived from the file stem, all three naming `bank-info.improv`. Pointed at `examples/demo.improv` instead — a document that exists, and one `tests/shipped_documents_load.rs` now proves loads, so the example cannot go stale the same way without a test saying so. Doc comment only; no code, no behaviour, no test moved. Still mentioning the file, deliberately: the two test doc-comments that explain why it was deleted. Not touched, and not mine to touch: `roadmap.org` line 194 and the `improvise-1ey` / `improvise-cr3` issue bodies in `.beads/` name it as the demo document to point the browser at. Reported rather than edited. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-ws-server/src/main.rs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/crates/improvise-ws-server/src/main.rs b/crates/improvise-ws-server/src/main.rs index 8a51710f..279f109e 100644 --- a/crates/improvise-ws-server/src/main.rs +++ b/crates/improvise-ws-server/src/main.rs @@ -1,12 +1,12 @@ //! `improvise-ws-server` — serve improvise documents over a websocket. //! //! ```text -//! improvise-ws-server bank-info.improv --port 8787 -//! improvise-ws-server bank-info.improv q3.improv # two documents +//! improvise-ws-server examples/demo.improv --port 8787 +//! improvise-ws-server examples/demo.improv q3.improv # two documents //! ``` //! //! Each file becomes a document named after its file stem, and a client -//! names the one it wants in the URL path: `ws://127.0.0.1:8787/bank-info`. +//! names the one it wants in the URL path: `ws://127.0.0.1:8787/demo`. //! With exactly one document, `ws://127.0.0.1:8787/` reaches it. //! //! Every document is opened here, at startup. A client cannot ask the server From 6a45edcfc9c851c53db08d96430440549867b21c Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:08:59 -0700 Subject: [PATCH 55/72] fix(fit): six more Copilot findings, and a false premise of my own MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The second review round. All six correct; five fixed, one documented and filed. **My fixture's determinism premise was false, and the comment asserting it was the worst part.** Both `document()` fixtures left one regular category with a single item and claimed the constant one-hot had a degenerate `(1, 1)` range that `simplify` would prune, so the champion "could not" reach an unresolvable feature. `data.rs::build_categorical` pushes `(0.0, 1.0)` for **every** one-hot whatever the column holds, so `simplify` cannot know it is constant — the success tests were passing on seed luck, and three comments, the PR body and a proposed repo-map gotcha all repeated the claim. Every category in both fixtures is now a `Label`, so the only feature is the numeric measure `Cost`, which is also the only shape a fit can currently store. **Resolving a name is not the same question as meaning the right column.** A numeric category called `Cost` encodes to a feature called `Cost`; with a `_Measure` item of the same name the emitted `Expr::Ref("Cost")` resolves cleanly — to the measure — and the stored rule reads a different column from the one it learned. `fitted_formula` deliberately asks only whether a name can be *spelled*, which is right for that check and wrong as the only one. So the question is now also asked of `preprocess`'s own plan: a used feature is accepted only where `Dataset::columns` says it is a raw `Numeric` **measure** column, which rules out one-hots, `.missing` flags, date parts and category coordinates by role rather than by spelling. A document naming the same thing as both a category and a measure is refused outright. Deliberately *not* a reconstruction of the expansion order index-by-index — that is improvise-4zl7's job and its ticket warns against a second implementation. **The width guard was on the wrong side of the cost.** It ran after `preprocess` had already materialised the rows × rows one-hot matrix it was meant to prevent. `encoded_width` now counts what the expansion *would* produce, from the raw table, without producing it. **`importance_ranking` misreported small importances.** `.max(1e-9)` on the divisor is right for the only input `fit_csv` ever had and wrong as a contract: a lone importance of `1e-10` divided by the floor and reported 10%. Importances this small are ordinary — error differences over a standardized target — so the zero case returns early and the shares sum to 100% at every scale. **A name can contain a newline.** `.improv` pipe quoting makes it legal, so a document could hold a measure whose name split the TUI status bar in two. `status_line` flattens the composed line once, rather than sanitising each interpolated part and forgetting one. **Declined and documented on `Quality`:** preprocessing runs over all rows before `cv_regress` picks folds, so imputation means and categorical vocabularies leak from the validation fold and `Measured` is slightly optimistic. Bounded — a fit over complete numeric measures, the only shape that currently stores, leaks nothing — and closing it means preprocessing per training fold, which `cv_regress` cannot express because it takes rows that are already encoded. That is a `gp-tree` API change, not a line here. 1225 passing (+5); clippy, fmt and the wasm gate clean; the CLI's output still byte-identical, which is the check that gates the `importance_ranking` change. Co-Authored-By: Claude Opus 5 (1M context) --- crates/gp-tree/src/fit.rs | 38 +- .../improvise-command/src/command/cmd/fit.rs | 19 +- crates/improvise-gp/src/fit.rs | 330 +++++++++++++++--- crates/improvise-gp/src/lib.rs | 17 +- 4 files changed, 347 insertions(+), 57 deletions(-) diff --git a/crates/gp-tree/src/fit.rs b/crates/gp-tree/src/fit.rs index bdb1a7eb..148d66d3 100644 --- a/crates/gp-tree/src/fit.rs +++ b/crates/gp-tree/src/fit.rs @@ -335,8 +335,20 @@ pub fn regression_quality(pooled: &[(f32, f32)]) -> RegressionQuality { /// 40% of what the model knows". A total of zero — every feature irrelevant, or /// a tree that is one leaf — gives an empty ranking rather than a division by /// zero. +/// +/// That zero case is answered by **returning early**, not by flooring the +/// divisor. `fit_csv` used `.max(1e-9)`, which is right for the only input it +/// ever had and wrong as a contract: a lone importance of `1e-10` then divides +/// by the floor rather than by itself and reports **10%**, where the one +/// feature the model uses carries all of it (Copilot on PR #25). Importances +/// this small are ordinary — they are error differences over a standardized +/// target — so the shares have to sum to 100% at every scale, not only at the +/// CLI's. pub fn importance_ranking(imp: &[f32]) -> Vec<(usize, f32)> { - let total: f32 = imp.iter().sum::().max(1e-9); + let total: f32 = imp.iter().sum(); + if total <= 0.0 { + return Vec::new(); + } let mut order: Vec = (0..imp.len()).collect(); order.sort_by(|&a, &b| { imp[b] @@ -592,6 +604,30 @@ mod quality_tests { assert!(importance_ranking(&[0.0, 0.0]).is_empty()); } + /// Copilot on PR #25. The shares must sum to 100% at **every** scale, and + /// `fit_csv`'s `.max(1e-9)` floor made that true only above the floor: one + /// feature carrying an importance of `1e-10` carries all of what the model + /// knows, and used to be reported at 10%. Importances this small are + /// ordinary — they are error differences over a *standardized* target. + #[test] + fn a_tiny_lone_importance_is_still_the_whole_of_it() { + let ranked = importance_ranking(&[1e-10]); + assert_eq!(ranked.len(), 1); + assert!((ranked[0].1 - 100.0).abs() < 1e-3, "{ranked:?}"); + } + + /// The general form of the same claim, across four orders of magnitude. + #[test] + fn the_shares_sum_to_a_hundred_at_any_scale() { + for scale in [1e-12f32, 1e-6, 1.0, 1e6] { + let total: f32 = importance_ranking(&[scale, 3.0 * scale]) + .iter() + .map(|&(_, share)| share) + .sum(); + assert!((total - 100.0).abs() < 1e-2, "at {scale}: {total}"); + } + } + /// `denormalize` is the inverse of the affine map `standardize` applied, /// so a tree fitted on standardized targets predicts in the user's own /// units — which is the only form that can be written into a document. diff --git a/crates/improvise-command/src/command/cmd/fit.rs b/crates/improvise-command/src/command/cmd/fit.rs index 973c537a..5c650fec 100644 --- a/crates/improvise-command/src/command/cmd/fit.rs +++ b/crates/improvise-command/src/command/cmd/fit.rs @@ -168,18 +168,23 @@ mod tests { const TAG: &str = "Tag"; /// A document a tree can learn exactly — `Score` steps at `Cost` 50 — in - /// the units a real document uses, and with `Region` single-valued so its - /// one-hot has the degenerate range `(1, 1)` that `simplify` prunes. The - /// champion therefore cannot reach an unresolvable feature and the success - /// path is deterministic rather than seed-dependent; `Tag` is a `Label` - /// category, which `dataset` keeps out of the features while still making - /// the twelve records distinct. See `improvise_gp::fit`'s own fixture. + /// the units a real document uses. + /// + /// **Every category is a `Label`**, so the only feature is the numeric + /// measure `Cost` and the champion cannot reach an unresolvable one. An + /// earlier version left `Region` regular with one item and claimed + /// `simplify` would prune the constant one-hot; it does not, because + /// `build_categorical` records `(0.0, 1.0)` for every one-hot whatever the + /// column holds — so these tests were passing on seed luck (Copilot on + /// PR #25). See `improvise_gp::fit`'s own fixture for the same correction. fn workbook() -> crate::workbook::Workbook { let mut wb = crate::workbook::Workbook::new("sales"); wb.add_category(REGION).expect("first category"); wb.add_item(REGION, "North").expect("Region exists"); wb.add_category(TAG).expect("second category"); - wb.model.category_mut(TAG).expect("just added").kind = CategoryKind::Label; + for cat in [REGION, TAG] { + wb.model.category_mut(cat).expect("just added").kind = CategoryKind::Label; + } for i in 0..12u32 { wb.add_item(TAG, format!("r{i}")).expect("Tag exists"); } diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index 6020311e..f32b9a1b 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -45,7 +45,7 @@ //! so the two cannot be confused, and a host that wants the numbers has to name //! the case where there are none. -use gp_tree::data::{Overrides, Target}; +use gp_tree::data::{Overrides, Target, preprocess}; use gp_tree::fit::{ FitConfig, RegressionQuality, cv_regress, denormalize, fit, importance_ranking, mse, permutation_importance, regression_quality, standardize, @@ -55,7 +55,42 @@ use improvise_core::model::Model; use improvise_core::view::View; use improvise_formula::Formula; -use crate::dataset::dataset; +use crate::dataset::{FeatureTable, feature_table}; + +/// What `preprocess` *would* expand this table to, counted without expanding +/// it. +/// +/// Deliberately an over-estimate and deliberately cheap: one column per +/// distinct string value, which is what `build_categorical` emits, plus one for +/// a column `preprocess` may read as numeric or as a date. It never allocates +/// the matrix, which is the whole point — the width has to be refused *before* +/// a rows × rows one-hot is materialised, not after. +/// +/// It is not a second implementation of the expansion, and must not become +/// one: it answers "roughly how wide" for a budget check, where +/// `Dataset::feature_names` answers "exactly which" for everything else. +fn encoded_width(table: &FeatureTable) -> usize { + (0..table.header.len().saturating_sub(1)) + .map(|col| { + table + .rows + .iter() + .filter_map(|row| row.get(col)) + .collect::>() + .len() + .max(1) + }) + .sum() +} + +/// The same text with every CR and LF turned into a space, so a name a +/// document is allowed to hold cannot split a one-line status bar. +fn one_line(text: String) -> String { + match text.contains(['\r', '\n']) { + false => text, + true => text.replace(['\r', '\n'], " "), + } +} /// The widest encoded feature matrix `fit_measure` will search synchronously. /// @@ -121,6 +156,21 @@ impl Default for FitRequest { /// them one: `Quality::Measured` carrying an all-zero [`RegressionQuality`] /// means "this tree is no better than the mean", which is the loudest thing a /// fit can say, and it must not be reachable by simply not having asked. +/// # A known optimism in `Measured`, filed rather than fixed +/// +/// Preprocessing runs over **all** rows before `cv_regress` picks its folds, so +/// an imputation mean and a categorical vocabulary are learned partly from the +/// rows they are then validated against (Copilot on PR #25). The held-out error +/// is therefore a little better than truly held-out wherever a *used* feature +/// had a missing cell. +/// +/// Two things bound it, and neither dismisses it. Only imputed numerics and +/// categorical vocabularies leak — a fit over complete numeric measures, which +/// is the only shape `fit_measure` currently *stores*, leaks nothing. And +/// closing it means preprocessing per training fold and encoding each +/// validation fold with that fold's plan, which `gp_tree::fit::cv_regress` +/// cannot express: it takes rows that are already encoded. That is an API +/// change in `gp-tree`, not a line here. #[derive(Clone, Copy, Debug, PartialEq)] pub enum Quality { /// Pooled `k`-fold cross-validation against the predict-the-mean baseline. @@ -196,11 +246,19 @@ impl FitOutcome { } else { format!(" \u{2014} {}", self.warnings.join("; ")) }; - format!( + // One line, and it has to be one line *by construction* rather than by + // the interpolated parts happening not to contain a newline. They can: + // pipe quoting makes `\n` legal inside a category or item name (the + // `.improv` grammar spells the escape), so a document can name a + // measure that splits the TUI status bar in two (Copilot on PR #25). + // The target, the feature names and `preprocess`'s warnings all reach + // this string, so the whole of it is flattened once at the end rather + // than each part being sanitised and one being forgotten. + one_line(format!( "{} = fitted tree: {quality}; {leans}; seed {:#x}{tail}", self.formula.target, self.seed() - ) + )) } } @@ -268,18 +326,40 @@ pub fn fit_measure( "{into:?} is already a measure; a formula on it would replace what is there. \ Name a measure that does not exist yet" ); - let data = dataset(model, view, target, &Overrides::default())?; + let table = feature_table(model, view, target)?; + // The width guard runs on the RAW table, before `preprocess` materialises + // anything. Checking it afterwards was the earlier version and it guards + // the wrong side of the cost: a category with a distinct value per record + // allocates a rows x rows one-hot matrix *during* preprocessing, so the + // refusal arrived after the memory and the O(rows^2) work had already been + // spent (Copilot on PR #25). `encoded_width` counts what the expansion + // would produce without producing it. + // + // Wide is a *freeze*: permutation importance is O(repeats x rows x + // features) evaluations on the thread that draws the next frame, and the + // bound this module's header quotes was measured on a document whose + // row-identity category is a `Label`, so it does not cover this shape. + // `Label` is the fix a user wants, and the message says so. + let width = encoded_width(&table); + anyhow::ensure!( + width <= MAX_FEATURES, + "fitting {target:?} would encode {width} features from {} records, which is too wide \ + to search on the UI thread. A category with a distinct value per record becomes one \ + column per record — mark such a category as a Label so it is not a feature", + table.rows.len(), + ); + + let feature_columns = table.header.len() - 1; + let category_columns = model.regular_category_names().len(); + let mut data = preprocess(&table.header, &table.rows, &Overrides::default()); + data.warnings.splice(0..0, table.warnings); let rows = regression_rows(data.target, target)?; - // Two guards on the *encoded* width, which is a different number from the - // one `feature_table` checked: `preprocess` both drops columns (entirely - // missing) and multiplies them (one-hot). - // - // Empty is a **panic**, not a poor fit: `random_tree` draws `rng.below(0)`, - // which is `0` by `SplitMix64`'s own answer to the empty range, and - // `sample_thresh` then indexes `spec.ranges[0]`. So a document whose only - // candidate feature is entirely missing crashes the command for any seed - // that does not open on a leaf. + // Empty is a **panic**, not a poor fit: `preprocess` drops an entirely + // missing column, and `random_tree` then draws `rng.below(0)`, which is `0` + // by `SplitMix64`'s own answer to the empty range, so `sample_thresh` + // indexes `spec.ranges[0]`. A document whose only candidate feature is + // empty crashed the command for any seed that did not open on a leaf. anyhow::ensure!( !data.feature_names.is_empty(), "nothing left to fit {target:?} against once the empty columns were dropped{}", @@ -288,22 +368,6 @@ pub fn fit_measure( false => format!(" — {}", data.warnings.join("; ")), } ); - // And wide is a *freeze*. A regular category with one distinct value per - // record one-hots to one column per record, and permutation importance is - // O(repeats × rows × features) evaluations — quadratic in records, on the - // thread that draws the next frame. The bound this module's header quotes - // was measured on a document whose row-identity category is a `Label`, so - // it does not cover this shape and refusing is honest where a silent - // multi-second freeze is not. `Label` is the fix a user wants, and the - // message says so. - anyhow::ensure!( - data.feature_names.len() <= MAX_FEATURES, - "fitting {target:?} would encode {} features from {} records, which is too wide to \ - search on the UI thread. A category with a distinct value per record becomes one \ - column per record — mark such a category as a Label so it is not a feature", - data.feature_names.len(), - rows.len(), - ); let names: Vec<&str> = data.feature_names.iter().map(String::as_str).collect(); let spec = Spec { @@ -348,6 +412,14 @@ pub fn fit_measure( &mut rng, ); + only_raw_measure_features( + &champion, + &spec, + &data.columns, + feature_columns, + category_columns, + )?; + Ok(FitOutcome { formula: crate::fitted_formula( &champion, @@ -366,6 +438,75 @@ pub fn fit_measure( }) } +/// Refuse a champion that splits on anything but a **raw numeric measure** +/// column, asked of `preprocess`'s own plan rather than of the feature's name. +/// +/// `fitted_formula` asks the *document* whether it can resolve a name, and that +/// check has a hole a name cannot close (Copilot on PR #25): a numeric category +/// called `Cost` produces a feature called `Cost`, and if a `_Measure` item is +/// also called `Cost` the emitted `Expr::Ref("Cost")` resolves — to the +/// measure, which is **not the column the tree was fitted on**. `Region=North` +/// and `Cost.missing` collide the same way with a document that happens to +/// contain an item so named, which the emitter's doc comment records as +/// deliberately accepted; accepting it is right for *that* check, whose question +/// is "can this be spelled", and wrong as the only check. +/// +/// So the question is asked of the plan instead. `columns` is one +/// [`ColumnSpec`] per **raw** column in `header` order (`preprocess` pushes one +/// per iteration), so `columns[i]` is `header[i]` and the first +/// `category_columns` of them are what `feature_table` contributed from +/// `Model::regular_category_names`. A feature index is accepted only when the +/// spec it *uniquely* names is a measure column of kind `Numeric` — which rules +/// out every one-hot, every `.missing` flag, every date part and every category +/// coordinate, by role and not by spelling. +/// +/// Note what this deliberately does **not** do: reconstruct the expansion order +/// index-by-index. That is improvise-4zl7's job and its ticket warns against a +/// second implementation of it. Matching a used feature's name against the raw +/// plan is sound in the *refusing* direction — `preprocess` names a plain +/// numeric column exactly `col` and gives every expanded feature a suffix or an +/// infix — and a false refusal costs a fit, where a false acceptance costs a +/// wrong rule. +fn only_raw_measure_features( + champion: &gp_tree::Tree, + spec: &Spec, + columns: &[gp_tree::data::ColumnSpec], + feature_columns: usize, + category_columns: usize, +) -> anyhow::Result<()> { + let raw_measure_names: Vec<&str> = columns + .iter() + .take(feature_columns) + .enumerate() + .filter(|(i, spec)| { + *i >= category_columns && matches!(spec.kind, gp_tree::data::ColumnKind::Numeric { .. }) + }) + .map(|(_, spec)| spec.name.as_str()) + .collect(); + let ambiguous: Vec<&str> = columns + .iter() + .take(feature_columns) + .map(|spec| spec.name.as_str()) + .filter(|name| columns.iter().filter(|s| s.name == *name).count() > 1) + .collect(); + anyhow::ensure!( + ambiguous.is_empty(), + "this document names {ambiguous:?} both as a category and as a measure, so a fitted \ + formula could not say which one it meant. Rename one of them" + ); + let not_measures: Vec = crate::used_features(champion, spec) + .into_iter() + .filter(|name| !raw_measure_names.contains(&name.as_str())) + .collect(); + anyhow::ensure!( + not_measures.is_empty(), + "the fitted tree splits on {not_measures:?}, which is not a numeric measure of this \ + document — a category coordinate or an encoding artefact, and a formula has no way \ + to spell either. Fit over numeric measures for now" + ); + Ok(()) +} + /// The regression rows, or a refusal naming why a categorical target is not a /// formula. /// @@ -406,14 +547,20 @@ mod tests { /// steps at `Cost` 50, and the numbers are in the **thousands** so a fit /// that forgets to de-normalize is visibly wrong rather than subtly so. /// - /// `Region` carries a single item on purpose. `preprocess` one-hots a - /// categorical column, and a one-hot that is constant across every record - /// has the degenerate range `(1, 1)`, which `simplify` prunes — so the - /// champion *cannot* split on an unresolvable feature and the success path - /// is deterministic rather than "whatever this seed happened to pick". - /// `Tag` is what makes the twelve records distinct, and it is a `Label` - /// category so `dataset` excludes it from the features (it would otherwise - /// one-hot into one column per row: a perfect memoriser). + /// **Every category here is a `Label`, and that is the only thing that + /// makes the success path deterministic.** An earlier version of this + /// fixture left `Region` regular with a single item and claimed the + /// constant one-hot had a degenerate `(1, 1)` range that `simplify` would + /// prune. That claim is false — `data.rs::build_categorical` pushes + /// `(0.0, 1.0)` for every one-hot whatever the column contains, so + /// `simplify` cannot know it is constant and the champion was free to split + /// on `Region=North`. These tests were passing on seed luck, and the + /// comment asserting otherwise was the worst part (Copilot on PR #25). + /// + /// A `Label` category is excluded from the features by `dataset`, so the + /// only feature is the numeric measure `Cost` — which is also the only + /// shape a fit can currently *store*, so it is the right success path to + /// pin. `Tag` is what makes the twelve records distinct. fn document() -> (Model, View) { let mut model = Model::new("sales"); for cat in [REGION, TAG] { @@ -423,7 +570,9 @@ mod tests { .category_mut(REGION) .expect("just added") .add_item("North"); - model.category_mut(TAG).expect("just added").kind = CategoryKind::Label; + for cat in [REGION, TAG] { + model.category_mut(cat).expect("just added").kind = CategoryKind::Label; + } for i in 0..12u32 { model .category_mut(TAG) @@ -559,6 +708,88 @@ mod tests { assert_eq!(a.request, b.request); } + /// Copilot on PR #25, and the sharpest finding of the round: asking the + /// *document* whether it can resolve a feature name is not the same + /// question as asking whether that name means the column the tree was + /// fitted on. A numeric category called `Cost` encodes to a feature called + /// `Cost`; with a `_Measure` item of the same name the emitted + /// `Expr::Ref("Cost")` resolves cleanly — **to the measure** — and the + /// stored rule silently reads a different column from the one it learned. + /// + /// A document naming the same thing twice is refused outright, because no + /// formula could say which of them it meant. + #[test] + fn a_category_and_a_measure_of_the_same_name_are_refused() { + let (mut model, view) = document(); + model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Regular; + model.add_category("Cost").expect("under the limit"); + for i in 0..12u32 { + model + .category_mut("Cost") + .expect("just added") + .add_item(format!("{}", i % 3)); + } + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err("a name that means two columns is not a fit"); + assert!(err.to_string().contains("Cost"), "{err}"); + } + + /// The other half of the same rule, and the one a *name* cannot catch: a + /// category coordinate is refused **by role**, even where the document + /// happens to contain an item spelled the same way. `fitted_formula`'s own + /// check accepts that name deliberately — its question is "can this be + /// spelled" — which is exactly why it cannot be the only check. + #[test] + fn a_category_coordinate_is_refused_even_when_an_item_shares_its_name() { + let (mut model, view) = document(); + model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Regular; + model + .category_mut(REGION) + .expect("Region exists") + .add_item("South"); + // An item literally named `Region=North`, so the document *can* + // resolve the one-hot's name and the older check would pass it. + model + .category_mut(MEASURE) + .expect("_Measure exists") + .add_item("Region=North"); + for i in 0..12u32 { + model.clear_cell(&cell(&format!("r{i}"), "Cost")); + } + for (i, region) in (0..12u32).map(|i| (i, if i < 6 { "North" } else { "South" })) { + model.set_cell( + CellKey::new(vec![ + (REGION.to_string(), region.to_string()), + (TAG.to_string(), format!("r{i}")), + (MEASURE.to_string(), "Score".to_string()), + ]), + CellValue::Number(if region == "North" { 1000.0 } else { 4000.0 }), + ); + } + assert!( + model.find_item_category("Region=North").is_some(), + "the fixture must make the one-hot's name resolvable, or this test \ + proves nothing the previous one does not" + ); + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err("a one-hot is not a measure however it is spelled"); + assert!(err.to_string().contains("not a numeric measure"), "{err}"); + } + + /// Copilot on PR #25. `.improv` pipe quoting makes a newline legal inside + /// a name, so a document can hold a measure whose name would split the TUI + /// status bar in two. The whole composed line is flattened once. + #[test] + fn a_name_containing_a_newline_cannot_split_the_status_line() { + let (model, view) = document(); + let outcome = fit_measure(&model, &view, "Score", "two\nlines", &request()) + .expect("a newline in a name is legal"); + assert_eq!(outcome.formula.target, "two\nlines"); + let line = outcome.status_line(); + assert!(!line.contains('\n'), "{line:?}"); + assert!(line.contains("two lines"), "{line:?}"); + } + /// Copilot on PR #25, and it is right: the outcome used to carry `seed` /// plus `config`, which is the pair gotcha 37b names — and that pair is /// not sufficient *here*. Cross-validation draws from the same generator @@ -596,13 +827,13 @@ mod tests { #[test] fn a_document_with_no_usable_feature_is_refused_rather_than_panicking() { let (mut model, mut view) = document(); - // Both halves are needed to reach zero encoded features, and finding - // that out is the point: clearing `Cost` alone leaves `Region`'s - // one-hot standing, so the fit returns a constant leaf rather than - // crashing. The panic needs *every* candidate column gone — no regular - // category to one-hot, and the one measure entirely missing. - model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Label; - view.set_axis(REGION, Axis::None); + // The fixture has no regular category, so `Cost` is the only column + // there is — and `preprocess` drops it once it is entirely missing, + // leaving zero features. With a regular category present its one-hot + // survives (range `(0.0, 1.0)` whatever it holds) and the fit returns + // a constant leaf instead of crashing, which is why this test needs + // the all-`Label` fixture rather than merely an empty measure. + let _ = &mut view; for i in 0..12u32 { model.clear_cell(&cell(&format!("r{i}"), "Cost")); } @@ -721,6 +952,9 @@ mod tests { #[test] fn a_fit_that_needs_a_category_is_refused_naming_the_feature() { let (mut model, view) = document(); + // `document()` makes every category a Label; this test is about what + // happens when one is not. + model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Regular; model .category_mut(REGION) .expect("Region exists") @@ -753,8 +987,8 @@ mod tests { "{msg}" ); assert!( - msg.contains("not an item") || msg.contains("not items"), - "{msg}" + msg.contains("not a numeric measure"), + "the refusal must say what kind of thing the feature is: {msg}" ); // The refusal is read off a one-line status bar, so it has to *be* one // line. A `rustfmt` pass over the multi-line literal that spells it had diff --git a/crates/improvise-gp/src/lib.rs b/crates/improvise-gp/src/lib.rs index b875d3df..1ca51961 100644 --- a/crates/improvise-gp/src/lib.rs +++ b/crates/improvise-gp/src/lib.rs @@ -175,12 +175,27 @@ pub fn fitted_formula( /// Every feature the tree splits on that `Expr::Ref` could not resolve, in the /// order the spec names them and without repeats. fn unresolvable_features(tree: &Tree, spec: &Spec, model: &Model) -> Vec { + used_features(tree, spec) + .into_iter() + .filter(|name| model.find_item_category(name).is_none()) + .collect() +} + +/// Every feature the tree splits on, in the order the spec names them and +/// without repeats. +/// +/// Public to the crate because two different questions are asked of the same +/// set: whether the *document* can resolve each name (here) and whether +/// `preprocess`'s own plan says each one is a raw numeric measure +/// ([`fit::fit_measure`]). Walking the tree twice, two ways, is how those two +/// answers would drift apart. +pub(crate) fn used_features(tree: &Tree, spec: &Spec) -> Vec { let mut used = vec![false; spec.names.len()]; mark_features(tree, &mut used); spec.names .iter() .zip(&used) - .filter(|(name, u)| **u && model.find_item_category(name).is_none()) + .filter(|(_, u)| **u) .map(|(name, _)| (*name).to_string()) .collect() } From 6a929c1d216eb4c4c9de48d06aaa6b7a2a085287 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:10:33 -0700 Subject: [PATCH 56/72] feat(records): a `+cat` column, so a schema can be built without leaving the grid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-2zq step 2, on the user's ruling. Step 1 made this coherent by stopping `on_category_added` from ejecting the user; this adds the column. Typing a name into `+cat` creates that category, and because `GridLayout::records_columns` filters on `kind.is_regular()` and never looks at items or at an axis, the coordinate column appears immediately and is typeable on the next keystroke. **The two questions the ruling left open, and the answers.** *Where the column goes.* Not a second trailing column, and not one `+` disambiguated by a prefix. A prefix would put a sigil into the name space — and a category and a measure may legitimately share a name, which is the exact ambiguity `_Dim` already carries a column *index* rather than a label to avoid (gotcha 1). A key was rejected too: the name still has to be typed somewhere, and a minibuffer lands in the records-exit class bug this PR characterizes but does not fix. So each `+` sits adjacent to what it adds to — `[coords…, +cat, measures…, +]`. Position tells the two apart instead of a label, which is the clutter the ticket was worried about. It costs a mid-grid column index, and the consequence worth naming is that a **blank** records grid now opens with the cursor on `+cat` rather than `+`: a record needs coordinates before it needs measures, so that order is defensible, but it does change what the first keystroke in an empty grid does. *Whether committing also sets the coordinate on the current row.* It does not. It is symmetric with the measure `+`, which creates an empty column and writes no value; and setting it would move **every** measure cell in the row onto the new coordinate (gotcha 1) — a data migration across N cells attached to the same Enter that made a schema change, in an app with no undo. The follow-up edit is one column away. **The refusal is visible, which took a change of order.** `MAX_CATEGORIES` is reachable here rather than theoretical. `AddCategory` now aborts the batch on refusal — same reason a failed save does (gotcha 21): what follows assumes the category exists, and the trailing `EnterEditAtCursor` would re-open the minibuffer *over* the message, since `draw_bottom_bar` renders the minibuffer instead of the status line. And `CommitAndAdvance` now emits its mode change **before** the commit rather than after, so an abort cannot eat it. That is the third time this PR has applied one rule — tidy-up goes where an abort cannot reach it — and the comment at that site had already described the pattern for the *advance* failing; the commit failing was simply not considered. The limit itself is still checked in exactly one place, `Model::add_category`. Evidence. Three new tests drive real keys: `the_new_category_column_creates_a_category_and_its_column` (the column is usable immediately), `..._writes_no_coordinate` (the deliberate answer to question 2, asserted over every row), and `..._refuses_past_the_limit_where_the_user_can_see_it`, which asserts the mode has **no minibuffer** — proved discriminating by swapping the commit back in front of the mode change and watching it report "Maximum of 12 categories reached" from inside `RecordsEditing`, where nothing would have drawn it. Four existing characterization tests were updated knowingly, not incidentally: the shared records fixture's column list (which accounted for 8 of the 9 initial failures on its own), the core layout and persistence smoke expectations, and the blank-model bootstrap, which now has to step to the `+` column and so records the cursor-position consequence above. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/commit.rs | 38 +++- crates/improvise-command/src/ui/app.rs | 202 ++++++++++++++++-- crates/improvise-command/src/ui/effect.rs | 17 +- crates/improvise-command/src/ui/project.rs | 10 +- crates/improvise-core/src/view/layout.rs | 44 +++- crates/improvise-core/src/view/mod.rs | 3 +- tests/records_wide_smoke.rs | 10 +- 7 files changed, 288 insertions(+), 36 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/commit.rs b/crates/improvise-command/src/command/cmd/commit.rs index 4a44a22f..29e16847 100644 --- a/crates/improvise-command/src/command/cmd/commit.rs +++ b/crates/improvise-command/src/command/cmd/commit.rs @@ -343,6 +343,7 @@ fn commit_plain_records_edit( commit_regular_cell_value(&row.cell_key(measure), value, effects); } RecordCol::NewMeasure => add_measure(value, effects), + RecordCol::NewCategory => add_category(value, effects), } } @@ -361,6 +362,29 @@ fn add_measure(value: &str, effects: &mut Vec>) { effects.push(effect::mark_dirty()); } +/// Typing a name into the `+cat` column creates that category, which gives +/// the row a coordinate column to type into (improvise-2zq). +/// +/// It creates the column and **does not set the coordinate on the current +/// row**, which is the deliberate answer to the ticket's third question. +/// Two reasons. It is symmetric with the measure `+`, which creates an empty +/// column and writes no value. And setting it would move *every* measure +/// cell in the row onto the new coordinate (gotcha 1) — a data migration +/// across N cells attached to the same Enter that made a schema change, in +/// an app with no undo. The follow-up edit is one `l` away in the column +/// that just appeared. +/// +/// Like `add_measure` this is a schema change, so it applies immediately +/// even inside a drill rather than being staged. +fn add_category(value: &str, effects: &mut Vec>) { + let name = value.trim(); + if name.is_empty() { + return; + } + effects.push(Box::new(effect::AddCategory(name.to_string()))); + effects.push(effect::mark_dirty()); +} + fn commit_cell_value( ctx: &CmdContext, key: &CellKey, @@ -372,8 +396,10 @@ fn commit_cell_value( return; }; match (&col, ctx.has_drill_state) { - // A new measure is a schema change, never a staged cell edit. + // A new measure or category is a schema change, never a staged + // cell edit: there is no cell to roll back with the drill. (RecordCol::NewMeasure, _) => add_measure(value, effects), + (RecordCol::NewCategory, _) => add_category(value, effects), (_, true) => effects.push(stage_drill_edit(record_idx, col, value)), (_, false) => commit_plain_records_edit(ctx, record_idx, &col, value, effects), } @@ -426,14 +452,22 @@ impl Cmd for CommitAndAdvance { } fn execute(&self, ctx: &CmdContext) -> Vec> { let mut effects: Vec> = Vec::new(); - commit_cell_value(ctx, &self.key, &self.value, &mut effects); // Pre-emptively drop to the normal counterpart of edit_mode. If the // advance succeeds, the trailing `EnterEditAtCursor` below will lift // us back into editing on the new cell. If the advance aborts // (e.g. already at bottom-right on Enter), `EnterEditAtCursor` is // skipped and we land in normal mode — which is the desired // "Enter at bottom-right commits and exits" behavior. + // + // It goes **before** the commit, not after (improvise-2zq). The + // commit can abort the batch itself — `AddCategory` does so when the + // document is at `MAX_CATEGORIES` — and an abort before this ran + // left the app in the editing mode, where `draw_bottom_bar` renders + // the minibuffer instead of the status line and the refusal the + // effect had just written was never drawn. Tidy-up goes where an + // abort cannot reach it; same rule as `ExecuteCommand`'s. effects.push(effect::change_mode(exit_mode_for(&self.edit_mode))); + commit_cell_value(ctx, &self.key, &self.value, &mut effects); match self.advance { AdvanceDir::Down => { let adv = EnterAdvance { diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 15b8a52f..3dae170a 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1391,6 +1391,155 @@ mod tests { ); } + /// improvise-2zq step 2: the `+cat` column creates a category and its + /// column, without leaving the grid. + /// + /// Driven through real keys. The assertion that earns its place is the + /// last one: the new column is **usable immediately**, which is the whole + /// point — `records_columns` filters on `kind.is_regular()` and never + /// requires items, so a category with no items is a column the moment it + /// exists. + #[test] + fn the_new_category_column_creates_a_category_and_its_column() { + let mut app = records_model_with_two_rows(); + let plus_cat = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == crate::view::NEW_CATEGORY_COL) + .expect("the records grid offers a +cat column"); + app.cursor_mut().selected = (0, plus_cat); + + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + for c in "Product".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + app.rebuild_layout(); + + assert!( + app.model_state.workbook.model.category("Product").is_some(), + "the category is created" + ); + assert!( + app.layout.is_records_mode(), + "and creating it does not eject the user (step 1)" + ); + let cols: Vec = (0..app.layout.col_count()) + .map(|i| app.layout.col_label(i)) + .collect(); + assert_eq!( + cols, + vec![ + "Region", + "Product", + crate::view::NEW_CATEGORY_COL, + "meas1", + "meas2", + crate::view::NEW_MEASURE_COL + ], + "the new coordinate column appears immediately, before the \ + still-available +cat, got {cols:?}" + ); + } + + /// Committing the `+cat` name creates the **column** and does not write + /// the coordinate on the row the cursor was on. + /// + /// The deliberate answer to the ticket's third question, and the reason + /// is gotcha 1: setting a coordinate moves *every* measure cell in the + /// row onto it. That is a data migration across N cells, and attaching it + /// to the same Enter that made a schema change — in an app with no undo — + /// is a bigger single keystroke than the measure `+` asks for. The + /// follow-up edit is one column away. + #[test] + fn the_new_category_column_creates_the_column_but_writes_no_coordinate() { + let mut app = records_model_with_two_rows(); + let plus_cat = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == crate::view::NEW_CATEGORY_COL) + .expect("+cat column"); + app.cursor_mut().selected = (0, plus_cat); + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + for c in "Product".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + app.rebuild_layout(); + + let product_col = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == "Product") + .expect("the new column exists"); + for row in 0..app.layout.row_count() { + assert_eq!( + app.layout.records_display(row, product_col).as_deref(), + Some(""), + "row {row} must have no coordinate for a category just created" + ); + } + } + + /// `MAX_CATEGORIES` is a reachable refusal on this path, and the refusal + /// has to be **visible**. + /// + /// `draw_bottom_bar` renders the minibuffer instead of the status line, + /// so a message written while an editor is open is never drawn. The + /// commit path therefore drops out of editing *before* running the add, + /// and `AddCategory` aborts the batch on refusal so the trailing + /// `EnterEditAtCursor` cannot re-open the editor over the message. + /// + /// The limit itself is checked in exactly one place — `Model::add_category` + /// — and nothing here restates it; this fills the document to the limit + /// by asking for categories until it refuses. + #[test] + fn the_new_category_column_refuses_past_the_limit_where_the_user_can_see_it() { + let mut app = records_model_with_two_rows(); + let mut n = 0; + while app + .model_state + .workbook + .add_category(format!("Filler{n}")) + .is_ok() + { + n += 1; + assert!(n < 100, "add_category must refuse eventually"); + } + app.rebuild_layout(); + app.view_state.status_msg.clear(); + + let plus_cat = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == crate::view::NEW_CATEGORY_COL) + .expect("+cat column"); + app.cursor_mut().selected = (0, plus_cat); + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + for c in "TooMany".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + + assert!( + app.model_state.workbook.model.category("TooMany").is_none(), + "the category is refused" + ); + assert!( + !app.view_state.status_msg.is_empty(), + "the refusal is reported" + ); + assert!( + app.view_state.mode.minibuffer().is_none(), + "and reported where it is drawn: a mode with a minibuffer hides \ + the status line, so the user would see nothing at all. mode is \ + {:?}, message {:?}", + app.view_state.mode, + app.view_state.status_msg + ); + } + /// improvise-2zq step 1: naming a category in records mode used to eject /// the user from the grid they were standing in. /// @@ -1545,10 +1694,16 @@ mod tests { ); } - /// A blank model has no measures, so wide format shows only the trailing - /// new-measure column. Typing a name there creates the measure, which - /// gives it a real column to type values into — records mode stays - /// self-sufficient for onboarding (improvise-c5v). + /// A blank model has no categories and no measures, so wide format shows + /// exactly the two creation columns. Typing a name into the new-measure + /// one creates the measure, which gives it a real column to type values + /// into — records mode stays self-sufficient for onboarding + /// (improvise-c5v). + /// + /// improvise-2zq widened that: a blank grid now offers `+cat` beside `+`, + /// so both halves of a schema can be bootstrapped without leaving the + /// grid. This test still pins the measure half; the category half is + /// `the_new_category_column_creates_a_category_and_its_column`. #[test] fn blank_model_bootstraps_its_first_measure_from_the_new_measure_column() { use crate::model::cell::CellKey; @@ -1561,8 +1716,8 @@ mod tests { .collect(); assert_eq!( cols, - vec![crate::view::NEW_MEASURE_COL], - "a blank model offers only the new-measure column" + vec![crate::view::NEW_CATEGORY_COL, crate::view::NEW_MEASURE_COL], + "a blank model offers exactly the two creation columns" ); // `o` seeds a row and opens an edit on it. @@ -1570,7 +1725,17 @@ mod tests { .unwrap(); assert_eq!(app.layout.row_count(), 1); - // Naming a measure in the + column creates it. + // The cursor lands at column 0, which in a blank grid is now `+cat` + // rather than `+` (improvise-2zq) — a record needs coordinates + // before it needs measures, so that order is deliberate. Move to the + // new-measure column to exercise the measure half. + // `o` left us editing column 0, so leave that edit before moving. + app.handle_key(KeyEvent::new(KeyCode::Esc, KeyModifiers::NONE)) + .unwrap(); + let new_measure_col = app.layout.col_count() - 1; + app.cursor_mut().selected = (0, new_measure_col); + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); for c in "Rev".chars() { app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) .unwrap(); @@ -1583,12 +1748,17 @@ mod tests { .collect(); assert_eq!( cols, - vec!["Rev", crate::view::NEW_MEASURE_COL], - "the named measure becomes a real column" + vec![ + crate::view::NEW_CATEGORY_COL, + "Rev", + crate::view::NEW_MEASURE_COL + ], + "the named measure becomes a real column, between the two \ + creation columns" ); // Typing into that column writes the value. - let rev_col = 0; + let rev_col = 1; app.cursor_mut().selected = (0, rev_col); app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) .unwrap(); @@ -1650,9 +1820,15 @@ mod tests { .collect(); assert_eq!( cols, - vec!["Region", "meas1", "meas2", crate::view::NEW_MEASURE_COL], - "setup: category column, one column per measure, then the \ - new-measure column" + vec![ + "Region", + crate::view::NEW_CATEGORY_COL, + "meas1", + "meas2", + crate::view::NEW_MEASURE_COL + ], + "setup: category columns, the new-category column, one column \ + per measure, then the new-measure column (improvise-2zq)" ); app } diff --git a/crates/improvise-command/src/ui/effect.rs b/crates/improvise-command/src/ui/effect.rs index 357d123f..138d2a56 100644 --- a/crates/improvise-command/src/ui/effect.rs +++ b/crates/improvise-command/src/ui/effect.rs @@ -95,8 +95,20 @@ macro_rules! simple_effect { pub struct AddCategory(pub String); impl Effect for AddCategory { fn apply(&self, app: &mut App) { + // A refusal aborts the batch, for the same reason a failed save does + // (gotcha 21): what follows assumes the category exists. From the + // `+cat` column (improvise-2zq) the rest of the batch would advance + // the cursor and re-open an editor on a column that was never + // created — and re-opening the minibuffer is also what would *hide* + // this message, since `draw_bottom_bar` renders the minibuffer + // instead of the status line. + // + // `MAX_CATEGORIES` is a reachable refusal on that path rather than a + // theoretical one, and the limit is checked in exactly one place — + // `Model::add_category` — so nothing here restates it. if let Err(e) = app.model_state.workbook.add_category(&self.0) { app.view_state.status_msg = e.to_string(); + app.abort_effects = true; } } fn kind(&self) -> EffectKind { @@ -658,8 +670,9 @@ impl Effect for ApplyAndClearDrill { apply_coordinate_rename(app, row, category, new_value); } // Never staged — `commit_cell_value` applies a new measure - // immediately, since it is a schema change and not a cell. - RecordCol::NewMeasure => {} + // or category immediately, since a schema change is not a + // cell edit and has nothing to roll back with the drill. + RecordCol::NewMeasure | RecordCol::NewCategory => {} } } app.model_state.dirty = true; diff --git a/crates/improvise-command/src/ui/project.rs b/crates/improvise-command/src/ui/project.rs index b0d87ff5..0340195d 100644 --- a/crates/improvise-command/src/ui/project.rs +++ b/crates/improvise-command/src/ui/project.rs @@ -315,10 +315,12 @@ fn cell_id(app: &App, row: usize, col: usize) -> Option { RecordCol::Measure(measure) => Some(measure.clone()), // Not cells to write into. Editing a category column renames a // coordinate across every measure cell in the row (gotcha 1) and - // the `+` column creates a measure; both are different intents - // from `SetCell`, so neither gets an id it could be composed - // with. - RecordCol::Category(_) | RecordCol::NewMeasure => return None, + // the `+`/`+cat` columns create a measure or a category; all are + // different intents from `SetCell`, so none gets an id it could + // be composed with. + RecordCol::Category(_) | RecordCol::NewMeasure | RecordCol::NewCategory => { + return None; + } }; return Some(CellId::Record(RecordCell { record: RecordHash::of(record), diff --git a/crates/improvise-core/src/view/layout.rs b/crates/improvise-core/src/view/layout.rs index 138b9af7..4df5af5e 100644 --- a/crates/improvise-core/src/view/layout.rs +++ b/crates/improvise-core/src/view/layout.rs @@ -10,6 +10,14 @@ use crate::view::{Axis, View}; /// measure column (improvise-c5v). pub const NEW_MEASURE_COL: &str = "+"; +/// The label of the column that creates a new **category** (improvise-2zq). +/// +/// It sits at the end of the category block rather than trailing the whole +/// grid, so each `+` is adjacent to what it adds to: `+cat` after the +/// coordinates, `+` after the measures. Two trailing columns would have made +/// the user read a label to tell them apart; position says it instead. +pub const NEW_CATEGORY_COL: &str = "+cat"; + /// Extract (record_index, column_index) from a synthetic records-mode /// CellKey. Returns None for normal pivot-mode keys. /// @@ -84,6 +92,9 @@ pub enum RecordCol { Measure(String), /// The trailing column that creates a new measure. NewMeasure, + /// The column that creates a new category (improvise-2zq). Sits after + /// the coordinates, where the column it creates will appear. + NewCategory, } /// One entry on a grid axis: either a visual group header or a data-item tuple. @@ -320,6 +331,7 @@ impl GridLayout { }) .map(|c| RecordCol::Category(c.to_string())) .collect(); + cols.push(RecordCol::NewCategory); cols.extend( model .category(MEASURE_CATEGORY) @@ -336,6 +348,7 @@ impl GridLayout { match col { RecordCol::Category(name) | RecordCol::Measure(name) => name.clone(), RecordCol::NewMeasure => NEW_MEASURE_COL.to_string(), + RecordCol::NewCategory => NEW_CATEGORY_COL.to_string(), } } @@ -367,7 +380,7 @@ impl GridLayout { ), // Always blank: it is where a new measure name is typed, never // a place a value is stored. - RecordCol::NewMeasure => Some(String::new()), + RecordCol::NewMeasure | RecordCol::NewCategory => Some(String::new()), } } @@ -605,8 +618,8 @@ fn cross_product(model: &Model, view: &View, cats: &[String]) -> Vec #[cfg(test)] mod tests { use super::{ - AxisEntry, GridLayout, NEW_MEASURE_COL, RecordCol, data_items, prune_axis_items, - synthetic_record_info, + AxisEntry, GridLayout, NEW_CATEGORY_COL, NEW_MEASURE_COL, RecordCol, data_items, + prune_axis_items, synthetic_record_info, }; use crate::model::cell::{CellKey, CellValue}; use crate::view::Axis; @@ -835,22 +848,33 @@ mod tests { .collect(); assert_eq!( cols, - vec!["Region", "Revenue", "Cost", NEW_MEASURE_COL], - "categories, then one column per measure, then the new-measure column" + vec![ + "Region", + NEW_CATEGORY_COL, + "Revenue", + "Cost", + NEW_MEASURE_COL + ], + "categories, the new-category column, one column per measure, \ + then the new-measure column — each `+` adjacent to what it adds \ + to (improvise-2zq)" ); assert_eq!( layout.record_col(0), Some(&RecordCol::Category("Region".to_string())) ); + assert_eq!(layout.record_col(1), Some(&RecordCol::NewCategory)); assert_eq!( - layout.record_col(1), + layout.record_col(2), Some(&RecordCol::Measure("Revenue".to_string())) ); - assert_eq!(layout.record_col(3), Some(&RecordCol::NewMeasure)); + assert_eq!(layout.record_col(4), Some(&RecordCol::NewMeasure)); - // Both measures of the one row are readable side by side. - assert_eq!(layout.records_display(0, 1).as_deref(), Some("100")); - assert_eq!(layout.records_display(0, 2).as_deref(), Some("50")); + // Both measures of the one row are readable side by side, and the + // `+cat` column between them and the coordinate is always blank. + assert_eq!(layout.records_display(0, 1).as_deref(), Some("")); + assert_eq!(layout.records_display(0, 2).as_deref(), Some("100")); + assert_eq!(layout.records_display(0, 3).as_deref(), Some("50")); } /// On initial entry into records mode, rows should be in a deterministic diff --git a/crates/improvise-core/src/view/mod.rs b/crates/improvise-core/src/view/mod.rs index d4fbf7fd..5e061ce2 100644 --- a/crates/improvise-core/src/view/mod.rs +++ b/crates/improvise-core/src/view/mod.rs @@ -4,6 +4,7 @@ pub mod types; pub use axis::{Axis, AxisColor}; pub use layout::{ - AxisEntry, GridLayout, NEW_MEASURE_COL, RecordCol, RecordRow, synthetic_record_info, + AxisEntry, GridLayout, NEW_CATEGORY_COL, NEW_MEASURE_COL, RecordCol, RecordRow, + synthetic_record_info, }; pub use types::{DRILL_VIEW, RECORDS_VIEW, VIRTUAL_VIEW_NAMES, View}; diff --git a/tests/records_wide_smoke.rs b/tests/records_wide_smoke.rs index a9742fcc..c121b25a 100644 --- a/tests/records_wide_smoke.rs +++ b/tests/records_wide_smoke.rs @@ -39,8 +39,10 @@ fn imported_model_shows_one_row_per_record_with_measure_columns() { .collect(); assert_eq!( cols, - vec!["product", "region", "cost", "revenue", "+"], - "categories, then one column per measure, then the new-measure column" + vec!["product", "region", "+cat", "cost", "revenue", "+"], + "categories, the new-category column, one column per measure, then \ + the new-measure column — each `+` sits next to what it adds to \ + (improvise-2zq)" ); assert_eq!(layout.row_count(), 2, "two source records, not four cells"); @@ -57,8 +59,8 @@ fn imported_model_shows_one_row_per_record_with_measure_columns() { assert_eq!( rows, vec![ - vec!["Pants", "West", "80", "200", ""], - vec!["Shirts", "East", "40", "100", ""], + vec!["Pants", "West", "", "80", "200", ""], + vec!["Shirts", "East", "", "40", "100", ""], ], "got {rows:?}" ); From 858a46e1a85e25bb96f8992e48f6894bee5e8a0f Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:13:49 -0700 Subject: [PATCH 57/72] docs: qualify the loopback claim, in both directions Copilot's sixth pass, and it is my own round-4 failure repeating: I wrote that "loopback is not a boundary a browser enforces" and that ANY open page can reach ws://127.0.0.1 -- an absolute where the evidence supports a qualified claim. Same shape as asserting a WS handshake carries no credential. CHECKED RATHER THAN TAKEN, and the finding is itself overstated. Copilot says "current Chromium gates WebSocket access from a public origin to localhost behind Local Network Access". Chrome's own documentation (developer.chrome.com /blog/local-network-access) says the opposite for the case at hand: WEBSOCKETS WERE NOT YET GATED when the permission prompt began shipping, with coverage planned. So the mitigation exists and is rolling out; whether it covers THIS socket, in a given browser at a given version, is exactly what neither of us can state and what no test here can settle. The text now says that, and keeps two gaps that are structural rather than version-dependent: LNA is a Chromium feature, so other browsers need not behave alike; and it engages only for a PUBLIC origin, so the locally-served page this paragraph is about falls outside it entirely. The threat is kept, the reassurance is not offered, and the durable fix is named -- the server checking Origin, which it does not do. Grepped the same claim through the rest of the section, per the rule this PR earned: the two remaining "any hostile page" statements are about the REMOTE wss:// endpoint, which LNA does not gate at all (it gates public->local), so they are correct unqualified and were left alone. The distinction is the whole point of the fix. 1193 passing, unchanged; no code touched. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index bc57a846..a1cd82b4 100644 --- a/README.md +++ b/README.md @@ -222,11 +222,24 @@ below is needed to develop, to demo locally, or to share a TUI-hosted document with another tab on the same machine. One caveat even there, for the reason spelled out below: a WebSocket is exempt -from the same-origin policy, and loopback is not a boundary a browser enforces -— so while the server is running, **any** page open in that browser can connect -to `ws://127.0.0.1:8787` and read or edit the document. Nothing in the server -checks `Origin`. That is tolerable for a local session you start and stop, and -it is the reason not to leave one running unattended. +from the same-origin policy, and **nothing in the server checks `Origin`** — so +a page you did not open can connect to `ws://127.0.0.1:8787` and read or edit +the document while the server runs. + +How much the *browser* mitigates that is in flux and should not be leaned on. +Chromium has been rolling out **Local Network Access**, which puts requests +from a public-network origin to a loopback or private-network address behind a +permission prompt; per Chrome's own documentation WebSockets were **not yet** +covered when the prompt began shipping, with coverage planned, so whether a +given browser and version gates *this* socket is not something this project can +state — and there is no browser here to test it in. Two gaps are structural +rather than version-dependent: it is a Chromium feature, so other browsers need +not behave the same; and it only engages for a **public** origin, so the +locally-served page this section is about is outside it entirely. + +Treat it, then, as tolerable for a local session you start and stop, and as a +reason not to leave one running unattended. The durable fix is the server +checking `Origin` itself, which it does not yet do. ### Sharing a TUI-hosted document with a remote browser From f90d1757be5c60fd1743b48a6876cc828a5301b0 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:14:16 -0700 Subject: [PATCH 58/72] test(harness): the four assertions the close-listener fix owes, and a third bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The harness is the one place a line of `web/*.js` is executed, not only the one place `storage_contract` reaches `OpfsFs`. PR #21 declined Copilot's request for regression tests on its close-listener fix because there was no JavaScript test infrastructure in this repository at all — no `package.json`, no runner, nothing in CI running a line of `web/app.js` — and standing up a parallel one beside this harness would have been two answers to one question. That decline is only honest if the assertions land here, so here they are, and `playwright.config.js` now says out loud that a change to `web/*.js` belongs in this spec directory. Playwright rather than a headless-DOM runner, because what goes wrong in these files is a real Worker, a real WebSocket, real OPFS and real event ordering — none of which a DOM shim has. `close` fires on a connection that never opened and on one that was established and dropped, and nothing on the event tells them apart: `readyState` is already CLOSED in any close listener, and both arrive as 1006 with `wasClean: false`. So: 1. **close-before-open must leave "cannot reach" standing.** `test.fail`, because the close handler on this branch is unconditional and overwrites it (improvise-m18f). That IS the regression guard, and it is aimed at the implementation actually here rather than at a mutant invented for the occasion — against an unconditional handler, "always say cannot reach" would pass and prove nothing. Verified by simulating PR #21's `opened` flag in the served copy: the test passes and Playwright reports "expected to fail, but passed", so the fix cannot land silently either. 2. **open-then-close must read "disconnected".** Green under both implementations — verified under the simulated fix too, since this is the arm that makes suppressing `close` wrong. 3. **The watchdog must stay out of the never-opened path.** Green, but for a reason its own comment does not lead you to expect: `armWatchdog` runs only after `transport.ready` resolves, which happens on `open`, so on a socket that never opened the watchdog is never armed at all. Asserted on the same page load as the "cannot reach" check, since both need the same 9-second wait. 4. **The watchdog must not overwrite a mid-session disconnect.** It does. Found by measuring rather than predicted, and it is a **third** bug: on the open-then-close arm the watchdog is armed, nothing disarms it, and 8s later "disconnected" is replaced by "accepted the connection but sent no document" — which is false, the peer connected and then went away. Same class as the bug `setShellFailure` fixed (a watchdog must be disarmed, not pre-empted) and it **survives** improvise-m18f's fix, which is about which message `close` writes rather than about the timer still running — confirmed under the simulated fix, where this one still fails. `test.fail` and filed; `web/app.js` is not this unit's file. Measured before asserting, which is how 3 and 4 came out opposite to the expectation they were handed: never-opened: connecting… -> cannot reach -> disconnected open-then-close: connecting… -> connected -> disconnected -> accepted the connection but sent no document 11 passing in the harness, 32s. The two `test.fail` entries are expected failures, so the suite is green and both bugs are pinned in a form that goes red the moment either is fixed. Co-Authored-By: Claude Opus 5 (1M context) --- harness/playwright.config.js | 18 ++++++ harness/tests/shell.spec.js | 119 ++++++++++++++++++++++++++++------- 2 files changed, 115 insertions(+), 22 deletions(-) diff --git a/harness/playwright.config.js b/harness/playwright.config.js index b526269c..130867c3 100644 --- a/harness/playwright.config.js +++ b/harness/playwright.config.js @@ -5,6 +5,24 @@ import { defineConfig, devices } from "@playwright/test"; // `OpfsFs` and to read `#status` out of a real page, not to grow a second // test framework beside `cargo test`. // +// # This is the one place a line of web/*.js is executed +// +// It is not only the OPFS harness. Before this there was no JavaScript test +// infrastructure in the repository at all — no package.json, no runner, +// nothing in CI running a single line of `web/app.js` or `web/worker.js` — so +// every bug in them was found by reading (gotcha 48) and the bug-first rule +// could not be honoured for a change to either. PR #21 declined to add +// regression tests for its close-listener fix on exactly that ground, and +// that decline is only honest because those assertions live here instead: +// see `tests/shell.spec.js`. +// +// So a change to `web/*.js` belongs in this spec directory, not in a second +// ad-hoc surface beside it — two answers to one question is the defect this +// codebase keeps curing. Playwright is the right vehicle rather than a +// headless-DOM runner (jsdom and friends) because the things that go wrong in +// these files are a real Worker, a real WebSocket, real OPFS and real event +// ordering, none of which a DOM shim has. +// // The server root is `harness/serve`, built by `build.sh` — the assembled // site plus this harness's own page. A module script will not load from // `file://`, so an http server is not optional. diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js index 9c054fa5..0e014d87 100644 --- a/harness/tests/shell.spec.js +++ b/harness/tests/shell.spec.js @@ -11,6 +11,23 @@ import { test, expect } from "@playwright/test"; const status = (page) => page.locator("#status"); +const json = (v) => JSON.stringify(v); + +/// `web/app.js`'s own HANDSHAKE_TIMEOUT_MS. Named here because two tests have +/// to outlast it, and a number that silently disagrees with the page would +/// make them pass for the wrong reason. +const HANDSHAKE_TIMEOUT_MS = 8_000; + +/// Record every distinct #status the page shows, not just the last one. +/// Several of these are about a message being REPLACED, which the final value +/// cannot see. +function logStatus() { + window.__statusLog = []; + new MutationObserver(() => { + const text = document.getElementById("status")?.textContent ?? ""; + if (text && window.__statusLog.at(-1) !== text) window.__statusLog.push(text); + }).observe(document, { childList: true, subtree: true, characterData: true }); +} test.describe("the shell reports what its peer is doing", () => { // improvise-hwpi: `on_frame` returns a Result, which wasm-bindgen throws. @@ -32,31 +49,89 @@ test.describe("the shell reports what its peer is doing", () => { // opened left #status blank, and "the server is not running" read exactly // like "still connecting". Port 9 (discard) refuses immediately. // - // It is asserted over the *history* of #status rather than its final value, - // because of improvise-m18f, which this reproduces: the socket's `close` - // listener fires after `error` and overwrites the precise message with - // "disconnected". That one is filed and deliberately not fixed here — - // suppressing `close` could hide a genuine mid-session disconnect, so it is - // not the same fix as the watchdog's (gotcha 48). Asserting both halves - // pins hwpi's fix AND m18f's bug, so whichever moves, this speaks up. - test("a peer that cannot be reached is named, not left blank", async ({ page }) => { - await page.addInitScript(() => { - window.__statusLog = []; - new MutationObserver(() => { - const el = document.getElementById("status"); - const text = el?.textContent ?? ""; - if (text && window.__statusLog.at(-1) !== text) window.__statusLog.push(text); - }).observe(document, { childList: true, subtree: true, characterData: true }); - }); + // The watchdog half is asserted on the same page load rather than in its + // own test, because it needs the same 9-second wait. It holds for a reason + // its own comment does not lead you to expect: `armWatchdog` runs only + // after `transport.ready` resolves, which happens on `open` — so on a + // socket that never opened the watchdog is never armed at all. + test("a peer that cannot be reached is named, and the watchdog stays out of it", async ({ + page, + }) => { + await page.addInitScript(logStatus); await page.goto("/index.html?ws=127.0.0.1:9"); - await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); + await expect(status(page)).not.toHaveText("", { timeout: 15_000 }); + await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); const log = await page.evaluate(() => window.__statusLog); - expect(log.some((t) => t.includes("cannot reach")), `#status said: ${JSON.stringify(log)}`) - .toBe(true); - // improvise-m18f, as it stands today. Delete this line when it is fixed — - // it is here so the fix cannot land silently. - expect(log.at(-1)).toContain("disconnected"); + expect(log.some((t) => t.includes("cannot reach")), `#status said: ${json(log)}`).toBe( + true, + ); + expect(log.some((t) => t.includes("sent no document")), `#status said: ${json(log)}`).toBe( + false, + ); + }); + + // improvise-m18f, and the regression guard for the fix in PR #21. + // + // `close` fires on both a connection that never opened and one that was + // established and dropped, and NOTHING on the event tells them apart: + // `readyState` is already CLOSED in any close listener, and both arrive as + // code 1006 with `wasClean: false`. Distinguishing them needs a flag set in + // the `open` handler. + // + // `test.fail` is the injected fault, aimed at the implementation actually + // on this branch rather than at a mutant invented for the occasion: the + // close handler is unconditional here, so it overwrites the precise + // "cannot reach" with "disconnected" and this assertion FAILS. That is what + // makes it a guard — against the unconditional handler, "always say cannot + // reach" would pass and prove nothing. When the `opened` flag lands, this + // starts passing, Playwright reports "expected to fail but passed", and the + // `test.fail()` line has to be deleted. It cannot land silently either way. + test("close before open leaves the precise message standing", async ({ page }) => { + test.fail(true, "improvise-m18f: the close handler is unconditional on this branch"); + await page.goto("/index.html?ws=127.0.0.1:9"); + // Wait for the status to settle on SOMETHING rather than for a particular + // message: waiting for "cannot reach" would retry against the value the + // page has already replaced, and waiting for "disconnected" would time + // out for the wrong reason once the fix lands. + await expect(status(page)).not.toHaveText("", { timeout: 15_000 }); + await page.waitForTimeout(1_000); + await expect(status(page)).toContainText("cannot reach", { timeout: 1_000 }); + }); + + // The other arm, which must keep working when the first one is fixed: a + // socket that really did open and then dropped IS a disconnect, and saying + // so is the whole reason `close` cannot simply be suppressed. + test("a socket that opened and then dropped reads as disconnected", async ({ page }) => { + await page.routeWebSocket("ws://127.0.0.1:8790/", (ws) => { + setTimeout(() => ws.close(), 300); + }); + await page.goto("/index.html?ws=127.0.0.1:8790"); + await expect(status(page)).toContainText("connected", { timeout: 15_000 }); + await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); + }); + + // A THIRD bug, found by measuring rather than predicted: on the + // open-then-close arm the watchdog IS armed (it is armed after + // `transport.ready`, which resolved on `open`), nothing disarms it, and 8 + // seconds later it replaces "disconnected" with "accepted the connection + // but sent no document" — which is false; the peer did connect and then + // went away. + // + // It is the same class as the bug `setShellFailure` fixed (a watchdog must + // be disarmed, not pre-empted) and it survives the improvise-m18f fix, + // because that fix is about which message `close` writes, not about the + // timer still running. Filed rather than fixed here: `web/app.js` is not + // this unit's file. + test("the watchdog does not overwrite a mid-session disconnect", async ({ page }) => { + test.fail(true, "the close handler does not disarm the handshake watchdog"); + await page.routeWebSocket("ws://127.0.0.1:8791/", (ws) => { + setTimeout(() => ws.close(), 300); + }); + await page.goto("/index.html?ws=127.0.0.1:8791"); + await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); + await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); + await expect(status(page)).toContainText("disconnected", { timeout: 1_000 }); }); // The first-frame watchdog (gotcha 46), found by reading. The condition is From f91c5a77dbfe797e5cc0b354ff2daa00d08f8722 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:17:43 -0700 Subject: [PATCH 59/72] docs: an allow-list is access control, not authentication Copilot's seventh pass, reporting no new open findings; this is the one carried item that is a real defect and it is cheap. The closing note called the IP and Origin allow-lists "the auth above", in a section whose opening sentence is that the server has NO AUTHENTICATION OF ANY KIND. That muddies the exact distinction the section is built on: an allow-list answers WHICH CLIENTS AND WHICH ORIGINS MAY CONNECT, never WHO is connecting. Calling it auth invites a reader to think the hole is closed when what has been added is a perimeter. Checked every other use of the word in the section rather than editing the one line flagged, per the grep rule this PR earned: "no authentication of any kind" (the server), "HTTP basic auth" (genuinely authentication, and being ruled out), "neither authenticates anything by default" (the tunnels) and `--basic-auth` (a product flag) are all correct as written. One site, one fix. The distinction is also exactly what improvise-kpnr has to decide, so the note now says so. 1193 passing, unchanged; no code touched. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a1cd82b4..8af430b2 100644 --- a/README.md +++ b/README.md @@ -391,7 +391,10 @@ Note the server binds `127.0.0.1` by default and `--bind` is the only way to widen that. Behind a proxy on the same host you should *not* widen it: the proxy reaches loopback, and leaving the default keeps the **unencrypted** port off the network. That is worth doing and is not access control — the proxy is -still the open door, which is why the auth above belongs there. +still the open door, which is why the **access controls** above belong there. +Those are access controls and not authentication, which the server still has +none of: they answer *which clients and which origins may connect*, never +*who* is connecting. The distinction is the whole of improvise-kpnr. ### Why not native `wss://` From 44295940ada2d80bb85b1b5d402ec16b70cc8028 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:18:22 -0700 Subject: [PATCH 60/72] test(persistence): corrupt the gzip CRC, so the checksum test tests the checksum MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's fourth finding on PR #22, and it was right about more than it claimed. `a_gzip_document_with_a_flipped_byte_fails_its_checksum` flipped a byte at `len / 2`, inside the DEFLATE payload, and asserted only `ErrorKind::InvalidInput` — which a *malformed compressed stream* also produces. The name was therefore a guess about which of two failures the arbitrary index happened to hit. Probed it before changing anything: flipping any byte in 10..40 gives `corrupt deflate stream` **thirty times out of thirty** and a checksum error never. `len / 2` happened to land somewhere that fails the checksum, so the test had been passing for a reason nobody chose. Now the flip is in the CRC32 field — the first four of the eight-byte gzip trailer — which leaves the compressed stream decodable, so the checksum is the only thing left to fail. And since both failures wear `InvalidInput`, the kind cannot separate them: the cause's own message must say `checksum`. That is asserted on the source error rather than on the `anyhow::Error`, because anyhow's `to_string` shows the outermost context, which is the sentence naming the document. Watched failing with the flip moved back into the payload (`got corrupt deflate stream`), and the sibling truncation test is untouched and still `UnexpectedEof`. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-io/src/persistence/mod.rs | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/crates/improvise-io/src/persistence/mod.rs b/crates/improvise-io/src/persistence/mod.rs index 49e0722e..de217116 100644 --- a/crates/improvise-io/src/persistence/mod.rs +++ b/crates/improvise-io/src/persistence/mod.rs @@ -316,14 +316,28 @@ mod tests { /// refuses damaged input" from "the decoder refuses short input". #[test] fn a_gzip_document_with_a_flipped_byte_fails_its_checksum() { + // The byte flipped is in the CRC32 field — the first four of the + // eight-byte gzip trailer — and not in the DEFLATE payload (Copilot, + // PR #22). A payload byte also yields `InvalidInput`, but it can do so + // for malformed compressed data rather than for the mismatch this test + // is named after, and the name would then be a guess about which. + // Corrupting the trailer leaves the compressed stream decodable, so + // the only thing left to fail is the checksum. let err = load_corrupt_gzip(|bytes| { - let mid = bytes.len() / 2; - bytes[mid] ^= 0xff; + let crc = bytes.len() - 8; + bytes[crc] ^= 0xff; }); let io = err .downcast_ref::() .unwrap_or_else(|| panic!("expected an io::Error, got {err:?}")); assert_eq!(io.kind(), std::io::ErrorKind::InvalidInput); + // The source error, not `err`: anyhow's `to_string` shows the outermost + // context only, and the context is the sentence naming the document. + assert!( + io.to_string().contains("checksum"), + "the kind alone does not separate a bad checksum from bad DEFLATE, \ + so the cause must say which, got {io}" + ); assert!( err.to_string().contains("model.improv.gz"), "the message must name the document, got {err}" From f535fba2f76cff709abad6a2ac1439c8c93e2d62 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:19:09 -0700 Subject: [PATCH 61/72] =?UTF-8?q?fix(fit):=20round=20three=20=E2=80=94=20t?= =?UTF-8?q?wo=20bugs=20in=20my=20own=20new=20guards,=20and=20a=20correctio?= =?UTF-8?q?n?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's third pass. Four fixed, one declined, and two of the four were defects in the checks the *previous* commit added. **The role check read positions where positions do not hold.** `preprocess` `continue`s past an entirely-missing column **without pushing a `ColumnSpec`**, so `Dataset::columns` is not index-aligned with the header once anything is dropped: one empty category shifts every later spec left, and the positional `i >= category_columns` test then read the surviving numeric measure as a category and refused a perfectly good fit. The role now comes from the *header* — which `feature_table` built and which never shifts — and the plan is consulted by name. Proved by re-injecting the positional form and watching `an_empty_category_before_a_measure_does_not_break_the_role_check` fail alone. **And the collision test never built the collision.** Adding a category named `Cost` with items puts no `Cost` coordinate on the existing records, so `feature_table` emitted an empty column, `preprocess` dropped it, and there was no duplicate name to find — the test passed on the positional bug instead. The records carry the coordinate now, the fixture asserts the header really has two `Cost` columns before fitting, and the assertion names the ambiguity diagnostic so it fails if that check is removed. Two findings, one root: a guard written against a premise nobody checked. **`encoded_width` treated every column as categorical**, so a numeric measure with 65 distinct values was refused as 65 features, while a date or a column with missing cells could expand past the estimate. It now approximates `preprocess`'s inference, and — the part that matters — the **exact** width check is restored after preprocessing. The preflight is advisory, and says so: its only job is to refuse a rows × rows one-hot before the memory is spent, and a drift in either direction is bounded by the exact check behind it. **Nothing bounded the work by record count.** `MAX_FEATURES` bounds the width; every stage scales with rows on the thread that draws the frame. `MAX_RECORDS` (20,000 — about a second at the measured 105 ms/2,000, roughly linear) is the default of a new `FitRequest::max_records`, so a host that has somewhere else to run a fit can raise it rather than being told no. **Declined, and it corrects something I got wrong.** `Spec::ranges` is computed over all rows and the same spec goes into every training fold, so the thresholds the search samples and the branches `simplify` prunes are bounded by a held-out extremum — which reaches a fit over *complete numeric measures* too. My earlier reply on the imputation thread said "the case that currently stores leaks nothing"; that was wrong, and the doc comment on `Quality` now says so. Still declined here for the same reason: closing it needs a `cv_regress` that takes a raw table and a plan-builder, which is a `gp-tree` API change and the same one improvise-4zl7 will want. 1228 passing (+3); clippy, fmt and the wasm gate clean; CLI output still byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-gp/src/fit.rs | 297 +++++++++++++++++++++++++++------ 1 file changed, 243 insertions(+), 54 deletions(-) diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index f32b9a1b..91f80bae 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -57,28 +57,50 @@ use improvise_formula::Formula; use crate::dataset::{FeatureTable, feature_table}; -/// What `preprocess` *would* expand this table to, counted without expanding -/// it. +/// A cheap upper bound on how wide `preprocess` would expand this table, +/// computed without expanding it. /// -/// Deliberately an over-estimate and deliberately cheap: one column per -/// distinct string value, which is what `build_categorical` emits, plus one for -/// a column `preprocess` may read as numeric or as a date. It never allocates -/// the matrix, which is the whole point — the width has to be refused *before* -/// a rows × rows one-hot is materialised, not after. +/// **Advisory, not authoritative.** Its one job is to refuse a catastrophic +/// shape — a category with a distinct value per record, which materialises a +/// rows × rows one-hot — *before* the memory is spent. The exact width is +/// `Dataset::feature_names.len()`, checked after preprocessing, and that is the +/// check that enforces [`MAX_FEATURES`]; this one exists only because the exact +/// answer arrives too late to prevent the allocation that produced it. /// -/// It is not a second implementation of the expansion, and must not become -/// one: it answers "roughly how wide" for a budget check, where -/// `Dataset::feature_names` answers "exactly which" for everything else. +/// It approximates `preprocess`'s type inference rather than sharing it: a +/// column whose present cells all parse as a number expands to one feature (two +/// with a missing indicator), and anything else is counted at its distinct-value +/// count, which is what `build_categorical` emits and an over-count for a date. +/// Copying the predicate is the lesser evil here — the alternative is exporting +/// the inference and calling this a second implementation of it — and the cost +/// of the copy drifting is bounded in both directions by the exact check: +/// over-counting costs a refusal the exact check would not have made, and +/// under-counting only lets the exact check do its job (Copilot on PR #25). fn encoded_width(table: &FeatureTable) -> usize { (0..table.header.len().saturating_sub(1)) .map(|col| { - table + let cells: Vec<&str> = table .rows .iter() .filter_map(|row| row.get(col)) - .collect::>() - .len() - .max(1) + .map(String::as_str) + .collect(); + let present: Vec<&str> = cells + .iter() + .copied() + .filter(|c| !gp_tree::data::is_missing(c)) + .collect(); + let has_missing = present.len() < cells.len(); + match !present.is_empty() && present.iter().all(|c| c.parse::().is_ok()) { + // A numeric column is one feature, plus the `.missing` flag. + true => 1 + usize::from(has_missing), + // Everything else: one column per distinct value. + false => present + .iter() + .collect::>() + .len() + .max(1), + } }) .sum() } @@ -103,6 +125,18 @@ fn one_line(text: String) -> String { /// shape that is really a mis-typed category. pub const MAX_FEATURES: usize = 64; +/// The most records `fit_measure` will search synchronously. +/// +/// The width bound alone does not bound the *work*: a one-feature table with a +/// million records still runs cross-validation, the final search and +/// permutation importance over all of them, on the thread that draws the next +/// frame (Copilot on PR #25). Measured at 105 ms for 2,000 records and roughly +/// linear, so 20,000 is about a second — which is where a pause stops being a +/// pause. Refusing is honest; a host that has somewhere else to run a fit can +/// still call this with a bigger budget, because the limit is the caller's +/// [`FitRequest::max_records`] and this is only its default. +pub const MAX_RECORDS: usize = 20_000; + /// What to ask of a fit. Everything a run's outcome depends on, so that /// `(FitRequest, document)` reproduces it. #[derive(Clone, Copy, Debug, PartialEq)] @@ -117,6 +151,10 @@ pub struct FitRequest { pub cv_folds: usize, /// Shuffles per feature when measuring permutation importance. pub importance_repeats: usize, + /// The most records this fit may search. The width bound does not bound + /// the work: every stage scales with rows. Defaults to [`MAX_RECORDS`]; + /// a host that runs fits off the draw thread can raise it. + pub max_records: usize, } impl Default for FitRequest { @@ -146,6 +184,7 @@ impl Default for FitRequest { }, cv_folds: 5, importance_repeats: 3, + max_records: MAX_RECORDS, } } } @@ -164,13 +203,21 @@ impl Default for FitRequest { /// is therefore a little better than truly held-out wherever a *used* feature /// had a missing cell. /// -/// Two things bound it, and neither dismisses it. Only imputed numerics and -/// categorical vocabularies leak — a fit over complete numeric measures, which -/// is the only shape `fit_measure` currently *stores*, leaks nothing. And -/// closing it means preprocessing per training fold and encoding each -/// validation fold with that fold's plan, which `gp_tree::fit::cv_regress` -/// cannot express: it takes rows that are already encoded. That is an API -/// change in `gp-tree`, not a line here. +/// **And it is wider than imputation.** `Spec::ranges` is computed over all +/// rows and the *same* spec goes into every training fold, so the thresholds +/// the search samples and the branches `simplify` prunes are bounded by a +/// held-out minimum or maximum. That reaches a fit over **complete numeric +/// measures** too — which is the only shape `fit_measure` currently stores, so +/// "the case that stores leaks nothing" was wrong, and this paragraph replaces +/// it. `Measured` is transductive rather than fully held out. +/// +/// It is declined here rather than dismissed. Closing it means building the +/// ranges and the preprocessing plan per training fold and encoding each +/// validation fold with *that* plan, which `gp_tree::fit::cv_regress` cannot +/// express: it takes one spec and rows that are already encoded. That is an API +/// change in `gp-tree` — a `cv_regress` taking a raw table and a plan-builder — +/// and it is the same change improvise-4zl7 will want, so the two are filed to +/// land together. #[derive(Clone, Copy, Debug, PartialEq)] pub enum Quality { /// Pooled `k`-fold cross-validation against the predict-the-mean baseline. @@ -340,6 +387,14 @@ pub fn fit_measure( // bound this module's header quotes was measured on a document whose // row-identity category is a `Label`, so it does not cover this shape. // `Label` is the fix a user wants, and the message says so. + anyhow::ensure!( + table.rows.len() <= request.max_records, + "fitting {target:?} would search {} records, over the {} this host will do \ + synchronously. Narrow the view — a Page or Filter selection scopes the rows \ + (gotcha 27) — or raise FitRequest::max_records if there is somewhere else to run it", + table.rows.len(), + request.max_records, + ); let width = encoded_width(&table); anyhow::ensure!( width <= MAX_FEATURES, @@ -349,8 +404,8 @@ pub fn fit_measure( table.rows.len(), ); - let feature_columns = table.header.len() - 1; let category_columns = model.regular_category_names().len(); + let header = table.header.clone(); let mut data = preprocess(&table.header, &table.rows, &Overrides::default()); data.warnings.splice(0..0, table.warnings); let rows = regression_rows(data.target, target)?; @@ -360,6 +415,17 @@ pub fn fit_measure( // by `SplitMix64`'s own answer to the empty range, so `sample_thresh` // indexes `spec.ranges[0]`. A document whose only candidate feature is // empty crashed the command for any seed that did not open on a leaf. + // The exact width, now that it is known. The preflight above is advisory + // and approximates `preprocess`'s type inference; this is the check that + // actually enforces the bound, and it catches both directions of a drift + // between the two (Copilot on PR #25). + anyhow::ensure!( + data.feature_names.len() <= MAX_FEATURES, + "fitting {target:?} encoded {} features, over the {MAX_FEATURES} this host will \ + search. A category with many distinct values becomes one column per value — \ + mark such a category as a Label so it is not a feature", + data.feature_names.len(), + ); anyhow::ensure!( !data.feature_names.is_empty(), "nothing left to fit {target:?} against once the empty columns were dropped{}", @@ -412,13 +478,7 @@ pub fn fit_measure( &mut rng, ); - only_raw_measure_features( - &champion, - &spec, - &data.columns, - feature_columns, - category_columns, - )?; + only_raw_measure_features(&champion, &spec, &header, category_columns, &data.columns)?; Ok(FitOutcome { formula: crate::fitted_formula( @@ -451,14 +511,23 @@ pub fn fit_measure( /// deliberately accepted; accepting it is right for *that* check, whose question /// is "can this be spelled", and wrong as the only check. /// -/// So the question is asked of the plan instead. `columns` is one -/// [`ColumnSpec`] per **raw** column in `header` order (`preprocess` pushes one -/// per iteration), so `columns[i]` is `header[i]` and the first -/// `category_columns` of them are what `feature_table` contributed from -/// `Model::regular_category_names`. A feature index is accepted only when the -/// spec it *uniquely* names is a measure column of kind `Numeric` — which rules -/// out every one-hot, every `.missing` flag, every date part and every category -/// coordinate, by role and not by spelling. +/// So the question is asked of the plan instead, and **by name, never by +/// position**. `columns` is *not* index-aligned with `header`: `preprocess` +/// `continue`s past an entirely-missing column without pushing a `ColumnSpec`, +/// so one empty category shifts every later spec left and a positional +/// `i >= category_columns` test then reads a numeric measure as a category and +/// refuses a perfectly good fit. That was the first version of this function +/// and it was wrong (Copilot on PR #25). The header, which `feature_table` +/// built, is the reliable statement of role: its first `category_columns` +/// entries are `Model::regular_category_names`, the rest up to the target are +/// measures. +/// +/// A used feature is accepted only when its name is a **measure** column of the +/// header *and* the plan gives that column kind `Numeric` — which rules out +/// every one-hot, every `.missing` flag, every date part and every category +/// coordinate, by role and not by spelling. Duplicate header names are refused +/// outright first, since a name that means two columns cannot be resolved by +/// either check. /// /// Note what this deliberately does **not** do: reconstruct the expansion order /// index-by-index. That is improvise-4zl7's job and its ticket warns against a @@ -470,30 +539,38 @@ pub fn fit_measure( fn only_raw_measure_features( champion: &gp_tree::Tree, spec: &Spec, - columns: &[gp_tree::data::ColumnSpec], - feature_columns: usize, + header: &[String], category_columns: usize, + columns: &[gp_tree::data::ColumnSpec], ) -> anyhow::Result<()> { - let raw_measure_names: Vec<&str> = columns + let feature_header = &header[..header.len().saturating_sub(1)]; + let mut ambiguous: Vec<&str> = feature_header .iter() - .take(feature_columns) - .enumerate() - .filter(|(i, spec)| { - *i >= category_columns && matches!(spec.kind, gp_tree::data::ColumnKind::Numeric { .. }) + .map(String::as_str) + .filter(|name| { + feature_header + .iter() + .filter(|h| h.as_str() == *name) + .count() + > 1 }) - .map(|(_, spec)| spec.name.as_str()) - .collect(); - let ambiguous: Vec<&str> = columns - .iter() - .take(feature_columns) - .map(|spec| spec.name.as_str()) - .filter(|name| columns.iter().filter(|s| s.name == *name).count() > 1) .collect(); + ambiguous.dedup(); anyhow::ensure!( ambiguous.is_empty(), "this document names {ambiguous:?} both as a category and as a measure, so a fitted \ formula could not say which one it meant. Rename one of them" ); + let raw_measure_names: Vec<&str> = feature_header + .iter() + .skip(category_columns) + .map(String::as_str) + .filter(|name| { + columns.iter().any(|c| { + c.name == *name && matches!(c.kind, gp_tree::data::ColumnKind::Numeric { .. }) + }) + }) + .collect(); let not_measures: Vec = crate::used_features(champion, spec) .into_iter() .filter(|name| !raw_measure_names.contains(&name.as_str())) @@ -718,20 +795,132 @@ mod tests { /// /// A document naming the same thing twice is refused outright, because no /// formula could say which of them it meant. + /// + /// The first version of this test **never built the collision** (Copilot on + /// PR #25): adding a category `Cost` with items puts no `Cost` coordinate + /// on the existing records, so `feature_table` emits an empty column, + /// `preprocess` drops it, and there was no duplicate name to find. It + /// passed on the *positional* role bug instead. The records here carry the + /// coordinate, and the assertion names the ambiguity diagnostic + /// specifically so it fails if that check is removed. #[test] fn a_category_and_a_measure_of_the_same_name_are_refused() { let (mut model, view) = document(); - model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Regular; model.add_category("Cost").expect("under the limit"); for i in 0..12u32 { model .category_mut("Cost") .expect("just added") - .add_item(format!("{}", i % 3)); + .add_item(format!("c{}", i % 3)); + // The coordinate, not merely the item: an empty column is dropped. + for measure in ["Cost", "Score"] { + let value = model + .get_cell(&cell(&format!("r{i}"), measure)) + .cloned() + .expect("the fixture wrote this cell"); + model.clear_cell(&cell(&format!("r{i}"), measure)); + model.set_cell( + CellKey::new(vec![ + (REGION.to_string(), "North".to_string()), + (TAG.to_string(), format!("r{i}")), + ("Cost".to_string(), format!("c{}", i % 3)), + (MEASURE.to_string(), measure.to_string()), + ]), + value, + ); + } } + assert_eq!( + feature_table(&model, &view, "Score") + .expect("the table builds") + .header + .iter() + .filter(|h| h.as_str() == "Cost") + .count(), + 2, + "the fixture must really produce two columns named Cost, or this \ + test proves nothing" + ); let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) .expect_err("a name that means two columns is not a fit"); - assert!(err.to_string().contains("Cost"), "{err}"); + assert!( + err.to_string().contains("could not say which one it meant"), + "{err}" + ); + } + + /// Copilot on PR #25. `preprocess` `continue`s past an entirely-missing + /// column without pushing a `ColumnSpec`, so `Dataset::columns` is not + /// index-aligned with the header once anything is dropped. A positional + /// `i >= category_columns` role test then read the surviving numeric + /// measure as a category and refused a perfectly good fit. + #[test] + fn an_empty_category_before_a_measure_does_not_break_the_role_check() { + let (mut model, view) = document(); + // A regular category whose coordinate no record carries: `feature_table` + // emits the column, `preprocess` drops it, and every later ColumnSpec + // shifts left. + model.add_category("Empty").expect("under the limit"); + model + .category_mut("Empty") + .expect("just added") + .add_item("none"); + let table = feature_table(&model, &view, "Score").expect("builds"); + assert_eq!(table.header.first().map(String::as_str), Some("Empty")); + let data = crate::dataset(&model, &view, "Score", &Overrides::default()).expect("builds"); + assert!( + data.columns.len() < table.header.len() - 1, + "the fixture must really drop a column, or this test proves nothing" + ); + fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect("a dropped empty category must not make Cost look like a category"); + } + + /// Copilot on PR #25. `MAX_FEATURES` bounds the width; nothing bounded the + /// *work*, and every stage — cross-validation, the search, permutation + /// importance — scales with records on the thread that draws the frame. + #[test] + fn more_records_than_the_budget_is_refused_before_any_search() { + let (model, view) = document(); + let tiny = FitRequest { + max_records: 4, + ..request() + }; + let err = fit_measure(&model, &view, "Score", FIT_INTO, &tiny) + .expect_err("twelve records is over a budget of four"); + let msg = err.to_string(); + assert!(msg.contains("would search 12 records"), "{msg}"); + assert!( + msg.contains("Page or Filter"), + "the message must say what to do: {msg}" + ); + } + + /// The preflight approximates `preprocess`'s type inference, so it can + /// disagree in both directions — and the exact check after preprocessing is + /// what actually enforces the bound. A numeric measure with more distinct + /// values than `MAX_FEATURES` encodes to **one** feature and must not be + /// refused for having many values (Copilot on PR #25). + #[test] + fn a_high_cardinality_numeric_measure_is_one_feature_not_many() { + let (mut model, view) = document(); + for i in 0..12u32 { + // Distinct numeric costs, far more of them than MAX_FEATURES would + // allow if they were counted as categories. + model.set_cell( + cell(&format!("r{i}"), "Cost"), + CellValue::Number(f64::from(i) * 1_000.0 + 0.5), + ); + } + let data = crate::dataset(&model, &view, "Score", &Overrides::default()).expect("builds"); + assert_eq!( + data.feature_names.len(), + 1, + "a numeric column is one feature: {:?}", + data.feature_names + ); + fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect("distinct numeric values are not distinct categories"); } /// The other half of the same rule, and the one a *name* cannot catch: a From 553b2cb90de74d37a57485680f2549190609ef39 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:21:28 -0700 Subject: [PATCH 62/72] chore(beads): file the CSWSH hole, the records-mode minibuffer exit, and route vvl3's assertions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From WU-I (PR #21) and WU-G (PR #27) hand-backs. Searched the open backlog first; none of these duplicates an existing bead. improvise-kc7n P2 — improvise-ws-server never checks Origin. Verified independently: `grep -rni origin crates/improvise-ws-server/src/` returns nothing. A WebSocket handshake is exempt from the same-origin policy, so any page an allowed user visits can drive their session from their own allowed address — the attack an IP allow-list structurally cannot see, because the request genuinely is from an allowed client. Loopback is not exempt and has no proxy in front of it, so the local path is the least protected. Related to improvise-kpnr rather than blocked by it: authentication and CSWSH are different questions and an answer to either leaves the other open. improvise-sihr P2 — every minibuffer mode entered from records mode exits to plain Normal while the layout is still records. Affects `:`, Ctrl-E and Ctrl-W; only the last is new. Declined in PR #27 because fixing one arm of a three-arm rule is not a fix, and characterized there instead. improvise-vvl3 also gains the four assertions improvise-taxx's fix owes, so that ticket need not re-derive them — including the one that must fail against the previous unconditional close handler, which is the guard and not the decoration. Recorded there because vvl3 now owns the JS test-infrastructure decision that PR #21 declined. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 91a343bc..3fffc45f 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,7 +1,7 @@ {"_type":"issue","id":"improvise-0yq8","title":"Move ClientState and keys.rs::step into improvise-protocol","description":"THE SINGLE HIGHEST-VALUE CONSOLIDATION LEFT, named by the agent that did the keymap move and measured rather than guessed.\n\ndesign-principles 5 (user ruling 2026-09-20): minimise the code a new UI framework needs; centralise logic in a redux-like store so a new front end DISPLAYS STATE rather than reimplementing behaviours. The test is: would a third front end have to write this again?\n\nkeys.rs::step is 99 LINES OF \"WHAT THIS COMMAND DOES\" and the answer is YES. It depends on NOTHING but ViewState + the protocol widget caches + the announced window -- every one of which is ALREADY A PROTOCOL TYPE. So it is in the wrong crate, and the only reason it is there is history.\n\nAFTER THIS MOVE A NEW FRONT END NEEDS EXACTLY TWO THINGS: browser-event (or terminal-event, or whatever) -\u003e KeyEvent, and painting. That is the whole of design-principles 5 made true rather than aspirational.\n\nWHY IT WAS NOT DONE IN THE SAME PASS: it is a crate move on top of a crate move (the keymap had just moved), and half-doing it would leave two homes. Deliberate, not forgotten.\n\nMEASURED CONTEXT from that pass: keys.rs grew 264 -\u003e 401 non-comment lines while GAINING the whole editing path, search typing, panels, transpose and tile select -- and the key TABLE left the crate entirely. web/app.js, the only genuinely framework-bound file, grew 344 -\u003e 359 while gaining a live editor and a hint line. So the trend is right; this ticket is what finishes it.\n\nNOTE THE SECOND-CONSUMER ARGUMENT: the more of the front end is SHARED, the more of it a second consumer can AUDIT. CellDisplay::edit_text exists because the browser asked a question the TUI had answered privately and the two answers differed -- see doctrine(being_the_second_consumer_audits_the_abstraction).","status":"open","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-oai9","title":"[bug] The worker's errors reach the console with their cause stripped","description":"FOUND BY THE FIRST REAL BROWSER RUN of the deployed page (2026-09-20), and it is the reason that run produced no actionable information.\n\nThe console said:\n\n improvise worker: failed to start\n Error: Cannot open demo.improv\n\n\"Cannot open demo.improv\" is the CONTEXT vfs::open attaches with with_context. anyhow Display prints ONLY THE OUTERMOST CONTEXT, so bindings::start mapping through e.to_string() kept the label and threw away the cause -- the Storage failure or the parse failure that actually happened.\n\nWHY THIS IS THE WORST POSSIBLE PLACE FOR IT: bindings.rs is cfg(target_arch = wasm32) and is the one module in the crate that NO TEST IN THE REPOSITORY CAN REACH. A devtools console is the only instrument pointed at it. So the single code path with no test coverage was also the one discarding its own diagnostics.\n\nFIXED: a report() helper rendering with the alternate form {:#}, which walks the chain joining with \": \". It lives in lib.rs rather than bindings.rs SO IT CAN BE TESTED AT ALL -- bindings is wasm-gated. Five call sites (start, on_message, flush, documents, refresh).\n\nTEST WRITTEN AGAINST THE OLD RENDERING FIRST: a_reported_error_carries_its_cause_not_just_its_label asserts that anyhow to_string() IS just the label (pinning the trap, so the helper stops being needed if that ever changes) and that report() contains BOTH the label and the cause.\n\nTHIS DOES NOT FIX THE UNDERLYING FAILURE -- it makes the next reload name it. Redeploy and read the console again.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:55:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-anr9","title":"Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once","description":"Two tickets turn out to be ONE piece of work, noticed by improvise-bpq2 while walking the bindings.\n\nimprovise-s1xw (multi-level grid labels -- the browser shows Food/Alice where the TUI shows two columns) and the bindings H / z / gz need THE SAME FIVE FACTS, which gotcha 44 already enumerates as the grid frame the cache does not carry:\n row_cats / col_cats -- how many header levels\n row_items -- multi-level label TUPLES and inline group headers\n col_group_for -- column group spans\n is_group_collapsed -- the collapse indicator\n(record_col is the sixth and is CORRECT to keep off: session-local, keys pending drill edits.)\n\nSo:\n * hide-selected-row-item (H) needs to know WHICH CATEGORY a row label belongs to -- impossible while row_labels slash-joins a tuple.\n * toggle-group / gz need group membership and collapsed state.\n * And the browser renders the slash-joined form, which the TUI own test asserts must not appear.\n\nDOING THEM SEPARATELY MEANS TOUCHING row_labels TWICE. It is a co-ordinated type change either way -- improvise-wasm-client/src/render.rs consumes both row_labels and col_labels -- so the cheap moment is once.\n\nDO NOT add row_levels beside row_labels. Two answers to \"what are the labels\" is the disease (improvise-edp), and bpq2 has just spent a wave demonstrating what one shared derivation buys.\n\nSupersedes/absorbs improvise-s1xw -- decide which id survives.","notes":"USER RULING 2026-09-20: 'Most of the view functionality should be a wasm blob that's shared between the TUI and the web UI' -- clarified as ONE SHARED CRATE, not the TUI loading wasm at runtime. The view derivations live in improvise-protocol; the browser gets them as wasm, the TUI links them natively, and ONLY THE PAINTING differs (ratatui vs DOM).\n\nTHIS TICKET IS NOW THE LAST STEP OF THAT, which is why it is P1.\n\nMeasured 2026-09-20: every other widget already takes a shared cache and no \u0026Model -- FormulaContent(\u0026FormulaCache), ViewContent(\u0026ViewListCache), TileBar(\u0026TileBarCache), CategoryContent(\u0026CategoryTreeCache), WizardWidget(\u0026WizardCache), HelpWidget(usize), WhichKeyWidget(\u0026[..]). THE REMAINING DUPLICATION IS ONE FILE: src/ui/grid.rs still derives 14 things straight off layout./view., and those are exactly the six frame facts in gotcha 44.\n\nSo closing this ticket is what makes 'most of the view functionality shared' TRUE rather than nearly true, and it fixes the Food/Alice divergence and unblocks H/z/gz as the same change.\n\nSEQUENCING NOTE: this touches improvise-protocol/src/render.rs, src/ui/grid.rs AND improvise-wasm-client/src/render.rs together -- it is a co-ordinated type change on row_labels/col_labels, not an additive one. Do not start it while another agent is in improvise-protocol; the client-side keymap work (improvise-bpq2 successor) is there now.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:26Z","started_at":"2026-09-20T05:32:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-vvl3","title":"A headless browser harness: run storage_contract and the shell assertions under Playwright","description":"THE CONDITION THAT DEFERRED improvise-xf1 HAS BEEN MET. The user deferred browser testing on 2026-09-20 with: \"I am happy just putting off the browser test for now: WE HAVE TO HAVE SOMETHING TO TEST.\" After improvise-9x6, djm, hwpi and the demo model, we do -- a complete self-contained application whose ENTIRE untested surface is the part a harness would cover.\n\nTHE EVIDENCE THAT IT HAS FLIPPED, not just an argument that it would be nice:\n * Three of improvise-hwpi four fixes are ONE-ASSERTION tests (status text after a bad frame, status text with no peer, one repaint on commit). That ticket itself said \"consider whether the right first move is a headless DOM -- otherwise the fixes are as unverified as the bugs.\" The fixes are now shipped, so that sentence is true of them.\n * The first-frame watchdog was found BY READING, and only because someone went looking for it. The next one will not be.\n * OPFS is the specific thing MemoryFs cannot stand in for, and it is now on the critical path of the DEFAULT deployment rather than a future one. storage_contract in storage.rs exists and is written down precisely so it can be run against OpfsFs in a browser; NOTHING BUT A BROWSER CAN RUN IT.\n * Persistence across reload is the one improvise-bck acceptance criterion no test in this repo can structurally reach.\n\nSCOPE, deliberately small: python3 -m http.server --directory site, plus one Playwright spec, reusing the assemble step CI already performs. Two parts -- (a) storage_contract against OpfsFs, which is the thing improvise-xf1 is actually about; (b) three shell assertions from hwpi.\n\nShould block improvise-bck sign-off. Un-defer or close improvise-xf1 into this, whichever the user prefers -- xf1 predates the worker and is narrower than what is now needed.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:52Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-vvl3","title":"A headless browser harness: run storage_contract and the shell assertions under Playwright","description":"THE CONDITION THAT DEFERRED improvise-xf1 HAS BEEN MET. The user deferred browser testing on 2026-09-20 with: \"I am happy just putting off the browser test for now: WE HAVE TO HAVE SOMETHING TO TEST.\" After improvise-9x6, djm, hwpi and the demo model, we do -- a complete self-contained application whose ENTIRE untested surface is the part a harness would cover.\n\nTHE EVIDENCE THAT IT HAS FLIPPED, not just an argument that it would be nice:\n * Three of improvise-hwpi four fixes are ONE-ASSERTION tests (status text after a bad frame, status text with no peer, one repaint on commit). That ticket itself said \"consider whether the right first move is a headless DOM -- otherwise the fixes are as unverified as the bugs.\" The fixes are now shipped, so that sentence is true of them.\n * The first-frame watchdog was found BY READING, and only because someone went looking for it. The next one will not be.\n * OPFS is the specific thing MemoryFs cannot stand in for, and it is now on the critical path of the DEFAULT deployment rather than a future one. storage_contract in storage.rs exists and is written down precisely so it can be run against OpfsFs in a browser; NOTHING BUT A BROWSER CAN RUN IT.\n * Persistence across reload is the one improvise-bck acceptance criterion no test in this repo can structurally reach.\n\nSCOPE, deliberately small: python3 -m http.server --directory site, plus one Playwright spec, reusing the assemble step CI already performs. Two parts -- (a) storage_contract against OpfsFs, which is the thing improvise-xf1 is actually about; (b) three shell assertions from hwpi.\n\nShould block improvise-bck sign-off. Un-defer or close improvise-xf1 into this, whichever the user prefers -- xf1 predates the worker and is narrower than what is now needed.","notes":"ASSERTIONS OWED BY improvise-taxx's FIX (from WU-I / PR #21, so vvl3 need not re-derive them). The close-listener now distinguishes never-connected from was-connected-and-dropped via an `opened` flag, because readyState is already CLOSED in any close listener and both cases are code 1006 / wasClean:false. Four assertions:\n\n 1. close-before-open -\u003e status stays 'cannot reach \u003curl\u003e ...'\n 2. open-then-close -\u003e status reads 'disconnected'\n 3. ASSERTION 1 MUST FAIL against the previous unconditional close handler. That arm IS the regression guard; without it, 'always say cannot reach' passes.\n 4. after either arm, the handshake watchdog must NOT fire 8s later and overwrite the message.\n\nAlso note WU-I DECLINED Copilot's request to add JS regression tests in PR #21, on the grounds that there is no JavaScript test infrastructure in this repo at all (no package.json, no runner, nothing in CI executing a line of web/*.js) and that a parallel ad-hoc JS surface beside this ticket would be two answers to one question. So this bead now owns that infrastructure decision, not just the OPFS half.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:52Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:09:55Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-gv9r","title":"[bug] A wire model command never marks the document dirty","description":"Found by improvise-9x6 (2026-09-20) the hard way: the worker first persist trigger was `if dirty`, and it SILENTLY NEVER FIRED.\n\nreduce::model_effects maps ModelCommand::SetCell to effect::SetCell, which writes the cell and does NOT touch model_state.dirty. Dirtiness is raised by a SEPARATE MarkDirty effect that only the command/key path emits -- so on every transport, a client edit leaves the authoritative document reporting CLEAN.\n\nTwo consequences, and the second is the one that will bite someone:\n * \"Save if dirty\" is a trigger that never fires. Any host built on it loses every client edit, silently.\n * Anything else that reads model_state.dirty about a client-edited document gets the wrong answer -- an unsaved-changes prompt, an autosave, a UI marker.\n\nWHY IT HAPPENED: MarkDirty being a separate effect is deliberate and reasonable on the key path, where a command composes the pair. The derivation that turned effects into ModelCommands (improvise-kmgs) took the effects ONE AT A TIME, so the pairing was lost -- the wire has SetCell without its MarkDirty companion. That is the interesting shape: a derivation that is correct per element and wrong for a set.\n\nCANDIDATE FIXES, and this wants deciding alongside improvise-9x6 sibling ticket about Save: make model_effects emit the MarkDirty companion where the key path does; or fold dirtiness into the effects that change the document so it cannot be forgotten (the unrepresentable-not-validated answer, and probably right); or decide that dirty is a host-local concept the wire has no opinion on, and then say so where someone will read it.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:28Z","started_at":"2026-09-20T05:32:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-368h","title":"[bug] No client on any transport can ask its host to save","description":"Found by improvise-9x6 (2026-09-20) while building the worker persist trigger, and it affects improvise-ws-server too -- which has shipped without anyone noticing.\n\neffect::Save is EffectKind::ViewOnly, and CORRECTLY so: it writes to THIS HOST file path and clears THIS HOST dirty flag, mutating no document state. But the wire vocabulary is DERIVED from the model effects (gotcha 38), so the derivation cannot reach it. There is no Command::Save and no ModelCommand::Save.\n\nConsequence: a connected client can edit the document and can never cause it to be written. On the socket deployment that means a ws-server session edits in memory and the file on disk never changes unless a human is at a TUI that is not there.\n\nTHIS IS A PROTOCOL DECISION, NOT A PATCH. Two coherent answers:\n * ADD Command::Save. But Save is genuinely ViewOnly and genuinely per-host -- a client asking a host to write to a path the client cannot see is a different thing from a client mutating the document, and it needs an answer for \"which path\" and \"what if there is none\" (improvise-lae already exists for the no-path case).\n * SAY EXPLICITLY THAT HOSTS OWN PERSISTENCE, and make each host trigger on its own terms. That is what improvise-9x6 did for the worker, and in a browser it is arguably RIGHT rather than a workaround: there is no file the user owns and no Ctrl-S semantics -- the document IS the OPFS entry. But then ws-server needs its own answer, and today it has none.\n\nDecide which. The gap is that neither was chosen, so the socket server silently does nothing.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:28Z","started_at":"2026-09-20T05:32:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ondj","title":"[bug] reduce_full/apply_command apply a Command::Model without the basis check","description":"Found by the code review of ADR 001 layer 1 (2026-09-19). CONFIRMED, not live today, and it should be closed BEFORE the browser worker lands.\n\nui/reduce.rs is documented as \"the one place a command is applied to the authoritative state\" and \"Every deployment mode -- the native TUI, a websocket server, a browser worker -- reaches the model through the functions here\". Both pub fn apply_command and pub fn reduce_full accept a \u0026Command including Command::Model, and:\n\n Command::Model { cmd, .. } =\u003e { // the basis is DISCARDED by `..`\n let effects = model_effects(app, cmd);\n app.apply_effects(effects)\n }\n\nThe check lives ONLY in Sessions::apply_model.\n\nWHY THIS IS THE INTERESTING SHAPE: making Command::Model a struct variant made the basis impossible to OMIT FROM THE MESSAGE. It did nothing to make it impossible to IGNORE ON ARRIVAL -- and `..` in a pattern is exactly the syntax for ignoring it, with no compile error. So the design got half of what a struct variant is usually reached for.\n\nNothing in-tree routes a client command through reduce_full (only tests), so nothing is broken now. But improvise-9x6 (the browser worker) is precisely the host that will reach for the function the module doc points it at, and it would apply client-supplied model commands unchecked.\n\nREMEDIES, pick deliberately:\n * Have apply_command take a ModelCommand from a private caller and expose only Sessions::apply for a wire Command -- makes the door unreachable rather than guarded.\n * Or have reduce_full return a Rejected-shaped result for Command::Model.\n * At absolute minimum, a compile-visible note AND a test that names the door, so the next person meets it.\n\nThe first is the one that matches this repo: invariants are made unrepresentable, not validated (doctrine(an_invariant_is_made_unrepresentable_not_validated)).","status":"closed","priority":1,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:54:29Z","closed_at":"2026-09-20T01:54:29Z","close_reason":"Fixed as option 1 -- make the door unreachable -- but as a TYPE rather than as visibility.\n\napply_command and reduce_full now take Checked\u003c'a\u003e, a private-field newtype over a \u0026Command whose ONLY constructor is Checked::against(\u0026App, \u0026Command) -\u003e Result\u003cSelf, Rejection\u003e, which is also the only caller of stale_basis. So there is no longer a spelling of 'apply this batch and ignore what it claims': the that discarded the basis is now a Rejection the caller must handle. Sessions::apply answers ONCE, before the dispatch, and hands the proof to apply_switch/apply_model -- so neither has to remember to ask, and apply_model lost its basis/model parameters entirely.\n\nWHY NOT pub(crate), which is what the ADR literally suggested: it works, but reduce_full then has NO non-test caller anywhere in the workspace, so it becomes a dead_code warning in the non-test build, and the only ways out are deleting a documented host API or cfg(test)-gating it. The newtype keeps both functions public and useful to a future host -- including the browser worker, which now gets a door that WORKS and is safe -- while making the unchecked path unconstructible. It also holds INSIDE the crate, which visibility does not: a second in-crate mutation path cannot be written without the check either.\n\nEVERYTHING THE TICKET SAID MUST SURVIVE, DOES: the check is still a property of the session door and not of every mutation (the key path, model_effects and App::apply_effects are untouched, and a host's own ModelCommand still needs no basis); caught_up(\u0026App, cmd) is unchanged and still not a bypass; the 10 basis_check tests are untouched and all pass.\n\nTHE REGRESSION GUARD IS THE SIGNATURE. The demonstration test (committed first, failed on the old code) NO LONGER COMPILES after the fix, which is the better outcome. What survives asserts that Checked::against returns Err(Rejection::LayoutMoved) and the document is untouched, with a doc-comment saying in prose that the signature is the real guard. Check-the-checker: making against() never refuse fails 8 tests -- the new one plus 7 of the 10 basis_check ones.\n\nNATIVE CLI EXERCISED FOR REAL, not merely compiled: improvise cmd and improvise script both land their changes on disk. They go through the keymap/registry path and never touched reduce_full.\n\nTHE THING THE TICKET DID NOT PREDICT, and it is the reason this defect could sit unnoticed: reduce_full HAS NO PRODUCTION CALLER IN THE ENTIRE WORKSPACE, and apply_command's view-slice arm is unreached outside tests. Every host reaches the model through Sessions::apply or reduce_key. So ui/reduce.rs's 'the signature the three hosts share' is ASPIRATIONAL -- a door pointed at by documentation and used by nobody, which is exactly the shape that hides a defect until the next host arrives. Filed as a successor.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -36,8 +36,10 @@ {"_type":"issue","id":"improvise-km8","title":"1.2 Fix Cargo.toml metadata","description":"Add missing package fields: description, repository, homepage, documentation, readme, keywords, categories, license. Description: 'Terminal pivot-table modeling in the spirit of Lotus Improv'. Keywords: tui, pivot, spreadsheet, data, improv. Categories: command-line-utilities, visualization.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:48Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:20:17Z","close_reason":"Metadata already present: description, license, repository, homepage, readme, keywords, categories all populated.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-902","title":"1.1 Audit and remove or update context/SPEC.md","description":"Compare context/SPEC.md against actual code. If spec contradicts code significantly, delete it. Optionally move salvageable conceptual content to docs/design-notes.md with a staleness disclaimer. Do not try to bring spec into sync.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:46Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:10:05Z","close_reason":"Audited SPEC.md against code. Found minor inaccuracies (storage internals, wizard step count, search mode representation) but no major architectural contradictions. Deleted SPEC.md since it was largely redundant with repo-map.md and design-principles.md. Salvaged product vision and non-goals into docs/design-notes.md with staleness disclaimer.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-tik","title":"Phase 1: Repository hygiene","description":"Raise the quality floor of what a stranger sees when they land on the repo. Subtasks cover spec audit, Cargo.toml metadata, publish-readiness, CSV audit, and example files.","status":"closed","priority":1,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:45Z","created_by":"Edward Langley","updated_at":"2026-04-09T07:22:34Z","closed_at":"2026-04-09T07:22:34Z","close_reason":"All 5 subtasks complete: SPEC.md audited, Cargo.toml metadata fixed, publish dry-run clean, CSV audit done, example files created.","dependencies":[{"issue_id":"improvise-tik","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T23:37:31Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-902","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-bv1","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-km8","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-n1h","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-sihr","title":"Every minibuffer mode entered from records mode exits to plain Normal while the layout is still records","description":"Found by Copilot on PR #27 against the new Ctrl-W save prompt, and DECLINED there with evidence because fixing the one arm the finding named would have been one arm of a three-arm rule.\n\nTHE BEHAVIOUR: every minibuffer mode entered from RecordsNormal exits to plain Normal while the LAYOUT IS STILL RECORDS. It affects `:` (CommandMode), Ctrl-E (ExportPrompt) and Ctrl-W (SavePrompt, new in PR #27). Only the last is new; the class predates it.\n\nTHE MECHANISM: the exit mode is a CONSTANT, baked into TextEntry::new(.., \"normal\") in the keymap builder and into each commit command's ChangeMode. Nothing carries where the user came FROM, so nothing can put them back.\n\nWHAT IS NOT THE MECHANISM, checked rather than reasoned (doctrine(binding_dump_diff)): the binding TABLE is fine. RecordsNormal goes 155 -\u003e 156 dumped lines across PR #27, the single addition being Char('w')+CONTROL -\u003e enter-mode [\"save-prompt\"], with nothing removed. with_parent inheritance (gotcha 13) is intact. What is lost is the RUNTIME MODE, not a binding.\n\nCHARACTERIZED, NOT FIXED: every_minibuffer_mode_entered_from_records_returns_to_plain_normal drives all three and asserts both halves -- the mode is Normal, and the layout is still records. So the current behaviour is pinned and a fix will have to update it deliberately.\n\nTHE FIX, and it is why this is its own bead: text-entry modes must carry their entry mode. That is a change to TextEntry, to AppMode (the exit is part of a mode's configuration, so it likely belongs beside MinibufferConfig -- note gotcha 11, MinibufferConfig::buffer_key is a String and not \u0026'static str precisely because ViewState must stay Deserialize-able, and the same constraint applies to anything added there), and to three commit commands. Wider than the PR that found it.\n\nWATCH TWO THINGS:\n * The exit is currently spelled in TWO places per mode -- the TextEntry spine's Esc binding and the commit command's ChangeMode. If they are made to carry an entry mode, make them read ONE source or the two will disagree on cancel versus commit.\n * SearchMode is hand-written on purpose and has no commit key (gotcha 11). It is not part of this class; do not sweep it in.\n\nRELATED: improvise-mai9 (a status message written while a minibuffer is open is never drawn) is the other half of why these modes are awkward -- both are consequences of the minibuffer owning the bottom row and nothing tracking what it displaced.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:12:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:12:20Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-kc7n","title":"improvise-ws-server never checks Origin, so any page a user visits can drive their session","description":"Found by Copilot on PR #21 round 5, against a reverse-proxy recipe whose IP allow-list had itself been added two rounds earlier to close a DIFFERENT hole. Verified independently by the orchestrator: `grep -rni origin crates/improvise-ws-server/src/` returns NOTHING. The server never inspects the Origin header and accepts every handshake.\n\nTHE HOLE: a WebSocket is EXEMPT FROM THE SAME-ORIGIN POLICY -- there is no CORS preflight on a handshake. So any hostile page that an allowed user visits can open a socket to the endpoint FROM THAT USER'S OWN ALLOWED ADDRESS, and read and edit every document the server holds. That is cross-site WebSocket hijacking.\n\nWHY NETWORK CONTROLS CANNOT SEE IT: an IP allow-list, a private network, loopback binding -- all answer \"may this CLIENT reach the endpoint\", and in this attack the answer is genuinely yes. The request really does come from an allowed client; it is simply being DRIVEN by a page the user did not intend to trust. Only Origin separates the two.\n\nLOOPBACK IS NOT EXEMPT, and this is the half a documentation-only fix cannot reach. In the local pairing case there is no proxy at all, so there is nothing in front to check Origin -- and loopback is not a boundary a browser enforces. While the server runs, ANY page open in that browser can reach ws://127.0.0.1:8787. So the local development path, which is the one the project actually recommends, is the least protected.\n\nDISTINCT FROM improvise-kpnr, and both are needed. kpnr is AUTHENTICATION: who is allowed to talk to this server at all. This is CSWSH: given that someone is allowed, what may put words in their mouth. An answer to either leaves the other open.\n\nTHE FIX IS CHEAP AND SERVER-SIDE: an allow-list of permitted origins checked at the handshake, defaulting to something safe for the local case (the page's own origin, and whatever --bind implies), with a flag to widen it. Refuse the upgrade rather than accepting and ignoring.\n\nTWO THINGS TO GET RIGHT:\n * A MISSING Origin header is not the same as a disallowed one. Non-browser clients (the repo's own tests, curl, a native client) send none, and the browser always does. Decide explicitly what absent means rather than letting the match arm decide by accident.\n * PR #21 has already documented Origin filtering in the PROXY as a required companion to the network control, phrased as \"in addition to, never instead of\". When this lands, the README should say the server does it too, and the proxy line becomes defence in depth rather than the only defence.\n\nTHE GENERAL LESSON, now doctrine(who_may_connect_is_two_questions): ask of any access control whether it constrains the CLIENT or what may DRIVE the client. Three consecutive rounds of that PR reasoned about the page's capabilities and none about the attacker's.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:09:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:09:42Z","dependencies":[{"issue_id":"improvise-kc7n","depends_on_id":"improvise-kpnr","type":"relates-to","created_at":"2026-09-20T06:09:49Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-mai9","title":"A status message written while a minibuffer is open is never drawn","description":"Found by WU-G while building improvise-lae (PR #27), and CONFIRMED THE HARD WAY in the same PR: Copilot then found a regression that this mechanism causes, which the agent had itself written down an hour earlier and not checked its own change against.\n\nTHE MECHANISM: src/draw.rs::draw_bottom_bar renders the minibuffer INSTEAD OF the status line. So a status_msg written while a minibuffer is open is never drawn at all -- not truncated, not overwritten, simply never rendered.\n\nWHO IS AFFECTED TODAY:\n * CategoryAdd / ItemAdd: a failed `commit-buffer category` is INVISIBLE. Worse, improvise-6t3 deliberately ordered that command's effects so the add effect is emitted LAST, specifically so its failure message would WIN over the command's optimistic status. It wins a race to a surface nobody can see.\n * A failed `:w \u003cpath\u003e` from CommandMode, which is the regression Copilot found on PR #27: the abort stops the batch before ReturnToNormalUnlessMoved and clear-buffer run, so the app stays in CommandMode with the stale command in the minibuffer and the new error hidden behind it.\n\nTHE WORKAROUND ALREADY SHIPPED, and it is a workaround: CommitSave emits [ChangeMode(Normal), SaveAs] so the error lands when the bottom row is the status bar again. That works for one command and does not generalise -- every other text-entry commit has the same hole.\n\nTWO CANDIDATE FIXES, and the choice is the ticket:\n * Draw both: give the minibuffer and the status line separate rows, or render the status into the minibuffer row when a message is pending. Cheap to describe, and it changes frame geometry, which is pinned by exact-frame characterization tests (see improvise-q0vb for what that costs).\n * Make it unrepresentable: a status message set while a minibuffer is open is a bug in the CALLER, so make the caller unable to do it -- e.g. commit paths always leave the mode before reporting. That is what CommitSave now does by hand, promoted to a rule.\n\nPREFER WHICHEVER MAKES THE FAILURE LOUD. The current state is the quiet kind: the code sets a message, the test asserts the message was set, and the user never sees it. A test asserting `app.status_msg == \"...\"` passes in every one of these cases, which is why this survived improvise-6t3's careful effect ordering.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:20Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependencies":[{"issue_id":"improvise-kpnr","depends_on_id":"improvise-kc7n","type":"relates-to","created_at":"2026-09-20T06:09:49Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-p1ym","title":"Project the page-axis item lists so [ and ] work in a browser","description":"The one key pair an independent smoke test found inert that the shared keymap did NOT fix, and it is a genuine projection gap rather than a resolution gap.\n\nGridCache::page_coords carries (category, SELECTED ITEM). PageNext/PagePrev need THE ITEM LIST TO CYCLE THROUGH, which nothing projects. So [ and ] resolve to a command name the client holds but cannot compose an argument for, and they reach Host::handle empty vec -- doctrine(an_upstream_key_is_a_black_hole), so they do nothing at all.\n\nCONFIRMED BY SMOKE TEST 2026-09-20 against the live page: \"[ / ] did not cycle Quarter (never saw Q2 ~17,100)\". The demo model puts Quarter on the page axis specifically so paging is a thing a visitor would try.\n\nTHE AGENT WAS TOLD NOT TO FORCE IT and did not -- a truthful \"these four keys work, [ / ] needs X\" is worth more than five half-working keys.\n\nSCOPE: extend page_coords (or add a sibling) to carry the item list per page-axis category. Note gotcha 27 -- \"what data is visible\" is View::filtering_cats() (Page UNION Filter) while \"what the user can page\" is categories_on(Axis::Page). This needs the SECOND one; projecting the first would let a client try to cycle a Filter axis, which by design it must not.\n\nWhile you are there: check whether the item list should be the EFFECTIVE items (hidden items excluded) rather than all of them, since [ and ] in the TUI cycle what the user can actually select.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-955n","title":"[bug] Host::commands applies a batch partially before rejecting","description":"Found by Copilot on PR #13 (the retro review of the unreviewed batch), DECLINED there with reasons on the thread, and filed rather than patched.\n\nADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a TYPE (Checked) to make the unchecked path unconstructible. But Checked is PER-COMMAND: Checked::against answers for one Command against the App AS IT STANDS. A ClientFrame::Commands is a SEQUENCE of them, and Host::commands applies in order and abandons at the first rejection. So a frame carrying SwitchView followed by a stale Command::Model has already applied the SwitchView.\n\nSO ADR 001 GUARANTEE IS CURRENTLY TRUE OF A SINGLE COMMAND AND FALSE OF A BATCH.\n\nA SECOND-ORDER EFFECT NOBODY HAD WRITTEN DOWN, found while declining it: Worker::on_frame skips persist() when ANY frame in the answer is Rejected. So a batch that DID apply a SwitchView before being refused leaves Workbook::active_view moved in memory and NOT WRITTEN BACK -- the document on disk and the document in the worker disagree, silently.\n\nWHY IT WAS DECLINED RATHER THAN PATCHED: closing it means the basis check moving from the COMMAND to the BATCH -- every command answered before any is applied. That is an ADR 001 design change with a real question inside it: WHAT DOES A BATCH BASIS MEAN WHEN ITS LATER COMMANDS WERE COMPOSED AGAINST ITS EARLIER ONES EFFECTS? A client resolving one keypress produces SetSelected plus offsets computed from the selection it just set. Answer that before writing code.\n\nREACHABILITY IS BOUNDED TODAY: no client in this workspace composes a batch containing more than one basis-carrying Command::Model -- the batches improvise-bpq2 produces are view commands, which carry no basis. Do this before the second tranche of client-side resolution creates a reason to batch two model commands.\n\nSupersedes the earlier ticket filed from bpq2 if there is one -- dedupe.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:29Z","started_at":"2026-09-20T05:32:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -151,6 +153,9 @@ {"_type":"issue","id":"improvise-pby","title":"2.2 Write the README","description":"Replace existing README.md with new one under 250 lines. Sections in order: Title, one-sentence pitch, inline demo GIF, why this exists (Lotus Improv reference), quick start, key bindings, installation (Nix/crates.io/prebuilt - no Homebrew), codebase overview, expectations disclaimer, license. No badges, TOC, or contributing guide.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:32Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:02:52Z","close_reason":"README written with all 10 required sections, committed. License updated to Apache-2.0 per user request.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ihv","title":"2.1 Add Nix tooling for asciinema, VHS, and cargo-dist","description":"Modify flake.nix to add pkgs.asciinema and pkgs.vhs to dev shell nativeBuildInputs. Add cargo-dist if packaged in nixpkgs. Add runtime deps for VHS (ttyd, ffmpeg) if needed. Verify tools work via nix develop --command. Add nix run app or shell script for demo recording workflow.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:30Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:30:19Z","close_reason":"asciinema, vhs, cargo-dist added to flake.nix dev shell and verified. cargo-dist wrapper enables 'cargo dist' subcommand. scripts/record-demo.sh added.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-yk4","title":"Phase 2: README and demo artifacts","description":"The main launch work. README is 80% of the launch, demo artifacts are 20%. Covers Nix tooling for asciinema/VHS, writing the README, creating demo GIF and asciinema casts, and verifying all artifacts.","status":"closed","priority":2,"issue_type":"feature","assignee":"spot","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:49Z","created_by":"Edward Langley","updated_at":"2026-04-14T07:47:43Z","closed_at":"2026-04-14T07:47:43Z","close_reason":"All 5 subtasks closed; artifacts verified present and tracked in git (README.md, docs/demo.gif, docs/demo.tape, docs/casts/*.cast).","dependencies":[{"issue_id":"improvise-yk4","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T23:37:37Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-ihv","type":"blocks","created_at":"2026-04-08T23:37:34Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-odx","type":"blocks","created_at":"2026-04-08T23:37:35Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-pby","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-rw3a","title":"Decide whether ZZ routes into the save prompt automatically","description":"The half of improvise-lae that was deliberately left open. PR #27 built the prompt (AppMode::SavePrompt, TextEntry builder, committing to SaveAs, bound Ctrl-W and reachable as `:enter-mode save-prompt`) and stopped short of making ZZ route into it.\n\nTODAY, after PR #27: ZZ on a pathless document refuses, stays in the app, and the status line names the prompt. That is already better than improvise-8tk left it -- the user is told where to go. The open question is whether being told is enough, or whether the vim reflex should just land somewhere useful.\n\nTHE OBSTACLE THE ORIGINAL TICKET NAMED, still true: CmdContext does not expose file_path, so a Cmd cannot currently decide \"is there a path\" at execute time. Adding the field is small; note improvise-2ud is separately proposing that CmdContext borrow the ModelState/ViewState slices rather than copying ~20 individual fields, which would make this free. Coordinate rather than adding a 21st copied field ahead of it.\n\nTHE CONSTRAINT PR #27 ESTABLISHED, and it must be respected by whatever lands here: the prompt is SESSION-LOCAL and was deliberately kept in the COMMAND layer rather than routed through effect::Save. Save is becoming a wire command (improvise-368h), and a prompt is addressed to precisely one session -- routing it through the effect would open a prompt on a host nobody asked, which is gotcha 38's argument for keeping AcceptWizardModel off the wire. So: a Cmd that inspects the path and emits a mode change, never an effect that decides to prompt.\n\nDECIDE ALSO what ZZ means once the prompt is open. ZZ is \"save and quit\"; if the prompt commits to SaveAs and the batch was aborted, the quit is gone. Should committing the path then quit? That is a second composition and the answer is not obviously yes -- a user who hit ZZ, got a prompt, and typed a path plausibly wants to leave, but making the prompt's Enter mode-dependent is the kind of hidden state improvise-ct5 just declined to add for `i`.\n","status":"open","priority":3,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:32Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:32Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-g9jb","title":"The text-entry mode list is hand-written in two crates and checked against AppMode in neither","description":"Found by WU-G on PR #27 while adding AppMode::SavePrompt.\n\nThe set of text-entry modes is written down BY HAND IN TWO PLACES, in two crates:\n * improvise-command::keymap::tests::text_entry_modes_with_exits\n * improvise-wasm-client::keys::tests::text_entry_modes\n\nand NOTHING CHECKS EITHER AGAINST AppMode. So adding a text-entry variant compiles, passes, and is silently absent from both lists -- which matters more since PR #27, because the browser now refuses to ENTER a prompt it cannot commit (keys.rs::strands_the_user), and that refusal is driven off one of these hand-written lists.\n\nTHE FAILURE MODE IS THE QUIET ONE: a new prompt mode omitted from the wasm list is one a browser will happily open and then swallow the user's typing in -- doctrine(a_doorway_to_a_black_hole_is_worse_than_the_key), which PR #27 established against SavePrompt and found ExportPrompt had already.\n\nTHE FIX: derive the list once, from AppMode itself. `MinibufferConfig` is already the thing that makes a variant text-entry -- AppMode::minibuffer() is an exhaustive match returning Option\u003cMinibufferConfig\u003e (gotcha 11), so \"the text-entry modes\" is derivable rather than enumerable. A const or a function in improvise-protocol beside AppMode, with both crates reading it, replaces both lists.\n\nCHECK ONE THING BEFORE ASSUMING THE DERIVATION IS FREE: the wasm list and the command list may not be the same set for a reason -- SearchMode is hand-written on purpose and has no commit key (gotcha 11), and the browser's question (\"can this client commit here\") is not identical to the keymap's (\"does this mode take text\"). If they genuinely differ, the answer is two derived predicates over one enum, not two hand-written lists.\n\ndoctrine(a_hand_enumerated_mirror_needs_an_exhaustive_match) already covers this; this is an instance with no match behind it at all.\n","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-z05r","title":"CommitExport does not bail on an empty path, and ExportCsv does not abort the batch","description":"Found by WU-G on PR #27 while fixing the save arms, and left deliberately: these are the same two defects one file over, at lower severity, and fixing them in that PR would have widened a save-focused change into an export-focused one.\n\nTWO ARMS, both mirroring what was just fixed for save:\n 1. CommitExport does not bail on an empty path, where CommitBuffer's three kinds all do (gotcha 20: buffer -\u003e trim -\u003e bail-on-empty -\u003e dirty -\u003e status is the shared spine).\n 2. ExportCsv does not set abort_effects on failure, where Save and (as of PR #27) SaveAs both do. So a `Binding::Sequence` or a `:` line composing an export with anything after it continues past a failed write.\n\nWHY IT IS LOWER SEVERITY THAN THE SAVE ARMS, and this is the honest reason it was deferred rather than a claim that it does not matter: an export failure discards nothing. improvise-8tk was a P1 because ZZ quit and the model was gone. Here the model is untouched and the user has simply not got the CSV they asked for -- so the cost is a missing file and a message, not lost work.\n\nNOTE THE SHAPE, because it is the third instance in this area: improvise-8tk fixed Save and not SaveAs; PR #27 found SaveAs; and this is the export arm of the same rule. doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only) exists for exactly this, and the general fix may be better than the third patch -- consider whether \"an effect that writes a file and fails aborts the batch\" can be made structural rather than remembered three times.\n\nALSO CHECK, while in there: whether the export error is visible at all, given that ExportPrompt is a minibuffer mode and a status message written while a minibuffer is open is never drawn (see the sibling bead on draw_bottom_bar).\n","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:21Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:21Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-44ad","title":"ColWidths is redundant under the label fold, and the test that would change is gotcha 33(c)'s evidence","description":"Found while closing improvise-hkt6 (PR #20, WU-K), tried, and DELIBERATELY REVERTED rather than landed -- filed so it is a decision rather than a rediscovery.\n\nhkt6(b) obliged grid_updates to diff `totals` against the LABEL-FOLDED cache rather than against the mirror (see gotcha 33). The symmetric move for `col_widths` looks obviously right and is not free: it BREAKS `a_client_on_a_socket_is_seeded_and_gets_its_edit_back`.\n\nWHY, and it is a real property rather than a brittle assertion: folding the labels RE-FITS the widths (gotcha 33(c) -- every arm that touches labels or widths re-fits, because a renderer zips widths with labels and a length mismatch indexes off the end). So under a fold-relative diff the ColWidths that a reframe currently carries becomes REDUNDANT and is suppressed. The client is correct either way; what changes is whether the reframe still SENDS the widths.\n\nTHE REASON IT WAS NOT SPENT IN PR #20: that assertion is gotcha 33(c)'s standing evidence -- it is the test that catches \"a late ColWidths put the mismatch straight back\". Rewriting it inside a PR whose subject was three totality gaps would have destroyed the evidence for a rule while fixing something else, which is exactly the shape improvise-q0vb was declined for on PR #12. Two right changes, different costs.\n\nTHE DECISION TO MAKE:\n * Leave widths diffed against the mirror. Costs one redundant ColWidths per reframe -- the same direction Dirty::of is deliberately sloppy in (gotcha 35: over-report pays one redundant Projection, under-report leaves a client wrong forever). Defensible and arguably correct on that principle alone.\n * Or move widths to the fold and REPLACE the socket assertion with one that still pins 33(c)'s property -- that a client never holds widths of a different length from its labels -- rather than one that pins the message count. That is the better test either way, and writing it is most of the work.\n\nPREFER THE SECOND ONLY IF the replacement test is written first and shown to fail against a deliberately late ColWidths. Otherwise prefer the first and say so in gotcha 33, so the asymmetry between totals and widths is STATED rather than discovered by the next person who notices it.\n","status":"open","priority":3,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:54:56Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:56Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-j7v6","title":"The typed-string classification lives in two crates, and its guard test asserts by hand","description":"Named by improvise-r4lo (PR #21) while deleting the OTHER duplication, and deliberately left rather than folded in -- r4lo was about two answers on the CLIENT side, and deleting Client::set_cell/set_cell_at settled that. This is the duplication one level out, and it crosses a crate boundary.\n\nwasm-client/src/socket.rs::typed classifies a user-typed string into a CellValue. So does improvise-command's commit_regular_cell_value, on the host's own key path. Two implementations of \"what does this string mean\", in two crates, with nothing holding them together.\n\nBOTH ARE LEGITIMATE, which is why this is not simply r4lo again. The browser has no App to ask and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; a host typing locally has an App and classifies there. The defect is that the RULE is written twice, not that it is applied in two places.\n\nTHE TEST THAT LOOKS LIKE A GUARD AND IS NOT: a_typed_string_is_classified_as_the_host_classifies_it states the obligation IN ITS NAME and does not discharge it -- it asserts three outcomes BY HAND rather than comparing against the host's function. So the two can drift and the test stays green, which is this repository's characteristic defect (doctrine(check_the_checker)): a test that passes while the thing it names is broken.\n\nTHE FIX: one classification function in improvise-protocol, which both crates already depend on, and both call sites calling it. Then the test either disappears (type identity replaces it) or becomes a real comparison. Note doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished) -- if the shared function lands and the hand-written three-outcome test survives beside it, the consolidation is not done.\n\nWATCH ONE THING WHILE MOVING IT: the classification must stay consistent with what the .improv format considers a number versus pipe-quoted text (gotcha 34, and format.rs::pipe_quote exists precisely so a text cell cannot read back as a number). A shared classifier that disagrees with the persistence layer would be a third answer wearing the badge of the first.\n","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-rucf.3","title":"Projection freshness for a source an external writer can change","description":"Child of improvise-rucf (ADR 003, live-backed models). Filed and DEFERRED 2026-09-19 alongside its parent.\n\nADR 001 governs concurrent EDITS. A live-backed model is read-only, so it has none and the basis check does not apply to it -- the guarantee is not voided, it is scoped to documents improvise owns, which is what it always meant.\n\nWHAT REMAINS IS A DIFFERENT AND SMALLER QUESTION, and this bead is it: an EXTERNAL WRITER changes what clients SEE. So a live-backed projection needs a freshness story -- polled, triggered, or explicitly stale-until-refreshed. That is projection freshness, not conflict resolution, and conflating the two is the trap.\n\nNOTE WHAT CHANGES ABOUT THE PROJECTION ARCHITECTURE: today's diff (ui/project.rs::grid_updates) compares a desired state derived from the App against each client's mirror. With a live source, \"the desired state\" involves latency -- a HashMap lookup becomes a query -- so the VIEWPORT-BOUNDED PROJECTION STOPS BEING AN OPTIMISATION AND BECOMES LOAD-BEARING. improvise-i5tk (narrow the TUI's GridCache to its viewport, benchmark first) is the related bead and its answer may be forced by this one.\n\nSHARED WITH THE SQLITE SOURCE CHILD: the RecordHash -\u003e record index becomes a cache needing invalidation rather than a structure maintained alongside writes. One invalidation story, not two.\n\nEXPLICITLY OUT OF SCOPE: anything resembling a lease, a lock, or a reservation. Those are ADR 001 layer 2 and they are about edits.\n","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:35Z","dependencies":[{"issue_id":"improvise-rucf.3","depends_on_id":"improvise-rucf","type":"parent-child","created_at":"2026-09-20T05:31:33Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -248,6 +253,7 @@ {"_type":"issue","id":"improvise-l36","title":"3.2 Publish to crates.io","description":"After cargo publish --dry-run is clean and user confirms, run cargo publish. Verify crate appears at crates.io/crates/improvise and cargo install improvise works.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:58Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-l36","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T21:09:29Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-l36","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-11a","title":"3.1 Configure cargo dist","description":"Run cargo dist init targeting x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin. Skip Windows and musl. Commit generated .github/workflows/release.yml and Cargo.toml additions. Test with v0.1.0-rc1 tag, verify builds, delete rc tag.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:55Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-11T07:49:57Z","close_reason":"Closed","dependencies":[{"issue_id":"improvise-11a","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-0s6","title":"Phase 3: Distribution","description":"Configure cargo dist, publish to crates.io, tag v0.1.0 release. Produces prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:51Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-0s6","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T23:37:39Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-3tj","type":"blocks","created_at":"2026-04-08T23:37:41Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-l36","type":"blocks","created_at":"2026-04-08T23:37:40Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-zx8t","title":"draw.rs::mode_style matches AppMode with a wildcard, so the repo map's exhaustiveness promise is false there","description":"Found by WU-G on PR #27 while adding AppMode::SavePrompt, and it is a correction to something context/repo-map.md currently promises.\n\nThe repo map says of AppMode: \"Per-variant UI facts live on the enum as exhaustive-match methods -- label(), hint_text(), panel(), minibuffer(), is_editing(), is_records() (improvise-2hi); ADD A VARIANT AND THE COMPILER NAMES EVERY ONE YOU MUST FILL IN.\"\n\nThat is TRUE of the five enum methods and FALSE of src/draw.rs::mode_style, which matches AppMode with a `_ =\u003e` WILDCARD. So a new variant silently inherits whatever the wildcard arm gives -- SavePrompt currently takes ExportPrompt's DarkGray by accident rather than by decision, which happens to be defensible and was nobody's choice.\n\nWHY IT IS P4 AND NOT HIGHER: the consequence is a colour, and the wrong colour is visible rather than silent. But the reason to fix it is that the repo map states a PROPERTY, and a property that holds in five places and not the sixth is worse than one nobody claimed -- the next person adding a variant will trust the sentence.\n\nTHE FIX IS ONE OF TWO, and they are not equivalent:\n * Make mode_style exhaustive, so the compiler does name it. Note mode_style stays in src/draw.rs deliberately, because Style is a ratatui type and improvise-protocol must not name one (gotcha 26, the AxisColor/MinibufColor pattern).\n * Or give AppMode a semantic colour method in the protocol crate -- the AxisColor treatment -- and let draw.rs do the one mapping. That is the established shape here and it would also give a browser something to use, which draw.rs's Style cannot.\n\nThe second is probably right and is the same question improvise-k099 asks about hint_text/label. Decide them together.\n","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} From bd3f89a74c68b6ed9be4abcaea59ed94c6fc369a Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:22:15 -0700 Subject: [PATCH 63/72] chore(beads): close the four WU-I beads with their reasoning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-oai9 closed on evidence with no code written — the report() helper was already in the tree, and the fault was injected ({e:#} → {e}) before closing rather than trusting the description. improvise-r4lo, improvise-taxx and improvise-o8fa closed by PR #21. Two of the three record that the TICKET was wrong rather than merely that the bug is fixed: r4lo's offered escape hatch does not exist, and taxx's proposed readyState mechanism cannot work because both cases arrive as code 1006 with wasClean:false. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 3fffc45f..a774b7d6 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,5 +1,5 @@ {"_type":"issue","id":"improvise-0yq8","title":"Move ClientState and keys.rs::step into improvise-protocol","description":"THE SINGLE HIGHEST-VALUE CONSOLIDATION LEFT, named by the agent that did the keymap move and measured rather than guessed.\n\ndesign-principles 5 (user ruling 2026-09-20): minimise the code a new UI framework needs; centralise logic in a redux-like store so a new front end DISPLAYS STATE rather than reimplementing behaviours. The test is: would a third front end have to write this again?\n\nkeys.rs::step is 99 LINES OF \"WHAT THIS COMMAND DOES\" and the answer is YES. It depends on NOTHING but ViewState + the protocol widget caches + the announced window -- every one of which is ALREADY A PROTOCOL TYPE. So it is in the wrong crate, and the only reason it is there is history.\n\nAFTER THIS MOVE A NEW FRONT END NEEDS EXACTLY TWO THINGS: browser-event (or terminal-event, or whatever) -\u003e KeyEvent, and painting. That is the whole of design-principles 5 made true rather than aspirational.\n\nWHY IT WAS NOT DONE IN THE SAME PASS: it is a crate move on top of a crate move (the keymap had just moved), and half-doing it would leave two homes. Deliberate, not forgotten.\n\nMEASURED CONTEXT from that pass: keys.rs grew 264 -\u003e 401 non-comment lines while GAINING the whole editing path, search typing, panels, transpose and tile select -- and the key TABLE left the crate entirely. web/app.js, the only genuinely framework-bound file, grew 344 -\u003e 359 while gaining a live editor and a hint line. So the trend is right; this ticket is what finishes it.\n\nNOTE THE SECOND-CONSUMER ARGUMENT: the more of the front end is SHARED, the more of it a second consumer can AUDIT. CellDisplay::edit_text exists because the browser asked a question the TUI had answered privately and the two answers differed -- see doctrine(being_the_second_consumer_audits_the_abstraction).","status":"open","priority":1,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-oai9","title":"[bug] The worker's errors reach the console with their cause stripped","description":"FOUND BY THE FIRST REAL BROWSER RUN of the deployed page (2026-09-20), and it is the reason that run produced no actionable information.\n\nThe console said:\n\n improvise worker: failed to start\n Error: Cannot open demo.improv\n\n\"Cannot open demo.improv\" is the CONTEXT vfs::open attaches with with_context. anyhow Display prints ONLY THE OUTERMOST CONTEXT, so bindings::start mapping through e.to_string() kept the label and threw away the cause -- the Storage failure or the parse failure that actually happened.\n\nWHY THIS IS THE WORST POSSIBLE PLACE FOR IT: bindings.rs is cfg(target_arch = wasm32) and is the one module in the crate that NO TEST IN THE REPOSITORY CAN REACH. A devtools console is the only instrument pointed at it. So the single code path with no test coverage was also the one discarding its own diagnostics.\n\nFIXED: a report() helper rendering with the alternate form {:#}, which walks the chain joining with \": \". It lives in lib.rs rather than bindings.rs SO IT CAN BE TESTED AT ALL -- bindings is wasm-gated. Five call sites (start, on_message, flush, documents, refresh).\n\nTEST WRITTEN AGAINST THE OLD RENDERING FIRST: a_reported_error_carries_its_cause_not_just_its_label asserts that anyhow to_string() IS just the label (pinning the trap, so the helper stops being needed if that ever changes) and that report() contains BOTH the label and the cause.\n\nTHIS DOES NOT FIX THE UNDERLYING FAILURE -- it makes the next reload name it. Redeploy and read the console again.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:55:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-oai9","title":"[bug] The worker's errors reach the console with their cause stripped","description":"FOUND BY THE FIRST REAL BROWSER RUN of the deployed page (2026-09-20), and it is the reason that run produced no actionable information.\n\nThe console said:\n\n improvise worker: failed to start\n Error: Cannot open demo.improv\n\n\"Cannot open demo.improv\" is the CONTEXT vfs::open attaches with with_context. anyhow Display prints ONLY THE OUTERMOST CONTEXT, so bindings::start mapping through e.to_string() kept the label and threw away the cause -- the Storage failure or the parse failure that actually happened.\n\nWHY THIS IS THE WORST POSSIBLE PLACE FOR IT: bindings.rs is cfg(target_arch = wasm32) and is the one module in the crate that NO TEST IN THE REPOSITORY CAN REACH. A devtools console is the only instrument pointed at it. So the single code path with no test coverage was also the one discarding its own diagnostics.\n\nFIXED: a report() helper rendering with the alternate form {:#}, which walks the chain joining with \": \". It lives in lib.rs rather than bindings.rs SO IT CAN BE TESTED AT ALL -- bindings is wasm-gated. Five call sites (start, on_message, flush, documents, refresh).\n\nTEST WRITTEN AGAINST THE OLD RENDERING FIRST: a_reported_error_carries_its_cause_not_just_its_label asserts that anyhow to_string() IS just the label (pinning the trap, so the helper stops being needed if that ever changes) and that report() contains BOTH the label and the cause.\n\nTHIS DOES NOT FIX THE UNDERLYING FAILURE -- it makes the next reload name it. Redeploy and read the console again.","status":"closed","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:55:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:22:07Z","started_at":"2026-09-20T05:32:34Z","closed_at":"2026-09-20T06:22:07Z","close_reason":"Closed by PR #21 (WU-I) ON EVIDENCE, with NO CODE WRITTEN -- the fix described in this ticket was already in the tree.\n\nVerified rather than assumed: report() renders with the alternate form {:#}, which walks the anyhow chain joining with \": \"; it lives in lib.rs rather than the wasm-gated bindings.rs so it can be tested at all; five call sites (start, on_message, flush, documents, refresh); and a_reported_error_carries_its_cause_not_just_its_label pins both halves -- that anyhow's to_string() IS just the outermost label, and that report() carries the cause too.\n\nFAULT INJECTED BEFORE CLOSING: changed {e:#} back to {e} and watched the test fail on contains(\"no such document\"). So the test is discriminating and the claim is evidence rather than description.\n\nThe ticket's own closing line still stands and is not discharged by this: it does not fix the underlying failure, it makes the next reload name it.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-anr9","title":"Project the five grid-frame facts: unblocks H, z, gz and fixes Food/Alice at once","description":"Two tickets turn out to be ONE piece of work, noticed by improvise-bpq2 while walking the bindings.\n\nimprovise-s1xw (multi-level grid labels -- the browser shows Food/Alice where the TUI shows two columns) and the bindings H / z / gz need THE SAME FIVE FACTS, which gotcha 44 already enumerates as the grid frame the cache does not carry:\n row_cats / col_cats -- how many header levels\n row_items -- multi-level label TUPLES and inline group headers\n col_group_for -- column group spans\n is_group_collapsed -- the collapse indicator\n(record_col is the sixth and is CORRECT to keep off: session-local, keys pending drill edits.)\n\nSo:\n * hide-selected-row-item (H) needs to know WHICH CATEGORY a row label belongs to -- impossible while row_labels slash-joins a tuple.\n * toggle-group / gz need group membership and collapsed state.\n * And the browser renders the slash-joined form, which the TUI own test asserts must not appear.\n\nDOING THEM SEPARATELY MEANS TOUCHING row_labels TWICE. It is a co-ordinated type change either way -- improvise-wasm-client/src/render.rs consumes both row_labels and col_labels -- so the cheap moment is once.\n\nDO NOT add row_levels beside row_labels. Two answers to \"what are the labels\" is the disease (improvise-edp), and bpq2 has just spent a wave demonstrating what one shared derivation buys.\n\nSupersedes/absorbs improvise-s1xw -- decide which id survives.","notes":"USER RULING 2026-09-20: 'Most of the view functionality should be a wasm blob that's shared between the TUI and the web UI' -- clarified as ONE SHARED CRATE, not the TUI loading wasm at runtime. The view derivations live in improvise-protocol; the browser gets them as wasm, the TUI links them natively, and ONLY THE PAINTING differs (ratatui vs DOM).\n\nTHIS TICKET IS NOW THE LAST STEP OF THAT, which is why it is P1.\n\nMeasured 2026-09-20: every other widget already takes a shared cache and no \u0026Model -- FormulaContent(\u0026FormulaCache), ViewContent(\u0026ViewListCache), TileBar(\u0026TileBarCache), CategoryContent(\u0026CategoryTreeCache), WizardWidget(\u0026WizardCache), HelpWidget(usize), WhichKeyWidget(\u0026[..]). THE REMAINING DUPLICATION IS ONE FILE: src/ui/grid.rs still derives 14 things straight off layout./view., and those are exactly the six frame facts in gotcha 44.\n\nSo closing this ticket is what makes 'most of the view functionality shared' TRUE rather than nearly true, and it fixes the Food/Alice divergence and unblocks H/z/gz as the same change.\n\nSEQUENCING NOTE: this touches improvise-protocol/src/render.rs, src/ui/grid.rs AND improvise-wasm-client/src/render.rs together -- it is a co-ordinated type change on row_labels/col_labels, not an additive one. Do not start it while another agent is in improvise-protocol; the client-side keymap work (improvise-bpq2 successor) is there now.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:26Z","started_at":"2026-09-20T05:32:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-vvl3","title":"A headless browser harness: run storage_contract and the shell assertions under Playwright","description":"THE CONDITION THAT DEFERRED improvise-xf1 HAS BEEN MET. The user deferred browser testing on 2026-09-20 with: \"I am happy just putting off the browser test for now: WE HAVE TO HAVE SOMETHING TO TEST.\" After improvise-9x6, djm, hwpi and the demo model, we do -- a complete self-contained application whose ENTIRE untested surface is the part a harness would cover.\n\nTHE EVIDENCE THAT IT HAS FLIPPED, not just an argument that it would be nice:\n * Three of improvise-hwpi four fixes are ONE-ASSERTION tests (status text after a bad frame, status text with no peer, one repaint on commit). That ticket itself said \"consider whether the right first move is a headless DOM -- otherwise the fixes are as unverified as the bugs.\" The fixes are now shipped, so that sentence is true of them.\n * The first-frame watchdog was found BY READING, and only because someone went looking for it. The next one will not be.\n * OPFS is the specific thing MemoryFs cannot stand in for, and it is now on the critical path of the DEFAULT deployment rather than a future one. storage_contract in storage.rs exists and is written down precisely so it can be run against OpfsFs in a browser; NOTHING BUT A BROWSER CAN RUN IT.\n * Persistence across reload is the one improvise-bck acceptance criterion no test in this repo can structurally reach.\n\nSCOPE, deliberately small: python3 -m http.server --directory site, plus one Playwright spec, reusing the assemble step CI already performs. Two parts -- (a) storage_contract against OpfsFs, which is the thing improvise-xf1 is actually about; (b) three shell assertions from hwpi.\n\nShould block improvise-bck sign-off. Un-defer or close improvise-xf1 into this, whichever the user prefers -- xf1 predates the worker and is narrower than what is now needed.","notes":"ASSERTIONS OWED BY improvise-taxx's FIX (from WU-I / PR #21, so vvl3 need not re-derive them). The close-listener now distinguishes never-connected from was-connected-and-dropped via an `opened` flag, because readyState is already CLOSED in any close listener and both cases are code 1006 / wasClean:false. Four assertions:\n\n 1. close-before-open -\u003e status stays 'cannot reach \u003curl\u003e ...'\n 2. open-then-close -\u003e status reads 'disconnected'\n 3. ASSERTION 1 MUST FAIL against the previous unconditional close handler. That arm IS the regression guard; without it, 'always say cannot reach' passes.\n 4. after either arm, the handshake watchdog must NOT fire 8s later and overwrite the message.\n\nAlso note WU-I DECLINED Copilot's request to add JS regression tests in PR #21, on the grounds that there is no JavaScript test infrastructure in this repo at all (no package.json, no runner, nothing in CI executing a line of web/*.js) and that a parallel ad-hoc JS surface beside this ticket would be two answers to one question. So this bead now owns that infrastructure decision, not just the OPFS half.","status":"in_progress","priority":1,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:52Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:09:55Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-gv9r","title":"[bug] A wire model command never marks the document dirty","description":"Found by improvise-9x6 (2026-09-20) the hard way: the worker first persist trigger was `if dirty`, and it SILENTLY NEVER FIRED.\n\nreduce::model_effects maps ModelCommand::SetCell to effect::SetCell, which writes the cell and does NOT touch model_state.dirty. Dirtiness is raised by a SEPARATE MarkDirty effect that only the command/key path emits -- so on every transport, a client edit leaves the authoritative document reporting CLEAN.\n\nTwo consequences, and the second is the one that will bite someone:\n * \"Save if dirty\" is a trigger that never fires. Any host built on it loses every client edit, silently.\n * Anything else that reads model_state.dirty about a client-edited document gets the wrong answer -- an unsaved-changes prompt, an autosave, a UI marker.\n\nWHY IT HAPPENED: MarkDirty being a separate effect is deliberate and reasonable on the key path, where a command composes the pair. The derivation that turned effects into ModelCommands (improvise-kmgs) took the effects ONE AT A TIME, so the pairing was lost -- the wire has SetCell without its MarkDirty companion. That is the interesting shape: a derivation that is correct per element and wrong for a set.\n\nCANDIDATE FIXES, and this wants deciding alongside improvise-9x6 sibling ticket about Save: make model_effects emit the MarkDirty companion where the key path does; or fold dirtiness into the effects that change the document so it cannot be forgotten (the unrepresentable-not-validated answer, and probably right); or decide that dirty is a host-local concept the wire has no opinion on, and then say so where someone will read it.","status":"in_progress","priority":1,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:10:13Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:28Z","started_at":"2026-09-20T05:32:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -36,9 +36,9 @@ {"_type":"issue","id":"improvise-km8","title":"1.2 Fix Cargo.toml metadata","description":"Add missing package fields: description, repository, homepage, documentation, readme, keywords, categories, license. Description: 'Terminal pivot-table modeling in the spirit of Lotus Improv'. Keywords: tui, pivot, spreadsheet, data, improv. Categories: command-line-utilities, visualization.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:48Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:20:17Z","close_reason":"Metadata already present: description, license, repository, homepage, readme, keywords, categories all populated.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-902","title":"1.1 Audit and remove or update context/SPEC.md","description":"Compare context/SPEC.md against actual code. If spec contradicts code significantly, delete it. Optionally move salvageable conceptual content to docs/design-notes.md with a staleness disclaimer. Do not try to bring spec into sync.","status":"closed","priority":1,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:06:46Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:10:05Z","close_reason":"Audited SPEC.md against code. Found minor inaccuracies (storage internals, wizard step count, search mode representation) but no major architectural contradictions. Deleted SPEC.md since it was largely redundant with repo-map.md and design-principles.md. Salvaged product vision and non-goals into docs/design-notes.md with staleness disclaimer.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-tik","title":"Phase 1: Repository hygiene","description":"Raise the quality floor of what a stranger sees when they land on the repo. Subtasks cover spec audit, Cargo.toml metadata, publish-readiness, CSV audit, and example files.","status":"closed","priority":1,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:45Z","created_by":"Edward Langley","updated_at":"2026-04-09T07:22:34Z","closed_at":"2026-04-09T07:22:34Z","close_reason":"All 5 subtasks complete: SPEC.md audited, Cargo.toml metadata fixed, publish dry-run clean, CSV audit done, example files created.","dependencies":[{"issue_id":"improvise-tik","depends_on_id":"improvise-2fr","type":"blocks","created_at":"2026-04-08T23:37:31Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-902","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-bv1","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-km8","type":"blocks","created_at":"2026-04-08T23:37:30Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-tik","depends_on_id":"improvise-n1h","type":"blocks","created_at":"2026-04-08T23:37:32Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-sihr","title":"Every minibuffer mode entered from records mode exits to plain Normal while the layout is still records","description":"Found by Copilot on PR #27 against the new Ctrl-W save prompt, and DECLINED there with evidence because fixing the one arm the finding named would have been one arm of a three-arm rule.\n\nTHE BEHAVIOUR: every minibuffer mode entered from RecordsNormal exits to plain Normal while the LAYOUT IS STILL RECORDS. It affects `:` (CommandMode), Ctrl-E (ExportPrompt) and Ctrl-W (SavePrompt, new in PR #27). Only the last is new; the class predates it.\n\nTHE MECHANISM: the exit mode is a CONSTANT, baked into TextEntry::new(.., \"normal\") in the keymap builder and into each commit command's ChangeMode. Nothing carries where the user came FROM, so nothing can put them back.\n\nWHAT IS NOT THE MECHANISM, checked rather than reasoned (doctrine(binding_dump_diff)): the binding TABLE is fine. RecordsNormal goes 155 -\u003e 156 dumped lines across PR #27, the single addition being Char('w')+CONTROL -\u003e enter-mode [\"save-prompt\"], with nothing removed. with_parent inheritance (gotcha 13) is intact. What is lost is the RUNTIME MODE, not a binding.\n\nCHARACTERIZED, NOT FIXED: every_minibuffer_mode_entered_from_records_returns_to_plain_normal drives all three and asserts both halves -- the mode is Normal, and the layout is still records. So the current behaviour is pinned and a fix will have to update it deliberately.\n\nTHE FIX, and it is why this is its own bead: text-entry modes must carry their entry mode. That is a change to TextEntry, to AppMode (the exit is part of a mode's configuration, so it likely belongs beside MinibufferConfig -- note gotcha 11, MinibufferConfig::buffer_key is a String and not \u0026'static str precisely because ViewState must stay Deserialize-able, and the same constraint applies to anything added there), and to three commit commands. Wider than the PR that found it.\n\nWATCH TWO THINGS:\n * The exit is currently spelled in TWO places per mode -- the TextEntry spine's Esc binding and the commit command's ChangeMode. If they are made to carry an entry mode, make them read ONE source or the two will disagree on cancel versus commit.\n * SearchMode is hand-written on purpose and has no commit key (gotcha 11). It is not part of this class; do not sweep it in.\n\nRELATED: improvise-mai9 (a status message written while a minibuffer is open is never drawn) is the other half of why these modes are awkward -- both are consequences of the minibuffer owning the bottom row and nothing tracking what it displaced.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:12:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:12:20Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-sihr","title":"Every minibuffer mode entered from records mode exits to plain Normal while the layout is still records","description":"Found by Copilot on PR #27 against the new Ctrl-W save prompt, and DECLINED there with evidence because fixing the one arm the finding named would have been one arm of a three-arm rule.\n\nTHE BEHAVIOUR: every minibuffer mode entered from RecordsNormal exits to plain Normal while the LAYOUT IS STILL RECORDS. It affects `:` (CommandMode), Ctrl-E (ExportPrompt) and Ctrl-W (SavePrompt, new in PR #27). Only the last is new; the class predates it.\n\nTHE MECHANISM: the exit mode is a CONSTANT, baked into TextEntry::new(.., \"normal\") in the keymap builder and into each commit command's ChangeMode. Nothing carries where the user came FROM, so nothing can put them back.\n\nWHAT IS NOT THE MECHANISM, checked rather than reasoned (doctrine(binding_dump_diff)): the binding TABLE is fine. RecordsNormal goes 155 -\u003e 156 dumped lines across PR #27, the single addition being Char('w')+CONTROL -\u003e enter-mode [\"save-prompt\"], with nothing removed. with_parent inheritance (gotcha 13) is intact. What is lost is the RUNTIME MODE, not a binding.\n\nCHARACTERIZED, NOT FIXED: every_minibuffer_mode_entered_from_records_returns_to_plain_normal drives all three and asserts both halves -- the mode is Normal, and the layout is still records. So the current behaviour is pinned and a fix will have to update it deliberately.\n\nTHE FIX, and it is why this is its own bead: text-entry modes must carry their entry mode. That is a change to TextEntry, to AppMode (the exit is part of a mode's configuration, so it likely belongs beside MinibufferConfig -- note gotcha 11, MinibufferConfig::buffer_key is a String and not \u0026'static str precisely because ViewState must stay Deserialize-able, and the same constraint applies to anything added there), and to three commit commands. Wider than the PR that found it.\n\nWATCH TWO THINGS:\n * The exit is currently spelled in TWO places per mode -- the TextEntry spine's Esc binding and the commit command's ChangeMode. If they are made to carry an entry mode, make them read ONE source or the two will disagree on cancel versus commit.\n * SearchMode is hand-written on purpose and has no commit key (gotcha 11). It is not part of this class; do not sweep it in.\n\nRELATED: improvise-mai9 (a status message written while a minibuffer is open is never drawn) is the other half of why these modes are awkward -- both are consequences of the minibuffer owning the bottom row and nothing tracking what it displaced.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:12:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:12:20Z","dependencies":[{"issue_id":"improvise-sihr","depends_on_id":"improvise-mai9","type":"relates-to","created_at":"2026-09-20T06:12:22Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-kc7n","title":"improvise-ws-server never checks Origin, so any page a user visits can drive their session","description":"Found by Copilot on PR #21 round 5, against a reverse-proxy recipe whose IP allow-list had itself been added two rounds earlier to close a DIFFERENT hole. Verified independently by the orchestrator: `grep -rni origin crates/improvise-ws-server/src/` returns NOTHING. The server never inspects the Origin header and accepts every handshake.\n\nTHE HOLE: a WebSocket is EXEMPT FROM THE SAME-ORIGIN POLICY -- there is no CORS preflight on a handshake. So any hostile page that an allowed user visits can open a socket to the endpoint FROM THAT USER'S OWN ALLOWED ADDRESS, and read and edit every document the server holds. That is cross-site WebSocket hijacking.\n\nWHY NETWORK CONTROLS CANNOT SEE IT: an IP allow-list, a private network, loopback binding -- all answer \"may this CLIENT reach the endpoint\", and in this attack the answer is genuinely yes. The request really does come from an allowed client; it is simply being DRIVEN by a page the user did not intend to trust. Only Origin separates the two.\n\nLOOPBACK IS NOT EXEMPT, and this is the half a documentation-only fix cannot reach. In the local pairing case there is no proxy at all, so there is nothing in front to check Origin -- and loopback is not a boundary a browser enforces. While the server runs, ANY page open in that browser can reach ws://127.0.0.1:8787. So the local development path, which is the one the project actually recommends, is the least protected.\n\nDISTINCT FROM improvise-kpnr, and both are needed. kpnr is AUTHENTICATION: who is allowed to talk to this server at all. This is CSWSH: given that someone is allowed, what may put words in their mouth. An answer to either leaves the other open.\n\nTHE FIX IS CHEAP AND SERVER-SIDE: an allow-list of permitted origins checked at the handshake, defaulting to something safe for the local case (the page's own origin, and whatever --bind implies), with a flag to widen it. Refuse the upgrade rather than accepting and ignoring.\n\nTWO THINGS TO GET RIGHT:\n * A MISSING Origin header is not the same as a disallowed one. Non-browser clients (the repo's own tests, curl, a native client) send none, and the browser always does. Decide explicitly what absent means rather than letting the match arm decide by accident.\n * PR #21 has already documented Origin filtering in the PROXY as a required companion to the network control, phrased as \"in addition to, never instead of\". When this lands, the README should say the server does it too, and the proxy line becomes defence in depth rather than the only defence.\n\nTHE GENERAL LESSON, now doctrine(who_may_connect_is_two_questions): ask of any access control whether it constrains the CLIENT or what may DRIVE the client. Three consecutive rounds of that PR reasoned about the page's capabilities and none about the attacker's.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:09:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:09:42Z","dependencies":[{"issue_id":"improvise-kc7n","depends_on_id":"improvise-kpnr","type":"relates-to","created_at":"2026-09-20T06:09:49Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-mai9","title":"A status message written while a minibuffer is open is never drawn","description":"Found by WU-G while building improvise-lae (PR #27), and CONFIRMED THE HARD WAY in the same PR: Copilot then found a regression that this mechanism causes, which the agent had itself written down an hour earlier and not checked its own change against.\n\nTHE MECHANISM: src/draw.rs::draw_bottom_bar renders the minibuffer INSTEAD OF the status line. So a status_msg written while a minibuffer is open is never drawn at all -- not truncated, not overwritten, simply never rendered.\n\nWHO IS AFFECTED TODAY:\n * CategoryAdd / ItemAdd: a failed `commit-buffer category` is INVISIBLE. Worse, improvise-6t3 deliberately ordered that command's effects so the add effect is emitted LAST, specifically so its failure message would WIN over the command's optimistic status. It wins a race to a surface nobody can see.\n * A failed `:w \u003cpath\u003e` from CommandMode, which is the regression Copilot found on PR #27: the abort stops the batch before ReturnToNormalUnlessMoved and clear-buffer run, so the app stays in CommandMode with the stale command in the minibuffer and the new error hidden behind it.\n\nTHE WORKAROUND ALREADY SHIPPED, and it is a workaround: CommitSave emits [ChangeMode(Normal), SaveAs] so the error lands when the bottom row is the status bar again. That works for one command and does not generalise -- every other text-entry commit has the same hole.\n\nTWO CANDIDATE FIXES, and the choice is the ticket:\n * Draw both: give the minibuffer and the status line separate rows, or render the status into the minibuffer row when a message is pending. Cheap to describe, and it changes frame geometry, which is pinned by exact-frame characterization tests (see improvise-q0vb for what that costs).\n * Make it unrepresentable: a status message set while a minibuffer is open is a bug in the CALLER, so make the caller unable to do it -- e.g. commit paths always leave the mode before reporting. That is what CommitSave now does by hand, promoted to a rule.\n\nPREFER WHICHEVER MAKES THE FAILURE LOUD. The current state is the quiet kind: the code sets a message, the test asserts the message was set, and the user never sees it. A test asserting `app.status_msg == \"...\"` passes in every one of these cases, which is why this survived improvise-6t3's careful effect ordering.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:20Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-mai9","title":"A status message written while a minibuffer is open is never drawn","description":"Found by WU-G while building improvise-lae (PR #27), and CONFIRMED THE HARD WAY in the same PR: Copilot then found a regression that this mechanism causes, which the agent had itself written down an hour earlier and not checked its own change against.\n\nTHE MECHANISM: src/draw.rs::draw_bottom_bar renders the minibuffer INSTEAD OF the status line. So a status_msg written while a minibuffer is open is never drawn at all -- not truncated, not overwritten, simply never rendered.\n\nWHO IS AFFECTED TODAY:\n * CategoryAdd / ItemAdd: a failed `commit-buffer category` is INVISIBLE. Worse, improvise-6t3 deliberately ordered that command's effects so the add effect is emitted LAST, specifically so its failure message would WIN over the command's optimistic status. It wins a race to a surface nobody can see.\n * A failed `:w \u003cpath\u003e` from CommandMode, which is the regression Copilot found on PR #27: the abort stops the batch before ReturnToNormalUnlessMoved and clear-buffer run, so the app stays in CommandMode with the stale command in the minibuffer and the new error hidden behind it.\n\nTHE WORKAROUND ALREADY SHIPPED, and it is a workaround: CommitSave emits [ChangeMode(Normal), SaveAs] so the error lands when the bottom row is the status bar again. That works for one command and does not generalise -- every other text-entry commit has the same hole.\n\nTWO CANDIDATE FIXES, and the choice is the ticket:\n * Draw both: give the minibuffer and the status line separate rows, or render the status into the minibuffer row when a message is pending. Cheap to describe, and it changes frame geometry, which is pinned by exact-frame characterization tests (see improvise-q0vb for what that costs).\n * Make it unrepresentable: a status message set while a minibuffer is open is a bug in the CALLER, so make the caller unable to do it -- e.g. commit paths always leave the mode before reporting. That is what CommitSave now does by hand, promoted to a rule.\n\nPREFER WHICHEVER MAKES THE FAILURE LOUD. The current state is the quiet kind: the code sets a message, the test asserts the message was set, and the user never sees it. A test asserting `app.status_msg == \"...\"` passes in every one of these cases, which is why this survived improvise-6t3's careful effect ordering.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:20Z","dependencies":[{"issue_id":"improvise-mai9","depends_on_id":"improvise-sihr","type":"relates-to","created_at":"2026-09-20T06:12:22Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependencies":[{"issue_id":"improvise-kpnr","depends_on_id":"improvise-kc7n","type":"relates-to","created_at":"2026-09-20T06:09:49Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-p1ym","title":"Project the page-axis item lists so [ and ] work in a browser","description":"The one key pair an independent smoke test found inert that the shared keymap did NOT fix, and it is a genuine projection gap rather than a resolution gap.\n\nGridCache::page_coords carries (category, SELECTED ITEM). PageNext/PagePrev need THE ITEM LIST TO CYCLE THROUGH, which nothing projects. So [ and ] resolve to a command name the client holds but cannot compose an argument for, and they reach Host::handle empty vec -- doctrine(an_upstream_key_is_a_black_hole), so they do nothing at all.\n\nCONFIRMED BY SMOKE TEST 2026-09-20 against the live page: \"[ / ] did not cycle Quarter (never saw Q2 ~17,100)\". The demo model puts Quarter on the page axis specifically so paging is a thing a visitor would try.\n\nTHE AGENT WAS TOLD NOT TO FORCE IT and did not -- a truthful \"these four keys work, [ / ] needs X\" is worth more than five half-working keys.\n\nSCOPE: extend page_coords (or add a sibling) to carry the item list per page-axis category. Note gotcha 27 -- \"what data is visible\" is View::filtering_cats() (Page UNION Filter) while \"what the user can page\" is categories_on(Axis::Page). This needs the SECOND one; projecting the first would let a client try to cycle a Filter axis, which by design it must not.\n\nWhile you are there: check whether the item list should be the EFFECTIVE items (hidden items excluded) rather than all of them, since [ and ] in the TUI cycle what the user can actually select.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} From 3c36a3b57f9a8a6b2afd04848f1262995804fc59 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:23:44 -0700 Subject: [PATCH 64/72] test(harness): Firefox runs the contract too, and test.fail stops hiding setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two Copilot findings on PR #24, both correct, both applied. **`test.fail` was masking setup failures.** Applied before navigation, it made any earlier failure — the page not loading, the script not starting, `#status` never updating — an *expected* failure, so CI stayed green through it. That is precisely the check-the-checker defect these tests exist to catch, wearing the name of its cure. Both tests now assert their setup first and apply `test.fail` immediately before the one assertion the known bug owns. Verified by breaking the setup (making `paintStatus` a no-op): before, 2 passed; after, **2 failed**. **Firefox now runs the whole battery, and it passes.** improvise-xf1's acceptance criterion names Chrome, Firefox and Safari, and the reason is real rather than ceremonial: `createSyncAccessHandle` is the one API this design rests on (gotcha 28) and each engine implemented it separately, so Chromium alone was one engine's opinion. `storage_contract` against `OpfsFs` now passes on **two** engines. **Safari is not covered, and the criterion is met for two of three.** Playwright's WebKit build provides no OPFS at all — `navigator.storage.getDirectory()` itself rejects with `UnknownError` — so there is nothing there for `OpfsFs` to be right or wrong about. Rather than delete the project and quietly narrow the claim, it is kept and the spec skips on a **runtime probe**: the day that build gains OPFS the checks start running with no CI change. The skip is conditional on the probe and not on the browser name, and an unexpected skip on any other engine **fails** with "that is a regression, not a known gap" — a silently skipped check being the failure this harness exists to prevent. The shell spec stays Chromium-only on purpose: it is about `web/app.js`'s own logic, which is engine-independent, and running `routeWebSocket` plus the worker handshake three times is slower and flakier to no benefit. 14 passing, 3 skipped, 36s. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 9 +++++++-- harness/playwright.config.js | 31 ++++++++++++++++++++++++++++++- harness/tests/opfs.spec.js | 30 +++++++++++++++++++++++++++++- harness/tests/shell.spec.js | 22 ++++++++++++++++------ 4 files changed, 82 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b67732f0..197caf39 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -120,9 +120,14 @@ jobs: - name: Install the harness working-directory: harness run: npm ci - - name: Install Chromium + # All three, because improvise-xf1's acceptance criterion names Chrome, + # Firefox and Safari and `createSyncAccessHandle` is implemented + # separately by each engine. WebKit is installed even though its OPFS is + # absent today: the spec skips on a runtime probe, so the day that build + # gains OPFS the check starts running without a CI change. + - name: Install browsers working-directory: harness - run: npx playwright install --with-deps chromium + run: npx playwright install --with-deps chromium firefox webkit # Builds both site bundles, the harness's own wasm bundle, and assembles # `harness/serve` — the same tree `pages-build` publishes, plus /opfs/. - name: Build the harness diff --git a/harness/playwright.config.js b/harness/playwright.config.js index 130867c3..907ea7e7 100644 --- a/harness/playwright.config.js +++ b/harness/playwright.config.js @@ -38,7 +38,36 @@ export default defineConfig({ baseURL: "http://127.0.0.1:8099", trace: "retain-on-failure", }, - projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }], + // improvise-xf1's acceptance criterion names Chrome, Firefox and Safari, + // and the OPFS spec is the reason: `createSyncAccessHandle` is the one API + // this whole design rests on (gotcha 28), its sync read/write/truncate + // signatures were added later than the handle itself, and each engine + // implemented it separately. Chromium alone would be one engine's opinion. + // + // The shell spec is Chromium-only on purpose — it is about `web/app.js`'s + // own logic, which is engine-independent, and `routeWebSocket` plus the + // worker handshake are slower and flakier to no benefit across three. + // + // Result today: Chromium and Firefox run the whole battery and pass. + // **Playwright's WebKit provides no OPFS at all** — `getDirectory()` itself + // rejects with `UnknownError` — so Safari is NOT covered and xf1's + // criterion is met for two engines of three. The project is kept rather + // than deleted so that it starts running the day that build gains OPFS; + // `opfs.spec.js` skips on a runtime probe, and an unexpected skip on any + // other engine is a failure. + projects: [ + { name: "chromium", use: { ...devices["Desktop Chrome"] } }, + { + name: "firefox", + use: { ...devices["Desktop Firefox"] }, + testMatch: /opfs\.spec\.js/, + }, + { + name: "webkit", + use: { ...devices["Desktop Safari"] }, + testMatch: /opfs\.spec\.js/, + }, + ], webServer: { command: "python3 -m http.server 8099 --bind 127.0.0.1 --directory serve", url: "http://127.0.0.1:8099/index.html", diff --git a/harness/tests/opfs.spec.js b/harness/tests/opfs.spec.js index 2fe6be7e..2d675af5 100644 --- a/harness/tests/opfs.spec.js +++ b/harness/tests/opfs.spec.js @@ -32,8 +32,36 @@ const EXPECTED = [ ]; test.describe("OpfsFs in a real browser", () => { - test.beforeEach(async ({ page }) => { + test.beforeEach(async ({ page, browserName }, testInfo) => { await page.goto("/opfs/index.html"); + + // improvise-xf1's acceptance criterion names Chrome, Firefox and Safari. + // Chromium and Firefox run the whole battery. Playwright's WebKit build + // provides **no OPFS at all** — `navigator.storage.getDirectory()` itself + // rejects with `UnknownError` — so there is nothing there for `OpfsFs` to + // be right or wrong about, and Safari stays uncovered. See the hand-back + // and `playwright.config.js`. + // + // The skip is conditional on the PROBE, not on the browser name, so it + // cannot quietly swallow a real regression: if Chromium or Firefox ever + // stops providing OPFS, this fails rather than skipping. A silently + // skipped check is exactly the failure this harness exists to prevent, so + // an unexpected skip also fails. + const opfs = await page.evaluate(async () => { + try { + await navigator.storage.getDirectory(); + return "yes"; + } catch (err) { + return String(err?.name ?? err); + } + }); + if (opfs !== "yes") { + expect( + browserName, + `${browserName} lost OPFS (${opfs}) — that is a regression, not a known gap`, + ).toBe("webkit"); + testInfo.skip(true, `this WebKit build provides no OPFS (${opfs})`); + } }); // Check the checker, and do it first: a green run against a worker that diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js index 0e014d87..af75a1e7 100644 --- a/harness/tests/shell.spec.js +++ b/harness/tests/shell.spec.js @@ -88,14 +88,20 @@ test.describe("the shell reports what its peer is doing", () => { // starts passing, Playwright reports "expected to fail but passed", and the // `test.fail()` line has to be deleted. It cannot land silently either way. test("close before open leaves the precise message standing", async ({ page }) => { - test.fail(true, "improvise-m18f: the close handler is unconditional on this branch"); await page.goto("/index.html?ws=127.0.0.1:9"); - // Wait for the status to settle on SOMETHING rather than for a particular - // message: waiting for "cannot reach" would retry against the value the - // page has already replaced, and waiting for "disconnected" would time - // out for the wrong reason once the fix lands. + // Everything up to here is SETUP, and it is asserted BEFORE `test.fail` + // is applied — otherwise a page that failed to load, or a status that + // never updated at all, would also be an "expected" failure and CI would + // stay green through it (Copilot, PR #24). Waiting for the status to + // settle on SOMETHING rather than on a particular message is what makes + // this arm work under both implementations: "cannot reach" would retry + // against a value the page has already replaced, and "disconnected" + // would time out for the wrong reason once the fix lands. await expect(status(page)).not.toHaveText("", { timeout: 15_000 }); await page.waitForTimeout(1_000); + + // From here on the known bug owns the outcome. + test.fail(true, "improvise-m18f: the close handler is unconditional on this branch"); await expect(status(page)).toContainText("cannot reach", { timeout: 1_000 }); }); @@ -124,13 +130,17 @@ test.describe("the shell reports what its peer is doing", () => { // timer still running. Filed rather than fixed here: `web/app.js` is not // this unit's file. test("the watchdog does not overwrite a mid-session disconnect", async ({ page }) => { - test.fail(true, "the close handler does not disarm the handshake watchdog"); await page.routeWebSocket("ws://127.0.0.1:8791/", (ws) => { setTimeout(() => ws.close(), 300); }); await page.goto("/index.html?ws=127.0.0.1:8791"); + // Setup, asserted before `test.fail` for the reason above: a socket that + // never opened would otherwise make this pass as an expected failure + // while testing nothing. await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); + + test.fail(true, "the close handler does not disarm the handshake watchdog"); await expect(status(page)).toContainText("disconnected", { timeout: 1_000 }); }); From a17ec9efacc2dbbdab628d226ee823b1463a135d Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:26:36 -0700 Subject: [PATCH 65/72] =?UTF-8?q?fix:=20two=20Copilot=20findings=20?= =?UTF-8?q?=E2=80=94=20a=20stale=20column=20index,=20and=20a=20false=20neg?= =?UTF-8?q?ative=20arm?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **A schema change shifted the grid under its own advance.** `add_category` inserts the new coordinate column immediately before `+cat`, so every index from `+cat` rightwards moves by one — while `CommitAndAdvance` computed its advance from `ctx`, the *pre-commit* layout (gotcha 23). On the last row `Enter` wraps to `col + 1`, which after the insert is `+cat` again. The failure is not "lands one column off": it **reopens category creation**, so a user naming two categories in a row never reaches the column they made and gets an unbounded `+cat` loop. `Tab` reached it the same way. A schema change that inserts a column now does not advance at all, which is the useful answer as well as the safe one — the insert happens exactly where the cursor is, so staying put lands on the new column. The measure `+` shares none of this and not by luck: it is the last column, so there is no index to its right to shift into, and both advances already special-case the bottom-right edge. `inserts_a_column` asks the *layout* what the column is rather than re-deriving it from the value, so gotcha 1 keeps one answer. Both arms were watched failing first, and both now assert the column landed on rather than merely "not `+cat`" — a `assert_ne` alone would pass anywhere. **And a negative arm that was asserting something false.** `a_prompt_this_client_cannot_commit_is_never_entered` used `:` to show that `strands_the_user` had not degenerated into "refuse every mode". But `:` is itself a dead room: it opens locally and types locally, while Enter is `[execute-command, clear-buffer]` and `step` has no `execute-command` arm, so the frame is dropped. The arm was **certifying the behaviour it existed to rule out**, which is worse than having no arm at all. `CommandMode` therefore joins the predicate. It costs no working surface — every ex-command behind `:` is inert for the same single reason — and the distinction from the two prompts is kept visible because it says who removes the line: they are dead *by construction* (no `ModelCommand` can name `Save`/`SaveAs`/`ExportCsv`), this is dead *by omission*, and improvise-e24v is scheduled to implement `execute-command` client-side. Listing it is therefore also the named gap gotcha 47 asks for: closing e24v has to come to this line and delete it, which a comment could not force. `the_command_line_cannot_be_committed_by_this_client` is the evidence rather than the assertion. The arm itself now has **two halves**, and the first version of the rewrite had neither — it pressed Enter in `Editing`, which never consults `strands_the_user`, so "refuse every mode" still passed. Half one enters a mode through `enter-mode` and so exercises the predicate; half two presses Enter and requires a real `Command::Model` back, because the property is finishing rather than entering. `?` is honest as half one where `:` was not: Help has no commit, so nothing typed into it can be swallowed. Proved in both directions: "refuse everything" fails half one, "refuse nothing" fails the positive arms and the named-gap list. Co-Authored-By: Claude Opus 5 (1M context) --- .../src/command/cmd/commit.rs | 34 +++++++ crates/improvise-command/src/ui/app.rs | 80 +++++++++++++++ crates/improvise-wasm-client/src/keys.rs | 98 ++++++++++++++++--- 3 files changed, 201 insertions(+), 11 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/commit.rs b/crates/improvise-command/src/command/cmd/commit.rs index 29e16847..4d1a7783 100644 --- a/crates/improvise-command/src/command/cmd/commit.rs +++ b/crates/improvise-command/src/command/cmd/commit.rs @@ -409,6 +409,19 @@ fn commit_cell_value( commit_regular_cell_value(key, value, effects); } +/// Whether committing `key` inserts a column, invalidating every column +/// index to its right. +/// +/// Asked of the *layout* — `record_col` resolves the synthetic `_Dim` +/// position into what the column actually is — rather than re-derived from +/// the value or the cursor, so there is one answer to "what is this column" +/// (gotcha 1). +fn inserts_a_column(ctx: &CmdContext, key: &CellKey) -> bool { + crate::view::synthetic_record_info(key) + .and_then(|(_, col_idx)| ctx.layout.record_col(col_idx)) + .is_some_and(|col| matches!(col, RecordCol::NewCategory)) +} + /// Direction to advance after committing a cell edit. #[derive(Debug, Clone, Copy)] pub enum AdvanceDir { @@ -468,6 +481,27 @@ impl Cmd for CommitAndAdvance { // abort cannot reach it; same rule as `ExecuteCommand`'s. effects.push(effect::change_mode(exit_mode_for(&self.edit_mode))); commit_cell_value(ctx, &self.key, &self.value, &mut effects); + if inserts_a_column(ctx, &self.key) { + // A schema change that inserts a column moves every index to its + // right, and the advance below is computed from `ctx` — the + // *pre-commit* layout, because `apply_effects` rebuilds only + // after the whole batch (gotcha 23). On the last row `Enter` + // wraps to `col + 1`, which after the insert is `+cat` again, so + // the editor reopened category creation and a user naming two + // categories in a row never reached the column they had made. + // `Tab` reached it the same way. + // + // Not advancing is also the *useful* answer rather than merely + // the safe one: `add_category` inserts the new column exactly + // where the cursor already is, so staying put lands on it. + // + // The measure `+` shares none of this, and not by luck — it is + // the last column, so there is no column to its right for an + // index to shift into, and both advances are already special- + // cased at the bottom-right edge. + effects.push(Box::new(effect::RebuildLayout)); + return effects; + } match self.advance { AdvanceDir::Down => { let adv = EnterAdvance { diff --git a/crates/improvise-command/src/ui/app.rs b/crates/improvise-command/src/ui/app.rs index 3dae170a..4dca39aa 100644 --- a/crates/improvise-command/src/ui/app.rs +++ b/crates/improvise-command/src/ui/app.rs @@ -1443,6 +1443,86 @@ mod tests { ); } + /// improvise-2zq, found by Copilot on PR #27: creating a category from + /// `+cat` shifted the grid under the advance that followed it. + /// + /// `add_category` inserts the new coordinate column **immediately before + /// `+cat`**, so every column index from `+cat` rightwards moves by one. + /// `CommitAndAdvance` then computed its advance from `ctx`, whose layout + /// is the pre-commit one (gotcha 23: `apply_effects` rebuilds only after + /// the whole batch). On the last row `Enter` wraps to the next column — + /// and the index it wrapped to was `+cat` *again* after the rebuild. + /// + /// So the failure is not "lands one column off". It **reopens category + /// creation**: a user naming two categories in a row never reaches the + /// column they just made, and gets an unbounded `+cat` loop instead. + /// + /// The last row is the arm that wraps, so it is the arm tested. + #[test] + fn enter_on_plus_cat_in_the_last_row_does_not_reopen_category_creation() { + let mut app = records_model_with_two_rows(); + let plus_cat = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == crate::view::NEW_CATEGORY_COL) + .expect("+cat column"); + let last_row = app.layout.row_count() - 1; + app.cursor_mut().selected = (last_row, plus_cat); + + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + for c in "Product".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Enter, KeyModifiers::NONE)) + .unwrap(); + + let (_, col) = app.cursor().selected; + let landed = app.layout.col_label(col); + assert_ne!( + landed, + crate::view::NEW_CATEGORY_COL, + "the advance must not land back on +cat — that is the loop, and \ + it is what a stale column index produces" + ); + assert_eq!( + landed, "Product", + "and it lands on the column just created, which is the useful \ + answer rather than merely a safe one: the insert happens at the \ + cursor, so staying put arrives there" + ); + } + + /// The same stale index reached `Tab`, which advances right rather than + /// wrapping. Both directions read the pre-commit layout, so both are + /// covered — one arm of a two-arm rule is the shape + /// `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` + /// names. + #[test] + fn tab_on_plus_cat_does_not_reopen_category_creation() { + let mut app = records_model_with_two_rows(); + let plus_cat = (0..app.layout.col_count()) + .find(|i| app.layout.col_label(*i) == crate::view::NEW_CATEGORY_COL) + .expect("+cat column"); + app.cursor_mut().selected = (0, plus_cat); + + app.handle_key(KeyEvent::new(KeyCode::Char('i'), KeyModifiers::NONE)) + .unwrap(); + for c in "Product".chars() { + app.handle_key(KeyEvent::new(KeyCode::Char(c), KeyModifiers::NONE)) + .unwrap(); + } + app.handle_key(KeyEvent::new(KeyCode::Tab, KeyModifiers::NONE)) + .unwrap(); + + let (_, col) = app.cursor().selected; + assert_ne!( + app.layout.col_label(col), + crate::view::NEW_CATEGORY_COL, + "Tab must not land back on +cat either" + ); + assert_eq!(app.layout.col_label(col), "Product"); + } + /// Committing the `+cat` name creates the **column** and does not write /// the coordinate on the row the cursor was on. /// diff --git a/crates/improvise-wasm-client/src/keys.rs b/crates/improvise-wasm-client/src/keys.rs index a8a8c9c7..faf79062 100644 --- a/crates/improvise-wasm-client/src/keys.rs +++ b/crates/improvise-wasm-client/src/keys.rs @@ -458,10 +458,37 @@ fn write_buffer(name: &str, value: String) -> Command { /// fixing one arm of a rule with two is the shape /// `doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only)` /// names. +/// +/// # `CommandMode` is dead by **omission**, and listed anyway +/// +/// Copilot found it on the next round, and it was a false negative in this +/// predicate's *first* version — which mattered more than its severity, +/// because `a_prompt_this_client_cannot_commit_is_never_entered` used `:` +/// as the negative arm that stops "refuse every mode" from passing. An arm +/// asserting something false does not merely fail to guard; it **certifies +/// the behaviour it was added to rule out.** +/// +/// `:` resolves locally into `CommandMode` and typing appends locally, but +/// Enter is `[execute-command, clear-buffer]` and [`step`] has no +/// `execute-command` arm, so `resolve` returns `Upstream` and the frame is +/// dropped. Every ex-command behind `:` is inert for the same reason — +/// there is one door and no client arm for it — so refusing the mode +/// removes **no working surface**, only a room that swallows typing. +/// `the_command_line_cannot_be_committed_by_this_client` is the evidence, +/// not the assertion. +/// +/// The distinction from the two prompts is worth keeping visible, because +/// it says who removes this line. They are dead **by construction**: no +/// `ModelCommand` can ever name `Save`/`SaveAs`/`ExportCsv`, so no resolver +/// would help. This is dead **by omission**: `execute-command` is +/// implementable client-side and improvise-e24v is scheduled to do it. +/// Listing it here is therefore also the named gap gotcha 47 asks for — +/// closing e24v has to come to this line and delete it, which a comment +/// could not have forced. fn strands_the_user(mode: &AppMode) -> bool { matches!( mode, - AppMode::SavePrompt { .. } | AppMode::ExportPrompt { .. } + AppMode::SavePrompt { .. } | AppMode::ExportPrompt { .. } | AppMode::CommandMode { .. } ) } @@ -1106,17 +1133,66 @@ mod tests { ); } - // ...but an `enter-mode` this client can finish is still resolved - // locally, or the fix above would be "refuse every mode". `:` is the - // sharpest negative: it is another minibuffer prompt, so it goes - // through the same `entered()` path and differs only in whether the - // commit behind it is reachable. - let entered = resolve(&state, ev(KeyCode::Char(':'))); + // The negative arm has **two halves**, and both are needed. + // + // Half one: a mode that is not refused is still entered. This is the + // half that goes through `strands_the_user`, so it is the only thing + // that can fail when the predicate degenerates to "refuse + // everything". `?` is honest here where `:` was not: Help has no + // commit at all, so nothing a user types into it can be swallowed, + // and Esc closes it locally. (Its page keys *are* upstream, which + // makes page navigation inert — but inert navigation loses no work, + // which is the line this predicate draws.) + assert_eq!( + resolve(&state, ev(KeyCode::Char('?'))), + KeyOutcome::Local(vec![Command::ChangeMode(AppMode::Help)]), + "a mode this client can finish must still be entered, or the \ + guard above is just \"refuse every mode\"" + ); + + // Half two: the property the predicate is *about* is finishing, not + // entering — so something has to press Enter and observe. The first + // version of this test asserted only that `:` entered its mode, and + // `:` is itself a dead room (see + // `the_command_line_cannot_be_committed_by_this_client`), so the arm + // was certifying the very behaviour it existed to rule out. + let mut client = client_on(record_cell()); + client.state_mut().view.mode = AppMode::editing(); + client + .state_mut() + .view + .buffers + .insert("edit".into(), "42".into()); + let KeyOutcome::Local(commands) = resolve(client.state(), ev(KeyCode::Enter)) else { + panic!("Enter in Editing must resolve here — that is the arm"); + }; assert!( - matches!(&entered, KeyOutcome::Local(cmds) - if cmds.iter().any(|c| matches!( - c, Command::ChangeMode(AppMode::CommandMode { .. })))), - ": must still open the command line locally, got {entered:?}" + commands.iter().any(|c| matches!(c, Command::Model { .. })), + "and it must actually write something, not merely change mode: \ + {commands:?}" + ); + } + + /// The evidence behind listing `CommandMode` in `strands_the_user`, + /// rather than the assertion that it is listed. + /// + /// `:` opens locally and typing appends locally, but Enter is + /// `[execute-command, clear-buffer]` and `step` has no `execute-command` + /// arm — so the whole sequence fails to resolve, `Host::handle` drops + /// the frame, and every ex-command behind `:` is inert. That is what + /// makes refusing the mode cost no working surface. + /// + /// When improvise-e24v gives this client `execute-command`, this test is + /// the one that should start failing, and the `CommandMode` arm of + /// `strands_the_user` is what it points at. + #[test] + fn the_command_line_cannot_be_committed_by_this_client() { + let mut state = in_mode(AppMode::command_mode()); + state.view.buffers.insert("command".into(), "q".into()); + assert_eq!( + resolve(&state, ev(KeyCode::Enter)), + KeyOutcome::Upstream(ev(KeyCode::Enter)), + "`:q` — and every other ex-command — resolves to nothing here" ); } From 359c157774969835ec04ba27fb917aed0e001957 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:28:27 -0700 Subject: [PATCH 66/72] chore: delete foo.improv, the second unloadable tracked document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sibling of bank-info.improv, deleted in PR #22. Both were force-added past .gitignore's `*.improv`, and both are hand-written fakes with no generator. foo.improv is worse: it has no version line at all, so it cannot parse under any grammar version. Note tests/shipped_documents_load.rs walks examples/ and web/ — the two directories .gitignore un-ignores — so neither root file was ever covered by it. Deleting them is what keeps that true rather than an omission. Co-Authored-By: Claude Opus 5 (1M context) --- foo.improv | 233 ----------------------------------------------------- 1 file changed, 233 deletions(-) delete mode 100644 foo.improv diff --git a/foo.improv b/foo.improv deleted file mode 100644 index 97fbe8bc..00000000 --- a/foo.improv +++ /dev/null @@ -1,233 +0,0 @@ -# Whatever - -## Category: Type -- Food -- Gas -- Medical -- Clothing - -## Category: Month -- Jan -- Feb -- Mar -- Apr -- May -- Jun - -## Category: Recipient -- Joe -- Bob -- Alice -- Jane - -## Category: Signer -- Ed -- Bernadette - -## Data -Month=Apr, Recipient=Alice, Signer=Bernadette, Type=Clothing = 39 -Month=Apr, Recipient=Alice, Signer=Bernadette, Type=Food = 756 -Month=Apr, Recipient=Alice, Signer=Bernadette, Type=Gas = 5645543 -Month=Apr, Recipient=Alice, Signer=Bernadette, Type=Medical = 654 -Month=Apr, Recipient=Alice, Signer=Ed, Type=Clothing = 582 -Month=Apr, Recipient=Alice, Signer=Ed, Type=Food = 3959 -Month=Apr, Recipient=Alice, Signer=Ed, Type=Gas = 28492 -Month=Apr, Recipient=Alice, Signer=Ed, Type=Medical = 53 -Month=Apr, Recipient=Bob, Signer=Bernadette, Type=Clothing = 49233 -Month=Apr, Recipient=Bob, Signer=Bernadette, Type=Food = 457 -Month=Apr, Recipient=Bob, Signer=Bernadette, Type=Gas = 34534 -Month=Apr, Recipient=Bob, Signer=Bernadette, Type=Medical = 345 -Month=Apr, Recipient=Bob, Signer=Ed, Type=Clothing = 293 -Month=Apr, Recipient=Bob, Signer=Ed, Type=Food = 6594 -Month=Apr, Recipient=Bob, Signer=Ed, Type=Gas = 2937 -Month=Apr, Recipient=Bob, Signer=Ed, Type=Medical = 62 -Month=Apr, Recipient=Jane, Signer=Bernadette, Type=Clothing = 9492 -Month=Apr, Recipient=Jane, Signer=Bernadette, Type=Food = 545 -Month=Apr, Recipient=Jane, Signer=Bernadette, Type=Gas = 453543 -Month=Apr, Recipient=Jane, Signer=Bernadette, Type=Medical = 7655 -Month=Apr, Recipient=Jane, Signer=Ed, Type=Clothing = 493 -Month=Apr, Recipient=Jane, Signer=Ed, Type=Food = 2957 -Month=Apr, Recipient=Jane, Signer=Ed, Type=Gas = 495 -Month=Apr, Recipient=Jane, Signer=Ed, Type=Medical = 72 -Month=Apr, Recipient=Joe, Signer=Bernadette, Type=Clothing = 9304 -Month=Apr, Recipient=Joe, Signer=Bernadette, Type=Food = 343 -Month=Apr, Recipient=Joe, Signer=Bernadette, Type=Gas = 7665 -Month=Apr, Recipient=Joe, Signer=Bernadette, Type=Medical = 76745 -Month=Apr, Recipient=Joe, Signer=Ed, Type=Clothing = 19 -Month=Apr, Recipient=Joe, Signer=Ed, Type=Food = 5658 -Month=Apr, Recipient=Joe, Signer=Ed, Type=Gas = 2974 -Month=Apr, Recipient=Joe, Signer=Ed, Type=Medical = 52 -Month=Feb, Recipient=Alice, Signer=Bernadette, Type=Clothing = 21 -Month=Feb, Recipient=Alice, Signer=Bernadette, Type=Food = 232 -Month=Feb, Recipient=Alice, Signer=Bernadette, Type=Gas = 23425 -Month=Feb, Recipient=Alice, Signer=Bernadette, Type=Medical = 2342 -Month=Feb, Recipient=Alice, Signer=Ed, Type=Clothing = 72 -Month=Feb, Recipient=Alice, Signer=Ed, Type=Food = 86940 -Month=Feb, Recipient=Alice, Signer=Ed, Type=Gas = 594 -Month=Feb, Recipient=Alice, Signer=Ed, Type=Medical = 23 -Month=Feb, Recipient=Bob, Signer=Bernadette, Type=Clothing = 22 -Month=Feb, Recipient=Bob, Signer=Bernadette, Type=Food = 23423 -Month=Feb, Recipient=Bob, Signer=Bernadette, Type=Gas = 6343 -Month=Feb, Recipient=Bob, Signer=Bernadette, Type=Medical = 2342 -Month=Feb, Recipient=Bob, Signer=Ed, Type=Clothing = 82 -Month=Feb, Recipient=Bob, Signer=Ed, Type=Food = 3494 -Month=Feb, Recipient=Bob, Signer=Ed, Type=Gas = 394 -Month=Feb, Recipient=Bob, Signer=Ed, Type=Medical = 65 -Month=Feb, Recipient=Jane, Signer=Bernadette, Type=Clothing = 1 -Month=Feb, Recipient=Jane, Signer=Bernadette, Type=Food = 232 -Month=Feb, Recipient=Jane, Signer=Bernadette, Type=Gas = 23443 -Month=Feb, Recipient=Jane, Signer=Bernadette, Type=Medical = 232 -Month=Feb, Recipient=Jane, Signer=Ed, Type=Clothing = 453 -Month=Feb, Recipient=Jane, Signer=Ed, Type=Food = 684 -Month=Feb, Recipient=Jane, Signer=Ed, Type=Gas = 4940 -Month=Feb, Recipient=Jane, Signer=Ed, Type=Medical = 34 -Month=Feb, Recipient=Joe, Signer=Bernadette, Type=Clothing = 233 -Month=Feb, Recipient=Joe, Signer=Bernadette, Type=Food = 2342 -Month=Feb, Recipient=Joe, Signer=Bernadette, Type=Gas = 4343 -Month=Feb, Recipient=Joe, Signer=Bernadette, Type=Medical = 3453 -Month=Feb, Recipient=Joe, Signer=Ed, Type=Clothing = 83 -Month=Feb, Recipient=Joe, Signer=Ed, Type=Food = 34534 -Month=Feb, Recipient=Joe, Signer=Ed, Type=Gas = 8394 -Month=Feb, Recipient=Joe, Signer=Ed, Type=Medical = 293 -Month=Jan, Recipient=Alice, Signer=Bernadette, Type=Clothing = 343 -Month=Jan, Recipient=Alice, Signer=Bernadette, Type=Food = 12 -Month=Jan, Recipient=Alice, Signer=Bernadette, Type=Gas = 12312 -Month=Jan, Recipient=Alice, Signer=Bernadette, Type=Medical = 12 -Month=Jan, Recipient=Alice, Signer=Ed, Type=Clothing = 593 -Month=Jan, Recipient=Alice, Signer=Ed, Type=Food = 232 -Month=Jan, Recipient=Alice, Signer=Ed, Type=Gas = 4543 -Month=Jan, Recipient=Alice, Signer=Ed, Type=Medical = 49 -Month=Jan, Recipient=Bob, Signer=Bernadette, Type=Clothing = 23 -Month=Jan, Recipient=Bob, Signer=Bernadette, Type=Food = 34 -Month=Jan, Recipient=Bob, Signer=Bernadette, Type=Gas = 646 -Month=Jan, Recipient=Bob, Signer=Bernadette, Type=Medical = 575 -Month=Jan, Recipient=Bob, Signer=Ed, Type=Clothing = 29394 -Month=Jan, Recipient=Bob, Signer=Ed, Type=Food = 343 -Month=Jan, Recipient=Bob, Signer=Ed, Type=Gas = 12331 -Month=Jan, Recipient=Bob, Signer=Ed, Type=Medical = 304 -Month=Jan, Recipient=Jane, Signer=Bernadette, Type=Clothing = 4545 -Month=Jan, Recipient=Jane, Signer=Bernadette, Type=Food = 34 -Month=Jan, Recipient=Jane, Signer=Bernadette, Type=Gas = 343 -Month=Jan, Recipient=Jane, Signer=Bernadette, Type=Medical = 5335 -Month=Jan, Recipient=Jane, Signer=Ed, Type=Clothing = 293 -Month=Jan, Recipient=Jane, Signer=Ed, Type=Food = 43 -Month=Jan, Recipient=Jane, Signer=Ed, Type=Gas = 675 -Month=Jan, Recipient=Jane, Signer=Ed, Type=Medical = 3939 -Month=Jan, Recipient=Joe, Signer=Bernadette, Type=Clothing = 53 -Month=Jan, Recipient=Joe, Signer=Bernadette, Type=Food = 23 -Month=Jan, Recipient=Joe, Signer=Bernadette, Type=Gas = 12 -Month=Jan, Recipient=Joe, Signer=Bernadette, Type=Medical = 232 -Month=Jan, Recipient=Joe, Signer=Ed, Type=Clothing = 29394 -Month=Jan, Recipient=Joe, Signer=Ed, Type=Food = 234 -Month=Jan, Recipient=Joe, Signer=Ed, Type=Gas = 23 -Month=Jan, Recipient=Joe, Signer=Ed, Type=Medical = 920 -Month=Jun, Recipient=Alice, Signer=Bernadette, Type=Clothing = 40 -Month=Jun, Recipient=Alice, Signer=Bernadette, Type=Food = 232 -Month=Jun, Recipient=Alice, Signer=Bernadette, Type=Gas = 544 -Month=Jun, Recipient=Alice, Signer=Bernadette, Type=Medical = 653 -Month=Jun, Recipient=Alice, Signer=Ed, Type=Clothing = 4032 -Month=Jun, Recipient=Alice, Signer=Ed, Type=Food = 29 -Month=Jun, Recipient=Alice, Signer=Ed, Type=Gas = 940 -Month=Jun, Recipient=Alice, Signer=Ed, Type=Medical = 9304 -Month=Jun, Recipient=Bob, Signer=Bernadette, Type=Clothing = 4930 -Month=Jun, Recipient=Bob, Signer=Bernadette, Type=Food = 343 -Month=Jun, Recipient=Bob, Signer=Bernadette, Type=Gas = 43 -Month=Jun, Recipient=Bob, Signer=Bernadette, Type=Medical = 3453 -Month=Jun, Recipient=Bob, Signer=Ed, Type=Clothing = 203 -Month=Jun, Recipient=Bob, Signer=Ed, Type=Food = 39 -Month=Jun, Recipient=Bob, Signer=Ed, Type=Gas = 920 -Month=Jun, Recipient=Bob, Signer=Ed, Type=Medical = 9402 -Month=Jun, Recipient=Jane, Signer=Bernadette, Type=Clothing = 60 -Month=Jun, Recipient=Jane, Signer=Bernadette, Type=Food = 343 -Month=Jun, Recipient=Jane, Signer=Bernadette, Type=Gas = 7657 -Month=Jun, Recipient=Jane, Signer=Bernadette, Type=Medical = 788 -Month=Jun, Recipient=Jane, Signer=Ed, Type=Clothing = 5738 -Month=Jun, Recipient=Jane, Signer=Ed, Type=Food = 84 -Month=Jun, Recipient=Jane, Signer=Ed, Type=Gas = 9204 -Month=Jun, Recipient=Jane, Signer=Ed, Type=Medical = 920 -Month=Jun, Recipient=Joe, Signer=Bernadette, Type=Clothing = 45 -Month=Jun, Recipient=Joe, Signer=Bernadette, Type=Food = 353 -Month=Jun, Recipient=Joe, Signer=Bernadette, Type=Gas = 232 -Month=Jun, Recipient=Joe, Signer=Bernadette, Type=Medical = 343 -Month=Jun, Recipient=Joe, Signer=Ed, Type=Clothing = 304 -Month=Jun, Recipient=Joe, Signer=Ed, Type=Food = 49 -Month=Jun, Recipient=Joe, Signer=Ed, Type=Gas = 83 -Month=Jun, Recipient=Joe, Signer=Ed, Type=Medical = 9203 -Month=Mar, Recipient=Alice, Signer=Bernadette, Type=Clothing = 5654 -Month=Mar, Recipient=Alice, Signer=Bernadette, Type=Food = 234 -Month=Mar, Recipient=Alice, Signer=Bernadette, Type=Gas = 343 -Month=Mar, Recipient=Alice, Signer=Bernadette, Type=Medical = 23456 -Month=Mar, Recipient=Alice, Signer=Ed, Type=Clothing = 8768689 -Month=Mar, Recipient=Alice, Signer=Ed, Type=Food = 543 -Month=Mar, Recipient=Alice, Signer=Ed, Type=Gas = 64 -Month=Mar, Recipient=Alice, Signer=Ed, Type=Medical = 6533 -Month=Mar, Recipient=Bob, Signer=Bernadette, Type=Clothing = 2346 -Month=Mar, Recipient=Bob, Signer=Bernadette, Type=Food = 3453 -Month=Mar, Recipient=Bob, Signer=Bernadette, Type=Gas = 2343 -Month=Mar, Recipient=Bob, Signer=Bernadette, Type=Medical = 343 -Month=Mar, Recipient=Bob, Signer=Ed, Type=Clothing = 8676 -Month=Mar, Recipient=Bob, Signer=Ed, Type=Food = 234 -Month=Mar, Recipient=Bob, Signer=Ed, Type=Gas = 345 -Month=Mar, Recipient=Bob, Signer=Ed, Type=Medical = 6533 -Month=Mar, Recipient=Jane, Signer=Bernadette, Type=Clothing = 234 -Month=Mar, Recipient=Jane, Signer=Bernadette, Type=Food = 564373 -Month=Mar, Recipient=Jane, Signer=Bernadette, Type=Gas = 2453 -Month=Mar, Recipient=Jane, Signer=Bernadette, Type=Medical = 463 -Month=Mar, Recipient=Jane, Signer=Ed, Type=Clothing = 5464 -Month=Mar, Recipient=Jane, Signer=Ed, Type=Food = 64 -Month=Mar, Recipient=Jane, Signer=Ed, Type=Gas = 453 -Month=Mar, Recipient=Jane, Signer=Ed, Type=Medical = 34543 -Month=Mar, Recipient=Joe, Signer=Bernadette, Type=Clothing = 8655 -Month=Mar, Recipient=Joe, Signer=Bernadette, Type=Food = 232 -Month=Mar, Recipient=Joe, Signer=Bernadette, Type=Gas = 3404 -Month=Mar, Recipient=Joe, Signer=Bernadette, Type=Medical = 343 -Month=Mar, Recipient=Joe, Signer=Ed, Type=Clothing = 7645645 -Month=Mar, Recipient=Joe, Signer=Ed, Type=Food = 234 -Month=Mar, Recipient=Joe, Signer=Ed, Type=Gas = 6434 -Month=Mar, Recipient=Joe, Signer=Ed, Type=Medical = 65343 -Month=May, Recipient=Alice, Signer=Bernadette, Type=Clothing = 23 -Month=May, Recipient=Alice, Signer=Bernadette, Type=Food = 5765 -Month=May, Recipient=Alice, Signer=Bernadette, Type=Gas = 304 -Month=May, Recipient=Alice, Signer=Bernadette, Type=Medical = 5 -Month=May, Recipient=Alice, Signer=Ed, Type=Clothing = 9628 -Month=May, Recipient=Alice, Signer=Ed, Type=Food = 673 -Month=May, Recipient=Alice, Signer=Ed, Type=Gas = 856 -Month=May, Recipient=Alice, Signer=Ed, Type=Medical = 9503 -Month=May, Recipient=Bob, Signer=Bernadette, Type=Clothing = 343 -Month=May, Recipient=Bob, Signer=Bernadette, Type=Food = 543 -Month=May, Recipient=Bob, Signer=Bernadette, Type=Gas = 405 -Month=May, Recipient=Bob, Signer=Bernadette, Type=Medical = 3 -Month=May, Recipient=Bob, Signer=Ed, Type=Clothing = 9602 -Month=May, Recipient=Bob, Signer=Ed, Type=Food = 5673 -Month=May, Recipient=Bob, Signer=Ed, Type=Gas = 23 -Month=May, Recipient=Bob, Signer=Ed, Type=Medical = 492 -Month=May, Recipient=Jane, Signer=Bernadette, Type=Clothing = 232 -Month=May, Recipient=Jane, Signer=Bernadette, Type=Food = 5494 -Month=May, Recipient=Jane, Signer=Bernadette, Type=Gas = 20 -Month=May, Recipient=Jane, Signer=Bernadette, Type=Medical = 9 -Month=May, Recipient=Jane, Signer=Ed, Type=Clothing = 9683 -Month=May, Recipient=Jane, Signer=Ed, Type=Food = 3434 -Month=May, Recipient=Jane, Signer=Ed, Type=Gas = 92 -Month=May, Recipient=Jane, Signer=Ed, Type=Medical = 9203 -Month=May, Recipient=Joe, Signer=Bernadette, Type=Clothing = 232 -Month=May, Recipient=Joe, Signer=Bernadette, Type=Food = 3424 -Month=May, Recipient=Joe, Signer=Bernadette, Type=Gas = 4030 -Month=May, Recipient=Joe, Signer=Bernadette, Type=Medical = 12 -Month=May, Recipient=Joe, Signer=Ed, Type=Clothing = 9204 -Month=May, Recipient=Joe, Signer=Ed, Type=Food = 34 -Month=May, Recipient=Joe, Signer=Ed, Type=Gas = 63 -Month=May, Recipient=Joe, Signer=Ed, Type=Medical = 2930 - -## View: Default (active) -Type: row -Month: column -Recipient: row -Signer: page, Ed -format: ,.0 -# -## View: Second -Type: row -Month: row -Recipient: column -Signer: page, Ed -format: ,.0 From 98e0c59ab9a7c0e43cb87eb685a3a5480868ae1b Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:29:06 -0700 Subject: [PATCH 67/72] chore(beads): close WU-L's four beads, correct improvise-1ey, file the date limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-odbc, improvise-7yx, improvise-g6s1 and improvise-q0vb closed by PR #22, with the reasoning on each. improvise-odbc's close records that the ticket's premise was FALSE and that testing it is what made the ticket cheap: quote_name mirrors bare_name character for character, so the round-trip property was complete and the generator needed nothing. The file had been hand-damaged through the v2025-04-09 migration. improvise-1ey's design field named bank-info.improv, which PR #22 deleted. roadmap.org:194 is generated from that field, so the fix belongs at the bd source and the file is regenerated — Copilot found the stale line and WU-L correctly declined to hand-edit a generated file. improvise-l8rh filed: the ambiguous-date heuristic reads American-first, so a European CSV imports with days and months swapped. Pinned rather than changed on the user's ruling — but it is now wrong DETERMINISTICALLY, which PR #22's HashSet fix is what bought. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index a774b7d6..623544fa 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -41,7 +41,7 @@ {"_type":"issue","id":"improvise-mai9","title":"A status message written while a minibuffer is open is never drawn","description":"Found by WU-G while building improvise-lae (PR #27), and CONFIRMED THE HARD WAY in the same PR: Copilot then found a regression that this mechanism causes, which the agent had itself written down an hour earlier and not checked its own change against.\n\nTHE MECHANISM: src/draw.rs::draw_bottom_bar renders the minibuffer INSTEAD OF the status line. So a status_msg written while a minibuffer is open is never drawn at all -- not truncated, not overwritten, simply never rendered.\n\nWHO IS AFFECTED TODAY:\n * CategoryAdd / ItemAdd: a failed `commit-buffer category` is INVISIBLE. Worse, improvise-6t3 deliberately ordered that command's effects so the add effect is emitted LAST, specifically so its failure message would WIN over the command's optimistic status. It wins a race to a surface nobody can see.\n * A failed `:w \u003cpath\u003e` from CommandMode, which is the regression Copilot found on PR #27: the abort stops the batch before ReturnToNormalUnlessMoved and clear-buffer run, so the app stays in CommandMode with the stale command in the minibuffer and the new error hidden behind it.\n\nTHE WORKAROUND ALREADY SHIPPED, and it is a workaround: CommitSave emits [ChangeMode(Normal), SaveAs] so the error lands when the bottom row is the status bar again. That works for one command and does not generalise -- every other text-entry commit has the same hole.\n\nTWO CANDIDATE FIXES, and the choice is the ticket:\n * Draw both: give the minibuffer and the status line separate rows, or render the status into the minibuffer row when a message is pending. Cheap to describe, and it changes frame geometry, which is pinned by exact-frame characterization tests (see improvise-q0vb for what that costs).\n * Make it unrepresentable: a status message set while a minibuffer is open is a bug in the CALLER, so make the caller unable to do it -- e.g. commit paths always leave the mode before reporting. That is what CommitSave now does by hand, promoted to a rule.\n\nPREFER WHICHEVER MAKES THE FAILURE LOUD. The current state is the quiet kind: the code sets a message, the test asserts the message was set, and the user never sees it. A test asserting `app.status_msg == \"...\"` passes in every one of these cases, which is why this survived improvise-6t3's careful effect ordering.\n","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:20Z","dependencies":[{"issue_id":"improvise-mai9","depends_on_id":"improvise-sihr","type":"relates-to","created_at":"2026-09-20T06:12:22Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-kpnr","title":"improvise-ws-server has no authentication, and the documented sharing path now exposes it","description":"Found by Copilot on PR #21 (improvise-o8fa, the reverse-proxy documentation), and it is a finding ABOUT the documentation that PR introduced rather than about code.\n\nimprovise-ws-server has NO AUTHENTICATION BY DESIGN. crates/improvise-ws-server/src/transport.rs says it outright: anyone who can open a socket to the port can rewrite every document it serves. The only control today is that it binds loopback.\n\nTHE README NOW TELLS USERS TO PUT A PUBLIC TLS PROXY IN FRONT OF IT (improvise-o8fa, per the user's ruling that the reverse proxy is the supported path to reaching a TUI-hosted document from an https origin). That instruction REMOVES THE ONLY CONTROL. TLS is confidentiality, not access control, and in a copy-pasteable recipe the two read alike.\n\nSo proxy-side auth is now LOAD-BEARING DOCUMENTATION rather than a server feature. PR #21 mitigated it -- the warning leads the section, the Caddyfile carries basic_auth marked not-optional, the --bind note now separates what loopback buys from what it does not, and the tunnel line says cloudflared/ngrok print public URLs and authenticate nothing -- but mitigation in prose is not the same as a property of the system.\n\nTHE DECISION TO MAKE, and it is a real fork rather than an oversight to correct:\n * The server grows a shared secret or token, checked at the websocket handshake. Small, and it makes the documented path safe by default rather than safe if followed.\n * Or \"auth is the proxy's job\" is the PERMANENT answer, stated in the server's own docs as a design position rather than left as an absence. Defensible -- it is what most single-binary dev servers do -- but then the README recipe is the security boundary and should say so in those words.\n\nDO NOT SPLIT THE DIFFERENCE. A token that is optional and off by default is the worst of both: it reads as a feature and defends nothing.\n\nNOTE THE GENERALISABLE PART, worth keeping whichever way this goes: DOCUMENTATION THAT INSTRUCTS AN ACTION CARRIES THAT ACTION'S CONSEQUENCES. The PR #21 section was scrupulous about one axis (build complexity, the cost of a second binary to a one-binary tool) and blind on another (exposure), and the --bind paragraph actively REASSURED the reader -- \"not reachable from anywhere else\", true of the port and false of the deployment. Neither the ticket, nor gotcha 48, nor the agent's own /code-review pass caught it; a reviewer reading the server's source did.\n","status":"open","priority":2,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T05:45:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:45:28Z","dependencies":[{"issue_id":"improvise-kpnr","depends_on_id":"improvise-kc7n","type":"relates-to","created_at":"2026-09-20T06:09:49Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-p1ym","title":"Project the page-axis item lists so [ and ] work in a browser","description":"The one key pair an independent smoke test found inert that the shared keymap did NOT fix, and it is a genuine projection gap rather than a resolution gap.\n\nGridCache::page_coords carries (category, SELECTED ITEM). PageNext/PagePrev need THE ITEM LIST TO CYCLE THROUGH, which nothing projects. So [ and ] resolve to a command name the client holds but cannot compose an argument for, and they reach Host::handle empty vec -- doctrine(an_upstream_key_is_a_black_hole), so they do nothing at all.\n\nCONFIRMED BY SMOKE TEST 2026-09-20 against the live page: \"[ / ] did not cycle Quarter (never saw Q2 ~17,100)\". The demo model puts Quarter on the page axis specifically so paging is a thing a visitor would try.\n\nTHE AGENT WAS TOLD NOT TO FORCE IT and did not -- a truthful \"these four keys work, [ / ] needs X\" is worth more than five half-working keys.\n\nSCOPE: extend page_coords (or add a sibling) to carry the item list per page-axis category. Note gotcha 27 -- \"what data is visible\" is View::filtering_cats() (Page UNION Filter) while \"what the user can page\" is categories_on(Axis::Page). This needs the SECOND one; projecting the first would let a client try to cycle a Filter axis, which by design it must not.\n\nWhile you are there: check whether the item list should be the EFFECTIVE items (hidden items excluded) rather than all of them, since [ and ] in the TUI cycle what the user can actually select.","status":"open","priority":2,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-o8fa","title":"improvise-ws-server speaks ws:// only, so the deployed page cannot reach it","description":"Found by opening the deployed page (2026-09-20). The guidance the page and the README carried was WRONG in a way that sends someone down a dead end.\n\nFROM AN https ORIGIN A ws:// ADDRESS IS BLOCKED AS MIXED CONTENT, so the deployed GitHub Pages site needs wss://. improvise-ws-server has NO TLS AT ALL -- no rustls, no native-tls, nothing in the crate or its dependencies -- so wss:// is not obtainable from it. The text said \"note that from an https origin a ws:// address is blocked\", which correctly named the symptom and then offered an address that can never work.\n\nTEXT CORRECTED ALREADY: both web/index.html and README.md now say the address must be wss:// from the deployed page, that the server speaks plain ws:// only, and that the ws:// form works when the page is served over http://localhost. This ticket is the missing CAPABILITY.\n\nTHREE WAYS TO PROVIDE IT, and they are not equivalent:\n * TERMINATE TLS IN FRONT (nginx, caddy, cloudflared, ngrok). Zero code, works today, and is what the corrected text points at. But it means the documented path to sharing a TUI-hosted document involves a second piece of software, which is a poor answer for a tool whose selling point is that it is one binary.\n * NATIVE wss:// IN improvise-ws-server -- rustls plus a cert. Real work, and it raises the certificate question: a self-signed cert makes the browser refuse the socket with no usable error, so this is only pleasant with a real cert or an explicitly trusted local CA (mkcert).\n * DECIDE THE DEPLOYED PAGE DOES NOT PAIR WITH A LOCAL SERVER at all, and say so. Defensible: the standalone worker deployment is the point of the Pages site, and ws:// pairing is a LOCAL development convenience that works fine from the local page. Cheapest, and it makes the docs shorter rather than longer.\n\nThe third is probably right for now and the first is the escape hatch; the second only earns its place if sharing a TUI-hosted document to a remote browser becomes a real use case. Do not implement the second by default.","status":"closed","priority":2,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:53:54Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:22:10Z","started_at":"2026-09-20T05:32:35Z","closed_at":"2026-09-20T06:22:10Z","close_reason":"Closed by PR #21 (WU-I). DOCUMENTATION ONLY, per the user's ruling; native wss:// DECLINED.\n\nTHE DECLINE'S SUBSTANCE IS THE CERTIFICATE, NOT rustls. A self-signed cert makes the browser refuse the socket with no usable error, because a WebSocket handshake has no interstitial -- there is no place to click through. A real cert or mkcert is a second piece of software too, so the choice is two-either-way, with one option additionally shipping a TLS stack and a cert-loading surface in a pivot-table binary.\n\nWHAT LANDED: a worked Caddy example, the nginx difference that bites (proxy_http_version 1.1 plus the Upgrade/Connection headers), the honest cost (a second binary, a poor answer for a one-binary tool), and the framing that the standalone worker deployment is the point of the Pages site while ws:// pairing works fine from a local http://localhost page.\n\nTHIS BEAD COST SEVEN COPILOT ROUNDS AND NINE FINDINGS, EVERY ONE IN PROSE. The sequence is the value and is recorded in kb.pl:\n * the first recipe exposed an unauthenticated world-writable server -- documenting a proxy documents an exposure, because TLS termination REMOVES the loopback binding that was the only control;\n * the \"fix\" put Caddy basic_auth in front of a socket the page opens as new WebSocket(url), an API that can set no headers over a handshake that shows no prompt -- BROKEN reads as protection, which is worse than absent;\n * an ngrok --cidr-allow recipe was written without being run (deprecated-but-accepted, so the reviewer's own \"fails at startup\" was also wrong);\n * \"the browser cannot send a credential on a WebSocket handshake\" over-generalised the governing fact and contradicted the Cloudflare Access bullet three paragraphs up, which works BECAUSE the handshake carries a cookie;\n * an IP allow-list cannot see a cross-site socket at all, which is improvise-kc7n;\n * an absolute loopback claim denied a browser mitigation that exists but is mid-rollout;\n * and \"the auth above\" called an allow-list authentication in a section whose opening sentence is that the server has none.\n\nThe governing fact was ALREADY IN THE SAME FILE at every step, as the basis of this very decline. The lesson the repo did not have: a reviewed paragraph is not evidence.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-955n","title":"[bug] Host::commands applies a batch partially before rejecting","description":"Found by Copilot on PR #13 (the retro review of the unreviewed batch), DECLINED there with reasons on the thread, and filed rather than patched.\n\nADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a TYPE (Checked) to make the unchecked path unconstructible. But Checked is PER-COMMAND: Checked::against answers for one Command against the App AS IT STANDS. A ClientFrame::Commands is a SEQUENCE of them, and Host::commands applies in order and abandons at the first rejection. So a frame carrying SwitchView followed by a stale Command::Model has already applied the SwitchView.\n\nSO ADR 001 GUARANTEE IS CURRENTLY TRUE OF A SINGLE COMMAND AND FALSE OF A BATCH.\n\nA SECOND-ORDER EFFECT NOBODY HAD WRITTEN DOWN, found while declining it: Worker::on_frame skips persist() when ANY frame in the answer is Rejected. So a batch that DID apply a SwitchView before being refused leaves Workbook::active_view moved in memory and NOT WRITTEN BACK -- the document on disk and the document in the worker disagree, silently.\n\nWHY IT WAS DECLINED RATHER THAN PATCHED: closing it means the basis check moving from the COMMAND to the BATCH -- every command answered before any is applied. That is an ADR 001 design change with a real question inside it: WHAT DOES A BATCH BASIS MEAN WHEN ITS LATER COMMANDS WERE COMPOSED AGAINST ITS EARLIER ONES EFFECTS? A client resolving one keypress produces SetSelected plus offsets computed from the selection it just set. Answer that before writing code.\n\nREACHABILITY IS BOUNDED TODAY: no client in this workspace composes a batch containing more than one basis-carrying Command::Model -- the batches improvise-bpq2 produces are view commands, which carry no basis. Do this before the second tranche of client-side resolution creates a reason to batch two model commands.\n\nSupersedes the earlier ticket filed from bpq2 if there is one -- dedupe.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:27Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:29Z","started_at":"2026-09-20T05:32:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-08wb","title":"[bug] A refused ClientFrame::Commands keeps the commands before the refusal","description":"Found by improvise-bpq2 (2026-09-20), stated on the variant, and currently UNREACHABLE from any client in this workspace -- filed so it is a decision rather than a discovery.\n\nClientFrame::Commands(Vec\u003cCommand\u003e) batches, because one cursor key resolved client-side is SetSelected plus the offsets viewport_effects returns, and a host that folds the selection but not the offset projects the wrong window. Host::commands applies IN ORDER and ABANDONS AT THE FIRST REJECTION.\n\nTHAT LEAVES A HOLE ADR 001 DOES NOT HAVE. ADR 001 is emphatic that a stale batch is never applied, NOT EVEN PARTIALLY -- and improvise-ondj went to the trouble of a type (Checked) to make the unchecked path unconstructible. But Checked is per-COMMAND, and a ClientFrame::Commands is a sequence of them, so a frame whose third command is refused has already applied the first two.\n\nWHY IT IS NOT LIVE: no client in this workspace composes a Commands frame containing more than one basis-carrying Command::Model -- the batches bpq2 produces are view commands (SetSelected + offsets), which carry no basis.\n\nTHE FIX IS TO MOVE THE BASIS CHECK TO THE BATCH: check every Command::Model in the frame before applying any of them. That is the shape ADR 001 already describes for an effect batch, one level up, and it would make Host::commands match the guarantee the ADR states.\n\nDo this before a client has reason to batch two model commands -- which the second tranche of client-side resolution (the ~15 bindings bpq2 left) may well create.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:42Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:29Z","started_at":"2026-09-20T05:32:29Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-ir7l","title":"[bug] A formula cell is offered as editable and silently reverts","description":"Pre-existing and consistent with the TUI, but improvise-bck just put it on the DEMO FRONT PAGE, which changes who meets it: the first thing a stranger sees is Profit beside Revenue and Cost.\n\nA formula target cell projects CellId::Record, so render.rs::editable says true, so the browser offers an editor. Type into Profit and the write lands -- then recompute_formulas overwrites it on the next pass and the value REVERTS with no explanation.\n\nWHY IT IS NOT SIMPLY A BUG IN editable: a formula target IS a stored coordinate; the cell is a record. What makes it unwritable is not its identity but that a FORMULA OWNS IT. So the fix is a fact the projection does not currently carry -- something like \"this cell is derived\" -- and it belongs beside numeric/error on CellDisplay, which already exists precisely because the SERVER is the side that knows.\n\nCHECK THE TUI BEFORE CHANGING ANYTHING: it has the same behaviour, so whatever is decided should land in both, or the divergence becomes the new bug (gotcha 44 is what that looks like). And note ADR 002 deliberately made settability a fact about the CellId VARIANT -- adding a second, independent \"but not really\" flag is exactly the two-answers-to-one-question shape this codebase keeps curing. Consider instead whether a formula-owned cell should project a different CellId at all.","notes":"VERIFIED CONCRETELY during the PR #13 review (2026-09-20), and the diagnosis is sharper than when this was filed.\n\nTHE ROOT CAUSE: project::cell_id decides Record vs Aggregate purely on whether a regular category sits on Axis::None. IT NEVER ASKS WHETHER THE _Measure COORDINATE NAMES A FORMULA TARGET. So a formula cell is a Record, render.rs::editable says true, and the browser offers an editor.\n\nMEASURED AGAINST THE SHIPPED DEMO:\n col 2 label=\"Profit\" is_record=true text=\"12,200\"\n after applying SetCell(key, Number(999.0)), Profit reads \"12,200\"\n\nThe write is ACCEPTED AND STORED. Model::evaluate_aggregated checks formula_cache BEFORE raw data (model/types.rs:409), so the typed value is shadowed forever with nothing telling the user. Not a revert -- a permanent silent shadow.\n\nAND THE TEST THAT COVERS THIS AREA IS LYING BY ITS NAME. every_visible_cell_in_the_web_demo_is_settable PASSES while its name is FALSE for the Profit column -- a third of the landing-page grid. It was proved discriminating against the one thing it CAN see (the aggregate at (0,0) in examples/demo.improv) and is structurally blind to this. That is exactly the shape AGENTS.md calls this repo's characteristic find, arriving in the test written to prevent it.\n\nTHE DECISION IS STILL THE DECISION, and ADR 002 is where it goes:\n * TEACH cell_id ABOUT FORMULA TARGETS -- changes what EVERY client calls a record, and record identity is also ADR 001's unit of atomicity, so this reaches the basis check too.\n * OR GIVE editable A SECOND INPUT -- but that is the conflation gotcha 42 exists to forbid, and ADR 002 deliberately made settability a fact about the CellId VARIANT precisely so there could not be a second answer.\n\nThe first is probably right and is not small. Until it lands, the demo's front page teaches a visitor that typing into Profit does nothing.","status":"open","priority":2,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T02:29:51Z","created_by":"Edward Langley","updated_at":"2026-09-20T03:35:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -124,7 +124,7 @@ {"_type":"issue","id":"improvise-6mq","title":"Introduce Storage/VFS abstraction in persistence layer","description":"Refactor persistence/mod.rs to go through a Storage trait instead of std::fs directly. Native build wires PhysicalFS; later issues wire OPFS for wasm. Behavior-preserving refactor — .improv files still save and load identically on native. Foundation for the standalone web deployment and for alternative formats (sqlite, parquet) that can target any backend.","design":"Option A: adopt the vfs crate directly (vfs::VfsPath, vfs::PhysicalFS). Pros: batteries included, filesystem-path-based API, open-source implementations. Cons: sync-only; wasm backends may need async. Option B: define our own narrow Storage trait with read_bytes/write_bytes/list/stat methods, sync for native and async-wrapped for wasm. Pros: tailored to our needs. Cons: more code to maintain. Recommend starting with Option A (vfs crate) and falling back to Option B if OPFS's async-only nature forces the issue. Refactor persistence::{save_md, load_md, save_json, load_json} to take \u0026dyn Storage (or VfsPath) as the target parameter. Autosave path resolution (currently uses the dirs crate) goes behind the abstraction too — native resolves to a dirs path, wasm resolves to a fixed OPFS path.","acceptance_criteria":"(1) Persistence functions take a storage parameter rather than reading/writing files directly. (2) Native build wires PhysicalFS (or equivalent) and all existing persistence tests pass unchanged. (3) dirs crate usage is isolated behind a function that can be stubbed for wasm. (4) Round-trip tests still pass. (5) A memory-backed test fixture exists for unit testing persistence without touching disk.","notes":"Depends on crate-split epic step 3 (improvise-8zh — improvise-io extraction) so the refactor happens in the right crate. Valuable independently of wasm: enables testing persistence without tempfile, simplifies fuzzing.","status":"closed","priority":2,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:33:47Z","created_by":"spot","updated_at":"2026-08-29T19:40:58Z","started_at":"2026-08-29T19:37:37Z","closed_at":"2026-08-29T19:40:58Z","close_reason":"persistence::save/load now go through a Storage trait; save/load remain thin PhysicalFs wrappers so every caller and all 83 persistence tests are untouched (AC 2).\n\nCHOSE OPTION B OVER THE RECOMMENDED OPTION A. The design section recommends starting with the vfs crate. I measured the actual surface first: persistence touches the filesystem in exactly two places — write the whole document in save, read the whole document in load. autosave_path is pure path manipulation with no I/O. That does not justify a dependency plus a VfsPath abstraction. The ticket's own con for Option A is decisive: it is sync-only while OPFS, the actual wasm target, is async — adopting A would mean doing B later anyway. The trait is two methods:\n\n fn read(\u0026self, path: \u0026Path) -\u003e Result\u003cVec\u003cu8\u003e\u003e\n fn write(\u0026self, path: \u0026Path, bytes: \u0026[u8]) -\u003e Result\u003c()\u003e\n\nPaths are opaque document names, not a directory tree, so an OPFS or map-backed backend implements it without pretending to have directories, metadata or seeking.\n\nAC 3 IS VACUOUS: 'dirs crate usage is isolated behind a function that can be stubbed for wasm' — there is no dirs usage. dirs = \"6\" is declared at root Cargo.toml:36 and referenced nowhere under src/ or crates/*/src/. It is simply an unused dependency; noted on improvise-v9w, which already covers removing unused root deps.\n\nGzip moved from streaming File+GzEncoder to encode-then-write. The .gz suffix still selects it, which makes gzip a property of the document name rather than of the backend. physical_and_memory_backends_write_identical_bytes pins that swapping backends does not change the bytes.\n\nAC 5 (memory-backed fixture): MemoryFs, plus three persistence tests that round-trip a workbook with categories, items and data through it — no tempfile, no disk. That is the shape a browser backend plugs into.\n\nVerified beyond unit tests: import to .improv, headless add-cat against it, and the .gz path all still work through the CLI.","dependencies":[{"issue_id":"improvise-6mq","depends_on_id":"improvise-8zh","type":"blocks","created_at":"2026-04-14T00:40:08Z","created_by":"spot","metadata":"{}"}],"dependency_count":1,"dependent_count":5,"comment_count":0} {"_type":"issue","id":"improvise-rbv","title":"Include _Measure in _Dim columns for records mode","description":"In records mode, _Dim columns are generated from regular (non-virtual) categories plus a synthetic 'Value' column. _Measure is excluded because it starts with '_'. This means records show a generic 'Value' column instead of showing which measure (Revenue, Cost, etc.) each record belongs to. _Measure items should appear as _Dim columns in records mode so the measure name is visible per-record.","status":"closed","priority":2,"issue_type":"bug","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:30:40Z","created_by":"spot","updated_at":"2026-04-15T11:16:12Z","closed_at":"2026-04-15T11:16:12Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-dz8","title":"Add Axis::Filter for drill-fixed coordinates","description":"Drill mode sets fixed coordinates (e.g. Product=Widgets, Region=North) as Axis::Page, but these should be immutable constraints, not user-navigable page dimensions. Add Axis::Filter variant that: (1) excludes categories from tile bar in tile select mode, (2) excludes from page cycling via [ and ], (3) visually distinguished from Page in the UI. Drill uses Filter instead of Page for its fixed coordinates.","status":"closed","priority":2,"issue_type":"feature","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-14T07:30:04Z","created_by":"spot","updated_at":"2026-09-01T05:45:01Z","started_at":"2026-09-01T05:38:46Z","closed_at":"2026-09-01T05:45:01Z","close_reason":"Axis::Filter added. DrillIntoCell fixes coordinates on Filter instead of Page. All three ticket requirements met: excluded from tile select (View::tileable_cats, used by tile_bar + MoveTileCursor + TileAxisOp), excluded from [/] page cycling (page_cat_data stays on categories_on(Axis::Page)), and visually distinct (Fix ⚲, yellow, via Axis::color -\u003e AxisColor::Filter). Data filtering preserved via View::filtering_cats (Page ∪ Filter), used by GridLayout and add-record-row. Axis::next maps Filter to itself so nothing can rotate into it. .improv grammar gains a 'filter' keyword with round-trip test — defensive, since Filter only occurs in the never-persisted _Drill view. Note: adding the variant caused exactly ONE compile error outside axis.rs, because improvise-rml had already consolidated the display matches.","dependencies":[{"issue_id":"improvise-dz8","depends_on_id":"improvise-rml","type":"blocks","created_at":"2026-04-16T16:02:39Z","created_by":"Ed L","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-1ey","title":"Browser frontend MVP: end-to-end working demo","description":"Wire everything together into a working browser demo: start the ws-server, open the HTML shell in a browser, it loads the wasm client, connects to the websocket, receives the initial snapshot, renders the grid, accepts keystrokes, and round-trips commands to the server. This is the milestone where the architecture proves itself end-to-end. Scope: open an existing .improv file server-side, view it in the browser, type a number into a cell, see the server-side value update and the browser's projection arrive with the new value.","design":"HTML shell: minimal index.html with a \u003cdiv\u003e for the grid and a \u003cscript\u003e tag bootstrapping the wasm module. JS bootstrap: loads wasm, opens websocket connection, wires on_message from the socket into wasm.on_message, wires wasm.on_key_event output into the socket send. Reconnection: on socket close, show a status indicator; on reconnect, request a fresh snapshot. No auth, no TLS, localhost only. Test by pointing at bank-info.improv or any existing test file.","acceptance_criteria":"(1) ws-server serves a file from CLI arg, accepts a websocket connection. (2) Browser loads page, wasm initializes, grid renders. (3) Arrow keys move the cursor locally (view effect) with no round-trip. (4) Typing a number + Enter in a cell round-trips to the server and the new value appears. (5) Scrolling fetches cells for the new viewport. (6) Reload recovers cleanly from current server state. (7) Video or screenshot of the demo attached to the issue.","notes":"Final milestone of the browser epic. Depends on projection layer (improvise-cqq), ws-server (improvise-q08), wasm client (wasm-client id), DOM renderer (dom-renderer id). No new logic — just glue and testing.","status":"open","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:24:44Z","created_by":"spot","updated_at":"2026-07-30T23:06:32Z","dependencies":[{"issue_id":"improvise-1ey","depends_on_id":"improvise-cqq","type":"blocks","created_at":"2026-08-30T10:08:33Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-1ey","depends_on_id":"improvise-cr3","type":"blocks","created_at":"2026-04-14T00:25:43Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-1ey","depends_on_id":"improvise-q08","type":"blocks","created_at":"2026-04-14T00:25:43Z","created_by":"spot","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"improvise-1ey","title":"Browser frontend MVP: end-to-end working demo","description":"Wire everything together into a working browser demo: start the ws-server, open the HTML shell in a browser, it loads the wasm client, connects to the websocket, receives the initial snapshot, renders the grid, accepts keystrokes, and round-trips commands to the server. This is the milestone where the architecture proves itself end-to-end. Scope: open an existing .improv file server-side, view it in the browser, type a number into a cell, see the server-side value update and the browser's projection arrive with the new value.","design":"HTML shell: minimal index.html with a \u003cdiv\u003e for the grid and a \u003cscript\u003e tag bootstrapping the wasm module. JS bootstrap: loads wasm, opens websocket connection, wires on_message from the socket into wasm.on_message, wires wasm.on_key_event output into the socket send. Reconnection: on socket close, show a status indicator; on reconnect, request a fresh snapshot. No auth, no TLS, localhost only. Test by pointing at examples/demo.improv or web/demo.improv.\n\n(Design field corrected 2026-09-19: it named bank-info.improv, which PR #22 deleted as unloadable -- hand-damaged during the v2025-04-09 migration and unreadable since April. roadmap.org line 194 is generated from this field, so the fix belongs here and the file is regenerated, never hand-edited. Note 'no auth, no TLS' is still an accurate description of improvise-ws-server and is now tracked as improvise-kpnr and improvise-kc7n rather than being merely a scope note.)","acceptance_criteria":"(1) ws-server serves a file from CLI arg, accepts a websocket connection. (2) Browser loads page, wasm initializes, grid renders. (3) Arrow keys move the cursor locally (view effect) with no round-trip. (4) Typing a number + Enter in a cell round-trips to the server and the new value appears. (5) Scrolling fetches cells for the new viewport. (6) Reload recovers cleanly from current server state. (7) Video or screenshot of the demo attached to the issue.","notes":"Final milestone of the browser epic. Depends on projection layer (improvise-cqq), ws-server (improvise-q08), wasm client (wasm-client id), DOM renderer (dom-renderer id). No new logic — just glue and testing.","status":"open","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:24:44Z","created_by":"spot","updated_at":"2026-09-20T06:28:41Z","dependencies":[{"issue_id":"improvise-1ey","depends_on_id":"improvise-cqq","type":"blocks","created_at":"2026-08-30T10:08:33Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-1ey","depends_on_id":"improvise-cr3","type":"blocks","created_at":"2026-04-14T00:25:43Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-1ey","depends_on_id":"improvise-q08","type":"blocks","created_at":"2026-04-14T00:25:43Z","created_by":"spot","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-cr3","title":"DOM renderer for browser grid (reads ViewState + render cache)","description":"JS/TS (or Rust via web-sys) layer that subscribes to the wasm client's state and renders the grid to the DOM. MVP: rebuild a \u003ctable\u003e element on each state change, no virtual-DOM diffing. Reads ViewState for mode indicator, cursor highlight, minibuffer text; reads render cache for cell contents, labels, column widths. Captures browser keyboard events and routes them into the wasm client's on_key_event.","design":"Option A: pure JS/TS module that reads wasm-exposed state via JsValue and updates DOM imperatively. Simpler for MVP. Option B: Rust + web-sys in the wasm crate, rendering from inside wasm. More code sharing but bigger bundle. Start with Option A. Renderer: single \u003ctable\u003e for the grid body, \u003cthead\u003e for column labels, \u003ctbody\u003e with rows. On state change, re-render the affected sections. Cursor highlight is a CSS class on the selected \u003ctd\u003e. Mode indicator is a \u003cdiv\u003e above the table. Minibuffer is a \u003cdiv\u003e shown conditionally when mode is Editing/FormulaEdit/etc.","acceptance_criteria":"(1) Grid renders from a ViewState + RenderCache snapshot. (2) Cursor highlight updates on cursor move without full re-render (nice to have, not required for MVP). (3) Mode indicator reflects current AppMode. (4) Keyboard events on document are captured and routed to wasm on_key_event. (5) Works in Chrome and Firefox. Help overlay, panels, import wizard, tile bar — all deferred post-MVP.","notes":"Consumes viewmodels (not the render cache directly) per improvise-edp. Specifically GridViewModel for grid rendering. Shares GridViewModel type and compute_grid_viewmodel function with ratatui's grid widget (improvise-n10) — one derivation, two rendering backends. DOM-specific concerns (device pixel ratio, CSS class names) live in the RenderEnv the viewmodel is computed against, not in the viewmodel itself.\nPRIORITY RAISED P3-\u003eP2 (groomed 2026-08-30): this gates P2 work (improvise-1ey improvise-6jk improvise-djm ), so leaving it P3 made 'bd ready' unworkable in priority order — the P2 cannot be reached without it. Priority should be non-decreasing along dependency edges. Revert if the intent was that the downstream P2 is genuinely not wanted yet.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:24:41Z","created_by":"spot","updated_at":"2026-09-20T00:37:49Z","closed_at":"2026-09-20T00:37:49Z","close_reason":"Landed as ad5d338 (merge of c72832f). 1090 tests passing, clippy/fmt clean, wasm clean on all six browser-bundle crates.\n\nWHAT IT IS: render.rs builds a RenderModel in which every question with a SILENT wrong answer is already decided in Rust under test, because the DOM half (web/app.js) is untested JavaScript -- editable is a fact about the CellId variant (Record only, per ADR 002; None is 'no projected cell', NOT 'aggregate'); the selection is placed by SUBTRACTING the scroll offset, because GridCache is viewport-indexed while ViewState::selected is absolute; the category panel is viewmodel::flatten_cat_tree, never a second walk. The page is four string calls: viewport, on_frame, render_model, typed_cell.\n\nVERIFIED HEADLESS END TO END with the real artifact: the built wasm module driven from Node over a real websocket to a real improvise-ws-server -- seed, render, type 98765, the value moves server-side, the projection comes back formatted as 98,765. That is improvise-1ey acceptance (4) with only the DOM missing.\n\nNOT VERIFIED, stated plainly: no browser and no headless DOM was available. web/app.js is syntax-checked and every call it makes is confirmed present in the generated .d.ts with the right arity, but NO PIXEL OF IT HAS BEEN RENDERED -- click-to-edit, the ResizeObserver extent arithmetic and the CSS are unexercised. That is what remains of 1ey along with keyboard nav (improvise-bpq2: ClientMsg has no Key variant, so a forwarded keypress is silently skipped today).\n\nFOLLOW-UPS FILED: improvise-zv4m/xks6 (protocol should own the framing -- socket.rs mirrors ws-server's enums because a native crate cannot be a wasm dependency), improvise-odbc (bank-info.improv does not parse, which matters because 1ey names it as the demo file).","dependencies":[{"issue_id":"improvise-cr3","depends_on_id":"improvise-edp","type":"blocks","created_at":"2026-04-14T00:53:44Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-cr3","depends_on_id":"improvise-gsw","type":"blocks","created_at":"2026-04-14T00:25:42Z","created_by":"spot","metadata":"{}"}],"dependency_count":2,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-gsw","title":"improvise-wasm-client crate (keymap + reduce_view in wasm)","description":"Create the wasm crate that runs in the browser. Depends on improvise-protocol (Command, ViewState, reduce_view) and on the extracted improvise-command crate for the keymap. Exposes wasm-bindgen entry points: init, on_key_event, on_message (incoming command from server), current_view_state (for the DOM renderer to read). Resolves browser keyboard events locally via the keymap, dispatches the resulting command through reduce_view, serializes the command for sending upstream.","design":"crates/improvise-wasm-client/. wasm-bindgen + web-sys. Entry points: #[wasm_bindgen] fn init() -\u003e Handle (allocates ViewState + RenderCache); fn on_key_event(handle, browser_key_event) -\u003e Option\u003cString\u003e (resolves via keymap, dispatches via reduce_view, returns serialized Command to send upstream or None if no-op); fn on_message(handle, serialized_command) (deserializes incoming Command, applies via reduce_view, marks render dirty); fn render_state(handle) -\u003e JsValue (exposes ViewState + RenderCache for the DOM renderer). Key conversion: browser KeyboardEvent → improvise neutral Key enum via a From impl. The keymap is the same Keymap type used server-side, but with no ratatui dependency (from improvise-command after crate epic step 5).","acceptance_criteria":"(1) Crate compiles to wasm32-unknown-unknown. (2) Size budget: under 500 KB compressed. (3) on_key_event correctly resolves common keys (arrows, letters, Enter, Esc) via the keymap and dispatches through reduce_view. (4) on_message correctly deserializes and applies projection commands. (5) A small JS test harness can drive the crate end-to-end without a real websocket.","notes":"Transport-agnostic at the Rust layer: on_message accepts serialized commands regardless of source, and outgoing commands are returned to the JS bootstrap which routes them to a WebSocket (thin-client epic 6jk) or a worker MessageChannel (standalone epic tm6). The same wasm artifact is reused literally in both deployments — only the JS bootstrap differs. Depends on improvise-cqi (protocol crate) and crate-epic step 5 (improvise-3mm) so Keymap is importable without ratatui. Does not include the DOM renderer itself — that is improvise-cr3, which consumes the state this crate exposes.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:24:37Z","created_by":"spot","updated_at":"2026-09-19T23:40:23Z","closed_at":"2026-09-19T23:40:23Z","close_reason":"crates/improvise-wasm-client landed complete, not a skeleton. 28 tests; cargo check -p improvise-wasm-client --target wasm32-unknown-unknown clean.\n\nTHE TICKET AC 3 IS A FALSE PREMISE and the design turns on it: a browser client resolves almost NO keys locally. A binding names an INTENT resolved against a CmdContext (Model, GridLayout, cursor, yanked cell) while a Command is the RESOLVED MUTATION -- the same reason improvise-gxi AC 4 was closed. So keys travel upstream UNRESOLVED as ClientMessage::Key { code, modifiers }.\n\nTHE ONE LOCAL PATH is minibuffer text entry, allowed for a STRUCTURAL reason rather than convenience: the target buffer is AppMode::minibuffer().buffer_key, the same field TextEntry::new reads (gotcha 11), so the two rules are DERIVED FROM ONE FACT rather than duplicated. Four things were considered and rejected with reasons in the module docs: cursor movement (GridCache is viewport-indexed, so a client cannot tell last-visible-row from last-row and cannot clamp), panel cursors (the flattening depends on expanded_cats), any modified character (KeyPattern::AnyChar matches under ANY modifier on the host, so mirroring the fallback would be a SUPERSET -- local resolution may only ever be a strict subset), and CommandMode Backspace (the single TextEntry::rebind, which leaves the mode on an empty buffer).\n\nimprovise-command is a DEV-dependency only, so the browser graph is formula -\u003e core -\u003e protocol -\u003e client: no App, Workbook, persistence or wizard. tests/lockstep.rs drives the real host and the client over the same keypress across all seven text-entry modes.\n\nAC 2 (under 500KB compressed) UNVERIFIED -- no wasm-pack/wasm-opt available; the dependency graph is the strongest evidence here and someone should measure a real release build. AC 5 (JS harness) not written, and belongs with improvise-cr3.","dependencies":[{"issue_id":"improvise-gsw","depends_on_id":"improvise-3mm","type":"blocks","created_at":"2026-04-14T00:25:41Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-gsw","depends_on_id":"improvise-cqi","type":"blocks","created_at":"2026-04-14T00:25:41Z","created_by":"spot","metadata":"{}"}],"dependency_count":2,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-q08","title":"improvise-ws-server binary (tokio + tungstenite session wrapper)","description":"New binary crate that wraps today's App behind a websocket. Accepts connections, creates a ClientSession per connection, receives Command messages from clients, dispatches them into the App, drains the outbound projection queue, sends projection commands back down. One App instance per session for the MVP (no shared authoritative state across sessions yet).","design":"crates/improvise-ws-server/. Tokio runtime, tokio-tungstenite for websocket. On connect: create App + ClientSession, send initial Snapshot (full ViewState + render cache filled from current viewport). Main loop: recv Command from client → feed into reduce_full (today's App::handle_key path, but command-keyed instead of key-keyed) → drain projection queue → send Commands down. On disconnect: drop session. Single session per connection for MVP. No auth, no collab, localhost only. Wire format: serde_json to start (easy to debug), bincode/postcard later if size matters.","acceptance_criteria":"(1) Binary crate builds and accepts websocket connections on a configurable port. (2) Initial snapshot is sent on connect. (3) Command round-trip works: client Command in → model updated → projection Command out. (4) Disconnection cleanly drops session state. (5) Integration test: spawn server, connect a test client, send a SetCell, verify a CellsUpdated comes back.","notes":"Depends on improvise-protocol (wire types) and improvise-mae/vb4/projection-emission for the server-side logic. The server itself is a thin transport wrapper — all the real logic lives in the App and projection layer.","status":"closed","priority":2,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-14T07:23:14Z","created_by":"spot","updated_at":"2026-09-19T23:46:08Z","closed_at":"2026-09-19T23:46:08Z","close_reason":"crates/improvise-ws-server landed whole; all five ACs met and tested. 14 tests (13 in host.rs + 1 real-socket integration). Host::handle is where every decision is made and every unit test points; transport.rs is framing, JSON and one handshake rule, with a doc comment saying that if the socket test ever becomes the place a behaviour is pinned, that behaviour belongs in host.rs.\n\nONE APP PER DOCUMENT, as ruled. Not per connection (the ticket premise, now false -- Sessions exists to hold many ClientSessions against one App, and per-connection Apps would make broadcast, apply_switch and apply_model dead code and give two tabs on one document two documents) and not per server. No singleton: Host is a value the transport is handed. The multi-document registry slots into transport.rs where host_loop is -- a HashMap\u003cDocumentId, Host\u003e plus a document id on Connect -- and nothing above it changes, because Sessions never owned the App.\n\nTHE ONLY ADDITION TO THE SHARED VOCABULARY IS ClientMsg::Viewport { rows, cols }, and it is derived rather than invented: the extent is the one session input no Command carries by construction. It is also the HANDSHAKE -- the bootstrap Snapshot seed carries the VISIBLE cells, so a snapshot built at extent zero is an empty grid needing an immediate second round trip. That rule lives in transport::connection alone, which keeps Host::connect total.\n\nNO KEYPRESS DOOR: the retraction arrived before one was built, so nothing had to be removed. ClientMsg carries Viewport and Command only.\n\nEvery non-obvious behaviour proven by injection, including both arms of the broadcast (relay to nobody FAILED; relay to everyone FAILED). The document-swap fixture keeps the incoming active view named Default per gotcha 39 fixture warning, so it cannot pass by accident.","dependencies":[{"issue_id":"improvise-q08","depends_on_id":"improvise-cqi","type":"blocks","created_at":"2026-04-14T00:25:39Z","created_by":"spot","metadata":"{}"},{"issue_id":"improvise-q08","depends_on_id":"improvise-cqq","type":"blocks","created_at":"2026-04-14T00:25:40Z","created_by":"spot","metadata":"{}"}],"dependency_count":2,"dependent_count":3,"comment_count":0} @@ -164,7 +164,7 @@ {"_type":"issue","id":"improvise-ktv1","title":"Move the help content into the shared layer so a browser can show help","description":"? and F1 now enter AppMode::Help in the browser -- the shared keymap delivered that for free -- BUT THE BROWSER HAS NO HELP OVERLAY, so the mode changes and nothing appears. src/ui/help.rs is ~600 lines of ratatui and nothing shared carries its content.\n\nTHE INTERIM, already shipped: the page draws AppMode::hint_text() in a #hint element, so each mode states its own keys. That is genuinely useful and is not a placeholder to rip out -- but it is one line, not five pages.\n\nWHY IT WAS DONE THAT WAY RATHER THAN SPECIAL-CASING: the alternative was to make the client refuse \"enter-mode help\", which is exactly the hand-written exception the keymap consolidation had just deleted. Declining to re-add one was right.\n\nTHE REAL FIX is the same shape as every other consolidation this week (design-principles 5): the help CONTENT is data -- pages of (key, description) -- and it belongs in improvise-protocol where both front ends read it. The TUI keeps its ratatui LAYOUT; the browser writes its own DOM; neither restates what the keys are.\n\nNOTE THE OPPORTUNITY: help content that lives beside the keymap can be DERIVED from it rather than written twice, which would close a drift nobody currently checks -- there is no test that the help text and the bindings agree, and doctrine(binding_dump_diff) exists because bindings have almost no direct tests. A help page generated from Keymaps::default_keymaps() cannot go stale.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T04:38:45Z","created_by":"Edward Langley","updated_at":"2026-09-20T04:38:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-e24v","title":"Second tranche of client-side key resolution: ~15 bindings needing no enrichment","description":"improvise-bpq2 closed the cursor half and DELIBERATELY STOPPED rather than half-doing a second tranche. These need NOTHING added to the protocol -- the operands are already in ViewState or a widget cache -- so they are the cheapest remaining keys, and each one currently does NOTHING in a browser (Host::handle answers ClientFrame::Key with an empty vec).\n\n * enter-mode \u003cx\u003e (: ? F1 C F V Esc Tab in most modes) -\u003e ChangeMode\n * toggle-panel-visibility / toggle-panel-and-focus -\u003e SetPanelOpen + ChangeMode\n * cycle-panel-focus\n * help-page-next / help-page-prev\n * transpose, toggle-prune-empty, view-back\n * switch-view-at-cursor / delete-view-at-cursor / create-and-switch-view (ViewListCache is flat)\n * commit-buffer category|formula|item (all operands in ViewState)\n * commit-cell-edit (ADR 002 CellId + the buffer)\n * clear-cell / paste (CellId + ViewState::yanked)\n * set-axis-for-tile / cycle-axis-for-tile\n\nALSO IN REACH, and bpq2 listed them under \"needs enrichment\" before establishing they do not: filter-to-item (Enter), open-item-add-at-cursor (a/o), delete-category-at-cursor (d), cycle-axis-at-cursor (Space) -- CatTreeEntry already carries the name and the axis. Each is a distinct Command shape, which is why they were left.\n\nMETHOD IS SETTLED, follow it: mirror each Cmd LINE FOR LINE including where siblings disagree, and add a lockstep test that drives the client and the real reduce_key over the same keypresses and compares ViewState. Do not tidy sibling commands into one rule -- that makes the client a second derivation, which is what lockstep testing exists to rule out.\n\nAnd update the_normal_mode_keys_that_still_need_the_host_are_these as you go: it is a named list precisely so closing one has to come there and say so.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T02:36:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T02:36:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-s17o","title":"reduce_full and apply_command's view-slice arm have no production caller","description":"Found while fixing improvise-ondj (2026-09-20), and it is the reason that defect could sit unnoticed.\n\nui/reduce.rs module doc says: \"the one place a command is applied to the authoritative state\" and \"Every deployment mode -- the native TUI, a websocket server, a browser worker -- reaches the model through the functions here.\"\n\nTHAT IS ASPIRATIONAL. reduce_full has NO PRODUCTION CALLER ANYWHERE IN THE WORKSPACE. Every host reaches the model through Sessions::apply or reduce_key. Nothing exercised reduce_full Command::Model arm, so nothing could notice it skipped the basis check -- a door advertised by documentation and walked by nobody is exactly the shape that hides a defect until the next host arrives.\n\nSECOND HALF, same shape: apply_command other =\u003e reduce_view(\u0026mut app.view_state, other) folds a wire VIEW command into the AUTHORITATIVE App own slice. Nothing in production does that -- Sessions::apply folds view commands into the CLIENT MIRROR instead. So view_slice_commands_land_in_the_authoritative_view_state is testing a path no host walks, and passing tells you nothing about any deployment.\n\nTHE DECISION: are these a host API worth keeping, or test-only scaffolding wearing a module doc? If the former, HAVE A HOST USE ONE -- improvise-9x6 (the browser worker) is the natural candidate and is being built now, so this is decidable immediately rather than in principle. If the latter, delete them and correct the module doc, which is currently making a promise the code does not keep.\n\nDo not just delete the test: decide what the function is for first.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:54:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-r4lo","title":"Two answers to what a typed string means, and they are adjacent","description":"Noticed during improvise-xks6 (2026-09-20) and left alone, because the envelopes were that ticket. Searched first: nothing open covers it.\n\nTWO CONTRADICTORY DOC COMMENTS, now one level below the envelopes they used to hide behind:\n\n * Client::set_cell_at says classifying a typed string \"is the HOST decision, not one to re-implement here\", and takes a SERIALIZED CellValue.\n * socket.rs::typed says the OPPOSITE -- \"Putting it in Rust rather than in the page JavaScript is what makes it testable\" -- and typed_cell takes a PLAIN STRING.\n\nBoth now return the same ClientFrame, so this is the same disease improvise-xks6 just cured, one level down: TWO ANSWERS TO ONE QUESTION, kept apart only by the fact that nothing called both.\n\nWHICH ONE IS RIGHT: typed_cell is the one web/app.js calls and the one repo-map gotcha 42 blesses -- the browser has no App to ask, and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; putting it in Rust rather than JS is what makes it testable and keeps one spelling per language boundary. set_cell_at is UNREFERENCED by the page.\n\nSo the likely answer is to delete set_cell_at and fix the doc, but check first whether anything else (a test, the lockstep harness, a future worker bootstrap) wants the pre-classified form -- a host that HAS an App genuinely can classify, and improvise-9x6 puts an App in the browser, which may make set_cell_at the right entry point for the WORKER deployment and typed_cell right for the socket one. If so the fix is to say that in both doc comments rather than to delete either. Decide it, do not leave two claims standing.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:52:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:35Z","started_at":"2026-09-20T05:32:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-r4lo","title":"Two answers to what a typed string means, and they are adjacent","description":"Noticed during improvise-xks6 (2026-09-20) and left alone, because the envelopes were that ticket. Searched first: nothing open covers it.\n\nTWO CONTRADICTORY DOC COMMENTS, now one level below the envelopes they used to hide behind:\n\n * Client::set_cell_at says classifying a typed string \"is the HOST decision, not one to re-implement here\", and takes a SERIALIZED CellValue.\n * socket.rs::typed says the OPPOSITE -- \"Putting it in Rust rather than in the page JavaScript is what makes it testable\" -- and typed_cell takes a PLAIN STRING.\n\nBoth now return the same ClientFrame, so this is the same disease improvise-xks6 just cured, one level down: TWO ANSWERS TO ONE QUESTION, kept apart only by the fact that nothing called both.\n\nWHICH ONE IS RIGHT: typed_cell is the one web/app.js calls and the one repo-map gotcha 42 blesses -- the browser has no App to ask, and ModelCommand::SetCell takes a CellValue, so somebody on the client side must decide; putting it in Rust rather than JS is what makes it testable and keeps one spelling per language boundary. set_cell_at is UNREFERENCED by the page.\n\nSo the likely answer is to delete set_cell_at and fix the doc, but check first whether anything else (a test, the lockstep harness, a future worker bootstrap) wants the pre-classified form -- a host that HAS an App genuinely can classify, and improvise-9x6 puts an App in the browser, which may make set_cell_at the right entry point for the WORKER deployment and typed_cell right for the socket one. If so the fix is to say that in both doc comments rather than to delete either. Decide it, do not leave two claims standing.","status":"closed","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:52:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:22:09Z","started_at":"2026-09-20T05:32:35Z","closed_at":"2026-09-20T06:22:09Z","close_reason":"Closed by PR #21 (WU-I). DECIDED: DELETE, do not re-document.\n\nThe ticket offered an escape hatch -- that improvise-9x6's browser App might want the pre-classified form, in which case the fix would be to say so in both doc comments rather than delete either. THAT ESCAPE HATCH DOES NOT EXIST. ModelCommand::SetCell carries a CellValue, so nothing that crosses a frame is ever a string, and the worker's App sits BEHIND the frame boundary and is handed a value. So set_cell_at's claim -- that classifying a typed string is \"the HOST's decision\" -- is false on every deployment, not just on the socket one.\n\nBoth set_cell_at AND set_cell are gone. set_cell cannot express ClearCell, which makes it a STRICTLY WEAKER DUPLICATE of typed_cell_frame rather than a peer -- and its the_json_wrapper_composes_the_same_message was exactly the compatibility test doctrine(a_consolidation_that_keeps_its_compatibility_test_is_unfinished) says a finished consolidation deletes.\n\nTEST COUNT DROPPED 5 AND IS FULLY ACCOUNTED: 2 aimed at deleted code; 3 were None-shape tests already asserted against typed_cell_frame in socket.rs. The 4th None shape -- ADR 001's absent basis -- had no twin, so a_client_with_no_basis_composes_nothing was RETARGETED rather than dropped, and proved discriminating by seeding a Projection::Basis into its fixture and watching it flip to Some.\n\nThree doc comments elsewhere named the deleted fn (keys.rs, state.rs, socket.rs); all re-pointed, per the grep rule.\n\nWHAT REMAINS, filed as improvise-j7v6: socket.rs::typed still duplicates commit_regular_cell_value on the host key path. Both are legitimate -- the browser has no App to ask -- but the RULE is written twice, and a_typed_string_is_classified_as_the_host_classifies_it states the obligation in its name while asserting three outcomes by hand rather than comparing against the host's function, so the two can drift and stay green.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i5tk","title":"Narrow the TUI's GridCache to its viewport -- benchmark first","description":"Disclosed as a caveat of PR #12 rather than discovered by it: it is in the PR body, in the aa7a6d3 commit message, and in repo-map gotcha 44. Copilot raised it independently on round 2.\n\nproject::grid_of evaluates EVERY CELL PER FRAME, where GridWidget used to evaluate only the visible ones.\n\nBE PRECISE ABOUT WHAT CHANGED: this is a WIDENING, not an introduction. compute_col_widths was already whole-grid and the totals loop was already O(rows x visible cols). And it is UNBENCHMARKED IN BOTH DIRECTIONS -- \"this can dominate TUI rendering\" is exactly as unevidenced as \"it is fine\" would be. Do not fix it on the strength of the reasoning.\n\nSO ACCEPTANCE CRITERION #1 IS A BENCHMARK. The fix and the evidence that it was needed are the same work.\n\nWHAT THE FIX COSTS, if the benchmark says do it: teaching GridWidget that a cache position is RELATIVE, which is the browser client arrangement (gotcha 44 -- the TUI builds at offsets zero deliberately, and that is what let the scrolling arithmetic stay untouched through improvise-n10). So this is not a narrowing of a loop, it is a change to what a cache index MEANS in the TUI, and every scroll site is in scope.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:47Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:26:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-6c6q","title":"[bug] apply_switch relays a refused switch to everyone but the asker","description":"Found by the code review of ADR 001 layer 1 (2026-09-19). Pre-existing -- the pre-wave broadcast: Option\u003cCommand\u003e was explicitly \"every OTHER client\" -- and the doc overclaims.\n\napply_switch doc says: \"a client that optimistically folded a name the document turned down is corrected by the same message that tells the others to follow.\" But it relays Broadcast::ToOthers, so THE ASKER IS NOT SENT THE CORRECTION. If the workbook refuses the switch (the view does not exist), the asker slice keeps the asked-for name while its mirror holds the accepted one -- gotcha 24 mismatch, and gotcha 29 says the two copies must move together.\n\nUNLIKE the apply_model case the review fixed, a blanket ToAll is WRONG here: the asker really did fold optimistically on the accepted path, so ToAll would double-apply on the common path.\n\nTHE HONEST FIX IS CONDITIONAL: ToAll when accepted != asked, ToOthers otherwise. Which is to say the audience is a function of whether the asker \"folded the right thing\", not of the command variant -- a slightly sharper version of doctrine(the_audience_follows_the_optimistic_fold).\n\nCheap. The reason it is P3 and not higher is that switching to a view that does not exist is not a thing a well-behaved client does; it is reachable through the : command line.","notes":"ALSO FIX THE DOC COMMENT, and treat it as the more dangerous half. apply_switch's doc (project.rs:~714, verbatim on main) actively claims 'a client that optimistically folded a name the document turned down is corrected by the same message that tells the others to follow.' It is not -- the relay is ToOthers. A comment that says the hole is closed is worse than the hole, because the next reader stops looking.\n\nCopilot surfaced this independently on PR #12 and its suggested shape is right and cheap: apply_switch ALREADY RECEIVES the asked-for name -- it takes the whole cmd, which IS Command::SwitchView(asked) -- it simply never destructures it. So the conditional audience (ToAll when accepted != asked, ToOthers otherwise) needs no new parameter.\n\nTHE REAL COST OF THIS TICKET IS THE REGRESSION TEST, NOT THE CHANGE. It must drive THE ASKER'S OWN SLICE, not the to_others helper -- that helper looks away from the asker by construction, and that is exactly how the sibling bug in apply_model (fixed in 9cb16ce) shipped green.","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:15Z","created_by":"Edward Langley","updated_at":"2026-09-20T01:26:45Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-z754","title":"Document name, dirty flag and file path have no home a client can read","description":"draw_title reads workbook.model.name, model_state.dirty and model_state.file_path. NONE IS PROJECTED. improvise-wasm-client RenderModel carries mode, status and view_name but not these, so A BROWSER TITLE BAR CANNOT SHOW THE DOCUMENT NAME OR THE DIRTY MARKER.\n\nFound closing improvise-n10, alongside the grid-frame gaps.\n\nWORTH DECIDING RATHER THAN JUST ADDING: dirty is a fact about the document that every client shares, and name/path likewise -- so these want a projection or a ViewState field, not a per-widget cache. But note file_path is also the thing Save uses and a browser has no filesystem path in the same sense (gotcha 2, gotcha 28), so \"what a remote client should be told about where the document lives\" is a real question and not obviously \"the path\".","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -254,7 +254,7 @@ {"_type":"issue","id":"improvise-11a","title":"3.1 Configure cargo dist","description":"Run cargo dist init targeting x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin. Skip Windows and musl. Commit generated .github/workflows/release.yml and Cargo.toml additions. Test with v0.1.0-rc1 tag, verify builds, delete rc tag.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:55Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-11T07:49:57Z","close_reason":"Closed","dependencies":[{"issue_id":"improvise-11a","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T21:09:28Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-0s6","title":"Phase 3: Distribution","description":"Configure cargo dist, publish to crates.io, tag v0.1.0 release. Produces prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:51Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:28:48Z","dependencies":[{"issue_id":"improvise-0s6","depends_on_id":"improvise-11a","type":"blocks","created_at":"2026-04-08T23:37:39Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-3tj","type":"blocks","created_at":"2026-04-08T23:37:41Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-0s6","depends_on_id":"improvise-l36","type":"blocks","created_at":"2026-04-08T23:37:40Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-zx8t","title":"draw.rs::mode_style matches AppMode with a wildcard, so the repo map's exhaustiveness promise is false there","description":"Found by WU-G on PR #27 while adding AppMode::SavePrompt, and it is a correction to something context/repo-map.md currently promises.\n\nThe repo map says of AppMode: \"Per-variant UI facts live on the enum as exhaustive-match methods -- label(), hint_text(), panel(), minibuffer(), is_editing(), is_records() (improvise-2hi); ADD A VARIANT AND THE COMPILER NAMES EVERY ONE YOU MUST FILL IN.\"\n\nThat is TRUE of the five enum methods and FALSE of src/draw.rs::mode_style, which matches AppMode with a `_ =\u003e` WILDCARD. So a new variant silently inherits whatever the wildcard arm gives -- SavePrompt currently takes ExportPrompt's DarkGray by accident rather than by decision, which happens to be defensible and was nobody's choice.\n\nWHY IT IS P4 AND NOT HIGHER: the consequence is a colour, and the wrong colour is visible rather than silent. But the reason to fix it is that the repo map states a PROPERTY, and a property that holds in five places and not the sixth is worse than one nobody claimed -- the next person adding a variant will trust the sentence.\n\nTHE FIX IS ONE OF TWO, and they are not equivalent:\n * Make mode_style exhaustive, so the compiler does name it. Note mode_style stays in src/draw.rs deliberately, because Style is a ratatui type and improvise-protocol must not name one (gotcha 26, the AxisColor/MinibufColor pattern).\n * Or give AppMode a semantic colour method in the protocol crate -- the AxisColor treatment -- and let draw.rs do the one mapping. That is the established shape here and it would also give a browser something to use, which draw.rs's Style cannot.\n\nThe second is probably right and is the same question improvise-k099 asks about hint_text/label. Decide them together.\n","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:34Z","started_at":"2026-09-20T05:32:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"closed","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:22:09Z","started_at":"2026-09-20T05:32:34Z","closed_at":"2026-09-20T06:22:09Z","close_reason":"Closed by PR #21 (WU-I). Fixed -- AND THE TICKET'S PROPOSED MECHANISM WAS WRONG, which is the part worth keeping.\n\nThe ticket said: \"distinguish 'never connected' from 'was connected and dropped', which the socket readyState at close time can tell you.\" IT CANNOT. readyState is already CLOSED by the time any close listener runs, whichever happened; and the CloseEvent cannot separate them either, because a refused connection and a mid-session drop are BOTH code 1006 with wasClean:false. Nothing on the event carries the distinction.\n\nThe only discriminator is whether `open` ever fired, which the page must REMEMBER ITSELF. Hence an `opened` flag set in the open handler.\n\nDeliberately NOT the watchdog's fix: the watchdog is disarmed because it exists for silence only, whereas suppressing `close` outright would hide a genuine mid-session disconnect, which is the one thing that listener is really for.\n\nA DETAIL FOUND WHILE FIXING IT, worth knowing before touching this again: armWatchdog runs only AFTER `transport.ready` resolves -- on `open` -- so on the never-opened path the watchdog never exists at all. That is not what its own comment leads you to expect, and it means the original watchdog bug could only ever have fired on a connection that SUCCEEDED.\n\nSTILL UNVERIFIED BY EXECUTION, and stated rather than glossed: web/app.js has never been run here. The change is argued from the WebSocket spec and checked with node --check (proved discriminating by appending an unbalanced paren and watching exit 1), not observed. The four assertions this fix owes are recorded on improvise-vvl3 so the browser harness can discharge them -- including the one that must FAIL against the previous unconditional handler, which is the guard and not the decoration.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"closed","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:44Z","started_at":"2026-09-20T05:32:38Z","closed_at":"2026-09-20T05:54:44Z","close_reason":"Closed by PR #20 (WU-K). All three gaps fixed, each fault-injected.\n\n(a) matching_positions total-ised via split_first. The fix is NOT vec![]: an empty partial constrains NOTHING, so EVERY cell matches -- answering \"none\" one level down would have made deleting a caller's fast path silently flip \"all\" to \"none\". The callers' partial.is_empty() branches are fast paths, not guards, and the test asserts the two doors agree.\n\n(b) invalidate_relabelled now blanks a relabelled COLUMN's total. Rows are the negative arm and stay untouched -- a total spans every row, so no one row frames it, and that arm is what stops \"blank on any relabel\" passing.\n\n(c) mark_features uses .get_mut().expect(...), matching expr_text.\n\nTHE THING THE TICKET DID NOT PREDICT, and the reason this bead was not independent of its own framing: (b) was filed as \"unreachable in practice, fix it anyway because gotcha 33(b) exists to stop apply trusting producers\". Right about the rule, wrong about the cost. Adding a blanking rule on the CONSUMER side changes what every PRODUCER owes, and grid_updates did not pay it: it diffed mirror.totals against desired.totals, so a column renamed while its total stayed the same NUMBER emitted ColLabels and no Totals -- the client blanked its own total and project_diff's self.grid = desired left the server believing it was still there. A client cache wrong forever, gotcha 35's exact failure direction. Found by Copilot. Fixed by diffing totals against the label-folded frame.\n\nWHEN YOU MAKE apply STOP TRUSTING A PRODUCER, GO READ THE PRODUCER.","dependency_count":0,"dependent_count":0,"comment_count":0} From d8a090288bf308245dd66bc923144ad312d2a80b Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:30:21 -0700 Subject: [PATCH 68/72] =?UTF-8?q?fix(fit):=20round=20four=20=E2=80=94=20a?= =?UTF-8?q?=20panic,=20a=20split=20status=20bar,=20and=20the=20question=20?= =?UTF-8?q?restated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four of five fixed. The fifth is the most valuable thing in the review. **A fitted rule means something different in an aggregated cell, and that is improvise-4zl7's blocking question in its general form.** A tree is fitted on individual *records*; an improvise formula is evaluated at a `CellKey`, and in a cell that aggregates over a hidden category `evaluate_aggregated` aggregates each `Expr::Ref` **first** and then evaluates the `IF`. Linear formulas survive that — `SUM(Revenue) - SUM(Cost)` is `SUM(Revenue - Cost)` — and a tree does not: aggregate-then-route is not route-then-aggregate. 4zl7 asks what a *categorical* split means where its coordinate is aggregated away; this is the same question for **every** fitted tree, numeric splits included, and it makes the bead larger and more clearly blocking rather than smaller. Characterized rather than answered, because answering it is the user's call. Worth recording what the fixture actually does: the aggregated cell comes back `Error("ref:Cost")`, not a wrongly-routed number — the loud failure this repo prefers, and *not* something to rely on, since nothing states it. The assertion is therefore the negative (it is not the sum of the per-record predictions), so it covers both outcomes and fails the day aggregation becomes record-wise. **An infinite target panicked out of a function that returns `Result`.** A `.improv` cell holding `inf` parses as an `f64`; `standardize` turns an infinite target into NaN leaves and `number_text` asserts. Verified by disabling the new guard and watching `number_text wants a finite value, got inf`. The two directions differ and the doc says so: an infinite *feature* poisons its range and quietly yields a constant leaf instead. A panic and a meaningless rule, both refused now. `NaN` needs no arm — `is_missing` already spells `"NaN"` as missing, so it is imputed away. **Only the success line was flattened**; refusals interpolate document names too, and a name may legally contain a newline — so a failed fit could split the status bar, which is when a user is most reading it. `one_line` is public and the command's error path uses it. **"a constant — no feature matters" overclaimed.** An empty importance ranking establishes that no shuffle raised the error, which a constant tree satisfies and so does a tree whose splits do not matter on these rows. It now reports what the data established: "nothing measurable — no feature changed the error". **Declined:** the record budget runs after `feature_table` has already grouped and sorted, and `RecordRow::group` linear-scans (`improvise-core/src/view/ layout.rs:43`) so grouping is O(cells x rows). Correct — but that cost is pre-existing and shared with records mode, it is in `improvise-core` which is outside this unit's boundary, and threading a limit into `feature_table` changes a signature `dataset()` and its tests share. Filed as an indexed-grouping ticket. 1230 passing (+2); clippy, fmt and the wasm gate clean; CLI byte-identical. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-command/src/command/cmd/fit.rs | 7 +- crates/improvise-gp/src/fit.rs | 127 +++++++++++++++++- 2 files changed, 131 insertions(+), 3 deletions(-) diff --git a/crates/improvise-command/src/command/cmd/fit.rs b/crates/improvise-command/src/command/cmd/fit.rs index 5c650fec..5178c043 100644 --- a/crates/improvise-command/src/command/cmd/fit.rs +++ b/crates/improvise-command/src/command/cmd/fit.rs @@ -58,6 +58,7 @@ //! ever needs shrinking, a default-on cargo feature that `web/build.sh` turns //! off is the lever, and this paragraph is the number to beat. +use improvise_gp::fit::one_line; use improvise_gp::{FitRequest, fit_measure, fitted_measure_name}; use crate::command::cmd::core::{Cmd, CmdContext}; @@ -150,7 +151,11 @@ impl Cmd for FitMeasure { target_category: outcome.formula.target_category.clone(), }), ], - Err(e) => vec![effect::set_status(format!("fit-measure: {e}"))], + // Flattened like the success line: several refusals interpolate a + // document name, and a name may legally contain a newline + // (Copilot on PR #25). A failure is when a user is most reading + // the status bar, so it is the last place to let one split. + Err(e) => vec![effect::set_status(one_line(format!("fit-measure: {e}")))], } } } diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index 91f80bae..d870a8a2 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -107,7 +107,14 @@ fn encoded_width(table: &FeatureTable) -> usize { /// The same text with every CR and LF turned into a space, so a name a /// document is allowed to hold cannot split a one-line status bar. -fn one_line(text: String) -> String { +/// +/// Public because the **refusals** need it too and they are composed by the +/// host: several interpolate a document name (`feature_table`'s missing-target +/// message, this module's role check), and a name may legally contain a +/// newline. Flattening only the success path left the failure path able to +/// split the bar — and a failure is when a user is most reading it (Copilot on +/// PR #25). +pub fn one_line(text: String) -> String { match text.contains(['\r', '\n']) { false => text, true => text.replace(['\r', '\n'], " "), @@ -284,7 +291,11 @@ impl FitOutcome { .map(|(name, share)| format!("{name} {share:.0}%")) .collect(); let leans = if top.is_empty() { - "a constant \u{2014} no feature matters".to_string() + // What an empty ranking establishes is that **no shuffle raised + // the error**, which a constant tree satisfies and so does a tree + // whose splits happen not to matter on these rows. Saying "a + // constant" claims the first (Copilot on PR #25). + "nothing measurable \u{2014} no feature changed the error".to_string() } else { format!("leans on {}", top.join(", ")) }; @@ -410,6 +421,23 @@ pub fn fit_measure( data.warnings.splice(0..0, table.warnings); let rows = regression_rows(data.target, target)?; + // Non-finite data is a **panic** too, and further down: `standardize` + // turns an infinite target into NaN leaves, an `inf..inf` range makes + // `sample_thresh` produce a NaN threshold, and `number_text` then asserts + // on a value it cannot spell — inside a function that promised a `Result` + // (Copilot on PR #25). A `.improv` cell holding `inf` parses as an `f64`, + // so this is reachable from a document rather than only from a bug. + anyhow::ensure!( + rows.iter() + .all(|r| r.target.is_finite() && r.feat.iter().all(|v| v.is_finite())) + && data + .ranges + .iter() + .all(|(lo, hi)| lo.is_finite() && hi.is_finite()), + "some of the values fitting {target:?} are infinite or not numbers, which a formula \ + cannot spell and a search cannot minimise. Correct or clear those cells first" + ); + // Empty is a **panic**, not a poor fit: `preprocess` drops an entirely // missing column, and `random_tree` then draws `rng.below(0)`, which is `0` // by `SplitMix64`'s own answer to the empty range, so `sample_thresh` @@ -965,6 +993,101 @@ mod tests { assert!(err.to_string().contains("not a numeric measure"), "{err}"); } + /// **The finding that matters most in this whole PR, and it is a product + /// question rather than a defect** (Copilot on PR #25). + /// + /// A tree is fitted on individual **records**. An improvise formula is + /// evaluated at a `CellKey`, and in any cell that aggregates over a hidden + /// category `evaluate_aggregated` aggregates each `Expr::Ref` **first** and + /// then evaluates the `IF`. For a linear formula that is the same answer — + /// `SUM(Revenue) - SUM(Cost)` is `SUM(Revenue - Cost)` — and for a tree it + /// is not: aggregate-then-route ≠ route-then-aggregate. + /// + /// So the stored rule means one thing on a record cell and a different + /// thing one row up. This test **characterises** that rather than asserting + /// it is right, because what to do about it is exactly improvise-4zl7's + /// blocking question in its general form: 4zl7 asks what a *categorical* + /// split means where the coordinate is aggregated away, and this is the + /// same question for every fitted tree, numeric splits included. + #[test] + fn a_fitted_rule_at_an_aggregated_cell_routes_on_the_aggregate_not_the_records() { + let (mut model, view) = document(); + let outcome = + fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands"); + model.add_formula(outcome.formula.clone()); + // `Tag` is the category the aggregated cell sums over, so it is what a + // view would put on `Axis::None` and what `App` passes here. + let none_cats = [TAG.to_string()]; + model.recompute_formulas(&none_cats); + + // Per record, the rule is the rule. + let per_record: Vec = (0..12u32) + .filter_map( + |i| match model.evaluate(&cell(&format!("r{i}"), FIT_INTO)) { + Some(CellValue::Number(n)) => Some(n), + _ => None, + }, + ) + .collect(); + assert_eq!(per_record.len(), 12, "every record must evaluate"); + + // One row up — the whole of `Tag` aggregated away — the formula sees + // the aggregate of `Cost` over every record and routes once on that. + let aggregated = model.evaluate(&CellKey::new(vec![ + (REGION.to_string(), "North".to_string()), + (MEASURE.to_string(), FIT_INTO.to_string()), + ])); + let sum_of_records: f64 = per_record.iter().sum(); + // What it is *not* is the sum of what the rule predicts per record. + // On this document it is in fact `Error("ref:Cost")` rather than a + // wrongly-routed number — which is the loud failure this repo prefers, + // and is **not** something to rely on: it is what this shape happens to + // produce, not a rule anything states. Asserting the negative covers + // both outcomes and fails the day aggregation becomes record-wise, + // which is the day this characterization should be replaced by the + // rule that answered the question. + assert!( + !matches!(aggregated, Some(CellValue::Number(n)) if (n - sum_of_records).abs() <= 1.0), + "the aggregated cell now agrees with the per-record predictions \ + ({aggregated:?} vs {sum_of_records}). improvise-4zl7's question \ + has been answered — replace this characterization with the rule." + ); + } + + /// Copilot on PR #25. A `.improv` cell holding `inf` parses as an `f64`, + /// and nothing checked: `standardize` turns an infinite target into NaN + /// leaves, an `inf..inf` range makes `sample_thresh` produce a NaN + /// threshold, and `number_text` then **asserts** — a panic out of a + /// function whose signature promised a `Result`. + /// + /// Verified by disabling the guard, and the two directions differ: an + /// infinite **target** really does panic in `number_text` ("wants a finite + /// value, got inf"), while an infinite **feature** poisons its range and + /// quietly yields a constant leaf. A panic and a meaningless rule, and both + /// are refused here. + /// + /// `NaN` is deliberately absent: `f64::NAN.to_string()` is `"NaN"`, which + /// `gp_tree::data::is_missing` already spells as *missing*, so it is + /// imputed away and never reaches the search. Only the infinities do. + #[test] + fn a_non_finite_cell_is_refused_rather_than_panicking() { + for (label, value) in [ + ("infinity", f64::INFINITY), + ("negative infinity", f64::NEG_INFINITY), + ] { + for measure in ["Cost", "Score"] { + let (mut model, view) = document(); + model.set_cell(cell("r3", measure), CellValue::Number(value)); + let err = fit_measure(&model, &view, "Score", FIT_INTO, &request()) + .expect_err(&format!("{label} in {measure} must be refused")); + assert!( + err.to_string().contains("infinite or not numbers"), + "{label} in {measure}: {err}" + ); + } + } + } + /// Copilot on PR #25. `.improv` pipe quoting makes a newline legal inside /// a name, so a document can hold a measure whose name would split the TUI /// status bar in two. The whole composed line is flattened once. From affb9a18128d196b0f4cb83a4b9712b27f92cef5 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:32:21 -0700 Subject: [PATCH 69/72] fix(harness): make the two expected-failures deterministic, and unstale 4 docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The browser job **failed in CI** on the previous head while passing locally, with both `test.fail` tests reporting "expected to fail, but passed". That is the worst possible outcome for an expected-failure: it is a check that is green when the bug is present and red when it is absent, so a flaky one inverts CI's meaning rather than merely being noisy. Copilot independently flagged the same test from a different direction on the same head. Two timing assumptions, both wrong off my machine: **A closed port is not a deterministic "refused before open".** Port 9's `close` timing is a property of the host's network stack, and on the GitHub runner it did not arrive inside the window, so `#status` still read "cannot reach" and the assertion passed. Replaced with the harness's **own http server**: it accepts the TCP connection and then fails the WebSocket upgrade, so `error` and `close` both fire promptly, everywhere. The page cannot tell the two apart and neither can `web/app.js` — in both cases the socket never opened. **A timer does not order a close after an open.** The watchdog test's mock closed 300ms in; on a slow host that can land before the page sees `open`, in which case `armWatchdog` never runs and the test passes having exercised nothing. Both mocks now close on the page's **first frame**, which the page sends only after `transport.ready` resolves — so "opened, then dropped" is true by construction rather than by luck. And the setup assertions now assert the **precondition of the bug** rather than that something happened: the m18f test polls the recorded `#status` history for "cannot reach" before enabling `test.fail`, because a bare non-empty check is already satisfied by "connecting to …" (Copilot's exact point). Verified three ways: `--repeat-each=3` green; breaking `paintStatus` gives 2 real failures, not expected ones; simulating PR #21's fix still flips the m18f guard to "expected to fail, but passed". **Four checked-in explanations this branch made false**, also Copilot's, and gotcha 48's third rule for the second time in this PR — when a fact is stated in more than one place, changing it is a grep, not an edit: - `worker-server/src/lib.rs` said `storage_contract` "has still never been run" against `OpfsFs` and that browser verification is deferred. - `bindings.rs` said it is "the only part of this crate that no test in the repository can reach". Still true of the shim, for a narrower reason now worth stating: the harness drives `OpfsFs` directly and never instantiates it. - `vfs.rs::holds_a_document` and its test both said `attach` indexes the zero-byte entry it creates. It no longer does, which is the whole of the fix three commits ago — so the check is now belt and braces rather than the mechanism, and says so. `web/app.js` and `web/worker.js` carry the same stale "untested JavaScript" claim and are **not this unit's files**; reported for the orchestrator rather than edited across the boundary. Co-Authored-By: Claude Opus 5 (1M context) --- .../improvise-worker-server/src/bindings.rs | 8 ++-- crates/improvise-worker-server/src/lib.rs | 15 ++++-- crates/improvise-worker-server/src/vfs.rs | 37 +++++++++----- harness/tests/shell.spec.js | 48 ++++++++++++++----- 4 files changed, 74 insertions(+), 34 deletions(-) diff --git a/crates/improvise-worker-server/src/bindings.rs b/crates/improvise-worker-server/src/bindings.rs index 3a37a440..87bdc775 100644 --- a/crates/improvise-worker-server/src/bindings.rs +++ b/crates/improvise-worker-server/src/bindings.rs @@ -5,9 +5,11 @@ //! is told lives in [`crate::worker`], and everything that could be wrong //! about *which document* is open lives in [`crate::vfs`]. Both are driven by //! host tests. What is left here is awaiting four promises and moving strings, -//! and it is the only part of this crate that no test in the repository can -//! reach — there is no browser here and `createSyncAccessHandle` exists in no -//! harness. +//! and it is the only part of this crate that no test reaches — `harness/` +//! now runs `OpfsFs` itself in a real browser (improvise-vvl3), but it drives +//! the backend directly and never instantiates this shim, so the four +//! `await`s below and the `#[wasm_bindgen]` surface around them are still +//! read rather than run. //! //! # The async seam, and the `Rc` it forces //! diff --git a/crates/improvise-worker-server/src/lib.rs b/crates/improvise-worker-server/src/lib.rs index 4a37f4fd..e6f15087 100644 --- a/crates/improvise-worker-server/src/lib.rs +++ b/crates/improvise-worker-server/src/lib.rs @@ -33,11 +33,16 @@ //! [`bindings`] is the shim, and it is deliberately thin enough to read: it //! awaits `OpfsFs::open`/`attach`/`refresh`, builds an `App` through //! [`vfs::open`], and forwards strings. It is compile-checked for wasm32 and -//! nothing more — `createSyncAccessHandle` exists in no test harness here, and -//! browser verification (improvise-xf1) is deferred by decision, so the -//! `storage_contract` that `OpfsFs` owes has still never been run against it -//! (gotcha 28). **If persistence misbehaves in a browser, `OpfsFs` is the -//! first place to look, not the last.** +//! nothing more — `createSyncAccessHandle` exists in no *host* harness. +//! +//! **What `OpfsFs` owes is no longer unrun.** `harness/` (improvise-vvl3) +//! executes `improvise_io::storage::contract::storage_contract` against it +//! inside a real dedicated Web Worker, under Chromium and Firefox, and the +//! browser CI job can fail the build. Two bugs fell out of the first run. +//! Still uncovered: Safari, because Playwright's WebKit build provides no +//! OPFS at all; and `bindings.rs` itself, which the harness does not +//! instantiate. So if persistence misbehaves in a browser, `OpfsFs` is a much +//! better-lit place to look than it was, and this shim is the dark one. //! //! # Nothing depends on this crate //! diff --git a/crates/improvise-worker-server/src/vfs.rs b/crates/improvise-worker-server/src/vfs.rs index cf23a76c..1f3b0531 100644 --- a/crates/improvise-worker-server/src/vfs.rs +++ b/crates/improvise-worker-server/src/vfs.rs @@ -120,18 +120,24 @@ pub fn open(storage: &dyn Storage, opening: &Opening<'_>) -> Result { /// Whether the backend holds a *document* under `path` — not merely an entry. /// -/// The distinction is the whole of improvise-9x6's first browser failure, and -/// it exists because `list` answers "is there an entry" while this asks "is -/// there something to load". `OpfsFs::attach` must create the file to obtain a -/// sync access handle for a document that does not exist yet, and it indexes -/// what it created — so by the time `boot` gets here, a **fresh origin already -/// lists `demo.improv`**, as a zero-byte file. Asking `documents` alone then -/// skips the seed and hands `load_from` an empty buffer, which is the -/// `Cannot open demo.improv` the deployed page reported. +/// The distinction is the whole of improvise-9x6's first browser failure. +/// `OpfsFs::attach` must create the file to obtain a sync access handle for a +/// document that does not exist yet, and it used to **index what it created** +/// — so a fresh origin already listed `demo.improv` as a zero-byte file, +/// `documents` said it was there, the seed was skipped, and `load_from` got an +/// empty buffer: the `Cannot open demo.improv` the deployed page reported. /// -/// `MemoryFs` has no `attach` — it is not on `Storage` — so no host test could -/// reach this, and the test below builds the empty entry by hand to stand in -/// for what `attach` does. +/// That cause is gone (improvise-vvl3): `attach` indexes nothing and `refresh` +/// skips zero-byte entries, so `list` names documents rather than entries. +/// This check is kept as the second half of the same rule — `list` answers +/// "is there an entry the backend will admit to" and this asks "is there +/// something to load" — and it is now belt and braces rather than the +/// mechanism. Removing it is a separate argument, since a backend that has +/// not been told the rule would put the bug straight back. +/// +/// `MemoryFs` has no `attach` — it is not on `Storage` — so no host test can +/// reach the browser path, and the test below builds the empty entry by hand +/// to stand in for what `attach` used to leave behind. fn holds_a_document(storage: &dyn Storage, path: &Path) -> Result { if !documents(storage)?.iter().any(|p| p == path) { return Ok(false); @@ -176,12 +182,17 @@ mod tests { /// test could see it. /// /// `OpfsFs::attach` creates the file — it must, to get a sync access - /// handle for a document that does not exist yet — and indexes what it - /// created. `boot` attaches *before* it opens, so on a fresh origin + /// handle for a document that does not exist yet — and used to index what + /// it created. `boot` attaches *before* it opens, so on a fresh origin /// `documents()` already contained `demo.improv` as a **zero-byte entry**, /// the seed was skipped as "already there", and `load_from` got an empty /// buffer: `Cannot open demo.improv`. /// + /// `OpfsFs` no longer indexes an attached-but-unwritten entry + /// (improvise-vvl3, asserted in `harness/` against a real browser), so + /// that cause is gone. This stays as the guard for any backend that lists + /// an entry it cannot load. + /// /// `MemoryFs` has no `attach` (it is not a `Storage` method), so the /// seeding tests above passed for a reason that cannot hold in a browser. /// This builds the empty entry by hand to stand in for it. diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js index af75a1e7..dcb4142f 100644 --- a/harness/tests/shell.spec.js +++ b/harness/tests/shell.spec.js @@ -11,6 +11,15 @@ import { test, expect } from "@playwright/test"; const status = (page) => page.locator("#status"); + +/// A peer that accepts the TCP connection and then refuses the WebSocket +/// upgrade — the harness's own static file server. Deterministic where a +/// closed port is not: a refused port's `close` timing is a property of the +/// host's network stack, and on the GitHub runner it did not arrive inside +/// the window, which made two expected-failure tests report "expected to +/// fail, but passed". The page cannot tell the two apart and neither can +/// `web/app.js`: in both cases the socket never opened. +const NO_WEBSOCKET_HERE = "127.0.0.1:8099"; const json = (v) => JSON.stringify(v); /// `web/app.js`'s own HANDSHAKE_TIMEOUT_MS. Named here because two tests have @@ -58,7 +67,7 @@ test.describe("the shell reports what its peer is doing", () => { page, }) => { await page.addInitScript(logStatus); - await page.goto("/index.html?ws=127.0.0.1:9"); + await page.goto(`/index.html?ws=${NO_WEBSOCKET_HERE}`); await expect(status(page)).not.toHaveText("", { timeout: 15_000 }); await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); @@ -88,16 +97,21 @@ test.describe("the shell reports what its peer is doing", () => { // starts passing, Playwright reports "expected to fail but passed", and the // `test.fail()` line has to be deleted. It cannot land silently either way. test("close before open leaves the precise message standing", async ({ page }) => { - await page.goto("/index.html?ws=127.0.0.1:9"); + await page.addInitScript(logStatus); + await page.goto(`/index.html?ws=${NO_WEBSOCKET_HERE}`); // Everything up to here is SETUP, and it is asserted BEFORE `test.fail` // is applied — otherwise a page that failed to load, or a status that // never updated at all, would also be an "expected" failure and CI would - // stay green through it (Copilot, PR #24). Waiting for the status to - // settle on SOMETHING rather than on a particular message is what makes - // this arm work under both implementations: "cannot reach" would retry - // against a value the page has already replaced, and "disconnected" - // would time out for the wrong reason once the fix lands. - await expect(status(page)).not.toHaveText("", { timeout: 15_000 }); + // stay green through it (Copilot, PR #24). + // + // The setup asserted is the PRECONDITION of the bug, not merely that + // something happened: the socket must have failed to open (so "cannot + // reach" was written) and then closed (so something could overwrite it). + // Asserted over the recorded history, because the whole subject is a + // message being replaced, which the instantaneous value cannot see. + await expect + .poll(() => page.evaluate(() => window.__statusLog ?? []), { timeout: 15_000 }) + .toEqual(expect.arrayContaining([expect.stringContaining("cannot reach")])); await page.waitForTimeout(1_000); // From here on the known bug owns the outcome. @@ -109,8 +123,10 @@ test.describe("the shell reports what its peer is doing", () => { // socket that really did open and then dropped IS a disconnect, and saying // so is the whole reason `close` cannot simply be suppressed. test("a socket that opened and then dropped reads as disconnected", async ({ page }) => { + // Closes on the page's first frame, so "it opened first" is true by + // construction rather than by timer. await page.routeWebSocket("ws://127.0.0.1:8790/", (ws) => { - setTimeout(() => ws.close(), 300); + ws.onMessage(() => ws.close()); }); await page.goto("/index.html?ws=127.0.0.1:8790"); await expect(status(page)).toContainText("connected", { timeout: 15_000 }); @@ -130,13 +146,19 @@ test.describe("the shell reports what its peer is doing", () => { // timer still running. Filed rather than fixed here: `web/app.js` is not // this unit's file. test("the watchdog does not overwrite a mid-session disconnect", async ({ page }) => { + // Close on the page's FIRST FRAME rather than on a timer. The page sends + // its viewport only after `transport.ready` resolves, which happens on + // `open`, so this makes "opened, then dropped" true by construction. A + // timer does not: on a slow host the close can land before the page sees + // `open`, in which case the watchdog is never armed and the test passes + // having exercised nothing — which is what happened in CI. await page.routeWebSocket("ws://127.0.0.1:8791/", (ws) => { - setTimeout(() => ws.close(), 300); + ws.onMessage(() => ws.close()); }); await page.goto("/index.html?ws=127.0.0.1:8791"); - // Setup, asserted before `test.fail` for the reason above: a socket that - // never opened would otherwise make this pass as an expected failure - // while testing nothing. + // Setup, asserted before `test.fail`: the socket really opened, and then + // really dropped. Both, in that order. + await expect(status(page)).toContainText("connected", { timeout: 15_000 }); await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); From 4abadb8e6dc61abf2dea28b0fb230a0f0fd1da37 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:38:09 -0700 Subject: [PATCH 70/72] test(harness): drop two expected-failures whose trigger is platform-dependent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An expected-failure is green when the bug is present and red when it is absent. One whose trigger depends on the host therefore does not merely flake — it **inverts CI's meaning**. These two failed the browser job twice with "expected to fail, but passed", and the second time was *after* the determinism work in the previous commit, which is the evidence that the problem is not the assertions' precision. Neither bug reproduces on the GitHub runner: - improvise-m18f, where the unconditional `close` listener overwrites "cannot reach …" with "disconnected" — on the runner `close` does not follow `error` inside the window, even against a server that accepts the TCP connection and then refuses the upgrade. - The handshake watchdog replacing a mid-session "disconnected" with "accepted the connection but sent no document" — on the runner it does not land, though the page demonstrably opened and then dropped, since both are asserted as setup. The trigger for both is the **order and timing of `error`, `close` and a timer on the host's network stack**. That is not a property of `web/app.js` and not something this harness can pin, so pretending otherwise was the mistake. Both are real, both were measured on macOS/Chromium, and the measurement is recorded in the file with the exact status sequences so the next person does not re-derive it — including the fact that the watchdog one **survives** the m18f fix, because that fix is about which message `close` writes rather than about the timer still running. What remains is the portable half, and it is the half that was doing the work: every message the page must write ("bad frame", "cannot reach", "connected"/"disconnected", "sent no document"), the one case where the watchdog must stay silent, and the two standalone-deployment tests. Filed for the orchestrator; `web/app.js` is not this unit's file. 12 passing, 3 skipped (WebKit has no OPFS), 23s. Co-Authored-By: Claude Opus 5 (1M context) --- harness/tests/shell.spec.js | 104 ++++++++++++------------------------ 1 file changed, 34 insertions(+), 70 deletions(-) diff --git a/harness/tests/shell.spec.js b/harness/tests/shell.spec.js index dcb4142f..96bf1cb6 100644 --- a/harness/tests/shell.spec.js +++ b/harness/tests/shell.spec.js @@ -80,45 +80,6 @@ test.describe("the shell reports what its peer is doing", () => { ); }); - // improvise-m18f, and the regression guard for the fix in PR #21. - // - // `close` fires on both a connection that never opened and one that was - // established and dropped, and NOTHING on the event tells them apart: - // `readyState` is already CLOSED in any close listener, and both arrive as - // code 1006 with `wasClean: false`. Distinguishing them needs a flag set in - // the `open` handler. - // - // `test.fail` is the injected fault, aimed at the implementation actually - // on this branch rather than at a mutant invented for the occasion: the - // close handler is unconditional here, so it overwrites the precise - // "cannot reach" with "disconnected" and this assertion FAILS. That is what - // makes it a guard — against the unconditional handler, "always say cannot - // reach" would pass and prove nothing. When the `opened` flag lands, this - // starts passing, Playwright reports "expected to fail but passed", and the - // `test.fail()` line has to be deleted. It cannot land silently either way. - test("close before open leaves the precise message standing", async ({ page }) => { - await page.addInitScript(logStatus); - await page.goto(`/index.html?ws=${NO_WEBSOCKET_HERE}`); - // Everything up to here is SETUP, and it is asserted BEFORE `test.fail` - // is applied — otherwise a page that failed to load, or a status that - // never updated at all, would also be an "expected" failure and CI would - // stay green through it (Copilot, PR #24). - // - // The setup asserted is the PRECONDITION of the bug, not merely that - // something happened: the socket must have failed to open (so "cannot - // reach" was written) and then closed (so something could overwrite it). - // Asserted over the recorded history, because the whole subject is a - // message being replaced, which the instantaneous value cannot see. - await expect - .poll(() => page.evaluate(() => window.__statusLog ?? []), { timeout: 15_000 }) - .toEqual(expect.arrayContaining([expect.stringContaining("cannot reach")])); - await page.waitForTimeout(1_000); - - // From here on the known bug owns the outcome. - test.fail(true, "improvise-m18f: the close handler is unconditional on this branch"); - await expect(status(page)).toContainText("cannot reach", { timeout: 1_000 }); - }); - // The other arm, which must keep working when the first one is fixed: a // socket that really did open and then dropped IS a disconnect, and saying // so is the whole reason `close` cannot simply be suppressed. @@ -133,38 +94,41 @@ test.describe("the shell reports what its peer is doing", () => { await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); }); - // A THIRD bug, found by measuring rather than predicted: on the - // open-then-close arm the watchdog IS armed (it is armed after - // `transport.ready`, which resolved on `open`), nothing disarms it, and 8 - // seconds later it replaces "disconnected" with "accepted the connection - // but sent no document" — which is false; the peer did connect and then - // went away. + // TWO BUGS THIS FILE DELIBERATELY DOES NOT ASSERT, and the reason is worth + // more than the assertions would have been. // - // It is the same class as the bug `setShellFailure` fixed (a watchdog must - // be disarmed, not pre-empted) and it survives the improvise-m18f fix, - // because that fix is about which message `close` writes, not about the - // timer still running. Filed rather than fixed here: `web/app.js` is not - // this unit's file. - test("the watchdog does not overwrite a mid-session disconnect", async ({ page }) => { - // Close on the page's FIRST FRAME rather than on a timer. The page sends - // its viewport only after `transport.ready` resolves, which happens on - // `open`, so this makes "opened, then dropped" true by construction. A - // timer does not: on a slow host the close can land before the page sees - // `open`, in which case the watchdog is never armed and the test passes - // having exercised nothing — which is what happened in CI. - await page.routeWebSocket("ws://127.0.0.1:8791/", (ws) => { - ws.onMessage(() => ws.close()); - }); - await page.goto("/index.html?ws=127.0.0.1:8791"); - // Setup, asserted before `test.fail`: the socket really opened, and then - // really dropped. Both, in that order. - await expect(status(page)).toContainText("connected", { timeout: 15_000 }); - await expect(status(page)).toContainText("disconnected", { timeout: 15_000 }); - await page.waitForTimeout(HANDSHAKE_TIMEOUT_MS + 1_000); - - test.fail(true, "the close handler does not disarm the handshake watchdog"); - await expect(status(page)).toContainText("disconnected", { timeout: 1_000 }); - }); + // 1. improvise-m18f: the socket's `close` listener is unconditional, so on + // a connection that never opened it overwrites "cannot reach …" with + // "disconnected". + // 2. On a connection that DID open and then dropped, nothing disarms the + // handshake watchdog, so 8s later "disconnected" is replaced by + // "accepted the connection but sent no document" — which is false. Same + // class as the bug `setShellFailure` fixed (a watchdog must be disarmed, + // not pre-empted, gotcha 48), and it SURVIVES the m18f fix, because that + // fix is about which message `close` writes rather than about the timer + // still running. + // + // Both are real and both were measured, on macOS/Chromium: + // + // never-opened: connecting… -> cannot reach -> disconnected + // open-then-close: connecting… -> connected -> disconnected + // -> accepted the connection but sent no document + // + // They were asserted here as `test.fail` expected-failures and **had to be + // removed**: neither reproduces on the GitHub runner, where `close` does + // not follow `error` within the window and the watchdog does not land. An + // expected-failure is green when the bug is present and red when it is + // absent, so one whose trigger is platform-dependent does not merely flake + // — it **inverts CI's meaning**, and it failed the build twice saying + // "expected to fail, but passed". + // + // What that leaves is a genuine limit rather than an oversight: the trigger + // for both is the ORDER AND TIMING of `error`/`close`/timer on the host's + // network stack, which is not a property of `web/app.js` and not something + // a harness can pin. The four assertions that remain in this file are the + // portable half — every message the page must write, and the one case where + // the watchdog must stay silent. Filed for the orchestrator with the + // recipe above; fixing them is `web/app.js`, which is not this unit's file. // The first-frame watchdog (gotcha 46), found by reading. The condition is // one no per-transport check would catch: the peer accepted the connection From 3bcc12609a62827ca352143b106e7a4db2b598fe Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:41:38 -0700 Subject: [PATCH 71/72] fix(fit): the fit's own name must not already mean something MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot's fifth pass, one finding and it is right. `into` was checked against `measure_item_names()`, and `Expr::Ref` does not ask that question: `Model::find_item_category` finds a **regular category's** item before it falls back into `_Measure` (gotcha 7). So naming the fit after an item of some category installed cleanly, and every later formula referring to the fitted measure then read the category coordinate instead — silently, because both resolve. The check now asks the document the same question `Expr::Ref` will ask, which is both stronger than the measure-only form and simpler than it. Two regressions: one that a category item's name is refused, and one that earns the refusal by asserting the property it protects — a second formula `Doubled = |Score (fit)| * 2` reads twice the fit, per record, against a real document. 1232 passing (+2); clippy, fmt and the wasm gate clean. Co-Authored-By: Claude Opus 5 (1M context) --- crates/improvise-gp/src/fit.rs | 60 +++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/crates/improvise-gp/src/fit.rs b/crates/improvise-gp/src/fit.rs index d870a8a2..9b0b8437 100644 --- a/crates/improvise-gp/src/fit.rs +++ b/crates/improvise-gp/src/fit.rs @@ -379,10 +379,17 @@ pub fn fit_measure( "a fit of {target:?} cannot be stored as {target:?} — the formula would shadow the \ cells it was fitted to. Name a different measure" ); + // **Any** existing item, not only an existing measure. `Expr::Ref` resolves + // through `Model::find_item_category`, which finds a *regular* category's + // item before it falls back into `_Measure` (gotcha 7) — so naming the fit + // after an item of some category installs cleanly and then makes every + // later formula that references it read the category coordinate instead + // (Copilot on PR #25). Asking the document the same question `Expr::Ref` + // will ask is both stronger than the measure-only check and simpler. anyhow::ensure!( - !model.measure_item_names().iter().any(|m| m == into), - "{into:?} is already a measure; a formula on it would replace what is there. \ - Name a measure that does not exist yet" + model.find_item_category(into).is_none(), + "{into:?} already names an item of this document, so a formula referring to the fit \ + would resolve to that instead. Name something that does not exist yet" ); let table = feature_table(model, view, target)?; // The width guard runs on the RAW table, before `preprocess` materialises @@ -1249,7 +1256,52 @@ mod tests { let (model, view) = document(); let err = fit_measure(&model, &view, "Score", "Cost", &request()).expect_err("Cost holds data"); - assert!(err.to_string().contains("already a measure"), "{err}"); + assert!(err.to_string().contains("already names an item"), "{err}"); + } + + /// Copilot on PR #25. The check used to ask only about *measures*, and + /// `Expr::Ref` does not: `Model::find_item_category` finds a regular + /// category's item **before** falling back into `_Measure` (gotcha 7). So + /// naming the fit after some category's item installed cleanly and then + /// made every later formula referring to it read the category coordinate. + #[test] + fn a_fit_named_after_a_category_item_is_refused() { + let (mut model, view) = document(); + model.category_mut(REGION).expect("Region exists").kind = CategoryKind::Regular; + assert!( + model.find_item_category("North").is_some(), + "the fixture must really have an item named North" + ); + let err = fit_measure(&model, &view, "Score", "North", &request()) + .expect_err("North is an item of Region"); + assert!(err.to_string().contains("already names an item"), "{err}"); + } + + /// And the property that refusal protects, asserted against a real + /// document: a second formula referring to the fitted measure reads the + /// **fit**, not something else that happens to share its name. + #[test] + fn a_later_formula_referring_to_the_fit_reads_the_fit() { + let (mut model, view) = document(); + let outcome = + fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands"); + model.add_formula(outcome.formula.clone()); + model.add_formula( + improvise_formula::parse_formula( + &format!("Doubled = {} * 2", improvise_formula::quote_name(FIT_INTO)), + MEASURE, + ) + .expect("parses"), + ); + model.recompute_formulas(&[]); + for i in 0..12u32 { + let fitted = model.evaluate(&cell(&format!("r{i}"), FIT_INTO)); + let doubled = model.evaluate(&cell(&format!("r{i}"), "Doubled")); + let (Some(CellValue::Number(a)), Some(CellValue::Number(b))) = (fitted, doubled) else { + panic!("record r{i} did not evaluate both measures"); + }; + assert!((b - a * 2.0).abs() < 1e-6, "r{i}: {b} is not twice {a}"); + } } /// improvise-8xhy reaching a real user, and the reason improvise-4zl7 From 3ba35a0ed82cd46d9e3a1036bd0a55974777a4f4 Mon Sep 17 00:00:00 2001 From: Edward Langley Date: Sat, 19 Sep 2026 23:44:52 -0700 Subject: [PATCH 72/72] chore(beads): close WU-J's three beads, narrow xf1 to Safari, file the watchdog bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit improvise-vvl3, improvise-vhcz and improvise-hmws closed by PR #24. vvl3's close records the finding that outlives the harness: a contract nobody can execute is prose. storage_contract had been cited in gotcha 28 for months as the spec OpfsFs must satisfy, and the first time OpfsFs was handed to it, it failed on its FIRST assertion — on a semantic disagreement about what list means, sitting in a trait doc everyone had read. hmws needed neither of its two proposed fixes: an orphan-rule refusal is a fact about where you are standing, not about the pair of types. The impl is simply allowed in the crate that owns the trait. improvise-xf1 narrowed rather than closed, and raised to P2. Chromium and Firefox now run the full battery; Playwright's WebKit provides no OPFS at all, so Safari is untested and the ticket's own 16.4+ claim is unverified. Parked as a runtime probe-skip so it starts running when that changes. improvise-xps8 filed: the handshake watchdog overwrites a genuine mid-session disconnect eight seconds later. A third bug in the same family, and it survives improvise-taxx's fix — confirmed by simulating that fix rather than assuming. Not a CI guard: its trigger is the host's network stack, and an expected-failure whose trigger is platform-dependent inverts CI's meaning rather than flaking. Co-Authored-By: Claude Opus 5 (1M context) --- .beads/issues.jsonl | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 623544fa..b4b2ef72 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -153,6 +153,8 @@ {"_type":"issue","id":"improvise-pby","title":"2.2 Write the README","description":"Replace existing README.md with new one under 250 lines. Sections in order: Title, one-sentence pitch, inline demo GIF, why this exists (Lotus Improv reference), quick start, key bindings, installation (Nix/crates.io/prebuilt - no Homebrew), codebase overview, expectations disclaimer, license. No badges, TOC, or contributing guide.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:32Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:02:52Z","close_reason":"README written with all 10 required sections, committed. License updated to Apache-2.0 per user request.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-ihv","title":"2.1 Add Nix tooling for asciinema, VHS, and cargo-dist","description":"Modify flake.nix to add pkgs.asciinema and pkgs.vhs to dev shell nativeBuildInputs. Add cargo-dist if packaged in nixpkgs. Add runtime deps for VHS (ttyd, ffmpeg) if needed. Verify tools work via nix develop --command. Add nix run app or shell script for demo recording workflow.","status":"closed","priority":2,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-09T04:07:30Z","created_by":"Edward Langley","updated_at":"2026-07-30T23:06:31Z","closed_at":"2026-04-09T07:30:19Z","close_reason":"asciinema, vhs, cargo-dist added to flake.nix dev shell and verified. cargo-dist wrapper enables 'cargo dist' subcommand. scripts/record-demo.sh added.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"improvise-yk4","title":"Phase 2: README and demo artifacts","description":"The main launch work. README is 80% of the launch, demo artifacts are 20%. Covers Nix tooling for asciinema/VHS, writing the README, creating demo GIF and asciinema casts, and verifying all artifacts.","status":"closed","priority":2,"issue_type":"feature","assignee":"spot","owner":"el-github@elangley.org","created_at":"2026-04-09T04:05:49Z","created_by":"Edward Langley","updated_at":"2026-04-14T07:47:43Z","closed_at":"2026-04-14T07:47:43Z","close_reason":"All 5 subtasks closed; artifacts verified present and tracked in git (README.md, docs/demo.gif, docs/demo.tape, docs/casts/*.cast).","dependencies":[{"issue_id":"improvise-yk4","depends_on_id":"improvise-abz","type":"blocks","created_at":"2026-04-08T23:37:37Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-d4w","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-ihv","type":"blocks","created_at":"2026-04-08T23:37:34Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-odx","type":"blocks","created_at":"2026-04-08T23:37:35Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-yk4","depends_on_id":"improvise-pby","type":"blocks","created_at":"2026-04-08T23:37:36Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":5,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-xps8","title":"The handshake watchdog overwrites a genuine mid-session disconnect eight seconds later","description":"Found by improvise-vvl3's harness (PR #24) -- the first time web/app.js had ever been executed -- and it is a THIRD bug in the same family, distinct from both of the ones already dealt with.\n\nTHE SEQUENCE, measured rather than argued:\n\n open-then-close: connecting... -\u003e connected -\u003e disconnected\n -\u003e \"accepted the connection but sent no document\" \u003c- EIGHT SECONDS LATER\n\nOn the open-then-close arm the handshake watchdog IS armed (armWatchdog runs after transport.ready, which resolves on `open`), and nothing disarms it when the socket later closes. So a genuine mid-session disconnect is correctly reported, and then silently replaced eight seconds later by a message that is FALSE -- the peer did not fail to answer a handshake, it answered and then went away.\n\nIT SURVIVES improvise-taxx'S FIX, confirmed rather than assumed: the harness simulated the `opened` flag and the assertion still fails. taxx distinguishes never-connected from was-connected-and-dropped in the CLOSE listener; this is the TIMER overwriting whatever that listener concluded. Different mechanism, same surface.\n\nIT IS THE SAME CLASS AS THE BUG setShellFailure ALREADY FIXED -- a timer firing later and replacing a more precise message (gotcha 48, \"a watchdog must be disarmed, not pre-empted\"). The fix for that one cleared the timer on a LOUD failure. This arm is not loud: a close after a successful open is an ordinary event, not an error, so it never reaches setShellFailure. The likely answer is that the close listener disarms the watchdog on the opened path, since once a socket has opened the handshake question is settled either way -- but check whether any path legitimately wants the watchdog to survive a close before assuming it.\n\nWHY IT IS NOT A CI GUARD, and this is the part worth reading before trying to test it: the harness shipped an assertion for this and HAD TO REMOVE IT. It reproduces on macOS/Chromium and NOT on the GitHub runner, because the trigger is the ORDER of error/close/timer on the host's network stack. A Playwright test.fail is green when the bug is present and red when it is absent, so an expected-failure whose trigger is platform-dependent does not merely flake -- IT INVERTS CI'S MEANING, and it failed the browser job twice reporting \"expected to fail, but passed\". See doctrine(an_expected_failure_must_not_be_platform_dependent). The measurement and the reproduction recipe are recorded in harness/tests/shell.spec.js.\n\nNOTE THE SAME CAVEAT NOW APPLIES TO improvise-taxx: its bug is also only locally reproducible, which is why the two assertions the fix owed could not be shipped as guards either. The fix is still right; it is the pinning that is unavailable.\n","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:44:26Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:44:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-l8rh","title":"The ambiguous-date heuristic reads American-first, so a European CSV imports with days and months swapped","description":"Named and pinned by WU-L on PR #22 (improvise-7yx item 2), and left UNCHANGED by the user's ruling 2026-09-19: leave it pinned, file it as a known limit.\n\nTHE BEHAVIOUR: import/analyzer.rs::detect_date_format is a FIRST-MATCH heuristic, so for a column of values that fit both %m/%d/%Y and %d/%m/%Y -- \"01/02/2025\", \"03/04/2025\" -- the AMERICAN reading wins. A European CSV therefore imports with days and months SWAPPED, silently, and every date in the column is wrong in a way that still looks like a date.\n\nWHY IT IS TRACKED RATHER THAN FIXED, and this is the argument that carried:\n * The import wizard SHOWS the detected format and lets the user override it, so the mis-read is visible at the point of import to a user who looks. That is the same argument detect_short_year_format already recorded for its own guess.\n * Changing the default to day-first only moves which half of the world imports silently wrong.\n * The honest fix is larger than the heuristic: an AMBIGUOUS column should surface its ambiguity rather than resolve it, which changes the analyzer's return shape (FieldProposal) and the wizard step that displays it. Worth doing deliberately, not as a patch.\n\nWHAT PR #22 DID CHANGE, and it is why this is now a stable known limit rather than a lottery: analyze_records used to iterate a HashSet, so the ten samples detect_date_format looks at were a DIFFERENT TEN EVERY RUN -- meaning a column whose eleventh value rules out %m/%d/%Y detected a different format run to run. Same CSV, same import, different dates. That is fixed (first-seen order, invariant(analyzer_first_seen_order)), so the heuristic is now WRONG DETERMINISTICALLY, which is the precondition for anyone being able to reason about it at all.\n\nPINNED BY: an_ambiguous_date_is_read_american_first, whose doc-comment states plainly that this silently mis-reads a European CSV, with the day-first reading asserted to parse so the test is a statement about AMBIGUITY rather than about parsing. a_day_first_only_value_settles_the_ambiguity is the discriminating other arm.\n\nSTILL TRUE AND UNCHANGED: the ten-sample window itself. A column whose first ten values are ambiguous and whose eleventh is not will still be read american-first. Now deterministic, still a limit.\n","status":"open","priority":3,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T06:28:43Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:28:43Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-rw3a","title":"Decide whether ZZ routes into the save prompt automatically","description":"The half of improvise-lae that was deliberately left open. PR #27 built the prompt (AppMode::SavePrompt, TextEntry builder, committing to SaveAs, bound Ctrl-W and reachable as `:enter-mode save-prompt`) and stopped short of making ZZ route into it.\n\nTODAY, after PR #27: ZZ on a pathless document refuses, stays in the app, and the status line names the prompt. That is already better than improvise-8tk left it -- the user is told where to go. The open question is whether being told is enough, or whether the vim reflex should just land somewhere useful.\n\nTHE OBSTACLE THE ORIGINAL TICKET NAMED, still true: CmdContext does not expose file_path, so a Cmd cannot currently decide \"is there a path\" at execute time. Adding the field is small; note improvise-2ud is separately proposing that CmdContext borrow the ModelState/ViewState slices rather than copying ~20 individual fields, which would make this free. Coordinate rather than adding a 21st copied field ahead of it.\n\nTHE CONSTRAINT PR #27 ESTABLISHED, and it must be respected by whatever lands here: the prompt is SESSION-LOCAL and was deliberately kept in the COMMAND layer rather than routed through effect::Save. Save is becoming a wire command (improvise-368h), and a prompt is addressed to precisely one session -- routing it through the effect would open a prompt on a host nobody asked, which is gotcha 38's argument for keeping AcceptWizardModel off the wire. So: a Cmd that inspects the path and emits a mode change, never an effect that decides to prompt.\n\nDECIDE ALSO what ZZ means once the prompt is open. ZZ is \"save and quit\"; if the prompt commits to SaveAs and the batch was aborted, the quit is gone. Should committing the path then quit? That is a second composition and the answer is not obviously yes -- a user who hit ZZ, got a prompt, and typed a path plausibly wants to leave, but making the prompt's Enter mode-dependent is the kind of hidden state improvise-ct5 just declined to add for `i`.\n","status":"open","priority":3,"issue_type":"decision","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:32Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:32Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-g9jb","title":"The text-entry mode list is hand-written in two crates and checked against AppMode in neither","description":"Found by WU-G on PR #27 while adding AppMode::SavePrompt.\n\nThe set of text-entry modes is written down BY HAND IN TWO PLACES, in two crates:\n * improvise-command::keymap::tests::text_entry_modes_with_exits\n * improvise-wasm-client::keys::tests::text_entry_modes\n\nand NOTHING CHECKS EITHER AGAINST AppMode. So adding a text-entry variant compiles, passes, and is silently absent from both lists -- which matters more since PR #27, because the browser now refuses to ENTER a prompt it cannot commit (keys.rs::strands_the_user), and that refusal is driven off one of these hand-written lists.\n\nTHE FAILURE MODE IS THE QUIET ONE: a new prompt mode omitted from the wasm list is one a browser will happily open and then swallow the user's typing in -- doctrine(a_doorway_to_a_black_hole_is_worse_than_the_key), which PR #27 established against SavePrompt and found ExportPrompt had already.\n\nTHE FIX: derive the list once, from AppMode itself. `MinibufferConfig` is already the thing that makes a variant text-entry -- AppMode::minibuffer() is an exhaustive match returning Option\u003cMinibufferConfig\u003e (gotcha 11), so \"the text-entry modes\" is derivable rather than enumerable. A const or a function in improvise-protocol beside AppMode, with both crates reading it, replaces both lists.\n\nCHECK ONE THING BEFORE ASSUMING THE DERIVATION IS FREE: the wasm list and the command list may not be the same set for a reason -- SearchMode is hand-written on purpose and has no commit key (gotcha 11), and the browser's question (\"can this client commit here\") is not identical to the keymap's (\"does this mode take text\"). If they genuinely differ, the answer is two derived predicates over one enum, not two hand-written lists.\n\ndoctrine(a_hand_enumerated_mirror_needs_an_exhaustive_match) already covers this; this is an instance with no match behind it at all.\n","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-z05r","title":"CommitExport does not bail on an empty path, and ExportCsv does not abort the batch","description":"Found by WU-G on PR #27 while fixing the save arms, and left deliberately: these are the same two defects one file over, at lower severity, and fixing them in that PR would have widened a save-focused change into an export-focused one.\n\nTWO ARMS, both mirroring what was just fixed for save:\n 1. CommitExport does not bail on an empty path, where CommitBuffer's three kinds all do (gotcha 20: buffer -\u003e trim -\u003e bail-on-empty -\u003e dirty -\u003e status is the shared spine).\n 2. ExportCsv does not set abort_effects on failure, where Save and (as of PR #27) SaveAs both do. So a `Binding::Sequence` or a `:` line composing an export with anything after it continues past a failed write.\n\nWHY IT IS LOWER SEVERITY THAN THE SAVE ARMS, and this is the honest reason it was deferred rather than a claim that it does not matter: an export failure discards nothing. improvise-8tk was a P1 because ZZ quit and the model was gone. Here the model is untouched and the user has simply not got the CSV they asked for -- so the cost is a missing file and a message, not lost work.\n\nNOTE THE SHAPE, because it is the third instance in this area: improvise-8tk fixed Save and not SaveAs; PR #27 found SaveAs; and this is the export arm of the same rule. doctrine(one_arm_of_a_symmetric_rule_is_evidence_about_that_arm_only) exists for exactly this, and the general fix may be better than the third patch -- consider whether \"an effect that writes a file and fails aborts the batch\" can be made structural rather than remembered three times.\n\nALSO CHECK, while in there: whether the export error is visible at all, given that ExportPrompt is a minibuffer mode and a status message written while a minibuffer is open is never drawn (see the sibling bead on draw_bottom_bar).\n","status":"open","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:21Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:21Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -172,7 +174,7 @@ {"_type":"issue","id":"improvise-x7sw","title":"Which-key hints have no protocol home (the open half of improvise-jb3)","description":"improvise-jb3 closed with its WIDGET half already done -- WhichKeyWidget::new(\u0026[(String, \u0026static str)]) was never \u0026App-coupled. This is the half that was never that ticket.\n\nThe derivation happens in draw.rs: App::armed_prefix_keymap() -\u003e Keymap::binding_hints(). ViewState::transient_prefix IS portable (gotcha 13 stores a prefix as an ADDRESS, not a keymap, precisely so ViewState stays free of improvise-command types) -- but THE KEYMAP SET IS NOT ON THE CLIENT, so a browser which-key cannot be derived from ViewState alone.\n\nTWO ANSWERS, and the choice is the ticket:\n * THE RESOLVED HINT LIST CROSSES -- a Projection, or a ViewState field. Cheap, and it keeps the keymap a client-only concept, which is the stated direction (\"keys should be converted into commands in the browser: the keymap should not be known server side at all\"). But it puts a UI string list on the wire, which is what improvise-k099 is already unhappy about for AppMode::hint_text/label.\n * THE CLIENT GETS THE KEYMAPS. Consistent with keys being a client concept, and the wasm client already has keys.rs -- but the hint list is currently derived from the SERVER-side KeymapSet, so this means the client owns the keymaps outright and the server stops having them. Larger, and probably where this is heading anyway.\n\nNote the second reading is the one that fits improvise-bpq2. Decide them together.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:33Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:33Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-2yc2","title":"DataStore::live_records walks a HashMap -- decide its order before writing the caller","description":"The next instance of improvise-shyt, found while fixing it, and deliberately LEFT ALONE rather than guessed at.\n\nDataStore::live_records walks the records: HashMap, so it yields a different order in every process -- the same defect that made records-mode rows reorder between sessions, misdirected drill staged edits, and drifted SUM/AVG in the last f64 bits.\n\nIt differs from shyt in one way that matters: IT HAS NO CALLERS YET. So this is not a bug report, it is \"decide the order deliberately before something depends on it\". The ADR 002 surrogate path will be the caller -- the RecordHash -\u003e RecordEntry index is what live_records walks -- and whoever writes it should pick an order on purpose rather than inherit a hash walk and then discover it, which is how shyt happened.\n\nTHE ANSWER SHYT LANDED ON, for reference: the index narrows, the primary IndexMap orders (matching_positions resolves each candidate with get_index_of, sorts, reads back with get_index). Measured at 18,000 cells that cost at worst +20% on one path and was 7% FASTER on the other. Whether that is the right answer here depends on what the surrogate path wants; a hash-keyed index may legitimately want no order at all, in which case say so and return something whose type admits it.","status":"closed","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:43Z","started_at":"2026-09-20T05:32:37Z","closed_at":"2026-09-20T05:54:43Z","close_reason":"Closed by PR #20 (WU-K). DECIDED: live_records answers in the STORE'S OWN ORDER -- it walks the primary IndexMap's cells and dedupes by RecordHash, so a record sits where its FIRST cell sits. That is the order matching_positions already gives (gotcha 18), projected onto records, rather than a second answer to \"what order is this store in\".\n\nDerived on each call rather than memoised on RecordEntry, because a stored position is a copy and the tombstone a deleted record leaves would keep it while its cells moved to the end. That case is now tested (a_recreated_record_takes_its_new_place_not_its_old_one) -- and it was found because Copilot noticed the doc comment ASSERTED the derive-don't-memoise choice mattered for delete-and-recreate while nothing tested it; a memoising implementation passed all three other tests.\n\nTHE TICKET'S ALTERNATIVE WAS REJECTED WITH A REASON WORTH KEEPING: \"a hash-keyed index may want no order at all -- return something whose type admits it\" cannot be built. Every iterator has an order, and a type that merely DISCLAIMS one still hands the caller hash order. Disclaiming is not making unrepresentable.\n\nCost moves O(records) -\u003e O(cells). With no caller to benchmark, correctness is the defensible default and the doc says what a hot-path caller should bring.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-4zl7","title":"Teach the emitter a categorical split (remedy 3 for improvise-8xhy)","description":"Successor to improvise-8xhy, which shipped remedy 1 (refuse at emit) as an explicit interim. This is the remedy that produces a formula A HUMAN WOULD HAVE WRITTEN, and it is the right end state -- but it was costed during 8xhy and it is much larger than it looks.\n\nANSWER THIS FIRST, BEFORE TOUCHING THE GRAMMAR. A fitted measure cell is evaluated at a CellKey context. In any view that aggregates over the split category -- Region on Page or None -- context.get(\"Region\") is None, and today Formula::filter falls straight through to aggregate_raw. So a categorical fit would SILENTLY DEGRADE to something else in exactly the views a pivot-table user spends most of their time in. That is a product decision about what a fit MEANS for an aggregated cell, not an emitter detail.\n\nSCOPE, once that is answered:\n * Expr::Is(Filter) evaluating to 1.0/0.0. This keeps the f64-only evaluator and avoids introducing string comparison into eval_expr. The primitive already exists in the right shape -- types.rs reads context.get(\u0026filter.category), the current cell own coordinate.\n * A grammar production plus tokenizer handling in the 1109-line parser.rs, with quoting rules for the category and item inside it (gotcha 34 -- one home, improvise_formula::quote).\n * Arms in BOTH evaluators: eval_expr and eval_expr_cached.\n * Coverage in the grammar-walking generator.\n * Thread Dataset::columns (the ColumnSpec/ColumnKind plan) into the emitter, so feature index -\u003e role is read off preprocess own plan. Do NOT reconstruct it: Spec::names is \u0026[\u0026str], and reconstructing index-\u003erole is a second implementation of preprocess expansion order, exactly the drift risk gotcha 41 names in another key.\n * .improv is free -- the writer emits Formula::raw.\n\nIT CLOSES ONLY A THIRD OF 8xhy EITHER WAY. A WHERE predicate covers the one-hots. col.missing means \"this record has no value for measure X\", a different vocabulary. The date sub-features (when.year, when.month) need date arithmetic improvise does not have at all. Those stay refused, or get their own tickets.\n\nPAIRS WITH improvise-vx3 (the fit-measure command), which is where a user would actually learn about the restriction.","status":"in_progress","priority":3,"issue_type":"feature","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:45:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:41Z","started_at":"2026-09-20T05:32:41Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-odbc","title":"[bug] bank-info.improv in the repo root does not parse","description":"Incidental finding from improvise-cr3 (2026-09-19), outside that ticket scope, unreproduced beyond the one load.\n\n Parse error at line 293, column 3: - JORDAN MARTINEZ (expected pipe_quoted)\n\nTWO READINGS, and they want different actions:\n * The file is STALE -- written before a grammar change -- in which case delete it or regenerate it, and the finding is that a sample file in the repo root is not exercised by anything.\n * The GRAMMAR HAS A GAP -- some name shape a writer emits that the reader rejects. That is the serious reading, because save/load round-trip is a property test (persistence proptests, parse(format(parse(generate())))) and a gap that survives it is a gap the GENERATOR does not reach.\n\nDecide which by checking whether improvise can WRITE that line today. If it can, the round-trip property is incomplete and the generator is the thing to fix, not the file.","status":"in_progress","priority":3,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:40Z","started_at":"2026-09-20T05:32:40Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-odbc","title":"[bug] bank-info.improv in the repo root does not parse","description":"Incidental finding from improvise-cr3 (2026-09-19), outside that ticket scope, unreproduced beyond the one load.\n\n Parse error at line 293, column 3: - JORDAN MARTINEZ (expected pipe_quoted)\n\nTWO READINGS, and they want different actions:\n * The file is STALE -- written before a grammar change -- in which case delete it or regenerate it, and the finding is that a sample file in the repo root is not exercised by anything.\n * The GRAMMAR HAS A GAP -- some name shape a writer emits that the reader rejects. That is the serious reading, because save/load round-trip is a property test (persistence proptests, parse(format(parse(generate())))) and a gap that survives it is a gap the GENERATOR does not reach.\n\nDecide which by checking whether improvise can WRITE that line today. If it can, the round-trip property is incomplete and the generator is the thing to fix, not the file.","status":"closed","priority":3,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:37:30Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:28:58Z","started_at":"2026-09-20T05:32:40Z","closed_at":"2026-09-20T06:28:58Z","close_reason":"Closed by PR #22 (WU-L). THE TICKET'S PREMISE WAS FALSE, AND TESTING IT RATHER THAN ASSUMING EITHER READING IS WHAT MADE THIS CHEAP.\n\nThe ticket offered two readings and hoped for the serious one: that the grammar had a gap some writer could emit, which would mean the round-trip property is incomplete and the GENERATOR is the thing to fix. It said to decide by checking whether improvise can WRITE the offending line. IT CANNOT. item_list's name is `pipe_quoted | bare_name`, bare_name is [A-Za-z_][A-Za-z0-9_-]* with no space, and quote_name mirrors it character for character -- so improvise writes |JORDAN MARTINEZ|. Reader and writer agree, the round-trip property is COMPLETE, and the grammar-walking generator needs no change.\n\nThe file was stale BY HAND: c48a5cd migrated it to v2025-04-09 by editing in place, pipe-quoting the Date items and not the Entity ones, and every line of its ## Data section names a category `Measure` where the file declares `_Measure`. Unloadable since April. Nothing loaded it.\n\nDELETED RATHER THAN REGENERATED, on the user's ruling 2026-09-19, together with foo.improv (the same class, one file over, with no version line at all). There is no generator for either -- they are hand-written fake data -- so regenerating meant inventing a canonical form and checking in 1200 lines whose only property is that they parse, which the grammar proptests already establish over a far wider input space.\n\nTHE EVIDENCE KEPT IS WORTH MORE THAN THE FILE:\n * an_item_name_the_item_line_cannot_carry_round_trips -- the deleted file's own names through format_md/parse_md.\n * a_bare_item_name_with_a_space_is_rejected -- the reader really refuses the unquoted line, with the quoted form parsing as a precondition. Without this the first test is consistent with BOTH spellings being accepted.\n * tests/shipped_documents_load.rs -- a DIRECTORY WALK over the two directories .gitignore un-ignores, not a list of names, because a list is a second place to remember and the forgotten file is the one nobody added to it. Fault injected and failed. Then HARDENED: it matched Path::extension() == \"improv\", which of x.improv.gz is \"gz\", so a gzipped shipped document would have been skipped -- the very hole the test exists to close, put back inside the fix for it.\n\nTHE FINDING, which is the opposite of what the ticket expected: a checked-in document in a format the repo parses is exercised by NOTHING unless something loads it, and grammar proptests cannot see it because they GENERATE their inputs.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-gd28","title":"Drill over the wire needs the addressed command form","description":"From the ADR 002 spine landing (2026-09-19). The RESOLVER exists and is tested -- App::resolve_aggregate, layout-only by necessity since an aggregate has no existence outside the layout that derives it -- and CellId::Aggregate is emitted. What is missing is the command.\n\nModelCommand::Drill was deliberately NOT added. Drilling creates the _Drill view and writes drill_state, which gotchas 30 and 38 rule session-local and therefore unaddressable by a NULLARY Command -- exactly the reason ApplyAndClearDrill was removed from the wire in improvise-3x3u.\n\nSo a wire Drill needs the ADDRESSED form: the variant carries a SessionId and the staging map moves off the single App onto the session. That is ADR 001 layer 2 (context/adr/001-concurrent-edits.md), so this is blocked on it rather than being independent work.\n\nThe seam is ready: emit, resolve and the type-level guarantee (Drill takes an Aggregate and nothing else) are all in place and tested.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:18:34Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:34Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-qddo","title":"Make DataStore an implementation of a source trait, not the implementation","description":"REWRITTEN 2026-09-19 to match context/adr/003-live-backed-models.md, and DEFERRED by user ruling in the same session, together with improvise-rucf and improvise-mjy7. The ADR calls this \"worth doing first regardless of the eventual shape\"; the user's ruling is that the whole direction waits, so it waits as a unit rather than landing a seam with nothing on the other side of it.\n\nTHE WORK IS UNCHANGED: introduce a source trait in improvise-core and make the current in-RAM DataStore AN implementation of it rather than THE implementation. No behaviour change -- doctrine(characterization_survives_the_move) applies and the existing tests must carry across unchanged in meaning.\n\nTODAY Model holds a DataStore directly and a Workbook loads ENTIRE; there is no partial-load path anywhere in the tree (verified while declining improvise-9ix). So \"where cell values come from\" is not a seam, it is an assumption spread across GridLayout, evaluation and persistence.\n\nWHAT ADR 003 ADDS TO THE ORIGINAL FRAMING, and it is the part worth re-reading before starting:\n\n * AGGREGATION IS THE INTERFACE, AND IT IS ALREADY SQL. apply_agg computes Sum/Avg/Min/Max/Count over a matching cell set -- exactly SQL's aggregates -- and GridLayout asks for a VALUE, not for the rows behind it. Shape the trait around \"give me the aggregate for this coordinate set\", never around \"give me the cells\": shaping it around cells forecloses pushdown, which is the entire point of the eventual live source.\n\n * WRITES MUST NOT BE IN THE TRAIT. This is new since the original text and it is a real constraint. If writing were in the trait a query source implements, a read-only source would have to FAIL AT RUNTIME for a method it structurally cannot serve. Keep reads and aggregates in the trait every source implements; let writing be a capability only the owned in-RAM implementation has. Read-only then becomes UNREPRESENTABLE rather than enforced -- the same discipline as CellId (ADR 002) and doctrine(an_invariant_is_made_unrepresentable_not_validated).\n\n * FORMULAS DO NOT MOVE. Evaluation is fixed-point and derived values live in Model::formula_cache, not in DataStore, so the split is clean: raw data is queried, derived data stays in RAM.\n\nCONSTRAINT: forbidden(improvise_core, io). The TRAIT belongs in improvise-core; any IO-touching implementation belongs in improvise-io. Do not let the trait signature drag IO types into core; if it needs Result, that is a boundary decision worth arguing rather than defaulting (AGENTS.md keeps Option/Result at the edges).\n\nWHY IT IS STILL WORTH DOING ON ITS OWN WHEN IT IS UN-DEFERRED: it is the seam every later option needs -- live-backing, a test double, or the import-once path ADR 003 rejected as the ONLY answer but did not forbid as A path; it makes \"the model is fully in RAM\" an explicit property of one implementation rather than an invisible assumption of the whole crate; and done as a pure refactor it is cheap and reviewable, where doing it AS PART OF adding a SQLite source mixes a large refactor with a large feature and makes both unreviewable.\n","notes":"\n--- READ-ONLY DECIDED (user, 2026-09-19), and it shapes the trait ---\nA live-backed model is READ-ONLY, and not reluctantly: for many use cases\nediting capabilities would be a DOWNSIDE. Pointing a pivot tool at a production\ndatabase and being unable to write to it is a feature.\n\nSO DO NOT PUT WRITES IN THE TRAIT. If writing were a trait method, a read-only\nsource would have to fail at RUNTIME for something it structurally cannot\nserve. Keep reads -- and crucially AGGREGATES, see the note above about shaping\nit around \"give me the aggregate for this coordinate set\" -- in the trait every\nsource implements, and let writing be a capability only the owned in-RAM\nimplementation has. Then read-only is UNREPRESENTABLE rather than enforced,\nwhich is the same discipline ADR 002 CellId uses and the same design-principles\n3 rule.\n\nRELATED CONVERGENCE WORTH KNOWING: ADR 002 already makes SetCell take a\nCellId::Record and nothing else, so a live-backed model that projects no Record\nids makes a client-side write unconstructible with no runtime check and no\n\"this model is read-only\" dialog. The settability distinction built for\naggregates carries read-only sources for free -- so the client half of this may\nalready be done.","status":"deferred","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:16:29Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-rucf","title":"Read arbitrary SQLite databases as a data source","description":"SUPERSEDED BY context/adr/003-live-backed-models.md, and rewritten 2026-09-19 to match it. The original text framed this as an open choice between import-once and live-backed, and said \"READ-ONLY for now\" as though that were a scoping convenience. ADR 003 decided both, and in the opposite direction from what this ticket assumed.\n\nWHAT THE ADR DECIDED, and this ticket now means:\n\n * LIVE-BACKED, not import-once. Import-once was explicitly REJECTED as the only answer -- it makes the model a snapshot, so source changes are invisible until re-import and the data must fit in memory. The model is a view over the source and queries go to it as the user pivots.\n * READ-ONLY IS A FEATURE, not a limitation accepted reluctantly. Pointing a pivot tool at a production database and being unable to write to it is the point. No write-through, no overlay, no reconciliation.\n * THEREFORE: Model becomes a FACADE OVER A DATA SOURCE, of which today's in-RAM DataStore is one implementation and a SQL query engine is another. That is a change to what Model IS, not to where its bytes come from.\n\nSTATUS: NOT A PRIORITY, per the ADR's own header. Deferred 2026-09-19 during the whole-backlog campaign, by user ruling, together with improvise-qddo and improvise-mjy7. Nothing here outranks the browser work. It is written down so the option stays open, because several decisions made recently could foreclose it by accident.\n\nTHE THREE THINGS THAT MAKE IT TRACTABLE, from the ADR, which should shape the design rather than be discovered late:\n 1. Aggregation is ALREADY the interface and it is ALREADY SQL. apply_agg computes Sum/Avg/Min/Max/Count over a matching cell set; those are exactly SQL's aggregates. GridLayout asks for a VALUE, not for the rows behind it. The pushdown is not a translation layer to be invented.\n 2. Formula results are ALREADY separate from stored data. Evaluation is fixed-point and derived values live in Model::formula_cache, not in DataStore. Raw data is queried; derived data stays in RAM. Formulas do not become SQL.\n 3. Clients are ALREADY insulated. A client sees CellDisplays and never learns where a value came from, so live-backing and the browser deployment compose rather than compete. A browser never needs SQLite at all.\n\nTHE CONSEQUENCE NOBODY ASKED FOR, and the reason this is three tickets rather than one: a live-backed document splits into OUR configuration and THEIR data, and .improv does not distinguish them. The format grows a SOURCE SPECIFICATION where a live model would otherwise have a ## Data section, and categories become derived (SELECT DISTINCT) rather than declared. improv.pest is the single source of truth, so the grammar comes first.\n\nCHILDREN (all filed and deferred alongside this one):\n * the .improv source specification and the grammar change\n * the SQLite source implementation, native-only and feature-gated\n * projection freshness for a source an external writer can change\n\nWHAT ADR 001 DOES AND DOES NOT OWE THIS: a read-only model has no concurrent EDITS, so the basis check simply does not apply -- the guarantee is scoped to documents improvise owns, which is what it always meant. What remains is a different and smaller question, which is the freshness child.\n\nTHE MACHINERY TO EXPRESS READ-ONLY ALREADY EXISTS, built for something else: ADR 002 makes SetCell take a CellId::Record and nothing else, so a source that projects no Record ids makes a write UNCONSTRUCTIBLE by the client -- no runtime check, no error path, no read-only dialog. One design consequence for the source trait follows: writes must NOT be in the trait a query source implements, or a read-only source has to fail at runtime for a method it structurally cannot serve.\n\nTHE DEPENDENCY CONSTRAINT THAT CARRIES OVER FROM improvise-9ix: improvise-io is in the wasm bundle (the wasm job in .github/workflows/ci.yml gates it) and libsqlite3-sys does not build for wasm32-unknown-unknown under this repo's nix cc-wrapper. Native-only and feature-gated, with the wasm job passing untouched. Verify the gate rather than assuming it.\n\nWHAT THIS IS NOT: improvise-9ix, which proposed SQLite as improvise's OWN PERSISTENCE FORMAT and was declined on measurements (22% larger than .improv, 4.7x gzipped, a second implementation of the pest grammar). Reading someone else's database has no round-trip obligation and no grammar duplication.\n","notes":"\n--- SUPERSEDED IN FRAMING by ADR 003 (user, 2026-09-19) ---\nThe user chose LIVE-BACKED over import-once, and the goal is stated as: have\nTHE OPTION of essentially using improvise as a DB CLIENT. NOT A PRIORITY --\nnothing here outranks improvise-1ey. Re-prioritised P2 -\u003e P3.\n\nSo this ticket read-only import framing is not the target shape. See\ncontext/adr/003-live-backed-models.md. The import-once seam it described\n(csv_parser -\u003e Vec\u003cValue\u003e -\u003e analyze_records) is still accurate and still\nuseful, but as ONE implementation of a source rather than as the answer.\n\nWHAT SURVIVES UNCHANGED and is worth keeping from this ticket: the type-mapping\nnotes (SQLite columns carry declared types, which is BETTER input than CSV\nstring sampling -- detect_date_format currently guesses a date by pattern-\nmatching text; dates may be TEXT, INTEGER epoch or REAL julian, so detect\nrather than assume; NULL is ABSENCE not the string \"null\", per improvise-ych),\nand the wasm constraint (improvise-io is in the bundle, libsqlite3-sys does not\nbuild for wasm32 under this repo nix cc-wrapper, so any SQLite dependency is\nfeature-gated and native-only).","status":"deferred","priority":3,"issue_type":"epic","owner":"el-github@elangley.org","created_at":"2026-09-20T00:14:11Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:18Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -235,7 +237,7 @@ {"_type":"issue","id":"improvise-aaw","title":"Replace CreateAndSwitchView with Binding::Sequence","description":"command/cmd/panel.rs:546-560 bundles create-view + switch-view + change-mode. Both primitives already exist via effect_cmds.\\n\\nFix: bind Sequence [(create-view nix-shell-env), (switch-view nix-shell-env), (enter-mode normal)] in the keymap. If the auto-generated view name is load-bearing, either thread through a minibuffer (matches other add flows) or keep a single create-view primitive that both creates and switches by default.\\n\\n~12 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nGROOMED 2026-09-19: still true; CreateAndSwitchView is cmd/panel.rs:658, not 546-560.\n\n--- INVESTIGATED 2026-09-19, NOT DONE: premise false ---\nCreateAndSwitchView is NOT a shim. It computes the new view's name from\ncontext -- format!(\"View {}\", ctx.workbook.views.len() + 1) -- and a static\nBinding::Sequence can only carry literal args, so [create-view X, switch-view X,\nenter-mode normal] would hard-code one name and produce duplicates on the\nsecond 'n'. The auto-generated name is load-bearing, so the command has a real\ndecision and is not replaceable by composition as written.\n\nThe ticket's own two fallbacks are both BEHAVIOUR changes needing a product\ndecision, not refactors:\n (a) thread the name through a new minibuffer mode -- that is a feature, and\n it changes what 'n' does in the view panel;\n (b) make the create-view primitive also switch -- that changes :create-view\n for scripts and duplicates what grid.rs already does by hand for the\n _Drill/_Records views.\nNeither was forced. RE-SCOPE this ticket to 'should creating a view from the\npanel prompt for a name?' or close it won't-fix.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:54Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:31Z","started_at":"2026-09-20T05:32:31Z","dependencies":[{"issue_id":"improvise-aaw","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:23Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-git","title":"Replace SaveAndQuit with Binding::Sequence in keymap","description":"command/cmd/mode.rs:232-241 is an 8-line command chaining Save + ChangeMode(Quit). Both primitives already exist.\\n\\nFix: delete the struct; bind via Sequence [(w, []), (force-quit, [])] in the keymap.\\n\\nTrivial application of the 'if the keybinding is A then B, the binding should be a Sequence, not a command named AAndB' principle.\\n\\n~8 LOC.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nUNBLOCKED (groomed 2026-08-30): improvise-8tk is closed, and its fix removes this ticket's stated objection rather than merely satisfying it.\n\nThe blocker was recorded as: 'a Sequence has no way to stop between steps, so converting first would cement the ZZ/:wq data-loss bug into the keymap'. That is no longer true. Keymap::dispatch flattens a Binding::Sequence into ONE effect vec applied by a single App::apply_effects call, and the Save effect now sets abort_effects when the save fails — so the abort carries across sequence steps exactly as it does within a command's own effect list. A [w, force-quit] Sequence would stop after a failed save.\n\nVerified by ui/effect.rs::tests::save_without_file_path_aborts_the_chain and failed_save_to_path_aborts_the_chain, plus ui/app.rs::tests::prefix_z_then_z_reaches_save_and_quit, which drives the real ZZ path.\n\nSo the conversion is now safe. Still check improvise-lae first: it may change what ZZ does when there is no file path, which would change what the Sequence should contain.\nGROOMED 2026-09-19: the \"still check improvise-lae first\" sentence in this ticket is a real ordering constraint, now recorded as a dependency edge rather than prose. Stale ref: SaveAndQuit is cmd/mode.rs:243 (cited 232-241).","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:48Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:31Z","started_at":"2026-09-20T05:32:31Z","dependencies":[{"issue_id":"improvise-git","depends_on_id":"improvise-8tk","type":"blocks","created_at":"2026-08-13T18:08:05Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-lae","type":"blocks","created_at":"2026-09-19T12:47:58Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-git","depends_on_id":"improvise-s13","type":"related","created_at":"2026-08-13T18:34:24Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-s13","title":"Delete mode-entry shim commands; use enter-mode with arg","description":"command/cmd/mode.rs:311-319 has EnterExportPrompt and similar shims (EnterSearchMode, EnterTileSelect, EnterFormulaEdit) that are each ~9-line wrappers around change_mode(AppMode::foo()). The generic EnterMode already exists with parse_mode_name() in the registry.\\n\\nFix: delete the shims; bind 'enter-mode' with the mode name as an arg in the keymap.\\n\\nCorrectness win: one code path for mode transitions. A new invariant on mode enter (e.g., 'always rebuild layout') needs to be added in one place instead of N shims.\\n\\n~40 LOC across all shims.","notes":"Groomed 2026-08-13 — considered merging improvise-git, improvise-aaw and improvise-s13 into one 'collapse composite commands into keymap composition' ticket and DECIDED AGAINST IT. They are the same principle (do not name a command AAndB; do not wrap a generic command in a shim) and they touch the same file cluster, so batching them in one session is worthwhile — hence the 'related' links. But merging the tickets would have taken improvise-git's new dependency on improvise-8tk (the ZZ/:wq silent-data-loss bug) and applied it to the whole group, blocking two pieces of work that are ready today behind a P1 bug they have nothing to do with. Keeping them separate keeps s13 and aaw actionable.\n\nIf batching them, do improvise-s13 first (biggest and cleanest, ~40 LOC across four shims), then improvise-aaw (has an unresolved design question about whether the auto-generated view name is load-bearing), and improvise-git last, only after improvise-8tk. Verify all of them in one pass with the (mode, key, modifier) binding-table dump described in improvise-4pg's notes.\nGROOMED 2026-09-19: still true; the shims have moved apart — EnterExportPrompt cmd/mode.rs:326, EnterSearchMode :338, EnterTileSelect :353, and EnterFormulaEdit is in cmd/panel.rs:465, not mode.rs.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:53:44Z","created_by":"Ed L","updated_at":"2026-09-19T20:35:53Z","closed_at":"2026-09-19T20:35:53Z","close_reason":"Deleted the two genuine shims: EnterExportPrompt and EnterFormulaEdit, rebinding Ctrl-e to 'enter-mode export-prompt' and formula-panel a/n/o to 'enter-mode formula-edit'. Premise was half false -- the ticket named four. EnterSearchMode changes no mode at all (SearchMode is Normal + a flag, not an AppMode) and EnterTileSelect guards on category count and seeds SetTileCatIdx(0); both are real commands and were left alone. Side effect, a fix not a regression: EnterMode clears the target mode's minibuffer, which the shims did not, though EnterFormulaEdit's doc-comment already claimed it did.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-7yx","title":"Close test gaps flagged by deep review","description":"Deep review identified several test-coverage gaps worth closing:\\n\\n1. Keymap fallback chain: command/keymap.rs has 22 tests but no case for 'transient keymap exists but key does not match' (app.rs:396-406).\\n2. Ambiguous date detection: import/analyzer.rs:49-77 first-match heuristic not tested against ['01/02/2025', '02/03/2025'] style ambiguity.\\n3. CSV adversarial inputs: no tests for BOM, CRLF, empty file, single-column file.\\n4. Records missing a category field mid-import: wizard.rs:160-229 has the valid=false/break path but it is not covered by build_model_cells_match_source_data or similar.\\n5. Gzip corruption error path: save_and_load_roundtrip_gzip tests happy path only; add a test that load() on a truncated .gz file surfaces an error.\\n\\nLow-effort, high-value additions. Batch into a single test-coverage PR.","notes":"GROOMED 2026-09-19 — two of the five items are now covered; three remain. (1) Keymap fallback COVERED: ui/app.rs::tests::unmatched_key_after_prefix_is_swallowed, added by improvise-jc8 when the transient keymap became a prefix address. (4) Records missing a category mid-import COVERED: wizard.rs::build_model_record_with_missing_category_value_skipped and build_model_warns_about_records_skipped_for_missing_category. STILL OPEN: (2) ambiguous date detection — analyzer.rs has detect_iso_date_format / detect_us_date_format / detect_no_date_format and no case where two readings both parse; (3) CSV adversarial — rfc4180 quoting and short rows ARE tested; BOM, CRLF, empty file and single-column file are not; (5) gzip corruption — only gzip_round_trips_through_memory_storage and save_and_load_roundtrip_gzip, both happy-path. Also all paths in the description predate the crate split: command/keymap.rs -\u003e crates/improvise-command/src/command/keymap.rs, import/* and persistence/* -\u003e crates/improvise-io/src/.","status":"in_progress","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:19:07Z","created_by":"Ed L","updated_at":"2026-09-20T05:32:38Z","started_at":"2026-09-20T05:32:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-7yx","title":"Close test gaps flagged by deep review","description":"Deep review identified several test-coverage gaps worth closing:\\n\\n1. Keymap fallback chain: command/keymap.rs has 22 tests but no case for 'transient keymap exists but key does not match' (app.rs:396-406).\\n2. Ambiguous date detection: import/analyzer.rs:49-77 first-match heuristic not tested against ['01/02/2025', '02/03/2025'] style ambiguity.\\n3. CSV adversarial inputs: no tests for BOM, CRLF, empty file, single-column file.\\n4. Records missing a category field mid-import: wizard.rs:160-229 has the valid=false/break path but it is not covered by build_model_cells_match_source_data or similar.\\n5. Gzip corruption error path: save_and_load_roundtrip_gzip tests happy path only; add a test that load() on a truncated .gz file surfaces an error.\\n\\nLow-effort, high-value additions. Batch into a single test-coverage PR.","notes":"GROOMED 2026-09-19 — two of the five items are now covered; three remain. (1) Keymap fallback COVERED: ui/app.rs::tests::unmatched_key_after_prefix_is_swallowed, added by improvise-jc8 when the transient keymap became a prefix address. (4) Records missing a category mid-import COVERED: wizard.rs::build_model_record_with_missing_category_value_skipped and build_model_warns_about_records_skipped_for_missing_category. STILL OPEN: (2) ambiguous date detection — analyzer.rs has detect_iso_date_format / detect_us_date_format / detect_no_date_format and no case where two readings both parse; (3) CSV adversarial — rfc4180 quoting and short rows ARE tested; BOM, CRLF, empty file and single-column file are not; (5) gzip corruption — only gzip_round_trips_through_memory_storage and save_and_load_roundtrip_gzip, both happy-path. Also all paths in the description predate the crate split: command/keymap.rs -\u003e crates/improvise-command/src/command/keymap.rs, import/* and persistence/* -\u003e crates/improvise-io/src/.","status":"closed","priority":3,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-04-16T18:19:07Z","created_by":"Ed L","updated_at":"2026-09-20T06:28:59Z","started_at":"2026-09-20T05:32:38Z","closed_at":"2026-09-20T06:28:59Z","close_reason":"Closed by PR #22 (WU-L). Items 2, 3 and 5 done; items 1 and 4 were already closed and both tests were verified to exist rather than assumed.\n\nITEM 2 FOUND A LIVE BUG ONE LINE AWAY, and it is the most consequential change on the branch though no ticket names it. analyze_records collected each string column's distinct values into a HashSet AND ITERATED IT -- invariant(indexmap_order) / gotcha 18 in a second place. build_model adds items in that order, IndexMap keeps it, persistence relies on it: THE SAME CSV IMPORTED TWICE PRODUCED TWO DIFFERENT .improv FILES AND TWO DIFFERENT ROW ORDERS. A probe saw five different orders in five runs of one process. It reached the date heuristic too: detect_date_format tests only the first ten samples, so WHICH ten varied, and a column whose eleventh value rules out %m/%d/%Y detected a different format run to run -- every date in it off by the American/European difference. Fixed with a filtered insert: the set still deduplicates and is never iterated. First-seen order. The two determinism tests were watched failing on the parent commit.\n\nThe heuristic ITSELF is pinned, not changed, per the user's ruling -- see improvise-l8rh. an_ambiguous_date_is_read_american_first states in its doc-comment that this silently mis-reads a European CSV, with the day-first reading asserted to parse so the test is about ambiguity rather than parsing; a_day_first_only_value_settles_the_ambiguity is the discriminating arm.\n\nITEM 3, five CSV tests (BOM, CRLF, empty, header-only, single column). All four inputs already behaved correctly, SO EACH TEST CARRIES A CONTROL proving it can see its own fault -- a BOM in the SECOND header is not leading, is not stripped, and does appear in the key; a LONE \\r is itself a record terminator to this reader and splits one row into two.\n\nITEM 5, three gzip tests kept apart by cause: truncation -\u003e UnexpectedEof; CRC corruption -\u003e InvalidInput AND a cause naming \"checksum\"; a valid gzip of invalid UTF-8 -\u003e the UTF-8 diagnosis. One non-test line changed: the gzip branch of load_from used a bare ? where the UTF-8 branch two lines below carried with_context, so a corrupt .gz reported \"incomplete deflate stream\" and no filename.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-61f","title":"Add invariant test for command registry name mapping","description":"crates/improvise-command/src/command/cmd/registry.rs is 586 lines with 0 tests. Each registration manually duplicates the command name as a string alongside the Cmd::name() implementation on the struct. Nothing verifies they match. Fix: add a test that iterates all registered entries and asserts registry_name == command.name() after construction. Low effort, locks down a class of typo bugs.","notes":"Groomed 2026-06-09: complementary to improvise-9cn (register_cmd! macro). The macro prevents future drift at compile/registration time; this test catches drift that predates or escapes the macro. Doing 9cn first likely shrinks this to a trivial registry-walk test — sequence them together.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:55Z","created_by":"Ed L","updated_at":"2026-08-29T19:10:27Z","closed_at":"2026-08-29T19:10:27Z","close_reason":"Delivered as part of improvise-9cn. See that issue for why the invariant as specified here ('assert registry_name == command.name() for all entries') is necessary but NOT sufficient: a wrong-variant prototype collides with its sibling rather than producing a mismatch, so the roundtrip test passes while a command name vanishes from the registry. Shipped alongside registered_names_are_unique, which is the check that actually reaches that class, plus a keymap-side test that every bound command name resolves. All four verified by fault injection.","dependencies":[{"issue_id":"improvise-61f","depends_on_id":"improvise-9cn","type":"blocks","created_at":"2026-04-16T11:54:34Z","created_by":"Ed L","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-nr5","title":"Split persistence/mod.rs into focused sub-modules","description":"crates/improvise-io/crates/improvise-io/src/persistence/mod.rs is 2402 lines mixing four cohesive units: pipe quoting (lines ~20-82), format_md (~155-282), parse_md (~285-580), export_csv (~581-630). Split into persistence/{quoting,format,parse,export}.rs behind the existing pub facade; no public-API change. Improves navigability and makes adding new format features easier to review.","notes":"GROOMED 2026-09-19 — STALE REFS. Path was written twice: crates/improvise-io/crates/improvise-io/src/persistence/mod.rs -\u003e crates/improvise-io/src/persistence/mod.rs. Size is 2783 lines now, not 2402, and every line range in the description is stale. Current boundaries by symbol (prefer these; they survive edits): quoting = is_bare_name/escape_pipe/unescape_pipe/quote_name/pipe_quote (24-86) plus format_number/parse_number_cell; entry points save/load/save_to/load_from/autosave_path (128-178); format_md (197); parse_md (330); coord_str (630); export_csv/export_csv_to/render_csv (639-700). Tests start at line 706 (#[cfg(test)]) and are roughly three quarters of the file — the split should move each function with its tests.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:52Z","created_by":"Ed L","updated_at":"2026-09-19T20:36:35Z","closed_at":"2026-09-19T20:36:35Z","close_reason":"Split persistence/mod.rs (2783 lines) into six files: mod.rs (259, document lifecycle + facade + test_helpers), quoting.rs (63, the five pipe-quoting helpers as pub(super)), format.rs (350), parse.rs (1378, incl. the four existing test modules moved whole), export.rs (156), roundtrip_tests.rs (635). Two deviations from the ticket sketch, both deliberate: format_number/parse_number_cell went to their own direction's file rather than into quoting.rs, leaving quoting.rs a real shared leaf; and the 34 tests that call BOTH format_md and parse_md got their own file rather than being arbitrarily assigned. Pure relocation verified body-by-body: every function body identical modulo indentation, only signature-level pub(super)/pub changes; persistence test count 97 before and after; 815 workspace tests pass; wasm, clippy and fmt clean. Grammar untouched.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-t8s","title":"View::axis_of panics on missing category","description":"crates/improvise-core/src/view/types.rs:118 uses expect('axis_of called for category not registered with this view'). The invariant is that Workbook::add_category registers every category with every view, but the panic surface is fragile for new callers. Fix: either audit all callers and document the invariant clearly, or change signature to fn axis_of(\u0026self, cat: \u0026str) -\u003e Option\u003cAxis\u003e and let callers handle None.","status":"closed","priority":3,"issue_type":"task","assignee":"Edward Langley","owner":"el-github@elangley.org","created_at":"2026-04-16T18:18:48Z","created_by":"Ed L","updated_at":"2026-06-10T04:46:47Z","closed_at":"2026-06-10T04:46:47Z","close_reason":"Fixed (test-first where applicable): records_display returns None for OOB columns (f04fe51); try_axis_of added + axis_of invariant documented + cycle_axis no longer panics on unregistered categories (f04fe51); scrolled header suppression fixed via pure show_sublabel helper for both row and column headers (df9a02b); CyclePanelFocus cycles from current panel with wrap (9e02245); AGENTS.md merge target corrected (9634c95).","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -256,9 +258,9 @@ {"_type":"issue","id":"improvise-zx8t","title":"draw.rs::mode_style matches AppMode with a wildcard, so the repo map's exhaustiveness promise is false there","description":"Found by WU-G on PR #27 while adding AppMode::SavePrompt, and it is a correction to something context/repo-map.md currently promises.\n\nThe repo map says of AppMode: \"Per-variant UI facts live on the enum as exhaustive-match methods -- label(), hint_text(), panel(), minibuffer(), is_editing(), is_records() (improvise-2hi); ADD A VARIANT AND THE COMPILER NAMES EVERY ONE YOU MUST FILL IN.\"\n\nThat is TRUE of the five enum methods and FALSE of src/draw.rs::mode_style, which matches AppMode with a `_ =\u003e` WILDCARD. So a new variant silently inherits whatever the wildcard arm gives -- SavePrompt currently takes ExportPrompt's DarkGray by accident rather than by decision, which happens to be defensible and was nobody's choice.\n\nWHY IT IS P4 AND NOT HIGHER: the consequence is a colour, and the wrong colour is visible rather than silent. But the reason to fix it is that the repo map states a PROPERTY, and a property that holds in five places and not the sixth is worse than one nobody claimed -- the next person adding a variant will trust the sentence.\n\nTHE FIX IS ONE OF TWO, and they are not equivalent:\n * Make mode_style exhaustive, so the compiler does name it. Note mode_style stays in src/draw.rs deliberately, because Style is a ratatui type and improvise-protocol must not name one (gotcha 26, the AxisColor/MinibufColor pattern).\n * Or give AppMode a semantic colour method in the protocol crate -- the AxisColor treatment -- and let draw.rs do the one mapping. That is the established shape here and it would also give a browser something to use, which draw.rs's Style cannot.\n\nThe second is probably right and is the same question improvise-k099 asks about hint_text/label. Decide them together.\n","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T06:04:22Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:04:22Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-taxx","title":"[bug] The socket close listener overwrites a precise error with disconnected","description":"Found during the PR #13 review, reported and not fixed -- same family as the watchdog bug Copilot found, different question.\n\nweb/app.js socket close listener writes \"disconnected\" UNCONDITIONALLY, so a connection that failed loudly (the error listener already wrote \"cannot reach ... -- is improvise-ws-server running?\") is immediately followed by a close event that replaces it with the vaguer message. The user sees the least informative of the two things that happened.\n\nTHE WATCHDOG BUG WAS THE SAME SHAPE AND WAS FIXED: a timer fired 8s later and overwrote whatever a loud failure had already written; setShellFailure now clears it. This one was left because the question is genuinely different -- what should the page say when a connection BOTH failed AND went away? -- and because suppressing close unconditionally could hide a GENUINE MID-SESSION DISCONNECT, which is the one thing that listener is really for.\n\nLIKELY ANSWER: distinguish \"never connected\" from \"was connected and dropped\", which the socket readyState at close time can tell you. Not \"skip close if error fired\".\n\nUntestable here (no browser, no headless DOM), which is the standing caveat on every web/*.js finding -- see the browser harness ticket.","status":"closed","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T03:35:28Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:22:09Z","started_at":"2026-09-20T05:32:34Z","closed_at":"2026-09-20T06:22:09Z","close_reason":"Closed by PR #21 (WU-I). Fixed -- AND THE TICKET'S PROPOSED MECHANISM WAS WRONG, which is the part worth keeping.\n\nThe ticket said: \"distinguish 'never connected' from 'was connected and dropped', which the socket readyState at close time can tell you.\" IT CANNOT. readyState is already CLOSED by the time any close listener runs, whichever happened; and the CloseEvent cannot separate them either, because a refused connection and a mid-session drop are BOTH code 1006 with wasClean:false. Nothing on the event carries the distinction.\n\nThe only discriminator is whether `open` ever fired, which the page must REMEMBER ITSELF. Hence an `opened` flag set in the open handler.\n\nDeliberately NOT the watchdog's fix: the watchdog is disarmed because it exists for silence only, whereas suppressing `close` outright would hide a genuine mid-session disconnect, which is the one thing that listener is really for.\n\nA DETAIL FOUND WHILE FIXING IT, worth knowing before touching this again: armWatchdog runs only AFTER `transport.ready` resolves -- on `open` -- so on the never-opened path the watchdog never exists at all. That is not what its own comment leads you to expect, and it means the original watchdog bug could only ever have fired on a connection that SUCCEEDED.\n\nSTILL UNVERIFIED BY EXECUTION, and stated rather than glossed: web/app.js has never been run here. The change is argued from the WebSocket spec and checked with node --check (proved discriminating by appending an unbalanced paren and watching exit 1), not observed. The four assertions this fix owes are recorded on improvise-vvl3 so the browser harness can discharge them -- including the one that must FAIL against the previous unconditional handler, which is the guard and not the decoration.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7v62","title":"Checked does not bind the App it was checked against","description":"Residual hole in improvise-ondj, found and documented by the agent that closed it. Not reachable today; filed so it is a decision rather than a discovery.\n\nChecked\u003ca\u003e is a private-field newtype over a \u0026Command whose only constructor, Checked::against(\u0026App, \u0026Command), runs the basis check. But the resulting Checked carries no evidence of WHICH App it was checked against, so a multi-document host could in principle check against one App and apply to another -- and the versions would be meaningless, since document_generation, layout_version and record_versions all start at zero in a fresh App and are per document (ADR 001 scopes them that way deliberately).\n\nWHY IT IS NOT REACHABLE TODAY: a Sessions is bound to one document (connect/apply/project_all all take the App as a parameter), and Registry hands out one (App, Sessions) pair per DocumentId. It is documented on the type.\n\nTHE OPTIONS, in increasing cost: leave it documented (defensible -- this is a coordination check, not a security boundary, and the misuse requires a host to actively mix two documents); give Checked a lifetime tied to the \u0026App so the borrow checker forbids the mix; or carry a DocumentId. The middle one is probably free and worth trying before accepting the first.\n\nWON T-FIX IS A LEGITIMATE OUTCOME -- but say so deliberately, because improvise-j31x made this a multi-document server on purpose and ADR 001 notes that a document is never evicted precisely so a reconnecting client cannot have a stale basis accepted as current. The invariant is load-bearing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:54:31Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:30Z","started_at":"2026-09-20T05:32:30Z","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"in_progress","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-q0vb","title":"Pad the import wizard's hint line to the panel width","description":"Pre-existing, found by Copilot on PR #12 and deliberately NOT fixed there -- see the pushback at https://github.com/fiddlerwoaroof/improvise/pull/12#discussion_r4055516143\n\nFour buf.set_string(x, hint_y, ...) sites in src/ui/import_wizard_ui.rs write the hint without padding to the panel width, so a SHORTER message leaves the tail of the longer previous string visible.\n\nTHE REPRODUCTION IS ALREADY IN THE SUITE, as an EXPECTATION: the DefineFormulas characterization test asserts \"|something went wrongcel\" -- the \"cel\" being the tail of \"Esc: cancel\". The artifact is currently pinned as correct output.\n\nWHY IT WAS NOT FIXED IN THE MIGRATION PR: those exact-frame characterization tests were that PR's ONLY evidence that the widget migration changed nothing (doctrine(characterization_survives_the_move)). Rewriting them in the same change would have destroyed the evidence for the change. The sibling finding in the same file (the non-ASCII truncate panic) WAS applied, precisely because it is byte-identical on ASCII and therefore costs no evidence. The two findings differ in what they cost, not in whether they are right.\n\nACCEPTANCE: the tail is gone AND the affected characterization expectations are updated IN THE SAME COMMIT, so the frame change is the point of the commit rather than a side effect of one.","status":"closed","priority":4,"issue_type":"bug","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:26:46Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:29:00Z","started_at":"2026-09-20T05:32:39Z","closed_at":"2026-09-20T06:29:00Z","close_reason":"Closed by PR #22 (WU-L). The acceptance clause was met exactly: the tail is gone AND the affected characterization expectation moved in the same commit, with the commit message naming it.\n\nTHE DIAGNOSIS WAS BIGGER THAN THE TICKET'S. The bottom row had FIVE WRITERS -- four step hints plus the message that overrides them -- and none owned clearing it. So the defect was not a forgotten pad at four call sites; it was a row with five writers and no owner. One helper, set_bottom_line, now truncates to the panel width and pads to it, and all five go through it.\n\nEXACTLY ONE EXPECTATION MOVED, of eight:\n FORMULA_STEP bottom row: \"|something went wrongcel ...\" -\u003e \"|something went wrong ...\"\nThe \"cel\" was three characters of a hint the message had replaced; it was never part of the message. The other seven frames are BYTE-IDENTICAL, because padding with spaces is invisible to buf_text (which trims each row) -- so the change can only show where it ERASES something. That is the evidence this is a fix and not a rewrite of the characterization PR #12 was protecting.\n\nTWO OF COPILOT'S FOUR FINDINGS LANDED IN THIS FILE and both were about measurement rather than logic: padding by chars().count() over-pads a wide character past the panel border and under-pads a combining mark (now UnicodeWidthStr::width, as grid.rs does); and the test written to catch the second of those PASSED FOR THE WRONG REASON, because drawing through the widget cannot see under-padding at all -- the hint ran first and had already padded the row, so the cells the message failed to reach held spaces either way. Replaced with a sentinel-filled row asserted directly against set_bottom_line.\n\nTHE GENERAL LESSON, now doctrine(an_ascii_fixture_cannot_see_a_width_bug): the eight exact-frame fixtures in that file are ASCII, so bytes, chars and cells all coincide and they are blind to every question about display width. Eight green frames sat either side of both bugs.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-hkt6","title":"Three small totality gaps found in review","description":"Three PLAUSIBLE/minor findings from the 2026-09-19 wave review, each unreachable today and each one refactor away from not being. Grouped because none is worth a ticket alone.\n\n1. DataStore::matching_positions panics on an empty partial -- it indexes sets[0]. Private, and both callers guard with `if partial.is_empty()`, so it is unreachable. But THE GUARD NOW LIVES ONE LEVEL AWAY FROM THE INDEXING, which is how this class of bug returns. A sets.first()?-shaped total function costs nothing.\n\n2. invalidate_relabelled does not blank `totals` the way Invalidated does. By gotcha 33(b) own argument -- preserve-by-position is only right where the label did not change -- a RENAMED column should drop its total. Unreachable in practice: grid_updates diffs mirror.totals != desired.totals and emits Totals after ColLabels in the same batch, so a changed total is always sent. Worth fixing anyway, because 33(b) exists PRECISELY to stop `apply` trusting producers.\n\n3. improvise-gp mark_features indexes used[*feat] without a bound, where expr_text uses .get().expect(...) for the same precondition. Same failure, less legible.","status":"closed","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T01:05:16Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:54:44Z","started_at":"2026-09-20T05:32:38Z","closed_at":"2026-09-20T05:54:44Z","close_reason":"Closed by PR #20 (WU-K). All three gaps fixed, each fault-injected.\n\n(a) matching_positions total-ised via split_first. The fix is NOT vec![]: an empty partial constrains NOTHING, so EVERY cell matches -- answering \"none\" one level down would have made deleting a caller's fast path silently flip \"all\" to \"none\". The callers' partial.is_empty() branches are fast paths, not guards, and the test asserts the two doors agree.\n\n(b) invalidate_relabelled now blanks a relabelled COLUMN's total. Rows are the negative arm and stay untouched -- a total spans every row, so no one row frames it, and that arm is what stops \"blank on any relabel\" passing.\n\n(c) mark_features uses .get_mut().expect(...), matching expr_text.\n\nTHE THING THE TICKET DID NOT PREDICT, and the reason this bead was not independent of its own framing: (b) was filed as \"unreachable in practice, fix it anyway because gotcha 33(b) exists to stop apply trusting producers\". Right about the rule, wrong about the cost. Adding a blanking rule on the CONSUMER side changes what every PRODUCER owes, and grid_updates did not pay it: it diffed mirror.totals against desired.totals, so a column renamed while its total stayed the same NUMBER emitted ColLabels and no Totals -- the client blanked its own total and project_diff's self.grid = desired left the server believing it was still there. A client cache wrong forever, gotcha 35's exact failure direction. Found by Copilot. Fixed by diffing totals against the label-folded frame.\n\nWHEN YOU MAKE apply STOP TRUSTING A PRODUCER, GO READ THE PRODUCER.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-g6s1","title":"WizardStep::Preview has no characterization fixture","description":"The one import-wizard step improvise-764 could not pin. It needs a document that parses to ZERO RECORDS without needing path selection, and neither existing fixture reaches it.\n\nSmall, and worth doing while the WizardCache shape is fresh: every other step is pinned at buffer level, so Preview is the single arm where a change to wizard_cache could alter what is drawn with nothing failing.","status":"in_progress","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:32:39Z","started_at":"2026-09-20T05:32:39Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-g6s1","title":"WizardStep::Preview has no characterization fixture","description":"The one import-wizard step improvise-764 could not pin. It needs a document that parses to ZERO RECORDS without needing path selection, and neither existing fixture reaches it.\n\nSmall, and worth doing while the WizardCache shape is fresh: every other step is pinned at buffer level, so Preview is the single arm where a change to wizard_cache could alter what is drawn with nothing failing.","status":"closed","priority":4,"issue_type":"task","assignee":"wave-1","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:36Z","created_by":"Edward Langley","updated_at":"2026-09-20T06:28:59Z","started_at":"2026-09-20T05:32:39Z","closed_at":"2026-09-20T06:28:59Z","close_reason":"Closed by PR #22 (WU-L). WizardStep::Preview pinned byte for byte.\n\nPreview is only ever an ENTRY step -- no path selection needed, zero records parsed -- so nothing advances into it and neither existing fixture starts there. An empty array is the smallest document that reaches it. The step is asserted as a PRECONDITION so the fixture cannot quietly stop reaching Preview and keep passing, which is the failure mode a characterization test of an unreachable step would otherwise have.\n\nProved discriminating by perturbing project::wizard_cache (uppercasing preview_summary) and watching it fail.\n\nIt also records a wart AS-IS rather than quietly fixing it: with no records to take keys from, the summary ends \"Sample keys:\" with nothing after the colon. Pinning the wart is the point of a characterization test; fixing it silently inside one is how the evidence for a later change gets destroyed.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-k69y","title":"[bug] draw_status reads the document's active view, not the client's","description":"src/draw.rs::draw_status uses app.model_state.workbook.active_view; ViewState::active_view is the CLIENT copy (gotcha 29).\n\nHARMLESS TODAY -- they are in lockstep and the_two_active_view_names_never_diverge guards it. WRONG THE DAY improvise-jytm LANDS, which is exactly the ticket that makes them diverge on purpose (the document keeps the DEFAULT, the client is authoritative). On that day the TUI badge names the wrong view.\n\nOne line. Left unfixed because it is a behaviour change and belongs with jytm rather than ahead of it. Note the wasm client already reads the correct one (self.view.active_view), so the TUI is the outlier.","status":"open","priority":4,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-20T00:58:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:58:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-poia","title":"The records-mode + column has no projected identity","description":"From the ADR 002 spine landing (2026-09-19). In records mode the trailing + (NewMeasure) column creates a new measure on the row (gotcha 1). It carries no CellId, so creating a measure over the wire has no address.\n\nThat is correct for now rather than an oversight -- a new measure is not a write to an existing cell, so it is not a SetCell and should not borrow one identity. But it means a browser client cannot do something the TUI can.\n\nDecide what it should be: a distinct command carrying the record surrogate plus the new measure name (derived from the effect that does it today, per doctrine(derive_the_enum_from_the_effects)), or a client-side flow that adds the measure and then writes the cell as two commands. The second is probably right -- it composes from operations that already exist -- but check whether the TUI treats it as one atomic action first, because splitting an atomic action across two commands is how a half-applied state appears.","status":"open","priority":4,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-20T00:18:35Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:18:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-mjy7","title":"Export a model as SQLite for interop (one-way, native-only)","description":"DEFERRED 2026-09-19 by user ruling during the whole-backlog campaign, together with improvise-qddo and improvise-rucf, and rewritten to say where it sits relative to ADR 003.\n\nTHE CLAIM IS STILL GOOD AND IS UNAFFECTED BY THE ADR. This is EXPORT for interop -- a user opening their model in sqlite3 or DB Browser to run their own queries. One-way, no round-trip obligation, and therefore none of the grammar-duplication risk that killed improvise-9ix: a second FORMAT must track every future change to improv.pest forever, a second VIEW need not, because nothing loads it back. It sits naturally beside export_csv/export_csv_to in crates/improvise-io/src/persistence/export.rs.\n\nWHY IT IS DEFERRED ANYWAY: it is the one bead in the SQLite cluster with no dependency on ADR 003, but it carries the same cost as the cluster's real work -- a rusqlite dependency and a wasm-gate verification -- for a capability nobody has asked for concretely. The user's ruling was to defer the cluster as a unit. Un-defer on a concrete interop request, or alongside improvise-rucf.2 (the SQLite source), where the dependency is being added anyway and the incremental cost of an exporter is small.\n\nNOTE THE DIRECTION IS OPPOSITE TO ADR 003'S. ADR 003 is about READING someone else's database as a live source. This is about WRITING ours out as a convenience. They share only the rusqlite dependency and the gating constraint; do not let one justify the other.\n\nTHE ONE REAL COST: cargo check --target wasm32-unknown-unknown fails in the libsqlite3-sys C build under this repo's nix cc-wrapper, so this must be NATIVE-ONLY AND FEATURE-GATED with the wasm CI job passing untouched. VERIFY that rather than assume it -- the improvise-9ix agent was careful to note the failure is the nix wrapper injecting -fzero-call-used-regs rather than proof that bundled SQLite cannot target wasm, so the gating is about protecting the browser bundle, not about an impossibility.\n\nSCOPE WHEN IT LANDS: choose a schema that is USEFUL TO QUERY rather than one that mirrors internal storage -- a reader wants cells joined to their coordinates, not an interned key blob. Do not add a reader; that is improvise-rucf's job and it has a different answer.\n","status":"deferred","priority":4,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-09-20T00:08:37Z","created_by":"Edward Langley","updated_at":"2026-09-20T05:31:48Z","dependency_count":0,"dependent_count":0,"comment_count":0}