(Stretch) Hybrid mode: native TUI with remote browser observer

Once the native binary has the in-process client/server split (issue improvise-e0u), allow a second subscriber to attach via a local ws-server thread. Result: a developer running the native TUI can share their session with a teammate in a browser, or have their own browser tab observing while they work in the terminal. Both subscribers receive projection commands from the same in-process server; commands from either side flow through the same reduce_full.

Native binary gains a --serve PORT flag (or similar) that, when set, spawns a ws-server thread alongside the native client. The ws-server is exactly improvise-q08's server logic, attached to the same in-process App as the native client. Two ClientSessions: one for the local ratatui client (zero-transport), one (or more) for remote websocket clients. Each has its own ViewState + render cache + subscription set. The projection layer fans out effect-driven updates to every session. The two subscribers see independent cursor/scroll/mode state (because view state is per-session) but the same model state (because model is shared). This is the deployment mode that fully justifies per-session view state tracking on the server.

(1) Native binary has an optional --serve flag that spawns a ws-server in the same process. (2) A remote browser can connect and see the native user's current session (shared model, independent view). (3) Edits from either side show up on the other after a round trip. (4) Graceful disconnect/reconnect of remote clients while native keeps running.

Stretch goal — not required for the core unification. Depends on issue e0u (binary split done) and improvise-q08 (ws-server exists). Orthogonal to the browser and standalone epics. The architectural investment in per-session view state and subscription-based projections is what makes this possible with minimal additional code.


Migrate import wizard widget and state to consume RenderCache

Rewrite ui/import_wizard_ui.rs (347 lines) to consume a WizardCache instead of the ImportWizard struct directly. Also decide where the import wizard state itself lives in the new crate layout — probably needs to stay in improvise-command (not improvise-io) so it can be referenced from App without creating a dep cycle.

WizardCache mirrors the ImportWizard state: current step, per-field decisions, preview rows, current validation errors. The widget reads cache + ViewState and draws. The ImportWizard state machine stays wherever it ends up living after the crate-split epic step 3 resolves its placement — either improvise-command (portable state machine) or improvise-io (tangled with CSV parsing). The separation between wizard state (session/view-ish) and import pipeline (pure data processing) is worth getting right because it affects the worker-server's ability to run imports.

(1) import_wizard_ui.rs no longer reads the wizard state directly — takes WizardCache. (2) ImportWizard is split into state + pipeline; state lives in an appropriate crate that doesn't cycle. (3) Native TUI wizard rendering unchanged. (4) Unit tests with fixture WizardCaches for each step.

Depends on improvise-edp (ViewModel layer). Wizard widget consumes WizardViewModel derived from WizardCache + ViewState. The wizard state machine itself (step tracking, field decisions) lives in the ImportWizard struct on the server side; the cache is a snapshot of its renderable state; the viewmodel adds any styling/layout derivation the widget needs.


Migrate help and which_key widgets to consume RenderCache + ViewState

Rewrite the small static-content widgets: ui/help.rs (5-page help overlay, 617 lines but mostly static text) and ui/which_key.rs (prefix-key hint popup). Lowest-risk widgets in the migration because their content is almost entirely derived from static data plus a small amount of ViewState (help_page index, active transient_keymap).

HelpCache: largely static; just the page-index state and the content pages (constant). Could even be stateless if the renderer reads help_page directly from ViewState. WhichKeyCache: derived from the active transient_keymap — a flat list of (key, binding description) entries. Each widget becomes a pure render function taking its cache + ViewState for minimal dynamic bits.

(1) help.rs and which_key.rs no longer import App. (2) Unit-testable with fixture caches (or no cache for help). (3) Native TUI rendering unchanged.

Depends on improvise-edp (ViewModel layer). Help widget consumes HelpViewModel (derived from the static help content + current help_page); which_key consumes WhichKeyViewModel (derived from the active transient_keymap). Low-complexity widgets, but go through the viewmodel layer for consistency with the rest of the migration.


SQLite persistence format (alternative save/restore)

Implement .sqlite as an alternative file format alongside .improv. Tables for categories, items, cells, formulas, views. Works over any VFS backend — native filesystem, OPFS, etc. — because the underlying storage layer is abstracted. Enables structured queries over model state and provides a more scalable container than plaintext for very large models. Orthogonal to the core standalone deployment; filed as a backlog follow-on since the MVP doesn't need it.

Use rusqlite with the 'bundled' feature so sqlite itself compiles into the wasm bundle. Schema: one row per cell keyed by (coord_hash, category_coords...); one row per category with kind + ordering; one row per item with group membership; one row per formula with raw + target. Views stored as JSON or as structured rows. Persistence API mirrors .improv: save_sqlite(storage, path) / load_sqlite(storage, path). Rusqlite on wasm may need custom VFS hookup — investigate sqlite-wasm-rs or similar for browser-friendly builds. Round-trip tests: model→sqlite→model must be identity.

(1) save_sqlite and load_sqlite functions exist and round-trip cleanly. (2) rusqlite links against the VFS abstraction correctly on both native and wasm. (3) Unit tests for schema, round-trip, and edge cases. (4) File manager UI (if any) offers .sqlite as a save format option.

Depends on improvise-6mq (VFS abstraction). Independent of the standalone deployment critical path — can be tackled any time after the Storage trait lands. Filed as backlog (P3).


Static HTML shell + GitHub Pages deploy workflow

HTML/JS shell that hosts the standalone wasm deployment. Registers the worker-server (dedicated worker for MVP, service worker as a later PWA enhancement), instantiates the MessageChannel transport, loads the main-thread wasm-client, wires them together, and presents the DOM renderer. Also publishes the whole bundle to GitHub Pages via a build + deploy workflow so anyone can open improvise in a browser with no install.

docs/ or web/ directory with index.html (#app div), main.js (main-thread wasm loader + worker registration + MessageChannel setup + keyboard event wiring), worker.js (loads the worker-server wasm bundle and handles postMessage), style.css. No websocket code — transport is local MessageChannel only. GitHub Actions: on push to main, build both wasm bundles (main-thread and worker) with cargo + wasm-bindgen, run wasm-opt for size, assemble into a static site, push to gh-pages branch. Path-filtered to skip if neither bundle changed. Worker registration: start with Dedicated Worker (simpler lifecycle, same-tab scope) until PWA install requirements justify Service Worker complexity.

(1) Static site loads improvise in a modern browser with no backend. (2) GitHub Actions workflow builds and deploys on every main push. (3) Deployed URL is listed in the README. (4) Demo .improv file is bundled so first-time visitors see something, not an empty grid.

Depends on improvise-djm (main-thread entry point) and the worker-server issue. Mostly configuration and glue — heavy lifting lives in the upstream wasm crates.


DOM renderer for browser grid (reads ViewState + render cache)

JS/TS (or Rust via web-sys) layer that subscribes to the wasm client's state and renders the grid to the DOM. MVP: rebuild a <table> element on each state change, no virtual-DOM diffing. Reads ViewState for mode indicator, cursor highlight, minibuffer text; reads render cache for cell contents, labels, column widths. Captures browser keyboard events and routes them into the wasm client's on_key_event.

Option A: pure JS/TS module that reads wasm-exposed state via JsValue and updates DOM imperatively. Simpler for MVP. Option B: Rust + web-sys in the wasm crate, rendering from inside wasm. More code sharing but bigger bundle. Start with Option A. Renderer: single <table> for the grid body, <thead> for column labels, <tbody> with rows. On state change, re-render the affected sections. Cursor highlight is a CSS class on the selected <td>. Mode indicator is a <div> above the table. Minibuffer is a <div> shown conditionally when mode is Editing/FormulaEdit/etc.

(1) Grid renders from a ViewState + RenderCache snapshot. (2) Cursor highlight updates on cursor move without full re-render (nice to have, not required for MVP). (3) Mode indicator reflects current AppMode. (4) Keyboard events on document are captured and routed to wasm on_key_event. (5) Works in Chrome and Firefox. Help overlay, panels, import wizard, tile bar — all deferred post-MVP.

Consumes viewmodels (not the render cache directly) per improvise-edp. Specifically GridViewModel for grid rendering. Shares GridViewModel type and compute_grid_viewmodel function with ratatui's grid widget (improvise-n10) — one derivation, two rendering backends. DOM-specific concerns (device pixel ratio, CSS class names) live in the RenderEnv the viewmodel is computed against, not in the viewmodel itself.


Formula tokenizer: support quoted identifiers for ambiguous names

The formula tokenizer currently uses heuristics to handle multi-word identifiers (greedy space consumption with keyword/operator break rules). This is fragile — we just fixed a bug where WHERE inside aggregates was consumed as part of an identifier.

A more robust approach: support SQL/CL-style quoted identifiers. Two candidate syntaxes:
- SQL style: double-quotes for identifiers, e.g. "Total Revenue" = "Base Revenue" + Bonus
- CL style: pipe-delimited, e.g. |Total Revenue| = |Base Revenue| + Bonus

SQL double-quoting is natural since the formula syntax already uses quotes for WHERE filter string literals. Disambiguation: in expression position, a quoted string becomes Token::Ident; in WHERE value position, it becomes Token::Str (parser already distinguishes these contexts).

This would let us simplify the tokenizer's space-handling heuristics and cleanly support category/item names that collide with keywords (WHERE, SUM, IF, etc.).

Implemented pipe-quoted identifiers |...| in the formula tokenizer. Pipes produce Token::Ident, work in expressions, aggregates, and WHERE clauses. split_where and parse_where also updated. 6 new tests.


Bug: WHERE inside aggregate parens broken by greedy identifier tokenizer

The tokenizer treats multi-word identifiers greedily — 'Revenue WHERE' becomes a single token. This means SUM(Revenue WHERE Region=East) doesn't parse correctly. The WHERE keyword is consumed as part of the identifier. Top-level WHERE (outside parens) works because split_where handles it before tokenization. Fix: either stop allowing spaces in identifiers inside parens, or detect WHERE as a keyword break point in the tokenizer.

Fixed tokenizer to break multi-word identifiers at keywords (WHERE, SUM, AVG, MIN, MAX, COUNT, IF). Two-pronged: (1) break when current identifier IS a keyword, (2) break when next word IS a keyword. SUM(Revenue WHERE Region=East) now parses correctly.


Test audit: persistence/mod.rs at 82% coverage

persistence/mod.rs is at 82% - just above the floor. Missing coverage for edge cases: empty models, special characters in names, view state persistence details, and formula preservation corner cases.

Added 16 tests covering save/load roundtrip (plain + gzip), autosave_path, export_csv, collapsed groups, page-without-selection, none axis, number format, text values, multiple views, and full feature roundtrip. Coverage from 82% to 97%.


Test audit: formula/parser.rs at 76% coverage

formula/parser.rs is at 76% line coverage, just under the 80% target. Missing coverage for MIN, MAX, COUNT aggregate functions, complex nested expressions, and error/edge case paths.

Added 19 tests covering MIN/MAX/COUNT aggregates, comparison operators, power/unary/mul/div, WHERE with quotes, error paths, multi-word identifiers, aggregate-name-as-ref edge case. Coverage from 76% to 89%. Found bug: WHERE inside aggregate parens broken by greedy tokenizer (improvise-0zf).


3.3 Tag v0.1.0 release

Create git tag v0.1.0, push it, verify cargo dist workflow produces release artifacts. Update README prebuilt binaries link to point at actual release.


3.2 Publish to crates.io

After cargo publish --dry-run is clean and user confirms, run cargo publish. Verify crate appears at crates.io/crates/improvise and cargo install improvise works.


3.1 Configure cargo dist

Run cargo dist init targeting x86_64-unknown-linux-gnu, aarch64-apple-darwin, x86_64-apple-darwin. Skip Windows and musl. Commit generated .github/workflows/release.yml and Cargo.toml additions. Test with v0.1.0-rc1 tag, verify builds, delete rc tag.


Phase 3: Distribution

Configure cargo dist, publish to crates.io, tag v0.1.0 release. Produces prebuilt binaries for Linux x86_64 and macOS (Intel + Apple Silicon).


Parquet columnar export (read-only)

One-way export to Apache Parquet for interop with data tools (pandas, Arrow, BI tools). Columnar layout maps naturally to improvise's category/measure/cell structure: each category becomes a column of the key space, each measure a value column. Read-back not required for MVP; this is primarily a data-out path. Orthogonal to the core standalone deployment; filed as a backlog follow-on.

Use arrow-rs + parquet crates (or polars as a simpler wrapper). Model shape → Arrow RecordBatch: one row per cell, columns are (category_1, category_2, ..., measure_name, value). Write the batch to a parquet file via the VFS. Read-back (parquet → model) is a stretch goal — most users will export once and consume the file elsewhere. Browser build: parquet + arrow crates are heavy; may push the wasm bundle size significantly. Consider making parquet export a separate optional wasm chunk loaded on demand.

(1) Export function writes a valid parquet file from the current model. (2) File opens correctly in pandas / pyarrow and reproduces the improv data. (3) Works over VFS (native filesystem + OPFS).

Depends on improvise-6mq (VFS abstraction). Very low priority — a nice data-science handoff story but not required for core functionality. Filed as P4 backlog.


4.2 Enable GitHub Pages

Enable Pages in GitHub repo settings: source=main branch, folder=/docs. Verify site is live at the GitHub Pages URL.


4.1 Create docs/index.html landing page

Vanilla HTML, single file, under 200 lines. Dark background, monospace headings. Embed asciinema-player from jsdelivr CDN. Sections: title/tagline, pivot cast, drill cast, formulas cast, import cast, install commands + GitHub link. Player config: rows 30, cols 100, theme monokai, autoPlay false, loop true.


Phase 4: Landing page (optional)

Create docs/index.html with embedded asciinema casts and enable GitHub Pages. Optional but recommended for launch.


App::new calls recompute_formulas before building initial layout so formula values appear on first frame. render() test helper also calls recompute_formulas.


lib.rs created to enable examples to import from improvise crate. main.rs uses 'use improvise::*' instead of mod declarations.


Deep-review batches work best with narrowly-scoped Explore agents (one per layer/principle lens: model+formula, command+ui, persistence+import, LoD violations, OCP violations). Prompt each with: (1) specific files with sizes, (2) the lens/principle to apply, (3) exactly how to report (count + file:line refs, prioritized, ~800 words). Parallel launches worked cleanly when file scopes did not overlap. Session 2026-04-16 produced a ~20-issue backlog this way.


Beads issues created by agents (owner 'spot') can duplicate users actively in-progress work. Example 2026-04-16: improvise-dwe (Split App into AppState + App wrapper) was a coarser restatement of improvise-vb4 (Split AppState into ModelState + ViewState) which was already in_progress and assigned to the user; dwe was filed 2 days later by an agent unaware of vb4. Before filing a structural refactor, run bd search for keywords from its core concept AND check in_progress issues.


DrillIntoCell strips _Measure coordinate from drill key when it matches a formula target, so matching_cells finds raw data records instead of returning empty.


gen-grammar example: generates random valid .improv content from pest grammar rules. Uses word pools for realistic output. pretty-print example: parses stdin and prints formatted output. Both in examples/ directory.


Running theme across the 2026-04-16 refactor backlog: the compilers exhaustive-match check is being bypassed. (1) string compares against virtual-category names (improvise-2lh); (2) registered command names as strings vs Cmd::name() (improvise-9cn, improvise-61f); (3) minibuffer buffer_key strings threaded through 7 AppMode constructors vs command lookups (improvise-k8h); (4) AppMode per-variant logic scattered across 52 match sites in 8 files (improvise-2hi); (5) duplicated Axis display matches in tile_bar + category_panel (improvise-rml). Common fix: push dispatch onto a method on the enum/type so the exhaustive match has one home.


Dynamic _Measure: formula targets are dynamically included via Model::measure_item_names() and effective_item_names(). add_formula no longer adds items to _Measure category. find_item_category falls back to formula targets. CommitFormula defaults target_category to _Measure.


Persistence: _Index and _Dim categories are never written to .improv files. _Measure only persists non-formula items. Formulas targeting _Measure omit the [_Measure] suffix (it's the default). Parser defaults to _Measure when no [Category] suffix present.


Keymap parent inheritance added (linked list of keymaps). All minibuffer modes use Binding::Sequence for Enter and Esc to include clear-buffer command. ClearBufferCmd registered in registry.