improvise/context/repo-map.md

Repository Map (LLM Reference)

Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture. Apache-2.0. Root binary+lib improvise; sub-crates under crates/:

• improvise-core — Model, View, Workbook, number formatting. No UI/IO.
• improvise-formula — formula parser, AST, parse_formula.
• improvise-io — .improv save/load, CSV/JSON import, Storage backends. No UI/commands.
• improvise-command — command/effect layer + semantic App state (app.rs, effect.rs, cat_tree.rs, viewport.rs). No ratatui/crossterm.

The main crate (and improvise-command) re-export the sub-crates as crate::{model, view, workbook, format, formula, persistence, import} so consumer paths stay stable when crates shuffle. The main crate also re-exports improvise_command::command as crate::command and improvise_command::ui::{app, cat_tree, effect, viewport} under crate::ui.

Architectural intent lives in context/design-principles.md — read that for the "why". This doc is the "where".

---

How to Find Things

I need to...	Look in
Add a keybinding	command/keymap.rs → default_keymaps()
Add a user command	command/cmd/<submodule>.rs, register in registry.rs
Add a state mutation	ui/effect.rs → implement Effect
Change formula eval	model/types.rs → eval_formula / eval_expr
Change cell storage / lookup	model/cell.rs → DataStore
Change category/item behavior	model/category.rs → Category
Change view axis logic	view/types.rs → View
Change grid layout	view/layout.rs → GridLayout
Change .improv format	persistence/improv.pest + persistence/mod.rs
Change where documents are stored	storage.rs → Storage impls
Change number display	format.rs → format_f64
Change CLI args	main.rs (clap)
Change import logic	import/wizard.rs → ImportPipeline
Change import-wizard keys/steps	command/cmd/wizard.rs (keys) + import/wizard.rs (steps)
Change frame layout	draw.rs → draw()
Change app state / modes	ui/app.rs → App, AppMode
Write a test for model logic	model/types.rs → mod tests / formula_tests
Write a test for a command	command/cmd/<module>.rs → colocated mod tests; helpers in cmd/mod.rs::test_helpers

---

Central Pattern: Cmd → Effect

keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
                            (pure, read-only ctx)       (list of mutations)    (only mutation site)

pub trait Cmd: Debug + Send + Sync {
    fn name(&self) -> &'static str;
    fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
}

pub trait Effect: Debug {
    fn apply(&self, app: &mut App);
    fn kind(&self) -> EffectKind;              // NO default — the compiler makes you classify
    fn changes_mode(&self) -> bool { false }   // override when Effect swaps AppMode
}

// EffectKind::{ViewOnly, Model(ModelChange)}
// ModelChange::{CellData, LayoutGeometry, WholeWorkbook}

CmdContext (see command/cmd/core.rs) holds immutable refs to Model, GridLayout, AppMode, cursor/offsets, buffers, yanked cell, key code, expanded cats, and panel/tile cursors.

Add a command: implement Cmd in a command/cmd/ submodule, register in registry.rs, bind in default_keymaps(). Simple wrappers go through the effect_cmd! macro in effect_cmds.rs.

Add an effect: implement Effect in ui/effect.rs, add a constructor fn if it helps composition.

---

Key Types (skim; read the source for fields/methods)

Model (model/types.rs): categories: IndexMap<String, Category>, data: DataStore, formulas: Vec<Formula>, views: IndexMap<String, View>, active_view, measure_agg. MAX_CATEGORIES = 12 (regular only). Virtual categories _Index, _Dim, _Measure always exist. Use regular_category_names() for user-facing pickers.

CellKey (model/cell.rs): Vec<(cat, item)> — always sorted by category. Build with CellKey::new(coords) / with(cat, item) / without(cat). Never construct the inner Vec directly.

CellValue: Number(f64) | Text(String) | Error(String). Errors surface from formula eval (circular, div/0, etc.).

DataStore (model/cell.rs): interned symbols + secondary index (Symbol, Symbol) → {InternedKey}. Hot-path lookups go through matching_values / matching_cells.

Formula AST (improvise-formula/src/ast.rs):

• Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }
• BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }
• AggFunc { Sum, Avg, Min, Max, Count }
• Formula { raw, target, target_category, expr, filter }

View (view/types.rs): category_axes: IndexMap<cat, Axis>, page selections, hidden items, collapsed groups, number_format, prune_empty. Axis { Row, Column, Page, None }. cycle_axis rotates Row → Column → Page → None.

GridLayout (view/layout.rs): pure function of Model + View. cell_key(r,c), cell_value(r,c), drill_records(r,c). Records mode auto-detects when _Index is on Row and _Dim is on Column.

AppMode (ui/app.rs): 16 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, CommandMode, ImportWizard, Help, Quit, RecordsNormal, RecordsEditing). Text-entry variants carry a MinibufferConfig. SearchMode is Normal + search_mode: bool, not its own variant. Per-variant UI facts live on the enum as exhaustive-match methods — label(), hint_text(), panel(), minibuffer(), is_editing(), is_records() (improvise-2hi); add a variant and the compiler names every one you must fill in. mode_style() stays in src/draw.rs because Style is a ratatui type. App state is split into ModelState (workbook, file_path, dirty) and ViewState (mode, cursor/offsets, buffers, panels, search) — see improvise-x2c/ew0.

Keymap (command/keymap.rs): Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }. Lookup fallback: exact(key,mods) → Char(NONE) → AnyChar → Any → parent. 16 mode keymaps built by KeymapSet::default_keymaps(); mode resolved via ModeKey::from_app_mode().

---

.improv File Format

Plain-text, markdown-like, defined by persistence/improv.pest. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via pest_meta).

v2025-04-09
# Model Name
Initial View: Default

## View: Default
Region: row
Measure: column
|Time Period|: page, Q1
hidden: Region/Internal
collapsed: |Time Period|/|2024|
format: ,.2f

## Formulas
- Profit = Revenue - Cost               # defaults to [_Measure]
- Tax = Revenue * 0.1 [CustomCat]

## Category: Region
- North, South, East, West              # bare items, comma-separated
- Coastal_East[Coastal]                 # grouped item, one per line
> Coastal                               # group definition

## Data
Region=East, Measure=Revenue = 1200
Region=West, Measure=Revenue = |pending|   # pipe-quoted text

• Name quoting: bare [A-Za-z_][A-Za-z0-9_-]*, else CL-style |…| with escapes \|, \\, \n. Same convention in the formula tokenizer.
• Write order: Views → Formulas → Categories → Data. Parser tolerates any order.
• Gzip: .improv.gz (same text, gzipped).
• Legacy JSON: auto-detected by leading {; never written.
• Virtual categories: _Index/_Dim never persist; _Measure persists only non-formula items (formula targets are rebuilt from the ## Formulas section).
• Virtual views: _Records/_Drill (view::VIRTUAL_VIEW_NAMES) are never written and are dropped on load if an older file carries them; saving from records/drill mode writes the first real view as Initial View. Their commands rebuild them on demand.

---

Gotchas (read before editing)

1. Records mode is wide format. A row is one coordinate tuple — the cells sharing everything but _Measure — and each raw measure gets its own editable column, followed by a + column that creates a new measure (improvise-c5v). Consequences: two measures on one coordinate are ONE row; editing a category column moves every measure cell in the row, not just the one under the cursor; _Dim in the synthetic key carries the column index, not a label, because a category and a measure may share a name — resolve it with GridLayout::record_col into a RecordCol. Formula targets are not columns (nothing to type into them). Drill pending edits are keyed by (row, RecordCol).
2. Every effect declares an EffectKind, and there is no default. ViewOnly never reaches other clients; Model(CellData | LayoutGeometry | WholeWorkbook) does (improvise-mae). A new effect that forgets to classify itself fails to compile — that is deliberate, since the failure mode is silent: a mis-declared ViewOnly edit would simply never reach connected clients. Watch two traps: the View lives inside model_state.workbook, so "touches model_state" does NOT mean "is a model change" (axis/page/format are LayoutGeometry, cursor and scroll are ViewOnly — see improvise-8uz); and Save/SaveAs/MarkDirty touch model_state for bookkeeping only and are ViewOnly.
3. Commands never mutate. &CmdContext is read-only; return Vec<Box<dyn Effect>>. If you want to touch &mut App, add or extend an Effect.
4. CellKey is always sorted. Use the constructors; equality and hashing rely on canonical form.
5. No Model::add_item. With a Workbook in hand use the facade: wb.add_item("Region", "East")? / add_item_in_group / remove_item (improvise-tvt). With only a Model, go through the category: m.category_mut("Region").unwrap().add_item("East").
6. Virtual categories. _Measure items = explicit items ∪ formula targets. Use measure_item_names() / effective_item_names("_Measure"). add_formula does not add items to _Measure. Formula-target lookup also falls back into _Measure via Model::find_item_category. Cell aggregation goes through apply_agg, formula SUM(…)/AVG(…) through eval_agg — they differ on empty input (absent vs. 0 for Sum/Count, error for Avg/Min/Max).
7. Display rounding is view-only. format_f64 (half-away-from-zero) is only called in rendering. Formula eval uses full f64. Never feed a rounded value back into eval.
8. Formula eval is fixed-point. recompute_formulas(none_cats) iterates until stable, bounded by MAX_EVAL_DEPTH; circular refs converge to CellValue::Error("circular"). App::new runs it before the first frame so formula cells render on startup.
9. Drill into formula cells strips the _Measure coordinate when it names a formula target, so matching_cells returns the raw records that feed the formula instead of returning empty.
10. Minibuffer keymaps come from TextEntry. The seven text-entry modes are built by the TextEntry builder in keymap.rs, which binds the shared spine (AnyChar→append-char <buf>, Backspace→pop-char <buf>, Esc→[clear-buffer <buf>, enter-mode <exit>]) and welds every commit key to its clear-buffer via .commit(key, cmd, args) (improvise-w3q). TextEntry::new takes the AppMode and reads <buf> off its MinibufferConfig, so the buffer the prompt names and the buffer the keys edit are the same string by construction (improvise-k8h) — the keymap never spells a buffer name. .rebind exists only for CommandMode's Backspace. SearchMode is hand-written on purpose — no commit key, and exit-search-mode owns the query.
11. Panel keymaps come from PanelKeymap. The three panel modes share a spine (Esc/Tab→Normal, arrows and j/k→move-panel-cursor <panel>, every panel's toggle key, :→command mode) built by PanelKeymap::new(panel); the keys that act on the panel's contents arrive through .bind/.bind_args/.delete because they vary in shape, not just command name (improvise-4pg). Panel::ALL/name()/key() in ui/app.rs drive the toggle keys for Normal and all three panels, so a fourth panel reaches every keymap from one ALL entry. name() must spell what parse_panel accepts.
12. Prefix keys are stored as an address, not a keymap. ViewState::transient_prefix holds the (KeyCode, KeyModifiers) that opened a prefix; KeymapSet::prefix_keymap resolves it back to the sub-keymap, walking parents so RecordsNormal reaches Normal's g/y/Z (improvise-jc8). ViewState must stay free of improvise-command types — it is destined for the protocol crate, which improvise-command will depend on. Resolution assumes prefixes are flat; prefix_keymaps_are_flat fails the build if a nested one is added.
13. Keybindings are per-mode. ModeKey::from_app_mode() resolves the mode; Normal + search_mode=true maps to SearchMode. Minibuffer modes bind Enter/Esc as Binding::Sequence so clear-buffer fires alongside commit/cancel.
14. Registry names come from prototypes, not strings. Every register* call reads the name off a prototype Cmd instance, so a typo is impossible (improvise-9cn) — there is no name literal in registry.rs. What types can't reach: four commands name themselves off a field (ViewNavigate, TileAxisOp, ToggleGroupAtCursor, CommitAndAdvance), so a prototype can name the wrong variant and still compile. That shows up as a duplicate registration, not a mismatch — the wrong-variant entry collides with its sibling and the sibling's name vanishes. registered_names_are_unique catches it; every_bound_command_name_is_registered catches the keymap side.
15. effect_cmd! macro wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written Cmd once there's real decision logic.
16. Float equality inside formula eval is currently mixed — =/!= use a 1e-10 epsilon but the div-by-zero guard checks rv == 0.0 exactly. Don't assume one or the other; tracked for cleanup.
17. IndexMap preserves insertion order for categories, views, and items. Persistence relies on this.
18. Commit paths for cell edits split across commit_cell_value → commit_regular_cell_value / commit_plain_records_edit in command/cmd/commit.rs. Keep the synthetic-records branch in sync with the plain branch when changing behavior.
19. Minibuffer commits go through one command. commit-buffer <formula|category|item> (CommitBuffer { kind: CommitKind }) owns buffer→trim→bail-on-empty→dirty→status for all three; the varying parts are methods on CommitKind. The add effect is emitted last so its failure message wins over the command's optimistic status (improvise-6t3). CommitKind::Item carries its target category, resolved from AppMode::ItemAdd by parse_commit_kind at registry time — so commit-buffer item is not constructible from a script line.
20. A failed Save aborts the batch. The Save effect sets abort_effects when there's no file path or the write fails (improvise-8tk), so anything chained after it — the Quit in wq/ZZ — is skipped and the error stays on screen. A Binding::Sequence flattens into one effect vec, so the abort carries across sequence steps too.
21. The import wizard's back path is a history stack, not a reverse table. advance pushes the step it leaves onto ImportWizard::history; previous pops it, so back always retraces the path actually taken through the conditional transitions (improvise-907). Preconditions live in validation_error(), checked on the step where the user can still fix them — never deferred to build_model. NameModel is terminal for advance: Enter there calls build(), which parks the workbook in built on Done, and AcceptWizardModel takes it from there. Going back from Done discards built.
22. app.layout goes stale mid-batch. apply_effects rebuilds only after the whole batch, so an effect that reads the layout after an earlier effect changed the grid needs a RebuildLayout effect in front of it — the command decides the order (see EnterEditAtCursorCmd, CommitAndAdvance). Read the cursor's cell text via App::display_value_at_cursor(); don't build a CmdContext inside apply (it recomputes column widths).

---

File Inventory

Line counts are static; test counts are informational — run cargo test --workspace for live numbers. Files under 100 lines and render-only widgets omitted.

improvise-core (crates/improvise-core/src/)

model/types.rs         2062 / 70t   Model, formula eval (fixed-point), CRUD
model/cell.rs           650 / 28t   CellKey (sorted), CellValue, DataStore (interned + index)
model/category.rs       222 /  6t   Category, Item, Group, CategoryKind
model/symbol.rs          79 /  3t   Symbol interning
view/layout.rs         1140 / 24t   GridLayout, drill, records mode
view/types.rs           531 / 28t   View config (axes, pages, hidden, collapsed, format)
view/axis.rs             21         Axis enum
workbook.rs             342 / 17t   Workbook: Model + cross-view ops, item facade
format.rs               229 / 29t   format_f64, parse_number_format (display only)

improvise-formula (crates/improvise-formula/src/)

parser.rs              1080 / 65t   pest grammar + tokenizer → Formula AST
ast.rs                   77         Expr, BinOp, AggFunc, Formula, Filter

improvise-io (crates/improvise-io/src/)

storage.rs               130 /  5t   Storage trait, PhysicalFs, MemoryFs (byte-level I/O)
persistence/improv.pest 124         PEG grammar — single source of truth
persistence/mod.rs     2402 / 83t   save/load/gzip/legacy-JSON, CSV export
import/wizard.rs       1121 / 38t   ImportPipeline + ImportWizard
import/analyzer.rs      292 /  9t   Field kind detection (Category/Measure/Time/Skip)
import/csv_parser.rs    300 /  8t   CSV parsing, multi-file merge

Command layer (crates/improvise-command/src/command/)

cmd/core.rs             319 /  2t   Cmd trait, CmdContext, CmdRegistry, parse helpers
cmd/registry.rs         604 /  0t   default_registry() — all registrations (no tests yet)
cmd/navigation.rs       501 / 10t   Move, EnterAdvance, Page*
cmd/cell.rs             198 /  6t   ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
cmd/commit.rs           517 /  9t   CommitAndAdvance, CommitBuffer(CommitKind), CommitExport
cmd/effect_cmds.rs      466 /  5t   effect_cmd! macro, 25+ simple wrappers
cmd/grid.rs             494 /  7t   ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
cmd/mode.rs             354 /  8t   EnterMode, Quit, EditOrDrill, EnterTileSelect
cmd/panel.rs            587 / 13t   Panel toggle/cycle/cursor, formula/category/view panels
cmd/search.rs           202 /  4t   SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
cmd/text_buffer.rs      240 /  7t   AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
cmd/tile.rs             160 /  5t   MoveTileCursor, TileAxisOp
cmd/wizard.rs           199 /  4t   HandleWizardKey — per-step import-wizard key tables
keymap.rs              1327 / 24t   KeyPattern, Binding, Keymap, TextEntry, ModeKey, 16 mode keymaps
parse.rs                236 / 19t   Script/command-line parser (prefix syntax)

Semantic UI state (crates/improvise-command/src/ui/)

ui/effect.rs           2021 / 54t   Effect trait, 50+ effect types
ui/app.rs              1867 / 55t   App, ModelState, ViewState, AppMode (16) + dispatch methods, Panel, handle_key, autosave, HELP_PAGE_COUNT
ui/cat_tree.rs          156 /  6t   Category tree flattener for panel
ui/viewport.rs          128         compute_col_widths / row_header_width / visible_cols (pure geometry)

TUI rendering, draw, main (src/ui/, src/draw.rs, src/main.rs)

ui/grid.rs              998 / 13t   GridWidget (ratatui); geometry helpers live in improvise-command viewport.rs
ui/help.rs              603         5-page help overlay (render only)
ui/import_wizard_ui.rs  347         Import wizard rendering
draw.rs                 422         TUI event loop, frame composition
main.rs                 388         CLI entry (clap): open, import, cmd, script
# other ui/*.rs are small panel renderers — skip unless changing layout/style

Examples

examples/gen-grammar.rs             Grammar-walking random .improv generator (pest_meta)
examples/pretty-print.rs            Parse stdin, print formatted .improv

Context docs

context/design-principles.md        Architectural principles & testing doctrine
context/plan.md                     Show HN launch plan
context/repo-map.md                 This file
docs/design-notes.md                Product vision & non-goals

---

Wasm

The browser bundle is improvise-core, improvise-formula, improvise-io and improvise-command. All four cargo check --target wasm32-unknown-unknown clean with no feature flags (improvise-ywd), gated by the wasm job in .github/workflows/ci.yml. The root improvise crate is the terminal binary and does not build for wasm — it depends on crossterm; that is correct, not a gap.

cargo check passing is not the same as running. Two known runtime hazards it cannot see: Instant::now() panics on wasm (hence web-time in improvise-command), and any std::fs call compiles but fails in a browser — see improvise-fuj for the two effects that still bypass Storage.

CLI

improvise [model.improv]                          # open TUI (default)
improvise import data.csv [--no-wizard] [-o out]  # import CSV/JSON
improvise cmd 'add-cat Region' -f model.improv    # headless command(s)
improvise script setup.txt -f model.improv        # run script file

Import flags: --category, --measure, --time, --skip, --extract, --axis, --formula, --name.

---

Testing — the short version

Full guidance lives in context/design-principles.md §6. Quick reminders:

• Suite runs in <2s. Don't let that drift.
• Commands: build a CmdContext, call execute, assert on returned effects. No terminal needed.
• Property tests (proptest, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, parse(format(parse(generate()))) stability.
• Bug-fix workflow: write a failing test before the fix (regression guard). Document the bug in the test's doc-comment (see model/types.rs::formula_tests).
• Coverage target ~80% line/branch on logic code; skip ratatui render paths.

---

Key dependencies

ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.