improvise/context/repo-map.md

Repository Map (LLM Reference)

Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture. Apache-2.0. Root binary+lib improvise; sub-crates under crates/:

• improvise-core — Model, View, Workbook, number formatting. No UI/IO.
• improvise-formula — formula parser, AST, parse_formula.
• improvise-io — .improv save/load, CSV/JSON import. No UI/commands.

The main crate re-exports each as crate::{model, view, workbook, format, formula, persistence, import} so consumer paths stay stable when crates shuffle.

Architectural intent lives in context/design-principles.md — read that for the "why". This doc is the "where".

---

How to Find Things

I need to...	Look in
Add a keybinding	command/keymap.rs → default_keymaps()
Add a user command	command/cmd/<submodule>.rs, register in registry.rs
Add a state mutation	ui/effect.rs → implement Effect
Change formula eval	model/types.rs → eval_formula / eval_expr
Change cell storage / lookup	model/cell.rs → DataStore
Change category/item behavior	model/category.rs → Category
Change view axis logic	view/types.rs → View
Change grid layout	view/layout.rs → GridLayout
Change .improv format	persistence/improv.pest + persistence/mod.rs
Change number display	format.rs → format_f64
Change CLI args	main.rs (clap)
Change import logic	import/wizard.rs → ImportPipeline
Change frame layout	draw.rs → draw()
Change app state / modes	ui/app.rs → App, AppMode
Write a test for model logic	model/types.rs → mod tests / formula_tests
Write a test for a command	command/cmd/<module>.rs → colocated mod tests; helpers in cmd/mod.rs::test_helpers

---

Central Pattern: Cmd → Effect

keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
                            (pure, read-only ctx)       (list of mutations)    (only mutation site)

pub trait Cmd: Debug + Send + Sync {
    fn name(&self) -> &'static str;
    fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
}

pub trait Effect: Debug {
    fn apply(&self, app: &mut App);
    fn changes_mode(&self) -> bool { false }   // override when Effect swaps AppMode
}

CmdContext (see command/cmd/core.rs) holds immutable refs to Model, GridLayout, AppMode, cursor/offsets, buffers, yanked cell, key code, expanded cats, and panel/tile cursors.

Add a command: implement Cmd in a command/cmd/ submodule, register in registry.rs, bind in default_keymaps(). Simple wrappers go through the effect_cmd! macro in effect_cmds.rs.

Add an effect: implement Effect in ui/effect.rs, add a constructor fn if it helps composition.

---

Key Types (skim; read the source for fields/methods)

Model (model/types.rs): categories: IndexMap<String, Category>, data: DataStore, formulas: Vec<Formula>, views: IndexMap<String, View>, active_view, measure_agg. MAX_CATEGORIES = 12 (regular only). Virtual categories _Index, _Dim, _Measure always exist. Use regular_category_names() for user-facing pickers.

CellKey (model/cell.rs): Vec<(cat, item)> — always sorted by category. Build with CellKey::new(coords) / with(cat, item) / without(cat). Never construct the inner Vec directly.

CellValue: Number(f64) | Text(String) | Error(String). Errors surface from formula eval (circular, div/0, etc.).

DataStore (model/cell.rs): interned symbols + secondary index (Symbol, Symbol) → {InternedKey}. Hot-path lookups go through matching_values / matching_cells.

Formula AST (improvise-formula/src/ast.rs):

• Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }
• BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }
• AggFunc { Sum, Avg, Min, Max, Count }
• Formula { raw, target, target_category, expr, filter }

View (view/types.rs): category_axes: IndexMap<cat, Axis>, page selections, hidden items, collapsed groups, number_format, prune_empty. Axis { Row, Column, Page, None }. cycle_axis rotates Row → Column → Page → None.

GridLayout (view/layout.rs): pure function of Model + View. cell_key(r,c), cell_value(r,c), drill_records(r,c). Records mode auto-detects when _Index is on Row and _Dim is on Column.

AppMode (ui/app.rs): 15 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, CommandMode, ImportWizard, Help, Quit). SearchMode is Normal + search_mode: bool, not its own variant.

Keymap (command/keymap.rs): Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }. Lookup fallback: exact(key,mods) → Char(NONE) → AnyChar → Any → parent. 14 mode keymaps built by KeymapSet::default_keymaps(); mode resolved via ModeKey::from_app_mode().

---

.improv File Format

Plain-text, markdown-like, defined by persistence/improv.pest. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via pest_meta).

v2025-04-09
# Model Name
Initial View: Default

## View: Default
Region: row
Measure: column
|Time Period|: page, Q1
hidden: Region/Internal
collapsed: |Time Period|/|2024|
format: ,.2f

## Formulas
- Profit = Revenue - Cost               # defaults to [_Measure]
- Tax = Revenue * 0.1 [CustomCat]

## Category: Region
- North, South, East, West              # bare items, comma-separated
- Coastal_East[Coastal]                 # grouped item, one per line
> Coastal                               # group definition

## Data
Region=East, Measure=Revenue = 1200
Region=West, Measure=Revenue = |pending|   # pipe-quoted text

• Name quoting: bare [A-Za-z_][A-Za-z0-9_-]*, else CL-style |…| with escapes \|, \\, \n. Same convention in the formula tokenizer.
• Write order: Views → Formulas → Categories → Data. Parser tolerates any order.
• Gzip: .improv.gz (same text, gzipped).
• Legacy JSON: auto-detected by leading {; never written.
• Virtual categories: _Index/_Dim never persist; _Measure persists only non-formula items (formula targets are rebuilt from the ## Formulas section).

---

Gotchas (read before editing)

1. Commands never mutate. &CmdContext is read-only; return Vec<Box<dyn Effect>>. If you want to touch &mut App, add or extend an Effect.
2. CellKey is always sorted. Use the constructors; equality and hashing rely on canonical form.
3. No Model::add_item. Go through the category: m.category_mut("Region").unwrap().add_item("East").
4. Virtual categories. _Measure items = explicit items ∪ formula targets. Use measure_item_names() / effective_item_names("_Measure"). add_formula does not add items to _Measure. Formula-target lookup also falls back into _Measure via find_item_category.
5. Display rounding is view-only. format_f64 (half-away-from-zero) is only called in rendering. Formula eval uses full f64. Never feed a rounded value back into eval.
6. Formula eval is fixed-point. recompute_formulas(none_cats) iterates until stable, bounded by MAX_EVAL_DEPTH; circular refs converge to CellValue::Error("circular"). App::new runs it before the first frame so formula cells render on startup.
7. Drill into formula cells strips the _Measure coordinate when it names a formula target, so matching_cells returns the raw records that feed the formula instead of returning empty.
8. Keybindings are per-mode. ModeKey::from_app_mode() resolves the mode; Normal + search_mode=true maps to SearchMode. Minibuffer modes bind Enter/Esc as Binding::Sequence so clear-buffer fires alongside commit/cancel.
9. effect_cmd! macro wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written Cmd once there's real decision logic.
10. Float equality inside formula eval is currently mixed — =/!= use a 1e-10 epsilon but the div-by-zero guard checks rv == 0.0 exactly. Don't assume one or the other; tracked for cleanup.
11. IndexMap preserves insertion order for categories, views, and items. Persistence relies on this.
12. Commit paths for cell edits split across commit_cell_value → commit_regular_cell_value / commit_plain_records_edit in command/cmd/commit.rs. Keep the synthetic-records branch in sync with the plain branch when changing behavior.

---

File Inventory

Line counts are static; test counts are informational — run cargo test --workspace for live numbers. Files under 100 lines and render-only widgets omitted.

improvise-core (crates/improvise-core/src/)

model/types.rs         2062 / 70t   Model, formula eval (fixed-point), CRUD
model/cell.rs           650 / 28t   CellKey (sorted), CellValue, DataStore (interned + index)
model/category.rs       222 /  6t   Category, Item, Group, CategoryKind
model/symbol.rs          79 /  3t   Symbol interning
view/layout.rs         1140 / 24t   GridLayout, drill, records mode
view/types.rs           531 / 28t   View config (axes, pages, hidden, collapsed, format)
view/axis.rs             21         Axis enum
workbook.rs             259 / 11t   Workbook: Model + cross-view ops
format.rs               229 / 29t   format_f64, parse_number_format (display only)

improvise-formula (crates/improvise-formula/src/)

parser.rs               776 / 65t   pest grammar + tokenizer → Formula AST
ast.rs                   77         Expr, BinOp, AggFunc, Formula, Filter

improvise-io (crates/improvise-io/src/)

persistence/improv.pest 124         PEG grammar — single source of truth
persistence/mod.rs     2410 / 83t   save/load/gzip/legacy-JSON, CSV export
import/wizard.rs       1117 / 38t   ImportPipeline + ImportWizard
import/analyzer.rs      292 /  9t   Field kind detection (Category/Measure/Time/Skip)
import/csv_parser.rs    300 /  8t   CSV parsing, multi-file merge

Command layer (src/command/)

cmd/core.rs             297 /  2t   Cmd trait, CmdContext, CmdRegistry, parse helpers
cmd/registry.rs         586 /  0t   default_registry() — all registrations (no tests yet)
cmd/navigation.rs       475 / 10t   Move, EnterAdvance, Page*
cmd/cell.rs             198 /  6t   ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
cmd/commit.rs           330 /  7t   CommitFormula, CommitCategoryAdd/ItemAdd, CommitExport
cmd/effect_cmds.rs      437 /  5t   effect_cmd! macro, 25+ simple wrappers
cmd/grid.rs             409 /  7t   ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
cmd/mode.rs             308 /  8t   EnterMode, Quit, EditOrDrill, EnterTileSelect
cmd/panel.rs            587 / 13t   Panel toggle/cycle/cursor, formula/category/view panels
cmd/search.rs           202 /  4t   SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
cmd/text_buffer.rs      256 /  7t   AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
cmd/tile.rs             160 /  5t   MoveTileCursor, TileAxisOp
keymap.rs              1066 / 22t   KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
parse.rs                236 / 19t   Script/command-line parser (prefix syntax)

UI, draw, main (src/ui/, src/draw.rs, src/main.rs)

ui/effect.rs            942 / 41t   Effect trait, 50+ effect types
ui/app.rs               914 / 30t   App state, AppMode (15), handle_key, autosave
ui/grid.rs             1036 / 13t   GridWidget (ratatui), column widths
ui/help.rs              617         5-page help overlay (render only)
ui/import_wizard_ui.rs  347         Import wizard rendering
ui/cat_tree.rs          165 /  6t   Category tree flattener for panel
draw.rs                 400         TUI event loop, frame composition
main.rs                 391         CLI entry (clap): open, import, cmd, script
# other ui/*.rs are small panel renderers — skip unless changing layout/style

Examples

examples/gen-grammar.rs             Grammar-walking random .improv generator (pest_meta)
examples/pretty-print.rs            Parse stdin, print formatted .improv

Context docs

context/design-principles.md        Architectural principles & testing doctrine
context/plan.md                     Show HN launch plan
context/repo-map.md                 This file
docs/design-notes.md                Product vision & non-goals

---

CLI

improvise [model.improv]                          # open TUI (default)
improvise import data.csv [--no-wizard] [-o out]  # import CSV/JSON
improvise cmd 'add-cat Region' -f model.improv    # headless command(s)
improvise script setup.txt -f model.improv        # run script file

Import flags: --category, --measure, --time, --skip, --extract, --axis, --formula, --name.

---

Testing — the short version

Full guidance lives in context/design-principles.md §6. Quick reminders:

• Suite runs in <2s. Don't let that drift.
• Commands: build a CmdContext, call execute, assert on returned effects. No terminal needed.
• Property tests (proptest, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, parse(format(parse(generate()))) stability.
• Bug-fix workflow: write a failing test before the fix (regression guard). Document the bug in the test's doc-comment (see model/types.rs::formula_tests).
• Coverage target ~80% line/branch on logic code; skip ratatui render paths.

---

Key dependencies

ratatui 0.30, crossterm 0.28, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.