# Repository Map (LLM Reference)

Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture.
Apache-2.0. Root binary+lib `improvise`; sub-crates under `crates/`:

- `improvise-core` — `Model`, `View`, `Workbook`, number formatting. No UI/IO.
- `improvise-formula` — formula parser, AST, `parse_formula`.
- `improvise-io` — `.improv` save/load, CSV/JSON import. No UI/commands.

The main crate re-exports each as `crate::{model, view, workbook, format, formula, persistence, import}` so consumer paths stay stable when crates shuffle.

Architectural intent lives in `context/design-principles.md` — read that for the "why". This doc is the "where".

---

## How to Find Things

| I need to...                        | Look in                                         |
|-------------------------------------|-------------------------------------------------|
| Add a keybinding                    | `command/keymap.rs` → `default_keymaps()`       |
| Add a user command                  | `command/cmd/<submodule>.rs`, register in `registry.rs` |
| Add a state mutation                | `ui/effect.rs` → implement `Effect`             |
| Change formula eval                 | `model/types.rs` → `eval_formula` / `eval_expr` |
| Change cell storage / lookup        | `model/cell.rs` → `DataStore`                   |
| Change category/item behavior       | `model/category.rs` → `Category`                |
| Change view axis logic              | `view/types.rs` → `View`                        |
| Change grid layout                  | `view/layout.rs` → `GridLayout`                 |
| Change `.improv` format             | `persistence/improv.pest` + `persistence/mod.rs` |
| Change number display               | `format.rs` → `format_f64`                      |
| Change CLI args                     | `main.rs` (clap)                                |
| Change import logic                 | `import/wizard.rs` → `ImportPipeline`           |
| Change frame layout                 | `draw.rs` → `draw()`                            |
| Change app state / modes            | `ui/app.rs` → `App`, `AppMode`                  |
| Write a test for model logic        | `model/types.rs` → `mod tests` / `formula_tests` |
| Write a test for a command          | `command/cmd/<module>.rs` → colocated `mod tests`; helpers in `cmd/mod.rs::test_helpers` |

---

## Central Pattern: Cmd → Effect

```
keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
                            (pure, read-only ctx)       (list of mutations)    (only mutation site)
```

```rust
pub trait Cmd: Debug + Send + Sync {
    fn name(&self) -> &'static str;
    fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
}

pub trait Effect: Debug {
    fn apply(&self, app: &mut App);
    fn changes_mode(&self) -> bool { false }   // override when Effect swaps AppMode
}
```

`CmdContext` (see `command/cmd/core.rs`) holds immutable refs to `Model`, `GridLayout`, `AppMode`, cursor/offsets, buffers, yanked cell, key code, expanded cats, and panel/tile cursors.

**Add a command**: implement `Cmd` in a `command/cmd/` submodule, register in `registry.rs`, bind in `default_keymaps()`. Simple wrappers go through the `effect_cmd!` macro in `effect_cmds.rs`.

**Add an effect**: implement `Effect` in `ui/effect.rs`, add a constructor fn if it helps composition.

---

## Key Types (skim; read the source for fields/methods)

**Model** (`model/types.rs`): `categories: IndexMap<String, Category>`, `data: DataStore`, `formulas: Vec<Formula>`, `views: IndexMap<String, View>`, `active_view`, `measure_agg`. `MAX_CATEGORIES = 12` (regular only). Virtual categories `_Index`, `_Dim`, `_Measure` always exist. Use `regular_category_names()` for user-facing pickers.

**CellKey** (`model/cell.rs`): `Vec<(cat, item)>` — **always sorted by category**. Build with `CellKey::new(coords)` / `with(cat, item)` / `without(cat)`. Never construct the inner `Vec` directly.

**CellValue**: `Number(f64) | Text(String) | Error(String)`. Errors surface from formula eval (circular, div/0, etc.).

**DataStore** (`model/cell.rs`): interned symbols + secondary index `(Symbol, Symbol) → {InternedKey}`. Hot-path lookups go through `matching_values` / `matching_cells`.

**Formula AST** (`improvise-formula/src/ast.rs`):
- `Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }`
- `BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }`
- `AggFunc { Sum, Avg, Min, Max, Count }`
- `Formula { raw, target, target_category, expr, filter }`

**View** (`view/types.rs`): `category_axes: IndexMap<cat, Axis>`, page selections, hidden items, collapsed groups, `number_format`, `prune_empty`. `Axis { Row, Column, Page, None }`. `cycle_axis` rotates Row → Column → Page → None.

**GridLayout** (`view/layout.rs`): pure function of `Model + View`. `cell_key(r,c)`, `cell_value(r,c)`, `drill_records(r,c)`. **Records mode** auto-detects when `_Index` is on Row and `_Dim` is on Column.

**AppMode** (`ui/app.rs`): 16 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, CommandMode, ImportWizard, Help, Quit, RecordsNormal, RecordsEditing). Text-entry variants carry a `MinibufferConfig`. `SearchMode` is Normal + `search_mode: bool`, not its own variant. `App` state is split into `ModelState` (workbook, file_path, dirty) and `ViewState` (mode, cursor/offsets, buffers, panels, search) — see improvise-x2c/ew0.

**Keymap** (`command/keymap.rs`): `Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }`. Lookup fallback: `exact(key,mods) → Char(NONE) → AnyChar → Any → parent`. 16 mode keymaps built by `KeymapSet::default_keymaps()`; mode resolved via `ModeKey::from_app_mode()`.

---

## `.improv` File Format

Plain-text, markdown-like, defined by `persistence/improv.pest`. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via `pest_meta`).

```
v2025-04-09
# Model Name
Initial View: Default

## View: Default
Region: row
Measure: column
|Time Period|: page, Q1
hidden: Region/Internal
collapsed: |Time Period|/|2024|
format: ,.2f

## Formulas
- Profit = Revenue - Cost               # defaults to [_Measure]
- Tax = Revenue * 0.1 [CustomCat]

## Category: Region
- North, South, East, West              # bare items, comma-separated
- Coastal_East[Coastal]                 # grouped item, one per line
> Coastal                               # group definition

## Data
Region=East, Measure=Revenue = 1200
Region=West, Measure=Revenue = |pending|   # pipe-quoted text
```

- **Name quoting**: bare `[A-Za-z_][A-Za-z0-9_-]*`, else CL-style `|…|` with escapes `\|`, `\\`, `\n`. Same convention in the formula tokenizer.
- **Write order**: Views → Formulas → Categories → Data. Parser tolerates any order.
- **Gzip**: `.improv.gz` (same text, gzipped).
- **Legacy JSON**: auto-detected by leading `{`; never written.
- **Virtual categories**: `_Index`/`_Dim` never persist; `_Measure` persists only non-formula items (formula targets are rebuilt from the `## Formulas` section).

---

## Gotchas (read before editing)

1. **Commands never mutate.** `&CmdContext` is read-only; return `Vec<Box<dyn Effect>>`. If you want to touch `&mut App`, add or extend an Effect.
2. **`CellKey` is always sorted.** Use the constructors; equality and hashing rely on canonical form.
3. **No `Model::add_item`.** Go through the category: `m.category_mut("Region").unwrap().add_item("East")`.
4. **Virtual categories.** `_Measure` items = explicit items ∪ formula targets. Use `measure_item_names()` / `effective_item_names("_Measure")`. `add_formula` does *not* add items to `_Measure`. Formula-target lookup also falls back into `_Measure` via `find_item_category`.
5. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full `f64`. Never feed a rounded value back into eval.
6. **Formula eval is fixed-point.** `recompute_formulas(none_cats)` iterates until stable, bounded by `MAX_EVAL_DEPTH`; circular refs converge to `CellValue::Error("circular")`. `App::new` runs it before the first frame so formula cells render on startup.
7. **Drill into formula cells** strips the `_Measure` coordinate when it names a formula target, so `matching_cells` returns the raw records that feed the formula instead of returning empty.
8. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the mode; Normal + `search_mode=true` maps to `SearchMode`. Minibuffer modes bind Enter/Esc as `Binding::Sequence` so `clear-buffer` fires alongside commit/cancel.
9. **`effect_cmd!` macro** wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written `Cmd` once there's real decision logic.
10. **Float equality inside formula eval is currently mixed** — `=`/`!=` use a `1e-10` epsilon but the div-by-zero guard checks `rv == 0.0` exactly. Don't assume one or the other; tracked for cleanup.
11. **`IndexMap`** preserves insertion order for categories, views, and items. Persistence relies on this.
12. **Commit paths for cell edits** split across `commit_cell_value` → `commit_regular_cell_value` / `commit_plain_records_edit` in `command/cmd/commit.rs`. Keep the synthetic-records branch in sync with the plain branch when changing behavior.

---

## File Inventory

Line counts are static; test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted.

### `improvise-core` (`crates/improvise-core/src/`)
```
model/types.rs         2062 / 70t   Model, formula eval (fixed-point), CRUD
model/cell.rs           650 / 28t   CellKey (sorted), CellValue, DataStore (interned + index)
model/category.rs       222 /  6t   Category, Item, Group, CategoryKind
model/symbol.rs          79 /  3t   Symbol interning
view/layout.rs         1140 / 24t   GridLayout, drill, records mode
view/types.rs           531 / 28t   View config (axes, pages, hidden, collapsed, format)
view/axis.rs             21         Axis enum
workbook.rs             266 / 11t   Workbook: Model + cross-view ops
format.rs               229 / 29t   format_f64, parse_number_format (display only)
```

### `improvise-formula` (`crates/improvise-formula/src/`)
```
parser.rs              1080 / 65t   pest grammar + tokenizer → Formula AST
ast.rs                   77         Expr, BinOp, AggFunc, Formula, Filter
```

### `improvise-io` (`crates/improvise-io/src/`)
```
persistence/improv.pest 124         PEG grammar — single source of truth
persistence/mod.rs     2402 / 83t   save/load/gzip/legacy-JSON, CSV export
import/wizard.rs       1121 / 38t   ImportPipeline + ImportWizard
import/analyzer.rs      292 /  9t   Field kind detection (Category/Measure/Time/Skip)
import/csv_parser.rs    300 /  8t   CSV parsing, multi-file merge
```

### Command layer (`src/command/`)
```
cmd/core.rs             319 /  2t   Cmd trait, CmdContext, CmdRegistry, parse helpers
cmd/registry.rs         601 /  0t   default_registry() — all registrations (no tests yet)
cmd/navigation.rs       501 / 10t   Move, EnterAdvance, Page*
cmd/cell.rs             198 /  6t   ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
cmd/commit.rs           455 /  7t   CommitFormula, CommitCategoryAdd/ItemAdd, CommitExport
cmd/effect_cmds.rs      466 /  5t   effect_cmd! macro, 25+ simple wrappers
cmd/grid.rs             494 /  7t   ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
cmd/mode.rs             354 /  8t   EnterMode, Quit, EditOrDrill, EnterTileSelect
cmd/panel.rs            587 / 13t   Panel toggle/cycle/cursor, formula/category/view panels
cmd/search.rs           202 /  4t   SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
cmd/text_buffer.rs      256 /  7t   AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
cmd/tile.rs             160 /  5t   MoveTileCursor, TileAxisOp
keymap.rs              1248 / 22t   KeyPattern, Binding, Keymap, ModeKey, 16 mode keymaps
parse.rs                236 / 19t   Script/command-line parser (prefix syntax)
```

### UI, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`)
```
ui/effect.rs           1860 / 41t   Effect trait, 50+ effect types
ui/app.rs              1475 / 30t   App, ModelState, ViewState, AppMode (16), handle_key, autosave
ui/grid.rs             1063 / 13t   GridWidget (ratatui), column widths
ui/help.rs              603         5-page help overlay (render only)
ui/import_wizard_ui.rs  347         Import wizard rendering
ui/cat_tree.rs          156 /  6t   Category tree flattener for panel
draw.rs                 422         TUI event loop, frame composition
main.rs                 388         CLI entry (clap): open, import, cmd, script
# other ui/*.rs are small panel renderers — skip unless changing layout/style
```

### Examples
```
examples/gen-grammar.rs             Grammar-walking random .improv generator (pest_meta)
examples/pretty-print.rs            Parse stdin, print formatted .improv
```

### Context docs
```
context/design-principles.md        Architectural principles & testing doctrine
context/plan.md                     Show HN launch plan
context/repo-map.md                 This file
docs/design-notes.md                Product vision & non-goals
```

---

## CLI

```
improvise [model.improv]                          # open TUI (default)
improvise import data.csv [--no-wizard] [-o out]  # import CSV/JSON
improvise cmd 'add-cat Region' -f model.improv    # headless command(s)
improvise script setup.txt -f model.improv        # run script file
```

Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`.

---

## Testing — the short version

Full guidance lives in `context/design-principles.md` §6. Quick reminders:

- Suite runs in <2s. Don't let that drift.
- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed.
- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability.
- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Document the bug in the test's doc-comment (see `model/types.rs::formula_tests`).
- Coverage target ~80% line/branch on logic code; skip ratatui render paths.

---

## Key dependencies

ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
