improvise/context/repo-map.md

# Repository Map (LLM Reference)

Terminal pivot-table modeling app. Rust, Ratatui TUI, command/effect architecture.
Crate `improvise` v0.1.0, Apache-2.0, edition 2021.

---

## How to Find Things

| I need to...                          | Look in                                      |
|---------------------------------------|----------------------------------------------|
| Add a new keybinding                  | `command/keymap.rs` → `default_keymaps()`    |
| Add a new user-facing command         | `command/cmd.rs` → implement `Cmd`, register in `default_registry()` |
| Add a new state mutation              | `ui/effect.rs` → implement `Effect`          |
| Change formula evaluation             | `model/types.rs` → `eval_formula()`, `eval_expr()` |
| Change how cells are stored/queried   | `model/cell.rs` → `DataStore`                |
| Change category/item behavior         | `model/category.rs` → `Category`             |
| Change view axis logic                | `view/types.rs` → `View`                     |
| Change grid layout computation        | `view/layout.rs` → `GridLayout`              |
| Change .improv file format            | `persistence/mod.rs` → `format_md()`, `parse_md()` |
| Change number display formatting      | `format.rs` → `format_f64()`                 |
| Change CLI arguments                  | `main.rs` → clap structs                     |
| Change import wizard logic            | `import/wizard.rs` → `ImportPipeline`        |
| Change grid rendering                 | `ui/grid.rs` → `GridWidget`                  |
| Change TUI frame layout               | `draw.rs` → `draw()`                         |
| Change app state / mode transitions   | `ui/app.rs` → `App`, `AppMode`               |
| Write a test for model logic          | `model/types.rs` → `mod tests` / `mod formula_tests` |
| Write a test for a command            | `command/cmd.rs` → `mod tests`               |

---

## Core Types and Traits

### Command/Effect Pipeline (the central architecture pattern)

```
User keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
                (immutable)      (pure, read-only)            (state mutations)
```

```rust
// src/command/cmd.rs
pub trait Cmd: Debug + Send + Sync {
    fn name(&self) -> &'static str;
    fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
}

pub struct CmdContext<'a> {
    pub model: &'a Model,           // immutable
    pub layout: &'a GridLayout,     // immutable
    pub registry: &'a CmdRegistry,
    pub mode: &'a AppMode,
    pub selected: (usize, usize),   // (row, col) cursor
    pub row_offset: usize,
    pub col_offset: usize,
    pub search_query: &'a str,
    pub search_mode: bool,
    pub yanked: &'a Option<CellValue>,
    pub key_code: KeyCode,          // the key that triggered this command
    pub buffers: &'a HashMap<String, String>,
    pub expanded_cats: &'a HashSet<String>,
    // panel cursors, tile cursor, visible dimensions...
}

// src/ui/effect.rs
pub trait Effect: Debug {
    fn apply(&self, app: &mut App);
    fn changes_mode(&self) -> bool { false }  // override if effect changes AppMode
}
```

**To add a command**: implement `Cmd`, then in `default_registry()` call `r.register(...)` or use the `effect_cmd!` macro for simple cases. Bind it in `default_keymaps()`.

**To add an effect**: implement `Effect` in `effect.rs`, add a constructor function.

### Data Model

```rust
// src/model/types.rs
pub struct Model {
    pub name: String,
    pub categories: IndexMap<String, Category>,  // ordered
    pub data: DataStore,
    pub formulas: Vec<Formula>,
    pub views: IndexMap<String, View>,
    pub active_view: String,
    pub measure_agg: HashMap<String, AggFunc>,   // per-measure aggregation override
}
// Key methods:
//   add_category(&mut self, name) -> Result<CategoryId>  [max 12 regular]
//   category(&self, name) -> Option<&Category>
//   category_mut(&mut self, name) -> Option<&mut Category>
//   set_cell(&mut self, key: CellKey, value: CellValue)
//   evaluate(&self, key: &CellKey) -> Option<CellValue>          [formulas + raw data]
//   evaluate_aggregated(&self, key, none_cats) -> Option<CellValue>  [sums over hidden dims]
//   add_formula(&mut self, formula: Formula)       [replaces same target+category]
//   remove_formula(&mut self, target, category)
//   category_names(&self) -> Vec<&str>                    [includes virtual]
//   regular_category_names(&self) -> Vec<&str>            [excludes _Index, _Dim]

const MAX_CATEGORIES: usize = 12;  // virtual categories don't count
```

```rust
// src/model/cell.rs
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct CellKey(pub Vec<(String, String)>);  // always sorted by category name
// CellKey::new(coords) — sorts on construction, enforcing canonical form
// CellKey::with(cat, item) -> Self — returns new key with coord added/replaced
// CellKey::without(cat) -> Self — returns new key with coord removed
// CellKey::get(cat) -> Option<&str>

#[derive(Clone, PartialEq)]
pub enum CellValue {
    Number(f64),
    Text(String),
}
// CellValue::as_f64() -> Option<f64>

pub struct DataStore {
    cells: HashMap<InternedKey, CellValue>,
    pub symbols: SymbolTable,
    index: HashMap<(Symbol, Symbol), HashSet<InternedKey>>,  // secondary index
}
// DataStore::set(&mut self, key: &CellKey, value: CellValue)
// DataStore::get(&self, key: &CellKey) -> Option<&CellValue>
// DataStore::matching_values(&self, partial: &[(String,String)]) -> Vec<CellValue>
// DataStore::matching_cells(&self, partial) -> Vec<(CellKey, CellValue)>
```

```rust
// src/model/category.rs
pub struct Category {
    pub id: CategoryId,             // usize
    pub name: String,
    pub kind: CategoryKind,
    pub items: IndexMap<String, Item>,  // ordered
    pub groups: IndexMap<String, Group>,
    next_item_id: ItemId,           // private, auto-increment
}
// Category::add_item(&mut self, name) -> ItemId  [deduplicates by name]
// Category::ordered_item_names(&self) -> Vec<&str>  [respects group order]

pub enum CategoryKind { Regular, VirtualIndex, VirtualDim, Label }
```

### Formula System

```rust
// src/formula/ast.rs
pub enum Expr {
    Number(f64),
    Ref(String),                           // reference to an item name
    BinOp(BinOp, Box<Expr>, Box<Expr>),
    UnaryMinus(Box<Expr>),
    Agg(AggFunc, Box<Expr>, Option<Filter>),
    If(Box<Expr>, Box<Expr>, Box<Expr>),
}
pub enum BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }
pub enum AggFunc { Sum, Avg, Min, Max, Count }
pub struct Formula {
    pub raw: String,              // "Profit = Revenue - Cost"
    pub target: String,           // "Profit"
    pub target_category: String,  // "Measure"
    pub expr: Expr,
    pub filter: Option<Filter>,   // WHERE clause
}

// src/formula/parser.rs
pub fn parse_formula(raw: &str, target_category: &str) -> Result<Formula>
```

Formula evaluation is in `model/types.rs` → `eval_formula()` / `eval_expr()`. Operates at full f64 precision. Display rounding in `format.rs` is view-only.

### View and Layout

```rust
// src/view/axis.rs
pub enum Axis { Row, Column, Page, None }

// src/view/types.rs
pub struct View {
    pub name: String,
    pub category_axes: IndexMap<String, Axis>,
    pub page_selections: HashMap<String, String>,
    pub hidden_items: HashMap<String, HashSet<String>>,
    pub collapsed_groups: HashMap<String, HashSet<String>>,
    pub number_format: String,     // e.g. ",.0" or ",.2f"
    pub prune_empty: bool,
    // scroll/selection state...
}
// View::set_axis(&mut self, cat, axis)
// View::axis_of(&self, cat) -> Axis
// View::cycle_axis(&mut self, cat)        [Row→Column→Page→None→Row]
// View::transpose(&mut self)              [swap Row↔Column]
// View::categories_on(&self, axis) -> Vec<&str>
// View::on_category_added(&mut self, cat) [auto-assigns axis]

// src/view/layout.rs
pub struct GridLayout { /* computed from Model + View */ }
// GridLayout::new(model, view) -> Self
// GridLayout::cell_key(row, col) -> Option<CellKey>
// GridLayout::cell_value(row, col) -> Option<CellValue>
// GridLayout::row_label(row) -> &str
// GridLayout::col_label(col) -> &str
// GridLayout::drill_records(row, col) -> Vec<(CellKey, CellValue)>
// Records mode: auto-detected when _Index on Row + _Dim on Column
```

### App State

```rust
// src/ui/app.rs
pub enum AppMode {
    Normal,
    Editing { minibuf: MinibufferConfig },
    FormulaEdit { minibuf: MinibufferConfig },
    FormulaPanel,
    CategoryPanel,
    ViewPanel,
    TileSelect,
    CategoryAdd { minibuf: MinibufferConfig },
    ItemAdd { minibuf: MinibufferConfig },
    ExportPrompt { minibuf: MinibufferConfig },
    CommandMode { minibuf: MinibufferConfig },
    ImportWizard,
    Help,
    Quit,
}
// Note: SearchMode is Normal + search_mode:bool flag, not a separate variant.

pub struct App {
    pub model: Model,
    pub mode: AppMode,
    pub file_path: Option<PathBuf>,
    pub dirty: bool,
    pub help_page: usize,
    pub transient_keymap: Option<Arc<Keymap>>,  // for prefix keys
    // layout cache, drill_state, wizard, buffers, panel cursors, etc.
}
// App::handle_key(&mut self, KeyEvent) -> Result<()>  [main input dispatch]
// App::rebuild_layout(&mut self)
// App::is_empty_model(&self) -> bool  [true when only virtual categories exist]
```

### Keymap System

```rust
// src/command/keymap.rs
pub enum KeyPattern { Key(KeyCode, KeyModifiers), AnyChar, Any }
pub enum Binding {
    Cmd { name: &'static str, args: Vec<String> },
    Prefix(Arc<Keymap>),                          // Emacs-style sub-keymap
    Sequence(Vec<(&'static str, Vec<String>)>),   // multi-command chain
}
pub enum ModeKey {
    Normal, Help, FormulaPanel, CategoryPanel, ViewPanel, TileSelect,
    Editing, FormulaEdit, CategoryAdd, ItemAdd, ExportPrompt, CommandMode,
    SearchMode, ImportWizard,
}

// Keymap::lookup(&self, key, mods) -> Option<&Binding>
//   Fallback chain: exact(key,mods) → Char with NONE mods → AnyChar → Any

// KeymapSet::default_keymaps() -> Self  [builds all 14 mode keymaps]
// KeymapSet::dispatch(&self, ctx, key, mods) -> Option<Vec<Box<dyn Effect>>>
```

---

## File Format (.improv)

Plain-text markdown-like. **Not JSON** (JSON is legacy, auto-detected by `{` prefix).

```
# Model Name

## Category: Region
- North
- South
- East [Coastal]        ← item in group "Coastal"
- West [Coastal]
> Coastal               ← group definition

## Category: Measure
- Revenue
- Cost
- Profit

## Formulas
- Profit = Revenue - Cost [Measure]     ← [TargetCategory]

## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"   ← text value in quotes

## View: Default (active)
Region: row
Measure: column
Time: page, Q1          ← page axis with selected item
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f
```

Gzip variant: `.improv.gz` (same content, gzipped). Persistence code: `persistence/mod.rs`.

---

## CLI

```
improvise [model.improv]                          # open TUI (default)
improvise import data.csv [--no-wizard] [-o out]  # import CSV/JSON
improvise cmd 'add-cat Region' -f model.improv    # headless command(s)
improvise script setup.txt -f model.improv        # run script file
```

Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`.

---

## Key Dependencies

| Crate | Purpose |
|-------|---------|
| ratatui 0.29 | TUI framework |
| crossterm 0.28 | Terminal backend |
| clap 4.6 (derive) | CLI parsing |
| serde + serde_json | Serialization |
| indexmap 2 | Ordered maps (categories, views) |
| anyhow | Error handling |
| chrono 0.4 | Date parsing in import |
| flate2 | Gzip for .improv.gz |
| csv | CSV parsing |
| enum_dispatch | CLI subcommand dispatch |
| **dev:** proptest, tempfile | Property testing, temp dirs |

---

## File Inventory

Lines / tests / path — grouped by layer.

### Model layer
```
1692 / 66t  model/types.rs        Model struct, formula eval, CRUD, MAX_CATEGORIES=12
 621 / 28t  model/cell.rs         CellKey (canonical sort), CellValue, DataStore (interned)
 216 /  6t  model/category.rs     Category, Item, Group, CategoryKind
  79 /  3t  model/symbol.rs       Symbol interning (SymbolTable)
   6 /  0t  model/mod.rs
```

### Formula layer
```
 461 / 29t  formula/parser.rs     Recursive descent parser → Formula AST
  77 /  0t  formula/ast.rs        Expr, BinOp, AggFunc, Formula, Filter (data only)
   5 /  0t  formula/mod.rs
```

### View layer
```
1013 / 23t  view/layout.rs        GridLayout (pure fn of Model+View), records mode, drill
 521 / 28t  view/types.rs         View config (axes, pages, hidden, collapsed, format)
  21 /  0t  view/axis.rs          Axis enum {Row, Column, Page, None}
   7 /  0t  view/mod.rs
```

### Command layer
```
3373 / 74t  command/cmd.rs         Cmd trait, CmdContext, CmdRegistry, 40+ commands
1068 / 22t  command/keymap.rs      KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
 236 / 19t  command/parse.rs       Script/command-line parser (prefix syntax)
  12 /  0t  command/mod.rs
```

### UI layer
```
 942 / 41t  ui/effect.rs           Effect trait, 50+ effect types (all state mutations)
 914 / 30t  ui/app.rs              App state, AppMode (15 variants), handle_key, autosave
1036 / 13t  ui/grid.rs             GridWidget (ratatui), col widths, rendering
 617 /  0t  ui/help.rs             5-page help overlay, HELP_PAGE_COUNT=5
 347 /  0t  ui/import_wizard_ui.rs Import wizard overlay rendering
 165 /  6t  ui/cat_tree.rs         Category tree flattener for panel
 113 /  0t  ui/view_panel.rs       View list panel
 107 /  0t  ui/category_panel.rs   Category tree panel
  95 /  0t  ui/tile_bar.rs         Tile bar (axis assignment tiles)
  87 /  0t  ui/panel.rs            Generic panel frame widget
  81 /  0t  ui/formula_panel.rs    Formula list panel
  67 /  0t  ui/which_key.rs        Prefix-key hint popup
  12 /  0t  ui/mod.rs
```

### Import layer
```
 773 / 38t  import/wizard.rs       ImportPipeline + ImportWizard
 292 /  9t  import/analyzer.rs     Field kind detection (Category/Measure/Time/Skip)
 244 /  8t  import/csv_parser.rs   CSV parsing, multi-file merge
   3 /  0t  import/mod.rs
```

### Top-level
```
 400 /  0t  draw.rs                TUI event loop (run_tui), frame composition
 391 /  0t  main.rs                CLI entry (clap): open, import, cmd, script
 228 / 29t  format.rs              Number display formatting (view-only rounding)
 806 / 38t  persistence/mod.rs     .improv save/load (markdown format + gzip + legacy JSON)
```

### Context docs
```
context/design-principles.md Architectural principles
context/plan.md              Show HN launch plan
context/repo-map.md          This file
docs/design-notes.md         Product vision & non-goals (salvaged from former SPEC.md)
```

**Total: ~16,500 lines, 510 tests.**

---

## Testing Guidelines

### Coverage target

Aim for **~80% line and branch coverage** on logic code. This is a quality floor, not a
ceiling — go higher where the code warrants it, but don't chase 100% on rendering
widgets or write tests that just exercise trivial getters. Coverage should be run with
`cargo llvm-cov` (available via `nix develop`).

### What to test and how

| Layer | Approach | Notes |
|-------|----------|-------|
| **Model** (types, cell, category, symbol) | Unit tests + **proptest** | The data model is the foundation. Property tests catch invariant violations that hand-picked cases miss (see CellKey sort invariant, axis consistency). |
| **Formula** (parser, eval) | Unit tests per operator/construct | Cover each BinOp, AggFunc, IF, WHERE, unary minus, chained formulas, error cases (div-by-zero, missing ref). Ensure eval uses full f64 precision — never display-rounded values. |
| **View** (types, layout) | Unit tests + **proptest** | Property tests for axis assignment invariants (each category on exactly one axis, transpose is involutive, etc.). Unit tests for layout computation, records mode detection, drill. |
| **Command** (cmd, keymap, parse) | Unit tests | Test command execution by building a `CmdContext` and asserting on returned effects. Test keymap lookup fallback chain. Test script parser with edge cases (quoting, comments, dots). |
| **Persistence** | Round-trip tests | `save → load → save` must be identical. Cover groups, formulas, views, hidden items, legacy JSON detection. |
| **Format** | Unit tests | Boundary cases: comma placement at 3/4/7 digits, negative numbers, rounding half-away-from-zero (not banker's), zero, small fractions. |
| **Import** (analyzer, csv, wizard) | Unit tests | Field classification heuristics, CSV quoting (RFC 4180), multi-file merge, date extraction. |
| **UI rendering** (grid, panels, draw, help) | Generally skip | Ratatui widgets are hard to unit-test and change frequently. Test the *logic* they consume (layout, cat_tree, format) rather than the rendering itself. |
| **Effects** | Test indirectly | Effects are thin `apply` methods. Test via integration: send a key through `App::handle_key` and assert on resulting app state. The complex ones (drill reconciliation, import) deserve targeted unit tests. |

### Property tests (proptest)

Use property tests for **invariants that must hold across all inputs**, not as a
substitute for example-based tests. Good candidates:

- Structural invariants: CellKey always sorted, each category on exactly one axis,
  toggle-collapse is involutive, hide/show roundtrips.
- Serialization roundtrips: save/load identity.
- Determinism: `evaluate` returns the same value for the same inputs.

Keep proptest case counts reasonable. The defaults (256 cases) are fine for most
properties. Don't crank them up to thousands — the test suite should complete in
under 2 seconds. If a property needs more cases to feel confident, that's a sign
the input space should be constrained with better strategies, not brute-forced.

### Bug-fix workflow

Per CLAUDE.md: **write a test that demonstrates the bug before fixing it.** Prefer
a small unit test targeting the specific function over an integration test. The test
should fail on the current code, then pass after the fix. Mark regression tests
with a doc-comment explaining the bug (see `model/types.rs` `formula_tests` for
examples).

### What not to test

- Trivial struct constructors and enum definitions (`ast.rs`, `axis.rs`).
- Ratatui `Widget::render` implementations — these are pure drawing code.
- Module re-export files (`mod.rs`).
- One-line delegation methods.

---

## Patterns to Know

1. **Commands never mutate.** They receive `&CmdContext` (read-only) and return `Vec<Box<dyn Effect>>`.
2. **CellKey is always sorted.** Use `CellKey::new()` — never construct the inner Vec directly.
3. **`category_mut()` for adding items.** `Model` has no `add_item` method; get the category first: `m.category_mut("Region").unwrap().add_item("East")`.
4. **Virtual categories** `_Index` and `_Dim` always exist. `is_empty_model()` checks whether any *non-virtual* categories exist.
5. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full f64.
6. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the current mode, then the corresponding `Keymap` is looked up. Normal + `search_mode=true` maps to `SearchMode`.
7. **`effect_cmd!` macro** generates a command struct that just produces effects. Use for simple commands without complex logic.
8. **`.improv` format is markdown-like**, not JSON. See `persistence/mod.rs`. JSON is legacy only.
9. **`IndexMap`** is used for categories and views to preserve insertion order.
10. **`MAX_CATEGORIES = 12`** applies only to `CategoryKind::Regular`. Virtual/Label categories are exempt.