improvise/context/repo-map.md

Repository Map (LLM Reference)

Terminal pivot-table modeling app. Rust, Ratatui TUI, command/effect architecture. Crate improvise v0.1.0, Apache-2.0, edition 2021.

---

How to Find Things

I need to...	Look in
Add a new keybinding	command/keymap.rs → default_keymaps()
Add a new user-facing command	command/cmd.rs → implement Cmd, register in default_registry()
Add a new state mutation	ui/effect.rs → implement Effect
Change formula evaluation	model/types.rs → eval_formula(), eval_expr()
Change how cells are stored/queried	model/cell.rs → DataStore
Change category/item behavior	model/category.rs → Category
Change view axis logic	view/types.rs → View
Change grid layout computation	view/layout.rs → GridLayout
Change .improv file format	persistence/mod.rs → format_md(), parse_md()
Change number display formatting	format.rs → format_f64()
Change CLI arguments	main.rs → clap structs
Change import wizard logic	import/wizard.rs → ImportPipeline
Change grid rendering	ui/grid.rs → GridWidget
Change TUI frame layout	draw.rs → draw()
Change app state / mode transitions	ui/app.rs → App, AppMode
Write a test for model logic	model/types.rs → mod tests / mod formula_tests
Write a test for a command	command/cmd.rs → mod tests

---

Core Types and Traits

Command/Effect Pipeline (the central architecture pattern)

User keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
                (immutable)      (pure, read-only)            (state mutations)

// src/command/cmd.rs
pub trait Cmd: Debug + Send + Sync {
    fn name(&self) -> &'static str;
    fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
}

pub struct CmdContext<'a> {
    pub model: &'a Model,           // immutable
    pub layout: &'a GridLayout,     // immutable
    pub registry: &'a CmdRegistry,
    pub mode: &'a AppMode,
    pub selected: (usize, usize),   // (row, col) cursor
    pub row_offset: usize,
    pub col_offset: usize,
    pub search_query: &'a str,
    pub search_mode: bool,
    pub yanked: &'a Option<CellValue>,
    pub key_code: KeyCode,          // the key that triggered this command
    pub buffers: &'a HashMap<String, String>,
    pub expanded_cats: &'a HashSet<String>,
    // panel cursors, tile cursor, visible dimensions...
}

// src/ui/effect.rs
pub trait Effect: Debug {
    fn apply(&self, app: &mut App);
    fn changes_mode(&self) -> bool { false }  // override if effect changes AppMode
}

To add a command: implement Cmd, then in default_registry() call r.register(...) or use the effect_cmd! macro for simple cases. Bind it in default_keymaps().

To add an effect: implement Effect in effect.rs, add a constructor function.

Data Model

// src/model/types.rs
pub struct Model {
    pub name: String,
    pub categories: IndexMap<String, Category>,  // ordered
    pub data: DataStore,
    pub formulas: Vec<Formula>,
    pub views: IndexMap<String, View>,
    pub active_view: String,
    pub measure_agg: HashMap<String, AggFunc>,   // per-measure aggregation override
}
// Key methods:
//   add_category(&mut self, name) -> Result<CategoryId>  [max 12 regular]
//   category(&self, name) -> Option<&Category>
//   category_mut(&mut self, name) -> Option<&mut Category>
//   set_cell(&mut self, key: CellKey, value: CellValue)
//   evaluate(&self, key: &CellKey) -> Option<CellValue>          [formulas + raw data]
//   evaluate_aggregated(&self, key, none_cats) -> Option<CellValue>  [sums over hidden dims]
//   recompute_formulas(&mut self, none_cats)               [fixed-point formula cache]
//   add_formula(&mut self, formula: Formula)       [replaces same target+category, adds item]
//   remove_formula(&mut self, target, category)
//   category_names(&self) -> Vec<&str>                    [includes virtual]
//   regular_category_names(&self) -> Vec<&str>            [excludes _Index, _Dim, _Measure]

const MAX_CATEGORIES: usize = 12;  // virtual categories don't count

// src/model/cell.rs
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct CellKey(pub Vec<(String, String)>);  // always sorted by category name
// CellKey::new(coords) — sorts on construction, enforcing canonical form
// CellKey::with(cat, item) -> Self — returns new key with coord added/replaced
// CellKey::without(cat) -> Self — returns new key with coord removed
// CellKey::get(cat) -> Option<&str>

#[derive(Clone, PartialEq)]
pub enum CellValue {
    Number(f64),
    Text(String),
    Error(String),       // formula evaluation error (circular ref, div/0, etc.)
}
// CellValue::as_f64() -> Option<f64>
// CellValue::is_error() -> bool

pub struct DataStore {
    cells: HashMap<InternedKey, CellValue>,
    pub symbols: SymbolTable,
    index: HashMap<(Symbol, Symbol), HashSet<InternedKey>>,  // secondary index
}
// DataStore::set(&mut self, key: &CellKey, value: CellValue)
// DataStore::get(&self, key: &CellKey) -> Option<&CellValue>
// DataStore::matching_values(&self, partial: &[(String,String)]) -> Vec<CellValue>
// DataStore::matching_cells(&self, partial) -> Vec<(CellKey, CellValue)>

// src/model/category.rs
pub struct Category {
    pub id: CategoryId,             // usize
    pub name: String,
    pub kind: CategoryKind,
    pub items: IndexMap<String, Item>,  // ordered
    pub groups: IndexMap<String, Group>,
    next_item_id: ItemId,           // private, auto-increment
}
// Category::add_item(&mut self, name) -> ItemId  [deduplicates by name]
// Category::ordered_item_names(&self) -> Vec<&str>  [respects group order]

pub enum CategoryKind { Regular, VirtualIndex, VirtualDim, VirtualMeasure, Label }

Formula System

// src/formula/ast.rs
pub enum Expr {
    Number(f64),
    Ref(String),                           // reference to an item name
    BinOp(BinOp, Box<Expr>, Box<Expr>),
    UnaryMinus(Box<Expr>),
    Agg(AggFunc, Box<Expr>, Option<Filter>),
    If(Box<Expr>, Box<Expr>, Box<Expr>),
}
pub enum BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }
pub enum AggFunc { Sum, Avg, Min, Max, Count }
pub struct Formula {
    pub raw: String,              // "Profit = Revenue - Cost"
    pub target: String,           // "Profit"
    pub target_category: String,  // "Measure"
    pub expr: Expr,
    pub filter: Option<Filter>,   // WHERE clause
}

// src/formula/parser.rs
pub fn parse_formula(raw: &str, target_category: &str) -> Result<Formula>

Formula evaluation is in model/types.rs → eval_formula() / eval_expr(). Operates at full f64 precision. Display rounding in format.rs is view-only.

View and Layout

// src/view/axis.rs
pub enum Axis { Row, Column, Page, None }

// src/view/types.rs
pub struct View {
    pub name: String,
    pub category_axes: IndexMap<String, Axis>,
    pub page_selections: HashMap<String, String>,
    pub hidden_items: HashMap<String, HashSet<String>>,
    pub collapsed_groups: HashMap<String, HashSet<String>>,
    pub number_format: String,     // e.g. ",.0" or ",.2f"
    pub prune_empty: bool,
    // scroll/selection state...
}
// View::set_axis(&mut self, cat, axis)
// View::axis_of(&self, cat) -> Axis
// View::cycle_axis(&mut self, cat)        [Row→Column→Page→None→Row]
// View::transpose(&mut self)              [swap Row↔Column]
// View::categories_on(&self, axis) -> Vec<&str>
// View::on_category_added(&mut self, cat) [auto-assigns axis]

// src/view/layout.rs
pub struct GridLayout { /* computed from Model + View */ }
// GridLayout::new(model, view) -> Self
// GridLayout::cell_key(row, col) -> Option<CellKey>
// GridLayout::cell_value(row, col) -> Option<CellValue>
// GridLayout::row_label(row) -> &str
// GridLayout::col_label(col) -> &str
// GridLayout::drill_records(row, col) -> Vec<(CellKey, CellValue)>
// Records mode: auto-detected when _Index on Row + _Dim on Column

App State

// src/ui/app.rs
pub enum AppMode {
    Normal,
    Editing { minibuf: MinibufferConfig },
    FormulaEdit { minibuf: MinibufferConfig },
    FormulaPanel,
    CategoryPanel,
    ViewPanel,
    TileSelect,
    CategoryAdd { minibuf: MinibufferConfig },
    ItemAdd { minibuf: MinibufferConfig },
    ExportPrompt { minibuf: MinibufferConfig },
    CommandMode { minibuf: MinibufferConfig },
    ImportWizard,
    Help,
    Quit,
}
// Note: SearchMode is Normal + search_mode:bool flag, not a separate variant.

pub struct App {
    pub model: Model,
    pub mode: AppMode,
    pub file_path: Option<PathBuf>,
    pub dirty: bool,
    pub help_page: usize,
    pub transient_keymap: Option<Arc<Keymap>>,  // for prefix keys
    // layout cache, drill_state, wizard, buffers, panel cursors, etc.
}
// App::handle_key(&mut self, KeyEvent) -> Result<()>  [main input dispatch]
// App::rebuild_layout(&mut self)
// App::is_empty_model(&self) -> bool  [true when only virtual categories exist]

Keymap System

// src/command/keymap.rs
pub enum KeyPattern { Key(KeyCode, KeyModifiers), AnyChar, Any }
pub enum Binding {
    Cmd { name: &'static str, args: Vec<String> },
    Prefix(Arc<Keymap>),                          // Emacs-style sub-keymap
    Sequence(Vec<(&'static str, Vec<String>)>),   // multi-command chain
}
pub enum ModeKey {
    Normal, Help, FormulaPanel, CategoryPanel, ViewPanel, TileSelect,
    Editing, FormulaEdit, CategoryAdd, ItemAdd, ExportPrompt, CommandMode,
    SearchMode, ImportWizard,
}

// Keymap::lookup(&self, key, mods) -> Option<&Binding>
//   Fallback chain: exact(key,mods) → Char with NONE mods → AnyChar → Any

// KeymapSet::default_keymaps() -> Self  [builds all 14 mode keymaps]
// KeymapSet::dispatch(&self, ctx, key, mods) -> Option<Vec<Box<dyn Effect>>>

---

File Format (.improv)

Plain-text markdown-like. Not JSON (JSON is legacy, auto-detected by { prefix).

# Model Name

## Category: Region
- North
- South
- East [Coastal]        ← item in group "Coastal"
- West [Coastal]
> Coastal               ← group definition

## Category: Measure
- Revenue
- Cost
- Profit

## Formulas
- Profit = Revenue - Cost [Measure]     ← [TargetCategory]

## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"   ← text value in quotes

## View: Default (active)
Region: row
Measure: column
Time: page, Q1          ← page axis with selected item
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f

Gzip variant: .improv.gz (same content, gzipped). Persistence code: persistence/mod.rs.

---

CLI

improvise [model.improv]                          # open TUI (default)
improvise import data.csv [--no-wizard] [-o out]  # import CSV/JSON
improvise cmd 'add-cat Region' -f model.improv    # headless command(s)
improvise script setup.txt -f model.improv        # run script file

Import flags: --category, --measure, --time, --skip, --extract, --axis, --formula, --name.

---

Key Dependencies

Crate	Purpose
ratatui 0.29	TUI framework
crossterm 0.28	Terminal backend
clap 4.6 (derive)	CLI parsing
serde + serde_json	Serialization
indexmap 2	Ordered maps (categories, views)
anyhow	Error handling
chrono 0.4	Date parsing in import
flate2	Gzip for .improv.gz
csv	CSV parsing
enum_dispatch	CLI subcommand dispatch
dev: proptest, tempfile	Property testing, temp dirs

---

File Inventory

Lines / tests / path — grouped by layer.

Model layer

1692 / 66t  model/types.rs        Model struct, formula eval, CRUD, MAX_CATEGORIES=12
 621 / 28t  model/cell.rs         CellKey (canonical sort), CellValue, DataStore (interned)
 216 /  6t  model/category.rs     Category, Item, Group, CategoryKind
  79 /  3t  model/symbol.rs       Symbol interning (SymbolTable)
   6 /  0t  model/mod.rs

Formula layer

 461 / 29t  formula/parser.rs     Recursive descent parser → Formula AST
  77 /  0t  formula/ast.rs        Expr, BinOp, AggFunc, Formula, Filter (data only)
   5 /  0t  formula/mod.rs

View layer

1013 / 23t  view/layout.rs        GridLayout (pure fn of Model+View), records mode, drill
 521 / 28t  view/types.rs         View config (axes, pages, hidden, collapsed, format)
  21 /  0t  view/axis.rs          Axis enum {Row, Column, Page, None}
   7 /  0t  view/mod.rs

Command layer

3373 / 74t  command/cmd.rs         Cmd trait, CmdContext, CmdRegistry, 40+ commands
1068 / 22t  command/keymap.rs      KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
 236 / 19t  command/parse.rs       Script/command-line parser (prefix syntax)
  12 /  0t  command/mod.rs

UI layer

 942 / 41t  ui/effect.rs           Effect trait, 50+ effect types (all state mutations)
 914 / 30t  ui/app.rs              App state, AppMode (15 variants), handle_key, autosave
1036 / 13t  ui/grid.rs             GridWidget (ratatui), col widths, rendering
 617 /  0t  ui/help.rs             5-page help overlay, HELP_PAGE_COUNT=5
 347 /  0t  ui/import_wizard_ui.rs Import wizard overlay rendering
 165 /  6t  ui/cat_tree.rs         Category tree flattener for panel
 113 /  0t  ui/view_panel.rs       View list panel
 107 /  0t  ui/category_panel.rs   Category tree panel
  95 /  0t  ui/tile_bar.rs         Tile bar (axis assignment tiles)
  87 /  0t  ui/panel.rs            Generic panel frame widget
  81 /  0t  ui/formula_panel.rs    Formula list panel
  67 /  0t  ui/which_key.rs        Prefix-key hint popup
  12 /  0t  ui/mod.rs

Import layer

 773 / 38t  import/wizard.rs       ImportPipeline + ImportWizard
 292 /  9t  import/analyzer.rs     Field kind detection (Category/Measure/Time/Skip)
 244 /  8t  import/csv_parser.rs   CSV parsing, multi-file merge
   3 /  0t  import/mod.rs

Top-level

 400 /  0t  draw.rs                TUI event loop (run_tui), frame composition
 391 /  0t  main.rs                CLI entry (clap): open, import, cmd, script
 228 / 29t  format.rs              Number display formatting (view-only rounding)
 806 / 38t  persistence/mod.rs     .improv save/load (markdown format + gzip + legacy JSON)

Context docs

context/design-principles.md Architectural principles
context/plan.md              Show HN launch plan
context/repo-map.md          This file
docs/design-notes.md         Product vision & non-goals (salvaged from former SPEC.md)

Total: ~16,500 lines, 510 tests.

---

Testing Guidelines

Coverage target

Aim for ~80% line and branch coverage on logic code. This is a quality floor, not a ceiling — go higher where the code warrants it, but don't chase 100% on rendering widgets or write tests that just exercise trivial getters. Coverage should be run with cargo llvm-cov (available via nix develop).

What to test and how

Layer	Approach	Notes
Model (types, cell, category, symbol)	Unit tests + proptest	The data model is the foundation. Property tests catch invariant violations that hand-picked cases miss (see CellKey sort invariant, axis consistency).
Formula (parser, eval)	Unit tests per operator/construct	Cover each BinOp, AggFunc, IF, WHERE, unary minus, chained formulas, error cases (div-by-zero, missing ref). Ensure eval uses full f64 precision — never display-rounded values.
View (types, layout)	Unit tests + proptest	Property tests for axis assignment invariants (each category on exactly one axis, transpose is involutive, etc.). Unit tests for layout computation, records mode detection, drill.
Command (cmd, keymap, parse)	Unit tests	Test command execution by building a CmdContext and asserting on returned effects. Test keymap lookup fallback chain. Test script parser with edge cases (quoting, comments, dots).
Persistence	Round-trip tests	save → load → save must be identical. Cover groups, formulas, views, hidden items, legacy JSON detection.
Format	Unit tests	Boundary cases: comma placement at 3/4/7 digits, negative numbers, rounding half-away-from-zero (not banker's), zero, small fractions.
Import (analyzer, csv, wizard)	Unit tests	Field classification heuristics, CSV quoting (RFC 4180), multi-file merge, date extraction.
UI rendering (grid, panels, draw, help)	Generally skip	Ratatui widgets are hard to unit-test and change frequently. Test the logic they consume (layout, cat_tree, format) rather than the rendering itself.
Effects	Test indirectly	Effects are thin apply methods. Test via integration: send a key through App::handle_key and assert on resulting app state. The complex ones (drill reconciliation, import) deserve targeted unit tests.

Property tests (proptest)

Use property tests for invariants that must hold across all inputs, not as a substitute for example-based tests. Good candidates:

• Structural invariants: CellKey always sorted, each category on exactly one axis, toggle-collapse is involutive, hide/show roundtrips.
• Serialization roundtrips: save/load identity.
• Determinism: evaluate returns the same value for the same inputs.

Keep proptest case counts reasonable. The defaults (256 cases) are fine for most properties. Don't crank them up to thousands — the test suite should complete in under 2 seconds. If a property needs more cases to feel confident, that's a sign the input space should be constrained with better strategies, not brute-forced.

Bug-fix workflow

Per CLAUDE.md: write a test that demonstrates the bug before fixing it. Prefer a small unit test targeting the specific function over an integration test. The test should fail on the current code, then pass after the fix. Mark regression tests with a doc-comment explaining the bug (see model/types.rs formula_tests for examples).

What not to test

• Trivial struct constructors and enum definitions (ast.rs, axis.rs).
• Ratatui Widget::render implementations — these are pure drawing code.
• Module re-export files (mod.rs).
• One-line delegation methods.

---

Patterns to Know

1. Commands never mutate. They receive &CmdContext (read-only) and return Vec<Box<dyn Effect>>.
2. CellKey is always sorted. Use CellKey::new() — never construct the inner Vec directly.
3. category_mut() for adding items. Model has no add_item method; get the category first: m.category_mut("Region").unwrap().add_item("East").
4. Virtual categories _Index, _Dim, and _Measure always exist. is_empty_model() checks whether any non-virtual categories exist. _Measure holds numeric data fields and formula targets; add_formula auto-adds the target item.
5. Display rounding is view-only. format_f64 (half-away-from-zero) is only called in rendering. Formula eval uses full f64. 5b. Formula evaluation is fixed-point. recompute_formulas(none_cats) iterates formula evaluation until values stabilize, using a cache. evaluate_aggregated checks the cache for formula results. Circular refs produce CellValue::Error("circular").
6. Keybindings are per-mode. ModeKey::from_app_mode() resolves the current mode, then the corresponding Keymap is looked up. Normal + search_mode=true maps to SearchMode.
7. effect_cmd! macro generates a command struct that just produces effects. Use for simple commands without complex logic.
8. .improv format is markdown-like, not JSON. See persistence/mod.rs. JSON is legacy only.
9. IndexMap is used for categories and views to preserve insertion order.
10. MAX_CATEGORIES = 12 applies only to CategoryKind::Regular. Virtual/Label categories are exempt.