255 lines
14 KiB
Markdown
255 lines
14 KiB
Markdown
# Repository Map (LLM Reference)
|
||
|
||
Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture.
|
||
Apache-2.0. Root binary+lib `improvise`; sub-crates under `crates/`:
|
||
|
||
- `improvise-core` — `Model`, `View`, `Workbook`, number formatting. No UI/IO.
|
||
- `improvise-formula` — formula parser, AST, `parse_formula`.
|
||
- `improvise-io` — `.improv` save/load, CSV/JSON import. No UI/commands.
|
||
|
||
The main crate re-exports each as `crate::{model, view, workbook, format, formula, persistence, import}` so consumer paths stay stable when crates shuffle.
|
||
|
||
Architectural intent lives in `context/design-principles.md` — read that for the "why". This doc is the "where".
|
||
|
||
---
|
||
|
||
## How to Find Things
|
||
|
||
| I need to... | Look in |
|
||
|-------------------------------------|-------------------------------------------------|
|
||
| Add a keybinding | `command/keymap.rs` → `default_keymaps()` |
|
||
| Add a user command | `command/cmd/<submodule>.rs`, register in `registry.rs` |
|
||
| Add a state mutation | `ui/effect.rs` → implement `Effect` |
|
||
| Change formula eval | `model/types.rs` → `eval_formula` / `eval_expr` |
|
||
| Change cell storage / lookup | `model/cell.rs` → `DataStore` |
|
||
| Change category/item behavior | `model/category.rs` → `Category` |
|
||
| Change view axis logic | `view/types.rs` → `View` |
|
||
| Change grid layout | `view/layout.rs` → `GridLayout` |
|
||
| Change `.improv` format | `persistence/improv.pest` + `persistence/mod.rs` |
|
||
| Change number display | `format.rs` → `format_f64` |
|
||
| Change CLI args | `main.rs` (clap) |
|
||
| Change import logic | `import/wizard.rs` → `ImportPipeline` |
|
||
| Change frame layout | `draw.rs` → `draw()` |
|
||
| Change app state / modes | `ui/app.rs` → `App`, `AppMode` |
|
||
| Write a test for model logic | `model/types.rs` → `mod tests` / `formula_tests` |
|
||
| Write a test for a command | `command/cmd/<module>.rs` → colocated `mod tests`; helpers in `cmd/mod.rs::test_helpers` |
|
||
|
||
---
|
||
|
||
## Central Pattern: Cmd → Effect
|
||
|
||
```
|
||
keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
|
||
(pure, read-only ctx) (list of mutations) (only mutation site)
|
||
```
|
||
|
||
```rust
|
||
pub trait Cmd: Debug + Send + Sync {
|
||
fn name(&self) -> &'static str;
|
||
fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
|
||
}
|
||
|
||
pub trait Effect: Debug {
|
||
fn apply(&self, app: &mut App);
|
||
fn changes_mode(&self) -> bool { false } // override when Effect swaps AppMode
|
||
}
|
||
```
|
||
|
||
`CmdContext` (see `command/cmd/core.rs`) holds immutable refs to `Model`, `GridLayout`, `AppMode`, cursor/offsets, buffers, yanked cell, key code, expanded cats, and panel/tile cursors.
|
||
|
||
**Add a command**: implement `Cmd` in a `command/cmd/` submodule, register in `registry.rs`, bind in `default_keymaps()`. Simple wrappers go through the `effect_cmd!` macro in `effect_cmds.rs`.
|
||
|
||
**Add an effect**: implement `Effect` in `ui/effect.rs`, add a constructor fn if it helps composition.
|
||
|
||
---
|
||
|
||
## Key Types (skim; read the source for fields/methods)
|
||
|
||
**Model** (`model/types.rs`): `categories: IndexMap<String, Category>`, `data: DataStore`, `formulas: Vec<Formula>`, `views: IndexMap<String, View>`, `active_view`, `measure_agg`. `MAX_CATEGORIES = 12` (regular only). Virtual categories `_Index`, `_Dim`, `_Measure` always exist. Use `regular_category_names()` for user-facing pickers.
|
||
|
||
**CellKey** (`model/cell.rs`): `Vec<(cat, item)>` — **always sorted by category**. Build with `CellKey::new(coords)` / `with(cat, item)` / `without(cat)`. Never construct the inner `Vec` directly.
|
||
|
||
**CellValue**: `Number(f64) | Text(String) | Error(String)`. Errors surface from formula eval (circular, div/0, etc.).
|
||
|
||
**DataStore** (`model/cell.rs`): interned symbols + secondary index `(Symbol, Symbol) → {InternedKey}`. Hot-path lookups go through `matching_values` / `matching_cells`.
|
||
|
||
**Formula AST** (`improvise-formula/src/ast.rs`):
|
||
- `Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }`
|
||
- `BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }`
|
||
- `AggFunc { Sum, Avg, Min, Max, Count }`
|
||
- `Formula { raw, target, target_category, expr, filter }`
|
||
|
||
**View** (`view/types.rs`): `category_axes: IndexMap<cat, Axis>`, page selections, hidden items, collapsed groups, `number_format`, `prune_empty`. `Axis { Row, Column, Page, None }`. `cycle_axis` rotates Row → Column → Page → None.
|
||
|
||
**GridLayout** (`view/layout.rs`): pure function of `Model + View`. `cell_key(r,c)`, `cell_value(r,c)`, `drill_records(r,c)`. **Records mode** auto-detects when `_Index` is on Row and `_Dim` is on Column.
|
||
|
||
**AppMode** (`ui/app.rs`): 15 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, CommandMode, ImportWizard, Help, Quit). `SearchMode` is Normal + `search_mode: bool`, not its own variant.
|
||
|
||
**Keymap** (`command/keymap.rs`): `Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }`. Lookup fallback: `exact(key,mods) → Char(NONE) → AnyChar → Any → parent`. 14 mode keymaps built by `KeymapSet::default_keymaps()`; mode resolved via `ModeKey::from_app_mode()`.
|
||
|
||
---
|
||
|
||
## `.improv` File Format
|
||
|
||
Plain-text, markdown-like, defined by `persistence/improv.pest`. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via `pest_meta`).
|
||
|
||
```
|
||
v2025-04-09
|
||
# Model Name
|
||
Initial View: Default
|
||
|
||
## View: Default
|
||
Region: row
|
||
Measure: column
|
||
|Time Period|: page, Q1
|
||
hidden: Region/Internal
|
||
collapsed: |Time Period|/|2024|
|
||
format: ,.2f
|
||
|
||
## Formulas
|
||
- Profit = Revenue - Cost # defaults to [_Measure]
|
||
- Tax = Revenue * 0.1 [CustomCat]
|
||
|
||
## Category: Region
|
||
- North, South, East, West # bare items, comma-separated
|
||
- Coastal_East[Coastal] # grouped item, one per line
|
||
> Coastal # group definition
|
||
|
||
## Data
|
||
Region=East, Measure=Revenue = 1200
|
||
Region=West, Measure=Revenue = |pending| # pipe-quoted text
|
||
```
|
||
|
||
- **Name quoting**: bare `[A-Za-z_][A-Za-z0-9_-]*`, else CL-style `|…|` with escapes `\|`, `\\`, `\n`. Same convention in the formula tokenizer.
|
||
- **Write order**: Views → Formulas → Categories → Data. Parser tolerates any order.
|
||
- **Gzip**: `.improv.gz` (same text, gzipped).
|
||
- **Legacy JSON**: auto-detected by leading `{`; never written.
|
||
- **Virtual categories**: `_Index`/`_Dim` never persist; `_Measure` persists only non-formula items (formula targets are rebuilt from the `## Formulas` section).
|
||
|
||
---
|
||
|
||
## Gotchas (read before editing)
|
||
|
||
1. **Commands never mutate.** `&CmdContext` is read-only; return `Vec<Box<dyn Effect>>`. If you want to touch `&mut App`, add or extend an Effect.
|
||
2. **`CellKey` is always sorted.** Use the constructors; equality and hashing rely on canonical form.
|
||
3. **No `Model::add_item`.** Go through the category: `m.category_mut("Region").unwrap().add_item("East")`.
|
||
4. **Virtual categories.** `_Measure` items = explicit items ∪ formula targets. Use `measure_item_names()` / `effective_item_names("_Measure")`. `add_formula` does *not* add items to `_Measure`. Formula-target lookup also falls back into `_Measure` via `find_item_category`.
|
||
5. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full `f64`. Never feed a rounded value back into eval.
|
||
6. **Formula eval is fixed-point.** `recompute_formulas(none_cats)` iterates until stable, bounded by `MAX_EVAL_DEPTH`; circular refs converge to `CellValue::Error("circular")`. `App::new` runs it before the first frame so formula cells render on startup.
|
||
7. **Drill into formula cells** strips the `_Measure` coordinate when it names a formula target, so `matching_cells` returns the raw records that feed the formula instead of returning empty.
|
||
8. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the mode; Normal + `search_mode=true` maps to `SearchMode`. Minibuffer modes bind Enter/Esc as `Binding::Sequence` so `clear-buffer` fires alongside commit/cancel.
|
||
9. **`effect_cmd!` macro** wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written `Cmd` once there's real decision logic.
|
||
10. **Float equality inside formula eval is currently mixed** — `=`/`!=` use a `1e-10` epsilon but the div-by-zero guard checks `rv == 0.0` exactly. Don't assume one or the other; tracked for cleanup.
|
||
11. **`IndexMap`** preserves insertion order for categories, views, and items. Persistence relies on this.
|
||
12. **Commit paths for cell edits** split across `commit_cell_value` → `commit_regular_cell_value` / `commit_plain_records_edit` in `command/cmd/commit.rs`. Keep the synthetic-records branch in sync with the plain branch when changing behavior.
|
||
|
||
---
|
||
|
||
## File Inventory
|
||
|
||
Line counts are static; test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted.
|
||
|
||
### `improvise-core` (`crates/improvise-core/src/`)
|
||
```
|
||
model/types.rs 2062 / 70t Model, formula eval (fixed-point), CRUD
|
||
model/cell.rs 650 / 28t CellKey (sorted), CellValue, DataStore (interned + index)
|
||
model/category.rs 222 / 6t Category, Item, Group, CategoryKind
|
||
model/symbol.rs 79 / 3t Symbol interning
|
||
view/layout.rs 1140 / 24t GridLayout, drill, records mode
|
||
view/types.rs 531 / 28t View config (axes, pages, hidden, collapsed, format)
|
||
view/axis.rs 21 Axis enum
|
||
workbook.rs 259 / 11t Workbook: Model + cross-view ops
|
||
format.rs 229 / 29t format_f64, parse_number_format (display only)
|
||
```
|
||
|
||
### `improvise-formula` (`crates/improvise-formula/src/`)
|
||
```
|
||
parser.rs 776 / 65t pest grammar + tokenizer → Formula AST
|
||
ast.rs 77 Expr, BinOp, AggFunc, Formula, Filter
|
||
```
|
||
|
||
### `improvise-io` (`crates/improvise-io/src/`)
|
||
```
|
||
persistence/improv.pest 124 PEG grammar — single source of truth
|
||
persistence/mod.rs 2410 / 83t save/load/gzip/legacy-JSON, CSV export
|
||
import/wizard.rs 1117 / 38t ImportPipeline + ImportWizard
|
||
import/analyzer.rs 292 / 9t Field kind detection (Category/Measure/Time/Skip)
|
||
import/csv_parser.rs 300 / 8t CSV parsing, multi-file merge
|
||
```
|
||
|
||
### Command layer (`src/command/`)
|
||
```
|
||
cmd/core.rs 297 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers
|
||
cmd/registry.rs 586 / 0t default_registry() — all registrations (no tests yet)
|
||
cmd/navigation.rs 475 / 10t Move, EnterAdvance, Page*
|
||
cmd/cell.rs 198 / 6t ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
|
||
cmd/commit.rs 330 / 7t CommitFormula, CommitCategoryAdd/ItemAdd, CommitExport
|
||
cmd/effect_cmds.rs 437 / 5t effect_cmd! macro, 25+ simple wrappers
|
||
cmd/grid.rs 409 / 7t ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
|
||
cmd/mode.rs 308 / 8t EnterMode, Quit, EditOrDrill, EnterTileSelect
|
||
cmd/panel.rs 587 / 13t Panel toggle/cycle/cursor, formula/category/view panels
|
||
cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
|
||
cmd/text_buffer.rs 256 / 7t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
|
||
cmd/tile.rs 160 / 5t MoveTileCursor, TileAxisOp
|
||
keymap.rs 1066 / 22t KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
|
||
parse.rs 236 / 19t Script/command-line parser (prefix syntax)
|
||
```
|
||
|
||
### UI, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`)
|
||
```
|
||
ui/effect.rs 942 / 41t Effect trait, 50+ effect types
|
||
ui/app.rs 914 / 30t App state, AppMode (15), handle_key, autosave
|
||
ui/grid.rs 1036 / 13t GridWidget (ratatui), column widths
|
||
ui/help.rs 617 5-page help overlay (render only)
|
||
ui/import_wizard_ui.rs 347 Import wizard rendering
|
||
ui/cat_tree.rs 165 / 6t Category tree flattener for panel
|
||
draw.rs 400 TUI event loop, frame composition
|
||
main.rs 391 CLI entry (clap): open, import, cmd, script
|
||
# other ui/*.rs are small panel renderers — skip unless changing layout/style
|
||
```
|
||
|
||
### Examples
|
||
```
|
||
examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta)
|
||
examples/pretty-print.rs Parse stdin, print formatted .improv
|
||
```
|
||
|
||
### Context docs
|
||
```
|
||
context/design-principles.md Architectural principles & testing doctrine
|
||
context/plan.md Show HN launch plan
|
||
context/repo-map.md This file
|
||
docs/design-notes.md Product vision & non-goals
|
||
```
|
||
|
||
---
|
||
|
||
## CLI
|
||
|
||
```
|
||
improvise [model.improv] # open TUI (default)
|
||
improvise import data.csv [--no-wizard] [-o out] # import CSV/JSON
|
||
improvise cmd 'add-cat Region' -f model.improv # headless command(s)
|
||
improvise script setup.txt -f model.improv # run script file
|
||
```
|
||
|
||
Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`.
|
||
|
||
---
|
||
|
||
## Testing — the short version
|
||
|
||
Full guidance lives in `context/design-principles.md` §6. Quick reminders:
|
||
|
||
- Suite runs in <2s. Don't let that drift.
|
||
- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed.
|
||
- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability.
|
||
- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Document the bug in the test's doc-comment (see `model/types.rs::formula_tests`).
|
||
- Coverage target ~80% line/branch on logic code; skip ratatui render paths.
|
||
|
||
---
|
||
|
||
## Key dependencies
|
||
|
||
ratatui 0.30, crossterm 0.28, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
|