chore: update repo-map
This commit is contained in:
Ed L
@ -1,292 +1,97 @@
|
||||
# Repository Map (LLM Reference)
|
||||
|
||||
Terminal pivot-table modeling app. Rust, Ratatui TUI, command/effect architecture.
|
||||
Cargo workspace, Apache-2.0, edition 2024. Root package `improvise` v0.1.0-rc2.
|
||||
Library + binary crate: `src/lib.rs` re-exports public modules (many from sub-crates), `src/main.rs` is the CLI entry.
|
||||
Sub-crates live under `crates/`:
|
||||
- `crates/improvise-core/` — pure-data core: `Model`, `View`, `Workbook`, and number formatting. Depends on `improvise-formula`. Re-exported from the main crate so `crate::model`, `crate::view`, `crate::workbook`, `crate::format` still resolve everywhere. Has no awareness of UI, I/O, or commands — builds standalone via `cargo build -p improvise-core`.
|
||||
- `crates/improvise-formula/` — formula parser, AST (`Expr`, `BinOp`, `AggFunc`, `Formula`, `Filter`), `parse_formula`. Re-exported as `crate::formula` from the main crate via `pub use improvise_formula as formula;`.
|
||||
- `crates/improvise-io/` — `.improv` persistence (parse/format, save/load, CSV export) and import pipeline (CSV/JSON wizard, field analyzer). Depends on `improvise-core` and `improvise-formula`; has no UI or command code. Re-exported from the main crate so `crate::persistence` and `crate::import` still resolve everywhere. Builds standalone via `cargo build -p improvise-io`.
|
||||
Terminal pivot-table app. Rust 2024, Ratatui TUI, command/effect architecture.
|
||||
Apache-2.0. Root binary+lib `improvise`; sub-crates under `crates/`:
|
||||
|
||||
- `improvise-core` — `Model`, `View`, `Workbook`, number formatting. No UI/IO.
|
||||
- `improvise-formula` — formula parser, AST, `parse_formula`.
|
||||
- `improvise-io` — `.improv` save/load, CSV/JSON import. No UI/commands.
|
||||
|
||||
The main crate re-exports each as `crate::{model, view, workbook, format, formula, persistence, import}` so consumer paths stay stable when crates shuffle.
|
||||
|
||||
Architectural intent lives in `context/design-principles.md` — read that for the "why". This doc is the "where".
|
||||
|
||||
---
|
||||
|
||||
## How to Find Things
|
||||
|
||||
| I need to... | Look in |
|
||||
|---------------------------------------|----------------------------------------------|
|
||||
| Add a new keybinding | `command/keymap.rs` → `default_keymaps()` |
|
||||
| Add a new user-facing command | `command/cmd/` → implement `Cmd` in the relevant submodule, register in `registry.rs` |
|
||||
| Add a new state mutation | `ui/effect.rs` → implement `Effect` |
|
||||
| Change formula evaluation | `model/types.rs` → `eval_formula()`, `eval_expr()` |
|
||||
| Change how cells are stored/queried | `model/cell.rs` → `DataStore` |
|
||||
| Change category/item behavior | `model/category.rs` → `Category` |
|
||||
| Change view axis logic | `view/types.rs` → `View` |
|
||||
| Change grid layout computation | `view/layout.rs` → `GridLayout` |
|
||||
| Change .improv file format | `persistence/improv.pest` (grammar), `persistence/mod.rs` → `format_md()`, `parse_md()` |
|
||||
| Change number display formatting | `format.rs` → `format_f64()` |
|
||||
| Change CLI arguments | `main.rs` → clap structs |
|
||||
| Change import wizard logic | `import/wizard.rs` → `ImportPipeline` |
|
||||
| Change grid rendering | `ui/grid.rs` → `GridWidget` |
|
||||
| Change TUI frame layout | `draw.rs` → `draw()` |
|
||||
| Change app state / mode transitions | `ui/app.rs` → `App`, `AppMode` |
|
||||
| Write a test for model logic | `model/types.rs` → `mod tests` / `mod formula_tests` |
|
||||
| Write a test for a command | `command/cmd/<module>.rs` → colocated `mod tests` |
|
||||
| I need to... | Look in |
|
||||
|-------------------------------------|-------------------------------------------------|
|
||||
| Add a keybinding | `command/keymap.rs` → `default_keymaps()` |
|
||||
| Add a user command | `command/cmd/<submodule>.rs`, register in `registry.rs` |
|
||||
| Add a state mutation | `ui/effect.rs` → implement `Effect` |
|
||||
| Change formula eval | `model/types.rs` → `eval_formula` / `eval_expr` |
|
||||
| Change cell storage / lookup | `model/cell.rs` → `DataStore` |
|
||||
| Change category/item behavior | `model/category.rs` → `Category` |
|
||||
| Change view axis logic | `view/types.rs` → `View` |
|
||||
| Change grid layout | `view/layout.rs` → `GridLayout` |
|
||||
| Change `.improv` format | `persistence/improv.pest` + `persistence/mod.rs` |
|
||||
| Change number display | `format.rs` → `format_f64` |
|
||||
| Change CLI args | `main.rs` (clap) |
|
||||
| Change import logic | `import/wizard.rs` → `ImportPipeline` |
|
||||
| Change frame layout | `draw.rs` → `draw()` |
|
||||
| Change app state / modes | `ui/app.rs` → `App`, `AppMode` |
|
||||
| Write a test for model logic | `model/types.rs` → `mod tests` / `formula_tests` |
|
||||
| Write a test for a command | `command/cmd/<module>.rs` → colocated `mod tests`; helpers in `cmd/mod.rs::test_helpers` |
|
||||
|
||||
---
|
||||
|
||||
## Core Types and Traits
|
||||
|
||||
### Command/Effect Pipeline (the central architecture pattern)
|
||||
## Central Pattern: Cmd → Effect
|
||||
|
||||
```
|
||||
User keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
|
||||
(immutable) (pure, read-only) (state mutations)
|
||||
keypress → Keymap lookup → Cmd::execute(&CmdContext) → Vec<Box<dyn Effect>> → Effect::apply(&mut App)
|
||||
(pure, read-only ctx) (list of mutations) (only mutation site)
|
||||
```
|
||||
|
||||
```rust
|
||||
// src/command/cmd/core.rs
|
||||
pub trait Cmd: Debug + Send + Sync {
|
||||
fn name(&self) -> &'static str;
|
||||
fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>>;
|
||||
}
|
||||
|
||||
pub struct CmdContext<'a> {
|
||||
pub model: &'a Model, // immutable
|
||||
pub layout: &'a GridLayout, // immutable
|
||||
pub registry: &'a CmdRegistry,
|
||||
pub mode: &'a AppMode,
|
||||
pub selected: (usize, usize), // (row, col) cursor
|
||||
pub row_offset: usize,
|
||||
pub col_offset: usize,
|
||||
pub search_query: &'a str,
|
||||
pub search_mode: bool,
|
||||
pub yanked: &'a Option<CellValue>,
|
||||
pub key_code: KeyCode, // the key that triggered this command
|
||||
pub buffers: &'a HashMap<String, String>,
|
||||
pub expanded_cats: &'a HashSet<String>,
|
||||
// panel cursors, tile cursor, visible dimensions...
|
||||
}
|
||||
|
||||
// src/ui/effect.rs
|
||||
pub trait Effect: Debug {
|
||||
fn apply(&self, app: &mut App);
|
||||
fn changes_mode(&self) -> bool { false } // override if effect changes AppMode
|
||||
fn changes_mode(&self) -> bool { false } // override when Effect swaps AppMode
|
||||
}
|
||||
```
|
||||
|
||||
**To add a command**: implement `Cmd` in the appropriate `command/cmd/` submodule, then register in `command/cmd/registry.rs`. Use the `effect_cmd!` macro (in `effect_cmds.rs`) for simple effect-wrapping commands. Bind it in `default_keymaps()`.
|
||||
`CmdContext` (see `command/cmd/core.rs`) holds immutable refs to `Model`, `GridLayout`, `AppMode`, cursor/offsets, buffers, yanked cell, key code, expanded cats, and panel/tile cursors.
|
||||
|
||||
**To add an effect**: implement `Effect` in `effect.rs`, add a constructor function.
|
||||
**Add a command**: implement `Cmd` in a `command/cmd/` submodule, register in `registry.rs`, bind in `default_keymaps()`. Simple wrappers go through the `effect_cmd!` macro in `effect_cmds.rs`.
|
||||
|
||||
### Data Model
|
||||
|
||||
```rust
|
||||
// src/model/types.rs
|
||||
pub struct Model {
|
||||
pub name: String,
|
||||
pub categories: IndexMap<String, Category>, // ordered
|
||||
pub data: DataStore,
|
||||
pub formulas: Vec<Formula>,
|
||||
pub views: IndexMap<String, View>,
|
||||
pub active_view: String,
|
||||
pub measure_agg: HashMap<String, AggFunc>, // per-measure aggregation override
|
||||
}
|
||||
// Key methods:
|
||||
// add_category(&mut self, name) -> Result<CategoryId> [max 12 regular]
|
||||
// category(&self, name) -> Option<&Category>
|
||||
// category_mut(&mut self, name) -> Option<&mut Category>
|
||||
// set_cell(&mut self, key: CellKey, value: CellValue)
|
||||
// evaluate(&self, key: &CellKey) -> Option<CellValue> [formulas + raw data]
|
||||
// evaluate_aggregated(&self, key, none_cats) -> Option<CellValue> [sums over hidden dims]
|
||||
// recompute_formulas(&mut self, none_cats) [fixed-point formula cache]
|
||||
// add_formula(&mut self, formula: Formula) [replaces same target+category]
|
||||
// remove_formula(&mut self, target, category)
|
||||
// measure_item_names(&self) -> Vec<String> [_Measure items + formula targets]
|
||||
// effective_item_names(&self, cat) -> Vec<String> [_Measure dynamic, others ordered_item_names]
|
||||
// category_names(&self) -> Vec<&str> [includes virtual]
|
||||
// regular_category_names(&self) -> Vec<&str> [excludes _Index, _Dim, _Measure]
|
||||
|
||||
const MAX_CATEGORIES: usize = 12; // virtual categories don't count
|
||||
```
|
||||
|
||||
```rust
|
||||
// src/model/cell.rs
|
||||
#[derive(Clone, PartialEq, Eq, Hash)]
|
||||
pub struct CellKey(pub Vec<(String, String)>); // always sorted by category name
|
||||
// CellKey::new(coords) — sorts on construction, enforcing canonical form
|
||||
// CellKey::with(cat, item) -> Self — returns new key with coord added/replaced
|
||||
// CellKey::without(cat) -> Self — returns new key with coord removed
|
||||
// CellKey::get(cat) -> Option<&str>
|
||||
|
||||
#[derive(Clone, PartialEq)]
|
||||
pub enum CellValue {
|
||||
Number(f64),
|
||||
Text(String),
|
||||
Error(String), // formula evaluation error (circular ref, div/0, etc.)
|
||||
}
|
||||
// CellValue::as_f64() -> Option<f64>
|
||||
// CellValue::is_error() -> bool
|
||||
|
||||
pub struct DataStore {
|
||||
cells: HashMap<InternedKey, CellValue>,
|
||||
pub symbols: SymbolTable,
|
||||
index: HashMap<(Symbol, Symbol), HashSet<InternedKey>>, // secondary index
|
||||
}
|
||||
// DataStore::set(&mut self, key: &CellKey, value: CellValue)
|
||||
// DataStore::get(&self, key: &CellKey) -> Option<&CellValue>
|
||||
// DataStore::matching_values(&self, partial: &[(String,String)]) -> Vec<CellValue>
|
||||
// DataStore::matching_cells(&self, partial) -> Vec<(CellKey, CellValue)>
|
||||
```
|
||||
|
||||
```rust
|
||||
// src/model/category.rs
|
||||
pub struct Category {
|
||||
pub id: CategoryId, // usize
|
||||
pub name: String,
|
||||
pub kind: CategoryKind,
|
||||
pub items: IndexMap<String, Item>, // ordered
|
||||
pub groups: IndexMap<String, Group>,
|
||||
next_item_id: ItemId, // private, auto-increment
|
||||
}
|
||||
// Category::add_item(&mut self, name) -> ItemId [deduplicates by name]
|
||||
// Category::ordered_item_names(&self) -> Vec<&str> [respects group order]
|
||||
|
||||
pub enum CategoryKind { Regular, VirtualIndex, VirtualDim, VirtualMeasure, Label }
|
||||
```
|
||||
|
||||
### Formula System
|
||||
|
||||
```rust
|
||||
// crates/improvise-formula/src/ast.rs
|
||||
pub enum Expr {
|
||||
Number(f64),
|
||||
Ref(String), // reference to an item name
|
||||
BinOp(BinOp, Box<Expr>, Box<Expr>),
|
||||
UnaryMinus(Box<Expr>),
|
||||
Agg(AggFunc, Box<Expr>, Option<Filter>),
|
||||
If(Box<Expr>, Box<Expr>, Box<Expr>),
|
||||
}
|
||||
pub enum BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }
|
||||
pub enum AggFunc { Sum, Avg, Min, Max, Count }
|
||||
pub struct Formula {
|
||||
pub raw: String, // "Profit = Revenue - Cost"
|
||||
pub target: String, // "Profit"
|
||||
pub target_category: String, // "Measure"
|
||||
pub expr: Expr,
|
||||
pub filter: Option<Filter>, // WHERE clause
|
||||
}
|
||||
|
||||
// crates/improvise-formula/src/parser.rs
|
||||
pub fn parse_formula(raw: &str, target_category: &str) -> Result<Formula>
|
||||
```
|
||||
|
||||
Formula evaluation is in `model/types.rs` → `eval_formula()` / `eval_expr()`. Operates at full f64 precision. Display rounding in `format.rs` is view-only.
|
||||
|
||||
### View and Layout
|
||||
|
||||
```rust
|
||||
// src/view/axis.rs
|
||||
pub enum Axis { Row, Column, Page, None }
|
||||
|
||||
// src/view/types.rs
|
||||
pub struct View {
|
||||
pub name: String,
|
||||
pub category_axes: IndexMap<String, Axis>,
|
||||
pub page_selections: HashMap<String, String>,
|
||||
pub hidden_items: HashMap<String, HashSet<String>>,
|
||||
pub collapsed_groups: HashMap<String, HashSet<String>>,
|
||||
pub number_format: String, // e.g. ",.0" or ",.2f"
|
||||
pub prune_empty: bool,
|
||||
// scroll/selection state...
|
||||
}
|
||||
// View::set_axis(&mut self, cat, axis)
|
||||
// View::axis_of(&self, cat) -> Axis
|
||||
// View::cycle_axis(&mut self, cat) [Row→Column→Page→None→Row]
|
||||
// View::transpose(&mut self) [swap Row↔Column]
|
||||
// View::categories_on(&self, axis) -> Vec<&str>
|
||||
// View::on_category_added(&mut self, cat) [auto-assigns axis]
|
||||
|
||||
// src/view/layout.rs
|
||||
pub struct GridLayout { /* computed from Model + View */ }
|
||||
// GridLayout::new(model, view) -> Self
|
||||
// GridLayout::cell_key(row, col) -> Option<CellKey>
|
||||
// GridLayout::cell_value(row, col) -> Option<CellValue>
|
||||
// GridLayout::row_label(row) -> &str
|
||||
// GridLayout::col_label(col) -> &str
|
||||
// GridLayout::drill_records(row, col) -> Vec<(CellKey, CellValue)>
|
||||
// Records mode: auto-detected when _Index on Row + _Dim on Column
|
||||
```
|
||||
|
||||
### App State
|
||||
|
||||
```rust
|
||||
// src/ui/app.rs
|
||||
pub enum AppMode {
|
||||
Normal,
|
||||
Editing { minibuf: MinibufferConfig },
|
||||
FormulaEdit { minibuf: MinibufferConfig },
|
||||
FormulaPanel,
|
||||
CategoryPanel,
|
||||
ViewPanel,
|
||||
TileSelect,
|
||||
CategoryAdd { minibuf: MinibufferConfig },
|
||||
ItemAdd { minibuf: MinibufferConfig },
|
||||
ExportPrompt { minibuf: MinibufferConfig },
|
||||
CommandMode { minibuf: MinibufferConfig },
|
||||
ImportWizard,
|
||||
Help,
|
||||
Quit,
|
||||
}
|
||||
// Note: SearchMode is Normal + search_mode:bool flag, not a separate variant.
|
||||
|
||||
pub struct App {
|
||||
pub model: Model,
|
||||
pub mode: AppMode,
|
||||
pub file_path: Option<PathBuf>,
|
||||
pub dirty: bool,
|
||||
pub help_page: usize,
|
||||
pub transient_keymap: Option<Arc<Keymap>>, // for prefix keys
|
||||
// layout cache, drill_state, wizard, buffers, panel cursors, etc.
|
||||
}
|
||||
// App::handle_key(&mut self, KeyEvent) -> Result<()> [main input dispatch]
|
||||
// App::rebuild_layout(&mut self)
|
||||
// App::is_empty_model(&self) -> bool [true when only virtual categories exist]
|
||||
```
|
||||
|
||||
### Keymap System
|
||||
|
||||
```rust
|
||||
// src/command/keymap.rs
|
||||
pub enum KeyPattern { Key(KeyCode, KeyModifiers), AnyChar, Any }
|
||||
pub enum Binding {
|
||||
Cmd { name: &'static str, args: Vec<String> },
|
||||
Prefix(Arc<Keymap>), // Emacs-style sub-keymap
|
||||
Sequence(Vec<(&'static str, Vec<String>)>), // multi-command chain
|
||||
}
|
||||
pub enum ModeKey {
|
||||
Normal, Help, FormulaPanel, CategoryPanel, ViewPanel, TileSelect,
|
||||
Editing, FormulaEdit, CategoryAdd, ItemAdd, ExportPrompt, CommandMode,
|
||||
SearchMode, ImportWizard,
|
||||
}
|
||||
|
||||
// Keymap::with_parent(parent: Arc<Keymap>) -> Self [Emacs-style inheritance]
|
||||
// Keymap::lookup(&self, key, mods) -> Option<&Binding>
|
||||
// Fallback chain: exact(key,mods) → Char with NONE mods → AnyChar → Any → parent
|
||||
// Minibuffer modes: Enter and Esc use Binding::Sequence to include clear-buffer
|
||||
|
||||
// KeymapSet::default_keymaps() -> Self [builds all 14 mode keymaps]
|
||||
// KeymapSet::dispatch(&self, ctx, key, mods) -> Option<Vec<Box<dyn Effect>>>
|
||||
```
|
||||
**Add an effect**: implement `Effect` in `ui/effect.rs`, add a constructor fn if it helps composition.
|
||||
|
||||
---
|
||||
|
||||
## File Format (.improv)
|
||||
## Key Types (skim; read the source for fields/methods)
|
||||
|
||||
Plain-text markdown-like, defined by a PEG grammar (`persistence/improv.pest`).
|
||||
Parsed by pest; the grammar is the single source of truth for both the parser
|
||||
and the grammar-walking test generator.
|
||||
**Model** (`model/types.rs`): `categories: IndexMap<String, Category>`, `data: DataStore`, `formulas: Vec<Formula>`, `views: IndexMap<String, View>`, `active_view`, `measure_agg`. `MAX_CATEGORIES = 12` (regular only). Virtual categories `_Index`, `_Dim`, `_Measure` always exist. Use `regular_category_names()` for user-facing pickers.
|
||||
|
||||
**Not JSON** (JSON is legacy, auto-detected by `{` prefix).
|
||||
**CellKey** (`model/cell.rs`): `Vec<(cat, item)>` — **always sorted by category**. Build with `CellKey::new(coords)` / `with(cat, item)` / `without(cat)`. Never construct the inner `Vec` directly.
|
||||
|
||||
**CellValue**: `Number(f64) | Text(String) | Error(String)`. Errors surface from formula eval (circular, div/0, etc.).
|
||||
|
||||
**DataStore** (`model/cell.rs`): interned symbols + secondary index `(Symbol, Symbol) → {InternedKey}`. Hot-path lookups go through `matching_values` / `matching_cells`.
|
||||
|
||||
**Formula AST** (`improvise-formula/src/ast.rs`):
|
||||
- `Expr { Number, Ref, BinOp, UnaryMinus, Agg, If }`
|
||||
- `BinOp { Add, Sub, Mul, Div, Pow, Eq, Ne, Lt, Gt, Le, Ge }`
|
||||
- `AggFunc { Sum, Avg, Min, Max, Count }`
|
||||
- `Formula { raw, target, target_category, expr, filter }`
|
||||
|
||||
**View** (`view/types.rs`): `category_axes: IndexMap<cat, Axis>`, page selections, hidden items, collapsed groups, `number_format`, `prune_empty`. `Axis { Row, Column, Page, None }`. `cycle_axis` rotates Row → Column → Page → None.
|
||||
|
||||
**GridLayout** (`view/layout.rs`): pure function of `Model + View`. `cell_key(r,c)`, `cell_value(r,c)`, `drill_records(r,c)`. **Records mode** auto-detects when `_Index` is on Row and `_Dim` is on Column.
|
||||
|
||||
**AppMode** (`ui/app.rs`): 15 variants (Normal, Editing, FormulaEdit, FormulaPanel, CategoryPanel, ViewPanel, TileSelect, CategoryAdd, ItemAdd, ExportPrompt, CommandMode, ImportWizard, Help, Quit). `SearchMode` is Normal + `search_mode: bool`, not its own variant.
|
||||
|
||||
**Keymap** (`command/keymap.rs`): `Binding { Cmd | Prefix(Arc<Keymap>) | Sequence(Vec<…>) }`. Lookup fallback: `exact(key,mods) → Char(NONE) → AnyChar → Any → parent`. 14 mode keymaps built by `KeymapSet::default_keymaps()`; mode resolved via `ModeKey::from_app_mode()`.
|
||||
|
||||
---
|
||||
|
||||
## `.improv` File Format
|
||||
|
||||
Plain-text, markdown-like, defined by `persistence/improv.pest`. Parsed by pest — the grammar is the single source of truth (the grammar-walking test generator reads it via `pest_meta`).
|
||||
|
||||
```
|
||||
v2025-04-09
|
||||
@ -296,49 +101,126 @@ Initial View: Default
|
||||
## View: Default
|
||||
Region: row
|
||||
Measure: column
|
||||
|Time Period|: page, Q1 ← pipe-quoted name, page with selection
|
||||
|Time Period|: page, Q1
|
||||
hidden: Region/Internal
|
||||
collapsed: |Time Period|/|2024|
|
||||
format: ,.2f
|
||||
|
||||
## Formulas
|
||||
- Profit = Revenue - Cost ← defaults to [_Measure]
|
||||
- Tax = Revenue * 0.1 [CustomCat] ← explicit [TargetCategory] for non-_Measure
|
||||
- Profit = Revenue - Cost # defaults to [_Measure]
|
||||
- Tax = Revenue * 0.1 [CustomCat]
|
||||
|
||||
## Category: Region
|
||||
- North, South, East, West ← bare items, comma-separated
|
||||
- Coastal_East[Coastal] ← grouped item (one per line)
|
||||
- Coastal_West[Coastal]
|
||||
> Coastal ← group definition
|
||||
|
||||
## Category: Measure
|
||||
- Revenue, Cost, Profit
|
||||
- North, South, East, West # bare items, comma-separated
|
||||
- Coastal_East[Coastal] # grouped item, one per line
|
||||
> Coastal # group definition
|
||||
|
||||
## Data
|
||||
Region=East, Measure=Revenue = 1200
|
||||
Region=East, Measure=Cost = 800
|
||||
Region=West, Measure=Revenue = |pending| ← pipe-quoted text value
|
||||
Region=West, Measure=Revenue = |pending| # pipe-quoted text
|
||||
```
|
||||
|
||||
### Name quoting
|
||||
- **Name quoting**: bare `[A-Za-z_][A-Za-z0-9_-]*`, else CL-style `|…|` with escapes `\|`, `\\`, `\n`. Same convention in the formula tokenizer.
|
||||
- **Write order**: Views → Formulas → Categories → Data. Parser tolerates any order.
|
||||
- **Gzip**: `.improv.gz` (same text, gzipped).
|
||||
- **Legacy JSON**: auto-detected by leading `{`; never written.
|
||||
- **Virtual categories**: `_Index`/`_Dim` never persist; `_Measure` persists only non-formula items (formula targets are rebuilt from the `## Formulas` section).
|
||||
|
||||
Bare names match `[A-Za-z_][A-Za-z0-9_-]*`. Everything else uses CL-style
|
||||
pipe quoting: `|Income, Gross|`, `|2025|`, `|Name with spaces|`.
|
||||
Escapes inside pipes: `\|` (literal pipe), `\\` (backslash), `\n` (newline).
|
||||
---
|
||||
|
||||
### Section order
|
||||
## Gotchas (read before editing)
|
||||
|
||||
`format_md` writes Views → Formulas → Categories → Data (smallest to largest).
|
||||
The parser accepts sections in any order.
|
||||
1. **Commands never mutate.** `&CmdContext` is read-only; return `Vec<Box<dyn Effect>>`. If you want to touch `&mut App`, add or extend an Effect.
|
||||
2. **`CellKey` is always sorted.** Use the constructors; equality and hashing rely on canonical form.
|
||||
3. **No `Model::add_item`.** Go through the category: `m.category_mut("Region").unwrap().add_item("East")`.
|
||||
4. **Virtual categories.** `_Measure` items = explicit items ∪ formula targets. Use `measure_item_names()` / `effective_item_names("_Measure")`. `add_formula` does *not* add items to `_Measure`. Formula-target lookup also falls back into `_Measure` via `find_item_category`.
|
||||
5. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full `f64`. Never feed a rounded value back into eval.
|
||||
6. **Formula eval is fixed-point.** `recompute_formulas(none_cats)` iterates until stable, bounded by `MAX_EVAL_DEPTH`; circular refs converge to `CellValue::Error("circular")`. `App::new` runs it before the first frame so formula cells render on startup.
|
||||
7. **Drill into formula cells** strips the `_Measure` coordinate when it names a formula target, so `matching_cells` returns the raw records that feed the formula instead of returning empty.
|
||||
8. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the mode; Normal + `search_mode=true` maps to `SearchMode`. Minibuffer modes bind Enter/Esc as `Binding::Sequence` so `clear-buffer` fires alongside commit/cancel.
|
||||
9. **`effect_cmd!` macro** wraps a pre-built effect as a parseable command. Use it for simple wrappers; reach for a hand-written `Cmd` once there's real decision logic.
|
||||
10. **Float equality inside formula eval is currently mixed** — `=`/`!=` use a `1e-10` epsilon but the div-by-zero guard checks `rv == 0.0` exactly. Don't assume one or the other; tracked for cleanup.
|
||||
11. **`IndexMap`** preserves insertion order for categories, views, and items. Persistence relies on this.
|
||||
12. **Commit paths for cell edits** split across `commit_cell_value` → `commit_regular_cell_value` / `commit_plain_records_edit` in `command/cmd/commit.rs`. Keep the synthetic-records branch in sync with the plain branch when changing behavior.
|
||||
|
||||
### Key design choices
|
||||
---
|
||||
|
||||
- Version line is exact match (`v2025-04-09`) — grammar enforces valid versions only.
|
||||
- `Initial View:` is a top-level header, not embedded in view sections.
|
||||
- Text cell values are always pipe-quoted to distinguish from numbers.
|
||||
- Bare items are comma-separated on one line; grouped items get one line each.
|
||||
## File Inventory
|
||||
|
||||
Gzip variant: `.improv.gz` (same content, gzipped). Persistence code: `persistence/mod.rs`.
|
||||
Line counts are static; test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted.
|
||||
|
||||
### `improvise-core` (`crates/improvise-core/src/`)
|
||||
```
|
||||
model/types.rs 2062 / 70t Model, formula eval (fixed-point), CRUD
|
||||
model/cell.rs 650 / 28t CellKey (sorted), CellValue, DataStore (interned + index)
|
||||
model/category.rs 222 / 6t Category, Item, Group, CategoryKind
|
||||
model/symbol.rs 79 / 3t Symbol interning
|
||||
view/layout.rs 1140 / 24t GridLayout, drill, records mode
|
||||
view/types.rs 531 / 28t View config (axes, pages, hidden, collapsed, format)
|
||||
view/axis.rs 21 Axis enum
|
||||
workbook.rs 259 / 11t Workbook: Model + cross-view ops
|
||||
format.rs 229 / 29t format_f64, parse_number_format (display only)
|
||||
```
|
||||
|
||||
### `improvise-formula` (`crates/improvise-formula/src/`)
|
||||
```
|
||||
parser.rs 776 / 65t pest grammar + tokenizer → Formula AST
|
||||
ast.rs 77 Expr, BinOp, AggFunc, Formula, Filter
|
||||
```
|
||||
|
||||
### `improvise-io` (`crates/improvise-io/src/`)
|
||||
```
|
||||
persistence/improv.pest 124 PEG grammar — single source of truth
|
||||
persistence/mod.rs 2410 / 83t save/load/gzip/legacy-JSON, CSV export
|
||||
import/wizard.rs 1117 / 38t ImportPipeline + ImportWizard
|
||||
import/analyzer.rs 292 / 9t Field kind detection (Category/Measure/Time/Skip)
|
||||
import/csv_parser.rs 300 / 8t CSV parsing, multi-file merge
|
||||
```
|
||||
|
||||
### Command layer (`src/command/`)
|
||||
```
|
||||
cmd/core.rs 297 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers
|
||||
cmd/registry.rs 586 / 0t default_registry() — all registrations (no tests yet)
|
||||
cmd/navigation.rs 475 / 10t Move, EnterAdvance, Page*
|
||||
cmd/cell.rs 198 / 6t ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
|
||||
cmd/commit.rs 330 / 7t CommitFormula, CommitCategoryAdd/ItemAdd, CommitExport
|
||||
cmd/effect_cmds.rs 437 / 5t effect_cmd! macro, 25+ simple wrappers
|
||||
cmd/grid.rs 409 / 7t ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
|
||||
cmd/mode.rs 308 / 8t EnterMode, Quit, EditOrDrill, EnterTileSelect
|
||||
cmd/panel.rs 587 / 13t Panel toggle/cycle/cursor, formula/category/view panels
|
||||
cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
|
||||
cmd/text_buffer.rs 256 / 7t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
|
||||
cmd/tile.rs 160 / 5t MoveTileCursor, TileAxisOp
|
||||
keymap.rs 1066 / 22t KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
|
||||
parse.rs 236 / 19t Script/command-line parser (prefix syntax)
|
||||
```
|
||||
|
||||
### UI, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`)
|
||||
```
|
||||
ui/effect.rs 942 / 41t Effect trait, 50+ effect types
|
||||
ui/app.rs 914 / 30t App state, AppMode (15), handle_key, autosave
|
||||
ui/grid.rs 1036 / 13t GridWidget (ratatui), column widths
|
||||
ui/help.rs 617 5-page help overlay (render only)
|
||||
ui/import_wizard_ui.rs 347 Import wizard rendering
|
||||
ui/cat_tree.rs 165 / 6t Category tree flattener for panel
|
||||
draw.rs 400 TUI event loop, frame composition
|
||||
main.rs 391 CLI entry (clap): open, import, cmd, script
|
||||
# other ui/*.rs are small panel renderers — skip unless changing layout/style
|
||||
```
|
||||
|
||||
### Examples
|
||||
```
|
||||
examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta)
|
||||
examples/pretty-print.rs Parse stdin, print formatted .improv
|
||||
```
|
||||
|
||||
### Context docs
|
||||
```
|
||||
context/design-principles.md Architectural principles & testing doctrine
|
||||
context/plan.md Show HN launch plan
|
||||
context/repo-map.md This file
|
||||
docs/design-notes.md Product vision & non-goals
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@ -355,219 +237,18 @@ Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axi
|
||||
|
||||
---
|
||||
|
||||
## Key Dependencies
|
||||
## Testing — the short version
|
||||
|
||||
| Crate | Purpose |
|
||||
|-------|---------|
|
||||
| ratatui 0.30 | TUI framework |
|
||||
| crossterm 0.28 | Terminal backend |
|
||||
| clap 4.6 (derive) | CLI parsing |
|
||||
| serde + serde_json | Serialization |
|
||||
| indexmap 2 | Ordered maps (categories, views) |
|
||||
| anyhow | Error handling |
|
||||
| chrono 0.4 | Date parsing in import |
|
||||
| pest + pest_derive | PEG parser for .improv format |
|
||||
| flate2 | Gzip for .improv.gz |
|
||||
| csv | CSV parsing |
|
||||
| enum_dispatch | CLI subcommand dispatch |
|
||||
| **dev:** proptest, tempfile, pest_meta | Property testing, temp dirs, grammar AST for test generator |
|
||||
Full guidance lives in `context/design-principles.md` §6. Quick reminders:
|
||||
|
||||
- Suite runs in <2s. Don't let that drift.
|
||||
- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed.
|
||||
- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability.
|
||||
- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Document the bug in the test's doc-comment (see `model/types.rs::formula_tests`).
|
||||
- Coverage target ~80% line/branch on logic code; skip ratatui render paths.
|
||||
|
||||
---
|
||||
|
||||
## File Inventory
|
||||
## Key dependencies
|
||||
|
||||
Lines / tests / path — grouped by layer.
|
||||
|
||||
### Core crate layers (sub-crate `improvise-core` under `crates/`)
|
||||
All of `model/`, `view/`, `workbook.rs`, `format.rs` now live under
|
||||
`crates/improvise-core/src/`. Paths below are relative to that root.
|
||||
The main crate re-exports each as `crate::model`, `crate::view`,
|
||||
`crate::workbook`, `crate::format` via `src/lib.rs`, so every
|
||||
consumer path stays unchanged.
|
||||
|
||||
#### Model layer
|
||||
```
|
||||
2062 / 70t model/types.rs Model struct, formula eval, CRUD, MAX_CATEGORIES=12
|
||||
650 / 28t model/cell.rs CellKey (canonical sort), CellValue, DataStore (interned, sort_by_key)
|
||||
222 / 6t model/category.rs Category, Item, Group, CategoryKind
|
||||
79 / 3t model/symbol.rs Symbol interning (SymbolTable)
|
||||
6 / 0t model/mod.rs
|
||||
```
|
||||
|
||||
#### View layer
|
||||
```
|
||||
1140 / 24t view/layout.rs GridLayout (pure fn of Model+View), records mode, drill
|
||||
531 / 28t view/types.rs View config (axes, pages, hidden, collapsed, format), none_cats()
|
||||
21 / 0t view/axis.rs Axis enum {Row, Column, Page, None}
|
||||
7 / 0t view/mod.rs
|
||||
```
|
||||
|
||||
#### Workbook + format (top-level in improvise-core)
|
||||
```
|
||||
259 / 11t workbook.rs Workbook wraps Model + views; cross-slice category/view ops
|
||||
229 / 29t format.rs format_f64, parse_number_format (display-only rounding)
|
||||
12 / 0t lib.rs Module declarations + `pub use improvise_formula as formula;`
|
||||
```
|
||||
|
||||
### Formula layer (sub-crate `improvise-formula` under `crates/`)
|
||||
```
|
||||
776 / 65t crates/improvise-formula/src/parser.rs PEG parser (pest) → Formula AST
|
||||
77 / 0t crates/improvise-formula/src/ast.rs Expr, BinOp, AggFunc, Formula, Filter (data only)
|
||||
5 / 0t crates/improvise-formula/src/lib.rs
|
||||
```
|
||||
|
||||
### Command layer
|
||||
```
|
||||
command/cmd/ Cmd trait, CmdContext, CmdRegistry, 40+ commands
|
||||
297 / 2t core.rs Cmd trait, CmdContext, CmdRegistry, parse helpers
|
||||
586 / 0t registry.rs default_registry() — all command registrations
|
||||
475 / 10t navigation.rs Move, EnterAdvance, PageNext/Prev
|
||||
198 / 6t cell.rs ClearCell, YankCell, PasteCell, TransposeAxes, SaveCmd
|
||||
330 / 7t commit.rs CommitFormula, CommitCategoryAdd/ItemAdd, CommitExport
|
||||
437 / 5t effect_cmds.rs effect_cmd! macro, 25+ parseable effect-wrapper commands
|
||||
409 / 7t grid.rs ToggleGroup, ViewNavigate, DrillIntoCell, TogglePruneEmpty
|
||||
308 / 8t mode.rs EnterMode, Quit, EditOrDrill, EnterTileSelect, etc.
|
||||
587 / 13t panel.rs Panel toggle/cycle/cursor, formula/category/view panel cmds
|
||||
202 / 4t search.rs SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
|
||||
256 / 7t text_buffer.rs AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
|
||||
160 / 5t tile.rs MoveTileCursor, TileAxisOp
|
||||
121 / 0t mod.rs Module declarations, re-exports, test helpers
|
||||
1066 / 22t command/keymap.rs KeyPattern, Binding, Keymap, ModeKey, 14 mode keymaps
|
||||
236 / 19t command/parse.rs Script/command-line parser (prefix syntax)
|
||||
12 / 0t command/mod.rs
|
||||
```
|
||||
|
||||
### UI layer
|
||||
```
|
||||
942 / 41t ui/effect.rs Effect trait, 50+ effect types (all state mutations)
|
||||
914 / 30t ui/app.rs App state, AppMode (15 variants), handle_key, autosave
|
||||
1036 / 13t ui/grid.rs GridWidget (ratatui), col widths, rendering
|
||||
617 / 0t ui/help.rs 5-page help overlay, HELP_PAGE_COUNT=5
|
||||
347 / 0t ui/import_wizard_ui.rs Import wizard overlay rendering
|
||||
165 / 6t ui/cat_tree.rs Category tree flattener for panel
|
||||
113 / 0t ui/view_panel.rs View list panel
|
||||
107 / 0t ui/category_panel.rs Category tree panel
|
||||
95 / 0t ui/tile_bar.rs Tile bar (axis assignment tiles)
|
||||
87 / 0t ui/panel.rs Generic panel frame widget
|
||||
81 / 0t ui/formula_panel.rs Formula list panel
|
||||
67 / 0t ui/which_key.rs Prefix-key hint popup
|
||||
12 / 0t ui/mod.rs
|
||||
```
|
||||
|
||||
### I/O crate layers (sub-crate `improvise-io` under `crates/`)
|
||||
All of `persistence/` and `import/` now live under `crates/improvise-io/src/`.
|
||||
Paths below are relative to that root. The main crate re-exports each
|
||||
as `crate::persistence` and `crate::import` via `src/lib.rs`, so every
|
||||
consumer path stays unchanged.
|
||||
|
||||
#### Persistence
|
||||
```
|
||||
124 / 0t persistence/improv.pest PEG grammar — single source of truth for .improv format
|
||||
2410 / 83t persistence/mod.rs .improv save/load (pest parser + format + gzip + legacy JSON), export_csv
|
||||
```
|
||||
|
||||
#### Import
|
||||
```
|
||||
1117 / 38t import/wizard.rs ImportPipeline + ImportWizard
|
||||
292 / 9t import/analyzer.rs Field kind detection (Category/Measure/Time/Skip)
|
||||
300 / 8t import/csv_parser.rs CSV parsing, multi-file merge
|
||||
3 / 0t import/mod.rs
|
||||
16 / 0t lib.rs Module decls + re-exports of improvise-core/formula modules
|
||||
```
|
||||
|
||||
### Top-level
|
||||
```
|
||||
400 / 0t draw.rs TUI event loop (run_tui), frame composition
|
||||
391 / 0t main.rs CLI entry (clap): open, import, cmd, script
|
||||
10 / 0t lib.rs Public module re-exports (routes to sub-crates)
|
||||
```
|
||||
|
||||
### Examples
|
||||
```
|
||||
examples/gen-grammar.rs Grammar-walking random file generator (pest_meta)
|
||||
examples/pretty-print.rs Parse stdin, print formatted .improv to stdout
|
||||
```
|
||||
|
||||
### Context docs
|
||||
```
|
||||
context/design-principles.md Architectural principles
|
||||
context/plan.md Show HN launch plan
|
||||
context/repo-map.md This file
|
||||
docs/design-notes.md Product vision & non-goals (salvaged from former SPEC.md)
|
||||
```
|
||||
|
||||
**Total: ~22,000 lines, 572 tests.**
|
||||
|
||||
---
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
### Coverage target
|
||||
|
||||
Aim for **~80% line and branch coverage** on logic code. This is a quality floor, not a
|
||||
ceiling — go higher where the code warrants it, but don't chase 100% on rendering
|
||||
widgets or write tests that just exercise trivial getters. Coverage should be run with
|
||||
`cargo llvm-cov` (available via `nix develop`).
|
||||
|
||||
### What to test and how
|
||||
|
||||
| Layer | Approach | Notes |
|
||||
|-------|----------|-------|
|
||||
| **Model** (types, cell, category, symbol) | Unit tests + **proptest** | The data model is the foundation. Property tests catch invariant violations that hand-picked cases miss (see CellKey sort invariant, axis consistency). |
|
||||
| **Formula** (parser, eval) | Unit tests per operator/construct | Cover each BinOp, AggFunc, IF, WHERE, unary minus, chained formulas, error cases (div-by-zero, missing ref). Ensure eval uses full f64 precision — never display-rounded values. |
|
||||
| **View** (types, layout) | Unit tests + **proptest** | Property tests for axis assignment invariants (each category on exactly one axis, transpose is involutive, etc.). Unit tests for layout computation, records mode detection, drill. |
|
||||
| **Command** (cmd, keymap, parse) | Unit tests | Test command execution by building a `CmdContext` and asserting on returned effects. Test keymap lookup fallback chain. Test script parser with edge cases (quoting, comments, dots). |
|
||||
| **Persistence** | Round-trip + grammar-generated | `save → load → save` must be identical. Grammar-walking generator produces random valid files from the pest AST; proptests verify `parse(generate())` and `parse(format(parse(generate())))`. Cover groups, formulas, views, hidden items, pipe quoting edge cases. |
|
||||
| **Format** | Unit tests | Boundary cases: comma placement at 3/4/7 digits, negative numbers, rounding half-away-from-zero (not banker's), zero, small fractions. |
|
||||
| **Import** (analyzer, csv, wizard) | Unit tests | Field classification heuristics, CSV quoting (RFC 4180), multi-file merge, date extraction. |
|
||||
| **UI rendering** (grid, panels, draw, help) | Generally skip | Ratatui widgets are hard to unit-test and change frequently. Test the *logic* they consume (layout, cat_tree, format) rather than the rendering itself. |
|
||||
| **Effects** | Test indirectly | Effects are thin `apply` methods. Test via integration: send a key through `App::handle_key` and assert on resulting app state. The complex ones (drill reconciliation, import) deserve targeted unit tests. |
|
||||
|
||||
### Property tests (proptest)
|
||||
|
||||
Use property tests for **invariants that must hold across all inputs**, not as a
|
||||
substitute for example-based tests. Good candidates:
|
||||
|
||||
- Structural invariants: CellKey always sorted, each category on exactly one axis,
|
||||
toggle-collapse is involutive, hide/show roundtrips.
|
||||
- Serialization roundtrips: save/load identity.
|
||||
- Determinism: `evaluate` returns the same value for the same inputs.
|
||||
|
||||
Keep proptest case counts reasonable. The defaults (256 cases) are fine for most
|
||||
properties. Don't crank them up to thousands — the test suite should complete in
|
||||
under 2 seconds. If a property needs more cases to feel confident, that's a sign
|
||||
the input space should be constrained with better strategies, not brute-forced.
|
||||
|
||||
### Bug-fix workflow
|
||||
|
||||
Per CLAUDE.md: **write a test that demonstrates the bug before fixing it.** Prefer
|
||||
a small unit test targeting the specific function over an integration test. The test
|
||||
should fail on the current code, then pass after the fix. Mark regression tests
|
||||
with a doc-comment explaining the bug (see `model/types.rs` `formula_tests` for
|
||||
examples).
|
||||
|
||||
### What not to test
|
||||
|
||||
- Trivial struct constructors and enum definitions (`ast.rs`, `axis.rs`).
|
||||
- Ratatui `Widget::render` implementations — these are pure drawing code.
|
||||
- Module re-export files (`mod.rs`).
|
||||
- One-line delegation methods.
|
||||
|
||||
---
|
||||
|
||||
## Patterns to Know
|
||||
|
||||
1. **Commands never mutate.** They receive `&CmdContext` (read-only) and return `Vec<Box<dyn Effect>>`.
|
||||
2. **CellKey is always sorted.** Use `CellKey::new()` — never construct the inner Vec directly.
|
||||
3. **`category_mut()` for adding items.** `Model` has no `add_item` method; get the category first: `m.category_mut("Region").unwrap().add_item("East")`.
|
||||
4. **Virtual categories** `_Index`, `_Dim`, and `_Measure` always exist. `is_empty_model()` checks whether any *non-virtual* categories exist. `_Measure` items come from two sources: explicit data items (in category) + formula targets (dynamically via `measure_item_names()`). `add_formula` does NOT add items to `_Measure` — use `effective_item_names("_Measure")` to get the full list. `_Index` and `_Dim` are never persisted to `.improv` files; `_Measure` only persists non-formula items.
|
||||
5. **Display rounding is view-only.** `format_f64` (half-away-from-zero) is only called in rendering. Formula eval uses full f64.
|
||||
5b. **Formula evaluation is fixed-point.** `recompute_formulas(none_cats)` iterates formula evaluation until values stabilize, using a cache. `evaluate_aggregated` checks the cache for formula results. Circular refs produce `CellValue::Error("circular")`.
|
||||
6. **Keybindings are per-mode.** `ModeKey::from_app_mode()` resolves the current mode, then the corresponding `Keymap` is looked up. Normal + `search_mode=true` maps to `SearchMode`.
|
||||
7. **`effect_cmd!` macro** generates a command struct that just produces effects. Use for simple commands without complex logic.
|
||||
8. **`.improv` format is defined by a PEG grammar** (`persistence/improv.pest`). Parsed by pest. Names use CL-style `|...|` pipe quoting when they aren't valid bare identifiers. JSON is legacy only.
|
||||
9. **`IndexMap`** is used for categories and views to preserve insertion order.
|
||||
10. **`MAX_CATEGORIES = 12`** applies only to `CategoryKind::Regular`. Virtual/Label categories are exempt.
|
||||
11. **Drill into formula cells** strips the `_Measure` coordinate from the drill key when it names a formula target, so `matching_cells` finds the raw data records that feed the formula instead of returning empty.
|
||||
12. **`App::new` calls `recompute_formulas`** before building the initial layout, so formula values appear on the first rendered frame.
|
||||
13. **Minibuffer buffer clearing** is handled by `Binding::Sequence` in keymaps: Enter and Esc sequences include `clear-buffer` to reset the text buffer. The `clear-buffer` command is registered in the registry.
|
||||
ratatui 0.30, crossterm 0.28, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
|
||||
|
||||
Reference in New Issue
Block a user