# Improvise — Multi-Dimensional Data Modeling Terminal Application

## Context

Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque
cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without
rewriting formulas. We are building a terminal application that treats data as a multi-dimensional,
semantically labeled structure — separating data, computation, and views into independent layers. The result
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
can be explored from multiple perspectives simultaneously.

The application compiles to a single static binary and provides a rich TUI experience.

---

## 1. Core Data Model

### 1.1 Categories and Items
- Data is organized into **categories** (dimensions) and **items** (members of a dimension).
  - Example: Category "Region" contains items "North", "South", "East", "West".
  - Example: Category "Time" contains items "Q1", "Q2", "Q3", "Q4".
- Items within a category can be organized into **groups** forming a hierarchy.
  - Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
- Groups are collapsible/expandable for drill-down.
- A model supports up to **12 categories** (virtual categories prefixed with `_` do not count).

### 1.2 Data Cells
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
- Cells hold numeric values, text, or empty/null.
- The underlying storage is a sparse multi-dimensional array (`HashMap<CellKey, CellValue>`).

### 1.3 Models
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
- Models are saved to and loaded from a single `.improv` file (plain-text markdown-like format; see §7.1).

---

## 2. Formula System

### 2.1 Named Formulas
- Formulas reference categories and items by name, not by cell address.
  - Example: `Profit = Revenue - Cost`
  - Example: `Tax = Revenue * 0.08`
  - Example: `Margin = Profit / Revenue`
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.

### 2.2 Formula Panel
- Formulas are defined in a **dedicated formula panel** (`F` key), separate from the data grid.
- All formulas are visible in one place for easy auditing.
- Formulas cannot be accidentally overwritten by data entry.
- Within the panel: `n`/`o` to create a new formula, `d` to delete.

### 2.3 Scoped Formulas (WHERE clause)
- A formula can be scoped to a subset of items:
  - Example: `Discount = 0.10 * Price WHERE Region = "West"`

### 2.4 Aggregation
- Built-in aggregation functions: `Sum`, `Avg`, `Min`, `Max`, `Count`.

### 2.5 Formula Language
- Expression-based (not Turing-complete).
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
- Conditionals: `IF(condition, then, else)`.
- `WHERE` clause for filtering: `Sum(Sales WHERE Region = "East")`.
- Parentheses for grouping.
- Literal numbers and quoted strings.

---

## 3. View System

### 3.1 Views as First-Class Objects
- A **view** is a named configuration specifying:
  - Which categories are assigned to **rows**, **columns**, **pages** (filters/slicers), or **none** (hidden).
  - Which items/groups are visible vs. hidden per-view.
  - Collapsed group state per-view.
  - Number formatting (per-view format string, e.g. `,.2f`).
- Multiple views can exist per model. Each is independent.
- Editing data in any view updates the underlying model; all other views reflect the change.

### 3.2 Category Tiles
- Each category is represented as a **tile** displayed in the tile bar.
- Press `T` (or `Ctrl+Arrow`) to enter tile-select mode.
  - `h`/`l` or `←`/`→` to select a tile.
  - `Space`/`Enter` to cycle axis (Row → Column → Page).
  - `r`/`c`/`p` to set axis directly to Row / Column / Page.
- Moving a tile triggers instant recalculation and re-render of the grid.

### 3.3 Page Axis (Slicing)
- Categories assigned to the page axis act as filters.
- The user selects a single item from a paged category using `[` and `]`.

### 3.4 Collapsing and Expanding
- Groups can be collapsed/expanded per-view with the `z` key.

### 3.5 Drill-Down and View History
- `>` drills into an aggregated cell, capturing a snapshot.
- `<` navigates back through view history.

### 3.6 Records Mode
- `R` toggles records mode: a long-format view activated when `_Index` is on Row and `_Dim` is on Column.
- `P` toggles "prune empty" to hide rows/columns with no data.

### 3.7 Transpose
- `t` swaps the Row and Column axes instantly.

---

## 4. Import Wizard

### 4.1 Purpose
- Users can import CSV or JSON files to bootstrap a model.
- Multiple CSV files can be merged with an automatic "File" category.

### 4.2 Wizard Flow (interactive TUI)

**Step 1: Preview** — Structural summary of the data.

**Step 2: Select Array Path** — (JSON only) If the JSON is not a flat array, the user selects which key path contains the primary record array.

**Step 3: Review Proposals** — Fields are analyzed and proposed as:
  - Category (small number of distinct string values)
  - Measure (numeric)
  - Time Category (date-like strings, with optional date component extraction: Year, Month, Quarter)
  - Skip (exclude from import)

**Step 4: Configure** — Set axis assignments, add formulas, name the model.

### 4.3 CLI Import
```bash
# Interactive wizard
improvise import data.json
improvise import sales.csv expenses.csv   # merge multiple CSVs

# Headless (skip wizard)
improvise import data.json --no-wizard -o model.improv

# With field overrides
improvise import data.csv \
  --category Region \
  --measure Revenue \
  --time Date \
  --extract Date:Month \
  --axis Region:row \
  --formula "Profit = Revenue - Cost" \
  --name "Sales Model" \
  -o output.improv
```

---

## 5. Terminal UI

### 5.1 Layout
```
+---------------------------------------------------------------+
| improvise  ·  Sales 2025 (model.improv) [+]     ?:help :q quit|
+---------------------------------------------------------------+
|  [Page: Region = East]                                        |
|              | Q1      | Q2      | Q3      | Q4      |        |
|--------------+---------+---------+---------+---------+--------|
| Shirts       |   1,200 |   1,450 |   1,100 |   1,800 |        |
| Pants        |     800 |     920 |     750 |   1,200 |        |
| ...                                                           |
|--------------+---------+---------+---------+---------+--------|
| Total        |   4,100 |   4,670 |   3,750 |   5,800 |        |
+---------------------------------------------------------------+
| Tiles: [Time ↔] [Product ↕] [Region ☰]       T to select     |
+---------------------------------------------------------------+
| NORMAL | Default                              ?:help  :q quit |
+---------------------------------------------------------------+
```

### 5.2 Panels and Modes
- **Grid** (main): Scrollable table of the current view.
- **Tile bar**: Category tiles with axis symbols. `T` or `Ctrl+Arrow` enters tile-select mode.
- **Formula panel**: `F` — list and edit formulas. `Ctrl+F` toggles visibility without focus.
- **Category panel**: `C` — manage categories, items, and axis assignments. `Ctrl+C` toggles visibility.
- **View panel**: `V` — switch, create, delete views. `Ctrl+V` toggles visibility.
- **Status bar**: Mode indicator, active view name, keyboard hints.
- **Help overlay**: `?` or `F1` — full key reference.

**Modes**: Normal, Insert (editing), Formula Edit, Formula Panel, Category Panel,
View Panel, Tile Select, Category Add, Item Add, Export Prompt, Command (`:` prefix),
Search, Import Wizard, Help, Quit.

### 5.3 Navigation and Editing
| Key | Action |
|-----|--------|
| ↑↓←→ / hjkl | Move cursor |
| gg / G | Jump to first / last row |
| 0 / $ | Jump to first / last column |
| Ctrl+D / Ctrl+U | Scroll 5 rows down / up |
| PageDown / PageUp | Page scroll |
| [ / ] | Page axis prev / next |
| i / a | Enter insert mode (or drill into aggregated cell) |
| Enter | Advance to next cell while editing |
| o | Add record row and begin editing |
| Esc | Cancel edit / return to Normal |
| x | Clear cell |
| yy | Yank (copy) cell value |
| p | Paste yanked value |
| / | Search grid |
| n | Next search match |
| N | New category quick-add (or previous search match context) |
| t | Transpose (swap rows ↔ columns) |
| z | Toggle group collapse under cursor |
| H | Hide current row item |
| > | Drill into aggregated cell |
| < | Navigate back (view history) |
| R | Toggle records mode |
| P | Toggle prune-empty rows/columns |
| T / Ctrl+Arrow | Enter tile-select mode |
| F | Toggle formula panel (focus) |
| C | Toggle category panel (focus) |
| V | Toggle view panel (focus) |
| Ctrl+F / Ctrl+C / Ctrl+V | Toggle panel visibility (no focus change) |
| Tab | Cycle focus to next open panel |
| : | Enter command mode |
| Ctrl+S | Save |
| Ctrl+E | Export CSV prompt |
| ZZ | Save and quit |
| Ctrl+Q | Force quit |
| ? / F1 | Help overlay |

### 5.4 Command Mode (`:`)
| Command | Description |
|---------|-------------|
| `:q` | Quit (warns if unsaved) |
| `:q!` | Force quit |
| `:wq` / `ZZ` | Save and quit |
| `:w [path]` | Save (path optional) |
| `:export [path.csv]` | Export active view to CSV |
| `:import <path>` | Open import wizard |
| `:add-cat <name>` | Add a category |
| `:add-item <cat> <item>` | Add one item to a category |
| `:add-items <cat> a b c…` | Add multiple items at once |
| `:formula <cat> <Name=expr>` | Add a formula |
| `:add-view [name]` | Create a new view |
| `:show-item <cat> <item>` | Restore a hidden item |

---

## 6. Command Layer (Headless Mode)

All model mutations go through a typed command layer. This enables:
- Scripting without the TUI
- Replay / audit log
- Testing without rendering

### 6.1 CLI Subcommands

```bash
# Open TUI (default)
improvise [model.improv]

# Single headless command(s)
improvise cmd 'set-cell Region/East Measure/Revenue 1200' -f model.improv

# Script file (one command per line, # or // comments)
improvise script setup.txt -f model.improv

# Import
improvise import data.json
```

### 6.2 Script Syntax
Commands use a prefix syntax (one per line). Multiple commands can be separated by `.` on a single line. Lines starting with `#` or `//` are comments.

### 6.3 Available Commands
Commands are registered internally and accessible via `:` in the TUI or via the `cmd`/`script` CLI subcommands. Key operations include:
- Cell manipulation: `set-cell`, `clear-cell`, `yank`, `paste`
- Navigation: `move-selection`, `scroll-rows`, `page-scroll`, `jump-first-row`, `jump-last-row`
- View: `transpose`, `toggle-records-mode`, `toggle-prune-empty`, `drill-into-cell`, `view-back`, `page-next`, `page-prev`
- Categories: `add-cat`, `add-item`, `add-items`, `delete-category-at-cursor`, `cycle-axis-at-cursor`, `filter-to-item`, `hide-selected-row-item`, `show-item`
- Formulas: `formula`, `enter-formula-edit`, `delete-formula-at-cursor`, `commit-formula`
- Views: `switch-view-at-cursor`, `create-and-switch-view`, `delete-view-at-cursor`, `add-view`
- Tiles: `enter-tile-select`, `move-tile-cursor`, `cycle-axis-for-tile`, `set-axis-for-tile`
- File: `save`, `wq`, `export`, `import`
- Modes: `enter-mode`, `search`, `enter-edit-mode`, `enter-export-prompt`

---

## 7. Persistence

### 7.1 File Format
Native format: plain-text markdown-like `.improv` file. Structure:

```
# Model Name

## Category: Region
- North
- South
- East [Coastal]
- West [Coastal]
> Coastal

## Formulas
- Profit = Revenue - Cost [Measure]

## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"

## View: Default (active)
Region: row
Measure: column
Time: page, Q1
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f
```

Compressed variant: `.improv.gz` (gzip, same payload).

Legacy JSON format is auto-detected (by `{` prefix) for backward compatibility.

### 7.2 Export
- `Ctrl+E` in TUI or `:export [path.csv]` command: exports active view to CSV.
- Respects current view axes and page filters.

### 7.3 Autosave
- Periodic autosave (every 30 seconds when dirty) to `.{filename}.autosave`.

---

## 8. Technology

| Concern | Choice |
|---------|--------|
| Language | Rust (stable) |
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
| Serialization | `serde` + `serde_json` (for legacy compat); native format is plain text |
| CLI | `clap` with subcommands (`open`, `import`, `cmd`, `script`) |
| Build | Standard `cargo build --release` (LTO, stripped); optional musl target via Nix |
| Dev environment | Nix flake with `rust-overlay` |
| No runtime deps | Single binary, no database, no network |

---

## 9. Non-Goals (v1)

- Scripting/macro language beyond the formula system.
- Collaborative/multi-user editing.
- Live external data sources (databases, APIs).
- Charts or graphical visualization.
- Multi-level undo history.

---

## 10. Verification

```bash
# Build
nix develop --command cargo build --release

# Import test (interactive wizard)
./improvise import sample.json

# Import test (headless)
./improvise import sample.json --no-wizard -o test.improv

# Headless commands
./improvise cmd 'add-cat Region' 'add-item Region East' -f new.improv

# Headless script
./improvise script tests/setup.txt -f new.improv

# TUI
./improvise model.improv
```