fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore: add launch plan

Browse Source
This commit is contained in:
Edward Langley
2026-04-08 21:12:00 -07:00
parent f4e4b75b20
commit 2dbedd36ce
2 changed files with 474 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

289
context/SPEC.md
View File

@ -9,8 +9,7 @@ semantically labeled structure — separating data, computation, and views into
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
can be explored from multiple perspectives simultaneously.
The application compiles to a single static binary (`x86_64-unknown-linux-musl`) and provides a rich TUI
experience.
The application compiles to a single static binary and provides a rich TUI experience.
---
@ -23,7 +22,7 @@ experience.
- Items within a category can be organized into **groups** forming a hierarchy.
- Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
- Groups are collapsible/expandable for drill-down.
- A model supports up to **12 categories**.
- A model supports up to **12 categories** (virtual categories prefixed with `_` do not count).
### 1.2 Data Cells
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
@ -32,7 +31,7 @@ experience.
### 1.3 Models
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
- Models are saved to and loaded from a single `.improv` file (JSON format).
- Models are saved to and loaded from a single `.improv` file (plain-text markdown-like format; see §7.1).
---
@ -46,23 +45,24 @@ experience.
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.
### 2.2 Formula Panel
- Formulas are defined in a **dedicated formula panel**, separate from the data grid.
- Formulas are defined in a **dedicated formula panel** (`F` key), separate from the data grid.
- All formulas are visible in one place for easy auditing.
- Formulas cannot be accidentally overwritten by data entry.
- Within the panel: `n`/`o` to create a new formula, `d` to delete.
### 2.3 Scoped Formulas (WHERE clause)
- A formula can be scoped to a subset of items:
- Example: `Discount = 0.10 * Price WHERE Region = "West"`
### 2.4 Aggregation
- Built-in aggregation functions: `SUM`, `AVG`, `MIN`, `MAX`, `COUNT`.
- Built-in aggregation functions: `Sum`, `Avg`, `Min`, `Max`, `Count`.
### 2.5 Formula Language
- Expression-based (not Turing-complete).
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
- Conditionals: `IF(condition, then, else)`.
- `WHERE` clause for filtering: `SUM(Sales WHERE Region = "East")`.
- `WHERE` clause for filtering: `Sum(Sales WHERE Region = "East")`.
- Parentheses for grouping.
- Literal numbers and quoted strings.
@ -72,49 +72,80 @@ experience.
### 3.1 Views as First-Class Objects
- A **view** is a named configuration specifying:
- Which categories are assigned to **rows**, **columns**, and **pages** (filters/slicers).
- Which items/groups are visible vs. hidden.
- Sort order (future).
- Number formatting.
- Which categories are assigned to **rows**, **columns**, **pages** (filters/slicers), or **none** (hidden).
- Which items/groups are visible vs. hidden per-view.
- Collapsed group state per-view.
- Number formatting (per-view format string, e.g. `,.2f`).
- Multiple views can exist per model. Each is independent.
- Editing data in any view updates the underlying model; all other views reflect the change.
### 3.2 Category Tiles
- Each category is represented as a **tile** displayed in the tile bar.
- The user can move tiles between row, column, and page axes to instantly pivot/rearrange the view.
- Moving a tile triggers an instant recalculation and re-render of the grid.
- Press `T` (or `Ctrl+Arrow`) to enter tile-select mode.
- `h`/`l` or `←`/`→` to select a tile.
- `Space`/`Enter` to cycle axis (Row → Column → Page).
- `r`/`c`/`p` to set axis directly to Row / Column / Page.
- Moving a tile triggers instant recalculation and re-render of the grid.
### 3.3 Page Axis (Slicing)
- Categories assigned to the page axis act as filters.
- The user selects a single item from a paged category using `[` and `]`.
### 3.4 Collapsing and Expanding
- Groups can be collapsed/expanded per-view (future: keyboard shortcut in grid).
- Groups can be collapsed/expanded per-view with the `z` key.
### 3.5 Drill-Down and View History
- `>` drills into an aggregated cell, capturing a snapshot.
- `<` navigates back through view history.
### 3.6 Records Mode
- `R` toggles records mode: a long-format view activated when `_Index` is on Row and `_Dim` is on Column.
- `P` toggles "prune empty" to hide rows/columns with no data.
### 3.7 Transpose
- `t` swaps the Row and Column axes instantly.
---
## 4. JSON Import Wizard
## 4. Import Wizard
### 4.1 Purpose
- Users can import arbitrary JSON files to bootstrap a model.
- Users can import CSV or JSON files to bootstrap a model.
- Multiple CSV files can be merged with an automatic "File" category.
### 4.2 Wizard Flow (interactive TUI)
**Step 1: Preview** — Structural summary of the JSON.
**Step 1: Preview** — Structural summary of the data.
**Step 2: Select Array Path** — If the JSON is not a flat array, the user selects which key path contains the primary record array.
**Step 2: Select Array Path** — (JSON only) If the JSON is not a flat array, the user selects which key path contains the primary record array.
**Step 3: Review Proposals** — Fields are analyzed and proposed as:
- Category (small number of distinct string values)
- Measure (numeric)
- Time Category (date-like strings)
- Label/Identifier (skip)
- Time Category (date-like strings, with optional date component extraction: Year, Month, Quarter)
- Skip (exclude from import)
**Step 4: Name the Model** — User names the model and confirms.
**Step 4: Configure** — Set axis assignments, add formulas, name the model.
### 4.3 Headless Import
```
improvise --cmd '{"op":"ImportJson","path":"data.json"}'
### 4.3 CLI Import
```bash
# Interactive wizard
improvise import data.json
improvise import sales.csv expenses.csv # merge multiple CSVs
# Headless (skip wizard)
improvise import data.json --no-wizard -o model.improv
# With field overrides
improvise import data.csv \
--category Region \
--measure Revenue \
--time Date \
--extract Date:Month \
--axis Region:row \
--formula "Profit = Revenue - Cost" \
--name "Sales Model" \
-o output.improv
```
---
@ -124,7 +155,7 @@ improvise --cmd '{"op":"ImportJson","path":"data.json"}'
### 5.1 Layout
```
+---------------------------------------------------------------+
| Improvise | Model: Sales 2025 [*] [F1 Help] [Ctrl+Q] |
| improvise · Sales 2025 (model.improv) [+] ?:help :q quit|
+---------------------------------------------------------------+
| [Page: Region = East] |
| | Q1 | Q2 | Q3 | Q4 | |
@ -135,39 +166,79 @@ improvise --cmd '{"op":"ImportJson","path":"data.json"}'
|--------------+---------+---------+---------+---------+--------|
| Total | 4,100 | 4,670 | 3,750 | 5,800 | |
+---------------------------------------------------------------+
| Tiles: [Time ↔] [Product ↕] [Region ☰] Ctrl+↑↓←→ tiles |
| Tiles: [Time ↔] [Product ↕] [Region ☰] T to select |
+---------------------------------------------------------------+
| NORMAL | Default | Ctrl+F:formulas Ctrl+C:categories ... |
| NORMAL | Default ?:help :q quit |
+---------------------------------------------------------------+
```
### 5.2 Panels
- **Grid panel** (main): Scrollable table of the current view.
- **Tile bar**: Category tiles with axis symbols. `Ctrl+Arrow` enters tile-select mode.
- **Formula panel**: `Ctrl+F` — list and edit formulas.
- **Category panel**: `Ctrl+C` — manage categories and axis assignments.
- **View panel**: `Ctrl+V` — switch, create, delete views.
- **Status bar**: Mode, active view name, keyboard hints.
### 5.2 Panels and Modes
- **Grid** (main): Scrollable table of the current view.
- **Tile bar**: Category tiles with axis symbols. `T` or `Ctrl+Arrow` enters tile-select mode.
- **Formula panel**: `F` — list and edit formulas. `Ctrl+F` toggles visibility without focus.
- **Category panel**: `C` — manage categories, items, and axis assignments. `Ctrl+C` toggles visibility.
- **View panel**: `V` — switch, create, delete views. `Ctrl+V` toggles visibility.
- **Status bar**: Mode indicator, active view name, keyboard hints.
- **Help overlay**: `?` or `F1` — full key reference.
**Modes**: Normal, Insert (editing), Formula Edit, Formula Panel, Category Panel,
View Panel, Tile Select, Category Add, Item Add, Export Prompt, Command (`:` prefix),
Search, Import Wizard, Help, Quit.
### 5.3 Navigation and Editing
| Key | Action |
|-----|--------|
| ↑↓←→ / hjkl | Move cursor |
| Enter | Edit cell |
| Esc | Cancel edit |
| Tab | Focus next open panel |
| / | Search |
| [ / ] | Page axis prev/next |
| Ctrl+Arrow | Tile select mode |
| Enter/Space (tile) | Cycle axis (Row→Col→Page) |
| r / c / p (tile) | Set axis directly |
| Ctrl+F | Toggle formula panel |
| Ctrl+C | Toggle category panel |
| Ctrl+V | Toggle view panel |
| gg / G | Jump to first / last row |
| 0 / $ | Jump to first / last column |
| Ctrl+D / Ctrl+U | Scroll 5 rows down / up |
| PageDown / PageUp | Page scroll |
| [ / ] | Page axis prev / next |
| i / a | Enter insert mode (or drill into aggregated cell) |
| Enter | Advance to next cell while editing |
| o | Add record row and begin editing |
| Esc | Cancel edit / return to Normal |
| x | Clear cell |
| yy | Yank (copy) cell value |
| p | Paste yanked value |
| / | Search grid |
| n | Next search match |
| N | New category quick-add (or previous search match context) |
| t | Transpose (swap rows ↔ columns) |
| z | Toggle group collapse under cursor |
| H | Hide current row item |
| > | Drill into aggregated cell |
| < | Navigate back (view history) |
| R | Toggle records mode |
| P | Toggle prune-empty rows/columns |
| T / Ctrl+Arrow | Enter tile-select mode |
| F | Toggle formula panel (focus) |
| C | Toggle category panel (focus) |
| V | Toggle view panel (focus) |
| Ctrl+F / Ctrl+C / Ctrl+V | Toggle panel visibility (no focus change) |
| Tab | Cycle focus to next open panel |
| : | Enter command mode |
| Ctrl+S | Save |
| Ctrl+E | Export CSV |
| F1 | Help |
| Ctrl+Q | Quit |
| Ctrl+E | Export CSV prompt |
| ZZ | Save and quit |
| Ctrl+Q | Force quit |
| ? / F1 | Help overlay |
### 5.4 Command Mode (`:`)
| Command | Description |
|---------|-------------|
| `:q` | Quit (warns if unsaved) |
| `:q!` | Force quit |
| `:wq` / `ZZ` | Save and quit |
| `:w [path]` | Save (path optional) |
| `:export [path.csv]` | Export active view to CSV |
| `:import <path>` | Open import wizard |
| `:add-cat <name>` | Add a category |
| `:add-item <cat> <item>` | Add one item to a category |
| `:add-items <cat> a b c…` | Add multiple items at once |
| `:formula <cat> <Name=expr>` | Add a formula |
| `:add-view [name]` | Create a new view |
| `:show-item <cat> <item>` | Restore a hidden item |
---
@ -178,63 +249,81 @@ All model mutations go through a typed command layer. This enables:
- Replay / audit log
- Testing without rendering
### 6.1 Command Format
JSON object with an `op` field:
```json
{"op": "CommandName", ...args}
```
### 6.1 CLI Subcommands
### 6.2 Available Commands
| op | Required fields | Description |
|----|-----------------|-------------|
| `AddCategory` | `name` | Add a category/dimension |
| `AddItem` | `category`, `item` | Add an item to a category |
| `AddItemInGroup` | `category`, `item`, `group` | Add an item in a named group |
| `SetCell` | `coords: [[cat,item],...]`, `number` or `text` | Set a cell value |
| `ClearCell` | `coords` | Clear a cell |
| `AddFormula` | `raw`, `target_category` | Add/replace a formula |
| `RemoveFormula` | `target` | Remove a formula by target name |
| `CreateView` | `name` | Create a new view |
| `DeleteView` | `name` | Delete a view |
| `SwitchView` | `name` | Switch the active view |
| `SetAxis` | `category`, `axis` (`"row"/"column"/"page"`) | Set category axis |
| `SetPageSelection` | `category`, `item` | Set page-axis filter |
| `ToggleGroup` | `category`, `group` | Toggle group collapse |
| `Save` | `path` | Save model to file |
| `Load` | `path` | Load model from file |
| `ExportCsv` | `path` | Export active view to CSV |
| `ImportJson` | `path`, `model_name?`, `array_path?` | Import JSON file |
### 6.3 Response Format
```json
{"ok": true, "message": "optional message"}
{"ok": false, "message": "error description"}
```
### 6.4 Invocation
```bash
# Single command
improvise model.improv --cmd '{"op":"SetCell","coords":[["Region","East"],["Measure","Revenue"]],"number":1200}'
# Open TUI (default)
improvise [model.improv]
# Script file (one JSON object per line, # comments allowed)
improvise model.improv --script setup.jsonl
# Single headless command(s)
improvise cmd 'set-cell Region/East Measure/Revenue 1200' -f model.improv
# Script file (one command per line, # or // comments)
improvise script setup.txt -f model.improv
# Import
improvise import data.json
```
### 6.2 Script Syntax
Commands use a prefix syntax (one per line). Multiple commands can be separated by `.` on a single line. Lines starting with `#` or `//` are comments.
### 6.3 Available Commands
Commands are registered internally and accessible via `:` in the TUI or via the `cmd`/`script` CLI subcommands. Key operations include:
- Cell manipulation: `set-cell`, `clear-cell`, `yank`, `paste`
- Navigation: `move-selection`, `scroll-rows`, `page-scroll`, `jump-first-row`, `jump-last-row`
- View: `transpose`, `toggle-records-mode`, `toggle-prune-empty`, `drill-into-cell`, `view-back`, `page-next`, `page-prev`
- Categories: `add-cat`, `add-item`, `add-items`, `delete-category-at-cursor`, `cycle-axis-at-cursor`, `filter-to-item`, `hide-selected-row-item`, `show-item`
- Formulas: `formula`, `enter-formula-edit`, `delete-formula-at-cursor`, `commit-formula`
- Views: `switch-view-at-cursor`, `create-and-switch-view`, `delete-view-at-cursor`, `add-view`
- Tiles: `enter-tile-select`, `move-tile-cursor`, `cycle-axis-for-tile`, `set-axis-for-tile`
- File: `save`, `wq`, `export`, `import`
- Modes: `enter-mode`, `search`, `enter-edit-mode`, `enter-export-prompt`
---
## 7. Persistence
### 7.1 File Format
Native format: JSON-based `.improv` file containing all categories, items, groups, data cells, formulas, and view definitions.
Native format: plain-text markdown-like `.improv` file. Structure:
Compressed variant: `.improv.gz` (gzip, same JSON payload).
```
# Model Name
## Category: Region
- North
- South
- East [Coastal]
- West [Coastal]
> Coastal
## Formulas
- Profit = Revenue - Cost [Measure]
## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"
## View: Default (active)
Region: row
Measure: column
Time: page, Q1
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f
```
Compressed variant: `.improv.gz` (gzip, same payload).
Legacy JSON format is auto-detected (by `{` prefix) for backward compatibility.
### 7.2 Export
- `Ctrl+E` in TUI or `ExportCsv` command: exports active view to CSV.
- `Ctrl+E` in TUI or `:export [path.csv]` command: exports active view to CSV.
- Respects current view axes and page filters.
### 7.3 Autosave
- Periodic autosave (every 30 seconds when dirty) to `.model.improv.autosave`.
- Periodic autosave (every 30 seconds when dirty) to `.{filename}.autosave`.
---
@ -244,8 +333,9 @@ Compressed variant: `.improv.gz` (gzip, same JSON payload).
|---------|--------|
| Language | Rust (stable) |
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
| Serialization | `serde` + `serde_json` |
| Static binary | `x86_64-unknown-linux-musl` via `musl-gcc` |
| Serialization | `serde` + `serde_json` (for legacy compat); native format is plain text |
| CLI | `clap` with subcommands (`open`, `import`, `cmd`, `script`) |
| Build | Standard `cargo build --release` (LTO, stripped); optional musl target via Nix |
| Dev environment | Nix flake with `rust-overlay` |
| No runtime deps | Single binary, no database, no network |
@ -266,17 +356,18 @@ Compressed variant: `.improv.gz` (gzip, same JSON payload).
```bash
# Build
nix develop --command cargo build --release
file target/x86_64-unknown-linux-musl/release/improvise # → statically linked
# Import test
./improvise --cmd '{"op":"ImportJson","path":"sample.json"}' --cmd '{"op":"Save","path":"test.improv"}'
# Import test (interactive wizard)
./improvise import sample.json
# Formula test
./improvise test.improv \
--cmd '{"op":"AddFormula","raw":"Profit = Revenue - Cost","target_category":"Measure"}'
# Import test (headless)
./improvise import sample.json --no-wizard -o test.improv
# Headless commands
./improvise cmd 'add-cat Region' 'add-item Region East' -f new.improv
# Headless script
./improvise new.improv --script tests/setup.jsonl
./improvise script tests/setup.txt -f new.improv
# TUI
./improvise model.improv