fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore: remove stale SPEC.md, salvage vision into docs/design-notes.md

Browse Source 
Audited SPEC.md against code — mostly accurate but redundant with
repo-map.md and design-principles.md. Minor drift in storage internals,
wizard step count, and mode representation. Salvaged product vision and
non-goals into docs/design-notes.md with staleness disclaimer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Langley
2026-04-09 00:11:20 -07:00
parent ecd001e589
commit 4751b800fd
3 changed files with 29 additions and 375 deletions
Download Patch File Download Diff File Expand all files Collapse all files

374
context/SPEC.md
View File

@ -1,374 +0,0 @@
# Improvise — Multi-Dimensional Data Modeling Terminal Application
## Context
Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque
cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without
rewriting formulas. We are building a terminal application that treats data as a multi-dimensional,
semantically labeled structure — separating data, computation, and views into independent layers. The result
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
can be explored from multiple perspectives simultaneously.
The application compiles to a single static binary and provides a rich TUI experience.
---
## 1. Core Data Model
### 1.1 Categories and Items
- Data is organized into **categories** (dimensions) and **items** (members of a dimension).
- Example: Category "Region" contains items "North", "South", "East", "West".
- Example: Category "Time" contains items "Q1", "Q2", "Q3", "Q4".
- Items within a category can be organized into **groups** forming a hierarchy.
- Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
- Groups are collapsible/expandable for drill-down.
- A model supports up to **12 categories** (virtual categories prefixed with `_` do not count).
### 1.2 Data Cells
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
- Cells hold numeric values, text, or empty/null.
- The underlying storage is a sparse multi-dimensional array (`HashMap<CellKey, CellValue>`).
### 1.3 Models
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
- Models are saved to and loaded from a single `.improv` file (plain-text markdown-like format; see §7.1).
---
## 2. Formula System
### 2.1 Named Formulas
- Formulas reference categories and items by name, not by cell address.
- Example: `Profit = Revenue - Cost`
- Example: `Tax = Revenue * 0.08`
- Example: `Margin = Profit / Revenue`
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.
### 2.2 Formula Panel
- Formulas are defined in a **dedicated formula panel** (`F` key), separate from the data grid.
- All formulas are visible in one place for easy auditing.
- Formulas cannot be accidentally overwritten by data entry.
- Within the panel: `n`/`o` to create a new formula, `d` to delete.
### 2.3 Scoped Formulas (WHERE clause)
- A formula can be scoped to a subset of items:
- Example: `Discount = 0.10 * Price WHERE Region = "West"`
### 2.4 Aggregation
- Built-in aggregation functions: `Sum`, `Avg`, `Min`, `Max`, `Count`.
### 2.5 Formula Language
- Expression-based (not Turing-complete).
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
- Conditionals: `IF(condition, then, else)`.
- `WHERE` clause for filtering: `Sum(Sales WHERE Region = "East")`.
- Parentheses for grouping.
- Literal numbers and quoted strings.
---
## 3. View System
### 3.1 Views as First-Class Objects
- A **view** is a named configuration specifying:
- Which categories are assigned to **rows**, **columns**, **pages** (filters/slicers), or **none** (hidden).
- Which items/groups are visible vs. hidden per-view.
- Collapsed group state per-view.
- Number formatting (per-view format string, e.g. `,.2f`).
- Multiple views can exist per model. Each is independent.
- Editing data in any view updates the underlying model; all other views reflect the change.
### 3.2 Category Tiles
- Each category is represented as a **tile** displayed in the tile bar.
- Press `T` (or `Ctrl+Arrow`) to enter tile-select mode.
- `h`/`l` or `←`/`→` to select a tile.
- `Space`/`Enter` to cycle axis (Row → Column → Page).
- `r`/`c`/`p` to set axis directly to Row / Column / Page.
- Moving a tile triggers instant recalculation and re-render of the grid.
### 3.3 Page Axis (Slicing)
- Categories assigned to the page axis act as filters.
- The user selects a single item from a paged category using `[` and `]`.
### 3.4 Collapsing and Expanding
- Groups can be collapsed/expanded per-view with the `z` key.
### 3.5 Drill-Down and View History
- `>` drills into an aggregated cell, capturing a snapshot.
- `<` navigates back through view history.
### 3.6 Records Mode
- `R` toggles records mode: a long-format view activated when `_Index` is on Row and `_Dim` is on Column.
- `P` toggles "prune empty" to hide rows/columns with no data.
### 3.7 Transpose
- `t` swaps the Row and Column axes instantly.
---
## 4. Import Wizard
### 4.1 Purpose
- Users can import CSV or JSON files to bootstrap a model.
- Multiple CSV files can be merged with an automatic "File" category.
### 4.2 Wizard Flow (interactive TUI)
**Step 1: Preview** — Structural summary of the data.
**Step 2: Select Array Path** — (JSON only) If the JSON is not a flat array, the user selects which key path contains the primary record array.
**Step 3: Review Proposals** — Fields are analyzed and proposed as:
- Category (small number of distinct string values)
- Measure (numeric)
- Time Category (date-like strings, with optional date component extraction: Year, Month, Quarter)
- Skip (exclude from import)
**Step 4: Configure** — Set axis assignments, add formulas, name the model.
### 4.3 CLI Import
```bash
# Interactive wizard
improvise import data.json
improvise import sales.csv expenses.csv # merge multiple CSVs
# Headless (skip wizard)
improvise import data.json --no-wizard -o model.improv
# With field overrides
improvise import data.csv \
--category Region \
--measure Revenue \
--time Date \
--extract Date:Month \
--axis Region:row \
--formula "Profit = Revenue - Cost" \
--name "Sales Model" \
-o output.improv
```
---
## 5. Terminal UI
### 5.1 Layout
```
+---------------------------------------------------------------+
| improvise · Sales 2025 (model.improv) [+] ?:help :q quit|
+---------------------------------------------------------------+
| [Page: Region = East] |
| | Q1 | Q2 | Q3 | Q4 | |
|--------------+---------+---------+---------+---------+--------|
| Shirts | 1,200 | 1,450 | 1,100 | 1,800 | |
| Pants | 800 | 920 | 750 | 1,200 | |
| ... |
|--------------+---------+---------+---------+---------+--------|
| Total | 4,100 | 4,670 | 3,750 | 5,800 | |
+---------------------------------------------------------------+
| Tiles: [Time ↔] [Product ↕] [Region ☰] T to select |
+---------------------------------------------------------------+
| NORMAL | Default ?:help :q quit |
+---------------------------------------------------------------+
```
### 5.2 Panels and Modes
- **Grid** (main): Scrollable table of the current view.
- **Tile bar**: Category tiles with axis symbols. `T` or `Ctrl+Arrow` enters tile-select mode.
- **Formula panel**: `F` — list and edit formulas. `Ctrl+F` toggles visibility without focus.
- **Category panel**: `C` — manage categories, items, and axis assignments. `Ctrl+C` toggles visibility.
- **View panel**: `V` — switch, create, delete views. `Ctrl+V` toggles visibility.
- **Status bar**: Mode indicator, active view name, keyboard hints.
- **Help overlay**: `?` or `F1` — full key reference.
**Modes**: Normal, Insert (editing), Formula Edit, Formula Panel, Category Panel,
View Panel, Tile Select, Category Add, Item Add, Export Prompt, Command (`:` prefix),
Search, Import Wizard, Help, Quit.
### 5.3 Navigation and Editing
| Key | Action |
|-----|--------|
| ↑↓←→ / hjkl | Move cursor |
| gg / G | Jump to first / last row |
| 0 / $ | Jump to first / last column |
| Ctrl+D / Ctrl+U | Scroll 5 rows down / up |
| PageDown / PageUp | Page scroll |
| [ / ] | Page axis prev / next |
| i / a | Enter insert mode (or drill into aggregated cell) |
| Enter | Advance to next cell while editing |
| o | Add record row and begin editing |
| Esc | Cancel edit / return to Normal |
| x | Clear cell |
| yy | Yank (copy) cell value |
| p | Paste yanked value |
| / | Search grid |
| n | Next search match |
| N | New category quick-add (or previous search match context) |
| t | Transpose (swap rows ↔ columns) |
| z | Toggle group collapse under cursor |
| H | Hide current row item |
| > | Drill into aggregated cell |
| < | Navigate back (view history) |
| R | Toggle records mode |
| P | Toggle prune-empty rows/columns |
| T / Ctrl+Arrow | Enter tile-select mode |
| F | Toggle formula panel (focus) |
| C | Toggle category panel (focus) |
| V | Toggle view panel (focus) |
| Ctrl+F / Ctrl+C / Ctrl+V | Toggle panel visibility (no focus change) |
| Tab | Cycle focus to next open panel |
| : | Enter command mode |
| Ctrl+S | Save |
| Ctrl+E | Export CSV prompt |
| ZZ | Save and quit |
| Ctrl+Q | Force quit |
| ? / F1 | Help overlay |
### 5.4 Command Mode (`:`)
| Command | Description |
|---------|-------------|
| `:q` | Quit (warns if unsaved) |
| `:q!` | Force quit |
| `:wq` / `ZZ` | Save and quit |
| `:w [path]` | Save (path optional) |
| `:export [path.csv]` | Export active view to CSV |
| `:import <path>` | Open import wizard |
| `:add-cat <name>` | Add a category |
| `:add-item <cat> <item>` | Add one item to a category |
| `:add-items <cat> a b c…` | Add multiple items at once |
| `:formula <cat> <Name=expr>` | Add a formula |
| `:add-view [name]` | Create a new view |
| `:show-item <cat> <item>` | Restore a hidden item |
---
## 6. Command Layer (Headless Mode)
All model mutations go through a typed command layer. This enables:
- Scripting without the TUI
- Replay / audit log
- Testing without rendering
### 6.1 CLI Subcommands
```bash
# Open TUI (default)
improvise [model.improv]
# Single headless command(s)
improvise cmd 'set-cell Region/East Measure/Revenue 1200' -f model.improv
# Script file (one command per line, # or // comments)
improvise script setup.txt -f model.improv
# Import
improvise import data.json
```
### 6.2 Script Syntax
Commands use a prefix syntax (one per line). Multiple commands can be separated by `.` on a single line. Lines starting with `#` or `//` are comments.
### 6.3 Available Commands
Commands are registered internally and accessible via `:` in the TUI or via the `cmd`/`script` CLI subcommands. Key operations include:
- Cell manipulation: `set-cell`, `clear-cell`, `yank`, `paste`
- Navigation: `move-selection`, `scroll-rows`, `page-scroll`, `jump-first-row`, `jump-last-row`
- View: `transpose`, `toggle-records-mode`, `toggle-prune-empty`, `drill-into-cell`, `view-back`, `page-next`, `page-prev`
- Categories: `add-cat`, `add-item`, `add-items`, `delete-category-at-cursor`, `cycle-axis-at-cursor`, `filter-to-item`, `hide-selected-row-item`, `show-item`
- Formulas: `formula`, `enter-formula-edit`, `delete-formula-at-cursor`, `commit-formula`
- Views: `switch-view-at-cursor`, `create-and-switch-view`, `delete-view-at-cursor`, `add-view`
- Tiles: `enter-tile-select`, `move-tile-cursor`, `cycle-axis-for-tile`, `set-axis-for-tile`
- File: `save`, `wq`, `export`, `import`
- Modes: `enter-mode`, `search`, `enter-edit-mode`, `enter-export-prompt`
---
## 7. Persistence
### 7.1 File Format
Native format: plain-text markdown-like `.improv` file. Structure:
```
# Model Name
## Category: Region
- North
- South
- East [Coastal]
- West [Coastal]
> Coastal
## Formulas
- Profit = Revenue - Cost [Measure]
## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"
## View: Default (active)
Region: row
Measure: column
Time: page, Q1
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f
```
Compressed variant: `.improv.gz` (gzip, same payload).
Legacy JSON format is auto-detected (by `{` prefix) for backward compatibility.
### 7.2 Export
- `Ctrl+E` in TUI or `:export [path.csv]` command: exports active view to CSV.
- Respects current view axes and page filters.
### 7.3 Autosave
- Periodic autosave (every 30 seconds when dirty) to `.{filename}.autosave`.
---
## 8. Technology
| Concern | Choice |
|---------|--------|
| Language | Rust (stable) |
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
| Serialization | `serde` + `serde_json` (for legacy compat); native format is plain text |
| CLI | `clap` with subcommands (`open`, `import`, `cmd`, `script`) |
| Build | Standard `cargo build --release` (LTO, stripped); optional musl target via Nix |
| Dev environment | Nix flake with `rust-overlay` |
| No runtime deps | Single binary, no database, no network |
---
## 9. Non-Goals (v1)
- Scripting/macro language beyond the formula system.
- Collaborative/multi-user editing.
- Live external data sources (databases, APIs).
- Charts or graphical visualization.
- Multi-level undo history.
---
## 10. Verification
```bash
# Build
nix develop --command cargo build --release
# Import test (interactive wizard)
./improvise import sample.json
# Import test (headless)
./improvise import sample.json --no-wizard -o test.improv
# Headless commands
./improvise cmd 'add-cat Region' 'add-item Region East' -f new.improv
# Headless script
./improvise script tests/setup.txt -f new.improv
# TUI
./improvise model.improv
```

2
context/repo-map.md
View File

@ -410,10 +410,10 @@ Lines / tests / path — grouped by layer.
### Context docs
```
context/SPEC.md Feature specification
context/design-principles.md Architectural principles
context/plan.md Show HN launch plan
context/repo-map.md This file
docs/design-notes.md Product vision & non-goals (salvaged from former SPEC.md)
```
**Total: ~16,500 lines, 510 tests.**