fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore: remove stale SPEC.md, salvage vision into docs/design-notes.md

Browse Source 
Audited SPEC.md against code — mostly accurate but redundant with
repo-map.md and design-principles.md. Minor drift in storage internals,
wizard step count, and mode representation. Salvaged product vision and
non-goals into docs/design-notes.md with staleness disclaimer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Langley
2026-04-09 00:11:20 -07:00
parent ecd001e589
commit 4751b800fd
3 changed files with 29 additions and 375 deletions
Download Patch File Download Diff File Expand all files Collapse all files

374
context/SPEC.md
View File

@ -1,374 +0,0 @@
# Improvise — Multi-Dimensional Data Modeling Terminal Application
## Context
Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque
cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without
rewriting formulas. We are building a terminal application that treats data as a multi-dimensional,
semantically labeled structure — separating data, computation, and views into independent layers. The result
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
can be explored from multiple perspectives simultaneously.
The application compiles to a single static binary and provides a rich TUI experience.
---
## 1. Core Data Model
### 1.1 Categories and Items
- Data is organized into **categories** (dimensions) and **items** (members of a dimension).
- Example: Category "Region" contains items "North", "South", "East", "West".
- Example: Category "Time" contains items "Q1", "Q2", "Q3", "Q4".
- Items within a category can be organized into **groups** forming a hierarchy.
- Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
- Groups are collapsible/expandable for drill-down.
- A model supports up to **12 categories** (virtual categories prefixed with `_` do not count).
### 1.2 Data Cells
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
- Cells hold numeric values, text, or empty/null.
- The underlying storage is a sparse multi-dimensional array (`HashMap<CellKey, CellValue>`).
### 1.3 Models
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
- Models are saved to and loaded from a single `.improv` file (plain-text markdown-like format; see §7.1).
---
## 2. Formula System
### 2.1 Named Formulas
- Formulas reference categories and items by name, not by cell address.
- Example: `Profit = Revenue - Cost`
- Example: `Tax = Revenue * 0.08`
- Example: `Margin = Profit / Revenue`
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.
### 2.2 Formula Panel
- Formulas are defined in a **dedicated formula panel** (`F` key), separate from the data grid.
- All formulas are visible in one place for easy auditing.
- Formulas cannot be accidentally overwritten by data entry.
- Within the panel: `n`/`o` to create a new formula, `d` to delete.
### 2.3 Scoped Formulas (WHERE clause)
- A formula can be scoped to a subset of items:
- Example: `Discount = 0.10 * Price WHERE Region = "West"`
### 2.4 Aggregation
- Built-in aggregation functions: `Sum`, `Avg`, `Min`, `Max`, `Count`.
### 2.5 Formula Language
- Expression-based (not Turing-complete).
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
- Conditionals: `IF(condition, then, else)`.
- `WHERE` clause for filtering: `Sum(Sales WHERE Region = "East")`.
- Parentheses for grouping.
- Literal numbers and quoted strings.
---
## 3. View System
### 3.1 Views as First-Class Objects
- A **view** is a named configuration specifying:
- Which categories are assigned to **rows**, **columns**, **pages** (filters/slicers), or **none** (hidden).
- Which items/groups are visible vs. hidden per-view.
- Collapsed group state per-view.
- Number formatting (per-view format string, e.g. `,.2f`).
- Multiple views can exist per model. Each is independent.
- Editing data in any view updates the underlying model; all other views reflect the change.
### 3.2 Category Tiles
- Each category is represented as a **tile** displayed in the tile bar.
- Press `T` (or `Ctrl+Arrow`) to enter tile-select mode.
- `h`/`l` or `←`/`→` to select a tile.
- `Space`/`Enter` to cycle axis (Row → Column → Page).
- `r`/`c`/`p` to set axis directly to Row / Column / Page.
- Moving a tile triggers instant recalculation and re-render of the grid.
### 3.3 Page Axis (Slicing)
- Categories assigned to the page axis act as filters.
- The user selects a single item from a paged category using `[` and `]`.
### 3.4 Collapsing and Expanding
- Groups can be collapsed/expanded per-view with the `z` key.
### 3.5 Drill-Down and View History
- `>` drills into an aggregated cell, capturing a snapshot.
- `<` navigates back through view history.
### 3.6 Records Mode
- `R` toggles records mode: a long-format view activated when `_Index` is on Row and `_Dim` is on Column.
- `P` toggles "prune empty" to hide rows/columns with no data.
### 3.7 Transpose
- `t` swaps the Row and Column axes instantly.
---
## 4. Import Wizard
### 4.1 Purpose
- Users can import CSV or JSON files to bootstrap a model.
- Multiple CSV files can be merged with an automatic "File" category.
### 4.2 Wizard Flow (interactive TUI)
**Step 1: Preview** — Structural summary of the data.
**Step 2: Select Array Path** — (JSON only) If the JSON is not a flat array, the user selects which key path contains the primary record array.
**Step 3: Review Proposals** — Fields are analyzed and proposed as:
- Category (small number of distinct string values)
- Measure (numeric)
- Time Category (date-like strings, with optional date component extraction: Year, Month, Quarter)
- Skip (exclude from import)
**Step 4: Configure** — Set axis assignments, add formulas, name the model.
### 4.3 CLI Import
```bash
# Interactive wizard
improvise import data.json
improvise import sales.csv expenses.csv # merge multiple CSVs
# Headless (skip wizard)
improvise import data.json --no-wizard -o model.improv
# With field overrides
improvise import data.csv \
--category Region \
--measure Revenue \
--time Date \
--extract Date:Month \
--axis Region:row \
--formula "Profit = Revenue - Cost" \
--name "Sales Model" \
-o output.improv
```
---
## 5. Terminal UI
### 5.1 Layout
```
+---------------------------------------------------------------+
| improvise · Sales 2025 (model.improv) [+] ?:help :q quit|
+---------------------------------------------------------------+
| [Page: Region = East] |
| | Q1 | Q2 | Q3 | Q4 | |
|--------------+---------+---------+---------+---------+--------|
| Shirts | 1,200 | 1,450 | 1,100 | 1,800 | |
| Pants | 800 | 920 | 750 | 1,200 | |
| ... |
|--------------+---------+---------+---------+---------+--------|
| Total | 4,100 | 4,670 | 3,750 | 5,800 | |
+---------------------------------------------------------------+
| Tiles: [Time ↔] [Product ↕] [Region ☰] T to select |
+---------------------------------------------------------------+
| NORMAL | Default ?:help :q quit |
+---------------------------------------------------------------+
```
### 5.2 Panels and Modes
- **Grid** (main): Scrollable table of the current view.
- **Tile bar**: Category tiles with axis symbols. `T` or `Ctrl+Arrow` enters tile-select mode.
- **Formula panel**: `F` — list and edit formulas. `Ctrl+F` toggles visibility without focus.
- **Category panel**: `C` — manage categories, items, and axis assignments. `Ctrl+C` toggles visibility.
- **View panel**: `V` — switch, create, delete views. `Ctrl+V` toggles visibility.
- **Status bar**: Mode indicator, active view name, keyboard hints.
- **Help overlay**: `?` or `F1` — full key reference.
**Modes**: Normal, Insert (editing), Formula Edit, Formula Panel, Category Panel,
View Panel, Tile Select, Category Add, Item Add, Export Prompt, Command (`:` prefix),
Search, Import Wizard, Help, Quit.
### 5.3 Navigation and Editing
| Key | Action |
|-----|--------|
| ↑↓←→ / hjkl | Move cursor |
| gg / G | Jump to first / last row |
| 0 / $ | Jump to first / last column |
| Ctrl+D / Ctrl+U | Scroll 5 rows down / up |
| PageDown / PageUp | Page scroll |
| [ / ] | Page axis prev / next |
| i / a | Enter insert mode (or drill into aggregated cell) |
| Enter | Advance to next cell while editing |
| o | Add record row and begin editing |
| Esc | Cancel edit / return to Normal |
| x | Clear cell |
| yy | Yank (copy) cell value |
| p | Paste yanked value |
| / | Search grid |
| n | Next search match |
| N | New category quick-add (or previous search match context) |
| t | Transpose (swap rows ↔ columns) |
| z | Toggle group collapse under cursor |
| H | Hide current row item |
| > | Drill into aggregated cell |
| < | Navigate back (view history) |
| R | Toggle records mode |
| P | Toggle prune-empty rows/columns |
| T / Ctrl+Arrow | Enter tile-select mode |
| F | Toggle formula panel (focus) |
| C | Toggle category panel (focus) |
| V | Toggle view panel (focus) |
| Ctrl+F / Ctrl+C / Ctrl+V | Toggle panel visibility (no focus change) |
| Tab | Cycle focus to next open panel |
| : | Enter command mode |
| Ctrl+S | Save |
| Ctrl+E | Export CSV prompt |
| ZZ | Save and quit |
| Ctrl+Q | Force quit |
| ? / F1 | Help overlay |
### 5.4 Command Mode (`:`)
| Command | Description |
|---------|-------------|
| `:q` | Quit (warns if unsaved) |
| `:q!` | Force quit |
| `:wq` / `ZZ` | Save and quit |
| `:w [path]` | Save (path optional) |
| `:export [path.csv]` | Export active view to CSV |
| `:import <path>` | Open import wizard |
| `:add-cat <name>` | Add a category |
| `:add-item <cat> <item>` | Add one item to a category |
| `:add-items <cat> a b c…` | Add multiple items at once |
| `:formula <cat> <Name=expr>` | Add a formula |
| `:add-view [name]` | Create a new view |
| `:show-item <cat> <item>` | Restore a hidden item |
---
## 6. Command Layer (Headless Mode)
All model mutations go through a typed command layer. This enables:
- Scripting without the TUI
- Replay / audit log
- Testing without rendering
### 6.1 CLI Subcommands
```bash
# Open TUI (default)
improvise [model.improv]
# Single headless command(s)
improvise cmd 'set-cell Region/East Measure/Revenue 1200' -f model.improv
# Script file (one command per line, # or // comments)
improvise script setup.txt -f model.improv
# Import
improvise import data.json
```
### 6.2 Script Syntax
Commands use a prefix syntax (one per line). Multiple commands can be separated by `.` on a single line. Lines starting with `#` or `//` are comments.
### 6.3 Available Commands
Commands are registered internally and accessible via `:` in the TUI or via the `cmd`/`script` CLI subcommands. Key operations include:
- Cell manipulation: `set-cell`, `clear-cell`, `yank`, `paste`
- Navigation: `move-selection`, `scroll-rows`, `page-scroll`, `jump-first-row`, `jump-last-row`
- View: `transpose`, `toggle-records-mode`, `toggle-prune-empty`, `drill-into-cell`, `view-back`, `page-next`, `page-prev`
- Categories: `add-cat`, `add-item`, `add-items`, `delete-category-at-cursor`, `cycle-axis-at-cursor`, `filter-to-item`, `hide-selected-row-item`, `show-item`
- Formulas: `formula`, `enter-formula-edit`, `delete-formula-at-cursor`, `commit-formula`
- Views: `switch-view-at-cursor`, `create-and-switch-view`, `delete-view-at-cursor`, `add-view`
- Tiles: `enter-tile-select`, `move-tile-cursor`, `cycle-axis-for-tile`, `set-axis-for-tile`
- File: `save`, `wq`, `export`, `import`
- Modes: `enter-mode`, `search`, `enter-edit-mode`, `enter-export-prompt`
---
## 7. Persistence
### 7.1 File Format
Native format: plain-text markdown-like `.improv` file. Structure:
```
# Model Name
## Category: Region
- North
- South
- East [Coastal]
- West [Coastal]
> Coastal
## Formulas
- Profit = Revenue - Cost [Measure]
## Data
Region=East, Measure=Revenue = 1200
Region=East, Measure=Cost = 800
Region=West, Measure=Revenue = "pending"
## View: Default (active)
Region: row
Measure: column
Time: page, Q1
hidden: Region/Internal
collapsed: Time/2024
format: ,.2f
```
Compressed variant: `.improv.gz` (gzip, same payload).
Legacy JSON format is auto-detected (by `{` prefix) for backward compatibility.
### 7.2 Export
- `Ctrl+E` in TUI or `:export [path.csv]` command: exports active view to CSV.
- Respects current view axes and page filters.
### 7.3 Autosave
- Periodic autosave (every 30 seconds when dirty) to `.{filename}.autosave`.
---
## 8. Technology
| Concern | Choice |
|---------|--------|
| Language | Rust (stable) |
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
| Serialization | `serde` + `serde_json` (for legacy compat); native format is plain text |
| CLI | `clap` with subcommands (`open`, `import`, `cmd`, `script`) |
| Build | Standard `cargo build --release` (LTO, stripped); optional musl target via Nix |
| Dev environment | Nix flake with `rust-overlay` |
| No runtime deps | Single binary, no database, no network |
---
## 9. Non-Goals (v1)
- Scripting/macro language beyond the formula system.
- Collaborative/multi-user editing.
- Live external data sources (databases, APIs).
- Charts or graphical visualization.
- Multi-level undo history.
---
## 10. Verification
```bash
# Build
nix develop --command cargo build --release
# Import test (interactive wizard)
./improvise import sample.json
# Import test (headless)
./improvise import sample.json --no-wizard -o test.improv
# Headless commands
./improvise cmd 'add-cat Region' 'add-item Region East' -f new.improv
# Headless script
./improvise script tests/setup.txt -f new.improv
# TUI
./improvise model.improv
```

2
context/repo-map.md
View File

@ -410,10 +410,10 @@ Lines / tests / path — grouped by layer.
### Context docs
```
context/SPEC.md Feature specification
context/design-principles.md Architectural principles
context/plan.md Show HN launch plan
context/repo-map.md This file
docs/design-notes.md Product vision & non-goals (salvaged from former SPEC.md)
```
**Total: ~16,500 lines, 510 tests.**

28
docs/design-notes.md Normal file
View File

@ -0,0 +1,28 @@
# Design Notes
> **Staleness warning:** This document captures conceptual framing from the
> original specification. It is not kept in sync with the code. For current
> architecture and types, see `context/repo-map.md` and
> `context/design-principles.md`.
## Product Vision
Traditional spreadsheets conflate data, formulas, and presentation into a single
flat grid addressed by opaque cell references (A1, B7). This makes models
fragile, hard to audit, and impossible to rearrange without rewriting formulas.
Improvise treats data as a multi-dimensional, semantically labeled structure --
separating data, computation, and views into independent layers. Formulas
reference meaningful names, views can be rearranged instantly, and the same
dataset can be explored from multiple perspectives simultaneously.
The application compiles to a single static binary and provides a rich TUI
experience.
## Non-Goals (v1)
- Scripting/macro language beyond the formula system.
- Collaborative/multi-user editing.
- Live external data sources (databases, APIs).
- Charts or graphical visualization.
- Multi-level undo history.