284 lines
9.9 KiB
Markdown
284 lines
9.9 KiB
Markdown
# Improvise — Multi-Dimensional Data Modeling Terminal Application
|
|
|
|
## Context
|
|
|
|
Traditional spreadsheets conflate data, formulas, and presentation into a single flat grid addressed by opaque
|
|
cell references (A1, B7). This makes models fragile, hard to audit, and impossible to rearrange without
|
|
rewriting formulas. We are building a terminal application that treats data as a multi-dimensional,
|
|
semantically labeled structure — separating data, computation, and views into independent layers. The result
|
|
is a tool where formulas reference meaningful names, views can be rearranged instantly, and the same dataset
|
|
can be explored from multiple perspectives simultaneously.
|
|
|
|
The application compiles to a single static binary (`x86_64-unknown-linux-musl`) and provides a rich TUI
|
|
experience.
|
|
|
|
---
|
|
|
|
## 1. Core Data Model
|
|
|
|
### 1.1 Categories and Items
|
|
- Data is organized into **categories** (dimensions) and **items** (members of a dimension).
|
|
- Example: Category "Region" contains items "North", "South", "East", "West".
|
|
- Example: Category "Time" contains items "Q1", "Q2", "Q3", "Q4".
|
|
- Items within a category can be organized into **groups** forming a hierarchy.
|
|
- Example: Items "Jan", "Feb", "Mar" grouped under "Q1"; quarters grouped under "2025".
|
|
- Groups are collapsible/expandable for drill-down.
|
|
- A model supports up to **12 categories**.
|
|
|
|
### 1.2 Data Cells
|
|
- Each data cell is identified by the intersection of one item from each active category — not by grid coordinates.
|
|
- Cells hold numeric values, text, or empty/null.
|
|
- The underlying storage is a sparse multi-dimensional array (`HashMap<CellKey, CellValue>`).
|
|
|
|
### 1.3 Models
|
|
- A **model** is the top-level container: it holds all categories, items, groups, data cells, formulas, and views.
|
|
- Models are saved to and loaded from a single `.improv` file (JSON format).
|
|
|
|
---
|
|
|
|
## 2. Formula System
|
|
|
|
### 2.1 Named Formulas
|
|
- Formulas reference categories and items by name, not by cell address.
|
|
- Example: `Profit = Revenue - Cost`
|
|
- Example: `Tax = Revenue * 0.08`
|
|
- Example: `Margin = Profit / Revenue`
|
|
- A formula applies uniformly across all intersections of the referenced categories. No copying or dragging.
|
|
|
|
### 2.2 Formula Panel
|
|
- Formulas are defined in a **dedicated formula panel**, separate from the data grid.
|
|
- All formulas are visible in one place for easy auditing.
|
|
- Formulas cannot be accidentally overwritten by data entry.
|
|
|
|
### 2.3 Scoped Formulas (WHERE clause)
|
|
- A formula can be scoped to a subset of items:
|
|
- Example: `Discount = 0.10 * Price WHERE Region = "West"`
|
|
|
|
### 2.4 Aggregation
|
|
- Built-in aggregation functions: `SUM`, `AVG`, `MIN`, `MAX`, `COUNT`.
|
|
|
|
### 2.5 Formula Language
|
|
- Expression-based (not Turing-complete).
|
|
- Operators: `+`, `-`, `*`, `/`, `^`, unary `-`.
|
|
- Comparisons: `=`, `!=`, `<`, `>`, `<=`, `>=`.
|
|
- Conditionals: `IF(condition, then, else)`.
|
|
- `WHERE` clause for filtering: `SUM(Sales WHERE Region = "East")`.
|
|
- Parentheses for grouping.
|
|
- Literal numbers and quoted strings.
|
|
|
|
---
|
|
|
|
## 3. View System
|
|
|
|
### 3.1 Views as First-Class Objects
|
|
- A **view** is a named configuration specifying:
|
|
- Which categories are assigned to **rows**, **columns**, and **pages** (filters/slicers).
|
|
- Which items/groups are visible vs. hidden.
|
|
- Sort order (future).
|
|
- Number formatting.
|
|
- Multiple views can exist per model. Each is independent.
|
|
- Editing data in any view updates the underlying model; all other views reflect the change.
|
|
|
|
### 3.2 Category Tiles
|
|
- Each category is represented as a **tile** displayed in the tile bar.
|
|
- The user can move tiles between row, column, and page axes to instantly pivot/rearrange the view.
|
|
- Moving a tile triggers an instant recalculation and re-render of the grid.
|
|
|
|
### 3.3 Page Axis (Slicing)
|
|
- Categories assigned to the page axis act as filters.
|
|
- The user selects a single item from a paged category using `[` and `]`.
|
|
|
|
### 3.4 Collapsing and Expanding
|
|
- Groups can be collapsed/expanded per-view (future: keyboard shortcut in grid).
|
|
|
|
---
|
|
|
|
## 4. JSON Import Wizard
|
|
|
|
### 4.1 Purpose
|
|
- Users can import arbitrary JSON files to bootstrap a model.
|
|
|
|
### 4.2 Wizard Flow (interactive TUI)
|
|
|
|
**Step 1: Preview** — Structural summary of the JSON.
|
|
|
|
**Step 2: Select Array Path** — If the JSON is not a flat array, the user selects which key path contains the primary record array.
|
|
|
|
**Step 3: Review Proposals** — Fields are analyzed and proposed as:
|
|
- Category (small number of distinct string values)
|
|
- Measure (numeric)
|
|
- Time Category (date-like strings)
|
|
- Label/Identifier (skip)
|
|
|
|
**Step 4: Name the Model** — User names the model and confirms.
|
|
|
|
### 4.3 Headless Import
|
|
```
|
|
improvise --cmd '{"op":"ImportJson","path":"data.json"}'
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Terminal UI
|
|
|
|
### 5.1 Layout
|
|
```
|
|
+---------------------------------------------------------------+
|
|
| Improvise | Model: Sales 2025 [*] [F1 Help] [Ctrl+Q] |
|
|
+---------------------------------------------------------------+
|
|
| [Page: Region = East] |
|
|
| | Q1 | Q2 | Q3 | Q4 | |
|
|
|--------------+---------+---------+---------+---------+--------|
|
|
| Shirts | 1,200 | 1,450 | 1,100 | 1,800 | |
|
|
| Pants | 800 | 920 | 750 | 1,200 | |
|
|
| ... |
|
|
|--------------+---------+---------+---------+---------+--------|
|
|
| Total | 4,100 | 4,670 | 3,750 | 5,800 | |
|
|
+---------------------------------------------------------------+
|
|
| Tiles: [Time ↔] [Product ↕] [Region ☰] Ctrl+↑↓←→ tiles |
|
|
+---------------------------------------------------------------+
|
|
| NORMAL | Default | Ctrl+F:formulas Ctrl+C:categories ... |
|
|
+---------------------------------------------------------------+
|
|
```
|
|
|
|
### 5.2 Panels
|
|
- **Grid panel** (main): Scrollable table of the current view.
|
|
- **Tile bar**: Category tiles with axis symbols. `Ctrl+Arrow` enters tile-select mode.
|
|
- **Formula panel**: `Ctrl+F` — list and edit formulas.
|
|
- **Category panel**: `Ctrl+C` — manage categories and axis assignments.
|
|
- **View panel**: `Ctrl+V` — switch, create, delete views.
|
|
- **Status bar**: Mode, active view name, keyboard hints.
|
|
|
|
### 5.3 Navigation and Editing
|
|
| Key | Action |
|
|
|-----|--------|
|
|
| ↑↓←→ / hjkl | Move cursor |
|
|
| Enter | Edit cell |
|
|
| Esc | Cancel edit |
|
|
| Tab | Focus next open panel |
|
|
| / | Search |
|
|
| [ / ] | Page axis prev/next |
|
|
| Ctrl+Arrow | Tile select mode |
|
|
| Enter/Space (tile) | Cycle axis (Row→Col→Page) |
|
|
| r / c / p (tile) | Set axis directly |
|
|
| Ctrl+F | Toggle formula panel |
|
|
| Ctrl+C | Toggle category panel |
|
|
| Ctrl+V | Toggle view panel |
|
|
| Ctrl+S | Save |
|
|
| Ctrl+E | Export CSV |
|
|
| F1 | Help |
|
|
| Ctrl+Q | Quit |
|
|
|
|
---
|
|
|
|
## 6. Command Layer (Headless Mode)
|
|
|
|
All model mutations go through a typed command layer. This enables:
|
|
- Scripting without the TUI
|
|
- Replay / audit log
|
|
- Testing without rendering
|
|
|
|
### 6.1 Command Format
|
|
JSON object with an `op` field:
|
|
```json
|
|
{"op": "CommandName", ...args}
|
|
```
|
|
|
|
### 6.2 Available Commands
|
|
|
|
| op | Required fields | Description |
|
|
|----|-----------------|-------------|
|
|
| `AddCategory` | `name` | Add a category/dimension |
|
|
| `AddItem` | `category`, `item` | Add an item to a category |
|
|
| `AddItemInGroup` | `category`, `item`, `group` | Add an item in a named group |
|
|
| `SetCell` | `coords: [[cat,item],...]`, `number` or `text` | Set a cell value |
|
|
| `ClearCell` | `coords` | Clear a cell |
|
|
| `AddFormula` | `raw`, `target_category` | Add/replace a formula |
|
|
| `RemoveFormula` | `target` | Remove a formula by target name |
|
|
| `CreateView` | `name` | Create a new view |
|
|
| `DeleteView` | `name` | Delete a view |
|
|
| `SwitchView` | `name` | Switch the active view |
|
|
| `SetAxis` | `category`, `axis` (`"row"/"column"/"page"`) | Set category axis |
|
|
| `SetPageSelection` | `category`, `item` | Set page-axis filter |
|
|
| `ToggleGroup` | `category`, `group` | Toggle group collapse |
|
|
| `Save` | `path` | Save model to file |
|
|
| `Load` | `path` | Load model from file |
|
|
| `ExportCsv` | `path` | Export active view to CSV |
|
|
| `ImportJson` | `path`, `model_name?`, `array_path?` | Import JSON file |
|
|
|
|
### 6.3 Response Format
|
|
```json
|
|
{"ok": true, "message": "optional message"}
|
|
{"ok": false, "message": "error description"}
|
|
```
|
|
|
|
### 6.4 Invocation
|
|
```bash
|
|
# Single command
|
|
improvise model.improv --cmd '{"op":"SetCell","coords":[["Region","East"],["Measure","Revenue"]],"number":1200}'
|
|
|
|
# Script file (one JSON object per line, # comments allowed)
|
|
improvise model.improv --script setup.jsonl
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Persistence
|
|
|
|
### 7.1 File Format
|
|
Native format: JSON-based `.improv` file containing all categories, items, groups, data cells, formulas, and view definitions.
|
|
|
|
Compressed variant: `.improv.gz` (gzip, same JSON payload).
|
|
|
|
### 7.2 Export
|
|
- `Ctrl+E` in TUI or `ExportCsv` command: exports active view to CSV.
|
|
|
|
### 7.3 Autosave
|
|
- Periodic autosave (every 30 seconds when dirty) to `.model.improv.autosave`.
|
|
|
|
---
|
|
|
|
## 8. Technology
|
|
|
|
| Concern | Choice |
|
|
|---------|--------|
|
|
| Language | Rust (stable) |
|
|
| TUI | [Ratatui](https://github.com/ratatui-org/ratatui) + Crossterm |
|
|
| Serialization | `serde` + `serde_json` |
|
|
| Static binary | `x86_64-unknown-linux-musl` via `musl-gcc` |
|
|
| Dev environment | Nix flake with `rust-overlay` |
|
|
| No runtime deps | Single binary, no database, no network |
|
|
|
|
---
|
|
|
|
## 9. Non-Goals (v1)
|
|
|
|
- Scripting/macro language beyond the formula system.
|
|
- Collaborative/multi-user editing.
|
|
- Live external data sources (databases, APIs).
|
|
- Charts or graphical visualization.
|
|
- Multi-level undo history.
|
|
|
|
---
|
|
|
|
## 10. Verification
|
|
|
|
```bash
|
|
# Build
|
|
nix develop --command cargo build --release
|
|
file target/x86_64-unknown-linux-musl/release/improvise # → statically linked
|
|
|
|
# Import test
|
|
./improvise --cmd '{"op":"ImportJson","path":"sample.json"}' --cmd '{"op":"Save","path":"test.improv"}'
|
|
|
|
# Formula test
|
|
./improvise test.improv \
|
|
--cmd '{"op":"AddFormula","raw":"Profit = Revenue - Cost","target_category":"Measure"}'
|
|
|
|
# Headless script
|
|
./improvise new.improv --script tests/setup.jsonl
|
|
|
|
# TUI
|
|
./improvise model.improv
|
|
```
|