fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

bench: scale bisect harness and report (improvise-felo.10)

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (scale-bisect)
2026-10-02 18:19:18 -07:00
committed by Edward Langley
parent 8220a06581
commit 13918bb924
7 changed files with 555 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
context/test-evidence/scale-bisect.md
+218
View File
@@ -0,0 +1,218 @@
# Scale bisect: how big a document does improvise handle today (improvise-felo.10)
Measured on `gh/main` @ 8220a065, **Apple M4 Max, 64 GB** (`sysctl`: `hw.memsize 68719476736`), macOS,
release build (`lto`, `opt-level 3`), native only (no wasm, no real OPFS, no terminal).
Harness: `benches/scale_bisect.rs` (one configuration per process) driven by `bench/scale_bisect.py`,
`bench/sweep.py`, `bench/edge.py`, `bench/full.py`. Reproduce:
`nix develop --command cargo build --release --bench scale_bisect`, copy the binary from
`target/release/deps/scale_bisect-*` to `/tmp/sb`, then `python3 bench/edge.py`.
Budgets (user ruling 2026-10-03): **8 ms is the headline** (120 FPS, hot path or defer to background);
16 ms, 100 ms (noticeable) and 1 s (broken) are reference crossings.
## Method, and what is real
Every number goes through the production path: `App::new` (open: formula recompute +
`GridLayout::with_frozen_records`), `App::dispatch_key(Down)` (**keypress/scroll**: it calls
`rebuild_layout`, then `cmd_context`, which runs `compute_col_widths` over the whole grid, then applies effects),
`project::grid_of` (TUI frame: whole grid; browser: a 45x12 window), `App::apply_effects` with `effect::SetCell`,
`SetAxis`/`HideItem`/`ToggleGroup`/`TogglePruneEmpty` (reshape), `Host::connect` and `Host::handle` with a
`caught_up` `ModelCommand::SetCell` (browser path incl. projection), `format_md`/`parse_md`/`save_to(MemoryFs)`.
Document: C0..C{cats-1} with K items, N distinct random tuples ("records", capped at K^cats/2), two measures
each (2N cells), `_Measure` on Column. Timings are the **median** of up to 3-8 in-process repeats; boundary
rows are the median of **3 separate processes** run sequentially (spread shown). Earlier exploratory sweeps
ran 2-3 processes in parallel and agree with the sequential runs within ~5%.
**`View::prune_empty` defaults to false** (`view/types.rs`), so "prune off" is what a user gets until they
toggle it; "prune on" is reported next to it. Rows below are the layout's `row_count`.
**Dominant cost (reasoned, not profiled):** the release profile strips symbols, so `sample` could not
symbolicate. The reason is in the code and the timings agree with it: `GridLayout::cell_key`/`row_item` do
`data_items(&self.row_items).nth(row)` (`view/layout.rs` ~253/292/807/870), O(row) per lookup, and
`compute_col_widths` (every keypress, via `cmd_context`) and `grid_of` visit every cell, so both are
O(rows^2). Evidence: with prune off, rows 729 -> 5.4 ms, 4096 -> 74.6 ms, 19683 -> 1248 ms (x6.7 rows ->
x14 time -> x17 time). A 45x12 browser window costs **the same** as the whole-grid TUI frame (K=30:
2274 vs 2298 ms) because the widths pass is whole-grid whatever the viewport says, so viewport narrowing buys
nothing today. `GridLayout` build itself is cheap (~90 ns/row, ADR 008).
## Attribution: state+projection vs render/serialize (K=10, 1000 rows, prune off; 3 runs)
| step | ms |
|---|---|
| keypress (state: rebuild + widths + effect) | 9.5-9.7 |
| `grid_of` whole grid (TUI projection) | 11.7-12.0 |
| `GridWidget` render into a 200x50 ratatui `Buffer` | 0.12 |
| `grid_of` 45x12 window (browser projection) | 9.1-9.3 |
| JSON encode of one SetCell's projected updates | 0.001-0.005 (0.6-1.1 KB) |
| JSON encode of the connect Snapshot | 0.02 (21 KB) |
Render and serialize are **<=0.5 ms in every row measured** (GridWidget 0.08-0.53 ms up to 14,500 records;
snapshot encode <=0.07 ms; records-mode SetCell updates 21 KB in 0.02 ms). The 8 ms is entirely state and
projection. Not measured: the rest of `draw()` (panels, status; `draw` is private to the binary), terminal I/O,
browser JSON *decode* and DOM paint.
## Axis B: items per category K, 3 categories on Row, N = 100k (capped at K^3/2), prune OFF
| K | rows | open | keypress | browser window | SetCell | maxrss MB |
|---|---|---|---|---|---|---|
| 5 | 125 | 0.1 | 0.5 | 0.5 | 0.2 | 8 |
| 9 | 729 | 0.2 | **5.4** (5.3-5.5) | 5.0 | 0.9 | 13 |
| 10 | 1,000 | 0.2 | **9.6** (9.5-9.7) | 9.2 | 1.2 | 14 |
| 11 | 1,331 | 0.3 | 12.7 | 12.7 | 1.6 | 10 |
| 12 | 1,728 | 0.3 | 20.0 | 19.3 | 2.1 | 11 |
| 15 | 3,375 | 0.5 | 65 | 64 | 4.0 | 14 |
| 16 | 4,096 | 0.6 | 74.6 (74.3-75.2) | 73.4 | - | - |
| 17 | 4,913 | 0.7 | **105** (105.0-105.5) | 103 | - | - |
| 20 | 8,000 | 1.1 | 302 | 301 | 9.4 | 21 |
| 25 | 15,625 | 1.9 | 773 | 771 | 18.8 | 33 |
| 26 | 17,576 | 2.2 | 967-977 | 962 | - | - |
| 27 | 19,683 | 2.4 | **1,248** (1,244-1,251) | 1,244 | - | - |
| 30 | 27,000 | 3.1 | 2,262 | 2,274 | 31 | 49 |
| 40 | 64,000 | 6.7 | 10,858 | 10,833 | 73 | 235 |
| 60 | 216,000 | 21.6 | timeout (>60 s) | - | - | - |
| 1000 | 10^9 | **refused** (see below) | | | | |
Hot-path keypress/scroll and browser window: **largest inside 8 ms: K=9 (729 rows)**; inside 16 ms: K=11;
first over 100 ms: **K=17 (4,913 rows)**; first over 1 s: **K=27 (19,683 rows)**. SetCell alone
(no widths pass; layout rebuild only, ~linear in rows): inside 8 ms to K~19 (K=15 4.0, K=20 9.4), 16 ms ~K=24,
100 ms ~K=45 (K=40 = 73 ms), 1 s extrapolated ~K=95 (**extrapolated, not measured**). Open stays <25 ms to K=60.
**3 x 1,000 on Row cannot be opened at any N**: the layout materializes the cross product (10^9 rows, ~150 B
each = ~150 GB, ADR 008). The harness refuses above 30M rows (`rows_cap`) rather than swap the machine, so
the 10^9 number is the ADR's estimate, not a measurement here.
### Prune ON (same documents)
| K | rows | open | keypress | browser window | SetCell |
|---|---|---|---|---|---|
| 7 | 171 | 2.0 | 4.5 (4.4-4.6) | 1.1 | 5.4 |
| 8 | 256 | 3.0 | **7.0** (7.0-7.1) | 1.6 | 8.5 |
| 9 | 364 | 5.3 | 12.8 | 2.7 | 15.3 |
| 11 | 665 | 12.2 | 29.6 | 6.2 | ~28 |
| 13 | 1,098 | 29.0 | 71 | 14.5 | ~70 |
| 14 | 1,372 | 48 | **119** | 24 | - |
| 20 | 4,000 | 300 | 747 | 145 | 903 |
| 21 | 4,630 | 401 | 968-992 | 190 | - |
| 25 | 7,812 | 767 | 1,899 | 366 | 2,328 |
| 30 | 13,500 | 2,255 | 5,555 | 1,047 | 6,804 |
Prune on: keypress inside 8 ms to **K=8**, first over 100 ms **K=14**, ~1 s at **K~21-22** (K=22 measured
0.93-0.95 s, a different N cap; first clearly over: K=25). Pruning shrinks the grid, so the browser window is
cheaper than prune off at the same K (K=20: 145 ms against 302 ms), but **every keypress re-runs
the prune** (`dispatch_key` calls `rebuild_layout`), so keypress and SetCell are *worse* with prune on
(K=10: 23 vs 9.9 ms). Prune is also quadratic (`nth`). Conclusion: prune on/off moves the boundary by about
one K at 8 ms and makes 100 ms/1 s *earlier* (K=14/21 vs 17/27); it never rescues the 3x1000 case.
## Axis A: records N at fixed K (K=30, 27,000 rows)
| N | prune off keypress | prune on rows | prune on keypress | prune on window |
|---|---|---|---|---|
| 100 | 451 | 100 | 861 | 0.3 |
| 1,000 | 527 | 1,000 | 1,059 | 6.4 |
| 5,000 | 1,013 | 5,000 | 2,178 | 140 |
| 13,500 (cap) | 2,267 | 13,500 | 5,555 | 1,047 |
Prune off: ~450 ms of widths over 27k empty rows + ~1.3 us per stored cell; **N never gets the keypress
under 100 ms at K=30**, and at K=1000 N is irrelevant because the layout cannot be built (rows = K^3).
The N-axis has no bisect point in the 3x1000 regime; N only matters once K is small enough that rows fit, and
there it is a second-order term. Open is flat (3.1-3.6 ms) in N with prune off.
## Axis C: categories on Row (prune off)
| K | cats | rows | open | keypress | note |
|---|---|---|---|---|---|
| 10 | 2 | 100 | 0.1 | 0.2 | |
| 10 | 3 | 1,000 | 0.2 | 9.9 | over 8 ms |
| 10 | 4 | 10,000 | 1.4 | **880** | ~1 s |
| 10 | 5 | 100,000 | 14 | timeout >100 s | |
| 3 | 6 | 729 | 0.3 | 27.9 | 5x worse than K=9 x 3 cats at the same rows |
| 3 | 8 | 6,561 | 2.0 | 2,798 | |
| 3 | 10-12 | 59k-531k | 18-210 | timeout | open alone passes 100 ms at 11 cats |
| 2 | 10 | 1,024 | 0.6 | 172 | |
| 2 | 12 | 4,096 | 2.2 | 3,190 | |
| 2 | 14 | 16,384 | 9 | 63,100 | |
The cap of 24 is unreachable: rows = product of items, so a view with c Row categories of K items dies by
K^c ~ 10^4 rows (1 s) regardless of N, and with two items per category by 12-14 categories. Per-row cost
also grows with category count (each row_item is a Vec of c items). Not measured: Row categories with prune
on, other axes absorbing categories (the realistic mitigation: only 1-2 on Row).
## Axis D: records mode over N records (`with_frozen_records`, the improvise-sk32 quadratic)
| N | open | keypress | browser window | SetCell (apply) | host SetCell+project | maxrss MB |
|---|---|---|---|---|---|---|
| 300 | 1.5 | 0.7 | 0.1 | 1.6 | - | 10 |
| 1,000 | 2.9 | 5.6 (5.5-5.7) | 0.3 | 10 | - | 12 |
| 1,200 | 3.9 | **7.9** (7.7-8.0) | 0.3 | - | - | - |
| 1,300 | 4.5 | 9.3 | 0.3 | 15.5 | 17 | 26 |
| 1,800 | 8.4 | 16.5 | 0.4 | - | - | - |
| 3,000 | 22.9 | 47 | 0.7 | 74 | - | 20 |
| 4,400-4,600 | 47-51 | **95-102** | 0.9 | 149 | 158 | 55 |
| 10,000 | 248 | 492 | 1.8 | 734 | - | 46 |
| 14,000-14,500 | 472-500 | **940-1,002** | 2.6-2.8 | 1,435 | 1,459 | 135-148 |
| 30,000 | 2,198 | 4,314 | 5.6 | 6,467 | - | 125 |
| 100,000 | **25,420** | **51,234** | 19.3 | not reached (killed at 100 s) | - | not reached |
Records mode, keypress/scroll: inside 8 ms **to N~1,200**; 16 ms N~1,800; 100 ms N~4,500 (first over: 4,600 =
102 ms); 1 s N~14,500. Open crosses 8 ms ~1,750, 16 ms ~2,300, 100 ms ~6,500, 1 s ~20,000. Time ~ N^1.95.
SetCell is ~1.5x keypress (it rebuilds and re-groups). The *browser window* is cheap (<20 ms even at 100k)
because records mode has two columns and the width walk is short, so the 51 s is `rebuild_layout`, not
projection. Prune on/off does not change records mode (prune applies to pivoted axes; the sweep ran off).
Records per KB: ~4 KB/record process-wide at 14k (maxrss includes the document generator, so read as an
upper bound); the 100k memory was not obtained.
## Axis E: save / load and per-command persistence
Mode `none` (all categories off-axis, so layout is trivial and only the store scales), 3 x 1,000, N records
= 2N cells:
| cells | format_md | text MB | parse_md | persist per accepted command (`save_to` MemoryFs) | maxrss MB |
|---|---|---|---|---|---|
| 2,000 | 13.8 | 0.1 | 4.5 | 12.4 | 18 |
| 20,000 | 156 | 0.86 | 37 | 161 | 80 |
| 60,000 | 527 | 2.5 | 119 | 539 | 236 |
| **200,000 (the target)** | **1,974** | 8.4 | **401** | **1,984** | 663 |
| 600,000 | 6,470 | 25 | 1,320 | 6,460 | 1,894 |
| 2,000,000 | 23,120 | 84 | 4,668 | 23,088 | 5,611 |
Linear: ~10 us/cell to format, ~2 us/cell to parse. Crossings, format_md: 8 ms ~700 cells, 16 ms ~1.5k,
100 ms ~11k, 1 s ~100k cells. parse_md: 8 ms ~4k cells, 100 ms ~50k, 1 s ~450k. **At the target (200k cells)
every accepted command in the browser worker costs ~2 s of format alone** before OPFS write time (not
measured: MemoryFs only). Persist per command also appears in smaller documents inside the grid axes
(K=10 grid: 5.5 ms; records N=4,400: 63 ms).
## Other operations (3 runs each; "grouped" fixtures start with C0 groups collapsed, so rows are not K^3)
Reshape (prune off, K=10, 1000 rows): hide item 0.25 ms, set axis 0.19 ms, **toggle prune 12.7 ms**.
K=26 grouped, 2,028 rows: hide 0.5, toggle group 1.8, set axis 1.8, **toggle prune 727-750 ms**.
Prune on K=13 grouped, 321 rows: hide 9.4, toggle group 37, set axis 37, toggle prune 37-39 ms.
`Host::connect` snapshot: K=10 6.5 ms, K=26 375 ms, records N=14,000 2.9 ms. Host SetCell+project: K=10 7.7 ms,
K=26 ~400 ms, records N=4,400 158 ms (equal to apply + a whole-grid width walk).
## Classification and feasibility of yielding
| operation | class | today's first size over 8 ms | dominated by | chunked/yielding form |
|---|---|---|---|---|
| keypress / scroll | **hot (<=8 ms)** | 1,000 rows (K=10) / N=1,300 records | `rebuild_layout` + whole-grid `compute_col_widths` (O(rows^2) `nth`), recomputed though nothing changed | not a slicing problem: cache widths/layout across keys, O(1) row lookup (ADR 008/009). Slicing would only hide it |
| browser window projection | **hot** | 1,000 rows | same width walk, whole grid regardless of window | needs the same fix; then window cost is O(window) |
| SetCell apply + project | **hot** | K~19 (prune off), 1,300 records | layout rebuild O(rows) + widths + projection | incremental update of one row/total, not a rebuild |
| column totals | hot | inside `grid_of` (not separable here) | whole-grid sum in the projection | maintained total (ADR 009) |
| open / layout build | deferrable | 40k rows (prune off); N~1,750 (records) | build is ~90 ns/row (sliceable by outer index); the prune/records grouping is the quadratic part | build is feasible in slices once the quadratic is removed; the quadratic itself is not sliceable |
| reshape (hide/collapse/axis) | deferrable | cheap prune-off (<2 ms to ~2k rows); 37 ms prune-on at 321 rows | layout rebuild + prune | one sort/remap at 100k occupied tuples was 2.3 ms in ADR 008's prototype, so a reshape can fit inline; prune toggle needs the quadratic fixed first |
| save (format_md) | deferrable | ~700 cells | per-cell formatting, linear | the Data section is a flat per-cell line stream, so chunking by cell range with a yield every ~400 cells looks feasible, but needs a consistent snapshot (clone or journal); not profiled by function |
| load (parse_md) | deferrable | ~4k cells | pest parse + tree walk, linear | tree walk is chunkable by line; the pest parse of one text is not, unless the Data section is parsed line by line (grammar is line oriented); not profiled by function |
| persistence per command | deferrable (debounce) | ~700 cells | the whole-document `format_md` | coalesce/debounce or append-journal: formatting 2 s per command at target is unusable inline at any slicing |
## What I could not measure
* wasm: all numbers are native; the browser worker is typically slower. Real OPFS I/O, JSON *decode* and
DOM paint, the full `draw()` frame, and Safari are not covered.
* Records mode at 100k: SetCell, host and RSS (killed after open and keypress, ~76 s).
* Per-function profile: symbols are stripped in `[profile.release]`; attribution is reasoned from the
code and the timing exponents, not a flame graph.
* SetCell 100 ms/1 s for prune off past K=40 are extrapolations.
* Memory (`maxrss`) includes the generator's hash set and `recs` vector; it is an upper bound on the
document. Layout cost ~150 B/row per ADR 008, consistent with K=40 (64k rows, 235 MB peak with prune off
including the generator and a second layout during rebuild).
* Noise: boundary rows are 3 sequential processes, spread <=3% (e.g. K=17: 105.0-105.5 ms). Non-boundary
rows are single runs, some taken with 2-3 concurrent processes on a 16+ core machine; treat them as +-5-10%.