From e09ec752261f8de48378a3de60afb308dd338a1f Mon Sep 17 00:00:00 2001 From: "Claude (agent-hier)" Date: Thu, 24 Sep 2026 21:54:23 -0700 Subject: [PATCH] docs: group hierarchies and fold lines, with the evidence for each rival (improvise-felo.6) The harness fixtures open their groups explicitly now that groups start closed. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze --- context/code/core.md | 133 +++++++++++++++++++++++- context/code/formula-and-io.md | 26 +++++ context/code/protocol.md | 29 +++++- context/test-evidence/command.md | 13 +++ context/test-evidence/core.md | 53 +++++++++- context/test-evidence/formula-and-io.md | 31 ++++++ context/test-evidence/protocol.md | 8 ++ context/test-evidence/tui.md | 15 +++ harness/fixtures/grouped.improv | 2 + harness/fixtures/tall.improv | 2 + 10 files changed, 304 insertions(+), 8 deletions(-) diff --git a/context/code/core.md b/context/code/core.md index ed64269e..b4d667c1 100644 --- a/context/code/core.md +++ b/context/code/core.md @@ -258,9 +258,11 @@ user drilled in to see all the raw data. On an axis with two grouped categories the runs **nest**: `North` in `Upper` holds `Jan` in `Q1`, and after `Q1` ends `North/Mar` is still in `Upper`. So a -marker names its level. `GroupHeader { level, .. }` opens a run at `level` and -closes every deeper one; `GroupEnd { level }` closes `level` and every deeper -one. `expand_category` reads the level off its prefix, which holds one item per +marker names its level. `GroupHeader { level, depth, .. }` opens a run at +`(level, depth)` and closes every deeper one; `GroupEnd { level, depth }` closes +that run and every deeper one. "Deeper" is the ordering of the pair: the level +first, then the depth inside one category's hierarchy (§19, improvise-felo.6). +`expand_category` reads the level off its prefix, which holds one item per outer category, so it is derived rather than passed. A row's group, for `row_group_for`/`col_group_for` and so for `z`, is the **innermost** run open at it. That is the user's ruling on the ticket: membership is per level, and @@ -1556,3 +1558,128 @@ editor stays open over the user's text (gotcha 21). `was` is not redundant with a stable id: a concurrent *replace* keeps the id and changes the text, which only `was` can see. + +## 19. Group hierarchies: parents nest runs, and every group starts closed (improvise-felo.6) + +`Group::parent` was parsed (`> Q1[2024]`), written, round-tripped and hashed, +and nothing read it: `Q1` was never drawn inside `2024`. It is now the +hierarchy. + +### The user's rulings + +- "I think hierarchies should start closed unless we somehow opt into a + specific one opened to a certain depth (view config maybe?)" — and the + reason: "in most cases, the closed by default start is preferable because it + makes the grid more dense." +- "Everything opens closed, including old files." There is no version gate: a + file written before this change opens with its groups closed, and its + `collapsed:` lines simply agree with the default. +- "I think we'll need a depth for expanded, something like expanded(2): ...". +- "expanded+collapsed in the same group is an error". + +### The model + +- **A group's chain** is the group and its ancestors, root first + (`Category::group_chain`). An item's chain is its group's + (`Category::item_chain`); an ungrouped item has none. A group's depth is its + number of ancestors. +- **A group covers every leaf beneath it.** `items_in_group` reads each item's + chain, so `2024` covers the months of `Q1-2024` and `Q2-2024`. That is the + whole change to evaluation: `evaluate_covering` already expanded a group + coordinate through `items_in_group` (§14), so a year's fold is the fold over + its months with no second path. +- **An item may sit directly in a parent group, beside its child groups.** + Chosen because nothing is simpler: the chain handles it (`Jun[H1]` after a + run of `Q2` inside `H1` ends the `Q2` run and stays in `H1`), and refusing it + would be a rule with nothing to protect. A date import never produces one. +- **A cycle is refused on load, naming its members** (`Refusal::HierarchyCycle`, + `persistence/parse.rs`). A cycle built in code cannot hang anything either: + `group_chain` stops where it would repeat. +- **There is no such thing as a dangling parent in this format.** A root group + is declared *only* by being named as a parent: the writer emits `>` lines + only for groups that have a parent, and nothing else can name `2024`. So + "a parent that names no group" and "a root" are the same line, and refusing + one refuses the other. + +### Which groups are open + +State lives on the `View`, in two places that cannot disagree about one group: + +- `expand_depth: cat → N` — `expanded(N): Cat`, the category's top N levels + open. +- `group_folds: cat → group → GroupFold` — ONE map, so a group has at most one + fold. `GroupFold::Collapsed` is `collapsed: Cat/Group`; + `GroupFold::Expanded(N)` is `expanded(N): Cat/Group`, the group open and + N-1 levels beneath it. Two sets would have made "both" representable, which + is the state the user ruled an error + (`doctrine(an_invariant_is_made_unrepresentable_not_validated)`). + +`View::is_group_collapsed(cat, chain)` answers, in order: + +1. the group's own fold, if it has one — so a `collapsed:` group inside an + `expanded(N)` region stays collapsed, and an `expanded:` group below the + category's depth opens; +2. open, if an ancestor's `Expanded(n)` reaches it (the ancestor `k` levels up + reaches it when `k < n`), or the category's depth does (`depth < N`); +3. closed. + +**`z` writes one fold, or none.** `toggle_group_collapse` flips the answer and +records the flip as `Collapsed` or `Expanded(1)` — unless the flip lands on +what the group would inherit from rule 2, in which case it removes the group's +fold. So `z z` leaves the view as it was, and no sequence of toggles writes a +file that names one group twice +(`no_sequence_of_toggles_writes_a_conflicting_file`). Closing a group that +carried `Expanded(3)` drops the 3: reopening it with `z` opens one level. + +### The layout + +`expand_category` walks each drawn item's chain. The **outermost** collapsed +group in the chain folds everything beneath it: the row's tuple is that group's +name, it folds once however non-contiguous its members are (§14), and the +headers above it are drawn. The run markers move from item to item exactly as +`remark` moves them after pruning — past the shared prefix of the two chains, +end the deeper runs innermost-out, then open what differs — so an unpruned axis +survives `prune_axis_items` marker for marker, and the enumeration in +`group_membership` (now with a `NESTED` fixture: a non-contiguous quarter, an +item directly in a half-year, an outer grouped category) asserts it. + +A header carries its own `collapsed`, computed once in `expand_category`, so +the frame projection and the frame hash read it rather than asking the view +again with a chain they would have to rebuild. + +### Persistence + +`expanded(N): Cat`, `expanded(N): Cat/Group`, bare `expanded:` for N = 1, and +the existing `collapsed: Cat/Group` (`improv.pest`, `expanded_line`). N is at +least 1 by the grammar (`'1'..'9' ~ ASCII_DIGIT*`), so `expanded(0)` is a parse +error rather than a value to explain. Two further decisions: + +- **The writer keeps what the view holds.** An old file's `collapsed:` lines + load as `Collapsed` folds and are written back verbatim, though they agree + with the default. They are the user's explicit closes, and they survive a + later `expanded(N): Cat` that would otherwise open those groups; dropping + them on save would change what that later line means. A `z` round trip + removes such a line, because `z` normalises (above). +- **A category depth is never written bare.** `axis_line` is tried before + `expanded_line`, so a category literally named `expanded` keeps its + `expanded: row`. The price is that `expanded: row` meaning "category `row`, + depth 1" would read as that axis line, so the writer always spells the + category form `expanded(N): Cat`. The group form has a `/` and is never + ambiguous. + +A view naming one group with two different folds, or one category with two +depths, is refused on load, naming the view and the group +(`Refusal::FoldConflict`). The same line twice is not a conflict. + +The grammar-walking generator emits cycles and conflicts with ordinary +probability (`> ||[||]` is a self-parent), so `generated_file_parses` accepts a +`Refusal` — a typed error, so it cannot accept a parse failure by accident. + +### Known limits + +- The TUI draws nested headers exactly as it draws any other: one line per + header, not indented by depth, and the column band shows each column's + innermost group only. Both are pre-existing presentation choices; the frame + carries `depth` for the day a front end wants to indent. +- Two `>` lines giving one group two different parents: the last wins, as it + did before this change. That makes the hierarchy a function of line order. diff --git a/context/code/formula-and-io.md b/context/code/formula-and-io.md index 2d07d8e6..3e101504 100644 --- a/context/code/formula-and-io.md +++ b/context/code/formula-and-io.md @@ -218,6 +218,32 @@ times. The probe was removed rather than kept: its rate depends on proptest's internal RNG, so as a permanent assertion it would fail on a dependency bump for a reason unrelated to the format. +### Folds are view lines: `collapsed:` and `expanded(N):` (improvise-felo.6) + +Groups start collapsed, so a view persists what it opens. `expanded(N): Cat` +opens a category's top N levels, `expanded(N): Cat/Group` opens that group and +N-1 levels beneath it, a bare `expanded:` is N = 1, and `collapsed: Cat/Group` +keeps its old meaning. The semantics, the precedence and the refusals are +`context/code/core.md` §19; what is particular to the format: + +- **No version bump.** The line is additive, and the user ruled that old files + open closed like new ones, so `v2025-04-09` still reads every file it did. +- **N >= 1 is grammar, not validation**: `depth_number = @{ '1'..'9' ~ + ASCII_DIGIT* }`. Spelled with a `Range` and `ASCII_DIGIT` rather than + `ASCII_NONZERO_DIGIT`, because the generator knows the first two and would + emit nothing for the third — `expanded(): ` — a file its own grammar refuses. +- **`axis_line` comes before `expanded_line`** in `view_entry`, so a category + named `expanded` keeps its axis line, and the writer never spells a category + depth bare (`expanded: row` would be that axis line). +- **Two refusals the grammar cannot express** — a cycle among `>` lines and one + group given two folds — are a typed `Refusal`, which `generated_file_parses` + accepts and nothing else does. The generator produces both at ordinary rates. +- **The generator reaches the new rule**, asserted by + `the_generator_writes_expanded_lines_with_and_without_a_depth` over 512 + deterministic files. Unlike the kind-line probe above, it asserts that each + spelling occurs at all, not a rate, so a proptest bump fails it only if the + walk stopped reaching the rule. + ### The tree walkers are infallible by construction Every `build_*` in `parser.rs` operates on a `Pair` pest has already diff --git a/context/code/protocol.md b/context/code/protocol.md index 81151eb8..044ddb9c 100644 --- a/context/code/protocol.md +++ b/context/code/protocol.md @@ -1474,6 +1474,22 @@ sequence. The header's level also gives a client the group's category, the dead field improvise-7xyx removed: `level` is read by `groups()`, and the category name is not carried twice. +**Within one category runs nest by `depth` (improvise-felo.6).** A group's +parent (`> Q1[2024]`) puts the quarter's run inside the year's on the SAME +level, so both markers also carry `depth`, the group's number of ancestors. +A run's key is `(level, depth)`, compared in that order: a header at a key +closes every run with a greater key, and so does a `GroupEnd`. With `Date` +alone on an axis, `2024-03` after `Q1-2024` ends sits in `2024`, and after +`2024` ends in nothing. Depth rides beside `level` rather than being folded +into one flat index, because a flat index would need every category's tree +height to be known before the first marker, and `level` would stop naming +`cats[level]`. Both front ends draw a nested header exactly as they draw any +other: one inline row per `Group` entry, the innermost group in the column +band. `the_two_front_ends_draw_the_same_nested_hierarchy_on_rows` and +`..._on_columns` in `tests/grid_frame_agreement.rs` hold them together, and +`the_frame_agrees_with_the_layout_inside_one_categorys_hierarchy` does the same for the +windowed frame against the layout. + `group_runs` is unchanged. It coalesces whatever `groups()` answers, by header identity (improvise-h0iv), so the band still paints one group per position, now the innermost. One band per level would be a separate feature. @@ -1554,14 +1570,19 @@ runs. Reversing it is one comparison, and the test is renamed ## §3b. A frame's levels are checked as it arrives -A `FrameGroup.level` and a `GroupEnd { level }` name a category: they index -`cats`. `AxisFrame::groups` keeps one open run per level and sizes that stack -with `open.resize(g.level, None)`. So a level arriving over the wire decides how -much it allocates. A frame whose level is `4294967295` would ask for a +A `FrameGroup.level` and a `GroupEnd`'s `level` name a category: they index +`cats`. `AxisFrame::groups` used to keep one open run per level and size that +stack with `open.resize(g.level, None)`, so a level arriving over the wire +decided how much it allocated. A frame whose level is `4294967295` would ask for a multi-gigabyte vector, and the client would be taken down instead of the frame being declined. ChatGPT Codex found this on #250, the PR that gave levels to the wire (improvise-e265). +Since improvise-felo.6 `groups()` keeps only the headers it has received, +ordered by `(level, depth)`, so no number on the wire sizes anything and the +new `depth` field needs no bound. The level check stays for its other reason: +`cats[level]` names the group's category, which a browser `z` reads. + The bound is enforced at deserialization, by `#[serde(try_from = "UncheckedAxisFrame")]`. That is the one place a frame enters from outside. A level that is not below `cats.len()` fails to parse, with an error that names the diff --git a/context/test-evidence/command.md b/context/test-evidence/command.md index 0444831b..08ce94cb 100644 --- a/context/test-evidence/command.md +++ b/context/test-evidence/command.md @@ -7214,3 +7214,16 @@ cleared the cell cannot pass. `the_shipped_demo_seeds_a_worker_whose_raw_cells_are_settable` wrote with `view: "Default"` against `web/demo.improv`, whose view is `Plan`. It passed only because a stored record ignored the view name. It now names `Plan`. + +### the_frame_agrees_with_the_layout_inside_one_categorys_hierarchy + +improvise-felo.6. The windowed frame carries the runs open above the window; +those are now keyed by `(level, depth)`. Rival: `axis_frame` closing carried +runs by level alone. Only this test fails; the existing agreement test has one +group per level, where the two keyings coincide. + +### z_opens_a_closed_hierarchy_one_level_at_a_time_and_closes_it_again + +`z` through a closed Year > Quarter > Month hierarchy, cursor at row 1. Rival: +`toggle_group_collapse` never removing a fold; the final assertion (no fold left +behind once the quarter is closed again) fails. diff --git a/context/test-evidence/core.md b/context/test-evidence/core.md index fb4b2e6e..d7d99df7 100644 --- a/context/test-evidence/core.md +++ b/context/test-evidence/core.md @@ -1865,5 +1865,56 @@ were run against the whole `improvise-core` suite: **What it cannot see is the allocation itself.** A visitor that clones the stack at every row returns the same answers, so no assertion here can tell the two apart without a counting allocator. The copy-free property is held by the -shape: the visitor receives a borrowed `&[Option]`, and no owned stack +shape: the visitor receives a borrowed slice of open runs (`&[OpenRun]` since +improvise-felo.6), and no owned stack leaves the walk. It is not held by a test. + +### a_parent_group_covers_every_leaf_beneath_its_child_groups + +improvise-felo.6. `items_in_group` read only the items whose own `group` named +the group, so a year covered nothing: its months sit in quarters. Rival run +against `improvise-core`: the direct-only filter restored. Four tests fail — +this one, `a_hierarchy_opens_closed_as_one_fold_per_root` (the 2024 fold reads +empty), `collapsing_a_parent_folds_its_children_whatever_their_state` and +`hidden_items_leave_every_fold_above_them`. The fixture adds `2024-03` last, so +the year and its first quarter are non-contiguous in insertion order. + +### a_year_quarter_month_hierarchy_nests_its_runs + +improvise-felo.6: the whole marker sequence of a Year > Quarter > Month axis +opened to depth 2, compared exactly. Rival: every header inside a category +emitted at depth 0 (a flat hierarchy). This test fails, and so do +`pruning_keeps_every_surviving_rows_runs_at_every_level` (the `NESTED` fixture), +`the_nested_fixture_draws_runs_inside_one_category` and +`pruning_drops_an_empty_inner_group_and_keeps_its_parent`. + +### a_hierarchy_opens_closed_as_one_fold_per_root + +The user's ruling on improvise-felo.6: groups start closed. Two rivals fail it. +The old open default (`inherits_collapsed` always false) fails twelve tests +across `improvise-core` and `improvise-io`, this one among them. The INNERMOST +collapsed group folding instead of the outermost (`.rev()` on the fold search) +fails this one, `hidden_items_leave_every_fold_above_them` and +`the_group_under_the_cursor_is_the_innermost_at_every_depth`: under it a closed +year shows its quarters as rows. + +### a_groups_depth_opens_it_and_n_minus_one_levels_beneath + +`expanded(2): Time/2024` opens 2024 and its quarters, not its months. Rival: an +ancestor reaching `k <= n` levels instead of `k < n`. Only this test fails, which +is why it exists: the boundary is the whole semantics of N. + +### a_toggle_overrides_what_a_group_inherits_either_way_and_toggling_back_clears_it + +`z` writes `Collapsed` or `Expanded(1)`, or removes the fold where the flip +lands on what the group inherits. Rival: never remove. This test and +`command::cmd::grid::tests::z_opens_a_closed_hierarchy_one_level_at_a_time_and_closes_it_again` +fail; every layout test passes, since a redundant fold draws the same grid. What +the rival breaks is the file (a line per toggle, forever) and `z z` being the +identity on the view. + +### pruning_drops_an_empty_inner_group_and_keeps_its_parent + +`prune_empty` at depth: an empty quarter goes, header and all, and its year's +header stays for the sibling quarter that has data. Rival: `prune_axis_items` +keying open runs by level alone. This test and the `NESTED` enumeration fail. diff --git a/context/test-evidence/formula-and-io.md b/context/test-evidence/formula-and-io.md index 3ef8badc..717ba987 100644 --- a/context/test-evidence/formula-and-io.md +++ b/context/test-evidence/formula-and-io.md @@ -1617,3 +1617,34 @@ that name, which must be refused. It was green on arrival, so its evidence is the reviewer's rival, **date-derived names dropped from the taken names**, which fails it. That fault would have let `add_category` reuse the existing category, and every cell key would then name `when_Year` twice. + +### a_group_both_expanded_and_collapsed_is_refused_naming_view_and_group + +The user's ruling on improvise-felo.6. Three spellings of one group given two +folds, each refused as `Refusal::FoldConflict` naming the view and the group, +plus a category given two depths. Rival: a second fold for a group silently +wins. Only this test fails. Its control is +`a_collapsed_group_inside_an_expanded_region_loads`: a DIFFERENT group +collapsed inside an expanded region, and the same line twice, both load. + +### a_cycle_among_group_parents_is_refused_naming_it + +`> Q1-2025[|2025|]` and `> |2025|[Q1-2025]`. Rival: the cycle check disabled; +only this test fails. The fixture's first version spelled `2024-01` bare, which +the grammar refuses (a bare name starts with a letter), so every test on it +passed or failed for the parse error, `a_malformed_depth_is_a_parse_error` +vacuously. It now carries a control that parses. + +### an_axis_line_for_a_category_named_expanded_is_still_an_axis_line + +`axis_line` is tried before `expanded_line`, so the writer must never spell a +category depth bare. Rival: the writer writing `expanded: Cat` for depth 1. This +test fails: the category `row` at depth 1 reads back as an axis line for a +category named `expanded`. + +### an_old_file_with_only_collapsed_lines_loads_closed_and_keeps_its_lines + +"Everything opens closed, including old files." Every group of the file opens +closed, the one its `collapsed:` line names and the three it does not, and the +line is written back. Rival: the old open default; this test fails with the +other eleven in that rival's list. diff --git a/context/test-evidence/protocol.md b/context/test-evidence/protocol.md index 8b1ead78..b5583639 100644 --- a/context/test-evidence/protocol.md +++ b/context/test-evidence/protocol.md @@ -1259,3 +1259,11 @@ the Enter commit. - **Rival** (the commit naming `state.view.active_view`, the view at `Enter`), run across the whole workspace: fails **only** this test, with `["Later"]` against `["Other"]`. + +### a_quarters_end_leaves_its_year_open_on_the_same_level + +improvise-felo.6: runs inside one category nest by `(level, depth)`. `Direct` +sits in `2024` after `Q1`'s `GroupEnd { level: 0, depth: 1 }`, and `Undated` +in nothing after `2024` ends. Rival: `groups()` closing by level alone, as it did +before depth existed. Only this test fails in `improvise-protocol`: every other +frame fixture puts one group per level. diff --git a/context/test-evidence/tui.md b/context/test-evidence/tui.md index 69cf6582..2e80651c 100644 --- a/context/test-evidence/tui.md +++ b/context/test-evidence/tui.md @@ -1429,3 +1429,18 @@ to the list's header row, which no item can cover. This test fills three rows with a three-line item and a second item after it. It asserts the first item is drawn whole and the header reads `1–1 of 2`. The rival **the header claims every item shown** fails it. + +### the_two_front_ends_draw_the_same_nested_hierarchy_on_rows + +improvise-felo.6: Year > Quarter > Month in one category, under a grouped outer +category, with a year open two levels, a quarter closed and a year folded. The +precondition pins the header sequence, so the agreement cannot pass on a frame +that drew no nested header. What makes the two ends agree is that neither +derives membership: both walk the same `FrameEntry` list. Rival: the browser's +`render.rs` skipping a `Group` entry whose `depth > 0`. This test fails on the +row count (the TUI draws the quarter line, the browser does not); the column +test does not, since the band shows only each column's innermost group, which +the rival leaves alone. Group names are short because the TUI +truncates a band label to its column's width, which the browser does not: the +first version named its years `Y2024` and the column test failed on `▶ Y2…` +against `▶ Y2025`, a truncation, not a disagreement. diff --git a/harness/fixtures/grouped.improv b/harness/fixtures/grouped.improv index 164bce39..8e8d6005 100644 --- a/harness/fixtures/grouped.improv +++ b/harness/fixtures/grouped.improv @@ -11,6 +11,8 @@ _Measure: page, Revenue _Index: none _Dim: none collapsed: Region/Upland +expanded(1): Month +expanded(1): Region format: ,.0 ## Category: Region diff --git a/harness/fixtures/tall.improv b/harness/fixtures/tall.improv index 5daa32d2..c16fc14d 100644 --- a/harness/fixtures/tall.improv +++ b/harness/fixtures/tall.improv @@ -8,6 +8,7 @@ _Dim: none _Measure: page Region: row format: ,.0 +expanded(1): Region ## View: Tall _Index: none @@ -15,6 +16,7 @@ _Dim: none _Measure: column Region: row format: ,.0 +expanded(1): Region ## Category: _Measure - Revenue, Cost