improvise/context/adr/007-stable-identities-amendment-1.md

ADR 007, amendment 1 — Stable identities

Ticket: improvise-sboz (this amendment improvise-cq1q.1; implementation epic improvise-cq1q) · Builds on: ADR 001, ADR 002

Status

Accepted (2026-10-09, PR #326). It amends ADR 007 (accepted 2026-09-25, PR #210, 007-stable-identities.md). This file states only what the amendment changes or adds, numbered by the accepted ADR's clauses; everything else in ADR 007 is still true. Where the amendment replaces an ADR 007 rule or an existing behaviour:

clause	accepted rule or existing behaviour	the amendment
1	groups have no id	a group has an id and a display, and its id behaves like every other id
1	_index, _dim, _measure are the only ids beginning with _	a leading _ is reserved for virtual ids in general, including the virtual views _records and _drill
1	the virtual categories are declared and named in the file (## Category: _Measure, _Measure: column)	they are defined by criteria with hardcoded ids and displays and referred to by id (_measure); the ## Category: _Measure header lists the stored measures but declares nothing
1	measure_item_names merges a formula into a stored measure of the same name	a formula never merges: it gets its own id, profit-<n> beside a stored profit
2	a display already in use in a scope is refused	equal displays in a scope are shown with a gapless #2, #3, … suffix in id order
2	CellKey pairs are (CategoryId, ItemId)	(CategoryId, ItemHandle)
4	ids sort by plain string order (CellKey::new sorts with String::cmp, and the writer orders data lines by their text)	ids sort in natural order. CellKey's canonical coordinate order changes with it, so RecordHash values change (they are on the wire, not in files), and a saved file's data lines and their coordinates are written in the new order; the loader canonicalises keys, so the data read back is the same
4	records mode string-sorts the store on entry (DataStore::sort_by_key, run by SortData)	records-mode rows come out in natural order, and so does the order matching_positions folds in (gotcha 18), so a SUM/AVG can change in the last bits
4	the writer string-sorts hidden:, expanded(…), collapsed: and ## Aggregation lines	those lines are written in natural order too
4	a GP fit string-sorts its records (improvise-gp dataset.rs)	the feature matrix's row order, and so a seeded fit's folds, follow natural order: a seeded fit can produce a different tree
4	app.rs view-shape fingerprints and Model's Debug string-sort ids	they sort naturally too; nothing visible changes
2, 5	a collapsed entry and a group reference name a group by name	by its id
4	casefold; a run of U+0020 spaces becomes one _; strip characters outside a–z, 0–9, -, _; remove a leading _	casefold; every character outside a–z, 0–9 and - becomes _; runs of _ collapse; leading and trailing _ and - are removed; n is prepended before a leading digit
5	a hand-written id uses the alphabet a–z, 0–9, -, _	a hand-written id obeys every syntax rule of a generated id
6	display names typed into minibuffers are turned into ids at the input edge	only CSV import and the .improv loaders read display names in files and command-language text, which names things by id; interactive input matches a display or an id
12	a typed name new to a formula edit is resolved against the document at commit	the client resolves it, ids first, and the wire carries only id-form text
12	the regression test commits the unedited text	the test edits a non-name token and commits

Context

Implementing the accepted ADR (epic improvise-cq1q) and using it raised points the accepted text does not settle: whether groups and measures carry ids, what a virtual category is in a file, exactly what slugify produces, which inputs may read display names, and which side resolves names in a formula edit. This amendment settles them.

Decision

1. What carries an identity

Groups have ids. A group has an id and a display, like a category or an item, and its id is an ordinary id: unique in the document, generated by slugify (group is an empty slug's <kind>), declared beside its display, referred to by id, created by display, and renamed by a display rename. Its wire address is GroupId. In code, PR #336 (improvise-cq1q.12) already adds a display to Group beside its key; that key becomes the group's id.

Virtual ids.

• A leading _ is reserved for virtual ids in general. The virtual views have ids _records and _drill, displayed _Records and _Drill: they are addressed on the wire (SwitchView), and a slugified _Drill would otherwise be drill and could collide with a user's view. Slugify never produces a leading _, so the grammar does not check the reservation.
• Virtual ids are lower case. The code, the wire, ViewState and the browser client spell the virtual categories _Index, _Dim and _Measure today, and move to _index, _dim and _measure in the re-key step.
• Virtual categories and measures are defined by criteria, not declared. Each contains whatever in the document meets its criterion: _measure holds the measures, and _index and _dim hold the record positions. Their ids and displays are hardcoded in the application, so a file never declares one, and a line that refers to one spells its id: _measure: column, region=east, _measure=revenue = 1200.
• The stored measures are still listed in a ## Category: _Measure section, each with its display and id (- Revenue(revenue), Cost(cost)). Its header names the hardcoded virtual category and is not a declaration of it. The section is complete: it lists every stored measure, so a file never relies on its data lines to discover one. A formula's measure is not listed there; it is declared by its ## Formulas line, which carries the id it shares with its formula. The loader recognises _Measure, _Index and _Dim by their hardcoded displays, and references are written _measure.
• Records mode's synthetic positions are values, not ids. The row coordinate under _index, and by the same reasoning the column coordinate under _dim, are values of those virtual categories. They take no part in id uniqueness and cannot collide with a user's item.

Formulas and measures. A measure is the normal formula target (profit = revenue - cost); a dimension target is rare, for a derived dimension.

• A formula is a kind of measure, so a _Measure formula's id is its measure's id. Profit = Revenue - Cost is one thing, profit: other formulas, each Profit cell's CellKey, and ReplaceFormula/RemoveFormula all name profit.
• Each measure's cells are keyed by one handle: ItemHandle::Item(id) for a stored measure, ItemHandle::FormulaTarget(id) for a formula's measure, never both.
• A formula never merges into a stored measure. With a stored profit (displayed Profit), adding Profit = Revenue - Cost creates a second measure with a fresh id, profit-<n>, by the ordinary collision rule. The stored measure keeps profit and its data; removing the formula removes only profit-<n>.
  • Both are named Profit in _measure's scope, so they are shown as Profit and Profit #2 (clause 2's suffix rule).
• A target outside _Measure is a real item with its own ItemId, and its formula has a separate FormulaId: a formula Tax in CustomCat gets item id tax, then formula id tax-<n>.
• Formulas may refer to anything by id. A function in a formula may accept any id-bearing thing (category, item, measure, formula, view, group) by its id. Whether a function takes the thing or its value (an item's display, say) is part of that function's signature.

2. Storage keys

• CellKey pairs are (CategoryId, ItemHandle), so a _Measure formula cell's coordinate is FormulaTarget without claiming an ItemId the document does not hold.
• A collapsed entry names its group by the group's id, and Group gains a display.
• Displays stay unique within a scope by suffixing. The accepted text refuses a display already in use in the scope; a formula's measure beside a stored measure of the same name (clause 1) cannot be refused. So when several things in one scope have the same display, they are shown with a suffix: the first keeps the bare display, and the rest get #2, #3, … (a space, then # and the number) as a gapless sequence in the order their ids sort.
  • The order is the ids' natural order (clause 4), so profit, profit-1, profit-2 and profit-10 show as Profit, Profit #2, Profit #3 and Profit #4.
  • Gapless: if the thing shown as Profit #2 is removed, Profit #3 becomes Profit #2.
  • The suffix is computed when a display is shown, from the things that share it, and is never stored or written to a file.
  • A suffixed display is a display like any other: interactive input that types Profit #2 picks that thing (clause 6), and the ids stay the unambiguous name.
  • Old files cannot hold two things with one name in a scope, so loading one adds no suffix.
  • A suffix never repeats a display already in the scope. If something is literally displayed Profit #2, the shared Profits take the next free numbers instead: Profit, Profit #3, Profit #4, still in id order. "Gapless" counts only the suffixes actually assigned, so a number taken by a literal display is skipped, like an id collision candidate that is already held.

4. How an id is made

Slugify's steps:

1. Casefold, with Unicode full case folding (Straße → strasse).
2. Replace every character outside a–z, 0–9 and - with _. Ids' alphanumerics are ASCII and include -; whitespace, punctuation and every non-ASCII letter become _.
3. Collapse each run of _ to one _, whether the underscores came from step 2 or were in the display.
4. Remove every leading and trailing _ and -, repeating until neither end holds either.
5. If the slug starts with a digit, prepend n, so no id is purely numeric or starts with a digit: 2024 → n2024.
6. If nothing is left, use <kind>-<n> from the shared counter. A display of only -, only _ or only non-ASCII letters ends here.

The syntax of an id: a lower-case ASCII letter, then a–z, 0–9, - and single underscores, never ending in _ or -. Slugify's output has this shape, so slugifying an id returns it unchanged, and a collision suffix -<n> keeps the shape.

display	id
Time Period, Time\tPeriod, Time \u{200B} Period	time_period
a_ b, a__b	a_b
Q1 - 2024	q1_-_2024
East , East_, -East-	east
Region(code)	region_code
Crème Brûlée	cr_me_br_l_e
2024, _2024	n2024
2024-Q1	n2024-q1
Ü, -, %	<kind>-<n>

A suffix and a hyphenated slug can look alike. East-1 slugs to east-1, which is also what the counter may make of east. They never collide, because every id a document holds is taken however it was made: a display slugging to a held east-1 becomes east-1-<n>, and a held counter candidate is skipped.

Ids are always sorted in natural order, wherever ids are ordered. Split each id into alternating runs of digits and non-digits; compare the runs pairwise, a non-digit run by byte order and a digit run by value: strip its leading zeros, then the shorter run is smaller, then compare bytes (so runs of any length work, with no fixed-width parse); when two digit runs have equal value, the one with fewer leading zeros sorts first; when one id runs out of runs first, it sorts first. So profit, profit-1, profit-2, profit-10, and a-7 before a-007.

Hand-written ids obey every syntax rule of a generated id. east-1, item-4 and q1_-_2024 load; -east, east-, 2024, a__b and _draft are refused at load.

Two branches of one file can merge into a file that will not load, if each adds a different thing that generates the same slug: the merged file declares that id twice, and the load is refused naming both. Stable identity across copies of a file is not promised (improvise-8tk5).

Built code. crates/improvise-core/src/model/id.rs implements this clause (context/code/core.md §20): slugify follows the six steps, is_id_syntax is the syntax, claim refuses what a generated id could not be, VIRTUAL_IDS holds the five virtual ids, IdKind::Group exists, and natural_cmp and display_suffixes implement natural order and clause 2's suffixes. No other code calls them yet.

5. Persistence

• The loader also refuses a hand-written id that breaks the syntax above, naming the line; the grammar's id_token stays permissive. A declared id never begins with _, since a file never declares a virtual category.
• Formula text quotes only keyword ids (|sum|). No id starts with a digit, and the formula grammar's bare identifier already admits -.
• Groups: a group reference ([coastal]) and the group in a collapsed: line are ids, and a group definition declares a display and an id (> Coastal(coastal)).
• A header such as ## Category: Budget (2024) in a new-version file gives the id 2024, which is refused. Import pipe-quotes Region(code) as ## Category: |Region(code)|(region_code).

An axis line is tried before format:, and a format that reads as an axis line is pipe-quoted (improvise-le4p). view_entry is an ordered choice and format_line takes the rest of the line, so a category with id format would lose its axis line to the number format; the new-version arm puts axis_line first. Then :set-format none would write format: none, read as an axis line, so the writer pipe-quotes a format that would also match axis_line (an axis keyword alone or followed by , <name>, or a value starting with |), and the new arm's format_line is pipe_quoted ~ &NEWLINE | rest_of_line. An old file's format: none is read as before and written quoted on first save. Required tests: a category displayed Format on a row axis round-trips; a view whose format is each axis keyword round-trips beside a category format on an axis.

A formula line. Today:

## Formulas
- Profit = Revenue - Cost
- Tax = Revenue * 0.1 [CustomCat]

Under the amendment:

## Formulas
- Profit(profit) = revenue - cost
- tax(tax-1) = revenue * 0.1 [customcat]

• A measure formula has no tag: _measure is implicit and never written as a tag. The line is the measure's only declaration: Profit is its display and (profit) the id it shares with its formula.
• A dimension formula has a tag, which is always the target category's id. The target is already declared with its display in its category, so the line names it by item id (tax), and (tax-1) is the formula's own id.
• So the loader tells them apart by the tag. The expression is over ids; a reference to a formula target, and a WHERE/IS value, is the id text of its ItemHandle. A target display that looks like Name(x) is pipe-quoted: - |Margin(%)|(margin) = ….

6. Display names and the wire

• Only CSV import and the .improv loaders turn display names in input text into ids, where input text means files and command-language text. They read text people wrote. An old file's names are resolved there as a migration.
• Command-language text names things by id. Anything that uses the command language as a scripting language is a script, wherever the text comes from (a file, improvise cmd, another program). A script naming something by display fails; it gets a display only through a dedicated function such as display_for(id).
• Interactive input (typing in the TUI or the browser, which is not input text) matches either a display or an id, as long as it is unambiguous which thing the user is picking. Ids win when it is not, and the display is shown as a preview (clause 12).
• Groups on the wire: AddItemInGroup's group is a group id, a new group is created by display, a group is renamed by id and new display, and FrameGroup carries the group's id and display.
• The formula panel treats a display equal to another thing's id as ambiguous, so it never shows a bare name that ids-first would bind elsewhere.
• ViewState::active_view and Workbook::active_view are keyed by id (improvise-p003).
• A _Measure formula-target row is removed by RemoveFormula and renamed by a display rename of the measure (improvise-i0n7).
• CheckedBatch's created-record exemption is removed (improvise-p003). It exists for frames like [AddItem(East), SetCell(East)], and a frame cannot address what it creates, so nothing produces one.

7. Rejections

Rejection::Gone(Named) gains Named::Group.

8. RecordHash

RecordHash hashes (CategoryId, ItemHandle) pairs spelled as id text. Ids are unique across kinds, so it stays canonical; a record excludes _measure, so a FormulaTarget is never part of one.

11. Migration

• Step 2 adds GroupId, a display on Group, and ItemHandle.
• Step 3 moves the virtual categories' keys to _index, _dim and _measure.
• A new step 7 builds clause 12's client-side capture and resolver, ids first from the start.

12. Who resolves names in a formula edit

The client resolves; the host never sees a display.

• The client holds the display→id mapping captured when the editor opened, as it holds OpenedCell. The TUI and the browser share one resolver in improvise-protocol.
• A name new to the edit is resolved by the client against its id-bearing projection: a typed name equal to an id names that thing, even if it is also another thing's display; otherwise a display exactly one thing has names it; a display matching several things is refused with their ids; a name matching nothing is refused. Ids are matched exactly. Ids win wherever there is ambiguity, now and once an autocomplete exists; the autocomplete prefers id matches and previews the display.
• The wire carries only id-form text. ReplaceFormula.raw and AddFormula.raw are in id form; an id that no longer exists is Rejection::Gone.

What ids-first costs. Item A is displayed East (id east) and item B is displayed east, so it got east-1. Typing east names A; B is reached as east-1. The panel therefore shows B as east-1, never as a bare east.

Tests.

• The accepted regression test's step 3 becomes: edit something that is not a name (Revenue - Cost * 1) and commit. Committing unedited text cannot tell a correct capture from none, because a client may pass unchanged text as a no-op (improvise-hexb).
• Ids-first: with A (East, east) and B (east, east-1), typing east in text not captured at open must bind A (and fail under displays-first), east-1 must bind B, and a display that is nobody's id must still resolve by display. A fixture where no display equals an id proves nothing.

Consequences

• Scripts that name things by display break and must be rewritten over ids.
• Two measures can share a display, and are shown as Profit and Profit #2.
• Old files' _Measure declarations and hyphen- or digit-led names change on first save.
• id.rs implements clause 4; the code that sorts, shows and creates ids does not call it yet.

Open questions

None.