Merge gh/main (5f8c3f8c) into adr007/display-2c
This commit is contained in:
Claude (adr007-2c)
@@ -25,7 +25,7 @@ them here would let the copy drift:
|
||||
| Source | What it holds |
|
||||
|---|---|
|
||||
| `context/design-principles.md` | The "why" — functional-first, polymorphism over conditionals, correctness by construction, layering, §7 testing doctrine |
|
||||
| `context/repo-map.md` | The "where", plus **numbered gotchas** — read the ones your diff touches |
|
||||
| `context/repo-map.md` | The "where", plus a **numbered gotcha digest** (the full text is `context/gotchas.md`) — read the long form of the ones your diff touches |
|
||||
| `prolog-sessions/kb.pl` | Executable knowledge: `invariant/2`, `doctrine/2`, `forbidden/2`, `decision/3`, and prior `work_done/3` |
|
||||
| `AGENTS.md` | Working rules: bug-first tests, `Option`/`Result`/`Box` at the edges, history reconciled only by merge, a new conventional commit by default (amend only an unpushed HEAD, for a small fix), and nothing that needs a force push |
|
||||
| `.agents/skills/wave/agent-rules.md`, "GitHub" | The reviewer's rule: what to read before your first comment, how to answer each existing review, and what every review you post must state. Read it before reviewing a PR |
|
||||
|
||||
@@ -45,6 +45,7 @@ nobody corrects. The brief adds only what varies per wave:
|
||||
1. **The expected base commit, by name.** Worktrees do not reliably branch from
|
||||
HEAD. One was created three merges behind and was only noticed because it
|
||||
was told to check. The agent checks against it: `agent-rules.md` rule 1.
|
||||
It then merges `gh/main` as it moves, not only at the start: rule 9.
|
||||
2. **A reference baseline**, so "N passing" is checkable and a drop is
|
||||
visible: Rust test count, harness counts, node counts, **and the machine**.
|
||||
The same commit once measured `improvise-io`'s binary at 1.4s on one machine
|
||||
|
||||
@@ -32,7 +32,7 @@ not in a second copy.
|
||||
6. **The Beads "Session Completion" block near the end of `AGENTS.md` does not
|
||||
set your git policy.** This repository does not opt in to the block's
|
||||
"team-maintainer" profile. On a wave branch that profile's pull and bare
|
||||
`git push` would be refused (rule 4), or worse. Rules 1–5, 7 and 8 are your
|
||||
`git push` would be refused (rule 4), or worse. Rules 1–5, 7, 8 and 9 are your
|
||||
git policy. The block is bd's and bd regenerates it whole, so it is not
|
||||
edited by hand: `AGENTS.md`, "Repository policy over the Beads blocks" (improvise-x1uk).
|
||||
7. **Pin `gh/main` to a literal SHA when the task starts.** Run
|
||||
@@ -101,6 +101,42 @@ not in a second copy.
|
||||
pinned SHA and `git cherry-pick -x` your own commits onto it (`-x` records
|
||||
where each came from). Close the old branch rather than force-pushing it:
|
||||
`AGENTS.md`, "Shipping a wave".
|
||||
9. **Merge main as soon as you can, and keep merging it.** The user:
|
||||
"Agents should merge main asap" and, when asked why, "Reduces conflicts"
|
||||
(2026-10-09 UTC; bd memory `agents-merge-main-asap`, written by the
|
||||
orchestrator; PR #335). Those two quotations are the user's words. The
|
||||
timing below is the orchestrator's operationalisation of them.
|
||||
- **When.** `git fetch gh`, then merge the new `gh/main` SHA (rule 7's
|
||||
re-pin, done on purpose) at the start of a task, before the first review
|
||||
request, before every push to a PR under review, and every time you
|
||||
resume. Main only gets further from your base, and a conflict met early
|
||||
is small. Never `origin/main`, never rebase, never squash (rule 8).
|
||||
Commit the merge through `scripts/agent-git`.
|
||||
- **After every merge of main, clean or not**, re-run the gates (see
|
||||
"Gates") on the merged tree, not on each side, and diff the test names
|
||||
against your pre-merge HEAD to see that none was lost. A clean merge is
|
||||
the dangerous case: it can delete a test module or break a shared test
|
||||
file with no conflict marker.
|
||||
- **Re-measure your baseline** after a mid-task merge, and report it
|
||||
against the merged SHA, so main's changes are not read as yours.
|
||||
- **Resolve a conflict by hand.** For each conflicted file, diff the
|
||||
result against both parents and the merge base, so a side's
|
||||
non-conflicting changes are not lost to a whole-file `--ours` or
|
||||
`--theirs`.
|
||||
- **A stacked PR merges its base branch, not `gh/main`.** GitHub diffs it
|
||||
against that base, so merging a `gh/main` ahead of the base puts main's
|
||||
commits in its diff (the #145/#148 symptom, which rule 3's numstat
|
||||
cannot see). The orchestrator retargets it to main once the base lands.
|
||||
- **The exception is narrow.** A durability push of WIP (see "Commit
|
||||
early") need not wait for a merge of main that conflicts. If you cannot
|
||||
resolve the conflict yet, push the WIP to a separate wip branch, never
|
||||
to the PR.
|
||||
- **Each merge-and-push moves the head SHA** an in-flight review is aimed
|
||||
at. Reviewers re-fetch the tip before posting, but do not merge main
|
||||
only to push a reply.
|
||||
- The mechanics are `AGENTS.md`, "Shipping a wave" (merge `gh/main`, never
|
||||
`origin/main`) and "Merging many PRs at once" (diff each parent; two
|
||||
clean changes to one test file can still fail together). Not restated here.
|
||||
|
||||
## Commit early: a worktree is not durable storage
|
||||
|
||||
@@ -305,7 +341,8 @@ out on purpose:
|
||||
## Files you must not write
|
||||
|
||||
`.beads/**` (never commit `issues.jsonl`, never `git checkout` it),
|
||||
`prolog-sessions/kb.pl`, `context/repo-map.md`: `AGENTS.md`, "Who owns what".
|
||||
`prolog-sessions/kb.pl`, `context/repo-map.md`, `context/gotchas.md`,
|
||||
`context/file-inventory.md`: `AGENTS.md`, "Who owns what".
|
||||
Also `AGENTS.md`, unless your brief grants you a named section. Report proposed
|
||||
text for any of these in your hand-back.
|
||||
|
||||
|
||||
@@ -151,7 +151,7 @@ that costs to get right. Everything in it was learned by getting it wrong first.
|
||||
|
||||
### Who owns what
|
||||
|
||||
Three files are **orchestrator-only**, and the rule exists because they are
|
||||
These files are **orchestrator-only**, and the rule exists because they are
|
||||
guaranteed conflicts — every agent has a reason to touch them and they merge
|
||||
badly:
|
||||
|
||||
@@ -160,8 +160,9 @@ badly:
|
||||
| `.beads/` (incl. `issues.jsonl`) | bd owns it. See "Never hand-edit" below. |
|
||||
| `prolog-sessions/kb.pl` | Written by `scripts/kb` only. See `context/code/kb-tool.md`. |
|
||||
| `context/repo-map.md` | Every agent wants to add a gotcha; every wave would conflict. |
|
||||
| `context/gotchas.md`, `context/file-inventory.md` | The long forms the map digests; same conflict, same owner. |
|
||||
|
||||
Wave agents **read** all three freely (`bd show`, `bd ready`, grep `kb.pl`) and
|
||||
Wave agents **read** all of them freely (`bd show`, `bd ready`, grep `kb.pl`) and
|
||||
**write** none of them. They report proposed repo-map lines and `kb.pl` facts as
|
||||
*text* in their hand-back, and the orchestrator applies them. Repo-map lines
|
||||
are batched at the end of the wave, after the review, because they conflict.
|
||||
|
||||
Generated
+165
-2
@@ -97,6 +97,12 @@ dependencies = [
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "arrayvec"
|
||||
version = "0.7.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56"
|
||||
|
||||
[[package]]
|
||||
name = "atomic"
|
||||
version = "0.6.1"
|
||||
@@ -178,6 +184,30 @@ dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "borsh"
|
||||
version = "1.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "553c5d846a6ba5150c65e3b1b8ec073bcf1abc20f9b7220de384a4443ea4e20a"
|
||||
dependencies = [
|
||||
"borsh-derive",
|
||||
"bytes",
|
||||
"cfg_aliases",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "borsh-derive"
|
||||
version = "1.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "12cdfe656708a01f89b451a7d36466e6fe6c414de0aa18fc54f864f6f9ca9f56"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"proc-macro-crate",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bumpalo"
|
||||
version = "3.20.3"
|
||||
@@ -705,6 +735,17 @@ dependencies = [
|
||||
"version_check",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"wasi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.3.4"
|
||||
@@ -1216,6 +1257,16 @@ dependencies = [
|
||||
"minimal-lexical",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-bigint"
|
||||
version = "0.4.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367"
|
||||
dependencies = [
|
||||
"num-integer",
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-conv"
|
||||
version = "0.2.2"
|
||||
@@ -1233,6 +1284,26 @@ dependencies = [
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-integer"
|
||||
version = "0.1.47"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b"
|
||||
dependencies = [
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-rational"
|
||||
version = "0.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824"
|
||||
dependencies = [
|
||||
"num-bigint",
|
||||
"num-integer",
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "num-traits"
|
||||
version = "0.2.19"
|
||||
@@ -1251,6 +1322,18 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "numtower"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"num-bigint",
|
||||
"num-integer",
|
||||
"num-rational",
|
||||
"num-traits",
|
||||
"proptest",
|
||||
"rust_decimal",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.4"
|
||||
@@ -1449,6 +1532,15 @@ dependencies = [
|
||||
"zerocopy",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro-crate"
|
||||
version = "3.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f"
|
||||
dependencies = [
|
||||
"toml_edit",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
@@ -1469,7 +1561,7 @@ dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"num-traits",
|
||||
"rand 0.9.5",
|
||||
"rand_chacha",
|
||||
"rand_chacha 0.9.0",
|
||||
"rand_xorshift",
|
||||
"regex-syntax",
|
||||
"rusty-fork",
|
||||
@@ -1510,6 +1602,8 @@ version = "0.8.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e058c7de0b26af77780c769414d6257830bb240f3c38477dbc2c16e5f54d6d4c"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"rand_chacha 0.3.1",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
@@ -1519,7 +1613,7 @@ version = "0.9.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"
|
||||
dependencies = [
|
||||
"rand_chacha",
|
||||
"rand_chacha 0.9.0",
|
||||
"rand_core 0.9.5",
|
||||
]
|
||||
|
||||
@@ -1534,6 +1628,16 @@ dependencies = [
|
||||
"rand_core 0.10.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_chacha"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
|
||||
dependencies = [
|
||||
"ppv-lite86",
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_chacha"
|
||||
version = "0.9.0"
|
||||
@@ -1549,6 +1653,9 @@ name = "rand_core"
|
||||
version = "0.6.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
|
||||
dependencies = [
|
||||
"getrandom 0.2.17",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rand_core"
|
||||
@@ -1713,6 +1820,23 @@ version = "0.8.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
|
||||
|
||||
[[package]]
|
||||
name = "rust_decimal"
|
||||
version = "1.43.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7653272e75dcac41dc199fbea6f5797633994fafd339943c06c9af16bf29cd3a"
|
||||
dependencies = [
|
||||
"arrayvec",
|
||||
"borsh",
|
||||
"bytes",
|
||||
"num-traits",
|
||||
"rand 0.8.8",
|
||||
"rand 0.9.5",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustc_version"
|
||||
version = "0.4.1"
|
||||
@@ -2161,6 +2285,36 @@ dependencies = [
|
||||
"tungstenite",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_datetime"
|
||||
version = "1.1.2+spec-1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_edit"
|
||||
version = "0.25.17+spec-1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3641d5bbb5349a79e1020a242d251efbc546ad8048d133958323ce9c40a9c9c"
|
||||
dependencies = [
|
||||
"indexmap",
|
||||
"toml_datetime",
|
||||
"toml_parser",
|
||||
"winnow",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "toml_parser"
|
||||
version = "1.1.5+spec-1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "baa693a8032d7e1cada7d0041e96126df243179ff061456783ac7f12bda4744c"
|
||||
dependencies = [
|
||||
"winnow",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tungstenite"
|
||||
version = "0.30.0"
|
||||
@@ -2525,6 +2679,15 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "winnow"
|
||||
version = "1.0.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wit-bindgen"
|
||||
version = "0.57.1"
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
[workspace]
|
||||
members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp"]
|
||||
members = [".", "crates/improvise-formula", "crates/improvise-core", "crates/improvise-protocol", "crates/improvise-io", "crates/improvise-command", "crates/improvise-wasm-client", "crates/improvise-ws-server", "crates/improvise-worker-server", "crates/gp-tree", "crates/improvise-gp", "crates/numtower"]
|
||||
|
||||
[package]
|
||||
name = "improvise"
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
# ADR 007, amendment 1 — Stable identities
|
||||
|
||||
**Ticket:** improvise-sboz (this amendment improvise-cq1q.1; implementation epic
|
||||
improvise-cq1q) · **Builds on:** ADR 001, ADR 002
|
||||
|
||||
## Status
|
||||
|
||||
**Accepted** (2026-10-09, PR #326). It amends ADR 007 (accepted 2026-09-25, PR #210,
|
||||
`007-stable-identities.md`). This file states only what the amendment changes or
|
||||
adds, numbered by the accepted ADR's clauses; everything else in ADR 007 is still
|
||||
true. Where the amendment replaces an ADR 007 rule or an existing behaviour:
|
||||
|
||||
| clause | accepted rule or existing behaviour | the amendment |
|
||||
|---|---|---|
|
||||
| 1 | groups have no id | a group has an id and a display, and its id behaves like every other id |
|
||||
| 1 | `_index`, `_dim`, `_measure` are the only ids beginning with `_` | a leading `_` is reserved for virtual ids in general, including the virtual views `_records` and `_drill` |
|
||||
| 1 | the virtual categories are declared and named in the file (`## Category: _Measure`, `_Measure: column`) | they are defined by criteria with hardcoded ids and displays and referred to by id (`_measure`); the `## Category: _Measure` header lists the stored measures but declares nothing |
|
||||
| 1 | `measure_item_names` merges a formula into a stored measure of the same name | a formula never merges: it gets its own id, `profit-<n>` beside a stored `profit` |
|
||||
| 2 | a display already in use in a scope is refused | equal displays in a scope are shown with a gapless ` #2`, ` #3`, … suffix in id order |
|
||||
| 2 | `CellKey` pairs are `(CategoryId, ItemId)` | `(CategoryId, ItemHandle)` |
|
||||
| 4 | ids sort by plain string order (`CellKey::new` sorts with `String::cmp`, and the writer orders data lines by their text) | ids sort in natural order. `CellKey`'s canonical coordinate order changes with it, so `RecordHash` values change (they are on the wire, not in files), and a saved file's data lines and their coordinates are written in the new order; the loader canonicalises keys, so the data read back is the same |
|
||||
| 4 | records mode string-sorts the store on entry (`DataStore::sort_by_key`, run by `SortData`) | records-mode rows come out in natural order, and so does the order `matching_positions` folds in (gotcha 18), so a `SUM`/`AVG` can change in the last bits |
|
||||
| 4 | the writer string-sorts `hidden:`, `expanded(…)`, `collapsed:` and `## Aggregation` lines | those lines are written in natural order too |
|
||||
| 4 | a GP fit string-sorts its records (`improvise-gp` `dataset.rs`) | the feature matrix's row order, and so a seeded fit's folds, follow natural order: a seeded fit can produce a different tree |
|
||||
| 4 | `app.rs` view-shape fingerprints and `Model`'s `Debug` string-sort ids | they sort naturally too; nothing visible changes |
|
||||
| 2, 5 | a collapsed entry and a group reference name a group by name | by its id |
|
||||
| 4 | casefold; a run of U+0020 spaces becomes one `_`; strip characters outside `a`–`z`, `0`–`9`, `-`, `_`; remove a leading `_` | casefold; every character outside `a`–`z`, `0`–`9` and `-` becomes `_`; runs of `_` collapse; leading and trailing `_` and `-` are removed; `n` is prepended before a leading digit |
|
||||
| 5 | a hand-written id uses the alphabet `a`–`z`, `0`–`9`, `-`, `_` | a hand-written id obeys every syntax rule of a generated id |
|
||||
| 6 | display names typed into minibuffers are turned into ids at the input edge | only CSV import and the `.improv` loaders read display names in files and command-language text, which names things by id; interactive input matches a display or an id |
|
||||
| 12 | a typed name new to a formula edit is resolved against the document at commit | the client resolves it, ids first, and the wire carries only id-form text |
|
||||
| 12 | the regression test commits the unedited text | the test edits a non-name token and commits |
|
||||
|
||||
## Context
|
||||
|
||||
Implementing the accepted ADR (epic improvise-cq1q) and using it raised points the
|
||||
accepted text does not settle: whether groups and measures carry ids, what a virtual
|
||||
category is in a file, exactly what slugify produces, which inputs may read display
|
||||
names, and which side resolves names in a formula edit. This amendment settles them.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. What carries an identity
|
||||
|
||||
**Groups have ids.** A group has an id and a display, like a category or an item, and
|
||||
its id is an ordinary id: unique in the document, generated by slugify (`group` is an
|
||||
empty slug's `<kind>`), declared beside its display, referred to by id, created by
|
||||
display, and renamed by a display rename. Its wire address is `GroupId`. In code,
|
||||
PR #336 (improvise-cq1q.12) already adds a display to `Group` beside its key; that key
|
||||
becomes the group's id.
|
||||
|
||||
**Virtual ids.**
|
||||
- **A leading `_` is reserved for virtual ids in general.** The virtual views have ids
|
||||
`_records` and `_drill`, displayed `_Records` and `_Drill`: they are addressed on the
|
||||
wire (`SwitchView`), and a slugified `_Drill` would otherwise be `drill` and could
|
||||
collide with a user's view. Slugify never produces a leading `_`, so the grammar does
|
||||
not check the reservation.
|
||||
- **Virtual ids are lower case.** The code, the wire, `ViewState` and the browser
|
||||
client spell the virtual categories `_Index`, `_Dim` and `_Measure` today, and move
|
||||
to `_index`, `_dim` and `_measure` in the re-key step.
|
||||
- **Virtual categories and measures are defined by criteria, not declared.** Each
|
||||
contains whatever in the document meets its criterion: `_measure` holds the measures,
|
||||
and `_index` and `_dim` hold the record positions. Their ids and displays are
|
||||
hardcoded in the application, so a file never declares one, and a line that refers to
|
||||
one spells its id: `_measure: column`, `region=east, _measure=revenue = 1200`.
|
||||
- **The stored measures are still listed in a `## Category: _Measure` section**, each
|
||||
with its display and id (`- Revenue(revenue), Cost(cost)`). Its header names the
|
||||
hardcoded virtual category and is not a declaration of it. The section is complete:
|
||||
it lists every stored measure, so a file never relies on its data lines to discover
|
||||
one. A formula's measure is not listed there; it is declared by its `## Formulas`
|
||||
line, which carries the id it shares with its formula. The loader recognises
|
||||
`_Measure`, `_Index` and `_Dim` by their hardcoded displays, and references are
|
||||
written `_measure`.
|
||||
- **Records mode's synthetic positions are values, not ids.** The row coordinate under
|
||||
`_index`, and by the same reasoning the column coordinate under `_dim`, are values of
|
||||
those virtual categories. They take no part in id uniqueness and cannot collide with
|
||||
a user's item.
|
||||
|
||||
**Formulas and measures.** A measure is the normal formula target (`profit = revenue -
|
||||
cost`); a dimension target is rare, for a derived dimension.
|
||||
- **A formula is a kind of measure, so a `_Measure` formula's id is its measure's id.**
|
||||
`Profit = Revenue - Cost` is one thing, `profit`: other formulas, each Profit cell's
|
||||
`CellKey`, and `ReplaceFormula`/`RemoveFormula` all name `profit`.
|
||||
- **Each measure's cells are keyed by one handle:** `ItemHandle::Item(id)` for a stored
|
||||
measure, `ItemHandle::FormulaTarget(id)` for a formula's measure, never both.
|
||||
- **A formula never merges into a stored measure.** With a stored `profit` (displayed
|
||||
`Profit`), adding `Profit = Revenue - Cost` creates a second measure with a fresh id,
|
||||
`profit-<n>`, by the ordinary collision rule. The stored measure keeps `profit` and
|
||||
its data; removing the formula removes only `profit-<n>`.
|
||||
- Both are named `Profit` in `_measure`'s scope, so they are shown as `Profit` and
|
||||
`Profit #2` (clause 2's suffix rule).
|
||||
- **A target outside `_Measure` is a real item** with its own `ItemId`, and its formula
|
||||
has a separate `FormulaId`: a formula `Tax` in `CustomCat` gets item id `tax`, then
|
||||
formula id `tax-<n>`.
|
||||
- **Formulas may refer to anything by id.** A function in a formula may accept any
|
||||
id-bearing thing (category, item, measure, formula, view, group) by its id. Whether a
|
||||
function takes the thing or its value (an item's display, say) is part of that
|
||||
function's signature.
|
||||
|
||||
### 2. Storage keys
|
||||
|
||||
- **`CellKey` pairs are `(CategoryId, ItemHandle)`**, so a `_Measure` formula cell's
|
||||
coordinate is `FormulaTarget` without claiming an `ItemId` the document does not
|
||||
hold.
|
||||
- **A collapsed entry names its group by the group's id**, and `Group` gains a
|
||||
`display`.
|
||||
- **Displays stay unique within a scope by suffixing.** The accepted text refuses a
|
||||
display already in use in the scope; a formula's measure beside a stored measure of
|
||||
the same name (clause 1) cannot be refused. So when several things in one scope have
|
||||
the same display, they are shown with a suffix: the first keeps the bare display, and
|
||||
the rest get ` #2`, ` #3`, … (a space, then `#` and the number) as a gapless
|
||||
sequence in the order their ids sort.
|
||||
- The order is the ids' natural order (clause 4), so `profit`, `profit-1`,
|
||||
`profit-2` and `profit-10` show as `Profit`, `Profit #2`, `Profit #3` and
|
||||
`Profit #4`.
|
||||
- Gapless: if the thing shown as `Profit #2` is removed, `Profit #3` becomes
|
||||
`Profit #2`.
|
||||
- The suffix is computed when a display is shown, from the things that share it, and
|
||||
is never stored or written to a file.
|
||||
- A suffixed display is a display like any other: interactive input that types
|
||||
`Profit #2` picks that thing (clause 6), and the ids stay the unambiguous name.
|
||||
- Old files cannot hold two things with one name in a scope, so loading one adds no
|
||||
suffix.
|
||||
- A suffix never repeats a display already in the scope. If something is literally
|
||||
displayed `Profit #2`, the shared `Profit`s take the next free numbers instead:
|
||||
`Profit`, `Profit #3`, `Profit #4`, still in id order. "Gapless" counts only the
|
||||
suffixes actually assigned, so a number taken by a literal display is skipped,
|
||||
like an id collision candidate that is already held.
|
||||
|
||||
### 4. How an id is made
|
||||
|
||||
**Slugify's steps:**
|
||||
1. **Casefold**, with Unicode full case folding (`Straße` → `strasse`).
|
||||
2. **Replace every character outside `a`–`z`, `0`–`9` and `-` with `_`.** Ids'
|
||||
alphanumerics are ASCII and include `-`; whitespace, punctuation and every non-ASCII
|
||||
letter become `_`.
|
||||
3. **Collapse each run of `_` to one `_`**, whether the underscores came from step 2 or
|
||||
were in the display.
|
||||
4. **Remove every leading and trailing `_` and `-`**, repeating until neither end holds
|
||||
either.
|
||||
5. **If the slug starts with a digit, prepend `n`**, so no id is purely numeric or
|
||||
starts with a digit: `2024` → `n2024`.
|
||||
6. **If nothing is left, use `<kind>-<n>`** from the shared counter. A display of only
|
||||
`-`, only `_` or only non-ASCII letters ends here.
|
||||
|
||||
**The syntax of an id:** a lower-case ASCII letter, then `a`–`z`, `0`–`9`, `-` and
|
||||
single underscores, never ending in `_` or `-`. Slugify's output has this shape, so
|
||||
slugifying an id returns it unchanged, and a collision suffix `-<n>` keeps the shape.
|
||||
|
||||
| display | id |
|
||||
|---|---|
|
||||
| `Time Period`, `Time\tPeriod`, `Time \u{200B} Period` | `time_period` |
|
||||
| `a_ b`, `a__b` | `a_b` |
|
||||
| `Q1 - 2024` | `q1_-_2024` |
|
||||
| `East `, `East_`, `-East-` | `east` |
|
||||
| `Region(code)` | `region_code` |
|
||||
| `Crème Brûlée` | `cr_me_br_l_e` |
|
||||
| `2024`, `_2024` | `n2024` |
|
||||
| `2024-Q1` | `n2024-q1` |
|
||||
| `Ü`, `-`, `%` | `<kind>-<n>` |
|
||||
|
||||
**A suffix and a hyphenated slug can look alike.** `East-1` slugs to `east-1`, which is
|
||||
also what the counter may make of `east`. They never collide, because every id a
|
||||
document holds is taken however it was made: a display slugging to a held `east-1`
|
||||
becomes `east-1-<n>`, and a held counter candidate is skipped.
|
||||
|
||||
**Ids are always sorted in natural order**, wherever ids are ordered. Split each id into
|
||||
alternating runs of digits and non-digits; compare the runs pairwise, a non-digit run by
|
||||
byte order and a digit run by value: strip its leading zeros, then the shorter run is
|
||||
smaller, then compare bytes (so runs of any length work, with no fixed-width parse); when
|
||||
two digit runs have equal value, the one with fewer leading zeros sorts first; when one
|
||||
id runs out of runs first, it sorts first. So `profit`, `profit-1`, `profit-2`, `profit-10`, and `a-7` before `a-007`.
|
||||
|
||||
**Hand-written ids obey every syntax rule of a generated id.** `east-1`, `item-4` and
|
||||
`q1_-_2024` load; `-east`, `east-`, `2024`, `a__b` and `_draft` are refused at load.
|
||||
|
||||
**Two branches of one file can merge into a file that will not load**, if each adds a
|
||||
different thing that generates the same slug: the merged file declares that id twice,
|
||||
and the load is refused naming both. Stable identity across copies of a file is not
|
||||
promised (improvise-8tk5).
|
||||
|
||||
**Built code.** `crates/improvise-core/src/model/id.rs` implements the accepted steps
|
||||
(`context/code/core.md` §20). A follow-up must replace rather than strip, collapse
|
||||
`_`, trim a trailing `_` and both ends' `-`, prepend `n`, make `claim` refuse what a
|
||||
generated id could not be, and add the virtual views and `IdKind::Group`.
|
||||
|
||||
### 5. Persistence
|
||||
|
||||
- **The loader also refuses a hand-written id that breaks the syntax above**, naming
|
||||
the line; the grammar's `id_token` stays permissive. A declared id never begins with
|
||||
`_`, since a file never declares a virtual category.
|
||||
- **Formula text quotes only keyword ids** (`|sum|`). No id starts with a digit, and
|
||||
the formula grammar's bare identifier already admits `-`.
|
||||
- **Groups:** a group reference (`[coastal]`) and the group in a `collapsed:` line are
|
||||
ids, and a group definition declares a display and an id (`> Coastal(coastal)`).
|
||||
- **A header such as `## Category: Budget (2024)`** in a new-version file gives the id
|
||||
`2024`, which is refused. Import pipe-quotes `Region(code)` as
|
||||
`## Category: |Region(code)|(region_code)`.
|
||||
|
||||
**An axis line is tried before `format:`, and a format that reads as an axis line is
|
||||
pipe-quoted** (improvise-le4p). `view_entry` is an ordered choice and `format_line`
|
||||
takes the rest of the line, so a category with id `format` would lose its axis line to
|
||||
the number format; the new-version arm puts `axis_line` first. Then `:set-format none`
|
||||
would write `format: none`, read as an axis line, so the writer pipe-quotes a format
|
||||
that would also match `axis_line` (an axis keyword alone or followed by `, <name>`, or a
|
||||
value starting with `|`), and the new arm's `format_line` is
|
||||
`pipe_quoted ~ &NEWLINE | rest_of_line`. An old file's `format: none` is read as before
|
||||
and written quoted on first save. Required tests: a category displayed `Format` on a
|
||||
row axis round-trips; a view whose format is each axis keyword round-trips beside a
|
||||
category `format` on an axis.
|
||||
|
||||
**A formula line.** Today:
|
||||
|
||||
```
|
||||
## Formulas
|
||||
- Profit = Revenue - Cost
|
||||
- Tax = Revenue * 0.1 [CustomCat]
|
||||
```
|
||||
|
||||
Under the amendment:
|
||||
|
||||
```
|
||||
## Formulas
|
||||
- Profit(profit) = revenue - cost
|
||||
- tax(tax-1) = revenue * 0.1 [customcat]
|
||||
```
|
||||
|
||||
- **A measure formula** has no tag: `_measure` is implicit and never written as a tag.
|
||||
The line is the measure's only declaration: `Profit` is its display and `(profit)`
|
||||
the id it shares with its formula.
|
||||
- **A dimension formula** has a tag, which is always the target category's id. The
|
||||
target is already declared with its display in its category, so the line names it by
|
||||
item id (`tax`), and `(tax-1)` is the formula's own id.
|
||||
- So the loader tells them apart by the tag. The expression is over ids; a reference to
|
||||
a formula target, and a `WHERE`/`IS` value, is the id text of its `ItemHandle`. A
|
||||
target display that looks like `Name(x)` is pipe-quoted: `- |Margin(%)|(margin) = …`.
|
||||
|
||||
### 6. Display names and the wire
|
||||
|
||||
- **Only CSV import and the `.improv` loaders turn display names in input text into
|
||||
ids**, where input text means files and command-language text. They read text people
|
||||
wrote. An old file's names are resolved there as a migration.
|
||||
- **Command-language text names things by id.** Anything that uses the command language
|
||||
as a scripting language is a script, wherever the text comes from (a file,
|
||||
`improvise cmd`, another program). A script naming something by display fails; it
|
||||
gets a display only through a dedicated function such as `display_for(id)`.
|
||||
- **Interactive input** (typing in the TUI or the browser, which is not input text)
|
||||
matches either a display or an id, as long as it
|
||||
is unambiguous which thing the user is picking. Ids win when it is not, and the
|
||||
display is shown as a preview (clause 12).
|
||||
- **Groups on the wire:** `AddItemInGroup`'s `group` is a group id, a new group is
|
||||
created by display, a group is renamed by id and new display, and `FrameGroup`
|
||||
carries the group's id and display.
|
||||
- **The formula panel** treats a display equal to another thing's id as ambiguous, so it
|
||||
never shows a bare name that ids-first would bind elsewhere.
|
||||
- **`ViewState::active_view` and `Workbook::active_view`** are keyed by id
|
||||
(improvise-p003).
|
||||
- **A `_Measure` formula-target row** is removed by `RemoveFormula` and renamed by a
|
||||
display rename of the measure (improvise-i0n7).
|
||||
- **CheckedBatch's created-record exemption is removed** (improvise-p003). It exists for
|
||||
frames like `[AddItem(East), SetCell(East)]`, and a frame cannot address what it
|
||||
creates, so nothing produces one.
|
||||
|
||||
### 7. Rejections
|
||||
|
||||
`Rejection::Gone(Named)` gains `Named::Group`.
|
||||
|
||||
### 8. `RecordHash`
|
||||
|
||||
`RecordHash` hashes `(CategoryId, ItemHandle)` pairs spelled as id text. Ids are unique
|
||||
across kinds, so it stays canonical; a record excludes `_measure`, so a `FormulaTarget`
|
||||
is never part of one.
|
||||
|
||||
### 11. Migration
|
||||
|
||||
- Step 2 adds `GroupId`, a display on `Group`, and `ItemHandle`.
|
||||
- Step 3 moves the virtual categories' keys to `_index`, `_dim` and `_measure`.
|
||||
- A new step 7 builds clause 12's client-side capture and resolver, ids first from the
|
||||
start.
|
||||
|
||||
### 12. Who resolves names in a formula edit
|
||||
|
||||
**The client resolves; the host never sees a display.**
|
||||
- The client holds the display→id mapping captured when the editor opened, as it holds
|
||||
`OpenedCell`. The TUI and the browser share one resolver in `improvise-protocol`.
|
||||
- **A name new to the edit** is resolved by the client against its id-bearing
|
||||
projection: a typed name equal to an id names that thing, even if it is also another
|
||||
thing's display; otherwise a display exactly one thing has names it; a display
|
||||
matching several things is refused with their ids; a name matching nothing is
|
||||
refused. Ids are matched exactly. Ids win wherever there is ambiguity, now and once an
|
||||
autocomplete exists; the autocomplete prefers id matches and previews the display.
|
||||
- **The wire carries only id-form text.** `ReplaceFormula.raw` and `AddFormula.raw` are
|
||||
in id form; an id that no longer exists is `Rejection::Gone`.
|
||||
|
||||
**What ids-first costs.** Item A is displayed `East` (id `east`) and item B is displayed
|
||||
`east`, so it got `east-1`. Typing `east` names A; B is reached as `east-1`. The panel
|
||||
therefore shows B as `east-1`, never as a bare `east`.
|
||||
|
||||
**Tests.**
|
||||
- The accepted regression test's step 3 becomes: edit something that is not a name
|
||||
(`Revenue - Cost * 1`) and commit. Committing unedited text cannot tell a correct
|
||||
capture from none, because a client may pass unchanged text as a no-op
|
||||
(improvise-hexb).
|
||||
- Ids-first: with A (`East`, `east`) and B (`east`, `east-1`), typing `east` in text not
|
||||
captured at open must bind A (and fail under displays-first), `east-1` must bind B,
|
||||
and a display that is nobody's id must still resolve by display. A fixture where no
|
||||
display equals an id proves nothing.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Scripts that name things by display break and must be rewritten over ids.
|
||||
- Two measures can share a display, and are shown as `Profit` and `Profit #2`.
|
||||
- Old files' `_Measure` declarations and hyphen- or digit-led names change on first save.
|
||||
- `id.rs` and its tests need the follow-up in clause 4.
|
||||
|
||||
## Open questions
|
||||
|
||||
None.
|
||||
@@ -1,5 +1,8 @@
|
||||
# ADR 007 — Stable identities for categories, items, formulas and views
|
||||
|
||||
**Amended:** by `007-stable-identities-amendment-1.md` (accepted 2026-10-09, PR #326),
|
||||
which states every rule it replaces.
|
||||
|
||||
**Status:** accepted (2026-09-25). The user approved revision 7. Revision 8 then
|
||||
followed the adversarial review of that head: it adds clause 12, the ADR's own reading
|
||||
of when a formula edit's names become ids, and corrects stale facts about main. The
|
||||
|
||||
@@ -0,0 +1,447 @@
|
||||
# ADR 009 — Incremental evaluation: an in-house DBSP core, plugged in behind a trait
|
||||
|
||||
**Status:** proposed. ·
|
||||
**Ticket:** improvise-felo.4 (the record of this ADR; implementation goes to its own
|
||||
epic) ·
|
||||
**Relates to:** ADR 001 (the basis a write claims), ADR 002 (surrogates on the wire),
|
||||
ADR 003, ADR 005, ADR 006, ADR 007, ADR 008
|
||||
|
||||
## Problem
|
||||
|
||||
Every change recomputes every formula cell (`Model::recompute_formulas`, called by
|
||||
`rebuild_layout` after each batch), and raw aggregates are re-folded on every read. That
|
||||
does not scale to a live source (ADR 003) streaming changes into a large table, nor to
|
||||
ADR 008's virtualized grid, which needs occupancy, totals and widths maintained without
|
||||
building the grid. A change should cost in proportion to what it touches.
|
||||
|
||||
## Decision
|
||||
|
||||
Formulas are evaluated incrementally by an in-house implementation of the DBSP core. A
|
||||
cell write becomes a delta, aggregates are maintained, and only the formula cells whose
|
||||
inputs moved are re-evaluated by the existing evaluator. The incremental results must
|
||||
equal today's eager results exactly.
|
||||
|
||||
**Principle: everything is as incremental as possible**, so the work done per model
|
||||
mutation is minimised. Where this ADR does work that grows with more than the
|
||||
mutation, it says so and names the incremental path that normally applies.
|
||||
|
||||
### 1. The `zset` crate
|
||||
|
||||
A problem-agnostic workspace crate with no dependencies beyond `std` (so it is
|
||||
wasm32-clean and serde-free). It provides:
|
||||
- `ZSet<K>`, a map from `K: Ord` to a non-zero `i64` weight, which is an abelian group;
|
||||
- a synchronous, single-threaded `Circuit` per process, with `delay`, `integrate`,
|
||||
`differentiate`, `map`, `filter`, `flat_map`, `index_by`, the bilinear delta join
|
||||
`join_indexed`, `distinct` and `aggregate`;
|
||||
- resumable rebuilds: building an operator from a snapshot is a `step(deadline)`, so it
|
||||
can run as a sliced job;
|
||||
- operator state behind one trace interface (look up, read a range, apply a delta), so
|
||||
later store-memory work can page or compress it.
|
||||
|
||||
**Operators are self-contained.** An operator's state and behaviour depend only on its
|
||||
inputs and its own state, never on a shared in-process global or on another operator's
|
||||
internals. That keeps the graph distributable: in principle the model could run across a
|
||||
cluster of machines. Nothing more is designed for that here.
|
||||
|
||||
There are no nested streams, no recursion operator and no timestamps beyond one clock.
|
||||
**The formula language is the query language.** What a request asks of the model beyond
|
||||
its axes and window (filters, aggregation, derived values) is written in formula text
|
||||
over ids (ADR 007). It goes through the same evaluator and circuit as a document
|
||||
formula, so there is no second query language.
|
||||
|
||||
### 2. The plug
|
||||
|
||||
improvise-core gains a trait, `Derived`, for the two things the evaluator reads: the
|
||||
formula result at a key, and the raw fold of the cells that match a partial key. The
|
||||
incremental backend implements it in a seam crate, `improvise-incremental`, which
|
||||
depends on `zset` and improvise-core. improvise-core never depends on `zset`.
|
||||
|
||||
A live source (ADR 003) that reports its changes feeds cell deltas like `DataStore`. A
|
||||
source that answers aggregate queries but reports no changes implements the derived-value
|
||||
operator directly, by query pushdown, with no change feed.
|
||||
|
||||
The circuit decides *which* formula cells to re-evaluate. The one existing evaluator
|
||||
computes each value, so formulas are never compiled into operators.
|
||||
|
||||
**`Eager` is temporary.** Today's code moves behind the trait as `Eager`, unchanged. It
|
||||
exists only to show discrepancies between the new evaluator and the old one: a
|
||||
differential property test applies one random edit sequence to both backends and
|
||||
compares every drawn cell, total and occupancy count. Eventually it is deleted, and
|
||||
nothing depends on it surviving. Long-term tests compare against exact references, such
|
||||
as `ExactSum` against rational sums, not against `Eager`.
|
||||
|
||||
### 3. The cell delta
|
||||
|
||||
`DataStore::set` and `DataStore::remove` emit a `ZSet<(CellKey, CellValue)>`:
|
||||
- `+(k, v)` for a new cell;
|
||||
- `−(k, v₀) + (k, v₁)` for an overwrite;
|
||||
- `−(k, v₀)` for a removal.
|
||||
|
||||
Values compare by normal float rules, except that every NaN is one canonical value:
|
||||
`-0.0` equals `+0.0`, and an equal-value overwrite consolidates to nothing. NaN payloads
|
||||
are not preserved, so nothing can be smuggled through them. Today's record tier compares
|
||||
bits, so overwriting `0.0` with `-0.0` stops counting as a change. Multisets order values
|
||||
by `f64::total_cmp` after mapping every NaN to the canonical one and `-0.0` to `+0.0`. `CellKey` gains `Ord`, which is its canonical order: the
|
||||
natural order of ids (ADR 007 amendment 1, clause 4), the order eager folds in. The
|
||||
circuit steps once per model effect.
|
||||
|
||||
The record tier and the dirty flag read this delta instead of scanning the store. The
|
||||
dirty flag fires on a non-empty `(key, value)` delta, or when a fingerprint of the
|
||||
formula set moves, because formula edits move no cell.
|
||||
|
||||
Under ADR 007 and its amendment, `CellKey` pairs are `(CategoryId, ItemHandle)`. So a
|
||||
rename is a display write and emits no delta. The formula-set fingerprint must use the id
|
||||
form too, or a rename reads as a formula change.
|
||||
|
||||
### 4. Aggregates
|
||||
|
||||
A group is the cell key minus the None categories. One aggregation circuit serves every
|
||||
shape key with the same None list, which removes the per-session model clone.
|
||||
|
||||
| aggregate | state per group | per delta element |
|
||||
|---|---|---|
|
||||
| `Count` | count of values, count of numbers | O(1) |
|
||||
| `Sum`, `Avg` | an exact accumulator (clause 8), and the count | O(1) |
|
||||
| `Min`, `Max`, text unanimity | a multiset of values | O(log d) |
|
||||
|
||||
A group is read with eager's rules, including NaN and signed zero. `LAST`/`FIRST` and
|
||||
semi-additive measures (improvise-felo.7, felo.8) reuse the multiset shape, keyed by the
|
||||
ordering category's order. Those tickets choose the key. It must keep a reorder or an
|
||||
insert incremental: only the entries under the moved items change, and nothing is
|
||||
rebuilt.
|
||||
|
||||
### 5. Formulas
|
||||
|
||||
**Stems** are `distinct` over the delta, keyed by "cell key minus the target and None
|
||||
categories". A formula counts at a stem where its top-level `WHERE` matches or raw data
|
||||
exists at the target.
|
||||
|
||||
**Inputs.** A formula cell is re-evaluated when one of its inputs moved:
|
||||
- a `Ref(R)`: the group at the stem with `R`'s category set to `R`;
|
||||
- an `Agg`: the group at the stem;
|
||||
- an `Agg` whose inner `WHERE` names `Cat = Item`: the group at the stem with `Cat` set
|
||||
to `Item`. This is the one join: stems indexed by "stem minus `Cat`".
|
||||
|
||||
The affected pairs are evaluated in the formula graph's topological order.
|
||||
|
||||
**An aggregate over an expression** (`SUM(Revenue * Price)`) is a pipeline:
|
||||
1. A per-record stage retracts and reinserts each touched record. It reads the record's
|
||||
previous cells as the store minus the delta, so it keeps no copy of the values. The
|
||||
store is the `integrate` operator's trace over the cell delta, read as an input of
|
||||
the stage, so the stage stays self-contained.
|
||||
2. A map produces `(context, order key, expr)`. The order key is (context rank, record
|
||||
key in `CellKey` order), which is eager's concatenation order.
|
||||
3. The group aggregate, with an error channel that reads the first `Err` in order.
|
||||
|
||||
No cross product is built. A formula whose expression reads other records is evaluated
|
||||
over the **occupied contexts** only, and `Eager` uses the same enumeration. The answer is
|
||||
unchanged, because a context with no records contributes nothing. One exception: an
|
||||
inner bare ref that names no item errors at every context, so for it the ref is checked
|
||||
before enumerating. This replaces `Expansion::members`.
|
||||
|
||||
### 6. Circular references
|
||||
|
||||
DBSP's least fixed point is not improvise's circularity: `X = X + 1` has none, and the
|
||||
answer must be `Error("circular")`. The formula graph's strongly connected components
|
||||
are maintained incrementally as formulas change. Acyclic formulas are evaluated once
|
||||
each, in topological order. A cyclic component is maintained incrementally too: a change
|
||||
re-iterates only the cells it reaches. A component that does not converge within the
|
||||
existing bound of sixteen passes (`MAX_EVAL_DEPTH`) propagates `Error("circular")`, as
|
||||
today. An optimisation that speeds convergence is allowed only where it is exact: it
|
||||
gives the same result as the plain iteration, and a test compares the two.
|
||||
|
||||
### 7. Operator state follows the request's shape key
|
||||
|
||||
A projection request's **shape key** selects operator state, shared by every request
|
||||
with the same key, from any client. The key is:
|
||||
- the categories on each axis, by level;
|
||||
- the None categories;
|
||||
- the filter keys: each filter category's item, and the hidden items of categories on
|
||||
None, Page or Filter;
|
||||
- the page categories, so each page is a maintained slice and paging rebuilds nothing;
|
||||
- `measure_agg`.
|
||||
|
||||
The key uses the canonical parsed id form of the request's query text, so two spellings
|
||||
of one query share their operator state. Everything else in a request is applied when output is read: hidden and collapsed items
|
||||
on axis categories, item order, the page selection and prune. Occupancy and column
|
||||
aggregates are keyed by axis **items**, so a collapse, hide, show, reorder or added item
|
||||
changes no operator. The visible axis is a remap.
|
||||
|
||||
- **A new key** builds its operators as a sliced job under ADR 008's job runner. Deltas
|
||||
arriving during the build are buffered and added at install. The finished result is the
|
||||
response to the request that asked for it, not a notice.
|
||||
- **A formula edit** is a formula delta fed to the circuit.
|
||||
- **A document swap** rebuilds everything, and a new key's first build is a full pass.
|
||||
These are the only rebuilds; every other change is a delta.
|
||||
- **Saving a view definition** touches no operator.
|
||||
- **Live keys are a cache.** A key is evicted after a period in which no request names
|
||||
it. The period is a local duration on the host, set by measurement, and never compared
|
||||
with another machine's clock. Every eviction is announced (clause 9).
|
||||
|
||||
**Column totals and hidden rows.** Totals are kept over all rows, because the shape key
|
||||
is shared and a hide is one client's. Whether hidden items travel in a request is ADR
|
||||
008's open question. If they do not, a total is read as it is kept. If they do, a total
|
||||
must leave the hidden rows out, so a read subtracts the request's H hidden cells as a
|
||||
temporary overlay:
|
||||
- `Count` subtracts;
|
||||
- `Min` and `Max` skip keys whose whole count is hidden;
|
||||
- text counts the distinct keys left;
|
||||
- `Sum` subtracts exactly through the accumulator.
|
||||
|
||||
That is O(H log d) per read for every aggregate, with no view-store state in the model
|
||||
store.
|
||||
Aggregate-over-expression totals index their entries by row tuple for the same
|
||||
subtraction.
|
||||
|
||||
### 8. Exact sums
|
||||
|
||||
An f64 sum maintained by `+v` and `−v` drifts. Measured by
|
||||
`crates/improvise-core/examples/retraction_drift.rs`, at 18,000 cells after a million
|
||||
cycles, it is 1.7e-9 from eager, past `FLOAT_EQ_EPSILON`, so `=` could answer
|
||||
differently on the two paths. So every `Sum` and `Avg` uses **`ExactSum`**, in both
|
||||
backends:
|
||||
- a fixed-point integer covering every finite f64, with counts of NaN and ±∞, which
|
||||
makes it an abelian group that rounds once, on read;
|
||||
- measured bit-equal to a fresh sum after a million cycles, and exact against rational
|
||||
sums on 20,000 cases;
|
||||
- 568 bytes per group, and about 1.3× the cost of an f64 add per update.
|
||||
|
||||
The sign of a zero sum follows IEEE rules for the same addends: it is `-0.0` only if
|
||||
every addend is `-0.0`, and an exact cancellation gives `+0.0`. It lives in ADR 005's
|
||||
`numtower` crate. **User-visible change:** displayed sums change in their last bits,
|
||||
because they become correctly rounded.
|
||||
|
||||
### 9. Delivery
|
||||
|
||||
The **model store** (the document model and the DBSP graph) and the **view store** (the
|
||||
cursor, the window and unsaved view definitions) are conceptual boxes, not a network
|
||||
boundary. Where a piece of state physically lives (a client, the host or a worker) is
|
||||
an implementation detail. The model store's decisions depend only on the model. A
|
||||
server-side twin of a client's state, such as its filter, watermark or window, may be
|
||||
held at the serving end of an edge where that makes the graph genuinely simpler. It is
|
||||
then part of the edge, not of the model store, and it is disposable: losing it never
|
||||
gives wrong data. A client that keeps its own watermark recovers with a correction from
|
||||
it. A client that relied on the twin for its watermark recovers with a full refetch.
|
||||
|
||||
**Every edge may cross a network.** An edge, meaning the stream from one operator to the
|
||||
next, may be in-process or remote. No edge assumes a function call or shared
|
||||
memory. An operator that sits at a wire boundary is just one implementation of an edge,
|
||||
an **edge adapter**. So every edge has the same semantics:
|
||||
- **A wire form.** Every element type that flows on an edge has a serializable form,
|
||||
defined in `improvise-protocol`. The adapter converts to and from it, and `zset` stays
|
||||
free of serde. `CellKey` is not `Serialize`, so on the wire a cell is named by its
|
||||
tuple of ids, its position or an ADR 002 surrogate.
|
||||
- **The wire form is lossless.** Nothing is lost in transport, numeric precision above
|
||||
all:
|
||||
- a value never crosses as a JSON or JS double. It crosses in ADR 005's canonical text:
|
||||
an inexact finite value as `#i` with its shortest round-trip spelling, which keeps
|
||||
the sign of zero, infinities as `inf` and `-inf`, and every NaN as the one token
|
||||
`nan`;
|
||||
- positions and counts below 2⁵³ are exact as JSON numbers. Wider integers cross as
|
||||
strings;
|
||||
- text crosses byte-exact, with no normalisation;
|
||||
- the delta is the unit that crosses, never `ExactSum`'s limbs. A receiver that needs
|
||||
a sum gets the host's exact read or rebuilds it from the exact deltas.
|
||||
|
||||
The stamp rule detects lost messages, but nothing would detect a lossy encoding. So
|
||||
every element type has a round-trip property test, where encoding then decoding is
|
||||
the identity for every value, including `-0.0`, infinities, 28-digit decimals and
|
||||
`i128`-wide fractions. A NaN round-trips as the canonical NaN.
|
||||
- **The clock is causal, never a wall clock.** Ordering, staleness and loss detection
|
||||
use only the stamp sequence (below), a logical clock the host advances once per
|
||||
committed command. Each process runs its own synchronous sub-circuit, and across an
|
||||
edge the stamped commands are the clock. No decision compares wall-clock times across
|
||||
machines, so clock skew cannot affect it. A delta crosses a wire only after its command
|
||||
commits, so a remote operator never sees a delta that is later rolled back.
|
||||
- **Ordering and backpressure.** An edge delivers in order. A slow edge coalesces
|
||||
exactly: deltas are summed, and notices merged. A merged notice carries the range of
|
||||
publications it covers, `(from, to]`, so the client that merges notices sees no gap
|
||||
where nothing was lost.
|
||||
- **An edge is a session, not a connection.** An edge may be one long-lived connection,
|
||||
or a sequence of requests carrying the same session token or cookie, for example
|
||||
several HTTP requests. The host decides nothing from the token: it is transport, and
|
||||
any later authentication is out of scope. What continues across connections is the
|
||||
client's own watermark and shape keys, which it sends with each request. Every request
|
||||
can be retried, and a resync from a watermark is idempotent. A client asks for
|
||||
"notices after my watermark", and the host answers from **one shared, bounded log
|
||||
of published notices**: a single log, not one per edge or client, holding the notices
|
||||
themselves (keys and flags, not values). A request older than the log's start is
|
||||
answered with a correction (below).
|
||||
- **Failure.** Timeouts are local monotonic durations, never compared across machines.
|
||||
On a long-lived connection, both ends keep alive, and a host-side timeout is tied to
|
||||
that open connection. On a sequence of requests, only the
|
||||
client detects failure: a request that fails or times out. In either case the client
|
||||
then asks for a correction from its watermark.
|
||||
|
||||
- **Correction is Z-set arithmetic.** A correction is `target − held`, where `target`
|
||||
is the current state for the client's window and `held` is what the client holds.
|
||||
Applying it with ordinary Z-set addition gives `held + (target − held) = target`, so
|
||||
the group structure of `ZSet` provides it, with no separate machinery. The client's
|
||||
position is a **watermark**: the whole stamp `(host instance, document_generation,
|
||||
publication)` up to which it has applied the stream, so everything at or before it is
|
||||
applied. A watermark from another host instance or generation is behind every current
|
||||
stamp. The client holds one per shape key it reads. A watermark is view-store state:
|
||||
the model store keeps none, so a client sends its own with each request unless a twin
|
||||
at the serving end holds it.
|
||||
- where the shared log covers the watermark and the covered range holds only moved
|
||||
keys, the host sends the current values of those keys, restricted to the client's
|
||||
window, and the client applies `target − held` over them. The log holds keys and
|
||||
flags, not values, so the model store keeps no view-store state;
|
||||
- where the covered range also holds axis changes, relabels or flags, which positions
|
||||
and labels depend on, the correction carries them too, in order, as a merged notice
|
||||
does, so the client re-derives positions incrementally;
|
||||
- only where the log no longer covers the watermark, or the covered range holds an
|
||||
oversized notice that lists no keys, does the host fall back to `target`
|
||||
for the whole window, and the client applies `target − held`;
|
||||
- a full refetch is `target − ∅`.
|
||||
|
||||
**Every read sees only the last committed state**, whether it is a correction, a
|
||||
fetch or `project`, and it is labelled with that state's stamp. A value a failed
|
||||
command wrote and then rolled back is never read. A correction is labelled with the
|
||||
publication range it covers, so it is applied only at the watermark it starts from,
|
||||
which makes it idempotent and resumable. Its elements are `(key, value)` pairs with
|
||||
weights, in the lossless wire form, so a client and a worker apply it with the same
|
||||
code.
|
||||
|
||||
Aggregation stays on the model side (ADR 008), so the operators that aggregate run on
|
||||
the model side. Which other edges cross a wire is a deployment choice. The engine's invalidation signal, which ADR 006's replay trigger
|
||||
relies on, is a correction trigger below, followed from a displayed cell's key back to
|
||||
the model.
|
||||
|
||||
**A request** is a shape key plus a window, given as an absolute cross-product top-left
|
||||
and dimensions. The window is view-store state, and a request costs O(visible).
|
||||
- **Positions** are bigints where they exceed 2⁵³, sent as JSON strings.
|
||||
- **Output** is keyed by tuple identity, with the position attached when sent.
|
||||
- **An item insert, removal or reorder** is sent as an axis change, and each client
|
||||
re-derives its positions.
|
||||
|
||||
**Pruning support.** Pruning is window-relative: a row or column empty within the
|
||||
displayed values is hidden. This replaces today's whole-grid `prune_empty`. The model
|
||||
answers one restricted successor query: "the next k rows after position p with a value
|
||||
in a column of S", and the same for columns. A per-column sorted index of occupied rows
|
||||
answers it in O(|S| log n + k|S|), holding O(occupied cells). Whether pruning runs on
|
||||
the client and which parameters travel are ADR 008's questions, and this design supports
|
||||
either answer. If the client prunes, it prunes its window, fills forward with that
|
||||
query, and, when scrolling up or left, finds a new top-left by galloping backwards in
|
||||
doubling jumps (about 14 to 28 probes at density 10⁻⁴).
|
||||
|
||||
**The change notice.** Each command commits on its own (ADR 006), and after each
|
||||
committed command the host emits one notice, addressed to no client. A frame
|
||||
whose later command fails keeps its earlier commands, and their notices, published. A
|
||||
notice carries:
|
||||
- the stamp `(host instance, document_generation, publication)`. The publication number
|
||||
increments once per committed command. The host instance changes on restart; it is a
|
||||
random id, never a wall-clock time;
|
||||
- axis changes, display relabels as `(id, display)`, and a flag when the formula set,
|
||||
`measure_agg`, the view definitions or the default view moved. A shared display's
|
||||
` #2` suffix is never sent: the client computes it from the displays in scope;
|
||||
- the shape keys evicted since the last notice;
|
||||
- for each live shape key, the moved output keys (cells, formula cells and totals), as
|
||||
the circuit computes them. Moved records alone would miss formulas with cross-stem
|
||||
inputs.
|
||||
|
||||
**A client receives only what it needs.** Filtering is part of the DBSP graph, not a
|
||||
separate mechanism. A client either pulls just the state it needs, or pushes a filter to
|
||||
the host's side of its connection: a filter operator on its shape's query, written in the
|
||||
query language, through which its notices pass. Whether the filter also carries the
|
||||
client's window is an optimisation, since the model store's decisions depend only on the
|
||||
model. A pushed filter is held by the open connection, or by a twin. Otherwise the
|
||||
client sends it with each request. While a connection's filter is open, it counts as a request
|
||||
naming its shape, so its shape is not evicted.
|
||||
|
||||
A notice whose key lists exceed a fixed size carries "correct" instead. That is a
|
||||
fallback, and it costs every reader a correction.
|
||||
|
||||
**What a client does.** It intersects the moved keys with its window. A fold row has no
|
||||
maintained key, so it maps member keys through its own remap first. It then fetches what
|
||||
moved. It asks for a correction when it sees a publication gap, a new
|
||||
`document_generation` or host instance, its shape in the eviction list, a notice it
|
||||
cannot read, or a dropped connection or failed request. A slow client may merge pending notices: the union of
|
||||
keys, the covered range of publications, and axis changes kept in order. Because the
|
||||
wire is lossless, outputs keep the circuit's value identity end to end.
|
||||
|
||||
**What this replaces.** Today's per-session mirrors and pushed diffs (`Sessions`,
|
||||
`ClientSession`, `grid_updates`, `Projection`, `Dirty`). The model store keeps no
|
||||
view-store state, only the shared notice log. `GridExtent` is derived by the client, and in records
|
||||
mode the record count is fetched. `Projection` also published the ADR 001 basis a write
|
||||
claims. Whether `project` returns it instead, and what replaces the layout tier, are
|
||||
ADR 008's questions.
|
||||
|
||||
**Edge adapters.** An edge that crosses a wire has one adapter at each end. Toward the
|
||||
circuit, the client's adapter is a sink for results and notices and a source of requests.
|
||||
Toward the client it is the mirror image. The client's end keeps the watermark. In
|
||||
process, an edge is a function call.
|
||||
|
||||
### 10. Column width hints
|
||||
|
||||
Every column incrementally maintains a width hint, which optimises grid sizing and
|
||||
reduces jitter. The hint is an integrated aggregate: a histogram from display width to
|
||||
count, so a retraction is O(log d). Where in the graph the hint is computed is a
|
||||
scheduling choice, made as an optimisation later.
|
||||
|
||||
### 11. Rollback and memory
|
||||
|
||||
- **Rollback.** The circuit's state joins ADR 006's per-command snapshot, which is taken
|
||||
before a command's effects and restored if the command fails. An inverse delta is not
|
||||
a rewind, because `delay` and `differentiate` keep history. The requirement is that
|
||||
this snapshot does not cost O(cells) per command against ADR 008's 8 ms budget, and
|
||||
that writes do not block reads. How is for the implementation epic.
|
||||
- **Memory.** Resident state beside the store is the multisets, `ExactSum` per summed
|
||||
group, occupancy per live shape, the successor-query index, the width histograms and
|
||||
the bounded notice log. The store itself is the integrated base and is never copied. It
|
||||
is measured at ADR 008's fixture before the store-memory work begins.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
- **Feldera's `dbsp` crate:** 67 required dependencies, including tokio and zstd, and it
|
||||
fails to build for wasm32.
|
||||
- **Compiling formulas into operators:** a second evaluator.
|
||||
- **An ad-hoc dirty-tracking cache:** one invalidation rule per derived fact, each a
|
||||
second description of evaluation.
|
||||
- **A topic or subscription mechanism beside the graph:** filtering is an operator in
|
||||
the graph (clause 9).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:**
|
||||
- A write's evaluation costs O(delta) plus the formula cells it reaches.
|
||||
- The store scans and the per-view model clones go.
|
||||
- ADR 008 gets occupancy, totals and widths from one evaluator.
|
||||
- Live sources that report changes plug in.
|
||||
- The model store holds no view-store state.
|
||||
|
||||
**Bad:**
|
||||
- Two backends exist until `Eager` goes.
|
||||
- Integrated state is O(cells) beside the store.
|
||||
- `ExactSum` costs 568 bytes per summed group.
|
||||
- Each command snapshots the circuit's state for rollback.
|
||||
|
||||
**Order** (each step is a ticket in the implementation epic, behind the differential
|
||||
test):
|
||||
1. `zset`;
|
||||
2. the `Derived` trait with `Eager`;
|
||||
3. the cell delta, the record tier and the dirty flag;
|
||||
4. order-free aggregates and occupancy, measured at ADR 008's fixture;
|
||||
5. formulas and the aggregate-over-expression pipeline;
|
||||
6. `Sum`, `Avg` and totals through `ExactSum`;
|
||||
7. live sources;
|
||||
8. delivery.
|
||||
|
||||
Step 8's required tests:
|
||||
- a frame `[AddItem, SetCell]` publishes one notice per committed command, numbered
|
||||
consecutively, and no client asks for a correction;
|
||||
- a dropped notice is detected;
|
||||
- a reconnect after a host restart asks for a correction;
|
||||
- a corrected state equals the target, `held + (target − held) = target`, and
|
||||
correcting a state that is already correct is the zero Z-set (property tests over
|
||||
random edit sequences and losses);
|
||||
- a refused command consumes no publication number, and the commands before it in its
|
||||
frame stay published;
|
||||
- a dropped connection or a failed request triggers a correction, and a later request
|
||||
with the same session resumes from the watermark.
|
||||
- a zero sum has the sign IEEE gives the same addends: `-0.0` only if every addend is
|
||||
`-0.0`, and `+0.0` for an exact cancellation (a property test against IEEE).
|
||||
|
||||
## Open questions
|
||||
|
||||
None.
|
||||
+30
-8
@@ -1691,11 +1691,12 @@ probability (`> ||[||]` is a self-parent), so `generated_file_parses` accepts a
|
||||
`slugify` and `IdAllocator` are the first piece of ADR 007. They are pure
|
||||
functions of the display and the allocator's own history. Nothing calls them
|
||||
yet: the creation sites move onto them in the re-key step (improvise-cq1q.4).
|
||||
Each rule traces to a numbered answer in the ADR's "What the user decided".
|
||||
Each rule traces to ADR 007's clause 4 (slugify, the counter, hand-written
|
||||
ids) or clause 1 (reserved ids).
|
||||
|
||||
**Case folding is full Unicode folding, and it comes from `unicase`.**
|
||||
`str::to_lowercase` is not folding: `ß` lowercases to itself and is then
|
||||
stripped, so `Straße` would slug to `strae` (answer 26). The same goes for `ſ`
|
||||
stripped, so `Straße` would slug to `strae` (clause 4, step 1: full case folding). The same goes for `ſ`
|
||||
(long s) and the `fi` ligature. `unicase` 2.9 is `no_std` and has no
|
||||
dependencies, so it passes the wasm gate. Its `to_folded_case` applies the
|
||||
C+F mappings of Unicode's CaseFolding.txt, and its own tests pin `Maße` →
|
||||
@@ -1704,20 +1705,39 @@ letter survives step 3. Deriving that ASCII-producing subset of the table by
|
||||
hand was rejected. It is a second copy of the Unicode data, and nothing would
|
||||
say when it drifted.
|
||||
|
||||
**A "space" is U+0020.** Answer 18 says "replace space with `_`", and answer 21
|
||||
says runs collapse. A tab or a no-break space is not a space here, so it is
|
||||
**A "space" is U+0020.** The accepted clause 4 says "replace each run of spaces
|
||||
with one `_`". A tab or a no-break space is not a space here, so it is
|
||||
stripped by step 3. Only a run of U+0020 becomes one `_`. An underscore the
|
||||
display already had is never merged with the one a space run makes. So `a_ b`
|
||||
slugs to `a__b`: the steps are applied as written, and no extra
|
||||
underscore-collapsing step exists. A trailing run survives, so `East ` slugs
|
||||
to `east_`. Only *leading* underscores are removed (step 4).
|
||||
|
||||
**ADR 007's amendment changes these rules, and this code predates it.** The
|
||||
amendment (`context/adr/007-stable-identities-amendment-1.md`, accepted 2026-10-09)
|
||||
says in its clause 4:
|
||||
- every character outside `a`–`z`, `0`–`9` and `-` becomes `_` instead of being
|
||||
stripped (not only U+0020), so `Region(code)` slugs to `region_code`, while
|
||||
`Q1-2024` keeps its `-`;
|
||||
- runs of `_` collapse, so `a_ b` and `a__b` both slug to `a_b`;
|
||||
- leading and trailing `_` and `-` are removed, so `East ` → `east`, `-East-` →
|
||||
`east`, and `-` alone takes `<kind>-<n>`;
|
||||
- `n` is prepended to a slug that starts with a digit, so `2024` → `n2024`;
|
||||
- a hand-written id must obey every syntax rule of a generated one.
|
||||
|
||||
This section, `slugify`, and the tests
|
||||
`slugify_collapses_a_run_of_spaces_to_one_underscore` (its `east_` case) and
|
||||
`slugify_keeps_an_underscore_the_display_already_had` describe the earlier rule
|
||||
above. They change in a follow-up the amendment calls for. The amendment's
|
||||
clause 4 lists every difference.
|
||||
|
||||
**The allocator records every id it admits and releases none.** That is the
|
||||
whole of retirement (clause 4, "within a session a removed id is never handed
|
||||
out again"). No release operation exists, so a removed id cannot be reissued
|
||||
by construction; no check refuses it. The counter starts at 1, is shared by
|
||||
every kind, and advances each time a suffix is tried, including a candidate
|
||||
that turns out to be taken by a hand-written id (answers 28 and 29). So
|
||||
that turns out to be taken by a hand-written id (clause 4: the `gensym` counter, and hand-written
|
||||
ids are canonical). So
|
||||
`east-7` may follow `east` directly, and no order is enforced among suffixes.
|
||||
**The walk past taken candidates is a loop, never recursion.** A
|
||||
new-version file saved from a non-ASCII import holds `item-1` … `item-N`, and
|
||||
@@ -1726,15 +1746,17 @@ version recursed once per candidate, not in tail position, and aborted the
|
||||
process on a stack overflow: in debug at N ≈ 11,000, in release at 50,000.
|
||||
An abort, not a panic, so the TUI lost unsaved edits (Codex P2 on PR #318).
|
||||
An empty slug always takes a suffix (`category-1`), even when the bare kind
|
||||
name happens to be free, because answer 16(e) spells the result
|
||||
name happens to be free, because clause 4 spells the result
|
||||
`<kind>-<n>`.
|
||||
|
||||
**`claim` is the only door for a hand-written id.** It checks, in this order:
|
||||
the alphabet (non-empty, `a`–`z`, `0`–`9`, `-`, `_`), the reserved leading
|
||||
`_` (only `_index`, `_dim` and `_measure` may carry one, per answer 23, and
|
||||
`_` (only `_index`, `_dim` and `_measure` may carry one, per clause 1, and
|
||||
the match is exact: `_indexer` and `_measure-1` are Reserved, since "no other
|
||||
id may start with `_`"), and
|
||||
duplicates. It never renames. Refusing is the loader's job, and the loader
|
||||
duplicates. It accepts `-`, `-x`, `x-`, `a__b` and `2024`, all of which the
|
||||
amendment refuses (a hand-written id obeys the syntax of a generated one). It never renames.
|
||||
Refusing is the loader's job, and the loader
|
||||
must name both lines of a duplicate; `ClaimError` carries only the id, so the
|
||||
loader keeps the line numbers. The virtual views `_records` and `_drill` are
|
||||
not yet in `VIRTUAL_IDS`. improvise-p003 item 1 asks for them, and they arrive
|
||||
|
||||
@@ -18,7 +18,7 @@ Published, in four sections:
|
||||
|
||||
| Section | Contents | Why |
|
||||
|---|---|---|
|
||||
| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md` | What the project is for, the doctrine, and the single best explanation of how it is built. |
|
||||
| Start here | `docs/design-notes.md`, `context/design-principles.md`, `context/repo-map.md`, `context/gotchas.md`, `context/file-inventory.md` | What the project is for, the doctrine, and the single best explanation of how it is built. The map is a digest; the last two are its long forms, published so a `gotcha N` citation and the per-file listing have somewhere to land. |
|
||||
| Decision records | `context/adr/*.md` | An ADR is written for an outside reader by construction — it states the options that were rejected, which is the part that is worth reading and the part that is nowhere else. |
|
||||
| The arguments, by area | `context/code/*.md` | This is the reasoning that used to live in code comments and was moved out under the two-line cap (design-principles §6). It is the most valuable prose in the repository and it is addressed to exactly the reader a documentation site has. |
|
||||
| Test evidence | `context/test-evidence/*.md` | One entry per test whose narrative is longer than its code. |
|
||||
@@ -43,7 +43,7 @@ Not published:
|
||||
will meet, and nobody visiting a website is about to read a PR thread.
|
||||
|
||||
None of the three is referenced by any published document except as a row in
|
||||
`repo-map.md`'s file inventory, which is inside a code fence, so leaving them
|
||||
the file inventory (`file-inventory.md`), which is inside a code fence, so leaving them
|
||||
out creates no dangling link. That was checked rather than assumed.
|
||||
|
||||
**And "not published" is now derived rather than assumed** (improvise-res9).
|
||||
@@ -233,11 +233,13 @@ recorded as a decision rather than left to be rediscovered as a bug.
|
||||
|
||||
## 6. Navigation
|
||||
|
||||
`repo-map.md` is one long document whose gotcha list is numbered 1 to 48, and
|
||||
splitting it per gotcha was rejected: the gotchas cross-reference each other
|
||||
constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a reader
|
||||
following those wants a scroll, not 48 page loads. It stays one page, with an
|
||||
id on every list item and a compact strip of numbered links under the heading.
|
||||
`repo-map.md` carries a digest of the numbered gotchas and `gotchas.md` the long
|
||||
form; the map was condensed because it was too long to put in every new agent's
|
||||
context. Splitting per gotcha was rejected: the gotchas cross-reference each
|
||||
other constantly ("see gotcha 35", "for the reason gotcha 39 watches…"), and a
|
||||
reader following those wants a scroll, not one page load per number. Each of the
|
||||
two stays one page, with an id on every list item and a compact strip of
|
||||
numbered links under the heading.
|
||||
The ids come from the list item's **own number**, not from a counter, so an
|
||||
inserted gotcha renumbers the anchors exactly as it renumbers the prose that
|
||||
cites them.
|
||||
|
||||
@@ -24,7 +24,7 @@ over `unescape_pipe_body`.
|
||||
|
||||
Before improvise-i2m5 there were **three** copies and they had drifted, over
|
||||
the keyword rule below. **`improvise-io` no longer has one** — its
|
||||
`persistence/quoting.rs` is deleted. `context/repo-map.md`'s file inventory
|
||||
`persistence/quoting.rs` is deleted. `context/file-inventory.md`
|
||||
kept claiming io "still has its own unguarded copy" long after the deletion —
|
||||
it was corrected in wave 1, but any *other* doc or comment still saying so is
|
||||
wrong, and that is the shape this particular fact goes stale in.
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
# `numtower`: the reasoning
|
||||
|
||||
`crates/numtower` is ADR 005 migration step 1 (improvise-59nw.1): the numeric tower,
|
||||
first and alone. Nothing in improvise depends on it yet, so it changes no behaviour.
|
||||
The decisions are the ADR's (`context/adr/005-exact-decimal-arithmetic.md`). This file
|
||||
records how the crate reads them, the places the ADR left a choice open or said two
|
||||
things, and the measurements. The evidence for each test is in
|
||||
`context/test-evidence/numtower.md`.
|
||||
|
||||
## The shape
|
||||
|
||||
| type | what it is | ADR |
|
||||
|---|---|---|
|
||||
| `Num` | the stored value: an exact `Decimal`, an exact `Ratio<i128>`, or an `f64` | clauses 1, 2 |
|
||||
| `NumRef` | a borrowed view of a `Num`'s arm, for exhaustive matching | — |
|
||||
| `Interim` | a value inside one evaluation; exact intermediates widen to a bigint rational | question 13 (b) |
|
||||
| `NumSum` | a retractable, order-free `SUM` over any `Num` | clause 8, question 16 |
|
||||
| `ExactSum` | ADR 009's f64 superaccumulator | question 15 (a) |
|
||||
| `parse_number`, `is_number`, `Typing` | clause 3's number language, gate and parse | clause 3, question 12 |
|
||||
| `Display` / `Num::from_canonical` | the canonical text: `12.50`, `10/3`, `#i0.1`, `inf`, `-inf`, `nan` | clauses 10, 11, 14 |
|
||||
| `Num::round_to_places` | display rounding, the mode a parameter | clause 12 "Format" |
|
||||
|
||||
**`Num` is a struct with private arms, not the ADR's `pub enum`.** The ADR's sketch
|
||||
(clause 2) shows `pub enum Num { Exact(Decimal), Fraction(Box<Ratio<i128>>),
|
||||
Inexact(f64) }`. A public enum would let any caller build values the tower never
|
||||
produces: `Fraction(1/2)` (which must be the decimal `0.5`), an unreduced fraction, or an
|
||||
exact `-0`. Each would break exact equality or the fingerprint of clause 9. So the arms
|
||||
are private, every constructor normalizes, and callers match on `NumRef`. That is
|
||||
`context/design-principles.md` §3, smart constructors. It is also what clause 14's table
|
||||
asks for, since improvise calls "`Num`'s accessors for exactness, mantissa, scale and
|
||||
fraction parts". The `Box` lives in one constructor, as `AGENTS.md` asks. `Num` is 24
|
||||
bytes, as the ADR measured, and `a_num_is_24_bytes` pins it.
|
||||
|
||||
**The normal form.** An exact zero has no sign. A fraction is reduced and has a positive
|
||||
denominator. A value whose reduced denominator is `2^p·5^q`, with `k = max(p, q) <= 28`
|
||||
and a mantissa that fits 96 bits, is always the `Decimal`, never a `Fraction`. So
|
||||
`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. Every `i128`
|
||||
is a valid numerator, `i128::MIN` included, because the arm is `Ratio<i128>`. That one
|
||||
value cannot be negated in `i128`, so `Operand::parts` keeps it out of the `i128` fast
|
||||
path and its arithmetic takes the bigint path (Copilot on PR #337).
|
||||
|
||||
## Arithmetic
|
||||
|
||||
### Why there are two types: `Num` and `Interim`
|
||||
|
||||
Question 13 (b) bounds only the **stored** result: "every intermediate inside one formula
|
||||
evaluation is unbounded up to the ceiling". The bounded `Num` cannot be such an
|
||||
intermediate, so `Interim` is the widened type. It is either a stored `Num` or an exact
|
||||
`BigQ` (a reduced `BigInt` fraction) that does not fit `i128`. `Interim::store` applies
|
||||
the stored bound. The operators on `&Num` are one-operation evaluations: each builds the
|
||||
`Interim`, applies the operation and stores the result. So `(a * a) / a` returns
|
||||
`a` exactly when computed through `Interim`, and is inexact when computed through `Num`
|
||||
(`an_intermediate_beyond_i128_comes_back_exact`). That difference is (b)'s whole point.
|
||||
Step 3's evaluators should evaluate through `Interim` and store once per cell.
|
||||
|
||||
**So `Interim` compares, and knows zero, without storing.** `IF(a*a/a = a, …)` and the
|
||||
division guard both run before the stored bound. If `=` had to store its operands first,
|
||||
its answer would depend on how the formula is spelled, which is what question 13 (b)
|
||||
rules out. `Interim::cmp_exact` is clause 6's exact order over every pairing: two stored
|
||||
values go through `Num::cmp_exact`, and anything wide compares as a bigint rational,
|
||||
against an f64 by its exact binary value. `Interim::is_zero` is the one zero predicate. A
|
||||
wide value is never zero, because zero always fits a stored arm. The comparison through
|
||||
storing is the rival (`an_interim_compares_exactly_without_storing_catches_storing_first`):
|
||||
`a` and `a ± 1/(3·10^30)` store to one f64 and compare equal. The review of `302594bf`
|
||||
asked for this or for a note naming it as a step-3 item; the crate does the first.
|
||||
|
||||
### Contagion, and how many roundings a demotion has
|
||||
|
||||
An inexact operand makes the result inexact. Each exact operand is converted by the
|
||||
correctly rounded conversion, and then the f64 operation runs. That is two roundings,
|
||||
which is clause 4's "computed in f64".
|
||||
|
||||
**An exact ∘ exact result that does not fit is rounded once.** It is the correctly rounded
|
||||
f64 of the exact value. Clause 4 still says a demoted sum or product is "rounded twice,
|
||||
once per operand and once in the operation". That text predates option (d). Option (d)'s
|
||||
"Operations" and question 13 (b) both say that a result failing the stored bound is "the
|
||||
correctly rounded f64 of the exact fraction", and under (b) the exact value is always in
|
||||
hand. The crate follows (d), because (d) is decided and later. Both texts cannot hold for
|
||||
one operation, so this is a reading, and it is flagged.
|
||||
|
||||
### Clause 4: the hint, the recompute, the stored scale
|
||||
|
||||
`Decimal + Decimal` and `Decimal × Decimal` first try `rust_decimal`'s checked operation.
|
||||
The result is trusted only when its scale equals the nominal one (`max(sa, sb)` for `+`
|
||||
and `−`, `sa + sb` for `×`). A lower scale is the **hint**, and so is a `None`. On the
|
||||
hint the exact integer result is recomputed in a 256-bit integer, and its trailing zeros
|
||||
are stripped. If the stripped mantissa fits 96 bits at a scale of at most 28, the result
|
||||
is a decimal at the **stored scale**, `min(nominal, 28)`, padded back with zeros while the
|
||||
mantissa fits. Otherwise the result goes to the fraction arm, then to a wide `Interim`.
|
||||
The ADR's own example holds: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28
|
||||
(`a_recomputed_result_keeps_the_stored_scale_clause_4_names`). A `None` counts as the
|
||||
hint, so that an overflow `rust_decimal` reports is never mistaken for an exact result.
|
||||
|
||||
### Division: the direct test, one implementation
|
||||
|
||||
Clause 5 is normative: write the quotient as a reduced `n / (2^p·5^q)`, then it is a
|
||||
decimal iff `k <= 28` and the padded mantissa fits. The crate does this the same way for
|
||||
every exact pairing, not only for decimals. It reduces the quotient (in `i128` when that
|
||||
cannot overflow, and in a bigint otherwise), then normalizes. `small_ratio` and
|
||||
`decimal_of` are the one normalization, so a decimal reached through any path has the
|
||||
same representation. Multiplying back is never used.
|
||||
|
||||
**A quotient carries its minimal scale** (`10.00 / 2` is `5`, `1 / 4` is `0.25`). The ADR
|
||||
does not say what scale a quotient has. A quotient's scale is not a function of its
|
||||
operands' scales (clause 5), and a result that passed through the fraction arm carries
|
||||
its minimal scale (option (d)), so the minimal scale is the consistent choice. It is a
|
||||
reading, and it is flagged.
|
||||
|
||||
**The ADR's argument against multiplying back is weaker than it says.** Clause 5 rejects
|
||||
it as incomplete, on `1e-28 / 0.5`: `q × 0.5` "needs scale 29, so the check flags it".
|
||||
That holds under revision 6's rule, which demotes on the hint
|
||||
(`division_decides_fit_directly_catches_multiplying_back` keeps that rival). But under
|
||||
revision 7's recompute the product `1e-28` is found exact after its zeros are stripped.
|
||||
So multiplying back by `Num`'s own `*` accepts the ADR's counterexample. In 20 runs of the
|
||||
division property (5,120 random cases plus the three seeds), it agreed with the direct
|
||||
test every time. That was an experiment and is not committed. The direct test stays the
|
||||
rule, because the ADR decided it and because it needs no second division.
|
||||
|
||||
### `POW`
|
||||
|
||||
- **The exponent.** It is exact when it is an exact **integer value**, so `2.0` counts.
|
||||
- **The size, decided before any multiplication.** The result's numerator has at least
|
||||
`|n|·(bits(num) − 1)` bits, and likewise its denominator. If either lower bound passes
|
||||
the ceiling, the result is `powf`, inexact, with nothing allocated. So
|
||||
`POW(1/3, 10^9)` costs nothing. Inside the bound the result is computed exactly, and it
|
||||
can still exceed the ceiling (`3^6000` is 9,510 bits). It then demotes, which is the
|
||||
ceiling's stated exception.
|
||||
- **The scale.** A positive power of a decimal takes clause 4's stored scale with nominal
|
||||
`s·n`, the scale repeated multiplication would give, so `1.50^2` is `2.2500`. A
|
||||
negative power is a division and has its minimal scale, **unit bases included**:
|
||||
`1.0^-2` is `1` and `(-1.0)^-3` is `-1`, as `2.0^-2` is `0.25`. Until the review of
|
||||
`302594bf` the unit-base arm padded every power to `s·|n|`, so `1.0^-2` read `1.00`
|
||||
(Copilot on PR #337). The power property now checks the scale of every decimal result
|
||||
against this rule, with unit bases among its seeds.
|
||||
- **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`,
|
||||
including one too large to compute.
|
||||
- **A demoted power takes its sign from the exact exponent.** `powf` gets the base's
|
||||
magnitude, and the sign is set from `base < 0 && n odd`. An exponent above 2^53 rounds
|
||||
to an even f64, so handing `powf` the signed base made `(-2)^(2^53 + 1)` read `+inf`
|
||||
(Copilot on PR #337).
|
||||
|
||||
### The ceiling
|
||||
|
||||
`CEILING_BITS = 8,192`. It applies to the bit length of an intermediate's numerator *or*
|
||||
denominator. The ADR states the ceiling as a width, not as which of the two it measures.
|
||||
Past it, the intermediate becomes the correctly rounded f64 of its exact value.
|
||||
|
||||
## Comparison
|
||||
|
||||
- **Exact with exact.** Cross-multiplication in 256 bits. A decimal is `m / 10^s` with
|
||||
`10^28 < 2^94`, so every product fits.
|
||||
- **Exact with inexact.** The ADR's method: convert the exact operand, and the rounded
|
||||
value decides unless it lands on the f64 itself. A tie is resolved against the f64's
|
||||
exact binary value, in a bigint.
|
||||
- **That fast path is sound only because the conversion is correctly rounded.** Injecting
|
||||
`rust_decimal`'s `to_f64` failed `exact_mixed_comparison_is_transitive` as well as the
|
||||
conversion test. A future integer conversion (clause 4 allows one) must keep that.
|
||||
|
||||
`MIN` and `MAX` skip NaN. Equal values break their tie as follows. Only the first rule is
|
||||
the ADR's (question 11); the other two are this crate's, and they make the result
|
||||
independent of argument order:
|
||||
- the exact value is preferred to the inexact one (question 11);
|
||||
- of two equal decimals, the larger scale is kept, for the same reason: it keeps more of
|
||||
what was typed;
|
||||
- of `0.0` and `-0.0`, `MIN` takes `-0.0` and `MAX` takes `0.0`, as IEEE 754's
|
||||
`minimum` and `maximum` do.
|
||||
|
||||
## The number language
|
||||
|
||||
**The gate is literally `f64::from_str` reading a finite value.** The parse is the
|
||||
crate's own lexer, and it never goes through an f64 on an exact path. The property test
|
||||
checks that the two agree, and that the exact value read rounds to the f64
|
||||
`f64::from_str` reads.
|
||||
|
||||
- **A lexeme's scale.** It is the scale the lexeme implies (`1.50` → 2, `1.5e3` → 0). When
|
||||
the implied scale is above 28 but the value fits a decimal (`0.1` written with 29
|
||||
places), clause 4's stored-scale rule applies: strip the zeros, then pad toward 28.
|
||||
The ADR only says "with the scale the lexeme implies", which cannot exceed 28.
|
||||
- **Zero.** It is exact, unsigned, at the implied scale clamped to 0–28, so
|
||||
`0e99999999999999999999` is `0`.
|
||||
- **An exponent too long to read** saturates, because no exact arm can reach it. A
|
||||
significand of more than 180 digits, or a leading digit outside 10^-130 to 10^40, goes
|
||||
straight to f64. Those bounds are wider than the fraction arm can reach (a reduced
|
||||
denominator `10^k / 2^a` is at least `2^k`, so `k < 127`).
|
||||
- **The forced `n/d` arm.** Its value is a decimal, then a fraction, then **the nearest
|
||||
f64**. The ADR says `n/d` "is a fraction or, once normalized, a decimal", and does not
|
||||
say what a forced `n/d` too large for `i128` becomes. The crate applies clause 3's order
|
||||
to it, as it does to every other spelled number. This is a reading, and it is flagged.
|
||||
**Except that the nearest f64 must be finite.** A forced `n/d` whose value rounds to
|
||||
infinity (`n` of 400 digits, `/1`) is text, exactly as the gate makes `1e999` text.
|
||||
Clause 3: "a finite lexeme must never read as infinity". One that underflows is the
|
||||
inexact zero, as `1e-999` is. `is_number` and `parse_number` share the one helper,
|
||||
`forced_fraction`, so the gate and the parse cannot disagree on it.
|
||||
|
||||
**The canonical text is `Display`.** An inexact finite value is `#i` followed by Rust's
|
||||
`{:?}` spelling, which is the shortest that reads back (`#i1e300`, `#i0.1`).
|
||||
`from_canonical` reads all six spellings back, and reads an exact spelling only as an
|
||||
exact value. Clause 14's list puts "`#i…`" in the crate, while its table says "the `#i`
|
||||
prefix and the grammar are improvise's". Those two contradict each other. The crate
|
||||
holds the prefix, because the wire and the file both need the one spelling, and step 7
|
||||
may move the prefix out if the grammar wants it.
|
||||
|
||||
## Rounding to places
|
||||
|
||||
Every arm goes through one exact path: the value as a bigint fraction,
|
||||
`n·10^places / d`, then rounded by the caller's mode (`HalfAwayFromZero` or `HalfEven`).
|
||||
An inexact value is first read exactly from its shortest round-trip spelling, which is
|
||||
improvise-felo.9's fix, so a typed-then-computed `1.005` shows `1.01`. A result that
|
||||
rounds to zero has no sign (`-0.001` at two places is `0.00`). The ADR does not decide
|
||||
that, and today's f64 path would print `-0.00`.
|
||||
|
||||
## Sums
|
||||
|
||||
`NumSum` keeps five things, and every one is an abelian group, so the state is a function
|
||||
of the multiset of terms and not of their order.
|
||||
|
||||
| part | holds | width |
|
||||
|---|---|---|
|
||||
| the decimal partition | every decimal, and every fraction whose denominator is `2^p·5^q` with `k <= 28`, as an integer at scale 28 | 320 bits |
|
||||
| the groups | per other reduced denominator, the summed numerator | 256 bits each, in a `BTreeMap` |
|
||||
| `group_floats` | an `ExactSum` of each group's correctly rounded value | ADR 009's |
|
||||
| `inexact` | an `ExactSum` of the f64 terms | ADR 009's |
|
||||
| `inexact_terms` | the net weight per f64 bit pattern, entries at zero dropped | a `BTreeMap` |
|
||||
|
||||
**The read.** It is exact when no inexact term is present and at most K = 8 groups
|
||||
remain. The groups and the decimal partition are then combined **over the lcm of their
|
||||
denominators, and reduced once**, and the stored bound applies to the result. Past K the
|
||||
read is the f64 accumulators: `group_floats`, the decimal partition converted once, and
|
||||
the inexact terms. That reading rounds each group, then the total. With inexact terms and
|
||||
at most K groups, the exact total is converted once and added to the inexact terms. A
|
||||
group whose numerator returns to zero is dropped, so "how many groups" is a function of
|
||||
state.
|
||||
|
||||
**"Is an inexact term present" is asked per value, never of the net weight.** The first
|
||||
version kept one net count. Inserting `1.0` and retracting `2.0` leaves the count at zero
|
||||
while the accumulator holds −1.0, so the read took the exact branch and dropped it
|
||||
(ChatGPT Codex, P1 on PR #337). Presence is now a map from f64 bit pattern to net weight,
|
||||
dropped at zero, the same rule the groups follow. It costs O(distinct inexact values),
|
||||
the same order as the groups. The K cutoff overrides representability: nine denominators whose exact total fits
|
||||
still read inexact (`more_than_k_denominators_read_inexact_even_when_the_total_fits`).
|
||||
That is question 13's second exception, and the ADR's flagged reading.
|
||||
|
||||
**`ExactSum` is ADR 009's prototype**, from `crates/improvise-core/examples/retraction_drift.rs`
|
||||
on PR #309. It has three changes:
|
||||
- it splits a weight into base-2^30 digits, as ADR 009 says the `numtower` version must,
|
||||
instead of looping per chunk;
|
||||
- it holds 70 limbs, not 67, because a digit's shift adds up to 60 bits;
|
||||
- it reads through the crate's one rational-to-f64 rounding.
|
||||
|
||||
It counts NaN and the infinities, so they can be retracted. A count that is nonzero in
|
||||
either direction is treated as present.
|
||||
|
||||
**`ExactSum`'s `==` compares sums, not representations.** A derived equality compared the
|
||||
raw limbs and the `pending` carry counter, so `0.5 + 0.5 != 1.0`, an insert then retract
|
||||
was not equal to an empty sum, and the same weights added in another order compared
|
||||
unequal (Copilot on PR #337). For a type whose whole point is order-freedom, and which
|
||||
ADR 009's group newtype will wrap, that equality is a trap. It now compares the
|
||||
propagated limbs, which are canonical (every limb but the top in `[0, 2^32)`), plus the
|
||||
three special-value counts, and ignores `pending`. `NumSum` derives no equality: compare
|
||||
its `read()`.
|
||||
|
||||
**For ADR 009's seam crate.** `NumSum` and `ExactSum` both have the shape a Z-set group
|
||||
needs: `add(&Num, i64)` / `add(f64, i64)`, `insert`, `retract`, `read`/`value`, `Clone`.
|
||||
`ExactSum` also has `absorb`, to merge two accumulators. Neither implements a group trait
|
||||
yet, because `zset` does not exist yet. The newtype of question 15 is where that goes.
|
||||
|
||||
## The wide integers
|
||||
|
||||
`Fixed<N>` is a two's-complement integer of `N` 64-bit limbs that never allocates. It is
|
||||
used for the clause 4 recompute (256 bits), exact comparison (256 bits), the decimal
|
||||
partition (320 bits) and the group numerators (256 bits). Clause 14 also names "256-bit
|
||||
helpers for the division reduction". Division reduces in `i128` when the products cannot
|
||||
overflow, and otherwise in a bigint. It does not have a hand-rolled 256-bit gcd. That
|
||||
path is the slow path, and it is measured below.
|
||||
|
||||
## Dependencies
|
||||
|
||||
The normal edges are `rust_decimal` 1.43.0 (std only), `num-rational` 0.4.2 (no default
|
||||
features), `num-bigint` 0.4.8, `num-integer` 0.1.47 and `num-traits` 0.2.19. The last two
|
||||
are direct dependencies, used for `gcd`/`lcm` and the numeric traits. They were already
|
||||
in the tree through `num-rational` and `num-bigint`, so they add no crate. The ADR names
|
||||
only the first three. There is no serde, and `cargo tree --edges normal` shows no
|
||||
`borsh`. `BigRational` is a **dev**-dependency feature only, for the oracle, so the
|
||||
oracle and the implementation do not share a rational type.
|
||||
|
||||
## Measured
|
||||
|
||||
On an Apple M4 Max (64 GB), rustc 1.98.1, `cargo run --release -p numtower --example
|
||||
bench`. Each range is two runs.
|
||||
|
||||
| what | figure |
|
||||
|---|---|
|
||||
| SUM, 18k and 90k typed two-place values, ns per element | f64 0.53; `rust_decimal` fold 2.7–3.0; `Num` `+` fold 7.8–8.2; `NumSum` insert and read 7.9–8.1 |
|
||||
| per operation over 18,000 typed pairs, ns | `Num` add 3.1–3.3, mul 3.0–3.2, div 146–153; `rust_decimal` checked add 1.4–1.5, mul 1.2, div 24.7–25.3 |
|
||||
| `(Rev − Cost) / Rev`, ns per record | 151–153 |
|
||||
| `to_f64` of a decimal (the text path), ns | 36.7–37.1 |
|
||||
| `round_to_places(2)`, ns | 370–382 |
|
||||
| `NumSum` read, K distinct 126-bit denominators, numerators of about 190 bits | K = 2: 5.2–5.4 µs; K = 8: 41.4–42.0 µs |
|
||||
| `ExactSum`, 1M values | 1.78–1.81 ns per insert, 0.29 µs per read |
|
||||
|
||||
Here is how these sit against the ADR:
|
||||
- **The grouped read.** At K = 8 it is inside the ADR's 52–127 µs. The ADR measured
|
||||
250-bit numerators, and these are about 190 bits, because `NumSum` can only reach a
|
||||
group's numerator through its public `add`.
|
||||
- **`ExactSum`.** It agrees with ADR 009's 2.0 ns per update and 0.58 µs per read.
|
||||
- **Division.** At about 150 ns it is above the ADR's *estimate* of 60–110 ns. Most
|
||||
quotients of typed values do not terminate, so each one pays two `i128` gcds and the
|
||||
`Box` of a `Fraction`.
|
||||
- **The rest.** The ADR's whole-recompute measurement at 18k and 90k cells waits for
|
||||
step 3, as the ADR says.
|
||||
|
||||
**The wasm32 cost** comes from a probe outside the repository, built exactly as the
|
||||
ADR's appendix builds one: a `cdylib`, `opt-level = "s"`, LTO, `wasm-opt -Oz
|
||||
--enable-bulk-memory --enable-nontrapping-float-to-int`, `gzip -9`. The baseline already
|
||||
parses and prints an f64.
|
||||
|
||||
| probe | after `wasm-opt -Oz` | gzipped |
|
||||
|---|---|---|
|
||||
| f64 only | 48,331 B | 29,162 B |
|
||||
| the ADR's five operations (parse, `+`, `−`, `/`, print) through `numtower` | +89,375 B | **+36,714 B** |
|
||||
| plus `NumSum`, `cmp_exact` and `round_to_places` | +119,277 B | **+48,221 B** |
|
||||
|
||||
The narrow figure confirms the ADR's estimate of "up to about +36 KB together". The full
|
||||
surface is about +47 KB gzipped. That is what the bundle pays once step 4 sums through
|
||||
the crate and step 6 rounds through it. It is more than the +15.4 KB the user answered
|
||||
question 8 against, and the ADR says question 8 "reopens" if the figure changes the
|
||||
answer. Neither figure is the real bundle, which shares allocator and formatting code
|
||||
with everything else.
|
||||
@@ -902,7 +902,7 @@ The feature is on for **every workspace crate with a normal dependency on
|
||||
serde_json** (improvise-dz8c). Each crate's own manifest names it, because each
|
||||
build is made from one crate alone: the `wasm` CI job and `wasm-pack` invoke
|
||||
per crate, and feature unification cannot reach across separate invocations
|
||||
(`context/repo-map.md`, `## Wasm`). Enabling it in one crate and relying on the
|
||||
(`context/file-inventory.md`, appendix "Wasm"). Enabling it in one crate and relying on the
|
||||
others to inherit it works in `cargo test` and nowhere else.
|
||||
|
||||
**Why "can parse" and not "parses a frame".** The first version of the rule
|
||||
|
||||
@@ -0,0 +1,430 @@
|
||||
# File Inventory (the long form)
|
||||
|
||||
The per-file listing that `context/repo-map.md` summarises. Moved here unchanged.
|
||||
|
||||
Line counts are a **dated snapshot**, not a maintained fact: measured 2026-09-24 at `c1772ca9`, and **70 of 105 had drifted past 10%** in the three days since the 2026-09-21 refresh — eleven had doubled or more (`client.rs` 255 → 943, `lockstep.rs` 1126 → 4064). Treat them as "is this file big", never as a figure to cite; re-derive with `wc -l`. The `ws-server/` and `wasm-client/` entries use a shorthand prefix, so a refresh must map it to `crates/improvise-<name>/{src,}/` by hand or they drift silently. Test counts are informational — run `cargo test --workspace` for live numbers. Files under 100 lines and render-only widgets omitted.
|
||||
|
||||
### `improvise-core` (`crates/improvise-core/src/`)
|
||||
```
|
||||
model/types.rs 3560 / 112t Model, formula eval (fixed-point), CRUD
|
||||
model/cell.rs 1358 / 47t CellKey (sorted), CellValue, DataStore (interned + index),
|
||||
live_records (insertion order — gotcha 18)
|
||||
model/record.rs 226 / 8t RecordHash / CellId surrogates: the opaque identities a
|
||||
thin client names a cell by (ADR 002)
|
||||
model/category.rs 310 / 9t Category, Item, Group, CategoryKind
|
||||
model/symbol.rs 89 / 3t Symbol interning
|
||||
view/layout.rs 2604 / 57t GridLayout, drill, records mode, records_columns; a private
|
||||
per-axis position index (`row_pos`/`col_pos`, rebuilt by `reindex`)
|
||||
so a data position resolves in O(1) (improvise-felo.13)
|
||||
view/types.rs 817 / 43t View config (axes, pages, hidden, collapsed, format)
|
||||
model/id.rs 290 slugify + IdAllocator: how a generated id is made (ADR 007
|
||||
clause 4, core.md §20). Nothing outside the file calls it yet
|
||||
view/axis.rs 190 / 5t Axis enum + label/next methods (no colour: that is a Face)
|
||||
workbook.rs 413 / 22t Workbook: Model + cross-view ops, item facade
|
||||
format.rs 519 / 40t format_f64, parse_number_format (display only)
|
||||
```
|
||||
|
||||
### `improvise-formula` (`crates/improvise-formula/src/`)
|
||||
```
|
||||
parser.rs 1394 / 79t pest grammar + tokenizer → Formula AST
|
||||
quote.rs 160 / 7t quote_name / is_bare_name / pipe_quote / unescape_pipe_body:
|
||||
the one home, both halves (improvise-i2m5). improvise-gp
|
||||
re-exports it; improvise-io calls it from
|
||||
persistence/{format,parse}.rs
|
||||
formula.pest 108 PEG grammar for formula text
|
||||
ast.rs 130 Expr, BinOp, AggFunc, Formula, Filter
|
||||
```
|
||||
|
||||
### `improvise-io` (`crates/improvise-io/src/`)
|
||||
```
|
||||
storage.rs 334 / 13t Storage trait (read/write/list/delete), PhysicalFs, MemoryFs,
|
||||
`impl Storage for Rc<T>` (improvise-hmws), and
|
||||
contract::storage_contract — public, not cfg(test), because
|
||||
a browser must run that sequence (harness/)
|
||||
opfs.rs 289 / 0t OpfsFs: Storage over the browser's Origin Private File
|
||||
System. wasm32 only. Run by harness/, not by cargo test
|
||||
persistence/improv.pest 179 PEG grammar — single source of truth
|
||||
persistence/mod.rs 441 / 11t document lifecycle: save/load/gzip/legacy-JSON over Storage;
|
||||
facade re-exporting format_md/parse_md/export_csv*
|
||||
persistence/format.rs 401 / 11t format_md: Workbook → .improv text
|
||||
persistence/parse.rs 1684 / 46t parse_md: pest tree-walk → Workbook (+ grammar generator)
|
||||
persistence/export.rs 243 / 5t export_csv / render_csv. Builds its layout with
|
||||
with_frozen_records, NOT new — `new` skips prune_empty
|
||||
(improvise-iot6)
|
||||
persistence/roundtrip_tests.rs
|
||||
1236 / 62t format_md ↔ parse_md characterization (tests only)
|
||||
import/wizard.rs 1967 / 62t ImportPipeline + ImportWizard. `Plan::place` is the one
|
||||
placement pass, shared by `collisions()` and the build so
|
||||
the two cannot disagree; `DuplicatePolicy` / `Collision`
|
||||
(improvise-cgqt)
|
||||
import/analyzer.rs 434 / 17t Field kind detection (Category/Measure/Time/Skip);
|
||||
analyze_records is first-seen order, not HashSet (gotcha 18)
|
||||
import/csv_parser.rs 590 / 26t CSV parsing, multi-file merge
|
||||
```
|
||||
|
||||
### `improvise-protocol` (`crates/improvise-protocol/src/`)
|
||||
```
|
||||
lib.rs 619 / 2t ViewState, ViewCursor, ViewFrame, AppMode (17), Panel,
|
||||
MinibufferConfig, MinibufColor, HELP_PAGE_COUNT
|
||||
reduce.rs 749 / 10t Command + ModelCommand (the wire format), HelpPage, reduce_view
|
||||
render.rs 2157 / 46t CellDisplay, WidgetCache trait + one cache per widget
|
||||
(GridCache/CategoryTreeCache/FormulaCache/ViewListCache/
|
||||
TileBarCache), AxisFrame/FrameEntry/FrameGroup and
|
||||
header_level_widths (the grid frame, gotcha 44),
|
||||
GridCache::seed (the bootstrap compaction), Projection,
|
||||
bootstrap-only Snapshot
|
||||
keymap.rs 975 / 0t KeyPattern, Binding, Keymap, ModeKey (17), TextEntry,
|
||||
PanelKeymap, Keymaps::default_keymaps/prefix_keymap — the
|
||||
shared table both front ends look up. Tests live with
|
||||
dispatch in improvise-command/src/command/keymap.rs
|
||||
viewmodel.rs 170 / 5t flatten_cat_tree — derivations between a cache and a
|
||||
renderer, called by both front ends (improvise-edp)
|
||||
face.rs 682 Face (a role: Axis, WizardField, Mode*, MinibufferOf…), Theme,
|
||||
AnsiSlot — appearance as data a front end applies (ADR 004,
|
||||
protocol.md §10); draw.rs::face_style is the TUI's one translation
|
||||
key.rs 96 / 3t KeyCode, KeyModifiers, KeyEvent (moved from command/)
|
||||
frame.rs 156 / 3t ClientFrame/ServerFrame — the ONE envelope pair, both
|
||||
deployments (improvise-xks6). Nothing else defines one.
|
||||
ClientFrame::Save is a request, not a Command (gotcha 45)
|
||||
basis.rs 257 / 6t Basis (the three tiers), Rejection, BasisCache — ADR 001
|
||||
layer 1. Basis has no Default and no public constructor
|
||||
help.rs 1241 / 14t HelpPage content and Hint { keys: Vec<(KeyCode, KeyModifiers)>,
|
||||
command, caption }. `compact` drops a chord by its MODIFIERS
|
||||
(anything beyond shift), never by parsing a label it just
|
||||
formatted (improvise-thpe) — the guard is the signature
|
||||
resolve.rs 4883 / 88t the browser's key resolvers: a BrowserKey to the Commands it
|
||||
composes, mirroring each Cmd line for line INCLUDING where
|
||||
siblings disagree (gotcha 47). The largest file in the crate
|
||||
client.rs 943 / 15t ClientState: ViewState + one cache per widget, and the fold;
|
||||
in_flight, `opened` (OpenedCell), notify, settle_status
|
||||
(gotcha 47). EditorSession is in lib.rs beside ViewState
|
||||
```
|
||||
|
||||
### `gp-tree` / `improvise-gp` (`crates/gp-tree/`, `crates/improvise-gp/`)
|
||||
```
|
||||
gp-tree/src/lib.rs 575 / 8t Tree, Spec, eval, simplify, render, breed, Population,
|
||||
SplitMix64 (vendored seeded Rng; seed() keeps the seed so
|
||||
a fit can be shown and recorded, pinned to the published
|
||||
reference vectors)
|
||||
gp-tree/src/data.rs 1000 / 21t raw string rows -> f32 feature vectors: per-column type
|
||||
inference, one-hot, mean-imputation + col.missing flag,
|
||||
ColumnSpec/encode_row, Dataset (incl. warnings),
|
||||
Model = trees + plan + seed + FitConfig
|
||||
gp-tree/src/fit.rs 961 / 32t the search: FitConfig (Default = the CLI's numbers), fit()
|
||||
-> simplified champion, cv_regress/cv_classify,
|
||||
permutation_importance, folds/standardize/mse.
|
||||
Returns data; never prints
|
||||
gp-tree/examples/fit_csv.rs 583 CLI over the above: flags, demo dataset, all printing
|
||||
gp-tree/tests/scoring_known_answers.rs
|
||||
250 / 6t f32 bit patterns + a next_u64 witness: the evidence any
|
||||
change to the scoring paths needs (gotcha 37c)
|
||||
improvise-gp/src/lib.rs 978 / 25t formula_text / fitted_formula: a Tree as IF(...) formula
|
||||
text, validated back through parse_formula. A threshold is
|
||||
the SHORTEST decimal that reads back as the same f64 (not
|
||||
the exact expansion), so a row on a split boundary routes
|
||||
the same in f64 as in f32 eval; non-finite values panic
|
||||
improvise-gp/src/dataset.rs 506 / 15t a View's records -> the feature matrix a fit consumes —
|
||||
the improvise-side counterpart of preprocess. Records mode
|
||||
is already a feature matrix (gotcha 1)
|
||||
improvise-gp/src/fit.rs 1778 / 35t one fit end to end: a View's records in, an improvise
|
||||
Formula and the evidence for believing it out. Not in a
|
||||
command, because none of it is terminal-specific
|
||||
```
|
||||
|
||||
### Command layer (`crates/improvise-command/src/command/`)
|
||||
```
|
||||
cmd/core.rs 475 / 2t Cmd trait, CmdContext, CmdRegistry, parse helpers
|
||||
cmd/registry.rs 903 / 13t default_registry() — all registrations
|
||||
cmd/navigation.rs 507 / 12t Move, EnterAdvance, Page*
|
||||
cmd/cell.rs 724 / 20t ClearCellCommand, YankCell, PasteCell, TransposeAxes, SaveCmd
|
||||
cmd/commit.rs 1304 / 27t CommitAndAdvance, CommitBuffer(CommitKind), CommitExport,
|
||||
CommitSave
|
||||
cmd/effect_cmds.rs 519 / 6t effect_cmd! macro, 25+ simple wrappers, `unless_unknown` (gotcha 51)
|
||||
cmd/fit.rs 293 / 8t fit-measure: evolve a tree over the active view and store it
|
||||
as an ordinary `## Formulas` line — nothing GP-specific is
|
||||
left in the document afterwards
|
||||
cmd/grid.rs 988 / 22t ToggleGroupAtCursor, ViewNavigate, DrillIntoCell, TogglePruneEmpty
|
||||
cmd/mode.rs 469 / 13t EnterMode, Quit, EditOrDrill, EnterTileSelect
|
||||
cmd/panel.rs 936 / 24t Panel toggle/cycle/cursor, formula/category/view panels
|
||||
cmd/search.rs 202 / 4t SearchNavigate, SearchOrCategoryAdd, ExitSearchMode
|
||||
cmd/text_buffer.rs 399 / 14t AppendChar, PopChar, CommandModeBackspace, ExecuteCommand
|
||||
cmd/tile.rs 170 / 6t MoveTileCursor, TileAxisOp
|
||||
cmd/wizard.rs 244 / 6t HandleWizardKey — per-step import-wizard key tables
|
||||
keymap.rs 1353 / 44t KeymapSet, Keymap::dispatch/dispatch_transient, effects_of,
|
||||
SetTransientPrefix — what a thin client cannot have. The
|
||||
table itself is improvise-protocol/src/keymap.rs
|
||||
parse.rs 350 / 23t Script/command-line parser (prefix syntax)
|
||||
```
|
||||
|
||||
### Semantic UI state (`crates/improvise-command/src/ui/`)
|
||||
```
|
||||
ui/effect.rs 4020 / 99t Effect trait. 63 effects: 53 `impl Effect for` blocks, 9
|
||||
simple_effect! expansions, and SetTransientPrefix in
|
||||
command/keymap.rs. The 10 outside the brace-match are all
|
||||
ViewOnly; 28 of the 53 are Model(_) (gotcha 38, h3y7)
|
||||
ui/app.rs 5558 / 134t App, ModelState, ViewState + dispatch methods, Panel, handle_key, dispatch_key, document_generation, autosave; apply_effects declares `dirty` for WholeWorkbook and observes it for CellData/LayoutGeometry (gotcha 45)
|
||||
ui/project.rs 6383 / 118t Dirty, Viewport, grid_of/category_rows/formula_texts/
|
||||
view_summaries/tile_entries, grid_updates (the diff),
|
||||
ClientSession + Sessions (connect/apply/rebase/project_all)
|
||||
ui/reduce.rs 1217 / 24t Checked / CheckedBatch / apply_command / model_effects /
|
||||
reduce_key — one door from a Command or a keypress to the
|
||||
authoritative App. `reduce_full` is GONE (improvise-h3y7)
|
||||
ui/host.rs 3410 / 67t Host: one App + one Sessions; handle, commands, project_all,
|
||||
rebase. Lives here, not in ws-server (gotcha 45)
|
||||
ui/cat_tree.rs 174 / 8t Category tree flattener for panel
|
||||
ui/viewport.rs 259 compute_col_widths / compute_row_header_width / visible_cols
|
||||
(pure geometry; the width rule itself is the protocol's);
|
||||
grid_width, SIDE_PANEL_WIDTH, MIN_GRID_WIDTH (improvise-gtn7.1);
|
||||
GridSummary / grid_summary / grid_window (improvise-felo.13)
|
||||
```
|
||||
|
||||
### TUI rendering, draw, main (`src/ui/`, `src/draw.rs`, `src/main.rs`)
|
||||
```
|
||||
ui/grid.rs 905 / 14t GridWidget (ratatui): no &Model, no &View — the frame comes
|
||||
off GridCache (gotcha 44). Its module doc is the best-written
|
||||
statement of that rule in the repo
|
||||
ui/help.rs 277 5-page help overlay (render only)
|
||||
ui/import_wizard_ui.rs 829 / 9t Import wizard rendering. `Panel` is the one writer, and
|
||||
every write is clipped to the panel's rows and width
|
||||
ui/text.rs 169 truncate and wrap, both measured in terminal cells, never bytes
|
||||
draw.rs 1122 / 20t TUI event loop, frame composition; `content_tests` draws a
|
||||
whole frame to check the selection beside a side panel
|
||||
main.rs 428 CLI entry (clap): open, import, cmd, script
|
||||
# other ui/*.rs are small panel renderers — skip unless changing layout/style
|
||||
tests/grid_frame_agreement.rs
|
||||
479 / 6t the two front ends draw the same grid frame (improvise-anr9)
|
||||
tests/records_wide_smoke.rs 67 / 1t
|
||||
tests/shipped_documents_load.rs 333 / 8t every TRACKED .improv/.improv.gz loads, and every record
|
||||
it projects is writable. "Shipped" is read off `git ls-files`,
|
||||
not a .gitignore reading (improvise-c0aj), and the walk's git
|
||||
is scrubbed of every hostile GIT_* setting the caller may carry
|
||||
tests/web_demo_model.rs 83 / 2t every visible cell in web/demo.improv is a Record
|
||||
```
|
||||
|
||||
### Browser host and client (`crates/improvise-ws-server/src/`, `crates/improvise-wasm-client/src/`, `web/`)
|
||||
```
|
||||
ws-server/registry.rs 616 / 14t Registry of Hosts keyed by DocumentId; ClientId {document,
|
||||
session} — a SessionId is unique only within its document.
|
||||
A document is never evicted while the server runs (ADR 001)
|
||||
ws-server/host.rs 4 / 0t a re-export shim only. The session loop is
|
||||
improvise-command/src/ui/host.rs (gotcha 45)
|
||||
ws-server/origin.rs 132 / 6t OriginPolicy: which origins may open a socket. Loopback by
|
||||
default, --allow-origin widens, absent allowed, two refused
|
||||
(improvise-kc7n). The only decision transport::admitted makes
|
||||
ws-server/transport.rs 330 / 4t tokio + tungstenite plumbing, plus the Origin check at the
|
||||
handshake — a refused upgrade, not an ignored socket
|
||||
ws-server/main.rs 190 / 5t CLI entry: serve a file on a port; --bind, --allow-origin
|
||||
ws-server/tests/socket.rs 389 / 6t the one test that uses a real socket: handshake, JSON
|
||||
framing, SessionId routing, which document a URL asked for
|
||||
wasm-client/lib.rs 455 / 8t Client: the wasm-bindgen surface (init, on_message, on_frame,
|
||||
viewport/set_viewport, render_model, set_cell_at, typed_cell)
|
||||
wasm-client/keys.rs 187 / 5t BrowserKey ONLY — a KeyboardEvent flattened to an improvise
|
||||
key. It does NOT resolve: `resolve` and KeyOutcome live in
|
||||
improvise-protocol/src/resolve.rs, shared by both front ends
|
||||
wasm-client/render.rs 1249 / 21t RenderModel: editable / selected / cat tree, decided here
|
||||
wasm-client/socket.rs 547 / 13t the fold (fold_frame/handle_frame), the two compositions
|
||||
and typed() text→CellValue. Defines no enums
|
||||
# state.rs is GONE — ClientState moved to improvise-protocol/src/client.rs
|
||||
wasm-client/tests/browser_round_trip.rs 2136 / 34t improvise-1ey's scope against a real Host
|
||||
wasm-client/tests/lockstep.rs 4064 / 78t host and client agree on a keypress
|
||||
web/app.js 816 string calls only, no protocol knowledge, no framework.
|
||||
Run by harness/tests/shell.spec.js (Chromium; the standalone half on Firefox too) — see
|
||||
gotcha 48 for what that does and does not reach
|
||||
web/worker.js 86 three string calls, no protocol knowledge. RUN by
|
||||
shell.spec.js's no-?ws path, with bindings.rs (gotcha 45)
|
||||
web/shell.js 399 everything app.js DECIDES that is not painting: no DOM,
|
||||
no browser API, one `node --test` away. Owns socketUrl,
|
||||
gridPlan, frameCostOf, documentTitle — so a chunk of the
|
||||
browser's render decisions is NOT in the file gotcha 42
|
||||
and gotcha 46 name. Reasoning: context/code/clients.md
|
||||
web/serial.js 19 the one rule worker.js exists to keep: one frame at a
|
||||
time, in arrival order (gotcha 48's async-listener race),
|
||||
extracted so it is testable without worker globals
|
||||
web/index.html, style.css the page; legibility only
|
||||
web/build.sh 38 wasm-pack build → web/pkg (gitignored). Needs
|
||||
wasm-opt = ["-O", "--enable-bulk-memory"] or the bundled
|
||||
wasm-opt rejects LLVM's memory.copy
|
||||
web/assemble.sh 112 assembles the deployable site (copies web/*.improv)
|
||||
web/demo.improv 65 the document the site ships and seeds from. Every cell a
|
||||
Record — asserted (gotcha 46)
|
||||
```
|
||||
|
||||
### Browser worker host (`crates/improvise-worker-server/`)
|
||||
```
|
||||
improvise-worker-server/src/worker.rs 765 / 17t Worker: one port, one client; handshake, Addressed
|
||||
filter, persist trigger (model_state.dirty). Every decision, every test
|
||||
improvise-worker-server/src/vfs.rs 348 / 8t documents/open: seeding is a Storage::write
|
||||
improvise-worker-server/src/bindings.rs 185 / 0t wasm32 shim: OpfsFs awaits + postMessage. No Rust test
|
||||
reaches it, but harness/tests/shell.spec.js DOES: the
|
||||
no-?ws path runs start()/OpfsFs::open under Chromium
|
||||
and Firefox.
|
||||
Called NEVER RUN here for months: see gotcha 45
|
||||
improvise-worker-server/src/lib.rs 51 / 1t the module doc that states exactly what is and is not
|
||||
covered — the one correct copy of that fact
|
||||
improvise-worker-server/tests/post_message.rs 407 / 8t main-thread traffic, strings only
|
||||
```
|
||||
|
||||
### Browser harness (`harness/`) — not a workspace member
|
||||
|
||||
```
|
||||
harness/playwright.config.js 50 projects: chromium (every spec), firefox (opfs, plus
|
||||
shell.spec's standalone half: grepInvert @mocked-peer),
|
||||
webkit (opfs only, which skips on a runtime probe: no
|
||||
OPFS). CI runs `--project=chromium --project=firefox` only:
|
||||
nixpkgs' WebKit aborts inside the runner VMs (ci.yml)
|
||||
harness/tests/shell.spec.js 345 the first execution of web/app.js. A change to web/*.js
|
||||
belongs here (gotcha 48). Playwright serves harness/serve/,
|
||||
which harness/build.sh ASSEMBLES: edit web/*.js, rebuild,
|
||||
or the specs run the old page
|
||||
harness/tests/opfs.spec.js 103 drives the OPFS contract page and reads back its result
|
||||
harness/opfs-contract/src/lib.rs 470 runs improvise_io::storage::contract::storage_contract
|
||||
against a real OpfsFs inside a real Web Worker
|
||||
harness/www/opfs/{index.html,worker.js} the page and worker that host it
|
||||
harness/build.sh 50 assembles harness/serve/ (web/assemble.sh, the fixtures,
|
||||
and the contract crate's wasm-pack build) — gitignored
|
||||
# gated by the `browser` job in .github/workflows/ci.yml, which can fail the build
|
||||
```
|
||||
|
||||
### Examples
|
||||
```
|
||||
examples/gen-grammar.rs Grammar-walking random .improv generator (pest_meta)
|
||||
examples/pretty-print.rs Parse stdin, print formatted .improv
|
||||
crates/improvise-core/examples/category_width_bench.rs
|
||||
The same records at 12 and at 24 regular categories
|
||||
(improvise-felo.1; context/code/core.md)
|
||||
```
|
||||
|
||||
### Context docs
|
||||
```
|
||||
.claude/skills/code-review/ Review skill: checks a change against the doctrine in these docs + kb.pl
|
||||
scripts/check-comment-runs.py The two-line comment cap AND --check-evidence. The flag is
|
||||
ADDITIVE since improvise-ijql — every invocation that can
|
||||
exit 0 answers the cap — guarded by a subprocess self-test,
|
||||
the only level a dispatch bug is visible at. Reasoning:
|
||||
context/code/tooling.md §2
|
||||
scripts/check-ci-graph.py pages-deploy must `needs` every job in ci.yml, and every
|
||||
suite `checkers.sh --list` knows must run in some CI job
|
||||
(improvise-dak5). It REFUSES what it cannot read rather than
|
||||
parsing YAML: the values it accepts are an allowlist
|
||||
(tooling.md). --self-test; run in the `test` job. README
|
||||
states the rule and lists no jobs (improvise-x7cq)
|
||||
scripts/checkers.sh The one entry point for every *-test.sh suite: scrubs the
|
||||
env to an allowlist, runs under the bash that runs it (no
|
||||
version floor since #297). `ci_suites` is the table of which
|
||||
suites each CI job runs (`--ci <job>`); ci.yml names none
|
||||
scripts/pr-watch Waits on a PR's checks (the pr-watch skill). Exit 5: the PR
|
||||
is already MERGED or CLOSED, so there is nothing to wait for
|
||||
scripts/assemble-stamp.sh The cache-bust stamp gate web/assemble.sh calls, extracted
|
||||
so its refusals can be exercised without a web/pkg
|
||||
scripts/assemble-stamp-test.sh Its check_the_checker, run first in the `build` CI job
|
||||
context/code/tooling.md The repository's own checkers: why an evidence heading's
|
||||
names come from cargo rather than from the source, and
|
||||
why a leaf name is not an identity
|
||||
context/test-evidence/tooling.md Evidence for the gates in scripts/ and for the tests that
|
||||
hold CI's shape
|
||||
scripts/nix-seed-flake-inputs.sh Seeds locked flake inputs over git so `nix develop` works in cloud sessions
|
||||
prolog-sessions/kb.pl Executable knowledge base: invariants, doctrine, decisions (CI-checked)
|
||||
context/adr/007-stable-identities.md Stable identities: ids unique across the document are the
|
||||
keys and names are display-only; gensym suffixes apply to
|
||||
generated ids only, and hand-written ids are canonical.
|
||||
Accepted 2026-09-25. (There is no ADR 005 or 006 file:
|
||||
improvise-l2b8 and improvise-ilvl claim those numbers.)
|
||||
context/adr/004-faces-and-themes.md Faces: appearance as UI-agnostic data a front end
|
||||
applies, emacs-style. Accepted 2026-09-22.
|
||||
context/adr/003-live-backed-models.md A model may be backed by a live query source —
|
||||
improvise as a DB client. Accepted, NOT a priority.
|
||||
context/adr/002-naming-a-cell.md How a client names a cell: an opaque surrogate on
|
||||
CellDisplay, whose absence means the cell is an
|
||||
aggregate. Records why _Index cannot serve.
|
||||
context/adr/001-concurrent-edits.md Concurrent edits between clients: record-level atomicity,
|
||||
optimistic basis check per batch, advisory reservations.
|
||||
Records why a causal clock was rejected.
|
||||
context/cloud-agents.md Cloud-session environment: tracker sync, bd pitfalls, what is blocked
|
||||
context/design-principles.md Architectural principles & testing doctrine
|
||||
context/plan.md Show HN launch plan
|
||||
context/repo-map.md The map: crates, how-to-find-things, the gotcha digest, a file map
|
||||
context/gotchas.md The long form of every numbered gotcha (the digest is in repo-map.md)
|
||||
context/file-inventory.md This file: the per-file listing the repo map summarises
|
||||
docs/design-notes.md Product vision & non-goals
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
# Wasm, CLI, testing and dependencies (long form)
|
||||
|
||||
The three repo-map sections of these names, as they stood before the map was condensed.
|
||||
|
||||
## Wasm
|
||||
|
||||
The browser bundle is `improvise-core`, `improvise-formula`, `improvise-io`,
|
||||
`improvise-protocol`, `improvise-command`, `improvise-wasm-client` and
|
||||
`improvise-worker-server`. All seven
|
||||
`cargo check --target wasm32-unknown-unknown` clean with no feature flags
|
||||
(improvise-ywd), gated by the `wasm` job in `.github/workflows/ci.yml`. That
|
||||
job gates **eight** crates, not seven: `improvise-gp` was added (improvise-z5eg)
|
||||
explicitly *not* as a bundle member but as a candidate, because staying
|
||||
wasm-clean is free now and expensive to retrofit — so the bundle is seven and
|
||||
the gate is eight, and neither number is the other one's typo.
|
||||
**A feature that adds a dependency to a bundle crate must be checked with `cargo tree --edges normal`** — the default output includes **dev** edges, so a plain `cargo tree -p improvise-protocol --target wasm32-unknown-unknown` reports a dev-only `serde_json` as *present* and reads exactly like a leak. Check per-crate too, because the `wasm` CI job invokes each crate separately and feature unification cannot reach across that (improvise-26dm added a `test-support` feature on `improvise-protocol` on those terms; it is off in every shipped build).
|
||||
|
||||
`improvise-ws-server` is the host and is native only — tokio, tungstenite and
|
||||
`Box<dyn Storage>` — so `improvise-wasm-client` cannot depend on it in anger. It
|
||||
no longer needs to: both ends `pub use` `improvise_protocol::frame` and neither
|
||||
defines a pair (gotcha 42). The **dev**-dependency remains, for
|
||||
`tests/browser_round_trip.rs`, which hands the client's own serialized frame to
|
||||
a real `Host::handle`. The root `improvise` crate is
|
||||
the terminal binary and does **not** build for wasm — it depends on
|
||||
crossterm; that is correct, not a gap.
|
||||
|
||||
`cargo check` passing is not the same as running. Two known runtime hazards
|
||||
it cannot see: `Instant::now()` panics on wasm (hence `web-time` in
|
||||
`improvise-command`), and any `std::fs` call compiles but fails in a browser
|
||||
— improvise-fuj closed the effects that bypassed `Storage`; autosave and the headless final write still do (improvise-31oe.11).
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
improvise [model.improv] # open TUI (default)
|
||||
improvise import data.csv [--no-wizard] [-o out] # import CSV/JSON
|
||||
improvise cmd 'add-cat Region' -f model.improv # headless command(s)
|
||||
improvise script setup.txt -f model.improv # run script file
|
||||
```
|
||||
|
||||
Import flags: `--category`, `--measure`, `--time`, `--skip`, `--extract`, `--axis`, `--formula`, `--name`, `--split-date <field>` (a date column imports as ONE hierarchical category by default; this restores the flat split, and `--extract` still picks the components — improvise-8jk6.3), `--strict` (with `--no-wizard`, any skipped record is a refusal and nothing is written, where the default writes and warns with a count — improvise-8jk6.6), and `--on-duplicate refuse|sum|separate` (default `refuse`; applies with `--no-wizard`, where the wizard would otherwise ask — improvise-cgqt). A headless import also refuses a `--split-date` that names a non-date field and an `--axis` that names a category the import did not build (exit 1).
|
||||
|
||||
---
|
||||
|
||||
## Testing — the short version
|
||||
|
||||
Full guidance lives in `context/design-principles.md` §7. Quick reminders:
|
||||
|
||||
- **Two numbers, not one** — the ambiguity between them is what let the old
|
||||
"suite runs in <2s" drift: **no single test binary over 2 seconds**, and the
|
||||
workspace wall clock is the sum. Measured at `63a5c13`: **1464 tests over 35
|
||||
binaries** (1 ignored), longest single binary **0.6–1.0s across two runs on one machine** —
|
||||
the spread being its own argument for reading your own run. Excludes compilation,
|
||||
and excludes `harness/` (Playwright, minutes, its own CI job). *Re-measure* rather
|
||||
than re-assert — `cargo test --workspace` prints a `finished in` per binary, and
|
||||
**the portable signal is the ratio, not the seconds**. **Not the ratio to the
|
||||
median, though: it is degenerate.** Most binaries print `0.00s` at two decimals,
|
||||
so the median is zero and every ratio to it is undefined (improvise-i5z6, 39
|
||||
binaries). Compare the slowest binary to the next few slowest instead: an outlier
|
||||
is one that sits well clear of them *in every run*. That is how i5z6 was read —
|
||||
`improvise-formula`'s lib longest of 39 in all three runs, at 0.91–1.95s against a
|
||||
runner-up's 0.50–0.92s. Both outliers so far had one cause: a generator re-parsing
|
||||
its pest grammar per case. `improvise-io` (improvise-8shu, 1.42s → 0.21s) and
|
||||
`improvise-formula` (improvise-i5z6, 0.84–0.98s → 0.06–0.07s) each read it once through a
|
||||
`OnceLock` now. A new grammar-walking strategy should do the same from the start.
|
||||
The full argument is `context/design-principles.md` §7.
|
||||
- Commands: build a `CmdContext`, call `execute`, assert on returned effects. No terminal needed.
|
||||
- Property tests (`proptest`, default 256 cases) cover invariants: CellKey sort, axis consistency, save/load roundtrip, `parse(format(parse(generate())))` stability.
|
||||
- **Bug-fix workflow**: write a failing test *before* the fix (regression guard). Record the bug in `context/test-evidence/`, **keyed by test name** — *not* in the test's doc comment, which the two-line comment cap forbids and `scripts/check-comment-runs.py` enforces. A heading may be module-qualified and must be when its leaf is not unique.
|
||||
- Coverage target ~80% line/branch on logic code; skip ratatui render paths.
|
||||
|
||||
---
|
||||
|
||||
## Key dependencies
|
||||
|
||||
ratatui 0.30, crossterm 0.29, clap 4.6 (derive), serde/serde_json, indexmap 2, anyhow, chrono 0.4, pest + pest_derive, flate2 (gzip), csv, enum_dispatch. Dev: proptest, tempfile, pest_meta.
|
||||
File diff suppressed because one or more lines are too long
+93
-524
File diff suppressed because one or more lines are too long
@@ -1935,14 +1935,14 @@ Rival: every space becomes `_`, with runs not collapsed. Only this test fails:
|
||||
|
||||
### model::id::tests::slugify_keeps_an_underscore_the_display_already_had
|
||||
|
||||
Rival: `.replace("__", "_")` after the space step, which reads answer 21 as
|
||||
"no doubled underscores". Only this test fails: `left: "a_b"`. The rule
|
||||
Rival: `.replace("__", "_")` after the space step, which reads the accepted
|
||||
clause 4 as "no doubled underscores". Only this test fails: `left: "a_b"`. The rule
|
||||
collapses runs of *spaces*. An underscore that was in the display is kept.
|
||||
|
||||
### model::id::tests::the_suffix_is_the_next_counter_value_not_the_lowest_free_number
|
||||
|
||||
Rival: the lowest free `<base>-<n>` for each base, which revision 5 of the ADR
|
||||
proposed and answer 28 replaced. Three tests fail: this one (`"west-1"` against
|
||||
Rival: the lowest free `<base>-<n>` for each base, which an early draft of
|
||||
ADR 007 proposed and clause 4's shared `gensym` counter replaced. Three tests fail: this one (`"west-1"` against
|
||||
`"west-2"`), `an_empty_slug_takes_its_kind_and_the_shared_counter` (`"item-1"`)
|
||||
and `a_slug_taken_by_another_kind_takes_a_suffix` (`"time_period-1"`). Only
|
||||
interleaving two bases separates one shared counter from per-base numbering,
|
||||
@@ -1952,8 +1952,9 @@ and this test is the one that does it on purpose.
|
||||
|
||||
Rival: `claim` validates the id but does not record it. This test and
|
||||
`claim_refuses_a_duplicate_a_malformed_and_a_reserved_id` fail. The second
|
||||
`East` is issued `east-1`, the file's own id, which is the collision answer 29
|
||||
forbids: "doing otherwise will corrupt the data section".
|
||||
`East` is issued `east-1`, the file's own id, which is the collision clause 4
|
||||
forbids: hand-written ids are canonical, and renaming one would corrupt the
|
||||
data section.
|
||||
|
||||
### model::id::tests::generated_ids_are_distinct_well_formed_and_never_reserved
|
||||
|
||||
@@ -1990,7 +1991,7 @@ exempt any id that begins with a virtual id (`VIRTUAL_IDS.iter().any(|v|
|
||||
id.starts_with(v))`), all 11 earlier tests passed, because
|
||||
`claim_refuses_a_duplicate_a_malformed_and_a_reserved_id` tries only `_draft`.
|
||||
This test fails on that rival (`left: Ok(())` for `_indexer`). The match is
|
||||
exact by the ADR's own wording: answer 23 says no other id may begin with `_`.
|
||||
exact by the ADR's own wording: clause 1 says no other id may begin with `_`.
|
||||
|
||||
### model::id::tests::a_display_with_leading_spaces_never_slugs_into_the_reserved_space
|
||||
|
||||
|
||||
@@ -0,0 +1,468 @@
|
||||
# Test evidence: `numtower`
|
||||
|
||||
What each test guards, which rival it was run against, and where that rival failed. The
|
||||
entries are keyed by exact test name, and a rename orphans its entry, which
|
||||
`scripts/check-comment-runs.py --check-evidence` catches. The reasoning these tests defend
|
||||
is in `context/code/numtower.md`. The decisions are ADR 005's.
|
||||
|
||||
**How the rivals are kept.** Every property in `crates/numtower/tests/properties.rs` is a
|
||||
checker that takes the implementation as a parameter. The property's own test runs it on
|
||||
the real code. A companion test, named `<property>_catches_<rival>`, runs the same checker
|
||||
on the rival, with a deterministic seed, and asserts two things:
|
||||
- the run fails;
|
||||
- the failure message carries **the property's own assertion text**, so a rival that
|
||||
fails in setup, panics, or fails some other assertion does not count (`assert_caught`
|
||||
in `tests/oracle/mod.rs`).
|
||||
|
||||
So the rival is watched failing on every run of the suite, not once by hand. Two rivals
|
||||
were also injected into the crate's source, and the run was recorded. Those runs are noted
|
||||
under their entries.
|
||||
|
||||
**The oracle** is `num-rational`'s `BigRational`, a dev-dependency feature. The crate
|
||||
itself never uses `BigRational`, so the oracle shares no rational type with the code under
|
||||
test. "Correctly rounded" is checked in the review's interval form (`is_nearest`). The
|
||||
exact value must lie inside the round-to-nearest-even interval of the returned f64, with
|
||||
the endpoints included only when the f64's mantissa is even. Comparing against
|
||||
`BigRational::to_f64` would have been circular in the way ADR 005's option (d) section
|
||||
describes.
|
||||
|
||||
---
|
||||
|
||||
## `crates/numtower/tests/properties.rs`
|
||||
|
||||
### an_exact_result_is_never_a_rounded_one
|
||||
|
||||
Every exact pairing of arms and every operation (`+ − × ÷`) is checked against the oracle.
|
||||
When the result claims to be exact, its value must be the exact value. The generator
|
||||
spans 96-bit mantissas, scales 0–28 and `i128` fractions. A fixed case pins ADR 005's own
|
||||
example, `1000 + 0.3333333333333333333333333333`. `rust_decimal` returns that at scale
|
||||
25, rounded, and the crate must not.
|
||||
|
||||
**Injected in the source.** In `arith.rs::decimal_sum`, the scale filter was deleted, so
|
||||
`checked_add`'s result was trusted. Three tests failed: this one,
|
||||
`contagion_is_total`, and `a_recomputed_result_keeps_the_stored_scale_clause_4_names`.
|
||||
The revert was followed by a full rerun.
|
||||
|
||||
### an_exact_result_is_never_a_rounded_one_catches_a_silently_rounding_add
|
||||
|
||||
The rival is `rust_decimal`'s own `checked_add`, trusted as exact. That is the one
|
||||
outcome ADR 005 clause 4 exists to prevent.
|
||||
|
||||
### contagion_is_total
|
||||
|
||||
Every pairing of the three arms, NaN and the infinities included, through `+ − × ÷`:
|
||||
- an inexact operand makes the result inexact;
|
||||
- an exact pair gives an exact result **iff** the oracle value fits a stored arm, and
|
||||
otherwise the correctly rounded f64 of that value;
|
||||
- division refuses iff the divisor is exactly zero, at either level.
|
||||
|
||||
"Total" means no pairing is missing. A pairing the code never handled would panic, and the
|
||||
runner reports a panic as a failure.
|
||||
|
||||
### contagion_is_total_catches_an_integral_inexact_kept_exact
|
||||
|
||||
The rival is a "harmless" promotion: an integral f64 is treated as exact, so `2 + 3.0` is
|
||||
exact. It is caught on the seed pair `(2, Inexact(3.0))`. That pair is in the generator on
|
||||
purpose. Integral f64s are rare among random floats, so without the seed the rival was
|
||||
caught only by luck.
|
||||
|
||||
### pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits
|
||||
|
||||
Bases from every arm, exponents −6 to 6. An exact base gives an exact result iff the
|
||||
oracle's power fits, and otherwise the correctly rounded value. `0^-n` refuses.
|
||||
|
||||
### pow_decides_its_size_before_computing_and_honours_the_ceiling
|
||||
|
||||
- `POW(1/3, 10^9)` returns at once, inexact. Its lower bound is past the ceiling, so
|
||||
nothing is allocated. Without the size check the test does not finish.
|
||||
- `(1/3)^6000` is computed, but is 9,510 bits and demotes. That is the ceiling's
|
||||
exception.
|
||||
- `1.05^28` is exact and `1.05^29` is not, the option (d) table's k = 29.
|
||||
- Also pinned: `(-1.0)^(10^20)`, `0^0`, the scale of `1.50^2`, and that a non-integer
|
||||
exponent is `powf`.
|
||||
|
||||
### the_f64_conversion_agrees_with_parsing_the_decimal_text
|
||||
|
||||
The generator spans **96-bit mantissas and scales 0–28**, as ADR 005 requires. The oracle
|
||||
text is built digit by digit, not by `rust_decimal`'s `Display`, which is what the crate
|
||||
itself calls. So the test does not compare the crate's path with itself. The review's
|
||||
case `63294.607176721429540940386554` is pinned.
|
||||
|
||||
**Injected in the source.** `num.rs::decimal_to_f64` was replaced with `rust_decimal`'s
|
||||
`ToPrimitive::to_f64`, and three tests failed:
|
||||
- this one, minimal case `-0.10806611601966961` at scale 17;
|
||||
- `the_number_languages_gate_and_parse_agree`, on `92240.01133710762224`;
|
||||
- `exact_mixed_comparison_is_transitive`.
|
||||
|
||||
The third was not expected. The comparison's fast path decides by the rounded value, and
|
||||
that is sound only if the rounding is correct.
|
||||
|
||||
### the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_to_f64
|
||||
|
||||
The rival is `rust_decimal`'s `to_f64`, which is not correctly rounded (ADR 005, B2). The
|
||||
test also asserts that the rival misses on the review's case, so the pinned case is known
|
||||
to discriminate.
|
||||
|
||||
### rust_decimal_to_f64_agrees_on_fifteen_digit_mantissas_so_a_narrow_generator_is_vacuous
|
||||
|
||||
This test **passes the rival** on purpose. With mantissas of at most 15 digits,
|
||||
`rust_decimal`'s `to_f64` agrees with the text every time, at this test's 256 cases as
|
||||
in ADR 005's 200,000 of 200,000. So a test whose generator stays in that range proves
|
||||
nothing about the conversion, and this test is why the generator above spans 96 bits. It
|
||||
ran at 4,096 cases until Copilot pointed out, on PR #337, that the repository's budget is
|
||||
256 (`context/design-principles.md` §7). The 200,000 is the ADR's measurement, and this
|
||||
test only pins its direction.
|
||||
|
||||
If `rust_decimal` ever fixes its conversion, this test still passes. What fails then is
|
||||
`the_f64_conversion_agrees_with_parsing_the_decimal_text_catches_rust_decimal_to_f64`,
|
||||
whose rival would agree. That is the signal to re-read both. An earlier version of this
|
||||
entry named this test instead, which was wrong.
|
||||
|
||||
### a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval
|
||||
|
||||
Reduced fractions with numerator and denominator widths from 20 to 126 bits, both signs:
|
||||
the review's independent form. It also checks the product of two such fractions, which
|
||||
reaches the bigint path and is stored. That covers `ratio_to_f64` on values wider than
|
||||
`i128`.
|
||||
|
||||
### a_fractions_conversion_lies_inside_the_round_to_nearest_even_interval_catches_an_f64_division
|
||||
|
||||
The rival is `n as f64 / d as f64`, which rounds three times.
|
||||
|
||||
### exact_mixed_comparison_is_transitive
|
||||
|
||||
The generator is a **crowd**: values crowded around one f64 `x`.
|
||||
- `x` itself, and its two neighbours;
|
||||
- the shortest decimal that spells `x`, and that decimal ± `k·10^-25`;
|
||||
- a third of the shortest decimal.
|
||||
|
||||
Every triple is checked for transitivity, and every pair for antisymmetry. Random values
|
||||
would almost never put two exacts and an inexact on one f64, which is exactly where the
|
||||
rival breaks.
|
||||
|
||||
### exact_mixed_comparison_is_transitive_catches_comparing_through_f64
|
||||
|
||||
The rival is revision 1's rule: convert the exact operand to f64, then compare. It fails
|
||||
as ADR 005 clause 6 says it does. `Exact(s) = Inexact(x)` and `Inexact(x) = Exact(s +
|
||||
10^-25)`, while the two exacts differ. The marker is `comparison: not transitive`, so it
|
||||
is the transitivity assertion that fails, not antisymmetry.
|
||||
|
||||
### comparison_agrees_with_the_oracle_at_every_pairing_of_arms
|
||||
|
||||
The differential half of comparison: the crowd, plus mixed random pools of decimals,
|
||||
fractions and finite f64s, against `BigRational`'s order. NaN is unordered, and an
|
||||
infinity orders past every exact value.
|
||||
|
||||
### min_and_max_prefer_the_exact_value_and_do_not_depend_on_order
|
||||
|
||||
Question 11's tie-break, plus the two the crate adds: the larger scale between equal
|
||||
decimals, and IEEE's `-0.0` for `MIN` and `0.0` for `MAX`. Each pair is asserted in both
|
||||
argument orders, the order-independence property the tie-breaks exist for.
|
||||
|
||||
### the_accumulator_is_independent_of_order
|
||||
|
||||
Random **weighted** term lists, from every arm, are summed in one order and in a shuffled
|
||||
order. The results must agree in **arm, value and scale**. The terms include fractions
|
||||
over a pool of fourteen non-decimal denominators, so groups receive several terms and
|
||||
lists often hold more than K of them, and arbitrary fractions too. ADR 005 clause 8's
|
||||
seed case, `[0.30000000000000004, 1e12, −1e12]`, is one of the generator's arms and is
|
||||
also pinned exactly. It caught the review's F12 (a group's old f64 not retracted), where
|
||||
the stale values make the past-K read depend on the order of updates.
|
||||
|
||||
**On weights** (the review of `302594bf`, B2). Until then every `NumSum` property called
|
||||
`insert` and `retract` only, so every weight was ±1. That was the cause the answer to
|
||||
Codex's P1 had itself named, and the generators had not changed. Two faults passed the
|
||||
whole suite: F7 (the decimal partition ignored the weight's magnitude) and F8 (a group
|
||||
multiplied by `weight.signum()`). Weights now come from `weight()`:
|
||||
- the units 1, −1 and 0;
|
||||
- −3 to 3;
|
||||
- magnitudes up to 2^40;
|
||||
- `±i64::MAX` and 2^30 − 1.
|
||||
|
||||
`i64::MIN` is left out so a weight always negates.
|
||||
|
||||
### the_accumulator_is_independent_of_order_catches_decimal_addition
|
||||
|
||||
The rival is clause 8's own rival, a fold with `Decimal`'s `+`, demoting to the tower
|
||||
where it overflows. On the seed case, one order rounds `0.30000000000000004 + 1e12` to
|
||||
scale 16 and the other does not.
|
||||
|
||||
### the_accumulator_retracts_exactly
|
||||
|
||||
Adding weighted `xs` and `ys`, then adding each `y` again with its weight negated in
|
||||
reverse order, must equal a fresh sum of `xs` in arm, value and scale. A group whose
|
||||
numerator returns to zero must be dropped, or a retracted denominator would still count
|
||||
toward K. With weights it catches F7 and F12 as well.
|
||||
|
||||
### the_accumulator_reads_the_exact_sum_or_its_stated_roundings
|
||||
|
||||
The differential test of the read against a `BigRational` oracle, over weighted terms. It
|
||||
states the value of all three branches:
|
||||
- **No inexact term, at most K live groups.** The read is exact iff the exact total fits,
|
||||
and is then that total. Otherwise it is the total's correctly rounded f64.
|
||||
- **Inexact terms present.** The exact total is rounded once, then the weighted f64s are
|
||||
added and the result is rounded once more.
|
||||
- **Past K groups.** The nearest f64 of Σ round(live group) + round(decimal partition) +
|
||||
Σ w·f.
|
||||
|
||||
**Until the review of `302594bf` (B1) the past-K branch asserted only that the read was
|
||||
inexact, and this entry claimed the test stated every branch. Both were wrong.** Three
|
||||
faults passed the whole suite:
|
||||
- F12: a group's old f64 was not retracted from `group_floats`;
|
||||
- F14: the past-K read dropped the decimal partition;
|
||||
- F15: the past-K read dropped every inexact term.
|
||||
|
||||
The test now has the reviewer's oracle, at the repository's 256-case budget. 34ab0fc3 ran
|
||||
it at 512, and Copilot pointed out the budget. At 256, every fault in the table below
|
||||
still failed by name, and F7, F8, F12, F14 and F15 failed in four of four passes. Half the lists come from the
|
||||
fourteen-denominator pool, up to 40 terms long, so groups receive several terms and K is
|
||||
passed often. **It also counts how many cases reach each branch and fails if any branch
|
||||
is reached fewer than 20 times**, so a generator change cannot silently starve one.
|
||||
|
||||
The oracle counts **live** groups, those whose summed numerator is nonzero. The first
|
||||
version counted denominators ever seen, and that disagrees with the state-based count
|
||||
whenever terms cancel. Inexact presence is likewise per bit pattern, by net weight.
|
||||
|
||||
**Every fault in the review's script, re-run after the fix.** Each was injected into the
|
||||
source and `cargo test -p numtower` was run on a confirmed rebuild. Each failed by name:
|
||||
|
||||
| fault | failed |
|
||||
|---|---|
|
||||
| F1 decimal sum's scale filter dropped | `a_recomputed_result_keeps_the_stored_scale_clause_4_names`, `an_exact_result_is_never_a_rounded_one`, `contagion_is_total` |
|
||||
| F2 `rust_decimal`'s `to_f64` | `exact_mixed_comparison_is_transitive`, `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `the_f64_conversion_agrees_with_parsing_the_decimal_text`, `the_number_languages_gate_and_parse_agree` |
|
||||
| F3 mixed comparison through f64 | `exact_mixed_comparison_is_transitive`, `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `an_interim_compares_exactly_without_storing` |
|
||||
| F4 one net inexact count | `the_accumulator_keeps_inexact_terms_whose_weights_cancel`, `the_accumulator_retracts_exactly`, this test |
|
||||
| F5 product, not lcm | `sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once` |
|
||||
| F6 `ExactSum` rounded per limb | `exact_sum_is_the_exact_weighted_total_rounded_once`, `more_than_k_denominators_read_inexact_even_when_the_total_fits`, `the_accumulator_keeps_inexact_terms_whose_weights_cancel`, this test |
|
||||
| F7 decimal weight magnitude ignored | this test, `the_accumulator_retracts_exactly` |
|
||||
| F8 group weight as its sign | this test |
|
||||
| F9 decimal product's scale filter dropped | `a_typed_value_and_the_same_value_computed_are_equally_exact`, `an_exact_result_is_never_a_rounded_one`, `contagion_is_total` |
|
||||
| F10 `ExactSum`'s low weight digit only | `exact_sum_is_the_exact_weighted_total_rounded_once`, this test |
|
||||
| F11 `POW` sign lost | `a_huge_odd_exponent_keeps_the_sign_of_a_negative_base` |
|
||||
| F12 old group f64 kept | this test, `the_accumulator_is_independent_of_order`, `the_accumulator_retracts_exactly` |
|
||||
| F13 mixed tie not checked exactly | `comparison_agrees_with_the_oracle_at_every_pairing_of_arms`, `exact_mixed_comparison_is_transitive`, `an_interim_compares_exactly_without_storing` |
|
||||
| F14 past K, decimal partition dropped | this test |
|
||||
| F15 past K, inexact terms dropped | this test |
|
||||
|
||||
- **F13's text now occurs twice.** `Interim`'s comparison added a second copy of the
|
||||
replaced line, so the review's script refuses to inject it. It was injected at its
|
||||
original site, in `cmp.rs::mixed`.
|
||||
- **The five former survivors were run twice and failed both times.** F12's set varied
|
||||
between runs: `the_accumulator_retracts_exactly` failed in one run and not the other,
|
||||
and this test failed in both.
|
||||
|
||||
### more_than_k_denominators_read_inexact_even_when_the_total_fits
|
||||
|
||||
`1/3 + 1/7 + … + 1/31` is nine distinct denominators. The exact total fits an `i128`
|
||||
fraction, yet the read is inexact. That is question 13's second exception, the K cutoff
|
||||
overriding representability. Eight of them read as an exact `Fraction`. The read past K is
|
||||
asserted to be **each group rounded, then the total rounded once**. The first version
|
||||
asserted one rounding of the exact total, and it failed: clause 8's maintained f64
|
||||
accumulator of group values rounds twice by design.
|
||||
|
||||
### exact_sum_is_the_exact_weighted_total_rounded_once
|
||||
|
||||
ADR 009's `ExactSum` against the oracle. Weights are drawn from small values, from the
|
||||
whole `i64` range and up to 2^40. A whole-range weight exercises all three base-2^30
|
||||
digits, which ADR 009's chunk loop could not reach in reasonable time. Values include
|
||||
subnormals.
|
||||
|
||||
### division_decides_fit_directly
|
||||
|
||||
ADR 005 clause 5's three seed cases, as both property inputs and pinned assertions:
|
||||
- `1 / 2^40` and `1 / 5^30` are fractions;
|
||||
- `1e-28 / 0.5` is the exact decimal `2e-28`.
|
||||
|
||||
Random decimal pairs must also be an exact decimal iff the quotient fits one, an exact
|
||||
fraction iff it fits only that, and otherwise inexact. The test also pins `AVG`'s example
|
||||
(`70/3`), `1/3 × 3 = 1`, and division by an inexact `-0.0`.
|
||||
|
||||
### division_decides_fit_directly_catches_multiplying_back
|
||||
|
||||
The rival is multiplying back with **revision 6's** scale test (demote on the hint). It
|
||||
fails on `1e-28 / 0.5`, as ADR 005 says. An experiment, not committed, ran multiplying
|
||||
back with `Num`'s own `*`, which carries revision 7's recompute. It agreed with the direct
|
||||
test on all 5,120 random cases and the seeds. The ADR's argument that multiplying back is
|
||||
incomplete holds only under the earlier rule (`context/code/numtower.md`, "Division").
|
||||
|
||||
### the_number_languages_gate_and_parse_agree
|
||||
|
||||
The generated lexemes have four sources:
|
||||
- structured lexemes: an optional sign, up to 40 digits, an optional fraction, and an
|
||||
optional exponent of up to 4 digits, so the empty parts give `1.`, `.5` and `.`;
|
||||
- junk over the alphabet of digits, `eE+-._/aifnIN` and space;
|
||||
- `n/d` lexemes;
|
||||
- the named spellings: `inf`, `NaN`, `Infinity`, `1_000`, `+.5`, `e5`.
|
||||
|
||||
For each lexeme, the parse succeeds iff `f64::from_str` reads a finite value. The exact
|
||||
value parsed must round to the f64 `f64::from_str` reads. `is_number` and `parse_number`
|
||||
must agree in both typings.
|
||||
|
||||
### the_number_languages_gate_and_parse_agree_catches_from_str_exact
|
||||
|
||||
The rival is `rust_decimal`'s `from_str_exact`. ADR 005 clause 3 names it as the parse
|
||||
whose language differs from the gate's: it rejects `1.5e3` and accepts `1_000`.
|
||||
|
||||
### the_number_language_seed_cases
|
||||
|
||||
Every seed case ADR 005's migration step 1 lists:
|
||||
- `1.5e3` is the exact `1500`;
|
||||
- `1_000` is text;
|
||||
- `-0.0` is the exact `0.0`, with no sign;
|
||||
- `3/4` is text when the type is inferred, and `0.75` when it is forced;
|
||||
- `1e999`, `-1e999`, `nan`, `inf` and `Infinity` are text;
|
||||
- `1e-999` is an inexact zero;
|
||||
- `1e-30` is the exact `1/10^30`;
|
||||
- `0.12345678901234567890123456789` is an exact fraction, not a rounded decimal;
|
||||
- `2^120` is an exact `n/1`;
|
||||
- `1e-40` is inexact.
|
||||
|
||||
### a_typed_value_and_the_same_value_computed_are_equally_exact
|
||||
|
||||
For random decimal pairs (and the seed `1e-15 × 1e-15`), the exact product is spelled as
|
||||
`<integer>e-<k>` and parsed. The parse must have the arm and the value of the computed
|
||||
product. An earlier draft built the text only when the reduced denominator was a power of
|
||||
ten, and rejected most cases. A reduced decimal product's denominator is `2^a·5^b`, so the
|
||||
text now scales by the smallest power of ten that makes the product an integer.
|
||||
|
||||
### a_typed_value_and_the_same_value_computed_are_equally_exact_catches_demoting_on_the_hint
|
||||
|
||||
The rival is revision 6's multiplication, which demotes whenever the returned scale drops.
|
||||
`1e-15 × 1e-15` then becomes an f64, while the typed `1e-30` is the exact fraction. That
|
||||
is the asymmetry revision 6's N2 was about.
|
||||
|
||||
### a_recomputed_result_keeps_the_stored_scale_clause_4_names
|
||||
|
||||
ADR 005 clause 4's example: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28,
|
||||
not scale 0 and not 32. The test also pins that a common-path sum keeps its returned scale
|
||||
(`12.50 + 1.5 = 14.00`), that `1000 + 0.333…` goes to the fraction arm, and that an exact
|
||||
zero result has no sign.
|
||||
|
||||
### an_intermediate_beyond_i128_comes_back_exact
|
||||
|
||||
The point of question 13 (b). `a = 10^30/7`; `a × a` does not fit `i128`, yet `(a × a) /
|
||||
a` through `Interim` is `a`, exactly. The same product stored through `Num` is inexact
|
||||
and correctly rounded.
|
||||
|
||||
### the_canonical_text_reads_back_as_the_same_number
|
||||
|
||||
Every arm's `Display` text reads back through `from_canonical` as the same arm, value and
|
||||
scale. The test pins the four spellings ADR 005 clause 11 tabulates, and that canonical
|
||||
exact text never reads back as an inexact value.
|
||||
|
||||
### the_accumulator_keeps_inexact_terms_whose_weights_cancel
|
||||
|
||||
ChatGPT Codex, P1 on PR #337, against `edbb50c4`. `NumSum` decided "is an inexact term
|
||||
present" from the **net** inexact weight. Adding `1.0` with weight 1 and `2.0` with weight
|
||||
−1 nets to zero while the accumulator holds −1.0. So `read()` took the exact branch and
|
||||
returned the exact partition alone, dropping the inexact term silently.
|
||||
|
||||
This test **failed at `edbb50c4`**, on its first assertion: the read of `[1.0 ×1, 2.0 ×−1,
|
||||
0.5]` claimed to be exact. Two things made the bug invisible to the existing properties:
|
||||
- every generated term was inserted with weight +1;
|
||||
- the retraction test retracts the very values it inserted, so per-value presence and net
|
||||
presence agree there by construction.
|
||||
|
||||
The test also asserts that retracting both inexact terms returns the exact `0.5`, so the
|
||||
fix cannot simply make every sum that ever saw an f64 inexact.
|
||||
|
||||
### a_huge_odd_exponent_keeps_the_sign_of_a_negative_base
|
||||
|
||||
Copilot on PR #337, against `edbb50c4`. When the size check demotes a power to `powf`, the
|
||||
exponent becomes an f64. Above 2^53 it rounds to an even value, so `(-2)^(2^53 + 1)`
|
||||
returned `+inf` where the sign must be negative, and the negative exponent gave `+0.0`.
|
||||
This test **failed at `edbb50c4`** on its first assertion. It also checks a fraction base,
|
||||
and that an even exponent stays positive, so a fix that always negates cannot pass.
|
||||
|
||||
### i128_min_is_an_exact_fraction_numerator
|
||||
|
||||
Copilot on PR #337, against `edbb50c4`. `Num::fit` rejected `i128::MIN` as a numerator
|
||||
because `to_i128` filtered it for both halves, so `Num::ratio(i128::MIN, 1)` became inexact
|
||||
although `Ratio<i128>` holds it. This test **failed at `edbb50c4`** on its first
|
||||
assertion. The oracle's `fits` had the same symmetric-range assumption, so no property
|
||||
could see the bug. Both are fixed, and `fraction()` now generates `i128::MIN / d`.
|
||||
|
||||
Allowing the value exposes a second hazard: it cannot be negated in `i128`, and the old
|
||||
`negate` computed `-n`, which panics in a debug build. So the test also negates
|
||||
`i128::MIN / 3` (an inexact, correctly rounded result), takes it through `Interim` twice
|
||||
and back, divides it, compares it and reads its canonical text.
|
||||
|
||||
### a_negative_power_has_its_minimal_scale
|
||||
|
||||
Copilot on PR #337, and B3 of the review of `302594bf`. The unit-base arm of `POW`
|
||||
padded every result to `s·|n|`, so `1.0^-2` was `1.00` and `(-1.0)^-3` was `-1.000`,
|
||||
while `2.0^-2` was `0.25`. `context/code/numtower.md` already said "a negative power …
|
||||
has its minimal scale", so the code contradicted the crate's stated rule. This test
|
||||
**failed at `302594bf`** on its first assertion. It also pins that a positive power keeps
|
||||
`s·n` (`(-1.0)^3 = -1.000`), so a fix that stripped every scale cannot pass.
|
||||
|
||||
The broader guard is the scale assertion now in
|
||||
`pow_is_exact_for_an_exact_base_and_integer_exponent_when_it_fits`, which had checked
|
||||
value and exactness but never scale. Its oracle, `power_scale`, is the rule restated:
|
||||
- start from the minimal scale;
|
||||
- pad toward `min(s·k, 28)` for a decimal base with `k > 0` only, while the mantissa
|
||||
fits 96 bits.
|
||||
|
||||
Unit bases (`1.0`, `-1.0`, `1.000`, `-1`, `0.00`) are among its seeds. It **failed at
|
||||
`302594bf`** with the minimal case `1.0^-1 = 1.0`. The injection G1 (a negative power
|
||||
keeps the base's scale) fails both tests.
|
||||
|
||||
### exact_sums_are_equal_when_their_values_are
|
||||
|
||||
Copilot on PR #337, and B3 of the review. `ExactSum` derived `PartialEq` over its raw
|
||||
limbs and its `pending` carry counter. This test **failed at `302594bf`** on its first
|
||||
assertion, `0.5 + 0.5 == 1.0`. It also asserts:
|
||||
- insert then retract equals an empty sum;
|
||||
- weights `(2^30 − 1, 2)` added in either order are equal;
|
||||
- different values are unequal, and a counted NaN is part of the value, so an `==` that
|
||||
always answers true cannot pass.
|
||||
|
||||
Injection G2, equality back on the representation, fails it.
|
||||
|
||||
### a_forced_fraction_never_reads_as_infinity
|
||||
|
||||
A should-fix from the review of `302594bf`. A forced `n/d` with a 400-digit `n` read as
|
||||
`Inexact(inf)`, though clause 3 says a finite lexeme never reads as infinity. This test
|
||||
**failed at `302594bf`**. It also asserts:
|
||||
- `is_number` agrees with the parse;
|
||||
- a negative one is text too;
|
||||
- an underflowing `1/9…9` is still the inexact zero, so a fix that refused every inexact
|
||||
forced fraction cannot pass.
|
||||
|
||||
Injection G3 fails it.
|
||||
|
||||
### an_interim_compares_exactly_without_storing
|
||||
|
||||
New API, from a should-fix in the review: `Interim` had no comparison and no zero test
|
||||
while step 3 is told to evaluate through it. For two wide products `a`, it compares `a`
|
||||
against:
|
||||
- another wide product;
|
||||
- `a ± 1/(3·10^30)`;
|
||||
- `a` itself;
|
||||
- the f64 nearest `a`, and that f64's upper neighbour.
|
||||
|
||||
Each comparison is made in both directions, against the `BigRational` order. It also
|
||||
asserts `a − a` is zero and `a` is not. Injection G4, storing before comparing, fails it,
|
||||
and so do F3 and F13, because the stored arms delegate to `Num::cmp_exact`.
|
||||
|
||||
### an_interim_compares_exactly_without_storing_catches_storing_first
|
||||
|
||||
The rival is the comparison the review warned step 3 would otherwise have to make: store
|
||||
both operands, then compare. `a` and `a ± 1/(3·10^30)` store to one f64, so the rival
|
||||
calls them equal.
|
||||
|
||||
---
|
||||
|
||||
## `crates/numtower/src/sum.rs`
|
||||
|
||||
### sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once
|
||||
|
||||
The combine must return the **lcm** as its denominator, and the reduced result must equal
|
||||
the `BigRational` sum. Group numerators are up to `i128 × i64`, and the denominators share
|
||||
factors (3, 77, 2^20, 3^40), so the lcm and the product differ.
|
||||
|
||||
### sum::tests::the_grouped_read_combines_over_the_lcm_and_reduces_once_catches_the_product
|
||||
|
||||
The rival combines over the product of the denominators. Its value is right, so only the
|
||||
lcm assertion can see it. That is why the property asserts the denominator before the
|
||||
value.
|
||||
@@ -0,0 +1,458 @@
|
||||
//! How far a SUM maintained by insert/retract deltas drifts from the eager fold,
|
||||
//! in f64, and what an exact accumulator costs. ADR 009 (improvise-felo.4), clause 8.
|
||||
|
||||
use std::hint::black_box;
|
||||
use std::time::Instant;
|
||||
|
||||
use improvise_core::format::format_f64;
|
||||
use improvise_core::model::cell::{CellKey, CellValue};
|
||||
use improvise_core::model::types::{Aggregation, Model};
|
||||
|
||||
const MEASURE: &str = "_Measure";
|
||||
|
||||
/// SplitMix64: seeded, so a run is reproducible and a second seed is a new sample.
|
||||
struct Rng(u64);
|
||||
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
|
||||
let mut z = self.0;
|
||||
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
|
||||
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
|
||||
z ^ (z >> 31)
|
||||
}
|
||||
|
||||
fn below(&mut self, n: u64) -> u64 {
|
||||
self.next() % n
|
||||
}
|
||||
}
|
||||
|
||||
/// A user-entered decimal as an exact count of 1e-5 units, its magnitude spread
|
||||
/// over `decades` decades. The f64 the store holds is `typed(units)`.
|
||||
fn decimal(rng: &mut Rng, decades: u32) -> i64 {
|
||||
let scale = 10i64.pow(rng.below(u64::from(decades)) as u32);
|
||||
let cents = rng.below(100_000) as i64;
|
||||
let sign = if rng.below(4) == 0 { -1 } else { 1 };
|
||||
sign * cents * scale
|
||||
}
|
||||
|
||||
fn typed(units: i64) -> f64 {
|
||||
units as f64 / 1e5
|
||||
}
|
||||
|
||||
/// The exact decimal sum, rounded to f64 once: what ADR 005's arithmetic would show.
|
||||
fn exact(units: &[i64]) -> f64 {
|
||||
units.iter().map(|&u| i128::from(u)).sum::<i128>() as f64 / 1e5
|
||||
}
|
||||
|
||||
fn key(row: usize) -> CellKey {
|
||||
CellKey::new(vec![
|
||||
("Row".to_string(), format!("r{row}")),
|
||||
(MEASURE.to_string(), "Amount".to_string()),
|
||||
])
|
||||
}
|
||||
|
||||
/// `cells` rows of one measure; the eager SUM is the one the grid draws.
|
||||
fn document(rng: &mut Rng, cells: usize, decades: u32) -> (Model, Vec<i64>) {
|
||||
let mut m = Model::new("drift");
|
||||
m.add_category("Row").expect("under the limit");
|
||||
let values: Vec<i64> = (0..cells).map(|_| decimal(rng, decades)).collect();
|
||||
for (row, v) in values.iter().enumerate() {
|
||||
m.category_mut("Row")
|
||||
.expect("just added")
|
||||
.add_item(format!("r{row}"));
|
||||
m.set_cell(key(row), CellValue::Number(typed(*v)));
|
||||
}
|
||||
(m, values)
|
||||
}
|
||||
|
||||
fn eager(m: &Model) -> f64 {
|
||||
let total = CellKey::new(vec![(MEASURE.to_string(), "Amount".to_string())]);
|
||||
m.evaluate_aggregated_f64(&total, Aggregation::OverHidden)
|
||||
}
|
||||
|
||||
/// Neumaier's compensated running sum: the cheap mitigation, measured beside the naive one.
|
||||
#[derive(Default)]
|
||||
struct Compensated {
|
||||
sum: f64,
|
||||
carry: f64,
|
||||
}
|
||||
|
||||
impl Compensated {
|
||||
fn add(&mut self, x: f64) {
|
||||
let t = self.sum + x;
|
||||
self.carry += match self.sum.abs() >= x.abs() {
|
||||
true => (self.sum - t) + x,
|
||||
false => (x - t) + self.sum,
|
||||
};
|
||||
self.sum = t;
|
||||
}
|
||||
|
||||
fn value(&self) -> f64 {
|
||||
self.sum + self.carry
|
||||
}
|
||||
}
|
||||
|
||||
const LIMB_BITS: u32 = 32;
|
||||
/// Every finite f64 is `mant * 2^(shift - 1074)` with `shift <= 2045` and `mant < 2^53`.
|
||||
const LIMBS: usize = 67;
|
||||
/// The most weight one limb absorbs between carries: each add moves a limb by < 2^32.
|
||||
const CHUNK: i64 = 1 << 30;
|
||||
|
||||
/// An exact sum of f64s: a signed fixed-point integer in units of 2^-1074, wide enough
|
||||
/// for every finite f64, with non-finite values counted so they can be retracted too.
|
||||
#[derive(Clone)]
|
||||
struct ExactSum {
|
||||
limbs: [i64; LIMBS],
|
||||
pending: u64,
|
||||
nan: i64,
|
||||
pos_inf: i64,
|
||||
neg_inf: i64,
|
||||
}
|
||||
|
||||
impl ExactSum {
|
||||
fn new() -> Self {
|
||||
ExactSum {
|
||||
limbs: [0; LIMBS],
|
||||
pending: 0,
|
||||
nan: 0,
|
||||
pos_inf: 0,
|
||||
neg_inf: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Add `x` with any `weight` (a consolidated Z-set weight). Exact: no rounding here.
|
||||
fn add(&mut self, x: f64, weight: i64) {
|
||||
match (x.is_nan(), x.is_infinite()) {
|
||||
(true, _) => self.nan += weight,
|
||||
(false, true) if x > 0.0 => self.pos_inf += weight,
|
||||
(false, true) => self.neg_inf += weight,
|
||||
(false, false) => {
|
||||
let mut left = weight;
|
||||
while left != 0 {
|
||||
let chunk = left.clamp(-CHUNK, CHUNK);
|
||||
self.add_finite(x, chunk);
|
||||
left -= chunk;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Bounded so each limb stays under 2^62: `|weight| <= CHUNK` and `pending`
|
||||
/// counts magnitude, not calls.
|
||||
fn add_finite(&mut self, x: f64, weight: i64) {
|
||||
if self.pending + weight.unsigned_abs() > CHUNK as u64 {
|
||||
propagate(&mut self.limbs);
|
||||
self.pending = 0;
|
||||
}
|
||||
let bits = x.to_bits();
|
||||
let biased = ((bits >> 52) & 0x7ff) as u32;
|
||||
let fraction = bits & ((1 << 52) - 1);
|
||||
let (mant, shift) = match biased {
|
||||
0 => (fraction, 0),
|
||||
e => (fraction | (1 << 52), e - 1),
|
||||
};
|
||||
let sign = if x.is_sign_negative() {
|
||||
-weight
|
||||
} else {
|
||||
weight
|
||||
};
|
||||
let wide = u128::from(mant) << (shift % LIMB_BITS);
|
||||
let k = (shift / LIMB_BITS) as usize;
|
||||
for i in 0..3 {
|
||||
let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64;
|
||||
self.limbs[k + i] += sign * part;
|
||||
}
|
||||
self.pending += weight.unsigned_abs();
|
||||
}
|
||||
|
||||
/// The exact sum rounded to the nearest f64, ties to even: the one rounding.
|
||||
fn value(&self) -> f64 {
|
||||
if self.nan > 0 || (self.pos_inf > 0 && self.neg_inf > 0) {
|
||||
return f64::NAN;
|
||||
}
|
||||
if self.pos_inf > 0 {
|
||||
return f64::INFINITY;
|
||||
}
|
||||
if self.neg_inf > 0 {
|
||||
return f64::NEG_INFINITY;
|
||||
}
|
||||
let mut limbs = self.limbs;
|
||||
propagate(&mut limbs);
|
||||
let negative = limbs[LIMBS - 1] < 0;
|
||||
if negative {
|
||||
limbs.iter_mut().for_each(|l| *l = -*l);
|
||||
propagate(&mut limbs);
|
||||
}
|
||||
let magnitude = round_to_f64(&digits(&limbs));
|
||||
match negative {
|
||||
true => -magnitude,
|
||||
false => magnitude,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Carry each limb into the next, leaving every limb but the top in [0, 2^32).
|
||||
fn propagate(limbs: &mut [i64; LIMBS]) {
|
||||
for k in 0..LIMBS - 1 {
|
||||
let carry = limbs[k] >> LIMB_BITS;
|
||||
limbs[k] -= carry << LIMB_BITS;
|
||||
limbs[k + 1] += carry;
|
||||
}
|
||||
}
|
||||
|
||||
/// A non-negative normalized accumulator as 32-bit digits, the top limb split out.
|
||||
fn digits(limbs: &[i64; LIMBS]) -> Vec<u64> {
|
||||
let mut out: Vec<u64> = limbs.iter().map(|&l| l as u64).collect();
|
||||
let mut top = out.pop().unwrap_or(0);
|
||||
loop {
|
||||
out.push(top & 0xffff_ffff);
|
||||
top >>= LIMB_BITS;
|
||||
if top == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn bit(d: &[u64], pos: usize) -> u64 {
|
||||
(d[pos / 32] >> (pos % 32)) & 1
|
||||
}
|
||||
|
||||
/// Round an exact multiple of 2^-1074 to the nearest f64, ties to even.
|
||||
fn round_to_f64(d: &[u64]) -> f64 {
|
||||
let Some(k) = d.iter().rposition(|&x| x != 0) else {
|
||||
return 0.0;
|
||||
};
|
||||
let msb = 32 * k + 63 - d[k].leading_zeros() as usize;
|
||||
if msb < 53 {
|
||||
let int = d[0] | (d[1] << 32);
|
||||
return int as f64 * f64::from_bits(1);
|
||||
}
|
||||
let mut mant = (0..53).fold(0u64, |m, i| m | (bit(d, msb - 52 + i) << i));
|
||||
let round = bit(d, msb - 53) == 1;
|
||||
let sticky = (0..msb - 53).any(|p| bit(d, p) == 1);
|
||||
let mut top = msb;
|
||||
if round && (sticky || mant & 1 == 1) {
|
||||
mant += 1;
|
||||
if mant == 1 << 53 {
|
||||
mant >>= 1;
|
||||
top += 1;
|
||||
}
|
||||
}
|
||||
let exp = top as i64 - 52 - 1074;
|
||||
if exp > 971 {
|
||||
return f64::INFINITY;
|
||||
}
|
||||
let scale = match exp >= -1022 {
|
||||
true => f64::from_bits(((exp + 1023) as u64) << 52),
|
||||
false => f64::from_bits(1u64 << (exp + 1074)),
|
||||
};
|
||||
mant as f64 * scale
|
||||
}
|
||||
|
||||
fn exact_over(values: &[i64]) -> f64 {
|
||||
let mut acc = ExactSum::new();
|
||||
values.iter().for_each(|&v| acc.add(typed(v), 1));
|
||||
acc.value()
|
||||
}
|
||||
|
||||
fn ulps(a: f64, b: f64) -> u64 {
|
||||
(a.to_bits() as i64).abs_diff(b.to_bits() as i64)
|
||||
}
|
||||
|
||||
struct Sums {
|
||||
exact: f64,
|
||||
eager: f64,
|
||||
naive: f64,
|
||||
comp: f64,
|
||||
xs_maintained: f64,
|
||||
xs_fresh: f64,
|
||||
}
|
||||
|
||||
/// Each running sum against the eager fold (the differential oracle) and against the
|
||||
/// exact decimal sum. `flip`: the gap reaches `FLOAT_EQ_EPSILON`, so `=` can answer differently.
|
||||
fn report(label: &str, cycles: usize, s: &Sums) {
|
||||
let shown = |v: f64| format_f64(v, true, 2);
|
||||
println!(
|
||||
"{label:<22} {cycles:>8} {:>10.2e} {:>6} {:>10.2e} {:>6} {:>5} {:>10.2e} {:>10.2e} {:>10.2e} {:>7} {:>10.2e}",
|
||||
(s.naive - s.eager).abs(),
|
||||
ulps(s.naive, s.eager),
|
||||
(s.comp - s.eager).abs(),
|
||||
ulps(s.comp, s.eager),
|
||||
(s.naive - s.eager).abs() >= 1e-10,
|
||||
(s.eager - s.exact).abs(),
|
||||
(s.naive - s.exact).abs(),
|
||||
(s.comp - s.exact).abs(),
|
||||
s.xs_maintained.to_bits() == s.xs_fresh.to_bits(),
|
||||
(s.xs_maintained - s.exact).abs(),
|
||||
);
|
||||
if shown(s.naive) != shown(s.eager) || shown(s.eager) != shown(s.exact) {
|
||||
println!(
|
||||
" shown at ,.2: naive {} eager {} exact {} maintained-exact-sum {}",
|
||||
shown(s.naive),
|
||||
shown(s.eager),
|
||||
shown(s.exact),
|
||||
shown(s.xs_maintained)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Each cycle retracts one live value and inserts a fresh one at the same key.
|
||||
fn random_updates(seed: u64, cells: usize, decades: u32, checkpoints: &[usize]) {
|
||||
let mut rng = Rng(seed);
|
||||
let (mut m, mut values) = document(&mut rng, cells, decades);
|
||||
let mut naive = eager(&m);
|
||||
let mut comp = Compensated::default();
|
||||
comp.add(naive);
|
||||
let mut xs = ExactSum::new();
|
||||
values.iter().for_each(|&v| xs.add(typed(v), 1));
|
||||
let mut done = 0;
|
||||
for &at in checkpoints {
|
||||
while done < at {
|
||||
let row = rng.below(cells as u64) as usize;
|
||||
let new = decimal(&mut rng, decades);
|
||||
naive = naive - typed(values[row]) + typed(new);
|
||||
comp.add(-typed(values[row]));
|
||||
comp.add(typed(new));
|
||||
xs.add(typed(values[row]), -1);
|
||||
xs.add(typed(new), 1);
|
||||
values[row] = new;
|
||||
m.set_cell(key(row), CellValue::Number(typed(new)));
|
||||
done += 1;
|
||||
}
|
||||
let sums = Sums {
|
||||
exact: exact(&values),
|
||||
eager: eager(&m),
|
||||
naive,
|
||||
comp: comp.value(),
|
||||
xs_maintained: xs.value(),
|
||||
xs_fresh: exact_over(&values),
|
||||
};
|
||||
report(&format!("{cells} cells, {decades} dec"), at, &sums);
|
||||
}
|
||||
}
|
||||
|
||||
/// A large value arrives, small ones accumulate under it, then it is retracted.
|
||||
fn absorbed_by_a_large_value() {
|
||||
let mut m = Model::new("absorb");
|
||||
m.add_category("Row").expect("under the limit");
|
||||
m.category_mut("Row").expect("just added").add_item("r0");
|
||||
let mut naive = 0.0;
|
||||
let mut comp = Compensated::default();
|
||||
let mut xs = ExactSum::new();
|
||||
m.set_cell(key(0), CellValue::Number(1e17));
|
||||
naive += 1e17;
|
||||
comp.add(1e17);
|
||||
xs.add(1e17, 1);
|
||||
for row in 1..=1000 {
|
||||
m.category_mut("Row")
|
||||
.expect("just added")
|
||||
.add_item(format!("r{row}"));
|
||||
m.set_cell(key(row), CellValue::Number(0.01));
|
||||
naive += 0.01;
|
||||
comp.add(0.01);
|
||||
xs.add(0.01, 1);
|
||||
}
|
||||
m.clear_cell(&key(0));
|
||||
naive -= 1e17;
|
||||
comp.add(-1e17);
|
||||
xs.add(1e17, -1);
|
||||
let mut fresh = ExactSum::new();
|
||||
(0..1000).for_each(|_| fresh.add(0.01, 1));
|
||||
let sums = Sums {
|
||||
exact: 10.0,
|
||||
eager: eager(&m),
|
||||
naive,
|
||||
comp: comp.value(),
|
||||
xs_maintained: xs.value(),
|
||||
xs_fresh: fresh.value(),
|
||||
};
|
||||
report("1e17, 1000 x 0.01, -1e17", 1001, &sums);
|
||||
}
|
||||
|
||||
/// Nanoseconds per operation over `n` values, best of three runs.
|
||||
fn per_op(n: usize, mut op: impl FnMut() -> f64) -> f64 {
|
||||
(0..3)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
black_box(op());
|
||||
t.elapsed().as_secs_f64() * 1e9 / n as f64
|
||||
})
|
||||
.fold(f64::INFINITY, f64::min)
|
||||
}
|
||||
|
||||
fn cost() {
|
||||
let n = 10_000_000;
|
||||
let mut rng = Rng(7);
|
||||
let values: Vec<f64> = (0..n).map(|_| typed(decimal(&mut rng, 9))).collect();
|
||||
let updates = 3 * n;
|
||||
let plain = per_op(updates, || {
|
||||
values
|
||||
.iter()
|
||||
.fold(0.0, |s, &v| black_box(s + v) - black_box(v) + v)
|
||||
});
|
||||
let exact_add = per_op(updates, || {
|
||||
let mut xs = ExactSum::new();
|
||||
values.iter().for_each(|&v| {
|
||||
xs.add(black_box(v), 1);
|
||||
xs.add(black_box(v), -1);
|
||||
xs.add(v, 1);
|
||||
});
|
||||
xs.limbs[0] as f64
|
||||
});
|
||||
let mut xs = ExactSum::new();
|
||||
values.iter().for_each(|&v| xs.add(v, 1));
|
||||
let read = per_op(1, || xs.value());
|
||||
println!();
|
||||
println!("cost, {n} values x (insert, retract, insert), best of 3, per single update:");
|
||||
println!(" plain f64 add/sub {plain:>8.2} ns per update");
|
||||
println!(" exact accumulator {exact_add:>8.2} ns per update");
|
||||
println!(" exact read (round) {read:>8.0} ns per read");
|
||||
println!(
|
||||
" state per group {} bytes, against 8 for an f64",
|
||||
std::mem::size_of::<ExactSum>()
|
||||
);
|
||||
}
|
||||
|
||||
/// A consolidated Z-set weight can be large; the accumulator must stay exact at 2^33.
|
||||
fn large_weight() {
|
||||
let x = f64::from_bits((1u64 << 32) - 1);
|
||||
let mut xs = ExactSum::new();
|
||||
xs.add(x, 1 << 33);
|
||||
let want = ((1u64 << 32) - 1) as f64 * 2f64.powi(33) * f64::from_bits(1);
|
||||
println!();
|
||||
println!(
|
||||
"large weight: (2^32-1) x 2^-1074 at weight 2^33 reads {:e}, exact {:e}, equal {}",
|
||||
xs.value(),
|
||||
want,
|
||||
xs.value().to_bits() == want.to_bits()
|
||||
);
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!(
|
||||
"{:<22} {:>8} {:>10} {:>6} {:>10} {:>6} {:>5} {:>10} {:>10} {:>10} {:>7} {:>10}",
|
||||
"case",
|
||||
"cycles",
|
||||
"naive-eag",
|
||||
"ulp",
|
||||
"comp-eag",
|
||||
"ulp",
|
||||
"flip",
|
||||
"eag-exact",
|
||||
"nai-exact",
|
||||
"cmp-exact",
|
||||
"xs=fresh",
|
||||
"xs-exact"
|
||||
);
|
||||
let checkpoints = [1_000, 10_000, 100_000, 1_000_000];
|
||||
for seed in [1, 2] {
|
||||
random_updates(seed, 1_000, 3, &checkpoints);
|
||||
random_updates(seed, 18_000, 3, &checkpoints);
|
||||
random_updates(seed, 18_000, 9, &checkpoints);
|
||||
}
|
||||
absorbed_by_a_large_value();
|
||||
large_weight();
|
||||
cost();
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
[package]
|
||||
name = "numtower"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
license = "Apache-2.0"
|
||||
description = "A three-level numeric tower: exact decimals, bounded exact fractions and inexact f64, with Scheme's contagion rules and exact, order-free sums."
|
||||
keywords = ["decimal", "rational", "exact", "numeric-tower"]
|
||||
categories = ["mathematics"]
|
||||
|
||||
# ADR 005 clause 14: rust_decimal (std only), num-rational without its default
|
||||
# features (for Ratio<i128>), and num-bigint always. No serde.
|
||||
[dependencies]
|
||||
rust_decimal = { version = "1.43", default-features = false, features = ["std"] }
|
||||
num-rational = { version = "0.4.2", default-features = false }
|
||||
num-bigint = { version = "0.4", default-features = false, features = ["std"] }
|
||||
num-integer = { version = "0.1", default-features = false }
|
||||
num-traits = { version = "0.2", default-features = false, features = ["std"] }
|
||||
|
||||
[dev-dependencies]
|
||||
num-rational = { version = "0.4.2", features = ["num-bigint-std"] }
|
||||
proptest = "1"
|
||||
|
||||
[[example]]
|
||||
name = "bench"
|
||||
test = false
|
||||
@@ -0,0 +1,187 @@
|
||||
//! ADR 005 migration step 1's benchmark: `cargo run --release -p numtower --example bench`.
|
||||
//! A microbenchmark, not the evaluator; what it does and does not show is in `context/code/numtower.md`.
|
||||
|
||||
use std::hint::black_box;
|
||||
use std::time::Instant;
|
||||
|
||||
use numtower::{Decimal, ExactSum, Num, NumSum, Rounding, Typing, parse_number};
|
||||
|
||||
struct SplitMix64(u64);
|
||||
|
||||
impl SplitMix64 {
|
||||
fn next(&mut self) -> u64 {
|
||||
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
|
||||
let mut z = self.0;
|
||||
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
|
||||
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
|
||||
z ^ (z >> 31)
|
||||
}
|
||||
}
|
||||
|
||||
/// Typed two-place values from 0.01 to 999,999.99, as the ADR's appendix draws them.
|
||||
fn typed_texts(seed: u64, n: usize) -> Vec<String> {
|
||||
let mut rng = SplitMix64(seed);
|
||||
(0..n)
|
||||
.map(|_| {
|
||||
let cents = 1 + rng.next() % 99_999_999;
|
||||
format!("{}.{:02}", cents / 100, cents % 100)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn nums(texts: &[String]) -> Vec<Num> {
|
||||
texts
|
||||
.iter()
|
||||
.map(|t| parse_number(t, Typing::Inferred).unwrap())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The best of `runs`, in nanoseconds per item.
|
||||
fn best(runs: usize, items: usize, mut f: impl FnMut()) -> f64 {
|
||||
(0..runs)
|
||||
.map(|_| {
|
||||
let t = Instant::now();
|
||||
f();
|
||||
t.elapsed().as_nanos() as f64 / items as f64
|
||||
})
|
||||
.fold(f64::INFINITY, f64::min)
|
||||
}
|
||||
|
||||
fn sums() {
|
||||
println!("SUM over typed two-place values, ns per element, best of 20");
|
||||
for n in [18_000, 90_000] {
|
||||
let texts = typed_texts(1, n);
|
||||
let floats: Vec<f64> = texts.iter().map(|t| t.parse().unwrap()).collect();
|
||||
let decimals: Vec<Decimal> = texts.iter().map(|t| t.parse().unwrap()).collect();
|
||||
let values = nums(&texts);
|
||||
let f64_fold = best(20, n, || {
|
||||
black_box(black_box(&floats).iter().sum::<f64>());
|
||||
});
|
||||
let decimal_fold = best(20, n, || {
|
||||
let s = black_box(&decimals)
|
||||
.iter()
|
||||
.fold(Decimal::ZERO, |a, b| a.checked_add(*b).unwrap());
|
||||
black_box(s);
|
||||
});
|
||||
let num_fold = best(20, n, || {
|
||||
let zero = Num::exact(Decimal::ZERO);
|
||||
black_box(black_box(&values).iter().fold(zero, |a, b| &a + b));
|
||||
});
|
||||
let accumulator = best(20, n, || {
|
||||
let mut s = NumSum::new();
|
||||
black_box(&values).iter().for_each(|v| s.insert(v));
|
||||
black_box(s.read());
|
||||
});
|
||||
println!(
|
||||
" {n:>6} cells: f64 {f64_fold:.2}, rust_decimal {decimal_fold:.2}, Num + fold {num_fold:.2}, NumSum {accumulator:.2}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn operations() {
|
||||
println!("Per operation over 18,000 typed pairs, ns, best of 20");
|
||||
let (a, b) = (
|
||||
nums(&typed_texts(21, 18_000)),
|
||||
nums(&typed_texts(22, 18_000)),
|
||||
);
|
||||
let decimal =
|
||||
|v: &[Num]| -> Vec<Decimal> { v.iter().map(|n| n.to_string().parse().unwrap()).collect() };
|
||||
let (da, db) = (decimal(&a), decimal(&b));
|
||||
let pairs = a.len();
|
||||
let rd_add = best(20, pairs, || {
|
||||
da.iter().zip(&db).for_each(|(x, y)| {
|
||||
black_box(x.checked_add(*y));
|
||||
})
|
||||
});
|
||||
let rd_mul = best(20, pairs, || {
|
||||
da.iter().zip(&db).for_each(|(x, y)| {
|
||||
black_box(x.checked_mul(*y));
|
||||
})
|
||||
});
|
||||
let rd_div = best(20, pairs, || {
|
||||
da.iter().zip(&db).for_each(|(x, y)| {
|
||||
black_box(x.checked_div(*y));
|
||||
})
|
||||
});
|
||||
let add = best(20, pairs, || {
|
||||
a.iter().zip(&b).for_each(|(x, y)| drop(black_box(x + y)))
|
||||
});
|
||||
let mul = best(20, pairs, || {
|
||||
a.iter().zip(&b).for_each(|(x, y)| drop(black_box(x * y)))
|
||||
});
|
||||
let div = best(20, pairs, || {
|
||||
a.iter()
|
||||
.zip(&b)
|
||||
.for_each(|(x, y)| drop(black_box(x.checked_div(y))))
|
||||
});
|
||||
let margin = best(20, pairs, || {
|
||||
a.iter()
|
||||
.zip(&b)
|
||||
.for_each(|(rev, cost)| drop(black_box((rev - cost).checked_div(rev))));
|
||||
});
|
||||
let to_f64 = best(20, pairs, || {
|
||||
a.iter().for_each(|x| {
|
||||
black_box(x.to_f64());
|
||||
})
|
||||
});
|
||||
let places = best(20, pairs, || {
|
||||
a.iter()
|
||||
.for_each(|x| drop(black_box(x.round_to_places(2, Rounding::HalfAwayFromZero))));
|
||||
});
|
||||
println!(" rust_decimal checked: add {rd_add:.1}, mul {rd_mul:.1}, div {rd_div:.1}");
|
||||
println!(" Num: add {add:.1}, mul {mul:.1}, div {div:.1}");
|
||||
println!(
|
||||
" Num (Rev - Cost) / Rev {margin:.1}, to_f64 {to_f64:.1}, round_to_places(2) {places:.1}"
|
||||
);
|
||||
}
|
||||
|
||||
/// K groups of near-`i128` denominators: the ADR's worst case for the exact read.
|
||||
fn grouped_read() {
|
||||
println!("NumSum read over K distinct 126-bit denominators, us per read, best of 200");
|
||||
for k in [2usize, 8] {
|
||||
let mut rng = SplitMix64(0x5EED);
|
||||
let mut sum = NumSum::new();
|
||||
for _ in 0..k {
|
||||
let den = ((rng.next() as i128) << 62 | rng.next() as i128 >> 2) | 1 | (1 << 125);
|
||||
let num = (rng.next() as i128) << 60 ^ rng.next() as i128;
|
||||
let term = Num::ratio(num, den).unwrap();
|
||||
sum.add(&term, i64::MAX);
|
||||
sum.add(&term, i64::MAX);
|
||||
}
|
||||
assert_eq!(sum.groups(), k);
|
||||
let read = best(200, 1, || drop(black_box(sum.read())));
|
||||
println!(
|
||||
" K = {k}: {:.1} us (exact: {})",
|
||||
read / 1000.0,
|
||||
sum.read().is_exact()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn exact_sum() {
|
||||
println!("ExactSum (f64 superaccumulator), best of 20");
|
||||
let floats: Vec<f64> = typed_texts(1, 1_000_000)
|
||||
.iter()
|
||||
.map(|t| t.parse().unwrap())
|
||||
.collect();
|
||||
let mut acc = ExactSum::new();
|
||||
let insert = best(20, floats.len(), || {
|
||||
acc = ExactSum::new();
|
||||
floats.iter().for_each(|x| acc.add(*x, 1));
|
||||
});
|
||||
let read = best(20, 1, || {
|
||||
black_box(acc.value());
|
||||
});
|
||||
println!(
|
||||
" {insert:.2} ns per insert, {:.2} us per read",
|
||||
read / 1000.0
|
||||
);
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("numtower bench (release); name the machine when you quote these");
|
||||
sums();
|
||||
operations();
|
||||
grouped_read();
|
||||
exact_sum();
|
||||
}
|
||||
@@ -0,0 +1,445 @@
|
||||
//! Arithmetic and contagion (ADR 005 clauses 4 and 5, question 13 (b)).
|
||||
//! The reasoning for each rule is in `context/code/numtower.md`, "Arithmetic".
|
||||
|
||||
use std::ops::{Add, Mul, Neg, Sub};
|
||||
|
||||
use num_bigint::{BigInt, BigUint};
|
||||
use num_integer::Integer;
|
||||
use num_traits::{One, Signed, ToPrimitive, Zero};
|
||||
use rust_decimal::Decimal;
|
||||
|
||||
use crate::fixed::I256;
|
||||
use crate::num::{MANTISSA_LIMIT, MAX_SCALE, Num, NumRef, decimal, pow10, small_ratio};
|
||||
use crate::ratio::BigQ;
|
||||
|
||||
/// The widest an exact intermediate may grow, in bits of numerator or denominator.
|
||||
pub const CEILING_BITS: u64 = 8192;
|
||||
|
||||
/// A value inside one evaluation: exact intermediates are unbounded up to the ceiling.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Interim(Inner);
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum Inner {
|
||||
Stored(Num),
|
||||
Wide(Box<BigQ>),
|
||||
}
|
||||
|
||||
/// The divisor was exactly zero, exact or inexact (clause 5).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub struct DivisionByZero;
|
||||
|
||||
impl From<Num> for Interim {
|
||||
fn from(n: Num) -> Interim {
|
||||
Interim(Inner::Stored(n))
|
||||
}
|
||||
}
|
||||
|
||||
impl Interim {
|
||||
fn wide(q: BigQ) -> Interim {
|
||||
Interim(Inner::Wide(Box::new(q)))
|
||||
}
|
||||
|
||||
/// The stored bound of question 13 (b): a wide value is the nearest f64.
|
||||
pub fn store(self) -> Num {
|
||||
match self.0 {
|
||||
Inner::Stored(n) => n,
|
||||
Inner::Wide(q) => Num::inexact(q.to_f64()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_exact(&self) -> bool {
|
||||
match &self.0 {
|
||||
Inner::Stored(n) => n.is_exact(),
|
||||
Inner::Wide(_) => true,
|
||||
}
|
||||
}
|
||||
|
||||
fn operand(&self) -> Operand<'_> {
|
||||
match &self.0 {
|
||||
Inner::Stored(n) => Operand::of(n),
|
||||
Inner::Wide(q) => Operand::Big(q),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn checked_div(&self, o: &Interim) -> Result<Interim, DivisionByZero> {
|
||||
divide(self.operand(), o.operand())
|
||||
}
|
||||
|
||||
pub fn pow(&self, o: &Interim) -> Result<Interim, DivisionByZero> {
|
||||
power(self.operand(), o.operand())
|
||||
}
|
||||
|
||||
/// Exactly zero; a wide value never is, because zero fits a stored arm.
|
||||
pub fn is_zero(&self) -> bool {
|
||||
match &self.0 {
|
||||
Inner::Stored(n) => n.is_zero(),
|
||||
Inner::Wide(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Clause 6's exact order, before the stored bound; `None` only when either is NaN.
|
||||
pub fn cmp_exact(&self, o: &Interim) -> Option<std::cmp::Ordering> {
|
||||
match (&self.0, &o.0) {
|
||||
(Inner::Stored(a), Inner::Stored(b)) => a.cmp_exact(b),
|
||||
_ => match (self.operand().big(), o.operand().big()) {
|
||||
(Some(a), Some(b)) => Some(a.cmp_value(&b)),
|
||||
(Some(a), None) => crate::cmp::exact_against_f64(&a, o.operand().float()),
|
||||
(None, Some(b)) => {
|
||||
crate::cmp::exact_against_f64(&b, self.operand().float()).map(|c| c.reverse())
|
||||
}
|
||||
(None, None) => self.operand().float().partial_cmp(&o.operand().float()),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Num {
|
||||
pub fn checked_div(&self, o: &Num) -> Result<Num, DivisionByZero> {
|
||||
divide(Operand::of(self), Operand::of(o)).map(Interim::store)
|
||||
}
|
||||
|
||||
pub fn pow(&self, o: &Num) -> Result<Num, DivisionByZero> {
|
||||
power(Operand::of(self), Operand::of(o)).map(Interim::store)
|
||||
}
|
||||
}
|
||||
|
||||
macro_rules! binary_ops {
|
||||
($($trait:ident $method:ident $f:ident;)*) => {$(
|
||||
impl $trait<&Interim> for &Interim {
|
||||
type Output = Interim;
|
||||
fn $method(self, o: &Interim) -> Interim {
|
||||
$f(self.operand(), o.operand())
|
||||
}
|
||||
}
|
||||
impl $trait<&Num> for &Num {
|
||||
type Output = Num;
|
||||
fn $method(self, o: &Num) -> Num {
|
||||
$f(Operand::of(self), Operand::of(o)).store()
|
||||
}
|
||||
}
|
||||
)*};
|
||||
}
|
||||
|
||||
binary_ops! {
|
||||
Add add plus;
|
||||
Sub sub minus;
|
||||
Mul mul times;
|
||||
}
|
||||
|
||||
impl Neg for &Interim {
|
||||
type Output = Interim;
|
||||
fn neg(self) -> Interim {
|
||||
negate(self.operand())
|
||||
}
|
||||
}
|
||||
|
||||
impl Neg for &Num {
|
||||
type Output = Num;
|
||||
fn neg(self) -> Num {
|
||||
negate(Operand::of(self)).store()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
enum Operand<'a> {
|
||||
Dec(&'a Decimal),
|
||||
Small(i128, i128),
|
||||
Big(&'a BigQ),
|
||||
Float(f64),
|
||||
}
|
||||
|
||||
impl<'a> Operand<'a> {
|
||||
fn of(n: &'a Num) -> Operand<'a> {
|
||||
match n.kind() {
|
||||
NumRef::Exact(d) => Operand::Dec(d),
|
||||
NumRef::Fraction(r) => Operand::Small(*r.numer(), *r.denom()),
|
||||
NumRef::Inexact(x) => Operand::Float(x),
|
||||
}
|
||||
}
|
||||
|
||||
/// The `i128` fast path's operands; `i128::MIN` cannot be negated, so it takes the bigint path.
|
||||
fn parts(self) -> Option<(i128, i128)> {
|
||||
match self {
|
||||
Operand::Dec(d) => Some((d.mantissa(), pow10(d.scale()))),
|
||||
Operand::Small(n, d) => (n != i128::MIN).then_some((n, d)),
|
||||
Operand::Big(_) | Operand::Float(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn big(self) -> Option<BigQ> {
|
||||
match self {
|
||||
Operand::Dec(d) => Some(BigQ::from_i128(d.mantissa(), pow10(d.scale()))),
|
||||
Operand::Small(n, d) => Some(BigQ::from_i128(n, d)),
|
||||
Operand::Big(q) => Some(q.clone()),
|
||||
Operand::Float(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn float(self) -> f64 {
|
||||
match self {
|
||||
Operand::Dec(d) => crate::num::decimal_to_f64(d),
|
||||
Operand::Small(n, d) => crate::ratio::ratio_to_f64(&BigInt::from(n), &BigInt::from(d)),
|
||||
Operand::Big(q) => q.to_f64(),
|
||||
Operand::Float(x) => x,
|
||||
}
|
||||
}
|
||||
|
||||
fn is_float(self) -> bool {
|
||||
matches!(self, Operand::Float(_))
|
||||
}
|
||||
|
||||
fn is_zero(self) -> bool {
|
||||
match self {
|
||||
Operand::Dec(d) => d.is_zero(),
|
||||
Operand::Small(..) | Operand::Big(_) => false,
|
||||
Operand::Float(x) => x == 0.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn inexact(x: f64) -> Interim {
|
||||
Interim::from(Num::inexact(x))
|
||||
}
|
||||
|
||||
/// An exact result in the first arm it fits; wide while under the ceiling, else demoted.
|
||||
fn settle_wide(q: BigQ) -> Interim {
|
||||
match Num::fit(&q) {
|
||||
Some(n) => Interim::from(n),
|
||||
None if q.bits() > CEILING_BITS => inexact(q.to_f64()),
|
||||
None => Interim::wide(q),
|
||||
}
|
||||
}
|
||||
|
||||
fn exact_or_wide(
|
||||
a: Operand,
|
||||
b: Operand,
|
||||
small: impl Fn(i128, i128, i128, i128) -> Option<(i128, i128)>,
|
||||
big: impl Fn(&BigQ, &BigQ) -> BigQ,
|
||||
) -> Interim {
|
||||
let fast = a
|
||||
.parts()
|
||||
.zip(b.parts())
|
||||
.and_then(|((n1, d1), (n2, d2))| small(n1, d1, n2, d2));
|
||||
fast.and_then(|(n, d)| small_ratio(n, d))
|
||||
.map(Interim::from)
|
||||
.unwrap_or_else(|| match a.big().zip(b.big()) {
|
||||
Some((x, y)) => settle_wide(big(&x, &y)),
|
||||
None => unreachable!("both operands are exact here"),
|
||||
})
|
||||
}
|
||||
|
||||
fn plus(a: Operand, b: Operand) -> Interim {
|
||||
match (a, b) {
|
||||
(x, y) if x.is_float() || y.is_float() => inexact(x.float() + y.float()),
|
||||
(Operand::Dec(x), Operand::Dec(y)) => decimal_sum(x, y),
|
||||
_ => exact_or_wide(
|
||||
a,
|
||||
b,
|
||||
|n1, d1, n2, d2| {
|
||||
let n = n1.checked_mul(d2)?.checked_add(n2.checked_mul(d1)?)?;
|
||||
Some((n, d1.checked_mul(d2)?))
|
||||
},
|
||||
BigQ::add,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
fn minus(a: Operand, b: Operand) -> Interim {
|
||||
plus(a, negate(b).operand())
|
||||
}
|
||||
|
||||
fn times(a: Operand, b: Operand) -> Interim {
|
||||
match (a, b) {
|
||||
(x, y) if x.is_float() || y.is_float() => inexact(x.float() * y.float()),
|
||||
(Operand::Dec(x), Operand::Dec(y)) => decimal_product(x, y),
|
||||
_ => exact_or_wide(
|
||||
a,
|
||||
b,
|
||||
|n1, d1, n2, d2| {
|
||||
let (g1, g2) = (n1.gcd(&d2).max(1), n2.gcd(&d1).max(1));
|
||||
Some((
|
||||
(n1 / g1).checked_mul(n2 / g2)?,
|
||||
(d1 / g2).checked_mul(d2 / g1)?,
|
||||
))
|
||||
},
|
||||
BigQ::mul,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
fn negate(a: Operand) -> Interim {
|
||||
match a {
|
||||
Operand::Dec(d) => Interim::from(Num::exact(-*d)),
|
||||
Operand::Float(x) => inexact(-x),
|
||||
Operand::Small(n, d) => n
|
||||
.checked_neg()
|
||||
.and_then(|m| small_ratio(m, d))
|
||||
.map(Interim::from)
|
||||
.unwrap_or_else(|| settle_wide(BigQ::from_i128(n, d).neg())),
|
||||
Operand::Big(q) => settle_wide(q.neg()),
|
||||
}
|
||||
}
|
||||
|
||||
fn divide(a: Operand, b: Operand) -> Result<Interim, DivisionByZero> {
|
||||
match (a, b) {
|
||||
(_, y) if y.is_zero() => Err(DivisionByZero),
|
||||
(x, y) if x.is_float() || y.is_float() => Ok(inexact(x.float() / y.float())),
|
||||
_ => Ok(exact_or_wide(
|
||||
a,
|
||||
b,
|
||||
|n1, d1, n2, d2| {
|
||||
let (g1, g2) = (n1.gcd(&n2).max(1), d1.gcd(&d2).max(1));
|
||||
Some((
|
||||
(n1 / g1).checked_mul(d2 / g2)?,
|
||||
(d1 / g2).checked_mul(n2 / g1)?,
|
||||
))
|
||||
},
|
||||
|x, y| {
|
||||
x.div(y)
|
||||
.unwrap_or_else(|| unreachable!("the divisor is not zero"))
|
||||
},
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Clause 4's hint: a returned scale below the nominal one may mean rounding.
|
||||
fn decimal_sum(x: &Decimal, y: &Decimal) -> Interim {
|
||||
let nominal = x.scale().max(y.scale());
|
||||
match x.checked_add(*y).filter(|r| r.scale() == nominal) {
|
||||
Some(r) => Interim::from(Num::exact(r)),
|
||||
None => {
|
||||
let align = |d: &Decimal| {
|
||||
I256::from_i128(d.mantissa())
|
||||
.checked_mul_u128(pow10(nominal - d.scale()) as u128)
|
||||
.expect("an aligned mantissa is below 2^190")
|
||||
};
|
||||
let sum = align(x).checked_add(&align(y)).expect("below 2^191");
|
||||
settle_decimal_integer(sum, nominal, nominal)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn decimal_product(x: &Decimal, y: &Decimal) -> Interim {
|
||||
let nominal = x.scale() + y.scale();
|
||||
match x.checked_mul(*y).filter(|r| r.scale() == nominal) {
|
||||
Some(r) => Interim::from(Num::exact(r)),
|
||||
None => {
|
||||
settle_decimal_integer(I256::mul_i128(x.mantissa(), y.mantissa()), nominal, nominal)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The recompute of clause 4: strip trailing zeros, fit 96 bits at scale <= 28, else a fraction.
|
||||
fn settle_decimal_integer(m: I256, scale: u32, nominal: u32) -> Interim {
|
||||
let (mut m, mut s) = (m, scale);
|
||||
while s > 0 {
|
||||
let (q, r) = m.div_rem_u64(10);
|
||||
if r != 0 {
|
||||
break;
|
||||
}
|
||||
m = q;
|
||||
s -= 1;
|
||||
}
|
||||
let fits = m
|
||||
.to_i128()
|
||||
.filter(|v| v.unsigned_abs() < MANTISSA_LIMIT as u128);
|
||||
match fits {
|
||||
Some(v) if s <= MAX_SCALE => Interim::from(padded(v, s, nominal.min(MAX_SCALE))),
|
||||
_ => settle_wide(BigQ::new(m.to_bigint(), BigInt::from(10).pow(s))),
|
||||
}
|
||||
}
|
||||
|
||||
/// The stored scale of clause 4: pad toward `target` while the mantissa fits 96 bits.
|
||||
pub(crate) fn padded(mantissa: i128, scale: u32, target: u32) -> Num {
|
||||
let (mut m, mut s) = (mantissa, scale);
|
||||
while s < target && (m * 10).unsigned_abs() < MANTISSA_LIMIT as u128 {
|
||||
m *= 10;
|
||||
s += 1;
|
||||
}
|
||||
decimal(m, s).expect("a padded mantissa still fits")
|
||||
}
|
||||
|
||||
/// The exponent of an exact integer-valued operand.
|
||||
fn integer_value(b: Operand) -> Option<BigInt> {
|
||||
match b {
|
||||
Operand::Dec(d) if d.fract().is_zero() => {
|
||||
Some(BigInt::from(d.mantissa() / pow10(d.scale())))
|
||||
}
|
||||
Operand::Small(n, 1) => Some(BigInt::from(n)),
|
||||
Operand::Big(q) if q.den.is_one() => Some(q.num.clone()),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn power(a: Operand, b: Operand) -> Result<Interim, DivisionByZero> {
|
||||
match (a.is_float(), integer_value(b)) {
|
||||
(false, Some(n)) => exact_power(a, &n),
|
||||
_ => Ok(inexact(a.float().powf(b.float()))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Clause 5's `POW`: the size is checked before anything is computed.
|
||||
fn exact_power(a: Operand, n: &BigInt) -> Result<Interim, DivisionByZero> {
|
||||
let q = a.big().expect("an exact base");
|
||||
let base_scale = match a {
|
||||
Operand::Dec(d) => d.scale(),
|
||||
_ => 0,
|
||||
};
|
||||
let k = n.magnitude();
|
||||
let target = match n.is_negative() {
|
||||
true => 0,
|
||||
false => k
|
||||
.to_u32()
|
||||
.and_then(|k| k.checked_mul(base_scale))
|
||||
.map_or(MAX_SCALE, |s| s.min(MAX_SCALE)),
|
||||
};
|
||||
let unit = q.den.is_one() && q.num.magnitude().is_one();
|
||||
match (q.num.is_zero(), n.is_negative(), unit) {
|
||||
(_, _, _) if n.is_zero() => Ok(Interim::from(padded(1, 0, 0))),
|
||||
(true, true, _) => Err(DivisionByZero),
|
||||
(true, false, _) => Ok(Interim::from(padded(0, 0, target))),
|
||||
(false, _, true) => {
|
||||
let negative = q.num.is_negative() && k.is_odd();
|
||||
Ok(Interim::from(padded(
|
||||
if negative { -1 } else { 1 },
|
||||
0,
|
||||
target,
|
||||
)))
|
||||
}
|
||||
(false, negative, false) => {
|
||||
let lower = |x: &BigInt| BigUint::from(x.bits().saturating_sub(1)) * k;
|
||||
let too_big = lower(&q.num).max(lower(&q.den)) > BigUint::from(CEILING_BITS);
|
||||
let odd_negative = q.num.is_negative() && k.is_odd();
|
||||
match (too_big, k.to_u32()) {
|
||||
(false, Some(k)) => Ok(exact_integer_power(&q, k, negative, target)),
|
||||
_ => Ok(inexact(signed_powf(a.float(), n, odd_negative))),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `powf` on the magnitude, the sign taken from the exact exponent's parity.
|
||||
fn signed_powf(base: f64, n: &BigInt, odd_negative: bool) -> f64 {
|
||||
let magnitude = base.abs().powf(n.to_f64().unwrap_or(f64::INFINITY));
|
||||
match odd_negative {
|
||||
true => -magnitude,
|
||||
false => magnitude,
|
||||
}
|
||||
}
|
||||
|
||||
fn exact_integer_power(q: &BigQ, k: u32, negative: bool, target: u32) -> Interim {
|
||||
let (num, den) = (q.num.pow(k), q.den.pow(k));
|
||||
let value = match negative {
|
||||
true => BigQ::new(den, num),
|
||||
false => BigQ { num, den },
|
||||
};
|
||||
let settled = settle_wide(value);
|
||||
match (&settled.0, negative) {
|
||||
(Inner::Stored(n), false) => match n.kind() {
|
||||
NumRef::Exact(d) => Interim::from(padded(d.mantissa(), d.scale(), target)),
|
||||
_ => settled,
|
||||
},
|
||||
_ => settled,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
//! Exact comparison at every pairing of arms (clause 6), and the `MIN`/`MAX`
|
||||
//! tie-breaks (clause 8). Why: `context/code/numtower.md`, "Comparison".
|
||||
|
||||
use std::cmp::Ordering;
|
||||
|
||||
use crate::fixed::I256;
|
||||
use crate::num::{Num, NumRef};
|
||||
use crate::ratio::BigQ;
|
||||
|
||||
impl Num {
|
||||
/// The order of the two values, exactly; `None` only when either is NaN.
|
||||
pub fn cmp_exact(&self, o: &Num) -> Option<Ordering> {
|
||||
match (self.exact_parts(), o.exact_parts()) {
|
||||
(Some(a), Some(b)) => Some(cross(a, b)),
|
||||
(Some(_), None) => mixed(self, o.to_f64()),
|
||||
(None, Some(_)) => mixed(o, self.to_f64()).map(Ordering::reverse),
|
||||
(None, None) => self.to_f64().partial_cmp(&o.to_f64()),
|
||||
}
|
||||
}
|
||||
|
||||
/// The smaller, NaN skipped; ties keep the most information (clause 8, question 11).
|
||||
pub fn min_of<'a>(&'a self, o: &'a Num) -> &'a Num {
|
||||
pick(self, o, Ordering::Less)
|
||||
}
|
||||
|
||||
pub fn max_of<'a>(&'a self, o: &'a Num) -> &'a Num {
|
||||
pick(self, o, Ordering::Greater)
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialEq for Num {
|
||||
fn eq(&self, o: &Num) -> bool {
|
||||
self.cmp_exact(o) == Some(Ordering::Equal)
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialOrd for Num {
|
||||
fn partial_cmp(&self, o: &Num) -> Option<Ordering> {
|
||||
self.cmp_exact(o)
|
||||
}
|
||||
}
|
||||
|
||||
/// `a.0/a.1` against `b.0/b.1`, denominators positive, in 256 bits.
|
||||
fn cross(a: (i128, i128), b: (i128, i128)) -> Ordering {
|
||||
I256::mul_i128(a.0, b.1).cmp(&I256::mul_i128(b.0, a.1))
|
||||
}
|
||||
|
||||
/// An exact value against an f64: the rounded value decides unless it ties.
|
||||
fn mixed(exact: &Num, x: f64) -> Option<Ordering> {
|
||||
match (x.is_nan(), x.is_infinite()) {
|
||||
(true, _) => None,
|
||||
(false, true) => Some(if x > 0.0 {
|
||||
Ordering::Less
|
||||
} else {
|
||||
Ordering::Greater
|
||||
}),
|
||||
(false, false) => match exact.to_f64().partial_cmp(&x)? {
|
||||
Ordering::Equal => {
|
||||
let q = exact.exact_q().expect("an exact operand");
|
||||
Some(q.cmp_value(&BigQ::from_f64(x)))
|
||||
}
|
||||
decided => Some(decided),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// A wide exact value against an f64, with no fast path: the slow, total form of `mixed`.
|
||||
pub(crate) fn exact_against_f64(q: &BigQ, x: f64) -> Option<Ordering> {
|
||||
match (x.is_nan(), x.is_infinite(), x > 0.0) {
|
||||
(true, _, _) => None,
|
||||
(false, true, true) => Some(Ordering::Less),
|
||||
(false, true, false) => Some(Ordering::Greater),
|
||||
(false, false, _) => Some(q.cmp_value(&BigQ::from_f64(x))),
|
||||
}
|
||||
}
|
||||
|
||||
fn pick<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num {
|
||||
match (a.cmp_exact(b), is_nan(a), is_nan(b)) {
|
||||
(_, true, _) => b,
|
||||
(_, false, true) => a,
|
||||
(Some(o), _, _) if o == want => a,
|
||||
(Some(Ordering::Equal), _, _) => tie(a, b, want),
|
||||
_ => b,
|
||||
}
|
||||
}
|
||||
|
||||
fn is_nan(n: &Num) -> bool {
|
||||
matches!(n.kind(), NumRef::Inexact(x) if x.is_nan())
|
||||
}
|
||||
|
||||
/// Equal values: exact over inexact, the larger scale, and IEEE's signed zero.
|
||||
fn tie<'a>(a: &'a Num, b: &'a Num, want: Ordering) -> &'a Num {
|
||||
match (a.kind(), b.kind()) {
|
||||
(NumRef::Inexact(_), NumRef::Exact(_) | NumRef::Fraction(_)) => b,
|
||||
(NumRef::Exact(_) | NumRef::Fraction(_), NumRef::Inexact(_)) => a,
|
||||
(NumRef::Exact(x), NumRef::Exact(y)) if y.scale() > x.scale() => b,
|
||||
(NumRef::Inexact(x), NumRef::Inexact(y)) => {
|
||||
let negative_wanted = want == Ordering::Less;
|
||||
match y.is_sign_negative() == negative_wanted && x.is_sign_negative() != negative_wanted
|
||||
{
|
||||
true => b,
|
||||
false => a,
|
||||
}
|
||||
}
|
||||
_ => a,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
//! Fixed-width two's-complement integers for the hot paths: no allocation.
|
||||
//! Why they are hand-rolled: `context/code/numtower.md`, "The wide integers".
|
||||
|
||||
use std::cmp::Ordering;
|
||||
|
||||
use num_bigint::{BigInt, Sign};
|
||||
|
||||
/// A signed integer of `64 * N` bits, little-endian limbs, two's complement.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
|
||||
pub(crate) struct Fixed<const N: usize> {
|
||||
limbs: [u64; N],
|
||||
}
|
||||
|
||||
/// The 256-bit integer clauses 4–6 use, and the 320-bit decimal partition of clause 8.
|
||||
pub(crate) type I256 = Fixed<4>;
|
||||
pub(crate) type I320 = Fixed<5>;
|
||||
|
||||
impl<const N: usize> Fixed<N> {
|
||||
pub(crate) const ZERO: Self = Fixed { limbs: [0; N] };
|
||||
|
||||
pub(crate) fn from_i128(x: i128) -> Self {
|
||||
let fill = if x < 0 { u64::MAX } else { 0 };
|
||||
let mut limbs = [fill; N];
|
||||
limbs[0] = x as u64;
|
||||
limbs[1] = (x >> 64) as u64;
|
||||
Fixed { limbs }
|
||||
}
|
||||
|
||||
pub(crate) fn is_negative(&self) -> bool {
|
||||
self.limbs[N - 1] >> 63 == 1
|
||||
}
|
||||
|
||||
pub(crate) fn is_zero(&self) -> bool {
|
||||
self.limbs.iter().all(|&l| l == 0)
|
||||
}
|
||||
|
||||
fn wrapping_neg(&self) -> Self {
|
||||
let mut out = [0u64; N];
|
||||
let mut carry = 1u64;
|
||||
for (o, &l) in out.iter_mut().zip(&self.limbs) {
|
||||
let (v, c) = (!l).overflowing_add(carry);
|
||||
*o = v;
|
||||
carry = c as u64;
|
||||
}
|
||||
Fixed { limbs: out }
|
||||
}
|
||||
|
||||
fn wrapping_add(&self, other: &Self) -> Self {
|
||||
let mut out = [0u64; N];
|
||||
let mut carry = false;
|
||||
for (o, (a, b)) in out.iter_mut().zip(self.limbs.iter().zip(&other.limbs)) {
|
||||
let (s, c1) = a.overflowing_add(*b);
|
||||
let (s, c2) = s.overflowing_add(carry as u64);
|
||||
*o = s;
|
||||
carry = c1 || c2;
|
||||
}
|
||||
Fixed { limbs: out }
|
||||
}
|
||||
|
||||
pub(crate) fn checked_add(&self, other: &Self) -> Option<Self> {
|
||||
let sum = self.wrapping_add(other);
|
||||
let same = self.is_negative() == other.is_negative();
|
||||
(!same || sum.is_negative() == self.is_negative()).then_some(sum)
|
||||
}
|
||||
|
||||
pub(crate) fn checked_neg(&self) -> Option<Self> {
|
||||
let neg = self.wrapping_neg();
|
||||
(self.is_zero() || neg.is_negative() != self.is_negative()).then_some(neg)
|
||||
}
|
||||
|
||||
/// The magnitude as unsigned limbs, with the sign split out.
|
||||
fn magnitude(&self) -> (bool, [u64; N]) {
|
||||
match self.is_negative() {
|
||||
true => (true, self.wrapping_neg().limbs),
|
||||
false => (false, self.limbs),
|
||||
}
|
||||
}
|
||||
|
||||
fn from_magnitude(negative: bool, limbs: [u64; N]) -> Option<Self> {
|
||||
let unsigned = Fixed { limbs };
|
||||
match (negative, unsigned.is_negative()) {
|
||||
(false, false) => Some(unsigned),
|
||||
(true, _) => {
|
||||
let neg = unsigned.wrapping_neg();
|
||||
(neg.is_negative() || neg.is_zero()).then_some(neg)
|
||||
}
|
||||
(false, true) => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn checked_mul_u128(&self, k: u128) -> Option<Self> {
|
||||
let (negative, mag) = self.magnitude();
|
||||
let mut out = [0u64; N];
|
||||
for (j, kk) in [k as u64, (k >> 64) as u64].into_iter().enumerate() {
|
||||
if kk == 0 {
|
||||
continue;
|
||||
}
|
||||
let mut carry = 0u128;
|
||||
for (i, &limb) in mag.iter().enumerate() {
|
||||
let idx = i + j;
|
||||
let prod = limb as u128 * kk as u128 + carry;
|
||||
match idx < N {
|
||||
true => {
|
||||
let s = out[idx] as u128 + (prod as u64) as u128;
|
||||
out[idx] = s as u64;
|
||||
carry = (prod >> 64) + (s >> 64);
|
||||
}
|
||||
false if prod != 0 => return None,
|
||||
false => carry = 0,
|
||||
}
|
||||
}
|
||||
if carry != 0 {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
Self::from_magnitude(negative, out)
|
||||
}
|
||||
|
||||
/// Divides the magnitude by `k`, returning the quotient (sign kept) and the remainder.
|
||||
pub(crate) fn div_rem_u64(&self, k: u64) -> (Self, u64) {
|
||||
let (negative, mag) = self.magnitude();
|
||||
let mut out = [0u64; N];
|
||||
let mut rem = 0u128;
|
||||
for i in (0..N).rev() {
|
||||
let cur = (rem << 64) | mag[i] as u128;
|
||||
out[i] = (cur / k as u128) as u64;
|
||||
rem = cur % k as u128;
|
||||
}
|
||||
let q = Self::from_magnitude(negative, out).unwrap_or(Self::ZERO);
|
||||
(q, rem as u64)
|
||||
}
|
||||
|
||||
pub(crate) fn to_i128(self) -> Option<i128> {
|
||||
let fill = if self.is_negative() { u64::MAX } else { 0 };
|
||||
let hi_ok = self.limbs[2..].iter().all(|&l| l == fill);
|
||||
let v = (self.limbs[0] as u128 | ((self.limbs[1] as u128) << 64)) as i128;
|
||||
(hi_ok && (v < 0) == self.is_negative()).then_some(v)
|
||||
}
|
||||
|
||||
pub(crate) fn to_bigint(self) -> BigInt {
|
||||
let (negative, mag) = self.magnitude();
|
||||
let digits: Vec<u32> = mag
|
||||
.iter()
|
||||
.flat_map(|&l| [l as u32, (l >> 32) as u32])
|
||||
.collect();
|
||||
let sign = if negative { Sign::Minus } else { Sign::Plus };
|
||||
BigInt::from_slice(sign, &digits)
|
||||
}
|
||||
|
||||
/// The product of two `i128`s; needs `N >= 4`.
|
||||
pub(crate) fn mul_i128(a: i128, b: i128) -> Self {
|
||||
let negative = (a < 0) != (b < 0);
|
||||
let (x, y) = (a.unsigned_abs(), b.unsigned_abs());
|
||||
let (x0, x1) = (x as u64 as u128, x >> 64);
|
||||
let (y0, y1) = (y as u64 as u128, y >> 64);
|
||||
let p00 = x0 * y0;
|
||||
let p01 = x0 * y1;
|
||||
let p10 = x1 * y0;
|
||||
let p11 = x1 * y1;
|
||||
let mid = (p00 >> 64) + (p01 as u64 as u128) + (p10 as u64 as u128);
|
||||
let mut limbs = [0u64; N];
|
||||
limbs[0] = p00 as u64;
|
||||
limbs[1] = mid as u64;
|
||||
let hi = (mid >> 64) + (p01 >> 64) + (p10 >> 64) + p11;
|
||||
limbs[2] = hi as u64;
|
||||
limbs[3] = (hi >> 64) as u64;
|
||||
let unsigned = Fixed { limbs };
|
||||
match negative {
|
||||
true => unsigned.wrapping_neg(),
|
||||
false => unsigned,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<const N: usize> Default for Fixed<N> {
|
||||
fn default() -> Self {
|
||||
Self::ZERO
|
||||
}
|
||||
}
|
||||
|
||||
impl<const N: usize> Ord for Fixed<N> {
|
||||
fn cmp(&self, other: &Self) -> Ordering {
|
||||
match (self.is_negative(), other.is_negative()) {
|
||||
(true, false) => Ordering::Less,
|
||||
(false, true) => Ordering::Greater,
|
||||
_ => self.limbs.iter().rev().cmp(other.limbs.iter().rev()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl<const N: usize> PartialOrd for Fixed<N> {
|
||||
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
|
||||
Some(self.cmp(other))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn the_256_bit_product_of_two_i128s_matches_bigint() {
|
||||
let cases = [
|
||||
(i128::MAX, i128::MAX),
|
||||
(i128::MAX, -i128::MAX),
|
||||
(-1, 1),
|
||||
(0, i128::MIN),
|
||||
(123_456_789_012_345_678_901_234, -98_765_432_109_876_543_210),
|
||||
];
|
||||
for (a, b) in cases {
|
||||
let want = BigInt::from(a) * BigInt::from(b);
|
||||
assert_eq!(I256::mul_i128(a, b).to_bigint(), want, "{a} * {b}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checked_arithmetic_refuses_what_overflows_and_keeps_what_fits() {
|
||||
let max = I256::from_i128(i128::MAX)
|
||||
.checked_mul_u128(u128::MAX)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
max.to_bigint(),
|
||||
BigInt::from(i128::MAX) * BigInt::from(u128::MAX)
|
||||
);
|
||||
assert!(max.checked_mul_u128(u128::MAX).is_none());
|
||||
let neg = max.checked_neg().unwrap();
|
||||
assert_eq!(neg.to_bigint(), -max.to_bigint());
|
||||
let (q, r) = neg.div_rem_u64(10);
|
||||
assert_eq!(q.to_bigint() * 10 - BigInt::from(r), neg.to_bigint());
|
||||
assert_eq!(I256::from_i128(-5).to_i128(), Some(-5));
|
||||
assert_eq!(max.to_i128(), None);
|
||||
assert!(I256::from_i128(-1) < I256::ZERO);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
//! Clause 3's one number language: the gate, the exact parse, the forced `n/d` arm,
|
||||
//! and the canonical text. Why each rule: `context/code/numtower.md`, "The number language".
|
||||
|
||||
use std::fmt;
|
||||
|
||||
use num_bigint::BigInt;
|
||||
use num_traits::{One, Zero};
|
||||
|
||||
use crate::arith::padded;
|
||||
use crate::num::{MAX_SCALE, Num, NumRef};
|
||||
use crate::ratio::BigQ;
|
||||
|
||||
/// Whether the type of the text is being guessed or was forced (question 12).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Typing {
|
||||
Inferred,
|
||||
Forced,
|
||||
}
|
||||
|
||||
/// The gate: the strings `f64::from_str` accepts and reads as a finite value.
|
||||
pub fn is_number(text: &str, typing: Typing) -> bool {
|
||||
gate(text) || (typing == Typing::Forced && forced_fraction(text).is_some())
|
||||
}
|
||||
|
||||
/// A number if the text is one, read exactly; `None` means the text is text.
|
||||
pub fn parse_number(text: &str, typing: Typing) -> Option<Num> {
|
||||
match (gate(text), typing) {
|
||||
(true, _) => Lexeme::read(text).map(|lx| lx.value(text)),
|
||||
(false, Typing::Forced) => forced_fraction(text),
|
||||
(false, Typing::Inferred) => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// A forced `n/d`; one that rounds to infinity is text, as a finite lexeme never reads as one.
|
||||
fn forced_fraction(text: &str) -> Option<Num> {
|
||||
let n = Num::settle(&fraction_lexeme(text)?);
|
||||
n.to_f64().is_finite().then_some(n)
|
||||
}
|
||||
|
||||
fn gate(text: &str) -> bool {
|
||||
text.parse::<f64>().is_ok_and(f64::is_finite)
|
||||
}
|
||||
|
||||
/// `[+-]digits/digits` with a nonzero denominator, as an exact value.
|
||||
fn fraction_lexeme(text: &str) -> Option<BigQ> {
|
||||
let (n, d) = text.split_once('/')?;
|
||||
let digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit());
|
||||
let unsigned = n.strip_prefix(['+', '-']).unwrap_or(n);
|
||||
let ok = digits(unsigned) && digits(d);
|
||||
let num: BigInt = ok.then(|| n.trim_start_matches('+').parse().ok())??;
|
||||
let den: BigInt = d.parse().ok()?;
|
||||
(!den.is_zero()).then(|| BigQ::new(num, den))
|
||||
}
|
||||
|
||||
/// A lexeme of the gate's language: sign, significant digits, and the power of ten.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub(crate) struct Lexeme {
|
||||
pub(crate) negative: bool,
|
||||
pub(crate) digits: String,
|
||||
pub(crate) exp10: i64,
|
||||
}
|
||||
|
||||
/// Exponents beyond this cannot reach either exact arm, and saturate.
|
||||
const EXP_SATURATE: i64 = 1 << 40;
|
||||
|
||||
impl Lexeme {
|
||||
/// `[+-]? (d+ (. d*)? | . d+) ([eE] [+-]? d+)?`, and nothing else.
|
||||
pub(crate) fn read(text: &str) -> Option<Lexeme> {
|
||||
let (negative, rest) = match text.as_bytes().first()? {
|
||||
b'-' => (true, &text[1..]),
|
||||
b'+' => (false, &text[1..]),
|
||||
_ => (false, text),
|
||||
};
|
||||
let (mantissa, exponent) = match rest.find(['e', 'E']) {
|
||||
Some(i) => (&rest[..i], Some(&rest[i + 1..])),
|
||||
None => (rest, None),
|
||||
};
|
||||
let (int, frac) = mantissa.split_once('.').unwrap_or((mantissa, ""));
|
||||
let all_digits = |s: &str| s.bytes().all(|b| b.is_ascii_digit());
|
||||
let mantissa_ok =
|
||||
all_digits(int) && all_digits(frac) && !(int.is_empty() && frac.is_empty());
|
||||
let exp = match exponent {
|
||||
Some(e) => read_exponent(e)?,
|
||||
None => 0,
|
||||
};
|
||||
let digits = format!("{int}{frac}");
|
||||
mantissa_ok.then(|| Lexeme {
|
||||
negative,
|
||||
digits,
|
||||
exp10: exp.saturating_sub(frac.len() as i64),
|
||||
})
|
||||
}
|
||||
|
||||
/// Clause 3's order: a decimal at the lexeme's scale, then a fraction, then the nearest f64.
|
||||
fn value(&self, text: &str) -> Num {
|
||||
let significant = self.digits.trim_start_matches('0');
|
||||
let stripped = significant.trim_end_matches('0');
|
||||
let implied_scale = (-self.exp10).clamp(0, MAX_SCALE as i64) as u32;
|
||||
match stripped.is_empty() {
|
||||
true => padded(0, 0, implied_scale),
|
||||
false => {
|
||||
let shift = self
|
||||
.exp10
|
||||
.saturating_add((significant.len() - stripped.len()) as i64);
|
||||
self.exact(stripped, shift, implied_scale)
|
||||
.unwrap_or_else(|| Num::inexact(text.parse().unwrap_or(f64::NAN)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn exact(&self, stripped: &str, shift: i64, implied_scale: u32) -> Option<Num> {
|
||||
let leading = stripped.len() as i64 + shift;
|
||||
let reachable = stripped.len() <= 180 && (-130..=40).contains(&leading) && shift >= -400;
|
||||
let q = reachable.then(|| self.rational(stripped, shift))?;
|
||||
Num::fit(&q).map(|n| match n.kind() {
|
||||
NumRef::Exact(d) => padded(d.mantissa(), d.scale(), implied_scale),
|
||||
_ => n,
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn rational(&self, stripped: &str, shift: i64) -> BigQ {
|
||||
let m: BigInt = stripped.parse().unwrap_or_default();
|
||||
let m = if self.negative { -m } else { m };
|
||||
let ten = BigInt::from(10);
|
||||
match shift >= 0 {
|
||||
true => BigQ::new(m * ten.pow(shift as u32), BigInt::one()),
|
||||
false => BigQ::new(m, ten.pow((-shift) as u32)),
|
||||
}
|
||||
}
|
||||
|
||||
/// The exact value, whatever its size: for display rounding.
|
||||
pub(crate) fn exact_value(&self) -> BigQ {
|
||||
let stripped = self.digits.trim_start_matches('0');
|
||||
match stripped.is_empty() {
|
||||
true => BigQ::new(BigInt::zero(), BigInt::one()),
|
||||
false => self.rational(stripped, self.exp10),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn read_exponent(e: &str) -> Option<i64> {
|
||||
let (negative, digits) = match e.as_bytes().first()? {
|
||||
b'-' => (true, &e[1..]),
|
||||
b'+' => (false, &e[1..]),
|
||||
_ => (false, e),
|
||||
};
|
||||
let ok = !digits.is_empty() && digits.bytes().all(|b| b.is_ascii_digit());
|
||||
let significant = digits.trim_start_matches('0');
|
||||
let magnitude = match significant.len() > 12 {
|
||||
true => EXP_SATURATE,
|
||||
false => significant.parse::<i64>().unwrap_or(0).min(EXP_SATURATE),
|
||||
};
|
||||
ok.then_some(if negative { -magnitude } else { magnitude })
|
||||
}
|
||||
|
||||
/// The shortest text that reads back as `x`; `None` for a non-finite value.
|
||||
pub fn shortest_round_trip(x: f64) -> Option<String> {
|
||||
x.is_finite().then(|| format!("{x:?}"))
|
||||
}
|
||||
|
||||
const INEXACT_MARK: &str = "#i";
|
||||
|
||||
impl Num {
|
||||
/// Reads every spelling `Display` writes: a decimal, `n/d`, `#i…`, `inf`, `-inf`, `nan`.
|
||||
pub fn from_canonical(text: &str) -> Option<Num> {
|
||||
match text {
|
||||
"inf" => Some(Num::inexact(f64::INFINITY)),
|
||||
"-inf" => Some(Num::inexact(f64::NEG_INFINITY)),
|
||||
"nan" => Some(Num::inexact(f64::NAN)),
|
||||
_ => match text.strip_prefix(INEXACT_MARK) {
|
||||
Some(rest) if gate(rest) => rest.parse().ok().map(Num::inexact),
|
||||
Some(_) => None,
|
||||
None => parse_number(text, Typing::Forced).filter(Num::is_exact),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The canonical text: a decimal with its scale, `n/d`, `#i` and the shortest spelling, or a token.
|
||||
impl fmt::Display for Num {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self.kind() {
|
||||
NumRef::Exact(d) => write!(f, "{d}"),
|
||||
NumRef::Fraction(r) => write!(f, "{}/{}", r.numer(), r.denom()),
|
||||
NumRef::Inexact(x) if x.is_nan() => f.write_str("nan"),
|
||||
NumRef::Inexact(x) if x.is_infinite() => {
|
||||
f.write_str(if x > 0.0 { "inf" } else { "-inf" })
|
||||
}
|
||||
NumRef::Inexact(x) => write!(f, "{INEXACT_MARK}{x:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
//! A three-level numeric tower: exact decimal, exact bounded fraction, inexact f64.
|
||||
//! The decisions are ADR 005; the reasoning is `context/code/numtower.md`.
|
||||
|
||||
mod arith;
|
||||
mod cmp;
|
||||
mod fixed;
|
||||
mod lang;
|
||||
mod num;
|
||||
mod ratio;
|
||||
mod round;
|
||||
mod sum;
|
||||
|
||||
pub use arith::{CEILING_BITS, DivisionByZero, Interim};
|
||||
pub use lang::{Typing, is_number, parse_number, shortest_round_trip};
|
||||
pub use num::{Num, NumRef};
|
||||
pub use round::Rounding;
|
||||
pub use sum::{ExactSum, K, NumSum};
|
||||
|
||||
pub use num_rational::Ratio;
|
||||
pub use rust_decimal::Decimal;
|
||||
@@ -0,0 +1,198 @@
|
||||
//! The stored number: an exact decimal, an exact bounded fraction, or an f64.
|
||||
//! The arms are private so a value is always normalized: `context/code/numtower.md`.
|
||||
|
||||
use num_bigint::BigInt;
|
||||
use num_integer::Integer;
|
||||
use num_rational::Ratio;
|
||||
use num_traits::{One, ToPrimitive};
|
||||
use rust_decimal::Decimal;
|
||||
|
||||
use crate::ratio::{BigQ, ratio_to_f64};
|
||||
|
||||
/// The most significant digits a `Decimal` holds, and its mantissa bound.
|
||||
pub(crate) const MAX_SCALE: u32 = 28;
|
||||
pub(crate) const MANTISSA_LIMIT: i128 = 1 << 96;
|
||||
|
||||
/// A number of the tower (ADR 005 clause 2), always in its one normal form.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Num(Repr);
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum Repr {
|
||||
Exact(Decimal),
|
||||
Fraction(Box<Ratio<i128>>),
|
||||
Inexact(f64),
|
||||
}
|
||||
|
||||
/// A borrowed view of a `Num`'s arm, for exhaustive matching by callers.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum NumRef<'a> {
|
||||
Exact(&'a Decimal),
|
||||
Fraction(&'a Ratio<i128>),
|
||||
Inexact(f64),
|
||||
}
|
||||
|
||||
impl Num {
|
||||
/// An exact decimal. Zero loses its sign: the sign of zero is an f64 artefact.
|
||||
pub fn exact(d: Decimal) -> Num {
|
||||
let mut d = d;
|
||||
if d.is_zero() {
|
||||
d.set_sign_positive(true);
|
||||
}
|
||||
Num(Repr::Exact(d))
|
||||
}
|
||||
|
||||
pub fn inexact(x: f64) -> Num {
|
||||
Num(Repr::Inexact(x))
|
||||
}
|
||||
|
||||
/// The exact value `num / den`, normalized; `None` when `den` is zero.
|
||||
pub fn ratio(num: i128, den: i128) -> Option<Num> {
|
||||
(den != 0).then(|| Num::settle(&BigQ::from_i128(num, den)))
|
||||
}
|
||||
|
||||
fn fraction(num: i128, den: i128) -> Num {
|
||||
Num(Repr::Fraction(Box::new(Ratio::new_raw(num, den))))
|
||||
}
|
||||
|
||||
pub fn kind(&self) -> NumRef<'_> {
|
||||
match &self.0 {
|
||||
Repr::Exact(d) => NumRef::Exact(d),
|
||||
Repr::Fraction(r) => NumRef::Fraction(r),
|
||||
Repr::Inexact(x) => NumRef::Inexact(*x),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_exact(&self) -> bool {
|
||||
!matches!(self.0, Repr::Inexact(_))
|
||||
}
|
||||
|
||||
/// Exactly zero, at either level; the one zero predicate of clause 6.
|
||||
pub fn is_zero(&self) -> bool {
|
||||
match &self.0 {
|
||||
Repr::Exact(d) => d.is_zero(),
|
||||
Repr::Fraction(_) => false,
|
||||
Repr::Inexact(x) => *x == 0.0,
|
||||
}
|
||||
}
|
||||
|
||||
/// The correctly rounded f64 (clause 4): a decimal through its text, a fraction exactly.
|
||||
pub fn to_f64(&self) -> f64 {
|
||||
match &self.0 {
|
||||
Repr::Exact(d) => decimal_to_f64(d),
|
||||
Repr::Fraction(r) => ratio_to_f64(&BigInt::from(*r.numer()), &BigInt::from(*r.denom())),
|
||||
Repr::Inexact(x) => *x,
|
||||
}
|
||||
}
|
||||
|
||||
/// The exact value as a rational; `None` for an inexact number.
|
||||
pub(crate) fn exact_parts(&self) -> Option<(i128, i128)> {
|
||||
match &self.0 {
|
||||
Repr::Exact(d) => Some((d.mantissa(), pow10(d.scale()))),
|
||||
Repr::Fraction(r) => Some((*r.numer(), *r.denom())),
|
||||
Repr::Inexact(_) => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn exact_q(&self) -> Option<BigQ> {
|
||||
self.exact_parts().map(|(n, d)| BigQ::from_i128(n, d))
|
||||
}
|
||||
|
||||
/// The stored bound: a decimal if it fits, else an `i128` fraction, else the nearest f64.
|
||||
pub(crate) fn settle(q: &BigQ) -> Num {
|
||||
Num::fit(q).unwrap_or_else(|| Num::inexact(q.to_f64()))
|
||||
}
|
||||
|
||||
/// `q` in an exact arm, at its minimal scale, if it fits one.
|
||||
pub(crate) fn fit(q: &BigQ) -> Option<Num> {
|
||||
decimal_of(q).or_else(|| {
|
||||
let num = q.num.to_i128()?;
|
||||
let den = q.den.to_i128()?;
|
||||
Some(Num::fraction(num, den))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// `num / den` normalized without allocating; `None` hands the case to the bigint path.
|
||||
pub(crate) fn small_ratio(num: i128, den: i128) -> Option<Num> {
|
||||
let representable = num != i128::MIN && den != i128::MIN && den != 0;
|
||||
representable.then(|| {
|
||||
let g = num.gcd(&den);
|
||||
let (n, d) = match den < 0 {
|
||||
true => (-num / g, -den / g),
|
||||
false => (num / g, den / g),
|
||||
};
|
||||
let p = d.trailing_zeros();
|
||||
let mut rest = d >> p;
|
||||
let mut q5 = 0;
|
||||
while rest % 5 == 0 {
|
||||
rest /= 5;
|
||||
q5 += 1;
|
||||
}
|
||||
let k = p.max(q5);
|
||||
let as_decimal = (rest == 1 && k <= MAX_SCALE)
|
||||
.then(|| 2i128.pow(k - p) * 5i128.pow(k - q5))
|
||||
.and_then(|pad| n.checked_mul(pad))
|
||||
.and_then(|m| decimal(m, k));
|
||||
as_decimal.unwrap_or_else(|| Num::fraction(n, d))
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn pow10(k: u32) -> i128 {
|
||||
10i128.pow(k)
|
||||
}
|
||||
|
||||
pub(crate) fn decimal_to_f64(d: &Decimal) -> f64 {
|
||||
d.to_string().parse().unwrap_or(f64::NAN)
|
||||
}
|
||||
|
||||
/// A decimal from a mantissa and scale, if the mantissa fits 96 bits.
|
||||
pub(crate) fn decimal(mantissa: i128, scale: u32) -> Option<Num> {
|
||||
let fits = mantissa.unsigned_abs() < MANTISSA_LIMIT as u128 && scale <= MAX_SCALE;
|
||||
fits.then(|| Num::exact(Decimal::from_i128_with_scale(mantissa, scale)))
|
||||
}
|
||||
|
||||
/// The direct test of clause 5: a reduced `n / (2^p 5^q)` is a decimal iff it fits.
|
||||
fn decimal_of(q: &BigQ) -> Option<Num> {
|
||||
let p = q.den.trailing_zeros().unwrap_or(0);
|
||||
let mut rest = &q.den >> p as usize;
|
||||
let five = BigInt::from(5);
|
||||
let mut q5 = 0u64;
|
||||
while rest.is_multiple_of(&five) && q5 <= MAX_SCALE as u64 {
|
||||
rest /= &five;
|
||||
q5 += 1;
|
||||
}
|
||||
let k = p.max(q5);
|
||||
if !rest.is_one() || k > MAX_SCALE as u64 {
|
||||
return None;
|
||||
}
|
||||
let pad = (BigInt::one() << (k - p) as usize) * five.pow((k - q5) as u32);
|
||||
let mantissa = (&q.num * pad).to_i128()?;
|
||||
decimal(mantissa, k as u32)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_num_is_24_bytes() {
|
||||
assert_eq!(std::mem::size_of::<Num>(), 24);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fraction_that_is_a_fitting_decimal_is_stored_as_the_decimal() {
|
||||
let half = Num::ratio(1, 2).unwrap();
|
||||
assert_eq!(half.kind(), NumRef::Exact(&Decimal::new(5, 1)));
|
||||
let third = Num::ratio(-10, 3).unwrap();
|
||||
assert_eq!(third.kind(), NumRef::Fraction(&Ratio::new_raw(-10, 3)));
|
||||
let tiny = Num::ratio(1, 1 << 40).unwrap();
|
||||
assert!(matches!(tiny.kind(), NumRef::Fraction(_)), "{tiny:?}");
|
||||
let mut minus_zero = Decimal::new(0, 2);
|
||||
minus_zero.set_sign_negative(true);
|
||||
assert!(minus_zero.is_sign_negative());
|
||||
let zero = Num::exact(minus_zero);
|
||||
assert!(!zero.to_f64().is_sign_negative());
|
||||
assert!(Num::ratio(1, 0).is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
//! Exact rationals in bigints: the widened intermediate of question 13 (b), and
|
||||
//! the one correctly rounded conversion of a rational to f64 (clauses 4 and 5).
|
||||
|
||||
use std::cmp::Ordering;
|
||||
|
||||
use num_bigint::{BigInt, BigUint, Sign};
|
||||
use num_integer::Integer;
|
||||
use num_traits::{One, Signed, ToPrimitive, Zero};
|
||||
|
||||
/// A reduced fraction with a positive denominator.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct BigQ {
|
||||
pub(crate) num: BigInt,
|
||||
pub(crate) den: BigInt,
|
||||
}
|
||||
|
||||
impl BigQ {
|
||||
pub(crate) fn new(num: BigInt, den: BigInt) -> BigQ {
|
||||
let (num, den) = match den.is_negative() {
|
||||
true => (-num, -den),
|
||||
false => (num, den),
|
||||
};
|
||||
let g = num.gcd(&den);
|
||||
match g.is_zero() || g.is_one() {
|
||||
true => BigQ { num, den },
|
||||
false => BigQ {
|
||||
num: num / &g,
|
||||
den: den / &g,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn from_i128(num: i128, den: i128) -> BigQ {
|
||||
BigQ::new(BigInt::from(num), BigInt::from(den))
|
||||
}
|
||||
|
||||
pub(crate) fn add(&self, o: &BigQ) -> BigQ {
|
||||
BigQ::new(&self.num * &o.den + &o.num * &self.den, &self.den * &o.den)
|
||||
}
|
||||
|
||||
pub(crate) fn neg(&self) -> BigQ {
|
||||
BigQ {
|
||||
num: -&self.num,
|
||||
den: self.den.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn mul(&self, o: &BigQ) -> BigQ {
|
||||
BigQ::new(&self.num * &o.num, &self.den * &o.den)
|
||||
}
|
||||
|
||||
/// `None` when `o` is zero.
|
||||
pub(crate) fn div(&self, o: &BigQ) -> Option<BigQ> {
|
||||
(!o.num.is_zero()).then(|| BigQ::new(&self.num * &o.den, &self.den * &o.num))
|
||||
}
|
||||
|
||||
pub(crate) fn cmp_value(&self, o: &BigQ) -> Ordering {
|
||||
(&self.num * &o.den).cmp(&(&o.num * &self.den))
|
||||
}
|
||||
|
||||
pub(crate) fn bits(&self) -> u64 {
|
||||
self.num.bits().max(self.den.bits())
|
||||
}
|
||||
|
||||
pub(crate) fn to_f64(&self) -> f64 {
|
||||
ratio_to_f64(&self.num, &self.den)
|
||||
}
|
||||
|
||||
/// The exact value of a finite f64.
|
||||
pub(crate) fn from_f64(x: f64) -> BigQ {
|
||||
let bits = x.to_bits();
|
||||
let biased = ((bits >> 52) & 0x7ff) as i64;
|
||||
let fraction = bits & ((1u64 << 52) - 1);
|
||||
let (mant, exp) = match biased {
|
||||
0 => (fraction, -1074),
|
||||
e => (fraction | (1 << 52), e - 1075),
|
||||
};
|
||||
let signed = match x.is_sign_negative() {
|
||||
true => -BigInt::from(mant),
|
||||
false => BigInt::from(mant),
|
||||
};
|
||||
match exp >= 0 {
|
||||
true => BigQ::new(signed << exp as usize, BigInt::one()),
|
||||
false => BigQ::new(signed, BigInt::one() << (-exp) as usize),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `num / den` rounded to the nearest f64, ties to even, subnormals and overflow included.
|
||||
pub(crate) fn ratio_to_f64(num: &BigInt, den: &BigInt) -> f64 {
|
||||
let negative = (num.sign() == Sign::Minus) != (den.sign() == Sign::Minus);
|
||||
let a = num.magnitude();
|
||||
let b = den.magnitude();
|
||||
let magnitude = match a.is_zero() {
|
||||
true => 0.0,
|
||||
false => magnitude_to_f64(a, b),
|
||||
};
|
||||
match negative {
|
||||
true => -magnitude,
|
||||
false => magnitude,
|
||||
}
|
||||
}
|
||||
|
||||
fn magnitude_to_f64(a: &BigUint, b: &BigUint) -> f64 {
|
||||
let shift = 54 - (a.bits() as i64 - b.bits() as i64);
|
||||
let (n, d) = match shift >= 0 {
|
||||
true => (a << shift as usize, b.clone()),
|
||||
false => (a.clone(), b << (-shift) as usize),
|
||||
};
|
||||
let (q, r) = n.div_rem(&d);
|
||||
let qbits = q.bits() as i64;
|
||||
let drop = (qbits - 53).max(shift - 1074);
|
||||
if drop > qbits + 1 {
|
||||
return 0.0;
|
||||
}
|
||||
let kept = (&q >> drop as usize).to_u64().unwrap_or(0);
|
||||
let round = q.bit(drop as u64 - 1);
|
||||
let sticky = !r.is_zero() || q.trailing_zeros().is_some_and(|t| (t as i64) < drop - 1);
|
||||
let up = round && (sticky || kept & 1 == 1);
|
||||
ldexp(kept + up as u64, drop - shift)
|
||||
}
|
||||
|
||||
/// `m * 2^e` for `m <= 2^53`, exact unless it overflows to infinity.
|
||||
fn ldexp(m: u64, e: i64) -> f64 {
|
||||
let top = e + 63 - m.leading_zeros() as i64;
|
||||
match (m == 0, top > 1023) {
|
||||
(true, _) => 0.0,
|
||||
(false, true) => f64::INFINITY,
|
||||
(false, false) => {
|
||||
let half = e / 2;
|
||||
m as f64 * pow2(half) * pow2(e - half)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn pow2(e: i64) -> f64 {
|
||||
match e {
|
||||
e if e >= -1022 => f64::from_bits(((e + 1023) as u64) << 52),
|
||||
e => f64::from_bits(1u64 << (e + 1074)),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn rounding_to_f64_reaches_subnormals_ties_and_overflow() {
|
||||
let one = BigInt::one();
|
||||
let min_sub = f64::from_bits(1);
|
||||
assert_eq!(ratio_to_f64(&one, &(BigInt::one() << 1074)), min_sub);
|
||||
assert_eq!(ratio_to_f64(&one, &(BigInt::one() << 1075)), 0.0);
|
||||
let three = BigInt::from(3);
|
||||
assert_eq!(ratio_to_f64(&three, &(BigInt::one() << 1076)), min_sub);
|
||||
assert_eq!(ratio_to_f64(&one, &BigInt::from(3)), 1.0 / 3.0);
|
||||
let tie = (BigInt::one() << 53) + 1;
|
||||
assert_eq!(ratio_to_f64(&tie, &one), 9007199254740992.0);
|
||||
let big = BigInt::one() << 1024;
|
||||
assert_eq!(ratio_to_f64(&big, &one), f64::INFINITY);
|
||||
assert_eq!(ratio_to_f64(&-big, &one), f64::NEG_INFINITY);
|
||||
let max = BigQ::from_f64(f64::MAX);
|
||||
assert_eq!(max.to_f64(), f64::MAX);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_f64_converts_to_its_exact_value_and_back() {
|
||||
for x in [0.1, -2.5e-310, 1e300, f64::MIN_POSITIVE, 123456.789, -7.0] {
|
||||
assert_eq!(BigQ::from_f64(x).to_f64().to_bits(), x.to_bits(), "{x}");
|
||||
}
|
||||
assert!(BigQ::from_f64(-0.0).num.is_zero());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
//! Rounding any value to `places` for display, exactly (clause 12, "Format").
|
||||
//! An inexact value is rounded over its shortest round-trip spelling (improvise-felo.9).
|
||||
|
||||
use num_bigint::BigInt;
|
||||
use num_integer::Integer;
|
||||
use num_traits::Signed;
|
||||
|
||||
use crate::lang::{Lexeme, shortest_round_trip};
|
||||
use crate::num::Num;
|
||||
use crate::ratio::BigQ;
|
||||
|
||||
/// The rounding mode is the caller's policy, not the crate's.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Rounding {
|
||||
HalfAwayFromZero,
|
||||
HalfEven,
|
||||
}
|
||||
|
||||
impl Num {
|
||||
/// Plain decimal text with exactly `places` fractional digits; `None` for a non-finite value.
|
||||
pub fn round_to_places(&self, places: u8, mode: Rounding) -> Option<String> {
|
||||
let q = match self.exact_q() {
|
||||
Some(q) => q,
|
||||
None => Lexeme::read(&shortest_round_trip(self.to_f64())?)?.exact_value(),
|
||||
};
|
||||
Some(render(round(&q, places, mode), places))
|
||||
}
|
||||
}
|
||||
|
||||
/// `q * 10^places`, rounded to an integer by `mode`.
|
||||
fn round(q: &BigQ, places: u8, mode: Rounding) -> BigInt {
|
||||
let scaled = q.num.abs() * BigInt::from(10).pow(places as u32);
|
||||
let (whole, rest) = scaled.div_rem(&q.den);
|
||||
let twice: BigInt = rest * 2;
|
||||
let up = match twice.cmp(&q.den) {
|
||||
std::cmp::Ordering::Greater => true,
|
||||
std::cmp::Ordering::Equal => mode == Rounding::HalfAwayFromZero || whole.is_odd(),
|
||||
std::cmp::Ordering::Less => false,
|
||||
};
|
||||
let magnitude = whole + BigInt::from(up as u8);
|
||||
match q.num.is_negative() {
|
||||
true => -magnitude,
|
||||
false => magnitude,
|
||||
}
|
||||
}
|
||||
|
||||
fn render(units: BigInt, places: u8) -> String {
|
||||
let digits = units.abs().to_string();
|
||||
let width = places as usize + 1;
|
||||
let padded = format!("{digits:0>width$}");
|
||||
let (int, frac) = padded.split_at(padded.len() - places as usize);
|
||||
let sign = if units.is_negative() { "-" } else { "" };
|
||||
match places {
|
||||
0 => format!("{sign}{int}"),
|
||||
_ => format!("{sign}{int}.{frac}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::lang::{Typing, parse_number};
|
||||
|
||||
fn typed(s: &str) -> Num {
|
||||
parse_number(s, Typing::Inferred).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_typed_value_rounds_as_it_was_typed() {
|
||||
let at2 = |n: &Num| n.round_to_places(2, Rounding::HalfAwayFromZero).unwrap();
|
||||
for (text, want) in [
|
||||
("1.005", "1.01"),
|
||||
("1.015", "1.02"),
|
||||
("0.285", "0.29"),
|
||||
("-2.675", "-2.68"),
|
||||
] {
|
||||
assert_eq!(at2(&typed(text)), want, "{text}");
|
||||
assert_eq!(
|
||||
at2(&Num::inexact(text.parse().unwrap())),
|
||||
want,
|
||||
"inexact {text}"
|
||||
);
|
||||
}
|
||||
assert_eq!(at2(&Num::ratio(10, 3).unwrap()), "3.33");
|
||||
assert_eq!(at2(&Num::ratio(2, 3).unwrap()), "0.67");
|
||||
assert_eq!(at2(&typed("-0.001")), "0.00");
|
||||
let even = typed("0.125")
|
||||
.round_to_places(2, Rounding::HalfEven)
|
||||
.unwrap();
|
||||
assert_eq!(even, "0.12");
|
||||
assert_eq!(
|
||||
typed("12.5")
|
||||
.round_to_places(0, Rounding::HalfEven)
|
||||
.unwrap(),
|
||||
"12"
|
||||
);
|
||||
assert_eq!(
|
||||
Num::inexact(f64::NAN).round_to_places(2, Rounding::HalfEven),
|
||||
None
|
||||
);
|
||||
let wide = Num::ratio(1, 3)
|
||||
.unwrap()
|
||||
.round_to_places(40, Rounding::HalfEven)
|
||||
.unwrap();
|
||||
assert_eq!(wide, format!("0.{}", "3".repeat(40)));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,396 @@
|
||||
//! Order-free, retractable sums (clause 8): ADR 009's f64 `ExactSum`, and `NumSum`,
|
||||
//! which adds the exact partitions. The design: `context/code/numtower.md`, "Sums".
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use num_bigint::{BigInt, BigUint};
|
||||
use num_integer::Integer;
|
||||
use num_traits::{One, Zero};
|
||||
|
||||
use crate::fixed::{I256, I320};
|
||||
use crate::num::{MAX_SCALE, Num, NumRef, pow10};
|
||||
use crate::ratio::{BigQ, ratio_to_f64};
|
||||
|
||||
/// The most non-decimal denominator groups a sum reads exactly (question 16: K = 8).
|
||||
pub const K: usize = 8;
|
||||
|
||||
const LIMB_BITS: u32 = 32;
|
||||
/// A finite f64 is `mant * 2^(shift - 1074)`, `shift <= 2045`; weights add 60 bits.
|
||||
const LIMBS: usize = 70;
|
||||
const CHUNK: u64 = 1 << 30;
|
||||
|
||||
/// An exact sum of weighted f64s, rounded once when read (ADR 009).
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct ExactSum {
|
||||
limbs: [i64; LIMBS],
|
||||
pending: u64,
|
||||
nan: i128,
|
||||
pos_inf: i128,
|
||||
neg_inf: i128,
|
||||
}
|
||||
|
||||
/// Equal sums, not equal representations: carries propagated, `pending` ignored.
|
||||
impl PartialEq for ExactSum {
|
||||
fn eq(&self, o: &ExactSum) -> bool {
|
||||
let (mut mine, mut theirs) = (self.limbs, o.limbs);
|
||||
propagate(&mut mine);
|
||||
propagate(&mut theirs);
|
||||
let counts = |s: &ExactSum| (s.nan, s.pos_inf, s.neg_inf);
|
||||
mine == theirs && counts(self) == counts(o)
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for ExactSum {}
|
||||
|
||||
impl Default for ExactSum {
|
||||
fn default() -> Self {
|
||||
ExactSum {
|
||||
limbs: [0; LIMBS],
|
||||
pending: 0,
|
||||
nan: 0,
|
||||
pos_inf: 0,
|
||||
neg_inf: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ExactSum {
|
||||
pub fn new() -> ExactSum {
|
||||
ExactSum::default()
|
||||
}
|
||||
|
||||
/// Adds `x` with a Z-set `weight`; the weight is split into base-2^30 digits.
|
||||
pub fn add(&mut self, x: f64, weight: i64) {
|
||||
match (x.is_nan(), x.is_infinite(), x > 0.0) {
|
||||
(true, _, _) => self.nan += weight as i128,
|
||||
(false, true, true) => self.pos_inf += weight as i128,
|
||||
(false, true, false) => self.neg_inf += weight as i128,
|
||||
(false, false, _) => {
|
||||
let magnitude = weight.unsigned_abs();
|
||||
for j in 0..3 {
|
||||
let digit = (magnitude >> (30 * j)) & (CHUNK - 1);
|
||||
if digit != 0 {
|
||||
let signed = if weight < 0 {
|
||||
-(digit as i64)
|
||||
} else {
|
||||
digit as i64
|
||||
};
|
||||
self.add_digit(x, signed, 30 * j);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn add_digit(&mut self, x: f64, weight: i64, extra: u32) {
|
||||
if self.pending + weight.unsigned_abs() > CHUNK {
|
||||
propagate(&mut self.limbs);
|
||||
self.pending = 0;
|
||||
}
|
||||
let bits = x.to_bits();
|
||||
let biased = ((bits >> 52) & 0x7ff) as u32;
|
||||
let fraction = bits & ((1 << 52) - 1);
|
||||
let (mant, shift) = match biased {
|
||||
0 => (fraction, 0),
|
||||
e => (fraction | (1 << 52), e - 1),
|
||||
};
|
||||
let sign = if x.is_sign_negative() {
|
||||
-weight
|
||||
} else {
|
||||
weight
|
||||
};
|
||||
let at = shift + extra;
|
||||
let wide = u128::from(mant) << (at % LIMB_BITS);
|
||||
let k = (at / LIMB_BITS) as usize;
|
||||
for i in 0..3 {
|
||||
let part = ((wide >> (LIMB_BITS * i as u32)) & 0xffff_ffff) as i64;
|
||||
self.limbs[k + i] += sign * part;
|
||||
}
|
||||
self.pending += weight.unsigned_abs();
|
||||
}
|
||||
|
||||
/// Adds another accumulator's contents.
|
||||
pub fn absorb(&mut self, o: &ExactSum) {
|
||||
let mut theirs = o.limbs;
|
||||
propagate(&mut theirs);
|
||||
propagate(&mut self.limbs);
|
||||
for (mine, t) in self.limbs.iter_mut().zip(theirs) {
|
||||
*mine += t;
|
||||
}
|
||||
propagate(&mut self.limbs);
|
||||
self.pending = 0;
|
||||
self.nan += o.nan;
|
||||
self.pos_inf += o.pos_inf;
|
||||
self.neg_inf += o.neg_inf;
|
||||
}
|
||||
|
||||
/// The exact sum rounded to the nearest f64, ties to even.
|
||||
pub fn value(&self) -> f64 {
|
||||
match (self.nan != 0, self.pos_inf != 0, self.neg_inf != 0) {
|
||||
(true, _, _) | (false, true, true) => f64::NAN,
|
||||
(false, true, false) => f64::INFINITY,
|
||||
(false, false, true) => f64::NEG_INFINITY,
|
||||
(false, false, false) => ratio_to_f64(&self.total(), &(BigInt::one() << 1074)),
|
||||
}
|
||||
}
|
||||
|
||||
/// The finite part as an integer in units of 2^-1074.
|
||||
fn total(&self) -> BigInt {
|
||||
let mut limbs = self.limbs;
|
||||
propagate(&mut limbs);
|
||||
let digits: Vec<u32> = limbs[..LIMBS - 1].iter().map(|&l| l as u32).collect();
|
||||
let low = BigInt::from(BigUint::from_slice(&digits));
|
||||
(BigInt::from(limbs[LIMBS - 1]) << (LIMB_BITS as usize * (LIMBS - 1))) + low
|
||||
}
|
||||
}
|
||||
|
||||
/// Carry each limb into the next, leaving every limb but the top in [0, 2^32).
|
||||
fn propagate(limbs: &mut [i64; LIMBS]) {
|
||||
for k in 0..LIMBS - 1 {
|
||||
let carry = limbs[k] >> LIMB_BITS;
|
||||
limbs[k] -= carry << LIMB_BITS;
|
||||
limbs[k + 1] += carry;
|
||||
}
|
||||
}
|
||||
|
||||
/// A `SUM` over any `Num`s: exact when every term is exact and at most K groups remain.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct NumSum {
|
||||
decimals: I320,
|
||||
groups: BTreeMap<i128, I256>,
|
||||
group_floats: ExactSum,
|
||||
inexact: ExactSum,
|
||||
/// Net weight per f64 bit pattern: a read is inexact while any term is present.
|
||||
inexact_terms: BTreeMap<u64, i128>,
|
||||
}
|
||||
|
||||
impl NumSum {
|
||||
pub fn new() -> NumSum {
|
||||
NumSum::default()
|
||||
}
|
||||
|
||||
pub fn insert(&mut self, x: &Num) {
|
||||
self.add(x, 1);
|
||||
}
|
||||
|
||||
pub fn retract(&mut self, x: &Num) {
|
||||
self.add(x, -1);
|
||||
}
|
||||
|
||||
/// Adds `x` with a Z-set `weight`. Exact and associative: the order cannot matter.
|
||||
pub fn add(&mut self, x: &Num, weight: i64) {
|
||||
match x.kind() {
|
||||
NumRef::Inexact(f) => {
|
||||
self.inexact.add(f, weight);
|
||||
let net = self.inexact_terms.entry(f.to_bits()).or_insert(0);
|
||||
*net += weight as i128;
|
||||
if *net == 0 {
|
||||
self.inexact_terms.remove(&f.to_bits());
|
||||
}
|
||||
}
|
||||
NumRef::Exact(d) => {
|
||||
let unit = pow10(MAX_SCALE - d.scale()) as u128;
|
||||
self.add_scaled(d.mantissa(), unit, weight);
|
||||
}
|
||||
NumRef::Fraction(r) => match at_scale_28(*r.denom()) {
|
||||
Some(unit) => self.add_scaled(*r.numer(), unit, weight),
|
||||
None => self.add_to_group(*r.numer(), *r.denom(), weight),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn add_scaled(&mut self, n: i128, unit: u128, weight: i64) {
|
||||
let magnitude = weight.unsigned_abs() as u128;
|
||||
let scaled = match unit.checked_mul(magnitude) {
|
||||
Some(both) => I320::from_i128(n).checked_mul_u128(both),
|
||||
None => I320::from_i128(n)
|
||||
.checked_mul_u128(unit)
|
||||
.and_then(|t| t.checked_mul_u128(magnitude)),
|
||||
};
|
||||
let term = scaled
|
||||
.and_then(|t| if weight < 0 { t.checked_neg() } else { Some(t) })
|
||||
.expect("a weighted term is below 2^284");
|
||||
self.decimals = self
|
||||
.decimals
|
||||
.checked_add(&term)
|
||||
.expect("320 bits holds 2^36 terms");
|
||||
}
|
||||
|
||||
fn add_to_group(&mut self, n: i128, d: i128, weight: i64) {
|
||||
let old = self.groups.get(&d).copied().unwrap_or(I256::ZERO);
|
||||
let new = old
|
||||
.checked_add(&I256::mul_i128(n, weight as i128))
|
||||
.expect("256 bits holds 2^64 terms");
|
||||
let value = |num: I256| ratio_to_f64(&num.to_bigint(), &BigInt::from(d));
|
||||
if !old.is_zero() {
|
||||
self.group_floats.add(value(old), -1);
|
||||
}
|
||||
match new.is_zero() {
|
||||
true => {
|
||||
self.groups.remove(&d);
|
||||
}
|
||||
false => {
|
||||
self.group_floats.add(value(new), 1);
|
||||
self.groups.insert(d, new);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The current non-decimal denominator groups.
|
||||
pub fn groups(&self) -> usize {
|
||||
self.groups.len()
|
||||
}
|
||||
|
||||
/// One read: exact over at most K groups, else the f64 accumulators (clause 8).
|
||||
pub fn read(&self) -> Num {
|
||||
match (!self.inexact_terms.is_empty(), self.groups.len() > K) {
|
||||
(false, false) => Num::settle(&self.exact_total()),
|
||||
(_, true) => {
|
||||
let mut acc = self.inexact.clone();
|
||||
acc.absorb(&self.group_floats);
|
||||
acc.add(self.decimal_total().to_f64(), 1);
|
||||
Num::inexact(acc.value())
|
||||
}
|
||||
(true, false) => {
|
||||
let mut acc = self.inexact.clone();
|
||||
acc.add(self.exact_total().to_f64(), 1);
|
||||
Num::inexact(acc.value())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn decimal_total(&self) -> BigQ {
|
||||
BigQ::new(self.decimals.to_bigint(), BigInt::from(pow10(MAX_SCALE)))
|
||||
}
|
||||
|
||||
fn exact_total(&self) -> BigQ {
|
||||
let decimal = (!self.decimals.is_zero())
|
||||
.then(|| (self.decimals.to_bigint(), BigInt::from(pow10(MAX_SCALE))));
|
||||
let terms: Vec<(BigInt, BigInt)> = decimal
|
||||
.into_iter()
|
||||
.chain(
|
||||
self.groups
|
||||
.iter()
|
||||
.map(|(d, n)| (n.to_bigint(), BigInt::from(*d))),
|
||||
)
|
||||
.collect();
|
||||
let (num, den) = combine_over_lcm(&terms);
|
||||
BigQ::new(num, den)
|
||||
}
|
||||
}
|
||||
|
||||
/// `Σ n/d` over the lcm of the denominators, not yet reduced: the read reduces once.
|
||||
pub(crate) fn combine_over_lcm(terms: &[(BigInt, BigInt)]) -> (BigInt, BigInt) {
|
||||
let lcm = terms.iter().fold(BigInt::one(), |l, (_, d)| l.lcm(d));
|
||||
let num = terms
|
||||
.iter()
|
||||
.fold(BigInt::zero(), |s, (n, d)| s + n * (&lcm / d));
|
||||
(num, lcm)
|
||||
}
|
||||
|
||||
/// `10^28 / d` when `d = 2^p 5^q` with `max(p, q) <= 28`: the term joins the decimal partition.
|
||||
fn at_scale_28(d: i128) -> Option<u128> {
|
||||
let p = d.trailing_zeros();
|
||||
let mut rest = d >> p;
|
||||
let mut q = 0;
|
||||
while rest % 5 == 0 {
|
||||
rest /= 5;
|
||||
q += 1;
|
||||
}
|
||||
(rest == 1 && p.max(q) <= MAX_SCALE).then(|| (pow10(MAX_SCALE) / d) as u128)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use num_rational::BigRational;
|
||||
use proptest::prelude::*;
|
||||
use proptest::test_runner::{Config, RngAlgorithm, TestRng, TestRunner};
|
||||
|
||||
type Combine = fn(&[(BigInt, BigInt)]) -> (BigInt, BigInt);
|
||||
|
||||
/// Up to K groups of wide numerators whose denominators share factors.
|
||||
fn groups() -> impl Strategy<Value = Vec<(BigInt, BigInt)>> {
|
||||
let shared = prop::sample::select(vec![1i128, 3, 7 * 11, 1 << 20, 3i128.pow(40)]);
|
||||
let group =
|
||||
(any::<i128>(), any::<i64>(), shared, 1i128..(1 << 60)).prop_map(|(n, w, s, d)| {
|
||||
(
|
||||
BigInt::from(n) * BigInt::from(w),
|
||||
BigInt::from(s) * BigInt::from(d),
|
||||
)
|
||||
});
|
||||
prop::collection::vec(group, 1..=K)
|
||||
}
|
||||
|
||||
fn check_grouped_read(combine: Combine, deterministic: bool) -> Result<(), String> {
|
||||
let config = Config {
|
||||
cases: 256,
|
||||
failure_persistence: None,
|
||||
..Config::default()
|
||||
};
|
||||
let mut runner = match deterministic {
|
||||
true => {
|
||||
TestRunner::new_with_rng(config, TestRng::deterministic_rng(RngAlgorithm::ChaCha))
|
||||
}
|
||||
false => TestRunner::new(config),
|
||||
};
|
||||
let result = runner.run(&groups(), |terms| {
|
||||
let (num, den) = combine(&terms);
|
||||
let lcm = terms.iter().fold(BigInt::one(), |l, (_, d)| l.lcm(d));
|
||||
prop_assert_eq!(&den, &lcm, "the read did not combine over the lcm");
|
||||
let oracle: BigRational = terms
|
||||
.iter()
|
||||
.map(|(n, d)| BigRational::new(n.clone(), d.clone()))
|
||||
.sum();
|
||||
let read = BigQ::new(num, den);
|
||||
let got = BigRational::new(read.num.clone(), read.den.clone());
|
||||
prop_assert_eq!(got, oracle, "the read is not the sum");
|
||||
prop_assert!(read.num.gcd(&read.den).is_one(), "the read is not reduced");
|
||||
Ok(())
|
||||
});
|
||||
result.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_grouped_read_combines_over_the_lcm_and_reduces_once() {
|
||||
check_grouped_read(combine_over_lcm, false).unwrap();
|
||||
}
|
||||
|
||||
fn over_the_product(terms: &[(BigInt, BigInt)]) -> (BigInt, BigInt) {
|
||||
let den = terms.iter().fold(BigInt::one(), |p, (_, d)| p * d);
|
||||
let num = terms
|
||||
.iter()
|
||||
.fold(BigInt::zero(), |s, (n, d)| s + n * (&den / d));
|
||||
(num, den)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_grouped_read_combines_over_the_lcm_and_reduces_once_catches_the_product() {
|
||||
let err = check_grouped_read(over_the_product, true).expect_err("the rival passed");
|
||||
assert!(
|
||||
err.contains("the read did not combine over the lcm"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fraction_with_a_decimal_denominator_joins_the_decimal_partition() {
|
||||
let mut s = NumSum::new();
|
||||
s.insert(&Num::ratio(10i128.pow(35), 1).unwrap());
|
||||
s.insert(&Num::ratio(3, 1 << 20).unwrap());
|
||||
assert_eq!(s.groups(), 0);
|
||||
s.insert(&Num::ratio(1, 1i128 << 40).unwrap());
|
||||
s.insert(&Num::ratio(1, 3).unwrap());
|
||||
assert_eq!(
|
||||
s.groups(),
|
||||
2,
|
||||
"2^40 has scale 40, past 28: a group of its own"
|
||||
);
|
||||
s.retract(&Num::ratio(1, 3).unwrap());
|
||||
assert_eq!(
|
||||
s.groups(),
|
||||
1,
|
||||
"a group whose numerator returns to zero is dropped"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
//! The `BigRational` oracle and the generators the property tests share.
|
||||
//! Each property is a checker taking the implementation, so a rival can be run through it.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use std::cmp::Ordering;
|
||||
|
||||
use num_bigint::BigInt;
|
||||
use num_rational::BigRational;
|
||||
use num_traits::{One, Signed, ToPrimitive, Zero};
|
||||
use numtower::{Decimal, Num, NumRef};
|
||||
use proptest::prelude::*;
|
||||
use proptest::test_runner::{Config, RngAlgorithm, TestCaseError, TestRng, TestRunner};
|
||||
|
||||
pub fn q(n: &Num) -> Option<BigRational> {
|
||||
match n.kind() {
|
||||
NumRef::Exact(d) => Some(BigRational::new(
|
||||
d.mantissa().into(),
|
||||
BigInt::from(10).pow(d.scale()),
|
||||
)),
|
||||
NumRef::Fraction(r) => Some(BigRational::new((*r.numer()).into(), (*r.denom()).into())),
|
||||
NumRef::Inexact(x) => BigRational::from_float(x),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `v` fits a stored exact arm: a decimal at scale <= 28, or an `i128` fraction.
|
||||
pub fn fits(v: &BigRational) -> bool {
|
||||
let limit = BigInt::one() << 96;
|
||||
let decimal = (0..=28u32).any(|s| {
|
||||
let scaled = v * BigRational::from_integer(BigInt::from(10).pow(s));
|
||||
scaled.is_integer() && scaled.to_integer().abs() < limit
|
||||
});
|
||||
let (min, max) = (BigInt::from(i128::MIN), BigInt::from(i128::MAX));
|
||||
decimal || (&min <= v.numer() && v.numer() <= &max && v.denom() <= &max)
|
||||
}
|
||||
|
||||
/// Whether `f` is the round-to-nearest-even f64 of `v`: the review's interval form.
|
||||
pub fn is_nearest(f: f64, v: &BigRational) -> bool {
|
||||
let exact = |x: f64| BigRational::from_float(x).expect("finite");
|
||||
let two = BigRational::from_integer(BigInt::from(2));
|
||||
let half_ulp_past = |x: f64, up: bool| {
|
||||
let (next, inward) = match up {
|
||||
true => (x.next_up(), x.next_down()),
|
||||
false => (x.next_down(), x.next_up()),
|
||||
};
|
||||
match next.is_finite() {
|
||||
true => (exact(x) + exact(next)) / &two,
|
||||
false => exact(x) + (exact(x) - exact(inward)) / &two,
|
||||
}
|
||||
};
|
||||
match f.is_finite() {
|
||||
false => {
|
||||
let edge = half_ulp_past(f64::MAX.copysign(f), f > 0.0);
|
||||
(f > 0.0 && v >= &edge) || (f < 0.0 && v <= &edge)
|
||||
}
|
||||
true => {
|
||||
let lo = half_ulp_past(f, false);
|
||||
let hi = half_ulp_past(f, true);
|
||||
let even = f.to_bits() & 1 == 0;
|
||||
match even {
|
||||
true => &lo <= v && v <= &hi,
|
||||
false => &lo < v && v < &hi,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A decimal's text built digit by digit, without `rust_decimal`'s `Display`.
|
||||
pub fn decimal_text(mantissa: i128, scale: u32) -> String {
|
||||
let digits = format!(
|
||||
"{:0>width$}",
|
||||
mantissa.unsigned_abs(),
|
||||
width = scale as usize + 1
|
||||
);
|
||||
let (int, frac) = digits.split_at(digits.len() - scale as usize);
|
||||
let sign = if mantissa < 0 { "-" } else { "" };
|
||||
format!("{sign}{int}.{frac}0")
|
||||
}
|
||||
|
||||
/// The correctly rounded f64 of an exact value, independent of `numtower`'s own.
|
||||
pub fn independent_f64(n: &Num) -> f64 {
|
||||
match n.kind() {
|
||||
NumRef::Exact(d) => decimal_text(d.mantissa(), d.scale()).parse().unwrap(),
|
||||
NumRef::Fraction(_) => q(n).unwrap().to_f64().unwrap(),
|
||||
NumRef::Inexact(x) => x,
|
||||
}
|
||||
}
|
||||
|
||||
/// Same arm, same value and same scale: what a fingerprint would see.
|
||||
pub fn same(a: &Num, b: &Num) -> bool {
|
||||
match (a.kind(), b.kind()) {
|
||||
(NumRef::Exact(x), NumRef::Exact(y)) => {
|
||||
x.mantissa() == y.mantissa() && x.scale() == y.scale()
|
||||
}
|
||||
(NumRef::Fraction(x), NumRef::Fraction(y)) => x == y,
|
||||
(NumRef::Inexact(x), NumRef::Inexact(y)) => x.to_bits() == y.to_bits(),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn arm(n: &Num) -> &'static str {
|
||||
match n.kind() {
|
||||
NumRef::Exact(_) => "exact",
|
||||
NumRef::Fraction(_) => "fraction",
|
||||
NumRef::Inexact(_) => "inexact",
|
||||
}
|
||||
}
|
||||
|
||||
pub fn mantissa96() -> impl Strategy<Value = i128> {
|
||||
let max = (1i128 << 96) - 1;
|
||||
prop_oneof![
|
||||
-max..=max,
|
||||
-1_000_000i128..1_000_000,
|
||||
-(10i128.pow(15))..10i128.pow(15),
|
||||
]
|
||||
}
|
||||
|
||||
pub fn decimal() -> impl Strategy<Value = Num> {
|
||||
(mantissa96(), 0u32..=28).prop_map(|(m, s)| Num::exact(Decimal::from_i128_with_scale(m, s)))
|
||||
}
|
||||
|
||||
fn of_width(bits: u32) -> impl Strategy<Value = i128> {
|
||||
(any::<u128>(), any::<bool>()).prop_map(move |(r, neg)| {
|
||||
let v = ((r >> (128 - bits)) | (1u128 << (bits - 1))) as i128;
|
||||
if neg { -v } else { v }
|
||||
})
|
||||
}
|
||||
|
||||
pub fn fraction() -> impl Strategy<Value = Num> {
|
||||
let widths = (1u32..=126, 1u32..=126)
|
||||
.prop_flat_map(|(a, b)| (of_width(a), of_width(b)))
|
||||
.prop_map(|(n, d)| Num::ratio(n, d).unwrap());
|
||||
let edge = (1i128..i128::MAX).prop_map(|d| Num::ratio(i128::MIN, d | 1).unwrap());
|
||||
prop_oneof![8 => widths, 1 => edge]
|
||||
}
|
||||
|
||||
pub fn finite_f64() -> impl Strategy<Value = f64> {
|
||||
use proptest::num::f64::{NEGATIVE, NORMAL, POSITIVE, SUBNORMAL, ZERO};
|
||||
prop_oneof![
|
||||
(POSITIVE | NEGATIVE | NORMAL | SUBNORMAL | ZERO),
|
||||
-1e6f64..1e6,
|
||||
(-1000i64..1000).prop_map(|k| k as f64 / 8.0),
|
||||
]
|
||||
}
|
||||
|
||||
pub fn exact() -> impl Strategy<Value = Num> {
|
||||
prop_oneof![decimal(), fraction()]
|
||||
}
|
||||
|
||||
pub fn any_num() -> impl Strategy<Value = Num> {
|
||||
prop_oneof![
|
||||
3 => exact(),
|
||||
1 => finite_f64().prop_map(Num::inexact),
|
||||
1 => prop_oneof![Just(f64::NAN), Just(f64::INFINITY), Just(f64::NEG_INFINITY)].prop_map(Num::inexact),
|
||||
]
|
||||
}
|
||||
|
||||
/// Runs a property; `deterministic` for a rival, so its failure is not luck.
|
||||
pub fn run<S: Strategy>(
|
||||
strategy: S,
|
||||
cases: u32,
|
||||
deterministic: bool,
|
||||
test: impl Fn(S::Value) -> Result<(), TestCaseError>,
|
||||
) -> Result<(), String> {
|
||||
let config = Config {
|
||||
cases,
|
||||
failure_persistence: None,
|
||||
..Config::default()
|
||||
};
|
||||
let mut runner = match deterministic {
|
||||
true => TestRunner::new_with_rng(config, TestRng::deterministic_rng(RngAlgorithm::ChaCha)),
|
||||
false => TestRunner::new(config),
|
||||
};
|
||||
runner.run(&strategy, test).map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
/// A rival must fail on the property's own assertion, never on something else.
|
||||
pub fn assert_caught(result: Result<(), String>, marker: &str) {
|
||||
let err = result.expect_err("the rival passed: the property cannot see it");
|
||||
assert!(err.contains(marker), "failed for another reason: {err}");
|
||||
}
|
||||
|
||||
pub fn oracle_cmp(a: &Num, b: &Num) -> Option<Ordering> {
|
||||
Some(q(a)?.cmp(&q(b)?))
|
||||
}
|
||||
|
||||
pub fn integer(v: &BigRational) -> bool {
|
||||
v.is_integer() && !v.numer().is_zero()
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -46,6 +46,8 @@
|
||||
"improvise-wasm-client"
|
||||
"improvise-worker-server"
|
||||
"improvise-gp"
|
||||
# ADR 005 clause 14: nothing depends on it yet, and improvise-core will.
|
||||
"numtower"
|
||||
];
|
||||
pFlags = crates: lib.concatMapStringsSep " " (c: "-p ${c}") crates;
|
||||
|
||||
|
||||
@@ -34,6 +34,8 @@ const SECTIONS = [
|
||||
["docs/design-notes.md", "design-notes", "Design notes"],
|
||||
["context/design-principles.md", "design-principles", "Design principles"],
|
||||
["context/repo-map.md", "repo-map", "Repository map"],
|
||||
["context/gotchas.md", "gotchas", "Gotchas (long form)"],
|
||||
["context/file-inventory.md", "file-inventory", "File inventory (long form)"],
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -47,6 +49,8 @@ const SECTIONS = [
|
||||
["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"],
|
||||
["context/adr/005-exact-decimal-arithmetic.md", "adr/005-exact-decimal-arithmetic", "ADR 005 — Exact decimal arithmetic (proposed)"],
|
||||
["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"],
|
||||
["context/adr/007-stable-identities-amendment-1.md", "adr/007-stable-identities-amendment-1", "ADR 007, amendment 1"],
|
||||
["context/adr/009-incremental-evaluation.md", "adr/009-incremental-evaluation", "ADR 009 — Incremental evaluation (proposed)"],
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -63,6 +67,7 @@ const SECTIONS = [
|
||||
["context/code/tui.md", "code/tui", "The TUI"],
|
||||
["context/code/cli.md", "code/cli", "The headless CLI"],
|
||||
["context/code/gp.md", "code/gp", "gp-tree and improvise-gp"],
|
||||
["context/code/numtower.md", "code/numtower", "numtower, the numeric tower"],
|
||||
["context/code/kb-tool.md", "code/kb-tool", "The kb tool"],
|
||||
["context/code/tooling.md", "code/tooling", "The repository's own checkers"],
|
||||
],
|
||||
@@ -81,6 +86,7 @@ const SECTIONS = [
|
||||
["context/test-evidence/tui.md", "test-evidence/tui", "The TUI"],
|
||||
["context/test-evidence/cli.md", "test-evidence/cli", "The headless CLI"],
|
||||
["context/test-evidence/gp.md", "test-evidence/gp", "gp-tree and improvise-gp"],
|
||||
["context/test-evidence/numtower.md", "test-evidence/numtower", "numtower"],
|
||||
["context/test-evidence/tooling.md", "test-evidence/tooling", "The repository's own gates"],
|
||||
["context/test-evidence/layout-cost.md", "test-evidence/layout-cost", "Layout and keypress cost"],
|
||||
["context/test-evidence/scale-bisect.md", "test-evidence/scale-bisect", "Scale bisect"],
|
||||
@@ -610,7 +616,7 @@ for (const section of SECTIONS) {
|
||||
for (const [src, slug, title] of section.docs) {
|
||||
const depth = slug.split("/").length - 1;
|
||||
const ctx = { depth, cited, unresolved, anchors: new Set() };
|
||||
if (slug === "repo-map") ctx.listItemId = (n) => (n >= 1 && n <= 200 ? `gotcha-${n}` : null);
|
||||
if (slug === "repo-map" || slug === "gotchas") ctx.listItemId = (n) => (n >= 1 && n <= 200 ? `gotcha-${n}` : null);
|
||||
const md = readFileSync(join(root, src), "utf8");
|
||||
const { html, toc } = renderMarkdown(md, ctx);
|
||||
|
||||
|
||||
+6
-9
@@ -189,12 +189,9 @@ def read_rows(path):
|
||||
|
||||
|
||||
def gotchas(text):
|
||||
"""(number, lines, chars) for each numbered item of repo-map's Gotchas section."""
|
||||
out, cur, inside = [], None, False
|
||||
"""(number, lines, chars) for each numbered item of context/gotchas.md."""
|
||||
out, cur, inside = [], None, True
|
||||
for line in text.split("\n"):
|
||||
if line.startswith("## "):
|
||||
inside = line.startswith("## Gotchas")
|
||||
continue
|
||||
if inside and line.startswith("---"):
|
||||
inside = False
|
||||
if not inside:
|
||||
@@ -214,13 +211,13 @@ def doc_sizes(repo, sha):
|
||||
"context").split("\n") if p.endswith(".md")]
|
||||
show = lambda p: git(repo, "show", f"{sha}:{p}")
|
||||
lines = {p: len(show(p).splitlines()) for p in sorted(files)}
|
||||
gs = gotchas(show("context/repo-map.md")) if "context/repo-map.md" in files else []
|
||||
gs = gotchas(show("context/gotchas.md")) if "context/gotchas.md" in files else []
|
||||
longest = max(gs, key=lambda g: g[2], default=None)
|
||||
evidence = {p: sum(1 for l in show(p).split("\n") if re.match(r"^#{3,4} ", l))
|
||||
for p in sorted(files) if p.startswith("context/test-evidence/")}
|
||||
return {
|
||||
"context_lines": lines,
|
||||
"repo_map": {"gotchas": len(gs), "longest": longest and
|
||||
"gotchas": {"count": len(gs), "longest": longest and
|
||||
{"number": longest[0], "lines": longest[1], "chars": longest[2]}},
|
||||
"test_evidence": {"entries": sum(evidence.values()), "per_file": evidence},
|
||||
}
|
||||
@@ -270,9 +267,9 @@ def text(r, top):
|
||||
out += ["", TITLES[t]] + table(r["tables"][t])
|
||||
out += ["", "Doc sizes (lines)"]
|
||||
out += [f" {n:>6} {path}" for path, n in r["docs"]["context_lines"].items()]
|
||||
rm, te = r["docs"]["repo_map"], r["docs"]["test_evidence"]
|
||||
rm, te = r["docs"]["gotchas"], r["docs"]["test_evidence"]
|
||||
lg = rm["longest"]
|
||||
out.append(f" repo-map gotchas: {rm['gotchas']}" + (
|
||||
out.append(f" gotchas: {rm['count']}" + (
|
||||
f"; longest is {lg['number']} ({lg['lines']} lines, {lg['chars']} chars)"
|
||||
if lg else ""))
|
||||
out.append(f" test-evidence entries: {te['entries']}")
|
||||
|
||||
+10
-11
@@ -76,10 +76,9 @@ put context/code/x.md 8 x3
|
||||
mkdir -p "$repo/context/test-evidence"
|
||||
printf '### t_one\ntext\n#### t_two\n## not an entry\n### t_three\n' \
|
||||
> "$repo/context/test-evidence/y.md"
|
||||
cat > "$repo/context/repo-map.md" <<'EOF'
|
||||
# Map
|
||||
|
||||
## Gotchas (read before editing)
|
||||
printf '# Map\n' > "$repo/context/repo-map.md"
|
||||
cat > "$repo/context/gotchas.md" <<'EOF'
|
||||
# Gotchas
|
||||
|
||||
1. **One.** short.
|
||||
2. **Two.** a much longer gotcha, with plenty of words in it to be the longest one
|
||||
@@ -136,25 +135,25 @@ if run "$work/stats" text; then
|
||||
row "docs/*.md is other docs" by-area "other docs" '["2","0","0.000"]'
|
||||
row "context/code" by-area "context/code" '["8","0","0.000"]'
|
||||
row "context/test-evidence" by-area "context/test-evidence" '["5","0","0.000"]'
|
||||
row "repo-map is its own area" by-area "repo-map" '["13","0","0.000"]'
|
||||
row "other context" by-area "other context" '["2","0","0.000"]'
|
||||
row "repo-map is its own area" by-area "repo-map" '["1","0","0.000"]'
|
||||
row "other context" by-area "other context" '["13","0","0.000"]'
|
||||
row "kb.pl" by-area "kb.pl" '["5","0","0.000"]'
|
||||
row "scripts/CI takes .github, and a rename is a delete and an add" \
|
||||
by-area "scripts/CI" '["15","7","0.467"]'
|
||||
row "web and harness, and its deletion ratio" by-area "web and harness" '["14","9","0.643"]'
|
||||
row "a lock file, .beads/ and site/ are generated" by-area "generated" '["14","0","0.000"]'
|
||||
row "the pre-window README counts nowhere" by-area "other" '["4","0","0.000"]'
|
||||
row "prose is the context areas, other docs and agent docs" by-kind prose '["43","6","0.140"]'
|
||||
row "prose is the context areas, other docs and agent docs" by-kind prose '["42","6","0.143"]'
|
||||
row "code is rust, scripts/CI and web" by-kind code '["82","45","0.549"]'
|
||||
row "prose to code is prose over code" by-kind ProseToCode '["0.524","0.133",""]'
|
||||
row "prose to code is prose over code" by-kind ProseToCode '["0.512","0.133",""]'
|
||||
is "weeks are columns, first-seen first" '.tables["area-by-week"][0][1:]' '["2026-W24","2026-W23"]'
|
||||
row "an area is split by ISO week" area-by-week "web and harness" '["3","11"]'
|
||||
is "every day's churn.csv total equals code-maat's abs-churn" \
|
||||
'[.tables["by-day"][1:][]|.[1]==.[2]]|all' 'true'
|
||||
row "a day's total is every file changed that day" by-day 2026-06-04 '["72","72"]'
|
||||
is "gotchas end with their section" '.docs.repo_map.gotchas' '3'
|
||||
row "a day's total is every file changed that day" by-day 2026-06-04 '["71","71"]'
|
||||
is "gotchas end with their section" '.docs.gotchas.count' '3'
|
||||
is "the longest gotcha is measured over its lines" \
|
||||
'.docs.repo_map.longest|[.number,.lines]' '[2,2]'
|
||||
'.docs.gotchas.longest|[.number,.lines]' '[2,2]'
|
||||
is "test-evidence entries are ### and #### headings" '.docs.test_evidence.entries' '3'
|
||||
is "coupling finds the 15 pairs of six files, and web/app.js with scripts/foo.sh" \
|
||||
'.coupling|length' '16'
|
||||
|
||||
Reference in New Issue
Block a user