fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(adr): ADR 005 folds in the user's answers to questions 10 and 13-15 and answers the review of 33833f28 (improvise-l2b8)

Browse Source 
Revision 6. The user decided question 10 (a) with a 320-bit decimal
partition, question 13 (b), question 14 `numtower`, and question 15 (a).

N1: the number language's gate is the finite results of f64::from_str,
which is #321's gate; nan, inf and 1e999 stay text at import as
improvise-jc95 decided, and reversing jc95 is the rejected alternative.
N2: entry tries the fraction arm before f64, so a typed value is as exact
as the same value computed.

Also: raw_value's restructuring cost is stated; stale text is corrected;
the ceiling is sized to K*127+256; the exact grouped read is measured
(0.67-1.9 ms at K = 32, about 50x the estimate), which adds question 16
on the value of K; and the last-bit miss condition is stated on the
terms, matching ADR 009 at 4973845f.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (agent-adr005)
2026-09-25 12:08:41 -07:00
committed by Edward Langley
parent 33833f2822
commit 02bcfcb158
1 changed files with 300 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
context/adr/005-exact-decimal-arithmetic.md
+300 -86
View File
@@ -20,7 +20,7 @@ ADR 007 was first filed under this number. They belong to ADR 007, not to this d
**Citations.** Every `file:line` below is at `gh/main` `b639266b`. Line numbers are
dated, so re-derive them before citing this document elsewhere. "ADR 009" means the
head of PR #309 at `25ebf641`.
head of PR #309 at `4973845f` (revision 6; it was `25ebf641` through revision 5).
**Revision 2** answers the adversarial review of `1fe3a85d` (a BLOCK) and the four
ChatGPT Codex findings on the same head. The review's two blocking points are both
@@ -55,6 +55,23 @@ type is forced. The same revision answers the rest of that review (B2–B4, D1's
correction, question 13, the crate homes). Questions 10 and 13–15 stay open, and nothing
is renumbered.
**Revision 6** folds in **the user's answers to questions 10, 13, 14 and 15**, left
inline on `33833f28` at 17:45–17:48 UTC on 2026-09-25: the 320-bit decimal partition
and option (a) for question 10, (b) for question 13, `numtower` for question 14, and (a)
for question 15. It answers the review of `33833f28` (issuecomment-5836990563):
- **N1.** The number language's gate is now the **finite** results of `f64::from_str`,
so it is #321's gate. `nan`, `inf` and `1e999` stay text at import, as improvise-jc95
decided. Reversing jc95 is named as the rejected alternative (clause 3).
- **N2.** Entry tries the **fraction arm before f64**, so a typed value is exactly as
exact as the same value computed (clause 3).
- The should-fix items: `raw_value`'s cost to the import pipeline is stated (clause 12,
step 5); the stale text is corrected; the ceiling is sized to the grouped read; and
the exact read's cost is now **measured**. It is about 50× the revision 5 estimate,
which is why question 16 is new.
It also corrects when a last-bit miss is visible to `=`. The bound is on the **terms**,
not the total, as ADR 009 now says at `4973845f` (Codex on #309).
## What the user decided
The user answered inline on `e4acf643`, one comment per question. Each answer is quoted
@@ -132,13 +149,26 @@ Decided: **accept the skew**. The handshake version is filed as **improvise-szi8
> "Don't we need a much bigger accumulator (2000ish bits) to handle exact accumulation
> generically?"
This is a question back to the ADR, not a ruling, so question 10 stays open. It is
answered with numbers in clause 8, "How wide, generically". In short:
That was a question back to the ADR, answered with numbers in clause 8, "How wide,
generically". In short:
- **Decimals need 320 bits**, because their range is small.
- **f64 values need about 2,100 bits.** That is ADR 009's accumulator.
- **Fractions have no fixed generic width.** Combining k arbitrary `i128` denominators
needs up to 127·k bits. The user's ~2,000 bits is exactly k = 16.
Then, on `33833f28`, on the line "decimals need 320 bits":
> "320bits is fine"
and on option (a):
> "(a)"
Decided: **(a)**. A 320-bit decimal partition, a 256-bit numerator per fraction group,
and a read that combines at most K groups in a bigint rational and then applies the
stored bound. One figure changed after the answer, and it is flagged here: the exact
read was an estimate of 16–34 µs at K = 32, and it measures **0.67–1.9 ms** in the worst
case (option (d), "What that costs"). K is a constant inside `numtower`, not a format
or wire fact, so its value is split out as question 16 rather than reopening this one.
**11. The `MIN`/`MAX` tie-break.**
> "Yeah, exact."
@@ -153,6 +183,25 @@ is forced.** So a typed or imported `3/4` whose type is being guessed stays text
keeps dates. A column the user has forced to a measure reads `3/4` as a fraction.
Clause 12 defines the one number language this uses.
**13. Where the fraction bound applies.**
> "(b)"
Decided: **(b)**. The bound applies to the **stored** result, and every intermediate
inside one formula evaluation is unbounded up to the ceiling. The ceiling is the one
exception, and it is now sized to the grouped read (question 13).
**14. The numeric crate's name.**
> "this is fine"
Decided: **`numtower`** (clause 14).
**15. Where ADR 009's `ExactSum` lives.**
> "(a)"
Decided: **(a)**, in `numtower`, beside the exact decimal accumulator. ADR 009's seam
crate adapts it to `zset`'s group trait through a newtype, and `numtower` can land
holding `ExactSum` alone, ahead of this ADR (clause 14).
## The decision this ADR writes down
1. **Three levels** (decided, question 1). A number is an **exact decimal**, an **exact
@@ -160,7 +209,8 @@ Clause 12 defines the one number language this uses.
There are no unbounded rationals. They were measured, and they fail on the most
ordinary pivot-table question, the sum of ratios (clause 1).
2. **Exact is the default.** Every number a person types, every number in a `.improv`
file, every CSV or JSON number and every formula literal is exact when it fits.
file, every CSV or JSON number and every formula literal is exact when it fits
either exact arm: a decimal first, then a bounded fraction (clause 3).
3. **Exactness is lost only in named places.** `+`, `-`, `*` and `/` keep it: a
quotient is a decimal when it terminates and fits, and otherwise a bounded fraction
(decided, question 2). `POW` keeps it for an integer exponent. A non-integer exponent,
@@ -173,7 +223,7 @@ Clause 12 defines the one number language this uses.
5. **The stored type is `CellValue::Number(Num)`**, where `Num` is `Exact(Decimal)`,
`Fraction(Box<Ratio<i128>>)` or `Inexact(f64)`. The crates are `rust_decimal` and
`num-rational`. `Num` and everything numeric live in a
problem-agnostic workspace crate of their own, proposed as `numtower` (clause 14).
problem-agnostic workspace crate of their own, `numtower` (decided, question 14; clause 14).
6. **Each kind of number has its own exact accumulator.** ADR 009's f64 accumulator
makes `SUM` over *inexact* values order-free, and this ADR does not re-specify it.
The *exact* values get a wide fixed-point integer of their own (clause 8), because
@@ -260,8 +310,9 @@ bug (improvise-shyt), fixed by pinning iteration order rather than by removing i
**The epsilon is a patch, and it does not scale.** It was chosen because it "absorbs
accumulated f64 rounding noise" (`context/code/core.md` §4). But one ulp of 1,000,000 is
1.16e-10. So above about 2¹⁹ (524,288), a one-ulp error is already larger than the
epsilon, and `=` sees it. Below about 1e-10, the epsilon calls different numbers equal.
1.16e-10. So once a value is above about 2¹⁹ (524,288), a one-ulp error in it is
already larger than the epsilon, and `=` sees it. Below about 1e-10, the epsilon calls
different numbers equal.
### What ADR 009's exact accumulator fixes, and what it does not
@@ -305,12 +356,28 @@ corrected here. For two summands a superaccumulator is exactly `+`, because one
addition is already correctly rounded. So `0.1 + 0.2` is still `0.30000000000000004`
under the accumulator.
**The misses are in the last bit, so no display format shows them.** A person reading a
three-line subtotal at two places sees the right number. The miss shows in two places:
in `=` against a typed value, once the epsilon is gone or once the magnitude is above
2¹⁹, and in the editor, which opens on the shortest round-trip spelling
(`0.30000000000000004`). That is where 16% of positive 3-term sums, and about 40% of
signed sums of any length, go wrong.
**The misses are in the last bit of the largest term, not of the total.** Each stored
term is off from its typed value by at most half an ulp of that term. So the exact sum of
the stored terms is off from the typed total by at most about ½·Σ ulp(term), plus the
total's own half-ulp at the final rounding (revision 6; revision 5 put the condition on
the total, and ADR 009 made the same correction at `4973845f`, from ChatGPT Codex on
#309). When terms cancel, a miss can be far larger than the total's ulp:
- **The large-term, small-total case.** `1000000000000000.10 + (-1000000000000000.00)`
has a typed total of 0.10. The first term is stored as `1e15 + 0.125`, because the ulp
at 1e15 is 0.125. So the exact sum of the stored terms is **0.125**, and the miss is
0.025 on a total of 0.1.
- **When `=` sees one today.** `=` is an absolute `1e-10` (gotcha 17), so a miss can
reach it only when ½·Σ ulp(term) reaches `1e-10`. That needs at least one term of
about 2¹⁹ ≈ 5.2e5 or larger, or many terms just below it. It does not depend on the
total. The ±1,000 values are far below 2¹⁹, so today's epsilon hides their misses;
the positive values up to 999,999.99 pass it.
- **Once this ADR deletes the epsilon** (clause 6), every miss is visible to `=`.
A person reading a subtotal at two places usually sees the right number, because the
miss is below the displayed digits unless the terms are large. The miss also shows in
the editor, which opens on the shortest round-trip spelling (`0.30000000000000004`).
That is where 16% of positive 3-term sums, and about 40% of signed sums of any length,
go wrong.
So the division of labour is:
@@ -328,7 +395,7 @@ So the division of labour is:
summed, it is summed by ADR 009's accumulator, and this ADR only says so. ADR 009's
"what each exact domain covers" table says that accumulator makes `=` against a typed
decimal right "only for sums". The measurement above narrows that to *large same-sign*
sums. ADR 009's head `8ebcf889` now says "no, in general" and cites both distributions,
sums. ADR 009 says "no, in general" since `8ebcf889`, and cites both distributions,
so the two tables agree.
### The crates
@@ -419,8 +486,12 @@ stays, as the explicitly lossy accessor for the seams that want a binary float (
`Expr::Number(f64)` becomes **`Expr::Number(Num)`**, not `Expr::Number(Decimal)`. The
formula grammar has no digit bound, so a literal such as
`999999999999999999999999999999` is legal formula text and has no `Decimal`. It takes
clause 3's fallback, like every other spelled number: exact when it fits, else the
nearest f64. Bounding the grammar instead was rejected, because it would make a formula
clause 3's order, like every other spelled number: a decimal if it fits, else a fraction
if it fits (that literal is below 2¹²⁷, so it is the exact `Fraction` n/1), else the
nearest f64. A literal is a number by the grammar, so clause 3's gate, which decides
number-or-text for data, does not apply to it: a literal that overflows f64 (`1e999`)
keeps today's meaning, inexact infinity, the same as an inexact overflow in `*`.
Bounding the grammar instead was rejected, because it would make a formula
that parses today stop parsing, over a limit of the representation rather than of the
language. (Codex, P2 on `1fe3a85d`.)
@@ -435,16 +506,16 @@ values.
decimals a person typed, and "decimal math by default" is the user's instruction. So
Scheme's `#e` is improvise's default.
A spelled number is **inexact** only when it cannot be exact:
- it is `inf`, `-inf` or `nan` (clause 7);
- it has more than 28 significant digits, or its magnitude is outside `Decimal`'s range
(for example `1e300` or `1e-40`). Then it reads as the nearest f64.
A spelled number is **inexact** only when it cannot be exact: it fits neither a
`Decimal` nor an `i128` fraction (for example `1e300`, `1e-40`, or a 40-digit integer).
Then it reads as the nearest f64. The tokens `inf`, `-inf` and `nan` are inexact too,
but they are clause 7's tokens, not numbers of the language (below).
**The parse must refuse, not round.** `Decimal::from_str` rounds a 30-digit input to 28
digits **silently** (`0.12345678901234567890123456789` comes back as
`0.1234567890123456789012345679`). That is a rounded value calling itself exact, the one
outcome clause 4 forbids.
outcome clause 4 forbids. Under the order below, that input is the exact fraction
`12345678901234567890123456789/10²⁹` instead.
**One number language, for every entry point** (revision 5, answering B4 of the review
of `87ae21ca`). Revision 4 said "parse with `from_str_exact`", but the gate that decides
@@ -453,21 +524,43 @@ whether a string is a number at all accepts different strings. `from_str_exact`
language is defined once, and the gate and the parse cannot disagree. `numtower` owns
it.
- **What is a number.** The gate is exactly the strings Rust's `f64::from_str`
accepts, which is #321's `number_in` gate. That is an optional sign; digits with an
optional fraction (`1.`, `.5`); an optional exponent `e`/`E` with its own sign; and
the tokens `inf`, `infinity` and `nan` in any case. **`1_000` is not a number, so it
is text.**
- **How it parses.** A finite lexeme is parsed **exactly** from its digits and exponent:
a mantissa and a scale shift, never through an f64.
- If the exact value fits a `Decimal`, it is `Exact`, with the scale the lexeme
implies: `1.50` has scale 2, `1.5e3` is `1500` at scale 0, and `1.5e-3` is
`0.0015`.
- If it does not fit, it is the nearest f64 (`f64::from_str`, which is correctly
rounded): `Inexact`.
- **What is a number.** The gate is the strings Rust's `f64::from_str` accepts **and
reads as a finite value** (revision 6, N1 of the review of `33833f28`). That is #321's
`number_in` gate, which drops anything `Number::from_f64` cannot hold. So a number is
an optional sign; digits with an optional fraction (`1.`, `.5`); and an optional
exponent `e`/`E` with its own sign, whose value does not overflow f64.
- **`1_000` is not a number**, so it is text.
- **`nan`, `inf`, `infinity` in any case, and a finite lexeme that overflows f64**
(`1e999`, `-1e999`) are not numbers either, so they are text. That is
improvise-jc95's decision, which #321's
`a_nonfinite_numeric_token_stays_text_rather_than_becoming_zero` pins for `NaN`,
`inf`, `-inf`, `1e999` and `-1e999`. A finite lexeme must never read as infinity.
- A lexeme that **underflows** (`1e-999`) passes, because f64 reads it as a finite
zero. It is then too small for either exact arm, and it reads as that zero,
inexact. #321 treats it the same way.
- **The non-finite values are clause 7's tokens, not numbers.** `typed()`, the `.improv`
reader and the wire accept exactly `inf`, `-inf` and `nan`, the three spellings
`nonfinite_token` writes (`cell.rs:65`), as inexact values. Import does not, so a
CSV `nan` stays text, as jc95 decided. This is a behaviour change for `typed()`, and
it is stated: today it hands any string to `str::parse::<f64>` (`resolve.rs:1438`), so
a typed `Infinity` or `1e999` becomes infinity and a typed `NaN` becomes NaN. Under
this clause each is text.
- **How it parses.** A lexeme that passes the gate is parsed **exactly** from its digits
and exponent: a mantissa and a scale shift, never through an f64. Then it takes the
first arm it fits, in this order (revision 6, N2 of the same review):
- **A decimal.** If the exact value fits a `Decimal`, it is `Exact`, with the scale
the lexeme implies: `1.50` has scale 2, `1.5e3` is `1500` at scale 0, and `1.5e-3`
is `0.0015`.
- **A fraction.** Otherwise, if its reduced value `m / 10^k` fits `Ratio<i128>`, it is
a `Fraction`. So a typed `1e-30` is the exact `1/10³⁰`, and an integer between
`Decimal`'s maximum (about 7.9 × 10²⁸) and 2¹²⁷ is the exact `n/1`. This is the
order clause 4 already gives arithmetic, so the same value is equally exact typed
and computed: `1e-15 * 1e-15` was already the exact `1/10³⁰`. A fraction carries
no scale (the option (d) section).
- **The nearest f64.** Otherwise it is `f64::from_str`'s correctly rounded value,
`Inexact`. The gate has already made it finite.
- A lexeme that passes the gate is never refused. **Failing to fit falls back; failing
to parse cannot happen.**
- The non-finite tokens are inexact (clause 7).
- **Zero has no sign at the exact level.** `-0.0` reads as exact `0.0`. The sign of zero
is an f64 artefact. This is a behaviour change, and it is stated: an f64 `-0.0` never
survives entry as exact.
@@ -482,14 +575,22 @@ it.
deciding a column's kind. That keeps `3/4` as a date or a label.
The entry points are `typed()`, the `.improv` reader (whose `number` rule is a subset of
the gate), formula literals, the wire and import.
the gate before the finite check), formula literals, the wire and import.
**Reversing jc95 was rejected.** The alternative was a gate of every string
`f64::from_str` accepts, so that a CSV `nan` imports as `Inexact(NaN)` and a CSV `1e999`
as `Inexact(inf)`. It would make a finite lexeme read silently as infinity, which is the
invented-value class jc95 closed, and it would flip #321's test. The gain, a measure
column that can hold a NaN, is not worth either: a column carrying `nan` is shown as
text in the wizard, where the user can see it (jc95's close reason).
**A JSON number is read from its source lexeme, never through its f64** (decided,
question 6: "the round trip through f64 seems dangerous"). `serde_json` normally parses
a number to f64 before improvise sees it. The import therefore reads each record's
values with `serde_json`'s `raw_value` feature, which hands over the number's exact
source text, and parses that text by the language above. So a JSON `12.50` keeps its
scale, and a 20-digit JSON literal stays exact. `arbitrary_precision` was the other way
scale, and a 20-digit JSON literal stays exact. That costs a restructuring of the import
pipeline, stated in clause 12. `arbitrary_precision` was the other way
to do this, and it stays rejected, because it breaks serde's buffered deserialization of
numbers (the user's answer to question 6). Revision 4's shortest-spelling recovery of a
JSON number is withdrawn.
@@ -680,7 +781,8 @@ well (Consequences).
An exact number is always finite. `inf`, `-inf` and `nan` are inexact, as in Scheme
(`+inf.0`, `+nan.0`). They keep their one spelling (improvise-55v2) in the file, on the
wire, and in `typed()`. Where they come from is unchanged: stored data, and inexact
wire, and in `typed()`: `inf`, `-inf` and `nan`, and no other (clause 3). Import keeps
them as text (improvise-jc95). Where they come from is unchanged: stored data, and inexact
overflow in `*` or `POW`. Exact overflow becomes inexact (clause 4), and it may overflow
further to `inf` only at the inexact level.
@@ -759,7 +861,9 @@ each group is an integer:
- a group's numerator is at most `i128` × an `i64` weight × 2⁶⁴ terms, so **256 bits**
per group;
- the only unbounded step is the **read**, which combines the groups over their lcm;
- with at most K = 32 groups that lcm is at most about 4,100 bits, or 2,030 at K = 16.
- with at most K = 32 groups that lcm is at most about 4,100 bits, or 2,030 at K = 16,
and the combined numerator at most K·127 + 256 ≈ 4,320 bits (measured: 3,924 and
4,053 bits for 32 random 126-bit denominators and 250-bit numerators).
Under question 13 (b) that read runs in a bigint rational (`num-bigint`), so it needs no
fixed register at all. Its result is then held to the stored bound: it is exact if it
@@ -825,9 +929,12 @@ use the crate's `Hash`. It hashes the representation explicitly.
### 10. `.improv` persistence and migration
**Reading.** Every `number` token reads as clause 3 says: exact if it fits, else inexact.
The grammar's `number` rule (`improv.pest:110-112`) already accepts everything a decimal
needs, including an exponent.
**Reading.** Every `number` token reads as clause 3 says: a decimal if it fits, else a
fraction if it fits, else the nearest f64. The grammar's `number` rule
(`improv.pest:110-112`) already accepts everything a decimal needs, including an
exponent. A `number` token that overflows f64 (`1e999`, which only a hand-edited file
can hold) is text, as a CSV field is (clause 3). `inf`, `-inf` and `nan` read through
the `bare_value` arm as clause 7's tokens, as they do today (`parse.rs:268-273`).
**Writing.** An exact decimal is written as its decimal, with its scale. An exact
fraction is written `n/d`, with the sign on `n` (`-10/3`), through the grammar's
@@ -924,11 +1031,12 @@ aligns the contract with PR #321.
- **CSV.** PR #321 keeps every non-empty CSV field as its verbatim text
(`Value::String`, carried as `Scalars::Text`). It does so to preserve numeric-looking
category identifiers. So a CSV measure is parsed **from that text** by clause 3's
number language. #321's `number_in` gate *is* that language's gate, so the analyzer and
the parse agree on every string: `1.5e3` is the exact `1500`, and `1_000` is text. It
keeps its significant digits and its **scale**, so a CSV `12.50` opens in the editor
as `12.50`. It falls back to the nearest f64 only when it does not fit a `Decimal`,
never because it failed to parse.
number language. #321's `number_in` gate *is* that language's gate, finite values only
(revision 6), so the analyzer and the parse agree on every string: `1.5e3` is the
exact `1500`, and `1_000`, `nan`, `inf` and `1e999` are text. It keeps its significant
digits and its **scale**, so a CSV `12.50` opens in the editor as `12.50`. It falls
back to the nearest f64 only when it fits neither exact arm, never because it failed
to parse.
- **Inferred and forced columns differ on `n/d`** (decided, question 12). While the
analyzer is inferring a column's kind, `3/4` is text, so a column of dates stays a
column of dates. Once the user **forces** a column to a measure (a wizard override,
@@ -942,6 +1050,18 @@ aligns the contract with PR #321.
`raw_value`, and never through the f64 `as_f64` (`wizard.rs:425`) reads today
(decided, question 6; clause 3). So a JSON `12.50` keeps its scale, and a 19-digit
integer stays exact. Revision 4's shortest-spelling rule for JSON is withdrawn.
- **This restructures the JSON half of the import pipeline** (revision 6, from the
review of `33833f28`). `ImportPipeline` carries `serde_json::Value` (ten uses in
`wizard.rs`, and more in `analyzer.rs`), and a `Value`'s numbers are already f64.
`raw_value` hands over a lexeme only to a reader that deserializes into `RawValue`
through a hand-written visitor. It does not work through a `Value`, or through any
buffered path such as an untagged enum. So the JSON reader becomes a hand-written
deserializer that keeps each number as its source text, and the pipeline carries
that text to the number language, as #321 already does for CSV.
- That is the restructuring revision 3 cited as a reason not to carry the text. It is
paid here because the user ruled out the round trip through f64 (question 6), and
this is the only way to honour that for JSON. It is migration step 5's largest
part.
The collision "sum" policy (`wizard.rs:307`) then sums exactly, by clause 8.
@@ -957,7 +1077,7 @@ stays improvise's and does not become the crate's.
### 13. The crate: `rust_decimal`
Chosen on the table above. Four reasons:
Chosen on the table above. Three reasons:
- **Size and speed.** It is `Copy` and 16 bytes, with no heap. So a `CellValue` does
not start allocating for every number. It is about 2.5× faster than `bigdecimal` on
@@ -967,12 +1087,15 @@ Chosen on the table above. Four reasons:
significant digits holds any money amount and any count. `bigdecimal`'s unbounded
add costs a heap allocation for a precision no pivot table uses. Its division still
rounds, so it does not buy exact division either.
- **Serde.** Its serializer is a decimal string by default, which is clause 11's wire
spelling.
Its serde support is not a reason. `numtower` has no serde (clause 14), and the wire
spells a value in its own encoding (clause 11). Revision 5 still listed it, and that
contradicted both.
The cost is three duties the crate does not do for us, each named where it applies:
detecting rounding at the bound (clause 4), refusing an over-long parse with
`from_str_exact` (clause 3), and converting to f64 by a correctly rounded path rather
detecting rounding at the bound (clause 4), parsing (clause 3 parses the digits and
exponent itself, because `from_str` rounds silently and `from_str_exact` rejects
`1.5e3`), and converting to f64 by a correctly rounded path rather
than `to_f64` (clause 4). A fourth is avoided rather than done: its serde deserializer
accepts JSON numbers, so the wire does not use it. The wire parses its own string
encoding through `numtower` (clause 11).
@@ -986,8 +1109,8 @@ At the user's direction (revision 4): "I think ADR 005 should implement the math
operations in a separate crate". The numbers are a **problem-agnostic workspace crate**,
like `gp-tree` and ADR 009's `zset`, and it knows nothing about improvise.
**Name.** Proposed: **`numtower`**. It names the idea, Scheme's numeric tower, rather
than this program. That is question 14.
**Name.** **`numtower`** (decided, question 14). It names the idea, Scheme's numeric
tower, rather than this program.
**What it holds:**
- `Num`, with its `Exact(Decimal)`, `Fraction` (question 2, decided (d)) and `Inexact(f64)`
@@ -1004,8 +1127,8 @@ than this program. That is question 14.
(clause 12, "Format", and improvise-felo.9);
- the **f64 → decimal shortest-spelling** conversion, which the display rounding and the
`#i` spelling on the wire and in the file all use;
- parsing that refuses rather than rounds (`from_str_exact` and the f64 fallback,
clause 3);
- parsing that never rounds silently: digits and exponent to an exact value, then the
decimal, fraction and f64 arms in that order (clause 3);
- the canonical text of a value: the decimal with its scale, `n/d`, `#i…`, and the
non-finite tokens;
- clause 3's **number language**, both its gate and its parse, including the forced
@@ -1073,8 +1196,8 @@ normal ones. That is what lets it be committed at all (appendix).
**ADR 009's `ExactSum`** is an f64 superaccumulator, in the proposed `zset` crate.
Clause 8 needs it for the inexact partition, and a mixed sum's read combines it with the
exact partition. The recommendation (question 15) is that **`ExactSum` moves into
`numtower`**, the one home for exact summation. `zset` stays independent of numbers: it
exact partition. **`ExactSum` moves into `numtower`** (decided, question 15 (a)), the
one home for exact summation. `zset` stays independent of numbers: it
keeps its abelian-group trait, and ADR 009's seam crate (`improvise-incremental`, its
question 2) wraps `numtower`'s accumulators in a newtype that implements that trait. The
orphan rule forbids implementing `zset`'s trait for `numtower`'s type anywhere else, and
@@ -1186,10 +1309,12 @@ There are three ways to sum one:
term touches only its own group. So the state is independent of order, and
retraction is exact.
At read, the groups are combined in ascending order of denominator with checked
`i128` arithmetic. If that fits, the sum is exact. If it overflows, the sum is the
inexact total: each group's correctly rounded f64, summed in ADR 009's accumulator.
How it is read is the recommendation below: exactly, in a bigint rational, when
there are at most K non-decimal groups (question 13 (b)); otherwise from a maintained
f64 accumulator of the groups' values, with the decimal partition converted once.
Both paths are deterministic and independent of the order the terms arrived in.
(Revision 5 still described a checked-`i128` read here, with a fallback that summed
each group's f64 at read. That contradicted the maintained accumulator below.)
The cost is memory. It is O(distinct denominators) per summed group, bounded by
the cells, the same order as ADR 009's multisets.
@@ -1233,11 +1358,29 @@ integer and reads in O(1). The review proposed a mitigation, and it is adopted:
numerator over an `i128` denominator, estimated at about 1 µs through a bigint
rational. Plus two accumulator updates at 2.0–3.6 ns (ADR 009's figures). An update
of the decimal partition is one 320-bit add.
- **Exact read (an estimate).** At most 32 combinations. The groups hold 256-bit
numerators, and under question 13 (b) the read runs in a **bigint rational**. At the
measured `BigRational` rate of 490–1,060 ns per add (the SUM benchmark), that is about
**16–34 µs**. Revision 4's "at most about 15 µs" assumed `Ratio<i128>` operands, which
the groups are not.
- **Exact read (measured in revision 6).** At most K combinations of groups that hold
256-bit numerators, in a **bigint rational** (question 13 (b)). The worst case is K
distinct denominators near `i128`'s size, where the lcm grows about 127 bits per
group. Measured with 250-bit numerators and random 126-bit denominators, 200 reads
each (appendix):
| K | lcm bits | `BigRational` add per group | lcm, then one reduce |
|---|---|---|---|
| 2 | 252 | 16 µs | 7 µs |
| 8 | 988 | 127 µs | 52 µs |
| 16 | 1,957 | 448 µs | 174 µs |
| 32 | 3,924 | 1.9 ms | 0.67 ms |
- Revision 5 estimated **16–34 µs** at K = 32 from the SUM benchmark's rate. Those
operands had small typed denominators, so the estimate was about **50× too low** at
this size. The cost grows roughly as K², because each step works on an lcm that
grows linearly.
- **Combining over the lcm and reducing once** is 2.3–2.8× faster than reducing after
every add, and it gives the same value (the benchmark asserts it). `numtower`'s read
should do that.
- Denominators a person produces (`/ 3`, `/ 12`, a percentage) are small, and the read
is then far cheaper. The table is the bound, not the common case. The value of K is
question 16.
- **f64 read:** ADR 009's 0.6–1.1 µs.
- **Memory is still O(distinct denominators)**, because the map must survive
retraction to know when the count falls back under K. For a 90k-record margin sum
@@ -1248,8 +1391,9 @@ integer and reads in O(1). The review proposed a mitigation, and it is adopted:
This keeps (d)'s wins, because `SUM(v / 3)`, averages and percentages have one or a
handful of denominators, and it bounds the rest. The sum of margins reads the f64
accumulator and is inexact, as under (a). Its exact combination would overflow by the
6th term anyway (below). The 32 is a judgement: large enough for a `/ N` over a few
different Ns, and small enough to bound the read. It is not tuned.
6th term anyway (below). The 32 was a judgement: large enough for a `/ N` over a few
different Ns, and small enough to bound the read. It was not tuned, and the read's
measured cost above is why question 16 asks for a smaller one.
### The file and the wire
@@ -1343,6 +1487,8 @@ the user answered question 2 "(d)":
- **Clause 1:** three levels: exact decimal, exact bounded fraction, inexact.
- **Clause 2:** `Num` gains the boxed `Fraction` arm.
- **Clause 3:** entry tries the fraction arm before f64, as arithmetic does
(revision 6).
- **Clause 4:** a decimal operation that exceeds 28 digits tries the fraction arm
before f64.
- **Clause 5:** a non-terminating quotient is a `Fraction`. `AVG` is exact.
@@ -1382,8 +1528,8 @@ In order. Each step lands on its own, with the suite green.
detection. It holds the correctly rounded conversion to f64, the division and `POW`
rules (clause 5), exact comparison (clause 6), the exact-sum accumulator (clause 8),
rounding to places, and clause 3's number language. It also holds the fraction arm and
its grouped sums (question 2 is decided (d)), and the widening rule if question 13
chooses unbounded intermediates. Nothing in improvise depends on it yet, so it lands
its grouped sums (question 2 is decided (d)), and the widening rule with its 8,192-bit
ceiling (question 13 is decided (b)). Nothing in improvise depends on it yet, so it lands
with its own tests and changes no behaviour. It may be preceded by a `numtower`
holding only `ExactSum`, for ADR 009 (clause 14).
@@ -1402,8 +1548,18 @@ In order. Each step lands on its own, with the suite green.
- division's direct fit test: `1/2^40` and `1/5^30` are fractions, `1e-28 / 0.5` is
the exact decimal `2e-28`, and multiplying back is not used (clause 5);
- the number language's gate and parse agree on every generated lexeme. The seed
cases are `1.5e3` (exact `1500`), `1_000` (text), `-0.0` (exact `0.0`), and `3/4`
(text when inferred, a fraction when forced).
cases are:
- `1.5e3` (exact `1500`), `1_000` (text), `-0.0` (exact `0.0`), and `3/4` (text
when inferred, a fraction when forced);
- `1e999` and `-1e999` (text: a finite lexeme never reads as infinity), `nan`,
`inf` and `Infinity` (text in the gate), and `1e-999` (an inexact zero);
- `1e-30` (the exact fraction `1/10³⁰`), `0.12345678901234567890123456789` (an
exact fraction, not a rounded decimal), an integer between `Decimal`'s maximum
and 2¹²⁷ (an exact `n/1`), and `1e-40` (inexact: it fits neither arm);
- a typed value and the same value computed are equally exact: `1e-30` parsed
equals `1e-15 * 1e-15`, arm and value both;
- the grouped read combines over the lcm and reduces once, and agrees with a
`BigRational` sum of the groups.
The differential tests run against a `BigRational` oracle. The benchmark goes in
`numtower/examples/`. Add the crate to the `wasm` CI job. Measure one whole recompute
@@ -1423,10 +1579,17 @@ In order. Each step lands on its own, with the suite green.
`[0.30000000000000004, 1e12, −1e12]` through the evaluator. `Decimal`'s own `+` must
fail it.
5. **Entry points:** `typed()`, the import pipeline's measure read, formula literals
(clauses 3 and 12). The import tests assert:
(clauses 3 and 12). The largest part is the JSON reader: a hand-written deserializer
that keeps each number's source text through `raw_value`, because a
`serde_json::Value` has already made it an f64 (clause 12). The import tests assert:
- a CSV `12.50` and a JSON `12.50` both keep their scale, because both are parsed
from their lexeme;
- `1.5e3` imports as the exact `1500`, and `1_000` imports as text;
- `nan`, `inf`, `1e999` and `-1e999` import as text, so #321's
`a_nonfinite_numeric_token_stays_text_rather_than_becoming_zero` stays green
unchanged;
- `typed()` reads `inf`, `-inf` and `nan` as clause 7's tokens, and `Infinity`,
`NaN` and `1e999` as text (the stated behaviour change);
- `3/4` is text in an inferred column and the fraction `3/4` in a column forced to
a measure.
6. **Display and edit text** (clause 9). The `1.005` case is the regression test, and
@@ -1439,9 +1602,10 @@ In order. Each step lands on its own, with the suite green.
## Open questions
The user answered questions 1–9, 11 and 12 inline on `e4acf643`. Each is marked
**decided** below, and quoted in full under "What the user decided". Questions 10, 13, 14
and 15 are open. Nothing is renumbered.
The user answered questions 1–9, 11 and 12 inline on `e4acf643`, and 10, 13, 14 and
15 inline on `33833f28`. Each is marked **decided** below, and quoted in full under
"What the user decided". Question 16 is new in revision 6, and it is the only open one.
Nothing is renumbered.
1. **The tower's shape.** **Decided (the user): three levels**: exact decimal, exact
bounded fraction, inexact f64 (clause 1). The original recommendation, two levels,
@@ -1483,7 +1647,10 @@ and 15 are open. Nothing is renumbered.
9. **Version skew on the wire.** **Decided (the user): accept it**, and the handshake
version is filed as **improvise-szi8** (clause 11).
10. **The exact-sum accumulator's width** (revised in revision 5, after the user asked:
10. **The exact-sum accumulator's width.** **Decided (the user): (a)**, with "320bits
is fine" for the decimal partition. The value of K is split out as question 16,
because the read's cost was measured after the answer. The question as it was put
(revised in revision 5, after the user asked:
"Don't we need a much bigger accumulator (2000ish bits) to handle exact accumulation
generically?"). Clause 8's "How wide, generically" answers with numbers:
- **decimals need 320 bits**;
@@ -1504,7 +1671,8 @@ and 15 are open. Nothing is renumbered.
- 2,048 bits pays 256 bytes per summed group for a combination that rarely fits the
stored bound anyway.
- The bigint read is paid only when a sum holds 2 to K distinct fraction
denominators: about 16–34 µs at K = 32, an estimate.
denominators: about 16–34 µs at K = 32, an estimate. (Revision 6 measured it:
0.67–1.9 ms at K = 32 in the worst case. See question 16.)
- (c) breaks order-independence (revision 2).
11. **The `MIN`/`MAX` tie-break.** **Decided (the user): prefer the exact value**
@@ -1518,7 +1686,8 @@ and 15 are open. Nothing is renumbered.
(`resolve.rs:1421`); the TUI must be held to the same rule (the improvise-umi4
lesson).
13. **Where the fraction bound applies** (open; added in revision 4). Is the `i128`
13. **Where the fraction bound applies.** **Decided (the user): (b)**. Added in
revision 4, and put as follows. Is the `i128`
bound checked (a) after every operation, or (b) only on the **stored** result, with
every intermediate inside one formula evaluation unbounded? Under (b) an
intermediate uses the `i128` fast path and widens to a bigint rational only when a
@@ -1534,8 +1703,13 @@ and 15 are open. Nothing is renumbered.
**The rule has one exception, and it is stated rather than hedged** (revision 5,
the review of `87ae21ca`):
- **The ceiling.** Widening carries a size ceiling, for example 4,096 bits, so that a
pathological expression cannot grow a bigint without limit. Past it, the
- **The ceiling.** Widening carries a size ceiling so that a pathological
expression cannot grow a bigint without limit. It must be at least
K·127 + 256 bits, so that the grouped read (below) never reaches it: 4,320 bits
at K = 32. Revision 5's example of 4,096 bits was below that, and a worst-case
K = 32 read would have hit it (the review of `33833f28`). Proposed: **8,192
bits**, which covers K = 32 with room, and costs nothing until a value is that
large. Past it, the
intermediate demotes, and by contagion the stored result is inexact **even when
the exact result would fit**. That is spelling-dependent, so distributivity fails
past the ceiling. It is the one exception to "exact iff it fits", and no ledger
@@ -1553,15 +1727,18 @@ and 15 are open. Nothing is renumbered.
**Cost.** This is an estimate, not measured on this path.
- The common case is unchanged: no intermediate overflows `i128`.
- Only a cell whose expression overflows `i128` part-way through pays for bigint
rational steps. `BigRational` addition measured 490–1,060 ns per add, so about a
microsecond per widened step.
rational steps. A widened step works on operands near 256 bits, not on the SUM
benchmark's small typed values: combining two such fractions measured 7–16 µs
(option (d), "What that costs"). So a widened step costs microseconds, not the
490–1,060 ns revision 5 cited.
- Such a cell usually demotes at the store anyway.
14. **The numeric crate's name** (open; added in revision 4). *Recommendation:*
`numtower`. It is problem-agnostic like `gp-tree` and `zset`, so it should not carry
14. **The numeric crate's name.** **Decided (the user): `numtower`** ("this is
fine"). Added in revision 4, with the recommendation `numtower`. It is problem-agnostic like `gp-tree` and `zset`, so it should not carry
improvise's name.
15. **Where ADR 009's `ExactSum` lives** (open; added in revision 4).
15. **Where ADR 009's `ExactSum` lives.** **Decided (the user): (a)**. Added in
revision 4, and put as follows.
- (a) In `numtower`, beside the exact decimal accumulator, with ADR 009's seam crate
adapting it to `zset`'s group trait through a newtype.
- (b) In `zset`, as ADR 009 proposes, with `numtower` depending on `zset` to
@@ -1572,6 +1749,27 @@ and 15 are open. Nothing is renumbered.
`ExactSum` alone, ahead of this ADR, so ADR 009 does not wait for it (clause 14).
This question is shared with ADR 009 (PR #309).
16. **The value of K** (open; added in revision 6). K is the most non-decimal groups a
`SUM` reads exactly (question 10). Above it the read is the inexact f64
accumulator. Revision 5 proposed K = 32 on an estimate of 16–34 µs per read, and
the measured worst case is **0.67–1.9 ms** (option (d), "What that costs"). Under
ADR 009 a maintained `SUM` reads after every edit that touches it, so that is a
per-edit cost, paid by each such cell.
- **(a)** K = 32, as decided with question 10: worst case 0.67–1.9 ms per read.
- **(b)** K = 8: worst case 52–127 µs per read. A `SUM` over more than 8 distinct
denominators is inexact.
- **(c)** K = 16: worst case 174–448 µs per read.
*Recommendation:* **(b), K = 8**, with the read combining over the lcm and reducing
once (the lower figure in each range).
- The ceiling (question 13) stays at 8,192 bits under any of the three.
- The common sums keep exactness under K = 8: `SUM(v / 3)` has one denominator, and
an average or a percentage has a handful.
- The sum of margins has one denominator per record, so it passes any of these Ks
long before it is large, and reads inexact under each of them.
- K is a constant inside `numtower`, not a format or wire fact, so it can be tuned
later against the evaluator benchmark (migration step 1) without a new ADR.
## Consequences
- Typed values and their `+ - *` are exact. `0.1 + 0.2 = 0.3`, `1234.56 - 1234.55 =
@@ -1616,6 +1814,13 @@ and 15 are open. Nothing is renumbered.
language are shared.
- `n/d` is text wherever a type is inferred and a fraction wherever it is forced
(decided, question 12).
- `nan`, `inf` and a finite lexeme that overflows f64 (`1e999`) are text at import, as
improvise-jc95 decided. `typed()` narrows to the three canonical tokens, so a typed
`Infinity`, `NaN` or `1e999` becomes text where today it is a number (clause 3).
- A value typed past `Decimal`'s range is an exact fraction when it fits `i128`
(`1e-30`, a 30-digit integer), as the same value computed already is (clause 3).
- A `SUM` reads exactly over at most K distinct fraction denominators, at a worst-case
cost that grows about as K² (question 16), and past K it is inexact.
- Formulas gain `EQUAL(a, b, epsilon)`, and `EQUAL` becomes a keyword the writer quotes.
- **Costs:** `Num` is 24 bytes where an `f64` is 8, but **`CellValue` stays 32 bytes**
under both (a) and (d), because its `String` arm already sets the size (the review
@@ -1698,3 +1903,12 @@ The `to_f64` check compared `Ratio<i128>::to_f64` with `BigRational::to_f64` on
fractions (seed 5). That comparison is **circular**, because the two share one routine,
so the ADR now relies on the review's independent interval check instead (option (d),
"Operations"). Every figure is the range over two runs on the machine above.
**The grouped read** (revision 6), in the same crate. For K = 2, 8, 16 and 32, 200 reads
each. A read combines K groups whose numerators are random 250-bit integers and whose
denominators are random odd 126-bit integers (xorshift64, one fixed seed), sorted
ascending by denominator. Two methods are timed on the same groups: `BigRational`
addition group by group, which reduces by a gcd after every add, and an lcm
accumulation with one gcd reduction at the end. The two results are asserted equal.
One run on the machine above; a second run agreed within 10%. Where the text gives a
range such as 0.67–1.9 ms, it runs from the lcm method to the per-add method.