docs(adr): ADR 005 exact decimal arithmetic (proposed) (improvise-l2b8)
A two-level numeric tower: exact decimals (rust_decimal) by default, inexact f64 by Scheme's contagion rule. Measured against f64, bigdecimal and num-rational, and against ADR 009's exact f64 accumulator. Registered on the doc site. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (agent-adr005)
committed by
Edward Langley
Edward Langley
parent
b639266b0d
commit
1fe3a85d1e
@@ -0,0 +1,656 @@
|
||||
# ADR 005 — Exact decimal arithmetic: a two-level numeric tower for formulas
|
||||
|
||||
**Status:** proposed. Only the user accepts an ADR. The numbered questions at the end
|
||||
are theirs to answer on the PR. ·
|
||||
**Ticket:** improvise-l2b8 (parent improvise-felo) ·
|
||||
**Direction from:** the user, 2026-09-22 (below) ·
|
||||
**Relates to:** ADR 009 (incremental evaluation, improvise-felo.4, PR #309), which now
|
||||
carries an exact f64 accumulator; ADR 007 (stable identities), whose new file-version
|
||||
literal this ADR may share ·
|
||||
**Governed by:** `context/design-principles.md` §3 and §5
|
||||
|
||||
> "Formulas should default to decimal math (ideally following Scheme's numeric tower
|
||||
> rules for promotion)."
|
||||
>
|
||||
> — the user, 2026-09-22, recorded on improvise-l2b8
|
||||
|
||||
**Numbering.** This is the ADR that improvise-l2b8 claimed as 005 before ADR 007 was
|
||||
written. improvise-8tk5 and improvise-80rm say "ADR 005" in their titles only because
|
||||
ADR 007 was first filed under this number. They belong to ADR 007, not to this document.
|
||||
|
||||
**Citations.** Every `file:line` below is at `gh/main` `b639266b`. Line numbers are
|
||||
dated, so re-derive them before citing this document elsewhere. "ADR 009" means the
|
||||
head of PR #309 at `a2836f44`.
|
||||
|
||||
## The decision this ADR writes down
|
||||
|
||||
1. **Two levels, not four.** A number is **exact** (a decimal) or **inexact** (an f64).
|
||||
Scheme's exact integers and exact decimals are one level here, and there are no exact
|
||||
rationals. Rationals were measured and they fail on the most ordinary pivot-table
|
||||
question, the sum of ratios (clause 1).
|
||||
2. **Exact is the default.** Every number a person types, every number in a `.improv`
|
||||
file, every CSV or JSON number and every formula literal is exact when it fits.
|
||||
3. **Exactness is lost only in named places.** `+`, `-` and `*` keep it. Division keeps
|
||||
it when the quotient is a finite decimal. `POW` keeps it for an integer exponent. A
|
||||
non-integer exponent, a non-terminating quotient, an overflow, and any inexact operand
|
||||
make the result inexact. That is Scheme's contagion rule.
|
||||
4. **The epsilon goes.** `=` and `!=` compare exactly, and the division-by-zero guard
|
||||
tests for exactly zero. The two still share one predicate, which is the property
|
||||
gotcha 17 exists to keep.
|
||||
5. **The stored type is `CellValue::Number(Num)`**, where `Num` is `Exact(Decimal)` or
|
||||
`Inexact(f64)`. The crate is `rust_decimal`.
|
||||
6. **This ADR does not cover summation order.** ADR 009's exact f64 accumulator already
|
||||
makes `SUM` over f64 values order-free. What this ADR adds is everything that
|
||||
accumulator cannot reach: per-cell arithmetic, typed money-like values, exact
|
||||
comparison, and showing a typed value back as it was typed. Clause 8 states the
|
||||
boundary, and the measurement behind it.
|
||||
|
||||
## Context
|
||||
|
||||
### How a number flows today
|
||||
|
||||
Every number is an `f64` from the moment it is parsed until the moment it is drawn.
|
||||
|
||||
| where | what | `file:line` |
|
||||
|---|---|---|
|
||||
| the stored type | `CellValue::Number(f64)`, serialized through `wire_f64` | `crates/improvise-core/src/model/cell.rs:116-122` |
|
||||
| non-finite tokens | `nonfinite_token` / `parse_nonfinite` (`inf`, `-inf`, `nan`) | `cell.rs:63-82` |
|
||||
| the wire adaptor | JSON number when finite, token string when not (improvise-55v2) | `cell.rs:84-114` |
|
||||
| editor seed | `CellValue::edit_text`: Rust's shortest round-trip spelling (improvise-umi4) | `cell.rs:137-145` |
|
||||
| screen spelling | `Display`: integer, else `{:.4}` | `cell.rs:148-161` |
|
||||
| cell aggregation | `apply_agg` over `&[f64]`; `aggregate_cell_values` filters `as_f64` | `crates/improvise-core/src/model/types.rs:20-44` |
|
||||
| formula aggregation | `eval_agg`, `values.iter().sum()` in store order | `types.rs:57-87` |
|
||||
| the epsilon | `FLOAT_EQ_EPSILON = 1e-10` (improvise-0bf) | `types.rs:16` |
|
||||
| record-level eval | `eval_expr`: `lv + rv`, `lv / rv` behind the epsilon, `lv.powf(rv)` | `types.rs:229-282` |
|
||||
| comparison | `eval_bool`: `(lv - rv).abs() < FLOAT_EQ_EPSILON` | `types.rs:314-315` |
|
||||
| the cached twin | `eval_expr_cached`, the same arithmetic again | `types.rs:1090-1104`, `1156-1157` |
|
||||
| formula literal | `Expr::Number(f64)`, from `number = digits ("." digits)?` | `crates/improvise-formula/src/ast.rs:35`, `formula.pest:105-108`, `parser.rs:192` |
|
||||
| typed input | `typed()`: `text.parse::<f64>()`, else text (improvise-j7v6) | `crates/improvise-protocol/src/resolve.rs:1438-1446` |
|
||||
| display | `round_half_away` scales by `10^d` in f64, then `format!` | `crates/improvise-core/src/format.rs:125-160` |
|
||||
| `.improv` write | `format_number`: integer, else shortest `Display`, else `{:?}` | `crates/improvise-io/src/persistence/format.rs:38-54` |
|
||||
| `.improv` grammar | `number = "-"? digits ("." digits)? (exponent)?` | `crates/improvise-io/src/persistence/improv.pest:110-112` |
|
||||
| `.improv` read | `parse_number_cell`: `s.parse()` into f64 | `crates/improvise-io/src/persistence/parse.rs:141-146` |
|
||||
| CSV field | `i64`, else `f64`, into a `serde_json::Number` | `crates/improvise-io/src/import/csv_parser.rs:167-186` |
|
||||
| import measures | read back with `as_f64` | `crates/improvise-io/src/import/wizard.rs:425` |
|
||||
| JSON import | `serde_json::Value`, numbers already f64 (`float_roundtrip`) | `src/main.rs:365`, `crates/improvise-command/src/ui/effect.rs:1151`, `1280` |
|
||||
| import collisions | the "sum" policy adds in f64, in record order (improvise-cgqt) | `crates/improvise-io/src/import/wizard.rs:307`, `330` |
|
||||
| the wire | `ModelCommand::SetCell { value: CellValue }`, `CellDisplay::value`, `SetYanked` | `crates/improvise-protocol/src/reduce.rs:33`, `88-92`; `render.rs:88-104` |
|
||||
| gp seam | `cell_text`: a number crosses **as text**, `n.to_string()` | `crates/improvise-gp/src/dataset.rs:50-56` |
|
||||
|
||||
`CellValue::Number` appears 660 times across 43 files (`grep -rn`, tests included).
|
||||
That is the blast radius of any change to the stored type.
|
||||
|
||||
### What is already wrong, measured
|
||||
|
||||
The measurements in this ADR were taken on an **Apple M4 Max (64 GB), rustc 1.98.1,
|
||||
release builds**, by a scratch program described in the appendix. They were not
|
||||
committed, for the reason the appendix gives.
|
||||
|
||||
**Per-cell arithmetic is wrong in the last place, and the user sees it.**
|
||||
|
||||
- `0.1 + 0.2` is `0.30000000000000004`, and ten `0.1`s sum to `0.9999999999999999`.
|
||||
- `1234.56 - 1234.55` is `0.009999999999990905`, and `100.10 - 100` is
|
||||
`0.09999999999999432`.
|
||||
|
||||
A display format hides most of this. It does not hide it from `=`, from an editor, or
|
||||
from the next formula.
|
||||
|
||||
**Display rounding is wrong for a typed value.** `format.rs:127-136` rounds by scaling
|
||||
the f64. A typed `1.005` is stored as `1.00499999999999989…`, so at two places the grid
|
||||
shows **`1.00`**. A person who typed `1.005` and asked for two places expects `1.01`.
|
||||
Half-away-from-zero on the decimal gives `1.01`. `2.675`, `0.125` and `1.115` happen to
|
||||
agree on both paths. So the bug is real but intermittent, which is the worst kind for a
|
||||
spreadsheet.
|
||||
|
||||
**`SUM` depends on order, and on how far it is from what was typed.** SUM over
|
||||
18,000 typed two-place decimals (0.01 to 999,999.99, seed 1):
|
||||
|
||||
- the forward and the reverse f64 fold differ by **5.0e-5**;
|
||||
- the forward fold is **5.3e-5** from the exact decimal total (1.4e-4 at 90,000 cells);
|
||||
- the decimal sum (`rust_decimal`) is identical in both orders.
|
||||
|
||||
ADR 009's own measurement found the same thing with five-place values: the eager sum is
|
||||
up to 5.8e-10 from the exact decimal sum, above `FLOAT_EQ_EPSILON` at 7 of 8
|
||||
checkpoints (improvise-l2b8's comment). Gotcha 18 recorded the order dependence as a real
|
||||
bug (improvise-shyt), fixed by pinning iteration order rather than by removing it.
|
||||
|
||||
**The epsilon is a patch, and it does not scale.** It was chosen because it "absorbs
|
||||
accumulated f64 rounding noise" (`context/code/core.md` §4). But one ulp of 1,000,000 is
|
||||
1.16e-10. So above about 2¹⁹ (524,288), a one-ulp error is already larger than the
|
||||
epsilon, and `=` sees it. Below about 1e-10, the epsilon calls different numbers equal.
|
||||
|
||||
### What ADR 009's exact accumulator fixes, and what it does not
|
||||
|
||||
ADR 009 now proposes a **superaccumulator**: a fixed-point integer wide enough for every
|
||||
finite f64. It sums f64 values exactly and rounds once, when the sum is read. It makes
|
||||
`SUM` independent of order, and its retraction exact, **without this ADR**. That changes
|
||||
what this ADR is for, so it was measured rather than assumed.
|
||||
|
||||
The question: after a perfect f64 accumulator, does the sum of typed decimals equal the
|
||||
typed total? The stand-in for the accumulator was an exact rational sum of the stored
|
||||
f64 values, rounded once (appendix).
|
||||
|
||||
| typed two-place values summed | trials | accumulator ≠ typed total | naive f64 fold ≠ typed total |
|
||||
|---|---|---|---|
|
||||
| 2 | 10,000 | **2,120** (21%) | 2,120 (21%) |
|
||||
| 3 | 10,000 | **1,621** (16%) | 2,683 (27%) |
|
||||
| 10 | 10,000 | **752** (7.5%) | 4,039 (40%) |
|
||||
| 100 | 10,000 | **250** (2.5%) | 7,944 (79%) |
|
||||
| 18,000 | 8 seeds | **0** | (all 8: see above) |
|
||||
| 90,000 | 8 seeds | **0** | (all 8) |
|
||||
|
||||
**The accumulator solves the large sum and not the small one.** Over thousands of cells
|
||||
the input rounding errors cancel far below the total's ulp, so the one rounding lands on
|
||||
the typed total. Over two or three cells they do not. For two summands a
|
||||
superaccumulator is exactly `+`, because one f64 addition is already correctly rounded.
|
||||
So `0.1 + 0.2` is still `0.30000000000000004` under the accumulator. A subtotal of three
|
||||
line items is the sum a person checks by eye, and it misses in 16% of cases.
|
||||
|
||||
So the division of labour is:
|
||||
|
||||
| | ADR 009 accumulator | this ADR |
|
||||
|---|---|---|
|
||||
| `SUM`/`AVG` independent of order, exact retraction | **yes** | yes for exact values; it uses the accumulator for inexact ones (clause 8) |
|
||||
| a large `SUM` equal to its typed total | yes, measured above | yes |
|
||||
| a small `SUM` equal to its typed total | **no**: 16% of 3-term sums miss | yes |
|
||||
| per-cell arithmetic (`Revenue - Cost`, `* 1.1`, `0.1 + 0.2`) | **no** | yes |
|
||||
| exact `=` against a typed value, with no epsilon | only where the sum happens to round right | yes |
|
||||
| a typed value shown back as typed (`12.50`, `1.005` at two places) | no | yes |
|
||||
| cost | 568 bytes per group, about 3.2 ns per update | 16 bytes per value; about 5–8 ns per add, 60–110 ns per division |
|
||||
|
||||
**This ADR does not re-specify the accumulator.** Where an inexact value must be summed,
|
||||
it is summed by ADR 009's accumulator, and this ADR only says so.
|
||||
|
||||
### The crates
|
||||
|
||||
Three crates were evaluated. All three build for `wasm32-unknown-unknown`.
|
||||
|
||||
| | `rust_decimal` 1.43.0 | `bigdecimal` 0.4.10 | `num-rational` 0.4.2 (`BigRational`) |
|
||||
|---|---|---|---|
|
||||
| value | 96-bit integer and a scale up to 28 | `BigInt` and an `i64` scale | `BigInt` / `BigInt` |
|
||||
| inline size | **16 bytes, `Copy`**, no heap | 40 bytes and a heap buffer | 64 bytes and two heap buffers |
|
||||
| `+ - *` | exact within 28 significant digits; beyond that it **rounds silently** (1000 + 1/3 comes back at scale 25) | always exact | always exact |
|
||||
| `/` | rounds to 28 significant digits (`1/3*3 ≠ 1`) | rounds to 100 digits by default | exact |
|
||||
| normal-edge deps | `arrayvec`, `num-traits`, `serde` | `libm`, `num-bigint`, `num-integer`, `num-traits` | `num-bigint`, `num-integer`, `num-traits` |
|
||||
| wasm size, marginal | **+35.9 KB, +15.4 KB gzipped** | +77.9 KB, +30.7 KB gzipped | +59.6 KB, +20.8 KB gzipped |
|
||||
| SUM, ns per element (18k / 90k) | **5.7–8.1 / 4.5–8.5** | 14.5–22.9 / 10.8–20.7 | 609–1,057 / 491–989 |
|
||||
| `(Rev − Cost) / Rev`, summed, ns per record | **65–111 / 58–112** | 2,429–5,353 | **5.3–6.8 ms**, abandoned (below) |
|
||||
|
||||
The f64 references are 0.54–1.18 ns per element for SUM and 0.56–1.24 ns per record for
|
||||
the formula. Each range is two full runs. `1.5 == 1.50` and the two hash equally in
|
||||
`rust_decimal`. The wasm figures are `wasm-opt -Oz` over a module that already carries
|
||||
f64 parse and print, as the real bundle does, so they are the marginal cost. They are
|
||||
from a five-operation probe, not the real bundle, and the real cost should be re-read
|
||||
when the dependency lands (question 8).
|
||||
|
||||
**Exact rationals fail on the sum of ratios, and it is not close.** The ticket asked
|
||||
what bounds denominator growth. Summing `(Revenue − Cost) / Revenue` over random typed
|
||||
records, the denominator of the running sum grew as follows:
|
||||
|
||||
| ratios summed | 256 | 512 | 1,024 | 2,048 | stopped (20 s budget) |
|
||||
|---|---|---|---|---|---|
|
||||
| denominator bits | 4,699 | 9,103 | 17,015 | 31,697 | 44–55 k, after 2,936–3,741 records |
|
||||
|
||||
Nothing bounds it. Each new revenue brings new prime factors into the least common
|
||||
denominator, so it grows linearly in bits and each addition costs more than the last.
|
||||
At the stopping point one addition cost milliseconds. **An 18,000-record average margin
|
||||
does not finish.** That is the most ordinary question a pivot table is asked, so exact
|
||||
rationals as the *default* are rejected on measurement, not taste.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Two levels: exact decimal and inexact binary
|
||||
|
||||
Scheme's tower is integer ⊂ rational ⊂ real ⊂ complex, and each level has an exact and an
|
||||
inexact form. Improvise keeps **two** levels:
|
||||
|
||||
- **Exact:** a decimal, `rust_decimal::Decimal`. Integers are decimals with scale 0.
|
||||
- **Inexact:** an `f64`, Scheme's flonum.
|
||||
|
||||
No exact rationals, for the reason measured above. No complex numbers, because nothing
|
||||
produces one. The exact level is **bounded**: 28 significant digits and a magnitude
|
||||
below about 7.9 × 10²⁸. Scheme's exact numbers are unbounded, so this is the one place
|
||||
the tower deviates from Scheme by design. Clause 4 says what happens at the bound.
|
||||
|
||||
### 2. The stored type: `CellValue::Number(Num)`
|
||||
|
||||
```rust
|
||||
pub enum Num {
|
||||
Exact(Decimal),
|
||||
Inexact(f64),
|
||||
}
|
||||
|
||||
pub enum CellValue {
|
||||
Number(Num),
|
||||
Text(String),
|
||||
Error(String),
|
||||
}
|
||||
```
|
||||
|
||||
`Num` lives in `improvise-core` beside `CellValue`. The arithmetic and the contagion
|
||||
rules are methods on `Num`, in one module. Both evaluators (`eval_expr` and
|
||||
`eval_expr_cached`) call the same methods, so the rules cannot drift between them the
|
||||
way the epsilon once did (improvise-0bf).
|
||||
|
||||
A new arm beside `Number(f64)` was rejected. It would double every numeric match at 660
|
||||
sites, and "is this cell a number" could then be asked two ways. `CellValue::as_f64`
|
||||
stays, as the explicitly lossy accessor for the seams that want a binary float (clause
|
||||
12).
|
||||
|
||||
`Expr::Number(f64)` becomes `Expr::Number(Decimal)`. A formula literal is always exact,
|
||||
because it is decimal text (clause 3).
|
||||
|
||||
### 3. What is exact on entry
|
||||
|
||||
**Everything a person or a file spells in decimal is exact, when it fits.** That covers
|
||||
typed input (`typed()`), `.improv` numbers, CSV fields, formula literals and pasted
|
||||
values.
|
||||
|
||||
**This deviates from Scheme on purpose.** Scheme reads `1.5` as *inexact* and needs
|
||||
`#e1.5` for exact. Improvise reads `1.5` as exact, because a pivot table's inputs are
|
||||
decimals a person typed, and "decimal math by default" is the user's instruction. So
|
||||
Scheme's `#e` is improvise's default.
|
||||
|
||||
A spelled number is **inexact** only when it cannot be exact:
|
||||
|
||||
- it is `inf`, `-inf` or `nan` (clause 7);
|
||||
- it has more than 28 significant digits, or its magnitude is outside `Decimal`'s range
|
||||
(for example `1e300` or `1e-40`). Then it reads as the nearest f64.
|
||||
|
||||
**A JSON number** arrives as an f64, because `serde_json` parses it before improvise sees
|
||||
it. It is read as the exact decimal of its **shortest round-trip spelling**. The shortest
|
||||
spelling of `0.1_f64` is `0.1`, not `0.1000000000000000055…`. With `float_roundtrip`
|
||||
enabled workspace-wide, that recovers the *value* the file spelled whenever the literal
|
||||
had at most 15 significant digits, though not its trailing zeros (clause 12). A longer
|
||||
literal may come back different in its last digits. Enabling `serde_json`'s
|
||||
`arbitrary_precision` instead is question 6. The import pipeline carries CSV fields as
|
||||
`serde_json` numbers too, so the same rule reads them.
|
||||
|
||||
### 4. How exactness propagates
|
||||
|
||||
| operation | exact ∘ exact | any inexact operand |
|
||||
|---|---|---|
|
||||
| `+`, `-`, `*`, unary `-` | exact, **if** the result is exactly representable; else inexact | inexact |
|
||||
| `/` | clause 5 | inexact |
|
||||
| `POW` | clause 5 | inexact |
|
||||
| `IS`, `IF` | not arithmetic: `IS` yields exact 0 or 1 | — |
|
||||
|
||||
**Contagion is Scheme's rule.** An inexact operand makes the result inexact, and an
|
||||
exact result never comes from an inexact input.
|
||||
|
||||
**"If exactly representable" needs care, because `rust_decimal` rounds silently.** An add
|
||||
whose result needs more than 28 significant digits comes back rounded, with no signal.
|
||||
The measurement caught it: `1000 + 0.3333333333333333333333333333` returns scale 25.
|
||||
Overflow of the magnitude is reported (`checked_add` returns `None`), but precision loss
|
||||
is not. So `Num`'s arithmetic must **detect** either one and return `Inexact`, computed in
|
||||
f64 from the operands. It must never return a rounded `Exact`. A rounded value that
|
||||
calls itself exact is the one outcome this design exists to prevent. Under clause 5 no
|
||||
computed exact value carries a long fraction, so this arises only from a typed value with
|
||||
more than about 20 decimal places, or from magnitudes near 10²⁸.
|
||||
|
||||
### 5. Division, `AVG` and `POW`
|
||||
|
||||
**Division.** Exact ÷ exact is **exact when the quotient is a finite decimal** that fits
|
||||
(`1 / 4 = 0.25`, `Profit / Revenue` when it happens to terminate). Otherwise it is
|
||||
**inexact**: the f64 nearest the true quotient. So `1 / 3` is inexact, and `1 / 3 * 3`
|
||||
is the f64 `1.0` or a neighbour, exactly as today. The implementation must decide
|
||||
termination without a check that itself rounds. For example, multiplying the quotient
|
||||
back is unsound, because that multiply can round too.
|
||||
|
||||
Scheme would answer `1/3` exactly. That is the choice the rational measurement rules
|
||||
out, and question 2 puts it to the user with the alternatives.
|
||||
|
||||
**`AVG`** is `SUM / COUNT` under the division rule. The average of `10.00`, `20.00` and
|
||||
`40.00` is 70 / 3, which does not terminate, so it is inexact. The average of `10.00`,
|
||||
`20.00`, `30.00` and `40.00` is exactly `25`. **Many averages will be inexact.** That is
|
||||
honest: they are not finite decimals. They display exactly as today.
|
||||
|
||||
**`POW`.** `x ^ n` with exact `x` and an exact integer `n ≥ 0` is exact (repeated
|
||||
multiplication, with clause 4's overflow rule). A negative integer `n` is `1 / x^|n|`,
|
||||
under the division rule. Any non-integer exponent, or any inexact operand, is `powf`,
|
||||
inexact. This is the ticket's "obvious candidate" for an inexact operation, and it is
|
||||
the only one besides division: improvise has no transcendental functions.
|
||||
|
||||
**Division by zero** is `Error("div/0")` for every combination of exactness, as today.
|
||||
The test is **exactly zero**, with no epsilon (clause 6).
|
||||
|
||||
### 6. Comparison, and the end of `FLOAT_EQ_EPSILON`
|
||||
|
||||
`=`, `!=`, `<`, `>`, `<=` and `>=` compare **exactly**. `FLOAT_EQ_EPSILON` and its six
|
||||
uses are deleted.
|
||||
|
||||
- **Exact with exact:** decimal comparison. `1.5 = 1.50` is true, because scale is
|
||||
presentation (clause 9), not value.
|
||||
- **Inexact with anything:** the exact operand is converted to f64, and the two f64s are
|
||||
compared exactly. That is the arithmetic contagion rule applied to comparison. It can
|
||||
be non-transitive at the last bit, as mixed comparison is in most Schemes. Question 3
|
||||
asks whether inexact `=` should keep a tolerance instead.
|
||||
|
||||
**Gotcha 17's invariant survives in a new form.** The epsilon was shared by `=` and the
|
||||
division guard so that `IF(x = 0, a, y / x)` is coherent: the condition and the guard
|
||||
must agree on what zero is (improvise-0bf). Under this ADR they agree because both test
|
||||
exactly zero. The property is "one predicate for zero", not "one epsilon". The test that
|
||||
pins it keeps its name and its `x = 1e-11` case, which is now non-zero on both sides.
|
||||
|
||||
### 7. NaN and the infinities live only at the inexact level
|
||||
|
||||
An exact number is always finite. `inf`, `-inf` and `nan` are inexact, as in Scheme
|
||||
(`+inf.0`, `+nan.0`). They keep their one spelling (improvise-55v2) in the file, on the
|
||||
wire, and in `typed()`. Where they come from is unchanged: stored data, and inexact
|
||||
overflow in `*` or `POW`. Exact overflow becomes inexact (clause 4), and it may overflow
|
||||
further to `inf` only at the inexact level.
|
||||
|
||||
### 8. `SUM` and the other aggregates
|
||||
|
||||
`apply_agg` and `eval_agg` partition their values by exactness:
|
||||
|
||||
- **Exact values are summed exactly**, as a decimal. The sum is independent of order.
|
||||
That removes gotcha 18's order-dependent `SUM` as a class for exact data, which is
|
||||
nearly all data.
|
||||
- **Inexact values are summed by ADR 009's exact f64 accumulator** when it lands, and by
|
||||
today's fold until then.
|
||||
- **The result is exact if every value was exact.** Otherwise it is inexact: the f64 of
|
||||
the exact part plus the accumulator's part, rounded once. **It must not be a fold with
|
||||
contagion part-way through.** Such a fold switches to f64 at the first inexact value
|
||||
in store order, so the answer would depend on where that value sits.
|
||||
|
||||
`COUNT` is exact (an integer). `MIN` and `MAX` compare by clause 6 and return the chosen
|
||||
value with its own exactness. Text unanimity and `mixed` are unchanged (gotcha 7).
|
||||
|
||||
**For ADR 009:** its accumulator stays the mechanism for inexact values. The exact part is
|
||||
an abelian group without it: decimal addition is exact under clause 4's detect-or-demote
|
||||
rule. So a maintained `Sum` over exact values is bit-equal to eager on every path, with
|
||||
no accumulator at all.
|
||||
|
||||
### 9. Display, edit text and typed scale
|
||||
|
||||
**Display** of an exact value rounds the decimal, half away from zero, with
|
||||
`round_dp_with_strategy(d, MidpointAwayFromZero)`, then groups. So a typed `1.005` shows
|
||||
as `1.01` at two places. An inexact value displays exactly as today. `format_value` and
|
||||
`NumberFormat::number` take `&Num`. Gotcha 8 is unchanged: rounding is view-only and
|
||||
never fed back.
|
||||
|
||||
**Edit text** of an exact value is its decimal **with its scale**. A typed `12.50` opens
|
||||
as `12.50`, not `12.5`, so a typed value is shown back as it was typed, up to canonical
|
||||
spelling: `+1`, `1e3` and `012` become `1`, `1000` and `12`. An inexact value keeps
|
||||
today's shortest round-trip spelling (improvise-umi4). A non-finite value keeps its token.
|
||||
|
||||
**Scale is presentation, not value.** `12.5` and `12.50` are equal under `=` and under
|
||||
`CellValue`'s `PartialEq`. So rewriting one as the other does not bump the record's
|
||||
version in ADR 001's record tier. `rust_decimal` hashes them equally, so that property
|
||||
holds for free.
|
||||
|
||||
### 10. `.improv` persistence and migration
|
||||
|
||||
**Reading.** Every `number` token reads as clause 3 says: exact if it fits, else inexact.
|
||||
The grammar's `number` rule (`improv.pest:110-112`) already accepts everything a decimal
|
||||
needs, including an exponent.
|
||||
|
||||
**Writing.** An exact number is written as its decimal, with its scale. An inexact
|
||||
*finite* number needs a spelling that reads back as inexact, or a save and reload would
|
||||
silently make it exact. The format must say which one it holds: the ticket's `1/3`
|
||||
against `0.333…` point. This ADR proposes Scheme's own marker, **`#i`**:
|
||||
`#i0.3333333333333333`. The non-finite tokens are already inexact and need no marker.
|
||||
The spelling is question 4.
|
||||
|
||||
**Inexact stored values are rare.** A `## Data` line holds raw cells: typed, imported or
|
||||
pasted. None of them is inexact unless it was pasted from an inexact formula result, is
|
||||
non-finite, or is out of `Decimal`'s range. The marker is a correctness requirement,
|
||||
not a common sight.
|
||||
|
||||
**Migration: an existing number becomes the exact decimal it spells.** Every existing
|
||||
`.improv` loads unchanged in the sense that matters: every value reads as the number its
|
||||
text names. That is a behaviour change at the 17th significant digit, and it goes
|
||||
toward what the text says. An old file containing `0.30000000000000004` (possible where a
|
||||
computed f64 was pasted and saved) loads as exactly that decimal, not as the binary
|
||||
fraction the old program held. No document changes in any digit a person can see.
|
||||
|
||||
**The version literal.** `#i` is a grammar change. An old reader would take
|
||||
`#i0.5` as text (`bare_value`), so a file that uses it must carry a new version literal.
|
||||
ADR 007 clause 10 already introduces one ("the regime is chosen only by the first line").
|
||||
If both ADRs land close together, they share one bump. A file with no inexact finite
|
||||
number can be written under either literal, because it reads the same under both. That
|
||||
choice is question 5.
|
||||
|
||||
**The round-trip tests** must be re-read, as the ticket says. `format_number`'s three
|
||||
spellings collapse to two (decimal, `#i` shortest), and the grammar-walking generator
|
||||
gains the `#i` arm. The property `parse(format(parse(x)))` stable is unchanged, and it
|
||||
is now joined by **exactness preserved**, which the old format could not state.
|
||||
|
||||
### 11. The wire
|
||||
|
||||
`CellValue` crosses the wire in `ModelCommand::SetCell`, `SetYanked` and
|
||||
`CellDisplay::value`. `CellDisplay::text` and `edit_text` are strings and are unaffected.
|
||||
So a thin client *is* affected, and the ticket was right to ask for confirmation rather
|
||||
than assume.
|
||||
|
||||
`Num` serializes through the existing untagged adaptor, extended by one arm:
|
||||
|
||||
| value | JSON |
|
||||
|---|---|
|
||||
| exact | a **string** of its decimal: `"12.50"` |
|
||||
| inexact, finite | a **number**, as today: `0.3333333333333333` |
|
||||
| inexact, non-finite | the token string, as today: `"inf"` |
|
||||
|
||||
A decimal string never collides with a token, because `inf`, `-inf` and `nan` are not
|
||||
decimals. A finite inexact cell's bytes are unchanged.
|
||||
|
||||
**Version skew fails loudly in one direction and silently in the other.** An old peer
|
||||
reading `"12.50"` tries it as a token and refuses the frame, which is loud. A new peer
|
||||
reading an old peer's JSON number takes it as inexact, which is silent. That only
|
||||
matters when a deployed browser page talks to a `ws-server` of a different version. The
|
||||
protocol has no version handshake today. This ADR does not add one, and names the hole.
|
||||
|
||||
`typed()` stays the one home of text → value (improvise-j7v6). It parses exact first,
|
||||
so a browser composes exact values with no second rule.
|
||||
|
||||
### 12. The seams
|
||||
|
||||
**gp-tree is out of scope and needs no change.** A number crosses into a fit **as text**
|
||||
(`dataset.rs:50-56`), which `preprocess` parses into f32. An exact `12.50` crosses as
|
||||
`"12.50"`, and it parses to the same f32. The emitted thresholds are decimal text
|
||||
(`improvise-gp/src/lib.rs:285-298`), so they become exact literals, and exact comparison
|
||||
against exact data routes the same way as the f64 comparison does today, except within
|
||||
f32 rounding of a threshold. That boundary difference already exists between f32 and
|
||||
f64. `scoring_known_answers.rs`'s bit patterns (gotcha 37c) are untouched, because
|
||||
gp-tree never sees a `Num`.
|
||||
|
||||
**Import.** The import pipeline carries every record as a `serde_json::Value`, CSV
|
||||
included: `parse_csv_field` (`csv_parser.rs:167-186`) turns a field into an `i64` or an
|
||||
`f64` `Number`, and the pipeline reads measures back with `as_f64` (`wizard.rs:425`). A
|
||||
`Value` has no decimal. So the one change is at that read: a measure becomes a `Num` by
|
||||
clause 3's shortest-spelling rule, for CSV and JSON alike. A CSV field of at most 15
|
||||
significant digits is recovered exactly. Carrying the field's text through the pipeline
|
||||
instead would restructure the analyzer for digits no CSV of money carries. One thing is
|
||||
lost and stated: the **scale**. A CSV `12.50` becomes the f64 `12.5`, whose shortest
|
||||
spelling is `12.5`, so an imported value's editor opens on `12.5`. Its value is exact.
|
||||
Only typed and `.improv` values keep their trailing zeros. The collision "sum" policy
|
||||
(`wizard.rs:307`) then sums exactly, by clause 8.
|
||||
|
||||
**Format.** `format_f64` stays for inexact values and gains a decimal twin. `FormatPlan`
|
||||
is unaffected: it chooses a format, not a value.
|
||||
|
||||
### 13. The crate: `rust_decimal`
|
||||
|
||||
Chosen on the table above. Four reasons:
|
||||
|
||||
- **Size and speed.** It is `Copy` and 16 bytes, with no heap. So a `CellValue` does
|
||||
not start allocating for every number. It is about 2.5× faster than `bigdecimal` on
|
||||
SUM, and about 40× faster on the formula workload.
|
||||
- **Bundle.** It is the smallest marginal addition to the wasm bundle.
|
||||
- **Bound.** It is bounded, and the bound is the right size for this domain. 28
|
||||
significant digits holds any money amount and any count. `bigdecimal`'s unbounded
|
||||
add costs a heap allocation for a precision no pivot table uses. Its division still
|
||||
rounds, so it does not buy exact division either.
|
||||
- **Serde.** Its serializer is a decimal string by default, which is clause 11's wire
|
||||
spelling.
|
||||
|
||||
The cost is clause 4's detection duty, because it rounds silently at the bound.
|
||||
`num-rational` is rejected on measurement (above). `bigdecimal` is the fallback if the
|
||||
bound ever matters. Other crates (`fastnum`, `dashu`) were not evaluated.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
- **Exact rationals as the default (Scheme proper).** Measured above: an 18,000-record
|
||||
average of ratios does not finish. It would be the right answer for a symbolic
|
||||
calculator and the wrong one for a pivot table.
|
||||
- **A rounded decimal that still calls itself exact** (SQL `NUMERIC`: `1/3` becomes
|
||||
`0.3333…` at 28 places, marked exact). It keeps "decimal math" true everywhere, but
|
||||
then "exact" means "exact except after a division". Equality, ADR 009's group law and
|
||||
every future reasoning about exactness would carry that exception silently. It is
|
||||
question 2's option (b), for the user to overrule.
|
||||
- **Keep f64 and adopt only ADR 009's accumulator.** It fixes large sums. It leaves
|
||||
per-cell arithmetic, small sums (16% of 3-term sums), display of typed values and the
|
||||
epsilon exactly as they are (Context).
|
||||
- **Three levels (exact decimal, inexact decimal, f64).** An "inexact decimal" arm for
|
||||
rounded quotients would add a third arm to every numeric match. It would buy only a
|
||||
28-digit rather than a 17-digit approximation of a number that is not a decimal
|
||||
anyway.
|
||||
- **A new `CellValue` arm beside `Number(f64)`.** Rejected in clause 2.
|
||||
|
||||
## Migration
|
||||
|
||||
In order. Each step lands on its own, with the suite green.
|
||||
|
||||
1. **`Num` and its arithmetic**, in `improvise-core`, with the contagion table (clause 4),
|
||||
the division and `POW` rules (clause 5) and exact comparison (clause 6). Property
|
||||
tests: exact addition is associative and commutative; contagion is total; an `Exact`
|
||||
result is never a rounded one (inject a silently-rounding add and watch it fail).
|
||||
Measure one whole recompute at 18,000 and 90,000 cells against today, and name the
|
||||
machine. The arithmetic figures above are a microbenchmark, not the evaluator.
|
||||
2. **`CellValue::Number(Num)`** across the 660 sites. This is mechanical, and `as_f64`
|
||||
keeps the seams compiling.
|
||||
3. **Both evaluators** move to `Num`. `FLOAT_EQ_EPSILON` is deleted. The zero-predicate
|
||||
test is re-pointed (clause 6).
|
||||
4. **Aggregates** partition by exactness (clause 8).
|
||||
5. **Entry points:** `typed()`, the import pipeline's measure read, formula literals
|
||||
(clauses 3 and 12).
|
||||
6. **Display and edit text** (clause 9). The `1.005` case is the regression test, and
|
||||
it fails at HEAD today.
|
||||
7. **`.improv`:** read exact, write with scale, `#i` for inexact finite, the version
|
||||
literal (clause 10).
|
||||
8. **The wire** adaptor (clause 11), with the retired-spelling discipline of gotcha 42:
|
||||
assert the error kind, not merely that an error occurred.
|
||||
|
||||
## Open questions
|
||||
|
||||
Each question has a recommendation. The user's answers will be folded in under "What
|
||||
the user decided", as ADRs 004 and 007 did.
|
||||
|
||||
1. **The tower's shape.** Two levels, exact decimal and inexact f64, with Scheme's
|
||||
contagion and a bounded exact level (clauses 1 and 4)?
|
||||
*Recommendation:* yes. It is the smallest tower that makes typed input and its
|
||||
arithmetic exact, and the measurements rule out the larger one.
|
||||
|
||||
2. **Non-terminating exact division.** What is `1 / 3` for two exact operands?
|
||||
(a) inexact f64, the nearest binary value; (b) a 28-digit decimal that stays marked
|
||||
exact (SQL `NUMERIC`); (c) an exact rational (Scheme proper).
|
||||
*Recommendation:* (a). (c) does not finish an average of ratios (measured). (b) makes
|
||||
"exact" false after every division, silently, and it breaks ADR 009's group law for
|
||||
those values: in the measurement, a `rust_decimal` sum of ratios differed between
|
||||
forward and reverse order. (a) is Scheme's own answer for any operation that cannot
|
||||
stay exact.
|
||||
|
||||
3. **Tolerance on inexact equality.** With the epsilon deleted, should `=` between two
|
||||
*inexact* values keep a tolerance?
|
||||
*Recommendation:* no. Exact comparison everywhere, as Scheme does. A tolerance at a
|
||||
fixed absolute size is wrong at large magnitudes (Context), and a relative one is a
|
||||
new semantic that belongs in its own function (for example `APPROX(a, b, tol)`) if it
|
||||
is ever wanted. Exact values, which are nearly all values, need none.
|
||||
|
||||
4. **The spelling of an inexact finite number in `.improv`.** `#i0.333…` (Scheme's
|
||||
marker), `~0.333…`, or no marker at all (an inexact value reloads as exact)?
|
||||
*Recommendation:* `#i`. It is Scheme's spelling, and the user asked for Scheme's
|
||||
rules. No marker is rejected: a save and reload would change a value's exactness,
|
||||
and then the exactness-preserved round-trip property could not be stated.
|
||||
|
||||
5. **The version literal.** Share ADR 007 clause 10's new literal, or take its own?
|
||||
*Recommendation:* share it if the two land within one release, and take its own
|
||||
otherwise. Either way the first line alone picks the regime, as ADR 007 decided.
|
||||
|
||||
6. **JSON import precision.** Read a JSON number through its shortest f64 spelling
|
||||
(clause 3), or enable `serde_json`'s `arbitrary_precision` so the literal survives?
|
||||
*Recommendation:* shortest spelling. `arbitrary_precision` is a workspace-unified
|
||||
feature. It would change `serde_json::Number` for the wire and every other JSON user,
|
||||
including the browser bundle, to fix a literal of more than 15 significant digits in
|
||||
an import file.
|
||||
|
||||
7. **Mixed-exactness comparison.** Compare an exact and an inexact value by converting
|
||||
the exact one to f64 (clause 6), or exactly, by converting the f64 to its exact
|
||||
binary expansion?
|
||||
*Recommendation:* convert to f64. It is the same rule as arithmetic, and the
|
||||
non-transitive corner needs a value that is exactly between two f64s.
|
||||
|
||||
8. **The dependency.** `rust_decimal` in `improvise-core`, a browser-bundle crate, at a
|
||||
marginal +35.9 KB (+15.4 KB gzipped) by the probe?
|
||||
*Recommendation:* yes, with the real bundle re-measured in migration step 1, and
|
||||
checked with `cargo tree --edges normal` per crate (the repo map's Wasm section).
|
||||
|
||||
9. **Version skew on the wire.** Accept clause 11's silent direction (an old peer's
|
||||
number read as inexact), or add a protocol version to the handshake first?
|
||||
*Recommendation:* accept it, and file the handshake version as its own ticket. The
|
||||
skew needs a deployed page and a `ws-server` of different versions, and inexact is
|
||||
the honest reading of a value an old peer computed in f64.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Typed values and their `+ - *` are exact. `0.1 + 0.2 = 0.3`, `1234.56 - 1234.55 =
|
||||
0.01`, and a typed `1.005` shows as `1.01` at two places.
|
||||
- `SUM` over exact data is independent of order, with no accumulator. Gotcha 18's `SUM`
|
||||
drift goes as a class for exact data, not just by pinned iteration order.
|
||||
- `FLOAT_EQ_EPSILON` is gone, and gotcha 17 is rewritten as "one zero predicate".
|
||||
- A typed `12.50` stays `12.50` in the editor.
|
||||
- Division that does not terminate, `AVG` over most counts, and `POW` with a fractional
|
||||
exponent are inexact, and display as today.
|
||||
- `.improv` gains `#i` and a version literal. Existing files load as the decimals they
|
||||
spell.
|
||||
- The wire spells exact values as strings. A thin client needs no new rule, because
|
||||
`typed()` is shared.
|
||||
- **Costs:** a 16-byte number where there was an 8-byte one, and `rust_decimal` in the
|
||||
bundle (+15 KB gzipped, probed). Arithmetic is about 8× slower per add and about 100×
|
||||
per division than f64 (measured, microbenchmark). A 90,000-record formula that
|
||||
divides for every record pays about 5–10 ms in arithmetic alone. Whether the
|
||||
evaluator notices is migration step 1's measurement, and it is **not** measured here.
|
||||
- **Not covered:** the order-free sum of *inexact* values, which is ADR 009's
|
||||
accumulator; a protocol version; conversion functions such as `EXACT()`/`INEXACT()`,
|
||||
which no ticket asks for.
|
||||
|
||||
## Appendix: the measurements
|
||||
|
||||
A scratch crate outside the workspace, run with `cargo run --release`. It was not
|
||||
committed. Committing it would add three decimal crates to the workspace's dev
|
||||
dependencies, and every `cargo test --workspace` would then build them, to hold a
|
||||
proposal's evidence. If the user accepts this ADR, migration step 1 commits the
|
||||
`rust_decimal` half as an example in `improvise-core`, with no new dependency beyond the
|
||||
one the ADR adds.
|
||||
|
||||
**Data.** A typed value is a two-place decimal from 0.01 to 999,999.99, drawn by
|
||||
SplitMix64 (the same generator as ADR 009's `retraction_drift.rs`), and spelled as the
|
||||
text a person types. Each representation parses that text: f64 by `str::parse`,
|
||||
`Decimal`/`BigDecimal` by `from_str`, and `BigRational` as mantissa over 10^scale.
|
||||
|
||||
**SUM.** A fold over N parsed values, N = 18,000 and 90,000, best of 50 runs (20 for
|
||||
`bigdecimal`, 5 for `BigRational`), reported per element. The order check folds forward
|
||||
and reversed.
|
||||
|
||||
**Formula.** For N records with independent Revenue and Cost, `SUM((Rev − Cost) / Rev)`.
|
||||
`bigdecimal` is run at its default precision (100 digits) and at 28. `BigRational` is
|
||||
summed incrementally, printing the running denominator's bit length at powers of two,
|
||||
and abandoned after 20 seconds.
|
||||
|
||||
**Accumulator stand-in.** Each stored f64 is converted to its exact rational
|
||||
(`BigRational::from_float`), summed exactly, and rounded once to f64. That is precisely
|
||||
what a correct superaccumulator returns. It is compared with the typed total, the
|
||||
decimal sum of the texts converted to f64 once. Small N: 10,000 trials each at
|
||||
N = 2, 3, 10 and 100, seed 7. Large N: seeds 1 to 8.
|
||||
|
||||
**Display.** `format.rs:127-136`'s `round_half_away`, copied verbatim, against
|
||||
`Decimal::round_dp_with_strategy(2, MidpointAwayFromZero)`, for `2.675`, `1.005`,
|
||||
`0.125` and `1.115`.
|
||||
|
||||
**Wasm size.** A `cdylib` exporting one function that parses two strings, then does
|
||||
`(a + b) − b`, divides by `b` and prints, in each representation. It also parses and
|
||||
prints an f64 in every variant, so f64 formatting is in the baseline as it is in the real
|
||||
bundle. Profile `opt-level = "s"`, `lto = true`, `codegen-units = 1`,
|
||||
`panic = "abort"`. Then `wasm-opt -Oz --enable-bulk-memory
|
||||
--enable-nontrapping-float-to-int`, and `gzip -9` for the compressed figure.
|
||||
@@ -45,6 +45,7 @@ const SECTIONS = [
|
||||
["context/adr/002-naming-a-cell.md", "adr/002-naming-a-cell", "ADR 002 — Naming a cell"],
|
||||
["context/adr/003-live-backed-models.md", "adr/003-live-backed-models", "ADR 003 — Live-backed models"],
|
||||
["context/adr/004-faces-and-themes.md", "adr/004-faces-and-themes", "ADR 004 — Faces"],
|
||||
["context/adr/005-exact-decimal-arithmetic.md", "adr/005-exact-decimal-arithmetic", "ADR 005 — Exact decimal arithmetic (proposed)"],
|
||||
["context/adr/007-stable-identities.md", "adr/007-stable-identities", "ADR 007 — Stable identities"],
|
||||
],
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user