Merge pull request #337 from fiddlerwoaroof/adr005/numtower

feat(numtower): ADR 005 step 1, the numeric tower crate (improvise-59nw.1)

context/code/numtower.md

@@ -0,0 +1,328 @@
+# `numtower`: the reasoning
+
+`crates/numtower` is ADR 005 migration step 1 (improvise-59nw.1): the numeric tower,
+first and alone. Nothing in improvise depends on it yet, so it changes no behaviour.
+The decisions are the ADR's (`context/adr/005-exact-decimal-arithmetic.md`). This file
+records how the crate reads them, the places the ADR left a choice open or said two
+things, and the measurements. The evidence for each test is in
+`context/test-evidence/numtower.md`.
+
+## The shape
+
+| type | what it is | ADR |
+|---|---|---|
+| `Num` | the stored value: an exact `Decimal`, an exact `Ratio<i128>`, or an `f64` | clauses 1, 2 |
+| `NumRef` | a borrowed view of a `Num`'s arm, for exhaustive matching | — |
+| `Interim` | a value inside one evaluation; exact intermediates widen to a bigint rational | question 13 (b) |
+| `NumSum` | a retractable, order-free `SUM` over any `Num` | clause 8, question 16 |
+| `ExactSum` | ADR 009's f64 superaccumulator | question 15 (a) |
+| `parse_number`, `is_number`, `Typing` | clause 3's number language, gate and parse | clause 3, question 12 |
+| `Display` / `Num::from_canonical` | the canonical text: `12.50`, `10/3`, `#i0.1`, `inf`, `-inf`, `nan` | clauses 10, 11, 14 |
+| `Num::round_to_places` | display rounding, the mode a parameter | clause 12 "Format" |
+
+**`Num` is a struct with private arms, not the ADR's `pub enum`.** The ADR's sketch
+(clause 2) shows `pub enum Num { Exact(Decimal), Fraction(Box<Ratio<i128>>),
+Inexact(f64) }`. A public enum would let any caller build values the tower never
+produces: `Fraction(1/2)` (which must be the decimal `0.5`), an unreduced fraction, or an
+exact `-0`. Each would break exact equality or the fingerprint of clause 9. So the arms
+are private, every constructor normalizes, and callers match on `NumRef`. That is
+`context/design-principles.md` §3, smart constructors. It is also what clause 14's table
+asks for, since improvise calls "`Num`'s accessors for exactness, mantissa, scale and
+fraction parts". The `Box` lives in one constructor, as `AGENTS.md` asks. `Num` is 24
+bytes, as the ADR measured, and `a_num_is_24_bytes` pins it.
+
+**The normal form.** An exact zero has no sign. A fraction is reduced and has a positive
+denominator. A value whose reduced denominator is `2^p·5^q`, with `k = max(p, q) <= 28`
+and a mantissa that fits 96 bits, is always the `Decimal`, never a `Fraction`. So
+`Fraction(n/1)` exists only when `n` does not fit 96 bits, as with `10^35/1`. Every `i128`
+is a valid numerator, `i128::MIN` included, because the arm is `Ratio<i128>`. That one
+value cannot be negated in `i128`, so `Operand::parts` keeps it out of the `i128` fast
+path and its arithmetic takes the bigint path (Copilot on PR #337).
+
+## Arithmetic
+
+### Why there are two types: `Num` and `Interim`
+
+Question 13 (b) bounds only the **stored** result: "every intermediate inside one formula
+evaluation is unbounded up to the ceiling". The bounded `Num` cannot be such an
+intermediate, so `Interim` is the widened type. It is either a stored `Num` or an exact
+`BigQ` (a reduced `BigInt` fraction) that does not fit `i128`. `Interim::store` applies
+the stored bound. The operators on `&Num` are one-operation evaluations: each builds the
+`Interim`, applies the operation and stores the result. So `(a * a) / a` returns
+`a` exactly when computed through `Interim`, and is inexact when computed through `Num`
+(`an_intermediate_beyond_i128_comes_back_exact`). That difference is (b)'s whole point.
+Step 3's evaluators should evaluate through `Interim` and store once per cell.
+
+**So `Interim` compares, and knows zero, without storing.** `IF(a*a/a = a, …)` and the
+division guard both run before the stored bound. If `=` had to store its operands first,
+its answer would depend on how the formula is spelled, which is what question 13 (b)
+rules out. `Interim::cmp_exact` is clause 6's exact order over every pairing: two stored
+values go through `Num::cmp_exact`, and anything wide compares as a bigint rational,
+against an f64 by its exact binary value. `Interim::is_zero` is the one zero predicate. A
+wide value is never zero, because zero always fits a stored arm. The comparison through
+storing is the rival (`an_interim_compares_exactly_without_storing_catches_storing_first`):
+`a` and `a ± 1/(3·10^30)` store to one f64 and compare equal. The review of `302594bf`
+asked for this or for a note naming it as a step-3 item; the crate does the first.
+
+### Contagion, and how many roundings a demotion has
+
+An inexact operand makes the result inexact. Each exact operand is converted by the
+correctly rounded conversion, and then the f64 operation runs. That is two roundings,
+which is clause 4's "computed in f64".
+
+**An exact ∘ exact result that does not fit is rounded once.** It is the correctly rounded
+f64 of the exact value. Clause 4 still says a demoted sum or product is "rounded twice,
+once per operand and once in the operation". That text predates option (d). Option (d)'s
+"Operations" and question 13 (b) both say that a result failing the stored bound is "the
+correctly rounded f64 of the exact fraction", and under (b) the exact value is always in
+hand. The crate follows (d), because (d) is decided and later. Both texts cannot hold for
+one operation, so this is a reading, and it is flagged.
+
+### Clause 4: the hint, the recompute, the stored scale
+
+`Decimal + Decimal` and `Decimal × Decimal` first try `rust_decimal`'s checked operation.
+The result is trusted only when its scale equals the nominal one (`max(sa, sb)` for `+`
+and `−`, `sa + sb` for `×`). A lower scale is the **hint**, and so is a `None`. On the
+hint the exact integer result is recomputed in a 256-bit integer, and its trailing zeros
+are stripped. If the stripped mantissa fits 96 bits at a scale of at most 28, the result
+is a decimal at the **stored scale**, `min(nominal, 28)`, padded back with zeros while the
+mantissa fits. Otherwise the result goes to the fraction arm, then to a wide `Interim`.
+The ADR's own example holds: `1.2500000000000000 × 4.0000000000000000` is `5` at scale 28
+(`a_recomputed_result_keeps_the_stored_scale_clause_4_names`). A `None` counts as the
+hint, so that an overflow `rust_decimal` reports is never mistaken for an exact result.
+
+### Division: the direct test, one implementation
+
+Clause 5 is normative: write the quotient as a reduced `n / (2^p·5^q)`, then it is a
+decimal iff `k <= 28` and the padded mantissa fits. The crate does this the same way for
+every exact pairing, not only for decimals. It reduces the quotient (in `i128` when that
+cannot overflow, and in a bigint otherwise), then normalizes. `small_ratio` and
+`decimal_of` are the one normalization, so a decimal reached through any path has the
+same representation. Multiplying back is never used.
+
+**A quotient carries its minimal scale** (`10.00 / 2` is `5`, `1 / 4` is `0.25`). The ADR
+does not say what scale a quotient has. A quotient's scale is not a function of its
+operands' scales (clause 5), and a result that passed through the fraction arm carries
+its minimal scale (option (d)), so the minimal scale is the consistent choice. It is a
+reading, and it is flagged.
+
+**The ADR's argument against multiplying back is weaker than it says.** Clause 5 rejects
+it as incomplete, on `1e-28 / 0.5`: `q × 0.5` "needs scale 29, so the check flags it".
+That holds under revision 6's rule, which demotes on the hint
+(`division_decides_fit_directly_catches_multiplying_back` keeps that rival). But under
+revision 7's recompute the product `1e-28` is found exact after its zeros are stripped.
+So multiplying back by `Num`'s own `*` accepts the ADR's counterexample. In 20 runs of the
+division property (5,120 random cases plus the three seeds), it agreed with the direct
+test every time. That was an experiment and is not committed. The direct test stays the
+rule, because the ADR decided it and because it needs no second division.
+
+### `POW`
+
+- **The exponent.** It is exact when it is an exact **integer value**, so `2.0` counts.
+- **The size, decided before any multiplication.** The result's numerator has at least
+  `|n|·(bits(num) − 1)` bits, and likewise its denominator. If either lower bound passes
+  the ceiling, the result is `powf`, inexact, with nothing allocated. So
+  `POW(1/3, 10^9)` costs nothing. Inside the bound the result is computed exactly, and it
+  can still exceed the ceiling (`3^6000` is 9,510 bits). It then demotes, which is the
+  ceiling's stated exception.
+- **The scale.** A positive power of a decimal takes clause 4's stored scale with nominal
+  `s·n`, the scale repeated multiplication would give, so `1.50^2` is `2.2500`. A
+  negative power is a division and has its minimal scale, **unit bases included**:
+  `1.0^-2` is `1` and `(-1.0)^-3` is `-1`, as `2.0^-2` is `0.25`. Until the review of
+  `302594bf` the unit-base arm padded every power to `s·|n|`, so `1.0^-2` read `1.00`
+  (Copilot on PR #337). The power property now checks the scale of every decimal result
+  against this rule, with unit bases among its seeds.
+- **The edge values.** `0^0` is `1`. `0^-n` is `div/0`. `±1^n` is `±1` for any `n`,
+  including one too large to compute.
+- **A demoted power takes its sign from the exact exponent.** `powf` gets the base's
+  magnitude, and the sign is set from `base < 0 && n odd`. An exponent above 2^53 rounds
+  to an even f64, so handing `powf` the signed base made `(-2)^(2^53 + 1)` read `+inf`
+  (Copilot on PR #337).
+
+### The ceiling
+
+`CEILING_BITS = 8,192`. It applies to the bit length of an intermediate's numerator *or*
+denominator. The ADR states the ceiling as a width, not as which of the two it measures.
+Past it, the intermediate becomes the correctly rounded f64 of its exact value.
+
+## Comparison
+
+- **Exact with exact.** Cross-multiplication in 256 bits. A decimal is `m / 10^s` with
+  `10^28 < 2^94`, so every product fits.
+- **Exact with inexact.** The ADR's method: convert the exact operand, and the rounded
+  value decides unless it lands on the f64 itself. A tie is resolved against the f64's
+  exact binary value, in a bigint.
+- **That fast path is sound only because the conversion is correctly rounded.** Injecting
+  `rust_decimal`'s `to_f64` failed `exact_mixed_comparison_is_transitive` as well as the
+  conversion test. A future integer conversion (clause 4 allows one) must keep that.
+
+`MIN` and `MAX` skip NaN. Equal values break their tie as follows. Only the first rule is
+the ADR's (question 11); the other two are this crate's, and they make the result
+independent of argument order:
+- the exact value is preferred to the inexact one (question 11);
+- of two equal decimals, the larger scale is kept, for the same reason: it keeps more of
+  what was typed;
+- of `0.0` and `-0.0`, `MIN` takes `-0.0` and `MAX` takes `0.0`, as IEEE 754's
+  `minimum` and `maximum` do.
+
+## The number language
+
+**The gate is literally `f64::from_str` reading a finite value.** The parse is the
+crate's own lexer, and it never goes through an f64 on an exact path. The property test
+checks that the two agree, and that the exact value read rounds to the f64
+`f64::from_str` reads.
+
+- **A lexeme's scale.** It is the scale the lexeme implies (`1.50` → 2, `1.5e3` → 0). When
+  the implied scale is above 28 but the value fits a decimal (`0.1` written with 29
+  places), clause 4's stored-scale rule applies: strip the zeros, then pad toward 28.
+  The ADR only says "with the scale the lexeme implies", which cannot exceed 28.
+- **Zero.** It is exact, unsigned, at the implied scale clamped to 0–28, so
+  `0e99999999999999999999` is `0`.
+- **An exponent too long to read** saturates, because no exact arm can reach it. A
+  significand of more than 180 digits, or a leading digit outside 10^-130 to 10^40, goes
+  straight to f64. Those bounds are wider than the fraction arm can reach (a reduced
+  denominator `10^k / 2^a` is at least `2^k`, so `k < 127`).
+- **The forced `n/d` arm.** Its value is a decimal, then a fraction, then **the nearest
+  f64**. The ADR says `n/d` "is a fraction or, once normalized, a decimal", and does not
+  say what a forced `n/d` too large for `i128` becomes. The crate applies clause 3's order
+  to it, as it does to every other spelled number. This is a reading, and it is flagged.
+  **Except that the nearest f64 must be finite.** A forced `n/d` whose value rounds to
+  infinity (`n` of 400 digits, `/1`) is text, exactly as the gate makes `1e999` text.
+  Clause 3: "a finite lexeme must never read as infinity". One that underflows is the
+  inexact zero, as `1e-999` is. `is_number` and `parse_number` share the one helper,
+  `forced_fraction`, so the gate and the parse cannot disagree on it.
+
+**The canonical text is `Display`.** An inexact finite value is `#i` followed by Rust's
+`{:?}` spelling, which is the shortest that reads back (`#i1e300`, `#i0.1`).
+`from_canonical` reads all six spellings back, and reads an exact spelling only as an
+exact value. Clause 14's list puts "`#i…`" in the crate, while its table says "the `#i`
+prefix and the grammar are improvise's". Those two contradict each other. The crate
+holds the prefix, because the wire and the file both need the one spelling, and step 7
+may move the prefix out if the grammar wants it.
+
+## Rounding to places
+
+Every arm goes through one exact path: the value as a bigint fraction,
+`n·10^places / d`, then rounded by the caller's mode (`HalfAwayFromZero` or `HalfEven`).
+An inexact value is first read exactly from its shortest round-trip spelling, which is
+improvise-felo.9's fix, so a typed-then-computed `1.005` shows `1.01`. A result that
+rounds to zero has no sign (`-0.001` at two places is `0.00`). The ADR does not decide
+that, and today's f64 path would print `-0.00`.
+
+## Sums
+
+`NumSum` keeps five things, and every one is an abelian group, so the state is a function
+of the multiset of terms and not of their order.
+
+| part | holds | width |
+|---|---|---|
+| the decimal partition | every decimal, and every fraction whose denominator is `2^p·5^q` with `k <= 28`, as an integer at scale 28 | 320 bits |
+| the groups | per other reduced denominator, the summed numerator | 256 bits each, in a `BTreeMap` |
+| `group_floats` | an `ExactSum` of each group's correctly rounded value | ADR 009's |
+| `inexact` | an `ExactSum` of the f64 terms | ADR 009's |
+| `inexact_terms` | the net weight per f64 bit pattern, entries at zero dropped | a `BTreeMap` |
+
+**The read.** It is exact when no inexact term is present and at most K = 8 groups
+remain. The groups and the decimal partition are then combined **over the lcm of their
+denominators, and reduced once**, and the stored bound applies to the result. Past K the
+read is the f64 accumulators: `group_floats`, the decimal partition converted once, and
+the inexact terms. That reading rounds each group, then the total. With inexact terms and
+at most K groups, the exact total is converted once and added to the inexact terms. A
+group whose numerator returns to zero is dropped, so "how many groups" is a function of
+state.
+
+**"Is an inexact term present" is asked per value, never of the net weight.** The first
+version kept one net count. Inserting `1.0` and retracting `2.0` leaves the count at zero
+while the accumulator holds −1.0, so the read took the exact branch and dropped it
+(ChatGPT Codex, P1 on PR #337). Presence is now a map from f64 bit pattern to net weight,
+dropped at zero, the same rule the groups follow. It costs O(distinct inexact values),
+the same order as the groups. The K cutoff overrides representability: nine denominators whose exact total fits
+still read inexact (`more_than_k_denominators_read_inexact_even_when_the_total_fits`).
+That is question 13's second exception, and the ADR's flagged reading.
+
+**`ExactSum` is ADR 009's prototype**, from `crates/improvise-core/examples/retraction_drift.rs`
+on PR #309. It has three changes:
+- it splits a weight into base-2^30 digits, as ADR 009 says the `numtower` version must,
+  instead of looping per chunk;
+- it holds 70 limbs, not 67, because a digit's shift adds up to 60 bits;
+- it reads through the crate's one rational-to-f64 rounding.
+
+It counts NaN and the infinities, so they can be retracted. A count that is nonzero in
+either direction is treated as present.
+
+**`ExactSum`'s `==` compares sums, not representations.** A derived equality compared the
+raw limbs and the `pending` carry counter, so `0.5 + 0.5 != 1.0`, an insert then retract
+was not equal to an empty sum, and the same weights added in another order compared
+unequal (Copilot on PR #337). For a type whose whole point is order-freedom, and which
+ADR 009's group newtype will wrap, that equality is a trap. It now compares the
+propagated limbs, which are canonical (every limb but the top in `[0, 2^32)`), plus the
+three special-value counts, and ignores `pending`. `NumSum` derives no equality: compare
+its `read()`.
+
+**For ADR 009's seam crate.** `NumSum` and `ExactSum` both have the shape a Z-set group
+needs: `add(&Num, i64)` / `add(f64, i64)`, `insert`, `retract`, `read`/`value`, `Clone`.
+`ExactSum` also has `absorb`, to merge two accumulators. Neither implements a group trait
+yet, because `zset` does not exist yet. The newtype of question 15 is where that goes.
+
+## The wide integers
+
+`Fixed<N>` is a two's-complement integer of `N` 64-bit limbs that never allocates. It is
+used for the clause 4 recompute (256 bits), exact comparison (256 bits), the decimal
+partition (320 bits) and the group numerators (256 bits). Clause 14 also names "256-bit
+helpers for the division reduction". Division reduces in `i128` when the products cannot
+overflow, and otherwise in a bigint. It does not have a hand-rolled 256-bit gcd. That
+path is the slow path, and it is measured below.
+
+## Dependencies
+
+The normal edges are `rust_decimal` 1.43.0 (std only), `num-rational` 0.4.2 (no default
+features), `num-bigint` 0.4.8, `num-integer` 0.1.47 and `num-traits` 0.2.19. The last two
+are direct dependencies, used for `gcd`/`lcm` and the numeric traits. They were already
+in the tree through `num-rational` and `num-bigint`, so they add no crate. The ADR names
+only the first three. There is no serde, and `cargo tree --edges normal` shows no
+`borsh`. `BigRational` is a **dev**-dependency feature only, for the oracle, so the
+oracle and the implementation do not share a rational type.
+
+## Measured
+
+On an Apple M4 Max (64 GB), rustc 1.98.1, `cargo run --release -p numtower --example
+bench`. Each range is two runs.
+
+| what | figure |
+|---|---|
+| SUM, 18k and 90k typed two-place values, ns per element | f64 0.53; `rust_decimal` fold 2.7–3.0; `Num` `+` fold 7.8–8.2; `NumSum` insert and read 7.9–8.1 |
+| per operation over 18,000 typed pairs, ns | `Num` add 3.1–3.3, mul 3.0–3.2, div 146–153; `rust_decimal` checked add 1.4–1.5, mul 1.2, div 24.7–25.3 |
+| `(Rev − Cost) / Rev`, ns per record | 151–153 |
+| `to_f64` of a decimal (the text path), ns | 36.7–37.1 |
+| `round_to_places(2)`, ns | 370–382 |
+| `NumSum` read, K distinct 126-bit denominators, numerators of about 190 bits | K = 2: 5.2–5.4 µs; K = 8: 41.4–42.0 µs |
+| `ExactSum`, 1M values | 1.78–1.81 ns per insert, 0.29 µs per read |
+
+Here is how these sit against the ADR:
+- **The grouped read.** At K = 8 it is inside the ADR's 52–127 µs. The ADR measured
+  250-bit numerators, and these are about 190 bits, because `NumSum` can only reach a
+  group's numerator through its public `add`.
+- **`ExactSum`.** It agrees with ADR 009's 2.0 ns per update and 0.58 µs per read.
+- **Division.** At about 150 ns it is above the ADR's *estimate* of 60–110 ns. Most
+  quotients of typed values do not terminate, so each one pays two `i128` gcds and the
+  `Box` of a `Fraction`.
+- **The rest.** The ADR's whole-recompute measurement at 18k and 90k cells waits for
+  step 3, as the ADR says.
+
+**The wasm32 cost** comes from a probe outside the repository, built exactly as the
+ADR's appendix builds one: a `cdylib`, `opt-level = "s"`, LTO, `wasm-opt -Oz
+--enable-bulk-memory --enable-nontrapping-float-to-int`, `gzip -9`. The baseline already
+parses and prints an f64.
+
+| probe | after `wasm-opt -Oz` | gzipped |
+|---|---|---|
+| f64 only | 48,331 B | 29,162 B |
+| the ADR's five operations (parse, `+`, `−`, `/`, print) through `numtower` | +89,375 B | **+36,714 B** |
+| plus `NumSum`, `cmp_exact` and `round_to_places` | +119,277 B | **+48,221 B** |
+
+The narrow figure confirms the ADR's estimate of "up to about +36 KB together". The full
+surface is about +47 KB gzipped. That is what the bundle pays once step 4 sums through
+the crate and step 6 rounds through it. It is more than the +15.4 KB the user answered
+question 8 against, and the ADR says question 8 "reopens" if the figure changes the
+answer. Neither figure is the real bundle, which shares allocator and formatting code
+with everything else.