fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #203 from fiddlerwoaroof/collapse-fu/eccl

Browse Source 
improvise-eccl/f2oo: a fold evaluates formulas over itself and covers only drawn members
This commit is contained in:
Ed Langley
2026-09-23 08:28:26 -07:00
committed by GitHub
parent 194d8f529c 8c797720ba
commit e406edc8e8
9 changed files with 702 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
context/code/core.md
+69 -13
View File
@@ -869,11 +869,14 @@ an involution without anything being taught about involution, and neither
### The cell
`Model::evaluate_aggregated` expands, at its head, any coordinate that names a
group rather than an item into one key per member (a cross product where several
coordinates do). Doing it in the **model** is what makes `display_text`, prune,
column totals, `grid_of`, search, yank and CSV export all follow with no change.
Three decisions:
`Model::evaluate_covering` expands, at its head, any coordinate that names a group
rather than an item into one key per covered member (a cross product where
several coordinates do). The grid reads every pivot key through
`GridLayout::evaluate`, which passes the members it draws. Since
improvise-eccl/f2oo, display text, pruning, column totals, `grid_of`, search,
yank and CSV export all go through that one evaluation, rather than each calling
`Model::evaluate_aggregated`. That remains the view-less answer, and it covers
every member. Five decisions:
- **Membership is read off each item's own `group` field, never off
`Category::groups`.** `add_item_in_group` registers no `Group`; that list is
@@ -882,14 +885,67 @@ Three decisions:
- **An item with the same name as the group wins.** `items.contains_key`
short-circuits the expansion, so an existing document with an item `Q1` reads
exactly as before.
- **Stored values fold once over every record the group covers; a derived value
folds its members' results.** Folding the members' *aggregates* is right for
`Sum` and wrong for `Avg`/`Count` under `Axis::None` — `avg(avg(2,4), 9) = 6`
where the group's average is `5`. A formula target has no stored records to
pool, so its members' evaluated results are folded with the measure's
aggregation; for a ratio formula that is the usual pivot-table limitation, not
a new one. Hidden items are included in the fold, as on `Axis::None`: hiding
decides what is drawn, not what an aggregate covers.
- **Stored values fold once over every record the group covers.** Folding the
members' *aggregates* is right for `Sum` and wrong for `Avg`/`Count` under
`Axis::None`: `avg(avg(2,4), 9) = 6` where the group's average is `5`.
- **A formula is evaluated over the fold, never folded itself** (improvise-eccl).
This bullet used to say a formula target "has no stored records to pool", so
its members' results were folded with the measure's aggregation, and it called
that "the usual pivot-table limitation". **The premise was wrong.** A formula's
*inputs* are stored and can be pooled, and the model already evaluated a
formula over pooled inputs whenever a category sat on `Axis::None`:
`recompute_formulas` strips the None categories from the stems. Folding the
members' results summed ratios instead, so `Margin = (Revenue - Cost) / Revenue`
at 0.5 and 0.9 showed **1.4** on the Q1 row where Month-on-None showed **0.8**.
Spreadsheet calculated fields do not have that limitation either. The formula
is now evaluated at the group key, with each reference read through the fold
(`Reach::Fold` in `model/types.rs`):
- a raw measure reads the stored fold;
- another formula target is evaluated over the fold in turn;
- an aggregate inside the formula (`SUM`, `MAX`, …) pools every covered
member's records. `MAX(Revenue)` had been *summing* the members' maxima.
The collapsed row now agrees with the None-axis answer. **Test with a ratio**:
a sum agrees under both rules, which is how this was missed.
- **The folded category reads as absent** (improvise-lnkb, the orchestrator's
ruling). `Month IS Q1` is false on the Q1 fold, and a formula's own
`WHERE Month = "Q1"` does not apply there. Both answers are what the None axis
gives, and what an inline `SUM(… WHERE Month = "Q1")` already gave. This
PR's first version said `IS` saw the group's name and called that "stated
rather than hidden". That made the fold row the one place a formula could
observe collapse state, breaking this section's own "fold = None over the
drawn members" rule. `Model::read_coordinate` is the one reading of a
coordinate by name at a fold.
- A formula with its own `WHERE Month = "Jan"` shows **empty** on the Q1 fold,
because the fold is not Jan. The None axis gives the same.
- **A fold evaluates each formula once** (improvise-o1y0). A reference to
another formula target is evaluated over the fold in turn, so without memory
a doubling chain re-evaluates exponentially. The first version did exactly
that: a chain of eight took 511 formula evaluations for one cell, and the
reviewer measured 9.83 s on an 18k-cell grid. `evaluate_covering` now carries
a memo for one evaluation, and a key met while still in progress is reported
`circular`. Measured on an Apple M4 Max in release, on 100 products × 12
months folded to quarters with the F0…F8 chain (4,000 cells): 0.94–1.04 s
without the memo, about 60 ms with it, and about 10 ms on gh/main. gh/main
was cheap because it read the members out of the formula cache, which is
what gave the wrong ratios. The remaining ~6× is each column's cell
re-deriving the chain below it (45 evaluations per row where 9 would do). A
memo scoped to the whole grid would remove it; it is not built.
- **A fold covers the members the grid draws** (improvise-f2oo). Hidden members
are not in it. So a fold is exactly the rows it replaces, the total of a
**stored** measure under Sum or Count does not move on `z`, and a group whose
every member is hidden has no row. Under Avg, Min or Max a stored measure's
total does move, and so does a **formula** column's (PR #203's review measured
Margin 1.90 expanded, 1.30 collapsed). The strip sums the rows' values rather
than folding the column the way a fold does, so it sums a Max row's maximum
and a formula row's result. That is improvise-wm20, and it is not this rule. That last
point was already true, and it contradicted the old rule, under which a hidden
value appeared only while a sibling was visible. This deliberately **differs
from `Axis::None`**, where hidden items still count, because the two replace
different things. A fold replaces *rows the user sees*. A None axis replaces a
*dimension*, and hiding an item on it hides nothing that was drawn. The view's
hidden set reaches the model as the `covers` predicate `GridLayout::evaluate`
passes, since the model knows no view.
### Settability and drill
context/test-evidence/command.md
+11
View File
@@ -5712,6 +5712,17 @@ cannot see the feature removed from `improvise-command` alone, because
`improvise-io`, a normal dependency, enables it in the same build. That case is
held by `every_crate_that_can_parse_json_enables_float_roundtrip` (`tooling.md`).
### the_grid_and_a_csv_export_agree_on_a_fold_row
The projection-level pin PR #203's review asked to keep. The review's own
scratch test is made permanent here. Q1 = {Jan, Feb, Mar} with March hidden and
funded (50/50), so a fold that covered hidden members would read 90 / 58 / 0.36.
`grid_of` (what a client is sent) and `export_csv_to` (what a file gets) must
both read `40.00, 8.00, 0.80` on the Q1 row. The two go through different
producers (`viewport::grid_body` and `render_csv`), and both reach
`GridLayout::evaluate`. Rival: that evaluation covering every member. This test
fails with `["90.00", "58.00", "0.36"]`.
### p_onto_a_sparse_record_writes_it
improvise-q9eg, a regression introduced by PR #197 and found in its review. In
context/test-evidence/core.md
+80 -2
View File
@@ -1346,8 +1346,8 @@ this fixture and the value is absent — this test, the next, and
Under `Axis::None` (`Store`), `Avg` and `Count` over `Jan/A=2, Jan/B=4, Feb/A=9`.
The group's average is `5` and its count `3`; folding the members' aggregates gives
`avg(3, 9) = 6` and `count(2, 1) = 2`. Rival: always fold members' results (the
formula-target branch taken for stored values) — this test fails. `Sum` cannot see
`avg(3, 9) = 6` and `count(2, 1) = 2`. Rival: always fold members' results (what the
formula-target branch did before improvise-eccl) — this test fails. `Sum` cannot see
the difference, which is why the fixture sets `measure_agg`.
### model::types::model_tests::an_item_named_like_a_group_is_read_as_the_item
@@ -1412,3 +1412,81 @@ are exercised. It also lists the key's coordinates out of sorted order, which ch
that the steps follow `CellKey`'s canonical order. Rival: `Axis::Page` in place of
`Axis::Filter` for a fixed coordinate, which reopens improvise-dz8 (a drilled
coordinate the user could page away from). This test fails.
### a_folds_ratio_formula_is_the_ratio_of_its_folded_inputs
improvise-eccl. `Margin = (Revenue - Cost) / Revenue` with Jan `10/5` and Feb `30/3`
under `Q1`, and Apr in `Q2` with no data, so the Q1 fold is row 1. The member ratios
are 0.5 and 0.9, so the three candidate rules give three different answers: summed
1.4, averaged 0.7, pooled `(40 - 8) / 40` 0.8. The expected value is the **pooled**
one, and it is also asserted equal to the same document with Month on `Axis::None`,
which is the ticket's criterion. The None-axis document works because Apr carries
no data, so the None axis pools exactly Q1's months. At `a164726f` the fold read
**1.4**. The ticket's own fixture (0.5 and 0.5) would have separated sum from
pool, but not mean from pool. The second formula set chains
(`Profit = Revenue - Cost`, `Margin = Profit / Revenue`), so a reference to
another formula target is read over the fold as well. A rule that folds a formula
ref's members while pooling raw refs fails there.
### a_folds_aggregate_formula_pools_its_members_records
The guard on the other side of improvise-eccl. Once a formula at a fold is
evaluated at the group key, an aggregate inside it (`SUM`, `MAX`) must pool every
covered member's records, because `Month=Q1` matches no record. `MAX(Revenue)` is
the discriminating case. It **failed before the change too**: the old rule folded
the members' `MAX`es with the measure's `Sum`, giving 40 where the answer is 30.
`SUM` alone agrees under the old rule and the new one. Rival: `contexts_of` stops
pooling at a fold (`Reach::Fold { .. } => Vec::new()`). This test alone fails.
### collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was
improvise-f2oo. `Q1 = {Jan 3, Feb 4}`, `Q2 = {Apr 100}`, Feb hidden. Expanded, the
drawn rows total 103. At `a164726f` collapsing gave a Q1 fold of 7 and a total of
**107**: `z` moved the total. The ruling: a fold covers the members the grid
draws. The test asserts 103 both ways and a fold of 3. It then hides Jan too and
asserts that there is no Q1 row and a total of 100. That half already held, and it
is the inconsistency the ticket named: before, a hidden value appeared only while
a sibling was visible. Rival: `GridLayout::evaluate` covering every member
(`&|_, _| true`). This test alone fails, with 107.
### a_folds_aggregate_filtered_on_the_folded_category_counts_each_record_once
Found by the ChatGPT Codex review of PR #203, at head `14acf46d`. Pooling an
aggregate over a fold's member contexts ran `agg_values` once per member, and an
inline `WHERE` on the **folded** category overwrites that coordinate, as it does
at a record. Every member therefore became the same record set.
`SUM(Revenue WHERE Month = "Jan")` on the Q1 fold read **20**, Jan twice. The
test asserts 10, and equality with the Month-on-None document. The fix dedupes
the member contexts after the filter is applied, so each record set is pooled
once. The rival is the pre-fix code, which read 20. `a_folds_aggregate_formula_pools_its_members_records`
could not see this, because none of its aggregates carried a filter.
### a_fold_evaluates_each_formula_it_reaches_once
improvise-o1y0, from PR #203's review. A reference from a fold's formula to
another formula target is evaluated over the fold in turn, with nothing
remembered, so a doubling chain `F{i} = F{i-1} + F{i-1}` re-evaluated
exponentially. The test **counts** formula evaluations rather than timing them,
through a test-only thread-local counter. At `2a4f35d7` evaluating `F8` on one
fold took **511** evaluations (2⁹ − 1); with the memo it takes 9. It also
asserts the chain's value (7 × 256), so a memo that returned the wrong entry
cannot pass. Rival: keep the in-progress marker but drop finished values from
the memo (`.filter(Option::is_none)`). This test fails with 511.
### a_cycle_at_a_fold_is_still_circular
improvise-o1y0's other half. `P = Q + 1`, `Q = P + 1` at a fold must read
`Error("circular")`. The memo's in-progress marker is what answers it now,
because the depth bound the fold path used before is gone. Without the marker
the evaluation recurses until the stack runs out, which is why the marker is
there rather than being an optimisation.
### a_fold_answers_is_and_where_on_its_category_as_the_none_axis_does
improvise-lnkb, ruled by the orchestrator: on a fold, the folded category reads
as absent. `Flag = Month IS "Q1"` and `Marked = Revenue * 2 WHERE Month = "Q1"`
are each compared with the same document with Month on `Axis::None`. At
`689c7665` `Flag` read **1** on the fold and 0 on None. The final assertion pins
`Flag` at `Some(0.0)`, false rather than absent, so a fix that dropped the cell
cannot pass. Rival: `read_coordinate` ignoring the reach (`if false && …`).
This test fails.
crates/improvise-command/src/command/cmd/cell.rs
+1 -4
View File
@@ -1,4 +1,3 @@
use crate::model::Aggregation;
use crate::model::cell::{CellKey, CellValue};
use crate::ui::effect::{self, Effect, ViewTarget};
use crate::view::synthetic_record_info;
@@ -585,9 +584,7 @@ impl Cmd for YankCell {
// A records cell is the layout's one resolution, as the browser yanks it (gotcha 1).
let value = match synthetic_record_info(&self.key) {
Some((row, col)) => ctx.layout.records_value(row, col),
None => ctx
.model
.evaluate_aggregated(&self.key, Aggregation::of(ctx.none_cats())),
None => ctx.layout.evaluate(ctx.model, &self.key),
};
vec![
Box::new(effect::SetYanked(value)),
crates/improvise-command/src/command/cmd/search.rs
+1 -5
View File
@@ -1,4 +1,3 @@
use crate::model::Aggregation;
use crate::model::cell::CellValue;
use crate::ui::app::AppMode;
use crate::ui::effect::{self, Effect, Panel};
@@ -113,10 +112,7 @@ impl Cmd for SearchNavigate {
Some(k) => k,
None => return false,
};
let s = match ctx
.model
.evaluate_aggregated(&key, Aggregation::of(ctx.none_cats()))
{
let s = match ctx.layout.evaluate(ctx.model, &key) {
Some(CellValue::Number(n)) => format!("{n}"),
Some(CellValue::Text(t)) => t,
Some(CellValue::Error(e)) => format!("ERR:{e}"),
crates/improvise-command/src/ui/project.rs
+60
View File
@@ -5293,6 +5293,66 @@ mod cell_identity {
"another record's basis was invalidated for nothing"
);
}
/// improvise-eccl/f2oo at the projection boundary: the grid a client is sent and
/// the CSV an export writes agree on a fold row — see the evidence file.
#[test]
fn the_grid_and_a_csv_export_agree_on_a_fold_row() {
let mut wb = Workbook::new("T");
wb.add_category("Month").unwrap();
wb.add_item_in_group("Month", "Apr", "Q2").unwrap();
for month in ["Jan", "Feb", "Mar"] {
wb.add_item_in_group("Month", month, "Q1").unwrap();
}
for (month, revenue, cost) in [("Jan", 10.0, 5.0), ("Feb", 30.0, 3.0), ("Mar", 50.0, 50.0)]
{
for (measure, n) in [("Revenue", revenue), ("Cost", cost)] {
wb.add_item("_Measure", measure).unwrap();
let key = CellKey::new(vec![
("Month".into(), month.into()),
("_Measure".into(), measure.into()),
]);
wb.model.set_cell(key, CellValue::Number(n));
}
}
wb.model.add_formula(
crate::formula::parse_formula("Margin = (Revenue - Cost) / Revenue", "_Measure")
.unwrap(),
);
wb.active_view_mut().set_axis("Month", Axis::Row);
wb.active_view_mut().set_axis("_Measure", Axis::Column);
wb.active_view_mut().hide_item("Month", "Mar");
wb.active_view_mut().toggle_group_collapse("Month", "Q1");
let app = App::new(wb, None);
let grid = grid_of(&app, WHOLE);
let q1 = grid.row_frame.leaf_labels().iter().position(|l| l == "Q1");
let q1 = q1.expect("a Q1 fold row");
let drawn: Vec<String> = (0..grid.col_frame.leaf_labels().len())
.map(|c| grid.cell(q1, c).map(|d| d.text.clone()).unwrap_or_default())
.collect();
let storage = crate::storage::MemoryFs::new();
let path = std::path::Path::new("out.csv");
crate::persistence::export_csv_to(&storage, &app.model_state.workbook, "Default", path)
.expect("the export writes");
let bytes = crate::storage::Storage::read(&storage, path).unwrap();
let csv = String::from_utf8(bytes).unwrap();
let exported: Vec<String> = csv
.lines()
.find_map(|l| l.strip_prefix("Q1,"))
.expect("a Q1 line")
.split(',')
.map(String::from)
.collect();
assert_eq!(
drawn,
["40.00", "8.00", "0.80"],
"March is hidden, so not folded"
);
assert_eq!(exported, drawn, "the export and the grid disagree:\n{csv}");
}
}
/// ADR 001's basis check, both arms per tier (improvise-xwi6/fkfg). The fixtures
crates/improvise-command/src/ui/viewport.rs
+2 -4
View File
@@ -6,8 +6,8 @@ use unicode_width::UnicodeWidthStr;
use improvise_protocol::header_level_widths;
use crate::format::{FormatPlan, NumberFormat};
use crate::model::Model;
use crate::model::cell::{CellKey, CellValue};
use crate::model::{Aggregation, Model};
use crate::view::{AxisEntry, GridLayout};
/// Minimum column width — enough for short numbers/labels + 1 char gap.
@@ -110,9 +110,7 @@ fn cell_value(
true => (None, layout.records_value(row, col)),
false => {
let key = layout.cell_key(row, col);
let value = key
.as_ref()
.and_then(|k| model.evaluate_aggregated(k, Aggregation::of(&layout.none_cats)));
let value = key.as_ref().and_then(|k| layout.evaluate(model, k));
(key, value)
}
}
crates/improvise-core/src/model/types.rs
+289 -67
View File
@@ -51,17 +51,50 @@ fn member_records(model: &Model, partial: &CellKey) -> Vec<CellKey> {
seen.into_iter().collect()
}
/// A formula-level aggregate over the data matching `context` minus the
/// target. Deliberately *not* [`apply_agg`]'s emptiness rule — §2.
/// A formula-level aggregate over the data matching each of `contexts` minus the
/// target, pooled. Deliberately *not* [`apply_agg`]'s emptiness rule — §2.
fn eval_agg(
model: &Model,
context: &CellKey,
contexts: &[CellKey],
target_category: &str,
func: &AggFunc,
inner: &Expr,
agg_filter: Option<&Filter>,
eval_at: &dyn Fn(&Expr, &CellKey) -> Result<f64, String>,
) -> Result<f64, String> {
// A filter overrides its coordinate, so members it rewrites to one item name
// one record set: pool each distinct context once (core.md §14).
let distinct: IndexSet<CellKey> = contexts
.iter()
.map(|c| match agg_filter {
Some(f) => c.clone().with(&f.category, &f.item),
None => c.clone(),
})
.collect();
let values = distinct
.iter()
.map(|c| agg_values(model, c, target_category, inner, agg_filter, eval_at))
.collect::<Result<Vec<_>, String>>()?
.concat();
match func {
AggFunc::Sum => Ok(values.iter().sum()),
AggFunc::Count => Ok(values.len() as f64),
AggFunc::Avg | AggFunc::Min | AggFunc::Max => {
apply_agg(&values, func).ok_or_else(|| "empty".to_string())
}
}
}
/// The values an aggregate folds at one context: a stored ref's cells, or the
/// expression asked of each member record — §13.
fn agg_values(
model: &Model,
context: &CellKey,
target_category: &str,
inner: &Expr,
agg_filter: Option<&Filter>,
eval_at: &dyn Fn(&Expr, &CellKey) -> Result<f64, String>,
) -> Result<Vec<f64>, String> {
let mut partial = context.without(target_category);
// The fast path is the fold specialised to a ref that names a STORED
// cell; where it does not, the premise fails — §13, s08t, s2w3.
@@ -83,15 +116,15 @@ fn eval_agg(
if let Some(f) = agg_filter {
partial = partial.with(&f.category, &f.item);
}
let values: Vec<f64> = match select_raw {
match select_raw {
// A bare ref asks for the values that exist; anything else asks the
// expression of each member record and folds the answers — §13.
true => model
true => Ok(model
.data
.matching_values(&partial.0)
.into_iter()
.filter_map(|v| v.as_f64())
.collect(),
.collect()),
false => member_records(model, &partial)
.into_iter()
.map(|record| {
@@ -101,28 +134,48 @@ fn eval_agg(
.fold(partial.clone(), |k, (cat, item)| k.with(cat, item));
eval_at(inner, &at)
})
.collect::<Result<Vec<f64>, String>>()?,
};
match func {
AggFunc::Sum => Ok(values.iter().sum()),
AggFunc::Count => Ok(values.len() as f64),
AggFunc::Avg | AggFunc::Min | AggFunc::Max => {
apply_agg(&values, func).ok_or_else(|| "empty".to_string())
}
.collect(),
}
}
// How many formulas a fold has evaluated: what improvise-o1y0's regression counts.
#[cfg(test)]
thread_local! {
static FOLD_FORMULA_EVALUATIONS: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
}
/// One fold evaluation's memory: each formula key is evaluated once, and a key met
/// while still in progress (`None`) is a cycle (improvise-o1y0).
type FoldMemo = std::cell::RefCell<HashMap<CellKey, Option<Option<CellValue>>>>;
/// Which members of a collapsed group its fold covers. The model knows no
/// view, so the grid passes the members it draws (core.md §14).
pub type Covers<'a> = &'a dyn Fn(&str, &str) -> bool;
/// Where a formula's references are read: at one record, or across the members
/// a collapsed group's fold covers, as a None axis reads them (core.md §14).
#[derive(Clone, Copy)]
enum Reach<'a> {
Record,
Fold {
agg: Aggregation,
covers: Covers<'a>,
memo: &'a FoldMemo,
},
}
/// The **record-level** evaluator: no cache, because a member key is not a stem
/// the cache holds. Errors propagate immediately — §3, §13.
/// the cache holds. Errors propagate immediately — §3, §13. `reach`: §14.
fn eval_expr(
expr: &Expr,
context: &CellKey,
model: &Model,
target_category: &str,
depth: u8,
reach: Reach,
) -> Result<f64, String> {
use crate::formula::BinOp;
let eval = |e: &Expr| eval_expr(e, context, model, target_category, depth, reach);
match expr {
Expr::Number(n) => Ok(*n),
Expr::Ref(name) => {
@@ -130,15 +183,21 @@ fn eval_expr(
.find_item_category(name)
.ok_or_else(|| format!("ref:{name}"))?;
let new_key = context.clone().with(cat, name);
match model.evaluate_depth(&new_key, depth) {
let value = match reach {
Reach::Record => model.evaluate_depth(&new_key, depth),
Reach::Fold { agg, covers, memo } => {
model.evaluate_fold(&new_key, agg, covers, memo)
}
};
match value {
Some(CellValue::Number(n)) => Ok(n),
Some(CellValue::Error(e)) => Err(e),
_ => Err(format!("ref:{name}")),
}
}
Expr::BinOp(op, l, r) => {
let lv = eval_expr(l, context, model, target_category, depth)?;
let rv = eval_expr(r, context, model, target_category, depth)?;
let lv = eval(l)?;
let rv = eval(r)?;
match op {
BinOp::Add => Ok(lv + rv),
BinOp::Sub => Ok(lv - rv),
@@ -153,22 +212,26 @@ fn eval_expr(
}
}
}
Expr::UnaryMinus(e) => Ok(-eval_expr(e, context, model, target_category, depth)?),
Expr::UnaryMinus(e) => Ok(-eval(e)?),
// A fold's aggregate pools every member record it covers, never the
// members' own aggregates (§14).
Expr::Agg(func, inner, agg_filter) => eval_agg(
model,
context,
&model.contexts_of(context, reach),
target_category,
func,
inner.as_ref(),
agg_filter.as_ref(),
&|e, at| eval_expr(e, at, model, target_category, depth),
&|e, at| eval_expr(e, at, model, target_category, depth, Reach::Record),
),
Expr::Is(filter) => Ok(eval_is(context, filter)),
Expr::Is(filter) => Ok(eval_is(
model.read_coordinate(context, &filter.category, reach),
filter,
)),
Expr::If(cond, then, else_) => {
let cv = eval_bool(cond, context, model, target_category, depth)?;
match cv {
true => eval_expr(then, context, model, target_category, depth),
false => eval_expr(else_, context, model, target_category, depth),
match eval_bool(cond, context, model, target_category, depth, reach)? {
true => eval(then),
false => eval(else_),
}
}
}
@@ -182,12 +245,13 @@ fn eval_bool(
model: &Model,
target_category: &str,
depth: u8,
reach: Reach,
) -> Result<bool, String> {
use crate::formula::BinOp;
match expr {
Expr::BinOp(op, l, r) => {
let lv = eval_expr(l, context, model, target_category, depth)?;
let rv = eval_expr(r, context, model, target_category, depth)?;
let lv = eval_expr(l, context, model, target_category, depth, reach)?;
let rv = eval_expr(r, context, model, target_category, depth, reach)?;
match op {
BinOp::Eq => Ok((lv - rv).abs() < FLOAT_EQ_EPSILON),
BinOp::Ne => Ok((lv - rv).abs() >= FLOAT_EQ_EPSILON),
@@ -211,10 +275,8 @@ fn eval_bool(
/// `Cat IS Item` as a number. A cell with no `Cat` coordinate is **not** that
/// item, matching `Formula::filter`'s own `unwrap_or(false)` — §13.
fn eval_is(context: &CellKey, filter: &Filter) -> f64 {
context
.get(&filter.category)
.is_some_and(|v| v == filter.item.as_str()) as i32 as f64
fn eval_is(coordinate: Option<&str>, filter: &Filter) -> f64 {
coordinate.is_some_and(|v| v == filter.item.as_str()) as i32 as f64
}
/// Whether a grid hides dimensions a cell key does not name. The emptiness of
@@ -523,33 +585,88 @@ impl Model {
}
pub fn evaluate_aggregated(&self, key: &CellKey, agg: Aggregation) -> Option<CellValue> {
let members = self.group_expansion(key);
if !members.is_empty() {
return self.evaluate_group(key, &members, agg);
self.evaluate_covering(key, agg, &|_, _| true)
}
/// [`Self::evaluate_aggregated`], with a collapsed group's fold covering only
/// the members `covers` admits — the ones a grid draws (core.md §14).
pub fn evaluate_covering(
&self,
key: &CellKey,
agg: Aggregation,
covers: Covers,
) -> Option<CellValue> {
self.evaluate_fold(key, agg, covers, &FoldMemo::default())
}
/// A key naming a group folds; any other key reads as before. A formula is
/// evaluated OVER the fold, never folded itself — why: core.md §14.
fn evaluate_fold(
&self,
key: &CellKey,
agg: Aggregation,
covers: Covers,
memo: &FoldMemo,
) -> Option<CellValue> {
let members = self.group_expansion(key, covers);
match (members.is_empty(), self.formula_for(key)) {
(true, _) => self.evaluate_unfolded(key, agg),
(false, Some(formula)) => {
self.fold_formula_once(formula, key, &members, (agg, covers, memo))
}
(false, None) => self.fold_stored(key, &members, agg),
}
}
/// A formula at a group key, evaluated at most once per fold evaluation; a key
/// already in progress is a cycle, so `circular` still comes out (improvise-o1y0).
fn fold_formula_once(
&self,
formula: &Formula,
key: &CellKey,
members: &[CellKey],
(agg, covers, memo): (Aggregation, Covers, &FoldMemo),
) -> Option<CellValue> {
let seen = memo.borrow().get(key).cloned();
match seen {
Some(Some(value)) => value,
Some(None) => Some(CellValue::Error("circular".into())),
None => {
memo.borrow_mut().insert(key.clone(), None);
#[cfg(test)]
FOLD_FORMULA_EVALUATIONS.with(|n| n.set(n.get() + 1));
let reach = Reach::Fold { agg, covers, memo };
let value = self.eval_formula_folded(formula, key, members, agg, reach);
memo.borrow_mut().insert(key.clone(), Some(value.clone()));
value
}
}
}
/// A key that names no group: exact, the formula cache, or raw aggregation.
fn evaluate_unfolded(&self, key: &CellKey, agg: Aggregation) -> Option<CellValue> {
if agg == Aggregation::Exact {
return self.evaluate(key);
}
// Check the formula cache first.
if let Some(cached) = self.formula_cache.get(key) {
return Some(cached.clone());
}
// Aggregate raw data across all None-axis categories.
aggregate_cell_values(&self.data.matching_values(&key.0), self.agg_for(key))
}
/// One key per member where `key` names a collapsed GROUP rather than an item —
/// the cross product over every such coordinate. Empty when none does (§14).
fn group_expansion(&self, key: &CellKey) -> Vec<CellKey> {
/// One key per covered member where `key` names a collapsed GROUP rather than
/// an item — the cross product over every such coordinate. Empty when none does.
fn group_expansion(&self, key: &CellKey, covers: Covers) -> Vec<CellKey> {
let mut expanded = vec![key.clone()];
for (cat_name, name) in &key.0 {
let members = self
let members: Vec<&str> = self
.category(cat_name)
.filter(|cat| !cat.items.contains_key(name.as_str()))
.map(|cat| cat.items_in_group(name))
.unwrap_or_default();
.unwrap_or_default()
.into_iter()
.filter(|m| covers(cat_name, m))
.collect();
if members.is_empty() {
continue;
}
@@ -564,29 +681,85 @@ impl Model {
}
}
/// A group's cell: stored values fold ONCE over every record the group covers,
/// a derived one folds its members' results — why the two differ: core.md §14.
fn evaluate_group(
/// A coordinate as a formula reads it. At a fold a group's name is no item, so
/// the folded category reads as absent, as on `Axis::None` (improvise-lnkb).
fn read_coordinate<'k>(
&self,
context: &'k CellKey,
cat: &str,
reach: Reach,
) -> Option<&'k str> {
let value = context.get(cat)?;
match reach {
Reach::Fold { .. } if self.names_a_group(cat, value) => None,
Reach::Fold { .. } | Reach::Record => Some(value),
}
}
/// Whether `name` is a group of `cat` rather than one of its items.
fn names_a_group(&self, cat: &str, name: &str) -> bool {
self.category(cat)
.is_some_and(|c| !c.items.contains_key(name) && !c.items_in_group(name).is_empty())
}
/// The contexts an aggregate inside a formula pools over at `context`.
fn contexts_of(&self, context: &CellKey, reach: Reach) -> Vec<CellKey> {
let members = match reach {
Reach::Record => Vec::new(),
Reach::Fold { covers, .. } => self.group_expansion(context, covers),
};
match members.is_empty() {
true => vec![context.clone()],
false => members,
}
}
/// A group's stored values fold ONCE over every record the group covers.
fn fold_stored(
&self,
key: &CellKey,
members: &[CellKey],
agg: Aggregation,
) -> Option<CellValue> {
let values: Vec<CellValue> = match self.formula_for(key) {
Some(_) => members
.iter()
.filter_map(|k| self.evaluate_aggregated(k, agg))
.collect(),
None => members
.iter()
.flat_map(|k| match agg {
Aggregation::Exact => self.data.get(k).into_iter().collect(),
Aggregation::OverHidden => self.data.matching_values(&k.0),
})
.cloned()
.collect(),
};
aggregate_cell_values(&values.iter().collect::<Vec<_>>(), self.agg_for(key))
let values: Vec<&CellValue> = members
.iter()
.flat_map(|k| match agg {
Aggregation::Exact => self.data.get(k).into_iter().collect(),
Aggregation::OverHidden => self.data.matching_values(&k.0),
})
.collect();
aggregate_cell_values(&values, self.agg_for(key))
}
/// A formula at a group key, its refs read through the fold — the value the
/// same formula has with the group's category on `Axis::None` (§14).
fn eval_formula_folded(
&self,
formula: &Formula,
key: &CellKey,
members: &[CellKey],
agg: Aggregation,
reach: Reach,
) -> Option<CellValue> {
let applies = formula
.filter
.as_ref()
.is_none_or(|f| self.read_coordinate(key, &f.category, reach) == Some(f.item.as_str()));
if !applies {
return self.fold_stored(key, members, agg);
}
let depth = Self::MAX_EVAL_DEPTH;
match eval_expr(
&formula.expr,
key,
self,
&formula.target_category,
depth,
reach,
) {
Ok(n) => Some(CellValue::Number(n)),
Err(e) => Some(CellValue::Error(e)),
}
}
/// [`Self::evaluate_aggregated`] as `f64`, with `0.0` for an empty cell.
@@ -744,16 +917,19 @@ impl Model {
)?),
Expr::Agg(func, inner, agg_filter) => eval_agg(
model,
context,
std::slice::from_ref(context),
target_category,
func,
inner.as_ref(),
agg_filter.as_ref(),
// The fold evaluates at a MEMBER key, which is not a stem the
// cache holds — so the record-level evaluator, in both paths.
&|e, at| eval_expr(e, at, model, target_category, Model::MAX_EVAL_DEPTH),
&|e, at| {
let depth = Model::MAX_EVAL_DEPTH;
eval_expr(e, at, model, target_category, depth, Reach::Record)
},
),
Expr::Is(filter) => Ok(eval_is(context, filter)),
Expr::Is(filter) => Ok(eval_is(context.get(&filter.category), filter)),
Expr::If(cond, then, else_) => {
let cv = eval_bool_cached(cond, context, model, target_category, none_cats)?;
if cv {
@@ -841,6 +1017,7 @@ impl Model {
self,
&formula.target_category,
depth,
Reach::Record,
) {
Ok(n) => Some(CellValue::Number(n)),
Err(e) => Some(CellValue::Error(e)),
@@ -891,7 +1068,17 @@ mod agg_helper_tests {
let empty = Expr::Ref("Revenue".to_string());
let ctx = CellKey::new(vec![]);
let never = |_: &Expr, _: &CellKey| unreachable!("a bare Ref never folds per record");
let run = |func| eval_agg(&model, &ctx, "_Measure", &func, &empty, None, &never);
let run = |func| {
eval_agg(
&model,
std::slice::from_ref(&ctx),
"_Measure",
&func,
&empty,
None,
&never,
)
};
assert_eq!(run(AggFunc::Sum), Ok(0.0));
assert_eq!(run(AggFunc::Count), Ok(0.0));
@@ -1493,6 +1680,41 @@ mod model_tests {
);
}
/// improvise-o1y0: a fold evaluates each formula it reaches once — see
/// context/test-evidence/core.md.
#[test]
fn a_fold_evaluates_each_formula_it_reaches_once() {
let mut m = quarter_model(&[("Jan", "A", 3.0), ("Feb", "A", 4.0)]);
m.add_formula(crate::formula::parse_formula("F0 = Revenue", "_Measure").unwrap());
for i in 1..=8 {
let text = format!("F{i} = F{p} + F{p}", p = i - 1);
m.add_formula(crate::formula::parse_formula(&text, "_Measure").unwrap());
}
let key = coord(&[("Month", "Q1"), ("Store", "A"), ("_Measure", "F8")]);
super::FOLD_FORMULA_EVALUATIONS.with(|n| n.set(0));
let value = m.evaluate_covering(&key, Aggregation::Exact, &|_, _| true);
let evaluations = super::FOLD_FORMULA_EVALUATIONS.with(|n| n.get());
assert_eq!(
value,
Some(CellValue::Number(7.0 * 256.0)),
"the chain's value"
);
assert_eq!(evaluations, 9, "F0..F8 once each, not 2^9 - 1");
}
/// improvise-o1y0's other half: a key met while still in progress is a cycle.
#[test]
fn a_cycle_at_a_fold_is_still_circular() {
let mut m = quarter_model(&[("Jan", "A", 3.0), ("Feb", "A", 4.0)]);
m.add_formula(crate::formula::parse_formula("P = Q + 1", "_Measure").unwrap());
m.add_formula(crate::formula::parse_formula("Q = P + 1", "_Measure").unwrap());
let key = coord(&[("Month", "Q1"), ("Store", "A"), ("_Measure", "P")]);
assert_eq!(
m.evaluate_covering(&key, Aggregation::Exact, &|_, _| true),
Some(CellValue::Error("circular".into()))
);
}
/// Evidence: context/test-evidence/core.md.
#[test]
fn a_groups_stored_values_fold_once_not_per_member() {
crates/improvise-core/src/view/layout.rs
+189 -3
View File
@@ -1,3 +1,4 @@
use std::collections::{HashMap, HashSet};
use std::rc::Rc;
use crate::model::category::{DIM_CATEGORY, INDEX_CATEGORY, MEASURE_CATEGORY};
@@ -377,6 +378,8 @@ pub struct GridLayout {
/// In records mode: what each column holds, parallel to `col_items`.
/// Empty for normal pivot views.
record_cols: Vec<RecordCol>,
/// The view's hidden items: a fold covers only the members this grid draws.
hidden: HashMap<String, HashSet<String>>,
}
impl GridLayout {
@@ -460,6 +463,7 @@ impl GridLayout {
none_cats,
records: None,
record_cols: Vec::new(),
hidden: view.hidden_items.clone(),
}
}
}
@@ -493,6 +497,7 @@ impl GridLayout {
none_cats,
records: Some(Rc::new(records)),
record_cols,
hidden: HashMap::new(),
}
}
@@ -611,7 +616,7 @@ impl GridLayout {
for (ci, cell) in row.iter_mut().enumerate() {
*cell = self
.cell_key(ri, ci)
.and_then(|k| model.evaluate_aggregated(&k, Aggregation::of(&self.none_cats)))
.and_then(|k| self.evaluate(model, &k))
.is_some();
}
}
@@ -649,6 +654,13 @@ impl GridLayout {
axis_label(&self.col_items, col)
}
/// The value of a pivot key as THIS grid reads it: aggregated over its None
/// axis, and a fold over the members it draws (core.md §14). One evaluation.
pub fn evaluate(&self, model: &Model, key: &CellKey) -> Option<CellValue> {
let drawn = |cat: &str, item: &str| !self.hidden.get(cat).is_some_and(|h| h.contains(item));
model.evaluate_covering(key, Aggregation::of(&self.none_cats), &drawn)
}
/// Display text for (row, col) in either mode.
pub fn display_text(
&self,
@@ -662,8 +674,7 @@ impl GridLayout {
} else {
self.cell_key(row, col)
.and_then(|key| {
let value =
model.evaluate_aggregated(&key, Aggregation::of(&self.none_cats))?;
let value = self.evaluate(model, &key)?;
Some(plan.at(Some(&key)).value(Some(&value)))
})
.unwrap_or_default()
@@ -1780,6 +1791,181 @@ mod tests {
);
}
/// Q2 = {Apr}, Q1 = {Jan, Feb}, so the Q1 fold is row 1. Jan and Feb carry
/// ratios 0.5 and 0.9 whose sum, mean and pooled ratio all differ; Apr is empty.
fn margin_book(formulas: &[&str]) -> Workbook {
let mut wb = Workbook::new("T");
wb.add_category("Month").unwrap();
wb.add_item_in_group("Month", "Apr", "Q2").unwrap();
wb.add_item_in_group("Month", "Jan", "Q1").unwrap();
wb.add_item_in_group("Month", "Feb", "Q1").unwrap();
wb.add_item("_Measure", "Revenue").unwrap();
wb.add_item("_Measure", "Cost").unwrap();
for (month, revenue, cost) in [("Jan", 10.0, 5.0), ("Feb", 30.0, 3.0)] {
for (measure, n) in [("Revenue", revenue), ("Cost", cost)] {
wb.model.set_cell(
CellKey::new(vec![
("Month".into(), month.into()),
("_Measure".into(), measure.into()),
]),
CellValue::Number(n),
);
}
}
for f in formulas {
wb.model
.add_formula(crate::formula::parse_formula(f, "_Measure").unwrap());
}
wb.active_view_mut().set_axis("Month", Axis::Row);
wb.active_view_mut().set_axis("_Measure", Axis::Column);
wb
}
/// `measure` at the row labelled `row_label`, as the grid reads it.
fn grid_value(wb: &mut Workbook, row_label: &str, measure: &str) -> Option<f64> {
wb.model.recompute_formulas(&wb.active_view().none_cats());
let layout = GridLayout::new(&wb.model, wb.active_view());
let row = (0..layout.row_count()).find(|&r| layout.row_label(r) == row_label)?;
let col = (0..layout.col_count()).find(|&c| layout.col_label(c) == measure)?;
layout
.evaluate(&wb.model, &layout.cell_key(row, col)?)
.and_then(|v| v.as_f64())
}
/// improvise-eccl — see context/test-evidence/core.md.
#[test]
fn a_folds_ratio_formula_is_the_ratio_of_its_folded_inputs() {
for formulas in [
&["Margin = (Revenue - Cost) / Revenue"][..],
&["Profit = Revenue - Cost", "Margin = Profit / Revenue"][..],
] {
let mut folded = margin_book(formulas);
folded
.active_view_mut()
.toggle_group_collapse("Month", "Q1");
let fold = grid_value(&mut folded, "Q1", "Margin").expect("a Q1 fold");
let mut none = margin_book(formulas);
none.active_view_mut().set_axis("Month", Axis::None);
let over_none = grid_value(&mut none, "", "Margin").expect("a None-axis cell");
assert!((fold - 0.8).abs() < 1e-9, "{formulas:?}: fold {fold}");
assert!(
(fold - over_none).abs() < 1e-9,
"{formulas:?}: {fold} vs {over_none}"
);
}
}
/// improvise-lnkb: the folded category reads as absent, as on `Axis::None` — see
/// context/test-evidence/core.md.
#[test]
fn a_fold_answers_is_and_where_on_its_category_as_the_none_axis_does() {
let formulas = [
"Flag = Month IS \"Q1\"",
"Marked = Revenue * 2 WHERE Month = \"Q1\"",
];
for measure in ["Flag", "Marked"] {
let mut folded = margin_book(&formulas);
folded
.active_view_mut()
.toggle_group_collapse("Month", "Q1");
let fold = grid_value(&mut folded, "Q1", measure);
let mut none = margin_book(&formulas);
none.active_view_mut().set_axis("Month", Axis::None);
let over_none = grid_value(&mut none, "", measure);
assert_eq!(
fold, over_none,
"{measure}: the fold and the None axis differ"
);
}
let mut folded = margin_book(&formulas);
folded
.active_view_mut()
.toggle_group_collapse("Month", "Q1");
assert_eq!(
grid_value(&mut folded, "Q1", "Flag"),
Some(0.0),
"false, not absent"
);
}
/// improvise-eccl's guard on the other side: an aggregate INSIDE the formula
/// still pools every record the fold covers.
#[test]
fn a_folds_aggregate_formula_pools_its_members_records() {
let mut wb = margin_book(&["Top = MAX(Revenue)", "Total = SUM(Revenue)"]);
wb.active_view_mut().toggle_group_collapse("Month", "Q1");
assert_eq!(grid_value(&mut wb, "Q1", "Total"), Some(40.0));
assert_eq!(grid_value(&mut wb, "Q1", "Top"), Some(30.0));
}
/// An inline `WHERE` on the folded category makes every member context the
/// same record set — see context/test-evidence/core.md.
#[test]
fn a_folds_aggregate_filtered_on_the_folded_category_counts_each_record_once() {
let formula = ["Jans = SUM(Revenue WHERE Month = \"Jan\")"];
let mut folded = margin_book(&formula);
folded
.active_view_mut()
.toggle_group_collapse("Month", "Q1");
let fold = grid_value(&mut folded, "Q1", "Jans");
let mut none = margin_book(&formula);
none.active_view_mut().set_axis("Month", Axis::None);
assert_eq!(fold, Some(10.0), "Jan's revenue, once");
assert_eq!(
fold,
grid_value(&mut none, "", "Jans"),
"the None-axis answer"
);
}
/// Every row's value in column 0, summed: the totals strip's arithmetic.
fn column_total(wb: &Workbook) -> f64 {
let layout = GridLayout::new(&wb.model, wb.active_view());
(0..layout.row_count())
.filter_map(|r| layout.evaluate(&wb.model, &layout.cell_key(r, 0)?))
.filter_map(|v| v.as_f64())
.sum()
}
/// improvise-f2oo — see context/test-evidence/core.md.
#[test]
fn collapsing_a_group_with_a_hidden_member_leaves_the_total_where_it_was() {
let mut wb = Workbook::new("T");
wb.add_category("Month").unwrap();
wb.add_category("Type").unwrap();
wb.add_item("Type", "Food").unwrap();
for (month, quarter, n) in [("Jan", "Q1", 3.0), ("Feb", "Q1", 4.0), ("Apr", "Q2", 100.0)] {
wb.add_item_in_group("Month", month, quarter).unwrap();
wb.model.set_cell(
CellKey::new(vec![
("Month".into(), month.into()),
("Type".into(), "Food".into()),
]),
CellValue::Number(n),
);
}
wb.active_view_mut().set_axis("Month", Axis::Row);
wb.active_view_mut().set_axis("Type", Axis::Column);
wb.active_view_mut().hide_item("Month", "Feb");
assert_eq!(column_total(&wb), 103.0, "expanded: Jan and Apr are drawn");
wb.active_view_mut().toggle_group_collapse("Month", "Q1");
assert_eq!(column_total(&wb), 103.0, "z moved the total");
assert_eq!(
grid_value(&mut wb, "Q1", "Food"),
Some(3.0),
"Feb is hidden"
);
// Every member hidden: no fold is drawn, and the total still agrees.
wb.active_view_mut().hide_item("Month", "Jan");
assert_eq!(grid_value(&mut wb, "Q1", "Food"), None, "no Q1 row");
assert_eq!(column_total(&wb), 100.0);
}
/// improvise-t0ma: the steps themselves, not their translation into effects —
/// see context/test-evidence/core.md.
#[test]