feat(fit): fit-measure — a fitted tree as an ordinary improvise formula
improvise-szrt / improvise-vx3. `improvise-gp` was a bridge with nothing
crossing it; this is the crossing.
`improvise_gp::fit::fit_measure` is the sequence between the two halves that
already existed: `dataset` (a View's records as a feature matrix) and
`fitted_formula` (a champion as formula text the document can resolve). It
lives in `improvise-gp` rather than in the command because none of it is
terminal-specific — a browser fitting the same measure must run exactly these
steps in exactly this order or it gets a different tree from the same seed.
`command/cmd/fit.rs` is the user-facing surface: `fit-measure <target>
[<measure-name>] [<seed>]`, returning `SetStatus` + `MarkDirty` + `AddFormula`.
After it there is nothing GP-specific in the document — the rule is a
`## Formulas` line, recomputed by the fixed-point evaluator, editable in the
formula panel, drillable, saved with no grammar change.
**Where the fit runs, and why.** In `execute`, on the command side. A fit is
decision logic — a pure function of (Model, View, target, seed, FitConfig)
that mutates nothing — so it belongs where decision logic belongs; putting an
expensive stochastic search inside `Effect::apply` would mix it into the one
place the app mutates. That buys purity, not asynchrony, so vx3's AC 4 is met
on its "bounded" arm: `FitRequest::default` is a smaller search than the CLI's,
**measured** at 6ms / 12ms / 105ms over 12 / 200 / 2,000 records in release.
**Three things the sequence had to get right**, each of which fails quietly:
* The fit is standardized and the formula must not be. Written down untouched,
a champion predicts standard deviations — a well-formed formula evaluating to
±5 where the user's data is in the thousands, and nothing downstream can see
it. `denormalize` folds the affine map back into the leaves first.
* The rule is stored as a *new* measure (`<target> (fit)`), never as the target
itself: gotcha 7 makes a formula target and an explicit item one item, so the
cells the fit was learned from would stop being visible.
* `Quality` is an enum. `cv_regress` pools nothing below two folds and scoring
an empty pool gives R² exactly 0 — "nobody checked" reading as "this rule is
worth nothing", which is the loudest thing a fit can say.
**A bug this found, fixed with its own failing test first.** `_Measure` is a
filtering category, and `Workbook::new`'s default view puts it on `Axis::Page`
— so `dataset::filter_coords` pinned it to one measure and the feature matrix
collapsed. On a document whose target is `_Measure`'s first item every feature
column came back blank; otherwise the table was empty and the error blamed the
data ("no visible record has a Revenue value"). `_Measure` is the coordinate
the columns are spread across, so paging it is not "which records am I looking
at", it is "throw away every column". Every existing test in that module built
its view with `View::new` + `on_category_added`, under which an underscore
category lands on `Axis::None` — the fixture was building the one world the
code worked in. The two new regression tests compare row *contents*, because
asserting the header and the row count passes with the bug present.
Evidence, and the faults injected to earn it. Dropping the `denormalize` call
fails `the_fitted_formula_predicts_in_the_documents_own_units` and nothing
else. Both `_Measure`-on-Page tests were watched failing before the one-line
fix. Every new test asserts against a real document — never a helper that
fabricates a `_Measure` item per feature name, which is what let improvise-8xhy
hide. 1216 passing, up 27 from 1189; suite still well under 2s.
Also fixes the refusal message, which a `rustfmt` pass over its multi-line
literal had left with runs of ten spaces mid-sentence — invisible in the source
and to every test that only asked whether a substring was present, and visible
to precisely the user it exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edward Langley
Generated
+1
@@ -862,6 +862,7 @@ dependencies = [
|
||||
"anyhow",
|
||||
"improvise-core",
|
||||
"improvise-formula",
|
||||
"improvise-gp",
|
||||
"improvise-io",
|
||||
"improvise-protocol",
|
||||
"serde",
|
||||
|
||||
@@ -11,6 +11,7 @@ improvise-core = { path = "../improvise-core" }
|
||||
improvise-protocol = { path = "../improvise-protocol" }
|
||||
improvise-formula = { path = "../improvise-formula" }
|
||||
improvise-io = { path = "../improvise-io" }
|
||||
improvise-gp = { path = "../improvise-gp" }
|
||||
anyhow = "1"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
//! `fit-measure`: evolve a decision tree over what the active view shows and
|
||||
//! store it as an ordinary improvise formula.
|
||||
//!
|
||||
//! After this command there is nothing GP-specific left in the document — the
|
||||
//! rule is a `## Formulas` line like any other, recomputed by the fixed-point
|
||||
//! evaluator, editable in the formula panel, drillable, and saved with no new
|
||||
//! grammar (improvise-21d's enabling fact, and improvise-7na's answer).
|
||||
//!
|
||||
//! # The fit runs inside `execute`, and that is the architecture rather than a
|
||||
//! shortcut
|
||||
//!
|
||||
//! A `Cmd` is pure over `&CmdContext` and returns effects; an `Effect` is the
|
||||
//! only mutation site. A fit is **decision logic** — a pure function of
|
||||
//! `(Model, View, target, seed, FitConfig)` that mutates nothing — so it
|
||||
//! belongs on the command side, and what the effects carry is its *result*.
|
||||
//! Putting the search inside an `Effect::apply` would have mixed an expensive
|
||||
//! stochastic computation into the one place the app mutates, which is the
|
||||
//! split's whole point.
|
||||
//!
|
||||
//! What this does **not** buy is asynchrony. The command runs on the thread
|
||||
//! that draws the next frame, so a fit is bounded rather than cancellable:
|
||||
//! [`FitRequest::default`] is a deliberately smaller search than the CLI's
|
||||
//! (improvise-vx3's AC 4 asks for "cancellable **or** bounded"). Moving it off
|
||||
//! the main loop is a change to `draw.rs`'s event loop, not to this file, and
|
||||
//! nothing here forecloses it — the fit is already a pure function that could
|
||||
//! be handed to a worker and have its `FitOutcome` posted back as the same
|
||||
//! effects.
|
||||
//!
|
||||
//! # What a refusal is for
|
||||
//!
|
||||
//! `improvise_gp::fit_measure` fails rather than storing something misleading
|
||||
//! — most sharply when the champion splits on an encoding artefact like
|
||||
//! `Region=North`, which is not an item of anyone's document, so every cell of
|
||||
//! the stored measure would be `Error("ref:…")` (improvise-8xhy, repo-map
|
||||
//! gotcha 37d). This command is where a user actually meets that restriction,
|
||||
//! so the refusal goes to the status line **whole**: it names the features that
|
||||
//! could not be resolved, because "the fit failed" is not something a user can
|
||||
//! act on and "it split on Region, which is a category rather than a measure"
|
||||
//! is.
|
||||
|
||||
use improvise_gp::fit::{FitRequest, fit_measure, fitted_measure_name};
|
||||
|
||||
use crate::command::cmd::core::{Cmd, CmdContext};
|
||||
use crate::ui::effect::{self, Effect};
|
||||
|
||||
/// Fit `target` from the other measures and categories the view shows, storing
|
||||
/// the rule as the measure `into`.
|
||||
#[derive(Debug)]
|
||||
pub struct FitMeasure {
|
||||
pub target: String,
|
||||
pub into: String,
|
||||
pub request: FitRequest,
|
||||
}
|
||||
|
||||
impl FitMeasure {
|
||||
/// A nameless instance for `register_pure`, which reads a command's name
|
||||
/// off a prototype rather than off a string literal (gotcha 15).
|
||||
pub fn prototype() -> Self {
|
||||
Self {
|
||||
target: String::new(),
|
||||
into: String::new(),
|
||||
request: FitRequest::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// `fit-measure <target> [<measure-name>] [<seed>]`.
|
||||
///
|
||||
/// The seed is an argument rather than a clock read for the reason the CLI
|
||||
/// fixes its own default: two fits of the same document should agree, and a
|
||||
/// user who wants a different search asks for one. It is reported back on
|
||||
/// the status line so the fit that landed can be asked for again.
|
||||
pub fn parse(args: &[String]) -> Result<Box<dyn Cmd>, String> {
|
||||
let [target, rest @ ..] = args else {
|
||||
return Err("fit-measure requires a target measure".into());
|
||||
};
|
||||
let (into, seed) = match rest {
|
||||
[] => (fitted_measure_name(target), None),
|
||||
[into] => (into.clone(), None),
|
||||
[into, seed] => (
|
||||
into.clone(),
|
||||
Some(
|
||||
seed.parse::<u64>()
|
||||
.map_err(|e| format!("fit-measure: {seed:?} is not a seed: {e}"))?,
|
||||
),
|
||||
),
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"fit-measure takes 1-3 arguments, got {}",
|
||||
args.len()
|
||||
));
|
||||
}
|
||||
};
|
||||
Ok(Box::new(FitMeasure {
|
||||
target: target.clone(),
|
||||
into,
|
||||
request: FitRequest {
|
||||
seed: seed.unwrap_or(FitRequest::default().seed),
|
||||
..FitRequest::default()
|
||||
},
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
impl Cmd for FitMeasure {
|
||||
fn name(&self) -> &'static str {
|
||||
"fit-measure"
|
||||
}
|
||||
|
||||
fn execute(&self, ctx: &CmdContext) -> Vec<Box<dyn Effect>> {
|
||||
match fit_measure(ctx.model, ctx.view, &self.target, &self.into, &self.request) {
|
||||
// The report first and `AddFormula` last, so a formula the emitter
|
||||
// spelled but the parser will not take leaves *its* message on
|
||||
// screen rather than this command's optimistic one (gotcha 20).
|
||||
Ok(outcome) => vec![
|
||||
effect::set_status(outcome.status_line()),
|
||||
Box::new(effect::MarkDirty),
|
||||
Box::new(effect::AddFormula {
|
||||
raw: outcome.formula.raw.clone(),
|
||||
target_category: outcome.formula.target_category.clone(),
|
||||
}),
|
||||
],
|
||||
Err(e) => vec![effect::set_status(format!("fit-measure: {e}"))],
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::command::cmd::test_helpers::*;
|
||||
use crate::model::category::CategoryKind;
|
||||
use crate::model::cell::{CellKey, CellValue};
|
||||
use crate::view::Axis;
|
||||
|
||||
const MEASURE: &str = "_Measure";
|
||||
const REGION: &str = "Region";
|
||||
const TAG: &str = "Tag";
|
||||
|
||||
/// A document a tree can learn exactly — `Score` steps at `Cost` 50 — in
|
||||
/// the units a real document uses, and with `Region` single-valued so its
|
||||
/// one-hot has the degenerate range `(1, 1)` that `simplify` prunes. The
|
||||
/// champion therefore cannot reach an unresolvable feature and the success
|
||||
/// path is deterministic rather than seed-dependent; `Tag` is a `Label`
|
||||
/// category, which `dataset` keeps out of the features while still making
|
||||
/// the twelve records distinct. See `improvise_gp::fit`'s own fixture.
|
||||
fn workbook() -> crate::workbook::Workbook {
|
||||
let mut wb = crate::workbook::Workbook::new("sales");
|
||||
wb.add_category(REGION).expect("first category");
|
||||
wb.add_item(REGION, "North").expect("Region exists");
|
||||
wb.add_category(TAG).expect("second category");
|
||||
wb.model.category_mut(TAG).expect("just added").kind = CategoryKind::Label;
|
||||
for i in 0..12u32 {
|
||||
wb.add_item(TAG, format!("r{i}")).expect("Tag exists");
|
||||
}
|
||||
let measure = wb
|
||||
.model
|
||||
.category_mut(MEASURE)
|
||||
.expect("_Measure always exists");
|
||||
for item in ["Cost", "Score"] {
|
||||
measure.add_item(item);
|
||||
}
|
||||
for i in 0..12u32 {
|
||||
let cost = 10.0 * f64::from(i);
|
||||
wb.model.set_cell(cell(i, "Cost"), CellValue::Number(cost));
|
||||
wb.model.set_cell(
|
||||
cell(i, "Score"),
|
||||
CellValue::Number(if cost > 50.0 { 4000.0 } else { 1000.0 }),
|
||||
);
|
||||
}
|
||||
wb.active_view_mut().set_axis(REGION, Axis::Row);
|
||||
wb
|
||||
}
|
||||
|
||||
fn cell(i: u32, measure: &str) -> CellKey {
|
||||
CellKey::new(vec![
|
||||
(REGION.to_string(), "North".to_string()),
|
||||
(TAG.to_string(), format!("r{i}")),
|
||||
(MEASURE.to_string(), measure.to_string()),
|
||||
])
|
||||
}
|
||||
|
||||
fn cmd(args: &[&str]) -> Box<dyn Cmd> {
|
||||
FitMeasure::parse(&args.iter().map(|s| s.to_string()).collect::<Vec<_>>())
|
||||
.expect("the arguments parse")
|
||||
}
|
||||
|
||||
/// Run `fit-measure` the way the app does: build the read-only context
|
||||
/// from a workbook, execute, and apply the effects to a real `App`.
|
||||
fn run(args: &[&str]) -> crate::ui::app::App {
|
||||
let wb = workbook();
|
||||
let layout = make_layout(&wb);
|
||||
let registry = make_registry();
|
||||
let effects = cmd(args).execute(&make_ctx(&wb, &layout, ®istry));
|
||||
let mut app = crate::ui::app::App::new(workbook(), None);
|
||||
app.apply_effects(effects);
|
||||
app
|
||||
}
|
||||
|
||||
/// The end-to-end claim of improvise-vx3, asserted against a real
|
||||
/// document: a command fits, the formula lands through the ordinary
|
||||
/// `add_formula` path, and the new measure evaluates to numbers.
|
||||
#[test]
|
||||
fn the_command_installs_a_formula_the_document_evaluates() {
|
||||
let app = run(&["Score"]);
|
||||
let model = &app.model_state.workbook.model;
|
||||
assert!(
|
||||
model.formulas().iter().any(|f| f.target == "Score (fit)"),
|
||||
"no fitted formula was stored: {:?} / {:?}",
|
||||
model.formulas().iter().map(|f| &f.raw).collect::<Vec<_>>(),
|
||||
app.view_state.status_msg,
|
||||
);
|
||||
for i in 0..12u32 {
|
||||
match model.evaluate(&cell(i, "Score (fit)")) {
|
||||
Some(CellValue::Number(_)) => {}
|
||||
other => panic!("record r{i} came back as {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Adding a rule is a change to the document, so the command marks it
|
||||
/// dirty — otherwise the fit is lost on the next quit without a prompt.
|
||||
#[test]
|
||||
fn a_fit_marks_the_document_dirty() {
|
||||
assert!(
|
||||
!crate::ui::app::App::new(workbook(), None).model_state.dirty,
|
||||
"the fixture starts clean"
|
||||
);
|
||||
assert!(run(&["Score"]).model_state.dirty);
|
||||
}
|
||||
|
||||
/// improvise-vx3 is emphatic that the rule alone is not the deliverable:
|
||||
/// a tidy tree that generalizes no better than the mean is worse than
|
||||
/// useless. So the status line carries the held-out comparison, what the
|
||||
/// rule leans on, and the seed that reproduces it.
|
||||
#[test]
|
||||
fn the_status_line_reports_quality_against_the_baseline() {
|
||||
let app = run(&["Score"]);
|
||||
let status = &app.view_state.status_msg;
|
||||
for expected in ["R\u{b2}", "baseline", "leans on", "seed"] {
|
||||
assert!(
|
||||
status.contains(expected),
|
||||
"{expected:?} missing from {status:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A refusal reaches the user *whole*. "fit failed" is not actionable;
|
||||
/// naming the measure that does not exist is (improvise-8xhy's argument,
|
||||
/// arriving at the place a user meets it).
|
||||
#[test]
|
||||
fn a_refusal_names_what_went_wrong_and_stores_nothing() {
|
||||
let wb = workbook();
|
||||
let layout = make_layout(&wb);
|
||||
let registry = make_registry();
|
||||
let effects = cmd(&["Margin"]).execute(&make_ctx(&wb, &layout, ®istry));
|
||||
assert_eq!(effects.len(), 1, "a refused fit must emit no model change");
|
||||
let mut app = crate::ui::app::App::new(workbook(), None);
|
||||
app.apply_effects(effects);
|
||||
assert!(
|
||||
app.view_state
|
||||
.status_msg
|
||||
.contains("not a measure holding data"),
|
||||
"{:?}",
|
||||
app.view_state.status_msg
|
||||
);
|
||||
assert!(app.model_state.workbook.model.formulas().is_empty());
|
||||
}
|
||||
|
||||
/// gotcha 37b: the seed is half of what reproduces a fit, so it is an
|
||||
/// argument — and a different one really does search differently.
|
||||
#[test]
|
||||
fn the_seed_argument_changes_the_search() {
|
||||
let wb = workbook();
|
||||
let layout = make_layout(&wb);
|
||||
let registry = make_registry();
|
||||
let ctx = make_ctx(&wb, &layout, ®istry);
|
||||
assert_ne!(
|
||||
effects_debug(&cmd(&["Score", "A", "1"]).execute(&ctx)),
|
||||
effects_debug(&cmd(&["Score", "A", "99"]).execute(&ctx)),
|
||||
"two seeds produced the identical batch of effects"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_target_is_required_and_a_bad_seed_is_named() {
|
||||
assert!(FitMeasure::parse(&[]).is_err());
|
||||
let err = FitMeasure::parse(&["Score".into(), "A".into(), "later".into()])
|
||||
.expect_err("\"later\" is not a seed");
|
||||
assert!(err.contains("is not a seed"), "{err}");
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ pub mod cell;
|
||||
pub mod commit;
|
||||
pub mod core;
|
||||
pub mod effect_cmds;
|
||||
pub mod fit;
|
||||
pub mod grid;
|
||||
pub mod mode;
|
||||
pub mod navigation;
|
||||
|
||||
@@ -15,6 +15,7 @@ use super::cell::*;
|
||||
use super::commit::*;
|
||||
use super::core::*;
|
||||
use super::effect_cmds::*;
|
||||
use super::fit::FitMeasure;
|
||||
use super::grid::*;
|
||||
use super::mode::*;
|
||||
use super::navigation::*;
|
||||
@@ -54,6 +55,7 @@ pub fn default_registry() -> CmdRegistry {
|
||||
},
|
||||
);
|
||||
r.register_pure(&AddFormulaCmd(vec![]), AddFormulaCmd::parse);
|
||||
r.register_pure(&FitMeasure::prototype(), FitMeasure::parse);
|
||||
r.register_pure(&RemoveFormulaCmd(vec![]), RemoveFormulaCmd::parse);
|
||||
r.register_pure(&CreateViewCmd(vec![]), CreateViewCmd::parse);
|
||||
r.register_pure(&DeleteViewCmd(vec![]), DeleteViewCmd::parse);
|
||||
|
||||
@@ -113,12 +113,23 @@ fn cell_text(value: &CellValue) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
/// The coordinates every visible record shares, exactly as `GridLayout::new`
|
||||
/// derives them: one selection per filtering category, defaulting to its first
|
||||
/// item when the view names none.
|
||||
/// The coordinates every visible record shares — one selection per filtering
|
||||
/// category, defaulting to its first item when the view names none, exactly as
|
||||
/// `GridLayout::new` derives them, **except** that `_Measure` never
|
||||
/// constrains.
|
||||
///
|
||||
/// That exception is not a special case, it is the difference between a grid
|
||||
/// and a feature matrix. `_Measure` is the coordinate the *columns* are spread
|
||||
/// across, so pinning it to one item is not "which records am I looking at" but
|
||||
/// "throw away every column except this one" — and `Workbook::new`'s default
|
||||
/// view puts it on `Axis::Page`, so that was the behaviour on a freshly opened
|
||||
/// document. Either the target's own column survived and every feature came
|
||||
/// back blank, or, on a document whose target is not `_Measure`'s first item,
|
||||
/// the table was empty and the error blamed the data.
|
||||
fn filter_coords(model: &Model, view: &View) -> Vec<(String, String)> {
|
||||
view.filtering_cats()
|
||||
.into_iter()
|
||||
.filter(|cat| !CategoryKind::of_name(cat).is_measure())
|
||||
.filter_map(|cat| {
|
||||
let selected = view
|
||||
.page_selection(cat)
|
||||
@@ -395,6 +406,53 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Regression (improvise-vx3): **`_Measure` is a filtering category and
|
||||
/// must not filter a fit.** `Workbook::new`'s default view puts it on
|
||||
/// `Axis::Page`, so `filtering_cats()` named it, `filter_coords` pinned it
|
||||
/// to the first measure, and every "record" then carried exactly one
|
||||
/// measure cell — never the target. The whole table came back as "no
|
||||
/// visible record has a Revenue value to learn from" on a document plainly
|
||||
/// full of them, which reads as a data problem rather than as an axis one.
|
||||
///
|
||||
/// The rule that fixes it is not a special case: a fit's target *and* its
|
||||
/// features are measures, spread across the `_Measure` coordinate, so
|
||||
/// paging to one is not "which records am I looking at" but "throw away
|
||||
/// every column". Page ∪ Filter still scopes the rows in every other
|
||||
/// category, which is what the test below this one pins.
|
||||
///
|
||||
/// Every test in this module used `View::new` + `on_category_added`, under
|
||||
/// which a leading-underscore category lands on `Axis::None` — so the
|
||||
/// fixture was quietly building the one world the code worked in, and no
|
||||
/// test here could see the default a real `Workbook` hands out.
|
||||
#[test]
|
||||
fn a_measure_on_the_page_axis_does_not_filter_away_the_training_rows() {
|
||||
let (model, mut view) = fixture();
|
||||
let unfiltered = table(&model, &view);
|
||||
view.set_axis(MEASURE_CATEGORY, Axis::Page);
|
||||
let paged = feature_table(&model, &view, "Revenue")
|
||||
.expect("paging the measure axis is not a reason to have no data");
|
||||
// The *contents*, not the shape. Pinning only the header and the row
|
||||
// count passes on this fixture even with the bug present: `_Measure`'s
|
||||
// first item is `Revenue`, so every row still has a target and only
|
||||
// the feature columns come back blank — the shape survives while the
|
||||
// data does not, which is exactly the silent failure to guard against.
|
||||
assert_eq!(paged, unfiltered);
|
||||
}
|
||||
|
||||
/// The other half, and the sharper one: which measure `filter_coords`
|
||||
/// would have pinned to is the *first* item, so on a document whose target
|
||||
/// is not first the table came back empty and the error blamed the data —
|
||||
/// "no visible record has a Revenue value to learn from", on a document
|
||||
/// plainly full of them.
|
||||
#[test]
|
||||
fn a_paged_measure_that_is_not_the_target_does_not_empty_the_table() {
|
||||
let (model, mut view) = fixture();
|
||||
let unfiltered = table(&model, &view);
|
||||
view.set_axis(MEASURE_CATEGORY, Axis::Page);
|
||||
view.set_page_selection(MEASURE_CATEGORY, "Cost");
|
||||
assert_eq!(table(&model, &view), unfiltered);
|
||||
}
|
||||
|
||||
/// gotcha 27: what is *visible* is Page ∪ Filter. Scoping by
|
||||
/// `categories_on(Axis::Page)` would fit against rows a drill has
|
||||
/// already excluded, and say nothing about it.
|
||||
|
||||
@@ -0,0 +1,616 @@
|
||||
//! One fit, end to end: a [`View`]'s records in, an improvise [`Formula`] and
|
||||
//! the evidence for believing it out.
|
||||
//!
|
||||
//! [`dataset`] already turns what a view shows into the feature matrix a search
|
||||
//! consumes, and [`crate::fitted_formula`] already turns a champion into
|
||||
//! formula text the document can store. This module is the sequence between
|
||||
//! them, and it exists here rather than in a command because none of it is
|
||||
//! terminal-specific: a browser fitting the same measure over the same document
|
||||
//! must run exactly these steps in exactly this order, or it gets a different
|
||||
//! tree from the same seed.
|
||||
//!
|
||||
//! **Nothing here prints** (repo-map gotcha 37a). The quality numbers, the
|
||||
//! importance ranking and `preprocess`'s dropped columns all come back as data
|
||||
//! on [`FitOutcome`], and deciding they become a status line, a panel or a
|
||||
//! stderr note is the host's business. [`FitOutcome::status_line`] is a
|
||||
//! *rendering* of that data offered so a second front end does not compose its
|
||||
//! own — the same argument that put `flatten_cat_tree` in the shared layer —
|
||||
//! and it is a returned `String`, never a `println!`.
|
||||
//!
|
||||
//! # Three things the sequence has to get right
|
||||
//!
|
||||
//! **A fit is standardized and the formula must not be.** The crate's leaf
|
||||
//! scores live in a ±5 range, so a target in the thousands cannot be reached by
|
||||
//! leaf mutation and [`gp_tree::fit::standardize`] is not optional. Its
|
||||
//! consequence is that the champion predicts *standard deviations*: `fit_csv`
|
||||
//! answers that by printing `≈ score × std + mean` beside the tree, which a
|
||||
//! document has nowhere to put. So the affine map is folded back into the
|
||||
//! leaves with [`gp_tree::fit::denormalize`] before a single character of
|
||||
//! formula text is spelled. Omitting it emits a perfectly well-formed formula
|
||||
//! whose every cell is out by an affine map — wrong in the quiet way, since
|
||||
//! small numbers in the right shape look like an answer.
|
||||
//!
|
||||
//! **The seed alone does not reproduce a fit** (gotcha 37b), so
|
||||
//! [`FitOutcome`] carries the [`FitConfig`] beside it. Every stochastic step
|
||||
//! draws from one generator in a fixed order — cross-validate, then fit, then
|
||||
//! explain — so the pair really is sufficient.
|
||||
//!
|
||||
//! **Cross-validating nothing is not the same as explaining nothing.**
|
||||
//! [`gp_tree::fit::cv_regress`] pools no pairs below two folds, and scoring an
|
||||
//! empty pool gives an R² of exactly zero — which reads as "this rule is worth
|
||||
//! nothing" when what happened is that nobody checked. [`Quality`] is an enum
|
||||
//! so the two cannot be confused, and a host that wants the numbers has to name
|
||||
//! the case where there are none.
|
||||
|
||||
use gp_tree::data::{Overrides, Target};
|
||||
use gp_tree::fit::{
|
||||
FitConfig, RegressionQuality, cv_regress, denormalize, fit, importance_ranking, mse,
|
||||
permutation_importance, regression_quality, standardize,
|
||||
};
|
||||
use gp_tree::{Spec, SplitMix64};
|
||||
use improvise_core::model::Model;
|
||||
use improvise_core::view::View;
|
||||
use improvise_formula::Formula;
|
||||
|
||||
use crate::dataset::dataset;
|
||||
|
||||
/// What to ask of a fit. Everything a run's outcome depends on, so that
|
||||
/// `(FitRequest, document)` reproduces it.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct FitRequest {
|
||||
/// The seed for every stochastic step.
|
||||
pub seed: u64,
|
||||
/// The search's knobs — necessary beside the seed, not implied by it.
|
||||
pub config: FitConfig,
|
||||
/// Cross-validation folds. Below two, no held-out evidence is produced and
|
||||
/// [`Quality::Unmeasured`] says so.
|
||||
pub cv_folds: usize,
|
||||
/// Shuffles per feature when measuring permutation importance.
|
||||
pub importance_repeats: usize,
|
||||
}
|
||||
|
||||
impl Default for FitRequest {
|
||||
/// Interactive defaults: bounded hard enough to run inside a keystroke.
|
||||
///
|
||||
/// `FitConfig::default()` is the CLI's budget — 80 members over 600
|
||||
/// generations — which is the right answer for a batch tool and the wrong
|
||||
/// one for a synchronous command, because a `Cmd` here runs on the thread
|
||||
/// that draws the next frame. These numbers are a deliberate quality/
|
||||
/// latency trade, not a tuned optimum; a caller wanting the CLI's search
|
||||
/// passes `FitConfig::default()`.
|
||||
///
|
||||
/// **Measured**, release build, on a document of the shape this module's
|
||||
/// tests use: **6ms** at 12 records, **12ms** at 200, **105ms** at 2,000 —
|
||||
/// the whole sequence, five-fold CV included. So it is a pause rather than
|
||||
/// a freeze at any document size in this repo's own benchmarks (18,000
|
||||
/// cells), and roughly linear in records beyond that. There is no timing
|
||||
/// *test*: a wall-clock assertion is flaky on shared CI and would fail for
|
||||
/// a reason that is not this code's. Re-measure after changing these.
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
seed: 0x0F17_C0FF_EE00_D00D,
|
||||
config: FitConfig {
|
||||
pop: 40,
|
||||
generations: 120,
|
||||
..FitConfig::default()
|
||||
},
|
||||
cv_folds: 5,
|
||||
importance_repeats: 3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Held-out evidence, or the statement that there is none.
|
||||
///
|
||||
/// The two are different things to tell a user and an `Option` would have made
|
||||
/// them one: `Quality::Measured` carrying an all-zero [`RegressionQuality`]
|
||||
/// means "this tree is no better than the mean", which is the loudest thing a
|
||||
/// fit can say, and it must not be reachable by simply not having asked.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub enum Quality {
|
||||
/// Pooled `k`-fold cross-validation against the predict-the-mean baseline.
|
||||
Measured { folds: usize, of: RegressionQuality },
|
||||
/// Fewer rows than folds asked for: nothing was held out, so nothing is
|
||||
/// claimed.
|
||||
Unmeasured { rows: usize },
|
||||
}
|
||||
|
||||
/// A fitted rule and the evidence for it.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FitOutcome {
|
||||
/// The champion as an improvise formula, already parsed and already
|
||||
/// checked against the document it is to be stored in.
|
||||
pub formula: Formula,
|
||||
/// How the rule did on data it was not fitted to.
|
||||
pub quality: Quality,
|
||||
/// `(feature name, percentage share)`, most important first, features the
|
||||
/// rule does not use omitted.
|
||||
pub importance: Vec<(String, f32)>,
|
||||
/// Records and columns the selection dropped — `preprocess`'s notices and
|
||||
/// the feature table's, in that order.
|
||||
pub warnings: Vec<String>,
|
||||
/// The seed this fit ran under, and…
|
||||
pub seed: u64,
|
||||
/// …the knobs it ran under. Both, because neither alone reproduces it.
|
||||
pub config: FitConfig,
|
||||
}
|
||||
|
||||
impl FitOutcome {
|
||||
/// One line naming the rule's worth, what it leans on, and anything the
|
||||
/// selection dropped — in that order, because a user who reads only the
|
||||
/// first clause has read the part that decides whether to believe it.
|
||||
pub fn status_line(&self) -> String {
|
||||
let quality = match self.quality {
|
||||
Quality::Measured { folds, of } => format!(
|
||||
"R\u{b2} {:.2}, RMSE {:.3} vs baseline {:.3} ({folds}-fold CV)",
|
||||
of.r2,
|
||||
of.rmse,
|
||||
of.baseline_mse.sqrt(),
|
||||
),
|
||||
Quality::Unmeasured { rows } => {
|
||||
format!("no held-out check \u{2014} only {rows} record(s)")
|
||||
}
|
||||
};
|
||||
let top: Vec<String> = self
|
||||
.importance
|
||||
.iter()
|
||||
.take(3)
|
||||
.map(|(name, share)| format!("{name} {share:.0}%"))
|
||||
.collect();
|
||||
let leans = if top.is_empty() {
|
||||
"a constant \u{2014} no feature matters".to_string()
|
||||
} else {
|
||||
format!("leans on {}", top.join(", "))
|
||||
};
|
||||
let tail = if self.warnings.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!(" \u{2014} {}", self.warnings.join("; "))
|
||||
};
|
||||
format!(
|
||||
"{} = fitted tree: {quality}; {leans}; seed {:#x}{tail}",
|
||||
self.formula.target, self.seed
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// The measure a fit of `target` is stored under, unless a caller names
|
||||
/// another: the fit is a *second, derived* column beside the data it explains.
|
||||
///
|
||||
/// Storing it under `target` itself would make the formula shadow the very
|
||||
/// cells it was fitted to — gotcha 7's `_Measure` items are explicit items ∪
|
||||
/// formula targets, so the two would be one item with a computed value, and
|
||||
/// the user's typed data would stop being visible. A fit is a rule *about* the
|
||||
/// data, never a replacement for it.
|
||||
pub fn fitted_measure_name(target: &str) -> String {
|
||||
format!("{target} (fit)")
|
||||
}
|
||||
|
||||
/// Fit `target` from everything else the `view` shows, storing the rule as the
|
||||
/// measure `into`, and hand back the rule with the evidence for it.
|
||||
///
|
||||
/// Fails, rather than storing something misleading, when: the selection cannot
|
||||
/// produce a table at all (see [`crate::feature_table`]); `into` is a measure
|
||||
/// that already holds data; the target is categorical, which a single
|
||||
/// regression tree does not express; or the champion splits on a feature the
|
||||
/// document cannot resolve (improvise-8xhy, and the message names them).
|
||||
pub fn fit_measure(
|
||||
model: &Model,
|
||||
view: &View,
|
||||
target: &str,
|
||||
into: &str,
|
||||
request: &FitRequest,
|
||||
) -> anyhow::Result<FitOutcome> {
|
||||
anyhow::ensure!(
|
||||
into != target,
|
||||
"a fit of {target:?} cannot be stored as {target:?} — the formula would shadow the \
|
||||
cells it was fitted to. Name a different measure"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
!model.measure_item_names().iter().any(|m| m == into),
|
||||
"{into:?} is already a measure; a formula on it would replace what is there. \
|
||||
Name a measure that does not exist yet"
|
||||
);
|
||||
let data = dataset(model, view, target, &Overrides::default())?;
|
||||
let rows = regression_rows(data.target, target)?;
|
||||
|
||||
let names: Vec<&str> = data.feature_names.iter().map(String::as_str).collect();
|
||||
let spec = Spec {
|
||||
ranges: &data.ranges,
|
||||
names: &names,
|
||||
};
|
||||
let mut rng = SplitMix64::new(request.seed);
|
||||
|
||||
// Held-out evidence first, and from the same generator, so the sequence
|
||||
// reproduces as a whole rather than step by step.
|
||||
let folds = request.cv_folds.min(rows.len());
|
||||
let quality = match folds >= 2 {
|
||||
true => Quality::Measured {
|
||||
folds,
|
||||
of: regression_quality(&cv_regress(&rows, &spec, &request.config, folds, &mut rng)),
|
||||
},
|
||||
false => Quality::Unmeasured { rows: rows.len() },
|
||||
};
|
||||
|
||||
// Then the rule itself, over every row. Standardized to fit, de-normalized
|
||||
// to be written down — see this module's header.
|
||||
let (normalized, mean, std) = standardize(&rows);
|
||||
let champion = denormalize(
|
||||
&fit(&normalized, &spec, &request.config, &mut rng),
|
||||
mean,
|
||||
std,
|
||||
);
|
||||
|
||||
let importance = permutation_importance(
|
||||
|shuffled| mse(&champion, shuffled),
|
||||
&rows,
|
||||
names.len(),
|
||||
request.importance_repeats,
|
||||
&mut rng,
|
||||
);
|
||||
|
||||
Ok(FitOutcome {
|
||||
formula: crate::fitted_formula(
|
||||
&champion,
|
||||
&spec,
|
||||
into,
|
||||
improvise_core::model::category::MEASURE_CATEGORY,
|
||||
model,
|
||||
)?,
|
||||
quality,
|
||||
importance: importance_ranking(&importance)
|
||||
.into_iter()
|
||||
.map(|(j, share)| (data.feature_names[j].clone(), share))
|
||||
.collect(),
|
||||
warnings: data.warnings,
|
||||
seed: request.seed,
|
||||
config: request.config,
|
||||
})
|
||||
}
|
||||
|
||||
/// The regression rows, or a refusal naming why a categorical target is not a
|
||||
/// formula.
|
||||
///
|
||||
/// `preprocess` infers classification the moment a target cell does not parse
|
||||
/// as a number, and a classification fit is one tree *per class* plus an
|
||||
/// `argmax` — three things improvise formulas have none of. Refusing here is
|
||||
/// the same choice `fitted_formula` makes for an unresolvable feature: fail
|
||||
/// where the formula would have been written, rather than store something that
|
||||
/// looks like an answer.
|
||||
fn regression_rows(
|
||||
target_data: Target,
|
||||
target: &str,
|
||||
) -> anyhow::Result<Vec<gp_tree::fit::Row<f32>>> {
|
||||
match target_data {
|
||||
Target::Regress(rows) => Ok(rows),
|
||||
Target::Classify { labels, .. } => anyhow::bail!(
|
||||
"{target:?} holds text, so a fit over it is a classification into {labels:?} — \
|
||||
that is one tree per class chosen by argmax, which improvise formulas cannot \
|
||||
express. Fit a numeric measure instead"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use improvise_core::model::category::CategoryKind;
|
||||
use improvise_core::model::cell::{CellKey, CellValue};
|
||||
use improvise_core::view::Axis;
|
||||
|
||||
const MEASURE: &str = "_Measure";
|
||||
const REGION: &str = "Region";
|
||||
const TAG: &str = "Tag";
|
||||
const FIT_INTO: &str = "Score (fit)";
|
||||
|
||||
/// A document with one regular category, one feature measure and one
|
||||
/// target measure standing in a rule a tree can express exactly: `Score`
|
||||
/// steps at `Cost` 50, and the numbers are in the **thousands** so a fit
|
||||
/// that forgets to de-normalize is visibly wrong rather than subtly so.
|
||||
///
|
||||
/// `Region` carries a single item on purpose. `preprocess` one-hots a
|
||||
/// categorical column, and a one-hot that is constant across every record
|
||||
/// has the degenerate range `(1, 1)`, which `simplify` prunes — so the
|
||||
/// champion *cannot* split on an unresolvable feature and the success path
|
||||
/// is deterministic rather than "whatever this seed happened to pick".
|
||||
/// `Tag` is what makes the twelve records distinct, and it is a `Label`
|
||||
/// category so `dataset` excludes it from the features (it would otherwise
|
||||
/// one-hot into one column per row: a perfect memoriser).
|
||||
fn document() -> (Model, View) {
|
||||
let mut model = Model::new("sales");
|
||||
for cat in [REGION, TAG] {
|
||||
model.add_category(cat).expect("under the category limit");
|
||||
}
|
||||
model
|
||||
.category_mut(REGION)
|
||||
.expect("just added")
|
||||
.add_item("North");
|
||||
model.category_mut(TAG).expect("just added").kind = CategoryKind::Label;
|
||||
for i in 0..12u32 {
|
||||
model
|
||||
.category_mut(TAG)
|
||||
.expect("just added")
|
||||
.add_item(format!("r{i}"));
|
||||
}
|
||||
let measure = model.category_mut(MEASURE).expect("_Measure always exists");
|
||||
for item in ["Cost", "Score"] {
|
||||
measure.add_item(item);
|
||||
}
|
||||
for i in 0..12u32 {
|
||||
let cost = 10.0 * f64::from(i);
|
||||
model.set_cell(cell(&format!("r{i}"), "Cost"), CellValue::Number(cost));
|
||||
model.set_cell(
|
||||
cell(&format!("r{i}"), "Score"),
|
||||
CellValue::Number(if cost > 50.0 { 4000.0 } else { 1000.0 }),
|
||||
);
|
||||
}
|
||||
let mut view = View::new("Default");
|
||||
for cat in model.category_names() {
|
||||
view.on_category_added(cat);
|
||||
}
|
||||
view.set_axis(REGION, Axis::Row);
|
||||
(model, view)
|
||||
}
|
||||
|
||||
fn cell(tag: &str, measure: &str) -> CellKey {
|
||||
CellKey::new(vec![
|
||||
(REGION.to_string(), "North".to_string()),
|
||||
(TAG.to_string(), tag.to_string()),
|
||||
(MEASURE.to_string(), measure.to_string()),
|
||||
])
|
||||
}
|
||||
|
||||
fn request() -> FitRequest {
|
||||
FitRequest {
|
||||
config: FitConfig {
|
||||
pop: 24,
|
||||
generations: 40,
|
||||
..FitRequest::default().config
|
||||
},
|
||||
..FitRequest::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// The whole point, asserted against a real document: the fit's formula
|
||||
/// goes in through the ordinary `add_formula` path and every cell of the
|
||||
/// new measure evaluates to a number — not to `Error("ref:…")`, which is
|
||||
/// what improvise-8xhy was.
|
||||
#[test]
|
||||
fn a_fitted_formula_evaluates_in_the_document_it_was_fitted_on() {
|
||||
let (mut model, view) = document();
|
||||
let outcome =
|
||||
fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands");
|
||||
model.add_formula(outcome.formula.clone());
|
||||
model.recompute_formulas(&[]);
|
||||
for i in 0..12u32 {
|
||||
match model.evaluate(&cell(&format!("r{i}"), &outcome.formula.target)) {
|
||||
Some(CellValue::Number(_)) => {}
|
||||
other => panic!("record r{i} came back as {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The de-normalization regression, and the reason it needs a target in
|
||||
/// the thousands. `fit` runs against a standardized target, so a champion
|
||||
/// written down untouched predicts standard deviations: a well-formed
|
||||
/// formula, parsing cleanly, evaluating to ±5 where the user's data is
|
||||
/// 1000–4000. Nothing downstream can see that — it is a number in a cell.
|
||||
///
|
||||
/// Injecting the fault (dropping the `denormalize` call in `fit_measure`)
|
||||
/// fails this and only this test.
|
||||
#[test]
|
||||
fn the_fitted_formula_predicts_in_the_documents_own_units() {
|
||||
let (mut model, view) = document();
|
||||
let outcome =
|
||||
fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("the fit lands");
|
||||
model.add_formula(outcome.formula.clone());
|
||||
model.recompute_formulas(&[]);
|
||||
for i in 0..12u32 {
|
||||
let Some(CellValue::Number(n)) = model.evaluate(&cell(&format!("r{i}"), FIT_INTO))
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
assert!(
|
||||
(500.0..=5000.0).contains(&n),
|
||||
"record r{i} predicted {n}, which is standardized units, not the \
|
||||
document's 1000–4000"
|
||||
);
|
||||
}
|
||||
// …and the same claim read off the formula text, so the assertion does
|
||||
// not depend on the target item having been resolvable above.
|
||||
let numbers = leaf_numbers(&outcome.formula.expr);
|
||||
assert!(!numbers.is_empty(), "the fit emitted no leaf at all");
|
||||
assert!(
|
||||
numbers.iter().any(|n| *n > 500.0),
|
||||
"every leaf is {numbers:?} — standardized units, not the document's 1000–4000"
|
||||
);
|
||||
}
|
||||
|
||||
/// Every `Number` literal in the emitted expression, leaves and thresholds
|
||||
/// alike. A threshold here is a `Cost` (0–110) and a leaf is a `Score`
|
||||
/// (1000–4000), so "some literal exceeds 500" can only be a leaf.
|
||||
fn leaf_numbers(expr: &improvise_formula::Expr) -> Vec<f64> {
|
||||
use improvise_formula::Expr;
|
||||
match expr {
|
||||
Expr::Number(n) => vec![*n],
|
||||
Expr::If(c, t, e) => [c, t, e].iter().flat_map(|e| leaf_numbers(e)).collect(),
|
||||
Expr::BinOp(_, l, r) => [l, r].iter().flat_map(|e| leaf_numbers(e)).collect(),
|
||||
Expr::UnaryMinus(e) => leaf_numbers(e),
|
||||
Expr::Ref(_) | Expr::Agg(..) => Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// gotcha 37b: the seed alone does not reproduce a fit, so the outcome
|
||||
/// carries the config beside it — and the pair really is sufficient.
|
||||
#[test]
|
||||
fn the_same_seed_and_config_reproduce_the_same_rule() {
|
||||
let (model, view) = document();
|
||||
let a = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits");
|
||||
let b = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits");
|
||||
assert_eq!(a.formula.raw, b.formula.raw);
|
||||
assert_eq!(a.seed, b.seed);
|
||||
assert_eq!(a.config, b.config);
|
||||
}
|
||||
|
||||
/// And the other half: a different *config* under the same seed searches
|
||||
/// differently, which is why recording the seed alone would not do.
|
||||
#[test]
|
||||
fn the_same_seed_under_a_different_config_is_a_different_search() {
|
||||
let (model, view) = document();
|
||||
let base = request();
|
||||
let wider = FitRequest {
|
||||
config: FitConfig {
|
||||
generations: base.config.generations * 3,
|
||||
..base.config
|
||||
},
|
||||
..base
|
||||
};
|
||||
assert_eq!(base.seed, wider.seed);
|
||||
assert_ne!(
|
||||
fit_measure(&model, &view, "Score", FIT_INTO, &base)
|
||||
.expect("fits")
|
||||
.formula
|
||||
.raw,
|
||||
fit_measure(&model, &view, "Score", FIT_INTO, &wider)
|
||||
.expect("fits")
|
||||
.formula
|
||||
.raw,
|
||||
"the same seed gave the same rule under different knobs, so recording \
|
||||
the seed alone would have been enough after all"
|
||||
);
|
||||
}
|
||||
|
||||
/// A fit is a rule *about* the data, never a replacement for it. Stored
|
||||
/// under its own target the formula would shadow the very cells it was
|
||||
/// fitted to (gotcha 7: `_Measure` items are explicit items ∪ formula
|
||||
/// targets, so the two would be one item), and the user's typed numbers
|
||||
/// would simply stop being visible.
|
||||
#[test]
|
||||
fn a_fit_stored_under_its_own_target_is_refused() {
|
||||
let (model, view) = document();
|
||||
let err =
|
||||
fit_measure(&model, &view, "Score", "Score", &request()).expect_err("self-shadowing");
|
||||
assert!(err.to_string().contains("shadow"), "{err}");
|
||||
}
|
||||
|
||||
/// The same hazard one step removed: any existing measure would be
|
||||
/// replaced by the rule, so the name has to be a new one.
|
||||
#[test]
|
||||
fn a_fit_stored_over_an_existing_measure_is_refused() {
|
||||
let (model, view) = document();
|
||||
let err =
|
||||
fit_measure(&model, &view, "Score", "Cost", &request()).expect_err("Cost holds data");
|
||||
assert!(err.to_string().contains("already a measure"), "{err}");
|
||||
}
|
||||
|
||||
/// improvise-8xhy reaching a real user, and the reason improvise-4zl7
|
||||
/// exists: with a categorical feature carrying the signal, the champion
|
||||
/// splits on a one-hot — `Region=North`, an encoding artefact no document
|
||||
/// has an item for — and the whole fit is refused *by name* rather than
|
||||
/// written in as a measure whose every cell is `Error("ref:…")`.
|
||||
///
|
||||
/// Note what this costs in practice: which features a champion picks is a
|
||||
/// property of the search, so on a document with several categories the
|
||||
/// same command refuses or succeeds depending on the seed.
|
||||
#[test]
|
||||
fn a_fit_that_needs_a_category_is_refused_naming_the_feature() {
|
||||
let (mut model, view) = document();
|
||||
// Region is the only thing left that can explain Score.
|
||||
model
|
||||
.category_mut(REGION)
|
||||
.expect("Region exists")
|
||||
.add_item("South");
|
||||
for i in 0..12u32 {
|
||||
model.clear_cell(&cell(&format!("r{i}"), "Cost"));
|
||||
}
|
||||
for (i, region) in (0..12u32).map(|i| (i, if i < 6 { "North" } else { "South" })) {
|
||||
model.set_cell(
|
||||
CellKey::new(vec![
|
||||
(REGION.to_string(), region.to_string()),
|
||||
(TAG.to_string(), format!("r{i}")),
|
||||
(MEASURE.to_string(), "Score".to_string()),
|
||||
]),
|
||||
CellValue::Number(if region == "North" { 1000.0 } else { 4000.0 }),
|
||||
);
|
||||
}
|
||||
let err = fit_measure(&model, &view, "Score", FIT_INTO, &request())
|
||||
.expect_err("the champion can only split on a one-hot");
|
||||
let msg = err.to_string();
|
||||
assert!(
|
||||
msg.contains("Region=North") || msg.contains("Region=South"),
|
||||
"{msg}"
|
||||
);
|
||||
assert!(
|
||||
msg.contains("not an item") || msg.contains("not items"),
|
||||
"{msg}"
|
||||
);
|
||||
// The refusal is read off a one-line status bar, so it has to *be* one
|
||||
// line. A `rustfmt` pass over the multi-line literal that spells it had
|
||||
// left runs of ten spaces mid-sentence — invisible in the source and in
|
||||
// every test that only asked whether a substring was present, and
|
||||
// visible to precisely the user this message exists for.
|
||||
assert!(!msg.contains('\n'), "{msg:?}");
|
||||
assert!(!msg.contains(" "), "{msg:?}");
|
||||
}
|
||||
|
||||
/// A text-valued target is a *classification*, which is k trees and an
|
||||
/// argmax — not something an improvise formula can hold. Refused by name,
|
||||
/// at the point the formula would have been written.
|
||||
#[test]
|
||||
fn a_text_target_is_refused_as_a_classification() {
|
||||
let (mut model, view) = document();
|
||||
for i in 0..12u32 {
|
||||
model.set_cell(
|
||||
cell(&format!("r{i}"), "Score"),
|
||||
CellValue::Text(if i % 2 == 0 { "up" } else { "down" }.into()),
|
||||
);
|
||||
}
|
||||
let err =
|
||||
fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect_err("text target");
|
||||
assert!(err.to_string().contains("classification"), "{err}");
|
||||
}
|
||||
|
||||
/// Cross-validating nothing is not "explained nothing". With fewer records
|
||||
/// than folds asked for, the outcome says so rather than reporting an R²
|
||||
/// of zero that a user would read as a verdict on the rule.
|
||||
#[test]
|
||||
fn too_few_records_to_cross_validate_says_so_rather_than_scoring_zero() {
|
||||
let (mut model, view) = document();
|
||||
for i in 1..12u32 {
|
||||
for measure in ["Cost", "Score"] {
|
||||
model.clear_cell(&cell(&format!("r{i}"), measure));
|
||||
}
|
||||
}
|
||||
let outcome = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits");
|
||||
assert!(
|
||||
matches!(outcome.quality, Quality::Unmeasured { rows: 1 }),
|
||||
"{:?}",
|
||||
outcome.quality
|
||||
);
|
||||
assert!(
|
||||
outcome.status_line().contains("no held-out check"),
|
||||
"{}",
|
||||
outcome.status_line()
|
||||
);
|
||||
}
|
||||
|
||||
/// The status line is one line and leads with the verdict, because a user
|
||||
/// who reads only its first clause has read the part that decides whether
|
||||
/// to believe the rule at all.
|
||||
#[test]
|
||||
fn the_status_line_leads_with_the_verdict_and_names_the_seed() {
|
||||
let (model, view) = document();
|
||||
let outcome = fit_measure(&model, &view, "Score", FIT_INTO, &request()).expect("fits");
|
||||
let line = outcome.status_line();
|
||||
assert!(!line.contains('\n'), "{line}");
|
||||
assert!(
|
||||
line.starts_with(&format!("{FIT_INTO} = fitted tree: R")),
|
||||
"{line}"
|
||||
);
|
||||
assert!(line.contains("baseline"), "{line}");
|
||||
assert!(
|
||||
line.contains(&format!("{:#x}", outcome.seed)),
|
||||
"the seed is what reproduces the fit: {line}"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -84,8 +84,10 @@
|
||||
//! it looks and what it would and would not cover.
|
||||
|
||||
pub mod dataset;
|
||||
pub mod fit;
|
||||
|
||||
pub use dataset::{FeatureTable, dataset, feature_table};
|
||||
pub use fit::{FitOutcome, FitRequest, Quality, fit_measure};
|
||||
|
||||
use gp_tree::{Spec, Tree};
|
||||
use improvise_core::model::Model;
|
||||
@@ -153,7 +155,13 @@ pub fn fitted_formula(
|
||||
let unresolvable = unresolvable_features(tree, spec, model);
|
||||
anyhow::ensure!(
|
||||
unresolvable.is_empty(),
|
||||
"the fitted tree splits on {unresolvable:?}, which {} not {} of this document — a formula refers to its inputs by name, so every cell of {target:?} would be a reference error. Encoding artefacts (`Cat=Item` one-hots, `col.missing` flags, date parts) cannot be written as a formula: fit over numeric measures for now",
|
||||
// One line, because it is read off a status bar. A `rustfmt` pass over
|
||||
// the previous multi-line literal left runs of ten spaces mid-sentence
|
||||
// in the message a user actually sees.
|
||||
"the fitted tree splits on {unresolvable:?}, which {} not {} of this document \
|
||||
— a formula refers to its inputs by name, so every cell of {target:?} would be \
|
||||
a reference error. Encoding artefacts (`Cat=Item` one-hots, `col.missing` flags, \
|
||||
date parts) cannot be written as a formula: fit over numeric measures for now",
|
||||
if unresolvable.len() == 1 { "is" } else { "are" },
|
||||
if unresolvable.len() == 1 {
|
||||
"an item"
|
||||
|
||||
Reference in New Issue
Block a user