Merge pull request #322 from fiddlerwoaroof/feat/8jk6.3-date-hierarchy

feat(import): a date column imports as one hierarchical category (improvise-8jk6.3)
This commit is contained in:
Ed Langley
2026-10-02 22:32:35 -07:00
committed by GitHub
12 changed files with 919 additions and 20 deletions
+2 -2
View File
@@ -40,9 +40,9 @@ improvise import examples/demo.csv \
--no-wizard \ --no-wizard \
--category Region --category Product --category Customer \ --category Region --category Product --category Customer \
--measure Revenue --measure Cost \ --measure Revenue --measure Cost \
--time Date --extract Date:Month \ --time Date --extract Date:Month --split-date Date \
--axis Region:row --axis Product:row \ --axis Region:row --axis Product:row \
--axis Date_Month:column --axis Measure:column \ --axis Date_Month:column --axis _Measure:column \
--axis Customer:page --axis Date:none \ --axis Customer:page --axis Date:none \
--formula "Profit = Revenue - Cost" \ --formula "Profit = Revenue - Cost" \
--name "Acme Sales Demo" \ --name "Acme Sales Demo" \
+53
View File
@@ -613,6 +613,59 @@ different row order on screen, and a different detected date format, since
varied. The `HashSet` in `analyze_records` is there to deduplicate and **never varied. The `HashSet` in `analyze_records` is there to deduplicate and **never
to iterate**. to iterate**.
### A date column imports as ONE hierarchical category (improvise-8jk6.3)
The user: "I think dates should usually be one category and we can use
formulas to crosstab." So the default `DateShape::Hierarchy` builds one
category named after the column, where the old import built up to three flat
ones beside it (`Date_Year`, `Date_Quarter`, `Date_Month`):
- **Its items are the finest chosen component**, spelled so they stay unique
across years: `2024-01`, `2024-Q1`, `2024`, which are
`extract_date_component`'s existing spellings. A month is never `Jan`.
- **The coarser chosen components are its groups**, coarsest first
(`DateComponent::COARSE_TO_FINE`): a month sits in its quarter, and the
quarter's `>` line names its year. A level left unchosen is skipped rather
than drawn empty, so Year + Month groups months straight into years.
- **It opens folded**, one row per year, with nothing to set, because every
group starts closed (`context/code/core.md` §19).
- **It costs one of the category slots**, not up to four, and
`proposed_category_count` counts it that way.
- **The raw dates are gone from that category.** The day is not a component, so
two records of one month land on one cell and the collision step asks, as it
does for any other collision. `Split` keeps the day.
- **A value the detected format cannot read keeps its raw text** as an
ungrouped item, with a warning naming the column and the row count, rather
than being dropped. The warning is separate from the split shape's, because
that one reports an absent coordinate and this one a coordinate that is there
and ungrouped. **Unless its spelling is one the hierarchy generates**
(improvise-d0n6): a raw `2024-01` or `2024` is the same name as the generated
month or year, so it would join the grouped item and the warning would be
false. The build **refuses**, naming the column and the spellings and pointing
at the split shape. That follows the collision rule below: where two records
would silently be one thing, the user is asked and nothing is defaulted. A
fallback to the split shape was rejected as a silent change of the user's
choice, and a renamed raw item as altering the data. The check is
`Plan::spelling_clash`.
- **Item order is first-seen, not chronological.** Order is a correctness
property here (gotcha 18), and a group need not be contiguous, so a CSV that
is not in date order still nests correctly. It is just drawn in the order
it was read.
`DateShape::Split` is the old behaviour, unchanged, and stays a choice. `s` on
the wizard's date step toggles the field under the cursor, and the header line
for each field says which it will get ("one category: Year > Quarter > Month"
or "split: Date, Date_Year, …"). It is worded once, in
`project.rs::date_shape_sentence`. Headless, `--split-date <field>` does the
same, and `--extract` keeps its meaning: which components to build.
**Crosstabs by formula are not available yet.** A formula can name a month
(`SUM(Amount WHERE Date = "2024-01")`, `Date IS "2024-01"`), but a `WHERE` or
`IS` naming a GROUP matches no record: both compare the coordinate to the name
exactly. `SUM(Amount WHERE Date = "2024")` is an empty sum, and `Date IS "2024"`
is 0 on every month of 2024. That was measured on this branch and reported for
filing, not built here.
### The ambiguous-date heuristic reads American-first, deliberately ### The ambiguous-date heuristic reads American-first, deliberately
`01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data `01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data
+97
View File
@@ -1685,6 +1685,44 @@ closed, the one its `collapsed:` line names and the three it does not, and the
line is written back. Rival: the old open default; this test fails with the line is written back. Rival: the old open default; this test fails with the
other eleven in that rival's list. other eleven in that rival's list.
### a_date_column_imports_as_one_category_of_months_in_quarters_in_years
improvise-8jk6.3, the user's "dates should usually be one category". The
fixture has two days of one month, so that month is one summed cell, and
January of the next year, so a month must stay unique across years. Rivals, each
run against the `date_hierarchy` module, and each caught here:
- The split as the default shape.
- Levels taken finest first, so the year hangs under the month.
- Groups registered without their parents.
- The raw dates still added beside the months.
- A hierarchy counted as `1 + components` slots.
### a_level_left_unchosen_is_skipped_not_emptied
Year + Month nests months straight into years, and Quarter alone is a flat
category of quarters. Caught the finest-first rival and the raw-dates rival.
### an_imported_date_hierarchy_opens_folded_to_its_years
The import sets nothing: the hierarchy opens folded, one row per year, because
every group starts closed (improvise-felo.6). The 2024 row is the fold of its
months, 111. Caught three of the rivals above.
### an_unparsed_date_keeps_its_value_and_is_warned_about
A format that reads none of the values keeps each raw value as an ungrouped
item, and the build says so. Its first version passed on the base commit: the
shape was ignored there and the SPLIT's warning happened to contain the same
words. It now also asserts there is no `Date_Year`. Rival: the unparsed value
not counted; this test fails.
### a_headless_date_import_splits_when_told_to
`--split-date` is the wizard's `s`, headless. It failed first as an unknown
argument. Rival: the flag parsed and ignored; this test fails, and
`a_headless_date_import_is_one_hierarchical_category` does not, which is the
point of having both.
### every_fold_the_api_can_build_reads_back_from_the_file_it_writes ### every_fold_the_api_can_build_reads_back_from_the_file_it_writes
ChatGPT Codex, P2 on #317 at `e09ec752`: `GroupFold::Expanded(usize)` and ChatGPT Codex, P2 on #317 at `e09ec752`: `GroupFold::Expanded(usize)` and
@@ -1939,6 +1977,45 @@ predates this PR (improvise-ych), but the sparse-record rule now rests on it:
without it the value becomes an item named `" "`. Rival F4 (the reviewer's without it the value becomes an item named `" "`. Rival F4 (the reviewer's
fault) fails it. fault) fails it.
### a_raw_date_spelled_like_a_generated_item_is_refused
improvise-d0n6 (Codex P2 on PR #322). A `%Y-%m-%d` column imported Year > Month
with `2024-01-15` and a raw `2024-01`. Before the refusal the build succeeded,
put both records on the grouped item `2024-01` (summing them) and warned that
the raw value was "kept ungrouped", which was false. Fault injected: removing
the `spelling_clash` call makes this test fail with `unwrap_err()` on an `Ok`.
### a_raw_date_spelled_like_a_generated_group_is_refused
improvise-d0n6. The same with a raw `2024`, a generated group. Removing the
`spelling_clash` call fails it the same way. The control,
`a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped`, passes either
way, so a blanket refusal of every unparsed value fails it.
### the_date_step_will_not_advance_over_a_spelling_clash
improvise-d0n6 (Copilot on PR #322, and the adversarial review's B1). The
spelling-clash refusal sat only in the build, so a wizard with a raw `2024` in
a `%Y-%m-%d` column passed ConfigureDates and failed on NameModel, three steps
from the `s` that fixes it. Before the check joined `validation_error` the
test failed with the step already `DefineFormulas`; removing that call again
reproduces it. It also toggles to the split shape and requires the step to
advance, so a check that blocks forever fails.
### split_date_naming_an_unknown_or_non_date_field_is_refused
improvise-tpm8 class. `--split-date Nope` and `--split-date Revenue` (a
measure) exited 0 and did nothing. Both now exit 1 naming the flag and the
field; removing the check in `run_headless_import` fails both rows.
### an_axis_naming_a_category_the_import_did_not_build_is_refused
improvise-8jk6.3 review. `--extract Date:Month --axis Date_Month:column` used
to build one `Date` category and drop the `--axis` for the vanished
`Date_Month`, exit 0. It now exits 1 naming the category and `--split-date`.
This also refuses any `--axis` naming an absent category, which was silently
ignored before this PR as well.
### a_folder_of_only_header_only_files_keeps_no_record_and_refuses ### a_folder_of_only_header_only_files_keeps_no_record_and_refuses
improvise-8jk6.6 meeting PR #314's rule that a header-only file is still a improvise-8jk6.6 meeting PR #314's rule that a header-only file is still a
@@ -1949,3 +2026,23 @@ items and empty views: the category check alone cannot refuse. The `setup`
assertion proves that seeded proposal exists, so the test is not passing only assertion proves that seeded proposal exists, so the test is not passing only
because there was no category. **R7**, the check removed from the build, fails because there was no category. **R7**, the check removed from the build, fails
it. The user confirmed the refusal on PR #321: "Refuse (as #321 does)". it. The user confirmed the refusal on PR #321: "Refuse (as #321 does)".
### an_absent_axis_category_that_is_no_date_part_names_what_was_built
improvise-8jk6.3 review (B2). The README's own recipe passed
`--axis Measure:column`, a silent no-op because the category is `_Measure`; the
first refusal blamed dates for it. The test uses a non-date absent name and
asserts the message lists the built categories and carries no date hint.
Fault injected: making the hint unconditional fails it.
### the_readmes_import_recipe_runs
improvise-8jk6.3 review (B2). Runs the README's recipe as printed. Fault
injected: restoring `--axis Measure:column` in the README fails it with the
refusal.
### split_date_on_a_time_field_with_no_date_format_is_refused
improvise-8jk6.3 review. `--time Revenue --split-date Revenue` made the field a
time category with no date format and the split was a silent no-op. Without
the `date_format.is_some()` condition the import exits 0 and this fails.
@@ -65,6 +65,7 @@ fn review_proposal_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> { fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
match key { match key {
KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent), KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent),
KeyCode::Char('s') => op(WizardOp::ToggleDateShape),
KeyCode::Enter => op(WizardOp::Advance), KeyCode::Enter => op(WizardOp::Advance),
_ => cursor_keys(key), _ => cursor_keys(key),
} }
@@ -1053,6 +1053,7 @@ pub enum WizardOp {
ToggleProposal, ToggleProposal,
CycleProposalKind, CycleProposalKind,
ToggleDateComponent, ToggleDateComponent,
ToggleDateShape,
StartFormulaEdit, StartFormulaEdit,
DeleteFormula, DeleteFormula,
ConfirmFormula, ConfirmFormula,
@@ -1078,6 +1079,7 @@ impl WizardOp {
Self::ToggleProposal => wizard.toggle_proposal(), Self::ToggleProposal => wizard.toggle_proposal(),
Self::CycleProposalKind => wizard.cycle_proposal_kind(), Self::CycleProposalKind => wizard.cycle_proposal_kind(),
Self::ToggleDateComponent => wizard.toggle_date_component(), Self::ToggleDateComponent => wizard.toggle_date_component(),
Self::ToggleDateShape => wizard.toggle_date_shape(),
Self::StartFormulaEdit => wizard.start_formula_edit(), Self::StartFormulaEdit => wizard.start_formula_edit(),
Self::DeleteFormula => wizard.delete_formula(), Self::DeleteFormula => wizard.delete_formula(),
Self::ConfirmFormula => wizard.confirm_formula(), Self::ConfirmFormula => wizard.confirm_formula(),
+62 -1
View File
@@ -11,7 +11,7 @@ use improvise_core::model::record::{AggregateHash, RecordHash};
use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol}; use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol};
use improvise_core::view::{Axis, View}; use improvise_core::view::{Axis, View};
use improvise_core::workbook::Workbook; use improvise_core::workbook::Workbook;
use improvise_io::import::analyzer::{DateComponent, FieldKind}; use improvise_io::import::analyzer::{DateComponent, DateShape, FieldKind};
use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep}; use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep};
use improvise_protocol::{ use improvise_protocol::{
AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay, AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay,
@@ -662,6 +662,7 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
((*label).to_string(), p.date_components.contains(component)) ((*label).to_string(), p.date_components.contains(component))
}) })
.collect(), .collect(),
shape: date_shape_sentence(p),
}) })
.collect(), .collect(),
collisions: w collisions: w
@@ -700,6 +701,27 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
} }
} }
/// What a date field's chosen components become, worded for every front end.
fn date_shape_sentence(p: &improvise_io::import::analyzer::FieldProposal) -> String {
let chosen = |order: &[DateComponent]| -> Vec<&str> {
order
.iter()
.filter(|c| p.date_components.contains(c))
.filter_map(|c| DATE_COMPONENTS.iter().find(|(d, _)| d == c))
.map(|(_, label)| *label)
.collect()
};
match (p.date_shape, chosen(&DateComponent::COARSE_TO_FINE)) {
(_, none) if none.is_empty() => "one category of the raw dates".to_string(),
(DateShape::Hierarchy, levels) => format!("one category: {}", levels.join(" > ")),
(DateShape::Split, _) => {
let parts = chosen(&DATE_COMPONENTS.map(|(c, _)| c));
let names: Vec<String> = parts.iter().map(|l| format!("{}_{l}", p.field)).collect();
format!("split: {}, {}", p.field, names.join(", "))
}
}
}
/// The date components the wizard offers, **in the order it offers them**: /// The date components the wizard offers, **in the order it offers them**:
/// `ImportWizard::date_config_at_cursor` counts these same rows. /// `ImportWizard::date_config_at_cursor` counts these same rows.
const DATE_COMPONENTS: [(DateComponent, &str); 3] = [ const DATE_COMPONENTS: [(DateComponent, &str); 3] = [
@@ -6522,3 +6544,42 @@ mod column_totals {
); );
} }
} }
/// improvise-8jk6.3: the wizard's date step says what each date field becomes,
/// and `s` switches it between one category and the split.
#[cfg(test)]
mod date_shape {
use super::wizard_cache;
use crate::import::wizard::{ImportWizard, WizardStep};
use serde_json::json;
#[test]
fn the_date_step_says_what_a_date_field_becomes_and_s_switches_it() {
let mut w = ImportWizard::new(json!([
{"Region": "East", "When": "2024-01-15", "Revenue": 1.0},
{"Region": "West", "When": "2024-04-02", "Revenue": 2.0},
]));
w.advance();
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
let shape = |w: &ImportWizard| wizard_cache(w).date_fields[0].shape.clone();
assert_eq!(shape(&w), "one category of the raw dates");
w.toggle_date_component();
w.move_cursor(1);
w.toggle_date_component();
w.move_cursor(1);
w.toggle_date_component();
assert_eq!(shape(&w), "one category: Year > Quarter > Month");
w.toggle_date_shape();
assert_eq!(
shape(&w),
"split: When, When_Year, When_Month, When_Quarter"
);
let built = w.pipeline.build_model().unwrap();
assert!(built.model.category("When_Quarter").is_some());
w.toggle_date_shape();
let built = w.pipeline.build_model().unwrap();
assert!(built.model.category("When_Quarter").is_none());
let when = built.model.category("When").unwrap();
assert_eq!(when.item_chain("2024-04"), vec!["2024", "2024-Q2"]);
}
}
@@ -22,6 +22,26 @@ pub enum DateComponent {
Quarter, Quarter,
} }
impl DateComponent {
/// The components as a hierarchy nests them, coarsest first.
pub const COARSE_TO_FINE: [DateComponent; 3] = [
DateComponent::Year,
DateComponent::Quarter,
DateComponent::Month,
];
}
/// How a date column's components become categories (improvise-8jk6.3).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum DateShape {
/// ONE category named after the column: its items are the finest chosen
/// component, grouped by the coarser ones. The user's default.
#[default]
Hierarchy,
/// The column's raw values, plus one flat category per chosen component.
Split,
}
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct FieldProposal { pub struct FieldProposal {
pub field: String, pub field: String,
@@ -32,6 +52,8 @@ pub struct FieldProposal {
pub date_format: Option<String>, pub date_format: Option<String>,
/// Which date components to extract as new categories. /// Which date components to extract as new categories.
pub date_components: Vec<DateComponent>, pub date_components: Vec<DateComponent>,
/// Whether those components nest in one category or split into several.
pub date_shape: DateShape,
} }
impl FieldProposal { impl FieldProposal {
@@ -132,6 +154,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
accepted: true, accepted: true,
date_format: None, date_format: None,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
}; };
} }
@@ -159,6 +182,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
accepted: true, accepted: true,
date_format, date_format,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
}; };
} }
@@ -170,6 +194,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
accepted: true, accepted: true,
date_format: None, date_format: None,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
}; };
} }
@@ -180,6 +205,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
accepted: true, accepted: true,
date_format: None, date_format: None,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
}; };
} }
@@ -191,6 +217,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
accepted: true, accepted: true,
date_format: None, date_format: None,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
} }
}) })
.collect() .collect()
+449 -11
View File
@@ -1,10 +1,10 @@
use std::collections::HashMap; use std::collections::{HashMap, HashSet};
use anyhow::{Result, anyhow}; use anyhow::{Result, anyhow};
use serde_json::Value; use serde_json::Value;
use super::analyzer::{ use super::analyzer::{
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path, DateComponent, DateShape, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
extract_date_component, find_array_paths, extract_date_component, find_array_paths,
}; };
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile}; use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile};
@@ -239,13 +239,23 @@ impl ImportPipeline {
.filter(|p| p.accepted) .filter(|p| p.accepted)
.map(|p| match p.kind { .map(|p| match p.kind {
FieldKind::Category => 1, FieldKind::Category => 1,
FieldKind::TimeCategory if p.date_format.is_some() => 1 + p.date_components.len(), FieldKind::TimeCategory if p.date_format.is_some() => match p.date_shape {
DateShape::Hierarchy => 1,
DateShape::Split => 1 + p.date_components.len(),
},
FieldKind::TimeCategory => 1, FieldKind::TimeCategory => 1,
FieldKind::Measure | FieldKind::Label => 0, FieldKind::Measure | FieldKind::Label => 0,
}) })
.sum() .sum()
} }
/// Why the date step cannot advance: see [`Plan::spelling_clash`].
pub fn spelling_clash(&self) -> Option<String> {
let plan = self.plan();
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
plan.spelling_clash(&placements)
}
/// Every coordinate tuple more than one record lands on, in first-seen /// Every coordinate tuple more than one record lands on, in first-seen
/// order. Known before the build, so the wizard can ask (improvise-cgqt). /// order. Known before the build, so the wizard can ask (improvise-cgqt).
pub fn collisions(&self) -> Vec<Collision> { pub fn collisions(&self) -> Vec<Collision> {
@@ -300,6 +310,7 @@ impl ImportPipeline {
&& p.kind == FieldKind::TimeCategory && p.kind == FieldKind::TimeCategory
&& p.date_format.is_some() && p.date_format.is_some()
&& !p.date_components.is_empty() && !p.date_components.is_empty()
&& p.date_shape == DateShape::Split
}) })
.flat_map(|p| { .flat_map(|p| {
let fmt = p.date_format.as_deref().unwrap(); let fmt = p.date_format.as_deref().unwrap();
@@ -314,11 +325,17 @@ impl ImportPipeline {
}) })
}) })
.collect(); .collect();
let categories = accepted(&[FieldKind::Category, FieldKind::TimeCategory]);
let date_hierarchies = categories
.iter()
.filter_map(|p| DateHierarchy::of(p))
.collect();
Plan { Plan {
categories: accepted(&[FieldKind::Category, FieldKind::TimeCategory]), categories,
measures: accepted(&[FieldKind::Measure]), measures: accepted(&[FieldKind::Measure]),
labels: accepted(&[FieldKind::Label]), labels: accepted(&[FieldKind::Label]),
date_extractions, date_extractions,
date_hierarchies,
scalars: self.scalars, scalars: self.scalars,
} }
} }
@@ -339,6 +356,9 @@ impl ImportPipeline {
let formulas = self.parsed_formulas()?; let formulas = self.parsed_formulas()?;
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect(); let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
if let Some(clash) = plan.spelling_clash(&placements) {
return Err(anyhow!(clash));
}
let collisions = collisions_among(&placements); let collisions = collisions_among(&placements);
let occurrence = match (&self.on_duplicate, collisions.is_empty()) { let occurrence = match (&self.on_duplicate, collisions.is_empty()) {
(_, true) | (DuplicatePolicy::Sum, false) => None, (_, true) | (DuplicatePolicy::Sum, false) => None,
@@ -350,11 +370,16 @@ impl ImportPipeline {
for cat_proposal in categories { for cat_proposal in categories {
wb.add_category(&cat_proposal.field)?; wb.add_category(&cat_proposal.field)?;
for val in cat_proposal // A hierarchical date column's items are its components, added below.
.distinct_values let raw_items = match plan
.date_hierarchies
.iter() .iter()
.filter(|v| !v.trim().is_empty()) .any(|h| h.field == cat_proposal.field)
{ {
true => &[][..],
false => &cat_proposal.distinct_values[..],
};
for val in raw_items.iter().filter(|v| !v.trim().is_empty()) {
wb.add_item(&cat_proposal.field, val)?; wb.add_item(&cat_proposal.field, val)?;
} }
} }
@@ -387,7 +412,7 @@ impl ImportPipeline {
// Per-extraction tally of values the detected format could not parse. A // Per-extraction tally of values the detected format could not parse. A
// derived coordinate that is merely absent is improvise-yss5. // derived coordinate that is merely absent is improvise-yss5.
let mut unparsed_dates = vec![0usize; date_extractions.len()]; let mut unparsed_dates = vec![0usize; date_extractions.len() + plan.date_hierarchies.len()];
// Two rows sharing every coordinate are one cell; last-row-wins lost the // Two rows sharing every coordinate are one cell; last-row-wins lost the
// earlier one (improvise-2cw9). The map only deduplicates — gotcha 18. // earlier one (improvise-2cw9). The map only deduplicates — gotcha 18.
let mut cells: Vec<PendingCell> = vec![]; let mut cells: Vec<PendingCell> = vec![];
@@ -405,7 +430,11 @@ impl ImportPipeline {
mut coords, mut coords,
measures, measures,
unparsed, unparsed,
chains,
} => { } => {
for chain in &chains {
add_date_chain(&mut wb, chain)?;
}
for (cat, item) in &coords { for (cat, item) in &coords {
wb.add_item(cat, item)?; wb.add_item(cat, item)?;
} }
@@ -457,7 +486,21 @@ impl ImportPipeline {
} }
let mut warnings: Vec<String> = self.skip_warning().into_iter().collect(); let mut warnings: Vec<String> = self.skip_warning().into_iter().collect();
warnings.extend(unparsed_date_warnings(date_extractions, &unparsed_dates)); let (split, whole) = unparsed_dates.split_at(date_extractions.len());
warnings.extend(unparsed_date_warnings(date_extractions, split));
warnings.extend(
plan.date_hierarchies
.iter()
.zip(whole)
.filter(|(_, n)| **n > 0)
.map(|(h, n)| {
format!(
"'{}' kept its raw value in {n} row(s), ungrouped; it did not match \
the date format '{}'",
h.field, h.format
)
}),
);
warnings.extend(collision_warning(&summed)); warnings.extend(collision_warning(&summed));
warnings.extend(occurrence.map(|name| separated_note(name, &collisions))); warnings.extend(occurrence.map(|name| separated_note(name, &collisions)));
@@ -475,6 +518,7 @@ fn unseen(field: &str, kind: FieldKind, distinct_values: Vec<String>) -> FieldPr
accepted: true, accepted: true,
date_format: None, date_format: None,
date_components: vec![], date_components: vec![],
date_shape: DateShape::default(),
} }
} }
@@ -552,9 +596,43 @@ struct Plan<'a> {
measures: Vec<&'a FieldProposal>, measures: Vec<&'a FieldProposal>,
labels: Vec<&'a FieldProposal>, labels: Vec<&'a FieldProposal>,
date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>, date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>,
date_hierarchies: Vec<DateHierarchy<'a>>,
scalars: Scalars, scalars: Scalars,
} }
/// A date column imported as ONE category: its items are the finest chosen
/// component, grouped by the coarser ones (improvise-8jk6.3, core.md §19).
struct DateHierarchy<'a> {
field: &'a str,
format: &'a str,
/// The chosen components, coarsest first.
levels: Vec<DateComponent>,
}
impl<'a> DateHierarchy<'a> {
fn of(p: &'a FieldProposal) -> Option<Self> {
let levels: Vec<DateComponent> = DateComponent::COARSE_TO_FINE
.into_iter()
.filter(|c| p.date_components.contains(c))
.collect();
let hierarchical =
p.kind == FieldKind::TimeCategory && p.date_shape == DateShape::Hierarchy;
(hierarchical && !levels.is_empty()).then_some(Self {
field: &p.field,
format: p.date_format.as_deref()?,
levels,
})
}
/// Every level's name for one value, root first; the last is the item.
fn chain(&self, value: &str) -> Option<Vec<String>> {
self.levels
.iter()
.map(|c| extract_date_component(value, self.format, *c))
.collect()
}
}
/// Where one source record lands. Worked out once, for the build and for the /// Where one source record lands. Worked out once, for the build and for the
/// collision check alike, so the two cannot disagree (improvise-cgqt). /// collision check alike, so the two cannot disagree (improvise-cgqt).
enum Placement { enum Placement {
@@ -564,6 +642,7 @@ enum Placement {
coords: Vec<(String, String)>, coords: Vec<(String, String)>,
measures: Vec<(String, f64)>, measures: Vec<(String, f64)>,
unparsed: Vec<usize>, unparsed: Vec<usize>,
chains: Vec<DateChain>,
}, },
NotObject, NotObject,
} }
@@ -575,6 +654,7 @@ impl Plan<'_> {
}; };
let mut coords: Vec<(String, String)> = vec![]; let mut coords: Vec<(String, String)> = vec![];
let mut unparsed: Vec<usize> = vec![]; let mut unparsed: Vec<usize> = vec![];
let mut chains: Vec<DateChain> = vec![];
// No value is no coordinate: records are sparse (improvise-8jk6.2). // No value is no coordinate: records are sparse (improvise-8jk6.2).
let named = |field: &str| map.get(field).and_then(scalar_name); let named = |field: &str| map.get(field).and_then(scalar_name);
@@ -583,7 +663,24 @@ impl Plan<'_> {
.iter() .iter()
.filter_map(|p| Some((p, named(&p.field)?))) .filter_map(|p| Some((p, named(&p.field)?)))
{ {
coords.push((cat_proposal.field.clone(), v.clone())); let hierarchy = self
.date_hierarchies
.iter()
.enumerate()
.find(|(_, h)| h.field == cat_proposal.field);
let item = match hierarchy.map(|(at, h)| (at, h.chain(&v))) {
Some((_, Some(chain))) => {
let item = chain.last().cloned().unwrap_or_default();
chains.push((cat_proposal.field.clone(), chain));
item
}
Some((at, None)) => {
unparsed.push(self.date_extractions.len() + at);
v.clone()
}
None => v.clone(),
};
coords.push((cat_proposal.field.clone(), item));
// Extract date components from this field's value // Extract date components from this field's value
for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() { for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() {
@@ -615,10 +712,83 @@ impl Plan<'_> {
coords, coords,
measures, measures,
unparsed, unparsed,
chains,
} }
} }
} }
impl Plan<'_> {
/// Why the build stops when an unparsed date is spelled like a generated
/// item or group (improvise-d0n6; formula-and-io.md, date hierarchy).
fn spelling_clash(&self, placements: &[Placement]) -> Option<String> {
let generated = |field: &str| -> HashSet<&str> {
placements
.iter()
.filter_map(|p| match p {
Placement::Placed { chains, .. } => Some(chains),
Placement::NotObject => None,
})
.flatten()
.filter(|(cat, _)| cat == field)
.flat_map(|(_, chain)| chain.iter().map(String::as_str))
.collect()
};
for (at, h) in self.date_hierarchies.iter().enumerate() {
let tally = self.date_extractions.len() + at;
let names = generated(h.field);
let mut clashing: Vec<&str> = vec![];
for p in placements {
let Placement::Placed {
coords, unparsed, ..
} = p
else {
continue;
};
let raw = coords.iter().find(|(c, _)| c == h.field).map(|(_, v)| v);
let clash = raw.filter(|v| {
unparsed.contains(&tally)
&& names.contains(v.as_str())
&& !clashing.contains(&v.as_str())
});
clashing.extend(clash.map(String::as_str));
}
if !clashing.is_empty() {
return Some(format!(
"'{}' has value(s) that do not match the date format '{}' but are spelled \
like a month, quarter or year it generates: {}; fix them, or import the \
column with the split shape",
h.field,
h.format,
clashing.join(", ")
));
}
}
None
}
}
/// One hierarchical date coordinate: its category and its chain, root first.
type DateChain = (String, Vec<String>);
/// Register a date item under its groups, and each group under its parent.
fn add_date_chain(wb: &mut Workbook, (cat, chain): &DateChain) -> Result<()> {
let Some((item, groups)) = chain.split_last() else {
return Ok(());
};
match groups.last() {
Some(group) => wb.add_item_in_group(cat, item, group)?,
None => wb.add_item(cat, item)?,
};
let category = wb
.model
.category_mut(cat)
.ok_or_else(|| anyhow!("category '{cat}' vanished mid-import"))?;
for pair in groups.windows(2) {
category.add_group(crate::model::category::Group::new(&pair[1]).with_parent(&pair[0]));
}
Ok(())
}
/// A tuple, and each record on it: its 1-based position and the measures it writes. /// A tuple, and each record on it: its 1-based position and the measures it writes.
type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>); type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>);
@@ -921,7 +1091,9 @@ impl ImportWizard {
.nothing_to_import() .nothing_to_import()
.or_else(|| self.no_category_message()) .or_else(|| self.no_category_message())
.or_else(|| self.category_limit_message()), .or_else(|| self.category_limit_message()),
WizardStep::ConfigureDates => self.category_limit_message(), WizardStep::ConfigureDates => self
.category_limit_message()
.or_else(|| self.pipeline.spelling_clash()),
WizardStep::ResolveDuplicates => match &self.pipeline.on_duplicate { WizardStep::ResolveDuplicates => match &self.pipeline.on_duplicate {
DuplicatePolicy::Sum => None, DuplicatePolicy::Sum => None,
DuplicatePolicy::Separate(_) => self.separate_blocked(), DuplicatePolicy::Separate(_) => self.separate_blocked(),
@@ -1199,6 +1371,18 @@ impl ImportWizard {
} }
} }
/// Switch the field at the cursor between ONE hierarchical category and the
/// flat split into one category per component (improvise-8jk6.3).
pub fn toggle_date_shape(&mut self) {
if let Some((pi, _)) = self.date_config_at_cursor() {
let proposal = &mut self.pipeline.proposals[pi];
proposal.date_shape = match proposal.date_shape {
DateShape::Hierarchy => DateShape::Split,
DateShape::Split => DateShape::Hierarchy,
};
}
}
// ── Formula editing ──────────────────────────────────────────────────────── // ── Formula editing ────────────────────────────────────────────────────────
/// Buffer for typing a new formula in the DefineFormulas step. /// Buffer for typing a new formula in the DefineFormulas step.
@@ -1511,6 +1695,7 @@ mod tests {
prop.date_format = Some("%m/%d/%Y".to_string()); prop.date_format = Some("%m/%d/%Y".to_string());
prop.date_components.push(DateComponent::Month); prop.date_components.push(DateComponent::Month);
prop.date_components.push(DateComponent::Year); prop.date_components.push(DateComponent::Year);
prop.date_shape = crate::import::analyzer::DateShape::Split;
} }
} }
p p
@@ -1614,6 +1799,7 @@ mod tests {
for prop in &mut p.proposals { for prop in &mut p.proposals {
if prop.field == "Date" && prop.kind == FieldKind::TimeCategory { if prop.field == "Date" && prop.kind == FieldKind::TimeCategory {
prop.date_components.push(DateComponent::Month); prop.date_components.push(DateComponent::Month);
prop.date_shape = crate::import::analyzer::DateShape::Split;
} }
} }
let wb = p.build_model().unwrap(); let wb = p.build_model().unwrap();
@@ -2484,11 +2670,39 @@ mod tests {
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap(); let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
assert_eq!(date.kind, FieldKind::TimeCategory); assert_eq!(date.kind, FieldKind::TimeCategory);
date.date_components = vec![DateComponent::Month]; date.date_components = vec![DateComponent::Month];
date.date_shape = crate::import::analyzer::DateShape::Split;
let (wb, _) = p.build_model_with_warnings().unwrap(); let (wb, _) = p.build_model_with_warnings().unwrap();
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None); assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None); assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
} }
/// improvise-8jk6.3 under improvise-8jk6.12: the default shape puts the date
/// parts IN the file's own column, so that column is what its view seats.
#[test]
fn a_files_view_seats_its_own_hierarchical_date_column() {
use crate::import::analyzer::DateComponent;
let mut p = merged_pipeline(&[
(
"m/abs-churn.csv",
"date,added
2026-01-02,10
",
),
(
"m/age.csv",
"entity,age-months
src/a.rs,2
",
),
]);
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
date.date_components = vec![DateComponent::Year, DateComponent::Month];
let (wb, _) = p.build_model_with_warnings().unwrap();
assert!(wb.model.category("date_Month").is_none(), "one category");
assert_ne!(wb.views["abs-churn"].axis_of("date"), Axis::None);
assert_eq!(wb.views["age"].axis_of("date"), Axis::None);
}
/// improvise-8jk6.12 (2): two files of one batch never share a view name. /// improvise-8jk6.12 (2): two files of one batch never share a view name.
/// Evidence: `context/test-evidence/formula-and-io.md`. /// Evidence: `context/test-evidence/formula-and-io.md`.
#[test] #[test]
@@ -2744,6 +2958,7 @@ mod tests {
for prop in &mut p.proposals { for prop in &mut p.proposals {
if prop.field == "Date" { if prop.field == "Date" {
prop.date_components.push(DateComponent::Month); prop.date_components.push(DateComponent::Month);
prop.date_shape = crate::import::analyzer::DateShape::Split;
} }
} }
let wb = p.build_model().unwrap(); let wb = p.build_model().unwrap();
@@ -3256,6 +3471,7 @@ mod tests {
w.advance(); w.advance();
assert_eq!(w.step, WizardStep::ConfigureDates, "setup"); assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
w.toggle_date_component(); w.toggle_date_component();
w.toggle_date_shape();
w.advance(); w.advance();
assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup"); assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup");
w.move_cursor(1); w.move_cursor(1);
@@ -3379,3 +3595,225 @@ mod tests {
assert_eq!(msg, "Nothing to import: the source has no records"); assert_eq!(msg, "Nothing to import: the source has no records");
} }
} }
/// improvise-8jk6.3: a date column imports as ONE hierarchical category.
#[cfg(test)]
mod date_hierarchy {
use super::{DuplicatePolicy, ImportPipeline};
use crate::import::analyzer::{DateComponent, DateShape, FieldKind};
use crate::model::cell::CellKey;
use crate::view::GridLayout;
use serde_json::json;
/// Two days of one month, a month in the next quarter, and January of the
/// NEXT year, so a month must stay unique across years.
fn records() -> serde_json::Value {
json!([
{"Date": "01/15/2024", "Amount": 1.0},
{"Date": "01/20/2024", "Amount": 10.0},
{"Date": "04/02/2024", "Amount": 100.0},
{"Date": "01/03/2025", "Amount": 1000.0},
])
}
fn pipeline(components: &[DateComponent], shape: DateShape) -> ImportPipeline {
let mut p = ImportPipeline::new(records());
p.on_duplicate = DuplicatePolicy::Sum;
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
assert_eq!(date.kind, FieldKind::TimeCategory, "precondition");
date.date_components = components.to_vec();
date.date_shape = shape;
p
}
fn cell(month: &str) -> CellKey {
CellKey::new(vec![
("Date".into(), month.into()),
("_Measure".into(), "Amount".into()),
])
}
/// The default shape: ONE category named after the column, its items the
/// months, grouped into quarters grouped into years.
#[test]
fn a_date_column_imports_as_one_category_of_months_in_quarters_in_years() {
use DateComponent::*;
let p = pipeline(&[Month, Year, Quarter], DateShape::default());
assert_eq!(p.proposed_category_count(), 1, "one slot, not four");
let wb = p.build_model().unwrap();
let names: Vec<&str> = wb.model.regular_category_names();
assert_eq!(
names,
vec!["Date"],
"no Date_Year, Date_Quarter or Date_Month"
);
let date = wb.model.category("Date").unwrap();
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
assert_eq!(items, vec!["2024-01", "2024-04", "2025-01"]);
assert_eq!(date.item_chain("2024-01"), vec!["2024", "2024-Q1"]);
assert_eq!(date.item_chain("2025-01"), vec!["2025", "2025-Q1"]);
assert_eq!(
wb.model.get_cell(&cell("2024-01")).and_then(|v| v.as_f64()),
Some(11.0),
"two days of one month are one cell"
);
assert_eq!(
wb.model.get_cell(&cell("2025-01")).and_then(|v| v.as_f64()),
Some(1000.0),
"January 2025 is not January 2024"
);
}
/// Chosen levels only: without Quarter, months group straight into years.
#[test]
fn a_level_left_unchosen_is_skipped_not_emptied() {
use DateComponent::*;
let wb = pipeline(&[Year, Month], DateShape::Hierarchy)
.build_model()
.unwrap();
let date = wb.model.category("Date").unwrap();
assert_eq!(date.item_chain("2024-04"), vec!["2024"]);
let wb = pipeline(&[Quarter], DateShape::Hierarchy)
.build_model()
.unwrap();
let date = wb.model.category("Date").unwrap();
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
assert_eq!(
items,
vec!["2024-Q1", "2024-Q2", "2025-Q1"],
"flat quarters"
);
assert!(date.item_chain("2024-Q1").is_empty());
}
/// The user's ruling on improvise-felo.6: the hierarchy opens folded, one
/// row per year.
#[test]
fn an_imported_date_hierarchy_opens_folded_to_its_years() {
use DateComponent::*;
let wb = pipeline(&[Year, Quarter, Month], DateShape::Hierarchy)
.build_model()
.unwrap();
let layout = GridLayout::with_frozen_records(&wb.model, wb.active_view(), None);
let rows: Vec<String> = (0..layout.row_count())
.map(|r| layout.row_label(r))
.collect();
assert_eq!(rows, vec!["2024", "2025"]);
let v = layout.evaluate(&wb.model, &layout.cell_key(0, 0).unwrap());
assert_eq!(
v.and_then(|v| v.as_f64()),
Some(111.0),
"2024 folds its months"
);
}
/// The flat split stays available as a choice, and is what it was.
#[test]
fn the_split_shape_keeps_the_raw_dates_and_one_category_per_component() {
use DateComponent::*;
let p = pipeline(&[Year, Month], DateShape::Split);
assert_eq!(p.proposed_category_count(), 3);
let wb = p.build_model().unwrap();
let mut names: Vec<&str> = wb.model.regular_category_names();
names.sort_unstable();
assert_eq!(names, vec!["Date", "Date_Month", "Date_Year"]);
assert!(
wb.model
.category("Date")
.unwrap()
.items
.contains_key("01/15/2024")
);
}
/// A value the detected format cannot read keeps its raw text as an
/// ungrouped item, and the build says so rather than dropping the record.
#[test]
fn an_unparsed_date_keeps_its_value_and_is_warned_about() {
use DateComponent::*;
let mut p = pipeline(&[Year, Month], DateShape::Hierarchy);
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
date.date_format = Some("%Y-%m-%d".into());
let (wb, warnings) = p.build_model_with_warnings().unwrap();
assert!(wb.model.category("Date_Year").is_none(), "one category");
let date = wb.model.category("Date").unwrap();
assert!(
date.items.contains_key("01/15/2024"),
"the raw value is kept"
);
assert!(
date.item_chain("01/15/2024").is_empty(),
"and it is ungrouped"
);
assert!(
warnings
.iter()
.any(|w| w.contains("'Date' kept its raw value in 4 row(s)")),
"{warnings:?}"
);
}
/// A %Y-%m-%d column imported Year > Month, one valid date and one raw
/// value spelled `raw`.
fn with_raw(raw: &str) -> ImportPipeline {
use DateComponent::*;
let mut p = ImportPipeline::new(json!([
{"Date": "2024-01-15", "Amount": 1.0},
{"Date": raw, "Amount": 2.0},
]));
p.on_duplicate = DuplicatePolicy::Sum;
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
date.kind = FieldKind::TimeCategory;
date.date_format = Some("%Y-%m-%d".into());
date.date_components = vec![Year, Month];
date.date_shape = DateShape::Hierarchy;
p
}
/// A raw value spelled like a generated month would merge with it.
#[test]
fn a_raw_date_spelled_like_a_generated_item_is_refused() {
let err = with_raw("2024-01").build_model_with_warnings().unwrap_err();
let msg = err.to_string();
assert!(msg.contains("'Date'") && msg.contains("2024-01"), "{msg}");
assert!(msg.contains("split"), "{msg}");
}
/// A raw value spelled like a generated year would sit beside its group.
#[test]
fn a_raw_date_spelled_like_a_generated_group_is_refused() {
let err = with_raw("2024").build_model_with_warnings().unwrap_err();
let msg = err.to_string();
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
}
/// The control: a raw value spelled like nothing generated still imports,
/// so the refusal above is not a blanket one.
#[test]
fn a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped() {
let (wb, warnings) = with_raw("sometime").build_model_with_warnings().unwrap();
let date = wb.model.category("Date").unwrap();
assert!(date.item_chain("sometime").is_empty());
assert!(date.items.contains_key("2024-01"), "{warnings:?}");
assert!(
warnings.iter().any(|w| w.contains("1 row(s)")),
"{warnings:?}"
);
}
/// The clash is answered on the date step, where `s` fixes it, not on the
/// last step three screens later (improvise-907).
#[test]
fn the_date_step_will_not_advance_over_a_spelling_clash() {
use super::{ImportWizard, WizardStep};
let mut w = ImportWizard::from_pipeline(with_raw("2024"));
w.step = WizardStep::ConfigureDates;
w.advance();
assert_eq!(w.step, WizardStep::ConfigureDates);
let msg = w.message.clone().unwrap_or_default();
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
w.toggle_date_shape();
w.advance();
assert_ne!(w.step, WizardStep::ConfigureDates, "split clears it");
}
}
+3
View File
@@ -967,6 +967,9 @@ pub struct WizardDateField {
/// what is drawn, so the substitution is made once, here. /// what is drawn, so the substitution is made once, here.
pub format: String, pub format: String,
pub components: Vec<(String, bool)>, pub components: Vec<(String, bool)>,
/// What the chosen components become, as a sentence: one hierarchical
/// category, or a split (improvise-8jk6.3). Worded once, in the projection.
pub shape: String,
} }
/// Several source records sharing every coordinate (improvise-cgqt). Data, /// Several source records sharing every coordinate (improvise-cgqt). Data,
+55 -2
View File
@@ -38,6 +38,8 @@ trait Runnable {
fn run(self, model_file: Option<PathBuf>) -> Result<()>; fn run(self, model_file: Option<PathBuf>) -> Result<()>;
} }
// Parsed once per process, so a variant's size costs nothing worth boxing for.
#[allow(clippy::large_enum_variant)]
#[derive(Subcommand)] #[derive(Subcommand)]
#[enum_dispatch(Runnable)] #[enum_dispatch(Runnable)]
enum Commands { enum Commands {
@@ -72,10 +74,16 @@ struct ImportArgs {
#[arg(long)] #[arg(long)]
skip: Vec<String>, skip: Vec<String>,
/// Extract date component, e.g. "Date:Month" (repeatable) /// Extract date component, e.g. "Date:Month" (repeatable). The components
/// nest in ONE category named after the column unless --split-date names it
#[arg(long)] #[arg(long)]
extract: Vec<String>, extract: Vec<String>,
/// Split a date field's components into one flat category each, beside its
/// raw values, rather than one hierarchical category (repeatable)
#[arg(long)]
split_date: Vec<String>,
/// Set category axis, e.g. "Payee:row" (repeatable) /// Set category axis, e.g. "Payee:row" (repeatable)
#[arg(long)] #[arg(long)]
axis: Vec<String>, axis: Vec<String>,
@@ -157,6 +165,7 @@ impl Runnable for ImportArgs {
time_fields: self.time, time_fields: self.time,
skip_fields: self.skip, skip_fields: self.skip,
extractions: parse_colon_pairs(&self.extract), extractions: parse_colon_pairs(&self.extract),
split_dates: self.split_date,
axes: parse_colon_pairs(&self.axis), axes: parse_colon_pairs(&self.axis),
formulas: self.formula, formulas: self.formula,
name: self.name, name: self.name,
@@ -192,6 +201,7 @@ struct ImportConfig {
time_fields: Vec<String>, time_fields: Vec<String>,
skip_fields: Vec<String>, skip_fields: Vec<String>,
extractions: Vec<(String, String)>, extractions: Vec<(String, String)>,
split_dates: Vec<String>,
axes: Vec<(String, String)>, axes: Vec<(String, String)>,
formulas: Vec<String>, formulas: Vec<String>,
name: Option<String>, name: Option<String>,
@@ -211,7 +221,7 @@ fn parse_colon_pairs(args: &[String]) -> Vec<(String, String)> {
use import::wizard::{DuplicatePolicy, ImportPipeline}; use import::wizard::{DuplicatePolicy, ImportPipeline};
fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) { fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) {
use import::analyzer::{DateComponent, FieldKind}; use import::analyzer::{DateComponent, DateShape, FieldKind};
// Override field kinds // Override field kinds
for p in &mut pipeline.proposals { for p in &mut pipeline.proposals {
@@ -244,6 +254,13 @@ fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, confi
} }
} }
// The wizard's `s`, headless: which date fields split rather than nest.
for p in &mut pipeline.proposals {
if config.split_dates.contains(&p.field) {
p.date_shape = DateShape::Split;
}
}
// Set formulas // Set formulas
pipeline.formulas = config.formulas.clone(); pipeline.formulas = config.formulas.clone();
@@ -294,6 +311,23 @@ fn apply_axis_overrides(wb: &mut Workbook, axes: &[(String, String)]) {
} }
} }
/// Why `--axis` is refused: the categories the import did build, and a date
/// hint only for a name shaped like one of the split date parts.
fn axis_refusal(cat: &str, built: &[String]) -> String {
let date_part = ["_Year", "_Month", "_Quarter"]
.iter()
.any(|suffix| cat.ends_with(suffix));
let hint = if date_part {
" A date's parts are one category unless --split-date names it."
} else {
""
};
format!(
"--axis names '{cat}', which this import did not build.{hint} Built: {}",
built.join(", ")
)
}
fn run_headless_import( fn run_headless_import(
mut pipeline: ImportPipeline, mut pipeline: ImportPipeline,
config: &ImportConfig, config: &ImportConfig,
@@ -305,6 +339,15 @@ fn run_headless_import(
if let (true, Some(skipped)) = (config.strict, pipeline.skip_warning()) { if let (true, Some(skipped)) = (config.strict, pipeline.skip_warning()) {
anyhow::bail!("--strict: {skipped}"); anyhow::bail!("--strict: {skipped}");
} }
if let Some(field) = config.split_dates.iter().find(|f| {
!pipeline.proposals.iter().any(|p| {
&p.field == *f
&& p.kind == import::analyzer::FieldKind::TimeCategory
&& p.date_format.is_some()
})
}) {
anyhow::bail!("--split-date '{field}' is not a date column of this import");
}
if pipeline.on_duplicate == DuplicatePolicy::Refuse { if pipeline.on_duplicate == DuplicatePolicy::Refuse {
let collisions = pipeline.collisions(); let collisions = pipeline.collisions();
if !collisions.is_empty() { if !collisions.is_empty() {
@@ -315,6 +358,16 @@ fn run_headless_import(
for warning in &warnings { for warning in &warnings {
eprintln!("warning: {warning}"); eprintln!("warning: {warning}");
} }
if let Some((cat, _)) = config
.axes
.iter()
.find(|(cat, _)| wb.model.category(cat).is_none())
{
anyhow::bail!(axis_refusal(
cat,
&wb.model.categories.keys().cloned().collect::<Vec<_>>()
));
}
apply_axis_overrides(&mut wb, &config.axes); apply_axis_overrides(&mut wb, &config.axes);
if let Some(path) = output.or(model_file) { if let Some(path) = output.or(model_file) {
+7 -4
View File
@@ -180,7 +180,10 @@ impl<'a> Widget for ImportWizardWidget<'a> {
if y >= inner.bottom().saturating_sub(2) { if y >= inner.bottom().saturating_sub(2) {
break; break;
} }
let header = format!(" {} (format: {})", proposal.field, proposal.format); let header = format!(
" {} (format: {}) — {}",
proposal.field, proposal.format, proposal.shape
);
panel.put( panel.put(
x, x,
y, y,
@@ -221,7 +224,7 @@ impl<'a> Widget for ImportWizardWidget<'a> {
x, x,
hint_y, hint_y,
w, w,
"Space: toggle Enter: next \u{2190} back Esc: cancel", "Space: toggle s: one category / split Enter: next \u{2190} back Esc: cancel",
Style::default().fg(Color::DarkGray), Style::default().fg(Color::DarkGray),
); );
} }
@@ -1178,7 +1181,7 @@ mod tests {
"ConfigureDates\n", "ConfigureDates\n",
"┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n", "┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n",
"│Select date components to extract (Space toggle): │\n", "│Select date components to extract (Space toggle): │\n",
"│ When (format: %Y-%m-%d) │\n", "│ When (format: %Y-%m-%d) — one category of the raw dates │\n",
"│ [ ] Year │\n", "│ [ ] Year │\n",
"│ [ ] Month │\n", "│ [ ] Month │\n",
"│ [ ] Quarter │\n", "│ [ ] Quarter │\n",
@@ -1196,7 +1199,7 @@ mod tests {
"│ │\n", "│ │\n",
"│ │\n", "│ │\n",
"│ │\n", "│ │\n",
"│Space: toggle Enter: next ← back Esc: cancel │\n", "│Space: toggle s: one category / split Enter: next ← back Esc: cancel │\n",
"└──────────────────────────────────────────────────────────────────────────────┘\n", "└──────────────────────────────────────────────────────────────────────────────┘\n",
"\n", "\n",
"====\n", "====\n",
+161
View File
@@ -665,6 +665,167 @@ fn a_headless_import_keeps_duplicates_separate_when_told_to() {
assert!(!doc.contains("= 30"), "{doc}"); assert!(!doc.contains("= 30"), "{doc}");
} }
/// Dates across two years, one per month, so nothing collides.
fn dated_csv(scratch: &Scratch) -> PathBuf {
let csv = scratch.path("dated.csv");
std::fs::write(
&csv,
"Date,Revenue\n2024-01-15,1\n2024-04-02,10\n2025-01-03,100\n",
)
.unwrap();
csv
}
/// improvise-8jk6.3, headless: `--extract` builds ONE hierarchical category by
/// default, as the wizard does, and it is written with its `>` lines.
#[test]
fn a_headless_date_import_is_one_hierarchical_category() {
let scratch = Scratch::new("date_hierarchy");
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&[
"--extract",
"Date:Year",
"--extract",
"Date:Quarter",
"--extract",
"Date:Month",
],
);
assert!(out.status.success(), "{}", stderr_of(&out));
let doc = std::fs::read_to_string(&out_path).unwrap();
assert!(!doc.contains("Date_Month"), "{doc}");
assert!(doc.contains("- |2024-01|[|2024-Q1|]"), "{doc}");
assert!(doc.contains("> |2024-Q1|[|2024|]"), "{doc}");
assert!(!doc.contains("expanded"), "it opens folded: {doc}");
}
/// The flat split stays available headless, as `s` makes it in the wizard.
#[test]
fn a_headless_date_import_splits_when_told_to() {
let scratch = Scratch::new("date_split");
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&["--extract", "Date:Month", "--split-date", "Date"],
);
assert!(out.status.success(), "{}", stderr_of(&out));
let doc = std::fs::read_to_string(&out_path).unwrap();
assert!(doc.contains("## Category: Date_Month"), "{doc}");
assert!(doc.contains("2024-01-15"), "the raw dates stay: {doc}");
}
/// improvise-8jk6.3 review: a `--split-date` that names nothing splittable
/// said nothing and exited 0.
#[test]
fn split_date_naming_an_unknown_or_non_date_field_is_refused() {
let scratch = Scratch::new("split_date_unknown");
for field in ["Nope", "Revenue"] {
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&["--extract", "Date:Month", "--split-date", field],
);
assert!(!out.status.success(), "{field}: {}", stderr_of(&out));
let err = stderr_of(&out);
assert!(err.contains("--split-date") && err.contains(field), "{err}");
}
}
/// improvise-8jk6.3 review: `--axis Date_Month:column` names a category the
/// hierarchical import no longer builds, and was dropped without a word.
#[test]
fn an_axis_naming_a_category_the_import_did_not_build_is_refused() {
let scratch = Scratch::new("axis_unknown");
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&["--extract", "Date:Month", "--axis", "Date_Month:column"],
);
assert!(!out.status.success(), "{}", stderr_of(&out));
let err = stderr_of(&out);
assert!(
err.contains("Date_Month") && err.contains("--split-date"),
"{err}"
);
}
/// `--time` on a column with no date format makes it a time category without
/// dates, so there is nothing to split.
#[test]
fn split_date_on_a_time_field_with_no_date_format_is_refused() {
let scratch = Scratch::new("split_date_no_format");
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&["--time", "Revenue", "--split-date", "Revenue"],
);
assert!(!out.status.success(), "{}", stderr_of(&out));
assert!(
stderr_of(&out).contains("--split-date"),
"{}",
stderr_of(&out)
);
}
/// A name that is no date part gets the categories that exist, never the date
/// hint (the README's old `--axis Measure:column` was this).
#[test]
fn an_absent_axis_category_that_is_no_date_part_names_what_was_built() {
let scratch = Scratch::new("axis_unknown_plain");
let out_path = scratch.path("out.improv");
let out = run_import(
&dated_csv(&scratch),
&out_path,
&["--axis", "Measure:column"],
);
assert!(!out.status.success(), "{}", stderr_of(&out));
let err = stderr_of(&out);
assert!(
err.contains("'Measure'") && err.contains("_Measure"),
"{err}"
);
assert!(!err.contains("--split-date"), "no date hint: {err}");
}
/// The README's import recipe runs as printed.
#[test]
fn the_readmes_import_recipe_runs() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
let readme = std::fs::read_to_string(root.join("README.md")).unwrap();
let start = readme.find("improvise import examples/demo.csv").unwrap();
let recipe = readme[start..]
.split("```")
.next()
.unwrap()
.replace("\\\n", " ");
let mut args: Vec<String> = vec![];
for (i, part) in recipe.split('"').enumerate() {
match i % 2 {
1 => args.push(part.to_string()),
_ => args.extend(part.split_whitespace().map(String::from)),
}
}
let at = args.iter().position(|a| a == "-o").unwrap();
args.truncate(at);
let scratch = Scratch::new("readme_recipe");
let out = Command::new(bin())
.arg("import")
.arg(root.join("examples/demo.csv"))
.args(args.iter().skip(3))
.arg("-o")
.arg(scratch.path("out.improv"))
.output()
.unwrap();
assert!(out.status.success(), "{}", stderr_of(&out));
}
/// code-maat's reports: three schemas, a stray note, and room for the output. /// code-maat's reports: three schemas, a stray note, and room for the output.
fn code_maat_folder(scratch: &Scratch) -> PathBuf { fn code_maat_folder(scratch: &Scratch) -> PathBuf {
let dir = scratch.path("maat"); let dir = scratch.path("maat");