Merge pull request #322 from fiddlerwoaroof/feat/8jk6.3-date-hierarchy
feat(import): a date column imports as one hierarchical category (improvise-8jk6.3)
This commit is contained in:
Ed Langley
GitHub
@@ -40,9 +40,9 @@ improvise import examples/demo.csv \
|
||||
--no-wizard \
|
||||
--category Region --category Product --category Customer \
|
||||
--measure Revenue --measure Cost \
|
||||
--time Date --extract Date:Month \
|
||||
--time Date --extract Date:Month --split-date Date \
|
||||
--axis Region:row --axis Product:row \
|
||||
--axis Date_Month:column --axis Measure:column \
|
||||
--axis Date_Month:column --axis _Measure:column \
|
||||
--axis Customer:page --axis Date:none \
|
||||
--formula "Profit = Revenue - Cost" \
|
||||
--name "Acme Sales Demo" \
|
||||
|
||||
@@ -613,6 +613,59 @@ different row order on screen, and a different detected date format, since
|
||||
varied. The `HashSet` in `analyze_records` is there to deduplicate and **never
|
||||
to iterate**.
|
||||
|
||||
### A date column imports as ONE hierarchical category (improvise-8jk6.3)
|
||||
|
||||
The user: "I think dates should usually be one category and we can use
|
||||
formulas to crosstab." So the default `DateShape::Hierarchy` builds one
|
||||
category named after the column, where the old import built up to three flat
|
||||
ones beside it (`Date_Year`, `Date_Quarter`, `Date_Month`):
|
||||
|
||||
- **Its items are the finest chosen component**, spelled so they stay unique
|
||||
across years: `2024-01`, `2024-Q1`, `2024`, which are
|
||||
`extract_date_component`'s existing spellings. A month is never `Jan`.
|
||||
- **The coarser chosen components are its groups**, coarsest first
|
||||
(`DateComponent::COARSE_TO_FINE`): a month sits in its quarter, and the
|
||||
quarter's `>` line names its year. A level left unchosen is skipped rather
|
||||
than drawn empty, so Year + Month groups months straight into years.
|
||||
- **It opens folded**, one row per year, with nothing to set, because every
|
||||
group starts closed (`context/code/core.md` §19).
|
||||
- **It costs one of the category slots**, not up to four, and
|
||||
`proposed_category_count` counts it that way.
|
||||
- **The raw dates are gone from that category.** The day is not a component, so
|
||||
two records of one month land on one cell and the collision step asks, as it
|
||||
does for any other collision. `Split` keeps the day.
|
||||
- **A value the detected format cannot read keeps its raw text** as an
|
||||
ungrouped item, with a warning naming the column and the row count, rather
|
||||
than being dropped. The warning is separate from the split shape's, because
|
||||
that one reports an absent coordinate and this one a coordinate that is there
|
||||
and ungrouped. **Unless its spelling is one the hierarchy generates**
|
||||
(improvise-d0n6): a raw `2024-01` or `2024` is the same name as the generated
|
||||
month or year, so it would join the grouped item and the warning would be
|
||||
false. The build **refuses**, naming the column and the spellings and pointing
|
||||
at the split shape. That follows the collision rule below: where two records
|
||||
would silently be one thing, the user is asked and nothing is defaulted. A
|
||||
fallback to the split shape was rejected as a silent change of the user's
|
||||
choice, and a renamed raw item as altering the data. The check is
|
||||
`Plan::spelling_clash`.
|
||||
- **Item order is first-seen, not chronological.** Order is a correctness
|
||||
property here (gotcha 18), and a group need not be contiguous, so a CSV that
|
||||
is not in date order still nests correctly. It is just drawn in the order
|
||||
it was read.
|
||||
|
||||
`DateShape::Split` is the old behaviour, unchanged, and stays a choice. `s` on
|
||||
the wizard's date step toggles the field under the cursor, and the header line
|
||||
for each field says which it will get ("one category: Year > Quarter > Month"
|
||||
or "split: Date, Date_Year, …"). It is worded once, in
|
||||
`project.rs::date_shape_sentence`. Headless, `--split-date <field>` does the
|
||||
same, and `--extract` keeps its meaning: which components to build.
|
||||
|
||||
**Crosstabs by formula are not available yet.** A formula can name a month
|
||||
(`SUM(Amount WHERE Date = "2024-01")`, `Date IS "2024-01"`), but a `WHERE` or
|
||||
`IS` naming a GROUP matches no record: both compare the coordinate to the name
|
||||
exactly. `SUM(Amount WHERE Date = "2024")` is an empty sum, and `Date IS "2024"`
|
||||
is 0 on every month of 2024. That was measured on this branch and reported for
|
||||
filing, not built here.
|
||||
|
||||
### The ambiguous-date heuristic reads American-first, deliberately
|
||||
|
||||
`01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data
|
||||
|
||||
@@ -1685,6 +1685,44 @@ closed, the one its `collapsed:` line names and the three it does not, and the
|
||||
line is written back. Rival: the old open default; this test fails with the
|
||||
other eleven in that rival's list.
|
||||
|
||||
### a_date_column_imports_as_one_category_of_months_in_quarters_in_years
|
||||
|
||||
improvise-8jk6.3, the user's "dates should usually be one category". The
|
||||
fixture has two days of one month, so that month is one summed cell, and
|
||||
January of the next year, so a month must stay unique across years. Rivals, each
|
||||
run against the `date_hierarchy` module, and each caught here:
|
||||
- The split as the default shape.
|
||||
- Levels taken finest first, so the year hangs under the month.
|
||||
- Groups registered without their parents.
|
||||
- The raw dates still added beside the months.
|
||||
- A hierarchy counted as `1 + components` slots.
|
||||
|
||||
### a_level_left_unchosen_is_skipped_not_emptied
|
||||
|
||||
Year + Month nests months straight into years, and Quarter alone is a flat
|
||||
category of quarters. Caught the finest-first rival and the raw-dates rival.
|
||||
|
||||
### an_imported_date_hierarchy_opens_folded_to_its_years
|
||||
|
||||
The import sets nothing: the hierarchy opens folded, one row per year, because
|
||||
every group starts closed (improvise-felo.6). The 2024 row is the fold of its
|
||||
months, 111. Caught three of the rivals above.
|
||||
|
||||
### an_unparsed_date_keeps_its_value_and_is_warned_about
|
||||
|
||||
A format that reads none of the values keeps each raw value as an ungrouped
|
||||
item, and the build says so. Its first version passed on the base commit: the
|
||||
shape was ignored there and the SPLIT's warning happened to contain the same
|
||||
words. It now also asserts there is no `Date_Year`. Rival: the unparsed value
|
||||
not counted; this test fails.
|
||||
|
||||
### a_headless_date_import_splits_when_told_to
|
||||
|
||||
`--split-date` is the wizard's `s`, headless. It failed first as an unknown
|
||||
argument. Rival: the flag parsed and ignored; this test fails, and
|
||||
`a_headless_date_import_is_one_hierarchical_category` does not, which is the
|
||||
point of having both.
|
||||
|
||||
### every_fold_the_api_can_build_reads_back_from_the_file_it_writes
|
||||
|
||||
ChatGPT Codex, P2 on #317 at `e09ec752`: `GroupFold::Expanded(usize)` and
|
||||
@@ -1939,6 +1977,45 @@ predates this PR (improvise-ych), but the sparse-record rule now rests on it:
|
||||
without it the value becomes an item named `" "`. Rival F4 (the reviewer's
|
||||
fault) fails it.
|
||||
|
||||
### a_raw_date_spelled_like_a_generated_item_is_refused
|
||||
|
||||
improvise-d0n6 (Codex P2 on PR #322). A `%Y-%m-%d` column imported Year > Month
|
||||
with `2024-01-15` and a raw `2024-01`. Before the refusal the build succeeded,
|
||||
put both records on the grouped item `2024-01` (summing them) and warned that
|
||||
the raw value was "kept ungrouped", which was false. Fault injected: removing
|
||||
the `spelling_clash` call makes this test fail with `unwrap_err()` on an `Ok`.
|
||||
|
||||
### a_raw_date_spelled_like_a_generated_group_is_refused
|
||||
|
||||
improvise-d0n6. The same with a raw `2024`, a generated group. Removing the
|
||||
`spelling_clash` call fails it the same way. The control,
|
||||
`a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped`, passes either
|
||||
way, so a blanket refusal of every unparsed value fails it.
|
||||
|
||||
### the_date_step_will_not_advance_over_a_spelling_clash
|
||||
|
||||
improvise-d0n6 (Copilot on PR #322, and the adversarial review's B1). The
|
||||
spelling-clash refusal sat only in the build, so a wizard with a raw `2024` in
|
||||
a `%Y-%m-%d` column passed ConfigureDates and failed on NameModel, three steps
|
||||
from the `s` that fixes it. Before the check joined `validation_error` the
|
||||
test failed with the step already `DefineFormulas`; removing that call again
|
||||
reproduces it. It also toggles to the split shape and requires the step to
|
||||
advance, so a check that blocks forever fails.
|
||||
|
||||
### split_date_naming_an_unknown_or_non_date_field_is_refused
|
||||
|
||||
improvise-tpm8 class. `--split-date Nope` and `--split-date Revenue` (a
|
||||
measure) exited 0 and did nothing. Both now exit 1 naming the flag and the
|
||||
field; removing the check in `run_headless_import` fails both rows.
|
||||
|
||||
### an_axis_naming_a_category_the_import_did_not_build_is_refused
|
||||
|
||||
improvise-8jk6.3 review. `--extract Date:Month --axis Date_Month:column` used
|
||||
to build one `Date` category and drop the `--axis` for the vanished
|
||||
`Date_Month`, exit 0. It now exits 1 naming the category and `--split-date`.
|
||||
This also refuses any `--axis` naming an absent category, which was silently
|
||||
ignored before this PR as well.
|
||||
|
||||
### a_folder_of_only_header_only_files_keeps_no_record_and_refuses
|
||||
|
||||
improvise-8jk6.6 meeting PR #314's rule that a header-only file is still a
|
||||
@@ -1949,3 +2026,23 @@ items and empty views: the category check alone cannot refuse. The `setup`
|
||||
assertion proves that seeded proposal exists, so the test is not passing only
|
||||
because there was no category. **R7**, the check removed from the build, fails
|
||||
it. The user confirmed the refusal on PR #321: "Refuse (as #321 does)".
|
||||
|
||||
### an_absent_axis_category_that_is_no_date_part_names_what_was_built
|
||||
|
||||
improvise-8jk6.3 review (B2). The README's own recipe passed
|
||||
`--axis Measure:column`, a silent no-op because the category is `_Measure`; the
|
||||
first refusal blamed dates for it. The test uses a non-date absent name and
|
||||
asserts the message lists the built categories and carries no date hint.
|
||||
Fault injected: making the hint unconditional fails it.
|
||||
|
||||
### the_readmes_import_recipe_runs
|
||||
|
||||
improvise-8jk6.3 review (B2). Runs the README's recipe as printed. Fault
|
||||
injected: restoring `--axis Measure:column` in the README fails it with the
|
||||
refusal.
|
||||
|
||||
### split_date_on_a_time_field_with_no_date_format_is_refused
|
||||
|
||||
improvise-8jk6.3 review. `--time Revenue --split-date Revenue` made the field a
|
||||
time category with no date format and the split was a silent no-op. Without
|
||||
the `date_format.is_some()` condition the import exits 0 and this fails.
|
||||
|
||||
@@ -65,6 +65,7 @@ fn review_proposal_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
||||
fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
||||
match key {
|
||||
KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent),
|
||||
KeyCode::Char('s') => op(WizardOp::ToggleDateShape),
|
||||
KeyCode::Enter => op(WizardOp::Advance),
|
||||
_ => cursor_keys(key),
|
||||
}
|
||||
|
||||
@@ -1053,6 +1053,7 @@ pub enum WizardOp {
|
||||
ToggleProposal,
|
||||
CycleProposalKind,
|
||||
ToggleDateComponent,
|
||||
ToggleDateShape,
|
||||
StartFormulaEdit,
|
||||
DeleteFormula,
|
||||
ConfirmFormula,
|
||||
@@ -1078,6 +1079,7 @@ impl WizardOp {
|
||||
Self::ToggleProposal => wizard.toggle_proposal(),
|
||||
Self::CycleProposalKind => wizard.cycle_proposal_kind(),
|
||||
Self::ToggleDateComponent => wizard.toggle_date_component(),
|
||||
Self::ToggleDateShape => wizard.toggle_date_shape(),
|
||||
Self::StartFormulaEdit => wizard.start_formula_edit(),
|
||||
Self::DeleteFormula => wizard.delete_formula(),
|
||||
Self::ConfirmFormula => wizard.confirm_formula(),
|
||||
|
||||
@@ -11,7 +11,7 @@ use improvise_core::model::record::{AggregateHash, RecordHash};
|
||||
use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol};
|
||||
use improvise_core::view::{Axis, View};
|
||||
use improvise_core::workbook::Workbook;
|
||||
use improvise_io::import::analyzer::{DateComponent, FieldKind};
|
||||
use improvise_io::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep};
|
||||
use improvise_protocol::{
|
||||
AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay,
|
||||
@@ -662,6 +662,7 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
||||
((*label).to_string(), p.date_components.contains(component))
|
||||
})
|
||||
.collect(),
|
||||
shape: date_shape_sentence(p),
|
||||
})
|
||||
.collect(),
|
||||
collisions: w
|
||||
@@ -700,6 +701,27 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
||||
}
|
||||
}
|
||||
|
||||
/// What a date field's chosen components become, worded for every front end.
|
||||
fn date_shape_sentence(p: &improvise_io::import::analyzer::FieldProposal) -> String {
|
||||
let chosen = |order: &[DateComponent]| -> Vec<&str> {
|
||||
order
|
||||
.iter()
|
||||
.filter(|c| p.date_components.contains(c))
|
||||
.filter_map(|c| DATE_COMPONENTS.iter().find(|(d, _)| d == c))
|
||||
.map(|(_, label)| *label)
|
||||
.collect()
|
||||
};
|
||||
match (p.date_shape, chosen(&DateComponent::COARSE_TO_FINE)) {
|
||||
(_, none) if none.is_empty() => "one category of the raw dates".to_string(),
|
||||
(DateShape::Hierarchy, levels) => format!("one category: {}", levels.join(" > ")),
|
||||
(DateShape::Split, _) => {
|
||||
let parts = chosen(&DATE_COMPONENTS.map(|(c, _)| c));
|
||||
let names: Vec<String> = parts.iter().map(|l| format!("{}_{l}", p.field)).collect();
|
||||
format!("split: {}, {}", p.field, names.join(", "))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The date components the wizard offers, **in the order it offers them**:
|
||||
/// `ImportWizard::date_config_at_cursor` counts these same rows.
|
||||
const DATE_COMPONENTS: [(DateComponent, &str); 3] = [
|
||||
@@ -6522,3 +6544,42 @@ mod column_totals {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3: the wizard's date step says what each date field becomes,
|
||||
/// and `s` switches it between one category and the split.
|
||||
#[cfg(test)]
|
||||
mod date_shape {
|
||||
use super::wizard_cache;
|
||||
use crate::import::wizard::{ImportWizard, WizardStep};
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn the_date_step_says_what_a_date_field_becomes_and_s_switches_it() {
|
||||
let mut w = ImportWizard::new(json!([
|
||||
{"Region": "East", "When": "2024-01-15", "Revenue": 1.0},
|
||||
{"Region": "West", "When": "2024-04-02", "Revenue": 2.0},
|
||||
]));
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||
let shape = |w: &ImportWizard| wizard_cache(w).date_fields[0].shape.clone();
|
||||
assert_eq!(shape(&w), "one category of the raw dates");
|
||||
w.toggle_date_component();
|
||||
w.move_cursor(1);
|
||||
w.toggle_date_component();
|
||||
w.move_cursor(1);
|
||||
w.toggle_date_component();
|
||||
assert_eq!(shape(&w), "one category: Year > Quarter > Month");
|
||||
w.toggle_date_shape();
|
||||
assert_eq!(
|
||||
shape(&w),
|
||||
"split: When, When_Year, When_Month, When_Quarter"
|
||||
);
|
||||
let built = w.pipeline.build_model().unwrap();
|
||||
assert!(built.model.category("When_Quarter").is_some());
|
||||
w.toggle_date_shape();
|
||||
let built = w.pipeline.build_model().unwrap();
|
||||
assert!(built.model.category("When_Quarter").is_none());
|
||||
let when = built.model.category("When").unwrap();
|
||||
assert_eq!(when.item_chain("2024-04"), vec!["2024", "2024-Q2"]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,6 +22,26 @@ pub enum DateComponent {
|
||||
Quarter,
|
||||
}
|
||||
|
||||
impl DateComponent {
|
||||
/// The components as a hierarchy nests them, coarsest first.
|
||||
pub const COARSE_TO_FINE: [DateComponent; 3] = [
|
||||
DateComponent::Year,
|
||||
DateComponent::Quarter,
|
||||
DateComponent::Month,
|
||||
];
|
||||
}
|
||||
|
||||
/// How a date column's components become categories (improvise-8jk6.3).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum DateShape {
|
||||
/// ONE category named after the column: its items are the finest chosen
|
||||
/// component, grouped by the coarser ones. The user's default.
|
||||
#[default]
|
||||
Hierarchy,
|
||||
/// The column's raw values, plus one flat category per chosen component.
|
||||
Split,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FieldProposal {
|
||||
pub field: String,
|
||||
@@ -32,6 +52,8 @@ pub struct FieldProposal {
|
||||
pub date_format: Option<String>,
|
||||
/// Which date components to extract as new categories.
|
||||
pub date_components: Vec<DateComponent>,
|
||||
/// Whether those components nest in one category or split into several.
|
||||
pub date_shape: DateShape,
|
||||
}
|
||||
|
||||
impl FieldProposal {
|
||||
@@ -132,6 +154,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -159,6 +182,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -170,6 +194,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -180,6 +205,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -191,6 +217,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
use std::collections::HashMap;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
use serde_json::Value;
|
||||
|
||||
use super::analyzer::{
|
||||
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||
DateComponent, DateShape, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||
extract_date_component, find_array_paths,
|
||||
};
|
||||
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile};
|
||||
@@ -239,13 +239,23 @@ impl ImportPipeline {
|
||||
.filter(|p| p.accepted)
|
||||
.map(|p| match p.kind {
|
||||
FieldKind::Category => 1,
|
||||
FieldKind::TimeCategory if p.date_format.is_some() => 1 + p.date_components.len(),
|
||||
FieldKind::TimeCategory if p.date_format.is_some() => match p.date_shape {
|
||||
DateShape::Hierarchy => 1,
|
||||
DateShape::Split => 1 + p.date_components.len(),
|
||||
},
|
||||
FieldKind::TimeCategory => 1,
|
||||
FieldKind::Measure | FieldKind::Label => 0,
|
||||
})
|
||||
.sum()
|
||||
}
|
||||
|
||||
/// Why the date step cannot advance: see [`Plan::spelling_clash`].
|
||||
pub fn spelling_clash(&self) -> Option<String> {
|
||||
let plan = self.plan();
|
||||
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
|
||||
plan.spelling_clash(&placements)
|
||||
}
|
||||
|
||||
/// Every coordinate tuple more than one record lands on, in first-seen
|
||||
/// order. Known before the build, so the wizard can ask (improvise-cgqt).
|
||||
pub fn collisions(&self) -> Vec<Collision> {
|
||||
@@ -300,6 +310,7 @@ impl ImportPipeline {
|
||||
&& p.kind == FieldKind::TimeCategory
|
||||
&& p.date_format.is_some()
|
||||
&& !p.date_components.is_empty()
|
||||
&& p.date_shape == DateShape::Split
|
||||
})
|
||||
.flat_map(|p| {
|
||||
let fmt = p.date_format.as_deref().unwrap();
|
||||
@@ -314,11 +325,17 @@ impl ImportPipeline {
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let categories = accepted(&[FieldKind::Category, FieldKind::TimeCategory]);
|
||||
let date_hierarchies = categories
|
||||
.iter()
|
||||
.filter_map(|p| DateHierarchy::of(p))
|
||||
.collect();
|
||||
Plan {
|
||||
categories: accepted(&[FieldKind::Category, FieldKind::TimeCategory]),
|
||||
categories,
|
||||
measures: accepted(&[FieldKind::Measure]),
|
||||
labels: accepted(&[FieldKind::Label]),
|
||||
date_extractions,
|
||||
date_hierarchies,
|
||||
scalars: self.scalars,
|
||||
}
|
||||
}
|
||||
@@ -339,6 +356,9 @@ impl ImportPipeline {
|
||||
let formulas = self.parsed_formulas()?;
|
||||
|
||||
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
|
||||
if let Some(clash) = plan.spelling_clash(&placements) {
|
||||
return Err(anyhow!(clash));
|
||||
}
|
||||
let collisions = collisions_among(&placements);
|
||||
let occurrence = match (&self.on_duplicate, collisions.is_empty()) {
|
||||
(_, true) | (DuplicatePolicy::Sum, false) => None,
|
||||
@@ -350,11 +370,16 @@ impl ImportPipeline {
|
||||
|
||||
for cat_proposal in categories {
|
||||
wb.add_category(&cat_proposal.field)?;
|
||||
for val in cat_proposal
|
||||
.distinct_values
|
||||
// A hierarchical date column's items are its components, added below.
|
||||
let raw_items = match plan
|
||||
.date_hierarchies
|
||||
.iter()
|
||||
.filter(|v| !v.trim().is_empty())
|
||||
.any(|h| h.field == cat_proposal.field)
|
||||
{
|
||||
true => &[][..],
|
||||
false => &cat_proposal.distinct_values[..],
|
||||
};
|
||||
for val in raw_items.iter().filter(|v| !v.trim().is_empty()) {
|
||||
wb.add_item(&cat_proposal.field, val)?;
|
||||
}
|
||||
}
|
||||
@@ -387,7 +412,7 @@ impl ImportPipeline {
|
||||
|
||||
// Per-extraction tally of values the detected format could not parse. A
|
||||
// derived coordinate that is merely absent is improvise-yss5.
|
||||
let mut unparsed_dates = vec![0usize; date_extractions.len()];
|
||||
let mut unparsed_dates = vec![0usize; date_extractions.len() + plan.date_hierarchies.len()];
|
||||
// Two rows sharing every coordinate are one cell; last-row-wins lost the
|
||||
// earlier one (improvise-2cw9). The map only deduplicates — gotcha 18.
|
||||
let mut cells: Vec<PendingCell> = vec![];
|
||||
@@ -405,7 +430,11 @@ impl ImportPipeline {
|
||||
mut coords,
|
||||
measures,
|
||||
unparsed,
|
||||
chains,
|
||||
} => {
|
||||
for chain in &chains {
|
||||
add_date_chain(&mut wb, chain)?;
|
||||
}
|
||||
for (cat, item) in &coords {
|
||||
wb.add_item(cat, item)?;
|
||||
}
|
||||
@@ -457,7 +486,21 @@ impl ImportPipeline {
|
||||
}
|
||||
|
||||
let mut warnings: Vec<String> = self.skip_warning().into_iter().collect();
|
||||
warnings.extend(unparsed_date_warnings(date_extractions, &unparsed_dates));
|
||||
let (split, whole) = unparsed_dates.split_at(date_extractions.len());
|
||||
warnings.extend(unparsed_date_warnings(date_extractions, split));
|
||||
warnings.extend(
|
||||
plan.date_hierarchies
|
||||
.iter()
|
||||
.zip(whole)
|
||||
.filter(|(_, n)| **n > 0)
|
||||
.map(|(h, n)| {
|
||||
format!(
|
||||
"'{}' kept its raw value in {n} row(s), ungrouped; it did not match \
|
||||
the date format '{}'",
|
||||
h.field, h.format
|
||||
)
|
||||
}),
|
||||
);
|
||||
warnings.extend(collision_warning(&summed));
|
||||
warnings.extend(occurrence.map(|name| separated_note(name, &collisions)));
|
||||
|
||||
@@ -475,6 +518,7 @@ fn unseen(field: &str, kind: FieldKind, distinct_values: Vec<String>) -> FieldPr
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -552,9 +596,43 @@ struct Plan<'a> {
|
||||
measures: Vec<&'a FieldProposal>,
|
||||
labels: Vec<&'a FieldProposal>,
|
||||
date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>,
|
||||
date_hierarchies: Vec<DateHierarchy<'a>>,
|
||||
scalars: Scalars,
|
||||
}
|
||||
|
||||
/// A date column imported as ONE category: its items are the finest chosen
|
||||
/// component, grouped by the coarser ones (improvise-8jk6.3, core.md §19).
|
||||
struct DateHierarchy<'a> {
|
||||
field: &'a str,
|
||||
format: &'a str,
|
||||
/// The chosen components, coarsest first.
|
||||
levels: Vec<DateComponent>,
|
||||
}
|
||||
|
||||
impl<'a> DateHierarchy<'a> {
|
||||
fn of(p: &'a FieldProposal) -> Option<Self> {
|
||||
let levels: Vec<DateComponent> = DateComponent::COARSE_TO_FINE
|
||||
.into_iter()
|
||||
.filter(|c| p.date_components.contains(c))
|
||||
.collect();
|
||||
let hierarchical =
|
||||
p.kind == FieldKind::TimeCategory && p.date_shape == DateShape::Hierarchy;
|
||||
(hierarchical && !levels.is_empty()).then_some(Self {
|
||||
field: &p.field,
|
||||
format: p.date_format.as_deref()?,
|
||||
levels,
|
||||
})
|
||||
}
|
||||
|
||||
/// Every level's name for one value, root first; the last is the item.
|
||||
fn chain(&self, value: &str) -> Option<Vec<String>> {
|
||||
self.levels
|
||||
.iter()
|
||||
.map(|c| extract_date_component(value, self.format, *c))
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Where one source record lands. Worked out once, for the build and for the
|
||||
/// collision check alike, so the two cannot disagree (improvise-cgqt).
|
||||
enum Placement {
|
||||
@@ -564,6 +642,7 @@ enum Placement {
|
||||
coords: Vec<(String, String)>,
|
||||
measures: Vec<(String, f64)>,
|
||||
unparsed: Vec<usize>,
|
||||
chains: Vec<DateChain>,
|
||||
},
|
||||
NotObject,
|
||||
}
|
||||
@@ -575,6 +654,7 @@ impl Plan<'_> {
|
||||
};
|
||||
let mut coords: Vec<(String, String)> = vec![];
|
||||
let mut unparsed: Vec<usize> = vec![];
|
||||
let mut chains: Vec<DateChain> = vec![];
|
||||
|
||||
// No value is no coordinate: records are sparse (improvise-8jk6.2).
|
||||
let named = |field: &str| map.get(field).and_then(scalar_name);
|
||||
@@ -583,7 +663,24 @@ impl Plan<'_> {
|
||||
.iter()
|
||||
.filter_map(|p| Some((p, named(&p.field)?)))
|
||||
{
|
||||
coords.push((cat_proposal.field.clone(), v.clone()));
|
||||
let hierarchy = self
|
||||
.date_hierarchies
|
||||
.iter()
|
||||
.enumerate()
|
||||
.find(|(_, h)| h.field == cat_proposal.field);
|
||||
let item = match hierarchy.map(|(at, h)| (at, h.chain(&v))) {
|
||||
Some((_, Some(chain))) => {
|
||||
let item = chain.last().cloned().unwrap_or_default();
|
||||
chains.push((cat_proposal.field.clone(), chain));
|
||||
item
|
||||
}
|
||||
Some((at, None)) => {
|
||||
unparsed.push(self.date_extractions.len() + at);
|
||||
v.clone()
|
||||
}
|
||||
None => v.clone(),
|
||||
};
|
||||
coords.push((cat_proposal.field.clone(), item));
|
||||
|
||||
// Extract date components from this field's value
|
||||
for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() {
|
||||
@@ -615,10 +712,83 @@ impl Plan<'_> {
|
||||
coords,
|
||||
measures,
|
||||
unparsed,
|
||||
chains,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Plan<'_> {
|
||||
/// Why the build stops when an unparsed date is spelled like a generated
|
||||
/// item or group (improvise-d0n6; formula-and-io.md, date hierarchy).
|
||||
fn spelling_clash(&self, placements: &[Placement]) -> Option<String> {
|
||||
let generated = |field: &str| -> HashSet<&str> {
|
||||
placements
|
||||
.iter()
|
||||
.filter_map(|p| match p {
|
||||
Placement::Placed { chains, .. } => Some(chains),
|
||||
Placement::NotObject => None,
|
||||
})
|
||||
.flatten()
|
||||
.filter(|(cat, _)| cat == field)
|
||||
.flat_map(|(_, chain)| chain.iter().map(String::as_str))
|
||||
.collect()
|
||||
};
|
||||
for (at, h) in self.date_hierarchies.iter().enumerate() {
|
||||
let tally = self.date_extractions.len() + at;
|
||||
let names = generated(h.field);
|
||||
let mut clashing: Vec<&str> = vec![];
|
||||
for p in placements {
|
||||
let Placement::Placed {
|
||||
coords, unparsed, ..
|
||||
} = p
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let raw = coords.iter().find(|(c, _)| c == h.field).map(|(_, v)| v);
|
||||
let clash = raw.filter(|v| {
|
||||
unparsed.contains(&tally)
|
||||
&& names.contains(v.as_str())
|
||||
&& !clashing.contains(&v.as_str())
|
||||
});
|
||||
clashing.extend(clash.map(String::as_str));
|
||||
}
|
||||
if !clashing.is_empty() {
|
||||
return Some(format!(
|
||||
"'{}' has value(s) that do not match the date format '{}' but are spelled \
|
||||
like a month, quarter or year it generates: {}; fix them, or import the \
|
||||
column with the split shape",
|
||||
h.field,
|
||||
h.format,
|
||||
clashing.join(", ")
|
||||
));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// One hierarchical date coordinate: its category and its chain, root first.
|
||||
type DateChain = (String, Vec<String>);
|
||||
|
||||
/// Register a date item under its groups, and each group under its parent.
|
||||
fn add_date_chain(wb: &mut Workbook, (cat, chain): &DateChain) -> Result<()> {
|
||||
let Some((item, groups)) = chain.split_last() else {
|
||||
return Ok(());
|
||||
};
|
||||
match groups.last() {
|
||||
Some(group) => wb.add_item_in_group(cat, item, group)?,
|
||||
None => wb.add_item(cat, item)?,
|
||||
};
|
||||
let category = wb
|
||||
.model
|
||||
.category_mut(cat)
|
||||
.ok_or_else(|| anyhow!("category '{cat}' vanished mid-import"))?;
|
||||
for pair in groups.windows(2) {
|
||||
category.add_group(crate::model::category::Group::new(&pair[1]).with_parent(&pair[0]));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A tuple, and each record on it: its 1-based position and the measures it writes.
|
||||
type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>);
|
||||
|
||||
@@ -921,7 +1091,9 @@ impl ImportWizard {
|
||||
.nothing_to_import()
|
||||
.or_else(|| self.no_category_message())
|
||||
.or_else(|| self.category_limit_message()),
|
||||
WizardStep::ConfigureDates => self.category_limit_message(),
|
||||
WizardStep::ConfigureDates => self
|
||||
.category_limit_message()
|
||||
.or_else(|| self.pipeline.spelling_clash()),
|
||||
WizardStep::ResolveDuplicates => match &self.pipeline.on_duplicate {
|
||||
DuplicatePolicy::Sum => None,
|
||||
DuplicatePolicy::Separate(_) => self.separate_blocked(),
|
||||
@@ -1199,6 +1371,18 @@ impl ImportWizard {
|
||||
}
|
||||
}
|
||||
|
||||
/// Switch the field at the cursor between ONE hierarchical category and the
|
||||
/// flat split into one category per component (improvise-8jk6.3).
|
||||
pub fn toggle_date_shape(&mut self) {
|
||||
if let Some((pi, _)) = self.date_config_at_cursor() {
|
||||
let proposal = &mut self.pipeline.proposals[pi];
|
||||
proposal.date_shape = match proposal.date_shape {
|
||||
DateShape::Hierarchy => DateShape::Split,
|
||||
DateShape::Split => DateShape::Hierarchy,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── Formula editing ────────────────────────────────────────────────────────
|
||||
|
||||
/// Buffer for typing a new formula in the DefineFormulas step.
|
||||
@@ -1511,6 +1695,7 @@ mod tests {
|
||||
prop.date_format = Some("%m/%d/%Y".to_string());
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_components.push(DateComponent::Year);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
p
|
||||
@@ -1614,6 +1799,7 @@ mod tests {
|
||||
for prop in &mut p.proposals {
|
||||
if prop.field == "Date" && prop.kind == FieldKind::TimeCategory {
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
let wb = p.build_model().unwrap();
|
||||
@@ -2484,11 +2670,39 @@ mod tests {
|
||||
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
||||
assert_eq!(date.kind, FieldKind::TimeCategory);
|
||||
date.date_components = vec![DateComponent::Month];
|
||||
date.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
let (wb, _) = p.build_model_with_warnings().unwrap();
|
||||
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
|
||||
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3 under improvise-8jk6.12: the default shape puts the date
|
||||
/// parts IN the file's own column, so that column is what its view seats.
|
||||
#[test]
|
||||
fn a_files_view_seats_its_own_hierarchical_date_column() {
|
||||
use crate::import::analyzer::DateComponent;
|
||||
let mut p = merged_pipeline(&[
|
||||
(
|
||||
"m/abs-churn.csv",
|
||||
"date,added
|
||||
2026-01-02,10
|
||||
",
|
||||
),
|
||||
(
|
||||
"m/age.csv",
|
||||
"entity,age-months
|
||||
src/a.rs,2
|
||||
",
|
||||
),
|
||||
]);
|
||||
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
||||
date.date_components = vec![DateComponent::Year, DateComponent::Month];
|
||||
let (wb, _) = p.build_model_with_warnings().unwrap();
|
||||
assert!(wb.model.category("date_Month").is_none(), "one category");
|
||||
assert_ne!(wb.views["abs-churn"].axis_of("date"), Axis::None);
|
||||
assert_eq!(wb.views["age"].axis_of("date"), Axis::None);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.12 (2): two files of one batch never share a view name.
|
||||
/// Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
@@ -2744,6 +2958,7 @@ mod tests {
|
||||
for prop in &mut p.proposals {
|
||||
if prop.field == "Date" {
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
let wb = p.build_model().unwrap();
|
||||
@@ -3256,6 +3471,7 @@ mod tests {
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||
w.toggle_date_component();
|
||||
w.toggle_date_shape();
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup");
|
||||
w.move_cursor(1);
|
||||
@@ -3379,3 +3595,225 @@ mod tests {
|
||||
assert_eq!(msg, "Nothing to import: the source has no records");
|
||||
}
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3: a date column imports as ONE hierarchical category.
|
||||
#[cfg(test)]
|
||||
mod date_hierarchy {
|
||||
use super::{DuplicatePolicy, ImportPipeline};
|
||||
use crate::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
use crate::model::cell::CellKey;
|
||||
use crate::view::GridLayout;
|
||||
use serde_json::json;
|
||||
|
||||
/// Two days of one month, a month in the next quarter, and January of the
|
||||
/// NEXT year, so a month must stay unique across years.
|
||||
fn records() -> serde_json::Value {
|
||||
json!([
|
||||
{"Date": "01/15/2024", "Amount": 1.0},
|
||||
{"Date": "01/20/2024", "Amount": 10.0},
|
||||
{"Date": "04/02/2024", "Amount": 100.0},
|
||||
{"Date": "01/03/2025", "Amount": 1000.0},
|
||||
])
|
||||
}
|
||||
|
||||
fn pipeline(components: &[DateComponent], shape: DateShape) -> ImportPipeline {
|
||||
let mut p = ImportPipeline::new(records());
|
||||
p.on_duplicate = DuplicatePolicy::Sum;
|
||||
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||
assert_eq!(date.kind, FieldKind::TimeCategory, "precondition");
|
||||
date.date_components = components.to_vec();
|
||||
date.date_shape = shape;
|
||||
p
|
||||
}
|
||||
|
||||
fn cell(month: &str) -> CellKey {
|
||||
CellKey::new(vec![
|
||||
("Date".into(), month.into()),
|
||||
("_Measure".into(), "Amount".into()),
|
||||
])
|
||||
}
|
||||
|
||||
/// The default shape: ONE category named after the column, its items the
|
||||
/// months, grouped into quarters grouped into years.
|
||||
#[test]
|
||||
fn a_date_column_imports_as_one_category_of_months_in_quarters_in_years() {
|
||||
use DateComponent::*;
|
||||
let p = pipeline(&[Month, Year, Quarter], DateShape::default());
|
||||
assert_eq!(p.proposed_category_count(), 1, "one slot, not four");
|
||||
let wb = p.build_model().unwrap();
|
||||
let names: Vec<&str> = wb.model.regular_category_names();
|
||||
assert_eq!(
|
||||
names,
|
||||
vec!["Date"],
|
||||
"no Date_Year, Date_Quarter or Date_Month"
|
||||
);
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||
assert_eq!(items, vec!["2024-01", "2024-04", "2025-01"]);
|
||||
assert_eq!(date.item_chain("2024-01"), vec!["2024", "2024-Q1"]);
|
||||
assert_eq!(date.item_chain("2025-01"), vec!["2025", "2025-Q1"]);
|
||||
assert_eq!(
|
||||
wb.model.get_cell(&cell("2024-01")).and_then(|v| v.as_f64()),
|
||||
Some(11.0),
|
||||
"two days of one month are one cell"
|
||||
);
|
||||
assert_eq!(
|
||||
wb.model.get_cell(&cell("2025-01")).and_then(|v| v.as_f64()),
|
||||
Some(1000.0),
|
||||
"January 2025 is not January 2024"
|
||||
);
|
||||
}
|
||||
|
||||
/// Chosen levels only: without Quarter, months group straight into years.
|
||||
#[test]
|
||||
fn a_level_left_unchosen_is_skipped_not_emptied() {
|
||||
use DateComponent::*;
|
||||
let wb = pipeline(&[Year, Month], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
assert_eq!(date.item_chain("2024-04"), vec!["2024"]);
|
||||
let wb = pipeline(&[Quarter], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||
assert_eq!(
|
||||
items,
|
||||
vec!["2024-Q1", "2024-Q2", "2025-Q1"],
|
||||
"flat quarters"
|
||||
);
|
||||
assert!(date.item_chain("2024-Q1").is_empty());
|
||||
}
|
||||
|
||||
/// The user's ruling on improvise-felo.6: the hierarchy opens folded, one
|
||||
/// row per year.
|
||||
#[test]
|
||||
fn an_imported_date_hierarchy_opens_folded_to_its_years() {
|
||||
use DateComponent::*;
|
||||
let wb = pipeline(&[Year, Quarter, Month], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let layout = GridLayout::with_frozen_records(&wb.model, wb.active_view(), None);
|
||||
let rows: Vec<String> = (0..layout.row_count())
|
||||
.map(|r| layout.row_label(r))
|
||||
.collect();
|
||||
assert_eq!(rows, vec!["2024", "2025"]);
|
||||
let v = layout.evaluate(&wb.model, &layout.cell_key(0, 0).unwrap());
|
||||
assert_eq!(
|
||||
v.and_then(|v| v.as_f64()),
|
||||
Some(111.0),
|
||||
"2024 folds its months"
|
||||
);
|
||||
}
|
||||
|
||||
/// The flat split stays available as a choice, and is what it was.
|
||||
#[test]
|
||||
fn the_split_shape_keeps_the_raw_dates_and_one_category_per_component() {
|
||||
use DateComponent::*;
|
||||
let p = pipeline(&[Year, Month], DateShape::Split);
|
||||
assert_eq!(p.proposed_category_count(), 3);
|
||||
let wb = p.build_model().unwrap();
|
||||
let mut names: Vec<&str> = wb.model.regular_category_names();
|
||||
names.sort_unstable();
|
||||
assert_eq!(names, vec!["Date", "Date_Month", "Date_Year"]);
|
||||
assert!(
|
||||
wb.model
|
||||
.category("Date")
|
||||
.unwrap()
|
||||
.items
|
||||
.contains_key("01/15/2024")
|
||||
);
|
||||
}
|
||||
|
||||
/// A value the detected format cannot read keeps its raw text as an
|
||||
/// ungrouped item, and the build says so rather than dropping the record.
|
||||
#[test]
|
||||
fn an_unparsed_date_keeps_its_value_and_is_warned_about() {
|
||||
use DateComponent::*;
|
||||
let mut p = pipeline(&[Year, Month], DateShape::Hierarchy);
|
||||
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||
date.date_format = Some("%Y-%m-%d".into());
|
||||
let (wb, warnings) = p.build_model_with_warnings().unwrap();
|
||||
assert!(wb.model.category("Date_Year").is_none(), "one category");
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
assert!(
|
||||
date.items.contains_key("01/15/2024"),
|
||||
"the raw value is kept"
|
||||
);
|
||||
assert!(
|
||||
date.item_chain("01/15/2024").is_empty(),
|
||||
"and it is ungrouped"
|
||||
);
|
||||
assert!(
|
||||
warnings
|
||||
.iter()
|
||||
.any(|w| w.contains("'Date' kept its raw value in 4 row(s)")),
|
||||
"{warnings:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// A %Y-%m-%d column imported Year > Month, one valid date and one raw
|
||||
/// value spelled `raw`.
|
||||
fn with_raw(raw: &str) -> ImportPipeline {
|
||||
use DateComponent::*;
|
||||
let mut p = ImportPipeline::new(json!([
|
||||
{"Date": "2024-01-15", "Amount": 1.0},
|
||||
{"Date": raw, "Amount": 2.0},
|
||||
]));
|
||||
p.on_duplicate = DuplicatePolicy::Sum;
|
||||
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||
date.kind = FieldKind::TimeCategory;
|
||||
date.date_format = Some("%Y-%m-%d".into());
|
||||
date.date_components = vec![Year, Month];
|
||||
date.date_shape = DateShape::Hierarchy;
|
||||
p
|
||||
}
|
||||
|
||||
/// A raw value spelled like a generated month would merge with it.
|
||||
#[test]
|
||||
fn a_raw_date_spelled_like_a_generated_item_is_refused() {
|
||||
let err = with_raw("2024-01").build_model_with_warnings().unwrap_err();
|
||||
let msg = err.to_string();
|
||||
assert!(msg.contains("'Date'") && msg.contains("2024-01"), "{msg}");
|
||||
assert!(msg.contains("split"), "{msg}");
|
||||
}
|
||||
|
||||
/// A raw value spelled like a generated year would sit beside its group.
|
||||
#[test]
|
||||
fn a_raw_date_spelled_like_a_generated_group_is_refused() {
|
||||
let err = with_raw("2024").build_model_with_warnings().unwrap_err();
|
||||
let msg = err.to_string();
|
||||
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
|
||||
}
|
||||
|
||||
/// The control: a raw value spelled like nothing generated still imports,
|
||||
/// so the refusal above is not a blanket one.
|
||||
#[test]
|
||||
fn a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped() {
|
||||
let (wb, warnings) = with_raw("sometime").build_model_with_warnings().unwrap();
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
assert!(date.item_chain("sometime").is_empty());
|
||||
assert!(date.items.contains_key("2024-01"), "{warnings:?}");
|
||||
assert!(
|
||||
warnings.iter().any(|w| w.contains("1 row(s)")),
|
||||
"{warnings:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The clash is answered on the date step, where `s` fixes it, not on the
|
||||
/// last step three screens later (improvise-907).
|
||||
#[test]
|
||||
fn the_date_step_will_not_advance_over_a_spelling_clash() {
|
||||
use super::{ImportWizard, WizardStep};
|
||||
let mut w = ImportWizard::from_pipeline(with_raw("2024"));
|
||||
w.step = WizardStep::ConfigureDates;
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ConfigureDates);
|
||||
let msg = w.message.clone().unwrap_or_default();
|
||||
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
|
||||
w.toggle_date_shape();
|
||||
w.advance();
|
||||
assert_ne!(w.step, WizardStep::ConfigureDates, "split clears it");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -967,6 +967,9 @@ pub struct WizardDateField {
|
||||
/// what is drawn, so the substitution is made once, here.
|
||||
pub format: String,
|
||||
pub components: Vec<(String, bool)>,
|
||||
/// What the chosen components become, as a sentence: one hierarchical
|
||||
/// category, or a split (improvise-8jk6.3). Worded once, in the projection.
|
||||
pub shape: String,
|
||||
}
|
||||
|
||||
/// Several source records sharing every coordinate (improvise-cgqt). Data,
|
||||
|
||||
+55
-2
@@ -38,6 +38,8 @@ trait Runnable {
|
||||
fn run(self, model_file: Option<PathBuf>) -> Result<()>;
|
||||
}
|
||||
|
||||
// Parsed once per process, so a variant's size costs nothing worth boxing for.
|
||||
#[allow(clippy::large_enum_variant)]
|
||||
#[derive(Subcommand)]
|
||||
#[enum_dispatch(Runnable)]
|
||||
enum Commands {
|
||||
@@ -72,10 +74,16 @@ struct ImportArgs {
|
||||
#[arg(long)]
|
||||
skip: Vec<String>,
|
||||
|
||||
/// Extract date component, e.g. "Date:Month" (repeatable)
|
||||
/// Extract date component, e.g. "Date:Month" (repeatable). The components
|
||||
/// nest in ONE category named after the column unless --split-date names it
|
||||
#[arg(long)]
|
||||
extract: Vec<String>,
|
||||
|
||||
/// Split a date field's components into one flat category each, beside its
|
||||
/// raw values, rather than one hierarchical category (repeatable)
|
||||
#[arg(long)]
|
||||
split_date: Vec<String>,
|
||||
|
||||
/// Set category axis, e.g. "Payee:row" (repeatable)
|
||||
#[arg(long)]
|
||||
axis: Vec<String>,
|
||||
@@ -157,6 +165,7 @@ impl Runnable for ImportArgs {
|
||||
time_fields: self.time,
|
||||
skip_fields: self.skip,
|
||||
extractions: parse_colon_pairs(&self.extract),
|
||||
split_dates: self.split_date,
|
||||
axes: parse_colon_pairs(&self.axis),
|
||||
formulas: self.formula,
|
||||
name: self.name,
|
||||
@@ -192,6 +201,7 @@ struct ImportConfig {
|
||||
time_fields: Vec<String>,
|
||||
skip_fields: Vec<String>,
|
||||
extractions: Vec<(String, String)>,
|
||||
split_dates: Vec<String>,
|
||||
axes: Vec<(String, String)>,
|
||||
formulas: Vec<String>,
|
||||
name: Option<String>,
|
||||
@@ -211,7 +221,7 @@ fn parse_colon_pairs(args: &[String]) -> Vec<(String, String)> {
|
||||
use import::wizard::{DuplicatePolicy, ImportPipeline};
|
||||
|
||||
fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) {
|
||||
use import::analyzer::{DateComponent, FieldKind};
|
||||
use import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
|
||||
// Override field kinds
|
||||
for p in &mut pipeline.proposals {
|
||||
@@ -244,6 +254,13 @@ fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, confi
|
||||
}
|
||||
}
|
||||
|
||||
// The wizard's `s`, headless: which date fields split rather than nest.
|
||||
for p in &mut pipeline.proposals {
|
||||
if config.split_dates.contains(&p.field) {
|
||||
p.date_shape = DateShape::Split;
|
||||
}
|
||||
}
|
||||
|
||||
// Set formulas
|
||||
pipeline.formulas = config.formulas.clone();
|
||||
|
||||
@@ -294,6 +311,23 @@ fn apply_axis_overrides(wb: &mut Workbook, axes: &[(String, String)]) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Why `--axis` is refused: the categories the import did build, and a date
|
||||
/// hint only for a name shaped like one of the split date parts.
|
||||
fn axis_refusal(cat: &str, built: &[String]) -> String {
|
||||
let date_part = ["_Year", "_Month", "_Quarter"]
|
||||
.iter()
|
||||
.any(|suffix| cat.ends_with(suffix));
|
||||
let hint = if date_part {
|
||||
" A date's parts are one category unless --split-date names it."
|
||||
} else {
|
||||
""
|
||||
};
|
||||
format!(
|
||||
"--axis names '{cat}', which this import did not build.{hint} Built: {}",
|
||||
built.join(", ")
|
||||
)
|
||||
}
|
||||
|
||||
fn run_headless_import(
|
||||
mut pipeline: ImportPipeline,
|
||||
config: &ImportConfig,
|
||||
@@ -305,6 +339,15 @@ fn run_headless_import(
|
||||
if let (true, Some(skipped)) = (config.strict, pipeline.skip_warning()) {
|
||||
anyhow::bail!("--strict: {skipped}");
|
||||
}
|
||||
if let Some(field) = config.split_dates.iter().find(|f| {
|
||||
!pipeline.proposals.iter().any(|p| {
|
||||
&p.field == *f
|
||||
&& p.kind == import::analyzer::FieldKind::TimeCategory
|
||||
&& p.date_format.is_some()
|
||||
})
|
||||
}) {
|
||||
anyhow::bail!("--split-date '{field}' is not a date column of this import");
|
||||
}
|
||||
if pipeline.on_duplicate == DuplicatePolicy::Refuse {
|
||||
let collisions = pipeline.collisions();
|
||||
if !collisions.is_empty() {
|
||||
@@ -315,6 +358,16 @@ fn run_headless_import(
|
||||
for warning in &warnings {
|
||||
eprintln!("warning: {warning}");
|
||||
}
|
||||
if let Some((cat, _)) = config
|
||||
.axes
|
||||
.iter()
|
||||
.find(|(cat, _)| wb.model.category(cat).is_none())
|
||||
{
|
||||
anyhow::bail!(axis_refusal(
|
||||
cat,
|
||||
&wb.model.categories.keys().cloned().collect::<Vec<_>>()
|
||||
));
|
||||
}
|
||||
apply_axis_overrides(&mut wb, &config.axes);
|
||||
|
||||
if let Some(path) = output.or(model_file) {
|
||||
|
||||
@@ -180,7 +180,10 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
||||
if y >= inner.bottom().saturating_sub(2) {
|
||||
break;
|
||||
}
|
||||
let header = format!(" {} (format: {})", proposal.field, proposal.format);
|
||||
let header = format!(
|
||||
" {} (format: {}) — {}",
|
||||
proposal.field, proposal.format, proposal.shape
|
||||
);
|
||||
panel.put(
|
||||
x,
|
||||
y,
|
||||
@@ -221,7 +224,7 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
||||
x,
|
||||
hint_y,
|
||||
w,
|
||||
"Space: toggle Enter: next \u{2190} back Esc: cancel",
|
||||
"Space: toggle s: one category / split Enter: next \u{2190} back Esc: cancel",
|
||||
Style::default().fg(Color::DarkGray),
|
||||
);
|
||||
}
|
||||
@@ -1178,7 +1181,7 @@ mod tests {
|
||||
"ConfigureDates\n",
|
||||
"┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n",
|
||||
"│Select date components to extract (Space toggle): │\n",
|
||||
"│ When (format: %Y-%m-%d) │\n",
|
||||
"│ When (format: %Y-%m-%d) — one category of the raw dates │\n",
|
||||
"│ [ ] Year │\n",
|
||||
"│ [ ] Month │\n",
|
||||
"│ [ ] Quarter │\n",
|
||||
@@ -1196,7 +1199,7 @@ mod tests {
|
||||
"│ │\n",
|
||||
"│ │\n",
|
||||
"│ │\n",
|
||||
"│Space: toggle Enter: next ← back Esc: cancel │\n",
|
||||
"│Space: toggle s: one category / split Enter: next ← back Esc: cancel │\n",
|
||||
"└──────────────────────────────────────────────────────────────────────────────┘\n",
|
||||
"\n",
|
||||
"====\n",
|
||||
|
||||
@@ -665,6 +665,167 @@ fn a_headless_import_keeps_duplicates_separate_when_told_to() {
|
||||
assert!(!doc.contains("= 30"), "{doc}");
|
||||
}
|
||||
|
||||
/// Dates across two years, one per month, so nothing collides.
|
||||
fn dated_csv(scratch: &Scratch) -> PathBuf {
|
||||
let csv = scratch.path("dated.csv");
|
||||
std::fs::write(
|
||||
&csv,
|
||||
"Date,Revenue\n2024-01-15,1\n2024-04-02,10\n2025-01-03,100\n",
|
||||
)
|
||||
.unwrap();
|
||||
csv
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3, headless: `--extract` builds ONE hierarchical category by
|
||||
/// default, as the wizard does, and it is written with its `>` lines.
|
||||
#[test]
|
||||
fn a_headless_date_import_is_one_hierarchical_category() {
|
||||
let scratch = Scratch::new("date_hierarchy");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&[
|
||||
"--extract",
|
||||
"Date:Year",
|
||||
"--extract",
|
||||
"Date:Quarter",
|
||||
"--extract",
|
||||
"Date:Month",
|
||||
],
|
||||
);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||
assert!(!doc.contains("Date_Month"), "{doc}");
|
||||
assert!(doc.contains("- |2024-01|[|2024-Q1|]"), "{doc}");
|
||||
assert!(doc.contains("> |2024-Q1|[|2024|]"), "{doc}");
|
||||
assert!(!doc.contains("expanded"), "it opens folded: {doc}");
|
||||
}
|
||||
|
||||
/// The flat split stays available headless, as `s` makes it in the wizard.
|
||||
#[test]
|
||||
fn a_headless_date_import_splits_when_told_to() {
|
||||
let scratch = Scratch::new("date_split");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--extract", "Date:Month", "--split-date", "Date"],
|
||||
);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||
assert!(doc.contains("## Category: Date_Month"), "{doc}");
|
||||
assert!(doc.contains("2024-01-15"), "the raw dates stay: {doc}");
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3 review: a `--split-date` that names nothing splittable
|
||||
/// said nothing and exited 0.
|
||||
#[test]
|
||||
fn split_date_naming_an_unknown_or_non_date_field_is_refused() {
|
||||
let scratch = Scratch::new("split_date_unknown");
|
||||
for field in ["Nope", "Revenue"] {
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--extract", "Date:Month", "--split-date", field],
|
||||
);
|
||||
assert!(!out.status.success(), "{field}: {}", stderr_of(&out));
|
||||
let err = stderr_of(&out);
|
||||
assert!(err.contains("--split-date") && err.contains(field), "{err}");
|
||||
}
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3 review: `--axis Date_Month:column` names a category the
|
||||
/// hierarchical import no longer builds, and was dropped without a word.
|
||||
#[test]
|
||||
fn an_axis_naming_a_category_the_import_did_not_build_is_refused() {
|
||||
let scratch = Scratch::new("axis_unknown");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--extract", "Date:Month", "--axis", "Date_Month:column"],
|
||||
);
|
||||
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||
let err = stderr_of(&out);
|
||||
assert!(
|
||||
err.contains("Date_Month") && err.contains("--split-date"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
/// `--time` on a column with no date format makes it a time category without
|
||||
/// dates, so there is nothing to split.
|
||||
#[test]
|
||||
fn split_date_on_a_time_field_with_no_date_format_is_refused() {
|
||||
let scratch = Scratch::new("split_date_no_format");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--time", "Revenue", "--split-date", "Revenue"],
|
||||
);
|
||||
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||
assert!(
|
||||
stderr_of(&out).contains("--split-date"),
|
||||
"{}",
|
||||
stderr_of(&out)
|
||||
);
|
||||
}
|
||||
|
||||
/// A name that is no date part gets the categories that exist, never the date
|
||||
/// hint (the README's old `--axis Measure:column` was this).
|
||||
#[test]
|
||||
fn an_absent_axis_category_that_is_no_date_part_names_what_was_built() {
|
||||
let scratch = Scratch::new("axis_unknown_plain");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--axis", "Measure:column"],
|
||||
);
|
||||
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||
let err = stderr_of(&out);
|
||||
assert!(
|
||||
err.contains("'Measure'") && err.contains("_Measure"),
|
||||
"{err}"
|
||||
);
|
||||
assert!(!err.contains("--split-date"), "no date hint: {err}");
|
||||
}
|
||||
|
||||
/// The README's import recipe runs as printed.
|
||||
#[test]
|
||||
fn the_readmes_import_recipe_runs() {
|
||||
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||
let readme = std::fs::read_to_string(root.join("README.md")).unwrap();
|
||||
let start = readme.find("improvise import examples/demo.csv").unwrap();
|
||||
let recipe = readme[start..]
|
||||
.split("```")
|
||||
.next()
|
||||
.unwrap()
|
||||
.replace("\\\n", " ");
|
||||
let mut args: Vec<String> = vec![];
|
||||
for (i, part) in recipe.split('"').enumerate() {
|
||||
match i % 2 {
|
||||
1 => args.push(part.to_string()),
|
||||
_ => args.extend(part.split_whitespace().map(String::from)),
|
||||
}
|
||||
}
|
||||
let at = args.iter().position(|a| a == "-o").unwrap();
|
||||
args.truncate(at);
|
||||
let scratch = Scratch::new("readme_recipe");
|
||||
let out = Command::new(bin())
|
||||
.arg("import")
|
||||
.arg(root.join("examples/demo.csv"))
|
||||
.args(args.iter().skip(3))
|
||||
.arg("-o")
|
||||
.arg(scratch.path("out.improv"))
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
}
|
||||
|
||||
/// code-maat's reports: three schemas, a stray note, and room for the output.
|
||||
fn code_maat_folder(scratch: &Scratch) -> PathBuf {
|
||||
let dir = scratch.path("maat");
|
||||
|
||||
Reference in New Issue
Block a user