feat(import): a date column imports as one hierarchical category, the flat split a choice (improvise-8jk6.3)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (agent-hier)
committed by
Edward Langley
Edward Langley
parent
e09ec75226
commit
195f3b44b8
@@ -40,7 +40,7 @@ improvise import examples/demo.csv \
|
||||
--no-wizard \
|
||||
--category Region --category Product --category Customer \
|
||||
--measure Revenue --measure Cost \
|
||||
--time Date --extract Date:Month \
|
||||
--time Date --extract Date:Month --split-date Date \
|
||||
--axis Region:row --axis Product:row \
|
||||
--axis Date_Month:column --axis Measure:column \
|
||||
--axis Customer:page --axis Date:none \
|
||||
|
||||
@@ -613,6 +613,51 @@ different row order on screen, and a different detected date format, since
|
||||
varied. The `HashSet` in `analyze_records` is there to deduplicate and **never
|
||||
to iterate**.
|
||||
|
||||
### A date column imports as ONE hierarchical category (improvise-8jk6.3)
|
||||
|
||||
The user: "I think dates should usually be one category and we can use
|
||||
formulas to crosstab." So the default `DateShape::Hierarchy` builds one
|
||||
category named after the column, where the old import built up to three flat
|
||||
ones beside it (`Date_Year`, `Date_Quarter`, `Date_Month`):
|
||||
|
||||
- **Its items are the finest chosen component**, spelled so they stay unique
|
||||
across years: `2024-01`, `2024-Q1`, `2024`, which are
|
||||
`extract_date_component`'s existing spellings. A month is never `Jan`.
|
||||
- **The coarser chosen components are its groups**, coarsest first
|
||||
(`DateComponent::COARSE_TO_FINE`): a month sits in its quarter, and the
|
||||
quarter's `>` line names its year. A level left unchosen is skipped rather
|
||||
than drawn empty, so Year + Month groups months straight into years.
|
||||
- **It opens folded**, one row per year, with nothing to set, because every
|
||||
group starts closed (`context/code/core.md` §19).
|
||||
- **It costs one of the category slots**, not up to four, and
|
||||
`proposed_category_count` counts it that way.
|
||||
- **The raw dates are gone from that category.** The day is not a component, so
|
||||
two records of one month land on one cell and the collision step asks, as it
|
||||
does for any other collision. `Split` keeps the day.
|
||||
- **A value the detected format cannot read keeps its raw text** as an
|
||||
ungrouped item, with a warning naming the column and the row count, rather
|
||||
than being dropped. The warning is separate from the split shape's, because
|
||||
that one reports an absent coordinate and this one a coordinate that is there
|
||||
and ungrouped.
|
||||
- **Item order is first-seen, not chronological.** Order is a correctness
|
||||
property here (gotcha 18), and a group need not be contiguous, so a CSV that
|
||||
is not in date order still nests correctly. It is just drawn in the order
|
||||
it was read.
|
||||
|
||||
`DateShape::Split` is the old behaviour, unchanged, and stays a choice. `s` on
|
||||
the wizard's date step toggles the field under the cursor, and the header line
|
||||
for each field says which it will get ("one category: Year > Quarter > Month"
|
||||
or "split: Date, Date_Year, …"). It is worded once, in
|
||||
`project.rs::date_shape_sentence`. Headless, `--split-date <field>` does the
|
||||
same, and `--extract` keeps its meaning: which components to build.
|
||||
|
||||
**Crosstabs by formula are not available yet.** A formula can name a month
|
||||
(`SUM(Amount WHERE Date = "2024-01")`, `Date IS "2024-01"`), but a `WHERE` or
|
||||
`IS` naming a GROUP matches no record: both compare the coordinate to the name
|
||||
exactly. `SUM(Amount WHERE Date = "2024")` is an empty sum, and `Date IS "2024"`
|
||||
is 0 on every month of 2024. That was measured on this branch and reported for
|
||||
filing, not built here.
|
||||
|
||||
### The ambiguous-date heuristic reads American-first, deliberately
|
||||
|
||||
`01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data
|
||||
|
||||
@@ -1648,3 +1648,41 @@ category named `expanded`.
|
||||
closed, the one its `collapsed:` line names and the three it does not, and the
|
||||
line is written back. Rival: the old open default; this test fails with the
|
||||
other eleven in that rival's list.
|
||||
|
||||
### a_date_column_imports_as_one_category_of_months_in_quarters_in_years
|
||||
|
||||
improvise-8jk6.3, the user's "dates should usually be one category". The
|
||||
fixture has two days of one month, so that month is one summed cell, and
|
||||
January of the next year, so a month must stay unique across years. Rivals, each
|
||||
run against the `date_hierarchy` module, and each caught here:
|
||||
- The split as the default shape.
|
||||
- Levels taken finest first, so the year hangs under the month.
|
||||
- Groups registered without their parents.
|
||||
- The raw dates still added beside the months.
|
||||
- A hierarchy counted as `1 + components` slots.
|
||||
|
||||
### a_level_left_unchosen_is_skipped_not_emptied
|
||||
|
||||
Year + Month nests months straight into years, and Quarter alone is a flat
|
||||
category of quarters. Caught the finest-first rival and the raw-dates rival.
|
||||
|
||||
### an_imported_date_hierarchy_opens_folded_to_its_years
|
||||
|
||||
The import sets nothing: the hierarchy opens folded, one row per year, because
|
||||
every group starts closed (improvise-felo.6). The 2024 row is the fold of its
|
||||
months, 111. Caught three of the rivals above.
|
||||
|
||||
### an_unparsed_date_keeps_its_value_and_is_warned_about
|
||||
|
||||
A format that reads none of the values keeps each raw value as an ungrouped
|
||||
item, and the build says so. Its first version passed on the base commit: the
|
||||
shape was ignored there and the SPLIT's warning happened to contain the same
|
||||
words. It now also asserts there is no `Date_Year`. Rival: the unparsed value
|
||||
not counted; this test fails.
|
||||
|
||||
### a_headless_date_import_splits_when_told_to
|
||||
|
||||
`--split-date` is the wizard's `s`, headless. It failed first as an unknown
|
||||
argument. Rival: the flag parsed and ignored; this test fails, and
|
||||
`a_headless_date_import_is_one_hierarchical_category` does not, which is the
|
||||
point of having both.
|
||||
|
||||
@@ -65,6 +65,7 @@ fn review_proposal_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
||||
fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
||||
match key {
|
||||
KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent),
|
||||
KeyCode::Char('s') => op(WizardOp::ToggleDateShape),
|
||||
KeyCode::Enter => op(WizardOp::Advance),
|
||||
_ => cursor_keys(key),
|
||||
}
|
||||
|
||||
@@ -1053,6 +1053,7 @@ pub enum WizardOp {
|
||||
ToggleProposal,
|
||||
CycleProposalKind,
|
||||
ToggleDateComponent,
|
||||
ToggleDateShape,
|
||||
StartFormulaEdit,
|
||||
DeleteFormula,
|
||||
ConfirmFormula,
|
||||
@@ -1078,6 +1079,7 @@ impl WizardOp {
|
||||
Self::ToggleProposal => wizard.toggle_proposal(),
|
||||
Self::CycleProposalKind => wizard.cycle_proposal_kind(),
|
||||
Self::ToggleDateComponent => wizard.toggle_date_component(),
|
||||
Self::ToggleDateShape => wizard.toggle_date_shape(),
|
||||
Self::StartFormulaEdit => wizard.start_formula_edit(),
|
||||
Self::DeleteFormula => wizard.delete_formula(),
|
||||
Self::ConfirmFormula => wizard.confirm_formula(),
|
||||
|
||||
@@ -11,7 +11,7 @@ use improvise_core::model::record::{AggregateHash, RecordHash};
|
||||
use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol};
|
||||
use improvise_core::view::{Axis, View};
|
||||
use improvise_core::workbook::Workbook;
|
||||
use improvise_io::import::analyzer::{DateComponent, FieldKind};
|
||||
use improvise_io::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep};
|
||||
use improvise_protocol::{
|
||||
AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay,
|
||||
@@ -644,6 +644,7 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
||||
((*label).to_string(), p.date_components.contains(component))
|
||||
})
|
||||
.collect(),
|
||||
shape: date_shape_sentence(p),
|
||||
})
|
||||
.collect(),
|
||||
collisions: w
|
||||
@@ -682,6 +683,27 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
||||
}
|
||||
}
|
||||
|
||||
/// What a date field's chosen components become, worded for every front end.
|
||||
fn date_shape_sentence(p: &improvise_io::import::analyzer::FieldProposal) -> String {
|
||||
let chosen = |order: &[DateComponent]| -> Vec<&str> {
|
||||
order
|
||||
.iter()
|
||||
.filter(|c| p.date_components.contains(c))
|
||||
.filter_map(|c| DATE_COMPONENTS.iter().find(|(d, _)| d == c))
|
||||
.map(|(_, label)| *label)
|
||||
.collect()
|
||||
};
|
||||
match (p.date_shape, chosen(&DateComponent::COARSE_TO_FINE)) {
|
||||
(_, none) if none.is_empty() => "one category of the raw dates".to_string(),
|
||||
(DateShape::Hierarchy, levels) => format!("one category: {}", levels.join(" > ")),
|
||||
(DateShape::Split, _) => {
|
||||
let parts = chosen(&DATE_COMPONENTS.map(|(c, _)| c));
|
||||
let names: Vec<String> = parts.iter().map(|l| format!("{}_{l}", p.field)).collect();
|
||||
format!("split: {}, {}", p.field, names.join(", "))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The date components the wizard offers, **in the order it offers them**:
|
||||
/// `ImportWizard::date_config_at_cursor` counts these same rows.
|
||||
const DATE_COMPONENTS: [(DateComponent, &str); 3] = [
|
||||
@@ -6464,3 +6486,42 @@ mod column_totals {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3: the wizard's date step says what each date field becomes,
|
||||
/// and `s` switches it between one category and the split.
|
||||
#[cfg(test)]
|
||||
mod date_shape {
|
||||
use super::wizard_cache;
|
||||
use crate::import::wizard::{ImportWizard, WizardStep};
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn the_date_step_says_what_a_date_field_becomes_and_s_switches_it() {
|
||||
let mut w = ImportWizard::new(json!([
|
||||
{"Region": "East", "When": "2024-01-15", "Revenue": 1.0},
|
||||
{"Region": "West", "When": "2024-04-02", "Revenue": 2.0},
|
||||
]));
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||
let shape = |w: &ImportWizard| wizard_cache(w).date_fields[0].shape.clone();
|
||||
assert_eq!(shape(&w), "one category of the raw dates");
|
||||
w.toggle_date_component();
|
||||
w.move_cursor(1);
|
||||
w.toggle_date_component();
|
||||
w.move_cursor(1);
|
||||
w.toggle_date_component();
|
||||
assert_eq!(shape(&w), "one category: Year > Quarter > Month");
|
||||
w.toggle_date_shape();
|
||||
assert_eq!(
|
||||
shape(&w),
|
||||
"split: When, When_Year, When_Month, When_Quarter"
|
||||
);
|
||||
let built = w.pipeline.build_model().unwrap();
|
||||
assert!(built.model.category("When_Quarter").is_some());
|
||||
w.toggle_date_shape();
|
||||
let built = w.pipeline.build_model().unwrap();
|
||||
assert!(built.model.category("When_Quarter").is_none());
|
||||
let when = built.model.category("When").unwrap();
|
||||
assert_eq!(when.item_chain("2024-04"), vec!["2024", "2024-Q2"]);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,6 +22,26 @@ pub enum DateComponent {
|
||||
Quarter,
|
||||
}
|
||||
|
||||
impl DateComponent {
|
||||
/// The components as a hierarchy nests them, coarsest first.
|
||||
pub const COARSE_TO_FINE: [DateComponent; 3] = [
|
||||
DateComponent::Year,
|
||||
DateComponent::Quarter,
|
||||
DateComponent::Month,
|
||||
];
|
||||
}
|
||||
|
||||
/// How a date column's components become categories (improvise-8jk6.3).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum DateShape {
|
||||
/// ONE category named after the column: its items are the finest chosen
|
||||
/// component, grouped by the coarser ones. The user's default.
|
||||
#[default]
|
||||
Hierarchy,
|
||||
/// The column's raw values, plus one flat category per chosen component.
|
||||
Split,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FieldProposal {
|
||||
pub field: String,
|
||||
@@ -32,6 +52,8 @@ pub struct FieldProposal {
|
||||
pub date_format: Option<String>,
|
||||
/// Which date components to extract as new categories.
|
||||
pub date_components: Vec<DateComponent>,
|
||||
/// Whether those components nest in one category or split into several.
|
||||
pub date_shape: DateShape,
|
||||
}
|
||||
|
||||
impl FieldProposal {
|
||||
@@ -132,6 +154,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -159,6 +182,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -170,6 +194,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -180,6 +205,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -191,6 +217,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
date_shape: DateShape::default(),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
|
||||
@@ -4,7 +4,7 @@ use anyhow::{Result, anyhow};
|
||||
use serde_json::Value;
|
||||
|
||||
use super::analyzer::{
|
||||
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||
DateComponent, DateShape, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||
extract_date_component, find_array_paths,
|
||||
};
|
||||
use crate::formula::parse_formula;
|
||||
@@ -116,7 +116,10 @@ impl ImportPipeline {
|
||||
.filter(|p| p.accepted)
|
||||
.map(|p| match p.kind {
|
||||
FieldKind::Category => 1,
|
||||
FieldKind::TimeCategory if p.date_format.is_some() => 1 + p.date_components.len(),
|
||||
FieldKind::TimeCategory if p.date_format.is_some() => match p.date_shape {
|
||||
DateShape::Hierarchy => 1,
|
||||
DateShape::Split => 1 + p.date_components.len(),
|
||||
},
|
||||
FieldKind::TimeCategory => 1,
|
||||
FieldKind::Measure | FieldKind::Label => 0,
|
||||
})
|
||||
@@ -177,6 +180,7 @@ impl ImportPipeline {
|
||||
&& p.kind == FieldKind::TimeCategory
|
||||
&& p.date_format.is_some()
|
||||
&& !p.date_components.is_empty()
|
||||
&& p.date_shape == DateShape::Split
|
||||
})
|
||||
.flat_map(|p| {
|
||||
let fmt = p.date_format.as_deref().unwrap();
|
||||
@@ -191,11 +195,17 @@ impl ImportPipeline {
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let categories = accepted(&[FieldKind::Category, FieldKind::TimeCategory]);
|
||||
let date_hierarchies = categories
|
||||
.iter()
|
||||
.filter_map(|p| DateHierarchy::of(p))
|
||||
.collect();
|
||||
Plan {
|
||||
categories: accepted(&[FieldKind::Category, FieldKind::TimeCategory]),
|
||||
categories,
|
||||
measures: accepted(&[FieldKind::Measure]),
|
||||
labels: accepted(&[FieldKind::Label]),
|
||||
date_extractions,
|
||||
date_hierarchies,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -222,11 +232,16 @@ impl ImportPipeline {
|
||||
|
||||
for cat_proposal in categories {
|
||||
wb.add_category(&cat_proposal.field)?;
|
||||
for val in cat_proposal
|
||||
.distinct_values
|
||||
// A hierarchical date column's items are its components, added below.
|
||||
let raw_items = match plan
|
||||
.date_hierarchies
|
||||
.iter()
|
||||
.filter(|v| !v.trim().is_empty())
|
||||
.any(|h| h.field == cat_proposal.field)
|
||||
{
|
||||
true => &[][..],
|
||||
false => &cat_proposal.distinct_values[..],
|
||||
};
|
||||
for val in raw_items.iter().filter(|v| !v.trim().is_empty()) {
|
||||
wb.add_item(&cat_proposal.field, val)?;
|
||||
}
|
||||
}
|
||||
@@ -263,7 +278,7 @@ impl ImportPipeline {
|
||||
let mut skipped_non_objects = 0usize;
|
||||
// Per-extraction tally of values the detected format could not parse. A
|
||||
// derived coordinate that is merely absent is improvise-yss5.
|
||||
let mut unparsed_dates = vec![0usize; date_extractions.len()];
|
||||
let mut unparsed_dates = vec![0usize; date_extractions.len() + plan.date_hierarchies.len()];
|
||||
// Two rows sharing every coordinate are one cell; last-row-wins lost the
|
||||
// earlier one (improvise-2cw9). The map only deduplicates — gotcha 18.
|
||||
let mut cells: Vec<PendingCell> = vec![];
|
||||
@@ -287,7 +302,11 @@ impl ImportPipeline {
|
||||
mut coords,
|
||||
measures,
|
||||
unparsed,
|
||||
chains,
|
||||
} => {
|
||||
for chain in &chains {
|
||||
add_date_chain(&mut wb, chain)?;
|
||||
}
|
||||
for (cat, item) in &coords {
|
||||
wb.add_item(cat, item)?;
|
||||
}
|
||||
@@ -345,7 +364,21 @@ impl ImportPipeline {
|
||||
&skipped_per_category,
|
||||
skipped_non_objects,
|
||||
);
|
||||
warnings.extend(unparsed_date_warnings(date_extractions, &unparsed_dates));
|
||||
let (split, whole) = unparsed_dates.split_at(date_extractions.len());
|
||||
warnings.extend(unparsed_date_warnings(date_extractions, split));
|
||||
warnings.extend(
|
||||
plan.date_hierarchies
|
||||
.iter()
|
||||
.zip(whole)
|
||||
.filter(|(_, n)| **n > 0)
|
||||
.map(|(h, n)| {
|
||||
format!(
|
||||
"'{}' kept its raw value in {n} row(s), ungrouped; it did not match \
|
||||
the date format '{}'",
|
||||
h.field, h.format
|
||||
)
|
||||
}),
|
||||
);
|
||||
warnings.extend(collision_warning(&summed));
|
||||
warnings.extend(occurrence.map(|name| separated_note(name, &collisions)));
|
||||
|
||||
@@ -359,6 +392,40 @@ struct Plan<'a> {
|
||||
measures: Vec<&'a FieldProposal>,
|
||||
labels: Vec<&'a FieldProposal>,
|
||||
date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>,
|
||||
date_hierarchies: Vec<DateHierarchy<'a>>,
|
||||
}
|
||||
|
||||
/// A date column imported as ONE category: its items are the finest chosen
|
||||
/// component, grouped by the coarser ones (improvise-8jk6.3, core.md §19).
|
||||
struct DateHierarchy<'a> {
|
||||
field: &'a str,
|
||||
format: &'a str,
|
||||
/// The chosen components, coarsest first.
|
||||
levels: Vec<DateComponent>,
|
||||
}
|
||||
|
||||
impl<'a> DateHierarchy<'a> {
|
||||
fn of(p: &'a FieldProposal) -> Option<Self> {
|
||||
let levels: Vec<DateComponent> = DateComponent::COARSE_TO_FINE
|
||||
.into_iter()
|
||||
.filter(|c| p.date_components.contains(c))
|
||||
.collect();
|
||||
let hierarchical =
|
||||
p.kind == FieldKind::TimeCategory && p.date_shape == DateShape::Hierarchy;
|
||||
(hierarchical && !levels.is_empty()).then_some(Self {
|
||||
field: &p.field,
|
||||
format: p.date_format.as_deref()?,
|
||||
levels,
|
||||
})
|
||||
}
|
||||
|
||||
/// Every level's name for one value, root first; the last is the item.
|
||||
fn chain(&self, value: &str) -> Option<Vec<String>> {
|
||||
self.levels
|
||||
.iter()
|
||||
.map(|c| extract_date_component(value, self.format, *c))
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Where one source record lands. Worked out once, for the build and for the
|
||||
@@ -370,6 +437,7 @@ enum Placement {
|
||||
coords: Vec<(String, String)>,
|
||||
measures: Vec<(String, f64)>,
|
||||
unparsed: Vec<usize>,
|
||||
chains: Vec<DateChain>,
|
||||
},
|
||||
/// No value for category `cat_idx`; `partial` is what preceded it.
|
||||
Missing {
|
||||
@@ -388,6 +456,7 @@ impl Plan<'_> {
|
||||
// Charged only once this record survives the walk: a row dropped
|
||||
// for a LATER missing category never lands (context/test-evidence).
|
||||
let mut unparsed: Vec<usize> = vec![];
|
||||
let mut chains: Vec<DateChain> = vec![];
|
||||
|
||||
for (cat_idx, cat_proposal) in self.categories.iter().enumerate() {
|
||||
let Some(v) = map.get(&cat_proposal.field).and_then(scalar_name) else {
|
||||
@@ -396,7 +465,24 @@ impl Plan<'_> {
|
||||
partial: coords,
|
||||
};
|
||||
};
|
||||
coords.push((cat_proposal.field.clone(), v.clone()));
|
||||
let hierarchy = self
|
||||
.date_hierarchies
|
||||
.iter()
|
||||
.enumerate()
|
||||
.find(|(_, h)| h.field == cat_proposal.field);
|
||||
let item = match hierarchy.map(|(at, h)| (at, h.chain(&v))) {
|
||||
Some((_, Some(chain))) => {
|
||||
let item = chain.last().cloned().unwrap_or_default();
|
||||
chains.push((cat_proposal.field.clone(), chain));
|
||||
item
|
||||
}
|
||||
Some((at, None)) => {
|
||||
unparsed.push(self.date_extractions.len() + at);
|
||||
v.clone()
|
||||
}
|
||||
None => v.clone(),
|
||||
};
|
||||
coords.push((cat_proposal.field.clone(), item));
|
||||
|
||||
// Extract date components from this field's value
|
||||
for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() {
|
||||
@@ -428,10 +514,33 @@ impl Plan<'_> {
|
||||
coords,
|
||||
measures,
|
||||
unparsed,
|
||||
chains,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One hierarchical date coordinate: its category and its chain, root first.
|
||||
type DateChain = (String, Vec<String>);
|
||||
|
||||
/// Register a date item under its groups, and each group under its parent.
|
||||
fn add_date_chain(wb: &mut Workbook, (cat, chain): &DateChain) -> Result<()> {
|
||||
let Some((item, groups)) = chain.split_last() else {
|
||||
return Ok(());
|
||||
};
|
||||
match groups.last() {
|
||||
Some(group) => wb.add_item_in_group(cat, item, group)?,
|
||||
None => wb.add_item(cat, item)?,
|
||||
};
|
||||
let category = wb
|
||||
.model
|
||||
.category_mut(cat)
|
||||
.ok_or_else(|| anyhow!("category '{cat}' vanished mid-import"))?;
|
||||
for pair in groups.windows(2) {
|
||||
category.add_group(crate::model::category::Group::new(&pair[1]).with_parent(&pair[0]));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A tuple, and each record on it: its 1-based position and the measures it writes.
|
||||
type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>);
|
||||
|
||||
@@ -1026,6 +1135,18 @@ impl ImportWizard {
|
||||
}
|
||||
}
|
||||
|
||||
/// Switch the field at the cursor between ONE hierarchical category and the
|
||||
/// flat split into one category per component (improvise-8jk6.3).
|
||||
pub fn toggle_date_shape(&mut self) {
|
||||
if let Some((pi, _)) = self.date_config_at_cursor() {
|
||||
let proposal = &mut self.pipeline.proposals[pi];
|
||||
proposal.date_shape = match proposal.date_shape {
|
||||
DateShape::Hierarchy => DateShape::Split,
|
||||
DateShape::Split => DateShape::Hierarchy,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// ── Formula editing ────────────────────────────────────────────────────────
|
||||
|
||||
/// Buffer for typing a new formula in the DefineFormulas step.
|
||||
@@ -1327,6 +1448,7 @@ mod tests {
|
||||
prop.date_format = Some("%m/%d/%Y".to_string());
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_components.push(DateComponent::Year);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
p
|
||||
@@ -1431,6 +1553,7 @@ mod tests {
|
||||
for prop in &mut p.proposals {
|
||||
if prop.field == "Date" && prop.kind == FieldKind::TimeCategory {
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
let wb = p.build_model().unwrap();
|
||||
@@ -2197,6 +2320,7 @@ mod tests {
|
||||
for prop in &mut p.proposals {
|
||||
if prop.field == "Date" {
|
||||
prop.date_components.push(DateComponent::Month);
|
||||
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||
}
|
||||
}
|
||||
let wb = p.build_model().unwrap();
|
||||
@@ -2709,6 +2833,7 @@ mod tests {
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||
w.toggle_date_component();
|
||||
w.toggle_date_shape();
|
||||
w.advance();
|
||||
assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup");
|
||||
w.move_cursor(1);
|
||||
@@ -2728,3 +2853,161 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3: a date column imports as ONE hierarchical category.
|
||||
#[cfg(test)]
|
||||
mod date_hierarchy {
|
||||
use super::{DuplicatePolicy, ImportPipeline};
|
||||
use crate::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
use crate::model::cell::CellKey;
|
||||
use crate::view::GridLayout;
|
||||
use serde_json::json;
|
||||
|
||||
/// Two days of one month, a month in the next quarter, and January of the
|
||||
/// NEXT year, so a month must stay unique across years.
|
||||
fn records() -> serde_json::Value {
|
||||
json!([
|
||||
{"Date": "01/15/2024", "Amount": 1.0},
|
||||
{"Date": "01/20/2024", "Amount": 10.0},
|
||||
{"Date": "04/02/2024", "Amount": 100.0},
|
||||
{"Date": "01/03/2025", "Amount": 1000.0},
|
||||
])
|
||||
}
|
||||
|
||||
fn pipeline(components: &[DateComponent], shape: DateShape) -> ImportPipeline {
|
||||
let mut p = ImportPipeline::new(records());
|
||||
p.on_duplicate = DuplicatePolicy::Sum;
|
||||
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||
assert_eq!(date.kind, FieldKind::TimeCategory, "precondition");
|
||||
date.date_components = components.to_vec();
|
||||
date.date_shape = shape;
|
||||
p
|
||||
}
|
||||
|
||||
fn cell(month: &str) -> CellKey {
|
||||
CellKey::new(vec![
|
||||
("Date".into(), month.into()),
|
||||
("_Measure".into(), "Amount".into()),
|
||||
])
|
||||
}
|
||||
|
||||
/// The default shape: ONE category named after the column, its items the
|
||||
/// months, grouped into quarters grouped into years.
|
||||
#[test]
|
||||
fn a_date_column_imports_as_one_category_of_months_in_quarters_in_years() {
|
||||
use DateComponent::*;
|
||||
let p = pipeline(&[Month, Year, Quarter], DateShape::default());
|
||||
assert_eq!(p.proposed_category_count(), 1, "one slot, not four");
|
||||
let wb = p.build_model().unwrap();
|
||||
let names: Vec<&str> = wb.model.regular_category_names();
|
||||
assert_eq!(
|
||||
names,
|
||||
vec!["Date"],
|
||||
"no Date_Year, Date_Quarter or Date_Month"
|
||||
);
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||
assert_eq!(items, vec!["2024-01", "2024-04", "2025-01"]);
|
||||
assert_eq!(date.item_chain("2024-01"), vec!["2024", "2024-Q1"]);
|
||||
assert_eq!(date.item_chain("2025-01"), vec!["2025", "2025-Q1"]);
|
||||
assert_eq!(
|
||||
wb.model.get_cell(&cell("2024-01")).and_then(|v| v.as_f64()),
|
||||
Some(11.0),
|
||||
"two days of one month are one cell"
|
||||
);
|
||||
assert_eq!(
|
||||
wb.model.get_cell(&cell("2025-01")).and_then(|v| v.as_f64()),
|
||||
Some(1000.0),
|
||||
"January 2025 is not January 2024"
|
||||
);
|
||||
}
|
||||
|
||||
/// Chosen levels only: without Quarter, months group straight into years.
|
||||
#[test]
|
||||
fn a_level_left_unchosen_is_skipped_not_emptied() {
|
||||
use DateComponent::*;
|
||||
let wb = pipeline(&[Year, Month], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
assert_eq!(date.item_chain("2024-04"), vec!["2024"]);
|
||||
let wb = pipeline(&[Quarter], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||
assert_eq!(
|
||||
items,
|
||||
vec!["2024-Q1", "2024-Q2", "2025-Q1"],
|
||||
"flat quarters"
|
||||
);
|
||||
assert!(date.item_chain("2024-Q1").is_empty());
|
||||
}
|
||||
|
||||
/// The user's ruling on improvise-felo.6: the hierarchy opens folded, one
|
||||
/// row per year.
|
||||
#[test]
|
||||
fn an_imported_date_hierarchy_opens_folded_to_its_years() {
|
||||
use DateComponent::*;
|
||||
let wb = pipeline(&[Year, Quarter, Month], DateShape::Hierarchy)
|
||||
.build_model()
|
||||
.unwrap();
|
||||
let layout = GridLayout::with_frozen_records(&wb.model, wb.active_view(), None);
|
||||
let rows: Vec<String> = (0..layout.row_count())
|
||||
.map(|r| layout.row_label(r))
|
||||
.collect();
|
||||
assert_eq!(rows, vec!["2024", "2025"]);
|
||||
let v = layout.evaluate(&wb.model, &layout.cell_key(0, 0).unwrap());
|
||||
assert_eq!(
|
||||
v.and_then(|v| v.as_f64()),
|
||||
Some(111.0),
|
||||
"2024 folds its months"
|
||||
);
|
||||
}
|
||||
|
||||
/// The flat split stays available as a choice, and is what it was.
|
||||
#[test]
|
||||
fn the_split_shape_keeps_the_raw_dates_and_one_category_per_component() {
|
||||
use DateComponent::*;
|
||||
let p = pipeline(&[Year, Month], DateShape::Split);
|
||||
assert_eq!(p.proposed_category_count(), 3);
|
||||
let wb = p.build_model().unwrap();
|
||||
let mut names: Vec<&str> = wb.model.regular_category_names();
|
||||
names.sort_unstable();
|
||||
assert_eq!(names, vec!["Date", "Date_Month", "Date_Year"]);
|
||||
assert!(
|
||||
wb.model
|
||||
.category("Date")
|
||||
.unwrap()
|
||||
.items
|
||||
.contains_key("01/15/2024")
|
||||
);
|
||||
}
|
||||
|
||||
/// A value the detected format cannot read keeps its raw text as an
|
||||
/// ungrouped item, and the build says so rather than dropping the record.
|
||||
#[test]
|
||||
fn an_unparsed_date_keeps_its_value_and_is_warned_about() {
|
||||
use DateComponent::*;
|
||||
let mut p = pipeline(&[Year, Month], DateShape::Hierarchy);
|
||||
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||
date.date_format = Some("%Y-%m-%d".into());
|
||||
let (wb, warnings) = p.build_model_with_warnings().unwrap();
|
||||
assert!(wb.model.category("Date_Year").is_none(), "one category");
|
||||
let date = wb.model.category("Date").unwrap();
|
||||
assert!(
|
||||
date.items.contains_key("01/15/2024"),
|
||||
"the raw value is kept"
|
||||
);
|
||||
assert!(
|
||||
date.item_chain("01/15/2024").is_empty(),
|
||||
"and it is ungrouped"
|
||||
);
|
||||
assert!(
|
||||
warnings
|
||||
.iter()
|
||||
.any(|w| w.contains("'Date' kept its raw value in 4 row(s)")),
|
||||
"{warnings:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -961,6 +961,9 @@ pub struct WizardDateField {
|
||||
/// what is drawn, so the substitution is made once, here.
|
||||
pub format: String,
|
||||
pub components: Vec<(String, bool)>,
|
||||
/// What the chosen components become, as a sentence: one hierarchical
|
||||
/// category, or a split (improvise-8jk6.3). Worded once, in the projection.
|
||||
pub shape: String,
|
||||
}
|
||||
|
||||
/// Several source records sharing every coordinate (improvise-cgqt). Data,
|
||||
|
||||
+19
-2
@@ -38,6 +38,8 @@ trait Runnable {
|
||||
fn run(self, model_file: Option<PathBuf>) -> Result<()>;
|
||||
}
|
||||
|
||||
// Parsed once per process, so a variant's size costs nothing worth boxing for.
|
||||
#[allow(clippy::large_enum_variant)]
|
||||
#[derive(Subcommand)]
|
||||
#[enum_dispatch(Runnable)]
|
||||
enum Commands {
|
||||
@@ -72,10 +74,16 @@ struct ImportArgs {
|
||||
#[arg(long)]
|
||||
skip: Vec<String>,
|
||||
|
||||
/// Extract date component, e.g. "Date:Month" (repeatable)
|
||||
/// Extract date component, e.g. "Date:Month" (repeatable). The components
|
||||
/// nest in ONE category named after the column unless --split-date names it
|
||||
#[arg(long)]
|
||||
extract: Vec<String>,
|
||||
|
||||
/// Split a date field's components into one flat category each, beside its
|
||||
/// raw values, rather than one hierarchical category (repeatable)
|
||||
#[arg(long)]
|
||||
split_date: Vec<String>,
|
||||
|
||||
/// Set category axis, e.g. "Payee:row" (repeatable)
|
||||
#[arg(long)]
|
||||
axis: Vec<String>,
|
||||
@@ -153,6 +161,7 @@ impl Runnable for ImportArgs {
|
||||
time_fields: self.time,
|
||||
skip_fields: self.skip,
|
||||
extractions: parse_colon_pairs(&self.extract),
|
||||
split_dates: self.split_date,
|
||||
axes: parse_colon_pairs(&self.axis),
|
||||
formulas: self.formula,
|
||||
name: self.name,
|
||||
@@ -187,6 +196,7 @@ struct ImportConfig {
|
||||
time_fields: Vec<String>,
|
||||
skip_fields: Vec<String>,
|
||||
extractions: Vec<(String, String)>,
|
||||
split_dates: Vec<String>,
|
||||
axes: Vec<(String, String)>,
|
||||
formulas: Vec<String>,
|
||||
name: Option<String>,
|
||||
@@ -205,7 +215,7 @@ fn parse_colon_pairs(args: &[String]) -> Vec<(String, String)> {
|
||||
use import::wizard::DuplicatePolicy;
|
||||
|
||||
fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) {
|
||||
use import::analyzer::{DateComponent, FieldKind};
|
||||
use import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||
|
||||
// Override field kinds
|
||||
for p in &mut pipeline.proposals {
|
||||
@@ -238,6 +248,13 @@ fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, confi
|
||||
}
|
||||
}
|
||||
|
||||
// The wizard's `s`, headless: which date fields split rather than nest.
|
||||
for p in &mut pipeline.proposals {
|
||||
if config.split_dates.contains(&p.field) {
|
||||
p.date_shape = DateShape::Split;
|
||||
}
|
||||
}
|
||||
|
||||
// Set formulas
|
||||
pipeline.formulas = config.formulas.clone();
|
||||
|
||||
|
||||
@@ -190,7 +190,10 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
||||
if y >= inner.bottom().saturating_sub(2) {
|
||||
break;
|
||||
}
|
||||
let header = format!(" {} (format: {})", proposal.field, proposal.format);
|
||||
let header = format!(
|
||||
" {} (format: {}) — {}",
|
||||
proposal.field, proposal.format, proposal.shape
|
||||
);
|
||||
panel.put(
|
||||
x,
|
||||
y,
|
||||
@@ -231,7 +234,7 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
||||
x,
|
||||
hint_y,
|
||||
w,
|
||||
"Space: toggle Enter: next \u{2190} back Esc: cancel",
|
||||
"Space: toggle s: one category / split Enter: next \u{2190} back Esc: cancel",
|
||||
Style::default().fg(Color::DarkGray),
|
||||
);
|
||||
}
|
||||
@@ -1188,7 +1191,7 @@ mod tests {
|
||||
"ConfigureDates\n",
|
||||
"┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n",
|
||||
"│Select date components to extract (Space toggle): │\n",
|
||||
"│ When (format: %Y-%m-%d) │\n",
|
||||
"│ When (format: %Y-%m-%d) — one category of the raw dates │\n",
|
||||
"│ [ ] Year │\n",
|
||||
"│ [ ] Month │\n",
|
||||
"│ [ ] Quarter │\n",
|
||||
@@ -1206,7 +1209,7 @@ mod tests {
|
||||
"│ │\n",
|
||||
"│ │\n",
|
||||
"│ │\n",
|
||||
"│Space: toggle Enter: next ← back Esc: cancel │\n",
|
||||
"│Space: toggle s: one category / split Enter: next ← back Esc: cancel │\n",
|
||||
"└──────────────────────────────────────────────────────────────────────────────┘\n",
|
||||
"\n",
|
||||
"====\n",
|
||||
|
||||
@@ -612,3 +612,56 @@ fn a_headless_import_keeps_duplicates_separate_when_told_to() {
|
||||
assert!(doc.contains("= 10") && doc.contains("= 20"), "{doc}");
|
||||
assert!(!doc.contains("= 30"), "{doc}");
|
||||
}
|
||||
|
||||
/// Dates across two years, one per month, so nothing collides.
|
||||
fn dated_csv(scratch: &Scratch) -> PathBuf {
|
||||
let csv = scratch.path("dated.csv");
|
||||
std::fs::write(
|
||||
&csv,
|
||||
"Date,Revenue\n2024-01-15,1\n2024-04-02,10\n2025-01-03,100\n",
|
||||
)
|
||||
.unwrap();
|
||||
csv
|
||||
}
|
||||
|
||||
/// improvise-8jk6.3, headless: `--extract` builds ONE hierarchical category by
|
||||
/// default, as the wizard does, and it is written with its `>` lines.
|
||||
#[test]
|
||||
fn a_headless_date_import_is_one_hierarchical_category() {
|
||||
let scratch = Scratch::new("date_hierarchy");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&[
|
||||
"--extract",
|
||||
"Date:Year",
|
||||
"--extract",
|
||||
"Date:Quarter",
|
||||
"--extract",
|
||||
"Date:Month",
|
||||
],
|
||||
);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||
assert!(!doc.contains("Date_Month"), "{doc}");
|
||||
assert!(doc.contains("- |2024-01|[|2024-Q1|]"), "{doc}");
|
||||
assert!(doc.contains("> |2024-Q1|[|2024|]"), "{doc}");
|
||||
assert!(!doc.contains("expanded"), "it opens folded: {doc}");
|
||||
}
|
||||
|
||||
/// The flat split stays available headless, as `s` makes it in the wizard.
|
||||
#[test]
|
||||
fn a_headless_date_import_splits_when_told_to() {
|
||||
let scratch = Scratch::new("date_split");
|
||||
let out_path = scratch.path("out.improv");
|
||||
let out = run_import(
|
||||
&dated_csv(&scratch),
|
||||
&out_path,
|
||||
&["--extract", "Date:Month", "--split-date", "Date"],
|
||||
);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||
assert!(doc.contains("## Category: Date_Month"), "{doc}");
|
||||
assert!(doc.contains("2024-01-15"), "the raw dates stay: {doc}");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user