Merge pull request #322 from fiddlerwoaroof/feat/8jk6.3-date-hierarchy
feat(import): a date column imports as one hierarchical category (improvise-8jk6.3)
This commit is contained in:
@@ -40,9 +40,9 @@ improvise import examples/demo.csv \
|
|||||||
--no-wizard \
|
--no-wizard \
|
||||||
--category Region --category Product --category Customer \
|
--category Region --category Product --category Customer \
|
||||||
--measure Revenue --measure Cost \
|
--measure Revenue --measure Cost \
|
||||||
--time Date --extract Date:Month \
|
--time Date --extract Date:Month --split-date Date \
|
||||||
--axis Region:row --axis Product:row \
|
--axis Region:row --axis Product:row \
|
||||||
--axis Date_Month:column --axis Measure:column \
|
--axis Date_Month:column --axis _Measure:column \
|
||||||
--axis Customer:page --axis Date:none \
|
--axis Customer:page --axis Date:none \
|
||||||
--formula "Profit = Revenue - Cost" \
|
--formula "Profit = Revenue - Cost" \
|
||||||
--name "Acme Sales Demo" \
|
--name "Acme Sales Demo" \
|
||||||
|
|||||||
@@ -613,6 +613,59 @@ different row order on screen, and a different detected date format, since
|
|||||||
varied. The `HashSet` in `analyze_records` is there to deduplicate and **never
|
varied. The `HashSet` in `analyze_records` is there to deduplicate and **never
|
||||||
to iterate**.
|
to iterate**.
|
||||||
|
|
||||||
|
### A date column imports as ONE hierarchical category (improvise-8jk6.3)
|
||||||
|
|
||||||
|
The user: "I think dates should usually be one category and we can use
|
||||||
|
formulas to crosstab." So the default `DateShape::Hierarchy` builds one
|
||||||
|
category named after the column, where the old import built up to three flat
|
||||||
|
ones beside it (`Date_Year`, `Date_Quarter`, `Date_Month`):
|
||||||
|
|
||||||
|
- **Its items are the finest chosen component**, spelled so they stay unique
|
||||||
|
across years: `2024-01`, `2024-Q1`, `2024`, which are
|
||||||
|
`extract_date_component`'s existing spellings. A month is never `Jan`.
|
||||||
|
- **The coarser chosen components are its groups**, coarsest first
|
||||||
|
(`DateComponent::COARSE_TO_FINE`): a month sits in its quarter, and the
|
||||||
|
quarter's `>` line names its year. A level left unchosen is skipped rather
|
||||||
|
than drawn empty, so Year + Month groups months straight into years.
|
||||||
|
- **It opens folded**, one row per year, with nothing to set, because every
|
||||||
|
group starts closed (`context/code/core.md` §19).
|
||||||
|
- **It costs one of the category slots**, not up to four, and
|
||||||
|
`proposed_category_count` counts it that way.
|
||||||
|
- **The raw dates are gone from that category.** The day is not a component, so
|
||||||
|
two records of one month land on one cell and the collision step asks, as it
|
||||||
|
does for any other collision. `Split` keeps the day.
|
||||||
|
- **A value the detected format cannot read keeps its raw text** as an
|
||||||
|
ungrouped item, with a warning naming the column and the row count, rather
|
||||||
|
than being dropped. The warning is separate from the split shape's, because
|
||||||
|
that one reports an absent coordinate and this one a coordinate that is there
|
||||||
|
and ungrouped. **Unless its spelling is one the hierarchy generates**
|
||||||
|
(improvise-d0n6): a raw `2024-01` or `2024` is the same name as the generated
|
||||||
|
month or year, so it would join the grouped item and the warning would be
|
||||||
|
false. The build **refuses**, naming the column and the spellings and pointing
|
||||||
|
at the split shape. That follows the collision rule below: where two records
|
||||||
|
would silently be one thing, the user is asked and nothing is defaulted. A
|
||||||
|
fallback to the split shape was rejected as a silent change of the user's
|
||||||
|
choice, and a renamed raw item as altering the data. The check is
|
||||||
|
`Plan::spelling_clash`.
|
||||||
|
- **Item order is first-seen, not chronological.** Order is a correctness
|
||||||
|
property here (gotcha 18), and a group need not be contiguous, so a CSV that
|
||||||
|
is not in date order still nests correctly. It is just drawn in the order
|
||||||
|
it was read.
|
||||||
|
|
||||||
|
`DateShape::Split` is the old behaviour, unchanged, and stays a choice. `s` on
|
||||||
|
the wizard's date step toggles the field under the cursor, and the header line
|
||||||
|
for each field says which it will get ("one category: Year > Quarter > Month"
|
||||||
|
or "split: Date, Date_Year, …"). It is worded once, in
|
||||||
|
`project.rs::date_shape_sentence`. Headless, `--split-date <field>` does the
|
||||||
|
same, and `--extract` keeps its meaning: which components to build.
|
||||||
|
|
||||||
|
**Crosstabs by formula are not available yet.** A formula can name a month
|
||||||
|
(`SUM(Amount WHERE Date = "2024-01")`, `Date IS "2024-01"`), but a `WHERE` or
|
||||||
|
`IS` naming a GROUP matches no record: both compare the coordinate to the name
|
||||||
|
exactly. `SUM(Amount WHERE Date = "2024")` is an empty sum, and `Date IS "2024"`
|
||||||
|
is 0 on every month of 2024. That was measured on this branch and reported for
|
||||||
|
filing, not built here.
|
||||||
|
|
||||||
### The ambiguous-date heuristic reads American-first, deliberately
|
### The ambiguous-date heuristic reads American-first, deliberately
|
||||||
|
|
||||||
`01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data
|
`01/02/2025` parses under both `%m/%d/%Y` and `%d/%m/%Y` and nothing in the data
|
||||||
|
|||||||
@@ -1685,6 +1685,44 @@ closed, the one its `collapsed:` line names and the three it does not, and the
|
|||||||
line is written back. Rival: the old open default; this test fails with the
|
line is written back. Rival: the old open default; this test fails with the
|
||||||
other eleven in that rival's list.
|
other eleven in that rival's list.
|
||||||
|
|
||||||
|
### a_date_column_imports_as_one_category_of_months_in_quarters_in_years
|
||||||
|
|
||||||
|
improvise-8jk6.3, the user's "dates should usually be one category". The
|
||||||
|
fixture has two days of one month, so that month is one summed cell, and
|
||||||
|
January of the next year, so a month must stay unique across years. Rivals, each
|
||||||
|
run against the `date_hierarchy` module, and each caught here:
|
||||||
|
- The split as the default shape.
|
||||||
|
- Levels taken finest first, so the year hangs under the month.
|
||||||
|
- Groups registered without their parents.
|
||||||
|
- The raw dates still added beside the months.
|
||||||
|
- A hierarchy counted as `1 + components` slots.
|
||||||
|
|
||||||
|
### a_level_left_unchosen_is_skipped_not_emptied
|
||||||
|
|
||||||
|
Year + Month nests months straight into years, and Quarter alone is a flat
|
||||||
|
category of quarters. Caught the finest-first rival and the raw-dates rival.
|
||||||
|
|
||||||
|
### an_imported_date_hierarchy_opens_folded_to_its_years
|
||||||
|
|
||||||
|
The import sets nothing: the hierarchy opens folded, one row per year, because
|
||||||
|
every group starts closed (improvise-felo.6). The 2024 row is the fold of its
|
||||||
|
months, 111. Caught three of the rivals above.
|
||||||
|
|
||||||
|
### an_unparsed_date_keeps_its_value_and_is_warned_about
|
||||||
|
|
||||||
|
A format that reads none of the values keeps each raw value as an ungrouped
|
||||||
|
item, and the build says so. Its first version passed on the base commit: the
|
||||||
|
shape was ignored there and the SPLIT's warning happened to contain the same
|
||||||
|
words. It now also asserts there is no `Date_Year`. Rival: the unparsed value
|
||||||
|
not counted; this test fails.
|
||||||
|
|
||||||
|
### a_headless_date_import_splits_when_told_to
|
||||||
|
|
||||||
|
`--split-date` is the wizard's `s`, headless. It failed first as an unknown
|
||||||
|
argument. Rival: the flag parsed and ignored; this test fails, and
|
||||||
|
`a_headless_date_import_is_one_hierarchical_category` does not, which is the
|
||||||
|
point of having both.
|
||||||
|
|
||||||
### every_fold_the_api_can_build_reads_back_from_the_file_it_writes
|
### every_fold_the_api_can_build_reads_back_from_the_file_it_writes
|
||||||
|
|
||||||
ChatGPT Codex, P2 on #317 at `e09ec752`: `GroupFold::Expanded(usize)` and
|
ChatGPT Codex, P2 on #317 at `e09ec752`: `GroupFold::Expanded(usize)` and
|
||||||
@@ -1939,6 +1977,45 @@ predates this PR (improvise-ych), but the sparse-record rule now rests on it:
|
|||||||
without it the value becomes an item named `" "`. Rival F4 (the reviewer's
|
without it the value becomes an item named `" "`. Rival F4 (the reviewer's
|
||||||
fault) fails it.
|
fault) fails it.
|
||||||
|
|
||||||
|
### a_raw_date_spelled_like_a_generated_item_is_refused
|
||||||
|
|
||||||
|
improvise-d0n6 (Codex P2 on PR #322). A `%Y-%m-%d` column imported Year > Month
|
||||||
|
with `2024-01-15` and a raw `2024-01`. Before the refusal the build succeeded,
|
||||||
|
put both records on the grouped item `2024-01` (summing them) and warned that
|
||||||
|
the raw value was "kept ungrouped", which was false. Fault injected: removing
|
||||||
|
the `spelling_clash` call makes this test fail with `unwrap_err()` on an `Ok`.
|
||||||
|
|
||||||
|
### a_raw_date_spelled_like_a_generated_group_is_refused
|
||||||
|
|
||||||
|
improvise-d0n6. The same with a raw `2024`, a generated group. Removing the
|
||||||
|
`spelling_clash` call fails it the same way. The control,
|
||||||
|
`a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped`, passes either
|
||||||
|
way, so a blanket refusal of every unparsed value fails it.
|
||||||
|
|
||||||
|
### the_date_step_will_not_advance_over_a_spelling_clash
|
||||||
|
|
||||||
|
improvise-d0n6 (Copilot on PR #322, and the adversarial review's B1). The
|
||||||
|
spelling-clash refusal sat only in the build, so a wizard with a raw `2024` in
|
||||||
|
a `%Y-%m-%d` column passed ConfigureDates and failed on NameModel, three steps
|
||||||
|
from the `s` that fixes it. Before the check joined `validation_error` the
|
||||||
|
test failed with the step already `DefineFormulas`; removing that call again
|
||||||
|
reproduces it. It also toggles to the split shape and requires the step to
|
||||||
|
advance, so a check that blocks forever fails.
|
||||||
|
|
||||||
|
### split_date_naming_an_unknown_or_non_date_field_is_refused
|
||||||
|
|
||||||
|
improvise-tpm8 class. `--split-date Nope` and `--split-date Revenue` (a
|
||||||
|
measure) exited 0 and did nothing. Both now exit 1 naming the flag and the
|
||||||
|
field; removing the check in `run_headless_import` fails both rows.
|
||||||
|
|
||||||
|
### an_axis_naming_a_category_the_import_did_not_build_is_refused
|
||||||
|
|
||||||
|
improvise-8jk6.3 review. `--extract Date:Month --axis Date_Month:column` used
|
||||||
|
to build one `Date` category and drop the `--axis` for the vanished
|
||||||
|
`Date_Month`, exit 0. It now exits 1 naming the category and `--split-date`.
|
||||||
|
This also refuses any `--axis` naming an absent category, which was silently
|
||||||
|
ignored before this PR as well.
|
||||||
|
|
||||||
### a_folder_of_only_header_only_files_keeps_no_record_and_refuses
|
### a_folder_of_only_header_only_files_keeps_no_record_and_refuses
|
||||||
|
|
||||||
improvise-8jk6.6 meeting PR #314's rule that a header-only file is still a
|
improvise-8jk6.6 meeting PR #314's rule that a header-only file is still a
|
||||||
@@ -1949,3 +2026,23 @@ items and empty views: the category check alone cannot refuse. The `setup`
|
|||||||
assertion proves that seeded proposal exists, so the test is not passing only
|
assertion proves that seeded proposal exists, so the test is not passing only
|
||||||
because there was no category. **R7**, the check removed from the build, fails
|
because there was no category. **R7**, the check removed from the build, fails
|
||||||
it. The user confirmed the refusal on PR #321: "Refuse (as #321 does)".
|
it. The user confirmed the refusal on PR #321: "Refuse (as #321 does)".
|
||||||
|
|
||||||
|
### an_absent_axis_category_that_is_no_date_part_names_what_was_built
|
||||||
|
|
||||||
|
improvise-8jk6.3 review (B2). The README's own recipe passed
|
||||||
|
`--axis Measure:column`, a silent no-op because the category is `_Measure`; the
|
||||||
|
first refusal blamed dates for it. The test uses a non-date absent name and
|
||||||
|
asserts the message lists the built categories and carries no date hint.
|
||||||
|
Fault injected: making the hint unconditional fails it.
|
||||||
|
|
||||||
|
### the_readmes_import_recipe_runs
|
||||||
|
|
||||||
|
improvise-8jk6.3 review (B2). Runs the README's recipe as printed. Fault
|
||||||
|
injected: restoring `--axis Measure:column` in the README fails it with the
|
||||||
|
refusal.
|
||||||
|
|
||||||
|
### split_date_on_a_time_field_with_no_date_format_is_refused
|
||||||
|
|
||||||
|
improvise-8jk6.3 review. `--time Revenue --split-date Revenue` made the field a
|
||||||
|
time category with no date format and the split was a silent no-op. Without
|
||||||
|
the `date_format.is_some()` condition the import exits 0 and this fails.
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ fn review_proposal_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
|||||||
fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
fn configure_date_keys(key: KeyCode) -> Vec<Box<dyn Effect>> {
|
||||||
match key {
|
match key {
|
||||||
KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent),
|
KeyCode::Char(' ') => op(WizardOp::ToggleDateComponent),
|
||||||
|
KeyCode::Char('s') => op(WizardOp::ToggleDateShape),
|
||||||
KeyCode::Enter => op(WizardOp::Advance),
|
KeyCode::Enter => op(WizardOp::Advance),
|
||||||
_ => cursor_keys(key),
|
_ => cursor_keys(key),
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1053,6 +1053,7 @@ pub enum WizardOp {
|
|||||||
ToggleProposal,
|
ToggleProposal,
|
||||||
CycleProposalKind,
|
CycleProposalKind,
|
||||||
ToggleDateComponent,
|
ToggleDateComponent,
|
||||||
|
ToggleDateShape,
|
||||||
StartFormulaEdit,
|
StartFormulaEdit,
|
||||||
DeleteFormula,
|
DeleteFormula,
|
||||||
ConfirmFormula,
|
ConfirmFormula,
|
||||||
@@ -1078,6 +1079,7 @@ impl WizardOp {
|
|||||||
Self::ToggleProposal => wizard.toggle_proposal(),
|
Self::ToggleProposal => wizard.toggle_proposal(),
|
||||||
Self::CycleProposalKind => wizard.cycle_proposal_kind(),
|
Self::CycleProposalKind => wizard.cycle_proposal_kind(),
|
||||||
Self::ToggleDateComponent => wizard.toggle_date_component(),
|
Self::ToggleDateComponent => wizard.toggle_date_component(),
|
||||||
|
Self::ToggleDateShape => wizard.toggle_date_shape(),
|
||||||
Self::StartFormulaEdit => wizard.start_formula_edit(),
|
Self::StartFormulaEdit => wizard.start_formula_edit(),
|
||||||
Self::DeleteFormula => wizard.delete_formula(),
|
Self::DeleteFormula => wizard.delete_formula(),
|
||||||
Self::ConfirmFormula => wizard.confirm_formula(),
|
Self::ConfirmFormula => wizard.confirm_formula(),
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ use improvise_core::model::record::{AggregateHash, RecordHash};
|
|||||||
use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol};
|
use improvise_core::view::layout::{AxisEntry, GridLayout, RecordCol};
|
||||||
use improvise_core::view::{Axis, View};
|
use improvise_core::view::{Axis, View};
|
||||||
use improvise_core::workbook::Workbook;
|
use improvise_core::workbook::Workbook;
|
||||||
use improvise_io::import::analyzer::{DateComponent, FieldKind};
|
use improvise_io::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||||
use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep};
|
use improvise_io::import::wizard::{DuplicatePolicy, ImportWizard, WizardStep};
|
||||||
use improvise_protocol::{
|
use improvise_protocol::{
|
||||||
AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay,
|
AxisFrame, Basis, BasisCache, CategoryRow, CategoryTreeCache, CategoryTreeUpdate, CellDisplay,
|
||||||
@@ -662,6 +662,7 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
|||||||
((*label).to_string(), p.date_components.contains(component))
|
((*label).to_string(), p.date_components.contains(component))
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
|
shape: date_shape_sentence(p),
|
||||||
})
|
})
|
||||||
.collect(),
|
.collect(),
|
||||||
collisions: w
|
collisions: w
|
||||||
@@ -700,6 +701,27 @@ pub fn wizard_cache(w: &ImportWizard) -> WizardCache {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// What a date field's chosen components become, worded for every front end.
|
||||||
|
fn date_shape_sentence(p: &improvise_io::import::analyzer::FieldProposal) -> String {
|
||||||
|
let chosen = |order: &[DateComponent]| -> Vec<&str> {
|
||||||
|
order
|
||||||
|
.iter()
|
||||||
|
.filter(|c| p.date_components.contains(c))
|
||||||
|
.filter_map(|c| DATE_COMPONENTS.iter().find(|(d, _)| d == c))
|
||||||
|
.map(|(_, label)| *label)
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
match (p.date_shape, chosen(&DateComponent::COARSE_TO_FINE)) {
|
||||||
|
(_, none) if none.is_empty() => "one category of the raw dates".to_string(),
|
||||||
|
(DateShape::Hierarchy, levels) => format!("one category: {}", levels.join(" > ")),
|
||||||
|
(DateShape::Split, _) => {
|
||||||
|
let parts = chosen(&DATE_COMPONENTS.map(|(c, _)| c));
|
||||||
|
let names: Vec<String> = parts.iter().map(|l| format!("{}_{l}", p.field)).collect();
|
||||||
|
format!("split: {}, {}", p.field, names.join(", "))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// The date components the wizard offers, **in the order it offers them**:
|
/// The date components the wizard offers, **in the order it offers them**:
|
||||||
/// `ImportWizard::date_config_at_cursor` counts these same rows.
|
/// `ImportWizard::date_config_at_cursor` counts these same rows.
|
||||||
const DATE_COMPONENTS: [(DateComponent, &str); 3] = [
|
const DATE_COMPONENTS: [(DateComponent, &str); 3] = [
|
||||||
@@ -6522,3 +6544,42 @@ mod column_totals {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3: the wizard's date step says what each date field becomes,
|
||||||
|
/// and `s` switches it between one category and the split.
|
||||||
|
#[cfg(test)]
|
||||||
|
mod date_shape {
|
||||||
|
use super::wizard_cache;
|
||||||
|
use crate::import::wizard::{ImportWizard, WizardStep};
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_date_step_says_what_a_date_field_becomes_and_s_switches_it() {
|
||||||
|
let mut w = ImportWizard::new(json!([
|
||||||
|
{"Region": "East", "When": "2024-01-15", "Revenue": 1.0},
|
||||||
|
{"Region": "West", "When": "2024-04-02", "Revenue": 2.0},
|
||||||
|
]));
|
||||||
|
w.advance();
|
||||||
|
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||||
|
let shape = |w: &ImportWizard| wizard_cache(w).date_fields[0].shape.clone();
|
||||||
|
assert_eq!(shape(&w), "one category of the raw dates");
|
||||||
|
w.toggle_date_component();
|
||||||
|
w.move_cursor(1);
|
||||||
|
w.toggle_date_component();
|
||||||
|
w.move_cursor(1);
|
||||||
|
w.toggle_date_component();
|
||||||
|
assert_eq!(shape(&w), "one category: Year > Quarter > Month");
|
||||||
|
w.toggle_date_shape();
|
||||||
|
assert_eq!(
|
||||||
|
shape(&w),
|
||||||
|
"split: When, When_Year, When_Month, When_Quarter"
|
||||||
|
);
|
||||||
|
let built = w.pipeline.build_model().unwrap();
|
||||||
|
assert!(built.model.category("When_Quarter").is_some());
|
||||||
|
w.toggle_date_shape();
|
||||||
|
let built = w.pipeline.build_model().unwrap();
|
||||||
|
assert!(built.model.category("When_Quarter").is_none());
|
||||||
|
let when = built.model.category("When").unwrap();
|
||||||
|
assert_eq!(when.item_chain("2024-04"), vec!["2024", "2024-Q2"]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -22,6 +22,26 @@ pub enum DateComponent {
|
|||||||
Quarter,
|
Quarter,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl DateComponent {
|
||||||
|
/// The components as a hierarchy nests them, coarsest first.
|
||||||
|
pub const COARSE_TO_FINE: [DateComponent; 3] = [
|
||||||
|
DateComponent::Year,
|
||||||
|
DateComponent::Quarter,
|
||||||
|
DateComponent::Month,
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
/// How a date column's components become categories (improvise-8jk6.3).
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||||
|
pub enum DateShape {
|
||||||
|
/// ONE category named after the column: its items are the finest chosen
|
||||||
|
/// component, grouped by the coarser ones. The user's default.
|
||||||
|
#[default]
|
||||||
|
Hierarchy,
|
||||||
|
/// The column's raw values, plus one flat category per chosen component.
|
||||||
|
Split,
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct FieldProposal {
|
pub struct FieldProposal {
|
||||||
pub field: String,
|
pub field: String,
|
||||||
@@ -32,6 +52,8 @@ pub struct FieldProposal {
|
|||||||
pub date_format: Option<String>,
|
pub date_format: Option<String>,
|
||||||
/// Which date components to extract as new categories.
|
/// Which date components to extract as new categories.
|
||||||
pub date_components: Vec<DateComponent>,
|
pub date_components: Vec<DateComponent>,
|
||||||
|
/// Whether those components nest in one category or split into several.
|
||||||
|
pub date_shape: DateShape,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl FieldProposal {
|
impl FieldProposal {
|
||||||
@@ -132,6 +154,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format: None,
|
date_format: None,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -159,6 +182,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format,
|
date_format,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -170,6 +194,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format: None,
|
date_format: None,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -180,6 +205,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format: None,
|
date_format: None,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -191,6 +217,7 @@ pub fn analyze_records(records: &[Value]) -> Vec<FieldProposal> {
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format: None,
|
date_format: None,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
.collect()
|
.collect()
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
use std::collections::HashMap;
|
use std::collections::{HashMap, HashSet};
|
||||||
|
|
||||||
use anyhow::{Result, anyhow};
|
use anyhow::{Result, anyhow};
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
|
||||||
use super::analyzer::{
|
use super::analyzer::{
|
||||||
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
DateComponent, DateShape, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||||
extract_date_component, find_array_paths,
|
extract_date_component, find_array_paths,
|
||||||
};
|
};
|
||||||
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile};
|
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile};
|
||||||
@@ -239,13 +239,23 @@ impl ImportPipeline {
|
|||||||
.filter(|p| p.accepted)
|
.filter(|p| p.accepted)
|
||||||
.map(|p| match p.kind {
|
.map(|p| match p.kind {
|
||||||
FieldKind::Category => 1,
|
FieldKind::Category => 1,
|
||||||
FieldKind::TimeCategory if p.date_format.is_some() => 1 + p.date_components.len(),
|
FieldKind::TimeCategory if p.date_format.is_some() => match p.date_shape {
|
||||||
|
DateShape::Hierarchy => 1,
|
||||||
|
DateShape::Split => 1 + p.date_components.len(),
|
||||||
|
},
|
||||||
FieldKind::TimeCategory => 1,
|
FieldKind::TimeCategory => 1,
|
||||||
FieldKind::Measure | FieldKind::Label => 0,
|
FieldKind::Measure | FieldKind::Label => 0,
|
||||||
})
|
})
|
||||||
.sum()
|
.sum()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Why the date step cannot advance: see [`Plan::spelling_clash`].
|
||||||
|
pub fn spelling_clash(&self) -> Option<String> {
|
||||||
|
let plan = self.plan();
|
||||||
|
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
|
||||||
|
plan.spelling_clash(&placements)
|
||||||
|
}
|
||||||
|
|
||||||
/// Every coordinate tuple more than one record lands on, in first-seen
|
/// Every coordinate tuple more than one record lands on, in first-seen
|
||||||
/// order. Known before the build, so the wizard can ask (improvise-cgqt).
|
/// order. Known before the build, so the wizard can ask (improvise-cgqt).
|
||||||
pub fn collisions(&self) -> Vec<Collision> {
|
pub fn collisions(&self) -> Vec<Collision> {
|
||||||
@@ -300,6 +310,7 @@ impl ImportPipeline {
|
|||||||
&& p.kind == FieldKind::TimeCategory
|
&& p.kind == FieldKind::TimeCategory
|
||||||
&& p.date_format.is_some()
|
&& p.date_format.is_some()
|
||||||
&& !p.date_components.is_empty()
|
&& !p.date_components.is_empty()
|
||||||
|
&& p.date_shape == DateShape::Split
|
||||||
})
|
})
|
||||||
.flat_map(|p| {
|
.flat_map(|p| {
|
||||||
let fmt = p.date_format.as_deref().unwrap();
|
let fmt = p.date_format.as_deref().unwrap();
|
||||||
@@ -314,11 +325,17 @@ impl ImportPipeline {
|
|||||||
})
|
})
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
let categories = accepted(&[FieldKind::Category, FieldKind::TimeCategory]);
|
||||||
|
let date_hierarchies = categories
|
||||||
|
.iter()
|
||||||
|
.filter_map(|p| DateHierarchy::of(p))
|
||||||
|
.collect();
|
||||||
Plan {
|
Plan {
|
||||||
categories: accepted(&[FieldKind::Category, FieldKind::TimeCategory]),
|
categories,
|
||||||
measures: accepted(&[FieldKind::Measure]),
|
measures: accepted(&[FieldKind::Measure]),
|
||||||
labels: accepted(&[FieldKind::Label]),
|
labels: accepted(&[FieldKind::Label]),
|
||||||
date_extractions,
|
date_extractions,
|
||||||
|
date_hierarchies,
|
||||||
scalars: self.scalars,
|
scalars: self.scalars,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -339,6 +356,9 @@ impl ImportPipeline {
|
|||||||
let formulas = self.parsed_formulas()?;
|
let formulas = self.parsed_formulas()?;
|
||||||
|
|
||||||
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
|
let placements: Vec<Placement> = self.records.iter().map(|r| plan.place(r)).collect();
|
||||||
|
if let Some(clash) = plan.spelling_clash(&placements) {
|
||||||
|
return Err(anyhow!(clash));
|
||||||
|
}
|
||||||
let collisions = collisions_among(&placements);
|
let collisions = collisions_among(&placements);
|
||||||
let occurrence = match (&self.on_duplicate, collisions.is_empty()) {
|
let occurrence = match (&self.on_duplicate, collisions.is_empty()) {
|
||||||
(_, true) | (DuplicatePolicy::Sum, false) => None,
|
(_, true) | (DuplicatePolicy::Sum, false) => None,
|
||||||
@@ -350,11 +370,16 @@ impl ImportPipeline {
|
|||||||
|
|
||||||
for cat_proposal in categories {
|
for cat_proposal in categories {
|
||||||
wb.add_category(&cat_proposal.field)?;
|
wb.add_category(&cat_proposal.field)?;
|
||||||
for val in cat_proposal
|
// A hierarchical date column's items are its components, added below.
|
||||||
.distinct_values
|
let raw_items = match plan
|
||||||
|
.date_hierarchies
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|v| !v.trim().is_empty())
|
.any(|h| h.field == cat_proposal.field)
|
||||||
{
|
{
|
||||||
|
true => &[][..],
|
||||||
|
false => &cat_proposal.distinct_values[..],
|
||||||
|
};
|
||||||
|
for val in raw_items.iter().filter(|v| !v.trim().is_empty()) {
|
||||||
wb.add_item(&cat_proposal.field, val)?;
|
wb.add_item(&cat_proposal.field, val)?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -387,7 +412,7 @@ impl ImportPipeline {
|
|||||||
|
|
||||||
// Per-extraction tally of values the detected format could not parse. A
|
// Per-extraction tally of values the detected format could not parse. A
|
||||||
// derived coordinate that is merely absent is improvise-yss5.
|
// derived coordinate that is merely absent is improvise-yss5.
|
||||||
let mut unparsed_dates = vec![0usize; date_extractions.len()];
|
let mut unparsed_dates = vec![0usize; date_extractions.len() + plan.date_hierarchies.len()];
|
||||||
// Two rows sharing every coordinate are one cell; last-row-wins lost the
|
// Two rows sharing every coordinate are one cell; last-row-wins lost the
|
||||||
// earlier one (improvise-2cw9). The map only deduplicates — gotcha 18.
|
// earlier one (improvise-2cw9). The map only deduplicates — gotcha 18.
|
||||||
let mut cells: Vec<PendingCell> = vec![];
|
let mut cells: Vec<PendingCell> = vec![];
|
||||||
@@ -405,7 +430,11 @@ impl ImportPipeline {
|
|||||||
mut coords,
|
mut coords,
|
||||||
measures,
|
measures,
|
||||||
unparsed,
|
unparsed,
|
||||||
|
chains,
|
||||||
} => {
|
} => {
|
||||||
|
for chain in &chains {
|
||||||
|
add_date_chain(&mut wb, chain)?;
|
||||||
|
}
|
||||||
for (cat, item) in &coords {
|
for (cat, item) in &coords {
|
||||||
wb.add_item(cat, item)?;
|
wb.add_item(cat, item)?;
|
||||||
}
|
}
|
||||||
@@ -457,7 +486,21 @@ impl ImportPipeline {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let mut warnings: Vec<String> = self.skip_warning().into_iter().collect();
|
let mut warnings: Vec<String> = self.skip_warning().into_iter().collect();
|
||||||
warnings.extend(unparsed_date_warnings(date_extractions, &unparsed_dates));
|
let (split, whole) = unparsed_dates.split_at(date_extractions.len());
|
||||||
|
warnings.extend(unparsed_date_warnings(date_extractions, split));
|
||||||
|
warnings.extend(
|
||||||
|
plan.date_hierarchies
|
||||||
|
.iter()
|
||||||
|
.zip(whole)
|
||||||
|
.filter(|(_, n)| **n > 0)
|
||||||
|
.map(|(h, n)| {
|
||||||
|
format!(
|
||||||
|
"'{}' kept its raw value in {n} row(s), ungrouped; it did not match \
|
||||||
|
the date format '{}'",
|
||||||
|
h.field, h.format
|
||||||
|
)
|
||||||
|
}),
|
||||||
|
);
|
||||||
warnings.extend(collision_warning(&summed));
|
warnings.extend(collision_warning(&summed));
|
||||||
warnings.extend(occurrence.map(|name| separated_note(name, &collisions)));
|
warnings.extend(occurrence.map(|name| separated_note(name, &collisions)));
|
||||||
|
|
||||||
@@ -475,6 +518,7 @@ fn unseen(field: &str, kind: FieldKind, distinct_values: Vec<String>) -> FieldPr
|
|||||||
accepted: true,
|
accepted: true,
|
||||||
date_format: None,
|
date_format: None,
|
||||||
date_components: vec![],
|
date_components: vec![],
|
||||||
|
date_shape: DateShape::default(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -552,9 +596,43 @@ struct Plan<'a> {
|
|||||||
measures: Vec<&'a FieldProposal>,
|
measures: Vec<&'a FieldProposal>,
|
||||||
labels: Vec<&'a FieldProposal>,
|
labels: Vec<&'a FieldProposal>,
|
||||||
date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>,
|
date_extractions: Vec<(&'a str, &'a str, DateComponent, String)>,
|
||||||
|
date_hierarchies: Vec<DateHierarchy<'a>>,
|
||||||
scalars: Scalars,
|
scalars: Scalars,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A date column imported as ONE category: its items are the finest chosen
|
||||||
|
/// component, grouped by the coarser ones (improvise-8jk6.3, core.md §19).
|
||||||
|
struct DateHierarchy<'a> {
|
||||||
|
field: &'a str,
|
||||||
|
format: &'a str,
|
||||||
|
/// The chosen components, coarsest first.
|
||||||
|
levels: Vec<DateComponent>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<'a> DateHierarchy<'a> {
|
||||||
|
fn of(p: &'a FieldProposal) -> Option<Self> {
|
||||||
|
let levels: Vec<DateComponent> = DateComponent::COARSE_TO_FINE
|
||||||
|
.into_iter()
|
||||||
|
.filter(|c| p.date_components.contains(c))
|
||||||
|
.collect();
|
||||||
|
let hierarchical =
|
||||||
|
p.kind == FieldKind::TimeCategory && p.date_shape == DateShape::Hierarchy;
|
||||||
|
(hierarchical && !levels.is_empty()).then_some(Self {
|
||||||
|
field: &p.field,
|
||||||
|
format: p.date_format.as_deref()?,
|
||||||
|
levels,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every level's name for one value, root first; the last is the item.
|
||||||
|
fn chain(&self, value: &str) -> Option<Vec<String>> {
|
||||||
|
self.levels
|
||||||
|
.iter()
|
||||||
|
.map(|c| extract_date_component(value, self.format, *c))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Where one source record lands. Worked out once, for the build and for the
|
/// Where one source record lands. Worked out once, for the build and for the
|
||||||
/// collision check alike, so the two cannot disagree (improvise-cgqt).
|
/// collision check alike, so the two cannot disagree (improvise-cgqt).
|
||||||
enum Placement {
|
enum Placement {
|
||||||
@@ -564,6 +642,7 @@ enum Placement {
|
|||||||
coords: Vec<(String, String)>,
|
coords: Vec<(String, String)>,
|
||||||
measures: Vec<(String, f64)>,
|
measures: Vec<(String, f64)>,
|
||||||
unparsed: Vec<usize>,
|
unparsed: Vec<usize>,
|
||||||
|
chains: Vec<DateChain>,
|
||||||
},
|
},
|
||||||
NotObject,
|
NotObject,
|
||||||
}
|
}
|
||||||
@@ -575,6 +654,7 @@ impl Plan<'_> {
|
|||||||
};
|
};
|
||||||
let mut coords: Vec<(String, String)> = vec![];
|
let mut coords: Vec<(String, String)> = vec![];
|
||||||
let mut unparsed: Vec<usize> = vec![];
|
let mut unparsed: Vec<usize> = vec![];
|
||||||
|
let mut chains: Vec<DateChain> = vec![];
|
||||||
|
|
||||||
// No value is no coordinate: records are sparse (improvise-8jk6.2).
|
// No value is no coordinate: records are sparse (improvise-8jk6.2).
|
||||||
let named = |field: &str| map.get(field).and_then(scalar_name);
|
let named = |field: &str| map.get(field).and_then(scalar_name);
|
||||||
@@ -583,7 +663,24 @@ impl Plan<'_> {
|
|||||||
.iter()
|
.iter()
|
||||||
.filter_map(|p| Some((p, named(&p.field)?)))
|
.filter_map(|p| Some((p, named(&p.field)?)))
|
||||||
{
|
{
|
||||||
coords.push((cat_proposal.field.clone(), v.clone()));
|
let hierarchy = self
|
||||||
|
.date_hierarchies
|
||||||
|
.iter()
|
||||||
|
.enumerate()
|
||||||
|
.find(|(_, h)| h.field == cat_proposal.field);
|
||||||
|
let item = match hierarchy.map(|(at, h)| (at, h.chain(&v))) {
|
||||||
|
Some((_, Some(chain))) => {
|
||||||
|
let item = chain.last().cloned().unwrap_or_default();
|
||||||
|
chains.push((cat_proposal.field.clone(), chain));
|
||||||
|
item
|
||||||
|
}
|
||||||
|
Some((at, None)) => {
|
||||||
|
unparsed.push(self.date_extractions.len() + at);
|
||||||
|
v.clone()
|
||||||
|
}
|
||||||
|
None => v.clone(),
|
||||||
|
};
|
||||||
|
coords.push((cat_proposal.field.clone(), item));
|
||||||
|
|
||||||
// Extract date components from this field's value
|
// Extract date components from this field's value
|
||||||
for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() {
|
for (at, (field, fmt, comp, derived_name)) in self.date_extractions.iter().enumerate() {
|
||||||
@@ -615,10 +712,83 @@ impl Plan<'_> {
|
|||||||
coords,
|
coords,
|
||||||
measures,
|
measures,
|
||||||
unparsed,
|
unparsed,
|
||||||
|
chains,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl Plan<'_> {
|
||||||
|
/// Why the build stops when an unparsed date is spelled like a generated
|
||||||
|
/// item or group (improvise-d0n6; formula-and-io.md, date hierarchy).
|
||||||
|
fn spelling_clash(&self, placements: &[Placement]) -> Option<String> {
|
||||||
|
let generated = |field: &str| -> HashSet<&str> {
|
||||||
|
placements
|
||||||
|
.iter()
|
||||||
|
.filter_map(|p| match p {
|
||||||
|
Placement::Placed { chains, .. } => Some(chains),
|
||||||
|
Placement::NotObject => None,
|
||||||
|
})
|
||||||
|
.flatten()
|
||||||
|
.filter(|(cat, _)| cat == field)
|
||||||
|
.flat_map(|(_, chain)| chain.iter().map(String::as_str))
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
for (at, h) in self.date_hierarchies.iter().enumerate() {
|
||||||
|
let tally = self.date_extractions.len() + at;
|
||||||
|
let names = generated(h.field);
|
||||||
|
let mut clashing: Vec<&str> = vec![];
|
||||||
|
for p in placements {
|
||||||
|
let Placement::Placed {
|
||||||
|
coords, unparsed, ..
|
||||||
|
} = p
|
||||||
|
else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let raw = coords.iter().find(|(c, _)| c == h.field).map(|(_, v)| v);
|
||||||
|
let clash = raw.filter(|v| {
|
||||||
|
unparsed.contains(&tally)
|
||||||
|
&& names.contains(v.as_str())
|
||||||
|
&& !clashing.contains(&v.as_str())
|
||||||
|
});
|
||||||
|
clashing.extend(clash.map(String::as_str));
|
||||||
|
}
|
||||||
|
if !clashing.is_empty() {
|
||||||
|
return Some(format!(
|
||||||
|
"'{}' has value(s) that do not match the date format '{}' but are spelled \
|
||||||
|
like a month, quarter or year it generates: {}; fix them, or import the \
|
||||||
|
column with the split shape",
|
||||||
|
h.field,
|
||||||
|
h.format,
|
||||||
|
clashing.join(", ")
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One hierarchical date coordinate: its category and its chain, root first.
|
||||||
|
type DateChain = (String, Vec<String>);
|
||||||
|
|
||||||
|
/// Register a date item under its groups, and each group under its parent.
|
||||||
|
fn add_date_chain(wb: &mut Workbook, (cat, chain): &DateChain) -> Result<()> {
|
||||||
|
let Some((item, groups)) = chain.split_last() else {
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
match groups.last() {
|
||||||
|
Some(group) => wb.add_item_in_group(cat, item, group)?,
|
||||||
|
None => wb.add_item(cat, item)?,
|
||||||
|
};
|
||||||
|
let category = wb
|
||||||
|
.model
|
||||||
|
.category_mut(cat)
|
||||||
|
.ok_or_else(|| anyhow!("category '{cat}' vanished mid-import"))?;
|
||||||
|
for pair in groups.windows(2) {
|
||||||
|
category.add_group(crate::model::category::Group::new(&pair[1]).with_parent(&pair[0]));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// A tuple, and each record on it: its 1-based position and the measures it writes.
|
/// A tuple, and each record on it: its 1-based position and the measures it writes.
|
||||||
type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>);
|
type OnTuple<'a> = (&'a Vec<(String, String)>, Vec<(usize, Vec<&'a str>)>);
|
||||||
|
|
||||||
@@ -921,7 +1091,9 @@ impl ImportWizard {
|
|||||||
.nothing_to_import()
|
.nothing_to_import()
|
||||||
.or_else(|| self.no_category_message())
|
.or_else(|| self.no_category_message())
|
||||||
.or_else(|| self.category_limit_message()),
|
.or_else(|| self.category_limit_message()),
|
||||||
WizardStep::ConfigureDates => self.category_limit_message(),
|
WizardStep::ConfigureDates => self
|
||||||
|
.category_limit_message()
|
||||||
|
.or_else(|| self.pipeline.spelling_clash()),
|
||||||
WizardStep::ResolveDuplicates => match &self.pipeline.on_duplicate {
|
WizardStep::ResolveDuplicates => match &self.pipeline.on_duplicate {
|
||||||
DuplicatePolicy::Sum => None,
|
DuplicatePolicy::Sum => None,
|
||||||
DuplicatePolicy::Separate(_) => self.separate_blocked(),
|
DuplicatePolicy::Separate(_) => self.separate_blocked(),
|
||||||
@@ -1199,6 +1371,18 @@ impl ImportWizard {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Switch the field at the cursor between ONE hierarchical category and the
|
||||||
|
/// flat split into one category per component (improvise-8jk6.3).
|
||||||
|
pub fn toggle_date_shape(&mut self) {
|
||||||
|
if let Some((pi, _)) = self.date_config_at_cursor() {
|
||||||
|
let proposal = &mut self.pipeline.proposals[pi];
|
||||||
|
proposal.date_shape = match proposal.date_shape {
|
||||||
|
DateShape::Hierarchy => DateShape::Split,
|
||||||
|
DateShape::Split => DateShape::Hierarchy,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ── Formula editing ────────────────────────────────────────────────────────
|
// ── Formula editing ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
/// Buffer for typing a new formula in the DefineFormulas step.
|
/// Buffer for typing a new formula in the DefineFormulas step.
|
||||||
@@ -1511,6 +1695,7 @@ mod tests {
|
|||||||
prop.date_format = Some("%m/%d/%Y".to_string());
|
prop.date_format = Some("%m/%d/%Y".to_string());
|
||||||
prop.date_components.push(DateComponent::Month);
|
prop.date_components.push(DateComponent::Month);
|
||||||
prop.date_components.push(DateComponent::Year);
|
prop.date_components.push(DateComponent::Year);
|
||||||
|
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
p
|
p
|
||||||
@@ -1614,6 +1799,7 @@ mod tests {
|
|||||||
for prop in &mut p.proposals {
|
for prop in &mut p.proposals {
|
||||||
if prop.field == "Date" && prop.kind == FieldKind::TimeCategory {
|
if prop.field == "Date" && prop.kind == FieldKind::TimeCategory {
|
||||||
prop.date_components.push(DateComponent::Month);
|
prop.date_components.push(DateComponent::Month);
|
||||||
|
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let wb = p.build_model().unwrap();
|
let wb = p.build_model().unwrap();
|
||||||
@@ -2484,11 +2670,39 @@ mod tests {
|
|||||||
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
||||||
assert_eq!(date.kind, FieldKind::TimeCategory);
|
assert_eq!(date.kind, FieldKind::TimeCategory);
|
||||||
date.date_components = vec![DateComponent::Month];
|
date.date_components = vec![DateComponent::Month];
|
||||||
|
date.date_shape = crate::import::analyzer::DateShape::Split;
|
||||||
let (wb, _) = p.build_model_with_warnings().unwrap();
|
let (wb, _) = p.build_model_with_warnings().unwrap();
|
||||||
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
|
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
|
||||||
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
|
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3 under improvise-8jk6.12: the default shape puts the date
|
||||||
|
/// parts IN the file's own column, so that column is what its view seats.
|
||||||
|
#[test]
|
||||||
|
fn a_files_view_seats_its_own_hierarchical_date_column() {
|
||||||
|
use crate::import::analyzer::DateComponent;
|
||||||
|
let mut p = merged_pipeline(&[
|
||||||
|
(
|
||||||
|
"m/abs-churn.csv",
|
||||||
|
"date,added
|
||||||
|
2026-01-02,10
|
||||||
|
",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"m/age.csv",
|
||||||
|
"entity,age-months
|
||||||
|
src/a.rs,2
|
||||||
|
",
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
||||||
|
date.date_components = vec![DateComponent::Year, DateComponent::Month];
|
||||||
|
let (wb, _) = p.build_model_with_warnings().unwrap();
|
||||||
|
assert!(wb.model.category("date_Month").is_none(), "one category");
|
||||||
|
assert_ne!(wb.views["abs-churn"].axis_of("date"), Axis::None);
|
||||||
|
assert_eq!(wb.views["age"].axis_of("date"), Axis::None);
|
||||||
|
}
|
||||||
|
|
||||||
/// improvise-8jk6.12 (2): two files of one batch never share a view name.
|
/// improvise-8jk6.12 (2): two files of one batch never share a view name.
|
||||||
/// Evidence: `context/test-evidence/formula-and-io.md`.
|
/// Evidence: `context/test-evidence/formula-and-io.md`.
|
||||||
#[test]
|
#[test]
|
||||||
@@ -2744,6 +2958,7 @@ mod tests {
|
|||||||
for prop in &mut p.proposals {
|
for prop in &mut p.proposals {
|
||||||
if prop.field == "Date" {
|
if prop.field == "Date" {
|
||||||
prop.date_components.push(DateComponent::Month);
|
prop.date_components.push(DateComponent::Month);
|
||||||
|
prop.date_shape = crate::import::analyzer::DateShape::Split;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let wb = p.build_model().unwrap();
|
let wb = p.build_model().unwrap();
|
||||||
@@ -3256,6 +3471,7 @@ mod tests {
|
|||||||
w.advance();
|
w.advance();
|
||||||
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
assert_eq!(w.step, WizardStep::ConfigureDates, "setup");
|
||||||
w.toggle_date_component();
|
w.toggle_date_component();
|
||||||
|
w.toggle_date_shape();
|
||||||
w.advance();
|
w.advance();
|
||||||
assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup");
|
assert_eq!(w.step, WizardStep::ResolveDuplicates, "setup");
|
||||||
w.move_cursor(1);
|
w.move_cursor(1);
|
||||||
@@ -3379,3 +3595,225 @@ mod tests {
|
|||||||
assert_eq!(msg, "Nothing to import: the source has no records");
|
assert_eq!(msg, "Nothing to import: the source has no records");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3: a date column imports as ONE hierarchical category.
|
||||||
|
#[cfg(test)]
|
||||||
|
mod date_hierarchy {
|
||||||
|
use super::{DuplicatePolicy, ImportPipeline};
|
||||||
|
use crate::import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||||
|
use crate::model::cell::CellKey;
|
||||||
|
use crate::view::GridLayout;
|
||||||
|
use serde_json::json;
|
||||||
|
|
||||||
|
/// Two days of one month, a month in the next quarter, and January of the
|
||||||
|
/// NEXT year, so a month must stay unique across years.
|
||||||
|
fn records() -> serde_json::Value {
|
||||||
|
json!([
|
||||||
|
{"Date": "01/15/2024", "Amount": 1.0},
|
||||||
|
{"Date": "01/20/2024", "Amount": 10.0},
|
||||||
|
{"Date": "04/02/2024", "Amount": 100.0},
|
||||||
|
{"Date": "01/03/2025", "Amount": 1000.0},
|
||||||
|
])
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pipeline(components: &[DateComponent], shape: DateShape) -> ImportPipeline {
|
||||||
|
let mut p = ImportPipeline::new(records());
|
||||||
|
p.on_duplicate = DuplicatePolicy::Sum;
|
||||||
|
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||||
|
assert_eq!(date.kind, FieldKind::TimeCategory, "precondition");
|
||||||
|
date.date_components = components.to_vec();
|
||||||
|
date.date_shape = shape;
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cell(month: &str) -> CellKey {
|
||||||
|
CellKey::new(vec![
|
||||||
|
("Date".into(), month.into()),
|
||||||
|
("_Measure".into(), "Amount".into()),
|
||||||
|
])
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The default shape: ONE category named after the column, its items the
|
||||||
|
/// months, grouped into quarters grouped into years.
|
||||||
|
#[test]
|
||||||
|
fn a_date_column_imports_as_one_category_of_months_in_quarters_in_years() {
|
||||||
|
use DateComponent::*;
|
||||||
|
let p = pipeline(&[Month, Year, Quarter], DateShape::default());
|
||||||
|
assert_eq!(p.proposed_category_count(), 1, "one slot, not four");
|
||||||
|
let wb = p.build_model().unwrap();
|
||||||
|
let names: Vec<&str> = wb.model.regular_category_names();
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
vec!["Date"],
|
||||||
|
"no Date_Year, Date_Quarter or Date_Month"
|
||||||
|
);
|
||||||
|
let date = wb.model.category("Date").unwrap();
|
||||||
|
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||||
|
assert_eq!(items, vec!["2024-01", "2024-04", "2025-01"]);
|
||||||
|
assert_eq!(date.item_chain("2024-01"), vec!["2024", "2024-Q1"]);
|
||||||
|
assert_eq!(date.item_chain("2025-01"), vec!["2025", "2025-Q1"]);
|
||||||
|
assert_eq!(
|
||||||
|
wb.model.get_cell(&cell("2024-01")).and_then(|v| v.as_f64()),
|
||||||
|
Some(11.0),
|
||||||
|
"two days of one month are one cell"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
wb.model.get_cell(&cell("2025-01")).and_then(|v| v.as_f64()),
|
||||||
|
Some(1000.0),
|
||||||
|
"January 2025 is not January 2024"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Chosen levels only: without Quarter, months group straight into years.
|
||||||
|
#[test]
|
||||||
|
fn a_level_left_unchosen_is_skipped_not_emptied() {
|
||||||
|
use DateComponent::*;
|
||||||
|
let wb = pipeline(&[Year, Month], DateShape::Hierarchy)
|
||||||
|
.build_model()
|
||||||
|
.unwrap();
|
||||||
|
let date = wb.model.category("Date").unwrap();
|
||||||
|
assert_eq!(date.item_chain("2024-04"), vec!["2024"]);
|
||||||
|
let wb = pipeline(&[Quarter], DateShape::Hierarchy)
|
||||||
|
.build_model()
|
||||||
|
.unwrap();
|
||||||
|
let date = wb.model.category("Date").unwrap();
|
||||||
|
let items: Vec<&str> = date.items.keys().map(String::as_str).collect();
|
||||||
|
assert_eq!(
|
||||||
|
items,
|
||||||
|
vec!["2024-Q1", "2024-Q2", "2025-Q1"],
|
||||||
|
"flat quarters"
|
||||||
|
);
|
||||||
|
assert!(date.item_chain("2024-Q1").is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The user's ruling on improvise-felo.6: the hierarchy opens folded, one
|
||||||
|
/// row per year.
|
||||||
|
#[test]
|
||||||
|
fn an_imported_date_hierarchy_opens_folded_to_its_years() {
|
||||||
|
use DateComponent::*;
|
||||||
|
let wb = pipeline(&[Year, Quarter, Month], DateShape::Hierarchy)
|
||||||
|
.build_model()
|
||||||
|
.unwrap();
|
||||||
|
let layout = GridLayout::with_frozen_records(&wb.model, wb.active_view(), None);
|
||||||
|
let rows: Vec<String> = (0..layout.row_count())
|
||||||
|
.map(|r| layout.row_label(r))
|
||||||
|
.collect();
|
||||||
|
assert_eq!(rows, vec!["2024", "2025"]);
|
||||||
|
let v = layout.evaluate(&wb.model, &layout.cell_key(0, 0).unwrap());
|
||||||
|
assert_eq!(
|
||||||
|
v.and_then(|v| v.as_f64()),
|
||||||
|
Some(111.0),
|
||||||
|
"2024 folds its months"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The flat split stays available as a choice, and is what it was.
|
||||||
|
#[test]
|
||||||
|
fn the_split_shape_keeps_the_raw_dates_and_one_category_per_component() {
|
||||||
|
use DateComponent::*;
|
||||||
|
let p = pipeline(&[Year, Month], DateShape::Split);
|
||||||
|
assert_eq!(p.proposed_category_count(), 3);
|
||||||
|
let wb = p.build_model().unwrap();
|
||||||
|
let mut names: Vec<&str> = wb.model.regular_category_names();
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(names, vec!["Date", "Date_Month", "Date_Year"]);
|
||||||
|
assert!(
|
||||||
|
wb.model
|
||||||
|
.category("Date")
|
||||||
|
.unwrap()
|
||||||
|
.items
|
||||||
|
.contains_key("01/15/2024")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A value the detected format cannot read keeps its raw text as an
|
||||||
|
/// ungrouped item, and the build says so rather than dropping the record.
|
||||||
|
#[test]
|
||||||
|
fn an_unparsed_date_keeps_its_value_and_is_warned_about() {
|
||||||
|
use DateComponent::*;
|
||||||
|
let mut p = pipeline(&[Year, Month], DateShape::Hierarchy);
|
||||||
|
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||||
|
date.date_format = Some("%Y-%m-%d".into());
|
||||||
|
let (wb, warnings) = p.build_model_with_warnings().unwrap();
|
||||||
|
assert!(wb.model.category("Date_Year").is_none(), "one category");
|
||||||
|
let date = wb.model.category("Date").unwrap();
|
||||||
|
assert!(
|
||||||
|
date.items.contains_key("01/15/2024"),
|
||||||
|
"the raw value is kept"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
date.item_chain("01/15/2024").is_empty(),
|
||||||
|
"and it is ungrouped"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
warnings
|
||||||
|
.iter()
|
||||||
|
.any(|w| w.contains("'Date' kept its raw value in 4 row(s)")),
|
||||||
|
"{warnings:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A %Y-%m-%d column imported Year > Month, one valid date and one raw
|
||||||
|
/// value spelled `raw`.
|
||||||
|
fn with_raw(raw: &str) -> ImportPipeline {
|
||||||
|
use DateComponent::*;
|
||||||
|
let mut p = ImportPipeline::new(json!([
|
||||||
|
{"Date": "2024-01-15", "Amount": 1.0},
|
||||||
|
{"Date": raw, "Amount": 2.0},
|
||||||
|
]));
|
||||||
|
p.on_duplicate = DuplicatePolicy::Sum;
|
||||||
|
let date = p.proposals.iter_mut().find(|p| p.field == "Date").unwrap();
|
||||||
|
date.kind = FieldKind::TimeCategory;
|
||||||
|
date.date_format = Some("%Y-%m-%d".into());
|
||||||
|
date.date_components = vec![Year, Month];
|
||||||
|
date.date_shape = DateShape::Hierarchy;
|
||||||
|
p
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A raw value spelled like a generated month would merge with it.
|
||||||
|
#[test]
|
||||||
|
fn a_raw_date_spelled_like_a_generated_item_is_refused() {
|
||||||
|
let err = with_raw("2024-01").build_model_with_warnings().unwrap_err();
|
||||||
|
let msg = err.to_string();
|
||||||
|
assert!(msg.contains("'Date'") && msg.contains("2024-01"), "{msg}");
|
||||||
|
assert!(msg.contains("split"), "{msg}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A raw value spelled like a generated year would sit beside its group.
|
||||||
|
#[test]
|
||||||
|
fn a_raw_date_spelled_like_a_generated_group_is_refused() {
|
||||||
|
let err = with_raw("2024").build_model_with_warnings().unwrap_err();
|
||||||
|
let msg = err.to_string();
|
||||||
|
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The control: a raw value spelled like nothing generated still imports,
|
||||||
|
/// so the refusal above is not a blanket one.
|
||||||
|
#[test]
|
||||||
|
fn a_raw_date_spelled_like_nothing_generated_is_kept_ungrouped() {
|
||||||
|
let (wb, warnings) = with_raw("sometime").build_model_with_warnings().unwrap();
|
||||||
|
let date = wb.model.category("Date").unwrap();
|
||||||
|
assert!(date.item_chain("sometime").is_empty());
|
||||||
|
assert!(date.items.contains_key("2024-01"), "{warnings:?}");
|
||||||
|
assert!(
|
||||||
|
warnings.iter().any(|w| w.contains("1 row(s)")),
|
||||||
|
"{warnings:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The clash is answered on the date step, where `s` fixes it, not on the
|
||||||
|
/// last step three screens later (improvise-907).
|
||||||
|
#[test]
|
||||||
|
fn the_date_step_will_not_advance_over_a_spelling_clash() {
|
||||||
|
use super::{ImportWizard, WizardStep};
|
||||||
|
let mut w = ImportWizard::from_pipeline(with_raw("2024"));
|
||||||
|
w.step = WizardStep::ConfigureDates;
|
||||||
|
w.advance();
|
||||||
|
assert_eq!(w.step, WizardStep::ConfigureDates);
|
||||||
|
let msg = w.message.clone().unwrap_or_default();
|
||||||
|
assert!(msg.contains("'Date'") && msg.contains("2024"), "{msg}");
|
||||||
|
w.toggle_date_shape();
|
||||||
|
w.advance();
|
||||||
|
assert_ne!(w.step, WizardStep::ConfigureDates, "split clears it");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -967,6 +967,9 @@ pub struct WizardDateField {
|
|||||||
/// what is drawn, so the substitution is made once, here.
|
/// what is drawn, so the substitution is made once, here.
|
||||||
pub format: String,
|
pub format: String,
|
||||||
pub components: Vec<(String, bool)>,
|
pub components: Vec<(String, bool)>,
|
||||||
|
/// What the chosen components become, as a sentence: one hierarchical
|
||||||
|
/// category, or a split (improvise-8jk6.3). Worded once, in the projection.
|
||||||
|
pub shape: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Several source records sharing every coordinate (improvise-cgqt). Data,
|
/// Several source records sharing every coordinate (improvise-cgqt). Data,
|
||||||
|
|||||||
+55
-2
@@ -38,6 +38,8 @@ trait Runnable {
|
|||||||
fn run(self, model_file: Option<PathBuf>) -> Result<()>;
|
fn run(self, model_file: Option<PathBuf>) -> Result<()>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Parsed once per process, so a variant's size costs nothing worth boxing for.
|
||||||
|
#[allow(clippy::large_enum_variant)]
|
||||||
#[derive(Subcommand)]
|
#[derive(Subcommand)]
|
||||||
#[enum_dispatch(Runnable)]
|
#[enum_dispatch(Runnable)]
|
||||||
enum Commands {
|
enum Commands {
|
||||||
@@ -72,10 +74,16 @@ struct ImportArgs {
|
|||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
skip: Vec<String>,
|
skip: Vec<String>,
|
||||||
|
|
||||||
/// Extract date component, e.g. "Date:Month" (repeatable)
|
/// Extract date component, e.g. "Date:Month" (repeatable). The components
|
||||||
|
/// nest in ONE category named after the column unless --split-date names it
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
extract: Vec<String>,
|
extract: Vec<String>,
|
||||||
|
|
||||||
|
/// Split a date field's components into one flat category each, beside its
|
||||||
|
/// raw values, rather than one hierarchical category (repeatable)
|
||||||
|
#[arg(long)]
|
||||||
|
split_date: Vec<String>,
|
||||||
|
|
||||||
/// Set category axis, e.g. "Payee:row" (repeatable)
|
/// Set category axis, e.g. "Payee:row" (repeatable)
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
axis: Vec<String>,
|
axis: Vec<String>,
|
||||||
@@ -157,6 +165,7 @@ impl Runnable for ImportArgs {
|
|||||||
time_fields: self.time,
|
time_fields: self.time,
|
||||||
skip_fields: self.skip,
|
skip_fields: self.skip,
|
||||||
extractions: parse_colon_pairs(&self.extract),
|
extractions: parse_colon_pairs(&self.extract),
|
||||||
|
split_dates: self.split_date,
|
||||||
axes: parse_colon_pairs(&self.axis),
|
axes: parse_colon_pairs(&self.axis),
|
||||||
formulas: self.formula,
|
formulas: self.formula,
|
||||||
name: self.name,
|
name: self.name,
|
||||||
@@ -192,6 +201,7 @@ struct ImportConfig {
|
|||||||
time_fields: Vec<String>,
|
time_fields: Vec<String>,
|
||||||
skip_fields: Vec<String>,
|
skip_fields: Vec<String>,
|
||||||
extractions: Vec<(String, String)>,
|
extractions: Vec<(String, String)>,
|
||||||
|
split_dates: Vec<String>,
|
||||||
axes: Vec<(String, String)>,
|
axes: Vec<(String, String)>,
|
||||||
formulas: Vec<String>,
|
formulas: Vec<String>,
|
||||||
name: Option<String>,
|
name: Option<String>,
|
||||||
@@ -211,7 +221,7 @@ fn parse_colon_pairs(args: &[String]) -> Vec<(String, String)> {
|
|||||||
use import::wizard::{DuplicatePolicy, ImportPipeline};
|
use import::wizard::{DuplicatePolicy, ImportPipeline};
|
||||||
|
|
||||||
fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) {
|
fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, config: &ImportConfig) {
|
||||||
use import::analyzer::{DateComponent, FieldKind};
|
use import::analyzer::{DateComponent, DateShape, FieldKind};
|
||||||
|
|
||||||
// Override field kinds
|
// Override field kinds
|
||||||
for p in &mut pipeline.proposals {
|
for p in &mut pipeline.proposals {
|
||||||
@@ -244,6 +254,13 @@ fn apply_config_to_pipeline(pipeline: &mut import::wizard::ImportPipeline, confi
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The wizard's `s`, headless: which date fields split rather than nest.
|
||||||
|
for p in &mut pipeline.proposals {
|
||||||
|
if config.split_dates.contains(&p.field) {
|
||||||
|
p.date_shape = DateShape::Split;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Set formulas
|
// Set formulas
|
||||||
pipeline.formulas = config.formulas.clone();
|
pipeline.formulas = config.formulas.clone();
|
||||||
|
|
||||||
@@ -294,6 +311,23 @@ fn apply_axis_overrides(wb: &mut Workbook, axes: &[(String, String)]) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Why `--axis` is refused: the categories the import did build, and a date
|
||||||
|
/// hint only for a name shaped like one of the split date parts.
|
||||||
|
fn axis_refusal(cat: &str, built: &[String]) -> String {
|
||||||
|
let date_part = ["_Year", "_Month", "_Quarter"]
|
||||||
|
.iter()
|
||||||
|
.any(|suffix| cat.ends_with(suffix));
|
||||||
|
let hint = if date_part {
|
||||||
|
" A date's parts are one category unless --split-date names it."
|
||||||
|
} else {
|
||||||
|
""
|
||||||
|
};
|
||||||
|
format!(
|
||||||
|
"--axis names '{cat}', which this import did not build.{hint} Built: {}",
|
||||||
|
built.join(", ")
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
fn run_headless_import(
|
fn run_headless_import(
|
||||||
mut pipeline: ImportPipeline,
|
mut pipeline: ImportPipeline,
|
||||||
config: &ImportConfig,
|
config: &ImportConfig,
|
||||||
@@ -305,6 +339,15 @@ fn run_headless_import(
|
|||||||
if let (true, Some(skipped)) = (config.strict, pipeline.skip_warning()) {
|
if let (true, Some(skipped)) = (config.strict, pipeline.skip_warning()) {
|
||||||
anyhow::bail!("--strict: {skipped}");
|
anyhow::bail!("--strict: {skipped}");
|
||||||
}
|
}
|
||||||
|
if let Some(field) = config.split_dates.iter().find(|f| {
|
||||||
|
!pipeline.proposals.iter().any(|p| {
|
||||||
|
&p.field == *f
|
||||||
|
&& p.kind == import::analyzer::FieldKind::TimeCategory
|
||||||
|
&& p.date_format.is_some()
|
||||||
|
})
|
||||||
|
}) {
|
||||||
|
anyhow::bail!("--split-date '{field}' is not a date column of this import");
|
||||||
|
}
|
||||||
if pipeline.on_duplicate == DuplicatePolicy::Refuse {
|
if pipeline.on_duplicate == DuplicatePolicy::Refuse {
|
||||||
let collisions = pipeline.collisions();
|
let collisions = pipeline.collisions();
|
||||||
if !collisions.is_empty() {
|
if !collisions.is_empty() {
|
||||||
@@ -315,6 +358,16 @@ fn run_headless_import(
|
|||||||
for warning in &warnings {
|
for warning in &warnings {
|
||||||
eprintln!("warning: {warning}");
|
eprintln!("warning: {warning}");
|
||||||
}
|
}
|
||||||
|
if let Some((cat, _)) = config
|
||||||
|
.axes
|
||||||
|
.iter()
|
||||||
|
.find(|(cat, _)| wb.model.category(cat).is_none())
|
||||||
|
{
|
||||||
|
anyhow::bail!(axis_refusal(
|
||||||
|
cat,
|
||||||
|
&wb.model.categories.keys().cloned().collect::<Vec<_>>()
|
||||||
|
));
|
||||||
|
}
|
||||||
apply_axis_overrides(&mut wb, &config.axes);
|
apply_axis_overrides(&mut wb, &config.axes);
|
||||||
|
|
||||||
if let Some(path) = output.or(model_file) {
|
if let Some(path) = output.or(model_file) {
|
||||||
|
|||||||
@@ -180,7 +180,10 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
|||||||
if y >= inner.bottom().saturating_sub(2) {
|
if y >= inner.bottom().saturating_sub(2) {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
let header = format!(" {} (format: {})", proposal.field, proposal.format);
|
let header = format!(
|
||||||
|
" {} (format: {}) — {}",
|
||||||
|
proposal.field, proposal.format, proposal.shape
|
||||||
|
);
|
||||||
panel.put(
|
panel.put(
|
||||||
x,
|
x,
|
||||||
y,
|
y,
|
||||||
@@ -221,7 +224,7 @@ impl<'a> Widget for ImportWizardWidget<'a> {
|
|||||||
x,
|
x,
|
||||||
hint_y,
|
hint_y,
|
||||||
w,
|
w,
|
||||||
"Space: toggle Enter: next \u{2190} back Esc: cancel",
|
"Space: toggle s: one category / split Enter: next \u{2190} back Esc: cancel",
|
||||||
Style::default().fg(Color::DarkGray),
|
Style::default().fg(Color::DarkGray),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -1178,7 +1181,7 @@ mod tests {
|
|||||||
"ConfigureDates\n",
|
"ConfigureDates\n",
|
||||||
"┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n",
|
"┌ Import Wizard — Date Components ─────────────────────────────────────────────┐\n",
|
||||||
"│Select date components to extract (Space toggle): │\n",
|
"│Select date components to extract (Space toggle): │\n",
|
||||||
"│ When (format: %Y-%m-%d) │\n",
|
"│ When (format: %Y-%m-%d) — one category of the raw dates │\n",
|
||||||
"│ [ ] Year │\n",
|
"│ [ ] Year │\n",
|
||||||
"│ [ ] Month │\n",
|
"│ [ ] Month │\n",
|
||||||
"│ [ ] Quarter │\n",
|
"│ [ ] Quarter │\n",
|
||||||
@@ -1196,7 +1199,7 @@ mod tests {
|
|||||||
"│ │\n",
|
"│ │\n",
|
||||||
"│ │\n",
|
"│ │\n",
|
||||||
"│ │\n",
|
"│ │\n",
|
||||||
"│Space: toggle Enter: next ← back Esc: cancel │\n",
|
"│Space: toggle s: one category / split Enter: next ← back Esc: cancel │\n",
|
||||||
"└──────────────────────────────────────────────────────────────────────────────┘\n",
|
"└──────────────────────────────────────────────────────────────────────────────┘\n",
|
||||||
"\n",
|
"\n",
|
||||||
"====\n",
|
"====\n",
|
||||||
|
|||||||
@@ -665,6 +665,167 @@ fn a_headless_import_keeps_duplicates_separate_when_told_to() {
|
|||||||
assert!(!doc.contains("= 30"), "{doc}");
|
assert!(!doc.contains("= 30"), "{doc}");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Dates across two years, one per month, so nothing collides.
|
||||||
|
fn dated_csv(scratch: &Scratch) -> PathBuf {
|
||||||
|
let csv = scratch.path("dated.csv");
|
||||||
|
std::fs::write(
|
||||||
|
&csv,
|
||||||
|
"Date,Revenue\n2024-01-15,1\n2024-04-02,10\n2025-01-03,100\n",
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
csv
|
||||||
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3, headless: `--extract` builds ONE hierarchical category by
|
||||||
|
/// default, as the wizard does, and it is written with its `>` lines.
|
||||||
|
#[test]
|
||||||
|
fn a_headless_date_import_is_one_hierarchical_category() {
|
||||||
|
let scratch = Scratch::new("date_hierarchy");
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&[
|
||||||
|
"--extract",
|
||||||
|
"Date:Year",
|
||||||
|
"--extract",
|
||||||
|
"Date:Quarter",
|
||||||
|
"--extract",
|
||||||
|
"Date:Month",
|
||||||
|
],
|
||||||
|
);
|
||||||
|
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||||
|
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||||
|
assert!(!doc.contains("Date_Month"), "{doc}");
|
||||||
|
assert!(doc.contains("- |2024-01|[|2024-Q1|]"), "{doc}");
|
||||||
|
assert!(doc.contains("> |2024-Q1|[|2024|]"), "{doc}");
|
||||||
|
assert!(!doc.contains("expanded"), "it opens folded: {doc}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The flat split stays available headless, as `s` makes it in the wizard.
|
||||||
|
#[test]
|
||||||
|
fn a_headless_date_import_splits_when_told_to() {
|
||||||
|
let scratch = Scratch::new("date_split");
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&["--extract", "Date:Month", "--split-date", "Date"],
|
||||||
|
);
|
||||||
|
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||||
|
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||||
|
assert!(doc.contains("## Category: Date_Month"), "{doc}");
|
||||||
|
assert!(doc.contains("2024-01-15"), "the raw dates stay: {doc}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3 review: a `--split-date` that names nothing splittable
|
||||||
|
/// said nothing and exited 0.
|
||||||
|
#[test]
|
||||||
|
fn split_date_naming_an_unknown_or_non_date_field_is_refused() {
|
||||||
|
let scratch = Scratch::new("split_date_unknown");
|
||||||
|
for field in ["Nope", "Revenue"] {
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&["--extract", "Date:Month", "--split-date", field],
|
||||||
|
);
|
||||||
|
assert!(!out.status.success(), "{field}: {}", stderr_of(&out));
|
||||||
|
let err = stderr_of(&out);
|
||||||
|
assert!(err.contains("--split-date") && err.contains(field), "{err}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// improvise-8jk6.3 review: `--axis Date_Month:column` names a category the
|
||||||
|
/// hierarchical import no longer builds, and was dropped without a word.
|
||||||
|
#[test]
|
||||||
|
fn an_axis_naming_a_category_the_import_did_not_build_is_refused() {
|
||||||
|
let scratch = Scratch::new("axis_unknown");
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&["--extract", "Date:Month", "--axis", "Date_Month:column"],
|
||||||
|
);
|
||||||
|
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||||
|
let err = stderr_of(&out);
|
||||||
|
assert!(
|
||||||
|
err.contains("Date_Month") && err.contains("--split-date"),
|
||||||
|
"{err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `--time` on a column with no date format makes it a time category without
|
||||||
|
/// dates, so there is nothing to split.
|
||||||
|
#[test]
|
||||||
|
fn split_date_on_a_time_field_with_no_date_format_is_refused() {
|
||||||
|
let scratch = Scratch::new("split_date_no_format");
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&["--time", "Revenue", "--split-date", "Revenue"],
|
||||||
|
);
|
||||||
|
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||||
|
assert!(
|
||||||
|
stderr_of(&out).contains("--split-date"),
|
||||||
|
"{}",
|
||||||
|
stderr_of(&out)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A name that is no date part gets the categories that exist, never the date
|
||||||
|
/// hint (the README's old `--axis Measure:column` was this).
|
||||||
|
#[test]
|
||||||
|
fn an_absent_axis_category_that_is_no_date_part_names_what_was_built() {
|
||||||
|
let scratch = Scratch::new("axis_unknown_plain");
|
||||||
|
let out_path = scratch.path("out.improv");
|
||||||
|
let out = run_import(
|
||||||
|
&dated_csv(&scratch),
|
||||||
|
&out_path,
|
||||||
|
&["--axis", "Measure:column"],
|
||||||
|
);
|
||||||
|
assert!(!out.status.success(), "{}", stderr_of(&out));
|
||||||
|
let err = stderr_of(&out);
|
||||||
|
assert!(
|
||||||
|
err.contains("'Measure'") && err.contains("_Measure"),
|
||||||
|
"{err}"
|
||||||
|
);
|
||||||
|
assert!(!err.contains("--split-date"), "no date hint: {err}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The README's import recipe runs as printed.
|
||||||
|
#[test]
|
||||||
|
fn the_readmes_import_recipe_runs() {
|
||||||
|
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
|
||||||
|
let readme = std::fs::read_to_string(root.join("README.md")).unwrap();
|
||||||
|
let start = readme.find("improvise import examples/demo.csv").unwrap();
|
||||||
|
let recipe = readme[start..]
|
||||||
|
.split("```")
|
||||||
|
.next()
|
||||||
|
.unwrap()
|
||||||
|
.replace("\\\n", " ");
|
||||||
|
let mut args: Vec<String> = vec![];
|
||||||
|
for (i, part) in recipe.split('"').enumerate() {
|
||||||
|
match i % 2 {
|
||||||
|
1 => args.push(part.to_string()),
|
||||||
|
_ => args.extend(part.split_whitespace().map(String::from)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let at = args.iter().position(|a| a == "-o").unwrap();
|
||||||
|
args.truncate(at);
|
||||||
|
let scratch = Scratch::new("readme_recipe");
|
||||||
|
let out = Command::new(bin())
|
||||||
|
.arg("import")
|
||||||
|
.arg(root.join("examples/demo.csv"))
|
||||||
|
.args(args.iter().skip(3))
|
||||||
|
.arg("-o")
|
||||||
|
.arg(scratch.path("out.improv"))
|
||||||
|
.output()
|
||||||
|
.unwrap();
|
||||||
|
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||||
|
}
|
||||||
|
|
||||||
/// code-maat's reports: three schemas, a stray note, and room for the output.
|
/// code-maat's reports: three schemas, a stray note, and room for the output.
|
||||||
fn code_maat_folder(scratch: &Scratch) -> PathBuf {
|
fn code_maat_folder(scratch: &Scratch) -> PathBuf {
|
||||||
let dir = scratch.path("maat");
|
let dir = scratch.path("maat");
|
||||||
|
|||||||
Reference in New Issue
Block a user