Merge commit 'c91cc055c8ad81afd8f055b71c74af6819020637' into fix/headless-import-refusals
# Conflicts: # context/test-evidence/cli.md # context/test-evidence/formula-and-io.md # crates/improvise-io/src/import/wizard.rs # src/main.rs # tests/folder_import_views.rs # tests/headless_cli.rs
This commit is contained in:
Claude (agent-headless-import)
committed by
Edward Langley
Edward Langley
commit
9b8b061a93
@@ -180,8 +180,8 @@ The import wizard analyzes CSV columns and proposes each as a category
|
||||
year/month/quarter extraction), or skip. Multiple CSVs merge automatically
|
||||
with a synthetic "File" category, even when their headers differ. A record
|
||||
simply has no value for a column its file lacks. A folder argument imports
|
||||
every `*.csv` in it, in name order, and each merged file also gets its own view,
|
||||
filtered to that file and showing only its columns:
|
||||
every `*.csv` in it, in name order, even when it holds a single CSV. Each file
|
||||
also gets its own view, filtered to that file and showing only its columns:
|
||||
|
||||
```sh
|
||||
improvise import reports/ --no-wizard -o reports.improv
|
||||
|
||||
+3
-1
@@ -373,7 +373,9 @@ there is deliberately no flag to name the category yet.
|
||||
## 12 — `import` takes folders, and the CLI only says which arguments are folders (improvise-lpzu)
|
||||
|
||||
`improvise import` takes any mix of files and folders. A folder stands for its
|
||||
`*.csv` files, in name order, so `import dir/` is `import dir/*.csv`. The rules
|
||||
`*.csv` files, in name order. A folder always merges, even when it holds a
|
||||
single CSV, so that file keeps its `File` item and view. A single file named
|
||||
directly still imports alone (the user's ruling on the PR). The rules
|
||||
and their reasons are in `context/code/formula-and-io.md`, "A folder of CSVs is
|
||||
one sparse model, with a view per file". The CLI's part is the one question
|
||||
`Storage` cannot answer: whether a path is a folder. `get_import_data` asks
|
||||
|
||||
@@ -734,10 +734,28 @@ columns not in a given CSV file." So files with different headers merge into
|
||||
takes the arguments in order and replaces each folder with its `*.csv`
|
||||
entries. The extension check is case-insensitive, the listing is not recursive,
|
||||
and the order is `Storage::list`'s, which is sorted by contract. Then the
|
||||
existing path runs unchanged: two or more files merge through `merge_csvs_from`
|
||||
with a `File` category, and one file imports alone. So `import dir/` is
|
||||
`import dir/*.csv`, including when the folder holds a single CSV. Four cases
|
||||
follow from that equivalence rather than being chosen one by one:
|
||||
existing path runs, with one difference: **a folder always merges**. Two or more
|
||||
files merge through `merge_csvs_from` with a `File` category, and so does a
|
||||
folder holding a single CSV. That gives the one file its `File` item and its
|
||||
per-file view.
|
||||
|
||||
The single-CSV rule is **the user's ruling (2026-09-25)**, which replaced this
|
||||
PR's first answer, "`import dir/` is exactly `import dir/*.csv`". Their words:
|
||||
"yes, we might want to have some kind of update import eventually (don't plan
|
||||
this yet, but we'd want file identity for this)". A folder names a collection
|
||||
that may grow, so a document built from it should record which file each
|
||||
record came from from the start.
|
||||
|
||||
**A single CSV named directly, not through a folder, still imports alone,
|
||||
without `File`.** That is deliberate. Naming one file asks for that file, not
|
||||
for a collection, and the shipped recipes (`examples/demo.improv` from
|
||||
`examples/demo.csv`) and every existing single-file script would otherwise gain
|
||||
a `File` category and a second view. File identity for update-import can be
|
||||
asked for by naming a folder. The test is
|
||||
`a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not`.
|
||||
|
||||
Apart from that, a folder still behaves as the glob would, and four cases follow
|
||||
from that rather than being chosen one by one:
|
||||
|
||||
- **Non-CSV entries are skipped**, not refused, because a glob skips them. This
|
||||
includes the output `.improv`, which a second run finds in the folder.
|
||||
@@ -761,8 +779,41 @@ so `Plan::place` gives it no coordinate there (improvise-8jk6.2 above), and no
|
||||
measure cell. Nothing is fabricated: no `0` and no `""`. The analyzer already
|
||||
classified each column over the values present (improvise-l1f5), so a column
|
||||
only one file has is judged by that file's values alone. The category cap
|
||||
counts the **union** of every file's categories. Rows from different files
|
||||
cannot collide, because `File` differs.
|
||||
counts the **union** of every file's categories, header-only files included
|
||||
(see the next paragraph). Rows from different files cannot collide, because
|
||||
`File` differs. One caveat on "judged by its values": a JSON `""` is not yet
|
||||
dropped by the analyzer the way `null` is, so a JSON numeric column holding
|
||||
`""` is proposed as a Label (improvise-8jk6.11). CSV is unaffected, because
|
||||
its parser turns a blank field into `null`.
|
||||
|
||||
**A header-only file is still a file** (the Codex P2 on PR #314). code-maat
|
||||
writes a header and no rows whenever nothing passes its threshold, so
|
||||
`coupling.csv` is often exactly that. The pipeline is analyzed from *records*,
|
||||
and such a file has none, so it had no `File` item, its columns never reached
|
||||
the model, and its view filtered to an item that did not exist.
|
||||
`ImportPipeline::from_merge` therefore **seeds from `Merged::sources`**:
|
||||
|
||||
- the `File` proposal's items are the sources' items, in file order, so every
|
||||
file has its item whether it has rows or not;
|
||||
- a source column that no record has a value in is proposed as a **Label**.
|
||||
That is the analyzer's own verdict for an all-null column (improvise-l1f5),
|
||||
and a column with no values is exactly that: nothing can be classified, so it
|
||||
takes no pivot slot and does not count toward the category cap until the user
|
||||
reclassifies it in the wizard or with `--category`/`--measure`.
|
||||
|
||||
This is also the file identity the user wants kept for a future update-import.
|
||||
|
||||
**A file with no measure column gets a records view** (improvise-8jk6.13). A
|
||||
record with no measure makes no cell, so such a file contributes items and no
|
||||
data. Its view had nothing to page `_Measure` to, and paged onto another file's
|
||||
measure, which the same view hides; the cell there was a `Record`, editable,
|
||||
under this file's `File`. Now `source_view` gives such a file the records
|
||||
layout (`view::records_steps`, the one home of what a records view is) with
|
||||
`File` still on Filter at the file's item. It draws zero rows, because there
|
||||
are no cells. A header-only file whose columns are
|
||||
all Labels is the same case. In records mode the columns are the whole model's
|
||||
categories and measures, not just this file's: records mode has no per-view
|
||||
column filter. With no rows that is cosmetic.
|
||||
|
||||
**Each file gets a view** (the user, 2026-09-25: "dir import should do this:
|
||||
one view filtered by the file column and projected to the columns in that
|
||||
@@ -791,7 +842,12 @@ initial view. Each per-file view:
|
||||
records lack those, so nothing splits.
|
||||
- **Puts `_Measure` on Page**, as `Workbook::new` does, **with other files'
|
||||
measures hidden** and the page selection on the file's first measure. A
|
||||
formula target comes from no file, so it stays visible.
|
||||
formula target comes from no file, so it stays visible. A file with no measure
|
||||
gets a records view instead (above).
|
||||
- **Is never named like another view from the same import.** `Default.csv` and
|
||||
`Default 2.csv` become `Default 2` and `Default 2 2`; without the same-batch
|
||||
check both would be named `Default 2` and one view would silently replace the
|
||||
other.
|
||||
|
||||
A per-file view is made only while `File` is an accepted category. A user who
|
||||
skipped or reclassified it has no filter to build views from.
|
||||
@@ -852,9 +908,11 @@ test is `a_record_with_no_coordinate_is_drawn_only_where_that_category_is_off_th
|
||||
- **`.improv`:** its Data line omits the category, and it round-trips.
|
||||
- **Drill:** a drill that does not constrain the category reaches it.
|
||||
|
||||
Whether Row/Column/Page should give such records a place of their own (a
|
||||
"(none)" position) is **open, and it is the user's call**. It is raised on the
|
||||
lpzu PR. Nothing here invents one.
|
||||
**No "(none)" position, by the user's decision (2026-09-25).** They were asked
|
||||
whether Row, Column or Page should give such records a place of their own. Their
|
||||
answer: "I think leaving it out for now is fine, maybe a UI element somewhere
|
||||
that shows the number of records underneath the current view?" That indicator
|
||||
is improvise-gtn7.10, and it is not an import concern.
|
||||
|
||||
### Colliding records are a question, not a default (improvise-cgqt)
|
||||
|
||||
|
||||
@@ -444,7 +444,9 @@ or browser test was written. For each file's view:
|
||||
- the drawn cells are that file's alone: `2.00, 7.00` for age and
|
||||
`5.00, 3.00` for coupling.
|
||||
|
||||
Rivals R2, R7 and R9 each fail it.
|
||||
Rivals R2, R7 and R9 each fail it. Since the review of `4e68372c` the fixture
|
||||
also holds a header-only `soc.csv`. Its view must carry `File` at `soc`, be a
|
||||
records view, and draw zero rows. Rivals F7 and F8 fail that half.
|
||||
|
||||
### a_csv_column_forced_to_category_keeps_its_text_verbatim
|
||||
|
||||
@@ -494,3 +496,14 @@ decision and not a drift. It asserts the warning's count (`1 of 3`), exit 0,
|
||||
and a Data line from the record *after* the skipped one. That last check
|
||||
shows the skip did not end the import early. **R9**, making any skip fatal,
|
||||
fails it.
|
||||
|
||||
### a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not
|
||||
|
||||
improvise-lpzu, after the user's ruling on PR #314 ("we'd want file identity").
|
||||
At `4e68372c`, a folder holding one CSV took the single-file path, because
|
||||
`import dir/` was exactly `dir/*.csv`. That document had no `File` category and
|
||||
no per-file view, and the test failed at the `## Category: File` assertion. The
|
||||
other half pins the direction the ruling did *not* change: the same CSV named
|
||||
directly imports without any `File`. The rival that merges every all-CSV
|
||||
import, lone files included, fails that half ("a lone file keeps today's
|
||||
shape"). So the test separates "a folder merges" from "everything merges".
|
||||
|
||||
@@ -1712,6 +1712,14 @@ improvise-lpzu. `[z.csv, b/, a.csv]` must expand to
|
||||
is sorted within itself. A fixture listing only folders, or files in name
|
||||
order, could not tell "argument order" from "sort everything". R6 fails it.
|
||||
|
||||
### two_files_with_one_stem_union_their_columns
|
||||
|
||||
improvise-8jk6.12 (3). `a/x.csv` (`entity,one`) and `b/x.csv` (`entity,two`) are
|
||||
one `File` item, `x`, whose columns are `entity, one, two`. Rival F3 (the
|
||||
reviewer's fault: clear the stem's columns before each file) leaves
|
||||
`entity, two`, and the `x` view would then hide the measure `one` that half its
|
||||
records carry.
|
||||
|
||||
### files_with_different_headers_merge_without_inventing_the_missing_columns
|
||||
|
||||
improvise-lpzu, at the parser. Each merged record carries exactly its own
|
||||
@@ -1837,3 +1845,52 @@ where the user can still choose another array path). On the base the step
|
||||
refused with `No categories: … Press c to cycle a field's kind`, which offers
|
||||
a key that has no field to act on. **R8**, the step without the check, fails
|
||||
it. The build test does not see R8.
|
||||
|
||||
### a_header_only_file_keeps_its_file_item_its_columns_and_its_view
|
||||
|
||||
The ChatGPT Codex P2 on PR #314 at `4e68372c`, confirmed by the adversarial
|
||||
review. `age.csv` (one row), a header-only `coupling.csv` and `rev.csv` (one
|
||||
row). At `005b7a76` the test failed at the first assertion: `File` held only
|
||||
`age` and `rev`, so the `coupling` view filtered to an item that did not exist,
|
||||
and `coupled`, `degree` and `average-revs` were not in the model. Now `File`
|
||||
reads `age, coupling, rev` in file order, the three columns exist (as Labels),
|
||||
and the `coupling` view is a zero-row records view filtered to `coupling`, while
|
||||
the `age` view still draws `2`. Rival F5 (sources not seeded) and F6 (unseen
|
||||
columns not proposed) each fail it, and so do F7 and F8 below.
|
||||
|
||||
### a_file_with_no_measure_gets_a_records_view_not_another_files_measure
|
||||
|
||||
improvise-8jk6.13, from the review of PR #314. `a.csv` (`entity,x`) and
|
||||
`tags.csv` (`entity,tag`). At `005b7a76` the `tags` view had `_Measure` on Page
|
||||
with no selection, so the layout paged onto `x`, a measure the same view hides.
|
||||
The test failed there, with `row_count() == 1`: `a`'s record, drawn under
|
||||
`File=tags`'s view. Now the view has no `_Measure` page, is a records layout,
|
||||
and draws zero rows, while `a`'s view is still a pivot. Rival F7 (the old
|
||||
paging) fails it, and F8 (the File selection lost when `records_steps` moves
|
||||
`File` off Filter) fails it with the same `1`. F8 is the mistake the first
|
||||
version of the fix made.
|
||||
|
||||
### a_files_view_seats_the_date_parts_of_its_own_date_column
|
||||
|
||||
improvise-8jk6.12 (1). The reviewer replaced the date-parts clause of
|
||||
`source_view`'s `own` with `false` and every test passed, because no folder
|
||||
fixture extracted a date part. Here `abs-churn.csv` has a `date` column,
|
||||
extracted to `date_Month`, and `age.csv` does not. `date_Month` is seated in the
|
||||
`abs-churn` view and on None in the `age` view. Rival F1 (the reviewer's fault)
|
||||
puts it on None in both and fails the first assertion.
|
||||
|
||||
### two_files_whose_names_collide_in_one_batch_keep_a_view_each
|
||||
|
||||
improvise-8jk6.12 (2). `Default.csv` and `Default 2.csv`. The first takes
|
||||
`Default 2`, so the second must take `Default 2 2`. Without the same-batch
|
||||
check both are named `Default 2`, and `IndexMap::insert` silently replaces the
|
||||
first, leaving two views where three are owed. The test asserts all three names
|
||||
with their filters. Rival F2 (the reviewer's fault) fails it.
|
||||
|
||||
### a_whitespace_only_category_value_is_no_coordinate
|
||||
|
||||
improvise-8jk6.12 (4). A `region` of `" "` is kept as a record with no
|
||||
`region` coordinate, and `region` holds only `East`. The trim in `scalar_name`
|
||||
predates this PR (improvise-ych), but the sparse-record rule now rests on it:
|
||||
without it the value becomes an item named `" "`. Rival F4 (the reviewer's
|
||||
fault) fails it.
|
||||
|
||||
@@ -524,6 +524,22 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.12 (3): two files with one stem are one item, and its
|
||||
/// columns are both files'. Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn two_files_with_one_stem_union_their_columns() {
|
||||
let fs = folder(&[
|
||||
("a/x.csv", "entity,one\ne,1\n"),
|
||||
("b/x.csv", "entity,two\ne,2\n"),
|
||||
]);
|
||||
let merged = merge_csvs_from(&fs, &["a/x.csv", "b/x.csv"]).unwrap();
|
||||
let expected = SourceFile {
|
||||
item: "x".to_string(),
|
||||
columns: ["entity", "one", "two"].map(String::from).to_vec(),
|
||||
};
|
||||
assert_eq!(merged.sources, [expected]);
|
||||
}
|
||||
|
||||
// ── RFC 4180 edge cases ───────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -7,11 +7,11 @@ use super::analyzer::{
|
||||
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
|
||||
extract_date_component, find_array_paths,
|
||||
};
|
||||
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, SourceFile};
|
||||
use super::csv_parser::{self, CsvRecords, FILE_CATEGORY, Merged, SourceFile};
|
||||
use crate::formula::{Formula, parse_formula};
|
||||
use crate::model::MAX_CATEGORIES;
|
||||
use crate::model::cell::{CellKey, CellValue};
|
||||
use crate::view::{Axis, VIRTUAL_VIEW_NAMES, View};
|
||||
use crate::view::{Axis, VIRTUAL_VIEW_NAMES, View, ViewStep, records_steps};
|
||||
use crate::workbook::Workbook;
|
||||
|
||||
// ── Pipeline (no UI state) ────────────────────────────────────────────────────
|
||||
@@ -129,6 +129,32 @@ impl ImportPipeline {
|
||||
pipeline
|
||||
}
|
||||
|
||||
/// A pipeline over a CSV merge, carrying its sources for the per-file views.
|
||||
pub fn from_merge(merged: Merged) -> Self {
|
||||
let mut pipeline = Self::from_csv(merged.records);
|
||||
pipeline.seed_sources(merged.sources);
|
||||
pipeline
|
||||
}
|
||||
|
||||
/// Every file keeps its `File` item and its columns, rows or none: a
|
||||
/// header-only file is still a file (improvise-lpzu).
|
||||
fn seed_sources(&mut self, sources: Vec<SourceFile>) {
|
||||
let items: Vec<String> = sources.iter().map(|s| s.item.clone()).collect();
|
||||
match self.proposals.iter_mut().find(|p| p.field == FILE_CATEGORY) {
|
||||
Some(file) => file.distinct_values = items,
|
||||
None => self
|
||||
.proposals
|
||||
.push(unseen(FILE_CATEGORY, FieldKind::Category, items)),
|
||||
}
|
||||
for column in sources.iter().flat_map(|s| &s.columns) {
|
||||
if !self.proposals.iter().any(|p| &p.field == column) {
|
||||
self.proposals
|
||||
.push(unseen(column, FieldKind::Label, vec![]));
|
||||
}
|
||||
}
|
||||
self.sources = sources;
|
||||
}
|
||||
|
||||
pub fn select_path(&mut self, path: &str) {
|
||||
self.selected_path = path.to_string();
|
||||
if let Some(arr) = extract_array_at_path(&self.raw, path) {
|
||||
@@ -436,6 +462,19 @@ impl ImportPipeline {
|
||||
}
|
||||
}
|
||||
|
||||
/// A proposal for a column no record has a value in, which is what an
|
||||
/// all-null column already gets from the analyzer (improvise-l1f5).
|
||||
fn unseen(field: &str, kind: FieldKind, distinct_values: Vec<String>) -> FieldProposal {
|
||||
FieldProposal {
|
||||
field: field.to_string(),
|
||||
kind,
|
||||
distinct_values,
|
||||
accepted: true,
|
||||
date_format: None,
|
||||
date_components: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
/// One view per merged file, in file order, while `File` is a category. The
|
||||
/// shape and why Filter: `context/code/formula-and-io.md` (improvise-lpzu).
|
||||
fn source_views(wb: &Workbook, sources: &[SourceFile], plan: &Plan) -> Vec<View> {
|
||||
@@ -484,12 +523,26 @@ fn source_view(wb: &Workbook, source: &SourceFile, plan: &Plan, name: String) ->
|
||||
for measure in theirs {
|
||||
view.hide_item("_Measure", &measure.field);
|
||||
}
|
||||
if let Some(first) = mine.first() {
|
||||
view.set_page_selection("_Measure", &first.field);
|
||||
match mine.first() {
|
||||
Some(first) => view.set_page_selection("_Measure", &first.field),
|
||||
None => into_records(&mut view, wb, &source.item),
|
||||
}
|
||||
view
|
||||
}
|
||||
|
||||
/// A file with no measure has no measure to page to: its view is its records,
|
||||
/// still filtered to the file (improvise-8jk6.13).
|
||||
fn into_records(view: &mut View, wb: &Workbook, item: &str) {
|
||||
for step in records_steps(&wb.model) {
|
||||
match step {
|
||||
ViewStep::Axis { category, axis } => view.set_axis(&category, axis),
|
||||
ViewStep::PageSelection { category, item } => view.set_page_selection(&category, &item),
|
||||
}
|
||||
}
|
||||
view.set_axis(FILE_CATEGORY, Axis::Filter);
|
||||
view.set_page_selection(FILE_CATEGORY, item);
|
||||
}
|
||||
|
||||
/// The accepted fields by role, and the date parts derived from them.
|
||||
struct Plan<'a> {
|
||||
categories: Vec<&'a FieldProposal>,
|
||||
@@ -2280,9 +2333,7 @@ mod tests {
|
||||
let paths: Vec<&str> = files.iter().map(|(p, _)| *p).collect();
|
||||
let merged = merge_csvs_from(&fs, &paths).unwrap();
|
||||
assert!(merged.warnings.is_empty(), "{:?}", merged.warnings);
|
||||
let mut p = ImportPipeline::from_csv(merged.records);
|
||||
p.sources = merged.sources;
|
||||
p
|
||||
ImportPipeline::from_merge(merged)
|
||||
}
|
||||
|
||||
/// code-maat's age and coupling reports.
|
||||
@@ -2370,6 +2421,114 @@ mod tests {
|
||||
assert_eq!((drill("Payee", "Bolt"), drill("Date", "d1")), (1, 1));
|
||||
}
|
||||
|
||||
/// improvise-lpzu, the Codex P2 on PR #314: a header-only file keeps its
|
||||
/// identity. Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn a_header_only_file_keeps_its_file_item_its_columns_and_its_view() {
|
||||
use crate::view::GridLayout;
|
||||
let (wb, _) = merged_pipeline(&[
|
||||
("maat/age.csv", "entity,age-months\nsrc/a.rs,2\n"),
|
||||
("maat/coupling.csv", "entity,coupled,degree,average-revs\n"),
|
||||
("maat/rev.csv", "entity,n-revs\nsrc/a.rs,9\n"),
|
||||
])
|
||||
.build_model_with_warnings()
|
||||
.unwrap();
|
||||
let files: Vec<&String> = wb.model.category("File").unwrap().items.keys().collect();
|
||||
assert_eq!(files, ["age", "coupling", "rev"]);
|
||||
for column in ["coupled", "degree", "average-revs"] {
|
||||
assert!(wb.model.category(column).is_some(), "{column} is missing");
|
||||
}
|
||||
let view = &wb.views["coupling"];
|
||||
assert_eq!(view.page_selection("File"), Some("coupling"));
|
||||
let layout = GridLayout::with_frozen_records(&wb.model, view, None);
|
||||
assert!(layout.is_records_mode() && layout.row_count() == 0);
|
||||
assert_eq!(drawn(&wb, "age"), [2.0]);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.13: a view never pages onto a measure it hides.
|
||||
/// Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn a_file_with_no_measure_gets_a_records_view_not_another_files_measure() {
|
||||
use crate::view::GridLayout;
|
||||
let (wb, _) = merged_pipeline(&[
|
||||
("t/a.csv", "entity,x\ne1,1\n"),
|
||||
("t/tags.csv", "entity,tag\ne1,red\n"),
|
||||
])
|
||||
.build_model_with_warnings()
|
||||
.unwrap();
|
||||
let tags = &wb.views["tags"];
|
||||
assert_eq!(tags.page_selection("_Measure"), None);
|
||||
assert_ne!(tags.axis_of("_Measure"), Axis::Page, "{tags:?}");
|
||||
let layout = GridLayout::with_frozen_records(&wb.model, tags, None);
|
||||
assert!(layout.is_records_mode(), "{tags:?}");
|
||||
assert_eq!(
|
||||
layout.row_count(),
|
||||
0,
|
||||
"a record with no measure has no cell"
|
||||
);
|
||||
assert!(!wb.views["a"].is_records_layout());
|
||||
}
|
||||
|
||||
/// improvise-8jk6.12 (1): a date part derived from the file's own column
|
||||
/// is the file's. Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn a_files_view_seats_the_date_parts_of_its_own_date_column() {
|
||||
use crate::import::analyzer::DateComponent;
|
||||
let mut p = merged_pipeline(&[
|
||||
("m/abs-churn.csv", "date,added\n2026-01-02,10\n"),
|
||||
("m/age.csv", "entity,age-months\nsrc/a.rs,2\n"),
|
||||
]);
|
||||
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
|
||||
assert_eq!(date.kind, FieldKind::TimeCategory);
|
||||
date.date_components = vec![DateComponent::Month];
|
||||
let (wb, _) = p.build_model_with_warnings().unwrap();
|
||||
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
|
||||
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.12 (2): two files of one batch never share a view name.
|
||||
/// Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn two_files_whose_names_collide_in_one_batch_keep_a_view_each() {
|
||||
let (wb, _) = merged_pipeline(&[
|
||||
("d/Default.csv", "Region,Revenue\nEast,1\n"),
|
||||
("d/Default 2.csv", "Region,Revenue\nEast,2\n"),
|
||||
])
|
||||
.build_model_with_warnings()
|
||||
.unwrap();
|
||||
let filters: Vec<(&str, Option<&str>)> = wb
|
||||
.views
|
||||
.values()
|
||||
.map(|v| (v.name.as_str(), v.page_selection("File")))
|
||||
.collect();
|
||||
assert_eq!(
|
||||
filters,
|
||||
[
|
||||
("Default", None),
|
||||
("Default 2", Some("Default")),
|
||||
("Default 2 2", Some("Default 2")),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
/// improvise-8jk6.12 (4): a whitespace-only value is no coordinate, never
|
||||
/// an item. Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
fn a_whitespace_only_category_value_is_no_coordinate() {
|
||||
let (wb, _) = ImportPipeline::new(json!([
|
||||
{"region": "East", "revenue": 1.0},
|
||||
{"region": " ", "revenue": 2.0},
|
||||
]))
|
||||
.build_model_with_warnings()
|
||||
.unwrap();
|
||||
let items: Vec<&String> = wb.model.category("region").unwrap().items.keys().collect();
|
||||
assert_eq!(items, ["East"]);
|
||||
assert_eq!(
|
||||
stored(&wb),
|
||||
["_Measure=revenue, region=East = 1", "_Measure=revenue = 2"]
|
||||
);
|
||||
}
|
||||
|
||||
/// improvise-lpzu: each merged file gets a view of its own records and
|
||||
/// columns. Evidence: `context/test-evidence/formula-and-io.md`.
|
||||
#[test]
|
||||
|
||||
+6
-4
@@ -348,8 +348,12 @@ fn get_import_data(args: &[PathBuf]) -> Option<ImportPipeline> {
|
||||
};
|
||||
let paths = paths.as_slice();
|
||||
let all_csv = paths.iter().all(|p| csv_path_p(p));
|
||||
// A folder always merges, so even one CSV keeps its file identity (lpzu).
|
||||
let a_folder = classified
|
||||
.iter()
|
||||
.any(|a| matches!(a, ImportPath::Folder(_)));
|
||||
|
||||
if paths.len() > 1 {
|
||||
if paths.len() > 1 || a_folder {
|
||||
if !all_csv {
|
||||
eprintln!("Multi-file import only supports CSV files");
|
||||
return None;
|
||||
@@ -360,9 +364,7 @@ fn get_import_data(args: &[PathBuf]) -> Option<ImportPipeline> {
|
||||
for warning in &merged.warnings {
|
||||
eprintln!("warning: {warning}");
|
||||
}
|
||||
let mut pipeline = ImportPipeline::from_csv(merged.records);
|
||||
pipeline.sources = merged.sources;
|
||||
Some(pipeline)
|
||||
Some(ImportPipeline::from_merge(merged))
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("CSV merge error: {e:#}");
|
||||
|
||||
@@ -19,15 +19,16 @@ fn reloaded_folder_import() -> App {
|
||||
"maat/coupling.csv",
|
||||
"entity,coupled,degree,average-revs\nsrc/a.rs,src/b.rs,80,5\nsrc/b.rs,src/c.rs,40,3\n",
|
||||
),
|
||||
("maat/soc.csv", "entity,soc\n"),
|
||||
];
|
||||
for (path, body) in files {
|
||||
fs.write(std::path::Path::new(path), body.as_bytes())
|
||||
.unwrap();
|
||||
}
|
||||
let merged = merge_csvs_from(&fs, &files.map(|(p, _)| p)).unwrap();
|
||||
let mut pipeline = ImportPipeline::from_csv(merged.records);
|
||||
pipeline.sources = merged.sources;
|
||||
let (wb, _) = pipeline.build_model_with_warnings().unwrap();
|
||||
let (wb, _) = ImportPipeline::from_merge(merged)
|
||||
.build_model_with_warnings()
|
||||
.unwrap();
|
||||
App::new(parse_md(&format_md(&wb)).unwrap(), None)
|
||||
}
|
||||
|
||||
@@ -54,4 +55,11 @@ fn a_saved_per_file_view_projects_only_its_file() {
|
||||
.collect();
|
||||
assert_eq!(drawn, texts, "{view}");
|
||||
}
|
||||
|
||||
// A header-only file: its item exists, and its view has nothing editable.
|
||||
let soc = grid_in(&app, "soc", whole);
|
||||
let file = soc.page_coords.iter().find(|p| p.cat == "File").unwrap();
|
||||
assert_eq!(file.selected, "soc");
|
||||
assert!(soc.records, "a file with no measure opens in records mode");
|
||||
assert_eq!(soc.extent.rows, 0, "{soc:?}");
|
||||
}
|
||||
|
||||
@@ -822,3 +822,34 @@ fn an_import_that_skips_some_records_warns_with_a_count_and_still_writes() {
|
||||
let doc = std::fs::read_to_string(&out_path).unwrap();
|
||||
assert!(doc.contains("_Measure=x, a=m = 4"), "{doc}");
|
||||
}
|
||||
|
||||
/// improvise-lpzu, the user's ruling: a folder always merges, so even one CSV
|
||||
/// keeps its file identity. Evidence: `context/test-evidence/cli.md`.
|
||||
#[test]
|
||||
fn a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not() {
|
||||
let scratch = Scratch::new("folder_single");
|
||||
let dir = scratch.path("one");
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let csv = dir.join("age.csv");
|
||||
std::fs::write(&csv, "entity,age-months\nsrc/a.rs,2\n").unwrap();
|
||||
|
||||
let from_folder = scratch.path("folder.improv");
|
||||
let out = run_import(&dir, &from_folder, &[]);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&from_folder).unwrap();
|
||||
assert!(doc.contains("## Category: File\n- age\n"), "{doc}");
|
||||
assert!(doc.lines().any(|l| l == "File: filter, age"), "{doc}");
|
||||
assert!(
|
||||
doc.contains("File=age, _Measure=age-months, entity=|src/a.rs| = 2"),
|
||||
"{doc}"
|
||||
);
|
||||
|
||||
let from_file = scratch.path("file.improv");
|
||||
let out = run_import(&csv, &from_file, &[]);
|
||||
assert!(out.status.success(), "{}", stderr_of(&out));
|
||||
let doc = std::fs::read_to_string(&from_file).unwrap();
|
||||
assert!(
|
||||
!doc.contains("File"),
|
||||
"a lone file keeps today's shape: {doc}"
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user