fix(import): a header-only or measureless file keeps its identity and a view of its own (improvise-lpzu, improvise-8jk6.13, improvise-8jk6.12)

ImportPipeline::from_merge seeds the File items and every file's columns
from Merged::sources, so a header-only CSV keeps its File item and its
columns (unseen columns are proposed as Labels, as an all-null column
is). A file with no measure column gets a records view filtered to it
instead of paging onto another file's hidden measure. Four per-file-view
rules the review found untested now each have a test that fails when
the rule is broken.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (agent-lpzu)
2026-09-24 21:47:11 -07:00
committed by Edward Langley
parent 005b7a769a
commit c43ee72402
7 changed files with 295 additions and 17 deletions
+41 -3
View File
@@ -691,8 +691,41 @@ so `Plan::place` gives it no coordinate there (improvise-8jk6.2 above), and no
measure cell. Nothing is fabricated: no `0` and no `""`. The analyzer already
classified each column over the values present (improvise-l1f5), so a column
only one file has is judged by that file's values alone. The category cap
counts the **union** of every file's categories. Rows from different files
cannot collide, because `File` differs.
counts the **union** of every file's categories, header-only files included
(see the next paragraph). Rows from different files cannot collide, because
`File` differs. One caveat on "judged by its values": a JSON `""` is not yet
dropped by the analyzer the way `null` is, so a JSON numeric column holding
`""` is proposed as a Label (improvise-8jk6.11). CSV is unaffected, because
its parser turns a blank field into `null`.
**A header-only file is still a file** (the Codex P2 on PR #314). code-maat
writes a header and no rows whenever nothing passes its threshold, so
`coupling.csv` is often exactly that. The pipeline is analyzed from *records*,
and such a file has none, so it had no `File` item, its columns never reached
the model, and its view filtered to an item that did not exist.
`ImportPipeline::from_merge` therefore **seeds from `Merged::sources`**:
- the `File` proposal's items are the sources' items, in file order, so every
file has its item whether it has rows or not;
- a source column that no record has a value in is proposed as a **Label**.
That is the analyzer's own verdict for an all-null column (improvise-l1f5),
and a column with no values is exactly that: nothing can be classified, so it
takes no pivot slot and does not count toward the category cap until the user
reclassifies it in the wizard or with `--category`/`--measure`.
This is also the file identity the user wants kept for a future update-import.
**A file with no measure column gets a records view** (improvise-8jk6.13). A
record with no measure makes no cell, so such a file contributes items and no
data. Its view had nothing to page `_Measure` to, and paged onto another file's
measure, which the same view hides; the cell there was a `Record`, editable,
under this file's `File`. Now `source_view` gives such a file the records
layout (`view::records_steps`, the one home of what a records view is) with
`File` still on Filter at the file's item. It draws zero rows, because there
are no cells. A header-only file whose columns are
all Labels is the same case. In records mode the columns are the whole model's
categories and measures, not just this file's: records mode has no per-view
column filter. With no rows that is cosmetic.
**Each file gets a view** (the user, 2026-09-25: "dir import should do this:
one view filtered by the file column and projected to the columns in that
@@ -721,7 +754,12 @@ initial view. Each per-file view:
records lack those, so nothing splits.
- **Puts `_Measure` on Page**, as `Workbook::new` does, **with other files'
measures hidden** and the page selection on the file's first measure. A
formula target comes from no file, so it stays visible.
formula target comes from no file, so it stays visible. A file with no measure
gets a records view instead (above).
- **Is never named like another view from the same import.** `Default.csv` and
`Default 2.csv` become `Default 2` and `Default 2 2`; without the same-batch
check both would be named `Default 2` and one view would silently replace the
other.
A per-file view is made only while `File` is an accepted category. A user who
skipped or reclassified it has no filter to build views from.
+3 -1
View File
@@ -444,7 +444,9 @@ or browser test was written. For each file's view:
- the drawn cells are that file's alone: `2.00, 7.00` for age and
`5.00, 3.00` for coupling.
Rivals R2, R7 and R9 each fail it.
Rivals R2, R7 and R9 each fail it. Since the review of `4e68372c` the fixture
also holds a header-only `soc.csv`. Its view must carry `File` at `soc`, be a
records view, and draw zero rows. Rivals F7 and F8 fail that half.
### a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not
+57
View File
@@ -1702,6 +1702,14 @@ improvise-lpzu. `[z.csv, b/, a.csv]` must expand to
is sorted within itself. A fixture listing only folders, or files in name
order, could not tell "argument order" from "sort everything". R6 fails it.
### two_files_with_one_stem_union_their_columns
improvise-8jk6.12 (3). `a/x.csv` (`entity,one`) and `b/x.csv` (`entity,two`) are
one `File` item, `x`, whose columns are `entity, one, two`. Rival F3 (the
reviewer's fault: clear the stem's columns before each file) leaves
`entity, two`, and the `x` view would then hide the measure `one` that half its
records carry.
### files_with_different_headers_merge_without_inventing_the_missing_columns
improvise-lpzu, at the parser. Each merged record carries exactly its own
@@ -1767,3 +1775,52 @@ identical `.improv` text, and `format(parse(text)) == text`. The reloaded
document still has the three views in order, `File: filter` at the file's item,
and the same drawn values. This is the first persisted, non-virtual use of
`Axis::Filter` (gotcha 27). R7 and R9 fail it.
### a_header_only_file_keeps_its_file_item_its_columns_and_its_view
The ChatGPT Codex P2 on PR #314 at `4e68372c`, confirmed by the adversarial
review. `age.csv` (one row), a header-only `coupling.csv` and `rev.csv` (one
row). At `005b7a76` the test failed at the first assertion: `File` held only
`age` and `rev`, so the `coupling` view filtered to an item that did not exist,
and `coupled`, `degree` and `average-revs` were not in the model. Now `File`
reads `age, coupling, rev` in file order, the three columns exist (as Labels),
and the `coupling` view is a zero-row records view filtered to `coupling`, while
the `age` view still draws `2`. Rival F5 (sources not seeded) and F6 (unseen
columns not proposed) each fail it, and so do F7 and F8 below.
### a_file_with_no_measure_gets_a_records_view_not_another_files_measure
improvise-8jk6.13, from the review of PR #314. `a.csv` (`entity,x`) and
`tags.csv` (`entity,tag`). At `005b7a76` the `tags` view had `_Measure` on Page
with no selection, so the layout paged onto `x`, a measure the same view hides.
The test failed there, with `row_count() == 1`: `a`'s record, drawn under
`File=tags`'s view. Now the view has no `_Measure` page, is a records layout,
and draws zero rows, while `a`'s view is still a pivot. Rival F7 (the old
paging) fails it, and F8 (the File selection lost when `records_steps` moves
`File` off Filter) fails it with the same `1`. F8 is the mistake the first
version of the fix made.
### a_files_view_seats_the_date_parts_of_its_own_date_column
improvise-8jk6.12 (1). The reviewer replaced the date-parts clause of
`source_view`'s `own` with `false` and every test passed, because no folder
fixture extracted a date part. Here `abs-churn.csv` has a `date` column,
extracted to `date_Month`, and `age.csv` does not. `date_Month` is seated in the
`abs-churn` view and on None in the `age` view. Rival F1 (the reviewer's fault)
puts it on None in both and fails the first assertion.
### two_files_whose_names_collide_in_one_batch_keep_a_view_each
improvise-8jk6.12 (2). `Default.csv` and `Default 2.csv`. The first takes
`Default 2`, so the second must take `Default 2 2`. Without the same-batch
check both are named `Default 2`, and `IndexMap::insert` silently replaces the
first, leaving two views where three are owed. The test asserts all three names
with their filters. Rival F2 (the reviewer's fault) fails it.
### a_whitespace_only_category_value_is_no_coordinate
improvise-8jk6.12 (4). A `region` of `" "` is kept as a record with no
`region` coordinate, and `region` holds only `East`. The trim in `scalar_name`
predates this PR (improvise-ych), but the sparse-record rule now rests on it:
without it the value becomes an item named `" "`. Rival F4 (the reviewer's
fault) fails it.
@@ -500,6 +500,22 @@ mod tests {
);
}
/// improvise-8jk6.12 (3): two files with one stem are one item, and its
/// columns are both files'. Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn two_files_with_one_stem_union_their_columns() {
let fs = folder(&[
("a/x.csv", "entity,one\ne,1\n"),
("b/x.csv", "entity,two\ne,2\n"),
]);
let merged = merge_csvs_from(&fs, &["a/x.csv", "b/x.csv"]).unwrap();
let expected = SourceFile {
item: "x".to_string(),
columns: ["entity", "one", "two"].map(String::from).to_vec(),
};
assert_eq!(merged.sources, [expected]);
}
// ── RFC 4180 edge cases ───────────────────────────────────────────
#[test]
+166 -7
View File
@@ -7,11 +7,11 @@ use super::analyzer::{
DateComponent, FieldKind, FieldProposal, analyze_records, extract_array_at_path,
extract_date_component, find_array_paths,
};
use super::csv_parser::{FILE_CATEGORY, SourceFile};
use super::csv_parser::{FILE_CATEGORY, Merged, SourceFile};
use crate::formula::parse_formula;
use crate::model::MAX_CATEGORIES;
use crate::model::cell::{CellKey, CellValue};
use crate::view::{Axis, VIRTUAL_VIEW_NAMES, View};
use crate::view::{Axis, VIRTUAL_VIEW_NAMES, View, ViewStep, records_steps};
use crate::workbook::Workbook;
// ── Pipeline (no UI state) ────────────────────────────────────────────────────
@@ -82,6 +82,32 @@ impl ImportPipeline {
pipeline
}
/// A pipeline over a CSV merge, carrying its sources for the per-file views.
pub fn from_merge(merged: Merged) -> Self {
let mut pipeline = Self::new(Value::Array(merged.records));
pipeline.seed_sources(merged.sources);
pipeline
}
/// Every file keeps its `File` item and its columns, rows or none: a
/// header-only file is still a file (improvise-lpzu).
fn seed_sources(&mut self, sources: Vec<SourceFile>) {
let items: Vec<String> = sources.iter().map(|s| s.item.clone()).collect();
match self.proposals.iter_mut().find(|p| p.field == FILE_CATEGORY) {
Some(file) => file.distinct_values = items,
None => self
.proposals
.push(unseen(FILE_CATEGORY, FieldKind::Category, items)),
}
for column in sources.iter().flat_map(|s| &s.columns) {
if !self.proposals.iter().any(|p| &p.field == column) {
self.proposals
.push(unseen(column, FieldKind::Label, vec![]));
}
}
self.sources = sources;
}
pub fn select_path(&mut self, path: &str) {
self.selected_path = path.to_string();
if let Some(arr) = extract_array_at_path(&self.raw, path) {
@@ -350,6 +376,19 @@ impl ImportPipeline {
}
}
/// A proposal for a column no record has a value in, which is what an
/// all-null column already gets from the analyzer (improvise-l1f5).
fn unseen(field: &str, kind: FieldKind, distinct_values: Vec<String>) -> FieldProposal {
FieldProposal {
field: field.to_string(),
kind,
distinct_values,
accepted: true,
date_format: None,
date_components: vec![],
}
}
/// One view per merged file, in file order, while `File` is a category. The
/// shape and why Filter: `context/code/formula-and-io.md` (improvise-lpzu).
fn source_views(wb: &Workbook, sources: &[SourceFile], plan: &Plan) -> Vec<View> {
@@ -398,12 +437,26 @@ fn source_view(wb: &Workbook, source: &SourceFile, plan: &Plan, name: String) ->
for measure in theirs {
view.hide_item("_Measure", &measure.field);
}
if let Some(first) = mine.first() {
view.set_page_selection("_Measure", &first.field);
match mine.first() {
Some(first) => view.set_page_selection("_Measure", &first.field),
None => into_records(&mut view, wb, &source.item),
}
view
}
/// A file with no measure has no measure to page to: its view is its records,
/// still filtered to the file (improvise-8jk6.13).
fn into_records(view: &mut View, wb: &Workbook, item: &str) {
for step in records_steps(&wb.model) {
match step {
ViewStep::Axis { category, axis } => view.set_axis(&category, axis),
ViewStep::PageSelection { category, item } => view.set_page_selection(&category, &item),
}
}
view.set_axis(FILE_CATEGORY, Axis::Filter);
view.set_page_selection(FILE_CATEGORY, item);
}
/// The accepted fields by role, and the date parts derived from them.
struct Plan<'a> {
categories: Vec<&'a FieldProposal>,
@@ -2176,9 +2229,7 @@ mod tests {
let paths: Vec<&str> = files.iter().map(|(p, _)| *p).collect();
let merged = merge_csvs_from(&fs, &paths).unwrap();
assert!(merged.warnings.is_empty(), "{:?}", merged.warnings);
let mut p = ImportPipeline::new(Value::Array(merged.records));
p.sources = merged.sources;
p
ImportPipeline::from_merge(merged)
}
/// code-maat's age and coupling reports.
@@ -2263,6 +2314,114 @@ mod tests {
assert_eq!((drill("Payee", "Bolt"), drill("Date", "d1")), (1, 1));
}
/// improvise-lpzu, the Codex P2 on PR #314: a header-only file keeps its
/// identity. Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn a_header_only_file_keeps_its_file_item_its_columns_and_its_view() {
use crate::view::GridLayout;
let (wb, _) = merged_pipeline(&[
("maat/age.csv", "entity,age-months\nsrc/a.rs,2\n"),
("maat/coupling.csv", "entity,coupled,degree,average-revs\n"),
("maat/rev.csv", "entity,n-revs\nsrc/a.rs,9\n"),
])
.build_model_with_warnings()
.unwrap();
let files: Vec<&String> = wb.model.category("File").unwrap().items.keys().collect();
assert_eq!(files, ["age", "coupling", "rev"]);
for column in ["coupled", "degree", "average-revs"] {
assert!(wb.model.category(column).is_some(), "{column} is missing");
}
let view = &wb.views["coupling"];
assert_eq!(view.page_selection("File"), Some("coupling"));
let layout = GridLayout::with_frozen_records(&wb.model, view, None);
assert!(layout.is_records_mode() && layout.row_count() == 0);
assert_eq!(drawn(&wb, "age"), [2.0]);
}
/// improvise-8jk6.13: a view never pages onto a measure it hides.
/// Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn a_file_with_no_measure_gets_a_records_view_not_another_files_measure() {
use crate::view::GridLayout;
let (wb, _) = merged_pipeline(&[
("t/a.csv", "entity,x\ne1,1\n"),
("t/tags.csv", "entity,tag\ne1,red\n"),
])
.build_model_with_warnings()
.unwrap();
let tags = &wb.views["tags"];
assert_eq!(tags.page_selection("_Measure"), None);
assert_ne!(tags.axis_of("_Measure"), Axis::Page, "{tags:?}");
let layout = GridLayout::with_frozen_records(&wb.model, tags, None);
assert!(layout.is_records_mode(), "{tags:?}");
assert_eq!(
layout.row_count(),
0,
"a record with no measure has no cell"
);
assert!(!wb.views["a"].is_records_layout());
}
/// improvise-8jk6.12 (1): a date part derived from the file's own column
/// is the file's. Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn a_files_view_seats_the_date_parts_of_its_own_date_column() {
use crate::import::analyzer::DateComponent;
let mut p = merged_pipeline(&[
("m/abs-churn.csv", "date,added\n2026-01-02,10\n"),
("m/age.csv", "entity,age-months\nsrc/a.rs,2\n"),
]);
let date = p.proposals.iter_mut().find(|f| f.field == "date").unwrap();
assert_eq!(date.kind, FieldKind::TimeCategory);
date.date_components = vec![DateComponent::Month];
let (wb, _) = p.build_model_with_warnings().unwrap();
assert_ne!(wb.views["abs-churn"].axis_of("date_Month"), Axis::None);
assert_eq!(wb.views["age"].axis_of("date_Month"), Axis::None);
}
/// improvise-8jk6.12 (2): two files of one batch never share a view name.
/// Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn two_files_whose_names_collide_in_one_batch_keep_a_view_each() {
let (wb, _) = merged_pipeline(&[
("d/Default.csv", "Region,Revenue\nEast,1\n"),
("d/Default 2.csv", "Region,Revenue\nEast,2\n"),
])
.build_model_with_warnings()
.unwrap();
let filters: Vec<(&str, Option<&str>)> = wb
.views
.values()
.map(|v| (v.name.as_str(), v.page_selection("File")))
.collect();
assert_eq!(
filters,
[
("Default", None),
("Default 2", Some("Default")),
("Default 2 2", Some("Default 2")),
]
);
}
/// improvise-8jk6.12 (4): a whitespace-only value is no coordinate, never
/// an item. Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
fn a_whitespace_only_category_value_is_no_coordinate() {
let (wb, _) = ImportPipeline::new(json!([
{"region": "East", "revenue": 1.0},
{"region": " ", "revenue": 2.0},
]))
.build_model_with_warnings()
.unwrap();
let items: Vec<&String> = wb.model.category("region").unwrap().items.keys().collect();
assert_eq!(items, ["East"]);
assert_eq!(
stored(&wb),
["_Measure=revenue, region=East = 1", "_Measure=revenue = 2"]
);
}
/// improvise-lpzu: each merged file gets a view of its own records and
/// columns. Evidence: `context/test-evidence/formula-and-io.md`.
#[test]
+1 -3
View File
@@ -364,9 +364,7 @@ fn get_import_data(args: &[PathBuf]) -> Option<ImportPipeline> {
for warning in &merged.warnings {
eprintln!("warning: {warning}");
}
let mut pipeline = ImportPipeline::new(Value::Array(merged.records));
pipeline.sources = merged.sources;
Some(pipeline)
Some(ImportPipeline::from_merge(merged))
}
Err(e) => {
eprintln!("CSV merge error: {e:#}");
+11 -3
View File
@@ -19,15 +19,16 @@ fn reloaded_folder_import() -> App {
"maat/coupling.csv",
"entity,coupled,degree,average-revs\nsrc/a.rs,src/b.rs,80,5\nsrc/b.rs,src/c.rs,40,3\n",
),
("maat/soc.csv", "entity,soc\n"),
];
for (path, body) in files {
fs.write(std::path::Path::new(path), body.as_bytes())
.unwrap();
}
let merged = merge_csvs_from(&fs, &files.map(|(p, _)| p)).unwrap();
let mut pipeline = ImportPipeline::new(serde_json::Value::Array(merged.records));
pipeline.sources = merged.sources;
let (wb, _) = pipeline.build_model_with_warnings().unwrap();
let (wb, _) = ImportPipeline::from_merge(merged)
.build_model_with_warnings()
.unwrap();
App::new(parse_md(&format_md(&wb)).unwrap(), None)
}
@@ -54,4 +55,11 @@ fn a_saved_per_file_view_projects_only_its_file() {
.collect();
assert_eq!(drawn, texts, "{view}");
}
// A header-only file: its item exists, and its view has nothing editable.
let soc = grid_in(&app, "soc", whole);
let file = soc.page_coords.iter().find(|p| p.cat == "File").unwrap();
assert_eq!(file.selected, "soc");
assert!(soc.records, "a file with no measure opens in records mode");
assert_eq!(soc.extent.rows, 0, "{soc:?}");
}