fiddlerwoaroof/improvise
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge commit 'c91cc055c8ad81afd8f055b71c74af6819020637' into feat/felo6-group-parents

Browse Source 
# Conflicts:
#	context/test-evidence/formula-and-io.md
This commit is contained in:
Claude (agent-hier)
2026-09-24 22:42:49 -07:00
committed by Edward Langley
parent 8d3230e685 c91cc055c8
commit 44718e8ab3
13 changed files with 1494 additions and 166 deletions
Download Patch File Download Diff File Expand all files Collapse all files
context/code/formula-and-io.md
+170 -5
View File
@@ -658,6 +658,144 @@ one proves nothing.
backend rather than only a file (improvise-fuj); the path-taking wrappers remain
for the native CLI.
### A folder of CSVs is one sparse model, with a view per file (improvise-lpzu)
`improvise import <folder>` exists for improvise-1xfb.5.1. code-maat writes one
CSV per analysis into a directory, and the schemas differ: coupling has
`entity,coupled,degree,average-revs`, age has `entity,age-months`, and
abs-churn has `date,added,deleted,commits`. The user ruled (2026-09-25):
"Records are already sparse: the model can just leave null/empty values for
columns not in a given CSV file." So files with different headers merge into
**one** model. It is not refused, and it is not split into one model per schema.
**A folder argument means exactly its `*.csv` files.** `expand_import_paths`
takes the arguments in order and replaces each folder with its `*.csv`
entries. The extension check is case-insensitive, the listing is not recursive,
and the order is `Storage::list`'s, which is sorted by contract. Then the
existing path runs, with one difference: **a folder always merges**. Two or more
files merge through `merge_csvs_from` with a `File` category, and so does a
folder holding a single CSV. That gives the one file its `File` item and its
per-file view.
The single-CSV rule is **the user's ruling (2026-09-25)**, which replaced this
PR's first answer, "`import dir/` is exactly `import dir/*.csv`". Their words:
"yes, we might want to have some kind of update import eventually (don't plan
this yet, but we'd want file identity for this)". A folder names a collection
that may grow, so a document built from it should record which file each
record came from from the start.
**A single CSV named directly, not through a folder, still imports alone,
without `File`.** That is deliberate. Naming one file asks for that file, not
for a collection, and the shipped recipes (`examples/demo.improv` from
`examples/demo.csv`) and every existing single-file script would otherwise gain
a `File` category and a second view. File identity for update-import can be
asked for by naming a folder. The test is
`a_folder_of_one_csv_still_gets_a_file_category_and_its_view_but_a_lone_file_does_not`.
Apart from that, a folder still behaves as the glob would, and four cases follow
from that rather than being chosen one by one:
- **Non-CSV entries are skipped**, not refused, because a glob skips them. This
includes the output `.improv`, which a second run finds in the folder.
- **A folder with no CSV at all is refused**, naming the folder. A glob that
matches nothing is an error too, and importing nothing would write an empty
document.
- **Files and folders mix freely**, in argument order.
- **A file named twice is read twice.** For example, `dir/a.csv dir/` reads
`a.csv` twice, and its records collide with themselves. The headless path
refuses that like any other collision. Nothing deduplicates, because the
shell glob would not either.
The listing goes through `Storage::list` so the browser host can reuse it
(gotcha 2). `Storage` knows documents, not folders, so **whether an argument
is a folder is the caller's question**. The CLI answers it at the edge with
`Path::is_dir` and passes an `ImportPath::{File, Folder}`. An OPFS host would
answer it from its own UI.
**Sparse merge.** A record from a file without some column has no key for it,
so `Plan::place` gives it no coordinate there (improvise-8jk6.2 above), and no
measure cell. Nothing is fabricated: no `0` and no `""`. The analyzer already
classified each column over the values present (improvise-l1f5), so a column
only one file has is judged by that file's values alone. The category cap
counts the **union** of every file's categories, header-only files included
(see the next paragraph). Rows from different files cannot collide, because
`File` differs. One caveat on "judged by its values": a JSON `""` is not yet
dropped by the analyzer the way `null` is, so a JSON numeric column holding
`""` is proposed as a Label (improvise-8jk6.11). CSV is unaffected, because
its parser turns a blank field into `null`.
**A header-only file is still a file** (the Codex P2 on PR #314). code-maat
writes a header and no rows whenever nothing passes its threshold, so
`coupling.csv` is often exactly that. The pipeline is analyzed from *records*,
and such a file has none, so it had no `File` item, its columns never reached
the model, and its view filtered to an item that did not exist.
`ImportPipeline::from_merge` therefore **seeds from `Merged::sources`**:
- the `File` proposal's items are the sources' items, in file order, so every
file has its item whether it has rows or not;
- a source column that no record has a value in is proposed as a **Label**.
That is the analyzer's own verdict for an all-null column (improvise-l1f5),
and a column with no values is exactly that: nothing can be classified, so it
takes no pivot slot and does not count toward the category cap until the user
reclassifies it in the wizard or with `--category`/`--measure`.
This is also the file identity the user wants kept for a future update-import.
**A file with no measure column gets a records view** (improvise-8jk6.13). A
record with no measure makes no cell, so such a file contributes items and no
data. Its view had nothing to page `_Measure` to, and paged onto another file's
measure, which the same view hides; the cell there was a `Record`, editable,
under this file's `File`. Now `source_view` gives such a file the records
layout (`view::records_steps`, the one home of what a records view is) with
`File` still on Filter at the file's item. It draws zero rows, because there
are no cells. A header-only file whose columns are
all Labels is the same case. In records mode the columns are the whole model's
categories and measures, not just this file's: records mode has no per-view
column filter. With no rows that is cosmetic.
**Each file gets a view** (the user, 2026-09-25: "dir import should do this:
one view filtered by the file column and projected to the columns in that
file"). `merge_csvs_from` returns a `Merged` whose `sources` list each `File`
item with the columns its header named. Two files with one stem share an item,
so their columns are unioned. The pipeline carries `sources`, and the build
adds one view per source after the merged `Default` view, which stays the
initial view. Each per-file view:
- **Is named after the file's stem.** A name already taken, including the
virtual `_Records` and `_Drill`, gets ` 2`, ` 3` and so on. A view named
`_Drill` would never be written and would be dropped on load.
- **Puts `File` on `Axis::Filter`, fixed at that file's item.** Filter and
not Page, because the view *is* that file's projection. Paging to another
file would show that file's records through this file's columns and hidden
measures, which is wrong on every page but one. Filter constrains without
being navigable (gotcha 27). Until now it appeared only in the never-saved
`_Drill` view, but the `.improv` grammar has always round-tripped it.
`per_file_views_survive_a_round_trip_and_repeat_byte_for_byte` proves that on
a saved per-file view, and `a_saved_per_file_view_projects_only_its_file`
proves the shared `grid_in` projection, which both the TUI and the browser
draw from, carries it as a fixed coordinate with nothing to cycle.
- **Seats the file's own categories** (its columns, plus date parts derived
from them) through `View::on_category_added`, in model order: the rule a
single-file import uses. Every other category goes on `None`. The file's
records lack those, so nothing splits.
- **Puts `_Measure` on Page**, as `Workbook::new` does, **with other files'
measures hidden** and the page selection on the file's first measure. A
formula target comes from no file, so it stays visible. A file with no measure
gets a records view instead (above).
- **Is never named like another view from the same import.** `Default.csv` and
`Default 2.csv` become `Default 2` and `Default 2 2`; without the same-batch
check both would be named `Default 2` and one view would silently replace the
other.
A per-file view is made only while `File` is an accepted category. A user who
skipped or reclassified it has no filter to build views from.
Two things this does not do. Page cycling still steps onto a hidden item and
draws an empty grid: `[`/`]` in a per-file view can reach another file's
measure (improvise-yd2t; the Default view already behaved this way). And a
CSV with its own `File` column has it overwritten by the merge's, which is the
existing behaviour of `merge_csvs`.
### The wizard's back path is a history stack
`advance` pushes the step it leaves; `previous` pops it. Keeping the path
@@ -684,10 +822,35 @@ and a blank string are **absences, not names**; stringifying them used to invent
an item literally called "null" (improvise-ych). Every other value keeps its
printed form.
A record missing a value for an accepted Category field has no place in the grid
and is dropped; `build_model_with_warnings` counts it against the *first*
category it is missing and reports the total, because without a warning that
loss is invisible in a thousand-row import (improvise-5j9).
**A record with no value for a category keeps its cell, without that
coordinate** (improvise-8jk6.2). Records are sparse: a `CellKey` names only the
coordinates it has. So a missing field, a `null` and a blank string all mean
"no coordinate here". None of them makes an item: not `"null"`, not `""`, and
no synthetic placeholder. The same rule covers labels, which used to become an
item named `""`. The user's words, 2026-09-25: "this limit in import should be
fixed, it dates to when this wasn't sparse." That limit was improvise-5j9's
rule: such a record "has no place in the grid" and was dropped, counted and
reported as `Skipped N of M records missing category values`. 5j9's warning
made the loss visible, and the loss is now gone, so the warning went with it.
The only record still skipped is one that is not a JSON object.
Where such a record shows up is decided by the axes, not by the import. The
test is `a_record_with_no_coordinate_is_drawn_only_where_that_category_is_off_the_grid`:
- **The category on Row or Column:** no position holds the record. Its other
coordinates may still draw a row, but the cell under the category's items is
empty, and the column total leaves it out.
- **The category on Page:** no selection shows it.
- **The category on None:** it is aggregated like any other record.
- **Records mode:** its row shows a blank cell in that category's column.
- **`.improv`:** its Data line omits the category, and it round-trips.
- **Drill:** a drill that does not constrain the category reaches it.
**No "(none)" position, by the user's decision (2026-09-25).** They were asked
whether Row, Column or Page should give such records a place of their own. Their
answer: "I think leaving it out for now is fine, maybe a UI element somewhere
that shows the number of records underneath the current view?" That indicator
is improvise-gtn7.10, and it is not an import concern.
### Colliding records are a question, not a default (improvise-cgqt)
@@ -704,7 +867,9 @@ made it choose too. It chose `Refuse` and points at `:import`.
**Detection comes before the build and shares the build's placement.** Where a
record lands is `Plan::place`: its categories, then its date parts, then its
labels, then its measure values, or a reason it is dropped. `collisions()` and
labels, then its measure values. Only a record that is not an object is
dropped, since a missing value is no coordinate rather than a reason to drop
(improvise-8jk6.2). `collisions()` and
`build_model_with_warnings` both call it, so the question the wizard asks and
the thing the build does cannot disagree. Neither has a second walk of the
record. **A collision is a cell that more than one record writes**, meaning a