docs(tooling): cite each headless gap's ticket, and record the empty window and the tools package (improvise-1xfb.5.1)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdfEGXA7aJ6ftMsCc9PJze
This commit is contained in:
Claude (agent-stats)
2026-09-24 22:01:08 -07:00
committed by Edward Langley
parent 2dcf001b87
commit e32a4de406
2 changed files with 85 additions and 20 deletions
+53 -20
View File
@@ -1048,7 +1048,9 @@ pipeline is also an end-to-end run of improvise's headless product: `import`,
2. code-maat runs `coupling` (with `--min-revs`/`--min-shared-revs`), `abs-churn`,
`age` and `soc`. Each writes one CSV into the output directory, named after
the analysis. code-maat is pinned in `flake.nix` as a `fetchurl` of the
1.0.4 standalone jar, with a JRE, in both the dev and `ci` shells.
1.0.4 standalone jar, run by a JRE. It is in the dev shell and in the
`.#repo-stats-tools` package, but not in the `ci` shell: the JRE is about
0.6 GB on Linux, and every CI job enters that shell.
3. The script adds `churn.csv`, one row per (commit, file) with the date, the ISO
week, the path (`entity`, code-maat's own name for it), the area, the kind
(prose, code or other), and the lines added and deleted. It also adds two
@@ -1068,6 +1070,14 @@ pipeline is also an end-to-end run of improvise's headless product: `import`,
The `.improv` file is the artifact. It opens in the TUI, and CI uploads it with
the CSVs.
**A window in which no file changed builds no model.** Its commits are all
empty (`--allow-empty`), so there is no churn to import and nothing for
code-maat to analyse. The script writes `churn.csv` with its header alone, skips
code-maat and improvise, and the report says so. It exits 0. Running improvise
over an empty CSV was the other option. It was not taken, because it would test
only how improvise treats a CSV with no records, which is not what this pipeline
exists to exercise.
**No collisions, by construction.** A `churn.csv` record is keyed by commit and
file, which `--no-renames` makes unique. code-maat's records are keyed by
entity, by entity pair or by date. `File` separates the sources. So the import
@@ -1091,12 +1101,14 @@ and age) are read from code-maat's own CSVs, because improvise cannot sort rows
- **Areas**: first match wins, in the order of `AREAS` in the script.
`generated` comes first, so `harness/opfs-contract/Cargo.lock` counts as
generated and not as harness. A Rust test is a `.rs` file under a `tests/` or
generated and not as harness. `.beads/` and `site/` are generated too.
`docs/*.md`, `README.md` and `RELEASING.md` are "other docs", and `.codex/`
is agent docs and skills. A Rust test is a `.rs` file under a `tests/` or
`benches/` directory, or one named `tests.rs` or `*_test(s).rs`. A
`#[cfg(test)] mod tests` inside a source file counts as Rust src, because no
path can tell the two apart.
- **Prose** is `context/code`, `context/test-evidence`, other `context/`,
`repo-map`, and the agent docs and skills. **Code** is Rust src, Rust tests,
- **Prose** is `context/code`, `context/test-evidence`, other `context/`, other
docs, `repo-map`, and the agent docs and skills. **Code** is Rust src, Rust tests,
scripts/CI, and web and harness. `kb.pl`, generated files and other files are
neither.
- **The deletion ratio** is deleted lines over added lines. It is 0 when nothing
@@ -1128,8 +1140,11 @@ and age) are read from code-maat's own CSVs, because improvise cannot sort rows
`$GITHUB_STEP_SUMMARY`, and the directory is uploaded as the `repo-stats`
artifact.
- The report runs in `nix-build` because that job already builds
`.#improvise`, and `nix shell .#improvise` reuses that build. The job's
checkout fetches full history (`fetch-depth: 0`) for the report.
`.#improvise`. Its steps run under `nix shell .#repo-stats-tools`, which joins
that same build with code-maat, so nothing is built twice. It is one package
rather than two installables because `check-ci-graph.py` reads a counted
step as `nix shell <one> --command`. The job's checkout fetches full history
(`fetch-depth: 0`) for the report.
### Gaps in the headless product, and the workarounds used
@@ -1140,39 +1155,43 @@ names it.
1. **A forced `--category` value is still parsed as a number.** The commit
`00565796` imported as item `565796`, and `220944e2` imported as
`22094400.0`. Workaround: `churn.csv` writes commits as `g<sha>`.
improvise-8jk6.4.
2. **`import --formula` drops a formula that fails to parse, and exits 0.**
`--formula 'X = ('` saves a model with no formula and says nothing.
`add-formula` in a script refuses it and exits 1, so the pipeline's own
category formula is added there.
category formula is added there. improvise-8jk6.5.
3. **A record that lacks any category's value is skipped.** A merge of CSVs with
different headers, which is the user's sparse model, imports **no records**.
Workaround: every CSV is padded with `-` for the categories it lacks. The
padding shows up as a `-` item in each category.
padding shows up as a `-` item in each category. improvise-lpzu makes
the import sparse.
4. **An import that skipped records exits 0**, even when it skipped all of
them. It saves an empty model and prints only a warning.
them. It saves an empty model and prints only a warning. improvise-8jk6.6.
5. **`set-axis` and `set-page` accept names that do not exist, and exit 0.** An
unknown category, an unknown axis name (`rwo`) and an unknown item are all
silent no-ops.
silent no-ops. improvise-31oe.2.
6. **The command language does not read `|pipe quoting|`.**
`create-view |By area|` made a view named `|By`, and the second word was
ignored. Commands that take a fixed number of arguments ignore extra ones
without saying so. Workaround: `"double quotes"`.
without saying so. Workaround: `"double quotes"`. improvise-31oe.3.
7. **`create-view` puts every category it is not told about on Page**, with its
first item selected. That silently filters the view. Every view has to set
every category's axis, one `set-axis` line each.
every category's axis, one `set-axis` line each. improvise-31oe.4.
8. **A view cannot hide a category's column.** A per-file records view shows
every category and measure, including the `-` padding.
every category and measure, including the `-` padding. improvise-31oe.5.
9. **`export` of a records view writes the editor's `+cat` and `+` columns.**
improvise-8jk6.7.
10. **`export` writes display text.** It uses the view's number format, so the
import's default `,.2` gives `"17,512.00"`. Workaround: `set-format .0` on
every exported view, and `FORMAT .3` on the ratio formulas.
improvise-8jk6.8.
11. **Nothing sorts.** Items stay in first-seen order, so weeks come out
newest first. A top-N table cannot be produced, so code-maat's rankings are
read from its CSVs.
read from its CSVs. improvise-31oe.6.
12. **`--extract` has no ISO week.** It has year, month and quarter. Workaround:
`churn.csv` carries a `week` column.
`churn.csv` carries a `week` column. improvise-8jk6.9.
13. **`import --formula` targets only `_Measure`.** A formula on another
category needs `add-formula <category> …` in a script.
category needs `add-formula <category> …` in a script. improvise-8jk6.10.
14. **The saved model opens on the import's `Default` view**, which puts `File`
on rows and `area` on columns, and not on the view the script left active.
That is improvise-azhb.
@@ -1180,12 +1199,26 @@ names it.
folder import, and it will also make the per-file views. Both of the
script's steps for these are marked interim.
16. **There is no headless output but a file.** `export` writes a CSV, and the
tables are laid out in Python.
tables are laid out in Python. improvise-31oe.7.
17. **`create-view` of an existing name replaces that view, and exits 0.**
`Workbook::create_view` inserts into an `IndexMap`, so it overwrites without
a word. It matters here once folder import makes its own per-file views: the
script's interim `create-view` of the same name would replace them.
improvise-31oe.8.
### The suite
`scripts/repo-stats-test.sh` builds a fixture repository whose every number is
known, runs the whole pipeline, code-maat and improvise included, and asserts
improvise's exported tables. It runs in `nix-build`, which puts improvise on
`PATH`. Locally it runs `cargo build --bin improvise` when improvise is not on
`PATH`. The evidence and the rivals are in `context/test-evidence/tooling.md`.
improvise's exported tables. It runs in `nix-build`, which puts improvise and
code-maat on `PATH`. Locally it uses improvise from `PATH`, or else
`target/debug/improvise`, and it fails with a message when there is neither.
It does not build improvise itself. `checkers.sh` scrubs the environment to an
allowlist, and on macOS that removes what the linker needs: a `cargo build`
inside the suite failed with `library not found for -liconv`. Build first with
`cargo build --bin improvise`; `cargo test --workspace` also leaves the binary
there. The suite is the slowest in `checkers.sh`, at about 25 seconds, most of
it JVM start-up for code-maat's four analyses per run.
The evidence and the rivals are in `context/test-evidence/tooling.md`.
+32
View File
@@ -1847,3 +1847,35 @@ through `checkers.sh repo-stats`:
pipeline runs fail with exit 3.
- Months instead of ISO weeks: both week cases fail.
- A `warning:` from improvise ignored: the warning-stub case fails.
### A window of empty commits, and four faults the first fixture missed — `scripts/repo-stats-test.sh` (improvise-1xfb.5.1)
**What broke.** ChatGPT Codex found it on PR #313, and the adversarial review
reproduced it. When every non-merge commit in the window is `--allow-empty`,
the log has commit headers and no files. `build_dir` took `churn.csv`'s header
from `rows[0]`, raised `IndexError` and exited 1. That exit is outside the
pipeline's failure split: it is neither a number nor improvise's exit 3.
**Fix.** `churn.csv` has a fixed header. When no file changed, the script
skips code-maat and improvise, writes the header alone, and exits 0.
**Red first.** The new case is a second fixture repository whose only
in-window commit is empty. Restoring the `rows[0]` header fails its run case
with the traceback.
**The review's four surviving faults, each now killed:**
- **`--no-renames` dropped.** The fixture now renames `scripts/foo.sh` to
`scripts/bar.sh`. Without the flag git reports one `{foo.sh => bar.sh}` entity
with no lines changed, not a delete and an add. Four cases fail: the scripts/CI
row, the code row, the ratio row and the coupling count.
- **`crate_of` forgetting the root crate.** `src/main.rs` now co-changes with
both crates. The crossing list loses three pairs.
- **A failing `script` call ignored.** A third stub exits 0 for `import` and
refuses `script`. The pipeline then reads exports that were never written and
exits 1, not 3.
- **`.beads/` and `site/` left out of generated.** Both are in the fixture now.
The generated row and the other row both fail.
**Also counted now:** `docs/guide.md` as other docs and `.codex/config.toml` as
agent docs. The coupling count is 16, not 15: `web/app.js` and
`scripts/foo.sh` change together in two commits, the 06-04 one and the rename.