diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1dd44856..b295f781 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -146,13 +146,39 @@ jobs: # dependency crate each run. The NixOS runners substitute from titan's # harmonia first, and titan builds gh/main's `#improvise` hourly, so a PR that # leaves Cargo.lock alone rebuilds only the workspace crates it changed. + # + # The repo-stats pipeline runs here because it needs this job's binary: + # `.#repo-stats-tools` is that build plus code-maat and its JRE, which stay + # out of the ci shell every other job enters. + # Its suite gates; the report on a push to main never fails the build, and + # its step fails only when improvise refuses the pipeline's input + # (context/code/tooling.md section 7). Full history is for the report. nix-build: - name: Nix build (crate2nix) + name: Nix build (crate2nix) and the repo-stats pipeline runs-on: [self-hosted, nixos, microvm] steps: - uses: actions/checkout@v7 + with: + fetch-depth: 0 - name: nix build .#improvise run: nix build --print-build-logs .#improvise + - name: The repo-stats pipeline computes what it claims to + run: nix shell .#repo-stats-tools --command ./scripts/checkers.sh --ci nix-build + - name: Churn, coupling and doc-growth report (report only) + if: github.event_name == 'push' + continue-on-error: true + run: | + rc=0 + nix shell .#repo-stats-tools --command ./scripts/repo-stats --out repo-stats > repo-stats.txt || rc=$? + { echo '```'; cat repo-stats.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY" + exit "$rc" + - name: Upload the repo-stats model and its CSVs + if: github.event_name == 'push' + uses: actions/upload-artifact@v7 + with: + name: repo-stats + path: repo-stats + retention-days: 30 wasm: name: wasm32 check (browser bundle crates) diff --git a/context/code/tooling.md b/context/code/tooling.md index e70c874b..b8952421 100644 --- a/context/code/tooling.md +++ b/context/code/tooling.md @@ -1032,3 +1032,212 @@ changes. Two rivals were run: - reading every failure as stale fails the offline case (exit 1, wanted 2); - dropping `--locked` fails the stale case (exit 0, wanted 1). Without `--locked`, cargo quietly rewrites the lock and reports success. + +## 7. The repo-stats pipeline (improvise-1xfb.5.1) + +`scripts/repo-stats` reports churn, change coupling and doc growth over a window +of history: by default the last 7 days of `gh/main`, falling back to `HEAD`, +without merges. It reports and never gates. The user asked for code-maat to do +the analyses it has, and for improvise to be the engine for the rest. So the +pipeline is also an end-to-end run of improvise's headless product: `import`, +`script` and `export`. + +### The shape + +1. `git log --numstat --no-renames --no-merges` in code-maat's `git2` format. +2. code-maat runs `coupling` (with `--min-revs`/`--min-shared-revs`), `abs-churn`, + `age` and `soc`. Each writes one CSV into the output directory, named after + the analysis. code-maat is pinned in `flake.nix` as a `fetchurl` of the + 1.0.4 standalone jar, run by a JRE. It is in the dev shell and in the + `.#repo-stats-tools` package, but not in the `ci` shell: the JRE is about + 0.6 GB on Linux, and every CI job enters that shell. +3. The script adds `churn.csv`, one row per (commit, file) with the date, the ISO + week, the path (`entity`, code-maat's own name for it), the area, the kind + (prose, code or other), and the lines added and deleted. It also adds two + flags to code-maat's `coupling.csv`: `crosses-crate` and `agent-rules`. +4. `improvise import ` (improvise-lpzu) merges the directory's five CSVs + into **one** sparse model, `repo-stats.improv`. The merge's `File` category + names each source, and the import makes one view per file, named after its + stem and filtered to it. That is the user's ruling: "Records are already + sparse: the model can just leave null/empty values for columns not in a + given CSV file." A record keeps no coordinate in a category its file lacks. +5. `improvise script` runs the checked-in `scripts/repo-stats.improvise`. It + adds the prose-to-code formula, builds four views on top of the per-file + ones and `export`s each to `report/*.csv`. The four names (`By area`, + `By kind`, `Area by week`, `By day`) must never be a file's stem: + `create-view` of an existing name replaces that view silently (gap 17). The + suite's view count, 10, fails if one does. None of the report reads the + merged `Default` view, which can be empty for a typical code-maat pair + (improvise-8jk6.15). +6. The text report is those exports laid out as tables. The code-maat rankings, + doc sizes and the coupling flags are added to it. `--json` gives the same + data as JSON. + +The `.improv` file is the artifact. It opens in the TUI, and CI uploads it with +the CSVs. + +**A window in which no file changed builds no model.** Its commits are all +empty (`--allow-empty`), so there is no churn to import and nothing for +code-maat to analyse. The script writes `churn.csv` with its header alone, skips +code-maat and improvise, and the report says so. It exits 0. Running improvise +over an empty CSV was the other option. It was not taken, because it would test +only how improvise treats a CSV with no records, which is not what this pipeline +exists to exercise. + +**`--out` must hold no CSV the script did not write.** `improvise import .` +merges every CSV in the directory, so a stale or foreign one would silently +join the model. The script refuses one with exit 2 and names it. It refuses +rather than deleting, because deleting files in a directory the caller named is +the wrong default. Its own five CSVs are overwritten on every run, so a rerun +into the same `--out` works. + +**No collisions, by construction.** A `churn.csv` record is keyed by commit and +file, which `--no-renames` makes unique. code-maat's records are keyed by +entity, by entity pair or by date. `File` separates the sources. So the import +keeps its default, `--on-duplicate refuse`. A collision would be a bug in this +script, and the import refuses it. + +**What improvise computes and what it does not.** Everything that is a sum or a +ratio of sums is a view: +- churn by area, with `DeletionRatio = deleted / added` as a `_Measure` formula; +- prose, code and other, with `ProseToCode = prose / code` as a formula on the + `kind` category; +- area by ISO week; +- lines added per day from `churn.csv` next to code-maat's `abs-churn`. The two + columns must agree, and the suite checks that they do. + +Two kinds of output stay in Python. The ranked code-maat tables (coupling, soc +and age) are read from code-maat's own CSVs, because improvise cannot sort rows +(gap 11 below). Doc sizes are a property of files at one ref, not of the window. + +### Definitions + +- **Areas**: first match wins, in the order of `AREAS` in the script. + `generated` comes first, so `harness/opfs-contract/Cargo.lock` counts as + generated and not as harness. `.beads/` and `site/` are generated too. + `docs/*.md`, `README.md` and `RELEASING.md` are "other docs", and `.codex/` + is agent docs and skills. A Rust test is a `.rs` file under a `tests/` or + `benches/` directory, or one named `tests.rs` or `*_test(s).rs`. A + `#[cfg(test)] mod tests` inside a source file counts as Rust src, because no + path can tell the two apart. +- **Prose** is `context/code`, `context/test-evidence`, other `context/`, other + docs, `repo-map`, and the agent docs and skills. **Code** is Rust src, Rust tests, + scripts/CI, and web and harness. `kb.pl`, generated files and other files are + neither. +- **The deletion ratio** is deleted lines over added lines. It is 0 when nothing + was deleted, and blank when nothing was added. +- **A crate crossing**: both paths are in a crate (`crates//`, the root + crate's `src/`, `tests/`, `benches/`, `examples/` or `Cargo.toml`, or + `harness/opfs-contract/`), and the two crates differ. A pair with one side + outside every crate is not flagged. +- **Agent rules**: both paths are in the agent docs and skills area. Such a pair + means one rule stated in several places. +- **Gotchas** are the numbered items between `## Gotchas` and the next `---` or + `##`. The longest gotcha is measured in characters. Most gotchas are one long + line, so a line count would say little. +- **Test-evidence entries** are the `###` and `####` headings. + +### What fails, and what never does + +- **The numbers never fail anything.** No budget is checked. Budgets are a + separate decision under the compaction epic, improvise-1xfb.5. +- **improvise refusing the pipeline's input exits 3.** That covers a non-zero + exit from `import` or `script`, and any `warning:` on their stderr. The + warning rule is there because a skipped record is data loss that exits 0 + (gaps 3 and 4). It is a product regression, not a statistic. +- **In CI**, the suite runs on every PR in `nix-build` and gates like any other + suite. The report runs only on a push to `main`. Its step has + `continue-on-error`, so an improvise failure there marks the step failed and + the job and the deploy go on. Whether that failure should block the deploy + is a question for the user on the PR. The report appends to + `$GITHUB_STEP_SUMMARY`, and the directory is uploaded as the `repo-stats` + artifact. +- The report runs in `nix-build` because that job already builds + `.#improvise`. Its steps run under `nix shell .#repo-stats-tools`, which joins + that same build with code-maat, so nothing is built twice. It is one package + rather than two installables because `check-ci-graph.py` reads a counted + step as `nix shell --command`. The job's checkout fetches full history + (`fetch-depth: 0`) for the report. + +### Gaps in the headless product, and the workarounds used + +Each of these was found by running the pipeline, and each is a finding against +improvise's command and model layer. Where a workaround exists, the script +names it. + +1. **A forced `--category` value is still parsed as a number.** The commit + `00565796` imported as item `565796`, and `220944e2` imported as + `22094400.0`. Workaround: `churn.csv` writes commits as `g`. + improvise-8jk6.4. +2. **`import --formula` drops a formula that fails to parse, and exits 0.** + `--formula 'X = ('` saves a model with no formula and says nothing. + `add-formula` in a script refuses it and exits 1, so the pipeline's own + category formula is added there. improvise-8jk6.5. +3. **A record that lacks any category's value is skipped.** A merge of CSVs with + different headers, which is the user's sparse model, imports **no records**. + Closed by improvise-lpzu, whose sparse import this pipeline now uses. Until + it landed, every CSV was padded with `-` for the categories it lacked. A + related hole is still open: a record with no measure value stores nothing + (improvise-8jk6.14). +4. **An import that skipped records exits 0**, even when it skipped all of + them. It saves an empty model and prints only a warning. improvise-8jk6.6. +5. **`set-axis` and `set-page` accept names that do not exist, and exit 0.** An + unknown category, an unknown axis name (`rwo`) and an unknown item are all + silent no-ops. improvise-31oe.2. +6. **The command language does not read `|pipe quoting|`.** + `create-view |By area|` made a view named `|By`, and the second word was + ignored. Commands that take a fixed number of arguments ignore extra ones + without saying so. Workaround: `"double quotes"`. improvise-31oe.3. +7. **`create-view` puts every category it is not told about on Page**, with its + first item selected. That silently filters the view. Every view has to set + every category's axis, one `set-axis` line each. improvise-31oe.4. +8. **A view cannot hide a category's column.** A per-file records view shows + every category and measure in the model. improvise-31oe.5. +9. **`export` of a records view writes the editor's `+cat` and `+` columns.** + improvise-8jk6.7. +10. **`export` writes display text.** It uses the view's number format, so the + import's default `,.2` gives `"17,512.00"`. Workaround: `set-format .0` on + every exported view, and `FORMAT .3` on the ratio formulas. + improvise-8jk6.8. +11. **Nothing sorts.** Items stay in first-seen order, so weeks come out + newest first. A top-N table cannot be produced, so code-maat's rankings are + read from its CSVs. improvise-31oe.6. +12. **`--extract` has no ISO week.** It has year, month and quarter. Workaround: + `churn.csv` carries a `week` column. improvise-8jk6.9. +13. **`import --formula` targets only `_Measure`.** A formula on another + category needs `add-formula …` in a script. improvise-8jk6.10. +14. **The saved model opens on the import's `Default` view**, which puts `File` + on rows and `area` on columns, and not on the view the script left active. + That is improvise-azhb. +15. **No folder import.** Closed by improvise-lpzu (PR #314), which this + pipeline now uses. Until it landed, the script named the five files and + generated its own per-file views. +16. **There is no headless output but a file.** `export` writes a CSV, and the + tables are laid out in Python. improvise-31oe.7. +17. **`create-view` of an existing name replaces that view, and exits 0.** + `Workbook::create_view` inserts into an `IndexMap`, so it overwrites without + a word. Folder import makes a view per file stem, so a report view named + like a stem would replace that file's view. + improvise-31oe.8. + +### The suite + +`scripts/repo-stats-test.sh` builds a fixture repository whose every number is +known, runs the whole pipeline, code-maat and improvise included, and asserts +improvise's exported tables. It runs in `nix-build`, which puts improvise and +code-maat on `PATH`. Locally it uses improvise from `PATH`, or else +`target/debug/improvise`, and it fails with a message when there is neither. +The fallback does not check how fresh that binary is, so locally the suite can +test a `target/debug/improvise` older than the tree. Rebuild after changing +improvise. + +It does not build improvise itself. `checkers.sh` scrubs the environment to an +allowlist, and on macOS that removes what the linker needs: a `cargo build` +inside the suite failed with `library not found for -liconv`. That is a bug in +`checkers.sh`: its allowlist drops the Nix linker variables, and the +orchestrator has filed it separately. Build first with +`cargo build --bin improvise`; `cargo test --workspace` also leaves the binary +there. The suite is the slowest in `checkers.sh`, at about 25 seconds, most of +it JVM start-up for code-maat's four analyses per run. + +The evidence and the rivals are in `context/test-evidence/tooling.md`. diff --git a/context/test-evidence/tooling.md b/context/test-evidence/tooling.md index c6ea8088..305c172c 100644 --- a/context/test-evidence/tooling.md +++ b/context/test-evidence/tooling.md @@ -1811,3 +1811,107 @@ that failed would leave the owner-authored base at HEAD and fail the case too. `GIT_CONFIG_GLOBAL=/dev/null` and `GIT_CONFIG_NOSYSTEM=1`, and every `GIT_*` identity variable is unset first. So a caller's own identity cannot make the author or committer assertions agree by accident. + +### The repo-stats pipeline computes what it claims to — `scripts/repo-stats-test.sh` (improvise-1xfb.5.1) + +**What it pins.** The fixture repository has one commit before the window, a +side branch merged with `--no-ff`, a shrink in the next ISO week, and one file +in every area. So each number the report prints is known by construction. The +suite runs the real pipeline, code-maat and improvise included, and asserts +improvise's exported tables: each area's added, deleted and deletion ratio; +prose, code and the `ProseToCode` row; the week split; and the per-day totals. +It also checks the doc sizes and the coupling flags, and that an improvise that +refuses or warns makes the pipeline exit 3. + +**Two fixture choices are load-bearing.** +- The repo map's `4.` item sits after the `---` that closes the Gotchas + section, with no `##` heading between them. In an earlier fixture a `##` + heading followed the `---`, so the `---` rule could be deleted and the suite + stayed green. +- `churn.csv` must equal code-maat's `abs-churn` on every day. That compares + two independent derivations of one number, one of them improvise's + aggregation. The fixture has a binary-free, rename-free history, so the two + agree exactly. + +**Rivals**, each injected into the script or into `repo-stats.improvise` and run +through `checkers.sh repo-stats`: +- Rust src matched before Rust tests: the rust src and rust tests rows fail. +- `DeletionRatio = deleted / (added + deleted)`: six ratio rows fail. +- `ProseToCode = code / prose`: the `ProseToCode` row fails. +- `--no-merges` dropped: the commit count fails (6, wanted 5). +- The `---` end of the Gotchas section ignored: the gotcha count fails (4, wanted 3). +- A crate crossing as "the crates differ", so a crate against no crate counts: + the crate-crossing case fails. +- `kb.pl` counted as prose: the prose row and the ratio fail. +- The category padding removed: the import skipped every record, and both + pipeline runs failed with exit 3. That rival is retired: folder import + (improvise-lpzu) keeps sparse records, and the padding is gone. The suite + passes 41 of 41 without it. +- Months instead of ISO weeks: both week cases fail. +- A `warning:` from improvise ignored: the warning-stub case fails. + +### A window of empty commits, and four faults the first fixture missed — `scripts/repo-stats-test.sh` (improvise-1xfb.5.1) + +**What broke.** ChatGPT Codex found it on PR #313, and the adversarial review +reproduced it. When every non-merge commit in the window is `--allow-empty`, +the log has commit headers and no files. `build_dir` took `churn.csv`'s header +from `rows[0]`, raised `IndexError` and exited 1. That exit is outside the +pipeline's failure split: it is neither a number nor improvise's exit 3. + +**Fix.** `churn.csv` has a fixed header. When no file changed, the script +skips code-maat and improvise, writes the header alone, and exits 0. + +**Red first.** The new case is a second fixture repository whose only +in-window commit is empty. Restoring the `rows[0]` header fails its run case +with the traceback. + +**The review's four surviving faults, each now killed:** +- **`--no-renames` dropped.** The fixture now renames `scripts/foo.sh` to + `scripts/bar.sh`. Without the flag git reports one `{foo.sh => bar.sh}` entity + with no lines changed, not a delete and an add. Four cases fail: the scripts/CI + row, the code row, the ratio row and the coupling count. +- **`crate_of` forgetting the root crate.** `src/main.rs` now co-changes with + both crates. The crossing list loses three pairs. +- **A failing `script` call ignored.** A third stub exits 0 for `import` and + refuses `script`. The pipeline then reads exports that were never written and + exits 1, not 3. +- **`.beads/` and `site/` left out of generated.** Both are in the fixture now. + The generated row and the other row both fail. + +**Also counted now:** `docs/guide.md` as other docs and `.codex/config.toml` as +agent docs. The coupling count is 16, not 15: `web/app.js` and +`scripts/foo.sh` change together in two commits, the 06-04 one and the rename. + +### The pipeline on folder import — `scripts/repo-stats-test.sh` (improvise-1xfb.5.1) + +Once #314 merged, `improvise import ` replaced the file list, the `-` +padding and the generated per-file views. The suite needed no change of +expectations and still passed 41 of 41. That shows only that the numbers +agree. It does **not** show that the model is right: the suite could not tell +sparse from padded. The delta review re-added `pad_categories` and the suite +still passed 41 of 41, while the model then had a `|-|` item in 13 places and +the `churn` view seated `agent-rules` on rows and paged everything else. The +view case counted 10 views either way. + +**What separates them now:** +- **"the sparse model has no padding item"**: no line of the model names `|-|`. +- **"the churn view seats …"**: four cases, one per seat. `date` is on rows, and + `agent-rules`, `coupled` and `crosses-crate`, which are coupling-only, are on + `none`. The churn view puts `entity` on columns, not `area` as the review + guessed. What discriminates is `date` on rows and the coupling-only + categories on `none`, which a padded CSV cannot give, because it claims all + nine categories. + +**Rivals:** +- **Padding re-added** (the review's own injection): 5 cases fail. They are + the `|-|` case, with 161 lines naming it in the fixture model, and all four + seat cases. +- **A report view named like a file stem**: `By day` renamed to `churn` leaves + 9 views, and the view-count case fails. +- **The foreign-CSV guard removed**: "a foreign CSV in --out is refused with + exit 2" fails with exit 0. The guard is the answer to the review's + non-blocking finding that `import .` picks up any CSV in `--out`. It refuses + instead of clearing. Deleting files in a directory the caller named is the + wrong default, and the script's own five CSVs are overwritten on every run, + so a rerun into the same `--out` is still accepted. The same fixture run + writes JSON and then text into one directory, which shows that. diff --git a/flake.nix b/flake.nix index 47dfc078..46250d37 100644 --- a/flake.nix +++ b/flake.nix @@ -85,6 +85,17 @@ pkgs.jq ]; + # scripts/repo-stats runs code-maat (improvise-1xfb.5.1). The JRE is about + # 0.6 GB, so it is kept out of ciInputs; only `nix-build` takes it. + + codeMaatJar = pkgs.fetchurl { + url = "https://github.com/adamtornhill/code-maat/releases/download/v1.0.4/code-maat-1.0.4-standalone.jar"; + hash = "sha256-QoeuIDSQGERFD3aVR7xEl6DaGm0cf6b63IWHBeZ0O18="; + }; + codeMaat = pkgs.writeShellScriptBin "code-maat" '' + exec ${pkgs.jre_headless}/bin/java -jar ${codeMaatJar} "$@" + ''; + # Playwright's downloaded browsers are generic-Linux builds that cannot # start on NixOS, so nixpkgs' build is used. harness/package.json pins # @playwright/test to this playwright-driver's version: they must match. @@ -114,6 +125,7 @@ pkgs.global pkgs.universal-ctags beads.packages.${system}.default + codeMaat # Demo recording and release tooling pkgs.asciinema @@ -129,6 +141,13 @@ // { improvise = cargoNix.workspaceMembers.improvise.build; default = self.packages.${system}.improvise; + code-maat = codeMaat; + # What the repo-stats pipeline runs besides the ci shell: one + # installable, because check-ci-graph reads `nix shell --command`. + repo-stats-tools = pkgs.symlinkJoin { + name = "repo-stats-tools"; + paths = [self.packages.${system}.improvise codeMaat]; + }; }; # `nix flake check` builds every CI build: crane's checks plus the diff --git a/scripts/checkers.sh b/scripts/checkers.sh index 183678fc..8409e639 100755 --- a/scripts/checkers.sh +++ b/scripts/checkers.sh @@ -101,6 +101,8 @@ ci_suites() { case "$1" in build) echo assemble-stamp ;; js) echo node-test ;; + # Needs improvise and code-maat, which only this job has (tooling.md section 7). + nix-build) echo repo-stats ;; # check-harness-lock must never read "could not resolve" as "stale" (improvise-zs2r). kb) echo kb checkers pr-watch release-preflight pr-reviews fmt-workspaces bd-audit-ids check-harness-lock agent-git ;; *) return 1 ;; diff --git a/scripts/repo-stats b/scripts/repo-stats new file mode 100755 index 00000000..6444d2a8 --- /dev/null +++ b/scripts/repo-stats @@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +"""Churn, coupling and doc-growth report over a window of history. + +code-maat's analyses and this script's per-file churn go into one directory of +CSVs; improvise imports them into one model and computes the report's tables. +See context/code/tooling.md section 7. +""" + +import argparse +import csv +import datetime +import json +import os +import re +import subprocess +import sys +import tempfile + +# First match wins, so the order is part of the definition. +AREAS = [ + ("generated", re.compile(r"^(\.beads/|site/|Cargo\.nix$|roadmap\.org$)" + r"|\.(lock|cast)$|(^|/)package-lock\.json$")), + ("repo-map", re.compile(r"^context/repo-map\.md$")), + ("kb.pl", re.compile(r"^prolog-sessions/kb\.pl$")), + ("context/code", re.compile(r"^context/code/")), + ("context/test-evidence", re.compile(r"^context/test-evidence/")), + ("other context", re.compile(r"^context/")), + ("other docs", re.compile(r"^(docs/.*\.md|README\.md|RELEASING\.md)$")), + ("agent docs and skills", re.compile(r"^(AGENTS\.md|CLAUDE\.md|\.agents/|\.claude/|\.codex/)")), + ("scripts/CI", re.compile(r"^(scripts/|\.github/|nix/|flake\.nix$)")), + ("web and harness", re.compile(r"^(web|harness)/")), + ("rust tests", re.compile(r"(^|/)(tests|benches)/.*\.rs$|(^|/|_)tests?\.rs$")), + ("rust src", re.compile(r"\.(rs|pest)$")), +] +OTHER = "other" +PROSE = {"context/code", "context/test-evidence", "other context", "other docs", "repo-map", + "agent docs and skills"} +CODE = {"rust src", "rust tests", "scripts/CI", "web and harness"} +AGENT_RULES = "agent docs and skills" + +# One CSV per code-maat analysis, plus churn.csv. Each stem becomes an item of +# the model's File category, which the .improvise script pages on. +ANALYSES = ["coupling", "abs-churn", "age", "soc"] +CATEGORIES = ["commit", "date", "week", "entity", "area", "kind", "coupled", + "crosses-crate", "agent-rules"] +CHURN_HEADER = ["commit", "date", "week", "entity", "area", "kind", "added", "deleted"] +MEASURES = ["added", "deleted", "commits", "degree", "average-revs", "age-months", "soc"] +TABLES = ["by-area", "by-kind", "area-by-week", "by-day"] +HERE = os.path.dirname(os.path.abspath(__file__)) + + +class ImproviseFailed(Exception): + """improvise refused the pipeline's input: a product regression, not a statistic.""" + + +def area_of(path): + for name, rx in AREAS: + if rx.search(path): + return name + return OTHER + + +def kind_of(area): + return "prose" if area in PROSE else "code" if area in CODE else "other" + + +def crate_of(path): + """The crate a path belongs to, or None for a path in no crate.""" + m = re.match(r"^crates/([^/]+)/", path) + if m: + return m.group(1) + if path.startswith("harness/opfs-contract/"): + return "opfs-contract" + if re.match(r"^(src|tests|benches|examples)/", path) or path == "Cargo.toml": + return "improvise" + return None + + +def git(repo, *args): + return subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True, + check=True).stdout + + +def resolve_ref(repo, ref): + """The ref asked for; the default gh/main falls back to HEAD where it is absent.""" + for r in ([ref] if ref else ["gh/main", "HEAD"]): + out = subprocess.run(["git", "-C", repo, "rev-parse", "--verify", "--quiet", + r + "^{commit}"], capture_output=True, text=True) + if out.returncode == 0: + return r, out.stdout.strip() + sys.exit(f"repo-stats: cannot resolve {ref or 'gh/main or HEAD'}") + + +def git2_log(repo, sha, since): + """The log in code-maat's git2 format: the ref's history, without merges.""" + return git(repo, "log", sha, "--no-merges", f"--since={since}", "--numstat", + "--date=short", "--pretty=format:--%h--%ad--%aN", "--no-renames") + + +def is_header(line): + return line.startswith("--") and line.count("--") >= 3 + + +def churn_rows(log): + """One row per (commit, file). A commit is written `g`, because improvise + reads an all-digit or exponent-shaped sha as a number (tooling.md section 7).""" + rows, commit, date = [], None, None + for line in log.split("\n"): + if is_header(line): + _, commit, date, _ = line.split("--", 3) + elif "\t" in line: + a, d, path = line.split("\t", 2) + year, week, _ = datetime.date.fromisoformat(date).isocalendar() + area = area_of(path) + rows.append({"commit": "g" + commit, "date": date, + "week": f"{year}-W{week:02d}", "entity": path, "area": area, + "kind": kind_of(area), "added": 0 if a == "-" else int(a), + "deleted": 0 if d == "-" else int(d)}) + return rows + + +def write_csv(path, header, rows): + with open(path, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=header) + w.writeheader() + w.writerows(rows) + + +def read_csv(path): + with open(path, newline="") as f: + return list(csv.DictReader(f)) + + +def code_maat(logfile, analysis, out, *extra): + subprocess.run(["code-maat", "-l", logfile, "-c", "git2", "-a", analysis, + "-o", out, *extra], capture_output=True, text=True, check=True) + + +def flag_coupling(path): + """code-maat's coupling.csv, plus the two flags code-maat has no notion of.""" + rows = read_csv(path) + for r in rows: + a, b = crate_of(r["entity"]), crate_of(r["coupled"]) + r["crosses-crate"] = "yes" if a and b and a != b else "no" + both = area_of(r["entity"]) == area_of(r["coupled"]) == AGENT_RULES + r["agent-rules"] = "yes" if both else "no" + write_csv(path, ["entity", "coupled", "degree", "average-revs", "crosses-crate", + "agent-rules"], rows) + + +def build_dir(out, log, rows, min_revs, today): + """The directory improvise imports: churn.csv and one CSV per analysis.""" + with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as f: + f.write(log + "\n") + try: + revs = ["--min-revs", str(min_revs), "--min-shared-revs", str(min_revs)] + extra = {"coupling": revs, "age": ["--age-time-now", today]} + for a in ANALYSES: + code_maat(f.name, a, os.path.join(out, f"{a}.csv"), *extra.get(a, [])) + finally: + os.unlink(f.name) + flag_coupling(os.path.join(out, "coupling.csv")) + + +def improvise(out, *args): + exe = os.environ.get("IMPROVISE", "improvise") + p = subprocess.run([exe, *args], cwd=out, capture_output=True, text=True) + if p.returncode != 0 or "warning:" in p.stderr: + raise ImproviseFailed(f"improvise {' '.join(args)} exited {p.returncode}:\n" + f"{p.stderr}") + + +def run_improvise(out): + """Import the directory into one model, a view per file included, then build + the report's views and export them.""" + flags = [f"--category={c}" for c in CATEGORIES] + [f"--measure={m}" for m in MEASURES] + improvise(out, "import", ".", "--no-wizard", *flags, "--name=Repo stats", + "--formula=DeletionRatio = deleted / added FORMAT .3", + "-o", "repo-stats.improv") + os.makedirs(os.path.join(out, "report"), exist_ok=True) + improvise(out, "script", os.path.join(HERE, "repo-stats.improvise"), + "-f", "repo-stats.improv") + return {t: read_rows(os.path.join(out, "report", f"{t}.csv")) for t in TABLES} + + +def read_rows(path): + with open(path, newline="") as f: + return list(csv.reader(f)) + + +def gotchas(text): + """(number, lines, chars) for each numbered item of repo-map's Gotchas section.""" + out, cur, inside = [], None, False + for line in text.split("\n"): + if line.startswith("## "): + inside = line.startswith("## Gotchas") + continue + if inside and line.startswith("---"): + inside = False + if not inside: + continue + m = re.match(r"^(\d+)\. ", line) + if m: + cur = [int(m.group(1)), 0, 0] + out.append(cur) + if cur is not None and line.strip(): + cur[1] += 1 + cur[2] += len(line) + return [tuple(g) for g in out] + + +def doc_sizes(repo, sha): + files = [p for p in git(repo, "ls-tree", "-r", "--name-only", sha, "--", + "context").split("\n") if p.endswith(".md")] + show = lambda p: git(repo, "show", f"{sha}:{p}") + lines = {p: len(show(p).splitlines()) for p in sorted(files)} + gs = gotchas(show("context/repo-map.md")) if "context/repo-map.md" in files else [] + longest = max(gs, key=lambda g: g[2], default=None) + evidence = {p: sum(1 for l in show(p).split("\n") if re.match(r"^#{3,4} ", l)) + for p in sorted(files) if p.startswith("context/test-evidence/")} + return { + "context_lines": lines, + "repo_map": {"gotchas": len(gs), "longest": longest and + {"number": longest[0], "lines": longest[1], "chars": longest[2]}}, + "test_evidence": {"entries": sum(evidence.values()), "per_file": evidence}, + } + + +def report(repo, ref, since, min_revs, out): + name, sha = resolve_ref(repo, ref) + log = git2_log(repo, sha, since) + commits = sum(1 for line in log.split("\n") if is_header(line)) + today = git(repo, "show", "-s", "--date=short", "--format=%cd", sha).strip() + r = {"ref": name, "sha": sha, "since": since, "commits": commits, "out": out, + "docs": doc_sizes(repo, sha), "tables": {}, "coupling": [], "soc": [], "age": []} + rows = churn_rows(log) + write_csv(os.path.join(out, "churn.csv"), CHURN_HEADER, rows) + # Empty commits alone change no file: nothing to analyse, so no model is built. + if rows: + build_dir(out, log, rows, min_revs, today) + r["tables"] = run_improvise(out) + for a in ("coupling", "soc", "age"): + r[a] = read_csv(os.path.join(out, f"{a}.csv")) + return r + + +def table(rows): + """A CSV improvise exported, as aligned text; the first column left-aligned.""" + n = max(len(r) for r in rows) + widths = [max(len(r[i]) for r in rows if i < len(r)) for i in range(n)] + return [(" " + " ".join((c.ljust if i == 0 else c.rjust)(widths[i]) + for i, c in enumerate(r))).rstrip() for r in rows] + + +TITLES = { + "by-area": "Churn by area (improvise)", + "by-kind": "Prose, code and other; ProseToCode = prose / code (improvise)", + "area-by-week": "Lines added by area and ISO week (improvise)", + "by-day": "Lines added per day, churn.csv beside code-maat's abs-churn (improvise)", +} + + +def text(r, top): + out = [f"repo-stats: {r['ref']} ({r['sha'][:10]}), since {r['since']}, " + f"{r['commits']} non-merge commits"] + if not r["tables"]: + out += ["", "No file changed in the window, so no model was built."] + for t in TABLES: + if r["tables"].get(t): + out += ["", TITLES[t]] + table(r["tables"][t]) + out += ["", "Doc sizes (lines)"] + out += [f" {n:>6} {path}" for path, n in r["docs"]["context_lines"].items()] + rm, te = r["docs"]["repo_map"], r["docs"]["test_evidence"] + lg = rm["longest"] + out.append(f" repo-map gotchas: {rm['gotchas']}" + ( + f"; longest is {lg['number']} ({lg['lines']} lines, {lg['chars']} chars)" + if lg else "")) + out.append(f" test-evidence entries: {te['entries']}") + out += ["", f"Change coupling (code-maat; top {top})"] + for c in r["coupling"][:top]: + flags = [f for f, k in (("CROSSES-CRATE", "crosses-crate"), + ("AGENT-RULES", "agent-rules")) if c[k] == "yes"] + out.append(f" {c['degree']:>4}% {c['average-revs']:>4} revs {c['entity']}" + f" <-> {c['coupled']}" + (f" [{', '.join(flags)}]" if flags else "")) + out += ["", f"Sum of coupling (code-maat; top {top})"] + out += [f" {s['soc']:>6} {s['entity']}" for s in r["soc"][:top]] + out += ["", f"Code age in months (code-maat; first {top})"] + out += [f" {a['age-months']:>4} {a['entity']}" for a in r["age"][:top]] + return "\n".join(out) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--ref", help="the ref to report on (default gh/main, else HEAD)") + ap.add_argument("--since", default="7 days ago", help="git --since (default 7 days ago)") + ap.add_argument("--min-revs", type=int, default=5, + help="code-maat --min-revs and --min-shared-revs (default 5)") + ap.add_argument("--top", type=int, default=15, help="rows per ranked table") + ap.add_argument("--out", default="repo-stats", + help="the directory for the CSVs, the model and its exports") + ap.add_argument("--json", action="store_true", help="emit JSON instead of text") + ap.add_argument("-C", dest="repo", default=".", help="the repository (default .)") + a = ap.parse_args() + os.makedirs(a.out, exist_ok=True) + ours = {f"{n}.csv" for n in ["churn"] + ANALYSES} + foreign = sorted(f for f in os.listdir(a.out) if f.lower().endswith(".csv") and f not in ours) + if foreign: + print(f"repo-stats: {a.out} holds CSVs this script did not write, and " + f"`improvise import` would merge them: {', '.join(foreign)}", file=sys.stderr) + return 2 + try: + r = report(a.repo, a.ref, a.since, a.min_revs, os.path.abspath(a.out)) + except ImproviseFailed as e: + print(f"repo-stats: {e}", file=sys.stderr) + return 3 + print(json.dumps(r, indent=2) if a.json else text(r, a.top)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/repo-stats-test.sh b/scripts/repo-stats-test.sh new file mode 100755 index 00000000..2e138dbd --- /dev/null +++ b/scripts/repo-stats-test.sh @@ -0,0 +1,233 @@ +#!/usr/bin/env bash +# check_the_checker for scripts/repo-stats (improvise-1xfb.5.1): the whole pipeline, +# code-maat and improvise included, over a fixture repo whose every number is known. +# See context/test-evidence/tooling.md. +set -uo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +STATS="$here/repo-stats" +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT + +pass=0; fail=0 +ok() { printf ' ok %s\n' "$1"; pass=$((pass+1)); } +bad() { printf ' FAIL %s (%s)\n' "$1" "$2"; fail=$((fail+1)); } + +# The improvise under test: on PATH in CI, else this tree's debug build. No build +# here: checkers.sh scrubs the linker's environment (context/code/tooling.md §7). +if command -v improvise >/dev/null; then + IMPROVISE="$(command -v improvise)" +elif [ -x "$here/../target/debug/improvise" ]; then + IMPROVISE="$here/../target/debug/improvise" +else + echo "repo-stats-test: no improvise; run \`cargo build --bin improvise\` first" >&2 + exit 1 +fi +export IMPROVISE + +# $1 name, $2 jq expression over $work/out.json, $3 the value it must print +is() { + local got + got="$(jq -c "$2" "$work/out.json" 2>&1)" + [ "$got" = "$3" ] && ok "$1" || bad "$1" "got $got, wanted $3" +} + +# $1 name, $2 table, $3 row label, $4 the row's cells after the label, as JSON +row() { + is "$1" ".tables[\"$2\"][]|select(.[0]==\"$3\")|.[1:]" "$4" +} + +# $1 name, $2 fixed-string the text report must contain +says() { + grep -Fq -- "$2" "$work/out.txt" && ok "$1" || bad "$1" "no '$2' in the text report" +} + +export GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null +export GIT_AUTHOR_NAME=fixture GIT_AUTHOR_EMAIL=f@x GIT_COMMITTER_NAME=fixture GIT_COMMITTER_EMAIL=f@x +repo="$work/repo" +g() { git -C "$repo" "$@" >/dev/null; } + +# $1 path, $2 line count, $3 tag. Rewriting with a new tag replaces every line. +put() { mkdir -p "$(dirname "$repo/$1")"; seq -f "$3 %g" 1 "$2" > "$repo/$1"; } +# $1 date; commits everything staged or not. +commit() { + g add -A + GIT_AUTHOR_DATE="$1T12:00:00Z" GIT_COMMITTER_DATE="$1T12:00:00Z" g commit -q -m "at $1" +} + +git init -q -b main "$repo" + +# Before the window: must count nowhere. +put README.md 50 old; commit 2026-01-01 + +# Two commits co-change six files, so each pair couples at 100% over 2 revs. +# src/ is the root crate's, so it crosses a boundary with both crates/. +for f in crates/a/src/lib.rs crates/b/src/lib.rs; do put "$f" 10 x1; done +put src/main.rs 5 x1; put crates/a/tests/t.rs 4 x1 +put AGENTS.md 3 x1; put .agents/skills/wave/agent-rules.md 3 x1 +commit 2026-06-02 +for f in crates/a/src/lib.rs crates/b/src/lib.rs; do put "$f" 6 x2; done +put src/main.rs 5 x2; put crates/a/tests/t.rs 4 x2 +put AGENTS.md 3 x2; put .agents/skills/wave/agent-rules.md 3 x2 +commit 2026-06-03 + +# One file in each remaining area. The repo map is 13 lines. +put context/code/x.md 8 x3 +mkdir -p "$repo/context/test-evidence" +printf '### t_one\ntext\n#### t_two\n## not an entry\n### t_three\n' \ + > "$repo/context/test-evidence/y.md" +cat > "$repo/context/repo-map.md" <<'EOF' +# Map + +## Gotchas (read before editing) + +1. **One.** short. +2. **Two.** a much longer gotcha, with plenty of words in it to be the longest one + and a second line. + +3. **Three.** middling length. + +--- + +4. not a gotcha, because the section has ended +EOF +put context/plan.md 2 x3 +put prolog-sessions/kb.pl 5 x3 +put scripts/foo.sh 7 x3; put .github/workflows/ci.yml 1 x3 +put web/app.js 9 x3; put harness/x.js 2 x3 +put Cargo.lock 11 x3 +put crates/a/src/grammar.pest 3 x3 +put .beads/issues.jsonl 2 x3; put site/index.html 1 x3 +put docs/guide.md 2 x3; put .codex/config.toml 1 x3 +commit 2026-06-04 +g tag before-shrink + +# A side branch merged with a merge commit: the side commit counts, the merge does not. +g checkout -q -b side +put notes.txt 4 x5; commit 2026-06-05 +g checkout -q main +GIT_AUTHOR_DATE=2026-06-06T12:00:00Z GIT_COMMITTER_DATE=2026-06-06T12:00:00Z \ + g merge -q --no-ff -m merge side + +# A shrink in the next ISO week, so one area deletes and one row spans two weeks, +# and a rename, which --no-renames counts as a delete and an add. +put web/app.js 3 x4 +g mv scripts/foo.sh scripts/bar.sh +commit 2026-06-09 + +# $1 output dir, $2 "text" to also write the text report, rest: extra args. +run() { + local out="$1" text="$2"; shift 2 + "$STATS" -C "$repo" --since 2026-06-01 --min-revs 2 --out "$out" "$@" --json \ + > "$work/out.json" 2> "$work/err" \ + || { bad "repo-stats $* ran" "$(cat "$work/err")"; return 1; } + [ "$text" = text ] || return 0 + "$STATS" -C "$repo" --since 2026-06-01 --min-revs 2 --out "$out" "$@" \ + > "$work/out.txt" 2>&1 \ + || { bad "repo-stats $* ran (text)" "$(cat "$work/out.txt")"; return 1; } +} + +if run "$work/stats" text; then + is "no gh/main in the fixture, so the ref is HEAD" '.ref' '"HEAD"' + is "the merge commit is not counted, the side commit is" '.commits' '5' + row "rust src is .rs and .pest outside tests" by-area "rust src" '["45","25","0.556"]' + row "rust tests are the tests/ files" by-area "rust tests" '["8","4","0.500"]' + row "agent docs and skills take .codex" by-area "agent docs and skills" '["13","6","0.462"]' + row "docs/*.md is other docs" by-area "other docs" '["2","0","0.000"]' + row "context/code" by-area "context/code" '["8","0","0.000"]' + row "context/test-evidence" by-area "context/test-evidence" '["5","0","0.000"]' + row "repo-map is its own area" by-area "repo-map" '["13","0","0.000"]' + row "other context" by-area "other context" '["2","0","0.000"]' + row "kb.pl" by-area "kb.pl" '["5","0","0.000"]' + row "scripts/CI takes .github, and a rename is a delete and an add" \ + by-area "scripts/CI" '["15","7","0.467"]' + row "web and harness, and its deletion ratio" by-area "web and harness" '["14","9","0.643"]' + row "a lock file, .beads/ and site/ are generated" by-area "generated" '["14","0","0.000"]' + row "the pre-window README counts nowhere" by-area "other" '["4","0","0.000"]' + row "prose is the context areas, other docs and agent docs" by-kind prose '["43","6","0.140"]' + row "code is rust, scripts/CI and web" by-kind code '["82","45","0.549"]' + row "prose to code is prose over code" by-kind ProseToCode '["0.524","0.133",""]' + is "weeks are columns, first-seen first" '.tables["area-by-week"][0][1:]' '["2026-W24","2026-W23"]' + row "an area is split by ISO week" area-by-week "web and harness" '["3","11"]' + is "every day's churn.csv total equals code-maat's abs-churn" \ + '[.tables["by-day"][1:][]|.[1]==.[2]]|all' 'true' + row "a day's total is every file changed that day" by-day 2026-06-04 '["72","72"]' + is "gotchas end with their section" '.docs.repo_map.gotchas' '3' + is "the longest gotcha is measured over its lines" \ + '.docs.repo_map.longest|[.number,.lines]' '[2,2]' + is "test-evidence entries are ### and #### headings" '.docs.test_evidence.entries' '3' + is "coupling finds the 15 pairs of six files, and web/app.js with scripts/foo.sh" \ + '.coupling|length' '16' + is "files of two crates, the root crate's included, cross a boundary" \ + '[.coupling[]|select(.["crosses-crate"]=="yes")|[.entity,.coupled]|sort]|sort' \ + '[["crates/a/src/lib.rs","crates/b/src/lib.rs"],["crates/a/src/lib.rs","src/main.rs"],["crates/a/tests/t.rs","crates/b/src/lib.rs"],["crates/a/tests/t.rs","src/main.rs"],["crates/b/src/lib.rs","src/main.rs"]]' + is "two agent-rule docs are flagged" \ + '[.coupling[]|select(.["agent-rules"]=="yes")|[.entity,.coupled]|sort]' \ + '[[".agents/skills/wave/agent-rules.md","AGENTS.md"]]' + says "the text report flags a crate crossing" "[CROSSES-CRATE]" + says "the text report flags agent-rule docs" "[AGENT-RULES]" + says "the text report prints improvise's ratio row" "ProseToCode" + views="$(grep -c '^## View: ' "$work/stats/repo-stats.improv")" + [ "$views" = 10 ] && ok "the model keeps a view per file and the report's four" \ + || bad "the model keeps a view per file and the report's four" "$views views" + # Sparse, not padded: a padded merge has the same numbers and views, but a + # `-` item in every category and per-file views seated on the padding. + dashes="$(grep -c '|-|' "$work/stats/repo-stats.improv")" + [ "$dashes" = 0 ] && ok "the sparse model has no padding item" \ + || bad "the sparse model has no padding item" "$dashes lines name |-|" + churn_view="$(awk '/^## View: churn$/,/^$/' "$work/stats/repo-stats.improv")" + for seat in "date: row" "agent-rules: none" "coupled: none" "crosses-crate: none"; do + grep -qx -- "$seat" <<< "$churn_view" && ok "the churn view seats $seat" \ + || bad "the churn view seats $seat" "$(tr '\n' ' ' <<< "$churn_view")" + done +fi + +# An --out holding a CSV this script did not write is refused, since the folder +# import would merge it. The script's own CSVs, from a rerun, are not foreign. +mkdir -p "$work/foreign"; echo "x,y" > "$work/foreign/other.csv" +"$STATS" -C "$repo" --since 2026-06-01 --out "$work/foreign" > /dev/null 2>&1 +got=$? +[ "$got" = 2 ] && ok "a foreign CSV in --out is refused with exit 2" \ + || bad "a foreign CSV in --out is refused with exit 2" "exit $got" + +# The window is scoped to the ref: at the tag, the shrink has not happened. +if run "$work/at-tag" json --ref before-shrink; then + row "--ref scopes the log" by-area "web and harness" '["11","0","0.000"]' + is "--ref names the ref it read" '.ref' '"before-shrink"' +fi + +# A window whose only commits are empty changes no file: exit 0, and no model. +empty="$work/empty" +git init -q -b main "$empty" +repo="$empty"; put README.md 5 old; commit 2026-01-01 +GIT_AUTHOR_DATE=2026-06-02T12:00:00Z GIT_COMMITTER_DATE=2026-06-02T12:00:00Z \ + g commit -q --allow-empty -m empty +if run "$work/empty-out" text; then + is "an empty commit is counted" '.commits' '1' + is "and builds no tables" '.tables' '{}' + lines="$(wc -l < "$work/empty-out/churn.csv" | tr -d ' ')" + [ "$lines" = 1 ] && ok "churn.csv is written with its header alone" \ + || bad "churn.csv is written with its header alone" "$lines lines" + says "the text report says why there are no tables" "No file changed in the window" +fi +repo="$work/repo" + +# improvise refusing the pipeline is exit 3, whatever the numbers were. The +# third stub imports cleanly and refuses only the scripts. +mkdir -p "$work/bin" +printf '#!/usr/bin/env bash\necho "Parse error: no" >&2\nexit 1\n' > "$work/bin/refuses" +printf '#!/usr/bin/env bash\necho "warning: Skipped 3 of 3 records" >&2\nexit 0\n' \ + > "$work/bin/warns" +printf '#!/usr/bin/env bash\n[ "$1" = script ] || exit 0\necho "Parse error: no" >&2\nexit 1\n' \ + > "$work/bin/refuses-a-script" +chmod +x "$work/bin/refuses" "$work/bin/warns" "$work/bin/refuses-a-script" +for stub in refuses warns refuses-a-script; do + IMPROVISE="$work/bin/$stub" "$STATS" -C "$repo" --since 2026-06-01 --out "$work/$stub" \ + > /dev/null 2>&1 + got=$? + [ "$got" = 3 ] && ok "an improvise that $stub fails the pipeline with exit 3" \ + || bad "an improvise that $stub fails the pipeline with exit 3" "exit $got" +done + +printf '\nrepo-stats: %d passed, %d failed\n' "$pass" "$fail" +[ "$fail" = 0 ] diff --git a/scripts/repo-stats.improvise b/scripts/repo-stats.improvise new file mode 100644 index 00000000..ce08f088 --- /dev/null +++ b/scripts/repo-stats.improvise @@ -0,0 +1,87 @@ +# The views scripts/repo-stats builds over the model it imported, run by +# `improvise script` with the output directory as the working directory. +# Every `export` here is a table the text report prints. See +# context/code/tooling.md section 7. + +# The prose-to-code ratio is a formula over the kind category's own items. +add-formula kind "ProseToCode = prose / code FORMAT .3" + +# Churn by area, with the deletion ratio (a _Measure formula from the import). +create-view "By area" +switch-view "By area" +set-axis File page +set-page File churn +set-axis area row +set-axis _Measure column +set-axis kind none +set-axis week none +set-axis date none +set-axis commit none +set-axis entity none +set-axis coupled none +set-axis crosses-crate none +set-axis agent-rules none +toggle-prune-empty +set-format .0 +export report/by-area.csv + +# Prose, code and other, and the ratio of the first two. +create-view "By kind" +switch-view "By kind" +set-axis File page +set-page File churn +set-axis kind row +set-axis _Measure column +set-axis area none +set-axis week none +set-axis date none +set-axis commit none +set-axis entity none +set-axis coupled none +set-axis crosses-crate none +set-axis agent-rules none +toggle-prune-empty +set-format .0 +export report/by-kind.csv + +# Lines added per area per ISO week. +create-view "Area by week" +switch-view "Area by week" +set-axis File page +set-page File churn +set-axis area row +set-axis week column +set-axis _Measure page +set-page _Measure added +set-axis kind none +set-axis date none +set-axis commit none +set-axis entity none +set-axis coupled none +set-axis crosses-crate none +set-axis agent-rules none +toggle-prune-empty +set-format .0 +export report/area-by-week.csv + +# Lines added per day, from both sources side by side: this script's churn +# rows and code-maat's abs-churn. The two columns must agree. +create-view "By day" +switch-view "By day" +set-axis date row +set-axis File column +set-axis _Measure page +set-page _Measure added +set-axis area none +set-axis kind none +set-axis week none +set-axis commit none +set-axis entity none +set-axis coupled none +set-axis crosses-crate none +set-axis agent-rules none +toggle-prune-empty +set-format .0 +export report/by-day.csv + +switch-view "By area"