improvise/context/test-evidence/cli.md

Test evidence — the headless CLI (src/main.rs, tests/headless_cli.rs)

Keyed by exact test name, per design-principles §7. A renamed test orphans its entry; python3 scripts/check-comment-runs.py --check-evidence catches a heading that names no test.

---

tests/headless_cli.rs — the contract at the process boundary

improvise-q914 / improvise-j6x8 / improvise-z05h. The reasoning is context/code/cli.md; this file records what broke and what each test can see.

Why a spawned binary rather than a unit test. The three things under test are an exit code, a choice of stream, and whether a file was written — none of which exists below std::process::exit. Per design-principles §7 this is a boundary contract, so it is pinned once, slowly and honestly, and the command/cmd/ unit tests assume it. Measured on the orchestrator's machine the whole binary runs in ~0.5s, twelve process spawns included, which is well inside the per-binary budget.

What was broken. run_headless_commands applied effects and never read app.view_state.status_msg, and set exit_code only for parse errors. So a successful command and a refused one produced byte-identical output — nothing — and exit 0 for both. Measured at bf83294 before the fix: six of the twelve tests below failed.

a_refused_command_says_what_went_wrong

add-item Nonexistent Thing against a document that has no such category. Workbook::add_item's Err reaches status_msg and the effect does not set abort_effects, which is what makes this the load-bearing case: it is a genuine refusal that no flag in the codebase classifies as one.

Asserts the category's name appears, not merely that something was printed — "something was printed" is satisfied by a fixed ok on every command.

a_command_with_nothing_to_say_says_nothing

The rival this exists to kill, and it is the one the brief named: a fix that prints on the happy path and stays silent on the refusal. Injected as if said.trim().is_empty() { eprintln!("ok") } — the exact inversion — and it fails this test and a_refused_command_says_what_went_wrong, five tests in all. A test that only asked "was anything printed" passes that rival.

add-category Region is the input because it sets no status at all, so the assertion is about the writer rather than about this command's vocabulary.

an_aborted_batch_exits_non_zero_and_says_why

:w into a directory that does not exist. effect::Save reports through report_file_write, which sets the status and abort_effects (gotcha 21), so this is the one arm the exit code can honestly read. Before the fix this exited 0 with no output, which is what made improvise cmd … && next-step proceed over a failed write.

a_command_that_succeeds_while_reporting_still_exits_zero

The rival this exists to kill: making any emitted status a non-zero exit, which is the obvious way to widen the previous test and is wrong. effect::Save on success also writes a status, as does cmd/fit.rs — whose status carries the seed that reproduces the fit. So "it spoke" cannot mean "it failed".

The first assertion is a check-the-checker: if the save ever stops reporting, the test proves nothing and says so rather than passing quietly.

a_parse_error_still_exits_non_zero

The pre-existing arm, kept so the widening cannot silently drop it. Also asserts the message names the word typed, which the Unknown command arm reports post-alias-resolution.

what_a_command_says_goes_to_stderr_and_stdout_stays_empty

Pins the stream choice itself. Both halves are asserted: a rival that writes to both streams passes an stderr-only check.

a_script_reports_exactly_as_cmd_does

improvise script is the same function, so this cannot fail while cmd passes — which is the point. It is here because the ticket asked whether the hole was shared, and a claim that two paths are one is worth one cheap assertion rather than a sentence. The script also carries a # comment, so the comment-skipping arm is exercised on the way through.

a_run_that_changes_nothing_does_not_rewrite_the_document

move-cursor down is ViewOnly, so nothing is dirty. This test passed before the fix — a freshly written document re-serialises byte-identically — and it is kept as the control that says so. The discriminating one is below.

a_no_op_run_leaves_an_unnormalised_literal_alone

The reported instance. A hand-written document carrying 1e20 is rewritten as 100000000000000000000 by a run asked to change nothing, because persistence::save ran unconditionally. It fails before the fix where its sibling does not, which is exactly why both are here: the sibling cannot see a normalisation, only a reordering.

The first assertion checks the fixture still contains 1e20 before the run, so a future grammar change that rejects the literal fails loudly instead of making the test vacuous.

a_run_that_changed_the_document_still_writes_it

The regression the dirty-gate could have introduced. effect::Save sets dirty = false, so a gate reading the flag once at the end would see [add-item East, w other.improv] as unchanged and silently drop the edit from the -f document. Accumulating changed per command is what this asserts; a gate written the obvious way fails it.

the_cmd_help_does_not_advertise_a_language_it_refuses

Three clap doc strings said JSON three times over. Checks both subcommands, because script's <PATH> help carried the same claim and a fix to cmd alone reads as complete.

the_language_the_help_names_is_the_one_it_accepts

The other direction, and the reason the test above is not merely a spell check: it demonstrates the : command line works end to end and that the JSON form the help advertised is genuinely refused. Without this half, deleting the word "JSON" from the help would satisfy the check while the CLI still accepted neither language.

Scratch (helper)

A per-test temp directory named from the test's own tag and the process id, so the twelve tests cannot collide, with a Drop that removes it. Not tempfile::TempDir only because the fixture also has to be readable by a spawned child at a stable path.

seeded (helper)

Runs add-category Region to create the document. It asserts the file exists afterwards rather than trusting the seed, because every test that follows would otherwise misreport as "the command did nothing" when the seed was what failed.

a_script_can_move_the_cursor_and_the_next_command_sees_it

improvise cmd and improvise script resolved every command through CmdRegistry::parse — the pure arm, which builds CursorState::default() (row 0, col 0, zero rows, zero columns). The interactive arm, which builds CursorState::from_ctx(ctx), was never reached from a script, so every Move snapshotted a zeroed cursor at parse time, clamped to (0, 0) and emitted SetSelected(0, 0) — worse than a no-op, since it resets the cursor (improvise-6uva). The reasoning is context/code/cli.md section 7.

Reproduced at a9cad15 before the fix, on a seeded document with Region on row carrying North, South, East: move-selection 2 0 . hide-selected-row-item and a bare hide-selected-row-item both wrote hidden: Region/North. After the fix the first writes hidden: Region/East.

Why the test has two halves. Asserting only that the moved script hides East cannot separate "the move worked" from "the origin happens to be East". The second document asserts the origin is North, so the first assertion is discriminating, and its message says what breaks if that ever changes — three identical replays of a fixture whose moves are silent no-ops prove nothing, which is how this survived five months.

Two faults were injected, one per layer, and both fail this test:

• parse_segment_interactive calls ctx.registry.parse instead of .interactive — the bug itself, one level down. left: "hidden: Region/North" right: "hidden: Region/East".
• run_headless_commands builds the segment up front with parse_line rather than per segment against a live CmdContext — the composition half. Same failure. The unit test in parse.rs passes under this second one, which is why both tests exist.

a_script_cursor_move_survives_the_line_that_made_it

The same property across two -f arguments rather than one dot-separated line, because improvise cmd takes several and improvise script is one per line: the cursor is App state and a line boundary is not a reset.

a_segment_is_built_against_the_cursor_it_will_run_against

The unit test for the layer that owns the bug. It seats a CmdContext at (1, 0) in a two-by-two grid and asserts move-selection 0 1 executes to SetSelected(1, 1).

Its second half is what keeps it honest: it builds the same command through registry.parse and asserts that one gives SetSelected(0, 0). If the pure arm ever learns the cursor, the first assertion stops discriminating and this half fails first, saying so.

a_pure_commands_two_arms_agree_on_no_arguments

A second instance of the same shape, found by the first fix rather than predicted by the ticket. CmdRegistry::register_pure built an interactive arm that refused empty args while its own pure arm accepted them, so w with no path — save to the current document — parsed one way and refused the other. Once a script re-parsed through the interactive arm, improvise cmd 'w' and :w both broke; the suite caught it as a_failed_colon_write_lands_in_normal_where_its_error_is_drawn reporting ":w — w: this command requires arguments".

Both arms are now parse, which is what "needs no interactive context" was supposed to mean. Restoring the empty-args guard fails this test on the first expect: the interactive arm refuses a bare w the pure arm accepts.

segments_drops_comments_and_blanks

segments is the split half of parse_line_with, extracted so the headless loop can re-parse one segment at a time. It answers the same three cases the whole-line parser did — a # comment, a // comment and a blank line are no commands — plus the empty segment in a . . b.

an_unknown_segment_is_an_error_not_a_silent_skip

parse_segment_interactive returns Ok(None) for a segment with no word, which is a shape an unknown word must not be able to reach: a silently skipped command in a script is the partial-and-silent failure this repo keeps finding. The error also names the word, matching what CmdRegistry::parse does.

an_explicit_coordinate_still_beats_the_cursor

Found by the ChatGPT Codex review of PR #151 at head 64bcb82, and correct: the first version of improvise-6uva's fix introduced a regression it was not looking for. Four registry entries — clear-cell, yank, paste, drill-into-cell — were registered with register, and their interactive arms took |_args, ctx|: they discarded the arguments and used ctx.cell_key(). That was harmless while a script only ever reached the pure arm, which reads the coordinates. The moment the headless loop re-parsed interactively, clear-cell Region/East passed the syntax gate and then cleared whatever the cursor was sitting on — the origin, Region/North.

So improvise-6uva's own lesson had a third instance in its own fix: two construction arms that mean different things demote whichever caller reaches the weaker one, and here the "weaker" one was the arm that had the context. The four arms went through key_from_args_or_cursor: explicit coordinates win, the cursor is the fallback, which is also what the keymap path wants since a binding carries no arguments. That helper is gone since improvise-2bac: the cursor is now the registry's fallback, reached only when parse refused, so explicit coordinates win by construction (context/code/cli.md §8).

The fixture writes three cells and clears the one the cursor is not on, so the assertion separates "cleared the named cell" from "cleared the cursor cell" — clearing North by name would have passed against the broken code. Restoring the arg-discarding arm fails it: the cursor cell was cleared instead of the named one: ["Region=East = 3", "Region=South = 2"].

commit_cell_edit_writes_the_named_cell_and_spares_the_cursors

improvise-2bac, found by the post-merge adversarial review of PR #151: the fourth instance of that PR's own lesson. commit-cell-edit was registered with two free arms. The pure arm read <mode> <value> <coords>; the interactive arm read only <mode>, took the value from the edit buffer and the key from the cursor. Once #151 made scripts construct through the interactive arm, commit-cell-edit editing 5 Region/East passed the syntax gate and then committed an empty buffer to the cursor cell — deleting it — with exit 0 and nothing on stderr.

Reproduced at 40b8913 before the fix, through the real binary: the document came back as ["Region=East = 3", "Region=North = 1"] — South, the cell the cursor had been moved onto, was gone and East was unchanged.

The fixture writes three cells and moves the cursor to South in the same line before committing to East, so the cursor sits on neither the target nor the origin: a test with the cursor on the target is right by accident, and one that only asserts the target misses the destruction. It asserts all three.

Injected rivals, run: (a) contextualize for CommitAndAdvance overwriting key and value from the cursor and the buffer — fails this test and a_line_whose_gate_passes_runs_whole; (b) the registry dispatching to the fallback first whenever one exists (the pre-fix shape) — fails this test, a_line_whose_gate_passes_runs_whole, an_explicit_coordinate_still_beats_the_cursor and both registry unit tests below.

a_line_whose_gate_passes_runs_whole

improvise-2bac's second symptom, reported on the ticket: context/code/cli.md §7 said the whole-line parse_line gate preserved line atomicity. It did not, wherever the interactive arm refused an argument list the pure arm accepted. add-category Foo . commit-cell-edit editing 5 Foo/x on an empty document printed Parse error: commit-cell-edit: no cell at cursor after Foo had already been added and saved: half the line landed.

The assertion is on Parse error and on the committed cell, not on the exit code. The line exits 1 both before #151 and after this fix, for a different and pre-existing reason: the grid has no rows, so commit-cell-edit's advance emits AbortChain, and the CLI maps an aborted batch to exit 1 (context/code/cli.md §3). Asserting success would have made this test fail on the correct behaviour.

a_malformed_commit_line_is_refused_whole

The other half of atomicity, which must survive the fix: a segment the gate refuses (commit-cell-edit editing 5 — a value and no coordinates) refuses the whole line, so the preceding add-category Foo never lands. It passed before the fix and must keep passing; its job is to catch a "fix" that loosens the pure arm to make the gate agree.

commit_cell_edit_honours_an_explicit_value_and_cell_over_the_cursor

The unit-level form of the headless test, against CmdRegistry::interactive directly: cursor on Type=Food, Month=Jan, a decoy edit buffer, and an explicit 5 for Type/Clothing Month/Feb. It asserts the built command's key and value. Rival (a) above — a contextualize that overwrites the parsed fields — failed it. improvise-yf7p then removed contextualize altogether (context/code/cli.md §8), so rival (a) no longer compiles; the test stays as the behavioural statement of what the signature now guarantees.

every_argument_list_the_gate_accepts_the_run_accepts_too

Enumerates every registration (CmdRegistry::names, no hand-kept list) against every candidate argument list, and requires the interactive arm to accept whatever the pure arm accepts, in a context whose cursor sits on no cell — a category with no items. That context is what made the old commit-cell-edit arm refuse, so a context with a cell under the cursor would have passed the bug. The assertion that the cursor has no cell is checked first; a plain Workbook::new does not satisfy it, which is how the fixture came to add the empty category. Rival (b) fails it.

explicit_open_and_focus_arguments_beat_the_toggle

The fifth instance, named on improvise-2bac as plausible and not yet run: toggle-panel-and-focus <panel> false false ignored both flags and toggled, so on a closed panel it opened it. The fixture closes every panel first, because on an open-and-focused panel the toggle also closes it and the test would pass against the bug. Rival: execute ignoring target and always toggling — fails this test for every panel.

a_script_writes_the_same_record_whatever_terminal_ran_it

improvise-up8g. run_headless_commands read crossterm::terminal::size() into App::term_{width,height}, and CmdContext::visible_rows derives from the height, so page-scroll 1 . hide-selected-row-item over a 60-item row axis hid R13 at 24 rows and R40 at 60 — one script, two documents. It became live when #151 let a script's commands see the cursor (improvise-6uva); before that the cursor was always the origin and every run hid R01.

The test runs the same script against two copies of one document at 24x80 and 60x200 and asserts the same record is hidden. One size cannot see the bug, which is the ticket's own requirement. The size is driven through LINES and COLUMNS with the child detached from any terminal by setsid: crossterm asks /dev/tty first and falls back to tput, which honours those variables, so without the detach a developer's real terminal would override both sizes and the test would pass against the bug on a laptop and fail it only in CI.

Two guards against vacuity. The hidden record must not be R01, because a cursor that never moved agrees at every size (the pre-#151 shape, and the shape of a fix that re-broke 6uva). And something must be hidden at all. Failed at 9e78051 with R13 against R40. It cannot see a width leak, and does not need to: width moves only col_offset, which no command today turns into a written record.

a_commit_whose_advance_has_nowhere_to_go_still_exits_zero

improvise-c5ip. commit-cell-edit ends in EnterAdvance, which emits AbortChain when the cursor is already bottom-right; the CLI mapped last_batch_aborted() to exit 1, so a write that landed and was saved exited 1 with nothing on stderr. Two cases: the cursor moved to the last of three rows and the write named explicitly, and a commit on a grid that is empty until the same line creates it. The value is asserted first, so the test cannot pass by the write simply not happening. Failed at 487f0b6 on the exit assertion with South = 9 already written. Rival: keeping || last_batch_aborted() in main.rs beside the new refusal check — fails here, at the first exit assertion.

a_refusal_that_only_writes_a_status_exits_non_zero

improvise-udt2. add-item Nonexistent Thing writes the workbook's error to the status and does not abort, so it exited 0. It asserts the refusal was printed before asserting the exit, so it cannot pass on a run that failed for some other reason. Rival: AddItem writing status_msg directly instead of calling App::refuse — fails here. The two rivals the ticket named are the existing a_command_that_succeeds_while_reporting_still_exits_zero (a successful :w calling refuse fails it, measured) and a_fit_that_reports_its_seed_is_not_a_refusal.

a_refusal_marks_only_its_own_batch

A refusing batch followed by a reporting one. The second assertion is the one the default cannot supply: a flag that is never reset passes the first half and fails the second. Rival: deleting the reset in apply_effects — fails at the second assertion. Rival: AddItem not calling refuse — fails at the first.

a_chain_with_nowhere_to_go_aborts_without_refusing

AbortChain must abort — asserted, so the test cannot pass on a batch that did nothing — and must not refuse. Rival: AbortChain calling refuse — fails here, and fails the process-level c5ip test too.

a_failed_save_is_a_refusal_as_well_as_an_abort

The one place the two flags coincide, and the reason the old exit code looked right: a failed file write both stops the chain (gotcha 21) and refuses. It keeps an_aborted_batch_exits_non_zero_and_says_why meaningful now that the CLI reads only the refusal.

a_fit_that_reports_its_seed_is_not_a_refusal

udt2's second rival: fit-measure speaks on success, carrying the seed. It asserts the status mentions the seed first, so a fit that said nothing cannot pass. Rival: the success arm emitting refuse instead of set_status — fails here. The refusal arm is asserted in a_refusal_names_what_went_wrong_and_stores_nothing.

a_headless_import_with_duplicates_refuses_by_default_and_names_them

improvise-cgqt, CLI parity: import --no-wizard summed colliding records silently apart from a warning, as 2cw9 left it. Red before the flag existed, because the import succeeded. It asserts a non-zero exit, the colliding records named in the wizard's own wording (2 records (#1, #3): Region=East), the flag named, and no output file. It has two siblings. ..._sums_duplicates_when_told_to asserts = 30 and no Occurrence. ..._keeps_duplicates_separate_when_told_to asserts the category, = 10 and = 20, and no = 30. So the two flags cannot both be summing.

a_pipe_quoted_argument_is_one_argument_without_its_pipes

improvise-31oe.3. Red before the fix: the old tokenizer split on whitespace only, so create-view |By area| built CreateViewCmd(["|By", "area|"]). It asserts the whole built command, so a lexer that joined the words but kept the pipes fails as well as one that split them. The second case has two quoted arguments side by side, so a lexer that reads one pipe piece per line fails. Rival: bare_word pushing | as a plain character. Fails here and in a_coordinate_may_pipe_quote_either_half.

a_pipe_quoted_argument_reads_the_escapes_improv_does

The three escapes pipe_quote writes, \|, \\ and \n, each in its own case. Red before the fix. Rival: pipe_body returning the raw body without unescape_pipe_body. Fails here, at the first case, and nowhere else in the crate, which is why the case exists. A lexer that treated \| as a closing pipe would also fail the first case, because the pipe after b then opens a piece that never closes.

a_command_given_more_arguments_than_it_takes_is_refused

improvise-31oe.3's second half. Red before the fix: create-view By area parsed and dropped area. It asserts the refusal opens with the command's name and quotes the surplus word, and that the pipe-quoted spelling of the same line is the control that parses. Rival: MaxArgs::admit answering Ok always. Fails here, in no_command_silently_drops_an_argument_it_was_given, and in a_colon_line_naming_something_missing_refuses_on_the_status_line.

no_command_silently_drops_an_argument_it_was_given

The rule is one rule only if every registration obeys it, so this enumerates every registered name against every candidate argument list, on both arms. For each list a command accepts, it appends four surplus words. One would be absorbed by an optional argument (add-formula ["1"] plus one more is a valid two-argument call), and four is more than any optional span in the registry. The last word must be named in the refusal. The exceptions are stated, not inferred: the Any list, and three commands whose own shape refusal runs first. Rivals, each run: declaring add-item-in-group as Any fails here, naming the dropped list. Removing the check from the interactive arm alone fails here and in the : line test. A test of the pure arm alone would have missed that.

a_command_naming_something_the_document_lacks_is_a_refusal

improvise-31oe.2 and its siblings: set-axis, set-page, hide-item, show-item, toggle-group and switch-view, each naming a missing category, item, group or view. Red before the fix at the first row, set-axis Regoin column, with an empty status. Each row asserts the refusal flag, the missing name in the message, and that the document's .improv text and the active view are unchanged. The last two catch a refusal written after the write. The control is a_command_naming_what_the_document_has_is_not_refused, whose group row is the one that caught unknown_group reading only Category::groups: add_item_in_group registers no Group. Rivals, each run: unknown_item answering None fails at set-page ["Type", "Nowhere"]. unknown_category answering None fails at the first row. The switch-view check disabled fails at its row.

a_colon_line_naming_something_missing_refuses_on_the_status_line

The TUI path for both tickets, through ExecuteCommand and App::apply_effects, not through the command alone. It asserts the refusal flag, the name on the status line, and the mode back at Normal. Rivals: an unknown category accepted fails at :set-axis Regoin row, and the arity check removed from the interactive arm fails at :create-view By area.

an_unknown_category_axis_or_item_exits_non_zero_and_names_it

The three repros from improvise-31oe.2, at the process boundary. Red against the gh/main source with this test file: set-axis Regoin row exited 0 with an empty stderr. The control, set-page Region East, must exit 0. Without it, a fix that refused every set-page would pass.

a_pipe_quoted_name_reaches_the_document_whole

improvise-31oe.3's repro at the process boundary: the saved document must carry ## View: By area and no |By. The second half, create-view By area, must exit non-zero naming "area". Red against the gh/main source: the document carried the view |By.