diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 46e534fc..cddd8c56 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -142,7 +142,7 @@ {"_type":"issue","id":"improvise-ls3y","title":"[bug] A wire-driven SwitchView does not push the view history; the effect does","description":"Found during improvise-kmgs (2026-09-19), pre-existing, out of that ticket scope.\n\napply_command SwitchView arm calls app.switch_to_view(name) DIRECTLY, while effect::SwitchView first pushes a ViewFrame onto view_back_stack and clears the forward stack. So a switch driven over the wire is NOT undoable by ViewBack, and a key-driven one is -- two spellings of one user action that differ in whether history records it.\n\nSurfaced while building a ViewBack session test: the test had to walk forward through the EFFECT to fill the stack, because the command path never fills it. That asymmetry is the tell.\n\nDecide which is right rather than blindly making them agree: history is arguably UI state (the model is shared, the UI is not), in which case the per-client stack should be fed by whichever door the client used -- and then the command path is the one that is wrong. Note improvise-jytm will make active_view per-client, which touches the same ground.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T22:11:30Z","created_by":"Edward Langley","updated_at":"2026-09-19T22:50:02Z","closed_at":"2026-09-19T22:50:02Z","close_reason":"Found again by Copilot on PR #11 and fixed in ee767f9, merged as eb0c271. apply_command performed Command::SwitchView itself, calling App::switch_to_view directly instead of running effect::SwitchView -- and the EFFECT is the only thing that pushes onto view_back_stack and clears view_forward_stack. It was also the last hand-rolled mutation left in apply_command, which sat badly against the PR own \"effects are the only mutation site\" thesis.\n\nWAVE 4 IS WHAT MADE IT BITE: ModelCommand::ViewBack pops exactly that stack, so a wire client could switch views and then be unable to go back, while the same two keypresses worked. Before ModelCommand there was no wire way to ask, so the missing push was invisible.\n\nDemonstrated first: a_wire_view_switch_records_history_the_way_a_key_switch_does failed with the document still on Second. Nothing else in the suite could see it -- switch_view_moves_the_document_and_the_client_together checks where a switch LANDED, not what it RECORDED.\n\nPart of Copilot suggestion was declined with reasons on the thread: the arm still returns a hand-written Dirty rather than Dirty::of(Model(LayoutGeometry)), because Dirty::of is a per-KIND over-approximation and a view switch is the one member of that kind which leaves the view list alone -- which view is ACTIVE is not in ViewListCache at all. switching_views_projects_the_axis_widgets_only pins that.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-dkqn","title":"gp-tree::fit should take one slice of rows, not parallel x/y slices","description":"Filed from Copilot PR #10 round 2 (2026-09-19). The PR agent DECLINED the literal request and was right to; this is the real remedy.\n\nCopilot asked five public scoring functions (mse, cost, fit, cv_regress/cv_classify, permutation_importance) to enforce their basic preconditions -- i.e. validate that the parallel x and y slices have matching lengths. That was declined on the grounds that validating a parallel length is exactly the ASSERTED form improvise-n3wt removed from prune_axis_items in the very same PR (make the invariant unrepresentable, do not check it), and that Result on five functions is what AGENTS.md means by keeping Option/Result at the boundaries. The reasoning was posted on the PR thread citing both.\n\nTHE FIX IS A TYPE CHANGE: give these functions one slice of rows rather than parallel x/y. That removes the zip truncation Copilot identified -- the thing that made mse wrong -- and answers the empty-input NaN in standardize/misclass_rate/mse in the same move, because an empty row set is then one shape rather than four independent edge cases.\n\nCONSTRAINT: wave 3 established byte-identical CLI output across nine scenarios including predictions and the saved model. That evidence must be re-established after this change, not assumed -- the whole point of that harness is that a refactor of the search cannot be trusted on a green suite alone.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T22:01:11Z","created_by":"Edward Langley","updated_at":"2026-09-19T22:11:20Z","closed_at":"2026-09-19T22:11:20Z","close_reason":"Row\u003cT\u003e { feat: Vec\u003cf32\u003e, target: T } in fit.rs -- generic in the target because the two supervised shapes differ only there (Row\u003cf32\u003e regression, Row\u003cusize\u003e class code). mse/cost/fit/misclass_rate/standardize/cv_regress/cv_classify/permutation_importance all take one slice, so the parallel lengths zip used to truncate silently cannot be built. Rejected: a Rows newtype with a checked constructor (validation wearing a type -- the thing the doctrine declines), a bare tuple slice, and Result on five signatures. The pairing is made at the boundary that BUILDS the data: preprocess walks one row index and the rows live inside Target, because a Dataset carrying rows and target separately just moves the zip up one level into every caller. Empty input answered once in mean_over -- no rows scored is no error found, not the 0.0/0.0 NaN that propagated through cost and sorted unpredictably against real costs.\n\nBIT-IDENTITY ESTABLISHED, NOT ASSERTED: known-answer vectors were captured from the UNMODIFIED code first -- raw f32::to_bits for every scoring path plus rng.next_u64() after each stochastic run, which is what pins the generator call order and count (SplitMix64::seed() returns the original seed and witnesses nothing). Those literals are now tests/scoring_known_answers.rs (6 tests) and pass unchanged. 37 rows is prime, so no fold count divides evenly and the short last fold is exercised. Nine-scenario CLI byte-identity re-established including model.json sha256. Note the CLI prints 3-4 decimals, which is exactly why the bit-level tests exist: the diff alone could not see a moved accumulation.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-t3ux","title":"cv_regress and cv_classify are O(n-squared)","description":"Filed from Copilot PR #10 round 2 (2026-09-19). Confirmed, not speculative: both do test.contains(i) inside a filter over 0..y.len(), in both loops, so the inner membership test is linear in the fold size.\n\nA bool mask over the row indices fixes it and changes no results. Wants a benchmark rather than a bare assertion of improvement -- and note that results must be bit-identical afterwards, since gp-tree fits are reproducible from a seed and any change in accumulation order would invalidate every recorded seed (see the SplitMix64 known-answer work, improvise-8x9o).\n\nPairs naturally with the row-slice reshape ticket; doing them together avoids touching these functions twice.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T22:01:11Z","created_by":"Edward Langley","updated_at":"2026-09-19T22:11:21Z","closed_at":"2026-09-19T22:11:21Z","close_reason":"complement(n, test) marks the fold in a Vec\u003cbool\u003e and filters; both cross-validators call it. Measured on the split alone (release, 5 folds): n=1000 84us -\u003e 15us (5.8x), n=4000 1.01ms -\u003e 45us (22.7x), n=16000 16.2ms -\u003e 134us (120.6x), n=64000 258ms -\u003e 482us (535.5x). Quadrupling n multiplies the scan by ~16 and the mask by ~3.6 -- the shape changed, not just the constant.\n\nBit-identity: the_training_split_is_what_the_contains_scan_gave asserts complement(20, test) equals the old (0..20).filter(|i| !test.contains(i)) index-for-index over every fold of k in {1,2,3,5,7} -- the equivalence directly, not inferred from a fit that happened to agree. The known-answer vectors from improvise-dkqn also pass unchanged.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"improvise-pczv","title":".improv section headers are a second, unquoted naming rule","description":"Found during improvise-i2m5 (2026-09-19), pre-existing and deliberately left alone.\n\nPipe quoting now has one home and a category or item named SUM is written |SUM| in every REFERENCE -- axis lines, item lists, page selections, hidden:, data coords. But section headers never went through quote_name on either side and still do not: improv.pest has category_section = { \"## Category: \" ~ rest_of_line ~ ... } and view_section likewise, and parse.rs::section_title reads them with as_str().trim(). So a name is quoted everywhere it is referenced and raw in its own header.\n\nThat is coherent for a whole-line token, and it is why the round trip still works. But it means '# Model Name', 'Initial View:', '## Category:' and '## View:' are a second naming rule with different escaping. It matters if a name with leading/trailing whitespace, an embedded newline, or a line that looks like another section marker is ever reachable -- today the writer cannot produce one, but nothing enforces that.\n\nDECIDE: either route section headers through quote_name/extract_name like every other name position (grammar change first -- improv.pest is the source of truth), or state in the grammar that the whole-line token is the intended contract and add a test pinning what happens to an awkward name. The current answer is pinned only in a test comment.","notes":"\n--- CONTEXT (2026-09-19) ---\nThe user has confirmed the pipe-quoting canon itself is fine (\"Pipe quoting is\nfine\"), so this ticket is NOT about whether to quote -- it is only about the\none position that still does not: section headers. Decide it on its own merits\n(whole-line token vs. quoted name), not as a re-litigation of the guard.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T21:21:39Z","created_by":"Edward Langley","updated_at":"2026-09-19T22:10:04Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"improvise-pczv","title":".improv section headers are a second, unquoted naming rule","description":"Found during improvise-i2m5 (2026-09-19), pre-existing and deliberately left alone.\n\nPipe quoting now has one home and a category or item named SUM is written |SUM| in every REFERENCE -- axis lines, item lists, page selections, hidden:, data coords. But section headers never went through quote_name on either side and still do not: improv.pest has category_section = { \"## Category: \" ~ rest_of_line ~ ... } and view_section likewise, and parse.rs::section_title reads them with as_str().trim(). So a name is quoted everywhere it is referenced and raw in its own header.\n\nThat is coherent for a whole-line token, and it is why the round trip still works. But it means '# Model Name', 'Initial View:', '## Category:' and '## View:' are a second naming rule with different escaping. It matters if a name with leading/trailing whitespace, an embedded newline, or a line that looks like another section marker is ever reachable -- today the writer cannot produce one, but nothing enforces that.\n\nDECIDE: either route section headers through quote_name/extract_name like every other name position (grammar change first -- improv.pest is the source of truth), or state in the grammar that the whole-line token is the intended contract and add a test pinning what happens to an awkward name. The current answer is pinned only in a test comment.","notes":"\n--- CONTEXT (2026-09-19) ---\nThe user has confirmed the pipe-quoting canon itself is fine (\"Pipe quoting is\nfine\"), so this ticket is NOT about whether to quote -- it is only about the\none position that still does not: section headers. Decide it on its own merits\n(whole-line token vs. quoted name), not as a re-litigation of the guard.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T21:21:39Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:34:01Z","closed_at":"2026-09-20T00:34:01Z","close_reason":"IT WAS A CORRUPTION BUG, not a tidiness ticket -- reachability was verified rather than assumed, and it is reachable end to end with no hand-construction.\n\nCLASS 1, the file becomes unloadable: RFC 4180 permits a quoted CSV header field containing a newline; csv::Trim::All trims a field EDGES but preserves an embedded break, and build_model calls add_category on the field verbatim. So a header \"Region\\nName\" imports cleanly, format_md writes ## Category: Region / Name across two lines, and parse_md fails. The user imports, works, saves, and cannot reopen. A header containing \\n## Data wrote a second SECTION MARKER.\n\nCLASS 2, worse because it is silent: a name with leading/trailing whitespace round-tripped SUCCESSFULLY AND WRONGLY. The header was written raw and the reader trims, so the category came back as \"Region\" while every data coordinate still said |Region | -- references were already quoted. The cells survive in the file, attach to nothing, and render blank with no error anywhere. Reached via JSON import keys.\n\nCLASS 3, found while closing the others: a name literally spelled |Region| was written bare, consistent only while the reader never looked at pipes -- which stops being true the moment headers accept a quoted form.\n\nDECISION: keep the whole-line contract, STATE IT IN THE GRAMMAR, and quote only what the line cannot carry. improv.pest gains header_name = { pipe_quoted ~ \u0026NEWLINE | line_name }, used by model_name/initial_view/category_section/view_section; format.rs::quote_header is the writing half and names exactly four cases -- empty, edge whitespace, embedded \\n or \\r, and already spelled |...|. EACH CLAUSE IS ONE THING THE READER DOES, which is what makes the list a derivation rather than a grab-bag.\n\nREJECTED routing headers through quote_name, and not as over-caution: it quotes every name that is not a bare identifier, so \"# Q3 Budget\" becomes \"# |Q3 Budget|\" and every existing document header is rewritten on next save for zero correctness gain -- the DEFAULT model name \"Imported Model\" included. The whole-line contract is a genuine property of the position (a header is a token, a reference is not); the fix was to state it and make it TOTAL, not delete it.\n\n\u0026NEWLINE is load-bearing twice: it decides the one ambiguous line (## Category: |a| and |b| is a whole-line name) and it makes the change backward compatible, since a pre-pczv header has no closing pipe at end of line and falls to line_name. 7 tests including a proptest over awkward names on a POPULATED model; 5 faults injected, all caught by a check naming the specific thing broken -- and the pre-existing grammar-walking generated_file_roundtrips fired on two of them, having picked up header_name for free, which is the grammar-as-source-of-truth principle paying out.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-n89z","title":"Three Projection mutations have no producer","description":"Wave-2 review (2026-09-19), LOW/design note. document_projections only ever sends CategoryTreeUpdate::Categories(..). CategoryTreeUpdate::Items, CategoryTreeUpdate::Axes and GridUpdate::Invalidated have no emitter anywhere in the tree.\n\ndesign-principles 'nothing speculative': they are consumer-side machinery justified by prose rather than by a producer. Either emit them (Items and Axes are the fine-grained updates the producer would want once category-tree traffic matters) or record in the protocol why they exist ahead of a producer -- Invalidated has a stated meaning ('contents stale, frame unchanged') that the diff-based producer never needs, which is a legitimate reason to keep it, but it should be written down.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T21:04:42Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:04:42Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-jxp3","title":"Sessions::connect rebuilds ViewState instead of cloning, so a reconnect snapshot is inexpressible","description":"From improvise-cqq (2026-09-19). The producer agent worked from a branch where ViewState was not Clone, so Sessions::connect CONSTRUCTS the Snapshot's view a second time rather than cloning the session's. At connect time the two are identical by construction, so there is no live defect -- but it means a snapshot of an ALREADY-RUNNING session cannot be expressed, which is exactly what a reconnect or a late-joining second client needs.\n\nViewState now derives Clone + PartialEq (added on the wave-1 review-fix branch so Snapshot could have a round-trip test), so the blocker is gone. Change connect to clone the session's view, and add a test that snapshotting a session which has already folded some commands reproduces that session's state -- not merely a default one. That test is the point: the current code would pass any test taken at connect time.\n\nRelates to invariant(bootstrap_is_a_compaction): the seed is already the diff-from-nothing via Dirty::EVERYTHING, so only the view half needs this.","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-09-19T20:56:07Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:53:05Z","closed_at":"2026-09-19T21:53:05Z","close_reason":"Fixed in the same commit (8475b2d, PR #10) as Copilot's second finding on that PR. The comment in Sessions::connect justified rebuilding the snapshot's view by claiming ViewState is not Clone -- which stopped being true when it gained the derive on the wave-1 branch so that Snapshot could have a round-trip test (gotcha 25). The snapshot now CLONES the mirror in both connect and rebase, which is also the correct shape: reseed is what decides what a freshly seeded mirror holds, and a client told anything else starts out disagreeing with the server's picture of it. That closes this ticket's actual point -- a snapshot of an already-running session is now expressible.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-i2m5","title":"improvise-io still carries its own pipe-quoting copy, without the keyword guard","description":"Follow-up from the wave-1 review fixes (2026-09-19). Pipe quoting now has one home -- improvise_formula::quote (quote_name/is_bare_name) -- and improvise-gp re-exports it. improvise-io/src/persistence/quoting.rs still has its own pub(super) is_bare_name/escape_pipe/quote_name/pipe_quote, and that copy lacks the keyword guard. It also holds a second copy of the UNQUOTING rule (unescape_pipe vs parser.rs::unquote_pipe).\n\nSwitching io to improvise_formula::quote_name is not purely mechanical: it would change .improv output for a category or item literally named SUM/WHERE/etc to |SUM|. That still parses (improv.pest accepts pipe_quoted), and the persistence round-trip tests will show the delta -- so the change is safe but visible, and someone should look at the diff rather than assume.\n\nDecide the keyword guard deliberately. It is NOT a correctness fix: probing showed the grammar accepts bare keywords in every position ('WHERE = A', 'X = SUM', 'SUM(WHERE)' all parse). It was kept in improvise-formula because that acceptance is an emergent property of PEG ordering rather than a stated rule, and a pipe-quoted name is an identifier everywhere with nothing to fall through. Cost is cosmetic.","notes":"\n--- CONFIRMED BY THE USER (2026-09-19) ---\n\"Pipe quoting is fine.\" The keyword guard as canonical behaviour is a USER\nDECISION, not an agent preference. So .improv writing |SUM| for a category or\nitem literally named SUM stays, and this is not to be revisited as\nover-caution by a later reader who notices that bare keywords happen to parse.\n\nThey do parse -- agg_call needs a ( and PEG backtracking falls through to\nref_expr -- but that is rule ordering rather than a stated rule, and a\n## Formulas line carries formula text verbatim, so a name format_md writes is\nhanded straight back to parse_formula. One namespace, one spelling rule.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-09-19T20:50:39Z","created_by":"Edward Langley","updated_at":"2026-09-19T22:10:02Z","closed_at":"2026-09-19T21:21:38Z","close_reason":"improvise-io/src/persistence/quoting.rs deleted entirely. improvise_formula::quote is now the ONE home and owns both halves: is_bare_name/quote_name (writing a name, keyword guard included), pipe_quote (unconditional, for .improv text cells that must not read back as numbers), and unescape_pipe_body (reading either back). parser.rs::unquote_pipe is now a 3-line shim over it, down from a 26-line duplicate.\n\nKEYWORD GUARD MADE CANONICAL, with a reason beyond taste: a ## Formulas line carries formula text verbatim, so a name format_md writes is later handed to parse_formula -- one namespace, so one spelling rule, and the one that does not lean on emergent PEG backtracking is the safer canon. USER-VISIBLE FORMAT DELTA, measured not assumed: a characterization test was written FIRST and confirmed passing on the old code, then was the only failure in the workspace after the switch. .improv now writes |SUM| for a category or item literally named SUM, on axis lines, item lists and data coords. Section headers are unchanged (see the follow-up ticket).\n\nUNQUOTING DIVERGENCE ANSWERED: the two old copies agreed on every input either grammar's pipe_inner can produce. They differed on exactly one unreachable input, a trailing lone backslash, where io returned it and parser.rs panicked via .expect(GRAMMAR_INVARIANT). The total (io) behaviour is canonical and pinned by a_trailing_lone_backslash_is_kept, on the grounds that a caller arriving by a non-grammar route should get its bytes back rather than a panic. 5 tests added.","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -165,7 +165,7 @@ {"_type":"issue","id":"improvise-1c9","title":"ExportPrompt mode shows no status-bar hint","description":"AppMode::hint_text() returns \"\" for AppMode::ExportPrompt, so when the user hits the export prompt the status bar goes blank instead of telling them what to type (a path) or how to cancel (Esc).\n\nFound while making hint_text exhaustive in improvise-2hi: the old match had a '_ =\u003e \"\"' arm that hid the gap. The exhaustive match now names ExportPrompt explicitly, and app.rs::only_export_prompt_and_quit_have_no_hint pins the current behavior — update that test when adding the hint.\n\nQuit is also blank, but correctly so: it is never rendered.\n\nFix: give ExportPrompt a hint along the lines of 'Enter:export Esc:cancel — type a file path'.","notes":"GROOMED 2026-09-19: still true, new home — AppMode and hint_text moved to improvise-protocol in improvise-cqi. The empty arm is crates/improvise-protocol/src/lib.rs:218 (ExportPrompt and Quit share it; Quit having no hint is fine).","status":"closed","priority":3,"issue_type":"bug","owner":"el-github@elangley.org","created_at":"2026-08-01T19:17:44Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:35:55Z","closed_at":"2026-09-19T20:35:55Z","close_reason":"Bug confirmed by a failing characterization test: only_export_prompt_and_quit_have_no_hint became only_quit_has_no_hint and failed with left: [EXPORT, QUIT]. Added the hint in improvise-protocol's AppMode::hint_text: 'Enter:export Esc:cancel — type a file path', with keys matching the actual ExportPrompt keymap (TextEntry + .commit(Enter, commit-export)). Quit keeps its blank arm and the doc-comment now says why.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"improvise-7na","title":"Decide whether fitted gp-tree models are persisted alongside the formula","description":"The emitted formula is enough to recompute existing cells, so the .improv file needs nothing extra for the common case. But scoring NEW rows — the --predict path in fit_csv — needs the encoding plan (which columns are categorical, their training-set distinct values, imputation means), which the formula text does not carry.\n\nDecide: (a) formula only, and refitting is the way to score new data; (b) store the gp_tree Model as a sidecar/JSON blob; (c) store the seed + config so the fit is reproducible without storing the tree. Option (c) is cheap and matches the format's plain-text character, but only works if the training data is unchanged. This is a format decision, so settle it before the fit command ships and starts writing files.","acceptance_criteria":"A decision is recorded with its rationale, and the .improv grammar either changes or is explicitly confirmed unchanged.","status":"open","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:21Z","created_by":"Edward Langley","updated_at":"2026-08-01T17:25:21Z","dependencies":[{"issue_id":"improvise-7na","depends_on_id":"improvise-izk","type":"blocks","created_at":"2026-08-01T10:26:23Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"improvise-vx3","title":"fit-measure command: fit a tree and add it as a formula","description":"The user-facing surface. A command (:fit-measure \u003ctarget\u003e or similar) that extracts features from the active view, runs the fit, emits the champion as formula text and adds it as a new measure — after which it is an ordinary improvise formula: recomputed by the fixed-point evaluator, editable in the formula panel, saved in the ## Formulas section, drillable.\n\nReport quality alongside it, not just the rule: cross-validated R²/RMSE against the predict-the-mean baseline, and the permutation feature-importance ranking. A decision tree that looks tidy but generalizes no better than the mean is worse than useless — it is a rule the user will believe. Per the command/effect split, the fit belongs in a Cmd (pure over CmdContext) with effects that add the formula and set status.","acceptance_criteria":"(1) A command fits and installs the formula. (2) The formula survives save/load and recomputes. (3) Fit quality vs baseline is reported. (4) Fitting is cancellable or bounded — a GA run must not freeze the TUI.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:21Z","created_by":"Edward Langley","updated_at":"2026-08-01T17:25:21Z","dependencies":[{"issue_id":"improvise-vx3","depends_on_id":"improvise-0ai","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-4s8","type":"blocks","created_at":"2026-08-01T10:26:22Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-izk","type":"blocks","created_at":"2026-08-01T10:26:20Z","created_by":"Edward Langley","metadata":"{}"},{"issue_id":"improvise-vx3","depends_on_id":"improvise-lzn","type":"blocks","created_at":"2026-08-01T10:26:21Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":4,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"improvise-4s8","title":"Extract a gp-tree feature matrix from a View's records","description":"Turn the active view's data into gp-tree's fixed-width f32 feature vectors: pick a target measure, use the remaining measures as numeric features and the regular categories as categorical features (one-hot, via the preprocessing promoted out of fit_csv). One row per distinct combination of the non-measure coordinates — which is what records mode already enumerates, so GridLayout's records path is the natural source rather than a new traversal.\n\nOpen questions worth settling here: whether hidden items and page filters should scope the training rows (the user's mental model is probably 'fit what I can see'), whether formula-derived measures may be used as features (they can, but a feature that already depends on the target makes the fit trivially perfect and useless — needs a guard), and what to do with Label categories, which are high-cardinality per-row text and would one-hot into thousands of columns.","acceptance_criteria":"(1) A function maps Model + View + target measure to (feature matrix, targets, Spec with names and ranges). (2) Spec ranges are supersets of the actual values — gp-tree's simplify prunes reachable branches otherwise, silently changing outputs. (3) A feature that is the target, or a formula depending on it, is rejected or excluded. (4) Tests over a small fixture model.","status":"open","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:20Z","created_by":"Edward Langley","updated_at":"2026-08-01T17:25:20Z","dependencies":[{"issue_id":"improvise-4s8","depends_on_id":"improvise-0yr","type":"blocks","created_at":"2026-08-01T10:26:10Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"improvise-4s8","title":"Extract a gp-tree feature matrix from a View's records","description":"Turn the active view's data into gp-tree's fixed-width f32 feature vectors: pick a target measure, use the remaining measures as numeric features and the regular categories as categorical features (one-hot, via the preprocessing promoted out of fit_csv). One row per distinct combination of the non-measure coordinates — which is what records mode already enumerates, so GridLayout's records path is the natural source rather than a new traversal.\n\nOpen questions worth settling here: whether hidden items and page filters should scope the training rows (the user's mental model is probably 'fit what I can see'), whether formula-derived measures may be used as features (they can, but a feature that already depends on the target makes the fit trivially perfect and useless — needs a guard), and what to do with Label categories, which are high-cardinality per-row text and would one-hot into thousands of columns.","acceptance_criteria":"(1) A function maps Model + View + target measure to (feature matrix, targets, Spec with names and ranges). (2) Spec ranges are supersets of the actual values — gp-tree's simplify prunes reachable branches otherwise, silently changing outputs. (3) A feature that is the target, or a formula depending on it, is rejected or excluded. (4) Tests over a small fixture model.","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:25:20Z","created_by":"Edward Langley","updated_at":"2026-09-20T00:34:49Z","closed_at":"2026-09-20T00:34:49Z","close_reason":"crates/improvise-gp/src/dataset.rs: feature_table / dataset turn a View visible records into the string table gp_tree::data::preprocess reads. THE SHAPE WAS NOT INVENTED -- records mode ALREADY IS a feature matrix (gotcha 1), a row is one coordinate tuple and each measure a column -- so it enumerates the same rows against the USER OWN view rather than through GridLayout, which would need _Index/_Dim on the axes and therefore a different view from the one being looked at.\n\nFOUR SELECTION RULES, each a place a wrong answer is silent:\n * rows scoped by View::filtering_cats() (Page UNION Filter), NOT categories_on(Axis::Page) -- gotcha 27, and the Page answer quietly fits against rows a drill excluded;\n * hidden items scope rows AND feature columns (\"fit what I can see\"), an explicitly named target exempt;\n * every regular category is a feature WHATEVER AXIS it sits on including Axis::None, because a coordinate is part of what makes a record distinct -- drop a None category and two rows get identical feature vectors with different targets; Label categories excluded by not being is_regular, as in records_columns, since per-row text one-hots into a perfect memoriser;\n * NO FORMULA-DERIVED VALUE ENTERS AT ALL -- a formula target is not a feature and is refused AS the target, one rule covering both halves with no dependency analysis. Membership read off model.formulas(), never off absence from the explicit _Measure item list, because a name can be BOTH.\n\nTWO INJECTIONS FOUND WEAK TESTS AND CHANGED THE WORK, which is the discipline paying out: the formula-feature injection first PASSED because the fixture Profit was only implied by its formula and so never reached the explicit item list, making the filter dead code from the test point of view -- the fixture now lists Profit as an explicit item AS WELL AS a formula target, the only configuration in which that rule is observable. And dropping the record sort left records_come_out_in_coordinate_order green (2 rows), so that test was DELETED rather than kept vacuous, with the ordering claim moved to the exact-table test doc comment plus an honest note that a single-process test cannot distinguish sorted from hashed-that-way-today.\n\nRecords are sorted by coordinate tuple because matching_cells walks a HashSet and f32 accumulation is order-dependent. Numbers reach the table via f64::to_string, never format_f64 (gotcha 8). A record with no target value is dropped with a warning -- keeping it flips preprocess whole target to CLASSIFICATION on one unparseable cell. 13 tests.","dependencies":[{"issue_id":"improvise-4s8","depends_on_id":"improvise-0yr","type":"blocks","created_at":"2026-08-01T10:26:10Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-lzn","title":"Seeded Rng implementation for gp-tree fits in improvise","description":"gp_tree::Rng is a trait the caller implements, deliberately, so evolution is reproducible and the crate pulls in no RNG. improvise has no PRNG dependency today. Decide between a small vendored deterministic generator (xorshift/PCG, ~20 lines, no new dep) and taking on rand. Reproducibility is not optional here: a fit writes a formula into the user's file, so re-running the same fit on the same data must produce the same rule, and a bug report must be reconstructible from the seed. Whatever is chosen, the seed should be visible to the user and recorded with the fit.","acceptance_criteria":"(1) An Rng impl exists for improvise fits. (2) Same seed + same data + same config yields an identical formula, asserted by a test. (3) The seed is surfaced, not hidden.","status":"closed","priority":3,"issue_type":"task","owner":"el-github@elangley.org","created_at":"2026-08-01T17:24:34Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:38:43Z","closed_at":"2026-09-19T20:38:43Z","close_reason":"Vendored SplitMix64 as gp_tree::SplitMix64 beside the Rng trait rather than adding a rand dependency -- the same generator was already written out three times, and the requirement is reproducibility, not statistical quality. Seed is surfaced, not hidden: SplitMix64::seed() keeps the starting value, fit_csv takes --seed (decimal or 0x hex) and prints it on every fit, and --save records it in data::Model::seed as #[serde(default)] so earlier models still load (verified by --predict against a pre-change baseline). Determinism tests pair the_same_seed_fits_the_same_formula with a_different_seed_fits_a_different_formula, so 'reproducible' cannot be satisfied by a search that never moved.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-0ai","title":"gp-tree: expose a fit/evolve entry point in the library","description":"The library exposes the GP operators — random_tree, mutate_at, crossover, subtree_at, graft_at, simplify, plus Population, GaConfig and breed — but no loop that drives them. The search itself lives in examples/fit_csv.rs, so every consumer reimplements generation-stepping, elitism, parsimony penalty and stopping. Add a fit entry point taking a dataset, a GaConfig and an Rng, returning the champion (simplified). Keep the k-fold cross-validation and permutation feature-importance reporting available too — those are what make the output trustworthy rather than just plausible.","acceptance_criteria":"(1) A public fit function evolves a champion from data + config + Rng. (2) fit_csv uses it. (3) Same seed and config produce the same tree.","notes":"GROOMED 2026-09-19 — STALE REF: examples/fit_csv.rs does not exist at that path; it is crates/gp-tree/examples/fit_csv.rs (gp-tree is a workspace member, not the root crate).","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:24:33Z","created_by":"Edward Langley","updated_at":"2026-09-19T21:23:04Z","closed_at":"2026-09-19T21:23:04Z","close_reason":"New crates/gp-tree/src/fit.rs (425 lines, 7 tests): FitConfig with Default = the CLI historical numbers, fit() returning the simplified champion across restarts, plus cv_regress/cv_classify/folds/standardize/permutation_importance/mse. Nothing in the module prints -- CV returns pooled prediction/actual pairs and importance a score per feature; tables stay in the CLI, the same boundary d44s drew. fit_csv.rs drops 787 to 598 lines. Premise verified before designing: the generation loop really was Trainer::fit in the example with pop/gens as function-local consts. Fault injected: the generation loop bounded to zero, i.e. a fit returning its seed tree unevolved. The two DETERMINISM tests stayed GREEN -- they would have been fooled, which is why they are not sufficient -- while a_fit_improves_on_the_tree_it_started_from and the_config_changes_the_fit_at_a_fixed_seed failed. Reproducibility verified end to end: same seeds produce the same fits after promoting the loop, predictions and saved model included, so RNG call order is provably unchanged.","dependencies":[{"issue_id":"improvise-0ai","depends_on_id":"improvise-0yr","type":"blocks","created_at":"2026-08-01T10:26:10Z","created_by":"Edward Langley","metadata":"{}"}],"dependency_count":1,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"improvise-0yr","title":"gp-tree: promote fit_csv preprocessing into the library","description":"examples/fit_csv.rs carries 1261 lines of genuinely reusable preprocessing that no library consumer can reach: per-column type inference (numeric/date/categorical), one-hot encoding of categoricals, mean-imputation with a col.missing 0/1 indicator so absence is learnable, and the Model struct pairing fitted trees with the encoding plan needed to re-encode fresh rows.\n\nimprovise needs all of this to turn a view's records into gp-tree's fixed-width f32 feature vector — categorical dimensions are the norm in a pivot model, not the exception. Move it behind a public API; the example becomes a thin CLI over the library.","acceptance_criteria":"(1) Type inference, encoding, imputation and Model are public API. (2) examples/fit_csv.rs is a CLI wrapper with no preprocessing logic of its own. (3) Existing fit_csv behavior unchanged — same output for the same CSV.","notes":"GROOMED 2026-09-19 — STALE REF: examples/fit_csv.rs does not exist at that path; it is crates/gp-tree/examples/fit_csv.rs (gp-tree is a workspace member, not the root crate).","status":"closed","priority":3,"issue_type":"feature","owner":"el-github@elangley.org","created_at":"2026-08-01T17:21:15Z","created_by":"Edward Langley","updated_at":"2026-09-19T20:38:43Z","closed_at":"2026-09-19T20:38:43Z","close_reason":"Preprocessing promoted into a public gp_tree::data module (534 lines): ColumnKind, ColumnSpec, Target, Model, Dataset, preprocess, encode_row, read_csv, parse_datetime and friends. examples/fit_csv.rs went 1261 -\u003e 787 lines and now holds only CLI concerns. Two deliberate shape changes: preprocess returns a Dataset struct instead of a positional 5-tuple, and build_date/encode_row share a date_parts helper. Behaviour preservation verified by captured baselines compared before/after -- demo fit, a mixed numeric/date/categorical CSV with a missing cell, the saved model JSON and a --predict run are byte-identical; the only difference anywhere is one panic message, now that it is library code. 12 new tests.","dependency_count":0,"dependent_count":3,"comment_count":0} diff --git a/context/repo-map.md b/context/repo-map.md index cec3bd8e..0cd08a4f 100644 --- a/context/repo-map.md +++ b/context/repo-map.md @@ -247,7 +247,6 @@ opfs.rs 253 / 0t OpfsFs: Storage over the browser's Origin P persistence/improv.pest 124 PEG grammar — single source of truth persistence/mod.rs 259 / 7t document lifecycle: save/load/gzip/legacy-JSON over Storage; facade re-exporting format_md/parse_md/export_csv* -persistence/quoting.rs 63 / 0t CL-style |…| quoting, shared by format and parse persistence/format.rs 350 / 11t format_md: Workbook → .improv text persistence/parse.rs 1378 / 42t parse_md: pest tree-walk → Workbook (+ grammar generator) persistence/export.rs 156 / 3t export_csv / render_csv