perf+fix: faster feature builds — deterministic git fast-paths, scope-proportional planning, LLM shape normalization#103
Merged
Conversation
Live-build incident (deepseek-v4-pro): generate_fix_issues returned fix issues whose acceptance_criteria was a bare string. The raw dicts bypass validation (DAGState.all_issues is list[dict]), the executor's attribute assignment onto IssueResult.final_acceptance_criteria bypasses Pydantic, and the next DAGState re-validation (replanner context / checkpoint reload) failed with six list_type errors — killing a 48-minute build with zero deliverable after cleanup deleted the fix branches. Fix, both stacks: coerce str -> [str] at every LLM ingestion boundary (fix generator output, replan updates/additions) and make the re-validating models tolerant (IssueResult.final_acceptance_criteria, SplitIssueSpec and PlannedIssue list fields; Go StrList + asStringSlice string case) so old poisoned checkpoints still load. Internal trace: exec_20260720_110500_m36lxxcq. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live benchmark: 23 of 88 agent calls in a full build were LLM-driven git bookkeeping (7 workspace setups, 7 mergers, 9 cleanups) — ~20 minutes of a 103-minute run. Worktree creation and cleanup are always mechanical; merges are mechanical when conflict-free. Run them as plain git: the merger agent now only sees branches that actually conflict, and single-branch levels no longer request integration tests (per-issue tests already covered them). deterministic_git=true by default on BuildConfig/ExecutionConfig; false restores the fully agent-driven path, and any fast-path failure falls back to the agent, so behavior never gets worse. Branch/worktree naming matches prompts/workspace.py exactly. Mirrored 1:1 in Go. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…enerator Live benchmark: the sprint planner turned a 2-file change into 7 issues with 12 acceptance criteria, and the fix generator's 'exactly ONE failed criterion per fix issue' rule turned one verification failure into 7 fix issues — each costing a full coder+review cycle. The sprint planner now gets an explicit sizing rubric (1-3 file change = one issue; calibration table; 3-5 ACs per issue), and the fix generator groups failures that share a root cause or files into one fix issue. Go prompt mirrors + golden fixtures regenerated from the Python truth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
Live before/after benchmarkSame task (add a retry decorator + tests to a small Python repo), same model (
Evidence the fast paths engaged: the delivered branch's history contains the fast path's Remaining time is planning (~10m, unchanged) and coder/review iterations (model-bound) — the structural overhead this PR targets is gone. |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One PR for the full improvement round that came out of a live full-pipeline benchmark (1h43m for a 2-file task on
openrouter/deepseek/deepseek-v4-pro, 88 agent calls, plus a second run that crashed at 48m). Three commits, three concerns:fix:normalize LLM-emitted scalar shapes on issue data — the 48m run died with 6DAGStatevalidation errors on checkpoint reload: the replanner emittedacceptance_criteriaas a plain string, assignment bypassed validation (validate_assignmentoff), and the poisoned checkpoint failed re-validation. Now every LLM ingestion boundary (fix generator, replanner update/add paths) coercesstr → [str](ensure_str_list/normalize_issue_dict; GoStrListwith tolerantUnmarshalJSON), and the schemas carrymode="before"validators so even pre-fix poisoned checkpoints still load. The exact incident is a regression test.perf:deterministic git fast-paths — LLM-driven git bookkeeping was 23 of 88 calls (7 workspace setups, 7 mergers, 9 cleanups), ~20 minutes. Worktree creation and cleanup are always mechanical; merges are mechanical when conflict-free. They now run as plain git (git_fast_path.py/internal/dag/gitfast.go): the merger agent only ever sees branches that actually conflict, and single-branch levels no longer request integration tests (per-issue tests already covered them).deterministic_git=trueby default;falserestores the agent-driven path, and any fast-path failure falls back to the agent — behavior never gets worse. Branch/worktree naming matchesprompts/workspace.pyexactly, so fast-path and agent-created state stay interchangeable (resume, mixed fallbacks, cleanup).perf(prompts):scope-proportional decomposition — the sprint planner turned the 2-file task into 7 issues with 12 acceptance criteria, and the fix generator's "exactly ONE failed criterion per fix issue" rule turned one verification failure into 7 fix issues (each a full coder+review cycle). The sprint planner now carries an explicit sizing rubric (a 1–3 file change is ONE issue; calibration table; 3–5 ACs per issue), and the fix generator groups failures sharing a root cause or files into one fix issue. Go prompt mirrors updated; golden fixtures regenerated from the Python truth.Test plan
tests/test_llm_shape_normalization.py+ Gostrlist_test.go/normalize_llm_shapes_test.go— coercion at every ingestion boundary, tolerant re-validation, and a regression test reproducing the exact crashed-checkpoint reloadtests/test_git_fast_path.py+go/internal/dag/gitfast_test.go— naming parity with the agent convention, clean-merge/conflict-abort semantics (integration branch left clean), conflict-only agent handoff, resume reuse, cleanup, flag-off and fallback paths.env): 1079 passed, 1 skipped + compileall clean; Go gofmt/build/vet/-racewithGOWORK=off🤖 Generated with Claude Code