Skip to content

perf+fix: faster feature builds — deterministic git fast-paths, scope-proportional planning, LLM shape normalization#103

Merged
AbirAbbas merged 3 commits into
mainfrom
perf/deterministic-git-fast-paths
Jul 21, 2026
Merged

perf+fix: faster feature builds — deterministic git fast-paths, scope-proportional planning, LLM shape normalization#103
AbirAbbas merged 3 commits into
mainfrom
perf/deterministic-git-fast-paths

Conversation

@AbirAbbas

@AbirAbbas AbirAbbas commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

One PR for the full improvement round that came out of a live full-pipeline benchmark (1h43m for a 2-file task on openrouter/deepseek/deepseek-v4-pro, 88 agent calls, plus a second run that crashed at 48m). Three commits, three concerns:

  1. fix: normalize LLM-emitted scalar shapes on issue data — the 48m run died with 6 DAGState validation errors on checkpoint reload: the replanner emitted acceptance_criteria as a plain string, assignment bypassed validation (validate_assignment off), and the poisoned checkpoint failed re-validation. Now every LLM ingestion boundary (fix generator, replanner update/add paths) coerces str → [str] (ensure_str_list / normalize_issue_dict; Go StrList with tolerant UnmarshalJSON), and the schemas carry mode="before" validators so even pre-fix poisoned checkpoints still load. The exact incident is a regression test.
  2. perf: deterministic git fast-paths — LLM-driven git bookkeeping was 23 of 88 calls (7 workspace setups, 7 mergers, 9 cleanups), ~20 minutes. Worktree creation and cleanup are always mechanical; merges are mechanical when conflict-free. They now run as plain git (git_fast_path.py / internal/dag/gitfast.go): the merger agent only ever sees branches that actually conflict, and single-branch levels no longer request integration tests (per-issue tests already covered them). deterministic_git=true by default; false restores the agent-driven path, and any fast-path failure falls back to the agent — behavior never gets worse. Branch/worktree naming matches prompts/workspace.py exactly, so fast-path and agent-created state stay interchangeable (resume, mixed fallbacks, cleanup).
  3. perf(prompts): scope-proportional decomposition — the sprint planner turned the 2-file task into 7 issues with 12 acceptance criteria, and the fix generator's "exactly ONE failed criterion per fix issue" rule turned one verification failure into 7 fix issues (each a full coder+review cycle). The sprint planner now carries an explicit sizing rubric (a 1–3 file change is ONE issue; calibration table; 3–5 ACs per issue), and the fix generator groups failures sharing a root cause or files into one fix issue. Go prompt mirrors updated; golden fixtures regenerated from the Python truth.

Test plan

  • tests/test_llm_shape_normalization.py + Go strlist_test.go / normalize_llm_shapes_test.go — coercion at every ingestion boundary, tolerant re-validation, and a regression test reproducing the exact crashed-checkpoint reload
  • tests/test_git_fast_path.py + go/internal/dag/gitfast_test.go — naming parity with the agent convention, clean-merge/conflict-abort semantics (integration branch left clean), conflict-only agent handoff, resume reuse, cleanup, flag-off and fallback paths
  • Existing executor suites unchanged and green (fallback keeps scripted-agent tests working)
  • Full suite at CI parity (no .env): 1079 passed, 1 skipped + compileall clean; Go gofmt/build/vet/-race with GOWORK=off
  • Live before/after benchmark on the same task/model: 1h43m39s / 88 calls / 7 issues → 41m18s / 34 calls / 2 issues, verification passed all 10 ACs, zero git-agent calls — full numbers in the comment below

🤖 Generated with Claude Code

AbirAbbas and others added 3 commits July 20, 2026 16:21
Live-build incident (deepseek-v4-pro): generate_fix_issues returned fix
issues whose acceptance_criteria was a bare string. The raw dicts bypass
validation (DAGState.all_issues is list[dict]), the executor's attribute
assignment onto IssueResult.final_acceptance_criteria bypasses Pydantic,
and the next DAGState re-validation (replanner context / checkpoint
reload) failed with six list_type errors — killing a 48-minute build with
zero deliverable after cleanup deleted the fix branches.

Fix, both stacks: coerce str -> [str] at every LLM ingestion boundary
(fix generator output, replan updates/additions) and make the re-validating
models tolerant (IssueResult.final_acceptance_criteria, SplitIssueSpec and
PlannedIssue list fields; Go StrList + asStringSlice string case) so old
poisoned checkpoints still load. Internal trace: exec_20260720_110500_m36lxxcq.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live benchmark: 23 of 88 agent calls in a full build were LLM-driven git
bookkeeping (7 workspace setups, 7 mergers, 9 cleanups) — ~20 minutes of a
103-minute run. Worktree creation and cleanup are always mechanical; merges
are mechanical when conflict-free. Run them as plain git: the merger agent
now only sees branches that actually conflict, and single-branch levels no
longer request integration tests (per-issue tests already covered them).

deterministic_git=true by default on BuildConfig/ExecutionConfig; false
restores the fully agent-driven path, and any fast-path failure falls back
to the agent, so behavior never gets worse. Branch/worktree naming matches
prompts/workspace.py exactly. Mirrored 1:1 in Go.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…enerator

Live benchmark: the sprint planner turned a 2-file change into 7 issues
with 12 acceptance criteria, and the fix generator's 'exactly ONE failed
criterion per fix issue' rule turned one verification failure into 7 fix
issues — each costing a full coder+review cycle. The sprint planner now
gets an explicit sizing rubric (1-3 file change = one issue; calibration
table; 3-5 ACs per issue), and the fix generator groups failures that
share a root cause or files into one fix issue. Go prompt mirrors + golden
fixtures regenerated from the Python truth.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AbirAbbas

Copy link
Copy Markdown
Collaborator Author

Live before/after benchmark

Same task (add a retry decorator + tests to a small Python repo), same model (openrouter/deepseek/deepseek-v4-pro, open_code runtime), same config, fresh repo copy each run:

Before (main) After (#102 + #103)
Wall clock 1h 43m 39s 41m 18s (2.5×)
Agent calls 88 34
Issues planned 7 (12 ACs) 2
Git agent calls (setup/merge/cleanup) 23 0 — all deterministic
Replanner churn 1 full replan (~33 min) none
API cost ≈$4.50 ≈$1.31
Outcome 6/7 issues, 5 tests 2/2 issues, verification passed all 10 ACs, 6 tests

Evidence the fast paths engaged: the delivered branch's history contains the fast path's merge(level-N): <branch> commits, and the reasoner timeline has zero run_workspace_setup / run_merger / run_workspace_cleanup calls. A separate pre-fix run of main also crash-failed at 48m on the exact bug #102 fixes (internal trace: exec_20260720_110500_m36lxxcq), so the before-side is arguably worse than the table shows.

Remaining time is planning (~10m, unchanged) and coder/review iterations (model-bound) — the structural overhead this PR targets is gone.

@AbirAbbas AbirAbbas changed the title perf: deterministic git fast-paths + scope-proportional planning perf+fix: faster feature builds — deterministic git fast-paths, scope-proportional planning, LLM shape normalization Jul 21, 2026
@AbirAbbas
AbirAbbas merged commit e75546a into main Jul 21, 2026
3 checks passed
@AbirAbbas
AbirAbbas deleted the perf/deterministic-git-fast-paths branch July 21, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant