Skip to content

fix: normalize LLM-emitted scalar shapes on issue data#102

Closed
AbirAbbas wants to merge 1 commit into
mainfrom
fix/normalize-llm-issue-shapes
Closed

fix: normalize LLM-emitted scalar shapes on issue data#102
AbirAbbas wants to merge 1 commit into
mainfrom
fix/normalize-llm-issue-shapes

Conversation

@AbirAbbas

Copy link
Copy Markdown
Collaborator

Summary

Fixes a build-killing crash found by live-benchmarking swe-planner.build on openrouter/deepseek/deepseek-v4-pro: generate_fix_issues returned fix issues whose acceptance_criteria was a bare string, and 18 minutes later the replanner's DAGState re-validation failed with six list_type errors — a 48-minute build died with zero deliverable (cleanup then deleted the fix branches). Internal trace: exec_20260720_110500_m36lxxcq.

The mechanics: raw issue dicts bypass validation (DAGState.all_issues is list[dict]), and the executor's attribute assignment onto IssueResult.final_acceptance_criteria bypasses Pydantic (validate_assignment is off), so the poison persisted into the checkpoint and only detonated on the next model re-validation. A second identical run happened to get lists from the model and passed — the failure is a coin flip on weak-model output shape.

Fix (both stacks)

  • Boundary normalization: str → [str] coercion on issue list fields at every LLM ingestion point — fix-generator output, replanner updated_issues/new_issues (normalize_issue_dict / dagutil.NormalizeIssueDict).
  • Tolerant re-validation: mode="before" validators on IssueResult.final_acceptance_criteria, SplitIssueSpec and PlannedIssue list fields; Go StrList type + asStringSlice bare-string case — so pre-fix poisoned checkpoints still load and Go stops silently dropping bare-string values.
  • Both executor assignment sites now coerce explicitly.

Test plan

  • tests/test_llm_shape_normalization.py — the exact incident as a regression test (checkpoint reload with a poisoned completed_issue), boundary coercion, apply_replan normalization; Go mirrors in internal/dagutil + internal/schemas
  • Full suite at CI parity (no .env): 1079 passed, 1 skipped + compileall clean
  • Go: gofmt clean, build, vet, go test -race -count=1 ./... with GOWORK=off

🤖 Generated with Claude Code

Live-build incident (deepseek-v4-pro): generate_fix_issues returned fix
issues whose acceptance_criteria was a bare string. The raw dicts bypass
validation (DAGState.all_issues is list[dict]), the executor's attribute
assignment onto IssueResult.final_acceptance_criteria bypasses Pydantic,
and the next DAGState re-validation (replanner context / checkpoint
reload) failed with six list_type errors — killing a 48-minute build with
zero deliverable after cleanup deleted the fix branches.

Fix, both stacks: coerce str -> [str] at every LLM ingestion boundary
(fix generator output, replan updates/additions) and make the re-validating
models tolerant (IssueResult.final_acceptance_criteria, SplitIssueSpec and
PlannedIssue list fields; Go StrList + asStringSlice string case) so old
poisoned checkpoints still load. Internal trace: exec_20260720_110500_m36lxxcq.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AbirAbbas

Copy link
Copy Markdown
Collaborator Author

Folded into #103 — same commit (62ffbd0) is already the first commit of that branch, so everything from this round ships as one PR.

@AbirAbbas AbirAbbas closed this Jul 21, 2026
@AbirAbbas
AbirAbbas deleted the fix/normalize-llm-issue-shapes branch July 21, 2026 02:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant