fix(reviewer): require evidence chain and multi-round consistency

[wangke19]

Problem

The self-dev loop for #1 produced a reviewer output (self-dev-artifacts/issue-1-reviewer-deepseek.txt) that cannot actually justify its PASS verdict:

{
  "verdict": "PASS",
  "files_affected": [...],
  "summary": "Implementation meets all acceptance criteria: ...",
  "feedback": []
}

This is not a review — it is a confirmation. The Loop Agent design (per ARCHITECTURE.md and README.md:101-111) calls for two things this output lacks:

1. No evidence chain

The reviewer never saw the actual code. The prompt in cmd/selfdev/main.go:133-161 hands the reviewer a text summary of what was implemented and pre-marks every acceptance criterion with [x]:

Acceptance criteria from issue #1:
- [x] createOrUpdatePR makes real HTTP call to GitHub API
- [x] Idempotent: existing PR detection
- [x] task.PRURL populated
- [x] Error handling: 401 -> blocked, 5xx -> retryable
- [x] Unit tests with mocks pass
- [x] No breaking changes to handler interface

The reviewer is being asked to confirm an answer, not produce one. With feedback: [] and a generic summary, there is no way for the human (or the reconciler) to verify which line of code satisfies which criterion.

2. No multi-round consistency

The Loop Agent decision table requires 3 consecutive PASS before exiting the review loop:

Verdict	Count	Action
PASS	≥ 3	→ reconciling (exit loop)
PASS	< 3	→ re-review (continue)
FAIL	< 5	→ fixing (loop back)
FAIL	≥ 5	→ blocked (exit loop)

cmd/selfdev ran exactly one round and accepted the verdict. Per the design, a single PASS should trigger another review, not an exit. This is the weakest possible conclusion the design permits, and it is being treated as sufficient.

Impact

The self-dev demo currently proves nothing about whether the implementation is correct — only that the reviewer agrees with a self-assessment. If this is the bar for "the loop ran end-to-end", the Loop Agent thesis is not actually validated.

Proposed fix

Evidence chain

Reviewer prompt must:

1. Receive the actual git diff (or file contents), not a summary.
2. List acceptance criteria without [x] pre-marks.
3. Be forced to emit one feedback entry per criterion with this shape:

   {
     "criterion": "createOrUpdatePR makes real HTTP call to GitHub API",
     "status": "PASS | FAIL",
     "evidence": "internal/handlers/pr.go:42 — POST to /repos/{owner}/{repo}/pulls",
     "notes": "..."
   }

4. Verdict is PASS only if every criterion has status: PASS. Otherwise FAIL.

Multi-round consistency

Runner must:

1. Call reviewer in a loop with cap maxIterations=5.
2. Track passCount; on PASS, increment; on FAIL, reset to 0.
3. Exit when passCount >= 3 (consistency threshold) or iterations >= 5 (bounded termination).
4. Between rounds, do not mutate the diff — the same code is reviewed each round, so consistency means the reviewer reaches the same conclusion independently.

The FAIL→fix loop (routing to an executor to apply fixes) is out of scope for cmd/selfdev — that requires an executor agent and is the production conductor's job. Track separately.

Scope

• Affected files: cmd/selfdev/main.go (rewrite reviewer stage)
• May extract a reviewer package if the prompt/JSON schema logic gets large
• Artifacts: replace the single-round output with the multi-round trace

Acceptance criteria

• Reviewer prompt contains no [x] pre-marks on acceptance criteria
• Reviewer prompt embeds the actual git diff (or file contents)
• JSON schema requires {criterion, status, evidence, notes} per criterion
• verdict: PASS only when all criteria have status: PASS
• Runner loops until 3 consecutive PASS or 5 iterations
• Saved artifact shows all rounds, not just one
• Each round's evidence is non-empty (fails fast if model returns empty evidence)

Related

• feat(reconciler): implement real GitHub PR creation via API #1 reconciler PR creation stub (the original self-dev task)
• docs(selfdev): land real LLM-driven self-dev loop artifacts #3 selfdev artifacts PR (where the shallow review output lives)
• fix(agent): point glm.go and deepseek.go at Anthropic-compat proxies #4 agent layer endpoint migration (blocks real conductor from running this loop in production)