feat: goal evaluator + strategist (closes #10)

[Protocol-zero-0]

Summary

Implements Issue #10 / Phase 2 — two optional roles that let an evolution session stop on success instead of running to max_iterations:

• Goal Evaluator — after each accepted run, asks an LLM whether the mission is complete. On goal_reached=true the CLI exits 0 with halted=false, distinct from the hard-stop exit 3.
• Strategist — every N rounds, produces stage / next_milestone / taboo_directions and injects them into the next round's planner_input.json.

Hard-stop machinery (max_iterations / max_total_usd / max_total_tokens) is untouched. Both roles are off by default, so existing configs keep behaving the same.

Changes

File	Purpose
evolution_kernel/config.py	New GoalEvaluatorConfig, StrategistConfig; Roles gains goal_evaluator and strategist argv tuples
evolution_kernel/governor.py	run_once(..., strategy=...) merges strategy into planner_input.json
evolution_kernel/cli.py	_run_loop checks goal_reached after each accepted run, calls strategist every N rounds; new _check_goal_reached and _invoke_strategist helpers
roles/goal_evaluator.py	LLM role (same provider/model resolution as roles/planner.py)
roles/strategist.py	LLM role
tests/test_issue10.py	15 new tests
tests/fixtures/*	3 deterministic fixture roles for E2E test paths

Config

roles:
  goal_evaluator: ["python3", "roles/goal_evaluator.py"]
  strategist:     ["python3", "roles/strategist.py"]
goal_evaluator:
  enabled: true
strategist:
  enabled: true
  every_n_rounds: 3

Test plan

• 15 new unit tests pass: config parsing × 7, governor strategy injection × 2, CLI goal_reached path × 4, CLI strategist injection × 2
• Full suite 54/54 passing (39 existing + 15 new), no regressions
• Real-LLM end-to-end run (single iteration with ANTHROPIC_API_KEY) to confirm prompts produce parseable JSON

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds two optional LLM-driven roles for the evolution loop: a Goal Evaluator that allows early CLI exit (rc=0, halted=false) when the mission is judged complete, and a Strategist that injects high-level strategy into the planner every N rounds. Both roles are off by default, so existing behaviour is preserved.

Changes:

• New GoalEvaluatorConfig / StrategistConfig + Roles.goal_evaluator / Roles.strategist argv tuples in config.py, with validation for every_n_rounds.
• Governor.run_once now accepts an optional strategy and conditionally adds it to planner_input.json; CLI _run_loop orchestrates goal-evaluator check after accepted runs and strategist invocation every N iterations.
• New LLM role scripts roles/goal_evaluator.py and roles/strategist.py (planner.py-style), plus 3 deterministic test fixtures and 15 new unit tests in tests/test_issue10.py.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
evolution_kernel/config.py	Adds GoalEvaluatorConfig/StrategistConfig dataclasses, role argv slots, and parsers with validation.
evolution_kernel/governor.py	run_once gains optional strategy arg merged into planner_input.json.
evolution_kernel/cli.py	_run_loop adds goal-reached early exit and strategist scheduling; new _check_goal_reached / _invoke_strategist helpers.
roles/goal_evaluator.py	New LLM role; outputs {goal_reached, confidence, reason} JSON.
roles/strategist.py	New LLM role; outputs {stage, next_milestone, taboo_directions} JSON.
tests/fixtures/goal_evaluator_reached.py	Deterministic fixture returning goal_reached: true.
tests/fixtures/goal_evaluator_not_reached.py	Deterministic fixture returning goal_reached: false.
tests/fixtures/strategist.py	Deterministic fixture emitting a fixed strategy payload.
tests/test_issue10.py	15 new tests covering config parsing, strategy injection, goal-reached exit path, and strategist cadence.

Comments suppressed due to low confidence (3)

evolution_kernel/cli.py:314

• Both _check_goal_reached and _invoke_strategist swallow subprocess failures silently: a non-zero exit code, missing output file, or malformed JSON causes the helper to return False/None with no message and no recorded artifact. When a user enables these roles and they fail (e.g. missing dependency, API auth error, invalid JSON), the loop will simply behave as if the role was disabled, making the problem very hard to diagnose. Consider at minimum logging completed.stderr (and the failure reason) to stderr, similar to how other failure paths surface diagnostics.

    completed = subprocess.run(argv, text=True, capture_output=True, check=False)
    if completed.returncode != 0 or not output_path.exists():
        return False
    try:
        data = json.loads(output_path.read_text(encoding="utf-8"))
        return bool(data.get("goal_reached", False))
    except (json.JSONDecodeError, OSError):
        return False


def _invoke_strategist(cfg: EvolutionConfig, result, iteration: int) -> dict | None:
    input_path = result.run_dir / "strategist_input.json"
    output_path = result.run_dir / "strategy.json"
    input_data = {
        "mission": cfg.mission,
        "current_round": iteration,
        "latest_evaluation": dict(result.evaluation),
    }
    input_path.write_text(json.dumps(input_data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    argv = [
        *cfg.roles.strategist,
        "--input", str(input_path),
        "--output", str(output_path),
        "--worktree", str(result.run_dir),
    ]
    completed = subprocess.run(argv, text=True, capture_output=True, check=False)
    if completed.returncode != 0 or not output_path.exists():
        return None
    try:
        return json.loads(output_path.read_text(encoding="utf-8"))
    except (json.JSONDecodeError, OSError):
        return None

evolution_kernel/cli.py:290

• _check_goal_reached writes goal_eval_input.json and goal_evaluation.json into result.run_dir, but the governor finalizes the run directory contents inside run_once and returns. These post-hoc files are not visible to the planner's history view (built from previous run artifacts), and there's no record in the ledger summary that a goal evaluation was performed. For auditability, consider either having the governor own the goal-evaluator invocation (so it is part of the canonical run artifacts and history), or writing a separate marker that downstream tooling can inspect.

def _check_goal_reached(cfg: EvolutionConfig, result) -> bool:
    input_path = result.run_dir / "goal_eval_input.json"
    output_path = result.run_dir / "goal_evaluation.json"
    input_data = {
        "mission": cfg.mission,
        "latest_evaluation": dict(result.evaluation),
    }
    input_path.write_text(json.dumps(input_data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    argv = [
        *cfg.roles.goal_evaluator,
        "--input", str(input_path),
        "--output", str(output_path),
        "--worktree", str(result.run_dir),
    ]
    completed = subprocess.run(argv, text=True, capture_output=True, check=False)
    if completed.returncode != 0 or not output_path.exists():
        return False
    try:
        data = json.loads(output_path.read_text(encoding="utf-8"))
        return bool(data.get("goal_reached", False))
    except (json.JSONDecodeError, OSError):
        return False

roles/strategist.py:99

• If the LLM responds with a JSON snippet whose regex match parses to a non-dict (e.g. a list, number, or string), result will be set to that value and the subsequent result.setdefault(...) calls will raise AttributeError, leading to an unhandled exception from the role. After json.loads(...) succeeds, guard with isinstance(result, dict) before calling setdefault, and fall back to the default-shaped dict otherwise. The same pattern applies to roles/goal_evaluator.py lines 88–100.

    m = re.search(r"\{.*\}", text, re.DOTALL)
    result = None
    if m:
        try:
            result = json.loads(m.group())
        except json.JSONDecodeError:
            pass
    if result is None:
        result = {"stage": "unknown", "next_milestone": text[:200], "taboo_directions": []}

    result.setdefault("stage", "unknown")
    result.setdefault("next_milestone", "")
    result.setdefault("taboo_directions", [])

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Protocol-zero-0]

E2E verification (Scheme B) — both new LLM prompts PASS

Ran the two new role prompts through real Claude Sonnet 4.6 via `claude -p` (bypasses ANTHROPIC_API_KEY and aider, validates the one thing unit tests can't: real-model output parses with our regex+json.loads).

Test inputs: mission = "Improve test coverage of math_utils from 60% to 95%", metrics = {coverage: 0.62, tests_passing: 18/30}.

Goal Evaluator

```json
{
"goal_reached": false,
"confidence": 0.99,
"reason": "Current coverage is 62%, which is below the 95% target required by the mission."
}
```
✅ 3 expected keys, correct types, confidence in [0,1], reasoning accurate.

Strategist

```json
{
"stage": "exploration",
"next_milestone": "Reach 75% coverage by adding tests for untested edge cases in math_utils (zero inputs, negative numbers, overflow boundaries)",
"taboo_directions": [
"Adding duplicate tests that cover already-tested happy paths",
"Increasing test count without improving coverage percentage",
"Testing internal implementation details instead of observable behavior",
"Writing tests that always pass regardless of logic correctness"
]
}
```
✅ 3 expected keys, correct types, business-meaningful content.

Observation worth keeping

The model returns JSON wrapped in ```json … ``` fenced blocks. The role scripts' regex `r"{.*}"` with `re.DOTALL` strips the fences correctly — fallback path was not triggered for either prompt.

Coverage of the verification matrix

• Unit (54/54): governor↔role JSON I/O, --loop control flow, goal_reached exit code, strategist injection at round N+1, all hard-stop interactions ✅
• Real-LLM (this comment): both new prompts parse cleanly under real Sonnet 4.6 ✅
• Out of scope: full aider-driven mutation loop — unchanged from v0.2, no regression surface from this PR.

Ready to merge.