Learning loop: feed real failures to the mutator + held-out validation gate for meta-eval rubric changes

[teetangh]

Context

Two defects blunt the self-learning loop (both called out in the repo's own known-limitations notes as top fixes):

1. The mutator never sees failures: learning/loop.py cycle() calls propose_candidate(trials=[]) — the mutator LLM is supposed to receive prior-cycle failure exemplars to direct prompt mutations, but gets an empty list, so candidate prompts are mutated blind.
2. The meta-evaluator self-modifies without validation: apply_proposal adopts rubric changes immediately; the designed held-out validation gate was never wired, and the merge_rubrics proposal type is accepted by the schema but silently no-ops. A bad rubric proposal (the Darwin–Gödel layer changing its own grader) goes live unchecked — exactly the failure class the meta-eval demo (scripts/demo_meta_eval.py, planted 35% out-of-policy blind spot) is meant to catch.

Implementation plan

1. Feed real failures to the mutator: in loop.cycle(), select the K worst trials from the previous cycle's eval run (data/eval_runs/{cycle_id}.json rows — lowest composite, compliance violations first) and pass them to propose_candidate(trials=...); verify the mutator prompt template actually interpolates them (add if missing). Cap tokens (existing 2000-token agent budget) by truncating transcripts to the failing segments.
2. Held-out validation gate for rubric changes: split eval seeds (data/seeds/eval_seeds.yaml) into proposal/validation sets. apply_proposal becomes: apply to a candidate rubric → re-score the held-out set with old vs new rubric → adopt only if the new rubric preserves agreement on known-good trials and improves detection on the planted blind-spot cases; otherwise reject and log to data/eval_runs/meta_log.jsonl.
3. Implement or remove merge_rubrics: either wire it (merge candidate rubric criteria with dedup) or reject the proposal type explicitly — silent no-op is the worst option.
4. Tests: unit test that propose_candidate receives non-empty trials after a cycle with failures; validation-gate test with a deliberately degenerate rubric proposal (must be rejected); scripts/demo_meta_eval.py still catches the planted 35% blind spot end-to-end.

Acceptance criteria

• Mutator prompt contains failure exemplars from the previous cycle (assert in test via prompt capture).
• A rubric proposal that degrades held-out agreement is rejected and logged; the planted blind-spot demo still passes.
• merge_rubrics either works with a test proving it, or is explicitly rejected with a clear error.

Dependencies

• None (independent of the voice-path issues; costs ~1 extra eval pass per adopted rubric proposal — budget note in scripts/cost_report.py).