feat(eval): C3 cross-family EVAL panel — close same-family generator↔evaluator - #479
Merged
Conversation
…evaluator The EVAL drift monitor is meant to grade the claude reviewer with a different family, but that was only a comment and the task was wired extractor=None (inert). C3 makes cross-family a control: run_panel_drift_monitor grades each family independently (per-family majority, never pooled) and flags drift when ANY family's own majority disagrees with the signed answer — a dominant family can't mask its own drift. LiveFamilyExtractor runs each family via a shared build_member_argv (extracted verbatim from pr_review_watcher/main.py into member_runner.py — a pure move; C1 reviewer tests stay green) + codex stdout fallback. EvalPanelSettings ships OFF (panel=[]/enabled=False). Degraded panel => skipped loud, never a same-family collapse. Inert until an extraction corpus exists; wired + unit-tested with fakes. tests/unit 86.03%; reviewer 166 green. Council spec COUNCIL_VERDICT.md Phase 3 (C3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ty flagged self._extractor (CheckExtractor | None) at the run_drift_monitor call; the elif guard above already returns when it is None, so cast it (ruff bans assert). No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Council spec (
docs/design/COUNCIL_VERDICT.md) Phase 3 / C3 — the last council phase. Closes the guide-gap audit's HIGH finding: same-family generator↔evaluator.The EVAL drift monitor is supposed to grade the production reviewer (claude family) with a different family — but that requirement was only a code comment, and the task shipped wired
extractor=None(permanently inert). C3 turns cross-family from a comment into a control.How
eval/panel_critic.run_panel_drift_monitor(pure, 100% unit-tested): grades each configured family independently — a per-family majority vote that is never pooled across families for the drift decision — and flagsdrifted = any family's own majority ≠ the signed answer. This is what stops a dominant/larger family from masking its own drift by outvoting a smaller one. (C1'saggregate_councildoesn't fit — its unanimous-LGTM/merge shape is a decision, not a drift-vs-answer grade — so C3 has its own aggregator.)eval/panel_invoker.LiveFamilyExtractor: runs each family via the sharedbuild_member_argv+ a codex stdout-JSON fallback (verdict.last_json_object).member_runner.py:build_member_argvextracted verbatim frompr_review_watcher/main.py(a pure move) so the EVAL invoker never imports the merge-critical reviewer module. C1's 166 reviewer tests stay green.EvalPanelSettings:panel=[]/enabled=False⇒ OFF by default, mirroring C1's empty-guardrail-paths rollout.skippeds with a loud reason — it never silently grades with a single (same) family, which would re-open the finding.Status / prerequisites
Fully wired and unit-tested with injected fakes, but inert in production until an
extraction-kind corpus exists (the seed corpus isverdict-kind) — same inert-until-seeded postureBackendCheckExtractoralready had. The deterministic Lane-1 replay gate (model-free) is unchanged and remains the only blocking control.Tests
tests/unitfull suite 8531 pass at 86.03% (gate 85%); reviewer suite 166 pass (C1 intact after the extraction); ruff clean. New modules 100% covered (impure subprocess gluepragma: no cover). Per-family domination cases (claude drifts / codex agrees, and reverse) and degraded-family-skip are explicitly tested.Council spec Phase 3 (C3). Completes the C1 (#469) + C2 (ContextLifecycle #43) + C3 council trilogy.
🤖 Generated with Claude Code