diff --git a/.forgeplan/adrs/ADR-015-methodology-v0-2-5-frontier-judge-panel-deterministic-lint-typing-judges-score-subjective-axes-only.md b/.forgeplan/adrs/ADR-015-methodology-v0-2-5-frontier-judge-panel-deterministic-lint-typing-judges-score-subjective-axes-only.md new file mode 100644 index 0000000..9766105 --- /dev/null +++ b/.forgeplan/adrs/ADR-015-methodology-v0-2-5-frontier-judge-panel-deterministic-lint-typing-judges-score-subjective-axes-only.md @@ -0,0 +1,62 @@ +--- +depth: standard +id: ADR-015 +kind: adr +last_modified_at: 2026-06-03T21:00:08.397322+00:00 +last_modified_by: claude-code/2.1.156 +links: +- target: PRD-002 + relation: refines +status: draft +title: Methodology v0.2 — 5-frontier judge panel; deterministic lint+typing, judges score subjective axes only +--- + +## Status + +Draft (proposed 2026-06-03). Refines PRD-002 (judge-panel methodology) and ADR-005 (median + bootstrap-CI publication gate). Awaiting review before activation. + +## Context and Problem Statement + +The v0.1 judge panel runs 3 routes in practice (`claude-sonnet-4-6-judge`, `gpt-5-mini-judge`, and an un-isolated `gemini-3-flash`), with only 2 properly billing-isolated `-judge` aliases. Observed inter-judge agreement is low: Krippendorff α = 0.187 (EVID-047) → 0.358 after the reasoning-cap fix (EVID-048/050), still below the 0.70 publication gate. Two problems compound it: + +1. **Weak/undersized panel.** Mini-class judges are less self-consistent and a 2-judge panel has no tie-break. The user directs us to judge on the most powerful June-2026 models. +2. **Dual-path double-count.** The `be_01` rubric scores `type_safety` as a judge criterion (weight 0.15) while `scoring.md` also scores a deterministic `type_safety_score` (tsc, weight 0.10) — the same signal counted twice. Lint has the analogous risk. + +## Decision Drivers + +- Raise inter-judge agreement toward the α ≥ 0.70 gate without inventing scores. +- Judge **diversity** (uncorrelated vendor families) over raw count — prior-art: frontier judges reach α 0.80–0.91; family diversity de-biases more than a 4th same-family judge. +- No double-count: a signal a compiler can decide must not also be judged. +- Preserve run immutability (ADR-0002) and the median + bootstrap-CI gate (ADR-005). +- Cost-awareness: judge calls dominate spend. + +## Considered Options + +- **A — Status quo** (2–3 mixed judges, deterministic lint/tsc, `type_safety` also judged). +- **B — 5 strong judges that ALSO grade lint/typing** (keeps the double-count). +- **C — 5 frontier judges, 5 families; lint/typing DETERMINISTIC only; judges score subjective axes only.** ← chosen. +- **D — Defer to v0.3.** + +## Decision Outcome + +Chosen: **Option C.** + +**Judge panel** = the 5 most powerful June-2026 models, 5 distinct families (user decision 2026-06-03): Claude Opus 4.8 · GPT-5.5 · Gemini 3.1 Pro · Grok 4 · DeepSeek V4 Pro. Wired as `-judge` aliases in `infra/litellm-config.yaml`, billed to `OPENROUTER_API_KEY_JUDGE` (NFR-005), `reasoning_effort=low` so the rubric JSON fits the 2048-tok cap (EVID-050 pattern). + +**Lint + typing are DETERMINISTIC** (eslint/ruff + tsc), 0.10 each in the frozen coding formula, computed by `auto_metrics.py` on the harness's real file tree. The judge rubric's `type_safety` criterion becomes `design_appropriateness` (subjective: composition, idioms, boundaries — what a compiler can't decide). `rubric_version` → 2.0. + +**Evidence-backed levers** folded in (Research-B prior-art): chain-of-thought in the judge prompt, forced per-criterion scoring, calibration few-shot anchors, rubric criteria kept orthogonal to deterministic metrics. + +**Self-judging:** `_FAMILY_ALIASES` gains xai/deepseek/minimax so the guard normalises grok-4 / deepseek-* correctly. Until per-eval exclusion lands, the 5 frontier models are the **reference/judge tier** and the scored candidate roster excludes those families. + +### Consequences + +- New **MethodologyVersion v0.2.0**. Per ADR-0002, published v0.1.0 runs are NOT re-scored in place; new runs reference v0.2.0 with a `supersedes: methodology-v0.1.0` manifest link. The existing board is re-scored as a NEW run. +- **Refines** PRD-002 + ADR-005 — same median reducer + CI-lower-bound ≥ 0.70 gate, expanded roster + criteria clean-up. Not a supersede. +- Cost ≈ $0.10/eval (5 frontier judges @ ~3k in / 600 out); the current 45-cell board re-scores for ~$5–15. Tiering (strong panel on calibration only) is unnecessary at this scale; revisit when the grid grows. +- Follow-up: per-eval self-judging exclusion to re-admit grok-4 / deepseek-* as candidates; live route ping pending a funded key. + +## More Information + +Implemented in PR #59 (`feat/v0.2-infra-wave`): litellm-config 5 judge routes, judge_panel `_FAMILY_ALIASES`, build_real_board `_JUDGES`, be_01 rubric, `auto_metrics.py`. 766 tests green. Evidence: EVID-047/048 (α + cap fix), EVID-049/050 (gpt-5-mini reasoning fix). Prior-art: GPT-4o judge α 0.908 vs 70B 0.806 (arXiv 2506.13639); CoT +7–13pp (arXiv 2604.23178); structured per-criterion 31.5% SPB reduction (arXiv 2604.22891); 3-diverse-family panel beats single large judge at 7× lower cost (Verga et al.). + diff --git a/apps/eval-core-py/scripts/build_real_board.py b/apps/eval-core-py/scripts/build_real_board.py index 5a4149d..827ab09 100644 --- a/apps/eval-core-py/scripts/build_real_board.py +++ b/apps/eval-core-py/scripts/build_real_board.py @@ -115,7 +115,19 @@ } _SEEDS = [1, 2] _TASK = "be_01_jwt_auth" -_JUDGES = ["claude-sonnet-4-6-judge", "gpt-5-mini-judge", "gemini-3-flash"] +# methodology v0.2 (2026-06-03): 5 frontier judges, 5 distinct families +# (Anthropic/OpenAI/Google/xAI/DeepSeek) — the most powerful June-2026 models +# (user decision). These are the REFERENCE tier; candidates that share a judge +# family (grok-4, deepseek-*) need per-eval self-judging exclusion (follow-up) or +# curation out of the scored roster. Routes wired in infra/litellm-config.yaml +# (-judge aliases, billed to OPENROUTER_API_KEY_JUDGE, reasoning_effort=low). +_JUDGES = [ + "claude-opus-4-8-judge", + "gpt-5-5-judge", + "gemini-3-1-pro-judge", + "grok-4-judge", + "deepseek-v4-pro-judge", +] _RUN_HASH = "sha256:" + "realboard".ljust(58, "0")[:58] _SNAP = datetime(2026, 6, 3, tzinfo=UTC) @@ -191,6 +203,65 @@ def _rows_from_result(result: object, pricing: dict[str, PricingTuple]) -> list[ return rows +def _merge_into_board(existing: Board, partial: Board) -> Board: + """Merge *partial* cells/harnesses/models into *existing*, replacing by (model, stack) key. + + This is the shared merge logic reused by --add-stack, --fill, and --fill-missing. + ``partial`` is the newly-run sub-grid; ``existing`` is the current board on disk. + The returned Board is a new model instance (existing is not mutated). + """ + by_key = {(c.model_id, c.stack_id): c for c in existing.cells} + for c in partial.cells: + by_key[(c.model_id, c.stack_id)] = c + h_by_id = {h.stack_id: h for h in existing.harnesses} + for h in partial.harnesses: + h_by_id[h.stack_id] = h + m_by_id = {m.model_id: m for m in existing.models} + for m in partial.models: + m_by_id.setdefault(m.model_id, m) + cells = list(by_key.values()) + scored_now = sum(1 for c in cells if c.mean_score is not None) + return existing.model_copy( + update={ + "cells": cells, + "harnesses": list(h_by_id.values()), + "models": list(m_by_id.values()), + "scored": scored_now > 0, + } + ) + + +def _compute_gap( + out: Path, + stacks: list[str] | None, +) -> list[tuple[str, str]]: + """Return (model_id, stack_id) pairs that are in _STACK_MODELS but absent from board.json. + + Args: + out: Path to the current board.json. + stacks: Optional list of stack IDs to limit the desired grid. + If None, all stacks in _STACK_MODELS are included. + """ + # Build desired (model, stack) set. + stacks_to_check = stacks if stacks is not None else list(_STACK_MODELS.keys()) + desired: set[tuple[str, str]] = set() + for stack_id in stacks_to_check: + for model_id in _STACK_MODELS.get(stack_id, []): + desired.add((model_id, stack_id)) + + # Load present set from board.json (empty set if file missing/unreadable). + present: set[tuple[str, str]] = set() + if out.exists(): + try: + board = Board.model_validate_json(out.read_text(encoding="utf-8")) + present = {(c.model_id, c.stack_id) for c in board.cells} + except Exception: # board may be malformed/missing; default to empty + pass + + missing = sorted(desired - present) + return missing + + async def _main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--confirm-spend", action="store_true", help="real $ (~$0.25)") @@ -209,15 +280,53 @@ async def _main() -> int: "board.json, without re-spending on the other harnesses. e.g. " "--add-stack goose", ) + ap.add_argument( + "--fill-missing", + action="store_true", + help="run ONLY the (model, stack) cells absent from the current board.json " + "across all stacks in _STACK_MODELS (or a subset via --stacks), then " + "merge them in. Use --dry-run to preview the gap without spending.", + ) + ap.add_argument( + "--stacks", + default="", + help="comma-separated stack IDs to limit --fill-missing scope. " + "e.g. --stacks goose,opencode", + ) + ap.add_argument( + "--dry-run", + action="store_true", + help="with --fill-missing: print the missing (model, stack) list grouped " + "by stack and the count, then exit WITHOUT spending.", + ) args = ap.parse_args() os.chdir(REPO) _load_env(REPO) + out = REPO / "apps" / "site" / "public" / "board.json" + + # --fill-missing --dry-run: compute gap and print summary, NO spend. + if args.fill_missing and args.dry_run: + stacks_filter = [s.strip() for s in args.stacks.split(",") if s.strip()] or None + missing = _compute_gap(out, stacks_filter) + if not missing: + print("DRY-RUN fill-missing: gap is EMPTY — all cells present.") + return 0 + # Group by stack for readability. + by_stack: dict[str, list[str]] = {} + for model_id, stack_id in missing: + by_stack.setdefault(stack_id, []).append(model_id) + print(f"DRY-RUN fill-missing: {len(missing)} missing cells across {len(by_stack)} stacks:") + for stack_id in sorted(by_stack): + models = sorted(by_stack[stack_id]) + print(f" {stack_id} ({len(models)}): {', '.join(models)}") + return 0 + key = os.environ.get("LITELLM_MASTER_KEY", "") if not key: print("ERROR: LITELLM_MASTER_KEY not set (.env)", file=sys.stderr) return 2 - if not args.confirm_spend: + if not args.confirm_spend and not args.fill_missing and not args.add_stack and not args.fill: n = (len(_RAW_MODELS) + len(_AIDER_MODELS)) * len(_SEEDS) print( f"DRY: pass --confirm-spend to run {n} evals " @@ -260,10 +369,13 @@ async def _main() -> int: def caller_for(stack_id: str) -> object: return stack_caller if stack_id != "raw-llm" else inspect_caller - # candidate_model_id only drives self-judging exclusion; all candidates are - # open (non-judge-family), so any is safe here. + # candidate_model_id only drives the CONSTRUCTION-time self-judging guard. + # _RAW_MODELS[0] must be a non-judge-family (open) model — clash-free against + # the 5-frontier judge roster. Per-eval exclusion for candidates that share a + # judge family (grok-4, deepseek-*) is a follow-up; until then the scored + # candidate roster excludes those families. panel = JudgePanel( - judge_models=_JUDGES, candidate_model_id=_RAW_MODELS[0], rubric_version="1.0" + judge_models=_JUDGES, candidate_model_id=_RAW_MODELS[0], rubric_version="2.0" ) runner = GridRunner( caller=inspect_caller, @@ -283,7 +395,6 @@ def caller_for(stack_id: str) -> object: # per-task wall-clock budget by difficulty (be_01 is medium -> 600s), so slow # model x harness pairs aren't cut off at the 300s default. task_timeout = {t: timeout_of(t) for t in [_TASK]} - out = REPO / "apps" / "site" / "public" / "board.json" # --fill: re-run ONLY the named models on raw-llm and merge their cells into # the existing board.json (fills previously-failed cells, leaves the rest). @@ -334,32 +445,65 @@ def caller_for(stack_id: str) -> object: rows = _rows_from_result(result, _PRICING) partial = build_board(rows, stacks_root=stacks_root, run_hash=_RUN_HASH, run_type="smoke") existing = Board.model_validate_json(out.read_text(encoding="utf-8")) - # Replace-or-append cells by (model, stack); union harnesses + models. - by_key = {(c.model_id, c.stack_id): c for c in existing.cells} - for c in partial.cells: - by_key[(c.model_id, c.stack_id)] = c - h_by_id = {h.stack_id: h for h in existing.harnesses} - for h in partial.harnesses: - h_by_id[h.stack_id] = h - m_by_id = {m.model_id: m for m in existing.models} - for m in partial.models: - m_by_id.setdefault(m.model_id, m) - cells = list(by_key.values()) - scored_now = sum(1 for c in cells if c.mean_score is not None) - merged_board = existing.model_copy( - update={ - "cells": cells, - "harnesses": list(h_by_id.values()), - "models": list(m_by_id.values()), - "scored": scored_now > 0, - } - ) + merged_board = _merge_into_board(existing, partial) out.write_text(merged_board.model_dump_json(indent=2) + "\n", encoding="utf-8") + cells = merged_board.cells + scored_now = sum(1 for c in cells if c.mean_score is not None) print(f" board now {scored_now}/{len(cells)} cells scored. new {stack_id} cells:") for c in partial.cells: print(f" {c.stack_id} x {c.model_id}: score={c.mean_score} cost=${c.mean_cost_usd}") return 0 + # --fill-missing (with --confirm-spend): run only the cells absent from the + # current board.json, grouped by stack into one GridSpec per stack, then + # merge each partial result using the same logic as --add-stack. + if args.fill_missing: + if not args.confirm_spend: + print("ERROR: --fill-missing requires --confirm-spend (real $).", file=sys.stderr) + return 2 + stacks_filter = [s.strip() for s in args.stacks.split(",") if s.strip()] or None + missing = _compute_gap(out, stacks_filter) + if not missing: + print("fill-missing: gap is EMPTY — all cells present. Nothing to run.") + return 0 + # Group missing by stack so we run one GridSpec per stack. + by_stack: dict[str, list[str]] = {} + for model_id, stack_id in missing: + by_stack.setdefault(stack_id, []).append(model_id) + print(f"FILL-MISSING: {len(missing)} cells across {len(by_stack)} stacks — running ...") + for stack_id in sorted(by_stack): + models = by_stack[stack_id] + print(f" stack {stack_id}: {models}") + existing = Board.model_validate_json(out.read_text(encoding="utf-8")) + for stack_id, models in sorted(by_stack.items()): + print(f"\n--- fill-missing: {stack_id} x {models} ---", flush=True) + result = await runner.run( + GridSpec( + run_hash=_RUN_HASH, + models=models, + tasks=[_TASK], + stacks=[stack_id], + seeds=_SEEDS, + task_timeout_s=task_timeout, + ) + ) + rows = _rows_from_result(result, _PRICING) + partial = build_board( + rows, stacks_root=stacks_root, run_hash=_RUN_HASH, run_type="smoke" + ) + existing = _merge_into_board(existing, partial) + # Write after each stack so a crash mid-run leaves a partial board. + out.write_text(existing.model_dump_json(indent=2) + "\n", encoding="utf-8") + cells_done = sum(1 for c in existing.cells if c.mean_score is not None) + print(f" {stack_id} done — board now {cells_done}/{len(existing.cells)} scored.") + for c in partial.cells: + print( + f" {c.stack_id} x {c.model_id}: score={c.mean_score} cost=${c.mean_cost_usd}" + ) + total_scored = sum(1 for c in existing.cells if c.mean_score is not None) + print(f"\nfill-missing complete: {total_scored}/{len(existing.cells)} cells scored.") + return 0 + # Two specs so the grid is non-cartesian: raw-llm on every candidate, aider # only on the models that follow its edit format. Same run_hash + runner so # rows merge into one board. @@ -389,7 +533,6 @@ def caller_for(stack_id: str) -> object: ) rows = [] total_cost = Decimal("0") - out = REPO / "apps" / "site" / "public" / "board.json" def _emit() -> None: board = build_board(rows, stacks_root=stacks_root, run_hash=_RUN_HASH, run_type="smoke") diff --git a/apps/eval-core-py/src/evaluators/lint_evaluator.py b/apps/eval-core-py/src/evaluators/lint_evaluator.py index 2bf6ddb..008db6f 100644 --- a/apps/eval-core-py/src/evaluators/lint_evaluator.py +++ b/apps/eval-core-py/src/evaluators/lint_evaluator.py @@ -36,9 +36,17 @@ logger = logging.getLogger(__name__) # Task-id prefix -> canonical language (when file extension is absent/ambiguous). +# NOTE: "be_" is intentionally absent — the reference be_01_jwt_auth task is +# TypeScript (Express), and the file-extension scan already handles it correctly +# when real files are present. Including a Python fallback here would cause +# LintEvaluator to invoke ruff on a TypeScript submission when the path is a +# text blob with no extension, silently producing 0 findings rather than calling +# eslint. When the path is a directory with real .ts files the extension scan +# correctly returns "typescript" without consulting this map. _TASK_LANG_MAP: dict[str, str] = { - "be_": "python", # backend tasks -- JWT auth in Express/TS or Python "fe_": "typescript", # frontend tasks -- React / TS + "ts_": "typescript", # explicit TypeScript tasks + "fs_": "typescript", # fullstack tasks (TS frontend + TS backend) "doc_": "none", # documentation tasks -- no linting applicable } diff --git a/apps/eval-core-py/src/orchestrator/auto_metrics.py b/apps/eval-core-py/src/orchestrator/auto_metrics.py new file mode 100644 index 0000000..7d7a694 --- /dev/null +++ b/apps/eval-core-py/src/orchestrator/auto_metrics.py @@ -0,0 +1,200 @@ +"""Run deterministic automatic evaluators and compute the coding-task final_score. + +Two responsibilities kept in one module (they share the same weight table): + +1. ``run_auto_evaluators(submission_path, task_id)`` + Runs LintEvaluator + TypeSafetyEvaluator on a real filesystem path (the + candidate's produced file tree). Returns a dict suitable for + EvalRow.automatic_metrics. Evaluators self-report skipped=True when their + binary is absent — those components default to 0.0. + +2. ``compute_final_score(eval_row)`` + Applies the frozen coding-task formula from docs/02-methodology/scoring.md: + + final_score = 0.40*correctness + 0.15*coverage + 0.10*complexity + + 0.10*lint + 0.10*type_safety + + 0.15*pattern_match + + Reads ``automatic_metrics`` for the deterministic components; reads + ``judge_aggregate.median_per_criterion["pattern_match"]`` for the judge term + (already normalised to 0-10 by JudgePanel.aggregate). + + Returns ``None`` when neither automatic_metrics nor a judge aggregate is + present (raw-llm evals without a judge panel — no score possible yet). + Returns a partial score when some components are present and others are not + (skipped evaluators contribute 0.0 to their weight slot — the formula is + still computed, just with those slots zeroed out). + + The result is in [0.0, 10.0]. + +Usage: + # In stack_scoring.exec_result_to_eval_result (after writing submission to disk) + automatic_metrics = await run_auto_evaluators(str(submission_path), task_id) + # In grid_runner._invoke_judge_panel (after JudgePanel.aggregate) + final_row = compute_final_score(eval_row_with_judge_and_auto_metrics) +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any + +from src.contracts.eval_row import EvalRow +from src.evaluators.lint_evaluator import LintEvaluator +from src.evaluators.type_safety_evaluator import TypeSafetyEvaluator + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Coding-task weight table (frozen — docs/02-methodology/scoring.md v0.1.0) +# --------------------------------------------------------------------------- + +_CODING_WEIGHTS: dict[str, float] = { + "correctness": 0.40, + "coverage": 0.15, + "complexity": 0.10, + "lint": 0.10, + "type_safety": 0.10, + "pattern_match": 0.15, +} + +# Evaluators that run on the host (static analysis, no sandbox). +_LINT_EVALUATOR = LintEvaluator() +_TYPE_SAFETY_EVALUATOR = TypeSafetyEvaluator() + + +# --------------------------------------------------------------------------- +# 1. run_auto_evaluators +# --------------------------------------------------------------------------- + + +async def run_auto_evaluators(submission_path: str, task_id: str) -> dict[str, Any]: + """Run lint + type_safety on *submission_path* and return automatic_metrics dict. + + Both evaluators self-report skipped=True when their binary is absent or + inapplicable — those slots carry score=0.0, which is the correct + contribution to the final_score formula (no score invented, no bias). + + The dict shape matches EvalRow.automatic_metrics expectations: + { + "lint": 0.8, # 0.0..1.0 (0-1 range, not 0-10) + "lint_skipped": false, + "lint_skip_reason": null, + "type_safety": 0.7, + "type_safety_skipped": false, + "type_safety_skip_reason": null, + } + + Args: + submission_path: Absolute path to the candidate's file tree or a single + source file. For CLI stacks this is the snapshot dir written by + stack_scoring.extract_submission. For raw-llm (text blob) callers + should pass the path to the raw_output file; evaluators will try + the file-extension detection and skip gracefully if no TS/Py found. + task_id: e.g. "be_01_jwt_auth". Drives language detection fallback. + + Returns: + dict suitable for EvalRow.automatic_metrics. Never raises. + """ + lint_res, ts_res = await asyncio.gather( + _LINT_EVALUATOR.evaluate(submission_path, task_id), + _TYPE_SAFETY_EVALUATOR.evaluate(submission_path, task_id), + ) + + if lint_res.skipped: + logger.debug( + "LintEvaluator skipped for task=%s path=%s reason=%s", + task_id, + submission_path, + lint_res.skip_reason, + ) + if ts_res.skipped: + logger.debug( + "TypeSafetyEvaluator skipped for task=%s path=%s reason=%s", + task_id, + submission_path, + ts_res.skip_reason, + ) + + return { + "lint": lint_res.score, + "lint_skipped": lint_res.skipped, + "lint_skip_reason": lint_res.skip_reason, + "type_safety": ts_res.score, + "type_safety_skipped": ts_res.skipped, + "type_safety_skip_reason": ts_res.skip_reason, + } + + +# --------------------------------------------------------------------------- +# 2. compute_final_score +# --------------------------------------------------------------------------- + + +def compute_final_score(row: EvalRow) -> float | None: + """Apply the frozen coding-task weighted formula and return final_score. + + Formula (docs/02-methodology/scoring.md): + final_score_01 = + 0.40 * correctness + 0.15 * coverage + 0.10 * complexity + + 0.10 * lint + 0.10 * type_safety + + 0.15 * pattern_match + + final_score_10 = final_score_01 * 10 + + Component sources: + - correctness, coverage, complexity, lint, type_safety: + ``row.automatic_metrics`` (values in 0-1 range). + - pattern_match: + ``row.judge_aggregate.median_per_criterion["pattern_match"]`` + (value in 0-10 range; divided by 10 to normalise). + + Missing components default to 0.0 (contributes 0 * weight to the total). + Returns None only when BOTH automatic_metrics and judge_aggregate are + absent, meaning there is genuinely nothing to score yet. + + Args: + row: An EvalRow with status=SCORED and any combination of + automatic_metrics / judge_aggregate already set. + + Returns: + float in [0.0, 10.0], or None when no scoring material exists at all. + """ + metrics = row.automatic_metrics + agg = row.judge_aggregate + + has_metrics = bool(metrics) + has_judge = agg is not None and bool(agg.median_per_criterion) + + if not has_metrics and not has_judge: + return None + + def _get(key: str) -> float: + """Fetch a 0-1 value from automatic_metrics; default 0.0 if absent.""" + v = metrics.get(key, 0.0) + try: + return float(v) + except (TypeError, ValueError): + return 0.0 + + def _judge_crit(key: str) -> float: + """Fetch a 0-10 judge criterion, normalise to 0-1; default 0.0.""" + if agg is None or agg.median_per_criterion is None: + return 0.0 + v = agg.median_per_criterion.get(key, 0.0) + try: + return float(v) / 10.0 + except (TypeError, ValueError): + return 0.0 + + score_01 = ( + _CODING_WEIGHTS["correctness"] * _get("correctness") + + _CODING_WEIGHTS["coverage"] * _get("coverage") + + _CODING_WEIGHTS["complexity"] * _get("complexity") + + _CODING_WEIGHTS["lint"] * _get("lint") + + _CODING_WEIGHTS["type_safety"] * _get("type_safety") + + _CODING_WEIGHTS["pattern_match"] * _judge_crit("pattern_match") + ) + + return round(max(0.0, min(10.0, score_01 * 10.0)), 4) diff --git a/apps/eval-core-py/src/orchestrator/grid_runner.py b/apps/eval-core-py/src/orchestrator/grid_runner.py index 8304d28..72e65f0 100644 --- a/apps/eval-core-py/src/orchestrator/grid_runner.py +++ b/apps/eval-core-py/src/orchestrator/grid_runner.py @@ -25,6 +25,7 @@ from opentelemetry.trace import StatusCode from src.contracts import ErrorClass, EvalStatus +from src.orchestrator.auto_metrics import compute_final_score from src.orchestrator.cost import BudgetGate, PricingTuple, compute_cost from src.orchestrator.eval_caller import ( EvalCaller, @@ -456,14 +457,24 @@ async def _invoke_judge_panel( span.set_attribute("pollmevals.judge_alpha_point", aggregation.alpha_point) span.set_attribute("pollmevals.judge_cost_usd", float(judge_cost)) + # Assemble the row with judge results, then compute the weighted + # final_score from automatic_metrics + judge pattern_match. + # compute_final_score reads both fields and returns None when neither + # is present (e.g. raw-llm without auto metrics — score stays None). + judged_row = result.eval_row.model_copy( + update={ + "judgments": judgments, + "judge_aggregate": aggregation, + } + ) + final_score = compute_final_score(judged_row) + if final_score is not None: + judged_row = judged_row.model_copy(update={"final_score": final_score}) + span.set_attribute("pollmevals.computed_final_score", final_score) + return dataclasses.replace( result, - eval_row=result.eval_row.model_copy( - update={ - "judgments": judgments, - "judge_aggregate": aggregation, - } - ), + eval_row=judged_row, completed_at=datetime.now(UTC), ) diff --git a/apps/eval-core-py/src/orchestrator/judge_panel.py b/apps/eval-core-py/src/orchestrator/judge_panel.py index c66253c..6291d9e 100644 --- a/apps/eval-core-py/src/orchestrator/judge_panel.py +++ b/apps/eval-core-py/src/orchestrator/judge_panel.py @@ -82,6 +82,17 @@ # Qwen (Alibaba) "qwen": "qwen", "alibaba": "qwen", + # xAI (Grok) — added methodology v0.2 (2026-06-03). Without this, the proxy + # alias "grok-4-judge" and candidate "grok-4" both fall through to their raw + # names (which differ), so the self-judging guard would MISS a grok-judges-grok + # clash. The startswith("grok") branch maps both to "xai". + "x-ai": "xai", + "xai": "xai", + "grok": "xai", + # DeepSeek — same rationale (candidate "deepseek-v4-pro" vs "deepseek-v4-pro-judge"). + "deepseek": "deepseek", + # MiniMax (alternate cheap 5th judge / candidate family). + "minimax": "minimax", } # inspect_ai.eval_async is process-global and forbids concurrent invocation diff --git a/apps/eval-core-py/src/orchestrator/stack_caller.py b/apps/eval-core-py/src/orchestrator/stack_caller.py index dcfcf6e..4261fae 100644 --- a/apps/eval-core-py/src/orchestrator/stack_caller.py +++ b/apps/eval-core-py/src/orchestrator/stack_caller.py @@ -25,6 +25,7 @@ import yaml from src.contracts import ErrorClass, EvalRow, EvalStats, EvalStatus +from src.orchestrator.auto_metrics import run_auto_evaluators from src.orchestrator.eval_caller import ( EvalRequest, EvalResult, @@ -103,9 +104,26 @@ async def call(self, request: EvalRequest) -> EvalResult: exec_result = await self.executor.execute(exec_request) if exec_result.status is ExecStatus.OK: - return exec_result_to_eval_result( + eval_result = exec_result_to_eval_result( exec_result, log_dir=self.log_dir, run_hash=self.run_hash ) + # Run lint + type_safety on the real filesystem snapshot produced by + # the harness. The submission dir is the repo_snapshot_dir that the + # harness wrote its changes into (not the concatenated text blob). + # Evaluators self-skip gracefully when binaries are absent. + assert eval_result.eval_row is not None + auto_metrics = await run_auto_evaluators( + str(exec_request.repo_snapshot_dir), request.task_id + ) + import dataclasses + + eval_result = dataclasses.replace( + eval_result, + eval_row=eval_result.eval_row.model_copy( + update={"automatic_metrics": auto_metrics} + ), + ) + return eval_result return self._failed_result(request, exec_result, started_at) def _failed_result( diff --git a/apps/eval-core-py/tests/test_auto_metrics.py b/apps/eval-core-py/tests/test_auto_metrics.py new file mode 100644 index 0000000..9e9e1c9 --- /dev/null +++ b/apps/eval-core-py/tests/test_auto_metrics.py @@ -0,0 +1,366 @@ +"""Tests for orchestrator/auto_metrics.py — evaluator wiring + final_score formula. + +Coverage: + TestRunAutoEvaluators -- lint + type_safety scores appear in automatic_metrics + for a TS snippet with known errors (non-zero scores) + TestRunAutoEvaluatorsSkip-- evaluators skip gracefully when binaries absent + TestComputeFinalScore -- formula: metrics + judge pattern_match -> correct result + TestComputeFinalScoreEdges-- None when nothing, partial when some components present + TestLintTaskLangMap -- be_ task no longer defaults to python fallback +""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import AsyncMock, patch + +import pytest + +from src.contracts import EvalArtifactRefs, EvalStats, EvalStatus +from src.contracts.artifact_ref import ArtifactRef +from src.contracts.eval_row import EvalRow +from src.contracts.judge import JudgeAggregation +from src.orchestrator.auto_metrics import ( + _CODING_WEIGHTS, + compute_final_score, + run_auto_evaluators, +) + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_proc(stdout: str, stderr: str = "", returncode: int = 0) -> AsyncMock: + proc = AsyncMock() + proc.returncode = returncode + proc.communicate = AsyncMock(return_value=(stdout.encode(), stderr.encode())) + return proc + + +def _minimal_artifact_refs() -> EvalArtifactRefs: + ref = ArtifactRef( + sha256="a" * 64, size_bytes=1, uri="file:///tmp/x", mime_type="text/plain" + ) + return EvalArtifactRefs(raw_output=ref, normalized_output=ref, evaluator_json=ref) + + +def _minimal_row(automatic_metrics: dict | None = None, judge_aggregate=None) -> EvalRow: + return EvalRow( + eval_id="abcdef0123456789", + model_id="openrouter/qwen/qwen-3-14b", + stack_id="aider", + task_id="be_01_jwt_auth", + seed=1, + status=EvalStatus.SCORED, + artifact_refs=_minimal_artifact_refs(), + stats=EvalStats(input_tokens=100, output_tokens=200, wall_clock_ms=1000, cost_usd=0), + automatic_metrics=automatic_metrics or {}, + judge_aggregate=judge_aggregate, + ) + + +# --------------------------------------------------------------------------- +# TestRunAutoEvaluators — TS snippet, eslint + tsc produce non-trivial scores +# --------------------------------------------------------------------------- + + +class TestRunAutoEvaluators: + """Verify that lint+type_safety scores flow through run_auto_evaluators.""" + + @pytest.mark.asyncio + async def test_ts_snippet_with_lint_errors_returns_nonzero_scores( + self, tmp_path: Path + ) -> None: + """A .ts file with eslint warnings produces lint < 1.0 and type_safety = 1.0.""" + ts_file = tmp_path / "solution.ts" + ts_file.write_text("var x = 1;\n") # deliberate: var is a lint finding + + # eslint reports 3 messages (simulating var, no-unused-vars, etc.) + eslint_out = json.dumps( + [{"filePath": str(ts_file), "messages": [{"m": "1"}, {"m": "2"}, {"m": "3"}]}] + ) + # tsc reports 0 errors + tsc_out = "" + + def _which(bin_name: str) -> str | None: + return f"/usr/bin/{bin_name}" if bin_name in ("eslint", "tsc") else None + + async def _fake_exec(*args: str, **_kw: object) -> AsyncMock: + if args[0] == "eslint": + return _make_proc(eslint_out, returncode=1) + else: # tsc + return _make_proc(tsc_out, returncode=0) + + with ( + patch("src.evaluators.lint_evaluator.shutil.which", side_effect=_which), + patch("src.evaluators.type_safety_evaluator.shutil.which", side_effect=_which), + patch("src.evaluators.lint_evaluator._get_eslint_version", return_value="v9.0.0"), + patch("src.evaluators.type_safety_evaluator._get_tsc_version", return_value="v5.5.0"), + patch("asyncio.create_subprocess_exec", side_effect=_fake_exec), + ): + metrics = await run_auto_evaluators(str(tmp_path), "be_01_jwt_auth") + + # lint: 3 findings -> 1.0 - 3/10 = 0.7 + assert metrics["lint"] == pytest.approx(0.7) + assert metrics["lint_skipped"] is False + # type_safety: 0 errors -> 1.0 + assert metrics["type_safety"] == pytest.approx(1.0) + assert metrics["type_safety_skipped"] is False + + @pytest.mark.asyncio + async def test_ts_snippet_with_type_errors_returns_nonzero_type_safety( + self, tmp_path: Path + ) -> None: + """A .ts file with tsc errors produces type_safety < 1.0.""" + ts_file = tmp_path / "bad.ts" + ts_file.write_text("const x: number = 'wrong';\n") + + # eslint: clean + eslint_out = json.dumps([{"filePath": str(ts_file), "messages": []}]) + # tsc: 2 type errors + tsc_out = ( + "bad.ts(1,7): error TS2322: Type 'string' is not assignable to type 'number'.\n" + "bad.ts(1,9): error TS2304: Cannot find name 'wrong'.\n" + ) + + def _which(bin_name: str) -> str | None: + return f"/usr/bin/{bin_name}" if bin_name in ("eslint", "tsc") else None + + async def _fake_exec(*args: str, **_kw: object) -> AsyncMock: + if args[0] == "eslint": + return _make_proc(eslint_out, returncode=0) + else: + return _make_proc(tsc_out, returncode=1) + + with ( + patch("src.evaluators.lint_evaluator.shutil.which", side_effect=_which), + patch("src.evaluators.type_safety_evaluator.shutil.which", side_effect=_which), + patch("src.evaluators.lint_evaluator._get_eslint_version", return_value="v9.0.0"), + patch("src.evaluators.type_safety_evaluator._get_tsc_version", return_value="v5.5.0"), + patch("asyncio.create_subprocess_exec", side_effect=_fake_exec), + ): + metrics = await run_auto_evaluators(str(tmp_path), "be_01_jwt_auth") + + # lint: 0 findings -> 1.0 + assert metrics["lint"] == pytest.approx(1.0) + # type_safety: 2 errors -> 1.0 - 2/10 = 0.8 + assert metrics["type_safety"] == pytest.approx(0.8) + + @pytest.mark.asyncio + async def test_keys_present_even_when_skipped(self, tmp_path: Path) -> None: + """All four keys (lint, lint_skipped, type_safety, type_safety_skipped) always present.""" + # No TS/Py files, no binaries available + with ( + patch("src.evaluators.lint_evaluator.shutil.which", return_value=None), + patch("src.evaluators.type_safety_evaluator.shutil.which", return_value=None), + ): + metrics = await run_auto_evaluators(str(tmp_path), "doc_01_cli_readme") + + assert "lint" in metrics + assert "lint_skipped" in metrics + assert "type_safety" in metrics + assert "type_safety_skipped" in metrics + assert metrics["lint"] == 0.0 + assert metrics["type_safety"] == 0.0 + + +# --------------------------------------------------------------------------- +# TestRunAutoEvaluatorsSkip — graceful degradation when binaries absent +# --------------------------------------------------------------------------- + + +class TestRunAutoEvaluatorsSkip: + @pytest.mark.asyncio + async def test_scores_default_zero_when_both_binaries_absent( + self, tmp_path: Path + ) -> None: + """lint=0.0, type_safety=0.0, both marked skipped when binaries absent.""" + ts_file = tmp_path / "app.ts" + ts_file.write_text("const x = 1;\n") + + with ( + patch("src.evaluators.lint_evaluator.shutil.which", return_value=None), + patch("src.evaluators.type_safety_evaluator.shutil.which", return_value=None), + ): + metrics = await run_auto_evaluators(str(tmp_path), "be_01_jwt_auth") + + assert metrics["lint"] == 0.0 + assert metrics["lint_skipped"] is True + assert metrics["type_safety"] == 0.0 + assert metrics["type_safety_skipped"] is True + + +# --------------------------------------------------------------------------- +# TestComputeFinalScore — coding-task weighted formula +# --------------------------------------------------------------------------- + + +class TestComputeFinalScore: + def test_all_components_present_correct_weighted_sum(self) -> None: + """Full metrics + judge pattern_match → formula result.""" + auto = { + "correctness": 0.8, + "coverage": 0.6, + "complexity": 1.0, + "lint": 0.7, + "type_safety": 0.9, + } + agg = JudgeAggregation( + n_judges_used=3, + judge_status="OK", + median_per_criterion={"pattern_match": 8.0}, # 0-10 range + ) + row = _minimal_row(automatic_metrics=auto, judge_aggregate=agg) + result = compute_final_score(row) + assert result is not None + + # Manual: (0.40*0.8 + 0.15*0.6 + 0.10*1.0 + 0.10*0.7 + 0.10*0.9 + 0.15*(8.0/10)) * 10 + expected_01 = ( + 0.40 * 0.8 + + 0.15 * 0.6 + + 0.10 * 1.0 + + 0.10 * 0.7 + + 0.10 * 0.9 + + 0.15 * (8.0 / 10.0) + ) + expected = round(expected_01 * 10.0, 4) + assert result == pytest.approx(expected, abs=1e-3) + + def test_lint_and_type_safety_contribute_to_score(self) -> None: + """Non-zero lint + type_safety scores shift the result vs zero baseline.""" + auto_zero = {"lint": 0.0, "type_safety": 0.0} + auto_nonzero = {"lint": 0.8, "type_safety": 0.9} + + row_zero = _minimal_row(automatic_metrics=auto_zero) + row_nonzero = _minimal_row(automatic_metrics=auto_nonzero) + + score_zero = compute_final_score(row_zero) + score_nonzero = compute_final_score(row_nonzero) + + assert score_zero is not None + assert score_nonzero is not None + # lint weight=0.10, type_safety weight=0.10 → 0.17 difference in 0-10 range + assert score_nonzero > score_zero + + def test_weights_sum_respected(self) -> None: + """All components at 1.0 → final_score = 10.0 (perfect score).""" + auto = { + "correctness": 1.0, + "coverage": 1.0, + "complexity": 1.0, + "lint": 1.0, + "type_safety": 1.0, + } + agg = JudgeAggregation( + n_judges_used=3, + judge_status="OK", + median_per_criterion={"pattern_match": 10.0}, + ) + row = _minimal_row(automatic_metrics=auto, judge_aggregate=agg) + result = compute_final_score(row) + assert result == pytest.approx(10.0, abs=1e-3) + + def test_all_zero_components_returns_zero(self) -> None: + """All components 0.0 → final_score = 0.0.""" + auto = { + "correctness": 0.0, + "coverage": 0.0, + "complexity": 0.0, + "lint": 0.0, + "type_safety": 0.0, + } + agg = JudgeAggregation( + n_judges_used=3, + judge_status="OK", + median_per_criterion={"pattern_match": 0.0}, + ) + row = _minimal_row(automatic_metrics=auto, judge_aggregate=agg) + assert compute_final_score(row) == pytest.approx(0.0) + + +# --------------------------------------------------------------------------- +# TestComputeFinalScoreEdges — None / partial / clamping +# --------------------------------------------------------------------------- + + +class TestComputeFinalScoreEdges: + def test_returns_none_when_no_metrics_and_no_judge(self) -> None: + """Empty row (no auto_metrics, no judge) → None (nothing to score).""" + row = _minimal_row() + assert compute_final_score(row) is None + + def test_partial_metrics_only_no_judge(self) -> None: + """Only lint+type_safety present; other components default to 0 but score computed.""" + auto = {"lint": 1.0, "type_safety": 1.0} + row = _minimal_row(automatic_metrics=auto) + result = compute_final_score(row) + assert result is not None + # Only lint(0.10) + type_safety(0.10) contribute → 0.20 * 10 = 2.0 + assert result == pytest.approx(2.0, abs=1e-3) + + def test_judge_only_no_auto_metrics(self) -> None: + """Only judge pattern_match present → partial score from that term.""" + agg = JudgeAggregation( + n_judges_used=3, + judge_status="OK", + median_per_criterion={"pattern_match": 10.0}, + ) + row = _minimal_row(judge_aggregate=agg) + result = compute_final_score(row) + assert result is not None + # Only pattern_match(0.15) contributes → 0.15 * (10.0/10.0) * 10 = 1.5 + assert result == pytest.approx(1.5, abs=1e-3) + + def test_score_clamped_to_10(self) -> None: + """Formula result > 10.0 is clamped (defensive — correct weights sum to exactly 1).""" + auto = {k: 2.0 for k in ("correctness", "coverage", "complexity", "lint", "type_safety")} + agg = JudgeAggregation( + n_judges_used=3, + judge_status="OK", + median_per_criterion={"pattern_match": 20.0}, + ) + row = _minimal_row(automatic_metrics=auto, judge_aggregate=agg) + result = compute_final_score(row) + assert result is not None + assert result <= 10.0 + + +# --------------------------------------------------------------------------- +# TestLintTaskLangMap — be_ prefix no longer defaults to python +# --------------------------------------------------------------------------- + + +class TestLintTaskLangMap: + def test_be_task_with_ts_files_uses_eslint_not_ruff(self, tmp_path: Path) -> None: + """be_ task with a .ts file → LintEvaluator uses eslint, not ruff.""" + from src.evaluators.lint_evaluator import _detect_language + + ts_file = tmp_path / "solution.ts" + ts_file.write_text("export const x = 1;\n") + + lang = _detect_language(tmp_path, "be_01_jwt_auth") + assert lang == "typescript" + + def test_be_task_empty_dir_not_forced_to_python(self, tmp_path: Path) -> None: + """be_ task with empty dir → does NOT fall back to python (old broken behaviour).""" + from src.evaluators.lint_evaluator import _detect_language + + lang = _detect_language(tmp_path, "be_01_jwt_auth") + # No files → "none" (not "python") + assert lang == "none" + + def test_fe_task_still_falls_back_to_typescript(self, tmp_path: Path) -> None: + """fe_ task with no files → still falls back to typescript via map.""" + from src.evaluators.lint_evaluator import _detect_language + + lang = _detect_language(tmp_path, "fe_01_multistep_form") + assert lang == "typescript" + + def test_doc_task_maps_to_none(self, tmp_path: Path) -> None: + """doc_ task → none (no linting applicable).""" + from src.evaluators.lint_evaluator import _detect_language + + lang = _detect_language(tmp_path, "doc_01_cli_readme") + assert lang == "none" diff --git a/apps/eval-core-py/tests/test_fill_missing.py b/apps/eval-core-py/tests/test_fill_missing.py new file mode 100644 index 0000000..0a39421 --- /dev/null +++ b/apps/eval-core-py/tests/test_fill_missing.py @@ -0,0 +1,347 @@ +"""Tests for --fill-missing mode in scripts/build_real_board.py. + +Covers: +1. TestComputeGap -- _compute_gap computes desired - present correctly +2. TestMergeIntoBoard -- _merge_into_board is idempotent and correct +3. TestDryRun -- --fill-missing --dry-run prints gap, exits 0, NO spend +""" + +from __future__ import annotations + +import sys +from datetime import UTC, datetime +from decimal import Decimal +from pathlib import Path +from typing import Any + +import pytest + +from src.contracts import ( + ArtifactRef, + EvalArtifactRefs, + EvalRow, + EvalStats, + EvalStatus, +) +from src.leaderboard.board import Board, build_board + +# --------------------------------------------------------------------------- +# Add scripts/ to sys.path so `from build_real_board import ...` works +# --------------------------------------------------------------------------- + +_SCRIPTS_DIR = Path(__file__).parents[1] / "scripts" +if str(_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(_SCRIPTS_DIR)) + +from build_real_board import ( # noqa: E402 + _STACK_MODELS, + _compute_gap, + _merge_into_board, +) + +_STACKS_ROOT = Path(__file__).resolve().parents[3] / "stacks" +_RUN_HASH = "sha256:" + "t" * 58 + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _ref() -> ArtifactRef: + return ArtifactRef( + sha256="a" * 64, size_bytes=10, uri="file:///tmp/x.txt", mime_type="text/plain" + ) + + +def _row( + model_id: str, + stack_id: str, + *, + score: float = 7.0, +) -> EvalRow: + return EvalRow( + eval_id="b" * 16, + model_id=model_id, + stack_id=stack_id, + task_id="be_01_jwt_auth", + seed=1, + status=EvalStatus.SCORED, + error_class=None, + artifact_refs=EvalArtifactRefs( + raw_output=_ref(), normalized_output=_ref(), evaluator_json=_ref() + ), + stats=EvalStats( + input_tokens=100, output_tokens=50, wall_clock_ms=10_000, cost_usd=Decimal("0.05") + ), + final_score=score, + judge_aggregate=None, + started_at=datetime.now(UTC), + completed_at=datetime.now(UTC), + ) + + +def _board_from_rows(rows: list[EvalRow]) -> Board: + return build_board(rows, stacks_root=_STACKS_ROOT, run_hash=_RUN_HASH, run_type="smoke") + + +def _write_board(path: Path, rows: list[EvalRow]) -> Board: + board = _board_from_rows(rows) + path.write_text(board.model_dump_json(indent=2) + "\n", encoding="utf-8") + return board + + +# --------------------------------------------------------------------------- +# 1. TestComputeGap +# --------------------------------------------------------------------------- + + +class TestComputeGap: + """_compute_gap returns desired - present correctly.""" + + def test_empty_board_all_desired_are_missing(self, tmp_path: Path) -> None: + """When board.json has no cells, every desired pair is reported as missing.""" + board_path = tmp_path / "board.json" + # Write a board with zero cells. + empty_board = _board_from_rows([]) + board_path.write_text(empty_board.model_dump_json(indent=2) + "\n", encoding="utf-8") + + missing = _compute_gap(board_path, stacks=["aider"]) + expected = {(m, "aider") for m in _STACK_MODELS["aider"]} + assert set(missing) == expected + + def test_present_cells_excluded_from_gap(self, tmp_path: Path) -> None: + """Cells already on the board are NOT included in the gap.""" + board_path = tmp_path / "board.json" + present_model = _STACK_MODELS["aider"][0] + _write_board(board_path, [_row(present_model, "aider")]) + + missing = _compute_gap(board_path, stacks=["aider"]) + assert (present_model, "aider") not in missing + # Remaining aider models should be in the gap. + expected_missing = {(m, "aider") for m in _STACK_MODELS["aider"] if m != present_model} + assert set(missing) == expected_missing + + def test_all_cells_present_returns_empty_gap(self, tmp_path: Path) -> None: + """When every desired cell is on the board, gap is empty.""" + board_path = tmp_path / "board.json" + rows = [_row(m, "aider") for m in _STACK_MODELS["aider"]] + _write_board(board_path, rows) + + missing = _compute_gap(board_path, stacks=["aider"]) + assert missing == [] + + def test_stacks_filter_limits_desired_set(self, tmp_path: Path) -> None: + """Only stacks in the filter contribute to the desired set.""" + board_path = tmp_path / "board.json" + _write_board(board_path, []) + + missing = _compute_gap(board_path, stacks=["aider"]) + stacks_in_gap = {stack_id for _, stack_id in missing} + assert stacks_in_gap == {"aider"} + # goose should not appear when filter is aider-only. + assert "goose" not in stacks_in_gap + + def test_none_stacks_filter_includes_all_stacks(self, tmp_path: Path) -> None: + """stacks=None includes all stacks from _STACK_MODELS.""" + board_path = tmp_path / "board.json" + _write_board(board_path, []) + + missing = _compute_gap(board_path, stacks=None) + stacks_in_gap = {stack_id for _, stack_id in missing} + assert stacks_in_gap == set(_STACK_MODELS.keys()) + + def test_missing_board_file_treats_as_empty(self, tmp_path: Path) -> None: + """When board.json doesn't exist, all desired cells are missing.""" + board_path = tmp_path / "nonexistent.json" + missing = _compute_gap(board_path, stacks=["aider"]) + expected = {(m, "aider") for m in _STACK_MODELS["aider"]} + assert set(missing) == expected + + def test_gap_is_sorted(self, tmp_path: Path) -> None: + """Gap list is sorted for deterministic output.""" + board_path = tmp_path / "board.json" + _write_board(board_path, []) + missing = _compute_gap(board_path, stacks=["aider", "goose"]) + assert missing == sorted(missing) + + def test_cross_stack_no_bleeding(self, tmp_path: Path) -> None: + """A present (model, aider) cell does NOT suppress (model, goose) from the gap.""" + board_path = tmp_path / "board.json" + shared_model = _STACK_MODELS["aider"][0] + _write_board(board_path, [_row(shared_model, "aider")]) + + missing = _compute_gap(board_path, stacks=["aider", "goose"]) + missing_set = set(missing) + # aider cell is present → not in gap. + assert (shared_model, "aider") not in missing_set + # goose cell for same model is NOT present → must be in gap. + if shared_model in _STACK_MODELS.get("goose", []): + assert (shared_model, "goose") in missing_set + + +# --------------------------------------------------------------------------- +# 2. TestMergeIntoBoard +# --------------------------------------------------------------------------- + + +class TestMergeIntoBoard: + """_merge_into_board is correct and idempotent.""" + + def test_merge_adds_new_cell(self) -> None: + """A cell from partial that is absent from existing is appended.""" + existing = _board_from_rows([_row("qwen-3-14b", "raw-llm")]) + partial = _board_from_rows([_row("qwen-3-14b", "aider")]) + merged = _merge_into_board(existing, partial) + keys = {(c.model_id, c.stack_id) for c in merged.cells} + assert ("qwen-3-14b", "raw-llm") in keys + assert ("qwen-3-14b", "aider") in keys + + def test_merge_replaces_existing_cell(self) -> None: + """A cell from partial that already exists in existing is replaced.""" + existing = _board_from_rows([_row("qwen-3-14b", "aider", score=5.0)]) + partial = _board_from_rows([_row("qwen-3-14b", "aider", score=9.0)]) + merged = _merge_into_board(existing, partial) + cell = next(c for c in merged.cells if c.model_id == "qwen-3-14b" and c.stack_id == "aider") + assert cell.mean_score == 9.0 + + def test_merge_is_idempotent(self) -> None: + """Merging the same partial twice produces the same result as merging once.""" + existing = _board_from_rows([_row("qwen-3-14b", "raw-llm")]) + partial = _board_from_rows([_row("qwen-3-14b", "aider", score=7.0)]) + once = _merge_into_board(existing, partial) + twice = _merge_into_board(once, partial) + # Cells should be identical sets. + keys_once = {(c.model_id, c.stack_id) for c in once.cells} + keys_twice = {(c.model_id, c.stack_id) for c in twice.cells} + assert keys_once == keys_twice + # Score should not drift. + score_once = next( + c.mean_score for c in once.cells if c.model_id == "qwen-3-14b" and c.stack_id == "aider" + ) + score_twice = next( + c.mean_score + for c in twice.cells + if c.model_id == "qwen-3-14b" and c.stack_id == "aider" + ) + assert score_once == score_twice + + def test_merge_unions_harnesses(self) -> None: + """New harness entries from partial are added to the merged board.""" + existing = _board_from_rows([_row("qwen-3-14b", "raw-llm")]) + partial = _board_from_rows([_row("qwen-3-14b", "aider")]) + merged = _merge_into_board(existing, partial) + harness_ids = {h.stack_id for h in merged.harnesses} + assert "raw-llm" in harness_ids + assert "aider" in harness_ids + + def test_merge_does_not_mutate_existing(self) -> None: + """_merge_into_board returns a new Board; existing is unchanged.""" + existing = _board_from_rows([_row("qwen-3-14b", "raw-llm")]) + partial = _board_from_rows([_row("qwen-3-14b", "aider")]) + original_cell_count = len(existing.cells) + _ = _merge_into_board(existing, partial) + assert len(existing.cells) == original_cell_count + + def test_merge_scored_flag(self) -> None: + """scored=True when at least one merged cell has a score.""" + existing = _board_from_rows([]) + partial = _board_from_rows([_row("qwen-3-14b", "aider", score=7.0)]) + merged = _merge_into_board(existing, partial) + assert merged.scored is True + + +# --------------------------------------------------------------------------- +# 3. TestDryRun (CLI entry-point via async _main) +# --------------------------------------------------------------------------- + + +class TestDryRun: + """--fill-missing --dry-run must print gap summary and exit 0 without spending.""" + + @pytest.mark.asyncio + async def test_dry_run_exits_zero(self, tmp_path: Path, monkeypatch: Any) -> None: + """--fill-missing --dry-run exits 0 regardless of gap size.""" + import argparse + + import build_real_board as brb + + # Write a board with zero aider cells so gap is non-empty. + board_path = tmp_path / "board.json" + _write_board(board_path, []) + monkeypatch.setattr(brb, "REPO", tmp_path) + # Patch out env key requirement. + monkeypatch.setenv("LITELLM_MASTER_KEY", "fake-key-for-test") + # Provide the board.json at the expected path. + board_dest = tmp_path / "apps" / "site" / "public" + board_dest.mkdir(parents=True) + _write_board(board_dest / "board.json", []) + + # Invoke via argparse simulation. + args = argparse.Namespace( + confirm_spend=False, + fill="", + add_stack="", + fill_missing=True, + stacks="aider", + dry_run=True, + ) + + # Monkeypatch parse_args so _main() picks up our args namespace. + import argparse as ap_mod + + monkeypatch.setattr(ap_mod.ArgumentParser, "parse_args", lambda self, *a, **kw: args) + # chdir is called by _main; patch it to avoid side effects. + monkeypatch.setattr("os.chdir", lambda p: None) + # _load_env is harmless but avoid file I/O. + monkeypatch.setattr(brb, "_load_env", lambda repo: None) + + exit_code = await brb._main() + assert exit_code == 0 + + @pytest.mark.asyncio + async def test_dry_run_empty_gap_exits_zero(self, tmp_path: Path, monkeypatch: Any) -> None: + """When gap is empty, --dry-run exits 0 with 'EMPTY' message.""" + import argparse + + import build_real_board as brb + + # Write a fully-populated aider column. + board_dest = tmp_path / "apps" / "site" / "public" + board_dest.mkdir(parents=True) + rows = [_row(m, "aider") for m in _STACK_MODELS["aider"]] + _write_board(board_dest / "board.json", rows) + + monkeypatch.setattr(brb, "REPO", tmp_path) + monkeypatch.setenv("LITELLM_MASTER_KEY", "fake-key-for-test") + + args = argparse.Namespace( + confirm_spend=False, + fill="", + add_stack="", + fill_missing=True, + stacks="aider", + dry_run=True, + ) + + import argparse as ap_mod + + monkeypatch.setattr(ap_mod.ArgumentParser, "parse_args", lambda self, *a, **kw: args) + monkeypatch.setattr("os.chdir", lambda p: None) + monkeypatch.setattr(brb, "_load_env", lambda repo: None) + + exit_code = await brb._main() + assert exit_code == 0 + + def test_dry_run_computes_correct_gap_subset(self, tmp_path: Path) -> None: + """_compute_gap with one stack present returns exactly the remaining models.""" + board_path = tmp_path / "board.json" + present = _STACK_MODELS["aider"][:1] + absent = _STACK_MODELS["aider"][1:] + _write_board(board_path, [_row(m, "aider") for m in present]) + + missing = _compute_gap(board_path, stacks=["aider"]) + missing_models = [m for m, s in missing if s == "aider"] + assert set(missing_models) == set(absent) + # Present model must not appear. + assert present[0] not in missing_models diff --git a/evals/task-packs/be_01_jwt_auth/rubric.yaml b/evals/task-packs/be_01_jwt_auth/rubric.yaml index 915f3b4..f392d08 100644 --- a/evals/task-packs/be_01_jwt_auth/rubric.yaml +++ b/evals/task-packs/be_01_jwt_auth/rubric.yaml @@ -12,7 +12,7 @@ schema_version: pollmevals.rubric.v1 task_id: be_01_jwt_auth -rubric_version: "1.0" +rubric_version: "2.0" # methodology v0.2: type_safety criterion → design_appropriateness (lint/typing are deterministic; judges score subjective axes only) sourcing: own # ADR-007 Tier 1 license: spec: CC-BY-SA-4.0 @@ -85,24 +85,30 @@ criteria: next(err) when appropriate. No try/catch swallowing without logging. Logs structured (one JSON object per event). - type_safety: + design_appropriateness: weight: 0.15 description: | - Strict mode compliance. No `any`. No `as` casts on user-provided - values. JWT payload narrowed via type guard (jsonwebtoken returns - string | JwtPayload; narrowing required). Discriminated unions or - branded types for token kinds where appropriate. + Is the SOLUTION SHAPE well-chosen for the task — beyond what a compiler or + linter can check? (Type-correctness and lint are scored SEPARATELY by the + deterministic tsc/eslint evaluators per methodology v0.2 — judges do NOT + re-grade them, to avoid double-counting one signal.) Judge here: middleware + composition and layering, separation of auth concerns, naming that maps to + the JWT/refresh domain, idiomatic Express patterns, and whether the + abstraction boundaries (token verify vs cookie rotation vs store) are drawn + where a senior engineer would draw them. anchors: 0: | - Liberal `any` use; `as unknown as Whatever` casts on req.body; - strict mode disabled or many ts-ignore comments. + One monolithic handler; auth, rotation, and storage tangled together. + Structure fights the framework (manual header parsing where Express + idioms exist). A reviewer would redesign it from scratch. 5: | - Strict mode on, but one or two unsafe casts on jwt.verify return - value or req.cookies. Payload accessed without narrowing. + Reasonable decomposition but one boundary is off — e.g. cookie rotation + bleeds into the verify path, or the store contract leaks framework types. + Works, but a senior would ask for one structural change. 10: | - Zero `any`, zero `as` casts on external input. jwt.verify return - narrowed via type guard before use. Public API types match prompt - verbatim. No ts-ignore. + Clean, idiomatic composition. Verify / rotate / store are cleanly + separated with intention-revealing boundaries. The shape is what an + experienced Express+JWT engineer would ship. Nothing to restructure. code_clarity: weight: 0.10 @@ -148,7 +154,7 @@ output_schema: correctness: 0..10 security_posture: 0..10 error_handling: 0..10 - type_safety: 0..10 + design_appropriateness: 0..10 code_clarity: 0..10 test_alignment: 0..10 total_score: 0..10 # weighted sum per criteria.weight above diff --git a/infra/litellm-config.yaml b/infra/litellm-config.yaml index 7708531..36275e5 100644 --- a/infra/litellm-config.yaml +++ b/infra/litellm-config.yaml @@ -240,6 +240,49 @@ model_list: # the JUDGE alias only — candidate gpt-5 use is unaffected. reasoning_effort: low + # ── methodology v0.2 (2026-06-03): 5-frontier diverse judge panel ─────────── + # Decision (user, 2026-06-03): judge on the 5 most powerful June-2026 models, + # 5 distinct vendor families (Anthropic/OpenAI/Google/xAI/DeepSeek) — diversity + # reduces correlated bias more than raw count (Research-B prior-art; our 2-judge + # α was 0.17-0.36, frontier judges show 0.80-0.91 self-consistency). All billed + # to OPENROUTER_API_KEY_JUDGE (NFR-005 isolation). reasoning_effort=low caps the + # reasoning budget so the full rubric JSON fits the 2048-tok cap (EVID-050 + # pattern); providers that don't accept it drop it silently (drop_params=true). + - model_name: claude-opus-4-8-judge + litellm_params: + model: openrouter/anthropic/claude-opus-4.8 + api_key: os.environ/OPENROUTER_API_KEY_JUDGE + api_base: https://openrouter.ai/api/v1 + reasoning_effort: low + + - model_name: gpt-5-5-judge + litellm_params: + model: openrouter/openai/gpt-5.5 + api_key: os.environ/OPENROUTER_API_KEY_JUDGE + api_base: https://openrouter.ai/api/v1 + reasoning_effort: low + + - model_name: gemini-3-1-pro-judge + litellm_params: + model: openrouter/google/gemini-3.1-pro-preview + api_key: os.environ/OPENROUTER_API_KEY_JUDGE + api_base: https://openrouter.ai/api/v1 + reasoning_effort: low + + - model_name: grok-4-judge + litellm_params: + model: openrouter/x-ai/grok-4.20 + api_key: os.environ/OPENROUTER_API_KEY_JUDGE + api_base: https://openrouter.ai/api/v1 + reasoning_effort: low + + - model_name: deepseek-v4-pro-judge + litellm_params: + model: openrouter/deepseek/deepseek-v4-pro + api_key: os.environ/OPENROUTER_API_KEY_JUDGE + api_base: https://openrouter.ai/api/v1 + reasoning_effort: low + # General proxy settings litellm_settings: # Prometheus metrics at GET /metrics — DB available so this works.