feat(forge): real-LLM runs, S6 judge-ladder, real contamination probe, calibration queue - #1
Merged
Merged
Conversation
Complete handoff document for the Calibrated Evaluation Foundry: the three-part thesis (benchmarks die of absorption, not saturation; the ARC-style methodology is a tunable system whose parameters are opinions until measured; every property is provable with an artifact), the premises extracted from both Airbnb articles (EDD principles + judge calibration; deterministic foundation, dual indeterminacy, micro-adapters, seam validation) and the benchmark-design literature (ARC-AGI-1/2/3, JudgeBench, METR evals-as-scaling), the full pipeline map, all study results (S1-S5 with exact numbers verified against the artifacts), honest limits, open paths in priority order, and run instructions.
…, calibration queue Executes every HANDOFF §7 open path this pass can reach, plus the nits a verification pass found: - fix: study.py dead docstring code, benchmark.py double oracle.calibrate() call, stratify.py KL-divergence doc mismatch, LlmSolver silent except - fix: LlmSolver never sent an Authorization header — real endpoints silently failed every call. Added bearer-token auth + --llm-api-key/ --llm-name CLI flags. - feat: real-LLM run against DeepSeek at two scales (23-task pilot RHAE 0.826; 60-task confirmatory RHAE 0.907, 90.7% solve). First real-model data point this pipeline has produced. No local Ollama available this session — flagged as an honest limit, not run. - feat: run_llm_leak_probes + format_hint (contamination.py) — real-model contamination probe (S4b), 0/20 false-fire on DeepSeek, never-leaked tasks. - feat: S6 judge-ladder budget study — real DeepSeek judge cuts 10-18% of human-attempt budget at no monotonicity cost; the judge's own difficulty ratings turned out compressed, an empirical instance of S1's difficulty-compression failure mode. - feat: calibration_queue.py (CalibrationQueue + RealHumanOracle) — protocol-conformant real-human calibration path, proven to drop into run_benchmark(oracle=...) with zero other changes. No live human data; calibrate() raises rather than fabricate. - docs: deep rewrite of docs/blog-eval-foundry.md (pushed to the live Notion draft) — critiques Airbnb's prescriptive-not-measured gap explicitly, SOTA lit review (agentic trajectory eval, RLVR reward hacking, contamination detection beyond n-gram, judge reliability), absorption-vs-saturation thesis stress-tested against the obvious saturation-relabeling counterargument using S1/S3. docs/HANDOFF.md synced with the same numbers. 19 new tests (233/233 total green). Harder task domains (open path #4) explicitly scoped out this pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Executes every HANDOFF §7 open path this pass can reach, plus nits a verification pass found.
Summary
oracle.calibrate()call, stratify.py KL-divergence doc mismatch, LlmSolver silent exceptLlmSolvernever sent anAuthorizationheader — real endpoints silently failed every call. Added bearer-token auth +--llm-api-key/--llm-nameCLI flags.run_llm_leak_probes+format_hint(contamination.py) — real-model contamination probe (S4b), 0/20 false-fire on DeepSeek, never-leaked tasks.calibration_queue.py(CalibrationQueue+RealHumanOracle) — protocol-conformant real-human calibration path, proven to drop intorun_benchmark(oracle=...)with zero other changes. No live human data;calibrate()raises rather than fabricate.docs/blog-eval-foundry.md(pushed to the live Notion draft) +docs/HANDOFF.mdsynced.19 new tests (233/233 total green). Harder task domains (open path #4) explicitly scoped out this pass.
🤖 Generated with Claude Code