Adopt agent-034/030 judge conversions; rerun claude-fable-5 at the new pin#93
Merged
Conversation
jerilynzheng
approved these changes
Jul 3, 2026
…w pin - Bump the pinned next.js SHA to the merged canary commit for vercel/next.js#95440 (agent-034 + agent-030 converted to the agentic LLM judge; agent-040 renamed to agent-040-instant upstream). - Rerun claude-fable-5 on its outstanding evals and commit the results: agent-030 and agent-034 pass with the judge clauses executing in-sandbox; agent-040-instant fails 0/4 (classified model failure — never exported `instant`). fable-5 is fully current, so it has no ACCEPTED_STALE entry. - Reshape check-stale.mjs to (experiment, eval) pairs: ACCEPTED_STALE maps each experiment to exactly the evals it may stay stale on, so any new staleness on an accepted experiment still fails CI, and refreshing a model deletes its pairs.
gaojude
force-pushed
the
jude/sync-agent-024-judge
branch
from
b0f5857 to
9fac5e3
Compare
gaojude
changed the title
… eval set (#94) - gpt-5.6-xhigh + gpt-5.6-xhigh--agents-md: direct OpenAI codex, reasoningEffort=xhigh. Eval-level: 21/24 base, 22/24 with AGENTS.md. All 0% failures classified + transcript-verified as model failures. - export-results: score every experiment over the current evals/ set. Results whose eval is missing or stale (new/changed per agent-eval status --json) export as notAvailable, count against the success rate, and are excluded from flip detection and avg duration. - Restore missing MODEL_NAMES/AGENTS_MD_PAIRS entries for claude-opus-4.8, cursor-composer-2.5, gpt-5.5-pro.
Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid
effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base,
22/24 with AGENTS.md. All 0% evals verified as model failures at the
assertion level; agent-040-instant remains unbeaten by any model.
The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed.
This was referenced Jul 8, 2026
…95) * Add GPT 5.6 Sol (xhigh) eval results Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base, 22/24 with AGENTS.md. All 0% evals verified as model failures at the assertion level; agent-040-instant remains unbeaten by any model. * Make Claude Fable 5's effort level explicit (high) The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed.
vercel-ai-gateway/opencode with the model defined via extraProviders (released 2026-07-08, not in the opencode catalogs yet; GLM 5.1 binary reused; limits from AI Gateway metadata). Eval-level: 20/24 base, 23/24 with AGENTS.md (+13, the largest docs delta measured). All 0% evals verified as model failures: <a> vs next/link (agent-000), middleware.ts vs proxy.ts (agent-031/039), missing instant export (agent-040 — still unbeaten by any model).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts the regex→agentic-judge eval conversions that landed in vercel/next.js#95440, exercising the incremental staleness workflow from #92 end-to-end.
What changed upstream.
agent-034-async-cookiesandagent-030-app-router-migration-hardnow grade their false-negative-prone checks with@vercel/agent-eval's LLM judge (fixed haiku grader) instead of regex. Separately,agent-040-unstable-instantwas renamed toagent-040-instantbetween the old pin and canary, so it counts as a new eval with no results.What syncing detected.
pnpm sync-evals 0240fd3+agent-eval status --json: exactlyagent-030+agent-034changed andagent-040-instantnew, across all 44 experiments — the other 21 evals stayed cached everywhere.What this PR does.
0240fd3).claude-fable-5on its outstanding evals and commits the results; itsACCEPTED_STALEentry is removed, so CI enforces it fresh.scripts/check-stale.mjsto (experiment, eval) pairs:ACCEPTED_STALEnow maps each experiment to exactly the evals it may stay stale on. Tighter than the old experiment-level list — any new staleness on an accepted experiment still fails CI, and refreshing a model deletes its pairs so enforcement tightens automatically.Un-rerun models keep serving their existing results on the dashboard (the export doesn't gate on fingerprints); their agent-030/034 numbers were graded by the old regex evals until each is refreshed — for these two evals a refresh can only move numbers up, since the judge accepts strictly more correct solutions than the regex did.