Add Grok 4.5 eval results#96
Merged
Merged
Conversation
vercel-ai-gateway/opencode with the model defined via extraProviders (released 2026-07-08, not in the opencode catalogs yet; GLM 5.1 binary reused; limits from AI Gateway metadata). Eval-level: 20/24 base, 23/24 with AGENTS.md (+13, the largest docs delta measured). All 0% evals verified as model failures: <a> vs next/link (agent-000), middleware.ts vs proxy.ts (agent-031/039), missing instant export (agent-040 — still unbeaten by any model).
gaojude
added a commit
that referenced
this pull request
Jul 8, 2026
gaojude
added a commit
that referenced
this pull request
Jul 9, 2026
…w pin (#93) * Adopt agent-034/030 judge conversions; rerun claude-fable-5 at the new pin - Bump the pinned next.js SHA to the merged canary commit for vercel/next.js#95440 (agent-034 + agent-030 converted to the agentic LLM judge; agent-040 renamed to agent-040-instant upstream). - Rerun claude-fable-5 on its outstanding evals and commit the results: agent-030 and agent-034 pass with the judge clauses executing in-sandbox; agent-040-instant fails 0/4 (classified model failure — never exported `instant`). fable-5 is fully current, so it has no ACCEPTED_STALE entry. - Reshape check-stale.mjs to (experiment, eval) pairs: ACCEPTED_STALE maps each experiment to exactly the evals it may stay stale on, so any new staleness on an accepted experiment still fails CI, and refreshing a model deletes its pairs. * Add GPT 5.6 (xhigh) eval results; normalize export over the canonical eval set (#94) - gpt-5.6-xhigh + gpt-5.6-xhigh--agents-md: direct OpenAI codex, reasoningEffort=xhigh. Eval-level: 21/24 base, 22/24 with AGENTS.md. All 0% failures classified + transcript-verified as model failures. - export-results: score every experiment over the current evals/ set. Results whose eval is missing or stale (new/changed per agent-eval status --json) export as notAvailable, count against the success rate, and are excluded from flip detection and avg duration. - Restore missing MODEL_NAMES/AGENTS_MD_PAIRS entries for claude-opus-4.8, cursor-composer-2.5, gpt-5.5-pro. * Add GPT 5.6 Sol (xhigh) eval results Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base, 22/24 with AGENTS.md. All 0% evals verified as model failures at the assertion level; agent-040-instant remains unbeaten by any model. * Make Claude Fable 5's effort level explicit (high) The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed. * Add GPT 5.6 Sol (xhigh) evals; make Fable 5's effort explicit (high) (#95) * Add GPT 5.6 Sol (xhigh) eval results Direct OpenAI codex at reasoningEffort=xhigh ('ultra' is not a valid effort — the API rejects it; xhigh is the max). Eval-level: 20/24 base, 22/24 with AGENTS.md. All 0% evals verified as model failures at the assertion level; agent-040-instant remains unbeaten by any model. * Make Claude Fable 5's effort level explicit (high) The runs used the Claude Code default for Fable 5, which is high (code.claude.com/docs/en/model-config.md). agentOptions.effort now pins it and the display name shows it, matching Opus 4.7 (max). agentOptions is not part of the result fingerprint, so cached results stay valid — no rerun needed. * Add Grok 4.5 eval results (#96) vercel-ai-gateway/opencode with the model defined via extraProviders (released 2026-07-08, not in the opencode catalogs yet; GLM 5.1 binary reused; limits from AI Gateway metadata). Eval-level: 20/24 base, 23/24 with AGENTS.md (+13, the largest docs delta measured). All 0% evals verified as model failures: <a> vs next/link (agent-000), middleware.ts vs proxy.ts (agent-031/039), missing instant export (agent-040 — still unbeaten by any model).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Grok 4.5 (released today) via
vercel-ai-gateway/opencode, model defined throughextraProviderssince it's not in the opencode catalogs yet (GLM 5.1 binary reused, limits from AI Gateway metadata): 20/24 base, 23/24 with AGENTS.md — the largest docs delta measured (+13), flipping agent-000/031/039.All 0% evals verified as model failures at the assertion level: plain
<a>instead ofnext/link(agent-000),middleware.tsinstead ofproxy.ts(agent-031/039), missinginstantexport (agent-040, still unbeaten by any model).Stacked on #95 → #93 — retarget after those merge.