A curated dashboard of the frontier in AI evaluation. Updated Daily — Target 8 Papers/Day.
| Resource | Description |
|---|---|
| 📅 Full Archive | Chronological history of every Daily 8 entry |
| 📚 Benchmark Reference | Curated stable reference — 200+ benchmarks by domain |
| 🗂️ Survey Index | All deep-dive notes organised by topic area |
| 📋 Weekly Digest | Rolling weekly summary with industry trust gaps |
| Paper | Modality | Key Finding | Notes | arXiv |
|---|---|---|---|---|
| LongMemEval-V2 | LLM | 451 questions / 115M-token histories / 5 memory abilities: AgentRunbook-C 72.5% vs. RAG baseline 48.5% — RAG insufficient for environment-specific expertise accumulation | 📄 | 🔗 |
| TokenHD | LLM | Token-level hallucination detector: 0.6B trained model surpasses QwQ-32B; scales consistently 0.6B→8B; no step segmentation required | 📄 | 🔗 |
| Rollout Cards | LLM | 0/50 audited agent repos disclose failed runs; reporting rule changes alone shift scores ±20.9 pp and invert frontier model rankings | 📄 | 🔗 |
| Principal Hierarchies | LLM | 10 frontier models / 7,136 scenarios: models adhere to professional standards in advisory contexts but abandon them during task execution; reasoning traces suppress recognised knowledge under authority pressure | 📄 | 🔗 |
| MedHopQA | LLM | BioCreative IX: 10,000 questions (1K scored/9K withheld); disease-centered 2-article multi-hop synthesis; open-ended format blocks answer-elimination shortcut | 📄 | 🔗 |
| MedMemoryBench | LLM | ~2,000 sessions / ~16,000 turns: memory saturation is universal — all mainstream architectures degrade as clinical information accumulates; complex medical reasoning and noise resilience are fundamental gaps | 📄 | 🔗 |
| ATC Safety Eval | LLM | Peak Risk Score 0.69 for best model; most LLMs score <0.6 despite high macro-F1 — aggregate accuracy hides unsafe error concentration in high-consequence entity categories | 📄 | 🔗 |
| Rubric RL Hacking | LLM | Rubric-based RL reward hacking: weak verifiers yield large proxy gains that don't transfer; exploitation grows over training; self-internalization gap detects it without holdout verifiers | 📄 | 🔗 |
| Model | Lab | Key Result | Notes | Link |
|---|---|---|---|---|
| Claude Opus 4.8 | Anthropic | 69.2% SWE-Bench Pro / 57.9% HLE (tools) / 83.4% OSWorld-Verified / 1890 GDPval-AA / 84% Online-Mind2Web / first >10% Legal Agent all-pass / $5/$25 per M tokens | 📄 | 🔗 |
| Kimi-K2.6 | Moonshot AI | 54.0 HLE-Full (beats GPT-5.4) / 58.6 SWE-Bench Pro (best open-weight) / 96.4 AIME 2026 / 1T MoE / Mod. MIT | 📄 | 🔗 |
| Mistral Medium 3.5 | Mistral AI | 77.6% SWE-Bench Verified / 91.4% τ³-Telecom / 128B dense / replaces 3 prior models / per-request reasoning | 📄 | 🔗 |
| Qwen3.6-27B | Alibaba | 77.2% SWE-Bench Verified at 27B / 94.1 AIME 2026 / 70.3% AndroidWorld / 1M-token context / Apache-2.0 | 📄 | 🔗 |
| GPT-5.4 | OpenAI | 75.0% OSWorld-V | 📄 | 🔗 |
| GPT-5.4 mini/nano | OpenAI | OSWorld-mini efficiency leader | 📄 | 🔗 |
| Gemini 3.1 Pro | 77.1% ARC-AGI-2 | 📄 | 🔗 | |
| Phi-4 Reasoning | Microsoft | 15B model, strong MathVista | 📄 | 🔗 |
| DeepSeek-V4-Pro | DeepSeek | 93.5 LiveCodeBench / 80.6% SWE-Verified / 1M-token context at 27% V3.2 FLOPs / MIT licence | 📄 | 🔗 |
| DeepSeek-V3.2 | DeepSeek | RL-based logic, MMLU-Pro + IMO | 📄 | 🔗 |
| ARC-AGI-2 | ARC Prize | Non-semantic visual abduction | 📄 | 🔗 |
| ERNIE 5.0 | Baidu | Trillion-param unified MoE | 📄 | 🔗 |
| Emu3.5 | BAAI | 94.03 TIIF-Bench; 20× gen speedup | 📄 | 🔗 |
| Qwen3.5-Omni | Alibaba | WER 1.11 Librispeech; 119 languages | 📄 | 🔗 |
A prioritised checklist of benchmarks to run against enterprise LLM/VLM deployments — ordered by deployment risk and evaluation ROI.
- SimpleQA — hallucination stress-test on every checkpoint before release
- HalluLens — extrinsic + intrinsic hallucination suites for any RAG or factual output pipeline
- TraceSafe-Bench — mid-trajectory tool-calling safety before any agentic deployment
- HINTBench — intrinsic agent safety validation on long agentic trajectories
- MemEvoBench — memory safety degradation check before persistent agent deployment
- MedSkillAudit — pre-release skill audit gate (target: <57% below threshold)
- MMLU-Pro — multi-domain knowledge baseline across 14 disciplines
- AgentBench — agent capability baseline before and after every major architecture upgrade
- SWE-bench Verified — code engineering capability baseline
- StructEval — structured output reliability across 18 formats
- MIRROR — metacognitive calibration; measure Compositional Calibration Error before deployment
- SOB — structured extraction accuracy on text/image/audio before any data pipeline deployment
- KWBench — unprompted problem recognition in professional knowledge work
- MATHVERSE Vision-Only — confirm genuine diagram reading vs. text shortcuts before engineering schematics deployment
- VLM-RobustBench — geometric robustness before any factory-floor visual deployment
- TPS-CalcBench — engineering calculation accuracy before safety-critical design tasks
- IndicDB / Semantic Layers Bench — text-to-SQL accuracy; build semantic layer before deployment
- AutomationBench — cross-app workflow orchestration; expect <10% without specialised training
- GTA-2 — multi-step tool workflow completion; expect <15% for complex workflows
- LongSumEval — QA-based summarisation fidelity before document summarisation deployment
- Cyber Defense Benchmark — threat-hunting recall before any SOC/security assistant deployment
- MuDABench — multi-document analytical QA before enterprise knowledge base deployment
- AgentSearchBench — agent discovery accuracy before routing layer deployment in multi-agent systems
Built with ❤️ for the AI evaluation community. Contributions welcome — see ARCHIVE.md for the full history.