llm-benchmark

A curated list on AI token economics: what tokens cost, where they get wasted, and how to cut the bill. Tools, benchmarks, papers, and copy-paste configs for the token economy of LLMs and coding agents.

A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.

Bug Hunt Bench: 105 real bugs in two production repos, frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek...) find and fix them in their own CLI, graded blind. Live leaderboard + every receipt.

An Open Harness and Benchmark for AI in Cybersecurity Operations.

The AI-native wire format for structured data. 100% comprehension on every frontier model. 50-92% fewer tokens than JSON. 43B+ lossless round-trips across 17 formats. Spec v3.5.1 Stable.

Community benchmark database for running LLMs on Apple Silicon Macs

Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением в 1С — Claude, GPT, Gemini, DeepSeek, YandexGPT, GigaChat.

[ICCV 2025] AdsQA: Towards Advertisement Video Understanding Arxiv: https://arxiv.org/abs/2509.08621

ICRTL Benchmark: Industrial-level RTL design challenges for evaluating PPA optimization, code generation, and LLM applications in EDA.

An LLM Benchmark evaluation platform for automated model testing and analysis. 面向大语言模型的 Benchmark 自动化评测平台，支持模型测试与结果分析

Open-source benchmark for evaluating LLMs on 220 real professional tasks across 9 sectors and 44 occupations. Reproducible experiments, artifact validation, grading, and a live evidence dashboard.

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Daily LLM value rankings - compare 300+ models by intelligence, speed and price. OpenRouter + Artificial Analysis. 大模型性价比排行榜

Catch local LLMs spilling into the CPU. One run shows GPU placement, prefill, decode, memory, power and a verdict.

tool-eval-bench results for Prism ML Ternary-Bonsai-27B (Q2_0) — 8 trials, score 85, deployability 80

Open-source benchmark for evaluating LLMs on domain-specific geological reasoning, AI for geology, and source-grounded geological knowledge.

Исследовательский вопрос: можно ли измерить «офисный интеллект» LLM? Попытка — здесь. 100 сценариев, 10 критериев, русский корпоративный контекст.

[ICML 2026] CapBencher toolkit: Give your LLM benchmark a built-in alarm for leakage and gaming

Benchmark for evaluating AI epistemic reliability - testing how well LLMs handle uncertainty, avoid hallucinations, and acknowledge what they don't know.

Specification for a reproducible, provenance-bound multi-lane LLM benchmark suite.