Skill: public benchmark catalog with tag taxonomy and direct model score links

[DiTo97]

Take-home assignment. Self-contained and vertical: taxonomy design → skill definition → process doc → ingestion code → storage → tests. Read the whole issue before starting; the acceptance criteria are the contract.

Problem

We have no benchmark data at all. Grepping the tree for benchmark|mmlu|gpqa|swe.?bench|livebench|lmeval|artificialanalysis returns zero substantive hits outside a stray string in a lockfile.

For tokenvalue to answer "best model for this task", a price and an arena rank are not enough. We need a catalog of public benchmarks with real tags — so that "this is a long-context multilingual retrieval task" can be turned into "these benchmarks are the relevant evidence" — and we need direct (model, reasoning_effort) → benchmark → score links wherever a published score actually exists.

The tagging is the hard and valuable part. A flat list of benchmark names is nearly useless; a benchmark tagged by modality, focus, scope, complexity, context-window sizing, and language is a routing table.

As with the arena work, this ships as a skill — an agentskills.io-style SKILL.md plus an explicit process doc, harness-agnostic, with deterministic checked-in code doing the parsing and validation. No CI cron, no API keys or paid subscriptions we do not already hold.

Scope

• Benchmark registry. A catalog entity per benchmark: stable id, name, maintainer/source, canonical URL, what it measures, scoring scale and its direction (higher-is-better vs lower-is-better — getting this wrong silently inverts every ranking downstream), score bounds, evaluation protocol notes, and licensing/usage constraints where they matter.
• Tag taxonomy. A closed, documented controlled vocabulary across at least: modality, focus, scope, complexity, context-window sizing, language. Free-text tags are not acceptable — they cannot be queried reliably. Each dimension needs defined values with written definitions, and a rule for what to do when a benchmark does not fit. Justify your value sets; this taxonomy will be load-bearing for the query layer and for the estimation work.
• Score links. (model, reasoning_effort) → benchmark → value, keyed to the identity from the data-model foundation issue, with provenance (where the number came from), the date it was published or captured, and the evaluation conditions where they are stated. A score without provenance is not admissible.
• Sources. Cover a meaningful spread — arena.ai, LMEval / lm-evaluation-harness, Hugging Face papers and leaderboards, official benchmark repos and papers. Breadth of sources matters less than each ingested source being correct and honestly labelled; we would rather have four sources done properly than twelve done sloppily.
• Skill + process doc. skills/<name>/SKILL.md matching the conventions of skills/tokenpricing/SKILL.md, registered in .claude-plugin/marketplace.json, plus a process document detailing per-source acquisition: where to go, what to extract, how to normalize scales, when to refuse a number rather than guess it.
• Deterministic ingestion. Capture raw, then parse with tested code, then validate against a schema. The harness never writes canonical records directly.
• Storage. In database/, following the conventions in services/sync/src/tokenpricing_sync/paths.py. Current state plus history. Idempotent refresh — unchanged upstream must produce a no-op diff.
• Normalization across scales. Benchmarks report on incompatible scales (percentages, Elo, pass@k, absolute counts). Decide how comparability is achieved — and whether normalization belongs at ingestion or at query time. Argue your choice; both are defensible, one of them is wrong for this codebase.

Out of scope

• The remote-database / SQLite migration. Issues Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 are open and cover a derived SQLite read layer. Do not touch them or build on them. Storage stays git-committed JSON.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase.
• CI cron jobs, API keys, or paid subscriptions. Explicitly excluded.
• Artificial Analysis as a source. AA's eval columns (23 benchmarks including GPQA Diamond, LiveCodeBench, AIME 2025, MMMU Pro, Humanity's Last Exam, SciCode, Terminal-Bench) are acquired by Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85, which owns the AA scraper and its identity crosswalk. Do not build a second AA ingestion path here. What this issue does own for AA: registering those benchmarks in the registry with their scale, direction and bounds, so Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85's scores link against real registry entries rather than bare strings. Count AA as one of your four sources for registry purposes and let Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index #85 supply the score links.
• Estimating missing scores. This issue only records scores that genuinely exist. Filling gaps is the category→benchmark estimation issue — but design the schema so an estimate can be stored alongside a measurement and always distinguished from it.
• Running evaluations ourselves. We ingest published results only.

Suggested approach

1. Design the tag taxonomy before writing any scraper. Take ten real benchmarks spanning very different shapes, tag them by hand, and see where your vocabulary breaks. Iterate on paper. This is cheaper than discovering the gaps after ingestion.
2. Model the registry and the score links as separate entities with a clean join. Conflating "what this benchmark is" with "what this model scored" will hurt immediately.
3. Get scale direction and bounds into the schema as required fields, not optional metadata.
4. TDD the parsers against trimmed real fixtures checked into the test tree, as services/sync/tests/ does. No network in tests.
5. Write the process doc last and validate it cold — hand it to a fresh harness session or a colleague and see whether their output passes your validator.
6. Where a published score's conditions are ambiguous (which variant? which effort? which harness version?), prefer recording it as unresolved over guessing. An honest coverage gap is more useful than a confident wrong join.

Acceptance criteria

• Benchmark registry schema includes id, name, source, URL, scoring scale, scale direction, bounds, and protocol notes; implemented in Python with the TypeScript mirror kept consistent if the data is exposed through the SDK.
• Tag taxonomy is a closed controlled vocabulary covering modality, focus, scope, complexity, context-window sizing, and language, with each value defined in writing and the rationale recorded.
• Score links are keyed by (model, reasoning_effort) per the data-model foundation issue, with mandatory provenance and date, and a field distinguishing measured from (future) estimated values.
• At least four distinct public sources ingested end to end, each with a tested parser and no network access in tests.
• skills/<name>/SKILL.md exists, follows existing skill conventions, is registered in .claude-plugin/marketplace.json, and is accompanied by a per-source process document.
• Identity resolution reports coverage and an explicit unresolved list; nothing silently dropped or force-matched.
• Data lands in database/ per existing path conventions, with current state and history; unchanged upstream yields a no-op diff (tested).
• Cross-scale comparability approach is implemented and its placement (ingestion vs query time) argued in the PR.
• uv run pytest -q and uv run pre-commit run -a green.
• Docs explain the taxonomy, the sources, and the known limitations — including which benchmarks are known to be contaminated or disputed, where that is public knowledge.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python, pnpm for libraries/typescript, npm for services/dashboard. CI is path-filtered; pushes to main touching libraries/** auto-release.

The taxonomy design is the part we will read most closely. A defensible, well-argued vocabulary with four solid sources beats a shallow one with twelve.