Skill: Artificial Analysis acquisition - provider leaderboard (model x platform) + Openness Index

[DiTo97]

Take-home assignment. Self-contained and vertical: source characterisation → skill definition → process doc → deterministic ingestion → identity crosswalk → storage → tests. Read the whole issue before starting; the acceptance criteria are the contract.

Problem

We have no third-party measured serving performance and no openness/licensing signal at all. database/current/prices.json carries pricing, context window, and three capability booleans. Grepping the tree for latency|throughput|openness|artificialanalysis returns nothing substantive.

Artificial Analysis (AA) publishes both, and a feasibility spike has already proven we can acquire them reliably.

The spike is on branch spike/aa-data-acquisition (commit 95dc4ef), under experiments/aa-scrape-spike/. Read it before starting — it answers most of the "is this even possible" questions and records the traps. It contains a full writeup, a captured sample of both datasets, an internal reference summary of the Openness Index spec, and the extraction snippets that actually worked.

Two datasets, on two different join keys:

(a) Provider leaderboard — /leaderboards/providers, keyed by (model × serving provider). 512 rows at capture. The default view is collapsed; the expanded view exposes 51 columns: 23 eval/benchmark scores, pricing (input/output/cache-hit/cache-write), throughput (median + P5/P25/P75/P95), latency (median first chunk, first answer token, + percentiles), end-to-end response time, context window, capability flags, and the provider's own API ID string.

(b) Openness Index — /evaluations/artificial-analysis-openness-index, keyed by model only. 298 models at capture, covering open and proprietary models. Scored 0–18 across six subcomponents, published normalized to 0–100.

What the spike established

• There is no backing JSON API on either page. 107 network requests on a clean load, zero XHR/fetch to any data endpoint. Both pages are server-rendered Next.js; the table DOM is the interface.
• Volume is a non-issue. All rows are in the DOM on first paint. No pagination, no lazy-load, no virtualisation, no per-model detail fetches. Capturing everything is two page loads. The instinct that a 512-row leaderboard needs a carefully rate-limited crawl does not apply here — there is nothing to iterate, so there is nothing to throttle.
• The expanded view is React state, with no URL parameter, and element.click() does not trigger it — it needs a synthetic pointer-event sequence. Expanding fires no network request; the data is already client-side. A plain curl of the HTML gets you the collapsed 11 columns only, so this needs a real browser.
• The Openness Index is not on the provider leaderboard. Confirmed. The expanded view has a column called "Omniscience Index" — a different metric entirely, which goes negative. The names are one letter apart and this has already caused confusion once. They are separate datasets on separate keys and must stay that way.
• AA does publish an official API, but not usefully at the free tier — see Commercial position.

The identity problem — read this before designing anything

This is the hard part of the issue, and it is not a generic "names are messy" caveat. There is a specific structural mismatch, evidenced in the spike sample.

On the provider leaderboard, reasoning variants are distinguished by slug but can share an identical display name:

Display name	Model slug
Qwen3.5 397B A17B	qwen3-5-397b-a17b
Qwen3.5 397B A17B	qwen3-5-397b-a17b-non-reasoning

On the Openness Index, the same two models are distinguished by display name, and there are no slugs at all — zero anchors anywhere in that table body:

Display name	Slug
Qwen3.5 397B A17B (Reasoning)	none
Qwen3.5 397B A17B (Non-reasoning)	none

Each dataset carries the disambiguator in the field the other one lacks. Joining on display name silently collapses the two Qwen variants on the (a) side; joining on slug is impossible on the (b) side.

It compounds: which effort level owns the bare slug varies per model. gpt-oss-120b (high) → gpt-oss-120b but gpt-oss-120b (low) → gpt-oss-120b-low; Claude Opus 5 (max) → claude-opus-5 but Claude Opus 5 (xhigh) → claude-opus-5-xhigh. For gpt-oss the bare slug is high; for Opus 5 it is max. You cannot derive the slug from name + effort by rule. Across the leaderboard: 221 distinct display names, 183 distinct model slugs.

Coverage is also unaligned — at capture time Claude Opus 5 is on the provider leaderboard but only Claude Opus 4.5 is in the Openness Index. The join must tolerate genuine misses on the openness side.

This lands directly on the composite identity in #74. AA's own data model is internally inconsistent about where reasoning effort lives, so the crosswalk has to be explicit, tested data — not a parsing convention.

Scope

A skill that acquires both AA datasets and lands them as validated records keyed to the identity from #74.

• Skill + process doc. skills/<name>/SKILL.md following the conventions of skills/tokenpricing/SKILL.md, registered in .claude-plugin/marketplace.json, plus a process document covering both pages: how to reach the expanded view, what to extract, how to detect that the page shape changed, when to abort rather than guess. Same pattern as Skill: arena.ai category score scraping for model + reasoning-effort variants #75 and Skill: public benchmark catalog with tag taxonomy and direct model score links #76 — if those have landed, follow the pattern they established rather than inventing a second one.
• Deterministic ingestion. Capture raw (table.outerHTML for both pages) as the immutable artefact, then parse from that with tested code, then validate against a schema. The harness never writes canonical records directly. Saving the raw table matters more than usual here: when AA restructures the page we want the failing input on disk, and it lets us derive fields we did not originally extract without re-scraping.
• Provider-leaderboard records keyed by (model, reasoning_effort, serving_platform) per Data model: unified model x reasoning effort x serving platform pricing layer #74. Every latency and throughput value must be labelled with origin third-party measured (AA) — distinct from vendor-published and from measured-by-us — and carry AA's measurement conditions. Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM) #78 already establishes this requirement; AA slots in as a third origin value.
• Openness records keyed by model per Data model: unified model x reasoning effort x serving platform pricing layer #74's variant identity. Store the raw 0–18 subcomponents and derive the 0–100 figure on read, not the reverse. Version records against "Openness Spec V1.0" — the spec PDF is stamped PRELIMINARY DRAFT and its definitions may move.
• The crosswalk is data, not logic. A checked-in aa_model_slug → aa_openness_row mapping with an explicit unresolved list, a coverage figure, and review when it changes. Fail loudly on ambiguity rather than picking a winner. Nothing silently dropped or force-matched — same discipline Skill: arena.ai category score scraping for model + reasoning-effort variants #75 and Skill: public benchmark catalog with tag taxonomy and direct model score links #76 require.
• Filter-state capture. The leaderboard's default filter state includes Status: Current, which excludes deprecated/legacy endpoints. Decide deliberately whether to flip it, and record the filter state as metadata on every capture. This is directly relevant to Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM) #78's deprecation work and is a silent data loss if nobody notices.
• Shape assertions that fail the job. Guard on expanded header count (51), the specific expected column names, row counts within a sane band (>400 providers, >250 openness), and non-zero anchors on the provider table. If the expand did not take, abort without writing. The realistic failure mode is a silent partial capture writing a collapsed 11-column table into database/, not an HTTP error.
• Storage. In database/ following services/sync/src/tokenpricing_sync/paths.py conventions. Current state plus history. Idempotent — unchanged upstream yields a no-op diff, tested, because the sync pipeline commits to main and a noisy diff pollutes the changelog.
• Schedulability. Clear entrypoint, machine-readable success/failure, no interactive prompts. Do not add a scheduled workflow.
• Attribution. Any surface built on this carries "Data source: Artificial Analysis (artificialanalysis.ai)" and links the spec PDF. Federico has confirmed crediting AA publicly is fine — badge, references, links. Attribution is not a blocker; it is a requirement to include.

Parsing gotchas — all observed in the spike, all will bite

• Negative numbers use U+2212 MINUS SIGN (−), not ASCII hyphen. parseFloat("−31") → NaN, and the Omniscience Index goes negative routinely.
• Trailing asterisk means estimated/partial (33*). Preserve the flag, do not strip it silently — Estimate missing benchmark scores from arena category distributions #77 cares about this distinction.
• -- / — means "no data" and is not zero. Absent, unknown and zero must stay structurally distinct, per Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM) #78.
• Thousands separators inside values (1,715); $ on prices; % on scores.
• Context window is human-rendered (1M, 1.05M, 262k, 205k) and differs per provider for the same model (Kimi K3: 1.05M on most, 1M on Together AI, 205k on Databricks) — it is a per-offering attribute, not per-model.
• The leaderboard header row is two-deep (group row + column row).
• Some API ID values are full URLs rather than model strings.

Out of scope

• The remote-database / SQLite migration. Migration: JSON-driven database → derived SQLite read layer for faster UI & SDK queries #60 and SQLite read layer for the canonical pricing database — decision record + Phases 0 & 1 #61 cover a derived SQLite read layer. Do not touch or build on them. Storage stays git-committed JSON.
• Workload-specific leaderboard / arena-per-agent generation. Deferred to a later phase.
• CI cron jobs. Do not add a scheduled workflow; make it schedulable.
• Deprecation dates and rate limits (TPM/RPM). AA publishes neither. Those stay with Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM) #78, which retains them as its core scope.
• Arena.ai scraping (Skill: arena.ai category score scraping for model + reasoning-effort variants #75) and the general benchmark catalog + taxonomy (Skill: public benchmark catalog with tag taxonomy and direct model score links #76). AA's 23 eval columns are ingested here as scores against the AA source; if Skill: public benchmark catalog with tag taxonomy and direct model score links #76 has landed, register them against its benchmark registry rather than defining a competing one. Do not build a second taxonomy.
• Estimating missing values (Estimate missing benchmark scores from arena category distributions #77). Record what AA publishes, including its own * estimate markers; do not fill gaps.
• Running benchmarks or measuring latency ourselves.

Commercial position — decide before building

AA publishes an official API (artificialanalysis.ai/api/v2, x-api-key). Both datasets we want are behind paid tiers:

Tier	Limit	Relevant access
Free	100 req/24h	headline indices, median performance, input/output pricing. Model-keyed only — no provider dimension, no openness components.
Pro	500 req/24h	full model detail including the Openness Index breakdown — i.e. (b)
Commercial	custom	provider-level data, percentiles, performance over time, raw measurements — i.e. (a)

So "the free API is insufficient" is correct and precise: (a) is Commercial tier, (b) is Pro tier.

Which means the honest framing is that scraping the public pages gets us data AA sells access to. That is a commercial/ToS question, not a technical one, and it is not the assignee's to resolve. Before building the scraper, someone should price the Pro and Commercial tiers (hello@artificialanalysis.ai). The Pro tier in particular may be cheap enough to make (b) a non-issue, and an API contract is far more stable than a DOM.

If a licence is obtained, the acquisition half of this issue collapses into an API client and most of the scraping scope below falls away — but the schema, identity crosswalk, provenance labelling and storage work all stand unchanged. Design so the acquisition layer is swappable behind the parser.

Suggested approach

1. Read experiments/aa-scrape-spike/ first. The writeup, the methodology summary, and extract.js will save you a day, particularly the pointer-event workaround and the parsing gotchas.
2. Resolve the commercial question before writing acquisition code. If it stalls, build behind an interface so the scraper can be swapped for an API client without touching the parser.
3. Design the crosswalk before the scraper. It is the part that will actually be hard, and it determines whether the two datasets are usable together at all. Hand-join twenty models across both datasets on paper and see where it breaks.
4. TDD the parsers against trimmed real fixtures checked into the test tree, as services/sync/tests/ does. Use the spike's captured samples as the seed. No network in tests.
5. Playwright, not HTTP. Two page.goto(), one synthetic-pointer press, two page.evaluate() extractions, ~30s. Realistic UA, respect robots.txt, one visit per page per run, no concurrency.
6. Write the process doc last and validate it cold — hand it to a fresh harness session and see whether their output passes your validator.
7. Treat capture timestamp and filter state as first-class fields. A latency figure without a capture date is not usable for ranking in Toolkit query layer: rank best model for a task across quality, cost, SLO and budget #79.

Acceptance criteria

• skills/<name>/SKILL.md exists, follows existing skill conventions, is registered in .claude-plugin/marketplace.json, and ships with a process document covering both pages including expanded-view access and page-shape-change handling.
• Raw captures are stored as the immutable artefact; parsing happens in tested checked-in code against those artefacts, validated to a schema. No free-form harness writes.
• Provider-leaderboard records keyed by (model, reasoning_effort, serving_platform) per Data model: unified model x reasoning effort x serving platform pricing layer #74, carrying pricing, throughput (median + percentiles), latency (median + percentiles), end-to-end times, context window, capability flags and API ID.
• Every latency/throughput value is labelled origin third-party measured (AA) with measurement conditions, consistent with Skill: serving platform / CSP capability scraping (deprecation, TPM, RPM) #78's provenance rules. A test asserts unlabelled values cannot be stored.
• Openness records store raw 0–18 subcomponents, derive the 0–100 value, and are version-tagged to the spec version. A test reconciles at least four models against published totals.
• The published methodology limitation is documented: Model Availability is an aggregate of two subcomponents that cannot be separated, and the two Methodology subcomponents are only recoverable as a residual. (Derivation and verification are in the spike's openness-index-methodology.md.)
• Model-name crosswalk between (a) and (b) is checked-in data with a coverage figure and an explicit unresolved list. Ambiguous matches fail loudly; a regression test covers the Qwen3.5 397B A17B reasoning/non-reasoning collision and the varying-bare-slug case.
• Capture records filter state (including whether Status: Current was active) and a capture timestamp.
• Shape assertions abort the run without writing when the expanded view fails to open or row counts fall outside expected bands. Tested.
• Parsers handle U+2212 minus, * estimate markers, -- as absent-not-zero, thousands separators, currency/percent symbols, and human-rendered context windows. Each has a test.
• Data lands in database/ per existing path conventions with current state and history; unchanged upstream yields a no-op diff (tested).
• Tests pass offline with no network access. uv run pytest -q and uv run pre-commit run -a green.
• Docs updated: what the data means, that it is AA-measured rather than vendor-published or measured by us, the openness spec's draft status, the join's known coverage gaps, and AA attribution.

Working agreement

Follow the self-contained-task workflow in libraries/python/AGENTS.md:29-37: linked branch via gh issue develop <issue> --checkout, draft PR early with plan and acceptance criteria, then plan → review → TDD (failing first) → implement minimally → verify → report.

uv for Python. CI is path-filtered per package; pushes to main touching libraries/** auto-release, so keep the blast radius intentional.

Be candid about what will not survive. The DOM contract here is small — one <table>, one button label, two href prefixes — but AA restyling the leaderboard will break it, and the shape assertions are what turn that from silent corruption into a loud failure. A capture that fails loudly beats one that quietly writes eleven columns.