CompassAcademic: 4 of 7 objective boards have an unseparated #1, two are exact ties broken by display order

[ipezygj]

On the CompassAcademic leaderboard, 4 of 7 objective boards have a #1 that isn't separated from #2 — two of them are exact ties broken by display order.

Thanks for publishing the underlying numbers openly; this whole check was possible only because hf-academic.json is fetchable and complete. I'm not reporting a bug in OpenCompass's evaluation — the scores reproduce fine. It's about what the leaderboard displays: a strict ordering over models whose differences the benchmark sizes can't resolve.

Related but not the same as #2392 / #2391, which are about judge calibration for subjective evaluation. Worth noting that CompassArena already ships bootstrap confidence intervals for its Bradley-Terry rankings (compass_arena_bradley_terry.py) — the principle is already accepted in one surface; the academic leaderboard doesn't carry it.

Method

Denominators were recovered, not assumed. For each board I searched for every n in 1..15000 consistent with all published scores under round(100k/n, 2) (a GRIM-style consistency check). The minimum consistent n came out at exactly the published dataset sizes without being supplied:

board	metric	recovered n
IFEval	Prompt-level-strict-accuracy	541
GPQA_diamond	accuracy	198
math_prm800k_500	accuracy	500
aime2024	accuracy	30
lcb_code_generation	pass@1	400
openai_humaneval	pass@1	164
bigcodebench_hard_instruct	pass@1	148

Then SE = sqrt(p(1-p)/n), and 20,000 simulated draws per board asking how often noise alone would put a different model on top.

Result

board	n	models	#1	#2	gap in leader's SE	P(measured #1 isn't the best)
lcb_code_generation	400	58	90.00%	83.75%	4.17σ	0.5%
IFEval	541	58	91.87%	88.54%	2.83σ	7.9%
math_prm800k_500	500	57	97.60%	96.20%	2.05σ	11.3%
bigcodebench_hard_instruct	148	17	29.73%	26.35%	0.90σ	46.4%
aime2024	30	58	86.67%	83.33%	0.54σ	37.8%
GPQA_diamond	198	58	69.70%	69.70%	0.00σ	76.4%
openai_humaneval	164	58	98.17%	98.17%	0.00σ	63.3%

• GPQA_diamond: deepseek-chat-r1 and gpt-4.5-preview-2025-02-27 are both 69.70%.
• openai_humaneval: deepseek-chat-r1 and QwQ-32B are both 98.17%.

These are exact ties, not rounding coincidences. One item is worth 0.505 pp on GPQA_diamond and 0.610 pp on HumanEval, so two models displaying the same 2-decimal value must have the same raw count. The smallest per-item step across all seven boards is 0.185 pp (IFEval), far above 2-dp resolution — a displayed tie is a real tie everywhere here.

Across all seven boards, of 357 adjacent rank pairs only 5 (1.4%) clear 2.5σ and 23 (6.4%) clear 1.35σ — so 93.6% of neighbouring ranks are not separated at even a loose bar.

Reproduce

import json, math, urllib.request
U = "http://opencompass.oss-cn-shanghai.aliyuncs.com/dev-assets/hf-research/hf-academic.json"
d = json.loads(urllib.request.urlopen(U).read())
meta = {"dataset", "version", "metric", "mode"}
N = {"GPQA_diamond": 198, "openai_humaneval": 164}      # both GRIM-recovered

for row in d.values():
    nm = row["dataset"]
    if nm not in N:
        continue
    sc = sorted(((m, float(v)) for m, v in row.items()
                 if m not in meta and v not in ("-", "", "N/A", None)),
                key=lambda x: -x[1])
    print(f"{nm}: #1 {sc[0][0]} {sc[0][1]}  #2 {sc[1][0]} {sc[1][1]}  "
          f"tie={sc[0][1] == sc[1][1]}  one item = {100/N[nm]:.3f} pp")

Limits, stated

• The published score is treated as ground truth and noise as independent across models. That's the standard winner's-curse framing; it is a model, not a fact.
• Eight boards excluded and why: bbh, mmlu_pro, musr_average, cmmlu, mmlu, korbench_single are naive_average over subtasks, so there is no single denominator and sqrt(p(1-p)/n) doesn't apply. drop and hellaswag had no clean minimum consistent n (6978 and 6387), so I left them unpinned rather than guess.
• aime2024 at n=30 can barely resolve anything — one problem is 3.33 pp. That board's ordering is close to unusable regardless of what's displayed.

Suggestion

Significance tiers, or a CI column, on CompassAcademic — the same treatment CompassArena already gives Bradley-Terry. No score would change; the display would stop asserting an order the data can't support. Happy to open a PR if that's useful, and equally happy to be told I've misread the metric definitions — in which case I'll close this myself and say so.

[lawanmohammed860-star]

سلاو