TL;DR
In evaluation/benchmarks.py:174-177, when the model fails to produce a parseable answer letter, the benchmark code awards a fractional credit 1 / len(options) to the correct count. This means a model that always emits unparseable output still scores ~10% on MMLU-Pro (with 10 choices per question), which is misleading.
Details
# evaluation/benchmarks.py:174-177
stats[subset]["n"] += 1
if ans is None:
stats[subset]["invalid"] += 1
stats[subset]["correct"] += 1 / len(options)
elif ans == gt:
stats[subset]["correct"] += 1The same pattern shows up in StandardMCQBenchmark.compute_metrics (evaluation/benchmarks.py:221-222) and MMLU.compute_metrics (evaluation/benchmarks.py:266-267).
If the intent is to report a "random-guess-adjusted" accuracy, that's worth keeping but it should be reported under a separate metric name (e.g. acc_with_guess) so it doesn't get confused with the raw accuracy that the published benchmark numbers use.
Impact
Reported MMLU/ARC/HellaSwag numbers may be inflated relative to comparable papers. This makes the README table at lines 29-32 not strictly comparable to the upstream benchmark leaderboards.
References
evaluation/benchmarks.py:174-181 (MMLU-Pro)evaluation/benchmarks.py:215-230 (StandardMCQBenchmark)evaluation/benchmarks.py:257-271 (MMLU)
TL;DR
In
evaluation/benchmarks.py:174-177, when the model fails to produce a parseable answer letter, the benchmark code awards a fractional credit1 / len(options)to the correct count. This means a model that always emits unparseable output still scores ~10% on MMLU-Pro (with 10 choices per question), which is misleading.Details
The same pattern shows up in
StandardMCQBenchmark.compute_metrics(evaluation/benchmarks.py:221-222) andMMLU.compute_metrics(evaluation/benchmarks.py:266-267).If the intent is to report a "random-guess-adjusted" accuracy, that's worth keeping but it should be reported under a separate metric name (e.g.
acc_with_guess) so it doesn't get confused with the raw accuracy that the published benchmark numbers use.Impact
Reported MMLU/ARC/HellaSwag numbers may be inflated relative to comparable papers. This makes the README table at lines 29-32 not strictly comparable to the upstream benchmark leaderboards.
References
evaluation/benchmarks.py:174-181(MMLU-Pro)evaluation/benchmarks.py:215-230(StandardMCQBenchmark)evaluation/benchmarks.py:257-271(MMLU)