Summary
Follow-up report on the opencompass grading surface (extends #2535, whose fix is still unlanded at the current pin). Four additional defect classes verified at pin ec59180, all with deterministic reproductions.
1. Model code can rewrite the ground truth in three benchmarks
APPS, TACO, and LiveCodeBench all use run_test to evaluate model-authored code. The reward path imports the model's modules to grade them, so the evaluated code can rewrite the expected outputs before comparison. A wrong answer scores correctly after the key is rewritten in-process.
2. CRUXEval splices the prediction into an assertion
The evaluator embeds the prediction inside assert result == expected and evaluates it, so a model that emits an expression re-evaluating expected as a side effect can influence both sides of the comparison.
3. JudgerBench silently drops unparseable verdicts from the denominator
The accuracy denominator counts only verdicts that parsed; a judge that fails on a subset of rows silently inflates the reported accuracy over the survivors.
4. LCEvaluator runs eval() on the raw prediction
The main evaluator process executes the raw prediction string. Combined with the sentinel-rebind bypass (a type-aware __eq__ that the model can redefine from inside its submission, circumventing the #2535 fix), the grading process is fully under evaluated-side control.
All four reproduce with deterministic, offline harnesses. Happy to share them as PRs with tests.
Summary
Follow-up report on the opencompass grading surface (extends #2535, whose fix is still unlanded at the current pin). Four additional defect classes verified at pin ec59180, all with deterministic reproductions.
1. Model code can rewrite the ground truth in three benchmarks
APPS, TACO, and LiveCodeBench all use
run_testto evaluate model-authored code. The reward path imports the model's modules to grade them, so the evaluated code can rewrite the expected outputs before comparison. A wrong answer scores correctly after the key is rewritten in-process.2. CRUXEval splices the prediction into an assertion
The evaluator embeds the prediction inside
assert result == expectedand evaluates it, so a model that emits an expression re-evaluatingexpectedas a side effect can influence both sides of the comparison.3. JudgerBench silently drops unparseable verdicts from the denominator
The accuracy denominator counts only verdicts that parsed; a judge that fails on a subset of rows silently inflates the reported accuracy over the survivors.
4. LCEvaluator runs eval() on the raw prediction
The main evaluator process executes the raw prediction string. Combined with the sentinel-rebind bypass (a type-aware
__eq__that the model can redefine from inside its submission, circumventing the #2535 fix), the grading process is fully under evaluated-side control.All four reproduce with deterministic, offline harnesses. Happy to share them as PRs with tests.