lcb_codegeneration` accuracy always 0.000 — grader returns `pass@1=0.000` for demonstrably correct code

[Maze999]

aiperf version

0.11.0

Environment

• aiperf running in a Kubernetes pod (Python venv), vLLM in a separate pod
• Python 3.12, Linux
• datasets==3.6.0 (pinned <4; see secondary issue at the bottom)
• lighteval==0.13.0

Models tested

• Qwen/Qwen3.5-27B-fp8 — thinking enabled (run12)
• Qwen/Qwen3.5-27B-fp8 — thinking disabled (enable_thinking: false) (run13, run15)

All configurations score 0.000 on every run, every problem.

---

Commands used

Non-thinking, concurrency 25 (run13) — 25 problems:

aiperf profile \
  --url 'http://<SERVER>' \
  --model 'Qwen/Qwen3.5-27B-fp8' \
  --tokenizer 'Qwen/Qwen3.5-27B-fp8' \
  --tokenizer-trust-remote-code \
  --endpoint-type 'chat' \
  --accuracy-benchmark 'lcb_codegeneration' \
  --artifact-dir 'lcb-codegen-qwen-fp8-run13-think-disable' \
  --random-seed 547 \
  --num-requests 25 \
  --concurrency 25 \
  --export-level 'raw' \
  --accuracy-verbose \
  --use-server-token-count \
  --extra-inputs '{"temperature": 0, "chat_template_kwargs": {"enable_thinking": false}}'

Non-thinking, concurrency 1 (run15) — 25 problems:
Same command with --concurrency 1.

Thinking enabled, concurrency 25 (run12) — 25 problems:
Same command without the enable_thinking override.

Full dataset run — 268 problems:
Same command with --num-requests 268.

---

Observed results

Run	Thinking	Concurrency	Requests	correct	total	unparsed	accuracy
run12	enabled	25	25	0	25	3	0.0000
run13	disabled	25	25	0	25	2	0.0000
run15	disabled	1	25	0	25	2	0.0000
full	disabled	25	268	0	268	61	0.0000

--concurrency 1 produces the same 0.000 result. Concurrency is not a factor.

---

Key evidence

1 — Grader completes normally but returns 0.000 (not an exception/parse failure)

From aiperf.log in run13 (thinking disabled, --concurrency 25):

[accuracy] session=9 correct=False unparsed=False
  reason='lighteval codegen_metrics pass@1=0.000 (snippet length=291)'
  extracted='def main():\n    A, B, C = map(int, input().split())\n    if B < C:\n        if A < B or A >= C:\n            print("Yes")\n ...'
  response_len=8252

• unparsed=False — code was successfully extracted (291 chars)
• correct=False — the grader ran and returned 0.000; this is not an error/exception path
• The extracted snippet is a correct Python solution for AtCoder problem "Shout Everyday" (easy, 14 test cases)

The same pattern holds across all 25 sessions (see attached logs):

• 21–23 sessions: unparsed=False, pass@1=0.000 — code extracted, grader returns zero
• 2–3 sessions: unparsed=True — no code block emitted (token budget exhausted for harder problems)
• 0 sessions: correct=True

2 — Standalone test proves the code and grader are both correct

Taking the exact 291-char snippet from profile_export_raw.jsonl (session 9) and running it through the identical aiperf grading path outside of aiperf:

from aiperf.accuracy.graders.code_execution import (
    _payload_to_test_cases, _build_evaluation_sample, _run_codegen_metrics
)
from lighteval.tasks.tasks.lcb.codegen_metrics import extract_code
import orjson

payload = orjson.loads(ground_truth)   # dataset ground_truth, built via _build_ground_truth()
inputs, outputs, fn_name = _payload_to_test_cases(payload)   # 14 test cases, fn_name=None
ev_sample = _build_evaluation_sample(inputs, outputs, fn_name)
snippet = extract_code(response_text)  # → same 291-char snippet

metrics, _ = _run_codegen_metrics(ev_sample, [[snippet]])
# → {'pass@1': np.float64(1.0)}  — ALL 14 test cases PASS

The same grader function, same code snippet, same test cases returns pass@1=1.0 when called directly, and pass@1=0.000 when called from within aiperf's record-processor pipeline.

Unknown: what causes codegen_metrics to return 0.000 inside aiperf

The discrepancy between the standalone result (pass@1=1.0) and the in-aiperf result (pass@1=0.000) is consistent across all concurrency levels (1 and 25), all models, and both thinking modes. The failure is isolated to the aiperf execution environment — something about how aiperf's AccuracyRecordProcessor invokes _run_codegen_metrics causes check_correctness() to report every test case as failed.

Secondary bug: accuracy.overall metric tag not found (fires on every graded result)

Every time a graded result is published to the realtime metrics bus, the dashboard raises:

ERROR - AIPerfDashboardUI_... - Error running Hook ... for AIPerfDashboardUI:
  Metric class with tag 'accuracy.overall' not found
ERROR - aiperf.common.utils - Error calling function _on_realtime_metrics:
  AIPerfMultiError("AIPerfDashboardUI.on_realtime_metrics:
    Metric class with tag 'accuracy.overall' not found")
Traceback (most recent call last):
  ...
  aiperf.common.exceptions.AIPerfMultiError:
    Errors calling functions: AIPerfDashboardUI.on_realtime_metrics:
      Metric class with tag 'accuracy.overall' not found

This fires once per graded result (25× in a 25-request run) and once per HTTP completion too. The accuracy.overall metric tag appears not to be registered before the accuracy record processor begins emitting updates. Does not block grading but floods the log with ERROR-level noise and likely breaks the live accuracy panel in the TUI.

---

Expected behaviour

lcb_codegeneration reports non-zero accuracy for correct model outputs.
Standalone testing with the exact same grader code confirms both the model output and the grader itself are correct.

Actual behaviour

accuracy = 0.0000 for 100% of problems across all runs, all models, all seeds, both thinking modes, and both --concurrency 1 and --concurrency 25.

[Maze999]

sanitized-run12-accuracy-lines.log
sanitized-run13-accuracy-lines.log