mputed zero rows crash some metric hooks and are silently scored by others

[jubick1337]

Describe the bug

Both count_failure_classes_as_zero and count_missing_rollouts_as_zero put an identity-only row into metric aggregation: task index, rollout index, agent ref, and reward: 0.0. Benchmark compute_metrics hooks read the verifier's output, which such a row does not carry.

Three hooks raise, so the run dies after the rollouts are already written:

hook	raises
responses_api_agents/tau2/app.py:456	KeyError('config')
resources_servers/vlm_eval_kit/app.py:150	KeyError('benchmark_name')
responses_api_agents/scicode_agent/app.py:144	ValueError on mixed token-accounting versions

Some hooks accept the row and score it as a measurement of something that was never measured:

• resources_servers/simpleqa/app.py:182 defaults the verdict to not_attempted, so the run
  reports that the model abstained.
• resources_servers/omniscience/app.py:207 defaults to incorrect, which feeds
  judge_omni_hallucination: a rollout that never ran is counted as a hallucination.
• resources_servers/ragtruth/app.py:245 scores it a true negative, inflating precision,
  recall and F1.
• resources_servers/asr_with_pc/app.py:399 averages per, a phone error rate, defaulting to
  0.0. Zero is a perfect score there, so a rollout that never ran improves the result.
• resources_servers/math_with_judge/app.py:413 returns an empty score dict, so
  compute_pass_majority_metrics skips the row: mean/reward moves while
  pass@1/symbolic_accuracy does not.

There is no single set of fields that satisfies all of them, because they need verifier verdicts rather than identity. Supplying those would invent outcomes that no verifier produced.

Steps/Code to reproduce bug

Reachable on main today, without count_missing_rollouts_as_zero:

1. Take any benchmark whose compute_metrics reads verifier fields, for example tau2 or vlm_eval_kit.
2. Set count_failure_classes_as_zero: ['agent_run_error']. This is the value the option's own description suggests.
3. Run it so that at least one rollout fails in that class.

tau2 raises KeyError('config'), vlm_eval_kit raises KeyError('benchmark_name'). The failure surfaces from /aggregate_metrics as HTTP 500 and reaches raise_for_status in nemo_gym/rollout_collection.py, after the rollouts jsonl has been closed.

For the silent case, run math_with_judge over two tasks where one succeeds and one fails in a counted class: mean/reward becomes 0.5 while pass@1/symbolic_accuracy still reports 100.

Expected behavior

An imputed zero either enters the benchmark's metrics correctly, or the run refuses to start with a message naming the benchmark and the option, rather than crashing after the rollouts are written or silently changing what a metric means.

One safeguard would cover both options. A benchmark declaring whether it tolerates an identity-only row, with an enforced check, would do it.

Configs

Any config that sets either option on an affected benchmark. Minimal fragment:

count_failure_classes_as_zero: ['agent_run_error']

Environment details

• OS: Ubuntu 24.04.4 LTS
• Python: 3.13.14
• nemo-gym: 0.7.0rc0
• pydantic 2.13.4, numpy 2.3.2

Additional context
The row count_failure_classes_as_zero builds is thinner than the one count_missing_rollouts_as_zero builds: _agent_request_failure_row carries only the stamped task and rollout indices, the agent ref and reward, with no task_name. So this is present on main independently of the newer option, which widens the surface rather than opening it.