Evaluation Time Issue

[Lorsense]

Hello authors,

Thank you very much for open-sourcing this great work.

I am currently reproducing your work using 8 × NVIDIA A800 GPUs, and I have encountered some issues regarding the evaluation that is performed every 20 training steps.

During evaluation, I found that the evaluation process takes an extremely long time (more than 36 hours), and it still does not produce the final evaluation results. I also tried evaluating intermediate checkpoints using run_eval_aime.sh, but encountered the same issue.

It is worth noting that during evaluation, I frequently see errors such as:

API Request Error: Gateway Timeout
Sandbox API call failed. Last error: ...

Similar errors also occur during training. However, during training, each step takes approximately 30 minutes on average, whereas the evaluation process seems to run indefinitely without completing.

I would like to ask what might be causing this issue. Is such a long evaluation time expected, or could it be related to the sandbox service, API gateway timeouts, or some configuration issue?

I would greatly appreciate any advice or suggestions on how to diagnose and resolve this problem.

Thank you very much for your time and help!

[YoungZ365]

Thank you for your detailed report!

Based on the symptoms you described, this is very likely a sandbox configuration issue rather than an evaluation issue itself.

The repeated errors such as:

API Request Error: Gateway Timeout
Sandbox API call failed

indicate that the code execution requests are timing out. Since both training and evaluation in SOD rely heavily on tool execution, if the sandbox cannot respond reliably, evaluation may appear to run indefinitely because many requests are repeatedly timing out or retrying.

The most common causes are:

The configured sandbox endpoint is invalid or no longer available.
The sandbox service does not have sufficient concurrency for the rollout/evaluation workload.
The sandbox instances are under-provisioned or overloaded, leading to frequent request timeouts.

Please first verify that your sandbox service is functioning correctly and can consistently execute Python code under concurrent requests.

We recommend checking:

Whether the sandbox URL is valid and reachable.
Whether the sandbox has enough concurrent instances/resources.
Whether the logs contain a large number of Sandbox API call failed or Gateway Timeout messages.

For the recommended sandbox configuration, please refer to the README and Issue #2, where we discuss the recommended deployment and concurrency settings in more detail.

Under a properly configured sandbox environment, evaluation should finish normally and should not take dozens of hours.

If the issue persists after verifying the sandbox configuration, please let us know which sandbox service/provider you are using, along with your sandbox configuration and logs, and we'd be happy to help investigate further.