Training qwen2_5_vl_7b_geo3k_grpo.sh on Ubuntu 22.04 + CUDA 12.6 + 4×A100 80 GB throws an error：
(pid=22967) Running step 0: 7%|? | 4.00/60.0 [2:11:12<23:53:54, 1.54kTraceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,???| 4.00/4.00 [16:03<00:00, 174s/it]t]
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code28<00:00, 6.99s/it]t]
exec(code, run_globals)
File "/workspace/EasyR1/verl/trainer/main.py", line 133, in <module>
main()
File "/workspace/EasyR1/verl/trainer/main.py", line 125, in main
ray.get(runner.run.remote(ppo_config))
File "/usr/local/lib/python3.10/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 2849, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 937, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(OutOfMemoryError): ray::Runner.run() (pid=22967, ip=172.17.0.2, actor_id=4a188c63394530d9b186dc3101000000, repr=<main.Runner object at 0x7ee37d7e27a0>)
File "/workspace/EasyR1/verl/trainer/main.py", line 94, in run
trainer.fit()
File "/workspace/EasyR1/verl/trainer/ray_trainer.py", line 676, in fit
self._save_checkpoint()
File "/workspace/EasyR1/verl/trainer/ray_trainer.py", line 322, in _save_checkpoint
self.actor_rollout_ref_wg.save_checkpoint(actor_path, save_model_only=self.config.trainer.save_model_only)
File "/workspace/EasyR1/verl/single_controller/ray/base.py", line 47, in func
output = ray.get(output)
ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory.
Memory on the node (IP: 172.17.0.2, ID: 29686a73893e28b17c8098100c402862f85c87c0fef5275c5289a7e7) where the task (actor ID: 8a69b7749d17a47727a6aa3701000000, name=hyfD5YWorkerDict_0:1:WorkerDict.__init__, pid=24819, memory used=68.73GB) was running was 304.56GB / 314.66GB (0.967883), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: 55f43085e3d707e2a1b89e642a71c688071c35ce527f297ece9fdb54) because it was the most recently scheduled task; to see more information about memory usage on this node, use `ray logs raylet.out -ip 172.17.0.2`. To see the logs of the worker, use `ray logs worker-55f43085e3d707e2a1b89e642a71c688071c35ce527f297ece9fdb54*out -ip 172.17.0.2. Top 10 memory users:
PID MEM(GB) COMMAND
24820 69.81 ray::WorkerDict.actor_rollout_ref_save_checkpoint
24821 69.75 ray::WorkerDict.actor_rollout_ref_save_checkpoint
24582 69.25 ray::WorkerDict.actor_rollout_ref_save_checkpoint
24819 68.73 ray::WorkerDict.actor_rollout_ref_save_checkpoint
22967 2.11 ray::Runner.run
23191 1.48 ray::BatchFunctionRewardManager
27628 1.42 ray::Runner.run
27627 1.39 ray::Runner.run
27629 1.35 ray::Runner.run
27630 1.34 ray::Runner.run
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. Set max_restarts and max_task_retries to enable retry when the task crashes due to OOM. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
However, all attempts return the same error mentioned above.If there’s anything else I can provide to help debug, just let me know; your assistance would be greatly appreciated.
Training qwen2_5_vl_7b_geo3k_grpo.sh on Ubuntu 22.04 + CUDA 12.6 + 4×A100 80 GB throws an error：
ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory. Memory on the node (IP: 172.17.0.2, ID: 29686a73893e28b17c8098100c402862f85c87c0fef5275c5289a7e7) where the task (actor ID: 8a69b7749d17a47727a6aa3701000000, name=hyfD5YWorkerDict_0:1:WorkerDict.__init__, pid=24819, memory used=68.73GB) was running was 304.56GB / 314.66GB (0.967883), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: 55f43085e3d707e2a1b89e642a71c688071c35ce527f297ece9fdb54)Error details：
Since I only have 4 A100（80G），I simply changed the
n_gpus_per_nodefrom 8 to 4 both inconfig.yamlandqwen2_5_vl_7b_geo3k_grpo.sh.Docker : [ngc-th2.7.1-cu12.6-vllm0.10.0] && [ngc-th2.7.0-cu12.6-vllm0.9.1]
Mem : 128G && 360G
However, all attempts return the same error mentioned above.If there’s anything else I can provide to help debug, just let me know; your assistance would be greatly appreciated.