The gRPC API supports canceling a running job by terminating the worker process that owns it. That cancel usually succeeds from the client’s point of view (CANCELLED / NO
T_FOUND), and a replacement worker is respawned automatically.
However, killing a worker mid-CUDA can intermittently leave the GPU in a sticky error state (cudaErrorIllegalAddress, RMM / memory allocation failures). Respawning only t
hat worker often does not clear it; later jobs on the same device may keep failing.
Workaround: If jobs fail with CUDA / allocation errors after a cancel (or delete of a running job), shut down and restart the entire cuopt_grpc_server process. A clean re
start clears the bad device state in our testing. Send SIGINT to the server (Ctrl-C from the console), or otherwise stop and start the process/pod.
Queued-job cancel (job not yet claimed by a worker) is not affected by this issue.
Notes:
Preferring SIGTERM before SIGKILL, and delaying worker respawn, did not reliably prevent the poison in stress testing.
A durable fix likely needs either cooperative cancel inside the solver (between CUDA work) and/or a CUDA health check that treats this as fatal and exits so an orchestrat
or can recycle the server (similar to the Python cuOpt server).
The gRPC API supports canceling a running job by terminating the worker process that owns it. That cancel usually succeeds from the client’s point of view (CANCELLED / NO
T_FOUND), and a replacement worker is respawned automatically.
However, killing a worker mid-CUDA can intermittently leave the GPU in a sticky error state (cudaErrorIllegalAddress, RMM / memory allocation failures). Respawning only t
hat worker often does not clear it; later jobs on the same device may keep failing.
Workaround: If jobs fail with CUDA / allocation errors after a cancel (or delete of a running job), shut down and restart the entire cuopt_grpc_server process. A clean re
start clears the bad device state in our testing. Send SIGINT to the server (Ctrl-C from the console), or otherwise stop and start the process/pod.
Queued-job cancel (job not yet claimed by a worker) is not affected by this issue.
Notes:
Preferring SIGTERM before SIGKILL, and delaying worker respawn, did not reliably prevent the poison in stress testing.
A durable fix likely needs either cooperative cancel inside the solver (between CUDA work) and/or a CUDA health check that treats this as fatal and exits so an orchestrat
or can recycle the server (similar to the Python cuOpt server).