RDMA transfer to a stale donor aborts the process at libfabric with no fallback

[emricksini-h]

Description

We run ModelExpress on EKS with EFA, and one of our vLLM pods crashed on startup while loading a model with --load-format=mx. The worker picked a donor, set up the RDMA transfer, and then the whole process aborted inside libfabric with a bad queue pair error. It did not fall back to the RunAI streamer or any other strategy.

Here are the logs:

Worker 0 (Worker 1 is identical with agent mx-auto-worker1-a78963c5, port 5556, source 0c9121ec):

INFO 06-30 10:41:55 [rdma_strategy.py:154] [Worker 0] Found 1 ready source worker(s)
INFO 06-30 10:41:55 [rdma_strategy.py:114] [Worker 0] Trying source worker 313d4819 (988 tensors)
INFO 06-30 10:41:57 [rdma_strategy.py:280] [Worker 0] Receiving 988 tensors from source
INFO 06-30 10:41:57 [nixl_transfer.py:480] [TIMING] add_remote_agent: 0.034s (agent=b'mx-auto-worker0-37b45556', blob=238999 bytes)
INFO 06-30 10:41:57 [nixl_transfer.py:529] [TIMING] match_tensors: 0.001s (988 tensors, 35.71 GB)
INFO 06-30 10:41:57 [nixl_transfer.py:549] [TIMING] prep_xfer_dlist: 0.000s
Libfabric EFA provider has encountered an internal error:

Libfabric error: (107) Transport endpoint is not connected
EFA internal error: (9) Bad queue pair (QP) number (QP does not exist or is in error state)

Your application will now abort().

After digging in, the donor it picked was registered and listed as READY, but the worker behind it was already gone, so the queue pair it tried to read from no longer existed.

We would expect to be able to recover by falling back to another strategy instead of crashing in that case.

[emricksini-h]

It took us a bit of time to understand why a donor was marked as READY while it was already gone (the heartbeat mechanism should have prevented that). After some debugging we realized the following sequence of events occured:

• Our Redis (Valkey) data volume filled up. It is a 1Gi PVC, and the append-only file grew until the disk was at 100%.
• Once the disk was full, Valkey could not write its snapshot or and as stop-writes-on-bgsave-error was "on" it started rejecting every write with MISCONF and went effectively read-only.
• The reaper needs to write to mark a worker STALE and to garbage collect it. With writes blocked, it could not do either. We saw it fail repeatedly in the server logs on the exact workers involved.
• Because those two workers were never marked STALE, they stayed in the registry as READY even though their processes were gone.
• A new pod started, asked for ready sources, got one of these dead donors, and tried to RDMA-read from it. The remote queue pair no longer existed, and libfabric aborted, which is the crash in the issue