PLUGINS/UCX: Added option to forcibly close UCP endpoints. (#1846)

[ovidiusm]

What?

Added option ucx_ep_close_force to forcibly close UCP endpoints.

Why?

At the moment ucp_nb_close_nbx is called with zeroed flags, meaning UCP endpoints (connected remotely) are destroyed only when the corresponding UCP worker is destroyed. Passing UCP_EP_CLOSE_FLAG_FORCE forces the destruction of a UCP endpoint and corresponding UCT endpoints, releasing UCX transport layer resources, e.g. UCT CUDA IPC cache openucx/ucx#11335.
The issue was observed using nixl_ep examples in case of repeated connections and disconnections between ranks.

[github-actions]

👋 Hi ovidiusm! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[ovidiusm]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 39f23d5a

TL;DR: The "Run CPP tests" stages failed for two reasons: (1) HardwareWarningTest.NoWarningWhenIbAndCudaSupported throws Failed to create UCX context: No such device because it requests ib,cuda UCX transports that don't exist on this node, and (2) ucx_backend_test hit a UCX SIGABRT (failed to arm iface: Input/output error); the other 68 "failures" are benign exit-code-42 warnings and are not regressions.

Full analysis

Summary: CPP test stages (203/242) failed on one deterministic gtest assertion plus a UCX worker abort; a large tail of exit-42 results are environmental warning artifacts, not real failures.

Root cause:

• Real failure (stage 242): HardwareWarningTest.NoWarningWhenIbAndCudaSupported constructs a UCX context expecting ib and cuda transports to be usable and no warning to be raised, but on mizu02 UCX was built/configured without CUDA support and IB devices aren't usable (transports 'ib','cuda' are not available; no usable transports/devices (asked ib,cuda ...)), so context creation throws Failed to create UCX context: No such device instead of succeeding. The test hard-codes a hardware assumption not met on this runner.
• Real failure (stage 203): ucx_backend_test aborted in UCX itself — ucp_worker.c:669 Fatal: failed to arm iface 0x...: Input/output error (exit 134). This is a UCX/IB-interface-level fault (the node's IB device ibp75s0f0/f1 reports "No such device" throughout), not a NIXL logic bug.
• The remaining 68 tests returning exit code 42 are the test harness's "Unexpected NIXL warning detected" sentinel, all triggered by the benign, repeated warning 8 NVIDIA GPU(s) were detected, but UCX CUDA support was not found. These are an environment/UCX-build artifact on this node, not per-test regressions.

Implicated commit: unknown — the HardwareWarningTest.* cases (NoWarningWhenIbAndCudaSupported, WarnWhenIbPresentButRdmaNotSupported, EfaHardwareMismatch*) appear to be recently added hardware-detection tests; I could not resolve the exact test source path or authoring commit from the API. Related test/harness churn is in test/gtest by e-eygin (e.g. #1919, #1927).

File: UCX plugin hardware-warning gtest (path not confirmed; likely under test/gtest/.../hardware_warning_test.cpp); failing UCX call originates at ucp_context.c:1750 / ucp_worker.c:669. The benign warning is emitted at src/.../ucx_utils.cpp:639 and :646.

Suggested fix:

1. Make HardwareWarningTest.NoWarningWhenIbAndCudaSupported skip (GTEST_SKIP) when UCX lacks usable ib/cuda transports, rather than asserting success — detect transport availability up front and gate the assertion, so nodes without CUDA-enabled UCX / usable IB don't hard-fail.
2. Investigate the runner mizu02: UCX is built without CUDA support and its IB devices (ibp75s0f0/f1) enumerate as "No such device" — the CI image/UCX build should include CUDA transport and the IB devices must be present, otherwise these tests can never pass here.
3. Separately triage the ucx_backend_test failed to arm iface: Input/output error abort as a UCX/IB device-health issue on the same node.

Related: #1909 (open PR: "Fail on unexpectedly skipped tests and fix device tests") appears directly relevant to gating these device-dependent tests. none other found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 39f23d5a

TL;DR: The "Allocate DL EP Environment" stage failed because a Slurm salloc for a GB200 node on dlcluster.nvidia.com timed out after its 3600s --immediate window with no resources available — a cluster capacity/infrastructure issue, not a code or test failure.

Full analysis

Summary: Stage 245 "Allocate DL EP Environment" (build #323) failed; the Slurm allocation request for partition gb200nvl72_cx8 could not be satisfied.

Root cause: The salloc -N 1 -p gb200nvl72_cx8 --immediate=3600 --time=01:30:00 --account=blackwell request was queued at 14:22:00 and, after waiting the full --immediate=3600 (1 hour) window with no free nodes, exited at 15:22:09 with salloc: error: Unable to allocate resources: Connection timed out. The build and image push (through stage 168/150) all succeeded; only the cluster node allocation failed. This is an environment/capacity problem, not a regression from PR #1950. (The parallel "build_helper ucx-v1.22.x" matrix entry, stage 125, shows only a successful image build; its FAILURE marking is a side effect of the parallel stage being aborted, not an independent build error.)

Implicated commit: unknown — not code-related; commit 39f23d5 is not implicated.

File: N/A (Jenkins pipeline Slurm allocation step, slurm.allocation in swx-jenkins-lib)

Suggested fix: Re-run the job — this is a transient GB200 capacity shortage. If it recurs, either (a) raise the --immediate wait window / switch to a queuing (non-immediate) allocation with a sensible timeout, (b) confirm the blackwell account has quota on gb200nvl72_cx8, or (c) add a retry-with-backoff around the salloc step so a momentary lack of free nodes doesn't fail the whole pipeline. Do not treat this as a test/code failure for PR #1950.

Related: none found.