Refactoring: derive nixlUcxThreadPoolEngine from nixlUcxThreadEngine

[iyastreb]

What?

To unify the code and reuse the same functions
It's pre-requisite for completions API PR

Summary by CodeRabbit

• Bug Fixes

  • Improved UCX backend thread lifecycle so the shared progress thread is created and joined only when enabled and shared workers are available.
  • Notification retrieval now properly runs progress when the shared progress thread is not active.

• Refactor

  • Updated the UCX thread-pool engine to inherit from the shared thread-engine, centralizing shared-worker sizing and notification behavior and removing pool-specific notification overrides.

[github-actions]

👋 Hi iyastreb! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

UCX thread handling now creates shared progress threads only when enabled and workers exist, and thread-pool instances inherit the thread-engine’s notification and progress behavior instead of maintaining separate shared-thread state.

Changes

UCX Thread Engine Refactor

Layer / File(s)	Summary
Shared progress thread creation and getNotifs logic src/plugins/ucx/ucx_backend.cpp, src/plugins/ucx/ucx_backend.h	nixlUcxThreadEngine documentation, constructor, destructor, and getNotifs now describe and implement conditional shared-thread creation and synchronous progress when no thread exists.
ThreadPoolEngine base class change and member cleanup src/plugins/ucx/ucx_backend.h	nixlUcxThreadPoolEngine now derives from nixlUcxThreadEngine, removes its own notification overrides, and drops the shared-thread and mutex members from its state.
ThreadPoolEngine construction and notification delegation src/plugins/ucx/ucx_backend.cpp	nixlUcxThreadPoolEngine construction delegates to the base class, validates shared-worker sizing through the inherited interface, stops joining a shared progress thread in destruction, and removes the local notification overrides.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and accurately summarizes the main refactor in this PR.
Description check	✅ Passed	The description covers the What and Why at a basic level, and the optional How section is the only major omission.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[iyastreb]

/build

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/plugins/ucx/ucx_backend.h (1)

328-328: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Override getNotifs() in the thread-pool engine. nixlUcxThreadPoolEngine can still be created with enableProgTh == false && num_threads > 0, so inheriting nixlUcxThreadEngine::getNotifs() makes the null-thread_ fallback call progressLoop() while dedicated workers are already active. That can progress the same UCX workers concurrently.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/plugins/ucx/ucx_backend.h` at line 328, The thread-pool engine inherits
getNotifs() from nixlUcxThreadEngine, which can incorrectly fall back to
progressLoop() even when nixlUcxThreadPoolEngine has dedicated worker threads
running. Override getNotifs() in nixlUcxThreadPoolEngine so notification
handling uses the thread-pool path only and never starts the null-thread
fallback that can progress UCX workers concurrently; update the override near
nixlUcxThreadPoolEngine and align it with the existing thread-pool/progress
logic in nixlUcxThreadEngine.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/plugins/ucx/ucx_backend.h`:
- Line 328: The thread-pool engine inherits getNotifs() from
nixlUcxThreadEngine, which can incorrectly fall back to progressLoop() even when
nixlUcxThreadPoolEngine has dedicated worker threads running. Override
getNotifs() in nixlUcxThreadPoolEngine so notification handling uses the
thread-pool path only and never starts the null-thread fallback that can
progress UCX workers concurrently; update the override near
nixlUcxThreadPoolEngine and align it with the existing thread-pool/progress
logic in nixlUcxThreadEngine.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 76f8d202-60e9-4b8d-808d-05c26446014e

📥 Commits

Reviewing files that changed from the base of the PR and between eef947f and 062d2f3.

📒 Files selected for processing (1)

• src/plugins/ucx/ucx_backend.h

[brminich]

/build

[iyastreb]

/build

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/plugins/ucx/ucx_backend.cpp`:
- Around line 366-368: Validate `num_threads` in `UcxBackend` before computing
`numSharedWorkers_` so it cannot exceed `getWorkers().size()`, and fail fast
with a runtime error if the requested thread count is invalid. Update the logic
around `shared_count` and the worker-indexing path in `UcxBackend::...` so it no
longer relies on `NIXL_ASSERT` for production safety, and ensure the derived
worker range used later when indexing workers is always bounds-checked.

In `@src/plugins/ucx/ucx_backend.h`:
- Around line 315-322: Move getSharedWorkersSize() back under the protected
section alongside appendNotif() in ucx_backend.h so it remains an internal
helper instead of a public API. Keep the UCX backend member grouping consistent
by preserving the intended public/protected/private layout, and make sure any
existing access by the pool still works through the protected override rather
than exposing it through the public interface.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 469a4e81-8e2e-47a6-b6f0-3394bff8ec44

📥 Commits

Reviewing files that changed from the base of the PR and between 062d2f3 and ed2fe96.

📒 Files selected for processing (2)

• src/plugins/ucx/ucx_backend.cpp
• src/plugins/ucx/ucx_backend.h

[iyastreb]

/build

[iyastreb]

/build

[iyastreb]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 5bd5a942

TL;DR: The build compiled and pushed the Docker image successfully; the job failed only in the "Allocate DL EP Environment" stage because Slurm could not allocate a GB200 node within its 1-hour --immediate=3600 window (salloc: error: Unable to allocate resources: Connection timed out). This is a cluster-capacity/infra failure, not a code defect in PR #1906 — retry or investigate gb200nvl72_cx8 availability.

Full analysis

Summary: Stage "Allocate DL EP Environment" (node 183/200) failed after a Slurm salloc for the gb200nvl72_cx8 partition timed out; the NIXL EP Docker build and install (stages 150/168) succeeded.

Root cause: salloc -N 1 -p gb200nvl72_cx8 ... --immediate=3600 was issued at 04:34:07 and produced no output for ~60 min until 05:34:16, when it reported salloc: Pending job allocation 1547293 / job queued and waiting for resources followed by salloc: error: Unable to allocate resources: Connection timed out. The 60-minute silence exactly matches the --immediate=3600 window: the requested exclusive GB200 node never became free, so Slurm gave up. This is a resource-availability/scheduling failure on the DL cluster, external to the PR's code — the compile, link, wheel build, and image push all completed cleanly beforehand. (The two build_helper_dl_ep matrix legs marked FAILURE are the intentional early-abort of redundant ucx build variants via "pipline stop on build_helper"; the real ucx builds 142/160 succeeded.)

Implicated commit: none — not caused by commit [REDACTED:Hex High Entropy String]; failure is infrastructure (Slurm allocation).

File: none (CI/infra: Slurm salloc call in the "Allocate DL EP Environment" pipeline step, partition gb200nvl72_cx8, job nixl-ci-ep-master-275).

Suggested fix: Re-run the job when GB200 capacity is available. If these allocation timeouts recur, either (a) raise --immediate/immediateTimeout so the job can queue longer, (b) add automatic retry/backoff around the salloc step in the pipeline library, or (c) work with the cluster admins on gb200nvl72_cx8 capacity/reservation for CI. Do not treat this as a PR code issue.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4d03c53f-ae17-444b-95e6-c47198d52e48 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 5bd5a942

TL;DR: The "Run Nixlbench tests" stage hung for ~41 minutes (05:16:37 → 05:57:52) during UCCL/UCX engine teardown and was SIGTERM-killed (exit 143); the hang is in the UCX progress-thread destructor refactored by this PR (#1906, deriving nixlUcxThreadPoolEngine from nixlUcxThreadEngine), where one worker's progress thread never exits/joins.

Full analysis

Summary: Jenkins nixl-ci-gpu #2710 stage "Run Nixlbench tests" (node 337) was ABORTED after ~51 min because a nixlbench UCCL WRITE VRAM→VRAM (ETCD) run hung on engine shutdown.

Root cause: Not a slow test — a hang. Timeline evidence: the last application output is at 05:16:37.768Z (Engine destroyed / Error/HUP on connection for the UCCL WRITE VRAM→VRAM ETCD case). Only one of the two parallel processes completed its Destroying Engine… / Engine destroyed sequence; the peer never finished teardown. There is then a ~41-minute gap with zero output until Jenkins sends the interrupt at 05:57:52.729Z and the shell returns exit code 143. The stall is in the UCX progress-thread shutdown/join path. PR #1906 restructures exactly that path (making nixlUcxThreadPoolEngine derive from nixlUcxThreadEngine, sharing the progress-thread lifecycle in ~nixlUcxThreadEngine/~nixlUcxThreadPoolEngine), and a destructor that fails to signal/join the progress thread (or spins in the pending-request drain loop) produces exactly this "one side prints 'Engine destroyed', the other hangs forever" signature.

Implicated commit: The PR under test, #1906 "Refactoring: derive nixlUcxThreadPoolEngine from nixlUcxThreadEngine" (head [REDACTED:Hex High Entropy String]). Closely related prior fix in the same code path: 0225cd5a0923 (Eylon Eliyahu Krause, #1880 — "ucx: fix infinite loop dropping pending requests on dedicated-thread exit").

File: src/plugins/ucx/ucx_backend.h:305-367 (the nixlUcxThreadEngine / nixlUcxThreadPoolEngine classes and their destructors); the corresponding ~nixlUcxThreadEngine() / ~nixlUcxThreadPoolEngine() progress-thread stop/join implementation in src/plugins/ucx/ucx_backend.cpp.

Suggested fix: In PR #1906, ensure the refactored destructor unconditionally signals the progress thread to stop and joins it, and that the shared vs. dedicated thread ownership isn't dropped/double-managed when nixlUcxThreadPoolEngine now derives from nixlUcxThreadEngine (e.g. the base ~nixlUcxThreadEngine must not leave thread_ running, and the derived class must not race on sharedThread_/dedicatedThreads_). Verify the pending-request drain loop from #1880 still terminates. Do not raise the stage time limit — this is a deadlock/hang, not a slow test. To reproduce/isolate, run ./bin/nixlbench --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type VRAM --check_consistency under two parallel workers and inspect the thread that never joins (e.g. gdb -p on the hung process at teardown).

Related: PR #1906 (#1906); prior fix in same path #1880 (0225cd5a0923).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 48399cb6-7d0b-48b1-b9e9-50d28e7d79e3 in the triage console for the audit trail.

[iyastreb]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 08a65b52

TL;DR: The "Run Nixlbench tests" stage was killed (exit 143) after hanging ~43 minutes; the last real output was the first ETCD-runtime UCCL test (READ VRAM→VRAM) printing "Engine destroyed" at 08:53:08, after which the UCCL backend hung on teardown/next-iteration re-init and never proceeded. Fix the UCCL Engine shutdown/re-init hang rather than raising the CI time limit.

Full analysis

Summary: Jenkins nixl-ci-gpu #2713, stage "Run Nixlbench tests" (node 356) was ABORTED (SIGTERM, exit 143) after the nixlbench process hung inside the UCCL benchmark loop.

Root cause: A hang, not slowness. Timestamps show continuous activity until 08:53:08.160Z ("Engine destroyed" + "Error/HUP on connection: 1.1.101.2:40351" for the first ETCD-runtime UCCL test, READ VRAM→VRAM). The very next loop iteration (UCCL READ VRAM→DRAM under ETCD) never printed even its "Connecting to ETCD" line. Then there is a ~17-minute dead gap until 09:10:16Z ("Cannot contact ... ChannelClosedException") and finally a kill at 09:36:22Z ("Sending interrupt signal / Killing processes", exit 143). The UCCL RDMA engine hung during teardown/re-initialization — one of the paired nixlbench workers destroyed its engine while the peer was left blocked (the ASIO variants of the same UCCL cases each finished in ~10s, so the problem is specific to the UCCL backend's shutdown/re-init between iterations, exposed under the ETCD runtime). The subsequent "export failed" / channel-closed lines are downstream symptoms of the node being torn down, not the cause.

Implicated commit: unknown for build #2713's PR #1906 (commit 08a65b5); the hanging component is the UCCL backend (src/plugins/uccl), last substantively touched by f8e2221 "Enable local xfer for UCCL backend (#1428)" and 63c494e "UCCL: Simplify and Optimize for batch transfers (#1271)", both by Pravein Govindan Kannan.

File: src/plugins/uccl (UCCL Engine create/destroy path); benchmark driver benchmark/nixlbench (the run_nixlbench_two_workers_etcd loop over UCCL cases).

Suggested fix: Investigate the UCCL Engine destroy → next-iteration re-init path for a deadlock/blocked accept. The teardown logs show one worker reaching "Engine destroyed" while the peer's background accept ("Waiting to accept incoming connection...") appears not to unblock; add a bounded timeout/interrupt to the UCCL accept and connection-teardown so a destroyed peer cannot leave the other worker blocked forever. As an interim CI mitigation, wrap each nixlbench invocation with a per-test timeout so a single hung UCCL case fails fast instead of consuming the whole stage budget. Do not simply raise the stage time limit — the process was hung, not making progress. Track against PR #1724 ("UCCL crash Fix + GDB trap").

Related: PR #1724 (UCCL crash fix); prior UCCL work PR #1428, #1271, #1151, #895.

[brminich]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 70081dc4

TL;DR: The aarch64 Test CPP stage hung for ~22 minutes on the gtest ucx/TestErrorHandling.LoadRemoteThenFail/0 (the test right after the last-logged BasicXfer) and was killed by Jenkins with SIGTERM (exit 143); the hang is in the UCP endpoint error-path teardown introduced by the "forcibly close UCP endpoints" change, not a genuinely slow build.

Full analysis

Summary: Jenkins build #2345, stage 501 (aarch64/nixl-base-pytorch26.06-cuda13.3-ubuntu24.04 → Test CPP) failed with exit code 143 after the gtest run stalled during the UCX error-handling test group.

Root cause: A process hang, not a timeout. The stage log shows continuous test progress up to [14:15:48] [83/154] ucx/TestErrorHandling.BasicXfer/0 (954 ms), then zero output for ~22 minutes until [14:38:00] Sending interrupt signal to process / script returned exit code 143. The next test in the suite ([84/154] ucx/TestErrorHandling.LoadRemoteThenFail/0, which completes normally in <1 s on the x86_64 run in stage 449) started and never returned — it hung. This is the UCX error-injection/endpoint-teardown path, which is exactly what the just-merged endpoint-close change modifies; forcibly closing UCP endpoints on the failure path can block indefinitely in the drain/close/progress loop (same class of bug as the previously fixed infinite loop on thread exit).

Implicated commit: 22cdab3de2e9 — "PLUGINS/UCX: Added option to forcibly close UCP endpoints. (#1846)" by Raul Akhmetshin, 2026-07-15 (merged same day as this build; most recent change to src/plugins/ucx/ucx_backend.cpp).

File: src/plugins/ucx/ucx_backend.cpp — the UCP endpoint force-close / disconnect path exercised by ucx/TestErrorHandling.LoadRemoteThenFail (endpoint close + ucp_worker_progress drain loop). Related test driver: test/gtest UCX TestErrorHandling fixture.

Suggested fix: Do not raise the CI time limit — this is a hang. Investigate the endpoint force-close logic added in 22cdab3de2e9: the close/flush request is likely being waited on with an unbounded while (!completed) ucp_worker_progress(...) loop that never completes on the error/peer-failure path (or a lock/condition-variable that is never signaled when the endpoint is closed with UCP_EP_CLOSE_FLAG_FORCE). Bound the wait (timeout + break), ensure force-close uses the force flag and doesn't wait for peer acks, and make sure the progress thread is still pumping while the close request is pending. Reproduce locally with gtest-parallel --workers=1 --serialize_test_cases ./bin/gtest --gtest_filter='*TestErrorHandling.LoadRemoteThenFail*' on aarch64.

Related: PR #1906 (this build); commit/PR #1846 / re-land PR #1950 ("Added option to forcibly close UCP endpoints"); prior related fix PR #1880 ("fix infinite loop dropping pending requests on dedicated-thread exit").

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 70081dc4

TL;DR: The Python-test stage (Run Python tests, ucx-v1.22.x variant) failed because all four test_sync_mode_agent[True-*] cases hit RuntimeError: Failed to bind metadata listener socket — the agent's metadata listener could not bind its fixed default port ("Address already in use [98]"), a port collision on the shared host. Fix: have the test pass an OS-assigned/ephemeral listener port (or a per-nixl_concurrent_id unique port) instead of the fixed default.

Full analysis

Summary: test/python/test_nixl_api.py::test_sync_mode_agent[True-*] (4 params, all with enable_listen=True) fail during the ucx-v1.22.x "Run Python tests" stage; the ucx-v1.22.x build_helper (stage 125) FAILURE is just the pipeline-stop propagation of this test failure.

Root cause: When enable_listen_thread=True, the agent starts a metadata listener that binds a fixed default TCP port. The log shows E ... metadata_stream.cpp:83 Socket Bind failed while setting up listener for MD: Address already in use [98] immediately before the failures, and the enable_listen=False variants (no listener) all pass. The default port was already occupied — a collision on shared host mizu04, where the ucx-master and ucx-v1.22.x container variants run concurrently (and/or a stale agent held the port). The test creates the config at test/python/test_nixl_api.py:22 (nixl_agent_config(sync_mode=sync_mode, enable_listen_thread=enable_listen)) without specifying a listen port, so it can't avoid the collision. Not a timeout/hang — the failure is an immediate bind error.

Implicated commit: No single code commit broke this; it's an environment/port-collision issue. Relevant context: 9cb27fbf7f5b (Ilya Yanok, "Let Python API control the sync mode (#1501)") added test_sync_mode_agent, and 0797ede5c477 (Guy Ealey Morag, "Support OS-assigned ports for metadata listener (#1439)") added the mechanism this test should use.

File: test/python/test_nixl_api.py:22 (config construction for test_sync_mode_agent); underlying bind at metadata_stream.cpp:83.

Suggested fix: Make the listener use a non-fixed port in this test — pass an OS-assigned (ephemeral, port 0) or per-nixl_concurrent_id unique listen_port into nixl_agent_config(...) for the enable_listen=True cases (leveraging the OS-assigned-port support from PR #1439). Also ensure the test harness kills/reaps stale agents and that concurrent ucx-master/ucx-v1.22.x runs on the same node don't share the default metadata listener port range.

Related: PR #1439 (OS-assigned ports for metadata listener); PR #1501 (added test_sync_mode_agent). No existing issue found for this specific failure.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 70081dc4

TL;DR: The "Allocate DL EP Environment" stage failed because a Slurm salloc on the gb200nvl72_cx8 GPU partition sat queued for the full --immediate=3600 (1 hour) window and then died with Unable to allocate resources: Connection timed out — a cluster capacity/infra issue, not a code bug in PR #1906.

Full analysis

Summary: Stage 200 (Allocate DL EP Environment) failed at Slurm node allocation; the earlier compile/image stages all succeeded and stage 124's failure was just the parallel build-matrix being stopped alongside it.

Root cause: salloc -N 1 -p gb200nvl72_cx8 ... --immediate=3600 was issued at 14:33:51, Slurm reported job 1563286 queued and waiting for resources, and exactly one hour later (15:33:59) it returned salloc: error: Unable to allocate resources: Connection timed out. The ~60-minute gap equals the immediate-timeout window — the GB200 partition had no free node to grant within the hour, so allocation timed out. This is a genuine wait for busy/unavailable hardware (continuous "queued and waiting" state), not an application hang and not a code defect. Build artifacts were produced and pushed successfully before this step.

Implicated commit: none — not caused by commit [REDACTED:Hex High Entropy String] or PR #1906; failure is in CI infrastructure resource allocation.

File: Jenkins pipeline slurm.allocation step (partition gb200nvl72_cx8, --immediate=3600, headNode dlcluster.nvidia.com) — not a repository source file.

Suggested fix: Retry the build — this is a transient GB200 cluster-capacity/queue issue. If it recurs, (1) check gb200nvl72_cx8 partition availability/quota for the blackwell account, and (2) consider making the allocation step auto-retry or raise --immediate only after confirming nodes exist, since the SSH connection also timed out at exactly 1h (verify the SSH ConnectTimeout/ServerAliveInterval aren't cutting the wait short). Do not treat this as a code change to PR #1906.

Related: none (no matching issues/PRs for this allocation failure).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7df2d54b-c38a-4f82-83f1-8960c0191f79 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 70081dc4

TL;DR: The "Build sanity image" stage (#470) failed because apt couldn't reach ports.ubuntu.com ("Network is unreachable" / connection timed out) while installing curl/git in contrib/Dockerfile.vllm; this is a CI network/infra issue, not a code bug — retry the build (and ideally make the image use a mirror or a base image with the tools preinstalled).

Full analysis

Summary: aarch64 "Build sanity image" stage failed at docker build -f contrib/Dockerfile.vllm on the apt install -y curl git step with exit code 100. (The upstream build_helper_vllm/aarch64 stage #332 was cancelled as a consequence — the wheel build itself succeeded.)

Root cause: Transient network outage on the Jenkins build node. The apt update phase succeeded (from an NVIDIA/deadsnakes mirror), but downloading the .deb packages from ports.ubuntu.com failed repeatedly: Cannot initiate the connection to ports.ubuntu.com:80 (2620:2d:4002:1::10c). - connect (101: Network is unreachable) and connection timed out for the IPv4 hosts. apt then aborted with E: Unable to fetch some archives → exit status 100. Note the build was making continuous progress (package lists fetched, then immediate connection failures) — this is a hard network failure, not a hang or timeout. No source-code change is implicated.

Implicated commit: none — infrastructure failure, unrelated to commit [REDACTED:Hex High Entropy String] / PR #1906.

File: contrib/Dockerfile.vllm:22-25 (the RUN apt update && apt install -y curl git ... layer).

Suggested fix:

1. Re-run the failed build — the outage to ports.ubuntu.com is transient and the same commit should pass once network connectivity is restored.
2. To harden against recurrence, make the layer resilient: add retries and prefer IPv4 (the IPv6 ports.ubuntu.com addresses were unreachable), e.g. configure Acquire::ForceIPv4 "true"; and apt-get -o Acquire::Retries=3 install ..., or point the base image at an internal/NVIDIA apt mirror.
3. Better still, bake curl and git into the vllm-nixl-base image so the PR sanity build doesn't need external apt access at all.

Related: none found.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id bb9967c7-1dea-4543-842e-83699724e93d in the triage console for the audit trail.

[iyastreb]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit adac02f4

TL;DR: The "Run CPP tests" stage exited 42 because the gtest CI harness treats the benign environmental warning 8 NVIDIA GPU(s) were detected, but UCX CUDA support was not found! as a fatal error, flagging 67 otherwise-passing tests; plus two genuine failures (HardwareWarningTest.NoWarningWhenIbAndCudaSupported throwing "Failed to create UCX context: No such device", and MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote where sendLocalMD() returned -2). The test container's UCX build lacks CUDA/IB transport support, so those tests cannot pass in this environment.

Full analysis

Summary: Stages 203 and 242 ("Run CPP tests") failed with exit code 42; 69/154 tests reported failed, almost all due to a UCX build lacking CUDA/IB support in the test container.

Root cause: The UCX in the test image was built/loaded without CUDA and accelerated-IB transports (UCX WARN transports 'ib','cuda' are not available; ERROR no usable transports/devices (asked ib,cuda...)). The gtest harness raises "ATTENTION: Unexpected NIXL warning or error detected!" and returns exit code 42 for every test that emits the ucx_utils.cpp:639/:646 GPU/IB-not-supported warning — that accounts for ~67 of the 69 "failures" (they still print [ OK ]/[ PASSED ]). Two are real failures: HardwareWarningTest.NoWarningWhenIbAndCudaSupported (exit 1) forces a UCX context requiring ib,cuda and throws Failed to create UCX context: No such device because neither transport exists in this image; and MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote where sendLocalMD() returned -2 (NIXL_ERR_BACKEND, an etcd backend issue). Note the build_helper ucx-master/ucx-v1.22.x parallel stages (124/125) also failed first, consistent with a UCX build/setup problem feeding a bad UCX into the test image.

Implicated commit: unknown (recent history on test/gtest is all telemetry/tracing work by e-eygin, none touching UCX transport detection; the transport-less UCX build in the container is the environmental trigger).

File: src/utils/ucx/ucx_utils.cpp:639 and :646 (warning source); test HardwareWarningTest.NoWarningWhenIbAndCudaSupported and MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote in test/gtest.

Suggested fix: Fix the test container's UCX so it is built with CUDA and IB/RDMA transport support (investigate the failing build_helper/ucx-master and ucx-v1.22.x stages 124/125 — the UCX build there is what the image consumes). Until UCX has CUDA/IB, HardwareWarningTest.NoWarningWhenIbAndCudaSupported cannot pass and every UCX test trips the "unexpected warning" check. Separately, the harness should not convert the informational ucx_utils.cpp:639/:646 GPU/IB-capability warnings into exit code 42 on hosts where UCX legitimately lacks CUDA/IB. Also confirm the etcd server is reachable for sendLocalMD (the -2/NIXL_ERR_BACKEND suggests an etcd connectivity/backend problem).

Related: none found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit adac02f4

TL;DR: The "Run Nixlbench tests" stage was killed by Jenkins (exit 143) after an ~18-minute hang, not a slow test; the first UCCL+ETCD benchmark hung during engine teardown (one nixlbench worker never exited after "Engine destroyed"/epoll_client HUP), so parallel blocked until the wall-clock kill. Investigate the UCX engine shutdown/progress-thread lifecycle changed in PR #1906.

Full analysis

Summary: Nixlbench stage (node 356) hung during teardown of the first UCCL+ETCD benchmark and was SIGTERM-killed after ~18 min of no output.

Root cause: Engine/worker shutdown deadlock. At 06:47:45–48 both workers reached "Destroying Engine…"; one printed "Engine destroyed" but the peer hit event_loop rdma/epoll_client.h:207 Error/HUP on connection: 1.1.101.4:44479 and never completed teardown/exit. Because the two workers run under parallel --halt now,fail=1, the surviving hung process prevented parallel from returning, producing 18 minutes of silence until Jenkins killed the build (exit 143 is the symptom, not the cause). This is a hang, so raising the time limit is NOT the fix.

Implicated commit: PR #1906 (adac02f, "Refactoring: derive nixlUcxThreadPoolEngine from nixlUcxThreadEngine") — the change under test; closely related prior fix is 0225cd5 (#1880, "ucx: fix infinite loop dropping pending requests on dedicated-thread exit") by Eylon Eliyahu Krause, same shutdown code path.

File: UCX engine shutdown path exercised via src/plugins/ucx (thread/thread-pool engine destructor + pending-request drain); the visible hang site is rdma/epoll_client.h:207 (UCCL provider) during engine destroy.

Suggested fix: Reproduce the UCCL+ETCD READ VRAM→VRAM case and capture a stack trace (gdb/py-spy) of the stuck nixlbench at teardown. Audit PR #1906's changes to the nixlUcxThreadEngine/nixlUcxThreadPoolEngine destructor and progress-thread join/pending-request-drain logic — a base-class destructor that no longer stops/joins the progress thread (or double-drains) in the derived type would leave a worker blocked on a HUP'd connection. Ensure the shutdown ordering from #1880 (drain pending requests, then join thread, then destroy endpoints) is preserved in the refactored hierarchy. Do not simply increase the Jenkins timeout.

Related: PR #1906 (#1906); PR #1880 (#1880); PR #1846 (#1846, forcibly close UCP endpoints).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit adac02f4

TL;DR: The "Run CPP tests" stage crashed (SIGABRT, exit 134) in ucx_backend_multi because UCX failed to open the TCP interface over IB device ibp84s0f1 ("No such device") on node mizu02; this is an environment mismatch, not a hang or a code bug.

Full analysis

Summary: Jenkins nixl-ci-gpu #2783 stage 203 ("Run CPP tests") failed when ./bin/ucx_backend_multi aborted with a std::runtime_error: Failed to create UCX worker: No such device.

Root cause: UCX on test node mizu02 tried to open a TCP transport bound to IB device ibp84s0f1 (uct_iface_open(tcp/ibp84s0f1) failed: No such device) and the test threw an unhandled std::runtime_error from the failed worker creation, terminating the process (exit 134). The preceding warnings — "8 NVIDIA GPU(s) were detected, but UCX CUDA support was not found" and "4 IB device(s) were detected, but accelerated IB support was not found" — show the UCX in this container image lacks CUDA/IB-accelerated transports, so UCX falls back to a device/interface name that isn't valid on this host. This is an environment/build-config problem, not a source regression: the log shows continuous activity up to the abort (no multi-minute gaps), so it is a crash, not a timeout/hang. (Note the second failure, stage 124 build_helper/x86_64/ucx-master, is a separate short build_helper failure that the pipeline recovered from via retry — stage 142 succeeded.)

Implicated commit: unknown — no code change in the failing test path is implicated; the failure originates from the UCX runtime/device configuration on node mizu02. Recent CI-image change PR #1869 ("Switch CI base image to pytorch + CUDA 13.3") is a plausible contributor to the UCX-without-CUDA/IB-accel build and is worth checking.

File: test/test_cpp.sh:100 (invocation of ./bin/ucx_backend_multi); crash surfaces from UCX worker creation in the UCX backend (see ucx_utils.cpp:639/646 warnings and ucp_worker.c:1492).

Suggested fix:

1. Investigate node mizu02 / the CI image: the UCX build reports missing CUDA and accelerated-IB support and is selecting IB device ibp84s0f1 which is absent. Rebuild/install UCX with CUDA + IB (rdma-core/verbs) support in the CUDA 13.3 pytorch base image, or pin UCX_NET_DEVICES/UCX_TLS in the test environment so ucx_backend_multi doesn't attempt tcp/ibp84s0f1 on a host without it.
2. Harden the test: ucx_backend_multi should catch the worker-creation std::runtime_error and skip/xfail with a clear message instead of aborting, so an environment issue doesn't crash the whole CPP stage.
3. Re-run on a node whose IB devices match the expected names to confirm it's host-specific.

Related: PR #1869 (CI base image switch to pytorch + CUDA 13.3); PR #1743 (run gtest in single process). No existing issue matches the "No such device" UCX worker failure.