PLUGINS/UCX: Gate error handling mode via NIXL_UCX_ERR_HANDLING_MODE env var#32
Open
aviallon wants to merge 102 commits into
Open
PLUGINS/UCX: Gate error handling mode via NIXL_UCX_ERR_HANDLING_MODE env var#32aviallon wants to merge 102 commits into
aviallon wants to merge 102 commits into
Conversation
* Simplify nixl installation Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add option to toggle between release and dev builds Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Install correct pytorch for the image Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Update CI tag Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Fix copyright Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Address coderabbit comments Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Chain exception Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Update src/bindings/python/nixl-meta/nixl/__init__.py Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com> Signed-off-by: ovidiusm <ovidium@nvidia.com> * Fix indentation Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Address review comments Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add warning in case of CPU pytorch installation Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Bump CI tag to trigger base image rebuild Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove warning Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Signed-off-by: ovidiusm <ovidium@nvidia.com> Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Match the fallback header probe to NIXL's etcd include style and remove the invalid break so external etcd installs can enable HAVE_ETCD during configure. Co-authored-by: Zihao Zhao <zizhao@nvidia.com>
* CI: Bump UCX version from v1.20.x to v1.21.x
Update the default UCX version across all CI configs, Dockerfiles,
build scripts, and documentation to track the v1.21.x release.
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* ucx: map UCS_ERR_CANCELED to NIXL_ERR_REMOTE_DISCONNECT for UCX >= 1.21
UCX 1.21 changed the behavior of peer error handling
(UCP_ERR_HANDLING_MODE_PEER): when a remote endpoint fails, pending
in-flight requests are now completed with UCS_ERR_CANCELED instead of
UCS_ERR_ENDPOINT_TIMEOUT or UCS_ERR_CONNECTION_RESET as in prior
versions.
NIXL uses UCP_ERR_HANDLING_MODE_PEER exclusively, so UCS_ERR_CANCELED
on a posted transfer request is semantically equivalent to a remote
disconnect, not an application-initiated cancel. The previous mapping
of UCS_ERR_CANCELED -> NIXL_ERR_CANCELED was correct under UCX 1.20.x
but breaks under 1.21.x in two ways:
1. Test assertions expecting NIXL_ERR_REMOTE_DISCONNECT fail.
2. The guard in postXfer() that early-returns on NIXL_ERR_REMOTE_DISCONNECT
does not fire, causing waitForCompletion() to be called with an
already-completed (invalid) request handle, resulting in an infinite
hot-spin.
Fix: add UCS_ERR_CANCELED to the NIXL_ERR_REMOTE_DISCONNECT case group
in ucx_status_to_nixl(). The only explicit reqCancel() call site
(ucx_backend.cpp release()) does not inspect the return value, so this
remapping is safe.
Fixes: ucx_threadpool/TestErrorHandling.XferThenFail hanging under
UCX 1.21.x in TCP-only environments (no RC/InfiniBand).
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
---------
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* nixl_ep: Support reusing CUDA graphs post-scale * nixl_ep: Optional connect_ranks without rank activation * Fix precommit
* DOCKER: Update curl version to 8.19.0 in manylinux Dockerfile * DOCKER: Add wget retry flags to curl download in manylinux Dockerfile
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
…esh (#1570) * nixlbench: fix ttl for immature expiration because of late first refresh - add Check() in etcd runtime, and call at the start of transfer(), to surface any KeepAlive object's error in previous attempts to refresh the lease. Note that Check() internally refreshes the lease. - add a longer (30 sec) ttl as the first refresh attempt may be late. - The first refresh from the dedicated thread inside KeepAlive object may be late, due to overhead to start the thread. - The first refresh from the first Check() in transfer() may be late due to nixlbench start up overhead. * nixlbench: start the first liveness check right away * refactor(nixlbench): Hoist checkKeepAlive() into xferBenchRT base class --------- Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* CORE: Remove pointer indirection in transfer request.
…nd introduce option for other engines (#1444) RDMA GET responses carry an empty HTTP body with data delivered out-of-band. When ObjectScale returns checksum headers (e.g. for objects uploaded with checksums), the SDK validates against the empty body and fails with "Response checksums mismatch". Default the Dell ObjectScale accelerated engine to resp_checksum=required (WHEN_REQUIRED) so the SDK skips body-checksum validation. Transport integrity is already ensured by RoCEv2 iCRC. The setting is exposed as a common S3 client parameter so users can override it for all OBJ engines. Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com> Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
--------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
* bindings/python: add RW thread sync mode * api/python: allow setting sync_mode from Python
Test DL Matrix CI_IMAGE_TAG check added to init script Verify CI_IMAGE_TAG is increased in all CI files Signed-off-by: Iaroslav Sydoruk <isydoruk@nvidia.com>
CHECK_NIXL_ERROR expanded its parameter twice — once in the condition and once in the error message. When the argument was a function call (e.g. agent->registerMem()), the function executed twice on failure. Evaluate the argument once into a local variable before testing and printing. Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
Export missing symbols in the meta package
* Add DOCA Telemetry Exporter plugin for NIXL Integrate DOCA Telemetry Exporter as a telemetry plugin, following the same architecture as the existing Prometheus exporter. The plugin is conditionally compiled only when DOCA headers are present on the system, ensuring users without DOCA access are unaffected. Key details: - Exports metrics via DOCA Telemetry Exporter with Prometheus endpoint - Default port 9091 (separate from Prometheus exporter on 9090) - Env vars: NIXL_TELEMETRY_DOCA_PROMETHEUS_PORT, NIXL_TELEMETRY_DOCA_PROMETHEUS_LOCAL - Build auto-detects DOCA headers; gracefully skips if absent - DOCA headers are not exposed in any installed/public headers Signed-off-by: Batsheva Black <bblack@nvidia.com> * telemetry: share DOCA context across agents in same process DOCA's metrics API only supports one metrics context per process. Share the schema, source, and metrics context via a process-wide weak_ptr so the first agent creates the pipeline and subsequent agents reuse it. Each agent is distinguished by the agent_name variable label on every metric it exports. The shared context is cleaned up via RAII when the last agent's exporter is released. Signed-off-by: Batsheva Black <bblack@nvidia.com> * fix clang * address review comments on DOCA telemetry exporter - Throw std::runtime_error on init failure so the plugin creator returns nullptr instead of a broken exporter - Add const to lock_guard for const-correctness - Document setenv thread-safety constraint - Detect host architecture for DOCA lib path (ARM64 support) - Fix README grammar Signed-off-by: Batsheva Black <bblack@nvidia.com> * rename DOCA exporter header guard to repo convention * address review comments on DOCA telemetry exporter * fix comments * fix comments * fix clang * add mutex for DOCA metric recording (CLX API is not thread-safe) The CLX Metrics API that DTE is built on requires external synchronisation for all metric recording functions. Protect metrics_add_counter / metrics_add_gauge with a dedicated mutex and update the DocaSharedContext comment accordingly. * Update src/plugins/telemetry/doca/doca_exporter.cpp Co-authored-by: Colin Hirsch <chirsch@nvidia.com> Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com> * address review: cleanup DOCA exporter - Add [[nodiscard]] to getHostname() - Use std::array for hostname buffer - Move DocaSharedContext definition to .cpp, forward-declare in header - Remove dead initialized_ member (constructor throws on failure) - Make initializeDoca() throw directly instead of returning status * fix_clang * resolve comments * fix to const * cleanup in failure case * fix clang * Fix DOCA exporter build after telemetry API changes on main Update plugin API version from V1 to V2, add missing doca_error.h include, and replace removed event.timestampUs_/eventName_ with eventType_ enum and nixlEnumStrings::telemetryEventTypeStr(). * add nodiscard * restyle * add time * remove helper --------- Signed-off-by: Batsheva Black <bblack@nvidia.com> Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com> Co-authored-by: Colin Hirsch <chirsch@nvidia.com> Co-authored-by: ovidiusm <ovidium@nvidia.com> Co-authored-by: Batsheva Black <you@example.com>
* Add VRAM and Accelerated engine options for nixl_object_test
Examples:
./build/test/unit/plugins/object/nixl_object_test -n 100 -s 16M -t 5
./build/test/unit/plugins/object/nixl_object_test -n 100 -s 16M -t 5 -e http://s3.example
./build/test/unit/plugins/object/nixl_object_test -a -T dell -e http://10.10.10.10:9000 -
./build/test/unit/plugins/object/nixl_object_test -v -a -T dell -e http://10.10.10.10:9000
* Address code review comments
* Address code review comment about filling the validation buffer
* Addressed the allocation and fill code review comment
* • Help text — -v, --vram now shows (requires --accelerated and CUDA/GPU
support).
• goto past RAII — Moved host_buffer, expected_buffer, fill_test_pattern,
and object_prefix declarations to before createBackend, so the goto cleanup
at the createBackend failure path no longer jumps past their
initialization.
• CUDA device ID — Declared cuda_dev_id under #ifdef HAVE_CUDA, captured
via cudaGetDevice(&cuda_dev_id) after the device count check, and used it
in vram_buf[i].devId instead of the hardcoded 0.
---------
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
…#1496) DefaultObjEngineImpl::queryMem checks object existence by calling checkObjectExists synchronously in a loop — one S3 HeadObject request per descriptor, waiting for each to complete before issuing the next. For a list of N descriptors, this serializes N network round-trips to S3. The OBJ plugin already uses async patterns for putObjectAsync and getObjectAsync, but checkObjectExists had no async counterpart. Fix: Add checkObjectExistsAsync to the iS3Client interface with implementations for both the standard S3 and S3 CRT clients. Refactor queryMem to launch all HeadObject requests in parallel using futures/ promises, then wait for all to complete. Remove the now-dead synchronous checkObjectExists. Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
… printStats (#1588) Throughput needs total iterations (all threads combined) since wall_time covers all threads running in parallel. Latency needs per-thread iterations since each thread runs that many sequentially. Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
--------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
--------- Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
## What Update Jenkinsfile behavior on corner case failures ## Why ? In some very specific case the job may fail to run but will report as succeeded to GH checks ## How ? force update build result on any failure Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Signed-off-by: Raul Akhmetshin <rakhmetshin@nvidia.com>
* python: add Intel XPU device support in _api.py Map "xpu" device strings to VRAM_SEG alongside "cuda", and replace the binary cuda/cpu ternary with an explicit if/elif/else in all four descriptor-list helpers (nixlXferDList and nixlRegDList, both single- tensor and list-of-tensors paths). PyTorch's XPU backend reports device as "xpu:N" rather than "cuda:N", so without this change XPU tensors would be misclassified as DRAM. Signed-off-by: Daniel Socek <daniel.socek@intel.com> * python: dedupe device memtype mapping and guard inputs - Add get_mem_type_from_device() helper based on device.type - Replace duplicated tensor device parsing in transfer and registration descriptor paths with the shared helper - Add guards for empty descriptor inputs to avoid descs[0] index errors in descriptor builders * python: fix empty/scalar tensor and ndarray descriptor checks * python: harden descriptor empty checks for tensor and ndarray inputs * python: harden descriptor parsing in python API * python: use canonical C++ mem_type names for tensors Addresses review comments on the XPU support change. Rename get_mem_type_from_device() to get_mem_type() and return canonical segment names (DRAM, VRAM) instead of device strings. Derive the memory type from tensor.get_device(), and rename tensor_type to ref_device in list‑of‑tensors validation. Retain cpu/cuda/xpu entries in nixl_mems as legacy mem_type aliases for backward compatibility with downstream callers, and add a TODO to remove these aliases once downstream migration to canonical segment names is complete. * python: keep tensor mem_type change only * python: keep tensor mem type check inline --------- Signed-off-by: Daniel Socek <daniel.socek@intel.com>
* ca cert updates * updated list of file paths to check Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> * updates Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> * updates Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> * updates Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> * updates Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> * updates Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> --------- Signed-off-by: Anjali Ratnam <anjaliratnam@microsoft.com> Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
* NIXL/EP/WHEEL: pack cu in diff namespace * NIXL/EP/WHEEL: fix load cu fallback
* refactor(obj): introduce engine registry for accelerated backends Replace the #ifdef ladder in obj_backend.cpp with a self-registration pattern. Each accelerated engine (S3Accel, Dell) registers itself via a static objAccelEngineRegistrar, and obj_backend.cpp dispatches through the registry. This makes it trivial to add new engines without modifying the factory function. Extract mock S3 client classes into separate headers for reuse, and introduce a mock client registry mirroring the production engine registry pattern. Signed-off-by: Olivier Garaud <olivier.garaud@scality.com> * style: apply clang-format brace style to production code Add braces around single-line if bodies to comply with the project's clang-format configuration. Signed-off-by: Olivier Garaud <olivier.garaud@scality.com> --------- Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
…#1635)
Allow callers to declare files by path in nixlBlobDesc::metaInfo:
metaInfo := <modes>:<path> # path-mode
| <anything else> # fd-in-devId mode
modes := <access>[,<flag>]*
access := "ro" | "rw"
flag := "direct" | "sync" | "noatime" | "create"
Backends open the file in registerMem and close it in deregisterMem.
Wired in all four file-aware plugins via a shared helper at
src/utils/file/file_path_mode.{h,cpp} (parsePathMeta + nixlFilePathMD
owned-fd RAII base):
- POSIX reference implementation
- HF3FS code-only (needs HF3FS client lib to build)
- CUDA_GDS ::open() wraps existing cuFileHandleRegister call
- GDS_MT owned-fd plumbed through gdsMtFileHandle's RAII
Strictly additive: unknown access/flag tokens fall through to fd-mode,
so existing fd-in-devId callers are unaffected.
Motivation: SGLang's HiCache issues 128 os.open() calls per flush from
Python; GIL contention makes each batch take ~415ms. Moving the opens
to C++ gives ~24x batch-open speedup in our mock and ~6-12x end-to-end
warm-phase TTFT speedup in hibench against Qwen-32B.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* DDN Infinia NIXL plugin
commit 849540b737a961217c5a6d43582ed440a3729885
Merge: 59147d8f 187e0514
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Apr 20 12:17:57 2026 -0400
Merge pull request ROCm#23 from 3rdParty/upstream_merge
Upstream merge
commit 187e05148fdc616e2b20a13bb8969f7a088c2554
Merge: 59147d8f 4d80978
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Apr 20 16:13:05 2026 +0000
Merge remote-tracking branch 'upstream/main' into upstream_merge
commit 59147d8f05a2fcdf1afb5b010d2065ef1a8ed8b5
Merge: d263402c 14baff3b
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Apr 20 10:31:22 2026 -0400
Merge pull request ROCm#22 from 3rdParty/RED-39451-1
Cleaned up documentation and trimmed README
commit 14baff3b61977e3649d62fc0114e00199c22a211
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Apr 20 13:55:58 2026 +0000
Cleaned up documentation and trimmed README
commit d263402c1a2dea1fbf467555c9e5d3be28ee470c
Merge: f755d40e 152d6d50
Author: Keyur Desai <kdesai@ddn.com>
Date: Fri Apr 17 16:59:17 2026 -0400
Merge pull request ROCm#21 from 3rdParty/RED-39451
Convert DDN license to Apache 2 license
commit 152d6d50ce21ea53284ab35b54ba5ba4e4bd7380
Merge: d2c8e1c5 f755d40e
Author: Keyur Desai <kdesai@ddn.com>
Date: Fri Apr 17 16:32:17 2026 -0400
Merge branch 'main' into RED-39451
commit f755d40e8a8ea8113ded75303df0de40f1ae750b
Author: Joseph Skazinski <jskazinski@ddn.com>
Date: Fri Apr 17 12:57:21 2026 -0700
Simplify registerMem and use BatchTask<> (ROCm#20)
* Simplify registerMem and use BatchTask<>
- Refactored registerMem to eliminate duplication and clarify logic
- Switched xfer requests to use new BatchTask<> and removed old vector of BatchRequest
- Added reserveOperations() to prevent vector reallocation issues
- Validated with up to 10K operations per batch
* fix(infinia): remove deprecated red_async_executor_t usage
Update plugin for red_async.hpp API changes:
- Remove red_async_executor_t (no longer exists in new API)
- Fix RED_ASYNC_DEFAULT_MAX_RETRIES constant name
- Correct memory type check in deregisterMem (VRAM_SEG/DRAM_SEG only)
* infinia: Implement queryMem with RED async API and enhance test output
- Implement queryMem() using red_async::BatchTask with HEAD operations
- Add timing instrumentation for all test phases
- Standardize data/throughput display to MiB/MiB/s
- Add test summary with keys, object size, and phase breakdown
- Update plugin documentation with async API patterns
* Add RED async config parameters to INFINIA backend
Support sthreads, num_buffers, num_ring_entries, and coremask
configuration from infinia.conf. Update backend, client, tests,
and documentation with new defaults and parsing logic.
* enhance INFINIA init message
* allow an empty string for coremask
commit d2c8e1c513ad3a9a84ebb88e7ba4bc48bedba40b
Author: Keyur Desai <kdesai@ddn.com>
Date: Wed Apr 15 20:05:14 2026 +0000
Convert DDN license to Apache 2 license
commit 0cdb085a1cc9f7c552eca862ca2ac6002a08de42
Author: Joseph Skazinski <jskazinski@ddn.com>
Date: Mon Apr 13 10:42:08 2026 -0700
RED-40574: Infinia plugin - remove mutex, enhance tracing, optimize (ROCm#19)
commit b6025814c063e99143d7b0fc6eeb2336e26be29d
Author: Joseph Skazinski <jskazinski@ddn.com>
Date: Fri Apr 10 07:24:31 2026 -0700
remove chunking from infinia backend to storage (ROCm#18)
* remove chunking from infinia backend to storage
* revert changes to nixl worker
* revert changes to nixl worker
* incorporate hpeng changes and additional cleanup
commit 9f288d5dc7ba6c6d7eef1e8047171b44c7fc02b5
Merge: 7b55446c 13da437d
Author: Hongbo Peng <hpeng@ddn.com>
Date: Wed Apr 8 09:58:01 2026 +0800
Merge pull request ROCm#17 from hpeng/RED-39914
move memory registration/deregistration to registerMem/deregisterMem
commit 13da437d6f89009691727c6ee4fa7e4b3215375b
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Apr 7 04:52:02 2026 +0000
move memory registration/deregistration to registerMem/deregisterMem instead of prepXfer
commit 7b55446cfbd723e4a2b212daf85b96d3d764fbe8
Author: Joseph Skazinski <jskazinski@ddn.com>
Date: Mon Apr 6 10:45:55 2026 -0700
fix(infinia): thread-safe devId map, OBJ_SEG-only storage, remove uns… (ROCm#16)
* fix(infinia): thread-safe devId map, OBJ_SEG-only storage, remove unsafe cast
* Use NIXL_DEBUG level based on review comments
commit 9e3d4d7cd0ae1d022daaf3e966b5e211c24e9607
Merge: 7254c9c8 ea4d0e70
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Mar 23 22:49:30 2026 -0400
Merge pull request ROCm#15 from skocol/sean/performance-optimizations
nixlInfiniaBackendReqH::postTransfer: avoid large copy when crossing a thread boundary
commit ea4d0e70613d777fb04180f8283bc47a019393f5
Author: Sean Kocol <skocol@ddn.com>
Date: Tue Mar 24 00:48:51 2026 +0000
Avoid spawning a thread for each xfer request
commit 7254c9c81491d5d3c2c5edb3fd0126a830d40c53
Merge: edd3ca6e 201ff61a
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Mar 23 15:17:52 2026 -0400
Merge pull request ROCm#14 from 3rdParty/RED-39416
Add config-file parse support to infinia NIXL plugin
commit 201ff61a0ddf176c476c161f6288411d310cba50
Author: Keyur Desai <kdesai@ddn.com>
Date: Sun Mar 22 03:54:51 2026 +0000
Add --config-file parse support to infinia NIXL plugin
commit edd3ca6e4f04e76a27cb6ab4d1db27b4fb61f376
Merge: dd555160 859b059e
Author: Keyur Desai <kdesai@ddn.com>
Date: Fri Mar 20 20:03:03 2026 -0400
Merge pull request ROCm#12 from 3rdParty/RED-39416
RED-39416: Fix broken nixlbench after async lib integration
commit 859b059e6576b04a47abaee0d0d142e6b23719ad
Author: Keyur Desai <kdesai@ddn.com>
Date: Fri Mar 20 13:48:21 2026 +0000
RED-39416: Fix broken nixlbench after async lib integration
commit dd555160a78f4812aeeef1ff64c2f4d03e46c7bf
Author: Joseph Skazinski <jskazinski@ddn.com>
Date: Thu Mar 19 21:09:27 2026 -0700
RED-38463: async lib improvements (ROCm#11)
* RED-38463: {async_lib} library performance improvements
* RED-38463: {async_lib} readme doc updates
* RED-38463: {async_lib} update nixl plugin hpeng with sugegsted changes
* refactor(infinia): use red_async_executor_t for batching
Replace manual batching with library's batch executor API.
Adds auto-tuning, progress callbacks, and 6 tuning parameters.
Fixes queue exhaustion errors, 6x performance improvement.
* feat(infinia): add GPU Direct Storage for RDMA GPU-to-storage transfers
Auto-detect VRAM, register with register_gpu_memory(), fallback to CPU staging if unavailable
* RED-38463: fix(infinia): add executor null checks for GPU memory cleanup
* RED-38463: fix(infinia): add red_async:: namespace qualifiers
* RED-38463: {async_lib} renamed red_async.hpp
* RED-38463: correct argument parsing for INFINA plugin
commit 80ab90136b8502999f9a9ff987fbb3e6e468935f
Merge: d94e662d a795db55
Author: Keyur Desai <kdesai@ddn.com>
Date: Thu Mar 5 18:06:07 2026 -0500
Merge pull request ROCm#9 from 3rdParty/RED-26691
Initial commit of NIXL plugin with async kv interface
commit a795db553f55852f0416b9afc7313605b176d424
Author: Keyur Desai <kdesai@ddn.com>
Date: Tue Feb 24 18:25:40 2026 +0000
Initial commit of NIXL plugin with async kv interface
commit d94e662d38c7dd1fd79f52129bc3af2042a1ffeb
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Feb 24 03:55:06 2026 +0000
disable infinia backend as no red_aisdk lib at now
commit b05ef984fb51feaba3cca27e4e587be7ff54a508
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Feb 24 03:54:20 2026 +0000
add dependency for nixl common to support abseil path
commit 4391f688e4e7a417a0aeb9d4d898749c4f6e3b1d
Merge: 3edd527a 77b9ab7
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Feb 24 03:13:58 2026 +0000
Merge remote-tracking branch 'upstream/main'
commit 3edd527a32557c8b312c6b780eed8c3666aa69fa
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Feb 24 03:05:51 2026 +0000
Revert "Merge pull request ROCm#6 from ziliu/feat/nixlbench-support-infinia-plugin"
This reverts commit b1c788c76aa44c25517d0074e6aa1c5888ee15f6, reversing
changes made to 53d00bc661e030c27bac84420276d71f10470108.
commit b1c788c76aa44c25517d0074e6aa1c5888ee15f6
Merge: 53d00bc6 62e9c3cd
Author: Keyur Desai <kdesai@ddn.com>
Date: Thu Jul 10 22:20:54 2025 -0400
Merge pull request ROCm#6 from ziliu/feat/nixlbench-support-infinia-plugin
nixlbench support infinia plugin
commit 62e9c3cd919e5882bb0b5bcf751c1b59054a8c41
Author: Zirui Liu <ziliu@ddn.com>
Date: Fri Jul 11 02:11:03 2025 +0000
add explaination to populate()
commit 4eeaef84231e4db5f0426764d95a2ccf875dc28f
Author: Zirui Liu <ziliu@ddn.com>
Date: Thu Jul 3 10:56:43 2025 +0000
fix populate() to avoid unnecesary offset check
commit f0b8e625d68cf59c7cc2bba73bf64746ef69218d
Author: Zirui Liu <ziliu@ddn.com>
Date: Tue Jul 1 14:29:35 2025 +0000
generate the same series of random keys
commit 22bb0e67803d2805732b52bf24712171118217b4
Author: Zirui Liu <ziliu@ddn.com>
Date: Tue Jul 1 04:07:21 2025 +0000
clean up comments
commit 66aa8d89d6762c72951d3c1a3d92acdd180ae135
Author: Zirui Liu <ziliu@ddn.com>
Date: Tue Jul 1 01:52:34 2025 +0000
additional changes to exchangeMetadata and exchangeIOV
commit 2559a2b784ea154d095ca9d91097c82a6375813e
Author: Zirui Liu <ziliu@ddn.com>
Date: Fri Jun 27 09:10:03 2025 +0000
initialize keys to support infinia plugin
commit 5b231682072a7e4e780056684f16b4b9248d9a8b
Author: Zirui Liu <ziliu@ddn.com>
Date: Thu Jun 26 17:15:47 2025 +0000
add micros for infinia
commit 8c9fae653738704e5155a3ccd1812f99cc20185f
Author: Zirui Liu <ziliu@ddn.com>
Date: Thu Jun 26 17:14:56 2025 +0000
add empty functions to include infinia backend. todo: initialize keys and values
commit 53d00bc661e030c27bac84420276d71f10470108
Merge: 22b88867 e704cf35
Author: Hongbo Peng <hpeng@ddn.com>
Date: Wed Jun 25 09:57:12 2025 +0800
Merge pull request ROCm#5 from hpeng/RED-27683
eliminate memcpy in the infinia Nixl plugin and aisdk code path
commit e704cf350c410a4f599a3f232d9ff64e9ab48b81
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 24 16:00:32 2025 +0000
rename red_aisdk_client_get/put for iomem
commit ffdb0decc06ea633a725d49e5b5440ed8a5cb56e
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 24 15:04:03 2025 +0000
fix typo for GET
commit f9ad22f966fd9fb1a374b9b3a418ad8a8b498bc4
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 24 07:25:15 2025 +0000
eliminate memcpy in the infinia Nixl plugin and aisdk code path
free allocated mem in error case
commit 22b8886757d6bdf6580a9a434d06537b926e0786
Merge: ce7ff3a4 d3dc1953
Author: Hongbo Peng <hpeng@ddn.com>
Date: Fri Jun 20 08:52:32 2025 +0800
Merge pull request ROCm#4 from hpeng/RED-27294-2
remove unsupported opts and replace GDS with Infinia
commit d3dc19536d8039d3a177c9ac30076366669e4fe6
Author: Hongbo PENG <hpeng@ddn.com>
Date: Thu Jun 19 04:33:33 2025 +0000
remove unsupported opts and replace GDS with Infinia
commit ce7ff3a40c3a9efde3b24cedd39291cd870003b4
Merge: 3d41192a ab048cd8
Author: Hongbo Peng <hpeng@ddn.com>
Date: Wed Jun 18 16:22:00 2025 +0800
Merge pull request ROCm#3 from hpeng/RED-27294
RED-27294 Infinia support in nixl plugin with enhanced unit test
commit ab048cd8abdc4d14c4aaf43641be91871cf36f44
Merge: 0114d347 249833d2
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 16:18:15 2025 +0800
Merge branch 'main' into RED-27294 to resolve conflicts
commit 249833d20a1b6a7ce65fe63d3847be638da082b3
Merge: 3d41192a 0a60245a
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 15:29:13 2025 +0800
Merge branch 'hpeng-RED-27294'
commit 0a60245a2d8176912108c7e06de1d9fc3e575447
Merge: 3d41192a 0114d347
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 15:25:57 2025 +0800
Merge branch 'RED-27294' of github.red.datadirectnet.com:hpeng/nixl into hpeng-RED-27294
commit 0114d347ab6f1ba029235c74a2e6d1fa846a2325
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 06:01:50 2025 +0000
replace printf with std::cerr
commit 7fa6691a9f8b4201825eaa7e4e9f41df6657d5ad
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 05:57:50 2025 +0000
remove tail whitespace and TAB
commit 7c9185ca1b174f09f30ece8b6ca8db3f531ada4b
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 05:46:41 2025 +0000
red_kv_put does NOT support append. Set maximum transfer size to 10M.
commit 59bab12284d3fd353cb59247a4824c1cd1da7ee9
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 05:12:05 2025 +0000
set cpp_args correctly
commit 358663db870b07bf1e5cad1f54d717f23dd7f39d
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 04:52:46 2025 +0000
replace TAB with whitespace
set separate seed for key generator
commit bdd7e29610404e77f8fff956619ec1d49833a1aa
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 04:18:42 2025 +0000
fix typo
commit 884c929bbedc232463aa7e74907df413d44657e2
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 04:10:25 2025 +0000
Update return code from red_aisdk
remove Async code branch
commit 01f76922034aafdf4121903eee353b90a0c1120c
Author: Hongbo PENG <hpeng@ddn.com>
Date: Wed Jun 18 02:48:05 2025 +0000
remove unsupported options
commit 61782a9352cbb541b32d49780dfd59bef25a63dc
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 05:41:29 2025 +0000
generate reproducible uuid for test and remove uuid dep
commit 9007afeb68cb9182f012d40ee1fe3a0c3db5ae90
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 05:05:51 2025 +0000
no support to skip read/write
commit 5cb8bed42841fee64c1459cf23c726c8cc6d8975
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 03:11:50 2025 +0000
rename macro USE_VRAM as HAVE_CUDA
commit 6c928dde7951189989d7ba96a9a49d06e17fe328
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 02:33:28 2025 +0000
add macro USE_VRAM to run on DRAM only
commit 32c8798de6c0588e6b78ddebd7598eae401f8614
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 02:21:37 2025 +0000
Code cleanup for commit
commit 8714093925b7dcab39d159d9883b73b0ab387e4a
Author: Hongbo PENG <hpeng@ddn.com>
Date: Tue Jun 17 02:20:26 2025 +0000
Move check from postXfer to preXfer
Code cleanup for commit
commit 747b552688afdbfd40a78f80546378f96f149bc4
Author: Hongbo PENG <hpeng@ddn.com>
Date: Fri Jun 13 02:06:44 2025 +0000
Add more args for the Infinia backend unit test to support
different size of keys
different numbers of reqs
multiple iters
commit be8778de311bed41d9a0039393d65e2026a119fd
Author: Hongbo PENG <hpeng@ddn.com>
Date: Fri Jun 13 02:05:50 2025 +0000
Remove the memcpy for DRAM.
Replace gds with infinia
commit 3d41192a7b5bc6d63f977f521d1222ea8c1e6aa9
Merge: a5206749 dbc32b75
Author: Keyur Desai <kdesai@ddn.com>
Date: Mon Jun 9 22:42:41 2025 -0400
Merge pull request ROCm#2 from hpeng/RED-25299-2
code update to sync with upstream
commit dbc32b75572a556e1247d12c5a2728b680bda006
Author: Hongbo PENG <hpeng@ddn.com>
Date: Mon Jun 9 07:38:32 2025 +0000
1. code update for upstream changes: ai-dynamo/nixl#295
2. fix typos
commit a520674908464ce3235bb03da90de9c470c631bf
Merge: e0b34a3 a3dcf962
Author: Keyur Desai <kdesai@ddn.com>
Date: Tue May 27 22:00:17 2025 -0400
Merge pull request ROCm#1 from kdesai/RED-25299
Initial commit of NIXL Infinia backend plugin
commit a3dcf9624bcaa876adec21408c4827aa45a3fcca
Author: Keyur Desai <kdesai@bb-4.vms.virts.svc.devintel2.local>
Date: Fri May 23 02:36:55 2025 +0000
Initial commit of NIXL Infinia backend plugin
* SQUASHME: Addressed code review comments 1.
* SQUASHME: clang and precommit hook fix
* SQUASHME: GPU direct fix
* SQUASHME: Addressed 2nd round of coderabbit comments
* QUASHME: Addressed comments from Adit
* SQUASHME: clang fix
* SQUASHME: Addressed Adit's comment on remote support
* SQUASHME: Addressed comments from Colin - part 2
* SQUASHME: clang fixees
* SQUASHME: Removed unnecessary StrFormat
* SQUASHME: Removed C++ 20 override from Infinia meson.build
* SQUASHME: rem oved format, as reuested by Adit
---------
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
… (#1733) * build-wheel.sh: exclude DDN partner libs from auditwheel bundling Adds libred_client*, libred_async*, liblz4* to the auditwheel --exclude list. The INFINIA backend plugin links against DDN's Infinia client runtime, which the operator installs separately; the wheel must not bundle these libraries. Signed-off-by: Adit Ranadive <aranadive@nvidia.com> * build-wheel.sh: quote $TMP_DIR and $WHL_PLATFORM Avoid word-splitting/globbing breakage in the auditwheel call when paths contain whitespace. Globs preserved by quoting only the directory portion: "$TMP_DIR"/nixl*.whl. Signed-off-by: Adit Ranadive <aranadive@nvidia.com> * Add quotes to remaining variables Signed-off-by: Adit Ranadive <aranadive@nvidia.com> --------- Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
…647) * nixlbench: add AMD ROCm/HIP build support Adds ROCm/HIP build support for nixlbench, gated behind a default-off `use_rocm=true` Meson option. Validated on MI300X (gfx942) and MI355X (gfx950). Design follows @tvegas1's gist (https://gist.github.com/tvegas1/4b33b748ba30c4817e9a5a6e593bebab): - benchmark/nixlbench/meson_options.txt: add `use_rocm` (boolean) + `rocm_path` (string; empty defaults to /opt/rocm). - benchmark/nixlbench/meson.build: declare a separate `rocm_dep` (link `-lamdhip64 -lhiprtc`, include rocm/include), add `-D__HIP_PLATFORM_AMD__` as a nixlbench-scoped project arg, set `HAVE_ROCM` in `config.h`, and add `rocm_dep` to the top-level deps list when `use_rocm=true`. CUDA detection skips when `use_rocm=true` (mutually exclusive); a no-op `cuda_dep = declare_dependency()` is declared so unconditional cuda_dep references don't error. - src/utils/meson.build, src/worker/meson.build, src/worker/nixl/meson.build: pick `cuda_dep` OR `rocm_dep` per target via `if cuda_available / elif rocm_available`. Targets that unconditionally listed `cuda_dep` previously now skip GPU deps when neither backend is available (existing CUDA-less builds still worked because cuda_dep was a no-op). - src/utils/utils.{h,cpp}, src/worker/nixl/nixl_worker.cpp: add `#if HAVE_ROCM` source-side guards. Includes `hip/hip_runtime.h`, maps `CHECK_CUDA_ERROR` onto `hipSuccess` / `hipGetErrorString`, routes VRAM alloc / memset / memcpy / free through `hipMalloc`, `hipMemset`, `hipMemcpy`, `hipFree`, and disables VMM on ROCm (ROCm has no equivalent to CUDA's `cuMem*` fabric API today). Stacks on #1642 (which adds the `wheel_variant` Meson option used by ROCm wheel naming). A follow-up PR will add the ROCm Dockerfile (rocm/dev-ubuntu-24.04 base + UCX --with-rocm), the ROCm CI compile job @edgargabriel asked about, and any contrib/*.sh modifications needed for wheel generation, per @tvegas1's request to keep that surface in one place. Signed-off-by: Andy Luo <anluo@amd.com> Signed-off-by: andyluo7 <andy.luo@amd.com> * nixlbench: fix two ROCm preprocessor bugs flagged by CodeRabbit 1. resolveVramSegment() was missing a HAVE_ROCM branch, causing ROCm-only builds to fall through to the Neuron check and exit with "VRAM not supported without CUDA or Neuron". Add `#elif HAVE_ROCM return VRAM_SEG` to mirror the pattern used in getVramDesc() and cleanupBasicDescVram(). 2. utils.cpp VMM guard used `defined(HAVE_ROCM)` / `defined(HAVE_CUDA)` but meson generates config.h with these as 0/1 values, not define/undef. Switch to `#if HAVE_ROCM` / `#elif HAVE_CUDA && !HAVE_CUDA_FABRIC` so the guard evaluates correctly when the macros are 0. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> --------- Signed-off-by: Andy Luo <anluo@amd.com> Signed-off-by: andyluo7 <andy.luo@amd.com> Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
* Fix POSIX backend queue fallback handling Guard POSIX request posting against unavailable I/O queues and let tests use the backend's compiled default queue instead of forcing Linux AIO. Co-authored-by: OpenAI Codex <codex@openai.com> * Address POSIX backend review feedback Split POSIX queue initialization errors into separate logs and clarify request-handle deletion by using the POSIX cast only for validation. Co-authored-by: OpenAI Codex <codex@openai.com> * Fail POSIX test early for unsupported queues Teach the POSIX plugin test which queue implementations were compiled in and reject POSIX AIO-only or unavailable io_uring configurations at startup with a clear error message. Co-authored-by: OpenAI Codex <codex@openai.com> * Update test-dl-matrix.yaml Modified to reflect different slurm server for CI Signed-off-by: Ofer Achler <ofer.achler@gmail.com> * Update test-dl-matrix.yaml Reverted the head node back to the general cluster Signed-off-by: Ofer Achler <ofer.achler@gmail.com> * Address POSIX handle hot-path review comments Avoid RTTI in POSIX request-handle paths and mark the defensive missing I/O queue check as unlikely. Co-authored-by: OpenAI Codex <codex@openai.com> * fix: address POSIX review comments * Update test-dl-matrix.yaml Added environment variable to make CI pass Signed-off-by: Ofer Achler <ofer.achler@gmail.com> --------- Signed-off-by: Ofer Achler <ofer.achler@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Adit Ranadive <aranadive@nvidia.com> Co-authored-by: Ofer Achler <ofer.achler@gmail.com>
Use resp.getType() so prepMemView works for non-VRAM remote descriptors. Add a regression test and a reusable dual-agent gmock setup helper for future tests. Signed-off-by: Shahaf Meir Kiselnik <smeir@nvidia.com> Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
* Parallelize memory query for azure blob Similar to OBJ plugin, parallelize HEAD requests when batch memory queries are provided. This specifically improves initial query times for integrations that checks for a blob existence (e.g., checking for KV cache in LMCache). For the most part the implementation, matches the OBJ plugin to make it easy to update both plugins if there are adjustments made to the approach in the future. Signed-off-by: Kyle Knapp <kyleknapp@microsoft.com> * Clean up memory query wait logic Specific changes were: * Make sure we wait for all instances where the callback started within the timeout block and not just within the error block when queueing work. * Consolidated logic to single helper function to make it easier to follow and maintain in the future. Signed-off-by: Kyle Knapp <kyleknapp@microsoft.com> * Add more robust error handling in blob client Specifically catch any non-standard exceptions to safeguard against crashes in the asio thread pool. This is primarily defensive as prior to it, non-standard exceptions had yet to be encountered. It also adds logging for exceptions in async put and get to have Azure exceptions visible in NIXL error logs. Alternative was to turn on AZURE_LOG_LEVEL=verbose to know what the exceptions were * Add assertions for null clients in tests Prevents not constructing a client and trying to use a null client when building a test blob engine for testing. Signed-off-by: Kyle Knapp <kyleknapp@microsoft.com> --------- Signed-off-by: Kyle Knapp <kyleknapp@microsoft.com> Co-authored-by: Adit Ranadive <aranadive@nvidia.com> Co-authored-by: ovidiusm <ovidium@nvidia.com>
* Support for prepared transfer API in nixlbench
…#1728) * build: bump DOCA to 3.3 and build the telemetry exporter in CI The DOCA telemetry exporter Metrics API requires DOCA >= 3.3, and the exporter plugin was never compiled in CI: no build image installed the telemetry-exporter SDK, and meson only probed for it via a hardcoded header path. This bumps DOCA and makes the exporter a first-class, CI-built plugin on par with the GPUNETIO backend. - Bump DOCA 3.2 -> 3.3 in every install site (.gitlab/build.sh, contrib and nixlbench Dockerfiles, manylinux rpm) and CI_IMAGE_TAG in the three Jenkins matrices. - Install libdoca-sdk-telemetry-exporter-dev (+ CollectX) in the deb images and doca-sdk-telemetry-exporter in the manylinux image so the plugin builds and links in CI. - Detect the exporter via pkg-config (doca-telemetry-exporter + doca-common) instead of a hardcoded /opt/mellanox/doca path, mirroring the GPUNETIO backend. - Add TELEMETRY_DOCA to all_plugins and gate it with the same enable_plugins / disable_plugins / is_explicit_enable pattern as GPUNETIO (auto-detected by default, error on explicit enable without the SDK, skippable via -Ddisable_plugins=TELEMETRY_DOCA). Signed-off-by: Efraim Eygin <eeygin@nvidia.com> * docs: install DOCA telemetry exporter deps in nixlbench setup The manual DOCA setup snippet installed only the GPUNETIO packages. Now that the DOCA telemetry exporter is built when its SDK is present, add libdoca-sdk-telemetry-exporter-dev (+ collectx-clxapidev) to match the container/CI install path. Addresses a review comment on #1728. Signed-off-by: Efraim Eygin <eeygin@nvidia.com> * fix(ci): install CollectX in manylinux wheel build for DOCA telemetry exporter The DOCA telemetry exporter pkg-config file (doca-telemetry-exporter.pc) lists `Requires.private: libclx-api, doca-common`. libclx-api is shipped by CollectX, which the manylinux RPM path (rpm -ivh --nodeps) did not install, so `pkg-config --cflags --libs doca-telemetry-exporter` failed the build. Install the CollectX RPMs from the DOCA host repo. They use a different naming scheme (collectx_<ver>-...-clxapi[dev].rpm), so a collectx-clxapidev-* glob would match nothing; use collectx_*rpm instead. The deb-based images are unaffected: libdoca-sdk-telemetry-exporter-dev already Depends on collectx-clxapidev. Signed-off-by: Efraim Eygin <eeygin@nvidia.com> --------- Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* test: add standalone DOCA telemetry exporter test Add a self-contained gtest that exercises the DOCA Telemetry Exporter Metrics API end to end: set up schema/source/metrics, accumulate a counter via doca_telemetry_exporter_metrics_add_counter_increment, flush with doca_telemetry_exporter_metrics_flush, and scrape the CollectX-backed Prometheus endpoint to verify the cumulative value. It builds as its own executable under test/doca-telemetry, not linked into the main gtest binary, so the DOCA/CollectX gRPC stack stays out of every other test. It is only built when the DOCA telemetry headers are present, mirroring the plugin gate in src/plugins/telemetry/doca/meson.build. Signed-off-by: Efraim Eygin <eeygin@nvidia.com> * telemetry/doca: accumulate counters and add exporter flush Counter events carry a per-operation delta, but the DOCA exporter pushed each delta through doca_telemetry_exporter_metrics_add_counter as an absolute value, yielding a non-monotonic series that disagreed with the Prometheus exporter. Switch counter events to doca_telemetry_exporter_metrics_add_counter_increment so repeated deltas form a monotonic cumulative total. Rename registerCounter/registerGauge to appendCounterSample/appendGaugeSample to reflect the time-series append semantics. Expose flush() (doca_telemetry_exporter_metrics_flush) on the exporter. Production relies on CollectX interval auto-flush, but a handful of samples will not fill the buffer to trigger one, so tests need an explicit flush before scraping. Add doca_nixl_test, which drives the real exporter end to end (exportEvent x3 -> flush -> scrape) and verifies the counter accumulates to the cumulative total. It compiles doca_exporter.cpp in directly and links only nixl_infra/nixl_common (no etcd/gRPC), so it stays free of the DOCA/CollectX gRPC clash like the raw test. Extract the shared HTTP scrape helpers into scrape_util.h, used by both DOCA tests. Verified: - ninja -C build src/plugins/telemetry/doca/libtelemetry_exporter_doca.so - meson test -C build doca_test doca_nixl_test (2/2 OK) Signed-off-by: Efraim Eygin <eeygin@nvidia.com> * test: unset DOCA env vars in TearDown The doca_nixl_test fixture set NIXL_TELEMETRY_DOCA_PROMETHEUS_LOCAL and NIXL_TELEMETRY_DOCA_PROMETHEUS_PORT in SetUp but never cleared them. Add a paired TearDown that unsets both so the fixture does not leak process-wide environment state. Signed-off-by: Efraim Eygin <eeygin@nvidia.com> * telemetry/doca: mark flush() [[nodiscard]] flush() returns a nixl_status_t that callers must check, matching the existing [[nodiscard]] annotations on exportEvent, appendCounterSample, and appendGaugeSample. Mark it [[nodiscard]] so its result cannot be silently ignored. Signed-off-by: Efraim Eygin <eeygin@nvidia.com> --------- Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
--------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
aviallon
requested review from
akolliasAMD,
edgargabriel and
gaoikawa
as code owners
June 9, 2026 13:48
Author
|
In my disaggregated prefill / decode setup, this reduced latency between prefill and decode by almost 100x: it went down from 1800ms to barely 20ms. |
Collaborator
@aviallon thank you for your PR. We are in the process of deprecating the RIXL repository, since support for AMD GPUs has been merged directly in NIXL ai-dynamo/nixl#1642 Would you like to refile your PR there? |
Author
|
Ah, I wasn't aware. “whyyyy does it not use rocm_ipc” kind of headaches. |
…st. (#1742) * UTILS/CONFIG: Make config file reading unit-testable and re-enable test. * Simplify and keep ABI. * Fix typo. * Fix order. --------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
… provider does not provide PCIe bus details. (#1587) Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Zhenlong Ma <zhenlongm@nvidia.com> Co-authored-by: Guy Ealey Morag <gealeymorag@nvidia.com>
* telemetry: disable collection when no exporter is created
Previously, NIXL_TELEMETRY_ENABLE=y (or captureTelemetry config) set an
internal telemetryEnabled flag even when initializeTelemetry() created no
exporter, for example when neither NIXL_TELEMETRY_DIR nor an explicit
NIXL_TELEMETRY_EXPORTER was configured. The datapath then kept recording
transfer stats and events that could never be exported.
Add nixlTelemetry::create(), which returns nullptr when no exporter/sink
is configured, and make data->telemetry_ != nullptr the single source of
truth for whether telemetry is active. The redundant telemetryEnabled
flag is removed.
Behavior change: requesting telemetry without a usable sink no longer
half-enables telemetry; getXferTelemetry() returns NIXL_ERR_NO_TELEMETRY.
Tests (gtest, Python) and docs/telemetry.md are updated to reflect this,
and Rust binding telemetry tests now configure a buffer directory for the
success cases.
Verified:
- ninja -C build install
- build/test/gtest/gtest --gtest_filter='*Telemetry*' (22/22)
- pytest test/python/test_nixl_api.py -k xfer_telemetry (3 passed)
- cargo test -p nixl-sys --no-run
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: address review comments on no-exporter disable
- Clarify the no-sink debug log to say telemetry is disabled: the
no-NIXL_TELEMETRY_DIR / no-exporter path now yields an inactive telemetry
object (create() returns nullptr), so "enabled without any exporter" was
misleading.
- Document the nixlTelemetry::create() null-return-vs-throw contract.
- Rust tests: extract telemetry_cleanup() to remove duplicated teardown and
surface remove_dir_all errors.
- Python: annotate _run_xfer_telemetry_check return type.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: serialize and restore env in Rust telemetry tests
Address review feedback that the telemetry tests mutate the process-global
NIXL_TELEMETRY_* variables without serialization or panic-safe cleanup, so a
failing assertion (or Rust's parallel test runner) could leak telemetry env
state into other tests.
- Add a generic RAII EnvGuard (tests/env_guard/mod.rs) that locks a shared
mutex to serialize env-mutating tests and snapshots/restores the requested
variables on drop.
- Use tempfile::TempDir for the buffer directory so it is removed on drop,
including on panic.
- Convert the four env-mutating get_xfer_telemetry tests to the guard.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: configure sink in before-posting Rust test
The test_get_xfer_telemetry_before_posting test enabled NIXL_TELEMETRY_ENABLE
but no sink, so after this PR telemetry is inactive and get_telemetry failed
with NIXL_ERR_NO_TELEMETRY ("telemetry off") instead of exercising the intended
"transfer not posted" path. Configure a temporary NIXL_TELEMETRY_DIR so
telemetry is active, and assert BackendError, which is how the binding surfaces
the NIXL_ERR_NOT_POSTED status returned for an unposted request.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: drop redundant OnceLock around env test lock
Mutex::new is const, so the EnvGuard serialization lock can be a
const-initialized static instead of a OnceLock-wrapped lazy init. Removes the
env_lock() helper and the OnceLock import; the mutex itself is unchanged.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: make EnvGuard restore keys mutated via set/remove
EnvGuard::set/remove previously mutated arbitrary keys, but Drop only restored
the variables captured by new(vars); a key touched only via set/remove would
leak past the guard. Snapshot a key's original value on first mutation (using
interior mutability so the &self API and callers are unchanged), so Drop fully
restores anything the guard touches.
Add unit tests covering the restore-on-drop contract: an un-snapshotted key set
via set() is removed again on drop, a pre-existing value is restored after set(),
and remove() is undone on drop.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* telemetry: throw when exporter creation fails
initializeTelemetry() threw on a failed plugin load but only logged and
returned (silently disabling telemetry) when createExporter() returned
nullptr. Both are genuine setup failures for an explicitly requested
exporter, and create()'s contract is to throw on such errors. Throw a
runtime_error to surface the failure consistently with the plugin-load path.
Add a gtest covering the failure path: an explicitly requested exporter that
cannot be loaded now makes initialization throw.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
* test/python: configure telemetry sink in test_agent
Since PR #1716, telemetry is only active when a sink/exporter is configured
(nixlTelemetry::create() returns nullptr otherwise). test_agent set
NIXL_TELEMETRY_ENABLE=y but no sink, so telemetry stayed inactive and
getXferTelemetry() returned NIXL_ERR_NO_TELEMETRY.
Point NIXL_TELEMETRY_DIR at a temporary directory so the buffer exporter is
created and per-transfer telemetry is captured, matching the C++ telemetry
tests. Restore the no-sink default and remove the temp dir at the end so
telemetry stays inactive for subsequent tests.
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
---------
Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Use Artifactory instead of Harbor for Docker image operations Signed-off-by: Noam Tsemah <ntsemah@nvidia.com>
…p (#1689)
* refactor(nixlbench): use RAII for memory deregistration and fd cleanup
Replace explicit deregister/cleanup loops in deallocateMemory() with
RAII wrappers that automatically handle resource teardown, improving
exception safety and reducing boilerplate.
- xferFileState now owns its fd (close-on-destroy, move-only)
- NixlMemRegion wraps registered memory regions and calls
deregisterMem + per-IOV cleanup in its destructor
- deallocateMemory() reduced to clearing RAII containers in
the correct ordering (remote regions, then fds, then local)
- createFileFds error paths simplified (RAII closes open fds)
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
* refactor(nixlbench): tidy up xferFileState and NixlMemRegion
- Mark the xferFileState value constructor noexcept.
- Extract the duplicated fd-close logic into a private closeFd() helper
used by the destructor and move-assignment.
- Make NixlMemRegion::release() private; it is only invoked by the
destructor and move-assignment.
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
* refactor(nixlbench): move RAII types into nixl_mem_region.{h,cpp}
Extract the xferFileState and NixlMemRegion RAII helpers (along with the
shared CHECK_NIXL_ERROR macro and iovListToNixlRegDlist helper they
depend on) out of nixl_worker.{h,cpp} into their own translation unit.
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
* refactor(nixlbench): tidy NixlMemRegion and iovListToNixlRegDlist
- iovListToNixlRegDlist returns nixl_reg_dlist_t instead of filling an
out-param, and sizes the list up front via the init_size constructor
to avoid re-allocations while filling it.
- NixlMemRegion::agent_ is now a reference rather than a nullable
pointer; the released/moved-from state is tracked via empty iovs_.
Drops the unused default ctor and move-assignment (a reference member
can't be rebound; vector growth uses the noexcept move ctor).
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
* refactor(nixlbench): derive NixlMemRegion cleanup from seg_type
Drop the std::function cleanup injected into NixlMemRegion; the cleanup
is fully determined by seg_type, which the region already holds. Add a
free cleanupIov() dispatcher (declared in nixl_mem_region.h, defined in
nixl_worker.cpp) and call it from release().
cleanupBasicDescDram/Vram/Obj become file-local free functions. Remove
the now-dead cleanupBasicDescFile/Blk: FILE fds are owned by
xferFileState and BLK descriptors own nothing.
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
---------
Signed-off-by: Olivier Garaud <olivier.garaud@scality.com>
…env var The problem is that the UCX backend hardcodes UCP_ERR_HANDLING_MODE_PEER as the default error-handling mode. This demands every transport support peer failure detection. Neither shared-memory transports (sysv, posix, cma) nor ROCm transports (rocm_ipc, rocm_copy) satisfy this requirement, so UCP falls back to the only surviving transport: TCP loopback. This change gates the error-handling mode via a new environment variable NIXL_UCX_ERR_HANDLING_MODE. When set to none, it relaxes the peer-failure requirement and unlocks the full transport set (sysv/posix for RMA, rocm_ipc for control/data). No functional change when the env var is unset. Ref: ROCm#32
aviallon
force-pushed
the
fix/ucx-err-handling-gate
branch
from
June 16, 2026 12:43
f1ae8c2 to
d0efcf0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
The problem is that the UCX backend hardcodes
UCP_ERR_HANDLING_MODE_PEERas the default error-handling mode(
ucx_utils.cpp:39). This demands every transport support peerfailure detection. Neither shared-memory transports (sysv, posix,
cma) nor ROCm transports (rocm_ipc, rocm_copy) satisfy this
requirement, so UCP falls back to the only surviving transport: TCP
loopback.
This change gates the error-handling mode via a new environment
variable
NIXL_UCX_ERR_HANDLING_MODE. When set tonone, itrelaxes the peer-failure requirement and unlocks the full
transport set (sysv/posix for RMA, rocm_ipc for control/data).
No functional change when the env var is unset.
Why?
For same-host disaggregated serving on PCIe-only AMD GPUs, the NIXL
UCX backend is forced into TCP loopback for all inter-process transfers.
Throughput is limited to ~150 MB/s instead of GB/s achievable with
shared memory or GPU IPC transports.
How?
The C++ backend at
ucx_backend.cpp:821already readsucx_error_handling_modefrom the custom init-params dict,defaulting to
peerwhen absent. This Python-side change merelypipes the
NIXL_UCX_ERR_HANDLING_MODEenvironment variable intothat dict during
nixl_agent.__init__, before callingcreate_backend("UCX", init).One line added in
src/api/python/_api.py, gated on whether theenv var is set at all.