Skip to content

nixl re-sync 4-23-2026#29

Merged
edgargabriel merged 60 commits into
ROCm:developfrom
edgargabriel:topic/nixl-sync-4-23-2026
Apr 28, 2026
Merged

nixl re-sync 4-23-2026#29
edgargabriel merged 60 commits into
ROCm:developfrom
edgargabriel:topic/nixl-sync-4-23-2026

Conversation

@edgargabriel

Copy link
Copy Markdown
Collaborator

What?

Periodic resync with upstream repo

ovidiusm and others added 30 commits March 30, 2026 17:19
* NIXL EP: Add torch version check

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Pin torch version to 2.11 during wheel build

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Pin torch exactly to 2.11.*

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add linter ignore

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add copyright

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* NIXL EP: Add torch version check

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Pin torch version to 2.11 during wheel build

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Pin torch exactly to 2.11.*

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add linter ignore

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add copyright

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
`FI_THREAD_COMPLETION` semantics require locked access to all objects
bound to a completion queue, particularly endpoints. Expand the CQ mutex
to cover posting operations to the endpoint as well.

* In case we need to progress the CQ in retry while posting an
  operation, only do so if the progress thread is not enabled.
  Otherwise, let the progress thread handle CQ polling.

Signed-off-by: Eric Raut <eraut@amazon.com>
* use ucx_ref and ignore torch ver in module

* Remove file

* fix

* fix
…BJ plugin. (#1441)

* Add Dell ObjectScale accelerated engine support to plugin_gtest for OBJ plugin.
Introduce NIXL_OBJ_ENDPOINT_OVERRIDE environment variable for setting the endpoint of
the Dell ObjectScale cluster.  Add support for VRAM_SEG tests.

Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>

* Extract CUOBJ tests to their own plugin test file.  Make endpoint override a universaly applied environment variable to allow all tests to run against a specific endpoint

* Addressed code review comments

* Address code review comment regarding guard naming in memory_handler.h

Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>

* Address code review comment

Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>

* Addressed Code review comments

Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>

---------

Signed-off-by: Jason Goldschmidt <jason.goldschmidt@dell.com>
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
Invoke artifactory-cleanup-by-time.sh in pipeline_stop (dry-run) to list
Docker images that would be removed by age. Use REGISTRY_REPO_NAME,
REGISTRY_REPO_PATH, ARTIFACTORY_CLEANUP_AGE (default 3m) and existing
Artifactory credentials.

Also: split REGISTRY_REPO into REGISTRY_REPO_NAME and REGISTRY_REPO_PATH,
set ARTIFACTORY_REGISTRY_URL for push/echo, add containerSelector for
podman on relevant steps, and simplify taskName (drop axis_index).

Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* NIXL/EP: Use vmm API instead of cudaMalloc

* NIXL/EP: Use vmm API instead of cudaMalloc

* NIXL/EP: revert

* NIXL/EP: Support gdr copy with vmm.

* NIXL/EP: Improve.

* NIXL/EP: Format.

* NIXL/EP: check return val.

* NIXL/EP: Format.

* NIXL/EP:Improve.

* NIXL/EP: Format.

* NIXL/EP: Improve.

* NIXL/EP: Format.

* NIXL/EP: fallback to cudaMalloc if fabric not supported

* NIXL/EP: set default vals

* NIXL/EP: Fix.

* NIXL/EP: Fix comments.

* NIXL/EP: Fix.

* NIXL/EP: Fix.

* NIXL/EP: not needed.

* NIXL/EP: new files terms.

* NIXL/EP: Fix comments.

* NIXL/EP: Fix comment.

* NIXL/EP: Fix.

* NIXL/EP: fix.

* NIXL/EP: Fix comment.

* NIXL/EP: Fix comment.

* NIXL/EP: Fix comment.

* NIXL/EP: Fix comment.

* NIXL/EP: Fix comment.

---------

Co-authored-by: Ofir Farjon <ofarjon@funk11.mec01.labs.mlnx>
Move the p2p_ptr_get() call inside the is_rank_masked
guard in three locations in nixl_ep.cu (two in dispatch,
one in combine).

In elastic scenarios, ranks can be set up in non-consecutive
order — for example, ranks 0 and 2 may start before rank 1
has been initialized. During this early phase, rank 1's P2P
memory mapping is not yet established. Previously,
p2p_ptr_get() was called unconditionally for all ranks before
the is_rank_masked check, so it would attempt to access the
uninitialized P2P mapping for rank 1 and fail, even though
rank 1 is correctly masked and should be skipped.

In each of the three locations, the p2p_ptr_get(dst_ptr, dst_rank)
call is moved to inside the rank mask guard, ensuring the P2P
pointer lookup only occurs for ranks that are actually participating
and initialized.
* nixl_ep: Add high-throughput kernels

Port the DeepEP high-throughput dispatch and combine kernels to use
the NIXL device API V2 (memory views, nixlPut, nixlAtomicAdd).

Rename nixl_ep.cu -> nixl_ep_ll.cu

Co-authored-by: Roey Azran <roeya@nvidia.com>
Co-authored-by: Micha Dery <mdery@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>

* fixups pre-commit

* fix pre-commit 2

* fix pre-commit3

---------

Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Co-authored-by: Roey Azran <roeya@nvidia.com>
Co-authored-by: Micha Dery <mdery@nvidia.com>
* chore: update attributions for the latest release

* fix missing license

* update attr

* fix dup

* more changes

* add missing license

* add new pckage
PR #1258 changed two independent if-statements into if/else in
registerMem(), breaking the fallthrough that calls cudaSetDevice()
when the CUDA address workaround is disabled for device N>0, in
multi-GPU per process case.

Without cudaSetDevice(), cuMemGetAddressRange() fails inside EFA's
dmabuf path for large buffers, causing fi_mr_key to return
FI_KEY_NOTAVAIL. Blackwell (B200) requires the correct CUDA context
for cuMemGetAddressRange; Hopper (H200) continues to work based on tests.

Restore the original two-if pattern and downgrade the expected
multi-GPU detection log from WARN to INFO.

Validated with nixlbench --scheme tp --mode MG --num-initiator-dev 4
--num-target-dev 8 --check-consistency on p6 B200 instances.

Signed-off-by: Ye Xiang <yexiang@amazon.com>
Signed-off-by: Raul Akhmetshin <rakhmetshin@nvidia.com>
…resolution (#1402)

* BINDINGS/PYTHON/NIXL_META/NIXL: fix static analysis issues for nixl_agent and nixl_logger

---------

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Roie Danino <rdanino@nvidia.com>
Co-authored-by: Roie Danino <rdanino@nvidia.com>
Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
* Remove redundant count buffer

* nixl_ep: Fix internode destruction flows

* Re-guard p2p_ptr_get with is_rank_masked

* Merge two !low_latency_mode blocks

* Remove redundant condition
* Fix unbounded growth of in-memory telemetry event buffer

The events_ vector grew without limit between periodic flushes.
Enforce the configured buffer size (NIXL_TELEMETRY_BUFFER_SIZE) by
dropping new events when the buffer is full (drop-newest policy).

Signed-off-by: Batsheva Black <bblack@nvidia.com>

* Reduce naming redundancy for telemetry buffer size

Remove the buffer_size local variable in initializeTelemetry() and
assign directly to maxBufferedEvents_. Use maxBufferedEvents_ instead
of exporter_->getMaxEventsBuffered() in writeEventHelper() since they
hold the same value.

---------

Signed-off-by: Batsheva Black <bblack@nvidia.com>
elastic.py intentionally sends SIGTERM to simulate
rank failure during the elastic test, but PR #1427
treats those expected exits as worker errors. Ignore
planned SIGTERM exits when checking worker exit codes
after the test completes.
* Integration build scripts for vllm

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add copyright

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Remove CUDA version parameter, we can use CUDA_VERSION from base image instead

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Address AI comments

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* Count all matches in LogProblemCounter

* Fix version check for WarnWhenIbPresentButRdmaNotSupported

* Use constexpr variable for ucp version

* Exit after the loop if it matched any regex
* build: use Ninja + memory-aware parallelism for Mooncake build

Upgrade Mooncake from v0.3.9 to v0.3.10.post1 and switch from
Make to Ninja generator. v0.3.10.post1 includes limit_jobs.cmake
(Mooncake PR#1718) which auto-detects available memory and creates
Ninja job pools to cap compile and link parallelism separately.

Previously the build hardcoded `make -j4` as a workaround for OOM
during linking. With Ninja job pools, compilation can use all cores
while memory-heavy link steps are automatically throttled — no
manual -j tuning needed.

Signed-off-by: staryxchen <staryxchen@tencent.com>

* docs: update Mooncake version in ATTRIBUTIONS-CPP.md

Bump Mooncake attribution from v0.3.9 to v0.3.10.post1 to match
the version upgrade in .gitlab/build.sh.

Signed-off-by: staryxchen <staryxchen@tencent.com>

* build: disable Mooncake store module (nixl only needs transfer engine)

Add -DWITH_STORE=OFF to skip building mooncake-store, which nixl
does not use (the Mooncake plugin only links libtransfer_engine).

This also works around a Pimpl compilation error in
S3SnapshotObjectStore introduced in v0.3.10.post1 (see
kvcache-ai/Mooncake#1796).

Signed-off-by: staryxchen <staryxchen@tencent.com>

* ci: bump CI_IMAGE_TAG to 20260402-1

Mooncake version and build flags changed (v0.3.10.post1, Ninja,
-DWITH_STORE=OFF), so the CI image must be rebuilt.

Signed-off-by: staryxchen <staryxchen@tencent.com>

---------

Signed-off-by: staryxchen <staryxchen@tencent.com>
Signed-off-by: ovidiusm <ovidium@nvidia.com>
Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
Co-authored-by: ovidiusm <ovidium@nvidia.com>
Signed-off-by: Raul Akhmetshin <rakhmetshin@nvidia.com>
Use raw pointers instead of dereferencing std::optional in ternary expressions, following the same pattern as recv_topk_idx_ptr and recv_topk_weights_ptr. This prevents GCC's -Wmaybe-uninitialized warning, which becomes a build error with -Werror.

Made-with: Cursor
* nixlbench: fix poll() infinite loop, add SIGTERM handling

- Register SIGTERM handler in main.cpp alongside existing SIGINT handler
- Add !signaled() check to both do-while conditions in poll() so the loop
  exits immediately when terminate is set
- Add checkLiveness() lambda in poll() that queries arePeersAlive() every 5s;
  sets terminate=1 if peer's etcd key is gone, triggering the !signaled() break
- Thread &terminate through execTransfer() and execTransferIterations() so the
  hot transfer loop breaks early on signal instead of running all N iterations

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: add etcd lease-based peer liveness detection

Each rank attaches its presence key to an etcd v3 lease (TTL=15s). A
standalone etcd::KeepAlive object (independent gRPC channel) sends periodic
renewals. If the process dies (any signal including SIGKILL), the KeepAlive
thread dies, the gRPC connection closes, and etcd auto-deletes the key after
the TTL.

- runtime.h: add virtual arePeersAlive() default (returns true) to base class
- etcd_rt.h: add KeepAlive member + arePeersAlive() override declaration
- etcd_rt.cpp:
  - setup(): attach rank key to lease via standalone KeepAlive constructor
    (avoids sharing the main SyncClient gRPC channel)
  - ~xferBenchEtcdRT(): Cancel()+reset() keepalive before rmdir() on clean exit
  - arePeersAlive(): query each peer's rank key; detect absence via
    resp.value().key().empty() (is_ok() returns true even for missing keys)
  - Store etcd endpoints string for standalone KeepAlive constructor
- worker.cpp: use std::_Exit() instead of exit() in synchronize() to bypass
  gRPC atexit handlers that deadlock with the KeepAlive background thread

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* Fix clang

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* Fix copyright

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: address style review comments

- arePeersAlive(): add [[nodiscard]] attribute
- keepalive: change shared_ptr to unique_ptr (ownership is exclusive)
- LEASE_TTL_S: move from class header to anonymous namespace in .cpp,
  rename to lease_ttl_s (ALL_CAPS reserved for macros)
- checkLiveness: use chrono_literals (5s) and direct duration comparison
  instead of duration_cast + count(); add const to local 'now'

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: address safety review comments

- terminate flag: change to std::atomic<int> to prevent data races between
  the signal handler, poll() loop, and OpenMP transfer threads that all read
  or write it concurrently
- destructor null guard: check client \!= nullptr before calling rmdir() in
  ~xferBenchEtcdRT(); setup() may fail after partial init, leaving client null
- cleanupForExit(): add best-effort namespace cleanup before _Exit() so the
  non-leased 'size' key (and any others) are removed; a stale size key would
  corrupt rank allocation in the next run of the same benchmark_group

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: fix python_bindings.cpp for atomic<int> terminate

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: remove redundant keepalive.reset() from destructor

unique_ptr destroys the held object automatically when the destructor
returns; the explicit reset() was a no-op. cleanupForExit() retains
reset() because it runs before _Exit() which skips destructors.

Update header guard.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: handle keepalive and rank-key registration failures in setup()

Wrap KeepAlive construction in try/catch and check the put() result.
Both paths unlock the etcd lock and return -1 on failure, preventing
setup() from reporting success with the rank unregistered.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: address coderabbit review comments

- python_bindings: replace atomic<int>* constructor binding with a
  3-arg lambda factory; use raw pointer return so it works with all
  pybind11 versions (unique_ptr return requires pybind11 >= 2.6)
- nixl_worker: add terminate check in recreate_per_iteration loop to
  match the fast-exit behaviour already present in the standard path
- worker: add static_assert for lock-free atomic<int> and use explicit
  fetch_add(1, relaxed) in signalHandler instead of prefix ++

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: register leased rank key before bumping size counter

A crash between put("size") and put("rank/N", lease) left a stale
non-leased size key that poisoned subsequent runs.  Register the
leased rank key first so a mid-setup crash leaves size unchanged;
only bump size once rank/<n> is successfully attached to the lease.

Also add terminate_ptr check inside execSingleTransfer() polling loop
so threads don't spin in getXferStatus() after SIGTERM is received.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: rename arePeersAlive and use consistent atomic ordering

- Rename arePeersAlive() -> areAllPeersAlive() for clarity
- Use load(memory_order_relaxed) consistently at all terminate_ptr
  read sites instead of implicit seq_cst via operator*; use
  store(memory_order_relaxed) for the liveness-check write

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

* nixlbench: use default memory ordering for terminate flag

Drop explicit memory_order_relaxed from all load()/store()/fetch_add()
calls on the terminate flag. The default seq_cst is fine for a flag
checked a few times per second; cleaner and more readable.

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>

---------

Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
* libfabric: Fix Neuron mem type support regression

Signed-off-by: Ye Xiang <yexiang@amazon.com>

* feat(nixlbench): add Neuron VRAM support for non-CUDA builds

Enable nixlbench VRAM segment allocation, deallocation, and
consistency checks on Neuron (Trainium/Inferentia) devices when
CUDA is not available.

Signed-off-by: Ye Xiang <yexiang@amazon.com>

* address review: do-while(0) wrap macro, add #else guard in cleanupBasicDescVram

* address review: refactor nixlbench VRAM helpers

---------

Signed-off-by: Ye Xiang <yexiang@amazon.com>
The gtest subdir was gated on ucx_dep.found() which checks for the
system UCX library, not whether the UCX plugin is actually being
built. On machines with UCX installed, this enters the gtest subdir
even when UCX is not in enable_plugins, hitting an undefined
ucx_backend_lib variable in the mock backend build.

The mock backend has no UCX source dependency -- the link_with was
unnecessary. Remove it and gate the gtest subdir on the UCX plugin
being enabled rather than the system library being present.

Signed-off-by: Ben Walker <ben@nvidia.com>
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
…1523)

* Build UCX and NIXL at the end to allow easy bisection with cached builds

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Remove pip install step

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Cleanup

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add git log and clean whitespace

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Update help string

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Quote UCX_REF

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Co-authored-by: Adit Ranadive <aranadive@nvidia.com>
Initialize status = NIXL_ERR_NOT_FOUND before the backend creation
loop to silence g++ -Werror=maybe-uninitialized.

Signed-off-by: Ben Walker <ben@nvidia.com>
nv-nmailhot and others added 22 commits April 15, 2026 11:39
Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
* CI: added a new slurm cluster

Signed-off-by: Iaroslav Sydoruk <isydoruk@nvidia.com>

* Fixed python tests

partial_md_example.py:119 unconditionally creates agents with enable_listen_thread=True
and hardcoded default ports (5555 for target, 7777 for initiator),
regardless of whether --etcd or socket mode is used

```
[2026-03-02T17:34:37.516Z] + python3 partial_md_example.py --etcd
[2026-03-02T17:34:38.149Z] 2026-03-02 09:34:37 NIXL INFO    partial_md_example.py:94 Using NIXL Plugins from: /opt/nixl/lib/aarch64-linux-gnu/plugins
[2026-03-02T17:34:38.149Z] 2026-03-02 09:34:37 NIXL INFO    partial_md_example.py:102 NIXL_ETCD_ENDPOINTS is set, using endpoints: http://127.0.0.1:59501
[2026-03-02T17:34:38.149Z] E0302 09:34:37.704217  464337 metadata_stream.cpp:84] Socket Bind failed while setting up listener for MD: Address already in use [98]
```

Signed-off-by: Iaroslav Sydoruk <isydoruk@nvidia.com>

* restore options

Signed-off-by: Iaroslav Sydoruk <isydoruk@nvidia.com>

---------

Signed-off-by: Iaroslav Sydoruk <isydoruk@nvidia.com>
* NIXLBENCH: Runtime without ETCD.

* NIXLBENCH: Fix formatting.

* NIXLBENCH: Review comments.

* NIXLBENCH: Bump tag.

* Apply suggestion from @guy-ealey-morag

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: Colin Hirsch <chirsch@nvidia.com>

* NIXLBENCH: Review comments.

* NIXLBENCH: Fix formatting.

* Factor out constants.

* Bump image tags.

* Add ASIO fallback.

* Fix test run.

* Fix test run.

* Fix formatting.

* Remove libasio-dev install again.

* Catch listen error.

* Fix image tag.

* NIXLBENCH: Review comments.

* Restore CI_IMAGE_TAG.

---------

Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
…470)

* telemetry: share Prometheus Exposer across agents in same process

Each agent tried to create its own prometheus::Exposer HTTP server on
the same port (default 9090), causing the second agent to fail with
EADDRINUSE. The second agent's telemetry was silently disabled.

Share a single Exposer via a process-wide weak_ptr so the first agent
creates the HTTP server and subsequent agents reuse it. Each agent
still owns its own Registry with per-agent labels, so metrics remain
correctly attributed. The Exposer is destroyed when the last agent's
exporter is released.

Signed-off-by: Batsheva Black <bblack@nvidia.com>

* telemetry: share single Prometheus Registry across all agents

Refactor from per-agent registries to a single shared Registry so that
all metric families live in one place, producing cleaner /metrics output.
Each agent's metrics are still distinguished by the agent_name label.
A destructor removes agent-specific metrics from the shared Registry
when an agent is destroyed.

* fix clang

* fix duplicate agent names

* telemetry: fix header guard naming and reject duplicate agent names

Rename the include guard to NIXL_SRC_PLUGINS_TELEMETRY_PROMETHEUS_EXPORTER_H
to follow the repo convention. Change duplicate agent_name handling from
a warning to throwing std::runtime_error so the exporter is never
constructed for a duplicate, preventing use-after-free on destruction.

* resolve comment

* telemetry: address PR review feedback

* telemetry: use RAII destructors for CounterEntry/GaugeEntry cleanup

Move family->Remove(metric) into ~CounterEntry and ~GaugeEntry, then
replace explicit loops in the destructor with counters_.clear() and
gauges_.clear() under the lock.

* fix clang

---------

Signed-off-by: Batsheva Black <bblack@nvidia.com>
Co-authored-by: ovidiusm <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* telemetry: replace string event names with enum type

Replace per-event char[32] eventName_ with a uint8_t enum
nixl_telemetry_event_type_t, avoiding string copies on every
telemetry event and enabling compile-time type checking.

- Add nixl_telemetry_event_type_t enum with all known event types
- Add constexpr telemetryEventTypeStr() returning string_view
- Rename eventName_ to eventType_ for clarity
- Update all callers to use enum values instead of string literals
- Update Python/C++ readers and Prometheus exporter accordingly
- Move Python lookup dicts to module scope
- Remove unused cstring include

Signed-off-by: Batsheva Black <bblack@nvidia.com>

* fix clang

* telemetry: add TODO for C++20 string_view map lookup in Prometheus exporter.

* Update src/core/telemetry/telemetry.cpp

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>

* Update src/core/telemetry/telemetry_event.h

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>

* Update src/core/telemetry/telemetry_event.h

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>

* telemetry: per-status error event types

* resolve comments

* telemetry: assert error paths; move status→event mapping to .cpp

- Declare nixlTelemetryEventTypeForStatus in telemetry_event.h; define in
  telemetry.cpp with explicit NIXL_SUCCESS/NIXL_IN_PROG and NIXL_ASSERT_ALWAYS
  on unhandled values (addresses review: do not silently drop new error codes).
- updateErrorCount: NIXL_ASSERT_ALWAYS for negative status and for missing
  mapping instead of ignoring empty optional.

* fix_clang

* Update src/core/telemetry/telemetry.cpp

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>

* Update src/core/telemetry/telemetry.cpp

Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>

* resolve comment

* telemetry: handle NIXL_SUCCESS/IN_PROG in status→event switch

Explicit cases satisfy -Wswitch under -Werror and assert if
nixlTelemetryEventTypeForStatus is called with a non-error status.

* fix clang

---------

Signed-off-by: Batsheva Black <bblack@nvidia.com>
Signed-off-by: BatshevaBlack <132911331+BatshevaBlack@users.noreply.github.com>
Co-authored-by: Guy Ealey Morag <guymorag142@gmail.com>
Co-authored-by: ovidiusm <ovidium@nvidia.com>
…559)

* Use correct synchronization with etcd metadata comm thread

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Address style comments

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Refactor to reduce code duplication

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Style changes

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* clang-format

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
…y used (#1546)

* Defer plugin load to the point where they are actually used

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Fix unit tests

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Simplify logic

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Fix race due to incorrect sync mode with etcd metadata

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Fix clang format

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Try RW to see perf difference

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Switch back to strict locking

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Simplify code

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Split etcd fix into separate PR

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Revert unrelated change

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Code cleanup

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Clean up logs

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Clang-format

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Simplify python code

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Remove dead code from telemetry

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Const locks

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Add unit test for deferred load/unload flow

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

* Fix copyright

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>

---------

Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* PLUGINS/POSIX: Remove dangling pointer.

* PLUGINS/POSIX: Fix includes.
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@edgargabriel
edgargabriel force-pushed the topic/nixl-sync-4-23-2026 branch 2 times, most recently from 583a29b to 5a475b9 Compare April 27, 2026 16:35
add a cleaner solution for some of the CUDA/HIP incompatilibities, not
just disable -Werror
@edgargabriel
edgargabriel force-pushed the topic/nixl-sync-4-23-2026 branch from 5a475b9 to 39be1de Compare April 27, 2026 16:38
@edgargabriel
edgargabriel merged commit 39be1de into ROCm:develop Apr 28, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.