upstream nixl sync 3-13-2026#24
Merged
edgargabriel merged 45 commits intoMar 13, 2026
Merged
Conversation
--------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
--------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
* Simplify telemetry test Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Refactor code to simplify Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
--------- Signed-off-by: Colin Hirsch <chirsch@nvidia.com>
since we can only run 4 PRs GPU tests we need to increase the allocation timeout so the next does not fail on the 10 minutes timeout for allocation. Signed-off-by: Daniel Pressler <danielpr@nvidia.com>
Without this fix, I get the following error: > ERROR: Unknown variable "ucx_gpu_device_api_v2_available" meson 1.3.2 Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>
Build libnixl_capi.so shared library from wrapper.cpp (exports nixl_capi_* C symbols) so downstream consumers can load nixl at runtime without build-time linking
Rewrite stubs.cpp from abort-on-call to dlopen/dlsym lazy forwarding — stubs now dlopen("libnixl_capi.so") at runtime and forward all calls to the real implementation
Add libnixl.so existence check in build.rs before attempting to link, so the fallback to stubs works correctly when nixl-sys is used as a git dependency (headers available from source tree but libraries not installed)
Link -ldl in stub builds for dlopen/dlsym support
* Add nixl_object_test.cpp and test build configuration * prepare for review * cleanup * remove files not part of this review * clang fixes * Remove VRAM option from nixl_object_test. Address build issue in other plugin functional tests * copyright fixes * Removed accel_type option * Address code review comments * Added docstrings * Remove skip_read/skip_write flags * Generate unique object keys based off a timestamp * Clang and copyright changes * Addressed more code review comments. Cleaned up example. * More changes to examples and help message
Specifically allows to easily connect to Azurite for local testing instead of requiring authentication using Microsoft Entra Id and making acutal API requests to Azure Blob Storage. In addition, nixlbench and the end to end tests were updated to accept connection strings to be able to test directly against Azurite. Signed-off-by: Kyle Knapp <kyleknapp@microsoft.com> Co-authored-by: ovidiusm <ovidium@nvidia.com>
* Sweep less request parameters Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Bump worker count Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * bisect Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * bisect Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * bisect Signed-off-by: ovidiusm <ovidium@nvidia.com> * Revert change in test yaml Signed-off-by: ovidiusm <ovidium@nvidia.com> * Revert change in c++ tests Signed-off-by: ovidiusm <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Signed-off-by: ovidiusm <ovidium@nvidia.com>
--------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Raul Akhmetshin <rakhmetshin@nvidia.com>
* libfabric: fix supported memory types Should support VRAM_SEG only if FI_HMEM_CUDA is there. Signed-off-by: Feng Ji <fengnji@amazon.com> * libfabric: enalbe tcp provider tcp provider is the recommeneded one in libfabric, over sockets provider, which is deprecated. Signed-off-by: Feng Ji <fengnji@amazon.com> * libfabric: calculate offset with remote region base addr For providers without FI_MR_VIRT_ADDR, offset is need. Signed-off-by: Feng Ji <fengnji@amazon.com> * libfabric: fix requested key conflict requested_key is needed for providers without FI_MR_PROV_KEY. Today only 32-bit is used, while libfabric supports 64-bit keys. Signed-off-by: Feng Ji <fengnji@amazon.com> --------- Signed-off-by: Feng Ji <fengnji@amazon.com>
* Add support to prepare Fifo Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix errors Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Add vector read/write support Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Use prepare fifo APIs Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove RCMODE Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove rcmode in prepXfer Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove rcmode from CI Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix cleanup Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Maintain a single transfer ID Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix formatting Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix formatting after changes Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove redundant FIFO_ITEM_SIZE Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Update commit SHA of UCCL Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove unused variable Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove unused vars Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix error Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Add recent updates to README Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Fix copyright and format Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Increase CI image tag Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Use latest UCCL commit SHA Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Add recent UCCL Commit - Fixing build issue on ARM platform - Add TCP support Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Remove sleep during cleanup Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Add stop_accept API for graceful cleanup Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Use CodeRabbit suggestions Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Add cleanup change to address crash Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Bump up the IMAGE_TAG Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Revert CI_IMAGE_TAG Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> * Update IMAGE_TAG Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> --------- Signed-off-by: Pravein Govindan Kannan <pravein.govindan.kannan@ibm.com> Signed-off-by: Mikhail Brinskiy <brminich@users.noreply.github.com> Co-authored-by: Mikhail Brinskiy <brminich@users.noreply.github.com>
There is a bug in the currently used SDK 1.11.581 where the CrtCallback uses data after it was freed. SDK version 1.11.760 fixes this. Update dockerfiles to use latest version. Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
* libfabric: remove control rail Remove control rail infrastructure and move notifications to rail 0. This simplifies the architecture while maintaining notification functionality. Key changes: - Remove control_rails_ vector and createControlRails() - Use rails_[0] for postControlMessage() with fi_senddata - Mark rail 0 as active when posting notifications to ensure CQ progress - Remove RailType enum and simplify API (insertAllAddresses, cleanupConnection) - Rename data_rail -> rail throughout (data_rails_, getNumRails, etc.) Validated: 8-device nixlbench and vLLM test with efa & tcp provider pass with zero EAGAIN errors Signed-off-by: Ye Xiang <yexiang@amazon.com> * libfabric: address review comments - Add validation to reject empty remote endpoint lists in createAgentConnection - Remove stale log message referencing control rails - Add zero-rail validation and reset state in createRails - Fix MR cleanup rollback to use index-based iteration - Remove stale @param control_endpoints_out from documentation Fixes potential crashes and memory leaks identified in PR #1386 review. Signed-off-by: Ye Xiang <yexiang@amazon.com> * libfabric: address review comments - Replace vector with unordered_set for rails_to_process * Cleaner code: rails_to_process.insert(0) handles duplicates automatically * Better performance: O(1) insertion vs O(n) for vector * No ordering concerns for CQ progress (rails are independent) - Remove unnecessary 'data' prefix from log messages * 'data fi_addrs' -> 'fi_addrs' * 'active data rail' -> 'rail' - Remove stale comments about control rails - Propagate progress failures in postXfer * Don't mask transport errors from progressActiveRails() Addresses review comments from fengjica and CodeRabbit on PR #1386 --------- Signed-off-by: Ye Xiang <yexiang@amazon.com>
…NIXL backend) (#1302)
…tMinLimit (#1368) plugins/obj/s3_crt: align CRT MPU behavior, fix request lifetime, update tests and CI - Align CRT client config.partSize and multipartUploadThreshold with crtMinLimit. When crtMinLimit is set, route objects >= crtMinLimit to the CRT client and ensure multipart upload (MPU) is consistently used. The CRT SDK clamps partSize < 5 MiB to 5 MiB and logs a warning; use crtMinLimit >= 5242880 to avoid this. Updated README accordingly. - Heap-allocate PutObjectRequest and GetObjectRequest in putObjectAsync()/getObjectAsync() to fix a use-after-free issue caused by the CRT SDK retaining raw pointers beyond the request scope. - Mark parameters const where applicable. - Update CRT tests to use crtMinLimit=5 MiB and MPU-sized buffers: - obj_plugin.cpp: 10 MiB (two 5 MiB parts) - obj.cpp (objCrtTestFixture): 6 MiB (above threshold) and 1 MiB (below) - CI: - Use newer AWS SDK to avoid S3 CRT segfaults. - Refresh build matrix YAML for new CI tags. Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
* Deprecate notifMsg/hasNotif for optional field Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add API version 1 tag Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Nixl agent config: simplify object creation by allowing to set all fields directly Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Mark merge desc and connection info fields as deprecated Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add shorthand without agent_name Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format and copyright fix Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Replace class with public members with struct Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Bindings: pass backends as const reference Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Address comments Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Add test for empty notification string Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format & copyright Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove obsolete comment Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Allow deserialization of empty notif msg Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Remove nixlApiVersionV1 Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Simplify optional use Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Replace curly braces constructor Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Agent config constructor and destructor Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Address comments on notification params handling Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Address comments: remove V1 tag, update prep callers Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * clang-format Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Revert notification helper Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Revert "Revert notification helper" This reverts commit 3e22ce676c5b6eefd15199b26f8c07ff08974220. Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> * Reapply "Revert notification helper" This reverts commit 935eb91653974d4357bf19ac844d06d27b73c71f. Perf is now same as baseline. Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> --------- Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
…1394) Fixing regression in unit tests due to license notice XML comment in test topo files being out of xml tag. Also updating README and comments for PR 1302 (NUMA-aware rail selection for DRAM_SEG memory type).
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Nate Mailhot <nmailhot@nvidia.com>
file_offset calculation fixed for non-GUSLI plugins. The regression was introduced by commit 62a5b3e. Signed-off-by: Anton Nayshtut <anayshtut@nvidia.com>
…n (#1404) - Add 50 ms sleep in checkXfer/getNotifs polling loops to yield CPU to CRT SDK background threads, reducing multi-buffer test flakiness - Assert NIXL_SUCCESS (not just fall-through) after polling loop so a timeout fails with "Transfer timed out after 30 seconds" instead of a misleading data-mismatch error from checkLocalMem - Extract timeout into COMP_TIMEOUT constant (30 s) and apply it to both performTransfer and verifyNotifs - Add matching ASSERT_GT(num_notifs, 0) timeout guard in verifyNotifs Signed-off-by: Adit Ranadive <aranadive@nvidia.com>
Signed-off-by: Artemy Kovalyov <artemyko@nvidia.com>
* chore: update Attributions for release 1.0.0 * fix licenses * add duel crates * precommit * add missing package
gaoikawa
approved these changes
Mar 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
resync with upstream repo