Skip to content

[core] feat(rdt): enable driver-side ray.put with NIXL tensor transport - #65072

Open
n3sfan wants to merge 2 commits into
ray-project:masterfrom
n3sfan:fix/rdt-driver-put-nixl
Open

[core] feat(rdt): enable driver-side ray.put with NIXL tensor transport#65072
n3sfan wants to merge 2 commits into
ray-project:masterfrom
n3sfan:fix/rdt-driver-put-nixl

Conversation

@n3sfan

@n3sfan n3sfan commented Jul 28, 2026

Copy link
Copy Markdown

Allow ray.put(value, _tensor_transport='nixl') from a driver process. Previously this failed because get_runtime_context().current_actor raises RuntimeError in a non-actor context.

Changes:

  • Make RDTMeta.src_actor and TransferMetadata.src_actor Optional
  • Catch RuntimeError in put_object and set src_actor=None for driver
  • Reject two-sided transports (NCCL/GLOO) from driver with clear error
  • Guard src_actor dereferences in _abort_transport and transfer paths
  • Handle free_object_primary_copy locally for driver-owned objects
  • Raise ValueError for _use_object_store=True on driver-owned RDT objects
  • Align worker.py error to RuntimeError with 'one-sided transport' message

Fixes #64714

Tests

Three new tests in python/ray/tests/rdt/test_rdt_nixl.py:

test_driver_put_nixl

End-to-end driver-side NIXL flow:

  1. ray.put(tensor, _tensor_transport="nixl") from the driver
  2. Verify src_actor is None in RDT metadata
  3. Verify tensor registered in NIXL transport
  4. Actor consumes the driver-created ref via actor.sum.remote(ref)
  5. Driver-local ray.get(ref) returns the original tensor
  6. After del ref + gc.collect(), verify cleanup: tensor freed, RDT metadata removed, NIXL metadata removed, tensor descriptor cache cleared

test_driver_owned_rdt_rejects_object_store_fallback

Verify _use_object_store=True on a driver-owned RDT object raises ValueError matching _use_object_store=True instead of crashing with AttributeError on None.__ray_call__.

test_driver_rejects_two_sided_transport

Verify ray.put(tensor, _tensor_transport="nccl") from the driver raises RuntimeError matching one-sided transport. Two-sided transports require an actor context to coordinate the send.

Running

# New driver tests
pytest -sv python/ray/tests/rdt/test_rdt_nixl.py \
  -k "test_driver_put_nixl or test_driver_rejects_two_sided_transport or test_driver_owned_rdt_rejects_object_store_fallback"

Results

Test Result
test_driver_put_nixl PASSED
test_driver_owned_rdt_rejects_object_store_fallback PASSED
test_driver_rejects_two_sided_transport PASSED

Allow ray.put(value, _tensor_transport='nixl') from a driver process.
Previously this failed because get_runtime_context().current_actor raises
RuntimeError in a non-actor context.

Changes:
- Make RDTMeta.src_actor and TransferMetadata.src_actor Optional
- Catch RuntimeError in put_object and set src_actor=None for driver
- Reject two-sided transports (NCCL/GLOO) from driver with clear error
- Guard src_actor dereferences in _abort_transport and transfer paths
- Handle free_object_primary_copy locally for driver-owned objects
- Raise ValueError for _use_object_store=True on driver-owned RDT objects
- Align worker.py error to RuntimeError with 'one-sided transport' message

Fixes ray-project#64714

Signed-off-by: n3sfan <quangminhcantho43@gmail.com>
@n3sfan
n3sfan requested a review from a team as a code owner July 28, 2026 10:28
@n3sfan n3sfan changed the title feat(rdt): enable driver-side ray.put with NIXL tensor transport [core] feat(rdt): enable driver-side ray.put with NIXL tensor transport Jul 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for driver-owned RDT (Ray Distributed Tensor) objects, allowing ray.put() to be called from the driver using one-sided transports like NIXL. It updates the RDT manager to handle cases where src_actor is None during transport abortion, actor cleanup, and object freeing, and adds corresponding unit tests. The review feedback suggests correcting a misleading error message in worker.py regarding two-sided transports and changing a ValueError to RuntimeError in rdt_manager.py for consistency.

Comment thread python/ray/_private/worker.py
Comment thread python/ray/experimental/rdt/rdt_manager.py

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 9f2d119. Configure here.

Comment thread python/ray/_private/worker.py Outdated
- worker.py: Fix misleading two-sided transport error message;
  ray.put() never supports two-sided transports, not just from driver
- rdt_manager.py: Change ValueError to RuntimeError for consistency
  with other RDT transport checks

Signed-off-by: n3sfan <quangminhcantho43@gmail.com>
@ray-gardener ray-gardener Bot added core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community core Issues that should be addressed in Ray Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RDT: Enable ray.put from a non ray actor

2 participants