Support DLPack 1.0

[bdice]

Summary

Coordinate RAPIDS support for DLPack 1.0 and its versioned managed-tensor protocol. DLPack 1.0 retains the ABI of DLTensor and legacy DLManagedTensor, and introduces DLManagedTensorVersioned as the current exchange type. Libraries that expose DLPack through stable C APIs must support the versioned type before relying on DLPack 1.x inputs.

Background

NVIDIA/cuvs#2354 updated its build-time DLPack dependency to 1.0 for 26.08. However, rapidsai/cugraph#5602 is blocked because cuDF still constrains DLPack to >=0.8,<1.0 and the 26.08 release is frozen. We need to approach this version migration in a unified way across libraries, because of the interdependencies. Thankfully, we have mechanisms to do this without ABI breaks (which cuVS only allows every 6 months).

DLPack 1.0 introduces a new DLManagedTensorVersioned that is meant to replace DLManagedTensor. The legacy DLManagedTensor layout is unchanged from DLPack 0.8 to 1.0, so updating the header alone is not an ABI break. However, DLManagedTensorVersioned has a different layout and cannot be passed to an API expecting DLManagedTensor*.

DLPack documents ABI stability within a major version: https://github.com/dmlc/dlpack/blob/77aafa4d3b0f80feffce9ad4c718dd26751ee0e4/include/dlpack/dlpack.h#L45-L46. Its versioned protocol requires consumers to validate the major version before accessing a DLManagedTensorVersioned; a major-version mismatch may only be released via its deleter.

We will need to update RAPIDS libraries to support DLPack 1.0, and support the new DLManagedTensorVersioned.

Current usages of DLPack:

• cuDF: public C++ from_dlpack and to_dlpack APIs exchange legacy DLManagedTensor. Python APIs also produce and consume DLPack objects. Its dependency constraint currently excludes DLPack 1.0.
• cuVS: public C APIs take and return DLManagedTensor*; its language bindings construct those legacy tensors for the C API. This is the primary public ABI migration.
• cuGraph: uses cuDF-to-CuPy DLPack conversions in Python return paths, but has no managed-tensor C API. The proposed C API in #5602 needs to define its legacy and versioned behavior explicitly.
• cuCIM: public C++ image-loading and plugin interfaces use DLTensor directly, backed by a vendored legacy DLPack header. DLTensor itself is layout-compatible with 1.0, so this is a header/dependency and compatibility review rather than a versioned managed-tensor migration.
• cuGraph-GNN / WholeGraph: Cython bindings implement __dlpack__ by allocating legacy DLManagedTensor values and producing dltensor Python capsules. They need versioned capsule production support.

Proposed Work

• Set a RAPIDS-wide DLPack support policy of >=1.0,<2.0 once all affected libraries can consume versioned tensors.
• Update the CMake DLPack configuration to use a compatible installed DLPack header or conda package before downloading the DLPack source; retain downloading only as a fallback.
• Review DLPack pins in conda recipes and developer environments. Packages using dlpack C APIs should have a run pinning on dlpack >=1.0,<2.0.
• Update cuDF, cuVS, cuCIM, and cuGraph-GNN (WholeGraph) managed-tensor producers and consumers to support both DLManagedTensor and DLManagedTensorVersioned (with runtime version checks). Same for cuGraph if it adds C APIs.
• For public C APIs, add versioned entry points or an equivalent explicit versioned dispatch mechanism while retaining legacy entry points during the transition.

DLPack usage details:

• Normalize both managed tensor forms at library boundaries into internal tensor views based on their embedded DLTensor.
• Preserve current ownership semantics: inputs are borrowed and must not have their deleters invoked by the library; consumers invoke deleters for returned managed-tensor views as documented by the API.
• Update Python DLPack exchange to negotiate max_version=(1, 0) and accept both legacy dltensor and versioned dltensor_versioned capsules where the library consumes Python DLPack objects.
• Mark returned versioned views with {major=1, minor=0} and appropriate flags, including read-only for index-owned views.
• Align any retained conda and developer-environment constraints with the supported build-time DLPack versions after the consuming code paths support DLPack 1.0.

Release Plan

We need to revert NVIDIA/cuvs#2354 for 26.08, and target this work for 26.10 or later.

Because it should not be ABI-breaking (only adding new APIs), it does not have to wait for cuVS 27.02 (the next release permitted to break ABI).

We can retain legacy DLManagedTensor support until we have a reason to drop it, which can happen in a future ABI-breaking release.

Acceptance Criteria

• Each affected public API accepts either the legacy DLManagedTensor or the newer DLManagedTensorVersioned.
• CMake uses a compatible installed DLPack header when available and downloads DLPack only when it is unavailable.
• Tests cover legacy and versioned tensor inputs, a major-version mismatch, read-only inputs/outputs where applicable, and managed-tensor deleter behavior.

References

• https://github.com/dmlc/dlpack/blob/v1.0/include/dlpack/dlpack.h
• Update DLPack to 1.0 NVIDIA/cuvs#2354
• Update DLPack to v1.0 cugraph#5602

[seberg]

This sounds good from an API development PoV. For the dependency issue, it seems practical to me to rely on dlpack never breaking ABI on the header level: Just remove it as a runtime dependency entirely seems perfectly fine (dunno in theo.
Having a run-export for it to me seems unnecessary at this point. DLPack must not break ABI, just extend/modify API (I guess technically <0.8 may really not have been ABI stable, but to be honest, it seems so old that I can see this being an OK gamble to to simplify the dependencies).

Most projects vendor the header anyway and while many may not expose C++ API, I don't think that matters as we only have forward declared struct DLManangedTensor in the public API anyway, or can reasonably change to that.)

(Part of the confusion is that dlpack, I think, was initially designed in a way that ABI stability wasn't obvious. But 1.0+ are only major versions w.r.t. to the structs contained, not w.r.t. header ABI changes. The public API will only ever use forward declared struct DLManagedTensor/DLManagedTensorVersioned and we should assume that those will never break ABI -- of course what is inside a DLManagedTensorVersioned depends on the version).

[bdice]

Agreed, that seems reasonable.