-
Notifications
You must be signed in to change notification settings - Fork 215
[REVIEW] New Dataset API Clarifying Ownership #1846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
HowardHuang1
wants to merge
38
commits into
NVIDIA:release/26.08
Choose a base branch
from
HowardHuang1:HH-Dataset-API
base: release/26.08
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
f221341
Add Dataset API for CAGRA (host vs device / owning vs view / standard…
HowardHuang1 badb837
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 41a6f22
Fix failing hnsw host index serialize_to_hnswlib test case
HowardHuang1 b406041
squash all Tarang commits to remove 26.10 ancestry. Should be left wi…
HowardHuang1 23e4114
Merge pull request #1 from tarang-jain/hh-dataset-api
HowardHuang1 0882b6c
merge API updates
aamijar 56cb7e1
c-serialize
divyegala e2d7b53
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 b9dd705
rust: make CAGRA dataset ownership explicit
yan-zaretskiy 2354487
record type and layout in serialization
divyegala b28cc72
merge and reconcile conflicts
divyegala 07f91a1
rename make verb + noun
aamijar 5e334b2
skip dataset when reading graph only
divyegala dab5c6b
Add CAGRA build heuristics independent of HNSW parameters (#2345)
achirkin 195b04c
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 11af0a6
Fix failing rocky clib tests on CI
HowardHuang1 a306dc9
remove cuvsCagraIndex_t from dataset.h
aamijar 7e68c56
finish audit
divyegala 5f7b90b
Fix regression introduced by upstream merge still using old build() f…
HowardHuang1 fdfab57
review bug
divyegala 21481e0
unify dataset make and view factories
aamijar 6d534e2
Revert "unify dataset make and view factories"
aamijar 2b753ac
unify dataset make and view factories 2
aamijar b4f7d3e
Fix failing Java test cases. Java test call sites were outdated and m…
HowardHuang1 2ffe67b
Fix merge conflicts. Factory names were using old factory names prior…
HowardHuang1 ed911bf
conflicts
divyegala c503887
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 38cf6fc
Have downstream wrappers mirror dataset factory consolidation in C AP…
HowardHuang1 80905ea
Fix merge conflict
HowardHuang1 96ff183
Merge remote-tracking branch 'upstream/release/26.08' into HH-Dataset…
HowardHuang1 a6dd4eb
Fix merge conflict that brought back compression param in a ann_cagra…
HowardHuang1 089808a
merge
divyegala f52a0a7
merge
divyegala 501d066
undo java and python changes
divyegala 469f912
Fix merge conflict where upstream introduced a new cpp test case that…
HowardHuang1 33ce2bd
Update get_dlpack.cmake
divyegala 679553e
Merge pull request #2 from divyegala/c-serialize
HowardHuang1 8b20957
Fix styles. Fix copyright issue from merge
HowardHuang1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| /* | ||
| * SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| #pragma once | ||
|
|
||
| #include <cuvs/core/c_api.h> | ||
| #include <cuvs/neighbors/common.h> | ||
|
|
||
| #include <dlpack/dlpack.h> | ||
| #include <stdint.h> | ||
|
|
||
| #ifdef __cplusplus | ||
| extern "C" { | ||
| #endif | ||
|
|
||
| /** | ||
| * @brief Generic dataset layout kind for C API dataset handles. | ||
| */ | ||
| typedef enum { | ||
| CUVS_DATASET_LAYOUT_STANDARD = 0, | ||
| CUVS_DATASET_LAYOUT_PADDED = 1 | ||
| } cuvsDatasetLayout_t; | ||
|
|
||
| /** | ||
| * @brief Memory space holding a C API dataset handle's data. | ||
| */ | ||
| typedef enum { | ||
| CUVS_DATASET_MEM_TYPE_HOST = 0, | ||
| CUVS_DATASET_MEM_TYPE_DEVICE = 1 | ||
| } cuvsDatasetMemType_t; | ||
|
|
||
| /** | ||
| * @brief Owning dataset handle. | ||
| * | ||
| * `addr` points to C++ owning dataset storage managed by the C API. `mem_type` identifies the | ||
| * memory space and `layout` identifies the data layout (standard or padded). | ||
| */ | ||
| typedef struct { | ||
| uintptr_t addr; | ||
| void (*destroy_addr)(void*); | ||
| DLDataType dtype; | ||
| cuvsDatasetMemType_t mem_type; | ||
| cuvsDatasetLayout_t layout; | ||
| } cuvsDataset; | ||
| typedef cuvsDataset* cuvsDataset_t; | ||
|
|
||
| /** | ||
| * @brief Non-owning dataset view handle. | ||
| * | ||
| * `addr` points to C API-owned metadata that references caller-provided tensor memory. The | ||
| * `mem_type` and `layout` fields identify the concrete dataset view type. | ||
| */ | ||
| typedef struct { | ||
| uintptr_t addr; | ||
| void (*destroy_addr)(void*); | ||
| DLDataType dtype; | ||
| cuvsDatasetMemType_t mem_type; | ||
| cuvsDatasetLayout_t layout; | ||
| } cuvsDatasetView; | ||
| typedef cuvsDatasetView* cuvsDatasetView_t; | ||
|
|
||
| /** | ||
| * @brief Create an empty owning dataset handle. | ||
| * | ||
| * The dataset storage, memory type, layout, and dtype are populated by the operation that fills | ||
| * this handle. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetCreate(cuvsDataset_t* dataset); | ||
|
|
||
| /** | ||
| * @brief Create an owning padded dataset from a host- or device-resident tensor. | ||
| * | ||
| * Memory residency is inferred from the tensor. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakePadded(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDataset_t* padded_dataset); | ||
|
|
||
| /** | ||
| * @brief Create a non-owning padded dataset view from a host- or device-resident tensor. | ||
| * | ||
| * Memory residency is inferred from the tensor. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakePaddedView(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDatasetView_t* padded_dataset); | ||
|
|
||
| /** | ||
| * @brief Create a non-owning standard dataset view from a host- or device-resident tensor. | ||
| * | ||
| * Memory residency is inferred from the tensor. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakeStandardView(cuvsResources_t res, | ||
| DLManagedTensor* dataset, | ||
| cuvsDatasetView_t* standard_dataset); | ||
|
|
||
| /** | ||
| * @brief Create a non-owning view wrapper from an owning dataset. | ||
| * | ||
| * The returned view references the owning dataset's storage. The dataset must outlive the view. | ||
| * | ||
| * @param[in] dataset owning dataset handle | ||
| * @param[out] view output view handle | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetMakeViewWrapper(cuvsDataset_t dataset, | ||
| cuvsDatasetView_t* view); | ||
|
|
||
| /** @brief Destroy an owning dataset handle created by a `cuvsDatasetMake*` function. */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetDestroy(cuvsDataset_t dataset); | ||
|
|
||
| /** | ||
| * @brief Destroy a non-owning dataset view handle created by a dataset view factory or | ||
| * `cuvsDatasetMakeViewWrapper`. | ||
| */ | ||
| CUVS_EXPORT cuvsError_t cuvsDatasetViewDestroy(cuvsDatasetView_t dataset_view); | ||
|
|
||
| #ifdef __cplusplus | ||
| } | ||
| #endif | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.