Migrate brute-force k-NN benchmark to NVBench - #2371
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Hi @PointKernel. What's the justification for this change? Are you working on improving brute force knn? Not a great idea to cherry pick algorithms from cuVS bench outside of the existing infra, especially not to yet another benchmark harness. If we are going to make such a change, we should do so for all the supported algorithms. Of course, if this is just for your own internal testing then please feel free to disregard. Opening up a PR tells me you probably want this merged, and in that case, we need to have a much larger discussion about this. |
|
@cjnolet Thanks for raising your concern. I am not specifically working on brute force kNN. I noticed this benchmark was already doing its own GPU timing and synchronization, so it seemed like a simple place to show what NVBench could provide. We migrated all the cuDF GPU benchmarks to NVBench and have found it useful for kernel benchmarks. It takes care of timing, synchronization, warmups, repetitions, and parameter handling that would otherwise need to be implemented manually. Since this benchmark already contains much of that boilerplate, it felt like a good contained example for comparing the two approaches. I agree this should not become a one off in cuVS. I opened the PR so we could discuss the idea around a concrete example rather than an abstract proposal. If cuVS finds this direction useful, this could be the starting point for a broader and more consistent effort across the appropriate GPU benchmarks. |
|
Okay that does makes sense. Thank you for the explanation! |
|
Thank you @cjnolet for checking out this diff and thank you @PointKernel for putting it together. cuDF benchmarks were originally written in googlebench. We started migrating to nvbench in 2021 and just finished in 2025, so there is no need to rush adoption all at once. NVbench is the standard for CCCL and a great tool to make GPU benchmarking easier. My favorite feature is that benchmark parameters can be varied at runtime without recompiling anything. I asked @PointKernel to open the PR to start the conversation with cuVS about nvbench. |
|
Thanks guys. I should probably point out, though, that we are getting close to marking our c++ backend as legacy as we move more towards benchmarking the Python backend in cuVS-bench. It's not formal yet, because we are still in the process of adopting and fleshing out the new python backend, but that is the ultimate goal. Does nvbench support any type of Python layer? |
Yes, I believe so. @oleksandr-pavlyk could probably comment more on this. |
There is no restriction on types to use in the benchmark. You can peruse Python examples of benchmarking at the repo: https://github.com/NVIDIA/nvbench/tree/main/python/examples They include benchmarking of numba-cuda kernels, torch operators, cupy functions, cute-dsl operations, and kernels compiled from source using cuda-core. Benchmarking in Python currently expects that work is submitted on the stream supplied in Unlike C++, Python does not currently allow for |
4 participants
This PR migrates the standalone brute-force k-NN benchmark from generic Google Benchmark target plumbing and one-shot host wall-clock timing to NVBench. It follows the cuDF setup by fetching static NVBench through rapids-cmake, using a shared NVBench main, and registering typed and value axes.
NVBench provides CUDA-event timing, automatic warmup and repeated measurements with noise-based stopping, typed and value axes, and element-throughput reporting for this device-only search. Input allocation, data initialization, and index construction remain outside the timed region.
The benchmark audit found that focused single-GPU search companions for cuVS brute force, IVF-Flat, IVF-PQ, IVF-SQ, IVF-RaBitQ, and CAGRA are appropriate NVBench candidates. GGNN may also qualify once its search measurement boundary is isolated. Build workflows and the end-to-end ANN benchmarks should remain on Google Benchmark, including CPU HNSW/FAISS/DiskANN, FAISS GPU comparisons, multi-GPU, and hybrid CAGRA workflows, because they rely on JSON configuration, persisted indexes, recall, threaded QPS and latency, NVTX-derived counters, and the existing
cuvs_benchGoogle Benchmark JSON parser.