Repository navigation
Propagate invalid matrix layout errors in production paths - #1453
Merged
Merged
Conversation
Use fallible matrix construction in existing Result-returning paths and add the standard ANNError conversion for layout validation failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the private dataset conversion helper fallible so invalid matrix dimensions return through the existing anyhow result instead of panicking. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
All changed allocations correctly propagate layout errors through compatible existing error boundaries.
Review effort: Balanced
Findings: None
What changed in this PR
Propagates invalid matrix-layout failures through existing production error boundaries instead of panicking.
Changes:
- Adds
LayoutErrorconversion intoANNError. - Replaces 10 production allocations with
Matrix::try_new. - Preserves start-point and PQ-training domain errors.
| File | Description |
|---|---|
diskann/src/graph/start_point.rs |
Propagates allocation failures via StartPointError. |
diskann/src/error/ann_error.rs |
Adds LayoutError conversion. |
diskann-quantization/src/product/train.rs |
Wraps allocation failures in PQTrainingError. |
diskann-providers/src/model/graph/provider/async_/inmem/full_precision.rs |
Makes reranking allocation fallible. |
diskann-disk/src/search/provider/disk_provider.rs |
Propagates candidate-matrix layout failures. |
diskann-benchmark/src/utils/datafiles.rs |
Makes converted dataset allocation fallible. |
diskann-benchmark/src/exhaustive/spherical.rs |
Propagates compressed-store allocation errors. |
diskann-benchmark/src/exhaustive/product.rs |
Propagates PQ-store allocation errors. |
diskann-benchmark/src/exhaustive/minmax.rs |
Propagates min-max store allocation errors. |
diskann-benchmark-core/src/search/api.rs |
Makes fixed-result matrix allocation fallible. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Alex Razumov (arrayka)
marked this pull request as ready for review
September 30, 2026 00:47
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (39.28%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1453 +/- ##
==========================================
+ Coverage 90.86% 91.87% +1.01%
==========================================
Files 579 579
Lines 114556 114568 +12
==========================================
+ Hits 104088 105264 +1176
+ Misses 10468 9304 -1164
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Mark Hildebrand (hildebrandmw)
approved these changes
Sep 30, 2026
Harsha Simhadri (harsha-simhadri)
approved these changes
Sep 30, 2026
Mark Hildebrand (hildebrandmw)
added a commit
that referenced
this pull request
Oct 5, 2026
# DiskANN v0.60.0 ## Major Changes ### `diskann-utils`: Matrix API changes There have been significant changes to the matrix APIs in `diskann-utils`. The previous contents of `diskann_utils::views` have been moved to `diskann_utils::views::rowmajor`. The following type changes have been made | v0.59 | v0.60 | Remarks | |-|-|-| | `views::Matrix<T>` | `views::rowmajor::Owned<T>` | Size changed from 32 bytes to 24 bytes | | `views::MatrixView<'a, T>` | `views::rowmajor::Ref<'a, T>` | Size changed from 32 bytes to 24 bytes | | `views::MatrixViewMut<'a, T>` | `views::rowmajor::Mut<'a, T>` | Size changed from 32 bytes to 24 bytes | | `MatrixBase` inherent methods | `rowmajor::Matrix` and `rowmajor::MatrixMut` trait methods | | The `rowmajor::Matrix` and `rowmajor::MatrixMut` trait methods need to be imported to access most matrix functionality. In addition, the following methods have been renamed and changed: | v0.59 | v0.60 | Remark | |-|-|-| | `row_iter` | `rows` | No longer panics when `ncols == 0` | | `row_iter_mut` | `rows_mut` | No longer panics when `ncols == 0` | | `par_row_iter` | `par_rows` | No longer panics when `ncols == 0` | | `par_row_iter_mut` | `par_rows_mut` | | | `get_row_unchecked` | `row_unchecked` | | | `get_row_unchecked_mut` | `row_unchecked_mut` | | | `as_mut_view` | `as_view_mut` | | | `try_from` constructors | `try_from_data` | | In addition, * The `Index` and `IndexMut` implementations have been removed. Use `Matrix::element` and `Matrix::element_mut` instead. * The `Init` and `Generator` style constructors for `Matrix::new` have been removed. Instead, `rowmajor::Owned::from_fn` and `rowmajor::Owned::from_element` can be used instead. The former's closure has been changed to take an argument `rowmajor::RowCol` indicated the row and column being initialized. Both of these constructors take now take the initializer as the last argument. * Matrix constructors validate against integer overflow and `isize::MAX` allocations via a `rowmajor::Layout<T>` layout. Constructors panic if these validations fail. `try_*` variations of these constructors which return `rowmajor::LayoutError` instead. ### `diskann-utils`: Strided API changes * Structs `StridedBase`, `StridedView`, and `MutStridedView` were collapsed into a single, immutable `Strided<'a, T>`. * `strided::linear_length` was removed, use `strided::Layout::linear_length` instead. * `strided::TryFromErrorLight` was removed. * `strided::TryFromError` changed from a struct to an enum, and `into_inner`/`as_static` were removed. ### `diskann` `InsertStrategy` is no longer a `SearchStrategy` subtrait. Implementations must now provide ```rust type SearchAccessor; type SearchAccessorError; fn insert_search_accessor(...) -> Result<Self::SearchAccessor, Self::SearchAccessorError>; ``` Previously, `insert_search_accessor` had a provided implementation forwarding to `SearchStrategy::search_accessor`. That default is now gone. This change was done to decouple insert element types from search element types. ### `diskann-disk` The publicly re-exported types below were removed in favor of standard buffered I/O: ``` diskann_disk::storage::CachedReader diskann_disk::storage::CachedWriter ``` Consumers should migrate to `std::io::BufReader`/`BufWriter` and the relevant storage-provider reader/writer. `READ_WRITE_BLOCK_SIZE` also changed from `u64` to `usize`. ### `diskann-quantization` These public traits now require `Debug`: ``` spherical::iface::Quantizer spherical::iface::DynQueryComputer spherical::iface::DynDistanceComputer ``` ### `diskann-inmem` (experimental) The old `layers` abstraction was removed and replaced by the `Representation`/`Store`/`Slots` architecture. And support for a spherically quantized backed was added. # Full Change Log * [diskann-inmem] Prepare code for quantization and beyond by @hildebrandmw in #1352 * Add Neon inner product U8xU1, U8xU2 and U8xU4 kernels by @pfoxARM in #1372 * [CI] Check all workspace features and private docs by @wuw92 in #1400 * [CI] Check Rust license headers by @wuw92 in #1401 * [disk] Replace custom cached I/O with standard buffered readers and writers by @wuw92 in #1396 * [dikkann_wide] Integer SIMDMinMax by @suri-kumkaran in #1407 * Add Neon Squared L2 kernels for USlice2 and USlice4 by @pfoxARM in #1398 * Bump the github-actions group with 3 updates by @dependabot[bot] in #1406 * [CI] Check workspace direct dependency versions by @wuw92 in #1410 * Train fresh disk-search PQ codebooks for each index build by @xwj-ox in #1299 * [diskann-quantization] Make most things in `diskann-quantization/spherical/iface.rs` debuggable by @hildebrandmw in #1414 * [diskann] Make `InsertStrategy` independent of `SearchStrategy` by @hildebrandmw in #1393 * Use #[expect] to detect unnecessary lint suppressions by @wuw92 in #1409 * [diskann-garnet] Store attributes during insert, not after by @metajack in #1412 * [diskann-wide] Fix `alias!` by @hildebrandmw in #1417 * [diskann-utils] Simplify `StridedView`. by @hildebrandmw in #1376 * [diskann-wide] More Neon Zip/Unzip impls by @hildebrandmw in #1405 * [diskann-garnet] Deserialize max id for quantization on load by @metajack in #1419 * More ergonomic `spherical::iface::Quantizer` constructors by @hildebrandmw in #1418 * [diskann-bftree] Clamp number of concurrency test parallel workers by @hildebrandmw in #1433 * Tweak behavior of adaptive L by @magdalendobson in #1354 * Bump the github-actions group with 2 updates by @dependabot[bot] in #1436 * Reform Range Search by @magdalendobson in #1337 * PiPNN 2/6: add numerical kernels by @SeliMeli in #1287 * Matrix overhaul 1 of N - Data Accessors by @hildebrandmw in #1415 * Add `assert_contains!` helper by @hildebrandmw in #1421 * Matrix overhaul 2 of N - `Matrix` invariants by @hildebrandmw in #1420 * [CI] Run SDE tests for diskann with all features by @SeliMeli in #1440 * [diskann-wide] Add Neon `vabdq_u8` and `vabdq_s8`. by @hildebrandmw in #1437 * [diskann-inmem] Add support for spherical quantization by @hildebrandmw in #1432 * [benchmark-runner] Bump edition to 2024 by @hildebrandmw in #1444 * [diskann-wide] Add byte-vector reinterpretations for u32x8 and u64x8 by @partychen in #1445 * Propagate invalid matrix layout errors in production paths by @arrayka in #1453 * Add sparse vector distance kernels (f32/f16 L2, inner product, cosine) by @mana-agarwal in #1434 * Bump the github-actions group with 3 updates by @dependabot[bot] in #1454 * [diskann-utils] Matrix Update 3 of N (constructors) by @hildebrandmw in #1451 * [diskann-utils] Matrix Update 4 of 4 by @hildebrandmw in #1457 ## New Contributors * @mana-agarwal made their first contribution in #1434 **Full Changelog**: v0.59.0...v0.60.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#1420 introduced layout validation and the fallible
Matrix::try_newconstructor, but intentionally left existingMatrix::newusages untouched: changing every call site would create substantial noise and forceResultthrough otherwise infallible APIs. KeepingMatrix::newas the ergonomic default was the right general tradeoff.This PR follows up selectively on that work. Production paths that already return errors can propagate invalid runtime-derived dimensions with little or no signature churn, reducing the layout-panic surface without making unrelated APIs noisier.
What
LayoutErrortoANNErrorconversion.Matrix::try_newin 10/16 production allocations whose existing error boundaries allow local propagation, including search results, reranking, start-point generation, PQ training, benchmark compression, and dataset conversion.StartPointErrorandPQTrainingError).Matrix::new: their dimensions are controlled fixtures, and some directly exercise the panicking convenience constructor.IdAggregator::finishneeds result propagation through its collector;RandomRotation::newandSimplePivots::flattenare public APIs; Latin-hypercube sampling is a public trait method with additional input-validation panics that should be addressed through a dedicated fallible API rather than only changing allocation.Validated with formatting, affected-crate checks, and focused start-point, PQ training, determinant-diversity, disk search, and datafile tests.