Skip to content

feat(diskann): expose cross-platform Python update APIs - #106

Merged
huanglune merged 1 commit into
AlayaDB-AI:mainfrom
ScissorJack-ever:feat/python-update-api
Jul 22, 2026
Merged

feat(diskann): expose cross-platform Python update APIs#106
huanglune merged 1 commit into
AlayaDB-AI:mainfrom
ScissorJack-ever:feat/python-update-api

Conversation

@ScissorJack-ever

Copy link
Copy Markdown
Contributor

Description

Expose standalone DiskANN build, search, and update APIs to Python, using external IDs for all public update operations.

Related Issue

N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • CI/CD changes

Changes Made

  • Added Python bindings for DiskANN build, open, search, insert, batch insert, remove, batch remove, flush, and external-ID lookup.
  • Added external-ID validation and mapping persistence across insert, delete, flush, and reload.
  • Added macOS update support using pread, pwrite, ftruncate, and fsync.
  • Added Windows update support using overlapped ReadFile/WriteFile, file resizing, and FlushFileBuffers.
  • Allowed Windows IOCP search handles to coexist with the writable update handle.
  • Added a dedicated DiskANN CMake dependency target, independent from LASER.
  • Made libcoro available on every supported platform while keeping liburing Linux-only.
  • Fixed Linux ARM wheel imports by linking libaio even when LASER is disabled.
  • Restored tombstone and external-ID state for read-only reloads.
  • Added portable C++ update coverage and enabled Python update tests on all wheel platforms.
  • Fixed io_uring E2E tests to distinguish blocking-backend validation from reactor-specific result validation.

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
cmake --build build/Verify --target test_diskann_portable_update
ctest --test-dir build/Verify -R 'test_diskann_(portable_update|disk_page_io)$' --output-on-failure

python -m pytest python/tests/diskann/test_update.py -q

cmake --build build/Verify --target _alayalitepy
cmake --build build/VerifyNoLaser --target _alayalitepy

build/Verify/tests/diskann/test_diskann_update_e2e \
  --gtest_filter='UpdateE2ETest.PipelinedSearch*'

Results:

  • Python DiskANN update tests: 6 passed.
  • Portable DiskANN update test: passed.
  • Existing DiskPageIO tests: passed.
  • Blocking pipelined-search contract test: passed.
  • Reactor result-equivalence test: skipped when io_uring is unavailable.
  • Python extension builds successfully with LASER both enabled and disabled.
  • LASER-disabled extension correctly records libaio as a runtime dependency.
  • All configured commit hooks passed.

Checklist

  • My code follows the project's coding style
  • I have run the configured lint and commit hooks
  • I have added tests that prove my fix/feature works
  • I have run the complete make test suite
  • Documentation changes are not required
  • My commits follow the conventional commits format

Additional Notes

  • Regular DiskANN updates now use a portable blocking backend on macOS and Windows.
  • search_pipelined() remains a Linux io_uring-specific optimization and intentionally does not silently fall back to blocking I/O.
  • Native macOS and Windows wheel jobs provide the final platform-specific compilation and runtime verification.

@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@huanglune
huanglune merged commit 87a79a2 into AlayaDB-AI:main Jul 22, 2026
11 checks passed
@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants