Skip to content

feat: implement DiskANN index with PQ support #66

Description

@huanglune

Description

Implement DiskANN (Disk-based Approximate Nearest Neighbor) index to support large-scale vector search that exceeds memory capacity.

Features

  • DiskANN index implementation with disk-based graph storage
  • Product Quantization (PQ) for vector compression
  • Aligned I/O infrastructure for efficient disk access
  • KMeans clustering for PQ codebook training

Technical Details

DiskANN Index

  • Disk page layout for vectors and graph edges
  • Builder for constructing disk-based index
  • Searcher for efficient disk-based ANN search

Product Quantization

  • PQ codebook training using KMeans
  • PQ encoding/decoding for vector compression
  • Distance computation using lookup tables

Infrastructure

  • Aligned reader for O_DIRECT disk I/O
  • Memory alignment utilities

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions