Implement hybrid search (BM25 + vector with Reciprocal Rank Fusion)

[clafollett]

Intent

Pure vector search is no longer competitive in the code intelligence space as of 2026. Every major competitor (Sourcegraph, Augment Code, GitHub Copilot, Cursor) uses hybrid search combining lexical (BM25/sparse) and semantic (vector) retrieval. Codetriever currently relies solely on dense vector similarity via Qdrant, which fails on exact identifier lookups — searching for authenticate_user may not surface the exact function if the embedding doesn't capture it precisely.

Hybrid search with Reciprocal Rank Fusion (RRF) is the industry-standard fusion algorithm. It runs BM25 and vector queries independently, then fuses results by rank (not score), eliminating the need for score calibration between fundamentally different retrieval methods.

Business Rules

• The system MUST support both lexical (exact match / BM25) and semantic (vector similarity) search in a single query
• Results MUST be fused using Reciprocal Rank Fusion (RRF) to produce a unified ranking
• Exact symbol name matches (function names, class names, variable names) MUST rank higher than semantically similar but inexact matches
• The fusion strategy MUST NOT require manual score calibration between retrieval methods
• The hybrid search MUST respect all existing filters: tenant_id, repository_id, branch
• The system SHOULD support configurable weighting (α parameter) between lexical and semantic components to allow tuning per deployment
• Query classification MAY be used to dynamically weight lexical vs. semantic based on query type (e.g., identifier-like queries favor lexical, natural language queries favor semantic)
• Search latency MUST remain under 200ms for warm queries (p95)

Competitive Context

• Qdrant natively supports sparse vectors in recent versions, enabling SPLADE/BM25 without a second search engine
• Sourcegraph has offered hybrid search for years (regex + semantic)
• Augment Code's Context Engine uses hybrid retrieval as a foundation
• GitHub Copilot combines remote semantic index with local diff scanning
• ParadeDB demonstrates hybrid BM25+vector in PostgreSQL (potential alternative approach)

Acceptance Criteria

• Lexical search component is operational alongside existing vector search
• RRF fusion produces a single ranked result set from both retrieval methods
• Exact identifier searches reliably surface the correct definition
• All existing multi-tenancy and filter behavior is preserved
• Search performance remains within acceptable latency bounds
• Existing semantic search quality is not degraded