Skip to content

Add hybrid RRF retrieval, optional reranking, and evaluation #95

Description

@bcdonadio

Intent

Measure and optionally improve recall after #94 while keeping lexical-only PostgreSQL fully usable.

Design

Retrieve independent lexical and vector candidates, deduplicate by canonical source identity, and fuse ranks with Reciprocal Rank Fusion; default k=60 with configurable candidate/final limits. Never combine heterogeneous raw scores. Return optional provenance for ranks, RRF contributions, provider/model, rerank position, and timing.

Reranking is a separate opt-in OpenAI-v1-compatible chat-completions adapter. It receives redacted bounded snippets plus opaque IDs and must return a strictly validated ID order. Use temperature zero, budgets, timeouts, retries, circuit breaking, and RRF/lexical fallback.

Deliverables

  • Hybrid retrieval/configuration and backward-compatible metadata.
  • Strict reranker adapter.
  • Versioned offline evaluation fixtures/command/report.
  • Recall@k, MRR, nDCG@k, latency, provider usage, failure, and fallback metrics.
  • Explicit rollout switch.

Non-goals

Vendor-specific rerank APIs, online learning, automatic model selection, telemetry upload, default hybrid/reranked results, or parity dependency.

Dependencies

Blocked by #89 and #94. Does not block parity or #225; blocks #226.

Acceptance tests

Golden queries reproducibly compare lexical, vector, RRF, and reranked modes. Tests cover ties, duplicates, missing embeddings, profile/dimension mismatch, provider timeout/429/5xx/auth/malformed/unknown IDs, budgets, redaction, deterministic order, fallback, and public response compatibility. Reports pin dataset/config and capture query plans.

Documentation

Document fusion math, knobs, provenance, evaluation method, privacy/cost/latency, provider setup, circuit breaker/fallback, staged enablement, and rollback.

Rollout

Evaluate offline first. Require no material relevance regression and an explicit latency/cost decision before opt-in canarying. Lexical remains the default until product evidence says otherwise.

Failure handling

Provider or validation failure returns RRF or lexical results with degraded metadata; recall must not fail solely because inference is unavailable. Never accept unknown/duplicate IDs or expose snippets in logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestp2-mediumNext sprint, quality/completenessprj-multi-machineProject: LCM multi-machine

    Type

    No type

    Projects

    Status
    Todo

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions