Module retrieval Roadmap

Retrieval Module Roadmap

Current Status

Phase 3 (Issue #5416) delivered: LoRAPackage and PortableAdapterProduct artifact classes, serialization APIs, integrity helpers, and LoRAManifestStore implemented. 55 GTest cases and a Google Benchmark suite are in place.

Hybrid Retrieval Rollout: staged rollout plan mapped to module readiness in ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md (issue #5468). Four phases: A (exact-first), B (local tensor + ANN), C (distributed coordination), D (optional GPU).

In Progress

• [~] hybrid retrieval rollout plan — Phase A exact-first entry criteria (Target: Q3 2026)
• [~] planning-to-contract traceability cleanup across EPIC 1 module docs (Target: Q3 2026)
• [~] phase-gate readiness definitions for tests and benchmarks (Target: Q3 2026)

Planned Features

Short-term (3-6 months)

• Phase A gate: single-shard exact retrieval verified end-to-end (Target: Q3 2026)
• [~] Phase B gate: ANN + CPU parity tests passing for Category A kernels (Target: Q3 2026) — COMPLETED 2026-08-08
• Phase C gate: multi-shard exact routing verified under shard failure injection (Target: Q4 2026)

Mid-term (6-12 months)

• complete phase-6 acceptance documentation from measured behavior (Target: Q1 2027)
• complete phase-7 integration into default build/test pipelines (Target: Q1 2027)
• Phase D gate: GPU break-even benchmark results reviewed and published (Target: 2027)

Implementation Phases

LoRA Artifacts (EPIC 1.4) — COMPLETE

Phase 1: Design / API Contract

• EPIC 1 contract ownership mapped to headers and planning docs
• LoRAPackage / PortableAdapterProduct artifact taxonomy defined

Phase 2: Core Implementation

• src/retrieval/include/lora_package.h — full public API (2026-07-16)
• src/retrieval/src/lora_package.cc — serialization, integrity, store CRUD

Phase 3: Error Handling and Edge Cases

• from_json throws std::invalid_argument on missing required fields
• verifyIntegrity returns false on empty hash, tampered fields, bad signature
• Malformed import entries skipped gracefully in importPackages
• Architecture allow-list enforced in supportsArchitecture (case-insensitive)

Phase 4: Tests

• 55 GTest cases in tests/epic1_retrieval/lora_package_test.cc
• Coverage: serialization round-trips, lifecycle status, integrity hash, signature verifier callback, CRUD operations, bulk export/import, lifecycle integration test

Phase 5: Performance and Hardening

• Benchmark suite in benchmarks/epic1_retrieval/lora_loading_bench.cc (12 BM_ scenarios)
• Portable SHA-256 implementation (no third-party crypto dependency)
• Thread-safe LoRAManifestStore using std::mutex

Phase 6: Documentation and Acceptance

• Full Doxygen API documentation on all public types and methods
• docs/EPIC1_LORA_ARTIFACTS.md updated to reflect implemented state
• Acceptance docs tied to measured runtime evidence (Target: Q1 2027)

Phase 7: Production Integration

• src/retrieval/CMakeLists.txt — themis_retrieval library target activated
• tests/epic1_retrieval/CMakeLists.txt — GTest target activated
• benchmarks/epic1_retrieval/CMakeLists.txt — benchmark target activated
• cmake/CMakeLists.txt — src/retrieval subdirectory wired into build
• hybrid retrieval rollout plan: staged delivery mapped to module readiness (issue #5468)

Hybrid Retrieval Rollout — IN PROGRESS

Phase 1: Design / API Contract

• core retrieval module docs aligned to source-verifiable scaffold state
• hybrid retrieval rollout plan documented in ai_working/HYBRID_RETRIEVAL_ROLLOUT_PLAN.md

Phase 2: Core Implementation

• Phase A: single-shard exact retrieval wired (graph + query + index CPU fallback) (Target: Q3 2026)
• [~] Phase B: ANN + CPU validation layer wired with advisory-only acceleration (Target: Q3 2026)
• skeleton translation units to be added with remaining implementation PRs

Phase 3: Error Handling and Edge Cases

• [~] Phase A: exact graph traversal error injection tests (Target: Q3 2026)
• [~] Phase B: ANN fallback-to-CPU on error, timeout, and validation failure (Target: Q3 2026)
• runtime failure semantics implementation and verification

Phase 4: Tests

• test_ann_frontdoor_single_shard — Phase A gate (Target: Q3 2026) ✅ PASS
• test_ann_cpu_parity_phase_b — Phase B gate (Target: Q3 2026) ✅ IMPLEMENTED 2026-08-08
  • CPU exact retrieval engine (reference implementation)
  • Mock ANN index for advisory acceleration path
  • Deterministic test fixtures with 20 vectors, 8-dimensional
  • 15 test cases covering parity, fallback, advisory mode, batch consistency
  • Category A kernels: euclidean, cosine distance
  • Acceptance thresholds: FP32 relative error < 1e-5
• BATCH 4: Hybrid Retrieval Phase A/B Parity Validation (Target: Q3 2026) — COMPLETED 2026-08-15
  • Exact-first entry criteria validation
  • 8 parity tests (HYB-01..HYB-08):
    • HYB-01: Exact match found, bypass ANN
    • HYB-02: No exact match, use ANN
    • HYB-03: Mixed dataset exact + ANN candidates
    • HYB-04: High-cardinality exact candidates
    • HYB-05: Empty exact results, ANN fallback
    • HYB-06: Concurrent exact + ANN queries (thread-safety)
    • HYB-07: Latency comparison (exact vs ANN)
    • HYB-08: Edge case NULL/empty/malformed inputs
  • File: tests/retrieval/test_retrieval_hybrid_parity_focused.cpp
  • Parity contract verified: Rank correlation ≥ 0.95 (Spearman)
  • Deterministic fixtures for reproducible testing
• [~] test_sharding_multishard_exact — Phase C gate (explicitly registered as ShardingMultiShardExactPhaseCGate; full environment validation still pending — repo-wide build blockers gpu_vector_index.cpp/incremental_view.cpp resolved 2026-08-17)
• contract and integration tests implementation in tests/epic1_retrieval/

Phase 5: Performance and Hardening

• [~] bench_ann_distance_cpu_vs_flat — Phase B benchmark gate (implemented; environment validation pending)
• [~] bench_multishard_exact — Phase C benchmark gate (deterministic benchmark hygiene tightened; full environment validation still pending — repo-wide build blockers resolved 2026-08-17)
• [~] benchmark suite implementation in benchmarks/epic1_retrieval/ (initial gates added; full validation pending)

Phase 6: Documentation and Acceptance

• acceptance docs tied to measured runtime evidence (Target: Q1 2027)

Phase 7: Production Integration

• Phase A production enable: graph + query + index (CPU-only) (2026-08-09: gate test_ann_frontdoor_single_shard ✅ PASS; single-shard exact retrieval wired)
• Phase B production enable: ANN + advisory acceleration (2026-08-09: gate test_ann_cpu_parity_phase_b ✅ IMPLEMENTED 2026-08-08; ANN + CPU validation layer wired)
• Phase C production enable: multi-shard + tensor summary (Target: Q4 2026)
• default pipeline integration enabled after gates are met

Production Readiness Checklist

LoRA Artifacts (COMPLETE)

• contract ownership and scope boundaries documented
• security and performance expectations documented for scaffold phase
• runtime behavior hardening completed (Phase 3)
• test and benchmark gates implemented and passing (55 GTest + 12 Benchmark scenarios)
• CMake targets enabled and library compiles independently
• Full Doxygen API documentation on all public types and methods

Hybrid Retrieval Rollout (IN PROGRESS)

• hybrid retrieval rollout plan mapped to module readiness (issue #5468)
• Phase A ctest gate passed (test_ann_frontdoor_single_shard)
• [~] Phase B ctest and benchmark gates implemented (validation pending)
• [~] Phase C ctest and benchmark gates implemented (validation pending)
• Phase A production enable criteria fully met (ctest + observability gates)
• Phase B production enable criteria met (ctest + benchmark gates before enabling advisory acceleration)
• Phase C production enable criteria met (multi-shard + tensor summary gates)

Known Issues and Limitations

• LoRA artifacts implementation (Phase 1-7) complete and tested
• Hybrid retrieval rollout (Phase A-D) in progress; Phase A (exact-first) complete with tests
• Phase B (ANN + CPU parity) has tests and benchmarks implemented, awaiting environment validation
• Phase C (multi-shard) has tests and benchmarks implemented, awaiting environment validation
• GPU and distributed-shard paths are not enabled until Phase C/D gates are met
• Graph Truth Layer is CPU-first and exact in all phases; no GPU acceleration for exact paths

Breaking Changes

• No roadmap-level breaking change planned. Phase transitions are additive and gated.

Program Execution Model — Wave Context

This module is a contributing module in the program-level Wave A → B → C → D execution model. It does not own a primary wave deliverable but must remain release_critical-green throughout all waves and must deliver Wave D operability improvements in Q1 2027. See [[../../ROADMAP.md|ROADMAP]] for the full wave model and exit criteria.

Wave D Contribution for retrieval

• Deliver or validate distributed tracing, high-cardinality stress coverage, exporter reliability, and operator remediation hints as applicable to this module (Target: Q1 2027)
• Contribute to or validate long-duration soak test coverage for this module's primary paths (Target: Q1 2027)
• Ensure runbook coverage for operator-critical scenarios in this module (Target: Q1 2027)

Cross-Wave Requirements

• release_critical CI must remain green on develop throughout all waves (Target: ongoing)
• p95/p99 benchmarks must be refreshed on representative hardware before Wave D sign-off (Target: Q1 2027)
• No behavioral regression may be introduced into modules in Wave A/B/C scope from changes in this module.

Program-Level Success Criteria (contribution)

• This module's distributed/acceleration paths fail closed (Target: Q1 2027)
• Benchmark-backed p95/p99 baselines exist on representative hardware (Target: Q1 2027)
• Operator-critical paths have diagnostics, alerts, and runbooks (Target: Q1 2027)