Add configurable hybrid/auto/dense retrieval to the service query path

[mahikaw]

Summary
Adds configurable hybrid retrieval to the NeMo Retriever service query path while keeping dense as the default for backward compatibility.

Service /v1/query (and MCP, which proxies to it) can now run in dense, hybrid, or auto mode via vectordb.retrieval_mode. Hybrid combines LanceDB vector search with BM25/FTS; auto inspects the table and selects dense or hybrid based on available indexes. Evidence responses report semantic for dense and semantic + lexical for hybrid.

Changes
Add vectordb.retrieval_mode to service config, bundled retriever-service.yaml, and Helm (serviceConfig.vectordb.retrievalMode)
Extend vectordb_app with --retrieval-mode and mode-aware search using the existing LanceDB.retrieval() path
Build FTS indexes on write when hybrid/auto is configured
Wire retrieval mode through the vectordb deployment and document MCP behavior (retrieval mode is server-owned)

Description

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Summary

This PR adds configurable dense, hybrid, and auto retrieval modes to the NeMo Retriever vectordb service query path, with dense kept as the default for backward compatibility. The change is end-to-end: Pydantic config, YAML bundle, Helm chart, CLI argument, and the VectorDBState runtime class all gain the new retrieval_mode knob.

• VectorDBState gains resolve_effective_retrieval_mode() and _ensure_hybrid_indexes(); the search() method now returns (hits, strategies) and delegates to LanceDB.retrieval() for both dense and hybrid paths.
• FTS index lifecycle: indexes are built on first write and on startup for existing tables in hybrid/auto mode, with a comment-documented ordering guarantee relative to _table_exists.
• Helm/YAML: serviceConfig.vectordb.retrievalMode threads through configmap and deployment args with a \"dense\" default guard.

Confidence Score: 4/5

Safe to merge after addressing the two error-handling gaps in vectordb_app.py; all other changes are additive and backward-compatible.

Two bugs exist in the startup path of VectorDBState.init: (1) _table_exists is set True before _ensure_hybrid_indexes() runs, so an FTS build failure is silently swallowed by the outer except and the service starts in a broken state; (2) _ensure_hybrid_indexes has a bare except Exception: return with no logging, hiding open-table failures entirely. Both issues are confined to vectordb_app.py; config, Helm, CLI, and test changes are clean.

nemo_retriever/src/nemo_retriever/service/vectordb_app.py — specifically the init try/except block and the bare-except in _ensure_hybrid_indexes.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/service/vectordb_app.py	Core change: adds hybrid/auto retrieval via VectorDBState.retrieval_mode, _ensure_hybrid_indexes, and resolve_effective_retrieval_mode. Two defects: (1) if FTS build raises during init, the outer catch silently swallows it while _table_exists is already True; (2) bare except Exception: return in _ensure_hybrid_indexes drops open-table failures with no logging.
nemo_retriever/src/nemo_retriever/service/config.py	Adds ServiceVectorRetrievalMode Literal type and retrieval_mode field to VectorDbConfig with a Pydantic Field and sensible default — clean, backward-compatible addition.
nemo_retriever/tests/test_service_vectordb_hybrid_integration.py	New integration test file with @pytest.mark.integration on all six tests — exercises real LanceDB I/O for FTS index building and end-to-end hybrid/auto query paths.
nemo_retriever/tests/test_service_vectordb_app.py	Adds unit tests for retrieval mode config, health reporting, hybrid strategy propagation, and 422 on missing FTS index — good coverage of new branches.
nemo_retriever/helm/templates/configmap.yaml	Threads retrievalMode into the Helm-rendered YAML configmap with a
nemo_retriever/helm/templates/deployment-vectordb.yaml	Adds --retrieval-mode CLI arg to the vectordb container spec with a sensible dense default.
nemo_retriever/helm/values.yaml	Adds retrievalMode: dense with inline comments explaining the three options — clean values addition.
nemo_retriever/src/nemo_retriever/service/cli.py	Adds --vectordb-retrieval-mode CLI override; Pydantic validation in load_config will reject invalid values.
nemo_retriever/src/nemo_retriever/service/mcp_server.py	One-line doc update clarifying that retrieval mode is server-owned — no functional change.
nemo_retriever/src/nemo_retriever/service/routers/ingest.py	Minimal summary string update to the /query endpoint — no functional change.

Comments Outside Diff (1)

1. nemo_retriever/src/nemo_retriever/service/vectordb_app.py, line 152-160 (link)

   FTS build failure silently poisons startup state

   When retrieval_mode == "hybrid", _table_exists is set to True on line 154 before _ensure_hybrid_indexes() is called on line 158. If _ensure_hybrid_indexes() raises — for example because inspect_lancedb_table_object or vdb.write_to_index throws — the outer except Exception: swallows the exception and logs "LanceDB table does not exist yet", even though _table_exists is already True. The service then starts in a broken state: health reports table_exists=True, hybrid queries immediately return 422 ("both a vector column and FTS index are required"), and the startup log gives no actionable clue. Moving _ensure_hybrid_indexes() outside the "table doesn't exist" try/except, or at minimum re-raising on FTS failure, would prevent this silent corruption of startup state.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: nemo_retriever/src/nemo_retriever/service/vectordb_app.py
   Line: 152-160
   
   Comment:
   **FTS build failure silently poisons startup state**
   
   When `retrieval_mode == "hybrid"`, `_table_exists` is set to `True` on line 154 before `_ensure_hybrid_indexes()` is called on line 158. If `_ensure_hybrid_indexes()` raises — for example because `inspect_lancedb_table_object` or `vdb.write_to_index` throws — the outer `except Exception:` swallows the exception and logs "LanceDB table does not exist yet", even though `_table_exists` is already `True`. The service then starts in a broken state: health reports `table_exists=True`, hybrid queries immediately return 422 ("both a vector column and FTS index are required"), and the startup log gives no actionable clue. Moving `_ensure_hybrid_indexes()` outside the "table doesn't exist" try/except, or at minimum re-raising on FTS failure, would prevent this silent corruption of startup state.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
nemo_retriever/src/nemo_retriever/service/vectordb_app.py:152-160
**FTS build failure silently poisons startup state**

When `retrieval_mode == "hybrid"`, `_table_exists` is set to `True` on line 154 before `_ensure_hybrid_indexes()` is called on line 158. If `_ensure_hybrid_indexes()` raises — for example because `inspect_lancedb_table_object` or `vdb.write_to_index` throws — the outer `except Exception:` swallows the exception and logs "LanceDB table does not exist yet", even though `_table_exists` is already `True`. The service then starts in a broken state: health reports `table_exists=True`, hybrid queries immediately return 422 ("both a vector column and FTS index are required"), and the startup log gives no actionable clue. Moving `_ensure_hybrid_indexes()` outside the "table doesn't exist" try/except, or at minimum re-raising on FTS failure, would prevent this silent corruption of startup state.

_{Reviews (3): Last reviewed commit: "fixes" | Re-trigger Greptile}