[imthi16]

Summary

Refactored the RAG pipeline to be fully async, added thread-safety to FAISS operations, implemented proper L2-to-cosine similarity conversion, and improved retrieval quality with configurable score thresholds.

Key Changes

Core Pipeline & Async Support

• Made all agent nodes (router, retriever, analyst, critic, formatter) async functions
• Wrapped blocking FAISS and embedding operations with asyncio.to_thread() to keep the event loop responsive
• Added route_after_retrieval() conditional edge to short-circuit the pipeline when no relevant chunks are found, returning a user-friendly message instead of asking the analyst to synthesize from empty context

FAISS Thread Safety

• Added threading.Lock() to VectorStoreService to guard all mutations and reads of the FAISS index and docstore
• Extracted _save_locked() internal method for callers that already hold the lock
• All public methods (add_documents, similarity_search, delete_by_doc_id, save, get_stats) now acquire the lock before accessing the vectorstore

Retrieval Scoring & Quality

• Implemented l2_squared_to_cosine() function to convert FAISS squared-L2 distances to cosine similarity in [0, 1]
  • Formula: cos(a, b) = 1 - score / 2 for L2-normalized vectors
  • Clamps negative similarities to 0 (unrelated chunks)
• Added configurable RETRIEVAL_SCORE_THRESHOLD (default 0.2) to filter low-confidence chunks before passing to analyst
• Added configurable RETRIEVAL_K (default 5) for number of chunks to retrieve
• Updated similarity_search() to apply threshold filtering and return converted similarity scores

LLM & JSON Handling

• Added json_mode parameter to get_llm() to enable Ollama's JSON output format constraint
• Implemented parse_json_response() utility to robustly parse model responses with optional markdown fences
• Updated router, analyst, and critic nodes to use json_mode=True for structured outputs
• Formatter node left without json_mode for free-form markdown generation
• Added LLM instance caching via @lru_cache to avoid rebuilding clients on every invocation

Document Deletion

• Implemented proper delete_by_doc_id() by iterating the docstore and deleting all chunks matching the doc_id
• Previously was a no-op; now fully functional with logging

Revision Loop Improvements

• Analyst node now receives previous critique and analysis during revision passes
• Includes explicit feedback block in the prompt so the analyst can address specific issues
• Critic node updated with clearer instructions to provide actionable feedback when confidence is low

Testing & Code Quality

• Added test_retrieval.py with unit tests for L2-to-cosine conversion, text chunking, and JSON parsing
• Added conftest.py to ensure pytest can import the app package
• Improved logging throughout (replaced print() with structured logger calls)
• Fixed timestamp generation to use timezone.utc instead of deprecated utcnow()
• Code formatting and import organization improvements

Configuration

• Added RETRIEVAL_K and RETRIEVAL_SCORE_THRESHOLD to settings and .env.example
• Removed unused GOOGLE_API_KEY from CI workflow

Notable Implementation Details

• The L2-to-cosine conversion is mathematically sound for L2-normalized embeddings (which HuggingFace embeddings provide by default)
• Thread safety is critical because the async pipeline now calls FAISS from worker threads via asyncio.to_thread()
• The retriever short-circuits the pipeline early if no chunks pass the threshold, avoiding wasted LLM calls and providing immediate user feedback
• JSON mode in Ollama removes the fragile "strip markdown and hope" parsing pattern, making outputs more reliable

https://claude.ai/code/session_013WDZTHsDvmhUax6fydeLwx