Category: Retrieval Performance
Priority: High
Phase: Phase 6–7 (LangGraph + HITL)
Problem
Rerank can fire multiple times per LangGraph turn (tool calls, HITL resume, re-queries), so cross-encoder cost compounds per turn, not per query.
Solution
Enforce a per-turn retrieval-call cap as a LangGraph state field; cache rerank scores within a turn keyed on (query, chunk_id); keep model loading out of the request path.
Category: Retrieval Performance
Priority: High
Phase: Phase 6–7 (LangGraph + HITL)
Problem
Rerank can fire multiple times per LangGraph turn (tool calls, HITL resume, re-queries), so cross-encoder cost compounds per turn, not per query.
Solution
Enforce a per-turn retrieval-call cap as a LangGraph state field; cache rerank scores within a turn keyed on
(query, chunk_id); keep model loading out of the request path.