Summary
In /api/chat, the full message history is sent to the model, but retrieval — both the query-type router and the pgvector search — only ever embeds the last message. Any follow-up that depends on earlier context retrieves nothing useful, and the assistant then answers "I don't have sufficient information" — even for something it answered one turn earlier.
Steps to reproduce
- Ask: "Who is the MNA for NA-125?" → correct answer with the representative's details.
- Follow up: "What's his phone number?"
- The follow-up fails with a no-information response, even though the contact info is in the database.
Expected
The follow-up resolves against the representative from the previous turn and returns the contact info already present in the DB.
Actual
The second turn embeds only "What's his phone number?" — no name, no constituency — so the cosine search finds nothing above the threshold, the assembled context is empty, and per the system prompt's "answer EXCLUSIVELY from the provided information" rule, the model says it lacks the information.
Root cause
In app/api/chat/route.tsx:
const lastMessage = messages[messages.length - 1]
const queryTypes = await detectQueryTypes(lastMessage.content)
// findRelevantContent(lastMessage.content) / findRelevantRepresentatives(lastMessage.content)
messages (full history) is passed to streamText, but every retrieval call in lib/ai/embedding.ts (detectQueryTypes, findRelevantContent, findRelevantRepresentatives) receives only lastMessage.content. Follow-ups that use pronouns or ellipsis therefore embed a query with no retrievable signal.
Suggested fix
Build the retrieval query from the last few user turns (e.g. the last 3), and use that string for routing and vector search — while still passing the full messages to streamText unchanged. No extra model calls, so no added latency or cost.
A more robust follow-up (separate change) would add a lightweight "condense question" step, reusing the existing detectQueryTypes LLM call to also return a standalone rewritten query, for cases where the antecedent lives in the assistant's reply rather than a user message.
Scope
Retrieval only. Input validation, rate-limiting, and error handling on this endpoint are separate concerns.
Happy to open a PR for the retrieval fix if that's welcome.
Summary
In
/api/chat, the full message history is sent to the model, but retrieval — both the query-type router and the pgvector search — only ever embeds the last message. Any follow-up that depends on earlier context retrieves nothing useful, and the assistant then answers "I don't have sufficient information" — even for something it answered one turn earlier.Steps to reproduce
Expected
The follow-up resolves against the representative from the previous turn and returns the contact info already present in the DB.
Actual
The second turn embeds only
"What's his phone number?"— no name, no constituency — so the cosine search finds nothing above the threshold, the assembled context is empty, and per the system prompt's "answer EXCLUSIVELY from the provided information" rule, the model says it lacks the information.Root cause
In
app/api/chat/route.tsx:messages(full history) is passed tostreamText, but every retrieval call inlib/ai/embedding.ts(detectQueryTypes,findRelevantContent,findRelevantRepresentatives) receives onlylastMessage.content. Follow-ups that use pronouns or ellipsis therefore embed a query with no retrievable signal.Suggested fix
Build the retrieval query from the last few user turns (e.g. the last 3), and use that string for routing and vector search — while still passing the full
messagestostreamTextunchanged. No extra model calls, so no added latency or cost.A more robust follow-up (separate change) would add a lightweight "condense question" step, reusing the existing
detectQueryTypesLLM call to also return a standalone rewritten query, for cases where the antecedent lives in the assistant's reply rather than a user message.Scope
Retrieval only. Input validation, rate-limiting, and error handling on this endpoint are separate concerns.
Happy to open a PR for the retrieval fix if that's welcome.