First, I’d treat this as a production incident since 20% of responses are affected. I’d check dashboards and logs to confirm when it started and whether it lines up with the deploy, and also break it down by things like transcript size or customer to see if there’s a pattern.
Since there are no errors, this usually points to a parsing or validation issue rather than a hard failure. I’d compare raw model outputs between successful and failed requests to see if the score is missing, renamed, or malformed. In parallel, I’d review the recent code changes — especially prompt changes, parsing logic, and any schema updates.
To mitigate quickly, I’d likely roll back or patch the endpoint so invalid outputs don’t silently return null. After that, I’d add better logging around model responses and parsing, write a regression test from a failing case, and ship a proper fix with monitoring so we catch this earlier next time.
I’d clarify what “search by topic” actually means in practice. For example — are they expecting simple keyword matches, or more semantic queries like “agent showed empathy”? And what does a useful result look like — just the transcript, or specific highlighted segments?
Since there’s no spec, I wouldn’t overdesign upfront. I’d aim to get something usable in front of the founder quickly and iterate from there.
So my first step would be building a thin vertical slice — take a set of transcripts, chunk them, generate embeddings, and store them in a simple vector index. Then expose a basic search endpoint or internal UI that supports a few high-value queries like apologies, pricing discussions, or cancellation intent.
The goal here isn’t perfection — it’s to validate that semantic search actually solves the problem.
At the same time, I’d create a small evaluation set — maybe 25–50 transcripts with known expected matches — because without that, it’s very easy to build something that looks good but isn’t reliable.
Once we have that first version, I’d sit with the founder, review real queries, refine what “topics” mean, and improve ranking. Only after that would I invest in scaling, UX, and more advanced features.