Description
evaluate_rag_coverage() only checks whether a requirement retrieved any chunk, not
whether the chunk is actually a good semantic match. If chunking degrades (see #1), every
requirement "matches" the one giant chunk, producing a misleadingly perfect coverage score.
Suggested approach
Add a similarity-score threshold to retrieve_relevant_experience() and have the eval report
count a requirement as "covered" only if its best match is above that threshold, not just
present.
Description
evaluate_rag_coverage()only checks whether a requirement retrieved any chunk, notwhether the chunk is actually a good semantic match. If chunking degrades (see #1), every
requirement "matches" the one giant chunk, producing a misleadingly perfect coverage score.
Suggested approach
Add a similarity-score threshold to
retrieve_relevant_experience()and have the eval reportcount a requirement as "covered" only if its best match is above that threshold, not just
present.