Fix L1 quality for multi-retriever - #1000
Conversation
swethakann
changed the title
| long rescoreStart = System.nanoTime(); | ||
| topDocs = retrieverContext.getRescoreTask().rescore(topDocs, searchContext); | ||
| rescoreTimeMs = (System.nanoTime() - rescoreStart) / 1_000_000.0; | ||
| int topHits = retrieverContext.getTopHits(); |
There was a problem hiding this comment.
Why do we add this truncation here? we don't do this in the default path, I don't see particular reason why this shall be different.
There was a problem hiding this comment.
I went through the code again. We do slice based on the topHits in default path. Then we can reuse this method getHitsFromOffset(topDocs, 0, retriever.topHits)
| } | ||
|
|
||
| @Test | ||
| public void testL1RescorerTruncatesToTopHitsBeforeBlend() { |
There was a problem hiding this comment.
Ain't the two tests the same
There was a problem hiding this comment.
we may to do set up a stricter test:
topHits 3, rescorer window 10. docs ranked 8-10 are rescored to the top so they end up showing the results. This cannot be the case if only 3 got rescored
There was a problem hiding this comment.
I have combined the tests in the follow up commit if that makes it easier for you.
We don't need the quality check at the moment imo. Re-ranking at L1 is done by inference plugin in prod so we only need a functionality check. I had verified the inference code does score descending yesterday before this change. So that's good.
There was a problem hiding this comment.
I think the gap missed testing is:
all docs: big number - firstPassSearch(topHits:3) -> filter docs: 10 (verify this, not 3) -> rescorer(window: 10) -> blender&response 3
There was a problem hiding this comment.
After checking the code, the change is pretty trivial but the test set up was complex. Instead captured it in a simple method that is easy to isolate for testing.
Fix hits collected for retrievers
This reverts commit 041d52e.
Fix hits collected for retrievers
3 participants
When a multi-retriever has an L1 rescorer with windowSize > topHits, expand collection to windowSize so the rescorer sees the full candidate pool, then truncate back to topHits after rescoring (keeping the best N by L1 score) before feeding to the blender.