fix(dedup): filter duplicate advisories on cosine similarity, not display rank - #1721
Open
ebarkhordar wants to merge 1 commit into
Open
Conversation
…play rank Under the default rrf fusion, relevance_score is a position in a result list rather than a similarity: _normalize_scores min-max rescales the fused scores so the top hit is always exactly 1.0. The duplicate-check advisory compared that against dedup_score_threshold, so the highest scorer cleared the filter whatever the registry held. search() now carries similarity_score, the cosine of the hit against the query embedding, on both paths that have one. DuplicateCheckService ranks and filters on it and skips hits that arrive without one. relevance_score keeps its display meaning. Refs agentic-community#1696
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DuplicateCheckServicecomparesdedup_score_thresholdagainstrelevance_score, but under the defaultrrffusion that value is a position in a result list, not a similarity._normalize_scoresmin-max rescales the fused scores, and all three of its return paths hand exactly 1.0 to the highest scorer (search_repository.py:290). Since 1.0 always clears 0.7, the filter atduplicate_check_service.py:342cannot reject the top candidate whatever the registry holds.The normalization is doing its job.
d97a24b6added it so the UI would stop rendering RRF scores as "1-3% match"; the advisory is just reading a display number as if it were a similarity.That makes the shared
-mcp-serversuffix incidental to this report. Against a registry holding onlypayroll-mcp-serverandweather-mcp-server, registeringtopic-mcp-serverstill returns an advisory at 1.0, on true cosines of 0.2648 and 0.2313.Fix
search()now carriessimilarity_scorenext torelevance_score: the cosine of the hit against the query embedding, stamped by_attach_similarity_scoreson the hybrid and client-side paths.relevance_scoreand the UI are untouched. The advisory ranks and filters onsimilarity_score, and a hit arriving without one is treated as not comparable rather than admitted.This also takes the non-default linear branch's
(cosine + 1) / 2mapping out of the threshold's meaning, where a configured 0.7 was really asking for cosine 0.4.The existing advisory fixtures set only
relevance_score, which is no longer the field that gates, so I renamed it there to keep them testing what they were testing.Verification
test_duplicate_check_service.pyfail onmainand pass here, plustest_search_similarity_scores.pyfor the helper.tests/unitsuite on Python 3.14: 6594 passed, 8 skipped.ruff check,ruff formatandmypyclean at the pinned pre-commit versions.topic-mcp-serverandcategory-mcp-serversit at 0.7621, so the boilerplate effect described in the report survives that case.Stripping catalog tokens before embedding would close that second one, but it changes what gets indexed rather than how a score is read, so it seemed worth keeping separate.
Refs #1696