search relevance scoring measures length, not relevance

[tony]

Summary

search's relevance scorer has two independent defects, both verified against the shipped code:

1. It measures length, not relevance — a record containing the query verbatim can score far below a record that does not.
2. It scores against negated terms — search deploy NOT lunch ranks every survivor against the string deploy lunch, a word the engine has just guaranteed is absent from all of them.

Present in v0.1.0a33.

Defect 1 — WRatio over an already-matched set measures length

The engine AND-matches every candidate before the ranker runs, so every surviving record is guaranteed to contain the query terms. Scoring that set with WRatio — a fuzzy similarity between two whole strings — no longer asks "does this record match?" (they all do). It asks "how similar in shape is this record's entire text to the query string?", which is dominated by length.

Reproduction

from rapidfuzz import fuzz

query  = "deploy the staging cluster"
record = ("Yesterday I had to deploy the staging cluster after the rollout failed, "
          "and the whole pipeline needed a manual restart because the health checks "
          "were still pointing at the old revision, which took another twenty minutes "
          "to drain before anything could be scheduled again on the new nodes at all.")

fuzz.WRatio(query, record)         # -> 60.0
fuzz.partial_ratio(query, record)  # -> 100.0

The record contains the query verbatim. WRatio scores it 60.0. A short record that merely resembles the query string scores higher — the long, genuinely-relevant record is penalised for being long.

partial_ratio scores it 100.0, which is correct: the query is present.

fuzz.WRatio is the right tool for "are these two strings fuzzy-similar?". It is the wrong tool for "which of these known-matching records is most relevant?".

Note the docs-vs-code divergence this also closes: run_search_command's own docstring already claims partial_ratio is what is used.

Defect 2 — the relevance query descends into NOT nodes

The string handed to the ranker is " ".join(args.terms) — cli/render.py#L398.

args.terms is fed from the compiled query's text terms — cli/parser.py#L906 — and the collector that builds them deliberately descends into negation nodes — query/compile.py#L571-L572:

if isinstance(node, NotNode):
    return _collect_text_terms(node.child)

That descent is correct for its original purpose — the ripgrep prefilter benefits from knowing every term, even negated ones, as the function's own docstring explains. It is simply wrong to reuse that list as a relevance query.

Reproduction

The negated term is visible in the status line, so no instrumentation is needed:

$ agentgrep search --agent codex 'deploy NOT lunch'

Searching deploy lunch

The engine correctly excludes records containing lunch — and then scores the survivors against a string containing lunch. Capturing the ranker's argument directly confirms it:

CLI query 'deploy'            ->  ranker scored against: 'deploy'
CLI query 'deploy NOT lunch'  ->  ranker scored against: 'deploy lunch'

The negated word distorts the ranking of the very records it was used to exclude.

Suggested direction

Two options, both better than the status quo:

• Score only the positive terms, with a substring-aware scorer (partial_ratio rather than WRatio). This fixes both defects at once: the negated term is dropped from the query, and length stops dominating.
• Score the match spans the engine already computed. The matcher already knows where each term hit. Term frequency, span proximity and position are a far better relevance signal than whole-text fuzzy similarity, and they are already in hand — no second pass over the text is needed.

Where this interacts with other work

Relevance is being promoted to a first-class order="relevance" request parameter, executed inside the collector rather than as a CLI post-pass (see ADR 0014 and #100). The scorer should be fixed as part of that move, not before it — otherwise the same broken score simply gets computed one layer down, where more surfaces depend on it.

It matters more once it moves. Today only the CLI ranks. Once order="relevance" exists, MCP clients get it too — and an agent acting on a "most relevant" list that is really a "shortest" list is a worse failure than a human noticing the order looks odd.

Related: #113 (--limit caps before the ranker runs, so the best match is discarded before this scorer would have mis-ranked it — the two defects compound).

Goal

Idempotent completion condition for /goal. It asserts an end state, not an action: if the state already holds, the first evaluation passes and nothing is edited.

/goal Work only on branch issue-115-relevance in a worktree cut from master (create if absent), not the shared master checkout; run pytest as NO_COLOR=1 env -u VIRTUAL_ENV -u UV_NO_SYNC .venv/bin/python -m pytest. First report git rev-parse --abbrev-ref HEAD and that pytest on tests/test_relevance_scoring.py, which exits 4 today (absent); if it exits 0, stop with no edits. Baseline: rank_search_records uses rapidfuzz WRatio over " ".join(args.terms). End state: the score measures relevance, not length, and never scores a record against a term the query excluded. tests/test_relevance_scoring.py passes, asserting a long record containing the query verbatim outranks a short record that merely resembles it, and that for deploy NOT lunch the ranker's text contains deploy and not lunch. agentgrep.ranking keeps one named relevance entry point whose docstring states the scorer and policy version, and run_search_command's docstring in cli/render.py names that scorer. Constraints: _collect_text_terms in query/compile.py keeps descending into NotNode, since the prefilter needs every term; derive the ranking query from a separate positive-terms-only accessor rather than mutating SearchArgs.terms or SearchQuery.terms, so matching and membership are unchanged and only score and order move; no unrelated refactor, no commit or push. Non-goals: an order=relevance parameter, ranking in the collector, and #113's ordered-limit defect - do not wait on it. Seams: the score stays a replaceable versioned policy, not a frozen public number; the CLI is not the only future ranking surface; pin no cursor or id encoding. Proof: that module green plus NO_COLOR=1 just test and uv run ty check exiting 0 (move any untracked scripts/benchmark.local.toml aside). Or stop after 10 turns and report what remains.

• Proof — git rev-parse --abbrev-ref HEAD && NO_COLOR=1 env -u VIRTUAL_ENV -u UV_NO_SYNC .venv/bin/python -m pytest tests/test_relevance_scoring.py -q
• Idempotence — Re-running finds the two scorer assertions already green in the named worktree, so turn one ends the goal with no edit; on master the module is absent and pytest exits 4.
• Blocked by — nothing; both defects are self-contained in ranking.py and cli/render.py, and search --limit discards the best results (limit caps in scan order, not result order) #113 is named as an explicit do-not-wait non-goal
• Blocks — nothing; it adds no gate and keeps the single relevance entry point in place for Find similar prompts and conversations #82's separate similarity scorer
• Preserves — the score stays a replaceable versioned policy rather than a frozen public number, so a later collector-executed order can adopt it; _collect_text_terms keeps descending into NotNode so the ripgrep prefilter and engine matching are untouched; SearchArgs.terms and SearchQuery.terms keep their meaning, leaving search --limit discards the best results (limit caps in scan order, not result order) #113 and Backfill prompt sessions for origin filters #98 unaffected; no cursor or id encoding is pinned and the CLI is not asserted to be the only ranking surface