Better search: BM25 ranking, snippets, error surfacing - #1
Open
ezwep wants to merge 1 commit into
Open
Conversation
- Replace UNION-then-date-sort with two FTS5 queries that yield BM25
scores per matching session. Title hits get a 200pt boost so a query
match in the conversation title outranks a body-only match.
- Add a "Best match" sort option (default for active searches) and
preserve existing date/token/cost sorts when the user picks them.
- Pass the FTS5 snippet() output (with <mark> tags) back to the UI,
rendered as a highlighted excerpt under each card so users can see
why a session matches.
- Smarter query parser: multi-word input becomes implicit-AND with
prefix matching on each token; quoted phrases, NEAR, NOT, and column
filters still pass through to FTS5 verbatim.
- Surface invalid FTS syntax (e.g. unbalanced quotes) as an inline
status banner under the search input instead of silently returning
zero results.
- Return shape changed from list to {conversations, search} so the UI
can show match counts and errors. Backward-compatible array fallback
in the JS callsite.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three concrete improvements to the conversation search:
bm25(). Title hits get a 5× weight and a 200pt absolute boost so a query that matches a title outranks one that only matches body text. Old behaviour (date-sort while searching) is preserved as an option.<mark>-highlighted excerpt of the best matching message (or title/excerpt). Built on FTS5'ssnippet()function, sanitised in JS so only<mark>survives.Plus a smarter query parser:
claude code→claude* code*).NEAR,NOT, and column filters pass through unchanged so power users keep full FTS5.Code shape
get_conversations()now returns{conversations, search}instead of a bare list. The JS callsite handles both shapes (back-compat). When search is active each conversation row is annotated withsnippet_html,search_score, andmatch_kind(title,message, ortitle+message).Test plan
"hello world"matches verbatim"unclosedsurfaces error in banner, no crashsearch_scoreannotation