Skip to content

Better search: BM25 ranking, snippets, error surfacing - #1

Open
ezwep wants to merge 1 commit into
mainfrom
feat/search-bm25-snippets
Open

Better search: BM25 ranking, snippets, error surfacing#1
ezwep wants to merge 1 commit into
mainfrom
feat/search-bm25-snippets

Conversation

@ezwep

@ezwep ezwep commented Apr 27, 2026

Copy link
Copy Markdown
Owner

Summary

Three concrete improvements to the conversation search:

  • Relevance ranking via SQLite FTS5 bm25(). Title hits get a 5× weight and a 200pt absolute boost so a query that matches a title outranks one that only matches body text. Old behaviour (date-sort while searching) is preserved as an option.
  • Snippets with highlighting. Each result now shows a <mark>-highlighted excerpt of the best matching message (or title/excerpt). Built on FTS5's snippet() function, sanitised in JS so only <mark> survives.
  • Inline error reporting. Invalid FTS syntax (e.g. unbalanced quotes) used to silently return zero results; now a status banner under the search input shows the actual SQLite error and the parsed FTS query.

Plus a smarter query parser:

  • Multi-word input → implicit-AND with prefix matching on each token (e.g. claude codeclaude* code*).
  • Quoted phrases, NEAR, NOT, and column filters pass through unchanged so power users keep full FTS5.

Code shape

get_conversations() now returns {conversations, search} instead of a bare list. The JS callsite handles both shapes (back-compat). When search is active each conversation row is annotated with snippet_html, search_score, and match_kind (title, message, or title+message).

Test plan

  • Single-word search returns ranked results, snippet visible
  • Multi-word search treats words as AND-prefix
  • Quoted phrase "hello world" matches verbatim
  • Invalid syntax "unclosed surfaces error in banner, no crash
  • Empty/null search returns full list with no search_score annotation
  • Manual UI smoke test in pywebview window
  • BM25 weight tuning if title-boost feels too aggressive

- Replace UNION-then-date-sort with two FTS5 queries that yield BM25
  scores per matching session. Title hits get a 200pt boost so a query
  match in the conversation title outranks a body-only match.
- Add a "Best match" sort option (default for active searches) and
  preserve existing date/token/cost sorts when the user picks them.
- Pass the FTS5 snippet() output (with <mark> tags) back to the UI,
  rendered as a highlighted excerpt under each card so users can see
  why a session matches.
- Smarter query parser: multi-word input becomes implicit-AND with
  prefix matching on each token; quoted phrases, NEAR, NOT, and column
  filters still pass through to FTS5 verbatim.
- Surface invalid FTS syntax (e.g. unbalanced quotes) as an inline
  status banner under the search input instead of silently returning
  zero results.
- Return shape changed from list to {conversations, search} so the UI
  can show match counts and errors. Backward-compatible array fallback
  in the JS callsite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant