Add opt-in layered BM25 search over clustered FTS terms#31
Merged
Conversation
Member
|
Codex identified an issue with the configured stopword filter being applied before expansion. Normally these are filtered out before stemming and before they enter WHERE query_term IS NOT NULL
AND query_term <> ''Therefore, a stopword-only query such as We should probably filter raw query tokens against the stopwords table before stemming/expansion (except when |
ngrams, fuzzy search, incremental sidecar index maintenance
Member
Author
|
Thanks @Dtenwolde I fixed it. But |
Fix several query-side correctness issues in FTS macros: - filter query stopwords before stemming in match_bm25 - deduplicate layered query terms after stemming - deduplicate layered expansion candidates before applying term_limit - trim comma-separated fields filters in both BM25 macros
Dtenwolde
approved these changes
Jun 29, 2026
Merged
Member
|
I've merged the duckdb pin PR, so this should update the submodule |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds an opt-in layered_search mode to the FTS extension. The new path keeps the existing FTS occurrence storage, but adds a small dictionary-level sidecar for query-time expansion and BM25 scoring over clustered terms.
layered_search=truetoPRAGMA create_fts_index.layered_searchimplycluster_terms, so terms is physically ordered bytermid, fieldid, docid.term_statsterm_stats_by_lenterm_gramssearch_layered_bm25match_layered_bm25When
incremental=trueis used withlayered_search=true, the sidecar is maintained by triggers for inserts and deletes. The triggers update only affected terms derived from the statement's new/old rows, instead of refreshing the whole sidecar.Deletes also prune zero-df terms from the layered sidecar so removed vocabulary no longer contributes to query expansion.