Skip to content

Add opt-in layered BM25 search over clustered FTS terms#31

Merged
lnkuiper merged 9 commits into
duckdb:mainfrom
kryonix:multi_layer_fts
Jul 2, 2026
Merged

Add opt-in layered BM25 search over clustered FTS terms#31
lnkuiper merged 9 commits into
duckdb:mainfrom
kryonix:multi_layer_fts

Conversation

@kryonix

@kryonix kryonix commented Jun 26, 2026

Copy link
Copy Markdown
Member

This PR adds an opt-in layered_search mode to the FTS extension. The new path keeps the existing FTS occurrence storage, but adds a small dictionary-level sidecar for query-time expansion and BM25 scoring over clustered terms.

  • Adds layered_search=true to PRAGMA create_fts_index.
  • Makes layered_search imply cluster_terms, so terms is physically ordered by termid, fieldid, docid.
  • Builds extension-owned dictionary sidecar tables:
    • term_stats
    • term_stats_by_len
    • term_grams
  • Adds layered search macros:
    • search_layered_bm25
    • match_layered_bm25
  • Supports exact, prefix, substring, and fuzzy dictionary expansion.
  • Keeps numeric terms exact-searchable, but excludes them from trigram expansion.
  • Keeps this path postings-free; scoring still uses the existing FTS terms table.

When incremental=true is used with layered_search=true, the sidecar is maintained by triggers for inserts and deletes. The triggers update only affected terms derived from the statement's new/old rows, instead of refreshing the whole sidecar.

Deletes also prune zero-df terms from the layered sidecar so removed vocabulary no longer contributes to query expansion.

@kryonix kryonix requested a review from Dtenwolde June 26, 2026 10:32
@Dtenwolde

Copy link
Copy Markdown
Member

Codex identified an issue with the configured stopword filter being applied before expansion. Normally these are filtered out before stemming and before they enter dict/term_stats. But search_layered_bm25 only removes NULL and empty strings:

WHERE query_term IS NOT NULL
        AND query_term <> ''

Therefore, a stopword-only query such as the can slip through this and fuzzy-expand into terms like theme (eventhough the was supposed to be excluded from the index).

We should probably filter raw query tokens against the stopwords table before stemming/expansion (except when stopwords='none').

@kryonix

kryonix commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

Thanks @Dtenwolde I fixed it. But main moved again, so I need to update the submodule first for CI.

kryonix added 3 commits June 29, 2026 13:34
Fix several query-side correctness issues in FTS macros:

- filter query stopwords before stemming in match_bm25
- deduplicate layered query terms after stemming
- deduplicate layered expansion candidates before applying term_limit
- trim comma-separated fields filters in both BM25 macros
@Dtenwolde Dtenwolde mentioned this pull request Jul 2, 2026
@lnkuiper

lnkuiper commented Jul 2, 2026

Copy link
Copy Markdown
Member

I've merged the duckdb pin PR, so this should update the submodule

@kryonix kryonix force-pushed the multi_layer_fts branch from 413acf3 to 600bf3e Compare July 2, 2026 11:42
@lnkuiper lnkuiper merged commit dc45d7b into duckdb:main Jul 2, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants