feat: full-text BM25 search (1:1 with surql-rs 0.29.0)#71
Merged
Conversation
… index + fulltextSearch) Port the surql-rs v0.29.0 full-text search feature 1:1. Added: - src/schema/analyzer.ts: DEFINE ANALYZER builder (AnalyzerDefinition, Tokenizer, TokenFilter, edgeNgram/ngram/snowball, analyzer/standardAnalyzer, analyzerToSurql, validateAnalyzer) in the repo's frozen-immutable idiom. - bm25Index() convenience and bm25/highlights options on searchIndex(); IndexDefinition gains bm25?/highlights? and an optional analyzer. - generateAnalyzerSql() and an analyzers[] input to generateSchemaSql() that emits DEFINE ANALYZER ahead of the tables that reference it. - Query.fulltextSearch(field, reference, query) + Query.searchScore(reference, alias) + the fulltextSearchQuery() helper. Query text is inlined as a single-quoted, escaped literal (backslash then quote). Fixed: - Full-text index now emits the SurrealDB 3.x FULLTEXT keyword (was SEARCH, a v3 parse error) with `FULLTEXT ANALYZER <analyzer|ascii> [BM25] [HIGHLIGHTS]` in both generateTableSql/generateEdgeSql (sql.ts) and the migration differ (diff.ts). The INFO FOR TABLE parser recognises both spellings and normalises the historical `ascii` analyzer back to undefined so default-form indexes round-trip. Docs/version: - docs/v3-patterns.md gains a FULLTEXT/BM25 section (SEARCH->FULLTEXT rename; search::score returns 0 under the v3 streaming executor, so rank by scan order for RRF). README feature list updated. - CHANGELOG 1.6.0 entry; version bumped to 1.6.0 in deno.json (npm version is sourced from deno.json at build time). Gates: deno fmt (LF/CI-clean), deno lint (169 files), deno check, and deno task test (206 passed / 1782 steps; the single live-DB CLI failure is pre-existing and environment-dependent).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the full-text BM25 feature to parity with the sibling libraries (surql-rs v0.29.0, surql-py, surql-go).
Added: DEFINE ANALYZER builder (
analyzer/standardAnalyzer,Tokenizer/TokenFilter); BM25FULLTEXTindex (bm25Index,searchIndex(..., {bm25,highlights})); full-text query builder (fulltextSearch/searchScore+fulltextSearchQuery);generateAnalyzerSql.Fixed: SurrealDB 3.x renamed the full-text index keyword
SEARCH→FULLTEXT(the old form was a parse error on v3); parser accepts both. See docs/v3-patterns.md.Version 1.5.0 → 1.6.0. Gates:
deno check/deno lintclean;deno task testgreen (one pre-existing live-DB CLI test excepted). No release in this PR.