Funky matching of numbers against text with apostrophes in porter stemer

[StephenFlavin]

it seems that FTS search for numeric strings will always match values with apostrophes (maybe other special chars).

how to replicate

setup:

create table foo (id int primary key, info text not null);

insert into foo values (1, 'foo'), (2, 'bar''s');

pragma create_fts_index('foo', 'id', 'info', stopwords = 'none'); -- default stemmer (porter)

select *, fts_main_foo.match_bm25(id, '0') from foo;

result

┌───────┬─────────┬──────────────────────────────────┐
│  id   │  info   │ fts_main_foo.match_bm25(id, '0') │
│ int32 │ varchar │              double              │
├───────┼─────────┼──────────────────────────────────┤
│     1 │ foo     │                             NULL │
│     2 │ bar's   │               0.2649063961843035 │
└───────┴─────────┴──────────────────────────────────┘

you get the same result for any number in the query e.g. 0123456789

┌───────┬─────────┬───────────────────────────────────────────┐
│  id   │  info   │ fts_main_foo.match_bm25(id, '0123456789') │
│ int32 │ varchar │                  double                   │
├───────┼─────────┼───────────────────────────────────────────┤
│     1 │ foo     │                                      NULL │
│     2 │ bar's   │                        0.2649063961843035 │
└───────┴─────────┴───────────────────────────────────────────┘

similarly if your query contains a number then some text you get the same match score

select *, fts_main_foo.match_bm25(id, '0foo') from foo;
┌───────┬─────────┬─────────────────────────────────────┐
│  id   │  info   │ fts_main_foo.match_bm25(id, '0foo') │
│ int32 │ varchar │               double                │
├───────┼─────────┼─────────────────────────────────────┤
│     1 │ foo     │                  0.3485610476109256 │
│     2 │ bar's   │                  0.2649063961843035 │
└───────┴─────────┴─────────────────────────────────────┘

tokeniser results

select fts_main_foo.tokenize('0');

┌────────────────────────────┐
│ fts_main_foo.tokenize('0') │
│         varchar[]          │
├────────────────────────────┤
│ [, ]                       │
└────────────────────────────┘

(elements are empty strings)
and

┌───────────────────────────────┐
│ fts_main_foo.tokenize('0foo') │
│           varchar[]           │
├───────────────────────────────┤
│ [, foo]                       │
└───────────────────────────────┘

(first element is an empty string)

[StephenFlavin]

this issue doesn't happen when I set the stemmer e.g.

pragma create_fts_index('foo', 'id', 'info', stemmer = 'none', stopwords = 'none');

or

pragma create_fts_index('foo', 'id', 'info', stemmer = 'english', stopwords = 'none');