Current State
We have a simple substring matching with SQLite's LIKE keyword.
Possible Solution
The SQLite FTS5 Extension with its trigram tokenizer looks like a suitable option.
It should be able to tokenize text of all UTF-8 characters, not just ASCII.
Issues to Solve
Currently in the get_news_with_filter method we are using FirstFetchedAt as a cursor, however, with FTS5 we will get a ranked list of best matches first, so we need to use a different type of cursor and a special pagination for news searching.
Open Problems
Highlighting search matches in the displayed results is great for UX, but I'm not sure if this is Aleym Core's issue or the presenter's. If it was the presenter's then it will need an identical algorithm to the one used by FTS5 and it will do some redundant calculations that should be cheap.
Current State
We have a simple substring matching with SQLite's
LIKEkeyword.Possible Solution
The SQLite FTS5 Extension with its
trigramtokenizer looks like a suitable option.It should be able to tokenize text of all UTF-8 characters, not just ASCII.
Issues to Solve
Currently in the
get_news_with_filtermethod we are usingFirstFetchedAtas a cursor, however, with FTS5 we will get a ranked list of best matches first, so we need to use a different type of cursor and a special pagination for news searching.Open Problems
Highlighting search matches in the displayed results is great for UX, but I'm not sure if this is Aleym Core's issue or the presenter's. If it was the presenter's then it will need an identical algorithm to the one used by FTS5 and it will do some redundant calculations that should be cheap.