Opensearch compatible tokenization

[Dtenwolde]

Part of: https://github.com/duckdblabs/duckdb-internal/issues/9855

This PR adds Opensearch-compatible tokenizer mode:

  PRAGMA create_fts_index(
      'docs',
      'id',
      'body',
      tokenizer = 'opensearch_standard'
  );

The current regex tokenizer mode remains the standard for backwards compatibility reasons.

regex mode works with Unicode already, but is not compatible with Opensearch/Lucene tokenization, mostly w.r.t. to Japanese/Chinese alphabets. In particular, the current tokenizer keeps CJK runs such as 東京大学に行く or 北京大学 as whole tokens, while OpenSearch’s standard tokenizer emits Han and Hiragana characters as individual tokens.

The new opensearch_standard mode uses the ICU extension to align with OpenSearch:

• Han and Hiragana characters are emitted as single-character tokens.
• Katakana, Hangul, Latin, Hebrew, Cyrillic, Arabic, and numeric text are preserved as word/token runs.
• Common intra-token punctuation is handled for cases like john's, example.com, foo_bar, snake_case, and 3.14.

And because I am curious, emoji's are now also better handled than with the regex mode:

  hello🙂world 中文🙂test

is tokenized more like:

  hello, 🙂, world, 中, 文, 🙂, test

We're not fully Opensearch compatible just yet. That requires a couple more pieces, in particular around folding/normalization mode and strip_accents, but that will be addressed in a follow-up PR.

[lnkuiper]

Thanks! Just two minor comments:

[lnkuiper]

This only treats ASCII ' as intra-token punctuation. As a result, common text such as It’s and l’amour is split into It, s / l, amour, whereas OpenSearch’s standard tokenizer preserves It’s as one token. Could we implement the relevant UAX #29 apostrophe handling (at least U+2019), and add coverage for it?

# name: test/temp.test
# group: [fts]

require fts

query T
SELECT token
FROM unnest(fts_tokenize_opensearch_standard('It’s')) AS t(token)
----
It’s

[lnkuiper]

This compiles against ICU headers from FTS’s pinned duckdb submodule, but below we link duckdb_icu_* object targets from the host DuckDB build. That mixes two DuckDB/ICU revisions with no compatibility guarantee. Can we source both headers and objects consistently from the host build (or consistently from the submodule)?