columnar: pin CSE FTS Spec v1 on cloud-engine

[3pointer]

What problem does this PR solve?

Issue Number: N/A

Problem Summary:

TiDB/CSE FTS Spec v1 uses full_text_index.index_fields as the canonical ordered physical field layout. CSE now owns parsing, validation, and conversion of that metadata. This PR now pins the latest fts_spec_v1 after its cloud-engine rebase (36128849380c), including match-expression query protocol support from tidbcloud/cloud-storage-engine#5801.

The production TiFlash path builds CSE through contrib/tiflash-columnar-hub, so TiFlash only needs to pin that CSE revision. TiFlash C++ does not need to parse or validate the FTS fields.

What is changed and how it works?

columnar: pin CSE FTS Spec v1 on cloud-engine

• Bump contrib/cloud-storage-engine to 36128849380c, the latest fts_spec_v1 head after the cloud-engine rebase.
• Align the columnar-hub workspace's Tantivy patch with CSE's pingcap/tantivy:cse-0.22.1 and refresh Cargo.lock. This parent-workspace patch is required because Cargo ignores a child workspace's [patch.crates-io] when CSE is built by columnar hub.
• Keep TiFlash C++ unchanged. Schema parsing, Analyzer validation, ordered/repeated column_id handling, Analyzer parameters, and enable_bm25 remain in CSE.

Metadata compatibility:

• Spec v1 canonical metadata is ordered index_fields; old filter_columns and transitional fields are not used as the physical layout.
• Legacy parser_type remains readable in CSE for rolling upgrades of existing single-field indexes.

Deployment order:

1. Land and deploy the CSE change and this TiFlash/columnar-hub pin.
2. Only then allow TiDB to emit new multi-field FTS DDL.

Check List

Tests

• Unit test
• Integration test
• Manual test (details below)
• No code

Completed locally on Apple arm64 (revalidated with CSE 36128849380c):

• cargo check --locked --manifest-path hub-runtime/Cargo.toml
• make debug, producing an arm64 libtiflash_proxy.dylib
• cargo fmt --all -- --check

Pending:

• TiFlash X64/ARM64 CI builds.
• End-to-end round-trip using TableInfo JSON emitted by a real TiDB instance.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Maintenance
  • Updated the cloud storage engine component reference.
  • Updated the Tantivy package source used by the TiFlash columnar component.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (1)

• contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b27b0072-9af7-4162-8e12-20cba28c8cbc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e7d9e38b-809b-4a45-9ec9-ec8f4306c11c

📥 Commits

Reviewing files that changed from the base of the PR and between 8541997 and 320bb19.

⛔ Files ignored due to path filters (1)

• contrib/tiflash-columnar-hub/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (2)

• contrib/cloud-storage-engine
• contrib/tiflash-columnar-hub/Cargo.toml

Included review availability: Your plan provides up to 4 included reviews per hour; 2 remain after this review.

---

📝 Walkthrough

Walkthrough

The change updates the cloud-storage-engine submodule commit and changes the patched tantivy repository and branch in the tiflash-columnar-hub Cargo configuration.

Changes

Dependency and submodule updates

Layer / File(s)	Summary
Submodule reference update contrib/cloud-storage-engine	The submodule pointer changes to commit 9cbffb66ae79b72212cef4622b60d0c5e766856b.
Tantivy patch configuration contrib/tiflash-columnar-hub/Cargo.toml	The tantivy patch source changes to pingcap/tantivy branch cse-0.22.1.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: 🔵 Low · up to 320bb

The PR pins a new CSE revision and refreshes shared Rust dependency metadata; it is mergeable with owner awareness, but the private submodule commit and OpenSSL lockfile alignment should be confirmed because a mismatch could fail the columnar compatibility check.

Suggested reviewers: jayson-huang, yongman

Poem

A rabbit hops through crates so bright,
A submodule points to newer light.
Tantivy follows a fresh branch,
Two tiny updates in one quick dash.
“Review complete!” the rabbit sings.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the primary change: pinning the CSE FTS Spec v1 on the cloud-engine.
Description check	✅ Passed	The description includes all required template sections, explains the changes, lists validation results, and documents deployment and compatibility considerations.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

@glorv: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[glorv]

@yongman @JaySon-Huang PTAL

[JaySon-Huang]

@glorv @3pointer pls fix the compile error under nextgen columnar compile option

error: cannot update the lock file /home/jenkins/agent/workspace/pingcap/tiflash/pull_integration_next_gen_columnar/tiflash/contrib/tiflash-columnar-hub/Cargo.lock because --locked was passed to prevent this
help: to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.
make: *** [Makefile:36: debug] Error 101

[JaySon-Huang]

@3pointer and create an issue for the behavior changes pls

[yongman]

Please add compatible test result or report for legacy fts data.

[JinheLin]

Legacy FTS data compatibility report

FTS files created by the old single-column stack remain readable and writable after rolling TiFlash → CSE → TiDB to Spec v1, after turning off legacy field layout, after creating new multi-column indexes in the same cluster, and after a full process restart. The application query fts_match_word(token, content) does not need to change.

Scope

This report is about on-disk legacy FTS data, not about new multi-column DDL.

• Legacy data: a single-column STANDARD FULLTEXT index created and persisted on the old stack (old TiDB + old CSE v26.3.8 + old TiFlash). Physical layout is the pre–Spec v1 "body" field.
• Question: after this PR’s Spec v1 CSE pin is in the read path, can those files still be queried and maintained without rebuild?

Upgrade order

Same order as the PR description: land CSE + TiFlash first, only then let TiDB emit multi-field FTS DDL.

Stage	TiDB	TiFlash	CSE	What happens to the legacy index
S0	old	old	old	Create table, insert 100,000 rows, persist FTS files
S1	old	new	old	Rolling TiFlash only; read + insert/update/delete, restore baseline
S2	old	new	new	Rolling CSE; same R/W cycle
S3	new	new	new	Rolling TiDB; same R/W cycle
LAYOUT_SWITCH	new	new	new	fts-new-index-use-legacy-field-layout=false; R/W again
SPEC_V1	new	new	new	Create new multi-analyzer indexes; R/W legacy table after each
LEGACY_SINGLE	new	new	new	Add a multi-column peer table; ADD a second STANDARD index on the same content column; bulk UPDATE/INSERT/DELETE
Restart	new	new	new	Stop the cluster, bring S3 binaries back

Legacy table

Created only at S0; never dropped; idx_legacy_content never rebuilt.

CREATE TABLE fts_legacy_single (
  id BIGINT PRIMARY KEY,
  bucket INT NOT NULL,
  content TEXT NOT NULL,
  FULLTEXT INDEX idx_legacy_content(content) WITH PARSER STANDARD
);

• id = 1 … 100000, content = 'legacytok{id%1000} document number {id}'
• Probe: SELECT COUNT(*) FROM fts_legacy_single WHERE fts_match_word('legacytok42', content)
• Expected hits while baseline is intact: 100

EXPLAIN after restart still uses the single column (not rewritten to a multi-field expression):

TableFullScan table:fts_legacy_single
  textSearch:(legacytok42 IN fts_compat_test.content)

A smaller single-column table (fts_documents) carried generational rows: D0 written at S0, D2 at S1, D1 at S2, D3 at S3. All generations stayed visible on later stages and after restart.

Observed query results

Probe SQL used at every stage:

SELECT COUNT(*) FROM fts_legacy_single;                                          -- rows
SELECT COUNT(*) FROM fts_legacy_single WHERE fts_match_word('legacytok42', content);  -- MATCH

Expected MATCH while the S0 baseline is intact: (100000 - 42 - 1) / 1000 + 1 = 100.
Numbers below are live TiFlash/TiDB results recorded in legacy-single-*.tsv / wait_query (the case dies unless the cell matches).

1. Row count and MATCH hits per stage

Stage	COUNT(*) expected	COUNT(*) actual	MATCH legacytok42 expected	MATCH actual	Source
S0 seed	100000	100000	100	100	manifest/legacy-single-baseline.env; Region=28
S1 before I/U/D	100000	100000	100	100	S1/results/legacy-single-rw.tsv (pre-cycle assert)
S1 after I/U/D restore	100000	100000	100	100	same file probe_hits=100
S2 before / after restore	100000	100000	100	100	S2/results/legacy-single-rw.tsv
S3 before / after restore	100000	100000	100	100	S3/results/legacy-single-rw.tsv label=s3
LAYOUT_SWITCH before / after restore	100000	100000	100	100	LAYOUT_SWITCH/results/legacy-single-rw.tsv
After NGRAM	100000	100000	100	100	S3/results/legacy-single-rw.tsv label=ngram
After SPEC_V1	100000	100000	100	100	SPEC_V1/results/legacy-single-rw.tsv
After MEM9	100000	100000	100	100	MEM9_FTS/results/legacy-single-rw.tsv
LEGACY_READ_BEFORE_MULTI	100000	100000	100	100	read-before-multi.tsv probe_hits=100 topk_rows=10
After multi-column peer table	100000	100000	100	100	timing LEGACY_READ_AFTER_MULTI hits=100
After second STANDARD on same content	100000	100000	100	100	timing hits=100; indexes idx_legacy_content, idx_legacy_content_v2
After bulk UPDATE (5 rows)	100000	100000	95	95	update-after-multi.tsv updated=5 old_token_hits=95
After INSERT 500	100500	100500	96	96	timing total=100500; id 100042 is bucket 42 → +1 MATCH
After DELETE id<=50	100450	100450	96	96	final-snapshot.tsv rows=100450 probe_hits=96
After full restart	100450	100450	96	96	Restart/results/legacy-single-restart.tsv

TopK at LEGACY_READ_BEFORE_MULTI (actual 10 ids, all bucket=42): 2042, 3042, 4042, 5042, 6042, 7042, 8042, 9042, 10042, 52042.

Restart EXPLAIN still scans the post-write table (estRows=100450) with single-column textSearch:

TableFullScan table:fts_legacy_single
  textSearch:(legacytok42 IN fts_compat_test.content)

Restart Region ID = 3001 (was 28 at S0). Both FTS index names still present: idx_legacy_content, idx_legacy_content_v2.

2. Rolling I/U/D on the legacy index (one throwaway row per stage)

Each cycle: INSERT one row → UPDATE its token → DELETE it → restore baseline. MATCH is SELECT id FROM fts_legacy_single WHERE fts_match_word('<token>', content).

Stage	id	INSERT MATCH new token	UPDATE MATCH new token	UPDATE MATCH old token	DELETE MATCH new token	Restore rows	Restore MATCH legacytok42
S1	800000001	legacyrws1 → 800000001 (empty→1)	legacyrws1after → 800000001	legacyrws1 → empty	legacyrws1after → empty (1→0)	100000	100
S2	800000002	legacyrws2 → 800000002	legacyrws2after → 800000002	legacyrws2 → empty	legacyrws2after → empty	100000	100
S3	800000003	legacyrws3 → 800000003	legacyrws3after → 800000003	legacyrws3 → empty	legacyrws3after → empty	100000	100
LAYOUT_SWITCH	800000004	legacyrwlayout → 800000004	legacyrwlayoutafter → 800000004	legacyrwlayout → empty	legacyrwlayoutafter → empty	100000	100
NGRAM	800000007	legacyrwngram → 800000007	legacyrwngramafter → 800000007	legacyrwngram → empty	legacyrwngramafter → empty	100000	100
SPEC_V1	800000005	legacyrwspec → 800000005	legacyrwspecafter → 800000005	legacyrwspec → empty	legacyrwspecafter → empty	100000	100
MEM9	800000006	legacyrwmem9 → 800000006	legacyrwmem9after → 800000006	legacyrwmem9 → empty	legacyrwmem9after → empty	100000	100

Example S1 statements (from S1/sql/executed.sql):

INSERT INTO fts_legacy_single VALUES (800000001,999,'legacyrws1 unique document 800000001');
SELECT id FROM fts_legacy_single WHERE fts_match_word('legacyrws1',content);          -- actual 800000001
UPDATE fts_legacy_single SET content='legacyrws1after unique document 800000001' WHERE id=800000001;
SELECT id FROM fts_legacy_single WHERE fts_match_word('legacyrws1after',content);     -- actual 800000001
SELECT id FROM fts_legacy_single WHERE fts_match_word('legacyrws1',content);          -- actual empty
DELETE FROM fts_legacy_single WHERE id=800000001;
SELECT id FROM fts_legacy_single WHERE fts_match_word('legacyrws1after',content);     -- actual empty

3. Bulk write after multi-column indexes exist (not restored)

Step	SQL	Actual
UPDATE 5 baseline rows with bucket=42 and id<=5000	rewrite content to legacytok42changed …	updated=5; MATCH legacytok42 100 → 95
INSERT ids 100001–100500	legacytok{id%1000} inserted tail {id}	COUNT(*) 100000 → 100500; MATCH 95 → 96 (id 100042 is bucket 42)
DELETE id<=50	remove 50 rows	COUNT(*) 100500 → 100450; MATCH stays 96 (id 42 was already rewritten, not a legacytok42 hit)
Peer table (control)	fts_match_word('peeralpha', col_a)	9001

4. Restart (stop cluster, bring S3 binaries back)

Metric	Before restart (final snapshot)	After restart	File
COUNT(*)	100450	100450	Restart/results/legacy-single-restart.tsv
MATCH legacytok42	96	96	same
FTS indexes	idx_legacy_content, idx_legacy_content_v2	same two names	Restart/results/legacy-fts-index-names.tsv
textSearch	single-column content	legacytok42 IN …content	Restart-legacy-probe.explain.txt

Conclusion

Legacy single-column FTS data written by CSE v26.3.8 remains valid on Spec v1 CSE/TiFlash:

1. Readable at every rolling step (TiFlash-only, CSE, TiDB).
2. Writable at every rolling step; visibility of insert/update/delete is correct.
3. Still queryable with the original fts_match_word(token, content) plan (textSearch on content only).
4. Coexists with new multi-column indexes and with a second FULLTEXT index on the same column.
5. Survives process restart.

No rebuild of old FTS files is required for this upgrade path.

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456.

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}

[JinheLin]

Please add compatible test result or report for legacy fts data.

@3pointer @yongman #11026 (comment)

[ti-chi-bot]

@3pointer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-tsan	bb8035c	link	false	/test pull-sanitizer-tsan

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, JinheLin, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JinheLin,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-21 03:24:30.14671536 +0000 UTC m=+212905.317809463: ☑️ agreed by JinheLin.
• 2026-08-25 11:21:30.31871351 +0000 UTC m=+587125.489807623: ☑️ agreed by yongman.