feat(clickhouse): deepen text-index / full-text-search testing (6 units)

Extends the existing TextIndexLike + JsonSkipIndex coverage toward the full
ClickHouse text-index surface. Validated on dev-vm head 26.6.1.734: a 1-hour
full-fleet run (167k queries, all ~45 oracles) produced 0 false positives from
any of these units. Soundness rules below were established empirically there.

ClickHouse#3 Pure-optimization toggle arm: TextIndexLike gains a DIRECT_READ_OFF arm
   (query_plan_direct_read_from_text_index=0, query_plan_text_index_add_hint=0).
   A pure optimization toggle must never change results; UNKNOWN_SETTING is
   already tolerated so it degrades on older builds.

ClickHouse#4 Merge + delete-masked-part topology: TextIndexLike optionally issues a
   synchronous lightweight DELETE (lightweight_deletes_sync=2) and/or OPTIMIZE
   TABLE ... FINAL after the inserts, recording topology in assertion messages.
   The Java ground truth now counts over the live (non-deleted) rows. Targets
   the #107309 class (index honoring a delete-masked part).

ClickHouse#6 General-fleet predicate injection: generateTextSearchPredicate emits
   startsWith / endsWith / multiSearchAny over plain String columns from a fixed
   vocabulary, gated by --text-search-predicate-emission (default on). These
   three are the functions proven index==scan-equivalent across ALL tokenizers
   (incl. array). hasToken/hasAllTokens/hasAnyTokens are deliberately NOT emitted
   here: hasToken diverges index-vs-scan on the array tokenizer (whole-value
   token) and hasAllTokens/hasAnyTokens diverge on ngrams (the multi-token
   needle's space-spanning grams) -- by-design tokenizer-semantics differences,
   not bugs. Sound for NoREC because renderSkipIndex never carries a preprocessor.

#7a renderSkipIndex breadth: full tokenizer matrix (splitByNonAlpha, ngrams(N),
   array, asciiCJK, splitByString, sparseGrams) and text() eligibility extended
   to FixedString / LowCardinality(String) / Nullable(String) / Array(String)
   (bare) and Map(String,*) via mapKeys/mapValues expression indexes. New
   ClickHouseErrors.getTextIndexErrors() tolerances wired into table creation.

ClickHouse#5 TextIndexPreprocessor oracle (NEW): builds a private table with
   INDEX(s) preprocessor=lower(s), then asserts that a forced direct read
   (force_data_skipping_indices + direct_read=1, add_hint=0) over a mixed-case
   corpus equals a Java lower()-token-membership ground truth. (The documented
   INDEX(lower(s)) equivalent form cannot be force-engaged for hasToken(s,...) on
   26.6.1.734 -- it raises INDEX_NOT_USED -- so the ground-truth comparison
   replaces a second-table comparison.)

#7b TextIndexContainer oracle (NEW): Array(String) + array tokenizer with exact
   has/hasAny/hasAll List ground truth, and Map(String,String) key-vs-value
   isolation via mapContainsKey/mapContainsValue, across DEFAULT /
   ignore_data_skipping_indices / use_skip_indexes=0 arms.

ALTER lifecycle: ClickHouseAlterGenerator gains ADD_INDEX / MATERIALIZE_INDEX /
   CLEAR_INDEX / DROP_INDEX kinds. New ClickHouseTextIndexLifecycle oracle
   asserts CREATE-with-index == (index-free + ALTER ADD INDEX + MATERIALIZE
   INDEX SETTINGS mutations_sync=2) == use_skip_indexes=0 scan over a predicate
   battery (LIKE + hasToken always; hasAllTokens/hasAnyTokens only on the
   splitByNonAlpha arm, where they are index==scan-equivalent), exercising the
   distinct MATERIALIZE-over-historical-parts code path.

All three new oracles registered in ClickHouseOracleFactory and the run /
validity oracle-name lists. All oracles use private per-iteration tables, fixed
token vocabulary, and integer-key/count comparisons to stay clear of the repo's
known false-positive families.

Author: fm4v

.claude/collect-oracle-validity.sh

@@ -20,7 +20,7 @@ THREADS="${THREADS:-6}"
 OUTDIR="$ROOT/val"
 
 # Full current oracle set (from run-sqlancer.sh ALL_ORACLES).
-IFS=',' read -r -a ORACLES <<< "TLPWhere,TLPDistinct,TLPGroupBy,TLPAggregate,TLPHaving,NoREC,PQS,CERT,CODDTest,SEMR,SEMRMulti,EET,SetOpTLP,CombinatorTLP,QccCache,SortedUnionLimitBy,SchemaRoundtrip,JoinAlgorithm,Cast,Parallelism,PartitionMirror,KeyCondition,TableFunctionIN,ViewEquivalence,AggregateStateRoundtrip,MaterializedViewConsistency,FinalMerge,ProjectionToggle,PatchPartConsistency,DictGetVsJoin,WindowEquivalence,DynamicSubcolumn,SubqueryMaterialize,MutationAnalyzer,TextIndexLike,TopK,JoinReorder,NaturalJoin,JsonSkipIndex,MaterializedCte,StatsToggle,ExtendedDatetime,JoinUseNulls,QueryCache"
+IFS=',' read -r -a ORACLES <<< "TLPWhere,TLPDistinct,TLPGroupBy,TLPAggregate,TLPHaving,NoREC,PQS,CERT,CODDTest,SEMR,SEMRMulti,EET,SetOpTLP,CombinatorTLP,QccCache,SortedUnionLimitBy,SchemaRoundtrip,JoinAlgorithm,Cast,Parallelism,PartitionMirror,KeyCondition,TableFunctionIN,ViewEquivalence,AggregateStateRoundtrip,MaterializedViewConsistency,FinalMerge,ProjectionToggle,PatchPartConsistency,DictGetVsJoin,WindowEquivalence,DynamicSubcolumn,SubqueryMaterialize,MutationAnalyzer,TextIndexLike,TopK,JoinReorder,NaturalJoin,JsonSkipIndex,MaterializedCte,StatsToggle,ExtendedDatetime,JoinUseNulls,QueryCache,TextIndexPreprocessor,TextIndexContainer,TextIndexLifecycle"
 
 rm -rf "$OUTDIR"; mkdir -p "$OUTDIR"
 

.claude/oracle-validity-workflow.js

@@ -16,7 +16,7 @@ const ORACLES = [
   'FinalMerge', 'ProjectionToggle', 'PatchPartConsistency', 'DictGetVsJoin', 'WindowEquivalence',
   'DynamicSubcolumn', 'SubqueryMaterialize', 'MutationAnalyzer', 'TextIndexLike', 'TopK', 'JoinReorder',
   'NaturalJoin', 'JsonSkipIndex', 'MaterializedCte', 'StatsToggle', 'ExtendedDatetime', 'JoinUseNulls',
-  'QueryCache',
+  'QueryCache', 'TextIndexPreprocessor', 'TextIndexContainer', 'TextIndexLifecycle',
 ]
 
 const VERDICT_SCHEMA = {

.claude/run-sqlancer.sh

@@ -35,7 +35,7 @@ EXTRA_CH_ARGS=""
 # 26.x coverage oracles (TextIndexLike..StatsToggle) appended 2026-06-10 after their convergence
 # run: 3h x 41 oracles x 1.09M queries with --eet-26x-modes/--variant-where-emission on produced
 # 0 false positives and 1 genuine CH wrong-result (JoinReorder, ANTI/SEMI/INNER chain).
-ALL_ORACLES="TLPWhere,TLPDistinct,TLPGroupBy,TLPAggregate,TLPHaving,NoREC,PQS,CERT,CODDTest,SEMR,SEMRMulti,EET,SetOpTLP,CombinatorTLP,QccCache,SortedUnionLimitBy,SchemaRoundtrip,JoinAlgorithm,Cast,Parallelism,PartitionMirror,KeyCondition,TableFunctionIN,ViewEquivalence,AggregateStateRoundtrip,MaterializedViewConsistency,FinalMerge,ProjectionToggle,PatchPartConsistency,DictGetVsJoin,WindowEquivalence,DynamicSubcolumn,SubqueryMaterialize,MutationAnalyzer,TextIndexLike,TopK,JoinReorder,NaturalJoin,JsonSkipIndex,MaterializedCte,StatsToggle,ExtendedDatetime,JoinUseNulls,QueryCache"
+ALL_ORACLES="TLPWhere,TLPDistinct,TLPGroupBy,TLPAggregate,TLPHaving,NoREC,PQS,CERT,CODDTest,SEMR,SEMRMulti,EET,SetOpTLP,CombinatorTLP,QccCache,SortedUnionLimitBy,SchemaRoundtrip,JoinAlgorithm,Cast,Parallelism,PartitionMirror,KeyCondition,TableFunctionIN,ViewEquivalence,AggregateStateRoundtrip,MaterializedViewConsistency,FinalMerge,ProjectionToggle,PatchPartConsistency,DictGetVsJoin,WindowEquivalence,DynamicSubcolumn,SubqueryMaterialize,MutationAnalyzer,TextIndexLike,TopK,JoinReorder,NaturalJoin,JsonSkipIndex,MaterializedCte,StatsToggle,ExtendedDatetime,JoinUseNulls,QueryCache,TextIndexPreprocessor,TextIndexContainer,TextIndexLifecycle"
 
 usage() {
   cat <<EOF

src/sqlancer/clickhouse/ClickHouseErrors.java

@@ -190,13 +190,26 @@ public static List<String> getAlterErrors() {
                 "EMPTY_LIST_OF_COLUMNS_QUERIED",
 
                 "Projection with name", "NO_SUCH_PROJECTION_IN_TABLE", "ILLEGAL_PROJECTION",
-                "Projection is fully supported", "projection", "Cannot add projection");
+                "Projection is fully supported", "projection", "Cannot add projection",
+
+                "Cannot add index", "index with name", "NO_SUCH_DATA_PART", "already exists");
     }
 
     public static void addAlterErrors(ExpectedErrors errors) {
         errors.addAll(getAlterErrors());
     }
 
+    public static List<String> getTextIndexErrors() {
+        return List.of("Unknown Index type", "Unknown index type", "Unknown tokenizer",
+                "Unexpected text index arguments", "full-text index", "full_text_index", "SUPPORT_IS_DISABLED",
+                "INCORRECT_QUERY", "ILLEGAL_TYPE_OF_ARGUMENT", "ILLEGAL_INDEX", "BAD_ARGUMENTS",
+                "data type must be String", "is not supported by text index", "of text index");
+    }
+
+    public static void addTextIndexErrors(ExpectedErrors errors) {
+        errors.addAll(getTextIndexErrors());
+    }
+
     public static List<String> getMutationErrors() {
         return List.of("TIMEOUT_EXCEEDED", "Cannot UPDATE key column", "Cannot DELETE", "Mutation cannot be executed",
 

src/sqlancer/clickhouse/ClickHouseOptions.java

@@ -59,6 +59,9 @@ public class ClickHouseOptions implements DBMSSpecificOptions<ClickHouseOracleFa
     @Parameter(names = "--variant-where-emission", description = "Emit Variant-typed predicate fragments in WHERE context (26.1 Variant-in-all-functions surface, PR #90900 + use_variant_as_common_type default-on, PR #90677). WHERE-only by design: the client-v2 RowBinary reader cannot decode a projected Variant column (R4), so the fragments are self-contained Boolean expressions and never reach a fetch column. Default-on since the 2026-06-10 convergence run (0 reader deaths, 0 false positives; the toInt64 constant-fallback wrap is load-bearing).", arity = 1)
     public boolean variantWhereEmission = true;
 
+    @Parameter(names = "--text-search-predicate-emission", description = "Emit full-text-search predicates (hasToken/hasAllTokens/hasAnyTokens/startsWith/endsWith/multiSearchAny) over plain String columns in general WHERE context, from a fixed token vocabulary. Lets the whole oracle fleet (TLPWhere/NoREC/CODDTest/...) incidentally differential-test text-indexed columns against full scans. Sound for NoREC because the general schema never carries a preprocessor on a text index (index path == scan path for preprocessor-free hasToken); preprocessor coverage lives only in the dedicated TextIndexPreprocessor oracle.", arity = 1)
+    public boolean textSearchPredicateEmission = true;
+
     @Parameter(names = "--join-reorder-allow-dropped-key-ref", description = "Let the JoinReorder oracle build ON clauses that reference a key column dropped by a preceding SEMI/ANTI join. Default false, PERMANENTLY: ClickHouse#107073 was closed by the optimizer team as by-design non-determinism -- columns read from the eliminated side of a SEMI/ANTI join are ANY-like (filled from whichever matching row arrives first), so any plan change or physical row-order change legally flips the result and a differential oracle comparing such queries is unsound. The restriction is therefore a soundness rule, not a temporary known-bug pin. Set true only to demonstrate the documented non-determinism.", arity = 1)
     public boolean joinReorderAllowDroppedKeyRef = false;
 

src/sqlancer/clickhouse/ClickHouseOracleFactory.java

@@ -24,7 +24,10 @@
 import sqlancer.clickhouse.oracle.mutate.ClickHouseMutationAnalyzerOracle;
 import sqlancer.clickhouse.oracle.patch.ClickHousePatchPartConsistencyOracle;
 import sqlancer.clickhouse.oracle.stats.ClickHouseStatsToggleOracle;
+import sqlancer.clickhouse.oracle.textindex.ClickHouseTextIndexContainerOracle;
+import sqlancer.clickhouse.oracle.textindex.ClickHouseTextIndexLifecycleOracle;
 import sqlancer.clickhouse.oracle.textindex.ClickHouseTextIndexLikeOracle;
+import sqlancer.clickhouse.oracle.textindex.ClickHouseTextIndexPreprocessorOracle;
 import sqlancer.clickhouse.oracle.topk.ClickHouseTopKOracle;
 import sqlancer.clickhouse.oracle.partition.ClickHousePartitionMirrorOracle;
 import sqlancer.clickhouse.oracle.pqs.ClickHousePivotedQuerySynthesisOracle;
@@ -357,5 +360,26 @@ public TestOracle<ClickHouseGlobalState> create(ClickHouseGlobalState globalStat
         public TestOracle<ClickHouseGlobalState> create(ClickHouseGlobalState globalState) throws SQLException {
             return new ClickHouseStatsToggleOracle(globalState);
         }
+    },
+    TextIndexPreprocessor {
+
+        @Override
+        public TestOracle<ClickHouseGlobalState> create(ClickHouseGlobalState globalState) throws SQLException {
+            return new ClickHouseTextIndexPreprocessorOracle(globalState);
+        }
+    },
+    TextIndexContainer {
+
+        @Override
+        public TestOracle<ClickHouseGlobalState> create(ClickHouseGlobalState globalState) throws SQLException {
+            return new ClickHouseTextIndexContainerOracle(globalState);
+        }
+    },
+    TextIndexLifecycle {
+
+        @Override
+        public TestOracle<ClickHouseGlobalState> create(ClickHouseGlobalState globalState) throws SQLException {
+            return new ClickHouseTextIndexLifecycleOracle(globalState);
+        }
     }
 }

src/sqlancer/clickhouse/gen/ClickHouseAlterGenerator.java

@@ -2,6 +2,8 @@
 
 import java.util.List;
 
+import com.clickhouse.data.ClickHouseDataType;
+
 import sqlancer.IgnoreMeException;
 import sqlancer.Randomly;
 import sqlancer.clickhouse.ClickHouseErrors;
@@ -18,7 +20,8 @@ private ClickHouseAlterGenerator() {
     }
 
     private enum AlterKind {
-        ADD_COLUMN, DROP_COLUMN, MODIFY_COLUMN, RENAME_COLUMN, COMMENT_COLUMN, ADD_PROJECTION
+        ADD_COLUMN, DROP_COLUMN, MODIFY_COLUMN, RENAME_COLUMN, COMMENT_COLUMN, ADD_PROJECTION, ADD_INDEX,
+        MATERIALIZE_INDEX, CLEAR_INDEX, DROP_INDEX
     }
 
     public static SQLQueryAdapter getQuery(ClickHouseGlobalState state) {
@@ -51,12 +54,25 @@ public static SQLQueryAdapter getQuery(ClickHouseGlobalState state) {
         case ADD_PROJECTION:
             renderAddProjection(sb, table);
             break;
+        case ADD_INDEX:
+            renderAddIndex(sb, table);
+            break;
+        case MATERIALIZE_INDEX:
+            renderMaterializeIndex(sb, table);
+            break;
+        case CLEAR_INDEX:
+            renderClearIndex(sb, table);
+            break;
+        case DROP_INDEX:
+            renderDropIndex(sb, table);
+            break;
         default:
             throw new AssertionError(kind);
         }
 
         ExpectedErrors errors = ExpectedErrors.newErrors().with(ClickHouseErrors.getExpectedExpressionErrors())
-                .with(ClickHouseErrors.getAlterErrors()).build();
+                .with(ClickHouseErrors.getAlterErrors()).with(ClickHouseErrors.getTextIndexErrors())
+                .with(ClickHouseErrors.getMutationErrors()).build();
 
         return new SQLQueryAdapter(sb.toString(), errors, true);
     }
@@ -128,6 +144,52 @@ private static void renderAddProjection(StringBuilder sb, ClickHouseTable table)
         sb.append(")");
     }
 
+    private static void renderAddIndex(StringBuilder sb, ClickHouseTable table) {
+        ClickHouseColumn strCol = pickStringColumn(table);
+        if (strCol != null) {
+            sb.append(" ADD INDEX IF NOT EXISTS tidx_").append(strCol.getName()).append(" (").append(strCol.getName())
+                    .append(") TYPE text(tokenizer = ").append(pickTokenizer()).append(") GRANULARITY 1");
+        } else {
+            ClickHouseColumn col = Randomly.fromList(table.getColumns());
+            sb.append(" ADD INDEX IF NOT EXISTS sidx_").append(col.getName()).append(" (").append(col.getName())
+                    .append(") TYPE set(100) GRANULARITY 1");
+        }
+    }
+
+    private static void renderMaterializeIndex(StringBuilder sb, ClickHouseTable table) {
+        sb.append(" MATERIALIZE INDEX IF EXISTS ").append(pickIndexName(table));
+    }
+
+    private static void renderClearIndex(StringBuilder sb, ClickHouseTable table) {
+        sb.append(" CLEAR INDEX IF EXISTS ").append(pickIndexName(table));
+    }
+
+    private static void renderDropIndex(StringBuilder sb, ClickHouseTable table) {
+        sb.append(" DROP INDEX IF EXISTS ").append(pickIndexName(table));
+    }
+
+    private static String pickIndexName(ClickHouseTable table) {
+        ClickHouseColumn strCol = pickStringColumn(table);
+        if (strCol != null) {
+            return "tidx_" + strCol.getName();
+        }
+        return "sidx_" + Randomly.fromList(table.getColumns()).getName();
+    }
+
+    private static ClickHouseColumn pickStringColumn(ClickHouseTable table) {
+        List<ClickHouseColumn> strCols = new java.util.ArrayList<>();
+        for (ClickHouseColumn col : table.getColumns()) {
+            if (col.getType().getType() == ClickHouseDataType.String) {
+                strCols.add(col);
+            }
+        }
+        return strCols.isEmpty() ? null : Randomly.fromList(strCols);
+    }
+
+    private static String pickTokenizer() {
+        return Randomly.fromOptions("'splitByNonAlpha'", "ngrams(3)", "'array'", "splitByString([' '])");
+    }
+
     private static void renderCommentColumn(StringBuilder sb, ClickHouseTable table) {
         ClickHouseColumn col = Randomly.fromList(table.getColumns());
 

src/sqlancer/clickhouse/gen/ClickHouseExpressionGenerator.java

@@ -423,6 +423,38 @@ public ClickHouseExpression generateStringCall(List<ClickHouseColumnReference> c
         return new sqlancer.clickhouse.ast.ClickHouseRawText(sql);
     }
 
+    private static final List<String> TEXT_SEARCH_VOCABULARY = List.of("alpha", "bravo", "charlie", "delta", "echo",
+            "foxtrot", "golf", "hotel", "india", "juliet", "clickhouse", "olap", "search", "token", "index", "query");
+
+    public ClickHouseExpression generateTextSearchPredicate(List<ClickHouseColumnReference> columns) {
+        List<ClickHouseColumnReference> stringCols = new java.util.ArrayList<>();
+        for (ClickHouseColumnReference c : columns) {
+            if (c.getColumn().getType().getType() == ClickHouseDataType.String) {
+                stringCols.add(c);
+            }
+        }
+        if (stringCols.isEmpty()) {
+            return null;
+        }
+        String s = ClickHouseToStringVisitor.asString(Randomly.fromList(stringCols));
+        String w1 = TEXT_SEARCH_VOCABULARY.get((int) Randomly.getNotCachedInteger(0, TEXT_SEARCH_VOCABULARY.size()));
+        String w2 = TEXT_SEARCH_VOCABULARY.get((int) Randomly.getNotCachedInteger(0, TEXT_SEARCH_VOCABULARY.size()));
+        String fn = Randomly.fromOptions("startsWith", "endsWith", "multiSearchAny");
+        String sql;
+        switch (fn) {
+        case "startsWith":
+            sql = "startsWith(" + s + ", '" + w1 + "')";
+            break;
+        case "endsWith":
+            sql = "endsWith(" + s + ", '" + w1 + "')";
+            break;
+        default:
+            sql = "multiSearchAny(" + s + ", ['" + w1 + "', '" + w2 + "'])";
+            break;
+        }
+        return new sqlancer.clickhouse.ast.ClickHouseRawText(sql);
+    }
+
     public ClickHouseExpression generateDateTransform(List<ClickHouseColumnReference> columns) {
         List<ClickHouseColumnReference> dateCols = new java.util.ArrayList<>();
         boolean dateTimeResolution = false;
@@ -1307,6 +1339,16 @@ public ClickHouseExpression generatePredicate() {
             }
         }
 
+        if (globalState.getClickHouseOptions().textSearchPredicateEmission
+                && Randomly.getBooleanWithSmallProbability()) {
+            ClickHouseExpression textPred = generateTextSearchPredicate(columnRefs);
+            if (textPred != null) {
+                return Randomly.getBoolean() ? textPred
+                        : new ClickHouseBinaryLogicalOperation(base, textPred,
+                                ClickHouseBinaryLogicalOperation.ClickHouseBinaryLogicalOperator.AND);
+            }
+        }
+
         if (ClickHouseVariantPredicateFactory.gateOpen(globalState.getClickHouseOptions().variantWhereEmission,
                 Randomly.getBooleanWithSmallProbability())) {
             List<String> intExprs = integerColumns(columnRefs).stream()

src/sqlancer/clickhouse/gen/ClickHouseTableGenerator.java

@@ -52,6 +52,7 @@ public static SQLQueryAdapter createTableStatement(String tableName,
         chTableGenerator.start();
         ExpectedErrors errors = new ExpectedErrors();
         ClickHouseErrors.addExpectedExpressionErrors(errors);
+        ClickHouseErrors.addTextIndexErrors(errors);
         return new SQLQueryAdapter(chTableGenerator.sb.toString(), errors, true);
     }
 
@@ -399,25 +400,63 @@ private String renderProjection(int idx, List<ClickHouseSchema.ClickHouseColumn>
 
     private String renderSkipIndex(int idx, ClickHouseSchema.ClickHouseColumn col) {
         ClickHouseDataType t = col.getType().getType();
+        String textTarget = textIndexTarget(col);
 
-        List<String> typeChoices = new ArrayList<>();
-        typeChoices.add("bloom_filter(0.01)");
-        typeChoices.add("set(100)");
+        List<String[]> candidates = new ArrayList<>();
+        candidates.add(new String[] { col.getName(), "bloom_filter(0.01)" });
+        candidates.add(new String[] { col.getName(), "set(100)" });
         if (t == ClickHouseDataType.Int32 || t == ClickHouseDataType.String) {
-            typeChoices.add("minmax");
+            candidates.add(new String[] { col.getName(), "minmax" });
         }
         if (t == ClickHouseDataType.String) {
-            typeChoices.add("ngrambf_v1(3, 256, 2, 0)");
-
-            typeChoices.add("text(tokenizer = 'splitByNonAlpha')");
-            typeChoices.add("text(tokenizer = ngrams(3))");
+            candidates.add(new String[] { col.getName(), "ngrambf_v1(3, 256, 2, 0)" });
+        }
+        if (textTarget != null) {
+            candidates.add(new String[] { textTarget, "text(tokenizer = " + pickTokenizer() + ")" });
+            candidates.add(new String[] { textTarget, "text(tokenizer = " + pickTokenizer() + ")" });
         }
-        String type = Randomly.fromList(typeChoices);
-        int granularity = Randomly.fromOptions(1, 2, 4);
-        return String.format("INDEX idx_%s_%d %s TYPE %s GRANULARITY %d", col.getName(), idx, col.getName(), type,
+
+        String[] chosen = Randomly.fromList(candidates);
+        int granularity = chosen[1].startsWith("text(") ? 1 : Randomly.fromOptions(1, 2, 4);
+        return String.format("INDEX idx_%s_%d %s TYPE %s GRANULARITY %d", col.getName(), idx, chosen[0], chosen[1],
                 granularity);
     }
 
+    private static String pickTokenizer() {
+        return Randomly.fromOptions("'splitByNonAlpha'", "ngrams(2)", "ngrams(3)", "ngrams(4)", "'array'", "'asciiCJK'",
+                "splitByString([' '])", "splitByString([' ', '-', '::'])", "sparseGrams(3, 5)");
+    }
+
+    private static String textIndexTarget(ClickHouseSchema.ClickHouseColumn col) {
+        sqlancer.clickhouse.ClickHouseType u = col.getType().getTypeTerm().unwrap();
+        if (isStringLeaf(u)) {
+            return col.getName();
+        }
+        if (u instanceof sqlancer.clickhouse.ClickHouseType.Array arr && isStringLeaf(arr.inner().unwrap())) {
+            return col.getName();
+        }
+        if (u instanceof sqlancer.clickhouse.ClickHouseType.Map m) {
+            boolean keyStr = isStringLeaf(m.keyType().unwrap());
+            boolean valStr = isStringLeaf(m.valueType().unwrap());
+            if (keyStr && valStr) {
+                return Randomly.getBoolean() ? "mapKeys(" + col.getName() + ")" : "mapValues(" + col.getName() + ")";
+            }
+            if (keyStr) {
+                return "mapKeys(" + col.getName() + ")";
+            }
+            if (valStr) {
+                return "mapValues(" + col.getName() + ")";
+            }
+        }
+        return null;
+    }
+
+    private static boolean isStringLeaf(sqlancer.clickhouse.ClickHouseType t) {
+        return t instanceof sqlancer.clickhouse.ClickHouseType.Primitive p
+                && p.kind() == sqlancer.clickhouse.ClickHouseType.Kind.String
+                || t instanceof sqlancer.clickhouse.ClickHouseType.FixedString;
+    }
+
     private static final int CLAUSE_VALIDATION_RETRY_LIMIT = 5;
 
     private static ClickHouseExpression generateValidated(Supplier<ClickHouseExpression> factory,