Problem
DoltLite supports application-defined collations registered through sqlite3_create_collation* for expressions and unindexed columns, but rejects persisted indexes whose key columns use one. This includes explicit indexes, UNIQUE constraints, and non-integer primary keys. A column with a custom collation can currently be indexed only by overriding the index expression with BINARY, NOCASE, or RTRIM.
Custom-collated indexes are not safe with the current prolly-tree key encoding. Prolly keys must have a stable byte ordering whenever the database is reopened, read by another process, committed, merged, or replicated. An application collation is a process-local callback: it may be absent after reopen, and an application may register different behavior under the same name. Persisting keys ordered by that callback would allow index searches and integrity checks to disagree with the stored order, and could make content hashes and merge behavior depend on the application currently opening the database.
The built-in BINARY, NOCASE, and RTRIM collations are supported because DoltLite owns their deterministic sort-key encoding.
Required design
Supporting custom-collated indexes needs a durable ordering contract rather than only a collation name. A design should address:
- deterministic, portable sort-key generation for custom collations;
- persisted collation identity and versioning;
- behavior when the collation is missing or its implementation does not match the persisted version;
- explicit indexes, implicit
UNIQUE indexes, and non-integer primary keys;
- index rebuilds when collation behavior changes;
- deterministic commits, merges, and replication across processes and machines;
- storage-format compatibility, including a format-version bump if prolly key encoding or another nested persisted format changes.
Acceptance criteria
- Custom-collated index creation succeeds only when its ordering can be reproduced safely after reopen.
- Index lookup, range scan, uniqueness enforcement, integrity checking, commit, merge, and replication remain deterministic.
- Missing or mismatched collation implementations fail clearly without silently returning incorrect results.
- Existing version 12 databases retain an explicit compatibility or migration policy.
Related: #1343 fixed a path that accidentally allowed an unsafe custom-collated implicit index.
Problem
DoltLite supports application-defined collations registered through
sqlite3_create_collation*for expressions and unindexed columns, but rejects persisted indexes whose key columns use one. This includes explicit indexes,UNIQUEconstraints, and non-integer primary keys. A column with a custom collation can currently be indexed only by overriding the index expression withBINARY,NOCASE, orRTRIM.Custom-collated indexes are not safe with the current prolly-tree key encoding. Prolly keys must have a stable byte ordering whenever the database is reopened, read by another process, committed, merged, or replicated. An application collation is a process-local callback: it may be absent after reopen, and an application may register different behavior under the same name. Persisting keys ordered by that callback would allow index searches and integrity checks to disagree with the stored order, and could make content hashes and merge behavior depend on the application currently opening the database.
The built-in
BINARY,NOCASE, andRTRIMcollations are supported because DoltLite owns their deterministic sort-key encoding.Required design
Supporting custom-collated indexes needs a durable ordering contract rather than only a collation name. A design should address:
UNIQUEindexes, and non-integer primary keys;Acceptance criteria
Related: #1343 fixed a path that accidentally allowed an unsafe custom-collated implicit index.